生成时间: 2026-08-10 17:24:15 (UTC+8); Arxiv 发布时间: 2026-08-10 20:00 EDT (2026-08-11 08:00 UTC+8)

今天共有 30 篇相关文章

Keyword: reinforcement learning

Evaluating XAI Support From A Hierarchical Reinforcement Learning Policy in Human-Agent Collaboration

在人机协作中评估基于分层强化学习策略的XAI支持

• Authors: Mateus Levi Simões Fernandes, Alberto Sardinha
• Subjects: Subjects: Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.06381
• Pdf link: https://arxiv.org/pdf/2608.06381
• Abstract Explainable AI (XAI) has shown promise for human-agent collaboration, yet results rely on hand-crafted policies in custom environments, limiting generalizability to state-of-the-art teaming research. We provide the first systematic evaluation of XAI support generated from an intrinsically explainable learned policy in an established benchmark. Using the Hierarchical Ad Hoc Agents (HA$^2$) architecture in Overcooked-AI, we generate real-time explanations from hierarchical subtask selections, delivered through text or audio via a novel trigger-based system. Our between-subjects experiment (n=38) found no significant performance effects, though participants with explanations showed trends toward faster performance improvement. More notably, audio explanations produced a significant reduction in participants' working-alliance bond with the agent -- an effect absent under the text modality -- suggesting that spoken explanations activate partnership expectations the underlying reactive policy cannot meet. We provide the first modality comparison in real-time human-agent collaboration and establish a baseline methodology for evaluating intrinsically explainable reinforcement learning architectures in benchmark environments. Results point to matching explanation modality to the underlying policy's capacity of sustaining the partnership its delivery implies as a potential path for more effective collaborative XAI.
• 中文摘要 可解释人工智能（XAI）在人与代理协作方面展现出潜力，但结果依赖于定制环境中手工制定的策略，限制了对最先进团队研究的泛化性。我们首次系统评估了基于内在可解释的学习策略生成的XAI支持，且在既定基准测试中实现。利用Overcooked-AI中的层级临时代理（HA$^2$）架构，我们通过基于触发器的系统通过文本或音频，从层级子任务选择生成实时解释。我们的受试者间实验（n=38）未发现显著的性能影响，尽管接受解释的参与者表现出更快的绩效提升趋势。更值得注意的是，音频解释显著降低了参与者与代理的工作联盟关系——这是文本模态中不存在的效果——表明口头解释激活了底层反应策略无法满足的合作期望。我们首次提供了实时人主协作中的模态比较，并建立了评估基准环境中内在可解释强化学习架构的基线方法论。结果表明，解释模态与其实施所暗示的合作关系的基础策略能力相匹配，这可能成为更有效协作XAI的潜在路径。

WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader

WebGrader：使用自我演进程序化评分器训练大语言模型进行网页开发

• Authors: Boshui Chen, Huiping Liu, Shaolei Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.06474
• Pdf link: https://arxiv.org/pdf/2608.06474
• Abstract Large language models increasingly generate complete websites from natural-language descriptions, and reinforcement learning has become a central approach to closing their remaining functional gap. This training regime is bottlenecked by reward design. Hand-authored browser scripts are executable yet costly to write for open-ended requirements, while VLM and GUI-agent graders scale but may issue verdicts before observing the decisive state. We propose WebGrader, a self-evolving programmatic grader that autonomously derives the required interaction flows from each website request, represents each flow as an executable Flow Contract, and uses its execution outcome as an RL reward. WebGrader materializes the generated project in a live browser, grounds target actions against the source code and live DOM, and collects visual, DOM, response, and persistent-state evidence along the same browser trajectory. A residual-driven offline loop then discovers reusable verifier skills, screens them on disjoint validation pages, and freezes the promoted skill graph before policy training. By separating test planning, action grounding, evidence collection, and semantic judgment, WebGrader issues a Pass verdict only after observing the requested transition. On WebGen-Bench, WebGrader trains an 8B policy to a 52.01% functional success rate, outperforming a matched appearance-plus-script reward by 7.88 points and surpassing o4-mini and DeepSeek-v4-flash. On WG-core-250, the policy reaches a Full Score of 44.953 and surpasses Qwen3-Coder-480B.
• 中文摘要 大型语言模型越来越多地从自然语言描述生成完整网站，强化学习已成为弥补其剩余功能差距的核心方法。这种训练模式因奖励设计而成为瓶颈。手写浏览器脚本可执行但为开放式需求编写成本较高，而VLM和GUI-agent评分器可扩展，但可能在观察决定性状态前就下结论。我们提出WebGrader，一种自我进化的程序评分器，自主从每个网站请求中推导出所需的交互流，将每个流程表示为可执行的流程契约，并以其执行结果作为强化学习奖励。WebGrader将生成的项目具体化为实时浏览器，基于源代码和实时DOM将目标动作建立，并沿同一浏览器轨迹收集可视化、DOM响应和持久状态证据。残差驱动的离线循环随后发现可重复使用的验证者技能，在不相交的验证页面上筛选，并在策略训练前冻结晋升技能图。通过分离测试规划、动作基础、证据收集和语义判断，WebGrader仅在观察请求的转换后才发出通过判决。在WebGen-Bench上，WebGrader将8B策略训练至52.01%的功能成功率，比匹配外观加脚本奖励高出7.88分，超过o4-mini和DeepSeek-v4-flash。在WG-core-250中，策略达到44.953满分，超过Qwen3-Coder-480B。

Flowing Through States: Neural ODE Regularization for Reinforcement Learning

流过状态：强化学习中的神经常微分方程正则化

• Authors: Mohamed Ghanem, Bernd Finkbeiner
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.06595
• Pdf link: https://arxiv.org/pdf/2608.06595
• Abstract Neural networks applied to sequential decision-making tasks typically rely on latent representations of environment states. While environment dynamics dictate how semantic states evolve, the corresponding latent transitions are usually left implicit, creating a potential misalignment between the two. We propose to model latent dynamics explicitly by drawing an analogy between Markov decision process (MDP) trajectories and ordinary differential equation (ODE) flows: in both cases, the current state fully determines its successors. Building on this view, we introduce a neural ODE-based regularization method that enforces latent embeddings to follow consistent ODE flows, thereby aligning representation learning with environment dynamics. Although broadly applicable to deep learning agents, we demonstrate its effectiveness in reinforcement learning by integrating it into Actor-Critic algorithms. Our approach yields major performance gains across various standard Atari benchmarks for A2C and gridworld environments for PPO.
• 中文摘要 应用于顺序决策任务的神经网络通常依赖于环境状态的潜在表征。虽然环境动态决定了语义状态的演变，但相应的潜在转移通常隐含，导致两者之间可能出现不一致。我们提出通过将马尔可夫决策过程（MDP）轨迹与常微分方程（ODE）流进行类比，明确建模潜在动态：在这两种情况下，当前状态都完全决定了其后继状态。基于这一观点，我们引入了一种基于神经ODE的正则化方法，强制潜在嵌入遵循一致的常微分方程流，从而使表征学习与环境动态保持一致。虽然广泛适用于深度学习代理，我们通过将其集成到Actor-Critic算法中展示了其在强化学习中的有效性。我们的方法在A2C的Atari标准基准测试和PPO的网格世界环境中带来了显著的性能提升。

MARS: A Monte Carlo Tree Search-based Adaptive and Responsive Scheduler

MARS：基于蒙特卡洛树搜索的自适应响应调度器

• Authors: Yash Kurkure, Yihe Zhang, Zhiling Lan, Michael E. Papka
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2608.06629
• Pdf link: https://arxiv.org/pdf/2608.06629
• Abstract Modern High Performance Computing systems depend on static heuristics and manual administration for job scheduling and reservation management. Deep Reinforcement Learning (DRL) has shown promising scheduling performance but requires historical training data and fixes the optimization goal at training time, forcing operators to retrain whenever priorities shift. We introduce MARS (Monte Carlo Tree Search-based Adaptive and Responsive Scheduler), a training-free HPC scheduler whose optimization goal is configurable through a reward function rather than baked into a learned model. MARS uses a lightweight discrete-event simulator to explore the future consequences of scheduling decisions within a strict time budget, adapting to the configured reward at each scheduling cycle. We evaluate MARS on year-long production workloads from two systems at Argonne Leadership Computing Facility -- 4,360-node Theta and 560-node Polaris---under two reward functions: wait-time minimization (MARS-CW) and utilization maximization (MARS-CU). Unlike DRL and heuristics, which only react to the current queue or wait for backfill to find holes, MARS exploits look-ahead to proactively drain the system and plan around future reservations, packing the system to avoid the fragmentation and utilization drop that typically precede reservation windows. MARS-CW reduces tail wait time by 64% on Theta and 43% on Polaris over the production WFP heuristic, while MARS-CU recovers utilization in the 48 hours leading into maintenance, demonstrating that MARS can target either objective via reward reconfiguration.
• 中文摘要 现代高性能计算系统依赖静态启发式和手动管理来进行作业调度和预约管理。深度强化学习（DRL）显示出有前景的调度性能，但需要历史训练数据，并在训练时固定优化目标，迫使操作员在优先级变化时重新训练。我们介绍了MARS（基于蒙特卡洛树搜索的自适应响应式调度器），这是一种无需训练的高性能计算调度器，其优化目标可通过奖励函数配置，而非内置于学习模型中。MARS使用轻量离散事件模拟器，在严格的时间预算内探索调度决策的未来影响，并根据每个调度周期的配置奖励进行调整。我们在阿贡领导计算设施的两个系统——4360节点的Theta和560节点的Polaris——的全年生产工作负载上评估MARS的运行---采用两种奖励函数：等待时间最小化（MARS-CW）和利用最大化（MARS-CU）。与仅对当前队列做出反应或等待回填寻找漏洞的DRL和启发式不同，MARS利用前瞻性功能主动耗尽系统资源，围绕未来预约进行规划，将系统打包以避免通常在预约窗口前出现的碎片化和利用率下降。MARS-CW在Theta上将尾部等待时间缩短64%，Polaris在生产型WFP启发式上减少43%，而MARS-CU则在维护前48小时内恢复利用率，证明MARS可以通过奖励重配置针对任一目标。

Vehicle routing problem using deep reinforcement learning - A case study about truck planning in the industry

利用深度强化学习的车辆路由问题——关于行业卡车规划的案例研究

• Authors: Siliang Lu, Dan Hu, Lili Wu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.06668
• Pdf link: https://arxiv.org/pdf/2608.06668
• Abstract As an important component of the supply chain industry, transportation has experienced rapid development in the past decade with the assistance of digital platforms and intelligent algorithms. Within the field of transportation research, Vehicle Routing Problem (VRP) has remained a persistent and enduring challenge. In the realm of management science, experts, and scholars from both the industrial and academic sectors have continuously explored optimization models and algorithms to effectively address routing problems, from the classical Traveling Salesman Problem to the more general Vehicle Routing Problem. These models and algorithms are applied in real-world industrial scenarios to achieve cost optimization and reduce carbon footprints. However, due to the complexity of real-world problems, numerous specific constraints are often added, and challenges such as information opacity, uncertainty, and irrational human behavior may arise. Therefore, deploying and optimizing mathematical models for VRP in practical scenarios while maintaining optimal results poses numerous challenges. This paper discusses and provides solutions for three different logistic use cases involving external truck network design. Through these industrial case study, the paper introduces how deep reinforcement learning-based vehicle routing optimization has been implemented. As a result, it can be observed that the routes optimized by reinforcement learning agent have over 10% total cost compared to baseline results. Furthermore, the paper proposes that in future research, DRL algorithms for vehicle routing problems could be generalized into more variations of VRP.
• 中文摘要 作为供应链行业的重要组成部分，交通在过去十年中借助数字平台和智能算法实现了快速发展。在交通研究领域，车辆路由问题（VRP）一直是持续且持久的挑战。在管理科学领域，来自工业和学术领域的专家和学者不断探索优化模型和算法，以有效解决路由问题，从经典的推销员问题到更广泛的车辆路径问题。这些模型和算法被应用于现实工业场景，以实现成本优化和减少碳足迹。然而，由于现实问题的复杂性，常常会加入许多具体约束，并可能出现信息不透明性、不确定性和非理性人类行为等挑战。因此，在实际场景中部署和优化VRP数学模型，同时保持最佳结果，带来了诸多挑战。本文讨论并提供了三种涉及外部卡车网络设计的物流应用场景的解决方案。通过这些工业案例研究，本文介绍了基于深度强化学习的车辆路由优化是如何实现的。因此，可以观察到强化学习代理优化的路线总成本相较基线结果超过10%。此外，论文提出未来研究中，车辆路由问题的DRL算法可以推广为更多VRP的变体。

IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agents

IB-RL：战略对话代理的孤立双边强化学习

• Authors: Senhao Wang, Chenghao Cai, Haitao Hu, Mingxing Huang, Xingguang Wang, Wenhao Li, Zecheng Lin
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.06735
• Pdf link: https://arxiv.org/pdf/2608.06735
• Abstract Reinforcement learning (RL) has achieved strong results in improving large language models (LLMs) on tasks with stationary, verifiable rewards, such as mathematical reasoning and code execution. In these settings, the environment follows fixed rules and does not adapt strategically to the agent. Strategic dialogue differs in this respect: the environment is another agent that adapts to the policy, and success depends on the interaction between the two sides. Despite this interactive nature, current RL approaches typically train a target agent against a fixed counterpart or simulator. We find that this training paradigm encourages the policy to exploit counterpart-specific regularities rather than learn strategies that generalize across counterparts. We call this problem the static-counterpart mismatch, which we quantify directly in our experiments. To address it, we propose Isolated Bilateral Reinforcement Learning (IB-RL), in which the two roles coevolve through joint rollouts while each role optimizes its own reward through fully independent advantages, action masks, and update paths. We evaluate frozen policies against fully independent held-out counterparts in both domains. On Vehicle TeleSales, IB-RL achieves 89.6% Success@1, compared to 84.6% for the best unilateral RL baseline. On Deal-or-NoDeal, it reaches 98.4% agreement against DeepSeek V4 Pro, compared to 86.4% for the best unilateral baseline. These results indicate that jointly training both roles with strict peragent isolation produces policies that generalize more effectively to unseen counterparts.
• 中文摘要 强化学习（RL）在改进大型语言模型（LLMs）方面取得了显著成效，适用于具有固定且可验证奖励的任务，如数学推理和代码执行。在这些环境中，环境遵循固定规则，不会战略性地适应智能体。战略对话在这方面不同：环境是另一个适应策略的智能体，成功取决于双方的互动。尽管具有这种交互性质，当前的强化学习方法通常将目标智能体与固定对应者或模拟器对抗训练。我们发现，这种训练范式鼓励策略利用对应者的特定规律性，而非学习跨对应者推广的策略。我们将此问题称为静态对应不匹配，并在实验中直接量化。为此，我们提出了孤立双边强化学习（IB-RL），其中两个角色通过联合推广共同进化，同时各自通过完全独立的优势、动作掩码和更新路径优化自身奖励。我们评估了冻结政策与两个领域中完全独立且未完成的对应策略。在车辆电话销售方面，IB-RL实现了89.6%的Success@1，而最佳单边RL基线为84.6%。在Deal-or-NoDeal中，IB-RL对DeepSeek V4 Pro的认可度为98.4%，而最佳单边基线为86.4%。这些结果表明，联合训练并严格隔离两个角色，能产生更有效地推广到未见对应角色的政策。

Capek 0.5: An Execution-Centric Vision-Language Model for Embodied Intelligence

Capek 0.5：以执行为中心的具身智能视觉语言模型

• Authors: Ying Chen, Weizhen Li, Zhe Hu, Zhenjiang Li, Rui Jiang, Zhifeng Gu, Lihuang Fang, Jiangping Liu, Lei Yi, Jie Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.06756
• Pdf link: https://arxiv.org/pdf/2608.06756
• Abstract Vision-language models are increasingly serving as the reasoning core of embodied agents. Robot execution is inherently iterative: each action reshapes the scene and physical state, continually renewing what must be perceived, reasoned about, and verified. Meeting these demands requires complementary capabilities that differ in supervision signals, prediction formats, and verification criteria. Existing approaches typically develop these capabilities against isolated, task-specific objectives, leaving open how they should be organized and integrated around execution as a whole. We present Capek 0.5, an embodied vision-language model built around an execution-centric capability taxonomy. Rather than organizing training by datasets or tasks, the taxonomy groups embodied capabilities according to their functional roles throughout execution and comprises four capability families: Spatial Reasoning, Temporal Understanding, Action Guidance, and State Verification. Each capability is first acquired by a dedicated specialist through reinforcement learning with verifiable rewards from a shared backbone, and the specialists are then consolidated into a single inference-time model through weight-space merging followed by routed policy-space distillation. We instantiate Capek 0.5 at the 2B and 35B-A3B scales and evaluate it from three complementary perspectives: comprehensive benchmark suites including Capek-StateBench, a new benchmark for state verification; a controlled study of capability retention from specialists to the unified model; and closed-loop evaluation in simulated embodied environments. Capek 0.5 improves the large majority of matched benchmark rows over its initialization, retains all four specialized capabilities in one checkpoint with quantified losses, and transfers to closed-loop embodied task execution.
• 中文摘要 视觉语言模型正日益成为具身代理的推理核心。机器人执行本质上是迭代的：每一个动作都会重塑场景和物理状态，不断更新必须感知、推理和验证的内容。满足这些需求需要在监督信号、预测格式和验证标准上不同的互补能力。现有方法通常针对孤立的任务特定目标开发这些能力，留下围绕整体执行组织和整合的开放性。我们介绍Capek 0.5，一种围绕执行为中心能力分类法构建的具身视觉语言模型。这些分类组不是按数据集或任务组织训练，而是根据执行过程中的功能角色体现能力，包含四个能力族：空间推理、时间理解、动作指导和状态验证。每个能力首先由专属专家通过强化学习获得，并可验证的共享骨干奖励，随后通过权重空间合并和路由策略空间蒸馏，整合到单一推理时间模型中。我们在2B和35B-A3B尺度上实现Capek 0.5，并从三个互补视角评估：包括Capek-StateBench在内的综合基准套件，这是状态验证的新基准;专家对统一模型的能力保留度进行受控研究;以及模拟内涵环境中的闭环评估。Capek 0.5在初始化过程中改进了绝大多数匹配基准行，保留了四种专业能力并实现量化损耗，并转移至闭环内涵任务执行。

Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models

探索还是收敛？扩散模型的阶段引导逐步优化

• Authors: Renye Yan, Jikang Cheng, You Wu, Wei Peng, Zongwei Wang, Ling Liang, Yimao Cai
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.06768
• Pdf link: https://arxiv.org/pdf/2608.06768
• Abstract Diffusion models have strong generative capabilities. However, their maximum likelihood training objective only focuses on reconstructing the data distribution, making it difficult to align with specific preferences. Reinforcement learning (RL) for preference alignment in diffusion models is promising but limited by reward sparsity. Since a single reward cannot support optimization, existing RL methods usually backpropagate the final reward to all previous steps. However, denoising is stage-wise, with distinct semantics and controllability. Repeating the final reward across all steps creates a temporal objective mismatch, encouraging reward shortcuts that lead to reward hacking. At the same time, due to reward backfilling, each time step receives the same reward, making it impossible to distinguish between actions, thereby weakening the optimization process. To resolve this issue, we propose Stage-Guided Per-Step Optimization (SGPO) for diffusion models, which jointly leverages signal-to-noise ratio and semantic changes to identify generation stages and adaptively assign stage-specific objectives. Early denoising is chaotic and far from the final reward, resulting in weak reward-behavior correlation. This stage should prioritize exiting the chaotic state. In the mid stage, the latent transitions to a stable structure, where the final reward better corresponds to generative behavior. Therefore, this stage optimizes the final reward while exploring diversity to avoid early convergence to a single mode. In the late stage, the latent's core structure is largely fixed, and preference optimization mainly amplifies local details, risking overfitting. Therefore, stable convergence is preferred to avoid quality degradation. Results from 16 comparative experiments validate SGPO. Our method achieves 26.7% average gains in generative quality and 36.7% higher convergence speed.
• 中文摘要 扩散模型具有强大的生成能力。然而，其最大似然训练目标仅聚焦于重建数据分布，难以与特定偏好对齐。扩散模型中偏好对齐的强化学习（RL）前景看好，但受限于奖励稀疏性。由于单个奖励无法支持优化，现有的强化学习方法通常将最终奖励反向传播到之前所有步骤。然而，去噪是分阶段进行的，语义和可控性各异。在所有步骤重复最终奖励会导致时间客观不匹配，助长奖励捷径，导致奖励黑客行为。同时，由于奖励回填，每个时间步获得相同的奖励，使得无法区分动作，从而削弱优化过程。为解决这一问题，我们提出了阶段引导每步优化（SGPO）用于扩散模型，该方法结合信噪比和语义变化来识别生成阶段并自适应地分配阶段特定目标。早期去噪是混沌的，且距离最终奖励相距甚远，导致奖励-行为相关性较弱。该阶段应优先退出混沌状态。中期阶段，潜能转变为稳定结构，最终奖励更符合生成行为。因此，该阶段在探索多样性的同时优化最终奖励，避免过早趋同单一模式。在后期阶段，潜在因素的核心结构基本固定，偏好优化主要放大局部细节，存在过拟合风险。因此，优先选择稳定收敛以避免质量下降。16个比较实验的结果验证了SGPO。我们的方法在生成质量上实现了26.7%的平均提升，收敛速度提升了36.7%。

Retrieval-Constrained Policy Optimization for Attack Technique Extraction from Cyber Threat Intelligence

从网络威胁情报中提取攻击技术的检索受限策略优化

• Authors: Jiayun Zhang, Junshen Xu, Zejun Xie, Yi Fan
• Subjects: Subjects: Cryptography and Security (cs.CR); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.06778
• Pdf link: https://arxiv.org/pdf/2608.06778
• Abstract Mapping cyber threat intelligence (CTI) text to MITRE ATT&CK techniques is essential for structured threat analysis, yet manual annotation is costly and does not scale. The ATT&CK taxonomy comprises several hundred attack techniques, and a single CTI passage may describe multiple techniques, making accurate and complete extraction challenging. Existing automated approaches fall short in different ways: multi-label classifiers struggle with severe class imbalance and the large label space, while LLM-based methods--retrieval pipelines and fine-tuned generators--optimize token-level objectives that treat technique annotation as sequence generation rather than set prediction, lacking direct supervision on whether the predicted technique set is correct and complete. We propose TTP-R1, a two-stage framework that combines retrieval-augmented supervised fine-tuning (SFT) with reinforcement learning using verifiable rewards (RLVR). A hybrid retriever first narrows the large label space to a candidate set, and a fine-tuned LLM learns to select the correct techniques. We then apply Group Relative Policy Optimization with a decomposed reward that directly supervises the precision, recall, and output format of the predicted technique set. Across four CTI benchmarks, TTP-R1 achieves the best average F1, improving sub-technique-level F1 by 7.4 percentage points over Claude Sonnet 4.5 with retrieval augmentation, while running 28x faster when served as an 8B-parameter model on a single GPU.
• 中文摘要 将网络威胁情报（CTI）文本映射到MITRE AT&CK技术对于结构化威胁分析至关重要，但手动注释成本高且无法扩展。ATT&CK分类包含数百种攻击技术，单条CTI段落可能描述多种技术，使得准确且完整的提取具有挑战性。现有自动化方法在多方面存在不足：多标签分类器面临严重的类不平衡和大标签空间，而基于LLM的方法——检索流水线和微调生成器——则优化了令牌级目标，将技术注释视为序列生成而非集合预测，缺乏对预测技术集是否正确和完整的直接监督。我们提出了TTP-R1，这是一个两阶段框架，结合检索增强监督微调（SFT）与使用可验证奖励（RLVR）的强化学习。混合检索器首先将大标签空间缩小到候选集合，微调的LLM学习选择正确的技术。然后我们应用组相对策略优化，并以分解奖励直接监督预测技术集的精度、回忆和输出格式。在四个CTI基准测试中，TTP-R1实现了最佳的平均F1，比Claude Sonnet 4.5在检索增强后提升子技术层F17.4个百分点，同时作为单GPU上的8B参数模型运行速度提升28倍。

EvoRIC: Reinforcement Learning Fine-Tuned LLM-empowered RAN Intelligent Control Toward Autonomous O-RAN

EvoRIC：强化学习 微调的 LLM 赋能的 RAN 智能控制，实现自主 O-RAN

• Authors: Lingyan Bao, Jemin Lee, Tony Q.S. Quek
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.06789
• Pdf link: https://arxiv.org/pdf/2608.06789
• Abstract Despite recent advances in applying artificial intelligence (AI) techniques to radio access network (RAN), critical challenges remain: traditional machine learning (ML) algorithms suffer from limited generalization across varying network topologies, whereas general-purpose large language models (LLMs) face high computational demands and lack domain-specific knowledge. To address these gaps, this article introduces the evolving RAN intelligent controller (RIC) (EvoRIC) framework, a hierarchical architecture that enables continuous evolution by leveraging a non-real-time RIC (non-RT RIC) for global model updates and a near-real-time RIC (near-RT RIC) for local execution, dynamically empowering LLMs with domain-specific decision-making capabilities. Within this framework, we employ a reinforcement learning-based fine-tuning (RLFT) mechanism where an LLM operates as an actor within a proximal policy optimization (PPO) agent. By leveraging the interaction tuples collected from the wireless environment, the LLM's parameters are iteratively updated to align semantic reasoning with rigorous network performance objectives. We evaluate the generalization and efficacy of the proposed EvoRIC framework within integrated access and backhaul (IAB) networks, and finally, discuss the open challenges and future directions of the EvoRIC framework toward realizing autonomous O-RAN.
• 中文摘要 尽管人工智能（AI）技术在无线接入网络（RAN）中的应用近年来取得了进展，但关键挑战依然存在：传统机器学习（ML）算法在不同网络拓扑上的推广性有限，而通用大型语言模型（LLM）面临较高的计算需求且缺乏领域特定知识。为弥补这些不足，本文介绍了不断演进的RAN智能控制器（RIC）（EvoRIC）框架，这是一种层级架构，通过利用非实时RIC（非RT RIC）进行全局模型更新，以及近实时RIC（近RT RIC）进行本地执行，实现持续演进，动态赋能LLM具备领域特定的决策能力。在该框架下，我们采用基于强化学习的微调（RLFT）机制，LLM作为近端策略优化（PPO）代理中的行为者运行。通过利用从无线环境中收集的交互元组，LLM的参数迭代更新，使语义推理与严格的网络性能目标保持一致。我们评估了EvoRIC框架在集成接入与回程（IAB）网络中的推广性和有效性，并讨论了EvoRIC框架在实现自主O-RAN方面的未解决挑战和未来方向。

PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Model

过去：高效扩散模型的提示自适应采样终止

• Authors: Renye Yan, Jikang Cheng, You Wu, Wei Peng, Zongwei Wang, Ling Liang, Yimao Cai
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.06794
• Pdf link: https://arxiv.org/pdf/2608.06794
• Abstract While diffusion models have made significant progress in text-to-image tasks, they still exhibit limitations when directly optimizing downstream objectives. Although Reinforcement Learning (RL) enables targeted optimization, existing methods are generally constrained by low-efficiency fine-tuning and sparse rewards. To address these challenges, we propose PAST, which provides differentiated rewards while adaptively regulating training episode length by jointly perceiving denoising progress and prompt difficulty. Specifically, we design an intrinsic reward paradigm to compensate for sparse extrinsic rewards and guide the model to explore paths that diverge more efficiently from noise patterns. We further provide theoretical justification for intrinsic rewards. Then, PAST dynamically monitors denoising completion and semantic alignment between image structures and prompt semantics. When both metrics satisfy generation requirements, the system adaptively terminates training. This enables appropriate allocation of episode lengths based on prompt difficulty and the current generation process. Finally, based on the predicted residual noise level, we establish a dual adaptive coordination mechanism. Specifically, it not only balances the extrinsic and intrinsic rewards but also balances the exploration and convergence. Experimental results demonstrate that PAST enhances computational efficiency of existing RL fine-tuning methods by up to 66.7%, while improving preference optimization quality by up to 29.5% through its dual adaptive regulation mechanism.
• 中文摘要 尽管扩散模型在文本到图像任务中取得了显著进展，但在直接优化下游目标时仍存在局限性。尽管强化学习（RL）能够实现有针对性优化，但现有方法通常受限于低效率的微调和稀疏的奖励。为应对这些挑战，我们提出了PAST，它在通过联合感知去噪进展和提示难度来自适应调节训练周期长度的同时，提供差异化奖励。具体来说，我们设计了一个内在奖励范式，以补偿稀疏的外在奖励，并引导模型探索更高效地偏离噪声模式的路径。我们还进一步为内在奖励提供了理论上的依据。随后，PAST动态监控图像结构与提示语义之间的去噪完成度和语义对齐。当两个指标都满足生成要求时，系统自适应终止训练。这使得根据提示难度和当前生成过程合理分配剧集长度成为可能。最后，基于预测的残余噪声水平，我们建立了双适应协调机制。具体来说，它不仅平衡了外在和内在奖励，还平衡了探索与收敛。实验结果表明，PAST通过其双适应调控机制，将现有强化学习微调方法的计算效率提升了高达66.7%，同时提升偏好优化质量高达29.5%。

C2Dex: Contact-Consistent Reconstruction and Retargeting for Dexterous Manipulation from Monocular Video

C2Dex：单目视频中灵巧操作的接触一致性重建与重定向

• Authors: Jie Ren, Zhehao Jiang, Yinhong Yang, Haorui Jia, Han Jiang, Ben Li, Yao Yao, Cheng Lin, Qiu Shen, Zhenshan Bing, Xiao-Xiao Long, Xun Cao
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.07045
• Pdf link: https://arxiv.org/pdf/2608.07045
• Abstract High-quality demonstrations for dexterous robot manipulation are costly and difficult to collect, whereas monocular human videos provide a scalable source of diverse manipulation behaviors. However, transferring such demonstrations to dexterous robots remains challenging: monocular hand-object interaction (HOI) reconstruction often produces temporally unstable contacts and physically implausible interactions, while conventional retargeting methods struggle to preserve task-relevant contacts and local interaction geometry across different hand embodiments. We present C2Dex, a video-to-dexterous-manipulation framework built around a shared interaction representation: stable object-side contacts recovered by aggregating noisy frame-wise observations in the canonical object space. These stable contacts serve a dual role: as trajectory-level constraints that guide reconstruction toward temporally coherent and physically plausible human HOI trajectories, and as explicit transfer targets for the dexterous hand, where Laplacian interaction optimization preserves the local hand-object geometry across embodiments and residual reinforcement learning refines the trajectory in simulation. Experiments on DexYCB and TACO show that C2Dex achieves end-to-end trajectory success rates of 57.78% and 26.67%, respectively, substantially outperforming the strongest baselines (17.78% and 10.00%) under identical evaluation criteria. Real-robot replay experiments further demonstrate physical feasibility across diverse contact-rich manipulation tasks. Project page: this https URL
• 中文摘要 高质量的灵巧机器人操作演示成本高昂且难以收集，而单眼人类视频则提供了可扩展的多样化操作行为来源。然而，将此类演示转移到灵巧机器人上仍然具有挑战性：单眼手-物体交互（HOI）重建常常产生时间不稳定的接触和物理上不合理的交互，而传统的重定向方法则难以在不同手部表现中保留任务相关接触和局部交互几何。我们提出了C2Dex，一个基于共享交互表示构建的视频到灵巧操作框架：通过汇总规范对象空间中噪声的逐帧观测恢复的稳定对象端接触。这些稳定接触具有双重作用：作为轨迹级约束，引导重建朝向时间上连贯且物理上合理的人类HOI轨迹，同时作为灵巧手的明确转移目标，拉普拉斯交互优化保留了各实体间的局部手-物体几何，残余强化学习则在模拟中精炼轨迹。在DexYCB和TACO上的实验显示，C2Dex分别实现了端到端轨迹成功率为57.78%和26.67%，在相同评估标准下显著优于最强基线（17.78%和10.00%）。真实机器人回放实验进一步证明了在多种接触丰富操作任务中的物理可行性。项目页面：此 https URL

Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Control

超越隔离：释放强化学习组件协同效应，实现样本高效连续控制

• Authors: Qi Zhao, Guozheng Ma, Yilun Kong, Lu Li, Haoyu Wang, Zilin Wang, Tiantian Zhang, Yuxing Wang, Jian Sha, Yongzhe Chang, Xueqian Wang, Dacheng Tao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07086
• Pdf link: https://arxiv.org/pdf/2608.07086
• Abstract Reinforcement learning systems are significantly more complex than other machine learning paradigms due to inherent properties, causing RL system design to jointly account for many tightly coupled factors. Despite advances in individual algorithmic components, their functional interdependencies remain underexplored: do they exhibit mutual synergy or counterproductive interference? To bridge this gap, we conduct a systematic investigation and find that the efficacy of different components exhibits significant task-dependency, and naively stacking state-of-the-art techniques does not necessarily yield performance gains; instead, it often triggers emergent challenges, such as compounded non-stationarity. Building upon these findings, we distill a suite of actionable insights into the principled coordination of these components. Guided by these insights, we propose ROSER, an RL framework that coordinates three critical dimensions: Model-based Representation, Optimization Stability, and Experience Replay. Across diverse continuous-control benchmarks, ROSER consistently outperforms vanilla baselines and achieves 17.60% gains over naive stack. Our findings underscore the necessity of a holistic perspective in RL system design and paves the way for developing sample-efficient agents.
• 中文摘要 由于固有特性，强化学习系统比其他机器学习范式复杂得多，导致强化学习系统设计需要共同考虑许多紧密耦合的因素。尽管单个算法组件取得了进步，它们的功能相互依赖性仍未被充分探索：它们是相互协同还是适得其反的干扰？为弥合这一差距，我们进行了系统性调查，发现不同组件的效能表现出显著的任务依赖性，简单地堆叠最先进技术未必带来性能提升;相反，它往往会引发诸如复合非平稳性等突发挑战。基于这些发现，我们总结出一系列关于这些组件原则性协调的可操作洞见。基于这些见解，我们提出了ROSER，这是一个协调三个关键维度的RL框架：基于模型的表示、优化稳定性和体验重放。在多种连续对照基准测试中，ROSER持续优于普通基线，并实现了17.60%的优越。我们的发现强调了在强化学习系统设计中整体视角的必要性，并为开发样本高效智能体铺平了道路。

How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

那么，在哪里：多回合代理强化学习中的守信用动作到令牌分配

• Authors: Lichao Ma, Yang Sun, Shuaitao Zhao, Yangyi Fang, Cong Qin, Xiaoliang Fu, Yuhang Tian, Yuchen Wei, Junbo Zhu, Yang Wei, Lu Pan, Jiaye Lin
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07118
• Pdf link: https://arxiv.org/pdf/2608.07118
• Abstract Credit assignment in multi-turn agent reinforcement learning operates at two levels: assigning trajectory-level credit to actions and distributing each action's credit across its tokens. In this paper, we introduce FACTOR, which separates these decisions. FACTOR uses checkpoint-calibrated TD residuals to assign per-action credits that telescope to the trajectory advantage, and feedback-conditioned teacher-student likelihood gaps to allocate each credit across the realized action tokens. Per-action normalization preserves the action-average coefficient and prevents token-level sign flips. We pair this construction with an action-mean reduction, removing the implicit dependence of an action's scalar surrogate weight on its token length. At the behavior policy and before clipping, each action's inner action-mean surrogate equals its TD credit. FACTOR consistently improves over competitive baselines across ALFWorld, WebShop, and ScienceWorld, with every environment-seed comparison favoring FACTOR and the largest gains emerging on the longest-horizon environment. The same hyperparameters transfer without retuning to a larger backbone and to a different model family. Ablations identify TD action credit as the dominant driver of the improvement, with hindsight token allocation contributing complementary gains.
• 中文摘要 多回合代理强化学习中的信用分配在两个层面：为动作分配轨迹级信用，并将每个动作的信用分配到其代币中。本文介绍了FACTOR，将其区分这些决策。FACTOR 使用检查点校准的TD残差分配每个动作的信用，使得轨迹优势显现，并通过反馈条件的师生似然差距分配每个积分在已实现的动作代币中。每个动作归一化保持动作平均系数，防止代币级符号翻转。我们将此构造与动作均值约简结合，消除动作标量代理权重对其代币长度的隐式依赖。在行为策略和裁剪前，每个动作的内部动作均值代理指标等于其TD积分。FACTOR在ALFWorld、WebShop和ScienceWorld的竞争基线中持续提升，所有环境-种子比较均偏向FACTOR，且在视野最长的环境中获得最大收益。相同的超参数在不重调至更大骨干或不同模型族的情况下转移。分析显示TD行动积分是改进的主要驱动力，事后诸葛亮代币分配也贡献了互补收益。

DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

DiDPO：编码代理训练的差分中差分策略优化

• Authors: Xucong Wang, Zhe Zhao, Liheng Yu, Di Wu, Xiaofeng Cao, Pengkun Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07147
• Pdf link: https://arxiv.org/pdf/2608.07147
• Abstract Reinforcement learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for training coding agents, where the execution feedback from compilation and tests provides objective verification. However, unlike agent tasks, coding agents face a unique and finer-grained credit assignment challenge: at each step, coding actions simultaneously pack varying changes into different regions of a code version, which makes the contribution of independent change indistinguishable. Existing RLVR methods mostly leverage the outcome reward or step-level reward, which fails to dive into a code diff and makes unique properties of coding actions invisible to training. In this paper, we propose Diff-in-Diff Policy Optimization (DiDPO), a critic-free RL method that constructs fine-grained credit units directly from the structure of code diffs. DiDPO organizes multi-turn coding interactions into multiple thought--action steps and discovers code diffs across sampled trajectories. It then selects anchors by aggregating highly similar sub-diffs split from each whole diff by our ``groupability score'', which provides the splitting schema that optimally balances the semantic scope of anchors and the group mass they may form. Finally these anchors form advantage groups and project the diff-level advantage back to individual response tokens. Experiments on long-horizon coding and reasoning benchmarks show that DiDPO significantly outperforms strong agentic RL baselines. On Qwen2.5-7B-Coder, DiDPO exceeds comparable methods by over 10\% and narrows the gap with far larger models, offering a principled framework for fine-grained credit assignment in coding agent training. We also open-source verl-code, an agentic rl codebase that supports various RL methods and coding benchmarks.
• 中文摘要 带有可验证奖励（RLVR）的强化学习已成为训练编码代理的强大范式，其编译和测试的执行反馈提供了客观验证。然而，与代理任务不同，编码代理面临独特且更细粒度的信用分配挑战：每一步，编码动作同时将不同变化打包到代码版本的不同区域，使得独立变化的贡献难以区分。现有RLVR方法大多利用结果奖励或步骤级奖励，这种机制未能深入代码差异，使编码动作的独特属性对训练不可见。本文提出了差分策略优化（DiDPO），这是一种无批评的强化学习方法，直接从代码差分结构构建细粒度的信用单元。DiDPO将多回合编码交互组织为多个思维-行动步骤，并发现采样轨迹中的代码差异。然后，它通过聚合从每个差分中划分出的高度相似子差异，按我们的“可组合性评分”来选择锚点，从而提供一个最佳平衡锚点语义范围和其可能形成群体质量的拆分模式。最后，这些锚点形成优势组，并将差异层级优势投射回单个响应代币。长期编码和推理基准测试的实验显示，DiDPO显著优于强能动RL基线。在Qwen2.5-7B-Coder上，DiDPO比同类方法高出10%以上，并缩小了与更大模型差距，为编码主体训练中的细粒度学分分配提供了原则性框架。我们还开源了verl-code，这是一个支持多种强化学习方法和编码基准的代理强化强化学习代码库。

A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturing

以MARL为中心的参考架构，用于智能制造中的大型语言模型增强

• Authors: Fouad Bahrpeyma, Dirk Reichelt
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07148
• Pdf link: https://arxiv.org/pdf/2608.07148
• Abstract Modern manufacturing imposes six coupled demands on adaptive control: local decisions with global consequences, partial observability, nonstationarity, reflex speed response with long horizon effects, delayed and diffuse outcomes, and dynamics that resist explicit modeling. Cooperative multiagent reinforcement learning (MARL), posed as a Dec-POMDP under centralized training with decentralized execution, is a particularly natural formalism for these demands. This paper adopts a MARL centered scope and asks where large language models (LLMs) should augment, interface with, train, or, in the strongest competitive case, replace that coordination core. A taxonomy organizes the literature through four LLM attachment points: policy, reward design, communication between agents, and hierarchical planning. A conditional capability profile separates native mechanism, reported performance, formal guarantee, and engineering maturity, and a deployment readiness analysis identifies the evidence behind each role. These stages yield the principal contribution: a three layer MARL centered reference architecture, grounded in evidence, for semantic reasoning, adaptive cooperative control, and independently assured execution. The LLM-Augmented Dec-POMDP is a descriptive comparative notation for that architecture, recording four attachment choices without introducing a new decision process class or algorithm. Under the reviewed evidence, conventional MARL is better suited to frequent, structured, decentralized coordination after task specific training, whereas LLM components are promising for semantic interpretation, reward drafting, human interaction, and slower supervisory planning. Current LLM only manufacturing controllers do not yet establish equivalence for strict real time, decentralized, safety critical control; this conclusion is bounded by the available evidence and does not assert impossibility.
• 中文摘要 现代制造对自适应控制施加了六项耦合要求：带全局后果的局部决策、部分可观测性、非平稳性、具有长视野效应的反射速度响应、延迟和扩散的结果，以及抗拒显式建模的动态。协作多智能体强化学习（MARL）作为集中训练和去中心化执行下的Dec-POMDP，是这些需求的自然形式化。本文采用以MARL为中心的范围，探讨大型语言模型（LLM）应在哪里增强、接口、训练，或在最激烈竞争的情况下替代该协调核心。分类法通过四个LLM附属点组织文献：策略、奖励设计、代理间通信和层级规划。条件能力配置文件区分原生机制、报告性能、形式保证和工程成熟度，部署准备度分析识别每个角色背后的证据。这些阶段产生了主要贡献：一个三层MARL中心的参考架构，基于证据，用于语义推理、自适应合作控制和独立保证执行。LLM增强型Dec-POMDP是一种描述性比较符号，记录四个附加选项，无需引入新的决策过程类或算法。根据综述证据，传统MARL更适合任务特定训练后的频繁、结构化、去中心化协调，而LLM组件则在语义解释、奖励起草、人际互动和较慢的监督规划方面具有前景。目前仅限制造的LLM控制器尚未建立严格实时、去中心化、安全关键控制的等价性;这一结论受现有证据限制，并不断言不可能。

Interpretable reinforcement learning with decision-tree pruning

可解释的强化学习与决策树修剪

• Authors: Mark Leon Ringer, Michel Tokic
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07151
• Pdf link: https://arxiv.org/pdf/2608.07151
• Abstract Reinforcement learning policies are difficult to inspect, but interpreting them is a prerequisite for trustworthiness. Converting a trained policy into explicit decision-tree rules improves transparency and the resulting artifacts often remain too complex for human understanding. We present a pruning process that simplifies such rule-based policies while preserving task performance and making edits to the policy auditable. The process defines a small set of structural and usage-aware operators and evaluates candidate edits by re-executing the policy to measure return and interpretability proxies. This exposes an transformation process from complex to compact policy structures. We investigate this approach on classic control and MuJoCo benchmarks, where pruning traces reveal consistent interpretability improvements while maintaining high performance.
• 中文摘要 强化学习策略难以检查，但解释它们是可信的前提。将训练有素的策略转换为显式决策树规则提升透明度，而产生的产物往往过于复杂，难以人类理解。我们提出了一种修剪过程，简化了基于规则的策略，同时保持任务性能并使策略编辑可审计。该过程定义了一小部分结构性和使用意识操作符，并通过重执行策略评估候选编辑，以衡量返回和可解释代理。这揭示了从复杂到紧凑策略结构的转变过程。我们在经典控制和MuJoCo基准测试中研究了这一方法，剪枝痕迹显示出一致的可解释性改进，同时保持高性能。

Momba: Network Modernization Improves Multi-Objective Reinforcement Learning

Momba：网络现代化提升多目标强化学习

• Authors: Adam Štafa, Santeri Heiskanen, Petr Novotný, Joni Pajarinen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07180
• Pdf link: https://arxiv.org/pdf/2608.07180
• Abstract Recent advances in deep reinforcement learning (RL) have shown that improving neural network architectures can yield substantial gains in sample efficiency and asymptotic performance without altering the underlying algorithms. In contrast, work on multi-objective reinforcement learning (MORL), which aims to discover a set of policies that balance trade-offs among conflicting objectives, has predominantly focused on algorithmic innovations, leaving the area of architectures underexplored. While the optimal policies and value functions can differ significantly depending on the trade-offs, MORL algorithms commonly represent them with simple feedforward networks conditioned on the trade-off. This raises the question of whether the performance of the algorithms could be improved with more expressive function approximators. In this paper, we integrate recent advances in neural network design: (i) observation and feature normalization, (ii) weight normalization, and (iii) modeling of distributional returns with an entropy-regularized MORL algorithm. The empirical results across standard continuous control benchmarks demonstrate that these changes substantially improve the quality of the produced solution sets without requiring major changes to the underlying algorithm.
• 中文摘要 深度强化学习（RL）的最新进展表明，改进神经网络架构可以在不改变底层算法的情况下显著提升样本效率和渐近性能。相比之下，旨在发现一套平衡冲突目标权衡的策略的多目标强化学习（MORL）工作主要聚焦于算法创新，导致架构领域尚未被充分探索。虽然最优策略和价值函数可能因权衡而有显著差异，但MORL算法通常以基于权衡的简单前馈网络来表示。这引发了一个问题：是否可以通过更具表现力的函数近似器来提升算法的性能。本文整合了神经网络设计的最新进展：（i）观测与特征归一化，（ii）权重归一化，（iii）用熵正则化MORL算法建模分布收益。跨越标准连续控制基准的实证结果表明，这些变化显著提升了生成的解集质量，而无需对底层算法进行重大修改。

Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis

在部分可观测性：封闭形式分析下，学习比策略类更受影响

• Authors: Idil Gözel (University College London)
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2608.07228
• Pdf link: https://arxiv.org/pdf/2608.07228
• Abstract When a reinforcement learning agent cannot observe the full state, we usually blame its policies: it cannot see enough to represent a good one. We show that in a solvable case the bigger problem lies elsewhere. Even when a good policy is available and the agent's value function is expressive enough to describe it exactly, learning still ends up somewhere far worse. We study a partially observed linear-quadratic problem in which a standard actor-critic learner can be solved in closed form. At our default setting the best policy the agent can represent is already close to optimal, costing 10.4% more than the ideal controller that observes everything. Learning does not find it. The algorithm instead comes to rest at a policy that is 35% worse than the best one available to it, and we can say exactly where and why. The cause is a bias in what the critic learns rather than a limit on what the actor can express. Because the agent cannot attribute what it sees to the part of the state it cannot observe, the critic misreads that unexplained variation as sharp curvature in its own value estimates, and the actor follows that error away from the optimum. We derive closed-form expressions for the resulting policy, for its cost, and for the one design choice that removes the problem, which is how far the learner looks ahead before trusting its own value estimates. Deep reinforcement learning experiments follow these predictions closely. Notably, giving the agent memory of past observations does not help, while changing how far it looks ahead does.
• 中文摘要 当强化学习代理无法观察完整状态时，我们通常归咎于其策略：它无法看到足够多的策略来代表一个好的策略。我们证明，在可解的情况下，更大的问题存在于其他地方。即使存在一个好的策略，且代理的价值函数足够表达能够精确描述它，学习结果仍然会更糟。我们研究一个部分观察的线性二次问题，其中标准的演员-批评学习者可以用封闭形式求解。在我们的默认设置下，代理能代表的最佳策略已经接近最优，成本比观察一切的理想控制器高出10.4%。学习找不到它。算法最终停留在一个比最佳策略差35%的策略上，我们可以准确说明具体位置和原因。原因是批评者学习内容的偏差，而非行为者表达能力的限制。由于智能体无法将所见归因于其无法观察的状态部分，批评者将这种无法解释的变异误解为自身价值估计中的急剧曲率，行为者则将该误差从最优值之外追踪。我们推导出最终策略、成本以及唯一消除问题的设计选择的闭式表达式，即学习者在信任自身价值估计前展望多远。深度强化学习实验紧随这些预测。值得注意的是，给予代理过去观察的记忆无济于事，而改变其展望时间则有效。

Why Study Emergent Behavior When You Can Regulate It? Aligning Multi-Agent Systems with Reward Prediction

既然可以调节，为什么还要研究涌现行为？将多智能体系统与奖励预测对齐

• Authors: Assaf Caftory, Almog Zemach, Moshe Butman, Doron Friedman
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.07280
• Pdf link: https://arxiv.org/pdf/2608.07280
• Abstract Multi-agent simulations are widely used to study complex social and ecological systems, where rich and often unexpected emergent behaviors arise from local interactions. A large body of prior work has focused on analyzing such emergent dynamics across domains. In this paper, we move beyond analyzing emergent behavior and introduce a learning-based mechanism for actively shaping it via social reward modeling. We introduce Multi-Agent Reward Prediction (MARP), a simple framework that extends preference-based reward modeling to multi-agent reinforcement learning. While the framework is designed to be applicable across multi-agent settings, the present empirical validation is limited to a single environment, and we therefore present MARP as a proof of concept within the studied domain. Rather than relying on handcrafted rewards, MARP learns a shared reward model from episode-level evaluations of collective outcomes, enabling decentralized agents to align their behavior with global social objectives. We study MARP in the Harvest Game, a canonical sequential social dilemma modeling common-pool resource management and related real-world challenges. Our results show that MARP can be tuned to produce behavior that is more closely aligned with target social metrics than standard reward-based baselines, while the learned reward model captures subtle environmental structure without explicit programming. Crucially, MARP supports multiple and composite social objectives within a single training regime. By modifying only the high-level evaluation metric, the same framework seamlessly aligns agent behavior with diverse goals, including sustainability, equality, and peace, as well as combinations of individual and group-level objectives. These findings demonstrate that emergent multi-agent behavior can be treated not only as a phenomenon to study, but as a target of principled, data-driven regulation.
• 中文摘要 多智能体模拟被广泛用于研究复杂的社会和生态系统，这些系统中丰富且常常出人意料的涌现行为源自局部互动。大量先前研究聚焦于分析跨领域涌现动态。本文超越了对涌现行为的分析，引入了基于学习的机制，通过社会奖励建模主动塑造行为。我们引入了多智能体奖励预测（MARP），这是一个简单框架，将偏好奖励建模扩展至多智能体强化学习。虽然该框架设计适用于多智能体环境，但当前的实证验证仅限于单一环境，因此我们将MARP作为该领域内的概念验证呈现。MARP不再依赖手工设计的奖励，而是通过集数级集体结果评估学习共享奖励模型，使去中心化智能体能够使其行为与全球社会目标保持一致。我们在收获游戏中研究MARP模型，这是一种典型的序列社会困境，模拟公共资源管理及相关现实世界挑战。结果显示，MARP可以调优以产生比标准奖励基线更贴近目标社会指标的行为，而习得的奖励模型则捕捉微妙的环境结构，无需显式编程。关键是，MARP支持单一训练体系内多重且复合的社会目标。仅修改高层评估指标，同一框架无缝将代理行为与多样化目标对齐，包括可持续性、平等与和平，以及个人和群体层面目标的组合。这些发现表明，新兴的多代理行为不仅可作为研究现象，更可作为原则性、数据驱动监管的目标。

Learning Long-Term Educational Investment Policies under Residential Sorting

学习长期教育投资政策，采用住宅分拣法

• Authors: Honglei Guo, Shuo Chen, Mingjie Bi, Zeyang Sun, Xiaoxi Wang, Yuhan Zhao
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.07295
• Pdf link: https://arxiv.org/pdf/2608.07295
• Abstract Allocating public-school investment effectively and fairly is difficult when school access depends on residence. School improvements can raise nearby housing demand and prices, reshape enrollment, and potentially limit access for lower-income households. These effects evolve as residential sorting changes school composition, quality, and future investment needs. Existing approaches often study school funding, household choice, and housing markets separately, while static models can miss their interconnected, long-term effects. We address this gap with a dynamic multi-agent framework that links government investment, household sorting, housing prices, population turnover, enrollment, and evolving school quality. A government planner uses reinforcement learning (RL) to identify multiyear allocation policies that account for household responses while balancing aggregate educational access and equity. In simulations, our RL-based policy attains the highest access level (0.4780) and second-lowest access Gini coefficient (0.0164) among representative baselines, demonstrating a favorable effectiveness-equity balance. The results also indicate reduced socioeconomic stratification in educational access. By making education-housing feedback explicit, our framework supports long-term analysis of how school investment shapes educational opportunity over time.
• 中文摘要 当学校入学依赖于居住地时，有效且公平地分配公立学校投资变得困难。学校改进可以提高附近住房需求和价格，重塑入学人数，并可能限制低收入家庭的入学机会。随着住宿分类改变学校构成、质量和未来投资需求，这些效应不断演变。现有方法通常分别研究学校资金、家庭选择和住房市场，而静态模型则可能忽略它们相互关联的长期影响。我们通过动态多代理框架解决这一差距，该框架将政府投资、家庭分类、住房价格、人口流动、招生和学校质量的演变联系起来。政府规划者利用强化学习（RL）确定多年分配政策，既考虑家庭响应，又平衡整体教育机会与公平性。在模拟中，我们基于强化学习的政策在代表性基线中达到了最高入学水平（0.4780）和第二低的基尼系数（0.0164），显示出有利的效能与公平平衡。结果还显示教育机会中的社会经济分层减少。通过明确化教育-住房反馈，我们的框架支持对学校投资如何长期影响教育机会的长期分析。

TEMPO: Semantic-Action Decoupled RL Post-Training for Vision-Language-Action Models

TEMPO：视觉-语言-行动模型的语义-动作解耦强化学习后训练

• Authors: Ziheng Liu, Quantao Yang
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.07314
• Pdf link: https://arxiv.org/pdf/2608.07314
• Abstract Vision-language-action (VLA) models are commonly adapted to downstream manipulation tasks via supervised fine-tuning (SFT) or online reinforcement learning (RL) post-training. SFT is prone to distribution mismatch, and existing RL approaches typically apply a single, uniform update strategy to all model components, ignoring their distinct functional roles. We propose TEMPO, a semantic-action decoupled, two-timescale RL post-training framework for VLA models. TEMPO freezes the pretrained vision-language backbone to preserve general semantic representations, and restricts adaptation to two components with dedicated RL optimization loops: the semantic projection layer and the low-level action expert. We update them at different rates--the semantic projection layer infrequently, to keep the latent action stable, and the action expert frequently, to rapidly incorporate control feedback from online interaction. This decoupling RL fine-tuning strategy prevents fast policy updates from destabilizing high-level semantic representations while still allowing the action expert to learn efficiently from online feedback. Experiments on the CALVIN benchmark and real-world manipulation tasks demonstrate that TEMPO consistently outperforms both pretrained state-of-the-art VLA models and the RL post-training baseline, while reaching and maintaining higher evaluation rewards on two real-world tasks.
• 中文摘要 视觉-语言-动作（VLA）模型通常通过监督微调（SFT）或在线强化学习（RL）在训练后适应下游操作任务。SFT容易出现分布不匹配，现有的强化学习方法通常对所有模型组件采用单一统一的更新策略，忽略它们各自的功能角色。我们提出了TEMPO，一种语义-动作解耦的两时间尺度强化学习后训练框架，适用于VLA模型。TEMPO冻结预训练视觉语言骨干网以保留一般语义表示，并将适应限制在两个具有专用强化学习优化循环的组件：语义投射层和低级动作专家。我们以不同速率更新它们——语义投射层更新频率较低，保持潜在动作稳定，动作专家更新频率，快速整合在线交互的控制反馈。这种解耦RL微调策略防止了快速策略更新破坏高层语义表示，同时仍能让行动专家高效地从在线反馈中学习。CALVIN基准测试和现实操作任务的实验表明，TEMPO持续优于预训练的先进VLA模型和RL后训练基线，同时在两项真实任务中获得并保持更高的评估奖励。

Learning Fault-Tolerant Locomotion with Adaptive Gait Timing

利用自适应步态时机学习容错移动

• Authors: Giovanbattista Gravina, Luca Rossini, Carlo Rizzardo, Arturo Laurenzi, Nikos Tsagarakis
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.07328
• Pdf link: https://arxiv.org/pdf/2608.07328
• Abstract Hardware failures require legged robots to rapidly reorganize coordination and gait timing to maintain stability and mobility. This is particularly challenging for larger quadrupeds, where increased mass and tighter actuation limits reduce the feasibility of aggressive, high-frequency compensation strategies often observed on smaller platforms. In this work, we propose a deep reinforcement learning approach for fault-tolerant locomotion under actuator power loss. The method employs an asymmetric actor-critic architecture in which the critic has access to privileged information during training, while the actor learns to reconstruct a corresponding latent representation from proprioceptive observations. We introduce a latent-alignment loss that encourages consistency between actor and critic representations. Additionally, we augment the action space with a learnable gait frequency parameter, enabling adaptive gait timing in response to terrain variations and actuator degradation without predefined faulty-leg strategies. The approach is validated in high-fidelity simulation on uneven terrain and real-world experiments on flat ground using a 68 kg quadruped robot.
• 中文摘要 硬件故障需要腿部机器人快速重组协调和步态时机，以保持稳定性和机动性。这对大型四足动物尤其具有挑战性，因为质量增加和更严格的执行限制降低了在小型平台上常见的高频激进补偿策略的可行性。本研究提出一种深度强化学习方法，用于执行器断电时容错运动。该方法采用非对称的行为者-批评者架构，批评者在训练期间可访问特权信息，而执行者则通过本体感受观察学习重建相应的潜在表征。我们引入了潜在对齐损失，促进演员与批评者表征之间的一致性。此外，我们还通过可学习的步态频率参数增强动作空间，实现对地形变化和执行器退化的自适应步态时机，无需预设故障腿策略。该方法在不平坦地形的高保真模拟和使用68公斤四足机器人的平地真实实验中得到了验证。

Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks

Aftab：并行Q网络中CNN编码器和高级值函数的综合基准测试

• Authors: Taha Shieenavaz, Shabnam Zareshahraki, Loris Nanni
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07335
• Pdf link: https://arxiv.org/pdf/2608.07335
• Abstract Recent advancements in deep reinforcement learning have increasingly favored simplified, highly parallelized paradigms. Notably, the Parallelized Q-Network (PQN) algorithm achieves stable off-policy learning without relying on computationally expensive replay buffers or target networks. However, the representational capacity and parameter efficiency of visual encoders operating in these buffer-free settings remain underexplored. In this work, we systematically investigate the architectural design space of Convolutional Neural Networks for PQN. We design and rigorously evaluate eight distinct CNN topologies, optimizing for sample efficiency under strict parameter constraints. Furthermore, we study the impact of representation and value estimation enhancements by integrating the Hadamax encoding paradigm and advanced Q-learning extensions, including distributional, ensemble, and dueling heads. Extensive experiments on the Atari-57 benchmark demonstrate that our proposed composite architecture, Aftab, achieves an Interquartile Mean (IQM) Human-Normalized Score of 6.479, establishing a 0.86 Probability of Improvement over the standard PQN baseline. Additionally, structural resilience evaluations on the highly non-stationary Procgen Hard benchmark confirm out-of-distribution generalization, with Aftab yielding an IQM Procgen Normalized Score of 0.418 compared to the baseline's 0.382. Ultimately, this work establishes an efficient, probabilistically superior structural reference for model-free reinforcement learning, all while preserving the simplicity and memory efficiency of unbuffered, parallelized optimization. The complete Aftab framework, including all model definitions, training configurations, and raw experimental logs, is open-sourced and available on our GitHub repository: this https URL
• 中文摘要 深度强化学习的最新进展越来越倾向于简化、高度并行化的范式。值得注意的是，并行化Q网络（PQN）算法能够实现稳定的非策略学习，而无需依赖计算量高的重放缓冲区或目标网络。然而，在这些无缓冲环境中运行的视觉编码器的表征能力和参数效率仍未被充分探索。本研究系统地探讨了卷积神经网络在PQN中的架构设计空间。我们设计并严格评估了八种不同的CNN拓扑结构，在严格参数约束下优化样本效率。此外，我们通过整合Hadamax编码范式和先进的Q学习扩展（包括分布头、集合头和对立头）研究了表示和价值估计增强的影响。在Atari-57基准测试上的大量实验表明，我们提出的复合架构Aftab实现了四分位均值（IQM）人类归一化评分6.479，提升概率为标准PQN基线的0.86。此外，对高度非平稳的Procgen Hard基准测试进行结构弹性评估，确认了分布外的泛化，Aftab的IQM Procgen归一化分数为0.418，而基线为0.382。最终，这项工作建立了一个高效且概率优越的无模型强化学习结构参考，同时保持了无缓冲、并行化优化的简洁性和内存效率。完整的Aftab框架，包括所有模型定义、训练配置和原始实验日志，均为开源，可在我们的GitHub仓库获取：此链接 https URL。

Trajectory-Relative Hindsight Distillation for Agentic Reinforcement Learning

轨迹相对事后洞察提炼用于能动强化学习

• Authors: Haoyu Zheng, Yun Zhu, Qing Wang, Wenqiao Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.07371
• Pdf link: https://arxiv.org/pdf/2608.07371
• Abstract Recent agentic reinforcement learning methods use hindsight to complement sparse outcome rewards. However, a completed rollout can yield many such signals, leaving their appropriate allocation across turns unclear. We introduce TRIAL, a trajectory-relative hindsight distillation framework with a unified turn-aligned scoring protocol. For each decision turn, TRIAL extracts an outcome view of that decision's realized consequence and evaluates the same response under ordinary and hindsight-conditioned contexts. The signed log-probability gap determines the direction and local strength of token-level supervision, while turn-level magnitudes are normalized jointly over the realized trajectory. The resulting allocation multipliers have an eligible-token-weighted mean of one, redistributing dense supervision across turns while fixing its average multiplier. Experiments on WebShop and ALFWorld with different backbones show that TRIAL outperforms GRPO across all eight combinations of backbone, environment, and evaluation metric, while achieving the best or tied-best performance among six methods on six of them. On WebShop with Qwen3-1.7B, TRIAL improves the success rate from 56.4% to 75.2% and the task score from 78.7% to 85.7%. Controlled ablations further show that trajectory-relative turn allocation provides substantial gains beyond those of dense hindsight distillation alone.
• 中文摘要 近期的代理强化学习方法利用事后视角来补充稀疏的结果奖励。然而，完成的推广可能会产生许多此类信号，但它们在各回合的适当分配仍不明确。我们引入了TRIAL，一种轨迹相对的事后洞察提炼框架，采用统一的回合对齐评分协议。对于每个决策回合，TRIAL提取该决策实现后果的结果视图，并在普通和事后视角条件下评估相同的反应。带符号的对数概率差距决定了代币级监督的方向和局部强度，而回合级的幅度则在实现轨迹上联合归一化。所得分配乘数的合格标记加权均值为1，在固定平均乘数的同时重新分配密集监督于各回合。在WebShop和ALFWorld上使用不同骨架的实验显示，TRIAL在骨干、环境和评估指标的八种组合中均优于GRPO，且在其中六种方法中表现最佳或并列最佳。在WebShop与Qwen3-1.7B的结合上，TRIAL将成功率从56.4%提升至75.2%，任务得分从78.7%提升至85.7%。受控消融进一步表明，轨迹相对转向分配带来的显著提升，远超单纯的密集事后诸葛明提取。

LYRA: Label-Free Structural Synchronization and Resource Allocation for UAV Edge Networks

LYRA：无标签结构同步与无人机边缘网络资源分配

• Authors: Feng He, Alireza Furutanpey, Paolo Bellavista, Yu Qiu, Jiangchuan Liu, Jiannong Cao, Schahram Dustdar
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.07392
• Pdf link: https://arxiv.org/pdf/2608.07392
• Abstract While deploying hierarchical vision models to process mission-critical tasks, UAV edge systems must adaptively update the models to sustain inference reliability under low-level environmental corruption. However, existing work has overlooked the optimal timing for model updates, the impracticality of relying on real-time expert labels, and the significant bandwidth and energy constraints of UAVs. This paper proposes a joint model update scheduling and resource allocation framework, aiming to maximize long-term semantic fidelity and resource efficiency of UAV edge intelligence systems. To address the challenge of label-free semantic evaluation, we formulate the Online Semantic Disagreement Rate (OSDR) as a proxy for timely update triggering, thereby enabling fine-grained Sensitivity-Aware Structural Synchronization (SASS). Furthermore, to overcome the curse of dimensionality in hybrid action spaces and effectively bound long-term energy budgets, we propose a Lyapunov-guided discrete reinforcement learning algorithm that performs action space dimensionality reduction and transforms constraints into virtual queue stability problems. The reported experimental results, based on real traffic traces, demonstrate that the proposed framework consistently outperforms representative baselines in semantic recovery efficiency and update triggering precision, by satisfying long-term energy budget and by reducing average risk backlog by up to 33.3\% in the dynamic environmental corruption scenario.
• 中文摘要 在部署分层视觉模型处理关键任务时，无人机边缘系统必须自适应地更新模型，以在低层次环境破坏下维持推理可靠性。然而，现有工作忽视了模型更新的最佳时机、依赖实时专家标签的不切实际性，以及无人机显著的带宽和能耗限制。本文提出了一个联合模型更新调度与资源分配框架，旨在最大化无人机边缘智能系统的长期语义准确性和资源效率。为解决无标签语义评估的挑战，我们制定了在线语义不一致率（OSDR）作为及时更新触发的代理指标，从而实现细粒度的敏感度感知结构同步（SASS）。此外，为克服混合动作空间中的维度问题和有效绑定的长期能量预算，我们提出了一种李雅普诺夫引导的离散强化学习算法，该算法执行动作空间维度减小，并将约束转化为虚拟队列稳定性问题。基于真实流量痕迹的实验结果表明，所提出的框架在语义恢复效率和更新触发精度方面始终优于代表性基线，满足长期能源预算，并在动态环境腐败场景下将平均风险积压减少高达33.3%。

ResidencyRL: Reinforcement Learning in Simulated Clinical Environments

住院医师RL：模拟临床环境中的强化学习

• Authors: Valentin Liévin, Samuel Schmidgall, Tim Strother, Alex Bijamov, Akshay Goel, Anil Palepu, Chunjong Park, Vahid Balazadeh, Min Woo Sun, Marius Guerard, Justin Chen, Dave Steiner, Vikram Dhillon, Ibrahim Azar, Akhil Mehta, Nicholas Spetsieris, Shilpan Shah, Maen Abdelrahim, Amit Dahiya, Yun Liu, Katherine Chou, Yossi Matias, Avinatan Hassidim, Dale R. Webster, Quoc V. Le, Raia Hadsell, Joelle Barral, Carey Radebaugh, Aleksandra Faust, Shekoofeh Azizi, Mike Schaekermann, Po-Hsuan Cameron Chen, Tao Tu, David Racz, Lin Yang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.07418
• Pdf link: https://arxiv.org/pdf/2608.07418
• Abstract In medical education, physicians convert academic knowledge into clinical expertise through residency: years of training across thousands of encounters, with diverse sources of feedback and progressively greater autonomy. Much of clinical reasoning relies on the patient encounter, a dialogue in which a clinician elicits history, refines diagnostic hypotheses, and decides management under uncertainty. While large language models (LLMs) excel on static medical benchmarks, methods to optimize the full sequence of clinical decisions remain underdeveloped. We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters (up to 60 dialogue turns and 8 tool calls per trajectory). ResidencyRL pairs the policy agent with LLM simulators capable of complex, adversarial behaviors, training against a structured reward aligned to diagnostic accuracy, management quality, communication, documentation, and safety. On held-out evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions (88.0% vs. 81.0%) and reduces missed red flag rates by 31%, demonstrating rigorous mitigation of premature closure. Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons. The procedural competencies transfer to unseen benchmarks: the agent outperforms the base model across all six clinical axes of the AMIE multi-visit benchmark, and shows consistent directional improvements on AgentClinic and CRAFT-MD. Our findings demonstrate that sequential clinical decision-making can be effectively learned through multi-turn RL in simulation, yielding robust, generalizable capabilities, paving the way towards clinical mastery. Prospective validation with real-world workflows remains necessary to establish clinical utility.
• 中文摘要 在医学教育中，医生通过住院医师培训将学术知识转化为临床专长：经过数千次临床经历的多年培训，反馈来源多样，自主性逐渐增强。临床推理很大程度上依赖于患者接触，即临床医生在不确定性下提取病史、完善诊断假设并在不确定性下决定管理的对话。虽然大型语言模型（LLMs）在静态医学基准上表现出色，但优化临床决策全流程的方法仍然不成熟。我们介绍ResidencyRL，这是一种强化学习（RL）方法，用于通过模拟多回合临床会面（每条轨迹最多60次对话回合和8次工具调用）训练临床人工智能（AI）代理。ResidencyRL将政策代理与能够执行复杂对抗行为的LLM模拟器结合，针对符合诊断准确性、管理质量、沟通、文档和安全的结构化奖励进行训练。在长期评估中，ResidencyRL代理在恶劣条件下诊断准确率提升了7.0%（88.0%对81.0%），并降低了31%的遗漏红旗率，展现了对过早闭合的严格缓解。盲法专家临床医生验证了这些提升，在87.6%的并排比较中更倾向于受过训练的代理。操作能力转移到未见基准：代理在AMIE多次就诊基准的六个临床轴上均优于基础模型，并在AgentClinic和CRAFT-MD上持续展现出方向性提升。我们的发现表明，通过多轮强化学习模拟，可以有效学习顺序临床决策，带来稳健且可推广的能力，为临床精通铺平道路。在实际工作流中进行前瞻性验证仍是确立临床效用的关键。

Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

Fisher-R1：培训大型语言模型代理以实现可靠假设检验

• Authors: Jiacheng Miao, Jin Mu, Guanhua Chen, James Zou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07437
• Pdf link: https://arxiv.org/pdf/2608.07437
• Abstract Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeekV4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.
• 中文摘要 可靠的假设检验是许多实证科学主张的基础。大型语言模型（LLM）代理越来越多地被用于自动化这一过程，因为它们可以检查数据集、生成代码并进行端到端的分析。然而，我们表明它们经常存在细微的推理错误，导致即使分析正确，也会导致错误结论。现有基准测试未能捕捉这种失败模式，因为它们很少评估报告的p值是否在数据背后的假设下具有统计有效性。我们通过构建P-Bench来弥补这一空白，该基准测试包含425个开放式、现实的假设检验任务，涵盖经济学、生物学和医学领域。每个任务都需要代理选择统计方法，计算p值，并在仅给出科学假设和数据集的情况下得出结论。我们还进一步介绍了Fisher-R1，这是一款开权重LLM代理，经过训练用于通过合成任务和强化学习进行严格假设检验。在P-Bench上，Fisher-R1-14B在基础上有显著提升，超越了包括GPT-5.4和DeepSeekV4-Pro在内的强力专有和开源基线，单次试验成功率平均相较于DeepSeek-V4-Pro提升21%，最具挑战性的任务提升了最多26%。我们的结果表明，当前LLM代理缺乏可靠的统计推理来进行假设检验，而在具有验证统计奖励的任务上进行强化学习，其可靠性显著提升。

CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

CreativeInstruct：可扩展教学LLM以平衡质量、创造力和多样性

• Authors: Ananya Sahu, Mohit Bansal, Elias Stengel-Eskin
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07460
• Pdf link: https://arxiv.org/pdf/2608.07460
• Abstract While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.g., story generation) as well as those that require it implicitly, e.g., reinforcement learning (RL). We instead propose CreativeInstruct, a scalable instruction-tuning method that teaches LLMs to balance creative, base-model-like generations with the quality of post-trained models, by learning to inject special [StartCreativity] spans that bias generation toward creativity. Furthermore, we introduce a structural diversity metric based on graph edit distance, which captures narrative level variation missed by purely lexical and semantic metrics. On narrative generation, CreativeInstruct matches or exceeds the diversity of both multi-model baselines and distilled variants of their outputs, without sacrificing quality or requiring multiple models at inference time. These results are mirrored in our human evaluation, where we find that annotators rate CreativeInstruct generations as more creative than the post-trained LLMs' generations in 70.3% of cases. We also show the benefits of creative models as a substrate for RL: GRPO applied to a CreativeInstruct checkpoint improves by ~4% on AMC and ~5% points on MATH over the same training applied to the post-trained checkpoint.
• 中文摘要 虽然后训练提升了大型语言模型（LLMs）的能力，但通常降低了它们的输出多样性和创造力，对明确需要创造力的任务（如故事生成）以及隐性需要创造力的任务（如强化学习（RL）产生负面影响。我们提出了CreativeInstruct，这是一种可扩展的指令调优方法，通过学习注入特殊的[StartCreativity]跨度，从而平衡创造性、基模型类的生成与后训练模型的质量，从而学习注入这种偏向创造力的生成。此外，我们引入了基于图编辑距离的结构多样性指标，捕捉了纯词汇和语义指标所忽略的叙事层面变异。在叙事生成方面，CreativeIninstruction能够匹配或超越多模型基线及其精炼输出变体的多样性，同时不牺牲质量，也不要求在推理时使用多个模型。这些结果在我们的人类评估中也得到了体现，我们发现标注者在70.3%的案例中，将CreativeIninstruction的生成比后训练LLM的生成更具创造性。我们还展示了创造性模型作为强化学习基础的优势：在CreativeIninstruction检查点上，应用GRPO在AMC上提升了~4%，在MATH中提升~5%。

SimWAM: A Simple World Action Model for End-to-End Autonomous Driving

SimWAM：端到端自动驾驶的简单世界行动模型

• Authors: Zongchuang Zhao, Xin Zhou, Tianyang Xu, Zhengyang Sun, Kaixuan Zhou, Honglin Li, Dingkang Liang, Xiang Bai
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.07468
• Pdf link: https://arxiv.org/pdf/2608.07468
• Abstract World-Action Models (WAMs) improve end-to-end autonomous driving by transferring video dynamics priors to action prediction, but existing methods require costly future generation at inference. We present SimWAM, a simple yet effective WAM that uses video generation purely as a training signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing the video branch to be discarded after training and leaving a self-contained planner that directly predicts trajectories. Since the two experts share no parameters and interact only through a unified attention interface, the video backbone could be replaced and the action expert scaled independently without modifying the learning objective or inference pipeline. We further apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Our SimWAM achieves $91.5$ PDMS on NAVSIM, surpasses state-of-the-art WAM-based planners with substantially lower latency, and transfers zero-shot to nuScenes. These results position SimWAM as a simple yet solid baseline that could readily benefit from advances in video generation for efficient autonomous driving. The code and model weights are available at this https URL
• 中文摘要 世界动作模型（WAM）通过将视频动态先行传递到动作预测，提升端到端自动驾驶，但现有方法需要在推断时产生成本高昂的未来生成。我们介绍SimWAM，一种简单但高效的WAM，纯粹将视频生成作为训练信号。它通过联合流匹配共同训练预训练视频专家和轻量级动作专家。孤立注意力掩码使动作预测独立于未来帧，允许视频分支在训练后丢弃，留下一个自包含的计划器直接预测轨迹。由于两位专家不共享参数，仅通过统一的注意力接口互动，视频骨干可以被替换，动作专家可以独立扩展，而无需修改学习目标或推理流水线。我们进一步应用强化学习，优化超越轨迹模仿的组合驱动奖励。我们的SimWAM在NAVSIM上实现了91.5美元的PDMS计算，超越了基于WAM的先进规划器，延迟显著降低，并将零拍摄数据传输到nuScenes。这些结果使SimWAM成为一个简单而坚实的基线，有望借助视频生成技术的进步，实现高效的自动驾驶。代码和模型权重可在此 https URL 获取

Keyword: diffusion policy

There is no result