生成时间: 2026-08-14 17:08:44 (UTC+8); Arxiv 发布时间: 2026-08-14 20:00 EDT (2026-08-15 08:00 UTC+8)

今天共有 21 篇相关文章

Keyword: reinforcement learning

Multi-AUV Ad-hoc network-based Target Tracking: A Value Gradient Guidance Multi-Agent Diffusion Reinforcement Learning Approach

多AUV基于自组网络的目标跟踪：一种数值梯度引导多智能体扩散强化学习方法

• Authors: Jiaao Ma, Chuan Lin, Guangjie Han, Shengchao Zhu, Qian Zhu, Ying Liu, Zhenyu Wang
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.12436
• Pdf link: https://arxiv.org/pdf/2608.12436
• Abstract Multi-AUV ad-hoc network-based target tracking requires networked autonomous underwater vehicles (AUVs) to cooperatively track maneuvering targets under constrained acoustic communication, dynamic topology, and uncertain ocean disturbances. Although multi-agent reinforcement learning (MARL) enables decentralized coordination through centralized training, existing methods suffer from high-dimensional joint state-action modeling, noise-sensitive policy generation, leading to unstable training and degraded tracking. To address these issues, we propose VGG-MADiffRL, a value-gradient-guided multi-agent diffusion RL algorithm, and MDCA, a diffusion?based hierarchical control architecture. Leveraging underwater mission characteristics, we model sonar detection mechanisms and ocean current disturbances, formulating cooperative tracking for multi-AUV ad-hoc networks as an MDP. The proposed MDCA constitutes a three-tier closed-loop control framework: a global intelligent control layer, a local online training layer, and a physical action execution layer. This structure enables synergistic optimization across task allocation, local decision processes, and execution feedback. Within MDCA, the local online training layer is the policy learning framework; VGG-MADiffRL builds on diffusion policies and incorporates value gradients to guide action generation in the reverse denoising process, steering the generated actions towards higher expected returns. It employs twin value networks with joint optimization and soft target updates to mitigate overestimation and training oscillations, promoting more stable convergence. Experimental results show that VGG-MADiffRL consistently achieves faster convergence, higher tracking accuracy, and smoother training dynamics in cooperative tracking scenarios, validating its effectiveness and practical engineering value in dynamic underwater settings.
• 中文摘要 多AUV临时网络目标跟踪需要网络自主水下车辆（AUV）在受限声学通信、动态拓扑和不确定海洋扰动条件下协同跟踪机动目标。尽管多智能体强化学习（MARL）通过集中训练实现了分散协调，但现有方法存在高维联合状态动作建模和噪声敏感策略生成的问题，导致训练不稳定和跟踪性能下降。为解决这些问题，我们提出了VGG-MADiffRL（一种以数值梯度引导的多智能体扩散RL算法）和MDCA（基于扩散的分层控制架构）。利用水下任务特性，我们模拟声纳探测机制和洋流扰动，作为MDP构建多AUV临时网络的协作跟踪。拟议的MDCA构成三层闭环控制框架：全局智能控制层、局部在线训练层和物理动作执行层。该结构实现任务分配、局部决策过程和执行反馈之间的协同优化。在MDCA中，本地在线训练层是策略学习框架;VGG-MADiffRL基于扩散策略，结合值梯度指导反向去噪过程的动作生成，引导生成动作朝向更高的预期回报。它采用双值网络，结合联合优化和软目标更新，减少高估和训练振荡，促进更稳定的收敛。实验结果显示，VGG-MADiffRL在协同跟踪场景下持续实现更快收敛、更高跟踪精度和更平滑的训练动态，验证了其在动态水下环境中的有效性和工程实用价值。

Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces

推理陪审团：多模型共识评估推理痕迹

• Authors: Congchao Wang, Diwakar Singh, Qiaozi Gao, Spyros Matsoukas, Yang Liu, Mahdi Namazifar
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.12585
• Pdf link: https://arxiv.org/pdf/2608.12585
• Abstract Improving reasoning LLMs requires the ability to judge the quality of long reasoning traces for effective reasoning data curation, strong training signals during reinforcement learning, and an in-depth understanding of reasoning behaviors during model performance evaluation. Additionally, surfacing reasoning mistakes that the model makes would enable improving the model's performance at runtime through providing feedback. Due to the difficulty of this complex task on long reasoning traces, single-model judges (even frontier models) do not do well at identifying reasoning defects. Additionally, leveraging frontier models during online training of reasoning LLMs is generally prohibited due to guardrails in terms of use. In this work, we introduce Reasoning Jury, a system that replaces the single judge with a jury of LLMs and a moderated consensus mechanism, to improve the fidelity of judgments for identifying reasoning defects. In reasoning jury, defects of a reasoning trace and their severity are surfaced through a deliberation where a moderator conducts a discussion amongst the jury where the jurors critique each other's judgments and get to modify their initial votes. The moderator derives a consensus through deliberation amongst jurors or consolidation of judgements. We show that Reasoning Jury with a jury of open-weight models (e.g., gpt-oss-120b) is able to significantly outperform frontier models (opus-4.6, sonnet-4.6, and gemini-3.1-pro) at correctly identifying reasoning defects. Besides accuracy performance improvements, the aggregated cost of the jury (initial verdicts, deliberations, consolidation, etc.) is a fraction (8 to 15%) of the cost of running frontier models in LLM-as-a-judge setup. We also show how these judgements can be leveraged to understand failure modes of reasoning LLMs on benchmarks, which allows much deeper understanding of a model's performance.
• 中文摘要 提升推理大型模型需要判断长推理轨迹质量以有效管理推理数据，强化学习中的强训练信号，以及对模型性能评估推理行为的深入理解。此外，揭示模型在推理中犯下的错误，可以通过反馈提升模型运行时的性能。由于这一复杂任务在长推理轨迹上难以完成，单模型评判（即使是前沿模型）在识别推理缺陷方面表现不佳。此外，在线训练推理大型语言模型通常被禁止，原因是使用上的限制。本研究引入了推理陪审团系统，该系统用大型语言模型陪审团和一个受控共识机制取代单一评审，以提高判断判断的准确性，用于识别推理缺陷。在推理陪审团中，推理痕迹的缺陷及其严重程度通过审议揭示，主持人在评审团中进行讨论，评审员相互批评并修改初始投票。主持人通过陪审员间的审议或合并判断达成共识。我们展示了由开放权重模型（如gpt-oss-120b）组成的推理陪审团，在正确识别推理缺陷方面显著优于前沿模型（opus-4.6、sonnet-4.6和gemini-3.1-pro）。除了准确性性能提升外，陪审团的总成本（初步裁决、审议、整合等）仅为在LLM即评委环境中运行前沿模型成本的一小部分（8%至15%）。我们还展示了如何利用这些判断来理解基准推理大型语言模型的失败模式，从而更深入地理解模型的性能。

Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies

超越最佳猜测：利用Evolution Strategies提升LLM解决方案覆盖范围

• Authors: Conor F. Hayes, Elliot Meyerson, Kajetan Schweighofer, Roberto Dailey, Babak Hodjat, Risto Miikkulainen, Xin Qiu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2608.12679
• Pdf link: https://arxiv.org/pdf/2608.12679
• Abstract Large Language Models (LLMs) are increasingly deployed in discovery domains such as math and science. The usual approach is to present the problem to the model and use its answer as the proposed solution. However, beyond this best guess, discovery can be enhanced by increasing test-time compute. In a process called pass@k, the model is allowed to explore the solution space and generate diverse candidate solutions. Unfortunately, the standard approach to post-training LLMs through Reinforcement Learning (RL) may limit pass@k: the model's output distribution narrows around high-reward outputs, causing the solution coverage to collapse. The alternative is to use Evolution Strategies (ES), a population-based, gradient-free post-training method that optimizes directly in weight space through random perturbations. As this paper shows, ES achieves consistently higher pass@k than RL and produces a broader output distribution with greater solution coverage. This coverage in turn makes it possible to achieve better results in e.g. standard math benchmarks. Thus, ES provides a better foundation for post-training in discovery problems and other domains where diverse solution coverage is critical.
• 中文摘要 大型语言模型（LLMs）越来越多地被应用于数学和科学等发现领域。通常的方法是将问题呈现给模型，并以其答案作为提出的解。然而，除了这种最佳猜测，发现还可以通过增加测试时间计算来增强。在称为pass@k的过程中，模型可以探索解空间并生成多样化的候选解。不幸的是，通过强化学习（RL）进行后训练LLM的标准方法可能会限制pass@k：模型的输出分布在高奖励输出附近缩小，导致解决方案覆盖率崩溃。另一种选择是使用进化策略（ES），这是一种基于群体、无梯度的后训练方法，通过随机扰动直接在权重空间中优化。正如本文所示，ES始终比强化学习实现更高的pass@k，并产生更广泛的输出分布和更大的解覆盖。这种覆盖反过来使得在诸如标准数学基准测试中取得更好的结果成为可能。因此，ES为发现问题及其他需要多样化解覆盖的领域后培训提供了更好的基础。

Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence

空间记忆代理：基于经验的程序记忆，用于空间智能

• Authors: Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, Hao Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.12743
• Pdf link: https://arxiv.org/pdf/2608.12743
• Abstract Spatial intelligence is becoming a foundation for embodied agents, robotic planning, and multimodal assistants. To improve the spatial reasoning ability of VLM agents, existing work has mainly followed two lines. One line uses post-training methods, such as supervised fine-tuning and reinforcement learning. Another line adopts an agentic paradigm in which the model calls external spatial tools, such as depth estimation and 3D reconstruction tools, to gather intermediate spatial evidence. We study a complementary and underexplored route: Can a frozen VLM agent improve its spatial reasoning through \textbf{parameter-update-free self-evolution}, without depending on external expert spatial tools at inference time? We present \textbf{Spatial Memory Agent (SMA)}, an \textbf{experience-grounded runtime framework} that converts verified spatial experience into reusable transferable lessons. In a verifiable spatial environment, SMA queries the frozen VLM, obtains a predicted answer and reward, and uses \textbf{verifier-guided reflection} to distill compact transferable lessons from spatial experience. SMA further assigns each lesson a \textbf{Transfer Reliability Score (TRS)}, which is initialized uniformly and calibrated from later retrieval outcomes as visit evidence of future transfer reliability. During \textbf{read-only deployment}, SMA retrieves lessons by semantic filter and similarity-TRS combined ranking, allowing the retrieved memory to guide frozen model inference. Across five representative spatial benchmarks and four base VLMs, SMA achieves the highest macro average in every base-model block and the best accuracy among the evaluated methods in most of the 20 evaluations, establishing a practical parameter-update-free path for spatial self-evolution across the evaluated frozen model scales and environments.
• 中文摘要 空间智能正成为具身智能体、机器人规划和多模态助手的基础。为了提升VLM智能体的空间推理能力，现有工作主要遵循两条路线。一条线采用后训练方法，如监督微调和强化学习。另一条线采用智能体范式，模型调用外部空间工具，如深度估计和三维重建工具，以收集中间空间证据。我们研究一条互补且鲜为人知的路径：冻结的VLM智能体能否通过\textbf{参数更新无更新的自我进化}提升空间推理能力，而无需在推理时依赖外部专家空间工具？我们介绍\textbf{空间记忆代理（SMA）}，一个\textbf{基于体验的运行框架}，将验证的空间体验转化为可重复使用的可转移课程。在可验证的空间环境中，SMA查询冻结的VLM，获得预测答案和奖励，并利用\textbf{验证者引导反思}从空间经验中提取紧凑可转移的课程。SMA进一步为每个课程分配一个\textbf{迁移可靠性评分（TRS）}，该评分由后续检索结果统一初始化并校准，作为未来迁移可靠性的访问证据。在\textbf{只读部署}期间，SMA通过语义滤波和相似度-TRS联合排序检索课程，使检索到的记忆指导冻结模型推断。在五个具有代表性的空间基准测试和四个基础VLM中，SMA在每个基模型块中实现了最高的宏观平均值，在大多数20次评估中，评估方法中准确性最高，建立了在被评估的冻结模型尺度和环境中空间自我演化的实用无参数更新路径。

Decentralized Multi-Player Q-Learning in Episodic Markov Decision Processes with Information Asymmetry

信息不对称的分集式马尔可夫决策过程中的去中心化多玩家Q-学习

• Authors: Larissa Xu, King Bi, William Chang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.12753
• Pdf link: https://arxiv.org/pdf/2608.12753
• Abstract We study decentralized multi-player reinforcement learning in episodic tabular Markov decision processes (MDPs) under three forms of information asymmetry: (A) unobserved actions with common rewards, (B) observed actions with independent rewards, and (C) unobserved actions with independent rewards. Players cannot communicate during learning but may agree on a protocol a priori. For Problems A and B we propose \texttt{mQ-learning} and \texttt{mQ-learning-intervals}, achieving $\tilde{O}(\sqrt{H^4 S A_{\text{joint}}\, T})$ regret, where $H$ is the horizon, $S$ the state count, $T = KH$ the total steps, and $A_{\text{joint}} = \prod_{i=1}^M |\mathcal{A}i|$ the joint action space across $M$ players. For Problem C we give \texttt{mEXC} and \texttt{mEXC-Bellman}, two-phase explore-then-commit algorithms with regret $\tilde{O}(H (S A{\text{joint}})^{1/3} T^{2/3})$. Against the centralized joint-action benchmark, decentralized learning under information asymmetry matches the single-agent Q-learning rate of \cite{jin2018q} up to logarithmic factors. Because $A_{\text{joint}}$ grows exponentially in $M$, the bounds are most meaningful for small $M$ or small per-player action sets.
• 中文摘要 我们研究了在三种信息不对称形式下，情节表马尔可夫决策过程（MDP）中的去中心化多玩家强化学习：（A）未观察到的动作共享奖励，（B）观察到的动作获得独立奖励，以及（C）未观察到的行为且奖励独立。玩家在学习过程中无法进行交流，但可以先验达成协议。对于问题A和B，我们提出\texttt{mQ-learning}和\texttt{mQ-学习区间}，实现$ \tilde{O}（\sqrt{H^4 S A_{\text{joint}}\， T}）$ regret，其中$H$为地平线，$S$为状态计数，$T = KH$为总步数，$A_{\text{joint}} = \prod_{i=1}^M |\mathcal{A}i|$ 为跨$M$玩家的联合行动空间。对于问题C，我们给出了 \texttt{mEXC} 和 \texttt{mEXC-Bellman}，两阶段探索-然后提交算法，后悔为 $\tilde{O}（H （S A{\text{joint}}）^{1/3} T^{2/3}）$。针对中心化联合动作基准，信息不对称下的去中心化学习在对数因子下与单代理的 \cite{jin2018q} Q-学习率相匹配。由于 $A_{\text{joint}}$ 在 $M$ 中呈指数增长，边界对于小$M$或每玩家小动作集最有意义。

Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents

超越结果奖励：深度搜索代理的步骤级自我提炼策略优化

• Authors: Haoze Wu, Chuqiao Kuang, Tianyi Zhuang, Xiaoguang Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.12764
• Pdf link: https://arxiv.org/pdf/2608.12764
• Abstract Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment. On-policy self-distillation (OPSD) addresses this by using the model's own logits as dense token-level teachers, but extending it to search agents introduces a fundamental tension: the teacher, having access to privileged information such as the correct answer, produces a distribution that differs systematically from the student's exploration-based reasoning, and naive distillation causes the student to inherit this information asymmetry rather than learn better search strategies. We resolve this tension through two contributions. First, we construct Evidence Anchors, which are concise, step-level evidence snippets extracted from the web, as privileged information that captures key reasoning steps without revealing the entire answer path. Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher-student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories. This design decouples what to update from how much to update: the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories are left untouched, preserving their diversity. On Qwen3-8B, SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES, surpassing or matching GRPO trained with twice as many gradient steps while adding only about 5 percent overhead per step from a single additional forward pass.
• 中文摘要 深度搜索代理在跨越数十步的轨迹上运行，而标准强化学习每个轨迹仅提供单一的结果奖励，这对有效的学分分配来说过于稀疏。策略自蒸馏（OPSD）通过使用模型自身的logit作为密集的token级教师来解决这个问题，但将其扩展到搜索代理则引入了一种根本张力：教师能够访问如正确答案等特权信息，产生的分布与学生基于探索的推理系统性地不同，而朴素的提炼则使学生继承了这种信息不对称，而非学习更好的搜索策略。我们通过两个贡献解决了这种张力。首先，我们构建了证据锚点，即从网络中提取的简明的步骤级证据片段，作为特权信息，捕捉关键推理步骤而不揭示全部答案路径。其次，我们提出阶级自提炼策略优化（SSPO），将师生分歧转换为GRPO中的阶级优势权重，仅应用于错误轨迹。该设计将更新内容与更新量分离：结果奖励决定政策变更方向，教师在每一步调节其强度。正确轨迹保持不变，保持多样性。在Qwen3-8B中，SSPO在浏览计算、GAIA和FRAMES中持续优于GRPO，超过或匹敌以两倍梯度步训练的GRPO，且每步仅增加约5%的额外前向传递开销。

Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs

超越正确性：混合思维MLM中的基准对标与反应行为的对齐

• Authors: Xinming Wang, Weinong Wang, Hongming Yang, Yansong Lin, Zheng Ruan, Shangpin Peng, Qiming Peng, Nan Qiao, Fengyuan Lu, Guoqing Ma, Marito Li, Songyang Zhang, Saiyong Yang, Han Hu, Yonglong Tian, Xu-Yao Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.12781
• Pdf link: https://arxiv.org/pdf/2608.12781
• Abstract Hybrid-thinking multimodal large language models (MLLMs) allow a single model to alternate between deliberative thinking and latency-efficient non-thinking inference. Although these modes differ in reasoning budget, their delivered responses should satisfy the same user-facing standard. Correctness alone may not characterize this response quality; we therefore evaluate task accuracy and response-pattern failures as complementary outcomes. We study this gap through \textbf{response-pattern alignment}: whether thinking and non-thinking interfaces preserve acceptable final-response behavior. We introduce \textbf{PatternEval}, a failure-enriched diagnostic benchmark comprising 2,415 multimodal prompts spanning visual perception and grounding, structured image understanding, and multimodal knowledge reasoning. PatternEval tests four recurrent failures: chain-of-thought leakage, response repetition, logical contradiction, and performative reasoning. Response-pattern failures are widespread across models from different providers, with non-thinking inference exhibiting substantially higher failure rates and thereby creating systematic misalignment between thinking and non-thinking interfaces. Motivated by this diagnosis, we develop \textbf{PatternRM}, a response-level reward model, and \textbf{PatternRL}, which introduces pattern-specific penalties during reinforcement learning. Experiments on Qwen3-VL-4B and Qwen3-VL-8B show that incorporating pattern-specific penalties into reinforcement learning can mitigate cross-mode misalignment while incurring a marginal task performance trade-off. Together, PatternEval and PatternRL provide an evaluation-and-training framework for aligning user-visible response patterns across hybrid-thinking interfaces.
• 中文摘要 混合思维多模态大型语言模型（MLLM）允许单一模型在深思思考和延迟高效的非思考推断之间交替进行。尽管这些模式在推理预算上有所不同，但它们交付的响应应满足相同的面向用户标准。仅凭正确性可能无法定义这种响应质量;因此，我们将任务准确性和反应模式失效视为互补的结果。我们通过 \textbf{反应模式对齐}来研究这一差距：思考与非思考接口是否保持可接受的最终反应行为。我们介绍了 \textbf{PatternEval}，这是一个包含2,415个多模态提示的失误丰富诊断基准，涵盖视觉感知与基础、结构化图像理解以及多模态知识推理。PatternEval 测试了四种反复出现的失败：思维链泄漏、反应重复、逻辑矛盾和表演性推理。反应模式失败在不同提供者的模型中普遍存在，非思考推断的失败率显著更高，从而导致思考与非思考接口之间系统性错位。基于这一诊断，我们开发了 \textbf{PatternRM}，一个响应级奖励模型，以及 \textbf{PatternRL}，后者在强化学习中引入了模式特定的惩罚。对 Qwen3-VL-4B 和 Qwen3-VL-8B 的实验表明，将模式特定的惩罚纳入强化学习可以减轻跨模式错位，同时带来边际任务性能的权衡。PatternEval 和 PatternRL 共同提供了一个评估与训练框架，用于在混合思维界面中对齐用户可见的反应模式。

HumanoidVLN: A Physics-Grounded Simulator and Benchmark for Vision-Language Navigation Across Diverse Humanoid Embodiments

HumanoidVLN：基于物理的模拟器和跨类人形态视觉语言导航的基准测试

• Authors: Quan-Dung Pham, Anh Dao, The-Anh Nguyen, Minh Nguyen-Dinh, Phuong Nam Dang, Tri Pham, Hung Tran, Bach Dao, Tuyen P. Le, Truong Nguyen, Quan Nguyen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.12860
• Pdf link: https://arxiv.org/pdf/2608.12860
• Abstract Vision-Language Navigation (VLN) for humanoid robots poses challenges existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled agents, humanoid morphologies vary across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. We present HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments. Built on NVIDIA Isaac Sim, our platform supports an extensible set of humanoid configurations, demonstrated on four robots (Unitree G1, Unitree H1, Internal-A, Internal-B) spanning 10-12 lower-body DoF and heights from 1.17m to 1.80m, via a hierarchical control stack combining a reinforcement learning locomotion policy with interchangeable PD or MPC path trackers. New robots and VLN models integrate with minimal effort; we demonstrate compatibility with NaVILA, DualVLN, StreamVLN, and JanusVLN. Environments are drawn from artist-designed scenes and 3D Gaussian Splatting reconstructions, filtered for navigable areas exceeding 100 square meters. Instructions are generated by a dual generator-reviewer plus paraphraser multi-agent pipeline with human-in-the-loop verification, yielding 933 collision-aware reference episodes, each paired with one fine-grained instruction and three coarse-grained stylistic variants (formal, natural, casual). Across four models and four embodiments, JanusVLN achieves the highest mean success rate of 43.55% and nDTW of 48.38. In a 20-episode sim-to-real pilot with DualVLN and the Unitree G1, navigation errors correlate strongly (r=0.935), with a mean absolute difference of 0.68m and mean trajectory similarity of 0.782 (+/-0.188) nDTW. These results highlight the interaction between VLN models, controllers, and humanoid embodiments under physical execution. Code, benchmark, and data will be released upon acceptance at this https URL.
• 中文摘要 人形机器人的视觉语言导航（VLN）面临现有基准测试未能解决的挑战：双足行走施加了轮式智能体所缺乏的物理约束，类人形态在不同平台上存在差异，且自我中心的观察被运动引起的摄像动态扭曲。我们介绍了HumanoidVLN，一个基于物理的模拟器和VLN在多种类人生物表现中的基准测试。基于NVIDIA Isaac Sim，我们的平台支持一套可扩展的人形配置，已在四台机器人（Unitree G1、Unitree H1、Internal-A、Internal-B）上演示，覆盖10-12个下半身景深，高度从1.17米到1.80米，通过层级控制栈结合强化学习运动策略与可互换的PD或MPC路径追踪器实现。新型机器人和VLN模型集成轻松完成;我们展示了与NaVILA、DualVLN、StreamVLN和JanusVLN的兼容性。环境取材自艺术家设计的场景和3D高斯喷溅重建，并筛选出超过100平方米的可导航区域。指令由双生成器-审查器加释义多代理流水线生成，支持人工在环验证，生成933集碰撞感知参考集，每集配对一条细粒度指令和三种粗粒度风格变体（正式、自然、休闲）。在四个模型和四个实例中，JanusVLN实现了最高的平均成功率43.55%，nDTW达到48.38%。在一次20集的模拟到现实试点中，DualVLN和Unitree G1的导航误差高度相关（r=0.935），平均绝对差为0.68m，平均轨迹相似度为0.782（+/-0.188） nDTW。这些结果突出展示了VLN模型、控制器与类人生物在物理执行下的相互作用。代码、基准和数据将在接受后发布于该HTTPS网址。

SPARED: Reasoning-Based AI-Generated Image Detection via Adversarially Edited Data

SPARED：基于推理的AI生成图像检测，通过对抗性编辑数据进行

• Authors: Yicheng Bao, Xiahui Guo, Xuhong Wang, Xin Tan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.12876
• Pdf link: https://arxiv.org/pdf/2608.12876
• Abstract Detecting AI-generated images is only half the task: a deployed detector must also justify its verdict, yet existing detectors inherit three failure modes from their training data: real and fake images collected from different sources invite provenance shortcuts, supervised explanation corpora teach templated rationales, and a static forgery corpus leaves the decision boundary standing still while generators keep moving. We introduce \methodname{}, an adversarial reinforcement learning framework that pits two heterogeneous models against each other. A diffusion image editor learns to edit real photographs into fake counterparts of those same photographs that fool the current detector, while a reasoning MLLM learns to expose them with a verdict grounded in free-form reasoning. Both rewards are shortcut-proof by design: the attacker is credited only when its edit is faithfully executed, and the defender only when its verdict is correct. As the two models alternate, each round's attacker regenerates a harder training pool aimed at the current detector's blind spots, so the detector must generalize rather than memorize any fixed artifact distribution. Although the explanation is never rewarded, its quality rises round over round as a side effect of accuracy-only training. A detector trained within this loop improves monotonically across rounds on each of three external benchmarks.
• 中文摘要 检测AI生成图像只是任务的一半：部署的检测器还必须证明其判决的合理性，但现有检测器从训练数据中继承了三种失败模式：来自不同来源的真实和虚假图像会带来来源识别捷径，监督解释语料库教授模板化的理由，静态伪造语料库则使决策边界停滞不前，而生成器则继续运转。我们引入了 \methodname{}，这是一种对抗强化学习框架，将两个异构模型相互对抗。扩散图像编辑器学习将真实照片编辑成伪造的假照片以欺骗当前检测器，而推理型 MLLM 则学会以自由形式推理为基础的判决来曝光。这两种奖励设计上都是无可侵犯的：攻击者只有在其编辑忠实执行时才获得认可，只有当其裁决正确时才会被攻击者获得认可。当两个模型交替使用时，每轮攻击者会重新生成一个更难的训练池，针对当前探测器的盲区，因此探测器必须泛化而非记忆任何固定的伪影分布。虽然解释从未得到奖励，但其质量随着纯准确训练的副作用逐轮提升。在该循环中训练的探测器在三个外部基准测试中单调地在各轮次中单调地提升。

Towards Socially Compliant Navigation in Deep Reinforcement Learning via Proxemics-Based Reward Modeling

通过基于近距离的奖励建模，迈向深度强化学习中的社会合规导航

• Authors: Takieddine Soualhi (CHROMA), Jacques Saraydaryan (CPE, CHROMA), Laetitia Matignon (UCBL)
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.12917
• Pdf link: https://arxiv.org/pdf/2608.12917
• Abstract Developing effective robot navigation methods in crowded environments is essential for real-world applications. Although recent deep reinforcement learning (DRL) methods have improved navigation performance in crowded environments, they often focus primarily on task-centric objectives and underrepresent social compliance objectives. In this paper, we introduce a novel proxemics-based reward formulation for DRL social navigation that provides a dense, interpretable social learning signal while maintaining navigation efficiency. Our approach models each human's personal space as a radial Gaussian-mixture field derived from Hall's proxemics theory and computes a robot-centric local cost over the robot's field of view. We integrate the proposed reward into established DRL navigation methods and evaluate it in simulation across multiple crowd scenarios, reward baselines, and crowd densities using both navigation metrics and social metrics. Results show that the proposed reward consistently improves social metrics in simulation while maintaining competitive navigation performance relative to the compared reward models.
• 中文摘要 在拥挤环境中开发有效的机器人导航方法对于实际应用至关重要。尽管最新的深度强化学习（DRL）方法提高了拥挤环境中的导航性能，但它们往往主要关注任务中心目标，忽视了社会合规目标。本文介绍了一种基于近量学的新奖励表述，用于DRL社交导航，提供密集且可解释的社会学习信号，同时保持导航效率。我们的方法将每个人的个人空间建模为基于Hall近似学理论的径向高斯混合场，并计算机器人视野内的机器人中心局部成本。我们将拟议的奖励整合进既有的DRL导航方法，并在多个人群场景、奖励基线和人群密度中结合导航指标和社会指标进行模拟评估。结果表明，所提奖励在模拟中持续提升社会指标，同时保持相较于对比奖励模型的竞争导航性能。

Momentum as Residual-Driven Multiplier Correction for Deep Learning Optimization

动量作为深度学习优化的残差驱动乘数修正

• Authors: Zhixin Ren, Yau Lyu, Congrong Li, Liping Zhang, Shengbo Eben Li
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.12925
• Pdf link: https://arxiv.org/pdf/2608.12925
• Abstract Momentum-based optimizers are widely used in modern deep learning, yet the relations among momentum recursion, update geometry, and acceleration remain only partially understood. We develop an $\textbf{A}$DMM-$\textbf{I}$nspired $\textbf{M}$omentum (AIM) framework based on residual-penalty variable splitting, which interprets momentum as a multiplier-like correction driven by the splitting residual. AIM recovers the exponential moving average of gradients from an ADMM-style multiplier update and separates two mechanisms that are usually intertwined in practical optimizers: the residual penalty determines the update geometry, whereas the approximation of the objective-related subproblem determines the acceleration form. Building on AIM, we propose $\textbf{R}$elativistic $\textbf{A}$daptive gradient $\textbf{D}$escent with $\textbf{A}$ccelerated $\textbf{R}$esidual (RADAR), which combines relativistic adaptive geometry, decoupled residual correction, and second-order momentum filtering to improve the update direction and momentum estimation. We establish stochastic convergence through a variance-perturbed Lyapunov drift analysis. Experiments on supervised vision learning, language modeling, and reinforcement learning show that RADAR achieves consistent improvements over strong adaptive optimizer baselines.
• 中文摘要 基于动量的优化器在现代深度学习中被广泛应用，但动量递归、更新几何和加速度之间的关系仍仅部分理解。我们基于残差惩罚变量拆分开发了$\textbf{A}$DMM-$\textbf{I}$nspired $\textbf{M}$omentum（AIM）框架，该框架将动量解释为由分裂残差驱动的乘数式修正。AIM恢复了ADMM式乘数更新的梯度指数移动平均，并分离了两种在实际优化器中通常交织的机制：残差惩罚决定更新几何，而目标相关子问题的近似决定加速度形式。基于AIM，我们提出$\textbf{R}$elativistic$\textbf{A}$daptive梯度$\textbf{D}$escent，带$\textbf{A}$ccelerated $\textbf{R}$esidual（RADAR），结合相对论自适应几何、解耦残差修正和二阶动量滤波，以提升更新方向和动量估计。我们通过方差扰动的李雅普诺夫漂移分析建立了随机收敛性。监督视觉学习、语言建模和强化学习的实验表明，RADAR相较于强自适应优化基线实现了持续的改进。

Multi-perspective Imbalance-Conscious 6G Beamforming Optimization and Performance

多视角不平衡意识的6G波束成形优化与性能

• Authors: Chukwunonso Henry Nwokoye, Blessing Oluchi Iloka, Chikwue V. Umeugoji, Christopher Anene Egemba, Nnenna D. Duroha
• Subjects: Subjects: Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.12929
• Pdf link: https://arxiv.org/pdf/2608.12929
• Abstract The study presents a systematic machine learning (ML) study of 6G-IoT beamforming optimization (6GBO) using supervised and unsupervised approaches. We compared the predictive power of network, environmental, device, and vision feature groups for 6GBO. Additionally, it addressed other unsupervised perspectives that can enhance 6GBO, including clustering network scenarios using methods such as K-means, DBSCAN, and hierarchical clustering. Several imbalance-aware experiments revealed that network features possess better prediction power than device, environmental, and vision feature groups, as evidenced by their recall, F1-score and ROC-AUC values. For unsupervised ML exploration (assessed using Elbow, Silhouette score, and Davies-Bouldin Index methods), the results indicate that the deployment environment and type of device primarily influence clustering, rather than mobility-based attributes. Furthermore, the explainability analysis showed that bandwidth, IoT sensors, and mobility possess higher global feature importance across the feature groups. In the future, we would apply deep and reinforcement learning techniques to predict throughput/latency or to optimize rewards determined by performance indicators like SNR enhancement
• 中文摘要 本研究采用监督和非监督方法，系统地进行了机器学习（ML）研究，针对6G物联网波束成形优化（6GBO）。我们比较了网络、环境、设备和视觉特征组在6GBO中的预测能力。此外，研究还探讨了其他可增强6GBO的无监督视角，包括使用K-means、DBSCAN和分层聚类等方法对网络场景进行聚类。多项意识失衡的实验显示，网络特征比设备、环境和视觉特征组具有更强的预测能力，这一点可从其召回率、F1评分和ROC-AUC值中得到证明。对于无监督机器学习探索（使用肘部、Silhouette评分和Davies-Bouldin指数方法评估），结果表明部署环境和设备类型主要影响聚类，而非基于移动性的属性。此外，可解释性分析显示，带宽、物联网传感器和移动性在功能组中具有更高的全球特征重要性。未来，我们将应用深度学习和强化学习技术来预测吞吐量/延迟，或优化由SNR增强等性能指标决定的奖励

A Deep RL based Framework for Targeted White Matter Tractography

基于深度强化学习的靶向白质牵引图框架

• Authors: Ankita Joshi
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.12960
• Pdf link: https://arxiv.org/pdf/2608.12960
• Abstract Fiber tractography's ability to reconstruct the brain's structural pathways, has made it a crucial component of modern neuroimaging, enabling detailed, non-invasive mapping of structural connectivity and supporting a wide range of neurological research and clinical applications. However, despite its importance, tractography remains a challenging task due to the inherent complexity of white matter structure and its susceptibility to false positives, which can lead to the misrepresentation of critical pathways. To overcome these limitations, in this thesis, we propose a hybrid framework that integrates reinforcement learning with supervised learning for refining RL policies, specifically tailored for tract-specific tractography. Notably, our framework does not rely on ground-truth fibers for training. Moreover, the tract-specific formulation bypasses the need for an explicit segmentation process, simplifying the overall pipeline. Our work includes two main contributions, each building upon the previous. First, we introduce a hybrid approach that combines reinforcement learning with supervised learning (specifically, GPT-based policy learning) to refine policies in a tract-specific context. Second, we propose a scalable framework for data-driven multi-policy fusion, which leverages the complementary strengths of multiple RL policies to improve tractography performance and robustness. We demonstrate the effectiveness of our framework through extensive validation on benchmark public datasets including TractoInferno, HCP, and ISMRM-2015, highlighting its ability to generalize across data sources and accurately reconstruct brain white matter tracts. We believe that these contributions represent significant advancements in the field of tractography, improving robustness, reliability, and accuracy while reducing dependence on ground-truth annotations.
• 中文摘要 纤维牵引图重建大脑结构通路的能力使其成为现代神经影像学的重要组成部分，能够详细、非侵入性地绘制结构连接，支持广泛的神经学研究和临床应用。然而，尽管其重要性，牵引图仍是一项具有挑战性的任务，因为白质结构本身复杂且易发生假阳性，可能导致关键通路的误表。为克服这些局限性，本论文提出了一个混合框架，将强化学习与监督学习整合，用于优化强化学习策略，专为特定束的牵引图量身定制。值得注意的是，我们的框架不依赖地面真实纤维进行训练。此外，束特异性表述绕过了显式分割过程，简化了整体流程。我们的工作包括两项主要贡献，均基于前述。首先，我们引入一种混合方法，将强化学习与监督学习（特别是基于GPT的策略学习）结合，以在特定路线语境中优化策略。其次，我们提出了一个可扩展的数据驱动多策略融合框架，利用多重强化学习策略的互补优势，提升追踪记录的表现和稳健性。我们通过对包括TractoInferno、HCP和ISMRM-2015在内的基准公共数据集进行广泛验证，展示了该框架的有效性，突出其跨数据源的泛化能力，并准确重建大脑白质通道。我们认为这些贡献代表了文献学领域的重大进展，提升了鲁棒性、可靠性和准确性，同时减少了对真实注释的依赖。

OGR-MARL: Option-Guided Residual Multi-Agent Reinforcement Learning for Heterogeneous USV Cooperative Pursuit in Constrained Port Waterways

OGR-MARL：受限港口水道中异质USV合作追踪的选项引导残余多代理强化学习

• Authors: Mao Jiayang, Wang Lanfeng, Peng Zhao-Han
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.12995
• Pdf link: https://arxiv.org/pdf/2608.12995
• Abstract Heterogeneous USV cooperative pursuit in constrained port waterways requires evader interception under navigation, traffic, and role constraints. This paper proposes OGR-MARL, an option-guided residual multi-agent reinforcement learning framework that is decoupled from a specific MARL algorithm. OGR-MARL integrates shared evader belief, role-conditioned option targets, adaptive rule penalties, and residual policy learning, allowing different MARL algorithms to learn corrective actions on top of rule-guided behaviors rather than exploring constrained port environments from scratch. We instantiate OGR-MARL with representative continuous-control MARL backbones, including MADDPG, MATD3, MAPPO, and MASAC, yielding OGR-MADDPG, OGR-MATD3, OGR-MAPPO, and OGR-MASAC. Experiments in an abstract Xiazhimen port-waterway scenario show that the OGR-MASAC instantiation achieves a 75.0% capture rate, promising mission-effective rule compliance, and the best heterogeneous coordination among the tested methods. Without retraining, zero-shot transfer to a QGIS/AIS-informed Xiazhimen map achieves promising results, demonstrating the generalization potential of OGR-MARL in more complex port scenarios.
• 中文摘要 在受限港口水道中，异构USV协同追捕需要在导航、交通和角色约束下进行规避拦截。本文提出了OGR-MARL，一种选项引导的残余多代理强化学习框架，该框架与特定MARL算法解耦。OGR-MARL整合了共享规避信念、角色条件选项目标、自适应规则惩罚和残余策略学习，使不同的MARL算法能够基于规则引导行为学习纠正措施，而无需从零开始探索受限港口环境。我们实例化OGR-MARL与代表性的连续控制MARL骨干，包括MADDPG、MATD3、MAPPO和MASAC，生成OGR-MADDPG、OGR-MATD3、OGR-MAPPO和OGR-MASAC。抽象的下直门港口-水道场景实验显示，OGR-MASAC实现实现75.0%的捕获率，有望实现任务有效规则合规性，并在测试方法间实现最佳异构协调。无需重新训练，零射点转移至QGIS/AIS知情下直线图可取得良好效果，展示了OGR-MARL在更复杂港口场景中的泛化潜力。

Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

时间GRPO：视觉-语言-行动强化学习中的超越轨迹级学分

• Authors: Yao Zhou, Hang Gao, Fengge Wu, Changwen Zheng, Wenwen Qiang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.13026
• Pdf link: https://arxiv.org/pdf/2608.13026
• Abstract Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.
• 中文摘要 结果驱动强化学习提供了一种可扩展的方式，从稀疏的任务成功反馈中，进行视觉-语言-行动（VLA）策略的后续训练。在基于GRPO的VLA后训练中，轨迹中的每个动作都应用一个推广级优势。完成多个有效阶段但后续失败的推广，可能会惩罚那些产生早期进展的动作。我们称之为轨迹级信用混叠。时间GRPO通过构建可检测的任务阶段，将每个部署与阶段特定的动作间隔对齐，并仅比较进入同一阶段的部署来解决这一问题。由此产生的阶段优势将在一次策略更新中应用到相应的区间。在RoboTwin 2.0中，时间GRPO提升了任务成功率和采样效率，并在任务视野内实现了持续的提升。LIBERO-LONG的受控更新保留了共享的先决条件阶段，并将改进集中在推广结果差异的第一阶段。

Pareto-Aware Hierarchical Reinforcement Learning for Online Resource Allocation in RIS-assisted Large-Scale IoT Systems

帕累托感知层级强化学习用于RIS辅助的大型物联网系统中的在线资源分配

• Authors: Wenhan Xu, Jiashuo Jiang, Danny H.K. Tsang
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.13032
• Pdf link: https://arxiv.org/pdf/2608.13032
• Abstract With the rapid evolution of 5G and emerging 6G networks, reconfigurable intelligent surfaces (RIS) have become a critical technology for enhancing wireless communication scenarios. However, optimizing RIS-assisted multi-user systems typically introduces high-dimensional physical layer variables and non-convex Pareto-optimal rate sets, posing severe computational challenges for real-time applications. To address these limitations, this paper proposes a dimension-reduced, hierarchical reinforcement learning (RL) framework, termed Pareto-aware autoencoder-assisted RL (PAAERL), to optimize online resource allocation in RIS-assisted Internet of Things (IoT) networks. Our approach first substitutes high-dimensional continuous RIS beamforming variables with lower-dimensional weight vectors that strictly represent the Pareto-optimal frontier, theoretically avoiding geometric information loss across both convex and non-convex rate regions. To further mitigate the curse of dimensionality in dense networks, an autoencoder architecture is integrated to execute a secondary, data-driven compression phase, mapping the priority space into a highly condensed continuous latent action space. Extensive simulations conducted across practical communication scenarios, including multi-user mobile edge computing (MEC) networks, demonstrate that the proposed PAAERL framework drastically reduces offline training times, accelerates online policy convergence, and significantly decreases overall network costs compared to state-of-the-art benchmarks, underscoring its exceptional scalability and practical viability for next-generation intelligent IoT environments.
• 中文摘要 随着5G和新兴6G网络的快速发展，可重构智能表面（RIS）已成为提升无线通信场景的关键技术。然而，优化RIS辅助多用户系统通常会引入高维物理层变量和非凸帕累托最优速率集，给实时应用带来严重的计算挑战。为解决这些限制，本文提出了一种降维、层级强化学习（RL）框架，称为帕累托感知自编码器辅助RL（PAAERL），以优化RIS辅助物联网（IoT）网络中的在线资源分配。我们的方法首先用严格表示帕累托最优前沿的低维权重矢量替代高维连续RIS波束成形变量，理论上避免了凸率和非凸率区域的几何信息丢失。为进一步缓解密集网络维度的诅咒，集成了自编码器架构，执行次级数据驱动压缩阶段，将优先级空间映射为高度凝聚的连续潜在动作空间。在包括多用户移动边缘计算（MEC）网络在内的实际通信场景中进行的广泛模拟表明，所提PAAERL框架大幅缩短离线训练时间，加速在线策略融合，并显著降低整体网络成本，相较于最先进基准测试，凸显其卓越的可扩展性和对下一代智能物联网环境的实用性。

AoI-Guaranteed Dynamic Route Planning for Connected Vehicles

AoI保证的互联车辆动态路线规划

• Authors: Sajedeh Norouzi, Maryam Ansarifard, Farshad Zeinali, Ali Nouruzi, Nader Mokari, Hamid Saeedi, Nizar Zorba
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.13083
• Pdf link: https://arxiv.org/pdf/2608.13083
• Abstract The advancement of Intelligent Transportation Sys- tems (ITS) has been significantly driven by progress in radio communication technology. Dynamic route planning, a key com- ponent of ITS, traditionally focuses on metrics such as route capacity and travel time. This paper presents a novel dual- factor approach that integrates travel time estimation and radio resource availability into an innovative route-planning scheme for connected vehicles (CVs). To address this dual-objective route planning challenge, we employ Deep Reinforcement Learning (DRL). Our approach, called AoI-Guaranteed Dynamic Route Planning (AGDRP), effectively balances travel time and Age of Information (AoI), enhancing route planning performance through adaptive learning over time. Simulation results demon- strate that AGDRP outperforms the baseline scheme, which solely focuses on travel time optimization. In fact, we show that incor- porating AoI minimization significantly enhances route planning performance beyond conventional travel-time-based approaches.
• 中文摘要 智能交通系统（ITS）的发展在很大程度上受到无线通信技术进步的推动。动态路线规划是ITS的关键组成部分，传统上关注路线容量和旅行时间等指标。本文提出了一种创新的双因素方法，将旅行时间估计和无线资源可用性整合进一种创新的连接车辆（CV）路线规划方案中。为应对这一双重目标的路线规划挑战，我们采用了深度强化学习（DRL）。我们的方法称为AoI-Guaranteed Dynamic Route Planning（AGDRP），有效平衡了旅行时间与信息时代（AoI），通过自适应学习逐步提升路线规划性能。模拟结果表明，AGDRP优于仅专注于旅行时间优化的基线方案。事实上，我们表明，纳入的AoI最小化显著提升了路线规划性能，超越了传统的基于时间的路径规划方法。

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

教导规模，而非方向：多回合多步LLM代理的验证者界限信用分配

• Authors: Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo, Chenyi Zhuang, Leilei Gan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.13179
• Pdf link: https://arxiv.org/pdf/2608.13179
• Abstract Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce $\textbf{CrEST}$, a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher. $\textbf{CrEST}$ resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that $\textbf{CrEST}$ consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling.
• 中文摘要 带可验证奖励的强化学习（RLVR）为训练多回合工具使用代理提供了验证者限制的性能上限，但其轨迹级信用分配将异质的每回合结果混为一体。策略上提炼提供密集的每个代币监督，但要么受教师限制，要么易发生梯度集中崩溃。我们引入$\textbf{CrEST}$，一种层级学分分配框架，保留了RL验证者限制的上限，同时包含来自特权自学者的密集代币级信号。$\textbf{CrEST}$在两个层级解决信用：回合分段验证优势解决回合间稀释，熵门控自教师调节细化回合内代币贡献。在BFCL V3和WildToolBench上的实验显示，$\textbf{CrEST}$在两个模型尺度上始终优于强化学习和蒸馏基线，在长轨迹和严格会话级指标上提升最大。我们的研究表明，教师在策略优化中的角色可以从确定更新方向简化为调节更新幅度，解锁密集学分分配，同时不牺牲验证者界限上限。

Rules or Character? Scaling Laws for AI Safety Design

规则还是角色？人工智能安全设计的尺度法则

• Authors: Satoshi Takahashi, Nobuji Kouno, Masaaki Komatsu, Ryuji Hamamoto
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.13345
• Pdf link: https://arxiv.org/pdf/2608.13345
• Abstract Artificial Intelligence (AI) safety systems combine character shaping (e.g., Reinforcement Learning from Human Feedback [RLHF], Constitutional AI), which modifies behavioral distributions at training time, with rule enforcement (e.g., output filters, safety classifiers), which blocks harmful outputs at inference time, yet little formal analysis exists on how their optimal balance should change as deployment scales increase. We introduce a stylized comparative-statics model that parameterizes safety design as a resource allocation alpha in [0,1] between these two approaches, incorporating scale-dependent filter degradation, common-mode failures, and character fragility -- the risk that shaped behavior degrades or collapses under novel conditions. Under a multiplicative Pareto damage model, we derive closed-form expected harm and supplement it with tail-risk (CVaR) analysis via Monte Carlo simulation. Across three scenarios (optimistic, moderate, pessimistic), the optimal alpha is interior or at the rules-only boundary and shifts weakly toward character shaping as deployment scale T grows, from negligible (Delta alpha = +0.01) to pronounced (Delta alpha = +0.21) depending on scenario. The dominant parameter is the baseline character fragility rate p^(0)_frag, which shifts alpha by 0.50 across its range -- far exceeding the effect of tail severity, filter quality, or common-mode failure probability. CVaR and expected-harm optima converge at large T. These results suggest that safety architecture decisions depend less on deployment scale per se than on the reliability of character shaping under distributional shift.
• 中文摘要 人工智能（AI）安全系统结合了字符塑造（例如，来自人类反馈的强化学习[RLHF]，宪法人工智能），在训练时修改行为分布，与规则执行（如输出过滤器、安全分类器），在推断时阻断有害输出，但对于其最优平衡如何变化，几乎没有正式分析。我们引入了一个风格化的比较静态模型，将安全设计参数化为两者之间[0,1]的资源分配α，结合了规模依赖的滤波器劣化、共模失效和特征脆弱性——即塑造行为在新条件下退化或崩溃的风险。在乘法帕累托损伤模型下，我们推导出封闭形式的预期伤害，并通过蒙特卡洛模拟补充尾部风险（CVaR）分析。在三种情景（乐观、中等、悲观）中，最优α位于内部或仅规则边界，随着部署尺度T的增长，从可忽略的（Delta alpha = +0.01）到显著（Delta alpha = +0.21）而弱地向字符塑形移动，具体取决于情景。主导参数是基线字符脆弱性率p^（0）_frag，该值在其范围内α移动0.50——远超尾部严重度、滤波器质量或共模失效概率的影响。CVaR和预期伤害最优值在大T时收敛。这些结果表明，安全架构决策更多依赖于分布偏移下字符塑形的可靠性，而非部署尺度本身。

FIRE-VLA: Failure-Informed Self-Evolution for Vision-Language-Action Models in Autonomous Driving

FIRE-VLA：自动驾驶中视觉-语言-行动模型的失败知情自我进化

• Authors: Hao Dou
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.13395
• Pdf link: https://arxiv.org/pdf/2608.13395
• Abstract Reinforcement learning improves autonomous-driving vision-language-action (VLA) models by evaluating trajectories sampled from the current policy. Group relative policy optimization (GRPO) learns from reward differences within each rollout group. When all sampled trajectories are poor, this relative signal can rank failures without identifying behavior outside the failed region. We introduce FIRE-VLA, a failure-informed self-evolution framework that converts such unresolved failures into privileged supervision for the next policy. Low-reward, low-diversity groups trigger self-distillation from a frozen round-start copy of the same model. Teacher and student have the same parameter scale, but only the teacher observes the hidden future trajectory. Supervision follows the student's generated prefix and is restricted to answer tokens, while GRPO remains active for every group. The updated policy supplies the teacher for the next round, allowing the routed failure distribution to change with the policy without requiring a larger external teacher. Starting from the same Qwen2.5-VL-3B SFT checkpoint, the comparison matches student rollout and policy-update counts. On 6,019 examples from 150 held-out nuScenes scenes, FIRE-VLA retains comparable single-sample planning, reduces G=4 mean L2 from 1.848 to 1.500 m, and lowers evaluation-persistent failure prevalence from 13.03% to 11.20%. The reduction in mean error arises mainly from rare severe rollouts rather than uniform improvement across ordinary trajectories.
• 中文摘要 强化学习通过评估当前策略中抽样的轨迹，改进自主驾驶视觉-语言-行动（VLA）模型。群体相对策略优化（GRPO）通过每个推广组内的奖励差异进行学习。当所有抽样轨迹都不理想时，该相对信号可以对失败进行排名，而无需识别失败区域外的行为。我们引入了FIRE-VLA，一种基于失败的自我演化框架，将未解决的失败转化为下一策略的特权监督。低奖励、低多样性的小组从同一模型的冻结轮次起始副本中触发自我蒸馏。教师和学生拥有相同的参数尺度，但只有教师观察隐藏的未来轨迹。监督遵循学生生成的前缀，限制于答案标记，而GRPO对每个小组保持激活。更新后的策略为教师提供下一轮准备，允许路由失败分布随政策变化，而无需增加外部教师。从同一Qwen2.5-VL-3B SFT检查点出发，比较匹配学生的部署和策略更新计数。在150个保留的nuScenes场景中的6,019个示例中，FIRE-VLA保留了可比的单样本规划，将G=4的平均L2从1.848米降至1.500米，并将评估持续失败的发生率从13.03%降至11.20%。平均误差的减少主要来自罕见的严重推广，而非普通轨迹的均匀改善。

Intern-S2-Preview: Scientific Agentic Foundation Model

实习生第二季预览：科学代理基金会模型

• Authors: Lei Bai, Jiaqi Cao, Chiyu Chen, Guanzhou Chen, Kai Chen, Guangran Cheng, Erfei Cui, Xuanlang Dai, Shengyuan Ding, Shangheng Du, Yanhui Duan, Yue Fan, Youqing Fang, Quan Gan, Yuanyuan Gao, Jiaye Ge, Lixin Gu, Yuzhe Gu, Qipeng Guo, Junjun He, Xin Hong, Ming Hu, Zhouqi Hua, Haian Huang, Junhao Huang, Zixian Huang, Minxi Jin, Lingkai Kong, Alexander Lam, Zehao Li, Zonglin Li, Tianhao Liang, Dahua Lin, Junyao Lin, Tianyang Lin, Zhouhan Lin, Jiangning Liu, Jin Liu, Kuikun Liu, Wenran Liu, Yifei Liu, Yuhong Liu, Yuhong Liu, Zhoumianze Liu, Ziyan Liu, Ziyu Liu, Haijun Lv, Han Lv, Chengqi Lyu, Le Ma, Ningsheng Ma, Zerun Ma, Haoyang Peng, Runyu Peng, Jifei Shan, Zixin Shang, Kou Shi, Xiang Shi, Qisheng Su, Xuerui Su, Hao Sun, Xiao Sun, Yanan Sun, Yu Sun, Huanze Tang, Yinghao Tang, Wenhui Tian, Zhongbo Tian, Bingli Wang, Haomin Wang, Jiarui Wang, Jingzhi Wang, Rui Wang, Xiquan Wang, Yi Wang, Zhecan Wang, Ziyi Wang, Zun Wang, Rubin Wei, Lianyi Wu, Wen Wu, Yue Wu, Yuhan Wu, Zhenyu Wu, Zijian Wu, Shuhao Xing, Jun Xu, Xingle Xu, Xuenan Xu, Xiangchao Yan, Ziang Yan, Bowen Yang, Danni Yang, Lin Yang, Zhiqi Yang, Qian Yao, Haochen Ye, Peng Ye, Jinhui Yin, Jiashuo Yu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.13505
• Pdf link: https://arxiv.org/pdf/2608.13505
• Abstract Scientific discovery increasingly requires AI systems that can reason over scientific evidence of heterogeneous modalities, interact with scientific tools and environments, and sustain progress across long task horizons. We present Intern-S2-Preview, a series of scientific agentic foundation models designed to support multimodal scientific understanding, reasoning, generation, and long-horizon tasks. The training pipeline begins with scientific multimodal pre-training over rendered scientific documents, interleaved image-text data, and diverse scientific corpora. Starting from the pretrained checkpoint, we apply a unified post-training pipeline consisting of supervised fine-tuning, scalable multi-task reinforcement learning (RL), black- and white-box agentic RL, and on-policy distillation. This pipeline is supported by practical techniques that improve rollout and training stability and efficiency, including partial rollout with off-policy correction, adaptive length regularization, online speculative decoding, robust multi-task optimization, and trace-aware experience assembly for agentic tasks. At the architecture level, Intern-S2-Preview-397B extends time series modelling from efficient long-sequence understanding to numerical forecasting, while Memory Decoder is studied as a separate memory-augmented path for rapid scientific specialization without modifying the frozen 397B backbone. Evaluations across scientific, multimodal, agentic, and general-purpose benchmarks show that Intern-S2-Preview-397B achieves competitive or leading results in multiple settings. The time series modules improve scientific signal understanding and forecasting on SciTS, while the separate Intern-MemDec-4B extension improves the Biology-Instructions average score from 56.92 to 60.32 without modifying the frozen 397B backbone.
• 中文摘要 科学发现日益需要能够推理异质模态科学证据、与科学工具和环境互动，并持续长任务视野进展的人工智能系统。我们呈现Intern-S2-Preview，一系列科学代理基础模型，旨在支持多模态科学理解、推理、生成和长视野任务。训练流程始于对渲染科学文档、交错图像文本数据和多样化科学语料库的科学多模态预训练。从预训练检查点出发，我们应用统一的后期训练流程，包括监督微调、可扩展多任务强化学习（RL）、黑白盒智能强化学习（Agentic RL）以及策略中提炼。该流程支持实用技术，提升部署和训练的稳定性与效率，包括部分部署含非策略修正、自适应长度正则化、在线推测解码、稳健多任务优化以及针对代理任务的痕迹感知体验组装。在架构层面，Intern-S2-Preview-397B 将时间序列建模从高效的长序列理解扩展到数值预测，而 Memory Decoder 则作为独立的内存增强路径，用于快速科学专业化，且不修改冻结的 397B 骨干。跨科学、多模态、代理和通用基准的评估显示，Intern-S2-Preview-397B 在多个环境中均具竞争力或领先效果。时间序列模块提升了 SciTS 上的科学信号理解和预测，而独立的 Intern-MemDec-4B 扩展则将生物学指令的平均得分从 56.92 提升至 60.32，且未修改冻结的 397B 骨干。

Keyword: diffusion policy

There is no result