生成时间: 2026-07-28 18:39:20 (UTC+8); Arxiv 发布时间: 2026-07-28 20:00 EDT (2026-07-29 08:00 UTC+8)

今天共有 58 篇相关文章

Keyword: reinforcement learning

CRAFT: Learn the Schema, Execute the Plan

技巧：学习模式，执行计划

• Authors: Aakash Kolekar, Sahika Genc, Shahriar Shariat, Bunyamin Sisman, Tibor Mezi, Barbara Poblete, Shree Vandana Kachroo, Calvin Chi, Parth Parmar, Ari Singer, Prayaas Jain, Cindy Barker, Benoit Dumoulin
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Multiagent Systems (cs.MA); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.22642
• Pdf link: https://arxiv.org/pdf/2607.22642
• Abstract Enterprise coding agents translate natural-language analytical requests into executable code over proprietary APIs, schemas, and metric definitions. Yet the prevailing deployment pattern injecting exhaustive schema and tool documentation into each prompt increases inference overhead, complicates schema evolution, and undermines reliability in multi-turn analysis. We investigate whether stable schema knowledge and tool-use behavior can instead be acquired through post-training while preserving the consistency required for production-facing analytics. We present CRAFT, a two-stage post-training recipe for schema-grounded coding agents. First, schema-stripped PLAN supervised fine-tuning learns domain-structured plans and executable behaviors from validated trajectories without exhaustive prompt-time schema injection. Second, execution-shaped reinforcement learning aligns the policy for tool selection, code quality, plan-code consistency, and recovery from failed executions. Training trajectories are curated through a Tri-Gate filter combining execution validation, data-integrity checks, and LLM-judge reasoning audit. We evaluate CRAFT for planned rollout in advertising analytics, covering campaign performance analysis, metric drill-downs, entity-level performance analysis, and multi-turn analytical refinement. The enterprise evaluation environment incorporates beta APIs as the agent-facing tool surface and spans 25 schema-linked core entities and 30 agentic workflows. Relative to a schema-stuffed baseline, CRAFT improves composite Agent Score by +9.6 pp, consistency by +4.1 pp, and multi-turn coherence by +4.2 pp, while reducing input-token burden by approximately 9x and schema-discovery loops by up to 5x. We further report deployment tradeoffs, reward-shaping limitations, and training-infrastructure extensions required for multi-turn tool-use reinforcement learning in enterprise settings.
• 中文摘要 企业编码代理通过专有API、模式和度量定义将自然语言分析请求转换为可执行代码。然而，当前的部署模式将详尽的模式和工具文档注入每个提示，增加了推理开销，复杂化了模式演化，并削弱了多回合分析的可靠性。我们研究了稳定模式知识和工具使用行为是否可以通过后期训练获得，同时保持面向生产环境分析所需的一致性。我们介绍CRAFT，一种基于模式的编程代理的两阶段训练后方案。首先，模式剥离的PLAN监督微调通过验证轨迹学习领域结构化计划和可执行行为，无需穷尽提示时模式注入。其次，执行形强化学习使工具选择、代码质量、计划-代码一致性及执行失败后恢复策略保持一致。训练轨迹通过三门过滤器策划，结合执行验证、数据完整性检查和LLM裁判推理审计。我们评估CRAFT在广告分析中的计划推广，涵盖活动绩效分析、指标深入分析、实体级性能分析及多回合分析细化。企业评估环境采用测试版API作为面向代理的工具表盘，涵盖25个模式关联核心实体和30个代理工作流。相较于模式填充的基线，CRAFT使综合代理评分提升+9.6 pp，一致性提升+4.1 pp，多回合一致性提升+4.2 pp，同时将输入令牌负担降低约9倍，模式-发现循环最多减少5倍。我们还报告了部署权衡、奖励塑造限制以及企业环境中多回合工具使用强化学习所需的培训基础设施扩展。

DocHRL: A Hierarchical Reinforcement Learning Framework for Cost-Optimised Document Classification

DocHRL：一种用于成本优化文档分类的分层强化学习框架

• Authors: Mohammed Yousif, Prabhjot Singh, Arjun Pankajakshan, Madhu Reddiboina
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.22644
• Pdf link: https://arxiv.org/pdf/2607.22644
• Abstract Real-world document classification pipelines typically apply the same sequence of models to every incoming document, regardless of its complexity or type. This leads to inefficient use of compute and human resources: simple documents are over-processed while difficult ones may not receive enough scrutiny. We introduce DocHRL, a hierarchical reinforcement learning framework that learns to adaptively and dynamically select the most cost-effective classification policy on a per-document basis. DocHRL formulates document classification as a sequential decision problem with a two-level policy hierarchy: a top-level policy selects among broad options (vision classifiers, LLMs, OCR, and human-in-the-loop review), while option-specific sub-policies choose the concrete model or tool to invoke. The reward signal is the negative total expected cost, which captures inference cost, cost of misclassification, and cost of human labelling. Trained with Proximal Policy Optimisation on the RVL-CDIP benchmark, DocHRL achieves a macro F1 of 0.973 across 16 document classes while reducing average per-document cost to 2.74 normalised units compared to substantially higher costs incurred by fixed standalone classifiers. Our results demonstrate that cost-aware reinforcement learning can simultaneously improve classification performance and operational efficiency in document understanding systems.
• 中文摘要 现实世界的文档分类流程通常对每个入来文档应用相同的模型序列，无论其复杂度或类型如何。这导致计算和人力资源的低效利用：简单文档被过度处理，而复杂文档可能未得到足够审查。我们介绍了DocHRL，一种层级强化学习框架，能够自适应且动态地针对每文档选择最具成本效益的分类策略。DocHRL将文档分类表述为一个具有两级策略层级的顺序决策问题：顶层策略从广泛选项（视觉分类器、大型语言模型、OCR和人工审核）中选择，而选项专属子政策则选择具体模型或工具调用。奖励信号是负的总期望成本，涵盖推理成本、误分类成本和人工标记成本。通过基于RVL-CDIP基准的近端策略优化训练，DocHRL在16个文档类别中实现了0.973的宏F1，同时将每文档的平均归一化成本降低至2.74个归一化单位，相比固定独立分类器的成本大幅提升。我们的结果表明，成本感知型强化学习能够同时提升文档理解系统的分类性能和运营效率。

STAIF: A Stage-wise Optimization for Complex Instruction Following

STAIF：复杂指令跟随的分阶段优化

• Authors: Jian Hong, Chen Cheng, Quan Liu, Yuhao Chen, Enhong Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.22649
• Pdf link: https://arxiv.org/pdf/2607.22649
• Abstract Following complex instructions with multiple explicit constraints remains a fundamental challenge for large language models (LLMs). Existing alignment methods, such as DPO, optimize holistic reward signals that often underemphasize strict satisfaction of individual constraints, particularly under out-of-distribution or multi-constraint settings. In this paper, we propose STAIF, a stage-wise optimization framework that decouples the alignment of subjective (soft) constraints from the optimization of objectively verifiable (hard) constraints. Stage 1 applies preference optimization with multiple negative samples to sharpen sensitivity to soft constraints, while Stage 2 applies Reinforcement Learning with Verifiable Rewards (RLVR) to enforce strict compliance with hard constraints. To support this method, we construct STAINSTRUCT, a high-quality bilingual (English, Chinese) dataset of approximately 31,000 complex multi-constraint instructions. Extensive analyses validate the design of STAIF and show state-of-the-art performance on representative benchmarks against strong baselines, as well as genuine generalization.
• 中文摘要 遵循具有多个显式约束的复杂指令仍是大型语言模型（LLM）面临的根本挑战。现有的对齐方法，如DPO，优化了整体奖励信号，这些信号常常低估对单个约束的严格满足，尤其是在分布外或多约束条件下。本文提出了STAIF，一种分阶段优化框架，将主观（软）约束的对齐与客观可验证（困难）约束的优化解耦。第一阶段通过多负样本进行偏好优化以提升对软约束的敏感性，第二阶段则采用可验证奖励强化学习（RLVR）强制严格遵守硬约束。为支持该方法，我们构建了STAINSTRUCT，这是一个高质量的双语（英、中）数据集，包含约31,000条复杂的多约束指令。大量分析验证了STAIF的设计，并在强基线和真实的泛化条件下展示了代表性基准的先进性能。

How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Drift

LLM任务适应如何重塑对齐：行为与表征漂移的多维研究

• Authors: James Elcock, William F. Shen, Xinchi Qiu, Nicholas D. Lane
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.22676
• Pdf link: https://arxiv.org/pdf/2607.22676
• Abstract Post-training is a key mechanism for adapting large language models to downstream tasks. While prior work suggests that task adaptation can alter a model's pre-existing alignment, especially its safety behavior, its broader effects across alignment domains remain poorly understood. We address this gap through a systematic evaluation of representative task-adaptation methods, including supervised fine-tuning (SFT), KL-regularized SFT, and reinforcement learning with verifiable rewards (RLVR) across 15 alignment aspects spanning six key domains: safety, factuality, stance stability, social harm, controllability, and instructability. Our results reveal that post-training does not reshape alignment uniformly. RLVR improves task performance while inducing comparatively small, but non-zero, metric-specific shifts, while SFT leads to substantially larger alignment drift across domains. KL regularization mitigates this effect: stronger reference-model anchoring reduces alignment drift from the baseline, although KL-SFT still falls short of RLVR in preserving alignment. Representation-level analysis further supports this pattern, with shifts in alignment-relevant representations tracking behavioral drift. Together, these results show that task adaptation is not merely a capability-improving step, but an alignment intervention in its own right, motivating multi-dimensional alignment evaluation as a standard component of post-training pipelines.
• 中文摘要 后训练是将大型语言模型适应到下游任务的关键机制。尽管先前研究表明任务适应可以改变模型的既有对齐，尤其是其安全行为，但其在对齐领域的更广泛影响仍不充分。我们通过系统评估代表性任务适应方法，包括监督微调（SFT）、KL正则化SFT和可验证奖励强化学习（RLVR），涵盖安全、事实性、姿态稳定性、社会伤害、可控性和可指导性六大对齐方面，从而解决了这一空白。我们的结果显示，后训练并未均匀重塑对齐。RLVR在诱导相对较小但非零的度量特异性转变的同时提升任务表现，而SFT则导致跨领域显著更大的对齐漂移。KL正则化缓解了这一效应：更强的参考模型锚定减少了基线的比对漂移，尽管KL-SFT在保持比对方面仍不及RLVR。表征级分析进一步支持这一模式，比对相关表征的转变追踪行为漂移。综合来看，这些结果表明任务适应不仅是提升能力的步骤，更是一种对齐干预，促使多维比对评估作为训练后流程的标准组成部分。

LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory

LazyMem：广泛检索，选择性构造，以实现高效的长期代理记忆

• Authors: Jing Yu, Yibo Zhao, Jiaming Zhang, Xiang Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.22690
• Pdf link: https://arxiv.org/pdf/2607.22690
• Abstract Long-term memory lets LLM agents reuse past interactions, but raw dialogue histories are verbose and information-sparse. Retrieving broadly improves evidence coverage yet overwhelms downstream reasoning with noise; compressing at write time reduces noise but irreversibly discards details the future query may need. We introduce LazyMem, which sidesteps this dilemma by deferring all memory construction to query time. A lightweight 4B model processes the retrieved candidate pool in overlapping parallel windows, selectively retaining and compressing only query-relevant content. The model is trained through supervised fine-tuning followed by group-based reinforcement learning with a format-gated composite reward that combines a rule-based action signal measuring selection accuracy with an LLM-judged quality signal measuring source faithfulness and query utility. On the LongMemEval benchmark, LazyMem-4B achieves an LLM-judge accuracy of 0.85 with only 213 memory tokens, 68.7$\times$ fewer than retrieval-only, and generalizes to LoCoMo (0.68) without target-domain training, while reducing mean latency over the prior query-time baseline. The 32B variant reaches 0.93, surpassing oracle-context references on aggregation-heavy question types. The code associated with this work is publicly available at this https URL.
• 中文摘要 长期记忆允许LLM代理重用过去的交互，但原始对话历史冗长且信息稀疏。广泛检索提升证据覆盖率，但会用噪声淹没下游推理;写入时压缩减少噪声，但不可逆地丢弃未来查询可能需要的细节。我们引入了LazyMem，通过将所有内存构建推迟到查询时间，绕过了这一困境。一个轻量级的4B模型在重叠的并行窗口中处理检索的候选池，选择性地保留和压缩与查询相关的内容。该模型通过监督微调训练，随后采用基于组的强化学习，采用格式门槛的复合奖励，结合基于规则的动作信号测量选择准确性，以及LLM判断的质量信号，测量源的忠实度和查询效用。在LongMemEval基准测试中，LazyMem-4B仅用213个内存令牌实现了0.85的LLM判定准确率，比仅检索少68.7美元\时间美元，并且可推广到无目标域训练的LoCoMo（0.68），同时降低了前一查询时间基线的平均延迟。32B版本达到0.93，超过了聚合题型的oracle上下文引用。与该工作的相关代码在此HTTPS网址公开。

HiLLTS: Zero-Shot Hierarchical LLM-Guided Traffic Signal Control for Sustainable Transportation

HiLLTS：零射分层LLM引导交通信号控制，用于可持续交通

• Authors: Yue Ding, Tendai Mukande, Mingming Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.22691
• Pdf link: https://arxiv.org/pdf/2607.22691
• Abstract Urban traffic congestion significantly increases fuel consumption, greenhouse gas emissions, and commuter delays, resulting in substantial economic losses and environmental harm in modern cities. Traditional traffic signal control strategies such as fixed-time scheduling, actuated control, and reinforcement learning (RL)-based methods, offer different degrees of adaptability; however, RL-based methods can require extensive retraining, careful reward design, and substantial simulation data when transferred across networks or demand regimes. To address these challenges, we propose HiLLTS, an LLM-guided traffic signal control framework that employs a hierarchical three-layer architecture consisting of a central coordination agent, a district layer and multiple cluster-level intersection agents. Experimental results demonstrate consistent improvements in both congestion and environmental performance. Compared with the strongest non-LLM baseline in each scenario, HiLLTS reduces average waiting time by 36.73% under the low-congestion scenario and 14.71% under the high-congestion scenario, while reducing average CO2 emissions by 7.87% and 8.57%, respectively. Larger gains are observed against weaker baselines: under low congestion, HiLLTS achieves reductions of up to 18.00% in emissions and 62.07% in waiting time relative to Fixed-Time control; under high congestion, reductions of up to 28.89% in emissions and 40.36% in waiting time are observed relative to Max Pressure. The ablation study further validates the contribution of LLM-guided coordination over rule-based control
• 中文摘要 城市交通拥堵显著增加燃料消耗、温室气体排放和通勤延误，导致现代城市的重大经济损失和环境破坏。传统的交通信号控制策略，如固定时间调度、感应控制和强化学习（RL）方法，提供了不同程度的适应性;然而，基于RL的方法在跨网络或需求体系传输时，可能需要大量再训练、精心设计奖励和大量模拟数据。为应对这些挑战，我们提出了HiLLTS，这是一个基于LLM引导的交通信号控制框架，采用由中央协调代理、区域层和多个集群级交叉代理组成的分层三层架构。实验结果显示，拥堵和环境性能均有持续提升。与每个场景中最强的非大型语言模型基线相比，HiLLTS在低拥堵情景下平均等待时间减少了36.73%，在高拥堵情景下减少了14.71%，平均CO2排放分别减少了7.87%和8.57%。相比较弱基线，HiLLTS的增长更大：在低拥堵情况下，HiLLTS相较固定时间控制实现了最多18.00%的排放减少和62.07%的等待时间;在高拥堵情况下，相较最大压力可减少最多28.89%的排放和40.36%的等待时间。消融研究进一步验证了LLM引导协调相较于基于规则控制的贡献

Cortex: Compact Behavior Cloning for Quake with Frozen Visual Features

Cortex：带冻结视觉特征的Quake紧凑行为克隆

• Authors: Dzmitry Malyshau
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.22739
• Pdf link: https://arxiv.org/pdf/2607.22739
• Abstract We study how far a deliberately simple behavioral-cloning policy can progress in a visually rich first-person game before adding reinforcement learning or explicit memory. Cortex is a compact Quake policy with 10.98 million trainable parameters in a six-layer transformer over a frozen DINOv3 encoder. It is trained on the Quake subset of the public Pixels2Play corpus: 6,849 recordings (about 474.7 hours), represented as 17.09 million cached decision frames with keyboard and mouse actions. One sampled training epoch uses 517,048 four-frame windows and takes 3.3 minutes of policy-head optimization on one RTX 5080, excluding one-time feature extraction. We evaluate two independent batches of 20 stochastic, 120-second episodes on Quake E1M1. Cortex does not complete the level, but every episode reaches the opening door, button room, and gate descent; 19 of 20 episodes in each batch record at least one kill. Under the same time-controlled harness, released P2P-150M and NitroGen checkpoints remain shallower in five matched-duration episodes each. These comparisons are limited by small reference samples and different native interfaces. Ablations show that denser visual tokens improve combat and survival, while longer optimization and naive action history improve offline metrics without consistently improving play. The remaining failures are consistent with covariate shift and motivate targeted corrective data. We release the policy implementation, checkpoint, and a representative rollout.
• 中文摘要 我们研究了一个刻意简单的行为克隆策略在视觉丰富的第一人称游戏中能达到多远，然后才添加强化学习或显式记忆。Cortex 是一个紧凑的 Quake 策略，拥有 1098 万可训练参数，采用六层变换器，基于冻结的 DINOv3 编码器。它在公开 Pixels2Play 语料库中的 Quake 子集上训练：6,849 个录音（约 474.7 小时），以 1709 万个带有键盘和鼠标操作的缓存决策帧表示。一个采样训练纪元使用了 517,048 个四帧窗口，在一台 RTX 5080 上进行策略头优化需 3.3 分钟，不包括一次性特征提取。我们评估了 Quake E1M1 上两批独立的 20 集随机集，每集 120 秒。Cortex 未完成关卡，但每集都达到开门、按钮室和门下行;每批 20 集中有 19 集至少记录一次击杀。在同一时间控制束下，发布的 P2P-150M 和 NitroGen 检查点在五集匹配时长的集数中保持较浅。这些比较受限于少量参考样本和不同的原生界面。消融显示，更密集的视觉标记能提升战斗和生存，而更长的优化和朴素的行动历史则提升离线指标，但并未持续改善游戏体验。其余失败与协变量偏移一致，并激励有针对性的纠正数据。我们发布了策略实施、检查点和代表性发布。

Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias

令人沮丧的简单语言模型黑箱式通过Logit偏置

• Authors: Ofek I. Cohen, Lior Shani, Aviv Rosenberg, Ankur Samanta, Tal Wagner, Yonathan Efroni
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.22837
• Pdf link: https://arxiv.org/pdf/2607.22837
• Abstract Many organizations aim to adapt language models for internal use, both to improve performance on domain-specific tasks and to address privacy concerns around sensitive data. However, such adaptation remains non-trivial: it often requires operationally challenging fine-tuning of open-source models or ad hoc prompt optimization. We study a minimal alternative based on a simple API-level control: allowing users to bias the model's logits with a user-defined vector. We develop a black-box method for learning a single context-independent logit-bias vector, added at every decoding step, without modifying model weights or requiring gradients. Starting from a KL-regularized reinforcement learning (RL) objective, we characterize when such a fixed logit-bias vector can approximate the optimal prefix-dependent correction and derive a closed-form inverse-propensity estimator from rollouts, rewards, and token probabilities. Empirically, this simple decoding-time intervention improves over base models on mathematical and reasoning benchmarks while using far fewer trainable parameters than conventional fine-tuning. Our results suggest that learned logit bias is a lightweight mechanism for adapting language models under minimal access requirements.
• 中文摘要 许多组织致力于将语言模型调整为内部用途，既提升领域特定任务的性能，也解决敏感数据相关的隐私问题。然而，这种适应仍然不简单：通常需要对开源模型进行操作上的微调或临时提示优化。我们研究了一个基于简单API级控制的最小替代方案：允许用户用用户自定义的向量对模型的logit进行偏置。我们开发了一种黑箱方法，用于学习单个上下文无关的logit偏向量，该向量在每个译码步骤中添加，无需修改模型权重或要求梯度。从KL正则化强化学习（RL）目标出发，我们刻画了固定logit偏向向量何时能近似最优前缀相关修正，并从rollout、奖励和token概率中推导出闭式反倾向估计器。从经验角度看，这种简单的解码时间干预在数学和推理基准测试上优于基础模型，同时使用远少于传统微调的可训练参数。我们的结果表明，习得性对数偏倚是一种轻量级机制，可以在最低访问要求下调整语言模型。

A Replay-Constrained Simulation Framework for Personalization of Powered Knee--Ankle Prosthesis Controllers

一个用于个性化动力膝踝假肢控制器的回放约束模拟框架

• Authors: Duong Le, Ryan Posh, Shihao Cheng, Maani Ghaffari, Robert D. Gregg
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.22858
• Pdf link: https://arxiv.org/pdf/2607.22858
• Abstract Personalization of impedance controllers for powered prosthetic legs is critical to accommodating individual gait biomechanics but remains challenging. Existing methods rely on time-intensive human-in-the-loop exploration and/or constrain optimization to low-dimensional, single-joint parameter subspaces. Sim-to-real transfer has enabled high-dimensional locomotion control for legged robots, but in assistive device control the human partner remains un-modelable. We present a replay-constrained simulation framework: a MuJoCo-based simulator reproduces prosthetic knee-ankle dynamics while replaying recorded hip kinematics and feedback-based ground reaction forces from individual walking data, bypassing the need to model complex human neuromuscular control mechanisms. We demonstrate the framework with a deep reinforcement learning policy that personalizes phase-dependent stiffness, damping, and equilibrium angle at both joints simultaneously, maximizing a biomimicry-based reward computed solely from onboard prosthesis measurements. Experiments with three participants with transfemoral amputation during level-ground walking at 0.8~m/s demonstrate strong simulation-to-hardware predictive validity (Pearson $r=0.96$--$0.997$). The best-performing policy on hardware was consistently predicted within the top five simulation policies for all participants. The learned controllers improved overall biomimicry rewards by 42--59\% relative to the unpersonalized baseline. The framework supports scalable high-dimensional personalization of powered prosthetic legs and is amenable to extension to higher-dimensional controller parameterizations such as neural-network controllers.
• 中文摘要 为动力义肢量身定制阻抗控制器对于适应个体步态生物力学至关重要，但仍具挑战性。现有方法依赖于耗时的人机循环探索和/或限制优化至低维单关节参数子空间。模拟到现实的传输使腿部机器人实现了高维运动控制，但在辅助设备控制中，人类伙伴仍无法建模。我们提出了一个回放约束模拟框架：基于MuJoCo的模拟器在回放个体行走数据中录制的髋部运动学和基于反馈的地面反作用力的同时，重现假肢膝-踝动态，绕过了复杂人类神经肌肉控制机制的建模需求。我们演示了该框架，采用深度强化学习策略，同时个性化两个关节的相位依赖刚度、阻尼和平衡角，最大化基于仿生的奖励，仅凭机载假肢测量计算。三名参与者在0.8~m/s平地行走时进行经股骨截肢的实验显示了强烈的模拟到硬件预测效度（Pearson $r=0.96$-$0.997$）。硬件表现最佳的策略在所有参与者的前五模拟策略中均被一致预测。学习到的控制器相比非个性化基线提升了42-59%的整体仿生奖励。该框架支持可扩展的高维动力假肢个性化，并可扩展至神经网络控制器等高维控制器参数化。

Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests

空间智商：通过层级能力测试解构空间智能

• Authors: Patrick Rim, Tom Long, Ekta Prashnani, Ruth Rosenholtz, Ben Boudaoud, Peter Xenopoulos, Alex Wong, Joohwan Kim, Jae-Hyun Jung
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.22864
• Pdf link: https://arxiv.org/pdf/2607.22864
• Abstract Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably. Existing benchmarks evaluate these models as black boxes, limiting their ability to identify the underlying causes of lower performance: when a model fails a spatial reasoning task, it remains difficult to ascertain whether the hurdle is perceptual, such as recognizing object boundaries, or cognitive, such as reasoning about occlusion to infer hidden geometry. We introduce Spatial-IQ, a hierarchical diagnostic framework that decomposes object counting in stacked 3D structures into 9 perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition, with mental rotation as an additional target probe. Using NVIDIA Isaac Sim, we procedurally generated a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth. We evaluate models across three output formats (free-response text, multiple-choice images, and image editing) alongside a human baseline. The Spatial-IQ framework shows that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure. Finally, we demonstrate that training models with chain-of-thought (CoT) supervision over our hierarchical sub-tasks, combined with reinforcement learning with verifiable rewards, significantly improves both spatial consistency across sub-tasks and target-task accuracy, supporting the value of the proposed decomposition as both a diagnostic tool and a training signal.
• 中文摘要 多模态大型语言模型（MLLMs）在视觉解释方面表现出色，但在人类可靠解决的空间推理任务上表现不佳。现有基准将这些模型评为黑箱，限制了它们识别性能下降的根本原因：当模型未能完成空间推理任务时，很难判断障碍是感知层面，如识别物体边界，还是认知层面，如推理遮挡推断隐藏几何。我们引入了Spatial-IQ，一种层级诊断框架，将堆叠三维结构中的物体计数分解为9个感知和认知子任务，按人类空间认知的发展阶段组织，心理旋转作为额外目标探针。利用NVIDIA Isaac Sim，我们程序生成了一个包含约8万个叠加三维结构的多样化数据集，并基于每个任务进行地面真实数据。我们评估了三种输出格式（自由回答文本、多项选择图和图像编辑）的模型，同时也采用了人类基线。Spatial-IQ框架显示，表现最佳的模型往往在目标任务（对象计数）上取得成功，而在支持该任务的低层子任务上却未能成功，且模型在保留这些层级链条的程度上存在差异，常常揭示了单靠原始目标任务准确性无法掩盖的捷径行为。最后，我们证明，在层级子任务的思维链（CoT）监督下训练模型，结合可验证奖励的强化学习，显著提升了子任务间的空间一致性和目标任务的准确性，支持了所提分解作为诊断工具和训练信号的价值。

Not All LLM Reasoning is Visible in the Chain-of-Thought

并非所有LLM推理都能在思维链中显现

• Authors: Vatsal Baherwani, Tom Goldstein, Ashwinee Panda
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.22925
• Pdf link: https://arxiv.org/pdf/2607.22925
• Abstract A key question for AI safety is whether a language model expresses all of its reasoning in its output tokens. We demonstrate a concrete failure mode where frontier models exhibit invisible reasoning by leveraging semantically irrelevant filler tokens to improve performance on synthetic reasoning tasks. We evaluate 13 frontier language models across three tasks and find that many models benefit significantly from filler tokens, with accuracy improvements of up to 13 percentage points. The benefit depends on which tokens are used and differs across models. We further show that filler tokens enable Claude Opus 4.5 to satisfy a hidden modular arithmetic constraint without sacrificing accuracy on its primary task, demonstrating that invisible reasoning can serve objectives entirely invisible to CoT monitoring. Reinforcement learning gives Qwen3-235B strong preferences over filler token content, but neither RL nor supervised fine-tuning produces a filler token benefit that persists at test time. Our results indicate that frontier models already perform consequential computation with no interpretable trace in their output tokens.
• 中文摘要 人工智能安全的关键问题是，一个语言模型是否通过输出词表达了所有推理。我们展示了一种具体的失败模式，即前沿模型通过利用语义无关的填充词来提升合成推理任务的性能，从而表现出隐形推理。我们评估了13个前沿语言模型，涵盖三个任务，发现许多模型从填充词中显著受益，准确率提升高达13个百分点。这种优势取决于所使用的词元，且各模型间存在差异。我们还进一步证明，填充词使Claude Opus 4.5能够满足隐藏的模块算术约束而不牺牲其主要任务的准确性，证明隐形推理可以实现CoT监控完全看不见的目标。强化学习赋予Qwen3-235B对填充代币内容的强烈偏好，但无论是强化学习还是监督式微调，都不会产生在测试时持续存在的填充代币优势。我们的结果表明，前沿模型已经在输出代币中执行无可解释痕迹的后果计算。

Finite-Time Analysis of the Natural Policy Gradient in Finite-Horizon Markov Decision Processes

有限视界马尔可夫决策过程中自然政策梯度的有限时间分析

• Authors: Asha Barua, Sajad Khodadadian
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2607.22982
• Pdf link: https://arxiv.org/pdf/2607.22982
• Abstract Natural Policy Gradient (NPG) is a well-established Reinforcement Learning algorithm that underlies widely used methods such as Trust Region Policy Optimization and Proximal Policy Optimization, both of which have demonstrated strong empirical success. In this paper, we study exact NPG in finite-horizon Markov Decision Processes with known dynamics and horizon-dependent transition kernels. We provide the first finite-time convergence guarantees for this algorithm in this setting, for which we consider both constant and increasing step size regimes. With a constant step size $\eta_t=\eta$, we prove that NPG converges sublinearly with a rate of $\mathcal{O}(H^{2}/t)$ after $t$ iterations, where $H$ is the horizon length. We also extend this constant step size analysis to linear MDPs in an exact population-projection oracle under a full support projection distribution, recovering the same sublinear rate as in the tabular setting. Furthermore, with increasing step sizes, we prove that this algorithm achieves a linear convergence rate of $\mathcal{O}\left(\left(1-\frac{1}{\vartheta_\rho}\right)^t\right)$ for a problem-dependent constant $\vartheta_\rho > 1$, and the horizon-only robust schedule of the form $\eta_t=\eta_0(H/(H-1))^t$ where $\eta_0>0$ and $H \geq 2$, attains this same geometric rate.
• 中文摘要 自然策略梯度（NPG）是一种成熟的强化学习算法，是信任区域策略优化（Trust Region Policy Optimization）和近端策略优化（Proximal Policy Optimization）等广泛使用方法的基础，这两者都已证明了强有力的实证成功。本文研究了有限视野马尔可夫决策过程中已知动力学和视界依赖转移核的精确NPG。我们首次为该算法提供了有限时间收敛保证，考虑了常数和递增的步长范围。在步长$\eta_t=\eta$恒定的情况下，我们证明NPG在$t$迭代后以$\mathcal{O}（H^{2}/t）$的速率以亚线性收敛，其中$H$为视界长度。我们还将该常步长分析扩展到在完全支持投影分布下精确的种群投影预言机中的线性MDP，恢复与表格设置相同的亚线性速率。此外，随着步长增加，我们证明该算法在问题依赖常数$\>vartheta_vartheta_\rho}\right时实现线性收敛率$\mathcal{O}\left（\left（1-\frac{1}{\rho}\right）^t\right）$，且仅视界的稳健时程形如$\eta_t=\eta_0（H/（H-1））^t$（其中$\eta_0>0$和$H\geq 2$）也达到相同的几何收敛率。

Label-free Industrial Fault Detection via Adversarial Inverse Reinforcement Learning: A System for Run-to-Failure Prognostics

通过对抗性逆强化学习实现无标签工业故障检测：一个运行到故障预报系统

• Authors: Dhiraj Neupane, Mohamed Reda Bouadjenek, Richard Dazeley, Sunil Aryal
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.22987
• Pdf link: https://arxiv.org/pdf/2607.22987
• Abstract Machinery fault detection (MFD) remains heavily reliant on supervised learning, which struggles with the scarcity of fault labels in real-world settings. While reinforcement learning (RL) offers a framework to model the sequential nature of degradation, current ``RL-based'' MFD methods reduce the problem to a static contextual bandit (CB) formulation: by ignoring state transitions and discarding the temporal discount factor, they collapse to standard supervised classification. We propose an adversarial inverse reinforcement learning (AIRL) framework that treats MFD as an offline IRL problem. Unlike reconstruction-based approaches that rely on static error margins, or CBs that ignore dynamics, our method recovers an intrinsic "health" reward directly from observational state transitions, requiring neither manual reward engineering nor fault labels. On three run-to-failure benchmarks (HUMS2023, IMS, XJTU-SY), AIRL is the only method achieving non-saturated post-detection consistency across all datasets, while CB baselines fail to detect gradual degradation and reconstruction models collapse into always-anomalous states. Code and data: this https URL.
• 中文摘要 机械故障检测（MFD）仍然高度依赖监督式学习，而在现实环境中故障标签的稀缺性问题上存在困难。虽然强化学习（RL）提供了建模退化顺序性质的框架，但当前基于RL的MFD方法将问题简化为静态上下文盗垒（CB）形式：通过忽略状态转换和时间折扣因子，它们归结为标准的监督分类。我们提出了一种对抗性逆强化学习（AIRL）框架，将MFD视为离线现实问题。与依赖静态误差范围或忽略动态的重建方法不同，我们的方法直接从观察状态转换中恢复内在的“健康”奖励，无需人工奖励工程或故障标签。在三个运行失败基准测试（HUMS2023、IMS、XJTU-SY）中，AIRL是唯一实现所有数据集检测后非饱和一致性的方法，而CB基线无法检测到渐进性退化，重建模型崩溃为始终异常状态。代码和数据：此https URL。

Real2Sim2Real for Vision-Language-Action Manipulation: An AMD ROCm-Based Pipeline

Real2Sim2Real 用于视觉-语言-行动操作操作：基于 AMD ROCm 的流水线

• Authors: Qing Yang, Xun Wang, Ziguan Wang, Zhenjiang Li, Hongqiang Wang, Dongdong Weng
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Graphics (cs.GR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.22997
• Pdf link: https://arxiv.org/pdf/2607.22997
• Abstract Physical AI -- the integration of large vision-language-action (VLA) models with embodied agents that act in the real world -- has emerged as the next major frontier for AI, echoed by industry leaders such as Jensen Huang (`the next big thing is Physical AI, AI with a body,'' GTC Paris, June 2025) and Dr. Lisa Su (we're entering the world of Physical AI ... this is where AI enters the real world,' CES 2026). This paper presents an end-to-end, fully AMD-accelerated technology stack for embodied manipulation, spanning data-center training silicon, Radeon PRO simulation/rendering GPUs, and Ryzen AI edge compute, unified by the open ROCm software stack. We demonstrate that training and deploying VLA-based manipulation policies does not require a CUDA-locked ecosystem. Four progressive demonstrations are presented: (1) a Sim-to-Real manipulation pipeline trained with SmolVLA and deployed on a physical Franka arm; (2) a semantic, language-grounded object-selection task (`one-of-three'); (3) a Real2Sim synthetic-data generation pipeline that fuses 3D Gaussian Splatting (3DGS) reconstructions of real scenes with the Genesis physics engine; and (4) large-scale reinforcement learning for quadruped and humanoid locomotion benchmarked across multiple hardware platforms. All pipelines run natively on ROCm + PyTorch on RDNA4 (Radeon AI PRO R9700) and RDNA3.5 (Radeon PRO W7900) hardware and are reproducible on the free Radeon Cloud Platform.
• 中文摘要 物理人工智能——将大型视觉-语言-动作（VLA）模型与具身代理在现实世界中行动的集成——已成为人工智能的下一个重要前沿，行业领袖如黄仁森（《下一个大趋势是拥有身体的人工智能》，GTC 巴黎，2025年6月）和苏丽莎博士（“我们正进入物理人工智能的世界......这就是人工智能进入现实世界的地方，2026年CES》）。本文提出了一个端到端、完全由AMD加速的具身操作技术栈，涵盖数据中心训练硅、Radeon PRO模拟/渲染GPU和Ryzen AI边缘计算，并由开放的ROCm软件栈统一。我们证明了基于VLA的训练和部署操作策略并不需要CUDA锁定的生态系统。呈现四个渐进演示：（1）一个用SmolVLA训练并部署在Franka机械臂上的模拟到现实操作流水线;（2）一个语义、语言基础的对象选择任务（“三项之一”）;（3）一个Real2Sim合成数据生成流水线，将真实场景的3D高斯喷射（3DGS）重建与Genesis物理引擎融合;以及（4）跨多个硬件平台进行基准测试的四足和类人移动大规模强化学习。所有流水线均原生运行在RDNA4（Radeon AI PRO R9700）和RDNA3.5（Radeon PRO W7900）硬件上的ROCm + PyTorch上，并且可在免费的Radeon云平台上复现。

Online Policy Evaluation for MDPs with Dynamic UBSR Measures

针对具有动态UBSR措施的MDP在线政策评估

• Authors: Weikai Wang, Erick Delage
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.23030
• Pdf link: https://arxiv.org/pdf/2607.23030
• Abstract Developing efficient function-approximation methods for policy evaluation is a fundamental challenge in risk-aware reinforcement learning. Existing approaches either focus on restrictive classes of risk measures or rely on access to a simulator, limiting their applicability in fully online settings. In this work, we propose computationally efficient online learning algorithms for policy evaluation in Markov decision processes (MDPs) with dynamic utility-based shortfall risk (UBSR) measures under linear function approximation. Specifically, we introduce the UBSR-TD algorithm, establish conditions under which it converges almost surely, and develop several variants designed to accelerate convergence. Our formulation shows that existing policy evaluation algorithms for risk-neutral MDPs can be readily adapted to dynamic UBSR settings by incorporating a loss function into the temporal-difference error. Numerical experiments support our theoretical findings, and an application to a perishable inventory management problem with shelf-life uncertainty demonstrates the practical effectiveness of the proposed methods.
• 中文摘要 开发高效的策略评估函数近似方法是风险感知强化学习中的一个根本挑战。现有方法要么专注于限制性风险测量类别，要么依赖模拟器访问，限制了其在完全在线环境中的适用性。本研究提出计算高效的在线学习算法，用于马尔可夫决策过程（MDP）中基于动态效用的短缺风险（UBSR）线性函数近似下的策略评估。具体来说，我们引入了UBSR-TD算法，确定其几乎必然收敛的条件，并开发了多个旨在加速收敛的变体。我们的表述表明，现有的风险中性MDP策略评估算法可以通过在时间差分误差中加入损失函数，轻松适应动态UBSR环境。数值实验支持我们的理论发现，且应用于具有保质期不确定性的易腐库存管理问题，展示了所提方法的实际有效性。

DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding

DishSeg24k：基于随机专家解码的大规模食品分段基准

• Authors: Yilin Wang, Haochen Shi, Guanyu Chen, Weiqing Min, Jinkai Zheng, Chenggang Yan, Shuqiang Jiang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.23070
• Pdf link: https://arxiv.org/pdf/2607.23070
• Abstract Food segmentation is essential for applications such as intelligent catering, dietary assessment, and recommendation. However, existing benchmarks fail to capture the complexity of real-world dining scenes. The challenges of dense inter-dish overlap, fine-grained class similarity, and extreme long-tail class distributions exceed the fidelity of current datasets. To fill this gap, we introduce \textbf{DishSeg24k}, a large-scale dish-level segmentation benchmark with 24,096 images, 112,281 instances, and 278 fine-grained categories in real-world dining environments. Based on DishSeg24k, we further propose \textbf{Food Expert-Adaptive Segmentation Transformers (FEAST)} to address these challenges. FEAST models query-based decoding as a Markov Decision Process (MDP), where each decoder layer update is treated as a sequential decision step that explores uncertainty along dish boundaries. We further redesign the decoder with a reinforcement learning (RL)-guided Mixture-of-Experts (MoE) module, in which a dual-critic decoupled optimization scheme separates task-oriented query refinement from structure-aware expert routing. This design promotes expert specialization and prevents expert collapse under long-tail category distributions. Finally, extensive experiments on DishSeg24k demonstrate the state-of-the-art performance of FEAST, which outperforms previous methods by {+3.21\%} mIoU, {+3.68\%} mDice, and {+4.00\%} mAcc, respectively. We further validate the effectiveness of FEAST on FoodSeg103. The dataset and code will be publicly released.
• 中文摘要 食品细分对于智能餐饮、饮食评估和推荐等应用至关重要。然而，现有基准未能捕捉真实用餐场景的复杂性。密集的菜间重叠、细粒度类别相似性和极端长尾类别分布等挑战超出当前数据集的保真度。为填补这一空白，我们引入了\textbf{DishSeg24k}，这是一个大规模的菜级细分基准测试，包含24,096张图片、112,281个实例和278个细粒度类别，适用于真实用餐环境。基于DishSeg24k，我们进一步提出了\textbf{Food Expert-Adaptive Segmentation Transformers （FEAST）}以解决这些挑战。FEAST将基于查询的译码建模为马尔可夫决策过程（MDP），每次解码层更新都被视为一个顺序决策步骤，探索碟形边界的不确定性。我们进一步用强化学习（RL）引导的专家混合（MoE）模块重新设计了解码器，其中双批判者解耦优化方案将任务导向的查询细化与结构感知的专家路由分离开来。该设计促进了专家的专业化，防止了长尾类别分布下的专家崩溃。最后，在DishSeg24k上的大量实验展示了FEAST的先进性能，其性能分别优于以往方法的{+3.21\%} mIoU、{+3.68\%} mDice和{+4.00\%} mAcc。我们进一步验证了FEAST在FoodSeg103上的有效性。数据集和代码将公开发布。

Traceable LLM Reasoning for Fake-Order Fraud Detection

可追溯的大型语言模型推理用于假订单欺诈检测

• Authors: Siqi You, Bingsong Xu, Zhixian Zheng, Xinjian Peng, Yang Xie, Ying Wang, Jiarong Xu
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23075
• Pdf link: https://arxiv.org/pdf/2607.23075
• Abstract Detecting fake-order fraud at scale remains a critical challenge for large online-to-offline (O2O) service platforms, as existing approaches often rely on expert-designed features, produce black-box decisions, and provide limited interpretability. To address these limitations, we propose DeepScrub, a reinforcement learning framework built upon large language models (LLMs) for fake-order fraud detection with traceable reasoning. DeepScrub introduces three innovations. First, a semantic unification module converts heterogeneous risk signals into textual descriptions that LLMs can understand. Second, continued pre-training on risk-control corpora injects domain knowledge, and task rewards jointly evaluate prediction correctness and reasoning quality. Third, the SUggest-REflect (SURE) mechanism incorporates expert feedback and model self-checking to iteratively refine reasoning paths. On a real-world fake-order fraud detection dataset, DeepScrub achieves a macro-F1 score of 85.3%, outperforming the best baseline by 2.7 percentage points. Our task-optimized 8B model further surpasses a 32B model, showing that domain adaptation can matter more than model scale in this setting. In a four-week live pilot, DeepScrub achieved 91.8% precision and 88.5% recall, improving over first-stage human reviewers by 16.6 and 38.8 percentage points. It reduced first-stage manual review workload by 94% and saved nearly one million RMB annually. These results show that DeepScrub improves fraud review accuracy, reduces first-stage review workload, and provides traceable evidence for production risk-review workflows.
• 中文摘要 大规模检测假订单欺诈仍是大型线上到线下（O2O）服务平台面临的关键挑战，因为现有方法常依赖专家设计的功能，产生黑箱决策，且解释性有限。为解决这些局限性，我们提出了DeepScrub，一个基于大型语言模型（LLM）构建的强化学习框架，用于假订单欺诈检测，具有可追溯推理性。DeepScrub引入了三项创新。首先，语义统一模块将异构风险信号转换为LLM可理解的文本描述。其次，持续对风险控制语料库进行预训练，注入领域知识，任务奖励共同评估预测正确性和推理质量。第三，SUggest-REflect（SURE）机制结合专家反馈和模型自我检查，迭代优化推理路径。在真实的伪造订单欺诈检测数据集上，DeepScrub 的宏观 F1 得分为 85.3%，比最佳基线高出 2.7 个百分点。我们的任务优化 8B 模型进一步超越了 32B 模型，表明在该环境中，领域适应性可能比模型规模更为重要。在为期四周的现场试点中，DeepScrub 实现了 91.8% 的准确率和 88.5% 的召回率，比第一阶段人工审核员提升了 16.6% 和 38.8 个百分点。它将第一阶段人工审核的工作量减少了 94%，每年节省近百万人民币。这些结果表明，DeepScrub 提高了欺诈审查的准确性，减轻了第一阶段审查的工作量，并为生产风险审查工作流程提供了可追溯的证据。

AgentOmnia: Scaling Agentic Models for Full-Scenario Applications

AgentOmnia：全场景应用的智能模型尺度化

• Authors: Hao Jiang, Gangtao Xin, Yingdi Huang, Guojie Zhu, Jiangshan Zhang, Xinyuan Lin, Yunkun Xu, Chengyu Shen, Wenlong Fei, Jiawei Li, Yujie Fu, Sichen Kang, Tingyu Xie, Yedi Hu, Jingren Zhang, Hongcheng Gao, Jianshu Zeng, Chong Chen, Chang Guo, Chao Feng, Feng Wang, Fulin Lin, Jinchao Ma, Lang Mei, Li Huang, Liyan Liu, Qing He, Shuting Tao, Siyu Mo, Xiangnan Chen, Xiaohan Yu, Xiaoyang Li, Yanheng Hou, Yanyu Wu, Zhihan Yang, Wentao Zhang, Yang Gao, Zhao Cao
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.23124
• Pdf link: https://arxiv.org/pdf/2607.23124
• Abstract Large language model agents have advanced rapidly, yet progress remains fragmented across domains, capabilities, task difficulty, and interaction settings. We frame this as full-scenario agentic scaling and present AgentOmnia, a framework coordinating task-space definition, data synthesis, post-training, evaluation, and improvement across To-Consumer (ToC), To-Business (ToB), and To-Employee (ToE) applications. An extensible Domain x Capability x Atomic Difficulty taxonomy aligns these stages and enables fine-grained diagnosis with OmniaBench. AgentOmnia combines bidirectional environment-task synthesis with tool-dependency, program-structured, and solver-based pipelines, constructing 5,018 stateful environments with 255,375 tools and 52,361 tasks. Programs, solvers, and verifiers provide correctness signals, while supervised fine-tuning, online agentic reinforcement learning, and a rollback curriculum support post-training. Evaluation failures translate into Product Requirement Documents (PRDs) for targeted self-evolution. Starting from Qwen3-30B-A3B-Thinking-2507, AgentOmnia raises the pass rate on the OmniaBench challenging subset from 9.16% to 37.11% and the macro-average across OmniaBench, $\tau^2$-Bench, DeepPlanning, and VitaBench from 22.86% to 41.69%. Under a unified protocol,it leads the evaluated agentic post-trained baselines on OmniaBench and retains the highest four-benchmark macro-average. It also surpasses Qwen3-235B-A22B-Thinking-2507 on all four benchmarks and exceeds Qwen3.5-35B-A3B on the macro-average. Gains span three application splits, ten capability dimensions, eight atomic-difficulty factors, and 76 of 90 level-1 domains, indicating broad rather than category-specific improvement. A one-round study provides initial evidence for PRD-guided self-evolution, motivating validation at larger scales and in industrial settings.
• 中文摘要 大型语言模型代理发展迅速，但进展在不同领域、能力、任务难度和交互设置上仍处于碎片化状态。我们将此框架定义为全场景代理扩展，并呈现AgentOmnia，一个协调任务空间定义、数据综合、后期培训、评估和改进的框架，涵盖面向消费者（ToC）、业务到业务（ToB）和员工到员工（ToE）应用。可扩展的领域×能力×原子难度分类法对齐了这些阶段，并支持OmniaBench的细粒度诊断。AgentOmnia结合了双向环境-任务综合、工具依赖性、程序结构化和基于求解器的流程，构建了5,018个有状态环境，拥有255,375个工具和52,361个任务。程序、求解器和验证器提供正确性信号，同时监督式微调、在线智能体强化学习和回滚课程支持培训后。评估失败转化为产品需求文档（PRD），用于有针对性的自我演进。从Qwen3-30B-A3B-Thinking-2507开始，AgentOmnia将OmniaBench挑战子集的通过率从9.16%提升至37.11%，宏观平均值从22.86%提升至41.69%。在统一协议下，AgentOmnia在OmniaBench上领先评估的代理训练后基线，并保持最高的四个基准宏观平均值。它还在四个基准测试中超过了Qwen3-235B-A22B-Thinking-2507，并在宏观平均上超过Qwen3.5-35B-A3B。提升涵盖三个应用拆分、十个能力维度、八个原子难度因素以及90个一级领域中的76个，显示出广泛的而非类别特定的提升。一项单轮研究提供了PRD引导自我演化的初步证据，促使在更大规模和工业环境中进行验证。

Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation

带有噪音学生政策自我提炼的自我提升视觉语言模型

• Authors: Shuai Wang, Daoan Zhang, Zhe Tang, Hao Cheng, Jiaheng Wei
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23125
• Pdf link: https://arxiv.org/pdf/2607.23125
• Abstract Post-training enables vision-language models (VLMs) to understand human instructions and perform various downstream tasks. Current post-training methods usually rely on human-annotated data, distillation from external models, reinforcement learning with human feedback, or verifiable answers. This limits their ability to improve without external supervision. To tackle this, we propose NOPD (Noisy Student On-Policy Self-Distillation), a simple yet effective self-distillation approach that improves VLMs without any external models or ground-truth answers. Our key insight is that prediction discrepancies between clean and corrupted inputs naturally induce a self-supervision signal. In NOPD, the model learns from corrupted inputs while using its own predictions under clean inputs as token-level supervision. We show the effectiveness of NOPD on five visual reasoning tasks; it can match and even outperform reinforcement learning approaches or distillation from external models. Notably, when trained with 2.1K samples from Geometry3K, NOPD improves Qwen2.5-VL-7B by 20 points on its validation set. It also shows generalization on out-of-distribution test sets and achieves 7.4 point gains on MathVista. Furthermore, we demonstrate that NOPD is a general approach to enhance VLMs, achieving improvements across three models on 12 benchmarks.
• 中文摘要 后期训练使视觉语言模型（VLMs）能够理解人类指令并执行各种下游任务。当前的后期训练方法通常依赖于人类注释数据、外部模型的提炼、基于人类反馈的强化学习或可验证的答案。这限制了它们在无外部监督下提升的能力。为解决这个问题，我们提出了NOPD（噪声学生政策自蒸馏），这是一种简单但有效的自蒸馏方法，无需外部模型或真实答案即可改进VLM。我们的关键见解是，干净输入与损坏输入之间的预测差异自然会诱导自我监督信号。在NOPD中，模型从损坏输入中学习，同时使用自身在干净输入下的预测作为标记级监督。我们展示了NOPD在五个视觉推理任务上的有效性;它可以匹配甚至超过强化学习方法或外部模型的提炼。值得注意的是，在使用 Geometry3K 的 2.1K 样本训练时，NOPD 在其验证集上提升了 Qwen2.5-VL-7B 20 个百分点。它还在分布外测试集上展现了推广性，并在 MathVista 上实现了 7.4 个百分的提升。此外，我们证明了 NOPD 是一种通用方法，用于增强 VLM，在 12 个基准测试中实现了三个模型的改进。

SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

SeekJudge：计算机智能体强化学习的实用奖励框架

• Authors: Yang Wan, Zhenhao Zhang, Jierui Wang, Linchao Zhu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23263
• Pdf link: https://arxiv.org/pdf/2607.23263
• Abstract Deciding whether a trajectory actually fulfills its instruction governs how we measure computer-use agents on long-horizon graphical-user-interface tasks and how we train them with reinforcement learning. This judgment has long relied on rule-based evaluation, which struggles to align with human intention and goes stale when an app updates or its online content drifts. Existing model-based judges attempt to address these problems but still leave a performance gap to the rule-based evaluation. We propose the \textbf{SeekJudge} framework, in which four role-specialized agents, a Condense, a Ground, a Seek and an Analyze agent, reach a verdict through a Seek--Analyze loop over the trajectory. A seed-calibrated distillation pipeline trains one specialized $9$B model to serve as the shared backbone for all four agents. Measured by downstream success rate on held-out RL test goals, SeekJudge is the first practical model-based reward to match or surpass native rule-based supervision in online RL. Beyond accuracy, SeekJudge provides step-level judgments, runs far cheaper than a closed-source large model, and keeps a small per-call context that scales to much longer trajectories. We further contribute a general architectural improvement to the reward server that speeds up judging in RL. Together these make model-based reward a practical drop-in for rule-based supervision in CUA reinforcement learning.
• 中文摘要 判断轨迹是否真正完成指令，决定了我们如何衡量计算机代理在长期图形用户界面任务中的表现，以及如何通过强化学习进行训练。这种判断长期依赖于基于规则的评估，但该评估难以与人类意图保持一致，当应用更新或在线内容漂移时，评估就会变得乏味。现有基于模型的评判试图解决这些问题，但仍为基于规则的评估留下性能差距。我们提出了 \textbf{SeekJudge} 框架，其中四个角色专用代理——一个凝聚代理、一个地块代理、一个寻点代理和一个分析代理——通过轨迹上的寻-分析循环达成结论。一个种子校准蒸馏管道训练一个专门的9亿美元模型，作为四个代理的共享骨干。通过对既定强化学习测试目标的下游成功率衡量，SeekJudge是首个在在线强化学习中与原生规则监督匹配甚至超越的实用模型奖励。除了准确性，SeekJudge还提供步级判断，运行成本远低于闭源大型模型，并且保持了小规模的每次调用上下文，且可扩展到更长的轨迹。我们还为奖励服务器带来了一项通用架构改进，加快了强化学习中的判断速度。这些因素共同使模型奖励成为CUA强化学习中基于规则的监督的实用辅助工具。

On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards

关于无偏且长度不变的策略优化与结果奖励的不可能性

• Authors: Fei Ding, Yongkang Zhang, Yuhao Liao, Zijian Zeng, Huiming Yang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23364
• Pdf link: https://arxiv.org/pdf/2607.23364
• Abstract Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek-R1. The recent improvement Dr. GRPO (COLM 2025) identifies the response-level length bias caused by per-trajectory length normalization in GRPO and proposes removing this normalization, claiming the resulting optimizer is "unbiased." We show that this claim is incomplete. Specifically, we establish an impossibility theorem: under the standard outcome reward + GRPO setting, no length-based weighting scheme can simultaneously achieve the following two properties. (P1) Gradient unbiasedness: the gradient estimator is an unbiased estimate of the true policy gradient. (P2) Length invariance: each trajectory's effective contribution to the gradient is independent of its token length. GRPO approximately satisfies P2 but violates P1; Dr. GRPO satisfies P1 but violates P2. We characterize the complete tradeoff spectrum via the parametric family f_alpha(L) = L^{alpha - 1}, where alpha = 0 recovers GRPO, alpha = 1 recovers Dr. GRPO, and provide quantitative analysis showing that Dr. GRPO's length bias can cause longer trajectories to dominate gradient updates by a factor proportional to the length ratio. Our results reveal that neither algorithm is universally "done right"; they occupy opposite ends of a fundamental and unavoidable tradeoff.
• 中文摘要 群相对策略优化（GRPO）是大型语言模型中训练推理能力的主流强化学习算法，尤其被DeepSeek-R1采用。最新的改进Dr. GRPO（COLM 2025）识别了GRPO中每轨迹长度归一化引起的响应级长度偏差，并提出去除该归一化，声称最终优化器是“无偏的”。我们证明该主张不完整。具体来说，我们建立了一个不可能定理：在标准结果奖励+GRPO设置下，任何基于长度的权重方案都无法同时实现以下两个性质。（P1）梯度无偏性：梯度估计器是真实策略梯度的无偏估计。（P2）长度不变性：每个轨迹对梯度的有效贡献与其标记长度无关。GRPO近似满足P2但违反P1;DR. GRPO满足P1但违反P2。我们通过参数族 f_alpha（L） = L^{alpha - 1}来刻画完整的权衡谱，其中alpha = 0恢复GRPO，alpha = 1恢复Dr. GRPO，并提供定量分析，表明DR. GRPO的长度偏差会导致更长的轨迹以与长度比成正比的因子优势支配梯度更新。我们的结果表明，这两种算法都不是普遍“正确”的;它们处于基本且不可避免权衡的两端。

LA-RL: Label-Aware Self-Reflection for Reinforcement Learning in Information Extraction

LA-RL：信息提取中强化学习的标签感知自我反思

• Authors: Xiao You, Tianwei Yan, Zixu Shan, Longyu Du, Shan Zhao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.23420
• Pdf link: https://arxiv.org/pdf/2607.23420
• Abstract Large language models show strong promise for information extraction (IE), but existing reflection-based correction methods are often misaligned with structured extraction outputs. Free-form self-reflection can flag an error, yet it rarely identifies whether the failure is a missing span, wrong label, boundary mismatch, invalid relation type, or reversed argument order. We introduce LA-RL (Label-Aware Reflective Reinforcement Learning), an outcome-supervised framework that guides IE self-correction with task-grounded diagnostic labels. A single backbone first predicts an extraction, diagnoses task-specific error labels, and then revises its output conditioned on the diagnosis. Training starts from diagnostic data labeled by an annotation model for cold-start supervised fine-tuning and proceeds through two GRPO stages that reward final extraction quality, format validity, and first-pass correctness, without a process reward model. Experiments on named entity recognition, relation extraction, and event extraction show consistent same-backbone gains over SFT, including 6.83 average F1 on SciER relation extraction, about 20 F1 on out-of-distribution relation extraction, and 14.80 trigger F1 plus 17.50 argument F1 on DuEE1.0. Ablations show that reflection structure is task-sensitive: stronger constraints benefit relation extraction, whereas named entity recognition needs less restrictive correction under domain shift.
• 中文摘要 大型语言模型在信息提取（IE）方面展现出强烈潜力，但现有基于反射的纠正方法常常与结构化提取输出不匹配。自由形式自反思可以标记错误，但很少能识别失败是缺少跨度、错误标签、边界不匹配、无效关系类型或反向参数顺序。我们引入了LA-RL（标签感知反思强化学习），这是一个结果监督框架，指导IE通过基于任务的诊断标签进行自我纠正。单一骨干首先预测提取，诊断任务特定错误标签，然后根据诊断调整输出。训练从由注释模型标记的诊断数据开始，进行冷启动监督微调，经过两个GRPO阶段，奖励最终提取质量、格式有效性和首轮正确性，无需过程奖励模型。命名实体识别、关系提取和事件提取的实验显示，同骨干在SFT上均有稳定的提升，包括SciER关系提取的平均F1为6.83，分布外关系提取约为20为F1，在DuEE1.0为14.80触发F1加17.50参数F1。消融显示反射结构具有任务敏感性：更强的约束有利于关系提取，而命名实体识别在域移位下需要的限制性修正较少。

Learning to Optimize: Joint Routing and Flow Allocation on Sparse Non-Euclidean Networks

学习优化：稀疏非欧几里得网络上的联合路由与流量分配

• Authors: Haomiao Sun, Fang He, Congyuan Ji, Xindi Tang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23467
• Pdf link: https://arxiv.org/pdf/2607.23467
• Abstract We study an integrated pickup-and-delivery problem on sparse, non-Euclidean networks that jointly optimizes cyclic routing, cargo flow allocation, and cross-cycle service. The tight coupling of these operational constraints creates a complex discrete-continuous decision space with highly restricted feasible regions. To overcome these computational challenges, we propose Double-Channel Graph Attention (DCGA), an end-to-end reinforcement learning framework. DCGA isolates network reachability and demand-service logic into separate graph channels and constructs valid routes using a simulator-coupled, constraint-informed decoder. Experiments on LinerLib benchmarks demonstrate that DCGA achieves seconds-level inference and delivers state-of-the-art solution quality on instances beyond a specific scale, with its advantage over existing baselines widening significantly as problem size increases. Supported by extensive stability and ablation analyses, our results demonstrate that this structure-aware learning approach provides an effective, low-latency engine for realistic routing-and-flow optimization.
• 中文摘要 我们研究了一个在稀疏、非欧几里得网络上的集成取货与交付问题，该问题共同优化循环路由、货物流分配和跨周期服务。这些操作约束的紧密耦合创造了一个复杂且连续的决策空间，且可行区域极为受限。为克服这些计算挑战，我们提出了双通道图关注（DCGA），一种端到端强化学习框架。DCGA将网络可达性和需求服务逻辑分离为独立的图通道，并利用模拟器耦合的约束知情解码器构建有效路径。LinerLib基准测试的实验表明，DCGA在特定尺度之外的实例中实现了秒级推断，并提供最先进的解决方案质量，随着问题规模的增加，其相较现有基线的优势显著扩大。在大量稳定性和消融分析的支持下，我们的结果表明，这种结构感知学习方法为真实的路由和流量优化提供了高效、低延迟的引擎。

PRISM: Polynomial Representations for Interaction-Structured Motor Control

棱镜：交互结构运动控制的多项式表示

• Authors: Seung Hyun Lee, Stella X. Yu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.23473
• Pdf link: https://arxiv.org/pdf/2607.23473
• Abstract Robot policies are typically MLPs mapping observations to actions. Yet robot observations are physical variables, and many action-relevant cues arise not from individual variables but from their interactions; power, inertial effects, contact, slip, and compliance depend on products among observable signals. We introduce PRISM, a policy representation that makes polynomial interactions among observable physical variables explicit, learnable, and compact. Rather than listing all polynomial terms, PRISM uses a factorized polynomial module to expose higher-order interaction features efficiently. In reinforcement learning, it keeps the standard MLP backbone but applies a gradually activated element-wise polynomial function after it. In imitation learning, it replaces linear proprioceptive conditioning in Diffusion Policy with a polynomial layer trained end-to-end. Across humanoid locomotion and contact-rich manipulation, PRISM improves performance over standard MLP policies and larger MLPs with matched capacity, showing that interaction structure cannot be replaced by capacity alone. It also yields sensorless compliant behavior without force, wrench, tactile input, contact labels, or admittance control. These results suggest that polynomial representations should become a standard architectural choice for embodied motor control. The project page is available at this https URL
• 中文摘要 机器人策略通常是将观测值映射为动作的MLP。然而，机器人的观测是物理变量，许多与动作相关的线索并非来自单个变量，而是来自它们的相互作用;功率、惯性效应、接触、滑移和顺应性依赖于可观测信号之间的乘积。我们引入了PRISM，这是一种策略表示，使可观测物理变量之间的多项式相互作用显式、可学习且紧凑。PRISM不列举所有多项式项，而是使用分解后的多项式模高效暴露高阶交互特征。在强化学习中，它保留标准MLP骨干，但随后应用一个逐步激活的元素级多项式函数。在模仿学习中，它用端到端训练的多项式层取代扩散策略中的线性本体感受条件。在类人机动和接触丰富操作中，PRISM提升了比标准MLP策略和更大MLP匹配容量的MLP的性能，表明交互结构无法仅靠容量来替代。它还能实现无传感器的顺应行为，无需力、扳手、触觉输入、接触标签或导纳控制。这些结果表明，多项式表示应成为具身运动控制的标准架构选择。项目页面可访问此 https URL

Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

通过哈达玛德超参数化实现稀疏高斯混合模型Q函数的在线强化学习

• Authors: Minh Vu, Konstantinos Slavakis
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23474
• Pdf link: https://arxiv.org/pdf/2607.23474
• Abstract This paper develops an online, off-policy policy-iteration framework for reinforcement learning (RL), based on sparse Gaussian-mixture-model Q-functions (S-GMM-QFs). The framework reconciles streaming, non-stationary data with the Riemannian structure of the parameter space while handling distributional mismatch through experience replay. S-GMM-QFs are introduced via Hadamard overparametrization, enabling interpretable sparsification through smooth regularization that facilitates Riemannian-based optimization. Overparametrization allows the framework to adaptively identify meaningful components from a large initial pool, yielding sparse models where interpretability emerges naturally from geometry: each component's parameters (means and covariances) explicitly encode its geometric role in the ambient state-action space. These geometric roles are learned through online gradient descent on a smooth objective over a (Cartesian-product) Riemannian manifold. Numerical tests demonstrate that S-GMM-QFs match or exceed deep RL methods while using substantially fewer parameters and achieving faster improvement per observed transition. Notably, parameter efficiency and interpretability combine to maintain strong generalization in low-parameter regimes where sparsified deep RL approaches degrade.
• 中文摘要 本文开发了一个基于稀疏高斯混合模型Q函数（S-GMM-QF）的在线、非策略策略迭代（RL）强化学习框架。该框架通过经验回放处理分布不匹配，协调流的非平稳数据与参数空间的黎曼结构。S-GMM-QF通过哈达玛过参数化引入，通过平滑正则化实现可解释的稀疏化，便于基于黎曼优化。过参数化使该框架能够自适应地从大量初始池中识别有意义的组件，产生稀疏模型，其中几何自然产生可解释性：每个组件的参数（均值和协方差）显式编码其在环境状态-作用空间中的几何角色。这些几何角色通过在（笛卡尔积）黎曼流形上的光滑目标线上梯度下降来学习。数值检验表明，S-GMM-QFs在使用显著更少参数且每次观察跃迁提升更快的情况下，能够匹配或超过深度强化学习方法。值得注意的是，参数效率和可解释性结合在一起，在稀疏的深部强化学习方法退化的低参数环境中保持了强强的泛化性。

Learning Sampling Parameters for Diffusion Models

学习扩散模型的采样参数

• Authors: Arisrei Lim, Yossi Gandelsman
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.23488
• Pdf link: https://arxiv.org/pdf/2607.23488
• Abstract Text-to-image diffusion models expose many inference-time sampling parameters, including prompts, negative prompts, classifier-free guidance scales, and noise schedules. These parameters are typically manually chosen once and then held fixed across prompts and denoising timesteps, even though different prompts and stages of generation can benefit from different parameter values. We introduce LeSAMP, a framework for learning prompt-conditioned, timestep-varying sampling parameters. We formulate parameter selection as a reinforcement learning problem: Given a user prompt, a large language model is trained to emit schedules for the chosen sampling parameters. We optimize our model using rewards from human preference models and VLM-as-a-judge. We evaluate our model on Flux.1 [dev] and Stable Diffusion 3.5, and find that compared to baselines, LeSAMP has a win rate of up to 68.12% using human preference scores and 73.37% using VLM-as-a-judge. These gains are validated in a user study where we achieve win rates of up to 59.46% over previous baselines. Our results suggest that learned sampling-parameter policies provide a complementary approach to existing post-training methods for improving diffusion model outputs.
• 中文摘要 文本到图像扩散模型揭示了许多推理时间采样参数，包括提示词、负提示、无分类器的指导量表和噪声计划。这些参数通常手动选定一次，然后在提示和去噪时间步间保持固定，尽管不同提示和生成阶段可能受益于不同的参数值。我们引入了LeSAMP，一个用于学习提示条件、时间步变化采样参数的框架。我们将参数选择表述为强化学习问题：给定一个用户提示，一个大型语言模型被训练以发射所选采样参数的调度。我们利用人类偏好模型和VLM作为评判的奖励来优化模型。我们在Flux.1 [开发]和Stable Diffusion 3.5上评估模型，发现与基线相比，LeSAMP的胜率高达68.12%，使用VLM作为评判时为73.37%。这些提升在用户研究中得到验证，我们比之前基线实现了高达59.46%的胜率。我们的结果表明，学习到的采样参数策略为现有训练后方法提供了补充方法，以提升扩散模型输出。

LEACL: LLM-Enhanced Automatic Curriculum Learning for Reinforcement Learning in Long-Horizon Manipulation Tasks

LEACL：LLM增强型自动课程学习，用于长期视野操作任务中的强化学习

• Authors: Faraz Heravi, James Ouyang, Zifan Xu, Arjun Kumar, Yoonchang Sung, Peter Stone
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.23515
• Pdf link: https://arxiv.org/pdf/2607.23515
• Abstract Long-horizon manipulation tasks pose significant challenges for reinforcement learning due to sparse reward signals and long horizons. Automatic curriculum learning (ACL) has been proposed to tackle these challenges by progressively training agents on a sequence of tasks, from easier to more difficult. However, the success of ACL depends heavily on task-dependent specifications-such as well-defined task parameter spaces and difficulty measures-which are often manually crafted and difficult to generalize across diverse tasks. Recent advances in large language models (LLMs) offer a promising alternative by enabling the decomposition of complex tasks into meaningful subtasks using the LLMs' web-scale common-sense knowledge. This decomposition can provide a natural curriculum structure for efficient learning of long-horizon tasks. However, existing LLM-based methods typically rely on hand-designed dense reward functions to learn each subtask, which can introduce bias and still requires significant human supervision. In this work, we propose LLM-enhanced automatic curriculum learning (LEACL), a framework that integrates LLMs and ACL to address these limitations. Specifically, LLMs are used to both decompose tasks into subtasks and to generate task-dependent specifications for each subtask. These specifications are then used by ACL algorithms to guide learning using only sparse reward signals, eliminating the need for dense reward design. We evaluate LEACL on five long-horizon manipulation tasks from the LIBERO benchmark. LEACL achieves better asymptotic performance in terms of the success rates compared to human-designed dense rewards.
• 中文摘要 长视野操作任务因奖励信号稀疏和视野较长，对强化学习构成重大挑战。自动课程学习（ACL）被提出通过逐步训练代理完成一系列任务（从简单到更难）来应对这些挑战。然而，ACL的成功高度依赖于任务相关的规范——如明确定义的任务参数空间和难度度量——这些通常需要手动设计，难以在不同任务间推广。大型语言模型（LLM）的最新进展为复杂任务分解为有意义的子任务提供了有前景的替代方案，利用LLMs的网络级常识知识。这种分解为高效学习长视野任务提供了自然的课程结构。然而，现有基于LLM的方法通常依赖手工设计的密集奖励函数来学习每个子任务，这可能带来偏见，且仍需大量人工监督。本研究提出LLM增强型自动课程学习（LEACL），该框架整合LLM与ACL以解决这些限制。具体来说，LLM用于将任务分解为子任务，并为每个子任务生成任务相关的规范。ACL算法随后利用这些规范仅使用稀疏的奖励信号来引导学习，消除了密集奖励设计的需求。我们评估了LEACL在LIBERO基准测试中的五个长期视野操作任务。LEACL在渐近成功率方面优于人类设计的密集奖励。

ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awareness

ObsDriveBench：在恶劣天气下以可观测性意识进行多模态理解的基准测试

• Authors: Qiao Yan, Yihan Wang, Zhenghao Xing, Jiaqi Xu, Pheng-Ann Heng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23537
• Pdf link: https://arxiv.org/pdf/2607.23537
• Abstract Autonomous driving under adverse weather remains a critical challenge, yet existing vision-language benchmarks mainly evaluate under standard conditions, synthetic corruptions, or single modality. As a result, it remains unclear how vision-language models behave under real-world adverse weather with multi-modal inputs. We argue that a key difficulty lies in degraded environmental observability: under fog, rain, snow, and low illumination, multi-modal observations become unreliable and cross-modally inconsistent, posing challenges to scene understanding, and subsequent decision-making. To study this, we introduce \textbf{ObsDriveBench}, a real-world multi-modal benchmark for adverse-weather autonomous driving. Our benchmark is designed with three capability dimensions: \textbf{observability awareness}, \textbf{spatial reliability}, and \textbf{risk-aware decision-making}, enabling fine-grained diagnosis of model behavior under degraded observations. We construct the benchmark through observability meta-annotation, scene description, and capability oriented multiple-choice tasks over synchronized camera, LiDAR, and radar inputs, forming a benchmark with over 14k training and 13k test questions. Experiments reveal consistent performance degradation of existing vision-language models. We further introduce \textbf{ObsDrive} model with normal-weather supervised fine-tuning and adverse-weather reinforcement learning, improving robustness across all three capabilities. The dataset and evaluation code will be released at \href{this https URL}{\texttt{ObsDriveBench}}.
• 中文摘要 恶劣天气下的自动驾驶仍是一个关键挑战，但现有视觉语言基准主要在标准条件下、合成污染或单一模态下进行评估。因此，视觉语言模型在现实恶劣天气下多模态输入下的表现仍不明确。我们认为一个关键难题在于环境可观测性下降：在雾、雨、雪和低照条件下，多模态观测变得不可靠且跨模态不一致，给场景理解和后续决策带来挑战。为此，我们引入了\textbf{ObsDriveBench}，这是一个现实世界中恶劣天气自动驾驶的多模态基准测试。我们的基准测试设计包含三个能力维度：\textbf{可观测性意识}、\textbf{空间可靠性}和\textbf{风险感知决策}，实现模型在降级观测下行为的细致诊断。我们通过可观测性元注释、场景描述以及基于能力导向的多项选择题构建基准测试，涵盖同步摄像头、激光雷达和雷达输入，形成包含超过1.4万训练和1.3万测试题的基准。实验显示现有视觉语言模型持续性能下降。我们进一步引入了具有正常天气监督微调和恶劣天气强化学习的\textbf{ObsDrive}模型，提升了三项能力的稳健性。数据集和评估代码将在\href{this https URL}{\texttt{ObsDriveBench}}发布。

Anticipatory Risk-Guided Reinforcement Learning for Safe Flight Through Dynamic Clutter

前瞻性风险引导强化学习，助力在动态杂波中安全飞行

• Authors: Yuchao Mei, Guohao Zhang, Luxia Ai, Haopeng Chen, Wenbing Tao
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23565
• Pdf link: https://arxiv.org/pdf/2607.23565
• Abstract Safe quadrotor navigation in cluttered and dynamic environments depends not only on instantaneous geometric perception, but more critically on anticipating collision risks induced by relative motion. Conventional modular pipelines frequently suffer from perception latency, while end-to-end learning methods relying on implicit scalar rewards often struggle to extract reliable spatio-temporal features without physics-grounded supervision. To address this, we propose an anticipatory risk-guided reinforcement learning framework. Leveraging privileged simulator states, we construct a directionally aligned future collision risk map based on the Closest Point of Approach (CPA). Through an asymmetric actor-critic architecture, the network is trained to self-predict this structured risk, which explicitly guides the visual policy during deployment. A lightweight spatio-temporal encoder extracts motion cues directly from onboard depth sequences, bypassing explicit object tracking or optical flow estimation. Extensive simulated and real-world experiments demonstrate that our method effectively improves safety margins and flight efficiency in dense dynamic clutters compared to existing baselines. Furthermore, the learned policy achieves robust zero-shot Sim-to-Real transfer on a physical quadrotor, relying purely on abstracted spatio-temporal depth sequences and its self-predicted risk priors, validating the effectiveness of our approach and its robust generalization from simulation to reality.
• 中文摘要 在杂乱且动态环境中的安全四旋翼导航不仅依赖瞬时几何感知，更关键的是预测由相对运动引发的碰撞风险。传统模块化流水线常常存在感知延迟，而依赖隐式标量奖励的端到端学习方法，在没有物理基础监督的情况下常难以提取可靠的时空特征。为此，我们提出了一种前瞻性风险引导强化学习框架。利用特权模拟器状态，我们构建基于最近进近点（CPA）的方向对齐未来碰撞风险地图。通过非对称行为者-批判者架构，网络被训练为自我预测这种结构化风险，并在部署过程中明确指导视觉策略。轻量级时空编码器直接从机载深度序列中提取运动线索，绕过显式物体跟踪或光流估计。大量模拟和现实实验表明，我们的方法相较于现有基线，在密集动态杂波中有效提升安全裕度和飞行效率。此外，该策略在物理四旋翼上实现了稳健的零次模拟到真实传输，纯粹依靠抽象的时空深度序列及其自我预测的风险先验，验证了我们方法的有效性及其从模拟到现实的稳健推广。

Optimal Reward Shaping: Autonomous Car Parking Case Study

最佳奖励塑造：自动停车案例研究

• Authors: Emre Özkaya, Nicolas R. Gauger
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.23617
• Pdf link: https://arxiv.org/pdf/2607.23617
• Abstract Designing effective reward functions for model-free reinforcement learning under non-holonomic constraints remains a persistent challenge, often resulting in severe local minima such as policy paralysis or over-conservative hazard avoidance. In this work, we present a parameterized reward shaping framework featuring coverage-gated alignment feedback, drive-direction switch regularization, and an aligned episode termination mechanism evaluated on an autonomous parallel parking task. Crucially, we show that environmental reward parameters and algorithmic hyperparameters are deeply co-dependent, requiring joint meta-optimization to achieve stable convergence. By employing surrogate-based Bayesian optimization, our co-optimized Deep Q-Network (DQN) agent resolves characteristic control failure modes, significantly outperforming uncalibrated baselines across both success rate and trajectory smoothness.
• 中文摘要 在非全纯约束下设计无模型强化学习的有效奖励函数仍是一个持续的挑战，常导致严重的局部极小值，如策略瘫痪或过度保守的危险规避。本研究提出了一个参数化奖励塑造框架，包含覆盖门控对齐反馈、行驶方向开关正则化以及在自主平行停车任务中评估的对齐事件终止机制。关键是，我们表明环境奖励参数与算法超参数高度相互依赖，需要联合元优化以实现稳定收敛。通过采用基于代理的贝叶斯优化，我们协优化的深度Q网络（DQN）智能体解决了特征控制失效模式，在成功率和轨迹平滑性上显著优于未校准基线。

Breaking the Total Variance Barrier: Sharp Sample Complexity for Linear Heteroscedastic Bandits with Fixed Action Set

突破总方差障碍：带有固定动作集的线性异方差盗垒的锐利样本复杂度

• Authors: Heyang Zhao, Tianyuan Jin, Weixin Wang, Vincent Y. F. Tan, Pan Xu, Quanquan Gu
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2607.23679
• Pdf link: https://arxiv.org/pdf/2607.23679
• Abstract Recent years have witnessed increasing interests in tackling heteroscedastic noise in bandits and reinforcement learning. In these works, the cumulative variance of the noise $\Lambda = \sum_{t=1}^T \sigma_t^2$, where $\sigma_t^2$ is the variance of the noise at round $t$, is used to characterize the statistical complexity of the problem, yielding \emph{simple regret} bounds of order $\tilde{\cal{O}}(d \sqrt{\Lambda / T^2})$ for $d$-dimensional linear bandits with heteroscedastic noise. However, with a closer look, $\Lambda$ remains the same order even if the noise is close to zero at half of the rounds, which indicates that the $\Lambda$-dependence is not optimal. In this paper, we revisit the stochastic linear bandit problem with heteroscedastic noise, where the action set is prefixed throughout the learning process. We propose a novel variance-adaptive algorithm \texttt{VAEE} (Variance-Aware Exploration with Elimination) for large action set, which actively explores actions that maximizes the information gain among a candidate set of actions that are not eliminated. With the active-exploration strategy, we show that \texttt{VAEE} achieves a \emph{simple regret} with a nearly \emph{harmonic-mean} dependent rate. For finitely many actions, we propose a variance-aware variant of G-optimal design based exploration, which achieves a simple regret with sharper dependence on $d$. We also establish a nearly matching lower bound for the fixed action set setting indicating that \emph{harmonic-mean} dependent rate is unavoidable. To the best of our knowledge, this is the first work that breaks the $\sqrt{\Lambda}$ barrier for stochastic linear bandits with heteroscedastic noise.
• 中文摘要 近年来，人们对强化和强化学习中异方差噪声的解决兴趣日益增长。在这些研究中，噪声$\Lambda = sum_{t=1}^T \sigma_t^2$（其中$\sigma_t^2$是第$t轮噪声的方差），用于描述问题的统计复杂度，给出了具有异方差噪声的$d维线性强盗的\emph{简单遗憾}阶数$$_\tilde{\cal{O}}（d \sqrt{\Lambda / T^2}）$的阶数。然而，仔细观察后，即使噪声在半轮时噪声接近零，$\Lambda$的阶数依然保持不变，这表明$\Lambda$依赖性并非最优。本文重新审视了带有异方差噪声的随机线性盗垒问题，其中动作集在学习过程中始终前缀。我们提出了一种新颖的方差自适应算法 \texttt{VAEE}（方差感知探索与消元），适用于大型动作集，该算法主动探索能够最大化未消除候选动作集信息增益的动作。通过主动探索策略，我们证明 \texttt{VAEE} 实现了 \emph{简单遗憾}，且其速率几乎依赖于 \emph{调和均值}。对于有限多个动作，我们提出了一种基于 G-最优设计的方差感知变体，该变体实现了对 $d 美元更明显依赖的简单遗憾。我们还建立了固定动作集设置的近似下界，表明 \emph{调和均值}依赖率不可避免。据我们所知，这是首个突破随机线性盗垒器在异方差噪声中的$\sqrt{\Lambda}$障碍的工作。

Offline-Online Curriculum RL for Multimodal Reasoning

多模态推理的离线在线课程强化学习

• Authors: Wendi Deng, Hang Du, Guoshun Nan, Haokun Tian, Jiaqi Yu, Xinlei Cao, Jaile Li, Jingfeng Chen, Ling Deng, Ting Li, Hao Yang, Jun Liu, Xudong Jiang, Sicong Leng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23700
• Pdf link: https://arxiv.org/pdf/2607.23700
• Abstract Multimodal large language models exhibit capabilities on reasoning tasks, yet often produce flawed intermediate steps while yielding correct final answers. This behavior undermines interpretability and reliability, suggesting reliance on spurious shortcuts rather than faithful reasoning. Although efforts have explored step-level supervision, distinguishing decisive steps from redundant ones remains challenging. We propose $O^2$-CritiCuRL, a novel curriculum reinforcement learning framework that introduces critical-step awareness through an iterative offline-online paradigm. In the offline stage, $O^2$-CritiCuRL conducts multi-rollout analysis over step-annotated trajectories to estimate step-level importance, allowing the framework to distill critical reasoning steps and filter out redundant ones. In the online stage, we employ a progressive step-level reinforcement learning strategy, where truncated chains guide the model to infer missing steps and refine its reasoning, thereby sharpening its focus on critical steps and overcoming the limitations of static supervision. Extensive experiments on multimodal reasoning benchmarks show that our method achieves state-of-the-art performance while delivering superior training and inference efficiency. Code is available at this https URL.
• 中文摘要 多模态大型语言模型在推理任务中表现出能力，但常常在给出正确最终答案时产生错误的中间步骤。这种行为削弱了可解释性和可靠性，表明他们依赖于虚假的捷径而非忠实推理。尽管已有研究探索步骤级监督，但区分决定性步骤与冗余步骤仍具挑战性。我们提出了$O^2$-CritiCuRL，一种新型课程强化学习框架，通过迭代离线-在线范式引入关键步骤意识。在离线阶段，$O^2$-CritiCuRL对步骤注释轨迹进行多重推广分析，估计步骤重要性，使框架能够提炼关键推理步骤并过滤冗余步骤。在在线阶段，我们采用渐进式步骤级强化学习策略，截断链引导模型推断缺失步骤并优化推理，从而更聚焦关键步骤，克服静态监督的局限。多模态推理基准测试的广泛实验表明，我们的方法在提供更优训练和推理效率的同时，实现了最先进的性能。代码可在此 https URL 获取。

Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning

稀疏奖励长视界强化学习的层级软性演员-批评者

• Authors: Zahra Abdalla Elashaal, Afef Hfaiedh, Nahla Khraief, Issmail Ellabib, Giansalvo Cirrincione
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23726
• Pdf link: https://arxiv.org/pdf/2607.23726
• Abstract Exploration in sparse-reward long-horizon tasks poses significant challenges for reinforcement learning. To address these challenges, we propose a two-level Hierarchical Reinforcement Learning (HRL) framework. The first level handles high-level strategic planning, while the low-level uses the continuous-control Soft Actor-Critic (SAC) algorithm, and they utilize entropy-regularized policy optimization. The proposed framework was trained and evaluated using the Search-and-Rescue-2 (SAR-2) dataset. HRL-SAC effectively addresses sparse-reward long-horizon search problems characterized by delayed rewards and continuous control, and its outperforming the flat SAC baseline reinforcement learning in terms of success rates, coverage efficiency, and convergence. These findings indicate that hierarchical entropy-regularized policies are a promising solution to tackle long-horizon sparse-reward reinforcement learning tasks.
• 中文摘要 在稀疏奖励的长视野任务中探索对强化学习构成了重大挑战。为应对这些挑战，我们提出了一个两级的层级强化学习（HRL）框架。第一层处理高级战略规划，低层使用连续控制软演员-批判者（SAC）算法，并采用熵正则化策略优化。所提框架使用搜索与救援2（SAR-2）数据集进行训练和评估。HRL-SAC有效解决了以延迟奖励和持续控制为特征的稀疏奖励长视野搜索问题，并在成功率、覆盖效率和收敛性方面优于扁平的SAC基线强化学习。这些发现表明，分层熵正则化策略是解决长视野稀疏奖励强化学习任务的有前景方案。

TRUAV: Distributed Multi-Agent Reinforcement Learning for Trajectory Planning and Routing Enhancement in UAV-Aided IoT-Enabled VANETs

TRUAV：分布式多智能体强化学习，用于无人机辅助物联网VANET中的轨迹规划与路由增强

• Authors: Muhammad Umar Farooq Qaisar, Lin Zhang, Zhen Chen, Wajdy Othman, Shehzad Ashraf Chaudhry, Chang Liu
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.23734
• Pdf link: https://arxiv.org/pdf/2607.23734
• Abstract Unmanned aerial vehicles (UAVs) have emerged as a key enabler of next-generation Internet of Things (IoT) ecosystems, offering flexible aerial relaying to extend connectivity across dynamic vehicular ad hoc networks (VANETs) in smart city environments. However, conventional centralized approaches for UAV trajectory planning require continuous global network state aggregation, making them impractical under bandwidth and energy constraints typical of dense urban deployments. In this article, we present TRUAV, a distributed multi-agent reinforcement learning framework based on independent tabular Q-learning for joint UAV trajectory planning and routing enhancement in UAV-aided VANETs. Each UAV is equipped with a local Q-learning agent that operates purely on locally observable information, including vehicle density, packet queue states, and neighbor UAV positions, thereby eliminating the need for global state exchange. A potential-game-inspired reward design encourages spatial diversity and routing-aware UAV positioning among interacting agents while accounting for energy consumption. Numerical simulations over a large urban area with 200 mobile vehicles show that the proposed TRUAV framework achieves network coverage and packet delivery ratios comparable to centralized deep reinforcement learning methods, while also improving relay delay and energy efficiency. Finally, we discuss emerging challenges and future research directions for distributed multi-agent UAV-assisted IoT systems.
• 中文摘要 无人机（UAV）已成为下一代物联网（IoT）生态系统的关键推动力，提供了灵活的空中中继，扩展智能城市环境中动态车辆自组网（VANETs）之间的连接。然而，传统的集中式无人机轨迹规划方法需要持续的全球网络状态聚合，在密集城市部署典型的带宽和能源限制下难以实现。本文介绍了TRUAV，一种基于独立表格Q学习的分布式多智能体强化学习框架，用于无人机辅助VANET中的联合无人机轨迹规划和路线优化。每架无人机配备本地Q学习代理，仅基于本地可观测信息操作，包括车辆密度、数据包队列状态和邻近无人机位置，从而消除了全球状态交换的需求。潜在的游戏启发奖励设计鼓励交互代理间空间多样性和路径感知型无人机定位，同时考虑能耗。在拥有200辆移动车辆的大型城市区域进行数值模拟显示，拟议的TRAUV框架实现了与集中式深度强化学习方法相当的网络覆盖和数据包分包分配比率，同时提升了中继延迟和能效。最后，我们讨论了分布式多智能体无人机辅助物联网系统的新兴挑战和未来研究方向。

Zing: Social Mind for LLMs

Zing：面向大型语言模型的社交思维

• Authors: Zing Team, Ao Xiang, Bi Jingping, Chen Jiahui, Chen Lehan, Chen Yilin, Cheng Xueqi, Fan Yixing, Gan Kairong, Gao Haowen, Gao Jinhua, Gao Shuxuan, Gong Chang, Guo Jiafeng, Guo Ruijie, Han Zhouyu, He Guangfu, He Yichun, Jiang Shuo, Jing Shaoling, Jing Ya, Lei Chenhao, Lei Yan, Li Anqi, Li Chengao, Li Haoyu, Li Shitian, Liang Xinjian, Liu Zhaoge, Lyu Xingyu, Nie Zhuwei, Pang Liang, Quan Zeping, Shan Shiguang, Shen Huawei, Tang Xinran, Tian Feng, Wang Qian, Wang Ruiping, Wang Xiaohong, Xia Zaiyu, Xiao Yi, Xu Jiayuan, Xu Kehan, Xu Qianqian, Xu Tianyu, Xu Yongjun, Yang Haoming, Yang Jun, Yao Di, Yu Xiaoming, Zhang Futong, Zhang Jie, Zhang Shixuan, Zhang Yuxuan, Zhao Xinyu, Zhao Zhuoran, Zhong Yunfei, Zhu Shengyu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.23740
• Pdf link: https://arxiv.org/pdf/2607.23740
• Abstract As large language models move from isolated task solving toward long-term service in human environments, they require social intelligence: the ability to infer mental states, track social relations, reason over norms, and adapt behavior under context. This report presents Zhijing, an integrated framework for measuring, internalizing, and grounding social intelligence. For measurement, we introduce SoMBench, a psychology-grounded benchmark spanning 3 primary dimensions, 17 secondary dimensions, and 71 task paradigms. It controls question format, narrative perspective, and context length across 284 shared scenarios and 3,481 expert-verified instances. Evaluation of 20 representative LLMs reveals substantial headroom: the best model achieves only 72.08% overall accuracy, and none of the 17 secondary dimensions reaches the 90% near-ceiling band. For internalization, we develop Zing, a diagnosis-driven training recipe combining supervised fine-tuning, on-policy distillation, and rubric-based reinforcement learning. Across five social-cognition benchmarks, Zing consistently outperforms its base models, with Zing-27B-Stage2 achieving the best average score and Zing-32B-Stage2 remaining competitive with DeepSeek-V4-Pro. For deployment-time grounding, we build Actio, a harness-controlled inference architecture that routes four typed supports into reasoning: PRISM for procedural guidance, Starling for runtime mental-state representation, SAGE for reusable experience, and gated RAG for external social and normative knowledge. Across five base models and three benchmarks, the full harness improves 14 of 15 model-benchmark pairs and is best or tied for best in 8, demonstrating the effectiveness of typed runtime support. Together, these results show that socially intelligent LLMs require coordinated advances in evaluation, parametric internalization, and deployment-time grounding.
• 中文摘要 随着大型语言模型从孤立任务解决转向长期人类环境服务，它们需要社会智能：推断心理状态、追踪社会关系、超越规范推理，以及根据上下文调整行为的能力。本报告介绍了智镜，一个用于衡量、内化和扎根社会智力的综合框架。在测量方面，我们引入SoMBench，这是一个基于心理学的基准，涵盖3个主要维度、17个次要维度和71个任务范式。它控制了284个共享场景和3481个专家验证实例中的问题格式、叙述视角和上下文长度。对20个具有代表性的大型语言模型的评估显示出相当大的余裕：最佳模型整体准确率仅为72.08%，且17个次要维度中没有一个达到90%接近天花板区间。在内化方面，我们开发了Zing，一种结合监督微调、策略提炼和基于评分标准强化学习的诊断驱动训练方案。在五个社会认知基准中，Zing持续优于其基础模型，Zing-27B-Stage2获得最佳平均分，Zing-32B-Stage2与DeepSeek-V4-Pro保持竞争力。在部署时间基础方面，我们构建了Actio，一种基于约束的推理架构，将四个类型支持导入推理：PRISM用于程序性指导，Starling用于运行时心理状态表示，SAGE用于可重用经验，门控RAG用于外部社会和规范知识。在五个基础模型和三个基准测试中，完整框架提升了15对模型-基准中的14对，并在8对中表现最佳或并列最佳，展示了类型化运行时支持的有效性。综合来看，这些结果表明，具有社会智能的大型语言模型需要在评估、参数内化和部署时间基础上实现协调进步。

Training Language Models to Cooperate with Inference-Time Controllers

训练语言模型以配合推理时间控制器

• Authors: Moumita Choudhury, Vanshaj Khattar, Jing Liu, Toshiaki Koike-Akino, Ankush Chakrabarty, Shlomo Zilberstein, Ye Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23771
• Pdf link: https://arxiv.org/pdf/2607.23771
• Abstract Large language model (LLM) performance increasingly depends not only on the base model, but also on the inference-time controller used to organize reasoning. Existing post-training methods, however, typically optimize for a single fixed interaction pattern, despite real deployments relying on diverse controllers such as Chain-of-Thought, self-consistency, debate, planning, and verification pipelines. This creates a training--deployment mismatch and limits transfer to new workflows. We introduce CALM (Controller-Aware Language Models), a post-training framework that explicitly places controllers in the training loop. We formulate controller-aware post-training as multi-task reinforcement learning over controller-induced interaction protocols, where controllers are compositions of reusable local reasoning modules. This structure also induces a module-level decomposition of mixed-controller training under a turn-level GRPO objective, enabling a systematic study of controller and module-aware training strategies. We evaluate CALM on held-out controller compositions and broader controller shifts, showing that controller-aware post-training improves generalization across inference-time workflows beyond single-controller optimization.
• 中文摘要 大型语言模型（LLM）的性能越来越依赖于基础模型，还依赖于用于组织推理的推理时间控制器。然而，现有的后训练方法通常优化单一固定交互模式，尽管实际部署依赖于多种控制器，如思维链、自洽性、辩论、规划和验证流程。这造成了训练与部署的不匹配，限制了向新工作流程的迁移。我们引入了CALM（控制器感知语言模型），这是一个明确将控制器置于训练循环中的后训练框架。我们将控制器感知后的训练表述为多任务强化学习，基于控制器诱导的交互协议，其中控制器是可重用的局部推理模块组合。该结构还在回合级GRPO目标下诱导了模块层级的混合控制器训练分解，从而实现对控制器和模块感知训练策略的系统研究。我们评估了CALM在外显控制器组成和更广泛控制器位移上的应用，表明控制器感知的后训练提升了推理时间工作流程的泛化性，超越了单控制器优化。

$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

$N_0$-VTLA：带潜在触觉令牌的视野-触觉语言-动作模型缩放

• Authors: NeoteAI Team, Fudan TEAI Team
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.23782
• Pdf link: https://arxiv.org/pdf/2607.23782
• Abstract We present $N_0$-VTLA, a vision-tactile-language-action (VTLA) foundation model capable of (1) fine-grained contact-rich manipulation with tactile perception and tactile-feedback control, and (2) offline policy improvement from stored deployment data. Building on current vision-based backbones, we propose a training recipe for tactile integration consisting of visuo-tactile pre-training, staged tactile-pathway integration, and advantage-conditioned offline policy improvement. During pre-training, the policy learns broad contact priors from NeoData, our large-scale visuo-tactile robot dataset; to our knowledge, $N_0$-VTLA is the first VTLA model pretrained on tactile data at scale. During post-training, we augment the policy with a predictive tactile pathway that distills the contact patterns learned at scale into the fine motion adjustments required by downstream tactile-centric manipulation. For offline policy improvement, we introduce ALTER, an advantage-conditioned offline reinforcement learning method that converts relative progress and trajectory-event comparisons into binary advantage labels for policy training on a fixed deployment corpus, further improving task-specific learning on contact-rich skills such as deformable object manipulation. Across contact-rich benchmarks, $N_0$-VTLA outperforms strong baselines by wide margins: it wins all nine real-robot NeoReal tasks and reaches 63.8% mean success on a twenty-task simulation suite, against 44.0% for the strongest baseline. $N_0$-VTLA policies trained with ALTER reach 75-95% success on three long-horizon real-robot tasks. These results lay a foundation for versatile tactile-driven manipulation policies.
• 中文摘要 我们呈现$N_0$-VTLA，一种视觉-触觉-语言-行动（VTLA）基础模型，具备（1）细粒度的丰富接触操作，具备触觉感知和触觉反馈控制，以及（2）基于存储部署数据的离线策略改进。基于现有基于视觉的骨干，我们提出了一种触觉集成训练配方，包括视觉-触觉预训练、分阶段触觉-路径集成和优势条件离线策略改进。在预训练期间，策略从NeoData——我们的大规模视觉-触觉机器人数据集中学习广泛的接触先验;据我们所知，$N_0$-VTLA是首个大规模预训练的触觉数据VTLA模型。在后期训练阶段，我们通过预测触觉通路补充策略，将大规模学习的接触模式提炼为下游触觉中心操作所需的细致动作调整。为提升离线策略，我们引入了ALTER，这是一种优势条件离线强化学习方法，将相对进度和轨迹-事件比较转换为二元优势标签，用于固定部署语料库上的策略训练，进一步提升了对丰富接触技能（如可变形对象操作）的任务专属学习。在接触丰富基准测试中，$N_0$-VTLA在强基线上表现显著：它赢得了全部九个真实机器人NeoReal任务，并在20个任务模拟套件中平均成功率达到63.8%，而最强基线为44.0%。$N_0$-VTLA策略在三个长期真人机器人任务中成功率为75-95%。这些结果为多功能触觉驱动操作策略奠定了基础。

GNN-based Multi-Agent Control of Traffic Shockwaves in Sparse Vehicular Ad-hoc Networks

基于GNN的多代理控制稀疏车辆自组网络中的流量冲击波

• Authors: Prachi Nandi, Madhuri Malakar, Sonakshi Satpathy, Pabitra Mohan Khilar
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG); Numerical Analysis (math.NA)
• Arxiv link: https://arxiv.org/abs/2607.23792
• Pdf link: https://arxiv.org/pdf/2607.23792
• Abstract Traffic shockwaves are stop-and-go waves that propagate upstream through the streams of vehicles and are one of the major causes of traffic congestion, fuel inefficiency, and increased accident rates in modern transportation systems. Although Connected and Autonomous Vehicles (CAVs) offer a promising opportunity to mitigate such shockwaves, most existing control strategies rely on global traffic state information, making them impractical for early-stage deployment of Vehicular Ad-hoc Networks (VANETs). In this paper, we propose a decentralized Multi-Agent Reinforcement Learning (MARL) framework that integrates a Graph Neural Network (GNN) to enhance the control architecture of connected and autonomous vehicles. The proposed approach enables vehicles to learn cooperative control policies using locally available information and interaction with neighboring vehicles. The effectiveness of the proposed scheme is evaluated using a scalable simulation environment under realistic highway traffic conditions. Simulation results show that the proposed GNN-based MARL framework can reduce the propagation of traffic shockwaves by up to 80\%, even when only 10\% of the vehicles are connected.
• 中文摘要 交通冲击波是通过车辆流向上游传播的停走波，是现代交通系统中交通拥堵、燃油效率低下和事故率上升的主要原因之一。尽管互联和自动驾驶车辆（CAVs）为缓解此类冲击波提供了有前景的机会，但大多数现有控制策略依赖于全局交通状态信息，因此在车辆自组网（VANETs）的早期部署中不切实际。本文提出了一个去中心化的多智能体强化学习（MARL）框架，集成图神经网络（GNN），以增强互联和自动驾驶车辆的控制架构。该方法使车辆能够利用本地可用信息和与邻近车辆的交互学习协同控制策略。该方案的有效性通过可扩展的模拟环境在真实的高速公路交通条件下进行评估。模拟结果表明，基于GNN的MARL框架即使仅连接10%车辆，也能将交通冲击波传播减少多达80%。

PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis

PathScale-R1：病理图像分析的跨尺度推理

• Authors: Chi Phan, Tianyi Zhang, Yufeng Wu, Qiaochu Xue, Jiajie Zhang, Linghan Cai, Zeyu Liu, Sudong Wang, Yueming Jin, Dan Hu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23794
• Pdf link: https://arxiv.org/pdf/2607.23794
• Abstract Pathological diagnosis is inherently multi-scale, requiring the integration of global tissue architecture at low magnification with cellular morphology at higher magnification. However, existing pathology benchmarks and vision-language models (VLMs) are still largely developed under single-scale settings, limiting their ability to learn clinically meaningful multi-magnification reasoning. Moreover, naively constructed visual question answering (VQA) tasks may be susceptible to text-only or superficial visual shortcuts, leading to unreliable assessments of visual understanding. To address these limitations, we introduce a benchmark and training framework for shortcut-resistant cross-scale pathology reasoning. We design an Adversarial Text-only Screening strategy for semantic reasoning questions and a Structure-controlled Distractor Sampling strategy for visual grounding questions, encouraging models to rely on cross-scale visual evidence. Based on this pipeline, we construct PathScale-VQA, a high-quality cross-scale pathology VQA benchmark with 10,373 multiple-choice questions grounded in 1,368 diagnostic paths across multiple magnification levels. Building on the semantic reasoning set, PathScale-R1 is optimized through Difficulty-driven Reasoning Distillation supervised fine-tuning followed by reinforcement learning with a Scale-aware Reasoning Structure reward, which encourages the use of evidence across magnifications. Extensive experiments demonstrate state-of-the-art performance of PathScale-R1 on cross-scale reasoning tasks and effective transfer to conventional single-scale pathology VQA. Our code is available at this https URL.
• 中文摘要 病理诊断本质上是多尺度的，需要将低倍率下的全局组织结构与高倍率的细胞形态整合起来。然而，现有的病理基准和视觉语言模型（VLMs）仍大多在单一尺度环境中开发，限制了它们学习临床意义深远多倍推理的能力。此外，简单构建的视觉问答（VQA）任务可能容易受到仅文本或表面视觉捷径的影响，导致视觉理解评估不可靠。为解决这些局限性，我们引入了一个针对耐捷径跨尺度病理学推理的基准和训练框架。我们设计了针对语义推理问题的对抗性纯文本筛选策略和视觉基础问题的结构控制分心抽样策略，鼓励模型依赖跨尺度视觉证据。基于该流程，我们构建了PathScale-VQA，这是一个高质量的跨尺度病理学VQA基准，包含10,373道多项选择题，基于1,368条诊断路径，跨倍放大。基于语义推理集，PathScale-R1通过困难驱动推理蒸馏监督微调优化，随后通过强化学习和尺度感知推理结构奖励进行优化，鼓励跨倍数分析的证据应用。大量实验展示了PathScale-R1在跨尺度推理任务中的先进性能，并有效迁移到传统单一尺度病理VQA中。我们的代码可在此 https 网址获取。

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

从RLVR到RLSVR：任务转换为开放式LLM自我提升带来自我验证的奖励

• Authors: Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, Wentian Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23802
• Pdf link: https://arxiv.org/pdf/2607.23802
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verified. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference this http URL on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a multi-agent self-play environment inspired by Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）通过实现大规模优化，推动了推理导向大型语言模型（LLMs）的近期进展。然而，其适用性主要限于数学和编码等领域，这些领域正确性可以通过确定性验证。开放式任务通常依赖于人类偏好、奖励模型或基于LLM的评判者，这会引入评估偏差、评判能力瓶颈，并进一步推断。基于自监督学习原理，我们提出了基于任务转换的强化学习（RLSVR），这是一种基于任务转换的训练范式，用于将RLVR扩展到开放式任务。RLSVR将开放式任务转化为可验证的代理环境，其内部规则和交互结果自动生成奖励信号。我们用SpyRL实现RLSVR，这是一个多代理自玩环境，灵感来自《谁是间谍？》。代理接收非对称信息，完成同一目标任务，并投票识别指定的间谍。由于间谍身份是预先确定的，投票结果提供完全可验证的奖励，而成功识别仍与输出质量密切相关。文本摘要、创意写作和数学推理的实验表明，SpyRL在不可验证任务中优于现有自我提升方法，并在可验证推理任务上持续获得收益。这些结果表明，任务转换能够将基于RLVR的可扩展自我提升扩展到本可验证的领域之外。模型和代码已发布于此 https URL。

Understanding Human-like Solutions in Combinatorial Optimization via Learning and Search

通过学习与搜索理解组合优化中的类人解

• Authors: Haijiang Yan, Jian-Qiao Zhu, Liqiang Huang, Ming Meng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23854
• Pdf link: https://arxiv.org/pdf/2607.23854
• Abstract Humans often find good solutions to combinatorial optimization problems that are computationally hard even for advanced computer algorithms. In the Euclidean traveling salesman problems (TSP), people rapidly produce tours that are near-optimal, despite severe limits on time and computation. What makes a tour human-like, and how might such solutions be learned? Here we address these questions through a large-scale behavioral and computational investigation of human performance in Euclidean TSP. We sampled a broad space of TSP instances, collected human solutions, and compared them with neural policies based on Pointer Networks, which are recurrent neural networks with an attention-based pointing mechanism that define probability distributions over valid tours. We trained these networks under multiple objectives, including reinforcement learning (RL), supervised learning from optimal tours, supervised learning from human tours, and RL fine-tuning after optimal-supervised pretraining. Human tours were not identical to optimal tours, but occupied a near-optimal geometric basin: they shared many structural properties with optimal solutions while preserving systematic human-specific deviations. The best account of human tours was not direct imitation of optimal tours, but a model pretrained on optimal tours, fine-tuned by RL, and decoded through $\text{Best-of-}N$ sampling. These findings suggest that human-like solutions may emerge from a combination of structured supervised learning, RL, and test-time search, echoing computational principles underlying many modern artificial intelligence systems.
• 中文摘要 人类常常能找到对组合优化问题的良好解，这些问题即使对高级计算机算法来说计算难度极高。在欧几里得旅行推销员问题（TSP）中，人们尽管时间和计算受限，仍能迅速生成接近最优的旅行。是什么使旅行类人？这些解又如何被学习？这里我们通过对欧几里得TSP中人类表现的大规模行为和计算研究来探讨这些问题。我们采样了广泛的TSP实例，收集人类解，并将其与基于指针网络的神经策略进行比较，指针网络是具有注意力指向机制的循环神经网络，定义有效巡回的概率分布。我们根据多个目标训练这些网络，包括强化学习（RL）、从最优巡回中进行监督学习、从人类巡回中进行监督学习，以及在最优监督预训练后进行强化学习。人类旅行与最优路径并不完全相同，但占据了一个近乎最优的几何盆地：它们在保持系统性人类特异偏差的同时，具有许多结构性属性并具有最优解。人类旅行的最佳解释不是直接模仿最优行程，而是基于最优行程预训练的模型，经过强化学习微调，并通过$\text{最佳}N$采样解码。这些发现表明，类人类解可能源于结构化监督学习、强化学习和测试时搜索的结合，呼应许多现代人工智能系统背后的计算原理。

Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping

通过可行的动作映射桥接强化学习与最优控制

• Authors: Stefan Richter, Alberto Giammarino, Guillem Torrente, Sam Blakeman, Peter Dürr
• Subjects: Subjects: Systems and Control (eess.SY); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.23930
• Pdf link: https://arxiv.org/pdf/2607.23930
• Abstract Operating constrained dynamical systems requires controllers to efficiently solve complex tasks while enforcing recursive feasibility and safety constraints. To address these competing requirements, we present Feasible Action for Optimal Control (FAOC), a novel control framework integrating Reinforcement Learning (RL) and Optimal Control (OC). The key contribution is a computationally efficient, optimization-based mapping algorithm that transforms the RL agent's action from a static abstract set into a state-dependent feasible parameter set of the Optimal Control Problem (OCP), guaranteeing strict satisfaction of the dynamical system's constraints. Thus, FAOC effectively combines the predictable safety of OC with the flexibility of RL. In contrast to prior work, the abstract action space of the RL agent does not require expert or heuristic design, and the OCP formulation is not compromised by the inability of RL to guarantee feasibility. We apply our approach to real-time motion planning for robot table tennis, which encapsulates these challenges. Via simulated experiments, we show that FAOC outperforms state-of-the-art baselines in both sample efficiency and closed-loop performance.
• 中文摘要 运行受限动力系统要求控制器高效解决复杂任务，同时强制执行递归可行性和安全性约束。为应对这些相互竞争的需求，我们提出了可行的最优控制（FAOC），这是一种整合强化学习（RL）和最优控制（OC）的新颖控制框架。其关键贡献是一种计算高效的基于优化的映射算法，将强化学习代理的动作从静态抽象集转变为最优控制问题（OCP）中状态依赖的可行参数集，保证动力系统约束的严格满足。因此，FAOC有效地结合了OC的可预测安全性与强化学习的灵活性。与以往工作不同，强化学习代理的抽象作用空间不需要专家或启发式设计，OCP的表述也未因强化学习无法保证可行性而受到影响。我们将我们的方法应用于机器人乒乓球的实时运动规划，体现了这些挑战。通过模拟实验，我们证明FAOC在样品效率和闭环性能方面均优于最先进的基线。

EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff

EviBack：通过证据受限教师退让进行搜索-代理强化学习

• Authors: Xiao Ma, Zhiquan Hu, Yi Wei, Chenchen Zhao, Yijun Chen, Jicheng Zhao, Yuming Li Chuang Dai
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.23955
• Pdf link: https://arxiv.org/pdf/2607.23955
• Abstract Reinforcement learning enables Agentic RAG systems to learn multi-turn search from verifiable outcome rewards, but all- zero rollout groups provide no comparative signal and may hide useful search behavior. We present EviBack, an evidence- constrained Teacher backoff that supplies auxiliary super- vision to such groups while preserving verifiable Actor re- wards. It separates evidence assessment from answer refine- ment, preventing reference answers from overriding evidence- insufficiency judgments. A fully automated, end-to-end GPT- 5.5-assisted APE pipeline starts from a manually authored single-prompt dual-task Teacher, automatically partitions and labels rollout data, and performs ablation, task decomposition, evaluation, and selection to produce a gated two-stage Teacher. Compared with the manual design, the resulting Teacher im- proves downstream F1 and valid-answer rate while reduc- ing search, duplicate queries, and forced termination. Across seven open-domain QA benchmarks and three Qwen3 scales, EviBack improves F1 over Search-R1 and raises both single- and multi-hop macro F1. We guarantee that the code will be made publicly available at a later stage.
• 中文摘要 强化学习使智能RAG系统能够从可验证的结果奖励中学习多回合搜索，但所有零部署组不提供比较信号，可能隐藏有用的搜索行为。我们介绍EviBack，一种证据受限的教师备份，为此类组提供辅助监督，同时保持可验证的演员回报。它将证据评估与答案细化分离，防止参考答案覆盖证据不足判断。一个全自动化、端到端的GPT-5.5辅助APE流程从手动编写的单提示双任务教师出发，自动划分和标记推广数据，执行消融、任务分解、评估和选择，生成门禁的两阶段教师。与手动设计相比，最终的Teacher im-证明了下游F1和有效答案率，同时减少了搜索、重复查询和强制终止。在七个开放域QA基准和三个Qwen3尺度中，EviBack提升了F1相较于Search-R1，提升了单跳和多跳宏F1。我们保证代码将在后续阶段公开发布。

WARL: Wrench-Augmented Reinforcement Learning for Task-Agnostic Learning in Legged Robots

WARL：用于腿部机器人任务无关学习的扳手增强强化学习

• Authors: Keita Yoneda, Kento Kawaharazuka, Kei Okada
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.24036
• Pdf link: https://arxiv.org/pdf/2607.24036
• Abstract While reinforcement learning for legged robots has achieved high motor performance, it has been constrained by the limited exploration capability of actions confined to the joint space. To address this issue, this study proposes a new method, Wrench-Augmented Reinforcement Learning (WARL), which introduces a wrenche (force and torque) into the action space. The proposed method combines wrench-guided exploration with a success rate-based curriculum mechanism to expand exploration capabilities in the early stages of learning, with the ultimate goal of acquiring behaviors based solely on joint control. Experiments using a quadruped robot demonstrated that WARL can learn robustly across diverse terrains and motor tasks without requiring terrain-specific reward adjustments or complex curriculum designs. Furthermore, an ablation study verified the effectiveness of the Switching Curriculum, which gradually eliminates the wrench. On the other hand, we also show that introducing a wrench can encourage behaviors that do not sufficiently exploit the robot's physical embodiment. These findings suggest that while wrench-based exploration enhancement is effective for improving learning efficiency, designing it in a way that is consistent with the robot's physical structure is a critical future challenge.
• 中文摘要 虽然腿部机器人的强化学习已实现高运动表现，但受限于关节空间内动作探索能力的限制。为解决这一问题，本研究提出了一种新方法——扳手增强强化学习（WARL），将扳手（力和扭矩）引入动作空间。该方法结合扳手引导探索与基于成功率的课程机制，扩展学习早期阶段的探索能力，最终目标是仅凭关节控制获得行为。使用四足机器人的实验表明，WARL能够在不同地形和运动任务中进行强健学习，无需地形特定奖励调整或复杂课程设计。此外，消融研究验证了切换课程的有效性，该方法逐步消除扳手。另一方面，我们也表明，引入扳手可能鼓励那些未能充分发挥机器人物理本体的行为。这些发现表明，虽然基于扳手的探索增强对提升学习效率有效，但设计使其与机器人物理结构一致仍是一个关键的未来挑战。

Constrained Reinforcement Learning Using Successor Representations

使用继任表示的约束强化学习

• Authors: Michael Girstl, Alexander Mattick, Christopher Mutschler
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.24057
• Pdf link: https://arxiv.org/pdf/2607.24057
• Abstract Real-world Reinforcement Learning depends on the ability to formulate safety constraints into a policy. A common way to model such constraints is to introduce an additional cost signal in the Markov Decision Process, which notifies the agent of unwanted behavior independently of the reward signal. Unfortunately, current methods are hard to adapt to changes in the cost function introduced by, e.g., domain shift or obstacles moving over time. The lack of adaptability means that policies are too unflexible to deal with complex real-world conditions. We propose the Safe Deep Successor Representation (SafeDSR), a novel method that allows quick retraining of policies towards new cost structures. SafeDSR extends the Deep Successor Representation (Kulkarni et al., 2016) to Constrained Reinforcement Learning by introducing a single learnable weight matrix to decouple the learned value function across dynamics, rewards, and costs. This matrix can be updated in a supervised manner instead of having to adapt the whole network if the cost structure of the environment changes. We demonstrate this ability in a freely configurable two-dimensional navigation environment and show that our method is competitive on a simple navigation task while being considerably more flexible
• 中文摘要 现实世界的强化学习依赖于将安全约束构建策略的能力。建模此类约束的常见方法是在马尔可夫决策过程中引入额外的成本信号，该信号独立于奖励信号通知代理人不希望的行为。遗憾的是，当前方法难以适应如领域转移或障碍随时间移动带来的成本函数变化。缺乏适应性意味着策略过于灵活，难以应对复杂的现实环境。我们提出了安全深度继承表示（SafeDSR），这是一种新颖方法，允许策略快速重新训练以适应新的成本结构。SafeDSR通过引入单一可学习权重矩阵，将深度继承表示扩展到受限强化学习，跨动力学、奖励和成本进行解耦。该矩阵可以在监督下更新，无需在环境成本结构变化时调整整个网络。我们在可自由配置的二维导航环境中展示了这一能力，并证明我们的方法在简单导航任务中具有竞争力，同时更灵活

ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning

ACRL：稳定强化学习中训练-推断差异的自适应控制

• Authors: Wenwu Fan, Qihong Lin, Zhijie Xia, Zhuo Zheng, Sihao Wang, Qiang Chen, Liangsheng Zhu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.24062
• Pdf link: https://arxiv.org/pdf/2607.24062
• Abstract Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.
• 中文摘要 大型语言模型（LLM）的强化学习（RL）训练常因训练与推理之间的差异而存在不稳定性。这种训练-推理差异主要源于两个因素：训练引擎与推理引擎之间的架构分离，以及推理中使用低精度量化而非高精度计算。为解决由高训练-推理差异引起的训练不稳定性问题，我们提出了其自适应控制的原理和方法。我们提出了自适应控制强化学习（ACRL），该方法自适应地将训练-推理差异维持在合理范围内，以确保强化学习训练的稳定。除了稳定性外，ACRL本质上还会增加策略熵，从而增强探索和精度。实验结果表明，当推理引擎采用FP8量化时，ACRL能够持续将训练-推断差异保持在合理范围内，并稳定强化学习训练。此外，ACRL不仅能匹配BF16基线的准确性，还优于重要性抽样（IS）定位。

Learning Reusable Hybrid Motion Priors for Humanoid Locomotion from Motion Imitation

学习可重复使用的混合运动先验，用于模拟运动中的类人移动

• Authors: Valerio Belli (UNIROMA, UCL), Valerio Modugno (UCL), Enrico Mingo Hoffman (HUCEBOT), Fabio Amadio (HUCEBOT)
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.24083
• Pdf link: https://arxiv.org/pdf/2607.24083
• Abstract Reinforcement learning can produce robust humanoid controllers, but each new task is typically trained as a separate policy with its own reward design and training process. Motion imitation provides an alternative source of motor competence by training policies to track retargeted human motions, yet the resulting controllers remain reference trackers and are not directly usable as task policies. We propose a three-stage pipeline that turns motion-imitation skills into a reusable hybrid motion prior (HMP) for humanoid locomotion. First, an expert policy is trained to imitate retargeted human motion-capture clips. Second, the expert is distilled into a frozen architecture composed of a proprioceptive encoder, a residual vector-quantized (RVQ) codebook, and an action decoder. Third, task-level policies are trained to solve locomotion tasks by selecting discrete codebook entries while the HMP remains frozen. We evaluate the method on velocity tracking, point-goal navigation, and fall-recovery velocity tracking in simulation, and deploy the velocity-tracking policy on a real Unitree G1 robot. The distillation process preserves the tracking behavior of the expert, while the resulting HMP can be reused without retraining as the action interface for different downstream locomotion policies. The learned HMP reveals an interpretable codebook structure in which the number of active RVQ stages modulates the available gait patterns. We further show that training the codebook with the rotation trick improves latent organization and reduces downstream falls compared with a standard straight-through estimator.
• 中文摘要 强化学习可以生成强大的类人型控制器，但每个新任务通常作为独立策略进行训练，拥有其独立的奖励设计和训练流程。动作模仿通过训练策略追踪重新定向的人类动作，提供了另一种运动能力来源，但最终产生的控制器仍是参考跟踪器，不能直接用作任务策略。我们提出了一个三阶段流程，将动作模仿技能转化为可重复使用的混合运动先验（HMP）用于人形运动。首先，专家策略被训练以模拟重新定向的人类动作捕捉片段。其次，专家被提炼成由本体感觉编码器、残余矢量化（RVQ）码本和动作解码器组成的冻结架构。第三，任务级策略通过在HMP冻结时选择离散码本条目来解决运动任务。我们评估了该方法在仿真中的速度追踪、点目标导航和坠落恢复速度追踪，并将速度追踪策略部署在真实的Unitree G1机器人上。蒸馏过程保留了专家的跟踪行为，而所得的HMP可不需重新训练即可重复使用，作为不同下游移动策略的动作接口。所学HMP揭示了一个可解读的代码手册结构，其中活跃RVQ阶段的数量调节可用的步态模式。我们还进一步证明，使用旋转技巧训练代码手册相比标准直通估计器能改善潜在组织并减少下游坠落。

MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agents

MemChain：为内存增强的LLM代理学习可解释的记忆痕迹

• Authors: Yiwen Ma, Songjun Tu, Qichao Zhang, Dong Li, Linjing Li, Dongbin Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.24097
• Pdf link: https://arxiv.org/pdf/2607.24097
• Abstract Memory-augmented LLM agents typically answer queries by retrieving relevant memories and feeding them directly to an answer model. This retrieval-as-evidence paradigm assumes retrieved memories are already suitable for reasoning, leaving the answer model to resolve redundancy, conflicts, and weak relevance while incurring substantial context overhead in long-term memory tasks. We propose MemChain, a trainable post-retrieval memory policy that transforms retrieved candidates into answer-facing active memory, represented as a compact and grounded evidence context. Given a user query and retrieved candidates, MemChain first generates a question-conditioned evidence plan, then constructs an ordered grounded evidence trace that organizes retrieved memories according to their semantic roles and dependencies, and finally executes explicit memory actions to produce a concise evidence context for answer generation. To train the mediator, we introduce a two-stage learning framework. Supervised trace learning first teaches the policy to generate structurally valid plans, traces, actions, and evidence contexts. We then propose Trace-Guided Memory Policy Optimization (TMPO), a reinforcement learning objective that optimizes the memory policy using downstream answer quality while jointly encouraging trace grounding, evidence support, structural validity, and answer stability across multiple rollouts. Experiments on LoCoMo and LongMemEval-S demonstrate that MemChain consistently achieves state-of-the-art performance across both closed-source and open-weight frozen answer models while substantially reducing the memory context passed to the answer model.
• 中文摘要 记忆增强的LLM代理通常通过检索相关记忆并直接输入答案模型来回答查询。这种检索即证据范式假设检索到的记忆已适合推理，因此答案模型需要解决冗余、冲突和相关性不足，同时在长期记忆任务中承担大量上下文开销。我们提出了MemChain，一种可训练的检索后记忆策略，将检索的候选记忆转化为面向答案的主动记忆，表现为紧凑且扎根的证据上下文。给定用户查询和检索候选者，MemChain首先生成一个问题条件证据计划，然后构建有序的基础证据追踪，根据其语义角色和依赖组织检索的记忆，最后执行显式记忆动作，生成简洁的证据上下文以生成答案。为了训练中介，我们引入了一个两阶段学习框架。监督式追踪学习首先教策略生成结构有效的计划、追踪、动作和证据上下文。随后，我们提出了跟踪引导记忆策略优化（TMPO），这是一种强化学习目标，利用下游答案质量优化记忆策略，同时促进跟踪基础化、证据支持、结构性效度和多次部署中的答案稳定性。LoCoMo和LongMemEval-S的实验表明，MemChain在闭源和开权定冻答案模型中始终保持最先进性能，同时大幅减少传递给答案模型的内存上下文。

Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding

监管推理：交通规则理解的思路链

• Authors: Yueru Luo, Xu Yan, Changqing Zhou, Yiming Yang, Chao Zhan, Shuqi Mei, Chao Zheng, Zhen Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.24199
• Pdf link: https://arxiv.org/pdf/2607.24199
• Abstract Understanding and complying with traffic regulations is a safety-critical requirement for autonomous driving, yet remains challenging due to the diversity and context dependence of traffic signage. Importantly, regulation understanding is not a simple recognition task, but a reasoning problem: whether a rule applies depends on interpreting the sign in relation to the spatial layout of lanes and scene context. To support such reasoning, MapDR provide fine-grained annotations that link each traffic sign's regulatory rules to the specific lanes they govern. Existing methods, however, largely treat this as direct sequence prediction, ignoring the underlying reasoning that connects sign semantics and map structure. To address this limitation, we explicitly incorporate reasoning into this task and propose a framework that equips vision-language models (VLMs) with chain-of-thought (CoT) capabilities. We first design a scalable CoT curation pipeline that bootstraps rationales from a strong LLM through a two-round strategy and employs a VLM-based verifier to filter out incorrect cases, yielding a high-quality set of (CoT, answer) pairs. Building on this foundation, we adopt a two-stage training scheme: supervised fine-tuning (SFT) to teach rationale-to-answer generation, followed by GRPO reinforcement learning with answer-grounded, fine-grained rewards to further improve final answer accuracy. Extensive experiments on MapDR show that our approach significantly improves both interpretability and accuracy, establishing the first reasoning-based framework for regulation-aware autonomous driving.
• 中文摘要 理解和遵守交通法规是自动驾驶的安全关键要求，但由于交通标志的多样性和上下文依赖性，这一要求依然具有挑战性。重要的是，理解法规并非简单的识别任务，而是一个推理问题：规则是否适用取决于如何解释标志与车道空间布局和场景上下文的关系。为支持这种推理，MapDR提供了细粒度注释，将每个交通标志的监管规则与其具体车道连接起来。然而，现有方法大多将其视为直接序列预测，忽视了连接标志语义和地图结构的背后推理。为解决这一限制，我们明确将推理纳入任务，并提出了一个框架，使视觉语言模型（VLMs）具备思维链（Chain of-Thought，CoT）能力。我们首先设计了一个可扩展的CoT策展流程，通过两轮策略从强LLM中引导推理，并采用基于VLM的验证器过滤错误案例，生成高质量的（CoT，答案）对。在此基础上，我们采用了两阶段训练方案：监督式微调（SFT）教授从理据到答案生成，随后是基于答案的GRPO强化学习，提供基于答案的细粒度奖励，进一步提升最终答案准确性。在MapDR上的大量实验表明，我们的方法显著提升了可解释性和准确性，建立了首个基于推理的监管意识自动驾驶框架。

From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Search

从专有到开源：通过多智能体协议提炼弥合代理搜索中的分发差距

• Authors: Junlin Liu, Jiangwang Chen, Zixin Song, Shuaiyu Zhou, Chunji Lv, Hank Wu, Kailin Jiang, Jinyang Wu, Bohan Yu, Chenxi Zhou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.24280
• Pdf link: https://arxiv.org/pdf/2607.24280
• Abstract Agentic search enables large language models to solve knowledge-intensive tasks by interleaving multi-step reasoning with retrieval, yet optimizing this with outcome-based reinforcement learning (RL) provides only sparse supervision. Knowledge distillation can supply denser guidance, and advanced proprietary models with their strong reasoning capabilities are promising teachers. While distilling from proprietary models can densify this supervisory signal, conventional logit-matching is precluded by hidden logits and mismatched tokenizers, whereas raw natural language trajectory imitation transfers superficial stylistic artifacts rather than core reasoning competence. To address the heterogeneous distillation problem and bridge the distribution gap, we propose Multi-Agent Protocol Distillation (MAPD), a joint distillation and RL framework uses a structured, style-normalized protocol as an intermediate representation. An offline multi-agent system (MAS) decomposes each query, retrieves supporting evidence, repairs failed searches, and converts the resulting exploration trace into a JSON protocol containing the task type, reasoning plan, and extractive grounding facts. During training, the protocol is provided only to a privileged branch of the student policy, whose token distributions furnish a dense distillation signal alongside the sparse RL objective. Extensive evaluations across seven QA benchmarks demonstrate that MAPD consistently outperforms competitive distillation and RL, achieving average success rates of 39.4\% on Qwen3-1.7B and 44.4\% on Qwen3-4B. Crucially, the framework generalizes robustly across diverse proprietary teachers while effectively mitigating the student policy from style drift and verbosity degeneration.
• 中文摘要 智能搜索使大型语言模型能够通过多步推理与检索来解决知识密集型任务，但通过基于结果的强化学习（RL）优化仅提供稀疏的监督。知识蒸馏可以提供更密集的指导，先进的专有模型凭借其强大的推理能力，是有前景的教师。虽然从专有模型提炼可以使监督信号更密，但传统logit匹配因隐藏logit和不匹配分词器而被阻碍，而原始的自然语言轨迹模仿则转移的是表面的风格伪影，而非核心推理能力。为解决异构蒸馏问题并弥合分布差距，我们提出了多智能体协议蒸馏（Multi-Agent Protocol Distillation，MAPD）这一联合蒸馏和强化学习框架，使用结构化、样式规范化协议作为中间表示。离线多智能体系统（MAS）分解每个查询，检索支持证据，修复失败的搜索，并将由此产生的探索痕迹转换为包含任务类型、推理计划和提取基础事实的JSON协议。在培训期间，该协议仅提供给学生政策的特权分支，其令牌分发在稀疏的强化学习目标外，提供密集的提炼信号。对七个QA基准的广泛评估表明，MAPD始终优于竞争性提炼和强化学习，Qwen3-1.7B平均成功率为39.4%，Qwen3-4B为44.4%。关键是，该框架在不同专有教师之间能够强健推广，同时有效缓解学生政策对风格漂移和冗长退化的影响。

Learning Adaptive Multi-Task Guidance, Navigation, and Control via Hypernetworks

通过超网络学习自适应多任务引导、导航和控制

• Authors: Ricard Marsal I Castan, Aman Arora, Antoine Richard, Andrej Orsula, Cédric Pradalier, Miguel A. Olivares-Méndez
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.24292
• Pdf link: https://arxiv.org/pdf/2607.24292
• Abstract Autonomous free-flying robots in orbital environments require controllers that are both versatile and resource-efficient, yet maintaining a separate, task-specific policy for each mission profile is architecturally brittle and limits operational flexibility as requirements evolve. We introduce HYPER-GNC, a multi-task reinforcement learning framework in which a hypernetwork maps physics-informed task embeddings to the weights of a shared actor-critic policy, enabling a single compact controller to master four distinct GNC tasks: velocity tracking, docking, inspection, and navigation with obstacle avoidance. The continuous embedding space allows the controller to generalize to novel mission configurations at deployment time without any retraining. Extensive experiments demonstrate that HYPER-GNC achieves sample efficiency comparable to single-task specialists while maintaining stability under significant inertial perturbations and external body wrenches. We further validate the framework on a physical satellite emulator, successfully bridging the simulation-to-reality gap across all mission profiles. Code, trained models, and deployment scripts are made publicly available to support reproducibility.
• 中文摘要 轨道环境中的自主自由飞行机器人需要既多功能又资源高效的控制器，但为每个任务配置文件维护独立的任务特定策略在架构上较为脆弱，限制了需求演变的操作灵活性。我们引入了HYPER-GNC，一种多任务强化学习框架，其中超网络将物理知情的任务嵌入映射到共享的行为者-批判策略权重，使单个紧凑控制器能够掌握四项不同的GNC任务：速度跟踪、对接、检查和带障碍物规避的导航。连续嵌入空间使控制器能够在部署时推广到新颖的任务配置，无需重新训练。大量实验表明，HYPER-GNC在显著惯性扰动和外部体力扳手下保持稳定的样本效率可与单任务专家相当。我们还在物理卫星模拟器上验证了该框架，成功弥合了所有任务配置文件中仿真与现实的差距。代码、训练模型和部署脚本均公开，以支持可重复性。

CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models

CONSISTRE：一个统一一致性感知框架，用于大型语言模型的文档级关系提取

• Authors: Mingxuan Sun
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.24312
• Pdf link: https://arxiv.org/pdf/2607.24312
• Abstract Document-level relation extraction (DocRE) aims to extract relations among multiple entities across extended contexts while maintaining consistency across predicted triples. Although large language models (LLMs) show remarkable reasoning capabilities in information extraction, their predictions are typically generated independently for each candidate triple and may violate fundamental relational constraints such as transitivity, symmetry, and functional uniqueness, leading to contradictory and unreliable outputs. We propose CONSISTRE, a unified consistency-aware framework for DocRE that addresses this limitation through two complementary tracks. The first operates at inference time for black-box LLMs, combining constraint-aware prompting, constraint-based verification, and iterative self-reflection to refine predictions without task-specific fine-tuning. The second injects consistency knowledge into smaller open-source models via a knowledge distillation and reinforcement learning pipeline: reasoning traces from a powerful teacher are distilled into a student via supervised fine-tuning, followed by GRPO alignment using a composite reward that jointly optimizes extraction performance and relational consistency. Together, the two tracks cover both API-accessible and locally deployable scenarios under a unified consistency formulation. Experiments on DocRED show that both tracks outperform their baselines, with the inference-time track achieving competitive F1 using off-the-shelf black-box LLMs and the training-time track substantially narrowing the gap between 7--8B open-source models and state-of-the-art proprietary LLMs at a fraction of their inference cost. Ablation studies confirm that explicit consistency modeling mitigates relational contradictions and enhances the reliability of LLM-based DocRE across both deployment paradigms.
• 中文摘要 文档级关系提取（DocRE）旨在在扩展上下文中提取多个实体之间的关系，同时保持预测三元组的一致性。尽管大型语言模型（LLMs）在信息提取方面展现出卓越的推理能力，但它们的预测通常为每个候选三元组独立生成，可能违反传递性、对称性和功能唯一性等基本关系约束，导致输出矛盾且不可靠。我们提出了CONSISTRE，这是一个统一的一致性意识DocRE框架，通过两条互补路径解决这一限制。第一个在黑盒大型语言模型的推理阶段工作，结合了约束感知提示、基于约束的验证和迭代自我反思，以优化预测，无需任务特定微调。第二条通过知识蒸馏和强化学习流水线，将一致性知识注入较小的开源模型：通过监督微调，将强大教师的推理痕迹提炼到学生身上，随后通过复合奖励进行GRPO对齐，共同优化提取性能和关系一致性。这两条轨道共同涵盖API可访问和本地部署的场景，采用统一的一致性表述。DocRED上的实验显示，这两条路径均优于基线，推理时间轨道使用现成黑盒大型语言模型实现了竞争F1，训练时间轨道则以极低的推理成本大幅缩小了7-8B开源模型与最先进专有LLM之间的差距。消融研究证实，显式一致性建模能够减少关系矛盾，并提升基于LLM的DocRE在两种部署范式上的可靠性。

Continual-RL for Generalization in Autonomous Racing on the RoboRacer Platform

Continual-RL 用于 RoboRacer 平台上的自动竞速推广

• Authors: Joel Siegert, Edoardo Ghignone, Michele Magno
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.24320
• Pdf link: https://arxiv.org/pdf/2607.24320
• Abstract A key challenge in modern robotics is to adapt to changing environments, a challenge that is exacerbated when simulations cannot encompass every possible real-world configuration, and therefore Reinforcement Learning (RL) in the physical world becomes necessary. Continual Reinforcement Learning provides the tools to address this challenge; however, both the frameworks and the methods remain underexplored. Autonomous Racing and in particular the RoboRacer competition provide a testing ground for such methods, as learning to drive on a new track-floor combination with the least amount of new experience naturally frames a continual learning problem. This work tries to address this gap by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers. Furthermore, a comparison method based on offline RL is proposed, and a simulation analysis of the plasticity properties of the methods is conducted.
• 中文摘要 现代机器人学的一个关键挑战是适应不断变化的环境，当模拟无法涵盖所有可能的现实世界配置时，这一挑战更加复杂，因此物理世界中的强化学习（RL）变得必要。持续强化学习提供了应对这一挑战的工具;然而，无论是框架还是方法，都尚未被充分探索。自主竞速，尤其是RoboRacer竞赛，为这类方法提供了试验场，因为在新赛道与地面结合中学习驾驶，且新经验最少，自然构成了持续学习的问题。本研究试图通过提出基于持续反向传播的持续强化学习框架来弥补这一空白，该框架仅凭真实世界数据即可在一组赛道上训练通用策略，并在15分钟内对其进行微调，以超越经典控制器。此外，提出了基于离线强化学习的比较方法，并对这些方法的塑性特性进行模拟分析。

VulnGym: Evaluating Vulnerability Management Strategies against Advanced Persistent Threats

VulnGym：评估针对高级持续威胁的漏洞管理策略

• Authors: Sofia Della Penna, Lorenzo Parracino, Luciano Pianese, Vittorio Orbinato, Roberto Natella
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2607.24552
• Pdf link: https://arxiv.org/pdf/2607.24552
• Abstract Enterprise networks are continuously targeted by Advanced Persistent Threats (APTs), attack campaigns exploiting software vulnerabilities to compromise critical assets over time. As disclosed vulnerabilities grow, resource-constrained organizations must prioritize which ones to patch. Existing prioritization standards score vulnerabilities individually and cannot capture how a patching policy performs against an adversary that progresses through the network over time. Previous tools have simulated attack campaigns through Reinforcement Learning (RL), but either omit vulnerability management, leaving the attacker unopposed, or rely on synthetic networks disconnected from real threat data, and so cannot assess how a policy would fare against a realistic adversary. To fill this gap, we propose VulnGym, a simulation tool to evaluate vulnerability management policies. VulnGym simulates an RL-trained attacker, calibrated on real APT profiles, against a defender executing a configurable patching policy over a network with real Common Vulnerabilities and Exposures (CVEs). Both agents act on a shared, evolving network representation, so the attacker's progress is directly shaped by the defender's patching activity, allowing a given policy to be stress-tested against a realistic attack campaign. Experiments based on real-world vulnerabilities and two APTs show that vulnerability management must be tailored to organizational context, adversarial behavior, network topology, and asset criticality.
• 中文摘要 企业网络持续受到高级持续威胁（APT）攻击，这些攻击活动利用软件漏洞，逐步攻破关键资产。随着暴露漏洞的增加，资源有限的组织必须优先处理哪些漏洞。现有的优先级标准对漏洞进行单独评分，无法反映补丁策略在网络中逐步推进的攻击者时的表现。以往的工具通过强化学习（RL）模拟攻击活动，但要么省略漏洞管理，使攻击者无人反抗，要么依赖与真实威胁数据无关的合成网络，无法评估策略在现实对手面前的表现。为弥补这一空白，我们提出了VulnGym，一款用于评估漏洞管理策略的模拟工具。VulnGym 模拟一个基于真实 APT 配置文件校准的强化学习攻击者，对抗一个在具有真实公共漏洞与暴露（CVE）的网络上执行可配置补丁策略的防御者。两个代理都基于共享且不断演变的网络表示，因此攻击者的进展直接受防御者的补丁活动影响，使得某一策略能够在真实的攻击活动中进行压力测试。基于真实世界漏洞和两个 APT 的实验表明，漏洞管理必须根据组织上下文、对抗行为、网络拓扑和资产关键性进行定制。

Evaluating Fuzz Testing for Reinforcement Learning Agents

评估强化学习代理的模糊测试

• Authors: Zhibin Kang, Hanmo You, Dong Wang, Haiming Zheng, Junjie Chen
• Subjects: Subjects: Machine Learning (cs.LG); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.24577
• Pdf link: https://arxiv.org/pdf/2607.24577
• Abstract Reinforcement Learning (RL) agents are increasingly deployed in safety-critical domains such as robotics, autonomous driving, and drone control, where unexpected behaviors may lead to severe real-world consequences. Fuzz testing has recently emerged as a promising method for exploring the vast state spaces of RL agents and exposing crashes. Although numerous RL fuzzing methods have been proposed, existing studies often differ in evaluation settings, baselines, and metrics, making it difficult to draw reliable conclusions about their relative effectiveness and practical usefulness. To address this gap, we present the first comprehensive empirical study that systematically evaluates RL fuzzing methods from four complementary perspectives: effectiveness, diversity, efficiency, and practical utility. We benchmark five state-of-the-art methods alongside random testing under unified configurations across three environments of increasing complexity (MountainCar, BipedalWalker, and CARLA), and further assess the downstream usefulness of detected crashes for agent robustness improvement and safety monitoring. Our results reveal several key insights. For instance,throughput-oriented methods like MDPFuzz demonstrate superior effectiveness and efficiency in crash discovery, while methods explicitly designed to encourage exploration like SeqDivFuzz excel at uncovering diverse crash behaviors. We also show that fuzzing-generated crashes can meaningfully improve agent robustness and enable accurate safety monitoring with strong cross-method generalization. Beyond these empirical findings, we distill actionable guidance for both researchers and practitioners, highlighting the benefits of combining complementary fuzzing strategies and adopting multi-level diversity analysis to achieve more comprehensive and practical RL testing.
• 中文摘要 强化学习（RL）代理越来越多地应用于机器人、自动驾驶和无人机控制等安全关键领域，在这些领域，意外行为可能导致严重的现实后果。模糊测试最近成为探索强化学习代理庞大状态空间和揭示崩溃的有前景方法。尽管已有许多强化学习模糊检测方法被提出，但现有研究在评估环境、基线和指标上常存在差异，难以得出可靠结论，判断其相对有效性和实用性。为弥补这一空白，我们提出了首个综合实证研究，系统性地从有效性、多样性、效率和实用性四个互补视角评估强化学习模糊检测方法。我们对五种最先进方法及随机测试进行了基准测试，涵盖三个日益复杂化的环境（MountainCar、BipedalWalker和CARLA），并进一步评估检测到的碰撞在增强智能体鲁棒性和安全监控中的后续效用。我们的结果揭示了若干关键见解。例如，以吞吐量为导向的方法如MDPFuzz在碰撞发现中表现出优异的效果和效率，而专门鼓励探索的SeqDivFuzz方法则在发现多样化崩溃行为方面表现出色。我们还展示了模糊生成的碰撞能够显著提升代理的鲁棒性，并通过强有力的跨方法泛化实现准确的安全监测。除了这些实证发现外，我们还总结了对研究人员和实践者的可操作指导，强调结合互补模糊检测策略和采用多层次多样性分析以实现更全面、更实用的强化学习测试的好处。

Kimi K3: Open Frontier Intelligence

Kimi K3：开放边境情报

• Authors: Kimi Team: Tongtong Bai, Yifan Bai, Yiping Bao, M. C., Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y. Charles, H.S. Che, Guanduo Chen, Guangyu Chen, Guanzheng Chen, Huarong Chen, Jia Chen, Jianlong Chen, Jun Chen, Kexin Chen, Peng Chen, Ruijue Chen, Wentao Chen, Xin Chen, Yang Chen, Yanru Chen, Yifei Chen, Yingjiang Chen, Yuankun Chen, Yujie Chen, Yutian Chen, Zhirong Chen, Dazhi Cheng, Yean Cheng, Jialei Cui, Jingbing Cui, Anqi Dai, Jiaqi Deng, Hao Ding, Rui Ding, Shaofeng Ding, Mengfan Dong, Mengnan Dong, Yuhao Dong, Yuxin Dong, Angang Du, Chenzhuang Du, Dikang Du, Jusen Du, Yulun Du, Yu Fan, Jing Feng, Qiulin Feng, Yichen Feng, Kelin Fu, Qiang Fu, Fuxuan Gao, Hongcheng Gao, Jingyue Gao, Tong Gao, Weijia Gao, Shangyi Geng, Jie Gong, Linhu Gong, Shengao Gong, Xiaochen Gong, Qizheng Gu, Yicheng Gu, Shuhao Guan, Haiqing Guo, Shiqi Guo, Xiang Guo, Zhengyan Guo, Beixi Hao, Wenxin Hao, Xiaoru Hao, Dailan He, Haotian He, Lehan He, Qi He, Weiran He, Xinran He, Xinyi He, Yibo He, Yunjia He, Chao Hong, Tiange Hong, Hao Hu, Jiaxi Hu, Ruikun Hu, Weiming Hu, Yangyang Hu, Zhenxing Hu, Liang Hua, Jinbin Huang, Ke Huang, Ruiyuan Huang, Siying Huang, Weixiao Huang, Yan Huang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.24653
• Pdf link: https://arxiv.org/pdf/2607.24653
• Abstract We introduce Kimi K3, a 2.8T parameter Mixture-of-Experts model with 104 billion activated parameters, native vision capabilities, and a 1-million-token context window. Kimi K3 is built on Kimi Delta Attention and Attention Residuals, which improve information flow across sequence length and model depth. Together with Stable LatentMoE, which effectively activates 16 of 896 routed experts per token, and refined training and data recipes, these advances yield an approximately 2.5x improvement in overall scaling efficiency over Kimi K2. Post-training highlights reinforcement learning across general, agentic, and coding domains and multiple reasoning-effort levels, enabling compositional generalization and robust long-horizon execution. At 2.8T scale, Kimi K3 is supported by infrastructure advances in multiple areas: algorithm-system co-design for KDA, perfectly balanced expert-parallel training with efficient memory management, million-token agentic RL with persistent rollout and sandbox states, and deployment innovations. Extensive evaluations show that Kimi K3 achieves frontier-level performance across long-horizon coding, agentic, knowledge, reasoning, and vision tasks. While its overall performance still trails the most powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol, Kimi K3 consistently outperforms other open and proprietary models evaluated in our suite. We release the full Kimi K3 model weights to facilitate future research and accelerate the broader deployment and adoption of frontier intelligence.
• 中文摘要 我们引入了 Kimi K3，一个 2.8 吨参数的专家混合模型，拥有 1040 亿激活参数、原生视觉能力和 100 万令牌上下文窗口。Kimi K3 基于 Kimi Delta 注意力和注意力残差，改善了序列长度和模型深度中的信息流动。结合 Stable LatentMoE，每个标记有效激活 896 位路由专家中的 16 位，以及精炼的训练和数据配方，这些进步使整体扩展效率比 Kimi K2 提升约 2.5 倍。后期训练强调跨通用、代理和编码领域及多层推理努力的强化学习，支持组合泛化和稳健的长视野执行。在2.8T规模下，Kimi K3在多个领域的基础设施进步得到了支持：KDA的算法-系统协同设计、专家-并行训练与高效内存管理的完美平衡、百万令牌的代理强化学习（具持久展开和沙盒状态）以及部署创新。广泛评估显示，Kimi K3在长期编码、代理、知识、推理和视觉任务方面实现了前沿性能。虽然其整体性能仍落后于最强大的专有模型，即Claude Fable 5和GPT-5.6 Sol，但Kimi K3始终优于我们套件中评估的其他开放和专有模型。我们发布完整的Kimi K3模型权重，以促进未来研究并加速前沿智能的更广泛部署和采用。

Explainable Reinforcement Learning via Physics-Aware Policy Distillation

通过物理感知策略蒸馏的可解释强化学习

• Authors: Shaker Al-Tamari, Waled Kadour
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.24672
• Pdf link: https://arxiv.org/pdf/2607.24672
• Abstract In safety-critical sectors such as robotics and automotive engineering, the deployment of Deep Reinforcement Learning (DRL) is often hindered by the black-box nature of deep neural networks. This lack of transparency poses significant challenges for regulatory compliance and human-agent trust. This paper presents an experimental study aimed at making high-performance continuous control DRL systems interpretable. A policy distillation framework is implemented using the classic Inverted Pendulum benchmark. A high-performance Twin Delayed DDPG (TD3) agent serves as an opaque, continuous teacher model, whose policy is distilled into an interpretable student surrogate based on a shallow Decision Tree. By leveraging a custom physics-aware feature and "Noisy Oracle Rollouts" for dataset generation, the distillation process achieves performance equivalent to the expert teacher. Furthermore, comparative control theory analysis reveals a fundamental trade-off: transitioning from continuous to discrete rule-based control induces high-frequency Bang-Bang actuation and a stable bimodal limit cycle. Simulation results indicate that Bounded-Input Bounded-Output (BIBO) stability is maintained while providing both global and local interpretability for safe autonomous systems.
• 中文摘要 在机器人和汽车工程等安全关键领域，深度强化学习（DRL）的部署常常受到深度神经网络黑箱特性的阻碍。这种缺乏透明度对监管合规性和人-代理信任构成了重大挑战。本文提出了一项实验性研究，旨在使高性能连续控制DRL系统可被理解。采用经典的倒摆基准实现策略提炼框架。高性能的双延迟DDPG（TD3）代理作为不透明的连续教师模型，其策略被提炼成基于浅层决策树的可解释学生代理。通过利用自定义物理感知功能和“噪声神谕展开”进行数据集生成，提炼过程实现了与专家教师相当的性能。此外，比较控制理论分析揭示了一个根本性的权衡：从连续规则控制转向离散基于规则的控制，会诱导高频的砰砰驱动和稳定的双峰极限循环。模拟结果表明，在为安全自主系统提供全局和局部可解释性的同时，有界输入有界输出（BIBO）的稳定性得以保持。

Keyword: diffusion policy

PRISM: Polynomial Representations for Interaction-Structured Motor Control

棱镜：交互结构运动控制的多项式表示

• Authors: Seung Hyun Lee, Stella X. Yu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.23473
• Pdf link: https://arxiv.org/pdf/2607.23473
• Abstract Robot policies are typically MLPs mapping observations to actions. Yet robot observations are physical variables, and many action-relevant cues arise not from individual variables but from their interactions; power, inertial effects, contact, slip, and compliance depend on products among observable signals. We introduce PRISM, a policy representation that makes polynomial interactions among observable physical variables explicit, learnable, and compact. Rather than listing all polynomial terms, PRISM uses a factorized polynomial module to expose higher-order interaction features efficiently. In reinforcement learning, it keeps the standard MLP backbone but applies a gradually activated element-wise polynomial function after it. In imitation learning, it replaces linear proprioceptive conditioning in Diffusion Policy with a polynomial layer trained end-to-end. Across humanoid locomotion and contact-rich manipulation, PRISM improves performance over standard MLP policies and larger MLPs with matched capacity, showing that interaction structure cannot be replaced by capacity alone. It also yields sensorless compliant behavior without force, wrench, tactile input, contact labels, or admittance control. These results suggest that polynomial representations should become a standard architectural choice for embodied motor control. The project page is available at this https URL
• 中文摘要 机器人策略通常是将观测值映射为动作的MLP。然而，机器人的观测是物理变量，许多与动作相关的线索并非来自单个变量，而是来自它们的相互作用;功率、惯性效应、接触、滑移和顺应性依赖于可观测信号之间的乘积。我们引入了PRISM，这是一种策略表示，使可观测物理变量之间的多项式相互作用显式、可学习且紧凑。PRISM不列举所有多项式项，而是使用分解后的多项式模高效暴露高阶交互特征。在强化学习中，它保留标准MLP骨干，但随后应用一个逐步激活的元素级多项式函数。在模仿学习中，它用端到端训练的多项式层取代扩散策略中的线性本体感受条件。在类人机动和接触丰富操作中，PRISM提升了比标准MLP策略和更大MLP匹配容量的MLP的性能，表明交互结构无法仅靠容量来替代。它还能实现无传感器的顺应行为，无需力、扳手、触觉输入、接触标签或导纳控制。这些结果表明，多项式表示应成为具身运动控制的标准架构选择。项目页面可访问此 https URL