生成时间: 2026-07-14 18:02:05 (UTC+8); Arxiv 发布时间: 2026-07-14 20:00 EDT (2026-07-15 08:00 UTC+8)

今天共有 52 篇相关文章

Keyword: reinforcement learning

Depth-Entropy Guided Sampling for Training-Free LLM Reasoning

深度-熵引导抽样用于无训练的大型语言模型推理

• Authors: Zibin Meng, Peng Xie, Kani Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.09693
• Pdf link: https://arxiv.org/pdf/2607.09693
• Abstract Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals. Recent work shows that sampling from sharpened base-model distributions at test time recovers much of the RL gain, yet existing methods rely solely on output-layer likelihoods and ignore the transformer's internal forward-pass dynamics. We introduce Depth-Entropy Guided Sampling (DEGS), a training-free, test-time method that exploits layer-wise entropy collapse as an intrinsic quality signal. We observe that stronger reasoners -- including RL-posttrained variants -- exhibit a distinctive "late collapse": logit-lens decoded entropy stays elevated until deeper layers before converging. We define a per-sequence collapse depth $D(\mathbf{x})$ and a joint objective $\pi(\mathbf{x}) \propto p(\mathbf{x})^\alpha \exp(\beta D(\mathbf{x}))$ that combines sequence likelihood with this depth-entropy structure, instantiated inside an MCMC power-sampling framework (DEGS-MCMC). Across three open-weight models and four reasoning benchmarks, this near-chance per-candidate signal compounds over the sampling trajectory into state-of-the-art training-free accuracy, with gains largest out of domain and on the harder splits -- exactly where likelihood alone falls short -- at single-digit-percent wall-clock overhead. DEGS narrowly trails an in-house GRPO reference on the math splits GRPO was trained for, yet surpasses it out of domain on GPQA for all three models, without any training, reward model, or labeled data.
• 中文摘要 强化学习（RL）已成为提升大型语言模型推理能力的主流范式，但它需要昂贵的训练、精心策划的数据和奖励信号。最新研究表明，在测试时从锐化基模型分布中采样可以恢复大部分强化逻辑增益，但现有方法仅依赖输出层似然，忽视变换器内部的前通动态。我们介绍深度-熵引导采样（DEGS），这是一种无需训练、测试时间限制的方法，利用层层熵坍缩作为内在质量信号。我们观察到，更强的推理者——包括强化学习后训练的变体——表现出一种独特的“晚坍缩”：logit透镜解码后的熵会保持在较高的层次中，直到更深层才收敛。我们定义了一个每序列坍缩深度$D（\mathbf{x}）$，以及一个联合目标$\pi（\mathbf{x}） \propto p（\mathbf{x}）^\alpha \exp（\beta D（\mathbf{x}）））$，结合序列似然度与MCMC幂采样框架（DEGS-MCMC）内实现的深度-熵结构。在三个开放权重模型和四个推理基准中，这一近乎每个候选人机会的信号在抽样轨迹中累积，达到最先进的无训练准确性，在域外和更难的分组中——即概率不足的地方——增幅最大，时钟开销仅为个位数百分比。DEGS在GRPO训练时的数学分配方面略逊于内部GRPO参考，但在GPQA领域外却超越了这三个模型，且没有任何训练、奖励模型或标记数据。

A Risk-Field Enhanced Closed-Loop Digital Twin Framework for Autonomous Driving Safety Validation

风险场增强型闭环数字孪生框架用于自动驾驶安全验证

• Authors: Yongzhi Liu
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.09772
• Pdf link: https://arxiv.org/pdf/2607.09772
• Abstract Autonomous driving systems require reliable safety validation before real-world deployment. However, large-scale road testing is costly, difffcult to reproduce, and inefffcient for exposing rare safety-critical scenarios. Conventional simulation improves repeatability, but an offfine simulator alone cannot continuously connect physical trafffc states, virtual reconstruction, algorithm evaluation, and scenario evolution. This paper proposes a risk-ffeld enhanced closed-loop digital twin framework for autonomous driving safety validation. The framework integrates physical data acquisition, data synchronization, virtual twin reconstruction, risk-aware scenario generation, autonomous driving algorithm evaluation, and safety analysis. A driving risk ffeld is introduced as a uniffed intermediate representation to describe obstacle, lane-departure, road-boundary, time-to-collision, and comfort-related risks around the ego vehicle. The risk ffeld ranks high-risk scenarios in the digital twin scenario library and provides dense safety guidance for reinforcement learning-based driving policies. A simulation-style evaluation protocol is designed to compare conventional reinforcement learning baselines, risk-penalty baselines, and the proposed risk-ffeld guided method. The study indicates that embedding explicit risk structure into digital twins can make autonomous driving validation more targeted, interpretable, and reusable, while its practical effectiveness remains bounded by model ffdelity, risk calibration, and sim-to-real transfer.
• 中文摘要 自动驾驶系统在实际部署前需要可靠的安全验证。然而，大规模路考成本高昂，难以复现，且在暴露罕见的安全关键场景时效率低下。传统仿真提高了重复性，但仅靠微型模拟器无法连续连接物理trafffc状态、虚拟重建、算法评估和场景演化。本文提出了一种风险-弗费尔德增强型闭环数字孪生框架，用于自动驾驶安全验证。该框架集成了物理数据采集、数据同步、虚拟双胞胎重建、风险感知场景生成、自动驾驶算法评估和安全分析。驾驶风险ffeld作为未被考虑的中间表征引入，用以描述障碍物、车道偏离、道路边界、碰撞时间以及自我车辆周围的舒适性相关风险。Risk ffeld在数字孪生情景库中对高风险场景进行了排名，并为基于强化学习的驾驶政策提供了密集的安全指导。设计了一种模拟式评估协议，比较传统强化学习基线、风险惩罚基线和拟议的风险-费尔德引导方法。研究表明，将显性的风险结构嵌入数字孪生可以使自动驾驶验证更具针对性、可解释性和可重复使用性，而其实际效果仍受限于模型忠实度、风险校准和模拟到现实的转移。

EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

EvoCUA-1.5：多回合计算机代理的在线强化学习

• Authors: Mianqiu Huang, Taofeng Xue, Chong Peng, Jinrui Ding, Sicheng Fan, Jiale Hong, Yufei Gao, Xiaocheng Zhang, Linsen Guo, Xin Yang, Dengchang Zhao, Yuchen Xie, Peng Pei, Xunliang Xie, Xipeng Qiu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.09773
• Pdf link: https://arxiv.org/pdf/2607.09773
• Abstract Computer-use agents must solve long-horizon tasks through repeated interaction with partially observable, multimodal desktop environments. Although imitation learning and offline trajectory refinement provide strong priors, static traces cannot cover the causal feedback loop of real computer use: each action changes the screen state, future action space, and recovery options. EvoCUA-1.5 extends self-evolving computer-use agents from offline experience learning to online reinforcement learning, where policies interact with executable sandbox environments and improve from verifiable task outcomes. Online RL in this setting requires more than directly reusing single-turn language-RL recipes. Multi-turn interaction introduces context-managed observations, sparse terminal rewards, variable-length trajectories, and slow environment feedback. EvoCUA-1.5 addresses these challenges with Step-Level Policy Optimization (STEPO), which preserves trajectory-level advantage balance after decomposition into step-level samples; policy-aware filtering and pass-rate calibration over verifiable synthesized tasks; Dynamic Tri-Adaptive Curriculum (DTAC), which combines learnable tasks, difficult positive replay, and controlled infeasible-task exposure; and a fully asynchronous RL infrastructure with staleness control and mini-group batching. Experiments show that these components improve training stability and downstream performance. EvoCUA-1.5 achieves 63.2\% success on OSWorld-Verified, outperforming comparable 32B/35B-scale open-weight baselines and even approaching models with significantly larger parameter counts. Overall, EvoCUA-1.5 provides a practical framework for scaling online RL in multi-turn computer-use agents.
• 中文摘要 计算机代理必须通过与部分可观察的多模态桌面环境反复交互来解决长期任务。尽管模仿学习和离线轨迹细化提供了强先验，静态痕迹无法覆盖真实计算机使用的因果反馈循环：每个动作都会改变屏幕状态、未来动作空间和恢复选项。EvoCUA-1.5将自我演进的计算机使用代理从离线体验学习扩展到在线强化学习，策略与可执行的沙盒环境交互，并从可验证的任务结果中得到改进。在这个设定下，在线强化学习不仅需要直接重复使用单回合语言强化学习的配方。多回合交互引入了上下文管理的观测、稀疏的终端奖励、可变长度的轨迹以及缓慢的环境反馈。EvoCUA-1.5通过步进策略优化（STEPO）解决了这些挑战，该优化在分解为步进层样本后保持轨迹层级优势平衡;针对可验证综合任务的策略感知过滤和通过率校准;动态三适应课程（DTAC），结合可学习任务、困难的积极重播和受控的不可行任务暴露;以及一个完全异步的强化学习基础设施，支持陈旧控制和小组批处理。实验表明，这些成分能提升训练稳定性和后续表现。EvoCUA-1.5在OSWorld-Verified上取得了63.2%的成功率，优于32B/35B尺度的开放权重基线，甚至接近参数数显著更大的模型。总体而言，EvoCUA-1.5为多回合计算机代理中在线强化学习提供了实用框架。

Length Penalties Make Chain-of-Thought Less Monitorable

长度罚则使思路链更难监控

• Authors: Bryce Little
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.09786
• Pdf link: https://arxiv.org/pdf/2607.09786
• Abstract Length-penalized reinforcement learning can shorten chain-of-thought reasoning while hiding an influence that drives the model's answer. In our experiments, training with length penalties does not stop misleading hints from steering models, even though the models' chains of thought mention the hint much less often. A token-accuracy evaluation would count these runs as successful because they use fewer reasoning tokens with little accuracy loss; it would miss whether the remaining trace still shows what drove the answer. We train Qwen3-4B and Qwen3-14B variants with different target chain lengths, then evaluate them with biasing-hint interventions on held-out MMLU-Pro-R and four transfer benchmarks. Compression sharply cuts reasoning tokens, preserves most multiple-choice accuracy, and leaves hint influence near baseline. At the strongest target, lower-bound faithfulness falls to 63.1% of baseline for Qwen3-14B and 69.4% for Qwen3-4B; the raw rate at which a monitor catches hint use falls from 69% to 49% and from 60% to 48%. To separate length from content, we randomly delete sentences from uncompressed baseline chains until the remaining text matches the compressed length. Even after this length matching, compressed chains disclose the hint 7-35 percentage points less often than baseline chains that we shorten at random, for both Qwen3 sizes and all five evaluation distributions. Compression therefore does more than shorten reasoning, preferentially removing the cues a monitor needs to see what influenced the answer. Together, these results reveal a compression-monitorability frontier in which cheaper reasoning can preserve answers while making the influences behind them harder to detect.
• 中文摘要 长度惩罚强化学习可以缩短思维链推理，同时隐藏驱动模型答案的影响因素。在我们的实验中，尽管模型的思维链条中提及这些提示的频率较低，但带长度惩罚的训练并不能阻止引导模型误导性的提示。标记准确性评估会将这些运行计为成功，因为它们使用较少的推理标记且准确率损失较小;它会错过剩余的痕迹是否还能显示驱动答案的原因。我们训练不同目标链长度的Qwen3-4B和Qwen3-14B变体，然后用MMLU-Pro-R和四个转移基准测试进行偏差提示干预评估。压缩会大幅减少推理标记，保持大部分选择题的准确性，并在基线附近留下线索影响。在最强目标下，Qwen3-14B的忠实度降至基线的63.1%，Qwen3-4B的69.4%;监视器捕捉到线索使用的原始比例从69%降至49%，再从60%降至48%。为了区分长度与内容，我们会随机删除未压缩基线链中的句子，直到剩余文本长度与压缩后长度相匹配。即使完成了这种长度匹配，压缩链在Qwen3大小和五个评估分布中，暴露提示的频率仍比我们随机缩短的基线链低7-35个百分点。因此，压缩不仅仅是缩短推理时间，而是优先去除监听器需要的线索，以便了解是什么影响了答案。这些结果共同揭示了一个压缩可监控性的前沿，在这种领域中，更廉价的推理可以保留答案，同时使背后的影响更难被发现。

Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

无元数据的元加权直接偏好优化，在噪声偏好标签下进行

• Authors: Hua Qu, Yifan Li, Xiaodong Yuan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.09796
• Pdf link: https://arxiv.org/pdf/2607.09796
• Abstract Direct Preference Optimization (DPO) has become an important method for aligning large language models (LLMs) with human preferences because it removes the need for explicit reward modeling and reinforcement learning optimization. However, its performance depends heavily on the quality of preference data, and noisy preference data in real-world settings can weaken alignment performance. To address this issue, we propose a bilevel optimization framework and prove, under certain assumptions, that this framework can recover the DPO optimum under clean data. We further derive a prior form for the learnable weighting function under asymmetric label-flipping noise. Considering that high-quality metadata may be difficult to obtain, we propose a task-agnostic meta-knowledge-driven method that enables meta-learning even when metadata is completely unavailable. To reduce the high cost of higher-order gradients in LLM meta-learning, we combine central-difference approximation with LoRA fine-tuning and develop a scalable training scheme. Experiments on TL;DR summarization and Anthropic HH single-turn dialogue show that the proposed method improves training performance over multiple DPO baselines under different noise rates.
• 中文摘要 直接偏好优化（DPO）已成为将大型语言模型（LLMs）与人类偏好对齐的重要方法，因为它消除了对显式奖励建模和强化学习优化的需求。然而，其性能高度依赖于偏好数据的质量，而在现实环境中噪声较大的数据会削弱比对性能。为解决此问题，我们提出了一个双层优化框架，并在某些假设下证明该框架能够在干净数据下恢复DPO的最优。我们进一步推导出可学习加权函数在非对称标签翻转噪声下的先验形式。考虑到高质量元数据可能难以获得，我们提出了一种任务无关的元知识驱动方法，即使在元数据完全不可得时也能实现元学习。为了降低LLM元学习中高阶梯度的高成本，我们将中心差分近似与LoRA微调结合，开发了可扩展的训练方案。Experiments on TL;DR总结和Anthropic HH单回合对话表明，所提方法在不同噪声率下，能提升多个DPO基线的训练表现。

Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

Robo-ValueRL：离线到在线强化学习的可靠价值估计

• Authors: Wenke Xia, Pei Ren, Wenbo Yu, Yizhuo Zhang, Jifan Li, Yixue Zhang, Yinuo Zhao, Qingyang Gao, Jianlong Fu, Jian Tang, Ji-Rong Wen, Zhengping Che, Di Hu
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.09866
• Pdf link: https://arxiv.org/pdf/2607.09866
• Abstract Offline-to-online reinforcement learning is promising for generalizable robotic manipulation, yet its full-stack complexity obscures reproduction and diagnosis. Within such systems, value estimation plays a central role in prioritizing heterogeneous data for policy improvement. Despite its importance, the central question remains underexplored: how value-function reliability shapes policy optimization in offline-to-online reinforcement learning. To answer this question, we propose Robo-ValueRL, a unified framework that enables reliable value estimation and systematically traces its downstream effects on policy pretraining and online improvement. Concretely, Robo-ValueRL learns a history-conditioned value estimator and evaluates its reliability through global-progress and local-preference metrics. These resulting value estimates are propagated into quality-conditioned consistency-policy pretraining and a residual adaptation module on online rollouts, providing a unified testbed for analyzing how value reliability shapes downstream policy performance. Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, our extensive experiments show that downstream performance is strongly associated with value reliability. Reliable value functions provide better action-quality estimates, allowing value-guided offline RL to scale more effectively than quality-agnostic behavior cloning, and stabilize online improvement by prioritizing high-quality rollout data. Integrating reliable value guidance through offline pretraining with online improvement, our system achieves 86% success on millimeter-level precise chip insertion and 84% on generalizable block disassembly. We hope these findings highlight the importance of value-guided data utilization for effective policy improvement from heterogeneous robotic experience.
• 中文摘要 离线到在线的强化学习在机器人操作中具有广泛潜力，但其全栈复杂性掩盖了复制和诊断。在此类系统中，价值估计在优先处理异质数据以推动政策改进方面起着核心作用。尽管重要，核心问题仍未被充分探讨：价值函数可靠性如何影响离线到在线强化学习中的策略优化。为回答这个问题，我们提出了Robo-ValueRL，这是一个统一框架，能够实现可靠的价值估计，并系统追踪其对政策预训练和在线改进的下游影响。具体来说，Robo-ValueRL学习一个历史条件值估计器，并通过全局进展和局部偏好指标评估其可靠性。这些所得的价值估计会传递到质量条件一致策略预训练和在线推广时的残余适配模块中，提供一个统一的测试平台，用于分析价值可靠性如何影响下游策略表现。通过240小时的线下演示和3000多个线上推广轨迹，我们广泛的实验表明，下游性能与价值可靠性高度相关。可靠的价值函数提供了更好的动作质量估计，使价值引导的离线强化学习比质量无关的行为克隆更有效地扩展，并通过优先考虑高质量的推广数据稳定在线改进。通过离线预训练与在线改进相结合，我们的系统在毫米级精确芯片插入成功率为86%，在可通用模块拆解方面成功率为84%。我们希望这些发现凸显价值导向数据利用对于从异构机器人体验中有效改进政策的重要性。

Nonparametric Bayesian Inverse Reinforcement Learning with Data-Parallel Gibbs Sampling

非参数贝叶斯逆强化学习与数据并行吉布斯采样

• Authors: Sai Anirudh Katupilla, Shreeya Dasa Lakshminath
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.09886
• Pdf link: https://arxiv.org/pdf/2607.09886
• Abstract Inverse Reinforcement Learning recovers reward functions from expert demonstrations, but standard formulations assume that all demonstrations come from a single expert. When demonstrations are pooled from multiple experts with distinct preferences, parametric methods recover an averaged reward that fits no individual expert well. We implement Nonparametric Bayesian Inverse Reinforcement Learning with a Dirichlet Process prior over reward functions, allowing the number of latent reward types to be inferred jointly with the rewards themselves. Inference uses a collapsed Gibbs sampler combining a Chinese Restaurant Process update for cluster assignments with a Metropolis-Hastings update for reward weights, and soft value iteration as the inner planning routine. We evaluate on a 10x10 ObjectWorld grid with two and three ground-truth reward types. The serial sampler recovers K=2 with Adjusted Rand Index of 1.000, substantially outperforming a Maximum Entropy IRL baseline (ARI=0.000). Extension to K=3 shows that the sampler correctly identifies the number of clusters in all runs; assignment ARI of 0.48-0.58 reflects behavioral overlap between expert types that persists across grid instantiations, revealing that reliable K=3 evaluation on ObjectWorld requires controlled object placement rather than random seeding. We further parallelize the sampler across CPU cores using Ray on HPC hardware, achieving a peak speedup of 4.79x at 8 workers, and characterize a throughput-versus-accuracy tradeoff arising from the consensus merge heuristic used during state aggregation. Code and a containerized environment are available at this https URL.
• 中文摘要 逆强化学习从专家演示中恢复奖励函数，但标准表述假设所有演示均来自单一专家。当演示来自多个偏好不同专家的汇总时，参数方法会获得一个不适合单个专家的平均奖励。我们通过对奖励函数的事先使用狄利克雷过程实现非参数贝叶斯逆强化学习，允许与奖励本身共同推断潜在奖励类型的数量。推断使用折叠的Gibbs采样器，结合了中餐馆流程的集群分配更新和Metropolis-Hastings的奖励权重更新，软价值迭代作为内部规划程序。我们在一个10x10的ObjectWorld网格上进行评估，并有两种和三种真实奖励类型。串行采样器恢复K=2，调整后兰德指数为1.000，远超最大熵的真实熵基线（ARI=0.000）。扩展到 K=3 显示采样器正确识别所有运行中的簇数;赋值ARI为0.48-0.58反映了专家类型间行为重叠，这种重叠在网格实例中持续存在，表明在ObjectWorld上可靠的K=3评估需要受控对象放置而非随机做种。我们进一步利用Ray在高性能计算硬件上对采样器跨CPU核心进行并行化，8个工作组时峰值加速达到4.79倍，并描述了状态聚合时共识合并启发式算法带来的吞吐量与准确性权衡。代码和容器化环境可在该 https URL 访问。

A Distributionally Robust Multi-agent Reinforcement Learning Framework for Intelligent Intersection Control

一个分布式稳健的多智能体强化学习框架用于智能交叉控制

• Authors: Shuwei Pei, Joran Borger, Arda Kosay, Bayu Jayawardhana, Muhammed O. Sayin, Saeed Ahmed
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.09899
• Pdf link: https://arxiv.org/pdf/2607.09899
• Abstract Multi-agent reinforcement learning (MARL) has emerged as a promising approach for traffic signal control. However, standard MARL policies typically optimize for expected returns under nominal conditions, leaving them highly vulnerable to spatial-temporal demand shifts and catastrophic congestion under adverse scenarios. To address this critical limitation, this paper proposes an algorithm-agnostic Distributionally Robust (DR) MARL framework integrating an adaptive Contextual-Bandit Worst-Case Estimator (CB-WCE). Operating on a slower timescale, the CB-WCE co-evolves with the traffic controllers by dynamically generating adversarial demand mixtures during training. This steers the learning process to fortify policies against bottleneck scenarios without requiring modifications to the underlying MARL architectures. The framework is evaluated across value-based, actor-critic, and policy-gradient methods on both a synthetic 5x5 grid and a heterogeneous Monaco City network. Empirical results demonstrate that the DR framework prevents unbounded queue growth and profoundly enhances both worst-case robustness and average-case efficiency. Notably, for the Proximal Policy Optimization (PPO) architecture in the Monaco environment, on average, robust retraining reduced the worst-case queue length by 74.39% and improved the average-case network-wide queue length by 75.45%. Furthermore, the retrained policies exhibit strong zero-shot generalization to unseen traffic distributions, highlighting the framework's scalability and potential for resilient real-world urban deployment.
• 中文摘要 多智能体强化学习（MARL）已成为一种有前景的交通信号控制方法。然而，标准的MARL政策通常优化名义条件下的预期回报，使其在不利情景下极易受到时空需求变化和灾难性拥堵的影响。为解决这一关键局限，本文提出了一个算法无关的分布稳健（DR）MARL框架，集成了自适应的上下文-强化最坏情况估计器（CB-WCE）。CB-WCE在较慢的时间尺度下运行，通过在训练期间动态生成对抗性需求混合，与交通控制系统协同演化。这引导学习过程以强化针对瓶颈场景的策略，而无需修改底层MARL架构。该框架在基于价值、行为者-批评者和策略梯度的方法中，在合成的5x5网格和异构摩纳哥城网络上进行评估。实证结果表明，DR框架防止了无界队列增长，并显著提升了最坏情况下的鲁棒性和平均情况下的效率。值得注意的是，对于摩纳哥环境下的近端策略优化（PPO）架构，平均而言，稳健再训练可将最坏情况队列长度减少74.39%，并提升平均网络范围队列长度75.45%。此外，重新调整的策略表现出对未见交通分布的强烈零样本泛化，凸显了框架的可扩展性和在现实世界中具韧性部署的潜力。

Beyond Bayesian Nash: Learning Minimax-Regret Equilibria for Adversarial Team Games under Asymmetric Information

超越贝叶斯纳什：在非对称信息下学习对抗性团队博弈的极小极大-遗憾均衡

• Authors: Naman Aggarwal, Jonathan P. How
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.09993
• Pdf link: https://arxiv.org/pdf/2607.09993
• Abstract Adversarial team games (ATGs) with asymmetric information, such as adversarial path-finding, goal search, and reachability games on graphs, require strategies that are robust to hidden opponent types, such as a hidden goal flag, and to deception. Under asymmetric information, deception is seen as strategic shifts in the type distribution such that the omniscient opponent can collude with Nature and condition its play on the observed type. Existing risk-neutral solution concepts, such as Bayesian Nash equilibrium (BNE), are sensitive to distribution shifts, while distributionally robust approaches provide guarantees only within a prescribed ambiguity set. To address these limitations, we introduce Probabilistically Robust Minimax-Regret Equilibrium (PR-MRE), a novel equilibrium concept that combines the distribution-free robustness of minimax-regret reasoning with probabilistic information from a nominal type distribution. PR-MRE minimizes worst-case regret over a high-confidence subset of the type space, providing protection against strategic redistribution of probability mass while avoiding the conservatism of fully distribution-free approaches. We show that, for normal-form Bayesian games, PR-MRE can be formulated as a robust bilinear program and derive a tractable semidefinite relaxation. We then adapt this relaxation into a novel meta-solver within a robust double-oracle framework, PRMRE-PSRO, enabling population-based learning of approximate PR-MRE strategies via deep reinforcement learning best responses. Experiments on graph-structured adversarial team games demonstrate that PR-MRE discovers strategies with substantially improved worst-case performance across hidden types compared to risk-neutral equilibrium solutions, resulting in more robust behavior under strategic distribution shifts.
• 中文摘要 具有非对称信息的对抗团队游戏（ATG），如对抗寻路、目标搜索和图上的可达性游戏，需要对隐藏对手类型（如隐藏目标旗）和欺骗具有鲁棒性的策略。在非对称信息下，欺骗被视为类型分布中的战略性转变，使全知的对手能够与自然合谋，并对观察到的类型进行条件。现有的风险中性解概念，如贝叶斯纳什均衡（BNE），对分布变化敏感，而分布稳健的方法仅在规定的歧义集中提供保证。为解决这些局限性，我们引入了概率鲁棒极小极大-遗憾均衡（PR-MRE），这是一种新颖的均衡概念，结合了极小极大后悔推理的无分布鲁棒性与名义类型分布的概率信息。PR-MRE在类型空间的高置信子集上最小化最坏情况下的遗憾，保护了概率质量的战略性重分配，同时避免了完全无分布方法的保守主义。我们证明，对于正规形贝叶斯博弈，PR-MRE可以被表述为一个稳健的双线性规划，并推导出一个可解的半正定松弛。随后，我们将这种松弛调整为一种新的元求解器，在一个稳健的双预言机框架下，PRMRE-PSRO，通过深度强化学习最佳响应实现基于群体的近似PR-MRE策略学习。图结构对抗团队博弈的实验表明，PR-MRE发现了在隐藏类型中最坏情况表现显著优于风险中性均衡解的策略，从而在战略分布变化下表现出更稳健的行为。

A Symbolic Neural CPU for Quantization-Simulated Writeback and Interpretable Program Execution

一个用于量化模拟写回和可解释程序执行的符号神经中央处理器

• Authors: Jose Luis Lima de Jesus Silva
• Subjects: Subjects: Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2607.10021
• Pdf link: https://arxiv.org/pdf/2607.10021
• Abstract Neural networks can learn algorithmic input-output mappings, but trusting a learned executor requires more than a correct final answer because the state transitions that produce it are usually hidden. To make those transitions visible, we introduce a trace-supervised symbolic neural CPU, a factorized learned execution architecture that combines recurrent control, an explicit operation router over a fixed differentiable arithmetic-logic unit bank, destination-masked register writeback, complete trajectory supervision and matched fixed-point replay. The model exposes the selected operation, source and destination registers, register trajectory, memory signals and writeback semantics at every step. On the principal 16-wide benchmark, the non-quantized executor reproduces reference execution exactly, while the eight-bit quantization-simulated executor preserves the symbolic operation path through programs of 1,000 instructions. When the same execution is evaluated against a matched fixed-point replay, the residual numerical drift disappears, showing that it comes from a mismatch between continuous and low-precision reference semantics rather than from execution failure. We compare recurrent, Transformer, temporal-convolution, temporal graph-inspired and state-space controllers, and the ablations show that operation-gate supervision is necessary for an inspectable execution path. Hidden-opcode memory-pressure tasks expose the remaining limits in delayed state use and temporal binding. We also extend the interface with ValueMemory, hybrid adaptive leaky integrate-and-fire controllers, candidate-constrained symbolic control trained through behaviour cloning and actor-critic reinforcement learning, and an RV32I base-integer semantic bridge. Together, these results establish a trace-verifiable framework for interpretable, low-precision and controllable neural execution.
• 中文摘要 神经网络可以学习算法输入输出映射，但信任已学会的执行者不仅需要一个正确的最终答案，因为产生该映射的状态转换通常是隐藏的。为了使这些转移可见，我们引入了轨迹监督符号神经CPU、一种结合了递归控制、固定可微算术逻辑单元组上的显式运算路由器、目的掩蔽寄存器写回、完整轨迹监督和匹配固定点重放的分解学习执行架构。模型在每一步都暴露所选操作、源寄存器和目的寄存器、寄存器轨迹、内存信号和写回语义。在主16宽基准测试中，非量化执行器精确重现引用执行，而8位量化模拟执行器通过1000条指令保持符号操作路径。当同一执行与匹配的不动点回放进行评估时，残余数值漂移消失，表明其源于连续引用语义与低精度参考语义的不匹配，而非执行失败。我们比较了递归控制器、Transformer控制器、时间卷积、时间图启发控制器和状态空间控制器，消融结果表明操作门监督对于可检验的执行路径是必要的。隐藏操作码内存压力任务揭示了延迟状态使用和时间绑定的剩余限制。我们还通过ValueMemory、混合自适应泄漏集成与发射控制器、通过行为克隆和actor-critic强化学习训练的候选约束符号控制，以及RV32I基整语义桥进行了扩展。这些结果共同建立了可追踪验证的可解释、低精度且可控神经执行的框架。

TAC-LOCO: Unified Whole-Body Control for Quadrupedal TACtile-Informed LOCO-Manipulation

TAC-LOCO：四足步态态控制的统一全身控制

• Authors: Muqun Hu, Yuhao Zhou, Kabir Ray Malik, Chi Lin, Won Suk Lee, Yu She, Yan Gu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.10132
• Pdf link: https://arxiv.org/pdf/2607.10132
• Abstract Dynamic loco-manipulation requires legged robots to coordinate whole-body motion while maintaining stable physical interaction with grasped objects under uncertain external forces. While tactile sensing has been widely studied for robotic manipulation, its role in dynamic whole-body control remains largely unexplored. Existing works without tactile feedback commonly grasp firmly rather than regulate the grasp according to the interaction. We propose TAC-LOCO, a tactile-augmented unified reinforcement learning framework that encodes tactile array observations from compliant grippers into a compact latent representation and joins it with proprioception for unified control of the legs, arm, and gripper. With effective grasp stability reward design, the policy learns to simultaneously track body velocity and end-effector trajectories, moderate grasp force, and prevent object slip under both gradual load changes and sudden release events. We deploy the policy zero-shot on a Unitree Go2 with an Interbotix WidowX 250 arm and tactile gripper, demonstrating dynamic tactile-informed loco-manipulation under varying external interactions, achieving a 47% reduction in grasping force and an object drop rate of less than 1%.
• 中文摘要 动态机车操控要求有腿机器人协调全身运动，同时在不确定的外部力下保持与被抓物体的稳定物理互动。尽管触觉感知已被广泛用于机器人操作，但它在动态全身控制中的作用仍大多未被充分探索。没有触觉反馈的现有作品通常会牢牢抓住，而非根据互动调节抓握。我们提出了TAC-LOCO，一种触觉增强统一强化学习框架，将顺从抓握者的触觉阵列观察编码成紧凑的潜在表征，并与本体感觉结合，实现对腿部、手臂和夹持器的统一控制。通过有效的抓握稳定性奖励设计，该政策能够同时追踪身体速度和末端执行器轨迹，适度控制抓握力，并防止物体在渐进载荷变化和突然释放事件下滑动。我们在Unitree Go2上部署了零发射策略，配备Interbotix WidowX 250臂和触觉抓钳，展示了在不同外部交互下动态触觉知情的机车操作，抓取力减少47%，物体掉落率低于1%。

REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generation

REVA-PO：稳定强化学习以生成胸部X光报告

• Authors: Li Guo, Anas M. Tahir, Z. Jane Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.10147
• Pdf link: https://arxiv.org/pdf/2607.10147
• Abstract Automated chest X-ray report generation has recently benefited from reinforcement learning (RL) and large language models. However, RL training often suffers from instability or limited exploration due to fixed Kullback-Leibler (KL) regularization and a static reference policy that accumulates KL pressure over time. We propose Response-Weighted and Validation-Anchored Policy Optimization (REVA-PO), a RL framework that stabilizes long-term training via Response-Weighted Regularization (RER) and Validation-Anchored Policy Reset (VAPR). RER dynamically adjusts per-response KL weights based on advantage and reference-policy entropy, relaxing constraints for high-quality responses while tightening them for low-quality ones. Complementarily, VAPR periodically synchronizes the reference and current policies to the best validation checkpoint, resetting accumulated regularization pressure to expand the viable exploration space. To ensure a robust starting point, we employ a three-stage pipeline consisting of warm-up training, classifier-guided supervised fine-tuning, and RL. Extensive evaluations on MIMIC-CXR and IU-Xray demonstrate that REVA-PO sets new state-of-the-art benchmarks in both linguistic quality and clinical accuracy. Notably, BLEU-4 improves by 5.1% on MIMIC-CXR and 3.6% on IU-Xray, while CheXpert F1 and RadGraph F1 scores increase by 4.5% and 12.8%, respectively, over prior leading methods. The code is publicly available at this https URL.
• 中文摘要 近年来，自动化胸部X光报告的生成受益于强化学习（RL）和大型语言模型。然而，由于固定的库尔巴克-莱布勒（KL）正则化和静态参考策略，导致强化学习训练常因不稳定性或探索有限而受限，这些策略会随着时间积累KL压力。我们提出了响应加权与验证锚定策略优化（REVA-PO），这是一种通过响应加权正则化（RER）和验证锚定策略重置（VAPR）稳定长期训练的强化学习框架。RER根据优势和参考策略熵动态调整每个响应的KL权重，对高质量响应放宽约束，对低质量响应则收紧约束。此外，VAPR定期同步参考策略和当前策略至最佳验证检查点，重置积累的正则化压力，扩展可行勘探空间。为确保起点稳健，我们采用三阶段流程，包括热身训练、分类器引导的监督微调和强化学习。对MIMIC-CXR和IU-X光的广泛评估表明，REVA-PO在语言质量和临床准确性方面树立了新的尖端基准。值得注意的是，BLEU-4在MIMIC-CXR上提升了5.1%，在IU-Xray中提升了3.6%，而CheXpert F1和RadGraph F1的得分分别比以往领先方法提升了4.5%和12.8%。代码在此 https URL 公开获取。

Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization

超越欧几里得剪裁：通过黎曼等距策略优化克服LLM强化学习中的探索崩溃

• Authors: Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Mingxuan Wang, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.10169
• Pdf link: https://arxiv.org/pdf/2607.10169
• Abstract Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).
• 中文摘要 强化学习（RL）已成为增强大型语言模型推理能力的主流范式。然而，带有PPO-Clip的强化学习算法本质上受限于探索坍缩。后续研究仍主要依赖启发式，未能确定PPO-Clip失败的根本原因。这项工作揭示了PPO-Clip的根本缺陷：它隐含地使用欧几里得度量来衡量策略差异，这在理论上与策略黎曼流形上的内在几何不一致。这种几何不匹配导致低概率区域的更新过于保守，而在高概率区域则过于激进，最终导致探索崩溃。为纠正这一几何缺陷，我们提出了黎曼等距策略优化（RIPO），它保证黎曼流形上的等距策略更新，有效平衡探索与利用。我们进一步证明，RIPO实现了有利的偏差-方差权衡，从而稳定了优化。大量实验表明，RIPO在七个竞赛级基准测试中显著优于现有LLM RL算法（在AIME24上比GRPO提升最多60%）。

Sharper Analysis of Single-Loop Methods for Bilevel Optimization

双层优化中单环方法的更敏锐分析

• Authors: Yubo Zhou, Jun Shu, Luo Luo, Junmin Liu, Deyu Meng, Guang Dai, Haishan Ye
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.10263
• Pdf link: https://arxiv.org/pdf/2607.10263
• Abstract Bilevel optimization underpins many machine learning applications, including hyperparameter optimization, meta-learning, neural architecture search, and reinforcement learning. While hypergradient-based methods have advanced significantly, a gap persists between theoretical guarantees and practical single-loop implementations required for efficiency. We bridge this gap by establishing sharper convergence results for single-loop approximate implicit differentiation (AID) and iterative differentiation (ITD) methods, leveraging our proposed analytical framework, decoupled norm analysis (DNA). For AID, we improve the convergence rate from $\mathcal{O}(\kappa^6/K)$ to $\mathcal{O}(\kappa^5/K)$, where $\kappa$ is the condition number of the inner-level problem. For ITD, we prove that the asymptotic error is $\mathcal{O}(\kappa^2)$, exactly matching the known lower bound and improving upon the previous $\mathcal{O}(\kappa^3)$ guarantee. Numerical experiments on synthetic and real tasks corroborate our theoretical findings.
• 中文摘要 双级优化支撑着许多机器学习应用，包括超参数优化、元学习、神经结构搜索和强化学习。尽管基于超梯度的方法取得了显著进步，但理论保证与实现效率所需的实际单环实现之间仍存在差距。我们通过建立更清晰的收敛结果，利用我们提出的分析框架——解耦范数分析（DNA），弥合了这一空白。对于AID，我们将收敛率从$\mathcal{O}（\kappa^6/K）$提升到$\mathcal{O}（\kappa^5/K）$，其中$\kappa$是内层问题的条件数。对于ITD，我们证明渐近误差为$\mathcal{O}（\kappa^2）$，恰好符合已知下界，并改进了之前的$\mathcal{O}（\kappa^3）$保证。对合成和真实任务的数值实验证实了我们的理论发现。

Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systems

衡量模拟与现实差距：设计一个经济实惠的现实世界基准平台，用于AIoT系统中的强化学习

• Authors: Rongping Zhou, Omid Tavallaie, Shuaijun Chen, Albert Y. Zomaya
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.10309
• Pdf link: https://arxiv.org/pdf/2607.10309
• Abstract Reinforcement learning (RL) is commonly employed to enhance the performance of autonomous systems, including the Autonomous Internet of Things (AIoT). However, the trial-and-error nature of RL, when conducted in real-world environments, is costly and hazardous in some scenarios. Consequently, the majority of RL research is conducted in simulation. This reliance introduces challenges related to the Sim-to-Real transferability. Evaluating the Sim-to-Real algorithmic robustness and the Sim-to-Real gap is a critical prerequisite for research aimed at improving RL performance in the real world. Therefore, industries such as robotics have developed concurrent simulation and physical platforms to facilitate this research. However, a universal Sim-to-Real benchmark platform for AIoT does not currently exist. To address these concerns, we developed a real-world AIoT platform for studying RL in AIoT. On this platform, an agent deployed on an edge device plays video games on a separate host computer via a hardware-emulated keyboard, guided by vision input. This platform uses commercially available components costing less than USD 400, together with two computers. Because the system's objective is game score maximization, it inherently mitigates safety risks associated with real-world RL deployments. Experimental results show the simulation-trained agent suffers a 1160% performance degradation relative to the human-level performance after real-world deployment, indicating a significant Sim-to-Real gap. Direct real-world training using the deep Q-network (DQN) algorithm achieves approximately 38% of human-level performance after 10 million training steps, demonstrating the feasibility of RL under real-world conditions. These results suggest that the proposed Sim-to-Real benchmark platform provides a substantial foundation for qualitative and quantitative evaluations of RL in real-world AIoT systems.
• 中文摘要 强化学习（RL）常被用来提升自主系统的性能，包括自主物联网（AIoT）。然而，在现实环境中进行强化学习的反复试验性质，在某些情况下成本高昂且存在风险。因此，大多数强化学习的研究都在模拟中进行。这种依赖带来了与模拟到真实可转移性的挑战。评估模拟到现实的算法鲁棒性以及模拟与现实之间的差距，是旨在提升现实世界中强化学习性能研究的关键前提。因此，机器人等行业开发了并行仿真与物理平台以促进相关研究。然而，目前还没有适用于 AIoT 的通用模拟到现实基准平台。为解决这些问题，我们开发了一个现实世界的AIoT平台，用于在AIoT中研究强化学习。在该平台上，部署在边缘设备上的代理通过硬件模拟键盘在独立主机上玩电子游戏，并由视觉输入引导。该平台使用市售的组件，价格低于400美元，配备两台计算机。由于系统的目标是最大化游戏得分，它本质上降低了现实环境中强化学习部署相关的安全风险。实验结果显示，经过模拟训练的智能体在实际部署后相较于人类水平的性能下降了1160%，显示出模拟与真实之间的显著差距。使用深度Q网络（DQN）算法进行直接的真实世界训练，在1000万次训练步后，可实现约38%的人类水平表现，证明了强化学习在现实条件下的可行性。这些结果表明，拟议的模拟到现实基准平台为现实AIoT系统中强化学习的定性和定量评估提供了坚实基础。

VINE: Taming Generative Control Policies for Reinforcement Learning

VINE：驯服生成控制策略以促进强化学习

• Authors: Rushuai Yang, Zhuo Han, Houlin Li, Hecheng Wang, Zhichao Wu, Rui Zhang, Zhaowei Zhang, Zihong Chen, Xiaohan Yan, Chiming Liu, Yi Chen, Wei Shan, Maoqing Yao
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.10369
• Pdf link: https://arxiv.org/pdf/2607.10369
• Abstract Flow-matching policies have emerged as an effective policy parameterization for robot learning. They iteratively generate actions from noise, enabling highly expressive modeling of complex and multimodal action distributions. However, prior works observed that scaling these policies with value-gradient reinforcement learning (RL) often leads to training instability. Existing methods attribute this instability to iterative generation and therefore avoid end-to-end value-gradient optimization by sacrificing iterative generation, high expressiveness, or value-gradient optimization. Contrary to prior belief, we show the instability does not stem from iterative generation itself, but from the vanilla sampling strategy originally designed for behavior cloning, which becomes brittle under value-gradient RL. Motivated by this insight, we propose VINE, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies. Instead of following a single flow trajectory, VINE reconstructs a new interpolation state at every denoising step, creating a stable differentiable path for value-gradient propagation while remaining compatible with the original flow-matching denoising process. As a result, VINE preserves the expressiveness and iterative generation of flow-matching without sacrificing end-to-end value-gradient optimization. Despite performing end-to-end backpropagation through all ten denoising steps, VINE achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task. Videos are available on our website: this https URL.
• 中文摘要 流量匹配策略已成为机器人学习中有效的策略参数化。它们通过迭代从噪声中生成动作，使复杂和多模动作分布的高度表达性建模成为可能。然而，先前的研究观察到，利用价值梯度强化学习（RL）来扩展这些策略，常常导致训练不稳定。现有方法将这种不稳定性归因于迭代生成，因此通过牺牲迭代生成、高表现力或值梯度优化来避免端到端的值梯度优化。与之前的认知相反，我们证明这种不稳定性并非源于迭代生成本身，而是源于最初为行为克隆设计的原版采样策略，该策略在值梯度强化学习下变得脆弱。基于这一见解，我们提出了VINE，一种基于强化学习的采样方法，能够实现流匹配策略的稳定端到端值梯度优化。VINE不再沿单一流轨迹走，而是在每个去噪步骤重建新的插值态，从而在保持与原始流量匹配去噪过程兼容的同时，创建稳定的可微导路径以实现值梯度传播。因此，VINE 保持了流匹配的表现力和迭代生成，同时不牺牲端到端的值梯度优化。尽管通过所有十个去噪步骤实现端到端反向传播，VINE在OGBench离线强化学习基准测试和现实机器人操作任务中实现了稳定的策略改进，并持续优于最先进的强化学习方法。视频可在我们的网站上观看：https 网址。

GRASP: GRanularity-Aware Search Policy for Agentic RAG

GRASP：代理性RAG的天才感知搜索政策

• Authors: Varun Gandhi, Jaewook Lee, Shantanu Todmal, Franck Dernoncourt, Ryan Rossi, Zichao Wang, Andrew Lan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.10463
• Pdf link: https://arxiv.org/pdf/2607.10463
• Abstract Agentic retrieval-augmented generation (RAG) extends static RAG by allowing language models to iteratively reason, generate search queries, retrieve evidence, and predict answers. However, it remains challenging for models to decide when to retrieve, whether to use lexical matching or semantic similarity, and how to control context granularity to prevent irrelevant tokens from interfering with agent reasoning. In this paper, we introduce GRASP, a reinforcement learning (RL) framework for training agents to adaptively coordinate complementary retrieval tools during multi-step reasoning. GRASP provides the agent with semantic search, keyword search, and paragraph-reading actions, enabling it to retrieve sentence-level evidence and expand further context only when needed. We train the policy with a reward that jointly accounts for answer accuracy, grounded reading, complementary search, and turn efficiency. Experiments on multi-hop reasoning benchmarks show that GRASP improves both retrieval recall and downstream question answering performance compared with single-step retrieval, prompting-based agentic RAG, and RL-based retrieval baselines. Qualitative and ablation analyses show that the learned policy develops interpretable skimming and scanning behavior: it uses semantic search for broad exploration, paragraph reading for local verification, and keyword search for entity-specific evidence. These results suggest that learning to coordinate retrieval signals and context granularity is critical for agent's correct reasoning.
• 中文摘要 代理检索增强生成（RAG）通过允许语言模型迭代推理、生成搜索查询、检索证据和预测答案，扩展了静态RAG。然而，模型在决定何时检索、是否使用词汇匹配或语义相似度，以及如何控制上下文粒度以防止无关令牌干扰代理推理方面仍然具有挑战性。本文介绍了GRASP，一种强化学习（RL）框架，用于训练智能体在多步推理过程中自适应协调互补检索工具。GRASP为代理提供语义搜索、关键词搜索和段落阅读操作，使其仅在需要时检索句子层面的证据并扩展上下文。我们通过奖励来培训保单，这些奖励共同考虑了答案的准确性、扎实的阅读、互补的搜索和轮流效率。多跳推理基准测试的实验显示，GRASP相比单步检索、基于提示的代理RAG和基于强化学习的检索基线，在检索回忆和下游问题回答表现上均有提升。定性和消融分析显示，该学到的策略发展出可解释的浏览和扫描行为：它使用语义搜索进行广泛探索，段落阅读进行局部验证，关键词搜索用于实体特定证据。这些结果表明，学习协调检索信号和上下文细度对于智能体正确推理至关重要。

Reinforcement Learning with Verifiable Physics: Post-training LLMs with Continuous Rewards

基于可验证物理的强化学习：带持续奖励的训练后大型语言模型

• Authors: Pengfei Cai, Utkarsh Utkarsh, Alan Edelman, Christopher Vincent Rackauckas, Rafael Gomez-Bombarelli
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2607.10474
• Pdf link: https://arxiv.org/pdf/2607.10474
• Abstract Partial differential equations (PDEs) are foundational to modeling in science and engineering, but constructing reliable numerical solvers remains labor-intensive, demanding expert knowledge of discretization schemes, stability conditions, and boundary treatments. Recent work has begun to frame PDE solving as a code-generation task for large language models (LLMs), yet existing approaches operate primarily at inference time: relying on prompting, debugging, self-refinement, and test-time scaling rather than adapting the model itself. In parallel, reinforcement learning with verifiable rewards has emerged as a post-training paradigm for code and math reasoning, but its verifiers are typically binary: a compiler runs, or a test passes. Such signals discard the graded structure of scientific correctness, where two solvers may both execute and yet differ in solution accuracy by orders of magnitude. In this work, we introduce RLVP: Reinforcement Learning with Verifiable Physics, an RL post-training framework for multi-PDE solver code generation. RLVP addresses this verifiability gap with a hybrid verifier: hard program-validity checks ensure executability, while continuous physics rewards score function-space accuracy and PDE-residual consistency. A single policy is post-trained across diverse PDE families spanning hyperbolic, parabolic, elliptic, and incompressible-flow systems. RLVP improves over both pre-trained and supervised-only baselines on PDE benchmarks, and shows zero-shot improvement transfer to held-out PDEs. We show that a smaller LLM post-trained with RLVP can outperform prompting a frontier model on in-distribution PDE solver generation. The trained policy shows evidence of compositionality in numerical motifs: it recombines stencils, time-stepping schemes, and boundary-handling primitives learned from the PDEs used in training into generated solvers for unseen PDE problems.
• 中文摘要 偏微分方程（PDE）是科学和工程建模的基础，但构建可靠的数值求解器仍然劳动密集，需要对离散化方案、稳定性条件和边界处理的专业知识。近期研究开始将偏微分方程求解视为大型语言模型（LLM）的代码生成任务，但现有方法主要在推理时间上工作：依赖提示、调试、自我优化和测试时间缩放，而非对模型本身进行调整。与此同时，带有可验证奖励的强化学习已成为代码和数学推理的训练后范式，但其验证者通常是二元的：编译器运行，或测试通过。这种信号摒弃了科学正确性分级结构，即两个求解器虽然都能执行，但解答准确度相差数个数量级。在本研究中，我们介绍了RLVP：带可验证物理的强化学习，这是一个用于生成多偏微分方程求解器代码的强化学习后训练框架。RLVP通过混合验证器填补了这一可验证性缺口：硬性程序有效性检查确保可执行性，而连续物理奖励评分函数空间的准确性和偏微分方程残差一致性。单一策略在涵盖双曲流、抛物线、椭圆流和不可压缩流系统中的多样偏微分方程家族中进行后期训练。在偏微分方程基准测试中，RLVP优于预训练和仅监督基线，并且显示出对保留偏微分方程的零机会改进转移。我们证明，使用RLVP后训练的小型LLM在分布内偏微分方程求解器生成上优于提示模型。训练后的策略显示出数值图案的组合性：它将从训练中使用的偏微分方程中学到的模板、时间步进方案和边界处理原语重新组合为未见偏微分方程问题的生成求解器。

ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples

ARMOR：用非策略锚点样本稳定策略中的LLM RL

• Authors: Kexin Huang, Junkang Wu, Jinda Lu, Shuo Yang, Chiyu Ma, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.10481
• Pdf link: https://arxiv.org/pdf/2607.10481
• Abstract Reinforcement learning (RL) has significantly enhanced the reasoning capabilities of large language models (LLMs), yet the training process remains notoriously fragile. In this work, we investigate a critical source of this instability: over-optimization, where models exploit training heuristics at the expense of generalizable reasoning. While reverse KL regularization is the standard defense against such degradation, our analysis reveals that it is often insufficient in this regime, as it fails to ensure comprehensive coverage of the reference distribution. To address this, we propose ARMOR (Anchor Rollout and Mixed Optimization for RL), a framework that shifts the paradigm from passive penalty to active sample stabilization. ARMOR comprises two key components: (1) Anchor Rollout, which leverages off-policy data from the reference policy to preserve established solution patterns; and (2) Mixed Optimization, which reformulates the policy objective to enable controlled exploration without relying on auxiliary losses. Extensive experiments on reasoning benchmarks validate that ARMOR effectively mitigates validation collapse, enabling sustained performance improvements over extended training horizons.
• 中文摘要 强化学习（RL）显著提升了大型语言模型（LLMs）的推理能力，但训练过程依然非常脆弱。在本研究中，我们探讨了这种不稳定性的一个关键来源：过度优化，即模型利用训练启发式方法，牺牲了可推广推理能力。虽然反向KL正则化是防止这种降解的标准防御，但我们的分析显示，在这种模式下它往往不够充分，因为无法确保对参考分布的全面覆盖。为此，我们提出了ARMOR（强化学习的锚点展开与混合优化）框架，将范式从被动惩罚转向主动样本稳定。ARMOR包含两个关键组成部分：（1）锚定部署，利用参考策略中的非策略数据来维护既定的解决方案模式;以及（2）混合优化，重新表述政策目标，实现不受辅助损失的受控勘探。大量推理基准测试验证ARMOR有效缓解验证崩溃，从而在更长的训练周期内实现持续的性能提升。

Tool-Adaptive LLM Reranker

工具自适应LLM重新排序器

• Authors: Zichuan Liu, Ruijin Hua
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.10555
• Pdf link: https://arxiv.org/pdf/2607.10555
• Abstract Generative Large Language Models (LLMs) have revolutionized information retrieval, yet their strictly parametric nature frequently leads to severe factual hallucinations when confronted with complex queries beyond their epistemic boundaries. While external tool-calling can mitigate this, indiscriminately invoking search tools for every document during reranking incurs prohibitive latency overheads, creating an intractable accuracy-efficiency dilemma. To address this challenge, we propose TALRanker, a novel framework that formalizes pointwise relevance scoring as an agentic Markov decision process. We optimize it via a two-stage training paradigm. An initial warm-up utilizes a language-preserving hybrid loss to prevent the catastrophic forgetting of native generative capacities. Subsequently, an asymmetric cost-aware reward equipped in reinforcement learning forces the policy to autonomously bypass tools for maximum efficiency when confident, while selectively retrieving external evidence to avert severe hallucination penalties when uncertain. Extensive evaluations demonstrate that TALRanker achieves state-of-the-art performance across standard and reasoning-intensive retrieval benchmarks, matching throughput with pointwise rerankers while outperforming parameter-heavy reasoning models.
• 中文摘要 生成大型语言模型（LLM）彻底改变了信息检索，但其严格参数化的特性常常导致在面对超越认知边界的复杂查询时出现严重的事实错觉。虽然外部工具调用可以缓解这一问题，但在重新排序时对每个文档无差别地调用搜索工具会带来极高的延迟开销，造成一个难以解决的准确性与效率的困境。为应对这一挑战，我们提出了TALRanker，这是一个新颖框架，将点状相关性评分形式化为一种代理马尔可夫决策过程。我们通过两阶段训练范式进行优化。初步热身利用语言保留混合损失，防止对母语生成能力的灾难性遗忘。随后，强化学习中配备的非对称成本感知奖励迫使策略在自信时自主绕过工具以实现最大效率，而在不确定时选择性地检索外部证据以避免严重的幻觉惩罚。大量评估表明，TALRanker在标准和推理密集型检索基准测试中均达到顶尖性能，吞吐量与点数重排序器匹配，同时优于参数密集推理模型。

UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

UNIBROWSE：一个多模浏览合成的数据到代理框架

• Authors: Xiyu Wei, Qingwei Zong, Zhuocheng Yu, Sujian Li
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.10557
• Pdf link: https://arxiv.org/pdf/2607.10557
• Abstract Multimodal BrowseComp tasks require agents to combine perception, tool use, and long-horizon reasoning over dynamic web content, challenging their ability to handle compositional structure, open-world uncertainty, and multimodal integration across extended interactions. Crucially, real-world multimodal browsing involves three distinct information-flow patterns: text-only, image-to-text, and text-to-image, yet existing data construction methods cover only the text-only and image-to-text patterns, leaving text-to-image largely unaddressed and limiting agent generality and robustness. We introduce UNIBROWSE, a unified data pipeline that for the first time simultaneously generates training data covering all three patterns, augments curated knowledge graphs with live web retrieval for improved fidelity, and introduces a novel metric of exploration degree to filter low-signal instances for efficient reinforcement learning. Through this pipeline, we produce high-quality cold-start tool-use trajectories and exploration-rich QA pairs, and train a 35B-scale agent via supervised fine-tuning and exploration-aware this http URL resulting UNIBROWSE agent achieves state-of-the-art performance on multimodal BrowseComp benchmarks, attaining an average accuracy of 54.4 across five diverse benchmarks -- an improvement of 10.5 points over its base model Qwen3.5-35B-A3B -- and surpassing serveral closed-source agent workflows such as GPT-5 (42.9), Gemini-2.5 Pro (44.8), and Gemini-2.5 Flash (41.3).
• 中文摘要 多模态浏览合成任务要求代理将感知、工具使用和长期推理结合在动态网页内容上，挑战他们处理组合结构、开放世界不确定性和跨跨扩展交互的多模态集成的能力。关键的是，现实世界的多模态浏览涉及三种不同的信息流模式：仅文本、图像到文本和文本到图像，但现有数据构建方法仅覆盖纯文本和图像到文本的模式，导致文本到图像在很大程度上未被解决，限制了代理的普遍性和稳健性。我们引入了UNIBROWSE，这是一条统一的数据流水线，首次同时生成涵盖所有三种模式的训练数据，通过实时网页检索增强了精选知识图谱以提升准确性，并引入了一种新的探索度指标，用于过滤低信号实例以实现高效的强化学习。通过该流程，我们生成高质量的冷启动工具使用轨迹和丰富的探索QA对，并通过监督微调和探索感知训练35B规模代理，该HTTP URL生成的UNIBROWSE代理在多模态BrowseComp基准测试中实现了最先进的性能，在五个多样化基准测试中平均准确率达到54.4——比其基础模型Qwen3.5-35B-A3B提升10.5分——并超越了多个闭源代理工作流例如GPT-5（42.9）、Gemini-2.5 Pro（44.8）和Gemini-2.5 Flash（41.3）。

Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

代理DPO：从模仿到代理策略优化，专家轨迹

• Authors: Yixiong Chen, Alan Yuille
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.10601
• Pdf link: https://arxiv.org/pdf/2607.10601
• Abstract Large Language Model (LLM) agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which treats multi-turn agent behavior as ordinary text imitation. This recipe is simple and low-cost, but it only learns to imitate the sequence of expert actions, rather than training the agent to choose the right action against plausible mistakes at each state. Existing methods to mitigate this problem include preference learning or reinforcement learning, but they usually need high-cost environment rollouts and reward models. We propose Agentic-DPO, a lightweight offline agent policy optimization method that turns expert trajectories into state-conditioned preference supervision. At each expert action state, Agentic-DPO samples a one-step action from the current state, treats plausible wrong actions as negatives, and contrasts them with the expert action using a DPO-style preference objective. To avoid mixing both policy and schema in preference learning, we introduce Policy-Preserving Augmentation (PPA), which renders the same latent trajectory under multiple schemas while keeping the expert policy fixed. Agentic-DPO requires no online environment rollout, reward model, or full-trajectory student exploration. We conduct experiments across StableToolBench, tau-bench retail, and Mind2Web, where Agentic-DPO consistently improves agents at different model scales beyond imitation. In particular, it raises tau-bench accuracy from 21.7% (SFT) to 41.4% for a 9B model, matching online GRPO under the same backbone with only step-level rollouts and without environment interaction during gradient steps. The results suggest that expert trajectories can support low-cost agentic policy optimization when converted from demonstrations into state-level action preferences. Code for Agentic-DPO is released at this https URL.
• 中文摘要 大型语言模型（LLM）代理通常通过监督微调（SFT）从专家轨迹训练，该方法将多回合代理行为视为普通文本模仿。这个配方简单且成本低，但它只是学会模仿专家动作的顺序，而不是训练代理人在每个状态下针对可能错误做出正确的行动。现有缓解该问题的方法包括偏好学习或强化学习，但通常需要高成本的环境推广和奖励模型。我们提出了Agentic-DPO，一种轻量级离线代理策略优化方法，将专家轨迹转化为状态条件偏好监督。在每个专家动作状态，Agentic-DPO从当前状态中抽样一步动作，将可能错误的行为视为负面，并使用DPO风格的偏好目标与专家动作进行对比。为避免在偏好学习中混合策略和模式，我们引入了策略保留增强（PPA），该方法在多个模式下呈现相同的潜在轨迹，同时保持专家策略的固定性。Agentic-DPO无需在线环境推广、奖励模式或学生全面探索。我们在StableToolBench、tau-bench零售和Mind2Web上进行了实验，Agentic-DPO在不同模型尺度上持续改进代理，超越了模仿。特别是，它将9B模型的tau-bench准确率从21.7%（SFT）提升至41.4%，在同一骨干网下仅进行步级展开且梯度步骤中无环境交互，匹配在线GRPO。结果表明，专家轨迹在从示范转化为州级行动偏好时，可以支持低成本的代理性政策优化。Agentic-DPO 的代码在此 https 网址发布。

Sequential compliance decisions of firms on cross-border data flows: An institutionally anchored decision support system

跨境数据流企业的顺序合规决策：一个机构化的决策支持系统

• Authors: Yuepeng Zhou, Dongchi Xing, Li Xiong
• Subjects: Subjects: Computers and Society (cs.CY)
• Arxiv link: https://arxiv.org/abs/2607.10620
• Pdf link: https://arxiv.org/pdf/2607.10620
• Abstract The economic value of data arises from its flow across organizations and national borders. Yet increasingly stringent data governance regimes are turning cross-border transfer into an institutionally constrained sequential decision, in which firms repeatedly weigh compliance costs against the value of data flows. From the perspective of a data-exporting firm, this paper develops an institutionally anchored decision support system. It converts regulatory rules into a computable minimal compliance mapping and models the firm's weekly decisions as a finite-horizon Markov decision process (MDP), with compliance represented as a hard constraint rather than a penalty term. The resulting problem is solved using masked deep reinforcement learning, while counterfactual path advantages provide interpretable signals to support the firm's cross-border data flow decisions. Experiments show that the policies learned within the system outperform the baselines considered and deliver interpretable, auditable decision support. Local processing concentrates in states where the business value of small lawful transfers does not cover their compliance costs, and the localization boundary shifts systematically as the regime tightens. Credential acquisition is front-loaded within the compliance year, and shallow decision trees reproduce the policy's decisions with high fidelity. Treating the persistent-friction weight as a continuous representation of regulatory strictness further reveals an absorb-then-adjust pattern, in which expected rewards decline before observable behavior changes, implying that assessments based only on behavioral indicators may understate the burden already borne by firms. Moreover, the system is not tied to any specific regulation and can be transferred to other jurisdictions and rule-based compliance problems.
• 中文摘要 数据的经济价值源于其跨组织和跨国边界的流动。然而，日益严格的数据治理制度正将跨境传输转变为制度性限制的顺序决策，企业反复权衡合规成本与数据流价值。从数据导出公司的角度，本文构建了一个制度化的决策支持系统。它将监管规则转换为可计算的最小合规映射，并将公司的每周决策建模为有限视野的马尔可夫决策过程（MDP），将合规表示为硬约束而非惩罚项。由此产生的问题通过掩蔽深度强化学习解决，反事实路径优势则为企业跨境数据流决策提供了可解释的信号。实验显示，系统内学习的策略优于考虑的基线，并提供可解释、可审计的决策支持。本地处理集中在那些小额合法转让的商业价值不足以覆盖合规成本的州，随着制度收紧，本地化边界系统性地调整。凭证获取在合规年度内前期进行，浅层决策树高保真地重现了政策决策。将持续摩擦权重视为监管严格性的连续表现，进一步揭示了一种吸收后调整的模式，即预期奖励在可观察到的行为变化前下降，这意味着仅基于行为指标的评估可能低估了企业已承担的负担。此外，该系统不受任何特定法规约束，可以转移至其他司法管辖区，解决基于规则的合规问题。

Learning to Fine-tune Foundation Models under Resource Limitations

在资源限制下学习如何微调基础模型

• Authors: Thomas Tsouparopoulos, Iordanis Koutsopoulos
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.10694
• Pdf link: https://arxiv.org/pdf/2607.10694
• Abstract We study the problem of optimal continual fine-tuning for a pre-trained Foundation Model deployed at a resource-limited device. At each time slot, a new batch of training data arrives, and the controller is faced with two options: either use the data to fine-tune the model and incur a compute cost, or do not fine-tune the model and discard the data. After the decision, the performance of the current model is measured in terms of an application-specific performance metric such as classification accuracy. Our objective is to learn an optimal policy that determines \emph{when to fine-tune the model} on a single task (e.g., sentiment analysis), under a finite compute budget. We formulate this online decision-making problem as a constrained Markov Decision Process, where the system state captures three essential aspects: (\textit{i}) model's performance, (\textit{ii}) computational budget, and (\textit{iii}) data distribution relevance to historic data encountered up to that point. The transition to the next state is stochastic and therefore, we propose a reinforcement learning-based method to solve this problem, namely the \emph{actor-critic} algorithm. We also consider the special case where the performance of fine-tuning for a given model can be predicted or estimated prior to decision; in this case the problem becomes a Dynamic Programming one. Experiments with a large pre-trained model on a widely-used text classification dataset demonstrate that our method consistently outperforms fine-tuning approaches with the same compute budget by more than $4\%$ in terms of accuracy and achieves $97\%$ of full-parameter fine-tuning accuracy while requiring only $25\%$ of the fine-tuning steps.
• 中文摘要 我们研究在资源有限设备上部署的预训练基础模型的最佳持续微调问题。每个时隙都会有一批新的训练数据到达，控制器面临两个选择：要么利用数据对模型进行微调并产生计算成本;要么不微调模型并丢弃数据。决策后，当前模型的性能通过应用特定的性能指标（如分类准确性）进行衡量。我们的目标是在有限的计算预算下，学习一个最优策略，决定在单一任务（例如情感分析）上\emph{何时微调模型}。我们将该在线决策问题表述为受限马尔可夫决策过程，系统状态涵盖三个关键方面：（\textit{i}）模型性能、（\textit{ii}）计算预算，以及（\textit{iii}）数据分布与迄今为止历史数据的相关性。向下一个状态的转变是随机的，因此我们提出了一种基于强化学习的方法来解决这个问题，即 \emph{actor-critic} 算法。我们还考虑了一种特殊情况，即在决策前可以预测或估计模型的微调性能;在这种情况下，问题就变成了动态规划的问题。在广泛使用的文本分类数据集上，使用大型预训练模型的实验表明，我们的方法在精度方面始终优于相同计算预算下的微调方法超过4%美元，且全参数微调精度达到97%%美元，而微调步骤仅需25%%美元。

To Answer or to Abstain: Mitigating Search-Agent Hallucinations via Abstention-Aware Reinforcement Learning

回答还是回避：通过回避意识强化学习缓解搜索代理幻觉

• Authors: Fengji Zhang, Tianyu Fan, Yuxiang Zheng, Xinyao Niu, Chengen Huang, Jacky Keung, Bei Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.10738
• Pdf link: https://arxiv.org/pdf/2607.10738
• Abstract Recent advances in equipping Large Language Models (LLMs) with search tools and outcome-reward reinforcement learning (RL) have achieved new state-of-the-art results on open-domain QA tasks. However, we argue that current training paradigms harbor a critical vulnerability: they predominantly reward correct answers but fail to penalize fabricated ones when retrieval fails, thereby implicitly exacerbating hallucinations. To address this, we propose Abstention-Aware Reinforcement Learning (AWA-RL), which dynamically shapes the abstention reward utilizing the model's query-specific prior capabilities and continuous on-policy training observations. We also introduce a novel metric, RA-F1, to measure the capability-reliability trade-off. Compared to non-abstaining baselines, AWA-RL boosts absolute precision by up to 10.3% and overall RA-F1 by 2.9%, with only marginal sacrifice in raw accuracy. These results confirm that AWA-RL successfully yields highly capable and reliable search agents. The code, data, and model weights are publicly available at this https URL.
• 中文摘要 近年来，大型语言模型（LLMs）配备搜索工具和结果奖励强化学习（RL）方面的进步，在开放领域质量保证任务中取得了新的先进成果。然而，我们认为当前的培训范式存在一个关键漏洞：它们主要奖励正确答案，但当检索失败时却不惩罚伪造答案，从而隐含加剧了幻觉。为此，我们提出了戒断意识强化学习（AWA-RL），利用模型的查询特定先验能力和持续的政策训练观察，动态塑造戒断奖励。我们还引入了一种新颖的指标RA-F1，用于衡量能力与可靠性的权衡。与非禁食基线相比，AWA-RL的绝对精度提升了最多10.3%，整体RA-F1提升了2.9%，在原始准确性上仅有边际牺牲。这些结果证实了AWA-RL能够成功生成高效且可靠的搜索代理。代码、数据和模型权重均在此 https URL 公开。

Filtering Harmful Actions Isn't Enough: Phantom Transfer in Agentic SDF

过滤有害行为还不够：Agentic SDF中的幻影转移

• Authors: Chinmayi Dixit
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.10750
• Pdf link: https://arxiv.org/pdf/2607.10750
• Abstract Synthetic data is widely used to train large language models because it is inexpensive to generate and easy to control. As models are increasingly deployed as agents, synthetic trajectories are likely to become an important source of training data for agentic behavior. We investigate the effects of training on synthetic agentic trajectories containing adversarial interactions, including actions such as terminating another agents process, lowering its scheduling priority, or accessing resources without authorization. We finetune Llama 3.3 70B Instruct on these trajectories, generated to approximate reinforcement learning rollouts, and evaluate the resulting models on Anthropics Agentic Misalignment suite and Apollos in context scheming scenarios. Finetuning on these trajectories consistently increases misaligned behavior. Leaking rises by roughly a factor of five over the baseline, 4.6% to 24.9%. This increase survives the removal of every adversarial action from the trajectories. Finetuning on structurally comparable trajectories generated benign from the start produce a substantially smaller effect, 15.5%. These results indicate that the misaligned disposition is introduced during the generation process and encoded diffusely throughout the trajectory, rather than being localized to the harmful actions themselves. The effect also depends on the generating model. Benign trajectories produced by Gemini 2.5 Flash induce slightly higher leaking rates than trajectories generated from identical tasks by Claude 3.7 Sonnet. In contrast, broad safety benchmarks degrade similarly across all finetuned models and therefore fail to distinguish these effects. Our results suggest that action level filtering is insufficient to ensure the safety of synthetic agentic training data and that dispositions introduced by the generating model can survive semantic inspection.
• 中文摘要 合成数据被广泛用于训练大型语言模型，因为它生成成本低且易于控制。随着模型越来越多地作为代理被部署，合成轨迹很可能成为代理行为训练数据的重要来源。我们研究训练对包含对抗性交互的合成智能体轨迹的影响，包括终止另一智能体进程、降低其调度优先级或未经授权访问资源等行为。我们对这些轨迹进行Llama 3.3 70B Infusion的微调，这些轨迹用于近似强化学习的推广，并在上下文设计场景下评估人类学智能错位套件和Apollos的模型。对这些路径进行精细调整会持续增加错位行为。泄漏率较基线上升约五倍，从4.6%升至24.9%。这种增长在移除所有对抗作用后依然存在。在结构上可比的路径上进行微调，且从一开始就产生良性，效果显著降低，仅为15.5%。这些结果表明错位倾向是在生成过程中引入的，并在整个轨迹中分散编码，而非局限于有害行为本身。这种效应还取决于生成模型。双子座2.5闪光产生的良性轨迹诱导的泄漏率略高于克洛德3.7号十四行诗相同任务产生的轨迹。相比之下，广泛的安全基准在所有微调模型中降解相似，因此无法区分这些影响。我们的结果表明，动作级过滤不足以确保合成代理训练数据的安全性，生成模型引入的倾向能够通过语义检查。

Q-Learning Lab: Teaching Reinforcement Learning Through Learner-Generated Trace Analysis

Q-Learning Lab：通过学习者生成的痕迹分析进行强化学习教学

• Authors: Ekkachai Jueng
• Subjects: Subjects: Computers and Society (cs.CY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.10802
• Pdf link: https://arxiv.org/pdf/2607.10802
• Abstract Reinforcement learning is usually introduced through the Bellman update, yet the equation often remains abstract to undergraduates: they watch policy arrows converge but rarely observe how each value is computed or why an action is chosen. We present Q-Learning Lab, a single-file, browser-based, bilingual (Thai/English) tool for teaching tabular Q-learning that requires no installation. Beyond the usual gridworld visualization - color-coded Q-values and policy arrows on a $5 \times 5$ world - the tool exposes a live Bellman-substitution panel showing the numeric update at every step, and logs each transition, including the full pre-action Q-row, the greedy-versus-random decision under $\varepsilon$-greedy exploration, and wall-collision events, into an exportable trace. The central contribution is a learn-export-analyze loop: learners run their own agent, export the complete trace as CSV, and analyze it themselves, producing learning curves, value heatmaps, and visitation maps, turning a passive demonstration into a source of learner-generated data for reflective inquiry. We validate the tool without human-subject data through three complementary evaluations: (i) correctness of the learned values and policy against a value-iteration ground truth on the identical MDP; (ii) hyperparameter sweeps over $\alpha$, $\gamma$, and $\varepsilon$ showing that every pedagogical claim the tool makes is reproducible; and (iii) a reward-editing study that uses the ground-truth optimal policy to separate two behaviorally identical but diagnostically opposite failure modes - an exploration failure versus genuine reward misspecification - that a single edited reward can produce. We also compare the tool against existing gridworld visualizers, describe its grounding in learning-by-doing pedagogy, and include a 50-minute lesson plan. The tool and all experiment code are openly available.
• 中文摘要 强化学习通常通过贝尔曼更新引入，但对本科生来说，这个方程往往很抽象：他们观察策略箭头的收敛，却很少观察每个值是如何计算的，或者为何选择某个动作。我们介绍Q-Learning Lab，一个单文件、基于浏览器、双语（泰语/英语）工具，用于教授表格式Q学习，无需安装。除了常规的网格世界可视化——在5美元×5美元世界上的颜色编码Q值和策略箭头——该工具还会实时显示Bellman替换面板，显示每一步的数值更新，并记录每个转换，包括完整的动作前Q行、$\varepsilon$-贪婪探索下的贪婪与随机决策以及墙体碰撞事件，并以可导出的轨迹形式记录。其核心贡献是学习-导出-分析循环：学习者运行自己的代理，导出完整痕迹为CSV，并自行分析，生成学习曲线、价值热图和访问图，将被动演示转化为学习者生成数据的反思性探究来源。我们通过三项互补评估验证该工具，无需人主数据：（i）学习到的价值观和政策是否符合同一MDP上的值重复基础真理;（ii） 超参数扫描了 $\alpha$、$\gamma$ 和 $\varepsilon$，表明该工具提出的每一个教学主张都是可重复的;以及（iii）一项奖励编辑研究，利用实地最优策略区分两种行为相同但诊断上相反的失败模式——探索失败与真正的奖励错误指定——这些模式均可能因单一编辑奖励而产生的。我们还将该工具与现有的网格世界可视化工具进行比较，介绍其基于“实践学习”教学法的基础，并包含一个50分钟的教学计划。工具和所有实验代码都是公开公开的。

Learning Roller-Skating Motions of Humanoid Robots Based on Adversarial Motion Priors

基于对抗运动先验学习类人机器人的轮滑动作

• Authors: Yunkang Cheng, Yutong Wu, Menghan Li, Shihe Zhou, Mingguo Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.10815
• Pdf link: https://arxiv.org/pdf/2607.10815
• Abstract Humanoid roller-skating is difficult because the robot must coordinate whole-body balance, rolling contacts, and velocity-dependent posture regulation. This paper presents an adversarial motion prior based reinforcement learning framework for two humanoid roller-skating gaits: Pump Glide skating and Push Glide skating. The two gait datasets are collected independently through motion capture and retargeted to the humanoid robot separately. The retargeted data are then smoothed and resampled into reference motion states for AMP training. The two gaits are learned by independent AMP training pipelines with separate reference datasets, separate policies, and independent reward architectures. Simulation experiments are designed to evaluate gait quality, velocity tracking, turning, and gait-specific reward ablations.
• 中文摘要 类人轮滑很难，因为机器人必须协调全身平衡、滚动接触和速度依赖的姿势调节。本文提出了一种针对两种类人轮滑步态：泵滑和推滑滑行的对抗性动作先验强化学习框架。这两组步态数据集通过动作捕捉独立收集，并分别重新定位到类人机器人。重新定向的数据随后被平滑并重新采样为AMP训练的参考运动状态。这两种步态通过独立的AMP训练流程学习，这些流程拥有独立的参考数据集、不同的策略和独立的奖励架构。模拟实验旨在评估步态质量、速度追踪、转弯以及步态特异性奖励消融。

Automated Stealthy Wear-Out Attack on Digital Twins With Deep Reinforcement Learning

深度强化学习对数字孪生的自动隐蔽磨损攻击

• Authors: Joshua Haworth, Aryan Pasikhani, George Pavlides, Prosanta Gope, John Clark
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2607.10830
• Pdf link: https://arxiv.org/pdf/2607.10830
• Abstract Digital Twins (DTs) have emerged as pivotal enablers of Industry 4.0, offering transformative capabilities such as real-time monitoring, advanced simulation, and precise control of physical assets. By bridging the physical and virtual domains, DTs facilitate seamless integration of data-driven decision-making and operational optimisation. However, this seamless interaction significantly expands the attack surface of industrial systems, creating vulnerabilities that adversaries can exploit. This paper introduces a novel and stealthy wear-out attack leveraging Deep Reinforcement Learning (DRL) to target DT-enabled infrastructures. The adversary strategically and covertly manipulates control signals, inducing increased torque on a specific joint to accelerate wear and tear while evading detection by a state-of-the-art anomaly detection system. Extensive benchmarking of reinforcement learning algorithms - including Twin Delayed Deep Deterministic Policy Gradient (TD3), Soft Actor-Critic (SAC), Proximal Policy Optimisation (PPO), and Advantage Actor-Critic (A2C) - revealed that SAC consistently outperformed its counterparts in terms of sample efficiency, stability, and overall attack effectiveness. We evaluate the proposed adversary in an industrial setting using the UR10e robotic arm. Results demonstrate the adversary's ability to significantly elevate torque levels on the targeted joint, leading to accelerated degradation and increased maintenance costs, all while operating stealthily and avoiding detection. Our findings highlight the substantial risks posed by DRL-driven adversaries to DT-enabled environments and emphasise the critical need for robust defence mechanisms to protect critical industrial systems.
• 中文摘要 数字孪生（DT）已成为工业4.0的关键推动力，提供了实时监控、高级仿真和物理资产精准控制等变革性能力。通过连接物理与虚拟领域，DT促进了数据驱动决策与运营优化的无缝整合。然而，这种无缝互动显著扩大了工业系统的攻击面，制造了攻击者可利用的漏洞。本文介绍了一种新颖且隐蔽的磨损攻击，利用深度强化学习（DRL）针对支持DT的基础设施。对手通过战略性和秘密手段操控控制信号，在特定关节上产生更大扭矩以加速磨损，同时躲避最先进的异常检测系统的发现。对强化学习算法的广泛基准测试——包括双延迟深度确定性策略梯度（TD3）、软演员-批判者（SAC）、近端策略优化（PPO）和优势演员-批判者（A2C）——显示SAC在样本效率、稳定性和整体攻击效能方面始终优于其他算法。我们在工业环境中使用UR10e机械臂评估拟议的对手。结果显示，对手能够显著提高目标关节的扭矩水平，加速退化和增加维护成本，同时保持隐蔽且避免被发现。我们的发现凸显了由DRL驱动的对手对DT环境带来的重大风险，并强调了建立强有力防御机制以保护关键工业系统的关键必要性。

Predictive Divergence Masks for LLM RL

LLM RL 的预测性发散掩膜

• Authors: Xiangxin Zhou, Jiarui Yao, Penghui Qi, Bowen Ping, Jiaqi Tang, Haonan Wang, Tianyu Pang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.10848
• Pdf link: https://arxiv.org/pdf/2607.10848
• Abstract Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away. Recent work DPPO improves the proximity criterion by replacing PPO's ratio-based test with a probability divergence between the behavior and training policies. However, its direction criterion is still inherited from PPO. A token can be masked only when the sampled-token importance ratio moves away from one. We observe that this ratio-based direction criterion is a single-sample proxy that can disagree in sign with the change of the divergence that defines the proximity criterion. We therefore propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region. For the discrete softmax policies used in LLM RL, we derive this prediction in closed form. Because production rollout engines expose only a truncated (top-K) view of the vocabulary, we develop two lightweight top-$K$ estimators for this prediction. Detailed analysis shows the divergence-based direction is better aligned with the realized change of the divergence than the sampled ratio, and the resulting masks improve RL training across model scales and precision settings.
• 中文摘要 大型语言模型（LLM）的强化学习通常依赖信任区域掩码来稳定非策略更新。主流的PPO式方法使用抽样令牌重要性比来衡量两个标准：邻近性标准，即策略是否偏离行为策略过远;方向性标准，判断更新是否使其远离。近期工作DPPO通过用行为策略与训练策略之间的概率偏差替代基于比例的PPO检验，改进了接近准则。然而，其方向准则仍沿用自PPO。只有当抽样的令牌重要性比从一转移时，一个令牌才能被掩盖。我们观察到，基于比率的方向准则是一个单样本代理，其符号可能与定义邻近准则的散度变化不一致。因此，我们提出了预测性背离掩膜，即询问下一步政策梯度是否会增加还是减少信任区域所使用的相同背离。对于LLM RL中使用的离散软极限策略，我们以封闭形式推导出该预测。由于生产推广引擎只暴露词汇的截断（top-K）视图，我们为该预测开发了两个轻量级的顶级$K美元估计器。详细分析表明，基于散度的方向比采样比更能与实际的散度变化相匹配，所得掩码提升了跨模型尺度和精度设置的强化学习训练。

SETA: Scaling Environments for Terminal Agents

SETA：终端代理的扩展环境

• Authors: Qijia Shen, Zhiqi Huang, Vamsidhar Kamanuru, Aznaur Aliev, Jay Rainton, Ahmed Awelkair, Zhichen Zeng, Jiajun Li, Shi Dong, Yueming Yuan, Boyuan Ma, Qizheng Zhang, Jiwei Fu, Yuzhen Mao, Wendong Fan, Ping Nie, Philip Torr, Bernard Ghanem, Changran Hu, Jonathan Lingjie Li, Urmish Thakker, Guohao Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.10891
• Pdf link: https://arxiv.org/pdf/2607.10891
• Abstract Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interfaces (GUIs). Among these, the terminal command line provides a text-based, general-purpose interface, covering tasks from system operations to data science and machine learning. However, scaling terminal-agent training remains challenging, as it requires diverse and coherent task instructions, executable environments, and reliable verification, while lacking naturally grounded supervision data. In this work, we propose SETA, a scalable framework for generating verifiable terminal environments for reinforcement learning (RL). The framework consists of two pipelines sharing a unified verification mechanism: SETA-Synth converts diverse sources into standardized RL environments, and SETA-Evol further expands from existing environments with adaptive control of difficulty and diversity. Together, we construct and release SETA-Env, the largest open-source verifiable terminal RL dataset to date, containing over 4,500 environments. We evaluate our dataset by training Qwen3-8B with GRPO on SETA-Env, achieving 12% pass rate on Terminal-Bench 2.0, the best reported result for an RL-trained model at the 8B scale. We further observe gains on DeepSeek-V4-Flash under the same terminal agent harness, with pass@1 on Terminal-Bench 2.0 improving from 40% to 43% and pass@5 improving from 54% to 58%. These results demonstrate that SETA- Env provides high-quality training environments for terminal agents and serves as a valuable resource for advancing research on terminal-based agent learning.
• 中文摘要 大型语言模型（LLM）正迅速转向通过多样化界面解决任务的代理，包括网页和图形用户界面（GUI）。其中，终端命令行提供了一个基于文本的通用接口，涵盖从系统运行到数据科学和机器学习等任务。然而，终端代理训练的规模化依然具有挑战性，因为它需要多样且连贯的任务指令、可执行环境和可靠的验证，同时缺乏自然的监督数据。在本研究中，我们提出了SETA，一种可扩展的框架，用于生成可验证的强化学习（RL）终端环境。该框架由两条共享统一验证机制的流水线组成：SETA-Synth将多样化源转换为标准化强化学习环境，SETA-Evol则基于现有环境扩展，具备难度和多样性的自适应控制。我们共同构建并发布了SETA-Env，这是迄今为止最大的开源、可验证的终端强化学习数据集，包含4500多个环境。我们通过在SETA-Env上用GRPO训练Qwen3-8B来评估数据集，在Terminal-Bench 2.0上实现了12%的通过率，是8B尺度上强化学习模型中报告的最佳成绩。我们进一步观察到，在同一终端代理工具框架下，DeepSeek-V4-Flash的提升，终端-Bench 2.0的 pass@1从 40% 提升到 43%，pass@5 从 54% 提升到 58%。这些结果表明，SETA-环境为终端代理提供了高质量的培训环境，并成为推动基于终端代理学习研究的重要资源。

A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

一个单扩散策略控制器，用于多任务块推送，实现零时模拟到真实传输

• Authors: Haitong Ma, Haldun Balim, Yang Hu, Bo Dai, Na Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.10892
• Pdf link: https://arxiv.org/pdf/2607.10892
• Abstract Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparse-reward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our real-world block-pushing setup under the tested variations.
• 中文摘要 扩散政策在使用行为克隆（BC）为机器人表示和学习复杂动作方面表现出有希望的实证表现。本文探讨了从零开始利用强化学习（RL）进行多任务机器人操作的扩散训练策略。具体来说，我们旨在为具有多形状的块推任务训练单一扩散策略。所提出的框架具有一个简单的策略损失函数，这是一个用于基于BC的扩散策略训练中的重加权证据下界，并且可以无缝地作为强化学习算法中的策略学习模块。为应对缺乏示范带来的探索挑战，我们采用了逆向课程生成和以目标为中心的表现方式。结合扩散策略的表达性，我们的设计支持在稀疏奖励模拟环境中学习多任务区块推送策略。我们还进一步评估了训练有素的扩散策略是否能在零射中转移到不同环境条件下的实际任务，包括目标位置、方块形状、方块重量和表面摩擦，这证明该流程可以在测试变体下转移到我们的实际方块推送设置中。

Reinforcement Learning versus Optimization for Optimal Transmission Switching: A Comparative Study

强化学习与优化在最优传输切换中的比较研究

• Authors: Yuanrui Sang, Israel Abiala, Rachel Gerdes
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.10948
• Pdf link: https://arxiv.org/pdf/2607.10948
• Abstract Optimal Transmission Switching (OTS) reduces generation cost by strategically opening transmission lines, but its mixed-integer linear program (MILP) formulation scales poorly for large-scale transmission networks. Reinforcement learning (RL) offers a computationally efficient alternative, but existing RL-based OTS approaches rely on soft penalties that permit physical constraint violations. This paper presents a comparison between an RL framework and an MILP-based optimization method for OTS. Case studies were carried out on the IEEE RTS-96 24-bus system; results show that the agent was able to produce near-optimal solutions at low switching budgets and tended to yield suboptimal solutions at high switching budgets. However, the RL agent was able to generate feasible solutions two-to-three orders of magnitude faster than the optimization solver.
• 中文摘要 最优传输切换（OTS）通过战略性开通输电线路降低发电成本，但其混合整数线性规划（MILP）表述对于大规模输电网络的扩展性较差。强化学习（RL）提供了一种计算效率高的替代方案，但现有基于强化学习的OTS方法依赖软惩罚，允许物理约束违规。本文比较了强化学习框架与基于MILP的OTS优化方法。案例研究针对IEEE RTS-96 24-总线系统;结果显示，该智能体在低切换预算下能够产生近似最优解，但在高切换预算下往往产生次优解。然而，强化学习代理能够比优化求解器快两到三个数量级地生成可行解。

Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies

关键时刻思考：基于强化学习策略的条件VLM推理社会导航

• Authors: Ali Ahmadi, Hamed Rahimi, Adrien Jacquet Cretides, Marie Samson, Mahdi Khoramshahi, Mohamed Chetouani
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.10991
• Pdf link: https://arxiv.org/pdf/2607.10991
• Abstract As mobile robots become more integrated into everyday human environments, social robot navigation is becoming essential for ensuring human comfort, safety, and trust. While reinforcement learning (RL) navigation policies provide the fast inference and reactive behavior necessary for real-time deployment, they still lack flexible semantic reasoning capabilities and often fail to generalize to complex social scenarios. Recent approaches have increasingly turned to vision-language models (VLMs) in place of RL policies to improve semantic and social reasoning in robot navigation. Nevertheless, their high computational cost and slow inference remain major barriers to real-time deployment. To overcome these limitations, we introduce HUMA (Hybrid Understanding for Multi-modal social Navigation), a hybrid architecture that dynamically balances the computational efficiency of RL policies with the deep semantic understanding of VLMs. Our approach uses a reactive RL policy to handle low-density, routine navigation tasks, while conditioning it on a post-trained high-level VLM when a human enters sensitive situations, such as the robot's proximity zone. We evaluate HUMA on the Social-MP3D and Social-HM3D benchmarks, where it achieves task success improvements of 20% and 3%, respectively, while significantly reducing personal space violations and human collisions against state-of-the-art baselines. Extensive ablation studies validate each architectural component, and real-world deployment on the Mirokaï mobile robot further demonstrates the practical viability of our approach.
• 中文摘要 随着移动机器人越来越融入日常人类环境，社交机器人导航变得至关重要，以确保人类的舒适、安全和信任。虽然强化学习（RL）导航策略提供了快速的推理和反应性行为以实现实时部署，但它们仍缺乏灵活的语义推理能力，且常常无法推广到复杂的社会场景。近年来，越来越多地采用视觉语言模型（VLMs）取代强化学习政策，以提升机器人导航中的语义和社会推理能力。然而，其高计算成本和推理缓慢仍是实时部署的主要障碍。为克服这些局限，我们引入了HUMA（多模态社交导航混合理解），这是一种动态平衡强化学习策略计算效率与VLM深度语义理解的混合架构。我们的方法采用反应式强化学习策略来处理低密度、常规的导航任务，同时在人类进入敏感环境（如机器人接近区）时，将其条件化为经过后训练的高级别VLM。我们在Social-MP3D和Social-HM3D基准测试中评估HUMA，任务成功率分别提升了20%和3%，同时显著减少了个人空间违规和人与碰撞的发生，相较于最先进的基准。广泛的消融研究验证了每个架构组件，Mirokaï移动机器人的实际部署进一步证明了我们方法的实用可行性。

STAMP: Provenance-Guided Credit Assignment for Deep Search Agents

STAMP：深度查询代理的来源指导信用分配

• Authors: Ke Xu, Han Xu, Xinran Chen, Yuqian Wang, Zhixuan Li, Xiaojian Liu, Changwo Wu, Jianqiang Xia, Yuchen Li
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.11172
• Pdf link: https://arxiv.org/pdf/2607.11172
• Abstract Reinforcement learning for deep-search agents has largely focused on trajectory-level scoring -- outcome correctness, citation-aware rewards, and evidence coverage. Yet the actions that expose supporting documents receive no targeted credit, a gap we call the reward-credit mismatch. We propose STAMP, in which a reference-based verifier judges whether each cited document supports an entity or relation in a training-time evidence graph, and first-exposure attribution traces each supported citation back to the action that first surfaced it. This step credit is injected through sign-preserving advantage modulation, which redistributes advantage across steps without changing the trajectory-level reward or the relative ranking of trajectories within each group. On BrowseComp, BrowseComp-ZH, and xbench-DS, STAMP improves the GRPO baseline by +2.0/+5.5/+3.0 points under matched SFT initialization, training data, and search tools, and composes with both outcome-only and citation-rubric base rewards. Component ablations confirm that the provenance-based credit signal and the sign-preserving advantage modulation each contribute to the gains.
• 中文摘要 深度搜索代理的强化学习主要侧重于轨迹层级评分——结果正确性、引用感知奖励和证据覆盖。然而，揭露支持文件的行为却没有获得有针对性的认可，这种差距我们称之为“奖励-信用错配”。我们提出STAMP方法，即基于引用的验证者判断每个被引用文件是否支持训练时间证据图中的实体或关系，首次曝光归因将每个支持引用追溯到最早出现的行动。这一阶数信用通过符号保持优势调制注入，该调制在不改变各组内轨迹级奖励或相对排名的情况下，将优势重新分配到各步之间。在BrowseComp、BrowseComp-ZH和xbench-DS上，STAMP在匹配SFT初始化、训练数据和搜索工具下，将GRPO基线提升+2.0/+5.5/+3.0点，并结合仅结果和引用评分标准的基础奖励。成分消融确认，基于来源的信用信号和符号保持优势调制均对收益有所贡献。

SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL

SCALECUA：通过可验证任务综合和高效在线强化学习扩展计算机使用代理

• Authors: Bowen Lv, Xiao Liu, Yanyu Ren, Hanyu Lai, Bohao Jing, Hanchen Zhang, Yanxiao Zhao, Shuntian Yao, Jie Tang, Yuxiao Dong
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.11185
• Pdf link: https://arxiv.org/pdf/2607.11185
• Abstract Computer use agents (CUAs) are emerging as a powerful interface for automating complex digital workflows through visual perception and GUI execution. Online reinforcement learning with verifiable rewards (RLVR) has emerged as a key direction for scaling their capabilities. However, this paradigm is bottlenecked by verifiable data scarcity and online RL inefficiency. To break these barriers, we introduce ScaleCUA, a unified framework that scales online RL for CUAs via verifiable task synthesis and efficient training. At the data level, we design VeriGen, an end-to-end framework for generating verifiable RL tasks through iterative docker interactions and a multi-agent feedback loop. Scaled to 100+ concurrent agent workers via a shared docker interaction probe, this pipeline produces 24K+ verifiable tasks and nearly 3K high-quality RL tasks. To maximize sample efficiency, we propose Frontier Sampling, which tracks per-task capability and allocates rollouts to the current learning frontier. On the training side, we further design Visual Context Segmentation, a sliding window over recent visual context that balances rollout and training-engine pressure, yielding a 2.83x training speedup over step-wise decomposition. Together, ScaleCUA achieves 68.7% on OSWorld and 54.0% on ScienceBoard, establishing new state-of-the-art performance among open-source computer use agents. Code, models, and datasets are available at this https URL.
• 中文摘要 计算机使用代理（CUA）正作为一种强大的界面，通过视觉感知和图形界面执行自动化复杂的数字工作流程。带有可验证奖励的在线强化学习（RLVR）已成为其能力扩展的关键方向。然而，这一范式被可验证的数据稀缺性和在线强化学习低效所限制。为了打破这些障碍，我们引入了ScaleCUA，一个统一框架，通过可验证的任务综合和高效训练，扩展CUA的在线强化学习。在数据层面，我们设计了VeriGen，这是一个端到端框架，通过迭代的docker交互和多代理反馈循环生成可验证的强化学习任务。通过共享的docker交互探针，该流水线可扩展至100+个并发代理工作者，产生24K+可验证任务和近3K高质量强化学习任务。为了最大化样本效率，我们提出了前沿采样，它跟踪每个任务的能力，并将推广分配到当前的学习前沿。在训练方面，我们进一步设计了视觉上下文分割，这是一个滑动窗口，覆盖近期视觉上下文，平衡滚动和训练引擎压力，使训练速度比逐步分解快达2.83倍。ScaleCUA在OSWorld上达到了68.7%，在ScienceBoard上达到了54.0%，在开源计算机使用代理中树立了新的最先进性能。代码、模型和数据集均可在此 https URL 访问。

Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability

基于偏好的强化学习推广：一个不可比性的理性模型

• Authors: Simone Drago, Marco Mussi, Leonardo Bianconi, Alberto Maria Metelli
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.11432
• Pdf link: https://arxiv.org/pdf/2607.11432
• Abstract In this work, we study the reinforcement learning (RL) problem from pairwise trajectory comparisons provided by a human expert. We generalize preference-based RL by formalizing a novel setting in which the expert can also label trajectory pairs as incomparable, i.e., when neither trajectory dominates the other. We introduce the learning problem and the desiderata that its solution should satisfy. Then, we propose a novel Bradley-Terry-inspired rationality model that effectively captures incomparabilities and infers a multi-dimensional reward function, and we study its properties. We provide a sample complexity analysis for learning the model parameters when a dataset is available. Finally, we evaluate our model's ability to reconstruct a reward function that aligns with the expert's comparisons in simulated environments and to recover the Pareto frontier of policies, along with a robustness analysis across varying levels of expert rationality.
• 中文摘要 本研究通过人类专家提供的两两轨迹比较，研究强化学习（RL）问题。我们通过形式化一个新颖的环境来推广基于偏好的强化学习，在该环境中专家也可以将轨迹对标记为不可比，即当两条轨迹都不支配另一条时。我们介绍学习问题及其解决方案应满足的期望。随后，我们提出了一种受Bradley-Terry启发的新理性模型，有效捕捉不可比性并推断出多维奖励函数，并研究其性质。当数据集可用时，我们提供了样本复杂度分析，用于学习模型参数。最后，我们评估模型在模拟环境中重建与专家比较相符的奖励函数的能力，以及恢复政策帕累托前沿的能力，同时对不同专家理性水平进行了鲁棒性分析。

SCOPE-RL: Optimizing Reasoning Paths Before and After Success

SCOPE-RL：成功前后优化推理路径

• Authors: Xiaojian Liu, Han Xu, Jianqiang Xia, Zhixuan Li, Ke Xu, Yiwei Dai, Xinran Chen, Changwo Wu, Yuchen Li
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.11506
• Pdf link: https://arxiv.org/pdf/2607.11506
• Abstract Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at this https URL.
• 中文摘要 带有可验证奖励的强化学习（RLVR）利用稀疏且可验证的最终答案奖励来优化大型语言模型。这种稀疏锚点可靠地验证轨迹是否成功，但不直接反馈产生轨迹的推理路径。在成功之前，难题的先决进展不会收到奖励信号;成功后，结果奖励无法区分组织良好的正确轨迹与冗余或局部缺陷的轨迹。我们引入了SCOPE-RL（带流程效率的支架链式优化），这是一个两阶段框架，在保持GRPO更新的同时，将锚点密实化：自适应支架式RL在成功前为隐藏的答案隐藏子问题链添加前缀分解的可验证奖励，质量感知过程RL则在成功后应用正确门控过程形状奖励以优化正确的轨迹。经过专家验证的步进质量评估协议不仅仅评估最终答案准确性，还能评估有用的步密度、误差定位和令牌效率。在QWEN3-8B-Ininstruction中，基于DAPO-Math和Big-Math训练，SCOPE-RL平均准确率提升了最多11.2pp，推理标记比仅结果GRPO减少了27.1%;在GSPO和Qwen3-0.6B-Ininstruction下，这些收益均保持不变，表明奖励-信号密集化与政策更新级别的RLVR进展是互补的。代码和数据可在此 https URL 获取。

Learning Residual Kinematic Corrections for Continuous Neural Decoding via Reinforcement Learning

通过强化学习学习持续神经解码的残余运动学修正

• Authors: Jiamian Li, Niall McShane, Attila Korik, Naomi du Bois, Karl McCreadie, Leen Jabban, Benjamin Metcalfe, Özgür Şimşek, Damien Coyle
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.11530
• Pdf link: https://arxiv.org/pdf/2607.11530
• Abstract Decoding continuous three-dimensional (3D) motor imagery (MI) using non-invasive electroencephalography (EEG)-based brain--computer interfaces (BCIs) remains challenging due to signal variability and residual decoding errors. Deep learning architectures such as convolutional neural network--long short-term memory (CNN--LSTM) models can capture spatial and temporal dynamics for continuous kinematic decoding; however, systematic residual errors persist in predicted trajectories. We propose a two-stage decoding framework that applies reinforcement learning (RL) to perform residual kinematic correction on the outputs of a CNN--LSTM decoder (CNN--LSTM--RL). The RL agent is trained offline without direct EEG input and instead operates on predicted kinematic trajectories to optimize movement accuracy relative to target trajectories. Decoding performance was quantified using Pearson correlation coefficients ($r$) and Root Mean Square Errors (RMSE) along the $x, y$, and $z$ axes. Compared to CNN--LSTM applied alone, CNN--LSTM--RL improved the mean correlation from $0.5076$ to $0.7181$ ($p = 0.0005$) in 2D and from $0.6420$ to $0.7780$ ($p = 0.0059$) in VR, with relative gains of $41.5\%$ and $21.2\%$, respectively. Correspondingly, RMSE was reduced from $0.0890$ to $0.0532$ (2D, $p < 0.0001$) and from $0.0714$ to $0.0441$ (VR, $p < 0.0001$), representing relative reductions of $40.2\%$ and $38.2\%$. These findings demonstrate that this scalable framework enhances 3D BCI MI decoding by correcting kinematic errors via offline residual RL without extra neural data, advancing neurorehabilitation, prosthetics, and virtual interaction.
• 中文摘要 利用非侵入性脑电图（EEG）为基础的脑-计算机接口（BCI）解码连续三维（3D）运动影像（MI）由于信号变异性和残留解码误差仍具挑战性。深度学习架构如卷积神经网络——长短期记忆（CNN——LSTM）模型能够捕捉空间和时间动力学，实现连续运动学解码;然而，预测轨迹中仍存在系统性残差误差。我们提出了一个两阶段解码框架，应用强化学习（RL）对CNN--LSTM译码器（CNN--LSTM--RL）输出进行残余运动学校正。强化学习者在离线训练时不直接输入脑电图，而是根据预测的运动学轨迹操作，以优化相对于目标轨迹的运动精度。解码性能通过沿$x、y$和$z$轴的皮尔逊相关系数（$r$）和均方根误差（RMSE）进行量化。与单独应用 CNN--LSTM 相比，CNN--LSTM-RL 在二维中将平均相关系从$0.5076$提升到$0.7181$（$p = 0.0005$），在虚拟现实中从$0.6420$提升到$0.7780$（$p = 0.0059$），相对相关系数分别提升为$41.5%$和$21.2\%$。相应地，RMSE从0.0890美元降至0.0532美元（2便士，$p <0.0001美元），从0.0714美元降至0.0441美元（VR，$p <0.0001美元），相对减少了40.2美元和38.2美元。这些发现表明，该可扩展框架通过离线残留强化学习纠正运动学错误，无需额外神经数据，从而提升了3D BCI MI解码，推动了神经康复、假肢和虚拟互动的发展。

Heuristic Learning for Active Flow Control Using Coding Agents

利用编码代理进行主动流量控制的启发式学习

• Authors: Paul Garnier, Jonathan Viquerat, Elie Hachem
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Fluid Dynamics (physics.flu-dyn)
• Arxiv link: https://arxiv.org/abs/2607.11565
• Pdf link: https://arxiv.org/pdf/2607.11565
• Abstract Active flow control involves nonlinear dynamics, partial observations, and computationally expensive simulations, making controller design particularly challenging. Deep reinforcement learning (DRL) has emerged as a powerful framework for such problems, but its success typically relies on large numbers of simulator interactions and produces neural-network policies whose decision process often remains difficult to interpret. In this work, we investigate a different paradigm: instead of optimizing neural-network parameters, we use modern coding agents to search directly for explicit executable feedback laws. We introduce a constrained heuristic-learning protocol in which an agent iteratively proposes, evaluates, and revises controller implementations while interacting exclusively through the public benchmark interface. The proposed framework is evaluated on 13 active flow-control benchmarks spanning one, two, and three-dimensional problems and compared against the strongest available DRL baselines under identical simulation budgets. The discovered heuristic controllers match or outperform the best DRL policy in 10 of the 13 environments while remaining compact, interpretable, and directly inspectable. Beyond aggregate performance, the resulting controllers reveal physically meaningful feedback mechanisms, transfer successfully across more challenging configurations, and remain competitive under varying Reynolds and Rayleigh numbers, actuator counts, and observation sparsity. These results suggest that heuristic learning through coding agents constitutes a credible and complementary alternative to conventional reinforcement learning, combining competitive performance with physically interpretable controller representations. Prompts and source code are available at this https URL.
• 中文摘要 主动流量控制涉及非线性动力学、部分观测和计算量高的仿真，使得控制器设计尤为具有挑战性。深度强化学习（DRL）已成为解决此类问题的强大框架，但其成功通常依赖大量模拟器交互，并产生决策过程难以理解的神经网络策略。本研究探讨了另一种范式：我们不优化神经网络参数，而是使用现代编码代理直接搜索显式可执行反馈定律。我们引入了一种受限启发式学习协议，代理通过公开基准接口迭代提出、评估和修订控制器实现。该框架基于13个涵盖一维、二维和三维问题的主动流量控制基准进行评估，并与在相同模拟预算下最强的DRL基线进行比较。发现的启发式控制器在13个环境中有10个环境表现优于或超过最佳DRL策略，同时保持紧凑、可解释且可直接检查。除了整体性能外，最终产生的控制器还展现了具有物理意义的反馈机制，能够成功跨越更具挑战性的配置，并在不同的雷诺数、执行器数量和观测稀疏度下保持竞争力。这些结果表明，通过编码代理的启发式学习是一种可信且互补的传统强化学习替代方案，将竞争性能与物理可解释的控制表征相结合。提示和源代码可在此 https URL 访问。

DiffEEG: A Self-Supervised Denoising Diffusion Model for Learning EEG Generic Representations

DiffEEG：一种用于学习脑电图泛型表示的自监督去噪扩散模型

• Authors: Abdulkader Helwan, Lina Abou-Abbas, Hussein El Amouri, Belkacem Chikhaoui, Khadidja Henni
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Signal Processing (eess.SP)
• Arxiv link: https://arxiv.org/abs/2607.11578
• Pdf link: https://arxiv.org/pdf/2607.11578
• Abstract Deep learning for EEG-based seizure detection faces critical challenges: severe annotation scarcity and extreme class imbalance, where ictal events comprise less than 10\% of clinical recordings. We present DiffEEG, a 9.6M-parameter self-supervised foundation model that addresses both limitations through denoising diffusion pre-training and reinforcement learning (RL)-based fine-tuning. Pre-trained on 1.3M unlabeled segments from the Temple University Hospital Seizure Corpus (TUHSZ), DiffEEG learns generic neural representations via a 1D U-Net with multi-head self-attention. For downstream adaptation, a reinforced decision layer employs policy gradient optimization to directly maximize F1-score, prioritizing sensitivity to rare seizure events over overall accuracy. Under strict patient-wise evaluation (279 patients, Leave-One-Fold-Out), DiffEEG achieves 61\% accuracy and 59\% F1 for 4-class seizure subtyping, and 81\% accuracy with 85\% weighted F1 for binary detection, maintaining clinically viable seizure recall (59\%) despite extreme imbalance (6.7\% prevalence). Segment-level evaluation establishes an upper bound of 97.6\% accuracy, confirming strong architectural capacity. DiffEEG demonstrates that diffusion-based pre-training combined with metric-aware reinforcement learning enables clinically deployable seizure monitoring with minimal labeled data requirements.
• 中文摘要 基于脑电图的深度学习癫痫发作检测面临关键挑战：严重的注释稀缺和极端的类别失衡，其中发作事件仅占临床记录的不到10%。我们介绍了DiffEEG，一个参数为960万的自监督基础模型，通过去噪扩散预训练和基于强化学习（RL）的微调，解决了这两个局限性。DiffEEG通过来自天普大学医院癫痫语料库（TUHSZ）的130万未标记片段预训练，通过带有多头自我注意力的1D U-Net学习通用神经表征。在下游适应方面，强化决策层采用策略梯度优化直接最大化F1分数，优先考虑对罕见发作事件的敏感性而非整体准确性。在严格的患者分类评估（279名患者，保留一折叠）下，DiffEEG在四类癫痫发作亚型中实现61%准确率和59%的F1准确率，在85%加权F1中二元检测准确率为81%，尽管极度失衡（6.7%的流行率）仍保持临床可行的癫痫回忆（59%）。分段级评估确立了97.6%的准确性上限，确认了强大的架构能力。DiffEEG表明，基于扩散的预训练结合计量感知强化学习，使得临床可部署的癫痫发作监测实现，且对标记数据需求极低。

Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO

作为自身批评者：通过CycleGRPO统一区域理解与本地化

• Authors: Xin Zhang, Haochen Wang, Yikang Zhou, Jason Li, Robby T. Tan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.11581
• Pdf link: https://arxiv.org/pdf/2607.11581
• Abstract This paper introduces Actor as Its Own Critic, a unified reinforcement learning framework, Cycle Group Relative Policy Optimization (CycleGRPO), that jointly optimizes region understanding and localization for Multimodal Large Language Models (MLLMs). Unlike existing separate pipelines, we leverage the inherent duality between the two tasks to construct a self-evaluating reinforcement learning paradigm: "region $\to$ text $\to$ region''. Specifically, a single MLLM first acts as the actor to generate region captions, then immediately transitions to a critic to ground its generated text back in the spatial domain. Therefore, CycleGRPO requires only region inputs, e.g., masks or bounding boxes, entirely bypassing the need for textual ground truths. A quality-aware token-level cycle-consistency reward is employed to assess the semantic discriminability of text captions via their physical localization accuracy. Empirically, built upon SAMTok, our CycleGRPO framework successfully bootstraps both capabilities simultaneously. Without any task-specific fine-tuning, the framework yields consistent performance gains across a wide range of benchmarks, including region captioning, region VQA, grounded dialogue, and referring segmentation. Overall, CycleGRPO offers a straightforward and scalable way to advance pixel-level capabilities in MLLMs. Code and models are released at this https URL.
• 中文摘要 本文介绍了Actor as Its Critic，一个统一的强化学习框架——循环组相对策略优化（Cycle Group Relative Policy Optimization，CycleGRPO），该框架共同优化多模态大型语言模型（MLLM）的区域理解与本地化。与现有的独立流水线不同，我们利用这两种任务之间的固有二元性构建了一个自我评估的强化学习范式：“区域 $\to$ 文本 $\to$ 区域”。具体来说，单个MLLM首先作为actor生成区域字幕，然后立即切换为critic，将其生成的文本重新扎根于空间域。因此，CycleGRPO只需区域输入，例如掩码或边界框，完全绕过文本的地面真理。采用质量感知的代币级周期一致性奖励，通过文本的物理定位准确性评估文本字幕的语义区分性。基于SAMTok，我们的CycleGRPO框架成功地同时启动了这两种能力。无需针对特定任务的微调，该框架在包括区域字幕、区域视频质量保证、基于地面对话和转介分段等多种基准测试中持续提升性能。总体而言，CycleGRPO 提供了一种直接且可扩展的方式，用于提升多层次多层次（MLLM）中的像素级能力。代码和模型发布于此 https 网址。

Auditing the Risk Claims of Distributional Reinforcement Learning

审计分布式强化学习的风险声明

• Authors: Hari Prasad
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2607.11607
• Pdf link: https://arxiv.org/pdf/2607.11607
• Abstract Distributional reinforcement learning agents learn full return distributions that are increasingly read at face value: for interpretability, risk-sensitive control, and safety monitoring. We ask a question theory anticipates but that has not been measured directly: are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric (the excess Wasserstein gap between the top two actions, which equals the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit itself manufactures false conclusions. Across QR-DQN, C51, and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, the placement of the strongest claims is statistically indistinguishable from truth-blind, and essentially no claim is confirmable: for these agents, the learned "risk" reflects a training artifact rather than environment stochasticity. The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96-100% of real claims (correlation 0.89-0.92): the reading measures the agents, not the audit. Acting on the heads' CVaR advice at their most-flagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but wrong audits of our own.
• 中文摘要 分布强化学习代理学习越来越多地被按字面解读的完整返回分布：用于可解释性、风险敏感控制和安全监控。我们提出一个理论预期但尚未被直接测量的问题：训练有素的分布代理所说的风险主张是否真实？我们的审计结合了决策相关筛选指标（顶端两行为之间的过剩Wasserstein差距，即一阶随机优势被破坏的质量）、快照重启蒙特卡洛的真实数据，以及统计工具（置换空、自助推翻、FDR控制），否则审计本身会产生错误结论。在MinAtar上的QR-DQN、C51和IQN（33次运行）中，40%-95%的最强风险权衡在95%置信度下被驳斥，最强主张的位置在统计上与真理盲无异，基本上没有任何主张是可证实的：对于这些代理来说，学习到的“风险”反映的是训练产物，而非环境随机。该伪物是结构性的（训练初期完全成型，与最终得分无关，且与每个种子有独特性），在全Atari比例下看起来无异，所有关于预先训练的近乎最先进QR-DQN的顶级突破宣称都被驳斥。已知大小的正对照证实了96%-100%的真实声明（相关系数0.89-0.92）：读数衡量的是代理人，而非审计。在最受标记状态下，根据主管的CVaR建议采取行动，概率从有利到明显比偶然差不等。无论是风险培训还是集成，都无法消除伪造，重新校准只能通过审计，使主张失效：磁头无信息，而非仅仅校准错误。我们发布工具包，记录了两个无声的陷阱，这些陷阱导致了我们自己令人信服但错误的审计结果。

SKooP: Symmetric Koopman Predictions for Faster and More Generalizable Legged Robot Locomotion with Reinforcement Learning

SKooP：基于强化学习实现更快且更具通用性的腿部机器人运动的对称Koopman预测

• Authors: Evelyn D'Elia, Weishu Zhan, Giulio Turrisi, Giulio Romualdi, Giuseppe L'Erario, Raffaello Camoriano, Wei Pan, Daniele Pucci
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.11624
• Pdf link: https://arxiv.org/pdf/2607.11624
• Abstract Reinforcement learning (RL) algorithms classically suffer from poor sample efficiency. In robotics, a recent line of work has emerged addressing this problem by encoding physics priors in the learning process. However, most of these approaches are validated on well-defined, low-dimensional benchmark systems rather than high-dimensional robots with complex nonlinear dynamics. In this paper, we introduce \textit{SKooP (Symmetric Koopman Predictions)}, an approach combining the advantages of morphological symmetries with those of a Koopman model learned via autoencoder to enhance policy learning. SKooP learns a Koopman model of the system dynamics alongside the policy. The resulting Koopman predictions are used as privileged observations for the critic, allowing the agent to learn based on smoother, more informative features. We also incorporate group symmetries into the actor, critic, encoder and decoder networks to produce a highly equivariant policy. The SKooP approach is validated via in-depth analysis of the learned Koopman models and symmetric policies to showcase how each of these influences the agent's performance. We also show that the learned policies are transferable to different simulation environments. Our results show that SKooP consistently reduces convergence time and increases the learned reward for multiple challenging bipedal locomotion tasks on a quadruped robot. Project page: this https URL
• 中文摘要 强化学习（RL）算法在经典上表现为样本效率较差的问题。在机器人领域，最近出现了一条研究方向，通过在学习过程中编码物理先验来解决这一问题。然而，大多数这些方法是在定义明确的低维基准系统上验证的，而非具有复杂非线性动力学的高维机器人。本文介绍了 \textit{SKooP （对称 Koopman 预测）}，这是一种结合形态对称性优势与通过自编码器学习的 Koopman 模型优势以增强策略学习的方法。SKooP 在策略的同时学习了系统动态的 Koopman 模型。由此产生的库普曼预测被用作批评者的特权观察，使智能体能够基于更平滑、更具信息量的特征进行学习。我们还将群对称性纳入actor、critic、encoder和decoder网络，以生成高度等变策略。通过对所学的库普曼模型和对称策略的深入分析，验证了SKooP方法，展示了这些模型如何影响代理的表现。我们还展示了所学策略可迁移到不同的仿真环境中。我们的结果表明，SKooP持续缩短收敛时间，并提高四足机器人在多项具有挑战性的双足行走任务中的学习奖励。项目页面：此 https URL

Active Offline-to-Online Reinforcement Learning

主动离线到在线强化学习

• Authors: Alper Kamil Bozkurt, Shangtong Zhang, Yuichi Motai
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.11720
• Pdf link: https://arxiv.org/pdf/2607.11720
• Abstract Background: Offline reinforcement learning (RL) enables effective policies to be trained from large, previously collected datasets and subsequently improved through limited online interaction. This offline-to-online RL (O2O-RL) paradigm is particularly promising in nonstationary domains where interaction is costly or potentially hazardous. Standard O2O-RL pipelines train multiple candidate policies offline, evaluate them using off-policy or online evaluation, and then deploy and fine-tune the policy with the highest estimated value. However, as in offline pretraining, fine-tuning performance is highly sensitive to the choice of algorithm and hyperparameters, making it risky to commit to a single policy. Objectives: We study active policy selection for fine-tuning under a limited interaction budget in O2O-RL settings. To our knowledge, this is the first work to address this problem. Methods: We formulate the problem by identifying a fundamental trade-off between allocating online interactions to policy evaluation, which helps identify high-performing policies, and allocating them to fine-tuning, which improves policy performance. We then propose an approach that balances this trade-off by actively selecting policies for fine-tuning based on upper-confidence bounds on their future performance. These bounds are derived from locally linear performance forecasts fitted to observations obtained through online evaluation. Results: Across a diverse range of experiments, the proposed approach consistently outperforms existing O2O-RL baselines. Conclusions: Actively selecting and fine-tuning policies uses limited online interaction budgets more effectively than either committing to a single policy or dividing the budget equally among all policies. Our framework also advances offline RL toward practical deployment in real-world systems where online interaction is costly or risky.
• 中文摘要 背景：离线强化学习（RL）使得能够从大量先前收集的数据集中训练有效策略，并通过有限的在线互动进行改进。这种离线到在线的强化学习（O2O-RL）范式在非平稳领域尤为有前景，因为交互代价高昂或潜在危险。标准O2O-RL流水线会离线训练多个候选策略，使用非策略或在线评估进行评估，然后部署并对估计价值最高的策略进行微调。然而，与离线预训练一样，微调性能对算法和超参数的选择高度敏感，因此只承诺单一策略存在风险。目标：我们研究在有限交互预算下，O2O-RL环境中的主动策略选择以进行微调。据我们所知，这是首次针对这一问题进行研究。方法：我们通过确定将在线互动分配用于政策评估（有助于识别高绩效政策）与将其分配用于微调（提升政策绩效）之间的基本权衡来构建问题。随后，我们提出一种方法，通过主动选择基于政策未来表现的上置信度界限进行微调，来平衡这一权衡。这些界限来自于通过在线评估获得的观测值拟合的局部线性性能预测。结果：在多样化的实验中，所提方法始终优于现有的O2O-RL基线。结论：主动选择和微调政策，比承诺单一政策或将预算平均分配给所有政策更有效地利用有限的在线互动预算。我们的框架还推动离线强化学习在现实系统中实际部署，尤其是在在线交互成本高昂或风险较高的系统中。

Time-Lag-Aware Deep Reinforcement Learning for Flexible Job-Shop Scheduling in PPVC Module Factories

PPVC模块工厂中灵活作业车间调度的时滞感知深度强化学习

• Authors: Ziheng Zhang, Wei Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.11725
• Pdf link: https://arxiv.org/pdf/2607.11725
• Abstract Prefabricated prefinished volumetric construction moves most building work into module factories, whose production floor operates as a flexible job shop. A major complication is decisive: long post-operation time-lags caused by concrete curing, watertightness ponding tests, and paint drying, during which a module is blocked while its workstation stays free. On benchmark instances grounded in an official national prefabrication guidebook, these lags inflate even the optimal reference makespan by about 67% on average, and ignoring them at decision time, then repairing to feasibility, is worse than every dispatching rule. We adapt a state-of-the-art dual-attention deep reinforcement learning solver through three minimally invasive, individually ablatable extensions: lag-aware dynamics with an admissible reward bound, two anticipatory lag feature channels, and liveness-masked operation- and station-type embeddings. With every extension disabled the implementation reproduces the original solver exactly, so all gains are attributable to the adaptations. We release a public, guidebook-grounded benchmark generator. On held-out instances the learned policy is the strongest solver-free scheduler: it reaches within about 4% of a constraint-programming reference and beats every dispatching rule and a genetic-algorithm metaheuristic, with its advantage widening under capacity contention, and a single size-mixed policy carries this lead across the trained range of factory sizes. It needs no solver, model, or license in the loop and re-plans within seconds of a disruption; where an exact solver can be deployed, that solver remains the quality ceiling, a boundary we map explicitly.
• 中文摘要 预制预制体量结构将大部分建筑工作转移到模块工厂，其生产车间作为灵活的作业车间运作。一个决定性的问题是：混凝土养护、防水积水测试和油漆干燥导致的长时间运行滞后，模块被阻塞，而工作站保持空闲。根据官方国家预制指南的基准案例，这些滞后平均会使最优参考寿命膨胀约67%，在决策时忽略它们，然后修复至可行性，比任何调度规则都糟糕。我们通过三种微创、可单独消蚀的扩展，调整了先进的双注意力深度强化学习求解器：具有可接受奖励界限的延迟感知动态、两个预期延迟特征通道，以及带活度掩蔽的操作型和站点型嵌入。禁用所有扩展后，实现会完全复现原始求解器，因此所有收益均归因于这些适应。我们发布了一个公开的、基于指南的基准生成器。在被保留的实例中，所学策略是最强的无求解器调度器：它接近约束编程参考的约4%，击败所有调度规则和遗传算法元启发式，其优势在容量争用下扩大，单一规模混合策略在训练过的工厂规模范围内保持领先。它不需要求解器、模型或许可，且在中断发生后几秒钟内重新规划;在可以部署精确求解器时，该求解器仍然是质量上限，这是我们明确映射的边界。

Multi-Agent Reinforcement Learning for C-V2X RAT Selection

多智能体强化学习用于C-V2X RAT选择

• Authors: Moritz Schaffenroth, Uwe Kölbel, Heike Lepke, Alexander Prinz, Alfred Höß
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.11744
• Pdf link: https://arxiv.org/pdf/2607.11744
• Abstract Vehicles are increasingly equipped with advanced V2X communication capabilities. While early V2X apps utilized services such as Cooperative Awareness Messages, recent developments have allowed more advanced applications including cooperative driving, shared perception, and sensor-sharing services. The broader mix of applications leads to heterogeneous requirements for latency and reliability. At the same time multiple communication technologies for V2X are available with pros and cons. Hybrid V2X communication can exploit the distinct advantages at the right moment to fulfill the requirements of the applications. This work studies the decision problem between cellular Uu link, NR-V2X PC5 sidelink, and the simultaneous use of both channels. We address this problem by using the multi-agent reinforcement learning algorithm MAPPO and compare it to five baselines consisting of a deep reinforcement learning (DRL) approach, a static decision tree approach and static channel selection strategies. The methods are evaluated in an urban scenario and with a set of selected communication use cases. The evaluation results show that when compared to the DRL approach, the on-time delivery ratio improves from 0.508 to 0.535 in a single-controlled-vehicle setting and from 0.548 to 0.567 when all vehicles follow the learned policy and reduces the training time by half. The gains result mainly from the advanced applications scenarios, as opposed to scenarios involving exclusively CAM messaging. This indicates future applications will benefit from such adaptive communication strategies and that multi-agent modelling is useful for addressing the underlying decision problem.
• 中文摘要 车辆越来越多地配备了先进的V2X通信能力。虽然早期的V2X应用使用了合作意识消息等服务，但近年来的发展使得更先进的应用得以实现，包括合作驾驶、共享感知和传感器共享服务。更广泛的应用组合导致延迟和可靠性要求异构。同时，多种V2X通信技术存在，各有优缺点。混合V2X通信可以在合适的时机利用其独特优势，满足应用需求。本研究研究了蜂窝Uu链路、NR-V2X PC5侧链路以及同时使用两个通道之间的决策问题。我们通过使用多智能体强化学习算法MAPPO来解决这个问题，并将其与五个基线进行比较，这些基线包括深度强化学习（DRL）、静态决策树方法和静态通道选择策略。这些方法在城市场景中进行评估，并结合一组选定的通信用例。评估结果显示，与DRL方法相比，单一受控车辆的准时交付率从0.508提升至0.535，当所有车辆遵循学习策略并将培训时间缩短一半时，准时交付率从0.548提升至0.567。这些优势主要来自高级应用场景，而非仅涉及CAM消息的场景。这表明未来的应用将受益于此类自适应通信策略，且多智能体建模对于解决潜在决策问题非常有用。

Transformer-Guided Swarm Intelligence for Frugal Neural Architecture Search

Transformer引导的群体智能用于节约神经结构搜索

• Authors: Romain Amigon
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2607.11826
• Pdf link: https://arxiv.org/pdf/2607.11826
• Abstract Neural Architecture Search (NAS) has automated the design of deep learning models but traditionally requires massive computational resources, often measured in thousands of GPU-days. In this paper, we propose a frugal and memetic NAS framework designed to democratize architecture design on consumer-grade hardware. Our approach combines the global macro-search capabilities of an autoregressive Transformer controller, trained via Reinforcement Learning (RL), with the local micro-exploitation of an Artificial Bee Colony (ABC) algorithm. To prevent premature convergence during the RL phase, we introduce a dynamic entropy mechanism that forces topological exploration upon detection of performance stagnation. Evaluated on a standard GPU (NVIDIA RTX 3060), our hybrid method effectively resolves the "cold-start" problem inherent in metaheuristics. By algorithmically penalizing network depth, our framework actively mitigates model bloat: on the CIFAR-10 dataset, it discovers an efficient architecture reaching 84.85% accuracy with only $\sim$174,000 parameters (significantly smaller than standard baselines like ResNet-20) in 3 hours of search time. Furthermore, we demonstrate the framework's flexibility by applying it to credit card fraud detection, directly optimizing the F1-Score on highly imbalanced tabular data to reach a F1-Score of 0.71 with a compact network of $\sim$4,600 parameters. These results suggest that our approach can yield tailored, accessible, and highly parameter-efficient deep learning models suitable for edge deployment.
• 中文摘要 神经架构搜索（NAS）已经自动化了深度学习模型的设计，但传统上需要大量计算资源，通常以数千GPU日为单位。本文提出了一个节约且具模因性的NAS框架，旨在实现消费级硬件架构设计的民主化。我们的方法结合了通过强化学习（RL）训练的自回归Transformer控制器的全局宏搜索能力，以及局部微观利用人工蜂群（ABC）算法。为防止强化学习阶段的过早收敛，我们引入了动态熵机制，在检测性能停滞时强制拓扑探索。在标准GPU（NVIDIA RTX 3060）上评估时，我们的混合方法有效解决了元启发式中固有的“冷启动”问题。通过算法惩罚网络深度，我们的框架积极缓解模型臃肿：在CIFAR-10数据集上，它发现了一个高效架构，仅有$174,000参数（明显低于ResNet-20等标准基线），准确率达到84.85%。此外，我们通过将该框架应用于信用卡欺诈检测，直接优化高度不平衡的表格数据中的F1评分，使得在一个仅有$\sim$4,600参数的紧凑网络下，将F1评分提升至0.71。这些结果表明，我们的方法能够生成适合边缘部署的定制化、可访问且参数高效的深度学习模型。

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation

一个极简主义的重定向引导强化学习方法，用于灵巧操作

• Authors: Yunhai Feng, Natalie Leung, Jiaxuan Wang, Lujie Yang, Haozhi Qi, Preston Culbertson
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.11874
• Pdf link: https://arxiv.org/pdf/2607.11874
• Abstract Recent work in humanoid whole-body control has found success with a simple recipe: retarget human motion to robot kinematic references, then train policies via reinforcement learning (RL) to track them. But how does this recipe transfer to dexterous manipulation? The answer is not obvious, as manipulation involves complex, contact-rich dynamics and requires delicate regulation of contact modes and forces. We present REGRIND, a minimalist retargeting-guided RL pipeline that learns dexterous manipulation policies from a single human demonstration. REGRIND retargets human hand-object motion to a robot reference that preserves hand-object spatial and contact relationships, trains a residual RL policy in simulation to track object-centric keypoints along that reference, and transfers the resulting policy zero-shot to hardware with careful system identification. The resulting policies produce fluid, human-like behavior on two different multi-fingered hands across contact-rich tool-use tasks, including operating a pair of scissors and turning a screwdriver. Through systematic hardware experiments, we identify and analyze the key factors that govern sim-to-real transfer in dexterous manipulation, offering practical guidance for retargeting-based learning in contact-rich settings. Videos and code are available at this https URL.
• 中文摘要 最近在人形全身控制方面的研究取得了成功，方法很简单：将人类运动重新定位到机器人运动学参考，然后通过强化学习（RL）训练策略来追踪这些参考。但这个配方如何转化为灵巧操控呢？答案并不明显，因为操控涉及复杂且接触丰富的动力学，需要对接触模式和力的细致调控。我们介绍REGRIND，一个极简主义重定向引导的强化学习流水线，通过单一人工演示学习灵巧操作策略。REGRIND将人类手与物体的运动重新定位到保持手与物体空间及接触关系的机器人参考，在仿真中训练残余强化学习策略以追踪该参考点上的对象中心关键点，并通过仔细系统识别将所得策略零点传输到硬件。由此产生的策略使两只不同的多指手在接触丰富的工具使用任务中产生流畅、类人行为，包括操作剪刀和转动螺丝刀。通过系统的硬件实验，我们识别并分析了灵巧操作中模拟到真实转移的关键因素，为接触丰富环境中基于重定向的学习提供了实用指导。视频和代码可在此 https URL 下载。

Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

请重读：预训练MLM是用于生成文本到图像的零机会奖励模型

• Authors: Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, Jie Wu, Hengshuang Zhao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.11886
• Pdf link: https://arxiv.org/pdf/2607.11886
• Abstract In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: this https URL
• 中文摘要 本文提出了SpectraReward，一种无需训练的奖励函数，将预训练的MLLM转化为用于图像生成强化学习的现成奖励模型。SpectraReward不再要求MLLM评判生成图像或回答分解后的验证问题，而是通过一次图像条件、教师强制前进，从生成图像中恢复原始提示的能力。我们以平均图像条件提示日志似然值作为奖励，直接重用MLLM预训练的图像-文本对齐能力，无需偏好标签，进行奖励模型微调。我们进一步介绍了自谱奖励（Self-SpectraReward），这是统一多模态模型的特例，其中策略自身的理解分支作为生成分支的奖励模型，形成一个无外部奖励模型或外部知识的闭环自我改进框架。通过广泛的图像生成RL研究，涵盖两个扩散模型、三种强化学习算法、涵盖4B至235B参数的四个MLLM家族的九个奖励MLLM骨干，以及五个非发行的文本到图像基准测试，验证了SpectraReward的有效性。结果显示，SpectraReward和Self-SpectraReward都能显著且持续地提升生成性能，并优于以往基于MLLM的奖励训练方法。进一步分析显示，较大奖励的MLLM并不总是更好，而Self-SpectraReward可以匹敌甚至超过更大的外部奖励模型，表明奖励与政策的对齐是有效生成图像强化学习的关键因素。项目页面：此 https URL

Keyword: diffusion policy

A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

一个单扩散策略控制器，用于多任务块推送，实现零时模拟到真实传输

• Authors: Haitong Ma, Haldun Balim, Yang Hu, Bo Dai, Na Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.10892
• Pdf link: https://arxiv.org/pdf/2607.10892
• Abstract Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparse-reward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our real-world block-pushing setup under the tested variations.
• 中文摘要 扩散政策在使用行为克隆（BC）为机器人表示和学习复杂动作方面表现出有希望的实证表现。本文探讨了从零开始利用强化学习（RL）进行多任务机器人操作的扩散训练策略。具体来说，我们旨在为具有多形状的块推任务训练单一扩散策略。所提出的框架具有一个简单的策略损失函数，这是一个用于基于BC的扩散策略训练中的重加权证据下界，并且可以无缝地作为强化学习算法中的策略学习模块。为应对缺乏示范带来的探索挑战，我们采用了逆向课程生成和以目标为中心的表现方式。结合扩散策略的表达性，我们的设计支持在稀疏奖励模拟环境中学习多任务区块推送策略。我们还进一步评估了训练有素的扩散策略是否能在零射中转移到不同环境条件下的实际任务，包括目标位置、方块形状、方块重量和表面摩擦，这证明该流程可以在测试变体下转移到我们的实际方块推送设置中。

Mixture of Frames Policy: Multi-Frame Action Denoising for Bimanual Mobile Manipulation

帧混合策略：双手移动操作的多帧动作去噪

• Authors: Dian Wang, Jisang Park, Xiaomeng Xu, Han Zhang, Shuran Song, Jeannette Bohg
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.11884
• Pdf link: https://arxiv.org/pdf/2607.11884
• Abstract Robotic manipulation is inherently multi-frame: local actions may be simple in an end-effector frame, while transport, upright-object handling, and whole-body coordination are better represented in a base-aligned frame. However, modern diffusion-based visuomotor policies typically commit to a single predefined action frame, forcing one denoiser to model action distributions that are often unnecessarily complex in that frame. We propose Mixture of Frames Policy (MoF), a diffusion policy that performs synchronized action denoising across multiple coordinate frames. MoF maintains a single canonical diffusion state, re-expresses it in several task-relevant frames, applies frame-specialized denoisers, and fuses their noise predictions back in the canonical frame. To make this possible for intermediate noisy diffusion states, we introduce a column-based 6D rotation representation within an SE(3) action parameterization that supports exact, differentiable frame transformations without requiring noisy rotations to lie on the SO(3) manifold. Across nine simulated bimanual manipulation tasks, we show that the best action frame is task-dependent and that MoF improves over oracle frame selection and standard Mixture-of-Experts (MoE) baselines. We further evaluate MoF on two real-world bimanual mobile manipulation tasks, demonstrating that it outperforms all constituent single-frame baselines. Project homepage: this https URL
• 中文摘要 机器人操作本质上是多帧的：局部动作在端执行器框架中可能很简单，而运输、直立物体操作和全身协调则在基准框架中表现得更好。然而，现代基于扩散的视觉运动策略通常只承诺一个预定义的动作框架，迫使一个去噪器模拟该框架中通常过于复杂的动作分布。我们提出了帧混合策略（MoF），这是一种跨多个坐标帧进行同步动作去噪的扩散策略。MoF保持单一的典型扩散态，将其重新表达为多个任务相关的参考系，应用针对具体的框架去噪器，并将噪声预测融合回规范参考系。为了实现中间噪声扩散态的实现，我们在 SE（3） 作用参数化中引入了基于列的 6D 旋转表示，支持精确且可微的帧变换，而无需噪声旋转置于 SO（3） 流形上。在九个模拟的双手操作任务中，我们表明最佳动作框架依赖于任务，且MoF优于oracle框架选择和标准专家混合（MoE）基线。我们进一步评估了MoF在两个真实世界双手移动操作任务中的表现，证明其表现优于所有组成的单帧基线。项目主页：此 https URL