生成时间: 2026-07-09 18:56:44 (UTC+8); Arxiv 发布时间: 2026-07-09 20:00 EDT (2026-07-10 08:00 UTC+8)

今天共有 29 篇相关文章

Keyword: reinforcement learning

Deep Reinforcement Learning for Reliability Based Bi-Objective Portfolio Optimization

基于可靠性的双目标组合优化深度强化学习

• Authors: Sounaq Das, Tanmay Sen, Raghu Nandan Sengupta, Aditya Gupta
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.06610
• Pdf link: https://arxiv.org/pdf/2607.06610
• Abstract Portfolio optimization under uncertainty is inherently a multi-objective decision problem involving complex interactions among return, risk, market dynamics, and practical investment constraints. Existing reliability based portfolio optimization approaches primarily rely on static optimization frameworks and often fail to capture sequential decision making, tail risk, and market frictions such as transaction costs. To address these limitations, we propose a deep reinforcement learning framework for multi-objective reliability based portfolio optimization (MORP-DRL). The proposed framework jointly optimizes expected return and downside risk using three complementary risk measures: variance, Conditional Value-at-Risk (CVaR), and Entropic Value-at-Risk (EVaR). To model uncertainty and heavy-tailed market behavior, asset returns are represented using GARCH(1,1), Extreme Value Theory, and a t-copula dependence structure, while realistic scenarios are generated through quasi-Monte Carlo simulation. A Proximal Policy Optimization (PPO) based strategy is developed under practical constraints including transaction costs and portfolio bounds, and is benchmarked against NSGA-II. Experiments on ten global equity indices across pre-COVID, COVID, and post-COVID market regimes demonstrate that MORP-DRL achieves competitive risk-return performance, reduced downside risk during periods of market stress, and scalability to high-dimensional portfolio settings.
• 中文摘要 在不确定性下进行投资组合优化本质上是一个多目标决策问题，涉及回报、风险、市场动态和实际投资约束之间的复杂相互作用。现有基于可靠性的投资组合优化方法主要依赖静态优化框架，常常未能捕捉连续决策、尾部风险和交易成本等市场摩擦。为解决这些局限性，我们提出了一个深度强化学习框架，用于多目标信度基础投资组合优化（MORP-DRL）。所提框架通过三种互补风险指标共同优化预期回报和下行风险：方差、条件风险价值（CVaR）和风险熵价值（EVaR）。为了模拟不确定性和重尾市场行为，资产回报采用GARCH（1,1）、极值理论和t-连环依赖结构表示，而现实情景则通过准蒙特卡洛模拟生成。基于近端策略优化（PPO）的策略是在包括交易成本和投资组合界限在内的实际约束条件下开发的，并以NSGA-II为基准测试。在疫情前、疫情及疫情后市场体系中对十个全球股票指数的实验表明，MORP-DRL实现了具有竞争力的风险回报表现，在市场压力时期降低了下行风险，并具备向高维投资组合的可扩展性。

HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing

HiFuzz：用于语义感知和自适应CPU模糊的层级强化学习

• Authors: Ya Wang, Hanwei Fan, Zhenguo Liu, Xiaofeng Zhou, Yangdi Lyu, Jiang Xu, Wei Zhang
• Subjects: Subjects: Hardware Architecture (cs.AR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.06619
• Pdf link: https://arxiv.org/pdf/2607.06619
• Abstract Modern processor verification struggles to reach deep architectural states due to the inefficiencies of traditional mutation-based fuzzing. We propose HiFuzz, a novel hierarchical reinforcement learning framework that replaces mutation with a structured, two-layer generation process: a Program Agent for global layout and a Basic Block Agent for precise instruction filling. To overcome reward sparsity, HiFuzz integrates an adaptive coverage reward mechanism and a semantic-aware basic block encoder providing intrinsic feedback. Extensive evaluations on three real-world RISC-V cores demonstrate that HiFuzz significantly outperforms state-of-the-art fuzzers in coverage and bug detection.
• 中文摘要 现代处理器验证难以达到深度架构状态，原因是传统基于突变的模糊效果效率低下。我们提出了HiFuzz，一种新型分层强化学习框架，用结构化的双层生成过程取代突变：用于全局布局的程序代理和用于精确填充指令的基本块代理。为了克服奖励稀疏性，HiFuzz集成了自适应覆盖奖励机制和语义感知基础块编码器，提供内在反馈。对三个真实世界RISC-V核心的广泛评估表明，HiFuzz在覆盖率和漏洞检测方面远远优于最先进的fuzzer。

ORAN-DEFEND: Subspace Detection and Sanitization of Backdoor DRL xApps in Open RAN

ORAN-DEFEND：开放RAN中后门DRL xApp的亚空间检测与净化

• Authors: Md Raihan Uddin, Fatemeh Lotfi, Tolunay Seyfi, Fatemeh Afghah
• Subjects: Subjects: Cryptography and Security (cs.CR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.06647
• Pdf link: https://arxiv.org/pdf/2607.06647
• Abstract Open Radio Access Networks (O-RAN) increasingly delegate near-real-time control to deep reinforcement learning (DRL) xApps obtained from third-party vendors, creating a new supply-chain attack surface. A backdoor policy behaves optimally until an adversary injects a covert trigger into the observed key performance indicator (KPI) telemetry, at which point it issues harmful control actions that degrade quality of service (QoS). We present ORAN-DEFEND, a retraining-free wrapper that sanitizes a frozen, potentially compromised xApp by projecting each KPI window onto a safe subspace estimated from a small number of trusted clean rollouts via singular value decomposition (SVD). We establish, both analytically and empirically, a precise recovery condition: the defense succeeds if the trigger energy concentrates in the orthogonal complement of the safe subspace, and we quantify this boundary through the trigger's $\Eperp$ energy fraction. On the Colosseum COLORAN dataset, we evaluate four structurally distinct DRL backdoor attacks, like TrojDRL, SleeperNets, BadRL, and Q-Incept, spanning inner-loop and outer-loop poisoning regimes and demonstrate $100\%$ return recovery and $\geq99.5\%$ defense success rate across all four when the subspace assumption holds. A geometry ablation reveals an intrinsic and previously uncharacterized limit of any linear projection defense: when the trigger collocates with the legitimate signal, the $\Eperp$ energy fraction governs recovery monotonically, and the linear residual detector collapses to chance even while a nonlinear classifier retains perfect separability.
• 中文摘要 开放无线接入网络（O-RAN）越来越多地将近实时控制权委托给第三方供应商提供的深度强化学习（DRL）xApp，形成新的供应链攻击面。后门策略表现最佳，直到攻击者在观察到的关键绩效指标（KPI）遥测中注入隐性触发器，随后执行有害的控制措施，降低服务质量（QoS）。我们介绍 ORAN-DEFEND，一种无需重新训练的包装程序，通过将每个关键绩效指标窗口投影到通过奇异值分解（SVD）少数可信的干净部署推算的安全子空间，净化冻结且可能被攻破的 xApp。我们通过解析和经验方法建立了一个精确的恢复条件：如果触发能量集中在安全子空间的正交补空间中，防御成功，并通过触发器的 $\Eperp$ 能量分数量化该边界。在Colosseum COLORAN数据集上，我们评估了四种结构上不同的DRL后门攻击，如TrojDRL、SleeperNets、BadRL和Q-Incept，跨越内环和外环毒害机制，并在亚空间假设成立时，展示了四种攻击均可实现100%%的回报回收和$/geq99.5%%的防御成功率。几何消融揭示了任何线性投影防御的内在且此前未表征的极限：当触发器与合法信号共处时，$\Eperp$能量分数单调支配恢复，线性残差探测器即因偶然坍缩，而非线性分类器仍保持完美可分离性。

When Does In-Context Search Help? A Sampling-Complexity Theory of Reflection-Driven Reasoning

上下文搜索什么时候有用？反思驱动推理的采样复杂性理论

• Authors: Yotam Wolf, Noam Wies, Amnon Shashua
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.06720
• Pdf link: https://arxiv.org/pdf/2607.06720
• Abstract Training large language models (LLMs) with extended reasoning has enabled in-context search, in which models iteratively generate, critique, and revise solution attempts. We provide a theoretical analysis of in-context search by modeling it as approximate inference over reasoning traces, where the base model defines a prior and self-reflection provides feedback for posterior updates, and study the resulting inference-time sampling complexity - the number of sequential attempts needed to achieve high success probability. We show that when reflections reliably localize early mistakes, in-context search can yield exponential improvements over the base model, solving problems with exponentially small zero-shot pass rates using only a polynomial number of sequential attempts, whereas when this property fails, conditioning on past attempts offers no asymptotic benefit over parallel sampling. We further show that these gains are robust and learnable: approximate posterior updates suffice, and cross-entropy training on search rollouts recovers the required behavior with polynomial sample complexity. Finally, we show that under a stagewise abstraction of reinforcement learning with verifiable rewards, the optimal policy extension implements the same posterior reweighting rule. We validate key qualitative predictions of the theory on real large reasoning models.
• 中文摘要 通过扩展推理训练大型语言模型（LLM），使得上下文内搜索成为可能，模型通过迭代生成、批评和修正解决方案尝试。我们通过对上下文搜索进行理论分析，将其建模为推理轨迹上的近似推断，其中基础模型定义先验，自我反思为后验更新提供反馈，并研究由此产生的推理时间采样复杂度——即实现高成功概率所需的连续尝试次数。我们证明，当反射能可靠定位早期错误时，上下文内搜索可以带来指数级的改进，仅用多项式次数的连续尝试解决零样本通过率指数级低的问题;而当该特性失效时，基于过去尝试的条件相比平行采样没有渐近优势。我们还进一步证明这些增益是稳健且可学习的：近似后验更新即可，搜索推广的交叉熵训练能以多项式样本复杂度恢复所需的行为。最后，我们证明在带有可验证奖励的强化学习分阶段抽象下，最优策略扩展实现了相同的后验重权规则。我们在实大推理模型上验证了该理论的关键定性预测。

Ad Headline Generation using Self-Critical Masked Language Model

使用自我批判的蒙面语言模型生成广告标题

• Authors: Yashal Shakti Kanungo, Sumit Negi, Aruna Rajan
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.06818
• Pdf link: https://arxiv.org/pdf/2607.06818
• Abstract For any E-commerce website it is a nontrivial problem to build enduring advertisements that attract shoppers. It is hard to pass the creative quality bar of the website, especially at a large scale. We thus propose a programmatic solution to generate product advertising headlines using retail content. We propose a state of the art application of Reinforcement Learning (RL) Policy gradient methods on Transformer based Masked Language Models. Our method creates the advertising headline by jointly conditioning on multiple products that a seller wishes to advertise. We demonstrate that our method outperforms existing Transformer and LSTM + RL methods in overlap metrics and quality audits. We also show that our model-generated headlines outperform human submitted headlines in terms of both grammar and creative quality as determined by audits.
• 中文摘要 对于任何电子商务网站来说，打造吸引消费者的持久广告都是一个不易的挑战。网站的创意质量标准很难通过，尤其是在大规模上。因此，我们提出了一种程序化解决方案，利用零售内容生成产品广告标题。我们提出了强化学习（RL）策略梯度方法在基于Transformer的蒙面语言模型上的应用。我们的方法是通过结合卖家希望推广的多个产品来创建广告标题。我们证明了该方法在重叠指标和质量审核方面优于现有的Transformer和LSTM + RL方法。我们还展示了，我们的模型生成标题在语法和创意质量方面均优于人工提交的标题，这些数据均经审计评估。

A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong

一项关于轻量级游戏特工强大条件的黄金标准研究

• Authors: Nima Kelidari, Mohammadsaeed Haghi, Mahdi Salmani
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2607.06854
• Pdf link: https://arxiv.org/pdf/2607.06854
• Abstract Reinforcement learning agents for imperfect-information card games are only as strong as the opponents they train against, and they are hard to grade, since they beat a random opponent over 99 percent of the time and only tie copies of themselves. So we build a strong, fixed, rule-based expert for Gin Rummy and use it only as a yardstick, never for training. It beats every agent we trained 70 to 99 percent of the time. Across more than a hundred runs, we isolate what makes a lightweight agent stronger. Trust region updates, a well-aimed reward, a curriculum of tougher opponents, warm starting, and keeping the best checkpoint all help, and stacking them lifts a self-play champion from about 30 to 36 percent against the expert. Several ideas did not pay off. Short-term and longer-term reward shaping, learned state embeddings, imitation and DAgger, and a live large language model opponent were each unhelpful, too slow, or too heavy to train at scale. Comparing MLP, convolutional, set-based, attention, and recurrent encoders shows that extra capacity does little to break the ceiling, suggesting the limit is information rather than network size. We add standard baselines (neural fictitious self-play and information set Monte Carlo search) and confirm the approach carries over to Leduc Hold'em, where the optimum is computable. The result is a lightweight, game-agnostic recipe that trains competitive agents without training on the expert, for any game a small model can handle, reported with robust statistics and released as a reusable package.
• 中文摘要 用于不完美信息卡牌游戏的强化学习代理的强度取决于训练对手的对手，而且评分较难，因为它们99%以上的概率能击败随机对手，且只打平自己的副本。所以我们打造一个强大、固定、基于规则的金拉米专家，只把它当作衡量标准，从不用于训练。它比我们培训的每个特工都超过70%到99%的时间。通过一百多次运行，我们分离出轻质剂的强效因素。信任地区更新、精准奖励、更强对手的课程、温暖的起始和保持最佳检查点，这些都很重要，叠加这些技能能让自玩冠军对专家的攻击率从大约30%提升到36%。但有几个想法未能奏效。短期和长期奖励塑造、学习状态嵌入、模仿和DAgger，以及一个活体大型语言模型对手，都无益、过慢或过于繁重，难以大规模训练。比较MLP、卷积、基于集合、注意力和递归编码器显示，额外容量对突破上限作用不大，表明限制在于信息而非网络规模。我们加入标准基线（神经虚构自玩和蒙特卡洛信息集搜索），并确认该方法适用于德州扑克，其中最优值是可计算的。结果是一个轻量级、游戏无关的配方，无需专家培训即可训练竞技代理，适用于任何小型模型能胜任的游戏，并以稳健统计数据报告并作为可重复使用的软件包发布。

Auditable Machine Unlearning for Privacy-Compliant Ransomware Detection Using Multi-Shard SISA and Deep Reinforcement Learning

可审计机器学习解读，利用多分片SISA和深度强化学习实现隐私合规勒索软件检测

• Authors: Jannatul Ferdous, Rafiqul Islam, Md Zahidul Islam
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2607.06860
• Pdf link: https://arxiv.org/pdf/2607.06860
• Abstract Ransomware poses an escalating cybersecurity threat as attackers continuously modify behavioral patterns to evade static defenses. Although existing machine learning-based detectors often achieve strong predictive performance, they generally assume fixed training data and do not support the selective removal of previously learned samples. This limitation conflicts with privacy regulations such as the GDPR and CCPA, which require the removal of sensitive user data upon request. To address this challenge, we propose an auditable ransomware detection and unlearning framework that integrates deep reinforcement learning with multi-shard SISA retraining. In the proposed system, a Double Deep Q-Network (DDQN) learns a reward-guided detection policy from behavioral features under asymmetric security costs, while multi-shard SISA enables privacy-compliant selective sample removal through shard-level retraining. The framework was evaluated using four criteria: utility preservation, oracle-based forgetting validation, membership inference auditing, and computational efficiency. On a balanced Windows 11 behavioral dataset comprising 2,000 samples and 103 features, the baseline DDQN detector achieved an F1 score of 0.9925 and an AUC of 0.9983. The experimental results show that single-shard unlearning maintains minimal utility degradation and low oracle disagreement, whereas moderate shard counts (M = 5-10) provide the best efficiency-performance trade-off, reducing retraining time to 5-30 s compared with 80-330 s for full retraining. In addition, the membership inference scores remain close to 0.5 across most configurations, indicating limited privacy leakage after unlearning. These findings demonstrate that a privacy-compliant ransomware detection framework can jointly achieve high detection performance, auditable deletion verification, and efficient sample removal.
• 中文摘要 随着攻击者不断改变行为模式以规避静态防御，勒索软件构成了不断升级的网络安全威胁。尽管现有基于机器学习的探测器通常能实现强有力的预测性能，但它们通常假设训练数据固定，不支持选择性地移除先前学习的样本。这一限制与GDPR和CCPA等隐私法规相冲突，后者要求在请求时删除敏感用户数据。为应对这一挑战，我们提出了一个可审计的勒索软件检测与去学习框架，将深度强化学习与多分片SISA再训练整合在一起。在拟议系统中，双深度Q网络（DDQN）通过行为特征在非对称安全成本下学习奖励引导的检测策略，而多碎片SISA则通过分片级再训练实现符合隐私的选择性样本移除。该框架通过四项标准进行评估：效用保持、基于预言机的遗忘验证、成员推断审计和计算效率。在包含2000个样本和103个特征的平衡Windows 11行为数据集中，基线DDQN检测器获得了F1得分0.9925，AUC为0.9983。实验结果显示，单片卸学习保持了最小的效用退化和低预言机冲突，而适度的碎片数量（M = 5-10）提供了最佳的效率与性能权衡，将再训练时间缩短至5-30秒，而完全再训练则为80-330秒。此外，大多数配置的成员推断得分仍接近0.5，表明在“解除学习”后隐私泄露有限。这些发现表明，符合隐私规范的勒索软件检测框架可以共同实现高检测性能、可审计的删除验证以及高效的样本移除。

SA-DRL: Security-Aware Deep Reinforcement Learning for Ransomware Detection with Asymmetric Reward Design

SA-DRL：安全感知深度强化学习，利用非对称奖励设计进行勒索软件检测

• Authors: Jannatul Ferdous, Rafiqul Islam, Md Zahidul Islam
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2607.06880
• Pdf link: https://arxiv.org/pdf/2607.06880
• Abstract Ransomware detection is a security-critical task in which false negatives and false positives have unequal operational consequences. Conventional machine learning detectors often use symmetric objectives that penalize missed ransomware detections and benign false alarms equally, although a false negative can cause irreversible encryption, operational disruption, and high recovery cost, whereas a false positive is usually reversible. This study proposes a Security-Aware Deep Reinforcement Learning (SA-DRL) framework that embeds false-negative and false-positive cost asymmetry into the reinforcement learning reward signal to prioritize missed-detection reduction. The framework also introduces a Security-Optimal Model Selection (SOMS) criterion and an adaptive episode-level sample-ordering mechanism. Four deep reinforcement learning agents, DQN, DDQN, PPO, and A2C, were evaluated using a symmetric baseline reward (R1) and a security-aware asymmetric reward (R2). Experiments used four discount factors, five-fold cross-validation, and three random seeds, resulting in 480 training runs on a balanced ransomware detection dataset. The SOMS criterion selects models by prioritizing false-negative rate, followed by F1-score and training time. Results show that asymmetric reward shaping improves security-oriented detection performance. The SOMS-selected configuration, DDQN with R2 and gamma = 0.1, achieved a false-negative rate of 0.0080, an F1-score of 0.9915, and an AUC of 0.998, reducing missed detections by 67.6% compared with the best supervised baseline. Across all configurations, R2 reduced the mean false-negative rate by 43% relative to R1. These findings show that reward-function design is important for security-sensitive ransomware detection.
• 中文摘要 勒索软件检测是一项安全关键任务，其中误报和误报在操作上产生不平等的后果。传统的机器学习检测器通常使用对称目标，对漏接勒索软件检测和良性误报同样惩罚，尽管误报可能导致不可逆加密、运行中断和高恢复成本，而误报通常是可逆的。本研究提出了一种安全感知深度强化学习（SA-DRL）框架，将假阴性和假阳性成本不对称嵌入强化学习奖励信号中，以优先减少漏检。该框架还引入了安全最优模型选择（SOMS）标准和自适应的集数级样本排序机制。采用对称基线奖励（R1）和安全意识非对称奖励（R2）评估了四个深度强化学习代理DQN、DDQN、PPO和A2C。实验采用了四个折扣因子、五重交叉验证和三个随机种子，结果在一个平衡的勒索软件检测数据集上进行了480次训练。SOMS 标准通过优先选择假阴性率，其次是 F1 分数和训练时间。结果表明，非对称奖励塑形提升了安全导向的检测性能。SOMS选定的配置DDQN（R2和γ=0.1）假阴性率为0.0080，F1评分为0.9915，AUC为0.998，较最佳监督基线减少了67.6%的漏检。在所有配置中，R2相较R1降低了43%的平均假阴性率。这些发现表明，奖励函数设计对于安全敏感的勒索软件检测至关重要。

Degradation-Aware Pumping Control of Variable-Speed Pumped Storage via Residual Reinforcement Learning

通过残差强化学习实现变速抽蓄的降解感知泵控制

• Authors: Kyung-bin Kwon, SangWoo Park, Dam Kim
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.06911
• Pdf link: https://arxiv.org/pdf/2607.06911
• Abstract Variable-speed pumped storage hydropower (VS-PSH) must honor short-block dispatch commitments while limiting the operational degradation that intensified regulation duty inflicts on its components. When a single controller pursues both aims at once, every tracking gain is paid for in degradation, a conflict that persists even under full model knowledge and look-ahead. This paper proposes a two-layer control architecture that separates the guaranteed commitment from the bounded learning. A deterministic feedforward-PI gate controller, auditable and certifiable for grid-connected operation, secures average power delivery over each five-minute block, while a residual reinforcement learning policy adjusts only the rotor speed within a fixed bound the gate loop can always absorb, so the worst-case command is bounded by construction. The speed policy tracks a demand-dependent best-efficiency-point reference and is trained against an operation-degradation index that combines off-best-efficiency hydraulic loss with power and actuation variation into one physically interpretable signal. Across normal and stressed dispatch, the proposed policy lowers best-efficiency-point tracking error by roughly 96\% relative to a fixed-speed baseline and cuts total degradation by up to about 56\% under the most demanding dispatch. It matches or slightly exceeds a full-information model-based optimizer in efficiency while preserving substantially tighter block tracking.
• 中文摘要 变速抽蓄水电（VS-PSH）必须履行短区块调度承诺，同时限制严格监管任务对其组件造成的运行劣化。当单个控制器同时追求这两个目标时，每一次跟踪收益都以退化为代价，这种冲突即使在完全了解模型和前瞻的情况下依然存在。本文提出了一种两层控制架构，将保证承诺与有界学习区分开来。确定性前馈PI门控器可审计并认证于并网运行，确保每个五分钟区块的平均功率传输，而残余强化学习策略仅调整门环能吸收的固定范围内转子速度，因此最坏情况的指令受结构限制。该速度策略跟踪一个需求依赖的最佳效率点参考，并基于一个运行衰减指数训练，该指数将偏向最佳效率的液压损失与功率和驱动变化结合为一个物理可解读的信号。在正常调度和应力调度中，拟议政策相较固定速度基线降低了约96%的最佳效率点跟踪误差，并在最苛刻调度下将总降级减少约56%。其效率与基于全信息模型的优化器相当或略高，同时保持了更紧凑的块跟踪。

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

UP：无界正不对称优化以打破勘探-稳定性困境

• Authors: Chongyu Fan, Pengfei Liu, Jingjia Huang, Sijia Liu, Yi Lin
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.06987
• Pdf link: https://arxiv.org/pdf/2607.06987
• Abstract Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs). To achieve sample efficiency, modern RL frameworks rely on importance sampling (IS). However, these algorithms suffer from an exploration-stability dilemma. Pure IS often leads to catastrophic training instability, while standard clipping mechanisms used to mitigate this instability strictly constrain the policy update budget. By formalizing the concept of Probability Capacity (Cap), we reveal that conservative clipping structurally stifles exploration by prematurely truncating the update budget for correct but low-confidence reasoning paths. To break free from these constraints, we propose Unbounded Positive Asymmetric Optimization (UP), a universal and plug-and-play objective. UP theoretically restructures the optimization process by anchoring the policy to its current state via the stop-gradient operator. This asymmetric design unleashes unclipped, stable gradients for positive advantages to maximize exploration, while maintaining standard clipping safeguards for negative advantages to prevent training instability. Furthermore, our formulation readily extends across different optimization granularities, including token-level (GRPO, DAPO) and sequence-level (GSPO) frameworks. Extensive experiments demonstrate that UP enhances exploration capacity and achieves superior reasoning accuracy across diverse RL algorithms (DAPO, GSPO, and GRPO), model architectures (Dense, MoE, and vision-language), and training modalities (language and multimodal), validating UP as a truly universal plug-and-play enhancement for RL-based training.
• 中文摘要 强化学习（RL）已成为提升大型语言模型（LLM）复杂推理能力的标准范式。为了实现样本效率，现代强化学习框架依赖于重要性抽样（IS）。然而，这些算法存在探索与稳定性的困境。纯信息系统常导致灾难性的训练不稳定性，而用于缓解这种不稳定性的标准裁剪机制严格限制策略更新预算。通过形式化概率容量（Cap）概念，我们揭示了保守裁剪结构性地扼杀了探索，因为提前截断了正确但低置信度推理路径的更新预算。为了摆脱这些限制，我们提出了无界正非对称优化（UP），这是一个通用且即插即用的目标。理论上，UP通过通过停止梯度算子将策略锚定到当前状态，重新构建了优化过程。这种不对称设计释放出未截断、稳定的梯度以获得正面优势以最大化勘探，同时保持标准的削减保护措施以防训练不稳定。此外，我们的表述能够跨越不同的优化粒度，包括代币级（GRPO、DAPO）和序列级（GSPO）框架。大量实验表明，UP增强了探索能力，并在多种强化学习算法（DAPO、GSPO和GRPO）、模型架构（Dense、MoE和视觉语言）以及训练模式（语言和多模态）上实现了更优越的推理精度，验证了UP作为真正通用的即插即用强化学习训练增强工具。

Large Behavior Model: A Promptable Digital Twin of the Retail Customer

大型行为模型：零售客户的可提示数字孪生

• Authors: Wachiravit Modecrua, Krittin Pachtrachai, Touchapon Kraisingkorn
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.06993
• Pdf link: https://arxiv.org/pdf/2607.06993
• Abstract Customer behavior modeling underpins recommendation, marketing, and decision support, yet existing approaches either optimize predictive accuracy without explaining decisions or simulate users without grounding them in real behavioral data. We present the Large Behavioral Model (LBM) that learns customer decision making directly from large-scale retail transactions through a unified Person-Environment formulation. Customer state is represented by a behavioral profile derived from historical purchases, while product context is incorporated through retrieval-augmented generation. The model is trained using continued pre-training on verbalized behavioral data, supervised fine-tuning for decision generation, and reinforcement learning with verifiable rewards for evidence-based calibration. We evaluate the proposed framework on purchase prediction, hard-negative discrimination, basket completion, promotion response, and cross-domain voucher redemption. The model consistently outperforms frontier general-purpose language models on in-domain retail tasks while demonstrating strong zero-shot and fine-tuned transfer across retailers and decision domains. Ablation studies show that continued pre-training is the primary driver of behavioral generalization, retrieval is most effective when applied during both training and inference, and reinforcement learning improves reliance on explicit behavioral evidence over generic language-model priors. These results demonstrate that behavioral knowledge encoded in transaction histories can be effectively learned by language models, providing a scalable foundation for customer digital twins and behavior simulation.
• 中文摘要 客户行为建模支撑着推荐、市场营销和决策支持，但现有方法要么优化预测准确性而不解释决策，要么模拟用户却不基于真实行为数据。我们提出了大型行为模型（LBM），通过统一的人-环境表述，直接从大规模零售交易中学习客户决策。客户状态由历史购买所得的行为画像表示，而产品上下文则通过检索增强生成技术被纳入。该模型通过持续的口头行为数据预训练、决策生成的监督微调，以及基于证据的校准的可验证奖励强化学习进行训练。我们评估了所提出的框架，包括购买预测、硬性负面辨别、篮子完成、促销响应和跨领域代金券兑换。该模型在领域零售任务中持续优于前沿通用语言模型，同时展现出强劲的零样本和微调转移能力，跨零售商和决策领域。消融研究显示，持续的预训练是行为泛化的主要驱动力，检索在训练和推断过程中最为有效，强化学习则改善了对显性行为证据的依赖，而非泛泛的语言模型先验。这些结果表明，编码在交易历史中的行为知识可以通过语言模型有效学习，为客户数字孪生和行为模拟提供可扩展的基础。

Online Data Selection Is Implicit Alignment

在线数据选择是隐式对齐

• Authors: Aoxiong Zeng, Yuxin Yang, Xiangquan Yang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07023
• Pdf link: https://arxiv.org/pdf/2607.07023
• Abstract Supervised fine-tuning (SFT) is often treated as a capability-adaptation step, while alignment is attributed to later preference optimization or reinforcement learning. This separation is incomplete: when examples are scored and kept online during fine-tuning, the choice of which data to train on already changes the model's behavioral preferences. We study online data selection as an implicit alignment mechanism. Given the same base model, optimizer, and selected-token budget, we compare random, loss-based, quality-based, and diversity-based online selectors and measure the behavioral drift they induce without any preference optimization. The proposed evaluation tracks helpfulness, refusal rate, verbosity, truthfulness, sycophancy, calibration, and jailbreak robustness, together with diagnostics for which behavioral modes are over-represented in the selected data. We formalize online selection as a reweighted SFT objective whose weights define an implicit preference over response styles and safety postures, so that an online scorer plays the role usually assigned to a reward model. This view predicts that high-scoring data can systematically favor longer, more assertive, more compliant, or more refusal-prone behaviors depending on how the online score is defined. Empirically, selectors that are statistically indistinguishable in task accuracy diverge sharply in refusal rate, verbosity, and sycophancy, and we show that the direction of the shift is predictable from the attribute mixture of the selected data. We introduce Alignment Drift Auditing (ADA), a controlled protocol for quantifying selection-induced behavioral movement, and Alignment-Aware Selection (AAS), a diagnostic online selector that retains data efficiency while constraining drift along safety and style axes.
• 中文摘要 监督微调（SFT）通常被视为能力适配步骤，而对齐则归因于后续的偏好优化或强化学习。这种分离是不完整的：当样本被评分并在微调期间在线时，选择训练数据本身就改变了模型的行为偏好。我们研究在线数据选择作为隐性对齐机制。在相同的基础模型、优化器和选择令牌预算的情况下，我们比较随机、基于损失、基于质量和基于多样性的在线选择器，并测量它们在无偏好优化情况下引发的行为漂移。本评估追踪了帮助性、拒绝率、冗长度、真实性、谄媚性、校准和越狱鲁棒性，以及在所选数据中行为模式过度代表的诊断。我们将在线选择形式化为一个加权后的SFT目标，其权重定义了对反应风格和安全姿态的隐性偏好，因此在线评分者扮演通常分配给奖励模型的角色。这种观点预测，高分数据可能系统性地偏向更长、更坚定、更顺从或更易拒绝的行为，这取决于在线评分的定义方式。从经验上看，在任务准确性上统计上无异的选择器在拒绝率、冗长度和谄媚度上有显著差异，我们证明了转变方向可从所选数据的属性组合中预测。我们介绍了对齐漂移审计（ADA），一种用于量化选择诱导行为动作的受控协议，以及对齐感知选择（AAS），一种诊断在线选择器，在限制安全轴和风格轴漂移的同时保持数据效率。

Gimitest: A Comprehensive Tool for Testing Reinforcement Learning Policies

Gimitest：测试强化学习策略的综合工具

• Authors: Dennis Gross, Quentin Mazouni, Helge Spieker, Arnaud Gotlieb
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.07029
• Pdf link: https://arxiv.org/pdf/2607.07029
• Abstract Reinforcement learning (RL) policies can be unsafe and vulnerable to attacks. Ensuring their reliability is often a pain point as existing automated testing methods target only selected environments, testing scenarios, and RL algorithms. To address this, we propose a comprehensive framework for testing single- and multi-agent RL policies under varying conditions. Our implementation of this framework, Gimitest, is an open-source tool that supports various gym frameworks and allows for modifications of their integrated components. This article describes the framework and details Gimitest's functionality and architecture. It showcases its effectiveness in testing multiple RL policies in environments such as the official Farama Gymnasium and PettingZoo.
• 中文摘要 强化学习（RL）策略可能不安全且易受攻击。确保其可靠性常常是一个痛点，因为现有的自动化测试方法只针对特定的环境、测试场景和强化学习算法。为此，我们提出了一个综合框架，用于在不同条件下测试单智能体和多智能体强化学习策略。我们实现的该框架Gimitest是一个开源工具，支持各种健身房框架，并允许修改其集成组件。本文介绍了该框架，并详细介绍了 Gimitest 的功能和架构。它展示了在官方法拉玛体育馆和宠物动物园等环境中测试多项强化学习策略的有效性。

ThermoDSE: A Thermal-Aware and Comprehensive Design Space Exploration for Chiplet-Based DNN Accelerators

ThermoDSE：基于芯片组的DNN加速器的热感知和综合设计空间探索

• Authors: Jian Peng, Hanwei Fan, Jingbo Jiang, Lin Jiang, Wei Zhang
• Subjects: Subjects: Hardware Architecture (cs.AR)
• Arxiv link: https://arxiv.org/abs/2607.07096
• Pdf link: https://arxiv.org/pdf/2607.07096
• Abstract Chiplet-based DNN accelerators provide a scalable path to balance performance and yield for modern AI workloads. However, such systems face critical challenges in area and thermal constraints. Design space optimization should jointly consider fine-grained task modeling, chiplet granularity, core granularity, and critical physical constraints. To the best of our knowledge, this is the first framework that involves all these factors. In this work, we propose ThermoDSE, a thermal-aware and comprehensive design space exploration framework for chiplet-based DNN accelerators. ThermoDSE integrates existing fine-grained modeling techniques into a unified simulation and optimization framework that jointly considers architecture design, task orchestration, and inter-chiplet communication under strict thermal and area constraints. Experimental results show that ThermoDSE achieves up to 3.5x improvement in Energy-Delay-Inverse-Yield, defined as E times D times inverse Y, compared with state-of-the-art Simba and other baselines. Furthermore, relative to simulated annealing and reinforcement learning-based methods, ThermoDSE converges to better design points with 3.7x and 29.4x runtime speedups, respectively.
• 中文摘要 基于芯片组的DNN加速器为现代AI工作负载提供了可扩展的路径，以平衡性能和产率。然而，这类系统在面积和热量限制方面面临严峻挑战。设计空间优化应结合细粒度任务建模、芯片组粒度、核心粒度及关键物理约束。据我们所知，这是第一个涵盖所有这些因素的框架。在本研究中，我们提出了ThermoDSE，一个基于芯片组的DNN加速器的热感知和全面设计空间探索框架。ThermoDSE将现有的细粒度建模技术整合为统一的仿真与优化框架，在严格的热和面积约束下共同考虑架构设计、任务编排和芯片间通信。实验结果显示，ThermoDSE在能量-延迟-反屈服率（定义为E乘以D乘以Y）方面，相比最先进的Simba及其他基线技术，提升了多达3.5倍。此外，相较于基于模拟退火和强化学习的方法，ThermoDSE在设计点上分别以3.7倍和29.4倍的运行加速收敛。

Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning

熵节奏策略优化用于多任务代理强化学习

• Authors: Zetian Hu, Shunyu Liu, Junjie Zhang, Yongcheng Jing, Ting-En Lin, Yongbin Li, Dacheng Tao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07178
• Pdf link: https://arxiv.org/pdf/2607.07178
• Abstract Recent breakthroughs of Reinforcement Learning (RL) have highlighted its potential for complex agentic Large Language Model (LLM) tasks. However, existing efforts largely focus on single-task settings, whereas real-world deployment necessitates a generalist agent capable of solving multiple tasks simultaneously. In this work, we identify a critical yet underexplored phenomenon in multi-task agentic RL: different tasks can exhibit exploration-exploitation pace mismatch. Specifically, easier tasks may converge early to low-entropy policies that hinder learning on harder tasks, while harder tasks can, in turn, push easier tasks back toward high-entropy exploration. This back-and-forth interaction creates inter-task entropy crossovers and frequent entropy spikes. Inspired by this observation, we introduce Entropy Pacing Policy Optimization (EPPO) for multi-task agentic LLMs, which coordinates entropy across tasks to stabilize multi-task optimization. At the core of EPPO is a task-wise dynamic clipping mechanism that replaces the fixed clipping threshold in Group Relative Policy Optimization (GRPO) with a task entropy-aware adaptive bound, tightening updates for over-confident tasks while relaxing them for under-explored ones. Experiments on the multi-task agentic benchmarks demonstrate that the proposed EPPO yields results superior to its counterparts.
• 中文摘要 强化学习（RL）的最新突破凸显了其在复杂代理大型语言模型（LLM）任务中的潜力。然而，现有工作主要集中在单一任务设置，而实际部署则需要一个能够同时解决多个任务的通用代理。本研究指出，我们识别出一个关键但尚未被充分探讨的多任务能动强化学习现象：不同任务可能表现出探索与利用速度不匹配。具体来说，较简单的任务可能提前趋向低熵政策，阻碍更难任务的学习，而更难的任务又可能将较简单的任务推回高熵探索方向。这种来回互动产生了任务间熵交叉和频繁的熵尖峰。受这一观察启发，我们引入了多任务智能体LLM的熵节奏策略优化（EPPO），它通过协调任务间的熵以稳定多任务优化。EPPO的核心是一个按任务进行动态裁剪机制，它用任务熵感知的自适应界限取代了Group Relative Policy Optimization（GRPO）中的固定裁剪阈值，收紧了过度自信任务的更新，同时对未被充分探索的任务则放宽了更新。多任务智能基准测试的实验表明，所提出的EPPO结果优于其他对应方案。

ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies

ORCAID：基于规则的斜向连续动作解释，用于深度强化学习策略

• Authors: Ignacio D. Lopez-Miguel, Ezio Bartocci, Thomas Eiter, Martin Tappler
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.07235
• Pdf link: https://arxiv.org/pdf/2607.07235
• Abstract Explainability remains a key issue in reinforcement learning (RL). Distilling an interpretable policy from an agent trained in a complex environment is particularly challenging when the action space is continuous. We introduce ORCAID, a novel method for extracting interpretable rule-based policies from RL agents operating in mixed continuous-discrete environments with continuous action spaces. Our main contribution is an efficient oblique decision tree training algorithm that partitions the state space by hyperplanes and fits local linear models. The key idea lies in a three-stage split search: efficient random initialization, local refinement, and backward elimination. Finally, adjacent leaves are merged to yield a concise set of interpretable rules describing a given deep RL policy. We evaluate ORCAID across multiple RL environments, demonstrating that the extracted rule-based policies maintain strong performance with a low number of parameters and can even be used to improve the performance of the original deep RL policy.
• 中文摘要 可解释性仍然是强化学习（RL）中的关键问题。当动作空间连续时，从在复杂环境中训练过的代理提炼出可解释的策略尤其具有挑战性。我们介绍了ORCAID，这是一种新颖的方法，用于从混合连续-离散环境下操作连续操作空间的强化学习代理中提取可解释的基于规则的策略。我们的主要贡献是一种高效的斜决策树训练算法，该算法通过超平面划分状态空间并拟合局部线性模型。核心思想在于三阶段的分拆搜索：高效的随机初始化、局部细化和向后消去。最后，相邻叶节点合并，生成一组简明可解释的规则，描述给定的深度强化学习策略。我们评估了ORCAID在多个强化环境中的应用，证明提取的基于规则的策略在低参数数量下依然保持强劲性能，甚至可以用于提升原始深度强化学习策略的性能。

Safe Reinforcement Learning using Ideas from Model Predictive Control

基于模型预测控制理念的安全强化学习

• Authors: Georg Schäfer, Jakob Rehrl, Stefan Huber, Simon Hirlaender
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.07252
• Pdf link: https://arxiv.org/pdf/2607.07252
• Abstract Reinforcement learning (RL) enables the synthesis of control policies directly from data, making it highly appealing for complex cyber-physical systems (CPSs) and robotics. A persistent challenge, however, is ensuring strict, hard safety constraints during the active learning phase. In real-world physical systems, violating mechanical limits can cause irreversible damage, necessitating that exploration remains strictly within safe operational regions. We propose a generalized framework that combines the adaptive, high-performance nature of deep reinforcement learning (DRL) with the formal safety guarantees of model predictive control (MPC). Using a mathematical model of the system dynamics, offline MPC computations define a feasible state-action space, representing all safe combinations of system states and control inputs that guarantee constraint satisfaction. During training and deployment, the RL agent's instantaneous actions are projected onto this globally verified feasible set via a safety filter. We systematically evaluate our generalized approach on a non-linear 1-DoF laboratory testbed, demonstrating successful exploration and stable policy convergence on physical hardware.
• 中文摘要 强化学习（RL）能够直接从数据综合控制策略，使其对复杂的网络物理系统（CPS）和机器人技术极具吸引力。然而，一个持续的挑战是确保在主动学习阶段严格且硬性的安全约束。在现实物理系统中，违反机械极限可能导致不可逆的损害，因此勘探必须严格限制在安全的操作区域内。我们提出了一个通用框架，结合了深度强化学习（DRL）的自适应高性能特性与模型预测控制（MPC）的形式安全保障。通过系统动力学的数学模型，离线MPC计算定义了一个可行的状态-作用空间，代表所有保证约束满足的系统状态和控制输入的安全组合。在训练和部署期间，强化学习代理的瞬时动作通过安全过滤器投射到该全球验证的可行集合上。我们在非线性1-DoF实验室测试平台上系统评估了这一广义方法，证明了在物理硬件上成功探索和策略趋同的稳定。

R^3: Advertisement Compliance Rectification via Group-Relative Experience Extractor and Curriculum Reinforcement

R^3：通过群体-亲属经验提取器和课程强化进行广告合规纠正

• Authors: Yuan Chen, Zhenyu Hu, Mengge Xue, Te Cao, Liqun Liu, Peng Shu, Huan Yu, Jie Jiang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07318
• Pdf link: https://arxiv.org/pdf/2607.07318
• Abstract Rigorous content moderation is crucial for online advertising but leads to millions of daily rejections. This scale renders manual rectification infeasible, particularly for video advertisements. However, existing safety-driven methods often suffer from aggressive over-editing, which compromises the advertiser's original semantic intent merely to satisfy compliance. In this work, we target the rectification of textual violations in video ads, covering both speech transcripts and on-screen text. We propose R^3, a novel framework designed to harmonize compliance with original semantic intent preservation. Our approach integrates three key innovations: (1) an experience-driven data synthesis framework that bootstraps high-quality supervision via a group-Relative compliance experience extractor; (2) a curriculum Reinforcement learning strategy with hierarchical rewards designed to enforce compliance while maximizing semantic consistency; and (3) a comprehensive video Rectification framework seamlessly integrating text recognition, rewriting, and re-rendering for industrial deployment. Extensive experiments on industrial datasets and online A/B testing demonstrate that R^3 significantly outperforms state-of-the-art baselines, achieving an optimal trade-off between violation rectification and intent preservation.
• 中文摘要 严格的内容审核对网络广告至关重要，但却导致每天数百万次拒绝。这种规模使得手工整流变得不可行，尤其是对于视频广告来说。然而，现有的安全驱动方法常常存在过度编辑，这破坏了广告主仅满足合规的原始语义意图。本研究重点是纠正视频广告中的文本违规，涵盖语音转录和屏幕文本。我们提出了R^3，这是一个旨在协调原始语义意图保持合规性的新框架。我们的方法整合了三项关键创新：（1）以经验为驱动的数据综合框架，通过群体-相对合规体验提取器自带高质量的监督;（2）课程强化学习策略，采用层级奖励，旨在强制遵守并最大化语义一致性;以及（3）一个全面的视频整流框架，无缝集成文本识别、重写和重渲染，适用于工业部署。工业数据集上的大量实验和在线A/B测试表明，R^3的表现显著优于最先进的基线，实现了违规纠正与意图保持之间的最佳权衡。

Towards Reliable Aerial Ground Vehicle Collaboration: An Integrated Planning and Autonomy Framework for Field Deployment

迈向可靠的空中地面载具协作：现场部署的综合规划与自主框架

• Authors: Md Safwan Mondal, Luca Russo, James D. Humann, James M. Dotterweich, Pranav Bhounsule
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.07350
• Pdf link: https://arxiv.org/pdf/2607.07350
• Abstract Limited flight endurance significantly restricts the operational range of unmanned aerial vehicles (UAVs) in long duration missions such as surveillance and inspection, where multiple spatially distributed Areas of Interest (AOIs) must be visited. These tasks require efficient routing determining the sequence of visits which directly impacts mission time, energy consumption, and overall feasibility. Pairing UAVs with unmanned ground vehicles (UGVs) for mobile recharging offers a promising solution, but introduces a tightly coupled cooperative routing problem involving UAV route planning, UGV road constrained movement, energy management, and rendezvous scheduling under uncertainty. In this work, we present an integrated planning and autonomy framework for reliable field deployment. We formulate the problem as an energy constrained cooperative routing task and solve it using a Deep Reinforcement Learning (DRL) based planner that jointly optimizes the UAV visitation sequence and rendezvous locations with the UGV, outperforming baseline heuristics in minimizing total mission time. To bridge the gap between planning and execution, we introduce a standardized two layer YAML based mission API that captures environment states and structures lightweight, synchronized action sequences. This framework is supported by a complete autonomy stack using PX4/MAVSDK for UAV control and ROS 2/Nav2 for UGV navigation. Furthermore, we propose a lightweight Rendezvous Aware Replanner (RARP) that operates online to handle environmental uncertainties, reducing energy margin violations from 83.33% to 20.00%. The full system is validated through outdoor field experiments, demonstrating robust cooperative navigation and adaptability in dynamic tasks, including a search and rescue scenario with vision language model (VLM) based hazard detection
• 中文摘要 飞行续航时间有限，极大限制了无人机（UAV）在长时间任务（如监视和检查）中的作战范围，这些任务需要访问多个空间分布的兴趣区（AOI）。这些任务需要高效的路线规划，确定访问顺序，这直接影响任务时间、能耗及整体可行性。将无人机与无人地面车辆（UGV）配对进行移动充电是一个有前景的解决方案，但也带来了紧密耦合的路由问题，涉及无人机路线规划、UGV道路受限的移动、能源管理和不确定条件下的会合安排。在本研究中，我们提出了一个集成的规划与自主性框架，以实现可靠的现场部署。我们将问题制定为一个能量受限的协作路由任务，并使用基于深度强化学习（DRL）的规划器，联合优化无人机访问顺序和与UGV的会合地点，在最小化总任务时间方面优于基线启发式。为了弥合规划与执行之间的差距，我们引入了基于YAML的标准化两层任务API，能够捕捉环境状态并构建轻量级、同步的动作序列。该框架由完整的自治协议栈支持，使用 PX4/MAVSDK 进行无人机控制，ROS 2/Nav2 用于 UGV 导航。此外，我们提出了一款轻量化的会面感知重规划器（RARP），在线运行以处理环境不确定性，将能差违规率从83.33%降至20.00%。通过户外实地实验验证了完整系统，展示了在动态任务中的稳健协作导航和适应性，包括基于视觉语言模型（VLM）的危险检测的搜救场景

BUS: Brain-Inspired Unsupervised Self-Reflection for Advanced Multimodal Reasoning

BUS：脑启发的无监督自我反思，用于高级多模态推理

• Authors: Jiacheng Yang, Tongying Xiao, Yunkai Dang, Cong Wang, Yuekun Yang, Qi Fan, Wenbin Li, Feng Miao, Yang Gao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.07361
• Pdf link: https://arxiv.org/pdf/2607.07361
• Abstract Current Vision-Language Models (VLMs) often struggle to handle complex visual tasks that require consistent and fine-grained reasoning. Recent methods aim to train models to facilitate self-reflective reasoning, i.e., reviewing and improving the generated reasoning. However, they require large volumes of annotated data and lack explicit reflective behavior during test time. This work aims to bridge this gap through inspiration from neuroscience. The human brain exhibits efficient backward prediction, i.e., predicting which current states are likely to precede a given future state. In this work, we first verify that mainstream VLMs can perform backward prediction, similar to the human brain. Then, we propose Brain-inspired Unsupervised Self-reflection (BUS), a label-free training framework to enhance reflective reasoning capability in challenging image analysis. BUS enables VLMs to perform backward prediction and provide explicit learning signals on data without ground-truth labels. In this way, BUS eliminates reliance on annotated data while improving reasoning performance. Notably, BUS is compatible with popular fine-tuning methods, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). Finally, extensive experiments on 8 benchmarks demonstrate the effectiveness of BUS across a wide range of complex visual tasks. It achieves notable improvements over the base model while using only unlabeled training data. Our experimental findings validate that backward prediction capability is critical for VLM reasoning.
• 中文摘要 当前的视觉语言模型（VLM）常常难以处理需要一致且细致推理的复杂视觉任务。最新方法旨在训练模型以促进自我反思推理，即审查和改进生成的推理。然而，它们需要大量注释数据，且在测试过程中缺乏明确的反射行为。本研究旨在通过神经科学的启发弥合这一鸿沟。人脑具有高效的向后预测能力，即预测哪些当前状态可能先于某一未来状态。本研究首先验证了主流VLM能够进行类似人脑的向后预测。随后，我们提出了脑启发的无监督自我反思（BUS），这是一种无标签的训练框架，旨在增强在复杂图像分析中的反思推理能力。BUS使VLM能够执行向后预测，并在没有地面真实标签的情况下对数据提供显式学习信号。通过这种方式，BUS消除了对注释数据的依赖，同时提升了推理性能。值得注意的是，BUS兼容流行的微调方法，如监督微调（SFT）和强化学习（RL）。最后，基于8个基准测试的广泛实验证明了BUS在广泛复杂视觉任务中的有效性。它在仅使用无标签训练数据的情况下，相较基础模型取得了显著改进。我们的实验发现验证了向后预测能力对VLM推理至关重要。

RLVP: Penalize the Path, Reward the Outcome

RLVP：惩罚路径，奖励结果

• Authors: Bojie Li, Noah Shi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07435
• Pdf link: https://arxiv.org/pdf/2607.07435
• Abstract Agents acting on our behalf in the real world (e.g. placing phone calls) must learn online from costly, often irreversible interactions rather than cheap simulator steps. Two things follow. First, deployability depends on the path, not only the outcome. An agent must respect outcome-neutral constraints such as not repeatedly calling an unresponsive user, respecting business hours, or completing required authentication constraints that outcome-based rewards cannot express, since violating them frequently improves apparent success. Second, because each interaction is expensive, the agent must learn efficiently from very few examples. Reinforcement learning from verifiable rewards (RLVR) is blind to both challenges: it optimizes solely on the outcome and wastes expensive rollouts on all-fail groups where group-relative advantage collapses to zero. Attempts to densify supervision by rewarding progress target the hard-to-verify direction. In contrast, real agentic environments can cheaply detect bad moves. Since group-relative advantage is equivalent to within-group variance, a dense signal helps only when it supplies variance the outcome lacks. A verifiable penalty on the path meets this condition reliably, while a progress potential helps only where partial progress is reachable. The resulting recipe "penalize the path, reward the outcome" achieves high task success with near-zero violations, where outcome-only training violates constraints on nearly every episode. We provide four design rules for effective penalties, including avoidance of the inaction trap that arises when a penalty is used in isolation.
• 中文摘要 在现实世界中代表我们行动的代理（例如拨打电话）必须通过在线学习昂贵且常常不可逆的互动，而非廉价的模拟器步骤。接下来有两点。首先，部署能力取决于路径，而不仅仅是结果。代理必须遵守与结果无关的约束，例如不反复呼叫无响应用户、遵守营业时间，或完成基于结果的奖励无法表达的认证约束，因为违反这些约束往往能提升表面上的成功率。其次，由于每次交互都代价高昂，智能体必须从极少的实例中高效学习。可验证奖励强化学习（RLVR）对这两个挑战视而不见：它仅优化结果，浪费昂贵的推广资源在全失败群体中，群体相对优势崩溃至零。通过奖励进步来加强监督的努力，针对难以验证的方向。相比之下，真实的代理环境可以廉价地发现错误的走法。由于群的相对优势等同于群内方差，密集信号只有在提供结果缺乏的方差时才有帮助。路径上的可验证惩罚可靠地满足这一条件，而进度潜力仅在部分进展可达时才有效。最终的“惩罚路径，奖励结果”的配方几乎无违规，任务成功率很高，而仅结果训练几乎每集都违反约束。我们提供了四条有效惩罚的设计规则，包括避免单独使用罚则时产生的无效陷阱。

EmbodiedGen V2: An Agentic, Simulation-Ready 3D World Engine for Embodied AI

EmbodiedGen V2：一款具备实体化、具备模拟能力的3D世界引擎

• Authors: Xinjie Wang, Liu Liu, Taojun Ding, Andrew Choi, Chaodong Huang, Mengao Zhao, Ziang Li, Jackson Jiang, Chunlei Yu, Shengxiang Liu, Wei Xu, Zhizhong Su
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.07459
• Pdf link: https://arxiv.org/pdf/2607.07459
• Abstract We present EmbodiedGen V2, a generative 3D world engine for building executable sim-ready environments for embodied intelligence. Sim-ready 3D asset generation has advanced rapidly, yet assembling such assets into policy-ready task environments remains largely manual, limiting scalable closed-loop learning. EmbodiedGen V2 addresses this gap through a unified sim-ready representation that connects cross-simulator assets, interaction affordances, task-driven worlds, large-scale multi-room scenes, and stateful Vibe Coding into a generative, editable, and reusable simulation pipeline. The generated environments support manipulation, navigation, mobile manipulation, cross-simulator deployment, and embodied policy training. In evaluation, the asset pipeline achieves 96.5% human acceptance and 98.6% collision success, and 83.3% of task-driven worlds are directly usable for downstream simulation without manual modification. Online reinforcement learning with generated environments further improves simulation success from 9.7% to 79.8%, and transfers to real robots with task success increasing from 21.7% to 75.0%. These results establish EmbodiedGen V2 as scalable simulation infrastructure for training, evaluating, and deploying embodied policies.
• 中文摘要 我们介绍EmbodiedGen V2，一款用于构建可执行模拟环境的生成式3D世界引擎，用于具身智能。模拟准备3D资产生成技术发展迅速，但将此类资产组装成策略准备任务环境仍大多手动完成，限制了可扩展的闭环学习。EmbodiedGen V2 通过统一的模拟就绪表示方式弥补了这一空白，将跨模拟器资产、交互功能、任务驱动世界、大规模多房间场景和有状态的 Vibe 编码连接成一个生成式、可编辑且可重复使用的仿真流水线。生成的环境支持操作、导航、移动操作、跨模拟器部署和具象政策培训。评估显示，该资产管线实现了96.5%的人类接受度和98.6%的碰撞成功率，且83.3%的任务驱动世界可直接用于下游模拟，无需手动修改。利用生成环境进行的在线强化学习进一步将模拟成功率从9.7%提升至79.8%，并转化为真实机器人，任务成功率从21.7%提升至75.0%。这些结果确立了 EmbodiedGen V2 作为可扩展的仿真基础设施，用于培训、评估和部署具象策略。

Reward-Adaptive Iterative Discovery: A Case Study on Automated Game Testing for NHL26

奖励自适应迭代发现：NHL26自动化游戏测试案例研究

• Authors: Florian Fuchs, Jessy Gosselin-Grant, Boris Skuin, Michele Petteni, Alessandro Sestini, Joakim Bergdahl, Amir Baghi, Linus Gisslén
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07498
• Pdf link: https://arxiv.org/pdf/2607.07498
• Abstract Testing is a major effort for the gaming industry, requiring a significant part of development budget and people power. We present a case study on a development version of the ice hockey game EA SPORTS NHL 26, for which human playtesters test the goalie AI for behavioral exploits. To reduce the effort of re-testing the goalie AI after every game or behavior modification in the development phase, we propose Reward-Adaptive Iterative Discovery (RAID), a novel approach to automatically find exploits using an iterative Reinforcement Learning (RL) approach that trains a population of goal scoring agents. While previous approaches can already successfully find exploits, RL algorithms tend to overfit to a single solution. We introduce a simple extension on top of existing RL algorithms, such that they find multiple diverse high-quality solutions. For our first deployment of this approach, within a single experiment we were able to find six hockey scoring exploit strategies that were qualitatively similar to those that playtesters had found in hours-long manual testing sessions.
• 中文摘要 测试是游戏行业的重要工作，需要大量开发预算和人力。我们展示了一个关于冰球游戏EA SPORTS NHL 26开发版本的案例研究，人类测试者测试守门员AI的行为漏洞。为了减少每场比赛后对守门员AI进行重测或开发阶段行为修改的工作，我们提出了奖励自适应迭代发现（RAID）的新方法，通过迭代强化学习（RL）方法自动寻找利用，训练一组得分代理。虽然以往方法已经能成功发现漏洞利用，但强化学习算法往往会过度拟合到单一解。我们在现有强化学习算法基础上引入了一个简单的扩展，使其能够找到多种多样化的高质量解。在我们首次部署这种方法时，在一次实验中，我们找到了六种冰球得分漏洞策略，这些策略在质上与测试者在长时间手动测试中发现的策略相似。

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

针对代理强化学习的单次推出异步优化

• Authors: Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07508
• Pdf link: https://arxiv.org/pdf/2607.07508
• Abstract Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).
• 中文摘要 强化学习（RL）在大型语言模型（LLMs）中变得越来越重要。此前用于大型语言模型的强化学习流水线大多是同步且批量交错的，这对于长期代理任务效率较低。近年来，异步强化学习作为一种更高效的替代方案出现，通过在推广时更新模型。然而，现有的异步强化学习系统往往强调吞吐量，而训练稳定性和任务效能则大多未被充分探讨。例如，一个关键挑战是广泛使用的GRPO框架中的群体抽样并不自然地适用于异步代理训练。本文介绍了单次部署异步优化（SAO），以解决异步强化学习的稳定性和非策略性挑战。为了减少非策略效应并提升泛化性，我们用单次展开抽样替代了按组抽样，即每个提示使用一次展开。我们通过实用的价值模型训练设计进一步改进了这一单次推广策略。为提升优化稳定性，我们引入严格的双面令牌级裁剪策略。SAO能够稳定训练一千步，并在代理编码和推理基准（如SWE-Bench Verified、BeyondAIME和IMOAnswerBench）上持续优于GRPO及其变体。我们还展示了单次推广强化学习在模拟在线学习环境中特别有效，因为模型必须适应不断变化的环境。为此，SAO已成功部署在agentic RL流水线中，用于训练开放GLM-5.2模型（750B-A40B）。

PHaul: A PPO-based forwarding agent for Sub6 enhanced Integrated Access and Backhaul networks

PHaul：基于PPO的转发代理，用于Sub6增强型综合接入和回程网络

• Authors: Jorge Pueyo, Daniel Camps-Mur and, Miguel Catalan-Cid
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2607.07584
• Pdf link: https://arxiv.org/pdf/2607.07584
• Abstract 3GPP Integrated Access and Backhaul (IAB) allows operators to deploy outdoor mm-wave access networks in a cost-efficient manner, by reusing the same spectrum in access and backhaul. In IAB networks the performance bottleneck is the wireless backhaul segment, where efficient forwarding strategies are needed to effectively use the available capacity. In addition, the performance of the mm-wave IAB backhaul segment is contingent on the availability of line of sight (LoS) conditions in the selected deployment sites. To mitigate LoS dependence, in this paper, we propose to complement the mm-wave backhaul segment of IAB networks with additional Sub6 backhaul links, which contribute to the capacity and robustness of the backhaul network. We refer to IAB networks combining Sub6 and mm-wave links in the backhaul as Sub6 enhanced IAB networks. In this context, the main contribution of this paper is PHaul, a forwarding engine for Sub6 enhanced IAB networks that accomodates different traffic engineering criteria, and combines an offline path selection heuristic with an online Deep Reinforcement Learning (DRL) agent based on Proximal Policy Optimization (PPO). By leveraging a network digital twin of the IAB wireless backhaul, PHaul periodically samples the input traffic of the backhaul network and updates flow to path mappings, with execution times below 10 seconds in realistic backhaul topologies. We present an exhaustive performance evaluation, where we demonstrate that PHaul can achieve gains of up to 36\% in throughput efficiency and of up to 20\% in fairness, when compared against two alternative heuristics in a wide range of network configurations. We also demonstrate that PHaul is robust to differences between the network topologies considered in the training and inference phases, which can occur in practice due to link failures.
• 中文摘要 3GPP集成接入与回传（IAB）允许运营商以成本效益高的方式部署户外毫米波接入网络，通过重复利用同一频谱进行接入和回传。在IAB网络中，性能瓶颈是无线回传段，需要高效的转发策略以有效利用可用容量。此外，毫米波IAB回传段的性能取决于所选部署站点的视距（LoS）条件可用性。为减少视信号依赖，本文提出用额外的Sub6回传链路来补充IAB网络的毫米波回传段，从而增强回程网络的容量和稳健性。我们将回程中结合Sub6和毫米波链路的IAB网络称为Sub6增强型IAB网络。在此背景下，本文的主要贡献是PHaul，这是一个面向Sub6增强型IAB网络的转发引擎，能够满足不同的流量工程标准，并将离线路径选择启发式与基于近端策略优化（PPO）的在线深度强化学习（DRL）代理结合起来。通过利用IAB无线回传的网络数字孪生，PHaul定期采样回程网络的输入流量，并将流量更新为路径映射，在真实回程拓扑下执行时间低于10秒。我们进行了详尽的性能评估，证明PHaul在与两种不同网络配置的启发式方法相比，能够实现最高36%的吞吐量效率提升和高达20%的公平性。我们还证明了PHaul对训练阶段和推理阶段考虑的网络拓扑差异具有鲁棒性，这些差异在实际中可能因链路故障而发生。

Unlearning to Protect: A Distilled Reinforcement Learning Framework with Privacy-Preserving Feature Unlearning and XAI for IoT Security

去学习保护：一个精炼强化学习框架，具隐私保护功能 Unlearning和XAI用于物联网安全

• Authors: Md. Nahid Hasan, Golam Rabiul Alam
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2607.07635
• Pdf link: https://arxiv.org/pdf/2607.07635
• Abstract Botnets pose a significant cybersecurity threat, enabling attacks such as DDoS, data theft, and service disruptions on IoT devices. These devices often lack built-in botnet traffic filtering, leaving them highly exposed. Existing AI-based solutions improve detection capabilities but have limitations: (i) they are too heavy for IoT deployment, and (ii) they lack unlearning capabilities to forget sensitive or outdated features without retraining. To address these challenges, we propose DiRLU, a lightweight, reinforcement learning driven framework, while ensuring privacy by selectively unlearning sensitive or outdated features without requiring retraining. The framework leverages knowledge distillation to transfer knowledge from a teacher model into a lightweight student model, with both models trained using A2C. A post-hoc unlearning mechanism modifies weights to remove targeted features, while restored features show negligible performance loss, confirming reversibility. Unlike many benchmark models that used only 5% of the BoT-IoT dataset, this research leverages 25%, allowing us to develop a strong teacher model. Both the teacher and student models were trained using the A2C reinforcement learning algorithm, achieving impressive results, with the student model achieving 99.60% accuracy and a 99.80% F1 score. To enhance transparency, we integrated Explainable AI (XAI), particularly LIME, which helps interpret the model's decisions and identify the key features influencing its predictions. Moreover, DiRLU requires only 2,370 FLOPS, approximately 3.87x more efficient than the state-of-the-art model, highlighting its efficiency for edge deployment. DiRLU combines efficiency with privacy, aligning with GDPR standards (right to be forgotten) to provide practical and scalable IoT security solution.
• 中文摘要 僵尸网络构成重大网络安全威胁，可能导致DDoS、数据盗窃和物联网设备服务中断等攻击。这些设备通常缺乏内置的僵尸网络流量过滤功能，导致它们高度暴露。现有基于AI的解决方案提升了检测能力，但存在局限性：（i）它们对物联网部署来说过于笨重，（ii）缺乏去除学习能力，无法在不重新训练的情况下忘记敏感或过时的功能。为应对这些挑战，我们提出了DiRLU，一个轻量级、以强化学习为驱动的框架，同时通过选择性地移除敏感或过时的功能，确保隐私，无需重新训练。该框架利用知识蒸馏技术将教师模型的知识转移到轻量级学生模型中，两者均使用A2C进行训练。事后还学习机制修改权重以移除目标特征，而恢复的功能性能损失可忽略不计，确认了可逆性。与许多仅使用5%的BoT-IoT数据集的基准模型不同，本研究利用了25%，使我们能够构建出强大的教师模型。教师和学生模型均使用A2C强化学习算法训练，取得了令人印象深刻的成绩，学生模型准确率达到99.60%，F1评分为99.80%。为提升透明度，我们整合了可解释人工智能（XAI），特别是LIME，帮助解读模型决策并识别影响预测的关键特征。此外，DiRLU仅需2370 FLOPS操作，比最先进模型高效约3.87倍，凸显其边缘部署效率。DiRLU将效率与隐私相结合，符合GDPR标准（被遗忘的权利），提供实用且可扩展的物联网安全解决方案。

Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

Agon：带有隐性竞争性竞争评分的跨模型强化学习

• Authors: Vladislav Beliaev
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.07690
• Pdf link: https://arxiv.org/pdf/2607.07690
• Abstract Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.
• 中文摘要 基于可验证奖励（如GRPO）的强化学习是当今推理模型的引擎，但它只评分最终答案。在困难问题上，这训练模型写得更多，而非思考得更好，因为描图本身从未被评分，也没有良好思考的标签。我们引入了Agon，它使两个竞争模型相互评分。两者都尝试解决同样的问题;在轮流扮演的角色中，一人起草解答，另一人边解边读，双方因解答优于对方而获得奖励。要获胜，模型必须用理性击败了见过其工作的竞争对手，因此在培训过程中，推理是隐含的，没有过程标签和奖励模型。由于两种模型都经过优化，它们面临着越来越强的竞争对手，而单一模型强化学习无法满足这一需求。两者只需力量相当且行为不同即可。推理时，这对模型在训练时部署，这是一个两阶段的级联，一个模型先草稿，另一个模型在阅读草稿后回答。在深度数学与Qwen3的硬分裂中，这使GRPO的pass@1翻倍，约为同一基底未训练混合剂传输的八倍。该排序在竞争性编程代码和模型族（Qwen3.5，Gemma 4）上可重复。目前模特们用文字交流;下一步是让他们在潜在空间里一起推理。

Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

选择性时间步加权和基于优势的重放用于样本高效扩散RLHF

• Authors: Eric Zhu, Abhinav Shrivastava, Soumik Mukhopadhyay
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.07693
• Pdf link: https://arxiv.org/pdf/2607.07693
• Abstract Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to diffusion models remains highly feedback inefficient, as existing approaches typically require large amounts of human or reward model evaluations. This limitation reduces the practicality of diffusion RLHF in realworld settings where feedback is the primary bottleneck. In this paper, we propose two complementary strategies that substantially improve the feedback efficiency of diffusion RLHF while preserving generalization to unseen prompts. Our key observation is that reward information in diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning from a reward signal. By emphasizing informative timesteps and trajectories during optimization, we obtain more effective gradient updates. First, we introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. We theoretically connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and approximate the resulting weighting trend empirically. Second, we introduce a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly improve the feedback efficiency of diffusion RLHF. Under identical hyperparameter settings, our approach achieves up to a 6$\times$ improvement in sample efficiency compared to widely used diffusion RLHF baselines.
• 中文摘要 来自人类反馈的强化学习（RLHF）已成为一种强大的范式，用于将生成模型与人类偏好对齐。然而，将RLHF应用于扩散模型仍然高度低效，因为现有方法通常需要大量人工或奖励模型评估。这一限制降低了扩散RLHF在反馈作为主要瓶颈的实际环境中的实用性。本文提出了两种互补策略，显著提升扩散RLHF的反馈效率，同时保持对未见提示的泛化性。我们的关键观察是，扩散轨迹中的奖励信息分布不均：并非所有去噪时间步或轨迹对从奖励信号中学习贡献均等。通过在优化过程中强调有意义的时间步和轨迹，我们获得了更有效的梯度更新。首先，我们引入了一种每时间步的加权方案，用于在策略优化过程中重新加权去噪步骤。我们理论上将这种加权与近端策略优化（PPO）的最优收敛性质联系起来，并对由此产生的加权趋势进行实证近似。其次，我们引入了一种重放机制，优先考虑信息轨迹，使模型能够重用过去的样本，而无需反复查询新的奖励。这些策略共同显著提升了扩散RLHF的反馈效率。在相同的超参数设置下，我们的方法相比广泛使用的扩散RLHF基线，样品效率提升可达6$\倍$。

Keyword: diffusion policy

GeoProp: Grounding Robot State in Vision for Generalist Manipulation

GeoProp：在视觉中接地机器人状态以实现通用操作

• Authors: Guoyang Zhao, Quanhao Qian, Gongjie Zhang, Wenhao Li, Jiuniu Wang, Xiaowei Lu, Deli Zhao, Ran Xu
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07101
• Pdf link: https://arxiv.org/pdf/2607.07101
• Abstract Proprioception is fundamental to robotic manipulation, yet standard fusion methods often treat it as an isolated vector lacking explicit alignment with visual tokens. Without a direct correspondence between 3D kinematics and 2D feature maps, manipulation policies struggle to ground the robot's state within the scene, frequently underperforming even vision-only baselines. To address this, we introduce GeoProp, a lightweight, plug-and-play adapter that aligns proprioception with vision through explicit geometric grounding and spatial feature sampling. GeoProp projects the robot state onto the image plane to sample localized visual features, constructing a grounded state token. It then injects state-derived spatial priors into the corresponding visual features via FiLM modulation. To capture motion intent, GeoProp further samples features at a short-horizon predicted coordinate derived from recent kinematics, providing look-ahead visual context. Across 67 tasks, GeoProp improves Diffusion Policy by 8.7% on 63 simulation tasks and pi_0 by 4.0% on the RoboTwin subset, and yields a 10.6% average gain across both policy families in the real world, while adding only 2-3% to the parameter count. These results demonstrate that GeoProp is a simple yet high-impact inductive bias for generalist embodied policies. Project page: this https URL.
• 中文摘要 本体感觉是机器人操作的基础，但标准融合方法通常将其视为孤立的向量，缺乏与视觉符号的明确对齐。由于三维运动学与二维特征图缺乏直接对应，操作策略难以将机器人状态扎根于场景中，常常表现不佳，甚至仅凭视觉的基线也难以实现。为此，我们引入了GeoProp，一款轻量化的即插即用适配器，通过显式几何基础和空间特征采样，将本体感觉与视觉对齐。GeoProp 将机器人状态投射到图像平面上，以采样局部视觉特征，构建有基状态标记。然后通过FiLM调制，将状态导出的空间先验注入相应的视觉特征中。为了捕捉运动意图，GeoProp 进一步采样了基于近期运动学预测坐标的短视距特征，提供前瞻性的视觉上下文。在67个任务中，GeoProp 在63个模拟pi_0任务中提升了扩散策略8.7%，在机器人双胞胎子集上提升了4.0%，在现实世界中，两个政策家族的平均提升均为10.6%，而参数数量仅增加了2-3%。这些结果表明，GeoProp 是一种简单但高影响力的归纳偏见，适用于通用的具象政策。项目页面：这个 https URL。