生成时间: 2026-08-12 17:13:32 (UTC+8); Arxiv 发布时间: 2026-08-12 20:00 EDT (2026-08-13 08:00 UTC+8)

今天共有 33 篇相关文章

Keyword: reinforcement learning

SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning

洞察未来：深度强化学习的前瞻性解释

• Authors: Tamar Gozlan, Claudia V. Goldman
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.09967
• Pdf link: https://arxiv.org/pdf/2608.09967
• Abstract Deep reinforcement learning (DRL) agents achieve strong performance in complex environments, yet their decision-making processes remain difficult to interpret. We introduce SPOT (Sampling Policy Observation Tree), a novel model-agnostic, sampling-based framework for interpreting DRL policies. Given access to the policy and an environment simulator, SPOT constructs an interpretable finite-horizon tree by sampling actions and recursively simulating the resulting successor states. The tree provides an empirical representation of the policy's action preferences and their possible downstream evolution. We provide formal guarantees establishing SPOT's asymptotic recovery of the policy's unique most probable action and characterizing its disagreement behavior under high-entropy policies. We demonstrate SPOT in the SUMO-RL traffic-signal control domain. The case study illustrates how its tree-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single-timestep feature-attribution methods.
• 中文摘要 深度强化学习（DRL）智能体在复杂环境中表现优异，但其决策过程仍然难以解读。我们介绍了SPOT（抽样策略观察树），这是一个新型的模型无关、基于抽样的DRL策略解释框架。在获得策略和环境模拟器的访问权限后，SPOT 通过采样动作并递归模拟所得的后继状态，构建了可解释的有限视界树。该树提供了策略行动偏好及其可能的后续演变的实证表现。我们提供了形式保证，确立SPOT对该策略唯一最可能行为的渐近恢复，并描述了其在高熵政策下的分歧行为。我们在SUMO-RL交通信号控制领域演示了SPOT。案例研究展示了其基于树的表示如何用于检查政策偏好、比较未来替代轨迹，并揭示单时间步特征归因方法无法察觉的下游行为。

Navigating the Proximity-Safety Balance: Constraint Decomposition for Human Following in Pedestrian Crowds

驾驭接近与安全平衡：行人人群中人类跟随的约束分解

• Authors: Shiting Gong, Jianpeng Yao, Jinfeng Wang, Marco Pavone, Jiachen Li
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.10056
• Pdf link: https://arxiv.org/pdf/2608.10056
• Abstract Following a target human in crowded environments involves an inherent conflict between staying close to the target and navigating safely among surrounding pedestrians and obstacles. This conflict becomes more severe in dense scenarios, where aggressive following risks collisions and conservative margins lead to target loss, especially when pedestrian behaviors are unfamiliar or unpredictable. Existing reinforcement learning (RL) methods typically encode these competing objectives into a single dense reward, but the resulting proximity-safety balance is implicit and difficult to adjust across conditions. To address this, we decompose the human-following task into a sparse task reward and independent cost constraints within a multi-constraint RL formulation, where each constraint is managed through cost thresholds with direct behavioral meaning rather than implicit reward weight ratios, allowing explicit and tunable control over the trade-off. We further quantify the prediction uncertainty of human motions and integrate these estimates into the RL costs to enhance safety under unpredictable conditions. Extensive experiments across both in-distribution and out-of-distribution settings demonstrate that our method achieves an effective proximity-safety balance compared to baselines. Real-robot deployment further validates the feasibility of our method in real-world scenarios. More details are available on our project page: this https URL.
• 中文摘要 在拥挤环境中跟踪目标人类，涉及紧贴目标与安全穿越周围行人和障碍物之间的内在冲突。在密集场景中，这种冲突更加严重，激进跟随风险、碰撞和保守的余距会导致目标丢失，尤其是在行人行为陌生或不可预测时。现有的强化学习（RL）方法通常将这些竞争目标编码为单一的密集奖励，但由此产生的接近与安全平衡是隐性的，且难以在不同条件下调整。为此，我们将人跟随任务分解为稀疏任务奖励和独立成本约束，采用多约束强化学习（RL）形式，每个约束通过具有直接行为意义的成本阈值管理，而非隐含的奖励权重比，从而实现显式且可调节的权衡控制。我们进一步量化人体运动的预测不确定性，并将这些估算整合到强化学习成本中，以提升不可预测条件下的安全性。在分布内外环境下的广泛实验表明，我们的方法相较于基线实现了有效的邻近安全平衡。真实机器人的部署进一步验证了我们方法在现实场景中的可行性。更多详情请见我们的项目页面：https 网址。

CHORUS: Complementary Experts for High-Coverage Testbench Stimulus Generation

合唱：高覆盖测试台刺激生成的补充专家

• Authors: Hejia Zhang, Sheng Lu, Zhongming Yu, Chia-Tung Ho, Brucek Khailany, Jishen Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.10090
• Pdf link: https://arxiv.org/pdf/2608.10090
• Abstract Large language models (LLMs) have advanced code generation, where executable feedback provides a more reliable learning signal than textual imitation alone. Hardware verification is an important application of code generation and accounts for a substantial fraction of modern chip design effort, with high-coverage testbench stimulus generation as a key task. We present CHORUS, a post-training framework that pushes performance beyond what a conventional supervised fine-tuning (SFT)-to-reinforcement learning (RL) pipeline achieves. CHORUS builds on two observations. First, staged SFT produces behaviorally diverse checkpoints, and dense-reward RL turns them into strong experts with comparable aggregate performance but distinct task-level strengths. Second, these complementary strengths can be exploited through either training-free model merging or further post-training to outperform the best individual expert. By consolidating the resulting specialists into a single 4B model, CHORUS achieves 88.0% Pass@1 on CVDP-ECov, outperforming DeepSeek-R1 (671B) by 13.5 percentage points.
• 中文摘要 大型语言模型（LLM）拥有先进的代码生成能力，可执行反馈比单纯的文本模仿更为可靠地提供学习信号。硬件验证是代码生成的重要应用，占现代芯片设计工作的很大一部分，其中高覆盖率测试平台刺激生成是关键任务。我们介绍了CHORUS，一种培训后框架，将绩效提升到传统监督式微调（SFT）到强化学习（RL）流程之外的水平。《合唱团》基于两个观察。首先，分阶段SFT产生行为多样化的检查点，密集奖励强化学习将他们培养为具备强大专家的强大专家，整体表现相当但任务层面优势不同。其次，这些互补优势可以通过无训练模型合并或进一步培训后优化来超越最佳个别专家。通过将所有专家整合到单一的4B模型中，CHORUS在CVDP-ECov的应用中获得了88.0%的Pass@1，比DeepSeek-R1（671B）高出13.5个百分点。

Procedural Fairness Failures in RLHF from Preference Averaging

RLHF中偏好平均法中的程序公平性失败

• Authors: M P V S Gopinadh, Karthik Kamuju, Kummari Avinash, John Joshua, Srinivasa Raju Rudraraju
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.10126
• Pdf link: https://arxiv.org/pdf/2608.10126
• Abstract Reinforcement Learning from Human Feedback (RLHF) aggregates heterogeneous preferences into a single reward model, assuming preference homogeneity. When preferences are heterogeneous, this aggregation induces a procedural fairness failure where majority preference groups dominate reward learning while minority preferences are systematically under-represented. This work defines procedural fairness in alignment as preserving distinct preference signals during reward modeling and shows that standard RLHF violates this via preference averaging. Preference-Aware RLHF (PA-RLHF) is introduced, separating optimization across preference modes at the reward learning stage. In a controlled setting, PA-RLHF improves overall alignment accuracy from 46.9% to 67.9% and reduces the fairness gap between best and worst aligned groups from 15.9 to 9.6 percentage points. These results show that procedural fairness failures in alignment can arise from structural design choices in reward learning, even in controlled, noise-free settings, with direct implications for large language models and agentic systems, where biased reward models can compound inequities across sequential decisions.
• 中文摘要 人类反馈强化学习（RLHF）将异质偏好聚合到单一奖励模型中，假设偏好同质性。当偏好异质时，这种聚合会导致程序公平性失败，即多数偏好群体主导奖励学习，而少数偏好系统性地被低估。这项工作将程序性公平性定义为在奖励建模过程中保留不同的偏好信号，并证明标准RLHF通过偏好平均法违反了这一原则。引入了偏好感知 RLHF（PA-RLHF），在奖励学习阶段将偏好模式间的优化分离。在受控环境中，PA-RLHF将整体比对准确率从46.9%提升至67.9%，并将最佳与最差比对组之间的公平性差距从15.9%缩小到9.6个百分点。这些结果表明，过程公平性对齐失败可能源于奖励学习中的结构设计选择，即使在受控、无噪声环境中，这对大型语言模型和智能系统有直接影响，因为偏置的奖励模型会加剧顺序决策间的不平等。

ConnectionMind: Leveraging Social Networks and Large Language Models for Personalized Recommendation at Meta

ConnectionMind：利用社交网络和大型语言模型实现Meta个性化推荐

• Authors: Haoyu Han, Yuming Liu, Lei Huang, Lizhu Zhang, Jiliang Tang, Xiangjun Fan
• Subjects: Subjects: Information Retrieval (cs.IR); Social and Information Networks (cs.SI)
• Arxiv link: https://arxiv.org/abs/2608.10187
• Pdf link: https://arxiv.org/pdf/2608.10187
• Abstract Modern recommendation systems on social media platforms such as Meta must model complex social relationships, including friendships, group memberships, and creator interactions, alongside massive and heterogeneous content such as text and video. Traditional recommendation models, however, often omit these signals or treat them independently, lacking the reasoning capability to integrate multi-relational context for fine-grained personalization. We present ConnectionMind, a production-ready recommendation framework that tightly integrates the social network structure with large language models (LLMs) to enable scalable, interpretable, and reasoning-aware personalization in Meta. ConnectionMind constructs a heterogeneous graph connecting users, items, friends, groups, and creator pages, and formulates recommendation as a graph reasoning problem: discovering personalized paths from users to candidate items. An LLM-based policy is employed to reason over these graph structures and guide recommendation decisions. To train the system at scale, ConnectionMind adopts a two-stage learning strategy. We first perform supervised fine-tuning (SFT) on large-scale user-item interaction trajectories to initialize the reasoning policy, followed by end-to-end reinforcement learning (RL) to refine the model's ability to reason over social graphs for personalized recommendation. Extensive experiments on multiple real-world datasets demonstrate the effectiveness of ConnectionMind compared to representative baselines. More importantly, ConnectionMind has been deployed in Meta's large-scale recommendation pipeline and has been evaluated through online A/B tests, achieving a 0.43% improvement in video watch time. These results demonstrate measurable real-world impact in a production recommendation system.
• 中文摘要 像Meta这样的社交媒体平台的现代推荐系统必须建模复杂的社交关系，包括友谊、群组成员和创作者互动，同时还要支持大量且异质的内容，如文本和视频。然而，传统的推荐模型往往省略这些信号或单独处理，缺乏整合多关系上下文以实现细粒度个性化的推理能力。我们介绍ConnectionMind，一个面向生产环境的推荐框架，紧密集成社交网络结构与大型语言模型（LLM），实现Meta中的可扩展、可解释性和推理感知个性化。ConnectionMind 构建了一个异构图，连接用户、物品、好友、群组和创作者页面，并将推荐作为图推理问题提出：发现用户到候选物品的个性化路径。基于LLM的策略用于推理这些图结构并指导推荐决策。为了大规模训练系统，ConnectionMind 采用了两阶段学习策略。我们首先对大规模用户-项目交互轨迹进行监督微调（SFT）以初始化推理策略，随后进行端到端强化学习（RL），以优化模型在社交图谱上推理以实现个性化推荐的能力。在多个真实世界数据集上的大量实验展示了ConnectionMind相较于代表性基线的有效性。更重要的是，ConnectionMind已被纳入Meta大规模推荐流程，并通过在线A/B测试进行了评估，视频观看时间提升了0.43%。这些结果展示了生产推荐系统中可衡量的实际影响。

Boundary-Seeking Policy Gradient for Safe Reinforcement Learning

安全强化学习的边界寻求策略梯度

• Authors: Chenhua Fan, Jiahui Zhu, Yuhang Zhang, Honghao Wei
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.10204
• Pdf link: https://arxiv.org/pdf/2608.10204
• Abstract Safe reinforcement learning maximizes reward subject to safety constraints. For Constrained Markov Decision Processes, the linear-programming view over occupancy measures implies that whenever the constraint is active at optimality, the optimal policy lies exactly on the constraint boundary, yet standard gradient-based methods do not exploit this structure and often settle in the feasible interior. We introduce Boundary-Seeking Policy Gradient (BSPG), a first-order method whose update combines a tangential component that improves reward while preserving cost to first order with a signed, residual-driven normal component that regulates the policy toward the active boundary from either side; the combined direction admits an algebraic Lagrangian form with an induced coefficient and no learned dual variable. Under exact gradients and stated regularity conditions, the constraint residual converges to zero from either side with a finite-horizon $O(1/\sqrt{T})$ bound, the tangential component is a reward-ascent direction on the boundary, and any convergent parameter sequence is stationary on the active constraint set, satisfying the KKT conditions when the limit is also a local maximizer over the feasible set. This complements existing analyses, which certify feasibility but do not characterize the constraint value at convergence. On a standard Safety-Gymnasium navigation task, BSPG attains higher reward while tracking the boundary more tightly than the compared baselines.
• 中文摘要 安全强化学习在安全约束下最大化奖励。对于受限马尔可夫决策过程，线性规划视角对占用度量意味着，当约束在最优状态下激活时，最优策略恰好位于约束边界上，但基于梯度的标准方法并未利用该结构，且通常停留在可行的内部。我们引入了边界寻求政策梯度（BSPG），这是一种一阶方法，其更新结合了一个切向分量（提升奖励且保持成本至一阶）与一个带符号、残差驱动的正常分量，该分量调节政策向主动边界的任一侧;合并方向存在一个代数拉格朗日形式，具有诱导系数且没有学习到的对偶变量。在精确梯度和规定的正则条件下，约束残差从任一侧收敛为零，界限有限视界$O（1/\sqrt{T}）$，切向分量是边界上的奖励-上升方向，任何收敛参数序列在活动约束集上静止，满足KKT条件，当极限也是可行集的局部最大化子时。这补充了现有分析，现有分析证明可行性，但不表征收敛时的约束值。在标准的Safety-Gymnasium导航任务中，BSPG在跟踪边界时比对比基线更为严格，从而获得更高的奖励。

Whole-Body Planning for Humanoids Navigating Confined Spaces via Self-Collision Avoidance References

通过自我碰撞避免参考，为人形生物在狭窄空间中导航的全身规划

• Authors: Carlos Gonzalez, Luis Sentis
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.10220
• Pdf link: https://arxiv.org/pdf/2608.10220
• Abstract Humanoid locomotion in highly confined environments requires navigating dense environmental obstacles and complex self-collision bounds while maintaining multi-contact dynamic feasibility. Traditional trajectory optimizers frequently struggle in these restricted spaces, as navigating the large collision space with splines on particle abstractions is insufficient and leads to poor local minima. To address this, we propose a three-stage whole-body planning framework that formulates kinematic path planning directly over kinematically reachable rigid-body volumes. By integrating differentiable collision avoidance into a reachability-constrained formulation, our framework synthesizes volume-informed guides that reliably guide a full-order trajectory optimizer over long horizons. We show that these optimized plans serve as high-quality references to train a residual reinforcement learning policy for robust online execution. We validate our approach on the Unitree G1 humanoid across three benchmark testbeds exceeding NIST emergency response standards, achieving restricted confinement ratios ($C_r < 1.5$). Our framework generates feasible trajectories across 12-to-18-second tasks with complex foot and hand contacts where standard baselines fail, while the learned policy successfully tracks these plans under extensive domain randomization in physics simulation.
• 中文摘要 在高度受限环境中进行类人机动，需要在保持多接触动态可行性的前提下，穿越密集的环境障碍和复杂的自碰撞界限。传统的轨迹优化器在这些受限空间中常常遇到困难，因为在粒子抽象中用样条在大碰撞空间中导航不足，导致局部极小值较差。为此，我们提出了一个三阶段的全体规划框架，直接在运动学上可达的刚体体积上制定运动路径规划。通过将可微碰撞避免整合进可达性约束的表述中，我们的框架合成了以体积为导向的导引，能够可靠地引导全阶轨迹优化器在长视野内运行。我们表明，这些优化计划作为高质量参考，用于训练残余强化学习策略，实现稳健的在线执行。我们在Unitree G1人形机器人上验证了方法，在三个基准测试平台上超越了NIST应急响应标准，实现了限制约束比（$C r<1.5美元）。我们的框架能够生成12至18秒任务中、复杂足部和手部接触的可行轨迹，这些任务标准基线无法实现，而所学策略则在物理模拟中通过广泛的领域随机化成功跟踪这些计划。

Topological Feasibility Guarantees for Differentiable Predictive Control

拓扑可行性保证可微预测控制

• Authors: Guangyu Wu, Ján Drgoňa
• Subjects: Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.10332
• Pdf link: https://arxiv.org/pdf/2608.10332
• Abstract Differentiable predictive control (DPC), a self-supervised learning approach for approximating explicit model predictive control (MPC) policies, offers significant computational advantages over online optimization-based MPC. However, feasibility guarantees, a core requirement for safe control, are currently provided either probabilistically or via online safety filters. The lack of rigorous feasibility guarantees for offline policy optimization remains an open problem. This paper establishes deterministic feasibility guarantees for DPC using a novel topological analysis of the induced reachable safe set, without requiring online safety filters. By exploiting the inherent model-based nature of DPC, in which differentiable system dynamics are embedded directly into the computational graph, we analyze the properties of the learned control policies and the corresponding system states from topological and geometric perspectives. Inspired by our theoretical analysis, we propose a novel self-supervised offline policy learning strategy that utilizes a proxy loss with Control Barrier Functions (CBFs). Crucially, these properties not only significantly improve policy training but also enable the derivation of strict, deterministic feasibility guarantees from a finite number of training samples. Extensive closed-loop simulations validate our theoretical findings, demonstrating that the empirical constraint violations monotonically decrease to zero as the training sample size increases. Ultimately, this work illustrates that DPC policy optimization yields formal safety certificates that are structurally unattainable with conventional black-box methods, e.g., reinforcement learning (RL) or supervised learning-based approximate MPC, thereby providing a new perspective on feasibility guarantees in learning-based control.
• 中文摘要 可微预测控制（DPC）是一种自监督学习方法，用于近似显式模型预测控制（MPC）策略，相比基于在线优化的MPC在计算上具有显著优势。然而，可行性保证作为安全控制的核心要求，目前要么以概率方式提供，要么通过在线安全过滤器提供。离线策略优化缺乏严格可行性保障仍是一个悬而未决的问题。本文通过对诱导可达安全集进行新颖拓扑分析，无需在线安全过滤器，建立了DPC的确定性可行性保证。通过利用DPC固有的基于模型的特性——可微系统动力学直接嵌入计算图中——我们从拓扑和几何视角分析了学习控制策略的属性及相应系统状态。受我们的理论分析启发，我们提出了一种新颖的自监督离线政策学习策略，利用代理损失与控制障碍函数（CBFs）。关键是，这些特性不仅显著提升了政策培训，还使得从有限数量的训练样本中推导出严格的确定性可行性保证。大量的闭环模拟验证了我们的理论发现，证明随着训练样本量的增加，经验约束违背会单调地降至零。最终，这项工作表明，DPC策略优化产生了传统黑箱方法（如强化学习（RL）或监督式基于学习的近似MPC）结构上无法实现的正式安全证书，从而为基于学习的控制中的可行性保证提供了新的视角。

Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks

长远工具使用代理任务的高效强化学习

• Authors: Zelei Cheng, Amritansh Mishra, Sambit Sahu, William Campbell
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10357
• Pdf link: https://arxiv.org/pdf/2608.10357
• Abstract Long-horizon tool-using agents must reason over user goals, domain policies, tool calls, simulator state, and delayed verifiable rewards. Reinforcement learning (RL) is a natural fit for this setting, but multi-turn on-policy rollouts create long contexts, while model-specific attention layers may require custom masks and learned sink normalization. We present SINKFLEX-RL, a modular training system for RL in dual-control tool-use environments. The system combines a Gymnasium-compatible environment wrapper, a VERL-style rollout dataflow, group-relative policy optimization without a separate value model, and a sink-aware FlexAttention path designed to preserve model-specific sink scaling under causal and sliding-window masks. In a preliminary Tau2Bench retail run, validation reward (mean@1) rises from 0.25 early in training to $0.44$ later in the observed training window, while training-score and trajectory-reward proxies also trend upward. In a fixed-configuration memory benchmark, the optimized attention path reduces peak VRAM from 28.06GB to 22.52GB at 4096 tokens, a $19.7\%$ reduction, and runs the measured 8192-token configuration using $25.53$~GB where the eager baseline runs out of memory. These results illustrate the value of integrating environment interfaces, RL dataflow, and attention-kernel design for memory-feasible long-horizon agent training.
• 中文摘要 长期工具使用代理必须对用户目标、域策略、工具调用、模拟器状态以及延迟可验证奖励进行推理。强化学习（RL）非常适合这种环境，但多回合的策略部署会产生较长的上下文，而模型特定的注意力层可能需要自定义掩码和学习的汇总归一化。我们介绍SINKFLEX-RL，一个模块化的强化学习训练系统，适用于双控制工具使用环境。该系统结合了兼容Gymnasium的环境包装器、类似VERL的推出数据流、无独立价值模型的组相对策略优化，以及设计用于在因果和滑动窗口掩码下保持模型特定收益扩展的感知FlexAttention路径。在初步的Tau2Bench零售运行中，验证奖励（mean@1）从训练初期的0.25上升到观察到的训练窗口后期的0.44美元，而训练分数和轨迹奖励代理指标也呈上升趋势。在固定配置内存基准测试中，优化注意力路径将峰值显存从28.06GB降至22.52GB，4096个令牌，减少19.7美元，运行测量到的8192令牌配置，使用25.53美元~GB，基准内存耗尽。这些结果展示了整合环境接口、强化学习数据流和注意力内核设计对于内存可行的长视野代理训练的价值。

Dreamer-SAC: Off-Policy Learning in Latent World Models for Sample-Efficient Autonomous Driving

Dreamer-SAC：在潜在世界模型中实现采样高效自动驾驶的非政策学习

• Authors: Jiazhuo Li, Linjiang Cao, Qi Liu, Xi Xiong
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.10386
• Pdf link: https://arxiv.org/pdf/2608.10386
• Abstract Sample-efficient reinforcement learning for autonomous driving is often limited by the trade-off between data efficiency and model bias. While world models reduce the reliance on costly environment interactions, policy optimization over learned dynamics remains sensitive to prediction errors. This paper proposes the Dreamer-SAC framework, which integrates a recurrent state-space world model with an off-policy soft actor-critic algorithm trained directly in latent space. The framework uses a combination of real interactions and short-horizon generated trajectories with n-step target estimation and multi-objective supervision. Evaluated in autonomous driving scenarios with objectives encompassing driving efficiency and safety, the proposed framework consistently outperforms representative reinforcement learning baselines, including DreamerV3, SAC, and PPO, while achieving improved performance with substantially fewer real environment interactions. Experiments reveal an inverted-U relationship between rollout horizon and policy performance, where short-horizon latent rollouts achieve the best trade-off between additional training signals and accumulated model bias. Furthermore, n-step target estimation demonstrates more effectiveness over one-step temporal-difference targets in exploiting predicted experience for value learning.
• 中文摘要 自动驾驶的样本高效强化学习通常受限于数据效率和模型偏差之间的权衡。虽然世界模型减少了对昂贵环境交互的依赖，但策略优化对学习动态仍对预测误差敏感。本文提出了Dreamer-SAC框架，该框架将循环状态-空间世界模型与直接在潜空间训练的非策略软演员-批判算法集成。该框架结合了真实交互和短视距生成轨迹，并结合n步目标估计和多目标监督。在以驾驶效率和安全为目标的自动驾驶场景中评估，所提框架始终优于代表性的强化学习基线，包括DreamerV3、SAC和PPO，同时实现了显著减少真实环境交互的提升性能。实验显示，推广视野与政策绩效之间存在反U关系，短视野潜在推展在额外训练信号与累计模型偏差之间取得最佳权衡。此外，n步目标估计在利用预测经验进行价值学习方面，比一步时间差分目标更有效。

TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

TideRL：通过准备感知调度提升能动强化物流的善意输出

• Authors: Yanyu Ren, Xizheng Wang, Xiao Liu, Bowen Lv, Hanchen Zhang, Shudan Zhang, Hanyu Lai, Shuai Wang, Li Chen, Dan Li, Jie Tang
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2608.10402
• Pdf link: https://arxiv.org/pdf/2608.10402
• Abstract Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, $\textrm{RA}^2\textrm{P}$ selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6$\times$ over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58$\times$, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.
• 中文摘要 大型语言模型的强化学习（RL）正朝向多回合代理工作负载发展，即展开任务会反复暂停外部环境，随着上下文增长恢复，并在高度变异时间结束。在此环境中，以训练吞吐量衡量的强化学习（RL）训练质量比GPU占用率更重要：GPU等待和反复预填充计算纯粹是开销。我们介绍TideRL，这是一个具备度感知的弹性强化学习系统，支持连续任务批处理、资源感知的引用演员流水线和弹性资源缩放。CTB保留有用的推展状态，$\textrm{RA}^2\textrm{P}$从准备待办队列和到达间隔中选择解耦流和共址聚合，ERS则使用相同的准备信号在推送和训练之间移动排名。在纯文本和多模态代理工作负载中，TideRL使同步基线提升了最多5.6$/时间的RL训练成果，且在异步基线上提升超过33%，同时实现了类似的任务性能。它还将KV缓存命中率提升了1.58$\times$，每步训练时间减少了最多44.3%，总等待时间减少了最多77.6%。

Threat-guided Policy-aware Scene Perturbation for Safe Autonomous Driving with Online Reinforcement Learning

威胁引导的政策感知场景扰动，支持安全自动驾驶，配合在线强化学习

• Authors: Xincong Hu (1), Lei Ou (1), Maosen Li (2), Jingtao Zhang (2), Liguo Hou (2), Zongzhang Zhang (1) ((1) Nanjing University, (2) Yinwang Intelligent Technology Co., Ltd)
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10403
• Pdf link: https://arxiv.org/pdf/2608.10403
• Abstract Reinforcement learning (RL) has shown promising performance in autonomous driving, yet ensuring the safety of online RL policies remains challenging due to insufficient exposure to safety-critical driving scenes. The long-tailed nature of real-world traffic situations makes dangerous and rare interactions difficult to encounter through conventional sampling, limiting the ability of RL policies to learn robust safety behaviors. Existing methods improve training diversity by synthesizing challenging scenes or adversarial situations. However, these approaches typically optimize scene generation objectives separately from the evolving policy, without explicitly modeling how generated perturbations relate to the current policy's weaknesses and learning needs. In this paper, we propose Threat-guided Policy-aware Scene Perturbation (TPSP) for safe autonomous driving with online RL. TPSP introduces a policy-aware scene encoder to capture the interaction between policy behaviors and surrounding environments, enabling scene perturbation aligned with the current policy. Based on this representation, TPSP selectively perturbs critical objects rather than applying uniform modifications across the scene. Furthermore, we develop a threat-guided optimization strategy that evaluates perturbed scenes through threat-level differences between policy rollouts on original and perturbed scenes, guiding the generation of safety-critical scenes with higher training value. Comprehensive experiments demonstrate that TPSP improves safety learning efficiency, achieving strong safety performance on NAVSIM v2 with approximately 4 million kilometers of simulated driving data. Ablation studies verify that policy-aware targeted perturbations provide more informative safety-critical experiences than random or policy-unaware strategies, enabling safer driving under limited interaction budgets.
• 中文摘要 强化学习（RL）在自动驾驶方面表现出良好表现，但由于缺乏对安全关键驾驶场景的充分接触，确保在线强化学习政策的安全性仍具挑战性。现实世界交通情境的长尾特性使得通过传统采样难以遇到危险和罕见的交互，限制了强化学习策略学习稳健安全行为的能力。现有方法通过综合具有挑战性的场景或对抗情境来提升训练多样性。然而，这些方法通常将场景生成目标与演化策略分开优化，而未明确建模生成扰动如何与当前策略的弱点和学习需求相关。本文提出了威胁引导的策略感知场景扰动（TPSP）用于在线强化学习的自动驾驶。TPSP 引入了策略感知场景编码器，用于捕捉策略行为与周围环境之间的交互，实现与当前策略对齐的场景扰动。基于这种表示，TPSP选择性扰动关键物体，而不是在场景中均匀进行修改。此外，我们开发了一种威胁引导优化策略，通过原始场景和扰动场景策略推广之间的威胁级别差异来评估扰动场景，指导生成具有更高训练价值的安全关键场景。综合实验表明，TPSP提升了安全学习效率，在NAVSIM v2上实现了强大的安全性能，拥有约400万公里的模拟驾驶数据。消融研究证实，具备政策意识的定向扰动比随机或无政策意识策略提供更多信息的安全关键体验，使得在有限的互动预算下实现更安全的驾驶。

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning

无批评的高效在线强化学习微调预训练

• Authors: Daoyi Li, Yixian Zhang, Chao Yu, Wenbo Ding, Yu Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10473
• Pdf link: https://arxiv.org/pdf/2608.10473
• Abstract Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce \textbf{C}ritic-\textbf{F}ree \textbf{P}retraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.
• 中文摘要 离线到在线（O2O）强化学习旨在利用基于静态数据集预训练的策略，同时通过在线交互加以改进。然而，直接重复使用线下培训的批评者可能会阻碍在线微调：随着政策和数据分布快速变化，线下培训继承的价值估计可能与在线环境不匹配，导致政策改进不准确和探索效率低下。为解决这个问题，我们引入了 \textbf{C}ritic-\textbf{F}ree \textbf{P}retraining：一种高效的范式，完全放弃离线批评者训练的方法，使新初始化的批评者能够适应而不继承有偏估计值。CFP兼容多种主流O2O算法，并在多样化任务中持续匹配或改进传统O2O算法，在若干具有挑战性的任务中尤为显著。

Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation

探索驱动的个性化联合强化学习，通过内在动机

• Authors: Md Rafid Islam, Rafsan Jany, Zahid Hasan, Ratun Rahman
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10499
• Pdf link: https://arxiv.org/pdf/2608.10499
• Abstract Personalized Federated Reinforcement Learning (PFRL) takes a decentralized approach to storing and accessing information based on past experiences while keeping each client's data private during the learning of each client's policy. Many current methods for PFRL rely heavily on exploiting existing reinforcement learning reward signals to derive an optimal policy for each client, thereby neglecting exploration in non-stationary or sparse-reward environments. In this work, we introduce a new exploration-driven framework, Exploration-Driven Personalized Federated Reinforcement Learning via Intrinsic Motivation (EDPFRL-IM), that leverages an inherent curiosity-driven exploration at each client to promote local exploration and protect client privacy. Furthermore, to facilitate policy discovery via exploration in previously unexplored state spaces, clients add an intrinsic random network distillation (RND) signal to their extrinsic reward. Additionally, the server does not have access to clients' raw experiences or local gradient estimates; instead, the server sends global exploration priors and collects minimal novelty summaries from each client to enable both diverse and coordinated exploration among clients. Experiments in benchmark environments show that our framework outperforms average PFRL benchmarks in policy personalization and sample efficiency, primarily in delayed and sparse reward systems. Overall, EDPFRL-IM enables the integration of a flexible exploratory learning structure into federated reinforcement learning systems while preserving client privacy.
• 中文摘要 个性化联合强化学习（PFRL）采用去中心化方法，基于以往经验存储和访问信息，同时在学习每位客户保单时保持其数据私密。许多当前的PFRL方法高度依赖利用现有的强化学习奖励信号来为每个客户端推导最优策略，从而忽视了在非平稳或稀疏奖励环境中的探索。在本研究中，我们引入了一个新的探索驱动框架——探索驱动的内在动机个性化联合强化学习（EDPFRL-IM），该框架利用每个客户内在的好奇心驱动探索，促进本地探索并保护客户隐私。此外，为了通过探索此前未探索的状态空间进行策略发现，客户会在其外部奖励中添加内在随机网络提炼（RND）信号。此外，服务器无法访问客户端的原始体验或局部梯度估计;相反，服务器会发送全球探索先验，并从每个客户收集最少的新颖性摘要，从而实现客户之间多样化且协调的探索。基准环境中的实验表明，我们的框架在策略个性化和样本效率方面优于平均的PFRL基准，主要体现在延迟和稀疏奖励系统中。总体而言，EDPFRL-IM 能够将灵活的探索性学习结构集成到联邦强化学习系统中，同时保护客户隐私。

SafeCap: Improving LVLM Safety with Image Captioning Reinforcement Learning

SafeCap：通过图像字幕强化学习提升LVLM安全

• Authors: Caoyuan Ma, Wenpu Liu, Weichu Xie, Tian Gu, Shilei Zhao, Lingxi Min, Shuai Dong, Yuqi Xu, Ji Zhao, Ziyue Wang, Wenzheng Chang, Taiqiang Wu, Yongfu Zhu, Wenqi Shao, Yinqiang Zheng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10513
• Pdf link: https://arxiv.org/pdf/2608.10513
• Abstract Large vision-language models (LVLMs) remain vulnerable to jailbreak attacks that exploit visual inputs to bypass safety alignment inherited from their language backbones. We propose SafeCap, a reinforcement-learning framework that aligns LVLMs through learned self-captioning. SafeCap trains a policy model to first generate a safety-relevant image caption and then produce a final answer; the caption is further optimized by whether it enables a frozen LLM to reach a safety-aligned decision. This caption-mediated objective encourages the policy to expose visual cues relevant to safe response generation rather than relying solely on direct refusal supervision. Across five multimodal safety benchmarks and six vision-utility benchmarks, SafeCap substantially improves aggregate safety performance under its intended DirectCap protocol, with gains of 3.7-19.0 points in safety average across four model settings while maintaining comparable or improved vision utility. Under controlled comparisons on matched backbones and data, SafeCap outperforms safety SFT, DPO, and SafeGRPO, demonstrating the effectiveness of caption-mediated reinforcement learning for multimodal safety alignment.
• 中文摘要 大型视觉语言模型（LVLM）仍然容易受到越狱攻击的威胁，这些攻击利用视觉输入绕过其语言骨干继承的安全对齐。我们提出了SafeCap，一种通过学习自写字幕来对齐LVLM的强化学习框架。SafeCap训练策略模型，先生成安全相关的图片说明，然后生成最终答案;该说明还通过是否使冻结的大型语言模型能够做出安全对齐的决策来进一步优化。这一以字幕为媒介的目标鼓励政策揭示与安全反应生成相关的视觉线索，而非仅依赖直接拒绝监督。在五个多模态安全基准和六个视觉-实用基准中，SafeCap在其预期的DirectCap协议下显著提升了整体安全性能，在四个模型设置下安全平均提升了3.7-19.0分，同时保持了可比或提升的视觉效用。在匹配主干和数据的受控比较下，SafeCap优于安全SFT、DPO和SafeGRPO，证明了字幕介导强化学习在多模态安全对齐中的有效性。

An Asynchronous Triggered MAC Protocol for Underwater Acoustic Networks

一种用于水下声学网络的异步触发MAC协议

• Authors: Bingwen Huangfu, Jiani Guo, Shanshan Song, Nan Sun, Jun Liu
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.10533
• Pdf link: https://arxiv.org/pdf/2608.10533
• Abstract Time Division Multiple Access (TDMA)-based Medium Access Control (MAC) protocols have proven their practicality through extensive field trials in Underwater Acoustic Networks (UANs), attributable to their hardware-agnostic and easily implementable properties. Most existing protocols rely on a synchronized and fixed-length slot paradigm to mitigate channel contention and facilitate orderly transmissions. However, this paradigm imposes significant clock synchronization overhead in UANs with low and variable sound speed and struggles to improve scheduling flexibility. Although some protocols attempt to refine this slot paradigm (adjust the slot length to improve channel reuse efficiency or scheduling frequency), they are still constrained by the trade-off between channel utilization and scheduling complexity. To this end, this paper advocates a paradigm shift in underwater MAC design, transitioning from synchronized slot to asynchronous scheduling, and realizes it through an Asynchronous Triggered MAC (AT-MAC). AT-MAC introduces a triggered slot paradigm without time synchroniza?tion, decoupling transmission scheduling from a rigid timeline and enabling asynchronous, variable-length slots to accommodate the long and diverse propagation delays. To power this slot paradigm, AT-MAC augments conventional Multi-Agent Deep Reinforcement Learning to handle asynchronous interaction, achieving efficient coordinated channel access under partial ob?servations. It further devises a load-aware fairness guard mech?anism to enable network-wide fairness status inference solely through local overhearing, thereby guiding adaptive scheduling correction to maintain fairness. Trace-based and on-board ex?periments validate the feasibility and computational practicality of AT-MAC. Extensive simulation results further demonstrate its superiority and adaptability across various scenarios and traffic conditions.
• 中文摘要 基于时分多址（TDMA）的介质接入控制（MAC）协议通过在水下声学网络（UAN）中的广泛实地试验，证明了其实用性，这归功于其硬件无关且易于实现的特性。大多数现有协议依赖同步且固定长度的时隙范式来减轻信道争用并促进有序传输。然而，这种模式在低且可变的声速的UAN中会带来显著的时钟同步开销，且难以提升调度灵活性。尽管一些协议试图优化这一时隙范式（调整时隙长度以提高信道重用效率或调度频率），但仍受信道利用率与调度复杂性之间的权衡限制。为此，本文主张水下MAC设计的范式转变，从同步槽向异步调度过渡，并通过异步触发MAC（AT-MAC）实现这一目标。AT-MAC引入了无时间同步的触发时隙范式，将传输调度与固定时间线解耦，并支持异步、可变长度的时隙以适应漫长且多样化的传播延迟。为支持该槽范式，AT-MAC增强了传统的多智能体深度强化学习，以处理异步交互，实现部分观测下的高效协调通道访问。它还设计了一种负载感知的公平保护机制，仅通过本地监听实现全网络公平状态推断，从而指导自适应调度修正以维持公平性。基于跟踪和机载的实验验证了AT-MAC的可行性和计算实用性。大量模拟结果进一步证明了其在各种场景和交通状况下的优越性和适应性。

SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models

SKILLER：用于小型语言模型中可重复使用的技能提取的语言级强化学习

• Authors: Chenhao Dang, Siyuan Xiong, Conghui He, Weijia Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10538
• Pdf link: https://arxiv.org/pdf/2608.10538
• Abstract Agent skills represent a standardized format for packaging procedural knowledge and domain expertise, serving within agent harness systems as an essential mechanism to continually constrain a language model's behavior space for repeatable, high-quality task execution. However, because strong closed-source models entail high inference costs, current popular agent harnesses, such as Codex and OpenClaw, remain prohibitively expensive when deploying these skills to accomplish real-world tasks. The rapid capability enhancement of open-source models deployable on consumer-grade GPUs presents a compelling opportunity to drastically reduce these costs by leveraging skill-based behavioral constraints. Nevertheless, automatically generating effective skills tailored specifically for such compact models remains a significant practical challenge. To address this, we propose SKILLER, a natural-language-driven reinforcement learning framework designed to automatically generate executor-specific skills for small models, which employs a strong model as the actor and critic, treats the small-model agent system as the environment, and propagates all reinforcement learning signals entirely via natural language. Extensive experimental evaluations across five relevant benchmarks using Qwen3.5-9B and Qwen3.5-4B demonstrate that SKILLER outperforms three open-source and one closed-source skill generation or evolution methods, achieving absolute gains ranging from 4.3 to 20.4 percentage points for the 9B model and 1.8 to 13.3 points for the 4B model, while remarkably matching the performance of strong closed-source models on single-skill tasks in SkillsBench. The project is available at this https URL.
• 中文摘要 代理技能代表了一种标准化格式，用于包装程序知识和领域专业知识，在代理利用系统中作为持续约束语言模型行为空间、实现可重复且高质量任务执行的重要机制。然而，由于强大的闭源模型需要高昂的推理成本，当前流行的代理工具，如Codex和OpenClaw，在实际应用这些技能以完成现实任务时，依然成本过高。可部署于消费级GPU的开源模型的快速能力增强，为利用基于技能的行为约束大幅降低成本提供了极具吸引力的机会。然而，自动生成专门针对此类紧凑模型的有效技能仍是一个重大的实际挑战。为此，我们提出了SKILLER，一种自然语言驱动的强化学习框架，旨在自动生成执行者专属技能，使用强模型作为行为者和批评者，将小模型代理系统视为环境，并完全通过自然语言传播所有强化学习信号。在五个相关基准测试中，使用 Qwen3.5-9B 和 Qwen3.5-4B 进行的广泛实验评估显示，SKILLER 在三种开源和一种闭源技能生成或演进方法中表现优于三种开源和一种闭源技能生成或演进方法，9B 模型的绝对提升范围在 4.3 至 20.4 个百分点之间，4B 模型提升 1.8 至 13.3 个百分点，同时在 SkillsBench 中，性能与强闭源模型的单一技能任务表现相当。该项目可在此访问 https 网址。

Reinforcement Learning-Based Laser Cutting Machine Parameter Optimization

基于增强学习的激光切割机参数优化

• Authors: Khanh Quan Pham, Majid Kundroo, Geunwoo Ban, Seongho Bae, Taehong Kim
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10549
• Pdf link: https://arxiv.org/pdf/2608.10549
• Abstract Achieving high accuracy in laser-based cutting of optical films requires careful tuning of parameters such as focal length and laser power beam, adjusted according to the specific properties of each film type. Trial-and-error based traditional methods are used to find the most suitable cutting parameters for various films, but they are slow and inaccurate. To address this issue, this paper presents the Reinforcement Learning for Laser Cutting (RL$^{2}$C) algorithm, which uses Q-learning with an epsilon-greedy policy to dynamically optimize cutting parameters, significantly reducing taper size and film wastage. Additionally, RL$^{2}$C incorporates a dynamic environment space adaptability mechanism to allow it to adapt to new states encountered during the learning process over multiple batches of experiments. Experimental results demonstrate that RL$^{2}$C requires fewer steps and less time to find optimal cutting parameters compared to various RL-based optimization methods. Specifically, RL$^{2}$C reduces the number of optimization steps by up to 12.5\% and processing time by up to 81.8\% compared to existing methods. This study demonstrates the potential of RL in industrial laser-cutting processes by improving cut quality, reducing time and film wastage, and minimizing manual interventions.
• 中文摘要 在基于激光的光学薄膜切割中实现高精度，需要根据每种胶片类型的特性对焦距和激光功率光束等参数进行精确调校。采用基于反复试验的传统方法来寻找各种胶片最合适的剪辑参数，但这些方法速度慢且不准确。为解决这一问题，本文介绍了激光切割强化学习（RL$^{2}$C）算法，该算法采用ε贪婪策略的Q学习动态优化切割参数，显著减少锥度尺寸和胶片浪费。此外，RL$^{2}$C 还集成了动态环境空间适应机制，使其能够适应多批实验学习过程中遇到的新状态。实验结果表明，与各种基于强化学习的优化方法相比，RL$^{2}$C所需的步骤更少，时间更短，找到最优切割参数。具体来说，RL$^{2}$C 相比现有方法，优化步骤数减少了最多 12.5% 和处理时间高达 81.8% 的提升。本研究展示了强化学习在工业激光切割工艺中的潜力，通过提升切割质量、减少时间和胶片浪费，以及减少人工干预。

Dual-Loop Self-Evolution via Verifiable Emotion Feedback for Multi-Turn Empathetic Dialogue

通过可验证的情绪反馈实现双循环自我进化，实现多回合同理心对话

• Authors: Yi Wei, Shuo Jiang, Huaixia Dou, Jie Zhu, Junhui Li, Lifan Guo, Feng Chen, Chi Zhang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.10626
• Pdf link: https://arxiv.org/pdf/2608.10626
• Abstract Large language models have demonstrated conversational capabilities, yet empathetic competence remains challenging. Empathetic support is inherently multi-turn and path-dependent: users disclose concerns gradually, emotions evolve over time, and early responses shape trust and receptivity. Reinforcement learning with verifiable emotion rewards provides scalable supervision for long-horizon interactions. However, existing methods evolve the dialogue policy while keeping its training interaction distribution fixed, creating a mismatch between policy competence and training experience. We introduce a dual-loop self-evolution framework driven by verifiable emotion feedback. With the user simulator and verifier frozen, the inner loop optimizes the multi-turn policy using continuous emotion rewards, while the outer loop uses the same outcomes to estimate policy-relative interaction utility and adapt experience. To obtain estimates from sparse, stochastic rollouts, the framework holds the scenario and interaction state constant within each group and prioritizes conditions whose group pass rates lie near the policy's competence boundary. A hierarchical controller shares evidence across support intents, while uncertainty-guided exploration and uniform rehearsal prevent premature exclusion. The resulting distribution generates trajectories, closing both loops without increasing the rollout budget. On SAGE, our framework raises Qwen3-8B Overall from 53.87 to 79.24 and outperforms protocol-matched uniform emotion-reward reinforcement learning by 7.23 points.
• 中文摘要 大型语言模型已展现出会话能力，但同理心能力依然充满挑战。同理心支持本质上是多重且依赖路径的：用户逐步表达关切，情绪随时间演变，早期反应塑造信任和接受度。带有可验证情感奖励的强化学习为长期互动提供了可扩展的监督。然而，现有方法在保持对话策略的培训互动分布固定的同时演进，导致政策能力与培训经验之间存在不匹配。我们引入了一个由可验证的情绪反馈驱动的双循环自我进化框架。当用户模拟器和验证器冻结时，内环通过持续情感奖励优化多回合策略，而外环则利用相同结果估计策略相对交互效用并调整体验。为了从稀疏、随机的推广中获得估计值，该框架在每个组内保持场景和交互状态不变，并优先考虑组通过率接近策略能力边界的条件。层级控制者在支持意图间共享证据，而不确定性引导的探索和统一的排练则防止过早排除。由此产生的分布产生轨迹，封闭两个环路而不增加推广预算。在SAGE平台上，我们的框架将Qwen3-8B整体评分从53.87提升至79.24，并且比协议匹配的均匀情感-奖励强化学习高出7.23分。

IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning

IADD-TR：基于模型的强化学习中的干预感知动力学解耦与目标正则化

• Authors: Zefeng Liang, Jie Qiao, Ruichu Cai, Weilin Chen, Zhifeng Hao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.10634
• Pdf link: https://arxiv.org/pdf/2608.10634
• Abstract Model-based reinforcement learning (MBRL), which learns environment dynamics to generate synthetic experience, is a promising approach to sample-efficient decision making. Numerous methods have been developed to improve dynamics prediction and policy optimization for MBRL through uncertainty estimation, model regularization, and conservative value learning. However, these methods typically treat the transition model and critic as monolithic predictors, overlooking the policy-induced data bias. Consequently, action can become entangled with environmental evolution, while uneven action coverage may distort the counterfactual value estimates used for policy improvement. To address this, we propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR). IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization. Its latent and state-aligned components are identifiable up to an invertible within-block transformation and pointwise, respectively. For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional. TR augments the critic with an action-density-scaled residual correction and optimizes a targeted loss, yielding doubly robust policy-gradient estimation when either the critic or the replay action density is consistently specified. Extensive experiments on five MuJoCo tasks show that IADD-TR achieves competitive returns with improved sample efficiency.
• 中文摘要 基于模型的强化学习（MBRL）通过学习环境动态生成合成经验，是一种有前景的样本高效决策方法。通过不确定性估计、模型正则化和保守值学习，已开发出多种方法以提升MBRL的动态预测和策略优化。然而，这些方法通常将过渡模型和批判者视为单一预测变量，忽视了政策引发的数据偏差。因此，行动可能与环境演变纠缠在一起，而不均衡的行动覆盖可能扭曲用于政策改进的反事实价值估计。为此，我们提出了IADD-TR，这是一个结合干预感知动力学解耦（IADD）和目标正则化（TR）的统一框架。IADD将转变分解为动作干预阶段和无动作自然演化阶段，使用零作用锚点解决该两阶段分解的非唯一性，从而实现稳健推广。其潜在分量和状态对齐分量分别可通过可逆的块内变换和逐点识别。对于策略学习，我们从重放状态政策梯度泛函的高效影响函数中推导出TR。TR通过动作密度尺度的残差修正来增强批评者，并优化目标损失，当批判者或重放的行动密度一致指定时，能获得双重稳健的策略梯度估计。对五个MuJoCo任务的广泛实验表明，IADD-TR能够以更高的样本效率实现竞争回报。

Threshold-Based Spiking Neural Networks for Event-Driven Status Update Systems

基于阈值的尖峰神经网络用于事件驱动状态更新系统

• Authors: Marco Fries, Andrea Ortiz
• Subjects: Subjects: Information Theory (cs.IT)
• Arxiv link: https://arxiv.org/abs/2608.10640
• Pdf link: https://arxiv.org/pdf/2608.10640
• Abstract Event-driven sensing supports energy-efficient Internet-of-Things (IoT) devices by activating communication only when relevant events occur. In such systems, transmission decisions are governed by the monitored process rather than predefined schedules. Consequently, jointly optimising information freshness and energy consumption is challenging because transmission decisions are restricted to randomly occurring events. To address this challenge, we investigate an event-driven status update system in which wake-up events follow the dynamics of the monitored process. The problem of determining whether to transmit the sensing data or not is cast as a Markov Decision Process (MDP) that jointly minimises the Age of Information (AoI) and transmission energy. We prove the existence of an optimal threshold policy, thereby obtaining an interpretable characterisation of the optimal transmission strategy. Motivated by this result, we propose a lightweight Reinforcement Learning (RL) approach based on Spiking Neural Networks (SNNs) whose architecture explicitly represents threshold policies. The resulting policy representation has constant complexity with respect to the maximum AoI and enables a more energy-efficient implementation than a comparable Artificial Neural Network (ANN). Numerical results demonstrate that the proposed SNN reliably learns optimal thresholds across different operating regimes.
• 中文摘要 事件驱动传感支持节能的物联网设备，仅在相关事件发生时激活通信。在此类系统中，输电决策受监控过程控制，而非预先设定的时刻表。因此，联合优化信息新鲜度和能耗具有挑战性，因为传输决策仅限于随机发生的事件。为应对这一挑战，我们研究了一种事件驱动状态更新系统，其中唤醒事件遵循被监控过程的动态。决定是否传输传感数据的问题被归结为马尔可夫决策过程（MDP），共同最小化信息时代（AoI）和传输能量。我们证明了最优阈值策略的存在，从而获得了最优传输策略的可解释特征。基于这一结果，我们提出了一种基于尖峰神经网络（SNN）的轻量级强化学习（RL）方法，其架构明确表示阈值策略。由此产生的策略表示在最大 AoI 方面具有恒定复杂性，使得比同类人工神经网络（ANN）实现更能效。数值结果表明，拟议的SNN能够可靠地在不同操作模式下学习最优阈值。

Control of hybrid wind-wave energy systems using reinforcement learning

利用强化学习控制混合风波能源系统

• Authors: Zechuan Lin, Kemeng Chen, Maosen Fan, Xiaofan Li, Xi Xiao, John V. Ringwood
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.10754
• Pdf link: https://arxiv.org/pdf/2608.10754
• Abstract Integrating wave energy converters (WECs) with floating offshore wind turbines (FOWTs), to form hybrid wind-wave energy (HWWE) systems, is a promising approach to achieve further cost reduction for offshore renewable energy. In such systems, the control of the integrated WECs plays an important role, with the potential to generate additional wave energy while simultaneously suppressing floating platform motion. However, HWWE systems are characterized by complex dynamics, making accurate modelling only viable through numerical simulation, and posing significant challenges for control design. This paper proposes a reinforcement learning (RL) control framework for HWWE systems, in which the real-time control policy is learned directly through interactions with high-fidelity simulation. A numerical model is established for a HWWE system consisting of an IEA 15 MW wind turbine, a VolturnUS semi-submersible platform, and three torus-type WECs, which is then employed as the RL training environment. Control performance is evaluated in terms of both wave energy generation and platform motion reduction, two competing objectives, from a Pareto perspective. It is shown that the proposed RL controller achieves substantial Pareto improvements over conventional control strategies, e.g., over 75\% higher wave energy capture at the same platform motion level, or nearly 50\% lower motion at the same energy capture level, thereby significantly extending the attainable performance boundary of HWWE systems.
• 中文摘要 将波浪能转换器（WEC）与浮式海上风力涡轮机（FOWTs）集成，形成混合风波能源（HWWE）系统，是实现海上可再生能源进一步降低成本的有前景方案。在此类系统中，集成WECs的控制发挥着重要作用，既有可能产生额外的波浪能量，又能抑制浮动平台的运动。然而，HWWE系统具有复杂的动力学特性，使得精确建模只能通过数值仿真实现，这对控制设计带来了重大挑战。本文提出了一种针对HWWE系统的强化学习（RL）控制框架，实时控制策略通过与高保真模拟的交互直接学习。建立了一个由IEA 15兆瓦风力涡轮机、一台VolturnUS半潜式平台和三个环形WECS组成的HWWE系统数值模型，随后用作强化学习训练环境。控制性能从帕累托视角评估波浪能量生成和平台运动减少这两个竞争目标。研究表明，所提出的强化控制控制器相比传统控制策略实现了显著的帕累托改进，例如在相同平台运动水平下波浪能量捕获提升了超过75%或在相同能量捕获水平下运动量降低近50%，从而显著扩展了HWWE系统可达的性能边界。

FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding

FADE：从被动验证到主动发现，在反事实视频理解中

• Authors: Fufangchen Zhao, Jinhu Fu, Jiachen Lei, Jiahong Wu, Xiangxiang Chu, Danfeng Yan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.10764
• Pdf link: https://arxiv.org/pdf/2608.10764
• Abstract Counterfactual video understanding evaluates whether models grasp physical and commonsense regularities. However, existing multiple-choice question (MCQ) benchmarks inadvertently leak target events through their questions and candidate options. This reduces the core challenge from active discovery to text-guided verification. In this paper, we present FADE, an effective training framework for counterfactual discovery and explanation. Our method is built on an evidence-first, two-stage training paradigm. First, evidence-internalized supervised fine-tuning grounds the model's predictions in decisive visual anomalies. Second, we apply a fading-anchor reinforcement learning strategy that progressively removes textual guidance, compelling the model to independently discover and explain evidence. To rigorously evaluate this capability, we also introduce an effective pipeline that converts existing MCQ datasets into aligned MCQ, open-ended question answering (OQA), and captioning tasks without requiring additional data curation. Our simple approach yields strong results. Using Qwen3-VL-8B as the baseline, FADE achieves state-of-the-art strict paired scores across all three tasks on DualityVidQA-test and IPV-Bench, outperforming GPT-5.6. In specific, when transitioning from constrained MCQs to unconstrained OQA and captioning, our model demonstrates remarkable robustness. Its performance retention is 90.4% and 67.4% on DualityVidQA-test-substantially higher than the 48.1% and 30.7% retained by GPT-5.6. We hope this simple framework can serve as a solid baseline for future research in unconstrained counterfactual video understanding.
• 中文摘要 反事实视频理解评估模型是否掌握物理和常识规律。然而，现有的选择题（MCQ）基准会无意中通过题目和候选选项泄露目标事件。这将核心挑战从主动发现转向文本引导验证。本文介绍了FADE，一种用于反事实发现和解释的有效培训框架。我们的方法基于以证据为先的两阶段训练范式。首先，基于证据的内化监督微调将模型的预测建立在决定性的视觉异常之上。其次，我们采用渐进式锚点强化学习策略，逐步去除文本指导，迫使模型独立发现和解释证据。为严格评估该能力，我们还引入了一套有效的流程，将现有多选题数据集转换为对齐的多选题、开放式问答（OQA）和字幕任务，无需额外数据整理。我们简单的方法带来了显著的成果。以Qwen3-VL-8B为基准，FADE在DualityVidQA测试和IPV-Bench的三项任务中均获得最先进的严格配对分数，优于GPT-5.6。具体来说，当从受限选择题过渡到无约束的OQA和字幕时，我们的模型展现出了显著的稳健性。其在 DualityVidQA 测试中的性能保留率分别为 90.4% 和 67.4%，远高于 GPT-5.6 的 48.1% 和 30.7%。我们希望这一简单框架能为未来无限制反事实视频理解研究提供坚实基础。

Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation

多语言机器翻译开放大型语言模型的无引用后训练

• Authors: Chris Han, Pengzhi Gao, Pei Fu, Jian Luan
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10812
• Pdf link: https://arxiv.org/pdf/2608.10812
• Abstract We study reference-free post-training for multilingual machine translation with open large language models. Starting from the supervised-finetuned MiLMMT-46-v0.1 models, we apply Group Relative Policy Optimization (GRPO) with a reward that averages two reference-free quality estimation models and is gated by language identification. We then linearly interpolate the supervised fine-tuning (SFT) and reinforcement learning (RL) model checkpoints to obtain MiLMMT-46-v1.0. Across 46 languages, the resulting models consistently improve translation quality over their SFT counterparts, outperform strong recent open baselines, including Seed-X, HY-MT2, and TranslateGemma, and achieve leading reference-free scores against evaluated proprietary systems such as Google Translate, Gemini 3 Pro, and GPT-5. We further investigate on-policy distillation and find that it reaches, but does not surpass, the quality frontier achieved by RL with checkpoint interpolation. We release the models and code to facilitate future research.
• 中文摘要 我们研究了多语言机器翻译在开放大型语言模型中进行无引用后训练。从监督微调的MiLMMT-46-v0.1模型出发，我们应用了群相对策略优化（GRPO），其奖励平均了两个无参考的质量估计模型，并以语言识别为门槛。随后，我们线性插值监督微调（SFT）和强化学习（RL）模型检查点，得到MiLMMT-46-v1.0。在46种语言中，这些模型的翻译质量持续优于SFT版本，优于近期强劲的开放基线，包括Seed-X、HY-MT2和TranslateGemma，并在与Google Translate、Gemini 3 Pro和GPT-5等专有系统相比，取得了领先的无引用评分。我们进一步研究了政策上提炼，发现它达到了通过检查点插值实现的质量边界，但并未超越。我们发布模型和代码，以促进未来的研究。

MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training

MoE代理模型用于LLM强化学习后低成本失败复制与诊断

• Authors: Yikai Wang, Chuansai Zhou, Yuhang Zhou, Weiqiang Wu, Cong Wu, Yue Deng, Ben Feng, Mingming Zhu, Beirong Zhou, Zhibin Wang, Sheng Zhong, Chen Tian, Wangze Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.10823
• Pdf link: https://arxiv.org/pdf/2608.10823
• Abstract Reinforcement learning (RL) post-training of large language models (LLMs) is computationally intensive and involves complex system pipelines with substantial debugging overhead. In practice, factors such as framework adaptation, numerical precision, and operator implementation can cause failures, including gradient overflow and loss divergence. Reproducing such failures directly on large models requires considerable time and computational resources. This paper systematically analyzes failures encountered during large-scale RL training on the Huawei Ascend platform, summarizes representative failure types, and identifies three model-side factors relevant to fault reproduction. Based on these factors, we propose a proxy-model construction method for low-cost fault investigation and auxiliary diagnosis. It employs structure-preserving, clustering-based expert pruning to select representative experts while retaining the model's backbone architecture, routing mechanism, and basic task capabilities. Our experimental results show that the proxy models reduce accelerator requirements by 50%-87.5% and achieve up to a 33.3x reduction in per-step NPU-hour cost, while preserving major training dynamics and reproducing fault responses consistent with the original models. Overall, the proxy models can serve as low-cost surrogates for fault reproduction, targeted validation, and auxiliary diagnosis in RL post-training.
• 中文摘要 强化学习（RL）对大型语言模型（LLMs）的后期训练计算量大，涉及复杂的系统流水线，调试开销较大。实际上，框架适应性、数值精度和算符实现等因素可能导致失效，包括梯度溢出和损耗散度。直接在大型模型上重现此类故障需要大量时间和计算资源。本文系统分析了华为Ascend平台上大规模强化学习训练中遇到的故障，总结了具有代表性的故障类型，并识别了三个与故障重现相关的模型侧因素。基于这些因素，我们提出了一种代理模型构建方法，用于低成本的故障调查和辅助诊断。它采用结构保持、基于聚类的专家剪枝，在保留模型骨干架构、路由机制和基本任务能力的同时，选择具有代表性的专家。我们的实验结果显示，代理模型可将加速器需求降低50%-87.5%，每步NPU小时成本降低高达33.3倍，同时保持主要训练动态并重现与原始模型一致的故障响应。总体而言，代理模型可作为低成本的替代工具，用于强化学习后错误重现、靶向验证和辅助诊断。

MIRA: Medical Image Reflection for Agentic Diagnosis

MIRA：医学图像反射用于代理诊断

• Authors: Shengzhi Wang, Jun Yang, Kai Wu, Xiaozhong Ji, Yiwen Ye, Ziyang Chen, Mingliang Xiong, Wen Fang, Mingqing Liu, Mengyuan Xu, Miaoxuan Shan, Caiyan Liu, Bin He, Qingwen Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10827
• Pdf link: https://arxiv.org/pdf/2608.10827
• Abstract Medical visual agents can use tools to inspect images and retrieve external knowledge, but indiscriminate tool use may introduce noisy or misleading evidence. Reliable diagnosis therefore requires not only acquiring additional observations, but also verifying whether tool actions are necessary and whether the resulting evidence supports the current hypothesis. We introduce MIRA (Medical Image Reflection for Agentic Diagnosis), a medical visual diagnostic framework for autonomous evidence search and reflective verification. MIRA dynamically invokes image-processing operations, including zooming, grounding, pointing, rotation, and measurement, as well as web search, while evaluating the relevance and consistency of the acquired evidence. We develop MIRA through a two-stage training strategy. First, a tool-augmented Monte Carlo Tree Search data engine explores diverse diagnostic hypotheses and jointly verifies visual grounding accuracy and semantic consistency to construct supervised fine-tuning trajectories. Second, reinforcement learning further improves decision-making through online reflective principle evolution: failure cases are distilled into candidate principles, and only principles that improve held-out rollout rewards are retained. Across nine medical visual reasoning benchmarks, MIRA achieves an average score of 64.73, improving its Qwen3-VL-8B backbone by 7.44 points. It also increases useful tool-use judgments from 56.2% to 73.8% and reduces harmful judgments from 8.9% to 1.6%. Qualitative analyses show that MIRA can re-examine evidence, correct premature conclusions, and adapt its tool-use strategy. Project page: this https URL
• 中文摘要 医学视觉代理可以使用工具检查图像和获取外部知识，但不加区分的工具使用可能会引入噪声或误导性的证据。因此，可靠的诊断不仅需要获取额外的观测数据，还要验证工具动作是否必要，以及所得证据是否支持当前假设。我们介绍了MIRA（医学影像反射用于代理诊断），这是一种用于自主证据搜索和反射验证的医学视觉诊断框架。MIRA动态调用图像处理操作，包括缩放、接地、指向、旋转和测量，以及网页搜索，同时评估所获证据的相关性和一致性。我们通过两阶段培训策略开发MIRA。首先，一个工具增强的蒙特卡洛树搜索引擎探索了多种诊断假设，并共同验证视觉基础准确性和语义一致性，以构建监督微调轨迹。其次，强化学习通过在线反思原则演进进一步提升决策能力：失败案例被提炼为候选原则，只有能改善未完成推广奖励的原则被保留。在九项医学视觉推理基准中，MIRA的平均得分为64.73，提升了Qwen3-VL-8B骨干7.44分。它还将有用的工具使用判断从56.2%提高到73.8%，并将有害判断从8.9%降至1.6%。定性分析表明，MIRA能够重新审视证据，纠正过早结论，并调整其工具使用策略。项目页面：此 https URL

Partially Observable Learning for Multi-Platform Dispatch Optimization

多平台调度优化的部分可观察学习

• Authors: Fengming Yao, Man Luo
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.10897
• Pdf link: https://arxiv.org/pdf/2608.10897
• Abstract Instant delivery platforms have become a critical component of urban logistics, increasingly relying on crowdsourced couriers to fulfill highly dynamic orders. In real-world systems, couriers are not exclusive to a single platform and may concurrently serve multiple platforms, while each platform can only observe its own orders and couriers' interactions due to privacy and operational constraints. This results in a multi-platform dispatch environment with inherent partial observability. However, most existing works on dispatch optimization assume full courier observability and mandatory assignment acceptance, causing substantial performance degradation when deployed in realistic multi-platform settings. In this paper, we propose POLO, a partially observable multi-agent reinforcement learning framework for dispatching optimization in multi-platform instant delivery systems. POLO firstly models each platform-grid pair as an independent agent that learns dispatch policies solely from platform-local observations, aligning the learning process with real-world privacy and operational constraints. To support effective decision-making under incomplete and heterogeneous courier information, POLO introduces a novel attention-based policy representation that selectively aggregates inter-courier information. Moreover, we design a counterfactual reward shaping mechanism to mitigate the non-stationarity induced by joint actions across grids, leading to more stable and scalable learning. We develop a high-fidelity simulator to evaluate dispatch performance under varying numbers of platforms and system scales. Extensive experiments demonstrate that POLO consistently outperforms strong baselines in terms of platform revenue and courier travel efficiency, highlighting its robustness and effectiveness in realistic multi-platform settings.
• 中文摘要 即时配送平台已成为城市物流的关键组成部分，越来越依赖众包快递来完成高度动态的订单。在现实系统中，快递员并非独占于单一平台，可能同时服务多个平台，而每个平台由于隐私和运营限制，只能观察自己的订单和快递员的互动。这导致了一个多平台调度环境，具有固有的部分可观测性。然而，大多数现有的调度优化工作假设了完全可观测快递和强制分配接受，导致在现实的多平台环境中部署时性能显著下降。本文提出了POLO，一种部分可观察的多智能体强化学习框架，用于多平台即时交付系统中的调度优化。POLO首先将每个平台-网格对建模为独立代理，仅通过平台-本地观察学习调度策略，使学习过程与现实世界的隐私和运营约束保持一致。为了支持在不完整且异构的快递信息下有效的决策，POLO引入了一种基于注意力的新型政策表征，选择性地汇总快递间信息。此外，我们设计了一种反事实的奖励塑造机制，以减轻跨网格联合动作引起的非平稳性，从而实现更稳定和可扩展的学习。我们开发了高保真模拟器，用于评估不同平台和系统规模下的调度性能。大量实验表明，POLO在平台收入和快递旅行效率方面始终优于强劲基线，凸显其在现实多平台环境中的稳健性和有效性。

Evidence-Grounded Trustworthy Multimodal Reasoning and Evaluation Benchmark in Complex Urban Scenes

基于证据的可信多模态推理与评估基准，适用于复杂城市场景

• Authors: Zhaoyang Wei, Bowen Jiang, Xumeng Han, Jiashu Li, Xuehui Yu, Yuling Liu, Guorong Li, Zhenjun Han, Jianbin Jiao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.10954
• Pdf link: https://arxiv.org/pdf/2608.10954
• Abstract While Multimodal Large Language Models (MLLMs) demonstrate impressive performance in benign scenarios, their cognitive reliability deteriorates significantly in complex scenes under adverse conditions. In these settings, models often rely on implicit inference without sufficient visual evidence, leading to a disconnect between perception and reasoning. Meanwhile, existing outcome-oriented benchmarks evaluate only final predictions and fail to diagnose failures in the underlying reasoning process. To address this gap, the authors propose AD2-Bench, which introduces a Hierarchical Visual Diagnosis framework that decomposes reasoning into a structured Chain of Evidence (CoE). This fine-grained diagnosis reveals that robust multimodal reasoning fundamentally depends on accurate evidence acquisition. Building on this perspective, the authors formulate reasoning from a probabilistic viewpoint and identify two primary causes of reasoning failure: Spatial Ambiguity, where models fail to distinguish target objects from background clutter, resulting in localization errors; and Semantic Uncertainty, where degraded visual features lead to incorrect semantic interpretation, resulting in understanding errors. To overcome these evidence deficiencies, they further propose Evidence-grounded Visual Reasoning (EGVOR), which replaces implicit reasoning with the explicit generation of Evidence Atoms - structured spatial-semantic triplets that enforce tight alignment between localization and semantic understanding. The model is trained through a hierarchical curriculum that progresses from reflective supervision construction to reinforcement learning, where reducing reasoning variance is explicitly rewarded. Extensive experiments demonstrate that EGVOR substantially improves reasoning stability under adverse conditions, providing a more robust framework for trustworthy multimodal cognition.
• 中文摘要 虽然多模态大型语言模型（MLLM）在良性场景中表现出色，但在复杂场景和恶劣条件下，其认知可靠性显著下降。在这些环境中，模型常依赖隐性推断，缺乏足够的视觉证据，导致感知与推理之间存在脱节。与此同时，现有以结果为导向的基准只评估最终预测，未能诊断推理过程中的失败。为弥补这一空白，作者提出了AD2-Bench，引入了层级视觉诊断框架，将推理分解为结构化的证据链（CoE）。这种细致诊断表明，稳健的多模态推理根本依赖于准确的证据获取。基于这一观点，作者从概率视角提出推理，并指出推理失败的两个主要原因：空间模糊性，即模型未能将目标对象与背景杂乱区分开，导致定位错误;以及语义不确定性，即视觉特征退化导致语义解释错误，进而导致理解错误。为弥补这些证据不足，他们进一步提出了基于证据的视觉推理（EGVOR），用显式生成证据原子取代隐性推理——这些结构化的空间-语义三元组，强迫局域化与语义理解之间的紧密对齐。该模型通过层级课程进行训练，课程从反思监督构建逐步推进到强化学习，明确奖励减少推理方差。大量实验表明，EGVOR在逆境条件下显著提升了推理稳定性，为可信的多模态认知提供了更稳健的框架。

ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answering

ConRub-Med：基于共识评分标准的强化学习，用于开放式医学问答

• Authors: Taojie Zhu, Yuan Xia, Tao Sun, Yizhi Wang, Yan Chen, Qunshan He, Tian Guan, Jian Wang, Jinjie Gu, Junwei Liu, Yonghong He
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.10996
• Pdf link: https://arxiv.org/pdf/2608.10996
• Abstract Reinforcement learning with verifiable rewards has been especially effective in mathematics and coding, where answers can be checked automatically. Many open-ended medical questions lack comparably cheap outcome verifiers: responses may be partly correct, incomplete, or contain clinically consequential errors. Rubrics written or validated by physicians offer strong clinical grounding, but involving experts in every instance is costly. Model-generated rubrics make this supervision scalable. We introduce ConRub-Med to preserve useful distinctions as rubric feedback moves from construction to policy optimization. For each prompt, three heterogeneous language models propose atomic criteria independently; a separate model reviews them, retaining only criteria with semantic support from all three generators. Three-State scoring distinguishes correct coverage, missing information, and incorrect claims. Errors receive negative rather than zero credit. When every response in a complete Group Relative Policy Optimization (GRPO) group receives the same final reward, a pairwise judge provides sequence advantages only if both candidate orders agree, without changing the scalar rewards. Groups without ties use vanilla GRPO. In a blinded study matched by question, two medical experts rate panels from the full pipeline as more clinically relevant than panels produced by one generator. Across the evaluated open models, ConRub-Med ranks first on six of nine benchmarks and achieves the highest medical and generalization averages. Using the resulting rubric dataset of 5,166 prompts, it scores $38.98 \pm 1.04$ (mean $\pm$ SD) on HealthBench-Hard, compared with InfiMed-ORBIT's 33.60 with 8,000 samples and 37.30 with 28,000.
• 中文摘要 带有可验证奖励的强化学习在数学和编程领域尤为有效，因为答案可以自动核对。许多开放式医疗问题缺乏相对廉价的结果验证工具：回答可能部分正确、不完整，或包含临床上严重的错误。医生撰写或验证的评分标准提供了坚实的临床基础，但每例都请专家参与成本较高。模型生成的评分标准使这种督导具有可扩展性。我们引入ConRub-Med是为了在评分标准反馈从构建转向策略优化时保持有用的区分。对于每个提示，三个异构语言模型独立提出原子标准;一个独立模型对它们进行审查，仅保留三个生成器的语义支持标准。三州评分区分了正确的保障、缺失的信息和错误的理赔。错误会获得负面而非零分。当一个完整的组相对策略优化（GRPO）组中每个响应都获得相同的最终奖励时，只有当两个候选顺序一致时，两对裁判才会提供序列优势，且不改变标量奖励。没有关联的团体使用原版GRPO。在一项盲测研究中，两位医学专家将整个管道中的面板评为比单一生产者产生的面板更具临床相关性。在评估的开放模型中，ConRub-Med在九个基准中排名第六，并取得了最高的医学和泛化平均值。利用5,166个提示的评分标准数据集，在HealthBench-Hard上得分为$38.98 \pm 1.04$（平均$\pm SD），而InfiMed-ORBIT在8,000个样本中得分为33.60,28,000个样本时为37.30。

Efficient Hypergradient Descent for Inverse Reinforcement Learning

高效的超梯度下降用于逆强化学习

• Authors: Nikita Sevriukov, Anna Barabanova, Uliana Gagarina, Karina Ivanova, Sofiia Kasaeva, Ilya Levin, Marina Sheshukova
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2608.11052
• Pdf link: https://arxiv.org/pdf/2608.11052
• Abstract Inverse reinforcement learning (IRL) aims to recover a reward function under which the resulting policy reproduces the behavior observed in expert demonstrations. A natural approach is to formulate IRL as a bilevel optimization problem, in which the inner level corresponds to policy optimization under the learned reward and the outer level measures the discrepancy between the induced policy and expert data. However, this formulation is computationally challenging in practice because the outer update requires a hypergradient involving an inverse-Hessian-vector product for the inner objective. We address this challenge by showing that, at the inner optimum, the Hessian of the inner objective is proportional to the Fisher information matrix of the policy, yielding a structured Fisher-based hypergradient closely related to Natural Hypergradient Descent. To address the resulting scalability bottleneck associated with large Fisher matrices, we approximate the required inverse-Fisher-vector product using a streaming spectral sketch, avoiding explicit construction of the Fisher matrix. We evaluate our approach against a first-order stochastic bilevel baseline across discrete- and continuous-control environments. The results demonstrate competitive policy performance and strong reward-ranking quality, while Fisher sketching reduces curvature-storage complexity and can improve computational efficiency relative to an explicit Fisher solver.
• 中文摘要 逆强化学习（IRL）旨在恢复一个奖励函数，使得的策略能够重现专家演示中观察到的行为。一种自然的方法是将IRL表述为双层优化问题，其中内层对应于学习奖励下的策略优化，外层则衡量诱导策略数据与专家数据之间的差异。然而，这种表述在实际中计算上具有挑战性，因为外部更新需要一个涉及反黑森向量积的超梯度，用于内部目标。我们通过证明在内最优处，内目标的黑森矩阵与策略的费舍尔信息矩阵成正比，从而产生一个结构化的基于费舍尔的超梯度，与自然超梯度下降密切相关。为解决大型费舍尔矩阵带来的可扩展性瓶颈，我们利用流谱草图近似所需的反费舍尔向量积，避免显式构造费舍尔矩阵。我们将该方法结合一阶随机双水平基线，在离散和连续控制环境中进行评估。结果显示出具有竞争力的策略表现和强劲的奖励排名质量，而费舍尔草图则降低了曲率存储的复杂性，并相较于显式费舍尔求解器能够提高计算效率。

Scheduling Mixed RL Rollouts Beyond Prefix Locality

调度混合强化学习推广超出前缀区域

• Authors: Zetao Hong, Song Yuan, Yuanhao Ding, Yibo Zhu, Daxin Jiang, Zhibin Wang, Chen Tian
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.11152
• Pdf link: https://arxiv.org/pdf/2608.11152
• Abstract Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms. Prefix-aware routing improves inference efficiency through cache reuse and load balancing, but it does not control how heterogeneous rollout sessions compete for KV-cache capacity. When reinforcement learning with verifiable rewards (RLVR), reinforcement learning from human feedback (RLHF), and agentic rollouts share an asynchronous inference service, their distinct sequence structures, interaction patterns, and KV-residency times create substantially different serving demands. Rollout scheduling must account for this heterogeneity without distorting the workload mixture specified by the trainer. We present MISA-T, a routing-layer admission policy for mixed rollout serving. MISA-T combines adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting. In rollout-only ablations on Step3.7 and Qwen3.6-35B-A3B, MISA-T improves rollout throughput over a sweep-tuned cache-aware vLLM Router by 53.3% and 43.6%, respectively, while maintaining high prefix-cache hit rates. In a matched 50-iteration Step3.7 experiment, it increases rollout throughput by 35.6% and reduces mean iteration time by 22.8%, while keeping the consumed workload mixture close to the trainer target and achieving comparable task scores.
• 中文摘要 现代强化学习（RL）大型语言模型（LLM）的后训练流程越来越多地将多个领域和反馈范式的推广工作负载结合起来。前缀感知路由通过缓存重用和负载均衡提升推理效率，但它无法控制异构部署会话如何竞争KV缓存容量。当可验证奖励强化学习（RLVR）、人类反馈强化学习（RLHF）和代理推断共享异步推理服务时，它们独特的序列结构、交互模式和KV驻留时间会产生显著不同的服务需求。推广调度必须考虑到这种异质性，同时不扭曲培训师指定的工作负载组合。我们介绍MISA-T，一种用于混合部署服务的路由层准入政策。MISA-T结合了自适应会话准入、工作负载感知的KV容量分配和驻地时间感知的KV计量。在Step3.7和Qwen3.6-35B-A3B的仅限部署消融中，MISA-T分别将覆盖吞吐量提升了53.3%和43.6%，同时保持了较高的前缀缓存命中率。在匹配的50次迭代Step3.7实验中，它提高了35.6%的推广吞吐量，平均迭代时间减少22.8%，同时保持消耗的工作负载比例接近培训器目标，并实现了相当的任务得分。

Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

测试时间自我演进的图形界面视觉基础，通过反思引导的政策自我蒸馏

• Authors: Shiyu Xuan, Zechao Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.11191
• Pdf link: https://arxiv.org/pdf/2608.11191
• Abstract GUI Visual Grounding is a fundamental capability for GUI agents. Existing models typically freeze their parameters after deployment, limiting their ability to adapt to unseen interfaces. Although recent methods attempt to adapt models via test-time reinforcement learning, they cannot reflect upon failed exploration. To overcome this, we propose a Test-Time Self-Evolving framework that enables models to improve after deployment without human-annotated ground truth. It constructs a closed-loop of Exploration, Evaluation, Reflection, and Internalization. Specifically, the agent first explores unseen interfaces by predicting grounding coordinates for given instructions. To evaluate these explorations, we introduce an MLLM-based Reflector to assess the generated results and provide the corresponding reasoning reflections. To internalize reflection knowledge into the model weights, we propose Reflection-Guided On-Policy Self-Distillation, which translates high-level reasoning into dense token-level supervision via a conditioned self-teacher. Furthermore, we design a Contrastive Calibration method to prevent incorrect auto-regressive prefixes from corrupting the supervisory signals during failed explorations. Extensive experiments across six benchmarks demonstrate our framework's effectiveness, achieving an average accuracy improvement of 7.4% over the base model. To the best of our knowledge, this is the first work to successfully exploit on-policy self-distillation for test-time adaptation in GUI visual grounding. By filling the gap in post-deployment adaptation, our framework completes the self-evolving capability of GUI agents. The code will be released.
• 中文摘要 GUI 可视化接地是 GUI 代理的基本功能。现有模型通常在部署后冻结参数，限制了它们适应未知接口的能力。尽管近期方法尝试通过测试时强化学习来适应模型，但它们无法反映探索失败的情况。为克服这一问题，我们提出了一个测试时间自我演化框架，使模型在部署后无需人工注释的真实数据也能改进。它构建了一个闭环，涵盖探索、评估、反思和内化。具体来说，代理首先通过预测给定指令的接地坐标来探索未见的界面。为评估这些探索，我们引入基于MLLM的反射器，用于评估生成的结果并提供相应的推理反思。为了将反思知识内化到模型权重中，我们提出了反思引导政策自我蒸馏，通过条件化自教师将高层推理转化为密集的代币级监督。此外，我们设计了对比校准方法，以防止错误的自回归前缀在失败探索时损坏监督信号。在六个基准测试中进行了大量实验，展示了我们框架的有效性，平均准确率比基础模型提升了7.4%。据我们所知，这是首个成功利用策略自蒸馏技术实现图形界面可视化基础测试时间适配的研究。通过填补部署后适应的空白，我们的框架完成了图形界面代理的自我演进能力。代码会被释放。

VidForensics-M1: Meta-Detection Reinforcement Learning with Verifiable Temporal Grounding for AI-Generated Video Forensics

VidForensics-M1：具备可验证时间基础的元检测强化学习，用于AI生成视频取证

• Authors: Bowei Liu, Zheng Lu, Yuhan Bian, Xinchen Zhang, Xingming Shui, Yuesheng Huang, Xuhuan Li, Zihao Liu, Yifan Yang, Jun Zhou, Xiu Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.11201
• Pdf link: https://arxiv.org/pdf/2608.11201
• Abstract Recent advances in video generation models have significantly improved the realism of synthetic videos, blurring the boundary between generated and authentic content and raising concerns about misinformation. Existing MLLM-based detectors mainly rely on supervised fine-tuning or label-level reinforcement learning, where coarse supervision limits generalization to unseen scenarios and emerging video generators. To overcome these limitations, we are the first to introduce \textbf{meta-detection} into AI-generated video detection, enabling reliable forgery detection by jointly optimizing predicted labels and supporting evidence within reinforcement learning. This paradigm requires reliable evidence signals and effective mechanisms to integrate them into label-level optimization. Textual rationales provide semantic descriptions of forgery artifacts, but their generation and verification depend on external models, making supervision vulnerable to hallucinations and semantic biases. In contrast, temporal grounding provides more objective and verifiable evidence, as manipulated intervals can be precisely controlled during forgery construction. Based on this insight, we propose an automated data construction pipeline that generates paired real-fake videos by replacing temporal segments with boundary-frame-conditioned video generation models. Furthermore, we introduce \textbf{Evidence-Guided Reward Redistribution}, which performs evidence-aware credit assignment by redistributing rewards among label-correct responses according to evidence quality. This preserves reliable label supervision while encouraging detectors to acquire fine-grained and verifiable forgery localization capabilities. Extensive experiments demonstrate that \textbf{VidForensics-M1} effectively leverages verifiable temporal evidence to achieve robust and generalizable AI-generated video detection.
• 中文摘要 视频生成模型的最新进展显著提升了合成视频的真实性，模糊了生成内容与真实内容的界限，并引发了对错误信息的担忧。现有基于MLLM的检测器主要依赖监督微调或标签级强化学习，粗监督限制了对未见场景和新兴视频生成器的泛化。为克服这些局限，我们率先将 \textbf{meta-detection} 引入 AI 生成的视频检测，通过共同优化预测标签并支持强化学习中的证据，实现可靠的伪造检测。该范式需要可靠的证据信号和有效机制，将其整合进标签层级优化。文本逻辑提供了伪造文物的语义描述，但其生成和验证依赖外部模型，使监督容易产生幻觉和语义偏见。相比之下，时间基础提供更客观且可验证的证据，因为在伪造构造过程中可以精确控制操作的间隔。基于这一见解，我们提出了一种自动化数据构建流程，通过用边界帧条件视频生成模型替代时间段，生成配对的真实假视频。此外，我们引入了\textbf{证据引导奖励再分配}，通过根据证据质量在标签正确回答中重新分配奖励，实现证据意识的信用分配。这既保持了可靠的标签监督，又鼓励探测器获得细粒度且可验证的伪造定位能力。大量实验表明，\textbf{VidForensics-M1} 有效利用可验证的时间证据，实现了稳健且可推广的 AI 生成视频检测。

Keyword: diffusion policy

There is no result