生成时间: 2026-07-17 17:59:10 (UTC+8); Arxiv 发布时间: 2026-07-17 20:00 EDT (2026-07-18 08:00 UTC+8)

今天共有 37 篇相关文章

Keyword: reinforcement learning

Intelligent Three Level Learning Architecture for Autonomous UAV Swarms in Search and Rescue

为搜救中自主无人机群的智能三级学习架构

• Authors: Oleksii Bychkov
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.14093
• Pdf link: https://arxiv.org/pdf/2607.14093
• Abstract This paper presents a novel three level hierarchical learning architecture for autonomous UAV swarms performing search and rescue operations. Unlike conventional approaches that apply a single learning paradigm across all hierarchy levels, the proposed architecture integrates three qualitatively different learning mechanisms corresponding to the biological hierarchy of reflexes, skills, and reasoning such as Hebbian neuroplasticity for individual agent adaptation, multi agent reinforcement learning with graph neural networks and behavior trees for tactical coordination, and model agnostic meta learning with BDI reasoning and a digital twin for strategic decision making. The architecture is formalized through twenty two architectural contracts organized across six components such as BDI, Behavior Trees, GNN, MARL, Neuroplasticity, Meta Learning that collectively provide six classes of formal guarantees such as safety, budget correctness, optimality, liveness, starvation freedom, and inter level consistency. We introduce Swarm Meta Cognition as a compositional property arising from the structured interaction of all three levels, enabling the swarm to monitor its own cognitive state and switch between cognitive strategies. Five constructive progress functions for SAR task types bridge the gap between abstract optimization theory and concrete operational scenarios. The main integration theorem establishes that when all contracts are satisfied, the hybrid neuro-symbolic system preserves all six guarantee classes. For the dynamic case with active learning, five new contracts extend the framework with three additional guarantees such as cognitive resilience, graceful degradation, and monotonic meta improvement. Theoretical analysis demonstrates that the architecture addresses five fundamental limitations of existing hierarchical RL approaches.
• 中文摘要 本文提出了一种新的三级分层学习架构，用于自主无人机群执行搜救任务。与传统方法在所有层级中应用单一学习范式不同，所提架构整合了三种质上不同的学习机制，分别对应反射、技能和推理的生物层级，如针对个体个体适应的Hebbian神经可塑性、基于图神经网络和行为树的多智能体强化学习用于战术协调，以及结合BDI推理和数字的模型无关元学习双生子则负责战略决策。该架构通过22份架构合同形式化，这些合同分布在六个组件中，如BDI、行为树、GNN、MARL、神经可塑性、元学习，这些合同共同提供六类形式保障，如安全性、预算正确性、最优性、活力性、无饥饿性和层级间一致性。我们将群体元认知介绍为一种组合属性，源自三层结构化交互，使群体能够监测自身认知状态并在认知策略间切换。针对SAR任务类型的五个建设性进展函数，弥合了抽象优化理论与具体操作场景之间的鸿沟。主要积分定理证明，当所有契约都满足时，混合神经符号系统保持所有六个保证类。对于主动学习的动态案例，五个新合同扩展了框架，增加了三项额外保障，如认知韧性、优雅退化和单调元改进。理论分析表明，该架构解决了现有分层强化学习方法的五个基本局限。

Interpretable Language Model for Closed-Loop Type 1 Diabetes Control

闭环1型糖尿病控制的可解释性语言模型

• Authors: Maya Sarkar
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.14126
• Pdf link: https://arxiv.org/pdf/2607.14126
• Abstract Type 1 Diabetes (T1D) is a chronic, life-threatening autoimmune condition characterized by the complete destruction of insulin-producing pancreatic beta cells. While Artificial Pancreas Systems (APS) powered by Reinforcement Learning (RL) have shown promise in automating insulin delivery, their ``black-box'' nature makes it hard for patients and doctors to trust them fully. This paper presents LLM-T1D, a promising approach that combines the precision of RL with the clear, human-like reasoning of Large Language Models (LLMs) to create a more transparent and reliable insulin pump controller. By training an expert RL system and distilling its knowledge into fine-tuned LLaMA 3.1 8B and Qwen3 8B models, we developed a controller that not only surpasses the RL system's performance but also explains its decisions in plain, understandable language. Tested on the FDA-approved UVA/Padova T1D simulator, the LLM controllers deliver excellent blood sugar control (73.5% Time in Range) while maintaining strict formal safety verification against hallucinations.
• 中文摘要 1型糖尿病（T1D）是一种慢性、危及生命的自身免疫性疾病，其特征是胰岛素产生胰腺β细胞的完全破坏。虽然由强化学习（RL）驱动的人工胰腺系统（APS）在自动化胰岛素输送方面展现出潜力，但其“黑箱”特性使患者和医生难以完全信任。本文介绍了LLM-T1D，这是一种有前景的方法，结合了强化学习的精确性与大型语言模型（LLM）清晰、类人化的推理，打造出更透明、更可靠的胰岛素泵控制器。通过训练专家级强化学习系统，并将其知识精炼到经过精细调校的LLaMA 3.1 8B和Qwen3 8B模型中，我们开发出了一个不仅超越强化学习系统性能，还能用通俗易懂的语言解释决策的控制器。在FDA批准的UVA/Padova第一型模拟器上测试，LLM控制器实现了出色的血糖控制（73.5%的正常时间），同时保持了对幻觉的严格正式安全验证。

ToolAnchor: Anchoring Counterfactual Context to Boost Agentic Tool-use Capability

ToolAnchor：锚定反事实上下文以提升代理工具使用能力

• Authors: Weiting Liu, Jieyi Bi, Wanqi Zhou, Jianfeng Feng, Yining Ma, Ai Han, Wenlian Lu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.14145
• Pdf link: https://arxiv.org/pdf/2607.14145
• Abstract Tool-augmented large language model agents excel at long-horizon tasks, yet they are typically post-trained on fixed toolsets. When tasks demand new tools, these agents struggle to incorporate them effectively, and retraining from scratch is often impractical. We identify the core obstacle in such toolset expansion problem as behavioral inertia: the tendency of agents to fall back on familiar tools and established reasoning patterns despite having access to new ones. We demonstrate that injecting counterfactual anchor contexts at critical decision points can break this inertia, recovering failed trajectories by eliciting suppressed agent capabilities. To scale this insight, we propose ToolAnchor, a framework that uses teacher models to hypothesize these counterfactual contexts, verifies them via student rollouts, and internalizes the successful interventions through agentic post-training. Extensive evaluations across general AI assistant (GAIA), textual search (BrowseComp), and visual search (VDR-Bench) tasks demonstrate that ToolAnchor consistently exhibits competitive performance under expanded toolsets. Our work bridges the gap between static post-training and dynamic adaptation, charting a new path for scalable agentic reinforcement learning.
• 中文摘要 工具增强的大型语言模型代理在长期任务中表现出色，但通常在固定工具集上进行后训练。当任务需要新工具时，这些代理难以有效整合，从零开始再培训往往不切实际。我们指出工具集扩展问题的核心障碍是行为惯性：代理即使拥有新的推理模式，仍倾向于依赖熟悉的工具和既定的推理模式。我们证明，在关键决策点注入反事实锚点上下文可以打破这种惯性，通过引发抑制剂能力来恢复失败的轨迹。为了扩大这一洞见，我们提出了ToolAnchor框架，该框架利用教师模型假设这些反事实情境，通过学生推广验证，并通过能动的培训后期内化成功干预。对通用人工智能助手（GAIA）、文本搜索（BrowseComp）和视觉搜索（VDR-Bench）任务的广泛评估表明，ToolAnchor在扩展工具集下持续展现出竞争性能。我们的工作弥合了静态后训练与动态适应之间的鸿沟，开辟了可扩展的代理强化学习新路径。

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

分支策略优化：沙箱原生语言代理强化学习

• Authors: Bowei He, Yankai Chen, Xiaokun Zhang, Xue Liu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.14171
• Pdf link: https://arxiv.org/pdf/2607.14171
• Abstract Reinforcement learning has emerged as the dominant paradigm for training large language model (LLM) agents that interact with executable sandboxes. State-of-the-art algorithms such as PPO, RLOO, and GRPO inherit their rollout topology from RLHF: for each prompt, N independent trajectories are sampled from the initial state, and an advantage is computed by subtracting a group baseline. This design ignores a defining property of agent sandboxes. They are deterministic, snapshottable, and resumable from any intermediate state. We argue that this property enables a fundamentally different rollout topology: rather than N independent trees of depth T, one can construct a single tree of N leaves whose siblings share prefixes, and therefore share variance. We instantiate this idea as Branching Policy Optimization (BPO), a sandbox-native RL algorithm that (i) adaptively snapshots the sandbox at high-entropy decision points along a backbone trajectory, (ii) forks K alternative actions per branch point and rolls out each to termination, and (iii) computes per-step advantages from sibling returns rather than from independent prompts. We prove this estimator is unbiased and has strictly lower variance than the trajectory-level baseline, with the reduction equal to the prefix-explained portion of return variance. On WebShop, ALFWorld, and SWE-bench Verified with Qwen2.5-7B and Llama-3.1-8B backbones, BPO improves success by 3.6--6.1 absolute points over GRPO and RLOO at matched compute, halves gradient-norm variance, and matches the best baseline using 38% fewer policy updates.
• 中文摘要 强化学习已成为训练大型语言模型（LLM）代理的主导范式，这些代理与可执行沙盒交互。最先进的算法如PPO、RLOO和GRPO继承了RLHF的展开拓扑结构：每个提示从初始状态中抽取N条独立轨迹，并通过减去组基线计算优势。这种设计忽略了代理沙盒的一个定义性质。它们是确定性的、快照表的，并且可以从任何中间状态恢复。我们认为这一性质使得一种根本不同的展开拓扑：与其有 N 棵深度为 T 的独立树，不如构造一棵由 N 个叶子组成的单一树，其兄弟叶子共享前缀，因此共享方差。我们将这一理念实例化为分支策略优化（BPO），这是一种沙盒原生的强化学习算法，它（i）在骨干轨迹上高熵决策点自适应快照沙盒，（ii）每个分支点分叉K个备用动作并逐步展开至终止，（iii）通过兄弟节点而非独立提示计算每步优势。我们证明该估计量是无偏的，且其方差严格低于轨迹水平基线，其约简等于返回方差中前缀解释部分。在WebShop、ALFWorld和SWE-bench Verified，采用Qwen2.5-7B和Llama-3.1-8B骨干网时，BPO在匹配计算、梯度-范数方差减半、且策略更新次数减少38%的情况下，比GRPO和RLOO提升了3.6%-6.1个绝对点。

RENEW: Towards Learning World Models and Repairing Model Exploitation from Preferences

RENEW：迈向学习世界模型并修复偏好中的模型利用

• Authors: Logan Mondal Bhamidipaty, Mykel Kochenderfer, Subramanian Ramamoorthy
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.14180
• Pdf link: https://arxiv.org/pdf/2607.14180
• Abstract World models are widely used in offline reinforcement learning (RL) to improve sample efficiency and generate experience beyond a fixed dataset. However, they are vulnerable to model exploitation where data coverage is thin. Prior work addresses this either by collecting more expert demonstrations, which is often expensive, unsafe, or unavailable, or by conservative algorithms that avoid uncertain regions, which limits generalization. We propose instead to repair exploitation directly using human preferences over imagined rollouts, leveraging the strong intuitive physics that allows humans to easily spot egregious dynamics hallucinations. We formalize this as Dynamics Learning from Human Feedback (DLHF), a Bradley-Terry preference loss over trajectory log-likelihoods under a learned dynamics model. Unfortunately, naive DLHF is sample inefficient, so we introduce RENEW, which uses epistemic uncertainty to focus finetuning where the model is most exploitable. We evaluate on several Jumanji and classic control environments and find that while naive DLHF requires an outsize preference budget, RENEW makes the framework practical by improving sample efficiency, limiting catastrophic forgetting, and reducing exploitation in pretrained world models. Taken together, our results provide initial evidence that preferences can supervise world model dynamics directly, offering a new approach to addressing exploitation in offline model-based RL.
• 中文摘要 世界模型被广泛应用于离线强化学习（RL），以提升样本效率并生成超越固定数据集的经验。然而，在数据覆盖薄弱的情况下，它们容易受到模型利用的威胁。以往的工作要么通过收集更多专家演示来解决这个问题，这往往成本高昂、不安全或不可得;要么是通过避免不确定区域的保守算法来解决这个问题，这限制了泛化。我们建议直接通过人类偏好修复剥削，而非想象中的推广，利用强大的直觉物理学，使人类能够轻松发现严重的动态幻觉。我们将此形式化为“从人类反馈中学习动力学”（DLHF），即在学习动力学模型下，Bradley-Terry偏好损失相较轨迹对数似然。不幸的是，朴素的DLHF样本效率低下，因此我们引入RENEW，利用认识不确定性将微调聚焦于模型最易利用的地方。我们在多个Jumanji和经典控制环境中进行了评估，发现虽然朴素的DLHF需要超额的优先级预算，但RENEW通过提高样本效率、限制灾难性遗忘以及减少预训练世界模型中的利用，使该框架变得可行。综合来看，我们的结果初步证明偏好可以直接监督世界模型动态，为解决离线基于模型的强化学习中的利用问题提供了新方法。

Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control

动态人形全身控制的语义音频理解

• Authors: J. M. A. Marcelo, M. Brienza, E. Bugli, L. Comito, D. Nardi, D. D. Bloisi, V. Suriani
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.14182
• Pdf link: https://arxiv.org/pdf/2607.14182
• Abstract Recent advances in humanoid robotics and reinforcement learning have enabled the acquisition of highly expressive whole-body motion policies. However, most robotic performances remain based on pre-scripted sequences or externally triggered behaviors, limiting autonomy and responsiveness to dynamic environments. In this work, we introduce a novel multi-modal orchestration framework for semantic audio-driven humanoid control, enabling robots to autonomously select and execute appropriate motion skills in real time. The system processes continuous audio streams and routes them into music or speech branches. Music input is handled via audio fingerprinting and semantic embeddings to retrieve track identity and temporal alignment, allowing dynamic mapping between musical segments and motion policies. Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-robot interaction. Both modalities share a unified interface that schedules skill execution over a reinforcement learning control pipeline. We validate the approach in simulation and on a Unitree G1 humanoid, showing robust sim-to-real transfer and consistent audio-conditioned policy selection. Supplementary materials are available at the following site: this https URL
• 中文摘要 近年来，类人机器人和强化学习的进步使得高度表现力强的全身运动政策成为可能。然而，大多数机器人表演仍基于预设序列或外部触发行为，限制了对动态环境的自主性和响应能力。在本研究中，我们提出了一种新的多模态编排框架，用于语义音频驱动的人形控制，使机器人能够实时自主选择并执行适当的运动技能。该系统处理连续的音频流，并将其路由到音乐或语音分支。音乐输入通过音频指纹识别和语义嵌入处理，以获取轨道身份和时间对齐，从而实现音乐片段和运动策略之间的动态映射。语音输入建立在一个由模仿学来的独立技能库中，实现了人机直接交互。这两种模式共享一个统一接口，通过强化学习控制流程调度技能执行。我们在模拟和Unitree G1人形机器人上验证了该方法，显示出稳健的模拟到真实传输和一致的音频条件策略选择。补充材料可在以下网站获取：此 https URL

Closed-Loop Knowledge Dynamics: An Operational Framework for Saturation and Escape

闭环知识动力学：饱和与逃离的操作框架

• Authors: Xuening Wu, Shan Yu, Shenqin Yin
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.14185
• Pdf link: https://arxiv.org/pdf/2607.14185
• Abstract Feedback-driven loops support iterative improvement in large language models, reinforcement learning, and autonomous discovery, yet their gains often diminish under repeated internal feedback. We study why closed-loop knowledge systems saturate and what external information can move them beyond their current attractors. We introduce a three-level operational framework in which knowledge states $x_t$ evolve through transition kernels $K_{\theta}$ indexed by a structural parameter $\theta$. The governing structure is defined as the observational equivalence class of $\theta$ induced by these kernels, while attractors and basins are properties of the fixed-$\theta$ dynamics. A structural intervention changes $\theta$ and produces a detectable kernel discrepancy on pre-specified probe states, making structural change falsifiable. Using a Lyapunov drift condition, we show that stable internal dynamics approach bounded stability regions with exponentially attenuated transients and a noise-controlled residual floor. We characterize escape through a metric condition on intervention-induced attractor displacement and a baseline-relative KL lower bound for increasing escape probability. This analysis also explains why conditional mutual information alone cannot certify escape: it measures variation among intervention-conditioned updates rather than departure from the no-intervention law. Case studies in LLM code repair, sparse-reward reinforcement learning, and Bayesian optimization use matched continuation controls to illustrate how feedback strength and alignment affect quality-improving escape. Our contribution is an operational connection among stability tools, measurable intervention effects, and cross-domain diagnostics.
• 中文摘要 反馈驱动的循环支持大型语言模型的迭代改进、强化学习和自主发现，但其收益在反复内部反馈下往往会减弱。我们研究闭环知识系统为何会被饱和，以及哪些外部信息能使其超越当前的吸引源。我们引入了一个三层操作框架，其中知识状态 $x_t$ 通过由结构参数 $\theta$ 索引的转移核 $K_{\theta}$ 演变而来。控制结构定义为由这些核诱导的观测等价类 $\theta$，而吸引子和盆则是固定 $\theta$ 动力学的性质。结构干预会改变 $\theta$，并在预设探针状态上产生可检测的核差异，使结构变化可证伪。利用李雅普诺夫漂移条件，我们表明稳定的内部动力学趋近于具有指数衰减瞬变和噪声控制残余底的有界稳定区。我们通过干预诱导吸引子位移的度量条件和基线相对KL下限来描述逃逸概率增加。该分析还解释了为何仅凭条件互信息无法证明逃逸：它衡量的是干预条件更新之间的变异，而非偏离无干预定律。LLM代码修复、稀疏奖励强化学习和贝叶斯优化的案例研究利用匹配延续控制来说明反馈强度和对齐如何影响质量提升的逃逸。我们的贡献是稳定工具、可测量干预效果和跨领域诊断之间的操作连接。

Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning

通过模型无关的下游奖励学习实现长期用户参与优化

• Authors: Dingsu Wang, Filip Ryzner, Kelly He, Armando Ordorica, David Woo, Aditya Mantha, Liyao Lu, Usha Amrutha Nookala, Haoran Guo, Jiacong He, Olafur Gudmundsson, Matt Chun, Krystal Benitez, Dhruvil Deven Badani, Yijie Dylan Wang
• Subjects: Subjects: Machine Learning (cs.LG); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.14192
• Pdf link: https://arxiv.org/pdf/2607.14192
• Abstract As recommender systems mature in the past few years, their optimization objectives have evolved from a primary focusing on short-term behavioral signals to a broader emphasis on long-term user engagement and retention. However, directly optimizing retention is difficult because return signals are sparse, delayed, and only partially attributable to earlier recommendations. Prior work has addressed this challenge with sequential modeling and reinforcement learning, but these approaches typically require task specific reward engineering, substantial computational overhead, and surface specific implementations that are difficult to generalize. In this paper, we present a unified, model-agnostic downstream reward framework for optimizing long-term user value in large-scale recommendation systems. First, we formulate the downstream reward learning problem and develop an offline screening framework to identify session level behaviors that are both observable early and predictive of future retention. We then propose several model-agnostic downstream rewards signals derived from observed user action patterns across multiple sources. We further discuss the engineering effort to productionize the proposed rewards derivations and challenges we faced when adding them to our ranking models. Online A/B experiments demonstrate consistent improvements in engagement and retention-related metrics, and the framework has been deployed across multiple Pinterest surfaces, including Homefeed, Related Pins, Search, and Notifications.
• 中文摘要 随着推荐系统在过去几年中日益成熟，其优化目标已从最初关注短期行为信号，转变为更广泛地强调长期用户参与度和留存率。然而，直接优化保留率较为困难，因为回传信号稀疏、延迟，且仅部分归因于早期推荐。此前已有研究通过顺序建模和强化学习解决了这一挑战，但这些方法通常需要任务特定的奖励工程、大量的计算开销以及难以泛化的表面特定实现。本文提出了一个统一的、模型无关的下游奖励框架，用于优化大规模推荐系统中的长期用户价值。首先，我们提出了下游奖励学习问题，并开发离线筛选框架，以识别既可早期观察又能预测未来记忆的会话级行为。随后，我们提出了若干模型无关的下游奖励信号，这些信号源自多个来源中观察到的用户行为模式。我们还进一步讨论了将拟议奖励的工程生产工作、推导以及在将它们添加到排名模型时面临的挑战。在线A/B实验显示参与度和留存相关指标持续提升，该框架已部署在多个Pinterest表面，包括首页推送、相关图钉、搜索和通知。

Lyapunov Guidance: A Unified Framework for Stabilizing Generative Flows

李雅普诺夫指导：稳定生成流的统一框架

• Authors: Jingdong Zhang, Xinze Li, Yize Jiang, Luan Yang, Minkai Xu, Junhong Liu
• Subjects: Subjects: Machine Learning (cs.LG); Dynamical Systems (math.DS); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.14272
• Pdf link: https://arxiv.org/pdf/2607.14272
• Abstract Flow matching has emerged as an effective framework for learning complex data distributions, but adapting pretrained flow models to new tasks often requires computationally expensive retraining. Post-training guidance provides a more efficient alternative, but existing methods are largely heuristic and offer no explicit stability guarantees. We address this limitation by proposing LyaGuide, a unified Lyapunov-guided framework that formulates flow guidance as a Lyapunov control problem. Our main theoretical result establishes an equivalence between guided flow matching and Lyapunov control, thereby unifying common guidance strategies, such as classifier guidance, reward guidance, and energy-based guidance, within a single control-theoretic framework. To enforce the Lyapunov condition, we introduce a pseudo-projection operator with a closed-form expression that endows learned or heuristic guidance terms with explicit stability guarantees. LyaGuide supports two practical settings: a model-driven setting, where the target guidance distribution is specified through a known Lyapunov function, and a data-driven setting, where the guidance is adapted from task-specific downstream data. LyaGuide is compatible with existing guidance methods, introduces minimal additional computational overhead, and is straightforward to integrate in practice. Extensive experiments on synthetic benchmarks, image inverse problems, reinforcement learning planning, and energy-based modeling demonstrate consistent improvements in sample quality, guidance fidelity, and robustness, while maintaining computational efficiency.
• 中文摘要 流匹配已成为学习复杂数据分布的有效框架，但将预训练流模型适应新任务通常需要计算量大的再训练。训练后指导提供了更高效的替代方案，但现有方法大多是启发式的，没有明确的稳定性保证。我们通过提出LyaGuide来解决这一局限性，这是一个统一的李雅普诺夫引导框架，将流引导表述为李雅普诺夫控制问题。我们的主要理论结果确立了引导流匹配与李雅普诺夫控制之间的等价性，从而在单一的控制理论框架内统一了常见的引导策略，如分类器引导、奖励引导和基于能量的指导。为强制执行李雅普诺夫条件，我们引入一个伪投影算子，其闭式表达式赋予学习或启发式指导项显式稳定性保证。LyaGuide 支持两种实用设置：模型驱动设置，通过已知的李雅普诺夫函数指定目标引导分布;以及数据驱动设置，指导从任务特定的下游数据中调整。LyaGuide兼容现有的指导方法，增加的计算开销最小，且在实际中易于集成。在合成基准测试、图像逆问题、强化学习规划和基于能量的建模等方面的广泛实验显示，样本质量、导引忠实度和鲁棒性持续提升，同时保持计算效率。

SD-MAR: Multi-image Analytical Reasoning via Synthetic Data and Reinforcement Learning

SD-MAR：通过合成数据和强化学习实现多图像分析推理

• Authors: Shiyu Yuan, Sourav Sanjukta Bhabesh, Zhe Wang, Dmitriy Bespalov, Wesley Rose, Huzefa Rangwala
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.14333
• Pdf link: https://arxiv.org/pdf/2607.14333
• Abstract Vision Language Models (VLMs) demonstrate strong perceptual abilities but remain limited in tasks requiring analytical reasoning across multiple visual states, such as multi-image comparison, change detection, and multi-step visual inference. These capabilities are critical for real-world multimodal applications where reasoning must be grounded in systematic differences between visual contexts. However, existing benchmarks rarely require both explicit visual comparison and analytical reasoning, leaving this capability underexplored. To address this gap, we introduce SD-MAR (Synthetic Data for Multi-image Analytical Reasoning), a framework for training and evaluating VLMs on multi-image analytical reasoning. SD-MAR constructs paired visual scenarios through controlled perturbations and generates reasoning tasks spanning semantic change attribution and quantitative comparison. We further train VLMs using GRPO-lite with Backward Discounted Allocation (BDA), a reinforcement learning approach that removes KL regularization to encourage stronger policy optimization while allocating greater credit to the later reasoning steps where analytical conclusions are formed. Experiments on Qwen2.5-VL-7B and InternVL3-8B show that GRPO-lite fine-tuning on SD-MAR improves in-domain accuracy by up to 36.95%, with Qwen2.5-VL-7B outperforming GPT-4.1 on the SD-MAR benchmark. Importantly, out-of-domain generalization is preserved or improved: performance remains within 1% on MME, MMMU-Pro, and MathVista, while improving by up to 4% on MMBench. LLM-as-judge evaluation further demonstrates consistent improvements in logical coherence and explanation quality across both models.
• 中文摘要 视觉语言模型（VLMs）展现出强大的感知能力，但在需要跨多视觉状态进行分析推理的任务上仍有限，如多重图像比较、变化检测和多步视觉推断。这些能力对于现实世界的多模态应用至关重要，因为推理必须建立在视觉语境之间系统的差异基础上。然而，现有基准很少同时要求明确的视觉比较和分析推理，因此这一能力尚未被充分开发。为弥补这一空白，我们引入了SD-MAR（多图像分析推理合成数据）框架，用于训练和评估多图像分析推理VLMs。SD-MAR通过受控扰动构建成对视觉场景，并生成涵盖语义变化归因和定量比较的推理任务。我们进一步使用GRPO-lite和后退贴现分配（BDA）训练VLMs，这是一种强化学习方法，去除KL正则化，鼓励更强的策略优化，同时给予后期推理步骤更多贡献，形成分析结论。在Qwen2.5-VL-7B和InternVL3-8B上的实验显示，SD-MAR上的GRPO-lite微调能使领域内准确率提升高达36.95%，Qwen2.5-VL-7B在SD-MAR基准测试中优于GPT-4.1。重要的是，域外泛化得以保留或提升：MME、MMMU-Pro 和 MathVista 的性能保持在 1% 以内，而在 MMBench 上提升最多 4%。LLM作为评判的评估进一步展示了两种模型在逻辑连贯性和解释质量方面的持续提升。

A Noise-Robust Elicit-to-Optimize Framework for Distortion Riskmetrics via Inverse Reinforcement Learning

通过逆强化学习实现噪声稳健的失真风险指标诱导优化框架

• Authors: Yang Liu, Yuhao Liu, Yunran Wei
• Subjects: Subjects: Machine Learning (cs.LG); Risk Management (q-fin.RM)
• Arxiv link: https://arxiv.org/abs/2607.14373
• Pdf link: https://arxiv.org/pdf/2607.14373
• Abstract We propose a noise-robust elicit-to-optimize framework that integrates inverse reinforcement learning (IRL) and reinforcement learning (RL) for eliciting agents' risk preferences and optimizing policies under a broad class of risk objectives characterized by distortion riskmetrics. On the elicitation side, we propose an adaptive Bayesian IRL method that infers agents' latent risk objectives from their noisy observed decisions, explicitly allowing agents to take stochastic and suboptimal actions. We establish the existence of a finite set of distinguishing questions that identifies the preferred distortion riskmetric within the candidate class and prove that the convergence rate of the algorithm is of order $O(\exp(-cm+O(\sqrt{m\log m})))$ under general settings, where $c>0$ is a constant and $m$ denotes the number of algorithm iterations. On the optimization side, we develop a model-free RL algorithm for optimizing policies under conditional distortion riskmetrics. By representing the objective as an integral of the conditional cost quantile function with respect to the distortion function, the method unifies distortion-riskmetric objectives. We optimize diverse risk objectives by extending the Proximal Policy Optimization (PPO) algorithm with policy, value, and quantile neural networks, where the quantile network estimates the full conditional cost quantile function and enables numerical evaluation of general risk objectives. A comprehensive empirical study demonstrates the framework's elicitation accuracy and effectiveness in complex financial environments.
• 中文摘要 我们提出了一个噪声鲁棒的诱导优化框架，整合逆向强化学习（IRL）和强化学习（RL），用于引发智能体的风险偏好并在以扭曲风险指标为特征的广泛风险目标类别下优化策略。在诱导方面，我们提出了一种自适应的贝叶斯IRL方法，通过其噪声观察决策推断智能体潜在风险目标，明确允许智能体采取随机且次优的行为。我们建立了一组有限的区分问题，用于识别候选类别内的首选扭曲风险度量，并证明算法的收敛率在一般设置下为$O（\exp（-cm+O（\sqrt{m\log m}））））$，其中$c>0$为常数，$m$表示算法迭代次数。在优化方面，我们开发了一种无模型的强化学习算法，用于在条件扭曲风险指标下优化策略。通过将目标表示为条件成本分位函数相对于扭曲函数的积分，该方法统一了扭曲-风险指标目标。我们通过将近距离策略优化（PPO）算法扩展为策略、价值和分位数神经网络，优化多样化的风险目标，其中分位数网络估计完整的条件成本分位函数，并实现对一般风险目标的数值评估。一项全面的实证研究证明了该框架在复杂金融环境中的引出准确性和有效性。

An offline approach to fNIRS-guided reinforcement learning for robot behavior

fNIRS引导强化学习的离线方法，用于机器人行为

• Authors: Julia Santaniello, Madelaine Brower, Benson Jiang, Donatello Sassaroli, Robert Jacob, Jivko Sinapov
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.14393
• Pdf link: https://arxiv.org/pdf/2607.14393
• Abstract Human-in-the-loop Reinforcement Learning has become a popular approach to training, finetuning, and aligning robot behavior with user preferences. Our paper explores the feasibility of using brain signals via functional near-infrared spectroscopy (fNIRS) to modulate robot learning in simulation. We compare agents trained on passive (observational) versus active (demonstrative) interaction tasks, and test multiple methods for enhancing the RL algorithm with the neural signal, focusing on parameter augmentation rather than replacement. We further examine how model granularity and noise affect agent learning. Our results show that this framework is effective: the neural signal improves learning when augmenting trajectory priorities and state-action q-values. Additionally, the framework learns successfully from offline data, offering a practical alternative for settings where real-time BCI setups are impractical or only limited data is available.
• 中文摘要 人机参与式强化学习已成为训练、微调和与用户偏好对齐机器人行为的流行方法。我们的论文探讨了利用功能性近红外光谱（fNIRS）利用大脑信号调制机器人学习在模拟中的可行性。我们比较了被动（观察）与主动（演示）交互任务训练的代理，并测试多种增强强化学习算法的神经信号方法，重点是参数增强而非替代。我们还进一步探讨模型粒度和噪声如何影响代理学习。我们的结果表明该框架有效：神经信号在增强轨迹优先级和状态-动作q值时提升了学习效果。此外，该框架能够成功从离线数据中学习，为实时BCI设置不切实际或数据有限的环境提供了实用替代方案。

Safe Execution of RL Policies Via Acceleration-Based CBF-QP Constraint Enforcement for Real-World Robotic Deployments

通过基于加速的CBF-QP约束执行，安全执行现实世界机器人部署中的强化学习策略

• Authors: Bastien Muraccioli, Alice Cariou, Pierre-Alexandre Leziart, Mathieu Celerier, Arnaud Demont, Gentiane Venture, Mehdi Benallegue
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.14488
• Pdf link: https://arxiv.org/pdf/2607.14488
• Abstract Reinforcement Learning (RL) has demonstrated remarkable capabilities for solving complex robotic control problems, but its lack of safety guarantees severely limits deployment on hardware. In particular, as legged robots and manipulators often operate near safety-critical boundaries, out-of-distribution states can lead to failure upon deployment. To address this, we introduce Acc-CBF-QP, an acceleration-based Quadratic Program (QP) safety filter using Control Barrier Functions (CBFs) that constrains any RL policy onto a safe set at runtime without modifying training. The method applies to unconstrained and Safe-RL policies, and enforces joint position, velocity, torque, and collision constraints within a unified optimization framework. A key contribution is the formulation of RL+QP tasks that regulate deviation from the RL command when constraints would otherwise be violated. We introduce a TorqueTask, minimizing torque deviation, and a Forward Dynamics Task, minimizing induced acceleration deviation, thus providing principled control over safety-performance trade-offs. Experiments on a 7-DoF Kinova Gen3 manipulator and a 19-DoF Unitree H1 humanoid, both in simulation and on hardware, highlight substantial reductions in constraint violations. On the real H1 hardware, a Safe-RL policy alone yielded 10.04 violations/s, which were reduced by 92% to 0.80 violations/s when augmented with Acc-CBF-QP. On the Kinova Gen3, Acc-CBF-QP fully eliminated violations. Nominal task performance of the RL objective is preserved in violation-free regimes. Under aggressive velocity commands on H1, Acc-CBF-QP improves execution by preventing constraint-induced shutdowns, yielding longer survival times. The full pipeline is open-source.
• 中文摘要 强化学习（RL）已展现出解决复杂机器人控制问题的卓越能力，但其缺乏安全保障严重限制了硬件上的部署。特别是，由于有腿机器人和机械臂常常在安全关键边界附近运行，部署时的分发状态可能导致故障。为此，我们引入了Acc-CBF-QP，一种基于加速的二次计划（QP）安全过滤器，使用控制障碍函数（CBFs），在运行时将任意强化学习策略限制在安全集上，而无需修改训练。该方法适用于无约束和安全强化策略，并在统一的优化框架内强制执行联合位置、速度、扭矩和碰撞约束。一个关键贡献是制定了RL+QP任务，用于调节在约束可能被违反时偏离RL命令的情况。我们引入了扭矩任务（TorqueTask），以最小化扭矩偏差，以及前向动力学任务（Forward Dynamics Task），以最小化诱导加速度偏差，从而对安全与性能权衡提供原则性控制。在模拟和硬件上，对7景深Kinova Gen3操作臂和19景度Unitree H1人形生物的实验显示了约束违规的显著减少。在真实的H1硬件上，仅Safe-RL策略就产生了10.04次违规/秒，加上Acc-CBF-QP后，这一数字减少了92%，达到0.80次。在Kinova Gen3上，Acc-CBF-QP完全消除了违规。强化学习目标的名义任务表现在无违规的状态下得以保持。在H1的激进速度指令下，Acc-CBF-QP通过防止约束导致的停机来提升执行效率，从而延长存活时间。整个流程都是开源的。

Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards

具有可验证奖励的强化学习非空泛化界限

• Authors: Yuxuan Zhu, Rohan Alur, Daniel Kang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.14506
• Pdf link: https://arxiv.org/pdf/2607.14506
• Abstract While reinforcement learning with verifiable rewards (RLVR) is widely used to improve the reasoning capabilities of large language models (LLMs), the generalizability of the resulting models remains poorly understood. In this work, we establish the first non-vacuous generalization bounds for parameter-efficient RLVR fine-tuning at the billion-parameter scale. Our approach adapts PAC-Bayes compression bounds to this setting, and addresses the inherent stochasticity of token generation by applying the Gumbel-max reparameterization trick. To operationalize these bounds, we propose the Progressive RLVR framework, which integrates RLVR with on-policy distillation, TinyLoRA, and model quantization. Progressive RLVR empirically retains 84-97% performance of standard LoRA fine-tuning while producing models that are 14,796x more compressible. We show that this framework yields non-vacuous generalization bounds in four domains: mathematical problem-solving, programming, general-knowledge reasoning, and Text-to-SQL. Our bounds exceed the accuracy of the base model by 9-51% and lie within 6-11% of the accuracy of the fine-tuned models.
• 中文摘要 虽然带有可验证奖励的强化学习（RLVR）被广泛用于提升大型语言模型（LLMs）的推理能力，但所得模型的推广性仍不充分。在本研究中，我们建立了首个非空推广界，用于十亿参数尺度上参数高效的RLVR微调。我们的方法将PAC-贝叶斯压缩界限调整到该环境，并通过应用Gumbel-max重参数化技巧解决了代币生成的固有随机性。为实现这些界限，我们提出了渐进式RLVR框架，该框架将RLVR与策略内提炼、TinyLoRA和模型量子化整合。渐进式RLVR在实现标准LoRA微调性能的同时，经验上保留了84%-97%的性能，同时产生的模型可压缩性提升了14,796倍。我们证明该框架在四个领域产生了非空推广界限：数学问题解决、编程、泛识推理和文本转SQL。我们的界限比基础模型的准确率高出9-51%，也在微调模型的6-11%以内。

A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models

一个用于微调离散扩散模型的连续时间强化学习框架

• Authors: Zikun Zhang, Jiayuan Sheng, David D. Yao, Wenpin Tang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.14522
• Pdf link: https://arxiv.org/pdf/2607.14522
• Abstract We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces via a stochastic control approach, where the state dynamics are modeled as a controlled continuous-time Markov chain (CTMC). We consider policy optimization problems and derive the corresponding policy gradient methods, leading to continuous-time variants of proximal policy optimization (PPO) and group relative policy optimization (GRPO). As a primary application, we develop a complete continuous-time RL framework for fine-tuning score-based discrete diffusion models. The proposed framework enables reward-driven optimization without requiring differentiability on the reward signals. In contrast to the existing GRPO-based approaches that only rely on terminal rewards, our formulation allows intermediate reward or advantage signals to be incorporated throughout the denoising trajectory. Importantly, when specialized to masked diffusion models (MDMs), our framework encompasses a rich class of policy parameterizations over the vocabulary simplex with analytically tractable probability ratios, providing a unified perspective on exploration and policy optimization in MDMs. For masked diffusion large language models (dLLMs), we further propose trajectory subsampling techniques to efficiently estimate computationally prohibitive trajectory likelihoods, reducing the computational cost of computing per-position probability ratios. We showcase the effectiveness of our methods on both low-dimensional entropy-regularized optimization problems and RL post-training of dLLMs on mathematical reasoning and coding tasks.
• 中文摘要 我们通过随机控制方法，在连续时间内提出强化学习（RL），并可能包含离散状态空间，可能还有任意作用空间，其中状态动力学被建模为受控连续时间马尔可夫链（CTMC）。我们考虑策略优化问题，推导相应的策略梯度方法，从而产生近端策略优化（PPO）和群相对策略优化（GRPO）的连续时间变体。作为主要应用，我们开发了一个完整的连续时间强化学习框架，用于微调基于分数的离散扩散模型。该框架实现了奖励驱动的优化，而无需对奖励信号进行微分。与仅依赖终端奖励的现有GRPO方法不同，我们的表述允许在整个去噪轨迹中纳入中间奖励或优势信号。重要的是，当我们专门针对掩蔽扩散模型（MDM）时，我们的框架涵盖了丰富的策略参数化类别，具有可解析的概率比，为MDM中的探索和策略优化提供了统一视角。对于掩蔽扩散大型语言模型（dLLMs），我们进一步提出了轨迹子采样技术，以高效估计计算上难以承受的轨迹似然，从而降低计算每位置概率比的计算成本。我们展示了方法在低维熵正则化优化问题和dLLM后训练中的强化学习数学推理和编码任务中的有效性。

3D Geometric Tooth Alignment Planning via Deep Reinforcement Learning

通过深度强化学习实现三维几何牙齿对齐规划

• Authors: Yong Li, Jianwen Lou, Jiayue Ma, Yao-Xiang Ding, Youyi Zheng, Haihua Zhu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.14544
• Pdf link: https://arxiv.org/pdf/2607.14544
• Abstract 3D geometric tooth alignment planning, which determines sequential trajectories from initial malocclusion to the final target alignment, is a cornerstone of modern digital orthodontics. This paper presents a novel deep reinforcement learning (DRL) framework to automate the generation of these alignment paths. We formulate the planning process as a Markov Decision Process (MDP) to capture its sequential decision-making nature, focusing on optimizing geometric trajectories while integrating essential spatial constraints, such as inter-dental collision avoidance and path efficiency. The proposed method leverages the Deep Deterministic Policy Gradient (DDPG) algorithm, enhanced by three key innovations: (1) a Transformer-based agent to model complex spatial interactions between teeth and manage high-dimensional state-action spaces; (2) a dynamic masking scheme that restricts movement to a sparse subset of teeth per step, better reflecting the clinical logic of sequential alignment; and (3) a two-stage curriculum learning strategy that gradually increases task difficulty to ensure training stability and efficient path discovery. We evaluate our approach on a dataset of 10K expert-designed treatment plans based on clinical data. Experimental results demonstrate that our method outperforms existing baselines in terms of path safety and geometric efficiency, providing a robust and automated solution for 3D geometric orthodontic alignment planning.
• 中文摘要 三维几何牙齿对齐规划，通过确定从初始错位到最终目标排列的顺序轨迹，是现代数字正畸学的基石。本文提出了一种新型深度强化学习（DRL）框架，用于自动化生成这些对齐路径。我们将规划过程制定为马尔可夫决策过程（MDP），以体现其连续决策特性，重点优化几何轨迹，同时整合牙间碰撞避免和路径效率等关键空间约束。该方法利用了深度确定性策略梯度（DDPG）算法，并通过三项关键创新得到了增强：（1）基于Transformer的智能体用于建模牙齿间复杂的空间交互并管理高维状态-动作空间;（2）动态掩蔽方案，限制每步移动至稀疏的牙齿子集，更符合临床顺序排列逻辑;以及（3）两阶段课程学习策略，逐步增加任务难度，以确保训练稳定性和高效路径发现。我们基于临床数据，基于1万个专家设计的治疗方案数据集来评估我们的方法。实验结果表明，我们的方法在路径安全和几何效率方面优于现有基线，为三维几何正畸对准规划提供了稳健且自动化的解决方案。

HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents

HyMobileAgent：高效图形界面代理的数据环境共扩展

• Authors: Hy Vision Team, Huawen Shen, Zhengyang Tang, Shangpin Peng, Liang Wu, Anran Zhang, Weinong Wang, Yiduo Guo, Chenxin Li, Zhengyao Fang, Yang Ding, Junyi Li, Fei Tang, Zheng Ruan, Yi Zhang, Xingran Zhou, Dingchen Yang, Sunqi Fan, Zhiyi Wan, Han Hu, Xin Lai, Pengyuan Lyu, Chengquan Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.14548
• Pdf link: https://arxiv.org/pdf/2607.14548
• Abstract As large multimodal models move from understanding content to operating on digital environments, mobile GUI has emerged as a challenging and consequential testbed for digital embodied intelligence. Mobile agents operate under three coupled constraints: precise perception of complex interfaces, scalable acquisition of high-quality interaction data, and robust long-horizon decision making under compounding execution errors. This report presents HyMobileAgent, a mobile GUI agent built on Hy3.0-VL-A3B, a vision-native foundation model featuring native any-resolution input, an A3B-scale deployment budget, and a 32K context window to model extended interaction histories. Rather than relying solely on model scaling, we develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction. Our framework integrates a GUI perception flywheel combining mock-interface synthesis, rejection sampling, and icon-specific augmentation; a knowledge pipeline that transforms tutorial videos into structured interaction data; a million-scale action data pipeline deployed across more than 2000 sandbox and real-device instances with automated failure attribution; the PhoneWorld Mock App Factory, providing a resettable training environment with 34 mock applications and over 34000 tasks; and a structured Planning-and-Reflection mechanism with explicit dead-loop detection for reliable long-horizon execution. We also introduce a progressive training recipe consisting of mid-training, supervised fine-tuning, and reinforcement learning with task-specific reward designs.
• 中文摘要 随着大型多模态模型从理解内容转向数字环境中运行，移动图形用户界面已成为数字具身智能的挑战性且深远的试验场。移动代理在三个耦合约束下工作：对复杂界面的精确感知、高质量交互数据的可扩展获取，以及在累计执行错误下实现稳健的长期决策。本报告介绍了HyMobileAgent，一款基于Hy3.0-VL-A3B构建的移动图形界面代理，是一个基于愿景的基础模型，具备原生任意分辨率输入、A3B规模的部署预算，以及用于建模扩展交互历史的32K上下文窗口。我们不再仅依赖模型缩放，而是开发了一个联合的数据与环境为中心的扩展框架，以解决移动交互的关键瓶颈。我们的框架集成了一个图形用户界面感知飞轮，结合了模拟界面综合、拒绝采样和特定图标增强;将教程视频转化为结构化交互数据的知识管道;一个百万级行动数据流水线部署在2000多个沙盒和真实设备实例上，并实现了自动故障归因;PhoneWorld 模拟应用工厂，提供可重置的培训环境，包含34个模拟应用和超过34000个任务;以及结构化的规划与反思机制，带有显式的死循环检测，实现可靠的长视野执行。我们还引入了渐进式训练方案，包括训练中段、监督式微调和强化学习，并设计针对任务的奖励。

Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

超越熵：通过对比策略优化实现的正确性感知优势塑造

• Authors: Weiwen Xu, Jia Liu, Hou Pong Chan, Long Li, Deng Cai, Min Chen, Hao Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.14614
• Pdf link: https://arxiv.org/pdf/2607.14614
• Abstract Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.
• 中文摘要 带有可验证奖励的强化学习（RLVR）通常使用熵来塑造优势。然而，熵无法区分有用的不确定性和有害的混淆，限制了其作为正确性信号的有效性。我们提出了对比策略优化（CPO），利用引用引导分布与普通生成分布之间的代币级对比性差异，进行正确性感知优势塑造。理论和实证结果都表明，这种分歧可靠地表明了代币层面的正确性。我们进一步证明，On-policy 蒸馏是 CPO 的一个特例，其中后验分布由外部教师模型实现。CPO还解决了零优势问题。域内和域外基准测试的实验表明，CPO在保持强强泛化性的同时，远超基于熵的RLVR方法。进一步分析表明，正确和错误的回答自然地支持探索和利用，平衡两者能带来最佳性能。

NavCMPO: Critic-Guided MeanFlow Policy Optimization for Adaptive Navigation

NavCMPO：自适应导航的批评者引导平均流策略优化

• Authors: Junjie An, Yi Wu, Xiao Liu, Yiqun Zhou, Yuechen Wu, Xiaoqing Guan, You Wang, Guang Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.14643
• Pdf link: https://arxiv.org/pdf/2607.14643
• Abstract End-to-end diffusion-based policies have demonstrated strong performance in mapless visual navigation, but their iterative denoising process introduces substantial inference latency, while behavior cloning limits performance to the quality of expert demonstrations. We present NavCMPO, a two-stage adaptive navigation framework that combines few-step MeanFlow trajectory generation, critic-guided refinement, and reinforcement learning fine-tuning. During pre-training, an obstacle proximity prediction task encourages the visual representation to capture obstacle-aware spatial information. To compensate for the degradation in obstacle avoidance caused by few-step generation, Critic-Guided Trajectory Refinement (CGTR) uses gradients from a critic trained with obstacle-point-cloud supervision to refine intermediate trajectories. During adaptation, the MeanFlow policy is fine-tuned using Proximal Policy Optimization with behavior-cloning regularization, while the critic is updated to accommodate embodiment-specific observation changes. Under a matched training budget on the InternVLA-N1 benchmark, NavCMPO achieves an average success rate of 74.7\%, exceeding the retrained NavDP baseline by 6.4 percentage points, while reducing inference latency from 85\,ms to 60\,ms. Experiments on a Unitree Go2 further demonstrate effective sim-to-real transfer.
• 中文摘要 端到端扩散策略在无地图视觉导航中表现出优异性能，但其迭代去噪过程带来了显著的推理延迟，而行为克隆则将性能限制在专家演示的质量上。我们介绍NavCMPO，一个两阶段的自适应导航框架，结合了少步MeanFlow轨迹生成、批评者引导的细化和强化学习的微调。在预训练阶段，障碍物接近预测任务鼓励视觉表征捕捉对障碍物的空间感知信息。为补偿少步生成导致的障碍物规避能力下降，批判者引导轨迹细化（CGTR）利用经过障碍点云监督训练的批评者提供的梯度来细化中间轨迹。在适应过程中，均流策略通过近端策略优化和行为克隆正则化进行微调，同时更新批评者以适应具象特有的观察变化。在InternVLA-N1基准匹配训练预算下，NavCMPO的平均成功率为74.7%，比重新训练的NavDP基线高出6.4个百分点，同时将推理延迟从85毫秒降至60毫秒。Unitree Go2上的实验进一步证明了模拟到现实的有效传输。

Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignment

停止思考，开始寻找：通过无推理对齐实现多模态文档问答的高效后期培训

• Authors: Harikrishnan P M, Goutham Vignesh, Ganesh Parab, Saisubramaniam Gopalakrishnan, Vishal Vaddina, Varun V, Rohit Agrawal
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.14682
• Pdf link: https://arxiv.org/pdf/2607.14682
• Abstract Efficient multimodal document question answering with explicit visual grounding, locating the precise document region that supports each answer remains an open challenge. Current approaches bifurcate into Supervised Fine-Tuning (SFT), which requires large annotated datasets and reaches optimization plateaus, and reasoning-centric Reinforcement Learning (RL), which depends on verbose intermediate traces that inflate inference token cost without clear benefit. We introduce Perception-RFT, a training framework that applies Group Relative Policy Optimization (GRPO) to multimodal document QA, bypassing intermediate reasoning tokens to directly align visual features with structured grounding outputs. To rigorously evaluate the necessity of reasoning, we construct a reasoning variant under identical reward settings. We find that reasoning-enabled models suppress their reasoning traces during training, converging to direct perception-based policies at the 4B parameter scale, reducing per-query inference token length by more than 60%, while reasoning-enabled RL underperforms perception-only training. Through a fine-grained analysis of Qwen3-VL-4B optimization dynamics, we confirm that SFT saturation and cold-start RL instability established in text-domain post-training extend to multimodal, and identify a previously uncharacterized Grounding Divergence: a selective trade-off between semantic robustness and geometric precision on two out of distribution (OOD) benchmarks (4,828 samples) under joint RL optimization. We further show that an early SFT$\rightarrow$RL transition achieves comparable precision with 65% less training data.
• 中文摘要 高效多模态文档题答与明确的视觉基础，定位支持每个答案的精确文档区域仍是一个开放挑战。当前方法分为监督微调（SFT），需要大量注释数据集并达到优化平台;以及以推理为中心的强化学习（RL），依赖冗长的中间痕迹，推理标记成本膨胀却无明显收益。我们介绍了Perception-RFT，这是一个将群相对策略优化（GRPO）应用于多模态文档质量保证的训练框架，绕过中间推理标记，直接将视觉特征与结构化的基础输出对齐。为了严格评估推理的必要性，我们在相同的奖励设置下构建了一个推理变体。我们发现，推理驱动模型在训练过程中抑制推理痕迹，趋向于4B参数尺度的直接感知策略，使每次查询推理令牌长度减少60%以上，而推理驱动的强化学习在仅感知训练中表现不佳。通过对Qwen3-VL-4B优化动态的细致分析，我们确认了文本域训练后建立的SFT饱和和冷启动RL不稳定性可扩展至多模态，并识别出此前未被表征的基准发散：在两个非分布（OOD）基准测试（4,828个样本）下，联合RL优化下，语义鲁棒性与几何精度之间的选择性权衡。我们还进一步证明，早期的SFT$\rightarrow$RL转换在训练数据减少65%的情况下实现了相当的精度。

MIND-CAVs: Multi-Intelligence Negotiation and Decision System for CAVs based on Intent-Driven Autonomy

MIND-CAVs：基于意图驱动自主性的多智能协商与决策系统

• Authors: Mainak Mondal, Yihang Feng, Yangchao Luo, Han Song
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.14688
• Pdf link: https://arxiv.org/pdf/2607.14688
• Abstract Modern autonomous vehicles largely operate as isolated agents: they rely on on-board perception and decision modules and broadcast Basic Safety Messages (BSMs) that expose only low-level kinematic state. While existing cooperative driving frameworks enable limited sensor sharing, they rarely communicate high-level maneuver intentions, and edge computing is primarily used for content delivery rather than decision arbitration. As a result, current connected autonomy lacks a principled mechanism for making globally consistent, intent-aware coordination decisions across vehicles. To address this gap, we propose MIND-CAVs, a Multi-Intelligence Negotiation and Decision framework for connected autonomous vehicles (CAVs) based on intent-driven autonomy. Each vehicle abstracts raw sensor observations into structured intent representations, exchanges them over V2X links, and receives globally consistent coordination plans from roadside edge servers. Edge agents combine learned and rule-based arbitration mechanisms to negotiate conflicting intents among vehicles, while a cloud platform records decisions for auditing and continual retraining. We implement MIND-CAVs in a CARLA-based AI-in-the-loop platform and evaluate it in multi-lane highway scenarios involving conflicting maneuvers and route-constrained exits. Experimental results show improved maneuver completion time and reduced unsafe proximity and unnecessary braking compared with isolated autonomy, first-come-first-served arbitration, and multi-agent reinforcement learning baselines.
• 中文摘要 现代自动驾驶车辆大多作为独立的代理存在：它们依赖车载感知和决策模块，以及仅暴露低水平运动学状态的基本安全信息（BSM）。虽然现有的协作驱动框架支持有限的传感器共享，但很少传达高层次的操作意图，边缘计算主要用于内容分发而非决策仲裁。因此，当前的互联自动驾驶缺乏一种原则性的机制，能够在全球范围内做出一致且基于意图的协调决策。为弥补这一空白，我们提出了MIND-CAVs，这是一种基于意图驱动自主性的多智能协商与决策框架，专为连接自动驾驶车辆（CAVs）设计。每辆车辆将原始传感器观测数据抽象为结构化的意图表示，通过V2X链路交换，并从路边边缘服务器接收全球一致的协调计划。边缘代理结合了已学到的和基于规则的仲裁机制，在车辆间协商相互冲突的意图，而云平台则记录决策，供审计和持续再培训。我们在基于CARLA的AI环路平台中实施MIND-CAVs，并在多车道高速公路场景中评估其在涉及冲突机动和路径限制出口的场景中。实验结果显示，与孤立自主、先到先得仲裁和多智能体强化学习基线相比，机动完成时间更长，减少了不安全的接近和不必要的制动。

Reinforcement Learning for the Full Strawberry Harvesting Process: Obstacle Separation, Detachment, and Placement

完整草莓采收过程的强化学习：障碍物分离、脱离与安置

• Authors: Changyou Miao, Teng Li, Ya Xiong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.14708
• Pdf link: https://arxiv.org/pdf/2607.14708
• Abstract Severe occlusions and deformable plant structures introduce complex contact dynamics that challenge robotic strawberry harvesting. A policy-driven reinforcement learning (RL) framework with heuristic phase coordination was developed, in which obstacle separation, fruit detachment, and placement were formulated as a sequential decision-making task. A shared interaction-aware policy generated Cartesian motions across all task phases, while lightweight heuristic logic coordinated task progression and gripper events. A shared structured observation space was used to represent target, obstacle, end-effector, and task-context information. A hierarchical architecture combined the high-level policy with low-level Cartesian impedance control for compliant interaction. To support zero-shot sim-to-real transfer, feasibility-first observation alignment and domain randomization were adopted. The policy achieved success rates of 89.7% in simulation and 82.0% in real-world experiments. As the occlusion level increased from 1 to 5, the average execution time increased from 12.99 s to 21.73 s, reflecting greater interaction complexity. These results demonstrated effective transfer of interaction-aware harvesting behaviors to a structurally different robotic platform.
• 中文摘要 严重的阻塞和可变形的植物结构引入了复杂的接触动力学，挑战了机器人草莓采摘。开发了一个策略驱动强化学习（RL）框架，采用启发式阶段协调，将障碍物分离、果实脱落和放置作为顺序决策任务。共享的交互感知策略在所有任务阶段生成笛卡尔运动，而轻量级启发式逻辑则协调任务进展和抓取事件。使用共享结构化观察空间表示目标、障碍物、终端执行器和任务上下文信息。分层架构将高层策略与低层笛卡尔阻抗控制结合，实现合规交互。为支持零样本模拟到实物传输，采用了以可行性为先的观测比对和领域随机化。该政策在模拟中取得了89.7%的成功率，在真实世界实验中达到82.0%。随着遮挡等级从1提升到5，平均执行时间从12.99秒增加到21.73秒，反映出交互复杂度的增加。这些结果证明了交互感知采集行为能够有效地转移到结构不同的机器人平台上。

WorkDrive: Roadwork Chain of Causation for Autonomous Driving

WorkDrive：自动驾驶道路因果链

• Authors: Tianyi Jiang, Wen Zhang, Sihan Yang, Ming Lu, Wentao Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.14727
• Pdf link: https://arxiv.org/pdf/2607.14727
• Abstract Autonomous driving vision-language models (VLMs) struggle in roadwork zones, where familiar visual cues such as lane markings and permanent signs are altered or absent, and temporary devices such as cones and barriers redefine the drivable corridor. VLMs can detect these objects, but without explicit guidance they anchor their reasoning on familiar elements from pre-training and fail to connect work-zone observations to correct planning decisions. We propose WorkDrive, a framework that constructs perception-grounded causal reasoning for work zones and aligns it with trajectory prediction. An automated multitask perception pipeline extracts structured scene facts and injects them into a Chain-of-Causation (CoC) annotation pipeline, redirecting the annotator's attention to domain-specific elements. The resulting reasoning labels are used for supervised fine-tuning, followed by reinforcement learning with a single reward: consistency between lateral meta-actions and the predicted trajectory. On ROADWork, the largest public work-zone dataset, the proposed roadwork CoC reduces trajectory average displacement error (ADE) by 9.0\%, and consistency-based GRPO yields a further 3.0\%, achieving progressive improvement over the trajectory-only baseline. Code and data will be publicly released.
• 中文摘要 自动驾驶视觉语言模型（VLM）在道路施工区表现不佳，熟悉的视觉线索如车道标和永久标志被改变或缺失，临时装置如锥桶和护栏重新定义了可行驶走廊。VLM可以检测这些物体，但如果没有明确指导，它们的推理基于预训练中熟悉的元素，未能将工作区观察与正确的规划决策联系起来。我们提出了WorkDrive框架，该框架构建基于感知的因果推理工作区，并将其与轨迹预测对齐。自动化多任务感知流水线提取结构化场景事实，并将其注入因果链（CoC）注释流水线，引导标注者的注意力转向领域特定元素。由此产生的推理标签用于监督微调，随后进行强化学习，奖励为横向元动作与预测轨迹之间的一致性。在最大的公共工作区数据集ROADWork上，拟议的道路工程CoC可将轨迹平均位移误差（ADE）降低9.0%，基于一致性的GRPO进一步降低3.0%，实现了相较仅轨迹基线的逐步改善。代码和数据将公开发布。

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

SEED：针对代理强化学习的自我演进策略提炼

• Authors: Jinyang Wu, Shuo Yang, Zhengxi Lu, Fan Zhang, Yuhao Shen, Lang Feng, Haoran Luo, Zheng Lian, Shuai Zhang, Zhengqi Wen, Jianhua Tao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.14777
• Pdf link: https://arxiv.org/pdf/2607.14777
• Abstract Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at this https URL.
• 中文摘要 大型语言模型越来越多地被训练为涉及多回合交互、工具使用和环境反馈的长期任务的交互代理。基于结果的强化学习（RL）提供了实用的优化范式，但其稀疏的轨迹级奖励对中间决策的指导有限，导致剧集级结果与代币级策略学习之间存在监督空白。我们提出了SEED（SElf-Evolving On-Policy Distillation），这是一个自我演化的框架，将已完成的政策发展轨迹转化为训练时间的事后诸葛亮技能，并将其行为效应提炼回政策模型。SEED首先微调策略，分析已完成的轨迹，生成捕捉可重用工作流程、决定性观察或失败避免规则的自然语言技能。在强化学习期间，现行政策既收集轨迹，也作为分析工具，从中提取事后诸葛亮的技能。因此，政策更新能够同时提升后续决策和技能分析，使事后诸葛亮的监督能够随着政策的推进而演进。SEED随后在普通和技能增强上下文下重新评分采样动作，将技能诱导的概率偏移转换为密集的代币级策略提纯信号。该信号与基于结果的强化学习联合优化，保持辅助监督与当前轨迹分布对齐。基于文本和视觉的智能体任务的广泛实验表明，SEED 持续提升性能和样本效率，并展现出对未见场景的强有力泛化能力。我们的代码可在此 https URL 访问。

PAC Learning in Turn-Based Stochastic Games with Reachability Objectives: A Decentralized Private Approach via Expected Conditional Distance

具有可达性目标的回合制随机游戏中的PAC学习：通过期望条件距离的去中心化私有方法

• Authors: Ali Asadi, Krishnendu Chatterjee, Pavol Kebis
• Subjects: Subjects: Machine Learning (cs.LG); Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2607.14877
• Pdf link: https://arxiv.org/pdf/2607.14877
• Abstract Reachability is the most fundamental logical objective, yet it is notoriously difficult to learn in reinforcement learning settings: even for Markov decision processes, PAC learning of reachability is impossible without additional assumptions. This difficulty also holds in turn-based stochastic games (TBSGs), where two adversarial players interact on a finite state space. In this work, we consider turn-based stochastic games with reachability objectives. For such settings, adversarial learning, in which players are adversarial even in the learning phase, is impossible. Therefore, the goal is to consider learning, in which both players learn the unknown model together. In this spirit, previous literature on PAC learning in TBSGs considers (a)~public information shared by both players; and (b)~centralized learning, which means that players share the same learning algorithm. In this work, our contribution is two-fold. First, we relax these strong assumptions and ensure learning: (i)~with private information not shared with the other player; and (ii)~decentralized learning where the players do not share the same learning algorithm. To the best of our knowledge, this work is the first positive result for decentralized and private information learning of TBSGs with reachability objectives. Second, we introduce a game-theoretic generalization of the Expected Conditional Distance (ECD) parameter, which measures the expected length of reaching the target set. We establish a polynomial-sample complexity bound with respect to the number of states, actions, ECD parameter, and inverses of error tolerance and failure probability.
• 中文摘要 可达性是最基本的逻辑目标，但在强化学习环境中学习它极为困难：即使是马尔可夫决策过程，PAC学习可达性也不可能，除非有额外的假设。这种难度同样适用于回合制随机博弈（TBSG），其中两个对立玩家在有限状态空间上互动。在本研究中，我们考虑具有可达性目标的回合制随机博弈。在这种环境中，对抗式学习是不可能的，即玩家即使在学习阶段也处于对抗状态。因此，目标是考虑学习，即双方玩家共同学习未知模型。本着这种精神，以往关于TBSG中PAC学习的文献认为（a）~双方共享的公共信息;以及（b）~集中式学习，意味着玩家共享相同的学习算法。在这项工作中，我们的贡献有两个方面。首先，我们放宽这些强假设，确保学习：（i）~不与对方共享的私人信息;以及（ii）~分散式学习，参与者不共享相同的学习算法。据我们所知，这项工作是针对具有可达性目标的去中心化和私密信息学习的首个积极成果。其次，我们引入了期望条件距离（ECD）参数的博弈论推广，该参数衡量到达目标集的期望长度。我们建立了一个多项式样本复杂度，该复杂度与状态数、动作数、ECD参数以及容错容忍度和失败概率的倒数相关联。

Reachability-Aware Pretraining for Efficient Target-Oriented Path Exploration in Temporal Knowledge Graph Reasoning

在时间知识图推理中高效目标导向路径探索的可达性感知预训练

• Authors: Chien-Liang Liu, Tsao-Lun Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.14886
• Pdf link: https://arxiv.org/pdf/2607.14886
• Abstract Temporal Knowledge Graph (TKG) reasoning under the extrapolation setting focuses on forecasting future time-stamped events (facts) from historical data in a temporal knowledge graph. Existing approaches, reinforcement learning (RL)-based multi-hop reasoning methods are prominent for TKG reasoning because they produce human-interpretable predictions via explicit multi-hop path tracing. However, during RL training, rewards are typically sparse, and exploration is highly inefficient due to the vast, time-evolving action space. These issues hinder efficient training and often limit overall performance. To address these challenges, we propose RAPTOR (Reachability-Aware Pretraining for Efficient Target-Oriented Path Exploration), a self-supervised pretraining method that injects a reachability-aware inductive bias to the agent. By learning to estimate the reachability of candidate actions to the target entity, RAPTOR reduces exploration over unpromising paths and provides a strong initialization for downstream RL fine-tuning. Experimental results on the ICEWS14, ICEWS05-15, and ICEWS18 datasets demonstrate that RAPTOR pretraining markedly improves the training efficiency and consistently outperforms conventional baselines, establishing it as an effective approach for enhancing RL-based multi-hop reasoning methods for TKG reasoning.
• 中文摘要 时间知识图谱（TKG）推理在外推设置下，侧重于从时间知识图谱中的历史数据预测未来时间戳事件（事实）。现有方法中，基于强化学习（RL）的多跳推理方法在TKG推理中非常突出，因为它们通过显式多跳路径追踪产生了人类可解释的预测。然而，在强化学习训练中，奖励通常稀少，探索效率极低，因为动作空间庞大且不断变化。这些问题阻碍了训练的效率，常常限制整体表现。为应对这些挑战，我们提出了RAPTOR（高效目标导向路径探索的可达性感知预训练），这是一种自我监督的预训练方法，向智能体注入可达性感知的归纳偏置。通过学习估计候选动作对目标实体的可达性，RAPTOR 减少了对不确定路径的探索，并为下游强化学习微调提供了强有力的初始化。ICEWS14、ICEWS05-15和ICEWS18数据集的实验结果表明，RAPTOR预训练显著提升训练效率，并持续优于传统基线，确立其作为增强强化学习多跳推理方法用于TKG推理的有效方法。

Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation

利用指令调优和合并进行推理模型适配

• Authors: Yu-Du Feng, Niels Mündler-Sasahara, Mark Vero, Martin Vechev
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.14895
• Pdf link: https://arxiv.org/pdf/2607.14895
• Abstract Reasoning language models (RLMs) have demonstrated impressive performance in domains such as mathematics and coding. These domains permit reliable verification of model outputs, which is important for enabling the reinforcement learning that drives RLM performance gains. However, training RLMs on domains that lack reliable verifiers remains challenging. Meanwhile, for both verifiable and unverifiable domains, large amounts of unused supervised fine-tuning data with human-written solutions exist. In this work, we show that these data can be used efficiently to further improve RLM performance. For this, we first use classic instruction tuning, supervised fine-tuning without reasoning traces, on the RLM. Next, we merge our instruction-tuned model with the original reasoning model, recovering its reasoning behavior on the target domain. Our extensive evaluation demonstrates that our technique improves RLM performance in both verifiable and hard-to-verify domains, including coding and text summarization, while preserving RLM capabilities across other domains. Importantly, our method is highly cost-effective, enabling such improvements for less than USD $3.
• 中文摘要 推理语言模型（RLMs）在数学和编程等领域表现出了令人印象深刻的表现。这些领域允许对模型输出的可靠验证，这对于推动RLM性能提升的强化学习至关重要。然而，在缺乏可靠验证者的域名上训练RLM仍然具有挑战性。与此同时，无论是可验证还是不可验证领域，都存在大量未使用的监督微调数据，采用人工编写的解决方案。本研究表明，这些数据可以高效利用，进一步提升RLM的性能。为此，我们首先在RLM上使用经典指令调优，即无推理痕迹的监督微调。接下来，我们将指令调优模型与原始推理模型合并，恢复其在目标域上的推理行为。我们的广泛评估表明，我们的技术在可验证和难以验证的领域（包括编码和文本摘要）中提升了RLM的性能，同时保留了其他领域的RLM能力。重要的是，我们的方法成本效益高，能够以不到3美元的价格实现这些改进。

Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation

多轴Max@K强化学习：文本到图像生成中的代表性多样性

• Authors: Ku Onoda, Paavo Parmas, Hiroki Furuta, Soichiro Nishimori, Yuta Oshima, Shohei Taniguchi, Yutaka Matsuo
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.14962
• Pdf link: https://arxiv.org/pdf/2607.14962
• Abstract Text-to-image (T2I) models can synthesize realistic, prompt-aligned images, yet samples generated for the same prompt often cover only a small subset of visually distinct modes. This limits the diversity of images, and for person-centric prompts, can reflect or amplify demographic skew. We formalize this problem as coverage of a predefined set of semantically specified modes, which we call target-mode coverage. We then propose multi-axis max@K, a group-based reinforcement learning objective for improving such coverage in diffusion-based T2I models. Given a group of samples and one score per target category, multi-axis max@K first takes the maximum score across samples for each category and then sums these category-wise maxima. The resulting credit assignment gives a sample positive weight on a category only when it increases that category's group-wise maximum, allowing different samples to contribute to different categories. We first validate the credit-assignment mechanism on a synthetic mixture and on SD3.5-M using deterministic pixel-based color rewards. We then evaluate the same objective on perceived-appearance fairness. Across three automatic evaluators on held-out prompts, multi-axis max@K improves the Fairness Score by 0.23-0.36 relative to the base model, while maintaining image quality and text alignment.
• 中文摘要 文本转图像（T2I）模型可以合成真实且符合提示的图像，但为同一提示生成的样本通常只涵盖视觉上明显不同的模式的一小部分。这限制了图片的多样性，对于以人为中心的提示，可能会反映或放大人口统计的偏向。我们将此问题形式化为对一组预定义语义指定模式的覆盖，称为目标模式覆盖。随后，我们提出了多轴max@K，这是一种基于群体的强化学习目标，旨在改善基于扩散的T2I模型中的覆盖率。给定一组样本和每个目标类别的得分，多轴max@K首先取每个类别样本的最大分数，然后对这些类别的最大值求和。所得的信用分配只有在增加该类别的组别最大值时，才会对该类别赋予正权重，从而允许不同样本贡献于不同类别。我们首先在合成混合物和基于确定性像素的颜色奖励中验证了该积分分配机制，适用于SD3.5-M。然后我们对同样的目标进行外观公平性感知评估。在三个自动评估器对待提示时，多轴max@K相对于基础模型，公平性评分提升了0.23-0.36，同时保持图像质量和文本对齐。

SMC-ES: Automated synthesis of formally verified control policies

SMC-ES：正式验证控制策略的自动综合

• Authors: Riccardo Curcio, Toni Mancini, Enrico Tronci
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2607.15003
• Pdf link: https://arxiv.org/pdf/2607.15003
• Abstract The deployment of autonomous cyber-physical systems in safety-critical environments requires closed-loop control strategies (i.e., policies) that are not only performant but also provably safe and robust. While learning-based methodologies such as Reinforcement Learning offer flexible and scalable approaches to automatically synthesize such controllers, they typically lack the formal guarantees necessary for safe deployment. To bridge this gap, we propose a novel simulation-based methodology to automatically synthesize policies with formal guarantees regarding performance, safety, and robustness specifications. Specifically, given a set of properties to verify, a confidence parameter $\delta$ and an allowable failure probability $\varepsilon$, our method guarantees that the synthesized policy comes with a certificate: with confidence at least $1 - \delta$, the probability of encountering a scenario where the given properties are violated is at most $\varepsilon$. We demonstrate the feasibility of our approach by developing SMC-ES, an algorithm that integrates Evolutionary Strategies with Statistical Model Checking-based verification. We evaluate SMC-ES on a suite of continuous control tasks using Gymnasium and Safety Gymnasium testbeds. Results show that, at the price of a sustainable increase in computational cost, our algorithm provides formal guarantees regarding performance, safety, and robustness specifications, while performing competitively against leading model-free Deep Reinforcement Learning (DRL) and Safe-DRL baselines.
• 中文摘要 在安全关键环境中部署自主网络物理系统需要闭环控制策略（即策略），这些策略不仅性能优异，还需可验证安全且稳健。虽然基于学习的方法论如强化学习提供了灵活且可扩展的方法来自动合成此类控制器，但它们通常缺乏安全部署所需的正式保证。为弥合这一差距，我们提出了一种基于仿真的新方法论，能够自动综合具有性能、安全性和鲁棒性规范形式保证的政策。具体来说，给定一组需要验证的属性，置信度参数为$\delta$，允许的失败概率为$\varepsilon$，我们的方法保证综合策略附带证书：置信度至少为$1 - \delta$时，遇到违反给定属性场景的概率最多为$\varepsilon$。我们通过开发SMC-ES算法，展示了该方法的可行性，该算法将进化策略与基于统计模型检查的验证相结合。我们利用体育馆和安全体育馆测试平台，在一系列持续对照任务中评估SMC-ES的效果。结果表明，尽管计算成本持续上升，我们的算法在性能、安全性和鲁棒性规格上提供了形式保证，同时在领先无模型深度强化学习（DRL）和安全DRL基线中表现具有竞争力。

CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking

CosFly-VLA：一种空间感知的视觉-语言-行动模型用于无人机跟踪

• Authors: Ruilong Ren, Songsheng Cheng, Yunpeng Zhou, Hanxuan Chen, Xiangyue Wang, Tianle Zeng, Shuai Yuan, Binbo Li, Hanzhong Guo, Ji Pei, Da Zhang, Kangli Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.15004
• Pdf link: https://arxiv.org/pdf/2607.15004
• Abstract Dynamic target tracking is essential for Unmanned Aerial Vehicles (UAVs) operating in complex urban environments, where both the target and the camera viewpoint change continuously. Existing Vision-Language-Action (VLA) policies can track visible targets effectively, but their performance often degrades when buildings, vegetation, or roadside objects block the line of sight. During sustained occlusion, a policy may lose the target state, execute actions toward an incorrect region, and amplify this error through subsequent observations until re-acquisition becomes impossible. To this end, we present CosFly-VLA, a spatially aware VLA model that jointly grounds the target, estimates its visibility, and generates continuous flight actions through a structured prediction interface. To train this policy, we use a large-scale recipe over diverse data sources. Spatially Grounded Continued Pretraining (CPT) on a 500k mixed pool injects UAV-view depth, distance, and 3-D spatial reasoning. A three-stage Curriculum-based Supervised Fine-Tuning (SFT) process then specializes the tracker through multi-head warm-up followed by two-stage curriculum learning over natural and hard / long-occlusion data. Chain-of-Thought (CoT) training subsequently teaches recovery-oriented reasoning traces before structured answers. Finally, a closed-loop Reinforcement Learning (RL) stage optimizes tracking behavior with a multi-component reward covering stand-off tracking, grounding quality, collision avoidance, and task success. Relative to OpenVLA, CosFly-VLA-0.8B reduces open-loop Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test. Closed-loop optimization improves Success Rate (SR) by 29.8% and 2.5%, respectively. These results demonstrate progress from visible-frame imitation toward spatially grounded action-closed-loop control, evaluated under a shared oracle state history.
• 中文摘要 动态目标跟踪对于在复杂城市环境中运行的无人机（UAV）至关重要，因为目标和摄像机视角都会不断变化。现有的视觉-语言-行动（VLA）政策可以有效追踪可见目标，但当建筑物、植被或路边物体阻挡视线时，目标的性能常常会下降。在持续的遮挡期间，策略可能会丢失目标状态，执行指向错误区域的操作，并通过后续观测放大该误差，直到无法重新获取。为此，我们介绍了CosFly-VLA，一种空间感知VLA模型，能够联合实现目标的接地、其可见度估计，并通过结构化预测界面生成连续飞行动作。为了培训这项政策，我们采用了大规模的方案，覆盖了多种数据源。在500k混合池上进行空间接地持续预训练（CPT）注入无人机视角的深度、距离和三维空间推理。随后通过三阶段的基于课程的监督微调（SFT）过程，将追踪器专门化为多头热身，随后是对自然数据和硬性/长遮蔽数据进行两阶段课程学习。思维链（Chain-of-Thought，CoT）培训随后教授以恢复为导向的推理痕迹，再教授结构化答案。最后，闭环强化学习（RL）阶段通过多元奖励优化追踪行为，涵盖距离跟踪、接地质量、碰撞避免和任务成功。相较于OpenVLA，CosFly-VLA-0.8B在可见测试中开环平均位移误差降低34.1%，未观察测试降低35.3%。闭环优化分别提升了29.8%和2.5%的成功率（SR）。这些结果展示了从可见帧模仿向空间基准动作闭环控制的进展，并在共享预言机状态历史下进行评估。

Evaluating covariate balance for long time horizon Markov decision processes

评估长期马尔可夫决策过程的协变量平衡

• Authors: Joshua Spear, Rebecca Pope, Neil J Sebire
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15080
• Pdf link: https://arxiv.org/pdf/2607.15080
• Abstract This article explores the application of covariate balance diagnostics for detecting the presence of hidden confounding/model miss-specification in studies applying offline reinforcement learning (RL) to deriving optimal treatment recommendations. The results demonstrate that, either there is a high risk of bias within existing offline RL studies for treatment recommendations or, existing covariate balance metrics are not sufficient to assess such studies. Regardless, existing offline RL studies cannot be concluded as being statistically robust. The conclusions propose future research directions for obtaining more methodologically robust applications of offline RL to treatment recommendation problems.
• 中文摘要 本文探讨了协变量平衡诊断在离线强化学习（RL）应用中检测隐藏混杂因素/模型误指定存在的应用，以推导最优治疗建议。结果表明，现有的离线强化学习研究中治疗建议存在较高偏倚风险，或者现有协变量平衡指标不足以评估此类研究。无论如何，现有的离线强化学习研究尚不能得出统计学上的稳健性结论。结论提出了未来研究方向，以实现离线强化学习在治疗建议问题中更方法学稳健的应用。

Digital Pantheon: Simulating and Auditing Coalition Formation with LLM Agents

数字神祇：用大型语言模型代理模拟与审计联盟形成

• Authors: Dylan Van Mulders, Matthias Bogaert, Dirk Van den Poel
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.15095
• Pdf link: https://arxiv.org/pdf/2607.15095
• Abstract The formation of political coalitions is a complex negotiation driven by both concrete policy objectives and deep-seated ideological convictions. While Large Language Models (LLMs) open new avenues for computational political science, the neutrality and helpfulness biases instilled by Reinforcement Learning from Human Feedback (RLHF) prevent them from sustaining steadfast partisan behaviour. We present a multi-agent framework that reconciles factual grounding with ideological alignment by combining Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Retrieval-Augmented Generation (RAG): DPO instils aggressive party-specific personas, while a per-party RAG pipeline keeps each agent bounded to its official manifesto. We operationalize the framework on the 2019 Flemish election, deploying the partisan agents in a hub-and-spoke negotiation arbitrated by a formateur. To make the emergent negotiation interpretable, we introduce a Multi-Layered Information Lineage Topology (MILT) that traces every clause in the final agreement back to its manifesto origin and classifies it into five provenance states, a Coalition Influence Score (CIS) that aggregates these traceable contributions to identify which party shaped the agreement, and a real-world grounding pass that benchmarks each simulated provision against the historically adopted coalition agreement. Across three independent simulations the framework yields a stable winner and ranking (N-VA ahead of CD\&V and Open Vld), and manifesto-anchored lineage reliably predicts real-world materialization whereas hallucinated content does not. The result is a transparent, scalable testbed for the ex-ante exploration of party compatibility and formateur-mediated compromise.
• 中文摘要 政治联盟的形成是一场复杂的谈判，既有具体的政策目标，也有根深蒂固的意识形态信念。虽然大型语言模型（LLMs）为计算政治学开辟了新途径，但由人类反馈强化学习（RLHF）植入的中立性和助益性偏差，使其无法维持坚定的党派行为。我们提出了一个多代理框架，通过结合监督微调（SFT）、直接偏好优化（DPO）和检索增强生成（RAG）来调和事实基础与意识形态一致：DPO植入了具有攻击性的党派特定形象，而每党派的RAG流程则使每个代理都受限于其官方宣言。我们将该框架付诸实践，参与2019年弗拉芒选举，派遣党派代理人参与由组阁人仲裁的枢纽辐射谈判。为了使涌现的谈判变得可理解，我们引入了多层信息谱系拓扑（MILT），将最终协议中的每条条款追溯到其宣言起源，并将其分类为五个来源状态;一个联盟影响力评分（CIS），汇总这些可追溯贡献以识别哪个方塑造了协议;还有一个真实世界的基础测试，将每个模拟条款与历史上采纳的联盟协议进行对比。在三个独立模拟中，该框架给出稳定的胜者和排名（N-VA领先CD\&V和Open Vld），且以宣言为锚定的谱系可靠地预测现实世界的物质化，而幻觉内容则不能。结果是一个透明且可扩展的试验场，用于事先探索党派兼容性和组建者调解的妥协。

Learning in Infinitesimal Non-Compositional Sketches

在无穷小非构图草图中的学习

• Authors: Sridhar Mahadevan
• Subjects: Subjects: Machine Learning (cs.LG); Category Theory (math.CT)
• Arxiv link: https://arxiv.org/abs/2607.15107
• Pdf link: https://arxiv.org/pdf/2607.15107
• Abstract This paper develops a categorical framework -- Learning in Infinitesimal Non-Compositional Sketches (LINCS) -- as the repair of non-compositionality: failures of diagrams to factor through quotient sketches lifted to the tangent category setting. Machine learning problems are specified as sketches: graphs with commutativity conditions $\mathcal D$, limit cones $\mathcal L$, and colimit cocones $\mathcal K$, generalizing the usual scalarization of loss functions or vector space assumptions. Non-compositionality is defined purely as failure of a universal factorization problem, not as arithmetic error between the desired and actual predictions. Given a learning sketch $\mathbb S=(S,\mathcal D,\mathcal L,\mathcal K)$, whose underlying graph is $S$, and a model $D:J \rightarrow C$, the base defect is the obstruction to factorization $\mbox{Obs}(\mbox{Fact}{\mathbb S}(D))$. The tangent lift applies the tangent functor $T$ to obtain $TD:J \rightarrow C$, and LINCS is defined as the obstruction $\mbox{Obs}(\mbox{Fact}{\mathbb S}(TD))$ -- asking whether infinitesimal perturbations preserve the compositionality this http URL paper also introduces Tangent Learning Sketches, which are sketches equipped with Cockett-Cruttwell tangent structure. The paper defines the INC endofunctor, which iterates the tangent lift, producing a tower $D,TD,T^2D, \cdots$ of factorization problems. ML is thereby formulated as the search for a coalgebraic fixed point where successive tangent unfoldings stabilize ($\nu T_{\mbox{INC}}$). Using the Aczel--Mendler theorem, we prove existence of a final INC coalgebra whenever $T_{\mbox{INC}}$ admits a set-based class realization that creates its final carrier. A detailed experimental evaluation of LINCS is underway in a number of concrete ML settings, including deep learning, large language models, and reinforcement learning, and is described in companion papers.
• 中文摘要 本文发展了一个范畴框架——《在无穷小非合成草图中学习》（LINCS）——作为非组合性的修复：图示未能通过商草图分解到切线范畴的失败。机器学习问题被指定为草图：具有交换性条件的图 $\mathcal D$、极限锥 $\mathcal L$ 和余极限定 cocon $\mathcal K$，推广了通常的损失函数标量化或向量空间假设。非组合性纯粹定义为普遍分解问题的失败，而非期望预测与实际预测之间的算术误差。给定一个学习草图 $\mathbb S=（S，\mathcal D，\mathcal L，\mathcal K）$，其底层图为 $S$，模型为 $D：J \rightarrow C$，基础缺陷是因式分解的障碍 $\mbox{Obs}（\mbox{Fact}{\mathbb S}（D））$。切线提升应用切函子 $T$ 以获得 $TD：J \rightarrow C$，LINCS 定义为障碍 $\mbox{Obs}（\mbox{Fact}{\mathbb S}（TD））$——探讨无穷小扰动是否保持组合性。http URL 论文还介绍了切线学习草图，即配备 Cockett-Cruttwell 切线结构的草图。论文定义了INC内函子，它迭代切升，产生塔级分解问题的塔$D，TD，T^2D，\cdots$。因此，机器学习被表述为寻找一个连续切展开稳定的余代数不动点（$\nu T_{\mbox{INC}}$）。利用阿泽尔-门德勒定理，我们证明当 $T_{\mbox{INC}}$ 存在基于集合的类实现并生成其最终载体时，就存在一个最终 INC 余代数。LINCS的详细实验评估正在多个具体机器学习环境中进行，包括深度学习、大型语言模型和强化学习，相关论文中有详细描述。

Concept-Guided Spatial Regularization for World Models in Atari Pong

Atari Pong 中世界模型的概念引导空间正则化

• Authors: Yukuan Lu, Zaishuo Xia, Weyl Lu, Yubei Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15142
• Pdf link: https://arxiv.org/pdf/2607.15142
• Abstract World models are usually evaluated as components of model-based reinforcement learning (MBRL) systems, while the world models themselves are rarely studied in isolation. We examine five representative visual world-model agents in Atari Pong: DreamerV3, DIAMOND, TWISTER, Simulus, and STORM. After reproducing their training pipelines and matching the reported agent performance, we freeze the learned world models and evaluate them with a closed-loop rollout diagnostic: a policy trained separately from the corresponding MBRL agent interacts with each frozen model, and the generated video trajectories are inspected for visual and dynamical errors. Across all five models, the rollouts contain clear failures, including ball disappearance, incorrect ball motion, and invalid ball-paddle interactions. Beyond visual trajectories, we further evaluate them with pixel-space zero-shot MBRL, where a new policy is trained entirely inside a frozen world model and then evaluated in the real environment. Across all five models, the resulting policies substantially underperform those produced by the corresponding original MBRL training pipelines. The gap is particularly large for DreamerV3, whose mean return drops from -5.5 to -20.9, near the minimum Pong return of -21. We hypothesize that insufficient modeling of task-critical concepts, such as the ball in Pong, may contribute to these failures. We therefore propose Concept-Guided Spatial Regularization (CGSReg), an auxiliary pixel reconstruction loss applied to segmented concept regions. Experiments show that CGSReg improves both closed-loop rollouts and pixel-space zero-shot MBRL in DreamerV3, DIAMOND, and TWISTER. Its effects vary across the remaining models and evaluation metrics, indicating that CGSReg alone does not address all world-model bottlenecks.
• 中文摘要 世界模型通常作为基于模型的强化学习（MBRL）系统的组成部分进行评估，而世界模型本身很少被单独研究。我们考察了Atari Pong中的五个代表性视觉世界模型代理：DreamerV3、DIAMOND、TWISTER、Simulus和STORM。在重现其训练流水线并匹配报告的智能体表现后，我们冻结已学习的世界模型，并通过闭环推广诊断进行评估：一个与对应MBRL智能体单独训练的策略与每个冻结模型交互，并检查生成的视频轨迹是否有视觉和动态错误。在所有五种模型中，滚动都存在明显的故障，包括球消失、球运动错误以及球与球拍的交互无效。除了视觉轨迹外，我们还用像素空间零射点MBRL进一步评估它们，该技术完全在冻结世界模型中训练新策略，然后在真实环境中进行评估。在所有五种模型中，最终的策略表现远低于对应的原始MBRL培训流程。DreamerV3的差距尤为显著，其平均回报率从-5.5降至-20.9，接近Pong的最低回报-21。我们假设，任务关键概念（如乒乓球中的球）建模不足可能导致这些失败。因此，我们提出了概念引导空间正则化（CGSReg），这是一种应用于分割概念区域的辅助像素重建损失。实验显示，CGSReg 在 DreamerV3、DIAMOND 和 TWISTER 中提升了闭环滚动和像素空间零射击 MBRL。其效果在其他模型和评估指标中有所不同，表明仅靠CGSReg无法解决所有世界模型瓶颈。

On-Policy Delta Distillation

政策上的三角洲蒸馏

• Authors: Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.15161
• Pdf link: https://arxiv.org/pdf/2607.15161
• Abstract On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD$^2$). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD$^2$ consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at this https URL
• 中文摘要 策略提炼是一种强化学习中的替代训练后方法，通过提供教师模型的代币级监督，缓解了奖励模型带来的限制。尽管政策内提炼已被研究并应用于各种场景，但其基本设计仍未被充分探索。本文引入了一种新的蒸馏奖励，称为delta信号，而非直接模仿教师的输出分布。差异信号定义为教师模型与其基础模型在推理能力调整前的差值。因此，它捕捉了推理调谐引发的变化，并为推理能力的传递提供了更直接的信号。通过大量实证证据，我们表明δ信号显著改善了策略上蒸馏，并将新的蒸馏方法称为策略上蒸馏（OPD$^2$）。数学、科学和代码推理基准的实验表明，OPD$^2$ 持续优于传统的策略提取，使推理型大型语言模型在短训练后期内就能实现强劲表现。代码将在此 https URL 上提供

Mask-Aware Policy Gradients for Diffusion Language Models

扩散语言模型的掩膜感知策略梯度

• Authors: Haran Raajesh, Kulin Shah, Adam Klivans, Philipp Krähenbühl
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15200
• Pdf link: https://arxiv.org/pdf/2607.15200
• Abstract Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation. Existing approaches approximate this log-likelihood by modeling only the token predictions, ignoring the order in which positions are unmasked during generation. We observe that MDLM generation involves two decisions at each step: what tokens to place at each masked position and which positions to remask. We formalize this as a two-stage action MDP, showing that the policy gradient naturally decomposes into a token term and a masking term. Combining optimization of both terms leads to state-of-the-art outcomes on mathematical reasoning and coding benchmarks, with scores of 87.1% on GSM8K and 53.4% on MBPP.
• 中文摘要 强化学习已被证明在提升大型语言模型的推理能力方面有效，但由于对数似然估计的难度，将其推广到蒙面扩散语言模型（MDLMs）仍然具有挑战性。现有方法仅通过建模代币预测来近似这种对数似然，忽略生成过程中位置被揭露的顺序。我们观察到，MDLM生成在每一步涉及两个决策：每个掩码位置放置哪些令牌，以及重新掩码哪些位置。我们将此形式化为两阶段动作MDP，表明策略梯度自然分解为一个标记项和一个掩蔽项。结合这两个术语的优化，在数学推理和编码基准测试上取得了最先进的成果，GSM8K得分为87.1%，MBPP为53.4%。

MeanFlowNFT: Bringing Forward-Process RL to Average-Velocity Generators

MeanFlowNFT：将过程强化学习推向平均速度生成器

• Authors: Yushi Huang, Xiangxin Zhou, Jun Zhang, Liefeng Bo, Tianyu Pang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15273
• Pdf link: https://arxiv.org/pdf/2607.15273
• Abstract MeanFlow generators achieve fast few-step sampling by predicting average velocities over time intervals, making them attractive for efficient generation. Reinforcement learning (RL) has become a powerful way to align diffusion and flow models with human preferences and task-specific objectives. In particular, DiffusionNFT offers an efficient forward-process RL framework that does not require reverse-process trajectories or likelihood estimation. However, applying such RL methods to MeanFlow remains underexplored. DiffusionNFT optimizes instantaneous velocities, whereas MeanFlow samples with average velocities. To bridge this gap, we introduce MeanFlowNFT. Inspired by the MeanFlow identity, which bridges average and instantaneous velocities, we construct an induced instantaneous-velocity predictor. We apply the DiffusionNFT objective to this predictor, making reward optimization well-defined for MeanFlow. Sampling remains based on the average velocity, preserving MeanFlow's fast few-step generation. We further prove that MeanFlowNFT inherits DiffusionNFT's strict policy-improvement guarantee. Experiments on image and video generation show that MeanFlowNFT consistently improves baselines. Moreover, it outperforms prior state-of-the-art RL-tuned few-step generators on most metrics ($6$ of $8$ on SD3.5-M), and can even surpass multi-step RL-tuned diffusion while using only a few sampling steps. For instance, on Wan 2.1, $4$-step MeanFlowNFT reaches a VBench score of $84.33$, surpassing $50$-step LongCat-Video RL ($82.57$).
• 中文摘要 平均流量发生器通过预测不同时间区间的平均速度实现快速的少步采样，使其具有高效的生成潜力。强化学习（RL）已成为将扩散和流动模型与人类偏好和任务特定目标对齐的强大方式。特别是，DiffusionNFT提供了一个高效的前向过程强化学习框架，无需逆向过程轨迹或似然估计。然而，将此类强化学习方法应用于平均流仍然缺乏探索。DiffusionNFT 优化瞬时速度，而 MeanFlow 采样的是平均速度。为了弥合这一差距，我们推出了MeanFlowNFT。受平均流恒定式启发，该恒等式连接了平均速度和瞬时速度，我们构建了一个诱导瞬时速度预测器。我们将 DiffusionNFT 目标应用于该预测器，使 MeanFlow 的奖励优化定义得更清晰。采样仍基于平均速度，保持了MeanFlow快速的少步生成特性。我们还进一步证明，MeanFlowNFT继承了DiffusionNFT严格的政策改进保障。图像和视频生成的实验表明，MeanFlowNFT持续提升基线水平。此外，它在大多数指标上优于之前最先进的RL调谐的少步发生器（SD3.5-M约为6美元至8美元），甚至能在仅用少量采样步的情况下超越多步RL调谐扩散。例如，在 Wan 2.1 中，步进 4 美元的 MeanFlowNFT 获得了 84.33 美元的 VBench 分数，超过了 50 美元步长猫-视频 RL（82.57 美元）。

Keyword: diffusion policy

There is no result