生成时间: 2026-07-24 18:20:42 (UTC+8); Arxiv 发布时间: 2026-07-24 20:00 EDT (2026-07-25 08:00 UTC+8)

今天共有 28 篇相关文章

Keyword: reinforcement learning

AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics

AINTMA：具备生成智能、安全云通信和自适应质量分析的自主测试管理智能人工智能架构

• Authors: Vinil Pasupuleti, Shyalendar Reddy Allala, Siva Rama Krishna Varma Bayyavarapu, Shrey Tyagi, Srinivasateja Songa
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.20452
• Pdf link: https://arxiv.org/pdf/2607.20452
• Abstract Modern software quality assurance demands intelligent, autonomous systems capable of adaptive decision-making across distributed cloud environments. This paper presents AINTMA (Agentic Intelligent Test Management Architecture), a multi-agent agentic AI system that transforms traditional test management into an autonomous quality intelligence ecosystem. AINTMA deploys six specialized AI agents (Test Discovery, Risk Assessment, Reinforcement Learning Prioritization, Execution Orchestration, Generative Quality Intelligence, and Cloud Security Monitor) coordinated through a secure multi-agent communication framework over a cloud-native microservices infrastructure. The Generative Quality Intelligence agent employs large language models to produce plain language quality narratives, defect risk summaries, and data-augmented test recommendations. The RL Prioritization agent models test selection as a Markov Decision Process, learning contextual policies from large-scale historical test execution data (47 features, rolling 36-month window). Secure cloud communication is enforced through a zero-trust API gateway with OAuth2/JWT authentication, encrypted inter-agent messaging, and multi-tenant isolation. Evaluation across 12 heterogeneous software projects over 18 months demonstrates: 88.4% test prioritization accuracy (APFD, vs. 51.2% random, 82.1% best commercial baseline); 43% test cycle time reduction; defect escape rate reduced from 8.3% to 2.1%; 340% ROI at 9-month payback. The agentic architecture scales to 50,000+ test cases with sub-400ms response time, and the generative intelligence module achieves 4.3/5.0 developer usefulness rating. AINTMA demonstrates that agentic AI, combining autonomous multi-agent coordination, generative intelligence and secure smart connectivity, can fundamentally advance software quality management in cloud-scale enterprise environments.
• 中文摘要 现代软件质量保证需要智能、自主的系统，能够在分布式云环境中进行自适应决策。本文介绍了AINTMA（代理智能测试管理架构），这是一种多代理代理AI系统，将传统测试管理转变为自主质量智能生态系统。AINTMA部署了六个专业的AI代理（测试发现、风险评估、强化学习优先级、执行编排、生成质量智能和云安全监控），通过云原生微服务基础设施的安全多代理通信框架协调。生成质量智能代理利用大型语言模型生成语言质量叙述、缺陷风险摘要和数据增强测试建议。强化优先级化代理将测试选择建模为马尔可夫决策过程，从大规模历史测试执行数据（47个特征，滚动36个月窗口）学习上下文策略。安全云通信通过零信任API网关执行，支持OAuth2/JWT认证、加密代理间消息传递和多租户隔离。18个月内对12个异构软件项目的评估显示：测试优先级准确率为88.4%（APFD，对比随机51.2%，最佳商业基线82.1%）;测试周期时间缩短43%;缺陷逃逸率从8.3%降至2.1%;9个月回本时投资回报率340%。代理架构可扩展至50,000+个测试用例，响应时间低于400毫秒，生成智能模块的开发者实用性评分为4.3/5.0。AINTMA展示了智能人工智能结合自主多智能体协调、生成智能和安全智能连接，能够从根本上推动云规模企业环境中的软件质量管理。

Reliability-Aware LLM Alignment from Inconsistent Human Feedback

基于不一致的人类反馈实现可靠性感知的LLM对齐

• Authors: Jingyi Huang, Ruohan Zong, Yujun Feng, Liran Ma, Lanyu Shang, Yang Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.20515
• Pdf link: https://arxiv.org/pdf/2607.20515
• Abstract Reinforcement Learning from Human Feedback (RLHF) is critical for aligning Large Language Models (LLMs) with human preferences. However, its efficacy is often compromised by the inherent inconsistency and subjectivity of human annotations. Existing preference optimization frameworks, such as Direct Preference Optimization (DPO), typically treat ambiguous pairs with high annotator disagreement identically to those with unanimous consensus, forcing models to overfit to inconsistent supervision signals and leading to suboptimal alignment. In this work, we propose Reliability-Guided Preference Optimization (RGPO), a robust framework designed to mitigate the impact of inconsistent human feedback. RGPO estimates annotator reliability and infers latent ground truth labels from noisy human feedback to identify robust preferences. Furthermore, we introduce a reliability-aware consistency optimization that dynamically modulates the training objective based on the consensus level of annotations, ensuring the model prioritizes high-consensus supervision signals. Extensive experiments on LLM alignment benchmarks demonstrate that RGPO effectively reduces inconsistency and noise in training data and achieves superior performance compared to widely adopted RLHF baselines. Our code and configurations are available at this https URL.
• 中文摘要 来自人类反馈的强化学习（RLHF）对于使大型语言模型（LLMs）与人类偏好对齐至关重要。然而，其有效性常常因人类注释本身的不一致性和主观性而受到影响。现有的偏好优化框架，如直接偏好优化（DPO），通常将标注者高度不一致的歧义对与一致一致的对视为一模一样，迫使模型对不一致的监督信号进行过度拟合，导致比对效果不理想。在本研究中，我们提出了可靠性引导偏好优化（RGPO）这一稳健框架，旨在减轻不一致的人为反馈影响。RGPO估计注释者的可靠性，并从噪声的人类反馈中推断潜在的真实性标签，以识别强健的偏好。此外，我们引入了基于可靠性的一致性优化，基于共识级别的注释动态调制训练目标，确保模型优先处理高共识监督信号。大量LLM比对基准测试实验表明，RGPO有效减少训练数据中的不一致和噪声，并相较广泛采用的RLHF基线实现更优的性能。我们的代码和配置可在该 https URL 访问。

When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion

当RLVR缩小推理边界：诊断反转Pass@k

• Authors: Todd Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.20543
• Pdf link: https://arxiv.org/pdf/2607.20543
• Abstract Reinforcement learning with verifiable rewards (RLVR) can improve one-sample accuracy while making a model worse under repeated sampling. We study this pass@k inversion: after training, the policy may solve fewer distinct problems than its base model at large $k$. The failure concentrates on boundary prompts, where the base model contains rare correct trajectories that are recoverable by sampling but too sparse to reliably appear in finite RLVR rollout groups. We argue that a two-mode account explains this as an absence-of-evidence failure: rare correct trajectories may disappear before RLVR samples and reinforces them often enough. The main contribution is this diagnostic and mechanistic framing. Per-Problem Base Anchoring (PBA) is a deliberately simple proof-of-concept: sharpen prompts with sufficient frozen-base correct evidence, and anchor risky prompts to the base distribution. Across three training seeds on Omni-MATH-Test, with MATH500 as a secondary high-coverage validation benchmark, PBA improves both \PassK{1} and high-budget coverage over matched GRPO. A 3000-prompt regime-controlled diagnostic study is consistent across seeds with the expected signature: ordinary GRPO loses base-solvable boundary prompts, while PBA preserves rare verifier-positive trajectories. We use mathematical verifiers as a controlled testbed for verifier-guided optimization; the same pass@k inversion risk applies to ECCV-relevant vision-language agents when repeated visual, spatial, or chart-reasoning attempts are checked by external tools or verifiers. Reasoning post-training should decide not only how strongly to optimize, but which prompts are safe to optimize.
• 中文摘要 带有可验证奖励的强化学习（RLVR）可以提高单样本准确率，同时使模型在重复抽样下变差。我们研究这种pass@k反转：训练后，该策略在整体上可能解决的独立问题比其基础模型少$k美元。失败主要集中在边界提示上，基础模型包含可通过抽样恢复但过于稀疏的正确轨迹，无法可靠地出现在有限的RLVR展开组中。我们认为，双模式叙述解释了这一点，即证据缺失：罕见的正确轨迹可能在RLVR采样前消失，且频繁被强化。主要贡献在于这种诊断和机制框架。按问题库锚定（PBA）是一种刻意简单的概念验证：锐化具有足够固定基础正确证据的提示，并将有风险的提示锚定到基础分布。在Omni-MATH-Test的三个训练种子中，MATH500作为次要高覆盖验证基准，PBA提升了\PassK{1}和高预算覆盖率，优于匹配GRPO。一项3000个序列的对照诊断研究在所有种子间与预期特征一致：普通GRPO丢失可解边界提示，而PBA则保留罕见的验证者正向轨迹。我们使用数学验证器作为验证者引导优化的受控测试平台;当外部工具或验证者检查重复的视觉、空间或图表推理尝试时，ECCV相关的视觉语言代理也存在同样的pass@k反转风险。训练后的推理不仅应决定优化的强度，还应决定哪些提示是安全的优化。

Conflict Resolution under Degraded Surveillance in Air Corridors Using Multi-Agent Reinforcement Learning

利用多智能体强化学习在降级监控下解决空中走廊的冲突

• Authors: Esrat Farhana Dulia, Syed Arbab Mohd Shihab, Caleb Adams, Ruben Del Rosario
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.20547
• Pdf link: https://arxiv.org/pdf/2607.20547
• Abstract Safe Advanced Air Mobility operations require aircraft to maintain separation when surveillance information is noisy, delayed, incomplete, or temporarily unavailable. This study develops a Deep Q-Network-based Multi-Agent Reinforcement Learning framework for decentralized conflict resolution among heterogeneous small unmanned aerial vehicles and electric vertical takeoff and landing aircraft operating within a structured three-dimensional corridor. Separate policies are trained for the two aircraft categories using local observations and a 14-action space that includes maintaining course, turning, vertical maneuvering, landing, and speed control. The simulation incorporates aircraft-specific dynamics, energy use, corridor constraints, observation noise, communication delay, information dropout, wind disturbance, actuator uncertainty, and model uncertainty. The trained policies are evaluated across 90 combinations of traffic density and minimum separation thresholds. Loss-of-separation frequency and duration generally increase with traffic density and separation requirements, although most events are resolved within 1s. Under safe conditions, agents maintain their motion approximately 79% of the time. During conflicts, turning accounts for 33% of actions, followed by maintaining motion at 29%, speed control at 25%, and vertical maneuvers at 13%. Six Pareto-optimal configurations reveal trade-offs between safety and corridor capacity. The framework supports the simulation-based evaluation of safer AAM conflict-resolution strategies under degraded surveillance conditions.
• 中文摘要 安全的先进空中机动行动要求飞机在监视信息嘈杂、延迟、不完整或暂时无法获取时保持分离。本研究开发了一个基于深度Q网络的多智能体强化学习框架，用于在结构化三维走廊内运行的异构小型无人机与电动垂直起降飞机之间的去中心化冲突解决。针对两类飞机分别训练政策，采用本地观察和14个行动空间，包括保持航向、转向、垂直机动、着陆和速度控制。该模拟包含了飞机特定的动力学、能耗、走廊约束、观测噪声、通信延迟、信息丢失、风扰动、执行器不确定性和模型不确定性。培训政策会在90种交通密度和最小隔离阈值的组合中进行评估。分隔丢失的频率和持续时间通常随着交通密度和分隔要求的增加而增加，尽管大多数事件在1秒内解决。在安全条件下，代理人大约有79%的时间保持运动。在冲突中，转向占行动的33%，其次是保持运动29%，速度控制25%，垂直机动13%。六种帕累托最优配置揭示了安全性与走廊容量之间的权衡。该框架支持在恶化监控条件下基于模拟评估更安全的AAM冲突解决策略。

Beyond SBDD: Geometric Deep Learning in Polypharmacology and Multi-target Drug Design

超越SBDD：多药理学与多靶点药物设计中的几何深度学习

• Authors: Tianming Han, Zhijie Pan, Wenchi Ge, Qi Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.20550
• Pdf link: https://arxiv.org/pdf/2607.20550
• Abstract The traditional "one drug, one target" paradigm of structure-based drug design (SBDD) frequently proves inadequate for treating multifactorial diseases such as cancer and neurodegenerative disorders, owing to compensatory signaling pathways and the emergence of drug resistance. While polypharmacology offers a synergistic therapeutic strategy, the rational design of ligands capable of simultaneously satisfying the geometric constraints imposed by multiple targets remains a major computational bottleneck. This review positions geometric deep learning (GDL) as a powerful integrative approach to overcome these limitations. We systematically survey GDL architectures ranging from invariant graph neural networks to SE(3)-equivariant diffusion models that harness non-Euclidean molecular data to capture intrinsic three-dimensional (3D) structural interdependencies. We critically analyze GDL applications across three core dimensions, including the characterization of shared binding pockets via geometric embeddings, multi-target bioactivity prediction through heterogeneous graph fusion, and de novo generation of dual-target ligands. Particular emphasis is placed on emerging structure-conditioned generative algorithms that integrate diffusion models with reinforcement learning to autonomously resolve complex geometric conflicts between competing binding sites. Furthermore, we evaluate the pivotal role of multimodal omics integration and specialized geometric benchmarking infrastructures in validating these models. By synthesizing these methodological advances, this review elucidates the paradigm shift in drug discovery from serendipitous exploration to rational, structure-driven polypharmacological molecular engineering, thereby providing a clear, structured guide for navigating the complexities of next-generation therapeutics.
• 中文摘要 传统的结构型药物设计（SBDD）“一药一靶”范式常常无法用于治疗癌症和神经退行性疾病等多因素疾病，原因在于代偿性信号通路和药物耐药性的出现。虽然多药理学提供了协同治疗策略，但能够同时满足多靶点所施加几何约束的配体合理设计仍是主要的计算瓶颈。本综述将几何深度学习（GDL）定位为克服这些局限的强大集成方法。我们系统地调查了GDL架构，涵盖从不变图神经网络到SE（3）等变扩散模型，这些模型利用非欧几里得分子数据捕捉内在的三维（3D）结构相互依赖关系。我们批判性地分析GDL在三个核心维度上的应用，包括通过几何嵌入表征共享结合口袋、通过异构图融合预测多靶生物活性，以及双靶配体的新生成。特别强调新兴的结构条件生成算法，这些算法将扩散模型与强化学习结合相结合，自主解决竞争结合位点之间的复杂几何冲突。此外，我们评估了多模态组学集成和专业几何基准测试基础设施在验证这些模型中的关键作用。通过综合这些方法学进展，本综述阐明了药物发现从偶然探索向理性、结构驱动的多药分子工程的范式转变，从而为应对下一代治疗的复杂性提供了清晰且有结构的指导。

CMI-Mem: Toward Generalizable Long-Term Memory Management via CMI-Augmented Reinforcement Learning

CMI-Mem：通过CMI增强强化学习实现可推广的长期记忆管理

• Authors: Yubo Wang, Qiuyu Zhao, Zenghui Sun, Shichao Dong, Jinsong Lan, Xiaoyong Zhu, Haoyang Li, Bo Zheng, Lei Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.20553
• Pdf link: https://arxiv.org/pdf/2607.20553
• Abstract Memory Manager models are pivotal in agent systems. Existing methods rely predominantly on LLM-judged synthetic question-answer (QA) pairs, making memory valuation dependent on sampled queries and the downstream reader. To address this limitation, we propose \textbf{CMI-Mem}, a reinforcement learning(RL)-based lightweight memory manager model with a hybrid reward that combines downstream QA correctness and intrinsic Conditional Mutual Information (CMI). CMI evaluates the information contributed by new conversational inputs relative to the current memory state without conditioning on a sampled QA query, thereby complementing rather than replacing QA grounding. Our codes are available at: this https URL , and the CMI-Mem-4B model checkpoint is available at: this https URL
• 中文摘要 内存管理器模型在代理系统中至关重要。现有方法主要依赖LLM判定的合成问答（QA）对，使得内存估值依赖于采样查询和下游读取器。为解决这一限制，我们提出了 \textbf{CMI-Mem}，一种基于强化学习（RL）的轻量级记忆管理模型，采用混合奖励，结合了下游质量保证的正确性和内在的条件互信息（CMI）。CMI评估新会话输入相对于当前内存状态所贡献的信息，而不依赖采样的QA查询，从而补充而非替代QA的基础。我们的代码可在：此 https URL 获取，CMI-Mem-4B 型号检查点可在 this https URL 获取

SalesLoop: Reinforcement Learning from Performance Feedback for Sales Lead Ranking

SalesLoop：通过绩效反馈进行强化学习，以获得销售线索排名

• Authors: Chenyu Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.20655
• Pdf link: https://arxiv.org/pdf/2607.20655
• Abstract Lead ranking in Customer Relationship Management (CRM) systems faces a persistent challenge: models achieving high offline accuracy often underperform in production. We identify three fundamental gaps responsible for this disconnect: offline-online metric mismatch, pointwise-listwise objective misalignment, and temporal distribution drift. To address these gaps, we propose SalesLoop, a reinforcement learning framework that establishes a closed feedback loop between model predictions and real-world business outcomes. Our approach introduces (1) a performance-aware reward that encodes conversion outcomes weighted by ranking position and conversion velocity, and (2) Discriminative GRPO, a listwise optimization objective that adapts Group Relative Policy Optimization to discriminative ranking models. SalesLoop improves NDCG@K by +7.9\% and P@K by +15.8\% over the strongest static baseline. A 160-day production A/B test at a New Energy Vehicle manufacturer, spanning 16.5M leads and 280 sales specialists across two provincial markets, validates statistically significant cumulative lift of +4.7\% ($p=0.047$) and +8.7\% ($p=0.002$). In production, the ranking backbone achieves Top-10\% recall of 44.1\% and surfaces high-intent leads at $2.3\times$ the conversion rate of specialist baselines.
• 中文摘要 客户关系管理（CRM）系统中的潜在客户排名面临持续挑战：实现高离线准确性的模型在生产环境中常常表现不佳。我们识别出导致这种脱节的三个根本性缺口：线下与在线指标不匹配、点列表目标错位和时间分布漂移。为弥补这些空白，我们提出了SalesLoop，一种强化学习框架，在模型预测与现实业务成果之间建立封闭反馈环。我们的方法引入了（1）一种性能感知奖励，编码了按排名和转化速度加权的转化结果，以及（2）判别性GRPO，一种列表式优化目标，将群体相对策略优化调整到判别性排名模型。SalesLoop 的NDCG@K比最强静态基线提升了+7.9%，P@K提升了+15.8%。在一家新能源汽车制造商进行的160天量产A/B测试，涵盖了两个省级市场的1650万潜在客户和280名销售专家，验证了统计学显著的累计提升，分别为+4.7\%（$p=0.047$）和+8.7\%（$p=0.002$）。在生产过程中，排名骨干实现了44.1%的前10%回忆率，并以2.3美元的价格呈现高意图线索，转化率是专业基线的2.3倍。

Adaptive Multi-Horizon Reinforcement Learning

自适应多视界强化学习

• Authors: Manoosh Samiei, Doina Precup, Paul Masset
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.20656
• Pdf link: https://arxiv.org/pdf/2607.20656
• Abstract Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement learning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.
• 中文摘要 在复杂且不断变化的环境中做出有效决策需要平衡短期和长期后果。在强化学习（RL）中，这种权衡通常通过固定的折现因子控制，该折现因子施加一个指数折现的时间视野。然而，生物制剂表现出灵活且适应性的时间折现，表明有效规划需要多个时间尺度。本文提出一种多视野方法，自适应地选择并组合时间视野，实现对奖励结构变化的稳健适应，而无需手动贴现因子调优。这种灵活性使该方法特别适用于涉及任务切换和环境配置变化的持续学习场景。通过实证，我们证明了我们的方法能够识别出多种MiniGrid环境中的有效贴现因子，包括由三个连续变化任务组成的连续设置。这些结果表明，自适应时间折现可以提升参数效率，并增强人工和生物启发学习系统的适应性。

Learning to Detect UI Principle Violations via Reinforcement Learning

通过强化学习学习检测UI原则违规

• Authors: Nishi Mehta, Swathi Alse, Himani Kumavat, Yue Yu, Pratik Jayarao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.20690
• Pdf link: https://arxiv.org/pdf/2607.20690
• Abstract Small language models and coding agents increasingly generate web front-end code, yet their outputs are typically evaluated primarily for functional correctness. A generated interface may compile, render, and pass unit tests while still violating established interface quality principles, including accessibility barriers, deceptive design patterns, poor visual hierarchy, and excessive decision complexity. Existing auditing approaches face a trade-off between cost, coverage, and scalability: expert human review provides rich judgment but is slow and expensive; frontier vision-language models offer broader reasoning capabilities but remain costly to deploy at scale; and rule-based tools such as axe-core and Lighthouse are inexpensive but primarily capture mechanically checkable accessibility issues. We investigate whether a lightweight vision-language model can serve as an effective critic for generated interfaces. We unify 19 interface-quality principles from three complementary sources of HCI knowledge: WCAG 2.2 accessibility standards, deceptive design taxonomies, and established theories of perception, cognition, and interaction. To train this critic, we construct a verified dataset of approximately 10,000 generated web pages by synthetically injecting known violations into clean, LLM-generated Tailwind pages. Continued reinforcement learning on a 4B vision-language model improves micro-F1 from 36\% to 84\%, with 13 of 19 principles exceeding 80\% F1. The resulting critic can audit generated interfaces, filter low-quality interface training data, and provide a reward signal for design-aware code generation. We release our data-generation recipe and injection/verification prompts to support reproducible evaluation and future work on scalable interface-quality assessment.
• 中文摘要 小型语言模型和编码代理越来越多地生成网页前端代码，但其输出通常主要被评估为功能正确性。生成的界面可能在编译、渲染和通过单元测试时，仍然违反既定的界面质量原则，包括无障碍障碍、欺骗性设计模式、糟糕的视觉层级结构以及过高的决策复杂性。现有审计方法面临成本、覆盖范围和可扩展性的权衡：专家人工审核提供了丰富的判断，但速度缓慢且成本高昂;前沿愿景语言模型提供了更广泛的推理能力，但大规模部署成本仍然较高;基于规则的工具如axe-core和Lighthouse价格低廉，但主要反映机械上可检查的无障碍问题。我们探讨了轻量级视觉语言模型是否能有效批判生成界面。我们统一了来自三种互补人机交互知识来源的19条界面质量原则：WCAG 2.2无障碍标准、欺骗性设计分类法以及已知的感知、认知和交互理论。为了训练这位批评者，我们通过合成地将已知违规注入干净的大型语言模型生成的Tailwind页面，构建了一个约1万个生成网页的验证数据集。在4B视觉语言模型上进行持续强化学习，微F1的表现率从36%提升至84%，其中19条原则中有13项超过80%的F1。最终的批评者可以审计生成的接口，过滤低质量的接口训练数据，并为设计感知代码生成提供奖励信号。我们发布数据生成配方和注入/验证提示，以支持可重复的评估以及未来可扩展界面质量评估的工作。

Perspective Latents as an Architectural Condition for Causal Emergence in Active Inference Agents

透视潜在作为主动推理代理因果出现的建筑条件

• Authors: Hongju Pae
• Subjects: Subjects: Machine Learning (cs.LG); Neurons and Cognition (q-bio.NC)
• Arxiv link: https://arxiv.org/abs/2607.20708
• Pdf link: https://arxiv.org/pdf/2607.20708
• Abstract A recent line of work measures causal emergence in reinforcement learning agents through Integrated Information Decomposition, reporting that $\Phi_r$ grows with training and tracks reward improvement. For active inference, this raises the question of how reward-free predictive organization relates to such information-theoretic signatures. I test this within an active inference agent whose architecture separates a fast perception latent $z$ from a slow global latent $g$, where $g$ is driven by prediction error and structurally decoupled from policy gradients. In a reward-free environmental regime-switching protocol, $\Phi_r$ concentrates in $g$; its aggregate magnitude is largely architectural and decreases with training. The substantive effect of learning becomes legible only at the atom-compositional level: decoupling flips sign from negative to positive and becomes regime-invariant under environmental change, while downward causation carries the regime-dependent adjustment. These results identify $g$ as the architectural locus of $\Phi_r$-relevant temporal organization in an active inference agent, and argue against reading scalar $\Phi_r$ as a direct index of learned integration.
• 中文摘要 一项近期研究通过综合信息分解测量强化学习代理的因果显现，报告称 $\Phi_r$ 随着训练增长并跟踪奖励改进。对于主动推断，这引发了一个问题：无奖励的预测组织如何与此类信息理论特征相关。我在一个主动推理代理中测试此过程，其架构将快速感知的潜在$z$与慢的全局潜在$g$区分开来，其中$g$由预测误差驱动，且结构上与策略梯度解耦。在无奖励的环境体制切换协议中，$\Phi_r$集中于$g$;其总体规模主要是建筑性的，随着训练而下降。学习的实质效果仅在原子组成层面显现：解耦使符号从负转为正，并在环境变化下变得体制不变，而向下因果则带有依赖体制的调整。这些结果将$g$作为主动推理代理中$\Phi_r$相关时间组织的架构性点，并反对将标量$\Phi_r$视为学习积分的直接指标。

Robostral Navigate

机器人导航

• Authors: Arjun Majumdar, Avinash Sooriyarachchi, Benjamin Tibi, Chris Bamford, Elliot Chane-Sane, Guillaume Lample, Khyathi Raghavi Chandu, Ludovic Ho Fuh, Mathieu Poiree, Olivier Duchenne, Rosalie Millner, Srijan Mishra, Theo Cachet, Thomas Chabal
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.20785
• Pdf link: https://arxiv.org/pdf/2607.20785
• Abstract Deploying navigation systems at scale requires a recipe that minimizes sensor assumptions, generalizes across robot embodiments, and trains efficiently. Yet, today's best systems depend on depth sensors, multi-camera rigs, or pre-built maps, limiting the hardware they support and increasing deployment cost. We introduce Robostral Navigate, an 8B vision-language model built around this scalability objective. The model consumes only a stream of monocular RGB images - the most ubiquitous sensor across robotic platforms and predicts waypoints by pointing to the next target location in the current camera view. Operating purely in image space, rather than robot-specific coordinates, makes the policy naturally robust to changes in camera intrinsics and scene scale, enabling deployment across wheeled, legged, and aerial robots without recalibration. We generate 2.4 million trajectories across 350k simulated scenes to reduce the reliance on real-world data collection and scale easily. We further introduce a prefix-caching training recipe that packs entire episodes into single training sequences, reducing training tokens by 22x and cutting training time from months to days. A tree-based attention mask prevents conditioning on previous ground-truth actions, encouraging visually grounded action prediction, and reinforcement learning is used to further improve exploration and recovery capabilities. On the Room-to-Room and Room-Across-Room in Continuous Environments (R2R-CE and RxR-CE) benchmarks, Robostral Navigate sets a new state of the art. On R2R-CE, it achieves a 77.4% success rate, surpassing the best monocular method by 10.5 points and the strongest depth- or multi-camera system by 5.3 points despite using only a single RGB camera. On RxR-CE, it reaches 75.1% success rate, outperforming all monocular baselines.
• 中文摘要 大规模部署导航系统需要一个最小化传感器假设、跨机器人体型推广并高效训练的方案。然而，如今最好的系统依赖于深度传感器、多摄像头设备或预设地图，限制了支持的硬件并增加了部署成本。我们介绍Robostral Navigate，这是一个围绕这一可扩展目标构建的8B视觉语言模型。该模型仅消耗一串单眼RGB图像——这是机器人平台上最普遍的传感器，并通过指向当前相机视角中的下一个目标位置来预测路径点。纯粹在图像空间中运行，而非机器人特定坐标，使该政策自然能抵抗摄像机内在和场景规模的变化，支持在轮式、腿式和空中机器人间部署，无需重新校准。我们在35万个模拟场景中生成240万条轨迹，以减少对真实数据收集的依赖并轻松扩展。我们还引入了一种前缀缓存训练配方，将整集压缩在单一训练序列中，将训练代币减少22倍，训练时间从数月缩短至数天。基于树的注意力掩体防止对之前地面真实行为的条件化，鼓励视觉化的基于行动预测，并利用强化学习进一步提升探索和恢复能力。在连续环境中的房间对房间和房间跨房间（R2R-CE和RxR-CE）基准测试中，Robostral Navial引领了新的技术水平。在R2R-CE上，成功率达到77.4%，比最佳单眼方法高出10.5分，在仅使用单台RGB相机的情况下，领先最强的深度或多摄系统5.3分。在RxR-CE测试下，成功率达到75.1%，优于所有单眼基线。

Robust Asynchronous Q-Learning under Reward and State Corruption via Batching

在奖励和状态损坏下的稳健异步Q-学习，通过批处理实现

• Authors: Sreejeet Maity, Aritra Mitra
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.20822
• Pdf link: https://arxiv.org/pdf/2607.20822
• Abstract Motivated by reinforcement learning in harsh environments, we consider the problem of learning an optimal policy subject to adversarially corrupted feedback. Specifically, at each time-step, an adversary can perturb both the reward and state observations of the learner following the Huber contamination model. To defend against such data corruption, we propose {\texttt{BR-Async-Q}}: a novel, epoch-based, robust (Q)-learning algorithm built upon two key ideas: (i) partitioning the online data stream into batches to reduce variance, and (ii) constructing robust estimates of the Bellman optimality operator using such batched data. We prove a high-probability $\ell_\infty$ error bound for {\texttt{BR-Async-Q}} that matches that for vanilla (Q)-learning, up to a small additive term that scales with the fraction of corrupted samples. To our knowledge, this provides the first robustness guarantee for asynchronous (Q)-learning subject to both reward and state corruption. Furthermore, when only rewards are corrupted, the dependence of our algorithm's bound on the corruption fraction is minimax optimal.
• 中文摘要 基于在恶劣环境中的强化学习，我们考虑在对抗性腐败反馈下学习最优策略的问题。具体来说，在每个时间步，攻击者都可能扰乱学习者的奖励和状态观测，遵循休伯污染模型。为防止此类数据损坏，我们提出了{\texttt{BR-Async-Q}}：一种新颖的基于跨元的稳健\（Q\）学习算法，基于两个关键理念：（i）将在线数据流划分为批次以降低方差，（ii）利用此类批次数据构建Bellman最优算子的稳健估计。我们证明了一个高概率的 $\ell_\infty$ 误差上限，适用于 {\texttt{BR-Async-Q}}，与普通 \（Q\）学习相匹配，且可达一个随损坏样本比例缩放的小加法项。据我们所知，这为异步\（Q\）学习提供了首个稳健性保证，且同时存在奖励和状态腐败。此外，当只有奖励被破坏时，算法对腐败比例的依赖关系是极小极大最优的。

Offline RL with Hierarchical Action Chunking

带有层级动作分块的离线强化学习

• Authors: Ahad Jawaid
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.20834
• Pdf link: https://arxiv.org/pdf/2607.20834
• Abstract Offline goal-conditioned reinforcement learning (RL) holds the promise of learning general-purpose policies from static datasets. However, scaling these methods to long-horizon tasks remains a challenge due to the curse of horizon, where value estimation errors can compound through long chains of bootstrapped Bellman backups. Existing hierarchical approaches mitigate this by decomposing tasks into subgoals, yet they often rely on low-level controllers that suffer from myopic execution and biased value estimates. In this work, we propose Hierarchical Implicit Q-Chunking (HiQC), an offline goal-conditioned RL algorithm that combines high-level latent planning with low-level action chunking. By conditioning the low-level critic on temporally extended action sequences, HiQC enables unbiased k-step value backups, compressing the horizon at both the planning and execution levels. We theoretically demonstrate that this dual decomposition results in a tighter bound on value error under a bounded per-backup error model compared to standard hierarchy or flat chunking alone. Empirically, HiQC achieves the highest aggregate performance among the compared methods on the OGBench suite, with its largest gains on long-horizon navigation tasks such as humanoid-giant.
• 中文摘要 离线目标条件强化学习（RL）有望从静态数据集中学习通用策略。然而，由于视野诅咒，将这些方法扩展到长视野任务仍是一个挑战，在视野中，价值估计误差可能通过长链的自举Bellman备份而累积。现有的层级方法通过将任务分解为子目标来缓解这一问题，但它们往往依赖于低层级控制器，这些控制器执行力短浅且价值估计偏颇。本研究提出层次隐式Q块化（HiQC），一种离线目标条件强化学习算法，结合了高层潜规划与低层动作分块。通过让低级批评者对时间延长的动作序列进行条件，HiQC实现了无偏的k步值备份，压缩了规划和执行层面的视野。我们理论上证明，这种对偶分解在有界的每次备份误差模型下，使值误差的界限比标准层次结构或单纯平分块更为严格。从经验来看，HiQC在OGBench套件中实现了所有比较方法中最高的综合性能，在长视野导航任务（如人形巨型）上提升最大。

Multi-turn RL with Structural and Performance Aware Rewards for CUDA Kernel Generation

多回合强化学习，具结构性和性能感知奖励，用于CUDA内核生成

• Authors: Quazi Ishtiaque Mahmud, Nesreen K. Ahmed, Ali Jannesari
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.20908
• Pdf link: https://arxiv.org/pdf/2607.20908
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful technique to enhance the reasoning capacity of LLMs for optimized code generation. However, existing RLVR approaches primarily rely on outcome-based signals such as correctness and speedup, overlooking performance-critical structural properties of programs that are essential for generating optimized code. In this work, we propose CudaPerf, a reflective RL framework that incorporates both verifiable execution rewards and structural code-aware rewards derived from parallelization features (e.g., memory coalescing, occupancy, Arithmatic Intensity, and synchronization patterns). CudaPerf operates in two stages: (1) an offline pairwise ranking module that learns to distinguish strong and weak program candidates via contrastive comparisons, and (2) an online RL training phase that jointly optimizes for correctness, performance, and structural efficiency through a unified reward signal. To further enhance learning, CudaPerf utilizes iterative refinement using execution feedback enabling progressive improvement of generated candidates. We also introduce a dataset comprising 2.9k C to CUDA and 1k PyTorch to CUDA programs, each paired with diverse input configurations and multiple CUDA implementations encompassing diverse optimization strategies. CudaPerf is evaluated across multiple benchmarks comprising both C to CUDA and PyTorch to CUDA transformations. Empirical findings suggest that CudaPerf significantly outperforms strong baselines, including Qwen-3-32B (for C to CUDA) and CUDA Agent (for PyTorch to CUDA) by achieving up to 5X & 3.32X improvements in speedup, and 17% & 7% improvements in correctness, respectively.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为一种强大的技术，能够提升LLM的推理能力，实现优化代码生成。然而，现有的RLVR方法主要依赖基于结果的信号，如正确性和加速，忽视了程序对生成优化代码至关重要的性能结构特性。在本研究中，我们提出了CudaPerf，一种反思性强化学习框架，结合了可验证的执行奖励和基于并行化特征（如内存合并、占用率、算术强度和同步模式）衍生的结构化代码感知奖励。CudaPerf 分为两个阶段：（1）离线的两对排序模块，通过对比比较学习区分强弱项目候选项目;（2）在线强化学习训练阶段，通过统一的奖励信号共同优化正确性、性能和结构效率。为进一步提升学习效果，CudaPerf采用执行反馈的迭代优化，实现对生成候选人的逐步改进。我们还介绍了一个包含2900个C到CUDA和1000个PyTorch到CUDA程序的数据集，每个数据集配有不同的输入配置和多个CUDA实现，涵盖了不同的优化策略。CudaPerf 在多个基准测试中进行评估，包括 C 到 CUDA 和从 PyTorch 到 CUDA 的转换。实证结果表明，CudaPerf显著优于强基线产品，包括Qwen-3-32B（从C到CUDA）和CUDA Agent（从PyTorch到CUDA）分别实现了最高5倍和3.32倍的加速提升，正确性提升了17%和7%。

The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoning

沉默的重量：潜在国际象棋推理中权重在草稿板上的因果论证

• Authors: Ishan S. Kshirsagar
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.20952
• Pdf link: https://arxiv.org/pdf/2607.20952
• Abstract Latent, or silent, reasoning lets language models carry out intermediate computation in continuous vector space instead of words, and is widely assumed to function as an internal scratchpad the model actively consults during inference. Whether that assumption survives reinforcement learning has not been tested directly: existing causal analyses of latent reasoning are confined to math and logic tasks, and compare a model's reliance on its thoughts within a single checkpoint, never before and after an RL stage. We train a chess-playing model through a staged latent-reasoning curriculum followed by reinforcement learning, and find legality climbs monotonically to 61% (from a 48% pre-RL baseline) while checkmate confabulation is eliminated entirely. To locate this gain, we run a six-condition causal intervention suite on the same model before and after RL: substituting or adding matched noise to the latent thought vectors leaves performance unchanged, ablating them causes only mild degradation, and only exact-zero vectors cause collapse. This robustness gap is itself the finding: under exact-zero corruption, legality collapses to 1% pre-RL versus 9% post-RL, a gap that survives correction for testing across the full battery; milder conditions trend similarly without independently reaching significance. RL appears to add robustness to disruption, not reliance on thought content. These results push back against the field's default assumption that latent thoughts function as an actively consulted inference-time scratchpad, and instead indicate latent reasoning's principal effect here is shaping the model's parameters during training. We also demonstrate a working RL gain in chess, a domain outside the math and logic settings where multiple groups report the same latent-reasoning-plus-RL recipe failing to improve accuracy over SFT.
• 中文摘要 潜在或无声推理使语言模型能够在连续向量空间中进行中间计算，而非单词，并被广泛认为作为模型在推理过程中主动参考的内部草稿。这种假设是否经受强化学习的考验尚未被直接验证：现有的潜在推理因果分析仅限于数学和逻辑任务，比较模型对其思维的依赖程度，仅在单一检查点内进行，从未在强化学习阶段之前和之后进行。我们通过分阶段的潜在推理课程训练一个下棋模型，随后进行强化学习，发现合法性单调地攀升至61%（从强化学习前的48%），而将死虚构则完全被消除。为了找到这一增益，我们在同一模型的强化学习前后运行了六个条件的因果干预套件：在潜在思维向量中替换或添加匹配噪声，表现不变;消减它们仅带来轻微退化;只有精确零向量会导致崩溃。这一鲁棒性差距本身就是研究结果：在完全零腐败条件下，合法性在强化前降至1%，强化后为9%，这一差距经由全电池测试校正后依然存在;较温和的条件也呈类似趋势，但未独立达到显著性。强化学习似乎为颠覆增加了稳健性，而不是依赖思想内容。这些结果反驳了该领域默认假设潜在思维作为主动咨询的推理时间草稿，反而表明潜在推理的主要作用是在训练过程中塑造模型参数。我们还展示了国际象棋中有效的强化学习（RL）提升，这是一个数学和逻辑之外的领域，多个团队报告同一潜在推理加强化学习的配方未能比SFT提升准确性。

EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization

EmoAgent-R1：迈向基于强化学习的动态智能体专化的多模态情感理解

• Authors: Lihuang Fang, Yuchen Zou, kebin Jin, Jinghui Qin
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.21013
• Pdf link: https://arxiv.org/pdf/2607.21013
• Abstract Multimodal large language models (MLLMs) have achieved impressive performance in multimodal emotion recognition (MER) tasks and lifted MER to a new level that is complex emotion understanding with advanced video understanding abilities and natural language description. However, existing MLLM-based methods often use a fixed prompt to perceive the emotions, ignoring the dynamicity and complexity of the emotion source in the multimodal inputs. To address these issues, we propose a novel Reinforcement Learning-based Dynamic Agent Specialization framework (\textbf{EmoAgent-R1}) to optimize the emotion recognition, reasoning, and generalization abilities of an MLLM with dynamic agent specialization based on reinforcement learning. Specifically, we first adopt a cold start strategy to endow an MLLM with preliminary emotion recognition, reasoning, and agent routing ability by training with synthetic answer-conditioned chain-of-thought data and agent routing data. Then, we further train the MLLM with reinforcement learning to perceive emotions in a two-step agentic workflow with agent selection and agent specialization. To effectively train EmoAgent-R1, we propose a novel Progressive Group-Relative Policy Optimization (P-GRPO) to combine group-based relative advantages with a PMI-inspired progressive token-level modulation to transform sparse rewards into fine-grained learning signals, mitigating the coarse-grained uniform credit assignment issue in GRPO. Extensive experiments on MER benchmarks demonstrate the superiority of our EmoAgent-R1 in stronger emotion reasoning performance and improved optimization stability.
• 中文摘要 多模态大型语言模型（MLLMs）在多模态情绪识别（MER）任务中取得了令人印象深刻的表现，并将MER提升到一个新高度，即复杂情感理解，具备先进的视频理解能力和自然语言描述。然而，现有基于MLLM的方法通常使用固定提示来感知情感，忽视多模态输入中情感源的动态性和复杂性。为解决这些问题，我们提出了一种基于强化学习的动态代理专化框架（\textbf{EmoAgent-R1}），以优化基于强化学习的动态主体专精MLLM的情感识别、推理和泛化能力。具体来说，我们首先采用冷启动策略，通过使用合成的答案条件思维链数据和智能体路由数据，赋予MLLM初步的情感识别、推理和代理路由能力。随后，我们进一步通过强化学习训练MLLM，使其在两步代理工作流程中感知情绪，包含代理选择和代理专业化。为有效训练EmoAgent-R1，我们提出了一种新型的渐进式组相对策略优化（P-GRPO），将基于群体的相对优势与受PMI启发的渐进式代币级调制结合，将稀疏奖励转化为细粒度学习信号，缓解GRPO中粗粒度均匀的学分分配问题。对MER基准测试的广泛实验证明了我们的EmoAgent-R1在更强的情绪推理性能和优化稳定性方面的优势。

From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

从评估到优化：Python 中基于层级感知的训练信号用于 CWE 预测

• Authors: Muntasir Adnan, Manile Srun, Carlos C. N. Kuhn
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.21069
• Pdf link: https://arxiv.org/pdf/2607.21069
• Abstract The original ALPHA benchmark introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penalty could theoretically also serve as a training signal. This paper provides that validation. We compare three delivery mechanisms: supervised fine-tuning, a dual-head classification loss, and reinforcement learning with a dense reward derived from the normalised penalty. We find that supervised approaches consistently regress below the zero-shot baseline under distribution shift, while GRPO succeeds. Our best policy reduces the cumulative ALPHA penalty of Qwen2.5-Coder-7B on Security Hardening and Adversarial Testing (SVEN) dataset by 27.9% under greedy decoding, and by 25.5% under sampled decoding(p = 0.005, Welch's t-test), reaching statistical parity with its 4.5x larger zero-shot teacher. We conclude that the value of a hierarchical penalty as a training signal depends largely on the directness of its delivery.
• 中文摘要 最初的ALPHA基准引入了针对Python中CWE级别漏洞预测的分类法感知惩罚，并提出该惩罚理论上也可以作为训练信号。这篇论文提供了这种验证。我们比较了三种传递机制：监督式微调、双头分类损失，以及与归一化惩罚中得出的密集奖励的强化学习。我们发现，监督式方法在分布转移下持续回归到零射点基线以下，而GRPO则成功。我们的最佳策略在贪婪解码下，将Qwen2.5-Coder-7B在安全加固与对抗性测试（SVEN）数据集上的累计ALPHA惩罚降低了27.9%，在抽样解码（p = 0.005，Welch的t检验）下减少了25.5%，与其4.5倍的零样本教师实现了统计上的平等。我们得出结论，分层惩罚作为训练信号的价值很大程度上取决于其传递的直接性。

Training Large Language Models for Self-Explanation Faithfulness

训练大型语言模型以实现自我解释的忠实性

• Authors: Yeoktatt Cheah, María Pérez-Ortiz, Noah Y. Siegel, Oana-Maria Camburu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.21090
• Pdf link: https://arxiv.org/pdf/2607.21090
• Abstract We propose a Reinforcement Learning (RL) method to directly optimize the faithfulness of self-explanations - the extent to which a model's generated reasoning accurately reflects its internal decision-making process. While existing work focuses on evaluating faithfulness or using inference-time prompting frameworks to improve an LLM's self-explanation's tractability, these approaches do not provide a mechanism to directly optimize a model's parameters to generate faithful self-explanations. We bridge this gap by modifying existing faithfulness metrics into an RL training objective. We investigate (1) if models can be trained to accurately detect factors that affect their decisions, and (2) whether RL can directly optimize for the disclosure of these factors thereby improving LLM self-explanations' faithfulness. We experiment with two intervention types: random-word insertions and user-bias insertions, using a per-sample reward derived from the Phi-CCT correlation metric. RL fine-tuned Llama3.1-8B and Qwen3-8B show substantial improvements on the Phi-CCT faithfulness metric, with in-distribution scores rising from near-zero to as high as 0.664, and out-of-distribution scores reaching up to 0.691 on held-out tasks such as StrategyQA. Cross-intervention generalization is weaker but more interesting: a priori we would not expect a model trained only on random word insertions to generalize to user-bias phrases, yet Llama3.1-8B shows non-zero transfer in this direction. The reverse direction and Qwen3-8B do not replicate this, indicating model-dependent and setup-dependent effects we cannot yet explain. Lastly we analyze model behavior to rule out reward gaming behaviors that often plague RL training. Ultimately, we show that models can be trained to implicitly identify influential factors and disclose them, offering a scalable path toward reducing unfaithful reasoning in LLMs.
• 中文摘要 我们提出了一种强化学习（RL）方法，直接优化自我解释的忠实度——即模型生成的推理准确反映其内部决策过程的程度。现有研究聚焦于评估忠实性或利用推理时间提示框架提升LLM自我解释的可解性，但这些方法并未提供直接优化模型参数以生成忠实自我解释的机制。我们通过修改现有的忠实度指标，将这一差距转化为强化学习培训目标。我们调查：（1）模型是否可以训练以准确检测影响其决策的因素，（2）强化学习是否能直接优化这些因素的披露，从而提升大语言模型自我解释的忠实度。我们尝试了两种干预类型：随机词插入和用户偏倚插入，使用基于Phi-CCT相关指标的单样本奖励。强化学习微调的Llama3.1-8B和Qwen3-8B在Phi-CCT忠实度指标上有显著提升，分布内得分从接近零上升至高达0.664，而在未分配任务如StrategyQA中，分布外得分最高可达0.691。交叉干预泛化较弱但更有趣：先验来看，仅用随机词汇插入训练的模型不会推广到用户偏向短语，但Llama3.1-8B显示出在此方向上的非零转移。反向和Qwen3-8B则无法复制此效果，表明模型和设置依赖效应尚无法解释。最后，我们分析模型行为，排除经常困扰强化学习训练的奖励游戏行为。最终，我们证明模型可以被训练成隐性识别影响因素并披露它们，为减少大型语言模型中不忠推理提供了可扩展的路径。

TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning

巡演：离线强化学习的轨迹级去学习基准

• Authors: Chaofan Pan, Lingfei Ren, Xiangyu Jiang, Yanhua Li, Xuemei Cao, Xiangkun Wang, Hao Yu, Wei Wei, Xin Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.21111
• Pdf link: https://arxiv.org/pdf/2607.21111
• Abstract Offline Reinforcement Learning (RL) agents are trained on fixed behavioral trajectories, which makes trajectory-level deletion important when selected data must be removed after training. Evaluating such deletion is difficult because a lower membership score can reflect trajectory removal, residual memorization visible to another attack, or policy collapse that destroys useful behavior. We introduce Trajectory-level memOrization and Unlearning in offline RL (TOUR), a benchmark that combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing. Across D4RL locomotion experiments and an exploratory AntMaze extension, TOUR shows that common deletion baselines have environment-dependent privacy-utility behavior. Retraining and fine-tuning often provide stronger retained-utility references than uniform GA+Refit, while TrajDeleter remains a useful comparator but is not uniformly stronger under the same audit. Reference-model, threshold, deviation, equivalence, action-error, representation-based, and query-limited attacks further show that a single likelihood-based membership score can overstate deletion quality. In the evaluated settings, conclusions about offline RL unlearning are therefore not stable under single-score auditing. They depend on matched non-member construction, retraining-relative calibration, attack family, retained utility, and explicit scope for diagnostic architecture or component-level evidence.
• 中文摘要 离线强化学习（RL）智能体的训练基于固定的行为轨迹，因此当训练后必须删除所选数据时，轨迹级删除尤为重要。评估此类删除很难，因为较低的成员分数可能反映出轨迹被移除、对其他攻击可见的残余记忆，或策略崩溃导致有用行为的破坏。我们引入了轨迹级记忆与离线强化学习（TOUR）基准测试，结合轨迹级分区、匹配非成员控制、再训练引用、保留性能锚点和多重攻击隐私审计。通过D4RL的移动实验和探索性的AntMaze扩展，TOUR显示了共同删除基线具有环境依赖的隐私-效用行为。重训练和微调通常比统一的GA+Refit提供更强的保留效用参考，而TrajDeleter仍是有用的比较器，但在同一审计下并不均匀强。参考模型、阈值攻击、偏差攻击、等价攻击、动作错误攻击、基于表示攻击和查询限制攻击进一步表明，单一基于似然的成员评分可能高估删除质量。在评估环境中，关于离线强化学习逆学习的结论在单一评分审计下并不稳定。它们依赖于匹配非成员构建、再训练相对校准、攻击家族、保留效用以及明确的诊断架构或组件级证据范围。

Relative Value Learning

相对价值学习

• Authors: Marc Höftmann, Jan Robine, Stefan Harmeling
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.21120
• Pdf link: https://arxiv.org/pdf/2607.21120
• Abstract In reinforcement learning, critics typically estimate absolute state values $V(s)$, estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function $\Delta(s_i, s_j) = V(s_i) - V(s_j)$. We introduce a pairwise Bellman operator and prove it is a $\gamma$-contraction with a unique fixed point equal to the true value differences, derive well-posed $1$-step, $n$-step and $\lambda$-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.
• 中文摘要 在强化学习中，批评者通常估计绝对状态值$V（s）$，估算特定情境在孤立情况下的良好程度。然而，事实证明，只有价值差异才对控制有影响。基于此，我们提出了相对价值学习（RV）框架，通过一个反对称函数 $\Delta（s_i， s_j） = V（s_i） - V（s_j）$ 直接学习价值差异。我们引入一个两对贝尔曼算子，并证明其是一个$\gamma$收缩，且不动点唯一等于真实值差，推导出良态的$1$步长、$n$步长和$\lambda$收益目标，并从两两差重建广义优势估计，获得无偏策略梯度估计量（R-GAE）。超越理论结果，我们将RV与PPO整合，并在Atari基准测试（49款ALE游戏）中表现优于标准PPO，表明相对价值估计是对绝对批评者的有效替代方案。

Deep Reinforcement Learning for Adaptive Gain Tuning in Control of Teleoperation Manipulators with Joint Flexibility and Time-Varying Delays

深度强化学习用于控制具关节灵活性和时变延迟的远程操作机械臂中的自适应增益调谐

• Authors: Armin Attarzadeh, Mohammad Ali Ghaemifar, Alireza Khanzadeh, Soheil Ganjefar
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.21145
• Pdf link: https://arxiv.org/pdf/2607.21145
• Abstract Bilateral teleoperation systems that include joint flexibility better reflect real robotic systems used in surgery, space, and rehabilitation. However, joint flexibility together with time-varying communication delays makes it difficult to maintain stable and coordinated motion between the master and slave robots. To address this, we propose a hybrid control method that combines a stable Proportional-plus-Damping (P+d) controller with a model-free deep reinforcement learning agent based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The P+d controller provides basic stability under bounded delays, while the learning agent adjusts and tunes the remote-side proportional and damping gains in real time to reduce vibrations and improve tracking. Stability is guaranteed for bounded time-varying delays using Lyapunov-Krasovskii analysis. The approach provides a practical solution for teleoperation systems facing both joint flexibility and uncertain network delays.
• 中文摘要 具备关节灵活性的双侧远程操作系统更能反映实际用于手术、航天和康复的机器人系统。然而，关节的灵活性加上随时间变化的通信延迟，使得主从机器人之间难以保持稳定协调的运动。为此，我们提出了一种混合控制方法，结合稳定的比例加阻尼（P+d）控制器与基于双延迟深度确定性策略梯度（TD3）算法的无模型深度强化学习代理。P+d控制器在有界延迟下提供基本稳定性，而学习代理实时调整和调谐远端的比例增益和阻尼增益，以减少振动并改善跟踪。利用李雅普诺夫-克拉索夫斯基分析，保证有界时间变化延迟的稳定性。该方法为面临联合灵活性和网络延迟不确定性的远程操作系统提供了切实可行的解决方案。

FORGE-plus: Force-Budgeted Recovery for Contact-Rich Assembly with a Frozen LLM Supervisor

FORGE-plus：带冻结LLM Supervisor的强制预算恢复，用于接触丰富组装

• Authors: Kyupaeck Jeff Rah, Midum Oh
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.21227
• Pdf link: https://arxiv.org/pdf/2607.21227
• Abstract Force-conditioned reinforcement learning (RL) enables tight-clearance assembly under a commanded force ceiling, but practical deployment requires determining an appropriate force limit for each object and recovering from insertion failures without exceeding it. We present a two-layer framework in which a frozen, text-only large language model (LLM) assigns a per-object force ceiling before execution and selects recovery maneuvers from a fixed action menu using compact textual force signatures. The LLM never controls force directly: a low-level controller enforces the force ceiling, the recovery policy cannot increase it, and the hidden breaking-force threshold is known only to the evaluator. We evaluate the framework on fragile bottle placement and 0.4 mm diametral-clearance gear insertion using two grippers (Robotiq 2F-140 and Franka Panda hand). A single policy passes 256/256 evaluation episodes on both fragile and robust objects without breakage, correctly predicts release timing, and completes a full table-pick-and-insert pipeline with a mean peak force of 5.4 N. Under injected in-grip slip, the force-signature recovery strategy resolves 40% and 64% of failures on the two grippers, whereas a press-harder baseline is either ineffective or causes frequent breakage. We also report negative results, including the failure of PPO to solve the task under strict force constraints and unsuccessful learned release strategies. All experiments are conducted in rigid-body simulation with hidden force-threshold breakage; no sim-to-real claim is made.
• 中文摘要 力条件强化学习（RL）在指令力上限下实现紧密间隙组装，但实际部署需要为每个物体确定适当的力限，并在插入失败后恢复且不超过该限值。我们提出了一个两层框架，其中一个冻结的纯文本大型语言模型（LLM）在执行前为每个对象分配一个力量上限，并通过紧凑的文本力量签名从固定动作菜单中选择恢复机动。LLM从不直接控制力：低级控制器执行力上限，恢复策略无法增加上限，且隐藏的断力阈值只有评估者知道。我们使用两个夹具（Robotiq 2F-140和Franka Panda手）评估了脆弱瓶子放置和0.4毫米直径净隙齿轮插入的框架。单一策略可通过256/256评估，适用于脆弱和坚固物体，且无断裂，准确预测释放时间，并完成完整表格选入与插入流水线，平均峰值力为5.4牛顿。在注入式握把滑动下，力签名恢复策略解决了两个夹具40%和64%的失效，而用力踩的基线要么无效，要么频繁断裂。我们还报告了负面结果，包括PPO在严格兵力约束下未能完成任务，以及学习释放策略失败。所有实验均在刚体模拟中进行，具有隐藏的力阈值断裂;没有模拟到真实的说法。

Expert Behavior Prior Reinforcement Learning

专家行为：事前强化学习

• Authors: Gong Gao, Weidong Zhao, Xianhui Liu, Ning Jia
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.21302
• Pdf link: https://arxiv.org/pdf/2607.21302
• Abstract Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations. However, most existing BPRL methods rely on static offline datasets, which often suffer from low data diversity and suboptimal trajectory quality. This reliance restricts the effectiveness of policy priors, hindering both policy exploitation and stability during online training. Consequently, agents are prone to inefficient exploration and unstable learning dynamics. To address these limitations, we deviate from existing offline pre-training methods and propose an Expert Behavior Prior (EBP) algorithm. Specifically, we introduce a Q-guided conditional variational autoencoder (Q-CVAE) that learns to generate expert policy priors directly from the online replay buffer. This enables the generation of high-value actions for guiding policy updates without relying on pre-collected expert trajectories. To further enhance policy exploitation, we propose an expert policy guidance (EPG) mechanism that selects expert actions from a generative support set, and we integrate a policy gradient correction (PGC) module to harmonize Q-guidance with expert supervision, promoting stable and consistent policy improvement. Extensive experiments conducted on robotic control (Gym, PyBullet) and industrial control (DMControl) benchmarks demonstrate that EBP significantly outperforms state-of-the-art online RL algorithms, achieving higher sample efficiency and more stable convergence.
• 中文摘要 行为先验强化学习（BPRL）已成为一种有前景的范式，通过利用离线演示得出的策略先验，提升在线强化学习（RL）样本效率。然而，大多数现有BPRL方法依赖静态离线数据集，这些数据集常常存在数据多样性低和轨迹质量不佳的问题。这种依赖限制了政策先验的有效性，阻碍了政策利用和在线培训的稳定性。因此，智能体容易在探索过程中效率低下，学习动态不稳定。为解决这些局限性，我们偏离现有的离线预训练方法，提出了专家行为先验（EBP）算法。具体来说，我们引入了Q引导的条件变分自编码器（Q-CVAE），它能直接从在线重放缓冲区生成专家策略先验。这使得无需依赖预先收集的专家轨迹即可生成高价值的政策更新行动。为进一步加强政策利用，我们提出专家政策指导（EPG）机制，从生成支持集中选择专家行动，并整合政策梯度修正（PGC）模块，协调Q指导与专家监督，促进稳定且一致的政策改进。在机器人控制（Gym、PyBullet）和工业控制（DMControl）基准测试中进行的大量实验表明，EBP的表现远超最先进的在线强化学习算法，实现了更高的采样效率和更稳定的收敛。

How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning

适配器能写多少位？参数高效微调的容量与记忆度的测量

• Authors: Kaizhen Tan, Heqing Du, Yang Feng
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.21351
• Pdf link: https://arxiv.org/pdf/2607.21351
• Abstract A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it. We put that assumption on a scale. Extending compression-based memorization analysis to the frozen-base setting, we measure directly, in bits, how much a low-rank adapter writes into a model it never changes. The answer is both smaller than full fine-tuning and less lawful than parameter counting would predict. Adapters store a couple of bits per trainable parameter, well short of a full model's budget, but that figure turns less on how many parameters an adapter carries than on where they sit. Move the same parameter budget from attention into the MLP and it holds nearly twice as much; strip the frozen base of its structure and the capacity all but disappears. Applied to realistic fine-tunes of Qwen2.5, the same instrument shows privacy leakage rising with the bits an adapter writes rather than the parameters it nominally has, and it draws a clean line between supervised and reinforcement learning: the secrets that supervised fine-tuning copies down verbatim, an adapter trained on verifiable rewards never records. Measuring what fine-tuning writes, rather than attacking it after the fact, turns a piece of folklore into a quantity one can design against.
• 中文摘要 LoRA适配器只有几兆字节，几乎所有人都把它当作技能，而不是背后数据的记录。我们把这个假设放在一个天平上。将基于压缩的记忆分析扩展到冻结基设置，我们直接以比特为单位测量低阶适配器在一个从未改变的模型中写入的量。答案既小于完全微调，也比参数计数预测的更不合法。适配器每个可训练参数存储几位，远低于模型的预算，但这个数字更多取决于适配器携带多少参数，而是它们的位置。将同样的参数预算从注意力转移到MLP，它几乎能容纳两倍的参数;剥离冻结的基地结构，容量几乎消失。应用于Qwen2.5的真实微调，同一仪器显示隐私泄露随着适配器写入的比特数量而增加，而非其名义参数，并且在监督学习与强化学习之间划出了清晰的界限：监督微调的秘密逐字复制，而训练于可验证奖励的适配器从未记录。衡量微调所写的内容，而不是事后攻击它，能将一段民间传说变成一个可以设计的数量。

PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning

PATS：策略意识培训支架用于代理强化学习

• Authors: Yipeng Shi, Zhipeng Ma, Yue Wang, Qitai Tan, Yang Li, Peng Chen, Zhengzhou Zhu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.21419
• Pdf link: https://arxiv.org/pdf/2607.21419
• Abstract In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy. To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold. Our framework, Pats, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts. Concrete guidance helps weak policies to complete challenging tasks. As policy improves, redundant context is revised or removed to reduce reliance on explicit guidance while preserving useful rollout variation. The policy is optimized with environmental rewards using standard RLVR, and the training scaffold is discarded at deployment. On ALFWorld and WebShop, Pats improves over strong baselines by up to 18.6%. Across seven search-augmented QA benchmarks, it remains competitive while using 32.1% fewer prompt tokens than the baseline.
• 中文摘要 在长期视野LLM代理强化学习中，弱策略常常重复类似失败，导致无益的推广轨迹，限制有效的策略优化。现有的以技能为中心的方法通过优化、过滤或内化可重复使用的技能来提升探索效果。然而，这些政策仍以技能本身为中心，而非作为适应政策演变的自适应培训时间支持。为此，我们提出了一种以政策为中心的培训范式，将技能重新框架为动态的培训支架。我们的框架Pats将最新政策的部署组转换为证据卡，并通过任务特定评估调整后续推广中使用的上下文。具体的指导帮助薄弱的政策完成具有挑战性的任务。随着政策改进，冗余上下文会被修订或删除，以减少对明确指导的依赖，同时保留有用的推广变异性。该政策通过标准RLVR进行环境奖励优化，培训支架在部署时被舍弃。在ALFWorld和WebShop上，Pats在强势基线基础上提升了最多18.6%。在七项搜索增强的质量保证基准中，它依然具有竞争力，同时使用了比基线少32.1%的提示代币。

AREX: Towards a Recursively Self-Improving Agent for Deep Research

AREX：迈向一个递归自我改进的深度研究代理

• Authors: Shuqi Lu, Chaofan Li, Kun Luo, Zhang Zhang, Hui Wang, Hongwang Xiao, Zheng Liu, Lei Xiong, Jiahao Wang, Sen Wang, Xiyan Jiang, Wanli Li, Yuyang Hu, Hongjin Qian, Bingyu Yan, Ziyi Xia, Yingxia Shao, Kang Liu, Zhicheng Dou, Di He, Chaozhuo Li, Qiwei Ye, Zhongyuan Wang, Zheng Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.21461
• Pdf link: https://arxiv.org/pdf/2607.21461
• Abstract Deep research requires agents to find answers that jointly satisfy multiple constraints. Discovering such answers is costly, whereas verifying a candidate can often be decomposed into tractable constraint-wise checks. This discovery--verification asymmetry suggests that a research agent should do more than simply search longer: it should recursively improve its current answer by verifying intermediate results and using the partially verified state to guide subsequent refinement. We introduce AREX, a family of Recursively Self-Improving (RSI) deep research agents. AREX alternates between an inner research loop that gathers evidence and constructs a provisional answer, and an outer self-improvement loop that audits the answer constraint-wise, identifies unresolved claims, and launches targeted follow-up research. To sustain RSI over long horizons, AREX learns an autonomous context-update tool that compresses growing interaction history into a compact improvement state preserving verified evidence and unresolved constraints, without relying on an external model. We train AREX on verified synthetic tasks and high-quality trajectories through agentic mid-training and long-horizon reinforcement learning. To mitigate sparse final rewards during long horizon learning, we emphasize key steps where decisive evidence is acquired or erroneous research directions are corrected. We instantiate a dense 4B model and a 122B-A10B Mixture-of-Experts model. Across BrowseComp, WideSearch, DeepSearchQA, Humanity's Last Exam (HLE), and other reasoning and tool-use benchmarks, AREX substantially outperforms comparable-scale baselines and remains competitive with models using substantially more activated parameters.
• 中文摘要 深入研究要求代理找到同时满足多重约束的答案。发现这些答案成本高昂，而验证候选人通常可以分解为可处理的约束性检查。这一发现——验证不对称性表明，研究代理应不仅仅是延长搜索时间：它应递归地通过验证中间结果并利用部分验证状态指导后续细化来改进当前答案。我们介绍AREX，一类递归自我改进（RSI）深度研究代理。AREX在内部研究循环（收集证据并构建临时答案）和外部自我改进循环之间交替进行，后者在约束方面审计答案，识别未解决的主张，并启动有针对性的后续研究。为了长期维持RSI，AREX学习了一种自主上下文更新工具，将不断增长的交互历史压缩为紧凑的改进状态，保留已验证的证据和未解决的约束，无需依赖外部模型。我们通过能动的中期训练和长期强化学习，训练AREX进行经过验证的合成任务和高质量轨迹。为了减少长期学习中最终奖励稀疏，我们强调关键步骤，这些步骤是获得决定性证据或纠正错误研究方向的。我们实例化了一个稠密的4B模型和一个122B-A10B专家混合模型。在BrowseComp、WideSearch、DeepSearchQA、Humanity's Last Exam（HLE）及其他推理和工具使用基准中，AREX的表现远超同等规模的基线，并与使用更活跃参数的模型保持竞争力。

Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections

无信号灯路口自动驾驶车辆的紧凑潜在协调

• Authors: Gil Lifshits, Igal Bilik, Gilad Katz
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.21488
• Pdf link: https://arxiv.org/pdf/2607.21488
• Abstract Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multi-agent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-of-concept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-vehicle coordination.
• 中文摘要 在无信号灯路口协调自动驾驶车辆仍是多智能体强化学习（MARL）系统面临的关键挑战，这些系统通常难以适应组合动作空间、对特权信息的依赖或僵化的智能体设计。我们提出了主代理原型计划系统（MAPS），这是一种分层深度强化学习（DRL）架构，其中集中式主智能体生成紧凑、连续的嵌入，称为原型计划，编码全局协调策略。去中心化的Worker代理将这种嵌入与本地观察集成，执行车辆特定的控制，将战略意图与战术执行脱钩，实现每个模块的独立优化。作为该协调机制的概念验证评估，我们在HighwayEnv中测试了72个交叉口配置的MAPS。MAPS实现了无碰撞导航，同时显著缩短平均行车时间，优于最先进的基线。所学的原型计划还表现出强有力的泛化性：由三智能体训练的系统在零shot到五智能体场景中成功率达94%，证实基于原型计划的层级学习为多载体协调提供了有前景的框架。

MIRROR: Learning from the Other View for Multi-Modal Reasoning

镜像：从他者视角学习多模态推理

• Authors: Wen Ye, Yuxiao Qu, Aviral Kumar, Xuezhe Ma
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.21552
• Pdf link: https://arxiv.org/pdf/2607.21552
• Abstract Unlike large language models (LLMs) that exhibit strong reasoning capabilities, vision-language models (VLMs) struggle with visual reasoning, even on geometry problems that admit equivalent text, diagram, and combined diagram+text views. We show that these views often elicit different behaviors: a model may solve a problem from text but fail on the corresponding diagram, or succeed visually while failing textually. This inconsistency suggests that different views expose complementary reasoning paths and failure modes that standard multimodal post-training does not fully exploit. To study and exploit this phenomenon, we construct ODA-Data, a high-quality paired multimodal geometry dataset with text-dominant, image-dominant, and combined image+text views of the same problems, together with splits for training and evaluating modality-dependent reasoning behaviors. We then develop Modality-Informed Reciprocal Reasoning Optimization (MIRROR), a reinforcement learning approach for improving multimodal reasoning via self supervision. For each problem, MIRROR evaluates the model under all views, selects the best-performing view as a teacher, and trains other views with a reverse-KL objective towards the teacher. Across reasoning benchmarks that evaluate on geometry problems, MIRROR improves over standard RL and yields more accurate and consistent behavior across modalities
• 中文摘要 与展现强大推理能力的大型语言模型（LLMs）不同，视觉语言模型（VLMs）在视觉推理方面存在困难，即使在允许等价文本、图表和组合图+文本视图的几何问题上也是如此。我们展示了这些视图常常引发不同的行为：模型可能通过文本解决问题，但在相应的图示上失败，或者视觉上成功但文本上失败。这种不一致表明不同观点暴露了互补的推理路径和失败模式，而标准多模态后训练未能充分利用这些。为研究和利用这一现象，我们构建了ODA-Data，一个高质量的配对多模态几何数据集，包含文本主导、图像主导及图像+文本联合视图，并配有用于训练和评估模态依赖推理行为的拆分。随后，我们开发了模态知情互惠推理优化（MIRROR），这是一种通过自我监督提升多模态推理的强化学习方法。对于每个问题，MIRROR在所有视图下评估模型，选择教师表现最佳的视图，并以反KL目标训练其他面向教师的视图。在评估几何问题的推理基准中，MIRROR相较标准强化学习更为精准，且在不同模态间产生更准确和一致的行为

Keyword: diffusion policy

There is no result