生成时间: 2026-07-10 18:57:08 (UTC+8); Arxiv 发布时间: 2026-07-10 20:00 EDT (2026-07-11 08:00 UTC+8)

今天共有 29 篇相关文章

Keyword: reinforcement learning

Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation

选择性左移：将测试时计算和基于难度的策划转化为低资源代码生成的训练数据

• Authors: Didula Samaraweera, Anjana Supun, Srinath Perera
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07748
• Pdf link: https://arxiv.org/pdf/2607.07748
• Abstract Large Language Models achieve strong code generation for high resource languages like Python and Java but suffer sharp performance drops on Low-Resource Programming Languages~(LRPLs) such as Julia. Improving Small Language Models~(SLMs) for these languages faces a trilemma: Supervised Fine-Tuning~(SFT) is bottlenecked by data scarcity, inference-time scaling is too expensive for deployment, and Reinforcement Learning from scratch yields near zero advantages. We propose a three-phase pipeline that resolves this trilemma by decoupling syntax acquisition from algorithmic reasoning. First, we \emph{left-shift} inference-time compute to an offline data synthesis engine that uses iterative compiler and test feedback to generate verified training examples. Second, we fine-tune an SLM on this synthetic, verified data to embed strong syntactic priors. Third, we apply Reinforcement Learning with Verifiable Reward~(RLVR) grounded by language-agnostic Input/Output tests, where the SFT prior constrains exploration away from syntax errors. Applied to Qwen3-8B, our pipeline improves pass@1 by up to +7.6 points on MultiPL-E and +14.2 points on the Agnostics LiveCodeBench for Julia compared to SOTA results. Furthermore, the pipeline only used $\frac{1}{3}$ data and $\frac{1}{6}$ cost over the previous state-of-the-art. We further demonstrate that the pipeline generalizes to Ballerina achieving 49.7\% MultiPL-E Pass@1, a language with near-zero pretraining representation. Ablations confirm that both the SFT phase and execution-grounded rewards are necessary for stable training.
• 中文摘要 大型语言模型在 Python 和 Java 等高资源语言上实现了强代码生成，但在低资源编程语言~（LRPL）如 Julia 上性能明显下降。为这些语言改进小语言模型~（SLMs）面临三难境地：监督式微调~（SFT）因数据稀缺而瓶颈，推理时间扩展成本过高，而从零开始的强化学习几乎没有优势。我们提出了一种三阶段流水线，通过将语法获取与算法推理解耦来解决这一三难困境。首先，我们将推理时间计算到一个离线数据综合引擎，该引擎使用迭代编译器和测试反馈生成经过验证的训练示例。其次，我们对这些合成且经过验证的数据进行SLM微调，以嵌入强语法先验。第三，我们应用基于语言无关输入输出测试的可验证奖励强化学习~（RLVR），其中SFT先验限制了对语法错误的探索。应用于Qwen3-8B时，我们的流程在MultiPL-E上pass@1提升了最多+7.6分，在Agnostics LiveCodeBench上为Julia提升了+14.2分，相比SOTA结果。此外，该管道仅使用了$\frac{1}{3}$数据和$\frac{1}{6}$成本，而非之前最先进的。我们进一步证明，该流水线可推广至Ballerina，实现了49.7%的MultiPL-E Pass@1，这是一种近乎零的预训练表示语言。消融证实，SFT阶段和执行基础奖励对于稳定训练都是必要的。

A Transdiagnostic Space of Disorder Like Phenotypes in Reinforcement Learning Agents

强化学习代理中类似表型的跨诊断障碍空间

• Authors: Hari Prasad
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07753
• Pdf link: https://arxiv.org/pdf/2607.07753
• Abstract Modelling psychological disorders in artificial agents offers both a testbed for computational psychiatry and a lens on the failure modes of affective control. Prior work induces one or two disorders in a reinforcement learning (RL) agent by hand-tuned reward shaping, labels the behaviour post hoc, and reports single runs. We recast disorder modelling as dose-controllable manipulation of cognitive appraisal signals in an appraisal-guided PPO agent, expressing seven disorders (anxiety, mania, obsessive-compulsive checking, depression, impulsivity, addiction, and post-traumatic stress) each as a single knob grounded in a computational psychiatry account, with each symptom measured by a preregistered assay mapped to a recognised paradigm. Across more than a thousand runs (10 seeds, four controls, 95% confidence intervals) every disorder shows a graded, monotone dose-response that no control reproduces. Beyond these induced effects, three findings emerge that were not written into the reward: the disorders self-organise into a two-dimensional affective space in which mania mirrors anxiety; removing a knob remits reward distortion disorders (mania, checking, addiction) but not avoidance disorders (anxiety, PTSD), which instead recover under a graded exposure curriculum; and two simultaneous knobs interact nonadditively, yielding testable comorbidity predictions. Appraisal weights thus parameterise a controllable space of affective phenotypes in which the same knobs that induce a disorder can model its treatment. We also show that three disorder knobs (depression, addiction, anxiety) transfer to a three-dimensional pixel environment (MiniWorld) with a standard convolutional agent and no appraisal critic, with cross-assay dissociation confirmed across both domains, indicating the framework is not specific to grid worlds or to PPO's appraisal critic.
• 中文摘要 对人工代理心理障碍的建模既为计算精神病学提供了试验平台，也为情感控制的失效模式提供了视角。以往的研究通过手工调优的奖励塑造诱导强化学习（RL）代理出现一到两种障碍，事后标记行为，并报告单次运行。我们将障碍建模重新定义为评估引导的PPO药物中认知评估信号的剂量可控操作，表达七种疾病（焦虑、躁狂、强迫性检查、抑郁、冲动、成瘾和创伤后应激），每个障碍都以计算精神病学账户为基础，每个症状通过预注册的检测法测量，映射到公认范式。在超过一千次运行（10个种子，4个对照，95%置信区间）中，每种疾病都表现出分级、单调的剂量反应，且无对照组能重复出现。除了这些诱发效应外，还有三个未被写入奖励的发现：这些障碍自我组织成一个二维的情感空间，其中躁狂与焦虑相映成趣;移除一个旋钮可以缓解奖赏扭曲障碍（躁狂、检查、成瘾），但不会缓解回避障碍（焦虑、PTSD），这些障碍在分级暴露课程下会恢复;两个同时存在的旋钮以非加法方式相互作用，从而得出可检验的共病预测。因此，评估权重参数化了一个可控的情感表型空间，在该空间中，诱发疾病的同一旋钮可以模拟其治疗。我们还展示了三个障碍旋钮（抑郁、成瘾、焦虑）转移到带有标准卷积代理且无评估批评者的三维像素环境（MiniWorld），且在两个领域均有交叉测定解离，表明该框架并非针对网格世界或PPO的评估批评者。

Scalable and Trustworthy Earth Observation Foundation Models

可扩展且可信的地球观测基金会模型

• Authors: Syed Usama Imtiaz, Mitra Nasr Azadani, Nasrin Alamdari
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07758
• Pdf link: https://arxiv.org/pdf/2607.07758
• Abstract Foundation models (FMs) have transformed machine learning from isolated task-specific model development toward general-purpose models pretrained on broad data and adapted to multiple downstream tasks. Earth observation (EO) is an important domain for this paradigm because satellite and airborne archives are large, high-revisit, and increasingly multimodal, while reliable field labels are often sparse. Remote sensing foundation models (RSFMs) cannot be transferred reliably/optimally without domain-specific adaptation. This is because EO data are governed by measurement physics and operational decision constraints. This chapter reviews the design principles arising from these domain-specific constraints. It first defines the FMs paradigm in remote sensing (RS), then synthesizes the current model landscape, pretraining objectives, architecture designs, downstream adaptation and trustworthiness requirements. The chapter also incorporates recent benchmark evidence showing that no single geospatial foundation model is universally best and that inconsistent evaluation remains a major issue to fair comparison and reliable deployment. In addition, two brief environmental monitoring case studies; physics-informed spectral targeted masking for harmful algal bloom prediction and reinforcement learning for adaptive environmental monitoring station selection to illustrate the FMs domain-guided principles in practice. This chapter posits that next-generation RSFMs should be evaluated not only by benchmark accuracy, but also by modality-aware transfer and physically plausible representations for trustworthy EO decisions.
• 中文摘要 基础模型（FM）将机器学习从孤立的任务特定模型开发转变为基于广泛数据预训练并适应多个下游任务的通用模型。地球观测（EO）是这一范式的重要领域，因为卫星和空中档案规模庞大、重访率高且日益多模态化，而可靠的现场标签往往稀少。遥感基础模型（RSFM）若不进行特定领域的适应，无法可靠或最优地传输。这是因为EO数据受测量物理和操作决策约束的约束。本章回顾了这些领域特定约束所产生的设计原则。它首先定义了遥感（RS）中的FM范式，然后综合当前模型格局、预训练目标、架构设计、下游适配和可信度要求。本章还纳入了最新的基准证据，表明没有单一地理空间基础模型是普适最佳的，且评估不一致仍是公平比较和可靠部署的主要问题。此外，还有两个简短的环境监测案例研究;基于物理的谱准掩蔽用于有害藻华预测和强化学习，用于自适应环境监测站点选择，以展示FMs的领域导向原理在实际应用中。本章提出，下一代RSFM不仅应以基准准确性为评估标准，还应考虑模态感知的转移和物理上合理的表征，以实现可信的EO决策。

Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms

深度强化学习评估与设计范式的原则性分析

• Authors: Ezgi Korkmaz
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07769
• Pdf link: https://arxiv.org/pdf/2607.07769
• Abstract Starting from the utilization of deep neural networks to approximate the state-action value function that led to winning one of the most challenging games, to algorithmic advancements that allowed solving problems without even explicitly stating the rules of the challenge at hand, reinforcement learning research has been the center of remarkable scientific progress for the past decade. In this paper, we focus on the key ingredients of this research progress and we analyze the canonical evaluation and design paradigms in reinforcement learning. We introduce the theoretical foundations of scaling laws in reinforcement learning and show that the asymptotic performance of reinforcement learning algorithms does not have a monotone relationship between performance rankings and data-regimes. We conduct large-scale experiments and our results demonstrate that a line of reinforcement learning research under the canonical design and evaluation paradigms resulted in incorrect conclusions. Our analysis and results provide a core analysis on scaling, capacity and complexity of deep reinforcement learning.
• 中文摘要 从利用深度神经网络近似导致赢得最具挑战性游戏之一的状态-行动值函数，到算法的进步使得问题无需明确说明挑战规则即可解决问题，强化学习研究在过去十年中一直是科学进步的中心。本文聚焦于该研究进展的关键要素，并分析强化学习中的典型评估与设计范式。我们介绍了强化学习中尺度律的理论基础，并证明强化学习算法的渐近性能与数据体系之间并非单调关系。我们进行了大规模实验，结果表明，在规范设计和评估范式下的一系列强化学习研究结果导致了错误的结论。我们的分析和结果为深度强化学习的扩展性、容量和复杂性提供了核心分析。

DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment

DeepSearch-world：在可验证环境中为深度搜索代理进行自我蒸馏

• Authors: Xinyu Geng, Xuanhua He, Sixiang Chen, Yanjing Xiao, Fan Zhang, Shijue Huang, Haitao Mi, Zhenwen Liang, Tianqing Fang, Yi R. Fung
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.07820
• Pdf link: https://arxiv.org/pdf/2607.07820
• Abstract Training tool-use agents to improve from their own experience remains challenging, as supervised fine-tuning relies on fixed teacher-distilled trajectories, while sparse-reward reinforcement learning provides weak supervision for long-horizon interactions. We present DeepSearch-Evolve, a self-distillation framework for web agents built on DeepSearch-World, a deterministic and verifiable environment with reproducible search and page-reading tools. DeepSearch-World contains 420K multi-hop QA tasks constructed from entity-level random walks and supports key agentic cognitive behaviors useful for self-evolving, including progress verification, grounded reflection, and failure recovery. DeepSearch-Evolve iteratively performs trajectory generation, filtering, data mixing, and fine-tuning to train stronger agents. Without distillation from more capable models, DeepSearch-World-9B achieves competitive performance compared with open-source agents, reaching 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA, showing that verifiable environments enable scalable self-evolution for long-horizon web agents. We will release the environment, 420K training pool, validation set, model, and code to facilitate future research on self-improving deep search agents.
• 中文摘要 训练工具使用代理从自身经验中提升仍然具有挑战性，因为监督微调依赖于固定的教师提炼轨迹，而稀疏奖励强化学习则对长期互动提供了薄弱的监督。我们介绍DeepSearch-Evolve，这是一个基于DeepSearch-World的网络代理自我蒸馏框架，这是一个确定性且可验证的环境，具备可重复的搜索和阅读工具。DeepSearch-World 包含 420K 多跳 QA 任务，由实体级随机游走构建，支持对自我演化有用的关键智能认知行为，包括进展验证、扎根反思和失败恢复。DeepSearch-Evolve 通过迭代方式进行轨迹生成、过滤、数据混合和微调，以训练更强的代理。在不提炼更强大模型的情况下，DeepSearch-World-9B 在与开源代理相比实现了竞争性能，在 BrowseComp 上达到 31.2%，在 GAIA 上达到 61.5%，在 HotpotQA 上达到 93.4%，表明可验证环境能够实现长期网络代理的可扩展自我演进。我们将发布环境、420K训练池、验证集、模型和代码，以促进未来自我改进深度搜索代理的研究。

Physics-Guided Biomechanical Gait Adaptation for Humanoid Locomotion on Extreme Sloped Terrains

物理引导的生物力学步态适应，用于极端坡地的类人移动

• Authors: Xuanyu Chen, Mohan Liu, Dengchen Mei, Zhihao Gu, Haitian Zhang, Kaimin Mao, Haiyue Zhu, Shijun Yan, Lin Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.07830
• Pdf link: https://arxiv.org/pdf/2607.07830
• Abstract Model-free reinforcement learning has enabled impressive humanoid locomotion; however, control on steep slopes remains largely unexplored. Unlike flat or discrete terrains, sloped terrains impose a persistent gravitational bias that demands simultaneous stability and posture control. Consequently, under generic reward formulations, policies can converge to slow, conservative low-center-of-mass (CoM) crouched gaits. In this work, we propose a novel two-stage physics-guided framework, dubbed HumoSlope, dedicated to robust humanoid locomotion on diverse sloped terrains. Specifically, Stage I establishes a terrain-consistent balance prior by introducing a slope-adaptive Zero Moment Point (ZMP) regularizer evaluated directly on the local inclined support plane rather than a world-horizontal reference. To prevent the resulting policy from defaulting to a crouched posture, Stage II introduces the Biomechanical Slope Gait Adapter (BSGA). Utilizing extracted macroscopic terrain descriptors as privileged, training-only signals, BSGA dynamically gates soft reward priors to modulate CoM height and lower-limb coordination based on the estimated slope geometry -- encouraging hip-dominant uphill propulsion and knee-oriented downhill braking. Crucially, the deployed actor remains entirely proprioceptive, requiring no online exteroceptive sensing. Extensive Sim-to-Real experiments demonstrate that our framework effectively mitigates posture degeneration and enables blind, continuous traversal of outdoor grass slopes up to 62.7% ($32.1^\circ$), validating a physics-guided approach to challenging slope terrain adaptation.
• 中文摘要 无模型强化学习使得令人印象深刻的类人生物运动成为可能;然而，陡坡的控制区仍然大多未被探索。与平坦或离散地形不同，斜坡地形具有持续的重力偏置，要求同时保持稳定性和姿势控制。因此，在通用奖励表述下，政策可以趋向缓慢、保守的低质心（CoM）蹲伏步态。在本研究中，我们提出了一种新的两阶段物理指导框架，称为HumoSlope，致力于在多样坡度地形上实现稳健的人形运动。具体来说，第一阶段通过引入一个直接在局部倾斜支撑面而非世界水平参考上评估的坡度自适应零力矩点（ZMP）正则化器，建立了地形一致的先验平衡。为防止最终政策默认为蹲姿，第二阶段引入了生物力学坡道步态适配器（BSGA）。BSGA利用提取的宏观地形描述符作为特权的训练信号，动态门控软奖励先验，根据估计的坡度几何形态调节CoM高度和下肢协调——鼓励以髋部为主的上坡推进和膝盖侧下坡制动。关键是，部署的行为者完全保持本体感受，无需在线外感知。大量模拟到现实实验表明，我们的框架有效减轻姿势退化，并实现盲目连续穿越户外草坡，最高可达62.7%（约32.1^\circ$），验证了基于物理的坡地地形适应挑战方法。

Infinity-Parser2 Technical Report

Infinity-Parser2 技术报告

• Authors: Zuming Huang, Jun Huang, Kexuan Ren, Baode Wang, Weizhen Li, Jianming Feng, Yu Wang, Yichen Yao, Shijun Lin, Yige Tang, Cheng Peng, Weidi Xu, Wei Chu, Yinghui Xu, Yuan Qi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.07836
• Pdf link: https://arxiv.org/pdf/2607.07836
• Abstract We present Infinity-Parser2, a large multimodal model that couples a controllable data-synthesis pipeline with multi-task reinforcement learning for end-to-end document parsing, addressing the persistent scarcity of faithfully annotated parsing corpora. Our contributions are threefold. First, we build a scalable synthesis engine, pairing a controllable rendering framework with an iterative refinement loop, and use it to construct and open-source Infinity-Doc2-5M: a 5-million-sample bilingual (Chinese/English) corpus spanning diverse document types, annotated with element bounding boxes, canonical content forms (Markdown, HTML, LaTeX, SMILES, structured charts), and full-page reading order. Second, we introduce a verifiable, multi-task reward system that enables Joint Reinforcement Learning across eight co-trained objectives (document parsing, layout analysis, table parsing, math formula parsing, chart parsing, chemical formula parsing, document VQA, and general multimodal understanding), unifying perception, structure, and reasoning in a single optimization signal. Third, we release two variants under a shared architecture: Infinity-Parser2-Flash, optimized for low-latency inference with a $3.68\times$ throughput gain over Infinity-Parser-7B, and Infinity-Parser2-Pro, engineered for precision-critical settings. Infinity-Parser2-Pro reaches state-of-the-art 87.6% on olmOCR-Bench and 74.3% on ParseBench, surpassing DeepSeek-OCR-2, PaddleOCR-VL-1.5, and MinerU2.5, with strong generalization to charts, chemical formulas, and document VQA.
• 中文摘要 我们介绍Infinity-Parser2，这是一个大型多模态模型，将可控的数据综合流水线与多任务强化学习相结合，实现端到端的文档解析，解决了忠实注释解析语料库的持续稀缺问题。我们的贡献有三方面。首先，我们构建了一个可扩展的综合引擎，将可控渲染框架与迭代精炼循环结合，并用它构建并开源Infinity-Doc2-5M：一个包含500万个双语（中英）语料库，涵盖多种文档类型，标注元素边界框、规范内容表（Markdown、HTML、LaTeX、SMILES、结构化图表）和整页阅读顺序。其次，我们引入了可验证的多任务奖励系统，支持八个共同训练目标（文档解析、布局分析、表格解析、数学公式解析、图表解析、化学式解析、文档VQA及一般多模态理解）的联合强化学习，统一感知、结构和推理于单一优化信号中。第三，我们在共享架构下发布了两个变体：Infinity-Parser2-Flash，优化低延迟推理，吞吐量比Infinity-Parser-7B提升3.68美元，倍数倍;以及Infinity-Parser2-Pro，专为精度关键设置设计。Infinity-Parser2-Pro 在 olmOCR-Bench 上达到 87.6% 的先进性能，ParseBench 上的 74.3%，超过了 DeepSeek-OCR-2、PaddleOCR-VL-1.5 和 MinerU2.5，并且对图表、化学式和文档质量保证（VQA）具有强烈的推广能力。

When Does Continual Learning Require Learning

持续学习什么时候需要学习

• Authors: Anne Harrington, Nayan Saxena, Michael Murphy, Anastasia Borovykh, Zeyu Yun, Sridhar Kamath, Ara Eindra Kyi, Trevor Darrell, Jitendra Malik, Yutong Bai
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07847
• Pdf link: https://arxiv.org/pdf/2607.07847
• Abstract As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn? Today, the field largely frames this as a problem of context management and mitigating forgetting. We argue this framing is incomplete: continual learning is fundamentally about increasing model competence as the world changes. We disentangle this change along two axes -- space, where the model encounters new domains, and time, where the underlying data drifts under a fixed task. This framing lets us study continual learning under realistic conditions: new domains arrive over time, facts drift past their training cutoff, and agentic interactions accumulate state across episodes. To evaluate methods under this setting, we recast widely used LLM benchmarks as sequential problems and introduce a single mechanism-agnostic protocol that compares prompt-based methods (GEPA, ACE), supervised learning (SFT, SDFT), reinforcement learning (GRPO, SDPO), and context compression (Cartridges, In-place TTT). Prompt-based methods fit each new stage quickly but degrade on future tasks. Distillation-based methods accumulate knowledge stably but struggle to update outdated facts. Context compression improves efficiency without substantially improving the ability to learn new tasks. Online reinforcement learning adapts most effectively to knowledge updates but remains sensitive to noisy reward signals. Overall, our results suggest that continual learning is not a single capability: different patterns of environmental change require fundamentally different update behaviors, determining when adaptation must be learned inside model weights and when it can be achieved through external scaffolding. We hope that understanding where each method succeeds and fails will guide the design of stronger continual learning systems.
• 中文摘要 随着大型语言模型（LLM）能力的提升，接下来的问题是，我们如何让模型能够持续学习？如今，该领域主要将此视为情境管理和减轻遗忘的问题。我们认为这种框架是不完整的：持续学习本质上是随着世界变化而提升模型能力。我们将这种变化沿着两个轴线解开——空间，模型遇到新领域;时间，底层数据在固定任务下漂移。这种框架使我们能够在现实条件下研究持续学习：新领域随时间到来，事实超越训练截止点，能动互动在各剧集间积累状态。为在此背景下评估方法，我们将广泛使用的LLM基准测试重新定义为顺序问题，并引入了单一机制无关协议，比较基于提示的方法（GEPA、ACE）、监督学习（SFT、SDFT）、强化学习（GRPO、SDPO）和上下文压缩（Cartridges、原位TTT）。基于提示的方法能快速适应每个新阶段，但在后续任务中会退化。基于蒸馏的方法能够稳定积累知识，但难以更新过时的事实。上下文压缩提高了效率，但并未显著提升学习新任务的能力。在线强化学习最能有效适应知识更新，但对噪声奖励信号保持敏感。总体而言，我们的结果表明持续学习并非单一能力：不同的环境变化模式需要根本不同的更新行为，决定何时适应必须在模型权重范围内学习，何时可通过外部支架实现。我们希望通过了解每种方法的成功与失败，指导设计更强大的持续学习系统。

Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning

反馈操作正则化：实现离线代理对齐以实现模仿学习

• Authors: Benjamin Poole, Minwoo Lee
• Subjects: Subjects: Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07859
• Pdf link: https://arxiv.org/pdf/2607.07859
• Abstract Reinforcement learning (RL) research has increasingly shifted focus towards alignment, ensuring agents learn behaviors adhering to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches predominantly combine these signals using multi-stage pipelines designed for the contextual bandit framing of language generation. Yet little work explores how these complementary inputs can serve as a richer, interconnected signal for single-stage offline training in fully sequential decision-making environments. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments to be a principled testbed for alignment evaluation, demonstrating improved aptitude and up to a 98\% reduction in misalignment across a range of imitation learning algorithms. FMR remains robust in limited data regimes, even when learning from scarce aligned and uninformative noisy demonstrations.
• 中文摘要 强化学习（RL）研究越来越多地将重点转向对齐，确保智能体学习符合人类价值观的行为。虽然人工演示和反馈对对齐至关重要，但现有方法主要通过多阶段管道将这些信号结合起来，这些管道设计用于语言生成的情境盗垒框架。然而，研究很少探讨这些互补输入如何作为单阶段离线训练中更丰富、互联的信号，适用于完全顺序决策环境中的训练。我们提出了反馈操作正则化（FMR），这是一种算法无关的方法，利用评估反馈作为纠正信号，以改善模仿学习策略的对齐。我们将安全体育馆环境改造为有原则的比对测试平台，展示了能力提升，并在多种模仿学习算法中错位减少高达98%。即使在从稀缺且缺乏信息量且噪声的演示中学习，FMR在有限的数据范围内依然稳健。

Reachability-Preserving Bellman Operator for the Discounted Reach-Cost Value Function: Uniting Hamilton-Jacobi Reachability and Reinforcement Learning

折扣覆盖成本值函数的保持可达性贝尔曼算子：结合汉密尔顿-雅各比可达性与强化学习

• Authors: Isabelle El-Hajj, Prashant Solanki, Jasper van Beers, Coen de Visser, Erik-Jan van Kampen
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.07893
• Pdf link: https://arxiv.org/pdf/2607.07893
• Abstract Hamilton-Jacobi (HJ) reachability provides rigorous safety and reachability guarantees for continuous-time dynamical systems, but its numerical solution suffers from the curse of dimensionality. Deep reinforcement learning (DRL), by contrast, offers scalable sample-based methods. However, RL is typically built around additive cumulative rewards; whereas, reachability objectives are inherently non-additive. This mismatch makes a direct bridge between HJ reachability and RL nontrivial. Recent discounted formulations have either introduced contraction by altering the original reachability semantics, or preserved exact semantics on the HJ side without a corresponding Bellman fixed-point characterization. In this paper, we close this gap by building on a semantics-preserving discounted reach-based value function and deriving a non-additive Bellman operator whose unique fixed point exactly matches the value function in the HJ formulation. We prove that discounting makes this operator contractive, yielding existence, uniqueness, and convergence of value iteration. Furthermore, we establish the equivalence between the HJ and Bellman characterizations, and show that RL can be interpreted as a sample-based approximation scheme for the same fixed-point equation. This yields a principled and semantically exact connection between HJ reachability and RL, enabling learning-based methods to approximate reachability value functions while preserving their safety-critical meaning. As a result, the proposed framework opens the door to scalable, data-driven computation of reachable sets and safety certificates in high-dimensional systems. Numerical experiments demonstrate close agreement with HJ solutions, confirm preservation of reachability semantics via alignment of zero level sets, and support the interpretation of reinforcement learning as a sample-based solver of the proposed Bellman operator.
• 中文摘要 Hamilton-Jacobi（HJ）可达性为连续时间动力系统提供了严格的安全性和可达性保证，但其数值解受限于维度问题。相比之下，深度强化学习（DRL）提供了可扩展的基于样本的方法。然而，强化学习通常围绕累积奖励构建;而可达性目标本质上是非加法的。这种不匹配使得HJ可达性和RL之间的直接桥梁变得不易。最近的折现表述要么通过改变原始可达性语义引入收缩，要么在HJ侧保留了精确语义，但没有相应的贝尔曼不动点刻画。本文通过建立一个保持语义的折现基于reach的价值函数，并推导出一个非加法的Bellman算子，其唯一不动点与HJ表述中的值函数完全匹配，从而弥补了这一空白。我们证明了贴现使该算符是收缩的，从而产生存在性、唯一性和价值迭代收敛性。此外，我们建立了HJ与Bellman刻画之间的等价性，并证明了强化学习可以解释为同一不动点方程的基于样本的近似方案。这为HJ可达性与强化学习之间建立了原则性和语义精确的联系，使基于学习的方法能够近似可达性值函数，同时保持其安全关键意义。因此，该框架为高维系统中可达集和安全证书的可扩展、数据驱动计算打开了大门。数值实验与HJ解高度一致，通过零能级集的对齐确认了可达性语义的保持，并支持将强化学习解释为基于样本的Bellman算子求解器。

When Implausible Tokens Get Reinforced: Tail-Aware Credit Calibration for LLM Reinforcement Learning

当不合逻辑的代币被强化：尾部感知信用校准用于大型语言模型强化学习

• Authors: Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang, Hoang Anh Duy Le, Zhaozhuo Xu, Guanchu Wang, Vladimir Braverman
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07976
• Pdf link: https://arxiv.org/pdf/2607.07976
• Abstract Reinforcement learning (RL) has achieved remarkable success in enhancing the reasoning capabilities of large language models (LLMs). However, widely used critic-free RL methods rely on uniform credit assignment, broadcasting the same advantage to all tokens regardless of their differences. We identify a critical failure mode of this design, which we refer to as Positive-Credit Contamination: low-probability tail tokens that are contextually erroneous receive identical positive credit to plausible ones within the same trajectory, resulting in the indiscriminate reinforcement of flawed reasoning behavior. To mitigate this issue, we propose Tail-Aware Credit calibratiOn (TACO), a method that calibrates uniform credit assignment to suppress undesirable positive updates. TACO first computes a tail-risk score that incorporates the local generation context to assess each token's risk of falling into the unreliable tail, distinguishing unexpected rarity from uncertainty-driven exploration. TACO then uses this score to tune positive credit for risky tokens without removing their gradients entirely, so that recurring useful rare patterns can accumulate reinforcement while incidental noise is progressively dampened. Experimental results across three LLMs and eight benchmarks show that TACO consistently outperforms GRPO-style baselines. Notably, TACO improves training stability, supporting sustained performance gains in long-horizon RL. The source code is available at: this https URL.
• 中文摘要 强化学习（RL）在提升大型语言模型（LLMs）推理能力方面取得了显著成功。然而，广泛使用的无批评强化学习方法依赖于统一的信用分配，无论代币有何差异，都能向所有代币传递相同的优势。我们识别出该设计的一个关键失效模式，称为正向信用污染：低概率的尾部标记在上下文中错误，获得与同一轨迹内合理尾迹相同的正向认可，导致错误推理行为被无差别强化。为缓解此问题，我们提出了尾部感知信用calibratiOn（TACO）的方法，该方法校准统一信用分配以抑制不良的正向更新。TACO首先计算一个尾风险评分，结合本地生成情境，评估每个代币落入不可靠尾部的风险，区分意外稀有度与不确定性驱动的探索。TACO随后利用该评分调优高风险代币的正信用，而不完全去除其梯度，使得重复出现的有用稀有模式积累强化，同时偶发噪声逐渐被抑制。在三个大型语言模型和八个基准测试中的实验结果显示，TACO始终优于GRPO风格的基线。值得注意的是，TACO提升了训练稳定性，支持长期强化学习的持续表现提升。源代码可在以下 https URL 获取。

Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generator

幻觉自我游戏：通过进化生成器自举强化探测器

• Authors: Shiping Yang, Shining Liang, Weihao Liu, Wenbiao Ding, Linjun Shou, Lu Cheng, Angel X. Chang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.07993
• Pdf link: https://arxiv.org/pdf/2607.07993
• Abstract Identifying faithfulness hallucinations in LLM-generated outputs remains challenging due to the scarcity of high-quality annotated data. Recent work relies on advanced LLMs to synthesize training data, including rationales, labels, and hallucinated claims. However, these methods treat the generator as a static component, limiting iterative improvement of the detector. To address this limitation, we introduce Hallucination Self-Play (HSP), a novel framework that enables the detector to bootstrap with an evolved generator. HSP involves two roles initialized from the same base model, a detector that assesses the faithfulness of model outputs, and a generator that produces increasingly hard-to-detect hallucinated responses. Specifically, the detector is first fine-tuned on human-labeled data and then employed as a reward model to train the generator via reinforcement learning from AI feedback (RLAIF). In turn, the evolved generator synthesizes hallucination data to further optimize the detector through rule-based reinforcement learning. Experiments on RAGTruth benchmark and two model families demonstrate that the proposed framework can progressively enhance a small LLM to match or even outperform advanced LLMs without external supervision. Our code is available at this https URL .
• 中文摘要 由于高质量注释数据稀缺，识别LLM生成输出中的忠实幻觉仍然具有挑战性。近期工作依赖高级大型语言模型（LLM）来综合训练数据，包括理由、标签和幻觉主张。然而，这些方法将生成器视为静态组件，限制了探测器的迭代改进。为解决这一限制，我们引入了幻觉自我游戏（Hallucination Self-Play，HSP）这一新框架，使探测器能够通过进化后的生成器自启动。HSP包含两个角色，初始化于同一基础模型：一个检测器评估模型输出的忠实性，另一个生成器产生越来越难以检测的幻觉反应。具体来说，检测器首先基于人工标记的数据进行微调，然后作为奖励模型，通过AI反馈强化学习（RLAIF）训练生成器。进化后的生成器则综合幻觉数据，通过基于规则的强化学习进一步优化探测器。在RAGTruth基准测试和两系列模型上的实验表明，所提出的框架能够逐步提升小型LLM，使其在无需外部监督的情况下，与甚至超越高级LLM匹敌甚至超越。我们的代码可在此 https URL 获取。

MORES: Mobile Reasoning-as-a-Service via Distributed LLM Inference-Time Scaling

MORES：通过分布式LLM推理时间尺度实现的移动推理即服务

• Authors: Guanchen Liu, Hongyang Du, Kaibin Huang
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2607.08116
• Pdf link: https://arxiv.org/pdf/2607.08116
• Abstract Inference-time scaling has emerged as an effective approach for enhancing the capabilities of Large Language Models (LLMs), addressing the growing demand for stronger reasoning without increasing model size. This novel form of LLM scaling comprises two representative approaches: explicit reasoning, which generates intermediate chain-of-thought tokens during an explicit thinking phase, and implicit reasoning, which iteratively updates hidden states in the latent space without producing explicit outputs. Despite their effectiveness, both paradigms incur substantial computational and memory overhead, raising challenges for deployment on resource-constrained edge devices. To address these issues, we propose a Mobile Reasoning-as-aService (MORES) framework that treats reasoning as a computational service accessible to edge devices over wireless networks. Focusing on implicit reasoning, we leverage its recursive structure to partition hiddenstate updates between edge devices and servers, enabling cooperative inference that allows devices to access additional cloud computation on demand. To optimize long-term performance, we formulate a joint computation and communication scheduling problem and solve it using a semantic Mixture-of-Experts (MoE)-based Deep Reinforcement Learning (DRL) algorithm to address heterogeneity in wireless conditions and task demands. The agent adaptively allocates resources by adjusting the number of recurrent steps and the transmission pruning rate, while a semantic router enables high-speed gating for real-time expert selection. Experimental results show that the proposed method achieves an approximately 18% improvement in system throughput over the baseline Soft Actor-Critic (SAC) algorithm. Our code is available at this https URL.
• 中文摘要 推理时间尺度已成为提升大型语言模型（LLMs）能力的有效方法，能够满足对强推理需求的增长需求，同时不增加模型规模。这种新型的大型语言模型扩展包括两种代表性方法：显式推理，在显性思考阶段生成中间的思维链代币;以及隐性推理，迭代更新潜在空间中的隐藏状态而不产生显式输出。尽管有效，这两种模式都带来了巨大的计算和内存开销，给资源有限的边缘设备部署带来了挑战。为解决这些问题，我们提出了一个移动推理即服务（MORES）框架，将推理视为一种可通过无线网络边缘设备访问的计算服务。我们专注于隐式推理，利用其递归结构将隐藏状态更新划分到边缘设备和服务器之间，实现协作推理，使设备能够按需访问额外的云计算。为了优化长期性能，我们提出了一个联合计算与通信调度问题，并利用基于语义专家混合（MoE）的深度强化学习（DRL）算法求解，以解决无线条件和任务需求的异质性。代理通过调整重复步骤数和传输剪枝率来自适应分配资源，而语义路由器则实现了高速门控，实现了专家的实时选择。实验结果显示，所提方法相比基础软演员-批判者（SAC）算法，在系统吞吐量上提升了约18%。我们的代码可在此 https URL 访问。

Securing Autonomous Vehicle Systems via Twin-Aware Federated Reinforcement Learning

通过双重感知的联合强化学习保障自动驾驶车辆系统的安全

• Authors: Zifan Zhang, Minghong Fang, Dianwei Chen, Zhuqing Liu, Prashant Khanduri, Xianfeng Yang, Anupam Das, Yuchen Liu
• Subjects: Subjects: Cryptography and Security (cs.CR); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2607.08137
• Pdf link: https://arxiv.org/pdf/2607.08137
• Abstract Federated reinforcement learning (FRL) is crucial for enabling collaborative learning across multiple agents without sharing raw data, thereby enhancing privacy and scalability in the decision-making process within dynamic vehicular environments. However, poisoning attacks pose a significant threat to the security and reliability of FRL-based systems, particularly in safety-critical autonomous driving, where this vulnerability remains largely unexplored. These attacks can compromise the global control model by subtly injecting malicious system parameters, leading to potential hazards. To counter these challenges, we present \alg (\underline{Sec}ure \underline{A}ggregation with \underline{p}oisoning-\underline{p}revention and historical reinforcement) as a defensive framework aimed at enhancing the robustness of FRL systems designed for safety-critical driving scenarios. \alg strategically integrates digital twins for rehearsal-based learning and leverages historical aggregated model parameters along with a selected central gradient to ensure that only benign data is aggregated, effectively mitigating the influence of malicious agents. Theoretical guarantees are provided for the convergence performance of \alg in the presence of poisoning attacks. We also validate the effectiveness of \alg using developed digital twins that model realistic highway environments to evaluate the control of autonomous vehicles under adversarial conditions.
• 中文摘要 联邦强化学习（FRL）对于实现多智能体间的协作学习至关重要，无需共享原始数据，从而提升动态车辆环境中决策过程的隐私性和可扩展性。然而，毒害攻击对基于FRL系统的安全性和可靠性构成重大威胁，尤其是在安全关键的自动驾驶领域，这一漏洞在很大程度上尚未被探索。这些攻击可能通过微妙地注入恶意系统参数，从而破坏全局控制模型，从而引发潜在危害。为应对这些挑战，我们提出了\alg（\下线{Sec}ure \下线{A}集群，结合\下线{p}操作-\下线{p}修复和历史强化）作为防御框架，旨在提升设计用于安全关键驾驶场景的FRL系统的稳健性。\ALG战略性地整合数字孪生以实现基于排练的学习，利用历史汇总的模型参数及选定的中央梯度，确保仅汇总良性数据，有效减轻恶意行为的影响。理论上为\alg在存在毒攻击下的收敛性能提供了保证。我们还通过开发的数字孪生模型验证了\alg的有效性，这些数字孪生模拟了真实的高速公路环境，以评估在对抗条件下自动驾驶车辆的控制能力。

MuScriptor: An Open Model for Multi-Instrument Music Transcription

MuScriptor：多乐器音乐转录的开放模型

• Authors: Simon Rouard, Michael Krause, Axel Roebel, Carl-Johann Simon-Gabriel, Alexandre Défossez
• Subjects: Subjects: Sound (cs.SD); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.08168
• Pdf link: https://arxiv.org/pdf/2607.08168
• Abstract Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
• 中文摘要 现有的自动音乐转录方法通常仅限于单乐器录音，或者在复杂的真实音乐混音中失效。尽管以往工作利用了合成训练数据，但所得模型推广能力较差，导致在真实的多仪器环境中输出大多无法使用。本研究分析合成数据在预训练中的有效性，同时结合真实音乐音频的微调和强化学习后的后期训练。我们进一步引入对器械存在的条件条件，以定制转录结果。最后，我们发布了MuScriptor，这是一种开放权重的多乐器音乐转录模型，适用于来自多种音乐流派的真实音乐录音。

ASMR: Agentic Schema Generation for Ship Maintenance Report Writing

ASMR：船舶维护报告写作的智能模式生成

• Authors: Sohrab Namazi Nia, Amogh Dalal, Ning Sa, Peter Ly, Marti Zentmaier, Tomek Strzalkowski, Jay Miller, Rishi Singh, Senjuti Basu Roy
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.08177
• Pdf link: https://arxiv.org/pdf/2607.08177
• Abstract In this paper, we study the automatic schema generation problem: given a collection of historical ship maintenance and operational reports across multiple form categories, automatically discover compact and informative schemas that capture the essential information requirements of each report type. To address this challenge, we propose ASMR, a modular agentic framework consisting of two specialized agents. A Field Generation Agent extracts semantic concepts from historical narratives and generates candidate schema fields through adaptive multi-granularity clustering, while a Structural Optimizer Agent employs reinforcement learning to identify compact, informative, and non-redundant schema representations. The resulting schemas can guide report authors toward producing more complete, consistent, and actionable reports. Preliminary results demonstrate the promise of the proposed approach and highlight several open research challenges at the intersection of data management, agentic AI, and human-centered AI.
• 中文摘要 本文研究自动模式生成问题：给定涵盖多种表格类别的历史船舶维护和运营报告集合，自动发现紧凑且信息丰富的模式，捕捉每种报告类型的基本信息需求。为应对这一挑战，我们提出了ASMR，一个由两个专业代理组成的模块化代理框架。字段生成代理通过自适应多粒度聚类从历史叙事中提取语义概念，生成候选模式字段，而结构优化代理则通过强化学习识别紧凑、信息丰富且非冗余的模式表示。由此产生的结构可以指导报告作者产出更完整、一致且可操作的报告。初步结果显示了该方法的前景，并凸显了数据管理、代理人工智能和以人为本人工智能交叉点面临的若干未解研究挑战。

Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs

通过多模态大型语言模型对策略进行可视化检查的开放式多代理自课程

• Authors: Lorenzo Pantè, Andrea Fanti, Roberto Capobianco
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.08193
• Pdf link: https://arxiv.org/pdf/2607.08193
• Abstract Open-ended curricula in Reinforcement Learning (RL) aim to train generally-capable agents by identifying tasks that facilitate learning increasingly complex skills. A major challenge when designing such curricula is assessing task difficulty relative to the agent's current learning progress. While previous work has explored using scalar task scores or textual summaries of the agent's behavior, here we study a different approach: directly inspecting policy behavior via recorded episode videos. We introduce a simple yet effective instantiation of this approach which leverages a Video Language Model (VLM) to both process these videos and provide curriculum recommendations, which we call Visual Inspection of Policies (VIP). Since videos can naturally contain any number of controllable agents, we empirically study VIP on the StarCraft Multi-Agent Challenge (SMAC). We show that even with a lightweight and openly accessible VLM (VideoLLaMa2-7B), VIP can use policy videos to generate more effective curricula than both its text-only ablation and methods that rely on scalar task scores.
• 中文摘要 强化学习（RL）中的开放式课程旨在通过识别有助于学习日益复杂技能的任务，培训具备普遍能力的代理。设计此类课程时的一个主要挑战是评估任务难度相对于主体当前学习进展的情况。虽然以往研究尝试使用标量任务评分或代理行为的文本摘要，但这里我们研究另一种方法：通过录制的剧集视频直接检查策略行为。我们引入了一种简单但有效的方法实例，利用视频语言模型（VLM）来处理这些视频并提供课程建议，我们称之为策略可视化检查（VIP）。由于视频中自然可能包含任意数量的可控代理，我们通过实证研究了《星际争霸》多代理挑战赛（SMAC）上的VIP。我们证明，即使使用轻量且开放的VLM（VideoLLaMa2-7B），VIP仍能利用政策视频生成比纯文本消融和依赖标量任务评分的方法更有效的课程。

Diarization-Guided Qwen-ASR Adaptation for Multilingual Two-Speaker Conversational Speech

日音化引导的Qwen-ASR多语双语会话语言适应

• Authors: Hao Wu, RongQi Han, Zhen Wang, Wei Liang, Wei Xu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.08208
• Pdf link: https://arxiv.org/pdf/2607.08208
• Abstract This paper describes our self-designed system for Task 1 of the MLC-SLM 2026 Challenge for multilingual two-speaker conversational speech. The system combines a modular speaker diarization front end with a challenge-adapted Qwen3-ASR-1.7B recognizer. The diarization front end performs voice activity detection, subsegment generation, CAMPPlus speaker embedding extraction, two-speaker spectral clustering, and RTTM-based audio segmentation. The resulting speaker-attributed segments are grouped by language or region and decoded by the adapted ASR model. For ASR adaptation, we first perform supervised full fine-tuning on the official training data, then apply LoRA fine-tuning with synthetic speech generated by a three-pipeline TTS-based synthetic speech augmentation framework, and finally refine the model using GRPO reinforcement learning with rewards based on WER/CER and penalties for hallucination, repetition, and length deviation. On the official development set, the full system achieves an average tcpMER of 23.70, reducing the error rate by 6.83 absolute points relative to the released Qwen-ASR-1.7B performance. On the final evaluation set, the system achieves an average tcpMER of 17.97. Ablation results show that supervised fine-tuning provides the largest gain, while synthetic-speech LoRA adaptation and reinforcement learning further improve robustness.
• 中文摘要 本文介绍了我们为MLC-SLM 2026挑战赛中多语言双语会话语音任务1自设计的系统。该系统结合了模块化扬声器传输前端和挑战性适配的Qwen3-ASR-1.7B识别器。diarization前端支持语音活动检测、子片段生成、CAMPPlus扬声器嵌入提取、双扬声器频谱聚类以及基于RTTM的音频分割。由此产生的说话者属性片段按语言或地区分组，并由适应后的ASR模型解码。对于ASR适配，我们首先对官方训练数据进行监督式全微调，然后对由三流水线TTS合成语音增强框架生成的合成语音进行LoRA微调，最后利用GRPO强化学习对模型进行细化，奖励基于WER/CER以及对幻觉、重复和长度偏差的惩罚。在官方开发套件中，完整系统的平均tcpMER为23.70，相较于已发布的Qwen-ASR-1.7B性能，错误率降低了6.83个绝对点。在最终评估集中，系统平均 tcpMER 为 17.97。消融结果显示，监督微调带来最大收益，而合成语音LoRA适应和强化学习进一步提升了鲁棒性。

Compete Then Collaborate: Frontier AI Teachers Build a Verifiable Curriculum to Improve a Coding Student Beyond Imitation

先竞争后协作：前沿人工智能教师构建可验证的课程，提升编程学生的水平，超越模仿

• Authors: Miseong Shawn Kim
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.08255
• Pdf link: https://arxiv.org/pdf/2607.08255
• Abstract Large language models increasingly serve as teachers generating training data for smaller students. Prior multi-teacher knowledge distillation methods merge outputs without determining which frontier model teaches best, often relying on an LLM judge biased toward its own outputs. We introduce a compete-then-collaborate framework where four frontier AI teachers (Claude, Codex-GPT, Grok, Gemini) are ranked head-to-head by an execution-based judge (unit tests and stdin-stdout checks) with fairness controls, and then collaborate to build a verifiable curriculum for a student (Qwen2.5-Coder). We report three findings. (1) Under execution verification, all teachers solve standard problems near-perfectly after self-correction (99-100%) due to a saturation effect, but harder competition problems separate them (Gemini 77% > Claude 69% = Codex 69% > Grok 50%); however, the robust student-side results do not depend on teacher ranking. (2) Imitation (SFT) on verified solutions does not improve, and can degrade, an already-competent student at 7B and 32B (e.g., from 76.7% to 72.7% on MBPP-test, and 5.9% to 2.9% on competition problems). (3) Using the same collaborative curriculum as a reinforcement learning with verifiable rewards (RLVR) environment improves the student (from 5.9% to 8.8% peak on competition problems, a +49% relative gain), reversing SFT's direction. The value of AI-teacher collaboration lies not in pooling answers to imitate, but in jointly constructing a verifiable environment where the student learns by doing. We release a reproducible on-prem pipeline (NVIDIA GB10) with framework patches for running GRPO on a bleeding-edge stack.
• 中文摘要 大型语言模型越来越多地作为教师，为较小的学生生成培训数据。以往多教师知识提炼方法合并输出时，往往依赖偏向自身输出的LLM评审。我们引入了一个竞争后协作框架，四位前沿AI教师（Claude、Codex-GPT、Grok、Gemini）由基于执行的评审（单元测试和标准-标准检定）与公平性对比，然后合作构建可验证的学生课程（Qwen2.5-Coder）。我们报告三项发现。（1）在执行验证中，所有教师在自我纠正后几乎完美地解决标准问题（99-100%），因为存在饱和效应，但较难的竞争问题则将他们分隔开来（Gemini 77% > Claude 69% = Codex 69% > Grok 50%）;然而，学生方面强健的结果并不依赖于教师排名。（2）对已验证解答的模仿（SFT）不会提升，反而可能降低7B和32B阶段已经具备能力的学生（例如MBPP测试从76.7%降至72.7%，竞赛题从5.9%降至2.9%）。（3）将同一协作课程作为可验证奖励强化学习（RLVR）环境，学生在竞争问题的峰值中提升至8.8%，相对提升+49%，逆转SFT的方向。AI与教师合作的价值不在于集合答案模仿，而在于共同构建一个可验证的环境，让学生通过实践学习。我们发布了可重复的本地流水线（NVIDIA GB10），并附带框架补丁，用于在尖端技术栈上运行GRPO。

Deep Reinforcement Learning-Empowered Wireless Sensor Networking for 6G Closed-Loop Controls

深度强化学习——基于6G闭环控制的无线传感器网络

• Authors: Chengleyang Lei, Wei Feng, Yunfei Chen, Yongxu Zhu, Ning Ge, Shi Jin
• Subjects: Subjects: Information Theory (cs.IT); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.08272
• Pdf link: https://arxiv.org/pdf/2607.08272
• Abstract Robots are increasingly deployed in remote or hazardous areas for mission-critical control tasks. Due to their limited individual capabilities, they have to rely on other field sensors to obtain the state information of targets, and also a dedicated edge information hub (EIH) to enable information exchange, sensing data analysis and control command generation. Such configuration follows a sensing-communication-computing-control (SC3) closed loop. To optimize the whole closed-loop performance, this paper minimizes the linear quadratic regulator (LQR) control cost by designing the sensor-to-EIH bandwidth allocation. Specifically, we first model the distortion noise caused by limited communication data rate based on the mutual information theory. Next, under the control policy based on the Kalman filter and LQR controller, we formulate the control process as a partially observable Markov decision process (POMDP), and develop a deep reinforcement learning (DRL)-based sensor-to-EIH bandwidth allocation scheme. The proximal policy optimization (PPO) algorithm is utilized to train the DRL agent. Simulation results are provided to show the superiority of the proposed DRL-based scheme.
• 中文摘要 机器人越来越多地部署在偏远或危险地区，执行关键任务控制任务。由于其单独能力有限，必须依赖其他现场传感器获取目标状态信息，同时还需配备专用边缘信息中心（EIH）以实现信息交换、传感数据分析和控制指令生成。这种配置遵循一个传感-通信-计算-控制（SC3）闭环。为了优化整个闭环性能，本文通过设计传感器与EIH带宽分配，最大限度地降低了线性二次稳压器（LQR）控制成本。具体来说，我们首先基于互信息理论建模了由有限通信数据率引起的失真噪声。接下来，基于卡尔曼滤波器和LQR控制器的控制策略，我们将控制过程表述为部分可观测马尔可夫决策过程（POMDP），并开发基于深度强化学习（DRL）的传感器到EIH带宽分配方案。近端策略优化（PPO）算法被用于训练DRL代理。提供模拟结果以证明基于DRL方案的优越性。

Spectral Analysis of Dueling Q-Learning

对抗Q学习的谱分析

• Authors: Donghwan Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.08340
• Pdf link: https://arxiv.org/pdf/2607.08340
• Abstract Q-learning is a fundamental algorithm in reinforcement learning (RL) for solving discounted Markov decision processes (MDPs) when the transition kernel is unknown. The deep Q-network (DQN) extends Q-learning by using a deep neural network for Q-function approximation, which makes Q-learning applicable to more practical high-dimensional problems. Dueling Q-learning decomposes the Q-function into a value function and an advantage function and learns the two components jointly, which can improve learning efficiency. However, the theoretical understanding of dueling Q-learning is still limited. Recent work has initiated an analysis of tabular dueling Q-learning, but existing guarantees focus on a regularized formulation and leave the pure tabular update less completely understood. This paper strengthens that line of analysis by adding a direct interpretation of the centered tabular decomposition and by establishing convergence guarantees for the unregularized, unprojected constant step-size recursion. In particular, we derive an exact switching linear system representation for deterministic dueling Q-learning and a finite-time error bound in expectation for the sampled stochastic version. The analysis clarifies how the value and advantage updates act as different gains on the action-common (value function) and action-differential (advantage function) components of the Q-function.
• 中文摘要 Q-学习是强化学习（RL）中用于解决转移核未知时的折现马尔可夫决策过程（MDP）的基础算法。深度Q网络（DQN）通过使用深度神经网络进行Q函数近似，扩展了Q学习，使Q学习适用于更实用的高维问题。Q-学习将Q-函数分解为价值函数和优势函数，并共同学习这两个组成部分，从而提高学习效率。然而，对决Q学习的理论理解仍然有限。近期工作启动了对表对答Q学习的分析，但现有保证关注正则化表述，而纯表更新对其理解尚不完全。本文通过直接解释中心表分解，并建立了非正则化、未投影的恒定步长递归的收敛保证，进一步加强了这一分析方向。特别地，我们推导出确定性对抗Q学习的精确切换线性系统表示，以及采样随机版本期望的有限时间误差界限。分析阐明了价值和优势更新如何作为Q函数中作用-共同（价值函数）和动作-差分（优势函数）成分的不同增益。

When Synthetic Speech Is All You Have: Better Call GRPO

当合成语音是你唯一拥有的语言时：最好联系GRPO

• Authors: Shashi Kumar, Yanis Labrak, Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello, Kadri Hacioğlu, Petr Motlicek, Andreas Stolcke
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.08409
• Pdf link: https://arxiv.org/pdf/2607.08409
• Abstract LLM-based ASR adapted to regulated domains such as banking is bottlenecked by privacy: real speech is costly and legally constrained to collect, making synthetic text-to-speech (TTS) an attractive substitute. Yet synthetic speech stays acoustically mismatched with real recordings, and work on this gap has stayed within supervised fine-tuning (SFT). We instead turn to reinforcement learning, and show that Group Relative Policy Optimization (GRPO) extracts far more from the same synthetic speech than SFT. Synthetic-only adaptation of the model with GRPO, a critic-free method rewarding low-WER hypotheses, reduces WER by 40\% relative to SFT (36.71\%$\to$22.09\%), and an SFT-then-GRPO combination pushes this further to 45\%. We trace the gain to behavior rather than representation: GRPO reduces insertion errors by improving stopping calibration and speech-to-text alignment by better anchoring attention to audio, leaving early-layer representations intact. When synthetic speech is the main resource, reinforcement learning should be preferred over supervised fine-tuning.
• 中文摘要 基于LLM的ASR适用于银行等受监管领域，但因隐私问题而成为瓶颈：真实语音收集成本高昂且法律上受限，使合成文本转语音（TTS）成为有吸引力的替代品。然而，合成语音在声学上仍与真实录音不匹配，这一空白的研究一直处于监督微调（SFT）范围内。我们转而转向强化学习，展示了群体相对策略优化（GRPO）从相同的合成语音中提取的远多于SFT。仅合成的模型采用GRPO方法（一种无批评的低WER假设）相较SFT降低40%（36.71%$至22.09\%），SFT加GRPO组合则进一步推高至45%。我们将增益追溯到行为而非表征：GRPO通过更好地锚定对音频的注意力，改善停止校准和语音转文本对齐，减少插入误差，保持早期层表示完整。当合成语音作为主要资源时，强化学习应优先于监督微调。

ADORN: Adaptive Drift handling for Open RAN using Reinforcement Learning

ADORN：基于强化学习的开放RAN自适应漂移处理

• Authors: Ashit Kumar Subudhi, Bhargav Chirumamilla, Shubham Vaishnav, Mduduzi C. Hlophe, Praveen Kumar Donta, Andrea Fumagalli, Venkateswarlu Gudepu, Koteswararao Kondepu
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.08443
• Pdf link: https://arxiv.org/pdf/2607.08443
• Abstract Dynamic traffic variations in Open Radio Access Networks (O-RAN) lead to drift, which degrades the performance of Artificial Intelligence/Machine Learning (AI/ML) models. Traditional retraining approaches maintain forecasting accuracy but incur high computational cost and may lead to violations of Service Level Agreements (SLAs). This work proposes a Q-learning-based adaptive retraining approach that formulates the retraining decision as a Markov Decision Process (MDP), where a Reinforcement Learning (RL) agent learns a policy that balances forecasting accuracy and retraining cost. The proposed approach incorporates a multi-expert Long Short-Term Memory (LSTM) ensemble to mitigate catastrophic forgetting and improve robustness across diverse traffic conditions. Experimental results show that the proposed approach effectively reduces retraining overhead compared to greedy and random baselines, while maintaining system performance within predefined limits.
• 中文摘要 开放无线接入网（O-RAN）中的动态流量变化导致漂移，从而降低人工智能/机器学习（AI/ML）模型的性能。传统的再训练方法保持预测准确性，但计算成本高昂，可能导致服务水平协议（SLA）的违反。本研究提出了一种基于Q学习的自适应再训练方法，将再训练决策表述为马尔可夫决策过程（MDP），强化学习（RL）代理学习一个平衡预测准确性和再训练成本的策略。该方法结合了多专家的长短期记忆（LSTM）集合，以减少灾难性遗忘并提升在不同交通条件下的稳健性。实验结果表明，所提方法相较于贪婪和随机基线有效减少了重训练开销，同时将系统性能保持在预设的限制内。

Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing

认知结构多模态代理，用于多模态理解、生成与编辑

• Authors: Feng Wang, Canmiao Fu, Zhipeng Huang, Chen Li, Jing Lyu, Ge Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.08497
• Pdf link: https://arxiv.org/pdf/2607.08497
• Abstract Recent unified multimodal models show a single architecture can jointly perform vision/language understanding and image generation/editing. However, they repeatedly feed all historical visual and textual inputs into a shared context window, limiting long-horizon multimodal dialogue due to visual token explosion and unreliable cross-turn referencing. We propose a Cognitive-structured Multimodal Agent that externalizes visual information into an Episodic Visual Memory and selectively reactivates relevant episodes during reasoning. The agent consists of a Perceptual Abstraction Engine for structured visual abstraction, a Cognitive Retrieval Engine for cross-turn memory retrieval, and a Multimodal Executive Controller for autonomous task inference and action planning. To address the lack of turn-level retrieval supervision in existing datasets, we develop a Unified Scenario Engine that programmatically generates structured multi-turn conversations with fine-grained retrieval annotations, enabling reinforcement learning to optimize abstraction and retrieval policies. We also construct a long-horizon visual-dialogue benchmark stratified by difficulty to evaluate episodic visual recall. Our 8B agent achieves 91.4% retrieval accuracy over 20-turn sessions, surpassing 32B baselines by +8.2% while nearly halving per-turn inference time (23.1s -> 12.7s). We further present the Cognitive-structured Multimodal Agent Harness (CMA-Harness), a tool-augmented deployment of the same cognitive structure integrating persistent multimodal memory, web access, image generation/editing/composition tools, and OpenAI-compatible serving. Structured memory and modular decision-making offer a more scalable, efficient paradigm for long-horizon multimodal agents than monolithic parameter scaling. Code: this https URL ; Project page: this https URL
• 中文摘要 最近的统一多模态模型表明，单一架构可以共同执行视觉/语言理解和图像生成/编辑。然而，它们反复将所有历史视觉和文本输入到共享上下文窗口，由于视觉符号爆炸和不可靠的交叉引用，限制了长视野多模态对话。我们提出了一种认知结构多模态代理，能够将视觉信息外化为情节视觉记忆，并在推理过程中选择性地重新激活相关发作。该智能体由一个用于结构化视觉抽象的感知抽象引擎、用于跨回合记忆检索的认知检索引擎，以及一个用于自主任务推断和行动规划的多模态执行控制器组成。为解决现有数据集缺乏回合级检索监督的问题，我们开发了统一场景引擎，程序生成结构化的多回合对话，并附带细粒度检索注释，使强化学习优化抽象和检索策略成为可能。我们还构建了一个按难易度分层的长视野视觉-对话基准，以评估情节视觉回忆。我们的8B代理在20回合会话中实现了91.4%的检索准确率，比32B基线高出+8.2%，且每回合推断时间几乎减半（23.1秒->12.7秒）。我们还介绍了认知结构多模态代理束带（CMA-Harness），这是一种基于同一认知结构的工具增强部署，集成了持久多模态记忆、网页访问、图像生成/编辑/合成工具以及兼容OpenAI的服务。结构化内存和模块化决策为长视野多模态代理提供了比单一参数缩放更可扩展、更高效的范式。代码：此 https 网址;项目页面：此 https URL

Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning

多模态、多环境机器教学以实现稳健的奖励学习

• Authors: Ali Larian, Qian Lin, Chang Zong Wu, Daniel S. Brown
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.08647
• Pdf link: https://arxiv.org/pdf/2607.08647
• Abstract As autonomous agents are increasingly deployed across diverse operational contexts, aligning their behavior with human intent demands reward functions that remain robust to such changes rather than overfitting to any single environment. Inverse reinforcement learning (IRL) provides a principled way to infer such objectives from human feedback. However, existing analyses of optimal teaching approaches for IRL focus on single-environment, demonstration-only settings, leaving underexplored how heterogeneous feedback modalities and environment dynamics jointly constrain reward functions that generalize across multiple environments. Because demonstrations in one MDP entangle reward information with that environments specific structure, the resulting rewards frequently fail to generalize when the agent is deployed in a new setting. We first analyze how different feedback modalities constrain rewards, showing that, in the unlimited-data regime, comparisons impose strictly stronger global constraints than other modalities. Beyond this theoretical analysis, we introduce a hierarchical machine teaching algorithm for reward learning that operates across multiple MDPs. The algorithm first greedily selects informative environments that expose complementary reward constraints, then strategically queries low-cost feedback within those environments. Empirically, our method achieves substantially lower regret and stronger generalization to held-out environments than uniform teaching baselines under identical feedback budgets, demonstrating the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions.
• 中文摘要 随着自主智能体在多样化运营环境中的部署日益增加，使其行为与人类意图保持一致，要求能够有效应对这些变化，而非过度拟合于任何单一环境。逆向强化学习（IRL）提供了一种有原则的方法，可以从人类反馈中推断出这些目标。然而，现有关于IRL最优教学方法的分析主要集中在单一环境、仅演示的环境，因此对异质反馈模态和环境动态如何共同约束跨多个环境的奖励函数尚未充分探讨。由于在一个MDP纠缠中演示会奖励该环境特定结构的信息，因此当代理在新环境中部署时，所得奖励往往无法泛化。我们首先分析了不同反馈模式如何限制奖励，显示在无限数据模式下，比较施加的全局约束比其他模式更严格。除了理论分析外，我们还引入了一种跨多个MDP运行的分层机器学习算法。算法首先贪婪地选择暴露互补奖励约束的信息环境，然后战略性地在这些环境中查询低成本反馈。实证上，我们的方法在相同反馈预算下相比统一教学基线实现了显著更低的遗憾率和更强的对待环境的推广，证明了多环境、多模态教学对学习动态稳健奖赏函数的重要性。

Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution

你需要一个Frontier型号作为引用验证器吗？深度研究来源归因的基准评分标准LLMs

• Authors: Ethan Leung, Elias Lumer, Corey Feld, Austin Huber, Vamse Kumar Subbiah, Kevin Paul
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.08700
• Pdf link: https://arxiv.org/pdf/2607.08700
• Abstract Reinforcement learning increasingly relies on an LLM judge to score each rubric criterion, and that judge acts as the reward model during training. Before such a signal can be trusted, we need to know how capable the judge must be and how biased it is. We study this calibration question for citation quality in deep-research systems, where a search-grounded LLM must support each claim it writes with a cited source. Citation quality is a structured rubric task in which each attribution-citation pair is judged along two dimensions that require an LLM, source relevance and factual support. On an adversarial long-form benchmark, we score 8 off-the-shelf LLM judges from 3 model families against gold labels over 1,248 rubric decisions, all of which were human-reviewed and 378 of which were hard cases adjudicated from judge disagreements. Cheaper judges remain competitive across both dimensions, with GPT-5-mini attaining the strongest source-relevance pass-class F1 at 0.908 ($\kappa$=0.636), while on factual support the judges are statistically indistinguishable (overlapping confidence intervals), so no single model dominates. At comparable F1, the judges still differ substantially in pass-rate drift, false positive rate, and false negative rate. Scalar F1 obscures this directional bias, yet it is exactly what a downstream reinforcement learning loop would reinforce. Calibrating the judge is therefore a prerequisite for using citation rubrics as reward signals, and our results show that this calibration does not require the most expensive available model.
• 中文摘要 强化学习越来越依赖LLM评判者对每个评分标准进行评分，而该评判员在培训中充当奖励模型。在信任这样的信号之前，我们需要知道法官的能力以及其偏见程度。我们研究了深度研究系统中引用质量的校准问题，其中基于搜索的大型语言模型必须用引用来源支持其撰写的每个主张。引用质量是一个结构化的评分标准任务，每个归因-引用对根据两个维度进行评判，这两维需要LLM、来源相关性和事实支持。在对抗性长格式基准测试中，我们对来自3个模型家族的8名现成LLM法官与金标进行评分，涉及1248个评分标准判决，全部为人工审核，其中378个为因法官分歧而裁决的硬案。成本较低的评委在这两个维度上依然具有竞争力，GPT-5-mini以0.908（$\kappa$=0.636）获得了最强的来源相关性通过类F1，而在事实支持下，评委在统计学上难以区分（置信区间重叠），因此没有单一模型占主导地位。在类似的F1级别中，评委在通过率漂移、假阳性和假阴性率上仍有显著差异。标量F1掩盖了这种方向偏置，但它正是下游强化学习循环所会强化的。因此，校准评审是使用引用评分标准作为奖励信号的前提，我们的结果表明，这种校准不需要最昂贵的模型。

MPFlow: Learning Budgeted Max-Flow Optimization on the Lightning Network with Deep Graph Reinforcement Learning

MPFlow：利用深度图强化学习学习闪电网络上的预算最大流量优化

• Authors: Harrison Rush, Vincent Davis, Simone Antonelli, Vikash Singh, Jesse Shrader, Emanuele Rossi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.08703
• Pdf link: https://arxiv.org/pdf/2607.08703
• Abstract We address liquidity placement in the Bitcoin Lightning Network (LN): given a fixed budget, which channels should a node open to maximize its routing capacity? We cast this as a budget-constrained combinatorial optimization problem on graphs, selecting $k$ edge additions that maximize $s$--$t$ max-flow, a theory-grounded measure of routing capacity, and solve it with graph reinforcement learning. Our lightweight agent combines a message-passing policy network with proximal policy optimization (PPO) and action masking, and is trained under a hub-exclusion curriculum: the network's top hubs are removed from training subgraphs, forcing the policy to learn capacity-aware placement rather than hub attachment. In extensive experiments on real Lightning Network snapshots, our method consistently outperforms strong heuristic baselines on the max-flow objective across multiple seeds and unseen graphs. The agent has been deployed in production for peer recommendations, executing 4640 channel-open decisions that cumulatively allocate 267.3 BTC over $16 million across 30 managed nodes.
• 中文摘要 我们讨论比特币闪电网络（LN）中的流动性配置：在固定预算下，节点应开启哪些通道以最大化其路由容量？我们将此定位为一个预算受限的组合优化图问题，选择$k$边加法以最大化最大流量$s$-$t$，这是一种基于理论的路由容量度量，并通过图强化学习求解。我们的轻量级代理结合了消息传递策略网络、近端策略优化（PPO）和动作掩蔽，并采用枢纽排除课程训练：网络的顶端枢纽被从训练子图中移除，迫使策略学习容量感知的布局，而非枢纽附加。在对真实闪电网络快照的广泛实验中，我们的方法在多个种子和未见图的最大流量目标上，始终优于强启发式基线。该代理已部署于生产环境中，用于对等推荐，执行了4640次通道开放决策，累计分配了267.3个BTC，金额超过1600万美元，覆盖30个托管节点。

Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference

潜在记忆宫殿：作为自回归变分推理的控制推理推理

• Authors: Chuning Zhu, Eva Xu, Jose Barreiros, Krishnan Srinivasan, Paarth Shah, Abhishek Gupta
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.08724
• Pdf link: https://arxiv.org/pdf/2607.08724
• Abstract Human decision-making is highly flexible -- some actions are taken immediately; others require longer deliberation. Language models have exhibited a similar capacity for adaptive "reasoning." However, transferring this capability to continuous control policies has been challenging, as directly reasoning in language space may lack the granularity for spatial understanding and precise motions. In this work, we show that reasoning for control policies can emerge by organizing information in an autoregressive latent space reminiscent of a memory palace, where retrieval is iterative and adaptive. Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution. We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound. The resulting policy, LMP-$\pi$, achieves strong empirical performance in simulation and real-world domains while exhibiting interpretable, adaptive allocation of test-time compute. We further show that the same framework yields a variable-length action tokenizer, LMP-$\texttt{tok}$, which significantly improves the performance of downstream autoregressive policies. Together, these results present a new perspective on latent reasoning for control through the lens of variational inference.
• 中文摘要 人类决策极为灵活——有些行动是立即采取的;有些则需要更长时间的审议。语言模型也展现出类似的适应性“推理”能力。然而，将这种能力转移到持续控制策略中一直具有挑战性，因为在语言空间中直接推理可能缺乏空间理解和精确动作的细致度。本研究展示了控制策略的推理可以通过在类似记忆宫殿的自回归潜在空间中组织信息，从而实现反复和适应性。我们的方法“潜在记忆宫殿”（LMP）将推理表述为带有自回归潜在分布的变分推断。我们推导出一种潜空间强化学习技术，以可操作地优化其变分下界。最终策略LMP-$\pi$在仿真和现实世界领域实现了强劲的实证性能，同时展现了可解释、自适应的测试时间计算分配。我们还进一步证明，同一框架还能生成一个可变长度的动作分词器LMP-$\texttt{tok}$，显著提升了下游自回归策略的性能。这些结果共同为通过变分推断视角对控制的潜在推理提供了新的视角。

Keyword: diffusion policy

DexVerse: A Modular Benchmark for Multi-Task, Multi-Embodiment Dexterous Manipulation

DexVerse：一个多任务、多身体灵巧操作的模块化基准测试

• Authors: Yunchao Yao, Zhuxiu Xu, Tianqi Zhang, Zixian Liu, Sikai Li, Zhenyu Wei, Feng Chen, Dihong Huang, Kechang Wan, Chenyang Ma, Shuqi Zhao, Shenghua Gao, Masayoshi Tomizuka, Yi Ma, Mingyu Ding
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.08751
• Pdf link: https://arxiv.org/pdf/2607.08751
• Abstract Building general-purpose dexterous manipulation policies requires benchmarks that go beyond isolated tasks to systematically evaluate policies across diverse interaction modes, sensory conditions, and robot embodiments. However, existing benchmarks remain limited in task and data diversity, embodiment coverage, or controllable visual variation, hindering studies of cross-task and cross-embodiment generalization. We present DexVerse, a large-scale and modular benchmark for dexterous manipulation. DexVerse includes 100 tasks spanning a broad range of manipulation skills, including object grasping and relocation, articulated-object interaction, functional tool use, bimanual coordination, non-prehensile control, contact-rich behaviors, multi-goal execution, and long-horizon multi-stage task completion. It supports 3 robot arms and 6 dexterous hands, and is extensible to new tasks, assets, and embodiments. To evaluate visuomotor generalization, DexVerse provides configurable visual variations in textures, background, lighting, and camera viewpoints. We further provide a VR-based teleoperation interface and 3,180 demonstrations with synchronized proprioceptive, RGB, depth, point-cloud, and state observations. We benchmark representative methods, including Diffusion Policy, DP3, OpenVLA, and $\pi_{0.5}$, across 19 tasks. Results reveal substantial challenges in task generalization and visuomotor robustness, establishing DexVerse as a promising testbed for general-purpose dexterous manipulation. Project page: this https URL
• 中文摘要 构建通用的灵巧操作策略需要超越孤立任务的基准，系统性地评估不同交互模式、感官条件和机器人体型的策略。然而，现有基准在任务和数据多样性、身体覆盖度或可控视觉变化方面仍有限，阻碍了跨任务和跨身体推广的研究。我们介绍DexVerse，一个大规模且模块化的灵巧操作基准测试。DexVerse包含100个任务，涵盖广泛的操作技能，包括物体抓取与重新定位、关节与物体交互、功能性工具使用、双手协调、非抓握控制、丰富接触行为、多目标执行以及长视野多阶段任务完成。它支持3只机械臂和6只灵巧的手，并且可扩展到新的任务、资产和实体。为了评估视觉运动泛化，DexVerse提供了可配置的纹理、背景、光照和摄像机视角的视觉变化。我们还提供了基于虚拟现实的远程操作接口，并进行了3180个演示，包括同步本体感觉、RGB、深度、点云和状态观测。我们对包括扩散策略、DP3、OpenVLA和$\pi_{0.5}$在内的代表性方法进行了基准测试，涵盖19个任务。结果揭示了任务泛化和视觉运动稳健性方面的重大挑战，使DexVerse成为通用灵巧操作的有前景的试验平台。项目页面：此 https URL