生成时间: 2026-08-05 18:39:00 (UTC+8); Arxiv 发布时间: 2026-08-05 20:00 EDT (2026-08-06 08:00 UTC+8)

今天共有 50 篇相关文章

Keyword: reinforcement learning

Don't Regenerate, Debug: A Domain-Specific Agent for Repairing Near-Miss Hardware Operators

别再生，调试：一个专门用于修复近失控硬件操作员的领域专用代理

• Authors: Yansong Sun, Shenxiu Wu, Siyuan Chen, Runlin Hou, Junhao Qiu, Junming Cao, Shudi Shao, Zhichao Lu, Qingfu Zhang
• Subjects: Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.02712
• Pdf link: https://arxiv.org/pdf/2608.02712
• Abstract Kernel generation for hardware accelerators such as GPUs and NPUs has become a proving ground for large language models (LLMs), and state-of-the-art systems raise correctness through pipelines that couple LLMs with agentic reinforcement learning and evolutionary search. Such pipelines generate, compile, and execute large numbers of candidate kernels, discarding most of them and forgoing the opportunity to distill failures into reusable knowledge. Many discarded candidates are near-miss operators that compile and run but fail numerical validation; each embodies genuine domain knowledge and a nontrivial investment in LLM inference, cross-compilation, and hardware execution. We argue for a paradigm shift: rather than regenerate, debug. Debugging is far more constrained than generating from scratch: the search space is small and feedback is dense. We present a domain-specific debug agent that addresses three core challenges in autonomous repair: mitigating knowledge scarcity through retrieved patterns and diagnostic instrumentation, ensuring integrity through anti-cheat detection and full-coverage evaluation, and controlling cost via convergence guards and bounded iteration. Debugging serves two complementary roles: it extends the capability frontier by recovering operators that repeated regeneration fails to produce, and it lowers cost per deliverable operator. Debug Pass@1 achieves 66.7% versus Regenerate Avg Pass@1's 25.9% and Regenerate Pass@3's 40.7%, while consuming 92.8% fewer tokens per success than three-trial regeneration. Component ablations show that the knowledge base drives recovery, while integrity gates reject 12.5-33.3% of the successes the workflow itself accepted.
• 中文摘要 GPU和NPU等硬件加速器的内核生成已成为大型语言模型（LLM）的试验场，而最先进的系统通过将LLM与代理强化学习和进化搜索结合的流水线提高了正确性。这些流水线生成、编译并执行大量候选内核，丢弃了大部分，放弃将失败提炼为可重用知识的机会。许多被丢弃的候选内核是近乎失败的算子，虽然编译并运行，但数值验证失败;每个都体现了真正的领域知识，以及在LLM推理、交叉编译和硬件执行上的非凡投资。我们主张范式转变：与其再生，不如调试。调试比从零生成更受限：搜索空间小，反馈密集。我们提出了一个领域特定的调试代理，解决自主修复中的三大核心挑战：通过检索模式和诊断工具缓解知识稀缺性，通过反作弊检测和全覆盖评估确保完整性，以及通过收敛防护和有界迭代控制成本。调试有两个互补作用：通过恢复重复再生无法产生的操作员，扩展能力前沿;降低每个交付操作员的成本。调试Pass@1实现66.7%，而Regenerate Avg Pass@1为25.9%，Regenerate Pass@3为40.7%，且每次成功消耗的token比三次尝试再生少92.8%。组件消融显示知识库驱动恢复，而完整性门拒绝了12.5%至33.3%的工作流程本身接受的成功率。

Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

通过演进评分标准的强化学习作为音频推理的奖励

• Authors: Fangxu Yu, Tao Feng, Dehai Min, Zinan Lin, Weijia Xu, Michael Xu, Philip S. Yu, Ge Liu, Tianyi Zhou
• Subjects: Subjects: Sound (cs.SD); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.02831
• Pdf link: https://arxiv.org/pdf/2608.02831
• Abstract Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at this https URL.
• 中文摘要 音频推理对于机器理解声学世界至关重要。带有可验证奖励的强化学习可以引发此类推理，但现有的奖励设计在局限性上是互补的：基于结果的奖励只监督最终答案，让模型在不顾音频的情况下获得答案，而基于过程的奖励则对推理本身进行评分，但依赖粗糙、手工制作且固定的标准，既不适应每个问题，也不依赖声学证据。此外，问题在需求上各不相同，有些依赖感知，有些依赖多步推理，任何静态标准随着策略的改进都会削弱。因此，用细粒度、音频基础和自适应奖励来监督推理过程至关重要，但也具有挑战性，因为这些奖励无法为每个样本手工设计。为此，我们引入了AudioRubrics，一种强化学习框架，通过自我演进、基于音频的评分标准奖励来监督音频推理。AudioRubrics从原始波形中合成每个采样的评分标准，并根据模型自身的部署情况，重新生成和重新权重每个组的标准，提供持续学习信号，持续针对当前策略的弱点，随着静态标准的饱和。对三个音频推理基准的全面评估显示，AudioRubrics在众多开源和基于训练的基线中表现显著优于。此外，我们的分析显示，收益随评分标准生成器和评判器的能力而成比例，AudioRubrics趋向稳定的推理长度，避免了退化崩溃和无界增长。听觉感知的提升进一步证明了将监督锚定在声学证据中的有效性。我们的项目页面可通过此 https 网址访问。

BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?

BODHI：大型语言模型会不会扩展并发现异构推理？

• Authors: Soumadeep Saha, Krish Sharma, Akshay Chaturvedi, Nicholas Asher
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.02867
• Pdf link: https://arxiv.org/pdf/2608.02867
• Abstract Although reinforcement learning with verifiable rewards (RLVR) has improved the performance of large language models (LLMs) across a variety of reasoning tasks, there is significant debate as to whether RLVR expands the reasoning capability boundary, or just improves sampling efficiency. In this paper, we investigate the nature of test-time exploration in RLVR-trained LLMs by employing controlled maze-solving experiments and extracting a tree structure from mathematical reasoning traces (BODHI-Trees) based on semantic equivalence. This helps us delineate between entropy arising from stylistic variations and genuine inferential branching. Our findings demonstrate that the policy entropy collapse observed in RLVR models is not merely syntactic, and is accompanied by a significant reduction in semantic branching entropy. While RLVR improves adherence to environmental constraints and backtracking capabilities, it constricts the space of continuations; we provide evidence suggesting that this might be responsible for the sample efficiency gains of RLVR, albeit at the cost of genuine rollout diversity.
• 中文摘要 尽管带可验证奖励的强化学习（RLVR）提升了大型语言模型（LLMs）在多种推理任务中的表现，但关于RLVR是否扩展推理能力边界，还是仅仅提升采样效率，存在重大争议。本文通过采用受控迷宫解决实验，并基于语义等价从数学推理轨迹（BODHI树）中提取树状结构，研究RLVR训练的LLMs测试时间探索的本质。这有助于我们区分由风格变化引起的熵与真正的推理分支。我们的发现表明，RLVR模型中观察到的策略熵崩溃不仅仅是句法上的，且伴随着语义分支熵的显著降低。虽然RLVR提高了对环境约束的遵守性和回溯能力，但也限制了延续空间;我们提供了证据表明，这可能是RLVR样本效率提升的原因，尽管代价是真实的推广多样性。

FLARE: Few-shot Learning-based Adaptive Reflective Engine

FLARE：基于少数样本学习的自适应反射引擎

• Authors: Dhanasekar Sundararaman, Bharat Gandhi, Aashna Garg, Minjie Li
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.02919
• Pdf link: https://arxiv.org/pdf/2608.02919
• Abstract Large language models (LLMs) are increasingly deployed in complex, compound AI systems where performance hinges on the quality of prompts. Recent state-of-the-art optimizers like GEPA (Genetic-Pareto) have argued that reflective instruction evolution can outperform traditional reinforcement learning and few-shot optimization. In this work, we challenge this shift by introducing FLARE (Few-shot Learning-based Adaptive Reflective Engine), a framework that leverages advanced reflective mechanisms and a small set of few-shot reference examples to optimize instructions. We evaluate our method across a diverse suite of benchmarks -- spanning retrieval-augmented reasoning (HotPotQA, MedQA, 2WikiMultiHopQA), tool calling, and multi-label emotion classification (GoEmotions) -- using the GPT-5 series of models. Our results demonstrate that FLARE consistently outperforms GEPA, winning on every task-model pair: it achieves gains of up to +14.2 points on HotPotQA (52.2 vs. GEPA's 42.2 with GPT-5-Chat), reaches 87.0% on tool calling (vs. 81.0% for GEPA), and lifts GoEmotions micro-F1 to 52.7% (+15.3) with GPT-5.1 on the full 5408-example test split, more than doubling GEPA's +5.7 gain. Beyond raw accuracy, FLARE is also strikingly data-efficient: on GoEmotions it reaches its peak performance using as few as 100 validation examples, while remaining markedly more stable across random seeds than GEPA. Our findings suggest that while reflective instructions are powerful, the strategic optimization of few-shot learning remains a critical frontier for maximizing the potential of next-generation LLMs.
• 中文摘要 大型语言模型（LLMs）越来越多地被应用于复杂、复杂的复合人工智能系统中，而这些系统的性能取决于提示的质量。近年来，像GEPA（Genetic-Pareto）这样的最先进优化器认为，反思式指令演化可以优于传统强化学习和少数样本优化。在本研究中，我们通过引入FLARE（基于少数样本学习的自适应反思引擎，FARE）来挑战这一转变，该框架利用先进的反思机制和少量少量样本的参考示例来优化指令。我们通过多种基准测试来评估我们的方法——涵盖检索增强推理（HotPotQA、MedQA、2WikiMultiHopQA）、工具调用和多标签情绪分类（GoEmotions）——使用GPT-5系列模型。我们的结果表明，FLARE 持续优于 GEPA，在所有任务模型对上都获胜：它在 HotPotQA 上提升高达 +14.2 分（GPT-5-Chat 为 52.2 分，GPT-5-Chat 为 42.2），工具调用率达到 87.0%（GEPA 为 81.0%），并且在 GPT-5.1 的 5408 个样本测试分段中，将 GoEmotions micro-F1 提升至 52.7%（+15.3），是 GEPA +5.7 分的两倍多。除了原始准确性，FLARE 还极具数据效率：在 GoEmotions 上，仅用 100 个验证样本即可达到峰值性能，同时在随机种子间的稳定性明显高于 GEPA。我们的发现表明，虽然反思式指令强大，但少数样本学习的战略优化仍是最大化下一代大型语言模型潜力的关键前沿。

Rubrics as Privileged Information for Open-Ended Generation

评分标准作为开放式生成的特权信息

• Authors: Deepika Bablani, Ajay Gupta, Wanming Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.02948
• Pdf link: https://arxiv.org/pdf/2608.02948
• Abstract On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations. We extend OPSD to open-ended generation using soft PI in the form of rubrics that guide preferences but admit many valid responses. Rubrics have served as scalar rewards for reinforcement learning (RL); we show that they provide substantially richer signal as dense PI for distillation, and contrary to intuition, soft rubric PI provides a larger and more effective training signal on student roll-outs than hard reference completion PI in this regime. A reference completion is one point in a set of valid responses, so distilling towards it over-constrains the student, while rubrics specify the preference structure shared across the set of valid responses. We show the effectiveness of using rubrics as PI for open-ended generation across Qwen and Llama model families and show that it outperforms rubric-as-reward (RaR) RL using HealthBench, a benchmark that grades open-ended health responses against physician-created rubrics, providing dense token-level supervision for open-ended tasks; RuPI beats RaR RL by up to +0.10 absolute score and, under matched recipe and KL direction, beats reference-PI by +0.034 to +0.079 absolute score across three models. We further show that these findings generalize to training on the RubricHub Science corpus and evaluating on ResearchQA: soft rubric PI outperforms both reference-PI distillation and RaR RL (66.6% vs. 64.2% and 57.6%).
• 中文摘要 在策略自我蒸馏（OPSD）中，单一模型在不同情境下既作为学生又作为教师，在可验证领域如数学中展现出希望，硬特权信息（PI）以实地真相回答形式结构性限制有效延续。我们将OPSD扩展到开放式生成，使用软PI形式指导偏好但允许许多有效回应。规律作为强化学习（RL）的标量奖励;我们证明它们作为浓密PI提供了更丰富的信息，且与直觉相反，软指标PI在学生推广时比硬性引用补全PI提供更大且更有效的训练信号。参考补全是有效回答集合中的一个点，因此向它提炼会过度约束学生，而规律则指定有效响应集合共享的偏好结构。我们展示了在Qwen和Llama模型家族中，使用评分标准作为开放式生成PI的有效性，并展示了其优于使用HealthBench的评分标准即奖励（RaR）RL。HealthBench是一个基准工具，该基准将开放式健康反应与医生创建的评分标准进行评分，为开放式任务提供密集的代币级监督;RuPI在RaR RL中以+0.10绝对得分领先RaR RL，在配方和KL方向匹配不足的情况下，在三个模型中以+0.034至+0.079的绝对分数优于参考PI。我们还进一步表明，这些发现在RubricHub Science语料库上的训练和ResearchQA评估中也适用：软指标PI在参考PI提炼和RaR RL中表现均优于（66.6%对64.2%和57.6%）。

SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling

SP3O：无奖励建模的分段偏好强化学习

• Authors: Evan Assmus, Qining Zhang, Lei Ying
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.02951
• Pdf link: https://arxiv.org/pdf/2608.02951
• Abstract Preference-based reinforcement learning (PbRL) for general stochastic MDPs often requires training a reward model. Existing reward-model-free methods are either restricted to bandits or deterministic MDPs, such as DPO or P3O, or use zeroth-order, gradient-free optimization, which in general exhibits a slower convergence rate than gradient-based algorithms. Furthermore, existing reward-model-free preference-based RL algorithms almost exclusively use trajectory-level feedback, which can require significant effort from a human evaluator when trajectories are long. On the other hand, segments are much shorter, so they are easier to compare and evaluate. In this paper, we introduce a novel reward-model-free, critic-free, and gradient-based PbRL algorithm compatible with segment preferences named Segment Pairwise Proximal Policy Optimization (SP3O). SP3O utilizes segment-level preference feedback to construct an accurate policy value difference estimator via off-policy importance sampling, and then uses the estimator to compute the policy gradient via a PPO-type loss function. We provide a theoretical basis for the algorithm and analyze the tradeoff in choosing the segment length. We also evaluate it experimentally against other PbRL/RLHF algorithms in robotic control and LLM finetuning settings to show its improved performance, especially in long-horizon tasks.
• 中文摘要 基于偏好的强化学习（PbRL）通常需要训练一个奖励模型。现有的无奖励模型方法要么仅限于bandit（强盗）或确定性MDP，如DPO或P3O，要么使用零阶、无梯度优化，后者的收敛速度通常比基于梯度的算法慢。此外，现有无奖励模型偏好的RL算法几乎完全使用轨迹级反馈，当轨迹较长时，人工评估者可能需要大量工作。另一方面，片段更短，因此更容易比较和评估。本文介绍了一种新颖的无奖励模型、无批评、基于梯度的PbRL算法，兼容段偏好，称为段对近策略优化（SP3O）。SP3O 利用分段级偏好反馈，通过非策略重要性抽样构建准确的策略值差值估计，然后利用该估计器通过 PPO 类型的损失函数计算策略梯度。我们为该算法提供了理论基础，并分析了选择段长度的权衡。我们还通过实验性地与其他 PbRL/RLHF 算法在机器人控制和大型语言模型微调设置中进行了比较，以展示其在长视野任务中的性能提升。

ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies

ValueFormer：一种因果变换器价值函数，带有阶段感知标签，用于半自治视觉-语言-行动策略

• Authors: Inkyu Sa, Konstantin Stulov, Rajat Bhageria
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.02958
• Pdf link: https://arxiv.org/pdf/2608.02958
• Abstract Vision-Language-Action (VLA) policies trained by behavior cloning fail silently: from the action stream alone, a collapsing rollout looks much like one making clean progress, because imitation supplies no notion of progress. Reinforcement learning would supply one, but it is impractical here, where real-robot experience is costly and deformable food resists simulation. The cheap alternative, a terminal success / failure bit, is learnable in principle yet far too sparse to say when a rollout went wrong. We argue that the per-frame label, not the architecture, is the hard part: to be useful it must be dense, continuous, and correctly shaped. We present ValueFormer, a compact policy-agnostic causal transformer over a frozen DINOv3 backbone that emits two per-frame signals in one forward pass: a smooth Monte Carlo value, V_mc, for advantage estimation and a sharp binary value for online mistake detection, targets that pull in opposite directions by design. Failed episodes are labeled with a stage-aware, success-then-decay return that preserves the success curve before the failure stage, and detection is supervised from mistake intervals rather than a single failure time, so mistakes the policy recovers from also carry signal. On a real-robot bimanual sandwich-assembly task 1,427 episodes), a critic-derived per-frame training weight lifts task completion from 70% to 85% (within noise at n=20), and a batched bf16 encoder cuts the live serving cost 3~5 times so the critic runs at 2 Hz alongside the policy on a single GPU.
• 中文摘要 通过行为克隆训练的视觉-语言-行动（VLA）策略无声失效：仅从动作流来看，崩溃的滚动就像是干净利落的进展，因为模仿不提供进步的概念。强化学习可以提供进步，但在这里不切实际，因为真实机器人体验代价高昂，且可变形的食物难以模拟。廉价的替代方案——终端成功/失败部分，原则上可学习，但太稀疏，无法判断推广何时出错。我们认为，难点在于每帧标签，而非架构：要有用，它必须密集、连续且形状正确。我们呈现ValueFormer，一个紧凑型的策略无关因果变换器，基于冻结的DINOv3骨干，在一次前向传递中每帧发出两个信号：一个平滑的蒙特卡洛值V_mc用于优势估计，以及一个锐利的二进制值用于在线错误检测，这两个目标设计上会向相反方向拉动。失败的事件会以阶段感知、成功后衰减的返回标记，保持失败阶段前的成功曲线，且检测从错误间隔而非单次故障时间进行监督，因此策略恢复的错误也会传递信号。在真实机器人双手夹层组装任务（1427集）中，Critic生成的每帧训练权重将任务完成率从70%提升到85%（在噪声范围内n=20），而批量bf16编码器将实时服务成本降低3~5倍，使critic在单GPU上以2 Hz运行，同时执行政策。

Neurosymbolic Reasoning with Incremental Knowledge for Sample Efficient Hierarchical Reinforcement Learning

基于增量知识的神经符号推理，用于样本高效层级强化学习

• Authors: Subrat Prasad Panda, Blaise Genest, Arvind Easwaran
• Subjects: Subjects: Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.02993
• Pdf link: https://arxiv.org/pdf/2608.02993
• Abstract (Flat) Reinforcement Learning (RL) agents face significant challenges in environments with sparse rewards that require long-horizon reasoning. A compelling approach to improve sample efficiency is to incorporate knowledge into learning and decision-making. In standard Hierarchical RL (HRL), knowledge is encoded in a fixed, non-updatable form, such as architectural choices, and remains unchanged throughout learning. With fixed HRL, reasoning with incremental knowledge learned during exploration is impractical before sufficient environmental knowledge is acquired, leading to poor sample efficiency. In this work, we propose neurosymbolic HRL with {\em Incremental Knowledge (InK)}: symbolic high-level components perform {\em symbolic planning} (e.g. using $D^*$) on an updatable representation of current InK, while low-level goal-conditioned neural modules learn motion primitives through experience using reward shaping. Experiments on navigation tasks demonstrate that incorporating InK substantially improves sample efficiency. Additionally, to perform {\em optimal} symbolic planning given {\em prior} knowledge about the world, we develop Belief World Tree Search. The code is available at this https URL.
• 中文摘要 （平面）强化学习（RL）智能体在奖励稀疏且需要长期推理的环境中面临重大挑战。提高样本效率的一种有效方法是将知识融入学习和决策中。在标准的层级RL（HRL）中，知识以固定、不可更新的形式编码，如架构选择，且在整个学习过程中保持不变。在固定HRL中，在获得足够环境知识之前，基于探索中学到的增量知识推理不切实际，导致样本效率较低。本研究提出神经符号HRL与{\em增量知识（InK）}：符号高级组件在当前InK的可更新表示上执行{\em符号规划}（例如使用$D^*$）），而低级目标条件神经模块通过奖励塑形通过经验学习运动原语。导航任务的实验表明，采用InK能显著提升样本效率。此外，为了在{\em先验}世界知识下进行符号规划，我们开发了信念世界树搜索。该代码可在此 https URL 获取。

CastFSR: A Fast--Slow--Reflect Agentic Reasoning Framework for Context-Aware Time Series Forecasting

CastFSR：一种用于上下文感知时间序列预测的快速-慢-反射代理推理框架

• Authors: Xiaoyu Tao, Mingyue Cheng, Bokai Pan, Chuang Jiang, Huanjian Zhang, Tian Gao, Yaguo Liu, Qi Liu, Enhong Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03031
• Pdf link: https://arxiv.org/pdf/2608.03031
• Abstract Time series forecasting is fundamental to decision-making in complex systems, where future dynamics are influenced not only by historical observations but also by evolving contextual features. Recent advances in large language models (LLMs) have extended forecasting beyond numerical extrapolation toward context-aware reasoning. However, existing approaches often lack explicit mechanisms to identify relevant contexts, reason about their impacts, and validate forecasts against temporal and domain constraints. In this work, we propose CastFSR, an agentic framework that formulates context-aware forecasting as a Fast--Slow--Reflect workflow. In fast thinking, CastFSR profiles observations and selects lightweight forecasters to construct a data-driven forecast prior. In slow deliberation, it retrieves contextual evidence, adaptively determines informative look-back windows, and reasons about how contexts reshape future dynamics. In reflection, it iteratively refines forecasts to ensure temporal, contextual, and domain consistency. CastFSR supports both training-free inference with off-the-shelf LLMs and efficient deployment through a two-stage SFT and reinforcement learning strategy that transfers its orchestration capability to compact LLMs. Extensive experiments on public datasets demonstrate that CastFSR consistently outperforms representative baselines. Our code is available at this https URL.
• 中文摘要 时间序列预测是复杂系统决策的基础，未来动态不仅受历史观测影响，还受不断演变的情境特征影响。大型语言模型（LLMs）的最新进展将预测从数值外推扩展到情境感知推理。然而，现有方法往往缺乏明确机制来识别相关情境、推理其影响，以及根据时间和领域约束验证预测。本研究提出CastFSR，一种能动框架，将情境感知预测构建为快-慢-反映工作流。在快速思考中，CastFSR分析观测数据，并选择轻量级预测者构建数据驱动预测。在慢思考中，它检索情境证据，自适应确定有益的回顾窗口，并推理情境如何重塑未来动态。反思时，它通过迭代优化预测，确保时间、上下文和领域的一致性。CastFSR 支持现成大型语言模型的无训练推断，并通过两阶段的SFT和强化学习策略高效部署，将编排能力转移到紧凑型LLMs。在公共数据集上的大量实验表明，CastFSR 持续优于代表性基线。我们的代码可在此 https URL 访问。

LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs

LLM 在野外服务：框架、方法与系统设计的实证研究

• Authors: Forough Majidi, Mohammad Mehdi Morovati, Foutse Khomh, Heng Li
• Subjects: Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03036
• Pdf link: https://arxiv.org/pdf/2608.03036
• Abstract Large Language Models (LLMs) are integrated into software systems and AI services, making efficient LLM serving a concern for software engineering. Serving LLMs is challenging because inference requires computation, memory, GPU resources, and execution while maintaining latency and throughput. Although prior research has proposed LLM inference, optimization, and serving techniques and frameworks, little is known about how they are adopted in practice. In this study, we investigate the use of LLM serving frameworks and serving methods in open-source software systems. We identify and analyze five LLM-specific frameworks: vLLM, SGLang, TensorRT-LLM, LMDeploy, and FlashInfer. We examine how these frameworks and techniques are adopted individually and in combination, how adoption varies across categories of LLMs, and how repositories differ in intent, focus, use case, and architectural design. Our results show that vLLM is the most visible framework in popularity and adoption, while parallel computation, memory management, and network pruning are the most frequently used serving-method categories. Multi-framework usage is limited, suggesting that developers rely on a single serving framework; however, combined frameworks connect complementary capabilities across the serving stack. Framework adoption varies across model families, modalities, model sizes, domain specializations, and deployment settings. Repository-level analysis shows that LLM serving frameworks support applications and architectures, including Reinforcement Learning (RL)-based reasoning, multimodal generation and understanding, microservices, and cloud infrastructure. Overall, this study provides a large-scale empirical characterization of LLM serving framework adoption in practice and offers insights for researchers, framework maintainers, and practitioners working on LLM systems.
• 中文摘要 大型语言模型（LLM）集成于软件系统和人工智能服务中，使得高效服务成为软件工程关注的焦点。服务LLM具有挑战性，因为推理需要计算、内存、GPU资源和执行，同时保持延迟和吞吐量。尽管已有研究提出LLM推理、优化和服务技术及框架，但对其实际应用知之甚少。本研究探讨了开源软件系统中LLM服务框架及其方法的应用。我们识别并分析五个LLM专用框架：vLLM、SGLang、TensorRT-LLM、LMDeploy和FlashInfer。我们考察这些框架和技术如何单独或组合采用，不同类别的LLM采用差异，以及仓库在意图、重点、用例和架构设计上的差异。我们的结果显示，vLLM是最受欢迎的框架，而并行计算、内存管理和网络修剪则是最常用的服务方法类别。多框架的使用有限，表明开发者依赖单一服务框架;然而，组合框架连接了服务栈中的互补能力。框架的采用因模型家族、模式、模型大小、领域专化和部署设置而异。仓库级分析显示，LLM服务框架支持的应用和架构，包括基于强化学习（RL）的推理、多模态生成与理解、微服务以及云基础设施。总体而言，本研究对LLM服务框架的实际采用进行了大规模实证描述，并为研究者、框架维护者和从事LLM系统的实践者提供见解。

PLAN: Parallel Liquid-Inspired Approximation Network for Efficient Representation Learning in Flexible Job Shop Scheduling

计划：并行液体启发近似网络，用于灵活工坊排班中的高效表示学习

• Authors: Dhivya Dharshini Kannan, Wei Zhang, Jieyi Bi, Yingpeng Du, Tianjun Wei, Jie Zhang, Zuming Liu, Anupam Trivedi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03041
• Pdf link: https://arxiv.org/pdf/2608.03041
• Abstract Deep reinforcement learning (DRL) approaches for flexible job shop scheduling (FJSP) heavily rely on attention-centric architectures to achieve state-of-the-art performance. However, these models suffer from excessive parameter counts and prohibitive inference latency as problem scales expand. While liquid neural networks (LNNs) offer a parameter-efficient alternative for modeling adaptive state evolution, their inherently sequential dynamics bottleneck computational efficiency. To resolve this trade-off, we propose PLAN (Parallel Liquid-inspired Approximation Network), a lightweight representation learning framework that reformulates continuous liquid-state dynamics into a discretized and parallelizable formulation. PLAN structurally decouples state evolution from context aggregation, where liquid-inspired updates handle the primary evolving state representation, and a lightweight context aggregation module provides complementary global context. Furthermore, PLAN acts as a versatile, plug-and-play backbone that generalizes to complex FJSP variants, pairing with a compact stochastic module for stochastic FJSP and replacing heavy heterogeneous graph transformers in multi-faceted dynamic FJSP. Extensive evaluations across deterministic, stochastic, and multi-faceted dynamic FJSP benchmarks show that PLAN reduces the average makespan by 1.2%, 1.4%, and 2.3%, respectively, compared with the corresponding state-of-the-art baselines, with the improvement reaching 10.2% in one benchmark setting. PLAN also reduces average inference latency by 13.2%, 31.7%, and 26.9%, respectively, with a maximum reduction of 69.2% on the largest instances, while using only 22$-$47% of the baseline parameters.
• 中文摘要 深度强化学习（DRL）方法用于灵活作业车间调度（FJSP）高度依赖以注意力为中心的架构以实现最先进的性能。然而，这些模型存在过多的参数数量和随着问题规模扩大而难以承受的推理延迟。虽然液体神经网络（LNN）为建模自适应状态演化提供了参数高效的替代方案，但其固有的顺序动力学限制了计算效率。为解决这一权衡，我们提出了PLAN（并行液体启发近似网络）的轻量级表示学习框架，将连续液体状态动态重新表述为离散且可并行化的表述。PLAN在结构上将状态演化与上下文聚合解耦，后者由液体启发的更新处理主要的进化状态表示，轻量级上下文聚合模块则提供互补的全局上下文。此外，PLAN 作为一个多功能的即插即用骨干，可推广到复杂的 FJSP 变体，配合紧凑的随机模块支持随机 FJSP，并取代多面动态 FJSP 中的繁重异构图变换器。对确定性、随机和多面动态 FJSP 基准的广泛评估显示，PLAN 分别将平均完成时长缩短 1.2%、1.4% 和 2.3%，相比对应的先进基线，在一个基准测试设置中提升达到 10.2%。PLAN 还分别降低了平均推理延迟 13.2%、31.7% 和 26.9%，最大实例最大减少 69.2%，而仅使用了 22$-$47% 的基线参数。

PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory

PI-Mem：将长上下文推理推送到360万个并行迭代存储的token中

• Authors: Dawei Liu, Haixu Song, Shuang Cheng, Shijie Wang, Haozheng Hou, Kaifeng Liu, Ermo Hua, Zhonghang Yuan, Zhijie Zhong, Yuchen Fan, Biqing Qi, Bowen Zhou
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03048
• Pdf link: https://arxiv.org/pdf/2608.03048
• Abstract Long-context reasoning remains a critical bottleneck for large language models, as recent recurrent-memory approaches face two inherent challenges: sequential chunk-wise updates can overwrite early critical evidence with later irrelevant content, and serial inter-chunk dependencies limit parallelism and cause latency to increase with context length. To address these issues, we propose PI-Mem (Parallel-Iterative Memory), a mechanism that processes all chunks in parallel and iteratively refines a shared memory over a bounded number of turns. In each turn, PI-Mem reads all chunks in parallel conditioned on the current memory, selects new or complementary evidence from each chunk, and merges the selected evidence into a compact shared memory for the next turn. To discourage redundant turns, we optimize the workflow through reinforcement learning with an auxiliary turn-efficiency reward, enabling the model to adaptively exit once sufficient evidence has been accumulated. We evaluate PI-Mem with Qwen3.5-35B-A3B and Qwen2.5-7B on the HotpotQA benchmark across context lengths up to 3.6 million tokens and find that it outperforms the recurrent-memory baseline by +6.25 and +7.81 absolute points while achieving 6.1$\times$ and 2.1$\times$ inference speedups, respectively. These results demonstrate that PI-Mem breaks the accuracy--efficiency trade-off in long-context reasoning and provides a scalable approach to complex multi-hop question answering over extremely long documents.
• 中文摘要 长上下文推理仍是大型语言模型的关键瓶颈，因为近期的循环记忆方法面临两个固有挑战：顺序按块更新可能覆盖早期的关键证据，后期无关内容;串行区块间依赖限制并行性，导致上下文长度增加延迟。为解决这些问题，我们提出了PI-Mem（并行迭代记忆）机制，该机制并行处理所有块，并在有限制的回合内迭代细化共享记忆。在每个回合中，PI-Mem根据当前内存并行读取所有块，从每个块中选择新的或互补证据，并将所选证据合并为下一回合的紧凑共享内存。为防止重复重复，我们通过强化学习优化工作流程，并设置辅助回合效率奖励，使模型在积累足够证据后能够自适应退出。我们在HotpotQA基准测试中评估PI-Mem，涵盖上下文长度最高360万个token，结果其表现优于重复记忆基线+6.25和+7.81绝对点，分别实现了6.1$\times$和2.1$\times$的推理加速。这些结果表明PI-Mem打破了长上下文推理中的准确性与效率权衡，并为极长文档的复杂多跳问题解答提供了可扩展的方法。

Towards More Expressive Spoken LLMs: Fine-Grained Intent Benchmarking and Acoustic-Lexical Decoupled Policy Optimization

迈向更具表现力的口语大型语言模型：细粒度意图基准与声学-词汇解耦策略优化

• Authors: Xiang Lin, Tian-Hao Zhang, Chunfeng Wang, Zhou Pan, Kun Zhan, Liang Li
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2608.03054
• Pdf link: https://arxiv.org/pdf/2608.03054
• Abstract Spoken emotional dialogue requires a model to understand a user's spoken input and generate a response that is both semantically appropriate and emotionally expressive. This is challenging because communicative intent may be stated explicitly in lexical content or conveyed more implicitly through paralinguistic cues, which can complement or diverge from the words themselves. However, two limitations constrain progress in this area: the scarcity of benchmarks that distinguish these intent expressions, and the lack of reinforcement learning objectives that jointly account for response quality and emotional expression. To address the lack of suitable benchmarks, we introduce ParaIntent, a Chinese benchmark comprising 14 intent categories with balanced explicit and implicit samples, together with a multidimensional evaluation protocol covering intent fulfillment, response quality, and emotional expression. For policy optimization, existing approaches either use a shared objective for text and speech or apply reinforcement learning to only one modality, leaving modality-specific learning signals entangled within policy optimization. Motivated by this, we propose Acoustic-Lexical Decoupled Policy Optimization (ALPO), which computes independent textual and acoustic advantages and routes them to the corresponding text and speech tokens within a unified rollout. Under identical reward functions and training budgets, ALPO improves over standard GRPO on most automatic metrics and achieves the best subjective results among the fine-tuned variants, with particularly clear gains in emotional expressiveness on both the synthetic and human-recorded test sets.
• 中文摘要 口头情感对话需要模型理解用户的口头输入，并生成既语义上合适又富有情感表达的回应。这具有挑战性，因为交际意图可能在词汇内容中明确表达，或通过副语言线索更隐含地传达，这些线索可以补充或偏离词语本身。然而，这一领域有两个限制：区分这些意图表达的基准稀少，以及缺乏能够共同考虑反应质量和情感表达的强化学习目标。为解决合适基准的缺失，我们引入了ParaIntent，这是一个中文基准，包含14个意图类别，具有显性与隐性均衡的样本，并配备了涵盖意图实现、反应质量和情感表达的多维评估协议。在策略优化方面，现有方法要么对文本和语音使用共享目标，要么仅对一种模态应用强化学习，使模态特定的学习信号纠缠在策略优化中。基于此，我们提出了声学-词汇解耦策略优化（ALPO），该方法计算独立的文本和声学优势，并将其路由到对应的文本和语音标记，形成统一的推广。在相同的奖励函数和训练预算下，ALPO在大多数自动指标上优于标准GRPO，在微调变体中实现最佳主观结果，尤其是在合成和人工录制测试集中，情感表达力均有明显提升。

CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning

CVPO：通过价值方差适应和动态课程学习提升LLM强化学习推理能力

• Authors: Ziqi Jia, Yalu Ouyang, Bo Pang, Panpan Li, Hangfei Xu, Shengzhao Wen, Shiyong Li, Yanpeng Wang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03068
• Pdf link: https://arxiv.org/pdf/2608.03068
• Abstract Reinforcement learning (RL) has emerged as an effective method for enhancing the reasoning capabilities of large language models (LLMs). However, existing methods suffer from insufficient precision in feedback on generated answer trajectories and exhibit the phenomenon of problem difficulty drift. To address these challenges, we propose CVPO - Curriculum-guided Value-Variance Policy Optimization. At the response trajectory level, we find that token-level value-variance correlates with exploration intensity. Our theoretical analysis shows this variance bounds policy update magnitude. We then use the estimated trajectory value-variance to quantify the intrinsic randomness in generation. Based on this, we design a variance-aware advantage adjustment mechanism for different reward types. At the question level, we introduce a dynamic curriculum weighting method that adapts to question difficulty. This helps the model focus on tasks matched to its current ability during each training stage. Experimental results show our method outperforms strong value-based baselines like VAPO. It achieves better performance and stronger exploration, enabling more accurate and robust reasoning in language models across various math tasks.
• 中文摘要 强化学习（RL）已成为提升大型语言模型（LLMs）推理能力的有效方法。然而，现有方法在生成的答案轨迹反馈精度不足，且存在问题难度漂移现象。为应对这些挑战，我们提出了CVPO——课程引导价值-方差策略优化。在响应轨迹层面，我们发现代币级价值-方差与探索强度相关。理论分析显示，该方差界限决定策略更新的幅度。随后，我们利用估计的轨迹值-方差量化生成中的内在随机性。基于此，我们设计了针对不同奖励类型的方差感知优势调整机制。在问题层面，我们引入了动态课程权重方法，适应问题难度。这有助于模型在每个训练阶段专注于与当前能力匹配的任务。实验结果显示，我们的方法优于像VAPO这样的强基于价值的基线。它实现了更好的性能和更强的探索，使语言模型在各种数学任务中能够进行更准确、更稳健的推理。

PAMT: Process-Aligned Reinforcement Learning for Multi-Domain Machine Translation

PAMT：多域机器翻译的过程对齐强化学习

• Authors: Yongshi Ye, Biao Fu, Chongxuan Huang, Yidong Chen, Xiaodong Shi
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.03077
• Pdf link: https://arxiv.org/pdf/2608.03077
• Abstract Multi-domain machine translation (MDMT) requires more than fluent generation: it demands domain-sensitive translation decisions such as domain disambiguation, terminology control, and stylistic adaptation. Large reasoning models (LRMs) make such decisions explicit through intermediate translation steps, but our analysis across 15 domains and four translation directions shows that this explicit reasoning is double-edged: it improves long-form and high-difficulty translation, yet often drifts in terminology-intensive and stylistically constrained settings. We trace this failure to a credit-assignment bottleneck: existing methods optimize final outputs or coarse trajectories, but cannot identify which translation steps actually help the final translation. To address this, we propose PAMT, a process-aligned training framework that combines cold-start domain-aware Long-CoT supervision with reinforcement learning. PAMT uses sequence-level format and outcome rewards for the final translation, together with a step-level process reward that measures how much each explicit translation step increases the likelihood of the reference translation. Across two backbones, PAMT improves over base models, outperforms MT-specialized baselines on average, and remains competitive with strong LLMs/LRMs across in-domain, OOD, and multilingual settings.
• 中文摘要 多领域机器翻译（MDMT）不仅需要流利生成，还需要领域敏感的翻译决策，如领域消歧义、术语控制和风格调整。大型推理模型（LRM）通过中间翻译步骤明确做出此类决策，但我们在15个领域和四个翻译方向的分析显示，这种显式推理具有双刃剑：它提升了长格式和高难度翻译，但在术语密集和风格受限的环境中常常漂移。我们将这一失败归因于学分分配瓶颈：现有方法优化最终输出或粗化轨迹，但无法确定哪些翻译步骤真正有助于最终翻译。为此，我们提出了PAMT，一种过程对齐的训练框架，结合了冷启动领域感知的长CoT监督与强化学习。PAMT采用序列级格式和结果奖励进行最终翻译，并结合一步级流程奖励，衡量每个显式翻译步骤对引用翻译可能性的提升程度。在两个骨干链上，PAMT优于基础模型，平均优于MT专业基线，并在领域内、OOD和多语言环境中与强大的LLM/LRMS保持竞争力。

SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation

SMOPD：通过专精合并在线政策提炼实现的多奖励强化学习

• Authors: Wen Wang, Jiahua Bao, Tu Yongsiqi, Yihao Liu, Haotian Zhou, Haoxuan Ma, Mengyu Zhou, Wenkui Fan, Junwei He, Xiaoxi Jiang, Guanjun Jiang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.03092
• Pdf link: https://arxiv.org/pdf/2608.03092
• Abstract We aim to improve model performance in multi-reward reinforcement learning training process. Existing Group reward-Decoupled Normalization Policy Optimization (GDPO) has mitigated the issue of reward signals masking one another during direct scalarization by normalizing each reward dimension separately before aggregation. However, our experiments show that GDPO still struggles to balance reward signals with different granularities. Specifically, in some particular training tasks, the model may receive a dense reward that assigns fine-grained scores ranging from 0.1 to 1.0, together with a sparse reward that provides only binary feedback of either 0 or 1. In such cases, we find that the sparse reward may provide an insufficient optimization signal, preventing its corresponding capability from being effectively reinforced. Therefore, how can we strengthen the optimization signal from the sparse reward without sacrificing the capability already learned from the fine-grained reward? To overcome this limitation, we propose Specialize-and-Merge Online Policy Distillation (SMOPD), a two-stage training method for multi-reward optimization. Stage1-Specialize: SMOPD first employs reward-priority configurations to train multiple reward-specialized teachers, allowing each reward to be learned under conditions where its signal can effectively drive optimization. Stage2-Merge: SMOPD then utilizes online policy distillation to combine the reward-specialized capabilities of these teachers into a single student policy, while maintaining balanced task-level optimization. To validate our method, we conduct experiments on two multi-reward settings: complementary rewards(tool-calling accuracy and format) and conflicting rewards (helpful and harmless rewards). Based on above settings, SMOPD outperforms GDPO across 1.5B, 3B and 7B backbones.
• 中文摘要 我们的目标是提升模型在多奖励强化学习训练过程中的性能。现有的组奖励解耦规范化策略优化（GDPO）通过在聚合前分别规范每个奖励维度，缓解了直接标量化过程中奖励信号相互掩盖的问题。然而，我们的实验显示，GDPO在平衡不同粒度的奖励信号方面仍然存在困难。具体来说，在某些特定训练任务中，模型可能获得一个密集的奖励，分配细粒度分数范围为0.1到1.0，同时奖励稀疏，仅提供0或1的二元反馈。在这种情况下，我们发现稀疏奖励可能提供的优化信号不足，导致其对应能力无法有效强化。因此，如何在不牺牲从细粒度奖励中学到的能力的前提下，增强稀疏奖励的优化信号？为克服这一限制，我们提出了专业化合并在线政策蒸馏（SMOPD），这是一种多奖励优化的两阶段培训方法。第一阶段-专业化：SMOPD首先采用奖励优先配置培训多位奖励专业教师，允许每个奖励在其信号有效驱动优化的条件下学习。第二阶段-合并：SMOPD随后利用在线策略提炼，将这些教师的奖励专业能力整合为单一学生政策，同时保持任务层级的平衡优化。为验证我们的方法，我们在两种多奖励设置上进行了实验：互补奖励（工具调用的准确性和格式）和冲突奖励（有益且无害的奖励）。基于上述设置，SMOPD在1.5亿、3亿和7亿骨干网络中表现优于GDPO。

Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL

凸包-邻域光滑对偶推广：在离线强化学习中控制局部修正传播

• Authors: Yi Yang, Zhennan Chen, Mingfeng Lv, Hanlei Li, Zhengsen Ruan, Lvqing Yang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.03108
• Pdf link: https://arxiv.org/pdf/2608.03108
• Abstract Offline reinforcement learning (offline RL) can benefit from nearby out-of-distribution (OOD) actions, but estimation errors at these actions may be amplified by bootstrapping. Existing regularization and local-generalization methods control either the admissible OOD region or the influence of generalized targets, often through separate mechanisms. We propose Convex Hull Neighborhood Smooth Dual Generalization (CSDG), which expresses the Bellman backup as an in-sample value target plus a CHN-local correction. This formulation makes the generalized contribution explicit and separates it from the in-sample reference path. The correction is obtained by smoothing in-sample-oriented and OOD-oriented candidates sampled at different perturbation radii. A mixture coefficient lambda scales its contribution to each backup, while the recursive discount remains gamma. Under boundedness and fixed perturbation kernels, we derive an exact one-step correction identity, a time-varying iterate bound, and a fixed-point bound that depends only on the branch discrepancy at the fixed point. We further characterize the implicit policies induced by the idealized operators and give a conditional non-degradation criterion. The practical algorithm approximates these quantities using asymmetric bounded noise and expectile regression, without exact support classification or an additional pessimistic OOD penalty. Experiments on Gym-MuJoCo and AntMaze show strong aggregate performance and stable value estimation. Code is available at: this https URL
• 中文摘要 离线强化学习（离线强化学习）可以受益于近域分布（OOD）动作，但这些动作的估计误差可能会被自助法放大。现有的正则化和局部推广方法控制可接受的OOD区域或广义目标的影响，通常通过不同的机制。我们提出了凸包邻域光滑对偶推广（CSDG），将贝尔曼备份表示为样本内值目标加上CHN局部修正。该表述使广义贡献显式化，并将其与样本内参考路径分离。修正通过对采样内和面向OOD的候选对象进行平滑，取样于不同微扰半径。混合系数λ对每个备份的贡献进行比例，递归折现保持伽马值。在有界性和固定微扰核下，我们推导出一个精确的一步修正恒等式、一个随时间变化的迭代界限，以及一个仅依赖于不动点分支差异的不动点界限。我们进一步表征理想算符诱导的隐式策略，并给出条件非降解准则。实用算法通过非对称有界噪声和期望回归近似这些量，无需精确支持分类或额外的悲观的OOD惩罚。Gym-MuJoCo和AntMaze上的实验显示了强劲的聚合性能和稳定的值估计。代码可访问：此 https URL

Shooting for Contact: Contact-Implicit Multiple Shooting for Dynamic Motion Retargeting

接触射击：动态运动重定向的接触隐式多重射击

• Authors: Sergio A. Esteban, Jason H. K. Siu, Derrick Mach, Junheng Li, Vince Kurtz, Joel W. Burdick, Aaron D. Ames
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.03116
• Pdf link: https://arxiv.org/pdf/2608.03116
• Abstract Motion retargeting approaches often prioritize kinematic similarity over whole-body dynamics, contact consistency, and actuation limits, yielding references that are difficult for reinforcement learning (RL) policies to reproduce, particularly for contact-rich behaviors. We present a contact-implicit, direct simulation-based multiple shooting (DSMS) framework that transforms kinematically feasible references into dynamically feasible whole-body trajectories. By embedding a differentiable simulator within a nonlinear program, DSMS resolves contact, friction, impacts, self-collision, and joint limits internally while enforcing tracking, actuation, and task constraints without prescribing a contact schedule or introducing explicit contact constraints. Compared with existing retargeting methods, DSMS accelerates motion-imitation RL training and yields policies with high success rates and low tracking error. We further demonstrate zero-shot sim-to-real transfer on the Unitree G1 through command-conditioned contact-rich crawling and a highly dynamic 180-degree jump-turn.
• 中文摘要 运动重定向方法通常优先考虑运动学相似性，而非全身动力学、接触一致性和执行极限，从而产生强化学习（RL）策略难以重现的参考，尤其是在接触丰富行为时。我们提出了一种接触隐式、基于直接仿真的多重射击（DSMS）框架，将运动学上可行的参考转化为动态可行的全身轨迹。通过在非线性程序中嵌入可微模拟器，DSMS内部解决接触、摩擦、碰撞、自碰撞和关节限制，同时强制跟踪、执行和任务约束，而无需规定接触计划或引入显式接触约束。与现有重定向方法相比，DSMS加速了运动模拟RL训练，并生成高成功率和低跟踪误差的策略。我们还进一步演示了Unitree G1上的零射模拟到实物传输，通过指令条件下的接触丰富爬行和高度动态的180度跳转。

Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

别偷看答案：无标签RLVR的Outcome-Masked Group 相对策略优化

• Authors: Yongshi Ye, Liang Zhang, Yidong Chen, Xiaodong Shi, Biao Fu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03119
• Pdf link: https://arxiv.org/pdf/2608.03119
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability. Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples. However, collapse arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning. We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization. OM-GRPO masks gradients on the answer span while retaining answer-level rewards through a soft consensus signal, shifting optimization pressure away from answer tokens. We further introduce Contrast-Augmented Reward, which refines reward estimation via low-cost pairwise comparisons over existing trajectories without additional rollouts. Across diverse reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms existing label-free RLVR methods and matches supervised GT-reward training with stable optimization. This stability is particularly beneficial in the Test-Time Training setting, where OM-GRPO surpasses majority voting by 4.24 points.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了LLM推理能力，但通常依赖基于实地（GT）答案，限制了可扩展性。基于投票的无标签RLVR用模型样本中的答案级共识取代了黄金监督。然而，当同一答案级信号同时用于估计奖励并驱动代币级策略优化时，模型会崩溃，从而促使模型直接强化答案代币，而非提升推理能力。我们提出了OM-GRPO，一种无标签RLVR框架，将奖励估计与策略优化解耦。OM-GRPO通过软共识信号掩盖答案跨度的梯度，同时保留答案级奖励，将优化压力从答案代币转移开来。我们进一步引入对比增强奖励，通过低成本的两对比较对现有轨迹进行优化，无需额外推广。在多种推理基准和三大大型语言模型骨干中，OM-GRPO持续优于现有无标签RLVR方法，并匹配监督下的GT奖励训练与稳定优化。这种稳定性在测试时训练环境中尤为有利，OM-GRPO在多数投票中高出4.24个百分点。

Process-Knowledge-Embedded Safe DRL for Real-Time Dispatch of Process Loads in Industrial Microgrids

工业微电网中工艺知识嵌入式安全日程学习，用于实时调度工艺负载

• Authors: Daniyaer Paizulamu, Lin Cheng, Fashun Shi, Yuchi Zhang, Zhaoyang Dong
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.03149
• Pdf link: https://arxiv.org/pdf/2608.03149
• Abstract Steelmaking process loads (SPLs) are flexible resources that enhance local renewable-energy utilization and reduce electricity procurement costs in industrial microgrids. However, strong multistage coupling makes current decisions affect subsequent feasibility, challenging conventional deep reinforcement learning to reduce costs while maintaining process feasibility throughout production. This paper proposes a process-knowledge-embedded safe deep reinforcement learning framework for the real-time dispatch of SPLs in industrial microgrids. Specifically, a lossless active-frontier action space is constructed, and a process-distance-guided action-processing mechanism reallocates excluded-action probabilities according to process distance and the actor's safe-action preference. Recursive process feasibility is established to guarantee admissible execution and feasible continuation. Furthermore, the expected process-correction distance is incorporated into PPO through a correction budget and a primal-dual update to internalize process knowledge into the raw policy, while a derived bound quantifies the raw policy's dependence on safety processing. Case studies using real-world data demonstrate zero process losses, electricity-cost reductions of 49.2% and 25.9% relative to rule-based scheduling and rolling MILP, respectively, within an acceptable computation time.
• 中文摘要 钢铁制造工艺负荷（SPL）是一种灵活的资源，能够提升本地可再生能源的利用率，并降低工业微电网中的电力采购成本。然而，强的多级耦合使当前决策影响后续可行性，挑战传统的深度强化学习在保持过程可行性的同时降低成本。本文提出了一种嵌入工艺知识的安全深度强化学习框架，用于工业微电网中实时调度SPL。具体来说，构建了一个无损的主动前沿动作空间，并采用过程距离引导的动作处理机制，根据过程距离和行为者的安全动作偏好重新分配排除动作概率。递归过程可行性建立，以确保可接受的执行和可行的延续。此外，预期过程修正距离通过修正预算和原始-对偶更新被纳入PPO，将流程知识内化为原始策略，而导出界限则量化原始策略对安全处理的依赖。使用真实世界数据的案例研究显示，在可接受的计算时间内，过程损失为零，电力成本分别相较于基于规则的调度和滚动MILP降低了49.2%和25.9%。

Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

测试时对比大型视觉语言模型：轨迹引导结构化采样方法

• Authors: Tianbao Jiang, Weicong Ni, Gerard de Melo, Linlin Wang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03204
• Pdf link: https://arxiv.org/pdf/2608.03204
• Abstract Post-training reinforcement learning (RL) algorithms are commonly used to align large vision-language models (LVLMs) with human intent and the requirements of visual reasoning tasks. However, existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions. To bridge this gap, we propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement, achieving better alignment with visual grounding and ensuring logical consistency. Our approach begins with curating a reasoning memory bank via a trajectory learning algorithm, which decomposes complex question solving into ordered sequences of predefined reasoning patterns. It subsequently accomplishes inference-time alignment by first collecting trajectories from reasoning memory bank to establish a global structural reasoning prior, and then using an iterative Markov Chain Monte Carlo (MCMC) algorithm for localized multi-objective refinement of the reasoning trace. Experiments across multiple multimodal reasoning datasets demonstrate that our approach significantly improves accuracy without incurring prohibitive inference overhead. These results establish trajectory-guided test-time sampling as a scalable and effective alternative to traditional post-training alignment, particularly for complex visual reasoning tasks.
• 中文摘要 训练后强化学习（RL）算法常用于将大型视觉语言模型（LVLM）与人类意图及视觉推理任务需求对齐。然而，现有基于强化学习的对齐方法往往资源密集，且在训练目标与推理时间分布之间存在不匹配。为弥合这一差距，我们提出了一种新型测试时间对齐方法，利用轨迹引导结构抽样进行动态推理时间细化，实现与视觉基础更好的对齐并确保逻辑一致性。我们的方法始于通过轨迹学习算法整理推理记忆库，将复杂问题解答分解为预定义的有序推理模式序列。随后，它通过先从推理记忆库收集轨迹以建立全局结构推理先验，然后使用迭代马尔可夫链蒙特卡洛（MCMC）算法对推理轨迹进行局部多目标细化，实现推理时间比对。跨多个多模态推理数据集的实验表明，我们的方法显著提升了准确性，同时避免了过高的推理开销。这些结果确立了轨迹引导测试时间采样作为传统训练后比对的可扩展且有效的替代方案，尤其是在复杂的视觉推理任务中。

Agentic Reinforcement Learning with Self-Distilled Reward Shaping

带有自我提炼奖励塑造的能动强化学习

• Authors: Ranxu Zhang, Guinan Chen, Chenshaodong, Jinghao Lin, Xiaozhou Xu, Sunzhe, Yanyong Zhang, Chao Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.03223
• Pdf link: https://arxiv.org/pdf/2608.03223
• Abstract Agentic reinforcement learning enables LLM agents to learn through interaction, but sparse trajectory-level rewards reveal success without identifying which intermediate decisions deserve credit. Training-only privileged skills can provide denser supervision by allowing the same frozen policy snapshot to rescore fixed tokens from skill-free trajectories while conditioned on task-matched procedural skills. Existing methods, however, do not jointly calibrate teacher scores across interaction steps, relate teacher confidence to realized returns, and integrate the resulting signal into native reward-to-advantage construction. We introduce Agentic Reinforcement Learning with Self-Distilled Reward Shaping (ADRS), a framework for constructing return-associated token-level credit for multi-turn language agents. ADRS centers and normalizes privileged token scores within each step, modulates them with a return-associated Teacher Value Advantage (TVA) gate based on within-group confidence--return association, and incorporates the gated token signal into native RL credit construction. Together, these components determine what the teacher prefers, when that preference is return-relevant, and how it enters the native reinforcement-learning credit path, while keeping rollouts and inference skill-free. Finally, experiments across three interactive benchmarks show that ADRS consistently improves performance on long-horizon tasks, with gains persisting across RL backbones, reduced-data settings, unseen tasks, and extended training. For anonymous review, our code is available at the following the link: this https URL
• 中文摘要 代理强化学习使LLM代理通过交互学习，但稀疏的轨迹级奖励无法揭示成功，却无法识别哪些中间决策应获得认可。仅限培训的特权技能可以通过允许同一冻结的策略快照在任务匹配程序技能条件下，从无技能轨迹重新评分固定代币，从而提供更密集的监督。然而，现有方法并未联合校准教师在交互步骤中的得分，未能将教师信心与实现回报联系起来，并将所得信号整合进原生的奖励到优势构建中。我们引入了带有自我提炼奖励塑造的代理强化学习（ADRS），这是一个构建多回合语言代理回报相关代币级积分的框架。ADRS在每一步中中心化并规范特权令牌分数，基于组内置信-回报关联，通过与回报相关的教师价值优势（TVA）门调节评分，并将门禁令牌信号纳入原生强化学习学分构建中。这些组成部分共同决定教师偏好何种偏好、何时该偏好与回报相关，以及如何进入本地强化学习学分路径，同时保持推广和推理无技能负担。最后，三个交互基准测试的实验显示，ADRS在长期任务中持续提升表现，且在强化学习骨干、减少数据设置、未见任务和扩展培训中均有持续提升。为匿名审核，我们的代码可在以下链接获取：此 https URL

PFM-HR: Pose Flow Matching for Humanoid Robots

PFM-HR：人形机器人姿态流匹配

• Authors: Yukang Gao, Yi Gu, Yangchen Zhou, Xingyu Chen, Zhaorui Wang, Fanghai Zhang, Hanyang Cao, Zhengyang Shen, Ji Ma, Runhan Zhang, Lei Han, Renjing Xu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.03227
• Pdf link: https://arxiv.org/pdf/2608.03227
• Abstract Motion priors improve reinforcement learning for physics-based humanoid tracking, but temporal priors require ordered motion clips, while pose priors provide limited guidance for policy-induced pose transitions. We present Pose Flow Matching for Humanoid Robots (PFM-HR), a reusable flow matching prior trained directly on large scale unordered pose data. PFM-HR introduces the Pose Geometry Score (PGS), which quantifies how joint coordinate changes during rollouts align with the local geometry of pose variation captured by the prior. Using PGS to modulate the tracking reward guides policy exploration toward structured pose changes while keeping the prior frozen across tracking tasks. Experiments demonstrate that PFM-HR improves both single motion and general motion tracking, especially for highly dynamic motions.
• 中文摘要 运动先验提升了基于物理的人形跟踪的强化学习，但时间先验需要有序运动剪辑，而姿态先验则为策略诱导的姿态转换提供有限指导。我们介绍了人形机器人姿态流匹配（PFM-HR），这是一种可重复使用的流程匹配，直接在大规模无序姿态数据上训练。PFM-HR引入了姿态几何评分（PGS），该指标量化了滚动过程中关节坐标变化与先验捕获的局部姿态变化几何的匹配度。利用PGS调制追踪奖励，引导策略探索，朝向结构化姿态变化进行，同时保持前置在跟踪任务中。实验表明，PFM-HR在单次动作和通用动作跟踪方面均有提升，尤其适用于高度动态的动作。

TimeRLM: Recursive Language Models Enable Precise Anomaly Localization in Long-Context Time-Series

TimeRLM：递归语言模型实现了长上下文时间序列中的精确异常定位

• Authors: Nicolas Zumarraga, Lorenzo Steno, Ning Wang, Max Rosenblattl, Thomas Kaar, Maxwell A. Xu, Kevin O'Sullivan, Markus Kreft, Elgar Fleisch, Paul Schmiedmayer, Patrick Langer, Robert Jakob
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03391
• Pdf link: https://arxiv.org/pdf/2608.03391
• Abstract Precise anomaly localization over long-context time series is a crucial task in monitoring applications across clinical care, industrial operations, financial services, and logistics, where brief evidence may hide inside long spans of high-frequency data. Time-Series Language Models (TSLMs) are able to ingest time series data and verbalize findings on anomalies in natural language; however, recent benchmarks report a decrease in retrieval performance at long contexts, mirroring failure modes in text, vision, and audio. In the text domain, Recursive Language Models (RLMs) can recover much of this lost performance by keeping context external to the large language model (LLM), allowing the model to query it through code. We present TimeRLM, an RLM formulation for time-series that sequentially manipulates the signal using code and vision capabilities. We further introduce AnomalyXL, a synthetic long-context anomaly localization benchmark with programmatically injected anomalies that require precise retrieval. We implement five different task categories and two variants: AnomalyXL-MCQ and AnomalyXL-Localize. TimeRLM outperforms every evaluated TSLM and single-pass baseline on four of the five AnomalyXL-Localize tasks, reaching 0.682 IoU on localization and 0.745 on classify-with-evidence, versus at most 0.329 and 0.072 across all baselines. We post-train TimeRLM using reinforcement learning. The resulting model further improves performance and requires approximately one-third as many agent interaction turns as its untrained base model to produce a final answer. On unseen real-world ECG, sleep and software observability recordings, the post-trained TimeRLM retains or improves performance, surpassing TSLMs despite being trained exclusively on synthetic data. Our findings suggest recursive interaction with time-series is an effective approach for long-horizon retrieval.
• 中文摘要 在长上下文时间序列上精确定位异常是监测临床护理、工业运营、金融服务和物流应用的关键任务，因为在这些领域，短暂证据可能隐藏在长时间的高频数据中。时间序列语言模型（TSLM）能够导入时间序列数据并以自然语言表达异常发现;然而，近期基准测试报告显示，长上下文下的检索性能下降，反映文本、视觉和音频的失败模式。在文本领域，递归语言模型（RLM）通过将上下文保持在大型语言模型（LLM）外部，使模型通过代码查询上下文，从而恢复了大部分丢失的性能。我们介绍TimeRLM，这是一种RLM时间序列表述，利用代码和视觉功能顺序操作信号。我们进一步介绍了AnomalyXL，这是一个合成长上下文异常定位基准测试，包含程序注入的异常，需要精确检索。我们实现了五个不同的任务类别和两个变体：AnomalyXL-MCQ和AnomalyXL-Localize。TimeRLM在五个AnomalyXL-Localize任务中的四个中表现优于所有评估过的TSLM和单次基线，定位达到0.682 IoU，有证据分类中达到0.745，而所有基线最多为0.329和0.072。我们使用强化学习对TimeRLM进行了后训练。所得模型进一步提升了性能，所需代理交互次数约为未训练基础模型的三分之一，才能得出最终答案。在未见的真实心电图、睡眠和软件可观测性记录中，经过训练的TimeRLM能够保留或提升性能，尽管仅基于合成数据训练，仍超过TSLMs。我们的发现表明，递归交互时间序列是长视野检索的有效方法。

Hierarchical Constrained Reinforcement Learning with Dynamic Boundary for Spatio-Temporal Vehicle-to-Grid Scheduling

具有动态边界的层级约束强化学习用于时空车辆到网格调度

• Authors: Haoyu Yan, Shutong Ding, Jiebao Zhang, Xi Yao, Yu Liu, Haoyu Wang, Chenchi Luo, Ye Shi
• Subjects: Subjects: Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2608.03409
• Pdf link: https://arxiv.org/pdf/2608.03409
• Abstract The rapid proliferation of Electric Vehicles (EVs) introduces significant spatio-temporal uncertainties into power grids, while Vehicle-to-Grid (V2G) technology offers critical flexibility through bidirectional power flow. However, integrating large-scale EVs into the Optimal Power Flow framework presents substantial challenges due to computational bottlenecks arising from solver complexity and coupled spatio-temporal constraints. Existing Reinforcement Learning (RL) methods often struggle to balance strict constraint satisfaction with scalability in highly dynamic EV fleet environments. To address these challenges, this paper proposes a Hierarchical Policy for Constrained Reinforcement Learning (HPC-RL) framework for spatially and temporally coupled V2G scheduling. The framework adopts a two-layer architecture: the upper level utilizes a RL algorithm based on the Generalized Reduced Gradient method to strictly enforce spatial grid-level hard constraints; the lower level implements a novel dynamic boundary strategy to compute real-time feasible charging power bounds for individual EVs, thereby ensuring the satisfaction of temporal charging demands. This integrated design not only enables the simultaneous handling of spatially and temporally coupled constraints during the RL optimization process but also significantly enhances generalization capabilities for large-scale fleets through hierarchical decoupling. Extensive experiments on IEEE 14, 30, and modified 141-bus systems demonstrate that HPC-RL outperforms Model Predictive Control and state-of-the-art safe RL baselines across all metrics. The proposed method achieves near-optimal scheduling strategies and drastically reduces online computation time in large-scale scenarios from hours to minutes, while maintaining a near-zero constraint violation rate and nearly 100\% charging demand satisfaction.
• 中文摘要 电动汽车（EV）的快速普及为电网带来了显著的时空不确定性，而车对网（V2G）技术则通过双向电力流提供了关键灵活性。然而，将大规模电动汽车整合进最优电力流框架面临重大挑战，因为求解器复杂性和耦合时空约束带来计算瓶颈。现有强化学习（RL）方法常在高度动态的电动车车队环境中难以平衡严格约束满足与可扩展性。为应对这些挑战，本文提出了一种用于空间和时间耦合V2G调度的受限强化学习（HPC-RL）分层策略框架。该框架采用两层架构：上层采用基于广义减梯度方法的强化学习算法，严格执行空间网格层的硬约束;下层采用新颖的动态边界策略，计算单个电动车的实时可行充电功率范围，从而确保满足时间充电需求。这种集成设计不仅支持在强化学习优化过程中同时处理空间和时间耦合约束，还通过分层解耦显著增强了大规模车队的泛化能力。对IEEE 14、30及改良版141总线系统的广泛实验表明，HPC-RL在所有指标上均优于模型预测控制和最先进的安全强化学习基线。该方法实现了近乎最优的调度策略，并将大规模场景下的在线计算时间从数小时大幅缩短到数分钟，同时保持近乎零的约束违规率和近乎100%的收费需求满足率。

When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPO

当正确解决方案重复出现：GRPO稀有度感知信用再分配

• Authors: Zhe Cao, Miaowen Wen, Fangjiong Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03467
• Pdf link: https://arxiv.org/pdf/2608.03467
• Abstract Reinforcement learning with verifiable rewards (RLVR) com- monly optimizes each correct completion as an independent learning signal. In GRPO, this completion-level uniformity creates structure-level skew: recurring correct solution forms accumulate positive coefficient mass in proportion to how often they are sampled, while rare forms receive limited credit. We formalize this behavior as multiplicity-induced structure-level credit concentration and introduce a partition- conditioned rule that redistributes positive advantages accord- ing to cluster rarity. Cue-GRPO instantiates this rule with- out auxiliary-model inference by using deterministic Strategy Cues to construct rollout-local partitions of verified-correct traces. Across Qwen2.5-Math-7B and Llama-3.1-8B-Instruct, Cue-GRPO improves AIME repeated-sampling performance, with the largest gains at high sampling budgets. Credit Re- distribution (CR) under Judge Partitions (JP) further indi- cates that the proposed redistribution mechanism can oper- ate with judge-derived partitions. Cue-GRPO adds only 6% wall-clock training overhead over GRPO. These results sup- port structure-level credit redistribution as a practical design axis for RLVR, with Strategy Cues providing a low-overhead implementation for competition mathematics. Code is avail- able at this https URL Repeat-Rarity-Aware-Credit-Redistribution-for-GRPO.
• 中文摘要 带有可验证奖励的强化学习（RLVR）com-m仅将每个正确完成作为独立学习信号进行优化。在GRPO中，这种完成级一致性产生了结构级偏斜：重复出现的正确解形式会根据其采样频率积累正系数质量，而罕见形式则获得有限的积分。我们将这种行为形式化为多重性诱导的结构级积分集中，并引入了一种划分条件规则，根据聚类稀有性重新分配正向优势。Cue-GRPO通过确定性策略提示构建已验证正确痕迹的部署局部划分，实现了该规则，无需辅助模型推断。在Qwen2.5-Math-7B和Llama-3.1-8B-Instruct中，Cue-GRPO提升了AIME重复采样性能，在高采样预算下获得最大收益。评判分割（JP）下的学分再分配（CR）进一步表明，拟议的再分配机制可以与法官派生的分配方式相匹配。Cue-GRPO相比GRPO仅增加了6%的墙时钟训练开销。这些结果支持结构级学分再分配作为RLVR的实用设计轴，策略提示为竞争数学提供了低开销实现。代码可在此 https URL Repeat-Rarity-Aware-Credit-Redistribution-for-GRPO 获取。

ToolLIFT: Lifting Tool-Specific Trajectories into Function-Level Graphs for Generalizable Tool Planning

ToolLIFT：将工具特定的轨迹提升到功能层面的图表中，实现可通用的工具规划

• Authors: Xiuhui You, Jiayi Luo, Zichao Shen, Qingyun Sun, Ziwei Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03468
• Pdf link: https://arxiv.org/pdf/2608.03468
• Abstract Historical tool-use trajectories provide valuable experience for large language model (LLM) agents to plan and coordinate tool usage. Existing approaches directly construct tool-level graphs from these trajectories, but the resulting graphs remain tied to specific tools and are hard to generalize across tool sets. To tackle this challenge, we find that despite differences in the tools involved, analogous tasks often share a common function-level workflow structure, which serves as a potentially more transferable abstraction for tool planning. Based on this insight, we propose ToolLIFT, a framework that lifts tool-specific trajectories into a function-level workflow graph (FWG) for generalizable tool planning. Specifically, we first propose a trajectory-lifting mechanism that encodes workflow structures in the FWG and shares collaboration experience across tools. Then, building on the global structure of the FWG, we introduce decoupled workflow planning and tool selection to align individual tool choices with the overall workflow. Lastly, to ensure reliable tool dataflow, we adopt Reinforcement Learning (RL) and propose source-gated and skill-specific rewards to maintain source-traceable information flow across tool calls. Experiments on two in-distribution (ID) and three out-of-distribution (OOD) benchmarks show that ToolLIFT consistently outperforms state-of-the-art baselines, demonstrating strong generalization to unseen tool sets.
• 中文摘要 历史工具使用轨迹为大型语言模型（LLM）代理规划和协调工具使用提供了宝贵经验。现有方法直接从这些轨迹构建工具级图，但最终生成的图仍绑定于特定工具，难以跨工具集泛化。为应对这一挑战，我们发现尽管涉及的工具存在差异，类似任务通常共享共同的功能级工作流结构，这为工具规划提供了更具可迁移性的抽象。基于这一见解，我们提出了ToolLIFT框架，该框架将工具特定的轨迹提升到功能级工作流图（FWG）中，实现可推广的工具规划。具体来说，我们首先提出了一种轨迹提升机制，将工作流结构编码在FWG中，并在不同工具间共享协作经验。随后，基于FWG的全局结构，我们引入了解耦式工作流规划和工具选择，使单个工具选择与整体工作流程保持一致。最后，为确保工具数据流的可靠性，我们采用了强化学习（RL），并提出源门和技能专属奖励，以维持工具调用间可追溯的信息流。在两个分布内（ID）和三个外部分发（OOD）基准测试上的实验显示，ToolLIFT始终优于最先进的基线，展现出对未见工具集的强有力推广能力。

Continue or Replan? Bernoulli-Continuation Policy Learning for Adaptive Horizon Execution

继续还是重新规划？伯努利-延续策略学习：自适应视界执行

• Authors: Weichen Xu, Zhenhua Liu, Lin Luo, Yaobo Liang, Chengtang Yao, Qingyu Mei, Jian Cao, Xixin Cao, Xing Zhang, Jiaolong Yang, Baining Guo
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03483
• Pdf link: https://arxiv.org/pdf/2608.03483
• Abstract Existing chunk-based Vision-Language-Action (VLA) models execute a fixed number of actions (i.e., execution horizon) before replanning, turning replanning into a task-agnostic periodic schedule that is independent of task progress. As a result, when no replanning boundary falls before a critical manipulation stage, it is executed from a stale chunk rather than a freshly replanned one. To address this limitation, we propose Bernoulli-Continuation Policy (BCP), a lightweight, plug-and-play framework for adaptive horizon execution that keeps the base VLA frozen. Given a fixed-length action chunk, its continuation head decomposes execution-horizon selection into a sequence of continue-or-replan decisions, which imposes an ordinal, prefix-sharing inductive bias over candidate horizons rather than treating them as independent classes. Since the optimal horizon for each chunk is not observable, we train this head with reinforcement learning from trajectory-level outcomes and introduce a Replanning-Efficiency Reward that jointly rewards task success and efficient VLA usage, discouraging the policy from collapsing to unnecessarily short horizons. On RoboTwin 2.0 with LingBot-VLA as the base policy, BCP improves the average success rate by +11.08% on 13 low-success tasks and from 89.88% to 93.94% (+4.06%) across all 50 tasks. Although trained only under the Clean setting, BCP generalizes to the Randomized setting, raising the average success rate by +4.06%. It also transfers to a different base policy $\pi_{0.5}$, achieving a better result on LIBERO (+1.7%) and, notably, on the harder LIBERO-PRO (+6.8%). On a real robot, BCP lifts success from 74% to 92% and from 44% to 84% on two manipulation tasks. Meanwhile, its negligible overhead, combined with higher success, makes BCP's overall runtime even lower than the fixed-horizon baselines.
• 中文摘要 现有基于块的视觉-语言-动作（VLA）模型在重新规划前执行固定数量的动作（即执行视野），使重新规划变成一个与任务无关的周期性计划，且不受任务进度影响。因此，当关键操作阶段之前没有重规划边界时，执行的是陈旧的部分，而非新重新规划的。为解决这一限制，我们提出了伯努利延续策略（BCP），这是一种轻量级的即插即用自适应视野执行框架，保持基础VLA冻结。给定固定长度动作块，其延续头将执行视野选择分解为一系列继续或重规划的决策，这对候选视界施加了序数、前缀共享的归纳偏见，而非将它们视为独立类。由于每个区块的最佳视野不可观察，我们通过从轨迹级结果进行强化学习训练该头部，并引入了重新规划-效率奖励，联合奖励任务成功率和高效使用VLA，防止策略陷入不必要的短视野。在以LingBot-VLA为基础策略的RoboTwin 2.0上，BCP在13个低成功任务中平均成功率提升+11.08%，在所有50个任务中从89.88%提升至93.94%（+4.06%）。虽然仅在清洁设置下训练，BCP可推广至随机设置，平均成功率提升+4.06%。它还能转移到不同的基础策略$\pi_{0.5}$，在LIBERO（+1.7%）和更难的LIBERO-PRO（+6.8%）上取得更好成绩。在真实机器人上，BCP在两个操作任务中成功率从74%提升到92%，从44%提升到84%。与此同时，其极低的开销加上更高的成功率，使BCP的整体运行时间甚至低于固定视野的基线。

Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks

混合大型语言模型增强强化学习代理用于复杂顺序决策任务

• Authors: Christophe D. Hounwanou, John Emeka Eze, Yaé Ulrich Gaba
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.03502
• Pdf link: https://arxiv.org/pdf/2608.03502
• Abstract Large Language Models (LLMs) have recently shown strong capabilities in reasoning, planning, and tool-use, enabling new forms of autonomous agents. However, LLM-based agents struggle with long-horizon sequential decision tasks that require precise action optimization and environment interaction. Reinforcement Learning (RL), while effective for sequential control, often lacks the high-level abstraction and task decomposition abilities needed for complex scenarios. This paper introduces an LLM-Augmented Reinforcement Learning Agent that integrates LLM-driven planning with RL-based action optimization. The proposed architecture leverages the LLM to generate subgoals, structured plans, and contextual guidance, while the RL agent refines low-level actions through interaction with the environment. Experiments on sequential decision tasks demonstrate improved sample efficiency, higher success rates, and more coherent action trajectories compared to RL-only and LLM-only baselines. This hybrid paradigm highlights a promising direction for building more capable autonomous systems.
• 中文摘要 大型语言模型（LLM）近年来展现出推理、规划和工具使用等强大能力，使自主智能体的新型态得以实现。然而，基于LLM的智能体在执行需要精确动作优化和环境交互的长期顺序决策任务时遇到困难。强化学习（RL）虽然对顺序控制有效，但通常缺乏复杂场景所需的高级抽象和任务分解能力。本文介绍了一种LLM增强强化学习代理，将LLM驱动的规划与基于强化学习的动作优化整合在一起。该架构利用LLM生成子目标、结构化计划和上下文指导，而强化智能体则通过与环境交互细化低层次动作。顺序决策任务的实验显示，与仅RL和仅LLM基线相比，样本效率提升、成功率更高，行动轨迹更连贯。这种混合范式凸显了构建更具能力自主系统的有前景方向。

Training Documents Reranker with Search Rubrics for Deep Research Agent

深度研究代理的培训文档重排序工具及搜索评分标准

• Authors: Wenhan Liu, Yu Lu, Qiaolin Xia, Hui Xu, Tong Zhao, Jian Xi, Yutao Zhu, Haijin Liang, Haibo Shi, Hao Wang, Zhicheng Dou
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.03527
• Pdf link: https://arxiv.org/pdf/2608.03527
• Abstract Retrieval systems help deep research agents generate high-quality answers by providing relevant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top-$k$ documents may not form a \textit{set} that satisfies the complex information needs of an agent query (\eg, diverse, concise and authoritative documents). In this paper, we propose search-oriented rubrics that \textit{explicitly} define the requirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical structure and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker \textbf{RubricRanker} to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine-tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that RubricRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks.
• 中文摘要 检索系统通过提供相关文档帮助深度研究代理生成高质量答案。然而，现有检索器通常通过相关性匹配选择文档，而单个高度匹配的$k顶级文档可能无法形成满足代理查询复杂信息需求的 \textit{set}（例如，多样化、简洁且权威文档）。本文提出以搜索为导向的评分标准，\textit{明确}定义高质量文档集应满足每个代理查询的需求。我们的检索评分标准组织为层级结构，并利用强大的大型语言模型进行综合。基于这些搜索评分标准，我们进一步训练文档重排序器 \textbf{RubricRanker}，从检索的文档中选择高质量子集。我们设计了一个两阶段训练框架，包括评分标准引导的监督微调和基于评分标准的强化学习。大量实验表明，RubricRanker 在四个深度研究基准中优于最强基准 2.6 分，并且在五个 RAG 基准上也能良好推广。

Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning

Hi-TTRL：调控共识并提供测试时强化学习的提示

• Authors: Kunbin Xu, Xingzuo Li, Xuefeng Bai, Kehai Chen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.03545
• Pdf link: https://arxiv.org/pdf/2608.03545
• Abstract Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.
• 中文摘要 测试时强化学习（TTRL）通过通过多数投票构建伪标签更新策略，提升无标签大型语言模型的推理能力。虽然有效，但多数投票分配的奖励信号对共识强度高度敏感，共识强度定义为在推广组中最常见答案的频率。在TTRL中，共识强度起双重作用：它既反映了伪标签的可靠性，也反映了优势的分布。低共识可以通过不成比例的巨大优势放大不可靠伪标签的更新，而高共识则降低奖励对比度，最终导致梯度消失。本文介绍了Hi-TTRL，一种测试时强化学习框架，利用抽样中的提示来调节推广共识强度。Hi-TTRL首先估计部分推广组的共识强度。当共识强度超出目标区间时，调用马尔可夫链蒙特卡洛（MCMC）提示采样器。采样器针对幂变前缀分布，并使用有限步近似抽样生成作为提示的rollout前缀。通过调整功率指数，Hi-TTRL生成功率目标加尖或平坦的提示，引导rollout共识强度朝目标区间方向移动。多数据集和骨干上的实验表明，Hi-TTRL相较标准TTRL持续提升，消融和共识引导分析验证了自适应提示引导共识调控的有效性。

Robust General Utility for Reinforcement Learning

强化学习的强健通用实用性

• Authors: Zixuan Liu, Fangzheng Wu, Brian Summa, Zizhan Zheng
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03562
• Pdf link: https://arxiv.org/pdf/2608.03562
• Abstract Reinforcement learning (RL) with general utility extends classic RL by optimizing an arbitrary utility functional of the policy-induced occupancy measure, thereby enabling a broader range of applications. However, previous work on general utility RL typically assumes the evaluation utility is fixed and correctly specified. In practice, the utility used at deployment can deviate from the training one, creating a robustness gap that prior work does not address. Motivated by this, we propose robust general-utility RL, a minimax learning framework that trains policies against utility misspecification within a prescribed uncertainty set. Our framework strictly generalizes standard general-utility RL while also providing a unified view of many existing RL frameworks, including reward-robust RL and constrained RL, through appropriate choices of the utility uncertainty set. We further develop provably convergent stochastic algorithms for two regimes. For concave utilities, we develop a projected stochastic gradient descent-ascent method and establish stationarity guarantees. For the more challenging nonconcave regime, we propose a stochastic prox-extragradient algorithm that mitigates ill-posed behavior induced by nonconcavity, with convergence guarantees to approximate first-order stationarity. Experiments on LLM safety alignment and exploration maximization tasks further corroborate the convergence behavior consistent with our theory.
• 中文摘要 具有通用效用的强化学习（RL）通过优化策略诱导占用度量的任意效用函数，扩展了经典强化学习，从而实现更广泛的应用范围。然而，以往关于通用效用RL的研究通常假设评估效用是固定且正确指定的。实际上，部署时使用的效用可能偏离训练效用，造成之前工作未能解决的鲁棒性差距。基于此，我们提出了稳健通用效用RL，一种极小极大学习框架，用于在规定的不确定性集中内训练策略以防止效用错误指定。我们的框架严格推广了标准通用效用RL，同时通过适当选择效用不确定性集，统一了许多现有RL框架，包括奖励稳健强化学习和受限强化学习。我们还进一步开发了两种模式的可证明收敛随机算法。对于凹效用，我们开发了一种预测随机梯度下降-上升方法，并建立了平稳性保证。对于更具挑战性的非凹形态，我们提出了一种随机近超梯度算法，以减轻非凹性引起的错态行为，并保证收敛近似一阶平稳性。LLM安全比对和探索最大化任务的实验进一步证实了收敛行为与我们理论一致。

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs

SFT冲突，强化学习共存：大型语言模型多任务学习的理论与实证分析

• Authors: Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03573
• Pdf link: https://arxiv.org/pdf/2608.03573
• Abstract Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.
• 中文摘要 监督式微调（SFT）和强化学习（RL）在增强大型语言模型（LLM）多任务推理方面表现出根本不同的行为。我们的初步实验揭示了一个现象：SFT在多阶段训练下存在严重的任务冲突，而RL则实现了多样化任务间的稳定共存。我们实证地将这一现象追溯到参数层面，观察到RL在任务间诱导稀疏且近似正交的更新。我们通过分析多任务梯度干扰，理论上解释了这一机制。我们的结果揭示了一个区别：SFT中的干扰是范数限制的，随绝对梯度大小扩展，而强化学习中的干扰是方差限制的，受优势归一化和策略优化引起的梯度方差限制。这种小方差界限导致任务间近正交的优化方向。基于这一见解，我们提出了Parallel-RL，一种将多任务训练解耦的范式，显著提升效率和灵活性。

Language-Specialized Multi-Teacher On-Policy Distillation for Multilingual LLM-Based ASR

多语言语言模型ASR的多教师政策提炼

• Authors: Yuan Xie, Jiaqi Song, Xianliang Wang, Ming Lei, Jie Gao, Jie Wu
• Subjects: Subjects: Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2608.03610
• Pdf link: https://arxiv.org/pdf/2608.03610
• Abstract Modern LLM-based ASR systems have established multilingual capability as a standard feature, leveraging large-scale multilingual corpora and LLMs' cross-lingual knowledge to achieve competitive performance across multilingual benchmarks. However, joint modeling of languages with heterogeneous acoustic, phonological, and lexical characteristics inevitably introduces optimization conflicts, undermining language-wise specialization. To address this challenge, we propose Language-Specialized Multi-Teacher On-Policy Distillation (LS-MOPD), which decouples language-specific knowledge acquisition from multilingual capability integration: language-specialized teachers are independently optimized via reinforcement learning (RL), after which their expertise is integrated into a generalist multilingual student through language routing and token-level multi-teacher distillation, thereby reducing direct cross-lingual optimization conflicts. We further explore two acoustic-prefix configurations, static and dynamic, to examine how teacher--student prefix consistency influences the efficacy of on-policy distillation. Experiments on benchmarks covering Mandarin, Mandarin subdialects, Cantonese, and English demonstrate that LS-MOPD substantially outperforms RL baselines and consistently surpasses the empirical performance envelope defined by best-performing RL teachers, revealing its potential to generalize beyond all teachers in multilingual ASR.
• 中文摘要 现代基于LLM的ASR系统已将多语言能力确立为标准特征，利用大规模多语言语料库和LLM的跨语言知识，在多语言基准中实现竞争性能。然而，对具有异质声学、语音和词汇特征的语言联合建模不可避免地引入优化冲突，削弱语言专业化。为应对这一挑战，我们提出了语言专业化多教师政策蒸馏（LS-MOPD），该方法将语言特定知识的习得与多语言能力整合分离：语言专业教师通过强化学习（RL）独立优化，随后其专业知识通过语言路由和代币级多教师蒸馏整合整合入通才多语言学生，从而减少直接的跨语言优化冲突。我们进一步探讨了静态和动态两种声学前缀配置，以考察师生前缀一致性如何影响政策提炼的有效性。在涵盖普通话、普通话子方言、粤语和英语的基准测试中，LS-MOPD显著优于强化学习基准，并持续超越表现最佳强化学习教师所定义的实证表现范围，展现出其在多语言ASR中超越所有教师的潜力。

Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details

种子间交叉对战足够吗？评估零次协调算法与实现细节的稳健性

• Authors: Maksymilian Wolski, Nicholas Hoernle, Johannes Forkel, Jakob Foerster
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.03644
• Pdf link: https://arxiv.org/pdf/2608.03644
• Abstract AI agents deployed in real-world settings must be capable of coordinating with humans and other AI agents they have not encountered before. Zero-shot coordination (ZSC) algorithms aim to achieve this by specifying high-level learning rules such that independently engineered agents can coordinate with each other at test time. Rigorous evaluation of ZSC algorithms remains difficult: ideally, multiple independent implementations of each proposed algorithm must be used, reflecting the variation that arises when independent parties interpret and implement the same specification. In practice, however, ZSC algorithms have almost exclusively been evaluated using a single implementation trained across different random seeds, with only a handful of works additionally varying the neural network architecture. This leaves open questions about robustness to specification ambiguities and implementation details. In this work, we provide the first systematic evaluation of this robustness. We introduce a new evaluation scheme, cross-implementation cross-play, varying implementation details that prior work has shown to affect the performance of multi-agent reinforcement learning (MARL) algorithms, and we evaluate Other-Play, a popular ZSC algorithm, with this scheme. Our findings are encouraging and suggest that, for Other-Play, the standard ZSC evaluation is, in fact, a reasonable proxy for this more thorough cross-implementation evaluation.
• 中文摘要 在现实环境中部署的AI代理必须能够与人类及其他未曾接触过的AI代理协调。零样本协调（ZSC）算法旨在通过指定高级学习规则实现这一点，使独立工程的代理能够在测试时相互协调。对ZSC算法的严格评估仍然困难：理想情况下，每个提出的算法必须使用多个独立实现，反映出独立方解释和实现同一规范时产生的差异。然而，实际上ZSC算法几乎完全是通过单一实现在不同随机种子上训练来评估，只有少数作品进一步改变了神经网络架构。这也留下了关于规范模糊性和实现细节的鲁棒性问题。在本研究中，我们首次系统地评估了这种鲁棒性。我们引入了一种新的评估方案——跨实现跨平台，结合先前研究显示对多智能体强化学习（MARL）算法性能有影响的实现细节，并用该方案评估了流行的ZSC算法Other-Play。我们的发现令人鼓舞，并表明对于Other-Play来说，标准的ZSC评估实际上是更深入交叉实现评估的合理代理。

Group Perspective Matters: Regulating Debate Relationships Can Mitigate Blind Conformity in Multi-Agent Debate

群体视角的重要性：规范辩论关系可以缓解多代理辩论中的盲目从众

• Authors: Hao Wu, Shoucheng Song, Chang Yao, Haoyu Wang, Huaiyu Wan, Youfang Lin, Kai Lv
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.03648
• Pdf link: https://arxiv.org/pdf/2608.03648
• Abstract Multi-Agent Debate (MAD) improves the reasoning performance of Large Language Models (LLMs) through multi-round interaction. However, LLMs in MAD are highly susceptible to blind conformity. Existing individual evaluation methods, typically based on confidence or perplexity, fail to reflect the correctness of reasoning and may even exacerbate blind conformity. To address this, we shift the perspective from individual evaluation to group interaction. We define mutual referencing among LLMs as \textbf{Debate Relationships} and recognize that regulating these relationships is the key to mitigating blind conformity. In this paper, we propose a novel framework for \textbf{D}ynamically r\textbf{E}gulating deb\textbf{A}te \textbf{R}elationships (DEAR) from the group perspective. At first, DEAR quantifies consensus and divergence as \textit{group evidence} to capture the debate state. Then, DEAR operates through three stages: 1) What: perceiving group consultation tendency and uncertainty; 2) Who: introducing a Selection RL-Agent to dynamically select reference peers; and 3) How: adopting a Behavior RL-Agent to adaptively adjust generation behaviors. Notably, we formulate the execution of the two RL-Agents as a sequential decision-making process, jointly optimizing via multi-agent reinforcement learning. Extensive experiments demonstrate that DEAR achieves superior performance while significantly reducing token consumption.
• 中文摘要 多智能体辩论（MAD）通过多轮交互提升大型语言模型（LLMs）的推理性能。然而，MAD中的LLM极易出现盲从。现有基于置信度或困惑度的个体评估方法无法反映推理的正确性，甚至可能加剧盲从从。为此，我们将视角从个体评估转向群体互动。我们将LLM之间的相互引用定义为\textbf{辩论关系}，并认识到调节这些关系是减轻盲从从的关键。本文提出了一个新框架，用于从群体视角对_textbf{D}ynamically r\textbf{E}gulating deb\textbf{A}te \textbf{R}关系（DEAR）进行。首先，DEAR 以 \textit{group evidence} 形式量化共识和分歧，以捕捉辩论状态。随后，DEAR 通过三个阶段运行：1）什么：感知群体协商倾向和不确定性;2）谁：引入选择性 RL 代理以动态选择参考对等人;3）如何：采用行为强化智能体以自适应调整生成行为。值得注意的是，我们将两个 RL 代理的执行方式设定为顺序决策过程，通过多智能体强化学习共同优化。大量实验表明，DEAR 在显著降低令牌消耗的同时实现了更优的性能。

CausalOPD: First-Wrong-Step Supervision for Distilling Causal Chain Reasoning

因果OPD：提炼因果链推理的首次错误监督

• Authors: Jian Zhang, Bingyi Wang, Yizhi Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03673
• Pdf link: https://arxiv.org/pdf/2608.03673
• Abstract Many critical reasoning tasks, including clinical diagnosis, legal judgment, and industrial fault diagnosis, require step-dependent causal chains in which early errors propagate and correct conclusions can mask invalid reasoning. Although large language models perform well on such tasks, privacy, latency, and controllability motivate distillation into locally deployable models. Standard trajectory imitation does not correct process errors on the student's own rollout distribution. We propose CausalOPD, a curriculum online process distillation framework. A knowledge-augmented teacher first provides trajectories grounded in domain-specific causal rules, entity relations, and structural constraints. The student then generates on-policy trajectories, and the teacher identifies the first wrong step, defined as the earliest transition that verifiably violates available constraints. Starting from the verified prefix, short-horizon reinforcement learning repairs this localized failure. A causal-stage curriculum advances from evidence-level to mechanism-level and conclusion-level errors, following their propagation order. Across three domains, CausalOPD improves average path correctness by 23.4 percentage points over sequence-level online process distillation and reduces the right-label-wrong-reasoning rate from 15.7% to 4.4%. The domain-specific 8B students also surpass both evaluated proprietary references in path correctness across all domains.
• 中文摘要 许多关键推理任务，包括临床诊断、法律判断和工业故障诊断，都需要阶梯依赖的因果链，早期错误传播，正确结论可以掩盖无效推理。尽管大型语言模型在此类任务中表现良好，但隐私、延迟和可控性促使将推理提炼成本地可部署的模型。标准轨迹模仿无法纠正学生自身推广分布中的过程错误。我们提出了CausalOPD，一种课程在线流程提纯框架。知识增强教师首先提供基于领域特定因果规则、实体关系和结构约束的轨迹。学生随后生成策略轨迹，教师识别第一个错误步骤，定义为最早可验证违反可用约束的过渡。从验证前缀出发，短视野强化学习修复这一局部失败。因果阶段课程从证据层面推进到机制层面和结论层面的错误，遵循其传播顺序。在三个领域中，因果OPD的平均路径正确性比序列级在线过程提炼提高了23.4个百分点，并将正确标签错误推理率从15.7%降至4.4%。领域特定8B学生在路径正确性方面也超过了评估过的专有参考文献。

DiagLoop: A Counterfactual Data Flywheel with Stage-Localized Reinforcement for Diagnostic LLMs

DiagLoop：一个带有阶段局部强化的反事实数据飞轮，用于诊断大型语言模型

• Authors: Jian Zhang, Bingyi Wang, Yizhi Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.03674
• Pdf link: https://arxiv.org/pdf/2608.03674
• Abstract Causal diagnostic models must explain how conclusions follow from evidence because diagnoses guide repairs and treatments. Yet serious cases are scarce, records rarely contain reasoning paths, and data transfer poorly across configurations, complicating local deployment. We present DiagLoop, a counterfactual data flywheel that converts codified physical relations or clinical guidelines, authored once per mechanism family, into training supervision beyond recorded cases. A training-only teacher proposes counterfactual worlds by varying causes, contexts, and observations, while an independent hybrid checker admits only valid worlds. The student reasons through symptom abstraction, causal-chain construction, and root-cause attribution. Stage-specific criteria identify its earliest failure. For nonterminal failures, a bounded repair probes downstream competence, and the resulting weakness profile guides subsequent data generation. Stage-localized reinforcement learning updates only the model-generated continuation, while replay and preservation reduce forgetting. The same criteria govern admission, attribution, reward, and regeneration through checks separate from the proposer. Using only synthesized scenarios and no case-level expert reasoning annotations, the resulting 8B model improves strict path correctness over the strongest conventional baseline. Gains are 11.6 points across eight industrial systems and 5.5 points across ten disease categories. Gains over a deranged-routing control are 3.9 and 2.3 points, respectively. The model also exceeds the evaluated proprietary references in both domains, even when they receive few-shot examples or the specification in context.
• 中文摘要 因果诊断模型必须解释结论如何从证据中得出，因为诊断指导修复和治疗。然而，严重病例稀少，记录中很少包含推理路径，且数据在不同配置间传输不佳，增加了局部部署的复杂性。我们介绍DiagLoop，一种反事实数据飞轮，将每个机制家族编写一次的编码物理关系或临床指南转化为超越记录病例的培训监督。仅培训教师提出基于不同原因、情境和观察的反事实世界，而独立混合检查者仅接受有效世界。学生通过症状抽象、因果链构建和根本原因归因进行推理。阶段特定标准识别最早的故障。对于非终极性故障，有界修复探测下游能力，由此产生的弱点画像指导后续数据生成。阶段局部强化学习仅更新模型生成的延续，而重放和保留则减少遗忘。相同的标准也用于通过独立于提议者的检查来管理录取、归因、奖励和再生。最终的8B模型仅使用综合场景，且无案例级专家推理注释，提升了严格路径正确性，优于最强的传统基线。在八个工业系统中提升了11.6个点，在十个疾病类别中提升了5.5个点。相比于紊乱路由控制，提升分别为3.9点和2.3个点。该模型在两个领域中也超过了评估后的专有参考文献，即使它们采用了少数样本样本或上下文中的规范。

PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloud

PhyAI：边缘的实时物理AI，云端可扩展部署

• Authors: Chenghua Wang, Daliang Xu, Dongqi Cai, Duojin Sun, Hao Zhang, Haoze Qian, Huaiyuan Zhang, Jinshuo Cui, Kezhao Zhao, Longxi Gao, Mengwei Xu, Rongjie Yi, Tianyue Zhang, Weikai Xie, Xiyuan Tan, Xuanzhe Liu, Yingying Qin, Yiwen Lu, Yuan Yao, Yuezhi Zu, Yunhan Guo, Ziqi Guo
• Subjects: Subjects: Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.03682
• Pdf link: https://arxiv.org/pdf/2608.03682
• Abstract Physical AI policies require inference throughout their lifecycle, including model evaluation, cloud reinforcement learning rollout, edge GPU serving, and onboard deployment. Although these settings share the same checkpoint and action semantics, they often rely on separate inference programs. To unify them, we build PhyAI, a Physical AI inference engine with a single runtime that keeps architecture-specific conditioning, solver, cache, and output logic in model adapters while sharing graph execution, kernels, memory management, and parallel services. The same codebase runs vision-language-action (VLA) models and world-action models (WAMs) on single or multiple GPUs across onboard, edge, and cloud deployments. We used the adapter interface to add MiniCPM-Robot on the day of its release. PhyAI achieves 1.40x-4.65x speedups over the official implementations of pi0, pi0.5, GR00T N1.7, and MiniCPM-Robot. On Cosmos3-Nano-Policy-DROID it reduces latency from 2.46 to 1.18 s on eight H20 GPUs (CFG=2, TP=4), a 2.08x speedup. Specialized runtimes remain faster in several configurations, so our goal is one runtime with competitive latency rather than the fastest result in every case. Detailed profiles reveal why different models need different execution policies: on a Hopper-series GPU at batch size one, the pi0.5 action expert accounts for 8.8% of FLOPs but 57.2% of latency; at batch size 32 its share drops to 13.5% and throughput reaches about 100 samples/s. Cosmos3 remains generation-dominated and gains only 14.3% throughput as batch size increases from 1 to 16. We further introduce the control-time Roofline, which distinguishes inference-bound from environment-bound control; the measured pi0.5 points on four LIBERO suites are environment-bound while Cosmos3 stays inference-bound. Code and benchmarks: this https URL.
• 中文摘要 物理AI策略需要在其生命周期内进行推理，包括模型评估、云强化学习推广、边缘GPU服务和板载部署。虽然这些设置共享相同的检查点和动作语义，但它们通常依赖于独立的推理程序。为了统一它们，我们构建了PhyAI，一个拥有单一运行时的物理AI推理引擎，在模型适配器中保持架构特定的条件、求解器、缓存和输出逻辑，同时共享图执行、内核、内存管理和并行服务。同一代码库在单GPU或多GPU上运行视觉语言动作（VLA）模型和世界动作模型（WAM），覆盖板载、边缘和云部署。我们利用适配器接口在MiniCPM-Robot发布当天添加了MiniCPM-Robot。PhyAI 在官方实现的 pi0、pi0.5、GR00T N1.7 和 MiniCPM-Robot 实现上实现了 1.40 倍到 4.65 倍的加速。在 Cosmos3-Nano-Policy-DROID 上，它将延迟从 2.46 秒降至 1.18 秒，八颗 H20 GPU（CFG=2，TP=4），提升了 2.08 倍。专用运行时在多种配置中依然更快，因此我们的目标是一个具有竞争力延迟的运行时，而不是每种情况下都追求最快的结果。详细的配置文件揭示了为何不同模型需要不同的执行策略：在批次 1 的 Hopper 系列 GPU 上，pi0.5 动作专家占 FLOP 的 8.8%，但延迟为 57.2%;在批处理 32 时，其份额降至 13.5%，吞吐量约为 100 个采样/秒。Cosmos3仍以世代为主，批次规模从1增加到16，吞吐量仅提升14.3%。我们进一步引入控制时间Roofline，区分推理约束和环境约束控制;四个LIBERO套件的pi0.5点是环境约束，而Cosmos3仍保持推理约束。代码与基准测试：此https URL。

GORDON: Graph-based Object-centric Rewards for Decomposition of Long-Horizon Manipulation

戈登：基于图的以对象为中心的长视野操作分解奖励

• Authors: Andrea Protopapa, Davide Buoso, Francesca Pistilli, Georgia Chalvatzaki, Giuseppe Averta
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.03753
• Pdf link: https://arxiv.org/pdf/2608.03753
• Abstract Learning long-horizon manipulation skills with reinforcement learning remains challenging due to the complexity of reward design, the limited guidance of sparse rewards, and the high cost of manual subtask annotation. Visual demonstrations can provide supervision for reward learning, but rewards learned from raw pixels can be brittle and sensitive to visual variation, background appearance, and robot motion. In this work, we propose GORDON, a graph-based object-centric reward learning framework that learns dense rewards from action-free video demonstrations. Each visual scene is represented as a graph of detected objects and spatial relations, and a graph neural network is trained in a self-supervised manner to embed these graphs into a task-aligned latent space. To align the representation with semantic task progress, we introduce an activity-aware weighted pooling mechanism that emphasizes task-relevant objects while masking robot-dominated motion. The dense reward is then computed as distances in the learned latent space of the current state to demonstrated goal configurations, providing a measure of task progress. In long-horizon tasks, the temporal profile of this reward reveals stage-wise object-state transitions, enabling automatic subtask discovery without manual segmentation. The discovered segments are then used to train subtask-specific rewards and specialized policies that are composed sequentially. Experiments on seven manipulation tasks on MAGICAL and ManiSkill3 benchmarks show that our object-centric reward improves reinforcement learning in short-horizon settings and enables successful policy learning in complex long-horizon tasks through automatic decomposition, achieving an average success rate of 74.4% across the long-horizon tasks (on average approximately +35 p.p. vs. best learned baseline and approximately +25 p.p. vs. oracle).
• 中文摘要 由于奖励设计复杂、奖励稀疏指导有限以及手动子任务标注的高成本，学习强化学习的长视野操作技能仍然具有挑战性。视觉演示可以为奖励学习提供监督，但从原始像素中学习的奖励可能脆弱且对视觉变化、背景外观和机器人运动敏感。本研究提出GORDON，一种基于图的对象中心奖励学习框架，通过无动作视频演示学习密集奖励。每个视觉场景以检测对象和空间关系的图表示，图神经网络以自我监督方式将这些图嵌入任务对齐的潜在空间。为使表示与语义任务进展对齐，我们引入了活动感知加权池机制，强调任务相关对象，同时掩盖机器人主导的运动。然后将密集奖励作为当前状态学习潜空间中与已展示目标配置的距离计算，提供任务进展度量。在长视野任务中，该奖励的时间轮廓揭示了各阶段的对象-状态转变，实现自动子任务发现，无需手动分割。发现的分段随后用于训练子任务特定奖励和顺序组合的专用策略。在MAGICAL和ManiSkill3基准测试中对七个操作任务的实验表明，我们以对象为中心的奖励提升了短期视野环境下的强化学习，并通过自动分解实现复杂长期任务中的策略学习成功，长视野任务的平均成功率为74.4%（平均约+35p.对最佳学习基线，约+25p.对oracle）。

AgenticVAU: Multi-Agent Explore-Verify Reasoning for Video Anomaly Understanding

AgenticVAU：多智能体探索-验证视频异常理解的推理

• Authors: Yuxiang Duan, Huining Li, Ao Li, Shuai Feng, Lanju Kong, Ning Liu, Jian Zhang, Xingdong Sheng, Yuntao Du
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.03779
• Pdf link: https://arxiv.org/pdf/2608.03779
• Abstract Video anomaly understanding (VAU) focuses on comprehensively interpreting abnormal events in videos, requiring models to identify anomalous occurrences, discover their supporting evidence, and explain the underlying causes beyond simple anomaly detection. Existing VAU methods often rely on specialized training or limited observations, restricting generalization or evidence coverage. Although single-agent alternatives support adaptive video observation, they still integrate exploration, observation, and decision-making within a unified reasoning process, offering limited role specialization and structured evidence coordination. To address these limitations, we present AgenticVAU, a training-free multi-agent framework that casts VAU as an explore--verify process, where the system first discovers potential anomalies and then verifies them through targeted observations. To achieve this, four specialized agents are introduced to handle visual-rule construction, search planning, video observation, and final decision, respectively. These agents communicate through an anchor registry, a shared evidence memory that binds each observation. Guided by this agent framework, AgenticVAU interleaves broad temporal exploration, dense local verification, and cross-interval comparison until sufficient evidence is collected. We conduct extensive experiments on the ECVA, UCF-Crime, and MSAD subsets of VAU-Bench, the results show that AgenticVAU outperforms zero-shot inference and reinforcement learning-based baselines, demonstrating the value of multi-agent collaboration for video anomaly understanding.
• 中文摘要 视频异常理解（VAU）侧重于全面解读视频中的异常事件，要求模型识别异常现象，发现其支持证据，并解释超越简单异常检测的根本原因。现有的VAU方法通常依赖专业训练或有限的观察，限制了泛化或证据覆盖。尽管单代理替代方案支持自适应视频观察，但它们仍将探索、观察和决策整合在统一的推理过程中，提供有限的角色专精和结构化证据协调。为解决这些限制，我们提出了AgenticVAU，一个无需训练的多代理框架，将VAU定位为一个探索-验证过程，系统首先发现潜在异常，然后通过有针对性的观察进行验证。为此，引入了四个专业代理，分别负责视觉规则构建、搜索规划、视频观察和最终决策。这些代理通过锚点注册表进行通信，锚点注册表是绑定每个观察的共享证据记忆。在该代理框架的指导下，AgenticVAU交错进行广泛的时间探索、密集的局部验证和跨区间比较，直到收集到足够的证据。我们在VAU-Bench的ECVA、UCF-Crime和MSAD子集上进行了大量实验，结果显示AgenticVAU优于零样本推断和基于强化学习的基线，证明了多智能体协作在视频异常理解中的价值。

History Matters: Meta-policy Delegation with Heterogeneous Multi-agent Reinforcement Learning

历史重要：采用异构多智能体强化学习的元策略委派

• Authors: Ziqing Lu, Avinash Reddy Mudireddy, Sarra Alqahtani, Weiyu Xu
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.03833
• Pdf link: https://arxiv.org/pdf/2608.03833
• Abstract AI agents are expected to play an increasingly important role in future decision-making systems. In this paper, we consider collaborative systems composed of heterogeneous multi-agent systems (MAS), where their members have different capabilities and operating costs. We study how agents can delegate tasks to one another so that certain research tasks can be completed effectively under resource-constrained scenarios. We first develop a multi-agent reinforcement learning-based (MARL) delegation training that enables agents to make sequential delegation decisions while minimizing the total execution cost. We then extend this approach to MARL with prescribed delegation topologies. Furthermore, we introduce two new frameworks for collaboration and delegation in multi-agent systems. The first framework proposes that an agent's policy depends not only on the current state of the underlying Markov decision process but also on the interaction history, including previous joint actions. This history-dependent formulation can improve coordination even in fully observable environments, where conventional MARL methods typically restrict policies to depend only on the current state. The second framework proposes a novel, potentially multi-dimensional monetary mechanism to facilitate the collaboration and delegation for MAS.
• 中文摘要 预计人工智能代理将在未来决策系统中扮演越来越重要的角色。本文探讨了由异构多智能体系统（MAS）组成的协作系统，其中成员拥有不同的能力和运营成本。我们研究智能体如何相互委派任务，以便在资源受限场景下有效完成某些研究任务。我们首先开发了一种多智能体强化学习（MARL）委托培训，使智能体能够在最小化总执行成本的同时做出顺序委托决策。随后我们将该方法扩展到MARL，采用规定的委托拓扑结构。此外，我们引入了多智能体系统中的协作与委托两个新框架。第一个框架提出，智能体的策略不仅依赖于底层马尔可夫决策过程的当前状态，还取决于交互历史，包括之前的联合行动。这种依赖历史的表述即使在完全可观测的环境中也能改善协调，因为传统MARL方法通常限制政策仅依赖当前状态。第二个框架提出了一种新颖且潜在多维的货币机制，以促进MAS的协作与委托。

FedCritic-MIMO: Communication-Efficient Serverless Federated Critic Learning for Massive-MIMO Resource Control in Open and Disaggregated 6G RANs

FedCritic-MIMO：用于开放和拆分6G RAN中大规模MIMO资源控制的通信高效无服务器联邦批评者学习

• Authors: Amin Farajzadeh, Melike Erol-Kantarci
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.03852
• Pdf link: https://arxiv.org/pdf/2608.03852
• Abstract This paper proposes FedCritic-MIMO, a communication-efficient serverless federated multi-agent reinforcement learning framework for AI-native resource control across independently deployable cell-level controllers in open and disaggregated 6G RANs. Controllers share no trainer, retain local actors and personalized critic components, and exchange only compatible shared critic parameters. FedCritic-MIMO targets reuse-$1$ multi-cell massive-MIMO OFDMA deployments, where RAN controllers jointly manage user scheduling, per-stream power allocation, beamforming, interference, and long-term QoS with limited inter-controller signaling. Each base station locally executes its actor without centralized training or actor federation, while critic knowledge is exchanged peer-to-peer over an interference-aware graph. It enables this collaboration through wireless-aware event triggering, adaptive layer-wise top-$k$ sparse critic exchange with error feedback, and balanced interference-aware fusion. We establish conditional finite-time stationarity and consensus guarantees for the balanced, compressed peer-to-peer critic recursion under a fixed-policy, frozen-target critic-regression model. In strongly interference-coupled reuse-$1$ simulations, FedCritic-MIMO achieves the best performance-communication tradeoff among heuristic, independent-learning, centralized-training, and communication-ablation baselines. It achieves the highest held-out throughput, improves user-rate distribution and mean SINR, increases QoS satisfaction, and attains the lowest interference cost per delivered bit among learning baselines. It reduces critic-communication overhead by $76\%$ relative to uncompressed distributed critic exchange. These results demonstrate that serverless exchange of compatible shared critic parameters can coordinate RAN controllers without centralized trajectory collection or parameter-server aggregation.
• 中文摘要 本文提出了 FedCritic-MIMO，这是一种高效的通信无服务器联邦多代理强化学习框架，用于在开放和拆分的 6G RAN 中独立部署的单元级控制器间实现 AI 原生资源控制。控制器不共享训练器，保留本地参与者和个性化的批评组件，仅交换兼容的共享批评参数。FedCritic-MIMO 针对重用的 1 美元多单元大规模 MIMO OFDMA 部署，其中 RAN 控制器共同管理用户调度、每流功率分配、波束形成、干扰和长期服务质量，且仅限于控制器间信令。每个基站本地执行其演员，无需集中训练或演员联合，而 CRRITIC 知识则通过干扰感知图点对点交换。它通过无线感知事件触发、自适应的层级最高$k美元稀疏批评者交换和错误反馈和平衡的干扰感知融合实现了这种协作。我们为固定策略、冻结目标批评者回归模型下的平衡压缩点对点批评递归建立了条件有限时间平稳性和共识保证。在强干扰耦合重用1美元模拟中，FedCritic-MIMO在启发式、独立学习、集中训练和通信消融基线之间实现了最佳的性能-通信权衡。它实现了最高的保留吞吐量，改善了用户速率分布和平均SINR，提高了服务质量满意度，并在学习基线中实现了最低的每交付比特干扰成本。相较于无压缩分布式批评者交换，它减少了76美元。这些结果表明，无服务器交换兼容的批评者参数可以在无需集中式轨迹收集或参数-服务器聚合的情况下协调RAN控制器。

EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning

EvoHIL：自我演进的奖励与流量匹配策略优化，实现稳健的人机在环强化学习

• Authors: Shuoqin Zhang, Tongtong Cheng, Xiru Gao, Jinzhuo Peng, Bin Zheng, Jiahao Tu, Ke Wang, Jia Pan, Zhe Hu, Kai Liu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.03872
• Pdf link: https://arxiv.org/pdf/2608.03872
• Abstract Human-in-the-loop reinforcement learning (HIL-RL) enables robots to learn contact-rich manipulation from limited real-world interaction, but deployment exposes three coupled limitations: static visual reward models fail under scene changes; independently sampled actions cause temporally inconsistent motion; and vision-based policies remain sensitive to appearance shifts. We present EvoHIL, a unified framework that adapts the reward model, action generator, and visual do main within a staged human-in-the-loop learning process. First, self-evolving reward (SER) adapts the success classifier from human-confirmed positives and provisional weak negatives. Second, Action Flow Stabilization (AFS) generates temporally coherent action chunks through flow matching, grounding policy updates in executed action prefixes and demonstrated behavior. Third, retention-aware offline fine-tuning replays relit interaction data while anchoring the AFS actor-critic to prior behavior, adapting the visual domain without additional robot interaction. Across six manipulation tasks on Franka FR3 and SO-101 arms under a controlled lighting shift, EvoHIL improves task success, agreement with human-confirmation labels, motion smoothness, and completion time relative to human-in-the-loop and imitation this http URL page: this https URL
• 中文摘要 人机在环强化学习（HIL-RL）使机器人能够从有限的现实世界互动中学习丰富的接触操作，但部署过程中暴露出三个耦合的局限：静态视觉奖励模型在场景变化下失效;独立采样的动作会导致时间不一致的运动;基于视觉的策略对外观变化保持敏感。我们提出了EvoHIL，一个统一框架，在分阶段的人工循环学习过程中适配奖励模型、动作生成器和视觉执行主控。首先，自我进化奖励（SER）将成功分类器从人类确认的正面和暂时弱负面中调整。其次，动作流稳定（AFS）通过流匹配生成时间上连贯的动作块，将策略更新基于已执行的动作前缀和演示行为。第三，保持感意识的离线微调重放重启交互数据，同时将AFS演员-批评者锚定于先前行为，适应视觉领域而无需额外的机器人交互。在Franka FR3和SO-101臂上，受控光照移换下，EvoHIL提升任务成功率、与人工确认标签的一致性、运动平滑度及相对于人工参与循环和模拟的完成时间

Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning

通过结构感知微调增强VLM奖励模型

• Authors: Pyrros Koussios, Chenhao Li, Xin Chen, Andreas Krause
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03875
• Pdf link: https://arxiv.org/pdf/2608.03875
• Abstract Designing effective reward functions remains a major bottleneck in Reinforcement Learning (RL). Recent work uses large foundation Vision-Language Models (VLMs) as reward models, computing text-observation similarity to bypass manual reward engineering. Although promising, these rewards are often noisy and unreliable, limiting their direct utility during deployment. We present Structure-Aware Fine-Tuning (SAFT), a simple, self-supervised method that refines these imperfect reward signals online without access to ground-truth supervision. SAFT leverages intrinsic structural priors to regularize the VLM's latent space via LoRA adapters. We rigorously evaluate SAFT across a spectrum of base model capabilities to demonstrate its versatility. Our results show that SAFT consistently denoises the reward landscape, yielding faster policy convergence and substantially improved alignment (EPIC distance) relative to the underlying base model, suggesting that failures can often be attributed to structural brittleness rather than semantic misunderstanding. By replacing extensive human preference annotation with structural inductive biases inherent to the task, SAFT offers a scalable path for stabilizing text-conditioned RL and underscores the broader value of incorporating task structure as a general inductive bias.
• 中文摘要 设计有效的奖励函数仍是强化学习（RL）中的一大瓶颈。近期工作利用大型基础视觉语言模型（VLMs）作为奖励模型，计算文本观察相似度以绕过人工奖励工程。尽管前景看好，但这些奖励常常噪声较大且不可靠，限制了其在部署时的直接效用。我们提出了结构感知微调（SAFT），这是一种简单的自监督方法，可在无需地面真实监督的情况下在线优化这些不完美的奖励信号。SAFT利用内在结构先验通过LoRA适配器规范VLM的潜在空间。我们严格评估SAFT在多种基础模型能力上的应用，以展示其多功能性。我们的结果表明，SAFT持续去噪奖励景观，带来更快的策略收敛和相较底层基模型显著改善的对齐（EPIC距离），表明失败往往可归因于结构脆弱性而非语义误解。通过用任务固有的结构归纳偏误替代大量人类偏好注释，SAFT为稳定文本条件强化学习提供了可扩展的路径，并强调将任务结构纳入一般归纳偏见的更广泛价值。

Latent Reward Registers for Diffusion Preference Alignment

扩散偏好对齐的潜在奖励寄存器

• Authors: Yuanshen Guan, Zipeng Feng, Zhiwei Xiong, Peiqin Sun
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.03929
• Pdf link: https://arxiv.org/pdf/2608.03929
• Abstract Aligning diffusion models with human preferences usually relies on a sparse terminal reward evaluated on the final generated samples, presenting a severe temporal credit-assignment challenge across the multi-step denoising process. We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents by prepending learnable, position-free register tokens to the input sequence of a frozen Diffusion Transformer (DiT). This independent readout mechanism extracts latent reward evidence without altering the generator's hidden states or velocity field. The resulting dense, differentiable reward signal throughout the full denoising process facilitates two alignment strategies. For training, Reward-Gradient On-Policy Distillation (RG-OPD) distills reward-guided updates along on-policy trajectories, bypassing the computationally expensive rollouts of standard policy gradients. For inference, Reward-Guided Sampling (RGS) steers trajectories via magnitude-matched reward gradients without parameter updates. Empirically, at high noise levels (u = 0.8), the registers reach the highest pairwise accuracy among the evaluated latent reward models. Furthermore, RG-OPD outperforms online reinforcement learning baselines while reducing GPU hours by up to 33x, and RGS establishes a new state-of-the-art among training-free methods, strictly enhancing both alignment and perceptual metrics. Code and weights are available at this https URL
• 中文摘要 将扩散模型与人类偏好对齐通常依赖于对最终生成样本评估稀疏终端奖励，这在多步去噪过程中带来了严重的时间信用分配挑战。我们提出了潜在奖励寄存器机制，这是一种通过在冻结扩散变换器（DiT）输入序列前置可学习、无位置寄存器令牌，直接从中间噪声潜在变量估算终端偏好的机制。该独立读出机制在不改变发生器隐藏态或速度场的情况下提取潜在奖励证据。在整个去噪过程中，获得的密集且可微分的奖励信号促进了两种对齐策略。在训练方面，奖励-梯度策略上蒸馏（RG-OPD）沿策略轨迹提炼奖励引导更新，绕过计算量高的标准策略梯度展开。作为推断，奖励引导抽样（RGS）通过幅度匹配的奖励梯度引导轨迹，且不更新参数。在高噪声水平（u = 0.8）下，寄存器在评估的潜在奖励模型中达到最高的两两准确率。此外，RG-OPD在减少GPU小时数最多33倍的情况下优于在线强化学习基线，RGS在无训练方法中建立了最先进的技术，严格提升了对齐和感知指标。代码和权重可在此 https URL 获取

Stochastic Multiple Shooting Trajectory Optimization via Sequential Local Policy Evaluation

通过顺序局部策略评估实现随机多重射击轨迹优化

• Authors: Ashwin Gupta, Joseph Moore
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.03978
• Pdf link: https://arxiv.org/pdf/2608.03978
• Abstract Stochastic single shooting trajectory optimization methods such as Model Predictive Path Integral control (MPPI) have been widely adopted in robotics due to their ability to reason about probabilistic dynamics and provide solutions where model gradients are noisy, costly to evaluate, or unavailable. However, satisfaction of terminal constraints when shooting over long action sequences is often sample inefficient, requiring a large number of iterations for convergence. In this paper, we present a stochastic multiple shooting method that optimizes short control action sequences connected via local feedback policies to improve sample efficiency and convergence to a terminal set. Additionally, we show that we are able to synthesize approximate system Jacobians purely from rollouts, making the method suitable for model-based reinforcement learning with black-box dynamics. We demonstrate the algorithm has improved sample efficiency and terminal set convergence for three nonlinear, underactuated optimization problems: a classic cartpole swingup task with analytical dynamics, a cartpole swingup task with learned neural network dynamics, and a VTOL quadplane performing a high angle-of-attack, precision post-stall landing maneuver.
• 中文摘要 随机单次射击轨迹优化方法如模型预测路径积分控制（MPPI）因其能够推理概率动力学并在模型梯度噪声大、评估成本高或无法获得的情况下提供解，已被机器人广泛采用。然而，在长动作序列拍摄时，终端约束的满足往往效率较低，收敛需要大量迭代。本文提出了一种随机多射击方法，通过局部反馈策略优化连接的短控制动作序列，以提高样本效率和收敛到终端集。此外，我们证明能够仅从展开合成系统雅可比近似，使该方法适合基于黑盒动力学的基于模型的强化学习。我们展示了该算法在三个非线性、欠致动优化问题的样本效率和终端集收敛性方面有所提升：经典的解析动力学侧极摆动任务、带学习神经网络动力学的侧极摆动任务，以及执行高攻角、高精度失速后着陆机动的垂直起降四翼机。

TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

TurnSight：工具整合推理的转向层次回顾自我提炼

• Authors: Changle Qu, Sunhao Dai, Hengyi Cai, Yuqi Zhou, Xinran Chen, Simon, Jun Xu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.04007
• Pdf link: https://arxiv.org/pdf/2608.04007
• Abstract Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at this https URL.
• 中文摘要 工具集成推理（TIR）使大型语言模型通过迭代工具交互解决复杂任务。然而，现有的强化学习方法通常依赖轨迹级监督，限制了长期TIR场景中的细粒度学分分配。策略上自我蒸馏通过具有特权上下文的教师分支提供了更密集的信号，但现有方法通常从地面真相答案或检索到的技能中推导出这些上下文，这些技能可能不反映代理实际访问的状态。此外，令牌级监督未能捕捉工具交互的回合级结构。为此，我们提出了TurnSight，一种回合级事后洞察自我蒸馏框架，直接从执行条件的事后洞察中推导监督。它构建多个具有不同前瞻性视野的事后视角，并通过跨视界方向一致选择可靠的监督。最后，选定的事后诸葛亮信号会在兄弟部署间进行归一化，并用于自适应调制强化学习优势，同时保持其原始优化方向。对三个基准测试的广泛实验证明了TurnSight的有效性。我们的代码可在此 https URL 获取。

Keyword: diffusion policy

A Hierarchical Approach to Imitation Learning for Manipulation Tasks Requiring Time Varying Forces

针对需要时间变化力的操作任务的层级模仿学习方法

• Authors: Rishabh Shukla, Adithya Santhosh, Shaili Gandhi, Samrudh Moode, Satyandra K. Gupta
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.03103
• Pdf link: https://arxiv.org/pdf/2608.03103
• Abstract Diffusion policies have shown strong performance in learning complex, multi-modal behaviors for robotic manipulation. However, their application to contact-rich disassembly tasks remains limited by a key trade-off: the iterative denoising process introduces inference latencies that makes high frequency control difficult, which is essential for realizing dynamic interactions such as chiseling and prying. Recent action-chunking techniques mitigate latency but use an open-loop execution window, rendering the system blind to rapid force transients caused by fracture events. To bridge this gap, we introduce the Diffusion Policy Augmented by Fast Trajectory Generation (DPA-FTG). Compared to recent visual-tactile approaches that focus on positional correction, DPA-FTG decouples low-frequency planning from high-frequency force regulation. At the high level ($5$ Hz), a conditional diffusion model predicts a sequence of latent parameters for selecting a strategy from a learned vocabulary of task primitives. At the low level ($60$ Hz), a lightweight, force-conditioned policy acts as a neural impedance controller, modulating execution in real-time to maintain contact stability. We validate our approach on a bimanual battery disassembly task involving the separation of a compliant sheet. Experimental evaluation demonstrates that DPA-FTG outperforms state-of-the-art baselines, including Reactive Diffusion Policy (RDP).
• 中文摘要 扩散策略在学习复杂多模态机器人操作行为方面表现出优异表现。然而，其在富接触拆解任务中的应用仍受限于一个关键权衡：迭代去噪过程引入了推理延迟，使得高频控制变得困难，而高频控制对于实现如雕刻和撬动等动态交互至关重要。近期的动作分块技术降低了延迟，但采用了开环执行窗口，使系统对断裂事件引起的快速力瞬态视为盲点。为弥合这一差距，我们引入了由快速轨迹生成增强的扩散策略（DPA-FTG）。与近期专注于位置修正的视觉-触觉方法相比，DPA-FTG将低频规划与高频力调控解耦。在高层（约5美元Hz），条件扩散模型预测一系列潜参数，用于从任务原语的学习词汇中选择策略。在低层（约60美元Hz）中，轻量级的力条件策略作为神经阻抗控制器，实时调制执行以维持接触稳定性。我们在涉及分离合规片的双手电池拆解任务中验证了我们的方法。实验评估表明DPA-FTG优于包括反应扩散策略（RDP）在内的最先进基线。

Bimanual Manipulation Within an 8 GB Budget: Zero-Copy Sensing and Quantized ACT on an Entry-Level Jetson

8 GB 预算内的双手操作：入门级 Jetson 上的零拷贝感应与量化 ACT

• Authors: Ekansh Singh, Eva Samuel, Alessandra Reneau, Ryan Schmeelk, Yashvi Gandhi
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.03938
• Pdf link: https://arxiv.org/pdf/2608.03938
• Abstract Bimanual manipulation policies trained with imitation learning are typically evaluated on workstation or datacenter-class GPUs, leaving the cost of deploying them on embedded hardware largely uncharacterized. We present a bimanual SO-101 system running entirely on an NVIDIA Jetson Orin Nano Super (8 GB), the entry-level tier of NVIDIA's embedded line, using a desktop GPU (RTX 3070) only for offline training, evaluated on pick-and-place of a deformable beanbag. First, we build a GStreamer capture pipeline backed by NVMM buffers that removes redundant host-device copies from three-camera sensing. Contrary to expectation, the conventional path fit the memory budget and dropped no frames; what zero-copy sensing recovers is CPU headroom (peak single-core utilization 98.0% to 77.0%) and worst-case latency (117.31 ms to 101.52 ms). Second, we train ACT and Diffusion Policy on identical demonstrations, each at its own reference budget (100k gradient steps for ACT, 200k for Diffusion Policy). ACT converges to a task-competent policy (19/20 trials) while Diffusion Policy does not converge to a usable one (0/10) even at twice the step count, which we attribute to differing convergence costs rather than an accuracy ceiling. Third, we convert ACT to TensorRT. FP16 reduces mean inference latency from 114.02 ms to 17.93 ms (6.4x) and INT8 to 12.65 ms (9.0x), with task success preserved at all three precisions (19/20, 18/20, 19/20). We report two findings not previously documented for ACT: TensorRT's general-purpose INT8 calibration quantizes the ResNet18 backbone but accepts zero of 145 transformer layers, explaining INT8's negligible size reduction over FP16 (0.9%) despite a further 28% latency gain; and the need for quantization is conditional on ACT's action-chunking configuration, feasible in full precision at n_action_steps = 100 but not at the per-step re-prediction temporal ensembling requires.
• 中文摘要 通过模仿学习训练的双手操作策略通常在工作站或数据中心级GPU上进行评估，因此在嵌入式硬件上部署的成本基本未被明确表征。我们展示了一台双手SO-101系统，完全运行在NVIDIA Jetson Orin Nano Super（8 GB）上，这是NVIDIA嵌入式产品线的入门级，使用桌面GPU（RTX 3070）进行离线训练，评估方式是可变形的豆袋椅。首先，我们构建了由NVMM缓冲区支持的GStreamer捕获流水线，去除三摄像头检测中的冗余主机设备副本。与预期相反，传统路径符合内存预算，且未丢弃任何帧;零复制感知恢复的是CPU余量（单核峰值利用率98.0%至77.0%）和最坏情况下延迟（117.31毫秒至101.52毫秒）。其次，我们用相同的演示训练ACT和扩散策略，每个演示各自使用不同的参考预算（ACT为10万梯度步，扩散策略为20万步）。ACT收敛为任务胜任策略（19/20试验），而扩散策略即使步数增加两倍也未收敛到可用策略（0/10），我们将此归因于收敛成本不同而非准确率上限。第三，我们将ACT转换为TensorRT。FP16将平均推理延迟从114.02毫秒降至17.93毫秒（6.4倍），INT8降至12.65毫秒（9.0倍），任务成功率在三者精度（19/20、18/20、19/20）下均保持。我们报告了两个此前未被记录在ACT上的发现：TensorRT的通用INT8校准对ResNet18骨干进行量子化，但接受145层变换器中的零层，这解释了INT8相较FP16体积可忽略的0.9%缩小，尽管延迟增加了28%;量化的需求取决于ACT的作用分块配置，该配置在n_action_steps = 100时可行，但无法达到每步时间集合所需的每次重新预测。