生成时间: 2026-08-24 16:49:10 (UTC+8); Arxiv 发布时间: 2026-08-24 20:00 EDT (2026-08-25 08:00 UTC+8)

今天共有 20 篇相关文章

Keyword: reinforcement learning

Multilingual Verifier Bias in RLVR: Benchmark, Rollout Diagnosis, and the Cross-Lingual Selection Bottleneck

RLVR中的多语言验证偏差：基准、推广诊断与跨语言选择瓶颈

• Authors: Chenyu Zhou, Qiliang Jiang, Xu Zhou
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.20362
• Pdf link: https://arxiv.org/pdf/2608.20362
• Abstract Reinforcement learning with verifiable rewards (RLVR) is a standard recipe for training large language models on mathematical reasoning, where an answer verifier serves as a language-neutral reward function. We show that this assumption fails in multilingual settings: an exact-match verifier turns format and script variation into language-dependent false-negative reward noise. We introduce a reusable protocol for auditing multilingual RLVR rewards: a verifier-robustness suite, a rollout-diagnosis procedure, and language-conditioned reward-error metrics for Japanese, English, and Chinese answers. On MGSM rollouts with k=8, the exact-match proxy rejects trusted-correct answers at sharply different rates by language across Qwen3-4B, Qwen3-8B, and Llama-3.1-8B-Instruct; for Qwen3-8B, the false-negative rate reaches 0.642 on JP against 0.122 on EN and 0.073 on CN. A plain-numeric probe localizes the mechanism to the final-answer interface: an interface model drives reward-error VLB to zero while the residual accuracy gap is unchanged. We then expose a cross-lingual selection bottleneck: on MGSM250 rollouts, a target-local aggregation rule using no trusted labels closes 55-78% of the average selection gap, and over 95% of repairs require genuine cross-lingual support. The bottleneck replicates on a 483-problem MATH-500 set. A controlled training audit shows that rule-GRPO raises trusted accuracy while the reward-error VLB stays high. The unifying message is operational: multilingual RLVR rewards should be audited by language and by answer interface before they are optimized.
• 中文摘要 带可验证奖励的强化学习（RLVR）是训练大型语言模型数学推理的标准配方，其中答案验证器作为语言中立的奖励函数。我们证明该假设在多语言环境中失效：精确匹配验证器将格式和脚本变体转化为依赖语言的假阴性奖励噪声。我们引入了可重用的多语言RLVR奖励审计协议：验证者-鲁棒性套件、推广诊断程序，以及针对日语、英语和中文答案的语言条件奖励-错误度量。在k=8的MGSM推广中，精确匹配代理在Qwen3-4B、Qwen3-8B和Llama-3.1-8B-Instruct语言间以截然不同的语言拒绝可信正确答案;对于Qwen3-8B，JP的假阴性率达到0.642，而EN为0.122，CN为0.073。明数值探测将机制定位于最终答案接口：接口模型将奖励误差VLB降至零，而剩余准确率差距保持不变。随后我们暴露了一个跨语言选择瓶颈：在MGSM250部署中，使用无可信任标签的目标-本地聚合规则能弥补平均选择差距的55-78%，且超过95%的修复需要真正的跨语言支持。瓶颈在483个问题的MATH-500集合上得以复制。受控训练审计显示，规则-GRPO提高了可信度准确性，而奖励-误差VLB保持高位。统一的信息是操作性的：多语言RLVR奖励应在优化前按语言和答案界面进行审计。

World models of environment, agent and joint agent-environment systems

环境、代理及联合代理-环境系统的世界模型

• Authors: Manuel Baltieri, Filippo Torresan, Yivan Zhang, Alexander Boyd, Fernando E. Rosas
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.20401
• Pdf link: https://arxiv.org/pdf/2608.20401
• Abstract World models are a central component of model-based reinforcement learning. They are usually discussed in terms of what variables they predict, such as observations, rewards, states, latent or information states. We argue that there is a prior distinction: which channel they model. We consider three cases: the environment channel $O_{:} \mid A_{:}$, the agent channel $A_{:} \mid O_{:}$, and the realised joint process $(A, O)_{:}$, equivalently viewed as a channel with no inputs. Using computational mechanics, we define canonical predictive models for these three cases as $\epsilon$-transducers or $\epsilon$-machines. Canonical environment models recover standard predictive state representations, while the other two give analogous notions of canonical models for the agent and the joint system. We then build canonical support-restricted environment and agent models induced by closed-loop coupling, whose predictive equivalences range over continuations supported by the realised interaction. The key structural result is that canonical support-restricted environment states factor through the canonical joint causal states, and their transition structure is induced directly from the joint model; the agent-side construction is dual. Finally, we give a POMDP/controller example in which the unrestricted environment model has infinitely many states while the canonical support-restricted model induced by the coupling is finite. The framework clarifies what different world models are models of, and how coupling and support restriction can change their canonical predictive structure and complexity.
• 中文摘要 世界模型是基于模型的强化学习的核心组成部分。它们通常以预测变量为基础进行讨论，如观察、奖励、状态、潜在或信息状态。我们认为存在先行区分：它们建模的通道。我们考虑三种情况：环境通道 $O_{：} \mid A_{：}$，代理通道 $A_{：} \mid O_{：}$，以及实现的联合过程 $（A， O）_{：}$，等价地视为无输入的通道。利用计算力学，我们将这三种情况的典型预测模型定义为 $\ε$-换能器或 $\ε⁻⁹-机器。规范环境模型恢复标准的预测状态表示，而另外两种模型则为代理和联合系统提供了类似的规范模型概念。随后，我们构建了由闭环耦合诱导的典范支持限制环境和代理模型，其预测等价性涵盖于由实现相互作用支持的延续期。关键结构结果是，规范支持限制环境状态通过规范联合因果状态进行分解，其过渡结构直接由联合模型诱导;代理侧构造为对偶。最后，我们举例POMDP/控制器，其中无限制环境模型具有无限多个状态，而耦合诱导的典型支持限制模型是有限的。该框架阐明了不同世界模型的模型，以及耦合和支持限制如何改变它们的典型预测结构和复杂性。

From Thermal Preference Prediction to Adaptive Thermal Intervention: A Reinforcement Learning Approach Using Physiological and Environmental Sensing

从热偏好预测到适应性热干预：利用生理与环境感知的强化学习方法

• Authors: Isibor Kennedy Ihianle, Emmanuel Manu, Ehsan Asnaashari, Mojgan Jadidi, Pedro Machado, Amrit Sagoo, Ahmad Lotfi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.20423
• Pdf link: https://arxiv.org/pdf/2608.20423
• Abstract Personalised thermal comfort is essential for occupant wellbeing and for the development of more responsive building-control strategies, yet conventional Heating, Ventilation, and Air Conditioning (HVAC) systems rely on static setpoints and population-level comfort models that fail to capture individual physiological variability. This paper presents a two-stage personalised thermal comfort approach integrating multimodal physiological and environmental sensing with reinforcement learning-based decision-making.
• 中文摘要 个性化热舒适对居住者福祉和更响应的建筑控制策略的发展至关重要，但传统的供暖、通风与空调（HVAC）系统依赖静态设定点和人口层面舒适模型，未能捕捉个体生理变异。本文提出了一种两阶段个性化热舒适方法，将多模态生理和环境感知与基于强化学习的决策相结合。

Annotations as Rollouts: Efficient and Scalable Reinforcement Learning for Video MLLMs

注释作为推广：视频多层次营销的高效且可扩展的强化学习

• Authors: Yunheng Li, Guohong Mu, Hao Li, Shengsheng Qian, Dingwen Zhang, Qibin Hou, Ming-Ming Cheng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.20492
• Pdf link: https://arxiv.org/pdf/2608.20492
• Abstract Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation. In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target. Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage. Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts. Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.
• 中文摘要 多模态大型语言模型（MLLM）已成为统一视频感知的主流范式。然而，在大型多任务数据集上进行后期训练仍然具有挑战性，因为现有强化学习方法采样的策略组，即使使用了昂贵的思维链（CoT）生成，高质量的推广也很少。本文研究了视频MLLM后训练中强化学习的样本效率和可扩展性，并介绍了OraRL。我们识别了一个被忽视的角色：除了给推广评分外，每个注释还可以将其策略组作为预言机展开进入，作为直接的正向优化目标。然而，直接的预言机集成并非简单：高回报的预言机会提升组基线，并反转本应是积极的策略优势，这种失败被称为优势倒置。OraRL的核心是一个解耦优势估计器：策略推展确定无oracle基线，而oracle-policy缺口则调制方向性增益和独立的oracle优势。符号平衡剪枝提高了效率：通过仅保留每个符号的oracle和最强的推展，OraRL只需2.2倍于SFT的步长，不到CoT下GRPO所需4.9倍的一半。OraRL随模型规模和数据扩展，主干量从0.8B提升到9B，GRPO提示最多可达100k个。无需思考链，Video-ORA-9B解码时间为130毫秒，而非4780毫秒。与之前的最佳模型相比，它将时间mIoU从62.5提升到66.0，AO从73.0提升到78.2，从64.3提升到70.4，三个基准空间智能宏观平均值从51.0提升到56.1;在VSI-Bench上，它得分为73.1，GPT-5为55.0，Gemini-3-Pro为55.1。

AgentMercury: Your Agent Can Synthesize Verifiable Environments for Business Scenarios at scale

AgentMercury：您的代理可以大规模综合可验证的业务场景环境

• Authors: Minbyul Jeong, Chanwoong Yoon
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.20634
• Pdf link: https://arxiv.org/pdf/2608.20634
• Abstract Agents learn to act through interaction with environments, yet the environments used for training are often manually constructed or synthesized around predefined tasks and benchmarks. This task-centric paradigm makes it difficult to scale environments that reflect realistic and evolving workflows where diverse tasks can naturally emerge from the underlying world. We introduce AgentMercury, a scalable framework for synthesizing executable environments from high-level business scenarios. Rather than constructing an environment for a specific task, AgentMercury first instantiates a persistent world with entities, services, tools, state, and executable cross-service invariants, from which diverse tasks and interaction trajectories can subsequently emerge. We construct 4,783 executable environments spanning 14 industries and 50 countries, and use them as training substrates for reinforcement learning. Despite being generated without targeting the evaluation benchmarks, policies trained on these business-oriented environments improve substantially on both enterprise workflows and out-of-domain benchmarks spanning reasoning, coding, scientific computing, and tool use. In our experiments, Qwen3.5-4B improves from 12.3 to 15.7 on EnterpriseOps-GYM and from 45.9 to 56.0 on AIME26 after training on AgentMercury environments. We further show that the construction process itself can be learned: fine-tuning Qwen3.5-35B-A3B on construction traces increases executable-world authoring success from 3.3% to 83.3% on held-out business scenarios. These results show that scenario-grounded environments can provide useful and generalizable learning signals beyond benchmark-specific training, while their construction can itself become a learnable capability.
• 中文摘要 代理通过与环境的交互学习行动，但用于训练的环境通常是手动构建或综合的，围绕预定义的任务和基准测试。这种以任务为中心的范式使得能够反映现实且不断演变的工作流程的环境难以扩展，从而让多样任务自然地从底层世界中产生。我们介绍了AgentMercury，一个可扩展的框架，用于从高层业务场景中综合可执行环境。AgentMercury不是为特定任务构建环境，而是首先实例化一个持久世界，包含实体、服务、工具、状态和可执行的跨服务不变量，从中可以产生多样化的任务和交互轨迹。我们构建了涵盖14个行业和50个国家的4783个可执行环境，并将其用作强化学习的训练基础。尽管在这些面向业务的环境中训练的策略在企业工作流程和跨域基准测试（涵盖推理、编码、科学计算和工具使用）方面均有显著提升。在我们的实验中，Qwen3.5-4B在EnterpriseOps-GYM上的等级从12.3提升到15.7，在AIME26上从45.9提升到56.0，在AgentMercury环境中训练后。我们还进一步证明，构建过程本身是可以学习的：在构建跟踪上微调Qwen3.5-35B-A3B，可将可执行世界在未完成的业务场景中成功率从3.3%提升到83.3%。这些结果表明，基于场景的环境可以提供有用且可推广的学习信号，超越基准特定训练，而其构建本身也可成为可学习的能力。

Why2Speak: Faithful Reasoning for Abstaining Action Policies

Why2Speak：忠实理由支持弃权行动政策

• Authors: Shreya Mendi, Brinnae Bent
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.20670
• Pdf link: https://arxiv.org/pdf/2608.20670
• Abstract Many agentic systems must repeatedly choose between acting and abstaining, making faithful reasoning important for oversight: an explanation is useful only if it reflects the computation that produced the action. We study this problem through intervention timing in multi-party conversation, where an assistant must decide whether to speak or remain silent. This setting exposes class imbalance, asymmetric action costs, and the possibility that exposing reasoning changes the policy being audited. Using Qwen3-8B, decoded with or without chain-of-thought reasoning, we compare direct decision policies, reasoning policies, supervised fine-tuning, and reinforcement learning. We find a capability-auditability tradeoff: the strongest direct policy achieves higher quality but exposes no reasoning to inspect, while the reasoning policy provides a trace at the cost of lower performance, particularly recall of true intervention opportunities. Supervised fine-tuning either suppresses reasoning or preserves it without improving decision quality, while reinforcement learning also fails to improve the reasoning policy. We identify one mechanism underlying this failure: group relative objectives provide no learning signal on confidently wrong prompts when sampled rollouts all select the same action. Controlled activation probes and behavioral ablations show that standard faithfulness methods can overstate evidence that exposed reasoning reflects the underlying decision process. Probability-based metrics saturate under confident decisions, probes are vulnerable to class imbalance and textual leakage, and reasoning ablations can confound reasoning content with changes in inference mode. Together, these results show that exposing reasoning can change an agent's action policy rather than simply make it observable. We provide controls for evaluating reasoning-based oversight of agents that can act or abstain.
• 中文摘要 许多能动系统必须反复在行动与回避之间做出选择，使得忠实推理对监督至关重要：解释只有反映产生该行动的计算时才有用。我们通过多方对话中的干预时机来研究这个问题，助理必须决定是说话还是保持沉默。这种设定揭示了阶级失衡、行动成本不对称，以及暴露推理可能改变被审计策略的可能性。利用Qwen3-8B，无论是否使用思维链推理解码，我们比较了直接决策策略、推理策略、监督微调和强化学习。我们发现了能力与可审计性权衡：最强的直接策略能实现更高质量但不暴露可检视的推理，而推理策略则提供痕迹，但代价是性能下降，尤其是真正干预机会的回忆。监督式微调要么抑制推理，要么保留推理而不提升决策质量，而强化学习同样未能改善推理策略。我们识别出这一失败的一个机制：当抽样的推广全部选择相同动作时，群体相对目标对自信错误的提示无学习信号。受控激活探针和行为消融表明，标准忠实度方法可能夸大暴露推理反映潜在决策过程的证据。基于概率的指标在自信决策下过于饱和，探测器易受类失衡和文本泄露影响，推理消融可能使推理内容与推理模式变化混淆。综合来看，这些结果表明，暴露推理可以改变智能体的行动策略，而不仅仅是使其可被观察。我们提供了评估基于推理的智能体监督的控制，这些智能体可行动或回避。

Reinforcement Learning for Continuous-Time Jump Markov Decision Processes with Applications to Network Dynamic Pricing

连续时间跳跃马尔可夫决策过程的强化学习及其在网络动态定价中的应用

• Authors: Huiling Meng, Ningyuan Chen, Xuefeng Gao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.20680
• Pdf link: https://arxiv.org/pdf/2608.20680
• Abstract We study reinforcement learning (RL) in Continuous-Time Jump Markov Decision Processes (CTJMDPs) featuring general discrete state spaces (which need not possess a vector space structure) and continuous/discrete action spaces. The setup covers many well-known applications in operations such as multi-product dynamic pricing with capacitated resources (Gallego and van Ryzin 1997). To model the exploration-exploitation tradeoff, we formulate an entropy-regularized continuous-time control problem with stochastic policies. Recent continuous-time RL techniques such as $q$-learning for controlled diffusions in (Jia and Zhou 2023) focus on continuous state spaces $\mathbb{R}^d$ and rely heavily on semimartingale theory in $\mathbb{R}^d$ for their theoretical analysis. Consequently, their methods cannot be directly applied to CTJMDPs with general discrete state spaces, which may lack the algebraic addition and subtraction structures inherent to Euclidean spaces. To bridge this gap, we establish the theoretical foundations of $q$-learning for CTJMDPs and develop model-free $q$-learning algorithms. Compared to naïve time discretization and approximating CTJMDPs using discrete-time MDPs, our approach has several conceptual and empirical benefits. Numerical experiments in network dynamic pricing (Gallego and van Ryzin 1997) show that our proposed RL algorithm reliably learns near-optimal policies and consistently outperforms standard benchmark methods, demonstrating superior solution quality and effective scalability to large-scale network instances.
• 中文摘要 我们研究了连续时间跳跃马尔可夫决策过程（CTJMDPs）中的强化学习（RL），这些过程具有一般离散状态空间（不一定具有向量空间结构）和连续/离散作用空间。该构建涵盖了许多著名操作应用，如多产动定价与容量资源（Gallego 和 van Ryzin 1997）。为建模探索-利用权衡，我们构建了一个带有随机策略的熵正则化连续时间控制问题。近期连续时间强化学习技术，如（Jia 和 Zhou 2023）中的$q$学习，聚焦于连续状态空间 $\mathbb{R}^d$，并在理论分析中高度依赖 $\mathbb{R}^d$ 中的半马丁格尔理论。因此，他们的方法无法直接应用于具有一般离散状态空间的CTJMDP，这些空间可能缺乏欧几里得空间固有的代数加法和减法结构。为弥合这一差距，我们建立了CTJMDP$q$学习的理论基础，并开发了无模型$q学算法。与朴素时间离散化和使用离散时间MDP近似CTJMDP相比，我们的方法具有多项概念性和实证优势。网络动态定价中的数值实验（Gallego和van Ryzin 1997）表明，我们提出的强化学习算法能够可靠地学习近优策略，并持续优于标准基准测试方法，展现出对大规模网络实例更优的解决方案质量和有效的可扩展性。

CDRL: Certification-Driven Reinforcement Learning for Neutrino Flavor Model Discovery

CDRL：认证驱动的强化学习用于中微子味道模型发现

• Authors: Piyush Jha, Jake Rudolph, Victoria Knapp-Pérez, Max Fieg, Aishik Ghosh, Vijay Ganesh
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Logic in Computer Science (cs.LO); High Energy Physics - Phenomenology (hep-ph)
• Arxiv link: https://arxiv.org/abs/2608.20686
• Pdf link: https://arxiv.org/pdf/2608.20686
• Abstract Many scientific discovery problems require searching combinatorial hypothesis spaces under complex domain constraints. Reinforcement learning (RL) offers a promising approach, but existing methods rely on scalar rewards that provide limited information about why candidate solutions fail, leading agents to repeatedly explore invalid regions. We introduce Certification-Driven Reinforcement Learning (CDRL), a framework that leverages structured feedback from symbolic reasoning tools. When a candidate violates domain constraints, these tools produce certificates identifying the actions responsible for failure. CDRL converts these certificates into reusable constraints that eliminate classes of invalid solutions and guide exploration toward valid regions. We evaluate CDRL on neutrino flavor model discovery in theoretical particle physics, where the hypothesis space exceeds $10^{26}$ possible models, and compare it with the state-of-the-art RL approach previously used for this task. Across three theory spaces, CDRL achieves up to 1.95$\times$ higher valid model rates and up to 6.33$\times$ higher neutrino model rates while evaluating up to 4$\times$ fewer candidates. We further extract 40 interpretable rules from search trajectories using a post-hoc decision-tree framework and show that reusing them as soft constraints yields gains of up to 2$\times$ in valid model rates and 3$\times$ in neutrino model discovery across all three theory spaces. These results suggest that CDRL uncovers reusable structure in combinatorial search spaces and provides a general framework for scientific model discovery.
• 中文摘要 许多科学发现问题需要在复杂领域约束下搜索组合假设空间。强化学习（RL）提供了一种有前景的方法，但现有方法依赖标量奖励，提供有限的信息说明候选解失败的原因，导致智能体反复探索无效区域。我们引入了认证驱动强化学习（CDRL），这是一个利用符号推理工具结构化反馈的框架。当候选者违反领域约束时，这些工具生成证书，识别导致失败的行为。CDRL将这些证书转化为可复用的约束，消除无效解类并引导探索有效区域。我们评估CDRL在理论粒子物理中中微子味道模型发现中的假设空间超过$10^{26}$可能模型，并将其与此前用于该任务的最先进强化学习方法进行比较。在三个理论空间中，CDRL在评估候选对象数量减少多达4$\times的情况下，有效模型率高出1.95$\times$，中微子模型率高出6.33$\times$。我们进一步利用事后决策树框架从搜索轨迹中提取40条可解释规则，表明将其作为软约束重用后，有效模型率提升多达2$\times$，中微子模型发现提升3$\times$。这些结果表明CDRL揭示了组合搜索空间中的可重复使用结构，并为科学模型发现提供了通用框架。

Towards Faithful Simulation of Human Shopping Behavior

迈向对人类购物行为的忠实模拟

• Authors: Jiakai Tang, Yan Mi, Jing Yu, Yang Zhang, See-Kiong Ng, Qi Cao, Fei Sun, Xu Chen, Wen Chen, Jian Wu, Han Zhu, Bo Zheng
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2608.20707
• Pdf link: https://arxiv.org/pdf/2608.20707
• Abstract Simulating realistic user shopping behavior underpins offline evaluation and reinforcement learning in e-commerce scenarios. While recent LLM- and VLM-based simulators have made encouraging progress, reproducing a real browsing session remains difficult for two reasons. (i) Memory Challenge: a shopping session spans dozens of pages, yet existing agents either discard long-range observation histories, losing the evolving user state, or naively concatenate them, overwhelming the context window and even degrading simulation quality. (ii) Optimization Challenge: current user simulators are typically supervised to match each logged action via imitation or step-level rewards; the resulting sessions often display unrealistic patterns, such as over-exploration or excessive passivity, which per-step supervision can neither detect nor correct. To address the above challenges, we present RecVerse, a GUI-grounded simulation agent that perceives pages through screenshots and produces faithful multi-turn trajectories. For the memory challenge, RecVerse adopts a cognitive-inspired hierarchical memory: Working Memory for short-term focus, Episodic Memory for in-session traces, and Preference Memory for high-level intent, with memory updates treated as actions so that the agent adaptively learns when and what to memorize. For the optimization challenge, RecVerse is optimized with a trajectory-level RL objective that scores entire sessions, aligning both macro-level action-type distributions and micro-level shopping intent with real users. We further release USB (User Simulation Benchmark), an interactive e-commerce GUI trajectory dataset for multi-turn user simulation. Experiments show that RecVerse significantly outperforms existing baselines in both behavioral fidelity and intent consistency.
• 中文摘要 模拟真实的用户购物行为是电子商务场景中离线评估和强化学习的基础。尽管基于大型语言模型（LLM）和大型语言模型（VLM）的模拟器取得了令人鼓舞的进展，但重现真实浏览会话仍因两个原因而困难。（i）记忆挑战：购物会话跨越数十页，现有代理要么丢弃长期观察历史，丢失不断演变的用户状态，要么天真地将它们串接起来，导致上下文窗口过大，甚至降低仿真质量。（ii）优化挑战：当前的用户模拟器通常通过模拟或步骤级奖励来匹配每个记录的动作;结果的会话常常表现出不切实际的模式，如过度探索或过度被动，而每步监督既无法检测也无法纠正这些。为应对上述挑战，我们介绍了RecVerse，一个基于图形界面的模拟代理，通过截图感知页面并生成忠实的多回合轨迹。在记忆挑战中，RecVerse采用了认知启发的层级记忆：短期专注用工作记忆，会话中追踪用情景记忆，高级意图用偏好记忆，记忆更新被视为动作，使智能体自适应地学习何时何物记忆。在优化挑战中，RecVerse采用轨迹级强化学习目标，对整个会话进行评分，使宏观层面的动作类型分布和微观层面的购物意图与真实用户保持一致。我们还发布了USB（用户模拟基准），这是一个用于多回合用户模拟的交互式电商图形用户轨迹数据集。实验显示，RecVerse在行为忠实度和意图一致性方面显著优于现有基线。

CAS: Conformalized Agentic Search via Adaptive Retrieval and Policy Weighting

CAS：通过自适应检索和策略加权实现的规范化代理搜索

• Authors: Zixi Zhu, Jiayuan Su, Jian Zhang, Yu Lin, Hongwei Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.20771
• Pdf link: https://arxiv.org/pdf/2608.20771
• Abstract Search Agents face a severe reliability crisis during reinforcement learning (RL) fine-tuning. Heuristic Top-K retrieval often causes critical evidence loss or noise inclusion, while over-confidence induced by progressive RL leads to hallucinated answers and redundant searches. To build highly reliable agents, we introduce Conformal Prediction (CP) and propose Conformalized Agentic Search (CAS). This framework establishes reliability guarantees on both the retrieval and training sides: on the retrieval side, an Adaptive Prediction Set (APS), a specific CP realization, translates statistical coverage into dynamic document truncation to construct prediction sets that are adaptive in size; on the training side, Adaptive Conformal Inference (ACI), a dynamic CP algorithm, dynamically constructs prediction sets with controllable coverage to quantify answer confidence, which is then used to penalize low-confidence trajectories within the Group Relative Policy Optimization (GRPO) objective, ensuring the model learns only from reliable ones. Experiments across single-hop and multi-hop QA datasets demonstrate that our framework significantly improves reasoning accuracy while drastically reducing redundant tool invocations, establishing a highly reliable and efficient agent paradigm. Our code is available at this https URL.
• 中文摘要 搜索代理在强化学习（RL）微调过程中面临严重的可靠性危机。启发式Top-K检索常导致关键证据丢失或噪声包含，而渐进式增强学习引发的过度自信则导致错觉的答案和冗余搜索。为了构建高度可靠的代理，我们引入了共形预测（CP）并提出了共形化代理搜索（CAS）。该框架在检索和训练双方都建立了可靠性保证：在检索端，自适应预测集（APS）作为特定的CP实现，将统计覆盖转化为动态文档截断，构建具有适应性的预测集;在训练端，自适应共形推断（ACI）是一种动态CP算法，动态构建可控覆盖的预测集以量化答案置信度，随后用来惩罚组相对策略优化（GRPO）目标内低置信度轨迹，确保模型仅从可靠路径中学习。跨单跳和多跳质量保证数据集的实验表明，我们的框架显著提升了推理准确性，同时大幅减少了冗余工具调用，建立了高度可靠高效的代理范式。我们的代码可在该网址访问。

Enhancing Localized Reasoning for Long Video Understanding via Efficient Segment-to-Video Supervision

通过高效的分段到视频监督，增强长视频理解的本地化推理

• Authors: Beibei Zhang, Chao Xu, Jun Lan, Zongyi Li, Lai Wei, Huijia Zhu, Tongwei Ren
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.20814
• Pdf link: https://arxiv.org/pdf/2608.20814
• Abstract Though Multimodal Large Language Models (MLLMs) have shown impressive potential in video understanding, long video understanding (LVU) remains challenging since distracting noise in complex and lengthy contexts can obscure localized details, misleading MLLMs to produce incorrect answers. Recent works mitigate these issues by incentivizing deep reasoning to include relevant evidence. However, these methods have two main problems: First, the reinforcement fine-tuning framework (RFT) they leveraged incurs substantial training overheads, including high annotation costs and complicated reward designs. Second, the self-reflective and iterative-perception mechanism in some methods causes lengthy outputs and high inference latency. To alleviate these problems, we propose a novel Segment-to-Video Supervision} method (S2V) to efficiently enhance fine-grained reasoning in LVU. Specifically, we generate question answer pairs (VQA) based on localized segments, and then transfer these segment-based VQA back to the whole video for training. Due to focusing on short segments, segment-based VQA can naturally notice details which tend to be overlooked from a whole-video perspective. Training on such data can enforce MLLMs to correctly associate fine-grained details with QA while avoiding distracting noise in the whole video. The S2V training involves just reinforcement learning (RL) with a simple accuracy reward based on only 10K VQA samples and the resulting S2V model predicts answer using a single forward pass with limited output tokens. Experimental results demonstrate that S2V can consistently improve LVU performance across multiple LVU benchmarks, outperforming both general MLLMs and reasoning-based methods not only in LVU accuracy but also in training and inference efficiency.
• 中文摘要 尽管多模态大型语言模型（MLLMs）在视频理解方面展现出了显著潜力，但长视频理解（LVU）依然具有挑战性，因为在复杂且冗长的环境中，干扰噪声会遮蔽局部细节，误导MLLM产生错误答案。近期研究通过激励深度推理纳入相关证据来缓解这些问题。然而，这些方法存在两个主要问题：首先，它们所利用的强化微调框架（RFT）带来了大量训练开销，包括高注释成本和复杂的奖励设计。其次，某些方法中的自我反思和迭代感知机制会导致输出时间过长且推理延迟较高。为缓解这些问题，我们提出了一种新型的片段到视频监督（S2V）方法，以高效提升LVU中的细粒度推理能力。具体来说，我们基于局部片段生成问题答案对（VQA），然后将这些基于片段的VQA反馈到整个视频进行训练。由于聚焦短片段，基于片段的VQA可以自然地发现从全视频视角容易被忽视的细节。基于此类数据的训练可以强制MLLM正确将细粒度细节与QA关联，同时避免整个视频中的干扰噪声。S2V训练仅包含强化学习（RL），基于仅1万个VQA样本，提供简单的准确率奖励，最终S2V模型通过有限输出代币的单次前向传递预测答案。实验结果表明，S2V能够在多个LVU基准测试中持续提升LVU性能，不仅在LVU准确性上优于一般MLLM和基于推理的方法，在训练和推断效率方面也更优。

Natural Sit-to-Stand Motion Synthesis For Humanoids via Guided Assistance Curricula and Staged Rewards

通过引导辅助课程和分阶段奖励，自然坐立动作合成

• Authors: Meet Pal Singh, Vyankatesh Ashtekar, Ashish Dutta
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.20823
• Pdf link: https://arxiv.org/pdf/2608.20823
• Abstract A humanoid has infinitely many ways to stand up from sitting while maintaining balance, making sit-to-stand (STS) a challenging control problem. We synthesise natural humanoid STS motion from scratch using reinforcement learning, without demonstrations or reference trajectories. A single Proximal Policy Optimisation policy learns smooth, human-like rising driven by three complementary components. (i) A coupled force/chair-height curriculum is used. A vertical pelvis-assist force aids early trajectory exploration and decays over training. Taller chairs are unlocked with decaying assisting force. This ensures that the policy masters a viable STS trajectory at each chair height before being exposed to harder ones, avoiding the premature distribution shift that otherwise collapses generalisation. (ii) Motion robustness is achieved by randomly sampling from a large number of inverse kinematics-generated initial and target poses spanning over eight chair heights. (iii) A set of rewards is defined inspired from biomechanics and optimal control studies. They shape the robot's angular momentum for seat-off, and enable support-region transition via centre of pressure attraction function to ensure smooth low-effort actuation. On a deterministic force-free evaluator, the policy attains more than 97% balanced-standing success across eight chair heights. The policy generalises smooth motion across chair heights and enables the robot to rise from substantially deep-seated postures as compared to the state of the art.
• 中文摘要 类人生物有无限种从坐姿起立并保持平衡的方式，使得坐站（STS）成为一个具有挑战性的控制难题。我们通过强化学习从零合成自然的人形STS动作，无需演示或参考轨迹。单一的近端政策优化策略学习由三个互补组成部分驱动的平滑、类人化上升。（i）采用耦合的力/椅子高度课程。垂直骨盆辅助力帮助早期探索轨迹，并在训练过程中衰减。较高的椅子则通过衰减辅助力解锁。这确保政策在面对较难的椅子前，在每个椅子高度掌握可行的STS轨迹，避免过早的分布转移，避免泛化崩溃。（ii） 运动鲁棒性通过从大量逆运动学生成的初始和目标姿态中随机抽样，涵盖八个椅子高度实现。（iii） 一组奖励基于生物力学和最优对照研究的启发。它们塑造机器人离座时的角动量，并通过压力中心吸引函数实现支撑区转换，确保平稳且低力的驱动。在确定性无力评估器上，该政策在八个椅子高度上实现了超过97%的平衡站立成功率。该策略推广了跨椅子高度的平滑运动，使机器人能够从较深坐姿态中起身，相较于最先进的技术水平。

Demonstration-Guided Humanoid Stand-Up on an Emulated Deformable Surface

演示引导的人形单口喜剧，在模拟可变形表面上

• Authors: Aniruddh Kushwah, Vyankatesh Ashtekar, Ashish Dutta
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.20852
• Pdf link: https://arxiv.org/pdf/2608.20852
• Abstract This paper presents a reference-guided reinforcement learning framework to generate stand-up motion for a 29-DOF Unitree G1 humanoid on deformable soft ground, using a human demonstration recorded on hard ground. The terrain compliance is modelled using solref and solimp parameters from MuJoCo's rigid body soft-contact model. The rewards consists of (i) reference motion tracking through residual joint-position control and (ii) explicit recovery objectives such as pelvis height, torso uprightness, and the final posture. First, the policy is trained with the specified rewards considering hard ground. Next, the terrain stiffness is lowered by updating solref and the nominal surface penetration zone is expanded using solimp. Subsequent training enables the policy to adapt to the delayed support force generation due to significant surface penetration during contact-intensive phases while preserving the original demonstration pattern. The learned policy successfully completes the fallen-to-standing task in simulation, reaching the targeted pelvis height and uprightness, with a maximum contact penetration of approximately 40 mm during the process. The proposed method is demonstrated on two stand-up sequences and successfully achieves the final recovery objective on both hard and soft ground. Ablation studies show that reference tracking alone is insufficient for successful stand-up, and that explicit recovery rewards are essential.
• 中文摘要 本文提出了一个参考引导强化学习框架，用于在可变形软地面上为29自由度的Unitree G1人形生物生成站立动作，并利用在硬地面上录制的人工演示。地形适应性通过MuJoCo刚体软接触模型中的solref和solimp参数进行建模。奖励包括（i）通过残余关节位置控制进行参考运动跟踪，以及（ii）明确的恢复目标，如骨盆高度、躯干直立和最终姿势。首先，在考虑硬地面的情况下训练策略，满足指定的奖励。接着，通过更新solref降低地形刚度，并利用solimp扩展名义表面穿透区。后续训练使策略能够适应由于接触密集阶段显著穿透导致的支撑力生成延迟，同时保持原始示范模式。所学策略成功完成了模拟中的倒地至站立任务，达到目标骨盆高度和直立度，过程中最大接触穿透力约为40毫米。该方法在两个站立序列中演示，并成功实现了硬地和软地面的最终恢复目标。消融研究表明，仅靠参考跟踪不足以实现成功的站立，明确的恢复奖励至关重要。

Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks

深度强化学习与模型预测控制之间的控制权共享：多类别运输网络的应用

• Authors: Giray Onur, Azita Dabiri, Bart De Schutter
• Subjects: Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.20858
• Pdf link: https://arxiv.org/pdf/2608.20858
• Abstract Transportation networks, in particular multi-class transportation networks (i.e., networks with mixed vehicle types), are complex systems that are challenging to control. Recently, Deep Reinforcement Learning (DRL), which learns control policies from interactions with the environment, and Model Predictive Control (MPC), which uses a system model to optimize control inputs, have been increasingly utilized for transportation network control. However, nonlinear system dynamics and high-dimensional state spaces in large-scale networks limit DRL's learning capacity under time-constrained training and increase MPC's computation time, hindering real-time implementation with limited computational resources. Moreover, MPC depends on an accurate network model, which is often unavailable for complex systems such as multi-class transportation networks. This paper proposes a novel DRL-MPC framework for multi-class transportation networks that divides control authority between DRL and MPC, combining DRL's fast online computation and model independence with MPC's built-in optimization and constraint-handling capabilities. In the hierarchical framework, MPC operates at the higher level and determines low-frequency control inputs whose slower update rate accommodates its high computation time, while DRL operates at the lower level and determines high-frequency control inputs using its fast online deployment. The framework is evaluated on a multi-class freeway network against a hierarchical MPC controller and a hybrid state-feedback-MPC controller, including scenarios with model mismatch and noisy traffic demands. Results show that the proposed framework outperforms the hybrid state-feedback-MPC controller, substantially reduces online computation time compared with the hierarchical MPC controller, and provides more effective constraint enforcement under model mismatch.
• 中文摘要 交通网络，尤其是多类别交通网络（即混合车辆类型的网络），是复杂的系统，难以控制。近年来，深度强化学习（DRL）通过与环境的交互学习控制策略，以及利用系统模型优化控制输入的模型预测控制（MPC）越来越多地被用于交通网络控制。然而，大规模网络中的非线性系统动力学和高维状态空间限制了DRL在时间限制训练下的学习能力，增加了MPC的计算时间，阻碍了有限计算资源下的实时实现。此外，MPC依赖于精确的网络模型，而这通常无法用于多类交通网络等复杂系统。本文提出了一种新颖的DRL-MPC多类交通网络框架，将控制权分配给DRL和MPC，结合DRL的快速在线计算和模型独立性，以及MPC内置的优化和约束处理能力。在分层框架中，MPC在高层运行，确定低频控制输入，其较慢的更新率以适应高计算时间;而DRL则在低层运行，利用其快速在线部署确定高频控制输入。该框架在多类高速公路网络上对分层MPC控制器和混合状态反馈MPC控制器进行评估，包括模型不匹配和噪声交通需求的场景。结果显示，所提框架优于混合状态反馈MPC控制器，显著缩短在线计算时间，并在模型不匹配下提供更有效的约束执行。

Multi-Objective Deep Reinforcement Learning for Secure and Stable Power System Operation

多目标深度强化学习，实现电力系统安全稳定运行

• Authors: Ioannis Papadopoulos, Georgios Tsaousoglou, Johanna Vorwerk
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.20914
• Pdf link: https://arxiv.org/pdf/2608.20914
• Abstract The ongoing energy transition challenges the stable operation of power systems and increases the need for rapid decision-making under uncertainty. While reinforcement learning has emerged as a promising framework for power system control and operation, existing applications typically focus on a single operational criterion, such as thermal security or small-signal stability. However, power system operation is inherently multi-objective and may involve trade-offs between objectives. This paper develops a unified-control deep reinforcement learning agent that maintains thermal security under stochastic load variations while steering the system toward operating points with improved damping of the most critical mode. Compared to a thermal-security-only agent and a business-as-usual policy, the proposed agent achieves a better balance among the operational objectives considered, with notably improved damping and negligible thermal-security violations. Finally, the operational value of increased critical damping is demonstrated under small- and large-signal disturbances, where operating points with higher damping lead to faster oscillation decay and improved critical clearing times.
• 中文摘要 持续的能源转型挑战了电力系统的稳定运行，并增加了在不确定性下快速决策的需求。尽管强化学习已成为电力系统控制和运行的有前景框架，但现有应用通常专注于单一操作标准，如热安全或小信号稳定性。然而，电力系统运行本质上是多目标的，可能涉及目标间的权衡。本文开发了一种统一控制深度强化学习代理，在随机负载变化下保持热安全，同时引导系统朝向具有最佳阻尼性最关键模式的操作点。与仅热安全代理和照常运营策略相比，该代理在考虑的操作目标之间实现了更好的平衡，显著提升了阻尼效果，且热安全违规可忽略不计。最后，在小信号和大信号扰动下，提升临界阻尼的操作价值得到了验证，其中阻尼较高的工作点能加快振荡衰减并提高临界清除时间。

Teaching is a Process: The TOSS Framework for Modeling Human Teaching Decisions in Human-Interactive Robot Learning

教学是一个过程：TOSS框架用于模拟人类交互机器人学习中的人类教学决策

• Authors: Bernhard Hilpert, Kim Baraka, Joost Broekens
• Subjects: Subjects: Robotics (cs.RO); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2608.21083
• Pdf link: https://arxiv.org/pdf/2608.21083
• Abstract Successful Human-Robot Teaching assumes alignment between robot processing needs and human teaching intent. To better understand this alignment, this work seeks to uncover the underlying logic that humans intuitively apply when teaching. Through an exploratory, bottom-up study with N=34, participants observing two distinct robot Reinforcement Learning (RL) scenarios, we analyze 204 intuitive teaching responses across early, middle, and late learning phases. Results reveal that teaching decisions consist of a nuanced, interconnected network of Triggers (situational catalysts), Objectives (subjective teaching targets), Signals (communicative acts), and Strategies (high-level governance) in which teachers spontaneously adopt diverse roles, acting as coaches, engineers, or designers and prioritize different objectives. Based on these results, we introduce the TOSS Framework, which conceptualizes Human-Robot teaching as a procedural loop between robot behavior and human teaching actions, in which human teaching decisions are modeled as Trigger-Signal responses modulated by teaching Objectives and Strategies. It provides future research with an openly accessible dataset and a theoretical foundation for a) understanding teaching decisions and b) simulating realistic oracles as well as c) designing human-centered teaching settings and novel robot learning algorithms that go beyond the constraints of current robot learning settings.
• 中文摘要 成功的人机教学假设机器人处理需求与人类教学意图相符。为更好地理解这种对齐，本研究旨在揭示人类在教学时直觉应用的潜在逻辑。通过一项N=34的探索性自下而上研究，参与者观察两种不同的机器人强化学习（RL）场景，分析了204个直觉教学反应，涵盖早期、中期和晚期学习阶段。结果显示，教学决策由一个细致且相互关联的触发器（情境催化剂）、目标（主观教学目标）、信号（沟通行为）和策略（高层治理）组成，教师自发扮演多样角色，扮演教练、工程师或设计师，并优先考虑不同目标。基于这些结果，我们提出了TOSS框架，该框架将人机教学概念化为机器人行为与人类教学行为之间的程序循环，其中人类教学决策被建模为触发信号响应，并由教学目标和策略调制。该框架为未来研究提供了开放可访问的数据集和理论基础，支持a）理解教学决策，b）模拟现实预言机，c）设计以人为本的教学环境和突破现有机器人学习限制的新型机器人学习算法。

SRL-MPC: Shape-Aware Reinforcement Learned Model Predictive Control

SRL-MPC：形状感知强化学习模型预测控制

• Authors: Ruihua Han, Rui Gao, Zhe Liu, Xinyi Wang, Chang Chen, Shuai Wang, Qi Hao, Jia Pan, Hengshuang Zhao
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21175
• Pdf link: https://arxiv.org/pdf/2608.21175
• Abstract Safe and efficient shape-aware navigation in heterogeneous crowds and robot fleets remains challenging. Traditional approaches often assume homogeneous robots, sparse workspaces, simplified geometry, offline computation, or handcrafted parameters to make the problem tractable, which limits their deployment in dense crowd scenarios. Toward this end, we propose Shape-Aware Reinforcement Learned Model Predictive Control (SRL-MPC), a method for safe, efficient, and adaptive navigation in crowds with heterogeneous shapes without geometry simplification. To encode shape-aware safety, we formulate high-order control barrier function (HOCBF) constraints from geometric separation features (GSFs) based on support function transformation. A reinforcement learning (RL) framework then learns a neural policy that reads GSFs and outputs real-time MPC parameter updates, enabling the MPC solver to adapt to neighboring crowd geometries. The key advantage of SRL-MPC is that it preserves the safety structure and generalizability of MPC while integrating the adaptability and intelligence of RL. Experiments in randomized crowd scenarios with arbitrary shaped robot fleets demonstrate the effectiveness, scalability, and robustness of SRL-MPC. The results show that SRL-MPC substantially outperforms representative baselines in safety and adaptability. Project website: this https URL
• 中文摘要 在异构人群和机器人车队中实现安全高效的形状感知导航仍然充满挑战。传统方法通常假设机器人同质化、工作空间稀疏、几何结构简化、离线计算或手工参数，以使问题易于解决，这限制了它们在密集人群场景中的应用。为此，我们提出了形状感知强化学习型预测控制（SRL-MPC），这是一种在异构形状人群中实现安全、高效且自适应导航且无需几何简化的方法。为编码形状感知安全，我们基于支撑函数变换，从几何隔离特征（GSF）中构建高阶控制屏障函数（HOCBF）约束。强化学习（RL）框架随后学习神经策略，读取GSF并实时输出MPC参数更新，使MPC求解器能够适应邻近的人群几何结构。SRL-MPC的主要优势在于它保留了MPC的安全结构和泛化性，同时整合了RL的适应性和智能性。在随机人群场景中，任意形状的机器人车队实验展示了SRL-MPC的有效性、可扩展性和鲁棒性。结果显示，SRL-MPC在安全性和适应性方面远超代表性基线。项目网站：此 https URL

Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

超越模仿：通过非策略Q规划实现自我改进的机器人政策

• Authors: Varun Giridhar, Anant Khandelwal, Jeremy A. Collins, Ignat Georgiev, Animesh Garg
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.21204
• Pdf link: https://arxiv.org/pdf/2608.21204
• Abstract Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step Q-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.
• 中文摘要 行为克隆（BC）推动了机器人操作的显著进展，但其根本受限于自身改进能力：失败的策略无法从失败中学习，除非有额外的人工演示。强化学习微调提供了自我改进的路径，但难以扩展到支撑现代机器人政策的数十亿参数模型。我们提出了Q-规划，它为大型视觉运动BC策略配备了一个小型非策略Q函数。由于Q-函数是估计价值而非模仿行为，它可以基于与BC策略相同的成功演示进行训练，随后吸收成功和失败的部署推广，而BC不具备这种不对称性。我们利用这种不对称性实现了推理时的价值引导动作选择（BC抽取中的单步Q加权平均），并实现在线自我改进，仅微调Q函数，BC权重不变。在LIBERO和双手动RoboTwin中，十次自我提升提升了我们测试的所有基准分数（LIBERO-10 93%至99%，RoboTwin 83.8%至91.4%），并在接近天花板的套件（LIBERO-Object、LIBERO-Goal）上缩短了成功发作。在两个双手真机器人任务中，同一循环（BC冻结，无人工干预）仅通过自身部署部署提升：五次迭代中堆叠杯40%至90%，插入式口袋提升25%至80%，而仅成功部署时SFT停滞在55%和30%。在相同的在线预算下，Q-Planning是Best-of-N、筛选SFT、IBRL、DSRL和DAWR中唯一能稳定从失败中改善且无需训练辅助参与者的方法。

AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization

AUSO：从内化到利用的行动级统一技能优化

• Authors: Huizu Lin, Chengkai Huang, Tianqi Gao, Tao Huang, Daijiao Liu, Tongxin Li, Xiaoyan Sun, Lina Yao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21292
• Pdf link: https://arxiv.org/pdf/2608.21292
• Abstract Skills play different roles as an agent's policy evolves: they should first provide learnable knowledge, then support capability formation, and finally be invoked only when they improve individual decisions. Existing methods rarely model this lifecycle. They either keep skills outside the model, fully internalize them, or select among internalization and utilization objectives through noisy task-level success rates. Such designs fragment training and assign uniform importance to actions within the same trajectory, even though skill guidance may help some decisions while distracting others. To solve these problems, we introduce AUSO (Action-level Unified Skill Optimization), which unifies skill learning and skill use through a progressive, action-aware optimization process. At the beginning of training, AUSO jointly learns from teacher guidance and environmental outcomes, enabling the policy to acquire foundational skills without losing task-oriented feedback. It subsequently emphasizes outcome-based policy optimization to consolidate autonomous problem-solving ability. As the policy matures, AUSO evaluates each sampled action under both skill-conditioned and skill-free contexts. The resulting action-level information signal is coupled with the trajectory outcome advantage, allowing beneficial skill-sensitive actions to receive stronger updates and harmful ones to be suppressed. Therefore, skills gradually transition from an external source of supervision into decision knowledge whose utilization is adapted to its action-level benefit, while reinforcement learning remains the shared backbone across all stages. Experiments on ALFWorld, WebShop, and SearchQA show that AUSO consistently improves agent performance and out-of-distribution generalization over competitive baselines.
• 中文摘要 随着智能体政策的发展，技能扮演不同角色：它们应先提供可学习知识，然后支持能力形成，最终仅在个人决策改善时被调用。现有方法很少模拟这一生命周期。它们要么将技能置于模型之外，要么完全内化，要么通过任务层级的噪声成功率在内化和利用目标之间选择。此类设计将培训分割，并统一赋予同一轨迹内的行动重要性，尽管技能指导可能帮助某些决策，同时分散其他决策注意力。为解决这些问题，我们引入了AUSO（行动级统一技能优化），通过渐进的、行动意识的优化流程统一技能学习与技能使用。培训初期，AUSO共同从教师指导和环境成果中学习，使政策能够获得基础技能而不丢失任务导向反馈。随后强调基于结果的策略优化，以巩固自主问题解决能力。随着策略成熟，AUSO在技能条件和无技能上下文下评估每个抽样动作。所得动作级信息信号与轨迹结果优势相结合，使有益的技能敏感动作获得更强的更新，抑制有害行为。因此，技能逐步从外部监督转向决策知识，其利用方式适应其动作层级效益，而强化学习仍是各阶段的共享骨干。ALFWorld、WebShop和SearchQA上的实验显示，AUSO在竞争基线上持续提升代理表现和分布外泛化。

Re$^3$Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learning

Re$^3$Cap：通过强化学习实现图像字幕增强的检索引导精炼

• Authors: Haonan Jia, Shichao Dong, Zenghui Sun, Jiawen Zheng, Ziqi Miao, Gege Shi, Qiuyu Zhao, Jinsong Lan, Xiaoyong Zhu, Bo Zheng
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21305
• Pdf link: https://arxiv.org/pdf/2608.21305
• Abstract Reinforcement Learning (RL) has demonstrated significant gains in image captioning, yet it is still limited in encouraging Large Vision-Language Models (LVLMs) to explore novel reasoning strategies. This limitation leads to a performance gap between RL and Supervised Fine-Tuning (SFT). In this paper, we argue that multi-modal retrieval can serve as an effective reasoning signal for caption refinement. Based on this insight, we present the Retrieval-Guided Refinement for Image Captioning (Re$^3$Cap), a retrieval-guided reasoning strategy that enhances image captioning without requiring additional annotations. Instantiated by Caption Refinement Suggester (CRS) and Caption Quality Assessor (CQA), this strategy identifies hallucinations and omissions in image captions, leading to more accurate and detailed descriptions. Extensive experiments demonstrate the superiority of our method in image captioning, even compared with Supervised Fine-Tuning. Especially, Re$^3$Cap outperforms GRPO with an average improvement of 8.64% in relation reasoning on the COCO-LN500 benchmark.
• 中文摘要 强化学习（RL）在图像字幕方面取得了显著进步，但在鼓励大型视觉语言模型（LVLMs）探索新颖推理策略方面仍有限。这一限制导致强化学习与监督微调（SFT）之间存在性能差距。本文论证多模态检索可以作为字幕优化的有效推理信号。基于这一见解，我们提出了图片字幕检索引导细化（Rebar^3$Cap），这是一种检索引导推理策略，可在无需额外注释的情况下增强图像字幕。该策略由字幕细化建议器（CRS）和字幕质量评估器（CQA）实现，能够识别图像说明中的幻觉和遗漏，从而实现更准确和详细的描述。大量实验证明，我们方法在图像字幕方面优于监督式微调。特别是，Re$^3$Cap在COCO-LN500基准测试中关系推理平均提升8.64%，优于GRPO。

Keyword: diffusion policy

There is no result