生成时间: 2026-08-11 17:00:26 (UTC+8); Arxiv 发布时间: 2026-08-11 20:00 EDT (2026-08-12 08:00 UTC+8)

今天共有 58 篇相关文章

Keyword: reinforcement learning

Unified Hallucination Fuzzing for Multimodal Large Language Models

多模态大型语言模型的统一幻觉模糊

• Authors: Pengfei Zhou, Jiajun Song, Zhiwei Tang, Yixing Ma, Xiaopeng Peng, Donghui Si, Yuhang Xu, Huiqi Song, Yiyuan Miao, Yichen Qian, Weihua Chen, Wangbo Zhao, Bohan Zhuang, Jiasheng Tang, Yang You
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07525
• Pdf link: https://arxiv.org/pdf/2608.07525
• Abstract Hallucination remains a persistent challenge for Multimodal Large Language Models (MLLMs), severely limiting their reliability in high-stakes applications. Existing evaluations, predominantly based on static benchmarks, suffer from narrow taxonomical coverage and rapid performance saturation, failing to reflect model robustness in evolving real-world scenarios. To bridge this gap, we present a systematic evaluation framework integrating a comprehensive benchmark with self-evolving stress testing. First, we introduce UniHall, a fine-grained dataset grounded in a unified taxonomy spanning Object, Instruction, and Knowledge dimensions. Second, to address benchmark saturation, we propose Self-Adaptive Multimodal Fuzzing (SAMF), a self-adaptive framework that employs evolutionary mutation strategies to explore the boundaries of model hallucinations. Crucially, to ensure reliable assessment of dynamic inputs, SAMF incorporates a structured metric suite driven by an ensemble of multi-modal oracles. Our extensive experiments reveal that state-of-the-art MLLMs exhibit significant performance degradation under fuzzing compared to conventional settings, exposing a dissociation between reasoning capabilities and factual grounding. Furthermore, we identify a helpfulness-hallucination trade-off, where reinforcement learning alignment inadvertently exacerbates sycophancy in instruction-following tasks. The framework, code and benchmark are available at this https URL.
• 中文摘要 幻觉依然是多模态大型语言模型（MLLMs）面临的持续挑战，严重限制了其在高风险应用中的可靠性。现有评估主要基于静态基准，但由于分类覆盖范围狭窄且性能迅速饱和，未能反映模型在不断演变的现实场景中的稳健性。为弥合这一差距，我们提出了一个系统化的评估框架，将综合基准与自我演化的压力测试相结合。首先，我们介绍了UniHall，这是一个基于统一分类法的细粒度数据集，涵盖对象、指令和知识维度。其次，为应对基准测试饱和问题，我们提出了自适应多模模糊（SAMF）框架，采用进化突变策略探索模型幻觉的边界。关键是，为了确保动态输入的可靠评估，SAMF采用了由多模态预言机集合驱动的结构化度量套件。我们广泛的实验显示，最先进的MLLM在模糊测试下表现显著下降，相较于传统设置，暴露了推理能力与事实基础之间的脱离。此外，我们识别出一种助人性与幻觉的权衡，即强化学习对齐无意中加剧了指令跟随任务中的谄媚行为。该框架、代码和基准可在此 https 网址获取。

DocAtlas: Long-Document Understanding as Mutable-State Interaction

DocAtlas：作为可变状态交互的长文档理解

• Authors: Hongchen Wei, Yuanzhe Wang, Bei Liu, Yifan Yang, Qi Dai, Kai Qiu, Yunsheng Li, Dongdong Chen, Chong Luo, Zhenzhong Chen, Baining Guo
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07527
• Pdf link: https://arxiv.org/pdf/2608.07527
• Abstract Long-document understanding requires models to find and combine evidence across many pages, layouts, tables, figures, and charts. Existing retrieval-augmented systems usually select evidence from a static index before generation, while recent agentic systems add multi-turn tool use but often rely on frozen proprietary backbones whose behavior is set by prompts. We present DocAtlas, a system that treats long-document understanding as a mutable-state information-seeking process. We instantiate DocAtlas as a mutable document harness: an external environment that determines what document information is searched, read, stored, reviewed, and shown to the model at each step. Given a document and question, the harness exposes search, reading, note-taking, and review tools, maintains a hierarchical tree and note store, and updates both as the agent records evidence. DocAtlas combines self-improving retrieval, selective evidence access, and active working memory under a fixed context budget. The same harness supports inference-time use with large VLMs and end-to-end reinforcement learning for compact VLM agents. With GPT-5.4, DocAtlas reaches 71.4\% on MMLongBench-Doc, exceeding the human-expert reference of 65.8\%. A Qwen3.5-4B VLM trained with end-to-end RL in the DocAtlas environment reaches 63.7\%, compared with a 54.4\% direct-input baseline, showing that mutable document-harness design can improve compact document agents by a large margin.
• 中文摘要 长文档理解需要模型在多页、布局、表格、图表和图表中寻找并组合证据。现有的检索增强系统通常在生成前从静态索引中选择证据，而近期的代理系统增加了多回合工具的使用，但通常依赖由提示决定行为的冻结专有骨干。我们介绍了DocAtlas，一种将长文档理解视为可变状态信息寻求过程的系统。我们将DocAtlas实例化为可变文档束缚：一个外部环境，决定每步搜索、阅读、存储、审查并向模型展示哪些文档信息。给定文档和问题时，束带会暴露搜索、阅读、笔记和复习工具，维护层级树和笔记存储，并在代理记录证据时同步更新。DocAtlas结合了自我改进的检索、选择性证据访问和在固定上下文预算下的主动工作记忆。同一工具带支持大型VLM的推理时间使用，并支持紧凑型VLM代理的端到端强化学习。使用GPT-5.4时，DocAtlas在MMLongBench-Doc上达到71.4%，超过了人类专家参考的65.8%。在DocAtlas环境中，采用端到端强化学习训练的Qwen3.5-4B VLM可达63.7%，而直接输入基线为54.4%，表明可变文档束带设计能大幅提升紧凑文档代理。

Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards

Search-G1：通过基于表示的内在奖励实现基于基础的搜索代理

• Authors: Cheng Ruoxi, Ma Haoxuan, Zhang Hongyi, Zhang Junming, Duan Ranjie, Xia Qiaolin, Wang Hao, Lu Yu, Shi Haibo, Ma Xingjun
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07531
• Pdf link: https://arxiv.org/pdf/2608.07531
• Abstract Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Existing external rewards provide either sparse outcome supervision or richer feedback from process annotations and LLM judges. Outcome rewards scale readily but cannot distinguish grounded retrieval from redundant search, whereas richer signals require costly annotation or inference during training. Internal rewards based on policy-side signals such as entropy, likelihood, or information gain are graded and inexpensive to evaluate, yet mainly reflect model confidence rather than evidence grounding. We propose Search-G1, a representation-based intrinsic reward framework that measures the operational grounding of an agent's answers through two intervention-calibrated readouts. A prompt-state readout predicts closed-book sufficiency, whose complement defines policy-relative retrieval necessity; an answer-commit readout estimates evidence reliance from answer-stage sensitivity to evidence deletion. Together, they provide additional credit to correct searched trajectories when retrieval is estimated necessary and the answer is evidence-sensitive, favor correct direct answers when closed-book knowledge suffices, and penalize repeated search. After calibration, reward scoring requires neither process annotations nor LLM-as-judge inference during policy optimization. Because reinforcement learning changes policy representations, Search-G1 periodically refits both readouts on trajectories from the latest checkpoint, allowing the reward to co-evolve with the policy. Experiments across multiple search-based question-answering benchmarks and two model scales show that Search-G1 improves the grounding--search-cost trade-off, producing shorter response-side trajectories at competitive task accuracy. Code is available at this https URL.
• 中文摘要 搜索增强语言代理应仅在必要时检索外部信息，并将答案建立在检索到的证据基础上。现有的外部奖励要么提供稀疏的结果监督，要么提供来自过程注释和大型语言模型评判的更丰富的反馈。结果奖励易于扩展，但无法区分有基础的检索与冗余搜索，而更丰富的信号则需要在训练过程中进行昂贵的注释或推断。基于策略侧信号如熵、似然或信息获得的内部奖励是分级且评估成本低廉的，但主要反映的是模型置信度，而非证据基础。我们提出了Search-G1，一种基于表征的内在奖励框架，通过两个干预校准的读出来衡量代理回答的操作基础性。即时状态读出预测闭卷本充分性，其补集定义了策略相对检索的必要性;答案提交读出估计从答案阶段敏感度到证据删除的依赖度。它们共同为必要检索且答案具证据敏感性时给予正确搜索轨迹额外认可，当封闭知识足够时优先正确直接答案，并惩罚重复搜索。校准后，奖励评分无需过程注释，也无需作为法官的LLM推断。由于强化学习改变策略表示，Search-G1会定期根据最新检查点的轨迹重新调整两个读数，使奖励与策略协同演化。多个基于搜索的问答基准和两种模型尺度的实验表明，Search-G1改善了基础-搜索-成本权衡，在竞争任务准确性下产生更短的响应侧轨迹。代码可在此 https URL 获取。

Generalizing deep reinforcement learning across cable-driven parallel robot configurations with actuator-level policies

将深度强化学习推广到具有执行器级策略的电缆驱动并行机器人配置

• Authors: Abir Bouaouda, Mohamed Boutayeb (CRAN, UIR), François Charpillet (LARSEN), Dominique Martinez (LORIA, ISM), Rémi Pannequin (CRAN)
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07546
• Pdf link: https://arxiv.org/pdf/2608.07546
• Abstract Cable-driven parallel robots (CDPRs) present diverse configurations and complex control challenges, which can be addressed by deep reinforcement learning (DRL) by learning their nonlinear dynamics. However, DRL methods often require extensive training time, and the resulting policies do not generalize well to different robot configurations or varying numbers of actuators. In this article, we introduce a novel DRL approach for controlling CDPRs that does not depend on the specific robot configuration. Our method trains an actuator-level policy that controls each motor to achieve its target cable length, in contrast to conventional DRL approaches that learn to control the entire robot to reach a desired end-effector position. To the best of our knowledge, this is the first work to apply DRL to control CDPRs using an actuator-level policy. This approach offers two main advantages: (i) a single shared policy can be applied to any CDPR configuration, regardless of actuator count, and (ii) reliance on inverse kinematics, avoiding the more challenging forward kinematics problem. Training is performed in simulation, and the learned policy is successfully transferred to a real CDPR. Experimental results show that the actuator-level policy (ALP) surpasses traditional reinforcement learning methods in both robustness and precision. We further control a real 8-motor CDPR with 3D motion using a policy trained on a simulated 4-motor planar CDPR operating in 2D. This illustrates that the proposed method is applicable to any CDPR configuration, independent of actuator number or placement.
• 中文摘要 电缆驱动并行机器人（CDPR）呈现出多样的配置和复杂的控制挑战，这些问题可以通过深度强化学习（DRL）通过学习其非线性动力学来解决。然而，DRL方法通常需要大量训练时间，因此其策略无法很好地推广到不同机器人配置或不同数量的执行器。本文介绍了一种不依赖于特定机器人配置的新型DRL控制CDPR方法。我们的方法训练执行器级别策略，控制每个电机达到其目标电缆长度，这与传统DRL方法不同，后者学习控制整个机器人以达到期望的端端执行器位置。据我们所知，这是首次将DRL应用于使用执行器级策略控制CDPR的工作。该方法有两个主要优势：（i）可将单一共享策略应用于任何执行器数量的CDPR配置;（ii）依赖逆运动学，避免了更具挑战性的正向运动学问题。训练在模拟中进行，所学策略成功转移到真实CDPR中。实验结果表明，执行器级策略（ALP）在鲁棒性和精度上优于传统强化学习方法。我们进一步控制一个真实的8电机CDPR及其3D运动，策略训练于模拟的4电机平面CDPR上，该CDPR在2D中运行。这表明所提方法适用于任何CDPR配置，不依赖执行器编号或位置。

Learning an Interior Layout Policy in a Domain Specific Language Action Space

在领域特定语言动作空间中学习内部布局策略

• Authors: Yuhao Lu, Weichen Zhang, Wenyi Xiao, Haohui Chen, Yiyun Fei
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07547
• Pdf link: https://arxiv.org/pdf/2608.07547
• Abstract Indoor scene layout generation is a challenging task in interior design. Existing methods often oversimplify the task by reducing room conditions to coarse 3D bounding boxes and neglecting structural elements such as doors and windows. More fundamentally, many prior approaches formulate spatial reasoning as direct coordinate prediction, thereby casting interior layout design as continuous regression over raw geometric parameters, which hinders the model from learning the underlying reasoning logic of intelligent layout design. We propose \textbf{LayoutDSL}, a novel LLM-based framework for learning an interior layout policy in a domain-specific language (DSL) action space. The DSL provides an explicit symbolic representation of layout information and serves as a structured action space for layout reasoning, where each action corresponds to an interpretable design decision. Under this DSL-based policy learning paradigm, we construct 3D-FrontDSL, a dataset of room-structure annotations paired with synthetic DSL action sequences for supervised fine-tuning. To promote a more generalizable and scalable policy with verifiable feedback, we design rewards grounded in interior design principles and physical plausibility, and optimize the policy via reinforcement learning. Extensive experiments demonstrate that LayoutDSL substantially improves spatial plausibility and design logicality over strong baselines and existing methods.
• 中文摘要 室内场景布局生成是室内设计中的一项具有挑战性的任务。现有方法常常将任务简化为粗略的三维边界框，忽视门窗等结构元素。更根本的是，许多先前方法将空间推理表述为直接坐标预测，从而将室内布局设计视为对原始几何参数的连续回归，阻碍模型学习智能布局设计的底层推理逻辑。我们提出了 \textbf{LayoutDSL}，一种基于大型语言模型的新框架，用于在领域特定语言（DSL）动作空间中学习室内布局策略。DSL 提供了布局信息的显式符号表示，并作为布局推理的结构化动作空间，每个动作对应一个可解释的设计决策。在这一基于DSL的策略学习范式下，我们构建了3D-FrontDSL，这是一个由房间结构注释与合成DSL动作序列结合的数据集，用于监督微调。为了促进更具通用性和可扩展性的策略，并基于可验证的反馈，我们设计基于室内设计原则和物理可信度的奖励，并通过强化学习优化策略。大量实验表明，LayoutDSL相比强基线和现有方法显著提升了空间合理性和设计逻辑性。

What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systems

接下来要编辑什么：对话系统中的视觉对齐图像编辑后续建议

• Authors: Zhijing Zhang, Jinpeng Yu, Xin Song, Bingnan Li, Chuyue Li, Changhui Du, Xiaolin Fang, Jiaming Liu, Ruihua Huang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07565
• Pdf link: https://arxiv.org/pdf/2608.07565
• Abstract Conversational assistants increasingly recommend follow-up edits to help users continue a task. Existing systems primarily target text-only interactions, leaving image-creation conversations underexplored. In image-creation tasks, useful follow-up edit suggestions must reflect user preferences, offer diverse directions, and remain executable on the current image. We collected 100,000 real multi-turn image-creation conversation samples from Qwen App and found that 80.1% are image-dependent, underscoring the need for multimodal recommendation. We address this setting with a three-stage framework. In Stage 1, we use real online data to build a human-reviewed table of appropriate follow-up editing intents, then create SFT targets and fine-tune a multimodal policy. In Stage 2, to align rule-guided SFT suggestions with actual user choices, we use user click feedback to optimize the policy through multi-objective reinforcement learning. In Stage 3, to reduce visual inconsistencies between suggested edits and the current image, we introduce a visual verifier as additional training supervision. Extensive experiments demonstrate that our framework significantly outperforms baselines on both automatic and human evaluations. In a live user-randomized A/B test with millions of users, our final framework reduces visual inconsistency from 3.7% to 0.9%. Furthermore, it significantly improves recommendation CTR by 32.70%, image take-away rate by 16.32%, and average conversation turns per user by 39.90% (all p<0.05).
• 中文摘要 对话助手越来越多地建议后续编辑以帮助用户继续完成任务。现有系统主要针对纯文本交互，导致图像创建对话被忽视。在图像创建任务中，有用的后续编辑建议必须反映用户偏好，提供多样化方向，并且在当前图像上可执行。我们从Qwen应用收集了10万个真实的多回合图像创建对话样本，发现80.1%依赖图像，强调了多模态推荐的必要性。我们通过三阶段框架解决这一设定。第一阶段，我们利用真实在线数据建立一个由人类审核的适当后续编辑意图表，然后创建SFT目标并微调多模态策略。第二阶段，为了使规则引导的SFT建议与实际用户选择对齐，我们利用用户点击反馈通过多目标强化学习优化策略。在第三阶段，为了减少建议编辑与当前图片之间的视觉不一致，我们引入了视觉验证器作为额外的训练监督。大量实验表明，我们的框架在自动和人工评估中均显著优于基线。在一项涉及数百万用户的实时用户随机A/B测试中，我们的最终框架将视觉不一致率从3.7%降至0.9%。此外，它显著提升了推荐点击率32.70%，图片删除率提升了16.32%，平均每用户对话次数提升了39.90%（均为p<0.05）。

Multi-Branch Policy Optimization for Multimodal Large Language Models

多模态大型语言模型的多分支策略优化

• Authors: Shuai Lyu, Yuning Gong, Ruiling Gao, Xiaoran Shang, Zhonghong Ou, Ping Zong, Yifan Zhu, Yuan Sun, Yang Qin, Peng Hu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.07581
• Pdf link: https://arxiv.org/pdf/2608.07581
• Abstract Group-based reinforcement learning methods for multimodal large language models typically rely on trajectory-level credit assignment that applies a single advantage to all tokens in a response. However, multimodal reasoning involves substantially higher perceptual uncertainty than text-only settings, where the model must repeatedly re-examine visual information to verify intermediate interpretations, and different visual groundings can lead to divergent reasoning paths, making such uniform credit assignment particularly inadequate and causing relative advantages to progressively degenerate toward zero. To address these challenges, we propose Multi-Branch Policy Optimization (MBPO), a tree-based framework that constructs reasoning trees at vision-language decision boundaries, enabling sibling branches to explore diverse visual hypotheses and assigning segment-level credit through branch-relative advantages. We further introduce a temporal replay buffer to reuse informative segments while controlling policy staleness. Experiments on several multimodal reasoning benchmarks show that MBPO outperforms representative baselines, improving both learning signal quality and optimization efficiency. The code is publicly available at this https URL.
• 中文摘要 多模态大型语言模型的基于群体的强化学习方法通常依赖轨迹级的信用分配，对响应中的所有标记施加单一优势。然而，多模态推理涉及远高于纯文本环境的感知不确定性，后者模型必须反复重新审视视觉信息以验证中间解释，且不同的视觉基础可能导致推理路径分歧，使得这种统一的信用分配尤为不足，导致相对优势逐渐趋近于零。为应对这些挑战，我们提出了多分支策略优化（MBPO）这一基于树的框架，在视觉-语言决策边界构建推理树，使兄弟分支能够探索多样化的视觉假设，并通过分支相对优势分配分段级的信用。我们还引入了时间重放缓冲区，在控制政策陈旧的同时重用信息段。对多个多模态推理基准测试的实验显示，MBPO优于代表性基线，提升了学习信号质量和优化效率。代码公开于此https网址。

CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning

CODS：可复用离线强化学习的迭代贝尔曼残差数据选择

• Authors: Ibne Farabi Shihab, Sanjeda Akter, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Md Najmus Swaqeeb, Anuj Sharma
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.07719
• Pdf link: https://arxiv.org/pdf/2608.07719
• Abstract Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment. We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions before freezing a reusable subset. Unlike prioritized replay, CODS produces a static artifact; unlike one-shot residual selection, it refreshes scores as the critic changes. At a 10\% budget, CODS retains 96.6\% of eligible-pool performance across 20 valid D4RL task--algorithm cells. It exceeds ReDOR and OPER on 19/20 cells and every other subset baseline on 20/20; all six subset advantages remain significant under predeclared hierarchical inference with Holm correction. Holding total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate thereafter. Equal-pass and equal-hour evaluations clarify that reuse, rather than a single-run speedup, creates the compute advantage. Mechanism and corruption interventions expose both useful sparse-reward enrichment and sensitivity to outliers. Finally, a whole-trace extension retains 95.4\% of pooled ALFWorld success and 96.5\% of pooled GSM8K exact match. CODS is therefore a reusable selection procedure, not a formal coreset guarantee.
• 中文摘要 离线强化学习反复从固定转移池训练策略，使得种子和超参数间的冗余数据成本高昂，而朴素的子采样则可以去除长视野学分分配所需的稀有转移。我们引入了CODS，一种由批评者引导的选择器，交替拟合算法匹配的批评者和获取高残差转移，然后冻结可重用子集。与优先重放不同，CODS产生静态伪影;与一次性残差选择不同，它会随着批评者的变化刷新分数。在10%预算下，CODS保留了20个有效D4RL任务算法单元的合格池性能的96.6%。它在19/20单元格上超过ReDOR和OPER，在20/20单元上超过所有其他子集基线;在预先声明的层级推断和Holm修正下，所有六个子集优势都显著。在总选择器更新固定的情况下，五轮获取可使四个代表性单元在一轮内提升11.23分，之后实现饱和。等通过和等小时评估明确了重用而非单次加速，创造了计算优势。机制和损坏干预既揭示了有用的稀疏奖励丰富，也提高了对异常值的敏感性。最后，全迹扩展保留了95.4%的合并ALFWorld成功率和96.5%的GSM8K精确匹配。因此，CODS是一种可复用的选择程序，而非正式的核心集保证。

LUCID: Latent-Skill Unified Control via Imagined Dynamics for Long-Horizon Humanoid Loco-Manipulation

《清醒：通过想象动力学实现潜在技能统一控制，实现远景类人机车操控》

• Authors: Cheng Guo, Mingzhe Ni, Angelo Cangelosi, Arash Ajoudani
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.07746
• Pdf link: https://arxiv.org/pdf/2608.07746
• Abstract Long-horizon humanoid loco-manipulation requires composing versatile whole-body skills and reliable high-level decision making. Existing methods often coordinate pretrained skills with scripted planners, finite-state machines or task-specific model-free policies, restricting their ability to handle complex task sequences. To address this limitation, we propose \textbf{LUCID}, a hierarchical model-based reinforcement learning framework that plans over reusable skills through imagined rollouts of a learned dynamics model. LUCID first trains a structured latent-conditioned low-level policy via adversarial imitation and then freezes it while jointly learning a high-level policy and macro-dynamics world model. The world model predicts the temporally extended state transitions induced by latent decisions, enabling high-level policy optimization through imagined rollouts. We evaluate our framework across various simulated multi-object rearrangement scenarios. Experimental results show that LUCID improves the full-task success and partial-completion rates compared to prior baseline methods, demonstrating its effectiveness in complex sequential loco-manipulation tasks.
• 中文摘要 长视野类人机车操作需要组合多功能的全身技能和可靠的高层决策。现有方法常将预训练技能与脚本规划器、有限状态机或任务特定无模型策略协调，限制了其处理复杂任务序列的能力。为解决这一限制，我们提出了 \textbf{LUCID}，这是一个层级基于模型的强化学习框架，通过对学习动力学模型的想象展开来规划可重用技能。LUCID 首先通过对抗模仿训练结构化潜条件的低级策略，然后在联合学习高级策略和宏观动态世界模型的同时将其冻结。世界模型预测潜在决策引发的时间延长状态转变，支持通过想象展开实现高层策略优化。我们评估该框架在各种模拟多对象重排场景中的应用。实验结果显示，与以往基线方法相比，LUCID提高了全任务成功率和部分完成率，展示了其在复杂顺序机车操作任务中的有效性。

The Sample Complexity of Policy Learning with Mu-Resets

采用多重重置的策略学习示例复杂性

• Authors: Gene Li
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.07772
• Pdf link: https://arxiv.org/pdf/2608.07772
• Abstract We study policy-based reinforcement learning under the $\mu$-resets interaction protocol of Kakade and Langford [KL02]. This interaction protocol enables the learner to sample trajectories from a given exploratory reset distribution $\mu$, in addition to the starting distribution. We resolve the question raised by [KLS25] on the role of policy realizability for the sample complexity of this problem. Critically, the dependence on horizon $H$ is governed by the notion of coverage assumed of the reset distribution. Under bounded all-policy concentrability, we show a $\exp(\Omega(H))$ sample complexity lower bound; with bounded pushforward concentrability, we show the dependence on horizon is tightly characterized as $\exp(\Theta(\sqrt H))$.
• 中文摘要 我们研究基于策略的强化学习，采用Kakade和Langford [KL02]的$\mu$-reset交互协议。该交互协议使学习者能够从给定的探索性重置分布$\mu$中采样轨迹，此外还采样起始分布。我们解决了[KLS25]提出的关于策略实现性对该问题样本复杂度作用的问题。关键是，视界$H$的依赖性受重置分布假设覆盖的概念支配。在有界全策略聚焦性下，我们展示了样本复杂度的$\exp（\Omega（H））$;在有界推前聚焦性下，我们表明对视界的依赖紧密地表征为$\exp（\Theta（\sqrt H）））$。

V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control

V-Simba：释放强化学习在视觉连续控制中的架构潜力

• Authors: Donghu Kim, Youngdo Lee, Hojoon Lee, Johan Obando-Ceron, Byungkun Lee, Aaron Courville, Pablo Samuel Castro, Jaegul Choo, Clare Lyle
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.07870
• Pdf link: https://arxiv.org/pdf/2608.07870
• Abstract Improving sample efficiency remains a core challenge in reinforcement learning (RL), especially in real-world settings like robotics, where data collection is costly. This challenge is pronounced in visual RL, where high-dimensional inputs often obscure learning signals. While prior work in visual RL has focused on algorithmic solutions, such as better dynamics models or exploration strategies, recent advances in state-based RL show that architectural design alone can lead to significant gains in sample efficiency. This raises an important question: Can these architectural principles transfer to visual RL? In response, we introduce V-Simba, a simple yet effective visual RL architecture inspired by the Simba architecture from state-based RL. Built on top of Soft Actor-Critic (SAC) with data augmentation, V-Simba modifies the architecture by adding normalization layers to stabilize training and using pointwise convolutions to reduce computation. Despite its simplicity, V-Simba matches or outperforms the state-of-the-art methods across the DMC, Adroit, and Meta-World benchmarks, while being more computationally efficient than DrQ-v2. We make our code publicly available at this https URL.
• 中文摘要 提高样本效率仍是强化学习（RL）的核心挑战，尤其是在机器人学等实际环境中，数据采集成本高昂。这一挑战在视觉强化学习中尤为突出，因为高维输入常常遮蔽学习信号。此前视觉强化学习的研究主要集中在算法解决方案，如更好的动力学模型或探索策略，但基于状态的强化学习的最新进展表明，仅靠架构设计就能显著提升样本效率。这引出了一个重要问题：这些架构原则能否转移到视觉强化学习中？为此，我们引入了V-Simba，一种简单但高效的视觉强化学习架构，灵感来自基于状态的强化学习Simba架构。V-Simba基于软演员批判（SAC）并支持数据增强，通过添加归一化层稳定训练，并利用逐点卷积减少计算，对架构进行了修改。尽管简单，V-Simba在DMC、Adroit和Meta-World基准测试中表现优于甚至超越最先进的方法，同时计算效率也优于DrQ-v2。我们将代码公开于此 https URL。

Distilling Physical Priors into Streaming World Models

将物理先验提炼成流世界模型

• Authors: Liangliang Zhao, Junying Wang, Danni Yang, Yifan Chang, Bin Fu, Yu Qiao, Bowen Zhou, Yihao Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.07981
• Pdf link: https://arxiv.org/pdf/2608.07981
• Abstract Streaming world models predict future visual states online while maintaining physically coherent dynamics over long horizons. However, their rollouts often violate basic physical constraints. A common approach distills pretrained bidirectional DiTs into few-step causal generators. However, this paradigm suffers from two fundamental limitations: generic bidirectional teachers acquire limited physical priors from visually oriented pretraining, and the limited priors suffer further loss during bidirectional-to-causal distillation. We present PhyS, a three-stage framework for distilling physical priors into streaming world models. To acquire physical priors from real-world interactions, we construct PhyS-120K, a dataset of 120K real-world physical-interaction videos spanning rigid-body dynamics, soft-body deformation, fluid phenomena, and phase transitions. Each video is annotated with structured descriptions of object properties and causal state transitions. Physics-aware supervised fine-tuning injects the physical priors into a bidirectional 14B DiT teacher, which we then distill into a lightweight 1.3B causal DiT for few-step autoregressive streaming generation. Finally, we use online reinforcement learning to incentivize the distilled model to generate physically plausible rollouts and further propose Temporal Credit Routing (TCR) to address temporal credit assignment. TCR evaluates physical consistency over overlapping temporal windows and routes the resulting group-relative advantages to temporally aligned denoising actions. On PhysicsIQ, PhyS improves the Wan2.1-14B teacher by 18.2\% and the Self Forcing, Rolling Forcing, and Causal Forcing by 23.7\%, 14.8\%, and 31.4\%, respectively. Results also improve the physics-aware video benchmarks VideoPhy, VideoPhy2, and PhyGenBench. The dataset, code, and more sample videos are available on our Project Page.
• 中文摘要 流式世界模型在在线预测未来视觉状态的同时，在长时间内保持物理相干的动态。然而，它们的推广往往违反基本的物理约束。一种常见方法是将预训练的双向DiT提炼成少步因果生成器。然而，这种范式存在两个根本性局限：通用双向教师通过视觉导向的预训练获得有限的物理先验，而有限的先验在双向到因果的蒸馏过程中又会进一步丧失。我们提出了PhyS，一个三阶段框架，用于将物理先验提炼成流世界模型。为了从现实世界相互作用中获取物理先验，我们构建了PhyS-120K，这是一个包含12万段真实物理交互视频的数据集，涵盖刚体动力学、软体变形、流体现象和相变。每个视频都配有对对象属性和因果状态转变的结构化描述。物理感知的监督微调将物理先验注入双向14B的DiT教师中，然后将其提炼为轻量级13B因果DiT，实现少数步自回归流的生成。最后，我们利用在线强化学习激励提炼模型生成物理上合理的展开，并进一步提出时间信用路由（TCR）以解决时间信用分配。TCR评估重叠时间窗口的物理一致性，并将所得的群体相对优势引导至时间对齐的去噪动作。在PhysicsIQ中，PhyS提升了Wan2.1-14B教师18.2%，自强迫、滚动强迫和因果强迫分别提升了23.7%、14.8%和31.4%。结果还提升了物理感知视频基准测试VideoPhy、VideoPhy2和PhyGenBench。数据集、代码及更多示例视频可在我们的项目页面获取。

Ground-Truth Neighborhood Regularization for Reinforcement Learning Post-Training of Time Series Foundation Models

基于时间序列基础模型训练后强化学习的地面真实邻域正则化

• Authors: Jianqi Zhang, Xingyu Zhang, Zeen Song, Changwen Zheng, Fanjiang Xu, Wenwen Qiang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08010
• Pdf link: https://arxiv.org/pdf/2608.08010
• Abstract Time series forecasting (TSF) plays an important role in a wide range of real-world applications. Recently, time series foundation models (TSFMs), pretrained on large-scale datasets, have demonstrated strong generalization capabilities and emerged as an important paradigm for TSF. Reinforcement learning (RL) post-training has consequently attracted growing attention as a means of further improving their performance on downstream tasks. However, we find that, in certain forecast regions, RL post-training may gradually shift the output distributions of TSFMs away from the ground truth, thereby limiting their performance. We refer to this phenomenon as \textbf{suboptimal collapse}. Our analysis suggests that difficulty in initially sampling high-quality trajectories near the ground truth is an important contributing factor to suboptimal collapse. To address this issue, we propose Ground-Truth Neighborhood Regularization (GTN-R) for RL post-training of TSFMs. GTN-R uses the ground truth as a reference for locating high-quality regions and guides the model's probability mass toward the ground-truth neighborhood. This increases the probability of sampling high-quality trajectories, mitigates suboptimal collapse, and improves performance. Moreover, GTN-R can be flexibly integrated into various RL methods for TSFMs. Extensive experiments show its effectiveness.
• 中文摘要 时间序列预测（TSF）在广泛的现实应用中扮演着重要角色。近年来，在大规模数据集上预训练的时间序列基础模型（TSFMs）展现出强大的泛化能力，并成为TSF的重要范式。因此，强化学习（RL）在训练后成为进一步提升其在下游任务表现的手段，受到越来越多的关注。然而，我们发现，在某些预测区域，强化学习后训练可能会逐渐使TSFM的输出分布偏离真实数据，从而限制其性能。我们将此现象称为\textbf{次优坍缩}。我们的分析表明，在接近真实位置初步采样高质量轨迹的困难是导致次优坍缩的重要因素。为解决这一问题，我们提出了用于 TSFM 训练后强化的地面真实邻域正则化（GTN-R）。GTN-R 以地面真实为参考，定位高质量区域，并引导模型的概率质量趋向地面真实邻域。这提高了高质量轨迹采样的概率，减少了次优坍缩，并提升了性能。此外，GTN-R 可以灵活地集成到各种 TSFM 的强化学习方法中。大量实验证明了其有效性。

A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning

强化学习中动态奖励塑造的统一框架

• Authors: Fouad Bahrpeyma
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.08158
• Pdf link: https://arxiv.org/pdf/2608.08158
• Abstract Sparse, delayed, and weakly informative rewards remain central obstacles to efficient reinforcement learning. Reward shaping addresses these limitations by supplementing the task reward with an auxiliary signal that can accelerate learning while, in the classical setting, the original objective remains the evaluation criterion. Established theory guarantees safety for fixed shaping signals: potential-based reward shaping preserves optimal policies when the auxiliary term is the discounted difference of a time-invariant potential. In contemporary reinforcement learning systems, however, both the learner and the information available for guidance evolve during training: value estimates improve, novelty diminishes, feedback shifts, and predictive models are refined. Adaptive reward mechanisms occur across exploration, Bayesian inference, human-in-the-loop learning, automated reward design, and foundation-model-based approaches. This study introduces a unified analytical framework for comparing dynamic reward shaping and neighbouring adaptive reward mechanisms. The proposed framework distinguishes parametric revision from state-dependent variation, separates additive shaping from reward replacement and reward-adjacent guidance, and organises existing methods along temporal, informational, and theoretical dimensions. Using this framework, twelve method families are comparatively analysed. The framework further highlights the conditions under which optimality guarantees survive contemporary deep reinforcement learning pipelines, replay buffers, bootstrapped critics, and reward normalisation, while exposing the unresolved relationship between adaptation rate and learner stability.
• 中文摘要 稀疏、延迟且信息不足的奖励仍然是高效强化学习的主要障碍。奖励塑造通过在任务奖励中补充辅助信号来解决这些限制，辅助信号可以加速学习，而在经典环境中，原始目标仍是评估标准。既有理论保证固定形态信号的安全性：当辅助项为时不变势的贴现差时，基于势的奖励形态保持最优策略。然而，在现代强化学习系统中，学习者和可供指导的信息在训练过程中都会演变：价值估计提升，新颖性减弱，反馈转移，预测模型得到优化。自适应奖励机制存在于探索、贝叶斯推断、人机循环学习、自动奖励设计和基于基础模型的方法等多种方法。本研究提出了一个统一的分析框架，用于比较动态奖励塑造与邻近的自适应奖励机制。所提出的框架区分了参数化修正与状态依赖变异，区分了加法塑造与奖励替代和奖励邻近指导，并从时间、信息和理论维度组织现有方法。利用该框架，对十二个方法家族进行了比较分析。该框架进一步强调了最优性保证在当代深度强化学习管道、重放缓冲区、自助批评者和奖励规范化中存续的条件，同时揭示了适应率与学习者稳定性之间的未解决关系。

VTO: Visual Tool Orchestration for Video Anomaly Detection

VTO：视频异常检测的可视化工具编排

• Authors: Rui Wang, Yeteng Wu, Xianling Zhang, Mengshi Qi
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08219
• Pdf link: https://arxiv.org/pdf/2608.08219
• Abstract Video anomaly detection (VAD) is a critical yet challenging task due to the complex and diverse nature of real-world scenarios. Traditional deep learning approaches are fundamentally limited by poor generalization across diverse scenarios. While multimodal agents offer a promising tool-learning paradigm for VAD, current systems relying on supervised fine-tuning struggle with complex orchestration, and standard reinforcement learning often causes premature termination due to coarse-grained outcome rewards. To address these challenges, we propose VTO, a process-supervised reinforcement learning framework. Moving beyond static tool usage, VTO enables the agent to dynamically explore and interact with the environment. Specifically, we introduce a foundation model-driven cognitive evaluator to provide context-aware semantic feedback, which is seamlessly integrated into a Process-Supervised Cognitive Alignment that delivers fine-grained, step-wise supervision. By explicitly penalizing logical truncation and rewarding complete causal chains, the agent optimizes its multi-step reasoning policy for interrelated tool orchestration. To support our proposed framework, we meticulously crafted VAD-Tool, a hierarchical visual tool set comprising 12 specialized vision tools spanning from entity tracking to high-stakes hazard detection, and established the corresponding benchmark for rigorous multi-step reasoning evaluation. Extensive experiments on VAD-Tool demonstrate that VTO significantly outperforms baselines, achieving up to a 10.2\% absolute accuracy improvement in tool scheduling. Code and data are available at this https URL.
• 中文摘要 视频异常检测（VAD）是一项关键但具有挑战性的任务，因为现实场景复杂多样。传统的深度学习方法在不同场景间的泛化能力有限，根本上受限于其能力。虽然多模态代理为VAD提供了有前景的工具学习范式，但目前依赖监督微调的系统在复杂编排和标准强化学习中常因粗粒度结果奖励而导致提前终止。为应对这些挑战，我们提出了VTO，一种过程监督强化学习框架。超越静态工具使用，VTO使智能体能够动态探索和与环境交互。具体来说，我们引入了基于基础模型的认知评估器，提供上下文感知语义反馈，并无缝集成到过程监督认知对齐中，提供细粒度的分阶段监督。通过明确惩罚逻辑截断并奖励完整的因果链，智能体优化了其多步推理策略，以实现相互关联的工具编排。为支持我们提出的框架，我们精心设计了VAD-Tool，这是一套包含12个专业视觉工具的分层视觉工具集，涵盖从实体追踪到高风险危害检测，并建立了严格多步推理评估的相应基准。VAD-Tool的广泛实验表明，VTO在工具调度方面显著优于基线，绝对准确率提升高达10.2%%。代码和数据可在此 https URL 获取。

Metanormative Theory for RL-Based Moral Agents

基于强化学习的道德代理的元规范理论

• Authors: Aleks Knoks, Marija Slavkovik
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08220
• Pdf link: https://arxiv.org/pdf/2608.08220
• Abstract The overlapping disciplines of machine ethics and value alignment are concerned with designing artificial agents that are aligned with human values and that act in ethically acceptable ways. A recent trend in these disciplines is the use of reinforcement learning (RL) to design such agents, sidelining the philosophical literature that used to play a more central role. Against this backdrop, this paper pursues two goals. The first is to draw out ideas from recent work in metanormative theory that can be useful for designing artificial moral and value-aligned agents. The second is to examine the RL architecture through the lens of these ideas. This will give us clearer criteria for when an RL agent's behavior can be classified as moral, as well as a basis for evaluating and comparing different RL-based approaches to machine ethics and value alignment.
• 中文摘要 机器伦理学和价值对齐这两个重叠学科关注于设计符合人类价值观、并以伦理可接受方式行动的人工代理。这些学科的最新趋势是使用强化学习（RL）来设计此类代理，取代了曾经扮演更核心角色的哲学文献。在此背景下，本文追求两个目标。第一，从元规范理论的最新研究中提取有助于设计人工道德和价值对齐代理的思想。第二，通过这些思想的视角审视强化学习架构。这将为我们更明确地判断何时强化学习代理的行为可归类为道德，并为评估和比较不同基于强化学习的机器伦理与价值对齐方法提供基础。

Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training

控制多样性强化微调：解耦强化学习后共享控制瓶颈

• Authors: Binwen Tan, Jingchao Wang, Dengzhe Hou, Lingyu Jiang, Zeyuan Wu, Yunhan Shen, Fangzhou Lin, Kazunori Yamada, Atsushi Koike
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.08224
• Pdf link: https://arxiv.org/pdf/2608.08224
• Abstract Reinforcement learning post-training unlocks complex reasoning in LLMs. Yet benchmark scores reveal only whether a model improved, not what changed inside it, nor how it splits finite capability across tasks. A representative interpretability line attributes the success of RL fine-tuning to stronger and more diverse circuit activation. We challenge this activation-centered account by separating activation from control: an activated circuit need not control the post-training reward gain. Adapting Metabolic Control Analysis, we define the Post-training Control Coefficient to measure component control over reward gain and arrange these coefficients by task family into a control matrix, paired with an activation-magnitude matrix. We call cross-task control concentration the Shared Control Bottleneck and the difference between activation and control concentration the Activation-Control Gap. This reveals that highly shared activations can coexist with task-specific control, while a small gap indicates that control has collapsed onto a shared direction and lost task specificity. To reduce this collapse, we regularize the post-training loss with the Shared Control Bottleneck and propose Control-Diverse Reinforcement Fine-Tuning (CD-RFT). The exact regularizer gradient requires second-order automatic differentiation incompatible with flash attention, so we derive a first-order proxy with worst-case overhead below eight percent. On Qwen2.5-7B, CD-RFT achieves the largest control decoupling and improves multi-task capability over matched GRPO across mathematics, code, and logic. The no-KL variant leads on pass@1, and the KL-penalized variant leads on large-k pass@k coverage that KL otherwise degrades. Together, these results show that the Shared Control Bottleneck is both a mechanistic diagnostic and a training regularizer, and that control decoupling and capability gains transfer to Llama-3.2-3B.
• 中文摘要 训练后强化学习解锁了大型语言模型中的复杂推理能力。然而基准评分仅揭示模型是否改进，而非模型内部的变化，或其如何将有限能力划分于任务。一条具有代表性的可解释性线将强化学习微调的成功归因于更强且多样化的回路激活。我们通过将激活与控制分离来挑战这一以激活为中心的说法：激活后的电路不必控制训练后的奖励获得。基于代谢控制分析，我们定义了训练后控制系数，以衡量对奖励增益的组成控制，并将这些系数按任务族排列成控制矩阵，配合激活强度矩阵。我们将跨任务控制集中称为共享控制瓶颈，将激活与控制集中的差异称为激活-控制差距。这表明高度共享的激活可以与任务特定控制共存，而小差距则表示控制已崩溃到共享方向并失去任务特异性。为减少这种崩溃，我们用共享控制瓶颈正则化训练后丢失，并提出控制多样性强化微调（CD-RFT）。精确的正则化梯度需要二阶自动微分，不兼容闪烁注意力，因此我们推导出一个最坏情况开销低于8%的一阶代理。在Qwen2.5-7B上，CD-RFT实现了最大的控制解耦，并在数学、代码和逻辑上相较匹配GRPO提升了多任务能力。无KL版本在pass@1上领先，KL惩罚版本在大k pass@k覆盖方面领先，而KL否则会削弱这些覆盖。综合来看，这些结果表明共享控制瓶颈既是机械诊断也是训练规范器，控制解耦和能力提升可转移至Llama-3.2-3B。

Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning

从环境反馈中学习：跨多时间尺度的代理强化学习学分分配

• Authors: Yifu Huo, Shunjie Xing, Chenglong Wang, Peinan Feng, Qiaozhi He, Yan Ding, Anxiang Ma, Yuxin Gao, Tongran Liu, Tong Xiao, Jingbo Zhu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.08255
• Pdf link: https://arxiv.org/pdf/2608.08255
• Abstract Agentic reinforcement learning (RL) often suffers from delayed and sparse rewards in real-world environments. A promising solution to this challenge is credit assignment, which aims to decompose trajectory-level rewards and provide more fine-grained supervision for intermediate decisions. However, existing credit assignment approaches ignore the rich process information naturally generated during environment interaction, e.g., interaction history. We argue that such information provides valuable supervision for identifying the contribution of individual actions. To this end, we propose Environmental Feedback-based Credit Assignment (EFCA), a multi-timescale credit assignment approach for long-horizon agentic RL. EFCA complements the long-term outcome signal with two environment-grounded process signals: a short-term feedback signal that captures the immediate effect of the current action and a medium-term state-history signal that identifies ineffective patterns from recent interactions. Both signals are directly extracted from environment feedback and integrated through a return reweighting mechanism. Experiments on ALFWorld and WebShop demonstrate that EFCA consistently improves both task success and task quality over strong baselines, highlighting the effectiveness of environment-grounded multi-timescale credit assignment for long-horizon agentic RL.
• 中文摘要 代理强化学习（RL）在现实环境中常常存在延迟和稀疏的奖励问题。一个有前景的解决方案是信用分配，它旨在分解轨迹级奖励，并为中间决策提供更细致的监督。然而，现有的信用分配方法忽视了环境交互过程中自然产生的丰富过程信息，例如交互历史。我们认为，这些信息为识别个别行为的贡献提供了宝贵的监督。为此，我们提出了基于环境反馈的信用分配（EFCA），这是一种多时间尺度的长期代理性强化学习的信用分配方法。EFCA通过两种环境基础过程信号补充长期结果信号：一种短期反馈信号捕捉当前行为的即时影响，以及中期状态历史信号，识别近期互动中无效的模式。这两个信号均直接从环境反馈中提取，并通过返回重加权机制进行整合。ALFWorld和WebShop上的实验表明，EFCA在强基线下持续提升任务成功率和任务质量，凸显了环境基础多时间尺度学分分配对长期能动性强化学习的有效性。

StructReward: Efficient Structured Process Rewards for Self-Correcting Multimodal Reasoning

StructReward：高效且结构化的过程奖励，用于自我纠正多模态推理

• Authors: Yifan Li, Ruxin Sun, Tongzhou Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08326
• Pdf link: https://arxiv.org/pdf/2608.08326
• Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective approach for improving multimodal reasoning. However, most existing methods evaluate an entire response using a binary reward based only on final-answer correctness, thereby discarding the supervision available in intermediate reasoning steps. Process reward models offer finer-grained feedback, but they typically rely on separately trained verifiers, costly chain-of-thought annotations, or online judging by large language models (LLMs). In this work, we introduce StructReward, a compute-efficient framework that provides dense reinforcement signals through structured step-level reward alignment. StructReward represents each generated solution as a sequence of reasoning steps and aligns them with process-labeled reference steps using lightweight numerical, symbolic, and lexical matching rules. The aligned labels are aggregated into a dense process reward and combined with final-answer consistency and output-validity rewards through a gated Group Relative Policy Optimization (GRPO) objective. We further recycle policy rollouts into complementary supervision for response comparison and reflective self-correction, rather than discarding them after policy updates. Separately, we use a strong LLM to rewrite sampled correct trajectories into reflection-oriented training instances, further strengthening the policy's ability to evaluate and refine its reasoning. Since reward computation is performed online without an additional learned verifier or external LLM judge, StructReward substantially reduces the computational overhead of multimodal reinforcement learning. Experimental results show that structured process supervision and rollout recycling provide an efficient path toward self-improving multimodal reasoning.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为提升多模态推理的有效方法。然而，大多数现有方法仅基于最终答案正确性，使用二元奖励评估整个反应，从而摒弃了中间推理步骤中的监督。过程奖励模型提供更细粒度的反馈，但通常依赖于单独训练的验证器、昂贵的思维链注释，或由大型语言模型（LLMs）在线判断。本研究介绍了StructReward，一个计算高效的框架，通过结构化的步骤级奖励对齐提供密集的强化信号。StructReward将每个生成的解表示为一系列推理步骤，并使用轻量级数值、符号和词汇匹配规则将其与过程标记的参考步骤对齐。对齐标签被聚合成密集过程奖励，并通过门槛组相对策略优化（GRPO）目标与最终答案一致性和输出有效性奖励结合。我们还将策略推广回收为补充监督，用于响应比较和反思自我纠正，而非策略更新后丢弃。另外，我们使用强有力的大型语言模型将采样的正确轨迹重写为面向反思的训练实例，进一步强化策略评估和完善推理的能力。由于奖励计算在线完成，无需额外学习验证者或外部LLM评判，StructReward大幅降低了多模态强化学习的计算开销。实验结果表明，结构化过程监督和展开循环利用为自我改进多模态推理提供了高效路径。

TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models

TrustRoboReward：多范式机器人奖励模型的偏好排序等张评分编辑

• Authors: Yidong Wang, Yan Zhan, Ziteng Feng, Zhenyu Cui, Ziyi Zhou, Renzhao Liang, Jiaxuan Zhu, Zilei Yang, Yiran Zhao, Zhongkuan Mao, Bo Jia, Hanchu Ni, Chenggang Xie, Biao Liu, Yi Zhang, Yong Dai, Xiaozhu Ju, Wei Ye, Shikun Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08491
• Pdf link: https://arxiv.org/pdf/2608.08491
• Abstract Reward models are a bottleneck for reinforcement learning in embodied AI. Long-horizon robotic manipulation requires scalable vision feedback beyond handcrafted rewards or task-specific annotations. Existing open-source VLM reward judges like RoboReward adopt simple 1--5 trajectory progress scoring, lacking pairwise preferences for RLHF, DPO and Bradley-Terry frameworks, while failing to optimize video scene understanding. Augmenting RoboReward with pairwise comparison and video-QA supervision causes inconsistency between pairwise preferences and pointwise scores, introducing training noise and hurting downstream performance---an issue aggregation methods such as TrustJudge cannot resolve. To address this, we propose TrustRoboReward, a multi-paradigm reward modeling framework equipped with Preference-Ordered Isotonic Score Editing (POISE). We construct a unified four-paradigm dataset with trajectory progress scoring (Score-A), video-QA answer quality scoring (Score-B), and their pairwise counterparts (Pair-A, Pair-B). Pairwise labels align better with human judgment than pointwise scores, inspiring us to calibrate pointwise scores to avoid score-pair reversals against pairwise preferences. POISE rectifies pointwise scores and eliminates cross-paradigm reversal conflicts unresolved by TrustJudge. Theoretically, POISE reduces score-pair reversal conflicts from 20.15% to 0%, whereas TrustJudge retains 20.46% conflicts on the same corpus. Evaluated on our benchmark, Qwen3-VL-4B trained with POISE achieves an overall reward score of 77.96%, nearly matching GPT-5-mini (78.09%, gap 0.13%) and outperforming the strongest RoboReward-4B baseline by 10.13%. It also lifts test-time score-pair consistency to 71.90%, exceeding RoboReward-4B (57.26%) and GPT-5-mini (68.09%). Integrating TrustJudge aggregation during inference boosts the overall score to 78.57%, surpassing the GPT-5-mini teacher model.
• 中文摘要 奖励模型是具身人工智能强化学习的瓶颈。长视野机器人操作需要超越手工奖励或任务特定注释的可扩展视觉反馈。现有开源VLM奖励评判器如RoboReward采用简单的1-5轨迹进度评分，缺乏RLHF、DPO和Bradley-Terry框架的两对偏好，且未能优化视频场景理解。通过两对比较和视频-质量保证监督增强RoboReward会导致成对偏好与点数评分不一致，引入训练噪声并损害下游性能---这是TrustJudge等聚合方法无法解决的问题。为此，我们提出了TrustRoboReward，一个配备偏好排序等张分数编辑（POISE）的多范式奖励建模框架。我们构建了一个统一的四范式数据集，包含轨迹进展评分（Score-A）、视频-QA答案质量评分（Score-B）及其成对对应（Pair-A， Pair-B）。成对标签比分点评分更贴合人类判断，激励我们校准分点评分以避免与成对偏好的分数对逆转。POISE纠正了点数分，消除了TrustJudge未解决的跨范式逆转冲突。理论上，POISE将分数对逆转冲突从20.15%降至0%，而TrustJudge在同一语料库上保留20.46%的冲突。基于我们的基准测试，使用POISE训练的Qwen3-VL-4B整体奖励得分为77.96%，几乎与GPT-5-mini（78.09%，差距0.13%）相当，并且比最强的RoboReward-4B基线高出10.13%。它还将测试时的得分对一致性提升至71.90%，超过RoboReward-4B（57.26%）和GPT-5-mini（68.09%）。在推理过程中整合TrustJudge聚合，整体得分提升至78.57%，超过了GPT-5-mini教师模型。

Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies

结构化参数环境下的课程生成，以实现稳健的导航政策

• Authors: Prishita Ray
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.08545
• Pdf link: https://arxiv.org/pdf/2608.08545
• Abstract Robust navigation policies for autonomous agents must generalize across continuously varying environmental conditions such as turn rates, obstacles, friction, pits, and slopes. Curriculum generation provides a principled mechanism for improving generalization by progressively adapting training environments, but designing such curricula in a sample-efficient and automated manner remains challenging. This paper proposes a reparameterized curriculum generation framework for structured continuous environment parameters using unidirectional gradient-based optimization. To improve robustness in multimodal observation spaces consisting of image-based and scalar inputs, a distribution-shift regularization objective is incorporated to encourage the learning of finer-grained latent representations. The proposed method is evaluated across two continuous-control OpenAI Gym environments: a 2D obstacle-based Car Racing variant and Bipedal Walker variant, where coupled environment parameters jointly influence policy performance. Across five random seeds, our method consistently outperforms vanilla policy training, random parameter sampling, manual curricula, frontier-based methods, Self-Paced Reinforcement Learning (SPRL), Absolute Learning Progress with Gaussian Mixture Models (ALP-GMM), and reverse curriculum learning baselines. Ablation studies further demonstrate the effectiveness of the reparameterized curriculum mechanism across both environments, while highlighting environment-dependent benefits of the auxiliary regularization objective.
• 中文摘要 自主智能体的稳健导航策略必须推广到连续变化的环境条件，如转向率、障碍物、摩擦、坑洼和坡度。课程生成提供了通过逐步调整训练环境来提升泛化的原则机制，但以样本高效和自动化的方式设计此类课程仍具挑战性。本文提出了一个基于单向梯度优化的结构化连续环境参数重新参数化课程生成框架。为提升由图像和标量输入组成的多模态观察空间中的鲁棒性，采用分布-移位正则化目标，鼓励学习更细粒度的潜在表示。该方法在两种连续控制的OpenAI健身房环境中进行评估：基于障碍的二维赛车变体和双足步行变体，耦合环境参数共同影响策略表现。在五个随机种子中，我们的方法始终优于普通策略训练、随机参数抽样、手动课程、基于前沿的方法、自定进度强化学习（SPRL）、高斯混合模型的绝对学习进展（ALP-GMM）以及反向课程学习基线。消融研究进一步证明了重新参数化课程机制在两种环境中的有效性，同时强调辅助正则化目标的环境依赖性益处。

Knowledge-Distilled End-to-End Reinforcement Learning for Smooth 6-DOF Thrust Control and Rapid Adaptation to Ocean Currents in Remotely Operated Vehicles

知识蒸馏的端到端强化学习，实现平滑的6自由度推力控制及对洋流的快速适应，适用于遥控飞行器

• Authors: Tiankuang Wen, Huiping Li, Gang Liu, Yong Jiang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.08598
• Pdf link: https://arxiv.org/pdf/2608.08598
• Abstract With the continuous improvement of computational capabilities, end-to-end reinforcement learning has been rapidly developed for remotely operated vehicles control. Nevertheless, existing end-to-end reinforcement-learningbased methods still face challenges in achieving optimal control under oceancurrent disturbances. In particular, there remains a lack of a unified control framework that can simultaneously achieve low steady-state tracking error, rapid transient response, energy-efficient operation, and smooth controlforce outputs under disturbances. To address the issue, this paper proposes the thrust smoothness rapid current adaptation proximal policy optimization (TSRCA-PPO) method which learns a near-optimal strategy by a twostage distillation learning framework. The core innovations of this work lie in the reward-function design and the privileged multi-encoder architecture. Ablation studies validate the effectiveness of each module. Simulation results demonstrate that the proposed TSRCA-PPO method consistently outperforms the conventional cascaded P-PID controller across all evaluation metrics. Specifically, TSRCA-PPO reduces the steady-state position error, steady-state attitude error, settling time, energy index, and thrustsmoothness index to 42.7%, 76.5%, 10.6%, 93.5%, and 15.9% of the corresponding P-PID values, respectively.
• 中文摘要 随着计算能力的持续提升，端到端强化学习已被迅速发展用于遥控车辆控制。然而，现有基于端到端强化学习的方法在洋流扰动下实现最优控制仍面临挑战。特别是，目前尚缺乏能够同时实现低稳态跟踪误差、快速瞬态响应、节能运行以及在扰动下平稳输出控制力的统一控制框架。为解决这一问题，本文提出了推力平滑快速电流适应近端策略优化（TSRCA-PPO）方法，该方法通过两阶段蒸馏学习框架学习近似最优策略。本工作的核心创新在于奖励函数设计和特权多编码器架构。消融研究验证了每个模块的有效性。模拟结果表明，所提TSRCA-PPO方法在所有评估指标上始终优于传统的级联P-PID控制器。具体来说，TSRCA-PPO将稳态位置误差、稳态姿态误差、稳定时间、能量指数和推力平滑指数分别降低至42.7%、76.5%、10.6%、93.5%和15.9%的P-PID相关值。

Multi-Agent Reinforcement Learning via Agent-Specific Preference

通过智能体特定偏好进行多智能体强化学习

• Authors: Ni Mu, Yao Luan, Yiqin Yang, Qing-Shan Jia
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.08604
• Pdf link: https://arxiv.org/pdf/2608.08604
• Abstract Multi-agent reinforcement learning (MARL) is a powerful framework for solving complex collaborative tasks, but it relies heavily on well-defined global reward functions. Designing such rewards is challenging, especially in systems with heterogeneous agents, where a single scalar objective may fail to capture diverse behaviors. In this paper, we introduce Multi-AGent Preference-Integrated lEarning (MAGPIE), which addresses these challenges through agent-specific preference modeling. Each agent is evaluated by a dedicated expert through preference signals, eliminating the need for global evaluation. We theoretically prove that optimizing these decentralized preferences converges to a Nash equilibrium policy. To integrate local preferences into a coherent global objective, we construct agent-specific reward models from preference data and combine them via a monotonic aggregation mechanism. We further prove that optimizing this aggregate reward model is equivalent to training the Nash equilibrium policy. Extensive experiments on benchmark multi-agent tasks and a sequential production line task show that MAGPIE achieves performance comparable to reward-engineered baselines, demonstrating its potential to facilitate policy learning in scenarios where precise reward engineering is impractical.
• 中文摘要 多智能体强化学习（MARL）是一个强大的框架，用于解决复杂的协作任务，但它高度依赖于明确定义的全局奖励函数。设计此类奖励具有挑战性，尤其是在智能体异构的系统中，单一标量目标可能无法捕捉多样化行为。本文介绍了多元偏好集成lEarning（MAGPIE），通过智能体特定偏好建模解决这些挑战。每个智能体都由专属专家通过偏好信号进行评估，无需全局评估。我们理论上证明，优化这些去中心化偏好趋向纳什均衡策略。为了将局部偏好整合成连贯的全局目标，我们从偏好数据构建智能体特定奖励模型，并通过单调聚合机制将它们组合起来。我们进一步证明，优化该总体奖励模型等同于训练纳什均衡策略。对基准多智能体任务和顺序生产线任务的广泛实验表明，MAGPIE的性能可与奖励工程基线相当，展示了其在精确奖励工程不切实际场景下促进策略学习的潜力。

MedCalc-R1: Knowledge-Guided Reward Framework for Medical Mathematical Reasoning

MedCalc-R1：医学数学推理的知识引导奖励框架

• Authors: Haotian Wang, Lian Yan, Xingzhi Yao, Fanshu Meng, Ye He, Jingchi Jiang, Yi Guan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08623
• Pdf link: https://arxiv.org/pdf/2608.08623
• Abstract In Reinforcement Learning with Verifiable Rewards (RLVR) frameworks for mathematical reasoning tasks, floating-point results are typically evaluated using a tolerance-based reward. However, this strategy suffers from challenges such as difficulty in threshold calibration, unstable training dynamics, and limited accuracy, especially in clinical scenarios. To address these limitations, we propose a knowledge-guided hybrid reward framework (\textsc{MedCalc-R1}). Specifically, we introduce a knowledge verification reward mechanism that enforces explicit generation of computational formulas, which are further validated by an external verifier to enhance interpretability and reasoning reliability. Furthermore, we design a hybrid soft-hard reward scheme combining a hard constraint based on clinical safety thresholds with a soft, precision-sensitive reward that progressively guides learning within the acceptable range. Experimental results demonstrate that our method significantly outperforms existing baselines in both reasoning accuracy and generalization capability, validating the effectiveness and applicability in safety-critical domains.
• 中文摘要 在可验证奖励强化学习（RLVR）框架中，浮点数结果通常采用容忍型奖励来评估。然而，该策略存在阈值校准困难、训练动态不稳定以及临床场景中准确性有限等挑战。为解决这些局限性，我们提出了知识引导混合奖励框架（\textsc{MedCalc-R1}）。具体来说，我们引入了知识验证奖励机制，强制明确生成计算公式，并通过外部验证者进一步验证，以提升可解释性和推理可靠性。此外，我们设计了一种软硬奖励混合方案，结合基于临床安全阈值的硬约束与软性、精度敏感的奖励，逐步引导学习在可接受范围内。实验结果表明，我们的方法在推理准确性和泛化能力方面显著优于现有基线，验证了其在安全关键领域的有效性和适用性。

Trajectory Design and Budgeted Querying for Digital Twin Calibration

轨迹设计与数字孪生校准的预算查询

• Authors: Vladyslava Spitkovska, Dmytro Kuzmenko
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.08631
• Pdf link: https://arxiv.org/pdf/2608.08631
• Abstract Digital-twin calibration requires interaction data that is expensive to collect. We study two acquisition decisions: which trajectories to generate, and when to spend a limited budget on privileged parameter measurements. Our framework couples an excitation-oriented reinforcement learning controller, a recurrent parameter estimator with predictive uncertainty, and a budgeted query policy. In Pendulum, a Random Forest diagnostic recovers gravity only weakly from task-oriented trajectories and does not recover mass or length, while a GRU trained on excitation-oriented trajectories reaches a mean absolute error of 0.0066 with no queries. We then withdraw continuous oracle access partway through an episode, so that the twin must run on the estimator's output for the remainder. The estimator-plus-policy pipeline achieves a terminal error of 0.0092 under a three-query budget, against 0.2031 for an uncalibrated twin. In partially observable Waterworld, five controllers produce different observed error profiles across three hidden parameters, and an estimator trained on a five-controller mixture reaches online normalized errors of roughly 4-5%. These exploratory case studies are not controlled ablations, but they motivate treating trajectory design and query allocation as explicit design variables in data-scarce calibration.
• 中文摘要 数字孪生校准需要收集昂贵的交互数据。我们研究了两个获取决策：生成哪些轨迹，以及何时在特权参数测量上投入有限预算。我们的框架结合了面向激发的强化学习控制器、带有预测不确定性的重复参数估计器和预算查询策略。在Pendulum中，随机森林诊断仅能从任务导向轨迹中弱恢复重力，且不恢复质量或长度;而训练于激励导向轨迹的GRU在无查询时达到平均绝对误差0.0066。随后，在一集中途撤回连续预言机访问，使孪生节点在剩余时间内必须依赖估计器的输出运行。估计器加策略流水线在三查询预算下实现0.0092的终端误差，而未校准的双生组为0.2031。在部分可观测的Waterworld中，五个控制器在三个隐藏参数上产生不同的观察误差轮廓，训练于五个控制器混合的估计器在线归一化误差约为4-5%。这些探索性案例研究不是受控消融，但它们促使将轨迹设计和查询分配视为数据稀缺校准中的显式设计变量。

Catastrophic Forgetting in Continual Reinforcement Learning

持续强化学习中的灾难性遗忘

• Authors: Emma Graham
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.08673
• Pdf link: https://arxiv.org/pdf/2608.08673
• Abstract This work explores the relationship between task similarity and catastrophic forgetting in reinforcement learning. Catastrophic forgetting, the phenomenon in machine learning of losing the ability to effectively perform on previous tasks, is a significant impediment to continual learning. This study aims to understand the extent to which the similarity of a new task influences the performance on the previous task. Interpretable reinforcement learning, specifically Q-learning, is employed on graph-based tasks with the objective of minimising the number of steps to reach a goal. The study investigates the performance on a previously learned task after training on a new task, for tasks of varying relative levels of complexity. The experimental results reveal a complex dynamic between task similarity and forgetting, with significant fluctuations in forgetting severity observed across degrees of task similarities and task complexities, and are suggestive of an interdependence of forgetting on the similarity and complexity of tasks. The observations were accompanied by observations of high degrees of variability in forgetting and an uneven distribution of task similarity measures. The relationship between these variables remains unclear and no evidence of statistical significance that task similarity has an effect, independently, on forgetting is found in continual reinforcement learning. Further research is warranted to gain a comprehensive understanding of the potential interplay between task similarity and catastrophic forgetting.
• 中文摘要 本研究探讨了强化学习中任务相似性与灾难性遗忘之间的关系。灾难性遗忘，即机器学习中失去有效完成前一任务能力的现象，是持续学习的重要障碍。本研究旨在理解新任务的相似性对前一任务表现的影响程度。可解释强化学习，特别是Q-learning，应用于基于图的任务，旨在最小化实现目标所需的步骤数。本研究调查了在新任务训练后，针对不同复杂程度任务的先前学习任务的表现。实验结果显示任务相似性与遗忘之间存在复杂动态，遗忘严重度在任务相似度和复杂度程度上存在显著波动，暗示遗忘与任务相似性和复杂度之间的相互依赖。观察结果伴随着遗忘差异较高和任务相似度分布不均的观察。这些变量之间的关系尚不明确，且在持续强化学习中未发现任务相似性对遗忘产生独立影响的统计显著性证据。需要进一步研究以全面理解任务相似性与灾难性遗忘之间的潜在相互作用。

Improving Generalization Robustness of Multimodal RLVR

提升多模RLVR的泛化鲁棒性

• Authors: Pengfei Zhou, Zhiwei Tang, Xiaopeng Peng, Chenrui Zhou, Lama Moukheiber, Yixing Ma, Bin Xu, Jiajun Song, Zhenglin Wan, Wangbo Zhao, Jiasheng Tang, Bohan Zhuang, Fan Wang, Yang You
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08802
• Pdf link: https://arxiv.org/pdf/2608.08802
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models more accurate, but the gains are brittle: simply paraphrasing a question or changing the prompt template can degrade them, which challenges reliable deployment in high-stakes scenarios like medical VQA. We trace this to two issues of the standard RL objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training distribution covers only a thin slice of the real-world prompts that the model might meet at deployment, so policies that perform well on the training distribution can behave differently under unseen prompts during test. Both failures call for a robust post-training method that helps the policy cover a broader distribution of semantically equivalent prompts, and we identify two measures that help achieve this objective: separating format from semantics in the reward, and applying policy invariance across perturbed prompts with equivalent semantics. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer based on an embedding-space adversary. Under stress testing, PIRL's average accuracy on benchmarks drops by only $\le 1\%$, where GRPO drops ~3%. On dynamic evaluation, PIRL also achieves the smallest performance drop.
• 中文摘要 带可验证奖励的强化学习（RLVR）使多模态大型语言模型更准确，但其收益有限：仅仅改写问题或修改提示模板可能会降低其质量，这在医疗VQA等高风险场景中难以可靠部署。我们将此归因于标准强化学习目标的两个问题。首先，二进制验证器将格式与内容混淆，因此奖励信号无法区分错误答案和格式错误。其次，训练分布只覆盖模型在部署时可能满足的现实提示的一小部分，因此在测试中表现良好的策略在未见提示下可能表现不同。这两种失败都需要一种稳健的训练后方法，帮助策略覆盖更广泛的语义等效提示分布，我们确定了两个有助于实现这一目标的措施：在奖励中将格式与语义分离，以及在具有等效语义的扰动提示上应用策略不变性。因此，我们提出了提示不变RLVR（PIRL），由动态三元奖励和基于嵌入空间对手的一致性正则化器组成。在压力测试下，PIRL在基准测试中的平均准确率仅下降$\le 1\%$，而GRPO下降~3%。在动态评估中，PIRL也实现了最小的性能下降。

ML-Based Hierarchical Prediction for Practical Energy Scheduling in Dynamic NTN-WPT Systems

基于机器学习的层级预测，用于动态NTN-WPT系统中实际的能源调度

• Authors: Zhanyu Ju, Wenchi Cheng
• Subjects: Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.08804
• Pdf link: https://arxiv.org/pdf/2608.08804
• Abstract With advancements in long-distance wireless power transfer (WPT) and space-based energy technologies, integrating WPT into non-terrestrial networks (NTNs), referred to as NTN-WPT, is emerging as a promising approach for next-generation wireless networks. This paper proposes an energy-scheduling approach that jointly optimizes energy efficiency, task completion rate, and task waiting time for power transfer from low Earth orbit satellites to terrestrial mobile user devices (UDs). To address scheduling challenges caused by satellite and UD mobility and channel uncertainty from stochastic propagation effects, we decompose the problem into three subproblems within a three-layer predictive framework: 1) a state prediction layer forecasts UD and satellite states; 2) an interaction mapping layer uses a graph neural network (GNN) to model energy transfer efficiency; and 3) a decision-making layer determines the energy allocation plan. Distinct machine learning (ML) methods are tailored to each layer. To balance the competing objectives, we adopt a multi-objective reinforcement learning (MORL) technique that scalarizes them into a weighted-sum reward, transforming the multi-objective problem into a tractable single-objective problem. We further introduce a multi-agent deep learning model integrating self-attention with multi-agent proximal policy optimization (MAPPO) to improve objective balancing. Simulation results show that the proposed approach achieves a better overall trade-off than baseline methods, maintaining competitive task completion rates and energy efficiency while reducing task waiting times, and remains robust under highly variable conditions.
• 中文摘要 随着远程无线电力传输（WPT）和空间能源技术的进步，将WPT集成到非地面网络（NTN），即NTN-WPT，正成为下一代无线网络的一种有前景的方法。本文提出了一种能量调度方法，结合低地轨道卫星向地面移动用户设备（UD）的能量传输效率、任务完成率和任务等待时间进行优化。为应对卫星和UD移动性带来的调度挑战，以及随机传播效应带来的通道不确定性，我们将问题分解为三层预测框架下的三个子问题：1）状态预测层预测UD和卫星状态;2）交互映射层使用图神经网络（GNN）来建模能量传输效率;3）决策层确定能源分配计划。针对每个层级定制了不同的机器学习（ML）方法。为平衡竞争目标，我们采用多目标强化学习（MORL）技术，将其标量化为加权和奖励，将多目标问题转变为可处理的单目标问题。我们进一步引入了整合自我关注与多智能体近端策略优化（MAPPO）的多智能体深度学习模型，以改善目标平衡。模拟结果显示，所提方法比基线方法实现了更好的整体权衡，保持了具有竞争力的任务完成率和能效，同时减少任务等待时间，并且在高度可变条件下保持鲁棒性。

Hierarchical Topology-Aware Planning and Control of Underwater Vehicle-Manipulator Systems in Confined Environments

在受限环境中对水下载具-机械臂系统的层级拓扑感知规划与控制

• Authors: Mohamed Abdelwahab, Ruggero Carli, Damiano Varagnolo, Alberto Dalla Libera
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.08871
• Pdf link: https://arxiv.org/pdf/2608.08871
• Abstract This paper addresses autonomous intervention with an underwater vehicle--manipulator system (UVMS) in confined, cluttered, and partially known environments, where poor maneuverability, narrow passages, and uncertain execution may cause the robot to enter unrecoverable regions. We propose MANTA, a three-layer hierarchical planning-and-control framework that couples passage accessibility, manipulation feasibility, and closed-loop execution. The first layer performs global connectivity reasoning in a conservative reduced base space to extract traversable corridor candidates toward the task region. The second layer refines each candidate corridor by jointly optimizing the continuous base motion and arm trajectory, producing a collision-free base--arm trajectory. The third layer learns a reach-and-hold base policy using Gaussian-process model-based reinforcement learning (MBRL) through MC-PILCO, enabling trajectory tracking and station keeping at the planned manipulation state. During execution, the framework monitors map updates and can trigger recovery and route repair when the active passage becomes infeasible. MANTA is evaluated in confined UVMS planning and closed-loop tracking experiments. Across 120 matched planning queries, it achieves higher task success than full-state sampling-based baselines while producing larger clearance margins and lower arm motion. The learned MC-PILCO policy further reduces position and yaw tracking errors on both training and unseen tube-like references. These results show MANTA as a structured and data-efficient framework for safe autonomous underwater intervention in caves, tubes, and cluttered subsea structures.
• 中文摘要 本文探讨了在狭窄、杂乱且部分已知环境中，利用水下机械臂系统（UVMS）进行自主干预，在这些环境中，机动性差、通道狭窄和执行不确定可能导致机器人进入无法恢复的区域。我们提出了MANTA，这是一个三层分层的规划与控制框架，结合通道可及性、操作可行性和闭环执行。第一层在保守的约化基空间中进行全局连通性推理，提取可通行走廊候选路径，朝向任务区域。第二层通过联合优化连续的基底运动和臂轨迹，优化每个候选走廊，生成无碰撞的基底-臂轨迹。第三层通过MC-PILCO采用高斯过程模型强化学习（MBRL）学习达程并保持基线策略，实现轨迹跟踪和在计划操作状态下的定位。执行过程中，框架监控地图更新，并在活动通道变得不可行时触发恢复和路线修复。MANTA在受限UVMS规划和闭环跟踪实验中进行评估。通过120次匹配规划查询，MANTA在实现比全状态采样基线更高的任务成功率，同时实现更大的净空余裕和下臂运动。所学的MC-PILCO策略进一步减少了训练和未见管状参考点上的位置和偏航跟踪误差。这些结果表明MANTA是一个结构化且数据高效的框架，用于洞穴、管道和杂乱海底结构中安全自主水下干预。

DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

DistillCache：KL引导自适应KV-缓存驱逐，实现内存高效LLM推断

• Authors: Asaad Althoubi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Performance (cs.PF)
• Arxiv link: https://arxiv.org/abs/2608.08878
• Pdf link: https://arxiv.org/pdf/2608.08878
• Abstract Transformer-based large language models (LLMs) achieve strong performance across many tasks, but their Key-Value (KV) cache grows linearly with sequence length, creating a severe memory bottleneck for long-context inference. Existing heuristic eviction methods (e.g., H$_2$O and SnapKV) rely on static attention or positional signals that often fail to capture a token's future predictive influence. We propose DistillCache, a reinforcement learning framework that formulates KV-cache eviction as a sequential decision problem. DistillCache learns a lightweight policy network using rich internal model signals (attention statistics, value norms, entropy, and position) and trains it with REINFORCE via a per-step KL-divergence reward to preserve the full-cache output distribution. On a 7B-parameter instruction-tuned Transformer (Mistral-7B-Instruct-v0.3), DistillCache retains 94.2% of full-cache accuracy on LongBench at a 25% cache budget, outperforming both strong heuristic baselines (H$_2$O, SnapKV) by up to 2.7 absolute points and, under our re-implementations, concurrent RL-based methods (ForesightKV, RLKV) by up to 1.4 points on long-context tasks. On reasoning benchmarks, DistillCache is competitive with the best concurrent method and surpasses it under aggressive compression. It also delivers up to 2.1x full-cache throughput while maintaining competitive practical efficiency. These results highlight the effectiveness of learned, distribution-aware policies for memory-efficient long-context LLM inference.
• 中文摘要 基于Transformer的大型语言模型（LLM）在许多任务中都能实现强劲性能，但其键值（KV）缓存随序列长度线性增长，导致长上下文推断中严重的内存瓶颈。现有的启发式驱逐方法（如H$_2$O和SnapKV）依赖静态注意力或位置信号，常常无法捕捉代币的未来预测影响。我们提出了DistillCache，一种强化学习框架，将KV缓存驱逐表述为顺序决策问题。DistillCache通过丰富的内部模型信号（注意力统计、值规范、熵和位置）学习一个轻量级策略网络，并通过REINFORCE通过每步KL发散奖励训练，以保持完整缓存输出分布。在7B参数指令调优的Transformer（Mistral-7B-Instruct-v0.3）上，DistillCache在LongBench上以25%的缓存预算保持94.2%的全缓存准确率，在长时间上下文任务中，比强启发式基线（H$_2$O、SnapKV）高出最多2.7个绝对点，在我们重新实现的过程中，基于强RL的方法（ForesightKV、RLKV）则高出最多1.4个点。在推理基准测试中，DistillCache与最佳并发方法竞争，并在激进压缩下超越。它还能提供高达2.1倍的全缓存吞吐量，同时保持竞争性的实用效率。这些结果凸显了学习式、分布感知策略在内存高效长上下文LLM推断中的有效性。

LLM Reasoning for Subjective Tasks: Failure Modes, Mitigation, and Dynamic Reasoning Routing

主观任务的大型语言模型推理：失效模式、缓解措施与动态推理路由

• Authors: Juncheng Dong, Ding Tong, Ishan Gupta, Yuyan Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08889
• Pdf link: https://arxiv.org/pdf/2608.08889
• Abstract Recommendation systems thrive on personalization, where ''correctness'' is rarely a binary truth but a matter of subjective human preference. As Large Language Models (LLMs) are deployed as autonomous verifiers of safety and quality guidelines, they face a distinctive challenge: context-aware preference alignment. Recent gains in Reinforcement Learning with Verifiable Rewards (RLVR) are indexed mostly on objective, mathematical tasks. Through a large-scale study spanning both proprietary and open-source models on four real-world verification tasks from a production recommender platform, we ask whether explicit reasoning generalizes to subjective, human-centric industry rubrics. We expose a fundamental vulnerability: rigid, math-centric reasoning traces actively degrade verification, and applying standard RLVR triggers a phenomenon we term reasoning collapse, in which the policy abandons deliberation in favor of rapid heuristic guessing. We introduce a conditional length-penalized post-training algorithm that intertwines verification accuracy with bounded reasoning length, halting collapse and recovering performance. Finally, we show that a reasoning trace's efficacy is tightly coupled with its socio-linguistic framing: across 1500 synthesized personas, verification accuracy swings by nearly 0.38 macro-F1 depending solely on the adopted reasoning persona---evidence that much subjective-verification error is really reasoning-style mismatch. This observation motivates a mid-training architecture that routes reasoning through contextually aligned personas. This work offers both a scalable algorithmic patch and a long-term architectural blueprint for aligning reasoning models with real-world subjective constraints.
• 中文摘要 推荐系统依赖个性化，“正确性”很少是二元真理，而是主观人类偏好的问题。随着大型语言模型（LLMs）作为安全和质量指南的自主验证工具，它们面临一个独特挑战：情境感知偏好对齐。可验证奖励强化学习（RLVR）的最新进展主要基于客观的数学任务。通过一项涵盖专有和开源模型的大型研究，涵盖四个真实世界验证任务，我们探讨显式推理是否能推广到主观、以人为中心的行业评分标准。我们暴露了一个根本性漏洞：僵化、以数学为中心的推理轨迹会主动降低验证能力，而应用标准RLVR会触发我们称之为推理崩溃的现象，即该策略放弃审议，转而采用快速启发式猜测。我们引入了一种条件长度惩罚的训练后算法，将验证准确性与有界推理长度交织在一起，阻止崩溃并恢复性能。最后，我们证明推理痕迹的有效性与其社会语言框架紧密相关：在1500个合成人格中，验证准确率仅依赖于所采用的推理人格，其波动近0.38个宏F1---证据表明，许多主观验证误差实际上是推理风格的不匹配。这一观察促使采用了一种中期训练架构，将推理路由上下文对齐的人格进行。这项工作既提供了可扩展的算法补丁，也为推理模型与现实主观约束的长期架构蓝图提供了对齐。

Reading is not Reasoning: Bridging the Agentic Policy Gap in Vision-Text Compression

阅读不是推理：弥合视觉-文本压缩中的能动政策鸿沟

• Authors: Cheng Fan, Junyi Zhou, Tingzhang Luo, RongJian Xu, Qiyanhui Lu, Mingjian Zhu, Hanting Chen, Jianyuan Guo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.08960
• Pdf link: https://arxiv.org/pdf/2608.08960
• Abstract Multi-step language-model agents repeatedly process growing interaction histories, leading to substantial context costs. Vision--text compression reduces these costs by rendering history as images, but the resulting modality shift creates a marked capability gap. Through controlled evaluations of history recovery, matched-state decisions, and complete trajectories, we show that this gap cannot be explained by OCR quality alone. Visual-history agents exhibit systematic drift in action selection, query formulation, stopping, and evidence use, revealing an agentic policy gap. We introduce \textbf{CAPS}, a two-stage \textbf{C}ross-modal \textbf{A}gentic \textbf{P}olicy \textbf{S}elf-distillation framework that uses the same model's stronger text-history policy to supervise its visual-history counterpart. Offline trajectory self-distillation transfers successful text-policy behavior to visual-history inputs, while online policy self-distillation provides dense supervision on states visited by the visual-history policy during reinforcement learning. On SearchQA, CAPS improves over AgentOCR by 5.0\% and 3.4\% with 3B and 7B backbones, respectively. On full-history ALFWorld, the corresponding gains are 15.6\% and 14.5\%. Across settings, CAPS reduces average memory-context cost by up to 63.3\% and peak cost by up to 83.4\% relative to matched text-history policies. These results show that explicit cross-modal policy self-distillation can preserve agent capability under vision--text compression. Our code will be made publicly available in a future release.
• 中文摘要 多步语言模型代理反复处理不断增长的交互历史，导致显著的上下文成本。视觉-文本压缩通过将历史渲染为图像来降低这些成本，但由此产生的模态转变造成了明显的能力差距。通过对历史恢复、匹配状态决策和完整轨迹的受控评估，我们表明这一差距无法仅靠OCR质量解释。可视化历史代理在动作选择、查询表述、停止和证据使用上表现出系统漂移，揭示了代理策略的缺口。我们引入了\textbf{CAPS}，一个两阶段的\textbf{C}ross模态\textbf{A}生成体\textbf{P}olicy \textbf{S}elf蒸馏框架，使用同一模型更强的文本历史策略来监督其视觉历史对应的对应物。离线轨迹自蒸馏将成功的文本策略行为转移到可视化历史输入，而在线策略自蒸馏则在强化学习期间对可视化历史策略访问的状态提供密集监督。在SearchQA中，CAPS分别在3B和7B骨干链下比AgentOCR提升5.0%和3.4%。在全历史ALFWorld上，相应提升分别为15.6%和14.5%。在不同设置下，CAPS相较匹配的文本历史策略，平均内存上下文成本降低了最多63.3%，峰值成本降低了高达83.4%。这些结果表明，显式跨模态策略自蒸馏可以在视觉-文本压缩下保持代理能力。我们的代码将在未来版本中公开。

Motif 3: Technical Report

主题三：技术报告

• Authors: Junghwan Lim, Joon Son Chung, Sungmin Lee, Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Dongpin Oh, Hyeyeon Cho, Dahye Choi, Jaeheui Her, Hanbin Jung, Changjin Kang, Minjae Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Yeongjae Park, Bokki Ryu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09119
• Pdf link: https://arxiv.org/pdf/2608.09119
• Abstract We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.
• 中文摘要 我们引入了Motif 3，一个仅由解码器支持的专家混合语言模型，拥有3140亿总参数，每个代币激活参数132亿。每个稀疏MoE层包含384个路由专家，每个标记选中8个。这种细粒度稀疏性在限制计算的同时，提供了显著的专家容量。Motif 3基于分组差分潜在注意力（GDLA）构建，该模型将分组差分注意力与多头潜在注意力的压缩键值表示相结合。该架构进一步整合了修改流形约束的超连接、专家专用多范数激活和多标记预测，以提升优化稳定性、专家专精和推理效率。我们在约12.5万亿个代币上预训练Motif 3，涵盖网页文档、STEM、代码、数学、多语言内容和领域专用语料库。专家平衡和数值稳定技术支持大规模稳定训练，而选择性MXFP8计算与通信、内存高效融合核和窗口感知上下文并行性支持最高256K令牌的训练。我们的培训后流程结合了通用监督微调、六位接受强化学习培训的专业教师、一位接受监督微调培训的软件工程教师，以及多教师的策略提炼。最终的统一模型整合了推理、编码、工具使用、专业工作、长上下文理解、校准隐忘和指令遵循等互补能力。在广泛的评估套件中，Motif 3 在领先开放权重模型中展现出竞争力，包括在长期能动任务、数学推理、科学知识和幻觉敏感评估方面的强劲表现。

RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

RISE-RL：基于评分标准的开放式强化学习选择性探索

• Authors: Jinkun Hou, Zhuo Liu, Huimin Ren, Hongsheng Xin, Pan Zhou, Kun Zhan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09123
• Pdf link: https://arxiv.org/pdf/2608.09123
• Abstract Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning (RL) methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration. We propose $\textbf{RISE-RL}$ (Rubric-Informed Selective Exploration), which uses repeatedly missed rubric criteria to elicit privileged trajectories that are difficult to discover through unguided exploration alone. RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and then re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective and removed once its additional benefit diminishes. Experiments with 4B and 14B models across writing, chat, health, and science show that RISE-RL achieves the highest mean score on every evaluated benchmark under guidance-free evaluation. Compared with standard Rubric-RL, it improves the average score by 1.3 points at the 4B scale and $\textbf{3.3 points at the 14B scale}$, including a $\textbf{6.0-point}$ gain on CreativeWriting-V3. It also improves creative-writing diversity and yields gains on objectively scored medical and scientific benchmarks. These results indicate that selective internalization through reward filtering and policy support shaping is effective for open-ended reinforcement learning.
• 中文摘要 将大型语言模型（LLMs）与开放式任务对齐具有挑战性，因为响应必须满足多维标准，而不能遵循单一正确的生成轨迹。现有基于评分标准的强化学习（RL）方法将细粒度的标准级反馈压缩为标量奖励，使得在有限的政策探索下难以针对持续能力缺口。我们提出了$\textbf{RISE-RL}$（评分标准知情选择性探索），利用反复错过的评分标准，诱导出难以通过无指导探索发现的特权轨迹。RISE-RL仅保留完全评分标准奖励高于自然推广平均奖励的轨迹，然后在原始提示下重新评估，以强调那些自然策略支持较弱的行为。所得的指导信号通过独立的辅助目标进行优化，并在其额外益处减弱时移除。在写作、聊天、健康和科学领域对4B和14B模型的实验显示，RISE-RL在无指导评估下，在所有评估基准中均得分最高。与标准评分标准RL相比，RISE-RL在4B量表上平均得分提升了1.3分，在14B量表上提高了$\textbf{3.3分}$，包括在CreativeWriting-V3上增加了$\textbf{6.0分}$。它还提升了创意写作的多样性，并在客观评分的医学和科学基准中获得收益。这些结果表明，通过奖励过滤和政策支持塑造进行选择性内化，对开放式强化学习非常有效。

SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learning

SpeedTuning：用轻量级强化学习加速策略执行

• Authors: David D. Yuan, Tony Z. Zhao, Kaylee Burns, Chelsea Finn
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09138
• Pdf link: https://arxiv.org/pdf/2608.09138
• Abstract While learned robotic policies hold promise for advancing generalizable manipulation, their practical deployment is often hindered by suboptimal execution speeds. Imitation learning policies are inherently limited by hardware constraints and the speed of the operator during data collection. In addition, there are no established methods for accelerating policies learned via imitation, and the empirical relationship between execution speed and task success remains underexplored. To address these issues, we introduce SpeedTuning, a reinforcement learning framework specifically designed to enhance the speed of manipulation policies. SpeedTuning learns to predict the optimal execution speed for actions, thereby complementing a base policy without necessitating additional data collection. We provide empirical evidence that SpeedTuning achieves substantial improvements in execution speed, exceeding 2.4x speed-up, while preserving an adequate success rate compared to both the original task policy and straightforward speed-up methods such as linear interpolation at a fixed speed. We evaluate our approach across a diverse set of dynamic and precise tasks, including pouring, throwing, and picking, demonstrating its effectiveness and robustness in enhancing real-world robotic manipulation.
• 中文摘要 虽然学习的机器人策略有望推动可推广的操作，但其实际应用常常受执行速度不足的阻碍。模仿学习策略本质上受硬件限制和数据采集时操作员的速度限制。此外，目前尚无加速通过模仿学习策略的方法，执行速度与任务成功率之间的实证关系仍未被充分探讨。为解决这些问题，我们引入了SpeedTuning，一个专门设计用于提升操作策略速度的强化学习框架。SpeedTuning学习预测动作的最佳执行速度，从而补充基础策略而无需额外数据收集。我们提供了实证证据，表明SpeedTuning在执行速度上实现了显著提升，加速超过2.4倍，同时保持了与原始任务策略及线性插值等简单加速方法相比的足够成功率。我们在多样的动态和精确任务中评估我们的方法，包括倒酒、抛掷和采摘，展示了其在增强现实机器人操作能力方面的有效性和稳健性。

CodecArena: Codec Quality Assessment via Visual Reinforcement Learning

CodecArena：通过视觉强化学习进行编码器质量评估

• Authors: Jiaye Fu, Weiqi Li, Qiankun Gao, Yanchen Zhao, Xiandong Meng, Jian Zhang, Siwei Ma, Jiaqi Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.09139
• Pdf link: https://arxiv.org/pdf/2608.09139
• Abstract Video coding is advancing into the low and ultra-low bitrate regime, driven by end-to-end codecs that replace the hand-crafted pipeline with jointly optimized neural networks and generative codecs that exploit the priors of video generation models. Yet the dominant metrics, LPIPS and DISTS, measure feature and texture similarity rather than content fidelity: a reconstruction that hallucinates a wrong face or blurs text into convincing strokes can still score well, even when a human rejects it instantly. To address this, we propose CodecArena, the first vision-language framework for video coding quality assessment, casting codec evaluation as source-conditioned comparative reasoning between a reference and its reconstructions. We optimize CodecArena with Facet-GRPO, a visual reinforcement learning scheme that aligns pairwise codec preferences while grounding the verdict in five fidelity facets: identity, objects, text, texture, and temporal consistency. Its facet-anchored reward uses automatically derived facet directions as weak anchors, rather than human per-facet labels, to prevent any single sub-score from dominating the holistic preference and to yield interpretable fine-grained quality judgments. To support training and evaluation in this underexplored regime, we construct two complementary resources: CodecArena-1K, a fully automatic preference dataset of 1,500 comparison groups built from traditional, neural, and generative codec reconstructions with fused vision-language and objective supervision; and CodecArena-Bench, a human-ranked benchmark with source-disjoint videos for fair out-of-domain evaluation. Extensive experiments demonstrate that CodecArena achieves state-of-the-art agreement with human judgments on source-disjoint content across diverse codecs and bitrates, surpassing perceptual metrics and prior vision-language evaluators.
• 中文摘要 视频编码正逐步进入低和超低码率领域，这由端到端编解码器驱动，这些编解码器用联合优化的神经网络和生成编解码器取代了手工制作的流程，利用视频生成模型的先验优势。然而，主流指标LPIPS和DISTS衡量的是特征和纹理的相似度，而非内容的真实度：即使人工立即否定，也会有高分。为此，我们提出了CodecArena，首个视频编码质量评估的视觉语言框架，将编解码器评估定位为参考与其重建之间的源条件比较推理。我们用Facet-GRPO优化CodecArena，这是一种视觉强化学习方案，能够对齐成对编解码器偏好，同时将结论基于五个保真度方面：身份、对象、文本、纹理和时间一致性。其面向锚定奖励使用自动生成的面向方向作为弱锚点，而非人类的每个面向标签，以防止任何单一子评分主导整体偏好，并产生可解释的细粒度质量判断。为支持这一未被充分探索的环境中的培训和评估，我们构建了两个互补资源：CodecArena-1K，一个全自动偏好数据集，包含1500个对照组，结合传统、神经和生成编解码器重建，结合视觉语言和客观监督;以及CodecArena-Bench，一个基于人类排名的基准测试，包含源与不相交的视频，用于公平的域外评估。大量实验表明，CodecArena在多种编解码器和码率上实现了与人类对源不相交内容判断的最先进一致性，超越了感知指标和以往视觉语言评估器。

An Agentic Generative Large Language Model for Treatment Planning of Colorectal Cancer

一种用于结直肠癌治疗计划的代理生成大型语言模型

• Authors: Mengxian Lyu, Cheng Peng, Tim Jang, Ang Li, Mengyuan Zhang, Ziyi Chen, Leighton Elliott, Tianshi Liu, Lidice Galindo, Chiranjeevi Sainatham, Oscar F. Borja-Montes, Kaleb E. Smith, Ying Zhang, Lichao Sun, Jiang Bian, Gloria Lipori, Duane A. Mitchell, Elizabeth A. Shenkman, Yi Guo, Thomas J. George, Yonghui Wu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.09142
• Pdf link: https://arxiv.org/pdf/2608.09142
• Abstract Treatment planning in precision oncology requires synthesizing heterogeneous patient information with rapidly evolving clinical guidelines to ensure guideline-concordant care. While large language models (LLMs) show promise in many diagnostic tasks, their adoption for high-stakes treatment planning is hindered by complex reasoning, adherence to timely clinical guidelines, and safety concerns. In this study, we present GatorOnco, an agentic LLM for colorectal cancer (CRC) treatment planning. GatorOnco is developed using a total of 282 billion tokens of biomedical text, including healthcare system-scale clinical text comprising 166 billion tokens from UF Health. We implemented a domain-adaptation method that integrates pre-training, model merging, a two-stage post-training approach, and agent-based reinforcement learning. An agentic retrieval-augmented generation (RAG) approach dynamically integrates time-sensitive clinical guidelines into the reasoning process. In a blind, randomized clinical evaluation conducted by five UF Health oncologists, GatorOnco significantly outperformed open-source LLMs (P < 0.01) and achieved expert-level performance comparable to UF Health oncologists. Compared with expert oncologists, GatorOnco received significantly higher ratings for readability (4.46 vs. 4.19, P < 0.01) and completeness (3.91 vs. 3.52, P < 0.01), while showing statistically comparable performance in correctness (4.09 vs. 4.11, P = 0.921), currency (4.04 vs. 3.98, P = 0.478), and safety (4.22 vs. 4.22, P = 0.999). These findings demonstrate that integrating agentic reasoning with large-scale domain adaptation can help bridge the gap for generative AI in high-stakes cancer treatment planning.
• 中文摘要 精准肿瘤学中的治疗计划需要综合异质患者信息与快速演变的临床指南，以确保护理符合指南。虽然大型语言模型（LLMs）在许多诊断任务中展现出潜力，但其在高风险治疗计划中的应用受到复杂推理、及时临床指南的遵守以及安全性问题的阻碍。本研究介绍了GatorOnco，一种用于结直肠癌（CRC）治疗计划的代理型LLM。GatorOnco开发时总计有2820亿个生物医学文本标记，其中包括来自UF Health的1660亿个医疗系统规模临床文本。我们实施了一种域适应方法，整合了预训练、模型合并、两阶段后训练方法和基于主体的强化学习。一种代理检索增强生成（RAG）方法动态将时间敏感的临床指南集成到推理过程中。在五位UF Health肿瘤科医生进行的盲目随机临床评估中，GatorOnco显著优于开源LLM（P < 0.）。 01）并实现了与UF Health肿瘤科医生相当的专家级表现。与专家肿瘤科医生相比，GatorOnco在可读性（4.46对4.19，P<0.01）和完整性（3.91对3.52，P<0.01）方面获得了显著更高的评分，同时在正确性（4.09对4.11，P=0.921）、有效性（4.04对3.98，P=0.478）和安全性（4.22对4.22，P=0.999）方面表现出显著更高的评分。这些发现表明，将代理推理与大规模领域适应相结合，有助于弥合生成式人工智能在高风险癌症治疗计划中的差距。

How Roadside Units Enhance Intersection Safety? Cooperative Autonomous Driving System Design and A Proof of Concept

路边单元如何提升路口安全？协作自动驾驶系统设计与概念验证

• Authors: Taoyuan Yu, Kui Wang, Zongdian Li, Tao Yu, Walid Saad, Kei Sakaguchi
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.09144
• Pdf link: https://arxiv.org/pdf/2608.09144
• Abstract Intersections remain one of the most hazardous locations in urban road networks, where heterogeneous traffic participants and limited visibility frequently lead to severe traffic conflicts. In this paper, a vehicle-to-infrastructure-to-vehicle (V2I2V) cooperative system is proposed for improving road safety and traffic efficiency by using digital twins (DTs) deployed on roadside units (RSUs) to eliminate blind spots and centrally coordinate connected and automated vehicles (CAVs) in smart intersections. The proposed system integrates cloud-based global DTs for macroscopic guidance and RSU-based local DTs for real-time operations. Within this architecture, a hierarchical reinforcement learning (HRL) framework combines offline pre-training with online fine-tuning to achieve robust cooperative control. Experimental results show that the proposed system achieves substantial improvements in safety and efficiency in simulation experiments and real-world proof-of-concept (PoC) trials. In simulations, our system ensures high safety, efficiency, and smoothness under realistic communications and traffic constraints. In PoC trials, the RSU-centric control loop achieves a decision-making latency of approximately 42 ms and maintains a safe stopping distance of 8.5 m for pedestrians, while also shortening stop duration and overall traversal time. These results indicate that the proposed system provides robust and scalable performance at smart intersections.
• 中文摘要 路口仍是城市道路网络中最危险的地点之一，交通混杂且能见度有限，常导致严重的交通冲突。本文提出了一种车辆到基础设施再到车辆（V2I2V）合作系统，通过在路边单元（RSUs）上部署数字孪生（DT）消除盲区，集中协调智能路口的互联和自动驾驶车辆（CAV）来提升道路安全和交通效率。该系统集成了基于云的全球DT用于宏观引导，基于RSU的本地DT用于实时操作。在该架构中，一个层级强化学习（HRL）框架结合了离线预训练与在线微调，实现了稳健的协作控制。实验结果显示，该系统在模拟实验和实际概念验证（PoC）试验中实现了安全性和效率的显著提升。在模拟中，我们的系统在真实的通信和交通约束下确保了高安全性、效率和顺畅性。在PoC试验中，以RSU为中心的控制环路实现了约42毫秒的决策延迟，并为行人保持了8.5米的安全停车距离，同时缩短了停车时间和整体通行时间。这些结果表明，所提系统在智能路口提供了稳健且可扩展的性能。

SparsePilot: Belief-Guided Network Planning under Sparse Wireless Measurements

SparsePilot：基于稀疏无线测量的信念引导网络规划

• Authors: Xuanhao Luo, Jiayuan Huang, Longyu Zhou, Mingzhe Chen, Yuchen Liu
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.09199
• Pdf link: https://arxiv.org/pdf/2608.09199
• Abstract Unmanned aerial vehicles (UAVs) have emerged as a promising solution for on-demand wireless coverage planning in urban environments. Existing learning-based UAV control methods, however, typically rely on continuous access to dense user-level received signal strength (RSS) measurements. Such full-observation assumptions are difficult to satisfy in real-world deployments due to the high cost and limited availability of dense wireless feedback. Sparse-feedback decision making under severe observation constraints therefore represents a fundamental challenge. To fill this gap, we propose SparsePilot, a measurement-efficient sensing-control framework that couples active wireless probing with belief-guided network control. SparsePilot formulates spatial probing as a multi-armed bandit problem over grid cells, uses upper confidence bound probing to select informative regions, and aggregates sparse RSS measurements into a coverage belief map. A deep reinforcement learning controller then uses this belief state to generate continuous UAV mobility actions, while the full wireless state remains hidden from the policy. We further provide a theoretical analysis connecting sparse probing, belief estimation error, and the sparse-feedback performance gap. Experiments across seven urban digital twins show that SparsePilot achieves superior coverage restoration performance while using only about 3.1% of the full-observation measurement budget and demonstrates strong cross-scene generalization to unseen urban-scale wireless environments.
• 中文摘要 无人机（UAV）已成为城市环境中按需无线覆盖规划的有前景解决方案。然而，现有基于学习的无人机控制方法通常依赖于持续访问高密度用户级接收信号强度（RSS）测量数据。由于高成本且密集无线反馈有限，这种全观测假设在实际部署中难以满足。因此，在严苛观测约束下做出稀疏反馈决策是一个根本性的挑战。为填补这一空白，我们提出了SparsePilot，一种将主动无线探测与信念引导网络控制相结合的高效测量感测控制框架。SparsePilot将空间探测构建为网格单元的多臂盗垒问题，利用上置信界探测选择信息区，并将稀疏RSS测量汇聚为覆盖信念图。深度强化学习控制器随后利用该信念状态生成连续的无人机移动动作，而完整的无线状态则对策略保持隐藏。我们还进一步提供了理论分析，将稀疏探测、信念估计误差和稀疏反馈性能差距联系起来。跨越七个城市数字孪生的实验表明，SparsePilot在仅使用约3.1%的全观测测量预算的情况下，实现了更优越的覆盖恢复性能，并展现出对未见城市尺度无线环境的强力跨场景泛化能力。

Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

超越可解决性：任务可学习性作为LLM RL培训后静态先验

• Authors: Ting Zhou, Zhenqing Ling, Daoyuan Chen, Qianli Shen, Yilun Huang, Ying Shen, Yaliang Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09217
• Pdf link: https://arxiv.org/pdf/2608.09217
• Abstract Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models, yet uniform task sampling allocates compute without regard to differences in how tasks respond to optimization. Existing task-valuation methods mostly rely on snapshot-based signals such as current pass rate or reward, which estimate how solvable a task is under the current policy. However, tasks with similar current solvability can still differ substantially in how positively they respond to further training. We study this residual axis as task learnability: a regime-conditional measure of expected positive response to continued training under a fixed RL post-training regime. By analyzing per-task reward trajectories, we find that learnability is reproducible across independently sampled training contexts and predictive of downstream utility. To make this signal practical before training begins, we propose TrajVal, a lightweight probe-based estimator that approximates per-task learnability from a short probe run and two endpoint evaluations. TrajVal can be used either as a standalone static prior for task sampling or as a multiplicative prior for existing online schedulers. Experiments on mathematical and logical reasoning benchmarks across multiple model scales show that TrajVal improves data efficiency over uniform sampling and provides complementary gains when combined with online scheduling methods.
• 中文摘要 强化学习（RL）已成为在大型语言模型中引发推理能力的核心训练后范式，但统一任务抽样则不考虑任务对优化响应的差异。现有的任务估值方法主要依赖基于快照的信号，如当前通过率或奖励，以估计任务在当前策略下的可解性。然而，当前可解性相似的任务在对进一步训练的积极响应上仍可能有显著差异。我们将该残余轴研究为任务可学习性：一种在固定强化学习后训练体系下，对持续训练预期积极响应的制度条件度量。通过分析每个任务的奖励轨迹，我们发现可学习性在独立抽样的训练环境中可重复，并能预测后续效用。为了在训练开始前使该信号实用，我们提出了TrajVal，一种轻量级的探针估计器，通过短时间探测运行和两次端点评估近似各任务的可学习性。TrajVal既可作为任务采样的独立静态先验，也可以作为现有在线调度器的乘法先验。在多个模型尺度上的数学和逻辑推理基准测试实验表明，TrajVal在与在线调度方法结合时，能提升数据效率，且在与在线调度方法结合时，能带来互补的收益。

DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Models

DreOPD：流匹配模型的退化参考推断策略蒸馏

• Authors: Mingfeng Lin, Chengfei Cai, Lin Xu, Yuxiang Wei, Liang Han
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.09233
• Pdf link: https://arxiv.org/pdf/2608.09233
• Abstract Flow-matching models are now a mainstream method to image generation, but its adaptation to diverse downstream scenarios typically relies on post-training, which may cause conflicts among task-specific optimization objectives. Reinforcement learning enables direct optimization of task-specific rewards beyond the original models, yet trajectory-level optimization may incur high-variance gradients and cross-task interference. On-policy distillation (OPD) offers dense and stable supervision on student rollouts, but conventional teacher matching remains imitation-based. We propose DreOPD, a Degraded-reference extrapolative OPD method for flow-matching models that bridges these two paradigms. Our DreOPD converts implicit reward extrapolation into closed-form velocity regression, enabling extrapolative post-training with the stability of OPD. It further uses a mildly degraded reference to strengthen the teacher-reference contrast, yielding a clearer extrapolation direction. Experiments on single- and multi-teacher settings show that DreOPD outperforms OPD and multi-task RL baselines in average performance, while surpassing specialized teachers on most metrics.
• 中文摘要 流匹配模型现已成为图像生成的主流方法，但其适应多样化下游场景通常依赖于后期训练，这可能导致任务特定优化目标之间的冲突。强化学习允许对任务特定奖励的直接优化，超越原始模型，但轨迹级优化可能引发高方差梯度和跨任务干扰。策略上提炼（OPD）为学生推广提供密集且稳定的监督，但传统的教师匹配仍基于模仿。我们提出了DreOPD，一种用于流匹配模型的退化参考外推OPD方法，连接了这两种范式。我们的DreOPD将隐式奖励外推转换为闭式速度回归，支持在OPD稳定性下的训练后推算。它还利用轻度退化的参考强化教师-参考对比，提供更清晰的外推方向。单师和多教师环境的实验显示，DreOPD在平均表现上优于OPD和多任务强化学习基线，且在大多数指标上超过专业教师。

Task-Oriented Formation Decision via Reinforcement Learning: Herding an Attacking Swarm

通过强化学习实现任务导向编队决策：驱赶攻击群体

• Authors: Zhaozong Wang, Guibin Sun, Jinyong Chen, Rui Zhou
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.09258
• Pdf link: https://arxiv.org/pdf/2608.09258
• Abstract Multi-robot systems can accomplish tasks that are difficult for a single robot by organizing into task-specific formations. Different from existing studies on multi-robot shape formation, we here study the task-oriented formation decision problem, with a focus on the herding task. This task is challenging due to the attackers' superior maneuverability and their unknown strategies. To address these challenges, we propose the following novel results. First, we encode the formation shape using a low-dimensional parameter vector. This parametric representation reformulates the formation decision as a parameter optimization problem, thereby resolving the limited flexibility of predefined shapes. By optimizing these formation parameters, the defenders' maneuverability disadvantage is mitigated through a formation shape that continuously adapts to task requirements. Second, we develop a reinforcement learning-based policy to regulate the formation parameters. Trained offline in simulations covering diverse attacking strategies, the learned policy can effectively handle adversarial unpredictability during online deployment. Comparative simulations against three baselines demonstrate that our method can successfully accomplish challenging herding tasks. Additional scalability simulations further verify its applicability to simulated scenarios involving dozens of robots. We also validate the practical feasibility of our method on a physical robotic platform with 3 attackers and 7 defenders.
• 中文摘要 多机器人系统可以通过组织任务特定的编队来完成单一机器人难以完成的任务。与现有多机器人形状形成研究不同，我们这里研究任务导向编队决策问题，重点关注放牧任务。由于攻击方机动性优越且策略未知，这一任务具有挑战性。为应对这些挑战，我们提出了以下新颖结果。首先，我们用低维参数矢量编码编队形态。该参数化表示将编队决策重新表述为参数优化问题，从而解决预定义形状的有限灵活性。通过优化这些编队参数，防御方机动性劣势得以缓解，编队形态能持续适应任务需求。其次，我们开发基于强化学习的策略来调控编队参数。通过离线训练，模拟涵盖多样化攻击策略，所学策略能够有效应对在线部署中的对抗性不可预测性。与三个基线的对比模拟表明，我们的方法能够成功完成具有挑战性的牧羊任务。额外的可扩展性模拟进一步验证了其在涉及数十台机器人的模拟场景中的适用性。我们还验证了该方法在拥有3个攻击方和7名防御者的物理机器人平台上的实际可行性。

SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation

SoftmaxGRPO：学习使用 Softmax 优势群估计推理

• Authors: Jefferson Hernandez, Jaywon Koo, Zilin Xiao, Chen Wei, Vicente Ordonez
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09271
• Pdf link: https://arxiv.org/pdf/2608.09271
• Abstract Group-based reinforcement learning objectives such as GRPO can allocate learning signal poorly across prompt difficulty: under binary rewards, group normalization induces a divergent weighting on easy prompts. We introduce Softmax Advantage Group Estimation (SoftmaxGRPO), a drop-in alternative that replaces z-score-normalized group advantages with temperature-scaled softmax advantages, keeping weights bounded regardless of prompt difficulty. For binary rewards, we derive the exact finite-group population objective and identify MaxRL as its low-temperature limit. For bounded scalar rewards, we show that the large-group update exactly optimizes a log-moment-generating-function objective, while a universal finite-group scalar objective cannot exist without additional assumptions on the reward distribution. Empirically, SoftmaxGRPO reallocates measured gradient budget away from near-solved prompts and consistently improves over GRPO under identical rewards. It reaches 51.8% on DeepMath with verifiable rewards and improves a 1.5B instruction-tuned model from 35.0% to 68.0% on Poetry using only lightweight text-similarity rewards.
• 中文摘要 基于群体的强化学习目标如GRPO在提示难度下可能分配学习信号较差：在二元奖励下，组规范化会诱导简单提示的发散权重。我们引入Softmax优势群估计（SoftmaxGRPO），这是一种可替代的替代方案，用温度尺度软最大优势替代z分数归一化群体优势，无论提示难度如何权重均保持界限。对于二元奖励，我们精确推导有限组总体目标，并将MaxRL视为其低温极限。对于有界标量奖励，我们证明大组更新恰好优化了一个对数矩生成函数目标，而普遍有限组标量目标在没有额外奖励分布假设的情况下无法存在。经验上，SoftmaxGRPO将测量梯度预算从近解提示中重新分配，并在相同奖励条件下相较GRPO持续提升。在DeepMath上，它在可验证的奖励下达到了51.8%，并且仅用轻量级文本相似度奖励，将15亿指令调优模型从35.0%提升到68.0%。

CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

CoRE：通过均衡共识奖励测试时间强化学习

• Authors: Ambuj Mehrish, Sebastiano Vascon
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09324
• Pdf link: https://arxiv.org/pdf/2608.09324
• Abstract On unlabeled test data, reinforcement learning lacks a ground-truth reward; test-time RL methods derive one from the model's own roll-outs, rewarding those that match the majority vote over $N$ sampled answers. That vote discards a correct answer whenever it is a minority and scores every majority-matching roll-out identically. We replace it with \emph{CoRE} (Consensus Rewards via Equilibrium): the $N$ roll-outs form a graph whose edges combine answer agreement, reasoning similarity, and generation confidence, and replicator dynamics extract its dominant set, yielding a refined pseudo-label, a graded per-roll-out reward, and a per-question cohesiveness gate. CoRE strictly generalizes voting: majority voting is recovered as a special case; a block-value analysis gives a sharp threshold for when consensus recovers a correct minority against a larger wrong plurality; and confidence calibration provably lowers that threshold multiplicatively. Across seven backbones and five benchmarks (42 model--benchmark cells, three seeds each), \emph{CoRE} improves the untrained base by $+21.7$ points on average versus $+20.4$ for majority-vote TTRL, wins wherever agreement is contestable with margins over the vote of up to $+7.5$ points, and reaches the voting baseline's plateau accuracy in $54$--$70$\% fewer steps. Consensus, not counting: treating the roll-out group as a graph rather than a ballot box turns a brittle vote into a calibrated, graded, self-supervised reward at no extra roll-out cost.
• 中文摘要 在未标记的测试数据中，强化学习缺乏实地真实奖励;测试时强化学习方法从模型自身的展开中推导出一个，奖励那些在$N$样本答案中与多数票匹配的答案。该投票在正确答案为少数时丢弃，并且对每个多数匹配的滚动结果都进行完全一致的评分。我们用\emph{CoRE}（通过均衡实现共识奖励）替代它：$N$的展开形成一个图，其边结合了答案一致性、推理相似性和生成信心，复制者动态提取其主导集合，产生一个精细的伪标签、每个滚动的分级奖励和每个问题的凝聚门。CoRE严格推广投票：多数投票作为特例恢复;区块值分析给出了一个明确的阈值，说明共识在错误多数中恢复正确少数派时的阈值;置信度校准可证明以乘法降低该阈值。在七个骨干和五个基准点（42个模型基准单元，每个三个种子）中，\emph{CoRE} 平均提升未训练基数提升+21.7美元，而多数票TTRL为+20.4美元，在可争议的投票中胜出，投票差距最高达+7.5美元，且在54美元-70%%的步骤减少时达到投票基线的平台精度。共识，不计入：将推广组视为图表而非投票箱，将脆弱的投票转化为校准、分级、自我监督的奖励，且无额外推广成本。

Control-Oriented Scenario Tree Construction through Reinforcement Learning

通过强化学习构建控制型情景树

• Authors: Fabio Pavirani, Bert Claessens, Pierre Pinson, Chris Develder
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.09335
• Pdf link: https://arxiv.org/pdf/2608.09335
• Abstract Multistage stochastic model predictive control (MPC) handles uncertainty by optimizing over a scenario tree, a finite branching approximation of future outcomes constructed from sampled forecasts. To build such a tree, conventional methods focus on matching the underlying probability distribution---e.g., via Wasserstein-based scenario reduction---but improved distributional accuracy does not necessarily yield better control performance. We propose a control-oriented approach that learns scenario tree construction directly from its impact on downstream decisions. Fixing the tree topology, we formulate tree construction as a sequential assignment of sampled scenarios to leaves. This assignment is parameterized by an attention-based policy over the scenario set and trained using reinforcement learning, with closed-loop control profit as the objective. Training is stabilized by an asymmetric critic that leverages realized future trajectories. We evaluate the method on a risk-averse battery arbitrage problem. Across a range of forecast set sizes, the learned construction consistently achieves the highest profit, outperforming classical forward and backward reduction methods and certainty-equivalent (single-trajectory forecast) control. The learned policy also exhibits greater robustness on challenging instances, consistently demonstrating better tail-risk characteristics. Analysis of the resulting trees indicates that our method constructs compact, selectively branching structures that capture high-impact events while keeping most trajectories nearly deterministic. These findings highlight that the value of a scenario tree depends critically on the decisions it supports, and provide an effective framework to train scenario tree constructors merely based on the closed-loop control optimization signal.
• 中文摘要 多阶段随机模型预测控制（MPC）通过在场景树上优化不确定性，这是一种由抽样预测构建的未来结果的有限分支近似。为了构建此类树，传统方法侧重于匹配底层概率分布---例如通过基于Wasserstein的情景简化---但提高分布准确性并不一定带来更好的控制性能。我们提出了一种以控制为导向的方法，直接从场景树对后续决策的影响中学习。固定树状拓扑，我们将树构建表述为对采样场景的顺序分配。该赋值通过基于注意力的策略对场景集进行参数化，并通过强化学习进行训练，目标是实现闭环控制的盈利。训练通过一个利用实现的未来轨迹的非对称批评者来稳定。我们在风险厌恶电池套利问题上评估该方法。在不同预测集大小范围内，所学构建始终实现最高收益，优于经典的前向和后退还原方法及确定性等效（单轨迹预测）控制。所学策略在挑战性实例上表现出更强的鲁棒性，持续表现出更优的尾部风险特征。对所得树的分析表明，我们的方法构建了紧凑、选择性分支的结构，捕捉高影响事件，同时保持大多数轨迹接近确定性。这些发现表明，情景树的价值关键依赖于其支持的决策，并为仅基于闭环控制优化信号训练情景树构建器提供了有效框架。

Intent Speaks Louder: Controllable User Simulation Beyond Response Imitation

意图更响亮：可控用户模拟超越响应模仿

• Authors: Bo Wang, Ruixing Zhang, Yunqi Liu, Yang Zhang, Liangzhe Han, Tongyu Zhu, Leilei Sun
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.09420
• Pdf link: https://arxiv.org/pdf/2608.09420
• Abstract User simulators are widely used as scalable environments for training and evaluating interactive assistants. Generating the next user turn is inherently one-to-many: the same profile and dialogue context may support multiple plausible continuations with different local interaction intents. A fluent response may therefore advance the dialogue through an inappropriate intent, such as acceptance rather than repair. Our key insight is that controllable user simulation should separate which local interaction intent the next user turn should realize from how that intent is expressed in language. We introduce UserIDA (User Intent-Directive Alignment), which exposes interaction intent as an explicit per-turn directive. UserIDA defines a six-way intent interface, learns directive-conditioned generation through supervised fine-tuning, and uses intent-calibrated policy optimization during group-based reinforcement learning. The reward preserves composite response quality while ensuring that intent-violating candidates rank below compliant alternatives in mixed groups. On LMSYS-USP, UserIDA achieves 86.6\% intent accuracy, outperforming the strongest dedicated user-simulator baseline by 24.3 percentage points while improving semantic and stylistic similarity. In within-context interventions, it realizes at least four of the six target intents in 91.7\% of evaluated dialogue states, compared with 22.9\% for the strongest external baseline. These results establish per-turn intent control as a complementary dimension to response fidelity in user simulation.
• 中文摘要 用户模拟器被广泛用作可扩展环境，用于培训和评估交互式助手。生成下一个用户回合本质上是一对多的：相同的配置文件和对话上下文可能支持多个具有不同局部交互意图的合理延续。因此，流畅的回应可能通过不适当的意图推进对话，比如接受而非修复。我们的关键见解是，可控用户模拟应将下一轮用户应实现的本地交互意图与语言表达方式区分开来。我们引入了UserIDA（用户意图-指令对齐），它将交互意图作为显式的每回合指令公开。UserIDA定义了六向意图接口，通过监督微调学习指令条件生成，并在基于群体的强化学习中使用意图校准的策略优化。奖励保持了复合反应质量，同时确保违背意图的候选人在混合组中排名低于合规替代方案。在LMSYS-USP项目中，UserIDA实现了86.6%的意图准确率，比最强的专用用户模拟基线高出24.3个百分点，同时提升了语义和风格上的相似性。在上下文内干预中，它在91.7%的评估对话状态中实现了至少四个目标意图，而最强外部基线的22.9%实现了这一比例。这些结果确立了每回合意图控制作为用户模拟中响应忠实度的补充维度。

ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Models

ZetaGPT：定位--编码-自由状态-空间-注意力语言模型的参考实现

• Authors: Róisín Luo
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09432
• Pdf link: https://arxiv.org/pdf/2608.09432
• Abstract Transformer-based language models rely on self-attention, whose computation is permutation-equivariant and therefore lacks an intrinsic mechanism for representing token order. Existing architectures address this limitation by explicitly incorporating positional information through learned positional embeddings or hand-crafted positional encodings, such as rotary positional encoding (RoPE), treating positional information as an architecturally acquired capability rather than an inherent property of the model. Motivated by the pursuit of positional-encoding-free architectures, this work explores a language model architecture that integrates causal state-space equations to implicitly encode positional information before attention computation. Specifically, each model block applies a causal state-space equation before self-attention, allowing recurrent state dynamics to encode sequential information into token representations. Consequently, subsequent attention layers operate on position-aware representations without requiring explicit positional encodings while retaining the expressive modeling capacity of self-attention. We present \textsc{ZetaGPT}, a compact hybrid language model designed for research, rapid prototyping, algorithm verification, and educational applications. In addition to the proposed architecture, \textsc{ZetaGPT} provides a fully open-source, end-to-end training pipeline encompassing dataset construction, tokenizer training, pretraining, supervised fine-tuning, reinforcement learning from human feedback (RLHF), and chain-of-thought (CoT) reasoning via pure reinforcement learning. To the best of our knowledge, \textsc{ZetaGPT} is the first open-source small language model without explicit positional encoding and establishes a compact, reproducible reference implementation for the development and empirical study of positional-encoding-free language models.
• 中文摘要 基于Transformer的语言模型依赖自注意，其计算是置换等变的，因此缺乏表示标记顺序的内在机制。现有架构通过学习位置嵌入或手工设计的位置编码（如旋转位置编码，RoPE）明确纳入位置信息来解决这一限制，将位置信息视为架构获得的能力，而非模型固有属性。本研究旨在追求无位置编码的架构，探索一种语言模型架构，该架构集成因果状态空间方程，在注意力计算前隐式编码位置信息。具体来说，每个模型块在自注意前应用因果状态空间方程，使循环状态动态能够将顺序信息编码为标记表示。因此，后续的注意力层在不需显式位置编码的情况下，基于位置感知表示运行，同时保留了自我注意力的表达建模能力。我们介绍 \textsc{ZetaGPT}，一种紧凑的混合语言模型，设计用于研究、快速原型制作、算法验证和教育应用。除了所提架构外，\textsc{ZetaGPT} 还提供一个完全开源、端到端的训练流水线，涵盖数据集构建、分词器训练、预训练、监督微调、基于人类反馈的强化学习（RLHF）以及通过纯强化学习实现思维链（CoT）推理。据我们所知，\textsc{ZetaGPT} 是首个无显式位置编码的开源小语言模型，建立了一个紧凑且可重复的参考实现，用于无位置编码语言模型的开发和实证研究。

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

聆听、观察与跟踪：全模态语言模型的时空视听事件推理

• Authors: Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu, Di Zhang, Zihan Ma, Guodong Li, Zhou Yang, Yu Xiang, Yifei Zheng, Minnan Luo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09435
• Pdf link: https://arxiv.org/pdf/2608.09435
• Abstract Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83\% average semantic accuracy across the four levels, compared with 37.28\% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.
• 中文摘要 理解动态声源需要共同确定声音的产生方式、声源的位置以及声音随时间的传播方式。然而，现有的音频语言模型通常将片段表示为全局声学事件，而视觉语言模型缺乏定位和追踪单个声源所需的空间音频线索。为评估这一缺失能力，我们引入了ST-OmniQA，这是一个时空视听问答基准测试，由全景视频与同步一阶Ambisonics（FOA）移动声源音频结合构建。它包含40K视频和400K问答对，分为四个能力层级，涵盖声事件识别、到达方向、声源距离、运动轨迹以及基于时间的视听推理。基于该基准，我们提出了ST-Omni-R1，该系统将FOA衍生的语义和轨迹表示与全景视觉上下文整合，并通过渐进课程学习和推理树强化学习进行训练。ST-Omni-R1在四个层级的平均语义准确率为77.83%，而最佳评估基线为37.28%。三项公共空间音频基准结果进一步表明，其学习的空间和运动表征可超越ST-OmniQA。

Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control

学习调节，而非循环：软演员---评论家恢复逆变器式热泵控制

• Authors: Faizan Ahmed, Aniket Dixit, James Brusey
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09453
• Pdf link: https://arxiv.org/pdf/2608.09453
• Abstract On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.
• 中文摘要 开关循环是住宅热泵压缩机磨损的主要原因，但建筑中的强化学习（RL）控制器通常只优化能源成本和热舒适度，忽视所学策略的循环次数。我们在控制奖励中加入一个平准化的压缩机磨损项，并研究最终行为如何依赖于强化热泵算法。在对软演员---批判者（SAC）和近端策略优化（PPO）进行相同的BOPTEST最佳水热热泵案例的马尔可夫决策过程训练时，我们发现SAC学习了一个连续调制策略，使压缩机始终处于接入状态---这是逆变器驱动热泵的工作原理---实现每日零启动，而PPO则崩溃为“砰砰”控制，循环次数超过基线。在BOPTEST模拟器上，SAC政策将热不适降低最多90.7%，但成本增加了11.5%，同时取消了所有基线循环。

RecoverFly: A Failure-Aware Reinforcement Learning Post-Training Framework for Aerial Vision-Language Navigation

RecoverFly：一种针对空中视觉语言导航的失败感知强化后学习框架

• Authors: Boxiong Wang, Hui Kang, Geng Sun, Jiahui Li, Chao Yu, Daxin Tian
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09467
• Pdf link: https://arxiv.org/pdf/2608.09467
• Abstract Unmanned aerial vehicle vision-language navigation (UAV-VLN) requires agents to translate visual observations and language instructions into reliable flight actions in complex environments. Although recent end-to-end UAV vision-language-action (UAV-VLA) policies reduce reliance on separately designed perception, planning, and control modules, their behavior-cloning objectives provide limited corrective supervision for interactive closed-loop execution. Reinforcement learning (RL) offers a promising solution, while its effectiveness is constrained by inefficient use of samples, long-tailed scene distributions, and policy distribution shift during optimization. To this end, we propose RecoverFly, a failure-aware RL post-training framework for end-to-end UAV-VLA policies. Specifically, RecoverFly adapts token-level RL for stable optimization of grammar-constrained autoregressive UAV actions, revisits unresolved failure cases to strengthen corrective learning and sample utilization, and combines a two-stage long-tail scene curriculum with reference-policy regularization to improve scene adaptation while preserving acquired capabilities. Experiments on the TravelUAV benchmark demonstrate that RecoverFly achieves the best performance on the seen, unseen-map, and unseen-object splits. Moreover, compared to the AerialVLA initialization, RecoverFly improves success rate by 3.12 to 8.37 percentage points under a total rollout budget of about 30\% of the training-set size, validating its effectiveness, robustness, and generalization capabilities.
• 中文摘要 无人机视觉语言导航（UAV-VLN）要求智能体将视觉观察和语言指令转化为复杂环境中可靠的飞行动作。尽管近期端到端无人机视觉语言动作（UAV-VLA）策略减少了对单独设计的感知、规划和控制模块的依赖，但其行为克隆目标为交互式闭环执行提供了有限的纠正监督。强化学习（RL）提供了有前景的解决方案，但其效果受限于样本使用效率低、场景长尾分布和优化时策略分布偏移。为此，我们提出了RecoverFly，一种针对端到端无人机VLA策略的故障感知强化学习后培训框架。具体来说，RecoverFly 适配令牌级强化学习以稳定优化语法约束的自回归 UAV 动作，重新审视未解决的失败案例以强化纠正学习和样本利用，并将两阶段长尾场景课程与引用策略正则化结合，以提升场景适应性，同时保留已获得的能力。TravelUAV 基准测试的实验表明，RecoverFly 在可见、未可见地图和未可见对象分割上表现最佳。此外，与 AerialVLA 初始化相比，RecoverFly 在约 30% 训练集规模的总推广预算下，成功率提升了 3.12 至 8.37 个百分点，验证了其有效性、鲁棒性和泛化能力。

Learning Preference Adaptation for Large Language Model Personalization via Verbal Reinforcement Learning

通过言语强化学习实现大型语言模型个性化的学习偏好调整

• Authors: Yuting Liu, Wei Wu, Jianzhe Zhao, Guibing Guo
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09507
• Pdf link: https://arxiv.org/pdf/2608.09507
• Abstract Natural language user preferences provide an interpretable interface for LLM personalization. However, universal preference summaries often contain information irrelevant to a particular downstream task. Directly supplying the full preference summary therefore wastes context capacity and introduces cross-task distraction, while manually designing task-specific preference views is difficult to scale. In this work, we study \emph{task-specific preference adaptation}: given a universal user preference summary and a downstream task, derive a task-conditioned representation that preserves sufficient decision-relevant evidence while removing redundant context. To this end, we propose \textsc{AlignXada}, a training-free meta-learning framework that induces reusable textual refinement policies for adapting universal preference summaries to task-specific ones. The refinement policy is iteratively optimized by a meta learner through verbal reinforcement learning. Across 13 tasks and three downstream models (39 task--model cells), \textsc{AlignXada} achieves an average gain of 3.82 points, improving 33 cells while retaining only 22.8\% of the original profile tokens and outperforming RAG in 36 cells. An extended faithfulness analysis further shows that the refined profiles remain largely grounded in the source preferences while preserving task-relevant personalization signals, suggesting that profile-side adaptation serves as a practical complement to universal memory construction for lifelong personalized agents.
• 中文摘要 自然语言用户偏好为LLM个性化提供了可解释的界面。然而，通用偏好摘要通常包含与特定下游任务无关的信息。直接提供完整的偏好摘要因此浪费上下文容量并引入跨任务分心，而手动设计任务特定偏好视图则难以扩展。本研究中，我们研究\emph{任务特定偏好适应}：给定通用用户偏好摘要和下游任务，推导出任务条件表示，既保留足够的决策相关证据，又去除冗余上下文。为此，我们提出了\textsc{AlignXada}，一个无需训练的元学习框架，用于将通用偏好摘要调整为任务特定摘要，从而诱导可复用的文本精炼策略。该精炼策略由元学习者通过口头强化学习迭代优化。在13个任务和3个下游模型（39个任务模型单元格）中，\textsc{AlignXada}平均提升3.82个点，提升33个单元格，同时保留了仅22.8%的原始配置文件代币，并在36个单元格中优于RAG。进一步的忠实度分析显示，精细化后的配置文件在保持任务相关个性化信号的同时，基本基于源偏好，表明配置文件侧适应是终身个性化智能体通用记忆构建的实用补充。

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

双向上下文自我蒸馏用于基于技能的大型语言模型代理的强化学习

• Authors: Tianjun Pan, Yuan Li, Hongda Wang, Linbo Jin, Mengfei Song, Lei Gao, Qiming Shi, Shaokang Fu, Jiarong Zhao, Chengyu Wang, Chengfu Huo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09555
• Pdf link: https://arxiv.org/pdf/2608.09555
• Abstract External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.
• 中文摘要 外部自然语言技能为大型语言模型（LLM）代理提供了可复用和可编辑的指导，用于解决复杂任务。然而，其有效性不仅取决于技能质量，还取决于策略是否能将所提供的指导转化为适当的行动。然而，专门设计用于提升这种技能利用能力的方法仍然鲜有探索。在实践中，基于技能的智能体通常以任务级奖励为中心的强化学习目标进行训练，这种目标提供有限的监督，难以捕捉策略在有效利用这些技能上的细微差异。我们提出了BCSD（双向上下文自我蒸馏）框架，将自我蒸馏与强化学习结合起来，以训练LLM代理更有效地使用外部技能。与以往依赖单一特权上下文的自我蒸馏方法不同，BCSD从两个互补的技能-上下文视角评估每条路径。增强视图引入更高层元技能指导，而缩减视图则修剪通用指导以突出任务特定技能。它们互补的标记级信号被合并，重新调整强化学习优势。ALFWorld和WebShop上的实验表明，BCSD在模型尺度上实现了最强的整体性能，使代理更有效地利用外部技能。消融研究进一步验证了增强与简化上下文视图的互补贡献。代码将发布以确保完全可重复性。

AudioMap: Cloze-and-Choice Reinforcement Learning for Time-Aware Dense Audio Captioning

AudioMap：针对时间感知密集音频字幕的闭门选择强化学习

• Authors: Yan Rong, Fengji Ma, Xu Li, Jinting Wang, Chen Zhang, Li Liu
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2608.09559
• Pdf link: https://arxiv.org/pdf/2608.09559
• Abstract Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle to achieve these two goals and mainly rely on supervised fine-tuning, yielding sub-optimal performance. While reinforcement learning (RL) shows promise, applying it to TDAC faces two main challenges: (1) existing rewards are too coarse to supervise multi-event, multi-attribute, and multi-relation descriptions in a fine-grained manner; and (2) temporal supervision is difficult for free-form captions, where flexible event-time expressions make reliable event-time correspondence challenging. To address these challenges, we propose AudioMap, a novel RL-based TDAC framework, which shifts to a unified cloze-and-choice reward paradigm. Specifically, we introduce the Evidence Sufficiency Reward (ESR) with an asymmetric hierarchical scoring mechanism to promote fine-grained accuracy and descriptive richness across diverse acoustic dimensions. Furthermore, we design the Event-Conditioned Temporal Reward (ECTR) to structurally bind timestamps to event semantics via temporal IoU, accompanied by a dual-curriculum learning strategy to facilitate the training process. Finally, to support this task, we construct the first time-aware fine-grained audio captioning dataset, AudioMapCap-44K, which contains 44K carefully annotated captions. Extensive experiments across diverse benchmarks show that AudioMap achieves state-of-the-art (SOTA) performance among open-source models and delivers competitive or superior results relative to proprietary models. Project page and release updates are available at this https URL.
• 中文摘要 时间感知密集音频字幕（TDAC）旨在生成具有精确时间边界（时间感知）的多个细粒度属性（密度）。现有方法难以实现这两个目标，主要依赖监督式微调，导致性能不理想。虽然强化学习（RL）展现出潜力，但将其应用于TDAC面临两个主要挑战：（1）现有奖励过于粗糙，无法以细粒度方式监督多事件、多属性和多关系的描述;（2）时间监督对于自由形式字幕来说较为困难，因为灵活的事件时间表达式使得可靠的事件时间对应难以实现。为应对这些挑战，我们提出了AudioMap，一种基于强化学习的新TDAC框架，转向统一的闭门与选择奖励范式。具体来说，我们引入了证据充分奖励（ESR），采用非对称的层级评分机制，以促进多维声学的细粒度准确性和描述丰富性。此外，我们设计事件条件时间奖励（ECTR），通过时间意向（IoU）结构性地将时间戳与事件语义绑定，并配合双课程学习策略以促进训练过程。最后，为支持该任务，我们构建了首个时间感知的细粒度音频字幕数据集AudioMapCap-44K，其中包含44K条精心注释的字幕。跨多种基准测试的广泛实验表明，AudioMap在开源模型中达到了最先进的（SOTA）性能，并且相较于专有模型，呈现出更具竞争力甚至更优的结果。项目页面和发布更新可在此 https 网址获取。

Bayesian Symbolic Regression with Entropic Reinforcement Learning

贝叶斯符号回归与熵强化学习

• Authors: Oussama Boussif, Mohammed Mahfoud, Younesse Kaddar, Moksh Jain, Sida Li, Damiano Fornasiere, Xiaoyin Chen, Yoshua Bengio, Esmeralda S. Whitammer
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.09617
• Pdf link: https://arxiv.org/pdf/2608.09617
• Abstract Symbolic regression is the problem of finding an algebraic expression describing a stochastic dependence of a target variable on a set of inputs. Unlike forms of regression that fit parameters assuming a fixed model structure, symbolic regression is a search problem over the space of expressions, represented, for example, as abstract syntax trees using a library of operators. Symbolic regression is typically used in settings with limited, noisy data in the natural sciences. However, searching for a single best-fitting expression fails to capture the epistemic uncertainty about the expression, which motivates a Bayesian perspective that enables uncertainty quantification and specification of natural priors to constrain the search space. In this work, we propose ERRLESS (Entropy-Regularized Reinforcement Learning for Expression Structure Sampling), a scalable approach for sampling from the posterior distribution over expressions given data using maximum-entropy reinforcement learning. ERRLESS learns a neural policy that constructs expressions sequentially by building up their abstract syntax trees. At convergence, the policy samples expressions from the posterior. At test time, expressions can be sampled by rollouts of this policy. We demonstrate that ERRLESS achieves competitive results on the Feynman benchmark while producing short and interpretable expressions. Additionally, we demonstrate that the mean of the posterior predictive approximated by ERRLESS achieves a high coefficient of determination ($R^2$) compared to an SMC baseline, highlighting the benefits of the Bayesian perspective in symbolic regression.
• 中文摘要 符号回归是寻找一个描述目标变量对一组输入的随机依赖的代数表达式的问题。与假设固定模型结构的参数拟合形式不同，符号回归是一个覆盖表达式空间的搜索问题，例如，可以用运算子库表示为抽象语法树。符号回归通常用于自然科学中数据有限且噪声大的环境。然而，单一寻找最优拟合表达式无法捕捉表达式的认识论不确定性，这促使采用贝叶斯视角，使不确定性量化和自然先验的指定来约束搜索空间。本研究提出ERRLESS（表达结构采样的熵正则化强化学习），这是一种可扩展的方法，用于通过最大熵强化学习从给定数据的表达式后验分布抽样。ERRLESS学习一种神经策略，通过构建抽象语法树顺序构建表达式。在收敛阶段，策略采样后验表达式。测试时，表达式可通过该策略的推广进行采样。我们证明ERRLESS在费曼基准测试中取得了竞争性成绩，同时生成短小且可解释的表达式。此外，我们证明ERRLESS近似的后验预测平均值相比SMC基线具有较高的确定系数（$R^2$），突出了贝叶斯视角在符号回归中的优势。

SR-OPSD: Self-Referenced On-Policy Self-Distillation

SR-OPSD：自指政策自我提炼

• Authors: Zhuo Sun, Entong Li, Yanlong Zhao, Xiaoyuan Cheng, Wenxuan Yuan, Kaiyu Li, Che Liu, Huihang Liu, Harrison Bo Hua Zhu, Li Zeng
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2608.09745
• Pdf link: https://arxiv.org/pdf/2608.09745
• Abstract On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on trajectories generated by the policy to be optimized, providing a useful complement to reinforcement learning with sparse outcome rewards. However, the self-teacher policy used in OPSD is typically a stop-gradient or exponential-moving-average copy of the policy conditioned on additional context information, and thus co-evolves with both the student policy and its on-policy context distribution. Directly matching such a moving target with a fixed projection objective can lead to unstable optimization or excessive distributional concentration. This nature of OPSD motivates the proposed \emph{Self-Referenced On-Policy Self-Distillation (SR-OPSD)}. At fixed student-generated contexts, a token-level variational characterization identifies the effective distillation target as a geometric interpolation between the self-teacher policy and a reference policy. Meanwhile, we use the Rényi divergence family to generalize the projection geometry. This formulation separates \emph{where} the adaptive target is placed from \emph{how} the student is projected toward it: the interpolation coefficient controls underlying target, while the Rényi order controls the projection geometry and its sensitivity to token-level density ratios. Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings.
• 中文摘要 策略上自我蒸馏（OPSD）将反馈转化为对待优化策略生成轨迹的密集代币级监督，为奖励稀疏的强化学习提供了有用的补充。然而，OPSD中使用的自学策略通常是基于额外上下文信息的停止梯度或指数移动平均的策略副本，因此与学生策略及其政策内上下文分布共同演化。直接将移动目标与固定投影目标匹配可能导致优化不稳定或分布过度集中。OPSD的这一性质促使了提出的\emph{自指策略自我蒸馏（SR-OPSD）}。在固定学生生成上下文中，代币级变分特征识别有效蒸馏目标为自教师策略与参考策略之间的几何插值。同时，我们使用Rényi散度族来推广投影几何。该表述将自适应目标的放置位置与学生如何向该目标投射的过程分开：插值系数控制底层目标，而Rényi阶控制投影几何及其对标记级密度比的敏感度。在科学评估、数学推理和编码生成任务中，使用多个大型语言模型进行大量实验表明，SR-OPSD在各种环境中实现了最先进或具竞争力的性能。

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition

通过集中训练和批判分解，实现机器人操作的高效现实世界在线强化学习

• Authors: Changhao Li, Yifang Zhang, Heng Zhang, Davide Torielli, Damiano Gasperini, Arturo Laurenzi, Luca Muratore, Arash Ajoudani, Nikos Tsagarakis
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.09762
• Pdf link: https://arxiv.org/pdf/2608.09762
• Abstract Real-world online reinforcement learning (RL) provides a promising approach for training robotic manipulation policies directly in the physical world, avoiding the sim-to-real gap and enabling continuous policy refinement through human-in-the-loop interaction. Recent methods have demonstrated sample-efficient learning through human intervention but remain limited to small randomization ranges and encounter challenges with the non-stationarity induced by concurrently training multiple agents. To address these limitations, we introduce a unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA). This enables multiple actors to share a centralized multi-head critic. The critic is decomposed into task and grasp heads, corresponding to the sparse task reward and a potential-based grasping reward, respectively. We accordingly reformulate the critic and actor objectives to exploit the decomposed Q-values while explicitly accounting for the categorical action distribution of the discrete gripper policy. Experimental results demonstrate that the proposed framework substantially improves both sample efficiency and policy performance. We validate our approach on two robotic arms and a simulated humanoid robot across tennis ball and banana pick-and-place, pot reset, and simulated block relocation tasks under dimension-wise domain randomization, approximately 5-25x larger than those considered in prior work. Compared with a state-of-the-art baseline, our method improves the success rate from 60% to 80% on tennis ball pick-and-place, from 60% to 90% on banana pick-and-place, and from 25% to 95% on simulated block relocation, while also successfully accomplishing a task where the baseline consistently fails. Videos and more details are available at our project website: this https URL.
• 中文摘要 现实世界的在线强化学习（RL）为直接在物理世界中训练机器人操作策略提供了有前景的方法，避免了模拟与现实之间的差距，并通过人机互动实现了持续的策略优化。近期方法已证明通过人工干预实现了样本高效学习，但仍受限于较小的随机化范围，并遇到同时训练多个智能体所带来的非平稳性挑战。为解决这些局限性，我们引入了一个统一框架，结合了集中训练与去中心化执行（CTDE）以及混合奖励架构（HRA）。这使得多个参与者共享一个集中的多头批评者。批评者被分解为任务头和抓取头，分别对应稀疏任务奖励和基于潜力的抓取奖励。因此，我们重新表述了批判者和行动者目标，以利用分解后的Q值，同时明确考虑离散抓手策略的类别动作分布。实验结果表明，所提出的框架显著提升了样本效率和策略性能。我们在网球和香蕉拾取放置、锅具重置及模拟块块移动任务中，基于维度域随机化验证了我们的方法，这一方法比以往研究高出约5-25倍。与最先进的基线相比，我们的方法将网球拾取放放成功率从60%提升到80%，香蕉拾取放放从60%提升到90%，模拟积木移动成功率从25%提升到95%，同时成功完成了基线持续失败的任务。视频及更多详情可在我们的项目网站观看：此 https URL。

Parameter Exploration for RLVR via Variational Learning

通过变分学习进行RLVR参数探索

• Authors: Vatsal Venkatkrishna, Nico Daheim, Iryna Gurevych
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.09805
• Pdf link: https://arxiv.org/pdf/2608.09805
• Abstract Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.
• 中文摘要 探索长期以来一直是强化学习研究的重点。最近，越来越多的证据表明，探索也是大型语言模型强化学习配方中的重要组成部分，能够显著影响下游性能。许多现有方法通过温度缩放控制动作空间中的探索。然而，这些方法无法重排序令牌，只能影响输出分布的方差。这限制了探索，可能导致发散或训练停滞。本文探讨参数空间探索，即通过从后验采样不同策略生成的部署，每个策略可能探索不同的部署。采样多样性较低或更多样化的策略则成为探索的互补控制杠杆。我们介绍了一系列称为扰动参数策略优化（Perturbed Parameter Policy Optimization，3PO）的方法，利用不同的采样策略和不同的展开分组来估算奖励。在数学推理和代码生成任务中对OLMo-3-1025-7B和Qwen2.5-Math-7B的实验表明，这些方法在几乎相同的FLOP成本下，持续提升标准GRPO的平均下游性能。此外，使用多参数样本在训练过程中比GRPO和动作空间基线更少零优势组和错误或错误的展开。总体而言，我们的工作提供了参数空间探索能够改善LLM强化学习的证据。

Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation

将技能提炼为权重，而非提示：抽象技能作为策略性自我提炼的特权信号

• Authors: Yubo Jiang, Fengying Xie, Zhiguo Jiang, Haopeng Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.09826
• Pdf link: https://arxiv.org/pdf/2608.09826
• Abstract Reinforcement learning with verifiable rewards yields no group-relative signal when rollout groups are uniformly correct or uniformly wrong, which account for 63.0-68.0% of groups in our experiments. We propose SKALD (Skill-Anchored Latent Distillation), an on-policy self-distillation framework that uses two context views of the same Qwen3-Base model: a question-only student and a teacher conditioned on an abstract, explicit-answer-filtered skill card. The student is trained on its own prefixes, transferring the skill-induced advantage into shared parameters without privileged input at test time. To stabilize context-induced distribution mismatch, SKALD employs an annealed exponentially tilted objective that downweights teacher-preferred tokens with very low student likelihood; as the tilt vanishes, it converges to teacher cross-entropy and recovers the forward-KL student gradient. An empirical gate activates distillation only when verified rollouts estimate a positive teacher advantage. Across five held-out mathematics benchmarks, SKALD improves overall avg@8 over GRPO by +2.46, +4.85, and +12.01 at 0.6B, 1.7B, and 4B, respectively. At 1.7B, zero-variance-only distillation recovers 84.7% of the full gain, while SKALD remains +4.06 above FLOP-matched GRPO and exceeds contextual skill exposure by +3.77. These results show that abstract skills provide dense supervision where group-relative rewards become uninformative.
• 中文摘要 带有可验证奖励的强化学习在推广组一致正确或一致错误时，不产生组相对信号，这占实验中63.0-68.0%的组别。我们提出SKALD（技能锚定潜在蒸馏），这是一种基于策略的自蒸馏框架，使用同一Qwen3-Base模型的两种上下文视图：仅问题学生和基于抽象显式答案过滤技能卡的教师。学生在自身前缀上训练，将技能诱导的优势转移到共享参数中，测试时不需特权输入。为稳定上下文诱发的分布不匹配，SKALD采用退火指数倾斜目标，降低教师偏好标记的权重，学生可能性极低;当倾斜值消失时，趋向教师交叉熵并恢复前向KL学生梯度。只有当经过验证的推广估计教师优势为正时，经验门才激活蒸馏。在五个保留的数学基准测试中，SKALD在GRPO上avg@8整体提升了+2.46、+4.85和+12.01，分别为0.6B、1.7B和4B。在1.7B时，零方差仅提取回收了84.7%的全部增益，而SKALD仍比FLOP匹配GRPO高出+4.06，且比上下文技能暴露高出+3.77。这些结果表明，抽象技能提供了密集的监督，而群体相对奖励变得无益。

Keyword: diffusion policy

There is no result