生成时间: 2026-08-20 16:39:02 (UTC+8); Arxiv 发布时间: 2026-08-20 20:00 EDT (2026-08-21 08:00 UTC+8)

今天共有 17 篇相关文章

Keyword: reinforcement learning

Position: Profiling Game Worlds by Transition Complexity

立场：通过过渡复杂度分析游戏世界

• Authors: Lele Cao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.18079
• Pdf link: https://arxiv.org/pdf/2608.18079
• Abstract Game world modeling (GWM) and reinforcement learning (RL) are often confounded because research papers rarely quantify how difficult the underlying transition prediction problem is at the declared interface (pixels/tokens/latents with finite history). We propose the Transition Complexity Profile (TCP): a small, reproducible set of metrics that characterizes an environment's (or gameplay dataset's) induced transition kernel by (i) intrinsic one-step branching, (ii) interaction-induced uncertainty and opponent influence when observable, and (iii) temporal/spatial dependency span via standardized probe curves. TCP is reported with an explicit reference distribution, protocol stochasticity, and a versioned measurement budget (sampling/resampling and fixed probe compute), enabling comparable numbers across benchmarks. We outline how common game families and modern "neural game engine" domains populate this landscape and call for TCP to become standard benchmark metadata and a required statistic in GWM and RL papers.
• 中文摘要 游戏世界建模（GWM）和强化学习（RL）常被混淆，因为研究论文很少量化底层转换预测问题在声明接口（像素/标记/有限历史的潜在变量）下的难易度。我们提出了过渡复杂性曲线（TCP）：一组小型、可重复的指标，通过（i）内在的一步分支、（ii）交互引起的不确定性和对手影响（可观测时）以及（iii）通过标准化探针曲线实现的时间/空间依赖性跨度来描述环境（或游戏数据集）诱导的过渡核。TCP以显式参考分布、协议随机性和版本化测量预算（采样/重采样和固定探针计算）报告，使得基准测试数据可比。我们概述了常见游戏家族和现代“神经游戏引擎”领域如何填充这一领域，并呼吁将TCP作为标准基准元数据，并成为GWM和强化学习论文中的必备统计数据。

SingularClip: Preventing Spectral Collapse to Maintain Plasticity in Continual and Reinforcement Learning

SingularClip：防止光谱坍缩以维持持续和强化学习中的可塑性

• Authors: Tyler Kastner, Nimrod De La Vega, Amir-massoud Farahmand
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.18319
• Pdf link: https://arxiv.org/pdf/2608.18319
• Abstract Neural networks trained on nonstationary tasks frequently lose the ability to fit new targets, a phenomenon referred to as loss of plasticity. We identify a novel source of plasticity loss due to the growing anisotropy of weight matrices' singular values during training, and analyze this phenomenon both empirically and theoretically. To mitigate this issue, we introduce SingularClip, a procedure that periodically clips the singular values of all weight matrices. We show that SingularClip performs strongly against baselines across a range of tasks in both continual supervised learning and deep reinforcement learning.
• 中文摘要 训练于非平稳任务的神经网络经常失去拟合新目标的能力，这种现象称为可塑性丧失。我们发现了一种新源，源于训练过程中重量矩阵奇异值各向异性增加，并对这一现象进行了实证和理论分析。为缓解此问题，我们引入了SingularClip，一种定期裁剪所有权重矩阵奇异值的过程。我们证明，SingularClip在持续监督学习和深度强化学习等多种任务中，基线表现优异。

Adaptive Multi-Agent Feature Selection for Personalized Fall Risk Prevention

自适应多智能体功能选择以实现个性化跌倒风险预防

• Authors: Chang Liu, Ladda Thiamwong, Yanjie Fu, Rui Xie
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.18450
• Pdf link: https://arxiv.org/pdf/2608.18450
• Abstract Falls among older adults represent a major public health challenge driven by complex, time-varying interactions across multiple risk domains. Effective fall risk factor identification requires learning from heterogeneous longitudinal data while accounting for sparse and delayed fall-related outcome events. However, existing approaches are largely static and fail to adaptively model evolving, individualized risk factors across modalities and time. We propose PAFIR, a Personalized and Adaptive Feature selection framework for fall risk Identification and pRevention, which formulates adaptive feature selection as a reinforcement learning problem over longitudinal multimodal health data. PAFIR jointly models structural dependencies among correlated assessment variables and temporal dynamics in wearable-derived physical activity data, and learns adaptive selection policies across repeated study visits using reward signals derived from sparse fall incidence outcomes. We apply PAFIR to data from the Physio fEedback Exercise pRogram (PEER) cluster-randomized trial. Experimental results demonstrate that PAFIR more effectively captures longitudinal and structural patterns of feature relevance than state-of-the-art baselines, and enables dynamic, subject-specific feature selection. By adapting selected features over time, PAFIR supports more timely and personalized fall prevention strategies.
• 中文摘要 老年人的跌倒代表了一项重大公共卫生挑战，由多个风险领域中复杂且时间变化的相互作用驱动。有效的跌倒风险因素识别需要从异质纵向数据中学习，同时考虑稀疏和延迟的跌倒相关结局事件。然而，现有方法大多是静态的，无法适应性地建模不同模式和时间范围内不断演变的个性化风险因素。我们提出了PAFIR，一种个性化且自适应的特征选择框架，用于跌倒风险识别和预防，将适应性特征选择作为对纵向多模态健康数据的强化学习问题提出。PAFIR联合建模了可穿戴体力活动数据中相关评估变量与时间动态之间的结构依赖关系，并通过稀疏跌倒发生率结果得出的奖励信号，在重复研究访问中学习适应性选择策略。我们将PAFIR应用于Physio fEedback Exercise pRogram（PEER）集群随机试验的数据。实验结果表明，PAFIR比最先进的基线更有效地捕捉纵向和结构性的特征相关模式，并实现动态、针对受试者的特征选择。通过逐步调整部分特征，PAFIR支持更及时、更个性化的跌倒预防策略。

Continual Reasoning Gym: Diagnosing and Harnessing Shared Reasoning in Continual RLVR

持续推理健身房：在持续RLVR中诊断并运用共享推理

• Authors: Lirui Luo, Guoxi Zhang, Hongming Xu, Rongqing Li, Cong Fang, Lifeng Fan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.18574
• Pdf link: https://arxiv.org/pdf/2608.18574
• Abstract Reinforcement learning with verifiable rewards (RLVR) commonly post-trains reasoning models on multiple tasks, while rerunning multitask RLVR (MTRL) as new tasks are added makes capability expansion costly. We therefore study continual RLVR, which updates the existing model as each task arrives. The central question is whether a model updated this way can perform as well as a jointly trained model. To answer this question, we introduce Continual Reasoning Gym, a continual-RLVR environment that organizes text and visual reasoning tasks into five task sequences. In this setting, we identify two key observations: Sequential RLVR exhibits modest forgetting, yet its final performance remains below that of MTRL. To understand the latter, we decompose final performance and show that forgetting accounts for only part of the gap. To explain the former, we identify shared reasoning: transferable reasoning structure allows training on one task to support others on average. We therefore introduce Continual Prompt Replay (CPR), which harnesses shared reasoning to improve learning on the arriving and future tasks by replaying previous-task prompts and regenerating their responses with the current policy. On average, only CPR reaches MTRL-level performance.
• 中文摘要 带有可验证奖励的强化学习（RLVR）通常会对多个任务进行推理模型的后期训练，而随着新任务的加入，重复运行多任务RLVR（MTRL）使能力扩展成本高昂。因此，我们研究持续的RLVR，它会随着每个任务的到来更新现有模型。核心问题是，以这种方式更新的模型是否能与联合训练的模型表现相当。为了回答这个问题，我们引入了Continual Reasoning Gym，这是一个持续式RLVR环境，将文本和视觉推理任务组织成五个任务序列。在此情境下，我们发现了两个关键观察：顺序RLVR表现出适度遗忘，但其最终表现仍低于MTRL。为了理解后者，我们分解了最终表现，并证明遗忘仅占了缺口的一部分。为了解释前者，我们识别了共享推理：可转移推理结构允许对一项任务的训练平均支持其他任务。因此，我们引入了持续提示重放（CPR），它利用共享推理，通过重放之前任务提示并根据当前政策生成其响应，提升当前和未来任务的学习效果。平均而言，只有CPR达到MTRL水平。

VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation

VA-Judger：基于人类偏好反馈的奖励建模，用于联合视频-音频生成

• Authors: Yinming Huang, Shuyuan Tu, Xi Yan, Zihan Yang, Jianhua Han, Xu Hang, Yu-Gang Jiang, Zuxuan Wu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.18607
• Pdf link: https://arxiv.org/pdf/2608.18607
• Abstract Using reinforcement learning to post-train joint video-audio generation models requires a reward signal. Existing methods construct this reward by combining metrics for individual quality dimensions, including audio quality, visual fidelity, and synchronization. However, these metrics evaluate perceptual dimensions separately and fail to capture the overall semantic and temporal coherence among the text prompt, video, and audio that shapes human preferences. Optimizing models against these metrics encourages reward hacking, generating video-audio content that achieves high scores on these metrics yet appears incoherent or unfaithful to human viewers. To address this problem, we first construct a large-scale human-preference dataset VAPref-10K for joint video-audio generation, comprising 9K prompts and 10.3K fine-grained paired comparisons from open-source generation models. We also introduce the VA-Judger-Bench benchmark with both in-domain and out-of-domain model comparisons to evaluate whether reward models truly align with human preferences. We further propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation. In particular, VA-Judger first learns from pairs with clear quality gaps to establish structured output and coarse preference discrimination, then distills reliable preference explanations for harder near-quality comparisons via rejection sampling verified against human annotations, and finally performs dimension-wise reinforcement learning that decomposes human feedback into individual quality dimensions for denser reward signals than a single binary preference label. Experiments show that VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations. Using its human-aligned rewards for post-training audio-video generation model also yields significant improvements in generation quality.
• 中文摘要 利用强化学习对联合视频-音频生成模型进行后期训练需要奖励信号。现有方法通过结合各质量维度的指标来构建这种奖励，包括音频质量、视觉真实度和同步性。然而，这些指标分别评估感知维度，未能捕捉文本提示、视频和音频之间塑造人类偏好的整体语义和时间连贯性。针对这些指标进行优化模型，会鼓励奖励黑客行为，生成在这些指标上得分很高但对人类观众来说显得不连贯或不忠实的视频音频内容。为解决此问题，我们首先构建了一个大规模的人类偏好数据集VAPref-10K，用于联合视频-音频生成，包含9K个提示和10.3K个来自开源生成模型的细粒度配对比较。我们还引入了VA-Judger-Bench基准，结合了领域内和域外的模型比较，以评估奖励模型是否真正符合人类偏好。我们还提出了VA-Judger，一种用于联合视频-音频生成的思维链全奖励模型。特别是，VA-Judger首先从具有明显质量差距的配对中学习，建立结构化输出和粗略偏好区分，然后通过拒绝抽样与人类注释验证，提炼出更难的近质量比较的可靠偏好解释，最后进行按维度进行强化学习，将人类反馈分解为单个质量维度，获得比单一二元偏好标签更密集的奖励信号。实验表明，VA-Judger在预测人类偏好方面，无论是域内还是域外评估，都优于指标基线。其基于人类的奖励用于训练后音视频生成模型，也显著提升了生成质量。

PCQA-R1: Advancing Generalized 3D Point Cloud Quality Assessment with Reinforcement Learning

PCQA-R1：推进强化学习的通用三维点云质量评估

• Authors: Kangning Ye, Yunhao Li, Sijing Wu, Yucheng Zhu, Guangtao Zhai
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.18627
• Pdf link: https://arxiv.org/pdf/2608.18627
• Abstract No-reference point cloud quality assessment (PCQA) has been an active topic in recent years and is used to measure and optimize the visual experience of point clouds. However, large multimodal models (LMMs) have rarely been explored in this area. Previous LMM-based methods mainly rely on supervised fine-tuning to directly predict numerical quality scores, lacking the ability to generalize across datasets with heterogeneous MOS scales and limited annotations. A key difficulty is that absolute MOS regression can be brittle across datasets with different score scales and distortion distributions, whereas relative quality ranking is more stable under such shifts. In this paper, we present PCQA-R1, the first reinforcement learning LMM for 3D point cloud quality assessment to simultaneously model quality understanding and scoring. Built upon the group relative policy optimization (GRPO) strategy, PCQA-R1 first constructs a chain-of-thought dataset, PCQA-CoT, which serves as cold-start training data through a reverse reasoning strategy that teaches the LMM to generate its reasoning process. We further introduce a Gaussian proximity reward that prevents calibration drift by anchoring score predictions to the source MOS range. Experimental results demonstrate that PCQA-R1 achieves state-of-the-art cross-dataset generalization across five benchmarks and competitive in-domain accuracy. Ablation studies support the role of ranking, Gaussian reward, and cold-start traces.
• 中文摘要 无参考点云质量评估（PCQA）近年来成为一个活跃的话题，用于测量和优化点云的视觉体验。然而，大型多模态模型（LMMs）在该领域很少被探索。以往基于LMM的方法主要依赖监督微调直接预测数值质量评分，缺乏跨数据集的泛化能力，且MOS尺度和注释有限。一个关键难点是，绝对MOS回归在不同评分尺度和失真分布的数据集间可能较为脆弱，而相对质量排名在此类变化下更为稳定。本文介绍了PCQA-R1，这是首个用于3D点云质量评估的强化学习LMM，旨在同时建模质量理解和评分。基于群体相对策略优化（GRPO）策略，PCQA-R1首先构建了一个思维链数据集PCQA-CoT，作为冷启动训练数据，通过反向推理策略教导LMM生成其推理过程。我们进一步引入高斯邻近奖励，通过将分数预测锚定到源MOS范围，防止校准漂移。实验结果表明，PCQA-R1实现了跨五个基准测试的先进跨数据集泛化，并具备竞争的领域准确性。消融研究支持排名、高斯奖励和冷启动痕迹的作用。

RTPO: Reverse-Turn Policy Optimization for Stabilizing Agentic RL Training

RTPO：反向转向策略优化以稳定能动强化学习训练

• Authors: Yugu Li, Jimmy Cao, Jianglin Qiao, Siyi Hu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.18682
• Pdf link: https://arxiv.org/pdf/2608.18682
• Abstract Training multi-turn agentic workflows with reinforcement learning (RL) enables large language models to perform complex reasoning, use external tools, and conduct iterative search beyond single-turn settings. Yet multi-turn RL training remains highly unstable, often causing severe performance degradation as the number of turns increases. Through theoretical analysis, we identify three tightly coupled sources of instability: rollout-training context mismatch, weak turn-level credit assignment under sparse terminal rewards, and asynchronous policy drift when short and long trajectories are optimized under different policy versions. We show that these issues share a common structural origin in flattened trajectory optimization and address them through a unified reverse-turn formulation. We propose Reverse-Turn Policy Optimization (RTPO), which organizes multi-turn rollouts as sparse reverse trees and performs turn-level policy updates in temporal reverse order, aligning each decision with its downstream continuation. RTPO enables causally consistent turn-level credit assignment and on-policy continuation to control asynchronous drift. We provide theoretical guarantees showing that RTPO eliminates context mismatch and asynchronous drift under the proposed turn-level formulation, reduces credit bias, and converges to recursive optimality. Experiments on multi-turn agentic RL benchmarks show that RTPO improves upon trajectory- and turn-level baselines by 21.50% and 10.76%, respectively, highlighting its potential to support more stable training for tool-using agents.
• 中文摘要 通过强化学习（RL）训练多回合智能体工作流，使大型语言模型能够进行复杂推理、使用外部工具，并进行超越单回合设置的迭代搜索。然而，多匝强化学习训练依然高度不稳定，随着匝数增加，常常导致性能严重下降。通过理论分析，我们识别出三种紧密耦合的不稳定性来源：推广-训练上下文不匹配、终端奖励稀疏下转折级信用分配较弱，以及在不同策略版本下优化短长期轨迹时的异步策略漂移。我们展示了这些问题在扁平轨迹优化中有共同的结构性根源，并通过统一的反向转向表述加以解决。我们提出了反向转向策略优化（RTPO），该方法将多回合的推出组织为稀疏的反向树，并按时间逆序进行回合级政策更新，使每个决策与其后续对齐。RTPO支持因果一致的回合级信用分配和策略继续，以控制异步漂移。我们提供了理论保证，表明RTPO在所提的转向级表述下消除了上下文不匹配和异步漂移，减少了信用偏差，并趋向递归最优。多回合智能强化学习基准测试的实验显示，RTPO相比轨迹级和转向级基线分别提升了21.50%和10.76%，凸显了其支持工具使用智能体更稳定训练的潜力。

MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models

MLREF：通过大型语言模型在强化学习中高效模块重用奖励设计

• Authors: Chenglin Liu, Xun Wang, Ruishuo Chen, Zhuoran Li, Longbo Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.18827
• Pdf link: https://arxiv.org/pdf/2608.18827
• Abstract Reward function design remains a bottleneck in reinforcement learning. While large language models (LLMs) have enabled automated reward generation, existing methods generate and revise reward functions as monolithic programs, making it difficult to reliably preserve and reuse effective components discovered in earlier iterations, leading to unstable performance across iterations. To address this, we propose Module Level Reward Evolution Framework (MLREF). At the core of MLREF is a module pool, a persistent repository of reusable reward components. MLREF treats the module pool as the primary optimization object: the pool evolves across iterations by accumulating successful modules, refining underperforming ones, and reusing proven components; while reward functions are constructed as linear combinations of modules drawn from this pool. To drive this evolution, MLREF integrates three mechanisms: reflection-based refinement, hybrid credit assignment, and a merge strategy with rollback, which together improve the effectiveness and robustness of reward optimization. Experiments on 17 tasks show that MLREF outperforms strong baselines by 25.2% in locomotion and 6.6% in manipulation, with more stable optimization dynamics.
• 中文摘要 奖励函数设计仍然是强化学习中的瓶颈。虽然大型语言模型（LLMs）使得自动奖励生成成为可能，但现有方法将奖励函数生成和修订为单一程序，这使得早期迭代中发现的有效组件难以可靠地保存和复用，导致迭代间性能不稳定。为此，我们提出了模块级奖励进化框架（MLREF）。MLREF 的核心是一个模块池，一个可重用奖励组件的持久存储库。MLREF将模块池视为主要优化对象：模块池通过累积成功模块、优化表现不佳模块以及重复使用已验证组件来演进;而奖励函数则由从该池中提取的模的线性组合构成。为推动这一演进，MLREF整合了三种机制：基于反思的细化、混合信用分配以及带回滚的合并策略，这些共同提升了奖励优化的有效性和稳健性。对17个任务的实验显示，MLREF在移动方面比强基线高出25.2%，在操作方面比强基线高出6.6%，且优化动态更为稳定。

Think-to-Personalize: Unifying Reasoning and Retrieval for User-Centric Personalized Dense Retrieval

思维个性化：统一推理与检索，实现以用户为中心的个性化密集检索

• Authors: Angqing Jiang, Gaoming Zhang, Jianchun Song, Kena Qi, Dayao Chen, Wei Lin, Defu Lian
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2608.18855
• Pdf link: https://arxiv.org/pdf/2608.18855
• Abstract Dense retrieval has become a cornerstone of modern local-lifestyle e-commerce search by encoding queries and items into semantic embedding spaces. While recent advancements have transitioned from BERT-based embedding models to Large Language Models (LLMs), most approaches still treat LLMs as static text encoders, neglecting their inherent reasoning capabilities. Furthermore, standard dense retrieval models remain query-centric, which is insufficient in e-commerce scenarios where sparse and ambiguous queries create an intent gap that can only be bridged by the rich context of user history. Meanwhile, existing personalized retrieval methods typically rely on implicit embedding interactions, which lack the reasoning capability to effectively disambiguate user intent from noisy historical behaviors. To address these challenges, we propose Think-to-Personalize (TTP), a novel framework that unifies explicit user-centric intent reasoning with dense retrieval. By reasoning over the user's historical purchase sequence, TTP explicitly deduces latent personalized needs and generates an intent-enhanced query, which is then encoded into a unified dense embedding. Specifically, we design a two-stage training paradigm: (1) a Supervised Fine-Tuning (SFT) stage that establishes cold-start capabilities; and (2) a Reinforcement Learning (RL) stage that aligns the reasoning process with retrieval utility using Group Relative Policy Optimization (GRPO). Extensive experiments on both proprietary and public benchmarks demonstrate that TTP significantly outperforms state-of-the-art baselines. Furthermore, in online A/B tests, it achieved a +0.46% lift in order volume, validating its practical effectiveness and establishing a new paradigm for reasoning-driven personalized dense retrieval.
• 中文摘要 密集检索已成为现代本地生活方式电子商务搜索的基石，通过将查询和项目编码到语义嵌入空间中。尽管近年来的进展已从基于BERT的嵌入模型转向大型语言模型（LLM），但大多数方法仍将LLM视为静态文本编码器，忽视其固有的推理能力。此外，标准的密集检索模型仍以查询为中心，这在电子商务场景中不足以应对稀疏和模糊的查询，导致意图差距，而这种差距只能通过丰富的用户历史背景来弥合。与此同时，现有的个性化检索方法通常依赖隐式嵌入交互，缺乏有效区分用户意图与噪声历史行为的推理能力。为应对这些挑战，我们提出了“思考个性化”（Think-to-Personalize，TTP）新颖框架，将显式用户中心的意图推理与密集检索相结合。通过推理用户的历史购买顺序，TTP明确推断潜在的个性化需求，生成意图增强的查询，然后编码成统一的密集嵌入。具体来说，我们设计了一个两阶段的训练范式：（1）监督式微调（SFT）阶段，建立冷启动能力;以及（2）强化学习（RL）阶段，利用群相对策略优化（GRPO）使推理过程与检索效用对齐。在专有和公开基准测试上的大量实验表明，TTP显著优于最先进的基线。此外，在线A/B测试中，订单量提升了+0.46%，验证了其实用性，并确立了基于推理的个性化密集检索新范式。

Falcon Perception-HD: High Density Perception via Reinforcement Learning

猎鹰感知-HD：通过强化学习实现高密度感知

• Authors: Sofian Chaybouti, Yasser Dahou, Ngoc Dung Huynh, Reda Alami, Hilde Kuehne
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.18881
• Pdf link: https://arxiv.org/pdf/2608.18881
• Abstract Autoregressive perception models trained to localize visual entities under the open-vocabulary setting are mostly trained using Supervised fine-tuning (SFT) with maximum likelihood, yet it optimizes a proxy objective (per-token cross-entropy) that is fundamentally misaligned with perception metrics such as precision and recall. In this paper, we explore post-training reinforcement learning (RL), specifically GRPO, to directly align these models with their evaluation metrics. Building up on the recently introduced Falcon Perception, we design an RL framework that addresses perception-specific challenges: reward design for set-structured outputs and multi-head sampling control. We discover multiple benefits from RL for perception: first, RL unlocks state-of-the-art performance in very dense scenes (up to 500 objects per scene), a regime where most existing systems degrade sharply or collapse; furthermore it fixes common issues in autoregressive perception models like mask repetitions and removes almost entirely the need for NMS and coordinate deduplication, which improve both performance and efficiency and remove the need for hyperparameters tuning; overall, we notice improvements on all levels of difficulties in referring expression segmentation (on PBench and SACO-Gold), and we find an elegant way to preserve the knowledge of whether an object exists or not (as evaluated by MCC) without training on negative samples. We show that a simple reward that penalizes false negatives and positives is sufficient. We develop two hybrid self-annotation pipelines, respectively tailored for difficult referring expressions and very dense scenes, and show their benefits on RL-training. Model weights are released as a Falcon Perception revision~\footnote{this https URL}. Datasets will be published.
• 中文摘要 在开放词汇设置下训练用于定位视觉实体的自回归感知模型，大多采用监督微调（SFT）以最大似然进行训练，但它优化了一个代理目标（每个代币的交叉熵），但该指标与准确性和召回率等感知指标根本不匹配。本文探讨了训练后强化学习（RL），特别是GRPO，以直接将这些模型与其评估指标对齐。基于最近推出的猎鹰感知，我们设计了一个强化学习框架，解决感知特有的挑战：为集合结构输出设计奖励和多头抽样控制。我们发现强化学习对感知有多重益处：首先，强化学习在极为密集的场景（每场景最多可达500个物体）中解锁了最先进的性能，而大多数现有系统会急剧退化或崩溃;此外，它解决了自回归感知模型中常见的问题，如掩膜重复，几乎完全消除了对NMS和坐标去重的需求，这些技术既提升了性能和效率，也无需超参数调优;总体而言，我们在指称表达切割（如PBench和SACO-Gold）上各级难度均有所改善，并找到一种优雅的方法，在不训练负样本的情况下，保留对象是否存在（由MCC评估）。我们证明，只需一个惩罚假阴性和阳性的简单奖励就足够了。我们开发了两条混合自注释流水线，分别针对困难的指称表达和非常密集的场景，并展示了它们在强化学习训练中的优势。模型权重以猎鹰感知版本发布~\footnote{此 https URL}。数据集将会被发布。

Multi-Agent Off-Policy Deep Reinforcement Learning for Smart Campus Coverage

智能校园覆盖的多代理非策略深度强化学习

• Authors: Omar Rady, Mohamed Ayman, Ali Arafa, Mohamed Shalma
• Subjects: Subjects: Machine Learning (cs.LG); Signal Processing (eess.SP)
• Arxiv link: https://arxiv.org/abs/2608.19049
• Pdf link: https://arxiv.org/pdf/2608.19049
• Abstract Deep reinforcement learning (DRL) has recently gained a great attention due to its real-time adaptation and effectiveness in complex optimization problems. This paper investigates the optimal deployment of millimeter-wave (mmWave) base stations (BSs) in a realistic, non-convex campus topology. The optimization problem is NP-hard, due to the non-convex, non-smooth nature of the max-min fairness objective. To overcome these constraints, we formulate the BS placement as a Markov Decision Process (MDP) and systematically benchmark four DRL schemes: a discrete single-agent Deep Q-Network (DQN), a spatially partitioned Multi-Agent DQN, a continuous single-agent Deep Deterministic Policy Gradient (DDPG), and a geographically partitioned multi-agent DDPG framework. Numerical evaluations reveal that the multi-agent DDPG approach substantially outperforms single-agent in dense scenarios. Additionally full coverage is achieved, and a fairness Jain's index of 0.94 is obtained. Finally, the multi-agent demonstrates highly efficient computational convergence of dense scenarios with $400$ users.
• 中文摘要 深度强化学习（DRL）因其实时适应性和在复杂优化问题中的高效性，近年来备受关注。本文探讨毫米波（mmWave）基站（BS）在真实、非凸校园拓扑结构中的最佳部署方法。由于最大最小公平性目标的非凸性和非光滑性，优化问题是NP难的。为克服这些限制，我们将BS布局定为马尔可夫决策过程（MDP），并系统地对四种DRL方案进行了基准测试：离散单代理深度Q网络（DQN）、空间划分的多智能体DQN、连续单智能体深度确定性策略梯度（DDPG）以及地理划分的多智能体DDPG框架。数值评估显示，多智能体DDPG方法在密集场景中显著优于单智能体。此外，还实现了全面覆盖，公平度为贾因指数为0.94。最后，多智能体展示了高效率的密集场景计算收敛，用户价值400美元。

Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

开放式MOPD：多教师政策提炼中能力不平衡的诊断与修正

• Authors: Huan-ang Gao, Haohan Chi, Yong Yan, Shiyuan Feng, Hanlin Wu, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.19098
• Pdf link: https://arxiv.org/pdf/2608.19098
• Abstract Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.
• 中文摘要 多教师策略提炼（M-OPD）已成为一种有前景的范式，通过密集的代币级奖励监督，将领域专门化强化学习（RL）专家整合到单一通才学生中。尽管在实际中取得了成功，但管理多教师能力集成的优化动态仍难以理解，且明显缺乏开放且严格可复现的配方。本研究中，我们在SmolLM3-3B-Base上建立了受控的M-OPD基准测试，采用oracle路由，将能力集成与路由歧义隔离开来。我们的研究揭示了明显的能力集成缺口：标准M-OPD相较于域路由的预言机集合，仅占可用余量的35.6%，而指令跟随等简明任务严重退化和过早停滞。关键是，我们证明了这种失败并非源于梯度冲突，而是由于代币级优化预算的严重错置。这种病理由三个正交因素驱动：跨域结构序列长度差异、由于学习率不均匀导致的动态收敛漂移，以及异步策略更新带来的多步奖励陈旧。为解决这些不平衡，我们引入了Open-MOPD，这是一个原则性框架，结合了代币份额平衡、缺口感知动态预算分配和学生奖励刷新。这些机制共同系统地恢复了跨域平衡，将单个可部署学生的剩余余量恢复率从35.6%提升至83.4%。我们完全开源了端到端的培训后方案、培训轨迹和评估套件，且预算符合学术负担。

PGFS++: Molecular Property Improvement under Synthesis and Diversity Constraints

PGFS++：在合成与多样性约束下的分子性质改进

• Authors: Boqiao Zhang, Godbless James, Sai Krishna Gottipati, Andrew Fitzgibbon
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.19121
• Pdf link: https://arxiv.org/pdf/2608.19121
• Abstract Improving molecular properties, such as drug-likeness or binding affinity, is a recurring task in early-stage drug discovery. However, molecules optimized in an unconstrained chemical space have limited practical value if they cannot be synthesized. Policy Gradient for Forward Synthesis (PGFS) is a synthesis-aware reinforcement learning method for molecular improvement, but its use of reactant embedding prediction makes reactant selection indirect, which, as we show, limits learning effectiveness. We first develop PGFS+, in which reaction templates and second reactants are represented by trainable embedding lookup tables. Combined with a more effective scoring function and RL algorithm, PGFS+ significantly improves the desired property. However, it exposes a reward-hacking failure mode: a powerful reactant search can map diverse input molecules to the same high-reward magnet molecule, improving the reward while collapsing the output diversity. We therefore introduce PGFS++, a synthesis-aware reinforcement learning framework for input-specific molecular improvement. Given an input molecule, PGFS++ treats it as the start of a forward-synthesis trajectory, applies learned reaction templates with compatible in-stock building blocks, and produces a molecule with improved target properties, an explicit synthesis route, and structural similarity to the input. Experiments on molecular improvement tasks show that PGFS++ improves target properties while preserving high output diversity.
• 中文摘要 改善分子性质，如药物相似性或结合亲和力，是早期药物发现中的反复任务。然而，在无约束的化学空间中优化的分子如果无法合成，实际价值有限。前向合成策略梯度（PGFS）是一种基于合成感知的强化学习方法，用于分子改进，但其反应物嵌入预测使反应物选择是间接的，正如我们所示，这限制了学习效果。我们首先开发了PGFS+，其中反应模板和反应物通过可训练嵌入查找表表示。结合更有效的评分函数和强化学习算法，PGFS+显著提升了期望的特性。然而，它暴露出一种奖励黑客的失败模式：强大的反应物搜索可以将不同的输入分子映射到同一高奖励磁铁分子上，从而提升奖励，同时压缩输出多样性。因此，我们引入了PGFS++，一个基于输入特异性分子改进的合成感知强化学习框架。给定输入分子，PGFS++将其视为正向合成轨迹的起点，应用已学到的反应模板和兼容的库存构建模块，生成具有改进靶向性质、明确合成路径和结构相似性的分子。分子改进任务的实验表明，PGFS++在保持高输出多样性的同时，能够改善靶向性质。

Continuous-Time Reinforcement Learning for Controlled Hawkes Jump-Diffusions

受控鹰跳跃扩散的连续时间强化学习

• Authors: Tomasz R. Bielecki, Thibaut Mastrolia, Haoze Yan
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2608.19151
• Pdf link: https://arxiv.org/pdf/2608.19151
• Abstract We study stochastic control of multivariate Hawkes-driven stochastic differential equations with machine learning algorithms in a non-Markovian setting. Due to the path dependence of the memory of the Hawkes intensity, this problem does not fall within classical stochastic control theory outside particular Markovian kernels. We first develop a finite-dimensional Markovianization procedure and algorithm to approximate multivariate Hawkes processes with mixtures of exponential kernels. We prove the convergence of the Markovianized approximation of the Hawkes process, its intensity, and the value of the problem to the original non-Markovian processes and the value of the primal problem. We then formulate continuous-time deterministic policy gradient learning on the Markovianized approximation of the problem, called Hawkes-CT DDPG. We propose a model-free algorithm to solve the non-Markovian Hawkes-driven optimization by observing only the event times of the process, the realization of the solution to the SDE, and a chosen set of decay filters, while the Hawkes kernel coefficients remain unknown. We compare our continuous time reinforcement learning Hawkes-CT DDPG method with discrete time reinforcement learning techniques under three different types of kernels: simple exponential, Erlang, and power-law kernels.
• 中文摘要 我们研究在非马尔可夫环境中，利用机器学习算法对多元霍克斯驱动随机微分方程进行随机控制。由于霍克强度记忆的路径依赖性，该问题不属于经典随机控制理论中的特定马尔可夫核之外。我们首先开发了一种有限维马尔可夫化过程和算法，用于近似多元霍克斯过程与指数核的混合。我们证明了霍克斯过程的马尔可夫化近似收敛性、其强度，以及问题与原始非马尔可夫过程的价值，以及原始问题的价值。随后，我们基于问题的马尔可夫近似（称为Hawkes-CT DDPG）构建连续时间确定性策略梯度学习。我们提出一种无模型算法，通过仅观察过程的事件时间、SDE解的实现以及选定的衰减滤波器集合来求解非马可夫霍克斯驱动的优化，而霍克斯核系数则保持未知。我们将连续时间强化学习Hawkes-CT DDPG方法与离散时间强化学习技术在三种不同类型的核下进行比较：简单指数核、Erlang核和幂律核。

ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

ADEPT：通过训练前后训练，利用强化学习加速灵巧度

• Authors: Jayjun Lee, Jessica Yin, Asif Rana, Nicholas Blauch, Sam Mady, Mohak Bhardwaj, Nima Fazeli, Nathan Ratliff, Karl Van Wyk, Ankur Handa
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.19182
• Pdf link: https://arxiv.org/pdf/2608.19182
• Abstract We introduce Accelerating Dexterity via Pre-Training (ADEPT), a large-scale reinforcement learning (RL) framework for learning sim-to-real transferable dexterity across high degree-of-freedom (DoF) robot embodiments that can solve long-horizon tasks directly from raw visuo-tactile perception. ADEPT pretrains a dexterous policy on a generic object reposing task, then post-trains downstream policies with this pretrained behavior as a prior. ADEPT enables learning new behaviors that are otherwise difficult to discover from scratch on multi-fingered robots and avoids learning the same set of skills over again for every new downstream task. The pretrained policy zero-shots the reposing phase of downstream tasks, but naïve RL fine-tuning rapidly degrades this capability during transfer. We address this with a stable post-training recipe combining behavior-cloning distillation, critic warm-up, and conservative on-policy updates. To safely exploit the full kinematic dexterity, we introduce a joint-space Geometric Fabric that mediates between the RL policy and the robot. We distill post-trained teachers into perceptive students that zero-shot sim-to-real transfer on two embodiments: a 23 DoF Kuka-Allegro with two RGB cameras, and a 29 DoF Flexiv-Sharpa with two RGB cameras and five vision-based tactile sensors, and can solve long-horizon tasks from challenging initial states with dexterity at human-level speed.
• 中文摘要 我们介绍了通过预训练加速灵巧度（ADEPT），这是一个大规模强化学习（RL）框架，用于学习高自由度（DoF）机器人体型的模拟到现实的可转移灵巧度，能够直接通过原始视觉-触觉感知解决远程任务。ADEPT 先在通用对象重置任务上预训练一个灵活策略，然后用该预训练行为作为先验对下游策略进行后期训练。ADEPT使得在多指机器人上难以从零开始发现的新行为，避免了每次新任务都重复学习同一套技能。预训练策略在下游任务的休整阶段实现零射击，但简单的强化学习微调在转移过程中会迅速削弱这一能力。我们通过一套稳定的训练后配方来解决这个问题，结合了行为克隆的提炼、批评热身和保守的政策更新。为了安全发挥完整的运动学灵活性，我们引入了一个关节空间几何织物，作为强化学习策略与机器人之间的中介。我们将经过后期培训的教师提炼为敏锐的学生，他们在两种设备上实现零拍摄模拟到真实的迁移：一台配备两台RGB摄像机的23焦度Kuka-Allegro，以及配备两台RGB摄像机和五个基于视觉的触觉传感器的29景广度Flexiv-Sharpa，能够以人类水平的灵巧度完成从具有挑战性的初始状态开始的长视野任务。

Keyword: diffusion policy

Revisiting the "Push-T" Robot Manipulation Task with Agentic Robotics

与Agentic Robotics一起重温“Push-T”机器人操控任务

• Authors: Shuangyu Xie, Kaiyuan Chen, Ken Goldberg
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.18227
• Pdf link: https://arxiv.org/pdf/2608.18227
• Abstract Push-T is an iconic benchmark for learning manipulation policies from human demonstrations. The robot must use a single point of contact to push a T-shaped block into a target pose. In this short paper, we revisit the Push-T task in the context of emerging advances in Agentic Robotics where an LLM coding agent -- Claude Code with Fable 5 -- is prompted to create an algorithmic solution that does not require any demonstration data. We study how effective the agentic coding loop can solve the Push-T task, and compare the resulting code as policy with the visuomotor imitation learning policy. Results suggest that the agent found the 2D gym simulation online, and used sim experiments to learn push mechanics, iteratively optimizing to achieve 100% success rate using 46% fewer steps than the best diffusion policy trained with 200 human demonstrations. The coding agent also solve extensions from T to the full alphabet (Push-A to Push-Z) using a self generated curriculum and generated simulation code for the Franka and UR5 robot arms in 3D cross-embodiment simulations with visual feedback. Videos, policies and details will be posted online.
• 中文摘要 Push-T是通过人工演示学习操作策略的标志性基准。机器人必须利用单一接触点将T形方块推入目标姿势。在这篇简短的论文中，我们将在智能机器人新兴进展的背景下重新审视Push-T任务，其中一个LLM编码代理——Claude Code与Fable 5——被要求创建一个无需任何演示数据的算法解决方案。我们研究代理编码循环解决Push-T任务的有效性，并将所得代码作为策略与视觉运动模拟学习策略进行比较。结果表明，该智能体在线找到了2D健身房模拟，并利用模拟实验学习推力机制，迭代优化，以比200次人体演示训练的最佳扩散策略少46%的步骤，实现100%成功率。编码代理还通过自生成的课程和为Franka和UR5机械臂生成的模拟代码，在3D交叉身体模拟中实现从T到完整字母表（Push-A到Push-Z）的扩展，并结合视觉反馈。视频、政策和详细信息将在线发布。

SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

SoftVTBench：一个具备变形感知的视觉触觉数据集及可变形对象操作的基准测试

• Authors: Bowen Jing, Mingxin Wang, Ruiyang Hao, Chenchen Ge, Hanwen Shen, Junjie He, Yang Cui, Yiming Hou, Weitao Zhou, Jiawei Wang, Minglei Li, Dandan Zhang, Ding Zhao, Houde Liu, Xiaofan Li, Si Liu, Ping Luo, Haibao Yu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.18701
• Pdf link: https://arxiv.org/pdf/2608.18701
• Abstract Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the task while allowing slip or causing excessive compression. A primary bottleneck is the absence of visuo-tactile datasets that pair policy-visible contact observations with independent physical ground truth over complete tasks. We introduce SoftVTBench, a visuo-tactile dataset for physical-interaction-aware deformable-object manipulation. It contains 4,000 expert demonstrations and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. At 20 Hz, each episode synchronizes multi-view RGB, dual-finger tactile RGB and marker motion, proprioception, language, and binary and continuous gripper actions, alongside evaluator-only finite-element (FEM) states. Building upon this dataset, we establish a closed-loop benchmark that uses fixed object-specific calibration to define the Deformation-aware Success Rate (DSR), which counts a rollout as successful only when it completes the task and keeps peak normalized deformation within tolerance. Across Diffusion Policy, $\pi_{0.5}$, and FastWAM, all 12 in-distribution configurations contain successful rollouts that violate the deformation tolerance, accounting for 0.7--24% of each configuration's successes. Under distribution shift, visuo-tactile variants achieve higher task success in all six policy--suite comparisons and higher DSR in five, whereas their in-distribution benefits are mixed. These results show that making touch available does not by itself ensure effective multimodal fusion. SoftVTBench therefore provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.
• 中文摘要 物理交互质量是可变形对象操作的核心，但大多数基准测试仅评估任务成功率。保单可能在允许打滑或造成过度压缩的情况下完成任务。主要瓶颈是缺乏将政策可见接触观测与独立物理地面真实匹配的视觉触觉数据集。我们介绍SoftVTBench，一个用于物理交互感知可变形对象操作的视觉触觉数据集。它包含4000个专家演示和50多个资产，包括体积可变形物体和视觉匹配的刚性双胞胎。在20 Hz频率下，每集同步多视角RGB、双指触觉RGB和标记运动、本体感觉、语言以及二元和连续抓握器动作，同时还支持仅评估者有限元（FEM）状态。基于该数据集，我们建立了闭环基准，利用固定的对象特定校准定义变形感知成功率（DSR），该基准仅在完成任务且峰值归一化变形在容差范围内时才算作成功。在扩散政策、$\pi_{0.5}$和FastWAM中，所有12种分布内配置均包含违反变形容差的成功推广，占每个配置成功率的0.7%至24%。在分布转移中，视觉触觉变体在六种策略套件比较中任务成功率更高，五项DSR较高，而其分布内收益则参差不齐。这些结果表明，仅凭触觉的可用性并不保证多模融合的有效性。因此，SoftVTBench 提供了一个通用的视觉-触觉资源，不仅用于研究策略是否成功，还能研究其如何与可变形物体物理互动，以及何时触碰能改善这种交互。