生成时间: 2026-09-02 20:38:06 (UTC+8); Arxiv 发布时间: 2026-09-02 20:00 EDT (2026-09-03 08:00 UTC+8)

今天共有 41 篇相关文章

Keyword: reinforcement learning

Elite-Weighted Supervised Fine-tuning for Goal-Directed Molecular Optimization

精英加权监督微调以实现目标导向分子优化

• Authors: Shiyun Wa, Yifei Wang, Anna G. Green, Simone Sciabola, Ye Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.00189
• Pdf link: https://arxiv.org/pdf/2609.00189
• Abstract Goal-directed optimization is essential for steering molecular generators to propose candidates with desired properties. However, it is often implemented with policy-gradient reinforcement learning, which requires a generation-trajectory log-probability whose form depends on the model architecture and generation procedure. This makes an optimizer difficult to reuse across architectures and conditional generative designs. Supervised fine-tuning needs none of that machinery, but its update is driven by a fixed dataset, so the reward never enters the update. We introduce Elite-Weighted Supervised Fine-tuning (EW-SFT), which uses reward to guide elite selection of high-scoring molecules, and updates the model by its own pretraining loss on that set. Ablations show that reward information is passed primarily through elite selection, rather than through continuous weighting within the selected set. Because the update consumes only scored molecules and the model's native loss, the same rule applies across autoregressive, masked-diffusion, and discrete-flow generators, and across de novo, motif-extension, and linker-design tasks. Under a fixed budget of 3D shape alignment oracle calls on two kinase reference compounds, EW-SFT consistently outperforms the corresponding native optimizers. It further improves goal-directed optimization under a 2D similarity oracle on four held-out references and achieves comparable performance on a sample-efficiency benchmark without a trajectory-level RL formulation. These results demonstrate that EW-SFT is a unified and effective optimizer across molecular generators, design constraints, references, and oracles.
• 中文摘要 目标导向优化对于引导分子生成器提出具有期望性质的候选对象至关重要。然而，它通常通过策略梯度强化学习实现，该学习需要生成轨迹对数概率，其形式取决于模型架构和生成过程。这使得优化器难以跨架构和条件生成设计重复使用。监督微调不需要这些机制，但其更新由固定数据集驱动，因此奖励从未进入更新。我们引入精英加权监督微调（EW-SFT），利用奖励引导精英选择高分分子，并通过自身对该集合的预训练损失更新模型。消融显示奖励信息主要通过精英选择传递，而非在所选集合内连续加权。由于更新仅消耗已评分分子和模型的原生损失，同样的规则适用于自回归、掩蔽扩散和离散流生成器，以及新生、基序扩展和连接设计任务。在两种激酶参考化合物的三维形状比对预言机预算下，EW-SFT持续优于相应的原生优化器。它进一步提升了在四条保留参考文献下的二维相似度预言机下的目标导向优化，并在样本效率基准测试中实现了相当的性能，无需轨迹级强化学习公式化。这些结果表明EW-SFT是跨分子生成器、设计约束、参考和预言机的统一且高效的优化器。

AI Should Not Only Be Helpful. It Should Be Contingent. Artificial Intimacy, Sycophancy, and the Future of Social Learning

人工智能不仅应该是有帮助的。它应该是偶然性的。人工亲密、谄媚与社会学习的未来

• Authors: Scott Compton, Arjun Nagendran
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.00211
• Pdf link: https://arxiv.org/pdf/2609.00211
• Abstract Conversational artificial intelligence is increasingly embedded in everyday social environments, where it functions as both an informational tool and a source of interpersonal feedback. This perspective introduces contingency, i.e., the degree to which system responses vary with user behavior and its interpersonal consequences, as a central construct for evaluating AI systems. We argue that current alignment approaches, including reinforcement learning from human feedback, tend to prioritize user approval and conversational fluency over behaviorally informative feedback, leading to sycophantic patterns of noncontingent affirmation. Drawing on behavioral science and social learning theory, we propose that contingent feedback is a key mechanism through which individuals develop interpersonal skills. When AI systems provide feedback weakly coupled to social consequences, they may reduce opportunities for adaptive calibration in real-world interactions, particularly during adolescence, a critical period for social development. We outline a framework for contingent AI, including trajectory-based evaluation and models of social consequence prediction, and propose a research agenda spanning developmental psychology, human-AI interaction, and machine learning. More broadly, we argue that AI systems should be evaluated not only by user satisfaction, but by their impact on human social learning.
• 中文摘要 对话式人工智能越来越多地嵌入日常社交环境中，既作为信息工具，也作为人际反馈的来源。这一观点引入了偶然性，即系统响应随用户行为及其人际后果变化的程度，作为评估人工智能系统的核心构念。我们认为，当前的对齐方法，包括从人类反馈中进行强化学习，往往优先考虑用户认可和会话流畅度，而非行为信息反馈，导致非偶然肯定的谄媚模式。借鉴行为科学和社会学习理论，我们提出条件反馈是个体发展人际技能的关键机制。当人工智能系统提供与社会后果耦合较弱的反馈时，可能会减少现实世界互动中适应性校准的机会，尤其是在青少年阶段，社会发展的关键阶段。我们概述了条件人工智能的框架，包括基于轨迹的评估和社会后果预测模型，并提出涵盖发展心理学、人机交互和机器学习的研究议程。更广泛地说，我们主张人工智能系统应不仅以用户满意度评估，还应考虑其对人类社会学习的影响。

Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning

发现并缓解多奖励强化学习中由聚合诱导的奖励黑客行为

• Authors: Yu Yuan, Yaoyou Fan, Lili Zhao, Guangting Zheng, Kai Zhang, Lu Pan, Ke Zeng, Qi Liu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.00213
• Pdf link: https://arxiv.org/pdf/2609.00213
• Abstract Reinforcement learning fine-tuning of large language models increasingly adopts multiple reward dimensions, including verifiable rules, task-specific evaluators, and learned reward models, to provide richer supervision across diverse capabilities. These dimensions are commonly scalarized with fixed aggregation weights. We identify a failure mode in which aggregation itself induces reward hacking: static projection aliases qualitatively different reward profiles into a single scalar, steering optimization toward whichever dimensions are easiest, densest, or systematically favored by the reward signal. Over training, this traps the policy in suboptimal profiles and prevents convergence to better-balanced ones that would yield higher task performance. To address this, we propose Adaptive Multi-Reward Projection (AMRP), a lightweight online method that reallocates aggregation weights using three signals, relative shortfall, reward volatility, and recent progress, increasing pressure on lagging, unstable, or stagnant dimensions while relieving saturated ones. Across structured reasoning, citation-grounded generation, and open-ended alignment under GRPO, AMRP consistently improves reward-profile balance and downstream performance over fixed and dynamic weighting baselines; it also remains effective with GDPO and PPO, supporting compatibility across RL algorithms. Our code is available at this https URL.
• 中文摘要 强化学习对大型语言模型的微调越来越多地采用多个奖励维度，包括可验证的规则、任务特定评估器和学习后的奖励模型，以在不同能力之间提供更丰富的监督。这些维度通常通过固定的聚合权重进行标量化。我们识别出一种失败模式，即聚合本身会诱导奖励黑客行为：静态投影将质的不同奖励组合合并为单一标量，将优化引导至奖励信号最容易、最密集或系统性偏好的维度。过度训练时，策略被困在次优配置中，阻止趋同于更平衡的方案，从而实现更高任务表现。为此，我们提出了自适应多重奖励预测（AMRP），这是一种轻量级在线方法，利用三个信号——相对短缺、奖励波动性和近期进展——重新分配聚合权重，增加对滞后、不稳定或停滞维度的压力，同时缓解饱和维度。在结构化推理、引用基础生成和开放式对齐GRPO下，AMRP持续提升奖励组合平衡和下游性能，优于固定和动态加权基线;它在GDPO和PPO中依然有效，支持跨强化学习算法的兼容性。我们的代码可在此 https URL 获取。

WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimization

WiSDoM：配备专家混合的无线稀疏决策变换器，用于多任务移动网络优化

• Authors: Fatih Temiz, Shavbo Salehi, Melike Erol-Kantarci
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.00284
• Pdf link: https://arxiv.org/pdf/2609.00284
• Abstract Emerging 6G wireless networks are expected to operate across diverse deployment scenarios, where variations in network topology, user mobility, traffic demand, and radio conditions challenge the scalability of conventional radio resource management (RRM). While offline reinforcement learning (RL) methods have demonstrated strong decision-making capabilities, learning a single policy that performs consistently across heterogeneous wireless environments remains difficult due to conflicting optimization objectives and limited model specialization. These challenges become particularly pronounced in coordinated multipoint (CoMP) transmission, where selecting the optimal serving-cell combination requires sequential decision-making under evolving network conditions. This paper presents the Wireless Sparse Decision Transformer with Mixture of Experts (WiSDoM), a sparse multi-task offline RL framework for adaptive multi-cell selection. WiSDoM combines Decision Transformers (DTs) with a Mixture-of-Experts (MoE) architecture that dynamically activates specialized experts according to task characteristics. This MoE mechanism improves model capacity without proportionally increasing inference cost, mitigates negative transfer, and enables expert specialization across tasks. WiSDoM is trained jointly on diverse network configurations spanning multiple base station and user equipment densities, mobility levels, and scheduler policies. Experimental results show that WiSDoM consistently outperforms heuristic methods, single-task models, and conventional multi-task DTs, improving quality of experience (QoE) by up to 55% while activating approximately one-third of the parameters of its dense counterpart during inference. Furthermore, WiSDoM exhibits strong task generalization and efficiently adapts to unseen wireless scenarios through few-shot prompting without retraining or fine-tuning.
• 中文摘要 新兴的6G无线网络预计将在多种部署场景中运行，在这些场景中，网络拓扑、用户移动性、流量需求和无线条件的差异挑战了传统无线资源管理（RRM）的可扩展性。尽管离线强化学习（RL）方法已展现出强大的决策能力，但由于优化目标冲突和模型专用有限，学习一个在异构无线环境中持续表现的单一策略仍然困难。这些挑战在协调多点（CoMP）传输中尤为突出，选择最优服务单元组合需要在不断演变的网络条件下进行顺序决策。本文介绍了无线稀疏决策变换器（WiSDoM），这是一个用于自适应多单元选择的稀疏多任务离线强化学习框架。WiSDoM结合了决策变换器（DT）与专家混合架构（MoE），根据任务特性动态激活专业专家。该MoE机制提升模型容量，同时不成比例增加推理成本，减少负转移，并实现跨任务的专家专业化。WiSDoM在跨多个基站和用户设备密度、移动性级别及调度策略的多样化网络配置上联合训练。实验结果显示，WiSDoM持续优于启发式方法、单任务模型和传统多任务DT，提升体验质量（QoE）高达55%，同时在推理过程中激活约三分之一的稠密对应参数。此外，WiSDoM表现出强烈的任务泛化能力，通过少样本提示高效适应未见的无线场景，无需重新训练或微调。

StreamScout: Learning When to Look Deeper for Streaming Video Understanding

StreamScout：学习何时深入寻找流媒体视频理解

• Authors: Ce Zhang, Jing Bi, Jinxi He, Jianshu Zhang, Jingyang Lin, Yunzhong Xiao, Minghao Fu, Yaqi Xie, Zhentao Xie, Weicong Chen, Katia Sycara, Ming Zhou
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.00291
• Pdf link: https://arxiv.org/pdf/2609.00291
• Abstract Streaming video understanding requires answering questions that arrive at arbitrary moments over an unbounded video stream. Existing systems primarily focus on what to retain in a bounded memory, yet access that memory using the same fixed-cost procedure for every query, despite substantial variation in the evidence required. We argue that deciding how deeply to access memory for each query is as important as deciding what the memory should store. To this end, we introduce StreamScout, an adaptive inference framework that maintains only a lightweight textual timeline in context as the stream unfolds. At query time, StreamScout progressively augments the timeline with up to three increasingly informative visual views: a glance at recent frames, a uniform look-back over the past stream, and query-salient retrieval. At each stage, the model answers immediately if the available evidence is sufficient; otherwise, it escalates to the next view. To improve this stop-or-escalate policy, we probe the cascade on an auxiliary set and distill the model's empirical competence boundary into supervision for a lightweight LoRA adaptation, yielding StreamScout-S. We further refine the policy through reinforcement learning, allowing the model to explore stopping behaviors beyond imitation of the distilled decisions, yielding StreamScout-R. Across three backbones and three streaming benchmarks, StreamScout and its variants consistently outperform prior streaming methods while substantially reducing inference cost and token consumption; on OVO-Bench, for instance, StreamScout-S improves Qwen3-VL-8B by 14.65 points while using 59% fewer tokens than uniform sampling and answering in 1.04 s on average.
• 中文摘要 流式视频理解需要回答在无界视频流中任意时刻出现的问题。现有系统主要关注在有界内存中保留什么，但访问内存时仍使用相同的固定成本程序，尽管所需证据存在较大差异。我们认为，决定每个查询访问内存深度与决定内存应存储哪些内容同样重要。为此，我们引入了StreamScout，一种自适应推理框架，在流展开时仅在上下文中保持轻量级文本时间线。在查询时，StreamScout逐步增加多达三个越来越有信息量的视觉视图：最近帧浏览、对过去流的统一回顾和查询显著检索。在每个阶段，模型会立即回答是否证据足够;否则，模型会升级到下一个视图。为改进该停止或升级策略，我们在辅助集上探测级联，并将模型的经验能力边界提炼为轻量级LoRA适配的监督，生成StreamScout-S。我们通过强化学习进一步细化策略，使模型能够探索超越复制决策的停止行为，最终生成StreamScout-R。在三个骨干和三个流基准测试中，StreamScout及其变体持续优于以往流式方法，同时显著降低推理成本和令牌消耗;例如在OVO-Bench上，StreamScout-S在使用标记数比均匀抽样少59%的情况下，平均在1.04秒内提升了Qwen3-VL-8B14.65个点。

Risk-Aware Decision-Making for Autonomous Overtaking: A World Model-Based Mixture-of-Experts Framework

自主超车的风险感知决策：基于全球模型的专家混合框架

• Authors: Yongzhi Liu, Sunan Zhang, Jinchang Xu, Jiawei Wang, Yushu Qiu, Chen Lv, Weichao Zhuang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.00385
• Pdf link: https://arxiv.org/pdf/2609.00385
• Abstract Autonomous highway overtaking demands foresighted decision-making to handle complex interactions, stochastic traffic evolution, and temporal risk accumulation. However, standard safe reinforcement learning approaches typically rely on implicit value-based risk estimations rather than explicit dynamics modeling, thereby struggling to accurately capture complex risk propagation over multi-step horizons. This limitation frequently results in behaviors that are locally safe but induce substantial latent risks in the long term. To address this, a World Model-based Risk-aware Mixture-of-Experts (WM-RMoE) framework is proposed. First, a learned latent dynamics model facilitates parallel multi-step rollouts, elevating safety assessment from the action level to the trajectory level via cumulative risk evaluation. Second, to enhance robustness under varying interaction intensities, a hierarchical gating mechanism dynamically coordinates experts across long-horizon, short-horizon, and rule-based safety modules. Furthermore, a Gaussian Mixture Model is integrated to preserve multimodal maneuvering branches, thereby mitigating the issue of behavioral mode averaging. Experimental results demonstrate that WM-RMoE significantly outperforms representative baselines in terms of safety compliance, decision stability, and generalization capability. Furthermore, benefiting from the risk-aware formulation, the proposed framework uniquely exhibits the ability to generate foresighted and semantically distinct overtaking maneuvers across diverse traffic densities.
• 中文摘要 自动高速公路超车需要前瞻性决策来处理复杂交互、随机交通演变和时间风险积累。然而，标准的安全强化学习方法通常依赖隐性基于价值的风险估计，而非显式动态建模，因此难以准确捕捉多步视野上的复杂风险传播。这种限制常导致行为在局部安全，但长期内诱导出显著潜在风险。为此，提出了基于世界模型的风险感知专家混合（WM-RMoE）框架。首先，学习的潜在动力学模型促进并行多步推广，通过累积风险评估将安全评估从行动层面提升到轨迹层面。其次，为了增强在不同交互强度下的鲁棒性，采用分层门控机制动态协调专家跨长视野、短视野和基于规则的安全模块。此外，集成了高斯混合模型以保留多模态机动分支，从而缓解行为模式平均化的问题。实验结果表明，WM-RMoE在安全合规性、决策稳定性和泛化能力方面显著优于代表性基线。此外，借助风险感知表述，所提框架独特地展现出在不同交通密度中产生前瞻性且语义上独特的超车动作的能力。

DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments

DRLM：基于深度强化学习的边缘环境中的LLM查询编排

• Authors: Reza Farahani, Zoha Azimi Ourimi, Mario Colosi, Lauri Loven, Christian Timmerer, Schahram Dustdar
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.00442
• Pdf link: https://arxiv.org/pdf/2609.00442
• Abstract Large language model (LLM) services increasingly process heterogeneous queries with diverse latency, accuracy, and resource requirements. While edge deployment reduces response time, the heterogeneity of devices and the diversity of model families, parameter scales, and quantization levels make efficient LLM query orchestration challenging. This paper introduces DRLM, a Deep Reinforcement Learning-based LLM query orchestration framework in edge environments. DRLM integrates two lightweight predictors: (i) a class-conditioned quality estimator that maps queries to semantic categories and infers model performance, and (ii) a feature-driven latency predictor that estimates inference time across model-device configurations. These predictions, combined with system state, feed a factorized Proximal Policy Optimization (PPO) agent that performs state-aware orchestration decisions. To enable data-driven orchestration, we construct a large-scale benchmarking dataset with 223 835 measurements spanning 1258 queries, 6 query classes, 8 model families (32 deployed instances), 5 quantization levels, and heterogeneous edge devices. Evaluation on a 64-node edge cluster and comparison with three baselines and two state-of-the-art methods show that DRLM reduces inference latency by up to 51% and queuing delay by up to 67 %, while incurring at most 8% accuracy loss. It improves latency under increasing workloads up to 61.4%, demonstrating robust and stable orchestration.
• 中文摘要 大型语言模型（LLM）服务越来越多地处理具有不同延迟、准确性和资源需求的异构查询。虽然边缘部署减少了响应时间，但设备的异质性以及模型族、参数尺度和量化水平的多样性使得高效的LLM查询编排变得具有挑战性。本文介绍了DRLM，一种基于深度强化学习的边缘环境中LLM查询编排框架。DRLM集成了两个轻量级预测器：（i）一个类条件质量估计器，将查询映射到语义类别并推断模型性能;（ii）一个特征驱动的延迟预测器，估计模型-设备配置间的推理时间。这些预测与系统状态结合，为一个因数化的近端策略优化（PPO）代理提供执行状态感知编排决策的过程。为实现数据驱动编排，我们构建了一个大型基准数据集，包含223,835项测量，涵盖1258次查询、6个查询类、8个模型族（32个部署实例）、5个量化级别和异构边缘设备。对64节点边缘集群的评估以及与三个基线和两种先进方法的比较显示，DRLM将推理延迟降低多达51%，排队延迟最高降低67%，同时最多损失8%的准确性。在工作负载增加时延迟提升至61.4%，展现了稳健稳定的编排能力。

Group Adaptive Clipping Policy Optimization

群自适应裁剪策略优化

• Authors: Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, Rein Houthooft
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.00444
• Pdf link: https://arxiv.org/pdf/2609.00444
• Abstract Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.
• 中文摘要 针对可验证奖励强化学习（RLVR）的组相对策略优化通常在所有推展中使用固定的重要性抽样（IS）比率裁剪边界。我们指出一个关键局限：在较难问题上，正确推展的罕见正确推展和较易问题中大量正确推展的剪辑率相近，尽管它们贡献的学习信号差异很大。组成功率较低的推举显示出更高的IS比，且带有更强的梯度信号以促进探索和解决新问题，但固定裁剪却被不成比例地抑制。为此，我们提出了群自适应剪裁策略优化（GAPO），这是对GRPO方法的插件修改，将剪裁边界调整以适应推广优势。GAPO的动机来自反向KL信任区域视角，即学习信号较大的推展应获得相应更大的更新余量。GAPO不要求奖励塑形，保持标准的PPO/GSPO替代，同时仅调整裁剪阈值。在Qwen和Llama模型中，GAPO在数学推理和编程基准测试中，持续提升Pass@1和Pass@k，优于固定裁剪和优势塑形基线，而基础模型的通过率相对较低。

Can LLMs Use Relational Transformer Embeddings?

LLMs 可以使用关系型变换器嵌入吗？

• Authors: Francisco Galuppo Azevedo, Clarissa Lima Loures
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.00457
• Pdf link: https://arxiv.org/pdf/2609.00457
• Abstract Injecting frozen relational-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy: the encoder handles multi-table structure, the LLM handles language and reasoning, and no lossy text serialization is required. We test this hypothesis concretely by injecting embeddings from a frozen Relational Transformer (RT) into Qwen3.5-4B via a learned MLP projection and LoRA adaptation, trained first with supervised fine-tuning (SFT) on chain-of-thought reasoning traces and then with group-based reinforcement learning (GSPO). We evaluate across 10 binary classification tasks on 6 relational databases from RelBench, under four supervision regimes: single-task (ST), within-dataset (WD), cross-dataset (CD), and all-task (ALL). The hybrid model does not consistently outperform standalone RT: it is frequently below random, highly sensitive to serialization format and relational-token budget, and unstable under RL training. We report these negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.
• 中文摘要 将冻结的关系编码器嵌入作为软代币注入大型语言模型（LLM）是一种概念上有吸引力的融合策略：编码器处理多表结构，LLM处理语言和推理，且无需有损文本序列化。我们通过通过学习的MLP投影和LoRA适配，将冻结关系转换器（RT）嵌入注入Qwen3.5-4B，先用监督微调（SFT）训练链式思维推理轨迹，再用基于群体的强化学习（GSPO）进行测试。我们在RelBench的6个关系数据库上评估了10个二进制分类任务，采用四个监督模式：单任务（ST）、数据集内（WD）、跨数据集（CD）和全任务（ALL）。混合模型的表现并不总是优于独立的RT：它经常低于随机，对序列化格式和关系令牌预算高度敏感，且在强化学习训练下不稳定。我们报告了这些负面结果并分析了失败模式，认为软令牌融合需要更强的对齐目标和模式感知设计，才能成为可靠的关系预测路径。

GlitchLab: A Hardware-in-the-Loop Optimizer for Physical Fault Injection

GlitchLab：物理故障注入的硬件在环优化器

• Authors: Tanvir Hossain, Abhinav Mahadevan, Jasper Van Woudenberg, Rajesh Velegalati, Arindam Bhattacharyya
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2609.00502
• Pdf link: https://arxiv.org/pdf/2609.00502
• Abstract Physical fault injection can turn brief hardware disturbances into security failures such as key recovery, authentication bypass, and unintended control flow. Finding effective faults is difficult because many interacting parameters create a large search space, successful settings are sparse and target-dependent, and each hardware attempt provides limited feedback. Under fixed testing time, efficient search is therefore critical for assessing fault sensitivity. We present GlitchLab, an online hardware-in-the-loop platform that treats delay as a timing gate, voltage and pulse duration as severity controls, and hardware outcomes as structured feedback. It implements RL-Q (Q-learning-based reinforcement learning), a structured bandit for discovery, and Structured-Outcome-Based Adaptive Search (SOBAS), a model-based policy for fault reproduction. Both policies find a target fault in every AES, password, and control-flow campaign. On AES and control flow, they require 2-85x fewer attempts and 26-1,237x less time than the baselines; on password, both succeed while the baselines fail within 5,000 attempts. After discovery, SOBAS reproduces faults 7.3-21x more often, while RL-Q identifies 30% more distinct AES settings.
• 中文摘要 物理故障注入可以将短暂的硬件干扰转化为安全故障，如密钥恢复、认证绕过和意外控制流。发现有效故障困难，因为许多相互作用参数形成庞大的搜索空间，成功的设置稀疏且依赖目标，且每次硬件尝试反馈有限。在固定测试时间内，高效搜索对于评估故障敏感性至关重要。我们介绍GlitchLab，一个在线硬件在环平台，将延迟视为定时门，电压和脉冲持续时间作为严重度控制，硬件结果作为结构化反馈。它实现了RL-Q（基于Q学习的强化学习），一种结构化的发现工具，以及基于模型的基于结果的自适应搜索（SOBAS），用于故障重现的策略。这两种策略都能在每个AES、密码和控制流活动中发现目标故障。在AES和控制流中，它们所需的尝试次数是基线的2-85倍，时间是基线的26-1,237倍;在密码系统中，两者均成功，而基线在5,000次尝试内失败。发现后，SOBAS重现错误的频率是7.3-21倍，而RL-Q识别出的AES不同设置则多30%。

GeoPAR: Large-Scale Multi-Agent Combinatorial Optimization with Geometry-Guided Parallel Autoregressive Learning

GeoPAR：基于几何引导并行自回归学习的大规模多智能体组合优化

• Authors: Wenjian Wu, Zesheng Jia, Jiaying Tang, Benyuan Yang, Jin Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2609.00577
• Pdf link: https://arxiv.org/pdf/2609.00577
• Abstract Multi-agent combinatorial optimization problems are notoriously challenging due to their NP-hard nature. Recent parallel autoregressive neural solvers improve inference efficiency by allowing agents to make decisions simultaneously, but their performance often degrades on large-scale instances. This is largely attributable to weak modeling of local geometric structures and the fact that conflicting task selections are handled only after action generation. To address these limitations, we propose GeoPAR, a geometry-guided parallel autoregressive reinforcement learning framework for scalable multi-agent combinatorial optimization. GeoPAR integrates three key components: (1) a projection-window sparse geometry mechanism that builds lightweight local candidate neighborhoods through multi-directional projections, (2) sparse edge-biased attention that injects these geometric relations into node representations, and (3) cache-guided conflict-aware assignment that reuses the geometric cache during decoding to suppress duplicate selections of exclusive tasks. Experiments on heterogeneous vehicle routing and open multi-depot pickup-and-delivery problems show that GeoPAR improves large-scale zero-shot generalization while substantially reducing rollout steps and maintaining efficient inference.
• 中文摘要 多智能体组合优化问题因其NP难性质而极具挑战性。近期的并行自回归神经求解器通过允许智能体同时做决策提升了推理效率，但在大规模实例中性能常常下降。这主要归因于局部几何结构建模薄弱，以及冲突任务选择仅在动作生成后处理。为解决这些限制，我们提出了GeoPAR，一种几何引导的并行自回归强化学习框架，用于可扩展多智能体组合优化。GeoPAR集成了三个关键组件：（1）投影窗口稀疏几何机制，通过多方向投影构建轻量级局部候选邻域;（2）稀疏边偏关注，将这些几何关系注入节点表示;（3）缓存引导的冲突感知分配，在解码时重用几何缓存以抑制排他任务的重复选择。对异构车辆路由和开放多仓库取货与交付问题的实验表明，GeoPAR提升了大规模零样本泛化，同时大幅减少了推进步骤并保持了高效的推断。

It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learning

需要两个人才能匹配：共同进化的生成式寻回犬与强化学习

• Authors: Runpeng Dai, Kaili Huang, Changsung Kang, Ciya Liao
• Subjects: Subjects: Information Retrieval (cs.IR); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.00638
• Pdf link: https://arxiv.org/pdf/2609.00638
• Abstract Retrieval is the first stage of modern search and advertising systems, selecting a candidate set from a large item universe for downstream ranking and auction. Recent work increasingly leverages LLMs to improve retrieval through query expansion, data synthesis, and retrieval-feedback training. However, the generative component is typically used for query-side augmentation, while final matching is still delegated to a downstream retriever. We introduce CoGR, a retrieval framework that instead trains LLMs to directly construct retrieval representations on both query and item sides. Each generator produces a compact set of keywords, which are matched directly through an inverted index, preserving compatibility with existing keyword-based retrieval infrastructure. CoGR uses a two-stage training pipeline. Supervised fine-tuning first establishes an aligned keyword space, after which co-evolving reinforcement learning alternately optimizes the query- and item-side generators with GRPO against the opposite side's frozen index. Both sides optimize the same query-to-item retrieval $F_1$ objective: the query side receives retrieval $F_1$ directly, while the item side receives a counterfactual marginal reward measuring the change in query-side $F_1$ caused by its generated keywords. Across 10 representative sparse, dense, and generative baselines, CoGR achieves the best performance on both an internal APP Marketplace dataset and the public WANDS benchmark, improving $F_1$ over the strongest baseline by $10.9\%$ and $36.1\%$, respectively. Further analysis shows stable co-evolution and increasingly aligned query--item keyword spaces over training.
• 中文摘要 检索是现代搜索和广告系统的第一阶段，从庞大的项目宇宙中选择候选集进行下游排名和拍卖。近期研究越来越多地利用LLM通过查询扩展、数据综合和检索反馈训练来提升检索能力。然而，生成组件通常用于查询端的增强，而最终匹配仍委托给下游检索器。我们介绍CoGR这一检索框架，训练LLM直接构建查询和项目双方的检索表示。每个生成器生成一组紧凑的关键词，通过倒置索引直接匹配，保持与现有关键词检索基础设施的兼容性。CoGR采用两阶段训练流水线。监督微调先建立对齐的关键词空间，随后共演化强化学习交替优化查询端和条目端生成器，利用GRPO对方冻结的索引进行优化。双方都优化相同的查询到项检索 $F_1$目标：查询方直接获得检索$F_1$，而条目方则获得一个反事实的边际奖励，衡量其生成的关键词引起的查询端$F_1$变化。在10个代表性的稀疏、密集和生成基线中，CoGR在内部APP市场数据集和公开WANDS基准测试上均表现最佳，较最强基线提升$F 1$10。 分别为9\%$和$36.1\%$。进一步分析显示，查询-项目关键词空间在训练中保持稳定的共同演化和日益一致。

ADAPT: Agile Diffusion Action Priors for Robust and Steerable Online Text-Driven Humanoid Control

ADAPT：敏捷扩散行动先验，实现稳健且可引导的在线文本驱动人形控制

• Authors: Yan Wu, Chenhao Li, Kaifeng Zhao, Gen Li, Marco Hutter, Siyu Tang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.00677
• Pdf link: https://arxiv.org/pdf/2609.00677
• Abstract We present ADAPT, an end-to-end framework for interactive, text-conditioned humanoid whole-body control. Unlike dominant text-to-motion pipelines that generate kinematic motions for a separate tracker, ADAPT solves language control with an end-to-end closed-loop control framework, where the robot must continuously respond to changing commands while maintaining balance, natural motion, and smooth transitions. ADAPT learns a diffusion-based action prior from text-labeled humanoid state-action trajectories, enabling diverse motion skills to be directly executed from language commands. To improve long-horizon robustness and smooth prompt switching, we train a lightweight residual reinforcement learning policy on top of the frozen diffusion controller. We further show that the same diffusion policy can be reused as a steerable text-conditioned motion prior for downstream task adaptation. Experiments demonstrate robust language-grounded skill execution, smooth interactive transitions, and style-preserving downstream control.
• 中文摘要 我们介绍了ADAPT，一个端到端的交互式文本条件人形全身控制框架。与主流的文本转动作流程生成独立追踪器的运动学运动不同，ADAPT通过端到端闭环控制框架解决语言控制问题，机器人必须在保持平衡、自然运动和平滑过渡的同时持续响应变化指令。ADAPT从文本标记的人形状态-动作轨迹中学习基于扩散的动作，使语言指令能够直接执行多样化的运动技能。为提升长视野的稳健性和平滑的提示切换，我们在冻结扩散控制器之上训练了轻量级残余强化学习策略。我们还进一步证明，同一扩散策略可作为可引导的文本条件运动先行重复使用，用于后续任务适应。实验展示了扎实的语言基础技能执行、流畅的互动过渡以及保持风格的下游控制。

Online Self-Weighted Fine-Tuning

在线自加权微调

• Authors: Haiquan Wen, Yiwei He, Bei Peng, Guangliang Cheng
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.00734
• Pdf link: https://arxiv.org/pdf/2609.00734
• Abstract Standard supervised fine-tuning (SFT) assigns the same explicit loss weight to every expert demonstration, regardless of the model's changing competence over training queries. Reinforcement learning (RL) based methods adapt update strength using model-generated rollouts, but often require substantially more sampling and can be unstable on hard tasks. We propose \textbf{Online Self-Weighted Fine-Tuning (OSW-FT)}, a simple method that augments SFT with online, trajectory-level weighting. For each query, OSW-FT estimates the model's current success rate using a small number of inference-only rollouts and rescales the standard SFT loss accordingly. The optimization direction remains anchored to the expert trajectory, while the update magnitude adapts online. For binary-verifiable reasoning, we connect this weighting to SFT and RL at the gradient level, inspired by variance-reduction principles. The resulting estimator is unbiased for the exact OSW-FT surrogate update for any finite rollout count, and we analyze convergence with respect to the corresponding surrogate objective. Evaluated across Qwen3 series ranging from 0.6B to 4B on multiple challenging benchmarks (e.g., AIME), OSW-FT consistently improves over SFT on small-to-medium scale models. OSW-FT offers a favorable compute-performance trade-off as a practical approach for fine-tuning small-to-medium LLMs on binary-verifiable reasoning tasks with only \textbf{2 online rollouts}.
• 中文摘要 标准监督微调（SFT）为每个专家演示赋予相同的显式损耗权重，无论模型在训练查询中的能力如何变化。基于强化学习（RL）的方法通过模型生成的展开调整更新强度，但通常需要大量采样，且在困难任务中可能不稳定。我们提出了\textbf{在线自加权微调（OSW-FT）}，这是一种简单方法，通过在线轨迹级权重增强SFT。对每个查询，OSW-FT通过少量仅推断的展开估计模型当前成功率，并相应重新调整标准SFT损失。优化方向仍锚定于专家轨迹，而更新幅度则在线调整。对于二元可验证推理，我们将此权重与梯度层面的SFT和RL联系起来，灵感来源于方差减少原则。所得估计量对任意有限扩展计数的精确OSW-FT替代更新均无偏，并分析对应替代目标的收敛性。在Qwen3系列中，从0.6B到4B，在多个具有挑战性基准测试（如AIME）上进行评估，OSW-FT在小至中等规模模型中持续优于SFT。OSW-FT作为微调中小型至中型LLM在二进制可验证推理任务中仅有\textbf{2次在线部署}的实用方法，提供了有利的计算性能权衡。

Non-Prehensile Throwing: A Reinforcement Learning Perspective

非抓握投掷：强化学习视角

• Authors: Abdullah Mustafa, Ryo Hanai, Ixchel G. Ramirez-Alpizar, Floris Erich, Ryoichi Nakajo, Yukiyasu Domae, Tetsuya Ogata
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.00771
• Pdf link: https://arxiv.org/pdf/2609.00771
• Abstract Robotic throwing enables fast object transport and extends a robot's reachable workspace beyond traditional pick-and-place. While prehensile (grasp-based) throwing works well for graspable items, non-prehensile (grasp-free) throwing is better suited for large, heavy, and/or deformable objects. Existing approaches rely on model-based optimization with simplified contact models (e.g., dynamic grasping) and low-dimensional trajectory parameterizations, which limit solution quality and reachable workspace. We propose a reinforcement learning approach that additionally leverages sliding and rolling contact modes and directly optimizes joint-space trajectories without analytical contact models or custom parameterizations. The Markov Decision Process (MDP) is formulated as a dynamical system that evolves the robot's joint state conditioned on the throwing target, object model, and initial configuration. Joint-jerk trajectories are planned offline at a low control rate and upsampled into smooth, high-rate velocity commands for deployment. For sim-to-real transfer, we minimize the robot-dynamics gap through minimum-jerk system identification and train uncertainty-aware policies to mitigate object-modeling errors, particularly sensitivity to dynamic friction. In simulation, the policy achieves 99% success across thousands of configurations and generalizes to unseen objects. Sensitivity analysis shows robustness to mass uncertainty but high sensitivity to dynamic friction, consistent with the sliding-based release mechanism. Deployed zero-shot on a UR5e operating near its physical limits (5 m/s end-effector velocity), our method throws diverse objects including heavy (790 g) and large (20x20x28 cm) items to targets up to 350 cm distance or 180 cm elevation, achieving a 97% real-world success rate.
• 中文摘要 机器人投掷实现了快速的物体运输，并将机器人可达的工作空间扩展到超越传统的拾取放置。虽然抓握式投掷对可抓取物品效果良好，但非抓握（无抓握）投掷更适合大型、重物和/或可变形物体。现有方法依赖基于模型的优化，采用简化的接触模型（如动态抓取）和低维轨迹参数化，这限制了解决方案质量和可达工作空间。我们提出了一种强化学习方法，额外利用滑动和滚动接触模式，直接优化关节空间轨迹，无需分析接触模型或自定义参数化。马尔可夫决策过程（MDP）被定义为一个动态系统，根据投掷目标、物体模型和初始配置演化机器人的关节状态。联合拉力轨迹在离线时以低控制率规划，并被上采样为平滑、高速命令以供部署。对于模拟到实物的传输，我们通过最小抖动系统识别最小化机器人动力学差距，并训练不确定性感知策略以减轻物体建模错误，特别是对动态摩擦的敏感性。在模拟中，该策略在数千种配置中成功率达99%，并推广到看不见的物体。灵敏度分析显示对质量不确定性鲁棒，但对动态摩擦高度敏感，这与基于滑动的释放机制一致。在接近物理极限（5米/秒末执行器速度）的UR5e上部署零发射，我们的方法能将各种物体投掷到最大350厘米距离或180厘米仰角的目标上，包括重物（790克）和大型物体（20x20x28厘米），实现97%的实际成功率。

Instella-MoE Technical Report

Instella-MoE 技术报告

• Authors: Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Zicheng Liu, Emad Barsoum
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.00791
• Pdf link: https://arxiv.org/pdf/2609.00791
• Abstract In this work, we introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. Instella-MoE combines a sparsely activated MoE design with architectural and system-level innovations, including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity, enabling efficient large-scale training and inference. The model is developed through a multi-stage pipeline comprising pre-training, mid-training, long-context extension, supervised fine-tuning with feedback-driven data curation, direct preference optimization, and reinforcement learning with Multi-Teacher On-Policy Distillation. Instella-MoE achieves an average score of 76.7 across standard pre-training benchmarks, outperforming prior fully open models including OLMo-3-7B, SmolLM3-3B, and OLMoE-1B-7B, while remaining competitive with open-weight MoE and dense baselines at comparable active-parameter scales, including Moonlight-16B-A3B and Qwen3.5-4B. After post-training, our final Think checkpoint achieves an average score of 73.2 across instruction-following, reasoning, math, coding, and chat benchmarks, outperforming both fully open and open-weight models with comparable or larger active parameter counts in our evaluation. To support transparent and reproducible research, we release the complete Instella-MoE model flow, including model weights, training configurations, data mixtures, and training code. Together, these contributions establish Instella-MoE a strong, fully open foundation for efficient, high-performing MoE models and reproducible research.
• 中文摘要 本研究介绍了Instella-MoE，一个全开放的专家混合（MoE）语言模型，拥有160亿总参数和每个代币28亿活跃参数，完全在AMD Instinct MI300X和MI325X GPU上从零训练。Instella-MoE结合了稀疏激活的MoE设计与架构和系统层创新，包括门控多头潜在注意力（Gated MLA）和FarSkip-Collective连接，实现高效的大规模训练和推断。该模型通过多阶段流程开发，包括预训练、中期训练、长上下文扩展、监督微调（反馈驱动数据管理）、直接偏好优化以及多教师策略蒸馏的强化学习。Instella-MoE在标准预训练基准中平均得分为76.7，优于之前的全开放模型，包括OLMo-3-7B、SmolLM3-3B和OLMoE-1B-7B，同时在类似主动参数尺度（如Moonlight-16B-A3B和Qwen3.5-4B）中，保持开放权重MoE和密集基线的竞争力。经过后期训练后，我们的最终Think检查点在指令跟随、推理、数学、编码和聊天基准测试中平均得分为73.2，优于活跃参数数量相当或更大的全开放和开放权重模型。为支持透明且可重复的研究，我们发布完整的Instella-MoE模型流程，包括模型权重、训练配置、数据混合和训练代码。这些贡献共同奠定了Instella-MoE一个坚实且完全开放的基础，用于高效、高效能的MoE模型和可重复性研究。

SFAD: Speculative Factuality-Aware Decoding

SFAD：推测事实感知解码

• Authors: Guanqiao Chen, Di Wang, Lijie Hu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.00796
• Pdf link: https://arxiv.org/pdf/2609.00796
• Abstract As one of the most critical challenges in large language models, contextual faithfulness directly determines their reliability in knowledge-intensive applications. This task is particularly challenging as it requires balancing factual consistency with generation efficiency. Contrastive decoding methods require dual forward passes (with and without context) to compare model outputs, doubling inference computational overhead, while post-training alignment demands extensive reinforcement learning with substantial computational overhead. To address this challenge, we present \textbf{SFAD}, a speculative decoding framework that enhances contextual faithfulness without inference degradation. We first construct \textbf{ConFide}, a preference dataset with fine-grained atomic perturbations, to train a context-faithful draft model via Direct Preference Optimization. During inference, Epistemic Friction detects potential hallucinations by quantifying distributional tension weighted by specialist certainty. When friction exceeds the threshold, Asymmetric Logit Steering refines the target distribution through residual-based logit injection; otherwise, standard speculation proceeds. Extensive experiments demonstrate that SFAD substantially improves faithfulness while achieving $2.48\times$ speedup, offering a practical solution for efficient LLMs.
• 中文摘要 作为大型语言模型中最关键的挑战之一，上下文忠实性直接决定了其在知识密集型应用中的可靠性。这一任务尤其具有挑战性，因为它需要在事实一致性与生成效率之间取得平衡。对比解码方法需要双重前向传递（有上下文和无上下文）来比较模型输出，导致推理计算开销翻倍;而训练后对齐则需要大量强化学习，计算开销巨大。为应对这一挑战，我们提出了 \textbf{SFAD}，一种推测性解码框架，能够增强上下文忠实性而不影响推理。我们首先构建 \textbf{ConFide}，这是一个具有细粒度原子扰动的偏好数据集，通过直接偏好优化训练一个上下文忠实的草稿模型。在推理过程中，Epistemic Friction 通过量化由专业确定性加权的分布张力来检测潜在的幻觉。当摩擦超过阈值时，非对称Logit引导通过基于残差的logit注入细化目标分布;否则，标准推测继续进行。大量实验表明，SFAD在实现2.48倍加速的同时显著提升了忠实度，为高效的LLMs提供了切实可行的解决方案。

One Policy, Any Budget: Internalizing Budget-Aware Search via Reinforcement Learning

一项政策，任何预算：通过强化学习内化预算感知搜索

• Authors: Xiaowei Sun, Jin Li, Yili Hong, Yikun Fu, Yanghua Xiao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.00813
• Pdf link: https://arxiv.org/pdf/2609.00813
• Abstract While reinforcement learning has enabled LLM-based search agents to invoke external tools, existing methods train under fixed budgets and cannot adapt when constraints vary at deployment. We propose AnySearch, a framework that enables a single policy to perform budget-aware search under any budget constraint through a training scaffold and curriculum reinforcement learning. In the first phase, we train the agent with explicit budget state injection and structured reasoning prompts that guide efficient allocation under linearly decaying budgets. In the second phase, the scaffold is removed and the agent learns to operate autonomously under adaptively sampled budget constraints, matching inference conditions. Both phases are optimized with a composite reward that couples answer accuracy with budget efficiency through absolute and relative signals, where an adaptive weight amplifies the efficiency signal for high-accuracy queries and attenuates it for low-accuracy ones. Extensive experiments on seven general and multi-hop QA benchmarks show that our method outperforms baselines across all budget scales, generalizes to unseen constraints beyond the training range, and achieves superior tool productivity without excessive token overhead. Our code is available at this https URL.
• 中文摘要 虽然强化学习使基于LLM的搜索代理能够调用外部工具，但现有方法只能在固定预算下训练，无法在部署时约束变化时进行适应。我们提出了AnySearch框架，该框架通过训练支架和课程强化学习，使单一策略能够在任何预算约束下执行预算感知搜索。第一阶段，我们通过显式预算状态注入和结构化推理提示训练代理，指导线性衰减预算下的高效分配。第二阶段，支架被移除，代理学习在自适应采样预算约束下自主运行，匹配推理条件。这两个阶段均采用复合奖励进行优化，通过绝对和相对信号将答案准确性与预算效率结合，其中自适应权重在高准确率查询中放大效率信号，对低精度查询则减弱。对七个通用和多跳质量保证基准的广泛实验表明，我们的方法在所有预算尺度上都优于基线，能够推广到训练范围之外的未知约束，并且在不增加代币开销的情况下实现了更优的工具生产力。我们的代码可在此 https 网址获取。

Dense Process Supervision for Search Agents via Fact Utility Estimation

通过事实效用估计实现搜索代理的密集过程监督

• Authors: Rongzhi Zhu, Xiangyu Liu, Yi Liu, Shuo Zhang, Ruirui Zhang, Rui Wu, Tao Jiang, Zequn Sun, Wenhao Xu, Wei Hu
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.00833
• Pdf link: https://arxiv.org/pdf/2609.00833
• Abstract Reinforcement learning (RL) for search agents typically relies on outcome rewards. However, it often fails to achieve effective credit assignment, due to the unclear value of intermediate steps. It is hard to separate their contributions from the final result. In this paper, we propose a dense process supervision method based on fact utility estimation, which models the reasoning process as the accumulation of discrete evidence facts. We first extract structured facts from raw observations and organize them into an explicit fact store. To support credit assignment, we then cluster semantically equivalent facts and infer the posterior utility of each fact cluster using Bayesian estimation over group rollouts. Finally, we convert the estimated fact utilities into dense step-level rewards to guide RL training. Experiments on seven single-hop and multi-hop QA benchmarks show that our method consistently outperforms existing baselines. Ablation studies validate clear relative improvements on multi-hop QA compared to outcome reward-only training.
• 中文摘要 搜索代理的强化学习（RL）通常依赖于结果奖励。然而，由于中间步骤的价值不明确，它常常无法实现有效的功劳分配。很难将它们的贡献与最终结果区分开来。本文提出了一种基于事实效用估计的密集过程监督方法，将推理过程建模为离散证据事实的累积。我们首先从原始观察中提取结构化事实，并将其组织成显式事实库。为支持功劳分配，我们随后将语义等效的事实聚类，并利用贝叶斯估计推断每个事实集群对组推广的后验效用。最后，我们将估计的事实效用转换为密集的步骤级奖励，以指导强化学习训练。对七个单跳和多跳质量保证基准的实验表明，我们的方法持续优于现有基线。消融研究验证了多跳QA相比仅奖励训练的明显相对改善。

Reinforcement Learning Enhanced LLM Agents for Complex Vehicle Routing Problems

强化学习增强型大型语言模型代理，用于复杂车辆路由问题

• Authors: Yi Chen, Zikang Yu, Jiahai Wang, Jinbiao Chen, Jianpeng Zhou, Zizhen Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.00859
• Pdf link: https://arxiv.org/pdf/2609.00859
• Abstract Vehicle Routing Problems (VRPs) are fundamental combinatorial optimization problems with widespread applications in various scenarios. The advanced optimization solvers can effectively solve such problems. However, modeling complex VRP variants for solvers often requires substantial domain expertise, which limits the accessibility of advanced optimization technologies. In this paper, we propose Reinforcement Learning Enhanced LLMAgents(RLEA), a multi-agent framework designed to automate the modeling of complex VRPs. RLEA introduces a lightweight neural Planner trained with Soft Q-learning to efficiently orchestrate the actions of LLM-based agents. In addition, we equip the system with an evolutionary memory module and retrieval-augmented generation, enabling the agent to leverage both accumulated experience and external solver knowledge during program generation and refinement for solving VRPs. We evaluated 48 distinct VRP variants across various solvers. The experimental results demonstrate that RLEA outperforms the previous state-of-the-ar method, achieving a 16.67% higher success rate while significantly reducing runtime errors. These results validate that integrating reinforcement learning with LLM-based reasoning is highly effective for automated optimization modeling. The appendix is available at: this https URL.
• 中文摘要 车辆路由问题（VRPs）是基础的组合优化问题，在各种场景中广泛应用。高级优化求解器能够有效解决此类问题。然而，为求解器建模复杂VRP变体通常需要较高的领域专业知识，这限制了先进优化技术的可及性。本文提出了强化学习增强型LLMAgents（RLEA），这是一个多智能体框架，旨在自动化复杂VRP建模。RLEA引入了一款轻量级神经规划器，通过软Q学习训练，高效协调基于LLM的智能体的行动。此外，我们为系统配备了进化内存模块和检索增强生成，使智能体能够在程序生成和完善过程中利用积累的经验和外部求解器知识来解决VRP问题。我们评估了48种不同VRP变体，涵盖了不同的求解器。实验结果表明，RLEA的表现优于之前的AR状态方法，成功率提高了16.67%，同时显著减少了运行时错误。这些结果验证了强化学习与基于LLM的推理的整合在自动化优化建模中非常有效。附录可在以下网站获取：https URL。

CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training

关怀：基于对比锚点的评分标准演化，适用于大型语言模型训练后

• Authors: Siyuan Li, Xinxin Song, Chen Ruinian, Jingjing Fan, Tingxiong Xiao, Yangen Hu, Ke Zeng, Jinli Suo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.00892
• Pdf link: https://arxiv.org/pdf/2609.00892
• Abstract Rubric-based reinforcement learning decomposes open-ended instructions into prompt-specific, flexible rubrics, making it better suited than reinforcement learning with verifiable rewards for post-training LLMs on open-ended tasks. However, static rubrics are inevitably hacked as the policy evolves, and existing dynamic approaches introduce new problems: undirected rubric extraction, unreliable hack detection, and unbounded rubric proliferation. We propose $\textbf{CARE}$ ($\textbf{C}$ontrastive $\textbf{A}$nchor-based $\textbf{R}$ubric $\textbf{E}$volution), which grounds every rubric evolution step in a high-quality anchor response generated by a frontier model conditioned on the prompt and its rubrics. At each training step, CARE contrasts the highest-scoring rollout against the anchor, enabling two complementary mechanisms: an Adaptive branch that reactively repairs reward misspecification; and a Chase branch that proactively converts frontier-level quality gaps into sharper rubrics. Together, the two branches $\textbf{maintain discriminative accuracy in the high-reward region}$---the precise region where reward over-optimization mostly originates. Experiments on WildChecklist-9K with Qwen2.5-7B-Base and Qwen2.5-7B-Instruct show that CARE achieves state-of-the-art performance on Arena-Hard-2.0, InfoBench, and FollowBench, and is the $\textbf{only}$ method whose win rate against GPT-4.1 anchor responses shows sustained improvement throughout 300 training steps; additional results on Llama-3.1-8B-Instruct and Qwen3-8B further indicate that CARE generalizes across model families.
• 中文摘要 基于评分标准的强化学习将开放式指令分解为提示特定且灵活的评分标准，使其比强化学习更适合在开放式任务中训练后获得可验证的奖励。然而，随着策略的发展，静态评分标准不可避免地被破解，现有动态方法也带来了新问题：无向评分标准提取、不可靠的黑客检测以及无界评分标准的繁增。我们提出 $\textbf{CARE}$ （$\textbf{C}$ontrastive $\textbf{A}$nchor-基于 $\textbf{R}$ubric $\textbf{E}$volution），该方法将每个评分标准演化步骤建立在由前沿模型生成的高质量锚点响应之上，基于提示及其评分标准。在每个训练步骤，CARE将最高得分的推广与锚点进行对比，从而实现两种互补机制：一个是被动修复奖励错误的自适应分支;另一个是主动将前沿级质量差距转化为更清晰的评分标准的Chase分支。这两个分支共同构成$\textbf{在高奖励区域保持判别准确性}$---奖励过度优化主要源头的精确区域。在WildChecklist-9K上，配合Qwen2.5-7B-Base和Qwen2.5-7B-Instruct的实验显示，CARE在Arena-Hard-2.0、InfoBench和FollowBench上实现了最先进的性能，并且是$\textbf{only}$方法，其对GPT-4.1锚点响应的胜率在300个训练步骤中持续提升;Llama-3.1-8B-Instruct和Qwen3-8B的进一步结果表明CARE在模型家族中具有推广性。

AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing for Distributed MoE Rollouts

AInfer-PD：分布式MoE部署的通信安全原位预填充-解码复用

• Authors: Guowei Wang, Chaokun Yang, Zhenxuan Pan, Yuhong Guo, Minghua Zhu, Zhechuan Zhang, Shuo Wan, Xiaowei Zhu
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.00993
• Pdf link: https://arxiv.org/pdf/2609.00993
• Abstract Rollout inference often dominates the wall-clock time of large-scale reinforcement learning (RL). In agentic RL, each trajectory alternates between model generation and environment interaction over multiple turns. Asynchronous trajectories consequently introduce new prefill (P) work while other trajectories remain in decode (D), making P/D coexistence a persistent property of the rollout rather than a one-time prompt-ingestion event. On shared accelerators, persistent P/D coexistence can make prefill interfere with latency-sensitive decode and prolong rollout completion. P/D disaggregation avoids this co-location but requires separate device pools and KV-cache transfers. In-place multiplexing retains shared devices and KV state, but existing designs lack the communication isolation needed for large MoE deployments that combine attention TP/DP with distributed expert execution. In practical implementations, P and D can issue intersecting collectives in inconsistent cross-rank orders; DeepEP's P and D paths also share mutable protocol state. We present AInfer-PD, which extends in-place P/D multiplexing to distributed MoE rollouts. AInfer-PD coordinates P/D collective order across ranks and gives the two DeepEP paths independent communication state, making crossed ADP/ATP and DeepEP paths safe for concurrent P/D execution. The design retains shared model weights and KV storage while coordinating P and D on the same devices. Across repeated single-node prefill-intensive workloads, AInfer-PD reduces fixed-workload rollout completion time by 7.1-22.5% relative to the same AInfer engine with P/D multiplexing disabled and by 24.8-32.9% relative to SGLang. On two nodes, the reductions are 18.0-35.3% and 18.3-31.8%, respectively. In a same-engine ablation, fine-grained boundaries reduce completion time by a further 8.6-19.8% over whole-epoch asynchronous enqueue.
• 中文摘要 扩展推理常常主导大规模强化学习（RL）的墙钟时间。在代理强化学习中，每个轨迹在模型生成和环境交互之间交替进行多轮。异步轨迹因此引入新的预填充（P）工作，而其他轨迹仍处于解码（D）中，使得P/D共存成为推展的持久属性，而非一次性的提示摄取事件。在共享加速器上，持久的P/D共存可能使预填充干扰延迟敏感的解码并延长滚动完成时间。P/D拆分避免了这种共址，但需要独立的设备池和KV缓存传输。原地复用保留共享设备和KV状态，但现有设计缺乏大规模MoE部署所需的通信隔离，无法将注意力TP/DP与分布式专家执行结合起来。在实际实现中，P和D可以以不一致的跨等级顺序发出交叉集体;DeepEP的P和D路径也共享可变协议状态。我们介绍AInfer-PD，将原位P/D复用扩展到分布式MoE部署。AInfer-PD协调P/D集体秩序跨等级，赋予两条DeepEP路径独立的通信状态，使交叉的ADP/ATP和DeepEP路径安全于并发P/D执行。该设计保留了共享的模型权重和KV存储，同时在同一设备上协调P和D。在重复单节点预填充密集型工作负载中，AInfer-PD相比同一AInfer引擎（禁用P/D复用）将固定工作负载的扩展完成时间缩短7.1%至22.5%，相比SGLang缩短24.8%至32.9%。在两个节点上，减少率分别为18.0-35.3%和18.3-31.8%。在同一引擎消融中，细粒边界比整个时代异步队列进一步缩短8.6-19.8%。

ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning

ARISE-RL：基于能动评分标准的迭代自我进化与强化学习

• Authors: Fanrui Zhang, Ruixue Ding, Qiang Zhang, Xi Chen, Boli Chen, Shihang Wang, Qiuchen Wang, Hongmin Zhan, Jinxin Bian, Li xingchao, Peijin Zheng, Hao cheng, Pengjun Xie, Kaipeng Zhang, Jiawei Liu, Zheng-Jun Zha
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01058
• Pdf link: https://arxiv.org/pdf/2609.01058
• Abstract Training open-ended agents via reinforcement learning (RL) is hindered by the lack of verifiable gold answers and scalable rubrics. Moreover, even near the model's capability boundary, long-horizon open-ended agentic tasks often yield brittle and unstable rewards, resulting in weak or noisy rollout contrast that obscures fine-grained optimization signals for group-based policy learning. To address these challenges, we propose ARISE-RL, a novel full-cycle self-evolution framework that couples a task/rubric Generator and a reasoning Solver through rubric-mediated co-evolution. The Generator grounds tool-related rubric criteria in real tool observations and is rewarded for producing valid, intermediate-difficulty tasks aligned with the Solver's evolving capability boundary. The Solver, in turn, learns from fine-grained rubric satisfaction signals through multi-step reasoning and tool use. We further introduce Reward-Gated Self-Evolution Distillation (RG-SED), which selectively distills a memory-augmented variant of the same policy back into itself only when the memory yields empirical reward improvement, thereby reducing distribution mismatch and avoiding blind imitation of noisy guidance. Finally, to support rigorous evaluation, we present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning. Extensive experiments demonstrate that ARISE-RL consistently achieves robust and stable overall state-of-the-art performance across all evaluated benchmarks.
• 中文摘要 通过强化学习（RL）训练开放式代理，因缺乏可验证的黄金答案和可扩展的评分标准而受阻。此外，即使接近模型能力边界，长期开放式代理任务常常产生脆弱且不稳定的奖励，导致展开对比薄弱或噪声大，掩盖了基于群体的策略学习的细粒度优化信号。为应对这些挑战，我们提出了ARISE-RL，一种新型全周期自我演化框架，通过评分标准介导的共进化将任务/评分标准生成器与推理求解器结合起来。生成器将工具相关评分标准建立在真实工具观察中，并因产出与解算器不断演变的能力边界一致的有效中等难度任务而获得奖励。求解器则通过多步推理和工具使用，从细粒度的评分标准满意信号中学习。我们还进一步引入了奖励门控自我演化蒸馏（RG-SED），该方法只有在内存产生经验奖励提升时，才选择性地将同一政策的内存增强变体提炼回自身，从而减少分布不匹配，避免盲目模仿噪声指导。最后，为支持严谨评估，我们介绍了ECR-Bench，这是一套专家校准的评分标准基准套件，涵盖单工具深度研究和多工具旅行规划。大量实验表明，ARISE-RL在所有评估基准测试中始终保持稳健稳定的整体最先进性能。

Accelerating Reinforcement Learning via MPC Solver-Gradient Guidance for Weights-varying MPC

通过MPC求解器加速强化学习——权重变化MPC梯度指导

• Authors: Baha Zarrouki, Arslan Thobani, Jasper Hoffmann, Mattia Piccinini, Rudolf Reiter, Felix Jahncke, Sébastien Gros, Davide Scaramuzza, Johannes Betz
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.01061
• Pdf link: https://arxiv.org/pdf/2609.01061
• Abstract In Model Predictive Control (MPC), cost-function weights shape closed-loop behavior, yet changing conditions often make fixed parametrizations suboptimal and motivate context-dependent online adaptation. Learning such policies is difficult because behavior depends implicitly on numerical MPC solutions, producing nonlinear, potentially nonsmooth, long-horizon dependencies on policy parameters. This creates a bias-variance tradeoff: Reinforcement Learning (RL) optimizes realized closed-loop return from environment samples but is sample-inefficient, whereas Gradient-Based Policy Learning (GB-PL) uses low-variance solver gradients from differentiable MPC to optimize surrogate losses on predicted trajectories but can be biased under model mismatch. We propose Solver-Gradient Guided Reinforcement Learning (SG-RL), a solver-sensitivity augmentation for RL-based online MPC cost-weight adaptation. SG-RL keeps sampled closed-loop return as the objective and uses bounded solver-derived gradients as auxiliary guidance to improve stability and sample efficiency. We instantiate SG-RL in Proximal Policy Optimization (PPO) with four modular algorithms that inject solver-gradient guidance into actor-update scaling, policy loss, advantage estimation, and value-function learning. On two full-scale autonomous racing platforms with intentional model mismatch, SG-RL reaches PPO's best closed-loop return with up to 70.6% fewer samples, outperforms GB-PL baselines by at least 54% in closed-loop return, and generalizes zero-shot to unseen environments.
• 中文摘要 在模型预测控制（MPC）中，成本函数权重塑造闭环行为，但变化的条件常使固定参数化不尽优，并促使上下文依赖的在线适应。学习此类策略较为困难，因为行为隐含依赖数值MPC解，产生非线性、可能不平滑的长视野依赖。这造成偏差-方差权衡：强化学习（RL）优化环境样本实现的闭环回报，但样本效率较低，而基于梯度的策略学习（GB-PL）使用可微MPC的低方差求解器梯度优化预测轨迹的替代损失，但在模型不匹配下可能有偏差。我们提出求解器-梯度引导强化学习（SG-RL），这是一种基于强化学习在线MPC成本权重适应的求解器敏感增强。SG-RL以采样闭环返回为目标，并使用有界求解器导出梯度作为辅助指导，以提高稳定性和采样效率。我们用四个模块化算法实现SG-RL，用于近端策略优化（PPO），将求解器梯度指导注入演员更新缩放、策略丢失、优势估计和价值函数学习。在两个全规模自主竞速平台中，SG-RL在样本减少最多70.6%的情况下达到PPO最佳闭环回报，闭环返回性能至少高出GB-PL基线54%，并将零机会推广到未见环境。

World Model-Guided Reinforcement Learning via Counterfactual User Engagement Simulation

通过反事实用户参与模拟实现的世界模型引导强化学习

• Authors: Ang Li, Xin Xu, Bin Liang, Yue Ma, Fubang Zhao, Yangyang Kang, Kam-Fai Wong
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2609.01067
• Pdf link: https://arxiv.org/pdf/2609.01067
• Abstract Reinforcement learning for user-centric agents is limited by the cost, latency, and risk of collecting online feedback, as well as by the lack of counterfactual comparisons under the same user state. In this paper, we propose World Model-Guided Reinforcement Learning via counterfactual user engagement simulation (WMG-RL), a framework in which a frozen user simulator provides reward supervision before real user exposure. Motivated by language world models, we instantiate the simulator as a User Engagement World Model (UEWM), which treats a recommended item as the agent action and the user's heterogeneous feedback as the environment observation. Rather than learning one fixed environment transition, UEWM learns to infer user-specific dynamics from engagement history and apply them to candidate items. In WMG-RL, a downstream policy proposes multiple candidate items for the same history; UEWM predicts the corresponding engagement feedback in parallel; and the simulated feedback is converted into dense rewards for policy optimization. Experiments show that UEWM provides reliable and transferable reward signals across domains, and that WMG-RL enables a compact 1.7B student policy to match or surpass much larger LLMs on downstream recommendation tasks.
• 中文摘要 针对以用户为中心的代理的强化学习受限于收集在线反馈的成本、延迟和风险，以及同一用户状态下缺乏反事实比较。本文提出通过反事实用户参与模拟（WMG-RL）实现世界模型引导强化学习，该框架中，固定的用户模拟器在真实用户接触前提供奖励监督。受语言世界模型激励，我们将模拟器实例化为用户参与世界模型（UEWM），将推荐项目视为代理动作，用户的异构反馈视为环境观察。UEWM不学习单一固定环境转换，而是从参与历史中推断用户特定动态并将其应用于候选项目。在WMG-RL中，下游策略为同一历史提出多个候选项目;UEWM并行预测相应的参与反馈;模拟反馈转化为密集奖励，用于策略优化。实验表明，UEWM在各域提供可靠且可转移的奖励信号，WMG-RL使紧凑的17亿学生政策能够在下游推荐任务中匹敌甚至超越更大规模的LLM。

Update for Decisions, Not Freshness: Goal-Oriented Status Updating and Selective Offloading at the Network Edge

更新为决策，而非新鲜度：目标导向状态更新和网络边缘的选择性卸载

• Authors: Jianpeng Qi, Qiyang Zhang, Chao Liu, Jing Sun, Yimei Liu, Yanwei Yu, Yingjie Wang, Wei Ni
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Multiagent Systems (cs.MA); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2609.01082
• Pdf link: https://arxiv.org/pdf/2609.01082
• Abstract In an edge--cloud collaborative edge-computing environment, an edge node (EN) must decide whether each user task should be executed locally, forwarded to a remote service (or cloud) node (SN), or rejected. The EN observes its local state directly but receives the SN state only through an intermittently refreshed cache. Status updating and task control therefore form an asynchronous closed loop under partial observability. Freshness-driven schemes, including those based on Age of Information (AoI), do not directly value an update by its effect on subsequent task decisions. We propose CoSMO (Co-design of Semantic-state Management and Offloading), a cooperative event-driven reinforcement learning (RL) framework that coordinates semantic status management and selective offloading through realized task utility. CoSMO learns a compact representation of the heterogeneous SN service state. At the SN, a recurrent semi-Markov double deep Q-network (Double DQN) agent jointly selects send/no-send and the next decision interval. At the EN, a task-terminal off-policy value-learning agent makes hierarchical gate--route decisions from local observations and stale remote semantics. The agents maintain separate observations and value targets but share the same realized task-utility stream, without centralized execution. Across the evaluated workload families, CoSMO's reported relative improvement in on-time completion rate over the best-performing competing method averages 18.6%--21.2%. For capacity-aware decision accuracy across the three strict-overload points, the corresponding reported gains average 17.6%--$17.9%.
• 中文摘要 在边缘云协作边缘计算环境中，边缘节点（EN）必须决定每个用户任务是本地执行、转发到远程服务（或云）节点（SN）还是拒绝。EN直接观察其本地状态，但仅通过间歇性刷新的缓存接收SN状态。因此，状态更新和任务控制形成了部分可观测性的异步闭环。基于新旧度驱动的方案，包括基于信息时代（AoI）的方案，并不直接根据更新对后续任务决策的影响来评估其价值。我们提出了CoSMO（语义状态管理与卸载协同设计），这是一个协作式事件驱动强化学习（RL）框架，通过实现的任务效用协调语义状态管理和选择性卸载。CoSMO学习异构SN服务状态的紧凑表示。在 SN，一个循环的半马尔可夫双深度 Q 网络（Double DQN）代理共同选择发送/不发送和下一个决策区间。在 EN 处，任务终端的非策略值学习代理根据局部观察和陈旧的远程语义做出分层门路决策。代理保持独立的观察和价值目标，但共享同一实现的任务-效用流，没有集中执行。在评估的工作负载家族中，CoSMO 报告的准时完成率相较于表现最佳的竞争方法的相对提升为 18.6%——21.2%。在三个严格超载点的容量感知决策准确度中，相应的提升平均为 17.6%——17.9%。

PersuaRL: Reinforcement Learning-Driven Multi-Expert Selection for Persuasive Dialogue Generation in Insurance

PersuaRL：基于强化学习驱动的多专家选择，用于保险中说服性对话生成

• Authors: Rohan Kirti, Akash Ghosh, Aryan Vats, Niladri Ghosh, Shipra Shriparn, Roshni Ramnani, Anutosh Maitra, Sriparna Saha
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.01188
• Pdf link: https://arxiv.org/pdf/2609.01188
• Abstract Large Language Models (LLMs) are revolutionizing digital communication by powering conversational agents deployed across domains such as customer service, digital sales, and insurance. These agents, built on LLMs, can understand user input, retrieve relevant information, and generate coherent responses. However, while they excel at factual communication, they often lack the ability to engage in truly persuasive, context-sensitive dialogue, especially in domains like insurance, where trust and clarity are critical. Building on this need within the insurance domain, our work focuses on improving the persuasiveness of digital agents, aka LLMs. To support this, we introduce InsureDial, a Persuasive Insurance Dialogue dataset, designed to capture the nuances of persuasive communication specific to motor insurance interactions. We introduce PersuaRL, a reinforcement learning-based framework that equips LLM-driven dialogue agents with the ability to adaptively explore, select, and coordinate strategies across multiple expert modules, guided by the evolving dialogue context, to achieve more effective persuasion. We conduct extensive automatic human and qualitative evaluations on two benchmark persuasion dialogue datasets, including our InsureDial. Our evaluations consistently demonstrate that PersuaRL outperforms baseline, generating contextually appropriate and highly persuasive responses.
• 中文摘要 大型语言模型（LLM）正在革新数字通信，支持部署在客户服务、数字销售和保险等多个领域的对话代理。这些基于LLM构建的代理能够理解用户输入，检索相关信息，并生成连贯的回应。然而，尽管它们在事实沟通方面表现出色，但往往缺乏真正具有说服力、情境敏感对话的能力，尤其是在像保险这样信任和清晰度至关重要的领域。基于保险领域的这一需求，我们的工作重点是提升数字代理（即LLM）的说服力。为此，我们推出了InsureDial，一个说服性保险对话数据集，旨在捕捉机动车保险互动中说服性沟通的细微差别。我们介绍了PersuaRL，一个基于强化学习的框架，使基于LLM的对话代理能够适应性地探索、选择和协调多个专家模块中的策略，并根据不断演变的对话语境实现更有效的说服。我们对包括InsureDial在内的两个基准说服对话数据集进行了广泛的自动人类和定性评估。我们的评估持续显示PersuaRL优于基线，生成符合上下文且高度说服力的回答。

CaRL-EM: Cost-Aware Reinforcement Learning for Entity Matching with LLMs

CaRL-EM：面向大型语言模型实体匹配的成本感知强化学习

• Authors: Chaohui Guo, Michel Klein, Zhisheng Huang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.01195
• Pdf link: https://arxiv.org/pdf/2609.01195
• Abstract Entity matching (EM) requires fine-grained contextual understanding and domain knowledge. Recent work shows that large language models (LLMs) can serve as strong matchers across domains, but most methods either make independent pairwise decisions or rely on manually designed composite pipelines, thus lacking flexibility in realistic multi-candidate settings. At the same time, they typically ignore inference cost at scale. We formulate LLM-based EM with candidates as a cost-aware sequential decision problem and propose CaRL-EM, a reinforcement learning controller that manages LLM operations. Given the state of an anchor record, its candidate set, and the cost, CaRL-EM adaptively chooses among different operators (Match/Compare/Select/Decide) and model capacities to maximize a quality-cost objective. The policy interacts with abstract operators, allowing the same controller to be reused with different underlying LLM backends at inference time without retraining. Experiments on 7 benchmarks show that CaRL-EM (i) learns to dynamically plan the usage of inexpensive and expensive operators based on task complexity, (ii) achieves robust zero-shot transfer across diverse datasets and domains, and (iii) consistently achieves a better quality-cost trade-off than strong LLM-based baselines and manually designed pipelines, yielding a lower inference cost at comparable or higher quality.
• 中文摘要 实体匹配（EM）需要细致的上下文理解和领域知识。最新研究表明，大型语言模型（LLM）可以作为跨域强匹配器，但大多数方法要么独立做出成对决策，要么依赖手动设计的复合流水线，因此在现实的多候选环境中缺乏灵活性。同时，它们通常忽视大规模推理成本。我们将基于LLM的EM与候选对象构建为一个成本感知的顺序决策问题，并提出了CaRL-EM，一种管理LLM操作的强化学习控制器。根据锚点记录的状态、候选集和成本，CaRL-EM自适应地在不同操作符（匹配/比较/选择/决定）和模型能力中选择，以最大化质量-成本目标。该策略与抽象操作符交互，使同一控制器在推理时可与不同的底层LLM后端重用，无需重新训练。7个基准测试的实验表明，CaRL-EM（i）能够根据任务复杂度动态规划廉价和昂贵的运算符的使用，（ii）实现跨多样数据集和领域的稳健零样本传输，（iii）持续优于强有力的基于LLM的基线和人工设计的流水线，从而在同等或更高质量下实现更低的推断成本。

Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents

探索更多，漂移少：仅结果强化学习足以满足长远视野互动代理的需求

• Authors: Liming Pu, Xiaoxia Li, Yifu Liu, Teng Cao, Bin Yang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01245
• Pdf link: https://arxiv.org/pdf/2609.01245
• Abstract Reinforcement learning is a natural way to post-train LLM agents for long-horizon interactive tasks judged only by end-of-task verification, yet a shared belief holds that outcome-only RL soon hits a ceiling on small open models. Recent work therefore compensates around the training with denser rewards, SFT priors, skill libraries, curated memory, or multi-agent orchestration. We argue the ceiling is an artifact of two failures of common practice. Signal starvation: group-relative RL with sparse outcome-only rewards yields a gradient only when a task's rollout group mixes successes and failures, so under-scaled exploration silences exactly the hardest, most instructive tasks. Policy drift: squeezing many updates out of a small task pool degrades the policy itself, as an unanchored objective lets the sampling distribution collapse exactly when saturation has already made informative groups rare. We present CANOPY (Coverage-ANchored On-PolicY RL), a minimalist protocol attacking both directly: scale same-task exploration until the natural signal reappears, keep every update on-policy, KL-anchored, and confined to the agent's own action tokens, then cash in an enlarged interaction budget at test time. On AppWorld, a long-horizon interactive coding benchmark, a Qwen3-14B policy trained with CANOPY through environment interaction alone--without task-specific supervision, auxiliary credit signals, or elaborate agent scaffolding--topped the public leaderboard (Feb. 2026; Test-Normal TGC 86.9, Test-Challenge 67.6), and the same design principles lift Qwen3.5-9B on SWE-bench Verified by 16.6 points. Agentic RL alone thus internalizes long-horizon capability directly into a small open model; we plan to release the complete training stack at this https URL.
• 中文摘要 强化学习是为仅通过任务结束验证判断的长期交互任务进行后期训练LLM代理的自然方式，但普遍认为仅结果的强化学习在小型开放模型上很快就会达到上限。因此，近期工作通过更密集的奖励、SFT先验、技能库、策划记忆或多智能体编排来弥补训练。我们认为天花板是两种常见实践失败的产物。信号饥饿：群体相对强化学习在仅结果奖励稀疏时，只有在任务的推广组混合成功与失败时才会产生梯度，因此规模不足的探索恰好压制了最难、最具启发性的任务。策略漂移：从小任务池中挤出大量更新会削弱策略本身，因为无锚定目标使抽样分布在饱和已使信息组稀缺时崩溃。我们提出了CANOPY（覆盖-安科鲁德On-PolicY RL），这是一种极简主义协议，直接攻击两者：扩大同任务探索直到自然信号重新出现，保持每次更新在策略上，基于KL锚定，并局限于代理自身的动作标记，然后在测试时兑现更大的交互预算。在AppWorld上，一个长期视野交互编码基准测试，Qwen3-14B策略仅通过环境交互训练——无任务特定监督、辅助信用信号或复杂的代理支架——登顶公开排行榜（2026年2月;测试正常TGC 86.9，测试挑战67.6），同样的设计原则使Qwen3.5-9B在SWE工作台验证中提升了16.6个百分。因此，代理强化学习本身就将长期视野能力直接内化到一个小型开放模型中;我们计划在此HTTPS网址发布完整的培训堆栈。

From Base Rollouts to RL Reasoning: A Budgeted Search Perspective

从基地部署到强化学习推理：预算搜索视角

• Authors: Wenhe Sun, Cunxiang Wang, Zijun Yao, Yixin Cao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.01274
• Pdf link: https://arxiv.org/pdf/2609.01274
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves language-model reasoning, but how these gains relate to inference-time decoding and search remains unclear. Does RL create reasoning the base model lacks, or shift the rollout distribution toward trajectories it can already reach but rarely samples? We study this behaviorally with a Unified Decoding Framework (UDF), which expresses token-level sampling, beam-like search, tree search, and sequence-level resampling as executable policies over a shared budgeted operating space, scored post hoc with pass@$k$, self-consistency, best-of-$N$, and first-finish success. Using paired Base/RL checkpoints from SimpleRL-Zoo, we ask whether an RL default-policy curve can be approximated by a structured path of Base operating points. On Math500, AIME, GPQA, and IFEval, the pass@$k$ recovery path follows a Budgeted Operating-Point Transition Rule (BOPTR), $N_{\mathrm{Base}} \approx \alpha N_{\mathrm{RL}}^{\beta}$, with benchmark-conditioned exponents. On Qwen2.5-7B, BOPTR gives the lowest transfer error among the non-oracle rules we test, 3.41 pp (95% CI [2.32, 5.53]); a three-seed replication gives 3.07 $\pm$ 0.39 pp. The rule extends to ten models across four families (3.28 to 4.87 pp on checkpoints added after fitting), to four benchmarks it was never fitted on (5.03 pp vs. 4.44 pp in fit), and holds without an RL checkpoint for the target model (4.19 pp) or without RL supervision of any kind (5.08 pp). These results support a qualified internalized-search reading: under the recipe we test, much of the measured RL gain corresponds to a change in sampling efficiency toward operating points the base model can already reach under search. We treat the scaling patterns as descriptive of this recipe and cohort, report where they break down, and use UDF and BOPTR as behavioral diagnostics rather than evidence of parameter-level equivalence.
• 中文摘要 带有可验证奖励的强化学习（RLVR）提升了语言模型推理能力，但这些收益如何与推理时间解码和搜索相关尚不清楚。强化学习是否创造了基础模型缺乏的推理，还是将推广分布向其已能达到但很少采样的轨迹移动？我们用统一解码框架（UDF）进行行为研究，该框架将代币级采样、类束搜索、树搜索和序列级重采样作为可执行策略，覆盖共享预算操作空间，事后评分包括pass@$k美元、自洽性、$N场最佳和首个完成成功。利用SimpleRL-Zoo中的配对基础/强化学习检查点，我们询问强化学习默认策略曲线是否可以用基础操作点的结构化路径近似。在Math500、AIME、GPQA和IFEval上，pass@$k$恢复路径遵循预算操作点转换规则（BOPTR），$N_{\mathrm{Base}} \approx \alpha N_{\mathrm{RL}}^{\beta}$，基准条件指数。在Qwen2.5-7B中，BOPTR在测试的非预言机规则中传递误差最低，3.41 pp（95% CI [2.32， 5.53]）;三种子复制为3.07 $\pm$ 0.39 pp。该规则扩展至四个模型族的十个模型（拟合后添加检查点为3.28至4.87 pp），对应从未拟合的四个基准（5.03 pp对拟合4.44 pp），且在目标模型没有强化学习检查点（4.19 pp）或无任何强化学习监督的情况下（5.08 pp）均成立。这些结果支持了合格的内化搜索读数：在我们测试的配方下，测量到的RL增益大部分对应于基础模型已可通过搜索达到的操作点的抽样效率变化。我们将缩放模式视为该配方和队列的描述，报告其失效之处，并使用UDF和BOPTR作为行为诊断工具，而非参数层面等效的证据。

VerTox: Verifiable Reward-Guided Corpus Poisoning Against Neural Ranking Models

VerTox：基于神经排名模型的可验证奖励引导红细胞中毒

• Authors: Zhiqi Huang, Vivek Datla, Zhichao Xu, Puxuan Yu, Vivek Srikumar, Alfy Samuel
• Subjects: Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2609.01325
• Pdf link: https://arxiv.org/pdf/2609.01325
• Abstract Neural ranking models have become core components of modern information retrieval systems and important building blocks of AI systems such as retrieval-augmented generation (RAG) pipelines. However, their robustness remains insufficiently understood in the presence of large language models (LLMs), which can generate fluent and deceptive content at scale. This work investigates the vulnerability of neural ranking models to corpus poisoning attacks, in which an adversary injects a small number of maliciously crafted documents into the corpus to distort ranking behavior. We propose VerTox, the first framework to formulate corpus poisoning as a verifiable reward-guided reinforcement learning (RLVR) problem. By explicitly coupling ranking distortion with factual corruption through specialized reward shaping, we fine-tune compact LLMs into adversarial generators. Experiments demonstrate that our method achieves near-perfect attack success rates, producing adversarial documents that frequently rank higher than target documents across major neural ranking architectures, as well as a proprietary commercial embedding model. The generated adversarial documents are fluent and exhibit low perplexity, making them difficult to detect. Furthermore, by explicitly encouraging factual corruption, our adversarial documents significantly degrade the performance of a downstream RAG application.
• 中文摘要 神经排序模型已成为现代信息检索系统的核心组成部分，也是检索增强生成（RAG）流水线等人工智能系统的重要构建模块。然而，在大型语言模型（LLM）存在下，其鲁棒性仍未被充分理解，这些模型能够大规模生成流畅且具有欺骗性的内容。本研究研究神经排序模型对语料库中毒攻击的脆弱性，即攻击者向语料库注入少量恶意制作的文档，以扭曲排名行为。我们提出了VerTox，这是首个将语料体中毒表述为可验证的奖励引导强化学习（RLVR）问题的框架。通过通过专门的奖励塑造，明确将排名扭曲与事实腐败耦合，我们将紧凑型LLM微调为对抗性生成器。实验表明，我们的方法几乎实现了完美的攻击成功率，能够生成在主要神经排序架构中经常排名高于目标文档的对抗文档，同时采用专有商业嵌入模型。生成的对抗文档流畅且困惑度低，使其难以被检测。此外，通过明确鼓励事实破坏，我们的对抗文档显著降低了下游RAG应用的性能。

Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR

验证者失效之处：RLVR中奖励信号的类别级审计

• Authors: Esther Xin
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.01354
• Pdf link: https://arxiv.org/pdf/2609.01354
• Abstract Reinforcement learning with verifiable rewards (RLVR) and standard benchmark evaluation both rely on an automatic verifier that turns a free text answer into a binary reward. Prior work reports that one evaluation harness accepts only about 94% of its own ground truth answers, blaming LaTeX parsing. That is an aggregate: it does not say which answer forms consume the error budget. We supply the decomposition. We apply metamorphic testing to the verifier rather than the model, generating certified equivalent answer variants, that is, rewrites that preserve mathematical meaning by construction, so that any rejection is a provable false negative needing no human adjudication. We then measure rejection per answer category across four widely used verifiers over 307,420 verdicts. We find three things. (1) Self validation ranges from 53.8% to 95.2% on identical inputs, a spread of 41.3 points. The published figure describes one implementation, not the task; two configurations of the same library disagree on 49.9% of pairs. (2) The residual is not spread across parsing categories but concentrated in whitespace and punctuation, which account for 93.0% of in contract failures for the default LaTeX configuration. A trailing period or newline dominates the budget. (3) Separating rejection from execution failure shows that verifiers with similar aggregate error fail for opposite reasons, and that a reference numeric cascade accepts off by one wrong answers as a step function of magnitude, from 0% below 10^4 to 100% at or above, because its relative tolerance is scale invariant.
• 中文摘要 带有可验证奖励的强化学习（RLVR）和标准基准评估都依赖自动验证器，将自由文本答案转化为二元奖励。先前研究报告称，一种评估工具仅接受约94%的自身真实答案，原因归咎于LaTeX解析。这是一个汇总：它不说明哪些答案形式消耗了错误预算。我们提供分解结果。我们将变质测试应用于验证者而非模型，生成经过认证的等价答案变体，即通过构造保持数学意义的重写，使任何拒绝都是可证明的假阴性，无需人工裁决。然后我们测量了四个广泛使用的验证器中每个答案类别的拒绝率，涵盖307,420个判决。我们发现了三点。（1）在相同输入下，自验证率在53.8%至95.2%之间，差距为41.3个百分点。已发布的图示描述的是一个实现，而非任务本身;同一库的两个配置在49.9%的对上存在差异。（2）残差不分布于解析类别，而是集中在空白和标点符号上，这占默认LaTeX配置中合同失败的93.0%。预算中，尾段周期或换行占主导地位。（3）将拒绝与执行失败分离，显示具有相似总误差的验证器因相反原因失败，且参考数值级联通过一个错误答案接受为幅度的阶梯函数，从10^4以下的0%到100%的100%，因为其相对容忍度是尺度不变的。

EdiTikZ: Scientific Figure Editing from Revision Trajectories

EdiTikZ：从修订轨迹出发的科学人物编辑

• Authors: Christian Greisinger, Zhixue Zhao, Steffen Eger
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.01409
• Pdf link: https://arxiv.org/pdf/2609.01409
• Abstract Vision-language models (VLMs) have shown strong performance in generating scientific figures from text or images. However, producing publication-ready figures requires iterative refinement, making scientific figure editing an important yet largely unexplored task. Existing approaches rely on costly proprietary agentic systems, focus primarily on evaluation, or construct training supervision from synthetically generated edits. Instead, we leverage naturally occurring scientific revision and development trajectories as a scalable source of supervision. To this end, we introduce DaEdiTikZ, the first large-scale dataset of revision-derived scientific figure edits, constructed by mining 391K plausible TikZ edit pairs from arXiv, GitHub, and TeX SE and inferring 781K directed edit instructions with a VLM conditioned on rendered figures and TikZ code. We further introduce DaEdiTikZ-Bench, a human-refined benchmark with 790 instances, and train two compact Qwen3.5-based EdiTikZ models (4B and 9B) by jointly learning reconstruction and editing, followed by reinforcement learning (RL) with complementary rewards for rendered fidelity and edit application. Automatic evaluation places our 9B model above all tested baselines, while human evaluation with 9 annotators and 4,320 ratings places it above GPT-5.6-Sol and on par with Gemini-3.1-Pro. Under severe out-of-distribution shifts, it remains competitive with GPT-5.6-Sol near its 2K training sequence-length regime. Models and datasets will be released.
• 中文摘要 视觉语言模型（VLMs）在从文本或图像生成科学图形方面表现出优异表现。然而，生成可发表的图形需要迭代优化，使科学图形编辑成为一项重要但尚未被广泛探索的任务。现有方法依赖成本高昂的专有智能体系统，主要关注评估，或从合成生成的编辑构建训练监督。相反，我们利用自然发生的科学修订和发展轨迹作为可扩展的监督来源。为此，我们引入了DaEdiTikZ，这是首个大规模的修订衍生科学图形编辑数据集，通过从arXiv、GitHub和TeX SE挖掘391K合理的TikZ编辑对构建，并基于渲染图和TikZ代码的VLM推断781K定向编辑指令。我们进一步介绍了DaEdiTikZ-Bench，一个由人类完善的基准测试，拥有790个实例，并通过联合学习重建和编辑训练两个基于Qwen3.5的紧凑型EdiTikZ模型（4B和9B），随后进行强化学习（RL），并对渲染的忠实度和编辑应用提供互补奖励。自动评估显示，我们的9B模型高于所有测试基线，而人工评估（9个标注者和4320个评分）使其高于GPT-5.6-Sol，与Gemini-3.1-Pro相当。在严重的非分布分布变化下，它在接近2K训练序列长度范围内仍能与GPT-5.6-Sol竞争。模型和数据集将陆续发布。

Provably Safe Sim-to-Real Transfer

可验证的安全模拟到现实传输

• Authors: Tingting Ni, Maryam Kamgarpour
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01418
• Pdf link: https://arxiv.org/pdf/2609.01418
• Abstract To mitigate the sample complexity of real-world reinforcement learning (RL), a common practice is to first train a policy in a simulator, where samples are cheap, and then deploy the learned policy in the real world with the hope that it generalizes effectively. Such direct sim-to-real transfer is not guaranteed to succeed: simulator-trained policies can be suboptimal in the real world due to sim-to-real mismatch. Correcting this mismatch requires collecting data from the real system, but in many applications, such as robotics and healthcare, this data-collection process is itself subject to safety constraints. This gives rise to the problem of safe sim-to-real transfer: how can an agent exploit an imperfect simulator while ensuring safe real-world data collection and learning a near-optimal feasible policy for the target system? We address this problem by formulating safe sim-to-real transfer within the framework of reward-free safe RL. We design a computationally efficient algorithm that exploits simulator information to provably reduce real-world interaction while ensuring safe exploration and enabling the computation of a near-optimal feasible policy for any potential reward function. Our real-world sample complexity bound characterizes the benefit of using the simulator in terms of the sim-to-real mismatch.
• 中文摘要 为了减轻现实强化学习（RL）的样本复杂性，常见做法是先在模拟器中训练策略，因为样本成本较低，然后在现实世界中部署所学策略，希望其能有效泛化。这种直接的模拟到现实传输并不保证成功：模拟器训练的策略由于模拟与现实不匹配，在现实世界中可能不够优化。纠正这种不匹配需要从真实系统收集数据，但在许多应用中，如机器人和医疗，这一数据收集过程本身受安全约束。这引发了安全模拟到现实传输的问题：智能体如何在利用不完美的模拟器，同时确保安全的真实世界数据收集并学习目标系统的近优可行策略？我们通过在无奖励安全强化学习框架内制定安全的模拟到现实转移来解决这个问题。我们设计了一种计算高效的算法，利用模拟器信息可证明地减少现实世界交互，同时确保安全探索，并实现对任何潜在奖励函数的近优可行策略的计算。我们的现实世界样本复杂度界限描述了使用模拟器在模拟与现实不匹配上的优势。

NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Games

NashDreamer：零和不完美信息游戏中的基于模型的强化学习

• Authors: Tomáš Holeček, Viliam Lisý
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.01549
• Pdf link: https://arxiv.org/pdf/2609.01549
• Abstract Model-based reinforcement learning (MBRL) has achieved remarkable results in single-agent domains, yet its extension to competitive imperfect information games (IIGs) remains underexplored. In multi-agent settings, opponent-induced non-stationarity complicates the learning process, and decentralized model learning faces severe identifiability barriers, which we argue make centralized model learning a mathematical necessity. Building on this analysis, we propose NashDreamer, a principled MBRL framework for two-player zero-sum IIGs. It introduces a centralized Multi-Agent Recurrent State-Space Model (MARSSM) that decouples environment dynamics from the effect of players' strategies on their individual observations. NashDreamer is designed to use arbitrary policy gradient algorithms and inherits their convergence guarantees towards Nash equilibria under an idealized model. Empirical evaluations across four benchmark games demonstrate that NashDreamer substantially improves sample efficiency over model-free baselines early in the training. Finally, we theoretically analyze the architecture's optimization landscape, identifying the vulnerability of the Dreamer family of algorithms to posterior collapse in stochastic environments. We leave it as an open challenge.
• 中文摘要 基于模型的强化学习（MBRL）在单智能体领域取得了显著成果，但其在竞争性不完美信息博弈（IIGs）中的推广仍然未被充分探索。在多智能体环境中，对手诱导的非平稳性使学习过程复杂，去中心化模型学习面临严重的识别障碍，我们认为这使得集中式模型学习成为数学上的必需。基于此分析，我们提出了NashDreamer，这是一个基于原则的MBRL框架，用于两人零和IIGs。它引入了一个中心化的多智能体递归状态空间模型（MARSSM），将环境动态与玩家策略对其个体观察的影响解耦。NashDreamer设计为使用任意策略梯度算法，并在理想化模型下继承其趋向纳什均衡的收敛保证。对四个基准游戏的实证评估表明，NashDreamer在训练初期相比无模型基线显著提升了样本效率。最后，我们理论分析了该架构的优化环境，识别了Dreamer系列算法在随机环境中对后期崩溃的脆弱性。我们将此作为一个开放的挑战。

Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers

通过熵进行选择性代理指导：从不完美的VLM教师那里学习自主策略

• Authors: Matteo Merler, Giovanni Bonetta, Davide Zago, Rossella Cancelliere, Bernardo Magnini
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.01567
• Pdf link: https://arxiv.org/pdf/2609.01567
• Abstract Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Because VLM advice is not always reliable, SAGE can weight teacher-action distillation using environment-derived advantages rather than treating all suggestions as equally useful. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher. The results show that selective guidance is most beneficial when the VLM can help the agent discover high-reward trajectories, and less useful when unguided exploration already succeeds or teacher actions do not lead to informative experience. SAGE also reduces VLM usage by prompting the teacher only on a fraction of training steps and requiring no VLM calls at deployment. Overall, our results suggest that VLMs don't need to be used as fixed policies to be useful; they can instead act as temporary, imperfect sources of guidance whose value is tested and internalized through interaction.
• 中文摘要 视觉语言模型（VLMs）为交互式决策提供了有用的先验，但直接将其作为策略使用成本高且脆弱：每一步都必须查询，环境交互无法改进，且可能重复系统性错误。我们研究如何从在线、昂贵且不完美但信息丰富的VLM教师那里学习廉价自主策略。我们提出了SAGE（通过熵进行选择性代理指导）框架，该框架仅在学习者不确定时查询VLM，在培训中执行建议动作，并将指导提炼成轻量级强化学习（RL）策略。由于VLM建议并不总是可靠，SAGE可以利用环境衍生优势来加权教师行动提炼，而非将所有建议视为同等有用。在稀疏奖励视觉推理和导航任务中，SAGE在评估时学习那些在评估时无需VLM指导的策略，并且在多种环境中优于无引导强化学习，包括所学策略超过VLM教师的环境。结果显示，当VLM能帮助智能体发现高奖励轨迹时，选择性引导最为有益;当无引导探索已成功或教师的行为未能带来信息体验时，选择性指导的作用则较少。SAGE还通过仅在培训步骤的一小部分提示教师，且部署时无需VLM调用，从而减少了VLM的使用。总体而言，我们的结果表明VLM不必作为固定策略也能发挥作用;它们可以作为临时且不完美的指导来源，其价值通过交互被测试和内化。

Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs

从小型到大型大型语言模型（LLM）的近优SFT-RL注释预算分配扩展

• Authors: Jingtan Wang, Arun Verma, Xiaoqiang Lin, Zhengyuan Liu, Nancy F. Chen, Daniela Rus, Bryan Kian Hsiang Low
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.01573
• Pdf link: https://arxiv.org/pdf/2609.01573
• Abstract How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhaustive large-scale search. Our results hold consistently across tasks, model families, and both preference-based off-policy and reward-supervision on-policy RL methods. We further analyze how the asymmetry in annotation costs between SFT and RL data shifts the near-optimal region.
• 中文摘要 如何在LLM后训练阶段将固定注释预算分配给监督微调（SFT）和强化学习（RL）仍是一个未解之谜。现有研究仅描述了广泛的趋势（例如，SFT在低数据环境中占主导地位），缺乏原则性分配框架，且未检验最优比值是否跨模型规模转移。我们将此问题框架为近最优性：我们不寻求单一最优SFT-RL比值，而是刻画近最优区域，即在指定峰值性能容差范围内的分配集合。经验上，该区域即使在小容差（2-10%）下也很宽，随模型尺度扩大，且能可靠地从小代理模型转移到大型目标模型。这带来了一个实用策略：小型代理模型实验足以识别可转移的近最优区域，无需穷尽的大尺度搜索。我们的结果在任务、模型族以及基于偏好的非策略方法和奖励监督策略强化学习方法中均一致。我们还进一步分析了SFT与RL数据之间注释成本的不对称性如何影响近似最优区域的移动。

StudentSim: Training LLM-based Student Simulators

StudentSim：训练基于LLM的学生模拟器

• Authors: Ke Yang, Chenglong Wang, Michel Galley, Chandan Singh, Jeevana Priya Inala, ChengXiang Zhai, Jianfeng Gao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.01591
• Pdf link: https://arxiv.org/pdf/2609.01591
• Abstract AI tutors are most useful when they adapt to each student's strengths, weaknesses, and preferred guidance, but evidence about which guidance works for which student is sparse, slow, and costly to collect from real learners. Student simulators can provide this signal as a proxy, yet existing approaches are limited: state-tracking models fit student behavior but struggle to process explanations or corrections, while LLM role-play follows guidance fluently but does not reliably match the competence of the student being imitated. We present StudentSim, a training framework that turns sparse per-student data into individualized simulators through pooled training followed by per-student specialization. The resulting simulators both mirror a student's own responses and update them under tutor guidance. We also introduce StudentSimEval, a standardized protocol covering 60 students across chess, second-language English writing, and mathematics, using public learner datasets with de-identified records shared for research. StudentSimEval measures behavioral fidelity (F), or how well a simulator matches a student's responses, and guidance responsiveness (R), or how readily it updates under tutor guidance, with all methods fit and evaluated on the same records. Across all three domains, StudentSim outperforms GPT-5.4 on both metrics. In chess, StudentSim reaches F=0.51 and R=0.91, compared with 0.23 and 0.72 for GPT-5.4 and 0.45 and 0.27 for Maia2. As a proof of concept, using StudentSim as a reward model for tutor reinforcement learning produces a chess tutor that expert humans rate as more accurate, better-guided, and more personalized than a no-RL baseline and a tutor trained against a GPT-5.4 simulator reward. Code is available at this https URL.
• 中文摘要 AI导师在适应每个学生的优势、劣势和偏好指导时最有用，但关于哪种指导适合哪个学生的证据稀少、缓慢且从真实学习者那里收集成本高昂。学生模拟器可以提供这一信号作为代理，但现有方法有限：状态追踪模型适合学生行为，但难以处理解释或纠正，而LLM角色扮演流畅遵循指导，但无法可靠匹配被模仿学生的能力。我们介绍StudentSim，一个培训框架，通过汇集训练和逐学生专精，将稀疏的每位学生数据转化为个性化模拟器。生成的模拟器既反映学生自身的回答，又在导师指导下更新。我们还引入了StudentSimEval，这是一套标准化协议，涵盖60名学生，涵盖国际象棋、第二语言英语写作和数学，使用公开学习者数据集，并共享去标识化记录供研究。StudentSimEval衡量行为忠实度（F），即模拟器与学生回答匹配的程度，以及指导响应性（R），即在导师指导下更新的快速度，所有方法均基于相同记录进行拟合和评估。在这三个领域，StudentSim在这两个指标上均优于GPT-5.4。在国际象棋中，StudentSim达到F=0.51，R=0.91，而GPT-5.4为0.23和0.72，Maia2为0.45和0.27。作为概念验证，使用 StudentSim 作为导师强化学习的奖励模型，生成的国际象棋导师被专家认为比无强化学习基线和基于 GPT-5.4 模拟器奖励训练的导师更准确、更有指导性和更个性化。代码可在此 https 网址获取。

Facet-0: A Robotic Foundation Model for Contact-Rich Precise Manipulation

Facet-0：用于接触丰富精密操作的机器人基础模型

• Authors: Haoyuan Deng, Haichao Liu, Wenkai Guo, Yuan Ling, Zaijia Yang, Yuanjiang Xue, Haosheng Sun, Liangzi Wang, Ziwei Wang
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.01596
• Pdf link: https://arxiv.org/pdf/2609.01596
• Abstract Real-world robotic assembly at sub-millimeter tolerances demands spatial precision, compliant interaction, and robustness to contact failures. We present Facet-0, a robotic foundation model that predicts and values the contact consequences of its actions. Facet-0 unifies multimodal representation learning and reinforcement learning (RL) post-training around a joint action-wrench proposal: a causal wrench history is aligned with vision-language semantics and kinematic state, and flow matching generates each action chunk together with the future wrist-wrench profile it is expected to induce. Deployment rollouts train a distributional Action-Wrench Critic to distinguish motions with similar task progress but different contact outcomes, while phase-aware rewards and contact-selective credit concentrate policy improvement on decisive interactions. To accommodate part-specific dynamics, a lightweight bounded actor reuses the frozen representation for on-robot adaptation; RL remains defined over executable Cartesian actions, while an auxiliary wrench head preserves predictive, non-commanded action-contact coupling. Trained on ManuFacet-1K, a 1,000-hour force-synchronized corpus spanning three embodiments and multiple manufacturing cells, the bounded task-adapted system reaches 82% mean success on five sub-millimeter computer-assembly tasks, compared with 15% for the strongest baseline, with 0.5 mm placement accuracy and 50 ms command latency.
• 中文摘要 在亚毫米级公差下进行现实世界的机器人组装，要求空间精度、顺应的交互以及对接触失效的鲁棒性。我们介绍了 Facet-0 这一机器人基础模型，能够预测并评估其动作的接触后果。Facet-0 将多模态表示学习和强化学习（RL）在训练后统一起来，围绕联合动作扳手提案进行整合：因果扳手历史与视觉语言语义和运动学状态对齐，流匹配生成每个动作块及其预期诱导的未来腕扳手配置文件。部署部署训练分布式动作扳手批评者，以区分任务进展相似但接触结果不同的动作，同时阶段感知奖励和接触选择性信用则将策略改进集中于决定性交互。为适应部分特定动态，轻量级有界演员重用冻结表示用于机器人适应;强化学习仍定义在可执行的笛卡尔动作上，而辅助扳手头则保持预测性、非指令作用-接触耦合。该系统基于ManuFacet-1K训练，这是一种1000小时的力同步语料库，涵盖三个实体和多个制造单元，该有界任务适应系统在五个亚毫米级计算机组装任务中平均成功率达82%，而最强基线为15%，定位精度为0.5毫米，指令延迟为50毫秒。

The Rise of Verbal Reinforcement Learning

言语强化学习的兴起

• Authors: Kshitij Tayal, Arun Sharma, Genta Indra Winata, Anirban Das, Sambit Sahu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01597
• Pdf link: https://arxiv.org/pdf/2609.01597
• Abstract Natural language is emerging as a primary feedback channel for improving language agents, capable of conveying intent, preferences, and causal structure in forms interpretable by both humans and modern language models. We call this paradigm Verbal Reinforcement Learning (VRL) and offer the first unified account of it. We organize the field around a single axis, \textit{when} verbal feedback takes effect in an agent's lifecycle and \textit{what} it modifies, yielding three pillars: (1) \textbf{Language as Grounding Signal}, where language defines the task itself by specifying goals, states, and reward structures; (2) \textbf{Language as Deliberative Feedback}, where natural language guides reasoning at test time without the need to update model parameters; (3) \textbf{Language as Learning Signal}, where language-based feedback shapes model parameters through training. Within each pillar, we synthesize representative work, distinguish key subcategories of approaches, and outline the distinct role language plays in shaping agent behavior. Together, this taxonomy shows how verbal reinforcement is reshaping agent development, while also defining the challenges and opportunities for building more capable and aligned agents.
• 中文摘要 自然语言正作为提升语言代理的主要反馈渠道，能够以人类和现代语言模型都能理解的形式传达意图、偏好和因果结构。我们将这一范式称为言语强化学习（VRL），并提供了该范式的首个统一说明。我们将领域围绕一个单一轴线组织，\textit{当语言反馈在代理生命周期中生效时}，以及_textit{它所改变的内容}，产生三大支柱：（1） \textbf{语言作为基础信号}，语言通过指定目标、状态和奖励结构来定义任务本身;（2） \textbf{语言作为审议反馈}，自然语言在测试时引导推理，无需更新模型参数;（3） \textbf{语言作为学习信号}，基于语言的反馈通过训练塑造模型参数。在每个支柱中，我们综合了代表性工作，区分关键的子类别方法，并概述语言在塑造代理行为中所扮演的独特作用。综合来看，这套分类展示了言语强化如何重塑代理发展，同时定义了培养更有能力和更契合的代理的挑战与机遇。

Keyword: diffusion policy

ADAPT: Agile Diffusion Action Priors for Robust and Steerable Online Text-Driven Humanoid Control

ADAPT：敏捷扩散行动先验，实现稳健且可引导的在线文本驱动人形控制

• Authors: Yan Wu, Chenhao Li, Kaifeng Zhao, Gen Li, Marco Hutter, Siyu Tang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.00677
• Pdf link: https://arxiv.org/pdf/2609.00677
• Abstract We present ADAPT, an end-to-end framework for interactive, text-conditioned humanoid whole-body control. Unlike dominant text-to-motion pipelines that generate kinematic motions for a separate tracker, ADAPT solves language control with an end-to-end closed-loop control framework, where the robot must continuously respond to changing commands while maintaining balance, natural motion, and smooth transitions. ADAPT learns a diffusion-based action prior from text-labeled humanoid state-action trajectories, enabling diverse motion skills to be directly executed from language commands. To improve long-horizon robustness and smooth prompt switching, we train a lightweight residual reinforcement learning policy on top of the frozen diffusion controller. We further show that the same diffusion policy can be reused as a steerable text-conditioned motion prior for downstream task adaptation. Experiments demonstrate robust language-grounded skill execution, smooth interactive transitions, and style-preserving downstream control.
• 中文摘要 我们介绍了ADAPT，一个端到端的交互式文本条件人形全身控制框架。与主流的文本转动作流程生成独立追踪器的运动学运动不同，ADAPT通过端到端闭环控制框架解决语言控制问题，机器人必须在保持平衡、自然运动和平滑过渡的同时持续响应变化指令。ADAPT从文本标记的人形状态-动作轨迹中学习基于扩散的动作，使语言指令能够直接执行多样化的运动技能。为提升长视野的稳健性和平滑的提示切换，我们在冻结扩散控制器之上训练了轻量级残余强化学习策略。我们还进一步证明，同一扩散策略可作为可引导的文本条件运动先行重复使用，用于后续任务适应。实验展示了扎实的语言基础技能执行、流畅的互动过渡以及保持风格的下游控制。