生成时间: 2026-10-27 22:09:06 (UTC+8); Arxiv 发布时间: 2026-10-27 20:00 EDT (2026-10-28 08:00 UTC+8)

今天共有 72 篇相关文章

Keyword: reinforcement learning

Recursive Semantic Divergence for LLM Agent Consistency

LLM代理一致性的递归语义发散

• Authors: Harshavardhan Abichandani, Penny Chong, Atin Ghosh, Daniel Dahlmeier
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.33012
• Pdf link: https://arxiv.org/pdf/2610.33012
• Abstract Large Language Model (LLM) agents with tool use exhibit inconsistent behavior across independent runs from identical states, calling different tools, producing contradictory outputs, or pursuing divergent strategies. This variability compounds over time and undermines deployment reliability. Existing evaluation metrics primarily measure task correctness via completion-based scores and do not capture consistency in agent behavior across executions. Separately, agent behavior is analyzed either at the token level using probability distributions or at the single-response level using semantic clustering, but neither captures how inconsistency propagates across future turns of a trajectory. We introduce Recursive Semantic Divergence (RSD), an unsupervised trajectory-level consistency metric adapted from bisimulation metrics in reinforcement learning. RSD measures the expected cumulative semantic disagreement between two independent rollouts of the same agent from the same state, using natural language inference to detect logical inconsistency. It is defined recursively over future states and approximated by a neural network trained on independent rollouts. The learned metric estimates future divergence from the current state, providing a consistency score at each turn without requiring the full trajectory to complete. When used as a reward signal for fine-tuning, RSD reduces the variation in task progress from 132% to 91% on tau2-bench and from 90% to 44% on toolsandbox, while improving performance across benchmarks and model scales. Training with RSD as an additional reward signal matches supervised fine-tuning performance with substantially lower variance, without requiring ground-truth labels.
• 中文摘要 使用工具的大型语言模型（LLM）代理在相同状态下的独立运行中表现出不一致的行为，调用不同工具，产生矛盾输出，或采取发散策略。这种变异性随时间加剧，削弱部署可靠性。现有评估指标主要通过基于完成度的评分衡量任务正确性，未能捕捉执行间代理行为的一致性。另外，代理行为通过概率分布在令牌层面分析，或在单响应层面使用语义聚类进行分析，但都未能体现不一致如何在未来轨迹中传播。我们引入递归语义发散（RSD），这是一种基于强化学习中双模拟度量的无监督轨迹级一致性指标。RSD通过自然语言推断检测逻辑不一致，衡量同一代理在同一状态下两个独立展开时预期的累计语义不一致。它在未来状态上递归定义，并由训练于独立滚动的神经网络近似。该学到的指标估计未来与当前状态的分歧，在每个转折点提供一致性分数，而无需完成完整轨迹。作为微调的奖励信号使用，RSD在tau2-bench上将任务进度的变异从132%降至91%，在toolsandbox上从90%降至44%，同时提升了基准和模型尺度上的性能。作为额外奖励信号的RSD训练，使监督微调性能与显著降低的方差匹配，无需贴上真实标签。

OvisOCR3 Technical Report

OvisOCR3技术报告

• Authors: Yu Xia, Yinglun Li, Yuhui Chen, Jinlong Huang, Yuyang Wanyan, Shengze Shi, Yifei Song, Yue Duan, Shubo Lin, Jianshan Zhao, En Lin, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.33066
• Pdf link: https://arxiv.org/pdf/2610.33066
• Abstract We introduce OvisOCR3, a 0.8B model for a broad range of OCR-related tasks. OvisOCR3 unifies document parsing, general OCR, key information extraction (KIE), document figure captioning, and cross-page table reconstruction in a single model. We train the model on real and synthetic data through continued pretraining and use domain-balanced mid-training to balance supervision across tasks with different data volumes and response lengths. Multi-teacher on-policy distillation transfers expertise from five domain-specific 4B teachers into the shared model, followed by mixed-task reinforcement learning with domain-specific rewards. We compare OvisOCR3 with general-purpose vision-language models and specialized OCR models on public and in-house benchmarks covering all five task families. In these comparisons, it achieves an overall score of 96.74 on OmniDocBench v1.6 and the highest overall scores on PureDocBench, Real5-OmniDocBench, and MDPBench. On OCRBench v2, it ranks first, scoring 65.9 and 61.3 on the English and Chinese splits, respectively. It also outperforms the baselines on OCRBench and all four KIE benchmarks. On DocFigCapBench, it achieves the highest localization F1 and a caption score close to that of Qwen3.8-Max. For cross-page tables, it leads in Relation F1 on CrossTabBench and truncated-table TEDS on MPDocBench-Parse. These results demonstrate that a compact 0.8B model can achieve strong performance across diverse OCR-related tasks. OvisOCR3 is available at this https URL.
• 中文摘要 我们引入了OvisOCR3，这是一个0.8亿模型，适用于广泛的OCR相关任务。OvisOCR3将文档解析、通用OCR、关键信息提取（KIE）、文档图形说明和跨页表重建统一在一个模型中。我们通过持续的预训练，基于真实和合成数据训练模型，并在训练中途使用领域平衡，以平衡不同数据量和响应长度的任务之间的监督。多教师的策略提炼将五位领域特定4B教师的专业知识转移到共享模型中，随后是混合任务强化学习和领域特定奖励。我们将OvisOCR3与通用视觉语言模型及涵盖所有五个任务族的公共及内部基准的专业OCR模型进行比较。在这些比较中，它在 OmniDocBench v1.6 上获得了 96.74 的总分，在 PureDocBench、Real5-OmniDocBench 和 MDPBench 上获得最高总体得分。在 OCRBench v2 上，它排名第一，分别在英文和中文分割中得分为 65.9 和 61.3。它还优于 OCRBench 和所有四个 KIE 基准测试的基线数据。在 DocFigCapBench 上，它获得了最高的本地化 F1 和接近 Qwen3.8-Max 的说明得分。在跨页表方面，它在 CrossTabBench 的关系 F1 和 MPDocBench-Parse 的截断表 TED 中领先。这些结果表明，紧凑的 0.8B 模型在多种 OCR 相关任务中能够实现强劲表现。OvisOCR3 可在此 https 访问网址获取。

Of Chains and Gains: What Reinforcement Learning changes on Reasoning-based Image Quality Assessment

链条与收益：强化学习对基于推理的图像质量评估带来的改变

• Authors: Yixuan Li, Yuxuan Jiang, Hanwei Zhu, Hancheng Zhu, Fan Zhang, Shiqi Wang, David Bull
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.33203
• Pdf link: https://arxiv.org/pdf/2610.33203
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has improved the performance of large Vision-Language Models (VLMs) on image quality assessment (IQA). Existing approaches often attribute these gains to improved reasoning before scoring, supported by increasingly plausible Chain-of-Thought (CoT). However, whether these gains exceed sampling effects and how they depend on explicit reasoning remain unclear. We rigorously investigate these questions through sampling-aware evaluation, controlled reasoning interventions, and shortcut experiments. We find that RL-trained models outperform base models under large-budget $\operatorname{max}_g@k$ evaluation, while improvements beyond averaging are concentrated in-distribution. RLVR produces chains that raise base-model performance when transplanted and improves scoring with the chain held fixed. Yet scoring becomes less dependent on the chain during inference, more clearly at 7B than at 32B. Steering the probed quality direction in thinking tokens also has little effect on scoring. Finally, RL can improve scoring accuracy by exploiting textual cues correlated with mean opinion scores (MOSs), which could inflate benchmark performance. Together, these findings show that RLVR raises quality scoring and produces more useful chains, but that better scoring does not imply a stronger reliance on explicit reasoning, while score-based rewards can also encourage reliance on quality-irrelevant shortcuts. We hope this preliminary study can enlighten more and more concrete reasoning schemes in IQA.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了大型视觉语言模型（VLM）在图像质量评估（IQA）中的表现。现有方法常将这些提升归因于评分前推理能力的提升，并得到越来越合理的思维链（Chain of-Thought，CoT）支持。然而，这些提升是否超越抽样效应，以及它们如何依赖显性推理，仍不清楚。我们通过采样感知评估、受控推理干预和捷径实验，严格调查这些问题。我们发现，强化学习训练的模型在大预算的$\operatorname{max}_g@k$评估中优于基础模型，而超越平均的改进则集中在分布中。RLVR产生的链在移植时提升了基础模型性能，并在链固定时提升了评分。然而，在推理过程中评分对链的依赖性降低，7B比32B更明显。引导探询质量方向在思考代币时对评分影响不大。最后，强化学习通过利用与平均意见得分（MOS）相关的文本线索来提高评分准确性，这可能会推高基准表现。综合来看，这些发现表明RLVR提高了质量评分并产生更多有用的链条，但更好的评分并不意味着更依赖显性推理，而基于分数的奖励也可能鼓励依赖与质量无关的捷径。我们希望这项初步研究能为IQA中越来越具体的推理方案提供启发。

Alignment of Autonomous Cyber Defense Agents via Adaptive Policy Shaping

通过自适应政策塑造实现自主网络防御代理的对齐

• Authors: Dena Mujtaba, Bharadwaj Ravichandran, Brian Hu, Arslan Basharat
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2610.33234
• Pdf link: https://arxiv.org/pdf/2610.33234
• Abstract Cybersecurity operators increasingly encounter sophisticated and fast-paced attacks, making autonomous cyber defense (ACD) a critical approach for supporting timely and effective decision-making. Reinforcement learning (RL)-based ACD agents have the ability to autonomously observe network states and take actions to defend and recover system assets, making them well suited for ACD. However, these agents often face critical trade-offs, such as improving network security while inadvertently degrading service availability or violating other operational priorities such as confidentiality and integrity, raising the core challenge of aligning agent behavior with human-specified goals and intent. In this work, we study the problem of aligning RL-based ACD agents with human-specified operational objectives without requiring costly retraining. We formalize alignment for ACD as a policy shaping problem, where a pretrained RL decision-making policy can be adjusted at test time using predicted attributes over agent actions. We also propose an adaptive approach that dynamically controls the alignment--performance trade-off as operational priorities shift during an ongoing cyber attack. We evaluate our method in a simulated cyber defense environment, CybORG, analyzing performance across multiple alignment objectives using the Confidentiality, Integrity, and Availability (CIA) information security triad. Our results show that adaptive test-time policy shaping can significantly improve alignment while preserving core cyber defense performance, offering a practical and scalable alternative to training-time alignment methods in RL. Our code is publicly available at this https URL.
• 中文摘要 网络安全操作员越来越多地遇到复杂且节奏快速的攻击，使自主网络防御（ACD）成为支持及时有效决策的关键方法。基于强化学习（RL）的ACD代理能够自主观察网络状态并采取行动防御和恢复系统资产，非常适合ACD。然而，这些代理常面临关键权衡，比如在提升网络安全的同时无意中降低服务可用性或违反机密性和完整性等其他运营优先级，这带来了将智能体行为与人类指定目标和意图对齐的核心挑战。本研究中，我们研究如何在不增加高成本再培训的情况下，将基于强化学习的ACD智能体与人类指定的操作目标对齐的问题。我们将ACD的对齐形式化为一个策略塑造问题，预训练的强化学习决策策略可以在测试时通过预测属性对智能体行动进行调整。我们还提出了一种自适应方法，能够动态控制对齐——随着持续网络攻击中运营优先级的变化，性能权衡。我们在模拟网络防御环境CybORG中评估了我们的方法，利用机密性、完整性和可用性（CIA）信息安全三项分析多个对齐目标的表现。我们的结果表明，自适应测试时策略塑造能显著提升对齐性，同时保持核心网络防御性能，为强化学习中训练时对齐方法提供了实用且可扩展的替代方案。我们的代码公开于此 https URL。

Reinforcement Learning for Controller Parameters in the Auto-Bidding System at Pinterest

Pinterest 自动竞价系统中控制器参数的强化学习

• Authors: Yichu Zhou, Qinyi Chen, Fei Feng, Jacob Loh, Mehdi Ben Ayed, Lin Yang, Andrei Curelea, Jiajing Xu
• Subjects: Subjects: Machine Learning (cs.LG); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.33248
• Pdf link: https://arxiv.org/pdf/2610.33248
• Abstract Auto-bidding systems in online advertising must balance competing objectives-maximizing budget delivery while minimizing cost per action-across heterogeneous campaigns under non-stationary market conditions. This trade-off evolves over campaign lifespans: early on, cost optimization dominates when budgets roll over; near deadlines, budget delivery becomes urgent. While PID controllers are widely used in production for their stability and interpretability, fixed tunings cannot adapt to diverse campaigns and shifting traffic patterns. We present RLBP (Reinforcement Learning for Bidding Parameters), a hybrid architecture deployed at Pinterest that uses RL as a meta-controller to select campaign-specific PID parameters daily while retaining PID for high-frequency bid adjustments. RLBP trains a multi-head Q-network using a relative-lift reward formulation normalized against the production baseline, and performs inference-time fusion that adapts the cost-delivery trade-off to campaign states without retraining. Through online A/B testing at Pinterest, RLBP achieves a statistically significant reduction in effective cost-per-click (eCPC) with no revenue degradation, and two months of post-rollout measurements against a holdout also show consistent cost reductions. These results demonstrate that meta-controller RL can successfully augment production control systems at scale.
• 中文摘要 在线广告中的自动竞价系统必须在不同目标之间取得平衡——最大化预算交付，同时在非固定市场条件下最小化每次行动成本。这种权衡会随着活动生命周期的演变而变化：早期预算展期时，成本优化占主导;临近截止日期时，预算交付变得紧迫。虽然PID控制器因其稳定性和可解释性被广泛应用于生产领域，但固定调优无法适应多样化的广告活动和不断变化的流量模式。我们介绍RLBP（竞价参数强化学习），这是一种在Pinterest部署的混合架构，利用RL作为元控制器每天选择活动特定的PID参数，同时保留PID用于高频竞价调整。RLBP使用相对提升奖励公式与生产基线归一化，训练多头Q网络，并进行推理时间融合，将成本-交付权衡调整到广告活动状态，无需重新训练。通过Pinterest的在线A/B测试，RLBP在有效每次点击成本（eCPC）上实现了统计学上的显著降低，且针对未减少收入的两个月后部署测量也显示出持续的成本降低。这些结果表明元控制器RL能够成功大规模增强生产控制系统。

Concurrent Task Orchestration: Hierarchical Trusted Collaborator Group Formation for Constrained Resource Allocation

并发任务编排：基于受限资源分配的层级可信协作者组建

• Authors: Jiazhi Chen, Xianbin Wang, Xuemin "Sherman" Shen
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.33295
• Pdf link: https://arxiv.org/pdf/2610.33295
• Abstract Effective collaboration among resource-constrained mobile intelligent machines, such as Unmanned Aerial Vehicles (UAVs), is essential for the completion of complex tasks. However, concurrent task demands, limited distributed resources, and dynamic collaborator availabilities make it challenging to satisfy task-specific completion goals in diverse collaborative systems. To address these challenges simultaneously, we propose a new framework of trusted collaborator group formation and the supporting resource allocation strategy for concurrent task orchestration. Specifically, a collaborative task is decomposed into subtasks subject to dynamic resource availability and inter-subtask dependencies. A task utility model is developed for collaborator group evaluation support subtask allocation according to task completion goals and unified value-risk analysis. Based on this model, concurrent task orchestration is formulated as a long-term utility maximization problem under $\alpha$-fairness. To solve the problem efficiently, we develop a hierarchical matching strategy that first clusters tasks and collaborators into coarse-grained sets and subsequently performs fine-grained collaborator group formation and task allocation within matched sets. Task-specific trust is predicted through spatial-temporal Graph Neural Networks (GNNs) and embedded into a fairness-aware Deep Reinforcement Learning (DRL) framework to continuously update the matching policies. Simulation results demonstrate that the proposed strategy improves concurrent task completion performance by 22% over representative baselines under resource constraints.
• 中文摘要 资源受限的移动智能机器之间（如无人机）之间的有效协作对于完成复杂任务至关重要。然而，任务并发需求、分布式资源有限以及协作者动态可用性使得在多样协作系统中实现任务特定目标变得困难。为同时应对这些挑战，我们提出了一个新的可信协作组形成框架及支持任务编排的资源分配策略。具体来说，协作任务被分解为受动态资源可用性和子任务间依赖的子任务。开发了任务效用模型，用于协作组评估支持子任务，基于任务完成目标和统一的价值风险分析。基于该模型，并发任务编排被表述为在$\alpha$公平性下的长期效用最大化问题。为高效解决问题，我们开发了一种分层匹配策略，先将任务和协作者聚集成粗粒度集合，随后在匹配集合内进行细粒度协作组的组建和任务分配。任务特定信任通过时空图神经网络（GNN）预测，并嵌入公平意识深度强化学习（DRL）框架，持续更新匹配策略。模拟结果表明，在资源约束下，所提策略比代表基线提升22%的并发任务完成表现。

An Empirical Study of Reinforcement Learning Scaling Laws

强化学习尺度定律的实证研究

• Authors: Zeyu Zheng, Clare Lyle, Khimya Khetarpal, Will Dabney
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.33420
• Pdf link: https://arxiv.org/pdf/2610.33420
• Abstract The discovery of scaling laws enables prediction of the performance, the optimal resource allocation strategy, and the best hyperparameter configuration of a large-scale training job based on inexpensive experiments at smaller scales. Guided by these empirical scaling laws, the deep learning community has achieved great success in training large (self-)supervised models on Internet-scale datasets. In contrast, despite its potential and wide application, reinforcement learning (RL) has yet to find a guiding principle for predictable scaling. In this work, we conduct an empirical study of RL scaling laws of an actor-critic algorithm in two procedurally generated environments, Starpilot and Craftax. We discover empirical scaling laws with strong predictive performance in both environments. The discovered scaling laws accurately predict the aforementioned three quantities for larger-scale training jobs up to two orders of magnitude in total training compute. In particular, we employ a variant of Elo scores for the single-agent RL paradigm as an alternative performance metric than the commonly used mean episode return. Our results show that the Elo score is a much more predictable metric than the mean episode return, making it a better choice for fitting scaling laws. By applying our discovered scaling laws to increase training compute, we establish a new state of the art in the challenging Craftax environment. Beneath the surface of the global scaling trend, we also observe a non-monotonic scaling pattern of the compute-optimal model size in Craftax. We connect this observation to a pathology known as Ray interference.
• 中文摘要 缩放定律的发现使得基于成本实验在更小尺度上预测大规模训练任务的性能、最优资源分配策略和最佳超参数配置成为可能。在这些经验缩放定律的指导下，深度学习社区在互联网规模数据集上训练大型（自监督）模型方面取得了巨大成功。相比之下，尽管强化学习潜力巨大且应用广泛，但尚未找到可预测缩放的指导原则。本研究中，我们对Starpilot和Craftax两个程序生成环境中actor-critic算法的强化学习缩放定律进行了实证研究。我们发现了在两种环境中具有强预测性能的经验缩放定律。发现的缩放定律准确预测了上述三个量，适用于较大规模训练任务，总训练计算量高达两个数量级。特别是，我们采用了单智能体强化学习范式的Elo分数变体，作为替代常用平均集数回报的性能指标。结果显示，Elo分数比平均集数回报更为可预测，使其成为拟合缩放定律的更好选择。通过应用我们发现的缩放定律来提升训练计算量，我们在充满挑战的Craftax环境中建立了新的技术水平。在全球缩放趋势的表面之下，我们还观察到Craftax中计算最优模型规模的非单调缩放模式。我们将这一观察与一种称为射线干扰的病理现象联系起来。

PinGER: Token-Efficient Generative Retrieval with Semantic IDs

PinGER：带语义ID的高效令牌生成检索

• Authors: Xinyi Li, Sujan Reddy Ale, Yi-Ping Hsu, Zelun Wang, Yilin Chen, Hanyu Li, Hanlin Lu, Arkin Dharawat, Lulu Yao, Yaxin Li, Bella Huang, Xiangyi Chen, Jaewon Yang, Jiajing Xu, Charles Rosenberg
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.33435
• Pdf link: https://arxiv.org/pdf/2610.33435
• Abstract Generative retrieval replaces nearest-neighbor search with token-by-token decoding of an item's semantic ID (SID). But existing SID-based methods primarily use L levels of tokens sequentially to represent each item in the input history. As a result, a fixed context window holds L times fewer items than it would with one embedding per item. We present PinGER, a generative retriever deployed at Pinterest that retains SID outputs while making input histories compact, built on three ideas. (1) An asymmetric attention mask over interleaved SID and dense tokens lets later items attend only to the dense ones, so the model is supervised at every SID position in training yet serves one token per item while computing the same function. (2) Generation is controllable through goal-conditioned outputs, reinforcement learning, and a prefix-diversity penalty, which together steer retrieval toward surface-specific goals and diverse candidates. (3) A serving stack built for efficient wide-beam generation is paired with decoding constrained to valid identifiers and candidate resolution from a host-local index. Deployed in Related Pins, PinGER lifts saves of fresh Pins by 3.60% and overall saves by 0.62%, while reducing the surface's end-to-end latency by 0.7% and eliminating the cost of the nearest-neighbor search it replaces.
• 中文摘要 生成检索用逐词解码词义ID（SID）来取代最近邻搜索。但现有基于SID的方法主要依次使用L级词来表示输入历史中的每个项目。因此，固定上下文窗口存储的项目数量是每项嵌入一个词的L倍。我们介绍PinGER，一款在Pinterest部署的生成式检索器，它保留SID输出同时使输入历史紧凑，基于三个理念。（1）在交织SID和稠密令牌上的非对称注意力掩码允许后续项目只关注密度较高的标记，因此模型在训练中每个SID位置都被监督，同时在计算相同函数时为每个项目服务一个令牌。（2） 生成可通过目标条件输出、强化学习和前缀多样性惩罚控制，这些方法共同引导检索朝向表面特定目标和多样化候选对象。（3） 为高效宽束生成构建的服务栈与受限于有效标识符和宿主-本地索引候选解析的解码配合。部署于相关引脚中，PinGER可提升新引脚保存率3.60%，整体保存率降低0.62%，同时降低表面端到端延迟0.7%，并消除替代最近邻搜索的成本。

A Comprehensive Framework for All Stages of Large-Model Training: From Data and Tokenization, through Pretraining, to Post-Training Alignment

一个涵盖大型模型训练各阶段的综合框架：从数据和标记化，到预训练，再到训练后对齐

• Authors: Xiangjing An, Zhibin Jiao, Aiting Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.33451
• Pdf link: https://arxiv.org/pdf/2610.33451
• Abstract The capabilities of large language models (LLMs) and multimodal LLMs (MLLMs) arise from the synergy of the full training pipeline, including data and tokenization, pretraining, and post-training (supervised fine-tuning, preference alignment, reinforcement learning, and continual optimization), rather than any individual stage. Existing surveys mostly adopt a single-stage or single-technique perspective, treating related research as isolated systems without unified stage division or a horizontally comparable basis. This paper proposes a unified stage division and a six-dimensional analytical framework covering data, objective function, optimization strategy, system constraints, evaluation, and risk and governance. We systematically review 395 core references from 2017 to 2026, and present a conceptual map of cross-stage effects: forward constraints of pretraining on post-training, and reverse empowerment of post-training on data and pretraining. We derive three core findings. First, the comparability of scaling laws depends on data quality and mixture, a long-neglected variable. Second, preference alignment methods (RLHF, DPO) and reasoning reinforcement learning with verifiable rewards redefine post-training boundaries, while the Pass@1 versus Pass@K debate remains unresolved. Third, multimodal training poses structural challenges to the pure-text paradigm in tokenizer design, training objectives, and evaluation. The paper provides a cross-stage effects diagram, a method-family map, a practical decision tree, and at least 10 verifiable open problems, with an extensible skeleton for focused sub-surveys per stage. As of the retrieval date, peer-reviewed evidence (A+B) accounts for approximately 79% of the 395 references.
• 中文摘要 大型语言模型（LLMs）和多模态LLMs（MLLM）的能力源于完整训练流程的协同作用，包括数据和标记化、预训练以及训练后（监督微调、偏好对齐、强化学习和持续优化），而非单阶段。现有调查大多采用单阶段或单一技术视角，将相关研究视为无统一阶段划分或横向可比基础的孤立系统。本文提出统一阶段划分及涵盖数据、目标函数、优化策略、系统约束、评估以及风险与治理的六维分析框架。我们系统地回顾了2017年至2026年的395个核心参考文献，并呈现了跨阶段效应的概念图：预训练的前向约束对训练后，以及对数据和预训练的反向赋权。我们得出三个核心发现。首先，尺度定律的可比性依赖于数据质量和混合性，这一长期被忽视的变量。其次，偏好对齐方法（RLHF、DPO）和带有可验证奖励的推理强化学习重新定义了训练后边界，而Pass@1与Pass@K的争论仍未解决。第三，多模态训练对纯文本范式在分词器设计、训练目标和评估中构成结构性挑战。论文提供了跨阶段效应图、方法家族图、实用决策树，以及至少10个可验证的未解决问题，并为每个阶段的聚焦子调查提供可扩展骨架。截至检索日期，同行评审证据（A+B）约占395篇参考文献的79%。

Qwen-Image-PE: Learning Visual Design for Text-to-Image Prompt Enhancement

Qwen-Image-PE：学习文本转图像提示增强的视觉设计

• Authors: Zekai Zhang, Yanran Zhang, Shengming Yin, Yan Shu, Zhening Liu, Bingxuan Zhao, Chenyang Zhu, Donghao Zhou, Jiahao Li, Jiahao Wang, Jie Zhang, Kaiyuan Gao, Kun Yan, Lihan Jiang, Ningyuan Tang, Qihang Ai, Rui Song, Shuo Cao, Xiao Xu, Xiaoyue Chen, Yuxiang Chen, Zhendong Wang, Zhouyang Li, Zihao Liu, Zikai Zhou, Bo Zheng, Huishuai Zhang, Dongyan Zhao, Chenfei Wu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.33472
• Pdf link: https://arxiv.org/pdf/2610.33472
• Abstract Learning visual design is essential for prompt enhancement (PE) to turn underspecified user requests into coherent visual specifications for text-to-image (T2I) generation. Existing methods preserve user intent and improve overall visual appeal, with less explicit emphasis on fine-grained design dimensions. To address this gap, we introduce Qwen-Image-PE, which learns visual design through two-stage training. Supervised fine-tuning with Caption-Based Reverse Distillation (CRD) uses dense captions of well-designed images as design supervision. Reinforcement learning with Groupwise Design Reward (GDR) jointly compares all candidate images for the same request to refine lighting, composition, layout, and detail. On Qwen-Image-Bench with Qwen-Image-2.1, Qwen-Image-PE-397B improves over original requests by 7.81 points, while Qwen-Image-PE-9B outperforms all evaluated PE baselines. Further evaluations show gains in visual design, aesthetics, image quality, and human preference. We find that PE scales along three axes: training data, model size, and information density, with sufficient information density as a prerequisite for effective enhancement. Moreover, we demonstrate that Qwen-Image-PE generalizes to four additional T2I backbones, even surpassing native prompt enhancers on HunyuanImage-2.1 and FLUX 3 Image Klein. We release Qwen-Image-PE-9B at this https URL.
• 中文摘要 学习视觉设计对于提示增强（PE）至关重要，将用户请求转化为文本转图像（T2I）生成的连贯视觉规范。现有方法保留用户意图并提升整体视觉吸引力，较少显性地强调细粒度设计维度。为弥补这一空白，我们引入了Qwen-Image-PE技术，通过两阶段训练学习视觉设计。基于字幕的逆向蒸馏（CRD）监督微调通过设计良好图像的密集字幕作为设计监督。强化学习与分组设计奖励（GDR）联合比较同一请求的所有候选图像，以精细化光照、构图、布局和细节。在Qwen-Image-Bench配合Qwen-Image-2.1上，Qwen-Image-PE-397B比原始请求提升7.81分，Qwen-Image-PE-9B则优于所有评估的PE基线。进一步评估显示，PE在视觉设计、美学、图像质量和人类偏好方面均有提升。我们发现PE在三个轴上可扩展：训练数据、模型大小和信息密度，且有效增强的前提是足够的信息密度。此外，我们证明Qwen-Image-PE可推广至四个额外的T2I骨干，甚至超过HunyuanImage-2.1和FLUX 3 Image Klein的原生提示增强器。我们将Qwen-Image-PE-9B发布至此https URL。

MDPO: Multi-constraint Dexterous Policy Optimization with Reinforcement Learning

MDPO：多约束灵活策略优化与强化学习

• Authors: Ao Zhou, Zongyou Yu, Ke Wu, Tao Xu, Junqiao Zhao, Wenchao Ding, Alois Knoll, Guang Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.33480
• Pdf link: https://arxiv.org/pdf/2610.33480
• Abstract Dexterous manipulation requires high-dimensional control policies to accomplish complex contact-rich tasks while simultaneously satisfying multiple safety requirements, such as joint limits, unsafe finger configurations, arm collisions, and unstable object interactions. Constrained reinforcement learning (RL) provides a principled framework for incorporating these requirements into dexterous manipulation policy learning. However, existing constrained RL methods for dexterous manipulation primarily focus on cumulative cost constraints and struggle to simultaneously handle multiple heterogeneous constraints during dexterous policy learning. In this work, we propose Multi-constraint Dexterous Policy Optimization (MDPO) for safe dexterous manipulation. We first formulate dexterous manipulation as a multi-constraint policy optimization problem involving both cumulative cost constraints and state-wise cost constraints. Based on this formulation, we integrate multi-constraint projection into the dexterous manipulation policy optimization process. Specifically, we employ constraint-specific policy projections to map the intermediate policy that maximizes task performance onto the feasible regions defined by the cumulative cost constraint and the state-wise cost constraint. Extensive experiments on ReDMan, including safe joint control, finger safety, and grasping tasks, demonstrate that MDPO achieves superior constraint satisfaction while maintaining competitive task performance compared with the baselines. Our code is available at this link.
• 中文摘要 灵巧操作需要高维控制策略，以完成复杂且接触丰富的任务，同时满足多重安全要求，如关节限制、不安全手指配置、手臂碰撞和不稳定物体交互。受限强化学习（RL）为将这些需求纳入灵巧操作策略学习提供了原则框架。然而，现有的受限强化学习方法主要关注累积成本约束，难以同时处理多个异构约束。本研究提出多约束灵巧策略优化（MDPO）用于安全灵巧操作。我们首先将灵巧操作表述为一个涉及累积成本约束和状态成本约束的多约束策略优化问题。基于该表述，我们将多约束投影整合进灵巧操作策略优化过程。具体来说，我们采用约束特定策略预测，将最大化任务性能的中间策略映射到累积成本约束和状态成本约束定义的可行区域。在ReDMan上的广泛实验，包括安全关节控制、手指安全和抓取任务，表明MDPO在保持竞争任务性能的同时，相较于基线实现了更优的约束满足。我们的代码可在此链接获取。

Learning Dexterous Manipulation: Hands, Simulators, Training, and Evaluation

学习灵巧操作：双手、模拟器、训练与评估

• Authors: Luai Abuelsamen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.33486
• Pdf link: https://arxiv.org/pdf/2610.33486
• Abstract A dexterous hand can change an object's pose without setting it down. This survey covers 113 learning-based methods that attempt it, the hands they run on, the simulators they train in, and how they report results. The field has largely converged on one recipe: reinforcement learning on privileged state in a GPU simulator, distilled into a vision-only student for transfer to hardware. Almost half the corpus trains that way, nearly all of it in one simulator, whose own successors are barely used. A second and faster-growing branch learns from teleoperated demonstration instead, through vision-language-action models, diffusion policies, and systems whose contribution is the collection rig itself. Published work has tripled since 2022. Grasping remains the largest task family and bimanual coordination the fastest-rising, and almost every experiment runs on one of four hands. What the survey adds comes from reading released code and vendor specifications beside the papers. Eight hands that can be bought or built from published designs appear in no method paper here. Where the depth a hand sinks into the object it holds is reported at all, it is not measured in the physics that produced the motion. Section VII proposes a protocol for the unreported quantities; the body gives the counts behind every claim above.
• 中文摘要 一只灵巧的手可以在不放下物体的情况下改变物体的姿势。本次调查涵盖了113种基于学习的方法，尝试了这些方法，包括它们使用的手、训练的模拟器以及它们如何报告结果。该领域基本上趋同于一个方案：在GPU模拟器中特权状态进行强化学习，提炼成仅具视觉能力的学生，移植到硬件上。近一半的语料库都采用这种方式训练，几乎全部集中在一个模拟器中，而其后续模拟器几乎未被广泛使用。另一个且增长更快的分支则通过远程操作演示学习，通过视觉-语言-动作模型、扩散策略以及以收集设备本身为贡献的系统。自2022年以来，已发表的研究数量增长了三倍。抓握仍然是最大的任务家族，双手协调是增长最快的，几乎所有实验都依赖四手中的一只进行。调查新增的内容来自于阅读与论文并列的已发布代码和厂商规范。这里没有方法论文中出现八只可购买或制作的手，这些设计已发表。如果手沉入物体的深度被报告，则不在产生运动的物理学中测量。第七节提出了未报告量的方案;正文提供了上述每个主张背后的计数。

RBGA: Residual-Budget Guide Arbitration for Offline-to-Online Safe Reinforcement Learning in Dexterous Manipulation

RBGA：离线到线上灵巧操作安全强化学习的剩余预算仲裁指南

• Authors: Zongyou Yu, Ao Zhou, Xiongbo Shi, Jiayi Guan, Shihang Du, Xudong Zhang, Shangding Gu, Junqiao Zhao, Hanli Wang, Guang Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.33494
• Pdf link: https://arxiv.org/pdf/2610.33494
• Abstract Dexterous manipulation requires robots to leverage existing datasets for rapid skill acquisition while maintaining safety during real-world deployment to prevent damage to the robot or its surroundings. Offline-to-online safe reinforcement learning (O2O safe RL) provides a promising solution for this problem by leveraging prior experience for online improvement under safety constraints, thereby mitigating the sample inefficiency and exploration risk of learning safe policies from scratch. However, existing O2O safe RL approaches do not explicitly account for the safety budget already consumed when making decisions within an episode. Excessive cost incurred early can therefore leave little room for subsequent exploration and policy improvement. To address this limitation, we propose \textbf{R}esidual-\textbf{B}udget \textbf{G}uide \textbf{A}rbitration (RBGA), an algorithm that explicitly tracks the remaining episode safety budget and uses it to adaptively select between offline guidance and the evolving online policy. At each step, RBGA compares the cost-to-go estimates of the guide and online actions against the remaining budget, enabling step-wise action arbitration throughout the episode. We provide theoretical analysis characterizing the safety implications of residual-budget-based selection and its effect on guide-assisted exploration. Extensive experiments demonstrate that RBGA enables effective online policy improvement while maintaining strong safety performance. Our code is available at this \href{this https URL}{link}.
• 中文摘要 灵巧操作要求机器人利用现有数据集快速习得技能，同时在实际部署时保持安全，防止机器人或其环境受损。离线到在线安全强化学习（O2O安全强化学习）为该问题提供了有前景的解决方案，利用先前经验在安全约束下在线改进，从而降低从零学习安全策略时的样本低效和探索风险。然而，现有的O2O安全强化学习方法并未明确考虑在事件中决策时已消耗的安全预算。因此，早期产生的过高成本将为后续探索和策略改进留有极少空间。为解决这一限制，我们提出了 \textbf{esidual-\textbf{B}udget \textbf{G}uide \textbf{A}rbitration（RBGA）算法，该算法明确跟踪剩余的节目安全预算，并利用它在离线指导和不断演变的在线政策之间进行自适应选择。在每个步骤，RBGA 将指南和在线行动的成本估算与剩余预算进行比较，从而实现全集的分阶段行动仲裁。我们提供了理论分析，描述了基于剩余预算选择的安全影响及其对引导辅助探索的影响。大量实验表明，RBGA 能够有效改进在线策略，同时保持强劲的安全性能。我们的代码可在 \href{this https URL}{link} 获取。

Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

吃掉你的精准：精准目标缓解基于评分标准的强化学习中的奖励黑客问题

• Authors: Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, Eunho Yang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.33512
• Pdf link: https://arxiv.org/pdf/2610.33512
• Abstract Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).
• 中文摘要 基于评分标准的奖励使得对不可验证任务进行有效的强化学习。然而，由于这些奖励通常被定义为评分标准覆盖度的加权度量，模型主要训练以最大化回忆目标。因此，强化学习训练的模型产生的反应更完整，但也更冗长，包含更多不必要的或事实错误的内容，因此它们的偏好度低于基础模型的回答。为解决这一问题，我们提出了FORK奖励，它结合了回忆优化和一种新颖的精度目标，后者要求模型反应的每个单位既是必要且事实正确的，分别通过评分标准驱动的必要性积分和事实正确性评分器来衡量。我们的实验结果表明，FORK优于现有基于评分标准的强化学习方法，并实现了帕累托最优性能，平衡了评分标准覆盖率和整体响应质量。与最初基于回忆的奖励相比，FORK大幅提升了准确度，缩短了响应时长，同时实现了相似的评分标准覆盖率，这导致胜率显著高于基础政策（例如，RubricHub-Medical的胜率从10.6%降至68.0%，HealthBench从22.1%降至74.1%）。

HA-DiPO: Diffusion Policy Optimization for Heterogeneous Cooperative Transport

HA-DiPO：异构协同运输的扩散策略优化

• Authors: Xuebin Li, Jinpeng Xiao, Zong Chen, Pierluigi Arpenti, Bruno Siciliano, Jiebao Sun, Yiqun Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.33584
• Pdf link: https://arxiv.org/pdf/2610.33584
• Abstract Cooperative transport with heterogeneous legged robots requires coordinating different motion capabilities while supporting an unfixed shared payload. We present HA-DiPO (Heterogeneous-Agent Diffusion Policy Optimization), a hierarchical multi-agent reinforcement learning framework under centralized training with decentralized execution (CTDE). The framework decouples team-level coordination from embodiment-specific motion control: at the high level, a centralized critic uses team-value estimates to jointly train independent coordination policies, aligning robot-specific action responses with a shared transport objective; at the low level, embodiment-specific robust motion policies track the coordination commands. The high-level diffusion policies offer more flexible action distributions than single-Gaussian parameterizations, allowing potentially multimodal coordination responses to be represented. The coordination policies learn from scratch through reward-driven simulator interaction without demonstrations. At execution, each robot generates coordination commands using only local robot and payload observations, without partner-state inputs or inter-robot messages. Simulation comparisons show that HA-DiPO achieves higher final mean returns and velocity-tracking success rates, as well as lower velocity errors, than the evaluated baselines. Real-world experiments further demonstrate quadrupedal-wheeled-legged and human-robot cooperative transport.
• 中文摘要 与异构腿部机器人的协作传输需要协调不同的运动能力，同时支持非固定共享有效载荷。我们介绍HA-DiPO（异构代理扩散策略优化），这是一个基于集中训练与去中心化执行（CTDE）的分层多智能体强化学习框架。该框架将团队级协调与具身特定运动控制解耦：在高层面，中心化批评者利用团队价值估计联合训练独立协调策略，将机器人特定动作响应与共享传输目标对齐;在低层，具象特定稳健运动策略跟踪协调指令。高层扩散策略提供比单高斯参数化更灵活的动作分布，允许表示潜在的多模态协调响应。协调策略通过奖励驱动的模拟器交互从零学习，无需演示。执行时，每个机器人仅利用本地机器人和有效载荷观测生成协调指令，不涉及伙伴状态输入或机器人间消息。模拟比较显示，HA-DiPO在最终平均回报和速度跟踪成功率方面均高于评估基线，速度误差更低。真实实验进一步展示了四足轮式和人机协作运输。

Physics-Informed Graph-Based Safe RL with Constraint-Specific Critics for Community Battery Coordination in Distribution Network

基于物理的基于图的安全强化学习，并针对配电网络中社区电池协调提供约束特定批评

• Authors: Chenghao Huang, Yue Yang, Hao Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Systems and Control (eess.SY); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2610.33654
• Pdf link: https://arxiv.org/pdf/2610.33654
• Abstract High penetrations of distributed energy resources (DERs) and fast-varying electric vehicle (EV) charging intensify voltage and line-limit violations in distribution networks (DNs), making safe, low-cost operation increasingly challenging. This paper proposes a physics-informed safe deep reinforcement learning (DRL) framework for coordinating community battery energy storage systems to mitigate constraint violations while minimizing operating cost. To represent nonlocal spatiotemporal interactions, we develop a dual-level physics-informed graph representation in which line impedance and capacity shape spatial information propagation, while power-balance, voltage-profile, and flow-reconstruction losses regularize the latent state toward control-relevant physical consistency. Building on this representation, we develop Lagrangian Dedicated-Critic PPO (LDCPPO), which implements constraint-specific value decomposition: independent reward and cost critics preserve heterogeneous voltage- and line-loading-related learning signals and combine their advantages only during the Lagrangian policy update. Case studies on DNs with high rooftop photovoltaic penetration and EV charging demonstrate that the proposed method reduces the key voltage-violation degree by up to 80.5% compared with the tested baselines, while also achieving the lowest voltage-violation count and line-overload degree with competitive operating cost.
• 中文摘要 分布式能源资源（DER）和快速变化电动汽车（EV）充电的高渗透加剧了配电网络（DN）中的电压和线路限制违规，使安全低成本运行变得越来越具有挑战性。本文提出了一个基于物理的安全性深度强化学习（DRL）框架，用于协调社区电池储能系统，以减轻约束违规并最小化运行成本。为表示非局域时空交互，我们开发了一种双层物理驱动的图表示法，其中线阻抗和容量决定空间信息传播，而功率平衡、电压剖面和流重建损耗则规范潜态趋向控制相关物理一致性。基于该表示，我们开发了拉格朗日专用批判者PPO（LDCPPO），实现了约束特定值分解：独立的奖励和成本批评者保留异构电压和线路负载相关的学习信号，并仅在拉格朗日政策更新时结合其优势。高屋顶光伏渗透率和电动车充电的DN案例研究表明，所提方法将关键电压违规程度降低了高达80.5%，同时以具有竞争力的运营成本实现了最低的电压违规次数和线路过载程度。

SWE-Ladder: Learning to Localize, Reproduce, and Patch via Progressive Privileged Self-Distillation

软件阶梯：学习通过渐进特权自我蒸馏的定位、繁殖和补丁

• Authors: Xudong Cai, Zhengxi Lu, Yuchen Yan, Jian Shao, Jun Xiao, Yueting Zhuang, Yongliang Shen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.33667
• Pdf link: https://arxiv.org/pdf/2610.33667
• Abstract Resolving a repository-level issue typically requires a software-engineering (SWE) agent to localize the faulty code, reproduce the failure, and patch the code. Reinforcement learning trains such agents from a single test result on the final patch, while on-policy self-distillation (OPSD) adds token-level supervision from a teacher with privileged evidence, exposing the full reference patch at every step. Neither provides stage-specific guidance for localization or reproduction. We propose SWE-Ladder, which turns the verified repair of each training instance into stage-specific evidence: the changed files and lines for localization, the fail-to-pass tests for reproduction, and the diff for patching. A rule-based router assigns each step of an end-to-end rollout to a stage based on its tool call, a frozen teacher sees only the matching evidence, and the stages are distilled in repair order while GRPO optimizes the full trajectory. With Qwen3-4B-Instruct-2507 on SWE-bench Verified under the R2E-Gym scaffold, SWE-Ladder improves pass@1 from 9.6% to 16.4%, compared with 10.4% for GRPO and 12.4% for OPSD with the full patch. Under the OpenHands scaffold, it improves pass@1 from 13.0% to 17.6%. Reversing the stage order under R2E-Gym reduces pass@1 to 11.4%.
• 中文摘要 解决仓库级问题通常需要软件工程（SWE）代理本地化错误代码、重现失败并修补代码。强化学习通过最终补丁的单一测试结果训练此类代理，而策略自蒸馏（OPSD）则通过拥有特权证据的教师进行标记级监督，在每个步骤都暴露完整的参考补丁。这两种方法都不提供阶段特定的本地化或重现指导。我们提出SWE-Ladder，将每个训练实例的已验证修复转化为阶段特定证据：本地化时修改的文件和行，重现失败测试，补丁时差异。基于规则的路由器根据工具调用将端到端推广的每个步骤分配给某个阶段，冻结教师只看到匹配的证据，阶段按修复顺序提炼，同时GRPO优化完整轨迹。在 R2E-Gym 架构下，SWE-bench 验证的 Qwen3-4B-Instruct-2507 提升了 pass@1 从9.6%提升到16.4%，而完整补丁后 GRPO 为 10.4%，OPSD 为 12.4%。在 OpenHands 支架下，其pass@1从13.0%提升至17.6%。在 R2E-Gym 下反转阶段顺序，pass@1降至11.4%。

Human-Feedback-Guided Multi-Agent Inverse Reinforcement Learning for Interactive Driving Behavior Modeling

人类反馈引导的多智能体逆向强化学习用于交互式驾驶行为建模

• Authors: Longhao Yan, Jinhao Liang, Kaidi Yang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.33726
• Pdf link: https://arxiv.org/pdf/2610.33726
• Abstract Modeling human driving behavior is fundamental to understanding driver decision-making and supporting autonomous vehicles (AVs) in mixed traffic. Reward learning offers a principled way to infer the latent objectives behind driving behaviors, yet most existing methods are formulated in single-agent settings and treat surrounding vehicles as dynamic obstacles. Directly extending reward learning to multi-agent scenarios is difficult because the joint action space grows exponentially with the number of interacting vehicles. In addition, existing approaches mainly rely on physical demonstrations and rarely exploit human feedback during reward recovery. To address these limitations, this paper proposes a Human-Feedback-Guided Multi-Agent Inverse Reinforcement Learning (HFG-MAIRL) framework for interactive driving behavior modeling. First, we develop a multi-agent maximum-entropy inverse reinforcement learning module that explicitly models each vehicle and its directly interacting neighbors in a localized Markov game, while using aggregated traffic states to capture the influence of the broader traffic environment. The resulting coupled multi-vehicle responses are approximated through finite marginal-response iterations over factorized vehicle-level policies, avoiding explicit enumeration of the full joint action space. Second, we incorporate human feedback as ordinal preference supervision through a Bradley-Terry likelihood. In our experiments, driver emotion labels are used as one instantiation of such feedback and converted into pairwise ordinal preference constraints. Experiments on the AIDE dataset show that HFG-MAIRL outperforms representative IRL baselines in decision modeling and produces interpretable feedback-aligned reward patterns.
• 中文摘要 建模人类驾驶行为对于理解驾驶员决策和支持混合交通中的自动驾驶汽车（AV）至关重要。奖励学习提供了一种原则性的方法来推断驾驶行为背后的潜在目标，但大多数现有方法在单智能体环境中构建，并将周围车辆视为动态障碍。直接将奖励学习扩展到多智能体场景较为困难，因为联合行动空间随着交互车辆数量呈指数增长。此外，现有方法主要依赖物理演示，很少在奖励恢复过程中利用人类反馈。为解决这些局限性，本文提出了一种人反馈引导多智能体逆向强化学习（HFG-MAIRL）框架，用于交互式驾驶行为建模。首先，我们开发了一个多代理最大熵逆强化学习模块，在局部马尔可夫博弈中显式建模每辆车辆及其直接交互的邻居，同时利用聚合的交通状态捕捉更广泛交通环境的影响。通过对因数分解车辆级策略进行有限边际响应迭代，近似生成耦合多车响应，避免显式枚举完整的联合行动空间。其次，我们将人为反馈作为Bradley-Terry似然的序数偏好监督。在实验中，驾驶员情绪标签作为反馈的实例化，并转换为成对序数偏好约束。AIDE数据集上的实验表明，HFG-MAIRL在决策建模中优于代表性的真实基线，并产生可解释的反馈对齐奖励模式。

A Distribution-Matching Perspective on LLM Training Algorithms

分布匹配视角下的大型语言模型训练算法

• Authors: Yanxing Huang, Bingzhang Yan, Zihan Tang, Wenzhe Li, Peng Li, Yang Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.33740
• Pdf link: https://arxiv.org/pdf/2610.33740
• Abstract In this work, we propose a unified framework for training algorithms across different stages of modern large language model development, based on the perspective of distribution matching. We show that a broad class of language model training algorithms can be interpreted as minimizing a certain divergence between the current model and a target distribution, and that characterizing the properties of this target distribution provides substantial insight into the behavior of the corresponding training algorithms. By analyzing specific forms of the target distribution, we establish that On-Policy Distillation (OPD) enjoys an advantage over classical Knowledge Distillation (KD) when the current model distribution is close to the target distribution, while exhibiting a disadvantage when the two distributions are far apart. Furthermore, when the target distribution itself depends on the model being optimized, we show that the accumulation of estimation variance can induce a deterministic degradation of the target distribution. These results provide a theoretical explanation, at least in part, for the long-term behavior of existing Reinforcement Learning (RL) algorithms. With this intuition we further designed Conditional KL (CKL) regularization for mitigating the degradation of the target distribution. CKL can effectively preserve mode diversity while relaxing the constraint imposed by classical KL regularization on reward improvements. We believe this theoretical perspective can bring some new understanding of language model training algorithms and provide useful guidance for future research.
• 中文摘要 本研究基于分布匹配视角，提出了跨现代大型语言模型开发不同阶段训练算法的统一框架。我们展示了一类广泛的语言模型训练算法可被解释为最小化当前模型与目标分布之间的一定差异，且对该目标分布的特性特性的表征为对应训练算法的行为提供了重要洞察。通过分析目标分布的具体形式，我们确定当当前模型分布接近目标分布时，策略上蒸馏（OPD）相较于经典知识蒸馏（KD）具有优势，而当两者分布相距较远时则存在劣势。此外，当目标分布本身依赖于被优化模型时，我们表明估计方差的累积可以诱导目标分布的确定性退化。这些结果至少部分为现有强化学习（RL）算法的长期行为提供了理论解释。基于这一直觉，我们进一步设计了条件KL（CKL）正则化，以减轻目标分布的退化。CKL可以在放松经典KL正则化对奖励改进的约束的同时，有效保持模态多样性。我们相信，这一理论视角能够带来对语言模型训练算法的新理解，并为未来研究提供有用指导。

Computer Vision Navigation for an Autonomous Vehicle Model Using Reinforcement Learning

基于强化学习的自动驾驶车辆模型计算机视觉导航

• Authors: Dinis Fernandes, Tiago Ribeiro, António Ribeiro, Carolina Lopes, Gil Lopes, A.Fernando Ribeiro
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.33755
• Pdf link: https://arxiv.org/pdf/2610.33755
• Abstract This work aims to assess the capabilities of the reinforcement learning algorithm, Proximal Policy Optimization (PPO), in developing a policy capable of controlling the speed and steering angle of a vehicle model to navigate within simulated road lines. The control system's input is a camera, and its outputs are the rotational speed of the rear wheels and the steering angle of the front wheels, controlling the vehicle's speed and direction, respectively. The algorithm is trained and tested in a simulated environment, where it receives image frames from the vehicle's camera, processes them through a feature extraction algorithm, and feeds them into the policy. For each frame, the policy outputs speed and direction commands for the vehicle. The policy is iteratively updated to enhance its performance. The reinforcement learning algorithm successfully generates a policy capable of navigating the vehicle in a simplified environment. Potential improvements are suggested to enhance the system's robustness in more complex scenarios. Creating a simulation environment suitable for testing an advanced reinforcement learning algorithm is a complex task. This project offers a framework for other developers to test different approaches and algorithms. The tested algorithm, PPO, uses proximity intervals - an innovative approach in reinforcement learning. In some scenarios, the algorithm achieved better performance than predecessor algorithms, which are typically harder to implement and require tuning more hyperparameters.
• 中文摘要 本研究旨在评估强化学习算法——近端策略优化（PPO）在开发能够控制车辆模型在模拟道路线内导航速度和转向角度的策略中的能力。控制系统的输入是摄像头，输出为后轮转速和前轮转向角，分别控制车辆速度和方向。该算法在模拟环境中训练和测试，接收车辆摄像头的图像帧，通过特征提取算法处理，并输入策略。对每个帧，策略输出车辆的速度和方向指令。策略会迭代更新以提升其性能。强化学习算法成功生成能够在简化环境中导航车辆的策略。建议改进以增强系统在更复杂场景下的稳健性。创建一个适合测试高级强化学习算法的仿真环境是一项复杂的任务。本项目为其他开发者提供了一个测试不同方法和算法的框架。被测试的算法PPO采用了邻近区间——这是一种强化学习中的创新方法。在某些情况下，该算法的性能优于前代算法，而前代算法通常更难实现，需要调整更多超参数。

The Hitchhiker's Guide to System 1 Models: Fast Decision-Making Beyond LLMS

System 1模型搭便车指南：超越LLMS的快速决策

• Authors: Haggai Roitman
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.33775
• Pdf link: https://arxiv.org/pdf/2610.33775
• Abstract Large Language Models excel at reasoning and generation but their autoregressive, token-by-token inference makes them unnecessarily slow, expensive, and poorly calibrated for the many routine decisions (routing, gating, scoring, classification) that modern AI systems require at sub-second latency. A growing class of models addresses this gap: System 1 decision models, architectures that map an input directly to a typed, structured decision in a single forward pass. This guide provides a comprehensive, practitioner-oriented treatment of the field. We propose a five-family taxonomy covering encoder-based classifiers, reward models, non-autoregressive decision engines, contrastive decision models, and extraction models, and we provide in-depth coverage of each family's architecture, training methodology, and deployment trade-offs. A central theme is calibration: we develop the theory of proper scoring rules, analyze Reinforcement Learning for Calibrated Decisions (RLCD), and examine recently discovered phenomena including IIA violations, ordinal-scale bias, and few-shot recalibration under distribution shift. On the integration side, we describe how System 1 models serve as decision infrastructure within agentic workflows and System 1 + System 2 cascades that route between fast and slow models based on calibrated confidence. The guide covers 18 application domains, maps the rapidly growing open-source ecosystem (over 70 model reproductions in weeks), provides runnable code throughout, and includes a practitioner's checklist, an open-source model zoo, and 22 self-assessment questions with detailed answers. Spanning 190 references, it is intended as a unified entry point for researchers and engineers working at the intersection of fast decision-making, calibration, and production AI systems.
• 中文摘要 大型语言模型在推理和生成方面表现出色，但其自回归、逐个令牌的推理使其不必要地缓慢、成本高昂，且在现代AI系统以亚秒延迟下所需的众多常规决策（路由、门控、评分、分类）时校准不佳。越来越多的模型弥补了这一空白：System 1决策模型，即通过一次前向传递直接将输入映射到类型化、结构化决策的架构。本指南提供了全面且面向实践者的该领域论述。我们提出了五大族分类法，涵盖基于编码器的分类器、奖励模型、非自回归决策引擎、对比决策模型和提取模型，并深入介绍每个家族的架构、训练方法论和部署权衡。校准是一个核心主题：我们发展了适当评分规则理论，分析校准决策的强化学习（RLCD），并研究了近期发现的现象，包括IIA违规、序数尺度偏置和分布移位下的少数样本重新校准。在集成方面，我们描述了System 1模型如何作为代理工作流中的决策基础设施，以及System 1 + System 2基于校准置信度在快速和慢速模型之间路由的级联。本指南涵盖18个应用领域，绘制了快速发展的开源生态系统（数周内完成70多个模型复制），提供可运行的代码，并包含实践者检查表、开源模型动物园和22个带有详细答案的自我评估问题。该指南涵盖190个参考文献，旨在为快速决策、校准和生产AI系统交叉领域的研究人员和工程师提供统一的切入点。

EnvLunch: The Environment as a Free Lunch for Learning to Interact

环境午餐：环境作为学习互动的免费午餐

• Authors: Wei Liu, Min Gao, Yali Du, Yulan He
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Databases (cs.DB)
• Arxiv link: https://arxiv.org/abs/2610.33817
• Pdf link: https://arxiv.org/pdf/2610.33817
• Abstract For agentic data analysis, language-model agents often require a large number of annotated tasks to learn how to interact effectively with executable environments (i.e., databases). While high-quality annotated tasks require heavy manual effort, an executable environment already provides rich interaction structure and verifiable outcomes. In this sense, the environment itself offers a free lunch, an additional set of verifiable tasks that require no human annotation. We propose EnvLunch, a warm-up stage that turns this environmental signal into agent training without human-written or model-generated task descriptions, before downstream agentic reinforcement learning. EnvLunch samples valid action trajectories, executes them to obtain final observations, and trains an agent to recover a functionally equivalent trajectory given only its final observation. This label-free warm-up enables the model to outperform the baseline by over 9 points on average across all downstream training budgets, and with few labels, the baseline needs 8 to 32 times as many annotated examples to match the accuracy that the EnvLunch model reaches at its smallest training budget. Letting the environment alone define the task yields a larger gain than having the model write its own tasks over the same environments, and this warm-up also improves fine-tuning on benchmarks beyond data analysis. EnvLunch therefore saves annotated tasks and provides a better starting point for fine-tuning agentic models when such tasks are limited.
• 中文摘要 对于智能体数据分析，语言模型代理通常需要大量注释任务，以学习如何有效与可执行环境（即数据库）交互。虽然高质量注释任务需要大量人工操作，但可执行环境已经提供了丰富的交互结构和可验证的结果。从这个意义上说，环境本身提供了一顿免费午餐，是一组无需人工注释的可验证任务。我们提出EnvLunch，这是一个热身阶段，将环境信号转化为智能体训练，无需人工编写或模型生成的任务描述，然后进行下游智能体强化学习。EnvLunch 采样有效的动作轨迹，执行以获得最终观测值，并训练智能体仅基于最终观测恢复功能等效轨迹。这种无标签预热使模型在所有下游训练预算中平均领先基线9分以上，且标签较少时，基线需要8到32倍的注释示例，才能匹配EnvLunch模型在最小训练预算下达到的准确性。让环境单独定义任务比模型在同一环境中自行编写任务获得更大收益，且这种预热还提升了基准测试的微调能力，超越数据分析。因此，EnvLunch保存了注释任务，并在任务有限时为微调代理模型提供了更好的起点。

EdgeDeID: A 0.6B Clinical De-identifier Trained Without Real Patient Data

EdgeDeID：一个0.6亿临床去标识认证，训练时没有真实患者数据

• Authors: Sina Daneshgar, Raya Namazi, Amirali Khosravi, Pouria Rouzrokh, Bardia Khosravi
• Subjects: Subjects: Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.33839
• Pdf link: https://arxiv.org/pdf/2610.33839
• Abstract De-identifying clinical notes with a hosted language model discloses the information the step is meant to protect. We present EdgeDeID, a 0.6B-parameter decoder-only de-identifier that runs locally and is trained without real patient text: three LLM teachers write synthetic notes, a single labeler re-annotates them into one convention, and the model learns to emit a typed XML schema of the identifiers it finds. On a 100-note synthetic in-domain benchmark with manually verified annotations it reaches 0.994 micro-F1 and leaves no missed identifier in 99% of notes, against 0.857 and 12% for the strongest of eight off-the-shelf systems. Quantized, it runs on CPU in 615 MB. Of three additions to supervised training, only a verifier adapter helps: it separates complete from incomplete outputs of other systems at 0.94 AUROC, whereas reinforcement learning with a recall-weighted reward and a rule-based post-processing layer leave in-domain results unchanged. On MIMIC ICU nursing notes F1 falls to 0.48. The errors follow two properties our synthetic notes lack, telegraphic style and low identifier density, and three hand-written in-context examples recover 0.08 F1 without annotated target data.
• 中文摘要 用托管语言模型去识别化临床笔记，揭示了步骤旨在保护的信息。我们展示了EdgeDeID，这是一种0.6B参数的仅解码器去标识符，本地运行且训练时不包含真实患者文本：三位LLM教师编写合成笔记，一个标签器将其重新注释成一个约定，模型学习输出输入的XML模式。在100条合成的领域内基准测试中，经过人工验证的注释，其微F1达到0.994，且99%的笔记无遗漏标识符，而八个现成系统中最强的为0.857和12%。量化后，它在CPU上运行，占615 MB。监督训练的三项新增功能中，只有验证器适配器有帮助：它将0.94 AUROC下其他系统的完整输出与不完整输出区分开来，而带有回忆加权奖励和基于规则的后处理层的强化学习则保持域内结果不变。在MIMIC重症监护病房护理笔记中，F1降至0.48。错误基于我们的合成笔记所缺乏的两个特性：电报风格和低标识密度，三个手写上下文示例在无注释目标数据的情况下恢复了0.08 F1。

RoboLongR: Causally Consistent Rewards Teach Robots to Progressively Master Long-Horizon Manipulation

RoboLongR：因果一致的奖励教导机器人逐步掌握长视界操控

• Authors: Zixuan Chen, Lin Cao, Fenggang Zhang, Haotong Zhang, Yangtao Chen, Jieqi Shi, Jing Huo, Yang Gao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.33863
• Pdf link: https://arxiv.org/pdf/2610.33863
• Abstract Long-horizon manipulation requires actions whose validity depends on earlier interactions, yet visual progress can conceal missing task prerequisites. We propose RoboLongR, an offline-to-online framework that learns history-dependent rewards to guide policy adaptation to new tasks. Its central objective is causal consistency: aligning rewarded progress with prerequisites supported by execution history. Stage-shuffle supervision encourages execution-order sensitivity, recurrent memory preserves evidence of earlier interactions, and two-round training refines trajectory segments using predicted progress. The resulting rewards guide stage-conditioned policy refinement from behavior cloning to online reinforcement learning. Sparse terminal feedback connects policy experience to reward learning by triggering correction for repeated high-scoring failures. Pretrained on demonstrations from 88 RLBench tasks and Open-X data, RoboLongR achieves 31.8% average success on 18 long-horizon tasks excluded from reward pretraining, exceeding the strongest evaluated baseline by 9.2 percentage points. Across four real-robot tasks, it achieves 57.5% success, a 32.5-percentage-point improvement over the comparison method. Reward-level controls probe sensitivity to synthetic order perturbations, while ablations examine the contributions of individual components to policy improvement.
• 中文摘要 长视野操作需要的行动有效性依赖于早期互动，但视觉进展可以掩盖缺失的任务前置条件。我们提出了RoboLongR，一种离线到在线的框架，学习历史依赖的奖励以指导策略适应新任务。其核心目标是因果一致性：将奖励进展与执行历史支持的前提条件对齐。阶段洗牌监督促进执行顺序敏感性，循环记忆保留早期互动证据，双轮训练利用预测进展细化轨迹段。由此产生的奖励指导阶段条件策略的细化，从行为克隆到在线强化学习。稀疏终端反馈通过触发纠正反复高分失败，将策略经验与奖励学习连接起来。在88个RLBench任务和Open-X数据的演示基础上预训练后，RoboLongR在18个排除在奖励预训练之外的长期任务中平均成功率为31.8%，比最强评估基线高出9.2个百分点。在四个真实机器人任务中，成功率为57.5%，比对比方法提升32.5个百分点。奖励层级对照调查对合成秩序扰动的敏感性，而消融分析则考察各个组成部分对政策改进的贡献。

Intervention-Certified Amplification: A Validity Gate for $σ$-Weighted Post-Training

干预认证的增强：为加权σ美元加权的培训后培训设立的有效性门槛

• Authors: Leiyue Chen, Meng Wei, Xuefeng Hu, Wei Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.33882
• Pdf link: https://arxiv.org/pdf/2610.33882
• Abstract Hybrid post-training often mixes supervised fine-tuning (SFT) and reinforcement learning (RL), using training-side signals to set each sample's balance. Such signals characterize the anchor data and policy, but not whether the emphasized behavior helps in the target task and evaluation setting. This mismatch can reactivate and amplify pretrained behavior even after its literal form is removed from supervision. We demonstrate this with DIAA (Dual-Implicit Adversarial Amplification), a per-sample $\sigma$-weighted SFT/RL hybrid. With all literal Python fences removed from its anchor, DIAA emits unexecuted Tool-Integrated Reasoning (TIR) on $29%$ of MATH-500 prompts, three times vanilla GRPO. Paired interventions show $10.6$ percentage points of accuracy recovery when TIR is suppressed and comparable contributions from SFT imitation and $\sigma$-amplification; a student trained on a $0%$-TIR teacher anchor emits it at $45.8%$. Intervention-Certified Amplification (ICA) gates $\sigma$ with paired-rollout target evidence. Frozen and online ICA reduce TIR to $13.8%$ and $11.2%$, halve residual drag, improve accuracy from $47.5%$ to $48.7%$ and $49.3%$, and transfer to held-out GSM8K without retuning.
• 中文摘要 混合后训练通常混合监督微调（SFT）和强化学习（RL），使用训练侧信号来设定每个样本的平衡。这些信号描述锚点数据和策略，但不影响强调行为是否有助于目标任务和评估环境。这种不匹配即使其字面形式被移除，也可能重新激活并放大预训练行为。我们用DIAA（双隐式对抗放大）演示了这一点，这是一种每样本$\sigma$加权的SFT/RL混合。在所有Python围栏从锚点移除后，DIAA会在价值$29%的MATH-500提示上发出未执行的工具集成推理（TIR），是原版GRPO的三倍。配对干预在抑制TIR时显示出10.6美元百分点的准确性恢复，SFT模仿和$\sigma-放大的贡献相当;在0%$-TIR教师锚点上训练的学生发出的TIR为$45.8%。干预认证放大（ICA）门槛通过配对推广目标证据实现$\sigma$。冻结和在线ICA将TIR降至13.8%和11.2%美元，剩余阻力减半，准确率从47.5%提升至48.7%和49.3%，并将未调谐的GSM8K转移至未调。

A Sign Law for Entropy Drift under Sampler Mismatch in RLVR

RLVR中采样器不匹配下熵漂移的符号定律

• Authors: Zhiyuan Wang, Jacob Steeves, Chang Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.34044
• Pdf link: https://arxiv.org/pdf/2610.34044
• Abstract Entropy in reinforcement learning with verifiable rewards (RLVR) moves up or down depending on choices that should be neutral, such as whether over-length responses are masked or counted as failures. We trace this to sampler mismatch, where rollouts are drawn with top-k/top-p truncation or at a temperature the learner does not apply, and prove that a constant advantage shift then changes the entropy of a softmax policy, at first order and in expectation, with a sign fixed by the sampler: positive shifts sharpen the policy under any sampler whose likelihood ratio to the policy increases with token probability, including top-k, top-p, min-p, and temperatures below one, and flatten it under flatter samplers. In group-based RLVR with token-level losses, the baseline-controlled part of this drift is set by the token-weighted net advantage, which masking and failing over-length responses can push to opposite signs; the remainder, a covariance between advantages and the sampler's sharpness gap, can feed back over long horizons. On a 2B model, the entropy slope is linear in shifts of the net advantage under truncated sampling and indistinguishable from flat under exact sampling, and token-level importance ratios leave 98% of the drift. In 100-step runs, a net-zero baseline only delays an entropy explosion that exact sampling does not show, a fixed positive shift delays it further, and a baseline weighted by the sharpness gap keeps entropy stable without changing the learner. On a 9B reasoning model over ten steps, the uncorrected rule loses held-out accuracy, whereas zeroing the net advantage or renormalizing the learner gains 8.2 and 8.8 points over it, with no held-out effect under exact sampling.
• 中文摘要 带可验证奖励的强化学习（RLVR）中的熵会根据应中性的选择而上升或下降，例如超长响应是否被掩盖或计为失败。我们将此追溯到采样器不匹配，即以top-k/top-p截断或学习者不适用的温度绘制出卷，并证明常数优势移位会改变软最大策略的熵，且在一阶且期望中，且由采样器固定符号：正向偏移使任何与策略似然比随标记概率增加的采样器（包括top-k、top-p、最小-p及温度低于1）下策略趋于平坦，而在更平坦的采样器下则使其趋平。在基于组且有代币级损失的RLVR中，该漂移的基线控制部分由token加权净优势决定，掩蔽和失败的超长响应可以将其推向相反符号;剩余的优势与采样器锐利度差距之间的协方差，可以跨越较长的视野反馈。在2B模型中，熵斜率在截断抽样下净优势的平移是线性的，在精确采样下则与平坦无异，且标记级重要性比保留了98%的漂移。在100步运行中，净零基线仅延迟精确采样未显示的熵爆炸，固定的正偏移会进一步延迟熵爆炸，而由锐利度差距加权的基线则保持熵稳定而不改变学习者。在9B推理模型中，未校正规则会失去保留的准确性，而归零净优势或重整化学习者则能获得8.2和8.8分，且在精确抽样下无保留效应。

Minimax Regret for Sobolev Kernel-Based Reinforcement Learning

基于 Sobolev 核的 Minimax Regret

• Authors: Haichen Hu, David Simchi-Levi
• Subjects: Subjects: Machine Learning (cs.LG); Statistics Theory (math.ST); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.34045
• Pdf link: https://arxiv.org/pdf/2610.34045
• Abstract We study the minimax regret rate for episodic kernel-based reinforcement learning, where the expected reward and transition density belong to a Sobolev reproducing kernel Hilbert space, with continuous state and action spaces, and adversarial initial states. For Sobolev smoothness parameter $\nu>0$, state dimension $d_S$, and action dimension $d_A$, we prove that the minimax expected regret over $T$ episodes is $\widetilde\Theta(T^{\max{e_A,e_B}})$, where $e_A=\frac{\nu+d_A}{2\nu+d_A}$, $e_B=\frac{\nu+d_A+3d_S/2}{2\nu+d_A+2d_S}$ with the other model parameters fixed. The two exponents reflect the distinct difficulties of action selection and learning across states. We introduce Sobolev Optimal Value Iteration (SOVI), which attains this rate from observed rewards and transitions. For fixed smoothness and reference-kernel lengthscale, its regret bound has a prefactor polynomial in the state-action dimension $d=d_S+d_A$, horizon $H$, transition and reward RKHS norm bounds $u,B_r$, and kernel norm-comparison factor $\Lambda$, up to logarithmic factors. The algorithm separates initial action selection from continuation-value estimation and combines local polynomial approximation with the shared RKHS structure of the approximation errors. Our matching upper and lower bounds determine the optimal dependence on the number of episodes for Sobolev kernel-based reinforcement learning. Thus, we partially address the open problem in Vakili(2024) under the condition of the Sobolev kernel setup.
• 中文摘要 我们研究基于情节核强化学习的最小极大后悔率，其中期望奖励和转换密度属于Sobolev再现核希尔伯特空间，具有连续状态和动作空间，以及对抗初始状态。对于Sobolev光滑参数$\nu>0$，状态维度$d_S$，动作维度$d_A$，我们证明了$T$集数内的最小最大期望遗憾为$\widetilde\Theta（T^{\max{e_A，e_B}}）$，其中$e_A=\frac{\nu+d_A}{2\nu+d_A}$，$e_B=\frac{\nu+d_A+3d_S}{2\nu+d_A+2d_S}$，其他模型参数保持不变。这两个指数反映了跨状态动作选择和学习的独特难点。我们引入Sobolev最优值迭代（SOVI），该方法通过观察到的奖励和转移获得该速率。对于固定的光滑性和参考核长度尺度，其遗憾界限在状态-动作维数中具有前因多项式$d=d_S+d_A$，视界$H$，转移和奖励RKHS范数界限为$u，B_r$，核范数比较因子$\Lambda$，最高可达对数因子。该算法将初始动作选择与延续值估计分离，并将局部多项式近似与共享的RKHS近似误差结构结合起来。我们的匹配上下界确定了基于Sobolev核的强化学习对剧集数的最优依赖。因此，我们在 Vakili（2024） 中部分解决了 Sobolev 核设置条件下的未解决问题。

Leveraging Dale's Principle as an Inductive Bias in Recurrent Neural Networks

利用戴尔原理作为循环神经网络中的归纳偏置

• Authors: Jiyi Wang, Chenxiao Yang, Yujia Zhao, Jingzhao Zhang, Lu Mi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34052
• Pdf link: https://arxiv.org/pdf/2610.34052
• Abstract One of the fundamental differences between biological neural networks and artificial neural networks (ANNs) is that biological neurons obey Dale's principle: each neuron projects either exclusively excitatory or exclusively inhibitory outputs. While this constraint is central to biological circuits, previous attempts to impose Dale's principle in ANNs have often led to training instability or degraded performance. In this work, we introduce EISep RNN, a recurrent neural network architecture that implements Dale's principle through a learnable excitatory-inhibitory separation. Across a series of learning settings, including multi-task learning, continual learning, and reinforcement learning, EISep achieves more stable training dynamics, competitive or improved performance, and sparser, more modular connectivity than vanilla RNNs and previous Dale-constrained models. EISep also exhibits stable and informative latent dynamics. These results establish Dale's principle as a biologically grounded and computationally effective inductive bias for recurrent neural networks.
• 中文摘要 生物神经网络与人工神经网络（ANNs）之间的根本区别之一是，生物神经元遵循戴尔原则：每个神经元都投射纯兴奋性或纯抑制性输出。虽然这一约束是生物回路的核心，但此前试图将戴尔原理强加于人工神经网络的尝试常常导致训练不稳定或性能下降。在本研究中，我们介绍了EISep RNN，这是一种通过可学习的兴奋性-抑制性分离实现戴尔原理的循环神经网络架构。在多任务学习、持续学习和强化学习等多种学习环境中，EISep实现了比普通RNN和以往戴尔约束模型更稳定的训练动态、更具竞争力或提升的性能，以及更稀疏、更模块化的连接性。EISep还表现出稳定且富有信息量的潜在动态。这些结果确立了戴尔原理作为一种生物学基础且计算有效、适用于循环神经网络的归纳偏误。

BE-CTDE: Blockchain-Enabled Multi-Agent Reinforcement Learning with Anomaly Detection and Token Economics

BE-CTDE：区块链驱动的多智能体强化学习，结合异常检测和代币经济学

• Authors: Chih-Wei Huang, Hsin-Chieh Lee, Yun-Chen Yu, Ibrahim Althamary, Rongxing Lu, Phone Lin
• Subjects: Subjects: Multiagent Systems (cs.MA); Cryptography and Security (cs.CR); Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.34121
• Pdf link: https://arxiv.org/pdf/2610.34121
• Abstract Multi-agent reinforcement learning supports cooperative decision-making in Internet of Things systems, but strategic deviations can disrupt coordination and undermine confidence in the training process. This paper proposes Blockchain-Enabled Centralized Training and Decentralized Execution (BE-CTDE), which uses blockchain as an active training-management layer. A behavioral verifier compares executed trajectories with shared-policy recommendations; its flags drive a token mechanism combining slashing, burning, rewards, and compensation. Retained balances determine each agent's contribution to the global policy update, connecting auditable behavior to learning influence. We derive a one-round incentive condition that accounts for action-dependent compensation and the payoff from deviation, and a finite expected exclusion-time bound under explicit conditional detection and accounting assumptions, showing perfect detection is unnecessary. An exact finite-population capacity bound characterizes the limits of the population-relative detector. In multi-agent path finding experiments, BE-CTDE outperforms a state-of-the-art learning-based path planner with substantially higher success rates under attack, even when adversaries form the majority. It also outperforms partially defended baselines in all configurations, with collision rates up to 35% lower than under detection without economic consequences and up to 43% lower than under conventional stake slashing. Together, these results characterize how behavioral verification and token feedback can support accountable cooperative learning.
• 中文摘要 多智能体强化学习支持物联网系统中的协作决策，但战略偏差可能扰乱协调并削弱对训练过程的信心。本文提出了区块链驱动的集中训练与去中心化执行（BE-CTDE），利用区块链作为主动训练管理层。行为验证器将执行轨迹与共享策略建议进行比较;其标志驱动结合砍杀、烧毁、奖励和补偿的代币机制。保留余额决定每个代理对全球政策更新的贡献，将可审计行为与学习影响联系起来。我们推导出一个单轮激励条件，考虑依赖动作的补偿和偏差收益，以及在明确条件检测和会计假设下有限的预期排除时间限制，证明完美检测并非必要。精确的有限种群容量界限定义了种群相对检测器的极限。在多智能体路径寻找实验中，BE-CTDE 在攻击下成功率显著高于最先进的基于学习的路径规划器，即使对手占多数。它在所有配置中都优于部分防御基线，碰撞率比检测时低多达35%，且比传统权益切割低多达43%。这些结果共同表征了行为验证和令牌反馈如何支持负责任的合作学习。

Self-Evolution Reasoner: Continuous Optimization via Policy-Intrinsic Exploration and Exploitation

自我进化推理器：通过策略内在探索与利用实现持续优化

• Authors: Wenhang Shi, Yiren Chen, Zhe Zhao, Jinhao Dong, Wei Lu, Xiaoyong Du
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.34137
• Pdf link: https://arxiv.org/pdf/2610.34137
• Abstract Reinforcement learning with verifiable rewards (RLVR) enables large language models (LLMs) to continuously self-improve their reasoning capabilities. Compared to RLVR's scalar rewards, recent self-supervised on-policy distillation methods leverage denser supervision signals. However, both paradigms suffer from exploration bottlenecks, struggling to escape local optima when the policy fails to sample valid solutions. Relying on external demonstrations can bypass this issue; however, the off-policy nature hinders genuine self-improvement and the mismatched-distribution reasoning can not be fully exploited. In this work, we propose Self-Evolution Policy Optimization (SEPO), a framework that drives continuous reasoning capability evolution via guided exploration and in-distribution exploitation. Specifically, SEPO utilizes labels to elicit correct reasoning from the policy, unlocking the exploration potential. By treating current policy conditioned on this in-distribution reasoning as a self-teacher, SEPO distills the newly discovered solution back into the policy. Through this iterative cycle of policy-intrinsic exploration and exploitation, SEPO achieves robust and continuous capability evolution. Across scientific, tool-use, and complex logical reasoning domains, SEPO consistently achieves the best performance under identical training time or steps, demonstrating superior efficiency and convergence ceilings. Notably, SEPO maintains stable improvements even in scenarios where baselines severely underperform or fail entirely, such as on complex tasks or with small-scale models. Our code is available at this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）使大型语言模型（LLM）能够持续自我提升推理能力。与RLVR的标量奖励相比，近期的自监督策略内提炼方法利用了更密集的监督信号。然而，这两种范式都存在探索瓶颈，当策略未能采样有效解时，难以摆脱局部最优。依赖外部演示可以绕过这一问题;然而，非策略性质阻碍了真正的自我提升，且分布错配推理无法被充分利用。本研究提出自我进化策略优化（SEPO），这一框架通过引导探索和分布内利用推动持续推理能力的演进。具体来说，SEPO利用标签从策略中引出正确的推理，释放探索潜力。通过将基于这种分布内推理的当前策略视为自学者，SEPO将新发现的解决方案提炼回策略中。通过这一策略内在探索和利用的迭代循环，SEPO实现了稳健且持续的能力演进。在科学、工具使用和复杂逻辑推理领域，SEPO在相同的训练时间或步数下持续获得最佳性能，展示了卓越的效率和收敛上限。值得注意的是，即使在基线严重表现不佳甚至完全失败的场景中，如复杂任务或小规模模型，SEPO也能保持稳定的改进。我们的代码可在此 https 网址获取。

Reinforcement Learning for Adaptive Crowd Flow Control Using Dynamic Barriers

利用动态屏障进行自适应人群流动控制的强化学习

• Authors: Alhanof Alolyan, He Wang, Amani Alonayzan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34163
• Pdf link: https://arxiv.org/pdf/2610.34163
• Abstract Crowd barriers are widely used to regulate pedestrian movement during large public events such as pilgrimages. However, fixed barrier layouts often create unnecessary stopping and local congestion even when they successfully control the overall crowd flow. Finding better barrier configurations is difficult because there is no dataset that provides the best barrier placement for a given crowd scene. We address this challenge by introducing a simulation-driven reinforcement learning framework that learns barrier configurations without requiring labelled training data. Starting from a segmented crowd scene, we construct a realistic simulation environment that reproduces the observed pedestrian distribution. This environment is then used to train a Soft Actor-Critic (SAC) agent that continuously adjusts barrier positions to improve pedestrian movement. Unlike supervised approaches the proposed framework allows the agent to explore barrier configurations that have never been observed in real data. We evaluate the learned policy under unseen numbers of pedestrians. The learned policy consistently outperforms the existing manually designed barrier configuration by reducing pedestrian stops while maintaining effective flow regulation. These results demonstrate that combining computer vision crowd simulation and reinforcement learning provides an effective framework for optimizing crowd control infrastructure.
• 中文摘要 人群屏障被广泛用于在大型公共活动（如朝圣）中调节行人流动。然而，固定屏障布局常常造成不必要的停车和局部拥堵，即使它们成功控制了整体人群流动。寻找更好的屏障配置很困难，因为没有数据集能为特定人群场景提供最佳屏障布置。我们通过引入一个仿真驱动的强化学习框架来解决这一挑战，该框架无需标记训练数据即可学习屏障配置。从分段的人群场景出发，我们构建了一个真实的模拟环境，以再现观察到的行人分布。随后利用该环境训练一个软演员-批评者（SAC）代理，持续调整屏障位置以改善行人流动。与监督式方法不同，该框架允许代理探索从未在真实数据中观察过的障碍物配置。我们在未见人数的行人下评估所学策略。该政策通过减少行人停车次数，同时保持有效的流量调节，持续优于现有手动设计的障碍配置。这些结果表明，计算机视觉、人群模拟与强化学习相结合，为优化人群控制基础设施提供了有效框架。

CORMA: Constraint-Routed Decision Mamba for Generalizable Multi-Constraint Offline Safe Reinforcement Learning

CORMA：约束路由决策Mamba用于推广多约束离线安全强化学习

• Authors: Changxu Ni, Feiran You, Yongan Guo, Hongyang Du, Kang Wei, Le Liang, Yiyang Ni, Jun Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.34208
• Pdf link: https://arxiv.org/pdf/2610.34208
• Abstract Offline safe reinforcement learning (RL) learns safety-aware policies from static datasets while satisfying safety constraints during deployment. Motivated by the strong trajectorymodeling capabilities of Transformers, recent Decision Transformer (DT)-based methods formulate offline safe RL as conditional sequence modeling by conditioning action generation on return-to-go (RTG) and cost-to-go (CTG) tokens, thereby enabling adaptation to varying safety budgets. However, existing DT-based methods are developed for single-constraint tasks. For tasks with multiple constraints, generalizing learned policies to unseen combinations of safety budgets remains challenging due to the conflicting requirements imposed by heterogeneous constraints. Moreover, the quadratic complexity of Transformer architectures significantly increases the computational overhead in multi-constraint tasks, as representing additional constraints requires more CTG tokens. Recently, Mamba has emerged as an efficient sequence-modeling backbone, offering linear computational complexity and strong generalization capabilities. In this paper, we propose CORMA, a constraint-routed decision Mamba framework for generalizable multi-constraint offline safe RL. CORMA decomposes policy inference into constraint-aware RTG estimation and adaptive action generation, where constraint-specific experts model feasible RTG distributions and a router-guided product-of-experts mechanism fuses these distributions to provide safety-aware guidance for a Mamba-based action decoder. A three-stage optimization strategy is designed to improve training stability. Extensive experiments on OSRL benchmarks demonstrate that CORMA achieves superior reward-safety trade-offs, reduces inference latency by 37% compared with DT-based methods, and exhibits stronger zero-shot generalization to unseen safety-budget combinations than offline safe RL baselines.
• 中文摘要 离线安全强化学习（RL）在部署过程中满足安全约束的同时，从静态数据集中学习安全意识策略。受Transformer强大轨迹建模能力的启发，基于决策转换器（DT）的近期方法将离线安全强化学习以条件序列建模的形式制定，通过以返回到货（RTG）和成本到货（CTG）令牌为条件，从而适应不同的安全预算。然而，现有基于DT的方法主要为单约束任务开发。对于多约束任务，由于异构约束带来的冲突需求，将学习策略推广到未见的安全预算组合仍然具有挑战性。此外，Transformer架构的二次复杂度显著增加了多约束任务的计算开销，因为表示额外约束需要更多CTG标记。近年来，Mamba已成为高效的序列建模骨干，具备线性计算复杂性和强大的泛化能力。本文提出CORMA，一种约束路由决策Mamba框架，用于通用多约束离线安全强化学习。CORMA将策略推断分解为约束感知的RTG估计和自适应动作生成，约束特定专家建模可行的RTG分布，路由器引导的专家产品机制融合这些分布，为基于Mamba的动作解码器提供安全意识指导。设计了三阶段优化策略以提升训练稳定性。OSRL基准测试的大量实验表明，CORMA在奖励与安全权衡上表现优异，推理延迟比基于DT的方法降低37%，并且在零样本推广中对未见安全-预算组合的推广力强于离线安全强化基线。

DoubleFree:Exploiting Exploration History for Rollout-Free Multi-Task LLMs Training

DoubleFree：利用探索历史进行无部署的多任务大型语言模型训练

• Authors: Yingping Wang, Min Cai, Chenyu Yuan, Xi Ye, Zhijiang Guo, Jiayi Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.34272
• Pdf link: https://arxiv.org/pdf/2610.34272
• Abstract Multi-task post-training can develop domain specialists through independent reinforcement learning (RL) and consolidate their capabilities through multi-teacher on-policy distillation (MOPD). However, standard MOPD generates on-policy student responses and keeps multiple teachers online throughout consolidation, introducing substantial system costs. We view this pipeline as domain-wise exploration followed by cross-domain exploitation. Domain RL produces not only the final specialists, but also the trajectories collected throughout their training. Existing methods primarily use the former while leaving the latter underutilized. We introduce DoubleFree, which reuses these historical trajectories for rollout-free consolidation and precomputes teacher supervision for online-teacher-free student training. To understand why this works, we analyze when updates from historical trajectories can approximate on-policy distillation updates and how combining trajectories across exploration stages can better support an evolving student. DoubleFree requires only minimal changes to existing post-training frameworks. Across math, code, and instruction following on Qwen3-4B, Qwen3-8B, and SmolLM3-3B, DoubleFree achieves $2.99$-$5.75\times$ consolidation speedups and reduces GPU-hours by $80.9$-$90.1\%$, while maintaining comparable multi-domain performance.
• 中文摘要 多任务后培训可以通过独立强化学习（RL）培养领域专家，并通过多教师政策提炼（MOPD）巩固其能力。然而，标准MOPD生成符合政策的学生响应，并在整合过程中保持多名教师在线，带来显著的系统成本。我们将该流程视为域级探索，随后是跨域开发。领域RL不仅产生最终专家，还生成培训过程中收集的轨迹。现有方法主要使用前者，而后者则未被充分利用。我们引入DoubleFree，重用这些历史轨迹进行无推广的整合，并为在线无教师培训预计算教师监督。为理解其有效性，我们分析了历史轨迹更新何时能近似策略提炼更新，以及跨探索阶段结合轨迹如何更好地支持学生的进化。DoubleFree 只需对现有的培训后框架做最小修改。在数学、代码和后续教学中，Qwen3-4B、Qwen3-8B 和 SmolLM3-3B 实现了 2.99 美元至 5.75 美元乘以的加速，同时 GPU 小时数减少了 80.9 美元至 90.1% 美元，同时保持了相当的多域性能。

AndroidFlux: Evaluating and Reward Modeling Failure Recovery Capabilities in Mobile Use Multimodal Agents

AndroidFlux：评估并奖励移动端使用多模态代理中的故障恢复能力建模

• Authors: Gyubeum Lim, Yemo Koo, Ghunwoo Jang, Zsolt Kira
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.34384
• Pdf link: https://arxiv.org/pdf/2610.34384
• Abstract Mobile-use multimodal agents are typically evaluated by task success from a clean initial state, which does not isolate their ability to recover from intermediate errors. Because mobile environments change dynamically, a general mobile-use agent must complete tasks robustly even from unexpected states and in unseen environments. This requires knowing when and where an agent makes errors, and whether it can recover from errors and subsequent states. To evaluate this capability, we introduce AndroidFlux, a failure-recovery benchmark built on AndroidWorld. AndroidFlux restores recorded trajectories to controlled checkpoints before, at, and after the first recoverable error, so that different agents continue from the same device state and history. On these checkpoints, a complementary reward-modeling benchmark evaluates recovery-action preferences using downstream rollout outcomes. We further release an offline reinforcement learning (RL) dataset that improves both error recovery and overall task success. Our evaluations show that recovery performance only partially correlates with clean-start success, and that explicit reasoning helps only models trained with GUI-domain specific RL.
• 中文摘要 移动用途多模态代理通常通过初始状态干净的任务成功率来评估，这并不能隔离其从中间错误中恢复的能力。由于移动环境动态变化，通用移动使用代理即使在意外状态和未见环境中也必须稳健地完成任务。这需要知道代理何时何地犯错，以及是否能从错误和后续状态中恢复。为评估这一能力，我们引入了基于AndroidWorld的失败恢复基准测试AndroidFlux。AndroidFlux在首次可恢复错误之前、起点和之后还原记录到受控检查点的轨迹，使不同代理能从相同的设备状态和历史继续运行。在这些检查点上，一个互补的奖励建模基准测试利用下游推广结果评估恢复-行动偏好。我们还发布了一个离线强化学习（RL）数据集，提升错误恢复和整体任务成功率。我们的评估显示，恢复性能与干净启动成功仅部分相关，显式推理仅帮助使用GUI领域特定强化学习训练的模型。

From Human Intent to In-Orbit Action: A Self-Learning Large Language Model Agent for 6G Satellite Networks

从人类意图到轨道行动：6G卫星网络的自学大型语言模型代理

• Authors: Zhengqian Xu, Yasheng Zhang, Chenhua Sun, Shan Lu, Chengcheng Li
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34387
• Pdf link: https://arxiv.org/pdf/2610.34387
• Abstract Autonomous control of sixth-generation (6G) low Earth orbit satellite networks is hindered by the gap between high-level human intent and low-level control logic: translating a request such as "uninterrupted Arctic video" into a valid reinforcement learning reward still takes days of manual trial and error, and existing large language model translators cannot learn from the failures of the policies they generate. This article presents ALMOND (Agentic Large-Model-Orchestrated autonomous Network Design), a cognitive framework that closes this intent-to-action loop under on-orbit compute constraints. The contribution in artificial intelligence is a bidirectional semantic engine that translates natural-language demands into multi-objective reward functions and, conversely, mines implicit operational constraints from failed reinforcement-learning trajectories through counterfactually validated forensics, together with a large language model-guided graph neuroevolution that seeds and critiques policy architectures. The contribution in engineering is a ground-centric, compute-frugal architecture for autonomous 6G non-terrestrial network operation, in which satellites execute only distilled policies below 50,000 parameters with sub-15 ms inference and no on-board large language model. On a 350-satellite digital twin driven by real ephemeris data, ALMOND raises polar handover success from 55% to over 95%, converges about three times faster than baseline reinforcement learning, and reduces reward engineering from 48 hours to 4.5 hours, demonstrating a practical path toward trustworthy in-orbit autonomy.
• 中文摘要 第六代（6G）近地轨道卫星网络的自主控制受限于高层人类意图与低层控制逻辑之间的差距：将“不间断的北极视频”等请求转化为有效的强化学习奖励仍需数天手工反复试验，现有大型语言模型翻译器无法从其生成策略的失败中学习。本文介绍了ALMOND（代理大型模型编排自主网络设计），这一认知框架在轨道计算约束下闭合了意图到行动的循环。人工智能的贡献是一套双向语义引擎，将自然语言需求转化为多目标奖励函数，反过来通过反事实验证的取证挖掘失败强化学习轨迹中的隐性操作约束，同时结合大型语言模型引导的图神经进化，种下并批判策略架构。工程领域的贡献在于采用以地面为中心、计算节约的架构，实现自主6G非地面网络运行，卫星仅执行5万个以下参数的精炼策略，推理速度低于15毫秒，且无机载大型语言模型。在基于真实星历数据驱动的350颗卫星数字孪生上，ALMOND将极地切换成功率从55%提升至95%以上，收敛速度约为基线强化学习的三倍，并将奖励工程时间从48小时缩短至4.5小时，展示了实现可信轨道自主性的切实路径。

IntNet: A Communication-Driven Multi-Agent Reinforcement Learning Framework for Cooperative Autonomous Driving

IntNet：一个基于通信的多智能体强化学习框架，用于合作自动驾驶

• Authors: Leandro Parada, Kevin Yu, Panagiotis Angeloudis
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34466
• Pdf link: https://arxiv.org/pdf/2610.34466
• Abstract Achieving safety in autonomous driving through Multi-Agent Reinforcement Learning (MARL) is a critical yet challenging task due to non-stationarity, partial observability, and the need for effective coordination among agents. Although earlier cooperative MARL methods have aimed to improve coordination by sharing encoded state observations, we find these strategies are insufficient in safety-critical scenarios. To address this gap, we present IntNet, a novel MARL framework that enhances safety by incorporating the transmission of vehicle intent and adaptive communication scheduling into a unified end-to-end learning paradigm, jointly optimising all components for safe coordination. Central to our approach is an observation prediction module that enables agents to forecast subsequent policy outputs, which they can then share across a vehicle-to-vehicle network. Our model-free architecture employs a Graph Attention Network to process incoming messages and a scheduler component to dynamically optimise the communication graph for bandwidth efficiency. We compare IntNet against state-of-the-art communicative MARL methods in complex urban environments with both autonomous and human-operated vehicles, achieving improved coordination, lower collision rates, and reducing communication efforts by up to 60\%. Through extensive experiments, we assess the framework's learning efficiency, scalability, and the balance between information sharing and bandwidth usage for collision-free trajectories.
• 中文摘要 通过多智能体强化学习（MARL）实现自动驾驶安全，是一项关键但具有挑战性的任务，因为其非平稳性、部分可观测性以及代理间需要有效协调。尽管早期合作的MARL方法旨在通过共享编码状态观测来改善协调，但我们发现这些策略在安全关键场景下仍不够。为弥补这一空白，我们提出了IntNet，一种新型MARL框架，通过将车辆意图传输和自适应通信调度整合进统一的端到端学习范式，共同优化所有组件以实现安全协调。我们方法的核心是一个观察预测模块，使智能体能够预测后续的政策输出，并通过车辆间网络共享这些结果。我们的无模型架构采用图关注网络处理来消息，并配备调度组件动态优化通信图以提升带宽效率。我们将IntNet与复杂城市环境中最先进的通信MARL方法进行比较，无论是自动驾驶还是人力车辆，均实现了协调改进、碰撞率降低，并减少了多达60%的通信努力。通过大量实验，我们评估了该框架的学习效率、可扩展性以及信息共享与带宽使用之间的平衡，实现无碰撞轨迹。

Best Policy Tracking in Reinforcement Learning

强化学习中的最佳策略跟踪

• Authors: Judith Echevarrieta, Etor Arza, Aritz Pérez, Josu Ceberio
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34501
• Pdf link: https://arxiv.org/pdf/2610.34501
• Abstract Reinforcement learning commonly exhibits non-monotonic performance dynamics in single-task settings. During learning, a sequence of policies is generated whose true performance is unknown and may oscillate significantly across iterations. Despite this well-known phenomenon, standard experimental practice typically reports either the final visited policy or the best among periodically evaluated ones, implicitly assuming that these selection criteria reliably approximate the best-performing policy visited throughout the entire sequence. In this work, we present an empirical study on the implications of non-monotonicity for policy selection in single-task policy-based reinforcement learning. We introduce post-hoc diagnostic metrics to characterize performance degradation and assess the accuracy and efficiency of different selection criteria. These metrics allow us to evaluate commonly used selection strategies across standard continuous-control benchmarks using a well-established on-policy actor-critic method. Our results show that, in this representative experimental setting, widely adopted selection practices can deviate considerably from the best policy encountered during learning, leading to underperforming estimates and suboptimal use of computational resources. As an alternative, we recommend a validation-based policy selection strategy constrained by computational cost, which provides a more consistent and accurate approximation of the best visited policy while improving selection efficiency. Overall, our findings highlight policy selection as an underexplored factor in the evaluation protocols of reinforcement learning and motivate further study on explicit and standardized validation-based selection procedures.
• 中文摘要 强化学习在单任务环境中通常表现出非单调的性能动态。在学习过程中，生成一系列策略，其真实性能未知，且可能在迭代中显著振荡。尽管存在这一众所周知的现象，标准实验实践通常报告最终访问的策略或周期评估策略中的最佳，隐含假设这些选择标准可靠地近似整个序列中表现最好的策略。本研究提出了一项关于非单调性对单任务基于策略强化学习策略选择的影响的实证研究。我们引入事后诊断指标，以表征性能下降并评估不同选择标准的准确性和效率。这些指标使我们能够利用成熟的策略内行为者-批评者方法评估标准连续控制基准测试中常用的选择策略。我们的结果表明，在这一具有代表性的实验环境中，广泛采用的选择实践可能与学习过程中遇到的最佳策略有显著偏离，导致估计表现不佳，计算资源使用不优。作为替代方案，我们建议采用基于验证、受计算成本约束的策略选择策略，这种策略在提升选择效率的同时，更一致、更准确地近似最佳访问策略。总体而言，我们的发现凸显了策略选择作为强化学习评估方案中一个未被充分探讨的因素，并激励了对显式和标准化验证选择程序的进一步研究。

GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations

GRASP：无推理注释的医学VQA组组自适应混合RL-SFT后期培训

• Authors: Wanqi Dong, Wanyue Dong, Jingze Ge, Mehul Motani
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.34561
• Pdf link: https://arxiv.org/pdf/2610.34561
• Abstract Adapting general-purpose vision-language models (VLMs) to medical visual question answering (VQA) is valuable for helping clinicians read images and make diagnoses, but annotated data are scarce and provide only short answers, without reasoning annotations. Supervised fine-tuning (SFT) fits these labels but overfits to them and amplifies their biases, eroding out-of-domain (OOD) generalization. Reinforcement learning (RL) better preserves generalization but never uses the gold answer as a training target: GRPO-style algorithms learn from the contrast between correct and incorrect answers within a group of sampled responses, so all-correct or all-wrong groups give no correctness signal. Existing SFT-RL hybrids either apply SFT and RL in separate stages or switch between them per question, or need reasoning traces from experts or stronger models, which medical VQA data lack. We propose GRASP, which combines RL and SFT in every update using only short answers: RL lets the model explore its own reasoning, SFT teaches the gold answer after that reasoning to supply the direct supervision RL lacks, and their balance adapts to how well the model answers each question. Tuning only about 1% of the parameters with LoRA, GRASP outperforms all compared training objectives both in domain and on OOD open-ended questions across Qwen3-VL-8B, InternVL3-8B and Gemma 3-4B. On an OOD benchmark spanning eight imaging modalities, GRASP-trained InternVL3-8B and Qwen3-VL-8B beat all 10 compared general and medical VLMs in open-ended accuracy, including models trained on up to 2,600 times more data or with over 250 times more trainable parameters. We release MedTerse, a dataset of 12.9K short-answer questions, and our code at this https URL.
• 中文摘要 将通用视觉语言模型（VLMs）适应到医疗视觉问答（VQA）方面有助于临床医生阅读图像和做出诊断，但注释数据稀少，且仅提供简短答案，缺乏推理注释。监督式微调（SFT）符合这些标签，但会过度拟合并放大其偏见，侵蚀域外（OOD）泛化。强化学习（RL）更好地保持泛化，但从不将金答案作为训练目标：GRPO风格算法通过一组抽样回答中正确与错误答案的对比学习，因此全正确或全错组不提供正确信号。现有的SFT-RL混合方法要么分阶段应用SFT和RL，要么每题切换，要么需要专家或更强模型的推理痕迹，而医学VQA数据缺乏这些。我们提出GRASP，每次更新结合RL和SFT，仅使用简短答案：RL让模型探索自身推理，SFT教授金答案，提供RL所缺乏的直接指导，平衡则根据模型对问题的答复能力调整。GRASP仅调优约1%的参数，在领域和OOD开放式问题上均优于所有对比训练目标，涵盖Qwen3-VL-8B、InternVL3-8B和Gemma 3-4B。在涵盖八种成像模态的OOD基准测试中，GRASP训练的InternVL3-8B和Qwen3-VL-8B在开放式准确率上击败了所有10个普通和医学VLM的比较模型，包括训练数据多达2600倍或可训练参数超过250倍的模型。我们发布了MedTerse，一个包含12.9K短答题的数据集，以及我们的代码，地址为 https URL。

Large Language Models: A Plain-Language Overview for Engineers

大型语言模型：工程师的简明语言概述

• Authors: Bahaeddin Türkoğlu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34585
• Pdf link: https://arxiv.org/pdf/2610.34585
• Abstract Large language models (LLMs) are neural networks trained on huge text collections to generate text by predicting what comes next. Engineers using ChatGPT and other generative artificial intelligence (AI) tools need a clear picture of what training changes and prompts add. The article starts with the Transformer, which connects words in context, and pre-training, where models learn to predict text. It then explains how fine-tuning adapts models to requests, with reinforcement learning from human feedback using human answer rankings. Next, retrieval-augmented generation (RAG) supplies documents, while chain-of-thought prompting encourages intermediate steps and agents use tools. FlashAttention lowers training and running costs by reducing data movement during attention; quantized low-rank adaptation (QLoRA) saves fine-tuning memory. Readers get a life cycle figure and tables comparing methods and their limits. A short checklist for judging LLM evaluation claims flags test material in training (data contamination) and model-based grading (LLM-as-a-judge). It also covers very long inputs (long context) and hallucination, meaning false or unsupported claims. Check factual accuracy separately from whether an answer sounds helpful.
• 中文摘要 大型语言模型（LLM）是训练于庞大文本集合上的神经网络，通过预测接下来的内容生成文本。使用 ChatGPT 及其他生成式人工智能（AI）工具的工程师需要清晰了解训练变化和提示带来的额外内容。文章从 Transformer 开始，它连接词语上下文，以及预训练，模型学习预测文本。随后解释了微调如何适应模型以满足请求，强化学习通过人类反馈和人类答案排名进行学习。接下来，检索增强生成（RAG）提供文档，而思维链提示鼓励中间步骤，并让智能体使用工具。FlashAttention 通过减少注意力期间的数据移动降低训练和运行成本;量化低秩适应（QLoRA）节省微调内存。读者获得生命周期图和表格，比较方法及其限制。判断LLM评估主张标志的简短清单：培训中的测试材料（数据污染）和基于模型的评分（如LLM作为评判）。它还涵盖了非常长的输入（长上下文）和幻觉，即错误或无根据的主张。检查事实准确性与答案是否听起来有用分开。

Audio-R1: Audio Reasoning through Reinforcement Learning on Self-Supervised Tasks

音频-R1：通过强化学习进行自我监督任务的音频推理

• Authors: Jihoon Jeong, Mirco Ravanelli, Zhepei Wang, Cem Subakan
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2610.34609
• Pdf link: https://arxiv.org/pdf/2610.34609
• Abstract Reasoning is increasingly important for large audio-language models (LALMs), yet remains relatively underexplored compared with text-only and vision-language models. We introduce Audio-R1, a reinforcement learning (RL) post-training framework that revisits audio self-supervised learning as a source of verifiable rewards for improving audio understanding and reasoning task performance. Audio-R1 uses answers automatically determined during the construction of self-supervised tasks to verify model responses and compute rewards. Experiments across multiple LALMs and diverse audio benchmarks show consistent performance gains, while ablations demonstrate the benefits of each self-supervised task and additional gains from joint training. While both supervised fine-tuning and Audio-R1 improve self-supervised task accuracy, Audio-R1 further improves performance on downstream tasks distinct from the training tasks. These results highlight the potential of RL post-training with self-supervised audio tasks to improve downstream performance and reasoning-trace quality.
• 中文摘要 推理在大型音频语言模型（LALMs）中日益重要，但与纯文本和视觉语言模型相比，仍然相对较少被充分探索。我们介绍了Audio-R1，一种强化学习（RL）后训练框架，重新审视音频自监督学习作为提升音频理解和推理任务表现的可验证奖励来源。Audio-R1利用构建自我监督任务过程中自动确定的答案来验证模型响应并计算奖励。跨多个LALM和多样化音频基准的实验显示了持续的性能提升，而消融分析则展示了每个自监督任务的益处以及联合训练带来的额外收益。虽然监督微调和Audio-R1都能提升自我监督任务的准确性，但Audio-R1进一步提升了下游任务的表现，区别于训练任务。这些结果凸显了强化学习与自监督音频任务后训练提升下游性能和推理追踪质量的潜力。

Dream to Assist: Intent-Conditioned World Models for Shared Vehicle Control

梦想助力：意图条件世界模型共享车辆控制

• Authors: Jonathan DeCastro, Thomas Balch, Emily Sumner, Deepak Gopinath, Laporsha Dees, Patricio Reyes Gomez, Elliot Weiss, Megan Applegate-Kenton, Todd Rowell, John Leonard, Avinash Balachandran, Guy Rosman
• Subjects: Subjects: Robotics (cs.RO); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34650
• Pdf link: https://arxiv.org/pdf/2610.34650
• Abstract Assistive agents in shared-control settings face a dual challenge: acting proactively to align with the human's intentions while remaining legible enough for the human to comprehend and fluently respond in real time. We present Dream2Assist, a model-based reinforcement learning framework for tightly-coupled shared decision making that addresses both sides of this challenge. Our approach extends a recurrent state-space world model with intent inference, enabling an assistive policy trained via fictitious co-play to anticipate and support discrete human objectives without overriding human agency. Beyond the computational framework, we describe the full deployment pipeline needed to translate a world-model-based policy into a safe, human-centered system in a resource-efficient way. Crucially, we address the sim-to-real-human gap via policy down-selection and reward restructuring within an iterative human-machine interface design. We validate the framework through three human-subjects studies and a preliminary on-vehicle deployment spanning two qualitatively distinct domains: high-speed racing (N=30) and urban driving with richer intent structure (N=10). Our results demonstrate that intent conditioning yields measurable improvements in torque alignment and collision rate. We surface key design insights and practices for deploying learning-based assistive agents in safety-critical, real-time human-robot teaming.
• 中文摘要 共享控制环境中的辅助代理面临双重挑战：既要主动行动以符合人类意图，又要保持足够清晰，使人类能够实时理解和流畅响应。我们提出了Dream2Assist，一种基于模型的强化学习框架，用于紧密耦合的共享决策，能够解决这一挑战的两端。我们的方法扩展了一个带有意图推断的循环状态-空间世界模型，使通过虚构共演训练的辅助策略能够预判并支持离散的人类目标，而不覆盖人类能动性。除了计算框架外，我们还描述了将基于世界模型的策略转化为安全、以人为中心的系统所需的完整部署流程，且资源高效。关键是，我们通过策略下选和奖励重组，在人机界面迭代设计中解决了模拟与真实人的差距。我们通过三项人受试者研究和跨越两个质的不同领域——高速赛车（N=30）和意图结构更丰富城市驾驶（N=10）——验证了该框架。我们的结果表明，意图条件反射能显著改善扭矩对齐和碰撞率。我们揭示了在安全关键的实时人机协作中部署基于学习辅助代理的关键设计见解和实践。

Movable-Element Multi-Functional RIS-Assisted Hybrid RSMA-NOMA: A Constrained Hybrid Reinforcement Learning Approach

可移动元件多功能RIS辅助混合RSMA-NOMA：一种受限混合强化学习方法

• Authors: Li-Hsiang Shen, Chi-Te Kuo
• Subjects: Subjects: Artificial Intelligence (cs.AI); Signal Processing (eess.SP)
• Arxiv link: https://arxiv.org/abs/2610.34701
• Pdf link: https://arxiv.org/pdf/2610.34701
• Abstract In this work, we propose a movable-element (ME) multi-functional reconfigurable intelligent surface (MF-RIS) architecture, termed MM-RIS, which integrates element mobility with adaptive signal reflection, transmission, amplification, and energy harvesting (EH) to enhance communication flexibility and energy self-sustainability. A hybrid rate-splitting multiple access (RSMA) and non-orthogonal multiple access (NOMA) transmission scheme is further designed to improve spectral efficiency. We formulate an energy efficiency (EE) maximization problem by jointly optimizing transmit beamforming, power and rate allocation, MM-RIS configurations, panel deployment, and ME positions. To address the resulting non-convex mixed continuous-discrete optimization problem, we propose constrained hybrid deep reinforcement learning with federated transformer and parametrized sharing (CRAFTS). CRAFTS integrates multi-agent proximal policy optimization (MAPPO) and multi-agent deep Q-networks (MADQN) through bi-directional parametrized sharing to handle continuous and discrete actions, respectively. Moreover, a transformer-based architecture and federated learning (FL) are incorporated to enhance state representation and distributed knowledge sharing, whilst a Lagrangian-based adaptive penalty mechanism facilitates the constraint satisfaction. Simulation results have demonstrated that the proposed MM-RIS architecture and CRAFTS framework substantially improve EE and constraint satisfaction compared to the benchmarks in the open literature. Moreover, the results validate the benefits of spatially reconfigurable MM-RIS, hybrid RSMA-NOMA transmissions, and intelligent collaborative learning, highlighting the potential of the proposed framework for energy-efficient and adaptive next-generation wireless networks.
• 中文摘要 本研究提出一种移动元件（ME）多功能可重构智能表面（MF-RIS）架构，称为MM-RIS，将元件机动性与自适应信号反射、传输、放大和能量收集（EH）整合，以增强通信灵活性和能源自给自足性。还设计了混合速率分裂多址（RSMA）和非正交多址（NOMA）传输方案，以提升频谱效率。我们通过联合优化发射波束形成、功率与速率分配、MM-RIS配置、面板部署及ME位置，提出了能效最大化问题。为解决由此产生的非凸混合连续-离散优化问题，我们提出了结合联邦变换器和参数化共享（CRAFTS）的受限混合深度强化学习。CRAFTS通过双向参数共享整合多智能体近端策略优化（MAPPO）和多智能体深度Q网络（MADQN），分别处理连续和离散动作。此外，采用基于变换器的架构和联邦学习（FL）以增强状态表示和分布式知识共享，同时基于拉格朗日的自适应惩罚机制促进约束满足。模拟结果表明，所提出的MM-RIS架构和CRAFTS框架相比公开文献中的基准大幅提升了EE和约束满足度。此外，结果验证了空间可重构MM-RIS、混合RSMA-NOMA传输及智能协作学习的优势，凸显了该框架在节能且适应性高的下一代无线网络中的潜力。

Uncovering the Turing Test's Latent Signal: Indistinguishability as a Reward

揭示图灵测试的潜在信号：作为奖励的不可区分性

• Authors: Emanuele Rimoldi, Riccardo Neumarker, Tomaso Poggio, Daniel Mitropolsky
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.34781
• Pdf link: https://arxiv.org/pdf/2610.34781
• Abstract Can a model learn by trying to fool another model? We show that it can. Inspired by the Turing Test and its recent generalization to arbitrary distinguishers, we introduce a domain-based teacher-Turing-Test, in which a stronger teacher model serves as both the target of imitation and the distinguisher, and a weaker student model is trained to make the teacher attribute responses to itself. We use this self-attribution (or fooling) probability as a reinforcement learning reward, without ever providing the student ground-truth answers, nor requiring the teacher to generate responses. We test our approach independently in four domains with verifiable answers: CommonsenseQA (reasoning), GSM8K (mathematics), KodCode (programming), and Spider (SQL), training a Qwen2.5-1.5B student/imitator model using DeepSeek V4 Flash as teacher/distinguisher. Beyond improving at passing the Turing Test, the student improves on independent, held-out evaluations in all domains and runs. To probe why being mistaken for the teacher can improve capability, our analyses reveal that self-attribution is associated with correctness -- substantially more than formatting changes -- and that it rewards partial correctness. Our results demonstrate a new, simple teacher-student training paradigm with a private and/or high-cost teacher. They also suggest that indistinguishability need not merely measure behavior -- it can also be a mechanism for learning.
• 中文摘要 模型能否通过试图欺骗另一个模型来学习？我们证明了它确实可以。受图灵测试及其近期对任意区分器的推广启发，我们引入了基于领域的教师图灵测试，其中更强的教师模型既是模仿的目标，也是区分者，而较弱的学生模型则训练为教师将反应归于自身。我们将这种自我归因（或欺骗）概率作为强化学习奖励，从不向学生提供真实答案，也不要求教师生成答案。我们在四个可验证答案的领域独立测试我们的方法：CommonsenseQA（推理）、GSM8K（数学）、KodCode（编程）和Spider（SQL），并用DeepSeek V4 Flash作为教师/区分器训练Qwen2.5-1.5B学生/模仿者模型。除了提升通过图灵测试的能力外，学生在所有领域和运行中都能提升独立且长期的评估。为了探究为何被误认为教师能提升能力，我们的分析显示，自我归因与正确性相关——远大于格式变化——并且奖励部分正确性。我们的结果展示了一种新的简单师生培训范式，由私人和/或高成本教师指导。他们还表明，不可区分性不仅仅衡量行为——它也可以成为学习的机制。

Centralized Critics for N-Agent Ad Hoc Teamwork

N-agent临时团队合作的集中批评者

• Authors: Andrew McAlinden, Caroline Wang, Steven Shi, Arrasy Rahman, Elad Liebman, Peter Stone
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34799
• Pdf link: https://arxiv.org/pdf/2610.34799
• Abstract Centralized training with decentralized execution (CTDE) has become the dominant paradigm in multi-agent reinforcement learning (MARL), motivating extensive study of what privileged information should be made available to the critic during training. While prior work has examined the role of global state in fully cooperative settings, the design of centralized critics for N-agent ad hoc teamwork - in which dynamically varying set of learning agents must coordinate with an arbitrary set of uncontrolled teammates drawn from an unknown distribution - remains poorly understood. We study this question by comparing two natural classes of advantage estimators: controlled history estimators, which condition only on the observation and action histories of the learner's own agents, and joint history estimators, which additionally condition on the histories of the uncontrolled teammates. We prove that both estimators are unbiased and characterize the conditions under which the joint history estimate attains strictly lower variance than its controlled counterpart. Empirically, joint history critics match or outperform controlled history critics across a range of NAHT tasks, yielding improved sample efficiency and final returns. Together, these results provide both a theoretical justification and an empirical recommendation for incorporating uncontrolled agent histories into centralized critics for N-agent ad hoc teamwork.
• 中文摘要 去中心化执行的集中训练（CTDE）已成为多智能体强化学习（MARL）的主导范式，促使对在培训期间应向批评者开放哪些特权信息进行广泛研究。尽管此前研究探讨了全局状态在完全合作环境中的作用，但N代理临时团队合作的中心化批评者设计——即动态变化的学习主体必须与来自未知分布的任意一组未受控队友协调——仍鲜有理解。我们通过比较两类自然的优势估计器来研究这个问题：受控历史估计器，仅基于学习者自身代理的观察和行动历史;联合历史估计器，额外条件为未受控队友的历史。我们证明了两种估计均无偏，并描述了联合历史估计在何种条件下方差严格低于对照对应指标。实证上，联合历史分析者在多项NAHT任务中表现与受控历史分析者匹配或优于对控历史分析者，从而提高了样本效率和最终回报。这些结果共同为将非受控代理历史纳入N代理临时团队合作的集中批评者提供了理论依据和实证建议。

Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning

行为质量不是学习效用：机器人学习中人类行为模型的客观不匹配

• Authors: Haoyang Jiang, Dana Kulic, Michael Burke, Pamela Carreno-Medrano
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.34839
• Pdf link: https://arxiv.org/pdf/2610.34839
• Abstract Human behaviour models are increasingly used to provide simulated human interactions for training robots via reinforcement learning (RL). Existing approaches primarily evaluate these models according to how closely their generated behaviours resemble real human data, while their usefulness as training partners is less commonly assessed. This creates a potential objective mismatch: a model that performs well according to behavioural evaluation metrics may not necessarily provide useful training experiences for downstream robot policy learning. Across two human-robot interaction (HRI) domains, highway driving and object handover, we evaluate different types of human behaviour models using common generative-model evaluation metrics. We consider multiple human trajectory representations and compare the resulting evaluations with the downstream performance of RL policies trained using the corresponding models. We find that existing behavioural evaluation metrics do not consistently predict downstream policy performance across domains and trajectory representations, while their evaluations are sensitive to the choice of trajectory representation. We also show that even when the true testing behaviour distribution is known, training against a different distribution may yield better downstream performance than training directly against the true distribution.
• 中文摘要 人类行为模型越来越多地被用于通过强化学习（RL）为训练机器人提供模拟的人际互动。现有方法主要根据这些模型生成的行为与真实人类数据的相似度来评估这些模型，而它们作为训练伙伴的实用性则较少被评估。这导致了潜在的客观不匹配：一个根据行为评估指标表现良好的模型，未必能为下游机器人策略学习提供有用的训练体验。在两个人机交互（HRI）领域——高速公路驾驶和物体切换——中，我们使用常见的生成模型评估指标评估不同类型的人类行为模型。我们考虑多种人类轨迹表示，并将所得评估结果与使用相应模型训练的强化学习策略的下游表现进行比较。我们发现现有行为评估指标在不同领域和轨迹表示中并不能一致预测下游策略表现，而其评估对轨迹表示的选择则较为敏感。我们还表明，即使已知真实测试行为分布，针对不同分布进行训练可能比直接针对真实分布进行训练获得更好的下游表现。

End-to-End MXFP8 Reinforcement Learning on Blackwell: What Matching Buys, When It Pays Off, and How It Fails

Blackwell上的端到端MXFP8强化学习：匹配带来什么好处，何时有效，以及失败的原因

• Authors: Wengee Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.34915
• Pdf link: https://arxiv.org/pdf/2610.34915
• Abstract Low-precision arithmetic can make reinforcement learning (RL) for large language models cheaper, but RL requires the sampler and the learner to agree. Several frameworks now run FP8 on both sides. We implement end-to-end MXFP8 RL in verl (Megatron/Transformer Engine training; vLLM and SGLang rollout; dense and MoE models) and measure on B200 GPUs, with Qwen3-8B and Qwen3-30B-A3B, what this design buys, when it is faster, and how it fails. (1) Matching. For an MXFP8 learner, a sampler on the learner's grid is more consistent than a higher-precision BF16 sampler: in one engine at fixed weights, train-rollout KL is 11% lower with half of the sampler's linear layers on the grid and 16% lower with all of them, and the synchronized weights we checked equal the learner's FP8 forward weights byte for byte. On the MoE model, an FP8 pipeline whose sampler uses another grid has twice the shared-grid pipeline's KL from the first step. Without a quantized KV cache, BF16 end to end remains 2.5-10x more consistent. (2) Speed. MXFP8 training adds a fixed cost per micro-batch and saves a per-token cost, so in our dense setting it pays off only above about 4.5k tokens per micro-batch; a micro-batch sweep on identical data confirms the per-micro-batch cost, and a trace shows the GPU waiting for the host for most of the added time. End to end, MXFP8 cuts MoE step time by 10.7% and generation time by 24%, and slows a short-sequence dense workload by 5%. (3) Correctness. We state seven conditions every weight synchronization must preserve and checks that test them against the live engine. Each of the seven failures we met on hardware broke one condition, five silently; the checks caught two at their first opportunity, each missed by the other checks, and CPU fault-injection tests reproduce five. All runs are single-seed and at most 60 steps long; we make no claim about final model quality.
• 中文摘要 低精度算术可以使大型语言模型的强化学习（RL）更便宜，但强化学习需要采样器和学习者达成一致。现在有多个框架在FP8两侧都运行。我们在verl（Megatron/Transformer Engine训练;vLLM和SGLang部署;稠密和MoE模型）实现端到端MXFP8 RL，并在B200 GPU上，配合Qwen3-8B和Qwen3-30B-A3B，测量该设计带来的收益、速度更快以及失败机制。（1）匹配。对于MXFP8学习器，学习者网格上的采样器比高精度的BF16采样器更为一致：在一台固定权重下的发动机中，列车滑行的KL在采样器线性层的一半在网格上降低11%，所有线性层在网格上降低16%，我们检查的同步权重与学习者FP8的前向权重字节逐字节相等。在MoE模型中，采样器使用另一个网格的FP8流水线，从第一步起其共享网格流水线的基层级计算量是共享网格流水线的两倍。没有量化的KV缓存，BF16端到端的一致性保持2.5-10倍。（2）速度。MXFP8训练每微批增加固定成本，节省每个令牌成本，因此在我们的密集环境中，每微批次仅获得约4500个令牌的回报;对相同数据进行微批量扫描确认了每微批次的成本，跟踪显示GPU大部分时间都在等待主机。端到端，MXFP8将MoE步次时间缩短10.7%，生成时间减少24%，并降低短序列密集工作负载5%。（3）正确性。我们规定每个权重同步必须保持七个条件，并进行与运行引擎对照测试的检查。我们在硬件上遇到的七个失败中，每个都违反了一个条件，其中五个是悄无声息的;检查在第一次机会时发现了两个，其他检查都未完成，CPU故障注入测试重现了五个。所有运行均为单种子，最多60步;我们不保证最终模型质量。

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization

通过推广协议提升推理能力，以实现基于群体的策略优化

• Authors: Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.34951
• Pdf link: https://arxiv.org/pdf/2610.34951
• Abstract Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficiency. Despite these gains, the extent to which differences among correct-answer rollouts reflect useful alternative reasoning or unresolved errors remains poorly understood. We investigate this question through a blind paired analysis of a model trained with Group Relative Policy Optimization (GRPO), comparing the highest- and lowest-agreement correct-answer traces for the same problem. Across 126 pairs, low-agreement traces have more spurious reasoning than high-agreement traces (35% vs. 22%), including internal inconsistencies and misuse of problem conditions. Based on these findings, we propose RoAR (Rollout-Agreement Reward), which adds reasoning-level agreement to GRPO's rewards for verified-correct rollouts. RoAR computes this signal from existing on-policy rollout groups, requiring no human-written rationales, additional sampling, or separately trained process evaluator. On five mathematical reasoning benchmarks, the 1.5B GRPO-RoAR model reaches 55.5% mean pass@1 under zero-shot evaluation using 7K training examples, exceeding GRPO and GRPO-CARE by 6.0 and 5.5 percentage points. A separate blind audit of 100 problems answered correctly by all four 1.5B models finds spurious rates of 18% for GRPO-RoAR, 32% for GRPO, 27% for GRPO-CARE, and 37% for DRA-GRPO. Our findings identify reasoning validity as an important consideration when using rollout differences to guide learning. Our code is available at this https URL.
• 中文摘要 最新可验证奖励强化学习（RLVR）方法利用推广多样性提升推理表现或训练效率。尽管取得了这些进展，正确答案展开之间的差异是否反映了有用的替代推理或未解决错误，仍难以理解。我们通过对使用群体相对策略优化（GRPO）训练的模型进行盲对分析，比较同一问题中最高和最低一致正确答案的追踪。在126对中，低一致性的追踪比高一致性的追踪存在更多虚假推理（35%对22%），包括内部不一致和问题条件的误用。基于这些发现，我们提出了RoAR（推广-协议奖励），该机制在GRPO对验证正确推广的奖励中增加了推理层级的一致性。RoAR从现有的政策内推广组计算该信号，无需人工撰写理由、额外抽样或单独培训的过程评估员。在五个数学推理基准测试中，1.5亿GRPO-RoAR模型在零样本评估下均值pass@1达到55.5%，比GRPO和GRPO-CARE高出6.0个百分点和5.5个百分点。对四个1.5B模型全部正确回答的100个问题进行的单独盲审发现，GRPO-RoAR为18%，GRPO为32%，GRPO-CARE为27%，DRA-GRPO为37%。我们的发现表明推理效度是使用推广差异指导学习的重要考量。我们的代码可在此 https 网址获取。

SimPOTrans: Self-Adaptation for Offline-to-Simultaneous Speech Translation via Reinforcement Learning

SimPOTrans：通过强化学习实现离线到同步语音翻译的自我适应

• Authors: Haotian Tan, Hiroki Ouchi, Sakriani Sakti
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.35042
• Pdf link: https://arxiv.org/pdf/2610.35042
• Abstract Simultaneous speech translation (SimulST) aims to translate ongoing speech with low latency while preserving translation quality. Recent studies have adopted large language models (LLMs) for SimulST and formulated it as a multi-turn dialogue task for efficient generation, achieving state-of-the-art performance. However, existing dialogue-style SimulST approaches typically rely on specialized training schemes to acquire simultaneous translation behavior, either using synthesized read/write trajectories, which may provide suboptimal training signals, or jointly optimizing translation quality and latency, whose competing objectives may cause optimization instability. In this paper, we introduce SimPOTrans, a reinforcement learning (RL) framework that enables offline speech translation models to self-adapt to dialogue-style SimulST. Through a simple two-stage RL post-training pipeline, SimPOTrans rewards an offline model for exhibiting translation behaviors beneficial to simultaneous scenarios when given partial speech inputs, thereby enabling offline-to-simultaneous self-adaptation without synthesized read/write trajectories or explicit latency optimization. At inference time, SimPOTrans requires no external decision policy and enables flexible quality--latency control by simply adjusting the chunk size of speech input. Experiments on the ACL 60/60 and MCIF benchmarks across three language directions (En$\rightarrow$De, En$\rightarrow$Zh, and En$\rightarrow$Ja) demonstrate that SimPOTrans outperforms several strong baselines, achieving a superior quality--latency trade-off.
• 中文摘要 同声语音翻译（SimulST）旨在以低延迟翻译正在进行的语音，同时保持翻译质量。近期研究采用了大型语言模型（LLMs）作为SimulST的应用，并将其构建为多回合对话任务，实现高效生成，实现最先进的性能。然而，现有的对话式SimulST方法通常依赖专门的训练方案来获得同步翻译行为，要么使用合成的读写轨迹，这可能带来次优的训练信号，要么联合优化翻译质量和延迟，这两者的目标可能导致优化不稳定。本文介绍了SimPOTrans，一种强化学习（RL）框架，使离线语音翻译模型能够自我适应对话式SimulST。通过简单的两阶段强化学习后训练流程，SimPOTrans 奖励离线模型在获得部分语音输入时表现出有利于同时语场景的翻译行为，从而实现离线到同时进行的自我适应，无需合成读写轨迹或显式延迟优化。在推理阶段，SimPOTrans 无需外部决策策略，仅通过调整语音输入的块大小即可实现灵活的质量-延迟控制。在三个语言方向（En$\rightarrow$De、En$\rightarrow$Zh 和 En$\rightarrow$Ja）上的 ACL 60/60 和 MCIF 基准测试实验表明，SimPOTrans 优于多个强基线，实现了更优越的质量-延迟权衡。

DriveUniTok: A Unified Multimodal Tokenizer for Driving World-Action Models

DriveUniTok：一个统一的多模态分词器，用于驱动世界行动模型

• Authors: Jingbo Xu, Hanchao Leng, JiaHui Huang, Hanbing Li, Jingwei Zhao, Kuiyuan Yang, Chen Gao, Guang Chen, Hangjun Ye, Yong Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.35044
• Pdf link: https://arxiv.org/pdf/2610.35044
• Abstract World models are emerging as a fundamental component of policy learning for real-world agents, enabling agents to predict future world states and plan actions within a learned representation of the dynamic environment. Existing approaches, however, often model appearance, metric geometry represented by projected LiDAR depth, and ego motion either partially or independently, overlooking the fact that these modalities are different projections of a shared underlying world state. This separation can lead to information loss and cross-modal inconsistency, ultimately limiting downstream performance. Motivated by this unified view of the world, we propose DriveUniTok, a unified multimodal tokenizer for driving world-action models (WAMs) that encodes these modalities in a single decodable continuous latent. We freeze DriveUniTok and adapt a pretrained diffusion model, DriveUniWAM, to predict future unified latents; the tokenizer then decodes them into future appearance, projected LiDAR depth, and ego trajectories. Experiments on NAVSIM demonstrate that our unified design achieves better future prediction and planning performance at lower downstream computational cost than separate-stream alternatives. With reinforcement learning post-training, DriveUniWAM achieves a PDMS of 91.54, reaching state-of-the-art performance among the evaluated WAM-based planners.
• 中文摘要 世界模型正作为现实世界智能体策略学习的基本组成部分，使智能体能够预测未来世界状态并在动态环境的学习表现中规划行动。然而，现有方法常常部分或独立地建模外观、以投影LiDAR深度表示的度量几何和自我运动，忽视了这些模态是共享底层世界状态的不同投影。这种分离可能导致信息丢失和跨模态不一致，最终限制下游性能。受这种统一世界观的激励，我们提出了DriveUniTok，一种统一的多模态分词器，用于驱动世界动作模型（WAM），将这些模态编码为单一可解码的连续潜在变量。我们冻结DriveUniTok，并调整预训练扩散模型DriveUniWAM，预测未来的统一潜在因素;分词器随后将其解码为未来外貌、投影LiDAR深度和自我轨迹。NAVSIM上的实验表明，我们的统一设计在更优的预测和规划表现上，且下游计算成本低于分流方案。经过训练后强化学习，DriveUniWAM实现了91.54的PDMS，在评估的WAM规划者中达到最先进的性能。

Reinforcement Learning Done Right for Explicit Confidence Targets

针对明确信心目标的正确强化学习

• Authors: Zefeng He, Yichen He, Yuan Lin, Yang Yang, Yu Cheng
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.35059
• Pdf link: https://arxiv.org/pdf/2610.35059
• Abstract Large language models (LLMs) can produce plausible but incorrect answers when they are uncertain, which can mislead users and undermine model reliability. Training models to abstain with reinforcement learning is a natural approach, but most existing methods use a fixed error penalty and therefore cannot adapt their behavior to different confidence requirements at inference time. We study \emph{explicit confidence targets}, where a user specifies a target (p) in the prompt and the model directly chooses whether to answer or abstain, with the goal of achieving an accuracy of at least (p) among answered questions. However, learning to respond to different confidence targets with a single model has proven difficult, and prior attempts have failed to distinguish different targets. We show that this failure comes from two problems: the reward signal that distinguishes different values of $p$ is often lost during optimization, while the original error penalty grows without bound as $p$ approaches one. We propose RLEC (\textbf{R}einforcement \textbf{L}earning with \textbf{E}xplicit \textbf{C}onfidence Targets), which preserves the learning signal needed to distinguish confidence targets, introduces a stable reward family that keeps the desired answer condition unchanged, and uses reward scheduling to balance effective learning and training stability. Together, these designs enable the first successful training of a single LLM for explicit confidence target control. Extensive experiments across diverse benchmarks and model scales show that RLEC largely preserves general capabilities and achieves competitive calibration performance. RLEC also supports fine-grained target control without an external decision system or threshold calibration after training. By making explicit confidence targets part of the trained policy, RLEC opens a new and more direct path toward behavioral calibration.
• 中文摘要 大型语言模型（LLMs）在不确定时可能产生合理但错误的答案，这可能误导用户并削弱模型可靠性。训练模型避免强化学习是一种自然方法，但大多数现有方法采用固定的错误惩罚，因此无法在推理时调整其行为以适应不同的置信度要求。我们研究了\emph{显式置信目标}，即用户在提示中指定目标\（p\），模型直接选择回答或弃权，目标是实现至少答题中的\（p\）准确率。然而，用单一模型学习响应不同置信目标一直困难，且之前的尝试未能区分不同目标。我们证明了这种失败源于两个问题：区分不同$p$值的奖励信号在优化过程中常常丢失，而原始的错误惩罚则随着$p$接近1而无限制地增长。我们提出了RLEC（\textbf{R}einforcement \textbf{L}earning with \textbf{E}xplicit \textbf{C}onfidence Targets），该方法保留区分置信目标所需的学习信号，引入稳定的奖励家族，保持期望答案条件不变，并利用奖励调度平衡有效学习与训练的稳定性。这些设计共同实现了首次成功训练单个LLM以实现显式置信目标控制。跨不同基准测试和模型尺度的大量实验表明，RLEC在很大程度上保留了通用能力，并实现了竞争性校准性能。RLEC还支持在训练后无需外部决策系统或阈值校准的细粒度目标控制。通过将显式置信目标纳入训练策略，RLEC开辟了一条更直接的行为校准路径。

Learning to Remember: Advances in Memory Mechanisms for Large Language Models

学会记忆：大型语言模型记忆机制的进展

• Authors: Chenye Han, Hongche Yu, Mengyue Wu, Dell Zhang, Jixiang Luo
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.35089
• Pdf link: https://arxiv.org/pdf/2610.35089
• Abstract Memory has become a central mechanism for extending large language models (LLMs) beyond fixed context windows, stateless interaction, and static parametric knowledge. This survey organizes the literature around three complementary dimensions: the life cycle of memory, the storage substrate, and the system role that memory plays in LLM-based agents, while using cognitive-science-inspired memory concepts as a lightweight conceptual motivation rather than as a separate technical contribution or implementation family. On this basis, we distinguish three mechanism families: nonparametric memory, where information is stored outside model parameters in text, graph, multimodal, or database-like structures; parametric memory, where information is stored or accessed through activations, key-value caches, learned modules, adapter-based parameterizations, or model weights; and memory for agent evolution, where stored experiences support reinforcement learning, continual learning, procedural reuse, context engineering, and agent harness design. Evaluation is treated as a separate cross-cutting component rather than as a memory mechanism. Unlike surveys that focus primarily on retrieval-augmented generation or long-context modeling, this paper emphasizes the full memory life cycle: construction, maintenance, retrieval, consolidation, forgetting, governance, and evaluation. By aligning conceptual motivations, computational memory mechanisms, benchmarks, and deployment concerns, the survey provides a structured roadmap for designing memory-augmented LLM agents that are persistent, adaptive, auditable, and safe to update over time.
• 中文摘要 内存已成为将大型语言模型（LLM）扩展到固定上下文窗口、无状态交互和静态参数化知识之外的核心机制。本综述围绕三个互补维度组织文献：记忆生命周期、存储基质以及内存在基于LLM的代理中所扮演的系统角色，同时将受认知科学启发的记忆概念作为轻量级的概念动机，而非作为独立的技术贡献或实现家族。基于此，我们区分了三大机制家族：非参数记忆，信息存储在模型参数之外，如文本、图、多模态或类数据库结构;参数存储，信息通过激活、键值缓存、学习模块、基于适配器的参数化或模型权重来存储或访问;以及智能体进化记忆，存储经验支持强化学习、持续学习、过程重用、上下文工程和智能体约束设计。评估被视为一个独立的跨领域组成部分，而非记忆机制。与主要关注检索增强生成或长上下文建模的调查不同，本文强调完整的记忆生命周期：构建、维护、检索、巩固、遗忘、治理和评估。通过对齐概念动机、计算记忆机制、基准测试和部署关注点，调查为设计持久、自适应、可审计且安全可随时间更新的内存增强LLM代理提供了结构化路线图。

CalibraRL: Confidence-Calibrated Reinforcement Learning for Mitigating Hallucinations in RAG

CalibraRL：信心校准强化学习，用于缓解RAG中的幻觉

• Authors: Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, Zhiming Zheng
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.35142
• Pdf link: https://arxiv.org/pdf/2610.35142
• Abstract Large language models (LLMs) often produce plausible but incorrect answers, motivating reinforcement learning methods that reward correct answers, penalize incorrect answers, and allow abstention. However, fixed outcome-based rewards do not distinguish incorrect responses with different confidence levels or account for differences in answering reliability across queries. Consequently, such rewards may inadequately penalize overconfident errors and fail to discourage unnecessary abstention. We propose CalibraRL, a reinforcement learning framework that incorporates response confidence and query-level answering evidence into reward design. For non-abstaining responses, CalibraRL combines correctness-based utility with a Brier-based penalty that encourages agreement between confidence and correctness. For abstentions, it estimates conditional answering reliability using a Beta posterior derived from the correctness of non-abstaining responses within each sampled group. The abstention reward increases with the posterior probability that the model's accuracy on non-abstaining responses is below a predefined threshold. CalibraRL reuses training rollouts without requiring additional annotations indicating whether the model knows the answer to each question. Experiments on three QA datasets demonstrate that CalibraRL improves answer precision and reduces hallucinations compared with strong baselines. Further analysis shows that CalibraRL reduces mean expected calibration error (ECE) by 30.8\% across two backbone models and three datasets, highlighting its potential to improve the reliability of retrieval-augmented generation (RAG) in web search.
• 中文摘要 大型语言模型（LLM）常常产生合理但错误的答案，促使强化学习方法奖励正确答案，惩罚错误答案，并允许弃权。然而，基于结果的固定奖励并未区分置信度不同的错误答案，也无法考虑不同查询的答案可靠性差异。因此，此类奖励可能无法充分惩罚过度自信的错误，也无法阻止不必要的弃权。我们提出了CalibraRL，一种将回答信心和查询级回答证据纳入奖励设计的强化学习框架。对于非弃权回答，CalibraRL结合了基于正确性的实用性和基于Brier的惩罚，鼓励信心与正确性之间的一致性。对于弃权，它通过基于每个抽样组内非弃权回答正确性得出的Beta后验估计条件答案信度。弃权奖励随着模型非弃权回答准确率低于预设阈值的后验概率增加。CalibraRL可重复使用训练扩展，无需额外注释说明模型是否知道每个问题的答案。对三个QA数据集的实验表明，CalibraRL相比强基线提高了答案精度并减少了幻觉。进一步分析显示，CalibraRL在两个骨干模型和三个数据集中平均期望校准误差降低了30.8%的百分比，凸显了其提升网络搜索中检索增强生成（RAG）可靠性的潜力。

DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models

DTCA：扩散语言模型中强化学习中的去噪令牌学分赋值

• Authors: Hongze Chen, Miao Pan, Lizhou Liang, Qinfeng Li, Xinyin Ma, Yongliang Shen, Weishi Wang, Daniel Dahlmeier, Xuhong Zhang, Wenqi Zhang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.35201
• Pdf link: https://arxiv.org/pdf/2610.35201
• Abstract Diffusion language models (DLMs) generate text through iterative denoising, enabling bidirectional context modeling and parallel token refinement. Recent studies have applied reinforcement learning (RL) to DLMs and improved their reasoning performance. However, the iterative denoising process makes credit assignment more challenging, as final outcome rewards provide limited information about how different tokens contribute to the final output. In this paper, we propose Denoising Token Credit Assignment (DTCA), a framework for token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after token commitment and groups these signals according to part-of-speech (POS) categories. These contribution weights are used to select informative denoising timesteps and assign token-level learning signals during RL training. DTCA further calibrates these learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.
• 中文摘要 扩散语言模型（DLMs）通过迭代去噪生成文本，实现双向上下文建模和并行词源优化。近期研究将强化学习（RL）应用于DLM并提升了其推理性能。然而，迭代去噪过程使得信用分配更具挑战性，因为最终结果奖励提供的信息有限，说明不同代币如何对最终输出贡献。本文提出去噪令牌信用分配（DTCA），这是一个用于扩散语言模型强化学习中代币级信用分配的框架。DTCA通过测量代币承诺后剩余掩蔽位置的预测置信度提升来估算代币贡献，并根据词性（POS）类别对这些信号进行分组。这些贡献权重用于选择有用的去噪时间步，并在强化学习训练中分配代币级学习信号。DTCA进一步根据策略更新引起的令牌级熵变化校准这些学习信号。LLaDA-8B-Instruct上的实验显示，DTCA在不同世代预算中持续提升数学推理性能，MATH500在基础模型上提升了最多9.0点，GSM8K提升了5.3个百分点。

InterQ-RL: Interaction-Aware Correction of Execution Mismatch in Quantized Agentic RL

InterQ-RL：量化智能强化学习中执行不匹配的交互感知纠正

• Authors: Yingbo Hao, Song Liu, Guanghua Yu, Zhaojing Zhou, Jianchen Zhu, Yi Zou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.35303
• Pdf link: https://arxiv.org/pdf/2610.35303
• Abstract Multi-turn agent reinforcement learning (RL) relies on long closed-loop rollouts, making generation and KV-cache costs a major bottleneck. Quantized rollout reduces these costs, but its low-precision executor also serves as the behavior policy and can induce execution mismatch with the learner. Existing quantized-RL methods are designed primarily for single-turn generation. They align the learner forward with quantized rollout or correct the current-token rollout-learner ratio, without modeling the policy history that produces each agent state. Even after quantization alignment, residual rollout-learner mismatch accumulates along the state-reaching policy history, so a current action may look locally aligned despite a mismatched history. We introduce InterQ-RL (Interaction-aware Quantized Reinforcement Learning) to correct this post-alignment mismatch. It applies bounded execution-prefix correction over the state-reaching policy history, allocates correction strength across action segments by effective support under a fixed displacement budget, and requires no additional model forward pass. We evaluate dense and mixture-of-experts (MoE) backbones from 4B to 30B on ALFWorld, WebShop, and ScienceWorld, with all quantized arms sharing the same rollout with 8-bit weights, activations, and KV cache (W8A8-KV8). Across four environment-backbone settings, InterQ-RL achieves the strongest quantized endpoint result, matching BF16 to within 0.7 points and exceeding it in two settings. It matches or exceeds the learning speed of alignment alone while avoiding its late performance regression.
• 中文摘要 多回合智能体强化学习（RL）依赖于长时间的闭环滚动，使生成和KV缓存成本成为主要瓶颈。量化展开降低了这些成本，但其低精度执行器同时作为行为策略，可能诱导与学习者的执行不匹配。现有的量化强化学习方法主要设计为单回合生成。它们通过量化展开前进对齐学习者，或纠正当前令牌的滚动-学习者比例，而未建模产生每个代理状态的策略历史。即使在量化对齐后，残余的滚动-学习器不匹配仍会沿状态覆盖策略历史积累，因此当前动作可能看起来是局部对齐的，尽管历史不匹配。我们引入InterQ-RL（交互感知量化强化学习）来纠正这种后对齐不匹配。它对状态覆盖策略历史的执行前缀校正有界，在固定位移预算下通过有效支持在动作段间分配纠正强度，且无需额外的模型前向传递。我们在ALFWorld、WebShop和ScienceWorld上评估了4B至30B的密集和专家混合骨干，所有量化臂共享同一部署，均为8位权重、激活和KV缓存（W8A8-KV8）。在四种环境骨干设置中，InterQ-RL实现了最强的量化端点结果，将BF16匹配到0.7点以内，并在两个设置中超过。它仅匹配或超过对齐的学习速度，同时避免了后期性能回归。

Belief-Conditioned Multi-Agent Actor Critic for Decentralized LLM Collaboration

去中心化大型语言模型协作的信念条件多智能体演员批评者

• Authors: Tianle Chen, Shuo Liu, Christopher Amato
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.35398
• Pdf link: https://arxiv.org/pdf/2610.35398
• Abstract Recent work has explored optimizing LLM collaboration through Multi-Agent Reinforcement Learning (MARL), but many approaches still rely on predefined workflows, limiting efficiency and scalability. Decentralized LLM collaboration is challenging because agents execute independently with limited communication, share a joint objective, and must infer useful information from communication traces. In this paper, we propose BCMAAC, a Belief-Conditioned Multi-Agent Actor-Critic method that maintains beliefs over contextual information and uses them to facilitate training. Experiments on Minecraft game-playing, web searching and tool-using tasks show that BCMAAC improves task performance and collaborative behavior compared to naive actor-critic baselines. The improvements are particularly pronounced in multi-turn tasks with richer contextual information, where belief modeling provides a clear advantage.
• 中文摘要 近期研究探讨了通过多智能体强化学习（MARL）优化LLM协作，但许多方法仍依赖预设工作流，限制了效率和可扩展性。去中心化LLM协作具有挑战性，因为智能体独立执行且通信有限，共享共同目标，且必须从通信痕迹中推断有用信息。本文提出BCMAAC，一种信念条件多代理行为者-批评者方法，它维护信念而非上下文信息，并利用信念促进训练。Minecraft游戏、网页搜索和工具使用任务的实验显示，BCMAAC相比简单的演员-批评者基线提升了任务表现和协作行为。这些改进在多回合任务中尤为明显，且上下文信息更丰富，信念建模具有明显优势。

GPDPO: Learning from Reward Conflicts via Group Pareto-Dominance Policy Optimization

GPDPO：通过群体帕累托主导政策优化从奖励冲突中学习

• Authors: Zixuan Xu, Haoxuan Chen, Jiangxin Sun, Jian-Fang Hu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.35480
• Pdf link: https://arxiv.org/pdf/2610.35480
• Abstract Multi-reward reinforcement learning offers a promising framework for aligning large language models (LLMs) with diverse objectives, but conflicting reward signals can induce ambiguous update directions and hinder stable optimization. Existing approaches address such conflicts through reward reweighting, gradient coordination, or rollout filtering, yet they primarily operate on per-response or group-level optimization signals and underexploit relational information among sampled responses. As a result, conflict-filtered rollouts may still contain useful supervision that is discarded during training. We propose Group Pareto-Dominance Policy Optimization (GPDPO), a multi-reward policy optimization method that recovers pairwise supervision from conflict-filtered rollouts. GPDPO identifies Pareto-dominance relations among sampled responses and converts them into auxiliary preference signals, which are jointly optimized with the original policy objective. This enables more effective reuse of existing rollouts without additional sampling or reward evaluation. Experiments on API-Bank across multiple model scales and reward configurations show that GPDPO consistently outperforms competitive multi-reward optimization baselines. Further analysis demonstrates that GPDPO reduces both group-relative and pairwise reward conflicts, supporting the effectiveness of exploiting relational supervision for multi-reward alignment. Code will be released.
• 中文摘要 多奖励强化学习为将大型语言模型（LLMs）与多样目标对齐提供了有前景的框架，但冲突的奖励信号可能导致更新方向模糊，阻碍稳定优化。现有方法通过奖励加权、梯度协调或rollout filter来解决此类冲突，但它们主要对每个响应或组级优化信号进行操作，且未能充分利用抽样响应中的关系信息。因此，冲突过滤后的rollout可能仍包含有用的监督，但在训练过程中被舍弃。我们提出了Group Pareto-Dominance Policy Optimization（GPDPO），这是一种多奖励策略优化方法，能够从冲突过滤后的部署中恢复两两监督。GPDPO识别抽样响应间的帕累托优势关系，并将其转换为辅助偏好信号，并与原始策略目标共同优化。这使得现有推广更有效地重用，无需额外抽样或奖励评估。在多个模型尺度和奖励配置的API-Bank实验显示，GPDPO始终优于竞争性的多奖励优化基线。进一步分析表明，GPDPO减少了群体相对和成对奖励冲突，支持利用关系监督实现多奖励对齐的有效性。代码将发布。

S$^2$-RL: Sample-Set Dual Reinforcement Learning for Generative Semantic Segmentation Dataset Distillation

S$^2$-RL：生成语义切割数据集蒸馏的样本集双重强化学习

• Authors: Haoyu Wang, Fei Zhou, Qingqing Qiu, Lei Zhang, Wei Wei, Chen Ding
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.35491
• Pdf link: https://arxiv.org/pdf/2610.35491
• Abstract While dataset distillation has witnessed significant advances in image classification, its extension to dense prediction tasks such as semantic segmentation is plagued by two core bottlenecks: bi-level optimization-based approaches suffer from prohibitive computational costs stemming from pixel-wise gradient unrolling, whereas proxy-based generative methods merely infuse visual textures into fixed semantic masks derived from real data, thus failing to condense rich semantic knowledge into a compact set of informative, novel scene generations. To address both bottlenecks simultaneously, we propose a novel Sample-Set Reinforcement Learning (S$^2$-RL) framework for generative semantic segmentation dataset distillation (SSDD). Specifically, S$^2$-RL formulates SSDD as a diffusion model based text-to-image generation paradigm, enabling flexible generation of images with informative, novel semantic distributions via a concise text prompt. Furthermore, we fine-tune a diffusion policy via Group Relative Policy Optimization (GRPO), leveraging a sample-set dual reward paradigm: for individual samples, it enforces semantic alignment with input text prompts and intra-class feature diversity; for sample sets, it maximizes inter-sample semantic distribution diversity by solving a contextual multi-armed bandit problem. These design choices enable S$^2$-RL to distill large-scale semantic segmentation datasets into a compact set of generated samples that encapsulate diverse, informative semantic knowledge without the need of pixel-wise gradient unrolling. Extensive experiments on the ADE20K and COCO datasets demonstrate that S$^2$-RL achieves substantial and consistent improvements over state-of-the-art baselines, establishing a new benchmark for SSDD.
• 中文摘要 尽管数据集蒸馏在图像分类方面取得了显著进展，但其扩展到如语义分割等密集预测任务时，仍面临两个核心瓶颈：基于双层优化的方法因像素级梯度展开而导致的计算成本过高，而基于代理的生成方法仅将视觉纹理注入来自真实数据的固定语义掩体中，未能将丰富的语义知识浓缩成紧凑且富有信息量的新场景生成。为同时解决这两个瓶颈，我们提出了一种新颖的样本集强化学习（S$^2$-RL）生成语义分割数据集蒸馏（SSDD）框架。具体来说，S$^2$-RL将SSDD表述为基于扩散模型的文本到图像生成范式，通过简洁的文本提示实现灵活生成具有信息量且新颖语义分布的图像。此外，我们通过群相对策略优化（GRPO）微调扩散策略，利用样本集双重奖励范式：对单个样本，通过输入文本提示和类内特征多样性强制语义对齐;对于样本集，通过解决上下文多臂强盗问题，最大化样本间语义分布多样性。这些设计选择使S$^2$-RL能够将大规模语义分割数据集提炼成紧凑的生成样本集，封装多样且富有信息量的语义知识，无需像素梯度展开。在ADE20K和COCO数据集上的大量实验表明，S$^2$-RL相较于最先进基线实现了显著且一致的改进，树立了SSDD的新基准。

Enhancing Deep Reinforcement Learning for Robotic Motion Planning via Task Decomposition

通过任务分解增强机器人运动规划的深度强化学习

• Authors: Yuanzhi He, Victor Romero-Cano, Jiayu Xu, Juan David Hernández, Gualtiero Colombo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.35494
• Pdf link: https://arxiv.org/pdf/2610.35494
• Abstract Training robotic manipulation policies in simulation is widely adopted due to safety and cost efficiency; however, tasks involving obstacles remain under-performing and often converge slowly under Deep Reinforcement Learning (DRL). To address this challenge, we propose a task decomposition framework for DRL that integrates Transfer Learning (TL) with Reward Redesign (RR) to improve learning efficiency and robustness in obstacle environments. The TL strategy selectively transfers only the first-layer critic parameters across structurally related tasks, since that layer captures general state--action representations while deeper layers encode reward- and obstacle-specific value geometry; RR decomposes tasks into staged sub-objectives to provide denser optimization signals. Experiments on the PyBullet Panda platform show consistent gains in obstacle settings. On Push with obstacle, our method achieves $0.83 \pm 0.05$, outperforming the strongest baseline (Truncated Quantile Critics (TQC), $0.77 \pm 0.06$) by approximately $7.8\%$. On Slide with obstacle, performance reaches $0.95 \pm 0.02$, exceeding Soft Actor-Critic (SAC) ($0.84 \pm 0.06$) by about $11\%$ success rates. In addition to higher final performance, forward transfer reduces convergence time by up to $70\%$ in obstacle tasks, and RR shortens sparse-reward Pick-and-Place training from roughly $0.75$M to $0.20$M steps ($\approx73\%$ reduction). The best policy is deployed on a real Franka Emika FR3 robot as a qualitative feasibility demonstration.
• 中文摘要 由于安全性和成本效益，模拟中机器人操作策略的训练被广泛采用;然而，涉及障碍的任务表现仍然不理想，且在深度强化学习（DRL）下收敛速度较慢。为应对这一挑战，我们提出了一个任务分解框架，将转移学习（TL）与奖励重设计（RR）整合，以提升障碍环境中的学习效率和稳健性。TL策略仅选择性地将第一层批判参数传递到结构相关任务中，因为该层捕捉的是一般状态-动作表示，而更深层则编码奖励和障碍物特定的价值几何;RR将任务分解为分阶段子目标，以提供更密集的优化信号。在PyBullet Panda平台上的实验显示障碍设置持续提升。在带障碍推行中，我们的方法实现了0.83 \pm 0.05$，比最强的基线（截断分位批评者（TQC）0.77 \pm 0.06$）高出约7.8\%$。在带障碍滑行中，表现达到$0.95 \pm 0.02$，成功率比Soft Actor-Critic（SAC）（0.84美元/pm 0.06$）高出约11%%$。除了更高的最终表现外，前向转移还能将障碍任务的收敛时间减少多达70%美元，RR将稀疏奖励的挑选与放置训练步数从约0.75美元缩短到0.20美元（约73%%$减少）。最佳策略是在真实的Franka Emika FR3机器人上进行定性可行性演示。

When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing

细粒度学分作业既有帮助也有弊端：通过视频剪辑实现的多模态代理强化学习

• Authors: Lei Zhao, Zongheng Cao, Xinyu Hu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.35497
• Pdf link: https://arxiv.org/pdf/2610.35497
• Abstract Reinforcement learning (RL) has shown promise for training multimodal agents to reason and use tools. However, it remains unclear when fine-grained credit assignment improves learning and when it fails to improve task performance. Video editing provides a useful testbed for this question, combining multimodal perception, long-horizon planning, and tool use with delayed outcome feedback. We construct four movie-trailer editing task types---Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut---that reflect real-world editing workflows and span different levels of task complexity and interaction horizon. Decaying process rewards provides the largest benefit on Fine Cut, with a 45.4% relative gain over outcome-only GRPO on tasks averaging more than 60 turns under outcome-only GRPO. On the three shorter-horizon task types averaging 30--40 turns under the same baseline, outcome-only GRPO remains competitive, with decaying process rewards yielding at most a 10.4% relative gain. Process rewards from an LLM judge initially improve tool use and encourage visual inspection. As training progresses, however, GRPO+PRM with a fixed process-reward weight increasingly exhibits excessive reasoning and redundant tool calls: process rewards continue to rise while task completion rates and outcome scores decline. Decaying the process-reward weight to zero mitigates this reward-hacking behavior while preserving the early gains. Despite dense token-level supervision, on-policy self-distillation (OPSD) yields little improvement or substantial degradation and is highly sensitive to the choice of privileged information. Our analysis reveals differences in teacher--student disagreement and gradient magnitudes, and suggests possible dilution of informative signals over long trajectories.
• 中文摘要 强化学习（RL）已显示出在训练多模态代理推理和使用工具方面展现出潜力。然而，细粒度的学分分配何时能改善学习，何时未能提升任务表现，仍不明确。视频剪辑为这一问题提供了有用的试验平台，结合了多模态感知、长远视野规划和工具使用，同时延迟结果反馈。我们构建了四种电影预告片编辑任务类型---内容选择、叙事序列、粗剪和精细剪辑---反映了现实世界的剪辑工作流程，涵盖不同任务复杂度和交互视野。衰减过程奖励在细剪中带来最大优势，在单纯结果GRPO下，任务平均超过60回合，相较结果仅GRPO提升45.4%。在三种平均30-40回合的短视野任务类型中，仅结果的GRPO依然具有竞争力，递减的过程奖励最多带来10.4%的相对收益。LLM评审的过程奖励最初改善工具使用并鼓励视觉检查。然而，随着培训的推进，固定过程-奖励权重的GRPO+PRM越来越多地表现出过度推理和冗余的工具调用：进程奖励持续上升，而任务完成率和结果评分下降。将过程-奖励权重减至零可以减轻这种奖励黑客行为，同时保留早期收益。尽管存在密集的代币级监督，策略自蒸馏（OPSD）几乎没有带来显著改善或显著退化，且对特权信息的选择极为敏感。我们的分析揭示了师生分歧和梯度大小的差异，并暗示信息信号可能在长期轨迹中被稀释。

Test-Time Confidence Adaptation under RL-Induced Trajectory Shift

在强化学习引发轨迹偏移下的测试时间置信度适应

• Authors: Zhenzhen Ren, Xiaoqing Zhang, GuoBiao Li, Bowen Fang, Yanxi Wang, Xinpeng Zhang, Ke Han
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.35500
• Pdf link: https://arxiv.org/pdf/2610.35500
• Abstract Trajectory-based confidence estimation has shown promising performance for large language models (LLMs) by leveraging rich signals from their reasoning processes. However, we find that such confidence estimators can degrade substantially after agentic reinforcement learning (Agentic RL) as it induces substantial shifts in the reasoning trajectories. More importantly, despite these shifts, we uncover a reliability structure that is largely preserved across Base and RL models: $\textbf{more}$ $\textbf{stable}$ $\textbf{trajectories}$ $\textbf{remain}$ $\textbf{more}$ $\textbf{likely}$ $\textbf{to}$ $\textbf{produce}$ $\textbf{correct}$ $\textbf{answers}$. This preserved relationship provides a transferable, label-free reliability signal for post-RL confidence adaptation. Building on this finding, we propose $\textbf{StaR-TTA}$, a stability-guided framework for label-free test-time confidence adaptation. StaR-TTA jointly learns from correctness supervision and stability-derived reliability supervision on labeled Base trajectories, establishing the connection between trajectory stability and answer reliability. At test time, it reuses stability-induced reliability rankings from unlabeled RL trajectories to adapt confidence scores to the shifted trajectory distribution, without requiring target correctness labels. Across four QA benchmarks, StaR-TTA achieves an average AUROC of (0.721), with consistent improvements across unseen datasets, model families and scales, and different Agentic RL algorithms.
• 中文摘要 基于轨迹的置信度估计通过利用推理过程的丰富信号，在大型语言模型（LLMs）中展现出有希望的表现。然而，我们发现此类置信估计在代理强化学习（代理强化学习）后可能显著退化，因为它会引发推理轨迹的显著转变。更重要的是，尽管存在这些变化，我们发现了一个在基础模型和强化学习模型中基本保持的可靠性结构：$\textbf{more}$ $\textbf{stable}$ $\textbf{trajectories}$ $\textbf{remain}$ $\textbf{more}$ $\textbf{likely}$ $\textbf{to}$ $\textbf{produce}$ $\textbf{correct}$ $\textbf{correct}$ $\textbf{answers}$。这种保持的关系为后强化学习信心适应提供了可转移的无标签信度信号。基于此发现，我们提出了$\textbf{StaR-TTA}$，一个稳定指导的无标签测试时间置信度适应框架。StaR-TTA结合了正确性监督和稳定性导出的信度监督，在带标记的Base轨迹上学习，建立了轨迹稳定性与答案信度之间的联系。在测试时，它重用未标记强化学习轨迹中的稳定性诱导信度排名，将置信度分数调整到轨迹分布偏移，无需标注目标正确性标签。在四个质量保证基准中，StaR-TTA的平均AUROC为\（0.721\），在未见数据集、模型族和量表以及不同的能动强化学习算法中均有持续提升。

S3AC: Stabilized, Selective, and Scalable Actor-Critic for LLM Reasoning

S3AC：稳定、选择性和可扩展的LLM推理演员批评者

• Authors: Shuo Shen, Guangran Cheng, Chengqi Lyu, Xiangsen Wang, Yanan Sun, Wenwei Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.35518
• Pdf link: https://arxiv.org/pdf/2610.35518
• Abstract Critic-free methods are widely used in reinforcement learning with verifiable rewards (RLVR) for long chain-of-thought (CoT) reasoning, avoiding difficult and costly critic training. Common alternatives lack token-level credit assignment and typically rely on grouped rollouts to estimate prompt-specific baselines. A critic can address both limitations, but long-CoT critic training is hindered by insufficient coverage of trajectories with diverse lengths and early gradients dominated by the newly initialized value head. We introduce S3AC (Stabilized, Selective, and Scalable Actor-Critic) for LLM reasoning, built around a practical critic-training recipe. The trained critic provides token-specific advantages and prompt-specific baselines. Using the prompt-level baseline to estimate difficulty, we propose Difficulty-Aware Prompt Sampling (DAPS). DAPS combines this estimate with outcome history to select prompts before generation, reducing rollout waste without offline difficulty initialization. S3AC improves group-rollout performance across dense and mixture-of-experts models from 1.5B to 397B parameters, outperforming a modern GRPO variant (GRPO-X). At 35B, it sustains single-rollout training while tested critic-free baselines deteriorate after initial gains.
• 中文摘要 无批评方法广泛应用于可验证奖励强化学习（RLVR）中，用于长思考链（CoT）推理，避免了困难且昂贵的批评者培训。常见替代方案缺乏代币级的学分分配，通常依赖分组推广来估算提示特定的基线。批评者可以同时解决这两个局限，但长期CoT批评者训练受限于对不同长度和早期梯度的轨迹覆盖不足，且这些路径由新初始化的值头主导。我们引入了S3AC（稳定、选择性和可扩展的演员-批评者）用于大型语言模型推理，基于实用的批评者训练方案。训练有素的批评者提供代币特定的优势和提示特异的基线。利用提示级基线估算难度，我们提出了难度感知提示采样（DAPS）。DAPS将该估计与结果历史结合，在生成前选择提示，减少推广浪费，无需离线初始化困难。S3AC在密集和专家混合模型中提升了从1.5B到397B参数的组推广性能，优于现代GRPO变体（GRPO-X）。在35B时，它能维持单次推广训练，而无批评的基线在初始提升后会恶化。

Evolve on Graph: Self-Evolving Knowledge Graph Reasoning without Question Annotations

图谱演化：无问题注释的自我演化知识图谱推理

• Authors: Zongyuan Sun, Shuai Zhao, Ke Shu, Ziyue Zhu, Shangyang Wu, Yuanlin Lin, Tao Wang, Shiyue Li, Yingjun Du, Anh Tuan Luu, Haoran Luo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.35598
• Pdf link: https://arxiv.org/pdf/2610.35598
• Abstract Knowledge graphs (KGs) provide reliable external knowledge for multi-hop factual reasoning with large language models (LLMs), yet most KGs in real-world domains such as finance and industry lack question--answer (QA) annotations. Without such annotations, existing methods struggle to improve LLM graph reasoning: training-free agents cannot improve their exploration through interaction; reinforcement learning (RL) agents depend on QA data annotated by humans or synthesized by closed-source models; and training data is typically built offline, so its topology mix is fixed and cannot follow the agent's ability. We propose Evolve on Graph (EoG), a framework that lets an LLM self-evolve its graph reasoning without any QA annotations. Through Graph Self-Play, a single LLM alternates between a Proposer that poses questions and a Solver that answers them. To make questions verifiable, Symbolically Verifiable Topology Synthesis (SVTS) samples reasoning topologies on the KG and executes the corresponding first-order logic queries, yielding topology skeletons with correct and complete answer sets; guided by a skeleton, the Proposer explores the graph to pose a multi-hop question. To let task topologies follow the Solver, a Boundary-Aware Graph Curriculum rewards the Proposer by the Solver's score variance, keeping questions at the Solver's capability boundary so that complex topologies enter training as the Solver improves. On six benchmarks over Freebase, Wikidata, and an industrial KG, EoG consistently outperforms prompting-, retrieval-, and agent-based methods, showing that LLM graph reasoning can improve through self-play without QA annotations. Our code is available at this https URL
• 中文摘要 知识图谱（KG）为大型语言模型（LLM）的多跳事实推理提供了可靠的外部知识，但现实世界如金融和工业领域的大多数知识图谱缺乏问答（QA）注释。没有此类注释，现有方法难以提升LLM图推理：无训练代理无法通过交互提升探索能力;强化学习（RL）代理依赖于人工注释或闭源模型综合的质量保证数据;训练数据通常是离线构建的，因此其拓扑组合固定，无法遵循代理的能力。我们提出了Evolve on Graph（EoG）框架，允许LLM在无需QA注释的情况下自我演化其图推理。通过图自演，单个LLM在提出问题的提议者和回答问题的求解者之间交替运行。为了使问题可验证，符号可验证拓扑综合（SVTS）在 KG 上对推理拓扑进行抽样，并执行相应的一阶逻辑查询，生成具有正确且完整答案集的拓扑骨架;在骨架引导下，提议者探索图以提出多跳问题。为了让任务拓扑跟随求解器，边界感知图课程通过求解者的得分方差奖励提议者，保持问题在求解器能力边界，使复杂拓扑在求解器改进时进入训练状态。在 Freebase、Wikidata 和工业 KG 的六个基准测试中，EoG 持续优于提示、检索和基于代理的方法，表明通过自玩且无需质量保证注释，LLM 图推理可以提升。我们的代码可在此 https URL 获取

EvoRouter: Cost-Efficient Multi-Agent Routing via Insight-Guided On-Policy Distillation

EvoRouter：通过洞察引导的策略内提炼实现的多智能体高效路由

• Authors: Yitao Xiao, Xu Zhang, Ye Cheng, Shaoyong Guo, Yan Sun, Ziyue Zhu, Sitao Xie, Guo Tong, Haoran Luo
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.35690
• Pdf link: https://arxiv.org/pdf/2610.35690
• Abstract Serving web queries at scale with multi-agent systems requires jointly configuring large language models, collaboration modes, agent counts, interaction rounds, and skills. However, the resulting configuration space grows combinatorially, while sparse execution rewards provide limited fine-grained supervision for interdependent planning decisions. We propose EvoRouter, a self-evolving multi-agent routing framework driven by insight-guided on-policy distillation. A dedicated expert periodically synthesizes performance-cost feedback from configuration trajectories across tasks into reusable conditional routing insights about capability requirements, returns from resource scaling, and interactions among configuration choices. A privileged teacher combines these evolving insights with current execution feedback to provide dense token-level supervision for the router's own planning trajectories through on-policy distillation. Cost-aware reinforcement learning jointly optimizes task performance and computational cost. Together, these training signals internalize execution insights into the routing policy. The updated policy generates new trajectories that support subsequent insight extraction, forming an execution-driven self-improvement loop. We evaluate EvoRouter on 19 in-domain datasets spanning five domains and two out-of-domain benchmarks. EvoRouter achieves an average in-domain accuracy of 68.69%, outperforming the strongest routing baseline by 7.23 percentage points while reducing inference compute cost by 50.95%. Our code is open-sourced at this https URL.
• 中文摘要 用多智能体系统大规模提供网络查询需要联合配置大型语言模型、协作模式、代理数量、交互轮次和技能。然而，最终的配置空间是组合式增长的，而稀疏的执行奖励则为相互依赖的规划决策提供了有限的细粒度监督。我们提出了EvoRouter，一种由洞察导向策略提纯驱动的自我演进多智能体路由框架。专职专家定期将任务中配置轨迹的性能成本反馈综合为可重用的条件路由洞见，涵盖能力需求、资源扩展收益及配置选择之间的交互。特权教师将这些不断演变的洞见与当前执行反馈结合，通过策略中蒸馏为路由器自身规划轨迹提供密集的令牌级监督。成本感知强化学习共同优化任务性能和计算成本。这些训练信号共同将执行洞察内化到路由策略中。更新后的策略生成支持后续洞察提取的新轨迹，形成以执行驱动的自我改进循环。我们在跨越五个域的19个域内数据集和两个域外基准测试中评估了EvoRouter。EvoRouter的平均域内准确率为68.69%，比最强的路由基线高出7.23个百分点，同时降低了50.95%的推理计算成本。我们的代码开源于此 https URL。

Audit Where Hacks Pay: Budgeted Strong-Verifier Audits Against Reward Hacking in GRPO

黑客攻击的审计：预算强验证者审计针对GRPO中奖励黑客行为

• Authors: Shurun Li, Michael Hua Wang, Haibo Zeng, Long Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.35716
• Pdf link: https://arxiv.org/pdf/2610.35716
• Abstract Reinforcement learning with verifiable rewards trains on cheap checks, such as unit tests, that a policy can learn to game, while a stronger check is too expensive to run on every rollout. We study how to spend a limited strong-verifier budget in GRPO, separating which weakly passing rollouts to audit from what reward a caught hack receives. At a fixed policy state with centered advantages, the expected update depends on a design only through each rollout's marginal audit probability and correction; reflecting a caught hack's reward about the failure reward negates the advantages of an all-hack group; and auditing whole groups with the fewest weak passes first (low-pass budgeted audits, LB) minimizes the advantage mass left on hacks among pass-count designs with a fixed correction when a label's value falls with the pass count. We tested this analysis in actual LLM training: from each run's starting state, the sign of this mass on a frozen panel predicted whether hacking would be suppressed in 33 of 39 runs, including 13 of 16 preregistered ones. In LeetCode (Wong et al., 2026) and Countdown-Code (Khalifa et al., 2026), LB drives hacking to near zero, while random audits with the same penalty and budget do not and use more labels. We also find that the randomness of the update affects training: in Countdown-Code, two unbiased corrections, per-item reward-level IPW and a group-level advantage correction, bring hacking down too but end 6.1-11.8 and 8.0-11.5 points lower in held-out accuracy than LB at the same budget. A full audit, which the group-level correction reproduces in expectation, stays within about two points of LB in three seeds, while LB uses 28% of its labels, and allocating the same correction by pass count recovers most of the loss (one seed). Training outcomes thus depend not only on the expected update but also on the randomness of its estimate.
• 中文摘要 用可验证奖励的强化学习训练于廉价检查，如单元测试，策略可以学会“游戏”，而更强的验证则成本过高，不能每次部署都运行。我们研究如何在GRPO中使用有限的强验证者预算，将弱强验证者预算与被抓获黑客获得的奖励区分开来。在固定策略状态且优势居中时，预期更新仅依赖于每次部署的边际审计概率和修正;反映被抓破的失败奖励会抵消全黑客组的优势;审计弱优先次数最少的整组（低通过预算审计，LB）可最小化通过计数设计中剩余的优势，当标签值随通过计数下降时，修正固定。我们在实际的LLM训练中测试了该分析：从每次运行的起始状态，该质量在冻结面板上的符号预测了39次运行中的33次是否会被抑制，其中包括16次预注册中的13次。在LeetCode（Wong等，2026）和Countdown-Code（Khalifa等，2026）中，LB将黑客攻击率推向接近零，而在相同惩罚和预算下的随机审计则不会，且使用更多标签。我们还发现更新的随机性影响训练：在Countdown-Code中，两个无偏修正——按项目奖励水平的IPW和一个组级优势修正——也降低了黑客行为，但最终在被保留准确率上比同一预算的LB低6.1-11.8点和8.0-11.5点。完整的审计（组级修正以预期重现）在三个种子中保持在LB约两点以内，而LB使用其标签的28%，通过通过次数分配相同的修正可恢复大部分损失（一个种子）。训练结果因此不仅依赖于预期更新，还依赖于估计值的随机性。

Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

利用路由集成和轨迹归因的多目标强化学习

• Authors: Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Robotics (cs.RO); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.35733
• Pdf link: https://arxiv.org/pdf/2610.35733
• Abstract Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward. Finally, we consider trajectory selection as another approach to mitigate objective conflict, and design a kernel surrogate trajectory attribution algorithm that estimates how feedback signals from a set of trajectories influence policy learning on other trajectories. The kernel can capture nonlinear interactions such as synergy and antagonism between trajectories. We evaluate our approach across diverse environments and benchmarks: routed ensembles improve average performance by 4.3% over baselines for online environments from Meta-World and MO-Gymnasium, and kernel surrogate attribution improves correlation with leave-one-out analysis over existing baselines by 50%. When applied to trajectory selection, kernel-based attribution yields a 27% improvement in reward over baseline attribution methods.
• 中文摘要 多目标强化学习是现代人工智能的一个根本性问题，其应用涵盖了训练后、推理模型和机器人控制。一个核心挑战是学习平衡竞争目标的策略，如安全与对齐中的帮助性与诚实，或机器人学中的速度与能量。最优策略在目标加权组合中可能有显著差异，这种差异的程度关键取决于目标间的冲突程度，使得按偏好策略学习样本效率低下。本文引入了目标-冲突系数，并利用该指标推导出在线环境的近似最优遗憾界限，从而量化错位对学习效率的影响。随后，我们设计了路由式集成策略学习，这是一种维护少量基准策略并动态引导轨迹到潜在奖励最高的策略的算法。最后，我们将轨迹选择视为减少客观冲突的另一种方法，设计了核替代轨迹归因算法，估计一组轨迹的反馈信号如何影响其他轨迹的政策学习。核可以捕捉轨迹间的协同和对抗等非线性交互作用。我们评估了跨多种环境和基准测试的方法：路由集合相比Meta-World和MO-Gymnasium的在线环境基线提升平均性能4.3%，而核代理归因在现有基线上与“留一出”分析的相关性提升了50%。应用于轨迹选择时，基于核的归因相比基线归因方法提高了27%的奖励。

Preference-Guided Cooperation in Multi-Objective Multi-Agent Reinforcement Learning

多目标多代理强化学习中的偏好引导合作

• Authors: Qiyue Xia, J. Michael Herrmann
• Subjects: Subjects: Multiagent Systems (cs.MA); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.35736
• Pdf link: https://arxiv.org/pdf/2610.35736
• Abstract Multi-objective multi-agent reinforcement learning (MOMARL) addresses agent coordination and objective trade-offs simultaneously while optimising overall system performance. Existing work tends to treat multi-agent cooperation and multi-objective optimisation as largely independent components of a learning problem, which amounts to an exorbitant complexity except in trivial cases. We study here whether the structure provided by multiple objectives can itself be exploited to facilitate cooperation, rather than being treated as an additional optimisation challenge. We formulate MOMARL at both the team and individual levels and introduce objective-wise counterfactual credit assignment to evaluate each agent's contribution across the team objectives. We propose Meta-Gradient Informed Multi-Objective Multi-Agent Reinforcement Learning (MIRL), with static (MIRL-S) and dynamic (MIRL-D) variants. By incorporating meta-gradient information, MIRL enables agents to learn how to prioritise objectives individually while collectively optimising the system-level objective, allowing the multi-objective structure to directly guide cooperation. Experiments on a multi-objective multiple particle environment show that our methods improve Pareto-front quality over baselines, with the dynamic variant achieving the strongest performance.
• 中文摘要 多目标多智能体强化学习（MOMARL）同时解决智能体协调和目标权衡，同时优化整体系统性能。现有工作倾向于将多智能体协作和多目标优化视为学习问题的大部分独立组成部分，除极其简单的情况下外，复杂度极高。我们在此研究多目标提供的结构是否可以被利用来促进合作，而非被视为额外的优化挑战。我们在团队和个人层面制定MOMARL，并引入按目标的反事实信用分配，以评估每个智能体在团队目标中的贡献。我们提出了元梯度知情多目标多智能体强化学习（MIRL），具有静态（MIRL-S）和动态（MIRL-D）两种变体。通过整合元梯度信息，MIRL使智能体能够学习如何单独优先排序目标，同时集体优化系统级目标，使多目标结构直接引导合作。在多目标多粒子环境中的实验表明，我们的方法在基线上提升了帕累托前沿的质量，其中动态变体表现最为出色。

SEMA-Auto: Sighted and Efficient Multi-Turn Attack with Autonomy

SEMA-Auto：瞄准且高效的多回合攻击，具备自主性

• Authors: Mingqian Feng, Xiaodong Liu, Weiwei Yang, Chenliang Xu, Jialin Song, Qiyao Ma, Chengquan Guo, Carolyn Buractaon, Christopher White, Jianfeng Gao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.35743
• Pdf link: https://arxiv.org/pdf/2610.35743
• Abstract Existing jailbreak attackers fall into two classes: static planners fix their single- or multi-turn adversarial plans in advance, remaining simple but losing sight of the victim's replies, wasting turns and limiting transferability; interactive generators read each victim reply to generate the next-turn attack but require in-loop evaluators for success signals or hints, incurring substantial costs and inflated performance. Bridging this gap, we propose SEMA-Auto, a sighted and efficient multi-turn attacker with autonomy, interacting with the victim model and terminating the session by itself upon success. To stabilize its behaviors and boost its capabilities, SEMA-Auto is trained in two stages. Synthetic bootstrap tuning installs the notion of attack targets and conforming decisions by fine-tuning the base model with bootstrapped trajectories. Decision-supervised reinforcement learning then optimizes two objectives jointly yet separately, reinforcing the attack with higher jailbreak reward while supervising the decision at each turn against an absolute target. Our method achieves state-of-the-art (SOTA) attack success rates (ASR) across multiple datasets, victims, and judges, outperforming all static and interactive baselines, within almost the fewest turns, demonstrating both effectiveness and efficiency. For example, SEMA-Auto performs a $76.0\%$ ASR@1 within an average of $3.0$ turns against DeepSeek-V4-Flash on AdvBench, while the best static baseline is $54.8\%$ with fixed $7$ turns and the best interactive baseline is $43.8\%$ in $3.7$ turns. Against safety-hardened frontier models, GPT-oss-120B, GPT-5.4, Claude-Haiku-4.5, and GPT-6-Astra, our counter-trained attacker achieves an average $59.6\%$ ASR@1 on HarmBench, $13.8\%$ over prior SOTA. Our code is available at: this https URL
• 中文摘要 现有越狱攻击者分为两类：静态策划器提前确定单回合或多回合对抗计划，保持简单但忽视受害者的回复，浪费回合并限制可转移性;交互式生成器读取每个受害者回复生成下一回合攻击，但需要循环内评估器来获取成功信号或提示，导致成本高昂且性能过高。为弥合这一空白，我们提出SEMA-Auto，一种具备自主性、能与受害者模型交互并在成功时自行终止会话的高效多回合攻击者。为稳定其行为并提升能力，SEMA-Auto 经过两个阶段训练。合成引导调优通过微调基础模型，植入攻击目标和合规决策的概念。决策监督强化学习随后同时又分别优化两个目标，在每个回合对绝对目标的决策进行监督，同时以更高的越狱奖励强化攻击。我们的方法在多个数据集、受害者和法官中实现了最先进的（SOTA）攻击成功率（ASR），在几乎最短的回合内超越所有静态和交互基线，展现了效果和效率。例如，SEMA-Auto在AdvBench上对DeepSeek-V4-Flash在平均3.0美元回合内完成了76.0%$的攻击ASR@1，而固定7美元回合的最佳静态基线为54.8美元，3.7美元回合中最佳交互基线为43.8美元。面对安全防护的前沿模型，如GPT-oss-120B、GPT-5.4、Claude-Haiku-4.5和GPT-6-Astra，我们的反训练攻击者在HarmBench上平均获得59.6美元ASR@1，比之前的SOTA高出13.8美元。我们的代码可在以下网站获取：https URL

FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

重点：长期语言模型代理的主动批评学习

• Authors: Zhengping Jiang, William Jurayj, Liaoyaqi Wang, Chunsheng Zuo, Anqi Liu, Benjamin Van Durme
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.35801
• Pdf link: https://arxiv.org/pdf/2610.35801
• Abstract Sparse outcome rewards make it difficult for long-horizon language-model agents to assign credit to earlier actions. Learned critics provide value estimates for intermediate states, but these estimates can be inaccurate, while estimating intermediate values with repetitive continuation samples from points within a trajectory incurs substantial costs. To mitigate these issues, we introduce FOCUS, an RL method that uses selective branching to help the critic network learn its value estimator. The rollout controller facilitates active learning for the critic by sampling continuations from states that are expected to most reduce the critic's estimation error. These refined value estimates then score the segment between consecutive states with a TD($\lambda$) advantage, while the sampled evidence supplies training targets for the critic. FOCUS supports long-horizon agentic reinforcement learning through a modular context-management interface that decouples context management from rollout allocation. Our experiments show that using compaction-induced segment boundaries for value estimation and credit assignment, FOCUS outperforms GRPO and VinePPO with matched per-step generation budgets on legal reasoning, deep search, and code localization benchmarks, improving held-out accuracy at Qwen3.5-9B scale by 7.2 points over the strongest baseline on BrowseComp-Plus (55.4 vs. 48.2) and by 4.1 points on DeonticBench (56.5 vs. 52.4).
• 中文摘要 结果奖励稀疏使得长期视野语言模型代理难以将归功于早期行动。有经验的批评者会为中间状态提供价值估计，但这些估计可能不准确，而从轨迹内的点重复连续样本估计中间值则会产生巨大成本。为缓解这些问题，我们引入了FOCUS方法，这是一种通过选择性分支帮助批评者网络学习其价值估计器的方法。推广控制器通过从预期能最大程度减少批评者估计误差的状态中抽样延续，促进批评者的主动学习。这些精细化的价值估计随后对连续状态之间的段进行TD（$\lambda$）优势评分，而抽样证据则为批评者提供训练目标。FOCUS 通过模块化上下文管理接口支持长期代理强化学习，将上下文管理与推广分配解耦。我们的实验显示，利用压缩诱导的片段边界进行价值估计和功劳分配，FOCUS 在法律推理、深度搜索和代码本地化基准测试上的每步生成预算匹配，在 Qwen3.5-9B 尺度上比 BrowseComp-Plus 最强基线提升 7.2 分（55.4 对 48.2），在 DeonticBench 上提升 4.1 分（56.5 对 52.4）。

World Model for Energy Efficiency Maximization in Dynamic Metasurface Antennas

动态变曲面天线能效最大化的世界模型

• Authors: Abdolrasoul Sakhaei Gharagezlou, Mehdi Monemi, Matti Latva-aho, Mehdi Rasti
• Subjects: Subjects: Information Theory (cs.IT); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.35813
• Pdf link: https://arxiv.org/pdf/2610.35813
• Abstract Dynamic metasurface antennas (DMAs) have emerged as a promising technology for realizing large-scale antenna arrays in next-generation wireless networks owing to their ability to enable dense and energy-efficient antenna architectures. However, jointly optimizing digital beamforming and DMA element weights under time-varying channel conditions remains challenging due to their coupled effects on the received signal and interference powers. We formulate this problem to maximize time-averaged energy efficiency (EE) under time-varying channel conditions, subject to transmit power constraints, Lorentzian phase limitations, and minimum rate requirements. To enable long-term planning and improve sample efficiency compared with model-free reinforcement learning, we develop a world model framework that learns latent system dynamics from interactions with the environment and jointly optimizes digital beamforming and DMA element weights through imagined trajectories. Furthermore, we incorporate feasibility-preserving action mappings and safety cost signals to ensure constraint satisfaction during policy learning. Simulation results demonstrate that the proposed framework improves the time-averaged EE by approximately 20% compared with the deep Q-network baseline, while achieving higher sample efficiency and more reliable constraint satisfaction.
• 中文摘要 动态元曲面天线（DMA）因其能够实现密集且节能的天线架构，已成为下一代无线网络中实现大规模天线阵列的有前景技术。然而，由于数字波束成形和DMA元素权重对接收信号和干扰功率的耦合效应，在时变信道条件下共同优化仍然具有挑战性。我们提出该问题，旨在最大化在时间变化信道条件下的时间平均能效（EE），同时受发射功率约束、洛伦兹相位限制和最低速率要求。为实现长期规划并提升样本效率，我们开发了一个世界模型框架，从与环境交互中学习潜在系统动力学，并通过想象轨迹联合优化数字波束形成和DMA元素权重。此外，我们结合了可行性保持的动作映射和安全成本信号，以确保政策学习期间约束满足。模拟结果表明，所提出的框架相比深度Q网络基线，平均时间平均的EE提高了约20%，同时实现了更高的样本效率和更可靠的约束满足。

Keyword: diffusion policy

HA-DiPO: Diffusion Policy Optimization for Heterogeneous Cooperative Transport

HA-DiPO：异构协同运输的扩散策略优化

• Authors: Xuebin Li, Jinpeng Xiao, Zong Chen, Pierluigi Arpenti, Bruno Siciliano, Jiebao Sun, Yiqun Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.33584
• Pdf link: https://arxiv.org/pdf/2610.33584
• Abstract Cooperative transport with heterogeneous legged robots requires coordinating different motion capabilities while supporting an unfixed shared payload. We present HA-DiPO (Heterogeneous-Agent Diffusion Policy Optimization), a hierarchical multi-agent reinforcement learning framework under centralized training with decentralized execution (CTDE). The framework decouples team-level coordination from embodiment-specific motion control: at the high level, a centralized critic uses team-value estimates to jointly train independent coordination policies, aligning robot-specific action responses with a shared transport objective; at the low level, embodiment-specific robust motion policies track the coordination commands. The high-level diffusion policies offer more flexible action distributions than single-Gaussian parameterizations, allowing potentially multimodal coordination responses to be represented. The coordination policies learn from scratch through reward-driven simulator interaction without demonstrations. At execution, each robot generates coordination commands using only local robot and payload observations, without partner-state inputs or inter-robot messages. Simulation comparisons show that HA-DiPO achieves higher final mean returns and velocity-tracking success rates, as well as lower velocity errors, than the evaluated baselines. Real-world experiments further demonstrate quadrupedal-wheeled-legged and human-robot cooperative transport.
• 中文摘要 与异构腿部机器人的协作传输需要协调不同的运动能力，同时支持非固定共享有效载荷。我们介绍HA-DiPO（异构代理扩散策略优化），这是一个基于集中训练与去中心化执行（CTDE）的分层多智能体强化学习框架。该框架将团队级协调与具身特定运动控制解耦：在高层面，中心化批评者利用团队价值估计联合训练独立协调策略，将机器人特定动作响应与共享传输目标对齐;在低层，具象特定稳健运动策略跟踪协调指令。高层扩散策略提供比单高斯参数化更灵活的动作分布，允许表示潜在的多模态协调响应。协调策略通过奖励驱动的模拟器交互从零学习，无需演示。执行时，每个机器人仅利用本地机器人和有效载荷观测生成协调指令，不涉及伙伴状态输入或机器人间消息。模拟比较显示，HA-DiPO在最终平均回报和速度跟踪成功率方面均高于评估基线，速度误差更低。真实实验进一步展示了四足轮式和人机协作运输。

S$^2$-RL: Sample-Set Dual Reinforcement Learning for Generative Semantic Segmentation Dataset Distillation

S$^2$-RL：生成语义切割数据集蒸馏的样本集双重强化学习

• Authors: Haoyu Wang, Fei Zhou, Qingqing Qiu, Lei Zhang, Wei Wei, Chen Ding
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.35491
• Pdf link: https://arxiv.org/pdf/2610.35491
• Abstract While dataset distillation has witnessed significant advances in image classification, its extension to dense prediction tasks such as semantic segmentation is plagued by two core bottlenecks: bi-level optimization-based approaches suffer from prohibitive computational costs stemming from pixel-wise gradient unrolling, whereas proxy-based generative methods merely infuse visual textures into fixed semantic masks derived from real data, thus failing to condense rich semantic knowledge into a compact set of informative, novel scene generations. To address both bottlenecks simultaneously, we propose a novel Sample-Set Reinforcement Learning (S$^2$-RL) framework for generative semantic segmentation dataset distillation (SSDD). Specifically, S$^2$-RL formulates SSDD as a diffusion model based text-to-image generation paradigm, enabling flexible generation of images with informative, novel semantic distributions via a concise text prompt. Furthermore, we fine-tune a diffusion policy via Group Relative Policy Optimization (GRPO), leveraging a sample-set dual reward paradigm: for individual samples, it enforces semantic alignment with input text prompts and intra-class feature diversity; for sample sets, it maximizes inter-sample semantic distribution diversity by solving a contextual multi-armed bandit problem. These design choices enable S$^2$-RL to distill large-scale semantic segmentation datasets into a compact set of generated samples that encapsulate diverse, informative semantic knowledge without the need of pixel-wise gradient unrolling. Extensive experiments on the ADE20K and COCO datasets demonstrate that S$^2$-RL achieves substantial and consistent improvements over state-of-the-art baselines, establishing a new benchmark for SSDD.
• 中文摘要 尽管数据集蒸馏在图像分类方面取得了显著进展，但其扩展到如语义分割等密集预测任务时，仍面临两个核心瓶颈：基于双层优化的方法因像素级梯度展开而导致的计算成本过高，而基于代理的生成方法仅将视觉纹理注入来自真实数据的固定语义掩体中，未能将丰富的语义知识浓缩成紧凑且富有信息量的新场景生成。为同时解决这两个瓶颈，我们提出了一种新颖的样本集强化学习（S$^2$-RL）生成语义分割数据集蒸馏（SSDD）框架。具体来说，S$^2$-RL将SSDD表述为基于扩散模型的文本到图像生成范式，通过简洁的文本提示实现灵活生成具有信息量且新颖语义分布的图像。此外，我们通过群相对策略优化（GRPO）微调扩散策略，利用样本集双重奖励范式：对单个样本，通过输入文本提示和类内特征多样性强制语义对齐;对于样本集，通过解决上下文多臂强盗问题，最大化样本间语义分布多样性。这些设计选择使S$^2$-RL能够将大规模语义分割数据集提炼成紧凑的生成样本集，封装多样且富有信息量的语义知识，无需像素梯度展开。在ADE20K和COCO数据集上的大量实验表明，S$^2$-RL相较于最先进基线实现了显著且一致的改进，树立了SSDD的新基准。

Learning Force-Reactive Humanoid Loco-Manipulation

学习原力反应类人机车操控

• Authors: Sihang Li, Xiaofeng Guo, Jinkun Cao, Charlie Cheng, Ruihan Yang, Chen Feng, Guanya Shi, Haozhi Qi
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.35862
• Pdf link: https://arxiv.org/pdf/2610.35862
• Abstract Many everyday tasks require controlling contact force as well as motion. On a humanoid, this force must often be coordinated across both hands and the whole body, and cannot be specified by kinematic trajectories alone. Therefore, we present FLORE, a framework for learning force-reactive humanoid loco-manipulation. We collect robot-free demonstrations using two force-sensing UMI-style handheld grippers for hand motion and contact wrenches, and wearable trackers for body motion. A force-reactive diffusion policy takes vision, proprioception, and measured wrenches as input, then predicts actions and per-hand force commands. A hybrid compliance whole-body controller with force tracking executes these commands while maintaining balance: an integral of the force error measured by real force/torque (F/T) sensors lets each hand track force accurately along the commanded direction and remain compliant along the others. In simulation, the controller tracks per-hand force commands of varied magnitude and direction, with little loss in whole-body motion accuracy. Across three contact-rich tasks on the robot, the force-reactive policy succeeds in 80% of rollouts, compared with 26.7% for a position-only policy trained on the same demonstrations and executed by the same controller. For more details, please visit our website: this https URL.
• 中文摘要 许多日常任务需要控制接触力和运动。在类人生物身上，这种力通常需要双手和全身协调，不能仅靠运动轨迹来确定。因此，我们提出了FLORE，一种学习力反应类人机车操控的框架。我们收集无机器人演示，使用两个力感应UMI式手持夹具进行手部运动和接触扳手，以及可穿戴追踪器进行身体运动。力反应扩散策略将视觉、本体感觉和测量扳手作为输入，然后预测动作和每手力指令。带有力追踪功能的混合顺应全身控制器执行这些指令，同时保持平衡：通过真实力/扭矩（F/T）传感器测量的力误差积分，使每只手能准确沿指令方向追踪力，并保持对其他手的顺应。在模拟中，控制器跟踪不同强度和方向的每手力指令，几乎不损全身运动准确性。在机器人三项接触丰富的任务中，力被动策略成功率为80%，而同一控制器执行的仅位置策略成功率为26.7%。欲了解更多详情，请访问我们的网站：https URL。