生成时间: 2026-10-19 23:21:27 (UTC+8); Arxiv 发布时间: 2026-10-19 20:00 EDT (2026-10-20 08:00 UTC+8)

今天共有 41 篇相关文章

Keyword: reinforcement learning

MedSonor: Learning to Use Clinical Audio in Medical LLMs

MedSonor：学习在医疗大型语言模型中使用临床音频

• Authors: Geyang Yu, Junying Chen, Benyou Wang
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2610.22405
• Pdf link: https://arxiv.org/pdf/2610.22405
• Abstract Clinical audio conveys medical information that transcripts may not preserve, but using it in medical responses requires interpreting it alongside clinical requests and dialogue context. Existing medical audio supervision largely targets classification or recognition, while general audio-language models lack specialized medical knowledge and direct feedback for open-ended responses. We present MedSonor, a medical dialogue model integrating medical knowledge, spoken dialogue, and clinical audio understanding. We pair authentic medical audio with clinical requests and combine these audio-integrated dialogues with collected medical audio and synthesized spoken dialogues for supervised fine-tuning, followed by two stages of rubric-guided reinforcement learning. We introduce MedVoice-Dial and MedVoice?AID to evaluate responses within supplied multi-turn medical contexts and to requests involving clinical audio, respectively. Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference evaluation, while broadly retaining general speech capabilities.
• 中文摘要 临床音频传达了可能无法保存的医疗信息，但在医疗响应中使用时，需要与临床请求和对话上下文并行解读。现有的医疗音频监督主要针对分类或识别，而一般音频语言模型缺乏专业医学知识和开放式回答的直接反馈。我们介绍MedSonor，一种整合医学知识、口语对话和临床音频理解的医疗对话模型。我们将真实的医疗音频与临床请求结合，并将这些音频集成对话与收集的医疗音频和合成的口语对话结合起来，进行监督式微调，随后进行两个阶段的评分标准引导强化学习。我们引入了MedVoice-Dial和MedVoice？分别用于评估在提供的多回合医疗情境中的反应，以及涉及临床音频的请求。在七项医学基准中，MedSonor优于评估的通用开放权重音频语言基线，并在医生整体偏好评估中排名最高，同时大致保留了一般语音能力。

Lean4AgenticRL: Advancing Agentic RL with Verifiable Multi-level Process Rewards

Lean4AgenticRL：推进可验证多层次流程奖励的智能强化学习

• Authors: Ruida Wang, Gerald Friedland, Tong Zhang, Chris Kong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.22448
• Pdf link: https://arxiv.org/pdf/2610.22448
• Abstract Enhancing the agentic capabilities of Large Language Models (LLMs) to tackle complex software engineering (SWE) and general terminal-usage problems is a pressing challenge. Current state-of-the-art LLMs often go through large-scale agentic Reinforcement Learning (RL). However, they mostly rely on high-level outcome rewards, with minimal consideration of low-level toolcall and mid-level workflow-stage rewards, which makes the training less effective. To overcome this limitation, we propose Lean4AgenticRL, which, to the best of our knowledge, is the first framework that grounds multi-level process rewards in the unified Lean4 formalization schema for open-ended agentic RL. Lean4AgenticRL first builds a problem-specific workflow system that balances the agent's own capability and controllability during execution, laying the foundation for further verification. Subsequently, our workflow formalization and multi-level reward annotation methods enable agent behavior verification within a general harness system. It provides multi-level rewards to support more effective agentic RL with the workflow-stage-normalized GRPO algorithm. Using Lean4AgenticRL, we train LAR-9B/27B, a family of models with enhanced agentic capability. Extensive experiments indicate that our model outperforms the leading model of similar size. LAR-27B achieves 77.3% on TerminalBench-2.1. On average, our models outperform the base model by 9.89% and surpass the second-best model of comparable size we know by 5.07%. These results indicate the potential of integrating formal methods in enhancing open-ended agentic RL.
• 中文摘要 提升大型语言模型（LLMs）的代理能力以应对复杂的软件工程（SWE）和一般终端使用问题，是一项紧迫的挑战。当前最先进的大型语言模型通常会经历大规模代理强化学习（RL）。然而，它们大多依赖于高层次的结果奖励，几乎不考虑低层工具调用和中层工作流阶段奖励，这使得训练效果较差。为克服这一限制，我们提出了Lean4AgenticRL，据我们所知，这是首个将多层次流程奖励建立在统一的Lean4形式化模式中、面向开放式代理强化学习的框架。Lean4AgenticRL首先构建了一个针对问题的工作流程系统，在执行过程中平衡代理自身的能力与可控性，为进一步验证奠定基础。随后，我们的工作流形式化和多级奖励注释方法使得在通用的约束系统内验证代理行为成为可能。它提供多层级奖励，支持更高效的代理强化学习，采用工作流阶段归一化GRPO算法。利用Lean4AgenticRL，我们训练了LAR-9B/27B，这是一系列具备增强代理能力的模型。大量实验表明，我们的模型优于同规模的领先模型。LAR-27B在TerminalBench-2.1上达到77.3%。平均来看，我们的模型比基础模型高出9.89%，并且比我们已知的同规模第二优模型高出5.07%。这些结果表明整合形式方法在提升开放式智能强化学习方面具有潜力。

ProactGen: Learning Proactive Clarification for Video Generation

ProactGen：学习主动澄清视频生成

• Authors: Zhixiao Wang, Yang Chen, Yifan Xu, Chenbin Li, Yue Chang, Kai Long, Fengming Dong, Limin Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.22608
• Pdf link: https://arxiv.org/pdf/2610.22608
• Abstract Recent advances in video generation let users create high-quality videos from natural language descriptions, yet obtaining results that match user intent remains time-consuming: individual generations are computationally expensive, and underspecified prompts force repeated cycles of generation, inspection, and revision. Existing video prompt optimization methods improve generation through automatic rewriting, but cannot supply the ideas and preferences that were not in the initial description. We propose ProactGen, a video generation framework that learns proactive clarification, eliciting user intent through questions grounded in generated video drafts. Specifically, we train a vision-language model as a questioning policy with reinforcement learning: observing the initial prompt and a video draft, it learns to ask the questions most valuable to downstream generation, rewarded by the alignment between the final generated video and a target reference. We further find that effective questioning does not require drafts of final-output quality, and exploit this property by generating drafts with fewer denoising steps, keeping the added cost of clarification low. Experiments show that ProactGen improves target alignment over direct generation by at least 1.8x the gain of the strongest baselines. Moreover, a single round of ProactGen already exceeds the alignment that the strongest baseline reaches after four refinement rounds, in 29% of its time.
• 中文摘要 视频生成的最新进展使用户能够从自然语言描述中创建高质量视频，但获得与用户意图匹配的结果仍然耗时：单个生成计算成本高，且未明确的提示要求反复生成、检查和修改。现有视频提示优化方法通过自动重写提升生成效果，但无法提供初始描述中未包含的创意和偏好。我们提出了ProactGen，一个视频生成框架，它通过基于生成视频草稿的问题，学习主动澄清，激发用户意图。具体来说，我们将视觉语言模型训练为带有强化学习的提问策略：观察初始提示和视频草稿，它学会提出对后续生成最有价值的问题，并通过最终生成视频与目标参考的对齐获得回报。我们还发现，有效的提问不需要达到最终输出质量的草稿，并通过生成更少的去噪步骤来利用这一特性，降低澄清带来的额外成本。实验显示，ProactGen在直接生成过程中，目标比对提升至少是最强基线的1.8倍。此外，单轮ProactGen在29%的时间内已超过最强基线在四轮精炼后达到的比对。

Diverse Reasoning Paths Matter: Symbolic Equivalence Augmentation for Enhancing RL Exploration

多样的推理路径很重要：符号等价增强以增强强化学习探索

• Authors: Xiaoou Liu, Tiejin Chen, Hanyang Chen, Yuan Ling, Hua Wei
• Subjects: Subjects: Symbolic Computation (cs.SC); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22665
• Pdf link: https://arxiv.org/pdf/2610.22665
• Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for improving the reasoning capabilities of LLMs. However, the role of supervised fine-tuning (SFT) data in enabling effective downstream RL remains underexplored. In this paper, we investigate a key hypothesis: the diversity of reasoning steps in chain-of-thought (CoT) during SFT is a critical enabler of RLVR performance. To test this hypothesis, we introduce EquivAug, a data augmentation framework that uses symbolic equivalence transformations to generate diverse yet provably correct reasoning paths. By converting CoT solutions into formal symbolic representations, applying equivalence transformations, and reconstructing natural-language explanations, EquivAug produces training data that differs in reasoning steps rather than surface-level lexical variation. Experiments on GSM8K and FOLIO across model scales demonstrate that EquivAug consistently achieves the highest reasoning path diversity after SFT and the strongest accuracy after GRPO. Our results establish that CoT diversity during SFT is positively correlated with downstream RLVR performance.
• 中文摘要 带有可验证奖励的强化学习（RLVR）已成为提升大型语言模型推理能力的强大范式。然而，监督式微调（SFT）数据在实现有效下游强化学习中的作用仍未被充分探讨。本文探讨了一个关键假说：SFT过程中思维链（CoT）中推理步骤的多样性是RLVR性能的关键推动力。为验证这一假设，我们引入了EquivAug，一种数据增强框架，利用符号等价变换生成多样但可证明正确的推理路径。通过将CoT解决方案转换为形式符号表示，应用等价变换并重建自然语言解释，EquivAug生成的训练数据在推理步骤上有所不同，而非表层词汇变化。在GSM8K和FOLIO上跨模型尺度的实验表明，EquivAug在SFT后始终获得最高的推理路径多样性，GRPO后准确率最高。我们的结果证明，SFT期间CoT多样性与下游RLVR表现呈正相关。

Gossip Agents: Reputation through Opinion Dynamics over Evolving Networks

八卦代理：通过舆论动态建立声誉，超越不断演变的网络

• Authors: Yutong Wu, Arlei Silva
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.22674
• Pdf link: https://arxiv.org/pdf/2610.22674
• Abstract Sustained cooperation among self-interested learning agents is essential in many shared environments. Reputation systems are among the best-studied mechanisms for promoting such cooperation. Existing multi-agent reinforcement learning approaches, however, typically assume that reputation is either globally available to all agents or derived solely from an agent's own observations, leaving largely unexplored how reputational information propagates between agents. We study gossip, the exchange of information about third parties, as a mechanism for reputation diffusion over dynamic communication networks. Agents form private opinions from limited observations and exchange them with their neighbors following DeGroot opinion dynamics, while the communication network changes as agents interact and move. A tunable broadcast radius controls the reach of information, spanning private reputation, local gossip, and globally shared reputation. We study this framework in two (n)-player social dilemmas with distinct network dynamics: a Prisoner's Dilemma with randomly resampled interaction groups and Harvest, where communication neighborhoods emerge from agents' movements. Our results show that the reach of reputational information shapes cooperation in both environments. Moreover, in Harvest, reputation diffusion increases environmental rewards and leads to more peaceful and sustainable behavior.
• 中文摘要 在许多共享环境中，自利学习智能体之间的持续合作至关重要。声誉系统是促进此类合作研究最深入的机制之一。然而，现有的多智能体强化学习方法通常假设声誉对所有智能体均可得，或仅源自智能体自身的观察，因此声誉信息如何在智能体间传播方面大多未被探讨。我们研究关于第三方的信息交换（gossip）作为声誉在动态通信网络中传播的机制。智能体通过有限的观察形成私人观点，并按照德格鲁特意见动态与邻居交换，而通信网络则随着智能体的互动和移动而变化。可调节的广播半径控制信息的覆盖范围，涵盖私人声誉、本地闲话和全球共享声誉。我们在两种具有不同网络动态的\（n\）玩家社会困境中研究该框架：一个是随机重抽样交互组的囚徒困境，另一个是Harvest，其中通信邻域由代理的移动形成。我们的结果表明，声誉信息的覆盖范围在两种环境中都塑造了合作。此外，在Harvest中，声誉扩散增加了环境奖励，并带来更和平和可持续的行为。

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

AlphaBet：面向大型语言模型事件预测代理的强化学习

• Authors: Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, Chao Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22684
• Pdf link: https://arxiv.org/pdf/2610.22684
• Abstract Forecasting an unresolved event requires finding relevant evidence and deciding how strongly it supports each outcome. Yet outcome-based training of Large Language Model (LLM) forecasters often begins with preassembled evidence at a single prediction date, leaving evidence acquisition outside the learning loop. We introduce AlphaBet, an agentic reinforcement learning (RL) framework that trains a single LLM policy to search, reason, and report probabilities across multiple pre-resolution dates. Each dated forecast is scored by a proper scoring rule against the eventual outcome, allowing one resolved event to supervise decisions under different information states. We decompose forecasting regret into an evidence-acquisition gap and a probability-reporting gap, and show that Group Relative Policy Optimization (GRPO)'s standard-deviation normalization can distort the reporting target under multi-class proper scores. On GLM-4.5-Air, AlphaBet reduces the Brier score from 0.731 to 0.626 on Forecast-Dojo and from 0.612 to 0.528 on FutureX. Controlled ablations attribute these gains to multi-date training and show that they come from improvements in both information acquisition and probability reporting. Our empirical results validate that GRPO's std normalization degrades proper-scoring-rule performance while leaving top-outcome accuracy largely unchanged.
• 中文摘要 预测未解决事件需要找到相关证据并决定其支持每个结果的强度。然而，基于结果的大型语言模型（LLM）预测者训练通常从单一预测日期的预组证据开始，导致证据获取过程脱离学习循环。我们介绍了AlphaBet，一种代理强化学习（RL）框架，训练单一LLM策略以搜索、推理和报告多个预解决日期的概率。每个有日期的预测都通过适当的评分规则对最终结果进行评分，使一个已解决事件能够在不同信息状态下监督决策。我们将预测遗憾分解为证据获取差距和概率报告差距，并展示了Group Relative Policy Optimization（GRPO）的标准差归一化在多类适当分数下会扭曲报告目标。在GLM-4.5-Air上，AlphaBet将Forecast-Dojo上的Brier分数从0.731降至0.626，FutureX上的分数从0.612降至0.528。受控消融将这些提升归因于多日期训练，并显示其来自信息获取和概率报告的改进。我们的实证结果验证了GRPO的标准化规范降低了正确评分规则的性能，而最高结果的准确性基本保持不变。

Filter-Augmented Reinforcement Learning for Battery Fast Charging with State of Health and Plating-Aware Reward Design

滤波增强强化学习用于电池快速充电，结合健康状态和装甲感知奖励设计

• Authors: Meng Yuan, Yang Li, Changfu Zou
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.22787
• Pdf link: https://arxiv.org/pdf/2610.22787
• Abstract Fast charging of lithium-ion batteries is crucial for electric vehicle adoption, yet it increases safety risks and accelerates battery ageing. To address this, we propose a filter-augmented reinforcement learning (RL) framework for lifelong health-aware fast charging, based on the twin-delayed deep deterministic policy gradient (TD3) algorithm. The RL agent is guided by a reward function that balances charging speed and battery longevity through the incorporation of state of health (SoH) information and an SoH-dependent voltage penalty term derived from a constant--current constant--overpotential charging strategy. Since onboard SoH estimates are inevitably corrupted by noise, two filtering strategies are integrated: an exponential moving average (EMA) filter when noise statistics are unknown, and a Kalman filter when they are available, with the filtered estimate embedded in both the agent's state and reward computation. The RL policy is trained using data from high-fidelity life-cycle simulations. The evaluation uses a published electrochemical and degradation parameterisation for a commercial LG M50 cell. Results show that the proposed method achieves up to 21.4\% more equivalent full cycles (EFCs) than conventional constant current--constant voltage (CC--CV) charging under 2\% SoH estimation noise, and maintains this lifetime advantage even under 5\% noise, while sustaining comparable charging speed.
• 中文摘要 锂离子电池的快速充电对电动汽车的普及至关重要，但它增加了安全风险并加速电池老化。为此，我们提出了一种基于双延迟深度确定性政策梯度（TD3）算法的滤波增强强化学习（RL）框架，用于终身健康意识快速充电。RL代理由一个奖励函数引导，该函数通过纳入健康状态（SoH）信息和基于SoH的电压惩罚项（源自恒定-电流常数-过电位充电策略）来平衡充电速度和电池寿命。由于车载SoH估计必然会被噪声破坏，因此整合了两种滤波策略：当噪声统计未知时采用指数移动平均（EMA）滤波器，当噪声统计量可用时使用卡尔曼滤波器，滤波后的估计数据嵌入智能体状态和奖励计算中。强化学习策略基于高精度生命周期模拟数据进行训练。评估采用了LG M50商用电池的已发表电化学和降解参数化。结果显示，该方法在2%的SoH估计噪声下，在保持相当充电速度的同时，比传统恒流恒压（CC-CV）充电多达21.4/%的等效全周期（EFC），并且即使在5%噪声以下也能保持这一寿命优势，同时保持相当的充电速度。

Self-Evolving Coding Agents via Harness-Policy Optimization

通过约束策略优化实现自我演化的编码代理

• Authors: Zhepei Wei, Wenlin Yao, Baolin Peng, Danqing Wang, Qianhui Wu, Hao Cheng, Yu Meng, Jianfeng Gao
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22791
• Pdf link: https://arxiv.org/pdf/2610.22791
• Abstract Large language model (LLM) agents have shown promising potential in coding tasks, and their performance is jointly determined by the policy model and the harness that mediates their interaction with the environment. Indeed, harness and policy are mutually reinforcing and evolving only one component is suboptimal---a harness tuned for one checkpoint may not target the failure modes of an updated policy, while policy learning remains limited to the trajectories elicited by a fixed harness. In light of this, we propose Self-HPO, a recursive self-improving framework for coding agents that alternates harness and policy optimization, using the evolving policy as its own harness proposer. Specifically, in each round, the current policy first analyzes its own failures and proposes new harness candidates (H-step). The policy is then updated through reinforcement learning (RL) under the selected harness (P-step), and hard tasks identified during training form the next harness search set, keeping harness optimization targeted to the policy's evolving failure modes. Across three coding benchmarks, Self-HPO consistently outperforms harness-only and policy-only baselines, improving Qwen3.5-35B-A3B's average resolve rate from 57.8% to 64.8%, with gains generalizing to out-of-distribution settings. Further analysis shows that both optimized components retain gains even when paired with their base counterparts, while their joint gain exceeds the sum of their individual gains. Self-HPO also improves reliability (Pass@1) and expands the capability frontier (Pass@k) over the base agent. Additionally, we show that with a stronger external model (e.g., Opus 4.8) as the harness proposer, the same framework yields a further performance boost.
• 中文摘要 大型语言模型（LLM）代理在编码任务中展现出有前景的潜力，其性能由策略模型与其与环境交互的约束共同决定。事实上，约束与策略相互强化和演进，只有一个组件不够优---针对单一检查点的约束机制可能无法针对更新策略的失败模式，而策略学习仍限于固定约束机制引发的轨迹。鉴于此，我们提出Self-HPO，一种递归自我改进的编码代理框架，交替使用利用和策略优化，使用演化策略作为自身的约束提案。具体来说，在每一轮中，当前策略首先分析自身的失败，并提出新的约束候选方案（H步）。随后，策略通过强化学习（RL）在所选束（P步）下更新，训练中识别的难任务则形成下一个束带搜索集，使束体优化针对策略不断演变的失败模式。在三个编码基准测试中，Self-HPO持续优于仅限束和仅策略基线，将Qwen3.5-35B-A3B的平均解析率从57.8%提升至64.8%，且收益可推广至分布外环境。进一步分析显示，两个优化组件即使与基础组件配对，仍保留收益，而它们的联合收益超过各自收益之和。Self-HPO还提升了可靠性（Pass@1），并扩展了基础代理的能力前沿（Pass@k）。此外，我们表明，如果采用更强的外部模型（例如作品4.8）作为束束提案者，同一框架还能带来进一步的性能提升。

Reasoning Reranking with Semantic IDs as the Sole Item Representation

以语义ID作为唯一项目表示的重新排序的推理

• Authors: Zhenwei Tang, Jintao Liu, Di Bai, Xidong Wu, Xinwu Cheng
• Subjects: Subjects: Information Retrieval (cs.IR); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.22797
• Pdf link: https://arxiv.org/pdf/2610.22797
• Abstract Reasoning rerankers write a rationale before ranking a candidate slate, and existing systems read item metadata to do so, even though generative recommenders increasingly represent items as compact Semantic IDs (SIDs). Ranking from SIDs alone has two appeals: SID prompts are far shorter than the matched item text, and although item codes are fixed from content, the backbone learns their token embeddings in part from user-item interaction sequences, so the tokens can carry behavioral signal that item text lacks. That training targets next-item generation, and whether it supports ranking a supplied slate from SIDs alone is untested. We build ReRankID, a controlled testbed in which models reason and rank from a history and slate shown only as SIDs, varying item rendering, history length, and slate size one at a time. On this testbed, such models often copy the input order instead of reranking, and readable metadata does not repair the failure. We compare supervised fine-tuning, rejection-sampled self-training, on-policy distillation, and verifier-based reinforcement learning as post-training methods and find that they elicit the missing behavior. The strongest post-trained model, the 8B model trained with a graded ranking reward, gives the best SID-only ranking and matches strong text-reading LLMs whose prompts are about 20x longer. It ranks above random with no history at all, where a caption-reading reference does not, and it keeps its gains at long histories, where text prompts pass 50k tokens and the caption-reading reference stops improving. In a paired rationale audit, this model argues trade-offs better than that reference but describes items less accurately. The common post-training methods we test thus elicit ranking from SIDs but not faithful item descriptions, which calls for post-training that improves the reasoning itself, not only the ranking it produces.
• 中文摘要 推理重排序器在排序候选列表前会先写理由，现有系统则读取项目元数据来进行排序，尽管生成推荐器越来越多地将项目表示为紧凑语义ID（SID）。仅从SID排序有两个优点：SID提示远短于匹配的项目文本;虽然项目代码通过内容固定，骨干部分通过用户与项目交互序列学习其令牌嵌入，因此令牌可以携带条目文本所缺乏的行为信号。该训练针对下一代项目生成，是否支持仅凭SID对提供列表进行排名尚未测试。我们构建了ReRankID，一个受控测试平台，模型根据历史和仅显示为SID的平板进行推理和排序，逐一调整项目渲染、历史长度和平板大小。在这个测试平台中，这类模型常常复制输入顺序而非重新排序，且可读元数据无法修复失败。我们比较了监督式微调、拒绝抽样自训练、策略上提炼和基于验证者的强化学习作为后训练方法，发现它们能引发缺失的行为。最强的后训练模型是8B模型，训练时带有分级排名奖励，它提供了最佳的仅SID排名，且匹配了提示长度约20倍的强文本阅读大型语言模型。它在完全没有历史的情况下排名高于随机，而在阅读标题引用没有历史记录的情况下，它在长历史阶段保持收益，即文本提示传递5万个代币，且阅读引用停止改进。在配对理据审计中，该模型论证了权衡比参考更准确，但描述项目的准确性较低。因此，我们测试的常见训练后方法只能从SID中引出排名，但不会从忠实的项目描述中获得排名，这需要后训练以提升推理本身，而不仅仅是排序。

LLM-Guided Dual-Stage Latency Optimization for Closed-Loop 6G V2I Digital Twins

闭环6G V2I数字孪生的LLM引导双级延迟优化

• Authors: Liu Cao, Peide Li, Yuan Yang, Zhaoyu Liu, Lyutianyang Zhang, Jinke Ren, Shuguang Cui
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.22848
• Pdf link: https://arxiv.org/pdf/2610.22848
• Abstract Wireless Digital Twins are emerging as a key enabler for predictive network control in 6G Vehicle-to-Infrastructure (V2I) systems. However, closed-loop V2I DTs face two dual-stage communication bottlenecks arising from model parameter exchange for V2I DT reconstruction and multi-service V2I traffic delivery after each reconstruction. This paper proposes a closed-loop dual-stage End-to-End (E2E) latency optimization framework that integrates Compressed Personalized Federated Learning with a unified Large-Language-Model-guided Multi-Agent Reinforcement Learning approach. A CPFL-enabled decision-oriented V2I DT is first constructed by jointly predicting Path Loss (PL) and Quality-of-Experience (QoE)-oriented Key Performance Indicators (KPIs). Then the framework comprises two temporally coupled but latency-wise distinct optimization stages: Stage I minimizes the joint Uplink (UL) and Downlink (DL) E2E transmission latency of compressed shared backbone's parameters, and Stage II minimizes the service-aware E2E latency of multi-service V2I traffic. As the two stages share aligned agent roles and action space, we develop a unified Semantic-Aware Guidance-Enhanced Multi-Agent Proximal Policy Optimization framework, in which the LLM acts as a training-time semantic planner that provides high-level traffic splitting and power allocation guidance to reduce inefficient exploration, while the MARL agents execute feasible low-level actions. Simulation results demonstrate that the proposed framework enables efficient V2I DT reconstruction and significantly reduces both E2E latency for DT reconstruction and multi-service V2I traffic. The proposed SAGE-MAPPO supports cross-stage policy reuse by sequentially solving the two stages within each DT update cycle, which accelerates convergence relative to the existing baselines and facilitates timely DT updates in dynamic V2I scenarios.
• 中文摘要 无线数字孪生正作为6G车辆到基础设施（V2I）系统中预测性网络控制的关键推动力。然而，闭环V2I数字孪生面临两个双阶段通信瓶颈，分别是V2I DT重建时的模型参数交换和每次重建后的多服务V2I流量交付。本文提出了一个闭环双阶段端到端（E2E）延迟优化框架，将压缩个性化联合学习与统一的大语言模型引导多代理强化学习方法整合。基于CPFL的决策导向V2I数据分析仪首先通过联合预测路径丢失（PL）和体验质量（QoE）导向的关键绩效指标（KPI）构建。随后，框架包含两个时间耦合但延迟不同的优化阶段：第一阶段最小化压缩共享主干网参数的联合上行（UL）和下行（DL）E2E传输延迟，第二阶段最小化多服务V2I流量的服务感知E2E延迟。由于两阶段共享对齐的代理角色和动作空间，我们开发了一个统一的语义感知导引增强多代理近端策略优化框架，LLM作为训练时语义规划器，提供高级别流量分割和功耗分配指导以减少低效率探索，而MARL代理则执行可行的低层操作。模拟结果表明，所提出框架能够高效重建V2I数据分析，并显著降低DT重建和多服务V2I流量的E2E延迟。拟议的SAGE-MAPPO支持跨阶段策略重用，通过在每个DT更新周期内顺序解决两个阶段，加速了相对于现有基线的收敛，并促进动态V2I场景下的及时DT更新。

Sample-Efficient Reinforcement Learning for Policy Improvement via Flow-Reversed Action Anchors

通过流反向动作锚点进行样本高效强化学习以改进策略

• Authors: Carl Qi, Weijun Dong, Xiaojie Wang, Peiyi Chen, Rahul Sajnani, Mingyuan Zhang, Amy Zhang, Jose Barreiros, Manikatan Nambi, Lillian J. Ratliff, Dylan Randle
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22852
• Pdf link: https://arxiv.org/pdf/2610.22852
• Abstract Generalist robot policies offer strong behavioral priors, but achieving deployment-level reliability and throughput remains difficult. Reinforcement learning (RL) can improve these policies, yet its effectiveness is constrained by the cost of online interaction. Existing methods adapt frozen flow-matching policies by treating noise space as a continuous action space. We show that this behaviorally unstructured space makes exploration inefficient, often preventing these methods from extracting meaningful learning signals on challenging tasks. We introduce AnchorQ, which enables coarse-to-fine policy improvement through a structured, refinable latent action representation. At each decision step, AnchorQ uses flow reversal to map reference behaviors to state-conditioned latent anchors, samples local particles around those anchors, and applies value-guided sampling to explore promising refinements. Our analysis shows that anchors reduce identification complexity at the cost of support bias; particle refinement trades this bias against action-class complexity; and critic accuracy determines the reliability of KL-regularized, value-guided selection. Across simulated and real-world manipulation tasks, AnchorQ outperforms state-of-the-art post-training baselines in sample efficiency, delivering a 37% improvement in real-world success rate and 97% higher simulated task throughput under the same interaction budget. Additional visualizations are available on our website this https URL.
• 中文摘要 通用机器人策略提供了强的行为先验，但实现部署级别的可靠性和吞吐量仍然困难。强化学习（RL）可以改进这些策略，但其有效性受限于在线交互的成本。现有方法通过将噪声空间视为连续动作空间来适应冻结流匹配策略。我们表明，这种行为无结构空间使探索效率低下，常常阻碍这些方法在具有挑战性任务中提取有意义的学习信号。我们引入了AnchorQ，通过结构化、可精炼的潜在动作表示实现从粗略到细致的策略改进。在每个决策步骤中，AnchorQ利用流反转将参考行为映射到状态条件潜在锚点，采样这些锚点周围的局部粒子，并应用价值引导抽样以探索有前景的细化。我们的分析显示，锚点以牺牲支持偏倚为代价降低识别复杂性;粒子细化则在这种偏差与动作类复杂性之间进行平衡;而批评准确度决定了KL正则化、价值引导选择的可靠性。在模拟和真实世界操作任务中，AnchorQ在样本效率上优于最先进的训练后基线，在相同交互预算下实现了37%的真实成功率提升和97%的模拟任务吞吐量。更多可视化内容可在我们的网站获取，网址为 https URL。

Federated Multiagent Soft Actor Critic for IRS Assisted Cognitive Radio Networks under Partial Observability

联邦多智能体软演员批评者，针对部分可观察性下的IRS辅助认知无线网络

• Authors: Satvika Pullisani, Usama Mir, Ubaid Abbasi
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2610.22884
• Pdf link: https://arxiv.org/pdf/2610.22884
• Abstract Intelligent reflecting surfaces (IRSs) can improve spectrum efficiency and interference control in cognitive radio networks (CRNs) by reconfiguring the propagation environment. This paper presents FMASAC, a federated multi-agent soft actor-critic framework for IRS-assisted CRNs under partial observability, in which distributed secondary users (SUs) jointly select their transmit power and the phase shifts of an assigned subset of IRS elements using only local observations. Primary user (PU) protection is incorporated through an interference-penalized reward, and SUs coordinate by periodically averaging model parameters at a federated server, without exchanging raw observations, channel state information, or replay data. We formulate the problem as a partially observable Markov decision process, specify the local learning and aggregation procedure, and characterize its per-round communication cost. Illustrative performance trends contrast FMASAC with centralized SAC, independent SAC, IRS-fixed multi-agent reinforcement learning, and a heuristic CRN strategy. We also outline the simulation-based evaluation and the open challenges, namely constraint handling, agent heterogeneity, and scalability, that are needed to validate the framework.
• 中文摘要 智能反射面（IRS）可以通过重新配置传播环境，提升认知无线网络（CRN）中的频谱效率和干扰控制。本文介绍了FMASAC，一种联邦多代理软演员-批评框架，适用于IRS辅助的部分可观测CRN，分布式次级用户（SU）通过仅局部观测共同选择其发射功率和指定IRS元素子集的相位偏移。主要用户（PU）保护通过干扰惩罚奖励实现，主用户通过在联邦服务器上定期平均模型参数进行协调，无需交换原始观测值、信道状态信息或回放数据。我们将问题表述为部分可观测的马尔可夫决策过程，指定局部学习和聚合过程，并表征其每轮通信成本。示例性性能趋势对比了FMASAC与集中式SAC、独立SAC、IRS固定多智能体强化学习以及启发式CRN策略。我们还概述了基于仿真的评估以及验证框架所需的未解决挑战，即约束处理、代理异质性和可扩展性。

Batched Contextual Reinforcement II: Does RLVR Need Long Rollout Windows?

批量上下文强化 II：RLVR 需要较长的滚动窗口吗？

• Authors: Bangji Yang, Hongbo Ma, Ruihan Guo, Jiajun Fan, Xi Zhu, Ge Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.22887
• Pdf link: https://arxiv.org/pdf/2610.22887
• Abstract Long rollout windows make reinforcement learning with verifiable rewards (RLVR) expensive, yet simply shortening them can remove the correctness signal: when every single-problem rollout truncates before an answer, group-relative accuracy advantages vanish. As a technical continuation of Batched Contextual Reinforcement (BCR), we study how concurrent rewards relax this dependence on long windows. Concurrent Constraint Reinforcement (CCR) places three independently verifiable problems in one capped training completion. Correct answers completed before truncation retain credit, while a shared budget encourages shorter useful reasoning without an explicit length penalty. We connect this mechanism to conditional reward- and update-variance analyses and study 4B and 8B mathematical reasoning models. Against single-problem GRPO with a 32k window, CCR with an 8,000-token window reduces tokens per training step from 975,464 to 174,957 on 4B and from 793,832 to 165,788 on 8B, saving 82.1% and 79.1%, respectively. An 8B training trace shows increasing accuracy reward alongside shorter completions. Single-problem evaluations provide supporting evidence of retained reasoning quality. These results identify reward-preserving training structure as a practical route to lower per-step cost in short-window RLVR.
• 中文摘要 长的滚动窗口使得带可验证奖励的强化学习（RLVR）成本高昂，但仅仅缩短窗口可以消除正确性信号：当每个单一问题的滚动在答案前截断时，组相对准确率优势将消失。作为批量上下文强化（BCR）的技术延续，我们研究并发奖励如何放松对长窗口的依赖。并发约束强化（CCR）将三个独立可验证的问题集中在一个限制训练完成中。在截断前完成的正确答案可保留学分，而共享预算则鼓励更短的有用推理且不存在显式长度惩罚。我们将该机制与条件奖励和更新方差分析相结合，并研究4B和8B数学推理模型。针对32k窗口的单问题GRPO，CCR在8,000个令牌窗口下，4B每训练步的令牌数从975,464减少到174,957,8B从793,832减少到165,788，分别节省82.1%和79.1%。8B训练轨迹显示，准确率奖励在完成时间缩短的同时增加。单一问题评估为推理质量的保留提供了支持证据。这些结果表明，保持奖励的训练结构是短窗口RLVR中降低每步成本的实用途径。

Reward Engine: Reward-Centric Self-Evolution for Open-Ended Tool Agents

奖励引擎：以奖励为中心的开放式工具代理自我进化

• Authors: Litong Feng, Ji Qi, Yiran Cheng, Shaowu Pan, Xinyu Zhang, Qijun Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22915
• Pdf link: https://arxiv.org/pdf/2610.22915
• Abstract Open-ended tool agents can complete the same objective through different tool paths, and their quality depends jointly on the interaction trace, environment state, and final artifacts, so such tasks rarely admit an executable ground-truth reward. The central challenge is therefore to construct a reward from existing experience that exposes the current policy's failures while remaining stationary during optimization. We introduce Reward Engine, which, like a self-motivated student, first works out what success requires and then optimizes toward it. From the task and initial trajectories, it builds a RewardPack of testcases for required correctness and rubrics for graded quality, each executed as a check that returns a judgment with evidence. The frozen RewardPack then guides the agent to revise and select its outputs, and can also serve as a reward for reinforcement learning. On WorkBuddy Bench Office and Claw-Eval, the constructed reward largely agrees with the benchmark graders, not reversing their ordering on 82.3\% and 80.5\% of candidate pairs. Used to revise executions, it raises three-run consistency ($\mathrm{Pass}^{3}$) over Baseline from 45.0\% to 54.0\% and from 78.4\% to 83.7\%.
• 中文摘要 开放式工具代理可以通过不同的工具路径完成同一目标，其质量共同依赖于交互轨迹、环境状态和最终工件，因此此类任务很少能获得可执行的真实奖励。因此，核心挑战是从现有经验中构建一个奖励，既暴露当前策略的失败，又在优化过程中保持不变。我们引入了奖励引擎，它像自驱学生一样，首先确定成功所需的条件，然后朝此进行优化。从任务和初始轨迹出发，它构建了一个包含必要正确性测试案例和评分质量评分的评分标准的奖励包，每个测试用例作为一个检查返回带有证据的判断。冻结的奖励包随后引导代理修改和选择其输出，也可以作为强化学习的奖励。在WorkBuddy Bench Office和Claw-Eval中，构造奖励大体上与基准评分者一致，未反转82.3%和80.5%候选配对的排序。用于修订执行，将三次运行一致性（$\mathrm{Pass}^{3}$）提升至基线的45.0%至54.0%，从78.4%提升至83.7%。

Ego2Contact: Predicting Touch from Egocentric Human Video for Contact-Rich Robot Manipulation

Ego2Contact：从自我中心的人类视频预测触觉，用于接触丰富的机器人操作

• Authors: Qianqian Yang, Dayou Li, Bangya Liu, Zihao Zhu, Nuo Chen, Siyuan Xu, Hezhen Hu, Chen Yu, Rebecca Friesen, Minghui Zheng, Cynthia Hipwell, Xiaoning Qian, Kenji Shimada, Zhiwen Fan
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.22932
• Pdf link: https://arxiv.org/pdf/2610.22932
• Abstract Contact locations and interaction forces are critical for contact-rich robot manipulation but are often ambiguous from vision alone. Measuring them on robots requires specialized tactile hardware and platform-specific data collection, which keeps robot tactile data scarce. Human egocentric video is far easier to scale but lacks touch. Recent predictors trained on egocentric recordings paired with tactile-glove measurements can predict this missing signal from video alone. These predictors, however, remain confined to human hands, which differ from robot hands in appearance and tactile layout. We present Ego2Contact, which learns a visual-tactile prior from human interactions and transfers it to robot hands. Its predictor fuses temporal scene context with hand-local visual features to infer bimanual tactile fields from the RGB video. Anatomical region correspondences align varying robot tactile layouts with the predictor's canonical grid, turning robot measurements into fine-tuning targets and predictions into policy inputs. Ego2Contact outperforms prior tactile predictors on human data, including unseen objects, and on robot data after adaptation. The adapted predictor provides tactile inputs for both simulated reinforcement learning and real-robot policy post-training. In simulation, policies using predicted touch outperform non-tactile baselines and recover part of the benefit of ground-truth tactile sensing. These results show that human-learned tactile priors can supply touch when direct tactile sensing is limited.
• 中文摘要 接触位置和交互力对于接触丰富的机器人操作至关重要，但仅凭视觉往往存在模糊性。在机器人上测量这些力量需要专用的触觉硬件和平台特定数据收集，这使得机器人触觉数据极为稀缺。以自我为中心的视频更容易扩展，但缺乏触觉。最近基于自我中心录制并结合触觉手套测量训练的预测器，可以仅凭视频预测这一缺失信号。然而，这些预测器仍局限于人手，而人手在外观和触觉布局上与机器人手不同。我们介绍Ego2Contact，它通过人类互动学习视觉-触觉先验并将其转移到机器人手上。其预测器融合了时间场景上下文与手部局部视觉特征，从RGB视频推断出双手触觉场。解剖区域对应将不同的机器人触觉布局与预测器的典范网格对齐，将机器人测量转化为微调目标，将预测转化为策略输入。Ego2Contact 在人类数据（包括看不见物体）及适应后机器人数据上表现优于以往的触觉预测器。适应后的预测器为模拟强化学习和真实机器人策略训练后提供触觉输入。在模拟中，使用预测触摸的策略优于非触觉基线，并部分恢复了地面真实触觉感知的优势。这些结果表明，当直接触觉感知受限时，人类学习的触觉先验能够提供触觉。

Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism

多偏见，一电路：后训练调谐预训练提示路由机制

• Authors: Prakhar Gupta, Terry Jingchen Zhang, Zhijing Jin
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.22937
• Pdf link: https://arxiv.org/pdf/2610.22937
• Abstract Language models often abandon a correct answer when the prompt contains a cue, a hint toward a wrong option such as "I think the answer is B." Sycophancy and related biases show this failure. They are often studied separately and blamed on alignment. We test this view causally, tracing the attention heads behind these biases through the stages of two post-training pipelines. In multiple-choice tests on most models, four cue types rely largely on the same few heads, a shared circuit already present in pretrained models. Most base models follow cues in plain text. Fine-tuning a base model on about 200 generic assistant examples without cues activates these heads in chat format, so cue-following can be a side effect of assistant tuning. The final reinforcement learning (RL) stage strengthens the effect of cues in almost every comparison. In Tulu-3-8B and OLMo-2-7B, we swap the attention patterns of six middle layers between the RL checkpoint and its predecessor. For suggested-answer cues, this shifts each checkpoint's cue effect most of the way to the other's level. Post-training thus seems to change when and how strongly models use this circuit. Mitigation for these multiple-choice biases could target this circuit.
• 中文摘要 当提示包含提示（暗示错误选项，如“我认为答案是B”）时，语言模型常常放弃正确答案。谄媚及相关偏见显示了这种失败。它们常被单独研究，并归咎于对齐。我们通过两个训练后流程的阶段，通过因果方式测试这种偏见背后的注意力头。在大多数模型的多项选择测试中，四种提示类型主要依赖于同样的几个头，这是预训练模型中已有的共享电路。大多数基础模型遵循明文提示。在约200个无提示的通用助手示例上微调基础模型，会激活这些聊天格式的头，因此提示跟踪可能是辅助调优的副作用。最后一个强化学习（RL）阶段几乎在所有比较中都会加强提示的效果。在Tulu-3-8B和OLMo-2-7B中，我们在强化学习检查点与其前身之间交换了六个中间层的注意力模式。对于建议答案提示，这会将每个检查点的提示效应大部分移到另一个节点的水平。因此，后训练似乎改变了模型使用该回路的时间和强度。缓解这些选择偏差可以针对该回路。

Aligning Language Models Towards Counterfactual Invariance

使语言模型趋向反事实不变性

• Authors: Basar Demir, Emily Shen, Ksheeraj Sai Vepuri, Sourabh Kulkarni, Harihar Subramanyam, Ankit Jain, Nan Jiang, Chirag Nagpal
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.22942
• Pdf link: https://arxiv.org/pdf/2610.22942
• Abstract Small variations in prompt characteristics can meaningfully alter the semantics, tone, and contextual assumptions of large language model outputs. Such characteristics can involve group variables such as demographics or other sensitive latent attributes. Sensitivity to such input variables is undesirable when demographic information is causally irrelevant to the task, but might still be relevant for generation of semantically coherent responses. This makes the task of enforcing counterfactual invariance while preserving model utility difficult. To study this problem, we first show that standard model alignment recipes (such as reinforcement learning) increase differences in response quality across demographics, while improving overall response quality, ultimately amplifying what is known as counterfactual bias. To address this, we develop a policy gradient technique involving a group-level reward function that operates on matched pairs of counterfactual prompts. For completeness, we build a dataset of controlled counterfactual prompt pairs and design an evaluation that separately measures cases where demographic information is causally relevant to the answer. Finally, we validate our findings on several fairness benchmarks. We demonstrate that our methods reduce counterfactual differences while preserving overall response quality and that the models continue to adapt appropriately when such sensitive information is relevant.
• 中文摘要 提示特征的微小差异可以显著改变大型语言模型输出的语义、语气和上下文假设。这些特征可能涉及群体变量，如人口统计学或其他敏感潜在属性。当人口统计信息与任务因果无关时，对这些输入变量的敏感性是不理想的，但对于生成语义连贯的回答仍可能相关。这使得在维护模型效用的同时执行反事实不变性变得困难。为研究该问题，我们首先证明标准模型比对配方（如强化学习）会增加不同人口统计学的响应质量差异，同时提升整体响应质量，最终放大所谓的反事实偏见。为此，我们开发了一种策略梯度技术，利用组级奖励函数作用于匹配的反事实提示对。为完整性起见，我们构建了一个受控反事实提示对数据集，并设计了评估，分别测量人口统计信息与答案因果相关的案例。最后，我们在多个公平性基准上验证了结果。我们证明我们的方法在保持整体响应质量的同时减少了反事实差异，并且模型在涉及此类敏感信息时会持续适当调整。

DiffusionRVGE: Reinforcement Learning via Regression-Based Value Gradient Estimation

扩散RVGE：基于回归的值梯度估计强化学习

• Authors: Jiayuan Wang, Tongyao Pang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.22973
• Pdf link: https://arxiv.org/pdf/2610.22973
• Abstract Reinforcement learning improves diffusion models using rewards on generated images, but repeated denoising and gradient estimation make training expensive. We introduce \method, an efficient method that constructs useful update directions by regressing centered rewards on centered final denoised images. It requires neither reward derivatives nor adjoint backpropagation through the sampling trajectory. A unified first-variation formulation identifies the value-gradient target with or without a trajectory-KL penalty. A population analysis under Gaussian noising explains the regression geometry, while a subspace analysis motivates clean-image coordinates. The implementation adapts rollout bandwidth and query weights to reward continuity: local rollouts suit smooth rewards, whereas broader groups provide informative contrasts for discrete rewards. Trained jointly on five rewards, one SD3.5-Medium model matches or exceeds DiffusionNFT's joint model on seven of eight metrics, reaching GenEval $0.94$, OCR $0.92$, and PickScore $23.82$ at about one fifth of its estimated training compute. In single-reward training, \method reaches matched reward thresholds $2.1$--$6.7\times$ faster than reproduced DiffusionNFT on identical hardware. Ablations support the clean-image estimator, direct direction transfer, and reward-dependent rollout choices. Code is available at this https URL.
• 中文摘要 强化学习通过对生成图像的奖励改进扩散模型，但反复去噪和梯度估计使训练成本高昂。我们介绍了 \ 方法，这是一种高效方法，通过对中心最终去噪图像回归中心奖励构建有用的更新方向。它既不要求奖励导数，也不需要通过采样轨迹进行伴随反向传播。统一的第一变异表述识别了有或无轨迹-KL 惩罚的值梯度目标。高斯噪声下的总体分析解释了回归几何，而子空间分析则激励了干净图像坐标。实现调整了扩展带宽和查询权重以实现奖励连续性：局部展开适合平滑奖励，而更广泛的组则为离散奖励提供有益的对比。在五项奖励上联合训练时，一个SD3.5-Medium模型在八项指标中有七项与DiffusionNFT联合模型匹敌或超越，达到GenEval $0.94美元、OCR $0.92美元、PickScore约23.82美元，约占其估计训练计算量的五分之一。在单一奖励训练中，\方法比在相同硬件上复制的DiffusionNFT更快达到匹配奖励阈值$2.1美元-6.7美元\倍数。消融支持干净图像估计器、直接方向转移和基于奖励的展开选项。代码可在此 https URL 获取。

Where Research Ideation Begins: Forward-Citation Supervision for Actionable Limitation Generation

研究创意的起点：可操作限制生成的前引监督

• Authors: Liyang Zhao, Yong Cao, Haibin Ling
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23038
• Pdf link: https://arxiv.org/pdf/2610.23038
• Abstract Automated research ideation has advanced rapidly, with multi-agent workflows that generate whole research ideas end to end. These frameworks leave largely unexamined the step that seeds a good idea: identifying an actionable limitation from literature. In this paper, we focus on this first step in the research ideation pipeline. Starting from a pilot study, we find that as large language models (LLMs) scale from 0.6B to 32B parameters, they become far better at recognizing which limitation later work addressed, but no better at generating one, suggesting that the bottleneck lies in the supervision signal rather than model capacity. We therefore construct ActLim-Bench from forward citations, yielding 18,279 evidence-linked supervision triples with human verification. On this signal, we propose a three-stage post-training pipeline: supervised fine-tuning, verifier-filtered rejection-sampling fine-tuning, and reinforcement learning with the verifier as an online dense reward. Experimental results show that our method achieves the highest recall score compared to existing methods among all frontier models. We further show that our generated limitations are preferred in 64.9% of non-tied human comparisons and improve downstream ideation systems. We will release our data and code upon publication.
• 中文摘要 自动化研究构思发展迅速，多智能体工作流程能够从头到尾生成完整的研究想法。这些框架几乎未涉及孕育好想法的步骤：从文献中识别可操作的限制。本文重点介绍研究创意流程的第一步。从试点研究出发，我们发现当大型语言模型（LLMs）从0.6B到32B参数时，它们在识别后续研究解决的限制方面表现显著提升，但生成限制方面并无改善，表明瓶颈在于监督信号而非模型容量。因此，我们从前引构建ActLim-Bench，得出18,279个与人工验证的证据关联监督三元组。基于该信号，我们提出了三阶段训练后流程：监督微调、验证者过滤拒绝抽样微调，以及以验证者作为在线密集奖励的强化学习。实验结果显示，我们的方法在所有前沿模型中达到了比现有方法更高的回忆得分。我们还进一步表明，在64.9%的非绑定人类比较中，我们产生的限制性更受青睐，并改进了后续的思维系统。我们将在发布后发布数据和代码。

Agentic In-Context Learning: What Algorithms Can Transformers Learn in Closed-Loop Interaction?

代理上下文学习：变换金刚在闭环交互中可以学习哪些算法？

• Authors: Shiguang Wu, Yaqing Wang, Quanming Yao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23051
• Pdf link: https://arxiv.org/pdf/2610.23051
• Abstract The learning algorithm view of in-context learning (ICL) studies how fixed-parameter Transformers learn from supplied examples. Language model agents extend this setting through interaction: their actions determine what evidence enters context next. We call this regime Agentic In-Context Learning (Agentic ICL). Existing studies largely depend on externally supplied contexts or reward-bearing interaction histories, while work on general tool-using agents lacks a learning algorithm view. We represent an interactive learner by its state, update, and decision rule, covering active learning, bandits, and online reinforcement learning (RL). For expressivity, we construct causal Transformers that approximate these algorithms over a finite horizon, and bound the resulting loss in return. For training, we characterize when optimizing process rewards preserves behavior that maximizes the final outcome. Distillation experiments show that Transformers can learn to execute three representative recursions. We then train models from outcome or process rewards without oracle action or state targets, as in RL post-training. Different rewards yield distinct decision policies. Improving the alignment of process supervision with outcome-optimal action preferences shifts learned decisions accordingly. Such behavioral differences persist in a pretrained language model across textual paraphrases.
• 中文摘要 上下文学习（ICL）的学习算法视角研究固定参数变换器如何从提供的示例中学习。语言模型代理通过交互扩展这一环境：它们的行为决定下一步进入上下文的证据。我们称此机制为代理上下文学习（Agentic ICL）。现有研究主要依赖外部提供的上下文或奖励交互历史，而一般工具使用代理的工作缺乏学习算法视角。我们通过交互式学习者的状态、更新和决策规则表示，涵盖主动学习、强盗和在线强化学习（RL）。对于表达性，我们构建了在有限视野内近似这些算法的因果变换器，并对所得损失进行约束。对于训练，我们描述优化过程奖励时保持最大化最终结果的行为。蒸馏实验表明，Transformers可以学习执行三个代表性递归。然后我们从没有预言机动作或状态目标的结果或过程奖励中训练模型，如同强化学习后的训练。不同的奖励会产生不同的决策策略。改善过程监督与结果最优动作偏好的对齐，会相应地调整所学决策。这种行为差异在预训练语言模型中，跨文本释义中依然存在。

WebCoder: Scaling General Frontend Development with Artifact-Grounded Agentic Reinforcement Learning

WebCoder：利用基于工件的智能强化学习扩展通用前端开发

• Authors: Zijun Dou, Zhen Yang, Xijun Liu, Dongzhuoran Zhou, Aohan Zeng, Jie Tang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23104
• Pdf link: https://arxiv.org/pdf/2610.23104
• Abstract Real-world frontend development requires agents to compose reproduction, generation, modification, and repair over an evolving application state. Learning a general policy for this process requires translating heterogeneous execution feedback into task-aligned rewards and effective credit assignment across long sequences of interdependent actions. We introduce WebCoder, a general frontend development agent trained with artifact-grounded agentic reinforcement learning. Our key idea is to assess diverse development activities through their effects on a shared executable artifact and its satisfaction of task requirements. We build Webdev-World, an executable environment that integrates code editing, application execution, browser inspection, and on-demand skills while preserving project state across interactions. An independent verifier evaluates the final application against task requirements to provide terminal rewards, while changes in intermediate artifact progress redistribute the resulting rollout advantage across development segments. Context compaction and message-tree sample construction support training on trajectories beyond a single context window while avoiding repeated loss computation on shared generated history. We further introduce Webdev-Bench to evaluate composed development requirements within a single project. On Webdev-Bench, WebCoder-27B raises requirement satisfaction from 64.8 for its Qwen3.8-27B base model to 72.8, exceeding Claude-Opus-4.8 (65.1) and GPT-5.4 (59.6); on Vision2Web, it improves the average visual and functional score from 41.1 to 62.4, on par with Claude-Opus-4.8. Ablation studies confirm the contributions of artifact-grounded credit assignment and long-horizon training.
• 中文摘要 现实世界的前端开发要求代理在不断演变的应用状态上组合复制、生成、修改和修复。学习这一过程的通用策略需要将异构执行反馈转化为任务对齐的奖励和在长时间相互依赖操作序列中的有效信用分配。我们介绍WebCoder，一款通用前端开发代理，通过基于工件的代理强化学习训练。我们的核心思想是通过多样化的开发活动对共享可执行工件及其任务需求的满足情况进行评估。我们构建了Webdev-World，这是一个集成代码编辑、应用执行、浏览器检查和按需技能的可执行环境，同时保持项目状态在交互间的状态。独立验证者根据任务需求评估最终应用以提供终端奖励，而中间工件进度的变化则将由此产生的推广优势重新分配到开发部分。上下文压缩和消息树样本构建支持对单一上下文窗口之外轨迹的训练，同时避免对共享生成历史重复丢失计算。我们进一步引入Webdev-Bench，用于评估单个项目内的组合开发需求。在Webdev-Bench上，WebCoder-27B将Qwen3.8-27B基础模型的需求满意度从64.8提升至72.8，超过Claude-Opus-4.8（65.1）和GPT-5.4（59.6）;在Vision2Web上，平均视觉和功能得分从41.1提升至62.4，与Claude-Opus-4.8相当。消融研究证实了基于工件的信用分配和长期视野训练的贡献。

LED: Improving Masked Diffusion Language Models by Lookahead Editing

LED：通过前瞻编辑改进掩盖扩散语言模型

• Authors: Yuyan Zhou, Zeyuan Yang, Yuxuan Song, Zhicheng Cai, Hongli Yu, Weiying Ma, Hao Zhou, James Kwok
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.23122
• Pdf link: https://arxiv.org/pdf/2610.23122
• Abstract Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive language models, enabling non-autoregressive text generation through parallel decoding. Existing dLLMs can be broadly categorized into two families: masked diffusion models (MDMs) and uniform diffusion models (UDMs) In this work, we theoretically show that MDMs are easier to optimize due to their mask-based transition schedule but have lower computational expressivity, whereas UDMs enable iterative state revision and greater expressivity but are substantially harder to train. This trade-off raises a natural question: can we obtain the expressivity of iterative revision without giving up the simple, easy-to-optimize transition schedule of MDMs? Motivated by this analysis, we propose Lookahead Editing (\ours{}), which augments masked diffusion decoding with a rewritable proposal state. At each decoding step, the model performs several lookahead updates to iteratively refine future predictions in this proposal state before committing tokens to the masked sequence. This design introduces a rewritable proposal that uses additional compute to improve expressivity while keeping the simple mask-based transition schedule of MDMs. We fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-\texttt{7B} and LED-\texttt{8B}, respectively. Across eight benchmarks, LED SFT yields average gains of $+11.4$ and $+7.7$ points over standard MDM SFT. Moreover, using SFT alone, LED-\texttt{8B} outperforms LLaDA2.1-16B Q-mode, which is trained with continued pre-training, supervised fine-tuning, and reinforcement learning.
• 中文摘要 扩散大型语言模型（dLLMs）已成为自回归语言模型的有前景替代方案，使得通过并行译码实现非自回归文本生成。现有的dLLM大致可分为两类：掩蔽扩散模型（MDMs）和均匀扩散模型（UDM）。本研究理论上表明，MDM因其基于掩码的过渡调度而更容易优化，但计算表达性较低;而UDM支持迭代状态修正和更高的表达力，但训练起来相对困难。这一权衡引发了一个自然问题：我们能否在不放弃MDM简单、易于优化的过渡计划的情况下，获得迭代修订的表达力？基于此分析，我们提出了前瞻性编辑（\ours{}），它通过可重写的提案状态增强掩蔽扩散解码。在每个解码步骤中，模型会进行多次前瞻更新，在将标记提交到掩码序列前，迭代优化该提案状态下的未来预测。该设计引入了可重写的提案，利用额外计算提升表达力，同时保持MDM的简单基于掩码的过渡计划。我们对Qwen2.5-7B和Qwen3-8B进行了微调，分别获得LED-\texttt{7B}和LED-\texttt{8B}。在八个基准测试中，LED SFT平均比标准MDM SFT提升$+11.4和$+7.7$。此外，仅使用SFT时，LED-\texttt{8B}的表现优于LLaDA2.1-16B Q模式，后者通过持续的预训练、监督微调和强化学习进行训练。

BeJEPA: A Goal-Conditioned Humanoid Behavior World Model via Unsupervised RL

BeJEPA：通过无监督强化学习的目标条件类人行为世界模型

• Authors: Liyun Yan, Haoyang Li, Yang Zhang, Jialing Zhu, Zhanxiang Cao, Zihao Zhou, Keqi Zhu, Yizhi Chen, Huang Chuxuan, Shengcheng Fu, Zhong Jiangwei, Qiao Sun, Chen Zhang, Yue Gao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.23127
• Pdf link: https://arxiv.org/pdf/2610.23127
• Abstract Humanoid control requires coordinating whole-body posture with position and heading in the world frame. Mainstream controllers imitate trajectories in the root frame, but struggle to regain the target world pose after displacement. We introduce BeJEPA, a goal-conditioned Behavior World Model learned through unsupervised reinforcement learning from human motion priors and robot interaction. BeJEPA learns a shared representation of body configuration and world placement through a joint-embedding predictive architecture (JEPA). A Dynamic Predictor predicts the next state embedding under an action, while a Successor Predictor summarizes a goal-conditioned policy's expected discounted future state embeddings in a single vector. During joint training, Dynamic grounds the initial representation in observed physical transitions, while Successor increasingly organizes it around long-term behavior. Their composition guides policy improvement through one physical prediction and one behavioral summary, without long imagined rollouts. In 20-second motion-tracking experiments, BeJEPA with online action refinement using Dynamic outperforms all evaluated baselines in global body-position error, reducing it on AMASS by 61.7% relative to BFM-Zero with world-pose inputs and 58.7% relative to SONIC. Physical deployment on a Unitree G1 demonstrates joint attainment of posture, world position, and heading goals, recovery from the ground, and closed-loop return after disturbances and goal changes. Project page: this https URL.
• 中文摘要 类人生物控制需要协调全身姿势与世界框架中的位置和方向。主流控制器在根框架中模拟轨迹，但在位移后难以恢复目标世界姿态。我们介绍BeJEPA，一种通过无监督强化学习从人类运动先验和机器人交互中学习的目标条件行为世界模型。BeJEPA通过关节嵌入预测架构（JEPA）学习身体配置和世界位置的共享表示。动态预测器预测动作下的下一个状态嵌入，而继任预测器则总结目标条件策略的预期折现未来状态嵌入单一向量。在联合训练中，动态系统将初始表征建立在观察到的物理转变中，而继任者则逐渐围绕长期行为组织表征。它们的组成通过一个物理预测和一个行为总结指导政策改进，避免长期设想的推广。在20秒运动跟踪实验中，BeJEPA通过动态在线动作细化，在全局体位误差方面优于所有评估基线，在AMASS上相较于带世界姿势输入的BFM-Zero降低了61.7%，相较于SONIC降低了58.7%。在Unitree G1上的物理部署展示了联合实现姿态、世界位置和航向目标的实现，从地面恢复，以及扰动和目标变化后的闭环返回。项目页面：此链接 https URL。

Grounded Credit Assignment in Reinforcement Learning for Multi-Turn Search Agents

多回合搜索代理强化学习中的基础学分作业

• Authors: Qirui Liu, Hao Chen, Yi Wen, Weijie Shi, Yao Zhao, Xianhe Chen, Hao Zhao, Xing Tang, Weihong Luo, Yuyu Luo
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.23148
• Pdf link: https://arxiv.org/pdf/2610.23148
• Abstract Outcome-reward reinforcement learning such as GRPO gives every token of a search-agent rollout the same advantage, although the rollout exercises two capabilities: finding evidence and using it. We show that accuracy improves without grounding: Search-R1 raises accuracy on all seven QA benchmarks, yet on Bamboogle the evidence-supported share of correct answers falls from 84.0% to 42.9%. We attribute this to GRPO's credit assignment: rollouts that retrieve answer-bearing evidence are favoured in aggregate only in proportion to their accuracy gain over the rest, a grounding gap that answering from memory or misusing evidence shrinks or reverses. To address this, we propose Grounded Credit Assignment (GCA), which uses verifiable sub-outcomes (evidence retrieval and answer correctness) to correct search and answer credit separately: a wrong answer is blamed more than the search that found the evidence, and a correct answer that bypasses retrievable evidence is credited less. Never negative on evidence-finding search, the correction does not shrink with the grounding gap; capped at 10% of each token's advantage, it never overrides the outcome signal. Without extra models, rollouts, or training stages, GCA improves average EM over Search-R1 by 5.7 (Qwen2.5-3B) and 6.3 points (Qwen3-30B-A3B) and raises this share from 76.9% to 93.4%.
• 中文摘要 结果-奖励强化学习（如GRPO）赋予搜索代理推广中的每个代币相同的优势，尽管推广具有两种能力：寻找证据和利用证据。我们证明，准确性在不依赖基础的情况下会提升：Search-R1在所有七项QA基准中都提高了准确率，而在Bamboogle上，有证据支持的正确答案比例从84.0%降至42.9%。我们将此归因于GRPO的学分分配：检索有答案证据的推广仅按其准确率提升的比例被整体偏好，这一基础差距使记忆回答或滥用证据缩小或逆转。为此，我们提出了“基础学分分配”（GCA），该方法利用可验证的子结果（证据检索和答案正确性）分别正确搜索和回答署名：错误答案比找到证据的搜索被指责更多，而绕过可检索证据的正确答案则被认可更少。在证据查找搜索中从不负面，纠正不会随着基础差距缩小;每个代币优势的上限为10%，且不会覆盖结果信号。在没有额外模型、推广或训练阶段的情况下，GCA在搜索-R1上平均EM提升了5.7（Qwen2.5-3B）和6.3分（Qwen3-30B-A3B），并将该比例从76.9%提升到93.4%。

Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning

树感知后置蒸馏用于长视野代理强化学习

• Authors: Heming Zou, Qi Wang, Yixiu Mao, Yun Qu, Yuyi Guo, Zeyuan Liu, Chenglin Li, Weijie Liu, Saiyong Yang, Xiangyang Ji
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.23181
• Pdf link: https://arxiv.org/pdf/2610.23181
• Abstract Reinforcement learning with verifiable rewards has become a scalable post-training paradigm for large language models. However, training agents over long-horizon interactions presents two challenges: using only terminal rewards makes turn-level credit assignment sparse and high-variance, while sequential mini-batch updates on a fixed rollout batch introduce policy staleness as the current policy drifts from the behavior policy. Existing approaches obtain denser credit via learned critics, process reward models, or on-policy distillation, but they rely on a value network, step-level annotations, or a stronger external teacher that increase training complexity. Meanwhile, importance sampling and clipping mitigate policy drift, but they do not specify a finite target that stops sequential updates from overshooting. In this work, we introduce TreeOPSD, a critic-free, tree-aware posterior self-distillation framework for long-horizon agentic reinforcement learning. Treating each interaction turn as a tree node, TreeOPSD branches alternative continuations from shared prefixes and recursively backs up terminal rewards into implicit turn-level credit. It then maps each credit to a closed-form, bounded teacher distribution parameterized by the likelihood ratio relative to the frozen behavior policy. Minimizing the divergence between this teacher and the student policy recovers the policy-gradient direction as the tangent at the behavior policy while bounding updates toward a finite target across sequential mini-batches. Empirically, TreeOPSD improves agentic capabilities across several competitive benchmarks and model scales, especially for long-horizon tasks, e.g., raising the average accuracy of Qwen3-30B-A3B-Thinking-2507 by 4.97 points over GRPO on the Deep Research setting.
• 中文摘要 带有可验证奖励的强化学习已成为大型语言模型可扩展的训练后范式。然而，长期交互的训练代理面临两个挑战：仅使用终端奖励使回合级的信用分配稀疏且方差高;固定部署批次的顺序小批量更新则引入策略陈旧，因为当前策略偏离行为策略。现有方法通过学习批评者、过程奖励模型或策略内蒸馏获得更密集的信用，但它们依赖价值网络、步级注释或更强的外部教师，从而增加训练复杂度。同时，重要性抽样和裁剪可以减轻策略漂移，但它们未指定一个限制顺序更新超额的目标。本研究介绍了TreeOPSD，一种无批评、树感知的后验自我蒸馏框架，用于长期代理强化学习。TreeOPSD将每个交互回合视为树节点，从共享前缀分支替代延续，递归地将终端奖励备份为隐式回合级积分。然后将每个积分映射到一个闭合式、有界教师分布，该分布参数由相对于冻结行为策略的似然比参数化。最小化教师与学生策略之间的偏差，可以恢复行为策略的切线，同时在连续的迷你批次中向有限目标进行边界更新。从经验角度看，TreeOPSD在多个竞争基准和模型尺度上提升了代理能力，尤其是在长视野任务中，例如在深度研究设置下，Qwen3-30B-A3B-Thinking-2507的平均准确率比GRPO提高了4.97个点。

A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation

内在更强的教师：通过政策内双向提炼强化教育部模型

• Authors: Haoyu Chen, Jun Bai, Yang Liu, Jianfei Zhang, Wenge Rong
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23185
• Pdf link: https://arxiv.org/pdf/2610.23185
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved Mixture-of-Experts (MoE) LLMs, but its outcome-level rewards provide only sparse supervision. On-Policy Distillation offers dense token-level feedback, yet requires a stronger compatible teacher, while on-policy self-distillation relies on hand-designed privileged information. We observe that the MoE architecture itself offers a way to resolve this dilemma, as increasing the activation budget per token exposes latent expert capacity and yields a potentially stronger teacher within the model. However, naive expansion degrades performance because newly co-activated experts are not optimized to work together. We therefore propose MoEvolve, a bidirectional distillation framework following a recovery-then-improvement strategy: we first restore the activation-expanded high-$k$ model and further strengthen it by exploiting its additional active capacity, then reverse the direction of distillation to transfer the enhanced capabilities back into the original low-$k$ configuration, retaining its inference-time activation budget. Across diverse MoE models and benchmarks, MoEvolve improves the initial models by up to $22.2$ domain-average points and outperforms same-budget RLVR baselines by up to $6.5$ points. Notably, the final low-$k$ models often surpass their activation-expanded teachers.
• 中文摘要 带可验证奖励的强化学习（RLVR）大幅改进了专家混合（MoE）大型语言模型，但其成果级奖励仅提供稀疏监督。策略上提炼提供密集的代币级反馈，但需要更强的兼容教师，而策略内自我提炼依赖手工设计的特权信息。我们观察到，MoE架构本身提供了解决这一困境的方法，增加每个代币激活预算暴露潜在专家容量，并可能在模型中获得更强的教师。然而，天真的扩展会降低性能，因为新激活的专家并未优化协作。因此，我们提出了MoEvolve，一种双向蒸馏框架，采用恢复后改进策略：我们先恢复激活扩展的高$k美元模型，并通过利用其额外的主动容量进一步强化，然后反转蒸馏方向，将增强能力回归原始低$k美元配置，保留推理时间激活预算。在多种MoE模型和基准测试中，MoEvolve提升初始模型最多22.2美元领域的平均点，并比同预算RLVR基线高达6.5美元。值得注意的是，最终的低$k美元模型常常超过激活扩展后的教师。

Proportionality in Ranking Compression

排名压缩中的比例性

• Authors: Daniel Halpern, Xinyu Liu, Evi Micha, Yu Peng Ng, Warut Suksompong
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG); Theoretical Economics (econ.TH)
• Arxiv link: https://arxiv.org/abs/2610.23193
• Pdf link: https://arxiv.org/pdf/2610.23193
• Abstract Motivated by applications in reinforcement learning from human feedback, we consider a setting where a given collection of rankings needs to be compressed into a smaller collection of rankings. This can also be seen as a combination of two fundamental social choice frameworks: ranking aggregation and multiwinner voting. We propose three proportionality notions that capture different types of representation in this setting: positional proportionality, pairwise proportionality, and proportionality for solid coalitions (PSC). On the one hand, we show that positional proportionality and PSC are always satisfiable for any input rankings and target compression size, and a desired output can be found in polynomial time. On the other hand, we prove that pairwise proportionality cannot be satisfied in general, but can nevertheless be attained when the input rankings are single-peaked or single-crossing.
• 中文摘要 受人类反馈强化学习应用的激励，我们考虑了一个需要将给定排名集合压缩到更小排名集合中的情境。这也可以看作是两种基本社会选择框架的结合：排名聚合和多重赢家投票。我们提出了三种比例性概念，涵盖了该环境中不同类型的代表性：位置比例性、两两比例性和实体联盟比例性（PSC）。一方面，我们证明位置比例性和PSC对任何输入排名和目标压缩大小总是可满足，且可在多项式时间内找到期望输出。另一方面，我们证明成对比例性在一般情况下无法满足，但当输入排名为单峰或单交叉时，仍可实现。

SimpleVLN: Towards Simple and Frontier Vision-Language Navigation

SimpleVLN：迈向简单且前沿的视觉语言导航

• Authors: Lingyi Hong, Weiren Cai, Jiayu Chen, Xinyu Zhou, Bailin He, Yunfei Guo, Qiao Sun, Boyu Ma, Jianfei Yang, Wenqiang Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.23203
• Pdf link: https://arxiv.org/pdf/2610.23203
• Abstract Vision-and-Language Navigation (VLN) requires embodied agents to ground language in visual observations and make long-horizon decisions. Existing methods often address these demands through task-specific architectures. As multimodal large language models (MLLMs) become increasingly capable, an important question remains: Have we truly unlocked the potential of native MLLMs for VLN before introducing task-specific architecture? We present SimpleVLN, an architecture-preserving framework developed through studies of navigation formulation, data construction, and optimization. At the formulation level, SimpleVLN combines single-turn decision inputs with fixed-budget global-local memory to efficiently use navigation history without unbounded context growth, while action chunks enable efficient output of multiple primitive actions per model invocation. For data construction, we combine expert demonstrations with policy-dependent DAgger experience and develop a scalable synthesis pipeline that generates large-scale tasks for reinforcement learning. At the optimization level, a progressive IL-DAgger-RL pipeline advances from action imitation to task-level outcome optimization. Comprehensive experiments validate the effectiveness of these three aspects. We instantiate SimpleVLN across multiple MLLM backbones and model sizes, demonstrating its generality. SimpleVLN achieves 76.7% and 80.1% success rates on the R2R-CE and RxR-CE splits, respectively, substantially outperforming existing methods. These results show that the navigation capabilities of native MLLMs can be effectively unlocked without task-specific architectural designs.
• 中文摘要 视觉与语言导航（VLN）要求具身智能体在视觉观察中扎根语言，并做出长期决策。现有方法通常通过任务特定架构满足这些需求。随着多模态大型语言模型（MLLM）能力的提升，一个重要问题依然存在：在引入任务特定架构之前，我们是否真正释放了原生MLLM在VLN中的潜力？我们介绍SimpleVLN，一个通过导航表述、数据构建和优化研究开发的架构保持框架。在表述层面，SimpleVLN结合单回合决策输入与固定预算的全局-局部内存，高效利用导航历史且无无限上下文增长，而动作块则使每个模型调用都能高效输出多个原始动作。在数据构建方面，我们结合专家演示与策略依赖DAgger经验，开发可扩展的综合流水线，生成大规模强化学习任务。在优化层面，渐进式的IL-DAgger-RL流水线从动作模仿向任务级结果优化推进。全面的实验验证了这三个方面的有效性。我们将SimpleVLN实例化于多个MLLM骨干和模型规模，展示了其通用性。SimpleVLN在R2R-CE和RxR-CE区间分别实现了76.7%和80.1%的成功率，远超现有方法。这些结果表明，原生MLLM的导航能力无需任务特定架构设计即可有效解锁。

MultiAgent-OPD: End-to-End Training of Hierarchical Multi-Agent Systems with On-Policy Distillation

多智能体-OPD：基于策略提炼的层级多智能体系统的端到端训练

• Authors: Yuhan Sun, Jiaxin Zhao, Yongdong Luo, Yibo Hu, Meiguang Jin, Junfeng Ma, Weixun Wang, Weihang Pan, Zhongan Bi
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.23234
• Pdf link: https://arxiv.org/pdf/2610.23234
• Abstract In a hierarchical multi-agent system, a Leader assigns tasks and subagents carry them out. Jointly training the Leader and the subagents is difficult because a final answer gives only indirect feedback to the subagents. An LLM judge can score the performance of the subagents, but these scores depend on the judge's own assessment of the quality of subagents' responses. We present MultiAgent-OPD, which combines on-policy distillation with reinforcement learning to jointly train a shared-parameter Leader-Searcher-Answerer team end to end. We first train a Teacher in a Searcher-Answerer workflow using answer correctness. During team training, the frozen Teacher scores student-generated search and summary tokens to provide local process supervision for the Searchers, while the Leader learns from team success and the Answerer from candidate correctness. Across seven datasets, the trained system reaches 38.68% macro avg@4, exceeding Prompted Critic by 4.06 percentage points and C-MAS + RL, the trained cascaded multiagent system, by 2.61 percentage points. The team also solves more questions within four attempts.
• 中文摘要 在分层多智能体系统中，领导者分配任务，子智能体执行任务。联合培训领导者和子智能体困难，因为最终答案只能间接反馈子智能体。大型语言模型评委可以对子智能体的表现进行评分，但这些分数取决于评委自身对子智能体回答质量的评估。我们介绍了多智能体-OPD，它结合了策略提取与强化学习，以端到端共同训练共享参数的领导者-搜索者-回答者团队。我们首先使用答案正确性培训教师进行搜索-回答者工作流程。在团队培训过程中，冻结的教师对学生生成的搜索和总结令牌进行评分，以为搜索者提供局部过程监督，而领导者从团队成功中学习，回答者从候选人正确性中学习。在七个数据集中，训练系统的宏观avg@4达到38.68%，比Prompted Critic高出4.06个百分点，比训练级联多代理系统C-MAS + RL高2.61个百分点。团队还在四次尝试中解决了更多问题。

Beyond Reward Scoring: Online Multi-Agent Preference Learning for Multimodal Response Generation

超越奖励评分：在线多智能体偏好学习用于多模态反应生成

• Authors: Ziyun Zou, Jiwoo Hong, Yifan Lu, Shao Tang, Weiyi Wu, Aman Gupta, Jin Tian, Imran Razzak
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23256
• Pdf link: https://arxiv.org/pdf/2610.23256
• Abstract Multimodal preference alignment often follows a score-based RLHF pipeline: a reward model is first trained to learn human preferences, then used to score on-policy responses and optimize the response model's generation. This turns preference supervision into a scalar signal for generation: a model may learn to produce answers with higher preference scores, but is rarely trained explicitly to learn the reasons underlying human preferences. We therefore ask whether making preference understanding an internal capability of the response model can improve its generation. To investigate this question, we propose \textbf{PrefMind}, an online dual-agent reinforcement learning framework with two trainable agents: a response Generator and a preference Critic. Given a pair of candidate responses, the agents analyze and discuss which response better aligns with human preference. Human labels supervise their judgments and corrections. For pairs on which both the Generator and revised Critic correctly identify the human-preferred response, the Generator produces an improved answer, and the Critic compares it against the original preferred response to provide a generation reward. Both agents are jointly updated from interactions sampled with their current policies. Through this process, \textbf{PrefMind} trains the Generator to understand human preferences via explicit judgment and correction, and to translate this understanding into better responses. Experiments across three model scales show improvements in both standalone preference judgment and downstream response generation.
• 中文摘要 多模态偏好对齐通常遵循基于评分的RLHF流程：奖励模型先训练以学习人类偏好，然后用于对政策响应进行评分并优化响应模型的生成。这使偏好监督成为生成的标量信号：模型可能学习生成更高偏好分数的答案，但很少被明确训练以了解人类偏好背后的原因。因此，我们探讨将偏好理解作为响应模型内部能力是否能提升其生成能力。为探讨这个问题，我们提出了\textbf{PrefMind}，这是一个在线双智能体强化学习框架，具有两个可训练代理：响应生成器和偏好批评者。给定两个候选反应，代理们分析并讨论哪种反应更符合人类偏好。人类标签监督他们的判断和修正。对于生成器和修订后的Critic都正确识别了人类偏好反应的配对，生成器会生成更优的答案，批评者将其与原始偏好反应进行比较，从而提供生成奖励。两个代理都会根据当前策略的交互采样共同更新。通过这一过程，\textbf{PrefMind}训练生成器通过显式判断和纠正来理解人类偏好，并将这种理解转化为更好的反应。跨三个模型尺度的实验显示，独立偏好判断和后续反应生成均有提升。

Language Models Learn to Improve Language Models

语言模型学习改进语言模型

• Authors: Minhua Lin, Zijun Wang, Zewen Liu, Zhaotian Weng, Bing He, Zhan Shi, Benoit Dumoulin, Xiang Zhang, Suhang Wang, Hanqing Lu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23344
• Pdf link: https://arxiv.org/pdf/2610.23344
• Abstract Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.
• 中文摘要 大型语言模型（LLM）越来越多地被用于自动化人工智能研究，包括语言模型（LM）开发，为递归自我改进提供了潜在路径。然而，将实验反馈转化为跨模型提出有效改进的学习能力面临两个挑战。首先，用真实反馈训练此类能力成本高昂：强化学习（RL）需要评估数万个提案，每个提案都可能需要完整的模型训练运行。即使每次运行十个GPU小时，仅奖励收集本身也需耗费数十万GPU小时，促使更便宜的替代者。其次，随着策略优化偏向高估方案，替代方案可能变得不可靠，且提案分布发生变化。为应对这些挑战，我们提出了LM2LM，它通过强化学习训练LLM生成有效的LM改进提案。为了提供可扩展的反馈，我们使用研究世界模型（RWM），即一个固定的大型语言模型，预测基于上下文中提供的真实实验记录带来的提案收益，作为强化学习的奖励模型。为保持反馈可靠性，我们定期执行RWM排名较高的当前政策提案，并将其测量结果添加到其上下文中。在八个自回归和扩散的LM开发环境中，对未完成任务的实际执行评估显示，LM2LM使Qwen3.8-27B能够超越GPT-6-sol，即使GPT-6-sol在上下文中获得了300条真实实验记录，且数据生成成本降低33%。LM2LM还推广到训练中未见的模型族。

Bayesian Learning of Parametric Mean Field Games with Common Noise

具有共同噪声的参数均域博弈的贝叶斯学习

• Authors: Grégoire Lambrecht, Mathieu Laurière
• Subjects: Subjects: Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2610.23387
• Pdf link: https://arxiv.org/pdf/2610.23387
• Abstract We study the inverse problem of recovering unknown parameters governing the dynamics and rewards of Mean Field Games (MFGs) from partial observations of population trajectories, made stochastic by common noise. The observer seeks to recover a consistent parameter $\theta$ via Bayesian inference, facing the challenge that evaluating the likelihood requires solving a new Nash equilibrium (NE) for each candidate $\theta$. Due to common noise, the mean-field evolution is stochastic and equilibrium policies are population dependent, which makes solving for each parameter computationally expensive. To overcome this, we introduce the Bayesian-Extended Mean Field Game (BE-MFG), whose single NE simultaneously encodes equilibrium policies for all parameters, enabling efficient learning of the conditional density of mean-field trajectories. We formalize parameter consistency and identifiability in MFGs with common noise and investigate conditions under which consistent parameter recovery is guaranteed. On three synthetic environments of various complexity, we recover the ground-truth parameter accurately from a small number of partial, stochastic observations, outperforming an adaptation of inverse reinforcement learning for MFGs. A fourth experiment, on public electricity market data, suggests that the approach can also be applied to real-world data.
• 中文摘要 我们研究了通过对群体轨迹的部分观测（由普通噪声随机化）恢复未知参数的逆问题。观察者试图通过贝叶斯推断恢复一致参数 $\theta$，但评估似然需要为每个候选 $\theta$ 求解新的纳什均衡（NE）。由于共同噪声，平均场演化是随机的，均衡策略依赖总体，这使得求解每个参数计算成本过高。为克服这一问题，我们引入了贝叶斯扩展均场博弈（BE-MFG），其单一 NE 同时编码所有参数的均衡策略，从而高效学习平均场轨迹的条件密度。我们在具有共同噪声的MFG中形式化参数一致性和可识别性，并探讨在保证一致参数恢复的条件下。在三种不同复杂度的合成环境中，我们通过少量部分随机观测准确恢复了基层真实参数，优于MFG的逆强化学习适应。第四项实验，基于公共电力市场数据，表明该方法同样可应用于现实世界数据。

No Eyes, No Problem: What Does Touch Contribute to Blind Baoding Ball Manipulation?

没有眼睛，没问题：触觉对盲眼保丁球操控有什么影响？

• Authors: Ayush Deshmukh, Nalin Agarwal, Elle Miller, Sethu Vijayakumar
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.23438
• Pdf link: https://arxiv.org/pdf/2610.23438
• Abstract Humans manipulate objects within their hands through touch alone with remarkable ease, but for robots, achieving sustained real-world manipulation without vision remains an open challenge. Although reinforcement learning (RL) can generate sophisticated blind manipulation policies in simulation, transferring them to a real dexterous hand remains hindered by continuously changing contacts, motion-induced tactile information drift, and the gap between simulated and physical dynamics. We study this problem through Baoding ball rotation on the Shadow Dexterous Hand Lite, using only proprioception and binary tactile feedback as observations. We achieve successful zero-shot transfer through a combination of a sensor-aligned, coupling-aware embodiment in simulation, tactile calibration, and two novel episodic domain randomisations: slew-rate action limits and STuck-AT Taxels. On the physical hand, the policy achieves sustained blind Baoding rotation, completing 225 consecutive $\mathbf{180^\circ}$ rotations (112.5 full rotations) at $\sim0.36 \;\text{rot/s}$ in a single continuous trial without any vision or object-state information. Furthermore, we evaluate the contribution of tactile sensing relative to proprioception alone and identify the hand regions where tactile information is most important. This is the strongest reported hardware result for Baoding ball manipulation in both rotation count and speed, irrespective of the sensing modality used. By effectively bridging the sim-to-real gap, our framework sets a new benchmark for sustained tactile-driven dexterous manipulation on physical hardware.
• 中文摘要 人类仅凭触觉就能轻松操作手中的物体，但对于机器人来说，在没有视觉的情况下实现持续的现实操作仍是一个开放的挑战。尽管强化学习（RL）可以在模拟中生成复杂的盲操作策略，但将其转移到真正灵巧的手上仍受持续变化的接触点、运动引起的触觉信息漂移以及模拟与物理动力学之间的差距所阻碍。我们通过在Shadow Dexterous Hand Lite上进行保定球体旋转，仅以本体感觉和二元触觉反馈作为观察来研究该问题。我们通过模拟中传感器对齐、耦合感知的实例化、触觉校准以及两种新型的情节域随机化——斜率动作极限和STuck-AT Taxels——的结合，成功实现了零击中转移。在物理手部，该策略实现了持续盲测球旋转，在一次连续试验中完成了225次连续$\mathbf{180^\circ}$旋转（112.5个完整旋转），时间为$\sim0.36 \;\text{rot{s}$，且无任何视觉或物体状态信息。此外，我们评估了触觉感知相对于本体感觉的贡献，并识别了触觉信息最重要的手部区域。这是无论使用何种感知方式，在旋转次数和速度方面，保定球操控中报告的最强硬件结果。通过有效弥合模拟与现实的差距，我们的框架为物理硬件上持续的触觉驱动灵巧操作树立了新标杆。

RECREG: Recovery-based Shielding for Regular Safety Properties in Reinforcement Learning

RECREG：强化学习中常规安全属性的恢复式屏蔽

• Authors: Alexander W. Goodall, Francesco Belardinelli
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23445
• Pdf link: https://arxiv.org/pdf/2610.23445
• Abstract In many real-world scenarios we need to consider formal requirements, such as safety and constraint compliance, to deploy Reinforcement Learning (RL) agents. In this paper we introduce RecReg, a safe RL framework capable of enforcing non-Markovian regular safety properties via recovery-based shielding. RecReg is a unified approach that supports different levels of a priori knowledge and safety guarantees. The paper primarily focuses on building the theoretical foundations for RecReg in the tabular setting, providing rigorous safety and convergence guarantees that are not present in related work. We provide an extensive set of empirical results using the MASA-Safe-RL library, and we outline an extension of RecReg to deep RL domains using SOTA model-based RL. Our empirical results demonstrate that RecReg achieves efficient learning and matches the theory in this paper.
• 中文摘要 在许多现实场景中，我们需要考虑安全和约束合规等正式要求，以部署强化学习（RL）代理。本文介绍了RecReg，一个安全的强化学习框架，能够通过恢复屏蔽强制执行非马尔可夫正规安全属性。RecReg是一种统一方法，支持不同层次的先验知识和安全保障。本文主要聚焦于在表格化环境中构建RecReg的理论基础，提供相关工作中不具备的严谨安全性和收敛性保证。我们利用MASA-Safe-RL库提供了大量实证结果，并概述了利用SOTA模型驱动RL将RecReg扩展到深度强化学习领域的方案。我们的实证结果表明RecReg实现了高效学习，并与本文理论相匹配。

Reinforcement Learning-Based Pricing and Operation for Hybrid Electric and Hydrogen Vehicle Stations

基于强化学习的混合动力电动和氢能车站定价与运营

• Authors: Lowell Welburn, Yang Chen, Xudong Wang, Abhilasha Saroj, Shaked Regev
• Subjects: Subjects: Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2610.23464
• Pdf link: https://arxiv.org/pdf/2610.23464
• Abstract Electric vehicles (EVs) and hydrogen fuel cell vehi- cles (HFCVs) are promising technologies in the efforts to reduce carbon emissions. Challenges such as limited infrastructure, con- sumer dissatisfaction, and electrical grid strain inhibit progress in the adoption of these technologies. A zero emission vehicle (ZEV) station, capable of handling EVs and HFCVs, addresses these challenges by focusing on maximizing station profit and consumer utility, while minimizing grid impacts. The purpose of this research is to develop a realistic simulation of a ZEV station that is capable of handling real-time pricing decisions enabled by reinforcement learning (RL) and to compare the performance of the RL agent to static pricing methods. After training, the RL agent gains 10.83% greater station profit than the baseline manual parameter search. This work serves as a useful tool for ZEV station owners when handling complex pricing and purchasing decisions.
• 中文摘要 电动汽车（EV）和氢燃料电池车（HFCVs）是减少碳排放的有前景技术。基础设施有限、消费者不满和电网压力等挑战阻碍了这些技术的推进。能够处理电动汽车和高碳变速车的零排放车辆（ZEV）站通过最大化电站利润和消费者效用，同时最小化电网影响来应对这些挑战。本研究旨在开发一个能够通过强化学习（RL）实现实时定价决策的ZEV站真实模拟，并将RL代理的性能与静态定价方法进行比较。培训后，RL代理比基础手动参数搜索获得10.83%的空间站利润。这项工作为ZEV站点所有者在处理复杂定价和采购决策时提供了有用工具。

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

基于偏好的强化学习中的理性引导人类奖励建模

• Authors: Anshul Nayak, Shahil Shaik, Jonathan Smereka, Yue Wang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23550
• Pdf link: https://arxiv.org/pdf/2610.23550
• Abstract Reinforcement learning (RL) policies optimized for task rewards may produce behaviors that are efficient but misaligned with human expectations, safety preferences, or intent. This paper presents a reason-guided preference-based reinforcement learning (PbRL) framework that learns a human reward model from trajectory-level feedback. In addition to preference labels, the framework incorporates reason tags explaining why a behavior is preferred, enabling the learned reward to capture both human choices and their underlying rationale. Compared with preference-only reward learning, the reason-guided formulation provides additional supervision for learning human-aligned rewards and improves the interpretability of the resulting policy. The learned human reward is combined with the environment reward during policy optimization, allowing the agent to improve task performance while producing more interpretable, human- aligned, and preference-sensitive behavior
• 中文摘要 针对任务奖励优化的强化学习（RL）策略可能产生高效但与人类期望、安全偏好或意图不符的行为。本文提出了一个基于理由的偏好强化学习（PbRL）框架，该框架通过轨迹级反馈学习人类奖励模型。除了偏好标签外，该框架还包含解释为何某行为被优先考虑的理由标签，使学习奖励能够同时捕捉人类选择及其背后的理由。与仅偏好奖励学习相比，理性引导表述为学习人类对齐奖励提供了额外的监督，并提升了最终策略的可解释性。在策略优化过程中，学习到的人类奖励与环境奖励结合，使智能体在产生更具可解释性、人类对齐且偏好敏感的行为的同时，提升任务表现

RL-Guided Temporal Localization for Dual-Channel Retrieval in Long-Horizon Agent Memory

远程学习引导的时间定位，用于长视野代理内存中的双通道检索

• Authors: Dabin Sheng, Zhe Wu, Jinming Zhao, Junliang Xing, Yuanchun Shi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.23598
• Pdf link: https://arxiv.org/pdf/2610.23598
• Abstract Time-aware retrieval is crucial for long-horizon LLM agents to effectively leverage ever-growing long-term memory. Existing methods focus on lexical or semantic similarity, leading to target memories being crowded out by temporally mismatched yet semantically similar candidates. Even time-aware approaches inject temporal cues only during memory organization or answer generation, overlooking temporal constraints at the retrieval stage, where candidate selection occurs - the root cause of the displacement. Inspired by the temporal contiguity effect in human episodic memory, we propose TIDER (Temporal Interval-driven Dual-channel Evidence Retrieval), a time-aware agentic retrieval framework. Central to TIDER is a reinforcement learning (RL)-trained temporal interval locator that infers the temporal windows in which the evidence lies. RL optimizes window localization as a discrete, non-differentiable retrieval decision, using composite rewards to convert sparse answer supervision into fine-grained retrieval feedback. These temporal windows first define the temporal retrieval channel; the retrieved candidates are then merged with results from the semantic channel and reranked by proximity to the windows. This temporal grounding enables TIDER to retrieve temporally aligned evidence that would otherwise be drowned out by the volume of semantically similar yet temporally mismatched memories. With a lightweight 1.7B locator, TIDER achieves state-of-the-art accuracy on both long-horizon memory benchmarks - 61.52% on LifeBench (+8.82%) and 78.96% on LoCoMo (+3.66%) - which indicates that temporal cues are crucial for pinpointing target evidence amid a vast pool of semantically similar historical events in long-term daily-life memory.
• 中文摘要 时间感知检索对于长视野LLM代理有效利用不断增长的长期记忆至关重要。现有方法关注词汇或语义相似性，导致目标记忆被时间不匹配但语义相似的候选记忆挤压。即使是时间感知方法，也仅在记忆组织或回答生成时注入时间线索，忽视检索阶段的时间限制，候选选择发生——这正是位移的根本原因。受人类情节记忆中时间连续效应的启发，我们提出了TIDER（时间区间驱动双通道证据检索），这是一种时间感知的代理检索框架。TIDER的核心是一个强化学习（RL）训练的时间区间定位器，推断证据所在的时间窗口。强化学习将窗口定位优化为离散、不可微分的检索决策，利用复合奖励将稀疏的回答监督转化为细粒度的检索反馈。这些时间窗口首先定义了时间检索通道;检索到的候选窗口随后与语义通道的结果合并，并根据与窗口的接近程度重新排序。这种时间基础使TIDER能够检索时间对齐的证据，否则这些证据会被大量语义相似但时间不匹配的记忆淹没。凭借轻量级17亿定位器，TIDER在两个长视野记忆基准上都达到了最先进的准确率——LifeBench上的61.52%（+8.82%）和LoCoMo的78.96%（+3.66%）——这表明时间线索对于在大量语义相似的历史事件中精准定位目标证据至关重要。

JAXtari: High-Throughput and Easy-to-Modify Arcade Learning Environment

JAXtari：高通量且易于修改的街机学习环境

• Authors: Quentin Delfosse, Raban Emunds, Paul Seitz, Sebastian Wette, Jannis Blüml, Daniel Kirn, Dominik Mandok, Kristian Kersting
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.23650
• Pdf link: https://arxiv.org/pdf/2610.23650
• Abstract The Arcade Learning Environment (ALE) has been a cornerstone of Deep Reinforcement Learning research, providing a standardized benchmark for evaluating general agents. However, its lack of native support for object-extraction and environment modification restricts research into evaluating whether agents learn robust causal models or rely on brittle visual heuristics. Furthermore, the ALE's dependence on CPU-based emulation creates a significant bottleneck in modern GPU-accelerated training pipelines, limiting both experimental scale and iteration frequency. In this work, we introduce JAXtari, an open-source, high-performance reimplementation of Atari 2600 environments. By leveraging massive GPU parallelization, JAXtari allows PPO agents to finish training on 100M environment steps on a single A100-GPU in under an hour. Beyond speed, it provides object-centric observations and a flexible modification system, allowing researchers to, e.g. develop neuro-symbolic methods or discover novel shortcuts and subsequently train and adapt their agents rapidly, sometimes in only minutes.
• 中文摘要 街机学习环境（ALE）一直是深度强化学习研究的基石，为评估通用代理提供了标准化基准。然而，其缺乏原生对象提取和环境修改支持，限制了评估代理是学习稳健因果模型还是依赖脆弱视觉启发式的研究。此外，ALE对基于CPU的仿真依赖在现代GPU加速训练流程中造成重大瓶颈，限制了实验规模和迭代频率。本研究介绍了JAXtari，一款开源高性能的Atari 2600环境重实现。通过大规模GPU并行化，JAXtari使PPO代理能够在一小时内完成单个A100-GPU上1亿环境步的训练。除了速度，它还提供以对象为中心的观察和灵活的修改系统，使研究人员能够开发神经符号方法或发现新捷径，进而快速训练和适应代理，有时甚至仅需几分钟。

Not All Actions Contribute Equally: Learning Precise Progress Estimation from Imprecise Demonstrations

并非所有行动的贡献都相同：从不精确的演示中学习精确的进展估计

• Authors: Huanqian Wang, Boshi Tang, Botong Xiao, Jie Zhang, Yang Yue, Danhua Tao, Shiji Song, Gao Huang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.23769
• Pdf link: https://arxiv.org/pdf/2610.23769
• Abstract Progress estimators play an essential role in providing dense reward signals for reinforcement learning in robotic manipulation, but accurately assessing task progress can be much challenging. Existing methods heavily rely on expert data and naively assume that all actions within a demonstration contribute to task success. This assumption often breaks down in real-world scenarios, especially in high-precision manipulation, where subtle geometric errors can lead to task failure. Therefore, these methods can erroneously interpret human mistakes as meaningful progress, further corrupting progress learning. We argue that a key limitation is the lack of a global prior regarding task evaluation criteria, which is crucial for distinguishing meaningful progress from human mistakes. To this end, we propose PRIME, a novel progress estimator capable of learning precise progress estimation from imprecise demonstrations by aggregating experience across trajectories. PRIME constructs a weighted graph over observed states, connecting states across trajectories based on spatial proximity and visual consistency. Edge weights encode spatial distances as transition costs, and shortest-path search approximates the geodesic distance to success. Changes in this distance naturally distinguish advancement, stagnation, and setbacks without assuming monotonic progress over time. Extensive experiments in both simulation and real world on precise manipulation tasks show that PRIME is consistently effective across diverse tasks, and robust to data with varying degrees of suboptimality. Notably, the advantage of PRIME gets more pronounced in higher-precision scenarios, yielding up to a (5\times) improvement in policy success rate over the SFT baselines on charger insertion.
• 中文摘要 进度估计器在机器人操作中为强化学习提供密集的奖励信号中起着关键作用，但准确评估任务进展可能非常具有挑战性。现有方法高度依赖专家数据，并天真地假设演示中的所有动作都有助于任务成功。这一假设在现实场景中常常被推翻，尤其是在高精度操作中，细微的几何误差可能导致任务失败。因此，这些方法可能错误地将人类错误解读为有意义的进步，进一步破坏进展学习。我们认为一个关键局限是缺乏关于任务评估标准的全局先验，这对于区分有意义的进步与人类错误至关重要。为此，我们提出了PRIME这一新型进展估计器，能够通过汇总不同轨迹的经验，从不精确的演示中学习精确的进展估计。PRIME构建基于观测状态的加权图，基于空间接近度和视觉一致性连接轨迹中的状态。边权重编码空间距离作为转移成本，最短路径搜索近似成功测地线距离。该距离的变化自然区分进展、停滞和挫折，而不假设单调进展。在模拟和真实世界中对精确操作任务的大量实验表明，PRIME在不同任务中始终有效，且对不同次优程度的数据保持鲁棒性。值得注意的是，PRIME的优势在高精度场景中更加明显，使得在充能器插入时的策略成功率比SFT基线提升多达5倍。

CocoRNA: Collective RNA Design with Cooperative Multi-agent Reinforcement Learning

CocoRNA：集体RNA设计与合作多主体强化学习

• Authors: Tianmeng Hu, Biao Luo, Ke Li
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.23820
• Pdf link: https://arxiv.org/pdf/2610.23820
• Abstract Designing RNA sequences that reliably fold into specific secondary structures is essential for understanding their biological functions but remains a challenging computational problem. We propose CocoRNA, a cooperative multi-agent reinforcement learning framework for RNA inverse design. CocoRNA simplifies the design task by decomposing it into smaller sub-problems, each solved collaboratively by multiple agents. This approach reduces the complexity of the problem and improves the exploration of design policies. During training, a centralized critic uses global structural information to guide the agents, enabling them to jointly optimize their design strategies. As a result, CocoRNA learns high-quality RNA design policies that generalize effectively to unseen structures without additional training. Experiments on the Rfam dataset demonstrate that CocoRNA substantially outperforms state-of-the-art methods in both success rate and design speed. Further experiments on other biological sequence design tasks highlight the effectiveness and broad potential of CocoRNA for complex design tasks.
• 中文摘要 设计能够可靠折叠成特定二级结构的RNA序列对于理解其生物功能至关重要，但仍是一个具有挑战性的计算难题。我们提出了CocoRNA，一种用于RNA逆向设计的合作多代理强化学习框架。CocoRNA通过将设计任务分解为更小的子问题，每个子问题由多个智能体协作解决，从而简化了设计任务。这种方法降低了问题的复杂性，并提升了设计策略的探索。在培训过程中，集中式评审员利用全球结构信息指导代理，使他们能够共同优化设计策略。因此，CocoRNA能够学习高质量的RNA设计策略，能够有效泛化到未见结构，无需额外训练。在Rfam数据集上的实验表明，CocoRNA在成功率和设计速度上都远超最先进方法。其他生物序列设计任务的进一步实验凸显了CocoRNA在复杂设计任务中的有效性和广泛潜力。

Blind Precise Assembly: Proprioceptive Sim-to-Real Peg-in-Hole for Diverse Parts

盲目精确组装：本体感觉模拟到真实的多部件接口

• Authors: Antonia Bronars, Younghyo Park, Kushal Kedia, Pulkit Agrawal
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.23849
• Pdf link: https://arxiv.org/pdf/2610.23849
• Abstract Every robot already carries high-resolution contact sensors: its own joints. Legged robots and humanoids exploit them to walk, run, and track whole-body motions without vision, but there the contacts are large and forceful. Is joint sensing also enough for precise, contact-rich assembly, where the robot must find a millimeter-scale opening, mesh gear teeth, or catch a thread through light contact? We study this question by training blind assembly policies that observe only a short history of joint positions, velocities, and their own actions, without visual, force, torque, or tactile sensing. The poses of both the fixture and the part are unknown to the policy, and the grasp is randomized substantially. The policy must therefore assemble the parts through contact, sensed exclusively through its own proprioception. Trained with reinforcement learning entirely in simulation over randomized parts, grasps, and contacts, such policies learn peg insertion, gear meshing, and nut threading. The peg insertion policy transfers zero-shot to the real world and inserts pegs of different shapes and sizes in 73% of trials. Controlled comparisons with matched visual policies in simulation show that blind policies generalize far better to unseen part shapes and sizes, though they can be slower. Training such policies needs thousands of different assemblies, each simulated with its clearance intact, so we also release an open-source generator that builds them procedurally from exact convex pieces for MuJoCo; it is more accurate and faster than convex decomposition or MuJoCo's signed-distance collision. Videos and interactive simulations are at this https URL.
• 中文摘要 每个机器人都已携带高分辨率接触传感器：即自身的关节。有腿机器人和类人机器人利用这些传感器行走、奔跑和追踪全身运动而无需视觉，但接触点大且有力。关节感知是否也足以实现精确、接触丰富的组装，机器人必须找到毫米级开口、啮合齿轮齿或通过光线接触抓住线？我们通过训练盲装策略来研究这个问题，这些策略仅观察关节位置、速度和自身动作的短暂历史，没有视觉、力、力或触觉感知。夹具和零件的姿态对政策来说都是未知的，抓握方式在很大程度上是随机的。因此，策略必须通过接触组装零件，这种接触完全通过自身本体感觉感知。通过完全在模拟中对随机零件、抓把和接触进行强化学习训练，这些策略学习了销钉插入、齿轮啮合和螺母螺纹。销钉插入策略将零点量转移到现实世界，并在73%的试验中插入不同形状和尺寸的销钉。与模拟中匹配的视觉策略进行受控比较显示，盲策略对未见零件形状和尺寸的泛化效果更好，尽管速度可能较慢。训练此类策略需要数千个不同的组件，每个组件都保持其间隙完整，因此我们还发布了一个开源生成器，可以从精确凸部件程序化构建它们，供MuJoCo使用;它比凸分解或MuJoCo的签名距离碰撞更准确、更快。视频和交互式仿真请访问此 https 网址。

Keyword: diffusion policy

There is no result