生成时间: 2026-07-29 18:41:44 (UTC+8); Arxiv 发布时间: 2026-07-29 20:00 EDT (2026-07-30 08:00 UTC+8)

今天共有 28 篇相关文章

Keyword: reinforcement learning

HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising

HOBA：自适应在线广告的层级政策内竞标代理

• Authors: Ji Wu, Yunshan Peng, Wentao Bai, Yunke Bai, Wenzheng Shu, Jinan Pang, Yanxiang Zeng, Xialong Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.24779
• Pdf link: https://arxiv.org/pdf/2607.24779
• Abstract Online advertising bidding systems typically deploy multiple offline-trained expert models (e.g., PID controllers, model predictive control, offline RL policies) but face two critical limitations: lack of online adaptability to non-stationary auction markets, and reliance on costly manual tuning of hyperparameters such as bid bounds and budget pacing constraints. We propose HOBA (Hierarchical On-policy Bidding Agents), a hierarchical reinforcement learning framework that decouples strategic reasoning, model selection, and bid execution across three time scales. At the high level, a large language model infers hyperparameters from contextual signals through a Think-Act-Observe-Reflect loop with historical experience retrieval. At the mid level, a SARSA agent dynamically selects among expert models, incorporating causal adjustment to eliminate selection bias. At the low level, a dynamic expert pool (PID, MPC, IQL, Decision Transformer) executes bids under high-level constraints. This design confines online learning to discrete expert selection rather than continuous bid optimization, significantly reducing exploration risk while maintaining adaptability. Experiments on the AuctionNet benchmark and a large-scale A/B test demonstrate consistent improvements over state-of-the-art baselines. In a large-scale online deployment, HOBA delivered substantial business value, achieving a +3.6\% increase in target cost, proving the effectiveness of our hierarchical multi-agent bidding paradigm.
• 中文摘要 在线广告竞价系统通常部署多个离线训练的专家模型（如PID控制器、模型预测控制、离线强化学习策略），但面临两个关键限制：缺乏对非固定拍卖市场的在线适应能力，以及依赖昂贵的人工调优超参数，如竞价界限和预算节奏约束。我们提出了HOBA（分层策略竞标代理），这是一个分层强化学习框架，将战略推理、模型选择和叫牌执行在三个时间尺度上解耦。在高层次上，大型语言模型通过“思考-行动-观察-反思”循环通过上下文信号推断超参数，并结合历史经验检索。在中层，SARSA代理在专家模型中动态选择，结合因果调整以消除选择偏差。在低层级，动态专家池（PID、MPC、IQL、Decision Transformer）在高层约束下执行竞标。这种设计将在线学习限制在离散的专家选择中，而非持续的竞标优化，显著降低了探索风险，同时保持了适应性。在AuctionNet基准测试和大规模A/B测试上的实验显示，这些测试相较于最先进的基线数据持续有显著提升。在大规模线上部署中，HOBA带来了显著的商业价值，目标成本增长+3.6%，证明了我们分层多代理竞价范式的有效性。

NEXT: Reasoning-Driven Video Recommendation via a Vision-Language Model

下一集：通过视觉语言模型进行推理驱动的视频推荐

• Authors: Yuming Liu, Hongye Yang, Harrison Zhao, Ellie Zhu, Bokai Cao, Lei Huang, Lizhu Zhang, Xiangjun Fan
• Subjects: Subjects: Information Retrieval (cs.IR); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2607.24789
• Pdf link: https://arxiv.org/pdf/2607.24789
• Abstract We present NEXT (Next-interest EXploration Transformer), a reasoning-driven video recommendation framework that reasons over the video a user has just watched, infers the viewer's next intent, and retrieves concrete follow-up videos. Explicit continuations such as episodes are linked directly; implicit cases are handled by generating intent queries and searching for matching candidates. This Item-to-Intent-to-Item formulation produces directed recommendations beyond co-engagement correlation or semantic similarity. To make this framework reliable at scale, we train NEXT-8B, a purpose-trained 8B vision-language model with a three-stage recipe: Perception-Enhanced Reinforcement Learning for query-agnostic evidence extraction, Distribution-Aligned Supervised Fine-Tuning over real and synthetic visual QA mixtures, and Group Relative Policy Optimization for last-mile alignment. NEXT-8B achieves the best single-model DocVQA performance, ranking second overall only behind a multi-agent system while surpassing a substantially larger 200B+ scale model, and improves next-intent logic-wise quality by 3.3% over the base model in a task-specific LLM-as-a-judge evaluation. We deploy NEXT as an additional retrieval path in a large-scale social media recommendation system and observe statistically significant production gains, including +0.53% watch time and +0.51% distinct video exposure. Overall, NEXT shows that a carefully trained compact vision-language model can serve as a practical reasoning engine for next-interest exploration at production scale.
• 中文摘要 我们介绍NEXT（Next-interest EXploration Transformer），这是一个基于推理的视频推荐框架，能够基于用户刚观看的视频进行推理，推断观众的下一步意图，并提取具体的后续视频。明确的续集如剧集会直接关联;隐式情况通过生成意图查询并搜索匹配候选词来处理。该项目对意图项目的表述产生了超越共参与相关性或语义相似性的有方向性建议。为了使该框架在大规模上可靠，我们训练了NEXT-8B，一个专门训练的8B视觉语言模型，采用三阶段方案：感知增强强化学习用于查询无关证据提取，分布对齐监督微调，应用于真实和合成视觉质量保证混合，以及组相对策略优化，进行最后一公里对齐。NEXT-8B在单模型中实现了最佳的DocVQA表现，整体排名仅次于多智能体系统，同时超过了规模明显更大的200B+规模模型，并且在任务特定LLM作为评判的评估中，其逻辑质量比基础模型提升了3.3%。我们将NEXT作为大型社交媒体推荐系统的额外检索路径部署，观察到统计上显著的制作增长，包括观看时长+0.53%和独立视频曝光+0.51%。总体而言，NEXT表明，经过严格训练的紧凑视觉语言模型可以作为生产规模下next兴趣探索的实用推理引擎。

AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning

AdaKP：在线自适应知识点选择，用于推理导向强化学习

• Authors: Zibin Meng, Zhenyu Zhao, Chunqiang Run
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.24833
• Pdf link: https://arxiv.org/pdf/2607.24833
• Abstract Reinforcement learning with verifiable rewards is a powerful paradigm for eliciting reasoning in large language models, yet it suffers from severe reward sparsity on competition-level mathematics. A common remedy injects atomic knowledge points (KPs) - short natural-language hints distilled from gold solutions - into the prompt. Existing methods, however, either fix this selection once offline or merely scale the monolithic quantity of injected text, leaving untouched the most informative axis of choice: which subset of atomic KPs to inject, and when. We introduce AdaKP, an online selector that re-chooses each problem's KP subset over the course of RL training. At its core is an entropy proxy that scores a KP by the reduction in next-token entropy it induces - a single inexpensive forward pass, with a provable bound on its truncation bias - in place of expensive rollout-based estimation. Three lightweight mechanisms make this signal usable online: a momentum smoother that absorbs per-step noise, a retirement-and-revival manager that prunes weak KPs while preserving exploration, and an adaptive scheduler that front-loads re-evaluations into early training. AdaKP further contributes a pre-flight validation gate that certifies the proxy against a leave-one-out ground truth before any expensive run is launched, turning method-level risk into a falsifiable check. Realized as a fully additive fork of a standard DAPO+GRPO trainer with no optimizer changes, AdaKP improves over a strong static-selection baseline on all eight competition-mathematics benchmarks at negligible added cost, positioning online, validated KP-subset selection as a practical and as-yet under-explored axis for reasoning-oriented reinforcement learning.
• 中文摘要 带有可验证奖励的强化学习是大型语言模型中引发推理的强大范式，但在竞争级数学中却存在严重的奖励稀疏性。一种常见的解决方案是将原子知识点（KPs）——从金色溶液中提炼出来的简短自然语言提示——注入提示。然而，现有方法要么在离线后固定该选择，要么仅对注入文本的庞大数量进行扩展，保持最有信息量的选择轴：注入哪个原子KP子集，以及何时注入。我们引入了AdaKP，一个在线选择器，在强化学习训练过程中重新选择每个问题的KP子集。其核心是一个熵代理，通过减少下一代币熵来得分KP——一次廉价的前向传递，且其截断偏差有可证明的界限——取代昂贵的基于推广的估计。三种轻量化机制使该信号在线使用：一种吸收每步噪声的动量平滑器;一种在保留探索的同时修剪弱的KP的退休与复苏管理器;以及一种将重新评估提前加载到早期训练中的自适应调度器。AdaKP还贡献了一个预检验证门，在启动任何昂贵运行前，将代理基于一个未排除的真实情况认证，将方法层级风险转化为可证伪的检查。作为标准DAPO+GRPO训练器的全加法分支，未做优化器更改，AdaKP在八项竞赛数学基准测试中基于强静态选择基线，且成本极低，在线定位，验证KP子集选择作为推理导向强化学习的实用且尚未充分探索的轴。

SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task

SearchArt：用可扩展的合成和验证任务训练长视野搜索代理

• Authors: Lang Mei, Xiaohan Yu, Chong Chen, Liyan Liu, Xiangnan Chen, Jinchao Ma, Chao Feng, Li Huang, Siyu Mo, Sichen Kang, Yunkun Xu, Zhihan Yang, Zhujun Xue, Jingren Zhang, Qing He, Yingdi Huang, Hao Jiang, Ziao Ma, Zewei Pan, Minhao Sun, Zhuo Tao, Jinzhao Xiao, Gangtao Xin, Huanyao Zhang, Wenjian Zhang, Jiangshan Zhang, Guojie Zhu, Jiaxin Mao, Wentao Zhang
• Subjects: Subjects: Information Retrieval (cs.IR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.24850
• Pdf link: https://arxiv.org/pdf/2607.24850
• Abstract Recent advances in large language models (LLMs) have enabled search agents to autonomously tackle complex tasks across extended search and reasoning horizons. However, training effective search agents remains challenging due to the lack of scalable and long-horizon tasks, and the difficulty of evaluating and correcting intermediate reasoning and tool-use behaviors. We introduce SearchArt, a scalable framework for training long-horizon search agents through verification-driven task synthesis and a multi-stage post-training pipeline. SearchArt constructs large-scale datasets for complex search-, research- and user-oriented tasks by synthesizing diverse information-seeking QA pairs and corresponding search trajectories from web documents and automatically generated evidence graphs. To ensure the reliability of the synthesized data, we design a verification pipeline that jointly evaluates QA consistency, trajectory quality, and the relevance of retrieved evidence. The verified trajectories are subsequently used in a multi-stage training process comprising supervised fine-tuning and reinforcement learning-based policy optimization. Search agents trained with SearchArt exhibit adaptive search planning, iterative evidence aggregation, and complex reasoning over extended interaction horizons. Experimental results demonstrate that, with only (Qwen3.5-) 27B parameters, SearchArt scores 74.39 on BrowseComp-ZH, 70.06 on BrowseComp, and 52.55 on Deepresearch-bench, matching or surpassing frontier closed-source agents on both deepsearch and deepresearch benchmarks.
• 中文摘要 大型语言模型（LLM）的最新进展使搜索代理能够自主处理跨越更广泛搜索和推理视野的复杂任务。然而，由于缺乏可扩展且长期的任务，以及评估和纠正中间推理和工具使用行为的困难，培训有效的搜索代理仍然具有挑战性。我们介绍了SearchArt，一个可扩展的框架，通过验证驱动的任务综合和多阶段训练后流程，用于训练长视野搜索代理。SearchArt通过综合多种信息寻求QA对及相应的搜索轨迹，构建针对复杂搜索、研究和用户任务的大规模数据集，并自动生成证据图。为确保综合数据的可靠性，我们设计了一个验证流程，共同评估质量保证一致性、轨迹质量及检索证据的相关性。验证后的轨迹随后被用于多阶段训练过程，包括监督式微调和基于强化学习的策略优化。使用SearchArt训练的搜索代理展现出自适应搜索规划、迭代证据聚合以及在漫长交互视野内的复杂推理能力。实验结果显示，仅以（Qwen3.5-）27B参数，SearchArt在BrowseComp-ZZ得分为74.39，BrowseComp为70.06，Deepresearch-bench为52.55，在deepsearch和deepresearch基准测试中均与前沿闭源代理持平甚至超越。

Egocentric Station Holding of Robotic Fish in Unknown Turbulent Background Flow

在未知湍流背景流中自我中心的机械鱼站停留

• Authors: Xiaozhu Lin, Xu Huang, Hongru Dai, Xiaopei Liu, Junzhi Yu, Yang Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.24860
• Pdf link: https://arxiv.org/pdf/2607.24860
• Abstract Approaching a target position and holding station in flowing water is a fundamental and critical capability for robotic fish operating in natural aquatic environments. Despite decades of advances in enhancing swimming efficiency and maneuverability, this capability remains underdeveloped, largely owing to the insufficiently characterized, highly nonlinear fluid-structure interactions inherent to freely swimming robotic fish in flows. To bridge this gap, we propose the SWiFT framework, a Swimming With Flow Toolbox that enables the efficient exploration of an egocentric station-holding policy for a body and/or caudal fin (BCF) robotic fish in unknown and turbulent background flows via reinforcement learning (RL). Our SWiFT integrates a free-swimming flow-tank experimental setup with a highly efficient, physically consistent computational fluid dynamics (CFD)-based simulator and a systematic sim-to-real transfer pipeline. The resulting policy achieves substantial improvements over state-of-the-art methods across all metrics, most notably root-mean-square error (RMSE) of distance. Furthermore, we validated that egocentric feedback alone, without any explicit flow sensing, enables station-holding in unknown turbulent flows, closely mirroring the biological phenomenon of rheotaxis. Accordingly, the success of this egocentric station-holding policy not only advances robotic fish control toward real-world deployment, but also highlights SWiFT's promise as a foundation for tackling complex swimming tasks for underwater robots.
• 中文摘要 在流动水中接近目标位置并保持岗位，是机械鱼在自然水域环境中操作的基本且关键能力。尽管在提升游泳效率和机动性方面已有数十年进展，但这一能力仍然未完全发展，主要原因是自由游动的机械鱼在流动中固有的流体-结构相互作用特性不充分且高度非线性。为弥合这一空白，我们提出了SWiFT框架，即一个“随流游泳工具箱”，通过强化学习（RL）高效探索身体和/或尾鳍（BCF）机器人鱼在未知且湍流背景流中的自我中心站位持有策略。我们的SWiFT集成了自由流动流槽实验装置、高效且物理一致的计算流体力学（CFD）模拟器和系统化的模拟到实传输流水线。该政策在所有度量上都相较于最先进方法取得了显著改进，尤其是距离均方根误差（RMSE）。此外，我们验证了仅靠自我反馈，无需显式流传感，就能在未知湍流中保持站位，这与流变现象的生物学现象非常相似。因此，这一自我中心的空间站控制政策的成功不仅推动了机器人鱼类控制的实际应用，也凸显了SWiFT作为解决水下机器人复杂游泳任务基础的潜力。

Inverse RL Helps Align AI by Imitating Humans

逆强化学习通过模仿人类来帮助对齐人工智能

• Authors: Michał Wiliński, Liu Leqi, Chirag Nagpal
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.24900
• Pdf link: https://arxiv.org/pdf/2607.24900
• Abstract Language model alignment aims to make model behavior reliably reflect desirable properties such as helpfulness, safety, and instruction following. Current approaches typically use supervised fine-tuning on demonstrations or reinforcement learning with rewards derived from verifiers or human feedback. These paradigms leave an important question underexplored: can demonstrations alone yield an implicit reward that can be inspected, reused, and optimized on-policy to align AI? Motivated by inverse reinforcement learning, we introduce Projected Alignment Reward Estimated from Demonstrations (PARED). PARED recovers the implicit reward underlying expert demonstrations as an explicit function over a small set of response-level features, learned by a lightweight discriminator that separates demonstrations from the policy's own samples in this feature space. Unlike a standard reward model, PARED requires no task-specific preference annotations: demonstrations provide the task-specific supervision, which can be augmented with AI feedback as additional dimensions of supervision. Through experiments involving inference-time reranking and adversarial on-policy RL, we show that the recovered reward improves a base policy without a supervised loss and yields further gains when optimized after standard supervised fine-tuning. Additionally, we demonstrate that PARED can be used for contextual alignment, in which a single policy can be tailored to the preferences of different audiences.
• 中文摘要 语言模型对齐旨在使模型行为可靠反映诸如帮助性、安全性和指令遵循性等理想属性。当前方法通常在演示或强化学习中使用监督微调，奖励来自验证者或人类反馈。这些范式留下了一个重要问题尚未被深入探讨：仅靠演示是否能产生隐含奖励，并可被检查、重复使用并优化政策以使人工智能对齐？在逆强化学习的激励下，我们引入了“演示预测对齐奖励估计”（PARED）。PARED 恢复了专家演示背后的隐性奖励，作为对一小部分响应级特征的显式函数，这些函数由轻量级判别器学习，该判别器将演示与策略自身样本在该特征空间中分离出来。与标准奖励模型不同，PARED不需要针对任务的特定偏好注释：演示提供任务特定的监督，并可通过AI反馈作为额外的监督维度进行补充。通过涉及推理时间重新排序和对抗性策略强化学习的实验，我们表明回收的奖励在无监督损失的情况下改进基础策略，并在标准监督微调后优化后获得进一步收益。此外，我们展示了PARED可用于情境对齐，即单一政策可以根据不同受众的偏好进行定制。

Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

校准部分重置：防止持续强化学习中的策略崩溃

• Authors: Luc McCutcheon, Evangelos Chatzaroulas, Saber Fallah
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.24996
• Pdf link: https://arxiv.org/pdf/2607.24996
• Abstract Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning. Recently, neuron resets have been used to maintain gradient flow and restore plasticity. However, full unit reinitialization often sacrifices peak performance and can destabilize training, leading to policy collapse. To preserve plasticity without destabilizing training, we propose Calibrated Partial Resets (CPR), an optimizer that periodically pulls low-utility neurons toward their initialization, with pull strength scaled by each neuron's utility. Unlike binary reset methods, partial resets avoid brittleness; unlike uniform decay, calibrated utility-scaling concentrates adjustment on the units that need it most. Among compared methods, only CPR avoids policy collapse over 400M training steps in SlipperyAnt, and it outperforms prior decay and reset-based methods on Continual MetaWorld and Continual MinAtar benchmarks. Ablations reveal a tunable trade-off between plasticity and peak performance, highlighting utility-scaled reinitialization as a promising direction for continual learning.
• 中文摘要 神经网络在训练过程中会受到休眠神经元积累和表达力丧失的阻碍，尤其是在非平稳数据环境中，如持续监督学习和强化学习。近年来，神经元重置被用来维持梯度流动并恢复可塑性。然而，全单位重新初始化往往会牺牲巅峰表现，并可能破坏训练稳定，导致政策崩溃。为了保持可塑性而不破坏训练，我们提出了校准部分重置（CPR）的优化器，它定期拉动低效用神经元朝初始化方向拉动，拉力强度根据每个神经元的效用进行调整。与二进制复位方法不同，部分复位避免了脆性;与均匀衰减不同，校准效用尺度将调整集中在最需要调整的单元上。在比较方法中，只有CPR在SlipperyAnt中超过4亿次训练步骤避免策略崩溃，并且在连续元世界和连续MinAtar基准测试中优于以往基于衰减和重置的方法。消融揭示了可塑性与峰值性能之间的可调节权衡，凸显了效用尺度的重新初始化作为持续学习的有前景方向。

Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT

类似模型学习方式不同：最终窗口预训练形状 超越SFT训练后

• Authors: Cen Lu, Yung-Chen Tang, Andrea Cavallaro
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.25063
• Pdf link: https://arxiv.org/pdf/2607.25063
• Abstract Developers judge a model checkpoint by how it behaves. After supervised fine-tuning (SFT), two checkpoints that perform about the same across relevant benchmarks are treated as interchangeable, equally ready for the next alignment stage, typically preference optimization. We ask whether this judgment misses a pretraining imprint: a difference that no post-SFT benchmark reveals, yet that decides how each checkpoint responds to further training. To find out, we run a controlled experiment on the final window of pretraining, the last data trained on before instruction tuning. Six branches fork from one partially pretrained checkpoint and differ only in this window: 500 million tokens, 0.1% to 1% of the tokens that precede it. Each branch trains its window on a single data source: generic web text, filtered web text, normative discourse, safety text, mathematical text, or synthetic educational text. SFT and post-training are then identical. After SFT the branches behave near-identically, within about one point on instruction following, refusal, and capability, yet the same post-training carries them to very different endpoints, under both a direct preference optimization update and a reinforcement learning update with a verifiable reward. We measure this deviation through refusal of harmful requests: when post-training begins the safety text branch refuses no more than the web text branch, yet by the end it has lost far less of its refusal. The other four branches gain little or no protection, so the effect is selective to what the window contained. The protection requires the safety text to arrive last rather than earlier in pretraining, and it reproduces on a second model family. What a model is pretrained on last shapes how it reacts to alignment. Therefore, a checkpoint should not be evaluated by its post-SFT behavior alone, and what it was trained on last should be reported with it.
• 中文摘要 开发者通过模型检查点的行为来判断它。经过监督微调（SFT），两个在相关基准测试中表现大致相同的检查点被视为可互换，同样准备好进入下一阶段的比对阶段，通常是偏好优化。我们会问，这种判断是否遗漏了预训练印记：这是SFT后任何基准都未揭示的差异，但这决定了每个检查点如何应对后续训练。为此，我们对预训练的最后一个窗口进行了受控实验，即指令调优前最后一次训练的数据。六个分支从一个部分预训练的检查点分支，仅在这个窗口内不同：5亿个令牌，占前置令牌的0.1%到1%。每个分支的窗口针对单一数据源进行训练：通用网络文本、过滤网页文本、规范性话语、安全文本、数学文本或综合教育文本。SFT和培训后培训是相同的。SFT之后，这些分支在指令跟随、拒绝和能力上表现几乎相同，但在相同的后期训练中，它们会到达截然不同的终点，无论是直接的偏好优化更新，还是带有可验证奖励的强化学习更新。我们通过拒绝有害请求来衡量这种偏差：培训后，安全文本分支拒绝的程度与网页文本分支差不多，但到结束时，它失去的拒绝能力远小于此。其他四个分支几乎没有保护，因此效果对窗口内容有选择性。该保护要求安全文本在预训练中最后出现，而非更早，并且在第二系列车型上得以复制。模型预训练的最后形状，以及它对对比齐的反应。因此，检查点不应仅凭其SFT后的行为来评估，应将其最后训练的内容与检查点一同报告。

PLATO: Pointer Learner for Agent and Task Openness

柏拉图：代理与任务开放性的指针学习器

• Authors: Alireza Saleh Abadi, Leen-Kiat Soh, Daniel Alan Redder, Adam Eck, Prashant Doshi
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2607.25082
• Pdf link: https://arxiv.org/pdf/2607.25082
• Abstract Open agent systems (OASYS) are increasingly prevalent in real-world domains where the sets of agents and tasks change unpredictably over time. Such openness, including agent openness (AO) and task openness (TO), poses a fundamental challenge to multi-agent reinforcement learning (MARL), which typically assumes fixed state and action spaces. Existing methods address openness only partially: padding and masking approaches introduce artificial bounds, while recent graph-based or hypergraph methods handle one dimension of openness but still depend on restrictive assumptions. In this paper, we introduce Pointer Learner for Agent and Task Openness (PLATO), a pointer-network-based actor combined with a centralized graph neural network (GNN) critic, trained with multi-agent proximal policy optimization under a centralized training and decentralized execution paradigm. Our pointer-based actor outputs distributions directly over the current task set. This directly supports changing action spaces without masking or retraining. Our GNN critic encodes agent-task interactions as a graph that changes shape with task and agent composition. Together, these components consider AO and TO without the boundedness of existing approaches. We formalize PLATO in a Task-and-Agent-Open Markov Game (TaAgO-MG), extending prior task-open formulations, and prove it is well-defined over the resulting unbounded state and action spaces. We evaluate PLATO with the Methods for Open Agent Systems Evaluation Initiative (MOASEI) wildfire suppression domain, an environment designed for open multi-agent system evaluation, and we demonstrate strong performance and more consistent zero-shot generalization than state-of-the-art baselines in OASYS.
• 中文摘要 开放代理系统（OASYS）在现实世界中越来越普遍，因为代理和任务的集合随时间不可预测地变化。这种开放性，包括代理开放性（AO）和任务开放性（TO），对多智能体强化学习（MARL）构成了根本性挑战，后者通常假设固定状态和动作空间。现有方法仅部分解决开放性：填充和掩蔽方法引入了人为界限，而最新的基于图或超图的方法处理一个维度的开放性，但仍依赖于限制性假设。本文介绍了Pointer Learner for Agent and Task Openness（PLATO），这是一种基于指针网络的参与者，结合了集中式图神经网络（GNN）critic，采用多代理近端策略优化训练，采用集中训练和去中心化执行范式。我们的指针驱动的演员直接在当前任务集中输出分布。这直接支持在不掩饰或重新训练的情况下改变动作空间。我们的GNN批评者将代理-任务交互编码为一个随着任务和代理组成变化形状的图。这些组成部分共同考虑了AO和TO，但没有现有方法的界限。我们将PLATO形式化为任务与代理开放马尔可夫博弈（TaAgO-MG），扩展了之前的任务开放表述，并证明其在无界状态和动作空间上是良好定义的。我们采用开放代理系统评估倡议（MOASEI）野火抑制领域来评估PLATO，这是一个为开放多代理系统评估设计的环境，我们展示了比OASYS中最先进基线更强的性能和更一致的零样本泛化能力。

Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

迈向小尺度语言模型代理的稳健强化学习

• Authors: Md Rezwanul Haque, Md. Milon Islam, Fakhri Karray
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Optimization and Control (math.OC); Computation (stat.CO)
• Arxiv link: https://arxiv.org/abs/2607.25091
• Pdf link: https://arxiv.org/pdf/2607.25091
• Abstract The alignment of Small Language Models (SLMs) in the 70--500M parameter range using reinforcement learning is often considered unstable, though the underlying failure mechanisms have not been systematically investigated. In the State-of-the-Art (SOTA) research, fifteen (model, corpus) configurations were trained using Proximal Policy Optimization (PPO). The experiments included Pythia-70M, 160M, 410M and SmolLM2-135M, 360M on the TinyStories, CNN/DailyMail, and Wikitext-103 corpora. Three reproducible failure modes were identified in small-scale language models: silent LoRA parameter freezing in standard PEFT/TRL pipelines, numerical overflow in importance ratios when using bfloat16, and catastrophic policy collapse due to reward-model error. These issues were addressed using a merge-and-reinitialize adapter technique, float32 precision during PPO updates, and a three-layer safety mechanism comprising reward whitening, importance-ratio guarding, and weight rollback. In this paper, a capacity-headroom hypothesis is proposed, which states that PPO performance at the SLM scale depends on both a fluent supervised model ($\text{PPL}<20$) and a discriminative reward signal, rather than on the number of model parameters. The proposed system converged stably in all experiments and improved preference win rate over the SFT baseline in configurations with a fluent prior and an informative reward signal. Furthermore, it outperformed instruction-tuned baselines while requiring significantly less training data. All checkpoints, preference datasets, and training scripts are publicly released$^§$.
• 中文摘要 使用强化学习将小型语言模型（SLM）在70-5亿参数范围内的对齐通常被认为是不稳定的，尽管其潜在的失败机制尚未被系统性研究。在最先进（SOTA）研究中，十五种（模型、语料库）配置采用了近端策略优化（PPO）训练。实验包括Pythia-70M、160M、410M和SmolLM2-135M、360M，均见于TinyStories、CNN/DailyMail和Wikitext-103语料库。在小规模语言模型中识别出三种可重复的失败模式：标准PEFT/TRL流水线中LoRA参数静默冻结、使用bfloat16时重要性比率的数值溢出，以及因奖励模型错误导致的灾难性策略崩溃。这些问题通过合并再初始化适配器技术、PPO更新时的float32精度以及包括奖励白化、重要性比保护和权重回滚的三层安全机制得到解决。本文提出了容量-余裕假设，认为SLM尺度下的PPO表现依赖于流态监督模型（$\text{PPL}<20$）和判别性奖励信号，而非模型参数数量。所提系统在所有实验中稳定收敛，并在具有流先验和信息性奖励信号的配置中，优先胜率相较于SFT基线提升。此外，它优于指令调优的基线，且所需的训练数据显著减少。所有检查点、偏好数据集和训练脚本均公开发布$^§$。

Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning

端点重放：深度强化学习中的最新缓冲区压缩

• Authors: Parham Mohammad Panahi, Armin Ashrafi, Haoyu Du, Andrew Patterson, Martha White, Adam White
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.25123
• Pdf link: https://arxiv.org/pdf/2607.25123
• Abstract Experience replay remains one of the most practical and useful algorithmic tools in the deep reinforcement learning (DRL) toolbox. Aside from the limited success of prioritized replay and specialized approaches for large asynchronous systems, most DRL algorithms make use of a large, uniformly sampled recency buffer---even the size, one million, remains unchanged. Could we store less data, reduce redundancy, or more effectively chain experience together to speed up value propagation and still retain the performance of large buffers? In this paper, we investigate a simple compression approach that stores representative transitions derived from the end-points of a chain of connected $n$-step sequences. By curating these end-points in a smaller recency buffer, our method maintains an effective memory horizon comparable to a standard large buffer while requiring an order of magnitude less storage. Through empirical evaluation, we demonstrate that this approach prevents the systematic bias inherent in naive compression strategies and matches the performance of traditional large buffers in the Pinball environment and the Atari 2600 benchmark.
• 中文摘要 体验回放仍然是深度强化学习（DRL）工具箱中最实用且有用的算法工具之一。除了优先重放和大型异步系统专用方法的有限成功外，大多数DRL算法都使用一个大且均匀采样的最近缓冲区---即使大小为一百万也保持不变。我们能否存储更少的数据，减少冗余，或者更有效地将经验串联起来，加快价值传递，同时仍保持大型缓冲区的性能？本文探讨了一种简单的压缩方法，该方法存储由一系列连通$n$步序列的端点推导出的代表性转移。通过将这些端点整理在较小的近期缓冲区中，我们的方法保持了与标准大型缓冲区相当的有效内存视野，同时所需的存储空间减少了一个数量级。通过实证评估，我们证明了这种方法防止了朴素压缩策略固有的系统性偏见，并能匹配弹球环境和Atari 2600基准测试中传统大型缓冲区的性能。

A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics

一个适用于具有转移动态变化的表格MDP中的混合强化学习的统一算法框架

• Authors: Zheshun Wu, Renjie Zheng, Jinhang Zuo, Zenglin Xu, Fang Kong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.25207
• Pdf link: https://arxiv.org/pdf/2607.25207
• Abstract This paper investigates a hybrid reinforcement learning setting in tabular Markov Decision Processes (MDPs), where an agent aims to learn an optimal policy by combining online interactions with a target environment and offline data from a source environment. A central challenge is that offline data may be collected from outdated environments with shifted transition dynamics, making naive integration of historical data ineffective. To address this, we propose a unified algorithmic framework featuring two algorithms: MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification. Both algorithms leverage fine-grained bias information to more effectively exploit offline data under general transition shifts. We provide theoretical guarantees for our framework, including both instance-dependent and independent upper bounds on regret and sub-optimality gap. Furthermore, we establish matching lower bounds to demonstrate the optimality of our approach and validate our theoretical findings through extensive experiments.
• 中文摘要 本文探讨了表格马尔可夫决策过程（MDP）中的混合强化学习环境，其中代理通过结合目标环境的在线交互和源环境的离线数据，学习最优策略。一个核心挑战是离线数据可能来自过时环境，过渡动态发生变化，导致对历史数据的简单整合无效。为此，我们提出了一个统一的算法框架，包含两个算法：MIN-UCB-VI 用于遗憾最小化，MAX-LCB-VI 用于最佳策略识别。这两种算法都利用细粒度偏置信息，更有效地利用一般过渡转移下的离线数据。我们为框架提供了理论保证，包括实例依赖性和独立的遗憾上限和次最优性差距。此外，我们建立了匹配的下界，以展示方法的最优性，并通过大量实验验证理论结果。

Structure-aware Relative Policy Optimization for Ranking

结构感知的相对策略优化以实现排名

• Authors: Yiteng Tu, Weihang Su, Zitao Su, Yiqun Liu, Min Zhang, Qingyao Ai
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.25268
• Pdf link: https://arxiv.org/pdf/2607.25268
• Abstract Ranking is a fundamental component of modern information access systems. Reinforcement learning (RL) provides a flexible framework for directly optimizing coarse-grained feedback and system-level objectives defined over the complete ranking list. However, existing RL-based ranking methods typically treat each sampled permutation as an atomic output and evaluate it primarily through a scalar reward, overlooking the structural relationships among different ranking lists. Consequently, permutations with similar rewards but substantially different permutation patterns may receive comparable optimization signals, potentially leading to inaccurate credit assignment and overly aggressive policy updates. To address this limitation, we propose SRPO, a \textbf{S}tructure-aware \textbf{R}elative \textbf{P}olicy \textbf{O}ptimization framework for listwise ranking. SRPO measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances. It quantifies the reward improvement per unit of ranking change, thereby emphasizing efficient local refinements, particularly those involving top-ranked positions. Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking, with particularly favorable performance in limited-feedback and complex list-level optimization settings.
• 中文摘要 排名是现代信息访问系统的基本组成部分。强化学习（RL）提供了一个灵活的框架，用于直接优化粗粒度反馈和系统级目标，这些目标定义在完整的排名列表中。然而，现有基于强化学习的排序方法通常将每个采样置换视为原子输出，主要通过标量奖励来评估，忽略了不同排名列表之间的结构关系。因此，奖励相似但排列模式显著不同的排列可能会收到相似的优化信号，可能导致信用分配不准确和政策更新过于激进。为解决这一限制，我们提出了SRPO，一个基于结构的\textbf{R}elative \textbf{P}olicy \textbf{O}ptimization框架，用于列表排名。SRPO利用最高加权的Kendall-tau距离测量采样置换之间的差异，并对相应距离对其两对奖励差异进行归一化。它量化了每单位排名变化带来的奖励提升，从而强调高效的本地优化，尤其是涉及顶级排名的优化。两种排名场景的实验结果表明，明确建模置换层级差异能提升列表级排名的有效性和稳定性，尤其在有限反馈和复杂列表级优化环境中表现尤为理想。

CAST: Game Solvers as Turn-Level Teachers for LLM Agents

演员阵容：游戏解决者作为LLM代理的回合级教师

• Authors: Yu Wang, Yi-Kai Zhang, Wentao Shi, Ziang Ye, Yuchun Miao, Yueqing Sun, Qi Gu, Xunliang Cai, Lan-Zhe Guo, Han-Jia Ye, Fuli Feng
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.25308
• Pdf link: https://arxiv.org/pdf/2607.25308
• Abstract Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at this https URL.
• 中文摘要 训练大型语言模型（LLMs）参与长期游戏是迈向通用决策的有希望的一步，然而带有可验证奖励的强化学习（RLVR）依赖于稀疏的最终奖励，几乎无法揭示决定哪些决策决定成功。更密集的工艺信号可以补充这一缺失的转弯级信用，但现有的信号很难既便宜又准确。我们观察到，博弈求解器状态值的变化决定了某个动作是否推动该状态向成功推进。基于这一见解，我们提出了CAST（解题教师学分分配），将这些价值变化转化为解题者的优势，并将其注入RLVR作为回合级信号。我们进一步证明，在软最优求解器假设下，最大化求解器优势等价于从求解器进行策略提纯，只需标量值而非教师logit。在《Sokoban》、《扫雷者》和《尖峰时刻》中，CAST在所有游戏中无论是在场内还是未见难度评估中都优于所有训练基线，并在ALFWorld和WebShop上取得了最高的零射击平均表现。我们的代码可在此 https URL 访问。

A Control System, a Dataset, and a Recipe for Making Frozen LLM Agents Learn a Domain

一个控制系统、一个数据集，以及一个让冻结的大型语言模型代理学习域的配方

• Authors: Debjyoti Paul
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.25415
• Pdf link: https://arxiv.org/pdf/2607.25415
• Abstract Production LLM agents are increasingly assembled from a frozen model wrapped in a harness: a prompt template, a tool set, a memory/retrieval layer, a planning strategy, and a verification policy. Two 2026 systems, Meta-Harness (Lee et al., 2026) and HyperAgents (Meta AI, 2026), show that this harness can itself be optimized or even self-rewritten by an agentic proposer -- at the cost of either an expensive code-search loop or unconstrained self-modifying code, neither of which is auditable or usable with a fully black-box model API. We take a narrower, more constrained position: treat the harness as a small, fixed, human-legible action space and learn a policy over it online with classic sample-efficient reinforcement learning (an $\epsilon$-greedy contextual bandit and REINFORCE), scored against a multi-objective reward (task success, verifier score, policy compliance, cost, latency, and an unsupported-claim penalty). We instantiate this control system with DSPy (Khattab et al., 2024) as both the context assembler and the source of the strongest non-adaptive baseline (a DSPy BootstrapFewShot static prompt), and evaluate it across three verifiable task domains -- tool-use workflows, code generation (HumanEval), and multi-hop retrieval QA (HotpotQA) -- and two model providers (a local Ollama model and AWS Bedrock). We release the harness-control-system code, the cross-domain verifiable task suite, the full trajectory/reward-decomposition logs from training, and a provider-agnostic deployment recipe for applying this to a new organization's domain and verification setup.
• 中文摘要 生产LLM代理越来越多地由一个冻结的模型包裹在一个框架中组装而成：提示模板、工具集、内存/检索层、规划策略和验证策略。2026年发布的两个系统Meta-Harness（Lee等，2026年）和HyperAgents（Meta AI，2026年）表明，智能体提议者可以优化甚至自我重写该框架——但代价是昂贵的代码搜索循环或不受限制的自我修改代码，而这两种代码都无法通过完全黑箱模型API进行审计或使用。我们采取更狭窄、更受限的立场：将束带视为一个小型、固定、人类可读的动作空间，并通过经典的样本高效强化学习（如贪婪的情境强化和REINFORCE）在线学习策略，并以多目标奖励（任务成功率、验证者得分、策略合规性、成本、延迟和无支持申诉惩罚）进行评分。我们以DSPy（Khattab等，2024）作为上下文汇编器和最强非自适应基线（DSPy BootstrapFewShot静态提示）的源，实现了该控制系统，并在三个可验证任务领域——工具使用工作流、代码生成（HumanEval）和多跳检索质量保证（HotpotQA）——以及两个模型提供者（本地Ollama模型和AWS Bedrock）进行评估。我们发布了机束控制系统代码、跨域可验证任务套件、培训中的完整轨迹/奖励分解日志，以及一个不依赖于提供者的部署配方，用于将这些内容应用于新组织的域和验证设置。

Bayesian-Guided Cooperative RL Beamforming for Wireless Adversarial User Detection

贝叶斯引导的合作式强焰学习波束成形用于无线对抗用户检测

• Authors: Parmida Geranmayeh, Onur Günlü
• Subjects: Subjects: Information Theory (cs.IT); Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2607.25417
• Pdf link: https://arxiv.org/pdf/2607.25417
• Abstract In next-generation wireless networks, communication systems are expected to go beyond simple data transmission and simultaneously provide high data rates, efficiency, and security. This requirement has motivated the extensive adoption of machine learning methods to develop intelligent and real-time network management frameworks, enabling the system to continuously monitor and react to channel variations and user behavior while maintaining efficient information delivery. In this context, the integration of machine learning with beamforming enables adaptive and data-driven beam direction selection, improving both the efficiency and security of wireless links. In this work, a 3GPP-based system model is first implemented under a no-attacker scenario, and an exhaustive search is employed as a reference to identify the best beamforming configurations. The proposed framework is then evaluated in the presence of an attacker and under different network scalability conditions. We demonstrate that the reinforcement learning-based approaches, namely Q-learning and SARSA (State-Action-Reward-State-Action), consistently outperform random selection in terms of total channel capacity, attacker detection accuracy, and performance stability. Among the evaluated reinforcement learning methods, Q-learning achieves the best overall trade-off between detection accuracy and computational efficiency. Our results indicate that the proposed framework provides a stable, scalable, and effective solution for joint beamforming and security-aware decision-making in dynamic and adversarial wireless environments.
• 中文摘要 在下一代无线网络中，通信系统预计不仅仅是简单的数据传输，还能同时提供高数据速率、高效性和安全性。这一需求促使机器学习方法被广泛采用，开发智能且实时的网络管理框架，使系统能够持续监控和响应信道变化及用户行为，同时保持高效的信息传递。在此背景下，机器学习与波束成形的结合使得自适应且数据驱动的波束方向选择成为可能，提升了无线链路的效率和安全性。本研究首先在无攻击者情景下实现基于3GPP的系统模型，并采用穷尽搜索作为参考，以识别最佳波束成形配置。随后在攻击者存在且不同网络可扩展性条件下评估所提框架。我们证明，基于强化学习的方法，即Q学习和SARSA（状态-动作-奖励-状态-动作），在总信道容量、攻击者检测准确性和性能稳定性方面始终优于随机选择。在评估的强化学习方法中，Q-学习在检测准确率与计算效率之间实现了最佳的整体权衡。我们的结果表明，所提出的框架为动态和对抗性无线环境中的联合波束成形和安全意识决策提供了稳定、可扩展且高效的解决方案。

System-Aware Adaptive CSI Feedback via RL-Guided Autoencoder Switching in Multi-User MIMO System

通过多用户MIMO系统中的RL引导自编码器切换实现系统感知自适应CSI反馈

• Authors: Maryam Ansarifard, Mohit K. Sharma, George Exarchakos, Kishor C. Joshi
• Subjects: Subjects: Information Theory (cs.IT); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.25588
• Pdf link: https://arxiv.org/pdf/2607.25588
• Abstract This paper proposes a system-aware adaptive channel state information (CSI) feedback framework for massive multiple-input multiple-output (mMIMO) systems, aiming to dynamically optimize the trade-off between reconstruction fidelity and signaling overhead. While deep learning-based autoencoders (AEs) have enabled significant CSI compression, conventional fixed-ratio schemes fail to adapt effectively to non-stationary channel conditions. To address this limitation, we develop a reinforcement learning (RL)-driven control framework that operates over a bank of pretrained multi-rate AEs, each corresponding to a distinct compression ratio (CR). At each time step, a centralized RL agent selects the most suitable CR for each user based on observed channel conditions and system performance indicators. Distinct from conventional mean squared error (MSE)-centric designs, we introduce a system-aware reward formulation that jointly accounts for spectral efficiency via signal-to-interference-plus-noise ratio (SINR), feedback overhead constraints, and the computational cost of model adaptation. Simulation results on high-dimensional delay-domain CSI datasets demonstrate that the proposed RL-guided framework effectively balances the overhead-accuracy tradeoff and adapts to dynamic channel environments. The proposed method improves spectral efficiency and feedback efficiency compared with fixed compression schemes and adaptive baselines, while maintaining a modest computational and memory footprint. Averaged over different numbers of users and across all considered baselines, the proposed RL framework reduces the CSI feedback cost by more than 53.4%, improves the average downlink sum rate by 53.64%, and reduces the NMSE by 22.38%. These results demonstrate its ability to achieve a more efficient rate-accuracy-feedback tradeoff under dynamic wireless conditions.
• 中文摘要 本文提出了一种系统感知自适应信道状态信息（CSI）反馈框架，适用于大规模多输入多输出（mMIMO）系统，旨在动态优化重建保真度与信令开销之间的权衡。虽然基于深度学习的自编码器（AE）实现了显著的CSI压缩，但传统的固定比率方案未能有效适应非定稳信道条件。为解决这一限制，我们开发了一个强化学习（RL）驱动的控制框架，运行在一组预训练的多速率AE上，每个AE对应不同的压缩比（CR）。在每个时间步，集中式强化学习代理根据观察到的信道状况和系统性能指标，为每个用户选择最合适的CR。与传统的均方误差（MSE）中心设计不同，我们引入了一种系统感知的奖励公式，通过信干扰加噪声比（SINR）、反馈开销约束以及模型适配的计算成本共同考虑频谱效率。高维延迟域CSI数据集的模拟结果表明，所提出的强化学习引导框架有效平衡了开销与准确性权衡，并适应动态信道环境。与固定压缩方案和自适应基线相比，该方法提升了频谱效率和反馈效率，同时保持了适中的计算和内存占用。通过不同用户数和所有考虑基线的平均值，所提的强化学习框架降低了CSI反馈成本超过53.4%，平均下行链路总和率提升了53.64%，并降低了22.38%。这些结果表明它在动态无线条件下实现更高效的速率-准确率-反馈权衡。

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

CoRT：用于令牌级评分标准引导策略优化的反事实重放

• Authors: Bo-Wen Zhang, Junwei He, Wen Wang, Song-Lin Lv, Wentao Ma, Rongyi Lin, Shuhan Zhong, Lan-Zhe Guo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.25659
• Pdf link: https://arxiv.org/pdf/2607.25659
• Abstract Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.
• 中文摘要 基于评分标准的强化学习通过根据显式标准评估模型输出，丰富了语言模型训练。然而，在GRPO风格的管道中，这些结构化判断被简化为标量响应级奖励，并转化为响应级优势，并统一广播给所有生成的代币。这就没有明确机制来在回答中分配署名，即使不同标准基于不同的跨度、格式决策或语义选择。我们提出了CoRT，一种基于评分标准条件GRPO的代币级信用加权方法。CoRT不训练辅助标记评分模型，而是使用反事实回放，在原始评分标准条件提示和匹配无条件提示下重新评分相同的抽样回答。由此产生的标记对数似然对比作为对评分标准上下文依赖性的代理指标。CoRT将这些对比映射为有界的、响应归一化的权重，并利用它们在不引入辅助评分器或改变响应级奖励的情况下，重新分配带符号的GRPO优势。跨指令调优模型和奖励粒度的实验显示，在绝大多数比较中，CoRT优于匹配响应水平的GRPO，平均提升4.4个百分点。该方法在与已学习的代币级信用基线保持竞争力，同时避免了独立的相关性学习阶段。这些结果表明，政策与内部反事实似然对比为响应内学分分配提供了有效的训练信号，同时保持GRPO的简洁性和稳定性。

Cooperative Multi-UAV Navigation in Complex Environments via Systematic Multi-Agent Deep Reinforcement Learning

通过系统多智能体深度强化学习实现复杂环境中的协作式多无人机导航

• Authors: Yu Su, Nabil Aouf
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.25754
• Pdf link: https://arxiv.org/pdf/2607.25754
• Abstract Cooperative navigation of multi-agent UAVs in complex environments faces key challenges including local optima traps, sparse rewards, learning imbalance among agents, and insufficient cross-scenario generalisation. This paper proposes a multi-agent deep reinforcement learning framework that addresses these issues through coordinated exploration, demonstration exploitation, safe curriculum scheduling, and structure-aware generalisation. First, a perception mechanism combining memory of visited states, directional novelty estimates, and penalty backpropagation enables agents to proactively detect and escape local optima. Second, a hierarchical collaborative demonstration buffer with tiered behaviour cloning manages trajectories by degree of team collaboration and applies differential supervision to the actor network, improving demonstration utilisation under sparse collaborative signals. Third, a safety-aware dual-condition curriculum scheduling mechanism reviews mastered scenarios through back-testing and experience pre-filling during training, suppressing catastrophic forgetting while ensuring both task performance and flight safety. For generalisation, local geometric features computed from sensor readings are abstracted into a domain parameter, through which a structure-aware gating network and mixture-of-experts mechanism condition the policy on local structural patterns rather than scenario-specific coordinates, enabling cross-scenario transfer without exposure to the target environment. The framework is further validated under mixed static-dynamic obstacle settings, showing robust adaptability to dynamic disturbances. Simulation results confirm strong performance in collaboration success rate, navigation robustness, zero-shot cross-scenario generalisation, and dynamic environment adaptability.
• 中文摘要 多智能体无人机在复杂环境中的协作导航面临关键挑战，包括局部最优陷阱、奖励稀疏、智能体间学习不平衡以及跨场景泛化不足。本文提出了一种多智能体深度强化学习框架，通过协调探索、示范利用、安全课程安排和结构感知泛化来解决这些问题。首先，结合访问状态记忆、方向性新颖性估计和惩罚反向传播的感知机制，使智能体能够主动检测并逃离局部最优。其次，采用分层行为克隆的层级协作演示缓冲区，按团队协作程度管理轨迹，并对演员网络施加差异性监督，提升稀疏协作信号下的演示利用率。第三，安全意识的双条件课程排程机制通过回测和培训预填经验回顾已掌握的情景，抑制灾难性遗忘，同时确保任务执行和飞行安全。为推广，从传感器读数计算出的局部几何特征被抽象为域参数，通过该参数，结构感知的门控网络和专家混合机制根据局部结构模式而非特定场景坐标来调整策略，从而实现跨场景传输而无需暴露于目标环境。该框架在混合静态-动态障碍条件下得到了进一步验证，显示出对动态扰动的稳健适应性。模拟结果证实了协作成功率、导航鲁棒性、零次跨场景泛化能力以及动态环境适应性等方面表现出色。

A Hierarchical Optimisation Framework for Integrated Electric-Hydrogen-Transport Systems

集成电力-氢气运输系统的层级优化框架

• Authors: Fulong Yao, Yiming Xu, Liana Cipcigan, Maurizio Albano, Naeima Hamed, Nima Valizadeh, Omer Rana
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.25776
• Pdf link: https://arxiv.org/pdf/2607.25776
• Abstract Integrated electric-hydrogen infrastructures are becoming increasingly important with the growing deployment of electric vehicles (EVs) and hydrogen vehicles (HVs) in transport systems. However, the strong coupling between vehicle scheduling and multi-energy dispatch introduces significant operational challenges. This paper models an integrated electric-hydrogen-transport system (EHTS) and proposes a hierarchical optimisation framework that couples vehicle scheduling and downstream energy dispatch through a sequential, demand-driven two-layer structure. In the vehicle scheduling layer, a solver-free greedy heuristic (SFGH) algorithm is developed to avoid repeated optimisation solving, enabling real-time EV charging and HV refuelling under non-preemptive service and within-interval sequential assignment. The resulting charging and refuelling demands are subsequently passed to the energy dispatch layer, where a deep reinforcement learning (DRL)-based approach is designed to optimise battery operation, hydrogen-tank operation, and PV generation allocation to minimise the overall operational cost of the EHTS while satisfying the scheduled transport demand. Representative case studies, together with comparative, ablation, and generalisation analyses, demonstrate the effectiveness and robustness of the proposed framework. Furthermore, the learned dispatch policy maintains strong performance across diverse transport-demand scenarios without retraining, demonstrating robust generalisation capability for practical deployment.
• 中文摘要 随着电动汽车（EV）和氢能车辆（HV）在交通系统中的日益普及，集成的电力-氢能基础设施变得越来越重要。然而，车辆调度与多能量调度的紧密耦合带来了重大运营挑战。本文建模了一个集成的电力-氢气运输系统（EHTS），并提出了一个层级优化框架，将车辆调度与下游能源调度通过顺序、需求驱动的两层结构相结合。在车辆调度层，开发了无求解器的贪婪启发式（SFGH）算法，避免重复优化求解，实现了在非抢占式服务和区间内顺序分配下的实时电动汽车充电和高压加注。由此产生的充电和加注需求随后传递至能源调度层，采用基于深度强化学习（DRL）的方法，优化电池运行、氢罐运行和光伏发电分配，以最大限度地降低EHTS的整体运营成本，同时满足计划运输需求。代表性案例研究，以及比较、消融和泛化分析，展示了该框架的有效性和稳健性。此外，学习出的调度策略在多种运输需求场景下保持强劲性能，无需重新训练，展现了实用部署的强有力泛化能力。

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

推理时投机：教代理通过联合代理-投机者强化学习预测下一个工具调用

• Authors: Jiabao Ji, Yujian Liu, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.25816
• Pdf link: https://arxiv.org/pdf/2607.25816
• Abstract Large language model agents often spend substantial wall-clock time waiting for tool call results. Tool-call speculation can hide this latency by predicting and pre-executing an agent's next tool call if the prediction matches the agent's eventual tool call, but existing speculators are typically separate draft models or cached traces that are poorly aligned with the deployed agent's own behavior. We identify this speculator-agent gap and show that the target agent itself is a strong next-call speculator. This points to a simpler design: unifying the agent and speculator within the same model. In this paper, we introduce the self-speculating agent, a single model that both solves tasks in agent mode and predicts its next tool call from partial trajectories in speculator mode, fully reusing prefix KV cache. To enable this dual-mode agent without degrading performance, we propose a joint agent-speculator reinforcement learning method, which derives speculation targets from the agent's own rollouts and alternates agent and speculator updates. Across agentic search QA and conversational tool-use agentic tasks, our method improves average next tool-call Hit@1 from 44.1 to 61.2 for Qwen3-4B and from 48.9 to 66.3 for Qwen3.5-4B, while preserving agent task success.
• 中文摘要 大型语言模型代理常常花费大量时间等待工具调用结果。工具调用的推测可以通过预测并预执行代理的下一次工具调用来隐藏这种延迟，前提是预测与代理最终的工具调用相符，但现有的投机者通常是独立的草稿模型或缓存跟踪，这些跟踪与部署代理自身的行为不匹配。我们识别了投机者-代理之间的差距，并证明目标代理本身是一个强有力的下一次投机者。这表明设计更简单：将代理人和投机者统一于同一模型中。本文介绍了自推测代理，这是一种既能在代理模式下解决问题，又能从推测模式下部分轨迹预测下一个工具调用的模型，完全重用前缀KV缓存。为了实现这种双模代理而不降低性能，我们提出了一种联合代理-投机者强化学习方法，该方法从代理自身的部署中推导投机目标，并交替更新代理和投机者更新。在代理搜索QA和对话工具使用代理任务中，我们的方法将Qwen3-4B的平均下一次工具调用Hit@1从44.1提升至61.2，Qwen3.5-4B从48.9提升至66.3，同时保持代理任务的成功率。

WarmTuner: Program-Specific Warm Starts for Compiler Autotuning via Offline-to-Online Reinforcement Learning

WarmTuner：通过离线到在线强化学习实现编译器自动调优的程序专属热启动

• Authors: Tianlu Qiao, Mingxuan Zhu, Zeyu Sun, Dan Hao
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.25831
• Pdf link: https://arxiv.org/pdf/2607.25831
• Abstract Compilers are fundamental software tools that translate high-level programs into machine code. Modern compilers expose hundreds of optimizations, each turned on or off through an optimization flag, to improve the performance of the generated code. However, the number of possible flag combinations grows exponentially, making it difficult to find a flag configuration well suited to a given target program. Existing compiler auto-tuning techniques reduce tuning cost by pruning the search space, injecting search biases, or predicting configuration performance. Although some exploit program features, the knowledge they extract from historical data is frozen once search begins; runtime feedback then guides only the search itself, never the prior. As a result, when this prior mismatches the target program, these methods waste much of the limited online budget before the search reaches good configurations. We propose WarmTuner, an offline-to-online reinforcement learning framework that instead turns historical records into a program-conditioned policy that predicts each flag's setting over the full flag space and remains adaptable on the target program. Offline, WarmTuner learns this program-conditioned policy over the full flag space from historical good configurations. Online, it refines the same policy on the target program using real compile-run feedback, so that the policy is driven by measured speedups rather than limited to the historical data. We instantiate the online update with Group Relative Policy Optimization (GRPO), which compares candidates in the same round and avoids a separate value model. We evaluate WarmTuner on GCC 15.2.0 with cBench and PolyBench. The results show that WarmTuner achieves an average speedup of 1.732x over GCC -O3 and obtains the best result on 14/30 programs, significantly outperforming the compared techniques.
• 中文摘要 编译器是将高级程序转换为机器码的基本软件工具。现代编译器通过一个优化标志开放或关闭了数百个优化，以提升生成代码的性能。然而，可能的旗帜组合数量呈指数增长，这使得找到适合特定目标程序的旗帜配置变得困难。现有的编译器自动调优技术通过修剪搜索空间、注入搜索偏差或预测配置性能来降低调优成本。虽然有些利用程序具备功能，但它们从历史数据中提取的知识在搜索开始后会被冻结;运行时反馈仅指导搜索本身，而非先验。因此，当这种先验与目标项目不匹配时，这些方法会浪费大量有限的在线预算，直到搜索达到理想配置。我们提出了WarmTuner，一种离线到在线的强化学习框架，将历史记录转化为程序条件策略，预测每个旗帜在整个旗帜空间内的设置，并保持对目标程序的适应性。离线时，WarmTuner通过历史良好配置学习该程序条件策略，覆盖整个旗帜空间。在线时，它通过真实的编译运行反馈对目标程序进行优化，使策略由测量到的加速驱动，而非仅限于历史数据。我们用Group Relative Policy Optimization（GRPO）实现在线更新，该方法比较同一轮候选人，避免使用单独的价值模型。我们通过cBench和PolyBench评估了GCC 15.2.0上的WarmTuner。结果显示，WarmTuner平均速度比GCC -O3提升1.732倍，并在14/30程序中取得最佳成绩，显著优于对比技术。

RecoReward: Recommender-Guided Multimodal Description Generation for Recommendation

RecoReward：推荐者引导的多模态描述生成

• Authors: Guohong Mu, Yueyang Liu, Jiangxia Cao, Changxin Lao, Zijie Zhuang, Yuhui Zhang, Jiaqi Feng, Ruochen Yang, Shuang Yang, Zhaojie Liu, Qibin Hou
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.25901
• Pdf link: https://arxiv.org/pdf/2607.25901
• Abstract Multimodal large language models (MLLMs) can convert multimodal item content into structured descriptions used as semantic features for recommendation. Conventional content-only generation, however, cannot use downstream user signals to determine which semantics should be emphasized. Recent user-conditioned methods incorporate these signals through user histories or profiles, but they require user information at inference and make generation user-dependent. In this paper, we introduce RecoReward, which instead uses behavior-derived rewards during training and preserves content-only inference. To instantiate this idea in live-stream recommendation, we treat historically engaged users as a proxy for future target users and use observational non-target users to estimate affinity shared broadly across users. The Recommender Affinity Score (RAS) contrasts these signals to provide user-selective feedback for reinforcement learning, allowing the learned policy to generate a single shared description without user inputs. In our offline benchmark, RecoReward-9B outperforms its Qwen3.5-9B baseline and all other evaluated models across seven recall metrics. Online A/B testing also shows performance gains. These results show that RecoReward trains the MLLM to produce item features that benefit downstream recommendation while retaining content-only serving.
• 中文摘要 多模态大型语言模型（MLLM）可以将多模态项目内容转换为结构化描述，作为推荐的语义特征。然而，传统的纯内容生成无法利用下游用户信号来决定应强调哪些语义。最近的用户条件方法通过用户历史或配置文件来整合这些信号，但推断时需要用户信息，且生成依赖用户。本文介绍了RecoRewere，它在训练过程中使用行为衍生的奖励，并保持仅内容的推断。为了在直播推荐中体现这一理念，我们将历史参与的用户视为未来目标用户的代理，并利用观察性非目标用户来估算用户间广泛共享的好感度。推荐者亲和力评分（RAS）对比这些信号，提供用户选择性的反馈以促进强化学习，使所学策略能够生成单一共享描述，无需用户输入。在我们的离线基准测试中，RecoReward-9B在七个召回指标上优于其Qwen3.5-9B基线及所有其他评估模型。在线A/B测试同样显示性能提升。这些结果表明，RecoReward训练MLLM能够生成有利于下游推荐的项目特征，同时保留仅内容的服务内容。

Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification

交互式奖励代理：通过环境状态验证进行图形用户界面任务评估

• Authors: Chenrui Shi, Yuwei Wu, Yang Liu, Ruining Feng, Zirui Shang, Zhi Gao, Lifeng Fan, Che Sun
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.25904
• Pdf link: https://arxiv.org/pdf/2607.25904
• Abstract Graphical user interface task evaluation aims to determine whether a GUI agent has successfully completed a user instruction. Automated GUI task evaluation has received increasing attention because the evaluation results can serve as reward signals for both test-time scaling and post-training. However, reliable GUI task evaluation remains challenging because the judgments often require access to environment states, such as system configurations, file data, and application settings, beyond the screenshots of execution trajectories. In this paper, we propose an interactive reward agent (IRA) based on a propose-then-verify framework to acquire and verify evidence from the post-execution environment. Given a task instruction and a GUI environment after the GUI agent execution, IRA first proposes the task completion conditions and then verifies them by invoking system tools, application tools, and GUI tools. This design combines evidence from both visible interfaces and the environment state in an interactive process. We further introduce GUI-RewardBench, a benchmark of 321 GUI task trajectories spanning 10 Ubuntu desktop application categories. Experiments show that IRA achieves 86.9% accuracy on GUI-RewardBench, outperforming existing evaluator baselines. We further apply IRA to reinforcement learning of GUI agents, achieving a 34.0% OSWorld success rate, which demonstrates that IRA can provide effective reward signals for training GUI agents.
• 中文摘要 图形用户界面任务评估旨在确定图形界面代理是否成功完成了用户指令。自动化图形界面任务评估日益受到关注，因为评估结果可以作为测试时间扩展和训练后奖励信号。然而，可靠的GUI任务评估仍然具有挑战性，因为判断通常需要访问环境状态，如系统配置、文件数据和应用设置，超出执行轨迹的截图。本文提出基于先提出-验证框架的交互式奖励代理（IRA），用于从执行后环境中获取和验证证据。在GUI代理执行后，给定任务指令和GUI环境，IRA首先提出任务完成条件，然后通过调用系统工具、应用工具和图形界面工具进行验证。该设计结合了可见界面和环境状态的证据，形成交互式过程。我们还进一步介绍了GUI-RewardBench，这是一个涵盖10个Ubuntu桌面应用类别的321个GUI任务轨迹的基准测试。实验显示，IRA在GUI-RewardBench上的准确率达到86.9%，优于现有评估基线。我们还进一步将IRA应用于图形界面代理的强化学习，实现了34.0%的OSWorld成功率，证明IRA能够为训练图形界面代理提供有效的奖励信号。

Physics-Aware End-to-End Deep Reinforcement Learning for Quadcopter Control with Actuator Dynamics

基于执行器动力学的物理感知端到端深度强化学习，用于四旋翼控制

• Authors: Ya-Chia Shen, Woei-Leong Chan
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.25985
• Pdf link: https://arxiv.org/pdf/2607.25985
• Abstract Unmanned aerial vehicles (UAVs), particularly quadcopters, present unique challenges for autonomous control due to their underactuated dynamics: only four available control inputs must govern six degrees of freedom. This paper investigates a physics-aware, end-to-end deep reinforcement learning (DRL) approach that acts directly on low-level body inputs, total thrust and body torques $(T, \tau_x, \tau_y, \tau_z)$, and closes the loop through a high-fidelity Simulink environment. Our simulator integrates a 12-state rigid-body model (MATLAB Level-2 S-Function) with (i) an Action2RPM allocation based on the Moore-Penrose pseudo-inverse of a coefficient matrix derived from thrust and drag terms, and (ii) first-order actuator dynamics for each motor (time constant $T_m = 0.076$ s), including rotor gyroscopic coupling. A shaped reward balances goal-reaching and stability using an exponential position well, attitude penalties, and quadratic velocity costs. Four DRL algorithms, DDPG, TD3, PPO, and SAC, are evaluated in two stages: (S1) thrust-only hover and (S2) hover with pitch torque and a translated goal. Results show that SAC and TD3 achieve superior stability and exploration efficiency, while PPO is less sample-efficient. The study highlights the significance of modeling actuator lags and aerodynamic moments for stable low-level control and provides a reproducible benchmark for quadcopter DRL.
• 中文摘要 无人机（UAV），尤其是四旋翼飞行器，由于其动力学不足，对自主控制带来了独特挑战：只有四个可用的控制输入必须控制六个自由度。本文探讨了一种物理意识、端到端深度强化学习（DRL）方法，该方法直接作用于低级人体输入、总推力和物体扭矩 $（T， \tau_x， \tau_y， \tau_z）$，并通过高精度的 Simulink 环境实现闭环。我们的模拟器将12态刚体模型（MATLAB二级S函数）与（i）基于推力和阻力项推导系数矩阵的Moore-Penrose伪逆矩阵的Action2RPM分配，以及（ii）每个电机的一阶执行器动力学（时间常数$T_m = 0.076$ s），包括转子陀螺耦合。形状奖励通过指数级位置、态度惩罚和二次速度成本，很好地平衡了目标达成和稳定性。四个DRL算法——DDPG、TD3、PPO和SAC——分为两个阶段进行评估：（S1）仅推力悬停和（S2）带有俯仰扭矩和平移目标的悬停。结果显示，SAC和TD3在稳定性和勘探效率上更优，而PPO的样品效率较低。该研究强调了对执行器滞后和气动矩建模对于稳定低空控制的重要性，并为四旋翼日间移动（DRL）提供了可重复的基准。

Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing

超越缩放：学习多工具视觉推理以实现超高分辨率遥感

• Authors: Fengxiang Wang, Jiangnan Huang, Mingshuo Chen, Yueying Li, Yang Shi, Junwei Luo, Haoyu Wang, Yansheng Li, Jing Zhang, Haiyan Zhao, Wenjing Yang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.25993
• Pdf link: https://arxiv.org/pdf/2607.25993
• Abstract Ultra-high-resolution (UHR) remote-sensing (RS) imagery provides fine-grained Earth-observation evidence over city-scale scenes, but poses a fundamental challenge for multimodal large language models (MLLMs): task-relevant evidence is often sparse, local, and spatially dispersed across extremely large visual contexts. A natural solution is to equip MLLMs with zoom-in tools for active local inspection. However, through a pilot study on XLRS-Bench, we find that zoom-in is only partially effective: it resolves easy and medium-level tasks with locally recoverable evidence, but saturates on hard cases requiring global search, multi-region comparison, path planning, or dispersed-evidence reasoning. Motivated by this finding, we move beyond single-tool zoom-in and introduce GeoMTVR, a large-scale Geospatial Multi-Tool Visual Reasoning dataset built from wide-area satellite imagery. GeoMTVR contains 13K UHR VQA samples with interleaved reasoning trajectories, diverse visual tool calls, and returned visual observations, enabling models to learn question decomposition, tool selection, regional inspection, object-level grounding, auxiliary visual reasoning, and cross-tool evidence integration. Beyond supervised fine-tuning, we propose a tool-attention-focused reinforcement learning algorithm that concentrates optimization on critical tool-use decisions, including when to invoke tools, which tool to select, where to apply it, and how to interpret tool outputs. By combining SFT on GeoMTVR with our RL algorithm, we develop GeoLens, a multi-tool visual reasoning MLLM for UHR RS. Experiments show that GeoLens consistently outperforms direct reasoning and single-tool zoom-in baselines, achieving stronger accuracy, better evidence grounding, and more efficient tool-use trajectories.
• 中文摘要 超高分辨率（UHR）遥感（RS）影像在城市规模场景中提供了细粒度的地球观测证据，但对多模态大型语言模型（MLLMs）来说是一个根本性的挑战：任务相关证据往往稀疏、局部且空间分散于极大的视觉环境中。一个自然的解决方案是为MLLM配备放大工具，以便进行本地主动检查。然而，通过对XLRS-Bench的试点研究，我们发现放大效果有限：它解决了简单且中等层次且可本地恢复证据的任务，但在需要全局搜索、多区域比较、路径规划或分散证据推理的困难案例上则显得饱和。受这一发现启发，我们超越了单工具放大，推出了GeoMTVR，这是一个基于广域卫星图像构建的大规模地理空间多工具视觉推理数据集。GeoMTVR包含13,000个UHR VQA样本，具有交错推理轨迹、多样的视觉工具调用和返回的视觉观察，使模型能够学习问题分解、工具选择、区域检查、对象层面的基础化、辅助视觉推理以及跨工具证据整合。除了监督微调外，我们还提出了一种以工具注意力为中心的强化学习算法，重点优化关键工具使用决策，包括何时调用工具、选择哪种工具、应用位置以及如何解释工具输出。通过将GeoMTVR上的SFT与我们的强化学习算法结合，我们开发了GeoLens，一款面向UHR RS的多功能视觉推理MLLM。实验显示，GeoLens始终优于直接推理和单工具放大基线，实现更强的准确性、更好的证据基础和更高效的工具使用轨迹。

Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidance

重新知情的梦想者：通过潜在指导高效训练的非对称世界模型

• Authors: Gaspard Lambrechts, Adrien Bolland, Daniel Ebi, Damien Ernst
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2607.26040
• Pdf link: https://arxiv.org/pdf/2607.26040
• Abstract Much like humans benefit from guidance while learning, reinforcement learning algorithms may benefit from additional supervision beyond rewards. Leveraging additional information during training to learn better representations and behaviors has been the focus of asymmetric reinforcement learning. This learning paradigm has proven effective under partial observability when additional state information is available, but also under full observability when more refined state information is available. Focusing on model-based reinforcement learning, we study the effect of asymmetric learning on observation representations and on privileged information representations. First, we identify a limitation in the privileged information representations learned by an asymmetric model-based algorithm known as the Informed Dreamer. Then, we propose a novel asymmetric representation learning objective using latent guidance, resulting in a new algorithm called the Reinformed Dreamer. Experiments across several benchmarks show a more consistent improvement over Dreamer than previous asymmetric approaches.
• 中文摘要 就像人类在学习时能从指导中受益一样，强化学习算法也可能受益于超越奖励的额外监督。在训练过程中利用额外信息学习更好的表征和行为，一直是非对称强化学习的重点。当有额外状态信息时，这种学习范式在部分可观测性下已被证明有效，在完全可观测性下，当状态信息更精细时也能有效。我们聚焦于基于模型的强化学习，研究非对称学习对观察表征和特权信息表征的影响。首先，我们发现了一种被称为“知情梦想者”的非对称模型算法所学习的特权信息表示的局限性。随后，我们提出了一种利用潜在引导的新颖非对称表示学习目标，形成了名为“Reinformed Dreamer”的新算法。多个基准测试的实验显示，相比以往的非对称方法，其改进更为一致。

Keyword: diffusion policy

S2A2: Audio-Visual Imitation Learning for Manipulation Tasks Using Acoustic Spatial Information

S2A2：利用声学空间信息进行操作任务的视听模仿学习

• Authors: Kaneyoshi Hiratsuka, Benjamin Yen, Ryosuke Kojima
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.26047
• Pdf link: https://arxiv.org/pdf/2607.26047
• Abstract Acoustic information provides rich cues about object location, material properties, and changes caused by contact or motion. This paper introduces a new set of acoustic-aware manipulation tasks for imitation learning, in which robots must use auditory cues to determine manipulation targets. These tasks require sound source localization and identification for active exploration in robotic manipulation. Also, we propose a multimodal imitation learning framework, Spatial-Spectral Audio Action (S2A2), that integrates visual features with acoustic spatial and acoustic signal information for the acoustic-aware manipulation tasks. We implemented S2A2 models that integrates policies such as ACT, Diffusion Policy, VQ-BeT, and $\pi_0$, into our framework. Simulation experiments showed that the proposed method is the most effective for tasks requiring both position and timbre. Furthermore, real-robot experiments confirm the applicability of the proposed tasks and framework to real-world manipulation.
• 中文摘要 声学信息提供了关于物体位置、材料性质以及接触或运动引起的变化的丰富线索。本文介绍了一套新的声学感知操作任务，用于模仿学习，机器人必须利用听觉线索确定操作目标。这些任务需要声源定位和识别，以便在机器人操作中进行主动探索。此外，我们提出了一种多模态模仿学习框架——空间-频谱音频动作（S2A2），该框架将视觉特征与声学空间和声信号信息整合，用于声学感知操作任务。我们实施了S2A2模型，将ACT、扩散政策、VQ-BeT和$\pi_0$等政策整合进我们的框架中。模拟实验表明，所提方法对于需要位置和音色的任务最为有效。此外，真实机器人实验证实了所提任务和框架在现实操作中的适用性。