生成时间: 2026-09-01 21:14:04 (UTC+8); Arxiv 发布时间: 2026-09-01 20:00 EDT (2026-09-02 08:00 UTC+8)

今天共有 73 篇相关文章

Keyword: reinforcement learning

CDPR: Counterfactual Advantage-based Credit Assignment for Cost-Aware Sequential Medical Diagnosis

CDPR：基于成本的反事实优势信用分配，用于成本意识的顺序医疗诊断

• Authors: Qi Peng, Yi Cai, Changmeng Zheng, Xin Wu, Jiayuan Xie, Qing Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.28599
• Pdf link: https://arxiv.org/pdf/2608.28599
• Abstract Clinical diagnosis is a step-by-step, cost-aware process: a physician orders examinations one at a time, observes the results, and updates the diagnosis before reaching a final conclusion. Most medical language models instead treat diagnosis as a one-pass classification task and ignore the trade-off between a test's value and its cost. We model diagnosis as a cost-aware sequential decision process and train the policy with reinforcement learning. The main difficulty is credit assignment: the only reliable signal comes once at the end of a long trajectory, so it scores a wasteful workup the same as an efficient one. We propose CDPR (Counterfactual Diagnostic Process Reward), which needs no expert labels and no learned critic. CDPR first finds the states where the policy hesitates, using the uncertainty of its action distribution, and then scores the chosen action by its advantage over the alternatives the policy itself would consider, estimated with short rollouts under a utility that balances correctness against test count, cost, and infeasible requests. A rollout cache reuses within-batch trajectories to keep the cost low. We integrate CDPR into GRPO and test it on one in-domain (MIMIC-IV) and two out-of-domain (ClinicalBench and a private hospital dataset) benchmarks. CDPR improves diagnostic accuracy while clearly reducing the number and cost of examinations.
• 中文摘要 临床诊断是一个逐步、成本意识的过程：医生一次下单检查，观察结果，更新诊断后得出最终结论。大多数医学语言模型将诊断视为一次通过的分类任务，忽视测试价值与成本之间的权衡。我们将诊断建模为一个成本意识的顺序决策过程，并通过强化学习训练策略。主要难点在于信用分配：唯一可靠的信号出现在漫长路径的终点，因此其评估浪费与高效工作一样。我们提出CDPR（反事实诊断过程奖励），无需专家标签，也无需专业批评。CDPR首先利用策略行动分布的不确定性找到政策犹豫的状态，然后根据策略本身考虑的替代方案的优势对所选行动进行评分，这些行为通过短时间推广估计，该工具在正确性与测试次数、成本和不可行请求之间取得平衡。部署缓存重用批内轨迹以保持低成本。我们将CDPR集成到GRPO中，并在一个域内（MIMIC-IV）和两个域外基准测试（ClinicalBench和一个私立医院数据集）。CDPR提高了诊断准确性，同时明显减少了检查的数量和成本。

SHAPE of Chain-of-Thought in Math Reasoning

数学推理中思维链的形状

• Authors: Jonghyun Song, Sangjun Song, Minjae Oh, Haesung Pyun, Sungsik Lee, Yohan Jo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.28600
• Pdf link: https://arxiv.org/pdf/2608.28600
• Abstract Large language models (LLMs) achieve strong performance on mathematical reasoning benchmarks, yet the mathematically meaningful skills underlying their reasoning remain underexplored. We introduce \texttt{SHAPE}, a framework that analyzes Chain-of-Thought (CoT) trajectories through two lenses developed in mathematics education: (1) semantic spaces: the model's evolving mathematical interpretations of a problem (e.g., algebraic, geometric), and (2) heuristics: the specific mathematical actions taken within those spaces (e.g., simplifying the problem, working backward). We first use \texttt{SHAPE} to analyze the reasoning patterns of various models. Our findings reveal that the mathematical heuristics employed by a model better explain final answer correctness than traditional CoT features. Furthermore, models are likely to reach correct solutions by concentrating their reasoning effort within a few semantic spaces rather than exploring many disparate ones -- a pattern consistent with human behavior. Next, we utilize the \texttt{SHAPE} lens to evaluate whether post-training truly enhances mathematical proficiency. We find that reinforcement learning induces mode-seeking in heuristic usage. Lastly, we post-train LLMs by promoting diverse heuristics and demonstrate its effectiveness in improving accuracy. Overall, \texttt{SHAPE} provides a theoretically-grounded diagnostic framework for decoding LLM reasoning and offers a new path toward post-training LLMs for math reasoning. The code for our model is available at this https URL
• 中文摘要 大型语言模型（LLMs）在数学推理基准测试中表现出色，但其推理背后的数学意义技能仍未被充分探索。我们介绍了 \texttt{SHAPE}，这是一个通过数学教育中发展的两个视角分析思维链（Chain-of-Thought，简称 CoT）轨迹的框架：（1）语义空间：模型对问题不断演变的数学解释（如代数、几何），以及（2）启发式：在这些空间内采取的具体数学操作（例如，简化问题、逆向推算）。我们首先使用 \texttt{SHAPE} 来分析各种模型的推理模式。我们的发现表明，模型所采用的数学启发式比传统 CoT 特征更能解释最终答案的正确性。此外，模型更可能通过将推理工作集中在少数语义空间内而非探索多个不同语义空间来达成正确解决方案——这一模式与人类行为一致。接下来，我们利用 \texttt{SHAPE} 视角评估后训练是否真正提升数学能力。我们发现强化学习会在启发式使用中诱导模式寻求。最后，我们通过推广多样化启发式方法对大型语言模型进行后期训练，并展示了其提升准确率的有效性。总体而言，\texttt{SHAPE} 提供了一个理论基础的诊断框架用于解码大型语言模型推理，并为数学推理的后训练大型语言模型开辟了新路径。我们模型的代码可在此 https URL 获取

InternReviewer & InternAdvocate: Objective Reward and Evaluation for Agentic Reinforcement Learning in Peer Review and Rebuttal

InternReviewer 与 InternAdvocate：同伴评审与反驳中能动强化学习的客观奖励与评估

• Authors: Xuerui Su, Liya Guo, Qizhi Pei, Qipeng Guo, Zhongbo Tian, Lijun Wu, Kai Chen, Zun Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.28612
• Pdf link: https://arxiv.org/pdf/2608.28612
• Abstract Generating professional scholarly content, such as peer reviews and rebuttals, requires an intricate synergy between domain reasoning and factual grounding. This work presents a comprehensive framework for the development and evaluation of specialized scholarly agents, InternReviewer and InternAdvocate. We first establish a large-scale, high-quality scholarly dataset and integrate a high-efficiency arXiv retrieval tool to enable active evidence gathering. To optimize these agents, we implement an agentic Reinforcement Learning (RL) paradigm driven by a unified objective metric and reward system. This system avoids the biases of subjective model-based judging by employing multi-dimensional criteria, including reference-anchored semantic alignment, structural compliance, and a strict verification mechanism that cross-checks citations against real-time interaction logs to eliminate hallucinations. Experimental results demonstrate that agents trained within this closed-loop framework exhibit significant improvements in reasoning depth and citation accuracy.
• 中文摘要 生成专业学术内容，如同行评审和反驳，需要领域推理与事实基础之间的复杂协同。本研究为专业学术代理——InternReviewer和InternAdvocate——的开发与评估提供了全面的框架。我们首先建立了大规模、高质量的学术数据集，并集成了高效的arXiv检索工具，以实现主动证据收集。为优化这些代理，我们实施了由统一客观指标和奖励系统驱动的代理强化学习（RL）范式。该系统通过采用多维标准，包括基于引用的语义对齐、结构一致性以及严格的验证机制（将引用与实时交互日志交叉核对以消除幻觉）等，避免了主观基于模型的判断偏见。实验结果表明，在该闭环框架下训练的代理在推理深度和引用准确性方面有显著提升。

Cognitively-Grounded On-Device Runtime Learning for Ground Robots in Unknown Physical Environments

未知物理环境中地面机器人的认知基础设备运行时学习

• Authors: Yihao Cai, Yanbing Mao, Christian Lebiere
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.28677
• Pdf link: https://arxiv.org/pdf/2608.28677
• Abstract This paper presents \ul{CogRun}, a framework that enables safety-critical ground robots to perform cognitively-grounded runtime learning entirely on edge-AI devices in unknown physical environments, without prior maps or perceptual knowledge. CogRun consists of three components: a Learning-Agent, a Rational-Agent, and a Coordinator. The Learning-Agent is novel in cognitive-neural learning architecture, which featurs dedicated replay buffers, cognition-driven experience sampling, and a safety-aware action blending of actor-critic reinforcement learning (RL) with instance-based learning (IBL). The Rational-Agent is a non-learning module that complements the Learning-Agent by exclusively handling safety-critical functions, while the Coordinator manages interactions between the two agents to promote safe and efficient runtime learning. CogRun's full autonomy stack (i.e., perception, learning, and control) on edge-AI devices eliminates dependence on wireless communications, enabling broader applications in challenging environments with limited or no connectivity. Experiments on a quadruped robot in real-world wild forests and on an off-road autonomous vehicle in a simulated wild forest demonstrate that CogRun enables safe and efficient runtime learning, allowing robots to safely and continuously interact with the physical world for enhancing task performance in complex, unknown environments.
• 中文摘要 本文提出了 \ul{CogRun} 框架，使安全关键的地面机器人能够在未知物理环境中，完全在边缘 AI 设备上执行基于认知的运行时学习，无需先前的地图或感知知识。CogRun 由三个组成部分组成：学习代理、理性代理和协调者。学习代理在认知神经学习架构中具有创新性，具备专用回放缓冲区、认知驱动体验采样，以及将演员-批评者强化学习（RL）与基于实例学习（IBL）的安全意识动作相结合。理性代理是一个非学习模块，通过专门处理安全关键功能来补充学习代理，而协调者则管理两个代理之间的交互，促进安全高效的运行时学习。CogRun在边缘AI设备上的全自主技术栈（即感知、学习和控制）消除了对无线通信的依赖，使得在连接有限或无连接的复杂环境中实现更广泛的应用。在真实野生森林中对四足机器人和在模拟野生森林中越野自动驾驶车辆上的实验表明，CogRun实现了安全高效的运行时学习，使机器人能够安全且持续地与物理世界交互，从而提升复杂未知环境中的任务性能。

ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoning

ERR+：连续熵解析，实现高效且果断的大型语言模型推理

• Authors: Xin Jiang, Minhao Wang, Wen Wu, Zhentao Xie, Shangheng Du, Jinxin Shi, Jiabao Zhao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.28771
• Pdf link: https://arxiv.org/pdf/2608.28771
• Abstract Large reasoning models achieve strong performance on complex tasks by generating extended chain-of-thought (CoT) traces via reinforcement learning with verifiable rewards (RLVR). While current RLVR methods have achieved strong results with correctness-based reward signals, they provide limited guidance on the quality of the reasoning process itself, leaving the internal reasoning structure largely unoptimized. Through empirical analysis across multiple model families, we identify a consistent pattern: correct reasoning trac es exhibit more frequent and larger token-level entropy drops within the thinking phase than incorrect ones. We propose ERR+, a two-phase RLVR framework grounded in this observation. The first phase trains with the Entropy Relief Reward (ERR), a bonus proportional to cumulative token-level entropy drops in the thinking phase, log-normalized by response length. Unlike prior methods that suppress entropy, ERR rewards the resolution of uncertainty while leaving exploratory high-entropy states unconstrained. The second phase introduces the Robust Relative Efficiency Reward, which scores each response's length against co-generated peers via a $\tanh$-transformed within-group $z$-score. We provide a formal analysis showing that joint optimization of the two objectives induces gradient conflict in early training, motivating the sequential design . Experiments on five datasets demonstrate consistent improvements in both accuracy and response conciseness across model backbones. Our code is available at this https URL
• 中文摘要 大型推理模型通过通过可验证奖励的强化学习（RLVR）生成延伸思维链（CoT）痕迹，从而在复杂任务中表现出色。虽然当前基于正确性的RLVR奖励信号取得了强劲效果，但它们对推理过程本身质量的指导有限，导致内部推理结构大多未得到优化。通过跨多个模型家族的实证分析，我们发现了一个一致的模式：正确的推理轨迹在思考阶段表现出比错误阶段更频繁且更大的代币级熵下降。我们提出了ERR+，一个基于该观察的两阶段RLVR框架。第一阶段训练的是熵缓解奖励（ERR），该奖励与思考阶段累计的代币级熵下降成正比，按响应长度进行对数归一化。与以往抑制熵的方法不同，ERR奖励不确定性的解决，同时保持探索性高熵状态不受约束。第二阶段引入了稳健相对效率奖励，通过组内的$\tanh$变换$z$评分，对共生成的同伴对每个响应的长度进行评分。我们提供了形式分析，表明联合优化这两个目标在早期训练中引发梯度冲突，从而推动顺序设计。在五个数据集上的实验显示，模型骨干在准确性和响应简洁性方面均有持续提升。我们的代码可在 https URL 获取

Efficient Geothermal Well-Control Optimization via Diffusion-Surrogate Reinforcement Learning

通过扩散-替代强化学习实现高效的地热井控制优化

• Authors: Ruimin Dai, Guodong Chen, Randy Harsuko, Kunpeng Liu, Nori Nakata
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.28791
• Pdf link: https://arxiv.org/pdf/2608.28791
• Abstract Real-time decision-making for enhanced geothermal systems (EGS) is challenging because long-term production periods involve high-dimensional control spaces and a large number of time-consuming high-fidelity hydrothermal simulations. Reinforcement learning provides a natural framework for state-dependent sequential control, but direct policy training with numerical simulators is computationally expensive. To address this issue, we propose a diffusion-surrogate guided reinforcement learning framework for long-horizon EGS well-control optimization. The reservoir temperature and pressure fields are used as system states, while injection rates are selected as control actions. A learned surrogate environment is constructed using conditional diffusion models to predict the evolution of reservoir temperature and pressure fields and a separate reward model to estimate the corresponding economic return. The surrogate environment is then integrated with Proximal Policy Optimization (PPO) for efficient policy training. Experiments on a fractured EGS benchmark show that the diffusion surrogate can accurately reproduce reservoir-state evolution over multiple control stages. The resulting surrogate-assisted PPO policy achieves competitive well-control performance compared with direct simulator-based PPO and existing optimization methods, while substantially reducing the dependence on expensive high-fidelity simulations. These results demonstrate the potential of diffusion-based surrogate environments for efficient reinforcement learning in geothermal well-control optimization.
• 中文摘要 增强型地热系统（EGS）的实时决策具有挑战性，因为长期生产期涉及高维控制空间和大量耗时的高保真热液模拟。强化学习为状态依赖的顺序控制提供了自然框架，但直接用数值模拟器进行策略训练计算成本高昂。为解决这一问题，我们提出了一种扩散-代理引导强化学习框架，用于长视野EGS井控优化。储层温度和压力场作为系统状态，注入速率作为控制动作选择。通过条件扩散模型构建学习替代环境，预测油藏温度和压力场的演变，并构建独立奖励模型估算相应的经济回报。随后将替代环境与近端策略优化（PPO）集成，实现高效的策略训练。在破碎EGS基准测试上的实验表明，扩散替代体能够准确重现多个控制阶段的油藏态演变。由此产生的替代辅助PPO策略在井控性能上与基于模拟器的直接PPO和现有优化方法相比具有竞争力，同时大幅减少了对昂贵高保真模拟的依赖。这些结果展示了基于扩散的替代环境在地热井控制优化中高效强化学习的潜力。

Designing, Deployment and Field Testing of C2Stack for Networked Intelligent Software-Defined UAVs

C2Stack 的网络智能软件定义无人机设计、部署及现场测试

• Authors: Maxwell McManus, Zhaoxi Zhang, Sidharth Santhi Nivas, Yuqing Cui, Prem Sagar Pattanshetty Vasanth Kumar, Chenzhi Zhao, Nicholas Mastronarde, George Sklivanitis, Dimitris Pados, Elizabeth Serena Bentley, Zhangyu Guan
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.28854
• Pdf link: https://arxiv.org/pdf/2608.28854
• Abstract Unmanned Aerial Vehicles (UAVs) are emerging as critical enablers of next-generation wireless networking and autonomous systems. Despite their potential, deploying and testing networked UAV systems in real-world environments remains challenging, largely due to the absence of well-developed, end-to-end, ready-to-use protocol stacks. To fill this gap, we present C2Stack, a configurable protocol stack and experimental framework designed for real-time control, evaluation, and optimization of UAV networks. C2Stack incorporates a modular control plane, referred to as the~C2Stack Network Operating System (CNOS), alongside a programmable data plane that exposes APIs for cross-layer algorithm development, digital twin integration, and autonomous swarm control. In this article, we share our experience with the deployment and testing of C2Stack. We implemented C2Stack on a custom UAV swarm platform that integrates multiprocessor system-on-chip (MPSoC) radios with Intel NUC computing modules, enabling interoperability with various RF front ends. Field trials were conducted in both netted environments and large-scale outdoor test ranges, focusing on two representative use cases: (i) network utility maximization through online reinforcement learning, and (ii) collaborative interference source localization. The experiments demonstrate the feasibility of real-time, data-driven optimization in dynamic aerial environments, while also revealing practical challenges in field deployments of networked UAV systems, including power constraints, sensing limitations, and deployment logistics. We have made C2Stack source code available to the community under the MIT License, with the goal of establishing it as a foundational framework for experimental research on intelligent networked aerial systems.
• 中文摘要 无人机（UAV）正作为下一代无线网络和自主系统的关键推动力而兴。尽管具有潜力，但在现实环境中部署和测试网络无人机系统仍具挑战性，主要原因是缺乏成熟的端到端即用协议栈。为填补这一空白，我们介绍了C2Stack，一款可配置的协议栈和实验框架，旨在实时控制、评估和优化无人机网络。C2Stack集成了一个模块化控制平面，称为~C2Stack网络操作系统（CNOS），并配备可编程数据平面，提供跨层算法开发、数字孪生集成和自主群体控制的API。本文将分享我们对C2Stack部署和测试的经验。我们在一个定制的无人机群平台上实现了C2Stack，该平台集成了多处理器系统单片（MPSoC）无线电与Intel NUC计算模块，实现了与多种射频前端的互操作性。现场试验在网状环境和大型户外测试区进行了，重点关注两个典型用例：（i）通过在线强化学习实现网络效用最大化，以及（ii）协同干扰源定位。这些实验展示了动态空中环境中实时、数据驱动优化的可行性，同时也揭示了网络无人机系统现场部署中的实际挑战，包括功率限制、感测限制和部署物流。我们已将C2Stack源代码以MIT许可证形式提供给社区，旨在将其确立为智能网络空中系统实验研究的基础框架。

The Halt Vector: Internalizing a Causal Steering Intervention for Efficient Reasoning

停机向量：内化因果引导干预以实现高效推理

• Authors: Dylan Jayabahu, Tinuade Adeleke
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.28859
• Pdf link: https://arxiv.org/pdf/2608.28859
• Abstract Reasoning models do not stop when they know the answer. On DeepSeek-R1-Distill-Qwen-7B the chain of thought runs about twice as long as the model's own answer probability takes to settle, and how much of that excess is removable varies from problem to problem, so a global length penalty cannot take it out. We take it out by internalizing a causal interpretability finding into the weights. The mechanism is a halt vector: a difference-of-means direction at layer 18 of this model whose steering strength controls how long it thinks, while a replicated value axis does nothing. Installing that intervention in the weights is harder than it looks. Maximizing the scalar projection onto the direction corrupts the off-axis dimensions a frozen downstream reader depends on, and generation gets longer instead of shorter; what works is reconstructing the whole steered activation with those dimensions pinned to their natural values. Fit from 24 problems and no reinforcement learning, the halt removes about a quarter of the thinking at held accuracy across five unseen benchmarks, and the cut tracks each problem's own removable slack at 0.70. It also closes a non-termination pathology that grows with difficulty and that a decoding-time confidence hook makes worse. We do not claim to beat a well-tuned length penalty or decoding-time early exit on the raw trade-off; the contribution is how the halt is obtained.
• 中文摘要 推理模型在知道答案时不会停止。在DeepSeek-R1-Distill-Qwen-7B上，思考链大约是模型自身答案概率稳定所需的两倍，且可去除多余的部分因问题而异，因此全局长度惩罚无法消除。我们通过将因果可解释性发现内化到权重中来消除它。机制是停机矢量：该模型第18层的差值方向，其引导强度控制思考时间长短，而复制的值轴则无效。在权重中安装这种干预比看起来更难。最大化标量投影到方向会破坏冻结下游读器依赖的非轴维度，生成时间变长而非缩短;有效的方法是重建整个引导激活，并将这些维度钉在自然值上。该拟合基于24个问题且无强化学习，暂停在五个未见基准中去除约四分之一的保持准确性思维，切割追踪每个问题自身的可移除余裕为0.70。它还结束了一种难以加剧的非终止病态，而解码时间置信钩则加剧了这一问题。我们不声称能在原始权衡上克服适当调校的长度惩罚或解码时间提前退出;贡献在于停止的获得方式。

Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management

用于无线资源管理的混合离线-在线多代理决策变换器

• Authors: Yiming Zhang, Kun Yang, Cong Shen, Dongning Guo
• Subjects: Subjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.28878
• Pdf link: https://arxiv.org/pdf/2608.28878
• Abstract This paper develops a hybrid offline-online multi-agent reinforcement learning framework based on decision transformers. The policy is first pretrained offline via supervised sequence modeling of trajectories generated by existing policies, providing a safe and sample-efficient initialization. It is then fine-tuned online using a hybrid objective that incorporates critic-guided gradients, enabling performance improvements beyond the offline policy. To facilitate stable offline-to-online transfer and effective multi-agent coordination, the framework incorporates return-weighted sampling, a critic conditioned on neighbors' actions, and neighborhood-correlated exploration. The approach is fully distributed: both training and execution rely only on local observations and limited information exchange among neighboring agents. Evaluations with dynamic traffic arrivals in two settings: (i) joint scheduling and power allocation and (ii) coordinated beamforming, show that the proposed method achieves quality-of-service (QoS) performance comparable to centralized methods. Moreover, when pretrained on lower-quality datasets, online fine-tuning is also observed to surpass the initial offline policy. These results demonstrate a promising learning-based alternative for wireless resource management.
• 中文摘要 本文基于决策变换器开发了一个混合离线-在线多智能体强化学习框架。该策略首先通过对现有策略生成轨迹的监督序列建模在离线预训练，提供安全且样本高效的初始化。随后通过混合目标进行在线微调，结合批评者引导梯度，实现超越离线策略的性能提升。为促进离线到在线的稳定传输和有效的多智能体协调，该框架结合了返回加权抽样、基于邻居行为的批评者以及邻域相关探索。该方法完全分布式：训练和执行均仅依赖局部观察和邻近代理间有限的信息交换。在两种环境下对动态流量到达的评估：（i）联合调度与功率分配，以及（ii）协调波束成形，表明所提方法实现的服务质量（QoS）性能可与集中化方法相当。此外，在低质量数据集上预训练时，在线微调还能超越初始离线策略。这些结果展示了一种有前景的基于学习的无线资源管理替代方案。

Continuity-Free Near-Minimax Leading-Order Regret for CVaR-UCBVI

CVaR-UCBVI的无连续性近极大领先阶遗憾

• Authors: Yuanlong Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.28960
• Pdf link: https://arxiv.org/pdf/2608.28960
• Abstract For finite-horizon tabular CVaR reinforcement learning, prior work proves a $\widetilde{O}(\tau^{-1}\sqrt{SAK})$ leading regret bound for arbitrary normalized return laws and the sharper $\widetilde{O}(\sqrt{SAK/\tau})$ rate under a density lower bound. We show that the same Bernstein CVaR-UCBVI algorithm attains the sharper rate without continuity assumptions. The key is a selected-budget self-bound: the conditional variance of the episode shortfall is at most $\tau$ plus the value-estimation width. Substitution into the original Bernstein decomposition yields, with high probability, $\widetilde{O}(\sqrt{SAK/\tau}+(SAHK^{1/4}+S^2AH)/\tau)$ regret for arbitrary normalized return laws, including atomic, mixed, and continuous laws. The $\tau^{-1/2}$ leading term matches the expected-regret minimax lower bound up to logarithmic factors. Thus Bernstein CVaR-UCBVI is minimax-optimal over the full return-law class in the leading-order regime; the lower-order terms retain their $\tau^{-1}$ dependence.
• 中文摘要 对于有限视界表CVaR强化学习，先前工作证明了任意归一化返回律和密度下更锐利的$\widetilde{O}（\sqrt{SAK}）$为前导后悔界，适用于任意归一化返回定律和更尖锐的$\widetilde{O}（\sqrt{SAK/\tau}）}}率。我们证明了同一个Bernstein CVaR-UCBVI算法在不假设连续性的情况下也能达到更锐利的速率。关键是选定预算自限：集数缺失的条件方差最多为$\tau$加上价值估计宽度。将替换到原始伯恩斯坦分解中，对于任意归一化收益率律（包括原子律、混合律和连续律）的高概率产生$\widetilde{O}（\sqrt{SAK/\tau}+（SAHK^{1/4}+S^2AH）/\tau）$ 后悔。$\tau^{-1/2}$ 首项与预期-后悔极大极大值下界匹配，直达对数因子。因此，Bernstein CVaR-UCBVI 在主导阶态下对整个收益律类是极小极大最优;低阶项保留其 $\tau^{-1}$ 依赖性。

The Role of Network Topology and Opponent Information in Shaping Cooperation in Multi-Agent Reinforcement Learning Systems

网络拓扑和对手信息在塑造多智能体强化学习系统合作中的作用

• Authors: Seongho Son, Stephen Hailes, Mirco Musolesi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.28977
• Pdf link: https://arxiv.org/pdf/2608.28977
• Abstract Several works have investigated the influence of graph topology on cooperation among artificial agents, while the majority of the literature has focused on modelling agents' adaptation through strategy imitation, which relies solely on the cumulative payoffs of others. This paper investigates scenarios in which each agent learns to play the two-player Iterated Prisoner's Dilemma (IPD) using deep reinforcement learning. Each agent is represented as a node in a graph, where its neighbours constitute the pool of opponents with whom it can interact. During each IPD episode, agents are provided with different types of information about their opponent, consisting of action history and opponent identity. Experimental results across different graph topologies show that the number of neighbours per node and the average path length are the main factors affecting the emergence of cooperation. We also show that, while partner selection fosters mutual cooperation by limiting the diversity of the opponent pool, providing agents with the identity of their opponent hinders the proliferation of cooperative strategies.
• 中文摘要 多项研究研究图拓扑对人工代理间合作的影响，而大多数文献则聚焦于通过策略模仿来模拟代理的适应，该策略完全依赖于他人累积收益。本文探讨了每个代理通过深度强化学习学习进行双人迭代囚徒困境（IPD）的情景。每个代理被表示为图中的一个节点，其邻居构成了可交互的对手池。在每个IPD阶段，代理会获得关于对手的不同类型信息，包括行动历史和对手身份。不同图拓扑的实验结果表明，每个节点邻居数量和平均路径长度是影响合作出现的主要因素。我们还表明，虽然合作伙伴选择通过限制对手池的多样性促进了相互合作，但向智能体提供对手身份却阻碍了合作策略的扩散。

Teaching Robot Policies to Humans Using Erroneous Examples

用错误例子教导人类机器人政策

• Authors: Rithika Narayan, Suresh Kumaar Jayaraman, Henny Admoni
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.29023
• Pdf link: https://arxiv.org/pdf/2608.29023
• Abstract Human-robot collaboration describes the process of humans and autonomous agents working together to accomplish common goals. This process is facilitated best when robot policies, or behaviors in different situations, are made transparent to humans. Demonstration-based explanations have been a focus of human-robot collaboration research, and the field has frequently drawn upon literature from education to improve how humans are taught robot policies. However, no single teaching method has been proven effective across domains, difficulties, learners, and other variables; the question of how humans can most effectively be taught robot policies remains open. In traditional classrooms, learners are shown erroneous examples, in which they reflect on and correct incorrect responses to understand common pitfalls when learning a concept. We propose using erroneous examples to teach robot policies, extending an existing policy teaching framework. We conduct a user study in which participants view incorrect demonstrations of robot behavior and correct the actions to align with the actual policy. Our findings suggest that viewing these incorrect demonstrations and verbalizing one's reasoning in predicting a robot's actions improves retention of the policy over time, in agreement with the effect of erroneous examples in classrooms. We also categorize participants into distinct learning styles and establish that participants using inverse reinforcement learning-like reasoning perform best on policy prediction tasks. With this work, we aim to advance the methods by which robots educate humans on their policies.
• 中文摘要 人机协作描述了人类与自主智能体协作以实现共同目标的过程。当机器人政策或不同情境下的行为对人类透明时，这一过程最为顺利。基于演示的解释一直是人机协作研究的重点，该领域也经常借鉴教育文献来改进人类学习机器人政策的方式。然而，尚无单一教学方法被证明跨领域、困难、学习者及其他变量有效;如何最有效地教授人类机器人政策的问题仍然悬而未决。在传统课堂中，学习者会看到错误的例子，反思并纠正错误回答，以理解学习概念时常见的陷阱。我们提议使用错误的例子来教授机器人政策，扩展现有的政策教学框架。我们进行了用户研究，参与者看到错误的机器人行为演示，并纠正其行为以符合实际政策。我们的发现表明，观察这些错误演示并口头表达预测机器人行为的推理，有助于随着时间推移提升政策的记忆，这与课堂中错误示例的影响相符。我们还将参与者分类为不同的学习风格，并确定采用逆强化学习类推理的参与者在政策预测任务中表现最佳。通过这项工作，我们旨在推动机器人教育人类了解其政策的方法。

Learning to Follow In-Context Watermark Instructions via Self-Distillation

通过自我蒸馏学习遵循上下文中的水印说明

• Authors: Yepeng Liu, Tianyi Chen, Xuandong Zhao, Dawn Song, Yuheng Bu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29030
• Pdf link: https://arxiv.org/pdf/2608.29030
• Abstract In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response. It thus equips LLMs with a watermarking interface that third parties can invoke without access to model internals. Its reliability hinges on the LLM following the instruction without degrading answer quality, yet how well current LLMs do so has not been measured. We introduce $\mathsf{ICWBench}$, a benchmark of three verifiable ICW instruction families, each scored on both detectability and answer quality. Evaluating 14 frontier proprietary and open-source LLMs, we find that none of the evaluated LLMs achieves both objectives across all three families. To address this, we propose a self-contained two-stage training method, requiring no distillation from a stronger model, no manual annotation, and no pre-existing ICW IF ability. The first stage, self-distillation with logits perturbation (SDLP), uses the same base LLM as both teacher and student: an instruction-equivalent decoding-time logits perturbation makes the teacher follow the ICW instruction, and the student is trained to match the teacher's output distribution. The second stage applies reinforcement learning with the automatic verifier as the reward. Applied to Qwen3-14B and GPT-OSS-20B, our method raises average TPR@$1\%$FPR across three ICW instructions from $0.100$ to $0.974$ and from $0.337$ to $0.968$, respectively, while maintaining high response quality under both perplexity evaluation and LLM-as-a-Judge.
• 中文摘要 上下文水印（ICW）在查询前附加指令，要求模型在响应中嵌入统计可检测的信号。因此，它为LLM配备了一个水印接口，第三方可以在无需访问模型内部的情况下调用。其可靠性依赖于LLM遵循指令而不降低答案质量，但目前LLM的执行效果尚未被测量。我们引入$\mathsf{ICWBench}$，这是一个基于三个可验证ICW指令族的基准测试，每个族在可检测性和答案质量上均有评分。评估14个前沿专有和开源LLM时发现，没有任何评估的LLM在这三类LLM中同时实现这两个目标。为此，我们提出了一种自包含的两阶段训练方法，无需从更强模型提炼，无需手动注释，也无需预先的ICW IF能力。第一阶段是带logit扰动的自蒸馏（SDLP），作为教师和学生使用相同的基础LLM：指令等效解码时间logit扰动使教师遵循ICW指令，学生则被训练以匹配教师的输出分布。第二阶段应用强化学习，奖励自动验证器。应用于Qwen3-14B和GPT-OSS-20B，我们的方法分别将三条ICW指令的平均值提升TPR@$1\%$FPR从$0.100$提升到$0.974$，从$0.337$提升到$0.968$，同时在困惑度评估和LLM作为评判下保持高响应质量。

PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning

PathBridger：离线目标条件强化学习的子目标桥梁

• Authors: Soohyun Choi, Seonvin Cho, Songnam Hong
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.29061
• Pdf link: https://arxiv.org/pdf/2608.29061
• Abstract Offline goal-conditioned reinforcement learning (GCRL) aims to learn policies for reaching diverse goals entirely from fixed trajectory data. Long-horizon offline GCRL remains challenging because sparse goal-reaching signals must be propagated over many steps, while execution errors cannot be corrected through additional environment interaction. Existing methods address these challenges by improving long-range value estimation or reducing the effective decision horizon through subgoals, options, and action chunks. In several hierarchical methods, however, a selected subgoal specifies where to go, while the intervening state-space path remains implicit in an endpoint-conditioned low-level policy. To address this interface, we propose PathBridger, a hierarchical offline GCRL method that explicitly connects subgoal selection to short-horizon execution. PathBridger constructs a state-space bridge toward the selected intermediate endpoint and decodes it into a short executable action chunk using an inverse dynamics model. Experiments across the evaluated OGBench tasks demonstrate strong aggregate performance, with particularly large gains on the multi-object Cube manipulation tasks. Code: this https URL
• 中文摘要 离线目标条件强化学习（GCRL）旨在完全从固定轨迹数据中学习实现多样化目标的策略。长视野离线GCRL依然具有挑战性，因为稀疏的目标达成信号必须跨越多个步骤传播，而执行错误无法通过额外的环境交互纠正。现有方法通过改进长期价值估计或通过子目标、选项和动作块缩减有效决策视野来应对这些挑战。然而，在若干分层方法中，选定的子目标指定了去向，而中间的状态空间路径则隐含于端点条件的低层策略中。为解决这一接口，我们提出了PathBridger，一种分层离线GCRL方法，明确将子目标选择与短视距执行连接起来。PathBridger 构建一个状态空间桥，指向选定的中间端点，并利用逆动力学模型将其解码为一个短的可执行动作块。在评估的 OGBench 任务中，实验显示出强劲的总体性能，尤其是在多对象 Cube 操作任务上取得显著提升。代码：此 https URL

Titans-QFWP: A Regime-Aware Hybrid Quantum Fast Weight Programmer for Portfolio Optimization

Titans-QFWP：一款用于投资组合优化的区域感知混合量子快速权重编程器

• Authors: Ming-Kai Hung, Jun-Hao Chen, Yun-Cheng Tsai, Samuel Yen-Chi Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.29093
• Pdf link: https://arxiv.org/pdf/2608.29093
• Abstract We propose Titans-QFWP, a hybrid reinforcement learning architecture integrating a Quantum Fast Weight Programmer with Titans-style memory (Persistence, Surprise, and Forgetting) for adaptive portfolio optimization. To address high-dimensional market features, we introduce an enhanced A3C^2 framework with Hungarian-aligned K-means clustering and scaled log-return rewards. Evaluated on 468 S&P 500 stocks under an Equal-Parameter-Count (EPC) benchmark with approximately 3,000 trainable parameters, Titans-QFWP achieves strong performance (median ARR 0.4260, Calmar 8.5504, IR 0.8427). Ablation results reveal that quantum gating fundamentally reshapes memory component roles, with Persistence supporting drawdown control, Surprise contributing to return generation, and Forgetting providing additional stabilization. By stabilizing these quantum representations, the model enables defensive allocation during market drawdowns while preserving upside potential.
• 中文摘要 我们提出了Titans-QFWP，一种混合强化学习架构，将量子快速权重程序员与Titans风格的记忆（持久性、惊喜和遗忘）整合，实现自适应投资组合优化。为解决高维市场特征，我们引入了增强型A3C^2框架，采用匈牙利对齐的K均值聚类和缩放对数回报奖励。在468只标普500股票中，基于相同参数计数（EPC）基准，约3000个可训练参数，Titans-QFWP表现强劲（中位ARR 0.4260，Calmar 8.5504，IR 0.8427）。消融结果显示，量子门控从根本上重塑了记忆组成部分的角色，持久性支持回撤控制，惊喜促进回报生成，遗忘则提供额外稳定。通过稳定这些量子表示，模型在市场回撤时实现防御性配置，同时保持上涨潜力。

Beyond Correctness: Validity-Oriented Evaluation of Biomedical LLM Judges

超越正确性：生物医学LLM评审的效度导向评估

• Authors: Rodrigo de Oliveira, Federico Pittino, James Gwinnutt, Jay Nanavati
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29127
• Pdf link: https://arxiv.org/pdf/2608.29127
• Abstract We propose a scalable, validity-oriented pipeline for evaluating biomedical LLM judges when high-quality human judgments are scarce. First, we augment existing human-labelled biomedical benchmarks with deterministic, metric-grounded mutations that produce auditable preference pairs. Second, we evaluate judges beyond aggregate correctness using three deployment-relevant dimensions: correctness against metric-derived gold labels, robustness under repeated stochastic sampling, and compliance with the requested output format. We use this pipeline to assess Llama-3.1-8B-Instruct under four regimes: (1) base, using the instruct model as is; (2) SFT, distillation-based supervised fine-tuning only; (3) RL, GRPO-based reinforcement learning only; and (4) SFT$\rightarrow$RL, SFT followed by RL. The base and single-stage regimes struggle on structured medical discrimination such as PICO extraction and clinical calculations, whereas SFT$\rightarrow$RL performs best across correctness, compliance, and robustness; gains concentrate on decomposable tasks (PICO, MedCalc), at times matching or outperforming frontier models.
• 中文摘要 我们提出了一个可扩展、以效度为导向的流程，用于在高质量人类判断稀缺时评估生物医学LLM评判。首先，我们用确定性、度量基础突变补充现有的人类标记生物医学基准，产生可审计的偏好对。其次，我们通过三个部署相关维度评估评判，超越汇总正确性：针对度量衍生金标签的正确性、反复随机抽样下的鲁棒性以及符合要求的输出格式。我们利用该流程在四个模式下评估Llama-3.1-8B-Instruct：（1）基于基础，使用原有的intrain模型;（2）基于蒸馏的SFT，仅监督微调;（3）基于GRPO的强化学习，仅RL;以及（4）SFT$\rightarrow$RL，SFT后再进行RL。基础和单阶段方案在结构化医学区分（如PICO提取和临床计算）上表现不佳，而SFT$\rightarrow$RL在正确性、合规性和鲁棒性方面表现最佳;增益集中于可分解任务（PICO、MedCalc），有时甚至超过前沿模型。

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

入口处锁着，内部敞开：RLVR缩小了解答空间

• Authors: Qiancheng Zhou, Ruizhe Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.29188
• Pdf link: https://arxiv.org/pdf/2608.29188
• Abstract Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: this https URL.
• 中文摘要 带可验证奖励的强化学习（RLVR）显著提高了单样本准确率（pass@1），但会导致策略的解空间收缩，降低测试时间缩放的收益。本研究探讨了推理轨迹中这种广度的丧失：策略是否未能访问有效的解族，还是在启动后未能执行计算？为将访问与执行区分开来，我们分析了倒计时任务，其解空间可穷举为由第一个操作数和算符定义的离散入口族，跨越 PPO 在 Qwen2.5-3B 和 GRPO 在 Qwen2.5-3B-Instruct。在两种训练设置中，解的覆盖率下降了多达67%，甚至在所有检查点解决的问题上也减半。我们证明这种收缩在入口处高度集中：每个标记的似然偏移在第一次算术运算前比下游推理时大11倍至16倍。仅提供未被选中的入口前缀，低访问族的完成率可恢复一个数量级以上（PPO下为0.018->0.212），表明替代解仍可执行但不再启动。基于这一局部化，我们发现表面提示未能恢复多样性，但针对入口的干预成功：带早期检查点的后层参数插值使解覆盖率提升37%，且无pass@1损失。最后，我们展示了早期步骤熵坍缩在7B和14B模型的六个数学基准中反复出现，但并非推理优化的必然副产品：SFT基线保留了超过两倍的覆盖率，分阶段的SFT-DPO-RLVR流水线保留了早期步骤熵。总之，推理广度是在门口丢失，而非室内。代码：此https URL。

A-MADiff: Attention-Guided Multi-Agent DRL with Diffusion Policies for Memory-Aware Task Orchestration in Mobile AIGC Networks

A-MADiff：带有扩散策略的注意力引导多智能体日程学习，用于移动AIGC网络中的内存感知任务编排

• Authors: Chongzhi Wu, Zhengtao Li, Jiawen Kang, Jinbo Wen, Xiaohuan Li, Maomao Zhang, Ekram Hossain
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.29255
• Pdf link: https://arxiv.org/pdf/2608.29255
• Abstract Artificial Intelligence-Generated Content (AIGC) services employ Generative AI (GenAI) models to automatically generate diverse content. Mobile AIGC networks host GenAI models on edge-located AIGC Service Providers (ASPs) to deliver low-latency and personalized AIGC services for mobile users. However, AIGC inference tasks typically occupy GPU memory until task completion, causing GPU memory exhaustion at serving ASPs and triggering out-of-memory failures rather than merely increasing service latency. Existing studies on AIGC task orchestration have largely overlooked GPU memory feasibility constraints. To address this issue, we develop a cooperative multi-agent orchestration framework, in which each edge node is equipped with a scheduling agent to route tasks to local ASPs or neighboring edge nodes. Since scheduling agents make decisions based only on local observations, while peer offloading couples their resource states and long-term utilities, we formulate the orchestration process as a cooperative Decentralized Partially Observable Markov Decision Process (Dec-POMDP). To solve the Dec-POMDP, we propose an \underline{A}ttention-guided \underline{M}ulti-\underline{A}gent deep reinforcement learning algorithm with \underline{Diff}usion policies (A-MADiff) under the centralized training with a decentralized execution paradigm. A-MADiff employs diffusion-based decentralized actors to generate multi-modal preferences over feasible orchestration actions, and an attention-guided centralized critic to estimate per-agent values from cross-agent states under GPU memory heterogeneity. Numerical results demonstrate that A-MADiff significantly improves the cumulative reward over the state-of-the-art baseline.
• 中文摘要 人工智能生成内容（AIGC）服务采用生成式人工智能（GenAI）模型自动生成多样化内容。移动AIGC网络在边缘定位的AIGC服务提供商（ASP）上托管生成式AI模型，为移动用户提供低延迟且个性化的AIGC服务。然而，AIGC推理任务通常占用GPU内存直到任务完成，导致GPU内存耗尽，导致服务ASP时内存耗尽，并触发内存不足故障，而不仅仅是增加服务延迟。现有关于AIGC任务编排的研究大多忽视了GPU内存可行性限制。为解决这一问题，我们开发了一个协作式多代理编排框架，每个边缘节点配备调度代理，将任务路由到本地ASP或邻近边缘节点。由于调度代理仅基于局部观察做出决策，而对等卸载则将资源状态和长期效用耦合，我们将编排过程表述为合作式去中心化部分可观测马尔可夫决策过程（Dec-POMDP）。为求解Dec-POMDP，我们提出了一个\underline{ttention引导\underline{M}ulti-\underline{A}gent深度强化学习算法，采用\underline{Diff}usion策略（A-MADiff），在中心化训练和去中心化执行范式下。A-MADiff采用基于扩散的去中心化行为者生成多模态偏好，针对可行编排动作产生多模态偏好，并采用注意力引导的中心批评器，在GPU内存异构条件下估计跨代理状态的每代理值。数值结果表明，A-MADiff相比最先进的基线显著提升了累计奖励。

RACER: Reinforced Agent Collaboration for Explainable Reasoning on Knowledge Graphs

RACER：强化智能体协作，用于知识图谱的可解释推理

• Authors: Yuwei Lou, Hao Hu, Yuzhou Jiang, Zongfei Zhang, Liang Wang, Jincai Liu, Jidong Ge, Xianping Tao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29263
• Pdf link: https://arxiv.org/pdf/2608.29263
• Abstract Large Language Models (LLMs) often suffer from hallucination and struggle with complex reasoning tasks requiring multi-hop domain knowledge. While integrating Knowledge Graphs (KGs) provides a structured and verifiable information source, current KG-enhanced LLM paradigms usually rely on single-agent path extraction and fixed prompting, lacking adaptability and facing huge search spaces. To address these challenges, we propose RACER, a Reinforced Agent Collaboration framework for Explainable Reasoning on knowledge graphs. RACER employs a semantic-aware action pruning and teacher-guided reinforcement learning mechanism to efficiently extract high-quality reasoning pathways from large-scale KGs. Furthermore, to mitigate single-path generation pitfalls, we introduce a cross-task accumulated shared memory graph paired with an attention-driven multi-path knowledge refinement module. Finally, RACER orchestrates these components through a four-role multi-agent collaboration system (GraphAgent, TemplateAgent, AnswerAgent, and CriticAgent) to dynamically refine prompts and evaluate answers. Extensive experiments on CommonsenseQA and OpenBookQA datasets demonstrate that RACER significantly outperforms state-of-the-art KG-enhanced LLM baselines with an average improvement of 5\%, offering robust and highly interpretable reasoning capabilities.
• 中文摘要 大型语言模型（LLM）常常出现幻觉，并在处理需要多跳领域知识的复杂推理任务时遇到困难。虽然集成知识图谱（KGs）提供了结构化且可验证的信息源，但当前KG增强型LLM范式通常依赖单代理路径提取和固定提示，缺乏适应性且搜索空间巨大。为应对这些挑战，我们提出了RACER，这是一个用于知识图谱可解释推理的强化代理协作框架。RACER采用语义感知动作剪枝和教师引导强化学习机制，高效从大规模知识图中提取高质量的推理路径。此外，为减少单路径生成的陷阱，我们引入了跨任务累积共享记忆图，配合注意力驱动的多路径知识精炼模块。最后，RACER通过一个四角色多代理协作系统（GraphAgent、TemplateAgent、AnswerAgent和CriticAgent）协调这些组件，动态优化提示并评估答案。在CommonsenseQA和OpenBookQA数据集上的大量实验表明，RACER显著优于最先进的KG增强型LLM基线，平均提升5%，提供稳健且高度可解释的推理能力。

RAGDiffusion++: From Macro-Retrieval to Micro-Fidelity Alignment for Garment Generation

RAGDiffusion++：从宏检索到服装生成的微保真对齐

• Authors: Yuhan Li, Xianfeng Tan, Fangao Zeng, Wenxiang Shang, Pipei Huang, Hao Zhou, Zhiyu Jin, Wenjun Zhang, Bingbing Ni
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29280
• Pdf link: https://arxiv.org/pdf/2608.29280
• Abstract Standard clothing asset generation---restoring forward-facing flat-lay garment images from diverse real-world contexts---holds immense commercial value yet demands both macroscopic topological accuracy and microscopic physical fidelity. Although our previous work RAGDiffusion effectively eradicated large-scale structural hallucinations via retrieval-augmented macro-constraints, achieving industrial-grade micro-texture realism remains an unsolved bottleneck. We formally identify this limitation as High-Frequency Trajectory Collapse: supervised fine-tuning (SFT) converges to the conditional mean of the training distribution, which is dominated by smooth, low-frequency textures, causing high-frequency patterns (e.g., fabric weaves, intricate logos) to become nearly un-sampleable. Naively applying Reinforcement Learning (RL) post-training further triggers Artifact Hacking, where models exploit semantic biases in generic reward models by generating deceptive checkerboard noise. Our key insight is that RL can fundamentally reshape the sampling distribution of flow models---elevating the probability of high-fidelity trajectories under accurate reward guidance---while adversarial regularization prevents exploitation of reward blind spots. Realizing this principle requires three prerequisites: (i)inherent capacity, established through a 27,725-pair high-complexity garment dataset (STGarment-Plus) and a Dual-Image-Stream FLUX architecture upgrade; (ii)perceptive reward, provided by a novel attribute-aware reward model (Garment-RM) trained on 500K images via fine-grained contrastive learning, achieving 84.67% human preference accuracy; and (iii)hacking prevention, enforced by our Adversarial-Regularized GRPO (AR-GRPO) strategy that integrates a dynamic discriminator into the RL sampling trajectory to penalize artifacts while enriching authentic high-frequency details.
• 中文摘要 标准服装资产生成---从多种现实世界场景恢复前置平面服装图像---具有巨大的商业价值，但同时要求宏观拓扑精度和微观物理真实度。尽管我们之前的研究RAGDiffusion通过检索增强宏约束有效消除了大尺度结构幻觉，但实现工业级微纹理真实性仍是一个未解决的瓶颈。我们将这一限制正式称为高频轨迹坍缩：监督微调（SFT）收敛到训练分布的条件平均值，该分布以平滑的低频纹理为主，导致高频图案（如织物织物、复杂标志）几乎无法采样。天真地应用强化学习（RL）后训练会进一步触发人工物黑客，即模型利用通用奖励模型中的语义偏见，产生欺骗性的棋盘格噪声。我们的关键见解是，强化学习能够从根本上重塑流模型的采样分布---提高在精确奖励指导下高保真轨迹的概率---而对抗正则化则防止对奖励盲点的利用。实现这一原则需要三个前提条件：（i）通过27,725对高复杂度服装数据集（STGarment-Plus）和双图像流FLUX架构升级建立的固有容量;（ii）感知奖励，由一种新颖的属性感知奖励模型（Garment-RM）提供，该模型通过细粒度对比学习在50万张图像上训练，实现了84.67%的人类偏好准确率;以及（iii）黑客防护，通过我们的对抗正则化GRPO（AR-GRPO）策略执行，该策略将动态判别器整合进强化学习采样轨迹，惩罚伪影，同时丰富真实的高频细节。

When Do Larger Batches Help Scale LLM Reinforcement Learning?

什么时候，更大批量的学习有助于扩大LLM强化学习？

• Authors: Ziniu Li, Jinbo Wang, Guanhua Huang, Feiyuan Zhang, Pengbo Li, Alex Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29296
• Pdf link: https://arxiv.org/pdf/2608.29296
• Abstract Larger batches reduce the variance of stochastic gradients per update and are therefore often expected to accelerate training. Yet whether this statistical benefit translates into lower wall-clock time-to-target remains unclear, because each update consumes more samples and may take longer to execute. We study this tradeoff in reinforcement learning for large language models. We separate its algorithmic and systems effects by comparing learning and execution along their natural axes. At the algorithmic level, we compare configurations at equal cumulative sample counts while retuning batch-dependent hyperparameters. Over a bounded range of batch sizes, this procedure yields an approximately batch-size-invariant family whose members follow similar sample-indexed learning trajectories. At the systems level, we exploit the computational asymmetry between rollout generation and training: autoregressive generation is often memory-bandwidth-bound at low concurrency, whereas training work scales approximately with the number of processed tokens. Combining these two views yields a direct decision rule: a larger-batch configuration reduces time-to-target only when its throughput gain exceeds its samples-to-target penalty. Experiments with GRPO and PPO support both sides of this decomposition. At the algorithmic level, square-root learning-rate scaling with Adam produces approximately batch-size-invariant learning curves over a bounded range of batch sizes. At the systems level, larger batches improve generation throughput by up to 2.29x on fixed hardware. In GRPO, combining higher throughput with learning-rate retuning reduces time-to-target by up to 29%, whereas increasing the batch without retuning is slower despite its higher throughput.
• 中文摘要 更大批量减少每次更新随机梯度的方差，因此通常预期能加快训练速度。然而，这种统计优势是否转化为更低的壁钟到目标时间尚不清楚，因为每次更新消耗更多样本，执行时间可能更长。我们研究了大型语言模型强化学习中的这种权衡。我们通过比较学习和执行沿自然轴的算法和系统效应来区分。在算法层面，我们在相等的累计样本数下比较配置，同时重新调校批次依赖的超参数。在有界的批次规模范围内，该过程产生一个近似批次大小不变的家族，其成员遵循类似的样本索引学习轨迹。在系统层面，我们利用了rollout生成与训练之间的计算不对称性：自回归生成在低并发时通常受内存带宽限制，而训练工作则大致随处理令牌数量增长。结合这两种视角得出一个直接决策规则：大批量配置仅在其吞吐量增益超过样本到目标的惩罚时，才能缩短目标到达时间。GRPO和PPO的实验支持这种分解的两端。在算法层面，使用Adam的平方根学习率缩放在有界的批次大小范围内产生近似的批量不变学习曲线。在系统层面，较大的批次在固定硬件上使生成吞吐量提升高达2.29倍。在GRPO中，将更高吞吐量与学习率重新调谐结合可将目标时间缩短多达29%，而不重调的批量增加则较慢，尽管吞吐量更高。

Plant-Inspired AI: Plants as Inspiration for Novel Problem Formulations, and Two Case Studies

植物启发的人工智能：植物作为新颖问题表述的灵感，以及两个案例研究

• Authors: Deepayan Sanyal, Joel Michelson, Carla E. Cao, Adam B. Roddy, Maithilee Kunda
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29356
• Pdf link: https://arxiv.org/pdf/2608.29356
• Abstract Artificial Intelligence (AI) has long been inspired by studies of biological intelligence. Reinforcement learning, for instance, drew inspiration from studies involving animal learning and is now a powerful paradigm for solving many real-world problems. Recently, plant biologists have uncovered a wide range of complex behaviors in plants that enable them to flexibly adapt to variable environments. Here, we argue that such behavior can motivate new AI frameworks encompassing a range of problems overlooked by existing problem-solving frameworks such as supervised learning, tree search, and constraint satisfaction. We illustrate this idea with two examples of intelligent problem-solving in plants: (1) leaf mimicry in Boquila trifoliolata, a vine capable of altering its leaves' morphology to resemble those of multiple host trees simultaneously; and (2) coordinated root-shoot growth, wherein plants allocate resources across organ systems exploring distinct environments. While leaf mimicry is highly specific to Boquila, coordination of root-shoot growth is shared across most plants. For both examples, we capture underlying computational principles and identify problems fitting these frameworks that are currently unaddressed by AI. Finally, we outline preliminary task formulations and discuss how these formulations may be applied to non-plant problems.
• 中文摘要 人工智能（AI）长期以来一直受到生物智能研究的启发。例如，强化学习从涉及动物学习的研究中汲取灵感，如今已成为解决许多现实问题的强大范式。最近，植物生物学家发现了植物中各种复杂的行为，使它们能够灵活适应多变的环境。在这里，我们认为，这种行为能够激励新的人工智能框架，涵盖现有问题解决框架忽视的一系列问题，如监督学习、树木搜索和约束满足。我们用两个植物智能问题解决的例子来说明这一观点：（1）Boquila trifoliolata（一种能够同时改变叶片形态以模仿多种寄主树木的藤蔓）中的叶片拟态;以及（2）协调根茎生长，植物在探索不同环境的器官系统间分配资源。虽然叶片拟态在博基拉树上高度特异，但根茎生长的协调在大多数植物中是共享的。对于这两个例子，我们捕捉了潜在的计算原理，并识别了这些框架下目前AI尚未解决的问题。最后，我们概述了初步任务表述，并讨论这些表述如何应用于非植物问题。

LiteSearch-VL: Small Multimodal Search Agents via Trajectory Distillation and Synthetic Step-DPO

LiteSearch-VL：通过轨迹蒸馏和合成步进-DPO实现小型多模态搜索剂

• Authors: Saeed Khaki, Nima Safaei, Kamal Ginotra
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29357
• Pdf link: https://arxiv.org/pdf/2608.29357
• Abstract Multimodal search agents answer visual questions by interleaving image understanding, web retrieval, tool use, and evidence synthesis. Strong systems exist, but in two expensive regimes: proprietary frontier models such as GPT-5 and Gemini, or large open vision-language backbones trained with substantial agentic data and reinforcement learning. We ask a different question: when released agent trajectories are distilled into much smaller backbones under a single-node budget, what is actually transferred? We study this with LiteSearch-VL, a low-compute recipe for Qwen3-VL-2B and Qwen3-VL-4B that uses only released OpenSearch-VL trajectories, parameter-efficient LoRA adapters, and synthetic step-level preferences: DPO on GPT-5-generated hard negatives targeting five local failure modes (premature answer, wrong tool, weak query, repeated query, ignored image). Across 12,400 GPT-5-judged rollouts on SimpleVQA, FVQA, LiveVQA, and VDR-Bench-testmini, the dominant effect is behavioral rather than a uniform accuracy lift: full-trajectory supervised fine-tuning transfers the agent contract, taking the 2B model from almost never emitting a usable answer (1,237/1,240 no_answer rollouts) to 28.4% macro Pass@1, matching or slightly exceeding the off-the-shelf 4B base (25.6%). Synthetic preference learning and compact tool distillation act as refinements rather than phase transitions (best 4B configuration: 30.8% macro Pass@1). Finally, a controlled VDR step-budget ablation shows that extra search turns convert abstentions into wrong_entity errors rather than correct answers, identifying answer verification, not search depth, as the next bottleneck for small multimodal agents.
• 中文摘要 多模态搜索代理通过交织图像理解、网页检索、工具使用和证据综合来回答视觉问题。强系统存在，但存在两种昂贵的模式：专有前沿模型如GPT-5和Gemini，或是通过大量代理数据和强化学习训练的大型开放视觉语言骨干。我们提出另一个问题：当释放的代理轨迹被提炼成更小的骨干，且预算为单节点时，实际传输了什么？我们用LiteSearch-VL研究，这是一种低计算的Qwen3-VL-2B和Qwen3-VL-4B配方，仅使用发布的OpenSearch-VL轨迹、参数高效的LoRA适配器和合成的步级偏好：针对五种局部失败模式（过早回答、错误工具、弱查询、重复查询、忽略图像）的GPT-5生成硬否定进行DPO。在SimpleVQA、FVQA、LiveVQA和VDR-Bench-testmini上12,400次GPT-5评判的部署中，主导效应是行为层面，而非均匀的准确性提升：全轨迹监督微调转移了代理契约，使2B模型从几乎不输出可用答案（1,237/1,240次no_answer部署）提升到28.4%的宏Pass@1，达到或略高于现成的4B基础（25.6%）。合成偏好学习和紧凑工具蒸馏作为精细而非相变（最佳4B配置：30.8%宏Pass@1）。最后，受控的VDR步进预算消融显示，额外的搜索回合将弃权转化为wrong_entity错误而非正确答案，这表明小多模态代理的下一个瓶颈是答案验证而非搜索深度。

Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs

思考、观察与修订：多层次营销中对不一致性感知的视觉自我修正

• Authors: Yu Cheng, Arushi Goel, Hakan Bilen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.29374
• Pdf link: https://arxiv.org/pdf/2608.29374
• Abstract Tool-augmented multimodal reasoning integrates external tools (e.g., object detection, depth estimation) into multimodal large language models (MLLMs) to address perceptual bottlenecks in complex visual tasks. However, existing approaches rarely verify tool outputs, limiting their ability to detect and recover from tool failures. We propose ReVISE, a framework that equips MLLMs with verification and dynamic error recovery for tool-augmented reasoning. ReVISE introduces (1) a curated training dataset that supervises reflective behaviors, enabling models to validate tool-derived evidence, reformulate queries when visual mismatches arise, and fall back to intrinsic grounding when external tools are unreliable; and (2) a reinforcement learning based targeted rewards that encourage internal reflection and penalize spatial misalignment. Experiments on several benchmarks demonstrate consistent improvements over existing methods, highlighting the importance of error detection and correction in tool-augmented multimodal reasoning.
• 中文摘要 工具增强多模态推理将外部工具（如对象检测、深度估计）集成到多模态大型语言模型（MLLM），以解决复杂视觉任务中的感知瓶颈。然而，现有方法很少验证工具输出，限制了其检测和恢复工具故障的能力。我们提出了ReVISE，一种为MLLM配备验证和动态错误恢复功能，用于工具增强推理的框架。ReVIE引入了（1）一个策划的训练数据集，监督反思行为，使模型能够验证工具衍生证据，在视觉不匹配时重新表述查询，并在外部工具不可靠时回归内在基础;（2）基于强化学习的有针对性奖励，鼓励内在反思并惩罚空间错位。多个基准测试的实验显示，其对现有方法有持续的改进，凸显了工具增强多模推理中错误检测和纠正的重要性。

Personalized Recommender Systems for Gym Workouts: A Reinforcement Learning Approach

个性化健身房锻炼推荐系统：强化学习方法

• Authors: Roan Rosema, Helma Torkamaan, Masoud Mansoury
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2608.29409
• Pdf link: https://arxiv.org/pdf/2608.29409
• Abstract Workout recommender systems aim to help gym users complete effective and engaging training sessions. However, recommending exercises alone is insufficient, as a practical system must also determine appropriate sets, repetitions, and training loads, while adapting to user behavior such as skipping exercises. Existing approaches typically consider only a subset of these factors, limiting their applicability in real-world settings. In this paper, we extend workout recommendation from exercise selection to full workout prescription. We propose a reinforcement learning (RL)-based framework with four environments: exercise-only and full-prescription settings, each with and without skip-based interaction. The full-prescription environments recommend exercises, sets, repetitions, and load, while the skip-enabled environments use user skipping behavior for online personalization. Experiments with synthetic users show that modeling the full prescription task leads to higher rewards and greater user engagement than exercise-only recommendation, highlighting the importance of realistic workout planning in personalized gym recommender systems.
• 中文摘要 锻炼推荐系统旨在帮助健身房用户完成有效且有趣的训练课程。然而，仅仅推荐锻炼是不够的，因为实用系统还必须确定合适的组数、重复次数和训练负荷，同时适应用户行为，如跳过动作。现有方法通常只考虑这些因素的部分，限制了其在现实环境中的适用性。本文将锻炼建议从动作选择扩展到完整的锻炼处方。我们提出了一个基于强化学习（RL）的框架，包含四个环境：仅锻炼和全处方设置，每种环境均有跳跃交互和无跳跃交互。全处方环境推荐动作、组数、重复次数和负荷，而支持跳跃的环境则利用用户跳跃行为进行在线个性化。对合成用户的实验表明，建模完整处方任务相比仅限锻炼的建议，能带来更高的奖励和更高的用户参与度，凸显了个性化健身房推荐系统中现实化锻炼规划的重要性。

RL-based Network Slice Embedding over Space Division Multiplexed Elastic Optical Networks

基于强化学习的网络片嵌入空间分割复用弹性光网络

• Authors: Divya Khanure, Riti Gour†, Congzhou Li, Jason P. Jue
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.29444
• Pdf link: https://arxiv.org/pdf/2608.29444
• Abstract Network slicing over space-division-multiplexed elastic optical networks (SDM-EONs) requires jointly managing spectrum, spatial cores, and compute resources, a coupling that many existing studies ignore by treating compute placement independently from routing and spectrum decisions. This disconnect can cause the spectrum to be allocated along a path, only for the request to fail due to insufficient compute resources along the path, or may result in compute resources being allocated without consideration for spectrum resource availability on the path between compute nodes. We propose a path-constrained reinforcement learning framework that addresses compute node selection and RMCSA, being aware of both resources, restricting the RL agent's action space to nodes along $k$-shortest paths between request endpoints. Training incorporates reward shaping to improve robustness under high load. We propose PPO-Full (Proximal Policy Optimization-Full), which jointly selects compute nodes and routing paths via a multi-dimensional action space, against distance-based heuristics, a greedy baseline, and a decoupled VONE-DRL baseline on a 24-node USNET topology under hotspot traffic conditions. Results demonstrate consistent improvements in acceptance rate over all baselines at high load, with gains becoming more pronounced as traffic intensity increases.
• 中文摘要 在空间分割复用弹性光网络（SDM-EON）上的网络切片需要共同管理频谱、空间核心和计算资源，许多现有研究通过独立处理计算布置与路由和频谱决策而忽视了这一耦合。这种脱节可能导致频谱被分配到路径上，但由于路径上的计算资源不足而请求失败，或者可能导致计算节点间路径上的计算资源分配时未考虑频谱资源可用性。我们提出了一个路径约束强化学习框架，涵盖计算节点选择和RMCSA问题，同时关注这两种资源，限制RL代理的动作空间仅限于请求端点之间$k$最短路径上的节点。训练中包含奖励塑造，以提升高负载下的鲁棒性。我们提出了PPO-Full（近端策略优化-Full），它通过多维动作空间联合选择计算节点和路由路径，对抗基于距离的启发式、贪婪基线和解耦的VONE-DRL基线，且在热点交通条件下为24节点USNET拓扑。结果显示，在高负载下所有基线的接受率持续提升，随着流量强度的增加，这一提升也更加明显。

Reference-Grafting Matches Fine-Tuning at Eliciting Sandbagged Capabilities

参考嫁接与精细调谐匹配，诱导沙包能力

• Authors: Linh Le, Hong Kiat Tan, David Williams-King
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29458
• Pdf link: https://arxiv.org/pdf/2608.29458
• Abstract Sandbagging, in which a model deliberately underperforms on an evaluation despite retaining the underlying capability, threatens the safety evaluations that frontier-model governance depends on. The Elicitation Game found that fine-tuning elicits hidden capability from sandbagging model organisms whereas additive activation steering fails. We revisit that verdict with reference-grafting, which sets an activation's coordinate along a contrast direction to the value it takes in an honest reference, at a small set of circuits chosen by active learning. Across eleven fine-tuned password-locked models (1.5-32B, three architecture lineages), it recovers +94 to +101% of the honest-sandbagging gap, matching fine-tuning elicitation without weight updates or training labels; two to five paired examples suffice to fit the direction. Similar recovery holds for reinforcement-learning-induced sandbagging and for password-locked code generation. Grafting works because the fine-tuned lock is a thresholded gate: held-out accuracy stays at the sandbagged level until the grafted coordinate crosses a threshold near the honest reference, which is why additive steering and zeroing the coordinate often fail. The direction tracks how the sandbagging was induced rather than what is withheld -- fit on grade-school science (ARC) it elicits withheld hazardous knowledge (WMDP), yet prompting, fine-tuning, and reinforcement learning each carry a different direction. Circuit-breaking marks the boundary: it reroutes activations on every forward pass, so the fixed edits we test are re-broken downstream and do not restore coherent generation.
• 中文摘要 沙袋效应，即模型在评估中故意表现不佳，尽管保留了底层能力，威胁到前沿模型治理所依赖的安全评估。诱发游戏发现，微调能从模型生物的沙袋中引出隐藏能力，而加法激活引导则失败。我们通过引用嫁接重新审视这一结论，该方法将激活坐标与诚实参考中取值的对比方向设定，且该值由主动学习选定的一小组电路上。在11个微调密码锁定模型（1.5-32B，三条架构谱系）中，它恢复了诚实沙袋差距的+94%至+101%，匹配微调诱导，无需权重更新或训练标签;两到五个配对示例即可符合方向。类似的恢复方法也适用于强化学习引发的沙袋化和密码锁定的代码生成。嫁接之所以有效，是因为微调锁是一个阈值门槛：保持的准确性会保持在沙袋层面，直到嫁接坐标越过接近诚实参考的阈值，这也是为什么加法引导和坐标归零常常失败的原因。方向追踪的是沙袋化的诱导过程，而不是被隐瞒的内容——适合小学科学（ARC）引发隐匿危险知识（WMDP），但提示、微调和强化学习各自方向不同。断路标记了边界：它在每次前向传递时重新导向激活，因此我们测试的固定编辑在下游被重新破坏，无法恢复连贯生成。

A Causal Model for Locating and Unlocking Sandbagging in Model Organisms

定位和解锁模式生物中沙袋的因果模型

• Authors: Hong Kiat Tan, Linh Le, David Williams-King
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.29461
• Pdf link: https://arxiv.org/pdf/2608.29461
• Abstract Sandbagging models strategically underperform on evaluations while retaining the capabilities being measured. The evaluations that guide frontier-model deployment and governance then understate what these models can do. To understand the mechanism, we propose a causal model of how sandbagging is carried in the residual stream. Early layers write the sandbagging intent onto a single axis of the stream, and a later layer reads that axis and commits the answer. We study three instruction-tuned models (Qwen2.5-7B, Llama-3-8B, and Mistral-7B) and four ways of installing a sandbagging lock (prompting, fine-tuning, reinforcement learning, and circuit breaking). Each lock decides from the prompt whether to sandbag, and the fine-tuned and circuit-broken locks answer honestly whenever a password appears. The causal model predicts a window of layers, after the last sandbagging write and before the answer commit, in which a single-layer reference graft of the sandbagging axis to its honest value restores the full capability. The single-layer graft recovers the capability in 28 of the 33 runs of the prompted, fine-tuned, and RL-trained locks, with a median held-out recovery of 96%. The circuit-broken lock rewrites the whole state through a band of layers, and the single-layer graft fails at every layer. We therefore introduce a second intervention, context grafting, which replays the password's cached key/value activations so that every layer's attention reads them as additional context. Context grafting provably and empirically restores the full capability on all three models, and the recovery is surprisingly insensitive to the exact password content. More broadly, an auditor can use this causal model to design interventional auditing techniques for sandbagging models.
• 中文摘要 沙袋模型在保持被测量能力的同时，在评估中策略性地表现不佳。指导前沿模型部署和治理的评估则低估了这些模型的可行性。为理解机制，我们提出了沙袋化如何在残余流中承载的因果模型。早期层将沙袋化意图写入流的单一轴线，后期层读取该轴并提交答案。我们研究了三种指令调优模型（Qwen2.5-7B、Llama-3-8B 和 Mistral-7B）以及四种安装沙袋锁的方法（提示、微调、强化学习和断链）。每个锁根据提示决定是否沙袋，微调和电路破断锁在出现密码时诚实回答。因果模型预测在最后一次沙袋写入后至答复提交前的层窗口，在此期间，沙袋轴的单层参考嫁接到其真实值后，恢复了全部能力。单层嫁接在33次提示、微调和强化学习锁运行中恢复了28次，中位数保留恢复率为96%。电路断裂锁通过一带重写整个状态，单层嫁接在每一层都失败。因此我们引入第二种干预——上下文嫁接，它重放密码缓存的键值激活，使每一层的注意力都将其视为额外上下文。上下文嫁接可证明且实证地恢复了三种模型的全部能力，且恢复对密码具体内容的敏感度出乎意料地低。更广泛地说，审计员可以利用该因果模型设计沙袋模型的介入审计技术。

FuncRoom-Agent: Sequential Feed-Forward 3D Functional Indoor Scene Generation

FuncRoom-Agent：顺序前馈3D功能室内场景生成

• Authors: Hao Feng, Zhi Zuo, MingJian Liang, Jingyu Hu, Xiaowei Hu, Liupengfei Wu, Dian Zhang, Guoxin Fang, Zhengzhe Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.29519
• Pdf link: https://arxiv.org/pdf/2608.29519
• Abstract We introduce Function-Room Generation, a new indoor 3D scene generation setting that creates rooms supporting explicit functional goals rather than merely visually plausible layouts. Existing agentic and executable methods improve controllability, but often depend on costly test-time generate--evaluate--revise loops, making functional room generation slow and computationally expensive. We address this challenge with three technical contributions. First, we design a recursive domain-specific language to effectively organize the hierarchical object compositions required by functional rooms, from room structure and major furniture to dense support-surface and nested small objects. It represents rooms as staged executable programs with explicit geometric and functional relations. Second, we propose a sequential feed-forward scene construction framework that distills recursive construction traces into a scene construction expert. At inference time, the expert writes executable DSL code stage by stage, and a deterministic executor directly instantiates each stage without teacher agents, online critics, or iterative repair. Third, we introduce ScenePRM, an execution-grounded process reward framework that improves the expert through reinforcement learning with functional, geometric, relational, and future-constructability feedback. We further establish a function-oriented benchmark and show state-of-the-art performance on both general indoor scene generation and function-room generation, achieving stronger functional completeness, relation correctness, geometric executability, and generation efficiency.
• 中文摘要 我们引入了功能-房间生成，一种新的室内3D场景生成设置，创建支持明确功能目标的房间，而不仅仅是视觉上合理的布局。现有的代理和可执行方法提高了可控性，但通常依赖于昂贵的测试时间生成-评估-修订循环，使功能房间生成缓慢且计算成本高昂。我们通过三项技术贡献来应对这一挑战。首先，我们设计一种递归领域专用语言，有效组织功能房间所需的层级对象组合，从房间结构和主要家具到密集的支撑面和嵌套小物体。它将房间表示为具有显式几何和函数关系的分阶段可执行程序。其次，我们提出一个顺序前馈场景构建框架，将递归构建路径提炼成场景构建专家。在推理阶段，专家逐阶段编写可执行的DSL代码，确定性执行者直接实例化每个阶段，无需教师代理、在线批评者或迭代修复。第三，我们引入ScenePRM，这是一个基于执行的过程奖励框架，通过强化学习提升专家，提供功能性、几何性、关系性和未来可构造性反馈。我们进一步建立面向函数的基准，展示室内场景生成和功能室生成的先进性能，实现更强的功能完整性、关系正确性、几何可执行性和生成效率。

Beyond Surface Alignment: Grounding the Dynamics of Situational Understanding and Generative Control in LLMs

超越表面对齐：奠定大型语言模型中情境理解与生成控制动态的基础

• Authors: Chenghao Yang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.29610
• Pdf link: https://arxiv.org/pdf/2608.29610
• Abstract The current alignment tuning paradigm for Large Language Models (LLMs) prioritizes surface-level behaviors -- fluency, safety, and tonal consistency. While effective for casual chat, this thesis argues that such surface alignment masks a lack of grounding, creating models that are stylistically confident but situationally brittle. We propose a framework of Grounded Alignment, analyzing how models process context (Input) and structure generation (Output), then aligning these grounded behaviors to human needs. First, we evaluate failures in Situational Grounding. SitTest shows that despite large context windows, state-of-the-art models struggle to maintain a consistent "mental model" of a changing environment. ReCode further shows that models rely on surface heuristics rather than deep syntactic dependencies: they "read" extensive histories without truly "understanding" the evolving situation. Second, we evaluate Generative Grounding. We introduce the Branching Factor (BF) to map LLM generation, finding that standard alignment tuning constricts this landscape into premature stylistic collapse. Hindsight further shows that models often fail to understand their own generations. Finally, we propose Dynamic Control for grounded interaction. AI Realtor demonstrates context engineering to compensate for poor situational grounding. Base-Aligned Model Collaboration decouples exploration from stylistic constraints. We also present Annealed Sampling for verifiable reinforcement learning and apply these ideas to Addiction Support, where model-generated rationalization offers a communication interface for high-stakes domains. Collectively, this work moves beyond surface alignment toward agents anchored in both context and generation.
• 中文摘要 当前大型语言模型（LLMs）的对齐调优范式优先考虑表层行为——流利度、安全性和声调一致性。虽然对日常聊天有效，但本论点认为，这种表面对齐掩盖了缺乏基础感，从而创造出风格自信但情境脆弱的模型。我们提出了一个基础对齐框架，分析模型如何处理上下文（输入）和结构生成（输出），然后将这些基础行为与人类需求对齐。首先，我们评估情境基础化的失败。SitTest显示，尽管上下文窗口较大，最先进的模型仍难以维持对环境变化的一致“心理模型”。ReCode进一步表明，模型依赖表面启发式，而非深层语法依赖：它们“读取”大量历史，却未真正“理解”不断演变的情境。其次，我们评估生成基础化。我们引入分支因子（BF）来映射LLM生成，发现标准对齐调优会将该景观限制为过早的风格崩溃。事后诸葛亮进一步表明，模型往往无法理解自身的代际。最后，我们提出了动态控制以实现基础交互。AI Realtor展示了上下文工程，以弥补情境基础不足。基准对齐模型协作将探索与风格约束脱钩。我们还提出了退火采样技术，用于可验证的强化学习，并将这些理念应用于成瘾支持，在那里模型生成的合理化为高风险领域提供了沟通接口。总体而言，这项工作超越了表面对齐，转向同时锚定于上下文和生成的代理。

JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Prediction

JPO：刑事判决预测中结构化法律推理的法理政策优化

• Authors: Zhaolu Kang, Yantao Liu, Tailong Luo, Leqi Zheng, Lei Wei, Chenghua Zhu, Junhao Gong, Jiachen Qian, Eric Hanchen Jiang, Jiaxin Liu, Yuan Wang, Hao Zhang, Zixia Wang, Rong Fu, Zheng Lin, Richeng Xuan, Zhichao Hu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.29616
• Pdf link: https://arxiv.org/pdf/2608.29616
• Abstract Criminal judgment prediction requires models to infer statutory articles, charges, and sentencing outcomes from case facts. Unlike standard classification tasks, it involves a structured reasoning process in which statutes should be matched with facts, charges should be justified by statutes, and sentencing outcomes should remain consistent with charges. Existing approaches optimize final labels, and while some have attempted to evaluate reasoning quality, their evaluations are indirect, often relying on LLM-generated rubrics that reflect model-internal preferences rather than the inherent logical structure of legal adjudication. We propose Juris Policy Optimization (JPO), a post-training framework for structured legal reasoning in Chinese criminal judgment prediction. JPO first uses teacher-generated rationales to supervise a standardized four-step reasoning process, and then applies reinforcement learning with a composite reward over legal prediction quality, reasoning structure completeness, and cross-step consistency. JPO further introduces token-level advantage reweighting and adaptive clipping for legally salient reasoning segments. Experiments on multiple open-source language models and three Chinese legal benchmarks show that JPO consistently improves both judgment prediction and reasoning quality over supervised fine-tuning and reinforcement learning baselines.
• 中文摘要 刑事判决预测需要模型从案件事实推断出法定条款、指控和量刑结果。与标准分类任务不同，它涉及结构化推理过程，即法规应与事实匹配，指控应由法规正当化，量刑结果应与指控保持一致。现有方法优化最终标签，虽然部分方法尝试评估推理质量，但其评估是间接的，常依赖反映模型内部偏好的LLM生成的评分标准，而非法律裁决固有的逻辑结构。我们提出法理政策优化（JPO），这是中国刑事判决预测中结构化法律推理的后期框架。JPO首先使用教师生成的理由来监督标准化的四步推理过程，然后对法律预测质量、推理结构完整性和跨步骤一致性进行综合奖励的强化学习。JPO进一步引入了令牌级优势重权和自适应剪裁，应用于法律显著推理段。在多个开源语言模型和三个中国法律基准测试上的实验显示，JPO在判断预测和推理质量上均优于监督式微调和强化学习基线。

AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing

AgenticRag-R1：基于栈内存的代理强化学习，用于多步推理、检索和记忆

• Authors: Xinke Jiang, Yue Fang, Zhibang Yang, Jiaran Gao, Zhixin Zhang, Tao Feng, Rihong Qiu, Wentao Zhang, Hongxin Ding, Ruizhe Zhang, Yongxin Xu, Yuheng Huang, Xu Chu, Junfeng Zhao, Yasha Wang
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29622
• Pdf link: https://arxiv.org/pdf/2608.29622
• Abstract Retrieval-Augmented Generation (RAG) improves the factuality of large language models (LLMs), yet existing RAG systems often struggle with complex, multi-step reasoning that requires adaptive retrieval and continuous revision of intermediate contexts. Recent reinforcement learning (RL)-based agentic RAG methods partially alleviate this issue, but typically rely on coarse-grained action spaces and trajectory-level rewards, resulting in weak reward assignment and a bias toward short-horizon, stereotyped reasoning template. To address, we propose AgenticRag-R1, a RL framework that deeply integrates reasoning, retrieval, and memory via a memory stack and fine-grained action space, supported by hierarchical action-aware rewards and an information-aware trajectory rejection strategy to enable effective long-horizon learning. Experiments across a diverse set of multi-hop, open-domain, and agentic reasoning benchmarks, spanning multiple backbone model sizes, demonstrate that AgenticRag-R1 consistently outperforms strong baselines. Moreover, AgenticRag-R1 learns more robust, interpretable, and memory-aware reasoning behaviors, highlighting the effect of fine-grained action modeling and information-aware optimization for long-horizon reasoning. Our code is anonymous available at this https URL.
• 中文摘要 检索增强生成（RAG）提升了大型语言模型（LLM）的事实性，但现有RAG系统常常在复杂、多步推理方面遇到困难，需要自适应检索和持续修订中间语境。近期基于强化学习（RL）的能动性RAG方法部分缓解了这一问题，但通常依赖粗粒度动作空间和轨迹级奖励，导致奖励分配薄弱，且偏向短视野、刻板的推理模板。为此，我们提出了AgenticRag-R1，这是一个通过记忆堆栈和细粒度动作空间深度整合推理、检索和记忆的强化学习框架，辅以层级动作感知奖励和信息感知轨迹拒绝策略，以实现有效的长视野学习。在多跳、开放域和智能体推理基准测试中，跨越多个骨干模型规模的实验表明，AgenticRag-R1 始终优于强基线。此外，AgenticRag-R1 学习了更稳健、可解释且具内存感知的推理行为，突出了细粒度动作建模和信息感知优化对长视野推理的效果。我们的代码匿名，链接如下。

Harness-RL: Black-Box Reinforcement Learning with Action-Args Decoupling for Central-Agent Multi-Agent Harnesses

束带-RL：基于动作-Args解耦的黑匣强化学习，适用于中央代理多智能体束

• Authors: Xinke Jiang, Zhixin Zhang, Zhibang Yang, Jiaran Gao, Rihong Qiu, Shijin Chen, Xu Chu, Junfeng Zhao, Yasha Wang
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.29641
• Pdf link: https://arxiv.org/pdf/2608.29641
• Abstract Large language model agents increasingly solve long-horizon tasks through multi-agent harnesses in which a central agent coordinates specialized sub-agents, tools, and environments. Training the central policy in such a harness raises two challenges. First, an action label is a low-cardinality decision, whereas its args form a high-dimensional conditional sequence; optimizing both with a shared sequence-level signal can produce conflicting gradients. Second, dynamic scheduling creates interdependent sessions with branches, parallel calls, and rewritten contexts, which cannot be faithfully reduced to one flat token sequence. We introduce Harness-RL, a structured reinforcement learning framework that combines Conflict-Aware Policy Optimization (CAPO) with interface-level black-box trajectory construction. The black-box component captures Interface Call Records, builds per-session prefix trees, and aligns outcome and process rewards with trainable tokens. CAPO uses forward activations to identify parameter partitions associated with action and args tokens, then routes their policy gradients to the corresponding subspaces. Harness-RL supports both central-only and joint multi-agent training. Across seven multi-hop question answering and agentic retrieval benchmarks, it reaches average F1 scores of 42.93 and 47.79 with Qwen2.5-1.5B and Qwen2.5-3B, respectively, while ablations validate the contribution of CAPO and favor central-only optimization in the evaluated setting. Our code is available at this https URL.
• 中文摘要 大型语言模型代理越来越多地通过多代理框架解决长期任务，其中中央代理协调专业的子代理、工具和环境。在此类框架中训练中央策略带来了两个挑战。首先，动作标签是低基数决策，而其 args 则构成高维条件序列;用共享的序列级信号优化两者可能导致梯度冲突。其次，动态调度会产生相互依赖的会话，包含分支、并行调用和重写上下文，无法忠实地归约为单一扁平令牌序列。我们介绍 Harness-RL，一种结构化强化学习框架，结合冲突感知策略优化（CAPO）与接口级黑匣轨迹构建。黑匣子组件捕获接口调用记录，构建每会话前缀树，并将结果和进程奖励与可训练的令牌对齐。CAPO通过前向激活识别与动作和ARGS令牌相关的参数分区，然后将其策略梯度路由到相应子空间。Harness-RL支持仅集中和联合多智能体训练。在七个多跳问答和代理检索基准测试中，Qwen2.5-1.5B和Qwen2.5-3B分别达到42.93和47.79的平均F1分数，而消融验证了CAPO的贡献，并支持在评估环境中仅集中优化。我们的代码可在此 https 网址访问。

SmoothRL: Online Reinforcement Learning During Asynchronous Execution

SmoothRL：异步执行中的在线强化学习

• Authors: Guang Gao, Yuxuan Nong, Baifu Huang, Jianan Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.29768
• Pdf link: https://arxiv.org/pdf/2608.29768
• Abstract Deploying robot policies in the physical world requires satisfying two fundamental desiderata: reliability and smooth real-time execution. However, deploying state-of-the-art generalist models presents challenges on both fronts. Achieving the precision and robustness required for real-world deployment necessitates sample-efficient online reinforcement learning (RL) to adapt pretrained models. Meanwhile, the increasing scale of robot foundation models has led to higher inference latency. To satisfy real-time constraints under high latency, modern systems adopt asynchronous inference with action chunking, overlapping policy computation with chunk execution to hide latency and enable smooth control. Despite their complementary roles, integrating asynchronous execution with gradient-based online RL remains underexplored. We present SmoothRL, an online RL framework that fine-tunes a pretrained policy within an asynchronous inference loop. SmoothRL follows a value-gradient paradigm, directly updating policy parameters using gradients of the action-value function with respect to policy actions. To enable correct optimization under asynchronous execution, SmoothRL explicitly models the asynchronous inference process during training. Specifically, each generated action chunk is partitioned by frame index into three regions: a committed region, consisting of actions committed by the previous inference cycle; an execution region, containing newly generated actions executed by the robot; and a discarded region, containing actions superseded by the next inference cycle. Gradients are propagated only through the execution region, ensuring policy optimization aligns with the trajectory distribution induced by asynchronous execution. We evaluate SmoothRL on real-world robotic tasks requiring high precision, as well as highly dynamic tasks that necessitate asynchronous execution.
• 中文摘要 在物理世界中部署机器人策略需要满足两个基本需求：可靠性和流畅的实时执行。然而，部署最先进的通用模型在两方面都面临挑战。实现现实部署所需的精度和鲁棒性，需要采样高效的在线强化学习（RL）来适配预训练模型。与此同时，机器人基础模型规模的扩大导致推理延迟增加。为满足高延迟下的实时约束，现代系统采用异步推理，动作分块，策略计算与块执行重叠，以隐藏延迟并实现平滑控制。尽管功能互补，异步执行与基于梯度的在线强化学习整合仍未被充分探索。我们介绍SmoothRL，一种在线强化学习框架，可在异步推理循环中微调预训练策略。SmoothRL 遵循价值梯度范式，直接利用动作值函数相对于策略动作的梯度更新策略参数。为了实现异步执行下的正确优化，SmoothRL 在训练过程中显式建模异步推理过程。具体来说，每个生成的动作块按帧索引划分为三个区域：已提交区域，包含前一推理周期执行的动作;执行区域，包含机器人新生成的动作;以及被丢弃的区域，包含被下一个推理周期取代的动作。梯度仅在执行区域内传播，确保策略优化与异步执行诱导的轨迹分布一致。我们将 SmoothRL 应用于需要高精度的现实机器人任务，以及需要异步执行的高动态任务。

Beyond Global Realism: Virtual Try-On Evaluation and Optimization with Dimension-wise Garment Fidelity Assessment

超越全球真实性：虚拟试穿评估与优化，采用按尺寸进行服装真实度评估

• Authors: Kaidong Zhang, Yukang Ding, Xiaoyu Liu, Ying Chen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.29804
• Pdf link: https://arxiv.org/pdf/2608.29804
• Abstract Virtual try-on (VTON) requires not only realistic generation but also faithful preservation of garment characteristics. However, existing evaluation metrics such as PSNR, SSIM, KID and FID struggle to measure the consistency between the generated and reference garments, particularly in capturing the multi-dimensional characteristics of garment fidelity. To address this, we propose DAT: a Dimension-wise Assessment framework for virtual Try-on, which decomposes garment consistency into seven interpretable dimensions: silhouette, color, neckline and sleeve shape, major decoration and structure, material texture, fine-detail fidelity, and logo preservation, each formulated as a discrete attribute-level prediction task. To train this specialized assessment model, we adopt a two-stage learning paradigm comprising large-scale weak supervision on 50K samples, followed by refinement on 10K higher-quality annotations obtained via multi-model voting. Furthermore, we employ weighted cross-entropy loss to mitigate the severe label imbalance inherent across evaluation dimensions. Beyond its role as an evaluation framework, the assessment model can be integrated into reinforcement learning optimization of Qwen-Image-Edit for VTON, where dimension-wise rewards are adaptively aggregated to emphasize under-optimized aspects during training. Experimental results show that our method (8B parameters) achieves state-of-the-art performance in terms of balanced accuracy, SROCC, and PLCC, outperforming strong proprietary models such as Gemini-3.1, Qwen3.7-plus, and GPT-5.5, while also serving as an effective optimization signal for reward-guided VTON generation
• 中文摘要 虚拟试穿（VTON）不仅需要真实生成，还需忠实保存服装特性。然而，现有的评估指标如PSNR、SSIM、KID和FID在衡量生成服装与参考服装之间的一致性方面存在困难，尤其是在捕捉服装忠实度的多维特性方面。为此，我们提出了DAT：虚拟试穿的按尺寸评估框架，将服装一致性分解为七个可解释的维度：轮廓、颜色、领口和袖型、主要装饰与结构、材质质感、细节忠实度以及标志保存，每个维度都作为离散属性级预测任务表述。为训练这一专业评估模型，我们采用两阶段学习范式：对5万样本进行大规模弱监督，随后通过多模型投票对1万个更高质量注释进行细化。此外，我们采用加权交叉熵损失以缓解评估维度间固有的严重标签不平衡。除了作为评估框架的角色外，该评估模型还可集成到VTON的Qwen-Image-Edit强化学习优化中，在训练过程中对维度奖励进行自适应聚合，强调优化不足的方面。实验结果显示，我们的方法（8B参数）在平衡准确率、SROCC和PLCC方面达到了最先进的性能，优于Gemini-3.1、Qwen3.7-plus和GPT-5.5等强专有模型，同时也作为奖励引导VTON生成的有效优化信号

SymVD: Symmetric Vision Language Action Distillation for Robot Manipulation

SymVD：机器人操作的对称视觉语言动作提炼

• Authors: Hyewon Choi, Donggyu Kim, Soojean Han
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.29828
• Pdf link: https://arxiv.org/pdf/2608.29828
• Abstract While pretrained Vision-Language-Action (VLA) models offer broad generalization capabilities in robotic manipulation tasks, adapting them to real-world environments or handling task shifts often requires substantial additional data and retraining. To address this, we propose Symmetric VLA Distillation (SymVD), a distillation framework that transfers knowledge from a large VLA teacher to a compact student policy by explicitly exploiting geometric symmetries in manipulation tasks, such as rotational and reflectional invariance. SymVD employs an equivariant actor-critic architecture and trains the student using a symmetry-aware objective that aligns with teacher actions under group-invariant properties. We demonstrate that by enforcing the policy to respect equivariance, SymVD reduces redundant exploration across configurations related by group transformations and improves sample efficiency during distillation. To further stabilize and improve distillation, SymVD introduces an adaptive weighting scheme that dynamically balances the distillation objective and reinforcement learning updates based on training progress, enabling robust transfer even when the teacher signal is imperfect or misaligned. Experimental results on robotic manipulation tasks demonstrate that SymVD consistently improves over standard distillation and also outperforms SAC in terms of sample efficiency and generalization to previously unseen symmetric transformations of the environment.
• 中文摘要 虽然预训练的视觉-语言-行动（VLA）模型在机器人操作任务中具备广泛的泛化能力，但将其适应现实环境或处理任务转移通常需要大量额外数据和重新训练。为此，我们提出了对称VLA蒸馏（SymVD），这是一种蒸馏框架，通过明确利用旋转和反射不变性等几何对称性，将大型VLA教师的知识传递给紧凑的学生策略。SymVD采用等变演员-批评架构，并使用与群不变性质下教师动作一致的对称性意识目标训练学生。我们展示了通过强制执行尊重等变性的策略，SymVD减少了群变换相关配置间的冗余探索，并提升了蒸馏过程中的样本效率。为进一步稳定和提升蒸馏，SymVD引入了自适应加权方案，基于训练进度动态平衡蒸馏目标与强化学习更新，即使教师信号不完美或错位，也能实现稳健的转移。机器人操作任务的实验结果表明，SymVD在样本效率和对环境前所未见对称变换的泛化方面，持续优于SAC。

A^2Agent: Action-Aware Reinforcement Learning for Repository-Level Code Localization Agents

A^2Agent：基于仓库级代码本地化代理的动作感知强化学习

• Authors: Doyeon Kim, Suyoung Bae, Yumin Lee, Jee-Hyong Lee
• Subjects: Subjects: Computation and Language (cs.CL); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2608.29831
• Pdf link: https://arxiv.org/pdf/2608.29831
• Abstract Localizing issue-relevant code regions is a critical step in automated software engineering. However, due to their reliance on sparse trajectory-level signals, existing methods cannot identify which per-turn actions are effective and often discover correct code regions during exploration but fail to commit them. To address these limitations, we propose an action-aware reinforcement learning method that combines a per-turn reward sequence rewarding both the discovery and commitment of gold code regions with an action-level advantage estimation scheme that isolates each action's credit by grouping turns sharing the same exploration context. Extensive evaluations show that our method improves the average F1 over the state-of-the-art (SOTA) by 1.58% on SWE-Bench Verified and 8.55% on SWE-Bench Pro, with our 4B model outperforming baselines up to 8x larger. Our code is available at this https URL.
• 中文摘要 本地化与问题相关的代码区域是自动化软件工程中的关键一步。然而，由于依赖稀疏的轨迹级信号，现有方法无法识别哪些每回合动作有效，且常常在探索过程中发现正确的代码区域但未能提交。为解决这些局限性，我们提出了一种动作感知强化学习方法，结合了每回合奖励序列（奖励金码区域的发现和承诺）与动作级优势估计方案，通过分组共享相同探索上下文的回合来隔离每个动作的功劳。广泛评估显示，我们的方法在SWE-Bench Verified上平均F1提升了1.58%，在SWE-Bench Pro上提升了8.55%，我们的4B模型表现甚至高出基线的8倍。我们的代码可在此 https 网址获取。

GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

GenRubric：用于可扩展LLM评估的自我演进评分标准生成

• Authors: Yifan Chen, Haitao Li, Qingyao Ai, Fengbin Zhu, Tat-Seng Chua, Min Zhang, Yiqun Liu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.29856
• Pdf link: https://arxiv.org/pdf/2608.29856
• Abstract Large language models are increasingly used as scalable evaluators for open-ended tasks. However, many LLM judges derive query-specific criteria during scoring, leaving the evaluation requirements insufficiently specified and their coverage difficult to audit. Query-specific rubrics make these requirements explicit, but expert-written rubrics are costly to construct, while existing automatic methods typically rely on inference-time refinement or external supervision. We introduce GenRubric, a self-evolving framework that improves rubric generation from unlabeled queries without requiring additional human annotations during self-evolution. Our approach is based on rubric-induced self-consistency: independently sampled rubrics for the same query provide partial views of its latent evaluation requirements, and a comprehensive rubric should induce a response that generalizes across these complementary evaluation views. We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality. We train GenRubric models at 4B, 8B, and 14B scales across multiple domains. Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics. The improvements further generalize to held-out domains, demonstrating the potential of self-evolving rubric generation for scalable and query-specific LLM evaluation. Code and models are publicly available at this https URL.
• 中文摘要 大型语言模型越来越多地被用作开放式任务的可扩展评估器。然而，许多LLM评判在评分时推导出查询特定标准，导致评估要求定义不足，其覆盖范围难以审计。查询特定评分标准明确了这些要求，但专家编写的评分标准构建成本较高，而现有自动方法通常依赖推理时间的细化或外部监督。我们引入GenRubric，一种自我演化框架，能够从无标签查询中提升评分标准生成，无需在自我演化过程中额外人工注释。我们的方法基于评分标准诱导的自一致性：对同一查询独立抽样的评分标准，提供其潜在评估需求的部分视图，综合评分标准应能引发跨这些互补评估视图的推广响应。我们通过强化学习实现这一原则，将跨评分标准的全面性信号与组级和标准级别的评分标准质量奖励相结合。我们在多个领域以4B、8B和14B尺度训练GenRubric模型。对人工注释评分标准基准的实验显示，自我演化能改善由生成评分标准诱导的评估与专家编写评分诱导的评价之间的一致性。这些改进进一步推广到保留领域，展示了自我演化评分标准生成在可扩展且查询特定LLM评估中的潜力。代码和模型可在此 https URL 公开获取。

Uncertainty-Driven Replay Memory for Reinforcement Learning

不确定性驱动的回放记忆用于强化学习

• Authors: Sheeraja Rajakrishnan, Alexander G. Ororbia, Travis Desell, Daniel E. Krutz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.29860
• Pdf link: https://arxiv.org/pdf/2608.29860
• Abstract Uncertainty estimation provides promising capabilities for reinforcement learning (RL) agents. Notably, estimating uncertainty can reduce the training time and enable agents to obtain greater rewards over time by exploiting information related to whether an action would facilitate exploration of portions of an environment that are well-known versus those that are relatively unknown. In this work, we propose a novel formulation of the experience replay buffer commonly used in RL that we call uncertainty-driven replay memory (UDRM), which entails an update scheme for internally stored memories based on uncertainty estimates obtained by an RL model during training. In contrast to existing forms of RL, which typically use temporal difference error or the distribution of transitions to update the replay memory buffer and train RL controllers, our scheme biases the memory buffer to store more uncertain transitions that will improve an RL agent's generalization throughout training. Experimental results demonstrate that our proposed uncertainty-aware replay buffer enables an RL agent to obtain higher rewards during training compared to other existing uncertainty-aware RL frameworks.
• 中文摘要 不确定性估计为强化学习（RL）智能体提供了有前景的能力。值得注意的是，估计不确定性可以缩短训练时间，并通过利用与某一行为是否有助于探索已知环境部分或相对未知部分相关的信息，使智能体能够随着时间获得更大的奖励。在本研究中，我们提出了一种新颖的经验重放缓冲表述，称为不确定性驱动重放记忆（UDRM），该方案基于训练过程中由强化学习模型获得的不确定性估计，对内部存储的记忆进行更新方案。与现有强化学习通常利用时间差误差或转移分布来更新回放记忆缓冲区和训练强化学习控制器不同，我们的方案对记忆缓冲区施加偏向，以存储更多不确定的过渡，从而提升强化学习代理在训练过程中的泛化能力。实验结果表明，我们提出的不确定性感知重放缓冲使强化学习代理在训练期间获得比其他已知不确定性强化学习框架更高的奖励。

When History Is Multimodal: Rethinking Context Management for Long-Horizon Agents

当历史多模态时：重新思考长期视野代理的上下文管理

• Authors: Jiaqi Su, Cong Pang, Jiawei Hong, Tiankuo Yao, Zixuan Chen, Xin Lou, Lewei Lu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.29897
• Pdf link: https://arxiv.org/pdf/2608.29897
• Abstract Long-horizon agents need a context manager to compress growing interaction histories into a bounded working context, via passive strategies or active strategies that decide how memory is accessed and reorganized. Meanwhile, prior optical-memory work mainly treats pixels as a dense codec for textualized histories, often presupposing that rendering context into optical memory incurs a significant performance drop relative to text, thus coupling this representation with SFT, self-distillation, or reinforcement learning to close this gap, leaving unresolved (i) how visual rendering performs as a context manager under a fair, controlled comparison, and (ii) whether this carrier offers a native advantage when history is inherently multimodal. In this paper, we formulate context management as a budget-constrained history transformation and introduce Visual Rendering (VR) as a representational context manager. Under a shared harness, policy model, trigger, and task domain, we evaluate VR on four text-centric and three multimodal benchmarks against four baselines (No Compression, Discard-All, Sliding Window, Summarization), finding visual memory is a natural carrier of native visual evidence. Building on this finding, we propose VERA (Visual Evidence-Retaining strategy for long-horizon Agents), a training-free context manager built on deterministic rendering with no exposed memory operations: on text-centric benchmarks it renders textual history as VR does, while on multimodal benchmarks it retains native visual observations instead of translating them into text. Across nearly all benchmarks, VERA cuts cumulative non-cache tokens by 31.5%-63.1% versus No Compression, matches existing managers on text-centric tasks, and achieves the highest accuracy among all baselines on multimodal tasks, supporting a modality-preserving view of long-horizon context management.
• 中文摘要 长视野代理需要上下文管理器，将增长中的交互历史压缩为有界的工作上下文，通过被动策略决定如何访问和重组记忆。与此同时，先前的光学记忆研究主要将像素视为文本化历史的密集编解码器，通常假设将上下文渲染到光学记忆相对于文本性能显著下降，因此将该表示与SFT、自我蒸馏或强化学习结合以弥合这一差距，但未解决的问题：（i）视觉渲染在公平、受控比较下作为上下文管理器的表现，以及（ii）在历史本质上多模态时，该载体是否具有原生优势。本文将上下文管理表述为预算受限的历史变换，并引入视觉渲染（VR）作为表示上下文管理器。在共享的利用器、策略模型、触发器和任务领域下，我们基于四个以文本为中心的基准测试和三个多模态基准测试，针对四个基线（无压缩、全部丢弃、滑动窗口、摘要）评估了VR，发现视觉记忆是原生视觉证据的自然载体。基于这一发现，我们提出了VERA（长视野代理视觉证据保留策略），这是一种基于确定性渲染、无裸露记忆操作的无训练上下文管理器：在以文本为中心的基准测试中，它像VR一样呈现文本历史;而在多模态基准测试中，它保留原生视觉观察，而非将其转化为文本。在几乎所有基准测试中，VERA将累计非缓存令牌比例降低了31.5%-63.1%，在文本中心任务上与现有管理器匹配，并在多模态任务中实现了所有基线中最高的准确性，支持了长视野上下文管理的模式保持视角。

SearchWiki: Learning to Build and Navigate Knowledge Wikis for Active Information Seeking

SearchWiki：学习构建和导航知识维基以促进主动信息寻求

• Authors: Guransh Singh, Vishwajeet Kumar, Arkadeep Acharya, Adnan Qidwai, Jaydeep Sen, Sachindra Joshi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.29953
• Pdf link: https://arxiv.org/pdf/2608.29953
• Abstract Flat retrieval-augmented generation treats a corpus as a bag of chunks, discarding document hierarchy and cross document structure. We introduce SearchWiki, a harness framework that synthesizes a corpus into a hierarchical, typed, navigable wiki and trains an agent, WikiResearcher-9B, to retrieve information through multi-turn tool use. The wiki organizes knowledge into three layers - document overviews, cross- document topic pages, and page-level source records; enabling progressive refinement of retrieval when initial lookup misses. We optimize the agent's navigation policy with on-policy reinforcement learning with a multi-component reward function balancing answer correctness, retrieval quality and trajectory efficiency. Evaluation on ViDoRe-V3 (8 domains), FinanceBench, and memory benchmarks (LoCoMo, LongMemEval, PersonaMem-v2) shows that WikiResearcher- 9B which is our RL-tuned Qwen 9B model, significantly outperforms same-size untrained baselines and exceeds or matches larger external models. SearchWiki paired with WikiResearcher-9B demonstrates that learned navigation over structured corpora is a superior alternative to flat retrieval.
• 中文摘要 平坦检索增强生成将语料库视为一袋块，丢弃文档层级和跨文档结构。我们介绍SearchWiki，一个利用框架，将语料库综合成层级、类型化、可导航的维基，并训练代理WikiResearcher-9B通过多回合工具检索信息。该维基将知识组织为三层——文档概览、跨文档主题页面和页面级源记录;当初始查找失误时，支持逐步细化检索。我们通过策略上的强化学习优化代理的导航策略，采用多组件奖励函数平衡答案正确性、检索质量和轨迹效率。在ViDoRe-V3（8个领域）、FinanceBench和内存基准测试（LoCoMo、LongMemEval、PersonaMem-v2）上的评估显示，WikiResearcher-9B（我们基于强化学习调优的Qwen 9B模型）显著优于同尺寸未训练基线，并超过或匹配更大的外部模型。SearchWiki与WikiResearcher-9B结合证明，基于结构化语料库的学习导航是比平面检索更优的选择。

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

TEMPO：大型音频语言模型的时序基础多任务后期训练

• Authors: Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh, Utathya Aich, Ramani Duraiswami, Dinesh Manocha
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.29999
• Pdf link: https://arxiv.org/pdf/2608.29999
• Abstract Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.
• 中文摘要 大型音频语言模型（LALMs）描述的是片段级音频，但无法为其识别的事件、说话者或声音分配时间戳。尽管时间戳对于语音识别和密集音频字幕等后续任务至关重要，但它仍是大多数LALM的主要局限。我们呈现TEMPO（时间基础多任务后期训练），这是首个统一处理音频、语音和音乐时间戳任务的模型。我们的核心贡献是一个基于三项创新的监督微调（SFT）阶段：原子时间戳标记、将正弦壁钟编码注入音频帧嵌入的时间感知投影仪，以及距离感知高斯损耗。我们的培训基于合成到真实课程。我们还进一步介绍了据我们所知，强化学习首次应用于统一音频时间戳，利用带有可验证时间奖励的GRPO，直接优化评估目标。GRPO不再是性能提升的主要来源，而是作为SFT检查点上的一个细化阶段，提供了适度的额外改进。为支持这项工作，我们构建了一个包含11.9万个样本的训练数据集和一个包含1万个样本的评估基准，这些样本来自五个任务的既有语料库。在该基准测试中，TEMPO优于Audio Flamingo Next和Qwen3-Omni这两个基于时间戳数据训练的先进LALMs。实验证实SFT实现了大部分这些提升，GRPO提供了一致但适度的细化。

Small Language Models as Judges for Rubric-Based Reinforcement Learning

小语言模型作为基于评分标准的强化学习评判

• Authors: Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.30005
• Pdf link: https://arxiv.org/pdf/2608.30005
• Abstract Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7$\times$ more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.
• 中文摘要 基于评分标准的强化学习将强化学习扩展到任务之外，超越了精确答案或基于规则的验证器，通过根据实例特定标准对回答进行评分。然而，这使奖励计算成本高昂：训练需要反复进行评分标准判断，通常使用专有API或具有7B或更多参数的本地生成式LLM评判。我们研究了较小的语言模型是否能作为高效且可靠的基于评分标准的评判。为使这个问题可测量，我们构建了PointRubric和RaR-Science-Static，这两种基于点的评分标准评估数据集，具有实例特定标准和项目满意标签。我们比较了从小模型中提取标准级判断的三种方法：生成判决、是/否Logprob边际和探针评判。在这两个数据集中，Qwen3-1.7B探测评判在这些方法中达到了最强的标准层面一致性，优于生成评审和Logprob评审。作为GRPO奖励模型，它在RaR-Science评分上训练策略范围为0.232至0.643，而生成评审基线为0.594，而基线则需要多10.7$\times$的奖励评审时间。任务和领域转移实验进一步表明，探测评委在不同环境中保持标准级奖励结构。

COGTRL: Training LLMs for Scientific Discovery Assistance using Cognitive Traces via Reinforcement Learning

COGTRL：通过强化学习，利用认知痕迹训练大型语言模型进行科学发现协助

• Authors: Shrinidhi Kumbhar Santosh Mashetty Divij Handa Kevin Coutinho, Siddharth Sambhaji Ghule, Chitta Baral
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.30109
• Pdf link: https://arxiv.org/pdf/2608.30109
• Abstract Large Language Models (LLMs) trained on extensive scientific research are increasingly integrated as assistants for scientific discovery. However, most research papers omit the fine-grained cognitive process of examining constraints, failed alternatives, and iterative decisions required to achieve the desired goal. Such cognitive processes are vital for real-world scientists working toward specific goals under constraints. In this paper, we show that LLMs, when trained to produce such cognitive traces, perform better as scientific discovery assistants than when trained solely on scientific literature. We propose COGTRL, a trajectory-level reinforcement learning framework that trains LLMs to emulate cognitively grounded reasoning by jointly optimizing cognitive traces and the scientific steps produced in an interleaved manner. Across two 3B-parameter models and two scientific domains (AI and Materials Science), COGTRL improves method quality by an average of 7.85 points over comparable 3B model baselines and achieves competitive performance relative to 70B parameter models. Moreover, analysis by domain experts shows a preference for methods generated by COGTRL over the baselines.
• 中文摘要 经过大量科学研究训练的大型语言模型（LLM）越来越多地被整合为科学发现的助手。然而，大多数研究论文省略了对约束、失败替代方案和迭代决策的细致认知过程，以实现目标。这些认知过程对于现实科学家在约束下实现特定目标至关重要。本文展示了，当LLM被训练产生此类认知痕迹时，作为科学发现助手的表现优于仅仅训练科学文献时。我们提出了COGTRL，一种轨迹级强化学习框架，通过共同优化认知痕迹和交织产生的科学步骤，训练LLM模拟认知基础推理。在两个3B参数模型和两个科学领域（人工智能和材料科学）中，COGTRL比可比的3B模型基线平均提升了7.85个百分点的方法质量，并且相较于70B参数模型实现了竞争力。此外，领域专家的分析显示，COGTRL生成的方法相对于基线更受青睐。

Online Estimation of Dynamic Origin-Destination Matrices Using Reinforcement Learning with Link-Flow Propagation Guidance

利用强化学习结合链路流传播指导的动态起点-目的矩阵在线估计

• Authors: Donggyu Min, Dong-Kyu Kim
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.30317
• Pdf link: https://arxiv.org/pdf/2608.30317
• Abstract Online dynamic origin-destination (OD) matrix estimation (DODE) calibrates time-dependent OD demand to reproduce observed link-flow trajectories. In online, OD demand should be estimated from current observations and propagated network states while subsequent observations and stochastic dynamic network loading (DNL) outcomes remain uncertain. Recently, reinforcement learning (RL) has emerged as a promising alternative, reducing computational burden by replacing iterative algorithms while being applicable to stochastic environments. However, because the policy is trained offline and deployed online, it must handle varying target link-flow trajectories; since each target trajectory defines the link-flow error used in the reward, the same OD demand vector can require different adjustments, making conventional scalar feedback ambiguous. To address this gap, this study proposes LFPG-RL, which integrates link-flow propagation guidance (LFPG) into proximal policy optimization (PPO). LFPG combines link-flow error sensitivities with the contribution of each OD-time demand component to simulated link flows, transforming aggregate mismatch into OD-specific advantage shaping for PPO actor updates. At deployment, the policy requires only a single forward pass. LFPG-RL is developed and evaluated on 250 weekday trajectories of 15-min link-flow data from a Melbourne arterial network modeled by a link transmission model with stochastic route choice. On held-out trajectories, LFPG-RL achieved an RMSE of 4.69, MAPE of 20.15%, and Pearson correlation of 0.995. These results support the contention that our method is a more efficient and accurate online OD demand calibration method compared to existing ones.
• 中文摘要 在线动态起点-目的地（OD）矩阵估计（DODE）校准时间相关的OD需求，以重现观测到的链路流轨迹。在线中，OD需求应根据当前观测和传播的网络状态估计，而后续观测和随机动态网络加载（DNL）结果仍不确定。近年来，强化学习（RL）作为一种有前景的替代方案出现，它通过替代迭代算法减轻计算负担，同时适用于随机环境。然而，由于该策略在离线训练和在线部署，必须处理不同的目标链路流轨迹;由于每个目标轨迹定义了用于奖励的链路流误差，同一OD需求向量可能需要不同的调整，使得传统的标量反馈变得模糊。为弥补这一空白，本研究提出了LFPG-RL，将链路流传播引导（LFPG）整合进近端策略优化（PPO）中。LFPG将链路流误差敏感性与每个OD-时间需求组件对模拟链路流的贡献结合，将总量不匹配转化为针对OD的PPO演员更新优势塑造。部署时，该策略仅需一次前向传递。LFPG-RL基于墨尔本主干道网络中250条工作日轨迹、15分钟链路流数据开发和评估，该轨迹由链路传输模型随机选择建模。在保留轨迹上，LFPG-RL实现了RMSE为4.69，MAPE为20.15%，皮尔逊相关系数为0.995。这些结果支持了我们的方法比现有方法更高效、更准确的在线过载需求校准方法的观点。

Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking

超越极化：点数重排序中思维链的生成约束

• Authors: Xiaoyang Chen, Jie Liu, Haijin Liang, Haibo Shi, Jin Ma, Ben He, Yingfei Sun, Dezhi Ye
• Subjects: Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2608.30398
• Pdf link: https://arxiv.org/pdf/2608.30398
• Abstract In pointwise document reranking, Chain-of-Thought models typically underperform direct scoring models. While existing diagnostics attribute this to inferior classification, score polarization, or calibration breakdown, whether targeted training can bridge this gap remains unclear. Our empirical study first confirms that this gap is stable across scales up to 32B parameters, ruling out model and data capacity confounders. We then apply stress tests utilizing reinforcement learning, fine-grained supervision, and architectural decoupling to explicitly repair these deviations. Although these interventions improve classification accuracy and absolute scores, the relative ranking gap persists. These findings suggest that, within the pointwise scoring paradigm, routing continuous relevance semantics through discrete text constrains ranking signal resolution, revealing a bottleneck that is stable and difficult to overcome under current standard methods, rather than an easily resolvable training bias.
• 中文摘要 在逐点文档重新排序中，思维链模型通常表现不及直接评分模型。现有诊断将此归因于分类劣势、分数极化或校准失调，但目标训练是否能弥合这一差距尚不明确。我们的实证研究首先确认该差距在最高32B参数尺度上稳定，排除模型和数据容量混杂因素。随后，我们应用强化学习、细粒度监督和架构解耦的压力测试，明确修复这些偏差。尽管这些干预提高了分类准确性和绝对得分，但相对排名差距依然存在。这些发现表明，在点分评分范式中，通过离散文本路由连续相关语义限制了排名信号分辨率，揭示了一个稳定且难以克服的瓶颈，而非易于解决的训练偏差。

SemPOI-RL: Aligning LLM Semantic Reasoning for Interpretable Out-of-Town POI Sequential Generation

SemPOI-RL：对齐可解释的外地 POI 顺序生成的 LLM 语义推理

• Authors: Yunqi Liu, Yang Zhang, Ruixing Zhang, Liangzhe Han, Yi Qiao, Tongyu Zhu, Leilei Sun
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.30399
• Pdf link: https://arxiv.org/pdf/2608.30399
• Abstract Large language models (LLMs) exhibit strong semantic reasoning and open-ended generation abilities, but aligning these abilities with structured sequential generation remains challenging. This challenge is particularly evident in out-of-town (OOT) POI sequence generation, where a model must infer transferable travel intent from a user's hometown behaviors, adapt to cross-city interest drift, and generate a coherent destination trajectory under structural constraints. Existing approaches either rely on latent ID-based transfer with limited interpretability or directly use LLMs for sequence generation without explicitly grounding inferred semantics into position-aware predictions. To address this gap, we propose SemPOI-RL, a framework that aligns LLM semantic reasoning with structured sequence generation for interpretable OOT recommendation. Specifically, we first fine-tune an LLM to infer destination-oriented travel styles from users' hometown trajectories, using natural language as an interpretable semantic intermediate. We then introduce a Semantic POI Alignment Module (SPAM) to ground these inferred styles into a style-conditioned masked autoencoder for position-aware trajectory generation. Finally, we apply reinforcement learning with recommendation-oriented rewards to align LLM-generated styles with downstream sequence quality. Experiments on two real-world datasets show that SemPOI-RL consistently outperforms both traditional recommenders and direct LLM baselines, while providing interpretable style attribution across different phases of a trip. The code is available at this https URL .
• 中文摘要 大型语言模型（LLMs）展现出强大的语义推理和开放式生成能力，但将这些能力与结构化顺序生成对齐仍具挑战性。这一挑战在外地（OOT）POI序列生成中尤为明显，模型必须从用户家乡行为推断可转移的旅行意图，适应跨城兴趣漂移，并在结构约束下生成连贯的目的地轨迹。现有方法要么依赖有限的潜在ID转移，要么直接使用LLM生成序列，而未明确将推断语义置于位置感知预测中。为弥补这一空白，我们提出了SemPOI-RL，一个将LLM语义推理与结构化序列生成相结合的框架，以提供可解释的OOT推荐。具体来说，我们首先微调LLM，利用自然语言作为可解释的语义中间体，从用户家乡轨迹推断目的地导向的旅行风格。随后引入语义POI对齐模块（SPAM），将这些推断风格扎根于风格条件掩蔽自编码器中，实现位置感知轨迹生成。最后，我们应用基于推荐导向奖励的强化学习，使LLM生成的风格与下游序列质量对齐。在两个真实世界数据集上的实验表明，SemPOI-RL始终优于传统推荐者和直接LLM基线，同时在旅行的不同阶段提供可解释的风格归因。该代码可在此 https URL 获取。

Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic

本地引导的演员-批评者：用具备子目标意识的批评者训练目标条件演员

• Authors: Olivier Serris, Stéphane Doncieux, Olivier Sigaud
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.30406
• Pdf link: https://arxiv.org/pdf/2608.30406
• Abstract Goal-conditioned reinforcement learning struggles with long horizons when rewards are sparse. While a planner can provide subgoals to guide a low-level policy, its use at test time may introduce practical subgoal management difficulties. An alternative paradigm utilizes a high-level planner to assist learning, while the policy remains conditioned only on the final goal, enabling planner-free deployment. Among these methods, Reinforcement Learning with Imagined Subgoals (RIS) introduces a regularization term that encourages the policy to take the same actions for the final goal as it does for an intermediate goal. This regularization, however, may lead to goal-chaining issues when intermediate goals are low-dimensional. Potential-based reward shaping (PBRS) translates plans into an additional reward while ensuring that the optimal policy remains unchanged. Yet, it can generate deceptive rewards in terminal states. We study these failure cases and first propose an alternative reward shaping method (RS) that removes these deceptive rewards at the expense of theoretical guarantees of PBRS. Similar to this RS variant, we then propose another method named Locally-Guided Actor Critic (LG-AC) that rewards the agent for reaching intermediate goals. Unlike RS, where intermediate rewards are implicit in the shaping signal, we explicitly condition a value estimator on the full sequence of intermediate goals but represent the value function as a sum of subgoal-conditioned value functions, enabling dense hindsight relabeling. We evaluate all these methods in tasks with challenging goal-chaining requirements and empirically highlight specific cases in which either action regularization or reward shaping yield low performance, while LG-AC achieves the best overall performance across tasks.
• 中文摘要 当奖励稀疏时，目标条件强化学习在长时间视野上遇到困难。虽然规划器可以提供子目标来指导低层次策略，但在测试阶段使用可能会带来实际的子目标管理困难。另一种范式利用高层次规划器辅助学习，而策略仅以最终目标为条件，实现无规划者部署。在这些方法中，带有想象子目标的强化学习（RIS）引入了一个正则化术语，鼓励策略对最终目标采取与中间目标相同的行动。然而，当中间目标为低维时，这种正则化可能导致目标链化问题。基于潜力的奖励塑造（PBRS）将计划转化为额外奖励，同时确保最优策略保持不变。然而，它在终端状态下可能产生欺骗性的奖励。我们研究这些失败案例，首先提出一种替代的奖励塑造方法（RS），该方法以牺牲PBRS理论保证为代价去除这些欺骗性奖励。类似于该RS变体，我们提出了另一种方法，称为局部引导行为者批评者（LG-AC），该方法奖励代理达成中间目标。与RS中中间奖励隐含于塑形信号不同，我们明确对价值估计器对整个中间目标序列进行条件，但将价值函数表示为子目标条件值函数的和，从而实现密集的事后重标。我们在具有挑战性目标链要求的任务中评估所有这些方法，并通过实证突出某些情况下，动作正则化或奖励塑造表现较低，而LG-AC在各任务中实现最佳整体表现。

Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry

混淆伪装为改善：系统评估129,000名患者登记中中风抗血栓治疗的离线强化学习

• Authors: Kihun Rhee
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2608.30442
• Pdf link: https://arxiv.org/pdf/2608.30442
• Abstract Recent offline reinforcement learning (RL) studies report policies that outperform physician decisions on clinical outcomes. We conduct a systematic, partially crossed evaluation of five offline RL algorithm families and 14 reward designs in 44,894 post-2018 acute ischemic stroke patients from a nationwide registry (N = 129,033). Standard Fitted Q-Evaluation (FQE) yields an apparent policy-improvement estimate of +0.0069; adding an Early Neurological Deterioration penalty increases it to +0.0101. We identify reward-embedded confounding, in which a proxy terminal reward encodes baseline severity and prognosis as well as treatment efficacy. A 2 x 2 factorial analysis finds that terminal reward confounding accounts for 218.6% of the observed signal change, so its removal overshoots the null. After DML-inspired GBM reward residualization, the FQE estimate attenuates to +0.0033 (p = 0.132), and full deconfounding yields +0.0025 (p = 0.291). FQE-based diagnostics, T-learner analyses, and direct recurrence analyses converge away from a clinically meaningful aggregate improvement. A 1-year mRS factorial analysis replicates the attenuation. We provide an empirically motivated six-step evaluation checklist. NIHSS-stratified heterogeneity is hypothesis-generating for prospective trial design; hospital-level disagreement does not persist after full reward deconfounding.
• 中文摘要 近期离线强化学习（RL）研究报告指出，这些政策在临床结果上优于医生的决策。我们对5个离线强化学习算法家族和14个奖励设计进行了部分交叉评估，涵盖了全国登记的44,894名2018年后急性缺血性卒中患者（N = 129,033）。标准拟合Q评估（FQE）给出的表观策略改进估计为+0.0069;加入早期神经退化惩罚则使其提升为+0.0101。我们识别出奖励嵌入的混杂因素，其中代理终末奖励编码基线严重度、预后以及治疗疗效。2×2因子分析发现，终末奖励混杂因素占观察到信号变化的218.6%，因此其去除超过了无结果。经过DML激励的GBM奖励残留后，FQE估计值衰减至+0.0033（p = 0.132），完全混杂结果为+0.0025（p = 0.291）。基于FQE的诊断、T-学习者分析和直接复发分析趋于临床意义上的总体改善。1年mRS因子分析复现了减弱。我们提供了实证动机的六步评估检查清单。NIHSS分层异质性是前瞻性试验设计的假设生成;在完全奖励纠杂后，医院层面的分歧不再持续。

Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry

学习结果变化之处：多模几何的学分可定论推理

• Authors: Jiani Guo, Junjie Wang, Jie Wu, Pengxiang Zhao, Dongdong Zhang, Shaohan Huang, Yujiu Yang, Furu Wei
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.30457
• Pdf link: https://arxiv.org/pdf/2608.30457
• Abstract Multimodal geometry reasoning requires VLMs to extract precise visual relations and preserve them through multi-step deduction. Existing free-form traces obscure the decisions that determine the answer, and trajectory-level reinforcement learning distributes a single terminal signal across the entire response. We introduce credit-addressable reasoning, in which the semantic units exposed during inference also define where learning compares alternatives and assigns credit. We instantiate this principle with Code-CoT, which retains the diagram, represents visual relations as line-addressable executable code, and organizes reasoning into typed events, and CE-GRPO, which selects event boundaries using structural priors and type-normalized entropy, samples complete continuations from shared prefixes, and converts outcome differences into localized advantages. Across nine geometry benchmarks, CE-GRPO achieves an average accuracy of 76.04, outperforming Qwen3-VL-8B and trajectory-level GRPO by $8.09$ and 3.43 points, respectively. Its relative advantage increases with the number of intermediate events, demonstrating the value of representation--optimization co-design for long, dependency-heavy multimodal reasoning.
• 中文摘要 多模几何推理要求VLM提取精确的视觉关系，并通过多步演绎保持它们。现有的自由形式痕迹遮蔽了决定答案的决策，轨迹级强化学习在整个响应中分配单一终端信号。我们引入了可寻址的推理，其中推理过程中暴露的语义单元也定义了学习比较备选方案和归功的处境。我们通过Code-CoT实现这一原则，Code-CoT保留了图，将视觉关系表示为可行寻址的可执行代码，并将推理组织为类型事件;CE-GRPO则利用结构先验和类型归一化熵选择事件边界，采样共享前缀的完整延续，并将结果差异转化为局部优势。在九个几何基准测试中，CE-GRPO的平均准确率为76.04，分别比Qwen3-VL-8B和轨迹级GRPO高出8.09美元和3.43个百分点。其相对优势随着中间事件的增加而增强，展示了表征——优化协同设计在长期依赖性强的多模态推理中的价值。

PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs

PAC：多任务强化学习的进步增强优势课程

• Authors: Yuanqiang Yu, Yanzhao Zheng, Zhentao Zhang, Tianze Xu, Chao Ma, Jihuai Zhu, Jiashun Liu, Xinle Deng, Baohua Dong, Hangcheng Zhu, Ruohui Huang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.30528
• Pdf link: https://arxiv.org/pdf/2608.30528
• Abstract Reinforcement learning (RL) is used to improve the reasoning abilities of LLMs, while training data span heterogeneous tasks. However, most RL post-training pipelines rely on fixed or manually designed task mixtures, even though task usefulness changes as training progresses. Online curriculum methods often define learnability by update magnitude, ignoring whether the update translates into reward gains, which can misallocate rollout budget toward tasks with large but ineffective updates. We propose PAC, a Progress-Augmented Advantage Curriculum for multi-task RL of LLMs that combines two task-level signals: advantage-derived learnability, which measures the magnitude of the policy update a task can induce, and recent reward gains, which show whether those updates have improved task performance. A Bayesian Thompson Sampling controller uses these signals to allocate rollouts across tasks during GRPO training. We evaluate PAC under two settings: a multi-level reasoning setting and a multi-domain reasoning setting. PAC improves sample efficiency and final performance: it reaches comparable validation scores with fewer rollout steps and achieves higher final averages than random sampling and advantage-based curriculum baselines in both settings. These results show that jointly tracking advantage signals and actual reward gains yields an effective online curriculum for LLM post-training.
• 中文摘要 强化学习（RL）用于提升LLM的推理能力，而训练数据则涵盖异构任务。然而，大多数强化学习后流程依赖固定或手动设计的任务组合，尽管任务的实用性会随着训练进展而变化。在线课程方法通常通过更新幅度来定义可学习性，忽视更新是否转化为奖励收益，这可能导致将推广预算错误分配给更新幅度大但无效的任务。我们提出了PAC，一种针对LLM多任务强化学习的进步增强优势课程，结合了两种任务级信号：优势衍生的可学习性，衡量任务能引发的策略更新的幅度，以及近期奖励增益，显示这些更新是否提升了任务表现。贝叶斯汤普森抽样控制器利用这些信号在GRPO训练期间分配任务的推广。我们在两个环境下评估PAC：多层次推理和多领域推理。PAC提升了样本效率和最终表现：在这两种环境中，PAC以更少的推广步骤达到相当的验证分数，并且最终平均值优于随机抽样和基于优势的课程基线。这些结果表明，结合跟踪优势信号和实际奖励收益，可以为LLM培训后提供有效的在线课程。

DiffPDE: Masked Diffusion Language Models as PDE Solver

DiffPDE：掩盖扩散语言模型作为偏微分方程求解器

• Authors: Wenxuan Guo, Yuyang Hong, Lubin Fan, Zhaojin Fu, Lin Chen, Kun Ding, Shiming Xiang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.30532
• Pdf link: https://arxiv.org/pdf/2608.30532
• Abstract Existing approaches for synthesizing Partial Differential Equation (PDE) solvers predominantly rely on autoregressive models, yet their global left-to-right decoding incurs substantial redundancy when addressing inherently localized bugs. In this work, we challenge this inefficient paradigm and propose DiffPDE, a framework leveraging discrete diffusion language models for targeted code repair. By introducing a localized re-masking and infilling strategy, DiffPDE regenerates only erroneous regions while preserving correct context, naturally aligning generation with the sparse nature of PDE errors. Furthermore, to handle coupled bugs requiring sequential interventions, we present Iterative Debugging GRPO (ID-GRPO), a reinforcement learning scheme that enables multi-round debugging within single trajectories via intermediate rewards. Experiments on PDEBench show that DiffPDE achieves competitive accuracy, outperforms same-scale AR models, and significantly accelerates repair.
• 中文摘要 现有的偏微分方程（PDE）求解器合成方法主要依赖自回归模型，但其全局从左到右的解码在处理固有局部错误时会产生大量冗余。本研究挑战了这一低效范式，提出了利用离散扩散语言模型进行针对性代码修复的DiffPDE框架。通过引入局部的重新掩蔽和填充策略，DiffPDE仅重生成错误区域，同时保持正确的上下文，自然地将生成过程与PDE错误的稀疏性相匹配。此外，为处理需要顺序干预的耦合错误，我们提出了迭代调试GRPO（ID-GRPO），这是一种通过中间奖励实现单一轨迹内多轮调试的强化学习方案。PDEBench上的实验表明，DiffPDE实现了竞争精度，优于同比例AR模型，并显著加快了修复速度。

SPHERE: Automatic Music Upmixing via Audio Language Model Post-Training with Spatial Heuristic Rewards

SPHERE：通过音频语言模型的自动音乐上混，后期训练与空间启发式奖励

• Authors: Zixun Guo, Calvin Murdock, Sanjeel Parekh, W Owen Brimijoin, Simon Dixon, Joshua Reiss, Ishwarya Ananthabhotla
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2608.30559
• Pdf link: https://arxiv.org/pdf/2608.30559
• Abstract In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio language model (ALM) post-training, leveraging rich representations from existing ALMs, which encode both music semantics and mixing knowledge. Specifically, we propose a post-training recipe that first employs rejection sampling SFT, followed by reinforcement learning (RL) with verifiable rewards (RLVR) via GRPO. We propose Sphere (Spatial Heuristic Rewards), a deterministic reward suite inspired by music mixing conventions, to guide our post-training. It consists of 6 perceptually-motivated sub-rewards and encourages the output mix to be centered, balanced and spacious. More broadly, our results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into language models, without task-specific architectures.
• 中文摘要 本文研究自动音乐上混任务，即系统从多干录音中预测空间混音参数。与依赖任务特定音乐编码器的现有方法不同，我们通过音频语言模型（ALM）后训练来完成该任务，利用现有ALM中丰富的表征，既编码音乐语义也包含混音知识。具体来说，我们提出了一种训练后方案，先采用拒绝采样SFT，随后通过GRPO进行可验证奖励（RL）强化学习（RL）。我们提出Sphere（空间启发式奖励），这是一套受音乐混音惯例启发的确定性奖励套件，用于指导我们的后期训练。它包含6个感知动机的子奖励，鼓励输出组合居中、平衡且宽广。更广泛地说，我们的结果表明，专家领域知识可以被编码为可验证的奖励，并提炼成语言模型，而无需特定任务架构。

Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum

学习合成时空视频基础教学

• Authors: Xingjian Wang, Shijian Wang, Yibo Wang, Zihao Yu, Runhao Fu, Xuelian Cheng, Zongyuan Ge
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.30584
• Pdf link: https://arxiv.org/pdf/2608.30584
• Abstract Despite the impressive progress of recent MLLMs on spatio-temporal video grounding (STVG), existing evaluations and training data focus primarily on simple queries. They largely overlook the compositional queries prevalent in real-world scenarios, where a target must be disambiguated by jointly reasoning about its attributes and relations to other entities. To bridge this gap, we propose Compositional Spatio-Temporal Video Grounding (CompSTVG), a task that requires models to process complex textual queries where every intertwined attribute and relational cue is essential for disambiguation. To facilitate this task at scale, we build a synthetic data engine that leverages a spatio-temporal scene graph as a difficulty measure and casts difficulty-controlled query synthesis as a constraint programming problem, producing difficulty-graded data for both evaluation and training. Built on this engine, we introduce STVG-CompBench, a benchmark stratified by explicit difficulty levels that jointly capture temporal complexity and spatial interference. Evaluating 11 representative STVG models on STVG-CompBench reveals that current models perform poorly on compositional queries, exhibiting a sharp performance drop that is typically obscured by overall dataset-level averages. We further construct synthetic training data and propose CurrSTVG, a curriculum reinforcement learning framework that delivers consistent gains, with the largest improvements observed on the most challenging compositional queries.
• 中文摘要 尽管近期MLLM在时空视频基础（STVG）方面取得了显著进展，现有评估和训练数据主要聚焦于简单查询。它们大多忽视了现实场景中常见的合成查询，即必须通过共同推理目标的属性及与其他实体的关系来消除歧义。为弥合这一空白，我们提出了合成时空视频基础化（Compositional Spatio-Temporal Video Grounding，CompSTVG）任务，该任务要求模型处理复杂的文本查询，其中每个交织属性和关系线索对消歧义至关重要。为实现该任务的大规模化，我们构建了一个综合数据引擎，利用时空场景图作为难度衡量，并将难度控制查询综合定位为约束编程问题，生成难度分级数据，用于评估和训练。基于该引擎，我们引入了STVG-CompBench基准，该基准通过明确的难度等级分层，共同捕捉时间复杂性和空间干扰。在STVG-CompBench上评估11个代表性STVG模型，显示当前模型在组合查询中表现不佳，表现明显下降，通常被整体数据集层面的平均值所掩盖。我们进一步构建合成训练数据，提出了CurrSTVG，一种课程强化学习框架，能够持续带来持续的提升，在最具挑战性的组合查询中取得了最大提升。

GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning

GMTS：基于梯度幅度的令牌选择提升了 LLM 推理的 RLVR 训练

• Authors: Outongyi Lv, Yuanwei Zhang, Xiaoqun Zhang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.30632
• Pdf link: https://arxiv.org/pdf/2608.30632
• Abstract Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.
• 中文摘要 强化学习（RL），尤其是带有可验证奖励的强化学习（RLVR），最近已成为增强大型语言模型（LLMs）推理能力的核心范式，在各推理任务中展现出显著效果。最新研究表明，高熵代币在模型训练中扮演着极其重要的角色，因为仅使用最高20%熵代币进行训练能带来显著的性能提升。然而，为什么这种高熵代币有益，仍不够理解。在这项研究中，我们发现，尽管一个答案中的高熵代币往往与较大梯度相关，但单靠熵本身并不能一致反映不同答案间的代币重要性，因为考虑到答案级别奖励信号的差异。基于这一观察，我们引入了基于梯度的基于符号选择（Gradient Magnitude-based Token Selection，GMTS）方法来量化代币的重要性，该方法利用熵-梯度关系来近似对代币选择的梯度-幅度排名。我们发现，在GMTS排名前20%的代币上训练，在三个推理领域和不同模型规模下，始终优于基于熵的代币选择，表明GMTS为RLVR训练提供了更细致的代币贡献估计。

Test-time Reinforcement Learning in Imperfect Information Games

不完全信息博弈中的测试时强化学习

• Authors: Ondrej Kubicek, Viliam Lisy, Tuomas Sandholm
• Subjects: Subjects: Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2608.30635
• Pdf link: https://arxiv.org/pdf/2608.30635
• Abstract Test-time reasoning has significantly improved performance in domains ranging from games to language models. However, test-time policy changes with formal guarantees on the performance of the resulting strategy remain a challenge in two-player zero-sum imperfect-information games. Existing solutions are limited to tabular methods or single gradient step updates. In this work, we investigate policy-gradient algorithms as a method for scalable test-time reasoning. We extend the concept of gadget game, tabular technique for test-time search, to the reinforcement learning setting. Unlike prior approaches, we represent the gadget game implicitly by modified sampling and neural policy rather then explicitly by constructing it, thereby removing constraints on subgame size. Furthermore, we formally prove that, unlike prior tabular algorithms, regularized policy-gradient algorithms limit possible strategy degradation caused by test-time reasoning, even without the gadget games. Our evaluation across small- and large-scale games confirms that additional test-time training often substantially improves performance relative to the blueprint strategy.
• 中文摘要 测试时推理在从游戏到语言模型等领域显著提升了性能。然而，在两人零和不完美信息博弈中，测试时策略变更并对策略性能提供形式保证仍是挑战。现有解法仅限于表格方法或单梯度步更新。本研究中，我们研究策略梯度算法作为可扩展测试时间推理方法。我们将工具博弈（测试时搜索的表格技术）的概念扩展到强化学习环境。与以往方法不同，我们通过修改采样和神经策略隐式表示小工具博弈，而非显式构造，从而消除子博弈大小的约束。此外，我们正式证明，与以往的表格算法不同，正则化策略梯度算法即使不涉及小工具博弈，也能限制测试时间推理可能造成的策略退化。我们对小规模和大型游戏的评估证实，额外的测试时间训练通常能显著提升相对于蓝图策略的性能。

Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL

一次三步：对比强化学习中的动作序列表征

• Authors: Michal Korniak, Kamil Dybek, Benjamin Eysenbach, Marco Bagatella, Michał Bortkiewicz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.30640
• Pdf link: https://arxiv.org/pdf/2608.30640
• Abstract While self-supervised approaches to reinforcement learning have achieved strong results by learning representations of states and actions, a key open question is the time scale over which actions should be modeled. Departing from the standard formulation relying on single-step actions, we extend contrastive reinforcement learning (CRL), a prototypical self-supervised method, to operate over action chunks, and find that this results in large, pervasive gains across established offline and online benchmarks: +31.7% and +93.1% across 18 and 11 environments respectively. While action-chunking-driven gains are generally explained through the ability to model non-Markovian, temporally extended policies, and to propagate unbiased multi-step returns, interestingly, we find that these arguments only partially apply to CRL. Our empirical studies suggest that, in the context of CRL, an action chunk carries more information about the goal than a single action, measurably improving the critic's representations, and rendering the algorithm significantly more effective.
• 中文摘要 虽然自监督强化学习方法通过学习状态和动作的表示取得了显著成效，但一个关键的未解问题是动作应被建模的时间尺度。我们放弃了依赖单步动作的标准表述，将对比强化学习（CRL）这一典型的自监督方法扩展到行动块上，发现这在既有的线下和在线基准中带来了显著且普遍的收益：分别在18个和11个环境中达到+31.7%和+93.1%。虽然动作块驱动的收益通常通过建模非马尔可夫、时间延伸策略以及传播无偏多步收益来解释，但有趣的是，我们发现这些论点仅部分适用于CRL。我们的实证研究表明，在CRL的背景下，动作块携带的目标信息比单个动作更多，显著提升了批评者的表征，并显著提升了算法的效能。

HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving

HiRS-Agent：一个用于可靠远程遥感任务解决的分层多智能体系统

• Authors: Boyang Mu, Zhiwei Wei, Mugen Peng, Wenjia Xu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2608.30672
• Pdf link: https://arxiv.org/pdf/2608.30672
• Abstract Recent advances in large language models and multimodal models have pushed remote sensing (RS) processing from simple perception models to agentic systems designed to tackle complex, long-horizon RS tasks. However, existing systems often rely on monolithic decision-making frameworks, which fail to accommodate the multi-stage, interdependent nature of RS tasks. This centralized approach leads to challenges such as unstable task execution, incorrect tool usage, and error propagation across stages. To address these issues, we propose HiRS-Agent, a hierarchical multi-agent system for long-horizon RS task solving. HiRS-Agent adopts a two-level collaborative architecture: the Manager Layer handles dynamic routing, step-level verification, replanning, and termination control, while the Specialist Layer organizes domain-specific tools according to the RS workflow and is responsible for subtask reasoning and tool execution. To further enhance the system's capability, we introduce a two-stage supervised tuning strategy and a verification-guided hierarchical reinforcement learning stage to jointly optimize coordination and tool-use policies. Experiments on Earth-Agent Benchmark and ThinkGeo show that HiRS-Agent substantially improves long-horizon tool-use capability and final-task correctness, demonstrating the effectiveness of structured multi-agent collaboration for reliable RS agents. The code is publicly available at this https URL.
• 中文摘要 大型语言模型和多模态模型的最新进展，使遥感（RS）处理从简单的感知模型转向旨在处理复杂、长视野RS任务的代理系统。然而，现有系统常依赖单一的决策框架，无法适应RS任务的多阶段、相互依赖性。这种集中化方法带来了任务执行不稳定、工具使用错误以及跨阶段错误传播等挑战。为解决这些问题，我们提出了HiRS-Agent，一种用于长期RS任务解决的分层多代理系统。HiRS-Agent采用两级协作架构：管理层负责动态路由、步级验证、重新规划和终止控制，而专家层则根据RS工作流程组织领域专用工具，负责子任务推理和工具执行。为进一步提升系统能力，我们引入了两阶段监督调优策略和验证引导的层级强化学习阶段，共同优化协调和工具使用策略。Earth-Agent Benchmark 和 ThinkGeo 上的实验显示，HiRS-Agent 显著提升了长期工具使用能力和最终任务正确性，展示了结构化多智能体协作对可靠 RS 代理的有效性。该代码可在此 https 网址公开获取。

CIG-RL: Curiosity-Driven Information-Guided Reinforcement Learning for Source Term Estimation in Uncertain Environments

CIG-RL：在不确定环境中进行源词估计的好奇心驱动信息引导强化学习

• Authors: Junhee Lee, Seunghwan Kim, Hongro Jang, Hyungjin Kim, Hyoungho Park, Changseung Kim, Hyondong Oh
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.30673
• Pdf link: https://arxiv.org/pdf/2608.30673
• Abstract Source term estimation (STE), which aims to estimate key properties of the gas source, is essential for identifying hazardous gas releases. Information-theoretic approaches have been adopted for autonomous STE using mobile sensors due to robustness in noisy environments, yet their online action selection incurs substantial computational cost. Deep reinforcement learning (DRL) provides a promising alternative with its fast decision-making capability. In DRL-based STE, the agent selects actions based on belief states of the source term updated from noisy measurement sequences. However, existing methods rely on random exploration or solely on belief uncertainty reduction without an effective exploration strategy in DRL, which can limit policy robustness in noisy environments. To address this, we propose a curiosity-driven information-guided reinforcement learning for robust and efficient STE. The proposed method promotes active exploration of novel belief state transitions that have not been sufficiently explored during training. We further introduce an uncertainty-adaptive active perception reward to guide efficient source search under uncertainty. Simulations under high-noise conditions and real-world experiments demonstrate the robustness and feasibility of the proposed framework, highlighting its potential for practical STE problems.
• 中文摘要 源项估计（STE）旨在估计气体源的关键属性，对于识别有害气体泄漏至关重要。由于在噪声环境中具有鲁棒性，信息理论方法已被用于使用移动传感器进行自主STE的方法，但其在线动作选择带来了大量计算成本。深度强化学习（DRL）以其快速决策能力提供了有前景的替代方案。在基于DRL的STE中，智能体根据源词的信念状态（从噪声测量序列更新）选择动作。然而，现有方法依赖随机探索或仅仅减少信念不确定性，缺乏有效的DRL探索策略，这可能限制了在噪声环境中的策略鲁棒性。为此，我们提出了一种基于好奇心驱动的信息引导强化学习，以实现稳健高效的STE。该方法促进了对训练中未充分探索的新信念状态转变的主动探索。我们还引入了不确定性自适应的主动感知奖励，以指导在不确定性下高效的源搜索。高噪声条件下的模拟和真实实验展示了该框架的鲁棒性和可行性，突出其在实际STE问题上的潜力。

VisLens: Single-Pass Interpretable Visual Search for Multimodal LLMs

VisLens：多模大型语言模型的单次可解释视觉搜索

• Authors: Jingyi He, Sanghwan Kim, Zeynep Akata
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.30705
• Pdf link: https://arxiv.org/pdf/2608.30705
• Abstract Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into two families: (1) Training-free methods based on attention or confidence scores are accurate but slow, since they require multiple MLLM queries per example. (2) Reinforcement Learning (RL) trained tool-use models are faster at inference but opaque, since their tool calls remain uncontrollable and hard to interpret. To overcome this, we propose \emph{VisLens} (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head. VisLens further uses a lightweight tuned-lens that maps early hidden states into the final hidden state space, so visual tokens can be read out from early layers. These tokens are matched to target words in the query to generate a crop of the relevant region, which is fed back in alongside the original image to produce the final answer. The whole process, from decoding to the final answer, completes in a single forward pass without repeated queries. VisLens matches or exceeds prior baselines while delivering a substantial latency advantage, running $8.5$--$9.9\times$ faster than Thyme and up to $22.2\times$ faster than training-free multi-pass search methods.
• 中文摘要 多模态大型语言模型（MLLM）在处理细粒度视觉搜索时遇到困难，该任务是定位高分辨率图像中的小型或稀有物体。现有的解决方案分为两类：（1）基于注意力或信心评分的无训练方法准确但速度较慢，因为它们每个样本需要多次MLLM查询。（2）强化学习（RL）训练的工具使用模型推理速度更快，但由于工具调用仍不可控且难以解释，因此具有不透明度。为克服这一问题，我们提出了\emph{VisLens}（通过Logit Lens实现视觉聚焦），这是一种基于logit透镜的视觉搜索方法，通过LLM头部投射隐藏状态来解码隐藏状态中的语义。VisLens进一步使用轻量级调谐透镜，将早期隐藏状态映射到最终隐藏状态空间，从而从早期层读取视觉符号。这些标记与查询中的目标词匹配，生成相关区域的裁剪，并与原始图像一起反馈，生成最终答案。从解码到最终答案的整个过程，只需一次前向传递即可完成，无需重复查询。VisLens 在延迟方面达到或超过先前基线，同时提供显著的延迟优势，运行速度比 Thyme 快 $8.5 至 $9.9\times，最高可比无训练的多遍搜索方法快 $22.2\ 倍。

Learning from What You Retrieve: Online RL Fine-Tuning for Semantic Retrieval

从你所检索的内容中学习：在线强化学习语义检索的微调

• Authors: Shaowei Wei, Chong Huang, Songtao Fang, Jin Zhang, Zhuojun Wang, Chengfu Huo
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.30753
• Pdf link: https://arxiv.org/pdf/2608.30753
• Abstract In large-scale e-commerce retrieval, dual-encoder retrievers are op- timized for contrastive similarity, whereas downstream rerankers capture finer-grained relevance preferences; this objective mis- match limits end-to-end retrieval quality. Reinforcement Learning offers a way to use reward-model feedback for retriever adaptation, but we observe that standard policy-gradient updates can degrade embedding geometry, especially when the document index must remain frozen due to industrial constraints. To address this, we propose PAO (Positive-Advantage-Only), a selective RL optimization method. Our analysis reveals that in- discriminate penalization of negative samples (pushing away) in a frozen high-dimensional space disrupts pre-trained semantic man- ifolds. PAO selectively applies gradient updates only to retrieved items with positive advantages, effectively pulling query embed- dings toward high-reward regions while preserving global topo- logical stability. Experiments on both a massive industrial dataset and public benchmarks demonstrate that PAO significantly outper- forms standard RL and distillation baselines.
• 中文摘要 在大规模电子商务检索中，双编码器检索器通过操作时实现对比相似性，而下游重新排序器则捕捉更细粒度的相关性偏好;这种客观的不匹配限制了端到端检索的质量。强化学习提供了利用奖励模型反馈进行检索器适应的方法，但我们观察到标准策略梯度更新会降低嵌入几何，尤其是在文档索引因工业限制必须保持冻结时。为此，我们提出了PAO（仅正优势），一种选择性强化学习优化方法。我们的分析显示，在冻结的高维空间中对负样本进行无差别惩罚（推开）会破坏预先训练的语义人流。PAO仅对具有积极优势的检索项目进行梯度更新，有效将查询嵌入-数据拉向高回报区域，同时保持全局拓扑稳定性。在庞大的工业数据集和公开基准测试上的实验表明，PAO显著优于标准强化学习和蒸馏基线。

T3S: Improving Multi-Task Reinforcement Learning with Task-Specific Feature Selector and Scheduler

T3S：通过任务专属功能选择器和调度器提升多任务强化学习

• Authors: Yuanqiang Yu, Tianpei Yang, Yongliang Lv, Yan Zheng, Jianye Hao
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.30765
• Pdf link: https://arxiv.org/pdf/2608.30765
• Abstract Multi-task reinforcement learning (MTRL) is a technique to train multiple tasks simultaneously, where previous works usually train a single model to solve different tasks by sharing parameters across various tasks. However, these methods are faced with inter-task interference since what parameters should be shared across tasks is not addressed, dramatically reducing learning efficiency. To solve these problems, we propose a novel MTRL framework called Task-Specific feature Selector and Scheduler (T3S), which consists of two components: a feature selector and a task scheduler. Specifically, the feature selectors employ hypernetworks to construct task-specific soft masks, which can be applied by globally shared representation to construct task-specific features. The task scheduler selects tasks for learning through two metrics, where the selection probability is inversely proportional to task progress (e.g., success rate) and task learning speed. Experimental results show that T3S consistently outperforms the state-of-the-art MTRL algorithms on various robotics manipulation tasks.
• 中文摘要 多任务强化学习（MTRL）是一种同时训练多个任务的技术，而以往的研究通常通过在不同任务间共享参数来训练单一模型来解决不同任务。然而，这些方法存在任务间干扰的问题，因为未解决任务间应共享的参数，极大降低学习效率。为解决这些问题，我们提出了一种名为任务特定功能选择器与调度器（T3S）的新型MTRL框架，由两个组成部分组成：特征选择器和任务调度器。具体来说，特征选择器利用超网络构建任务特定的软掩码，这些软掩码可通过全局共享表示构建任务特定特征。任务调度器通过两个指标选择学习任务，其中选择概率与任务进度（如成功率）和任务学习速度成反比。实验结果显示，T3S在各种机器人操作任务中持续优于最先进的MTRL算法。

S3C-LLM: Skill-Code Guided Agentic Language Models for Spectrum-to-Structure Elucidation

S3C-LLM：用于谱到结构阐明的技能代码引导代理语言模型

• Authors: Xuanle Zhao, Xinyuan Cai, Xiang Cheng, Bo Xu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.30910
• Pdf link: https://arxiv.org/pdf/2608.30910
• Abstract Spectroscopic structure elucidation is central to molecular analysis, but recent Large Language Model (LLM)-based methods mostly formulate it as direct spectrum-to-SMILES generation. Although this paradigm can leverage paired spectral data, it does not explicitly model the analytical workflow used by spectroscopists, such as diagnostic peak interpretation, fragment reasoning, formula constraints, and chemical consistency checking. In this paper, we introduce S3C-LLM, a skill-guided and code-grounded agentic LLM for spectrum-to-structure elucidation. Rather than directly predicting a molecule, S3C-LLM retrieves modality-specific spectroscopy skills, executes analysis code to instantiate these skills on the input spectra, and integrates the resulting peak-level evidence and formula constraints before generating SMILES. Specifically, we contribute a self-evolving spectroscopy skill library, a thinking-augmented skill-code trajectory construction pipeline, and a two-stage training strategy that teaches Qwen3-4B through supervised fine-tuning (SFT) followed by our proposed step-level reinforcement learning (RL). Experiments on diverse benchmarks show that S3C-LLM consistently outperforms current general LLMs and spectrum-specific models across spectra, while using less than 1/10th of SpectraLLM's training corpus.
• 中文摘要 光谱结构解析是分子分析的核心，但近期基于大型语言模型（LLM）的方法大多将其表述为直接从谱到SMILES生成。尽管该范式可以利用配对谱数据，但它并未显式建模谱学家使用的分析流程，如诊断峰解释、片段推理、公式约束和化学一致性检查。本文介绍了S3C-LLM，一种基于技能指导和代码基础的能动性LLM，用于谱到结构的阐明。S3C-LLM不直接预测分子，而是检索特定模态的光谱技能，执行分析代码在输入光谱上实现这些技能，并在生成SMILE前整合由此产生的峰值级证据和公式约束。具体来说，我们贡献了一个自我演化的光谱学技能库、一个思维增强的技能代码轨迹构建流程，以及一个两阶段训练策略，通过监督微调（SFT）教授Qwen3-4B，随后是我们提出的步骤级强化学习（RL）。在多种基准测试上的实验显示，S3C-LLM在使用不到SpectraLLM训练语料的十分之一的情况下，持续优于现有通用LLM和光谱特定模型。

Beacon: LLM Multi-Agent Driven Hardware Design Space Exploration for Heterogeneous Multi-Chiplet Deep Learning Accelerators

信标：多智能体驱动硬件设计空间探索，面向异构多芯片深度学习加速器

• Authors: Boyu Li, Zongwei Zhu, Qianyue Cao, Xi Li, Xuehai Zhou
• Subjects: Subjects: Hardware Architecture (cs.AR)
• Arxiv link: https://arxiv.org/abs/2608.30932
• Pdf link: https://arxiv.org/pdf/2608.30932
• Abstract Heterogeneous multi-chiplet accelerators allow chiplets to be configured independently to better match different operator characteristics and improve inference efficiency. However, heterogeneity makes simulator evaluation expensive, limiting the number of iterations affordable for hardware design space exploration (HW-DSE). Mainstream data-driven methods rely mainly on final metrics and a few predefined states, and require many search iterations to implicitly learn the relationships between input parameters and optimization objectives, making them less effective in this setting. In practice, evaluators also generate detailed reports on execution timelines, resource utilization, memory accesses, and communication behavior. Large language models (LLMs) can combine domain knowledge with these reports to explicitly identify bottleneck locations, degradation causes, and parameter adjustment directions, thereby improving each design decision under limited iteration budgets. Based on this observation, we propose Beacon, a report-driven LLM multi-agent framework for heterogeneous multi-chiplet HW-DSE. Beacon employs hierarchical agents for bottleneck localization, root-cause diagnosis, and hardware candidate generation, together with an Analysis Toolbox and RAG memory for closed-loop search. Under the same limited iteration budget, Beacon reduces the composite latency-energy-monetary-cost objective by 25.1\%--93.5\% compared with random search, Bayesian optimization, and reinforcement learning.
• 中文摘要 异构多芯片加速器允许芯片组独立配置，以更好地匹配不同操作员特性，提高推理效率。然而，异构性使模拟器评估成本高昂，限制了硬件设计空间探索（HW-DSE）可负担的迭代次数。主流数据驱动方法主要依赖最终指标和少数预定义状态，需要多次搜索迭代以隐式学习输入参数与优化目标之间的关系，因此在这种环境中效果较差。实际上，评估者还会生成关于执行时间线、资源利用率、内存访问和通信行为的详细报告。大型语言模型（LLMs）可以将领域知识与这些报告结合，明确识别瓶颈位置、退化原因和参数调整方向，从而在有限迭代预算下改进每项设计决策。基于这一观察，我们提出了Beacon，一个基于报告的大型语言模型多智能体框架，用于异构多芯片组硬件-DSE。Beacon采用分层代理进行瓶颈定位、根因诊断和硬件候选生成，并配备分析工具箱和RAG存储器进行闭环搜索。在相同的有限迭代预算下，信标将复合延迟-能源-货币成本目标降低了25.1%--93.5%，相比随机搜索、贝叶斯优化和强化学习。

LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation

LightNav-0：引发VLM空间智能以实现通用具身导航

• Authors: Shaoan Wang, Aocheng Luo, Fei Huang, Jingyi Xu, Xiaoyang Wang, Yueyu Wang, Qianli Ma, Fan Yang, Ran Mei, Jia Wei, Jiangpeng Hu, Xuhao Liu, Hongming Chen, Yuanbin Shao, Yiyang Lin, Ziliang Li, Liang Pan, Xinhang Liu, Yuntao Ma, Tingxiang Fan
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.30935
• Pdf link: https://arxiv.org/pdf/2608.30935
• Abstract Embodied navigation requires agents to translate heterogeneous goals and visual observations into actions across tasks, environments, and robot embodiments. Modern vision-language models (VLMs) already encode spatial priors for visual grounding, spatial reasoning, and pointing, but these capabilities are rarely elicited directly for robot control. Existing navigation systems instead rely on task- or embodiment-specific components, fragmenting perception, reasoning, and action while offering limited generalization. Here we present LightNav-0, a compact generalist embodied navigation model that elicits the spatial intelligence of a pretrained VLM and aligns it with navigation, without task-specific prediction heads. LightNav-0 represents diverse navigation tasks through a unified token interface: dual-channel pointing expresses task-, scene-, and embodiment-agnostic spatial intent, while a residual vector-quantized action tokenizer maps this intent to precise, embodiment-specific trajectories. Together with temporally aware visual history compression, ER mid-training, supervised fine-tuning, and reinforcement learning, this formulation supports instruction following, open-vocabulary object navigation, and visual tracking within a single model. The navigation training corpus spans 2K+ scenes and 4K+ hours of embodied navigation data. LightNav-ER, the embodied-reasoning checkpoint used to initialize LightNav-0, attains the highest complete-set average across 8 embodied-reasoning benchmarks, while LightNav-0 achieves state-of-the-art monocular success rates across all 10 public navigation simulation settings. Real-world evaluations further demonstrate zero-shot generalization across robot embodiments, diverse scenes, and static and dynamic targets. These results establish compact VLMs as a unified and transferable backbone for generalist embodied navigation.
• 中文摘要 具身导航要求智能体将异构目标和视觉观察转化为跨任务、环境和机器人具象的行动。现代视觉语言模型（VLM）已编码空间先验以实现视觉基础、空间推理和指向，但这些能力很少直接用于机器人控制。现有导航系统依赖任务或身体特定组件，分散感知、推理和行动，同时提供有限的泛化。这里介绍LightNav-0，一种紧凑的通用具象导航模型，能够引发预训练VLM的空间智能并将其与导航对齐，无需任务特定预测。LightNav-0 通过统一的令牌界面表示多样化的导航任务：双通道指向表达任务、场景和具象无关的空间意图，而残余矢量量化动作分词器则将该意图映射到精准的具象特定轨迹。结合时间感知视觉历史压缩、ER 中训练、监督微调和强化学习，该表述支持单一模型内的指令跟踪、开放词汇对象导航和视觉跟踪。导航训练语料库涵盖 2K+ 场景和 4K+ 小时的具象导航数据。LightNav-ER 作为用于初始化 LightNav-0 的具象推理检查点，在 8 个具象推理基准测试中实现了最高的完整集平均值，而 LightNav-0 在所有 10 种公共导航模拟设置中实现了最先进的单眼成功率。实际评估进一步展示了零射击推广能力，适用于机器人体型、多样场景以及静态和动态目标。这些结果确立了紧凑型VLM作为通用具象导航统一且可转移的骨干。

One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning

一个策略就足够：单智能体强化学习在化学工具学习中优于树状搜索

• Authors: Armin Dariani, Sifan Wu, Bang Liu, Entao Yang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.30952
• Pdf link: https://arxiv.org/pdf/2608.30952
• Abstract Chemistry questions often demand exact computation and database lookups that a language model cannot supply from its parameters, so it must reach for external tools. Tool use here is a three-part problem: select the right tool from a large pool, fill it with correctly typed arguments, and chain calls so that each consumes the outputs of the last. CheMatAgent, a previously published system, addresses this with hierarchical evolutionary MCTS: separate policy and execution models searching tool-call trees under two learned critics, one regressed partly onto GPT-assigned scores. We show that a single policy suffices. Our model interleaves reasoning, tool calls, and returns in one left-to-right generation, trained by a supervised warm-up and then outcome-level reinforcement learning against a programmatic reward read directly off the gold call chain, which leaves no learned critic and no judge in the training loop. On ChemToolBench multiple-tool comprehensive chemistry, on both backbones CheMatAgent use, we improve Tool F1 by 5.5% and Return F1 by 9.6% on Qwen-2.5-7B, and by 3.7% and 3.9% on Llama-3.1-8B, compared with their strongest search configuration, at one model invocation per question, against a search whose cost grows with the tree; we also lead answer Pass Rate on Qwen-2.5-7B.
• 中文摘要 化学问题通常需要精确计算和数据库查找，而语言模型无法从参数中提供这些信息，因此必须依赖外部工具。工具的使用是三部分问题：从大工具池中选择合适的工具，填充正确类型的参数，并链式调用，使每个调用都消耗上一个的输出。CheMatAgent，一个先前发布的系统，通过分层进化MCTS解决了这个问题：分别使用策略和执行模型，在两个学习批评者下搜索工具调用树，其中一个部分回归到GPT分配的评分。我们证明，单一策略就足够了。我们的模型在一次从左到右的生成中交错推理、工具调用和返回，通过监督的热身训练，然后通过结果级强化学习与直接从金调用链读取的程序奖励进行训练，这样训练循环中没有学习批评者和评判者。在ChemToolBench多工具综合化学中，CheMatAgent使用的两种骨干链，Qwen-2.5-7B工具F1提升了5.5%，返回F1提升了9.6%，Llama-3.1-8B则分别提升了3.7%和3.9%，相比它们最强的搜索配置（每题调用一次模型，且成本随树增加而增加）;我们也领先Qwen-2.5-7B的回答通过率。

When Does Predictor-Based RL Align with Human Perception? A Study of Subjective Rewards in Codec-Based Speech Language Models

基于预测变量的强化学习何时与人类感知相匹配？基于编码器的语音语言模型中主观奖励的研究

• Authors: Joonyong Park, Jerry Li
• Subjects: Subjects: Computation and Language (cs.CL); Sound (cs.SD); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2608.31035
• Pdf link: https://arxiv.org/pdf/2608.31035
• Abstract Codec-based text-to-speech (TTS) models make language-model post-training applicable to speech generation, but it remains unclear when learned perceptual predictors can serve as reinforcement learning rewards without losing alignment with human listeners. We study this question with Group Relative Policy Optimization (GRPO) using learned rewards for anime-like speaking style, naturalness, likability, and arousal. To prevent perceptual rewards from being optimized through transcript drift, we introduce a character error rate (CER) zone constraint and compare policy optimization with Best-of-$N$ reranking under the same reward gate. Across single-reward runs, each reward primarily improves its own target metric, showing that subjective predictors are not interchangeable quality surrogates. Multi-rater A/B tests further show uneven human transfer, while a reward-gap analysis separates average transfer from within-axis calibration: signed reward gaps significantly predict listener choices in the pooled analysis, whereas residual CER gaps do not, but per-axis calibration remains heterogeneous. Best-of-8 is a strong human-level baseline and is not clearly worse than GRPO perceptually, suggesting that GRPO should be viewed as amortizing reward-selected behavior into the policy rather than uniformly outperforming reranking. These results support analyzing subjective speech rewards as predictor-axis-base tuples and provide practical diagnostics for selecting rewards before multi-reward speech post-training.
• 中文摘要 基于编码器的文本转语音（TTS）模型使语言模型后训练适用于语音生成，但尚不清楚何时学到的感知预测变量能作为强化学习奖励而不失去与人类听者的一致性。我们利用群体相对策略优化（GRPO）研究该问题，利用对动漫式口语风格、自然性、喜感和唤起的学习奖励。为防止感知奖励通过文字本漂移优化，我们引入字符错误率（CER）区域约束，并将策略优化与$N最佳奖励在同一奖励门下重新排序进行比较。在单次奖励运行中，每个奖励主要提升自身目标指标，表明主观预测变量并非可互换的质量替代。多评级A/B测试进一步显示人类转移不均，而奖励-差距分析则将平均转移与轴内校准区分开来：带符号奖励差距显著预测汇总分析中的听众选择，而残余的CER差距则不然，但每轴校准仍是异质性的。8局三胜是强有力的人类水平基线，感知上并不明显低于GRPO，表明GRPO应被视为将奖励选择行为摊销到策略中，而非统一优于重新排序。这些结果支持将主观语音奖励作为预测轴基础元组分析，并为训练后多奖励语音前选择奖励提供了实用诊断。

Normalized Low-Rank Adaptation

归一化低阶适应

• Authors: Jiale Kang, Ziyin Yue, Zheng Zhan, Yangyi Huang, Weiyang Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.31036
• Pdf link: https://arxiv.org/pdf/2608.31036
• Abstract While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA without requiring repeated normalization throughout training. Across pretraining, supervised finetuning, and reinforcement learning, NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting. These benefits require neither additional trainable parameters nor inference-time computation, making NoRA a simple and broadly applicable enhancement to LoRA.
• 中文摘要 虽然低秩适应（LoRA）被广泛用于参数高效模型适应，但如何使其训练动态正则化以实现稳定有效优化仍缺乏探索。由于LoRA将上投影初始化为零，其早期优化动态主要受降投影控制。基于这一观察，我们引入了归一化低秩适应（NoRA），这是一种简单但有效的方法，可在训练过程中规范下投影矩阵。我们还进一步证明，仅在初始化时应用相同的归一化，从而提升标准LoRA，而无需在整个训练过程中反复归一化。在预训练、监督微调和强化学习中，NoRA持续加速收敛，提升性能和训练稳定性，并减少灾难性遗忘。这些优势无需额外可训练参数或推理时间计算，使NoRA成为LoRA的简单且广泛适用的增强。

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

政策提炼真的能提炼吗？从吵闹的老师到自我提升

• Authors: Yi Ding, Ruqi Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.31046
• Pdf link: https://arxiv.org/pdf/2608.31046
• Abstract On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
• 中文摘要 策略提纯（OPD）提供了密集的代币级监督，作为与可验证奖励强化学习（RLVR）稀疏结果层级优势的替代方案。然而，教师评分的学生生成轨迹本质上不符合其策略，因此其监督的可靠性以及学生进步的来源仍不明确。我们定量分析了OPD培训期间教师监督，发现存在大量噪声，其普遍性随着教师规模的扩大而增加。令人惊讶的是，学生政策对这种噪声不敏感，无论是否保留或取消噪声监督，学生策略都趋向可比的表现。OPD会提炼吗？通过分析驱动其收益的因素，我们发现学习主要集中于低对数概率的标记，使用单一固定负优势的指标与教师提供代币的表现相匹配。这表明 OPD 主要通过抑制低对数概率的标记来工作，而这不需要教师。这些发现激励了 On-Policy 自我适应（OPSA），这是一种利用熵自适应负优势的无监督方法。它为高熵位置分配更强的学习信号，抑制尾部标记，并在头部标记间均匀分配概率质量。与基础 \texttt{Qwen3-1.7B} 相比，OPSA 在 AIME24 上提升了 35.41 个Avg@32，相对提升 263% 百分比，且在三个基准测试中Pass@32均超过翻倍。它在 AIME24 上Avg@32中也比 OPD 高出 16.77 个百分点。跨模型家族和任务的大量实验和分析进一步证明了其有效性和普适性。

Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence

超越人类监督的大规模推理模型扩展：迈向超级智能之路

• Authors: Zhiqin Yang, Jingwen Fu, Yuhan Liu, Hengyu Liu, Yonggang Zhang, Kainan Cao, Zizhuo Zhang, Chenxin Li, Ruibin Yuan, Jiahao Pan, Jiankai Sun, Zhenyuan Zhang, Yibo Li, Yunlong Lin, Jing Xiong, Sida Lin, Bo Han, Wei Xue, Yike Guo
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.31075
• Pdf link: https://arxiv.org/pdf/2608.31075
• Abstract Recent advances in large reasoning models (LRMs) have shown that reinforcement learning with verifiable rewards (RLVR) can substantially improve reasoning in mathematics and code, where outcomes can be checked automatically. Extending this progress to open-ended and agentic tasks remains difficult because reliable rewards are harder to obtain and direct human supervision cannot keep pace with the scale and complexity of model-generated experience. This paper studies how LRMs can continue to improve as human supervision gradually recedes from the learning loop. We examine two connected dimensions of this problem. The reward axis traces the development from per-instance human judgments to reusable verifiers and rewards that operate even without human feedback. The experience axis examines how learning can progress from human-curated tasks and environments toward self-generated curricula, constructed environments, and autonomous co-evolution. We connect these dimensions through a five-level ladder from L0 to L4 that identifies which parts of the learning process remain under continued human control. Our analysis further highlights the risks introduced by increasingly autonomous rewards and experience generation, including reward hacking, feedback drift, curriculum collapse, and environment errors. Consequently, we also provide the evaluation around three complementary objects: policy capability, feedback fidelity, and experience quality. This analysis provides a structured account of current approaches to scaling LRMs beyond human supervision and the open problems involved in developing self-sustaining learning systems toward superintelligence. Furthermore, we maintain a continuously updated \href{this https URL}{GitHub repository} to track the latest advances.
• 中文摘要 大型推理模型（LRM）的最新进展表明，带可验证奖励的强化学习（RLVR）能够显著提升数学和代码中的推理能力，因为结果可以自动检查。将这一进展扩展到开放式和代理性任务仍然困难，因为可靠的奖励更难获得，且直接的人类监督无法跟上模型生成体验的规模和复杂性。本文研究了随着人类监督逐渐退出学习循环，LRMs如何持续改进。我们考察了该问题的两个相互关联的维度。奖励轴追溯了从每实例的人类判断到即使没有人类反馈也能运作的可重复使用的验证器和奖励的发展。经验轴考察学习如何从人类策划的任务和环境，向自我生成的课程、构建环境和自主的共演进发展。我们通过从L0到L4的五级阶梯连接这些维度，识别学习过程中哪些部分仍处于持续的人类控制之下。我们的分析进一步强调了奖励和体验生成日益自主所带来的风险，包括奖励黑客、反馈漂移、课程崩溃和环境错误。因此，我们还围绕三个互补对象进行了评估：策略能力、反馈忠实度和体验质量。本分析结构化地介绍了当前将LRMS扩展到超越人类监督的方法，以及开发自我维持学习系统向超级智能发展过程中面临的未解问题。此外，我们维护一个持续更新的\href{this https URL}{GitHub repository}，以跟踪最新进展。

Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization

在代理性政策优化中调和流程监督与基于结果的信用

• Authors: Jingxiao Yang, Wangjie Gan, Yingxuan Zhuang, Wenqi Zhang, Jintao Chen, Xuhong Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.31077
• Pdf link: https://arxiv.org/pdf/2608.31077
• Abstract Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective.
• 中文摘要 基于结果的强化学习为语言模型代理提供经过验证的反馈，但对所有决策均一致分配轨迹级优势，从而在长期相互作用中获得粗略的认可。策略自提纯通过重新评估仅在训练期间可用的特权信息（PI）的抽样行为，提供更细致的监督。然而，细粒度监督不一定是细粒度的认可：PI诱导的似然变化描述了额外信息如何改变策略偏好，但并不能直接决定可执行动作如何继承已验证的任务结果。这造成了监督与认可之间的差距。特权信号可能与当前交互状态无关，操作的细节度与可执行决策不匹配，且缺乏强化所需的结果语义。我们引入了TASPO技术，将特权监督转化为基于结果的行动认可。TASPO根据经过验证的成功经验构建决策适用的PI，汇总PI在可执行动作层级的概然偏移，并将相对行动支持转换为对原始轨迹优势的正、有界、均值保持权重。因此，验证结果决定了更新方向和平均尺度，而PI仅重新分配了各行动的功劳。在三个代理基准中，TASPO比GRPO提升10.6%，并更好地推广到未见任务。进一步分析表明，TASPO减少了监督不匹配，动作级分配稳定了策略优化过程。这些发现为社区提供了另一个有趣的视角。

DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution

DreamX-Creator：2K分辨率下实现原生音视频生成的民主化

• Authors: Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2608.31106
• Pdf link: https://arxiv.org/pdf/2608.31106
• Abstract Recent video generators often omit audio or synthesize it in a separate stage, limiting reciprocal modeling of visual dynamics and acoustic events. We present DreamX-Creator 1.0, a compact native joint audio-video generation system centered on a 7B generator. Conditioned on a first frame and a text prompt, the generator jointly denoises modality-specialized audio and video streams. The streams are processed independently in the first half of the network and coupled in the latter half through Gated Cross-Modal Attention, whose token- and head-wise output gates modulate each active cross-modal attention-head output. A unified Audio-Video Data System constructs and filters temporally coherent clips, produces structured multimodal annotations, and organizes clips into capability-oriented data pools. Progressive Joint Training comprises two audio-video pre-training stages followed by High-Quality Finetuning. Audio-Video Reinforcement Learning further post-trains the generator with Modality-Aware Multimodal Feedback that routes video-, audio-, and cross-modal feedback to the corresponding streams. For high-resolution output, our Autoregressive 1-Step 2K Refinement pipeline adapts a bidirectional multi-step teacher into an autoregressive multi-step refiner and distills it into a student requiring one denoising evaluation per temporal chunk. Overall, DreamX-Creator 1.0 achieves native, synchronized audio-video generation with performance competitive with state-of-the-art open-source systems. By releasing our compact 7B generator and 2K Refiner, we seek to democratize native audio-video generation and provide an accessible foundation for future research in unified audio-video generative modeling.
• 中文摘要 近期视频生成器常常省略音频或在单独阶段合成，限制了视觉动态和声学事件的互惠建模。我们介绍DreamX-Creator 1.0，一个以7B生成器为核心的紧凑原生联合音视频生成系统。该生成器以第一帧和文本提示为条件，联合去噪针对特定模态的音频和视频流。网络前半部分的流独立处理，后半部分通过门控跨模态注意力（Gated Cross-Modal Attention）耦合，其令牌和头部输出门调制每个活跃的跨模态注意力-头输出。统一的音视频数据系统构建并过滤时间相干的剪辑，生成结构化的多模态注释，并将剪辑组织为面向能力的数据池。渐进式联合训练包括两个音视频预训练阶段，随后是高质量微调。音频-视频强化学习进一步通过模态感知多模态反馈后期训练生成器，将视频、音频和跨模态反馈导向相应的流。对于高分辨率输出，我们的自回归1步2K精炼流水线将双向多步教师改编为自回归多步精炼器，并将其提炼为每个时间块需进行一次去噪评估的学生。总体而言，DreamX-Creator 1.0实现了原生同步音频视频生成，性能可与最先进的开源系统媲美。通过发布我们的紧凑型7B生成器和2K Refiner，我们旨在普及原生音视频生成，并为未来统一音视频生成建模研究提供可访问的基础。

PaperGym: Rubric-Centered Evolution for Research-Plan Generation

PaperGym：以评分标准为中心的进化研究计划生成

• Authors: Yuhan Wang, Zhengxi Lu, Yuchen Yan, Kaitao Song, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.31119
• Pdf link: https://arxiv.org/pdf/2608.31119
• Abstract Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.
• 中文摘要 研究规划是人工智能科学家的关键能力。然而，研究计划没有可验证的答案，因此强化学习缺乏所需的环境：任务与批评者结合。从科学论文中提取的评分标准可以提供批评者。然而，现有流程从相同内容中抽取问题和标准，因此可以通过意译获得奖励。每次推广，评分标准进一步压缩为单一标量。我们引入了PaperGym，一个统一框架，将每篇研究论文转化为完整的培训环境。PaperGym利用论文结构：问题由研究目标和背景综合而成，标准则来自方法和实验。标准涵盖方法论创新和实验设计，标准泄漏率降至3.7%，而现有数据集中为11.90%至34.10%。训练中两次使用评分标准：第一次作为OPSD自学者的特权上下文，第二次作为GRPO的奖励。在Qwen3-1.7B/4B/8B中，该计划优于监督微调（任一阶段）和反向排序，提升了五个基准平均值+5.6、+5.0和+4.8分。在配方固定的情况下，PaperGym-20k训练的模型在三方比较中获胜率为58.1%，而RubricHub Science为28.2%。训练好的Qwen3-8B在ResearchQA上达到73.48，高于规模更大的Kimi K2.6。我们发布了流水线、2万个实例的PaperGym-20k语料库，以及基准测试PaperGym-Innov和PaperGym-Design。

Keyword: diffusion policy

There is no result