生成时间: 2026-08-25 16:44:17 (UTC+8); Arxiv 发布时间: 2026-08-25 20:00 EDT (2026-08-26 08:00 UTC+8)

今天共有 55 篇相关文章

Keyword: reinforcement learning

AI Learning and Conceptual Transfer in the Game of Hidden Rules

隐藏规则游戏中的人工智能学习与概念转移

• Authors: Christo Mathew, Wentian Wang, Jacob Feldman, Lazaros K. Gallos, Paul B. Kantor, Vladimir Menkov, Hao Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.21372
• Pdf link: https://arxiv.org/pdf/2608.21372
• Abstract This report summarizes the work conducted on the Game of Hidden Rules (GOHR), focusing on reinforcement learning agents trained to infer hidden rules from trial-and-error feedback, representation design, rule difficulty analysis, transfer learning, generalization, and pseudo-bot-assisted human learning analysis. The report focuses on the Transformer-based A2C framework, Feature-Centric and Object-Centric representations, experimental findings, and classification of human learning data.
• 中文摘要 本报告总结了关于隐藏规则博弈（GOHR）的工作，重点关注训练通过试错反馈推断隐藏规则的强化学习代理、表示设计、规则难度分析、迁移学习、泛化以及伪机器人辅助人类学习分析。报告聚焦基于Transformer的A2C框架、以特征为中心和以对象为中心的表示、实验发现以及人类学习数据的分类。

Model of Models: When Does Emitting a Specialist Beat Attending, Adapting, or Tuning?

模型中的模型：何时由专业人员生产胜过关注、适应或调谐？

• Authors: John C. Howell
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21386
• Pdf link: https://arxiv.org/pdf/2608.21386
• Abstract Given a task described by a few examples, how should a model be specialized to it? Four mechanisms are available -- zero-shot, in-context attention, test-time gradient adaptation, and emitting specialist weights from a hypernetwork -- yet the operating regime of the last is rarely mapped. We run the identical four-way comparison across six tasks spanning regression, generation, language modeling, reinforcement learning, and clinical and genomic classification, holding the specialist, the context, and (where we can) the training budget fixed. The clearest wins for emission are about cost at matched quality: it ties the state-of-the-art amortized tabular model (TabPFN) on clinical few-shot classification while emitting a reusable specialist instead of re-attending the support set per query, and reaches noise-floor shape generation with a $132$-float per-instance program. On few-shot sinusoid regression it is $2$--$3$ orders of magnitude below MAML at zero test-time gradient steps -- a margin that narrows to $\sim$$30\times$ but persists once training budgets are equalized. Emission cannot match in-context attention on high-dimensional sequence modeling: under matched-budget pre-training a one-pass adapter recovers only a minority of the in-context gain ($14.0\pm0.9\%$ at $5$M, $11.2\pm0.5\%$ at $15$M), and a LoRA-rank sweep shows this shortfall is a partial capacity limit -- capture climbs from $5\%$ to $21\%$ as rank grows but plateaus far below full recovery. Mechanism ablations confirm the emitted specialist is genuinely task-conditioned, not a memorized prior; and, more speculatively, emitted specialists compose in weight space -- interpolating two of them tracks the corresponding blend of their functions. We close with a falsifiable thesis, operationalized through a per-task resolution measure, bounding when each conditioning mechanism should be preferred.
• 中文摘要 对于一个任务，以几个例子为例，模型应如何专门化？有四种机制可用——零样本、上下文关注、测试时间梯度适应和从超网络发射专家权重——但后者的操作模式很少被映射。我们在六个任务中进行了相同的四向比较，涵盖回归、生成、语言建模、强化学习以及临床和基因组分类，保持专家、上下文以及（尽可能）训练预算的固定。发射最明显的优势在于成本匹配质量：它将最先进的摊销表模型（TabPFN）与临床少数样本分类结合起来，同时输出可重用的专家，而无需每次查询重复访问支持集，并通过每实例132美元的浮动量程序实现噪声底形状生成。在少样本正弦回归中，在零测试时间梯度步中，这一差值比MAML低2美元至3美元数量级——这一差距缩小到30美元，但一旦训练预算平衡，这一差距依然存在。发射无法匹配高维序列建模中的上下文关注：在匹配预算下，一次过的适配器只能回收上下文内收益的一小部分（500万美元时14.0\pm0.9\%$，15万美元时11.2\pm0.5\%$），LoRA排名扫描显示这一缺口是部分容量限制——捕获量从5美元攀升至21美元，随着等级增长但远低于完全恢复。机制消融确认发射的专家是真正的任务条件反射者，而非记忆的先验;更推测的是，发射专家在权重空间中构成——插值其中两个能追踪其相应功能的混合。我们以一个可证伪的论点结束，通过每个任务的解决度量操作化，界定了每个条件反射机制何时应优先。

Runtime Action Interference for AI Control of AlphaStar in StarCraft II

星际争霸II中AI控制AlphaStar的运行时动作干扰

• Authors: Jaymari Chua, Chen Wang, Liming Zhu, Lina Yao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.21398
• Pdf link: https://arxiv.org/pdf/2608.21398
• Abstract A trained reinforcement learning policy does not determine the complete behavior that users encounter: deployment code still schedules, admits, suppresses, or replaces its proposed actions. We contribute \emph{runtime action interference} (RAI), an AI control mechanism that preserves policy parameters while regulating action pacing and filtering configured action patterns after inference. RAI releases a proposed action only when its cooldown condition is satisfied and its content detector does not flag the action; otherwise, it dispatches a no-op. The detector covers specified toxic behaviors, including worker-unit harassment, while the cooldown controls action rate. We implement RAI in a replication of AlphaStar this http URL and make the implementation and reproducibility materials available through an open source code repository. We deployed RAI in a \textit{StarCraft~II} human participant study that compared two presentations of the same opponent with high capability and rate limited actions; we withheld its capability claim in one presentation and disclosed it in the other. On response scales from 1 to 5, we observed pooled fairness, trust, and toxicity means of 3.90, 3.50, and 2.00 under claim withholding, compared with 2.62, 4.31, and 2.85 under disclosure. Disclosure corresponded with lower perceived fairness and higher perceived toxicity across every expertise group, whereas trust increased among novices and experts but decreased among intermediate participants. Our human evaluation therefore shows that perceptions of an opponent controlled through RAI can vary substantially with the capability information presented to users, even when the configured control remains constant. We conclude that human-computer evaluations must separate control within the execution stack from capability disclosure and assess fairness, trust, and toxicity as distinct dimensions of human experience.
• 中文摘要 训练有素的强化学习策略并不能决定用户遇到的完整行为：部署代码仍会调度、承认、抑制或替换其拟议动作。我们贡献了\emph{runtime action interference}（RAI），这是一种AI控制机制，在调整动作节奏的同时，在推断后过滤已配置的动作模式。RAI仅在拟议动作冷却条件满足且内容检测器未标记该动作时发布;否则，它会发送no-op。检测器覆盖指定的有毒行为，包括工人单位骚扰，而冷却时间则控制行动速率。我们在AlphaStar的复制中实现了RAI，该http URL，并通过开源代码库提供实现和可重复性材料。我们在一项\textit{StarCraft~II}人类参与者研究中部署了RAI，该研究比较了两种同一对手的高能力和速率限制动作的表现;我们在一次报告中未披露其能力主张，在另一次报告中披露了。在1到5的响应量表中，我们观察到公允、信任和毒性的合并平均值分别为3.90、3.50和2.00，而披露中为2.62、4.31和2.85。披露与所有专业组中感知公平性较低和毒性感知较高对应，而新手和专家的信任增加，但中级参与者的信任下降。因此，我们的人类评估显示，通过RAI控制的对手的感知会随着向用户呈现的能力信息而有显著差异，即使配置控制保持不变。我们得出结论，人机评估必须将执行栈内的控制与能力披露区分开来，并评估公平性、信任度和毒性作为人类体验的不同维度。

Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

让学分跟随计算：大型语言模型强化学习的架构感知学分传输

• Authors: Qifan Shi, Zhaolu Kang, Chenghua Zhu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21501
• Pdf link: https://arxiv.org/pdf/2608.21501
• Abstract Credit assignment in large-language-model reinforcement learning (LLM RL) can be separated into three objects: evidence about success, a transport operator that converts this evidence into token-level advantages, and an update geometry that turns advantages into policy changes. Recent work has greatly improved evidence, sampling, and update geometry, but the transport operator is usually architecture-agnostic. Fixed-discount GAE applies a stationary geometric kernel along token time; group-relative methods broadcast an outcome statistic across an entire response. Neither operator represents the trajectory-specific computation used by the Transformer policy itself. We introduce computation-conditioned credit transport (CCT), a general framework in which a detached statistic of the behavior policy's internal computation parameterizes the causal kernel that transports downstream value through a rollout. Our concrete algorithm, CompPO, maps native attention concentration to a bounded per-token retention gate, uses the gate in both the one-step bootstrap and a path-dependent generalized-advantage trace (Comp-GAE), and co-designs a transport-aligned critic (TAC) that reuses the actor's hidden states and routing information without a second same-scale Transformer. The task reward and clipped PPO policy objective remain unchanged; a constant gate recovers fixed-coefficient GAE. Across five Qwen3-4B seeds, CompPO reaches 61.4% final held-out accuracy (95% CI [60.8,62.0]) versus 53.8% [52.9,54.7] for tuned GRPO. Neither Comp-GAE with a standard critic (55.2%) nor TAC with a fixed gate (56.4%) matches the full model (interaction +2.4 [1.9,2.9]). Shuffle and position controls confirm trajectory-specific alignment; CompPO is stable in 10/12 PPO-grid runs versus 3/12. Frozen evaluation improves over GRPO by 4.3 and 3.9 greedy pass@1 macro points on Qwen3-4B and Llama-3.1-8B-Instruct.
• 中文摘要 大型语言模型强化学习（LLM RL）中的学分分配可分为三个对象：成功的证据、将证据转化为令牌级优势的传输算子，以及将优势转化为策略变更的更新几何。近期工作大幅改进了证据、抽样和更新几何，但传输算子通常与架构无关。固定折扣GAE沿令牌时间应用一个平稳几何核;组相对方法在整个响应中广播结果统计量。这两种算符都不代表Transformer策略本身所使用的轨迹特定计算。我们引入计算条件信用传输（CCT），这是一个通用框架，其中行为策略内部计算的分离统计量参数化了通过滚动传输下游价值的因果核。我们的具体算法CompPO将本地注意力集中映射到有界的每个令牌保留门，同时在一步引导和路径依赖的广义优势痕迹（Comp-GAE）中使用该门，并共同设计了一种传输对齐批评者（TAC），可在无第二个同等尺度变换器的情况下重用演员的隐藏状态和路由信息。任务奖励和截断的PPO策略目标保持不变;恒定门恢复固定系数GAE。在五个Qwen3-4B种子中，CompPO达到61.4%的最终保留准确率（95% CI [60.8,62.0]），而调优GRPO为53.8% [52.9,54.7]。标准批判者（55.2%）的Comp-GAE和固定门的TAC（56.4%）均未与完整模型匹配（交互+2.4 [1.9,2.9]）。洗牌和位置控制确认轨迹特定对齐;CompPO在10/12次PPO网格运行中稳定，而3/12次。冻结评估较GRPO提升4.3，Qwen3-4B和Llama-3.1-8B-Ininstruction的贪婪pass@1点提升3.9。

Forgotten in Weights, Recovered by Tools: Agentic Tool Unlearning for LLM Agents

权重中被遗忘，工具恢复：大型语言模型代理的代理工具去学习

• Authors: Baicheng Chen, Zheyuan Liu, Jingyu Zhang, Kaize Ding, Ningshan Ma, Yue Huang, Meng Jiang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21544
• Pdf link: https://arxiv.org/pdf/2608.21544
• Abstract Large language models (LLMs) are increasingly deployed as tool-augmented agents, where responses can depend on tool calls and external observations rather than model parameters alone. This creates an evaluation mismatch for LLM unlearning: previous unlearning methods may suppress direct parametric recall, but an agent can still recover the same forget target through tools such as web search, retrieval, or database lookup. We identify this failure mode as tool-mediated recovery and study agentic tool unlearning, which aims to reduce both parametric recall and tool-mediated recovery while preserving normal tool use for retained knowledge. To address this challenge, we propose Agentic Tool Unlearning (ATU), a two-stage framework. The first stage applies parametric knowledge unlearning to suppress direct recall, while the second stage performs trajectory-level reinforcement learning in simulated tool-augmented environments to penalize target-seeking tool behavior and final-answer leakage. Experiments on RWKU and MUSE across different LLM architectures show that ATU achieves a better balance between target forgetting and retained utility, making unlearning more robust under tool-augmented agent deployment.
• 中文摘要 大型语言模型（LLM）越来越多地作为工具增强代理部署，响应可以依赖工具调用和外部观察，而非仅依赖模型参数。这导致LLM去学习存在评估不匹配：之前的去学习方法可能抑制直接参数回忆，但代理仍可通过网页搜索、检索或数据库查找等工具恢复同一遗忘目标。我们将这种失败模式称为工具介导恢复和研究代理工具去学习，旨在减少参数回忆和工具介导恢复，同时保留对保留知识的正常工具使用。为应对这一挑战，我们提出了代理工具去学习（ATU）两阶段框架。第一阶段应用参数化知识去学习抑制直接回忆，第二阶段在模拟工具增强环境中执行轨迹级强化学习，惩罚目标寻觅工具行为和最终答案泄露。在不同大型语言模型架构上对RWKU和MUSE的实验表明，ATU在目标遗忘和保留效用之间实现了更好的平衡，使得在工具增强代理部署下，复学更为稳健。

Force/Torque-Based Kinematic Adaptation for Robotic Manipulation Tasks

基于力/扭矩的运动学适应用于机器人操作任务

• Authors: Carl Glen Henshaw (US Naval Research Laboratory)
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.21592
• Pdf link: https://arxiv.org/pdf/2608.21592
• Abstract Contact-rich robotic manipulation requires an accurate model of the kinematic relationship between a robot's joints and the task features it senses. This relationship is rarely known exactly: it changes with each tool the robot picks up and shifts, sometimes almost instantaneously, as contact modes change --- especially for multi-fingered hands that make and break contact at points that are not exactly prescribed, as in full-hand grasping. This paper develops an adaptive scheme that estimates that relationship online, using only joint-angle sensing and a wrist-mounted force/torque sensor, with no exteroceptive measurement of the tool tip. We derive a provably stable kinematic update law that identifies the kinematics of an unknown tool from force/torque feedback alone, and prove stability of both the rigid case and the case with a compliance controller as an inner loop. We show that identification is confined to the directions the motion excites --- so that, for example, a tool's length is unobservable under a rigid insertion push, while a compliant loop's passive yielding partially excites it; and that with a second-order admittance the compliant certificate holds unconditionally in continuous time. We also pose the combined control and estimation problem as a Quadratic Program (QP): the formulation yields the prediction term of the update law exactly but, instructively, cannot reproduce the tracking adaptation term. We validate the scheme in simulation on a peg-in-hole insertion. This work is the first step in a research program aimed at factoring manipulation learning into a task policy which can be learned in isolation of the robot, for instance by reinforcement learning, and an adaptive kinematic component that adapts online to the particular robot, hand, or tool in use.
• 中文摘要 丰富的接触式机器人操作需要对机器人关节与其感知到的任务特征之间的运动学关系建立准确模型。这种关系很少被确切了解：它会随着机器人每拿起一个工具而变化，并且会随着接触模式---的变化而变化，有时甚至几乎瞬间发生变化，尤其是在多指手在非规定点建立和断开接触时，如全手抓握。本文开发了一种自适应方案，仅利用关节角度感测和腕部安装的力/力矩传感器，在线估算这种关系，且不对工具提示进行外感知测量。我们推导出一个可证明稳定的运动学更新定律，仅凭力/扭矩反馈识别未知工具的运动学，并证明刚性外壳和带有顺应控制器作为内环的外壳的稳定性。我们证明，识别仅限于运动---激发的方向，例如工具长度在刚性插入推力下不可观察，而顺应环的被动让步部分激发;并且在二阶准入下，顺应证书在连续时间内无条件保持。我们还将控制与估计结合问题提出为二次规划（QP）：该表述精确给出更新律的预测项，但具有启发性地无法重现跟踪适应项。我们在孔中插入的模拟验证了该方案。这项工作是研究计划的第一步，旨在将操作学习分解进任务策略中，该策略可通过强化学习等独立学习，以及一个在线适应特定机器人、手或工具的自适应运动学组件。

Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

扰乱思想，而非像素：潜在空间推广多样化以强化视觉语言模型学习

• Authors: Michael Jerge, Joseph Pelczar, Justin Downes
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.21595
• Pdf link: https://arxiv.org/pdf/2608.21595
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar $p \le 0.001$) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了视觉语言模型（VLM）的推理能力，且在每个优化组内多样化展开可放大其益处。现有方法通过解码温度或像素空间图像扭曲实现多样化;我们询问扰动是否应处于模型的潜伏空间。我们引入噪声对比GRPO（NC-GRPO），该方法将尺度校准的高斯噪声注入提示编码通道的最后隐藏层，针对每个滚动组的一半成员，将这些展开从位移的出发状态分支。尽管位移仍达到答案的分支会被强化，将分支点的敏感度转换为策略梯度信号;目标、奖励和推理协议保持不变。在Geometry3K训练的Qwen2.5-VL-7B上，NC-GRPO在五个未完成基准测试（合并McNemar $p \le 0.001$）中显著提升了域外数学推理，同时提升了领域内精度和幻觉鲁棒性——后者是图像空间噪声回归的轴，即使在感知重基准测试中发布更高的OOD平均值。机制消融表明，独立的随机多样性而非噪声预算或方向是主动因素，噪声尺度研究揭示了推理专业化与通用能力之间的平衡。NC-GRPO设计为模态无关性，并可作为推理引擎的~50行变更集成到标准RLVR流水线中。

Reinforcement Learning on Benign Facts Amplifies Leakage of Memorized Private Data

基于良性事实的强化学习加剧了记忆中的私人数据泄露

• Authors: Renfei Zhang, Niloofar Mireshghallah
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21727
• Pdf link: https://arxiv.org/pdf/2608.21727
• Abstract Reinforcement learning with verifiable rewards (RLVR) is deployed to make models better at reasoning tasks, but its side effect on what models will divulge is under studied. Here we show that RLVR on facts increases extraction of personally identifiable information (PII) the instruct model had already memorized. We first confirm that instruct models have already memorized PII but leave them latent, rarely surfacing one when asked. We then apply RL on benign factual data that contains no PII of any kind, and re-probe: a targeted probe over name->email pairs, and an untargeted free-recall prompt that simply asks the model to list the addresses it knows. PII extraction rises sharply under both: on DeepSeek-V3.1, verbatim recall@k increases from 0.155 to 0.370, a 2.4x gain. The effect scales with model size: across three models spanning 8B to 671B parameters, absolute leakage is largest in the biggest model. Meanwhile model's reasoning abilities and refusal rates are retained, indicating that RL selectively changes which memorized information is accessible rather than broadly altering the model. In summary, memorized private data can be made markedly more extractable by training that never touches it. This gives an adversary a route to memorized data that requires no privacy-relevant training signal and no access to the data itself -- only the ability to fine-tune on something innocuous.
• 中文摘要 带有可验证奖励的强化学习（RLVR）被用于让模型更擅长推理任务，但其对模型将透露内容的副作用研究不足。我们展示了对事实的RLVR能增加指令模型已记忆的个人识别信息（PII）提取能力。首先确认指令模型已经记忆了PII，但保持潜在状态，很少在被要求时显现。然后对无任何PII的良性事实数据应用强化学习，重新探测：针对姓名>邮件对进行有针对性探测，以及对模型列出已知地址的非目标自由回忆提示。PII提取在两者下均大幅上升：在DeepSeek-V3.1中，逐字的recall@k从0.155增加到0.370，增长2.4倍。该效应随模型规模增长：在涵盖8B至671B参数的三个模型中，最大模型中绝对泄漏最大。与此同时，模型的推理能力和拒绝率得以保留，表明强化学习选择性地改变哪些记忆信息可访问，而非广泛改变模型。总之，通过不接触这些数据的训练，记忆的私密数据可以显著提升提取性。这为对手提供了一条无需隐私相关训练信号、无需访问数据本身的路径——只需对无害的细节进行微调。

CounterAlign: Counterfactual Supervision for Vision-Language-Action Models

CounterAlign：视觉-语言-行动模型的反事实监督

• Authors: Haru Kondoh, Kei Ota, Asako Kanezaki, Yueh-Hua Wu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.21740
• Pdf link: https://arxiv.org/pdf/2608.21740
• Abstract Vision-Language-Action (VLA) models are typically trained with behavior cloning (BC) on expert demonstrations. However, BC provides only positive supervision for expert actions, without explicit negative supervision indicating which actions are instruction-inconsistent or otherwise inappropriate. Reinforcement learning (RL) can provide such corrective signals, but often relies on externally specified rewards or curated non-expert data, both of which are costly to obtain in robotics. We show that offline RL for VLA models need not rely on curated non-expert trajectories: successful expert demonstrations alone can be transformed into dense corrective supervision through instruction relabeling. Specifically, by pairing expert actions with mismatched alternative instructions, we synthesize counterfactual instruction-observation-action tuples from the dataset and combine them with adversarial discriminator training to learn an instruction-grounded reward model for offline RL, without collecting additional rollouts or annotations. On the robustness-focused LIBERO-PRO benchmark, our method improves robustness to object position and task perturbations over a strong state-of-the-art baseline. It also outperforms competitive baselines in real-robot experiments on the TX-G2 (compatible with AGIBot G2). More broadly, our results suggest that, for data-constrained VLA learning, extracting denser supervision from each demonstration can complement collecting additional data.
• 中文摘要 视觉-语言-行动（VLA）模型通常通过行为克隆（BC）在专家演示中进行训练。然而，BC仅对专家行为提供积极监督，没有明确的负面监督指示哪些行为与指令不一致或不合适。强化学习（RL）可以提供此类纠正信号，但通常依赖外部指定的奖励或策划的非专家数据，而这两者在机器人技术中获取成本较高。我们表明，VLA模型的离线强化学习不必依赖策划的非专家轨迹：仅靠成功的专家演示可以通过指令重新标记转化为密集的纠正监督。具体来说，通过将专家动作与不匹配的替代指令配对，我们从数据集中合成反事实指令-观察-动作元组，并结合对抗判别器训练，学习基于指令的离线奖励模型，无需额外部署或注释。在以鲁棒性为重点的LIBERO-PRO基准测试中，我们的方法在强大最先进基线上提升了对物体位置和任务扰动的鲁棒性。它在TX-G2（兼容AGIBot G2）的真实机器人实验中也优于竞争基线。更广泛地说，我们的结果表明，对于数据受限的VLA学习，从每个演示中提取更密集的监督可以补充收集额外数据。

SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering

SAFE-G：结构感知、忠实证据引导的知识型视觉问答生成

• Authors: Long Shu, Shuochen Liu, Wei Chen, Junda Lin, Zhi Zheng, Huijun Hou, Tong Xu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21796
• Pdf link: https://arxiv.org/pdf/2608.21796
• Abstract Knowledge-based Visual Question Answering (KB-VQA) aims to answer queries that necessitate reasoning over external knowledge sources beyond the visual content. Typically, current methods fuse multimodal features to retrieve external information, subsequently leveraging Multimodal Large Language Models (MLLMs) to derive answers from the retrieved evidence. However, these methods often struggle to capture structural associations within complex contexts to effectively filter noise. Furthermore, they frequently fail to ensure that the reasoning process remains strictly faithful to the retrieved evidence. To address these challenges, we propose SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning. Specifically, we first employ a coarse-grained hybrid search fusing visual and textual modalities to recall candidate documents, and subsequently implement a structure-aware fine-grained graph retrieval that captures structural dependencies to filter noise and pinpoint precise evidence. Moreover, we introduce a reinforcement learning (RL) strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct. This strict alignment constraint compels the model to anchor its response in the retrieved context, effectively enhancing its capability to locate evidence via multimodal features and perform faithful reasoning. Extensive experiments on the Encyclopedic-VQA and InfoSeek benchmarks demonstrate that SAFE-G outperforms prior methods by a margin of 8.9% and 3.5%, substantially enhancing the overall reasoning accuracy. Our source code is publicly available at: this https URL.
• 中文摘要 基于知识的视觉问答（KB-VQA）旨在回答需要对视觉内容之外外部知识源进行推理的查询。通常，当前方法融合多模态特征以获取外部信息，随后利用多模态大型语言模型（MLLM）从检索到的证据中推导答案。然而，这些方法常常难以在复杂语境中捕捉结构性联想，以有效过滤噪声。此外，它们常常未能确保推理过程严格忠实于检索到的证据。为应对这些挑战，我们提出了SAFE-G——一种结构感知、忠实证据导向生成框架，能够实现精确的证据定位和可信的推理。具体来说，我们首先采用粗粒度混合搜索，融合视觉和文本模式来回忆候选文档，随后实现结构感知的细粒度图检索，捕捉结构依赖性以过滤噪声并精准定位证据。此外，我们引入了一种基于证据的强化学习（RL）策略，只有当所选证据正确时，才给予正确答案认可。这一严格的对齐约束促使模型将响应锚定于检索的上下文，有效增强了通过多模态特征定位证据和忠实推理的能力。在Encyclopedic-VQA和InfoSeek基准测试上的大量实验表明，SAFE-G比以往方法高出8.9%和3.5%，显著提升了整体推理准确性。我们的源代码公开于：https URL。

MCite-RL: Towards Reliable Multimodal RAG via Citation-enhanced Agentic Reinforcement Learning

MCite-RL：通过引用增强的代理强化学习迈向可靠的多模态RAG

• Authors: Suifeng Zhao, Zida Liu, Xinyu Lei, Lei Sun, Jun Gao, Sujian Li
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.21808
• Pdf link: https://arxiv.org/pdf/2608.21808
• Abstract Multimodal Retrieval-Augmented Generation (RAG) with visual citation is crucial for ensuring the traceability and verifiability of MLLMs. However, current RAG and SFT-based methods struggle to achieve robust cross-modal reasoning, causing imprecise visual citations or decoupling between the citation and the generated answers. To address these limitations, we propose MCite-RL, a citation-enhanced agentic reinforcement learning framework designed for reliable multimodal RAG. MCite-RL introduces an Agentic Refinement module for visual citation that employs iterative retrieval, reasoning, and recursive cropping to progressively narrow the search space, transforming citation into a dynamic, evidence-driven reasoning process rather than a static step. Furthermore, we incorporate a Citation-enhanced Reward mechanism that integrates both process-level and outcome-level feedback within a reinforcement learning paradigm to jointly optimize answer accuracy and source traceability. Extensive experiments on benchmarks such as Wiki-VISA, FinRAGBench-V, and MMLongBench-Doc demonstrate that MCite-RL effectively achieves the joint optimization of citation precision and answer quality.
• 中文摘要 多模态检索增强生成（RAG）配合视觉引用对于确保MLLM的可追溯性和可验证性至关重要。然而，当前基于RAG和SFT的方法难以实现稳健的跨模态推理，导致视觉引用不精确，或引用与生成答案之间出现解耦。为解决这些限制，我们提出了MCite-RL，一个基于引用增强的代理强化学习框架，旨在实现可靠的多模态RAG。MCite-RL引入了适用于可视化引用的代理细化模块，采用迭代检索、推理和递归裁剪，逐步缩小搜索空间，将引用转化为动态、证据驱动的推理过程，而非静态步骤。此外，我们采用了引用增强的奖励机制，将过程层级和结果层级反馈整合在强化学习范式中，共同优化答案准确性和来源可追溯性。在Wiki-VISA、FinRAGBench-V和MMLongBench-Doc等基准测试上的大量实验表明，MCite-RL有效实现了引用精度与答案质量的联合优化。

Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning

提示、批评与教师：视觉语言数学推理中稀疏奖励强化学习的先例注入

• Authors: Qiqian Fu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.21811
• Pdf link: https://arxiv.org/pdf/2608.21811
• Abstract Reinforcement learning for vision-language math reasoning starves under sparse reward: on a pool of 20,830 visual-math problems where Qwen2-VL-2B answers 3.6% of rollouts correctly, 85-97% of GRPO rollout groups are entirely wrong and contribute zero gradient. We train eleven methods under identical conditions in this regime, each injecting a different prior: text (reference-solution hints), distribution (on-policy distillation from a 7B teacher), and value (a value-pretrained critic with an MSE or HL-Gauss categorical loss). A prior helps exactly when it is delivered: the six arms whose prior effectively reaches the policy separate with no overlap from the remaining five -- the no-prior baseline and four arms whose prior is teacher-capped, gated away, or lost to a mis-parameterized critic -- both on the pooled in-domain metric and on cross-domain transfer (DynaMath). The central finding, however, concerns evaluation: one slice of the in-domain pool -- long used as this project's general-distribution check -- anti-correlates with genuine cross-domain transfer (Spearman rho = -0.74, n = 11 arms, permutation p = 0.011), while the hardest in-domain slice predicts it closely (rho = +0.89, p < 0.001). We attribute the inversion to a near-chance multiple-choice subset that rewards models for not having changed; read through it, the best cross-domain method looked mediocre and the worst looked like the champion. Among the methods, hint-guided exploration -- not UFT's auxiliary loss -- drives hint gains, and replacing the critic's MSE loss with HL-Gauss cross-entropy is worth +14.4 points in-domain. All accuracies are blind-judged, with paired exact tests.
• 中文摘要 视觉语言数学推理的强化学习在奖励稀疏下难以实现：在20,830个视觉数学题池中，Qwen2-VL-2B正确回答了3.6%的推广，85-97%的GRPO推广组完全错误且贡献零梯度。我们在相同条件下训练了11种方法，每种方法注入不同的先验：文本（参考解提示）、分布（7B教师的策略提炼）和价值（价值预训练批评者，具有MSE或HL-Gauss类别损失）。先验在交付时恰好有用：六个先验有效进入策略且与其余五个臂无重叠——无先验基线和四个先验被教师限制、封锁或丢失给参数错误批评者的臂——无论是在合并的领域指标上还是跨域转移（DynaMath）。然而，核心发现涉及评估：领域池中的一部分切片——长期用作本项目的一般分布检验——与真正的跨域转移反相关（Spearman rho = -0.74，n = 11臂，置换p = 0.011），而最难的领域内切片预测得很接近（rho = +0.89，p < 0.001）。我们将反转归因于一个近概率的多项选择子集，该子集奖励未变模型;通读后，最佳跨域方法表现平平，最差则像冠军。在这些方法中，提示引导探索——而非UFT的辅助损失——推动了提示收益，用HL-Gauss交叉熵替代批评者的MSE损失在领域内价值+14.4分。所有准确度均采用盲判，采用配对精确检验。

Beyond Success and Failure: Length-Aware Contrastive Learning for GUI Agents

超越成功与失败：面向图形界面代理的长度感知对比学习

• Authors: Chengyang Gu, Le Zhang, Jingbo Zhou, Yize Chen, Yu Shi, Siqi Bao, Zheng-Fan Wu, Hua Wu, Hui Xiong
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21830
• Pdf link: https://arxiv.org/pdf/2608.21830
• Abstract Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) have shown strong potential for automating tasks across diverse digital environments, where reinforcement learning (RL) has become a dominant training paradigm. However, widely used methods such as Group Relative Policy Optimization (GRPO) suffer from reward-gradient misalignment, leading to inefficient and unstable optimization. Recent work addresses this issue by reformulating RL with verifiable rewards (RLVR) as contrastive or classification-based objectives, which improve stability by eliminating problematic gradient behaviors. Despite this progress, existing contrastive RLVR methods rely primarily on outcome-level supervision and fail to capture fine-grained differences in trajectory quality within the same outcome category. In this paper, we propose Length-Aware Contrastive Learning for GUI Agents (LACL-GUI), a contrastive RLVR framework that incorporates trajectory-level quality signals into policy optimization. LACL-GUI introduces structured preferences within both successful and failed trajectories, encouraging concise successful executions and differentiating failure quality based on divergence from successful trajectories, while preserving optimization stability. Experiments on GUI agent benchmarks show that LACL-GUI provides more effective learning signals and consistently improves agent performance over prior methods, highlighting the value of trajectory-level supervision in contrastive RLVR.
• 中文摘要 由多模态大型语言模型（MLLM）驱动的图形用户界面（GUI）代理在多样化数字环境中展现出强大自动化任务的潜力，强化学习（RL）已成为主流训练范式。然而，广泛使用的方法如群相对策略优化（GRPO）存在奖励梯度错位问题，导致优化效率低下且不稳定。近期研究通过将可验证奖励（RLVR）重新表述为对比或分类目标来解决这一问题，通过消除问题梯度行为提升稳定性。尽管取得了这些进展，现有的对比RLVR方法主要依赖结果层级监督，未能捕捉同一结果类别内轨迹质量的细致差异。本文提出了GUI代理的长度感知对比学习（LACL-GUI），这是一种对比RLVR框架，将轨迹级质量信号纳入策略优化。LACL-GUI在成功和失败轨迹中引入结构化偏好，鼓励简洁成功执行，并根据与成功轨迹的偏差区分失败质量，同时保持优化稳定性。GUI代理基准测试实验显示，LACL-GUI提供更有效的学习信号，且持续提升代理性能，突出对比RLVR轨迹级监督的价值。

LLMs are Few-Shot Decision-Makers: Generalized Context-Aware Microgrid Frequency Control through Prompt Decision Transformer

LLMs是少数方决策者：通过即时决策变换器实现的通用上下文感知微电网频率控制

• Authors: Xu Yang, Chenhui Lin, Haotian Liu, Kaihang Deng, Yunhe Li, Wenchuan Wu
• Subjects: Subjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21858
• Pdf link: https://arxiv.org/pdf/2608.21858
• Abstract The rapid evolution of energy structures has positioned microgrids as pivotal components of next-generation power systems, offering enhanced resilience and renewable energy integration. However, the inherent low inertia, complex dynamics, and poor model conditions of microgrids necessitate advanced data-driven frequency control strategies. Although reinforcement learning (RL) has demonstrated certain potential and advantages, existing RL methods often struggle with generalization across diverse microgrid configurations and lack adaptability to unseen environments, particularly when explicit system parameters are unavailable. To address these challenges, in this paper, we introduce a novel prompt decision transformer (Prompt-DT) architecture for microgrid frequency control. Unlike traditional approaches that rely on hard-to-obtain environmental characteristic parameters, the proposed method leverages few-shot expert historical trajectories as prompts to guide autonomous perception and adaptive decision-making. In addition, we propose a context-aware training and execution mechanism utilizing self-supervised contrastive learning to enhance environment recognition and prompt utilization efficiency. In addition, a physics-informed prompt design technique that filters prompts based on cumulative reward and frequency volatility is proposed, ensuring high-quality physical guidance during online execution. Finally, to ensure generalization in unseen environments with limited data, we develop a lightweight finetuning approach that achieves performance comparable to full-parameter finetuning with minimal adjustments.
• 中文摘要 能源结构的快速演变使微电网成为下一代电力系统的关键组成部分，增强了韧性和可再生能源整合。然而，微电网固有的低惯性、复杂的动态和糟糕的模型条件，要求采用先进的数据驱动频率控制策略。尽管强化学习（RL）展现出一定潜力和优势，现有强化学习方法在不同微电网配置的泛化上常常存在困难，且缺乏对未知环境的适应性，尤其是在系统参数不明确时。为应对这些挑战，本文介绍了一种新颖的提示决策变换器（Prompt-DT）架构用于微电网频率控制。与依赖难以获得的环境特征参数的传统方法不同，本方法利用少数样本专家历史轨迹作为引导自主感知和自适应决策的提示。此外，我们提出了一种基于上下文的训练和执行机制，利用自我监督对比学习，以增强环境识别和提示利用效率。此外，提出了一种基于物理的提示设计技术，基于累计奖励和频率波动性过滤提示，确保在线执行时的高质量物理指导。最后，为了确保在数据有限的未见环境中实现泛化，我们开发了一种轻量级微调方法，实现与全参数微调相当的性能，且调整极少。

HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoning

HiDiffTIR：多回合工具集成推理的层级难度感知策略优化

• Authors: Yucan Guo, Xiaohan Wang, Miao Su, Saiping Guan, Zhongni Hou, Jiajun Chai, Wei Lin, Guojun Yin, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21863
• Pdf link: https://arxiv.org/pdf/2608.21863
• Abstract Tool-Integrated Reasoning (TIR) is a fundamental capability for LLM agents to solve complex tasks by interacting with external tools iteratively. Reinforcement Learning (RL) has become the dominant paradigm for enabling this capability. However, existing approaches typically assign uniform trajectory-level advantages and treat all correct tool calls equally, ignoring the varying difficulty and learning value across trajectories and reasoning steps. This can lead to imprecise learning signals that do not adequately distinguish between trivial and challenging tool-use patterns. To address this limitation, we propose HiDiffTIR, a Hierarchical Difficulty-aware policy optimization framework for multi-turn TIR. HiDiffTIR performs difficulty-aware credit assignment at both trajectory and turn levels, enabling the policy to focus on more informative trajectories and harder reasoning steps. Notably, this fine-grained optimization is achieved without additional supervision, relying solely on group-level statistics derived from standard RL rollouts. Extensive experiments on three tool-using benchmarks demonstrate that HiDiffTIR consistently improves multi-turn TIR performance and tool invocation accuracy over strong RL baselines, highlighting the necessity of difficulty-aware credit assignment for effective policy optimization in tool-integrated LLM agents.
• 中文摘要 工具集成推理（TIR）是LLM代理通过迭代与外部工具交互来解决复杂任务的基本能力。强化学习（RL）已成为实现该能力的主导范式。然而，现有方法通常赋予统一的轨迹层优势，并平等对待所有正确的工具调用，忽略轨迹和推理步骤中难度和学习价值的差异。这可能导致学习信号不精确，无法充分区分琐碎与挑战的工具使用模式。为解决这一限制，我们提出了HiDiffTIR，一种针对多回合TIR的层级难度感知策略优化框架。HiDiffTIR在轨迹和转向层级执行难度感知的信用分配，使策略能够聚焦于更具信息量的轨迹和更难的推理步骤。值得注意的是，这种细粒度优化无需额外监督，完全依赖标准强化学习推广中得出的组级统计数据。对三个工具基准测试的广泛实验表明，HiDiffTIR在多回合TIR性能和工具调用准确性方面，优于强强强的强强化学习基线，凸显了难度感知的信用分配对于工具集成LLM代理有效策略优化的必要性。

BioMed-Agent-RL: A Meta Learning, All You Need for Biomedical Applications

BioMed-Agent-RL：一种元学习，生物医学应用所需的一切

• Authors: Md Asaduzzaman Jabin, Zihao Wu, Tianming Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.21864
• Pdf link: https://arxiv.org/pdf/2608.21864
• Abstract The current progress of Clinical Vision Large Language Models (C-VLLMs) has substantially improved digital diagnostics, still these frameworks often endure lesion noises, modality misalignment, hallucination, and missed contextual grounding in complex clinical cases. Moreover, prevailing agent systems usually depend on static and non-adaptable pipelines and lack the versatility necessary for complex medical reasoning. To resolve these difficulties, we present BioMed-Agent-RL, a unified medical agent that incorporates adaptive orchestration, policy, and reward-based reinforcement learning (RL) models for biomedical applications. To ensure reliability, it invokes clinical context-aware preference optimization (CPO), direct preference optimization (DPO), and group relative policy optimization (GRPO) with dynamic entropy regulation. This pipeline utilizes a multimodal meta-learning approach that operates as a field-specific expert and human judgment synthesizer. The agent adaptively utilizes a set of model-level expertise, such as clinical grounding and reasoner, lesion segmenter, and field-specific synthesizer, across various clinical modalities (e.g., X-ray) by utilizing an iterative and adaptive RL approach. The agent learns to seriously synthesize misleading, conflicting vision cues and trust in inherent reasoning, while specialist advice is faulty. An intensive ablation study is conducted across multiple benchmarks, and the agent significantly outperforms existing state of the art models, such as GPT-5, attaining up to ~73% accuracy (gain of ~5%) over contemporary baselines. As a result, the framework suggests a new standard for building factual, reliable, robust, and expert-like intelligent agent systems for independent clinical reasoning.
• 中文摘要 临床视觉大型语言模型（C-VLLM）的进展显著提升了数字诊断，但这些框架在复杂临床病例中常常存在病灶噪声、模态错位、幻觉和错失上下文。此外，现有代理系统通常依赖静态且不可适应的管道，缺乏复杂医学推理所需的灵活性。为解决这些困难，我们提出了BioMed-Agent-RL，一种整合自适应编排、策略和基于奖励的强化学习（RL）模型的统一医疗代理，适用于生物医学应用。为确保可靠性，它引入了临床情境感知偏好优化（CPO）、直接偏好优化（DPO）和群相对策略优化（GRPO），并结合动态熵调控。该流程采用多模态元学习方法，作为领域特定专家和人类判断综合器。该代理通过迭代和自适应的强化学习方法，自适应地利用一套模型级专业知识，如临床基础分析与推理器、病灶分割器和领域特定综合器，涵盖多种临床模式（如X光）。代理学会认真综合误导性、相互冲突的视觉线索，并信任内在推理，而专业建议则存在缺陷。进行了跨多个基准测试的深入消融研究，代理显著优于现有最先进模型，如GPT-5，较当前基线实现高达~73%的准确率（提升~5%）。因此，该框架提出了构建事实、可靠、稳健且类似专家的智能代理系统以实现独立临床推理的新标准。

The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning

追逐是课程，捕获是功劳的锚点：追踪-回避自我游戏，为零数据大型语言模型推理

• Authors: Jing Yu, Shengchao Chen, Yiyun Tan
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.21871
• Pdf link: https://arxiv.org/pdf/2608.21871
• Abstract Reinforcement learning with verifiable rewards has become the dominant recipe for improving large language model reasoning, yet it presumes large human-curated task collections. Zero-data self-play removes this dependency, but existing methods vet learnability only by probing candidates and rejecting post hoc, never learning where along an environment's difficulty axis to place a task, and credit the solver with sparse terminal rewards alone. We recast zero-data self-play as a pursuit-evasion game: in LURE, an LLM evader positions tasks along each environment's difficulty axis to stay one step ahead of a planner-executor pursuer that hunts it down through verifiable interaction. The evader is trained on a capture-frontier reward that peaks when the solver captures it on exactly half of its rollouts, turning barely catchable into a learned positioning strategy rather than a hand-tuned rejection band. The pursuer earns capture-anchored dense process credit, in which monotone verifier progress is group-normalized jointly with the terminal capture under a round-anchored KL that keeps the co-evolution stable. Across three verifiable reasoning environments and three backbone families, LURE outperforms advanced baselines under unified/specialist settings, while the unified model attains stronger aggregate OOD zero-shot accuracy than all trained baselines across nine held-out benchmarks from three task families.
• 中文摘要 带有可验证奖励的强化学习已成为提升大型语言模型推理的主导配方，但它假设了大量人类策划的任务集合。零数据自玩消除了这种依赖，但现有方法仅通过探查候选对象和事后拒绝来审查可学习性，从未知道在环境难度轴上应将任务放在哪里，且仅凭稀疏的终端奖励给解题者。我们将零数据自玩重新定义为追踪-规避游戏：在《诱导》中，LLM逃避者将任务按每个环境的难度轴排列，以领先通过可验证互动追踪的计划执行者一步。逃避者训练于捕获前沿奖励，当解算器在其部署的恰好一半捕获时达到顶峰，将几乎可捕捉的目标转变为学习的定位策略，而非手工调校的拒绝带。追诉方获得捕获锚定的稠密过程积分，其中单调验证器的进展与终端捕获在圆锚定KL下共同进行组归一化，保持共演稳定。在三种可验证推理环境和三大骨干家族中，LURE在统一/专业环境中优于高级基线，而统一模型在九个任务家族中九个基准测试中，整体OOD零样本准确率优于所有训练基线。

From Solver Feedback to Faithful Plans: Multi-Role Reinforcement Learning for Symbolic Planning

从求解器反馈到忠实计划：符号规划中的多角色强化学习

• Authors: Chenghao Zhang, Yikai Mao, Shanqi Liu, Haoyu Gao, SaiSai Hu, Dan Roth
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21897
• Pdf link: https://arxiv.org/pdf/2608.21897
• Abstract Reliable planning requires converting natural-language instructions into executable symbolic specifications, yet large language models remain brittle without costly PDDL annotations and may exploit solver success in semantically unfaithful ways. We study how to learn faithful natural-language-to-PDDL formalization using only solver feedback, without human-written demonstrations. We propose a solvergrounded multi-role reinforcement learning framework where a single language model acts as an Actor, Judge, and Editor for generation, verification, and repair. The Actor proposes PDDL specifications, the Judge provides a solver-calibrated quality signal, and the Editor performs bounded diagnostic-conditioned refinement. On PlanBench, our method improves average success from 35.5% for LLM+P to 70.8%, achieves 66.3% faithful success, and reduces semantic drift to 6.4%. These results show that organizing solver feedback into generation, verification, and repair roles enables more scalable and faithful annotation-free symbolic planning
• 中文摘要 可靠的规划需要将自然语言指令转换为可执行的符号规范，但大型语言模型若无昂贵的PDDL注释，仍脆弱，可能以语义不忠的方式利用求解器成功。我们研究如何仅凭求解器反馈学习忠实自然语言到PDDL的形式化，无需人工演示。我们提出一个基于求解器的多角色强化学习框架，单一语言模型作为生成、验证和修复的演员、裁判和编辑器。演员提出PDDL规范，裁判提供求解器校准的质量信号，编辑执行有界诊断条件细化。在PlanBench上，我们的方法将LLM+P的平均成功率从35.5%提升至70.8%，忠实成功率达到66.3%，语义漂移降至6.4%。这些结果表明，将求解器反馈组织为生成、验证和修复角色，能够实现更具可扩展性和忠实的无注释符号规划

CIDER: Continual Interactive Distillation for Embodied Reinforcement Learning

CIDER：具身强化学习的持续互动蒸馏

• Authors: Houlin Li, Minghui Xu, Guo Xu, Xuan Du, Xiaohan Yan, Chun Wang, Yuxiang Yan, Shukai Yang, Yongcheng Liu, Wei Shan, Maoqing Yao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.21899
• Pdf link: https://arxiv.org/pdf/2608.21899
• Abstract Human-in-the-loop real-world reinforcement learning enables rapid acquisition of effective robotic manipulation policies for individual tasks, often within tens of minutes. Yet it remains unclear how to extend this paradigm to continual learning, where a single policy must acquire new skills without losing previously learned behaviors. Existing real-world continual learning methods do not explicitly constrain prior behaviors, leading to severe catastrophic forgetting. We introduce Continual Interactive Distillation for Embodied Reinforcement Learning (CIDER), a continual reinforcement learning framework that freezes the accumulated historical policy as a teacher before learning each new task and interleaves task learning with distillation-based retention. We further introduce gradient routing to separate the gradients used for acquiring new tasks from those used for preserving prior behaviors. We evaluate our method with a single shared actor on six real-world household and industrial manipulation tasks. Interactive Distillation maintains high measured success on previously learned tasks across our six-task real-robot sequence while acquiring each new task in 10 to 20 minutes, whereas every baseline forgets at least one previous task. Additional ablations reveal the key design choices that govern the tradeoff between stability and plasticity in real-world continual reinforcement learning.
• 中文摘要 人在环中现实强化学习使得为单个任务快速习得有效的机器人操作策略，通常在数十分钟内完成。然而，如何将这一范式推广到持续学习仍不明确，即单一策略必须在不丢失既有行为的情况下获得新技能。现有的现实世界持续学习方法未明确限制先前行为，导致严重的灾难性遗忘。我们引入了具身强化学习的持续互动蒸馏（CIDER），这是一种持续强化学习框架，在学习每个新任务前冻结作为教师积累的历史策略，并将任务学习与基于蒸馏的保留交错。我们还进一步引入梯度路由，将用于获取新任务的梯度与保留先前行为的梯度区分开来。我们用单一共享参与者评估了六个现实家庭和工业操控任务的方法。交互式蒸馏在我们六个任务与机器人序列中，保持了先前学习任务的高成功率，同时每个新任务在10到20分钟内完成，而每个基线任务至少会忘记一个之前的任务。额外的消融揭示了控制现实世界持续强化学习中稳定性与可塑性权衡的关键设计选择。

Beyond Fixed Directions: Adaptive Representation Analysis of Reasoning and Memorization in LLMs

超越固定方向：大语言模型中推理与记忆的自适应表征分析

• Authors: Shaheen Nabi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.21919
• Pdf link: https://arxiv.org/pdf/2608.21919
• Abstract Recent work has proposed that reasoning and memorization in language models can be characterized by a single representation direction, including methods that keep this direction fixed during reinforcement learning. We test two assumptions behind this view. First, are reasoning-oriented and factual-recall task groups approximately single-direction separable? Second, does the resulting geometry remain stable after GRPO? Using Qwen3-0.6B and a controlled 400-example dataset, we find that a one-dimensional projection can match a full 1024-dimensional linear probe with AUROC = 1.00 on the studied task groups. However, after GRPO, the corresponding direction is substantially reorganized: mean-direction cosine averages 0.453, probe-direction cosine 0.445, while direct representation drift reaches 0.511 at the final layer. Probe AUROC nevertheless remains 1.00. The evidence therefore supports single-direction decodability for the studied task groups but challenges fixed-direction stability: the information persists while its geometric realization changes.
• 中文摘要 最新研究提出，语言模型中的推理和记忆可以通过单一的表示方向来表征，包括在强化学习过程中保持该方向固定的方法。我们测试了这一观点背后的两个假设。首先，推理导向和事实回忆任务组是否近似单方向可分离？其次，GRPO后所得几何是否保持稳定？利用Qwen3-0.6B和一个受控的400个样本数据集，我们发现一维投影可以匹配一个完整的1024维线性探针，且AUROC=1.00在研究的任务组上。然而，经过GRPO后，相应方向进行了大幅重组：平均方向余弦平均为0.453，探针方向余弦为0.445，而直接表示漂移在最后一层达到0.511。探针AUROC仍然保持1.00。因此，证据支持研究任务组的单向可解码性，但对固定方向稳定性提出了挑战：信息在几何实现发生变化时依然存在。

ESCRAG-R1: Retrieval-Augmented Reinforcement Learning for Emotional Support Conversation

ESCRAG-R1：情感支持对话中的检索增强强化学习

• Authors: Weichu Liu, Yuxuan Hu, Yirong Sun, Ningning Mao, Ziyun Zhang, Jian Chen, Mingyang Xu, Qishan Zhong, Chengming Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21925
• Pdf link: https://arxiv.org/pdf/2608.21925
• Abstract Emotional Support Conversation (ESC) systems aim to provide holistic support by balancing professional therapeutic competence with natural empathy. However, existing methods struggle to simultaneously achieve structured, stage-aware reasoning and seamless empathy-expertise alignment, often resulting in an artificial splicing of clinical strategies and generic reassurance. To overcome these limitations, we propose ESCRAG-R1, a unified framework that integrates retrieval-based psychological guidance into Group Relative Policy Optimization (GRPO). By incorporating retrieval into the reinforcement learning loop, ESCRAG-R1 transforms external knowledge into a robust learning signal that stimulates explicit internal reasoning prior to generation and fundamentally reshapes the model's internal policy. To provide the reliable supervision required for this optimization, we construct ESC-Preference, a high-quality dataset based on a Client--Counselor--Judge evaluation framework that delivers precise, empathy-aware reward signals. Extensive experiments demonstrate that ESCRAG-R1 significantly outperforms existing baselines by mitigating superficial splicing and realizing a natural integration of professional guidance and empathetic expression. Code and datasets are released at this https URL.
• 中文摘要 情绪支持对话（ESC）系统旨在通过平衡专业治疗能力与自然共情，提供整体支持。然而，现有方法难以同时实现结构化、阶段感知的推理和无缝的同理心-专业知识对齐，常导致临床策略和通用安抚的人工拼接。为克服这些局限性，我们提出了ESCRAG-R1，一个统一框架，将基于检索的心理指导整合进群体相对政策优化（GRPO）。通过将检索融入强化学习循环，ESCRAG-R1将外部知识转化为强健的学习信号，在生成前激发显性内在推理，并从根本上重塑模型的内部策略。为提供优化所需的可靠监督，我们构建了ESC-Preference数据集，这是一个基于客户-咨询师-法官评估框架的高质量数据集，能够提供精准且具同理心意识的奖励信号。大量实验表明，ESCRAG-R1显著优于现有基线，减少了表面拼接，实现了专业指导与同理心表达的自然整合。代码和数据集发布于此 https URL。

EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning

EDGE：引导探索代理强化学习的体验提炼

• Authors: Can Xie, Yuyi Zhou, Wen Yang, Ziyi zhang, Siyao Song, Yingzhuo Deng, Shuo Ren, Jiajun Zhang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.21946
• Pdf link: https://arxiv.org/pdf/2608.21946
• Abstract Reinforcement learning with outcome-based objectives such as GRPO enables LLM-based agents to solve complex, long-horizon tasks, yet the reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update. Existing experience-augmented approaches retrieve historical guidance at inference time, but they apply experiences without accounting for the policy's evolving capability and create persistent dependencies on external retrieval. We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds and progressively internalizes their benefits into the parametric policy. Concretely, EDGE partitions each rollout group into experience-conditioned and experience-free trajectories to estimate and admit only positive marginal gains without extra sampling, then distills the induced behavior into the base policy via a reverse-KL objective on its own empirical support. A co-evolutionary experience bank further synthesizes guidance from emerging failure modes and prunes obsolete entries as the policy evolves. On ALFWorld and WebShop, EDGE improves over GRPO by 8.3 and 12.5 success-rate points at the 7B scale and retains 96.0% of its scaffolded performance when external experiences are removed at inference time. The code is available at this https URL.
• 中文摘要 基于结果的强化学习如GRPO使基于LLM的智能体能够解决复杂的长期任务，但交互轨迹中嵌入的可重复使用探索模式在一次策略更新后基本被丢弃。现有的经验增强方法在推理时获取历史指导，但它们应用经验时未考虑策略不断演变的能力，并造成对外部检索的持续依赖。我们提出了EDGE（引导探索体验提炼）框架，将检索到的经验视为临时训练时间支架，并逐步将其收益内化到参数化策略中。具体来说，EDGE将每个推广组划分为体验条件和无体验轨迹，估计并只承认正边际收益而不进行额外抽样，然后通过逆KL目标基于自身实证支持将诱发行为提炼进基础策略。共进化经验库进一步综合了新兴失败模式的指导，并随着策略演进修剪过时条目。在ALFWorld和WebShop上，EDGE在7B尺度上比GRPO提升8.3点和12.5个成功率，且在推断时移除外部经验后，仍保持96.0%的支架性能。代码可在此 https 网址获取。

ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents

ToSCA：利用分层强化学习对会话代理的时间和战略抽象

• Authors: Xiaoyu Wang, Qingqing Gu, Yue Zhao, Teng Chen, Yuqi Cao, Xiaokai Chen, Hongyan Li, Luo Ji
• Subjects: Subjects: Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.21969
• Pdf link: https://arxiv.org/pdf/2608.21969
• Abstract Humans have multiple levels of temporal abstractions on daily interaction and thinking, such as concept perception and strategic planning. Inspired by this nature, we propose a two-level hierarchical reinforcement learning (RL) framework for conversational agents, bridging the gap between previous token-level or utterance-level RL methods. Developed on a two-level MDP, the token-level response decoding is conditioned on the utterance-level action, the explicit textual strategies. Based on theoretical derivation and efficiency consideration, we use DQN to solve the high-level critic and PPO to solve the low-level actor-critic. To further alleviate the reward sparsity and facilitate the convergence, we also design the dual-granularity reward mechanism, in which the utterance-level satisfaction score is integrated with token-level intrinsic motivation and K-L penalty. Experiments on both daily and emotional support conversations show that our method outperforms versatile baselines in strategy determination and response quality. Our implementation is available at this https URL.
• 中文摘要 人类在日常互动和思考中拥有多层次的时间抽象，如概念感知和战略规划。受此启发，我们提出了一个两级层级强化学习（RL）框架，用于对话代理，弥合了以往令牌级和话语级强化学习方法之间的差距。基于两级MDP开发，令牌级反应解码基于话语层动作、显式文本策略。基于理论推导和效率考虑，我们用DQN解决高层批评者，PPO解决低层行为者-批评者。为进一步缓解奖励稀疏并促进收敛，我们还设计了双粒度奖励机制，其中话语级满意度与令牌级内在动机和K-L惩罚整合。日常和情感支持对话的实验表明，我们的方法在策略决策和响应质量方面优于多功能基线。我们的实现可在此 https URL 访问。

DELTA: Deformable Elevation-Based Local Terrain Attention Encoder for Sparse-Terrain Quadrupedal Locomotion

DELTA：稀疏地形四足行走的可变形基于高程的局部地形注意编码器

• Authors: Sanghyun Park, Moonkyu Jung, Jemin Hwangbo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.22033
• Pdf link: https://arxiv.org/pdf/2608.22033
• Abstract Stable quadrupedal locomotion on sparse terrain requires selecting state-relevant terrain evidence for precise foot placement. Model-based foothold planners provide precise foothold selection but rely heavily on explicit model assumptions. Recent attention-based map encoding (AME) studies show that end-to-end reinforcement learning (RL) can learn implicit foothold guidance. However, the computational cost of dense AME encoding grows with map resolution, limiting its scalability to fine-grained sparse terrain. We propose DELTA, a Deformable Elevation-Based Local Terrain Attention encoder. DELTA predicts state-conditioned sampling locations, forms terrain evidence tokens from adaptive local elevation patches, and attends only to a fixed-size token set. With fixed sampling and patch settings, DELTA's encoder cost is independent of map resolution. Experiments show that DELTA achieves final traversal performance comparable to AME at the standard resolution while improving learning efficiency. This fixed encoder cost enables the use of higher-resolution terrain maps, improving traversal on fine-grained sparse terrain. DELTA also demonstrates strong generalization to unseen mixed evaluation courses composed of continuous and discrete terrain elements. Beyond simulation, DELTA demonstrates successful sim-to-real transfer on RAIBO2. Analysis of the learned sampling offsets and attention weights shows that DELTA samples steppable regions and attends to terrain evidence relevant to future touchdowns without foothold labels or attention supervision.
• 中文摘要 在稀疏地形上实现稳定的四足行走需要选择与状态相关的地形证据以实现精确的脚步位置。基于模型的脚点规划器提供精确的脚步选择，但高度依赖显式模型假设。近期基于注意力的地图编码（AME）研究表明，端到端强化学习（RL）可以学习隐式的足迹引导。然而，密集的AME编码的计算成本随着地图分辨率增加而增加，限制了其在细粒度稀疏地形中的可扩展性。我们提出了DELTA，一种可变形的基于高程的局部地形注意力编码器。DELTA预测状态条件采样位置，从自适应的局部高程补丁生成地形证据标记，并仅关注固定大小的标记集。在固定采样和补丁设置下，DELTA编码器的成本与地图分辨率无关。实验表明，DELTA在标准分辨率下实现与AME相当的最终遍历性能，同时提升学习效率。这种固定编码成本使得使用更高分辨率的地形图成为可能，从而提升在细粒度稀疏地形上的横移能力。DELTA还展示了对由连续和离散地形元素组成的未见混合评估路径的强推广能力。除了模拟外，DELTA还展示了在RAIBO2上成功模拟到现实的转移。对学习的采样偏移和注意力权重的分析显示，DELTA采样可逐步区域，关注与未来着陆相关的地形证据，无需脚点标签或注意力监督。

Decoupled Physical Modeling and Execution for Physics Reasoning

物理推理中的解耦物理建模与执行

• Authors: Ye Zhang, Xuehang Guo, Rui Pan, Pengfei Yu, Denghui Zhang, Manling Li, Qingyun Wang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.22126
• Pdf link: https://arxiv.org/pdf/2608.22126
• Abstract Physics reasoning requires constructing a consistent model of the underlying physical system rather than relying solely on symbolic or formula-based manipulation. Although large language models have shown strong ability in solving math and coding problems, they still struggle with physics problems, as these problems entangle the physical modeling process with mathematical calculations. Humans approach physics by first building a representation of the system before performing calculations. Inspired by this, we introduce a unified framework that distills intermediate representations that explicitly encode the physical modeling process and adopt a two-stage post-training strategy, where supervised fine-tuning establishes structured modeling, and reinforcement learning with rubric-based feedback improves the quality of the modeling process. Experiments on multiple multimodal physics benchmarks show that our approach leads to consistent improvements in reasoning performance across different models and datasets. On PhysReason, PhyX and SeePhys benchmarks, physical modeling output performs GRPO by an average ~3%, showing that explicit physical modeling is an efficient strategy of improving physics reasoning for small LLMs.
• 中文摘要 物理推理需要构建一个一致的底层物理系统模型，而非仅依赖符号或基于公式的操作。尽管大型语言模型在解决数学和编码问题方面表现出强大能力，但在物理问题上仍面临困难，因为这些问题将物理建模过程与数学计算纠缠在一起。人类在进行计算前，首先构建系统的表示。受此启发，我们引入了一个统一框架，提取出明确编码物理建模过程的中间表示，并采用两阶段训练后策略，其中监督微调建立结构化建模，基于评分标准的强化学习提升建模质量。多模态物理基准测试的实验表明，我们的方法在不同模型和数据集间的推理性能持续提升。在PhysReason、PhyX和SeePhys基准测试中，物理建模输出的平均GRPO表现为约3%，表明显式物理建模是提升小型大型语言模型物理推理的高效策略。

MCP-Universe RL: A Framework for Training MCP Tool-Use Agents via Reinforcement Learning

MCP-Universe RL：通过强化学习训练MCP工具使用代理的框架

• Authors: Ziyang Luo, Yan Yang, Xiangru Jian, Ziji Shi, Xiaoqiang Lin, Jun Hao Liew, Silvio Savarese, Junnan Li
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.22167
• Pdf link: https://arxiv.org/pdf/2608.22167
• Abstract Reinforcement learning (RL) has become an effective way to improve the tool-use ability of large language models (LLMs), but most existing RL frameworks stop at the policy update. For every new domain, the user is left with two hard systems problems: standing up an isolated environment for each of hundreds of concurrent trajectories and connecting it to training, and scheduling the rollout so that the GPU stays busy across long, multi-turn episodes that spend much of their time stalled on slow tool calls. We present MCP-Universe RL (MCP-U RL), an open-source framework that takes over both. It uses the Model Context Protocol (MCP) as the interface to the environment, so any tool already exposed as an MCP server plugs into training with no RL-specific integration code. It builds the two missing layers once and reuses them across domains: an environment-orchestration layer that provisions, isolates, and recycles the MCP environments over a pluggable container backend, and a rollout-orchestration layer whose staged pipeline overlaps trajectories to keep the GPU busy while episodes wait on tools. A backend-agnostic training layer then applies the update through an existing RL backend, with veRL and slime integrations. With one configuration, changing only the task specification, we train software-engineering, deep-research, and general tool-use agents on gpt-oss-20b and improve task reward in all three.
• 中文摘要 强化学习（RL）已成为提升大型语言模型（LLM）工具使用能力的有效方法，但大多数现有的RL框架都停留在策略更新时。每当有新领域，用户面临两个难题：为数百条并发轨迹建立一个隔离环境并将其连接到训练，以及安排部署，使GPU在长时间、多回合的过程中保持忙碌，这些阶段大部分时间都被拖慢处理工具调用。我们介绍MCP-Universe RL（MCP-U RL），一个开源框架，接管了两者。它使用模型上下文协议（MCP）作为环境接口，任何已暴露为MCP服务器的工具都可以插入训练，无需RL专用集成代码。它一次性构建两层缺失的，并在多个域间重复使用：环境编排层通过可插拔的容器后端配置、隔离和回收MCP环境;以及一个分阶段的流水线重叠轨迹，使GPU保持忙碌，等待工具的集成。然后，一个后端无关的训练层通过现有的强化学习后端应用更新，集成veRL和slime。通过一个配置，仅修改任务规格，我们在gpt-oss-20b上训练软件工程、深度研究和通用工具使用代理，并提升三者的任务奖励。

MARL-Based Sequential RIS Auctions: A Physical-Layer Security Analysis

基于MARL的顺序RIS拍卖：物理层安全分析

• Authors: Yuanyu Zhang, Yu Zhang, Jialu He, Zhixin Huang, Shuangrui Zhao, Yulong Shen
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2608.22169
• Pdf link: https://arxiv.org/pdf/2608.22169
• Abstract Reconfigurable intelligent surfaces (RISs) hold great potential to enhance coverage, spectral efficiency, and communication security by intelligently configuring their reflecting elements. When owned by a neutral RIS operator, these elements can be offered as resources for which legitimate receivers and eavesdroppers compete. This paper investigates such competition and evaluates its impact on the physical-layer security performance of legitimate receivers. To model the competition, we develop a sequential RIS auction (SRA) framework, in which a bundle of RIS elements is auctioned in each round through a first-price sealed-bid mechanism, with each bidder submitting its bid based on the achievable rate gain and remaining budget. We then formulate the sequential bidding process as a Markov game by specifying its states, actions, rewards, and state transitions. To solve the game, we propose a multi-bidder deep deterministic policy gradient (MADDPG)-based multi-bidder reinforcement learning (MARL) approach under centralized training and decentralized execution (CTDE), enabling legitimate receivers and eavesdroppers to learn bidding strategies that maximize their long-term economic surplus. Numerical results show that, under the considered eavesdropper bidding strategies, the RL-based strategy enables legitimate receivers to achieve the highest secrecy rate per unit cost, outperforming random and fixed strategies and approaching the ideal physical-layer upper bound.
• 中文摘要 可重构智能曲面（RIS）通过智能配置其反射元件，具有提升覆盖率、频谱效率和通信安全性的巨大潜力。当这些元素由中立的RIS运营商拥有时，可以作为资源提供，供合法接收器和窃听者竞争。本文探讨了这种竞争，并评估其对合法接收器物理层安全性能的影响。为建模竞争，我们开发了一个顺序RIS拍卖（SRA）框架，每轮通过首价密封竞标机制拍卖一组RIS元素，每个竞标者根据可达成的报价涨幅和剩余预算提交竞标。随后，我们将顺序竞价过程构建为马尔可夫博弈，明确其状态、动作、奖励和状态转换。为解决此问题，我们提出了一种基于多竞标者深度确定性策略梯度（MADDPG）的多竞标者强化学习（MARL）方法，采用集中训练与去中心化执行（CTDE），使合法接收者和窃听者能够学习最大化其长期经济剩余的竞价策略。数值结果表明，在所考虑的窃听者竞价策略下，基于强化学习的策略使合法接收者能够实现最高的单位成本保密率，优于随机和固定策略，接近理想的物理层上界。

UR$^{2}$-MLLM: Uncertainty-aware Revisit Reasoning in Multimodal Large Language Models for Radiology Report Generation

UR$^{2}$-MLLM：多模态大型语言模型中不确定性感知的再访推理以生成放射科报告

• Authors: Yucheng Chen, Yang Yu, Jiazhou Zhou, Yufei Shi, Yongying Lan, Yichi Zhang, Liyi Li, Si Yong Yeo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.22217
• Pdf link: https://arxiv.org/pdf/2608.22217
• Abstract Radiologists generate diagnostic reports through iterative and selective revisiting of suspicious regions to refine their interpretations. Recent multimodal large language models (MLLMs) for radiology report generation (RRG) have shifted from text-only reasoning toward a ``Thinking-with-Images'' paradigm, incorporating visual evidence into the reasoning process. However, existing methods provide static visual evidence without a dynamic revisit mechanism during reasoning, neglecting how radiologists re-examine uncertain observations. To this end, we propose an Uncertainty-aware Revisit Reasoning MLLM (UR$^{2}$-MLLM) framework that dynamically revisits uncertain regions during reasoning for RRG. UR$^{2}$-MLLM is first equipped with uncertainty perception by training on an uncertainty-aware dataset. We then construct a multimodal reasoning trajectory dataset together with a detect-and-copy mechanism, which guides when and where to revisit. Finally, a visual grounding reward refines this behavior through reinforcement learning, aligning the revisited regions with corresponding anatomical structures. Experiments on MIMIC-CXR and IU-Xray show that UR$^{2}$-MLLM achieves state-of-the-art performance, highlighting the value of uncertainty-aware visual revisit reasoning for reliable and clinically aligned report generation.
• 中文摘要 放射科医生通过迭代和选择性地重访可疑区域来生成诊断报告，以优化其解释。近年来用于生成放射报告（RRG）的多模态大型语言模型（MLLM）已从纯文本推理转向“用图像思考”范式，将视觉证据纳入推理过程。然而，现有方法在推理过程中提供静态视觉证据，却没有动态回访机制，忽视了放射科医生如何重新审视不确定的观察结果。为此，我们提出了一种不确定性感知的再访推理MLLM（UR$^{2}$-MLLM）框架，用于RRG推理时动态重访不确定区域。UR$^{2}$-MLLM首先通过在不确定性感知数据集上训练，获得不确定性感知。随后，我们构建了一个多模态推理轨迹数据集，并结合检测与复制机制，指导何时何地回访。最后，视觉接地奖励通过强化学习优化该行为，将重访区域与相应的解剖结构对齐。MIMIC-CXR和IU-Xray的实验显示UR$^{2}$-MLLM实现了最先进的性能，凸显了不确定性感知视觉回访推理在生成可靠且临床对齐报告中的价值。

Risk-Sensitive Reinforcement Learning with Smoothed Quantile Objectives

风险敏感强化学习与平滑分位数目标

• Authors: Mohammad Alipour-Vaezi, Huaiyang Zhong, Sajad Khodadadian
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2608.22227
• Pdf link: https://arxiv.org/pdf/2608.22227
• Abstract Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields, including healthcare, finance, etc. A popular approach to incorporate risk sensitivity is to optimize a specific quantile of the cumulative reward distribution. However, exact quantile objectives are non-smooth and can change abruptly under small perturbations of the return distribution, making them difficult to optimize reliably when the transition model must be learned from data. Motivated by this instability, we develop UCB-BQRL, a model-based optimistic learning algorithm that maintains confidence sets for the transition kernel and plans using a lower-buffered quantile criterion. The buffered criterion smooths the exact quantile objective by averaging nearby lower quantiles, thereby improving stability under transition-estimation error. To compute the buffered-quantile policy at each episode, we introduce EVI-BQ, an exact dynamic-programming procedure. We establish a high-probability regret bound for UCB-BQRL, which up to logarithmic factors scales as $\mathcal{O}(\mathrm{e}^{\tau/\rho_\tau}+H^2\sqrt{SAT})$, where $\rho_\tau$ is denoted as the root-level left-plateau threshold, which is a problem-dependent constant. Further, we establish an information-theoretic lower bound of $\Omega(H/\rho_\tau\sqrt{AT})$ for the regret of any algorithm dealing with a quantile objective function. Finally, we prove that the exact point-quantile evaluation and exact lower-buffered quantile evaluation are PP-hard under polynomial-time Turing reductions, even for a fixed policy in a two-state, one-action finite-horizon MDP.
• 中文摘要 强化学习（RL）近年来取得了巨大成功。然而，强化学习的经典基础并未考虑目标函数的风险敏感性，而这在医疗、金融等多个领域尤为关键。一种流行的风险敏感性整合方法是优化累积奖励分布的特定分位数。然而，精确的分位数目标是非平滑的，且在回报分布的微小扰动下可能突然变化，这使得在需要从数据中学习过渡模型时，目标难以可靠地优化。基于这种不稳定性，我们开发了UCB-BQRL，一种基于模型的乐观学习算法，利用缓冲较低的分位数准则维持转移核和计划的置信度集。缓冲判据通过平均附近的较低分位数来平滑精确的分位数目标，从而提升在过渡估计误差下的稳定性。为了计算每集的缓冲分位策略，我们引入了EVI-BQ，一种精确的动态规划过程。我们建立了UCB-BQRL的高概率遗憾上限，其对数级下可扩展为$\mathcal{O}（\mathrm{e}^{\tau/\rho_\tau}+H^2\sqrt{SAT}）$，其中$\rho_\tau$表示根级左平台阈值，是一个依赖问题的常数。此外，我们建立了信息理论下界$\Omega（H/\rho_\tau\sqrt{AT}）$，用于处理分位数目标函数的算法的遗憾值。最后，我们证明了精确点分位数评估和精确低缓冲分位数评估在多项式时间图灵归约下是PP难的，即使是在两态一作用有限视界MDP中的固定策略。

Learning from the Test: Self-Referential Differential Testing for Deep RL Agents

从测试中学习：深度强化学习代理的自指差分测试

• Authors: Junda He, Jieke Shi, Zhou Yang, Mingfei Cheng, David Lo
• Subjects: Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.22284
• Pdf link: https://arxiv.org/pdf/2608.22284
• Abstract Deep Reinforcement Learning (DRL) has achieved significant success in complex decision-making problems. As DRL systems are increasingly deployed in real-world applications, ensuring their quality and reliability is paramount. Current works primarily focus on detecting safety-critical failures, often neglecting policy optimality, which can lead to reduced efficiency, user distrust, and economic losses. This oversight, compounded by the inherent "testing oracle problem" for optimality, leaves a significant gap in comprehensively evaluating DRL systems. To address this gap, we propose Delta (Differential Testing for DRL Agents), a novel and comprehensive framework that automatically identifies both safety-critical and optimality bugs in DRL agents. Delta employs a two-phase approach: (1) Safety Testing, where the Agent Under Test (AUT) is evaluated for catastrophic failures while collecting data from its decision-making policy, and (2) Optimality Testing, where this collected data from the prior phase is used to train a challenger agent via Offline Reinforcement Learning. Differential testing is then performed by comparing the challenger agent against the AUT; instances where the challenger achieves higher cumulative rewards indicate optimality issues in the AUT. We demonstrate Delta's effectiveness across five environments. We investigate the effectiveness of three offline RL algorithms (BC, BCQ, and CQL) in generating challenger agents. Experimental results demonstrate that safety testing datasets are valuable for training competent DRL agents. Challenger agents trained with BCQ proved most effective for identifying optimality issues within the framework of Delta. Across the five environments, Delta uncovered an average of 2,518 optimality issues, outperforming the baseline methods by 50.2%.
• 中文摘要 深度强化学习（DRL）在复杂决策问题中取得了显著成功。随着DRL系统在实际应用中的应用日益普及，确保其质量和可靠性至关重要。当前工作主要聚焦于检测安全关键故障，常忽视策略最优性，这可能导致效率下降、用户不信任和经济损失。这一忽视，加上固有的“测试预言机问题”，使得全面评估DRL系统存在重大空白。为弥补这一空白，我们提出了Delta（DRL代理差分测试），这是一个新颖且全面的框架，能够自动识别DRL代理中的安全关键性和最优性缺陷。Delta采用两阶段方法：（1）安全性测试，在收集决策策略数据的同时评估被测智能体（AUT）是否发生灾难性故障;（2）最优性测试，利用前一阶段收集的数据通过离线强化学习训练挑战者智能体。随后通过比较挑战者智能体与AUT进行差异化测试;挑战者获得更高累计奖励的实例表明AUT存在最优性问题。我们展示了Delta在五个环境中的有效性。我们研究了三种离线强化学习算法（BC、BCQ和CQL）在生成挑战者智能体方面的有效性。实验结果表明，安全性测试数据集对于训练具备能力的DRL智能体具有价值。在Delta框架内，使用BCQ训练的挑战者智能体在识别最优性问题方面效果最佳。在这五个环境中，Delta平均发现了2,518个最优性问题，比基线方法高出50.2%。

WAM-OPD: On-Policy Distillation for World Action Models

WAM-OPD：世界行动模型的政策提炼

• Authors: Liuhaichen Yang, Zhuang Jiang, Chenchao Sheng, Zezhi Tang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.22364
• Pdf link: https://arxiv.org/pdf/2608.22364
• Abstract World action models (WAMs) couple visual future prediction with robot action generation, but accelerated students can lose task capabilities during distillation and later encounter states that are poorly represented by offline data. We study whether on-policy distillation (OPD) can repair such a student without requiring sparse-reward reinforcement learning. We introduce WAM-OPD, a deployment-consistent post-training recipe for a video-first WAM. The student acts in the environment and therefore determines the history distribution. A frozen teacher labels those student histories with coherent video and action targets, while the student action branch is trained under its own generated video plan, as it is at deployment. Joint video and action losses update lightweight adapters in the shared backbone, together with an action flow-matching regularizer. In preliminary RoboTwin 2.0 studies on two tasks, the released one-video/one-action-step Flash-WAM improves from 0.0% to 58.3% success on HANDOVER MIC, and from 16.7% to 33.3% on PUT OBJECT CABINET. These task-specific results are an initial capability proof rather than evidence of broad or uniform generalization. They nevertheless suggest that dense teacher supervision on student-induced histories is a promising post-training interface for video-first WAMs.
• 中文摘要 世界动作模型（WAMs）将可视化未来预测与机器人动作生成结合，但加速学生在提炼过程中可能会失去任务能力，后续遇到离线数据难以充分呈现的状态。我们研究策略中提纯（OPD）是否能在不需稀疏奖励强化学习的情况下修复此类学生。我们引入WAM-OPD，这是一种部署一致的培训后视频优先WAM方案。学生在环境中行动，从而决定历史分布。冻结教师用连贯的视频和动作目标标记学生历史，而学生行动分支则根据自身生成的视频计划进行训练，就像部署时一样。联合视频和动作损失更新共享骨干中的轻量级适配器，并配合动作流匹配正则化器。在对两项任务的初步RoboTwin 2.0研究中，发布的单视频/一步Flash-WAM在HANDOVER MIC的成功率从0.0%提升至58.3%，在PUT OBJECT CABINET中从16.7%提升至33.3%。这些任务特定结果是初步能力证明，而非广泛或统一的泛化证据。尽管如此，它们仍表明，对学生引发历史进行密集教师监督，是视频优先WAM的有前景的培训后接口。

Small Reasoning Models are Instruction Followers in Function Calling

小推理模型是函数调用中的指令跟随者

• Authors: Yalda Taheri, Mohammad Hassan Heydari, Erfan Naaman, Afsaneh Fatemi
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.22472
• Pdf link: https://arxiv.org/pdf/2608.22472
• Abstract Function calling represents the core capability of agentic large language models (LLMs). Existing research has focused on enhancing LLMs function-calling accuracy through fine-tuning, reinforcement learning (RL), and multi-agent frameworks, particularly for native function-calling LLMs. This work demonstrates that LLMs achieve superior accuracy in function calling in instruction-following contexts (i.e., standard user-assistant interactions) rather than a tool calling context. We introduce Instruction-Followed Function Calling (IFFC), a novel framework that decouples function-calling logic from the primary LLM and delegates it to a dedicated smaller model operating within the instruction-following paradigm. Our method consistently outperforms both native function calling (NFC) and prompt-based function calling (PFC) baselines, with particularly strong gains on reasoning-oriented LLMs. Furthermore, we demonstrate that IFFC maintains robust performance under aggressive quantization, enabling efficient on-device deployment without significant accuracy degradation. This work establishes a new paradigm for reliable, resource-efficient function calling in edge-computing scenarios.
• 中文摘要 函数调用代表了代理大型语言模型（LLM）的核心能力。现有研究重点是通过微调、强化学习（RL）和多代理框架提升LLM的功能调用准确性，特别是针对原生函数调用LLMs。这项工作表明，LLM在指令跟随上下文（即标准用户-助手交互）中，功能调用的准确性优于工具调用上下文。我们引入了指令跟随函数调用（IFFC），这是一种新颖框架，将函数调用逻辑从主LLM中解耦，委托给一个在指令跟随范式内运行的专用小型模型。我们的方法持续优于本地函数调用（NFC）和基于提示的函数调用（PFC）基线，尤其在推理导向的LLM中表现突出。此外，我们证明IFFC在激进量化下保持稳健性能，使得设备内部署高效且不显著降低准确率。这项工作为边缘计算场景中可靠且资源高效的函数调用建立了新的范式。

Scaling Curriculum Learning For Autonomous Driving

自动驾驶课程学习规模化

• Authors: Cevahir Koprulu, David Paz, Feng Tao, Yuliang Guo, Xinyu Huang, Ufuk Topcu, Liu Ren
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.22549
• Pdf link: https://arxiv.org/pdf/2608.22549
• Abstract Batched simulators for autonomous driving have recently enabled training reinforcement learning (RL) agents at scale, encompassing thousands of traffic scenarios and billions of interactions within a matter of days. Although such high-throughput feeds RL algorithms faster than ever, their sample-efficiency has not kept pace: As the standard training scheme, domain randomization uniformly samples scenarios, thereby consuming a vast number of interactions on cases that contribute little to learning. Curriculum learning offers a remedy by adaptively prioritizing scenarios that matter most to policy improvement. We present CL4AD, the first integration of curriculum learning into batched autonomous driving simulators by framing scenario selection as an unsupervised environment design problem. We introduce utility functions that shape curricula based on success rates and the realism of the agent's behavior, in addition to existing regret-estimation functions. Large-scale experiments in GPUDRIVE demonstrate that curriculum learning achieves a 99% success rate a billion steps earlier than domain randomization, reducing wall-clock time by 77%, and outperforms heuristic curricula with static and dynamic attributes, with only one exception at the largest scale. An ablation under limited compute shows that curriculum learning improves sample efficiency by 67%. We also investigate how utility functions behave at scale, and how prioritized scenarios evolve during training. We release an implementation of CLForAD in GPUDRIVE.
• 中文摘要 自动驾驶的批量模拟器最近使训练强化学习（RL）代理能够大规模实现，在几天内涵盖数千个交通场景和数十亿次交互。尽管如此高通量使强化学习算法比以往更快地供给算法，但其采样效率并未跟上：作为标准训练方案，域随机化统一采样场景，从而消耗大量对学习贡献不大的案例交互。课程学习通过自适应优先排序对政策改进最重要的场景，提供了解决方案。我们提出了CL4AD，这是首次将课程学习集成到批量自动驾驶模拟器中，将场景选择框架为一个无监督环境设计问题。我们引入了基于成功率和代理行为真实性塑造课程的效用函数，除了现有的遗憾估计函数外。GPUDRIVE中的大规模实验表明，课程学习比领域随机化早十亿步实现99%的成功率，将墙钟时间缩短77%，并且在最大尺度上仅有一个例外，优于具有静态和动态属性的启发式课程。在有限计算条件下的消融显示，课程学习能提升样本效率67%。我们还研究了效用函数在大规模下的表现，以及优先级场景在训练过程中的演变。我们在GPUDRIVE中发布了CLForAD的实现。

Coalition-Aware Skill Reliability for Self-Evolving Agents

联盟感知技能可靠性，适用于自我进化的特工

• Authors: Qiyan Zhao, Xiaofeng Zhang, Bo Liu, Minda Chen, Wei Xiong, Jingyang Chen, Guanting Ye, Wenhao Yu, Xiaosong Yuan, Shijie Han, Da-Han Wang, Jianmin Ji, Fei Huang, Xu-Yao Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.22610
• Pdf link: https://arxiv.org/pdf/2608.22610
• Abstract Agent skills, structured artifacts distilled from interaction trajectories and dynamically reused from skill banks, have become a central mechanism for enabling large language model (LLM)-based self-evolving agents to learn from past experience. Yet existing work has largely focused on the operational aspects of skills, such as acquisition, evolution, and retrieval, while leaving a more fundamental reliability question unresolved: Do accumulated skills in an agent's skill bank actually make positive mechanistic contributions? We investigate this question through systematic skill-bank audits across alternative bank compositions and deployment domains, measuring the resulting changes in agent behavior. These audits reveal two recurring reliability failures: coalition pollution, where bank-level gains conceal negative coalition-level skill contributions, and cross-domain utility reversal, where source-beneficial skills reverse their effects after transfer. These findings motivate two reliability interventions: coalition-aware skill selection during skill accumulation and label-free skill masking after transfer. Coalition-Aware Skill Selection (CASS) selects more reliable candidate skills for the current bank using sampled Shapley marginals. Unsupervised Skill-Masked Coalition Optimizer (u-SMCO) masks transferred skills whose exclusion improves retrieval quality on unlabeled target-domain data. Agentic experiments on LoCoMo, LongMemEval, HotpotQA, and ALFWorld show that CASS and u-SMCO consistently improve task performance and cross-domain generalization over strong skill-based self-evolving agent baselines. Beyond accuracy, coalition-conditioned reliability modeling reduces sensitivity to noisy outcome-reward fluctuations during reinforcement learning and exposes the limits of isolation-based skill evaluation.
• 中文摘要 代理技能是从交互轨迹中提炼并动态重用的结构化产物，已成为使基于大型语言模型（LLM）的自我进化代理从过往经验中学习的核心机制。然而，现有研究主要聚焦于技能的操作层面，如习得、进化和检索，而留下了一个更根本的可靠性问题未解：代理技能库中积累的技能是否真的会产生积极的机制贡献？我们通过系统性地对替代技能库组成和部署领域的技能库审计来探讨这个问题，测量代理行为的变化。这些审计揭示了两个反复出现的可靠性失败：联盟污染，即银行层面的收益掩盖了联盟层面的负面技能贡献;以及跨域效用逆转，即源头受益技能在转移后逆转其效应。这些发现促使了两种可靠性干预措施：技能积累期间的联盟感知技能选择和转移后的无标签技能掩蔽。联盟感知技能选择（CASS）利用抽样Shapley边缘筛选当前银行更可靠的候选技能。无监督的技能掩蔽联盟优化器（u-SMCO）掩蔽了移除可提升未标记目标域数据检索质量的转移技能。在LoCoMo、LongMemEval、HotpotQA和ALFWorld上的代理实验显示，CASS和you-SMCO在强技能基础自我演化代理基线上持续提升任务表现和跨域泛化能力。除了准确性外，联盟条件信度建模还降低了强化学习过程中对噪声结果-奖励波动的敏感性，并揭示了基于隔离的技能评估的局限性。

DeepSAGE: Stage-Aware Reinforcement Learning for Structured CBT Counseling Dialogue

DeepSAGE：结构化认知行为疗法咨询对话的阶段感知强化学习

• Authors: Qi Zhang, Heajun An, Prakriti Dumaru, Sang Won Lee, Lifu Huang, Pamela J. Wisniewski, Jin-Hee Cho
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.22615
• Pdf link: https://arxiv.org/pdf/2608.22615
• Abstract Large Language Model (LLM)-based counseling agents can generate fluent and supportive responses, but they often lack the structured, goal-directed progression required to conduct a coherent therapeutic session. We present DeepSAGE (Strategic AI Guidance Engine), a hybrid LLM--Deep Reinforcement Learning (DRL) framework for stage-aware counseling dialogue grounded in the first session of Cognitive Behavioral Therapy (CBT). DeepSAGE represents the session as eleven stages with explicit therapeutic objectives, with an external controller determines stage completion and the DRL model selects therapeutic intentions that guide LLM response generation. We evaluate DeepSAGE against six retrieval-, prompting-, stage-, and policy-based alternatives. DeepSAGE elicits higher simulated client engagement and openness and achieves the strongest balance of stage-goal completion and dialogue efficiency among stage-structured systems. Domain expert review further indicates that the generated conversations exhibit broadly plausible emotional trajectories and recognizable CBT processes. Because the evaluation relies primarily on simulated clients and model-based metrics, these findings demonstrate comparative dialogue-control improvements rather than clinical effectiveness. These results suggest that combining stage-structured dialogue with learned strategy selection is a promising approach for AI counseling, though clinical effectiveness, safety, and real-world utility require further human evaluation.
• 中文摘要 基于大型语言模型（LLM）的咨询代理能够生成流畅且支持性的反应，但它们通常缺乏进行连贯治疗会话所需的结构化、目标导向的进展。我们介绍DeepSAGE（战略AI指导引擎），这是一种基于认知行为疗法（CBT）第一节会话的混合型LLM-深度强化学习（DRL）框架，用于阶段感知的咨询对话。DeepSAGE将会话分为11个阶段，明确治疗目标，外部控制器决定阶段完成，DRL模型选择指导LLM反应生成的治疗意图。我们对DeepSAGE进行了六种基于检索、提示、阶段和策略的替代方案进行评估。DeepSAGE激发了更高的模拟客户参与度和开放性，并在阶段结构化系统中实现了阶段目标完成与对话效率之间最强的平衡。领域专家评审进一步表明，生成的对话表现出大致合理的情绪轨迹和可识别的认知行为疗法（CBT）过程。由于评估主要依赖模拟客户和基于模型的指标，这些发现显示的是对话控制的比较性提升，而非临床效果。这些结果表明，将阶段结构化对话与学习策略选择相结合，是人工智能咨询的一种有前景的方法，尽管临床效果、安全性和实际效用仍需进一步人工评估。

Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning

通过Real2Sim动力学估计和强化学习，增强扭矩控制机器人的Sim2Real传输

• Authors: Davide Bargellini, Alex Pasquali, Andrea Govoni, Riccardo Zanella, Gianluca Palli
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.22629
• Pdf link: https://arxiv.org/pdf/2608.22629
• Abstract Transferring reinforcement learning policies from simulation to Real-World robots remains a major challenge, particularly when dealing with low-level torque control, where even small modelling inaccuracies can lead to unstable or unsafe behaviours. In this work, we propose a Real2Sim2Real pipeline that improves Sim2Real transfer for torque-controlled robotic arms by combining trajectory matching, parameter optimization via genetic algorithms, and domain randomization. Using the 7-DOF Franka Emika Panda robot, we first identify friction, inertia, and gravity compensation parameters by minimizing the error between real and simulated joint trajectories. These calibrated dynamics are then used to train a TQC-based reinforcement learning agent in simulation. The trained policy is evaluated in both Gazebo and MuJoCo environments, and finally deployed on the real robot. Our results demonstrate a significant improvement in tracking accuracy and policy robustness after parameter tuning, with smooth policy transfer from simulation to the Real-World across multiple target-reaching tasks. This work highlights the effectiveness of accurate physical modelling in enabling stable and generalizable torque-based reinforcement learning policies.
• 中文摘要 将强化学习策略从模拟转移到现实机器人仍是一大挑战，尤其是在处理低级扭矩控制时，即使是小的建模不准确也可能导致不稳定或不安全行为。本研究提出一个Real2Sim2Real流水线，通过结合轨迹匹配、遗传算法参数优化和域随机化，改进扭矩控制机械臂的Sim2Real传输。利用7自由度的Franka Emika Panda机器人，我们首先通过最小化真实与模拟关节轨迹之间的误差，识别摩擦、惯性和重力补偿参数。这些校准后的动力学随后用于训练基于TQC的强化学习代理。训练好的策略在Gazebo和MuJoCo环境中评估，最终部署于真实机器人。我们的结果显示，参数调优后追踪准确性和策略鲁棒性显著提升，多个目标目标任务中策略从模拟到现实世界的策略传递平滑。本研究强调了准确物理建模在实现稳定且可泛化的基于扭矩的强化学习策略方面的有效性。

Learning Generalizable Behaviors for Terminal Agents

学习终端代理的可推广行为

• Authors: Yihang Yao, Bo Pang, Xuan Phi Nguyen, Ding Zhao, Shafiq Joty, Semih Yavuz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.22631
• Pdf link: https://arxiv.org/pdf/2608.22631
• Abstract Terminal agents are a compelling application of large language models (LLMs), with the potential to integrate deeply into users' daily workflows. Reinforcement learning (RL) is a key technique for improving their capabilities, making scalable training environments a central challenge. Since public real-user interaction data are scarce, synthetic environments provide a practical alternative, but often suffer from domain gaps and limited fidelity, leading to poor generalization. Existing work mainly scales the quantity and diversity of synthetic environments, while reward-signal quality and the mechanisms governing generalization remain under-explored. We study how RL improves terminal agents and propose the Agentic Compositional Generalization hypothesis: rather than teaching new domain-specific skills from scratch, RL primarily shapes high-level decision-making behaviors that compose and route low-level skills acquired during pre-training and supervised fine-tuning (SFT). This account is consistent with our empirical results and suggests that verifier quality, which determines which behaviors are reinforced, is more important than simply increasing environment quantity or diversity. Motivated by this insight, we propose River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization. Using this recipe, our RL-trained agent achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks. River also generalizes across model families, scales, agent harnesses, and RL objectives. Using fewer than 30% of the TMax training environments, River improves RL gains by 106% and 30% on average for models ranging from 2B to 27B on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively.
• 中文摘要 终端代理是大型语言模型（LLMs）的有力应用，具有深度融入用户日常工作流程的潜力。强化学习（RL）是提升其能力的关键技术，使可扩展的训练环境成为核心挑战。由于公共的真实用户交互数据稀缺，合成环境提供了实用的替代方案，但常常存在领域差距和有限的保真度，导致泛化能力较差。现有工作主要在扩展合成环境的数量和多样性，而奖励信号质量及泛化机制仍未被充分探讨。我们研究了强化学习如何改进终端代理，并提出了代理组合泛化假说：强化学习不是从零教授新的领域专属技能，而是主要塑造高层决策行为，这些行为组合并引导在预训练和监督微调（SFT）中获得的低层技能。这一观点与我们的实证结果一致，并表明验证者质量（决定哪些行为被强化）比单纯增加环境数量或多样性更为重要。基于这一见解，我们提出了River，一种简单的训练方案，通过过滤低质量环境并通过过程级行为规范化增强结果奖励来提升奖励质量。使用该方法，我们的强化学习代理在四个终端代理基准测试中，在评估的开源强化学习8B模型中表现最佳。River还对模型家族、尺度、代理工具和强化学习目标进行了推广。使用少于30%的TMax训练环境时，River在Terminal-Bench-Lite和Terminal-Bench-v2.1上2B至27B的模型平均提升了106%和30%的强化学习收益。

GeoRisk-RAG: A Hierarchy-Aware Risk Framework for Improving RAG Reliability through Selective Answering

GeoRisk-RAG：一个层级感知的风险框架，通过选择性回答提升RAG可靠性

• Authors: Meenu Ravi, Shailik Sarkar, Lulwah AlKulaib, Yordanos Tessema, Chang-Tien Lu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.22634
• Pdf link: https://arxiv.org/pdf/2608.22634
• Abstract Current work on improving reliability in large language model (LLM)- generated answers has primarily leveraged Retrieval-Augmented Generation (RAG), knowledge-graph augmentation, and reinforcement learning. While these methods are adept at enhancing and measuring reliability through semantic similarity and faithfulness, they often struggle to distinguish semantic similarity from geographic validity. This is especially critical in natural hazard management domains where geographic granularity (i.e., town vs. city vs. state) is significant for decision-making, as responses valid in one municipality may not transfer to another. In such domains, a confidently wrong answer carries greater risk than abstaining. We present GeoRisk-RAG, a novel hierarchy-aware framework that addresses this geographic-validity gap through selective answering. This framework explicitly estimates geographic applicability using a Directed Acyclic Graph (DAG)-based distance for context retrieval before response generation. Experiments on a novel held-out wildfire-related question-answering (QA) dataset show that GeoRisk-RAG significantly reduces false confidence rates for location-dependent questions, lowering the rate to 0.009 compared with ~0.090 for standard semantic similarity and reranking baselines, while consistently achieving higher human preference alignment. This work provides a more comprehensive assessment of end-to-end RAG pipelines by integrating geographic validity and selective-answering behavior for safer decision-making in geospatial domains.
• 中文摘要 当前关于提升大型语言模型（LLM）生成答案可靠性的研究主要利用检索增强生成（RAG）、知识图增强和强化学习。虽然这些方法擅长通过语义相似性和忠实度来增强和测量可靠性，但它们常常难以区分语义相似性与地理效度。这在自然灾害管理领域尤为关键，因为地理细度（如城镇、城市与州）对决策至关重要，因为一个市镇有效的回答可能无法转移到另一个市镇。在此类领域，自信错误的答案比弃权风险更大。我们介绍GeoRisk-RAG，一种新型层级感知框架，通过选择性回答解决这一地理有效性差距。该框架通过基于有向无环图（DAG）的距离，明确估计地理适用性，以实现上下文检索，再生成响应。在一个新型野火相关问答（QA）数据集上的实验显示，GeoRisk-RAG显著降低了位置相关问题的假置信率，标准语义相似度和基线重新排序率降至0.009，而标准语义相似度和基线重排序率为~0.090，同时持续实现更高的人类偏好对齐。该工作通过整合地理效度和选择性回答行为，更全面地评估了RAG流程，实现地理空间领域的更安全决策。

Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computing

持续控制的尖峰神经网络：传统计算中的神经形态强化学习

• Authors: Jessica Hunter, Md Maruf Hossain Shuvo, Krishna Roy
• Subjects: Subjects: Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE)
• Arxiv link: https://arxiv.org/abs/2608.22729
• Pdf link: https://arxiv.org/pdf/2608.22729
• Abstract Reinforcement learning (RL) algorithms have made strides over the past decade applying them to a wide range of problems and control tasks. However, the deployment of RL on neuromorphic hardware for continuous control tasks remains under-validated. Namely it is unclear whether replacing a conventional actor network with a spiking neural network (SNN) affects the performance of an agent before any hardware-specific benefits manifest. We provide a systematic validation of a minimal, neuromorphically viable spiking actor variant of Soft Actor-Critic (SAC) on conventional hardware, establishing a baseline for future neuromorphic RL research. In this paper, we propose the Spiking Actor Network Soft Actor Critic (SANSAC) to address the use of RL frameworks in continuous environments, designed as a framework that can be implemented on neuromorphic hardware. We compare a traditional Soft Actor Critic (SAC) network to SANSAC in a traditional computer. We demonstrate the near equivalent performance of SANSAC and SAC, while addressing the impact of hidden dimensions. Our results demonstrate the viability of SNN based algorithms in complex continuous environments, as well as competitive performance to traditional neural networks in traditional computers, providing a basis to continue exploring the use of SNNs in continuous RL frameworks.
• 中文摘要 强化学习（RL）算法在过去十年中取得了进展，将其应用于广泛的问题和控制任务。然而，在神经形态硬件上部署强化学习用于连续控制任务的验证仍然不足。具体来说，目前尚不清楚用尖峰神经网络（SNN）替代传统演员网络是否会影响代理的性能，而在硬件特定优势显现之前。我们系统地验证了一种在传统硬件上最小、神经形态可行的软演员-批判者（SAC）尖峰演员变体，为未来神经形态强化学习研究奠定了基础。本文提出尖峰演员网络软演员批评者（SANSAC），用于解决强化学习框架在连续环境中的应用，设计为可实现在神经形态硬件上的框架。我们将传统软演员批评者（SAC）网络与传统计算机中的SANSAC进行比较。我们展示了SANSAC和SAC的性能几乎等效，同时解决了隐藏维度的影响。我们的结果证明了基于SNN算法在复杂连续环境中的可行性，以及在传统计算机中与传统神经网络的竞争性能，为继续探索SNN在连续强化学习框架中的应用奠定了基础。

Can We Perform Online RL for Image Editing without Editing Rewards?

我们能否在不获得编辑奖励的情况下进行在线强化学习进行图像编辑？

• Authors: Qichao Ma, Jikang Cheng, Ling Liang, Zhaofei Yu, Tiejun Huang, Renye Yan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.22780
• Pdf link: https://arxiv.org/pdf/2608.22780
• Abstract Reinforcement learning (RL) enables direct preference optimization for image editing through editing-specific rewards, which remain less developed due to costly triplet supervision and complex task-dependent calibration. In contrast, text-to-image (T2I) generation benefits from a mature and diverse reward ecosystem spanning semantic alignment, aesthetics, realism, glyph shape, and other visual preferences. Extending this ecosystem to image editing would substantially broaden the range of visual preferences accessible to RL-based optimization, prompting the central question: \emph{Can We Perform Image Editing RL without Editing Rewards?} In this paper, we argue that the standard image editing dimensions have potential to be mapped to the T2I reward space: image quality can transfer directly, prompt following can be aligned through a description of the desired visual state, and reference consistency admits a coarse semantic conversion by encoding the source content to preserve. However, editing instructions specify relative changes, whereas T2I rewards require self-contained target descriptions; moreover, semantically valid captions from generic vision-language models may be incompatible with the frozen reward. Hence, we further introduce Lever-Edit, a two-stage framework that learns a reward-aligned captioner for counterfactual target descriptions, freezes it, and optimizes the editing policy solely with the transferred T2I reward. Experiments show competitive editing alignment and source preservation against editing-reward-based fine-tuning, while outperforming intuitive transfer baselines.
• 中文摘要 强化学习（RL）通过针对编辑的奖励实现了图像编辑的直接偏好优化，而这些奖励由于昂贵的三元组监督和复杂的任务依赖校准，这些奖励仍然较为有限。相比之下，文本到图像（T2I）生成受益于一个成熟且多样化的奖励生态系统，涵盖语义对齐、美学、真实感、字形形状及其他视觉偏好。将这一生态系统扩展到图像编辑，将大大拓宽基于强化学习优化的视觉偏好范围，从而引发核心问题：\emph{我们能否在没有编辑奖励的情况下进行图像编辑强化学习？}本文论证，标准图像编辑维度有潜力映射到T2I奖励空间：图像质量可以直接传递，提示跟随可以通过对期望的视觉状态描述进行对齐，引用一致性允许通过编码保存源内容实现粗略语义转换。然而，编辑指令指定了相对变化，而T2I奖励则需要自包含的目标描述;此外，来自通用视觉语言模型的语义有效说明可能与冻结奖励不兼容。因此，我们进一步引入了Lever-Edit，这是一个两阶段框架，学习一个与反事实目标描述相关的奖励对齐字幕，冻结它，并仅用转移的T2I奖励优化编辑策略。实验显示，编辑对齐和来源保存在竞争性上相较于基于编辑奖励的微调，同时优于直觉转移基线。

TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rollouts

TailSieve：用于LLM部署的部分展开引导尾路由

• Authors: Tianqi Xu, Lu Lv, Haoyang Huang, Wenjie Huang, Zhanming Shen, Yuhao Shen, Baolin Zhang, Xinyi Hu, Shuang Ge, Jun Dai, Tianyu Liu, Suorong Yang, Zhikai Li, Ye Bai, Jun Zhang, Lei Chen, Yue Li, Mingchen Wan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.22788
• Pdf link: https://arxiv.org/pdf/2608.22788
• Abstract Large-scale rollouts have become a core component of modern LLM systems, spanning reinforcement learning (RL) post-training, on-policy distillation (OPD), and sampling-heavy evaluation pipelines. Unlike online serving, which is typically optimized for request-level latency and throughput, a small number of long-tail generations can dominate the end-to-end makespan of an entire rollout step. In practice, rollout requests are often routed uniformly across replicas, which can place extremely long generations inside high-concurrency decoding batches. To address this, we present TailSieve, a partial-rollout-guided framework that jointly controls tail routing and replica allocation for LLM rollouts. In an idealized setting with known completion lengths, we show that makespan-optimal routing in the long-tail regime combines tail isolation with load balancing, and that a simple top-k policy closely approximates this offline optimum. Leveraging the observation that long-tail prompts tend to remain long-tailed across policy updates, TailSieve uses partial rollouts as a training-free signal for identifying candidate tail groups. A hierarchical controller then jointly adapts the number of isolated groups and the replica split between the tail and bulk pools using collected response-work history and a measured concurrency-throughput model. TailSieve achieves up to 1.67x routing-only speedup over uniform group routing. The resulting low-concurrency tail pool further enables route-specialized speculative decoding with MTP or DFlash, achieving up to 2.59x speedup over uniform routing. Selected prompts are regenerated under the current policy, preserving on-policy generation and avoiding additional routing-induced length bias in steady state.
• 中文摘要 大规模部署已成为现代LLM系统的核心组成部分，涵盖强化学习（RL）训练后、策略上提炼（OPD）和采样重度的评估流程。与通常针对请求级延迟和吞吐量优化的在线服务不同，少量长尾生成可能主导整个部署步骤的端到端完成时长。实际上，滚动请求通常均匀地在副本间路由，这使得高并发解码批次中出现极长的代数。为此，我们提出了TailSieve，一个部分部署引导框架，联合控制LLM推展的尾部路由和副本分配。在已知完成长度的理想环境中，我们展示了长尾状态下的makespan最优路由结合了尾隔离和负载均衡，且一个简单的top-k策略极其接近这一离线最优。利用长尾提示在策略更新期间往往保持长尾的观察，TailSieve 利用部分展开作为无训练信号来识别候选尾组。然后，一个分层控制器结合收集的响应工作历史和测量的并发吞吐量模型，联合调整孤立组数量和尾池与大块池的副本分布。TailSieve 在仅路由时可实现高达 1.67 倍的纯路由加速，相较于均匀组路由。由此产生的低并发尾池进一步支持基于 MTP 或 DFlash 的路由专用推测解码，在均匀路由下实现高达 2.59 倍的加速。选定的提示在当前策略下重新生成，保持策略中生成，避免稳态下路由引起的额外长度偏差。

FinixDoc: Rethinking Financial Document Parsing Beyond Saturated Benchmarks

FinixDoc：重新思考财务文件解析，超越饱和基准

• Authors: Hang Wang, Jin Zhang, Guoliang Xu, Pengyue Lu, Yao Li, Zijiao Zhang, Tianyu Huang, Weiqi Xiong, Yulong Wang, Chuqiao Lu, Wenkang Huang, Kai Yang, Yadong Li, Hui Li, Xingzhong Xu, Xiao Xu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.22842
• Pdf link: https://arxiv.org/pdf/2608.22842
• Abstract Financial document parsing requires accuracy, structural consistency, and verifiability that current benchmarks often fail to reflect. We present FinixDoc, an end-to-end agentic parsing system for real-world financial documents, with FinixDoc-VL, a 4B-scale vision-language model built on Qwen3-VL-4B, as its core parser. To characterize the gap between benchmark and deployment performance, we introduce a Document Parsing Capability Matrix organized along two practical axes: visual quality and document scale. Guided by this matrix, FinixDoc-VL is trained with a domain-adapted recipe combining homoglyph-aware contrastive learning and multi-stage reinforcement learning with composite domain-specific rewards. To better leverage our accumulated advantage in low-quality financial-document data and support large-scale, high-quality data production, we further build a human-in-the-loop Data Factory pipeline with confidence-aware expert review. For evaluation, we construct FinixDocBench, a financial-domain evaluation suite covering digital-native, camera-captured, ultra-large-page, and internal-workflow scenarios, with a compliance-reviewed subset released alongside this technical report. On its main subsets, FinixDoc-VL achieves the highest overall score (81.43) among evaluated baselines, outperforming the next-best open-source model by 5.13 points, with the largest gains on internal financial workflows (FinixInner: 84.08 vs. 78.73).
• 中文摘要 金融文件解析需要准确性、结构一致性和可验证性，而当前基准测试往往未能反映这一点。我们介绍FinixDoc，一个面向现实世界金融文档的端到端代理分析系统，核心解析器为FinixDoc-VL，一个基于Qwen3-VL-4B构建的4B规模视觉语言模型。为了刻画基准测试与部署性能之间的差距，我们引入了文档解析能力矩阵，沿两个实用轴线组织：视觉质量和文档规模。在该矩阵指导下，FinixDoc-VL采用领域适配配方训练，结合同形感知对比学习和多阶段强化学习，并提供复合领域特定奖励。为了更好地利用我们在低质量财务文档数据上的积累优势，支持大规模高质量数据生产，我们进一步构建了带有信心感知专家审核的人机数据工厂流水线。为评估，我们构建了FinixDocBench，一套涵盖数字原生、摄像头捕捉、超大页面及内部工作流程场景的金融领域评估套件，并与本技术报告一同发布合规审查子集。在其主要子集中，FinixDoc-VL在评估基线中获得最高总分（81.43分），比次佳开源模型高出5.13分，内部财务工作流程提升最大（FinixInner：84.08对78.73）。

GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis

GSAR：具有自我演进数据综合的移动图形界面代理的目标-状态-锚点奖励

• Authors: Long Zhang, Yuhan Chen, Chaoran Zhang, Wanxia Cao, Kun Huang, Pengzhi Gao, Wei Liu, Jian Luan, Chenliang Li, Lixin Zou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.22847
• Pdf link: https://arxiv.org/pdf/2608.22847
• Abstract Vision-Language Models (VLMs) based GUI agents stand to benefit significantly from online reinforcement learning (RL). However, their training is bottlenecked by two fundamental issues: current data synthesis methods for GUI Agents rely on specific environments and struggle to generate diverse data, while existing evaluators either suffer from limited scalability or provide inaccurate and unreliable reward signals. To overcome these challenges, we introduce GSAR (Goal-State-Anchor Reward), a RL reward framework that supports scalable task generation and delivers reliable reward signals for stable and efficient policy optimization. Our approach features self-evolving data synthesis, which produces multiple environments through task execution and generates diverse tasks and goal states. Complementing this, a state-anchor mechanism automatically annotates task-relevant UI elements in successful goal states as reference anchors. During RL training, these reference anchors provide accurate, scalable reward signals that substantially enhance efficiency. Extensive evaluations demonstrate that our framework achieves over 90% accuracy on offline trajectory verification and performs closest to rule-based methods. Furthermore, agents trained using our reward framework exhibit strong performance on both AndroidWorld and our constructed benchmark, establishing a scalable approach for GUI agent training.
• 中文摘要 基于视觉语言模型（VLMs）的图形用户智能体将从在线强化学习（RL）中获益显著。然而，它们的训练受到两个根本性问题的瓶颈：当前图形界面代理的数据综合方法依赖特定环境，难以生成多样化的数据;而现有评估器要么受限于可扩展性，要么提供不准确且不可靠的奖励信号。为克服这些挑战，我们引入了GSAR（目标-状态-锚点奖励）——一个支持可扩展任务生成并提供可靠奖励信号以实现稳定高效策略优化的强化学习奖励框架。我们的方法具有自我演化的数据综合，通过任务执行生成多个环境，并生成多样化的任务和目标状态。此外，状态锚机制自动在成功目标状态中注释任务相关的用户界面元素作为参考锚点。在强化学习训练中，这些参考锚点提供准确、可扩展的奖励信号，显著提升效率。广泛评估表明，我们的框架在离线轨迹验证中达到超过90%的准确率，且表现最接近基于规则的方法。此外，使用我们奖励框架训练的代理在AndroidWorld和我们构建的基准测试中表现出强劲表现，建立了可扩展的图形界面代理训练方法。

PsychJail: Exploring Psychological Jailbreaks via Multi-Turn Persuasion of LLM Policies

PsychJail：通过多回合说服LLM策略探讨心理越狱

• Authors: Zeyu Feng, Qingyu Wu, Yuzhe Luo, Hua Cheng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.23028
• Pdf link: https://arxiv.org/pdf/2608.23028
• Abstract Large language models (LLMs) are increasingly deployed in education, healthcare, policy advising, and other interactive settings, where users engage them as sustained social interlocutors rather than one-shot query engines. This shift makes jailbreaks a growing safety threat, yet most research emphasizes single-turn prompt optimization or iterative attack refinement, leaving psychologically grounded multi-turn vulnerabilities underexplored. We present PsychJail, a psychology-guided framework for red teaming aligned LLMs through theory-grounded, multi-turn persuasion. PsychJail maps established social-psychological persuasion techniques into a tactic-conditioned attack policy. It factorizes each attacker action into a Change-of-Meaning analysis, tactic selection, and victim-visible message, operationalizing the Persuasion Knowledge Model (PKM). The policy is refined with trajectory-level reinforcement learning using a PKM-gated reward that credits early jailbreak success only when every turn contains a well-formed Change-of-Meaning analysis. Across four aligned victim models, PsychJail achieves the highest average attack success rate (87.3%) and outperforms strong single-turn and multi-turn baselines on every model. We also measure susceptibility at the action that breaks each victim, revealing four distinct model-level fingerprints that identify which persuasion levers affect each model and how broadly. These fingerprints help explain cross-model transfer asymmetry. We interpret them as four candidate psychological profiles-rationalist, credibility-driven, narrative-monoculture, and broadly persuadable-while treating this interpretation as a conjecture requiring future validation. Our findings establish psychological jailbreaks as a distinct red-teaming frontier for increasingly interactive LLMs.
• 中文摘要 大型语言模型（LLMs）越来越多地被应用于教育、医疗、政策咨询及其他互动环境中，用户将其视为持续的社交对话者，而非一次性查询引擎。这一转变使越狱成为日益增长的安全威胁，然而大多数研究强调单回合提示优化或迭代攻击优化，导致心理基础的多回合脆弱性尚未被充分探索。我们介绍PsychJail，一个基于理论、多回合说服的心理学指导框架，用于红队对齐的LLMs。PsychJail将既定的社会心理学说服技巧映射到战术条件攻击策略中。它将每个攻击者行为分解为意义变化分析、战术选择和受害者可见信息，操作化说服知识模型（PKM）。该策略通过轨迹级强化学习优化，采用PKM门槛奖励，只有当每回合包含良好的意义变更分析时，才会计入早期越狱成功。在四个对齐的受害者模型中，PsychJail实现了最高的平均攻击成功率（87.3%），并且在所有模型上都优于强的单回合和多回合基线。我们还测量了破坏每个受害者的行动的易感性，揭示了四个不同的模型级指纹，识别出哪些说服杠杆影响每个模型及其影响范围。这些指纹有助于解释跨模型转移不对称。我们将它们解释为四种候选心理特征——理性主义、可信度驱动、叙事单一文化和广泛可说服性——同时将此解释视为需要未来验证的猜想。我们的研究结果确立了心理越狱作为日益互动的大型语言模型（LLM）中红队化的独特前沿。

Artificial Empathy: Towards a Framework for Unsupervised Agency Detection and Policy Reconstruction

人工同理心：迈向无监督机构检测与政策重建的框架

• Authors: Peter Kuhn, Chris Pang, Sonakshi Chauhan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.23030
• Pdf link: https://arxiv.org/pdf/2608.23030
• Abstract We study how an AI system can identify and model other agents in its environment from observation alone, which is a capability necessary for cooperative behaviour in the real world. This problem is less constrained than inverse reinforcement learning and remains largely unexplored. We propose a framework that uses a reinforcement learning agent, trained on an independent task as a prior about agentic dynamics, to perform agency detection and policy reconstruction.
• 中文摘要 我们研究人工智能系统如何仅凭观察识别和建模环境中的其他代理，这是现实世界中合作行为所必需的能力。该问题比逆强化学习受限性小，且尚未被广泛探索。我们提出一个框架，利用强化学习代理作为代理动态的先验任务训练，进行代理检测和策略重建。

MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks

MobilePA-Bench：为移动规划代理在复杂现实任务中的基准测试

• Authors: Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, Steven Hoi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.23035
• Pdf link: https://arxiv.org/pdf/2608.23035
• Abstract As on-device LLM agents evolve into personal copilots, the mobile operating system has become a key testbed for this paradigm, making rigorous capability evaluation essential. Yet existing benchmarks fall into two camps, each with a critical blind spot: GUI-centric benchmarks test surface-level screen manipulation while overlooking background tool use and long-horizon planning, whereas static function-calling benchmarks rely on offline API matching that is detached from real runtime constraints. To close this gap, we present \textbf{MobilePA-Bench}, an interactive, stateful, and tool-centric benchmark for evaluating the tool-calling and planning abilities of mobile planning agents. MobilePA-Bench runs on an executable sandbox that maintains live application databases and returns structured feedback, spanning $13$ functional domains and $212$ realistic mobile tools. Beyond basic tool use, it evaluates a central planning agent along three advanced dimensions: \emph{(1)~Sub-agent Collaboration}---decomposing a complex task and delegating specialized work to capable sub-agents; \emph{(2)~Memory Usage}---recalling stored memories, user profiles, and past preferences to resolve implicit requests; and \emph{(3)~Skill Usage}---invoking pre-packaged composite skills instead of planning every step from scratch. Extensive experiments show that current frontier LLMs remain unreliable in mobile settings: performance drops sharply under strict tool ordering, permission limits, and unexpected runtime errors. By pairing an interactive function-calling sandbox with evidence-based verification, MobilePA-Bench serves as both a practical diagnostic benchmark and an interactive foundation for agentic reinforcement learning---accelerating the development of dependable mobile agents.
• 中文摘要 随着设备内LLM代理演变成个人副驾驶，移动操作系统已成为这一范式的关键试验场，使得严谨的能力评估变得不可或缺。然而，现有基准测试分为两类，各自存在关键盲点：以图形界面为中心的基准测试表层屏幕操作，忽视了后台工具使用和长期规划;而静态函数调用基准依赖离线API匹配，远离真实运行时约束。为弥合这一差距，我们推出了\textbf{MobilePA-Bench}，一个交互式、有状态且以工具为中心的基准测试，用于评估移动规划代理的工具调用和规划能力。MobilePA-Bench运行在可执行沙盒上，维护实时应用数据库并返回结构化反馈，涵盖价值13美元的功能领域和212美元的真实移动工具。除了基础工具使用外，它还从三个高级维度评估中央规划代理：\emph{（1）~子代理协作}---分解复杂任务并将专业工作委托给有能力的子代理;\emph{（2）~内存使用}---回忆存储的记忆、用户配置文件和过去偏好以解决隐式请求;以及\emph{（3）~技能使用}---调用预包装的综合技能，而非从零规划每一步。大量实验表明，当前前沿大型语言模型在移动环境中依然不可靠：在严格的工具排序、权限限制和意外运行时错误下，性能急剧下降。通过将交互式函数调用沙箱与基于证据的验证结合，MobilePA-Bench既是实用的诊断基准，也是智能体强化学习的互动基础---加速了可靠移动代理的发展。

Macro-Action Topological Navigation under Noisy Localization using Reinforcement Learning

使用强化学习进行噪声定位下的宏动作拓扑导航

• Authors: Simon Hakenes, Tobias Glasmachers
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.23055
• Pdf link: https://arxiv.org/pdf/2608.23055
• Abstract Navigating large, photorealistic 3D apartments from raw pixels is widely considered infeasible for plain reinforcement learning. We build an agent that does it anyway, estimating its own pose from the camera alone. The agent has to reach several target objects in sequence, and their positions change between episodes, so it must explore to find them. It builds on our earlier object-centric topological controller, which still read the agent's true pose and its object detections from the simulator. Here we replace that true pose with an onboard, object-centric estimate. For each object we keep a bank of ORB features that, when the object is seen again, yield a rough pose measurement, which a minimal Extended Kalman Filter (EKF) fuses with a motion model. As on a real robot, the executed motions are noisy. The estimate drifts, but the agent and the nearby objects drift together, so a locally consistent pose is enough to follow each short edge and then home in visually on the target, which lets us replace full SLAM with a much smaller model, closer to how biological navigation appears to work. In the photorealistic Habitat simulator, the agent reaches its target objects from vision alone, with a pose that only needs to be locally consistent.
• 中文摘要 从原始像素导航大型逼真3D公寓，普遍认为对纯强化学习来说不可行。我们构建了一个代理，仅凭摄像头估计自身姿态。代理必须依次到达多个目标对象，且它们的位置在不同事件中变化，因此必须探索以找到它们。它基于我们早期的以对象为中心的拓扑控制器，该控制器仍读取智能体的真实姿态及其对象检测结果。这里我们用内置的以对象为中心的估计值替代真实姿态。每个物体我们都保留一组ORB特征，当物体再次被观察时，得到粗略姿态测量，通过最小扩展卡尔曼滤波器（EKF）与运动模型融合。与真实机器人一样，执行的运动有噪声。估计值会漂移，但代理和附近物体会一起漂移，因此局部一致的姿态足以跟踪每个短边，然后在视觉上聚焦目标，这使我们能够用更小的模型替代完整的SLAM，更接近生物导航的运作方式。在照片写实的Habitat模拟器中，代理仅凭视觉到达目标物体，姿态只需局部一致即可。

Distributed Trajectory Planning and Resource Allocation for Dynamic Multi-UAV Collaborative Computing

动态多无人机协作计算的分布式轨迹规划与资源分配

• Authors: Tiankui Zhang, Wenlong Xu, Tianyi Shi, Xiaoxia Xu, Arumugam Nallanathan
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2608.23123
• Pdf link: https://arxiv.org/pdf/2608.23123
• Abstract This paper investigates a multiple uncrewed aerial vehicles (UAVs)-enabled distributed mobile edge computing (MEC) framework, where the set of collaborative UAVs dynamically varies over time due to their energy states and service loads. The joint optimization of trajectory planning and resource allocation is formulated as a Stackelberg game, where UAVs and mobile terminals (MTs) are modeled as leaders and followers, respectively. UAVs aim to maximize their benefits by balancing executed workload, energy cost, and resource allocation revenue, while MTs seek to minimize their total overhead, composed of computing delay and resource costs, through offloading and resource-request decisions. A hierarchical joint optimization algorithm is developed within a multi-agent deep reinforcement learning (MADRL) framework to coordinate UAVs and MTs in a distributed manner. At the leader level, UAVs jointly determine their trajectories, task migration ratios, MT-UAV association, and unit computing resource pricing. Each UAV is modeled as an agent in a partially observable Markov decision process, and the agents are jointly trained via multi-agent proximal policy optimization (MAPPO) under the centralized-training-and-decentralized-execution paradigm. At the follower level, MTs determine their optimal task offloading ratios and requested computing resources using a two-stage iterative algorithm. Simulation results demonstrate stable convergence under dynamic UAV participation. Compared to the no-collaboration benchmark, the proposed algorithm improves UAV efficiency by 18.58% through inter-UAV task migration and reduces average MT overhead by 33.77% over the fully offloading scheme. It also outperforms other benchmarks under varying network scales and capabilities by jointly optimizing UAV operations and resource utilization.
• 中文摘要 本文探讨了一种多无人机（UAV）支持分布式移动边缘计算（MEC）框架，其中协作无人机集合会因其能量状态和服务负载而动态变化。轨迹规划与资源分配的联合优化被表述为Stackelberg游戏，其中无人机和移动终端（MT）分别被建模为领导者和跟随者。无人机旨在通过平衡执行工作负载、能源成本和资源分配收入来最大化其效益，而MT则通过卸载和资源请求决策，试图最小化其总开销（包括计算延迟和资源成本）。在多智能体深度强化学习（MADRL）框架内开发了一种分层联合优化算法，以分布式方式协调无人机和MT。在领导者层面，无人机共同确定其轨迹、任务迁移比率、MT-UAV关联以及单元计算资源定价。每架无人机被建模为部分可观测的马尔可夫决策过程中的代理，代理们通过多代理近端策略优化（MAPPO）在集中训练与去中心化执行范式下联合训练。在跟随者层面，MT通过两阶段迭代算法确定其最优任务卸载比率和请求的计算资源。模拟结果显示在动态无人机参与下实现稳定收敛。与无协作基准相比，该算法通过无人机间任务迁移提升了无人机效率18.58%，并使平均MT开销降低33.77%，在不同网络规模和能力下，通过联合优化无人机操作和资源利用率。

Guided Riemannian Optimization (GuRO): Bridging Model Predictive Control and Decision Transformers

引导黎曼优化（GuRO）：模型预测控制与决策变换器的桥接

• Authors: Hossein Abdi, Satya Prakash Dash, Mingfei Sun
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.23204
• Pdf link: https://arxiv.org/pdf/2608.23204
• Abstract Decision-making in high-dimensional, nonlinear systems remains a central challenge in robotics. While model-based methods like Model Predictive Control (MPC) offer sample efficiency and interpretability, their performance degrades when the dynamics model is inaccurate or long-horizon predictions are required. Conversely, model-free reinforcement learning (RL) learns policies directly from interaction but suffers from high sample complexity and unstable optimization. Recent advances in sequence modeling have inspired transformer-based decision-making frameworks that can unify MPC and RL, but their training typically faces significant optimization challenges due to highly non-convex loss landscapes. In this work, we propose a novel framework that integrates MPC with RL in a sequence decision-making framework and leverages a curvature-aware optimization to efficiently tackle non-convex loss landscapes. MPC provides predictions of locally optimal trajectories that guide the decision transformer, removing the need for extensive offline pretraining. To address the slow and unstable convergence of traditional optimizers, we train the policy in a Riemannian parameter space using an efficient Riemannian (curvature-aware) method, leading to faster and more robust optimization. We evaluate our framework on high-dimensional quadruped control tasks and demonstrate consistent improvements over strong baselines, including TRPO, SAC, and Online Decision Transformer, achieving higher returns and faster convergence.
• 中文摘要 高维非线性系统中的决策仍然是机器人学中的核心挑战。虽然基于模型的方法如模型预测控制（MPC）提供了样本效率和可解释性，但当动力学模型不准确或需要长视野预测时，其性能会下降。相反，无模型强化学习（RL）直接通过交互学习策略，但存在高样本复杂性和不稳定的优化问题。序列建模的最新进展激发了基于变压器的决策框架，能够统一MPC和RL，但由于高度非凸的损耗景观，其训练通常面临重大优化挑战。本研究提出了一个新颖框架，将MPC与强化学习集成在序列决策框架中，并利用曲率感知优化高效处理非凸损耗景观。MPC提供局部最优轨迹预测，指导决策变换器，消除大量离线预训练的需求。为解决传统优化器收敛缓慢且不稳定的问题，我们采用高效的黎曼（曲率感知）方法在黎曼参数空间中训练策略，实现更快更稳健的优化。我们评估了高维四足控制任务的框架，并展示了在强基线（包括TRPO、SAC和在线决策变换器）上持续改进，实现更高收益和更快收敛。

Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning

Agent-G$^2$：智能强化学习的高斯指导

• Authors: Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang, Yongliang Shen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.23318
• Pdf link: https://arxiv.org/pdf/2608.23318
• Abstract Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G$^2$, a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G$^2$ on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G$^2$ outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.
• 中文摘要 基于提示的强化学习通过在每次部署前保留专家轨迹的前缀，解决长期能动任务中的奖励稀疏性，使策略从更接近成功的状态进行探索。其有效性取决于指导深度：即应保留多少轨迹。现有方法将该深度视为确定性标量。计划方法在样本间共享一个值，忽略每个任务的异质性;逐样本探测则单独估计该值，但代价是增加滚动次数。我们发现有用的指导占据一条深度带，其信息量分布大致为高斯分布，而非集中于单一最优点。我们提出了Agent-G$^2$，这是一种高斯指导框架，从一个高斯任务的深度计算，其中心和扩散是在线从已收集的策略优化中收集的推测中估算的，无需探针推展或学习深度预测器。该中心结合了全局基线和每个簇的难度，以及簇内的扩散轨迹。我们在ALFWorld和Qwen2.5-1.5B / 7B-Instruct上的WebShop评估了Agent-G$^2$。Agent-G$^2$在ALFWorld上表现优于基于提示、无提示和Aux-RL的2.3 / 3.9 / 7.4分，且每个样本探测的推广成本不到三分之一。

Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents

超越视频的思考：统一视频推理与开放世界视频代理的深入研究

• Authors: Wenqi Liu, Shijie Ma, Yunxiao Wang, Meng Liu, Qile Su, Han Liu, Bohan Hou, Xuanyu Zheng, Changyi Liu, Tianke Zhang, Haonan Fan, Kaiyu Jiang, Yingxin Li, Jiankang Chen, Xu Wang, Bin Wen, Tingting Gao, Han Li, Jianhua Yin, Yinwei Wei, Xuemeng Song
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.23329
• Pdf link: https://arxiv.org/pdf/2608.23329
• Abstract Open-world video understanding often requires a model to locate sparse visual evidence and acquire external knowledge that is absent from the video and its parametric memory. While Thinking-with-Videos enables active temporal perception and Deep Research supports multi-step information seeking, the two capabilities are typically developed in isolation. We introduce VideoRover, a unified Video Deep Research framework that iteratively coordinates video cropping, multimodal search, and webpage browsing. Given a video-question pair, VideoRover uses each tool result to select the next action, so localized video clips guide external retrieval and retrieved evidence triggers further video inspection and verification. To develop this capability, we construct an automated data curation pipeline, producing 26K verified SFT trajectories and 3K challenging RL instances. We also introduce VideoRover-Bench, a benchmark stratified by video duration and research difficulty. Experiments on VideoDR and VideoRover-Bench show that our VideoRover-8B-RL achieves performance comparable to proprietary models in the direct-answer setting without tool use while outperforming larger open-source models equipped with the same tool suite. Ablation studies and training dynamics further validate the complementary roles of active video grounding, external retrieval, and long-horizon reinforcement learning.
• 中文摘要 开放世界视频理解通常需要模型定位稀疏的视觉证据，并获取视频及其参数记忆中缺失的外部知识。虽然“带视频思考”实现了主动的时间感知，深度研究支持多步信息寻求，但这两者通常是单独开发的。我们引入了VideoRover，一个统一的视频深度研究框架，迭代协调视频裁剪、多模态搜索和网页浏览。给定视频问题对，VideoRover利用每个工具结果选择下一步操作，因此本地化视频片段引导外部检索，检索到的证据触发进一步的视频检查和验证。为开发此功能，我们构建了自动数据整理流程，产生26K经过验证的SFT轨迹和3K挑战性的强化学习实例。我们还推出了VideoRover-Bench，这是一个按视频时长和研究难度分层的基准测试。VideoDR和VideoRover-Bench上的实验表明，我们的VideoRover-8B-RL在直接回答模式下，无需工具即可实现与专有模型相当的性能，同时优于配备相同工具套件的大型开源模型。消融研究和训练动态进一步验证了主动视频接地、外部检索和长视野强化学习的互补作用。

Temporal Property-driven Design Space Exploration with Reinforcement Learning for Cyber-Physical Systems

基于时间属性的设计空间探索与强化学习，适用于网络物理系统

• Authors: Tagir Fabarisov, Maxime Cordy
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.23440
• Pdf link: https://arxiv.org/pdf/2608.23440
• Abstract Design-space exploration of configurable Cyber-Physical Systems (CPS) requires executable evaluation when design choices affect timing, fault propagation, recovery behavior, and temporal-property satisfaction. Repeated stochastic executions make exhaustive exploration impractical for large design spaces. This paper presents a temporal-property-driven CPS design workflow using Reinforcement Learning (RL). At design time, the RL agent selects subsystem alternatives to assemble a candidate system model. The model is then evaluated through simulation, during which online temporal-property monitors observe runtime traces and produce functional-property violation indicators. These indicators are combined with evaluated non-functional terms for budget, recoverability, sustained compliance, and operational use to calculate the reward used for subsequent candidate selection. The workflow is evaluated on a methane-sensitive mine-pump CPS. The corresponding executable case-study model is provided as additional contribution. RL-guided search identifies the highest-reward design observed in the experiments after 26 episodes (corresponds to 130 executable simulations). These designs were reached with fewer simulations than surrogate-guided Bayesian Optimization and population-based Genetic Algorithm baselines under the same executable model and reward formulation. Ablation study results indicate that value-based feedback and reuse of previous simulation traces contribute to this reduction.
• 中文摘要 当设计选择影响时序、故障传播、恢复行为和时间属性满足时，对可配置网络物理系统（CPS）进行设计空间探索需要可执行的评估。反复的随机执行使得大规模设计空间的穷尽探索变得不切实际。本文提出了一种基于时间属性驱动的CPS设计流程，采用强化学习（RL）。在设计阶段，RL代理选择子系统替代方案以组装候选系统模型。模型随后通过模拟进行评估，在线时间属性监控员观察运行时痕迹并生成功能性违规指标。这些指标与预算、可恢复性、持续合规性和运营使用等非功能项结合，计算后续候选选择的奖励。工作流程在甲烷敏感矿泵CPS上进行评估。相应的可执行案例研究模型作为额外贡献提供。强化学习引导搜索识别实验中在26次实验后观察到的最高奖励设计（对应130次可执行模拟）。这些设计在同一可执行模型和奖励表述下，模拟次数少于代理引导贝叶斯优化和基于群体的遗传算法基线。消融研究结果表明，基于价值的反馈和对先前模拟痕迹的重复利用对这一减少有贡献。

Reward-Free Continual Adaptation for Resilient Space Robots

无奖励的持续适应，打造韧性空间机器人

• Authors: Andrej Orsula, Miguel Olivares-Mendez, Carol Martinez
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.23452
• Pdf link: https://arxiv.org/pdf/2608.23452
• Abstract Space robots operate in extreme environments where hardware degradation can critically compromise traditional control strategies. While continual reinforcement learning offers a promising mechanism for online adaptation, it inherently requires access to a reward signal during deployment. However, precise reward computation in space is often infeasible due to the lack of external tracking systems and the overall complexity of the environment. To address the challenge of unobservable rewards, we introduce a reward-free continual learning framework that leverages latent-state world models. By pre-training a model-based agent across diverse simulations, the world model learns a robust predictor of the reward structure within its latent space. Upon deployment to an environment with severe hardware degradation, we freeze the observation encoder and reward predictor to update only the transition dynamics of the world model through unsupervised rollouts. By training the policy entirely on imagined trajectories generated by this updated world model, the agent adapts to altered dynamics without receiving new rewards. We demonstrate our approach across simulated planetary traversal, orbital navigation, and precision assembly tasks subjected to severe morphological failures.
• 中文摘要 空间机器人在极端环境中运行，硬件劣化可能严重影响传统控制策略。虽然持续强化学习提供了在线适应的有前景机制，但本质上需要在部署时访问奖励信号。然而，由于缺乏外部跟踪系统和环境整体复杂性，精确的太空奖励计算往往不可行。为应对无法观察奖励的挑战，我们引入了一个无奖励的持续学习框架，利用潜态世界模型。通过对基于模型的智能体在多种模拟中预训练，世界模型学习其潜空间内奖励结构的稳健预测变量。部署到硬件严重劣化环境时，我们冻结观察编码器和奖励预测器，仅通过无监督展开更新世界模型的过渡动态。通过完全基于该更新世界模型生成的想象轨迹训练策略，智能体适应变化的动态而不获得新的奖励。我们展示了在模拟行星穿越、轨道导航和精密组装任务中，尽管这些任务存在严重的形态失效。

How to Train a Critic Stably and Efficiently

如何稳定高效地培训评论家

• Authors: Penghui Qi, Xiangxin Zhou, Wee Sun Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.23566
• Pdf link: https://arxiv.org/pdf/2608.23566
• Abstract Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop \textbf{Best-Practice Critic Optimization (BPCO)}, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at this https URL
• 中文摘要 基于群体的强化学习方法，如针对大型语言模型的GRPO，通过为每个提示抽样多个回答来避免训练批评者。可靠的批评者可以从一个回答中估算代币级优势，但标准的批评者训练方案往往不稳定。我们研究了这种不稳定性，并开发了\textbf{最佳实践批评优化（BPCO）}，这是一种结合DPPO、奖励区间边界价值预测、蒙特卡洛价值目标、未归一化策略优势和长度适应性广义优势估计的方案。由于批评者仅在训练中使用，BPCO还可以将其条件化为奖励定义的信息，如参考答案或评分评分标准，这些信息在策略中隐藏。受控实验分离出每个设计选择的影响。在从1.5亿参数到30B-A3B混合专家模型的数学推理任务中，BPCO持续提升基于批评者的强基线，并在每个提示抽样一个回答的前提下，与基于群体的基线相匹配甚至超过。同样的配方还通过基于评分标准的奖励提升学习效果。这些结果表明，精心设计的批评者为群体相对优势估计提供了可靠的替代方案。代码可在此 https URL 获取

Keyword: diffusion policy

ODG-NoMaD: Overhead-Camera Direction-Guided NoMaD

ODG-NoMaD：俯视摄像头方向制导NoMaD

• Authors: Blossom Treesa Bastian, Keerthi S. Shetty, Manish Kolachalam, Rani Malhotra, Ashish Dutta
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.21395
• Pdf link: https://arxiv.org/pdf/2608.21395
• Abstract NoMaD [31] is a learned vision-navigation policy that unifies goal-conditioned navigation and exploration in a single goal-masked diffusion policy. In an unseen environment, however - where neither a goal image nor a topological map is available - it can only explore undirectedly, wandering without global awareness. We present ODG-NoMaD, which gives NoMaD's exploration mode a global sense of where to proceed, without retraining the policy. An overhead depth camera is used once on deployment to build an occupancy map and plan a global path, which is segmented to yield a desired heading; a per-frame traversability map from the robot's onboard depth then refines this into a collision-free direction. The gradient of a cosine direction cost is injected into the final denoising steps, rotating sampled trajectories toward this direction while preserving the multimodality of exploration. In simulated office environments with and without random obstacles, ODG-NoMaD reduces the residual distance to the target by up to an order of magnitude over unguided exploration, outperforms the point-goal cost guidance of NaviDiffusor [37], and is the only configuration that remains collision-free on every trial.
• 中文摘要 NoMaD [31] 是一种学习型视觉导航策略，将目标条件导航和探索统一为单一的目标掩蔽扩散策略。然而，在无视环境下——既无目标图像也无拓扑图——它只能无方向探索，游荡无全局意识。我们介绍 ODG-NoMaD，为 NoMaD 的探索模式提供全局方向感，无需重新训练策略。部署时使用俯视深度相机绘制占用地图并规划全局路径，路径被分割以生成期望航向;然后从机器人机载深度的每帧可遍历性图细化为无碰撞方向。余弦方向成本梯度注入最终去噪步骤，使采样轨迹朝该方向旋转，同时保持探索的多模态性。在有无随机障碍的模拟办公环境中，ODG-NoMaD通过无导引探索将目标剩余距离缩短多达一个数量级，优于NaviDiffusor的点目标成本指导[37]，并且是唯一在每次试验中保持无碰撞的配置。