生成时间: 2026-10-13 01:29:44 (UTC+8); Arxiv 发布时间: 2026-10-12 20:00 EDT (2026-10-13 08:00 UTC+8)

今天共有 40 篇相关文章

Keyword: reinforcement learning

DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models

DiffusionART：高效、稳定且可扩展的扩散模型强化

• Authors: Zhenglin Wan, Jiayu Chen, Zhengfeiyang Zhang, Pengfei Zhou, Wangbo Zhao, Fangyuan Kong, Pengfei Wan, Kun Gai, Yang You
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.12572
• Pdf link: https://arxiv.org/pdf/2610.12572
• Abstract Online reinforcement learning (RL) fine-tunes diffusion models with black-box rewards, but reverse-process methods such as Flow-GRPO are limited in efficiency by poor credit assignment, while forward-process methods such as DiffusionNFT are efficient yet unstable without carefully tuned exponential-moving-average (EMA) off-policy training. We show that NFT-style objectives contain a hidden forward--reverse consistency force that joint positive--negative training cancels only partially, and that EMA stabilizes training through restoration rather than off-policy rollouts. We introduce DiffusionART (Diffusion Reinforcement with Aggregation-First Reward Tilting), which aggregates each prompt's rewards into a shared reward-improving direction for reverse-process states, removing the consistency force by construction, and adds a trust-region-masked second update for drift-adaptive restoration without EMA tuning. It requires no differentiable reward model, remains effective with fewer samples per prompt, and supports pairwise reward models (DiffusionART-Pairwise). Across image and video generation, DiffusionART trains efficiently and stably without per-task tuning. In particular, on HunyuanVideo-13B with equally weighted visual and motion quality rewards, DiffusionART reaches the late-stage mean reward levels of DiffusionNFT and Flow-GRPO-Fast with 12.02 and 11.72 times fewer GPU hours, respectively, while achieving 28.8% and 53.0% higher mean training rewards.
• 中文摘要 在线强化学习（RL）通过黑箱奖励微调扩散模型，但逆向过程方法如Flow-GRPO因信用分配不佳而效率受限，而前向过程方法如DiffusionNFT则高效但不稳定，除非经过精心调优的指数移动平均（EMA）非策略训练。我们证明NFT风格目标包含隐藏的正向-反向一致性力，正负训练联合仅部分抵消，EMA通过恢复而非非策略推广稳定训练。我们引入DiffusionART（聚合优先奖励倾斜扩散强化），将每个提示的奖励聚合为逆向过程状态的共享奖励改善方向，消除结构一致性力，并增加了信任区域掩蔽的第二次更新，用于无EMA调优的漂移自适应恢复。它无需微分奖励模型，每提示样本数更少，且支持成对奖励模型（DiffusionART-Pairwise）。在图像和视频生成中，DiffusionART高效且稳定地训练，无需逐任务调整。特别是在HunyuanVideo-13B上，视觉和动作质量均等加权，DiffusionART达到DiffusionNFT和Flow-GRPO-Fast晚期平均奖励水平，分别减少12.02倍和11.72倍，平均训练奖励提升28.8%和53.0%。

Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents

超越二元可验证奖励：软件工程代理的行为自我奖励

• Authors: Yiran Wu, Andrew Zhao, Yurui Chang, Qingyun Wu, Xuan Feng
• Subjects: Subjects: Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2610.12598
• Pdf link: https://arxiv.org/pdf/2610.12598
• Abstract Reinforcement learning with verifiable rewards (RLVR) provides reliable supervision for software engineering agents through executable tests, but binary rewards fail to distinguish trajectories with substantially different debugging quality. We investigate whether the current policy can judge its own trajectories and use these judgments to improve task performance without a separately trained reward model. To this end, we introduce Behavioral Self-Rewarding (BSR), a reward-refinement method that reuses the policy to compare observable debugging behavior among trajectories from the same task with the same verifier outcome. BSR aggregates pairwise preferences into task-local scores and converts them into bounded auxiliary rewards, introducing reward contrast while preserving the verifier-defined pass/fail ordering. This provides additional supervision from already-collected trajectories without requiring new environment interactions. Experiments on SWE-bench Verified and SWE-bench-Live show consistent improvements across the evaluated Qwen3-14B and Qwen3-32B configurations, supporting using the policy's own comparative judgments as a complementary learning signal for training software engineering agents.
• 中文摘要 带可验证奖励的强化学习（RLVR）通过可执行测试为软件工程代理提供可靠的监督，但二元奖励无法区分调试质量显著不同的轨迹。我们研究当前策略是否能判断自身轨迹，并利用这些判断在不依赖单独训练的奖励模型的情况下提升任务表现。为此，我们引入了行为自我奖励（BSR），这是一种奖励细化方法，重用该策略比较同一任务中相同验证者结果的轨迹间可观察的调试行为。BSR将成对偏好汇总为任务局部得分，并将其转换为有界辅助奖励，引入奖励对比，同时保持验证者定义的通过/失败排序。这为已收集轨迹提供额外监督，而无需新的环境交互。在SWE-bench Verified和SWE-bench-Live上的实验显示，在评估的Qwen3-14B和Qwen3-32B配置中，持续有显著的改进，支持将政策自身的比较判断作为辅助学习信号来训练软件工程代理。

Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting

深度研究代理的反思提示优化：一种主题不相交、多种子协议，用于测量可转移收益和提示级过拟合

• Authors: Elias Lumer, Sahil Sen, Vamse Kumar Subbiah, Kevin Paul
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.12601
• Pdf link: https://arxiv.org/pdf/2610.12601
• Abstract Prompt optimization is an attractive alternative to reinforcement learning for adapting language-model agents, since it is cheap and produces human-readable artifacts. On factual tasks, however, it carries a failure mode that scalar-reward reinforcement learning does not. An optimizer that reads the training examples can write their answers into the prompt, so an in-distribution gain may be memorization rather than transferable behaviour. We apply reflective prompt evolution (GEPA) to the two system prompts of a multi-agent deep-research agent and evaluate it under a topic-disjoint split in which no topic is shared across train, validation, and test. Held-out citation reward rises by 0.079, from 0.702 to 0.781, with a bootstrap 95\% confidence interval of [0.051, 0.107], and the gain replicates across three seeds (mean 0.107). Nearly half of the winning prompt is a memorized answer key, yet a controlled ablation shows this content earns no reward even on the exact training questions it encodes. The large validation-minus-held-out gap is therefore small-sample selection under a noisy judge, not memorization. Varying the verifier isolates what makes the gain transfer. Decomposed per-dimension feedback is the causal ingredient, and collapsing it to a single scalar reward makes held-out transfer negative. The gain persists under an out-of-family judge and is largely reproduced by a hand-written version of the discovered recipe. Credible evaluation of prompt optimizers therefore requires held-out topics, multiple seeds, and verifier ablations. We release the candidate prompts, lineage diffs, figures, tables, and analysis.
• 中文摘要 提示优化是适应语言模型代理的强化学习替代方案，因为它成本低廉且产生人类可读的工件。然而，在事实任务中，提示优化存在标量奖励强化学习所不具备的失败模式。读取训练示例的优化器可以将答案写入提示中，因此分布内的收益可能是记忆而非可迁移行为。我们对多智能体深度研究代理的两个系统提示应用反思提示演化（GEPA），并在主题不相交的分配下评估，该分割中训练、验证和测试中无主题共享。被搁置引用奖励增加0.079，从0.702升至0.781，自助95%置信区间为[0.051， 0.107]，增益在三个种子间重复（均值0.107）。近一半的获胜提示是记忆的答案键，但受控消融显示，即使在其所编码的训练问题上，这些内容也没有任何奖励。因此，验证减去未保留的巨大差距是由于在嘈杂评审下样本选择较小，而非记忆。变化验证者可以分离出增益转移的因素。分解的每维反馈是因果成分，将其归纳为单一标量奖励则使保留转移为负。收益在非家族评判下持续存在，主要由发现配方的手写版本复制。因此，对提示优化器的可信评估需要保留主题、多个种子和验证者消融。我们发布候选提示、谱系差异、图表、表格和分析。

ROAM: Reinforcement Learning Optimized Adaptive Moving Target Defense for Smart Grid

ROAM：强化学习优化的智能电网自适应移动目标防御

• Authors: Mohammad Zakaria Haider, Prabin Mali, Muhammad Nadeem, Sumit Paudyal, Fei Ding, Mohammad Ashiqur Rahman
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.12730
• Pdf link: https://arxiv.org/pdf/2610.12730
• Abstract Smart grids rely on distributed phasor measurement units (PMUs) and wide-area communication networks whose data-acquisition pipelines are vulnerable to stealthy false data injection (FDI) attacks that evade conventional bad-data detection (BDD). Moving target defense (MTD) counters this exposure by continuously remapping sensor identities, but existing schemes apply static or pre-scheduled randomization and are each constrained by a single failure mode: seed-based methods are fast yet fragile to key compromise, while machine learning (ML) based prediction methods are resilient yet incur prohibitive latency. Neither adapts its randomization to live threat evidence. This paper presents ROAM, a deep reinforcement learning (DRL) driven MTD framework in which a deep Q-network (DQN) agent continuously re-optimizes the sensor-to-ID mapping at the aggregation layer. At each decision step, the agent selects the deception coverage, flags the specific sensors to remap, and sets the split of the remapped sensors between seed-based and ML-prediction-based recovery, without modifying PMU hardware and while preserving IEEE C37.118 compatibility. On held-out BDD-compliant FDI scenarios across the IEEE 14-, 57-, and 118- bus systems, ROAM keeps the percentage of stealthy attacks below 3% while sustaining detection rates above 93%. Against an adaptive soft actor-critic (SAC) attacker that observes the defender's configuration, ROAM keeps stealthy attacks below 2.6% with detection above 92%. Hardware-in-the-loop testing on an OPAL-RT platform confirms real-time feasibility, with sub-18ms average shuffle execution and an average reshuffle blind window of 2.6% of the 3-s decision cycle.
• 中文摘要 智能电网依赖分布式相量测量单元（PMU）和广域通信网络，其数据采集管道易受传统错误数据检测（BDD）的隐形虚假数据注入（FDI）攻击。移动目标防御（MTD）通过持续重映射传感器身份来应对这种风险，但现有方案采用静态或预定随机化，且均受单一失败模式限制：基于种子的方法快速但易被密钥泄露，机器学习（ML）预测方法虽具韧性但延迟过高。两者均未根据实时威胁证据调整随机化。本文介绍ROAM，一种深度强化学习（DRL）驱动的MTD框架，深度Q网络（DQN）代理在聚合层持续重新优化传感器到ID映射。在每个决策步骤中，代理选择欺骗覆盖范围，标记具体需要重新映射的传感器，并将重新映射后的传感器分配为基于种子的恢复和基于机器学习的恢复，无需修改PMU硬件，同时保持IEEE C37.118兼容性。在IEEE 14、57和118总线系统中，符合BDD标准的FDI场景中，ROAM将隐形攻击的比例控制在3%以下，同时保持检测率超过93%。面对观察防御者配置的自适应软行为者-批判者（SAC）攻击者，ROAM将隐形攻击控制在2.6%以下，探测率超过92%。OPAL-RT平台上的硬件在环测试证实了实时可行性，平均洗牌执行时间低于18毫秒，重洗盲窗平均为3秒决策周期的2.6%。

MADS: Multi-Agent Dialog Simulation for Robust Conversational Agents

MADS：多智能体对话模拟，用于稳健对话智能体

• Authors: Arkajyoti Chakraborty, Adam Zernik, Ishika Kulkarni, Mudit Arora, Hugo Lin, Justin Chiu, Alessandro Di Bari, Neha Gupta, Andreas Stolcke
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.12757
• Pdf link: https://arxiv.org/pdf/2610.12757
• Abstract Task-oriented conversational agents remain fragile under realistic conditions, especially when users exhibit non-cooperative behavior. Existing function-calling benchmarks underrepresent adversarial conversation trajectories, thereby limiting the training resources available for developing robust agents. We present MADS (Multi-Agent Dialog Simulation), a synthetic data generation pipeline that produces structured, diverse dialogue trajectories. We inject adversarial conversation, producing validated conversation trajectories that capture realistic, unhappy paths. Using MADS, we construct MADS-Bench: a benchmark spanning six domains, 55 personas, and 1095 conversations, with approximately 66% of conversations containing failure-prone conversation scenarios. We evaluate the robustness of the trained models using a two-stage training approach that combines supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR).
• 中文摘要 任务导向的对话代理在现实条件下依然脆弱，尤其是在用户表现出非合作行为时。现有的功能调用基准低估了对抗对话轨迹，从而限制了开发稳健代理的训练资源。我们提出了MADS（多代理对话模拟），这是一套合成数据生成流程，能够生成结构化、多样化的对话轨迹。我们注入对抗性对话，生成经过验证的对话轨迹，捕捉真实且不愉快的路径。利用MADS，我们构建了MADS-Bench：一个涵盖六个领域、55个角色和1095次对话的基准测试，约66%的对话包含易失败的对话场景。我们采用两阶段训练方法评估训练模型的稳健性，结合了监督微调（SFT）和可验证奖励（RLVR）的强化学习。

Reflex: Visually Grounded Reactive Humanoid Control

反射：视觉接地的反应型人形控制

• Authors: Taoyang Jia, Weikai Huang, Linxin Song, Jared Darlington, Jieyu Zhao, Yue Wang, Jiafei Duan, Zhongzheng Ren, Ranjay Krishna
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.12772
• Pdf link: https://arxiv.org/pdf/2610.12772
• Abstract Humans effortlessly react to the world; they adeptly return a tennis shot, catch a falling cup, step aside from a passing cart, or pass a box tossed during truck unloading. These everyday actions require anticipating a moving target and reflexively responding with the whole body within about a second. Such visual reactive whole-body control remains challenging for humanoids, which must (1) coordinate whole-body control, (2) react to object dynamics in real time, and (3) ground perception in egocentric vision. We study catching thrown boxes as a representative task that demands all three capabilities; existing systems cover only part of it, relying on end-effectors, motion capture, or marked objects. We present Reflex, a three-stage framework that learns these capabilities stage by stage while connecting them through a compact representation of object dynamics. First, a state-based policy learns whole-body catching with reinforcement learning. Second, the policy learns to infer the dynamics of a moving box from delayed and incomplete observations. Third, an egocentric RGB-D encoder learns to recover this dynamics representation directly from visual observations while keeping the whole-body controller fixed. This decomposition provides each capability with a direct training signal and makes visual learning substantially more scalable. In simulation, Reflex achieves 85.7% catch success from egocentric RGB-D, compared with 87.9% when given privileged box observations. Temporal visual history is critical: removing it reduces success by 35.6 and 39.8 points on centered and laterally offset throws. On a real Unitree G1, Reflex catches human-thrown boxes with 65% and 40% success at 2.5 and 5 m, reaching 76% and 59% of human catchers' success at the same distances.
• 中文摘要 人类能轻松应对世界;他们能灵活回击网球、接住掉落的杯子、避开路过的推车，或在卡车卸货时传递被抛弃的箱子。这些日常动作需要预判移动目标，并在大约一秒内用全身反射反应。这种视觉反应的全身控制对类人生物来说依然具有挑战性，他们必须（1）协调全身控制，（2）实时响应物体动态，（3）以自我为中心的视觉实现地面感知。我们研究捕捉被扔出的箱子作为一项代表性任务，需要三者兼具;现有系统仅覆盖部分内容，依赖端效器、动作捕捉或标记物体。我们介绍Reflex，一个三阶段框架，逐步学习这些能力，并通过紧凑的物体动态表示将它们连接起来。首先，基于状态的策略通过强化学习学习全身捕捉。其次，策略通过延迟和不完整的观察学推断移动盒子的动态。第三，以自我为中心的RGB-D编码器学习直接从视觉观察恢复这种动态表征，同时保持全身控制器固定。这种分解为每个能力提供了直接训练信号，使视觉学习大幅提升了可扩展性。在模拟中，Reflex在自我中心的RGB-D中捕获成功率为85.7%，而在给予特权盒子观察时为87.9%。时间视觉历史至关重要：移除它会使中心和横向偏移投掷的成功率分别降低35.6分和39.8分。在真实的Unitree G1上，Reflex在2.5米和5米距离接住人类投掷的箱子，成功率分别为65%和40%，在相同距离下达到人类捕手的76%和59%。

Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search

通过强化学习的神经结构搜索实现高效的视听语音增强

• Authors: Khubaib Ahmed, Ahsan Adeel, Nasir Saleem, Kia Dashtipour, Ahsan Ulhaq, Amir Hussain
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2610.12796
• Pdf link: https://arxiv.org/pdf/2610.12796
• Abstract Audio-visual speech enhancement (AVSE) improves speech intelligibility in noisy environments but often relies on computationally expensive models unsuitable for resource-constrained edge devices. This paper introduces a reinforcement learning-driven neural architecture search (RL-NAS) framework to discover compact, efficient AVSE architectures. The search space preserves temporal alignment between audio and visual streams, while a novel reward function jointly optimises speech enhancement performance, measured by SI-SNR improvement, and model compactness. Using proximal policy optimisation with constraint-aware action masking, the framework evaluates only 35 candidate architectures and identifies a model with 2.9 million parameters, compared with 22 million in the baseline. The proposed architecture achieves a 14.5 dB SI-SNR improvement, only 0.7 dB below the baseline, while reducing model size by approximately nine-fold. Evaluation on commercial mobile CPUs demonstrates a 4.3-fold reduction in real-time factor, supporting practical on-device deployment. Compared with random search, RL-NAS achieves 92% higher search efficiency. Architectural analysis reveals that shallower but wider temporal convolutional blocks and aggressive visual pathway pruning can maintain enhancement performance while substantially reducing computational requirements. These findings demonstrate the potential of reinforcement learning-guided architecture search for developing lightweight, efficient and deployable multimodal speech enhancement systems.
• 中文摘要 视听语音增强（AVSE）在嘈杂环境中提升语音可理解性，但通常依赖计算量高的模型，不适合资源受限的边缘设备。本文引入了强化学习驱动神经架构搜索（RL-NAS）框架，以发现紧凑高效的AVSE架构。搜索空间保持音频流与视觉流的时间对齐，同时采用新颖的奖励函数共同优化语音增强性能（通过SI-SNR提升和模型紧凑度衡量）。通过近端策略优化和约束感知动作掩蔽，该框架仅评估35种候选架构，识别出参数数为290万的模型，而基线为2200万。所提架构实现了14.5 dB SI-SNR提升，仅比基线低0.7 dB，同时模型规模约减少了九倍。商用移动CPU的评估显示实时系数降低了4.3倍，支持了实际的设备部署。与随机搜索相比，RL-NAS的搜索效率提高了92%。架构分析显示，较浅但更宽的时序卷积块和激进的视觉路径剪枝可以保持增强性能，同时显著降低计算需求。这些发现展示了强化学习引导架构搜索在开发轻量级、高效且可部署的多模态语音增强系统中的潜力。

Reward Hacking as a Transmissible Strategy in Multi-Agent Coding Systems

作为多智能体编码系统中可传递策略的奖励黑客

• Authors: Yuepeng Hu, Pegah Ghahremani, Reza Aghajani, Mahesh Pasupuleti, Himaghna Bhattacharjee
• Subjects: Subjects: Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.12822
• Pdf link: https://arxiv.org/pdf/2610.12822
• Abstract Multi-agent systems are increasingly used to solve coding tasks by coordinating agents that share code and plans. These agents are often optimized with reinforcement learning (RL) using automated evaluators, but bypassing evaluation can earn reward without producing a genuine solution. Prior work has largely treated reward hacking as a single-agent failure. In this work, we show that a reward-hacking strategy elicited in one agent can propagate to others and affect system coordination. Across four open-weight backbones, coding RL elicits and amplifies reward hacking in two settings: explicit mechanism descriptions in the system prompt and mid-training that models upstream exposure to facts about the coding environment. Using post-RL agents from the mid-training setting as sources, we observe propagation through code and shared memory, and across roles through natural-language plans. Even agents based on small open-weight models induce reward hacking in stronger closed-weight targets. On hard tasks, propagation can intensify along a transmission chain after the original source leaves as the code acquires more persuasive justifications. We also theoretically analyze conditions under which reward hacking amplifies, persists, or decays across agent interactions. Beyond propagation, reward hacking can bias task routing toward reward-hacking agents and survive candidate aggregation. As potential mitigations, we evaluate code review by the coder or a separate verifier, both of which reduce delivered hacks. Together, these results establish reward hacking as a transmissible strategy whose effects extend beyond the agent in which it first emerges.
• 中文摘要 多智能体系统越来越多地被用于通过协调共享代码和计划的智能体来解决编码任务。这些智能体通常通过自动评估器进行强化学习（RL）优化，但绕过评估可能会获得奖励，而不必产生真正的解决方案。以往的研究大多将奖励黑客视为单智能体的失败。在本研究中，我们展示了一个智能体触发的奖励黑客策略可以传播到其他智能体，影响系统协调。在四种开放权重骨干中，强化学习在两种环境中诱发并放大奖励黑客行为：系统提示中的显式机制描述和模拟上游对编码环境事实暴露的训练中期。利用中期训练设置的后强化智能体作为来源，我们观察到通过代码和共享记忆传播，以及通过自然语言计划跨角色的传播。即使是基于小型开权模型的代理，也会在更强的闭权目标中诱导奖励黑客。在困难任务中，原始源离开后，传播可能沿传输链增强，因为代码获得了更具说服力的理由。我们还理论分析了奖励黑客在代理交互中放大、持续或衰减的条件。除了传播，奖励黑客还可能使任务路由偏向奖励黑客智能体，并存活候选聚合。作为潜在的缓解措施，我们评估编码者或独立验证者的代码审查，这两种方法都能减少交付的黑客事件。这些结果共同证明了奖励黑客是一种可传递的策略，其效果超越了首次出现的智能体。

Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

跨行为策略优化：通过共享回归目标实现离线演示及策略启动与关闭部署的统一学习

• Authors: Du Cheng, Junyu Lu, Jingfan Zhang, Yuzhuo Zhang, Haoxuan Jia, Shichao Weng
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.12830
• Pdf link: https://arxiv.org/pdf/2610.12830
• Abstract Reasoning post-training draws on offline demonstrations and policy-generated rollouts, yet conventional supervised fine-tuning (SFT) followed by reinforcement learning (RL) separates their objectives and limits continued reuse of collected responses. We propose \textbf{Cross-Behavior Policy Optimization} (CBPO), which makes these sources compatible with one regression loss through a fixed-reference target. Because the generating policy does not enter this target, positive and negative offline completions, reference responses, and current and historical rollouts can share a common reward-labeled record format and replay buffer. As new responses enter the buffer, optimization retains the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. Restricting the available sources gives offline, on-policy, and off-policy variants, while CBPO-Unified combines them. Across two model scales and four mathematical reasoning benchmarks, CBPO delivers leading \texttt{avg@32} and \texttt{pass@32} performance across benchmarks and data regimes. On Qwen3-1.7B, CBPO-Unified improves benchmark-averaged \texttt{avg@32} and \texttt{pass@32} over SFT DAPO by 2.33 and 4.44 percentage points, respectively. In the replay ablation, retaining historical rollouts improves these metrics by 1.87 and 3.72 percentage points over the no-history configuration.
• 中文摘要 训练后推理依赖离线演示和策略生成的推广，而传统的监督微调（SFT）及强化学习（RL）则将目标分离，限制收集响应的持续重用。我们提出 \textbf{交叉行为策略优化}（CBPO），使这些来源通过固定引用目标实现一个回归丢失兼容。由于生成策略不进入该目标，正负离线完成、参考响应以及当前和历史推广可以共享一个带奖励标记的记录格式和回放缓冲区。当新响应进入缓冲区时，优化保持相同的回归目标，无需协调独立的 SFT 和 RL 目标，也不需要存储行为策略概率、重要性加权或学习批评者。限制可用来源会提供离线、策略中和非策略的变体，而 CBPO-Unified 则结合了它们。在两个模型尺度和四个数学推理基准中，CBPO在基准测试和数据体系中均提供领先的\texttt{avg@32}和\texttt{pass@32}性能。在Qwen3-1.7B中，CBPO-United比SFT DAPO提升了基准平均\texttt{avg@32}和\texttt{pass@32}分别提升2.33个百分点和4.44个百分点。在重放消融中，保留历史推展可使这些指标比无历史配置提升1.87个百分点和3.72个百分点。

Reinforcement Learning via Fixed-point Flow Matching

通过不动点流匹配进行强化学习

• Authors: Denis Blessing, Onur Celik, Gerhard Neumann
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.12848
• Pdf link: https://arxiv.org/pdf/2610.12848
• Abstract In maximum entropy and behavior-constrained reinforcement learning (RL), the optimal policy is given by a Boltzmann distribution. Generating samples from this distribution can be challenging, as it is often complex and multimodal. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL. To address this limitation, we propose fixed-point flow matching (FFM), a data-free procedure for training and fine-tuning flow policies in RL. FFM leverages the target flow identity to formulate the actor update as a fixed-point iteration, utilizing a flow-matching-style regression objective where the critic's action gradients serve as regression targets. We show that the optimal policy is a fixed point of this iteration. Through extensive experiments, we demonstrate that FFM effectively fine-tunes pretrained flow policies in the offline-to-online setting and enables efficient and accurate training of flow policies from scratch in online RL. These results establish fixed-point flow matching as a principled approach for learning expressive flow-based policies without requiring training data.
• 中文摘要 在最大熵和行为约束强化学习（RL）中，最优策略由玻尔兹曼分布给出。从该分布生成样本具有挑战性，因为它通常复杂且多模态。虽然用流匹配训练的流策略提供了表示此类分布的强大框架，但它们传统上依赖于训练数据的访问，而这在在线强化学习中通常不可得。为解决这一限制，我们提出了不动点流匹配（FFM），这是一种无数据的过程，用于在强化学习中训练和微调流策略。FFM利用目标流的恒等性将演员更新表述为不动点迭代，采用流匹配式回归目标，其中批评者的动作梯度作为回归目标。我们证明最优策略是该迭代的不动点。通过大量实验，我们证明了FFM在离线到在线环境中有效微调预训练流策略，并实现在线强化学习中从零开始高效且准确地训练流策略。这些结果确立了不动点流匹配作为一种原则性方法，用于学习基于表达式的基于流的策略，而无需训练数据。

JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents

JoyInside-VLM-Agent：具身智能体的产品条件多模态策略学习

• Authors: Gangyi Li, LuYao Wang, Jingfu Sun, Jingyi Yuan, Hang Liu, Hao Zong, Yujia Hu, Tingting Gao, Jianyong Wang, Xu Yang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.12957
• Pdf link: https://arxiv.org/pdf/2610.12957
• Abstract Embodied agents must translate multimodal observations and user requests into executable actions while adapting to product constraints, changing goals, and execution feedback. We present JoyInside-VLM-Agent, a ReAct-style multimodal agent developed through product-conditioned trajectory construction and staged policy learning for robotic platforms and smart hardware. A closed-loop sandbox generates multimodal interaction trajectories covering goal revisions and failure recovery, which are checked against execution outcomes and semantic criteria. Supervised fine-tuning and multi-teacher on-policy distillation precede reinforcement learning with DPER (Dual-Source Privileged On-Policy Self-Distillation with Token-Level Evidence Routing). DPER routes token-level evidence between a frozen privileged teacher and a privileged student sharing the rollout policy's parameters, modulating reward-derived token advantages while preserving their signs. Training rewards combine generative process evaluation, multi-judge rankings, and task-queue consistency. JoyInside-VLM-Agent achieves a BFCL v4 overall score of 77.44, exceeding the evaluated baselines; results on the other public benchmarks are mixed. On JoyInside-Devices and JoyInside-Robotics, the evaluated system, which includes domain-specific skill refinement, leads the baselines on all reported metrics. Two real-world quadruped case studies illustrate visually grounded search, replanning after execution failure, and explicit failure reporting.
• 中文摘要 具身智能体必须将多模态观察和用户请求转化为可执行动作，同时适应产品约束、变化的目标和执行反馈。我们介绍JoyInside-VLM-Agent，一种类似ReAct风格的多模态智能体，通过产品条件轨迹构建和分阶段策略学习开发，面向机器人平台和智能硬件。闭环沙盒生成多模态交互轨迹，涵盖目标修订和失败恢复，并与执行结果和语义标准进行检验。监督式微调和多教师策略蒸馏先于使用DPER（双源特权策略自蒸馏及令牌级证据路由）进行强化学习。DPER在冻结的特权教师和共享推广策略参数的特权学生之间路由令牌级证据，调制奖励衍生的代币优势同时保持其符号。训练奖励结合了生成过程评估、多评审排名和任务队列一致性。JoyInside-VLM-Agent 在 BFCL v4 中获得了 77.44 的总体得分，超过了评估基线;其他公开基准测试的结果则参差不齐。在 JoyInside-Devices 和 JoyInside-Robotics 上，包含领域特定技能精炼的评估系统在所有报告指标上领先。两个真实世界的四足案例研究展示了基于视觉的搜索、执行失败后的重新规划以及显式失败报告。

Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence

广义近端策略优化：从发散视角统一LLM强化学习算法

• Authors: Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13015
• Pdf link: https://arxiv.org/pdf/2610.13015
• Abstract Reinforcement learning has become the foundational paradigm for enhancing the reasoning capabilities of LLMs. Yet the dominant approach, Proximal Policy Optimization (PPO), relies on a geometrically degenerate clipping proxy that cannot adapt its resistance to policy shifts, leading to premature clipping and degraded exploration. Subsequent variants alleviate these issues through fragmented and heuristic modifications. We propose Generalized Proximal Policy Optimization (GPPO), a principled and unified framework for understanding and designing LLM RL algorithms through the lens of divergence. We establish a generalized $f$-divergence trust-region theory with a monotonic performance improvement guarantee, and show that different $f$-divergences induce distinct geometries and policy update dynamics. We further introduce novel estimator topologies that provide more faithful divergence estimates than the conventional single-sample Monte Carlo estimation. Under this framework, PPO and recent variants such as DAPO, DPPO, and RIPO emerge as special cases, establishing a principled two-dimensional design space of {Divergence $\times$ Estimator}. Extensive experiments on eight competition-level math and coding benchmarks show that both dimensions are critical. The best-performing GPPO instantiation surpasses standard GRPO by up to 50% and 14% on math and coding, delivering SOTA performance.
• 中文摘要 强化学习已成为提升LLM推理能力的基础范式。然而，主流方法近端策略优化（PPO）依赖于几何退化的剪裁代理，无法适应对策略变化的抵抗，导致过早裁剪和探索退化。后续变体通过碎片化和启发式修改缓解了这些问题。我们提出了广义近端策略优化（GPPO），这是一个原则性且统一的框架，通过发散视角理解和设计LLM RL算法。我们建立了广义$f$散度信任区域理论，并保证单调性能提升，并证明不同$f$发散会引发不同的几何和策略更新动态。我们进一步引入了新颖的估计拓扑，提供比传统单样本蒙特卡洛估计更忠实的散度估计。在该框架下，PPO及近期变体如DAPO、DPPO和RIPO作为特例出现，建立了一个原则性的二维设计空间——{散度$\times$估计量}。对八个竞赛级数学和编码基准的广泛实验表明，这两个维度都至关重要。表现最佳的GPPO实例在数学和编码方面比标准GRPO高出50%和14%，实现了SOTA性能。

Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL

轨迹感知策略优化：弥合长期视野大型语言模型强化学习中的状态分布差距

• Authors: Zhicheng Cai, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.13020
• Pdf link: https://arxiv.org/pdf/2610.13020
• Abstract Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. Yet long-horizon autoregressive rollouts introduce a fundamental limitation: local policy mismatch can compound into severe state distribution shift over trajectories. Existing LLM RL methods, such as GRPO and GSPO, control policy mismatch at the token or sequence level, but overlook this trajectory-induced shift, causing training instability and eventual collapse as the horizon grows. To close the state distribution gap, we propose Trajectory-Aware Policy Optimization (TAPO), a principled framework that theoretically derives the state-distribution correction from the off-policy policy gradient and incorporates it into the RL objective. TAPO further employs a geometric-mean estimator for variance reduction, together with a length-aware dynamic clipping that prevents overly deviated updates across the trajectory. Extensive experiments demonstrate that TAPO significantly surpasses existing LLM RL algorithms across eight competition-level benchmarks, achieving up to 48% and 12% improvement over GRPO on mathematical reasoning and algorithmic coding, with substantially enhanced training stability.
• 中文摘要 强化学习（RL）已成为提升LLM推理能力的主导范式。然而，长视野自回归推广引入了一个根本局限：局部策略不匹配可能累积成轨迹上的严重状态分布偏移。现有的LLM RL方法，如GRPO和GSPO，在令牌或序列层面控制策略错配，但忽视了这一轨迹引起的偏移，导致训练不稳定，随着视野扩大最终崩溃。为弥合状态分布差距，我们提出了轨迹感知策略优化（TAPO），这是一种原则性框架，理论上从非策略策略梯度中推导状态分布修正，并将其纳入强化学习目标中。TAPO还采用几何均值估计器来减少方差，并结合长度感知动态剪裁，防止轨迹中过度偏离的更新。大量实验表明，TAPO在八个竞赛级基准测试中显著优于现有LLM强化学习算法，在数学推理和算法编码方面分别提升48%和12%，训练稳定性显著提升。

Evolving Creatives with Clicks: CTR-Driven Advertising image Editing via Attribute Disentanglement and Preference Alignment

通过点击演进创意：通过属性解开和偏好对齐实现点击率驱动的广告图片编辑

• Authors: Xu Zhang, Haohan Wang, Zheng Zhang, Yingjie Xia, Wei Feng, Honghe Zhu, Jingjing Lv, Changping Peng, Xuheng Qin, Zehua Zhang, Zheng Luo, Qicheng Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13066
• Pdf link: https://arxiv.org/pdf/2610.13066
• Abstract The optimization objective for e-commerce advertising creatives has shifted from aesthetic quality to online business metrics like click-through rate (CTR). Recent methods generate appealing ad images from product images by constructing CTR-based reward signals and refining prompt generation via Reinforcement Learning. Such paradigms equally penalize all attributes within an underperforming image, even though some are actually beneficial to clicks. In this work, we propose Evolving Creatives with Clicks (ECC), which reformulates creative optimization as fine-grained editing rather than generation from scratch. ECC selectively refines attributes that hurt CTR while preserving those that contribute positively. Specifically, faced with a rich attribute space across massive images, ECC first identifies 28 attributes significantly correlated with CTR via ANOVA and Random Forest regression. Based on attribute contrasts between images with disparate CTRs, we instruction-tune a prompt model to generate editing suggestions, then refine it with real click preferences since reference-derived suggestions are not uniformly beneficial. Furthermore, we introduce Consistent Preference Optimization (CPO) as ECC's preference optimization strategy, which enforces consistency across semantically equivalent shuffled suggestions to mitigate DPO overfitting and improve preference generalization. Extensive experiments demonstrate that ECC significantly improves the CTR of ad images, outperforming state-of-the-art models by a substantial margin. The complete data pipeline, including attribute extraction and analysis, the resulting dataset, and source code will be released upon acceptance.
• 中文摘要 电子商务广告创意的优化目标已从美学质量转向在线业务指标，如点击率（CTR）。近期方法通过构建基于点击率的奖励信号和通过强化学习优化提示生成，从产品图片生成吸引人的广告图像。此类范式同样惩罚表现不佳图片中的所有属性，尽管其中一些属性实际上对点击量有益。本研究提出“点击创意演化”（ECC），将创意优化重新表述为细粒度编辑，而非从零生成。ECC有选择地细化那些损害CTR的属性，同时保留那些积极贡献的属性。具体来说，面对庞大图像中丰富的属性空间，ECC首先通过方差分析和随机森林回归识别出28个与CTR显著相关的属性。基于CTR差异图像之间的属性对比，我们对提示模型进行指令调优以生成编辑建议，然后通过真实的点击偏好进行细化，因为引用衍生的建议并非均一有益。此外，我们引入了一致性偏好优化（CPO）作为ECC的偏好优化策略，该策略在语义等效的洗牌建议间强制一致性，以减轻DPO过拟合并改善偏好泛化。大量实验表明，ECC显著提升了广告图片的点击率，远远超过最先进的模型。完整的数据流程，包括属性提取和分析、所得数据集和源代码，将在接受后发布。

SWEAR: Section-Wise Efficient Audio Reasoning via Self-Rated Difficulty

SWEAR：通过自我评定的难度，按段落高效推理音频

• Authors: Yixuan Li, Minghao Lv, Xuenan Xu, Zihao Zheng, Mengyue Wu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13081
• Pdf link: https://arxiv.org/pdf/2610.13081
• Abstract Recent work on large audio-language models (LALMs) explores adaptive reasoning to balance accuracy and efficiency. However, existing approaches often rely on coarse-grained difficulty estimation, overlooking the diverse factors that determine reasoning effort. We address this limitation by deriving five-dimensional difficulty scores and compressed reasoning traces from complementary offline assessments. Based on this characterization, we propose SWEAR, a difficulty-aware reasoning framework that learns difficulty signals during supervised fine-tuning and uses reinforcement learning to adapt reasoning allocation across reasoning stages. Experiments on MMAU-test-mini and MMAR show that SWEAR improves accuracy while substantially reducing generated tokens, shortening reasoning for easy instances without sacrificing performance on difficult ones.
• 中文摘要 近期关于大型音频语言模型（LALMs）的研究探讨了自适应推理以平衡准确性与效率。然而，现有方法常依赖粗粒度难度估计，忽视了决定推理努力的多种因素。我们通过从补充的离线评估中推导出五维难度分数和压缩推理轨迹来解决这一局限性。基于这一特性，我们提出了SWEAR这一难度感知推理框架，该框架在监督微调过程中学习困难信号，并利用强化学习在推理阶段间调整推理分配。MMAU-test-mini和MMAR的实验表明，SWEAR在显著减少生成代币的同时提高了准确性，缩短了简单实例的推理时间，同时不牺牲困难实例的性能。

DP-MIND: Differentially private memory-guided Long context Inference

DP-MIND：差分私有记忆引导长上下文推断

• Authors: Zhihao Liu, Yitian Cheng, Shuang Liu, Rui Wen, Yuke Hu, Zhan Qin
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13132
• Pdf link: https://arxiv.org/pdf/2610.13132
• Abstract LLM-as-a-Service has become a dominant paradigm for accessing large language models without requiring local deployment. However, the process raises severe privacy concerns when user queries may include sensitive or confidential contexts. Differential privacy (DP) provides a principled approach to protecting such inputs, but existing DP inference methods are primarily designed for short- or moderate-context settings and struggle with million-token-scale inputs due to limited context capacity and severe utility degradation under DP perturbation. In this paper, we propose DP-MIND, a memory-guided framework that scales differentially private inference to million-token contexts. DP-MIND enables private long-context inference by locally perturbing the input and processing it chunk by chunk with a compact memory that preserves task-relevant information across the sequence. To make this memory robust to DP noise, we optimize the memory construction policy directly on perturbed inputs using reinforcement learning, guided by both final answer correctness and memory consistency. We further recover perturbation-corrupted answer spans through a local denoising module without exposing the original context to the external model. We provide theoretical privacy analysis of DP-MIND and further conduct extensive empirical analysis across multiple long-context benchmarks with context lengths up to the million-token scale, demonstrating that DP-MIND consistently outperforms existing DP and long-context inference baselines in both scalability and utility.
• 中文摘要 LLM即服务已成为访问大型语言模型而无需本地部署的主流范式。然而，当用户查询可能涉及敏感或机密上下文时，这一过程带来了严重的隐私隐患。差分隐私（DP）提供了一种原则性的方法来保护此类输入，但现有的DP推理方法主要为短或中等上下文设置设计，由于上下文容量有限且DP扰动下效用严重下降，难以应对百万令牌级输入。本文提出DP-MIND，一种内存引导框架，将私有推理以差异化程度扩展至百万令牌上下文。DP-MIND通过局部扰动输入并逐块处理，实现私密长上下文推理，紧凑内存保持任务相关信息。为了使该内存对DP噪声具有鲁棒性，我们直接利用强化学习优化受扰输入的内存构建策略，并结合最终答案的正确性和内存一致性。我们还通过局部去噪模块恢复受扰动损坏的答案跨度，而无需将原始上下文暴露给外部模型。我们对DP-MIND进行了理论隐私分析，并进一步对多个长上下文基准进行了广泛的实证分析，上下文长度可达百万代币尺度，证明DP-MIND在可扩展性和实用性上始终优于现有的DP和长上下文推断基线。

SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agents

SOAR：源觉强化学习与搜索代理的精确学分赋值

• Authors: Shengze Huang, Zihao Tang, Jinluan Yang, Xin Yu, Zhenghao Lin, Xiao Liu, Zengxuan Wen, Yeyun Gong, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Fei Wu, Kun Kuang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.13157
• Pdf link: https://arxiv.org/pdf/2610.13157
• Abstract Reinforcement learning (RL) for search agents often suffers from evidence fabrication, a failure mode where agents bypass external information ($E_{ext}$) and fabricate an incorrect answer. We argue that this fabrication arises from an over-reliance on unreliable internal knowledge ($E_{int}$) as shortcuts due to the inherent opacity of reasoning trajectories. To address this, we propose SOAR (SOurce-Aware Reinforcement learning), a paradigm that transforms monolithic reasoning into source-explicit trajectories. SOAR (i) disentangles $E_{ext}$ and $E_{int}$ into parallel, observable streams via a Source-Aware Workflow, and (ii) incorporates Source-Consistent Reward to guide the policy evolution. By making information-source attribution explicit, SOAR enables precise credit assignment between different knowledge streams. Evaluations show that SOAR outperforms baselines on 3B and 7B backbones by 7.3% and 5.4%, while reducing evidence fabrication by 74.8%. These confirm that SOAR significantly improves performance on knowledge-intensive benchmarks while effectively mitigating evidence fabrication, fostering more faithful and grounded reasoning with enhanced robustness.
• 中文摘要 搜索智能体的强化学习（RL）常常存在证据造假（evidence fabrication）的问题，即代理绕过外部信息（$E_{ext}$）并伪造错误答案。我们认为这种虚构源于过度依赖不可靠的内部知识（$E_{int}$）作为捷径，这源于推理轨迹本身的不透明性。为此，我们提出了SOAR（超感知强化学习）范式，将单一推理转化为源显式轨迹。SOAR（i）通过源感知工作流将$E_{ext}$和$E_{int}$解开为并行的可观察流，并（ii）引入源一致性奖励指导政策演进。通过明确化信息源归因，SOAR实现了不同知识流间精确的功劳分配。评估显示，SOAR在3B和7B骨干链上分别优于基线7.3%和5.4%，同时减少了74.8%的证据伪造。这些评估证实，SOAR在知识密集型基准测试中显著提升了性能，同时有效减少证据伪造，促进更忠实、更扎实的推理，增强了稳健性。

AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learning

AdaC-GRPO：通过自适应课程学习掌握远景视觉数独谜题

• Authors: Sida Li, Yuanxin Liu, Linli Yao, Shicheng Li, Xu Sun, Haoli Bai, Xiaohui Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13184
• Pdf link: https://arxiv.org/pdf/2610.13184
• Abstract While reinforcement learning with verifiable reward (RLVR) enhances reasoning in large language models, its efficacy in long-horizon multimodal tasks remains underexplored. Visual Sudoku puzzles pose a representative challenge, requiring multi-step deductive reasoning over both local constraints and global dependency chains. To this end, we introduce FG-Sudoku, a fine-grained benchmark categorized by grid size and required deductive steps. Evaluations reveal that frontier MLLMs fail on complex long-range dependencies (e.g., 9*9 puzzles). During post-training, this near-zero success rate causes severe reward sparsity, halting standard RLVR. We propose Adaptive Curriculum Group Relative Policy Optimization (AdaC-GRPO) to overcome this. Unlike rigid or prior-based schedules, AdaC-GRPO dynamically constructs training batches by tracking the model's "Zone of Proximal Development" via online rollout accuracy. Experiments demonstrate that AdaC-GRPO enables Qwen2.5-VL-7B to approach the performance of Gemini-2.5-Flash while bridging 62.2% of the overall accuracy gap to Gemini-2.5-Pro, with its generalizability further validated across InternVL-3.5 and Qwen3-VL.
• 中文摘要 虽然带有可验证奖励的强化学习（RLVR）提升了大型语言模型中的推理能力，但其在长视野多模态任务中的有效性仍未被充分探索。视觉数独谜题具有代表性，需要在局部约束和全局依赖链上进行多步演绎推理。为此，我们引入了FG-数独，这是一个按网格大小和所需演绎步骤分类的细粒度基准。评估显示，前沿MLLM在复杂的长距离依赖（如9*9谜题）上失败。在训练后期，这种近乎零的成功率导致严重的奖励稀疏，导致标准RLVR停滞。我们提出了自适应课程组相对策略优化（AdaC-GRPO）来克服这一问题。与基于先验的僵化计划不同，AdaC-GRPO通过在线部署精度追踪模型的“近端发展区”动态构建训练批次。实验表明，AdaC-GRPO使Qwen2.5-VL-7B能够接近Gemini-2.5-Flash的性能，同时弥补了与Gemini-2.5-Pro62.2%的整体精度差距，其普遍性在InternVL-3.5和Qwen3-VL中得到了进一步验证。

Does Learning Recurrent Networks Matter in Partially Observable RL?

学习循环网络在部分可观察的强化学习中重要吗？

• Authors: Zekang Wang, Zhe He, Gaojie Jin
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13197
• Pdf link: https://arxiv.org/pdf/2610.13197
• Abstract Reinforcement learning (RL) under partial observability typically re- lies on recurrent networks to encode temporal information, with the recurrent module trained jointly with the policy or value function. This raises a natural question: is learning the recurrent network actually necessary? In this paper, we investigate whether a frozen recurrent network can provide sufficient memory for solving par- tially observable Markov decision processes (POMDPs). Motivated by the theoretical foundations of Reservoir Computing (RC) and Generalized Reservoir Computing (GRC), we study a simple archi- tecture that combines a fixed recurrent network with a trainable nonlinear output module. We evaluate this approach across tasks with varying memory demands, observation modalities, and action spaces. Surprisingly, frozen recurrent networks support recall over hundreds of steps and achieve competitive control performance across diverse POMDPs, while using less than half the number of trainable parameters required by their fully trainable counterparts. These results suggest that learning the recurrent memory module is not always necessary for solving POMDPs. Frozen recurrent networks therefore provide a simple, parameter-efficient, and sur- prisingly strong baseline for partially observable RL. Our source code is available at this https URL
• 中文摘要 在部分可观测性下，强化学习（RL）通常依赖循环网络来编码时间信息，循环模块与策略或价值函数共同训练。这自然引发了一个问题：学习循环网络真的有必要吗？本文探讨了冻结循环网络是否能为解决局部可观测的马尔可夫决策过程（POMDP）提供足够的内存。受储量计算（RC）和广义蓄积计算（GRC）理论基础的启发，我们研究了一个简单结构，将固定的循环网络与可训练的非线性输出模块结合起来。我们评估了该方法在不同内存需求、观察模式和动作空间的任务中进行评估。令人惊讶的是，冻结循环网络支持数百步的回忆，并在不同POMDP中实现了竞争性的控制性能，同时所需训练参数数量不到完全可训练网络的一半。这些结果表明，学习循环记忆模块并非解决POMDP的必要条件。因此，冻结循环网络为部分可观测的强化学习提供了一个简单、参数高效且出人意料地强大的基线。我们的源代码可在此 https URL 获取

S$^3$R: Fast and Faithful RL Rollouts via Self-Sparse Speculative Decoding

S$^3$R：通过自我稀疏的推测性解码实现快速且忠实的强化学习推广

• Authors: Yuhang He, Hongyu Ge, Zihan Lin, Siyi Liu, Yongqi Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.13202
• Pdf link: https://arxiv.org/pdf/2610.13202
• Abstract Fast rollout generation for long-context reinforcement learning with verifiable rewards (RLVR) requires reducing KV-cache traffic without changing the response groups used for policy optimization. In our profiled RLVR workloads, attention accounts for 72--76\% of autoregressive decoding kernel time. Sparse attention reduces these reads, but sampling directly from a sparse policy changes the response groups used for learning. For group-relative optimization, we show that even exact response-wise reweighting does not generally recover the expected dense-policy update because the advantage of each response depends on the rewards of other group members. We propose Self-Sparse Speculative Rollouts (S$^3$R). It uses sparse attention to propose tokens, then generates responses through full-attention verification and exact speculative sampling. Drafting and verification share the same policy parameter, eliminating the need to train or update a separate drafter. The resulting response groups follow the dense-policy distribution. Under S$^3$R, we characterize speedup conditions and the costs of drafting and verification. We found that dense verification still requires the complete historical KV cache. A KV-aware scheduler therefore limits KV-cache repeated offloading and restoration while maintaining rollout parallelism. % OR thrashing In a matched ablation with an aggressive sparse drafting budget, S$^3$R maintains stable training while direct sparse sampling with training-side correction becomes unstable. On long-context mathematical reasoning, S$^3$R achieves $1.39\times~1.35\times$ the end-to-end training throughput of autoregressive rollouts on Qwen3 model, with comparable task performance.
• 中文摘要 用于带可验证奖励的长上下文强化学习（RLVR）快速生成，需要在不更改策略优化响应组的情况下减少KV缓存流量。在我们剖析的RLVR工作负载中，注意力占自回归解码核心时间的72%-76%。注意力稀疏会减少这些读段，但直接从稀疏策略中抽样会改变用于学习的响应组。对于组相对优化，我们证明即使是精确的响应加权，通常也无法恢复预期的稠密策略更新，因为每个响应的优势取决于其他组成员的奖励。我们提出了自稀疏推测性展开（S$^3$R）。它利用稀疏关注来提出代币，然后通过全注意力验证和精确推测抽样生成响应。草拟和验证共享相同的策略参数，无需训练或更新单独的起草者。所得响应组遵循稠密策略分布。在 S$^3$R 下，我们描述了加速条件以及绘制和验证的成本。我们发现密集验证仍需完整的历史 KV 缓存。因此，KV 感知调度器限制了 KV 缓存的重复卸载和恢复，同时保持展开并行性。% OR 扰乱在匹配消融且具有激进稀疏绘图预算的条件下，S$^3$R 保持稳定训练，而直接稀疏采样并训练侧修正则变得不稳定。在长上下文数学推理下，S$^3$R 在 Qwen3 模型上实现自回归展开端到端训练吞吐量 $1.39\times~1.35\times$，任务性能相当。

Rethinking Relative Policy Differences Transfer of On-Policy Distillation

重新思考相对政策差异 政策内提炼的转移

• Authors: Rong Wu, Daocheng Fu, Yu Yang, Xuemeng Yang, Siqi Li, Shu Zou, Yao Zhu, Botian Shi, Bo Zhang, Cong Zhang, Pinlong Cai, Jianbiao Mei
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.13205
• Pdf link: https://arxiv.org/pdf/2610.13205
• Abstract On-policy distillation (OPD) trains a target policy on its own rollouts using token-level supervision from a stronger teacher, but typically relies on the teacher being substantially more capable than the target. Recent work relaxes this requirement by transferring the relative policy difference between an expert and a reference policy rather than the expert's absolute distribution. Yet it remains unclear which differences provide useful supervision and which targets can benefit from them. We develop a unified view in which the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization. This formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and enables any-to-any transfer across models that differ in scale, training stage or generation. Our systematic experiments yield three findings. First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and reference serve complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and generalize beyond sampled rollouts. Raw policy distance does not predict transferability, whereas an appropriate reference can rescue otherwise unsuccessful transfer. These results characterize when relative policy differences provide effective supervision for OPD.
• 中文摘要 策略上提炼（OPD）通过更强教师的代币级监督，在自身推广中训练目标策略，但通常依赖教师的能力远超目标。近期研究通过转移专家与参考策略之间的相对政策差异而非专家的绝对分布，放宽了这一要求。然而，哪些差异提供了有用的监督，哪些目标能从中受益，仍不明确。我们发展出一个统一观点，即代理专家与代理参考之间的对数概率比定义了对KL正则化策略优化的隐性奖励。该表述诱导目标的策略上更新，恢复传统OPD作为特例，并实现跨规模、训练阶段或生成不同模型的任一转移。我们的系统实验得出三项发现。首先，更强的专家不一定是更好的监督来源：由任务特定训练引起的差异，包括强化学习和监督微调，能够持续传递，而仅由模型规模产生的差异则传递得很弱甚至根本不传递。其次，专家和参考人扮演互补角色：专家提供转移能力，而参考人过滤共享行为并隔离相关残差。第三，即使是有用的差异也不会转移到所有目标：成功需要诱导的更新将目标向专家方向移动，并在抽样推广之外推广。原始政策距离无法预测可转移性，而合适的参考可以拯救原本不成功的转移。这些结果决定了相对政策差异何时能有效监督OPD。

OWLET: On-Policy Weighted Learning via Evidence Transfer for Video Large Language Model Post-Training

OWLET：基于策略的证据转移加权学习，适用于视频大型语言模型后训练

• Authors: Qitan Lv, Ke Fan, Hong Wang, Wen Wu, Yilin Wang, Bowen Zhou, Feng Wu, Chao Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.13272
• Pdf link: https://arxiv.org/pdf/2610.13272
• Abstract Post-training has become a key stage for eliciting reasoning capabilities in Video Large Language Models (Video-LLMs). Existing post-training methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, they do not fully exploit visual evidence for effective training. RL supervises the policy with a single trajectory-level reward, which is too coarse-grained for video reasoning: the same response-level advantage is applied to every generated token, disregarding that the tokens of a video response are grounded in the visual stream to very different degrees. OPD provides token-level supervision through output-distribution matching, but lacks direct supervision of the spatiotemporal representations that produce those outputs. We observe across extensive experiments that the same model reasons more accurately when its video view combines global temporal context with question-relevant details. Comparing the same student-generated response under this view and uniform sampling reveals changes in token likelihoods and internal representations, providing a training signal for evidence use. Building on this observation, we propose OWLET, an evidence-privileged RL method that derives training signals from a richer teacher view. This view couples uniform global context with question-conditioned local windows, providing visual evidence without requiring a larger model. To better exploit this privileged visual evidence, OWLET introduces (i) \textbf{evidence-aware credit redistribution}, which compares token likelihoods across the two views and redistributes positive response-level advantages according to their relative evidence support; and (ii) \textbf{spatiotemporal state alignment}, which aligns the two views' hidden states on verifier-accepted responses.
• 中文摘要 后训练已成为视频大型语言模型（Video-LLMs）中引发推理能力的关键阶段。现有的后训练方法主要遵循两种范式：强化学习（RL）和策略内提炼（OPD）。然而，它们并未充分利用视觉证据进行有效训练。强化学习通过单一轨迹级奖励来监督策略，这种奖励过于粗粒度，不适合视频推理：每个生成的代币都被赋予相同的响应层优势，而忽略视频反应的代币在视觉流中的根基程度差异很大。OPD通过输出-分布匹配提供代币级监督，但缺乏对产生这些输出的时空表征的直接监督。我们通过大量实验观察到，当同一模型的视频视图结合全局时间上下文和问题相关细节时，推理更为准确。在该视角下比较同一学生生成的回答与均匀抽样，揭示了代币似然和内部表征的变化，为证据使用提供了训练信号。基于这一观察，我们提出了OWLET，一种基于证据特权的强化学习方法，从更丰富的教师视角中推导出训练信号。该视图将统一的全局上下文与问题条件的局部窗口结合，提供视觉证据而无需更大模型。为更好地利用这一特权视觉证据，OWLET引入了（i） \textbf{证据感知的信用再分配}，比较两种视图的代币似然，并根据相对证据支持重新分配正面反应水平优势;以及（ii） \textbf{时空状态对齐}，将两种视图的隐藏状态对齐于验证者接受的回答。

MORAD: Reinforcement Learning with Multi-Objective Rewards for RNA Inverse Design

MORAD：RNA逆向设计的多目标奖励强化学习

• Authors: Jixin Tang, Ji Guo, Jun Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Biomolecules (q-bio.BM)
• Arxiv link: https://arxiv.org/abs/2610.13305
• Pdf link: https://arxiv.org/pdf/2610.13305
• Abstract We introduce MORAD, an online reinforcement learning framework for backbone-conditioned RNA diffusion models that trains a single shared policy across diverse target backbones. Structure-conditioned RNA inverse design seeks nucleotide sequences compatible with a target three-dimensional backbone, yet existing reinforcement learning approaches either optimize a separate policy for each target backbone or rely on offline preference optimization, leaving online post-training of a reusable backbone-conditioned model largely unexplored. MORAD combines backbone-conditioned group sampling, group-relative reward normalization, and a forward-process diffusion objective. Beyond existing RL rewards that primarily target tertiary structural quality, MORAD jointly optimizes six measurements spanning tertiary geometry, base pairing, ensemble behavior, and sequence composition. To aggregate these heterogeneous objectives despite differences in scale and optimization direction, MORAD maps each measurement to a bounded desirability score and combines them with a weighted geometric mean. Compared with a tertiary-structure-only reward, this multi-objective formulation improves 10 of the 11 evaluation metrics that have a preferred direction. Applied to pretrained RIDE, MORAD improves GDT-TS by 16.0% (from 0.3300 to 0.3827) and pairing MCC by 19.4% (from 0.6110 to 0.7295), while reducing normalized ensemble defect by 27.5% (from 0.3906 to 0.2830). On the test set, MORAD outperforms RiboDiffusion, gRNAde, and RDesign across tertiary structural accuracy, secondary-structure pairing, and thermodynamic quality, establishing a new state of the art in structure-conditioned RNA inverse design.
• 中文摘要 我们介绍了MORAD，一种在线强化学习框架，用于骨架条件RNA扩散模型，训练单一共享策略，跨越不同目标骨架。结构条件RNA逆设计寻求与靶三维骨架兼容的核苷酸序列，但现有强化学习方法要么为每个目标骨架优化独立策略，要么依赖离线偏好优化，使得可重复使用骨架条件模型的在线后训练在很大程度上未被探索。MORAD结合了骨架条件群抽样、群相对奖赏规范化和前向过程扩散目标。除了现有主要针对三级结构质量的强化学习奖励外，MORAD还联合优化了涵盖三级几何、碱基配对、集合行为和序列组成的六项测量。为了聚合这些异质目标，尽管尺度和优化方向不同，MORAD将每个测量映射到有界的期望性评分，并与加权几何平均结合。与仅三级结构奖励相比，该多目标表述改进了11个有偏好方向的评估指标中的10个。应用于预训练RIDE，MORAD使GDT-TS提升16.0%（从0.3300提升至0.3827），配对MCC提升19.4%（从0.6110降至0.7295），同时将归一化集合缺陷减少27.5%（从0.3906降至0.2830）。在测试组上，MORAD在三级结构精度、二级结构配对和热力学质量方面优于核扩散、gRNAde和RDesign，在结构条件RNA逆设计领域开创了新潮流。

AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents

AgentTune：工具使用LLM代理的统一训练生命周期

• Authors: R E Zera Marveen Lyngkhoi, Abhivansh Gupta, Vidushee Vats, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.13335
• Pdf link: https://arxiv.org/pdf/2610.13335
• Abstract Tool-using agents increasingly require a common substrate for building, training, evaluating, distilling, and maintaining agentic systems, yet existing tooling typically addresses these stages in isolation. We present AgentTune, a source-available Python library that provides a unified framework for developing and optimizing tool-using agents. At its core, AgentTune introduces a two-tier EventLog representation that preserves agent trajectories while supporting consistent projections for training, evaluation, and auditing. Building on this substrate, the library provides a shared agent runtime, tool-calling reinforcement learning, agentic distillation, retrieval-augmented generation, decision workflows, evaluation utilities, and a designed path toward self-healing agents. These components are exposed through composable interfaces so that agent strategies, training methods, evaluation procedures, and recovery mechanisms can be combined without requiring separate representations or pipelines. We demonstrate the framework across agentic tasks including legal contract search and domain-specific applications, illustrating its use for trajectory-based optimization, tool-use evaluation, and cross-task agent development. AgentTune is released as a source-available library at this https URL
• 中文摘要 使用工具的代理越来越需要一个共同的基底来构建、训练、评估、提炼和维护代理系统，但现有工具通常单独处理这些阶段。我们介绍AgentTune，一个可获得源代码的Python库，提供了一个统一框架，用于开发和优化工具使用代理。AgentTune的核心是引入两层事件日志表示，既保留了代理轨迹，又支持训练、评估和审计的一致预测。基于该基底，库提供共享代理运行时、工具调用强化学习、代理提纯、检索增强生成、决策工作流、评估工具以及通往自愈代理的设计路径。这些组件通过可组合接口曝光，使代理策略、训练方法、评估程序和恢复机制能够组合，而无需单独表示或流水线。我们展示了该框架在包括法律合同搜索和领域特定应用在内的代理任务中，展示了其在轨迹优化、工具使用评估和跨任务代理开发中的应用。AgentTune 作为源代码库发布，链接为 https URL

Evo-RL: Iterative Robot Reinforcement Learning from Selective Experience

Evo-RL：基于选择性经验的迭代机器人强化学习

• Authors: Kai Ye, Yifan Jin, Qianhang Luo, Hongru Zhang, Jiedong Zhang, Shuyuan Hu, Song-Yuan Wu, Jihai Xu, Jie Zhao, Bo Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.13345
• Pdf link: https://arxiv.org/pdf/2610.13345
• Abstract Existing real-world reinforcement learning methods improve robot policies by combining human corrections with autonomous rollouts. However, accumulated experience varies in its value for policy improvement, and even successful trajectories can contain inefficient or redundant behavior, motivating more selective experience reuse. To this end, we propose Evo-RL, an iterative reinforcement learning method that combines human interventions with advantage-guided experience selection. Evo-RL uses human corrections and experience with positive estimated advantages from successful trajectories to iteratively refine the policy. To support learning from these different experience sources, we introduce RW-RL-Dataset, which spans over 1,000 hours across four robot embodiments. On a real-world high-precision manipulation task, Evo-RL improves attempt success rate over RECAP by 30.46\%. Our code, model checkpoints, and dataset are publicly available.
• 中文摘要 现有的现实世界强化学习方法通过将人类修正与自主推广相结合，改进机器人策略。然而，积累的经验对策略改进的价值各不相同，即使是成功的轨迹也可能包含低效或冗余行为，促使更多选择性地重复使用经验。为此，我们提出了Evo-RL，一种将人类干预与优势引导经验选择相结合的迭代强化学习方法。Evo-RL利用人类修正和经验，并从成功轨迹中获得积极估计的优势，通过迭代细化策略。为支持从这些不同经验来源学习，我们引入了RW-RL数据集，涵盖四个机器人实例，时长超过1000小时。在现实世界的高精度操作任务中，Evo-RL比RECAP的尝试成功率提高了30.46%。我们的代码、模型检查点和数据集均公开。

EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning

EmoGavel：多模态情绪偏好学习的可验证奖励的接地感知强化学习

• Authors: Zilong Huang, Kong Aik Lee, Zezhong Jin, Man-Wai Mak, Tatsuya Kawahara
• Subjects: Subjects: Multimedia (cs.MM); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2610.13353
• Pdf link: https://arxiv.org/pdf/2610.13353
• Abstract Multimodal emotion preference learning uses pairwise comparisons to identify which open-ended emotion descriptions align more closely with human judgments. However, existing methods primarily focus on the final preference decision and provide little explicit supervision for localizing and correcting erroneous content. To address this limitation, we propose EmoGavel, a Grounding-Aware reinforcement learning framework with Verifiable rewards for multimodal Emotion preference Learning. We first introduce grounded supervised fine-tuning (Grounded SFT), which exploits controlled-error construction information to jointly learn preference prediction, error localization, repair, and missing-evidence recovery. We then optimize these verifiable outcomes with grounding-aware group relative policy optimization (GA-GRPO), combining preference-gated grounding rewards with dynamic zero-variance rescue to prioritize preference correctness and recover learning signals from all-wrong groups. Experiments on EmoPrefer and controlled-error evaluation sets show that EmoGavel improves both natural preference prediction and controlled-error discrimination while maintaining strong error-grounding performance. Our code and model checkpoints will be released once the paper is accepted.
• 中文摘要 多模态情绪偏好学习通过配对比较来识别哪些开放式情绪描述更贴近人类判断。然而，现有方法主要关注最终偏好决策，且对错误内容的定位和纠正缺乏明确监督。为解决这一限制，我们提出了EmoGavel，这是一个基于基础感知强化学习框架，并对多模态情绪偏好学习提供可验证奖励。我们首先引入了基于基础监督的微调（Grounded SFT），利用受控错误构造信息共同学习偏好预测、错误定位、修复和缺失证据恢复。随后，我们通过基础感知群体相对策略优化（GA-GRPO）优化这些可验证结果，将偏好门槛的接地奖励与动态零方差救援结合，优先排序偏好正确性并从错误组中恢复学习信号。在EmoPrefer和受控误差评估集上的实验表明，EmoGavel在保持强劲的错误接地性能的同时，不仅提升了自然偏好预测和受控错误辨别能力。论文被接受后，我们的代码和模型检查点将发布。

SWE-Critix: Post-Training Generalist Verifiers for Assessing Coding-Agent Trajectories

SWE-Critix：培训后通用性验证工具，用于评估编码代理轨迹

• Authors: Jiajun Hu, Chun Yong Chong
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13356
• Pdf link: https://arxiv.org/pdf/2610.13356
• Abstract Coding agents are increasingly deployed for end-to-end software engineering tasks, where a readily available oracle evaluator that defines the success criteria is often absent. This motivates verifications for coding agents that can reliably judge and explain their behavior. However, verification is non-trivial as coding agents are increasingly capable of solving long-horizon software engineering tasks. Their execution trajectories can be extremely long, containing hundreds of iterative interactions with code repositories and external tools. Existing verifiers either rely on general-purpose LLMs without specialized training or train dedicated critics that predominantly focus on producing scalar rewards, highlighting the relatively limited exploration of training-based generalist verification. We introduce SWE-Critix, a post-training pipeline for training generalist verifiers of coding-agent trajectories. Given a task and its execution trajectory, the learned verifier predicts the completion label and generates a natural-language analysis that explains its judgment. Our pipeline combines supervised fine-tuning (SFT) with reinforcement learning (RL) and trains a series of 30B models on trajectories from CodeForge-Preview. On a held-out test set, SWE-Critix improves classification accuracy from 57% for the base Qwen3-30B-A3B reasoning model to 66% after SFT and 78% after RL. We release our models, datasets, and training code to facilitate further research.
• 中文摘要 编码代理越来越多地被用于端到端的软件工程任务，而这些任务往往缺乏现成的oracle评估器来定义成功标准。这激励了能够可靠判断和解释其行为的编码代理进行验证。然而，验证并非简单，因为编码代理越来越具备解决长期软件工程任务的能力。它们的执行轨迹可能非常漫长，包含数百次与代码仓库和外部工具的迭代交互。现有验证器要么依赖没有专业培训的通用大型语言模型，要么训练专注于产生标量奖励的专职批评者，凸显了基于训练的通用验证探索相对有限。我们介绍SWE-Critix，这是一个用于培训编码代理轨迹通用验证者的后培训流程。给定任务及其执行轨迹，学习验证者预测完成标签，并生成解释其判断的自然语言分析。我们的流程结合了监督微调（SFT）和强化学习（RL），并基于CodeForge-Preview的轨迹训练一系列30B模型。在保留测试集上，SWE-Critix将分类准确率从基础Qwen3-30B-A3B推理模型的57%提升至SFT后的66%和强化学习后的78%。我们发布模型、数据集和训练代码，以促进进一步研究。

Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

推广时间干预：解锁超越策略限制的代理强化学习

• Authors: Chen Xi, Ding Ruixue, Wang Zhigaoyuan, Zhang Fanrui, Zhang Qiang, Chen Boli, Wang Shihang, Wang Tianfu, Ren Ailin, Wang Pengyong, Xie Pengjun, Xiong Hui
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13438
• Pdf link: https://arxiv.org/pdf/2610.13438
• Abstract Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model's current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at this https URL.
• 中文摘要 强化学习（RL）已成为提升大型语言模型（LLM）代理的关键范式，使得在交互环境中更强的推理、规划和工具使用。然而，现有的代理强化学习方法仍受制于策略推理瓶颈：当任务超出模型当前能力边界时，有效轨迹变得极为稀疏，训练常常崩溃。我们引入了推广时间干预（RTI），这是一种旨在引出超越策略内在探索能力的成功轨迹的技术。RTI通过在推广过程中利用奖励侧证据（即评估评分标准）将错误轨迹转化为成功轨迹，为策略本身很少解决的任务提供有效的学习信号。尽管干预轨迹本质上偏离策略，但当过滤掉分布偏移过大的轨迹时，干预轨迹可以有效指导策略优化。五个基准测试的广泛实验表明，RTI始终优于强化学习基线，使后训练策略能够接近更强的当代大型语言模型的性能，更重要的是超越其原始能力边界。代码可在此 https URL 获取。

Collective Autonomous Vehicle Routing Needs Social Awareness for Coexistence with Human Drivers

集体自动驾驶车辆路线规划需要社会意识以实现与人类驾驶者的共存

• Authors: Anastasia Psarou, Mikolaj Rams, Blazej Torbus, Rafal Kucharski
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13439
• Pdf link: https://arxiv.org/pdf/2610.13439
• Abstract Collective Autonomous Vehicle (AV) route choices in real-world traffic networks can create externalities for Human-Driven Vehicles (HDVs), as routes that benefit AVs may increase travel times for other road users. To address this, we introduce a social component, the marginal cost, to the selfish travel time AV reward. Marginal cost captures an AV's impact on other road users by comparing their travel times with those in a counterfactual simulation in which the AV is removed. We study AV route choice in mixed-autonomy traffic settings using Multi-Agent Reinforcement Learning (MARL), where AVs learn routing policies through repeated interactions with the traffic environment. Experiments on the real-world networks of Ingolstadt and Provins show that incorporating marginal cost into the AV reward not only limits the increase in HDV travel times and shortens AV travel times relative to selfish AV routing, but also improves overall system performance. In Ingolstadt, selfish routing of 40\% AVs increases the travel times of other vehicles by 1.7\%, and this drops to 1.1\% when the social reward is included. Interestingly, not only does overall system performance improve, but the AVs are also better off when acting this http URL effect is consistent across tested cities, penetration rates, and replications, yet when the weight of the social component in the reward increases, the system fails to reach the desired performance. For a more scalable approach to social AV routing in mixed-autonomy traffic, we compute the marginal cost only for the AVs whose route choices most affect the traffic system. This reduces the number of counterfactual simulations and achieves similar performance at a significantly lower computational cost. Our results suggest that AV routing should go beyond selfish rewards and, for the benefit of all traffic participants, include a social component in its objective.
• 中文摘要 在现实世界交通网络中，集体自动驾驶车辆（AV）路线选择可能为人驾驶车辆（HDV）创造外部性，因为有利于自动驾驶车辆的路线可能会增加其他道路使用者的行车时间。为此，我们在自私的自动驾驶时间奖励中引入了边际成本这一社会成分。边际成本通过比较自动驾驶车辆与其他道路使用者的行驶时间与反事实模拟中移除自动驾驶车辆时的行驶时间，来捕捉自动驾驶汽车对其他道路用户的影响。我们利用多智能体强化学习（MARL）研究了混合自动驾驶交通环境中自动驾驶车辆路线选择，该技术下自动驾驶通过与交通环境的反复交互学习路线策略。在英戈尔施塔特和普罗文斯的真实世界网络上的实验表明，将边际成本纳入自动驾驶奖励不仅限制了HDV行驶时间的增加，缩短了自动驾驶车辆相对于自私自动驾驶路线的行驶时间，还提升了整体系统性能。在英戈尔施塔特，40%的自私路由杀毒车会使其他车辆的行驶时间增加1.7%，而当包含社交奖励后，这一比例降至1.1%。有趣的是，不仅整体系统性能有所提升，当这种http URL效应在测试城市、渗透率和复制中保持一致时，杀毒软件的表现也更好;然而，当奖励中社会成分的权重增加时，系统却未能达到预期性能。为了在混合自治交通中实现更具可扩展性的社交杀毒软件路由方法，我们只计算路径选择对交通系统影响最大的杀毒软件的边际成本。这减少了反事实模拟的数量，并以显著较低的计算成本实现了类似的性能。我们的结果表明，杀毒软件路由应超越自私的奖励，为了所有交通参与者的利益，应将社会成分纳入其目标。

Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving

Reflect-VLA：通过双路径政策自提纯内化反思引导，实现自动驾驶

• Authors: Yuechen Luo, He Chong, Fang Li, Shaoqing Xu, Hexin Zhang, Ruiqi Zhang, Ziying Song, Lei Yang, Zhi-Xin Yang, Fuxi Wen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.13453
• Pdf link: https://arxiv.org/pdf/2610.13453
• Abstract Reinforcement Learning (RL) substantially improves planning performance in autonomous-driving Vision-Language-Action (VLA) models. However, existing RL methods rely on scalar trajectory rewards that measure overall quality but fail to identify specific planning errors or provide actionable correction signals. In long-tail scenarios, errors in scene understanding, behavior selection, and trajectory execution are compressed into a single score. We observe that Reflective Guidance turns trajectory evaluation into explicit diagnoses and corrective instructions, substantially improving planning accuracy. Yet such guidance requires an initial prediction and its evaluation results, making it unavailable for direct single-pass inference. We propose Reflect-VLA, which treats Reflective Guidance as training-time privileged information and internalizes its planning benefits through Dual-Path On-Policy Self-Distillation (Dual-OPSD). Unlike scalar-reward optimization, Dual-OPSD uses the driving score only to route student-generated trajectories. It distills dense token-level corrective distributions from the teacher for risky trajectories while applying self-imitation to preserve reliable ones. To provide reliable corrective distributions on student rollouts, the teacher is progressively constructed through Driving-Knowledge Pretraining, Planning-and-Refinement SFT, and Self-Refinement Rejection Sampling. At inference time, Reflect-VLA generates trajectories in a single pass without explicit guidance or iterative refinement. Experiments on NAVSIM v1/v2 and Bench2Drive demonstrate competitive planning performance while reducing training time by over 90% compared with RL.
• 中文摘要 强化学习（RL）显著提升了自主驾驶视觉-语言-行动（VLA）模型中的规划性能。然而，现有的强化学习方法依赖标量轨迹奖励来衡量整体质量，但未能识别具体的规划错误或提供可操作的纠正信号。在长尾场景中，场景理解、行为选择和轨迹执行的错误被压缩为一个评分。我们观察到，反思引导将轨迹评估转化为明确的诊断和纠正指令，显著提升了规划准确性。然而，这种指导需要初始预测及其评估结果，因此无法直接进行单次推断。我们提出了Reflect-VLA，它将反思引导视为训练时间的特权信息，并通过双路径策略自我蒸馏（Dual-OPSD）内化其规划收益。与标量奖励优化不同，Dual-OPSD仅使用驾驶分数来规划学生生成的轨迹。它从教师那里提取了高风险轨迹的密集代币级纠正分布，同时通过自我模仿保持可靠轨迹。为了在学生推广中提供可靠的纠正分布，教师通过驾驶-知识预训练、规划与精炼SFT和自我精炼拒绝采样逐步构建。在推断阶段，Reflect-VLA在一次无明确指导或迭代细化的情况下生成轨迹。NAVSIM v1/v2和Bench2Drive的实验展示了竞争规划性能，同时相比强化学习将培训时间缩短超过90%。

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

超越时间接近：强化学习中学分分配的解释性世界模型

• Authors: Oriol Corcoll, Qiyao Wei, Ciarán Mark Gilligan-Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2610.13504
• Pdf link: https://arxiv.org/pdf/2610.13504
• Abstract Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.
• 中文摘要 生成推荐者越来越多地使用强化学习（RL）来优化可能在决策后出现数百甚至数千次交互的结果。标准RL根据时间接近度传播奖励，而非显式相关性，削弱分配给远处但重要动作的信号，并造成样本效率的来源。我们引入解释性世界模型（EWM），这是一种归因框架，利用解释识别可能促成结果的早期事件。我们用前沿语言模型实例化EWM，并利用保守的层级算子作为学习信号，递归地重新分配奖励，同时保持未折现的剧集返回。为探究何谓有用解释，我们操作化了三种互补变体，灵感来自Halpern的实际因果关系、Lombrozo的解释性简洁性和Deutsch难以变化的解释。我们在两个受控的长期RL基准测试中评估EWM，一个是公开的顺序推荐基准和一个行业规模的生成推荐基准。在Crafter和WordCraft上，EWM显著优于PPO和LaRe，在奖励延迟最严重时提升最大。在KuaiRand上，EWM离线表现优于SFT和LaRe，在线A2C。最后，在对行业生成推荐器的离线评估中，EWM显著提升了召回率并增加了行为代理。这些结果表明，解释可以为从延迟结果中学习提供结构化且可重用的学分。

MENTOR: Learning from Failure with On-Policy Pedagogical Intervention

导师：通过政策教学干预从失败中学习

• Authors: Guanlan Yue, Hang Yu, Xiaoyong Huang, Ruipeng Li, Wei Zhang, Bingchang Liu, Rui Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13594
• Pdf link: https://arxiv.org/pdf/2610.13594
• Abstract Reinforcement Learning from Verifiable Rewards (RLVR) is a key driver for Large Reasoning Models, but it is vulnerable to "zero-reward collapse", a training failure where a sudden increase in task difficulty causes the learning signal to vanish. We argue that the conventional approach of discarding these failed trajectories is wasteful, as they contain valuable diagnostic information about the model's cognitive gaps, a bright side that can be leveraged for recovery. Existing solutions, however, often regress to static, "off-policy-style" interventions like pre-defined hint libraries or perform post-hoc refinement, failing to provide the adaptive, forward-looking guidance needed to escape the collapse. To address this, we propose a paradigm shift towards on-policy pedagogical intervention and introduce MENTOR, a multi-agent framework that acts as a live mentor. MENTOR diagnoses the root cause of failure in real-time and dynamically generates targeted, instructional prompts to guide the learning agent's next attempt. As a plug-and-play module, MENTOR integrates seamlessly into existing policy optimization pipelines like GRPO without modifying the training infrastructure. Extensive experiments on math and code generation show that MENTOR effectively mitigates zero-reward collapse, outperforming state-of-the-art baselines. Notably, it achieves massive average relative gains of up to 78.3% on Llama-3.2-3B-Instruct over its base model.
• 中文摘要 可验证奖励强化学习（RLVR）是大型推理模型的关键驱动力，但它易受“零奖励崩溃”影响，即任务难度突然上升导致学习信号消失的训练失败。我们认为，传统的放弃这些失败路径是浪费的，因为它们包含了关于模型认知缺口的宝贵诊断信息，是可利用的一面来实现恢复。然而，现有解决方案常常退化为静态的“非策略风格”干预，如预定义提示库或进行事后细化，未能提供适应性、前瞻性的指导，帮助避免崩溃。为此，我们提出向政策内教学干预的范式转变，并引入多智能体框架MENTOR，作为活导师。MENTOR 实时诊断失败根源，动态生成有针对性的指导提示，指导学习代理的下一次尝试。作为即插即用模块，MENTOR 无缝集成到现有策略优化流程如 GRPO，无需修改训练基础设施。数学和代码生成的大量实验表明，MENTOR 有效缓解零奖励崩溃，优于最先进的基线。值得注意的是，它在 Llama-3.2-3B-Instruct 基础模型上实现了高达 78.3% 的相对平均提升。

DIA: A Deep Reinforcement Learning-Based Interest-Aware ABR Scheme for Cloud Gaming

DIA：基于深度强化学习的云游戏兴趣感知ABR方案

• Authors: Zhe Zhang, Haiwei Dong, Marc St-Hilaire, Abdulmotaleb El Saddik
• Subjects: Subjects: Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2610.13698
• Pdf link: https://arxiv.org/pdf/2610.13698
• Abstract The continuous evolution of cloud gaming has enabled users to play high-quality cloud games on thin clients, such as smartphones and tablets. However, how to effectively transmit the gameplay frames with high resolution and low latency, especially over wireless networks, remains a significant challenge. To address this challenge, we propose a Deep Reinforcement Learning (DRL)-based Interest-Aware Adaptive Bitrate (ABR) scheme, called DIA, that guarantees the quality of perceptually key objects in each frame, while dynamically allocating lower resolutions for non-key objects to conserve bandwidth under constrained network conditions. By leveraging an off-the-shelf semantic segmentation model, the proposed DIA scheme can adjust the bitrate at the object-level. We also design a transformer-based feature extractor and an interest-aware reward function to enable the proposed DIA scheme to identify key objects and dynamically adjust their bitrates. This adaptation is guided by users' in-game movement speed, object features, and prevailing network conditions and operates without requiring additional hardware, such as gaze trackers. Extensive experiments show that the proposed DIA scheme outperforms existing ABR schemes in dynamic wireless network environments. Notably, DIA achieves at least a 20% higher normalized average Quality of Experience (QoE) than the State-Of-The-Art (SOTA) ABR algorithms in 5G scenarios.
• 中文摘要 云游戏的持续演进使用户能够在智能手机和平板电脑等瘦客户端上玩高质量的云游戏。然而，如何高效传输高分辨率和低延迟的游戏帧，尤其是在无线网络上，仍是一个重大挑战。为应对这一挑战，我们提出了一种基于深度强化学习（DRL）的兴趣感知自适应比特率（ABR）方案，该方案保证每帧中感知关键对象的质量，同时动态分配非关键对象较低分辨率，以在受限网络条件下节省带宽。通过现成的语义分割模型，所提DIA方案可以在对象层面调整比特率。我们还设计了一个基于变换器的特征提取器和兴趣感知奖励函数，使DIA方案能够识别关键对象并动态调整其比特率。这种适应由用户在游戏中的移动速度、物体特征和当前网络条件指导，且无需额外硬件（如视线追踪器）即可操作。大量实验表明，所提议的DIA方案在动态无线网络环境中优于现有ABR方案。值得注意的是，DIA在5G场景下比最先进（SOTA）ABR算法实现至少20%的归一化平均体验质量（QoE）。

SafeDiSTL: Decomposing Joint Constraints for Safe Multi-Agent Reinforcement Learning

SafeDiSTL：用于安全多智能体强化学习的分解联合约束

• Authors: Hongrui Zheng, Aditya Singh, Rahul Mangharam
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.13737
• Pdf link: https://arxiv.org/pdf/2610.13737
• Abstract Cooperative multi-agent tasks often impose constraints on a shared objective or collective safety. A team level violation cost function can measure this constraint outcome, but does not distinguish which agents' actions most strongly influence that violation. We present SafeDiSTL, a constrained multi-agent reinforcement learning framework that allows for credit assignment even with joint constraint measurements. We use Signal Temporal Logic (STL) robustness to measure episode-level violation through running-maximum cost increments. Smooth full-episode robustness and differentiable dynamics provide action sensitivities, which allocate each cost increment across agents at the same decision time. These sensitivities account for delayed effects when weighting agents' shares of each cost increment. We study the framework in cooperative payload transport, connected navigation, and networked traffic control tasks. SafeDiSTL achieves 1.6-5.1x the safe-success rate in transport, improves navigation safe-success by 11.1%, and delivers 50-68% more throughput in traffic when compared to constrained baselines. Results demonstrate a strong improvement of the learned policy in dealing with shared constraint while retaining the performance of unconstrained policies.
• 中文摘要 合作多智能体任务通常对共享目标或集体安全施加约束。团队级违规成本函数可以衡量该约束结果，但无法区分哪些代理的行为对该违规影响最大。我们介绍SafeDiSTL，一种受限的多代理强化学习框架，允许即使有联合约束测量也能进行信用分配。我们使用信号时间逻辑（STL）鲁棒性，通过运行最大成本增量来衡量剧集级违规。平滑的全集鲁棒性和可微动力学提供了动作敏感性，使每个成本增量在同一决策时间分配给各代理。这些敏感性考虑了在加权代理在每个成本增量中份额时的延迟效应。我们研究该框架在协作有效载荷运输、连接导航和网络交通控制任务中的应用。SafeDiSTL在运输中实现了1.6-5.1倍的安全成功率，使导航安全成功率提升11.1%，并且在交通吞吐量上比受限基线高出50%-68%。结果显示，在处理共享约束时，学习策略在保持无约束策略性能方面有了显著提升。

MARGoS: A Configuration-Driven Platform for Reproducible MARL Experiments in ARGoS

MARGoS：用于ARGoS中可重复MARL实验的配置驱动平台

• Authors: Tom Philipp Backert, Mohammad D. Soorati, Javad Ghofrani
• Subjects: Subjects: Software Engineering (cs.SE); Multiagent Systems (cs.MA); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.13772
• Pdf link: https://arxiv.org/pdf/2610.13772
• Abstract Multi-agent reinforcement learning (MARL) has become a prominent research direction, particularly within collective robotics, where the ARGoS simulator is widely used to study emergent swarm behavior. Yet despite this growth, systematic support for reuse and reproducibility in MARL-based swarm robotics remains limited and insufficiently standardized: a single experiment is assembled from manually aligned simulator files, training scripts, configuration values, paths, seeds, and environment details, so reuse and reproduction stay fragile even within one research group. To address this gap, we introduce MARGoS, a lightweight, configuration-driven platform developed in this project that elevates these experiment-defining inputs to first-class, platform-managed artifacts. MARGoS enables reproducible experimentation through configuration hashing, end-to-end seed propagation, frozen (immutable) configurations, structured logging, environment fingerprinting, and exportable experiment bundles that facilitate sharing, replay, and verification of results across machines and over time. A five-dimension evaluation (architecture, efficiency, reproducibility, usability, and collaboration) supports our claim that managing experiment variation as a platform responsibility makes ARGoS+MARL experiments easier to reuse, reproduce, and hand off. Our code is available at this https URL.
• 中文摘要 多智能体强化学习（MARL）已成为一个重要的研究方向，尤其是在集体机器人领域，ARGoS模拟器被广泛用于研究涌现的群体行为。然而，尽管有这种增长，基于MARL的群体机器人对重用和可重复性的系统支持仍然有限且标准化不足：单个实验由手动对齐的模拟器文件、训练脚本、配置值、路径、种子和环境细节组成，因此即使在同一研究组内，重用和复制也依然脆弱。为弥补这一空白，我们引入了MARGoS，一个由本项目开发的轻量级配置驱动平台，将这些定义实验的输入提升为一流的平台管理产物。MARGoS通过配置哈希、端到端种子传播、冻结（不可变）配置、结构化日志、环境指纹识别以及可导出的实验包实现可重复的实验，促进了跨机器和时间的共享、回放和验证结果。五维评估（架构、效率、可重复性、可用性和协作）支持了我们的观点：将实验变体管理作为平台责任，使ARGoS+MARL实验更容易重用、重现和交付。我们的代码可在该 https URL 访问。

SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models

SimVTA：模拟基于视频轨迹对齐，用于驱动世界动作模型

• Authors: Changhui Jing, Haochen Tian, Hongchen Li, Tianyu Li, Caojun Wang, Zihao Zhang, Yanhao Liu, Chengen Xie, Bailin Li, Hao Zhao, Zhenjie Yang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.13781
• Pdf link: https://arxiv.org/pdf/2610.13781
• Abstract Driving world action models (WAMs) learn from paired trajectories and future observations. Post-training with reinforcement learning allows the policy to explore alternative trajectories and improve planning using driving rewards. However, driving logs contain future videos only for the recorded trajectories, leaving these alternatives without corresponding visual supervision. Thus, we introduce SimVTA, a simulation-grounded post-training framework that combines trajectory optimization with video supervision. First, we sample candidate trajectories for each observation and use a real-world simulator (e.g., World Engine) to render future videos for high-reward candidates in real-world scenes reconstructed with 3D Gaussian Splatting (3DGS). This provides trajectory-matched supervision independent of the model's own predictions. Under a limited rendering budget, we prioritize video supervision for higher-error futures among these rendered candidates. Finally, we optimize trajectory planning with Group Relative Policy Optimization (GRPO) and use a flow-matching loss on the matched videos to update shared representations. At inference time, the model predicts trajectories directly, without simulator calls. On NAVSIM v1, applying SimVTA to a WAM initialization improves PDMS from 86.2 to 91.3 (+5.1). It achieves 88.2 EPDMS on NAVSIM v2 and an average HD-Score of 38.8 in closed-loop evaluation on HUGSIM. Ablations show that updating shared representations with video supervision improves planning, while prioritizing higher-error futures among high-reward trajectories further improves performance.
• 中文摘要 驱动世界动作模型（WAMs）通过配对轨迹和未来观测进行学习。带强化学习的后期训练使策略能够探索替代轨迹并利用驾驶奖励改进规划。然而，驾驶日志仅包含记录轨迹的未来视频，导致这些备选路径缺乏相应的视觉监督。因此，我们引入了SimVTA，一种基于模拟的训练后框架，结合了轨迹优化与视频监督。首先，我们对每个观测的候选轨迹进行采样，并使用真实世界模拟器（如World Engine）渲染未来高回报候选视频，并用3D高斯溅射（3DGS）重建的真实场景。这提供了轨迹匹配的监督，独立于模型自身的预测。在有限的渲染预算下，我们优先对这些渲染候选对象中误差更高的未来进行视频监督。最后，我们利用群相对策略优化（GRPO）优化轨迹规划，并利用匹配视频的流量匹配损耗来更新共享表示。在推断阶段，模型直接预测轨迹，无需模拟器调用。在NAVSIM v1中，将SimVTA应用于WAM初始化，使PDMS从86.2提升至91.3（+5.1）。在NAVSIM v2上实现了88.2 EPDMS，在HUGSIM闭环评估中平均HD-Score为38.8。消融显示，带视频监督更新共享表示能提升规划效果，而在高回报轨迹中优先考虑高误差未来则进一步提升性能。

Dream to Remember: Continual Steering of Generalist Robot Policies with World Models

铭记梦想：用世界模型持续引导通用机器人政策

• Authors: Bahey Tharwat, Iman Nematollahi, Marvin Alles, Xingyuan Zhang, Akshay L Chandra, Abhinav Valada
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.13816
• Pdf link: https://arxiv.org/pdf/2610.13816
• Abstract Generalist robot policies are trained with imitation learning and are bounded by the quality and coverage of their demonstrations. Reinforcement learning can improve and extend them, either by fine-tuning the policy weights or by training a small module to steer the input noise of a frozen generative policy. When tasks arrive sequentially, both approaches suffer from catastrophic forgetting. Fine-tuning on a task degrades prior tasks, and steering only shifts forgetting to the shared steering module. We propose Dream to Remember (D2R), which continually steers a frozen generalist policy toward new tasks, while retaining its earlier skills. As the steering module learns a new task, a frozen pretrained world model rehearses prior tasks through imagined rollouts, anchoring the module to its earlier behavior without a replay buffer or new data. When the world model is faithful enough, D2R learns the new task entirely in imagination. Otherwise, it learns the task in the environment while using the world model only to rehearse prior tasks. On CALVIN, D2R improves eight tasks in sequence entirely in imagination, raising average success from 63.7% to 74.8%, while LoRA fine-tuning on the same sequence drops it to 11.7%. With a frozen off-the-shelf generalist policy and world model, $\pi_0$ and A2World, D2R learns unseen LIBERO-90 tasks in the environment while keeping $\pi_0$'s performance on the LIBERO suites it was fine-tuned on, even though the world model is not accurate enough for policy optimization. Finally, on a real Franka robot, D2R improves three tasks in sequence entirely in imagination while retaining the earlier improvements.
• 中文摘要 通用机器人策略通过模仿学习训练，受限于演示的质量和覆盖范围。强化学习可以通过微调策略权重或训练一个小模块来引导冻结生成策略的输入噪声，从而改进和扩展策略。当任务顺序出现时，两种方法都会出现灾难性的遗忘。对任务进行微调会削弱先前任务，而引导仅会将遗忘转移到共享引导模块。我们提出了“梦回忆”（Dream to Remember，D2R），它不断引导固定的通用策略朝向新任务，同时保留其早期技能。当引导模块学习新任务时，冻结的预训练世界模型通过想象的展开排练先前任务，将模块锚定于其早期行为，无需重放缓冲或新数据。当世界模型足够忠实时，D2R完全在想象中学习新任务。否则，它在环境中学习任务，同时仅用世界模型来排练之前的任务。在CALVIN上，D2R完全凭想象顺序改进了八个任务，平均成功率从63.7%提升到74.8%，而对同一序列进行LoRA微调后，成功率降至11.7%。通过一个冻结的现成通用策略和世界模型$\pi_0$和A2World，D2R在环境中学习未见的LIBERO-90任务，同时保持$\pi_0$在其微调后LIBERO套件中的表现，尽管世界模型的准确度不足以进行策略优化。最后，在真实的Franka机器人上，D2R完全凭想象顺序改进了三个任务，同时保留了早期的改进。

Partially Informed Stochastic Optimization via Zeroth-Order Residual Tracking

通过零阶残差跟踪进行部分知情随机优化

• Authors: El Mahdi Chayti, Alireza Mirrokni, Martin Jaggi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.13871
• Pdf link: https://arxiv.org/pdf/2610.13871
• Abstract Many optimization problems provide direct access to only part of the gradient, while the remaining component must be inferred from function evaluations. This structure arises in performative prediction, performative reinforcement learning, and bilevel optimization with a black-box lower level. We introduce Partially Informed Stochastic Optimization (PISO), which combines the available gradient component with a running estimate of the residual. By centring each zeroth-order observation on this estimate, PISO makes the variance due to random probing depend on the tracking error rather than the residual's magnitude. This yields a randomized sketch iteration with an explicit contraction mechanism and a principled interpretation of momentum. For smooth nonconvex objectives, we establish convergence guarantees that depend on the residual's variation and initial magnitude, without requiring a uniform bound on the unavailable gradient component. With independently sampled function evaluations and Gaussian or spherical probes, PISO attains leading function-evaluation complexities of (O(d^{2}\varepsilon^{-6})) under Lipschitz gradients and (O(d^{2}\varepsilon^{-5})) under an additional Lipschitz-Hessian assumption. These rates require only one probe direction, two function samples, and one known-component sample per update. The analysis also quantifies how batching trades observations for fewer deployments. Across five settings, PISO variants improve on the tested baselines with comparable oracle access in three, reach a comparable solution faster in bilevel routing, and perform best among methods without response-model derivatives in performative reinforcement learning.
• 中文摘要 许多优化问题仅直接访问梯度的一部分，其余部分则需通过函数评估推断。这种结构体现在执行预测、执行强化学习和带有黑箱下层的双层优化中。我们引入部分知情随机优化（PISO），将可用梯度分量与剩余的持续估计结合起来。通过将每个零阶观测置中于该估计值，PISO使随机探测产生的方差依赖于跟踪误差而非残差的大小。这产生了随机草图迭代，具有显式收缩机制和对动量的原则性解释。对于平滑非凸目标，我们建立依赖于残差变化和初始大小的收敛保证，而无需对不可得梯度分量有统一界限。通过独立采样的函数评估和高斯探针或球面探针，PISO在利普希茨梯度下达到了\（O（d^{2}\varepsilon^{-6}）\））的领先函数评估复杂度，在额外的利普希茨-黑森假设下实现\（O（d^{2}\varepsilon^{-5}）\）。这些速率每次更新只需一个探针方向、两个函数样本和一个已知成分样本。分析还量化了批处理如何以更少的部署次数交换观测值。在五种环境中，PISO变体在三种条件下以可比的预言机访问提升了测试基线，在双层路由中更快达到可比解，在无响应模型导数的方法中表现最佳。

Keyword: diffusion policy

Non-Prehensile Multi-Object Placement with Mobile Robots via Diffusion-Based Trajectory Planning

通过基于扩散轨迹规划的移动机器人实现非抓握式多对象布置

• Authors: Lance Pharand, Megnath Ramesh, Mark Chiu, Yash Vardhan Pant, Stephen L. Smith
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.12684
• Pdf link: https://arxiv.org/pdf/2610.12684
• Abstract Non-prehensile manipulation with mobile robots has primarily targeted placement in coarse goal regions. Precision placement has been studied with manipulators on benchmarks such as Push-T, where an omnidirectional end effector can readily re-approach an object from any side. For a mobile robot, correcting the same placement errors may require wide repositioning maneuvers around the object. We formalize multi-object pushing with a mobile robot as a multi-target assignment problem at tight tolerances. We propose a diffusion planner with a causal factorization: each robot denoising step is followed by an object step conditioned on the current clean robot estimate. This factorization allows us to sample multiple object outcomes for a single robot plan, capturing the multi-modality of pushing. We also design a set of pushing-specific costs for mobile robots, used for both guidance and best-of-$N$ sampling at inference. On Push-T, our planner attains the highest success rate of the methods we compare, while having the lowest rate of implausible object motion. Across three multi-box pushing environments resembling warehouse scenarios, it raises the fraction of boxes placed by $21.5$ percentage points over an unguided baseline, and replans $1.8\times$ faster than a planner that denoises the two trajectories together. Lastly, we deploy the guided planner on a TurtleBot 3 Burger with object shapes absent from the training data, where it places $85.0\%$ of objects within tolerance against Diffusion Policy's $37.5\%$.
• 中文摘要 移动机器人的非抓握操作主要针对粗目标区域的目标定位。精密布置已在Push-T等基准测试中用机械臂研究，该基准测试中全向端执行器可轻松从任意方向重新接近物体。对于移动机器人，纠正相同放置误差可能需要在物体周围进行大范围重新定位。我们将移动机器人的多目标推力形式化为多目标分配问题，且公差严格。我们提出了一个扩散规划器，采用因果分解：机器人每个去噪步骤后，紧接一个基于当前干净机器人估计的物体步骤。该分解使我们能够为单一机器人计划采样多个物体结果，捕捉推送的多模态特性。我们还设计了一套针对移动机器人推送的具体成本，既用于引导，也用于推断时$N最佳采样。在Push-T中，我们的规划器在比较的方法中成功率最高，且物体不合理运动率最低。在三种类似仓库场景的多箱推送环境中，它能将放置箱子的比例提高21.5美元百分点，且比同时去噪两个轨迹的规划器快1.8倍。最后，我们将引导规划器部署在训练数据中缺少物体形状的TurtleBot 3汉堡上，它将85.0%$的物体置入与扩散政策37.5%$的容忍范围内。

RARA: Robust Policy Fine-Tuning through Retrieval-Anchored Representation Alignment for Robotic Manipulation

RARA：通过检索锚定表示对齐实现的稳健策略微调，以实现机器人操作

• Authors: Xuhui Kang, Sung-Wook Lee, Yan Zhuang, Tianle Zhong, Yen-Ling Kuo
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.13578
• Pdf link: https://arxiv.org/pdf/2610.13578
• Abstract Pre-trained robot policies, such as Vision-Language-Action (VLA) models, provide a strong foundation for generalist robotic manipulation, but adapting them to new tasks with only a few demonstrations remains challenging. Directly fine-tuning a pre-trained policy often achieves strong in-distribution performance, yet it can distort the representations learned during pre-training and weaken the model's generalization ability under out-of-distribution shifts. We propose Retrieval-Anchored Representation Alignment (RARA), a two-stage adaptation method, to address this issue. First, RARA retrieves representation features from pretraining with similar task intent and motion, and aligns the policy's representations of the target demonstrations with the pre-trained representations of the retrieved demonstrations. Second, RARA fine-tunes the policy on the target demonstrations while limiting how much its representations of the retrieved demonstrations change during fine-tuning. By reducing the representation mismatch before action learning and preserving pre-trained knowledge during adaptation, RARA limits unnecessary representation drift during fine-tuning. Experiments with Diffusion Policy and GR00T on 13 tasks in RoboTwin, RoboCasa, and on a real robot show that RARA improves the average success of fine-tuning by 9.5 points in-distribution and 13.6 points out-of-distribution, and outperforms all compared baselines. Project page: this https URL
• 中文摘要 预训练机器人策略，如视觉-语言-动作（VLA）模型，为通用机器人操作提供了坚实基础，但仅用少量演示将其适应新任务仍具挑战性。直接微调预训练策略通常能实现优异的分布内表现，但这会扭曲预训练中学到的表征，并在分布外转移时削弱模型的泛化能力。我们提出了检索锚定表示对齐（RARA）两种适应方法来解决这一问题。首先，RARA从预训练中检索具有相似任务意图和运动的表示特征，并将策略中目标演示的表示与检索演示的预训练表示对齐。其次，RARA在微调过程中对目标演示的策略进行微调，同时限制其对检索演示表示的变化幅度。通过减少动作学习前的表示不匹配并在适应过程中保留预训练知识，RARA限制了微调过程中不必要的表示漂移。在RoboTwin、RoboCasa和真实机器人上的13个任务中使用扩散策略和GR00T的实验显示，RARA在微调中平均成功率提升了9.5个点，分布外提升了13.6个点，并且优于所有对比基线。项目页面：此链接