生成时间: 2026-10-22 21:35:10 (UTC+8); Arxiv 发布时间: 2026-10-22 20:00 EDT (2026-10-23 08:00 UTC+8)

今天共有 49 篇相关文章

Keyword: reinforcement learning

Reinforcement learning for hardware acceleration and system-level optimization on FPGA and integrated-circuit platforms: A survey

FPGA和集成电路平台上硬件加速和系统级优化的强化学习：一项综述

• Authors: Hamza Kheddar
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Hardware Architecture (cs.AR); Performance (cs.PF); Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.28699
• Pdf link: https://arxiv.org/pdf/2610.28699
• Abstract Reinforcement learning (RL) enables adaptive and autonomous decision-making in embedded, edge, and heterogeneous computing systems. Field-programmable gate arrays (FPGAs) and integrated circuits (ICs) offer low latency, energy efficiency, and architectural flexibility for RL deployment, while introducing challenges in resource constraints, implementation complexity, and hardware-algorithm co-design. This review surveys more than 100 studies on RL implementations across FPGA and IC platforms, emphasizing architectures, implementation strategies, performance evaluation, and applications. The literature is organized by RL family, including model-based Markov decision processes, classical value-based methods, deep RL, actor-critic, policy-gradient, and multi-agent approaches. We further classify studies by abstraction level, hardware realization strategy, and application domain, examining how RL algorithms map onto reconfigurable and fixed hardware. Unlike surveys restricted to particular algorithms or platforms, this work provides a unified analysis of FPGA and IC implementations, covering acceleration, reconfigurable architectures, energy-aware control, and system-level optimization. Comparative assessments consider resource utilization, latency, throughput, power consumption, and reported performance gains. Finally, we identify recurring architectural patterns, practical limitations, open challenges, and research directions for efficient RL-driven hardware systems. The survey offers a consolidated reference for researchers developing and evaluating RL solutions on specialized hardware.
• 中文摘要 强化学习（RL）使嵌入式、边缘和异构计算系统中的自适应和自主决策成为可能。现场可编程门阵列（FPGA）和集成电路（IC）为强化学习部署提供了低延迟、能效和架构灵活性，同时也带来了资源约束、实现复杂性和硬件-算法协同设计等挑战。本综述综述了100多项关于FPGA和集成电路平台上强化学习实现的研究，重点关注架构、实施策略、性能评估和应用。文献按强化学习家族组织，包括基于模型的马尔可夫决策过程、经典基于价值的方法、深度强化学习、演员批评法、策略梯度和多智能体方法。我们进一步按抽象层级、硬件实现策略和应用领域分类研究，考察强化学习算法如何映射到可重构和固定硬件上。与局限于特定算法或平台的调查不同，本研究提供了FPGA和集成电路实现的统一分析，涵盖加速、可重构架构、能耗控制和系统级优化。比较评估考虑资源利用率、延迟、吞吐量、功耗及报告的性能提升。最后，我们识别了高效强化学习驱动硬件系统的反复出现的架构模式、实际局限、未解决挑战及研究方向。该调查为开发和评估专用硬件强化学习解决方案的研究者提供了综合参考。

SimHack: Predicting the onset of detectable reward hacking before RL training

模拟黑客：在强化学习训练前预测可检测到的奖励黑客行为的起始

• Authors: Junwei Deng, Alex Bie, Natalia Ponomareva
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28707
• Pdf link: https://arxiv.org/pdf/2610.28707
• Abstract When training LLMs to code with reinforcement learning, reward hacking is often detectable but difficult to prevent, since (a) environment hardening is increasingly difficult as training environments grow more complex; and (b) training-time interventions incentivize models to obfuscate hacking behavior to bypass detection. We instead study whether we can predict when detectable reward hacking becomes dominant before executing an RL run, enabling us to select lower-risk recipes in advance. We find that the initial hacking rate alone does not determine onset, which varies substantially across training recipes and repeated runs. We thus formulate this problem as predicting the distribution of the reward hacking onset step under a target training recipe. We introduce SimHack, a simulator that learns reward hacking training dynamics from a handful of smaller-scale reference RL runs and generates counterfactual simulation runs of unseen target recipes without costly retraining. Across unseen target recipes, SimHack predicts the median onset to a relative error of 11.3% and achieves across-recipe rank correlation of 0.85, while generalizing to differences in model scale, curriculum, number of rollouts, and learning rate between reference and target recipes. As an application, we show that SimHack can guide training decisions by screening candidate RL curricula, discovering curricula over the same set of scenarios that train for 96.6% longer without hacking.
• 中文摘要 在用强化学习训练大型语言模型编程时，奖励黑客通常可检测但难以预防，因为（a）随着训练环境的复杂，环境加固变得越来越困难;（b）训练时间干预激励模型混淆黑客行为以绕过检测。我们转而研究是否能预测可检测奖励黑客何时成为主导，从而提前选择风险较低的配方。我们发现仅凭初始黑客率无法决定发病时间，且触发率在不同训练配方和重复运行间差异显著。因此，我们将此问题表述为预测目标训练配方下奖励黑客启动步骤的分布。我们介绍SimHack，这是一款模拟器，通过少量小规模参考强化学习学习奖励黑客训练动态，生成未见目标配方的反事实模拟运行，无需昂贵的重新训练。在未见目标配方中，SimHack预测中位数起因误差为11.3%，并实现了0.85的跨配方排名相关，同时推广到参考与目标配方之间模型规模、课程、推广次数和学习率的差异。作为应用，我们展示了SimHack可以通过筛选候选强化学习课程来指导培训决策，发现在同一场景下，这些场景在不使用黑客的情况下训练时间长96.6%。

ReAPO: Retry-Aware Policy Optimization for Efficient Exploration in LLM Reasoning

ReAPO：重试感知策略优化，促进LLM推理中的高效探索

• Authors: Nghi Dao, Calder Morton-Ferguson, Noah Golowich, Adam Block
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.28770
• Pdf link: https://arxiv.org/pdf/2610.28770
• Abstract RLVR improves language-model reasoning, but commonly relies on independent stochastic rollouts for exploration, which can repeatedly sample similar failed strategies. Prior work has seen success with approaches that guide exploration by introducing dependence into the sampling process by including incorrect solutions in the context for subsequent attempts. Still, these methods introduce the additional question of how to assign credit to each attempt. We propose to view the credit assignment through the lens of Stochastic Shortest Paths, a framework in classical reinforcement learning that seeks to minimize the expected cumulative cost to reach a terminal state (in our case, a verified successful response). We show that, under suitable simplifying assumptions, this perspective naturally leads to a simple modification to GRPO, which we call Retry-Aware Policy Optimization (ReAPO). ReAPO allocates additional generation to failed trajectories, allowing successful retries to be reinforced without rewarding preceding failures, and assigns credit in a theoretically principled way to each attempt. Across Qwen3 and OLMo-2 models from 0.6B to 4B and eight mathematical and scientific reasoning benchmarks, ReAPO improves first-attempt performance over GRPO as well as natural recent baseline algorithms, while additional ablations demonstrate the utility of both serial sampling and ReAPO's credit assignment. On Qwen3-0.6B, ReAPO also outperforms GRPO with twice the group size or twice the training steps, reaching their peak accuracy with 39% and 31% fewer generated tokens, respectively. Taken together, our results suggest that ReAPO is a simple yet effective approach for improving LM reasoning performance through improved exploration and retry-aware credit assignment.
• 中文摘要 RLVR改进了语言模型推理，但通常依赖独立的随机展开进行探索，这可以反复采样类似失败的策略。以往的研究已成功通过在采样过程中引入依赖性，在后续尝试中加入错误解来引导探索。然而，这些方法也带来了如何为每次尝试分配信用的问题。我们建议通过随机最短路径（Stochastic Shortest Paths）的视角来审视信用分配，这是一种经典强化学习中的框架，旨在最小化达到终极状态（在我们案例中为已验证的成功响应）的预期累计成本。我们表明，在适当的简化假设下，这种视角自然导致对GRPO的简单修改，我们称之为重试感知策略优化（ReAPO）。ReAPO为失败轨迹分配额外生成，使成功重试得以强化而不奖励之前的失败，并以理论原则性的方式为每次尝试分配功劳。在Qwen3和OLMo-2模型中，从0.6B到4B，以及八个数学和科学推理基准，ReAPO不仅提升了对GRPO的首次尝试性能，也提升了自然的近期基线算法，同时额外的消融展示了串行抽样和ReAPO功分分配的实用性。在Qwen3-0.6B中，ReAPO在组规模或训练步数的两倍方面也优于GRPO，分别以减少39%和31%的生成代币达到峰值准确率。综合来看，我们的结果表明，ReAPO是一种简单但有效的方法，通过改进探索和重试意识的学分分配，提升LM推理性能。

Complexity-Aware Surgical Video Reasoning Segmentation with Digital Twin and Context Memory

复杂性感知手术视频推理分割，结合数字孪生和上下文记忆

• Authors: Yiqing Shen, Mathias Unberath
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.28776
• Pdf link: https://arxiv.org/pdf/2610.28776
• Abstract Conventional surgical segmentation relies on closed-set categories or explicit text queries, which cannot resolve implicit queries that require reasoning about workflows, instrument functions, and temporal relationships. To address these limitations, we propose SurgReasonSeg, a complexity-aware method that brings reasoning segmentation over digital twin (DT) representations to surgical video. Specifically, SurgReasonSeg constructs DT representations from surgical foundation models and trains two low-rank adapters (LoRA) of one large language model with reinforcement learning (RL), each specialized for a different query complexity. A lightweight router, also trained with RL, selects between the two adapters to balance latency against accuracy. SurgReasonSeg also sets the temporal context with an adaptive sliding window conditioned on query complexity and workflow phase, and retrieves relevant historical cases from a surgical context memory by workflow similarity and procedural relevance. We further introduce EndoVis-ReasonSeg, a surgical reasoning segmentation benchmark with 1,460 training and 961 test samples across semantic, spatial, and temporal categories. On EndoVis-ReasonSeg, SurgReasonSeg improves J&F over the strongest baseline, ReSurgSAM2, by 0.178, 0.237, and 0.297 on semantic, spatial, and temporal queries, and it also achieves the best results on the Ref-EndoVis17 and Ref-EndoVis18 referring segmentation benchmarks. A backbone-matched attribution protocol separates the contribution of the upstream segmenter from that of the reasoning modules. Our method runs at 18.8 FPS averaged over the query distribution, enabling real-time reasoning segmentation in surgical videos.
• 中文摘要 传统的外科分割依赖于封闭集类别或显式文本查询，无法解析需要推理工作流程、仪器功能和时间关系的隐式查询。为解决这些局限性，我们提出了SurgReasonSeg，一种复杂度感知方法，将推理分割应用于数字孪生（DT）表示中。具体来说，SurgReasonSeg 从手术基础模型构建DT表示，并通过强化学习（RL）训练两个大型语言模型的低秩适配器（LoRA），每个适配器专门化不同查询复杂度。一个同样用强化学习训练的轻量级路由器在两个适配器之间选择，以平衡延迟与准确性。SurgReasonSeg 还通过自适应滑动窗口根据查询复杂度和工作流阶段设置时间上下文，并根据工作流相似性和程序相关性从手术上下文内存中检索相关历史案例。我们进一步介绍了EndoVis-ReasonSeg，这是一个外科推理分割基准测试，涵盖语义、空间和时间类别，包含1460个训练样本和961个测试样本。在EndoVis-ReasonSeg上，SurgReasonSeg在语义、空间和时间查询上将J&F提升了0.178、0.237和0.297，并且在Ref-EndoVis17和Ref-EndoVis18的引用分割基准测试中取得了最佳成绩。骨干匹配归因协议将上游分段子与推理模块的贡献分开。我们的方法在查询分布平均下运行为18.8 FPS，实现了手术视频中的实时推理分割。

Is Anyone Enforcing the Cartel? What Reinforcement Learning Teaches LLM Agents in Markets

有人在执行卡特尔吗？强化学习教会了市场上的LLM代理什么

• Authors: Hieu Tran, Hong Yu
• Subjects: Subjects: Computation and Language (cs.CL); Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28787
• Pdf link: https://arxiv.org/pdf/2610.28787
• Abstract Evaluations of large language model (LLM) agents report that pricing agents collude at others' expense. Yet collusion is defined by enforcement: a cartel, in which competitors share monopoly profits, holds only if a member who cheats is punished. We introduce an audit that tests enforcement by forcing a member to cheat in counterfactual copies of a market, and use it to study what reinforcement learning (RL) teaches agents in pricing and trading. Small LLMs trained by RL learn what their training rivals reward, but only what they readily try. In self-play, they learn fixed rules ignoring their rivals; against a punishing rival, to be deterred; against a cheating rival, to match its cuts, never the costly price war that would deter it, which they rarely try. A war forced into training is learned, but mostly only where it was forced. Against their own copies, deterred and matching agents can earn the same cartel profits, but only matching ones can enforce them, and then thinly: as a by-product and only against large cuts. Q-learners enforce collusion only in pricing. Prompted LLMs reach cartel-like outcomes mostly through rules that ignore the market. Outcome measures cannot tell these cases apart; the audit can.
• 中文摘要 对大型语言模型（LLM）代理的评估报告显示，定价代理以他人为代价串通。然而，勾结是由执行定义的：一个竞争者共享垄断利润的卡特尔，只有在作弊成员受到惩罚时才成立。我们引入了一种审计，通过强制成员在市场的反事实副本中作弊来检验执行能力，并用它研究强化学习（RL）在定价和交易中教导代理的内容。由强化学习训练的小型大型语言模型学习其训练竞争对手奖励的内容，但只学习他们愿意尝试的内容。在自我游戏中，他们学习固定规则，忽视竞争对手;对惩罚性强的竞争对手学习，以威慑;对作弊的竞争对手学习，以匹配其分成，从未经历过昂贵的价格战，而这场战争他们很少尝试。被迫进入训练的战争是被学习的，但大多仅在被迫的地方进行。针对自身复制品，被威慑和匹配的代理人可以获得相同的卡特尔利润，但只有匹配的代理人才能执行这些利润，而且执行很薄弱：作为副产品，且仅针对大规模削减。Q-学习者仅在定价中强制执行串通。被引导的大型语言模型主要通过忽视市场的规则达到类似卡特尔的结果。结果指标无法区分这些案例;审计可以。

ArcFlow: Training Flow Policies for Multiple Solver Budgets

ArcFlow：多求解器预算的训练流程策略

• Authors: Hieu Tran, Hong Yu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.28788
• Pdf link: https://arxiv.org/pdf/2610.28788
• Abstract Flow-matching policies adjust inference cost through the number of solver steps, but reinforcement learning at a single fixed solver budget can hide degradation at cheaper deployment budgets and even erase useful one-step behavior present in a pretrained policy. We introduce Action-Ratio-Clip Flow (ArcFlow), a deployment-aware algorithm for training one flow policy across a specified range of solver budgets. ArcFlow combines action-aligned supervision, budget exposure, and worst-budget checkpoint selection. For reconstruction objectives, its bridge aggregates temporal probes into one surrogate ratio and one PPO-style clip per action, with a boundary residual exactly equal to deployed one-step reconstruction error. Cycling exposes the shared field to the deployment range, and selection optimizes its performance floor. For native velocity objectives, multi-budget endpoint supervision applies the same principle without replacing the native update. Both realizations preserve the architecture and deployment-time cost. Across DM Control, manipulation, and locomotion, ArcFlow attains the highest mean one-step return among five flow-policy baselines on 25 of 30 tasks under each method's reported training settings. Component studies identify complementary improvements from boundary supervision, the fixed-grid bridge, budget exposure, and deployment selection. These results establish deployment-budget-aware training as an effective route to strong low-step control with one policy serving multiple inference costs.
• 中文摘要 流匹配策略通过求解器步骤数调整推理成本，但在单一固定求解器预算下进行强化学习，可以在更便宜的部署预算中隐藏退化，甚至抹去预训练策略中有用的一步行为。我们引入了动作-比率-剪辑流（ArcFlow），这是一种部署感知算法，用于在指定求解器预算范围内训练单一流策略。ArcFlow结合了动作对齐的监督、预算暴露和最差预算检查点选择。对于重建目标，其桥接将时间探针聚合为一个代理比率和一个PPO式剪辑，每个动作的边界残差恰好等于部署的一步重建误差。循环将共享字段暴露于部署范围，选择则优化其性能底线。对于原生速度目标，多预算端点监督应用相同原理，但不替代原生更新。这两种实现都保留了架构和部署时间成本。在DM控制、操作和移动方面，ArcFlow在5个流-策略基线中，在30个任务中达到了最高的平均一步回报。组件研究识别了边界监督、固定网格桥、预算暴露和部署选择的互补改进。这些结果确立了部署预算感知训练作为实现强大低步控制的有效途径，单一策略承担多重推断成本。

Brittle by Design? Multi-Scenario Reinforcement Learning for Robust Mission Coordination in Systems of Systems

设计上脆弱？多场景强化学习用于系统中的系统中稳健的任务协调

• Authors: İbrahim Oğuz Çetinkaya, Sajad Khodadadian, Taylan Topcu
• Subjects: Subjects: Systems and Control (eess.SY); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28818
• Pdf link: https://arxiv.org/pdf/2610.28818
• Abstract As systems engineering (SE) shifts from monolithic systems to Systems of Systems, Mission Engineering (ME) has emerged as a new line of thinking in SE. Mission environments are uncertain and dynamic, and outcomes depend on how mission assets interact with them, rendering static architectures brittle and calling for flexible yet analytically rigorous ME approaches. We propose an intelligent mission coordination methodology integrating high-fidelity digital mission models with Reinforcement Learning (RL) for adaptive task allocation. Using a Digital Engineering (DE) infrastructure comprising a digital mission model and agent-based simulation, we formulate mission tactics management as a Markov Decision Process and train an RL coordinator with Proximal Policy Optimization to map mission states to tactical actions from realized outcomes. An aerial firefighting mission with three scenarios demonstrates the approach. First, all trained coordinators significantly outperform random and heuristic baselines. Second, single-scenario coordinators can be brittle: the coordinator trained on the more forgiving scenario degrades to random tactics performance in more demanding scenarios, whereas the demanding-scenario coordinator retains most of its advantage when transferred. Third, within the same training budget, a coordinator trained on both scenarios concurrently matches the scenario-specific coordinators in their own scenarios and significantly outperforms every other policy. Sensitivity analyses on ignition location, weather, scenario, and training curricula show that the variability represented during training shapes post-deployment performance. This proof of concept shows DE-enabled mission simulations combined with RL offer a reusable methodology for improving ME practice, which can be extended to fleet design and selection from a mission-first perspective.
• 中文摘要 随着系统工程（SE）从单体系统转向系统中的系统，任务工程（ME）成为SE中的一种新思维方向。任务环境充满不确定性和动态性，结果取决于任务资产与其的互动，使静态架构变得脆弱，需要灵活但分析严谨的ME方法。我们提出了一种智能任务协调方法，将高精度数字任务模型与强化学习（RL）整合，实现自适应任务分配。利用包含数字任务模型和基于代理的模拟的数字工程（DE）基础设施，我们将任务战术管理制定为马尔可夫决策过程，并通过近端策略优化培训强化学习协调员，将任务状态与实现结果的战术行动映射。一个包含三种场景的空中灭火任务展示了该方法。首先，所有受过培训的协调员的表现显著优于随机和启发式基线。其次，单一场景协调员可能较为脆弱：在更宽容的情景中受训的协调员在更具挑战性的情境中会表现为随机的战术表现，而要求高的情景协调员在调动后仍保留大部分优势。第三，在同一培训预算内，同时接受两种情景培训的协调员在各自情景中与特定情景协调员匹配，且显著优于其他所有政策。对点火地点、天气、情景和训练课程的敏感性分析显示，训练期间表现出的变异性塑造了部署后的表现。这一概念验证表明，DE支持的任务模拟结合强化学习提供了一种可重复使用的改进ME实践方法，且可从任务优先的角度推广到舰队设计和选择。

LIFT: Scoped Constraints for Evaluating and Training Instruction Following

提升：评估和培训指导的范围约束

• Authors: Eran Goldstein, Yuval Belfer, Yonatan Belinkov, Daniel Gissin
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.28819
• Pdf link: https://arxiv.org/pdf/2610.28819
• Abstract Common instruction following (IF) benchmarks, such as IFEval, evaluate the ability of language models (LMs) to follow simple constraints that can easily be verified programmatically. These same constraints, in turn, are used as reward signals for reinforcement learning. This design may lead to reward hacking of constraints, pushing models away from the main task. Recent attempts to address this resulted in benchmarks with richer constraints, but required expert labor, remained closed-access, and cannot be easily extended. We present Lift-Bench, an IF benchmark that asks a model to draft a document given a topic and a structured outline. Constraints are scoped to certain document sections, providing finer control over constraint density. Lift-Bench includes both verifiable and judged constraints, as well as a training counterpart, Lift-Train. Importantly, Lift-Bench provides both main task and instruction-following constraints, mitigating shallow reward hacking. On experiments with multiple models, we find that frontier models struggle with the harder subsets of our benchmark and analyze model performance along multiple axes. We also find that training on Lift-Train is much less susceptible to reward hacking than training on the common IFTrain. Lift-Bench and Lift-Train are publicly available, as well as the data generating pipeline.
• 中文摘要 常见的指令跟随（IF）基准测试，如IFEval，评估语言模型（LM）遵循可轻松程序验证的简单约束的能力。这些约束反过来被用作强化学习的奖励信号。这种设计可能导致约束的奖励黑客攻击，使模型偏离主要任务。近期尝试解决这一问题，导致了约束更丰富的基准，但需要专家劳动，仍为封闭访问，且难以扩展。我们介绍Lift-Bench，一种IF基准测试，要求模型在给定主题和结构化大纲时起草文档。约束范围限定于特定文档部分，提供对约束密度的更细致控制。Lift-Bench包含可验证和判定约束，以及训练对应的Lift-Train。重要的是，Lift-Bench 既提供了主要任务约束，也提供了指令跟随约束，减少了浅层奖励黑客行为。在多个模型的实验中，我们发现前沿模型在基准测试的较难子集上存在困难，并分析了多个轴的模型性能。我们还发现，Lift-Train 上的训练比在常见的 IFTrain 上更不易受到奖励黑客攻击的影响。Lift-Bench 和 Lift-Train 以及数据生成流程均公开可用。

HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics

HorizonComposer：时空一致的驾驶视频编辑，增强交通语义

• Authors: Mauricio Soroco, Yuqiu Liu, Zaid Tasneem, Francesco Pittaluga, Abhishek Aich, Wuyang Chen, Manmohan Chandraker, Ziyu Jiang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.28829
• Pdf link: https://arxiv.org/pdf/2610.28829
• Abstract Instruction-guided video generative models offer a scalable solution for stress-testing autonomous driving systems by simulating diverse environmental conditions. However, driving scenes operate within a dense Operational Design Domain (ODD) governed by strict traffic semantics. When applied to global driving-condition edits, general-purpose instruction-guided video editors frequently corrupt these semantics, leading to structural hallucinations and ``temporal shock.'' This work distills and protects traffic semantics during video editing at three levels. First, we construct a hybrid dataset by combining temporally aligned real-world videos with synthetic edits, balancing foundational scene structure with unambiguous appearance signals. Second, during training, we adapt advantage conditioning from offline reinforcement learning to align the model using dense rewards. We extract dense, pixel-level rewards from the hybrid data to serve as a localized signal and actively steer generation toward structurally reliable pixels. Finally, our model mitigates temporal shock by introducing thinking frames, a transition mechanism that seamlessly reconciles static image conditioning with dynamic video context. Extensive experiments demonstrate that our method prevents the corruption of multi-actor dynamics, establishing state-of-the-art spatiotemporal consistency, yielding consistent user preference (39% relative improvement in content preservation and 12% in temporal consistency), and achieving a significant 5% gain in downstream road line segmentation and 7% gain in downstream multi-object detection.
• 中文摘要 指令引导视频生成模型为模拟多样环境条件的自动驾驶系统压力测试提供了可扩展的解决方案。然而，驾驶场景运行在一个由严格交通语义支配的密集操作设计域（ODD）中。当应用于全局驾驶状态编辑时，通用指令引导视频编辑器经常破坏这些语义，导致结构性幻觉和“时间冲击”。这项工作在视频编辑中提炼并保护了交通语义，涵盖三个层面。首先，我们构建了一个混合数据集，将时间对齐的现实世界视频与合成编辑结合，平衡基础场景结构与明确的外观信号。其次，在训练过程中，我们调整离线强化学习中的优势条件，以利用密集奖励对齐模型。我们从混合数据中提取密集的像素级奖励，作为局部信号，并主动引导生成过程朝向结构上可靠的像素。最后，我们的模型通过引入思维帧（thinking frames）来缓解时间冲击，这是一种无缝调和静态图像条件与动态视频上下文的过渡机制。大量实验表明，我们的方法防止了多元动态的破坏，建立了最先进的时空一致性，实现了用户偏好的一致性（内容保存相对提升39%，时间一致性提升12%），并在下游道路线分段上显著提升了5%，下游多对象检测提升了7%。

DualHeads: Reward-tilted few-step diffusion made simple

DualHeads：奖励倾斜的几步扩散简化

• Authors: Yuanzhi Zhu, Eleftherios Tsonis, Simo Alami Chehboune, Xi Wang, Stéphane Lathuilière, Vicky Kalogeiton
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28858
• Pdf link: https://arxiv.org/pdf/2610.28858
• Abstract Distilled diffusion models generate high-quality images in a few sampling steps, but aligning them with task requirements and human preferences is difficult. Methods developed for multi-step diffusion models rely on differentiable rewards, or on score fields and stochastic trajectories that distilled generators may lack. We ask whether a few-step generator can be adapted using only external reward evaluations and reference samples, and show that two binary classifiers suffice. Our method, DualHeads, implements them as lightweight heads on a frozen diffusion backbone. The reward head learns which samples of the same prompt score higher, the reference head separates reference images from generated ones, and the generator maximizes a weighted sum of their logits. When the heads are optimal, this objective is KL-regularized maximization of the learned reward, and the ratio of head weights sets the regularization strength. Empirically, with both rule-based and model-based rewards, a few-step DualHeads generator reaches task rewards close to those of 40-step reinforcement learning methods while preserving image quality, at no extra inference cost. We hope that this simple and elegant dual-head formulation will help advance few-step reward-tilted generation.
• 中文摘要 蒸馏扩散模型可在少数采样步骤内生成高质量图像，但使其符合任务需求和人类偏好较为困难。多步扩散模型开发的方法依赖可微奖励，或基于分数场和随机轨迹，而这些是蒸馏生成器可能缺乏的。我们探讨是否可以仅通过外部奖励评估和参考样本进行调整，并证明两个二元分类器即可。我们的方法DualHeads将其作为固定扩散骨干上的轻量级头实现。奖励头学习同一提示中哪些样本得分更高，参考头将参考图像与生成图像分离，生成器最大化其logit的加权和。当头值最优时，目标是学习奖励的KL正则化最大化，头权比决定正则化强度。从经验角度看，结合基于规则和模型的奖励，几步DualHeads生成器能达到接近40步强化学习方法的任务奖励，同时保持图像质量，且无额外推理成本。我们希望这种简单优雅的双头表述能推动少步奖励倾向生成的发展。

RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

RL-IF：追踪非本地训练数据对在线强化学习的影响

• Authors: Shixuan Liu, Cheng Tang, Yuzheng Hu, Fan Wu, Han Zhao, Jiaqi W. Ma
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.28883
• Pdf link: https://arxiv.org/pdf/2610.28883
• Abstract Online reinforcement learning (RL) has achieved significant success in decision-making tasks, but modern online RL remains highly sensitive to the quality of training experience. Understanding the contribution of individual training instances is therefore central to data-centric RL, with applications ranging from experience selection to policy interpretation. Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non-local data influence across multiple training rounds. In this work, we formalize data attribution in online RL through a trajectory-level leave-one-out quantity, RL-LOO, which measures how one training sample influences a downstream target after subsequent online updates have taken place. We then derive RL-IF, a first-order estimator that propagates data influence through both optimization effects and policy-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions. Our analysis further disentangles these two effects and shows both theoretically and empirically that the full RL-IF estimator can often be well approximated using the optimization effect alone. Building on this observation, we develop RINSE, a practical data filtering method for online RL. Experiments on standard control benchmarks and an RLHF-style toxicity-mitigation setting show that RINSE improves training efficiency and final performance over standard PPO and local attribution baselines. These results suggest that RL-IF provides a practical and principled framework for understanding and improving online RL.
• 中文摘要 在线强化学习（RL）在决策任务中取得了显著成功，但现代在线强化学习对训练体验质量仍高度敏感。因此，理解单个训练实例的贡献是数据中心强化学习的核心，应用范围涵盖体验选择到策略解释等。现有在线强化学习的数据归因方法主要关注单一训练轮次内的局部影响，忽视了多个训练轮次中非局部数据的影响。本研究通过轨迹级的遗漏量RL-LOO形式化在线强化学习中的数据归因，该量度训练样本在后续在线更新后如何影响下游目标。随后我们推导出RL-IF，这是一种一阶估计量，通过优化效应和策略诱导的抽样效应传播数据影响，并在平滑性假设下提供其近似准确性的理论保证。我们的分析进一步解开了这两种效应，理论和实证都表明，仅用优化效应即可很好地近似完整的RL-IF估计器。基于这一观察，我们开发了RINSE，一种用于在线强化学习的实用数据过滤方法。在标准对照基准和RLHF类毒性缓解设置上的实验表明，RINSE相较于标准PPO和局部归因基线，提升了训练效率和最终表现。这些结果表明RL-IF为理解和改进在线强化学习提供了一个实用且有原则的框架。

JaxMARL-BC: Multi-Agent Reinforcement Learning for Macroeconomic Modelling, in Seconds

JaxMARL-BC：多智能体强化学习用于宏观经济建模，秒内完成

• Authors: Daniele Montagnani, Andrea Coletta, Antonietta Mira, Aldo Glielmo
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); General Economics (econ.GN)
• Arxiv link: https://arxiv.org/abs/2610.28907
• Pdf link: https://arxiv.org/pdf/2610.28907
• Abstract Multi-agent reinforcement learning (MARL) offers a principled way of populating simulated economies with many heterogeneous decision makers. Its adoption, pioneered in finance, has nonetheless been slow in macroeconomics because of two main limitations: the lack of a shared frame of reference connecting MARL systems to dynamic stochastic general equilibrium (DSGE) models the profession already trusts, and the very long training times. The first limitation was recently addressed by the Multi-Agent Reinforcement Learning Business Cycle (MARL-BC), which places deep RL households in a real business cycle economy and reduces to classic DSGE models in specific limits. Here we address the second limitation. We present JaxMARL-BC, a JAX-accelerated simulator that trains MARL-BC economies in seconds (between 75x and 724x faster than the reference implementation), and scales to 200,000 independent learning households. We verify its correctness against the known DSGE limits, characterise its scaling through a simple cost law, and demonstrate its value by a proof-of-concept calibration of an extended model with heterogeneous returns on capital, in a two-hour experiment that would previously have taken over a month. We release JaxMARL-BC as open source.
• 中文摘要 多智能体强化学习（MARL）提供了一种原则性的方式，将模拟经济体中填充许多异质决策者。尽管如此，其在金融领域率先采用，但在宏观经济学中采用速度较慢，原因有两个主要限制：缺乏将MARL系统与该行业已信任的动态随机一般均衡（DSGE）模型连接的共享参考框架，以及训练时间极长。第一个局限性最近被多智能体强化学习商业周期（MARL-BC）解决，该方案将深度强化学习家庭置于真实的商业周期经济中，并在特定限制下简化为经典的DSGE模型。这里我们讨论第二个限制。我们介绍JaxMARL-BC，一款JAX加速模拟器，能在几秒内训练MARL-BC经济（比参考实现快75倍到724倍），并可扩展至20万个独立学习家庭。我们验证其在已知DSGE限制下的正确性，通过简单成本定律表征其扩展性，并通过一个资本回报异质的扩展模型的概念验证校准，验证其价值，实验时间为两小时，原本需要一个多月时间。我们将JaxMARL-BC开源发布。

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

基于离散时间非策略数据的无模型连续时间强化学习算法，具有端到端保证

• Authors: Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2610.28915
• Pdf link: https://arxiv.org/pdf/2610.28915
• Abstract In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.
• 中文摘要 本文研究连续时间强化学习，该阶段状态遵循未知的随机微分方程，但仅有固定的离散时间转移批次。随着采样区间缩小，标准离散时间Q学习变得困难，因为动作值之间的差异消失。连续时间替代方案，如深度优势更新和马丁格尔q学习，保留动作信号，同时学习价值函数和动作依赖的批判者。基于PhiBE-Q表述（arXiv：2606.21925），我们开发了一个无模型的PhiBE策略迭代（PhiBE-PI）框架，使用单一状态动作批评者，同时保持连续时间动作排名尺度。每次迭代中，框架直接从采样增量学习批评者，并用以改进策略。同一批数据在整个过程中被重复使用，但未明确估计漂移或扩散。我们进一步发展线性函数近似实现，并建立了所学策略的局部端到端次优性保证。我们的分析量化了时间离散化、函数近似和有限样本策略评估的误差如何通过策略迭代传播。它考虑了连续策略对重用数据的依赖，因此保证不需要每次迭代重新批次。此外，我们识别出偏差-方差权衡：较小的抽样区间提升连续时间问题的近似，但在固定样本量下可能增加统计误差。因此，更细时间离散化不一定能带来更好的策略性能。

One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?

一项政策，多种风格：在哪里汇总对可控类人生物控制的偏好？

• Authors: Ziqian Huang, Yuzhuo Chen, Shimin Zhang, Junyang He, Xinyue Annie Yang, Animesh Garg
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28925
• Pdf link: https://arxiv.org/pdf/2610.28925
• Abstract Modern robotic agents are increasingly expected to support steerable control, where a single policy can adjust its behavioral style at runtime under changing user preferences. Preference-conditioned reinforcement learning provides this interface, but conditioning alone does not ensure that a requested behavior change is strong, does not leak into unrelated behaviors, and does not compromise task performance. Our evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics. We use these measures to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks. No aggregation level performs best across all environments and criteria. Experiments show that aggregating objectives early produces a wider range of behavioral responses but greater unintended changes in other objectives, while aggregating them later improves control precision at some cost to responsiveness.
• 中文摘要 现代机器人代理越来越多地被期望支持可引导控制，即单个策略可以在运行时根据用户偏好的变化调整其行为风格。偏好条件强化学习提供了该接口，但仅靠条件反射无法确保请求的行为变化强大，不会影响无关行为，也不会影响任务性能。我们的评估结合了偏好奖励响应性（PRR）与覆盖率和效用指标。我们利用这些指标比较两种类人控制框架下的奖励水平、优势水平、损失水平和基于PCGrad的聚合。没有聚合级别在所有环境和标准下表现最佳。实验表明，早期聚合目标会产生更广泛的行为响应范围，但其他目标的意外变化更大，而后期聚合则能提高控制精度，但对响应性有所牺牲。

SLoSa: Small, Low-Power, and Safe Foundation Models for Robotic Task Planning

SLoSa：小型、低功耗且安全的机器人任务规划基础模型

• Authors: Weizhe Xu, Jialiang Fan, Essien Taylor, Zijun Wang, Russell Joseph, Fanxin Kong
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Logic in Computer Science (cs.LO)
• Arxiv link: https://arxiv.org/abs/2610.28955
• Pdf link: https://arxiv.org/pdf/2610.28955
• Abstract Safe robotic task planning requires planners that can reliably achieve task goals while satisfying safety constraints. Classical planners search over explicit symbolic task representations and can provide formal guarantees with respect to encoded safety constraints, but require rigid input formats. Their computational cost also increases rapidly with the search space. In contrast, large language models (LLMs) offer flexible task interfaces and strong generalization capabilities, but they are large, power-hungry, and pre-trained with little robot-specific safety data and knowledge. Thus, they can be difficult to deploy locally on robotic systems with limited resources. To address these limitations, we present SLoSa, a neuro-symbolic compression and fine-tuning framework for training foundation-model-based robotic task planners with Small size, Low power demand, and Safety awareness. The framework first constructs a safety-aware dataset using classical planners, an advanced teacher LLM, and formal verifiers, followed by two training stages: supervised fine-tuning and reinforcement learning (RL) fine-tuning guided by a verification-based reward machine to enforce safety alignment. We evaluate SLoSa across four Planning Domain Definition Language (PDDL) domains, two household simulation environments, local edge platforms, and a desktop robotic arm. Across the PDDL tasks, verifier-guided post-training raises the success rate of compact 4B-14B planners to 70.5-76.0%, while INT4 deployment on a Jetson Orin requires 17.85-26.41 W of average power. Results in household simulators and on the physical robot further demonstrate the applicability of SLoSa across diverse robotic task settings.
• 中文摘要 安全的机器人任务规划需要规划者能够可靠地实现任务目标，同时满足安全约束。传统规划者会搜索显式符号任务表示，并能就编码的安全约束提供形式保证，但需要严格的输入格式。它们的计算成本也会随着搜索空间的增加迅速增加。相比之下，大型语言模型（LLMs）提供灵活的任务接口和强大的泛化能力，但体积庞大、耗电大且预训练，缺乏机器人专用的安全数据和知识。因此，在资源有限的机器人系统上本地部署它们较为困难。为解决这些局限性，我们提出了SLoSa，一种用于训练基于基础模型、体积小、功耗低且具备安全意识的机器人任务规划器的神经符号压缩和微调框架。该框架首先利用经典规划器、高级教师LLM和形式验证器构建一个安全意识数据集，随后进行两个培训阶段：监督式微调和基于验证的奖励机引导的强化学习（RL）微调以强制安全对齐。我们评估了SLoSa在四个规划领域定义语言（PDDL）域、两个家庭模拟环境、本地边缘平台和桌面机械臂上的应用。在PDDL任务中，验证者引导的后期训练将紧凑型4B-14B规划器的成功率提升至70.5-76.0%，而在Jetson Orin上部署INT4则平均功率为17.85-26.41瓦。在家庭模拟器和实体机器人上的结果进一步展示了SLoSa在多种机器人任务环境中的适用性。

Black-Box Red-Teaming of Multi-Agent Systems via Reinforcement Learning

通过强化学习实现多智能体系统的黑箱红团队

• Authors: Yuqi Jia, Hangsheng Zhang, Chenxi Liu, Junjie Liang, Xun Chen
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.28966
• Pdf link: https://arxiv.org/pdf/2610.28966
• Abstract Large language model (LLM) agents are increasingly deployed in multi-agent systems (MAS) to accomplish complex tasks. Prior black-box red-teaming attacks mainly focus on single agents, but we find that these methods are far less effective in MAS, where multiple sub-agents may not directly interact with the user. Therefore, we introduce ReMAS, the first reinforcement learning-based red-teaming framework tailored to MASs, which fine-tunes attacker LLMs to generate effective adversarial prompts for system prompt extraction. The framework follows a two-step process: first, a rewrite stage refines base attack prompts to increase extraction success, and second, a template generation stage constructs attack templates that improve the likelihood of invoking specific sub-agents and thus revealing their system prompts. Extensive experiments show that our method substantially improves attack success rates compared to existing black-box attacks, with transferability across different backend LLMs and MAS structures. These results underscore the vulnerabilities of multi-agent systems and the importance of developing stronger defenses against adaptive black-box adversaries.
• 中文摘要 大型语言模型（LLM）代理越来越多地被部署在多代理系统（MAS）中，以完成复杂任务。此前的黑箱红队攻击主要针对单个代理，但我们发现这些方法在MAS中效果远不如MAS，因为多个子代理可能无法直接与用户交互。因此，我们引入了ReMAS，这是首个基于强化学习的红团队框架，专为MAS量身定制，它微调攻击者LLM以生成有效的对抗提示以提取系统提示。该框架遵循两步过程：首先，重写阶段对基础攻击提示进行细化以提高提取成功率;其次，模板生成阶段构建攻击模板，以提高调用特定子代理并揭示其系统提示的可能性。大量实验表明，我们的方法相比现有黑箱攻击显著提升了攻击成功率，并且能够跨越不同后端LLM和MAS结构进行迁移。这些结果凸显了多智能体系统的脆弱性，以及开发更强防御自适应黑箱对手的重要性。

Doob-Based Generative Policy Optimization

基于Doob的生成策略优化

• Authors: Maojiang Su, Qijie Zhu, Yu Xiao, Shuyang Yu, Minshuo Chen, Zhaoran Wang, Han Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.28990
• Pdf link: https://arxiv.org/pdf/2610.28990
• Abstract Diffusion and flow models provide expressive policy classes for robotic and sequential decision-making systems. However, their action likelihoods are generally intractable, limiting the direct use of widely adopted policy optimization methods such as PPO and GRPO. We propose Doob Policy Optimization (DoobPO), a framework that leverages Doob's $h$-transform for likelihood-free reinforcement learning fine-tuning of generative policies. DoobPO separates policy improvement from generative policy learning. It first learns a lightweight ratio model that represents the desired policy update, and then distills the resulting ratio-tilted target policy into the generative policy via conditional drift matching. This formulation applies uniformly to both diffusion and flow policies and supports a broad family of policy improvement rules, including PPO and GRPO. We prove that the practical conditional drift matching objective is gradient-equivalent to the ideal Doob drift matching objective. We further show that the induced diffusion process solves an endpoint-constrained stochastic optimal control problem. This establishes that it achieves the minimum-energy modification required to realize the desired policy tilt. Across MuJoCo continuous-control benchmarks, DoobPO achieves strong performance against competitive diffusion- and flow-policy baselines, showing a principled and practical route to RL fine-tuning for generative policies.
• 中文摘要 扩散和流模型为机器人和顺序决策系统提供了表达式策略类。然而，它们的行动似然通常难以处理，限制了广泛采用的策略优化方法如PPO和GRPO的直接使用。我们提出了Doob策略优化（DoobPO），这是一个利用Doob的$h$变换进行生成策略无似然强化学习微调的框架。DoobPO将策略改进与生成策略学习区分开来。它首先学习一个轻量级比率模型，代表期望的策略更新，然后通过条件漂移匹配将生成倾斜的目标策略提炼进生成策略。该表述统一适用于扩散策略和流策略，并支持包括PPO和GRPO在内的广泛策略改进规则家族。我们证明了实际条件漂移匹配目标与理想的Doob漂移匹配目标在梯度上等价。我们还进一步证明，诱导扩散过程解决了端点约束的随机最优控制问题。这证明了它实现了实现所需策略倾斜所需的最小能量调整。在MuJoCo连续控制基准测试中，DoobPO在竞争性扩散和流量策略基线中表现出色，展示了生成策略强化学习微调的原则性和实用路径。

FlashNext-RL: A System Optimization Study for Long-Horizon Agentic RL of Qwen3.8-Flash-Next

FlashNext-RL：Qwen3.8-Flash-Next 长视野能动强化学习的系统优化研究

• Authors: Zhewei Yao, Yuxiong He
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.28998
• Pdf link: https://arxiv.org/pdf/2610.28998
• Abstract Long-horizon agentic reinforcement learning (RL) requires both efficient rollout generation and training on long sequences. However, the memory requirements and model-specific operators of Qwen3.8-Flash-Next make this difficult with existing open-source implementations. To resolve these limitations, we introduce FlashNext-RL, a model-specific RL training optimization framework based on prime-rl, with vLLM for software engineering (SWE) rollout generation. Our main focus is BF16 training of the full 48-layer model on a single 8xB200 node, with the first two layers, the n-gram table, the sparse-attention indexer and the multi-token prediction head frozen. We optimize host offload, gradient accumulation and the kernels for mixture-of-experts, sparse attention and the gated residual stream. In the full-model development study, non-padding training throughput increases from approximately 395 tokens/s/GPU in the initial 32K-token configuration to 4,438 tokens/s/GPU in the optimized 131K-token configuration, with hardware matmul throughput (aka without considering the cost of attention components) increasing from approximately 20 to 213 TFLOPs/GPU. A controlled cumulative study on an 8-layer model further quantifies the contributions of individual optimizations. Together with three inference nodes, the system supports a live SWE RL run with a target batch of 1,024 trainable rollouts and a 131,072-token context limit. Over steps 5-15, an iteration takes 22.4 minutes on average, with bounded entropy and trainer-sampler mismatch KL. These results highlight the importance of tailoring RL training to the model's architecture, memory layout and kernel requirements, rather than relying only on general-purpose RL infrastructure.
• 中文摘要 长视野代理强化学习（RL）需要高效的展开生成和长序列训练。然而，Qwen3.8-Flash-Next 的内存需求和模型特定操作符使现有开源实现难以实现。为解决这些限制，我们引入了基于prime-rl的模型专用RL训练优化框架FlashNext-RL，并支持软件工程（SWE）推广生成的vLLM。我们的主要关注点是在单个8xB200节点上训练完整的48层模型，前两层——n-gram表、稀疏注意力索引器和多标记预测头——保持冻结。我们优化了宿主卸载、梯度累积和核，以适应专家混合、稀疏注意力和门控残差流。在完整模型开发研究中，非填充训练吞吐量从初始32Ktoken配置中的约395token/s/GPU提升至优化后的131Ktoken配置中的4,438token/s/GPU，硬件matmul吞吐量（即不考虑注意力组件成本）从约20增加到213 TFLOPs/GPU。一项针对8层模型的受控累积研究进一步量化了各优化的贡献。系统结合三个推理节点，支持实时SWE强化学习运行，目标批次为1,024次可训练展开，上下文限制为131,072个token。在第5至15步中，一次迭代平均耗时22.4分钟，采用有界熵和训练器-采样器KL不匹配。这些结果强调了根据模型架构、内存布局和内核需求定制强化学习训练的重要性，而不仅仅是依赖通用强化学习基础设施。

Dissimilarity-Based Markov Decision Processes for Stabilizing State Distributions

基于差异性的马尔可夫判定过程用于稳定状态分布

• Authors: Xun Shen, Yuepeng Wang, Daniyal Sajedi-Hosseini, Akifumi Wachi, Ken Kawano, Yoshihiko Fujisawa, Mehrshad Sadria, Sébastien Gros, Katsuki Fujisawa
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.29024
• Pdf link: https://arxiv.org/pdf/2610.29024
• Abstract Reinforcement learning for stochastic systems typically optimizes expected cumulative rewards, while reliable and repeatable behavior also depends on how the state distribution evolves over time. Optimizing expected return alone does not guarantee that this distribution converges to a desired target or remains stable. This paper develops a theoretical framework for learning policies that stabilize state distributions. We formulate the problem as a Dissimilarity-based Markov Decision Process (DMDP), in which the policy acts on the current state distribution and the objective measures its dissimilarity to a target distribution. We establish sufficient conditions under which the optimal policy makes the target distribution asymptotically stable. To support policy learning, we introduce a finite-dimensional representation of distributions and show how stability guarantees for its parameter dynamics extend to the underlying state distributions. We further develop certificates for the learned policies, including their deployment using estimated distribution parameters. These results quantify how learning and estimation errors affect the guarantees and identify conditions under which learned policies retain asymptotic stability. For practical implementation, we combine distribution estimates and target-dependent rewards with PPO and SAC. A computable example on a linear system illustrates the proposed theoretical certificates. Furthermore, experiments on a HalfCheetah task demonstrate the benefits of distribution feedback and explicit target matching for regulating the velocity distribution.
• 中文摘要 随机系统的强化学习通常优化期望累积奖励，而可靠和可重复的行为也取决于状态分布随时间的变化。仅仅优化期望回报并不能保证该分布收敛于目标或保持稳定。本文构建了一个理论框架，用于学习稳定状态分布的策略。我们将问题表述为基于差异性的马尔可夫决策过程（DMDP），其中策略作用于当前状态分布，目标测量其与目标分布的差异。我们建立了充分条件，使最优策略使目标分布渐近稳定。为支持策略学习，我们引入了有限维分布表示，并展示了其参数动态的稳定性保证如何延伸至底层状态分布。我们进一步开发了学习策略的证书，包括使用估计分布参数的部署。这些结果量化了学习和估计误差如何影响保证，并识别了学习策略保持渐近稳定性的条件。为实现实际，我们将分布估计和目标相关奖励与PPO和SAC结合起来。一个线性系统的可计算示例展示了所提出的理论证书。此外，HalfCheetah任务的实验展示了分布反馈和明确目标匹配在调控速度分布方面的益处。

Value-Rectified Distillation for Flow-based Offline Reinforcement Learning

基于流程的离线强化学习中的价值纠正蒸馏

• Authors: Ke Jiang, Wen Jiang, Yoshinobu Kawahara, Xiaoyang Tan
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.29075
• Pdf link: https://arxiv.org/pdf/2610.29075
• Abstract We aim to address the problem of Out-of-Distribution (OOD) Mode Averaging, a critical problem that emerges when distilling Flow Matching (FM) based behavior policies in offline Reinforcement Learning (RL): When learning over multi-modal datasets, standard distillation inadvertently forces the policy to interpolate across different behavioral modes, causing the generated actions to collapse into invalid, OOD regions. To resolve this, we propose Value-Rectified Distillation (VRD), a novel framework that reformulates offline policy distillation as a value-guided generative trajectory alignment problem. By employing a dynamic coupling mechanism, VRD actively rectifies flow dynamics to explicitly decouple distinct behavioral modes, cleanly isolating high-value actions. Specifically, we instantiate VRD through two complementary algorithms: VRD-Q, which utilizes value-quantile behavior distillation to statistically filter out low-value behaviors, and VRD-R, which reorganizes the latent space to geometrically isolate high-value generative trajectories from the FM-based behavior policy. Extensive empirical evaluations demonstrate that VRD improves existing generative baselines across diverse continuous control and planning tasks on the D4RL, OGBench, and Minari benchmarks, especially over those multi-modal, non-expert datasets.
• 中文摘要 我们旨在解决Out-of-Distribution（OOD）模式平均问题，这是离线强化学习（RL）中基于流匹配（FM）的行为策略提炼时出现的关键问题：在多模态数据集上学习时，标准提纯无意中迫使策略跨不同行为模式插值，导致生成的动作崩溃到无效的OOD区域。为解决这一问题，我们提出了价值纠正蒸馏（VRD）这一新框架，将离线策略提炼重新表述为价值引导的生成轨迹对齐问题。通过采用动态耦合机制，VRD主动纠正流动态，明确解耦不同的行为模式，干净利落地隔离高价值动作。具体来说，我们通过两种互补算法实现VRD：VRD-Q利用价值分位行为蒸馏以统计性过滤低价值行为，以及VRD-R，重组潜在空间，几何上隔离高价值生成轨迹与基于FM的行为策略。大量实证评估表明，VRD在D4RL、OGBench和Minari基准测试中，特别是在多模态非专家数据集上，改进了多样化连续控制和规划任务中的现有生成基线。

State-Dependent Objective Coordination for End-to-End Balance and Navigation

端到端平衡与导航的状态相关目标协调

• Authors: Xuhao Huang, Xitao Xu, Yuxin Zheng, Yunfan Ren, Zheng Gong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.29142
• Pdf link: https://arxiv.org/pdf/2610.29142
• Abstract Traditional robotic navigation systems separate mapping, planning, and low-level control through cascaded modules, introducing computational overhead and accumulated latency. End-to-end reinforcement learning offers a compact unified policy that maps onboard observations directly to control actions. However, on inherently unstable platforms such as two-wheeled inverted-pendulum robots, navigation progress and obstacle avoidance can conflict with attitude stabilization in a strongly state-dependent manner. We propose a lightweight state-dependent objective coordination method for integrated navigation and control. A pitch-conditioned scalarization coefficient $\beta(s)\in[0,1]$ adjusts the relative contributions of balance stabilization and navigation objectives during training, without extra control hierarchies or auxiliary attitude-stabilization controllers, such as proportional-integral-derivative (PID) or linear-quadratic regulator (LQR) controllers. To support zero-shot deployment, simulation training models execution delay, action hold, asynchronous and noisy observations, and dynamics uncertainty. Gradient analysis reveals persistent directional interference and state-dependent variation in the objectives' relative scales. Comparisons with alternative reward-weighting, constraint, training, and control-structure strategies show that PPO+$\beta(s)$, where PPO denotes proximal policy optimization, achieves 98% and 95% success rates with small attitude excursions on the fixed obstacle map and the randomly generated high-obstacle-density map, respectively. Zero-shot hardware experiments further demonstrate bounded attitude behavior and consistent obstacle avoidance using only onboard observations. Training and evaluation code will be made publicly available upon acceptance.
• 中文摘要 传统机器人导航系统通过级联模块将地图绘制、规划和低层控制分离，增加了计算开销和累积延迟。端到端强化学习提供了紧凑统一的策略，将机载观测直接映射到控制动作。然而，在本质上不稳定的平台，如双轮倒摆机器人，导航进度和避障可能与姿态稳定发生强烈状态依赖性冲突。我们提出了一种轻量级状态依赖目标协调方法，用于集成导航与控制。俯仰条件标量系数 $\beta（s）\in[0,1]$ 在训练过程中调整平衡稳定和导航目标的相对贡献，无需额外的控制层级或辅助姿态稳定控制器，如比例积分微分（PID）或线性二次调节器（LQR）控制器。为支持零样本部署，模拟训练模型对执行延迟、动作保持、异步和噪声观测以及动力学不确定性进行训练。梯度分析揭示了目标相对尺度中持续存在的方向干扰和状态依赖性变化。与替代奖励加权、约束、训练和控制结构策略的比较显示，PPO+$\beta（s）$（其中PPO表示近端策略优化）在固定障碍地图和随机生成的高障碍密度地图上，分别以小幅姿态偏移实现98%和95%的成功率。零样本硬件实验进一步展示了仅用机载观测即可实现有界姿态行为和一致的障碍物规避。训练和评估代码将在接受后公开。

A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment

基于学习的控制十年：从理论基础到实际应用

• Authors: Angela P. Schoellig, Matthias A. Müller, Sebastian Trimpe, Melanie N. Zeilinger, Mahathi Anand, Andrea Carron, Oliver Hausdörfer, Victor G. Lopez, Friedrich Solowjow, Haocheng Zhao, Siqi Zhou
• Subjects: Subjects: Systems and Control (eess.SY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.29187
• Pdf link: https://arxiv.org/pdf/2610.29187
• Abstract Motivated by ten years of Learning-Based Control Invited Sessions at the IEEE Conference on Decision and Control (CDC), this tutorial paper reviews key advances at the intersection of machine learning and control. We focus on methods that combine the flexibility of machine learning techniques with the structure and theoretical foundations of control to design autonomous systems that are capable, reliable, and safe. We first introduce core notions for establishing stability and safety guarantees, then review approaches that learn dynamics models to improve control under uncertainty, and finally discuss data-driven methods that directly synthesize or learn policies, including reinforcement learning, imitation learning, and Bayesian optimization. We further discuss representative applications and open challenges. The goal is to provide an accessible entry point to the field and to support continued progress toward reliable learning-based control in real-world systems. To support learning and practical exploration, we share implementations and illustrative examples along with our discussion, together with additional resources on our website: this https URL.
• 中文摘要 本教程论文受IEEE决策与控制会议（CDC）十年来基于学习的控制邀请会议的启发，回顾了机器学习与控制交叉点的关键进展。我们聚焦于将机器学习技术的灵活性与控制的结构和理论基础结合的方法，以设计具备能力、可靠且安全的自主系统。我们首先介绍建立稳定性和安全保障的核心理念，然后回顾学习动力学模型以改善不确定性控制的方法，最后讨论直接综合或学习策略的数据驱动方法，包括强化学习、模仿学习和贝叶斯优化。我们还进一步讨论了代表性的应用和未解决的挑战。目标是为该领域提供一个可访问的切入点，并支持在真实世界系统中实现可靠基于学习的控制的持续进展。为支持学习和实践探索，我们在讨论中分享实现和示例，并在我们的网站上提供更多资源：https URL。

Reasoning-Search-Augmented LLMs: An Operational Taxonomy, Benchmark Analysis and Controlled Evidence Synthesis

推理搜索增强型大型语言模型：操作分类法、基准分析与受控证据综合

• Authors: Biswas Poudel, Nilson Chapagain, Amit Kumar, Xianshun Jiang
• Subjects: Subjects: Information Retrieval (cs.IR); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.29189
• Pdf link: https://arxiv.org/pdf/2610.29189
• Abstract Large language models (LLMs) demonstrate impressive emergent reasoning capabilities, yet they remain constrained by fixed model parameters and finite context windows. Even when augmented with static retrieval, they can produce statements that are unsupported by, or even contradictory to, retrieved evidence. This leads to hallucinations, difficulty incorporating up-to-date information, and failures to correctly interpret or reconcile sources. Recent work addresses these limitations by tightly coupling step-by-step reasoning with interactive search, treating retrieval as an action within an iterative agentic loop rather than a one-shot preprocessing step. This study reviews frameworks that implement such reasoning-search loops by interleaving planning, querying, retrieval, integration, verification, and stopping. We analyze these systems through four architectural axes: retrieval coupling, controller design, knowledge source, and grounding strictness. We separately treat the learning signal as a cross-cutting dimension because prompting, supervision, preference optimization, and reinforcement learning can instantiate the same architecture while producing materially different search, stopping, and grounding behavior. We further distinguish benchmark difficulty from RSA-loop demand and synthesize controlled within-study ablations and quality-cost deltas, avoiding invalid pooling across heterogeneous model backbones, retrievers, corpora, and evaluation protocols. Our goal is to clarify the landscape of reasoning-search-augmented LLMs and outline a unified perspective on the design choices shaping future research.
• 中文摘要 大型语言模型（LLM）展现了令人印象深刻的涌现推理能力，但它们仍受制于固定的模型参数和有限的上下文窗口。即使加入静态检索，它们仍可能产生与检索证据不支持甚至矛盾的陈述。这导致幻觉、难以整合最新信息，以及无法正确解释或调和来源。近期研究通过将逐步推理与交互式搜索紧密结合，将检索视为迭代代理循环中的一个动作，而非一次性的预处理步骤，来解决这些局限性。本研究回顾了通过交错规划、查询、检索、集成、验证和停止来实现此类推理-搜索循环的框架。我们通过四个架构轴分析这些系统：检索耦合、控制器设计、知识源和基础严格性。我们将学习信号单独视为跨领域维度，因为提示、监督、偏好优化和强化学习可能实现同一架构，同时产生实质性不同的搜索、停止和接地行为。我们还区分了基准难度与RSA环需求，综合研究内受控消融和质量成本差值，避免跨异构模型骨干、检索器、语料库和评估协议的无效池化。我们的目标是澄清推理搜索增强LLM的格局，并统一地阐述未来研究设计选择的视角。

MA-UITD: Offline Multi-Agent Reward Inference for Characterizing Continuous Social Behavior

MA-UITD：用于刻画连续社会行为的离线多智能体奖励推断

• Authors: Zeyuan Zhang, Yiyun Yang, Yuan Cheng, Herbert Zheng Wu, Yusi Chen
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.29237
• Pdf link: https://arxiv.org/pdf/2610.29237
• Abstract Multi-agent inverse reinforcement learning offers a powerful framework not only for imitation learning but also for understanding the neural computations supporting social interactions and decision-making. Inferring rewards from interacting agents is challenging because observed choices reflect both personal preferences and expectations about others. Existing methods often require additional environment interaction, repeated policy optimization as an inner loop, or restrictive assumptions about environment dynamics and action spaces, limiting their applicability to fixed behavioral datasets. We introduce Multi-Agent Unilateral Inverse Temporal Difference Learning (MA-UITD), an offline method for inferring explicit agent-specific rewards from joint demonstrations in continuous-state environments with either discrete or bounded continuous actions. Its core idea is to represent the long-term consequences of each agent's own actions through belief-conditioned successor features that average over other agents' behavior. Shared reward features and agent-specific preference vectors connect these unilateral values to the target reward function, while inverse temporal-difference consistency and noise-contrastive behavioral learning enable inference without known dynamics or an inner policy-optimization loop. Across simulated cooperative, competitive, and general-sum tasks, MA-UITD outperforms baselines in discrete-action settings and infers behaviorally compatible reward profiles in continuous-action settings. When applied to experimentally collected social-foraging trajectories, MA-UITD inferred distinct spatial and interaction-dependent reward structures for leaders and followers, providing a quantitative characterization of latent behavioral objectives that may be relevant for understanding the neural computations underlying social behavior.
• 中文摘要 多智能体逆强化学习不仅为模仿学习提供了强大框架，也为理解支持社会互动和决策的神经计算提供了支持。从交互智能体中推断奖励具有挑战性，因为观察到的选择既反映了个人偏好，也反映了对他人的期望。现有方法通常需要额外的环境交互、反复策略优化作为内循环，或对环境动态和动作空间的限制性假设，限制了其适用于固定行为数据集。我们介绍了多智能体单边反时差学习（MA-UITD），这是一种离线方法，用于通过连续状态环境中具有离散或有界连续动作的联合演示推断显式智能体特定奖励。其核心思想是通过信念条件的继任特征来表示每个智能体行为的长期后果，这些特征对其他智能体的行为进行平均。共享奖励特征和代理特异偏好向量将这些单向值与目标奖励函数连接起来，而反时间差一致性和噪声对比行为学习则使得无需已知动态或内部策略优化循环即可推断。在模拟的合作、竞争和一般求和任务中，MA-UITD在离散行动环境中优于基线表现，并在连续行动环境中推断行为兼容的奖励组合。当应用于实验收集的社会采集轨迹时，MA-UITD推断出领导者和追随者的独特空间和互动依赖奖励结构，提供了潜在行为目标的定量特征，可能对理解社会行为背后的神经计算具有参考价值。

Training in Deployment: Human-in-the-loop Online Post-Training for Robot Manipulation

部署培训：机器人操作的在线人机后期培训

• Authors: Chun Gu, Xiaofei Wei, Yiming Chen, Bozhou Zhang, Juntong Zhang, Xiatian Zhu, Li Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.29264
• Pdf link: https://arxiv.org/pdf/2610.29264
• Abstract Large pretrained robot foundation models provide strong priors for manipulation, yet adapting them to target tasks with offline teleoperated demonstrations often fails to cover the contact-rich, recovery, and precision-critical states encountered during deployment. Existing methods mitigate this issue by augmenting offline demonstrations with human-in-the-loop data or online reinforcement learning. Despite this, offline post-training adapts slowly because it cannot immediately absorb deployment experience, while online RL typically relies on value functions or auxiliary actor-critic models, increasing both training complexity and instability. We present TIDE (Training In DEployment), a human-in-the-loop online post-training system. TIDE introduces a provenance-conditioned supervised learning framework that updates the policy online from human interventions and policy-generated rollouts collected during deployment, without a learned critic, reward model, or action scorer. Across four real-world bimanual manipulation tasks, TIDE improves milestone-normalized task progress and binary full-task completion over the warm-start policy and matched additional offline demonstrations, and performs comparably to the fixed-policy collection baseline while making updated checkpoints available during deployment. Each task-specific online post-training session, from the start of deployment rollouts until the final checkpoint is available, completes in less than one hour while learning and checkpoint handling run asynchronously.
• 中文摘要 大型预训练机器人基础模型为操作提供了强先验，但将其调整为目标任务并用离线远程操作演示，往往无法涵盖部署过程中接触丰富、恢复和精度关键状态。现有方法通过用人机在环数据或在线强化学习中补充离线演示来缓解这一问题。尽管如此，离线后训练适应缓慢，因为它无法立即吸收部署经验，而在线强化学习通常依赖价值函数或辅助演员-批评模型，增加了训练复杂性和不稳定性。我们介绍TIDE（Training In DEployment），这是一个人在环路的在线后培训系统。TIDE引入了一种来源条件的监督学习框架，在线更新策略，基于部署期间收集的人类干预和政策生成的推广，无需学习批评者、奖励模型或行动评分器。在四个实际的双手操作任务中，TIDE提升了里程碑归一化任务进度和二进制全任务完成率，相较于热启动策略和匹配的额外离线演示，表现与固定策略收集基线相当，同时在部署期间提供更新检查点。每次任务特定的在线后培训课程，从部署部署开始到最终检查点可用，在学习和检查点处理异步运行期间，不到一小时内完成。

Flat-RLVR: Sharpness-Aware Reinforcement Learning with Verifiable Rewards for Reasoning Capacity in LLMs

Flat-RLVR：大语言模型中推理能力可验证的锐利感知强化学习

• Authors: Yeonbum Kim, Hyun Kyu Lee, Sung Whan Yoon
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.29281
• Pdf link: https://arxiv.org/pdf/2610.29281
• Abstract Reinforcement learning with verifiable rewards (RLVR), particularly through group-relative policy optimization (GRPO), has emerged as a central paradigm for the post-training of large language models (LLMs). While RLVR is highly effective at enhancing the reasoning capabilities of LLMs, it commonly induces \textit{distribution sharpening}, whereby the policy distribution becomes excessively peaked and rare-yet-correct trajectories are progressively suppressed, thus diminishing sample diversity. To this end, we promote flat-reward optima in RLVR to smooth the policy distribution. We theoretically link flat reward RLVR to reduced distribution sharpening, and empirically show improved sample diversity while maintaining single-attempt accuracy. Our method, called \texttt{Flat}-RLVR, searches for flat reward regions via sharpness-aware minimization during LLM post-training, combining this with GRPO variants to achieve better trade-offs between sampling efficiency and reasoning capacity.
• 中文摘要 带可验证奖励的强化学习（RLVR），特别是通过群体相对策略优化（GRPO），已成为大型语言模型（LLM）后训练的核心范式。虽然RLVR在增强LLM推理能力方面非常有效，但它通常会引发\textit{分布锐化}现象，即策略分布变得过于峰值，且稀有但正确的轨迹逐渐被抑制，从而降低样本多样性。为此，我们在RLVR中推广扁平奖励最优，以平滑策略分布。理论上我们将平面奖励RLVR与减少分布锐化联系起来，实证显示样本多样性提升，同时保持单次尝试准确性。我们的方法\texttt{Flat}-RLVR，在LLM后训练期间通过锐利感应最小化寻找扁平奖励区域，结合GRPO变体，实现抽样效率与推理能力之间的更好权衡。

DISRL: Dual-Scale Incremental Semi-Supervised Reinforcement Learning for Dynamic Community Tracking

DISRL：双尺度增量半监督强化学习，用于动态社区追踪

• Authors: Tianming Zhang, Jinxu Wang, Yixuan Zhang, Zhengyi Yang, Chenyu Hou, Bin Cao, Jing Fan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.29315
• Pdf link: https://arxiv.org/pdf/2610.29315
• Abstract Dynamic local community detection tracks the evolving community of designated seed nodes across graph snapshots. However, it faces two critical challenges: temporal supervision uncertainty, where sparse and outdated labels fail to provide seed-specific guidance under topology changes, and unbounded error cascades, where sequential policy transfer accumulates exploration errors over long horizons. To address these challenges, we propose DISRL, a Dual-scale Incremental Semi-supervised Reinforcement Learning framework. DISRL introduces a supervision construction module that dynamically generates seed-specific guidance from sparse known communities, and a dual-scale community exploration module that stabilizes policy evolution by decoupling short-term adaptation within temporal windows from long-term parameter consolidation across windows. Furthermore, we construct a dynamic graph benchmark with diverse community evolution scenarios. Extensive experiments demonstrate that DISRL consistently outperforms state-of-the-art baselines under various dynamic settings.
• 中文摘要 动态局部社区检测跟踪指定种子节点在图快照中不断演变的社区。然而，它面临两个关键挑战：时间监督不确定性，稀疏且过时的标签在拓扑变化下无法提供种子特定指导;以及无界错误级联，即连续策略转移在长期范围内累积探索错误。为应对这些挑战，我们提出了DISRL，一种双尺度增量半监督强化学习框架。DISRL引入了一个监督构建模块，能从稀疏已知社区动态生成种子特定指导，以及一个双尺度社区探索模块，通过将时间窗口内的短期适应与跨窗口的长期参数整合解耦，稳定策略演变。此外，我们构建了一个动态图基准，包含多样社区演化场景。大量实验表明，DISRL在各种动态环境下始终优于最先进的基线。

AdvFidelity: Unifying Preference Alignment and Few-Step Image Generation within Reinforcement Learning

AdvFidelity：强化学习中统一偏好对齐与少步图像生成

• Authors: Sicong Kang, Fangao Zeng, Yuhan Li, Hao Zhou, Mengfei Xu, Wenxiang Shang, Wei Li, Bingbing Ni, Pipei Huang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.29380
• Pdf link: https://arxiv.org/pdf/2610.29380
• Abstract Practical image generation requires both human preference alignment and few-step efficiency, yet combining reinforcement learning (RL) with distillation is often multi-stage and may erode RL gains We observe that few-step RL progressively improves structural coherence, even with specialized rewards, but leaves low-level artifacts that these rewards may fail to penalize. To address this gap, we propose AdvFidelity, which integrates artifact-sensitive fidelity supervision into RL through an online adversarial reward. This enables joint reward alignment and high-fidelity few-step generation within a single RL stage, without distribution matching distillation (DMD) or other explicit distillation objectives. We train a discriminator to rank high-quality references (positives) above content-matched reconstructions (negatives) produced by the current generator. Reconstructing moderately noised references largely preserves style and scene layout while exposing low-level degradations. This encourages the discriminator to assess artifacts rather than content differences. With negatives reconstructed online, alternating updates adapt fidelity supervision to the generator's evolving failure modes. References can come from existing target-domain data or full-step model samples. Built on LeapAlign, AdvFidelity backpropagates gradients from both rewards through the few-step sampling trajectories used at inference. Across two model backbones and three task settings, AdvFidelity matches full-step LeapAlign in alignment, task performance, and overall generation quality and remains highly competitive with other evaluated full-step RL methods, using only 4 or 8 steps without CFG.
• 中文摘要 实际图像生成既需要人类偏好对齐，也需要少量步骤效率，但将强化学习（RL）与提纯结合往往是多阶段的，可能会侵蚀强化学习的收益。我们观察到，即使是专门奖励，也能逐步提升结构一致性，但会留下低层次的伪影，这些奖励可能无法惩罚这些缺陷。为弥补这一空白，我们提出了AdvFidelity，它通过在线对抗性奖励将工件敏感性保真度监督整合进RL。这使联合奖励比对和高保真度的少数步骤生成能够在同一RL阶段实现，而无需分布匹配蒸馏（DMD）或其他显式提炼目标。我们训练判别器将高质量参考（正值）排在当前生成器产生的内容匹配重建（负面）之上。重建中度噪声参考在很大程度保留了样式和场景布局，同时暴露了低层次退化。这鼓励判别器评估伪影而非内容差异。负片在线重建后，交替更新使保真度监督适应生成器不断演变的故障模式。参考可来自现有目标域数据或全步模型样本。基于LeapAlign，AdvFidelity通过推断时使用的少步采样轨迹反向传播两个奖励的梯度。在两个模型骨干和三种任务设置中，AdvFidelity在对齐、任务性能和整体生成质量上与全步LeapAlign匹配，且在仅4或8步且不含CFG的情况下，与其他评估过的全步强化学习方法保持高度竞争力。

The Predictive--Optimization Gap in GuanDan Self-Play

关丹自玩中的预测优化差距

• Authors: Qicheng Zou, Biao Yuan, Siyu Song, Wentao Liu, Xiaokun Wang
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.29411
• Pdf link: https://arxiv.org/pdf/2610.29411
• Abstract Outcome-predictive signals are often treated as promising optimization targets in reinforcement learning, but predictive validity and optimization utility are different claims. We expose this distinction in GuanDan, a multi-round imperfect-information game in which the implication of a round outcome depends on the public match state. Held-out analysis shows that state--gain interactions provide incremental predictive value beyond a constant signed-gain effect, establishing that match progress is predictively meaningful. We then use changes in a frozen match-win predictor, $\Delta\Phi$, as a controlled round-level supervision intervention in a matched DanZero-style Monte Carlo self-play protocol. The intervention can succeed in an individual lineage: a selected progress-augmented checkpoint substantially outperforms its matched Baseline on a locked evaluation. Yet the method-level result reverses this impression: none of five preregistered replication seeds is positive, the pooled ten-seed interval includes zero, and across-iteration effects repeatedly change sign without a common late-stage advantage. These results identify a predictive--optimization gap: under the studied learner, a signal can describe outcome-relevant progress while failing to provide a reliably beneficial optimization direction.
• 中文摘要 结果预测信号常被视为强化学习中的有前景的优化目标，但预测效度和优化效用则是不同的主张。我们在关丹（GuanDan）中揭示了这一区别，该博弈多轮不完美信息博弈，其中轮次结果的含义依赖于公开匹配状态。已有分析显示，状态-收益交互提供了超越恒定带符号增益效应的增量预测价值，证明匹配进展具有预测意义。随后，我们将冻结匹配-胜利预测变量$\Delta\Phi$的变化作为匹配DanZero风格蒙特卡洛自玩协议中的受控轮级监督干预。该干预在单个谱系中可能成功：选定的进度增强检查点在锁定评估中显著优于匹配基线。然而，方法层面的结果却颠覆了这一印象：五个预注册的复制种子都没有正面，合并后的十个种子间隔为零，且跨迭代效应反复变符号，但没有共同的后期优势。这些结果揭示了预测与优化之间的差距：在被研究学习者下，信号可以描述与结果相关的进展，但无法提供可靠且有益的优化方向。

BIRD-RL: Scaling Agentic Reinforcement Learning over Stateful Data-Centric Environments

BIRD-RL：在有状态数据中心环境中扩展智能力强化学习

• Authors: Ge Qu, Jinyang Li, Xiaohan Xu, Xiaolong Li, Bowen Qin, Nan Huo, Ziwei Tang, Chenhao Ma, Reynold Cheng
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.29426
• Pdf link: https://arxiv.org/pdf/2610.29426
• Abstract While agentic reinforcement learning (RL) has emerged as a promising paradigm for training small language models (SLMs), existing frameworks are predominantly tool-centric and struggle with data-centric environments. Specifically, applying tool-centric frameworks to stateful tasks like SQL debugging introduces severe spatial inefficiencies since proper environment isolation typically requires a separate database container per rollout. Naively reusing databases without tracking evolving states across multi-turn interactions causes trajectory contamination and cross-trajectory interference, which severely destabilizes policy updates. To overcome these bottlenecks, we introduce BIRD-RL, a novel four-stage agentic RL curriculum that alternately enhances foundational agent reasoning and trajectory-adaptive on-policy exploration to circumvent the instability inherently associated with direct end-to-end learning. Crucially, to support this curriculum, BIRD-RL is driven by a Trajectory-scoped Persistent Execution Infrastructure, which enables safe database replica reuse while preserving trajectory isolation. This design improves spatial RL efficiency by 12.8x, successfully enabling scalable agentic RL for complex data-centric tasks without needs of Kubernettes. Empirical evaluations on the BIRD-CRITIC-SQLite benchmark demonstrate that our 7B and 14B models achieve Success Rates of 44.40% and 48.00%, respectively, showing competitive performance against strong proprietary model-based agents such as Claude-Opus-4.6 Agent (48.20%). Furthermore, BIRD-RL successfully trains unified SLMs capable of both advanced SQL debugging and generation, highlighting its broader potential for multi-objective optimization across heterogeneous, real-world data-centric tasks.
• 中文摘要 虽然代理强化学习（RL）已成为训练小型语言模型（SLMs）的有前景范式，但现有框架主要以工具为中心，难以适应数据中心环境。具体来说，将工具中心框架应用于如SQL调试等有状态任务会带来严重的空间效率，因为适当的环境隔离通常每次部署都需要单独的数据库容器。如果不跟踪多回合交互中演变状态，简单地重复使用数据库会导致轨迹污染和交叉轨迹干扰，严重破坏策略更新。为克服这些瓶颈，我们引入了BIRD-RL，一种新的四阶段代理强化学习课程，交替增强基础代理推理和轨迹自适应策略探索，绕过直接端到端学习固有的不稳定性。关键是，为支持该课程，BIRD-RL采用轨迹范围的持久执行基础设施，实现数据库副本的安全复用，同时保持轨迹隔离。该设计将空间强化学习效率提升12.8倍，成功实现可扩展的代理强化学习，适用于复杂数据中心任务，无需Kubernettes。基于BIRD-CRITIC-SQLite基准测试的实证评估显示，我们的7B和14B模型分别实现44.40%和48.00%的成功率，与Claude-Opus-4.6代理（48.20%）相比具有竞争力。此外，BIRD-RL成功训练了能够进行高级SQL调试和生成的统一SLMs，凸显其在异构、以现实世界为中心的数据任务中实现多目标优化的更广泛潜力。

One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents

一行为，多重掩膜：长视角代理的效应边际策略优化

• Authors: Shaojin Ma, Ruitao Wang, Pengyi Li, Jie Yang, Liangyu Chen, Gangyi Zhao, Hongyao Tang, Yan Zheng, Jianye Hao
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.29427
• Pdf link: https://arxiv.org/pdf/2610.29427
• Abstract Recent advances in multi-turn reinforcement learning (RL) have substantially improved the performance of LLM agents on complex, long-horizon interactive tasks. However, despite progress in fine-grained credit assignment, policy optimization still applies these learning signals through response-specific probability ratios. Different textual responses can induce the same executable effect, yet credit differences arising from subsequent outcomes remain tied to their individual textual realizations. We identify this mismatch as credit-update misalignment. Our analysis reveals that credit-text pairing directly shapes policy-update directions, even when the policy context and executable effect are held fixed. Motivated by this finding, we propose Effect-Marginal Policy Optimization (EMPO), which aligns policy optimization with execution-level equivalence through an effect-level surrogate that couples updates across alternative textual realizations. EMPO preserves the original credit signals, integrates with both trajectory-level and step-level credit estimators, and requires no additional critics or rollouts. Experiments on ALFWorld, WebShop, and Sokoban show that EMPO outperforms strong baselines in both task performance and learning efficiency.
• 中文摘要 多回合强化学习（RL）的最新进展显著提升了LLM代理在复杂、长视野交互任务中的表现。然而，尽管细粒度信用赋值取得了进展，策略优化仍通过响应特异概率比应用这些学习信号。不同的文本响应可能诱导相同的可执行效果，但后续结果产生的功绩差异仍与其文本实现相关联。我们将这种不匹配称为功劳-更新错配。我们的分析显示，功劳-文本配对直接影响策略更新方向，即使政策语境和可执行效果固定。基于这一发现，我们提出了效果边际策略优化（EMPO），通过一个效应级代理将策略优化与执行层等价对齐，将更新与替代文本实现耦合。EMPO保留了原始的信用信号，集成了轨迹级和阶级级信用估算器，无需额外的批评或推广。在ALFWorld、WebShop和Sokoban上的实验显示，EMPO在任务表现和学习效率方面均优于强基线。

Distance Field Driven Reinforcement Learning for Reactive and Robust Collision Avoidance

远程场驱动强化学习用于被动和稳健碰撞避免

• Authors: Taiga Ikeshima, Sheila Sutjipto, Lan Wu, Francisco J. Naranjo-Campos, Taiki Ishigaki, Taisuke Kobayashi, Teresa A. Vidal-Calleja, Eiichi Yoshida
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.29452
• Pdf link: https://arxiv.org/pdf/2610.29452
• Abstract Robust and reactive collision avoidance remains a central challenge for deploying Reinforcement Learning (RL) policies on real robotic manipulators, particularly when transferring behaviours learned in simulation to the physical world. This paper presents an RL framework that integrates the Euclidean Distance Field (EDF) directly into the observation space, providing the agent with continuous geometric cues that support stable and responsive avoidance behaviours. By leveraging the rich spatial information encoded in the EDF, the method enables a streamlined reward formulation that achieves effective obstacle avoidance both with and without Artificial Potential Field guidance. The proposed approach is validated on a 7-DoF redundant manipulator, demonstrating zero-shot transfer from simulation to real hardware. Despite being trained exclusively with randomised static objects, the deployed policy generalizes to previously unseen real environments and exhibits robust online avoidance of dynamic obstacles. These results highlight the potential of EDF-based representations to mitigate the sim-to-real gap and support reliable and real-time collision-aware behaviour in complex and changing environments.
• 中文摘要 稳健且被动的碰撞规避仍是部署强化学习（RL）策略于真实机器人操作器时的核心挑战，尤其是在将仿真中学到的行为转移到物理世界时。本文提出了一个强化学习框架，将欧几里得距离场（EDF）直接集成到观察空间，为智能体提供连续的几何线索，支持稳定且响应灵敏的规避行为。通过利用EDF编码的丰富空间信息，该方法实现了简化的奖励表述，能够在有无人工势场引导的情况下实现有效的障碍避让。该方法在7-DoF冗余操作器上得到了验证，展示了从模拟到真实硬件的零射击转移能力。尽管仅用随机静态对象训练，该策略可推广到此前未见过的真实环境，并表现出对动态障碍物的强劲在线避让。这些结果凸显了基于EDF表示的潜力，能够缓解模拟与现实之间的差距，并支持在复杂且不断变化的环境中实现可靠且实时的碰撞感知行为。

Harness Scaling for Generalizable Agentic Reinforcement Learning

用于广义能动强化学习的束带尺度

• Authors: Fei Bai, Huatong Song, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.29458
• Pdf link: https://arxiv.org/pdf/2610.29458
• Abstract Agent harnesses have become a central component of modern agentic systems, shaping how models use tools, manage context, and coordinate long-horizon interaction. Realizing the benefits of these harnesses depends on how effectively the underlying model can make use of them, motivating reinforcement learning directly through the harness. However, existing approaches typically train through a single fixed harness, which can lead to specialization and limited transfer across harness configurations. To address this, we introduce \textbf{Harness Scaling}, a new scaling dimension for agentic RL that extends training from a single harness to diverse execution configurations. Concretely, we construct a factorized harness space, select a compact training subset with balanced factorial coverage, and organize cross-harness training to combine global coverage with controlled variation for the same task. Across multiple benchmarks, model scales, and RL algorithms, Harness Scaling consistently improves generalization to unseen white-box configurations. Moreover, models trained through structured white-box variation transfer effectively to unseen external black-box harnesses, outperforming models trained through a single complex black-box harness. These results establish harness diversity as a meaningful scaling dimension for agentic RL and highlight the importance of how this diversity is organized for cross-harness generalization.
• 中文摘要 代理工具工具已成为现代智能体系统的核心组成部分，塑造了模型如何使用工具、管理上下文以及协调长视野交互的方式。实现这些工具的优势取决于底层模型如何有效利用它们，从而直接通过框架激励强化学习。然而，现有方法通常通过单一固定框架进行训练，这可能导致专门化和限制了不同框架配置之间的转移。为此，我们引入了 \textbf{Harness Scaling}，这是一种新的智能强化维度，将训练从单一框架扩展到多样化的执行配置。具体来说，我们构建了一个因式化框架空间，选择一个均衡因子覆盖的紧凑训练子集，并组织跨框架训练，将全局覆盖与受控变异结合，用于同一任务。在多个基准测试、模型尺度和强化学习算法中，框架尺度持续提升对未见白盒配置的推广性。此外，通过结构化白盒变异训练的模型能够有效地转移到未被看到的外部黑匣子框架，优于通过单一复杂黑盒框架训练的模型。这些结果确立了束具多样性作为代理强化学习有意义的尺度维度，并凸显了这种多样性组织方式对于跨框架泛化的重要性。

Parallel Thinking for Large Language Model with Path Diversity

具有路径多样性的大型语言模型的并行思维

• Authors: Jiaru Qian, Yihong Dong, Xue Jiang, Chun-Wah Hsu, Zhi Jin, Ge Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.29501
• Pdf link: https://arxiv.org/pdf/2610.29501
• Abstract Parallel thinking has emerged as a promising paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs) by exploring multiple reasoning paths simultaneously. However, we identify that parallel thinking via Reinforcement Learning (RL) typically suffers from the mode collapse of reasoning paths, where parallel thinking trajectories degenerate into similar ones. This homogeneity undermines the fundamental motivation of parallel thinking, that is, maintaining multiple hypotheses concurrently to prevent premature loss into a suboptimal solution. In this paper, we propose Diverse Parallel Thinking (DPT), a simple yet effective RL approach for parallel thinking designed to encourage path diversity. DPT rewards the model to develop distinct reasoning strategies by penalizing the prefix similarity of parallel thinking paths, thereby enhancing a more comprehensive exploration of the reasoning space. We conduct various experiments and analyses across multiple reasoning benchmarks to demonstrate the effectiveness of DPT. Results show that DPT successfully outperforms previous parallel thinking methods, achieving state-of-the-art performance over baselines. We validate path diversity as a key element to improve parallel thinking. Our code is available at this https URL.
• 中文摘要 并行思维已成为一种有前景的范式，能够通过同时探索多条推理路径，增强大型语言模型（LLMs）的推理能力。然而，我们发现通过强化学习（RL）进行的并行思维通常存在推理路径模式崩溃的问题，即平行思维轨迹退化为相似路径。这种同质性削弱了并行思维的根本动机，即同时维持多个假设以防止过早丢失为次优解。本文提出了多元平行思维（DPT），这是一种简单但有效的强化学习并行思维方法，旨在促进路径多样性。DPT通过惩罚平行思维路径的前缀相似性来奖励模型开发不同推理策略，从而增强对推理空间更全面的探索。我们通过多个推理基准测试进行各种实验和分析，以展示DPT的有效性。结果显示DPT成功超越以往的并行思维方法，在基线上达到了最先进的性能。我们将路径多样性视为提升并行思维的关键要素。我们的代码可在此 https URL 获取。

Hard and Soft Real-Time DNN Inference Scheduling on Heterogeneous Multi-Accelerator Systems via Reinforcement Learning

通过强化学习实现异构多加速器系统的硬实时和软DNN推断调度

• Authors: Francesco Giulio Blanco, Enrico Russo, Giuseppe Ascia, Davide Patti, Maurizio Palesi
• Subjects: Subjects: Hardware Architecture (cs.AR); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2610.29503
• Pdf link: https://arxiv.org/pdf/2610.29503
• Abstract We present a scheduling framework for heterogeneous multi-accelerator systems executing both hard real-time and soft real-time QoS-aware DNN inference workloads. The framework combines an online reinforcement learning (RL) scheduler with a deterministic runtime safety layer. The RL scheduler dynamically generates candidate schedules by jointly selecting sub-job priorities and accelerator assignments, while runtime certification guarantees that all HRT deadlines remain satisfied independently of the learned policy. To enable runtime certification in heterogeneous precedence-constrained systems, we introduce a schedulability-preserving prefix replacement criterion based on machine-wise cuts. The proposed criterion guarantees that runtime-generated schedule prefixes preserve the existence of a feasible continuation without requiring complete finish-time dominance with respect to a feasible reference schedule. Experimental results on heterogeneous DNN accelerator platforms show that the proposed framework preserves HRT schedulability while improving QoS-aware deadline satisfaction, client-level service quality, and accelerator utilization compared to state-of-the-art scheduling approaches.
• 中文摘要 我们提出了一个调度框架，适用于执行硬实时和软实时QoS感知DNN推理工作负载的异构多加速器。该框架结合了在线强化学习（RL）调度器与确定性运行安全层。RL调度器通过联合选择子作业优先级和加速器分配动态生成候选调度，而运行时认证则保证所有HRT截止日期独立于学习策略而满足。为在异构优先级约束系统中实现运行时认证，我们引入了基于机器间割除的保档前缀替换标准。该标准保证运行时生成的调度前缀保持可行续写存在，而无需对可行参考计划完全占上主导。异构DNN加速器平台的实验结果表明，所提框架在提升QoS感知截止时间满意度、客户级服务质量和加速器利用率的同时，相较于最先进的调度方法，同时保持了HRT的可调度性。

Safe World Model for Joint Trajectory and Beamforming Design of Mobile Base Stations

移动基站联合轨迹与波束成形设计的安全世界模型

• Authors: Abdolrasoul Sakhaei Gharagezlou, Zeinab Askari Donbeh, Mehdi Monemi, Matti Latva-aho, Mehdi Rasti
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.29534
• Pdf link: https://arxiv.org/pdf/2610.29534
• Abstract Mobile base stations (MBSs), including aerial and ground-based platforms, provide flexible wireless coverage through adaptive positioning. However, jointly optimizing their trajectories and digital beamforming is challenging because movement changes channel conditions and interference, while beamforming determines the resulting signal and interference powers. These coupled decisions also account for how current movements affect future positions and communication performance under mobility and safety constraints. We formulate this problem as a constrained Markov decision process that maximizes the time-averaged sum signal-to-interference-plus-noise ratio (SINR), subject to mobility, restricted-zone avoidance, and transmit-power constraints. To support long-term planning and reduce the extensive environment interactions required by model-free reinforcement learning, we develop a SafeDreamer-based world model framework that learns latent system dynamics from experience and optimizes a joint trajectory and beamforming policy through imagined trajectories. Feasibility-preserving action mappings and safety cost signals incorporate operational constraints into policy learning. Simulation results demonstrate that the proposed framework improves the time-averaged sum SINR by an average of 7\% and 9\% compared with proximal policy and deep Q-network baselines, respectively, while achieving higher sample efficiency and more reliable constraint satisfaction.
• 中文摘要 移动基站（MBS），包括空中和地面平台，通过自适应定位提供灵活的无线覆盖。然而，联合优化它们的轨迹和数字波束形成具有挑战性，因为移动会改变通道条件和干扰，而波束形成决定最终的信号和干扰功率。这些耦合决策还考虑了当前运动如何影响未来位置以及在机动性和安全约束下的通信性能。我们将该问题表述为一个受限马尔可夫决策过程，最大化时间平均信号干扰加噪声比（SINR），同时受移动性、限制区规避和发射功率约束。为支持长期规划并减少无模型强化学习所需的广泛环境交互，我们开发了一个基于SafeDreamer的世界模型框架，从经验中学习潜在系统动力学，并通过想象轨迹优化联合轨迹和波束形成策略。可行性保持的动作映射和安全成本信号将操作约束纳入策略学习。模拟结果表明，所提出框架相比近端策略和深度Q网络基线，平均提升时间平均和SINR分别提升了7%和9%，同时实现了更高的采样效率和更可靠的约束满足。

Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding

用于细粒度多模理解与基础的广义循环GRPO

• Authors: Xin Zhang, Haochen Wang, Beibei Lin, Zhaoxiang Zhang, Robby T. Tan, Jason Li, Dacheng Tao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.29552
• Pdf link: https://arxiv.org/pdf/2610.29552
• Abstract Fine-grained multimodal intelligence requires models to both understand a designated region and localize it from language. Existing methods learn these directions with separate objectives and task-specific annotations. We present Generalized CycleGRPO, the first unified reinforcement learning framework coupling region understanding and localization across spatial, temporal, and spatio-temporal domains. Given a region, a multimodal large language model generates candidate descriptions and then localizes each description to reconstruct the input region. Reconstruction overlap provides a verifiable reward for captioning, while individual localization rewards improve the inverse mapping. A common region-space formulation extends this cycle from image masks and boxes to video intervals and timestamped spatial evidence, without reference captions or an external language-model judge during reinforcement learning. In videos, the cycle encourages descriptions that distinguish a target event from similar recurrences. We introduce DOVE (Distinctive Occurrence-level Video-caption Evaluation), a human-verified contrastive benchmark for this temporal distinctiveness. CycleGRPO improves SAMTok's overall gIoU from 57.5% to 68.2% on GroundingSuite and average gIoU from 71.2% to 78.7% on GRES. For temporal tasks, it achieves 51.9% tIoU on Charades and 33.3% Temporal Evidence Coverage on DOVE, compared with 45.6% and 14.8% for the Qwen3-VL initialization. These results establish region-text cycle consistency as a general self-evaluation principle for multimodal understanding and localization. Models, code, and DOVE are available at this https URL.
• 中文摘要 细粒度多模态智能需要模型既理解指定区域，又能从语言中定位该区域。现有方法通过分别的目标和任务特定注释学习这些方向。我们介绍广义循环GRPO，这是首个统一强化学习框架，将区域理解与定位结合到空间、时间和时空领域。给定一个区域，多模态大语言模型生成候选描述，然后对每个描述进行局部化以重建输入区域。重建重叠为字幕提供了可验证的奖励，而单个局部化奖励则改善了逆映射。一种常见的区域-空间表述将这一循环从图像遮罩和框扩展到视频间隔和带时间戳的空间证据，强化学习过程中无需引用字幕或外部语言模型评判。在视频中，循环鼓励描述以区分目标事件与类似重复事件。我们引入了DOVE（显著发生级视频字幕评估），这是一项经过人工验证的时间独特性的对比基准。CycleGRPO将SAMTok在GroundingSuite上的整体gIoU从57.5%提升至68.2%，在GRES上平均gIoU从71.2%提升至78.7%。在时间任务中，它在猜谜游戏中实现了51.9%的tIoU，在DOVE上实现了33.3%的时间证据覆盖率，而Qwen3-VL初始化分别为45.6%和14.8%。这些结果确立了区域-文本周期一致性作为多模态理解和定位的通用自我评估原则。模型、代码和DOVE可在此https URL获取。

CineEvolve: An Agentic Self-Evolution Framework for Cinematic Multi-Shot Video Generation

CineEvolve：一种用于电影多镜头视频生成的代理自我进化框架

• Authors: Jingzhe Zhang, Peixuan Zhang, Genying Li, Bing Ma, Shuchen Weng, Si Li, Boxin Shi
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.29590
• Pdf link: https://arxiv.org/pdf/2610.29590
• Abstract Cinematic multi-shot video generation requires both narrative planning and iterative refinement. Existing methods either lack narrative reasoning or rely on feed-forward workflows without policy updates. In this paper, we propose CineEvolve, an agentic self-evolution framework for cinematic multi-shot video generation that unifies a Director and a Critic within a single vision-language policy. Emulating real-world filmmaking that couples iterative direction with critical feedback, we train CineEvolve via alternating reinforcement learning, where the Director surfaces flaws to sharpen the Critic, and the updated Critic guides the Director's targeted revisions. To stabilize co-evolution, we introduce Evolutionary GRPO (E-GRPO), leveraging historical anchoring and revision safeguarding to mitigate advantage inflation. Additionally, we contribute CineTraj, a 40K cinematic trajectory dataset spanning creation, critique, and revision. Extensive experiments demonstrate that CineEvolve outperforms existing approaches. Our code will be available at this https URL.
• 中文摘要 电影多镜头视频生成既需要叙事规划，也需要迭代精炼。现有方法要么缺乏叙事推理，要么依赖前馈工作流程而没有策略更新。本文提出CineEvolve，一种能动自我演化的电影多镜头视频生成框架，将导演和评论家统一在单一视觉语言策略内。我们模拟现实电影制作，将迭代指导与批判反馈相结合，通过交替强化学习训练CineEvolve，导演揭示缺陷以提升批评者，更新后的批评者指导导演的针对性修订。为稳定共演化，我们引入了进化GRPO（E-GRPO），利用历史锚定和修订保护来缓解优势膨胀。此外，我们还贡献了CineTraj，这是一个涵盖创作、批评和修订的40K电影轨迹数据集。大量实验表明，CineEvolve优于现有方法。我们的代码将在此https URL中公开。

Uni-Mem: Learning a Long-Horizon Agentic Memory Manager with Unified Online Policy Optimization

Uni-Mem：学习一个带有统一在线策略优化的长期代理记忆管理器

• Authors: Zhiqian Xia, Jielong Tang, Ling-An Zeng, Shixiong Kai, Rong Yao, Hao Kong, Xiao Dong, Wei Wang, Mingxuan Yuan, Siyu Xia, Haifeng Xia
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.29671
• Pdf link: https://arxiv.org/pdf/2610.29671
• Abstract Long-term agent memory is shifting from predefined workflows toward learnable policies for memory construction and retrieval. Existing learning-based methods either optimize construction alone or combine retrieval-side online reinforcement learning (RL) with offline construction supervision, limiting co-adaptation between the two stages. In this paper, we explore end-to-end online RL for a unified memory manager (Uni-Mem) that learns both capabilities under a shared policy. However, joint optimization with existing RL algorithms such as GRPO faces confounded group comparisons and sparse feedback over long memory-management trajectories. To this end, we propose Stage-Decoupled GRPO, which alternates construction and retrieval optimization while controlling variation from the other stage within each rollout group, yielding stage-specific learning signals. We further introduce Event-Wise Credit Assignment (ECA), which localizes feedback to responsible events and propagates event-level advantages to their constituent operations. Experiments on LoCoMo and LongMemEval show that Uni-Mem outperforms the strongest RL-based memory, demonstrating the effectiveness of jointly learning memory construction and retrieval via Stage-Decoupled GRPO and ECA.
• 中文摘要 长期代理记忆正从预定义的工作流向可学习的记忆构建和检索策略转变。现有基于学习的方法要么单独优化构建，要么结合检索侧在线强化学习（RL）与离线构建监督，限制了两阶段之间的共适应。本文探讨了端到端在线强化学习，用于统一记忆管理器（Uni-Mem），在共享策略下学习这两种能力。然而，与现有强化学习算法如GRPO联合优化时，在长记忆管理轨迹中存在混淆的组比较和稀疏反馈。为此，我们提出了阶段解耦GRPO，交替进行构建和检索优化，同时控制每个推广组内另一阶段的变异，从而产生阶段特定的学习信号。我们进一步引入事件分项分配（ECA），将反馈本地化于负责事件，并将事件级优势传播到其组成操作中。LoCoMo和LongMemEval的实验表明，Uni-Mem优于最强的基于强化学习的记忆，证明了通过阶段解耦GRPO和ECA共同学习记忆构建和检索的有效性。

DriveMLP: When Training Strategy Outperforms Model Complexity in Autonomous Driving

DriveMLP：当训练策略在自动驾驶中超越模型复杂性时

• Authors: Alessandro Canevaro, Julian Schmidt, Hang Yu, Ye Jin, Sajad Marvi, Georg Martius, Julian Jordan
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.29676
• Pdf link: https://arxiv.org/pdf/2610.29676
• Abstract Learning motion planning policies that remain reliable under closed-loop execution is a central challenge in Autonomous Driving (AD). Imitation Learning (IL) scales well to large AD datasets but suffers from distribution shift, causal confusion, and compounding errors under closed-loop execution. In contrast, Reinforcement Learning (RL) typically requires handcrafted rule-based rewards and massive simulation. We study a simple closed-loop IL framework that uses RL purely as an optimization tool, guided by a trajectory-distance reward derived from logged expert data. We refer to this training strategy as Expert-Guided RL (EGRL). Through a controlled empirical study we find evidence that the training procedure, rather than model size and complexity, accounts for a substantial part of the performance gains. We then introduce DriveMLP, a lightweight MLP policy trained with EGRL, as a simple instantiation of these findings. It achieves a score of 86.51 on the nuPlan Val14 Reactive benchmark, a +3.2 point improvement over the previous best IL planner (Flow Planner), with >15x faster inference. Code is available at: this https URL .
• 中文摘要 在自动驾驶（AD）中，学习在闭环执行下保持可靠的运动规划策略是核心挑战。模拟学习（IL）在大型AD数据集上可良好扩展，但在闭环执行下存在分布偏移、因果混淆和错误叠加的问题。相比之下，强化学习（RL）通常需要手工设计的基于规则的奖励和大规模模拟。我们研究了一个简单的闭环IL框架，该框架纯粹将强化学习作为优化工具，由基于专家数据的轨迹距离奖励指导。我们将这种训练策略称为专家引导强化学习（EGRL）。通过受控实证研究，我们发现训练过程而非模型规模和复杂度，是性能提升的显著体现。随后，我们引入了DriveMLP，一种用EGRL训练的轻量级MLP策略，作为这些发现的简单实例化。它在nuPlan Val14反应式基准测试中得分为86.51分，比之前最佳IL规划器（Flow Planner）提升+3.2分，推断速度提升>15倍。代码可访问：此 https URL 。

RangeNav: Task-Aligned LiDAR Range-Image Pretraining for Robot Navigation

RangeNav：任务对齐激光雷达（LiDAR）距离图像预训练，用于机器人导航

• Authors: Samuel Foo Enze, Manthan Patel, Fan Yang, Linus Kramer, Pascal Roth, Marco Hutter, Cesar Cadena
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.29845
• Pdf link: https://arxiv.org/pdf/2610.29845
• Abstract Autonomous mobile robots operating in dense crowds and cluttered sites require $360^\circ$ perception to react to surrounding hazards. While panoramic LiDAR offers full-surround coverage, learning representations within reinforcement learning (RL) is sample-inefficient under sparse rewards, and existing 3D pretrained point-cloud encoders are too memory-heavy for massively parallel RL simulation. We present RangeNav, a compact (183k parameters) LiDAR range-image encoder pretrained entirely on synthetic data. RangeNav uses anisotropic convolutions and circular padding to handle the extreme aspect ratio and azimuthal seam of range images. We pretrain the network using a navigation-aligned objective combining range reconstruction, semantic segmentation, and bird's-eye-view (BEV) prediction, then freeze it for policy learning. In simulation, adding semantic and BEV supervision increases success rates over a reconstruction-only baseline by an average of 32.03 and 23.73 percentage points (pp) in procedurally generated training and out-of-distribution realistic test environments, respectively. Zero-shot real-world deployments on a Unitree B2-W quadruped demonstrate robust navigation through dense crowds, low-light settings, and construction zones where limited-field-of-view depth policies struggle. Code and models will be open-sourced.
• 中文摘要 在密集人群和杂乱场所运行的自主移动机器人需要360^\circ$的感知能力来应对周围的危险。虽然全景激光雷达提供全环绕覆盖，但在强化学习（RL）中的学习表现在奖励稀疏时样本效率低下，现有的3D预训练点云编码器对大规模并行强化学习来说内存过重。我们介绍RangeNav，一款紧凑型（183k参数）LiDAR距离图像编码器，完全预训练于合成数据。RangeNav采用各向异性卷积和圆形填充处理极端宽高比和方位角测量图像。我们使用结合范围重建、语义分割和鸟瞰图（BEV）预测的导航对齐目标预训练网络，然后冻结用于策略学习。在模拟中，添加语义和BEV监督，在程序生成的训练和非分布的真实测试环境中，相比仅重建基线，成功率分别提升了平均32.03个百分点和23.73个百分点（pp）。在Unitree B2-W四足动物上的零发射真实部署展示了在密集人群、低光环境和施工区中稳健导航的能力，这些环境在有限视角深度政策下难以应对。代码和模型将开源。

CORD: Conflict-Aware Optimization for Joint RL and Distillation in Agents

CORD：代理中联合强化学习和蒸馏的冲突感知优化

• Authors: Wenhao Yu, Zhengcheng Wang, Jiahong Liu, Zexuan Qiu, Minda Hu, Shaohang Wei, Jiafei Lyu, Junyan Lyu, Haobo Fu, Irwin King
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.29965
• Pdf link: https://arxiv.org/pdf/2610.29965
• Abstract Joint reinforcement learning and self-distillation combines task rewards with dense teacher guidance for agent training, but their gradients can partly cancel. A mixing coefficient controls relative scale without selectively removing opposing components. We present CORD, a general framework for conflict-aware optimization of two objectives. CORD separates the choice of protected objective from the condition that triggers projection, and combines the gradients once per optimizer mini-batch. A nearest-point formulation yields a closed-form update and characterizes its first-order effect on the protected objective before optimizer transformations. Gradient diagnostics connect conflict in agent training to lost local optimization progress and motivate selective correction. As a gradient-combination mechanism, CORD can be added to different joint training recipes while retaining their objectives. Across three such recipes, it improves ALFWorld success by 1.5-15.9 percentage points. On WebShop, it improves two of the three pairs, with SDAR rising from 60.7% to 64.6% success. Our analyses explain why protection direction and projection gate must be chosen together, and why the distillation objective and its mixing weight remain consequential after projection.
• 中文摘要 联合强化学习和自蒸馏结合了任务奖励与密集的教师指导，用于智能体培训，但其梯度可能部分抵消。混合系数控制相对尺度，同时不选择性地去除对立成分。我们介绍CORD，一个用于冲突感知优化两个目标的通用框架。CORD将受保护目标的选择与触发投射的条件分离，并在每个优化器小批次中将梯度合并一次。最近点表述产生闭式更新，并在优化器变换前表征其对受保护目标的一阶效应。梯度诊断将代理训练中的冲突与失去的局部优化进展联系起来，并激励选择性修正。作为梯度组合机制，CORD可以添加到不同的联合训练配方中，同时保持其目标。在三种此类配方中，CORD使ALFWorld的成功率提升1.5-15.9个百分点。在WebShop上，它提升了三对中的两对，SDAR成功率从60.7%升至64.6%。我们的分析解释了为何保护方向和投影门必须同时选择，以及蒸馏目标及其混合权重在预测后依然具有重要意义。

EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

EPIC：通过Exact-KL前缀重要性-比率裁剪重新思考自回归LLM策略中的信任区域

• Authors: Chengxi Li, Hanlin Wu, Zhicheng Cai, Huan-ang Gao, Wei-Ying Ma, Jingjing Liu, Hao Zhou
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.29999
• Pdf link: https://arxiv.org/pdf/2610.29999
• Abstract Reinforcement learning (RL) based on PPO-style clipping has become an effective approach to improving the reasoning capabilities of large language models (LLMs). However, standard token-level clipping measures policy change only through the conditional probability of the current token, while the prefix conditioning context is itself generated by the policy. This mismatch becomes especially consequential in long-response and off-policy training. We reformulate autoregressive generation as a finite-horizon Markov decision process and derive its performance-difference identity under behavior-policy sampling. The resulting change of measure identifies the cumulative likelihood ratio of the generated prefix as the exact importance weight for each stepwise policy-improvement term. Starting from the monotonic-improvement principle of trust-region policy optimization, we then construct a prefix-level trust region using the exact reverse-KL coordinate $r-1-\log r$, whose behavior-policy expectation equals the corresponding prefix-distribution KL divergence. This yields EPIC, a PPO-style sequence-clipping method that directly regulates prefix-level policy drift without relying on a local quadratic approximation or expert-routing replay. Experiments on off-policy reinforcement learning with a sparse mixture-of-experts language model show that EPIC consistently outperforms standard and sequence-level clipping baselines across mathematical-reasoning benchmarks, while remaining competitive with routing-replay-based alternatives. Code is available at this https URL.
• 中文摘要 基于PPO风格裁剪的强化学习（RL）已成为提升大型语言模型（LLM）推理能力的有效方法。然而，标准的令牌级裁剪仅通过当前令牌的条件概率来衡量策略变化，而前缀条件语境本身由策略生成。这种不匹配在长响应和非策略训练中尤为重要。我们将自回归生成重新表述为有限视野的马尔可夫决策过程，并在行为-策略抽样下推导其性能-差值身份。由此产生的度量变化确定了生成前缀的累计似然比，作为每个逐步策略改进项的精确重要权重。基于信任区域策略优化的单调改进原则，我们利用精确的反KL坐标$r-1-\log r$构造前缀级信任区域，其行为-策略期望等于相应的前缀-分布KL发散。这产生了EPIC，一种PPO风格的序列裁剪方法，直接调节前缀级策略漂移，无需依赖局部二次近似或专家路由重放。在稀疏专家混合语言模型中进行非策略强化学习实验表明，EPIC在数学推理基准测试中始终优于标准和序列级裁剪基线，同时在路由重放替代方案中保持竞争力。代码可在此 https 网址获取。

RLMuscle: Muscle-Driven Character Animation via Reinforcement Learning

RLMuscle：通过强化学习实现的肌肉驱动角色动画

• Authors: Chunlei Li, Siyuan Yu, Yang Gao, Shuai Li, Peng Yu, Aimin Hao, He Wang
• Subjects: Subjects: Graphics (cs.GR)
• Arxiv link: https://arxiv.org/abs/2610.30033
• Pdf link: https://arxiv.org/pdf/2610.30033
• Abstract Body deformation caused by muscle contraction plays a vital role in the visual realism of character animation, yet in most 3D animation pipelines artists have to manually specify muscle activations, which is time-consuming and gives no guarantee of physical realism. To address this, we propose a new method in which a reinforcement learning (RL) controller discovers per-muscle activations that track reference motions without manual tuning. Existing RL muscle controllers stop at 1D muscles; we go one step further, keeping the 1D model inside the RL control loop as a fast surrogate, and using a 3D volumetric simulation to produce the final deformation. Because both stages share the very same constitutive law, the muscle deformation is produced by the same activations that generated the motion. We further add a measured electromyography (EMG) prior that steers the redundant activation solution toward real human muscle recruitment. To achieve stable, efficient, and realistic simulation, we propose a novel geometric multigrid for Extended Position-Based Dynamics (XPBD) that is built on a coarse tetrahedral cage with barycentric-coordinate prolongation and closes a V-cycle on tetrahedral meshes. Within this GPU-parallelized solver, we inject the same Hill-type force-length law as an energy-based fiber constraint that turns per-fiber activation into realistic muscle contraction and bulging. We demonstrate volumetric muscle simulation on a 194-muscle human body with a diverse set of motions (walk, kick, swim, overhead-squat, etc.), simulating the full-body model at 18 ms per frame on a single RTX 4090. Our work takes a step toward character animation that is not only visually convincing but also biomechanically grounded. The website with code and videos is at this https URL.
• 中文摘要 肌肉收缩引起的身体变形在角色动画的视觉真实感中起着关键作用，但在大多数3D动画流程中，艺术家必须手动指定肌肉激活，这耗时且无法保证物理真实性。为此，我们提出了一种新方法，由强化学习（RL）控制器发现每块肌肉激活，无需手动调校即可追踪参考动作。现有的强化学习肌肉控制器止于一维肌肉;我们更进一步，将一维模型置于强化学习控制循环内作为快速替代，并使用三维体积模拟生成最终变形。由于两阶段共享相同的本构定律，肌肉变形是由产生运动的相同激活产生的。我们还进一步添加了测量肌电图（EMG）事先，将冗余激活方案引导到真实人体肌肉招募。为了实现稳定、高效和逼真的仿真，我们提出了一种新型的扩展位置动力学（XPBD）几何多栅格，该结构基于粗四面体笼，具有重心坐标延长，并在四面体网格上闭合V环。在这个GPU并行求解器中，我们注入了与基于能量的光纤约束相同的Hill型力-长度定律，将每纤维激活转化为真实的肌肉收缩和膨胀。我们在一个194块肌肉的人体上演示体积肌肉模拟，动作多样（行走、踢腿、游泳、过头深蹲等），在单个RTX 4090上模拟全身模型，每帧18毫秒。我们的工作迈向了不仅视觉上令人信服且生物力学扎实的角色动画。包含代码和视频的网站位于 https URL。

AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation

AtomicRL：用于远程机器人操作的原子强化学习

• Authors: Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang, Xiaodan Liang, Liang Lin
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.30097
• Pdf link: https://arxiv.org/pdf/2610.30097
• Abstract Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.
• 中文摘要 视觉-语言-行动（VLA）模型在机器人操作方面展现出潜力，但仅靠模仿学习（IL）往往难以掌握复杂的长期视野技能。强化学习（RL）为进一步改进VLA策略提供了有效途径。然而，现有的VLA-RL方法通常因轨迹级奖励稀疏而导致信用分配延迟和偏颇，而子任务掌握不平衡则造成瓶颈，限制整体性能。为应对这些挑战，我们提出了AtomicRL，一种利用原子任务结构实现可靠信用分配和自适应探索的强化学习后期框架。AtomicRL引入了语义感知策略优化（SAPO），利用语义感知门控限制失败原子子任务的TD残差跨任务边界传播，减轻信用污染，同时保持子任务内的信用分配和价值自助。此外，瓶颈自适应目标探索（BATE）基于对单个原子子任务的掌握情况动态协调任务采样和局部探索激励，引导学习方向解决剩余瓶颈。最后，我们开发了Real2Sim2Real流水线，连接在线模拟与实际部署，便于向物理机器人的实际迁移。在LIBERO-LONG上，AtomicRL在两种VLA架构上实现了3.6%和4.0%的绝对性能提升，分别相较PPO提升4.0%。在CALVIN ABC-D上，它将全任务性能提升高达9.5%。结合Real2Sim2Real流水线，AtomicRL进一步提升了实际任务的平均表现21.3%。这些结果表明，AtomicRL有效提升了长期任务中VLA策略的学习效率和操作性能。

Differentially Private Sparse Reward Estimation with Preference Feedback

差分私有稀疏奖励估计与偏好反馈

• Authors: Meng Ding, Mingxi Lei, Jie Zhang, Jinyan Liu, Di Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.30099
• Pdf link: https://arxiv.org/pdf/2610.30099
• Abstract Reinforcement Learning with Human Feedback (RLHF) has become a central paradigm for aligning Large Language Models with human values. However, the reliance on sensitive preference data necessitates rigorous differential privacy (DP) guarantees to prevent user information leakage. Existing private alignment methods typically assume dense reward parameterizations, resulting in estimation error bounds that scale polynomially with the ambient {model size $d$}. Such dependence is prohibitive in modern {overparameterized models} where $d$ is vast. In this work, we study \emph{Private Sparse RLHF} problem to address this limitation, assuming that human preferences (reward model) are governed by a subset of $s^ \ll d$ relevant features. We provide the first comprehensive theoretical framework for differentially private sparse reward estimation from pairwise preference feedback under the Bradley-Terry-Luce model, and introduce efficient algorithms for both Sample-DP and Label-DP settings. Theoretically, under the squared $\ell_2$ parameter estimation error, we establish upper bounds of $\widetilde{O}(s^/n + {s^}^2/n^2\varepsilon^2)$ for $(\varepsilon,\delta)$-Sample-DP and $\widetilde{O}(s^/n + s^*/n\varepsilon^2)$ for $\varepsilon$-Label-DP based on randomized response, where $n$ is the training data size. For the lower bounds, we show that the Sample-DP rate is nearly optimal, and that the guarantee for our Label-DP method cannot be further improved within the randomized-response framework. We also provide a general lower bound for the problem with Label-DP. Empirically, our framework demonstrates superior efficacy over dense baselines across sentiment generation and safety alignment benchmarks, offering a significantly more favorable privacy-utility trade-off.
• 中文摘要 基于人类反馈的强化学习（RLHF）已成为将大型语言模型与人类价值观对齐的核心范式。然而，依赖敏感偏好数据需要严格的差别隐私（DP）保障以防止用户信息泄露。现有的私有比对方法通常假设奖励参数化密集，导致估计误差边界与环境{模型大小$d$}多项式式扩展。这种依赖在现代{参数过载模型}中极为严重，$d$极为庞大。本研究研究\emph{私有稀疏RLHF}问题以解决这一限制，假设人类偏好（奖励模型）由$s^ \ll d$相关特征子集控制。我们提供了首个基于Bradley-Terry-Luce模型，基于成对偏好反馈进行差异私密稀疏奖励估计的综合理论框架，并引入了适用于样本-DP和标签-DP设置的高效算法。理论上，在平方$\ell_2$参数估计误差下，我们建立了$\widetilde{O}（s^/n + {s^}^2/n^2\varepsilon^2）$的上界，用于$（\varepsilon，\delta）$-Sample-DP和$\widetilde{O}（s^/n + s^*/n\varepsilon^2）$ for $\varepsilon$-Label-DP，基于随机响应，其中$n$为训练数据大小。对于下界，我们表明样本-DP率几乎最优，且在随机响应框架下，Label-DP方法的保证无法进一步提升。我们还为Label-DP的问题提供了通用的下界。从经验上看，我们的框架在情感生成和安全对齐基准中表现出优越的高效性，提供了更有利的隐私与效用权衡。

Machine Learning-Enabled Wireless Power Transfer: A Task-Oriented Review of Structural Optimization, State and Parameter Estimation, and Adaptive Control

机器学习驱动的无线电力传输：结构优化、状态与参数估计及自适应控制的任务导向综述

• Authors: Denghao Luo, Yiheng Jiang, Jiaqi Zhao, Yuan Gao, Changsong Cai, Weiyi Huang, Yun Zhang
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2610.30101
• Pdf link: https://arxiv.org/pdf/2610.30101
• Abstract Wireless power transfer (WPT) systems must maintain efficient and safe energy delivery under variations in geometry, coupling, load, alignment, and operating conditions. Machine-learning methods are increasingly applied to accelerate structural design, estimate operating parameters and states, and support adaptive control and scheduling. This review develops a task-oriented framework organized around three interconnected domains: structural design and optimization, parameter and state estimation, and control and optimization. Supervised learning, reinforcement learning, and hybrid or physics-informed methods are compared in terms of data requirements, dependence on physical models, information sources, computational requirements, accuracy, online response, application scope, and optimization objectives. Supervised learning provides rapid surrogate modeling, regression, and classification when representative labeled data are available. Reinforcement learning addresses sequential control and scheduling but introduces challenges in reward design, safe exploration, and simulation-to-reality transfer. Hybrid methods integrate physical knowledge with data-driven models to improve data efficiency and physical consistency. The principal barriers to engineering deployment include limited and biased datasets, weak out-of-distribution generalization, insufficient physical consistency, unsafe online adaptation, embedded-resource constraints, and the absence of standardized benchmarks. Future research should prioritize uncertainty-aware and physics-informed learning, safe simulation-to-reality transfer, lightweight edge deployment, distributed coordination, reproducible evaluation, and digital-twin-assisted development.
• 中文摘要 无线电力传输（WPT）系统必须在几何形状、耦合、负载、对准和运行条件的变化下保持高效且安全的能量传输。机器学习方法越来越多地应用于加速结构设计、估计运行参数和状态，以及支持自适应控制和调度。本综述构建了一个围绕结构设计与优化、参数与状态估计以及控制与优化三个相互关联领域组织的任务导向框架。在数据需求、对物理模型、信息源、计算需求、准确性、在线响应、应用范围和优化目标方面进行了比较，比较了监督学习、强化学习以及混合或物理知情方法。监督学习在有代表性标记数据时提供快速的替代建模、回归和分类。强化学习解决顺序控制和调度，但也带来了奖励设计、安全探索和模拟到现实转移的挑战。混合方法将物理知识与数据驱动模型整合，以提升数据效率和物理一致性。工程部署的主要障碍包括数据集有限且有偏见、分布外泛化薄弱、物理一致性不足、不安全的在线适应、嵌入资源限制以及缺乏标准化基准。未来研究应优先考虑不确定性和物理驱动学习、安全的模拟到现实传输、轻量级边缘部署、分布式协调、可重复评估以及数字孪生辅助开发。

Asymmetric On-Policy Distillation Learns In-Context Exploration

非对称策略提炼学习上下文探索

• Authors: Sriyash Poddar, Yanda Bao, Jacob Krantz, Matthew Chang, Xavier Puig, Roozbeh Mottaghi, Natasha Jaques, Abhishek Gupta
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30105
• Pdf link: https://arxiv.org/pdf/2610.30105
• Abstract Sequential decision-making under partial observability is difficult because agents must explore to infer hidden, task-relevant information before acting effectively. Learning such information-gathering behaviors tabula rasa via online reinforcement learning is sample-inefficient, whereas offline imitation averages expert actions across hidden states and fails to learn exploration. In this work, we propose ASTEROID, a simple and effective student--teacher supervised learning framework that can learn test-time exploration. Although a privileged expert never needs to explore, we demonstrate that a history-conditioned student can learn to do so when trained with on-policy contexts: the student first collects on-policy context under partial observability, and the privileged expert provides action labels conditioned on that context. By distilling expert actions directly conditioned on the student's on-policy history, the student learns to systematically resolve partial observability, i.e., search while uncertain and exploit once the hidden state is revealed. This enables efficient test-time exploration (similar to Bayesian posterior sampling) without incurring the impractical training-data costs of reinforcement learning. Across a diverse range of eleven different environments, including discrete exploration benchmarks, robotic manipulation, navigation, and Procgen environments, ASTEROID achieves ~2x higher returns under fixed exploration budgets, uses ~100x fewer interactions than RL baselines, and outperforms frontier models at exploration. Finally, our method transfers policies trained in simulation to vision-denied real-world robot manipulation, exhibiting efficient exploration and increasing success by +40% over standard distillation. Overall, it provides a simple yet surprisingly effective framework to learn efficient test-time exploration.
• 中文摘要 在部分可观察性下进行顺序决策困难，因为智能体必须探索以推断隐藏的任务相关信息，才能有效行动。通过在线强化学习这种信息收集行为的白纸状学习样本效率较低，而离线模仿则在隐藏状态中平均专家行为，且无法学会探索。本研究提出ASTEROID，一种简单有效的学生-教师监督学习框架，可在考试时进行探索。虽然特权专家无需探索，但我们证明，历史条件学生在接受政策上下文训练时可以学会探索：学生首先在部分可观察性下收集策略上下文，特权专家根据该上下文提供行动标签。通过直接基于学生的政策历史提炼专家行为，学生学会系统地解析部分可观测性，即在不确定时搜索，并在隐藏状态揭示后加以利用。这使测试时间探索高效（类似贝叶斯后验采样）成为可能，同时避免强化学习中不切实际的训练数据成本。在包括离散探索基准、机器人操作、导航和Procgen环境在内的11种不同环境中，ASTEROID在固定探索预算下实现约2倍回报，使用比强化学习基线少约100倍的交互，并在探索中优于前沿模型。最后，我们的方法将模拟中训练的策略转化为视觉否定的现实世界机器人操作，展现出高效的探索效果，并比标准蒸馏提升+40%。总体而言，它提供了一个简单但出乎意料地有效的学习框架，帮助你学习高效的测试时探索。

LongZero: Enhancing Long-Context Reasoning in LLMs from Zero Data

LongZero：从零数据提升大型语言模型中的长上下文推理

• Authors: Zhennan Wan, Jianfei Chen
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.30132
• Pdf link: https://arxiv.org/pdf/2610.30132
• Abstract Reinforcement learning (RL) is an effective way to strengthen the long-context reasoning of large language models (LLMs), but it needs training data that are long, diverse, challenging, and verifiable. Human annotation of such data is expensive, naturally occurring data are too scarce to sustain RL, and existing synthesis methods depend on external corpora or stronger teacher models, which become harder to obtain as the trained models get stronger. Recently, RL from zero data, in which a model learns from tasks it proposes itself, has achieved excellent performance, but existing methods target short-context tasks. Therefore, we propose $\textbf{LongZero}$, a method that lets an LLM improve its own long-context reasoning without any external data. Instead of writing questions and answers over existing documents, LongZero first has the model synthesize the question and answer with its supporting evidence in a compact structured form, and only then separately expands the evidence into a long multi-document context. LongZero combines these self-written multi-hop reasoning tasks with self-written multi-round retrieval tasks and trains the model with GRPO. We evaluate LongZero on two LLMs across five long-context reasoning benchmarks and compare it with RL on data synthesized from external corpora. Experimental results show that LongZero can generalize to real tasks, and outperforms both baselines, suggesting a viable and promising path to improving long-context reasoning of LLMs.
• 中文摘要 强化学习（RL）是加强大型语言模型（LLM）长上下文推理的有效方法，但它需要长、多样、具有挑战性和可验证的训练数据。人工对此类数据的注释成本高昂，自然生成的数据过于稀缺，难以维持强化学习，现有综合方法依赖外部语料库或更强的教师模型，随着训练模型的增强，这些模型的获取变得越来越困难。最近，从零数据开始的强化学习取得了优异性能，但现有方法主要针对短上下文任务。因此，我们提出了$\textbf{LongZero}$的方法，使LLM能够在不依赖外部数据的情况下提升自身的长上下文推理能力。LongZero 不在现有文档上写出问题和答案，而是先让模型将问题和答案及其支持证据以紧凑结构化的形式综合，然后再单独将证据扩展为长的多文档上下文。LongZero 将这些自写的多跳推理任务与自写的多轮检索任务结合起来，并用 GRPO 训练模型。我们在五个长上下文推理基准测试中对两个大型语言模型进行了评估，并与外部语料库综合数据的强化学习进行了比较。实验结果显示，LongZero 能够推广到真实任务，并且表现优于两个基线，这为提升 LLM 长上下文推理提供了可行且有前景的路径。

Keyword: diffusion policy

There is no result