生成时间: 2026-11-05 00:01:15 (UTC+8); Arxiv 发布时间: 2026-11-04 20:00 EST (2026-11-05 09:00 UTC+8)

今天共有 39 篇相关文章

Keyword: reinforcement learning

EvoMemState: Evolving Memory State for Personal Agents

EvoMemState：个人代理的进化内存状态

• Authors: Zhishang Xiang, Yunbo Tang, Zhongqiao Liu, Junkai Chen, Zerui Chen, Qingguo Hu, Yuxuan Zhang, William Grant, Qinggang Zhang, Jinsong Su
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.03743
• Pdf link: https://arxiv.org/pdf/2611.03743
• Abstract Long-term memory enables large language model (LLM) agents to provide continuous personalized assistance by retaining user information across interactions. Existing systems support this process through fact extraction, organization, updating, and retrieval, but largely maintain memories as independent facts. This fact-centric paradigm may overlook how new facts update, constrain, or conflict with earlier ones, while leaving their applicability, role, and priority for the current request implicit. In this paper, we introduce EvoMemState, a state-centric framework that augments persistent facts with two complementary states: Relational States that capture how facts remain valid and relate to one another as memory evolves, and a Reasoning State that determines how these facts should be applied to the current request. Specifically, EvoMemState consists of three key components: (i) Ontology-Guided Memory State Induction extracts Persistent Facts and constructs Relational States that maintain their validity, scope, and evolving relations. (ii) RL-Based State Evolution internalizes memory-use experience into a policy through reinforcement learning to generate a request-specific Reasoning State. (iii) State-Aware Reasoning then combines the relevant facts with their Relational States and the generated Reasoning State to guide the personal agent. Extensive experiments on long-term dialogue and memory-use benchmarks demonstrate consistent improvements over existing memory systems in retrieval completeness and memory-grounded response accuracy. Our code and implementation details are available at this https URL.
• 中文摘要 长期记忆使大型语言模型（LLM）代理能够通过在交互中保留用户信息，提供持续的个性化协助。现有系统通过事实提取、组织、更新和检索支持这一过程，但大体上仍将记忆作为独立事实。这种以事实为中心的范式可能忽视新事实如何更新、限制或与早期事实冲突，同时隐含其适用性、角色和优先级对当前请求的适用性、作用和优先级。本文介绍了EvoMemState，一种以状态为中心的框架，通过两种互补状态来增强持久性事实：关系状态，捕捉事实如何保持有效并相互关联，随着记忆演变，以及判断这些事实如何应用于当前请求的推理状态。具体来说，EvoMemState 由三个关键组成部分：（i） 本体引导的记忆状态归纳提取持久事实，构建保持其有效性、范围和不断演变关系的关系状态。（ii） 基于强化学习的状态进化通过强化学习将内存使用经验内化为策略，生成针对请求的推理状态。（iii） 状态感知推理随后将相关事实与其关系状态及生成的推理状态结合起来，指导个人智能体。长期对话和内存使用基准测试的广泛实验显示，在检索完整性和基于内存的响应准确性方面，相较现有记忆系统有持续的改进。我们的代码和实现详情可在此 https URL 获取。

AIRoA-0: Long-Horizon, Precise, Cross-Embodiment Manipulation from Handheld Data

AIRoA-0：基于手持数据的长视野、精确、跨实体操作

• Authors: Yueh-Hua Wu, Kei Ota, Hanlong Li, Hayata Yuasa, Jason Wang, Kengo Ikeuchi, Kensuke Chiyoma, Makoto Sugiura, Masafumi Takahashi, Masatoshi Tateno, Naoaki Kanazawa, Nicola Scianca, Petr Khrapchenkov, Satoshi Otsubo, Takehiko Ohkawa, Takuya Okubo, Tatsuya Matsushima, Yuki Noguchi, Yuma Shin, Zixuan Zhong
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03815
• Pdf link: https://arxiv.org/pdf/2611.03815
• Abstract AIRoA-0 (A-0) is a bimanual manipulation system learned from 16,151 hours of handheld demonstrations, with no robot data collected at any stage. The demonstrations are recorded with YUBI, our open-source handheld gripper, which lets people record manipulation data anywhere with no robot present. From a single instruction, A-0 sorts more than a dozen small parts into their numbered cells and changes the mop pads and brush of a robot vacuum; among other tasks, it unzips a pencil case and folds a towel. It performs these tasks using only a camera on each wrist, with no head or scene camera, at a site where none of its training data was recorded. Across the ten tasks of our evaluation, it averages a success rate of 78%, against 27% for the strongest of four other vision-language-action models post-trained on the same task demonstrations. We collect no teleoperation data and use no reinforcement learning or on-robot corrections, and the robot appears only at test time. The system learns which regions of each wrist view matter for the task, predicts the stage the task has reached, and decodes actions as a continuous stream. A-0 is embodiment-agnostic: it runs on robots that mount the YUBI gripper, such as a bimanual Franka, TX-G2, OpenArm and UR. We are releasing the dataset, to our knowledge the largest open dataset of handheld-gripper demonstrations with task and subtask instructions, together with the pre-trained weights and the post-training recipe. Videos of A-0 on every task are on the project page, this https URL. Anyone can build the gripper, record a task of their own and post-train the model on it in under four hours on a single node of eight H200 GPUs.
• 中文摘要 AIRoA-0（A-0）是一款双手操作系统，基于16,151小时的手持演示学习，且在任何阶段都未收集机器人数据。演示使用我们的开源手持握持器YUBI录制，允许人们在无机器人在场的任何地方记录操作数据。从一条指令出发，A-0将十多个小部件分类到编号单元，并更换机器人吸尘器的拖把垫和刷子;此外，它还能拉开笔盒拉链和折叠毛巾。这些任务仅使用每只手腕上的摄像头完成，没有头部或场景摄像头，且现场未记录任何训练数据。在我们评估的十个任务中，A-0的平均成功率为78%，而其他四个视觉语言动作模型中较强的27%被记录。我们不收集远程操作数据，也不使用强化学习或机器人内修正，机器人仅在测试时出现。系统学习每个手腕视角中哪些区域对任务重要，预测任务已达到的阶段，并将动作以连续流方式解码。A-0 不依赖身体：它运行在安装 YUBI 抓手的机器人上，如双手 Franka、TX-G2、OpenArm 和 UR。据我们所知，我们将发布该数据集，这是迄今为止最大的手持握力器演示数据集，包含任务和子任务指令，以及预训练权重和训练后配方。每个任务的 A-0 视频均可在项目页面（https URL）上观看。任何人都可以在八台 H200 GPU 的单节点上组装夹具、录制自己的任务，并在四小时内完成模型后训练。

Dory1.0-4B: A Looped Normalized-GPT

Dory1.0-4B：循环归一化GPT

• Authors: Maxim Fishman, Simeng Sun, Shantanu Acharya, Faisal Ladhak, Brian Chmiel, Santiago Akle Serano, Ron Banner, Boris Ginsburg, Krishna C. Puvvada
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.03998
• Pdf link: https://arxiv.org/pdf/2611.03998
• Abstract We introduce Dory1.0-4B, a 4-billion-parameter looped (latent-recurrent) language model with elastic depth: a single checkpoint serves different inference compute budgets. Dory1.0-4B is built on a normalized Transformer (nGPT) backbone with interleaved sliding-window and full-attention layers, organized into an input block, a shared recurrent block, and an output block. At inference, the recurrent block can be applied a variable number of times. We pre-trained Dory1.0-4B on 6 trillion tokens. The first 4.5 trillion tokens were trained in NVFP4, and additional pre-training then extended the context length to 256K tokens. Next, we post-trained the model using Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR), with a difficulty-aware loop curriculum that routes harder prompts to deeper loops. Increasing the loop count improves accuracy, with the largest gains on coding, agentic, and long-context benchmarks. With two or more loops, Dory1.0-4B outperforms larger models such as Qwen3.5-9B, Gemma 4 12B, and Gemma 4 26B-A4B on average across 12 benchmarks. Looping also makes Dory1.0-4B more token-efficient on agentic benchmarks: with four loops, it generates 1.3-2.5x fewer tokens than with one loop, as the extra computation is spent in latent space rather than on longer outputs. To reduce memory during looping, Dory1.0-4B caches keys and values only from the last loop, keeping its KV-cache footprint equal to that of a non-looped model at any loop count while retaining most of the accuracy gains. We open-source the BF16, MXFP8, NVFP4, and GGUF quantized checkpoints.
• 中文摘要 我们引入了Dory1.0-4B，一个拥有40亿参数的循环（潜-递归）语言模型，具有弹性深度：单个检查点承担不同的推理计算预算。Dory1.0-4B基于规范化的Transformer（nGPT）骨干，带有交错滑动窗口和全注意力层，组织为输入块、共享递归块和输出块。推理时，递归块可以应用可变次数。我们对Dory1.0-4B进行了6万亿个令牌的预训练。最初的4.5万亿个令牌在NVFP4中训练，随后通过额外的预训练将上下文长度扩展到256K令牌。接下来，我们使用监督微调（SFT）和可验证奖励强化学习（RLVR）对模型进行了后期训练，采用难度感知的循环课程，将更难的提示引导到更深层次的循环。增加循环数量提升准确性，在编码、智能和长上下文基准测试中获得最大收益。在两个或更多循环中，Dory1.0-4B在12个基准测试中平均优于Qwen3.5-9B、Gemma 4 12B和Gemma 4 26B-A4B等大型模型。循环还使Dory1.0-4B在智能基准测试中更高效：四个循环时，产生的代币数量比单循环少1.3-2.5倍，因为额外的计算集中在潜在空间，而非更长的输出。为了减少循环时的内存，Dory1.0-4B 只缓存上一个循环的键和值，保持其 KV 缓存占地等于非循环模型在任意循环计数下，同时保留大部分准确率提升。我们开源了 BF16、MXFP8、NVFP4 和 GGUF 量化检查点。

Language Model Augmented Decision Transformer for Multi-Task Offline Reinforcement Learning

多任务离线强化学习语言模型增强决策转换器

• Authors: Xinbo Zhao, Yuhang Liu, Xin Zhang, Lei Le, Zhaohan Xi, Yu Yang, Yingxue Zhang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04031
• Pdf link: https://arxiv.org/pdf/2611.04031
• Abstract Offline multi-task reinforcement learning (MTRL) aims to learn a unified policy from heterogeneous offline datasets that generalizes across tasks. However, existing offline MTRL methods often struggle with two fundamental issues: severe per-task data scarcity and limited cross-task generalization. In this paper, we propose Language-Model Augmented Decision Transformer (LaMA-DT), a unified sequence-modeling framework that leverages a single large language model (LLM) for both trajectory augmentation and multi-task policy learning. LaMA-DT uses natural-language task descriptions to guide task-aligned trajectory augmentation through a set of supervised generation and alignment objectives, and incorporates a scoring-and-filtering mechanism to retain high-quality augmented trajectories. The same LLM is then repurposed as a language-conditioned Decision Transformer and fine-tuned on a mixture of real and augmented data. Experiments on Meta-World and MuJoCo multi-task benchmarks demonstrate that LaMA-DT consistently outperforms competitive baselines, particularly in low-data regimes, showing that language-guided data augmentation substantially improves cross-task generalization.
• 中文摘要 离线多任务强化学习（MTRL）旨在从异构离线数据集中学习统一策略，并推广到跨任务。然而，现有的离线MTRL方法常常面临两个根本问题：每任务数据严重稀缺和跨任务泛化有限。本文提出语言模型增强决策变换器（LaMA-DT），这是一种统一序列建模框架，利用单一大型语言模型（LLM）进行轨迹增强和多任务策略学习。LaMA-DT利用自然语言任务描述引导任务对齐轨迹增强，通过一组监督生成和对齐目标，并结合评分和过滤机制以保留高质量的增强轨迹。同一LLM随后被重新利用为语言条件决策转换器，并在真实和增强数据混合上进行微调。Meta-World和MuJoCo多任务基准测试的实验表明，LaMA-DT在低数据环境中持续优于竞争基线，显示语言引导数据增强显著提升了跨任务泛化能力。

Forgetting to Improve: Principled Data Removal in Active Learning

忘记改进：主动学习中的原则性数据移除

• Authors: Manuel Wendl, Erik Englesson, Andreas Krause, Carl Henrik Ek
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04054
• Pdf link: https://arxiv.org/pdf/2611.04054
• Abstract The uncertainty of a statistical model is most commonly factorized into an aleatoric and an epistemic part. This factorization changes how predictions are interpreted in downstream decision tasks. Importantly, except for the idealistic scenario with no model mismatch, the quantification is a characteristic of the model and not the data generating process. In this paper, we propose Forgetting to Improve a method that reduces this discrepancy by incorporating the task into the modeling framework. Our key insight is to acknowledge that in scenarios of model mismatch, data can have a detrimental effect on the modeling for a specific task. Based on this insight, we propose an influence function for Gaussian process models that allows for principled removal of detrimental data samples. We showcase the flexibility of this approach by demonstrating significant improvements across a range of tasks, including Bayesian optimization, model-based reinforcement learning, and transductive learning.
• 中文摘要 统计模型的不确定性通常被分解为偶然部分和认识部分。这种分解改变了下游决策任务中预测的解读方式。重要的是，除了无模型不匹配的理想情景外，量化是模型的特征，而非数据生成过程本身。本文提出了一种“遗忘改进”方法，通过将任务纳入建模框架来减少这种差异。我们的关键见解是承认，在模型不匹配情景中，数据可能对特定任务的建模产生不利影响。基于这一见解，我们提出了一种高斯过程模型的影响函数，允许原则性地去除有害数据样本。我们通过展示在贝叶斯优化、基于模型的强化学习和转导学习等多种任务上的显著改进，展示了该方法的灵活性。

On the Primacy Bias in RLVR Training

关于RLVR培训中的优先性偏差

• Authors: Phuc Minh Nguyen, Ngoc-Hieu Nguyen, Parshin Shojaee, Chinh Duc La, Duy Minh Ho Nguyen, Rui Zhang, Khoa D. Doan, Ting Hua, Nitesh V. Chawl
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04061
• Pdf link: https://arxiv.org/pdf/2611.04061
• Abstract Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a promising approach for enhancing the reasoning capabilities of large language models. However, it still remains unclear whether RLVR can extend reasoning capabilities beyond what is already encoded in the base model. In this paper, we study this question through the lens of primacy bias: the tendency of models to over-rely on already-learned representations, disproportionately improving performance on problems they already handle well while underexploring harder ones that require departing from those representations. Through our controlled experiments and large-scale analysis, we show that RLVR training on mixed data tends to further sharpen performance on easier problems while yielding only marginal gains on harder ones. We find that although hard problems lead to modest gains overall, they are actually the ones that help most in expanding the model's reasoning capabilities and pushing the representations. However, prolonged training on these hard problems may also hurt and lead to forgetting on easy problems. These findings help to better understand some of the prior conflicting observations about RLVR, specifically, why on (OOD) domains where the base model performs poorly, improvements on hard problems can expand reasoning; while on well-trained domains, forgetting can reduce performance and mask genuine gains obtained from hard problems.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为增强大型语言模型推理能力的有前景方法。然而，RLVR是否能将推理能力扩展到基础模型已编码之外，仍不明确。本文通过首重视偏差的视角研究这个问题：模型倾向于过度依赖已学会的表征，导致在已处理得好的问题上表现不成比例地提升表现，而忽视需要偏离这些表征的更难问题。通过我们的受控实验和大规模分析，我们表明，基于混合数据的RLVR训练往往能进一步提升简单问题的性能，而在较难问题上仅带来边际提升。我们发现，虽然困难问题整体带来适度提升，但实际上它们最有助于扩展模型推理能力并推动表征。然而，长期训练这些困难问题也可能带来困难，导致简单问题遗忘。这些发现有助于更好地理解关于RLVR的一些先前相互矛盾的观察，特别是为什么在基础模型表现不佳的（OOD）领域，困难问题的改进可以扩展推理能力;而在训练良好的领域，遗忘可能会降低性能，掩盖从困难问题中获得的真实收益。

Finite-Sample Convergence in Networked Average Reward MARL: Decentralization Pitfalls and Entropy Remedies

网络平均奖励MARL中的有限样本收敛：去中心化陷阱与熵补救

• Authors: Yizhou Zhang, Yashaswini Murthy, Laixi Shi, Adam Vierman
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.04072
• Pdf link: https://arxiv.org/pdf/2611.04072
• Abstract We study the problem of average reward Multi-Agent Reinforcement Learning (MARL) where agents interact within a network. Each agent in the network conducts local updates with information within its $k$-hop neighborhood and collaboratively maximizes the overall reward of the entire network. We first provide impossibility results in achieving convergence to the global optimal joint policies in our setting. These impossibility results highlight the challenges of decentralized learning with local information in multi-agent systems, namely the issues of multiagency, where each agent independently optimizes its policy, and partial observability, where agents have limited visibility into the global state. Given these challenges which indicate that decentralized policy optimization can generally be certified only up to stationarity, we provide finite-sample convergence guarantees for a decentralized actor-critic algorithm with linear function approximation that converges to an approximate stationary point with small gradient norm. To overcome such impossibility results, we further show that adding fixed-state entropy regularization reshapes the objective landscape which leads to even stronger global convergence guarantees for our proposed actor-critic algorithm.
• 中文摘要 我们研究平均奖励多智能体强化学习（MARL）问题，即智能体在网络中的交互。网络中的每个智能体在其$k跳邻域内进行局部更新，协同最大化整个网络的整体奖励。我们首先提供了在我们设定中实现与全局最优联合策略收敛的不可能结果。这些不可能结果凸显了多智能体系统中去中心化学习与本地信息的挑战，即多代理问题，即每个智能体独立优化策略，以及部分可观察性问题，智能体对全局状态的可见性有限。鉴于这些挑战表明去中心化策略优化通常只能在平稳性以下认证，我们为一个去中心化的actor-critic算法提供了有限样本收敛保证，该算法采用线性函数近似，收敛到一个带有小梯度范数的近似平稳点。为克服这种不可能结果，我们进一步证明，添加固定状态熵正则化会重塑客观景观，从而为我们提出的actor-critic算法带来更强的全局收敛保证。

AlgoPilot: Cross-Paradigm Reasoning in Language Models via Strategy Selection and Guidance

AlgoPilot：通过策略选择与指导实现语言模型中的跨范式推理

• Authors: Yilun Hao, Ruixiao Yang, Hudson Hilal, Yongchao Chen, Kaizhi Qian, Yada Zhu, Yang Zhang, Chuchu Fan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04079
• Pdf link: https://arxiv.org/pdf/2611.04079
• Abstract Real-world problems span diverse domains, including program synthesis, symbolic reasoning, planning, and optimization, and often require fundamentally different solution paradigms. A central challenge for LLM-based reasoning is twofold: identifying an appropriate problem-solving approach and formulating it correctly. In this paper, we propose AlgoPilot, a cross-paradigm reasoning framework that enables adaptive selection of the solving approaches and structured problem formulation prior to execution. The AlgoPilot framework contains three components: 1) a SteerLM, trained via supervised fine-tuning (SFT) and reinforcement learning (RL), that adaptively selects the solving strategy (paradigm and modes) based on input problems, 2) the corresponding expert GuideLM, trained via SFT, that generates structured formulation guidance tailored to the selected strategy, and 3) an ExecutionLM that generates solutions conditioned on both the selected strategy and its formulation. Unlike prompting-based methods that rely on fixed reasoning templates, our approach learns to adaptively choose and structure problem-solving strategies based on the input. We build AlgoPilot on Qwen3-8B and evaluate across 5 problem categories and 13 benchmarks with 66 unique domains, comparing against both the latest similarly scaled small models and large LLMs. AlgoPilot significantly improves average accuracy for the same model from 38.4% to 72.9%, outperforming the best large LLM prompting baseline GLM-5 by 11.7%. Ablation studies confirm the effectiveness of each component. Notably, we show that the learned steering and formulation modules can transfer across execution models, suggesting their potential to improve other LLMs. Project page: this https URL.
• 中文摘要 现实问题涵盖多个领域，包括程序综合、符号推理、规划和优化，且通常需要根本不同的解决方案范式。基于LLM推理的核心挑战有两个方面：确定合适的问题解决方法并正确表述。本文提出了AlgoPilot，一种跨范式推理框架，允许在执行前自适应选择求解方法和结构化问题表述。AlgoPilot框架包含三个组成部分：1）通过监督微调（SFT）和强化学习（RL）训练的SteerLM，基于输入问题自适应选择解决策略（范式和模式），2）通过SFT训练的专家GuideLM，生成针对所选策略的结构化表述指导，3）基于所选策略及其表述生成解决方案的ExecutionLM。与依赖固定推理模板的提示方法不同，我们的方法学习根据输入自适应选择和结构化解决问题的策略。我们在Qwen3-8B上构建AlgoPilot，评估涵盖5个问题类别和13个基准测试，涵盖66个独特领域，并与最新类似规模的小型模型和大型大型语言模型进行比较。AlgoPilot显著将同一模型的平均准确率从38.4%提升至72.9%，比最佳大型大型语言模型提示基线GLM-5高出11.7%。消融研究证实了每个组件的有效性。值得注意的是，我们展示了学习到的引导和表述模块可以跨执行模型迁移，显示它们有潜力改进其他大型语言模型。项目页面：此 https URL。

ModelBlaster: Closing the Performance Gap through Learning-Augmented Agents Across Models, Kernels and Hardware Backends

ModelBlaster：通过跨模型、内核和硬件后端的学习增强代理缩小性能差距

• Authors: Shengjun Kris Dong, Dima Nikiforov, Sahil Modi, Sana Damani, Edward C Lin, Siva Kumar Sastry Hari, Christos Kozyrakis
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04104
• Pdf link: https://arxiv.org/pdf/2611.04104
• Abstract Optimizing accelerator code across hardware architectures is difficult. ML compilers search predefined spaces of schedules and micro-kernels, while LLM-guided optimization can explore alternative algorithms, operator decompositions, and hardware-specific techniques. However, existing LLM agents typically target a single programming interface, toolchain, or accelerator family, focus on kernel-level optimization without model-level transforms, and offer limited reuse of experience across hardware targets. We present ModelBlaster, a Memory-Augmented In-Context Reinforcement Learning (MAIC-RL) framework for optimizing data-parallel kernels and whole models across heterogeneous SIMD and SIMT accelerators. Its mutable policy is a persistent knowledge base updated at inference time by Actor-Critic Textual Gradient Descent (AC-TGD). The knowledge base maps performance states to actions, outcomes, and constraints, enabling experience reuse across kernels, models, and hardware targets. A model-level optimizer uses kernel-level knowledge to inform whole-model decisions. The same optimization loop and knowledge schema support CUDA on NVIDIA GPUs, OpenCL on Qualcomm GPUs, and RVV intrinsics on RISC-V vector cores, with only lowering and measurement layers specific to each target. Compared with PyTorch, ModelBlaster achieves geometric-mean speedups of 1.85x, 5.58x, and 5.18x on KernelBench Levels 1, 2, and 3, and 1.69x on SOL-ExecBench Levels 1 and 2 on a B200 GPU. On Qualcomm Adreno GPUs, it achieves 5.8x over the vendor SDK and 1.7x over Apache TVM; on RISC-V vector cores, it achieves 2.7x over ExecuTorch with XNNPACK across 70 KernelBench problems.
• 中文摘要 跨硬件架构优化加速器代码较为困难。机器学习编译器搜索预定义的调度空间和微内核，而LLM引导优化则可探索替代算法、算符分解和硬件特定技术。然而，现有LLM代理通常针对单一编程接口、工具链或加速器家族，专注于内核级优化而不进行模型级转换，且有限度地提供跨硬件目标的经验重用。我们介绍ModelBlaster，一个内存增强上下文强化学习（MAIC-RL）框架，用于优化跨异构SIMD和SIMT加速器的数据并行内核和整体模型。其可变策略是一个持久的知识库，在推理时由Actor-Critic文本梯度下降（AC-TGD）更新。知识库将性能状态映射到动作、结果和约束，实现内核、模型和硬件目标间的体验重用。模型级优化器利用内核级知识来指导整体模型决策。相同的优化循环和知识模式支持NVIDIA GPU上的CUDA、高通GPU上的OpenCL和RISC-V向量核上的RVV内在技术，仅针对每个目标设置特定的下层和测量层。与PyTorch相比，ModelBlaster在KernelBench 1、2和3级实现了1.85倍、5.58倍和5.18倍的几何平均加速，在B200 GPU上的SOL-ExecBench 1级和2级实现了1.69倍。在高通Adreno GPU上，它比厂商SDK实现了5.8倍，比Apache TVM提升1.7倍;在 RISC-V 向量核上，在 70 个 KernelBench 问题中，它在 XNNPACK 的 ExecuTorch 上实现了 2.7 倍的效能。

Where Did the Update Go? Certifying Declared Change Envelopes for Model Updates

更新去哪了？认证模型更新的声明变更信封

• Authors: Muhammad Usama, Muhammad Saad Qadir
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04192
• Pdf link: https://arxiv.org/pdf/2611.04192
• Abstract Deployed language models are updated continually, and regulators increasingly accept changes that were declared in advance. Showing that an update changed behavior only where it was declared to is a different statistical problem from bounding how much it changed: a useful update is meant to change some decisions, so global disagreement and regression bounds cannot certify it. We formalize the problem with declared change envelopes. Before training, the developer commits to a region of inputs, and every decision that changes outside it counts as leakage. Leakage compares two models, so it can be certified from unlabeled traffic with anytime-valid tests whose sample cost we bound from above and below, and per-update certificates compose into a bound on lifetime drift outside all declared scopes. We measure leakage for LoRA and full fine-tuning, preference optimization, reinforcement learning, and knowledge editing on models from 1.5B to 72B parameters. Narrow updates leak widely: domain and policy LoRA updates keep only about 7-36% of the decisions they change inside the declared scope. Across 256 trained variants, leakage tracks the update's KL divergence to the incumbent on out-of-scope inputs (Spearman 0.94). None of the customary remedies we tested makes a rule-like update certifiable without giving up much of its intended change. Distilling the incumbent's decisions on unlabeled prompts in the deployment format does: anchored LoRA, GRPO, and (at a small learning rate) full fine-tuning updates certify 5% leakage from a few hundred to a few thousand unlabeled samples, and the certificates replicate on held-out traffic and on written answers. The certificates cover decision-like outputs only; on open-ended chat, re-batching alone changes 11-21% of responses and no update certifies. Code: this https URL
• 中文摘要 部署的语言模型不断更新，监管机构越来越接受事先声明的变更。证明更新只在被声明的地方改变行为，与限制其变化程度是不同的统计问题：有用的更新旨在改变某些决策，因此全局不同意和回归界限无法证明该变化。我们通过声明变更包络来形式化问题。在训练前，开发者承诺对一个输入区域进行承诺，所有超出该区域的决策都计入泄漏。泄漏比较了两个模型，因此可以通过任意时间有效测试认证，这些测试的样本成本从上下界限确定，且每次更新的证书组成了所有声明范围之外的终身漂移受限。我们测量LoRA的泄漏，并对15B至72B参数的模型进行全面微调、偏好优化、强化学习和知识编辑。狭窄更新泄露广泛：域和策略LoRA更新只保留约7-36%的决策在宣告范围内。在256种训练过的变体中，泄漏追踪更新与当前在域外输入（Spearman 0.94）的基层更新的分歧。我们测试的任何惯常补救措施都无法使类似规则的更新在不放弃大部分预期变更的情况下被认证。在部署格式中提取现有者对未标记提示的决策则有效：锚定的LoRA、GRPO和（以较低学习率）完整微调更新，能从几百到几千个未标记样本中认证5%的泄漏，证书在保留流量和书面回答上复制。证书仅覆盖决策类输出;在开放式聊天中，仅重新批处理仅改变11-21%的回复，且无更新认证。代码：这个 https URL

Harness-S: Learning from Sibling Harnesses for Generalizable Agent Training

Harness-S：从兄弟框架学习以实现通用代理培训

• Authors: Kangning Zhang, Wenxiang Jiao, Bowen Fang, Shuai Shao, Jiachen Zhu, Rong Shan, Shijian Wang, Hao Wang, Yuan Lu, Weiwen Liu, Weinan Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04277
• Pdf link: https://arxiv.org/pdf/2611.04277
• Abstract Harness evolution from a common base exhibits Locality, Complementarity, and Nonstationarity: variants repair different failures, and their utility changes as the policy learns. Rather than seeking a universally superior harness, we introduce Harness-S to selectively rescue failures of a base execution system through sibling harnesses and transfer the resulting capabilities into a single policy that remains effective under the original harness. Directional Harness Evolution develops specialized sibling harnesses along distinct repair directions, forming a reusable Sibling Harness Pool that remains fixed during policy training. Complementary Harness Rescue Training invokes this pool only when all initial attempts under the base harness fail. The same policy generates fresh rollouts under selected auxiliary harnesses, and these trajectories replace the failed group for reinforcement learning. An Adaptive Harness Router selects the Top-k variants using Harness Behavior Profiles and voting across multiple LLMs. Policy-aware profile re-anchoring refreshes this evidence as the policy learns. Across AppWorld, SWE-Gym, and DBBench, the trained policy substantially outperforms policy-only training when returned to the original base harness. Behavioral analyses and held-out cases support the transfer of harness-elicited behaviors into the policy itself. Reusing the router for Rank-1 inference provides additional gains at the same executor rollout count.
• 中文摘要 从共同基础演进出的束带演进表现出局部性、互补性和非固定性：变体修复不同的故障，其效用随着策略学习而变化。我们不寻求普遍优越的束带，而是引入束带S，通过兄弟束选择性救援基础执行系统的故障，并将所得能力转移到单一策略中，在原束下依然有效。方向性束带演化沿不同修复方向开发专用的兄弟束，形成可重复使用的兄弟束池，在策略训练期间保持固定。补充束带救援训练仅在基础束下所有初始尝试失败时调用该池。同一策略在选定辅助束下生成新的部署，这些轨迹替代失败的组进行强化学习。自适应束束路由器通过束缚行为配置文件和跨多个大型语言模型的投票选择顶k个变体。策略感知型配置重新锚定会刷新这些证据，随着策略的学习。在AppWorld、SWE-Gym和DBBench中，训练策略在返回原始基础框架后，显著优于仅策略训练。行为分析和保留案例支持将束缚引发的行为转移到策略本身。重复使用路由器进行Rank-1推断，在相同的执行者部署次数下提供了额外收益。

CORA: Teacher-Guided Causal O-RAN Adaptation for Jamming Detection

CORA：教师引导因果O-RAN干扰检测适配

• Authors: Sneihil Gopal, Richard A. Rouil
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04330
• Pdf link: https://arxiv.org/pdf/2611.04330
• Abstract In mission-critical networks, a missed jamming event is costlier than a false alarm, so detectors should favor recall, operate causally, and work without labeled attack data. Retrospective Key Performance Indicator (KPI) detectors meet the label-free requirement but rely on future context and cannot run in streaming operation. We present CORA, a teacher-guided reinforcement learning (RL) framework that transfers the hindsight of a retrospective detector to a causal one without labels during training or deployment. Offline, a retrospective KPI detector evaluated on unlabeled historical logs defines a recall-weighted reward for a masked controller that adapts the sensitivity, persistence, and grace of a causal KPI detector. At deployment, the teacher is removed and the frozen controller acts causally, with no ground-truth labels and no target-specific calibration. The design maps onto offline training in the Non-Real-Time RIC and causal control in the Near-Real-Time RIC of an Open Radio Access Network (O-RAN). Under leave-one-out transfer across ten 5G O-RAN testbed configurations, CORA reaches recall-oriented operating points on the deployable frontier that none of the evaluated baselines covers. A Proximal Policy Optimization (PPO) controller raises mean recall from 0.687 to 0.771 over a fixed default detector on all ten configurations, while a Deep Q-Network (DQN) achieves a comparable F2-score to the best teacher-selected static configuration, with 61% fewer false alarms. Removing the teacher lowers recall on every configuration.
• 中文摘要 在关键任务网络中，漏接干扰事件比误报更昂贵，因此探测器应优先回忆、因果性工作，且在无标记攻击数据的情况下工作。回顾性关键绩效指标（KPI）探测器满足无标签要求，但依赖未来上下文，无法在流式运行中运行。我们介绍CORA，一种教师引导的强化学习（RL）框架，将回顾性探测器的后见之明转移到培训或部署期间无标签的因果检测器。离线时，基于无标签历史日志评估的回顾性KPI检测器，定义了针对掩蔽控制器的回忆加权奖励，该控制器适应因果KPI检测器的敏感性、持久性和优雅性。部署时，教师被移除，冻结的控制者以随意方式行动，没有真实标签，也无针对特定目标的校准。该设计将开放无线接入网（O-RAN）的非实时RIC离线训练和近实时RIC中的因果控制映射。在十个5G O-RAN测试平台配置的留一输出传输下，CORA达到可部署前沿的回忆导向操作点，这些点评测基线未涵盖。近端策略优化（PPO）控制器在所有十种配置中均值回忆率从0.687提升至0.771，而深度Q网络（DQN）则获得与最佳教师选择静态配置相当的F2分数，误报减少61%。移除教师会降低所有配置的回忆率。

Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs

针对专家混合大型语言模型（LLM）的端到端FP4强化学习的培训和推广政策进行对齐

• Authors: Hei Yi Mak, Hoang Le, Mehran Taghian Jazi, Yunke Peng, Huan Lin, Zhengzhi Tang, Zhexu Liu, Yao Wang, Yaoyuan Wang, Anandharaju Durai Raju, Tanzila Rahman
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04376
• Pdf link: https://arxiv.org/pdf/2611.04376
• Abstract In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoeResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoeResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.
• 中文摘要 本研究中，我们研究了专家混合（MoE）大型语言模型的FP4强化学习（RL）训练后端到端。近期FP4强化学习引入了有效的恢复配方，可在密集模型的激进低精度量化下恢复准确性，但我们发现相同方法在MoE环境中并未优于朴素FP4基线。我们进行了深入研究以理解原因。我们的分析指出MoE独特路由机制的根本原因：由于路由器做出离散的专家选择决策，训练-推理策略的对齐对是否在训练和推理双方对称应用的恢复技术变得极为敏感。当使用非对称配方——包括仅纠正推广-训练-推断对齐的稠密模型方法时，训练与推断概率的Pearson相关系数及路由器选择匹配率将显著恶化。基于这一发现，我们提出了MoeResQ，一种针对训练后MoE强化的量化-恢复配方，强制训练侧与推断侧修正之间的对称性。MoeResQ恢复训练-推断对齐，并大幅缩小准确率差距至全精度，确立对称恢复作为FP4 RL中MoE模型的关键原则。

CANDO: Conservative Action-Noise Directional Optimization for Residual Reinforcement Learning

CANDO：残余强化学习中的保守动作噪声方向优化

• Authors: Zeyu Gao, Chengkai Hou, Chaoyue Li, Zhuangzhe Wu, Jiaming Liu, Yao Mu, Peng Jia, Li Zhang, Shanghang Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04543
• Pdf link: https://arxiv.org/pdf/2611.04543
• Abstract Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult. A central challenge is how to improve a frozen generative policy without destroying the structure learned from offline demonstrations. Noise-space residual methods address this by modifying the policy's input noise, but existing noise-space methods typically regulate adaptation through fixed latent feasible regions, which may exclude valuable noise edits. In contrast, action-space residual methods are more sample-efficient, but their efficiency can still be further improved. We propose Conservative Noise Directional Optimization (CNDO), a noise-space residual RL method that improves a frozen generative policy by learning state- and noise-conditioned directions around Gaussian latent samples. CNDO removes predefined latent feasible regions and instead regulates noise-space adaptation through the soft conservative constraint, which consists of conservative value estimation and gated execution. We further introduce Conservative Action-Noise Directional Optimization (CANDO), which adds an action-space residual, combining structured latent-space improvement with local action correction. Experiments show that, once CNDO narrows the gap between noise-space and action-space residual learning, CANDO further improves over single-space optimization, achieving 76.1 NAUC and a 91.4% final success rate, surpassing the best single-space results by 6.6 NAUC and 4.1% success rate. More demos and info are available on this https URL.
• 中文摘要 预训练生成策略为机器人控制提供了强有力的行为先验，但通过在线强化学习高效适应这些策略仍然困难。一个核心挑战是如何在不破坏离线演示中学到的结构的情况下改进冻结生成策略。噪声空间残差方法通过修改策略的输入噪声来解决这个问题，但现有噪声空间方法通常通过固定的潜在可行区域来调节适应，这可能排除有价值的噪声编辑。相比之下，作用空间残差方法采样效率更高，但其效率仍有提升空间空间。我们提出了保守噪声方向优化（CNDO），这是一种噪声空间残差强化学习方法，通过学习高斯潜在样本周围的状态和噪声条件方向来改进冻结生成策略。CNDO去除预设的潜在可行区域，转而通过软保守约束调节噪声空间适应，该约束包括保守值估计和门控执行。我们进一步引入保守作用噪声方向优化（CANDO），该技术增加了作用空间残差，结合了结构化潜空间改进与局部作用修正。实验表明，一旦CNDO缩小噪声空间与动作空间残差学习之间的差距，CANDO在单空间优化方面进一步提升，达到76.1 NAUC和91.4%的最终成功率，超越最佳单空间结果6.6 NAUC和4.1%的成功率。更多演示和信息可在此 https 网址获取。

Policy Network Diffusion

政策网络扩散

• Authors: Jiate Liu, Deheng Ye
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04614
• Pdf link: https://arxiv.org/pdf/2611.04614
• Abstract Diffusion models have shown remarkable progress in AI-generated content, including images and videos. In this paper, we explore whether diffusion models can generate the parameters of policy networks in reinforcement learning (RL), and whether the generated policies exhibit generalization capabilities. To achieve this, we propose Diffusion-based Conditional Policy Network Parameter Generation (DPPG), which consists of a Transformer-based autoencoder and a task-conditioned diffusion model. The autoencoder achieves a high compression rate by patchifying network parameters, while the task-conditioned diffusion model generates corresponding parameters conditioned on different environments. Our experiments demonstrate that, with a single unified DPPG model, we can synthesize high-quality policies across a diverse set of tasks from different environment domains, including Classic Control, Atari games, and MuJoCo. Additionally, our method can generate networks with a larger number of parameters compared to previous approaches. Furthermore, we find that in previously unseen environments, DPPG can generate parameters that achieve faster convergence and reach competitive performance, showcasing the potential future applications of parameter generation via diffusion.
• 中文摘要 扩散模型在AI生成内容（包括图片和视频）方面取得了显著进展。本文探讨扩散模型是否能在强化学习（RL）中生成策略网络参数，以及生成策略是否具备泛化能力。为此，我们提出了基于扩散的条件策略网络参数生成（DPPG），该方法由基于Transformer的自编码器和任务条件扩散模型组成。自编码器通过补缀网络参数实现高压缩率，而任务条件扩散模型则根据不同环境生成相应参数。我们的实验表明，通过单一统一的DPPG模型，我们可以综合涵盖来自不同环境域的多样化任务，包括经典控制、雅达利游戏和MuJoCo。此外，我们的方法能够生成参数数量更多的网络。此外，我们发现在前所未有的环境中，DPPG能够生成更快收敛并达到竞争性能的参数，展示了通过扩散实现参数生成的潜在未来应用。

Self-Privileged Fine-Tuning

自我特权微调

• Authors: Yangrong Hu, Xueyu Zhou, Yue Wu, Jian Huang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.04632
• Pdf link: https://arxiv.org/pdf/2611.04632
• Abstract Supervised fine-tuning (SFT) is a key component of language model post-training, enabling effective task adaptation and serving as an important foundation for subsequent reinforcement learning. Recent methods seek to improve SFT by introducing token-level weights into the training objective, where these weights are typically derived from a model's output distribution at each token position. To better understand how such weighting schemes affect SFT, we analyze their optimization objectives and theoretically characterize the target distributions they induce. Our analysis shows that dynamic weighting methods, such as Dynamic Fine-Tuning (DFT), induce a heavily sharpened distribution, whereas static weighting methods fit a target distribution proportional to the product of the data distribution and a frozen prior. These findings suggest that static weighting provides a more stable training target, while its effectiveness critically depends on the quality of the prior distribution. Building on this observation, we propose Self-Privileged Fine-Tuning (SPFT), which uses a frozen copy of the initial model conditioned on privileged information (such as ground truth for mathematical questions) as a prior to provide guidance for token-level weighting during fine-tuning. Through a Bayesian decomposition, we further show that SPFT reweights the standard static prior according to how informative each target token is for predicting the ground-truth outcome, thereby assigning larger weights to more predictive tokens. Experiments on mathematical reasoning and code generation show that SPFT improves over standard SFT and other baselines across the main settings.
• 中文摘要 监督微调（SFT）是语言模型训练后的关键组成部分，能够有效适应任务，并作为后续强化学习的重要基础。近期方法通过在训练目标中引入代币级权重来改进SFT，这些权重通常来自模型在每个标记位置的输出分布。为了更好地理解此类加权方案如何影响SFT，我们分析了它们的优化目标，并理论上描述了它们诱导的目标分布。我们的分析表明，动态加权方法如动态微调（DFT）会诱导出高度锐化的分布，而静态加权方法则拟合与数据分布与冻滞先验乘积成正比的目标分布。这些发现表明静态加权提供了更稳定的训练目标，而其有效性则关键于先验分布的质量。基于这一观察，我们提出了自我特权微调（SPFT），该方法以初始模型的冻结副本为条件，基于特权信息（如数学问题的基层真值）作为先验，为微调时的令牌级权重提供指导。通过贝叶斯分解，我们进一步表明，SPFT根据每个目标令牌对预测基层结果的信息量来重权标准静态先验，从而赋予更具预测性的权重。数学推理和代码生成的实验表明，SPFT在主要设置下优于标准SFT及其他基线。

BrainwashedLLM: Jailbreaking via Automated Adversarial Chain of Thoughts Discovery

BrainwashedLLM：通过自动对抗性思维链发现越狱

• Authors: Masoud Poorghaffar Aghdam, Göktuğ Serdar Yıldırım, Sinem Sav
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2611.04717
• Pdf link: https://arxiv.org/pdf/2611.04717
• Abstract Large Reasoning Models (LRMs) achieve strong performance on complex tasks by generating explicit Chain-of-Thought (CoT) reasoning, but the safety implications of this intermediate reasoning surface remain underexplored. We demonstrate that the reasoning trace itself is a powerful attack vector. Safety alignment in large models is substantially less robust to malicious reasoning embedded in the context than to direct imperative prompts. We introduce BrainwashedLLM, a weak-to-strong adversarial framework that automates the discovery of reasoning-based jailbreaks. First, we use Group Relative Policy Optimization (GRPO), a Reinforcement Learning (RL) algorithm, with a multi-objective rule-based reward to fine-tune a small, white-box source model to generate adversarial reasoning traces. Crucially, the reward is computed entirely over the source's own reasoning trace. Second, we show that these traces transfer in a zero-shot manner. When traces are injected as input prefixes, they steer larger black-box LRMs to continue the adversarial reasoning and produce harmful outputs, without requiring prompt optimization, gradient access, or any form of output feedback from the target model. Across four state-of-the-art LRMs that span distinct architectures, BrainwashedLLM achieves ASR@5 between 0.74 and 0.98 across targets, within 16 points of query-based baselines, without querying the target during attack construction. To our knowledge, this is the first framework to use RL to directly optimize the CoT of LRMs as an attack surface without any target feedback, and the first to demonstrate zero-shot weak-to-strong transfer of reasoning traces. Our results suggest that exposed CoT is a vulnerability in LRMs and motivate architectural defenses that isolate the reasoning buffer. Warning: This paper contains potentially offensive and harmful text.
• 中文摘要 大型推理模型（LRM）通过生成显式思维链（CoT）推理，在复杂任务中表现出色，但这一中间推理表面的安全性影响尚未被充分探讨。我们证明推理追踪本身是一种强大的攻击向量。大型模型中的安全对齐对嵌入上下文中的恶意推理的鲁棒性远低于对直接命令提示的抵抗。我们介绍了BrainwashedLLM，一个弱到强的对抗框架，用于自动化发现基于推理的越狱。首先，我们使用群相对策略优化（GRPO），这是一种强化学习（RL）算法，通过多目标基于规则的奖励，微调一个小型白箱源模型以生成对抗性推理追踪。关键是，奖励完全基于源自身的推理追踪计算。其次，我们证明这些痕迹以零射击方式传输。当输入前缀注入痕迹时，它们引导更大的黑匣子LRM继续对抗推理，产生有害输出，无需即时优化、梯度访问或目标模型的任何输出反馈。在四种跨越不同架构的先进LRM中，BrainwashedLLM在攻击构建过程中无需查询目标，在查询基线16点内实现目标间0.74至0.98的 ASR@5。据我们所知，这是首个利用强化语言直接优化LRM的CoT作为攻击面且无目标反馈的框架，也是首个展示零shot弱到强推理痕迹传输的框架。我们的结果表明暴露的CoT是LRM中的一个漏洞，并促使建筑防御以隔离推理缓冲区。警告：本文包含可能冒犯和有害的文字。

OneTrans-GSR: A Generative Slate Ranking Paradigm Unifying Reranking and Mix-Ranking with One Autoregressive Transformer

OneTrans-GSR：一种生成式平板排名范式，统一重新排名和混合排名，采用一个自回归变换器

• Authors: Feng Cheng, Yu-Hsuan Huang, Fengyi Wang, Chaofan Zhou, Zhennan Zhu, Huan He, Bo Zhang, Ruoqi Zhang, Zhanfeng Mo, Ziwei Liu, Lanyi Li, Yifeng Pan, Junhua Long, Ziqiu Zhou, Yue Lin, Shaowei Liu
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.04720
• Pdf link: https://arxiv.org/pdf/2611.04720
• Abstract Industrial recommender systems allocate substantial computation to pointwise candidate scoring, yet separate reranking and mix-ranking stages still assemble the final exposure slate using hand-crafted rules and specialized low-capacity models. Mix-ranking combines product cards, short videos, livestreams, and advertising cards supplied by distinct upstream business channels. Although these rules and models encode valuable domain knowledge, they are inherently limited in leveraging additional data and computation. They constrain the combinatorial space of candidate slates and determine objective trade-offs through manual tuning. To enable scalable and effective slate generation, we reformulate reranking and mix-ranking as a unified learning problem whose performance can improve with computation. Specifically, we instantiate this formulation across the reranking and mix-ranking stages and propose OneTrans-GSR, a single decoder-only policy that extends our deployed OneTrans backbone to generate the exposure slate end-to-end. During training, a reward model evaluates complete slates and guides reinforcement learning toward slate-level and long-term objectives. During decoding, business constraints are explicitly enforced. Offline evaluation shows that slate quality consistently improves as reward-model and policy compute increase. Deployed in a large-scale industrial recommendation system, OneTrans-GSR mixes heterogeneous content types within a unified slate. In online A/B tests, it improves GMV/u by 2.63%, advertiser value by 2.06%, and repurchase rate by 2.03%, while reducing end-to-end latency across the reranking and mix-ranking stages by 13%. These results mark a paradigm shift in slate generation: moving beyond human-engineered heuristics toward a general learned policy creates a scalable path for further algorithmic optimization and compute-driven improvement.
• 中文摘要 工业推荐系统为点数候选人评分分配大量计算，但分别的重新排序和混合排序阶段仍通过手工规则和专门的低容量模型组装最终曝光表。混合排名结合了产品卡、短视频、直播和广告卡，这些内容来自不同的上游业务渠道。尽管这些规则和模型编码了宝贵的领域知识，但在利用额外数据和计算方面本质上存在限制。它们限制候选名单的组合空间，并通过手动调整确定客观权衡。为了实现可扩展且高效的板块生成，我们将重新排序和混合排序重新表述为一个统一学习问题，其性能可随着计算提升而提升。具体来说，我们在重新排序和混合排名阶段实现了这一表述，并提出了OneTrans-GSR，这是一项仅适用于解码器的策略，扩展了我们部署的OneTrans骨干网，实现了端到端的曝光表生成。在培训过程中，奖励模型评估完整的表单，引导强化学习朝向板级和长期目标。在解码过程中，业务约束会被明确执行。离线评估显示，随着奖励模型和策略计算量的增加，板块质量持续提升。在大规模工业推荐系统中部署时，OneTrans-GSR在统一的板块内混合了异构内容类型。在在线A/B测试中，它提升了2.63%的GMV/u，提升了2.06%的广告主价值，2.03%的回购率提升了2.03%，同时在重新排名和混合排名阶段的端到端延迟减少了13%。这些结果标志着板块生成的范式转变：超越人机工程启发式，转向通用的学习策略，为进一步的算法优化和计算驱动改进创造了可扩展的路径。

Cooperative Multi-Agent Reinforcement Learning with Submodular Trajectory Rewards

合作多智能体强化学习与亚模块轨迹奖励

• Authors: Yasaman Zolfimoselo, Nikolas Geroliminis, Maryam Kamgarpour
• Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.04774
• Pdf link: https://arxiv.org/pdf/2611.04774
• Abstract We study cooperative multi-agent reinforcement learning with decoupled dynamics and a monotone submodular reward function over joint trajectories. Due to the non-additive reward, an optimal policy may depend on the history of all agents. This dependence makes exact optimization exponential in the horizon and the number of agents. To overcome this challenge while retaining theoretical guarantees, we introduce pre-window curvature $c_m$, which quantifies how states visited more than $m$ steps earlier reduce current marginal gains. Using this curvature to control the error from restricting the policy of each agent to depend only on its last $m+1$ states, we propose a greedy algorithm that exploits the decoupled dynamics to optimize local policies sequentially. For fixed $m$, the algorithm achieves a $\frac{1-c_m}{2-c_m}$ approximation guarantee up to estimation error, and its computation time is polynomial in the number of agents and the horizon. We recover the known $1/2$-factor approximation for time-additive rewards and obtain an approximation factor no smaller than the known total-curvature bound for a single agent. We formulate multi-agent traffic monitoring as an example of this framework and evaluate our algorithm on it.
• 中文摘要 我们研究了结合耦合动力学和单调子模奖励函数在联合轨迹上的合作多智能体强化学习。由于非加性奖励，最优策略可能依赖于所有代理的历史。这种依赖性使得精确优化在视野和代理数量上呈指数级。为克服这一挑战同时保留理论保证，我们引入了窗口前曲率$c_m$，该方法量化了早于$m步的状态如何减少当前边际收益。利用该曲率控制每个代理策略仅依赖其最后$m+1$状态的误差，我们提出了一种贪婪算法，利用解耦动力学顺序优化局部策略。对于固定$m$，算法实现了在估计误差范围内的$\frac{1-c_m}{2-c_m}$的近似保证，其计算时间与代理数量和视界等于多项式。我们恢复了已知的$2$因子近似，并获得一个不小于单个代理已知总曲率界限的近似因子。我们以多代理流量监控为例，并基于此评估算法。

ReSI-Math: Exploring Recoverable Capability Frontiers for Efficient Reasoning Improvement

ReSI-Math：探索可恢复能力前沿以提升高效推理

• Authors: Yao Cui, Zhen Yang, Yuandong Wang, Yangfu Zhu, Tianyu Wo, Zhenzhou Shao
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04787
• Pdf link: https://arxiv.org/pdf/2611.04787
• Abstract Reinforcement learning with verifiable rewards (RLVR) has become an effective approach for improving mathematical reasoning in large language models. However, a fundamental question remains: when a model fails to solve a problem, does it truly lack the underlying capability? In this work, we reveal that direct failure does not necessarily imply inability. Some reasoning failures can be recovered by exploring the model's own trajectories, exposing a recoverable capability frontier: a policy-dependent region where solutions are not reliably generated through direct decoding but can be elicited through trajectory-level search. Based on this observation, we introduce ReSI-Math, a framework that explores recoverable capability frontiers and converts recovered capabilities into data-efficient RLVR supervision. ReSI-Math uses Self-Judge-guided evolutionary search to identify recoverable failures under a fixed policy and adaptively selects frontier problems for sequential RLVR as the policy evolves. On Qwen3-VL-8B, correct-incorrect trajectory crossover produces 272 verifier-confirmed correct offspring from 621 attempts (43.8%). After three sequential one-shot RLVR stages, the MATH500 and WE-MATH average improves from 78.71 to 80.16. Under matched single-problem RLVR, random selection reaches only 53.31 versus 54.16 for frontier-guided Q1.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为提升大型语言模型数学推理能力的有效方法。然而，一个根本性问题依然存在：当模型未能解决问题时，它是否真的缺乏潜在能力？在本研究中，我们揭示了直接失败并不一定意味着无法实现。一些推理失败可以通过探索模型自身轨迹来恢复，揭示了一个可恢复能力前沿：一个依赖策略的区域，解不是通过直接解码可靠生成，而是可以通过轨迹级搜索引出。基于这一观察，我们引入了ReSI-Math框架，探索可恢复能力前沿，并将恢复的能力转化为数据高效的RLVR监督。ReSI-Math 采用自判引导的进化搜索，识别固定策略下的可恢复失败，并随着策略演进，自适应地选择前沿问题进行顺序 RLVR。在 Qwen3-VL-8B 上，正确-错误轨迹交叉在 621 次尝试中产生 272 个验证者确认的正确后代（43.8%）。经过三个连续的单次 RLVR 阶段后，MATH500 和 WE-MATH 平均值从 78.71 提升至 80.16。在匹配单问题 RLVR 下，随机选择仅为 53.31，而前沿引导 Q1 为 54.16。

Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling

密度感知奖励尺度：一种用于奖励比例抽样的强化学习框架

• Authors: Ria Shekhawat, Abhijeet Sinha, Yingtao Zhu, Dianbo Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04792
• Pdf link: https://arxiv.org/pdf/2611.04792
• Abstract Reinforcement learning objectives that maximize expected return can concentrate probability mass on a small subset of high-reward outcomes, limiting diversity when multiple distinct solutions are desirable. We introduce Density-Aware Reward Scaling (DARS), a simple reward-processing mechanism that promotes reward-proportional sampling in continuous outcome spaces while retaining standard policy-gradient training. DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by this density. The method requires no learned density model, auxiliary objective, critic, or modification to the underlying policy-gradient objective. In a controlled multimodal navigation task, DARS-GRPO avoids the single-mode collapse exhibited by GRPO and closely recovers the target mode allocation. Applied to text-to-image prompt adaptation, DARS-GRPO consistently improves image and prompt diversity across datasets, Policy LMs, and reward settings while maintaining good semantic alignment. These results show that density-aware reward scaling provides a lightweight way to improve coverage of diverse high-reward outcomes within existing reinforcement-learning pipelines.
• 中文摘要 最大化期望回报的强化学习目标可以将概率质量集中在一小部分高回报结果上，限制多样性，当需要多个不同解时。我们引入了密度感知奖励尺度（DARS），这是一种简单的奖励处理机制，在连续结果空间中促进奖励比例抽样，同时保留标准策略梯度训练。DARS 估计每个政策内推广组内的结果局部密度，并按该密度反向调整奖励。该方法无需学习密度模型、辅助目标、批判者或对底层策略梯度目标的修改。在受控多模态导航任务中，DARS-GRPO 避免了 GRPO 表现出的单模态崩溃，并紧密恢复目标模式分配。应用于文本到图像提示的适应，DARS-GRPO在保持良好语义对齐的同时，持续提升了数据集、策略LM和奖励设置的图像和提示多样性。这些结果表明，密度感知奖励尺度为提升现有强化学习流程中多样化高奖励结果的覆盖提供了一种轻量级方法。

Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates

在手机上训练专家大型语言模型：仅通过块谱坐标实现前向优化

• Authors: Liyan Tan, Yequan Zhao, Ziyue Liu, Yupeng Su, Ruijie Zhang, Zheng Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.04797
• Pdf link: https://arxiv.org/pdf/2611.04797
• Abstract Large language models (LLMs) on personal devices keep user data local, but limited memory restricts their general capability. Since most user-specific tasks do not require the breadth of frontier models, a practical solution is to adapt a shared on-device model into lightweight specialists. However, backpropagation is prohibitive in memory, and users rarely provide labeled data. We present MobileSpec, a system that trains specialists with forward passes alone and learns from automatically checked task outcomes, as in reinforcement learning with verifiable rewards (RLVR). Because forward-only updates are built from random perturbations, training is efficient only when these perturbations resemble the weight changes of RLVR, which largely preserve the singular values of pretrained weights and rotate their singular vectors. The core algorithm of MobileSpec, FuRA-S, produces such perturbations while training only the block-spectral coordinates of pretrained weights, namely the singular values of their column blocks. As we show theoretically and empirically, rescaling these blocks independently shifts their relative contributions to each weight matrix and thereby rotates its singular vectors. Since this update can be rewritten as a transformation of the layer input, FuRA-S also adapts INT4 models without modifying their weights. With specialists of only 1-5 MB that update 0.015-0.05% of the parameters, MobileSpec raises the success rate of a phone-use agent from 6-23% up to 89-100% and trains a personalized 8B model on an iPhone 16 Pro overnight. On RLVR benchmarks, FuRA-S is competitive with full-parameter evolution strategies (ES) and outperforms ES over LoRA adapters and quantized ES on INT4 models.
• 中文摘要 个人设备上的大型语言模型（LLMs）将用户数据保存在本地，但有限的内存限制了其通用能力。由于大多数用户特定任务不需要前沿模型的广泛性，一个实用的解决方案是将共享的设备内模型改编为轻量级专家模型。然而，反向传播在内存中占用量很大，用户很少提供带标签的数据。我们介绍MobileSpec系统，该系统仅通过前向传递训练专家，并从自动检查的任务结果中学习，类似于带有可验证奖励的强化学习（RLVR）。由于仅前向更新是由随机扰动构建的，只有当这些扰动类似于RLVR的权重变化时，训练才有效，后者在很大程度上保留预训练权重的奇异值并旋转其奇异向量。MobileSpec的核心算法FuRA-S在仅训练预训练权重的块谱坐标（即列块的奇异值）时，会产生此类扰动。正如我们理论和实证所示，独立重缩放这些块会改变它们对每个权重矩阵的相对贡献，从而旋转其奇异向量。由于此次更新可重写为层输入的变换，FuRA-S也能在不修改权重的情况下适配INT4模型。只需1-5 MB的专家，更新0.015-0.05%的参数，MobileSpec将手机代理的成功率从6-23%提升至89-100%，并在iPhone 16 Pro上一夜间训练个性化的8B模型。在RLVR基准测试中，FuRA-S在全参数演化策略（ES）中具有竞争力，并且在INT4模型上优于LoRA适配器和量化ES。

Reconstructing Fraudulent Strategies in Online Payments via Interpretable Inverse Reinforcement Learning

通过可解释的逆向强化学习重建在线支付中的欺诈策略

• Authors: Huilin He, Weiyu Han, Linbo Shao, Ruobing Zuo, Xiao Tan, Shuai Chen, Dawei Cheng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04819
• Pdf link: https://arxiv.org/pdf/2611.04819
• Abstract Payment fraud causes financial losses and undermines users' trust in web payment services. Although fraud detection is a critical defense for payment security, complex and diverse fraudulent behaviors can still evade detection. Generating fraud samples for data augmentation can broaden detectors' coverage of different fraud patterns. However, existing methods matching the distribution of transaction attributes alone do not ensure that generated sequences preserve the sequential relationships among transaction choices, payment feedback, and subsequent behavioral adjustments. To this end, we propose PaySeqGen, an interpretable inverse reinforcement learning framework that reconstructs fraud behavior policies from fraudsters' observed payment trajectories. We design a mechanism for constructing candidate actions to support plausible payment sequence generation while enabling exploration. The framework also learns interpretable behavioral rewards that capture fraudsters' preferences across different transaction contexts. By alternating between reward learning and policy optimization, the framework reconstructs policies that generate new fraud sequences for detector augmentation and learns interpretable rewards supporting sequential fraud-pattern analysis in real trajectories. Experiments in real-world payment scenarios at Ant International, a leading global digital payment provider, show that PaySeqGen outperforms state-of-the-art generative augmentation methods. Reward-based trajectory analysis also identifies fraud patterns of practical value, helping analyst locating suspicious segments and examine multistep fraud tactics. Evaluation in the deployed system of online payment further confirms its effectiveness.
• 中文摘要 支付欺诈导致经济损失，削弱用户对网络支付服务的信任。尽管欺诈检测是支付安全的关键防御，但复杂多样的欺诈行为仍可能规避检测。生成用于数据增强的欺诈样本可以拓宽检测对不同欺诈模式的覆盖范围。然而，仅匹配交易属性分布的方法无法确保生成的序列保持交易选择、支付反馈及后续行为调整之间的顺序关系。为此，我们提出了PaySeqGen，一种可解释的逆强化学习框架，能够根据欺诈者观察到的支付轨迹重建欺诈行为策略。我们设计了一个构建候选动作的机制，支持合理的支付序列生成，同时支持探索。该框架还学习可解释的行为奖励，捕捉欺诈者在不同交易情境下的偏好。通过在奖励学习和策略优化之间交替，该框架重建了生成新欺诈序列用于检测增强的策略，并学习支持实际轨迹中顺序欺诈模式分析的可解释奖励。全球领先的数字支付服务提供商Ant International在真实支付场景中的实验显示，PaySeqGen的表现优于最先进的生成增强方法。基于奖励的轨迹分析还能识别具有实际价值的欺诈模式，帮助分析师定位可疑片段并分析多步骤欺诈策略。在已部署的在线支付系统中的评估进一步证实了其有效性。

FourSight: Stable FP4 RL Post-Training via Sequence Filtering and Sparse Channel Compensation

FourSight：通过序列滤波和稀疏通道补偿实现稳定的FP4强化学习后训练

• Authors: Yining Wei, Tianyuan Wu, Gu Gong, Yuechen Tao, Yinghao Yu, Xiaowen Chu, Wei Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.04887
• Pdf link: https://arxiv.org/pdf/2611.04887
• Abstract Reinforcement learning (RL) post-training increasingly targets low-bit deployment, with NVFP4 weight-and activation quantization (W4A4) becoming the state-of-the-practice format. However, integrating NVFP4 into RL degrades rollout quality and amplifies cross-engine training-inference mismatch (TIM), destabilizing training. We show that these failures are concentrated rather than uniform - severely mismatched tail tokens concentrate in a few outlier sequences, while policy weights remain largely intact and quantization error concentrates in a few high-variance activation channels. Guided by this, we present FourSight, an algorithm-system co-design with two targeted solutions: TIM-aware sequence filtering that excludes drifted trajectories from policy updates, and sparse precision compensation that keeps only top-variance channels in BF16 while retaining native NVFP4 elsewhere. We further design a fused SpliceGEMM kernel with specialized prefill and decode datapaths to minimize overhead. FourSight sustains near-BF16 accuracy and strong FP4 deployment quality across models and engines, while improving step-to-accuracy by 1.76-2.60x and staying within 2.3% of FP4 step time up to 120B-scale models.
• 中文摘要 强化学习（RL）训练后越来越多地针对低位部署，NVFP4权重与激活量化（W4A4）成为实践现状。然而，将NVFP4集成到RL中会降低推广质量，并放大跨引擎训练-推断不匹配（TIM），从而破坏训练稳定性。我们表明这些失败是集中而非均匀的——严重错配的尾令牌集中在少数离群序列中，而策略权重基本保持不变，量化误差集中在少数高方差激活通道。基于此，我们提出了FourSight算法-系统，协同设计，采用两种针对性解决方案：TIM感知序列过滤，排除策略更新中漂移轨迹;以及稀疏精度补偿，仅保留BF16顶部方差通道，同时保留原生NVFP4。我们还设计了一个融合的SpliceGEMM内核，配备专用的预填充和解码数据通路，以最大限度减少开销。FourSight在各模型和引擎上保持接近BF16的精度和强大的FP4部署质量，同时将步进精度提升1.76-2.60倍，且在120B级模型的步进时间保持在FP4的2.3%以内。

Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

视频生成背后：Kling 对多模态理解的观察与见解

• Authors: Kling Team, Dong Chen, Jialu Chen, Xinlong Chen, Yifan Dai, Haotian Dong, Binzong Geng, Yushuo Guan, Minghui Guo, Guanlin He, Guanxiao He, Jingyun Hua, Yanwei Huang, Yiyan Ji, Xueping Jia, Yizhuo Jia, Bozhou Li, Jiaze Li, Min Li, Shihao Li, Shujie Li, Shengyi Liao, Weihong Lin, Shiyuan Liu, Xinyu Liu, Siyang Mao, Xiaoru Qu, Dianmo Sheng, Yang Shi, Jiafu Tang, Longxiang Tang, Yuqi Tang, Chengzhuo Tong, Kuo Wang, Qixun Wang, Rui Wang, Tianduo Wang, Yuran Wang, Zekun Wang, Zihao Wang, Boyang Xia, Yifan Xu, Liu Yang, Zhenlong Yuan, Bohan Zeng, Mujin Zhang, Yuanxing Zhang, Zhicheng Zhang, Zhuoran Zhang, Yuanhong Zheng, Kun Gai, Chengru Song, Pengfei Wan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.04916
• Pdf link: https://arxiv.org/pdf/2611.04916
• Abstract Video generation has progressed from isolated clips to coherent short-form audiovisual sequences, and the bottleneck is shifting from rendering to semantic control. This report summarizes the observations and insights on multimodal understanding gained from successive generations of Kling. We argue that generation quality is the projection of multimodal understanding through the entire data--conditioning--alignment--evaluation chain, and we examine this chain module by module. Kling-MLLM, a shared multimodal foundation model that unifies image, video, audio, synchronized audio--video, and interleaved inputs, is specialized for each downstream module. Because a DiT is trained on far fewer training samples than a language model, we treat captioning as a semantic contract that maximizes description density, and extend it to reference-grounded captioning that makes reference bindings explicit. Prompt enhancement translates user intent into this caption distribution: basic PE is derived from captions of compelling videos and refined with text-level and generator-in-the-loop reinforcement learning, while agentic PE refines the story and retrieves compelling examples. We show that a reliable text encoder comes from perceptual training rather than from stronger language ability, and that candidates can be screened from text alone before end-to-end training. Reward modeling pairs a general reward model for explainable, localized critiques with a decision reward model for fast multidimensional scoring, treating perceptual bandwidth as a first-class design parameter. Finally, we review task-specific automatic evaluation, focusing on protocols that expose scoring shortcuts and help trace failures to their upstream source. We hope these lessons offer practical guidance for building the understanding stack behind controllable video generation.
• 中文摘要 视频生成已从孤立片段发展为连贯的短视频视听序列，瓶颈正从渲染转向语义控制。本报告总结了历代Kling对多模态理解的观察和见解。我们认为生成质量是多模态理解在整个数据-条件-对齐-评估链中的投射，逐模块审视这一链。Kling-MLLM是一个共享多模态基础模型，统一图像、视频、音频同步音频和交织输入，针对每个下游模块进行专门化。由于DiT训练的训练样本远少于语言模型，我们将字幕视为最大化描述密度的语义契约，并将其扩展到基于引用的字幕，使引用绑定显式化。提示增强将用户意图转化为这种标题分布：基础PE源自引人入胜视频的字幕，并通过文本级和生成器在环路的强化学习进行优化，而代理PE则细化故事并提取引人注目的示例。我们表明，可靠的文本编码器来自感知训练，而非更强的语言能力，且候选人可在端到端训练前仅通过文本筛选。奖励建模将用于可解释、局部批评的通用奖励模型与快速多维评分的决策奖励模型结合，并将感知带宽视为一流设计参数。最后，我们回顾了任务特定的自动评估，重点介绍揭示评分捷径并帮助追踪失败源头的协议。我们希望这些经验能为构建可控视频生成背后的理解栈提供实用指导。

Learning Where to Look: Observation Policy Optimization for Thinking with Images

学习寻找哪里：用图像思考的观察策略优化

• Authors: Junfeng Wang, Jiawei Liu, Yongchao Xu, Tao Jiang, Jiangbo Ai, Jin Zhang, Chong Wang, Qixing Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.04943
• Pdf link: https://arxiv.org/pdf/2611.04943
• Abstract Complex visual question answering often requires vision-language models to think with images, i.e., actively acquire local visual evidence by zooming into informative regions of high-resolution images. Our error analysis shows that many failures in this setting originate from incorrect observation actions, where the model fails to localize key regions or obtain crops that contain sufficient evidence for answering. We further observe that bounding-box tokens exhibit substantially higher generation entropy than ordinary language tokens, indicating high uncertainty when the model decides where to look. However, existing reinforcement learning methods based on final-answer rewards provide only trajectory-level feedback, making it difficult to assign precise credit to the observation actions. We propose Observation Policy Optimization (OPO), an RL algorithm for optimizing observation actions in tool-augmented thinking with images. OPO consists of two core modules. First, Uncertainty-Guided Observation Branching (UOB) treats each image zoom-in tool call as an observation event and selectively branches at high-uncertainty, low-redundancy events during rollout, enabling efficient exploration of alternative cropping decisions and their resulting visual evidence. Second, Evidence-Localized Advantage Attribution (ELAA) compares sibling branches using local evidence-sufficiency rewards and assigns credit only to the corresponding observation-action tokens. By decoupling observation-action optimization from trajectory-level outcome supervision, OPO improves visual evidence acquisition and further enhances visual reasoning performance. Experiments show that OPO consistently improves Qwen3-VL models across scales and achieves competitive performance against larger models and tool-augmented baselines. Code is publicly available.
• 中文摘要 复杂的视觉问答通常需要视觉语言模型通过图像来思考，即通过放大高分辨率图像的信息区域主动获取局部视觉证据。我们的错误分析显示，该场景中的许多失败源于错误的观察动作，即模型未能定位关键区域或未能获得包含足够证据的作物。我们还观察到，边界框令牌的生成熵明显高于普通语言令牌，表明模型在决定查找地点时存在高度不确定性。然而，基于最终答案奖励的现有强化学习方法仅提供轨迹级反馈，难以精确归功于观察行动。我们提出了观察策略优化（OPO），这是一种用于优化工具增强思维中观察行为的强化学习算法。OPO由两个核心模块组成。首先，不确定性引导观察分支（UOB）将每次图像放大工具调用视为观察事件，并在推广过程中选择性分支高不确定性、低冗余事件，从而高效探索替代裁剪决策及其视觉证据。其次，证据局部优势归因（ELAA）使用局部证据充分奖励比较兄弟分支，仅将功劳归属于相应的观察-行动标记。通过将观察-行动优化与轨迹级结果监督解耦，OPO提升了视觉证据的获取，并进一步提升了视觉推理性能。实验显示，OPO在不同尺度上持续改进Qwen3-VL模型，并在与更大模型和工具辅助基线竞争中取得竞争力。代码已公开。

Safeguarding Multi-Agent Reinforcement Learning for Connected and Automated Vehicles with Recursively Feasible Control Barrier Functions

保护具有递归可行控制屏障功能的联网和自动车辆的多智能体强化学习

• Authors: Jianye Xu, Yousef Sayed, Bassam Alrifaee
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.04957
• Pdf link: https://arxiv.org/pdf/2611.04957
• Abstract Multi-Agent Reinforcement Learning (MARL) shows promise in coordinating connected and automated vehicles but offers no safety guarantee. Although Control Barrier Functions (CBFs) can add this guarantee, their optimization problem can become infeasible in practice due to control bounds. Moreover, they often overapproximate vehicle and road geometry, which makes them conservative. This paper proposes the Backup-Horizon CBF (BH-CBF), which evaluates safety along a horizon until all vehicles stop. BH-CBF is recursively feasible under control bounds, and it considers the exact vehicle and road geometry. Based on BH-CBF, we derive a fully decentralized safety filter that only needs a single broadcast of vehicle states and routes per step and stays real-time in dense traffic. We use BH-CBF to safeguard a MARL policy over its life cycle: as a reward during pre-training, as a run-time filter at deployment, and as a filter in the training loop that keeps fine-tuning on unseen maps collision-free. In simulations on real-world maps of a roundabout and an intersection, BH-CBF avoids all collisions of unsafe MARL policies and keeps its safety guarantee at every step. Existing filters either collide, lose the guarantee, or fall back to full braking far more often. In recorded roundabout traffic with 1264 human-driven vehicles over 115 km, our safeguarded policies shorten the median travel time by 14% compared with the human drivers, without any collision.
• 中文摘要 多智能体强化学习（MARL）在协调联网和自动驾驶车辆方面展现出潜力，但不提供安全保障。尽管控制障碍函数（CBF）可以增加这一保障，但由于控制界限，其优化问题在实际中可能变得不可行。此外，它们常常过度近似车辆和道路几何，因此较为保守。本文提出了备份地平线CBF（BH-CBF），该方法在视野线上评估安全，直到所有车辆停下。BH-CBF在控制范围内递归可行，并考虑车辆和道路的具体几何形状。基于BH-CBF，我们推导出一个完全去中心化的安全过滤器，每步只需广播一次车辆状态和路线，且在密集交通中保持实时。我们使用 BH-CBF 来保障 MARL 政策的生命周期：作为预培训期间的奖励，部署时的运行时过滤器，以及训练循环中的过滤器，确保在未见地图上进行无碰撞微调。在环岛和交叉口的真实地图模拟中，BH-CBF 避免所有不安全 MARL 政策的碰撞，并在每一步都保持安全保障。现有过滤器要么碰撞、失去保证，要么更频繁地恢复全制动。在记录的环岛交通中，1264辆人类驾驶车辆行驶115公里，我们的受保护政策使中位数行驶时间比人类驾驶者缩短14%，且无碰撞。

LLMs as MDP Designers: A Dependency-Aware Agentic Framework for Automated Robot Policy Generation

作为MDP设计者的LLMs：一种依赖感知的智能体框架用于自动机器人策略生成

• Authors: Hong Je-Gal, Jinwoo Kim, Chanbin Yi, Hyun-Suk Lee
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.04974
• Pdf link: https://arxiv.org/pdf/2611.04974
• Abstract We investigate the automated generation of executable robot reinforcement learning (RL) policies from natural-language task descriptions. Rather than using large language models (LLMs) as execution-time decision makers, we use them as designers that construct RL pipelines before deployment, including Markov decision process (MDP) formulation and simulator implementation. We identify a phase-wise structure in this problem: MDP formulation is dependency-structured and sequential, whereas simulator implementation becomes modular once the formulation is fixed. Based on this structure, we propose ARPG, a dependency-aware agentic framework that performs sequential MDP modeling, localized verification and correction, and simulator-structured implementation while externalizing intermediate artifacts. ARPG targets a key failure mode of automated policy generation: executable code may still yield non-learnable policies when state, action, reward, and termination artifacts are not semantically aligned. Experiments in Isaac Sim show that ARPG improves modeling correctness, environment executability, and policy-generation success compared with monolithic LLM baselines and off-the-shelf agentic systems. Additional demonstrations on custom robotics, standard RL, and engineering-domain MDPs further suggest that the same modeling pipeline can be reused beyond the main benchmark suite.
• 中文摘要 我们研究了从自然语言任务描述中自动生成可执行机器人强化学习（RL）策略的方法。我们不使用大型语言模型（LLMs）作为执行时间决策者，而是将它们作为设计者，在部署前构建强化学习流水线，包括马尔可夫决策过程（MDP）表述和模拟器实现。我们识别出该问题的阶段结构：MDP表述是依赖结构化且顺序化的，而模拟器实现一旦表述固定后变为模块化。基于该结构，我们提出了依赖感知型代理框架，执行顺序MDP建模、局部验证与修正以及模拟器结构化实现，同时外部化中间产物。ARPG针对自动化策略生成的关键失败模式：当状态、动作、奖励和终止产物语义对齐时，可执行代码仍可能产生不可学习的策略。Isaac Sim 中的实验表明，与单体大型语言模型基线和现成代理系统相比，ARPG 在建模正确性、环境可执行性和策略生成成功率方面有提升。关于定制机器人、标准强化学习和工程领域 MDP 的进一步演示表明，同一建模流水线可以在主基准测试套件之外重复使用。

CRONOS: Benchmarking Non-Episodic Reinforcement Learning for Robotic Manipulation

CRONOS：机器人操作中非情节式强化学习的基准测试

• Authors: Po-Yi Wu, Djengo Cyun-Jyun Fang, Dian Cheng, Tsung-Wei Ke
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05080
• Pdf link: https://arxiv.org/pdf/2611.05080
• Abstract Reinforcement learning (RL) is promising for adapting robot policies to unstructured real-world environments. However, standard RL pipelines rely on episodic training with frequent scene resets, which is impractical for real-world deployment due to the need for substantial human intervention. While prior work has studied non-episodic RL in relatively small policies trained from scratch, its behavior when fine-tuning modern large-scale pre-trained vision-language-action (VLA) models remains underexplored. In this work, we introduce CRONOS, a simulation benchmark for RL fine-tuning of VLAs under non-episodic scene resets. CRONOS leverages high-fidelity physics simulators and adopts shared-scene multi-task settings. Using CRONOS, we show that naively fine-tuning pre-trained policies fails in non-episodic settings due to shifts in object configurations and end-effector poses; however, these challenges can be mitigated through targeted reset allocation and by addressing biases in pre-trained models. Moreover, we find that non-episodic training enhances long-horizon manipulation and improves generalization to previously unseen object configurations and task sequences. Project page: this https URL
• 中文摘要 强化学习（RL）在适应非结构化现实环境中的机器人策略方面具有前景。然而，标准的强化学习流水线依赖于情节训练，频繁场景重置，这在现实世界部署中不切实际，因为需要大量人类干预。虽然此前已有研究从零开始训练的相对小型策略中的非情节性强化学习，但其在微调现代大规模预训练视觉语言动作（VLA）模型时的行为仍未被充分探索。本研究介绍了CRONOS，这是一个用于在非场景重置下进行强化学习微调的模拟基准测试。CRONOS 利用高精度物理模拟器，并采用共享场景多任务设置。利用CRONOS，我们证明了在非情节环境中，简单微调预训练策略会因对象配置和端效器姿态的变化而失败;然而，这些挑战可以通过有针对性的重置分配和处理预训练模型中的偏差来缓解。此外，我们发现非情境训练增强了长期视野的操作能力，并提高了对此前未见对象配置和任务序列的泛化能力。项目页面：此 https URL

WayTrace: A Map-to-Language-to-Trace Hierarchy for Embodied Urban Navigation

WayTrace：一个用于具身城市导航的地图到语言再到路径的层级结构

• Authors: Manthan Patel, Jonas Frey, Pascal Roth, Marco Hutter, Cesar Cadena
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.05129
• Pdf link: https://arxiv.org/pdf/2611.05129
• Abstract Web services provide walking directions for nearly every city, but following them reliably over long routes remains challenging for autonomous robots. Point-goal navigation policies rely heavily on accurate GNSS, while most vision-language navigation models struggle to execute even short instructions outdoors. We present WayTrace, a hierarchical system that connects web-map directions to robot motion through language. An off-the-shelf orchestrator vision-language model reads the turn-by-turn directions, a satellite map and recent camera images, and issues one atomic language command at a time, such as "cross the road to the left". An onboard trace predictor, trained on over one million automatically labelled samples from street-level images, draws the path for this command in the image, and a reinforcement learning policy tracks it. In long-range simulated missions, WayTrace remains robust to GNSS noise and rarely leaves crosswalks or sidewalks, while point-goal baselines degrade sharply with about ten times more violations. Deployed on a wheeled-legged robot, WayTrace completes real-world missions of up to 1.37 km at about 1.1 m/s through a busy city centre, rural trails and a university campus, handling crosswalks, staircases and construction sites. Project page: this https URL
• 中文摘要 网络服务几乎为每个城市提供步行路线，但自主机器人在长途路线上可靠地遵循路线仍然具有挑战性。点目标导航策略高度依赖准确的GNSS，而大多数视觉语言导航模型甚至难以在户外执行短指令。我们介绍WayTrace，这是一个通过语言将网络地图方向与机器人运动连接起来的分层系统。现成的编排视觉语言模型读取逐条路线、卫星地图和近期摄像头图像，并逐条发出原子语言命令，如“向左过马路”。车载追踪预测器基于超过一百万条自动标记的街道级图像样本训练，绘制该命令的路径，并由强化学习策略追踪。在远程模拟任务中，WayTrace对GNSS噪声保持强劲，很少离开斑马线或人行道，而点目标基线则急剧下降，违规次数约多十倍。WayTrace部署在带轮腿机器人上，能以约1.1米/秒的速度完成最长1.37公里的真实任务，穿越繁忙的市中心、乡村小径和大学校园，处理斑马线、楼梯和施工现场。项目页面：此 https URL

Catalyzing Interactive HTML Generation Capabilities via High-Fidelity Reinforcement Learning

通过高保真强化学习催化交互式HTML生成能力

• Authors: Xujia Wang, Yixuan Liu, Shaohan Huang, Furu Wei, Lei Hou, Bin Xu, Juanzi Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.05151
• Pdf link: https://arxiv.org/pdf/2611.05151
• Abstract Generating HTML from natural-language specifications is an important application of code large language models (LLMs). Despite producing renderable pages, current models often fail to ensure functionality and visual quality during actual interaction. Existing methods typically target task-specific webpage supervision, leaving demanding interactive artifact optimization underexplored. Using mini-games as a challenging testbed, we expose limitations in interaction experience, reward signal, and credit assignment fidelity that are obscured by shallow webpage interactions. We introduce WebCatalyst, a browser-interaction reinforcement learning framework with three complementary components: a game-level GUI assessor for sustained interaction and evaluation, Visual Anchor for mitigating VLM reward hacking, and Action-Trace Credit for grounding token-level credit assignment in runtime execution traces. Trained on fewer than 1K requests, predominantly for games, WebCatalyst raises the Cookie-Bench Mini Games score from 68.8 to 75.3 and also the WebGen-Bench accuracy from 48.8 to 56.9 on Qwen3.6-35B-A3B, outperforming Claude Opus 4.7 in average rank across evaluated metrics. Further analysis shows that Visual Anchor prevents aesthetic degradation, while runtime-grounded credit assignment accelerates functional improvement. Together, our results show that improving feedback fidelity is the key to substantially advancing interactive HTML generation.
• 中文摘要 从自然语言规范生成HTML是代码大型语言模型（LLM）的重要应用。尽管现有模型能生成可渲染页面，但它们常常无法在实际交互中保证功能和视觉质量。现有方法通常针对任务特定的网页监督，导致要求高的交互式工件优化尚未被充分探索。我们以小游戏作为挑战性测试平台，揭示了交互体验、奖励信号和信用分配忠实度的限制，这些都被浅层网页交互所掩盖。我们介绍了WebCatalyst，一个浏览器交互强化学习框架，包含三个互补组件：用于持续互动和评估的游戏级图形界面评估器、用于缓解VLM奖励黑客的Visual Anchor，以及用于将代币级信用分配建立在运行时执行痕迹中的Action-Trace Credit。WebCatalyst在少于1000个请求中训练，主要针对游戏，将Cookie-Bench迷你游戏得分从68.8提升至75.3，并将Qwen3.6-35B-A3B的WebGen-Bench准确率从48.8提升至56.9，在评估指标的平均排名中优于Claude Opus 4.7。进一步分析显示，Visual Anchor防止了美观退化，而基于运行时的学分分配加速了功能提升。综合结果显示，提升反馈忠实度是显著推进交互式HTML生成的关键。

JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks

JHiMA-WebRL：一个面向长期网络任务的联合分层多智能体强化学习框架

• Authors: Shanka Subhra Mondal, Jiayun Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.05263
• Pdf link: https://arxiv.org/pdf/2611.05263
• Abstract Multimodal web agents have shown promising capabilities for interacting with websites through graphical user interfaces (GUIs), yet realistic web tasks often require long-horizon, multi-step planning that remains challenging for monolithic policies. Such agents must jointly infer high-level plans, navigate through intermediate states, track progress over extended interaction histories, and execute visually grounded actions. Although prior work has explored modular multi-agent systems with a separate planner, the planner is typically not updated jointly with the actor. We investigate whether explicitly separating and jointly training planning and execution provides a stronger inductive bias for long-horizon web navigation. We introduce JHiMA-WebRL, a joint hierarchical multi-agent reinforcement learning framework for web agents that couples a task decomposer with an actor. To train the decomposer, we use a strong teacher model to construct structured subtask supervision from multimodal GUI trajectories, and use these annotations to fine-tune both the decomposer and the subtask-conditioned actor. We then extend Group Relative Policy Optimization (GRPO) to jointly optimize the decomposer and actor in the interactive WebArena environment through hierarchical rollout grouping and role-specific advantages. On WebArena-Lite-v2, JHiMA-WebRL-14B, consisting of a 7B actor and a 7B decomposer, achieves an average success rate of 37.2%, compared with 24% for a ScaleCUA-32B agent that we fine-tune with RL on the same training tasks and evaluate under the same protocol. Ablations show that jointly updating the decomposer during RL outperforms updating the actor alone. These results suggest that jointly learned role factorization with smaller models can be a competitive alternative to scaling monolithic agents for long-horizon web tasks.
• 中文摘要 多模态网络代理已展现出通过图形用户界面（GUI）与网站交互的有前景能力，但现实中的网络任务通常需要长期、多步规划，这对单一策略来说仍具挑战性。此类代理必须共同推断高层计划，穿越中间状态，跟踪扩展交互历史的进展，并执行视觉化的动作。尽管此前已有研究探索过带有独立规划器的模块化多智能体系统，但规划器通常不会与参与者联合更新。我们研究显式分离并联合训练规划与执行是否能为长视野网页导航提供更强的归纳偏向。我们介绍JHiMA-WebRL，这是一个联合层级多代理强化学习框架，将任务分解器与演员结合。为了训练分解器，我们使用强教师模型，从多模态图形界面轨迹构建结构化子任务监督，并利用这些注释对分解器和子任务条件演员进行微调。随后，我们扩展了组相对策略优化（GRPO），通过层级展开分组和角色特定优势，共同优化交互式WebArena环境中的分解器和演员。在WebArena-Lite-v2上，JHiMA-WebRL-14B（由一个7B演员和一个7B分解器组成）平均成功率为37.2%，而我们在同一训练任务下用强化学习微调并评估的ScaleCUA-32B代理为24%。消融结果显示，在强化学习期间联合更新分解器优于单独更新演员。这些结果表明，联合学习的角色因数与较小模型相比，可以成为长期网络任务中单一智能体扩展的竞争替代方案。

iGC: Antibody Optimization via Reinforcement Learning over a Learned Somatic Hypermutation Prior

iGC：通过强化学习优化抗体，超越已学习的体细胞超突变先验

• Authors: Thao Nguyen, Robert G. Alberstein, Ewa Nowara, Homa MohammadiPeyhani, Holly R. Steach, Sarah A. Robinson, Yuval Shechter, Donald W. Lee, Jason A. Vander Heiden, Franziska Seeger, Andrew Leaver-Fay, Richard Bonneau, Joshua Yao-Yu Lin
• Subjects: Subjects: Machine Learning (cs.LG); Biomolecules (q-bio.BM)
• Arxiv link: https://arxiv.org/abs/2611.05270
• Pdf link: https://arxiv.org/pdf/2611.05270
• Abstract Antibody affinity maturation is a natural optimization process in which B cells acquire mutations through somatic hypermutation (SHM), undergo selection for improved antigen binding, and iteratively evolve toward higher affinity. Computationally modeling this process requires capturing both biologically plausible mutations and antigen-specific selection. However, existing mutation models do not directly optimize binding, while affinity optimization often neglects realistic mutation trajectories. We introduce iGC (in silico Germinal Centers), a two-stage reinforcement learning (RL) framework for antigen-specific antibody optimization. Stage 1 learns an SHM mutation prior from parent-child pairs in B-cell lineage trees and adapts it to a target affinity-maturation dataset. Stage 2 initializes an RL policy from this prior and optimizes antigen binding with Kullback-Leibler (KL) regularization toward the frozen prior. To prevent reward exploitation, including repetitive high-reward substitutions and structurally tolerated framework mutations, we introduce a Reward-Tax Meta-Gradient objective. RewardNet smooths the affinity landscape, while TaxNet penalizes reward overexploitation, jointly defining a learned meta-reward for policy optimization. We evaluate iGC on mutation prediction, SHM consistency, affinity optimization, and mutation diversity. The mutation prior more than doubles the strongest baseline in mutation-site prediction. Following RL optimization, iGC maintains SHM sequence structure. It also achieves a strong exploration-exploitation balance, yielding competitive reward-diversity products across candidate-selection cutoffs. These results demonstrate that iGC integrates biologically plausible mutation generation with antigen-specific affinity selection, providing an in silico framework for iterative antibody affinity maturation.
• 中文摘要 抗体亲和力成熟是一个自然优化过程，B细胞通过体细胞高突变（SHM）获得突变，经过选择以提升抗原结合，并迭代进化至更高的亲和力。计算建模该过程需要同时捕捉生物学上合理的突变和抗原特异性选择。然而，现有突变模型并未直接优化结合，亲和力优化常常忽视现实中的突变轨迹。我们引入iGC（计算机生成中心），这是一种两阶段强化学习（RL）框架，用于抗原特异性抗体优化。第一阶段从B细胞谱系树中的亲子对中学习SHM突变，并将其适配到目标亲和力成熟数据集。第二阶段从该先验初始化强化策略，并利用Kullback-Leibler（KL）正则化优化抗原结合，朝向冻结先验。为防止奖励利用，包括重复的高奖励替换和结构容忍框架突变，我们引入了奖励税元梯度目标。RewardNet平滑亲和力景观，而TaxNet惩罚奖励过度利用，共同定义了策略优化的学习元奖励。我们评估iGC在突变预测、SHM一致性、亲和力优化和突变多样性方面的表现。突变前置于突变位点预测中最强基线的两倍多。经过强化学习优化后，iGC保持SHM序列结构。它还实现了强的探索-利用平衡，在候选选择截止点之间产生具有竞争力的奖励多样性产物。这些结果表明，iGC将生物学上合理的突变生成与抗原特异性亲和力选择相结合，提供了一个计算机模拟的抗体亲和力成熟框架。

GSBR: Geometric Support Behavior Regularization for Online RL

GSBR：在线强化语言的几何支撑行为正则化

• Authors: Charles Xu, Jacob Berg, Liyiming Ke, Sergey Levine
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05295
• Pdf link: https://arxiv.org/pdf/2611.05295
• Abstract Regularizing online reinforcement learning (RL) toward a pretrained base policy can improve sample efficiency and learning stability, but regularization that is too restrictive limits how much the policy can improve. Base policies -- typically diffusion or flow-matching models trained by imitation learning on large, diverse datasets -- capture many behaviors that are reasonable but suboptimal. Existing regularizers using such base policies are too restrictive in one of two ways: behavioral cloning regularizers pull the actor toward the full base-policy distribution, penalizing it for discarding suboptimal modes, while hard support constraints forbid improvements that lie just outside the base policy's support. We propose Geometric Support Behavior Regularization (GSBR), which penalizes actions by their distance from the base policy's support. Actions within the support incur no penalty, so the actor can concentrate on high-value modes; outside the support, where value estimates are less reliable, the penalty grows smoothly with distance while still permitting improvement when the value gained outweighs the penalty. GSBR estimates this distance from a finite set of base-policy samples, so it requires only sample access to the base policy. Across 19 tasks on four robotic benchmarks, GSBR raises the average success rate from 33% for the base policy to 95%, achieving the highest average success rate and throughput among all compared methods, with about 23% higher normalized throughput than the next-best RL method. Project website and open-source code at this https URL .
• 中文摘要 将在线强化学习（RL）正则化到预训练的基础策略可以提高样本效率和学习稳定性，但过于严格的正则化会限制策略的改进幅度。基础策略——通常是通过模仿学习在大型多样数据集上训练的扩散或流匹配模型——捕捉了许多合理但次优的行为。现有使用此类基础策略的正则化器在两种方面过于限制：行为克隆正则化器将行为者拉向完整的基础策略分布，惩罚其丢弃次优模式;而硬支持约束则禁止在基础策略支持之外进行改进。我们提出几何支持行为正则化（GSBR），该方法惩罚动作与基础策略支持的距离。支持内的动作不会产生惩罚，因此行为者可以专注于高价值模式;在支持之外，价值估计不够可靠，惩罚随距离平滑增长，同时当获得的价值超过惩罚时仍允许改进。GSBR从有限的基础策略样本中估算该距离，因此只需样本访问基础策略。在四个机器人基准测试的19个任务中，GSBR将基础策略的平均成功率从33%提升至95%，在所有比较方法中达到最高的平均成功率和吞吐量，比次优强化学习方法高约23%的标准化吞吐量。项目网站及开源代码见此 https URL。

MDSKILL: Learning to Internalize Skill Memory through Multi-View Distillation for Efficient Dynamic Agent Self-Evolution

MDSKILL：通过多视角提炼学习内化技能记忆，实现高效的动态代理自我进化

• Authors: Dian Jin, Jinluan Yang, Kairong Han, Baohong Li, Zijing Hu, Yi He, Wenlin Zhong, Yunze Tong, Minqin Zhu, Linchuan Li, Yanan Niu, Kun Kuang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.05313
• Pdf link: https://arxiv.org/pdf/2611.05313
• Abstract Skill memories encode past experiences as textual prompts to improve reinforcement learning (RL) for self-evolving agents. Complementing RL, On-Policy Self-Distillation (OPSD) provides dense, fine-grained supervision to internalize knowledge from skill-guided trajectories and accelerate policy learning. However, self-generated memories often contain noisy supervision and functionally entangled, redundant, or conflicting knowledge, making direct distillation difficult. Under limited training budgets, these issues can hinder knowledge internalization and even degrade performance. To address this problem, we propose MDSKILL, a multi-view distillation approach for skill memory internalization. MDSKILL organizes skill memories into three complementary functional views (state grounding, action planning, and execution control) and selects high-quality skills using dynamic metrics to construct view-specific and integrated reasoning branches. OPSD then transfers and consolidates complementary experiential knowledge across these branches into a shared policy, enabling efficient skill internalization and continuous agent self-evolution. We evaluate MDSKILL on three agent benchmarks against RL, skill-augmented, and self-distillation baselines that do not rely on stronger external teacher models. Without additional computational overhead, MDSKILL improves upon SOTA performance by up to 7.1% and 8.1% on ALFWorld and WebShop. Extensive ablations and evolution analyses further validate its effectiveness in knowledge internalization and policy improvement. The code is available at this https URL.
• 中文摘要 技能记忆将过去的经验编码为文本提示，以改善自我进化智能体的强化学习（RL）。作为强化学习的补充，策略上自我蒸馏（OPSD）提供密集、细粒度的监督，内化技能引导轨迹的知识，加速策略学习。然而，自生成记忆通常包含噪声监督和功能纠缠、冗余或冲突的知识，使直接提炼变得困难。在有限的培训预算下，这些问题可能阻碍知识内化，甚至降低性能。为解决这一问题，我们提出了MDSKILL，一种多视角的技能记忆内化提炼方法。MDSKILL将技能记忆组织为三种互补的功能视图（状态基础、行动规划和执行控制），并利用动态度量选择高质量技能，构建视角特定且集成的推理分支。OPSD随后将这些分支间互补的经验知识转移并整合成共享策略，实现高效的技能内化和持续的代理自我演进。我们基于三种代理基准测试，基于强化学习、技能增强和自我蒸馏基线，这些基准不依赖更强的外部教师模型。在不增加额外计算开销的情况下，MDSKILL在ALFWorld和WebShop上比SOTA性能提升了最多7.1%，分别提升8.1%。通过广泛的消融和演化分析，进一步验证了其在知识内化和策略改进方面的有效性。代码可在此 https URL 获取。

Suppressing Misaligned Behaviors in Language Models, Durably

持久地抑制语言模型中的错位行为

• Authors: Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han, Danqi Chen
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05358
• Pdf link: https://arxiv.org/pdf/2611.05358
• Abstract Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT. Together, our framework and method offer a step toward more durable post-hoc removal of undesired behaviors.
• 中文摘要 不希望的模型行为可以在训练的任何阶段出现或重新出现。一旦观察到，我们能多有效地抑制它们？为了研究这个问题，我们首先构建了两个模型生物——经过刻意训练表现出特定错位的语言模型——它们在触发行为的因素上有所不同。狭触发生物只有在提示包含特定提示时才会表现出异常：它在多项选择题中作弊，跟随指向错误答案的植入提示，否则表现正常。宽触发生物则没有这样的提示：在对不安全代码进行微调后，它会在广泛的开放式提示上给出有害的反应。然后，我们正式化事后行为抑制，并根据三个标准评估方法：即时抑制目标行为、在进一步训练中的耐久性以及保持一般能力。我们比较了标准方法——SFT、GRPO和梯度上升逆学习——在这些维度上。基于先前对比对浅层的研究，我们假设更持久的抑制需要从更广泛的错位状态中恢复的能力。我们引入了前缀扩展对抗强化学习（PERL），这是一种GRPO变体，通过从缓存生物轨迹采样的中间状态延续来增强推广。在两种生物体环境和两个模型家族（Qwen3-4B，GPT-OSS-20B）中，PEARL在保持任务准确性的同时实现了强有力的即时抑制效果，并且在对生物体数据或无关数据的进一步微调下依然持久。当不良行为范围广泛且无法通过定向SFT解决时，PEARL获得的最大益处。我们的框架和方法共同为更持久的事后消除不良行为迈出了一步。

PolicyIK: High-Precision Inverse Kinematics for Robot Arms with Reinforcement Learning

PolicyIK：机器人手臂的高精度逆运动学与强化学习

• Authors: Yuhan Liu, Xinqi Liu, Zixuan Dong, Keith W. Ross, Shenji Wan
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05372
• Pdf link: https://arxiv.org/pdf/2611.05372
• Abstract Inverse kinematics (IK) converts end-effector pose commands into joint motion, and many vision-language-action (VLA) policies rely on it to act on a given arm and to transfer across embodiments. Precise IK remains laborious: redundancy and singularities complicate the inverse mapping, execution depends on hand-tuned controllers, and training data on physical robots are costly to collect. We present PolicyIK, a goal-conditioned reinforcement learning policy for high-precision IK. It updates joint-position references from pose feedback and observes its accumulated reference relative to the measured joints, so it can retain the offset that the position controller needs at rest. Two-scale pose errors with an error-dependent action gain resolve millimeter residuals while preserving large motions. Progress and dwell rewards teach it to both reach the tolerance and stay within it. In simulation, a single attempt by the trained PPO policy keeps the mean errors over the final second within 1 mm and 0.2° on 93.5% of held-out tasks, and a deployment protocol raises success to 98.9%. When trained directly on a physical arm with an off-policy RL learner, PolicyIK achieves higher reaching and holding success than all four classical IK controllers under the same time budgets. In simulated pick-and-place, PolicyIK executes the end-effector commands of the $\pi_{0.5}$ VLA policy with 95.7% success. These results show that reinforcement learning can provide high-precision inverse kinematics for robot arms. Code is available at this https URL.
• 中文摘要 逆运动学（IK）将末端执行器姿态指令转换为关节运动，许多视觉-语言-动作（VLA）策略依赖其作用于特定臂并跨身体传递。精确的IK仍然繁琐：冗余和奇点使逆映射复杂，执行依赖手工调校控制器，且物理机器人训练数据收集成本高昂。我们介绍PolicyIK，一种针对高精度IK的目标条件强化学习策略。它通过姿态反馈更新关节位置参考，并观察相对于测量关节的累计参考，从而保持位置控制器静止时所需的偏移量。带有误差依赖动作增益的双尺度姿态误差解析毫米残差，同时保持大运动。进展和停留奖励教会它既达到容忍范围，又保持在容差内。在模拟中，训练好的PPO策略一次尝试，在93.5%的拖延任务中，最后一秒的平均误差保持在1毫米和0.2°之间，部署协议的成功率可提升至98.9%。当直接在物理臂上训练并使用非策略的强化学习器时，PolicyIK在相同时间预算下，达到的伸程和保持成功率优于四种经典IK控制器。在模拟选位中，PolicyIK以95.7%的成功率执行$\pi_{0.5}$ VLA策略的末端执行器命令。这些结果表明强化学习能够为机器人手臂提供高精度的逆运动学。代码可在此 https URL 获取。

Keyword: diffusion policy

Spectrally Routed Diffusion Policy for Multi-Task Robotic Manipulation

多任务机器人操作的光谱路由扩散策略

• Authors: Mingcong Li, Yihan Zhou, Jiayu Qu, Xueyang Guo, Yilin Mo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.04477
• Pdf link: https://arxiv.org/pdf/2611.04477
• Abstract Diffusion-based policies have achieved strong performance on single-task robotic manipulation. However, a single model trained jointly on multiple tasks consistently underperforms the per-task specialists trained on the same demonstrations. We focus on two challenges. First, the convolutional encoders commonly used in diffusion-based policies produce visual conditions that are not discriminative enough. Second, gradients from incompatible action modes update the shared denoiser's dense parameters along the same directions, causing destructive interference during joint training. To address both, we propose SR-DP (Spectrally Routed Diffusion Policy). A frozen VGGT encoder with a spatial-compression adapter produces compact multi-scale semantic-geometric tokens, a task-discriminative visual condition for tasks that share a workspace. We first train the denoiser as a standard diffusion policy, then decompose selected linear layers by SVD into a frozen basis of orthogonal rank-one experts, adapted through spectral routing and Householder input rotations. On six RoboTwin tasks, SR-DP averages a 68.4% success rate, surpassing multi-task VO-DP by 21.8 points and the strongest per-task baseline by 3.8 points. On four real-world bimanual tasks, Stage 2 reaches a 43.0% average success rate and improves over Stage 1 by 19.0 points.
• 中文摘要 基于扩散的策略在单任务机器人操作方面取得了优异表现。然而，一个在多个任务上联合训练的单一模型，始终表现得比在同一演示上训练的每任务专家表现不佳。我们关注两个挑战。首先，基于扩散策略常用的卷积编码器产生的视觉条件判别力不足。其次，来自不兼容动作模式的梯度会沿相同方向更新共享去噪器的密集参数，导致联合训练时的破坏性干扰。为解决这两者，我们提出了SR-DP（频谱路由扩散策略）。一个带空间压缩适配器的冻结VGGT编码器生成紧凑的多尺度语义几何标记，这是一种任务判别视觉条件，适用于共享工作空间的任务。我们首先将去噪器训练为标准扩散策略，然后通过 SVD 将选定的线性层分解为一个由正交一级专家组成的冻结基，并通过谱路由和 Householder 输入旋转进行调整。在六个机器人双生任务中，SR-DP 平均成功率为 68.4%，比多任务 VO-DP 高出 21.8 分，且是单任务最强基线高出 3.8 分。在四个真实双手任务中，第二阶段的平均成功率达到 43.0%，较第一阶段提升 19.0 分。

Physical Diffusion Policy via MPM-Guided 3D Gaussian Simulation

通过MPM引导的3D高斯模拟实现物理扩散政策

• Authors: Can Wang, Qiyuan Qiao, Lei Liu, Ge Yuan, Xinhui Liu, Dong Xu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.04493
• Pdf link: https://arxiv.org/pdf/2611.04493
• Abstract Learning robust manipulation policies for diverse everyday objects remains challenging because objects with different materials exhibit distinct deformation and contact dynamics. While diffusion policies provide a powerful framework for modeling multimodal manipulation behaviors, they typically lack explicit physical reasoning and may produce averaged actions when facing heterogeneous objects. We present a physics-aware diffusion policy that augments visual imitation learning with object physical understanding. Our framework recovers material properties from visual demonstrations by coupling 3D Gaussian Splatting with differentiable Material Point Method simulation, where video guidance and particle-level constraints enable accurate physical parameter estimation. The estimated physics conditions are further integrated into a material-aware Mixture-of-Experts diffusion policy, allowing adaptive behavior generation across different physical regimes. For practical deployment, we introduce a physics memory bank that provides efficient physical retrieval without requiring online simulation optimization. Experiments on a wide range of simulated and real-world objects demonstrate improved manipulation success and generalization over existing diffusion-based approaches. These results show that incorporating physical knowledge is critical for scalable robot manipulation in diverse environments.
• 中文摘要 学习针对不同日常物体的稳健操作策略仍具挑战性，因为不同材料的物体表现出明显的变形和接触动力学。虽然扩散策略为建模多模态操作行为提供了强大框架，但通常缺乏显式物理推理，面对异构物体时可能产生平均动作。我们提出了一种物理感知扩散策略，通过物体物理理解增强视觉模仿学习。我们的框架通过将3D高斯喷溅与可微分的材料点方法模拟结合，从视觉演示中恢复材料性质，视频引导和粒子级约束实现了准确的物理参数估计。估计的物理条件进一步集成到材料感知的专家混合扩散策略中，允许跨不同物理区间的自适应行为生成。为实际部署，我们引入了物理记忆库，提供高效的物理检索，无需在线模拟优化。在各种模拟和现实世界物体上的实验显示，操作成功率和泛化能力优于现有扩散方法。这些结果表明，融入物理知识对于在多样化环境中实现可扩展的机器人操作至关重要。