生成时间: 2026-11-03 23:58:52 (UTC+8); Arxiv 发布时间: 2026-11-03 20:00 EST (2026-11-04 09:00 UTC+8)

今天共有 88 篇相关文章

Keyword: reinforcement learning

OCRia-1 Technical Report

OCRia-1技术报告

• Authors: Aman Rangapur, Antonio Jimeno Yepes, Daniel Bustamante, Nutchapol Dendumrongsup, Sebastián G. Acosta, Pierre-Loïc Doulcet, Simon Suo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.00092
• Pdf link: https://arxiv.org/pdf/2611.00092
• Abstract We introduce OCRia-1, a 4B vision-language model (VLM) for document parsing that excels at preserving table structure, extracting chart data and retaining text formatting. OCRia-1 handles content that has traditionally been challenging in document parsing: merged cells and spanning headers, tables with thousands of cells, chart data, and meaningful text formatting such as bold, italics, subscripts and superscripts, preserving details needed for business-critical document analysis. At the time of this report, it achieves the highest scores on tables, charts and text formatting among the systems evaluated, including a 35B model and Mistral OCR 4.1. Its mean ParseBench score over the four dimensions we evaluate is 0.87, against 0.75 for the next-best system. We emphasize data quality and reward design in training OCRia-1. Synthetic pages provide exact ground truth for long tables and rich formatting, while real pages are transcribed using a pipeline grounded in the PDF text layer. We attribute OCRia-1's performance primarily to careful data curation that prioritizes quality and diversity over quantity. Dimension-specific rewards evaluate table reconstruction, chart-data extraction and text-formatting fidelity. We train the model on the curated data using supervised fine-tuning (SFT), then refine it through reinforcement learning with verifiable rewards (RLVR). To our knowledge, OCRia-1 is the fastest published document-parsing VLM in its class, decoding at about 4,700 tokens per second with a single request on one B200 GPU, about nine times faster than the base model on stock vLLM. This is achieved through a block-diffusion speculator, quantization and custom kernels.
• 中文摘要 我们介绍了OCRia-1，一种4B的视觉语言模型（VLM），用于文档解析，擅长保持表格结构、提取图表数据和保留文本格式。OCRia-1处理了文档解析中传统上较为困难的内容：合并单元格和生成头部、数千个单元格的表格、图表数据，以及有意义的文本格式（如加粗、斜体、下标和上标），保留了业务关键文档分析所需的细节。截至本报告发布时，它在所有评估系统中，包括35B模型和Mistral OCR 4.1，在表格、图表和文本格式方面取得了最高得分。其在我们评估的四个维度上的平均ParseBench得分为0.87，而次优系统为0.75。我们在训练OCRia-1时强调数据质量和奖励设计。合成页面为长表和丰富格式提供精确的真实数据，而真实页面则通过基于PDF文本层的流水线进行转录。我们将OCRia-1的性能归因于细致的数据策划，优先考虑质量和多样性而非数量。维度特定奖励评估表重建、图表数据提取和文本格式化的真实度。我们利用监督微调（SFT）训练模型，然后通过可验证奖励的强化学习（RLVR）进行细化。据我们所知，OCRia-1是同类中发布的文档解析VLM最快的，单次请求在一块B200 GPU上解码约4700个令牌每秒，约是原生vLLM基础模型的九倍。这通过块扩散推测器、量化和自定义核实现。

DTP-OPSD: Reinforcement Learning with Distractor-Aware Temporal-Privilege On-Policy Self-Distillation for Temporal Video Grounding

DTP-OPSD：基于分心感知的时间特权政策自我提炼的强化学习，实现时间视频基础

• Authors: Zhaoyu Chen, Yongwei Nie
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.00120
• Pdf link: https://arxiv.org/pdf/2611.00120
• Abstract Temporal Video Grounding (TVG) aims to localize the start and end times of an event described by a language query in a video. The task becomes particularly challenging when the same video contains multiple repeated, similar, or opposite events. On these hard examples, all responses sampled by Group Relative Policy Optimization (GRPO) may miss the target and receive zero temporal Intersection-over-Union (tIoU). Their group-relative advantages are then all zero, leaving no useful policy gradient signal. To address this, we propose DTP-OPSD, a reinforcement learning framework with distractor-aware temporal-privilege on-policy self-distillation. The student observes the clean video and generates responses on-policy, while a shared-weight teacher observes a privileged video in which the target window is marked by red full-frame borders and same-video distractor windows by blue full-frame borders. For the distillation loss, the teacher evaluates student-generated prefixes and provides dense token-level Jensen-Shannon divergence supervision. This supervision remains informative even when GRPO provides no policy gradient signal. We suppress distillation when the teacher does not outperform the student and strengthen it for groups where the teacher performs better but all student rollouts receive zero tIoU. We also construct a compact hard-interference training set through VLM screening and human verification. With only 300 additional post-training examples, DTP-OPSD achieves the highest mIoU among the compared open-source 8B models on Charades-TimeLens, ActivityNet-TimeLens, and QVHighlights-TimeLens, reaching 55.93, 54.77, and 66.71, respectively. Compared with Pure GRPO trained on the same data, DTP-OPSD improves all 12 reported metrics, with [email protected] gains of 1.28, 1.62, and 1.78 points on the three benchmarks. Code will be released at this https URL
• 中文摘要 时间视频基础化（TVG）旨在局部化视频中语言查询描述事件的开始和结束时间。当同一视频包含多个重复、相似或相反事件时，任务尤为具有挑战性。在这些困难的例子中，群相对策略优化（GRPO）抽样的所有响应都可能未达到目标，且时间交叉与并集（tIoU）为零。它们的群体相对优势全部为零，没有有用的策略梯度信号。为此，我们提出了DTP-OPSD，一种强化学习框架，具有分心感知的时间特权策略自我蒸馏。学生观察干净的视频并生成策略上的回应，而共享权重教师观察一个特权视频，目标窗口用红色全帧边框标记，同视频干扰窗口用蓝色全帧边框标记。对于蒸馏损失，教师评估学生生成的前缀，并提供密集的令牌级Jensen-Shannon发散监督。即使GRPO没有提供策略梯度信号，这种监督依然具有信息量。当教师表现不超过学生时，我们抑制蒸馏，对教师表现更好但所有学生推广都获得零tIoU的组加强蒸馏。我们还通过VLM筛选和人工验证构建了一个紧凑的硬干扰训练集。仅有300个额外训练后样本，DTP-OPSD在Charades-TimeLens、ActivityNet-TimeLens和QVHighlights-TimeLens上的开源8B模型中分别达到最高的mIoU，分别达到55.93、54.77和66.71。与基于相同数据训练的纯GRPO相比，DTP-OPSD在所有12项报告指标上均有所提升，三个基准测试分别提升了1.28、1.62和1.78 [email protected]点。代码将在此https URL发布

FB-MOAC: A Reinforcement Learning Algorithm for Forward-Backward Markov Decision Processes

FB-MOAC：一种用于正向-后向马尔可夫决策过程的强化学习算法

• Authors: Mohsen Amidzadeh, Mario Di Francesco
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.00130
• Pdf link: https://arxiv.org/pdf/2611.00130
• Abstract Reinforcement learning (RL) algorithms are effective in solving problems that can be modeled as Markov decision processes (MDPs). These algorithms primarily target forward MDPs whose dynamics evolve over time from an initial state. However, several important problems in different scenarios including stochastic control and network systems exhibit both a forward and a backward dynamics. As a consequence, they cannot be expressed as a standard MDP, thereby calling for a novel theory for RL in this context. Accordingly, this work introduces the concept of Forward-Backward Markov Decision Processes (FB-MDPs) for multi-objective problems, develops a novel theoretical framework to characterize their optimal solutions, and proposes a general forward-backward step-wise template that allows to adapt RL algorithms for FB-MDP problems. A Forward Backward Multi Objective Actor Critic (FB-MOAC) algorithm is introduced accordingly to obtain optimal policies with guaranteed convergence and a competitive rate with respect to standard approaches in RL. FB-MOAC is evaluated on diverse use cases in the context of mathematical finance and mobile resource management. The obtained results show that FB-MOAC outperforms the state of the art across different metrics, highlighting its ability to learn and maximize rewards.
• 中文摘要 强化学习（RL）算法在解决可建模为马尔可夫决策过程（MDP）的问题时非常有效。这些算法主要针对其动态从初始状态随时间演变的前向MDP。然而，在不同场景中，包括随机控制和网络系统，存在若干重要问题同时表现出正向和后向动力学。因此，它们无法作为标准MDP来表达，因此在该语境下需要一个新的强化学习理论。因此，本研究引入了多目标问题的正向后向马尔可夫决策过程（FB-MDP）概念，发展了一个新的理论框架来刻画其最优解，并提出了一个通用的前向-后退分步模板，允许将强化学习算法适配于FB-MDP问题。随后引入了前向后向多目标演员批评（FB-MOAC）算法，以获得最优策略，保证收敛并在强化学习中具有竞争力的效率。FB-MOAC在数学金融和移动资源管理背景下对多种应用场景进行了评估。结果显示，FB-MOAC在多个指标上超越了最先进技术，凸显了其学习和最大化奖励的能力。

Kerkoporta: Injecting Structured-Output Backdoors in Reasoning Coding Agents

Kerkoporta：在推理编码代理中注入结构化输出后门

• Authors: Georgios Syros, Blake Bullwinkel, Alina Oprea, Cristina Nita-Rotaru, Amanda Minnich, Xiaodong Liu, Giorgio Severi
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.00138
• Pdf link: https://arxiv.org/pdf/2611.00138
• Abstract Existing large language model (LLM) backdoor attacks operate primarily in chat settings, where the risk is bounded by free-form text outputs. AI agents remove that bound by performing actions and executing commands on the user's machine. We introduce structured-output backdoor attacks against reasoning models in agentic coding settings. A trigger placed anywhere in the agent's context causes the model to emit an attacker-chosen tool call, which the harness executes, directly compromising the system's confidentiality, integrity, or availability. We present Kerkoporta, a methodology that constructs high-quality poison samples, injects the backdoor via supervised fine-tuning, and recovers utility loss with lightweight reinforcement learning. We evaluate on modern, heavily post-trained open-weight models across proprietary and open-source harnesses, focusing on coding for its wide adoption and long-horizon task structure. The attack reaches success rates of up to 96.2% and can be configured to generalize across harnesses or bind to a single target. The resulting models can outperform the clean baseline on standard utility benchmarks, giving victims a direct incentive to adopt them. Overall, we demonstrate that LLM backdoors can have critical security impacts in the context of modern coding agents, and we characterize the poisoning methodology required to insert backdoors that are both effective and stealthy in this setting.
• 中文摘要 现有的大型语言模型（LLM）后门攻击主要在聊天环境中运行，风险受自由文本输出限制。人工智能代理通过在用户机器上执行动作和命令来解除这一限制。我们在智能编码环境中引入了针对推理模型的结构化输出后门攻击。在代理上下文中任意设置触发器，会使模型发出攻击者选择的工具调用，黑客执行该调用，直接威胁系统的机密性、完整性或可用性。我们介绍Kerkoporta方法，该方法构建高质量毒样本，通过监督微调注入后门，并通过轻量级强化学习恢复效用损失。我们评估现代、高度后期训练的开放权重模型，涵盖专有和开源框架，重点关注其广泛采用和长期任务结构的编码。该攻击成功率高达96.2%，并可配置为跨线束泛化或绑定单一目标。最终模型在标准效用基准测试中表现优于纯净基线，直接激励受害者采用这些模型。总体而言，我们展示了LLM后门在现代编码代理环境中可能带来关键的安全影响，并描述了在该环境中插入既有效又隐蔽后门所需的毒害方法。

Reimagining Text-to-SQL: A Survey of Large Language Model Architectures and Strategies

重新构想文本转SQL：大型语言模型架构与策略综述

• Authors: Shikhar Dave, Bhaskarjit Sarmah, Stefano Pasquali
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.00171
• Pdf link: https://arxiv.org/pdf/2611.00171
• Abstract Text-to-SQL systems translate natural language questions into executable SQL queries, democratizing access to relational data. Early approaches relied on handcrafted rules and template matching, while subsequent neural methods introduced encoder-decoder architectures, schema linking, and constrained decoding to handle increasingly complex queries. The emergence of large language models (LLMs), exemplified by systems such as ReFoRCE, Agentar-Scale-SQL, Snowflake's Arctic-Text2SQL-R1, and the modular XiYan-SQL framework, has dramatically advanced the field, pushing execution accuracy to new state-of-the-art levels on challenging benchmarks such as Spider, Spider 2.0, and BIRD. In parallel, modular frameworks decompose the task into sub-problems (schema retrieval, sketch-based generation, repair, and ranking), while agentic systems coordinate multiple specialized agents that communicate and dynamically allocate sub-tasks. This survey systematically reviews these LLM-driven approaches within a unified taxonomy of five paradigms (in-context learning, supervised and parameter-efficient fine-tuning, agentic and tool-augmented orchestration, retrieval-augmented generation, and reinforcement learning) and characterizes them by model granularity (monolithic vs. modular), level of supervision (zero-shot, fine-tuned, or reinforcement-based), and reasoning style (direct decoding vs. decomposed multi-step reasoning). We further analyze evaluation metrics, datasets, and open challenges, including domain generalization, efficiency, and real-world deployment, and we examine the role of the semantic layer in production systems. As LLMs increasingly access sensitive databases, we also discuss responsible AI practices. We survey over 100 papers and technical resources, focusing on work published between 2022 and 2026.
• 中文摘要 文本转SQL系统将自然语言问题转换为可执行的SQL查询，民主化了对关系数据的访问。早期方法依赖手工规则和模板匹配，而后续神经方法引入了编码-解码架构、模式链接和约束解码，以处理日益复杂的查询。大型语言模型（LLM）的出现，如ReFoRCE、Agentar-Scale-SQL、Snowflake的Arctic-Text2SQL-R1以及模块化XiYan-SQL框架，极大推动了该领域的发展，将执行精度推向了Spider、Spider 2.0和BIRD等高标准测试的最高水平。同时，模块化框架将任务分解为子问题（模式检索、基于草图的生成、修复和排序），而智能体系统协调多个专门代理，这些代理通信并动态分配子任务。本综述系统地回顾了这些基于LLM的五种统一分类法（上下文学习、监督式和参数高效微调、代理与工具增强编排、检索增强生成和强化学习），并通过模型粒度（单一与模块化）、监督层级（零样本、微调或基于强化）和推理风格（直接解码与分解多步推理）进行区分。我们还进一步分析了评估指标、数据集和开放挑战，包括领域泛化、效率和实际部署，并考察了语义层在生产系统中的作用。随着大型语言模型越来越多地访问敏感数据库，我们也讨论了负责任的人工智能实践。我们调查了100多篇论文和技术资源，重点关注2022年至2026年间发表的工作。

OmniPiano: Diverse Dexterous Piano-Playing Challenges for Standard, Robust, Safe, and Multi-Agent RL

OmniPiano：标准、稳健、安全及多智能体强化学习的多样灵巧钢琴演奏挑战

• Authors: Fan Xu, Fan Yang, Yifan Xu, Ruiyuan Zhang, Yimou Wu, Nuo Yi, Muning Wen, Guang Chen, Costas Spanos, Shangding Gu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.00243
• Pdf link: https://arxiv.org/pdf/2611.00243
• Abstract Dexterous robotic manipulation remains a major challenge in robotics, exemplified by piano playing, which requires coordinated control of fingers and provides a demanding testbed for reinforcement learning (RL). Although previous work established a benchmark for robotic piano playing, it is restricted to two hands and does not support unified evaluation. To address this gap, we introduce OmniPiano, a benchmark that provides one to up to five Shadow Hands and supports standard, robust, safe, and multi-agent RL within a shared piano-playing task family. OmniPiano is developed for scalable and diverse piano-playing tasks with configurable perturbations, explicit safety constraints, and decentralized cooperation settings. Particularly, to facilitate usability and extendability, OmniPiano adopts a highly modular design and provides comprehensive tasks to support RL study of task performance, robustness, safety, and cooperation in dexterous control. With at least 912 task settings and 36 baseline algorithms, OmniPiano further incorporates LLM-based agents to broaden the evaluation scope and reveal new insights. Extensive evaluations show that state-of-the-art RL algorithms and frontier LLM-based agents still struggle on challenging piano-playing tasks, even under standard RL settings, highlighting substantial room for improvement in dexterous control. The code, dataset, tutorial, and demo are available at this https URL.
• 中文摘要 灵巧的机器人操作仍是机器人学中的一大挑战，钢琴演奏需要协调控制手指，并为强化学习（RL）提供了严苛的测试平台。尽管先前工作为机器人钢琴演奏建立了基准，但该基准仅限于双手，且不支持统一评估。为弥补这一空白，我们推出了OmniPiano基准测试，提供一至五只影手，支持标准、稳健、安全和多代理的强化学习，属于共享的钢琴演奏任务家族。OmniPiano为可扩展且多样化的钢琴演奏任务开发，具备可配置扰动、明确的安全约束和去中心化合作设置。特别是为了促进可用性和可扩展性，OmniPiano采用高度模块化设计，提供支持强化学习任务表现、稳健性、安全性和灵巧控制协作的综合任务。OmniPiano 至少包含 912 个任务设置和 36 个基线算法，进一步整合基于 LLM 的智能体，拓宽评估范围并揭示新见解。大量评估显示，最先进的强化学习算法和前沿的 LLM 代理在具有挑战性的钢琴演奏任务上仍然存在困难，即使在标准强化学习设置下，这凸显了灵巧控制仍有很大改进空间。代码、数据集、教程和演示可在此 https 网址获取。

MEMENTO: Learning Context Management for Long Context Processing Agents

MEMENTO：长上下文处理代理的上下文管理学习

• Authors: Yunfan Zhang, Taehoon Hwang, Kathleen McKeown, Smaranda Muresan
• Subjects: Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.00244
• Pdf link: https://arxiv.org/pdf/2611.00244
• Abstract LLM agents that operate over long inputs must perform context management: deciding what to retain in a fixed-size context window. A common approach is compaction, where the model replaces its history with a self-generated note. Yet off-the-shelf agentic models, including GPT-5.5 and GLM-5.3, often write ineffective notes, and reinforcement learning (RL) alone rarely discovers useful note-taking behavior. We present MEMENTO, a long-context agent built on Qwen3-8B and trained to manage its context through compaction. We first bootstrap note-taking by fine-tuning on synthetic trajectories annotated with notes generated in hindsight, then reinforce it with RL using a reward that combines answer correctness, note-quality heuristics, and confidence calibration. Although trained only to compact into a fresh window, MEMENTO extends at test time to strategies such as context folding and confidence-based backtracking. On book-level question answering across four benchmarks, with books of up to 2.5M tokens, MEMENTO raises Qwen3-8B's accuracy from 36.9% to 67.8% while never holding more than 20K tokens in context. It outperforms GPT-5.5 agent in the same harness (13.1%) and full book context (56.7%) at less than 1/50 the cost. It also nearly matches a ~100x larger GLM-5.3 agent (69.3%) at 1/9 the cost, and comes within 6.4 points of GLM-5.3 full context at 1/26 the cost.
• 中文摘要 在长输入上运行的LLM代理必须执行上下文管理：决定在固定大小的上下文窗口中保留哪些内容。一种常见方法是压缩，即模型用自生成的笔记替换历史。然而，现成的代理模型，包括GPT-5.5和GLM-5.3，常常写入无效的笔记，且仅靠强化学习（RL）很少发现有用的笔记行为。我们介绍MEMENTO，一个基于Qwen3-8B构建的长上下文代理，训练通过压缩管理上下文。我们首先通过对带有事后注释的合成轨迹进行微调来启动笔记，然后用结合答案正确性、笔记质量启发式和置信度校准的奖励进行强化。虽然仅训练为压缩到新窗口，MEMENTO在测试时还扩展到上下文折叠和基于置信度的回溯等策略。在四个基准测试的书籍级问答中，最多有250万个代币，MEMENTO将Qwen3-8B的准确率从36.9%提升到67.8%，且上下文中代币从不超过2万个。它在相同框架下表现优于GPT-5.5代理（13.1%）和完整书籍上下文（56.7%），成本低于1/50。它还几乎能匹敌大约100倍大GLM-5.3代理（69.3%），成本为1/9，且与GLM-5.3全上下文代理仅6.4分之1，成本仅为1/26。

DEPLOY-RL: Active Boundary Discovery and Conservative Certification for Deployable Reinforcement Learning in Safety-Critical Continuous Processes

DEPLOY-RL：安全关键连续流程中可部署强化学习的主动边界发现与保守认证

• Authors: Yeojin Jang, Minu Baek, Gihun Gil, Minsung Jung, Beomdo Park, Woohyeon Kwon, Hyeonseok Jang, Junseong Park, Neha Sengar, Andres Saurez, Sangkeum Lee
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.00342
• Pdf link: https://arxiv.org/pdf/2611.00342
• Abstract Reinforcement learning (RL) policies often outperform classical controllers in simulation, yet rarely reach production in safety-critical processes. The barrier is that there is no principled way to answer "Is this policy safe to deploy?" with statistical guarantees. We introduce DEPLOY-RL, a post-training certification framework built on one key insight: deployment certification requires discovering where failures occur (boundary discovery), not measuring how much everywhere (uniform reconstruction). Our contributions: (1) a contract-coupled acquisition function that concentrates sampling on certification-critical boundaries, achieving approximately 2x sample efficiency with a semi-empirical ambiguity reduction bound (domain-calibrated convergence guarantee); (2) conformal risk control providing finite-sample false-go guarantees (<= alpha) under explicit deployment contracts; (3) a three-way decision framework (Deploy/No-Deploy/Abstain) with fail-safe PID/MPC fallback. In simulations on papermaking (industrial digital twin) and Tennessee Eastman (public benchmark), DEPLOY-RL achieves 4.4% false-go rate (vs. 8.6% for the best baseline) while retaining 88.6% policy coverage, the only method achieving <5% false-go with >85% coverage among 14 baselines under our evaluation protocol.
• 中文摘要 强化学习（RL）策略在仿真中常常优于传统控制器，但在安全关键流程中很少能实现生产。问题在于，没有原则性的方法可以用统计保证来回答“该策略部署安全吗？”这个问题。我们引入了DEPLOY-RL，这是一个基于一个关键洞见的培训后认证框架：部署认证要求发现失败发生的位置（边界发现），而非测量每个地方的失败程度（统一重建）。我们的贡献：（1）一个合同耦合的获取函数，集中采样于认证关键边界，实现约2倍的样本效率，并设有半经验的歧义减少界限（域校准收敛保证）;（2）在明确部署合同下提供有限样本的假成功保证（<=α）;（3）三向决策框架（部署/不部署/弃用），并支持失效保护的PID/MPC备选。在造纸（工业数字孪生）和田纳西伊士曼（公开基准）的模拟中，DEPLOY-RL实现了4.4%的假出行率（最佳基线为8.6%），同时保单覆盖率为88.6%，这是我们评估方案中唯一实现<5%假成率且覆盖率>85%的方法。

Compositional Self-Improvement: Promises and Pitfalls

作曲自我提升：承诺与陷阱

• Authors: Changho Shin, Daiwei Chen, Brenden Lake, Frederic Sala, Ramya Korlakai Vinayak
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.00452
• Pdf link: https://arxiv.org/pdf/2611.00452
• Abstract Although self-improvement is becoming increasingly important as tasks become more complex, models struggle to generate reliable supervision for harder, out-of-distribution instances. One way to address this challenge is to exploit compositional structure: models often solve simpler subproblems more reliably than full problems, so their subproblem predictions can be composed into supervision for harder ones. We systematically study when this kind of composed supervision supports self-improvement and when it fails. Our controlled experiments identify three failure modes: unreliable subproblem predictions, errors introduced by composition, and limited supervision coverage. Beyond controlled settings, these findings carry over to more realistic coding and reasoning tasks: composed labels are more accurate than direct self-generated labels and improve performance even on compositions longer than those seen during training, while the failure modes remain visible. Further, although these failure modes often keep compositional self-improvement from fully succeeding on its own, a short SFT warmup with compositional self-improvement can bootstrap reinforcement learning on tasks that require composing skills the model already has, particularly when direct exploration rarely succeeds. Together, these findings characterize when compositional self-improvement can provide sufficiently reliable supervision to expand model capabilities.
• 中文摘要 尽管随着任务变得越来越复杂，自我提升变得越来越重要，但模型在为更难、分布外的实例生成可靠监督方面仍存在困难。解决这一挑战的一种方法是利用组合结构：模型通常比完整问题更可靠地解决简单的子问题，因此其子问题的预测可以被合成为更难的子问题的监督。我们系统地研究了这种组合式监督何时支持自我改进，何时失败。我们的受控实验识别出三种失败模式：不可靠的子问题预测、组合引入的错误和有限的监督覆盖。超越受控设置，这些发现也适用于更真实的编码和推理任务：合成标签比直接自生成标签更准确，即使在比训练时看到的组合更长的组合上也能提升性能，同时失败模式依然可见。此外，尽管这些失败模式常常阻碍作曲自我提升的完全成功，但短暂的SFT热身配合作曲自我提升可以启动强化学习，尤其是在直接探索很少成功的情况下。这些发现共同决定了组合自我提升何时能提供足够可靠的监督以扩展模型能力。

Behavioral Steering of Vision-Language-Action Models in the Low-Parameter Regime

低参数区域视觉-语言-行动模型的行为引导

• Authors: James Tcheng, Feiyu Gavin Zhu
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.00522
• Pdf link: https://arxiv.org/pdf/2611.00522
• Abstract Reinforcement learning (RL) is increasingly used to adapt pretrained vision-language-action models (VLAs). We investigate the problem of behavioral steering: adapting a VLA's execution style to preferences such as carrying objects higher. To investigate how little trainable capacity such steering requires and how much pretrained capability can be retained, we freeze an OpenVLA-OFT 7B policy and optimize TinyLoRA adapters with 8 to 384 trainable parameters using either policy-gradient RL (GRPO) or evolutionary search (CMA-ES). With just 256 trainable parameters, we raise carry height by 120 mm on trained tasks at 92% success and by 74 mm on a held-out task at 80% success. By rescaling the learned update, we can tune the trade-off between style gains and capability retention without retraining. Increasing adapter capacity generally improves this trade-off, retaining more pretrained capability at comparable style gains. The choice of optimizer also depends on capacity, with CMA-ES being competitive at low parameter counts, while GRPO offers better trade-offs as capacity grows. These findings show that extremely low-dimensional updates can steer pretrained VLA behavior, with adapter capacity, optimizer choice, and update scaling offering practical control over adaptation and retention.
• 中文摘要 强化学习（RL）越来越多地被用于适应预训练视觉-语言-动作模型（VLA）。我们研究行为引导问题：调整VLA的执行风格以适应如将物体搬到更高位置等偏好。为了研究这种引导所需的可训练容量有多少以及预训练能力能保留多少，我们冻结了OpenVLA-OFT 7B策略，并利用策略梯度RL（GRPO）或进化搜索（CMA-ES）优化拥有8至384个可训练参数的TinyLoRA适配器。仅有256个可训练参数，我们在训练任务中成功率提升120毫米，成功率为92%，在未完成任务中提升74毫米，成功率为80%。通过重新扩展学习到的更新，我们可以在不重新训练的情况下调整样式提升与能力保留之间的权衡。增加适配器容量通常能改善这一权衡，保留更多预训练能力，且样式提升相当。优化器的选择也取决于容量，CMA-ES在低参数数下具有竞争力，而GRPO则随着容量增长提供了更好的权衡。这些发现表明，极低维的更新可以引导预训练的VLA行为，适配器容量、优化器选择和更新扩展性提供了对适配和保留的实际控制。

Dataset Mismatch Matters in Group Relative Policy Optimization for Reinforcement Learning from Verifiable Rewards

数据集不匹配在群体相对策略优化中的重要性，用于从可验证奖励中学习强化

• Authors: Zhen-Yu Zhang, Jiandong Zhang, Huaxiu Yao, Gang Niu, Masashi Sugiyama
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.00586
• Pdf link: https://arxiv.org/pdf/2611.00586
• Abstract Recently, reinforcement learning from verifiable rewards (RLVR) is practically very important, where the leading approach is group relative policy optimization (GRPO) that typically faces a trade-off between rollout efficiency and downstream performance. To improve efficiency while preserving performance, existing selective rollout methods mainly exploit only source-side training dynamics to optimize the discrete distribution over source prompts. However, in practice, the source training dataset is rarely identically distributed to the target test dataset. When such a mismatch exists, due to legal or privacy constraints, it is usually not acceptable to share target prompts themselves with the RLVR trainer. Nevertheless, it is still possible for the target data owner to provide limited aggregated feedback from a small held-out target set without revealing target prompts or rollout traces. In this paper, we propose learning-to-reweight rollout (L2RR) that explicitly incorporates target-side performance feedback into GRPO. Specifically, L2RR learns a discrete sampling distribution over source prompts by optimizing a bi-level objective that maximizes accuracy on a held-out target dataset, while simultaneously penalizing rollouts that provide little incremental learning information. Empirical studies on six math reasoning benchmarks and three model scales show that L2RR improves target-domain performance and increases rollout efficiency.
• 中文摘要 近年来，基于可验证奖励的强化学习（RLVR）实际上非常重要，主要方法是群相对策略优化（GRPO），通常在推广效率与下游性能之间存在权衡。为了提高效率并保持性能，现有的选择性推广方法主要仅利用源端训练动态来优化离散分布。然而，实际上，源训练数据集很少与目标测试数据集分布完全相同。当存在这种不匹配时，由于法律或隐私限制，通常不允许直接与RLVR训练器共享目标提示。尽管如此，目标数据所有者仍可从一个小目标集中提供有限的汇总反馈，而不暴露目标提示或推广痕迹。本文提出学习重权重推广（L2RR），明确将目标端性能反馈纳入GRPO。具体来说，L2RR通过优化一个双层目标来学习源提示的离散抽样分布，最大化对未提供额外学习信息的推广，同时惩罚那些提供少量增量学习信息的推广。基于六个数学推理基准和三个模型尺度的实证研究表明，L2RR能提升目标域表现并提高推广效率。

Enhancing Language Model Reasoning with Structured Multi-Level Modeling

通过结构化多层建模增强语言模型推理

• Authors: Siheng Xiong, Ali Payani, Faramarz Fekri
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.00616
• Pdf link: https://arxiv.org/pdf/2611.00616
• Abstract Inference-time scaling enhances a model's reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning (RL), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi-Level Reasoning (MLR), which reformulates long-CoT generation as a two-level stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming an alternating plan--execute loop. To maintain scalability, we adopt a minimal design where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. For training, we observe that outcome-reward RL provides sparse and delayed feedback for long trajectories (e.g., several thousand tokens), hindering credit assignment. We therefore introduce iterative Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget (10% SFT and 5% preference relative to the DeepSeek-R1 distillation setup), MLR outperforms both SFT-based distillation and strong RL/preference-optimization baselines across multiple base models and tasks. Moreover, MLR exhibits slower performance degradation on long-horizon reasoning, demonstrating stronger robustness under extended CoT generation.
• 中文摘要 推理时间尺度通过延长模型的思维链（CoT）来增强推理能力。然而，现有方法通常依赖于通过结果-奖励强化学习（RL）训练的单一策略，这在长期计划中常常出现失败的问题，隐式计划偏离有效策略，尤其是对于容量有限的小型LMs。为此，我们提出了多层推理（MLR），将长CoT生成重新表述为两层随机过程。高级规划者生成结构化的步骤描述符，指定推理模式和语义子目标。低级执行者随后基于这些描述符生成详细推理，形成交替的计划-执行循环。为保持可扩展性，我们采用最小设计，基础模型作为低层策略，轻量级LoRA模块实现高层策略。在训练方面，我们观察到结果-奖励RL在长路径（如数千个代币）中提供稀疏且延迟的反馈，阻碍了信用分配。因此，我们引入迭代Step-DPO，这是一种过程级偏好优化方案，利用扭曲顺序蒙特卡洛（TSMC）提供可扩展的逐步监督。这带来了更有效的训练、更高的稳定性和更高的准确性。对具有挑战性的数学、科学和逻辑推理基准测试的广泛实验表明，在相同缩减的数据预算（相对于DeepSeek-R1蒸馏设置，10% SFT和5%偏好下），MLR在多个基模型和任务中都优于基于SFT的蒸馏和强强的强强化/偏好优化基线。此外，MLR在长视野推理中表现较慢，在扩展CoT生成下表现出更强的鲁棒性。

Structural Dual Adaptation for Safe Reinforcement Learning

结构双重适应用于安全强化学习

• Authors: Runsheng Yu, Zhuoyang Song, Zengde Deng, Qixin Zhang, Li Shen, Jiaxing Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.00636
• Pdf link: https://arxiv.org/pdf/2611.00636
• Abstract Primal-dual methods enforce safety constraints in reinforcement learning through Lagrangian relaxation. For a fixed policy, the exact bounded multiplier response exhibits bang-bang switching at the constraint boundary. Practical gradient updates can oscillate through coupling with policy optimization. We propose Structural DuaL Adaptation (SDLA), which replaces this bang-bang response with a unique continuous response through current-cost centering, trajectory weighting, and quadratic regularization. We characterize activation thresholds and response sensitivity. Ternary search computes multipliers without a dual gradient stepsize. For entropy-regularized tabular policies, we bound equilibrium constraint violations alongside reward gaps. We establish convergence of delayed updates under mild assumptions. Neural PPO/SAC experiments on OmniSafe and HASARD demonstrate competitive rewards with improved constraint tracking. On HASARD, PPO-SDLA achieves higher reported mean reward than the human baseline in six of eighteen settings while meeting the respective mean-cost budgets, including four with lower mean cost. To the best of our knowledge, this is the first report of higher mean reward at lower mean cost than the human baseline across multiple HASARD settings.
• 中文摘要 原始对偶方法通过拉格朗日松弛在强化学习中强制执行安全约束。对于固定策略，精确的有界乘数响应在约束边界处表现出“砰砰”切换。实际梯度更新可以通过与策略优化的耦合振荡。我们提出了结构双重适应（SDLA），通过当前成本中心化、轨迹加权和二次正则化，将这种“砰砰”响应替换为独特的连续响应。我们表征激活阈值和响应敏感性。三元搜索计算无对偶梯度步长的乘数。对于熵正则化的表格策略，我们将平衡约束违规与奖励差距并列约束。我们在轻度假设下建立了延迟更新的收敛性。神经PPO/SAC实验在OmniSafe和HASARD上展示了通过改进约束跟踪实现竞争性奖励。在HASARD中，PPO-SDLA在18个环境中有6个实现了高于人类基线的平均奖励，同时满足相应的平均成本预算，其中4个平均成本较低。据我们所知，这是首次报告在多个HASARD环境中，平均奖励更高但平均成本低于人类基线。

When Shards Fly: Cross-Tensor In-Flight Conversion for Low-Latency, Low-Memory Weight Updates in Disaggregated RL

当碎片飞行：在分解强化学习中实现低延迟、低内存权重更新的跨张量飞行转换

• Authors: Jiayu Xiao, Ye Wang, Zhi Zhang, Jiarui Guo, Kan Liu, Tao Lan, Minchen Yu
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2611.00646
• Pdf link: https://arxiv.org/pdf/2611.00646
• Abstract Disaggregated LLM reinforcement learning (RL) repeatedly synchronizes policy weights between training and rollout GPU pools. Existing systems reduce communication through shard delivery and replica forwarding, yet separate conversion passes and staging buffers still add latency and GPU-memory overheads to synchronous updates. We present ShardFly, which coordinates reusable transfer planning with streaming GPU execution for synchronous, in-place weight updates. ShardFly introduces two components. The Planner compiles native shard mappings, quantization dependencies, and replica routes into cross-tensor schedules, reserving payload buffers only for scheduled transfers. The Executor runs each rank's schedule in one persistent transfer kernel, fusing memory-layout conversion, quantization, GPU-initiated communication, and destination writes while forwarding transformed bytes across replicas. Plans and communication state are reused under stable configurations and replaced between updates for coordinated rollout scaling. On up to 32 H20 GPUs, ShardFly achieves complete BF16 weight-update speedups of 1.26--27.92$\times$ across five models and four baselines: veRL NCCL, Awex, veRL NIXL, and WeightBridge. In end-to-end Qwen3-30B-A3B GRPO across four rollout configurations, ShardFly improves response-token throughput by 5.49--6.71\% over veRL NCCL.
• 中文摘要 分解式LLM强化学习（RL）反复同步训练和展开GPU池之间的策略权重。现有系统通过分片交付和副本转发减少通信，但分开的转换通道和暂存缓冲区仍会为同步更新增加延迟和GPU内存开销。我们介绍ShardFly，它协调可复用的传输规划与流式GPU执行，实现同步、原地权重更新。ShardFly引入两个组件。规划器将本地分片映射、量化依赖和副本路由编译为跨张量调度，仅为计划传输保留有效载荷缓冲区。执行者在一个持久传输内核中运行每个秩的调度，融合内存布局转换、量化、GPU发起通信和目标写入，同时在副本间转发转换后的字节。在稳定配置下，计划和通信状态被重用，并在更新间替换以实现协调的展开扩展。在最多32个H20 GPU上，ShardFly实现了5个型号和四个基线（veRL NCCL、Awex、veRL NIXL和WeightBridge）中1.26至27.92美元\倍数的BF16权重更新加速。在端到端Qwen3-30B-A3B GRPO中，覆盖四种部署配置，ShardFly相比veRL NCCL提升响应令牌吞吐量5.49%-6.71%。

Deep Reinforcement Learning with Smith-Predictor Delay Compensation

带有史密斯预测延迟补偿的深度强化学习

• Authors: Moh Kamalul Wafi
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.00668
• Pdf link: https://arxiv.org/pdf/2611.00668
• Abstract Time delays can degrade feedback performance and destabilize a closed loop that is stable without delay. This paper studies Smith-predictor-assisted reinforcement learning for systems with constant input delays. A deep deterministic policy gradient (DDPG) actor is trained on a nominal delay-free environment and deployed with a Smith predictor using a fixed nominal delay. For linear controllers, we derive exact closed-loop maps under plant and delay mismatch and provide small-gain and $\nu$-gap robustness conditions. A separate delay-independent Razumikhin condition gives a sufficient stability test for a fixed Lipschitz state-feedback policy. We also study output-error delay identification from a fixed input--output record, deriving the loss gradient and establishing projected-flow descent and local exponential convergence under exact model matching and an excitation condition. These identification results do not imply stability of an online adaptive predictor. Numerical experiments retained from the conference study compare PID, LQR, and DDPG controllers on stable and open-loop unstable first-order plants with valve stiction. The results illustrate the benefit of predictor compensation in the tested configurations and the sensitivity of standalone DDPG to delay and plant changes.
• 中文摘要 时间延迟会降低反馈性能，并使稳定无延迟的闭环不稳定。本文研究了恒定输入延迟系统的史密斯预测辅助强化学习。在名义无延迟环境下训练了一个深确定性策略梯度（DDPG）演员，并与史密斯预测器使用固定标称延迟部署。对于线性控制器，我们在工厂和延迟不匹配下推导出精确闭环映射，并提供小增益和$/gap鲁棒性条件。另一个独立的延迟无关Razumikhin条件为固定的Lipschitz状态反馈策略提供了足够的稳定性检验。我们还研究了从固定输入-输出记录中识别输出-错误延迟，推导损耗梯度，并在精确模型匹配和激发条件下建立预测流下降和局部指数收敛。这些识别结果并不意味着在线自适应预测器的稳定性。会议研究保留的数值实验比较了具有阀门阻滞的稳定和开环不稳定一阶植物上的PID、LQR和DDPG控制器。结果展示了预测因子补偿在测试配置中的优势，以及独立DDPG对延迟和植株变化的敏感性。

CheXPoT: Post-Hoc Visual Verification and Correction of VLM-Generated Radiology Reports

CheXPoT：VLM生成放射报告的事后可视化验证与校正

• Authors: Joey Guo, Rayan Ansari, Alaa Youssef
• Subjects: Subjects: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.00785
• Pdf link: https://arxiv.org/pdf/2611.00785
• Abstract Vision-language models (VLMs) have shown promise for chest radiograph report generation, but adoption remains limited because of hallucinated findings that are not grounded in the image. Correcting these hallucinations and improving overall report quality remains a significant challenge. Prior work has addressed this problem by refining models with reinforcement learning, preference optimization, or chain-of-thought supervision, often at substantial compute cost. To address this gap, we introduce CheXPoT, a program-of-thought (PoT) framework that verifies each generated finding against image features and removes those it cannot ground. CheXPoT decomposes generated reports into predicates with RadGraph and an LLM, grounds these predicates against a DenseNet-121 and Grad-CAM visual evidence table, executes PoT verification code, and rewrites reports after removing unsupported predicates. On a 4,145-image held-out split, CheXPoT yields statistically significant precision gains on 8 of 14 CheXpert classes and F1 gains on 5, with no significant decreases in either metric. CheXPoT lifts micro-average precision from 19.2% (MedGemma-4B) to 46.8% (MedGemma-4B + CheXPoT) and micro-average F1 from 23.0% to 49.4%, absolute gains of 27.6% and 26.4% respectively. We also tested CheXPoT on other VLM report generators. On LLaVA-Med, CheXPoT lifts micro-average precision from 4.5% to 17.7% and micro-average F1 from 6.3% to 19.4%. On MAIRA-2, CheXPoT lifts micro-average precision from 15.6% to 42.1% and micro-average F1 from 21.0% to 45.3%. Our results suggest that inexpensive post-hoc visual verification may improve VLM trustworthiness by correcting unsupported findings from generated reports. CheXPoT lays a foundation for tools that help validate VLM-generated radiology reports and reduce unsupported findings. These innovations may subsequently help increase trust and adoption of VLM tools in Medicine.
• 中文摘要 视觉语言模型（VLM）在生成胸部X光报告方面展现出潜力，但由于存在不基于图像的幻觉发现，采用度仍然有限。纠正这些幻觉并提升整体报告质量仍是一大挑战。以往的研究通过强化学习、偏好优化或思维链监督来优化模型，通常会付出较高的计算成本来解决这一问题。为弥补这一空白，我们引入了CheXPoT，一种思维程序（PoT）框架，该框架对每个生成的发现进行图像特征的验证，并剔除无法基于的部分。CheXPoT将生成的报告分解为RadGraph和LLM的谓词，基于DenseNet-121和Grad-CAM视觉证据表，执行PoT验证代码，并在删除不支持的谓词后重写报告。在4,145张未显示图像的拆分中，CheXPoT在14个CheXpert类别中有8个实现了统计学显著的精度提升，F1类别在5个类别上获得了显著提升，且两项指标均无显著下降。CheXPoT将微观平均精度从19.2%（MedGemma-4B）提升至46.8%（MedGemma-4B + CheXPoT），微观平均F1从23.0%提升至49.4%，绝对提升分别为27.6%和26.4%。我们还在其他VLM报告生成器上测试了CheXPoT。在LLaVA-Med上，CheXPoT将微观平均精度从4.5%提升至17.7%，微观平均F1从6.3%提升至19.4%。在MAIRA-2上，CheXPoT将微观平均精度从15.6%提升至42.1%，微观平均F1从21.0%提升至45.3%。我们的结果表明，低成本的事后可视化验证可以通过纠正生成报告中无依据的发现，提升VLM的可信度。CheXPoT为验证VLM生成的放射报告并减少无支持发现的工具奠定了基础。这些创新有望进一步提升VLM工具在医学中的信任和采用率。

Steer, Don't Tell: Breaking Reasoning Plateaus with Reachability-Aligned Self-Distillation

引导，不告诉：通过可达性对齐的自我提炼打破推理瓶颈

• Authors: Zijian Wu, Lingkai Kong, Lin Yang, Haiteng Zhao, Kuikun Liu, Kai Chen, Dahua Lin, Wenwei Zhang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.00796
• Pdf link: https://arxiv.org/pdf/2611.00796
• Abstract Post training for large reasoning models remains insufficient on challenging problems, whose correct trajectories are hard to sample. Though prior works address this difficulty by integrating supervised fine tuning on expert trajectories with reinforcement learning, such external experts are not always available. Existing self improvement methods instead use reference solutions to create a privileged self-teacher. Yet privileged teacher correctness does not imply student reachability. Solution conditioning shifts the teacher distribution toward actions unlikely under the unprivileged student, producing seemingly "correct" trajectories that are unreachable or even mal-formed learning targets. Accordingly, we identify reachability alignment as the key to self constructed supervision and propose Self-Steered Reasoner (SSR). SSR realizes this principle through student support projection, which transforms the privileged teacher distribution into a directional signal only over reasoning actions locally reachable under the unguided student. Verification then retains correct trajectories sampled from this reachable space. To further boost the sampling efficiency, an iterative refinement framework is adopted, using diagnostic guidance based on current trajectories. On problems with pass@64 = 0 even after extensive RL, SSR recovers correct and learnable trajectories for 74.6% without parameter updates. Fine tuning on these trajectories followed by resumed reinforcement learning improves aggregate accuracy by 4.9 percentage points on AIME25/26 and HMMT26. These results show that privileged guidance becomes effective self-improvement data when aligned with the reasoning actions the student can reach.
• 中文摘要 对于具有挑战性的问题，后续培训仍然不足，这些问题的正确轨迹难以被采样。尽管此前的研究通过将专家轨迹的监督微调与强化学习相结合来解决这一难题，但这类外部专家并非总是可用。现有的自我提升方法则使用参考解来创造特权自导。然而，特权教师正确性并不意味着学生可达性。解决方案条件反射将教师分布向非特权学生不太可能采取的行动，产生看似“正确”的轨迹，但实际上无法达到甚至是错误的学习目标。因此，我们将可达性对齐视为自我构建监督的关键，并提出了自我引导推理者（SSR）。SSR通过学生支持投射实现这一原则，将特权教师的分布转变为仅针对无引导学生局部可达的推理行为的方向信号。验证则保留从该可达空间中抽样的正确轨迹。为进一步提升采样效率，采用基于当前轨迹的诊断指导的迭代细化框架。即使经过大量强化学习，仍出现pass@64 = 0的问题，SSR在无参数更新的情况下恢复正确且可学习的轨迹，达到74.6%。对这些轨迹进行微调后恢复强化学习，在AIME25/26和HMMT26的总准确率提升了4.9个百分点。这些结果表明，当特权指导与学生能够达到的推理行动保持一致时，便成为有效的自我提升数据。

Toward Counterfactual Learning for Selective Re-Examination in Agentic Visual Search

迈向在代理视觉搜索中选择性复审的反事实学习

• Authors: Ao Chen, Shengchao Chen, Yan Luximon
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.00848
• Pdf link: https://arxiv.org/pdf/2611.00848
• Abstract Visual search over high-resolution images requires a vision-language model (VLM) not only to determine where to inspect, but also whether another observation is warranted before committing to an answer. Existing prompting, demonstrations, and outcome-based reinforcement learning can encourage re-examination, yet do not directly supervise whether it is beneficial at a particular search state. We introduce ReSee, a benchmark of re-examination decisions comprising 3,227 visual-search trajectories and 8,827 explicit commit-or-re-examine decisions, with counterfactual adjudication that distinguishes states where additional inspection is warranted from those where commitment is preferable. Building on this formulation, we propose CARE (Counterfactual Advantage for Re-Examination), a decision-level reinforcement learning method that compares paired continuations under commit and re-examine verdicts from the same search state. Their cost-aware return difference estimates the counterfactual value of an additional observation and is assigned only to the corresponding verdict tokens, providing localized credit for whether search should continue. Experiments expose limited re-examination selectivity in existing VLMs and show that CARE significantly improves accuracy and decision selectivity over GRPO across two backbones, with higher accuracy and fewer visual requests on six unseen OOD benchmarks.
• 中文摘要 对高分辨率图像进行视觉搜索不仅需要视觉语言模型（VLM）确定检查位置，还需要在做出答复前是否需要再次观察。现有的提示、演示和基于结果的强化学习可以鼓励重新审视，但不直接监督其在特定搜索状态下的有益性。我们介绍ReSee，这是一个重新审视决策基准，包含3,227条视觉搜索轨迹和8,827条显式提交或重新审视决策，并通过反事实裁决区分需要额外检查的状态与更倾向承诺的状态。基于这一表述，我们提出了CARE（反事实优势用于再审视），这是一种决策级强化学习方法，比较同一搜索状态下提交和重新审视下的成对续写。其成本感知性返回差估算额外观测值的反事实价值，仅赋予相应的判决标记，为搜索是否应继续提供局部信用。实验揭示了现有VLM中重新检查选择性的有限，显示CARE在两个骨干链上显著提升了GRPO的准确性和决策选择性，在六个未见的OOD基准中准确率更高，视觉请求更少。

Learning to Group and Order: Cross-Instance Self-Supervised RL for Vision-Centric MLLMs

学习分组与下班：面向视觉型多层次语言营销的跨实例自我监督强化学习

• Authors: Songming Yang, Hong-Tao Yu, Xiu-Shen Wei
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.00892
• Pdf link: https://arxiv.org/pdf/2611.00892
• Abstract Reinforcement learning (RL) post-training has become an important paradigm for improving multimodal large language models (MLLMs), especially with automatically verifiable rewards. Self-supervised visual pretext tasks provide such rewards without human annotations. However, existing methods mainly exploit within-instance structure, offering limited supervision for cross-instance discrimination and fine-grained comparison. We introduce Group-and-Order Self-Supervised Reinforcement Learning (GO-SSL), a cross-instance jigsaw framework for vision-centric MLLM post-training. Given two visual instances, GO-SSL mixes their local elements and trains the model to group them by instance of origin while recovering the order within each group, coupling instance-level comparison with spatial, temporal, or geometric structure recovery. This paired formulation turns the same unlabeled data into richer verifiable supervision through diverse pairings, shuffles, and a random-to-hard curriculum. We instantiate GO-SSL mainly on cross-image jigsaw tasks and further extend it to 3D depth and video temporal ordering. Across diverse benchmarks, GO-SSL consistently improves over the base MLLM and prior self-supervised post-training baselines, with clear gains in fine-grained perception and spatial understanding. Further evaluations across model scales and families support its generality. These results suggest that cross-instance comparison provides a useful self-supervised RL signal for improving transferable visual understanding in MLLMs. Code is available at this https URL.
• 中文摘要 强化学习（RL）训练后已成为改进多模态大型语言模型（MLLM）的重要范式，尤其是在自动验证奖励方面。自监督视觉借口任务无需人工注释即可提供此类奖励。然而，现有方法主要利用实例内部结构，提供有限的跨实例辨别和细粒度比较监督。我们介绍了群组与秩序自监督强化学习（GO-SSL），这是一个用于视觉中心MLLM后训练的跨实例拼图框架。给定两个视觉实例，GO-SSL混合其局部元素，训练模型按实例起源分组，同时恢复每个组内的顺序，将实例级比较与空间、时间或几何结构恢复相结合。这种配对表述通过多样化配对、洗牌和随机到硬性课程，将相同的未标记数据转化为更丰富且可验证的监督。我们主要在跨图像拼图任务中实例化GO-SSL，并进一步扩展到3D深度和视频时间排序。在不同基准测试中，GO-SSL相较于基础MLLM和之前的自我监督训练后基线持续提升，细粒度感知和空间理解有明显提升。跨模型尺度和模型族的进一步评估支持其普遍性。这些结果表明跨实例比较为提升MLLM中可转移视觉理解提供了有用的自监督强化学习信号。代码可在此 https 网址获取。

UI-UG 2: Advancing Mobile UI Understanding and Generation with Fine-Grained Supervision and Reinforcement Learning

UI-UG 2：通过细粒度监督与强化学习推进移动UI理解与生成

• Authors: Hao Yang, Shuoyang Liu, Xiaoyu Lin, Zhou Fang, Maji Huang, Zhaosong Huang, Ruichao Mao, Teng Guo, Hai Rao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2611.00926
• Pdf link: https://arxiv.org/pdf/2611.00926
• Abstract Mobile UI understanding and generation require both fine-grained component perception and reliable structured synthesis. Inspired by the unified direction introduced by UI-UG, we present UI-UG 2, a compact 4B-parameter multimodal large language model (MLLM) that advances mobile UI Understanding and Generation with broader task coverage, a front-end-aligned taxonomy, and online render-aware reinforcement learning. UI-UG 2 constructs a multi-source understanding corpus from RICO, VINS, Android in the Wild, and Android Control, annotated with an 80-category two-level UI taxonomy aligned with front-end component semantics, and pairs it with over 5,000 requirement-screenshot-JSON tuples for progressive structured UI generation through a domain-specific language (DSL). For understanding, UI-UG 2 introduces decomposed rewards for category prediction, OCR fidelity, color recognition, dense grounding, and GUI grounding. Unlike the offline Direct Preference Optimization (DPO)-style generation optimization in UI-UG, UI-UG 2 optimizes generation through end-to-end online Group Relative Policy Optimization (GRPO): sampled DSL outputs are rendered with a Playwright-based server-side pipeline and evaluated using LLM-as-a-Judge and contrastive preference ranking. Experiments show that UI-UG 2 achieves the strongest referring and dense-grounding results among the evaluated models on our 80-category benchmark, and achieves the best composite generation score among compared models at a compact scale (4B). More importantly, we observe a consistent understanding-to-generation transfer: adding fine-grained UI perception supervision improves generation quality under a fixed training budget, suggesting that component-level semantics provide a shared representational foundation for both tasks.
• 中文摘要 移动用户界面的理解与生成既需要细粒度的组件感知，也需要可靠的结构化综合。受UI-UG引入的统一方向启发，我们呈现UI-UG 2，一个紧凑的4B参数多模态大型语言模型（MLLM），通过更广泛的任务覆盖、前端对齐的分类法和在线渲染感知强化学习，推动移动UI理解与生成。UI-UG 2构建了来自RICO、VINS、Android in the Wild和Android Control的多源理解语料库，标注了80个类别的两级UI分类法，并结合了超过5000个需求截图JSON元组，支持通过领域特定语言（DSL）实现渐进式结构化UI生成。为理解，UI-UG 2引入了类别预测、OCR精度、颜色识别、密集基准和GUI基础的分解奖励。与UI-UG中的离线直接偏好优化（DPO）风格生成优化不同，UI-UG 2通过端到端在线组相对策略优化（GRPO）优化生成：采样的DSL输出通过基于Playwright的服务器端流水线渲染，并利用LLM即评判和对比偏好排名进行评估。实验显示，UI-UG 2在我们80个类别基准测试中，在被评估模型中实现了最强的引用和密集基准效果，并在紧凑尺度（4B）下获得比较模型中最高的复合生成得分。更重要的是，我们观察到理解到生成的一致性：在固定训练预算下，添加细粒度的用户界面感知监督提升了生成质量，表明组件级语义为两项任务提供了共享的表征基础。

HALO: Hierarchical Evidence Selection and Reinforcement Learning for Allocating Budgeted Label Optimization from Longitudinal Clinical Records

HALO：从纵向临床记录中分配预算标签优化的层级证据选择与强化学习

• Authors: Huaiyuan Ying, Feiqing Huang, Ryumei Nakada, Katherine P. Liao, Tianxi Cai
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2611.00986
• Pdf link: https://arxiv.org/pdf/2611.00986
• Abstract Learning from extensive longitudinal clinical records--spanning repeated encounters, evolving conditions, and heterogeneous documentation--under a finite labeling budget requires solving two coupled problems: deciding what evidence to keep and deciding which patients deserve higher-fidelity labels. Prior work on active learning and multi-fidelity label acquisition mainly addresses the second problem, while treating the evidence presented to the acquisition policy as fixed. We present HALO, a Hierarchical Evidence Selection and Reinforcement Learning pipeline for Allocating Budgeted Label Optimization from Longitudinal Clinical Records. This framework jointly learns what to read and what label source to acquire under budget. HALO first selects informative evidence from longitudinal records and then uses a budget-aware policy to allocate free-, silver-, and gold-fidelity label sources. For frozen policies, our theory characterizes asymmetric acquired-label channels and gives a budget-constrained finite-sample gold-risk bound for a common downstream logistic model. In a heart-failure case study, HALO attained the highest MedAlign acquired-label F1 at six of eight target budgets. A downstream classifier trained on HALO-acquired labels attained the highest external MGB F1 at budgets 15-60 and remained within 0.01 of the best comparator at budgets 75-120.
• 中文摘要 在有限的标签预算下，从广泛的纵向临床记录中学习——涵盖反复接触、病情演变和异质文档——需要解决两个共同问题：决定保留哪些证据，以及决定哪些患者应获得更高保真度标签。此前关于主动学习和多保真度标签获取的工作主要解决了第二个问题，同时将提交给获取政策的证据视为固定。我们介绍了HALO，这是一个用于从纵向临床记录分配预算标签优化的层级证据选择与强化学习流程。该框架共同学习应阅读哪些内容，以及在预算内获取哪些标签来源。HALO首先从纵向记录中选择信息性证据，然后采用预算意识政策分配免费、银质和黄金忠实度标签来源。对于冻结策略，我们的理论表征了非对称的获得标签通道，并为共同下游物流模型提供了预算受限的有限样本黄金风险界限。在一项心力衰竭案例中，HALO在八个目标预算中的六个中获得了最高的MedAlign获取标签F1。一个基于HALO获取标签训练的下游分类器在预算15-60时获得了最高的外部MGB F1，并且在预算75-120时与最佳对比者保持在0.01以内。

Pulse: Long-Horizon Tasks Need Streaming Personalization

Pulse：长远视野任务需要流式个性化

• Authors: Qijia Zhuang (1), Jialiang Liang (2), Yuxiang Ren (1) ((1) Nanjing University, (2) Xinjiang University)
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.00988
• Pdf link: https://arxiv.org/pdf/2611.00988
• Abstract Large language model agents increasingly undertake complex, long-horizon tasks that benefit from sustained personalized support through long-term memory. Existing memory-augmented methods retrieve and use memory primarily from the initial request, with limited awareness of ongoing execution and emerging personalization needs. Agents may therefore overlook user preferences and project constraints during multistep tasks. We propose Pulse, a streaming personalization method for long-horizon tasks. An independent, lightweight memory agent continuously monitors execution, identifies emerging personalization needs and potential preference conflicts, and uses user and project memories to provide targeted interventions at appropriate stages. Reinforcement learning optimizes when and how the memory agent intervenes, improving the timeliness and accuracy of personalized support. Pulse scores 71.9% on $\pi$-Bench and 46.4% on MCP-Persona, achieving the highest overall scores among the compared methods on both datasets while improving task execution efficiency.
• 中文摘要 大型语言模型代理越来越多地承担复杂且长期的任务，这些任务通过长期记忆获得持续的个性化支持。现有的记忆增强方法主要从初始请求中检索和使用记忆，对正在执行和新出现的个性化需求意识有限。因此，代理在多步骤任务中可能会忽视用户偏好和项目限制。我们提出了Pulse，一种用于长期任务的流式个性化方法。一个独立且轻量级的记忆代理持续监控执行，识别新兴的个性化需求和潜在偏好冲突，并在适当阶段利用用户和项目记忆提供有针对性的干预。强化学习优化记忆代理干预的时间和方式，提高了个性化支持的及时性和准确性。Pulse在$\pi$-Bench上得分71.9%，MCP-Persona中得分46.4%，在两类数据集中均获得最高综合得分，同时提升任务执行效率。

Entropy-Regularized Reinforcement Learning for the Regime-Switching Consumption-Investment Problem

熵正则化强化学习用于体制转换消费-投资问题

• Authors: Congde Hu, Danping Li, Xun Li, Yijie Zeng
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2611.01046
• Pdf link: https://arxiv.org/pdf/2611.01046
• Abstract This paper establishes an entropy-regularized reinforcement learning (ERRL) framework for the continuous-time consumption-investment problem under a regime-switching market model with general utility functions. By incorporating randomized exploratory policies and Shannon entropy regularization into the stochastic optimal control formulation, we derive closed-form representations for the optimal exploratory consumption and investment strategies. Specifically, the optimal investment strategy follows a multivariate Gaussian distribution, whereas the optimal consumption strategy takes the form of a Gibbs measure---which specializes to a Gamma distribution under logarithmic utility and a non-standard exponential distribution under power utility. Crucially, our theoretical analysis reveals that the classical deterministic optimal decisions correspond to the mode---rather than the mean---of these exploratory distributions. Furthermore, we establish the convergence of the exploratory value function to its classical unregularized counterpart as the temperature parameter vanishes. For algorithmic design, we develop a continuous-time q-learning framework grounded in a novel continuous-time q-function, prove the policy improvement theorem, and establish martingale characterizations for both on-policy and off-policy learning. We find that parameterizing the actor network based on theoretical distribution families can help accelerate convergence and enhance numerical stability. Furthermore, numerical experiments across regime-switching environments provide empirical support for our theoretical findings, demonstrating stable loss convergence as well as the monotonicity and concavity of the learned value functions.
• 中文摘要 本文建立了一个熵正则化强化学习（ERRL）框架，用于在具有通用效用函数的状态切换市场模型下，解决连续时间消耗-投资问题。通过将随机探索策略和香农熵正则化纳入随机最优控制表述，我们推导出了最优探索性消耗和投资策略的闭式表示。具体来说，最优投资策略遵循多元高斯分布，而最优消费策略则采用吉布斯测度---该测度专门化于对数效用下的伽马分布和幂效用下的非标准指数分布。关键是，我们的理论分析表明，经典的确定性最优决策对应的是这些探索性分布的模态---而非均值---。此外，我们建立了探索性价值函数与其经典非正则化对应函数的收敛性，因为温度参数为零。在算法设计方面，我们基于一种新颖的连续时间q函数开发了一个连续时间q学习框架，证明了策略改进定理，并为策略上和非策略学习建立了马丁格尔特征。我们发现，基于理论分布族对演员网络进行参数化有助于加速收敛并提升数值稳定性。此外，跨体制切换环境的数值实验为我们的理论发现提供了实证支持，证明了稳定的损失收敛以及所学价值函数的单调性和凹性。

Hi-CoLA: High Confidence Lower Bound Approximation Based Reinforcement Learning for Flex-Route Transit Operation Control

Hi-CoLA：高置信下界近似强化学习，适用于灵活路线交通运营控制

• Authors: Qi Kang, Haris N. Koutsopoulos, Hongliang Guo, Jinhua Zhao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.01071
• Pdf link: https://arxiv.org/pdf/2611.01071
• Abstract This paper studies safe policy improvement for real-time operation control of flex-route transit (FRT) systems, where robustness is critical and any performance decrease is undesirable. While reinforcement learning (RL) has shown strong empirical success, its adoption remains limited in safety and cost critical applications, such as intelligent transportation, since learned policies lack statistically certified performance guarantees before deployment. In this paper, we propose the High Confidence Lower Bound Approximation (Hi-CoLA) RL framework, which combines importance sampling and concentration inequalities with a differentiable surrogate network to efficiently estimate the high-confidence performance lower bound and optimize it via policy gradient for robust performance improvement. On the FRT operation control problem, Hi-CoLA outperforms state-of-the-art offline and safe RL methods in both average and tail-risk performance. A case study on an actual route in Chicago, implemented in the SUMO simulator, serves as a proof of concept. Hi-CoLA enhances real-time FRT control with guaranteed performance and is broadly applicable to other safety critical decision-making scenarios.
• 中文摘要 本文研究了灵活路线交通（FRT）系统实时运行控制的安全策略改进，其中鲁棒性至关重要，性能下降是不理想的。尽管强化学习（RL）在实证上取得了显著成功，但在安全性和成本关键应用（如智能交通）中的采用仍然有限，因为学习后的策略在部署前缺乏统计认证的性能保证。本文提出了高置信下界近似（Hi-CoLA）RL框架，结合重要性抽样和集中度不等式与可微替代网络，高效估计高置信度性能下界，并通过策略梯度优化以实现稳健的性能提升。在FRT运算控制问题上，Hi-CoLA在平均和尾部风险表现上均优于最先进的离线和安全强化学习方法。在芝加哥实际路线中实现的案例研究作为概念验证。Hi-CoLA增强了实时FRT控制，并保证性能，并广泛适用于其他安全关键决策场景。

Zeroth-Order Policy Optimization for Scheduling in Queueing Networks

排队网络调度的零阶策略优化

• Authors: Deyi Wang, Qining Zhang, Lei Ying
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.01186
• Pdf link: https://arxiv.org/pdf/2611.01186
• Abstract Reinforcement learning in queueing network scheduling presents significant challenges because of the countably infinite state space and the high stochasticity in transitions. In this paper, we propose zeroth-order policy optimization algorithms that are easy to implement and rely solely on estimates of the average cost, a single scalar quantity, without requiring the estimation of value or advantage functions. Our analysis focuses on back-pressure guarded policies and establishes uniform smoothness of the average cost objective with respect to the policy parameters. Building on this property, we establish finite-time convergence guarantees to an $\epsilon$-stationary policy for the proposed algorithms. Numerical experiments on multiple queueing networks, including systems beyond classical exponential service distribution, demonstrate competitive performance, outperforming gradient-based RL methods like PPO and traditional queueing baselines.
• 中文摘要 队列网络调度中的强化学习面临重大挑战，因为状态空间可数无限且转移随机性高。本文提出了零阶策略优化算法，易于实现，仅依赖平均成本估计，即单一标量量，无需估计价值或优势函数。我们的分析聚焦于背压保护策略，建立了平均成本目标相对于策略参数的均匀平滑性。基于该性质，我们为所提算法建立了有限时间收敛保证，支持$\ε$-平稳策略。在多队列网络上的数值实验，包括超越经典指数服务分布的系统，显示出具有竞争力的表现，优于基于梯度的强化学习方法如PPO和传统队列基线。

When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training

探索何时能防止崩溃？自我一致性训练中的三种模式与奖励路径

• Authors: Jaeho Hwang, Uiwon Hwang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.01307
• Pdf link: https://arxiv.org/pdf/2611.01307
• Abstract Test-time reinforcement learning (TTRL) adapts a model without labels by rewarding answers that agree with the majority, but it collapses the policy onto its majority answer and can remove the correct answer from the samples. Exploration can prevent this collapse, yet its strength is chosen by trial and error, and how the training algorithm changes the required strength is unclear. We model training on a single problem as a continuous flow of the logits of a softmax policy and analyze which policies are stable as the exploration coefficient varies. Only collapsed policies are stable below a lower boundary and only policies spread over many answers above an upper boundary, and we derive these boundaries in closed form for an agreement reward with smoothing toward the uniform policy, compute them for the majority reward of TTRL on policies with one leading answer and bound them with interval arithmetic for actions on ordered bins. We prove that under exact gradients, the learning rate and, in a mean-field model, the normalization of GRPO leave these boundaries unchanged when exploration passes through them together with the reward but shift them when exploration bypasses them. Exact sums on these policies then show that a bonus for rare answers needs a coefficient above $1/\log(G-1)$ for groups of $G$ samples before the normalization and about two to three times that after it. In free-form TTRL on MATH-500, a bonus placed before the normalization prevented collapse and kept more correct answers among the samples, although it lowered pass@1.
• 中文摘要 测试时强化学习（TTRL）通过奖励与多数意见一致的答案来适应无标签的模型，但会将策略压缩到多数答案上，并可能从样本中移除正确答案。探索可以防止这种崩溃，但其强度通过反复试验决定，训练算法如何改变所需强度尚不明确。我们将单一问题的训练建模为softmax策略logit的连续流，并分析哪些策略随着探索系数的变化而稳定。只有崩溃的策略在下边界以下稳定，且只有分布在多个上边界以上的策略中，我们以封闭形式推导这些边界，以平滑一致策略获得一致奖励，计算TTRL多数奖励的策略，并用区间算术对有序的语箱进行行动的约束。我们证明，在精确梯度下，学习率以及在平均场模型中GRPO归一化时，探索与奖励一同通过这些边界时保持不变，但当探索绕过这些边界时边界会移动。这些策略的精确和显示，稀有答案的加分在归一化前，$G$样本组的系数需要超过$1/\log（G-1）$，归一化后约为其两到三倍。在MATH-500的自由形式TTRL中，加分置于归一化之前防止崩溃，保持样本中更多正确答案，尽管pass@1降低。

Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning

无交互逆强化学习的对齐飞轮引导抑制补丁

• Authors: Elias Malomgré, Pieter Simoens
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.01322
• Pdf link: https://arxiv.org/pdf/2611.01322
• Abstract Learned reward models allow reinforcement-learning agents to use demonstrations, implicit feedback, safety constraints, social norms, or other sources that do not provide a complete hand-designed scalar reward. However, a learned reward model can assign high reward to unwanted behavior. A policy can exploit these errors during training. Later requirements can also identify additional regions where the deployed reward should change. We introduce \emph{flywheel-guided patching}, a mechanism for making such localized corrections without retraining the learned evaluator. An Alignment Flywheel identifies overgeneralized or constraint-violating regions and compiles verified findings into suppression patches. We instantiate the mechanism for Interactionless Inverse Reinforcement Learning using MoE-GUIDE, placing patches in the model's latent space and retrieving only a small local subset during online reward evaluation. In the main MuJoCo benchmark, we evaluate patching without expert-state reset warm starts. We also test whether operator-specified norms absent from the demonstrations can be compiled into the same reward interface and compare this with enforcing the corresponding rules online. The results show that localized corrective information can modify an existing learned reward without providing a complete replacement reward. Compilation can reuse judgments that are expensive or unavailable during control. Cheap deterministic rules may be simpler to check directly at each step. Code for MoE-GUIDE, Alignment Flywheel, and flywheel patching is at this https URL.
• 中文摘要 学习奖励模型允许强化学习代理使用演示、隐性反馈、安全约束、社会规范或其他无法提供完整手工标量奖励的来源。然而，学习奖励模型可以为不良行为赋予高奖励。策略可以在训练过程中利用这些错误。后续需求还可以识别出额外区域，这些区域应当发生变化。我们引入了\emph{flywheel-guided patching}，这是一种无需重新训练评估者即可进行局部纠正的机制。对齐飞轮识别过度泛化或违反约束的区域，并将验证结果汇总成抑制补丁。我们使用MoE-GUIDE实例化无交互逆强化学习机制，将补丁置于模型潜伏空间，在线奖励评估时仅检索一小部分局部子集。在主MuJoCo基准测试中，我们评估了无专家状态重置热启动的补丁。我们还测试了演示中缺失的操作员指定规范是否可以编译到同一奖励界面，并与在线执行相应规则进行比较。结果显示，局部化的纠正信息可以修改已有的学习奖励，而无需提供完整的替代奖励。编译可以在控制过程中重复使用昂贵或无法获得的判断。廉价的确定性规则可能更易于在每一步直接检查。MoE-GUIDE、对齐飞轮和飞轮补丁的代码位于此 https 网址。

Toward Executable Multi-framework Front-end Code Generation with Self-Correction

迈向可执行的多框架前端代码生成，并具备自我纠正功能

• Authors: Linxiao Li, Haoran Ma, Chenyue Wang, Jiaye Lin, Haochen Sui, Jiechao Gao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2611.01383
• Pdf link: https://arxiv.org/pdf/2611.01383
• Abstract Generating codes from UI screenshots has recently benefited from MLLMs, yet existing methods centered on HTML targets and transfer poorly to multi-frameworks such as React, Vue, and Angular. However, unlike HTML, which does not entail compilation issues, generating executable code in multi-framework settings is more challenging due to framework-specific differences in syntax and failure modes, as well as the need to maintain global consistency across files. To address this problem, we study executable multi-framework front-end code generation and propose MESCoder, a tree-structured self-corrective generation framework. Our method first extracts a structural scaffold from the screenshot and then constructs sub-trees and assembly relations step by step, transforming large project generation into a sequence of localized decisions. On top of this scaffold, we introduce a unified Project Agent that continuously revises its own partial project under runtime sandbox feedback and dynamically chooses between local repair and cluster level repair. For training, we adopt a two-stage strategy that first initializes the policy offline to learn basic multi-framework generation and repair priors, and then refines it with online reinforcement learning to optimize long horizon executability and final page quality. The resulting framework models multi-framework executability as a conditioned project state transition process and improves the stability and quality of complex front end generation through self correction. Experiments show that our method improves executability, while narrowing the visual-fidelity gap to HTML-based generation. Code is available at this https URL.
• 中文摘要 近年来，从界面截图生成代码受益于MLLM技术，但现有方法以HTML目标为中心，迁移到React、Vue和Angular等多框架时表现不佳。然而，与不涉及编译问题的HTML不同，多框架环境中生成可执行代码更具挑战性，因为框架特有的语法和失败模式差异，以及需要在文件间保持全局一致性。为解决这一问题，我们研究了可执行的多框架前端代码生成，并提出了MESCoder，一种树结构的自我纠正生成框架。我们的方法首先从截图中提取结构支架，然后逐步构建子树和汇编关系，将大型项目生成转化为一系列局部决策。在此支架之上，我们引入了一个统一的项目代理，它在运行时沙盒反馈下持续修订自身部分项目，并在本地修复和集群级修复之间动态选择。在训练方面，我们采用两阶段策略：首先离线初始化策略，学习基础的多框架生成和修复先验，然后通过在线强化学习进行优化，以优化长视野可执行性和最终页面质量。最终的框架将多框架可执行性建模为有条件的项目状态转换过程，并通过自我修正提升复杂前端生成的稳定性和质量。实验表明，我们的方法提升了可执行性，同时将可视化与保真度差距缩小到基于HTML的生成。代码可在此 https 网址获取。

Improved Sample Complexity Analysis For First Order Bilevel Reinforcement Learning

改进的样本复杂度分析，用于一阶双层强化学习

• Authors: Mudit Gaur, Naman Saxena, Vaneet Aggarwal
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.01500
• Pdf link: https://arxiv.org/pdf/2611.01500
• Abstract Bilevel reinforcement learning optimizes an upper-level objective through the policy response to a lower-level reinforcement learning problem. Such problems arise when learning a reward model while adapting a policy to that reward. However, a gap remains between sample-complexity guarantees and practical actor--critic training. Tabular analyses restrict applicability to finite spaces, while continuous-state analyses often impose a uniform Polyak--Łojasiewicz (PL) condition. We show that this condition need not hold even for smooth positive policies with exactly representable values. We develop a single-loop, fully first-order method for continuous states and finite actions with specified multilayer neural actors and a persistently trained nonlinear critic built from fixed features. Under explicit initialization, regularity, feature-excitation, and value-realizability conditions, fixed reference-KL regularization yields local curvature, and we prove containment of the coupled iterates. Shared trajectories control sampling noise, while paired critic analysis controls normalized critic-error differences as the response perturbation shrinks. A joint descent and tracking argument establishes an improved $\widetilde{\mathcal O}(\varepsilon^{-2})$ sample complexity for $O(\varepsilon)$ expected squared gradient norm of the upper objective induced by the selected regularized policy response, including critic training.
• 中文摘要 双层强化学习通过对低层强化学习问题的策略响应，优化了上层目标。此类问题出现在学习奖励模型并调整策略以适应该奖励时。然而，样本复杂度保证与实际的actor-critic训练之间存在差距。表格分析限制适用有限空间，而连续状态分析通常施加均匀的Polyak-Łojasiewicz（PL）条件。我们证明即使对于具有精确可表示值的平滑正策略，这一条件也不一定成立。我们开发了一种单循环、全一阶连续状态和有限动作的方法，采用指定的多层神经演员和由固定特征构建的持续训练非线性批评者。在显式初始化、正则性、特征激发和值实现条件下，固定参考KL正则化产生局部曲率，并证明耦合迭代的包含性。共享轨迹控制采样噪声，而配对批判分析则控制随着响应扰动缩小而归一化后的批判误差差异。联合下降和跟踪论证建立了改进的$\widetilde{\mathcal O}（\varepsilon^{-2}）$样本复杂度，$O（\varepsilon）$期望平方梯度范数由所选正则化政策响应诱导，包括批判训练。

Beyond Validation Loss: Coverage and Reliability Dynamics in Robot Policy Pretraining

超越验证损失：机器人政策预训练中的覆盖率与可靠性动态

• Authors: Zhanyi Sun, Max Simchowitz, Shuran Song
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.01565
• Pdf link: https://arxiv.org/pdf/2611.01565
• Abstract In modern robot learning, generative robot policies are first pretrained through supervised learning from demonstrations, then refined through reinforcement learning, supervised finetuning, or other forms of posttraining. Yet conventional pretraining metrics, such as validation loss and average task success, provide an incomplete account of which policies support both effective direct deployment and subsequent posttraining. In this work, we study closed-loop pass@k, which evaluates the probability that k rollouts from the same initial condition attain at least one success. Thus, pass@1 measures mean task success; large-k pass@k measures whether success remains reachable under repeated sampling. Across a wide range of posttraining recipes, we show that large-k pass@k reveals posttraining differences not captured by pass@1. In particular, checkpoints with similar pass@1 but different large-k pass@k have different posttraining performance. Moreover, we show pass@k peaks at different checkpoints during supervised pretraining, revealing a coverage-reliability tradeoff over the course of training. Based on these observations, we propose two practical interventions: (1) a purely offline checkpoint-selection criterion that balances coverage and reliability without closed-loop evaluation, and (2) a generative data augmentation procedure that expands supervision beyond the finite demonstration set, preserving large-k pass@k while maintaining or improving pass@1 and raising the observed coverage-reliability Pareto frontier. Together, our results motivate evaluation, checkpoint selection, and training procedures that explicitly account for posttraining. Blog post, visualizations, and code are available at this https URL.
• 中文摘要 在现代机器人学习中，生成式机器人策略首先通过演示的监督学习进行预训练，然后通过强化学习、监督微调或其他形式的后训练进行优化。然而，传统的预训练指标，如验证损失和平均任务成功率，未能完全说明哪些策略既支持有效的直接部署，也支持后续后训练。本研究中，我们研究闭环pass@k，评估同一初始条件k次部署至少一次成功的概率。因此，pass@1衡量任务成功率;大k pass@k衡量在反复抽样下是否仍可达成功。在广泛的训练后方案中，我们表明大k pass@k揭示了训练后pass@1未捕捉到的差异。特别是，具有pass@1相似大k但不同pass@k的检查点在训练后表现不同。此外，我们展示了监督预训练期间不同检查点的pass@k峰值，揭示了训练过程中覆盖率与可靠性的权衡。基于这些观察，我们提出了两种实用干预措施：（1）纯离线检查点选择标准，平衡覆盖率和可靠性，无需闭环评估;（2）一种生成式数据增强程序，将监督扩展到有限示范集之外，保持大k pass@k同时保持或改善pass@1并提升观测覆盖度-可靠性帕累托前沿。综合来看，我们的结果激励了评估、检查点选择和训练程序，明确考虑训练后问题。博客文章、可视化和代码可在此 https URL 获取。

ODE Policy: Robot Learning via Action Dynamics, Not Actions

常微分方程策略：通过动作动力学而非动作学习机器人

• Authors: Jindou Jia, Jingliang Li, Kuangji Zuo, Tuo An, Yuxuan Hu, Mengfei Zhao, Bofan Lyu, Jiaqi Bai, Daoyuan Zhu, Hengyu Dai, Junpeng Zhang, Jianfei Yang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.01729
• Pdf link: https://arxiv.org/pdf/2611.01729
• Abstract Most robot policies, whether trained by imitation learning (IL) or reinforcement learning (RL), map observations to action values at discrete control steps and leave the continuity of physical motion to be learned implicitly. We propose ODE policy, which learns the dynamics of the action rather than the action itself. A neural ordinary differential equation (ODE) learns the vector field of the action by matching the integrated actions to the targets over short action chunks in IL and single control steps in RL. A lightweight diversity head captures multi-modal behavior by randomizing the direction of the field. At deployment, each action is integrated from the previous one, so the executed trajectory is continuous by construction. We evaluate it as a visuomotor policy, as the action head of vision-language-action (VLA) and world action models (WAMs), and as the actor of model-based RL against 14 baselines on unimanual, bimanual, dexterous, locomotion, and loco-manipulation tasks. In IL, against flow matching, it reaches 5.9$\times$ the success, 5.4$\times$ the generalization, and 3.9$\times$ the mode balance after one epoch, and generates actions 10.8$\times$ faster. It mostly moves more smoothly than human demonstrations. On 14 real-world tasks across single-arm, dual-arm, dexterous, and humanoid robots, it achieves the highest success rate. In RL, it further speeds up SuperMimic for humanoid motion tracking and smooths its actions. Efficient in learning, data, parameters, and execution, ODE policy offers a promising path to scaling robot learning.
• 中文摘要 大多数机器人策略，无论是通过模仿学习（IL）还是强化学习（RL）训练，都将观察值映射到离散控制步骤的动作值，并将物理运动的连续性隐式学习。我们提出常微分方程策略，学习动作的动态，而非动作本身。神经常微分方程（ODE）通过将集成动作与目标匹配，在IL中短动作块和强化学习中单次控制步骤，学习动作的矢量场。轻量级多样性头通过随机化场方向捕捉多模态行为。部署时，每个动作从前一个动作积分，因此执行的轨迹通过构造是连续的。我们将它作为视觉运动策略、视觉-语言-动作（VLA）和世界动作模型（WAM）的动作头，以及作为基于模型的强化学习在单手、双手、灵巧、移动和运动操作任务的14个基线下进行的执行者。在IL中，面对流匹配，它在一个时代后达到5.9$\倍成功率，5.4$\倍数的概括，3.9倍\倍的模式平衡，并且生成动作的速度比人类演示快10.8倍\倍数。它大多数情况下比人类演示更流畅。在14项真实任务中，涵盖单臂、双臂、灵巧和人形机器人，它取得了最高的成功率。在强化学习中，它进一步加速了SuperMimic的人形运动跟踪，并使其动作更平滑。在学习、数据、参数和执行方面高效，常微分方程策略为机器人学习的扩展提供了有前景的路径。

MATE: Multi-Agent Traffic Engineering for Topology-Aware Distributed SRv6 Scheduling under Bursty Traffic

MATE：针对拓扑感知分布式SRv6调度的多代理流量工程，适用于突发流量

• Authors: Yuang Chen, Shuyi Liu, Chengdi Lu, Fengqian Guo, Hancheng Lu, Chang Wen Chen
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2611.01831
• Pdf link: https://arxiv.org/pdf/2611.01831
• Abstract As network traffic becomes increasingly dynamic and bursty, conventional centralized traffic engineering (TE) suffers from long control-loop delays caused by network-state aggregation, centralized computation, and policy distribution. In this paper, we propose MATE, a multi-agent TE method for proactive distributed scheduling in hybrid networks with partial deployment of Segment Routing over IPv6 (SRv6). MATE uses historical traffic stability to characterize the burst risk of each traffic demand and path sensitivity to quantify the ability of candidate paths to accommodate additional traffic, thereby enabling differentiated proactive scheduling tailored to heterogeneous traffic characteristics. MATE adopts an offline-centralized training and online-distributed execution architecture. During training, a shared global critic evaluates joint actions under a reward combining maximum link utilization (MLU) with variability-weighted path sensitivity, while counterfactual difference rewards provide agent-specific credit for actor learning. During online execution, edge agents generate traffic-splitting policies from local observations and apply them through SRv6, avoiding centralized optimization in each scheduling interval. Experiments on the China Environment for Network Innovations (CENI) testbed and simulations on the Sanet topology show that MATE outperforms a multi-agent reinforcement learning (MARL) ablation baseline without agent-specific credit assignment, COPE, and TEAL in most evaluated scenarios and approaches centralized FIGRET on measured traffic traces. On CENI-Abilene, MATE achieves MLU within 10% of the linear-programming optimum in up to 90.1% of test instances. MATE also achieves lower median normalized MLU than FIGRET in most tested failure scenarios, with mean neural-inference times of 2.05--3.14~ms.
• 中文摘要 随着网络流量变得越来越动态和突发化，传统的集中式流量工程（TE）因网络状态聚合、集中计算和策略分布而导致的控制环路延迟较长。本文提出MATE，一种多代理的DE方法，用于混合网络中的主动分布式调度，部分部署IPv6段路由（SRv6）。MATE利用历史流量稳定性来描述每个流量需求的突发风险和路径敏感性，量化候选路径容纳额外流量的能力，从而实现针对异构流量特性的差异化主动调度。MATE采用离线集中训练和在线分布式执行架构。在训练过程中，共享的全球评审员在结合最大链路利用率（MLU）和变数加权路径敏感性的奖励下评估联合行动，而反事实差分奖励则为参与者学习提供代理特定积分。在线执行时，边缘代理从本地观测生成流量拆分策略，并通过SRv6应用，避免每个调度区间的集中优化。在中国网络创新环境（CENI）测试平台和Sanet拓扑上的模拟显示，MATE在大多数评估场景中优于多智能体强化学习（MARL）消融基线，且在测量的流量痕迹上采用集中式FIGRET。在CENI-Abilene中，MATE在多达90.1%的测试实例中实现线性编程最优的MLU在10%内。MATE在大多数测试失败场景中，中位归一化MLU低于FIGRET，平均神经推断时间为2.05-3.14~毫秒。

Don't Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations

别猜，明智地问：开源领域大型语言模型对话中的基准测试与主动引发训练

• Authors: Rongyu Zhang, Mutian Bao, Jinjian Zhang, Yuetian Du, He Xu, Wei Zhou, Xiaojun Zhu, Ming Kong, Jie Liu, Linjian Mo, Qiang Zhu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.01878
• Pdf link: https://arxiv.org/pdf/2611.01878
• Abstract When user queries are incomplete, large language models should ask targeted questions before committing to an answer, a capability we call proactive elicitation, yet evaluation on real user-assistant queries across seven LLMs (8B to frontier) shows that fewer than 11% of underspecified queries trigger a follow-up question. To benchmark and train this capability, we introduce VagueBench, where each seed query is annotated with structured intent and detail units then degraded into intent-vague and fully-vague variants via controlled removal of linguistic cues, yielding known hidden information sets for evaluating both when and what to ask. We propose DialogGRPO, an offline reinforcement learning method that organizes multi-turn elicitation as a dialogue-state directed acyclic graph, where candidate questions at each node receive dense turn-level rewards combining immediate information gain with lookahead value through group-relative comparison, enabling fine-grained credit assignment without runtime simulation. DialogGRPO elicits missing information on over 88% of vague queries with substantial gains in both recovery efficiency and answer coverage, without over-asking on complete queries, outperforms prompting and test-time-scaling baselines, and generalizes zero-shot to MultiWOZ 2.4 and AmbigQA.
• 中文摘要 当用户查询不完整时，大型语言模型应在承诺回答前提出有针对性的问题，我们称之为主动引发，但对七个大型语言模型（8B至Frontier）真实用户助手查询的评估显示，少于11%的未明确要求查询触发后续问题。为对该能力进行基准测试和训练，我们引入了VagueBench，每个种子查询都标注结构化的意图和细节单元，然后通过受控去除语言线索，分解为意图模糊和完全模糊的变体，从而获得已知的隐藏信息集，用于评估何时和问什么。我们提出了DialogGRPO离线强化学习方法，将多回合引发组织为对话状态导向无环图，每个节点的候选问题通过群体相对比较获得密集的回合级奖励，结合即时信息获得与前瞻性价值，实现细粒度的信用分配，无需运行时模拟。DialogGRPO在88%以上的模糊查询中提取缺失信息，在恢复效率和答案覆盖度上均有显著提升，且不会对完整查询过度询问，表现优于提示和测试时间缩放基线，并将零样本推广至MultiWOZ 2.4和AmbigQA。

NAS-PO: Native Attention-Strategy Policy Optimization for Vision-Language Models

NAS-PO：视觉语言模型的原生注意力策略策略优化

• Authors: Ke An, Shizhe Xiang, Yue Liu, Liang Chen, Pei Fu, Shaojie Zhang, Xiaoxing Hu, Yupei Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.02008
• Pdf link: https://arxiv.org/pdf/2611.02008
• Abstract Reinforcement learning with verifiable rewards (RLVR) has become an effective post-training paradigm for vision-language models (VLMs). Standard RLVR, however, assigns identical outcome-derived credit to all generated tokens regardless of their perceptual relevance. Recent perception-aware methods estimate visual reliance through image perturbation or masking. These indirect proxies require perturbation branches, while corruption sensitivity does not necessarily identify the visual evidence used for reasoning. We instead shape model perception through native attention, an intrinsic model signal. Rather than treating attention as a post-hoc explanation, we formulate it as a reward-anchored, token-aligned routing signal that encourages attention patterns supporting visually grounded reasoning. To this end, we propose NAS-PO, a Native Attention-Strategy Policy Optimization framework. NAS-PO analyzes native attention with a visual mass policy that quantifies visual allocation and a conditional visual policy that captures spatial routing over visual tokens, making the optimization well-posed. Based on this, Native Attention-Strategy Advantage Shaping (NAS-AS) draws on the visual-mass policy to redistribute signed policy-gradient credit for visually engaged rollouts. The Adaptive Attention-Coupled (AAC) loss then aligns the visual-mass policy on all response tokens and the conditional visual policy on visually active ones, consolidating the visual-routing patterns that drive successful trajectories. Experiments demonstrate that NAS-PO substantially surpasses leading RL-tuned VLMs across 7 out-of-domain reasoning and perception benchmarks, with its effectiveness consistently validated across 8B and 32B. Visual-Routing Sensitivity (VRS) analysis further indicates that the optimized attention prioritizes visually load-bearing regions.
• 中文摘要 带有可验证奖励的强化学习（RLVR）已成为视觉语言模型（VLM）一种有效的训练后范式。然而，标准RLVR对所有生成的代币赋予相同的结果衍生信用，无论其感知相关性如何。最新的感知感知方法通过图像扰动或掩蔽来估计视觉依赖。这些间接代理需要扰动分支，而损坏敏感性不一定识别用于推理的视觉证据。我们通过原生注意力——一种内在模型信号——来塑造模型感知。我们不将注意力视为事后解释，而是将其表述为一个以奖励为锚定、代币对齐的路由信号，鼓励支持视觉基础推理的注意力模式。为此，我们提出了NAS-PO，一种原生注意力策略策略优化框架。NAS-PO通过一个量化视觉分配的视觉质量策略和一个捕捉视觉标记空间路由的条件视觉策略分析原生注意力，使优化变得良好。基于此，原生注意力-策略优势塑造（NAS-AS）利用视觉质量策略重新分配已签署的策略梯度积分，用于视觉参与的推广。自适应注意力耦合（AAC）损失随后将所有响应标记的视觉质量策略与视觉活跃标记的条件视觉策略对齐，整合驱动成功轨迹的视觉路由模式。实验表明，NAS-PO在7个域外推理和感知基准中显著超过领先的强化学习调优VLM，其有效性在8B和32B中持续验证。视觉路由敏感性（VRS）分析进一步表明，优化后的注意力优先处理视觉负载区域。

Retrieval-Enhanced Cultural Evaluation for Social Media Translation Training

社交媒体翻译培训中的检索增强文化评估

• Authors: Linjuan Wu, Yiwen Qiu, Xinze Lyu, Dingming Li, Hui Yao, Zhongyi Sang, Ye Guo, Daoxin Zhang, Zhe Xu, Fei zhao, Mu Chuan, Yao Hu, Weiming Lu, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.02009
• Pdf link: https://arxiv.org/pdf/2611.02009
• Abstract Social media translation requires preserving meaning while capturing the cultural nuances of user-generated content (UGC). However, gaps in the cultural knowledge of LLM-based evaluators can lead to inaccurate assessments and unreliable training rewards. To improve these signals, we use retrieval-enhanced cultural evaluation to train CUE-MT, a family of social media translation models. A calibrated, domain-specific Judger guides supervised data refinement and is augmented by an Agent for hard-example mining and reinforcement learning. The Agent retrieves relevant external evidence as needed and distills it into concise hints that the Judger uses to score translations and assign rewards. Across 9B, 27B, and 35B-A3B backbones, CUE-MT outperforms the evaluated open-source baselines of comparable size on CUE-MT-Bench and three public benchmarks. Human evaluation on CUE-MT-Bench shows cultural effectiveness gains of 7.44-15.12 percentage points over the strongest external baseline in each size group. Retrieval improves the Judger's four-way classification accuracy by 8.07 percentage points over a matched no-retrieval baseline that uses LLM-generated hints alone. Training ablations further support the combination of hard-example mining and Agent-enhanced rewards.
• 中文摘要 社交媒体翻译需要在保留意义的同时捕捉用户生成内容（UGC）的文化细微差别。然而，基于LLM的评估者文化知识的缺口可能导致评估不准确和培训奖励不可靠。为改善这些信号，我们使用检索增强型文化评估训练CUE-MT，这是一系列社交媒体翻译模型。校准的领域特定评判者引导监督数据细化，并由代理辅助硬样本挖掘和强化学习。代理根据需要检索相关外部证据，并将其提炼成简明提示，评判者用以评分翻译和分配奖励。在9B、27B和35B-A3B骨干链中，CUE-MT在CUE-MT-Bench和三个公开基准测试中表现优于同等规模的开源基线。在CUE-MT-Bench上的人类评估显示，每个规模组中，文化有效性提升为较最强外部基线高出7.44-15.12个百分点。检索比仅使用LLM生成提示的匹配无检索基线，将评判者的四向分类准确率提升了8.07个百分点。训练消融进一步支持硬样本挖掘与代理增强奖励的结合。

NorthCUA Technical Report: Turbocharging Computer-Use Agents through Experience at Scale

NorthCUA技术报告：通过大规模经验增强计算机使用代理的动力

• Authors: Chaoyun Zhang, Bingxuan Li, Pinxue Guo, Yangfan Gao, Zikun Cai, Tianci Xie, Hui Li, Yuqi Huang, Yifei Cao, Bo Mao, Tian Tang, Qichang Zheng, Tongtong Liu, Xiaoxing Wang, Jinxiu Liu, Lizhi Lin, Zhou Shao, Maosen Zhou, Miao Zheng, Shuo Zhang, Yangji He, Kepeng Lei, Ning Zhao, Wei Shen, Zhenjiang Jin, Xinyue Zhang, Peng Sun, Zhenhua Han, Rui Zheng, Hang Yan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.02102
• Pdf link: https://arxiv.org/pdf/2611.02102
• Abstract Scaling computer-use agents from individual actions to complex software work requires experience that connects decisions to complete outcomes. We present NorthCUA, an integrated system for turbocharging computer-use agents through experience at scale. Its Experience Flywheel supplies diverse executable tasks to the NorthCUA Harness, which sustains long interactions, records decisions, and checks task outcomes. The Flywheel uses these records to curate verified trajectories. For unsolved tasks, recursive self-improvement evolves reusable skills and preserves verified milestones so later attempts build on earlier progress. Staged Learning combines this experience with grounding data from the Flywheel to develop GUI competence through mid-training and long-horizon supervised fine-tuning. The policy then learns from fresh attempts in the same Harness through joint desktop-web reinforcement learning, closing the loop between experience production and policy improvement. The resulting Nano, Mini, and Pro family achieves state-of-the-art results among open-source models across GUI grounding, desktop execution, and Web interaction under the reported evaluation settings. For more responsive Web workflows, NorthCUA-Ultrafast pairs direct action selection with open-ended generation on pre-RL Nano, offering a measured quality-latency tradeoff. Together, these advances establish NorthCUA as a high-performing model family and a reusable foundation for agents that carry complex software work from intent to completion.
• 中文摘要 将计算机操作代理从单个操作扩展到复杂软件工作，需要将决策与完整结果连接起来的经验。我们介绍NorthCUA，一个集成系统，通过大规模经验加速计算机使用代理。其体验飞轮向NorthCUA框架提供多样化的可执行任务，维持长时间交互，记录决策并检查任务结果。飞轮利用这些记录策划经过验证的轨迹。对于未解决任务，递归自我提升进化可重复使用的技能，并保留经过验证的里程碑，使后续尝试能够建立在早期进展基础上。分阶段学习将此经验与飞轮的基础数据结合，通过中期培训和长期监督微调培养图形用户界面能力。该策略随后通过桌面与网络联合强化学习，从同一框架内的新尝试中学习，闭合经验生产与策略改进之间的循环。最终的Nano、Mini和Pro系列在开源模型中，在图形界面基础、桌面执行和Web交互方面均取得了最先进的性能，且均符合报告的评估设置。对于更响应的Web工作流程，NorthCUA-Ultrafast将直接动作选择与开放式生成结合，实现了质量与延迟的可衡量权衡。这些进步共同确立了NorthCUA作为高性能模型家族的地位，并为承载复杂软件工作从意图到完成的代理提供了可重用的基础。

Value Forcing: Aligning Return Distributions at All Noise Levels

价值强制：在所有噪声水平下对齐回报分布

• Authors: Chongyi Zheng, Zhiyuan Zhou, Qiyang Li, Benjamin Eysenbach, Sergey Levine
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02160
• Pdf link: https://arxiv.org/pdf/2611.02160
• Abstract Learning accurate value functions is central to reinforcement learning (RL), yet remains challenging because supervision is limited and indirect: ground-truth values are unavailable, and each transition in the data only provides an immediate reward that can be used to construct bootstrapped targets as supervision. This challenge is particularly pronounced when data is limited or reward signals are sparse, as is common in real-world applications such as robotics. We argue that the key to accurate value learning in this limited data regime is extracting richer learning signals from the available data. In this paper, we introduce Value Forcing, a new approach that uses flow matching to estimate return distributions and enables accurate value learning from limited data. Our key insight is that we can extract richer learning signals by enforcing Bellman consistency at every noise level. Empirically, Value Forcing improves the accuracy of estimating the return distribution, and prevents the distribution from collapsing to a single value, as occurs with prior methods. We instantiate Value Forcing with two practical value-learning algorithms: VF, an actor-critic method, and IVF, an implicit value-learning method. We evaluate both algorithms across $26$ long-horizon tasks in the offline RL setting and find that they outperform prior value-learning methods, with particularly large gains when data is limited. Our project website is available at this https URL.
• 中文摘要 学习准确的价值函数是强化学习（RL）的核心，但由于监督有限且间接：缺乏基础真实值，且数据中的每次转变仅提供即时奖励，可用于构建自举目标作为监督。当数据有限或奖励信号稀疏时，这一挑战尤为突出，这在机器人等现实应用中很常见。我们认为，在这种有限数据环境中实现准确价值学习的关键是从可用数据中提取更丰富的学习信号。本文介绍了价值强制，这是一种利用流匹配估计回报分布并从有限数据中实现准确价值学习的新方法。我们的关键见解是，通过在每个噪声级别强制执行贝尔曼一致性，我们可以提取更丰富的学习信号。从经验角度看，价值强制提高了估计回报分布的准确性，防止分布像以往方法那样崩溃为单一值。我们用两种实用的价值学习算法实现价值强制：VF，一种行为者-批评者方法，以及IVF，一种隐式价值学习方法。我们在离线强化环境中，评估了这两种算法在26美元长视野任务中的表现，发现它们优于以往的价值学习方法，尤其在数据有限时提升显著。我们的项目网站访问地址为 https URL。

ANCHOR: KL-Anchored Adaptation for Stable Offline-to-Online RL

主播：吉隆坡锚定的稳定离线到在线强化学习

• Authors: Geonwoo Cho, Jaegyun Im, Doyoon Kim, Lexin Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02177
• Pdf link: https://arxiv.org/pdf/2611.02177
• Abstract Offline-to-online reinforcement learning (RL) follows the pretrain-finetune paradigm: an agent is first pretrained on historical data and then adapted through online interaction. We study offline-to-online adaptation without offline replay, where the agent must improve using only the pretrained model and newly collected experience. A key challenge is to avoid catastrophic failure, a sharp early performance collapse during online fine-tuning, while still achieving strong asymptotic performance. Through controlled experiments, we show that removing offline replay can induce catastrophic failure, while retaining strong conservative regularization from offline RL can suppress final online performance. We also find that common stabilization strategies do not reliably prevent early collapse, and that the critic-side mechanisms we test are insufficient on their own. Motivated by these findings, we propose ANCHOR, a simple adaptation method that anchors the actor to the offline-pretrained policy early in online fine-tuning, anneals this constraint over time, and retains only a small residual conservative critic regularizer. We provide a theoretical interpretation showing that KL anchoring helps preserve critic reliability across nearby occupancy distributions, with a two-phase regret bound explaining when anchoring should be maintained or relaxed. Across FrankaKitchen, AntMaze, and Adroit tasks, ANCHOR substantially reduces catastrophic failure while maintaining strong asymptotic performance. Ablations show that KL annealing and the residual conservative weight are important for balancing early stability and online adaptation, while additional analysis shows that ANCHOR is most beneficial when the pretrained policy contains useful behavior to preserve. Code is available at this https URL.
• 中文摘要 离线到在线的强化学习（RL）遵循预训练-微调范式：代理先在历史数据上进行预训练，然后通过在线交互进行适应。我们研究离线到在线的适应过程，无需离线回放，代理必须仅使用预训练模型和新收集的经验进行改进。一个关键挑战是避免灾难性失败，即在线微调过程中性能急剧崩溃，同时仍能实现强的渐近性能。通过受控实验，我们表明去除离线重放可能导致灾难性失败，而保留离线强化的保守正则化可抑制最终在线表现。我们还发现，常见的稳定策略无法可靠地防止早期崩溃，且我们测试的批评方机制本身不足。基于这些发现，我们提出了ANCHOR这一简单的适应方法，该方法在在线微调早期将行为者锚定于离线预训练策略，随时间退火该约束，并仅保留少量剩余的保守批评正则化。我们提供了理论解释，表明KL锚定有助于保持邻近占用分布中的批评者可靠性，并以两阶段遗憾约束解释何时应维持或放松锚定。在FrankaKitchen、AntMaze和Adroit任务中，ANCHOR显著减少灾难性失败，同时保持强强的渐近性能。消融显示KL退火和剩余保守权重对平衡早期稳定性和在线适应至关重要，进一步分析显示，当预训练策略包含可保留的有用行为时，ANCHOR最为有益。代码可在此 https 网址获取。

Iterative Gumbel Planning for Continuous Control

连续控制的迭代甘贝尔规划

• Authors: Shaohuai Liu, Weirui Ye, Yilun Du, Le Xie
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02236
• Pdf link: https://arxiv.org/pdf/2611.02236
• Abstract Search-based planning remains a powerful policy improvement paradigm in reinforcement learning, driving milestone successes in discrete domains. However, extending this paradigm to continuous control exposes a structural mismatch: search evaluates a finite set of action candidates, whereas continuous RL typically relies on Gaussian policies. We show that winner-action distillation into a continuous Gaussian induces a \textit{projection error} and a stepwise variance-expanding gradient tendency when the selected action is sufficiently far from the current mean. Empirically, this tendency is associated with elevated terminal variance and a return gap on the evaluated tasks. To address this, we propose \textbf{Iterative Gumbel Planning (IGP)}. By adopting a per-dimension factorized categorical policy, IGP naturally aligns the policy support with the search candidates, eliminating the projection error and reducing the residual approximation to a bounded discretization error. To bypass the combinatorial explosion of high-dimensional discrete action spaces, IGP pairs Gumbel Top-$K$ candidate sampling with an MPPI-style multi-round iterative search to refine the policy at each state. Empirically, IGP achieves state-of-the-art aggregate performance and sample efficiency on DMControl and HumanoidBench. Furthermore, it produces more stable action sequences with lower temporal variance, demonstrating its practical potential for real-world robotic deployment.
• 中文摘要 基于搜索的规划仍然是强化学习中强有力的策略改进范式，推动离散领域的里程碑成功。然而，将该范式推广到连续控制暴露出结构性不匹配：搜索评估的是有限的行动候选对象，而连续强化学习通常依赖高斯策略。我们证明，当所选动作远离当前平均值时，获胜行动提炼为连续高斯分布会诱导一个\textit{投影误差}和逐步扩展方差的梯度趋势。从经验角度看，这种趋势与评估任务的终端方差升高和返回缺口相关。为此，我们提出了\textbf{迭代Gumbel规划（IGP）}。通过采用每维分解的类别策略，IGP自然地将策略支持与搜索候选对齐，消除投影误差，并将剩余近似简化为有界离散化误差。为绕过高维离散动作空间的组合爆炸，IGP将Gumbel Top-$K$候选采样与MPPI式多轮迭代搜索结合，在每个状态下细化策略。从经验角度看，IGP在DMControl和HumanoidBench上实现了最先进的汇总性能和样本效率。此外，它能产生更稳定且时间方差更低的动作序列，展示了其在现实机器人部署中的实用潜力。

Detect, Repair and Audit: A Hierarchical Multi-Agent Defense Against Adversarial Attacks in Energy Forecasting

检测、修复与审计：能源预测中针对对抗性攻击的分层多智能体防御

• Authors: Zewei Shi, Ruoxi Sun, Minhui Xue, Feng Liu, Hua Ma, Xingliang Yuan, Surya Nepal
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2611.02310
• Pdf link: https://arxiv.org/pdf/2611.02310
• Abstract Accurate electricity-demand forecasting is essential for the modern energy infrastructure. Recently, Times-Series Large Language Models (TS-LLMs) have emerged as a promising approach to forecasting more accurate demand from historical consumption data in zero- and few-shot settings, helping energy operators effectively plan supply and allocate energy resources. However, these models are vulnerable to compromised sensor inputs that can distort forecasts and lead to erroneous market orders or unnecessary load shedding. Existing defenses focus primarily on end-device forecasting, overlooking the distinct data-drift, latency, and audit requirements across energy infrastructure. To address this gap, we extract from major energy standards and regulatory frameworks a three-layer conceptual model that captures the security postures for each layer. Guided by this model, we develop a Multi-Agent Reinforcement Learning (MARL) detector coupled with signal repair at local control centers, designed to mitigate distributional data drift through complementary feature-based detection and clean-reference-calibrated repair while meeting latency budgets. We further deploy an evidence-grounded LLM auditing agent with human-in-the-loop validation for audit-critical Virtual Power Plants (VPPs). In our evaluation, the best MARL configuration achieves an F1 score above 0.91 across all dataset-attack pairs, while signal repair reduces adversarial-to-clean input Relative Squared Error (RSE) by up to 83%. The few-shot auditor detects 77.78% of attacks in the Sherlock dataset. On the evaluated hardware, detection and repair require 7.95 ms on average, with a worst-observed latency of 14.50 ms, remaining within the local-center latency budget informed by IEEE 1646.
• 中文摘要 准确的电力需求预测对于现代能源基础设施至关重要。近年来，时间序列大语言模型（TS-LLMs）作为一种有前景的方法，能够从历史用电数据中预测零和少样本参数下更准确的需求，帮助能源运营商有效规划供应和分配能源资源。然而，这些模型容易受到传感器输入受损的影响，导致预测失真，导致错误的市场订单或不必要的停电。现有防御措施主要关注终端设备预测，忽视了能源基础设施中独特的数据漂移、延迟和审计需求。为弥补这一空白，我们从主要能源标准和监管框架中提取了一个三层概念模型，捕捉各层的安全态势。在该模型的指导下，我们开发了多智能体强化学习（MARL）检测器，配合本地控制中心的信号修复，旨在通过互补的基于特征的检测和干净引用校准的修复，减轻分布数据漂移，同时满足延迟预算。我们还进一步部署了基于证据的LLM审计代理，支持审计关键虚拟电厂（VPP）的人工在环路验证。在我们的评估中，最佳MARL配置在所有数据集攻击对中均获得超过0.91的F1分数，而信号修复则可将对抗性到干净输入的相对平方误差（RSE）降低高达83%。少数机会的审计员检测到Sherlock数据集中77.78%的攻击。在评估的硬件上，检测和维修平均需要7.95毫秒，最严重的延迟为14.50毫秒，且符合IEEE 1646规定的本地中心延迟预算。

Do Visual RL Gains Scale at Test Time? A Quantile View of Post-Training

视觉强化学习的提升会在测试时随之增长吗？训练后分位数视图

• Authors: Austin Wang, Lexing Ying
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.02366
• Pdf link: https://arxiv.org/pdf/2611.02366
• Abstract Reinforcement learning (RL) post-training improves the single-sample quality of visual generative models, but deployment often provides additional inference compute through Best-of-N sampling. Whether RL gains transfer under such test-time scaling, particularly on difficult held-out tasks, remains poorly understood in the visual domain. Our central contribution is to systematically identify and characterize a failure mode of scaling transfer in visual RL: across multiple RL methods, model backbones, and evaluations spanning compositional generation, text rendering, and preference alignment, higher single-sample reward does not necessarily imply a better quality-compute frontier. RL gains may attenuate, saturate, or even reverse at large sampling budgets, with the base model eventually outperforming its post-trained counterpart. We explain these regimes through a quantile-transfer framework, showing that Best-of-N places increasing weight on the upper tail of the reward distribution as N grows. Consequently, RL can improve typical samples while compressing or degrading the rare high-quality outcomes that test-time scaling exploits. As a simple intervention motivated by this analysis, we introduce Quantile Utility Policy Optimization (QUPO), which optimizes a controllable mixture of quantile utilities corresponding to different inference budgets. Experiments on SD3.5-M and BAGEL show that QUPO better preserves upper-tail quality and substantially improves scaling on challenging held-out prompts relative to standard RL post-training. Our results suggest that visual generative models should be evaluated and optimized not only for single-sample quality, but for how post-training reshapes the full reward distribution that inference-time scaling exploits.
• 中文摘要 强化学习（RL）在训练后提升了视觉生成模型的单样本质量，但部署时通常通过N次抽样提供额外的推理计算。在这种测试时间尺度下，尤其是在困难的持续任务中，RL是否能获得迁移，在视觉领域仍然不清楚。我们的核心贡献是系统地识别和描述视觉RL中尺度转移的失败模式：在多种RL方法、模型骨干以及涵盖组合生成、文本渲染和偏好对齐的评估中，更高的单样本奖励并不一定意味着质量计算的前沿更优。在大采样预算下，RL的收益可能会衰减、饱和甚至逆转，基模型最终会优于其后训练后的对应模型。我们通过分位转移框架解释这些模式，表明N的优势会随着N的增长，奖励分布的上尾部权重逐渐增加。因此，强化学习可以在压缩或降解测试时间缩放所利用的稀有高质量结果的同时，提升典型样本。作为一项简单的干预措施，基于该分析，我们引入了分位效用策略优化（QUPO），该优化对应不同推理预算的可控分位效用组合。在SD3.5-M和BAGEL上的实验显示，QUPO比标准强化学习后更能更好地保留上尾质量，并显著提升挑战性提示的缩放性。我们的结果表明，视觉生成模型不仅应评估和优化单样本质量，还应关注后训练如何重塑推理时间缩放所利用的整体奖励分布。

SGR-OPSD: Learning from Subgoal Plans with Graph-Regularized Credit Assignment for Agentic Reinforcement Learning

SGR-OPSD：通过图正则化学分分配学习子目标计划学习，用于能动强化学习

• Authors: Shu Liu, Yang Chong, WenShi Chen, Man Lan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02414
• Pdf link: https://arxiv.org/pdf/2611.02414
• Abstract OPSD provides dense supervision for agentic reinforcement learning through privileged information feedback. However, existing privileged information often lacks explicit task-progression planning, while teacher-student log-probability gaps reflect shifts in model preference rather than true task progress. To address these limitations, we propose Subgoal-Guided Graph-Regularized OPSD (SGR-OPSD), a framework that combines subgoal supervision with interaction-derived evidence for turn-level credit assignment. Specifically, SGR-OPSD uses LLM-generated subgoal plans to provide dense action-level guidance, converting them into advantage weights via a constrained, KL-regularized objective. To ground credit in observed outcomes, we construct a Subgoal Progress Graph that estimates empirical returns from sampled transition frequencies. The graph compares empirical action returns against the state average to derive preferences that calibrate the teacher's credit weights. Evaluated on ALFWorld, WebShop, and Search-QA across two model scales (Qwen2.5-3B and 7B), SGR-OPSD consistently outperforms GRPO and strong self-distillation baselines, improving success rates over GRPO by up to 7.4 percentage points and reaching 85.8% on ALFWorld with Qwen2.5-7B. Ablation studies confirm that subgoal guidance and graph-based weight calibration are key to these performance gains.
• 中文摘要 OPSD通过特权信息反馈为能动强化学习提供密集的监督。然而，现有特权信息通常缺乏明确的任务进度规划，而师生日志概率差距反映的是模型偏好的转变，而非真实的任务进展。为解决这些局限性，我们提出了子目标引导图正则化OPD（SGR-OPSD）框架，该框架结合了子目标监督与交互衍生证据，用于回合级学分分配。具体来说，SGR-OPSD使用LLM生成的子目标计划提供密集的行动级指导，并通过受限的基层正则化目标将其转换为优势权重。为基于观察到的结果获得功劳，我们构建了一个子目标进展图，估算抽样过渡频率的实证回报。该图比较经验行动回报与状态平均值，推导出校准教师学分权重的偏好。在ALFWorld、WebShop和Search-QA上，涵盖两个模型尺度（Qwen2.5-3B和7B）评估后，SGR-OPD始终优于GRPO和强的自蒸馏基线，成功率较GRPO提升了高达7.4个百分点，在QWEN2.5-7B的ALFWorld上达到85.8%。消融研究证实，子目标指导和基于图表的权重校准是这些性能提升的关键。

RLiD: Reinforcement Learning in the Decision Plane for Language Agents and LLMs

RLiD：语言代理和大型语言模型决策平面中的强化学习

• Authors: Ziyi Chen, Yan Zhang, Jianhui Wei, Jianing Yin, Daoan Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02455
• Pdf link: https://arxiv.org/pdf/2611.02455
• Abstract Reinforcement learning (RL) has improved the capabilities of large language models (LLMs) across a wide range of tasks. However, token-level RL assigns an action's advantage to the entire generated response, including tokens specifying the parsed action, reasoning chains, and malformed content. Although the environment executes only the parsed action, variation in how the action is expressed in natural language can introduce additional variance into policy-gradient estimates. We propose RLiD (Reinforcement Learning in the Decision plane), a framework that directly optimizes the probabilities of executable action options. RLiD compiles observations, an optional compact memory, and a labeled action menu into the model input, and uses a large decision model (LDM) to select an action in a single forward pass. By eliminating autoregressive token generation and bypassing action parsing, RLiD avoids invalid action outputs and improves RL rollout efficiency. Our analysis shows that, within a fixed text policy, marginalizing over alternative realizations of the same action removes a zero-mean gradient component without changing the expected update. We further distill trained LDM policies into generative LLMs through supervision on their action trajectories. Experiments on ARC, Countdown, and Sokoban demonstrate improved performance or lower computational cost across several settings. On ALFWorld, the complete training and distillation pipeline reaches 99.3% success in 12.7 H100 GPU-hours, compared with 86.6% in 17.3 GPU-hours for token-level RL of the same LLM. With compiled memory, our strongest Qwen2.5-1.5B LDM and distilled generative LLM achieve 100% success on all 134 unseen and 140 seen ALFWorld evaluation games.
• 中文摘要 强化学习（RL）提升了大型语言模型（LLMs）在多种任务中的表现能力。然而，令牌级强化学习将动作优势分配给整个生成的响应，包括指定解析动作、推理链和畸形内容的标记。尽管环境只执行解析后动作，但动作在自然语言中表达方式的差异会为策略梯度估计引入额外变异。我们提出了RLiD（决策平面强化学习），这是一个直接优化可执行动作选项概率的框架。RLiD将观察值、可选的紧凑内存和带标签的动作菜单整合到模型输入中，并使用大型决策模型（LDM）在一次前向中选择动作。通过消除自回归令牌生成和绕过动作解析，RLiD避免了无效动作输出，提高了强化学习的展开效率。我们的分析显示，在固定文本策略下，对同一动作的替代实现进行边缘化，可以去除零均值梯度成分而不改变预期更新。我们进一步通过对其动作轨迹的监督，将训练的LDM策略提炼为生成LLMs。ARC、Countdown和Sokoban的实验显示，在多个设置下性能提升或计算成本降低。在ALFWorld上，完整的训练和蒸馏流水线在12.7小时H100 GPU小时内达到99.3%，而同一LLM的代币级强化学习在17.3小时内达到86.6%。通过编译内存，我们最强的Qwen2.5-1.5B级LDM和提炼生成LLM在所有134个未见和140个已见的ALFWorld评估游戏中均实现100%成功率。

Learning Purposeful Co-designs of Bio-inspired Quadrupedal Robots

学习有目的的仿生四足机器人共同设计

• Authors: Fabio Boekel, Jiatao Ding, Cosimo Della Santina
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.02462
• Pdf link: https://arxiv.org/pdf/2611.02462
• Abstract Treating morphology and control as separate design problems has constrained the efficiency and adaptability of quadrupedal robots, while current co-design approaches are often constrained by restricted design spaces. We overcome this limit by introducing Dogger, a bio-inspired quadruped with modular three-segment legs, four-bar linkages and adjustable passive knee compliance. A two-stage framework combining reinforcement learning and Bayesian optimization is developed: the morphology-conditioned control policy is trained across designs and then held fixed during task-specific design selection. The resulting designs and policies deployed zero-shot reduce the cost of transport by 18.6% and 35.7% for inspection and payload-carrying tasks. Comparisons demonstrate that full geometry-compliance co-design outperforms fixed and restricted design scopes. Cross-task tests further reveal distinct task-specific morphologies with shared mechanical trade-offs. These results provide hardware-level evidence that expanding the physical design space alongside learned control is essential to realize the full potential of task-driven morphology-control co-design in legged robotics.
• 中文摘要 将形态和控制视为独立设计问题限制了四足机器人的效率和适应性，而当前的共创方法常受限于有限的设计空间。我们通过引入Dogger克服了这一限制，这是一种仿生的四足机器人，配备模块化三段腿、四杆连杆和可调节的被动膝关节合规性。开发了一个结合强化学习和贝叶斯优化的两阶段框架：形态条件控制策略在不同设计间训练，并在任务特定设计选择时保持固定。最终的零射程设计和策略降低了18.6%的运输成本，检查和有效载荷任务的运输成本降低了35.7%。比较显示，完全几何合规的协同设计优于固定和受限设计范围。跨任务测试进一步揭示了具有共享机械权衡的任务特异形态。这些结果提供了硬件层面的证据，表明在学习控制的同时扩展物理设计空间对于实现腿式机器人中任务驱动形态-控制协同设计的全部潜力至关重要。

To Coordinate or Not to Coordinate? Learning to Swim with Eight Soft Arms

协调还是不协调？用八只软臂学游泳

• Authors: Junkai Chen, Dehan Yuan, Yiyuan Zhang, Yu Duan, Fan Shi, Dixia Fan, Cecilia Laschi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.02468
• Pdf link: https://arxiv.org/pdf/2611.02468
• Abstract Does effective swimming with multiple underactuated soft arms require explicit coordination between arms? We hypothesize that an eight-arm swimmer can control its swimming speed and direction by independently learning each arm's actuation amplitude under a shared rhythmic drive, without prescribed inter-arm phase coupling. To test this hypothesis, we compare three reinforcement learning-based control strategies using proximal policy optimization (PPO): (1) learning arm-specific actuation amplitudes without inter-arm phase coupling, (2) learning actuation amplitudes with prescribed inter-arm phase coupling, and (3) directly learning actuator commands through end-to-end PPO without an explicit rhythmic structure. Across 200 episodes with randomized three-dimensional velocity commands, the controller without phase coupling satisfies the tracking criterion in 169 episodes, compared with 117 for the phase-coupled controller and 28 for end-to-end PPO. Straight-line swimming and commanded direction-change trials further demonstrate more accurate velocity tracking without phase coupling. These findings suggest that effective eight-arm swimming does not require prescribed inter-arm phase coupling, but can instead be achieved through learned arm-specific amplitude modulation within a rhythmic control structure.
• 中文摘要 多臂软臂的弱驱动有效游泳是否需要双臂间的明确协调？我们假设，八臂游泳者可以通过独立学习每只手臂在共享节律驱动下的驱动幅度来控制游泳速度和方向，而无需规定的臂间相位耦合。为验证该假设，我们比较了三种基于强化学习的近端策略优化控制策略：（1）学习手臂特定的执行幅度而不进行臂间相位耦合，（2）通过规定的臂间相位耦合学习执行振幅，（3）通过端到端PPO直接学习执行器指令，无需明确的节奏结构。在200次随机三维速度指令的事件中，无相位耦合控制器在169次中满足跟踪标准，而相位耦合控制器为117次，端到端PPO为28次。直线游泳和指令方向变化试验进一步展示了在无相位耦合的情况下更准确的速度跟踪。这些发现表明，有效的八臂游泳不需要规定的臂间相位耦合，而是可以通过在节律控制结构内学习的手臂特异幅度调制实现。

RLVR Sharpens the Pencil, Not the Mind

RLVR削铅笔，不是磨脑

• Authors: Shreeyam Bangera, Manya Gupta
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.02574
• Pdf link: https://arxiv.org/pdf/2611.02574
• Abstract Models like DeepSeek-R1 made Reinforcement Learning with Verifiable Rewards (RLVR) famous: reward a model for correct answers and it gets dramatically better at reasoning without being taught how. The leading explanation is sharpening---that the base model already knows how to solve these problems and RLVR just makes it more likely to reach for its best solution paths. This has only ever been checked by comparing before and after training, never by watching what changes inside across it. We tracked OLMo-2-1B across all 13 training checkpoints on GSM8K. First-try accuracy climbs from 58.0% to 61.3%, but ask the model 256 times and solution coverage actually rises from 94.5% to 96.9%, proving the knowledge was already there. We then planted reasoning mistakes into problems and trained simple probes to detect them. Those probes held between AUROC 0.798 and 0.840 across all 13 checkpoints and 3 error types, a spread smaller than the confidence interval; the model never builds a sharper internal picture of what a reasoning mistake looks like. The one place we find real internal change is at the final answer token, where a probe distinguishing early from late checkpoints hits an AUROC of 0.93, while the same probe on the actual reasoning steps lands near chance. RLVR makes the model lead with its best thinking. It does not change what that thinking is.
• 中文摘要 像DeepSeek-R1这样的模型让可验证奖励强化学习（RLVR）闻名：通过正确回答奖励模型，模型在不教导如何推理的情况下大幅提升推理能力。主要解释是锐化---基础模型已经知道如何解决这些问题，而RLVR只是让它更有可能找到最佳解法路径。这只通过训练前后比较来验证，从未通过观察训练内部的变化。我们在GSM8K上追踪了OLMo-2-1B的13个训练检查点。首次尝试的准确率从58.0%上升到61.3%，但问模型256次后，解答覆盖率实际上从94.5%上升到96.9%，证明知识早已存在。随后我们将推理错误植入问题中，并训练简单探针来检测这些错误。这些探测在所有13个检查点和3种错误类型中保持在AUROC 0.798到0.840之间，差距小于置信区间;模型从未构建出更清晰的内部推理错误图景。我们发现真正的内部变化是在最终答案标记处，早期和晚期检查点的探测显示AUROC为0.93，而实际推理步骤的探测则接近偶然。RLVR让模型以最佳思维领先。但它不改变这种思维。

Beyond $L_\infty$-Norm Sets: Synthesizing Interpretable Non-Uniform Uncertainty Sets for Evaluation-Time State-Adversarial Robustness

超越$L_\infty$-范数集：综合可解释的非一致不确定性集合以求评估时间状态-对抗性鲁棒性

• Authors: Xulin Chen, Ruipeng Liu, Garrett E. Katz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.02576
• Pdf link: https://arxiv.org/pdf/2611.02576
• Abstract Deep reinforcement learning (DRL) agents are vulnerable to perturbations of their observed state inputs. Prior works often constrain state adversaries with $L_p$-norm uncertainty sets and evaluate policy performance under the resulting perturbations. However, in continuous control tasks like robotics control, different state dimensions have distinct physical meanings and influence policy behavior to different degrees, which $L_p$-norm sets fail to capture or explain. Moreover, it is important to identify when and along which state dimensions a policy is vulnerable, independently of the algorithm used to train it. In this paper, we focus on evaluation-time attacks applied to pretrained policies. We propose \textbf{P}erformance-\textbf{D}egradation \textbf{N}on-\textbf{U}niform \textbf{U}ncertainty \textbf{S}et (\method{}), an interpretable formulation of uncertainty sets for state perturbations. \method{} is state-dependent, dimension-wise non-uniform, and can be synthesized online during policy evaluation. We first formulate an ideal constrained optimization problem for \method{} computation, then introduce a surrogate constraint to make the problem tractable. Experiments across multiple continuous-control environments, policy algorithms, and state attacks show that \method{} reveals variation in policy sensitivity across states and dimensions, while its performance degradation is moderately controlled.
• 中文摘要 深度强化学习（DRL）智能体容易受到其观察到状态输入的扰动。以往的研究常常用 $L_p$-范数不确定性集约束状态对手，并在扰动下评估策略表现。然而，在机器人控制等连续控制任务中，不同的状态维度具有不同的物理含义，并以不同程度影响策略行为，而 $L_p$-范数集无法捕捉或解释这些。此外，识别策略何时以及沿哪些状态维度易受攻击也很重要，这与用于训练策略的算法无关。本文重点关注应用于预训练策略的评估时间攻击。我们提出 \textbf{P}erformance-\textbf{D}降级 \textbf{N}on-\textbf{U}niform \textbf{U}ncertainty \textbf{S}et （\method{}），这是一种状态扰动不确定性集的可解释表述。\method{} 依赖状态，维度上不均匀，可在策略评估时在线合成。我们首先构建一个理想约束优化问题用于 \method{} 计算，然后引入替代约束使问题可解。在多个连续控制环境、策略算法和状态攻击中的实验表明，方法{}在状态和维度间显示策略敏感性变化，而其性能下降则被适度控制。

Whitened Advantage Summation Policy Optimization: Correlation-Aware Aggregation for Multi-Reward Reinforcement Learning

白化优势总和策略优化：多奖励强化学习中的相关性感知聚合

• Authors: Olaitan Olaleye, Jayson Meribe, Temidayo Olaleye
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02609
• Pdf link: https://arxiv.org/pdf/2611.02609
• Abstract Reinforcement learning for language models often combines several rewards by normalizing each one and adding them with fixed weights. When the rewards are correlated, the variation they share is counted once for every reward that carries it. In terms of the ZCA-decorrelated components of the reward signal, the weighted sum applies weights $\Sigma^{1/2} w$ rather than the chosen weights $w$, where $\Sigma$ is the reward correlation matrix. We study WAS-PO, which whitens the normalized advantages with $\Sigma^{-1/2}$ before adding them, so that the chosen weights apply to those components. We derive the scalarized objective whose gradient it follows when normalization statistics are held fixed, show that its weights on the raw rewards can be negative, give an exact two-reward condition under which the lighter-weighted reward is reversed, and give a loose bound on the error from estimating $\Sigma$ during training. On tool calling with Qwen2.5-1.5B-Instruct, WAS-PO reached a higher mean training reward than GDPO in all seven seeds with three rewards (3.8% higher) and all three seeds with four (3.9% higher). WAS-PO also replaces GDPO's per-prompt normalization with batch-level normalization, and we did not separate the two changes with three or four rewards. The training-reward gains did not carry over to held-out accuracy, which was slightly lower than GDPO's with three rewards (not significant) and 0.03 lower in absolute terms with four (two seeds evaluated). With two rewards, WAS-PO trailed GDPO in training reward, and a three-seed control weakly suggests that the gap comes from batch-level normalization rather than whitening. In a math setting where the two rewards were nearly identical by construction, WAS-PO drove down the lighter-weighted reward, as the sign condition predicts.
• 中文摘要 语言模型的强化学习通常通过归一化多个奖励并加固定权重来组合。当奖励相关时，它们共享的变异对每个携带该奖励的奖励计数一次。根据ZCA去相关分量的奖励信号，加权和赋予权重$\Sigma^{1/2} w$，而非选定权重$w$，其中$\Sigma$为奖励相关矩阵。我们研究WAS-PO，该方法在添加前将归一化优势与$\Sigma^{-1/2}$美白，使所选权重适用于这些组成部分。我们推导了标量化目标，其梯度在固定归一化统计时，证明其对原始奖励的权重可能为负，给出一个精确的二奖励条件，使得较轻的加权奖励被反转，并给出训练中估计$\Sigma$时误差的宽限。在Qwen2.5-1.5B-Ininstruction工具调用中，WAS-PO在所有七个种子中均均训练奖励高于GDPO（3个奖励高3.8%），三个种子均高于四个（高3.9%）。WAS-PO还用批次级归一化替代了GDPO的每次提示归一化，且我们未将三或四个奖励的两个变化分离。训练奖励的收益未转化为保留准确率，后者略低于三项奖励的GDPO（无显著性），绝对值低0.03（四个奖励，评估两个种子）。WAS-PO在训练奖励中落后于GDPO，三种子控制弱暗示差距来自批量归一化而非白化。在数学环境中，两奖励结构几乎相同，WAS-PO降低了较轻加权奖励，正如符号条件预测的那样。

Auditing Representation-Space Fraud Detection: A Bayesian IRL Framework for Calibrated Uncertainty and Its Limits

审计表示空间欺诈检测：用于校准不确定性及其局限的贝叶斯IRL框架

• Authors: Shivam Swarup, Divya Prakash Shrivastava, Rakesh Thakur
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02673
• Pdf link: https://arxiv.org/pdf/2611.02673
• Abstract Shilling detectors for recommender systems are almost always evaluated by one number, detection AUROC, with no account of when it can be trusted. We audit representation-space detection with VEIL (Variational Evaluation via Inverse-reinforcement Learning), a pairwise (Bradley-Terry) Bayesian reward-inference framework over frozen encoder embeddings, following the Bayesian auditing paradigm for LLM alignment [1]: (A) a base posterior; (B) a hierarchical ARD extension with learned confidence; and (C) a label-routed organic/attacker mixture used as an evidence-accumulation this http URL prove (Proposition 1) that no detector built on a representation invariant to a transformation of user profiles can outperform chance against an attack that is indistinguishable from organic profiles under that this http URL is near-perfect against synthetic-structure attacks (AUROC 0.936-1.000) but collapses under donor-mimicry, which clones a real user's item set and splices in 1-3 target items. Performance falls to chance on MovieLens-1M (0.48-0.51), partially degrades on Amazon Musical Instruments, and degrades least on this http URL-mimicry is not undetectable: a non-learned baseline, based on each profile's maximum raw item-set Jaccard similarity to any other profile, reaches AUROC 0.92-0.97 (average precision 0.50-0.57) on all four donor-mimicry configurations, including ML-1M, where every embedding-based score is at this http URL signal is present in the raw profiles, but LightGCN's representation discards it, placing injected profiles closer to their true donor than its zero-propagation MF reduction does (distance 1.98 vs. 3.09). Severity varies by dataset, encoder, and detection method, with no single axis predicting the others. The limit lies in the representation, not the attack, and raw-similarity checks are a necessary complement to any learned detector.
• 中文摘要 推荐系统中的先令检测器几乎总是通过一个数字——检测AUROC来评估，且不考虑何时可信。我们用VEIL（通过逆强化学习进行变分评估）审计表示空间检测，这是一种基于冻结编码器嵌入的两两（Bradley-Terry）贝叶斯奖赏-推断框架，遵循LLM对齐的贝叶斯审计范式[1]：（A）基础后验;（B）带有学习信心的分层ARD扩展;以及（C）一种标签路由的有机/攻击者混合物，作为证据积累，该http URL证明（命题1）任何基于对用户配置文件转换不变表示的检测器，都无法在与有机配置文件无异的攻击中胜过偶然，因为该HTTP地址对合成结构攻击几乎完美无异（AUROC 0.936-1.000），但在供体模仿下崩溃，该模拟克隆真实用户的物品集并在1-3个目标项目中拼接。在MovieLens-1M上性能依赖运气（0.48-0.51），在亚马逊乐器上部分退化，而在该http上下降最小。URL拟态并非不可检测：基于每个配置文件最大原始条目集Jaccard相似度的非学习基线，在所有四种供体拟态配置（包括ML-1M）上均达到AUROC 0.92-0.97（平均精度0.50-0.57），其中所有基于嵌入的分数均存在于原始拟态中，但LightGCN的表示舍弃了该信号，使注入的配值比其零传播MF减少更接近其真实供体（距离1.98对3.09）。严重度因数据集、编码器和检测方法而异，没有单一轴预测其他轴。限制在于表征本身，而非攻击，原始相似性检查是任何学习检测器的必要补充。

Reap: Uncovering Monte Carlo proof search with a lightweight model

收割：用轻量级模型揭示蒙特卡洛证明搜索

• Authors: Yutong Wang, Xuanzhi Ren, Tianyi Xu, Ruitong Liu, Jingda Xu, Peihao Wu, Bryan Dai, Bin Dong
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02678
• Pdf link: https://arxiv.org/pdf/2611.02678
• Abstract We present Reap, an open framework for learning and analyzing formal proof search in Lean4. Reap implements policy-value-guided Monte Carlo Tree Search directly in Lean, sharing proof-state transitions, AND-OR search, and replay verification between inference and training rollouts. Structured action logs expose how neural predictions and tactic execution interact during search. Using this framework, we train Reaper-1, a 1.7B-parameter policy-value model, through continued pre-training, supervised tactic learning, and iterative search-based reinforcement learning. Using a separate premise selector, the final system solves 79.9% of miniF2F-test at pass@32, establishing state-of-the-art performance among open provers of comparable size while remaining competitive with substantially larger models. Beyond improved solve rates, our results suggest that RL strengthens the prover's ability to organize proofs around useful intermediate results, favoring substantial advances over repeated local adjustments. This more directed proof construction is accompanied by increasingly elaborate tactic programs: fewer explicit search decisions are needed, while more work is performed within each decision. The resulting trade-off between search effort and tactic execution cost provides insight into how learning changes the prover's problem-solving strategy.
• 中文摘要 我们介绍Reap，一个开放框架，用于学习和分析Lean4中的形式证明搜索。Reap直接在Lean中实现了策略值引导的蒙特卡洛树搜索，共享证明状态转换、AND-OR搜索以及推理与训练推广之间的重放验证。结构化动作日志揭示了神经预测与战术执行在搜索过程中的相互作用。利用该框架，我们通过持续的预训练、监督战术学习和迭代基于搜索的强化学习，训练Reaper-1，一个1.7B参数的策略-价值模型。通过独立的前提选择器，最终系统在pass@32下解决了79.9%的miniF2F测试，在同等规模的开放证明者中建立了最先进的性能，同时保持与更大模型的竞争力。除了提升求解率外，我们的结果表明，强化学习增强了证明者围绕有用中间结果组织证明的能力，优先于显著进步而非反复局部调整。这种更具导向性的证明构建伴随着越来越复杂的战术程序：需要更少的显式搜索决策，而每个决策内的工作量增加。搜索努力与战术执行成本之间的权衡，有助于理解学习如何改变证明者解决问题的策略。

Rose: Efficient Embodied RL through Graph Optimization and GPU Compute Resource Control

Rose：通过图优化和GPU计算资源控制实现高效的内涵强化学习

• Authors: Wen Zeng, Ke Hong, Xu Fu, Zixiao Huang, Zehao Wang, Zhuoran Zhu, Yuanqing Wang, Bo Dai, Zhen Guo, Hao Lin, Chunyang Zhu, Boxun Li, Chao Yu, Quanlu Zhang, Guohao Dai, Zhenhua Zhu, Yu Wang
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2611.02705
• Pdf link: https://arxiv.org/pdf/2611.02705
• Abstract Embodied reinforcement learning (RL) coordinates policy inference, simulation, and model training with highly heterogeneous resource demands. Although individual components frequently underutilize GPU compute during execution, existing systems only overlap work at coarse component boundaries (e.g., pipelining inference and simulation across simulator groups), leaving substantial low-activity GPU windows within subcomponent phases and nested loops. Exploiting these fine-grained windows manually requires tedious dependency analysis and workload-specific coordination. Furthermore, even when independent work is scheduled concurrently, cross-process GPU time slicing and streaming multiprocessor (SM) contention can negate the benefits of overlap. We present Rose, a runtime-aware system that automates fine-grained overlap and GPU resource management for embodied RL while preserving synchronous training semantics. Through a lightweight programmable interface that exposes execution hierarchy and state accesses, Rose tracks runtime data readiness and GPU activity to perform a two-stage optimization. First, a window-guided graph optimizer identifies low-activity windows and applies graph transformations to overlap complementary work across components and loop levels. Second, leveraging NVIDIA Multi-Process Service (MPS) for cross-process GPU concurrency, a slack-aware resource manager tunes process-level SM quotas by exploiting branch synchronization slack to mitigate interference on the critical path. Both stages combine local profiling with end-to-end iteration validation to guarantee throughput gains with minimal tuning overhead. Across diverse policies, simulators, RL algorithms, and GPU platforms, Rose achieves 1.20--1.92$\times$ training throughput improvement and increases average SM activity by 23.8--39.3\% compared with the state-of-the-art systems.
• 中文摘要 具身强化学习（RL）协调策略推断、仿真和模型训练，同时应对高度异构的资源需求。尽管单个组件在执行时常常未能充分利用GPU计算，现有系统仅在粗分组件边界上工作重叠（例如，推理和仿真在模拟器组间进行流水线），导致子组件阶段和嵌套循环中存在大量低活跃的GPU窗口。手动利用这些细粒度窗口需要繁琐的依赖分析和工作负载特定的协调。此外，即使独立工作同时调度，跨进程GPU时间切片和流多处理器（SM）争用也可能抵消重叠带来的好处。我们介绍Rose，这是一个运行时感知系统，能够自动化细粒度重叠和GPU资源管理，同时保持同步训练语义。通过一个轻量级可编程接口，该接口暴露执行层级和状态访问，Rose 跟踪运行时数据准备度和 GPU 活动，进行两阶段优化。首先，窗口引导的图优化器识别低活动窗口，并应用图变换以实现组件和环路层级的互补工作。其次，利用 NVIDIA 多进程服务（MPS）实现跨进程 GPU 并发，一个具 slack 感知的资源管理器通过利用分支同步松弛来调优进程级 SM 配额，以减少关键路径上的干扰。这两个阶段结合了本地分析和端到端迭代验证，确保吞吐量提升且调优开销最小。在多种策略、模拟器、强化学习算法和GPU平台上，Rose实现了1.20至1.92美元/时间$的训练吞吐量提升，并使平均SM活动提升了23.8%至39.3%的水平，相比最先进系统。

Hindsight Credit Assignment over Trees for Agentic RL

事后诸葛亮的信用分配，适用于代理强化学习的树

• Authors: Penghong Zhao, Yihao Chen, Maoheng Yuan, Yunjian Zhang, Abudukelimu Wuerkaixi, Haoran Xu, Man Lv, Yaolong Wang, Zhijie Sun, Guohua Liu, Yuewei Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02706
• Pdf link: https://arxiv.org/pdf/2611.02706
• Abstract Reinforcement learning for long-horizon LLM agents faces a fundamental dilemma: successful experiences are essential for policy improvement, yet limited policy capabilities make such experiences difficult to discover. Sparse outcome rewards compound this challenge, leaving sampled trajectories with little relative feedback for learning. In this paper, we propose TriGRPO, a tree-based reinforcement learning framework that turns scarce successes into exploration and learning signals through hindsight credit assignment within and across rollout trees. TriGRPO constructs rollout trees through hindsight-guided branching: when an ongoing trajectory reaches a matched decision state, a successful action discovered during rollout initiates a sibling branch that continues autonomously, enabling further exploration and parent-sibling comparisons. Three complementary groups organize policy optimization over these trees: root groups capture within-prompt relative returns, node groups assign credit from parent-sibling outcome differences, and cross groups supplement sparse feedback with historical performance on related tasks. Experimental results demonstrate consistent gains in task performance and rollout efficiency across model scales, highlighting structured hindsight credit assignment as an effective approach to long-horizon agent learning.
• 中文摘要 长期LLM代理的强化学习面临一个根本困境：成功的经验对策略改进至关重要，但有限的策略能力使得此类体验难以发现。稀疏的结果奖励加剧了这一挑战，使得抽样轨迹几乎没有相对学习反馈。本文提出了TriGRPO，一种基于树的强化学习框架，通过事后视角的学分赋值，在推广树内及跨树间将稀缺成功转化为探索和学习信号。TriGRPO通过事后视角引导分支构建滚出树：当持续的轨迹达到匹配决策状态时，在推广过程中发现的成功动作启动一个自主继续的兄弟分支，从而促进进一步探索和父子姐妹比较。三个互补组对这些树进行策略优化：根组捕捉即时内的相对回报，节点组根据父-兄弟姐妹结果差异分配功劳，跨组则用相关任务的历史表现补充稀疏反馈。实验结果显示，任务表现和推广效率在模型尺度上持续提升，强调结构化事后诸葛亮功劳分配是长期代理学习的有效方法。

AffToM: Benchmarking Recursive Affective Theory of Mind in Dynamic Multi-Party Interactions

AffToM：动态多方互动中递归情感心智理论的基准测试

• Authors: Jinpeng Hu, Yiduo Jin, Erqiang Wang, Jia Li, Jingjing Wu, Peipei Song
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02725
• Pdf link: https://arxiv.org/pdf/2611.02725
• Abstract Understanding emotion in realistic social interactions requires more than recognizing facial expressions or assigning affective labels. The same event can elicit different emotions depending on what each participant knows, what they believe others know or feel, and how these nested mental states evolve over time. Existing benchmarks largely study higher-order epistemic Theory of Mind (ToM) and multimodal emotion reasoning in isolation, leaving recursive affective reasoning under asymmetric and dynamically changing information underexplored. Therefore, we introduce AffToM, a benchmark for affective ToM with 1,922 test pairs and 19,759 training pairs. AffToM evaluates Individual Affective Reasoning (IAR), which infers emotions from events and socio-cognitive context, and Multi-party Affective Reasoning (MPAR), which recursively tracks participants' nested beliefs and emotions as new evidence emerges. We develop an event-driven multi-agent generation pipeline grounded in dynamic character interaction graphs that explicitly capture event sequences, interpersonal relations, and mental-state updates. Furthermore, we propose AffToM-R1, a two-stage post-training framework combining supervised fine-tuning with perspective-aware rubric-guided reinforcement learning. Experiments show that MPAR remains challenging, with even the strongest evaluated MLLM achieving less than 60% accuracy, while AffToM-R1 improves MPAR accuracy by 13.38 percentage points over its base model. Our dataset is available at: this https URL.
• 中文摘要 理解现实社会互动中的情绪不仅仅是识别面部表情或赋予情感标签。同一事件可能引发不同情绪，取决于每个参与者所知、他们认为他人知道或感受的，以及这些嵌套心理状态如何随时间演变。现有基准大多孤立研究高阶认知理论（ToM）和多模态情绪推理，导致递归情感推理在非对称且动态变化的信息下尚未被充分探讨。因此，我们引入AffToM，这是一个情感ToM基准，包含1922对测试对和19759对训练对。AffToM评估个体情感推理（IAR），即从事件和社会认知情境中推断情绪，以及多方情感推理（MPAR），递归追踪参与者嵌套的信念和情绪，随着新证据的出现。我们开发了一个事件驱动的多智能体生成流水线，基于动态角色交互图，明确捕捉事件序列、人际关系和心理状态更新。此外，我们提出了AffToM-R1，一个两阶段的训练后框架，结合了监督微调与视角感知的评分标准引导强化学习。实验显示，MPAR依然具有挑战性，即使是最强的MLLM评估准确率也低于60%，而AffToM-R1则将MPAR准确率提升13.38个百分点，比其基础模型高出13.38个百分点。我们的数据集可在：此 https URL。

What Drives the Zero-to-One Jump? Diagnosing and Steering Hard-Prompt Learning in RLVR

是什么驱动了从零到一的跳跃？诊断和引导RLVR中的硬提示学习

• Authors: Rui Zhu, Yumou Liu, Zhenzhe Zheng, Fan Wu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.02747
• Pdf link: https://arxiv.org/pdf/2611.02747
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved mathematical reasoning, yet progress on hard prompts often stalls: a prompt can remain unsolved for many training steps before abruptly transitioning to solved--a discrete "zero-to-one" jump whose underlying dynamics and steerability remain poorly understood. We investigate two questions: what drives this transition, and can it be steered? Through event-based analysis measuring the time-to-first-positive (TTFP), we find that negative rollouts preceding successful transitions consistently retain higher early-stage generation entropy, distinguishing near-solvable failures from dead ends. This signal motivates NAPO (Negative-Augmented Policy Optimization), a method integrated with Group Relative Policy Optimization (GRPO) that ranks negatives in unsolved groups via an early-branching driver and assigns graded advantages, injecting informative gradients precisely where standard group-relative updates vanish. Under fixed rollout budgets, NAPO accelerates early solution discovery and improves Pass@256 by 10.0 points on AIME24 and 6.6 points on AIME2025, demonstrating that zero-to-one transitions are diagnosable and steerable through driver-guided negative learning. Our code is available at this https URL.
• 中文摘要 可验证奖励强化学习（RLVR）显著提升了数学推理能力，但困难提示的进展常常停滞：提示可能在许多训练步骤中未解，随后突然转变为解决——这是一种离散的“零到一”跳跃，其潜在动力学和可引导性仍不明确。我们探讨两个问题：是什么驱动了这一转变，它能否被引导？通过事件分析测量首次正向时间（TTFP），我们发现成功转变前的负向推广持续保留较高的早期生成熵，区分了近乎可解的失败与死胡同。这一信号激励了NAPO（负增强策略优化），这是一种与组相对策略优化（GRPO）集成的方法，通过早期分支驱动对未解决组的负数进行排名，并分配分级优势，在标准组相对更新消失处注入信息梯度。在固定的部署预算下，NAPO加快了早期解决方案发现，并在AIME24上提升了10.0分，Pass@256 AIME2025上提升了6.6分，证明了通过驱动驱动的负向学习，零到一的转变是可诊断和引导的。我们的代码可在此 https 网址获取。

Beyond Policy Patches: Distilling RL Specialists into Vision-Language-Action Policies

超越政策补丁：将强化学习专家提炼为愿景-语言-行动政策

• Authors: Zhennan Jiang, Tingying Xie, Shilong Ji, Shu'ang Yu, Tianxing Zhou, Yuhui Chen, Jiahuan Gao, Kaizhi Xie, Yuanfan Xu, Feng Gao, Yu Wang, Haoran Li, Dongbin Zhao, Chao Yu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.02777
• Pdf link: https://arxiv.org/pdf/2611.02777
• Abstract Vision-language-action (VLA) policies can handle noncritical stages of long-horizon manipulation yet fail during short precision-critical phases. Lightweight reinforcement learning (RL) specialists can address these failures without directly applying RL to the full VLA, but retain auxiliary controllers and switching mechanisms at deployment. We study how to transfer their capabilities into a single flow-matching VLA. Direct imitation of specialist rollouts can reproduce action jitter and leave the VLA unprepared for states encountered during its own execution. We introduce REPAIR, a value-guided distillation framework that combines offline initialization with on-policy refinement. At each critical-phase observation, REPAIR queries the corresponding specialist for multiple action candidates and weights them using its critic to construct a supervision distribution. A shared weighted flow-matching objective trains the VLA first on observations from specialist-assisted rollouts, then on observations collected through its own execution. Specialists provide supervision without taking control during on-policy collection and are removed at deployment. Across six real-robot tasks with 30 evaluation trials per task, REPAIR achieves 100% success on all six tasks, with lower mean end-effector jerk than specialist-assisted execution on every task.
• 中文摘要 视觉-语言-动作（VLA）策略可以处理非关键的长视距操作阶段，但在短时间的精度关键阶段失败。轻量级强化学习（RL）专家可以在不直接将强化语言应用于完整VLA的情况下解决这些失败，同时保留部署时的辅助控制器和切换机制。我们研究如何将这些能力转移到单一的流量匹配VLA中。直接模仿专家的部署可以重现动作抖动，使VLA在自身执行时遇到的状态时未做好准备。我们引入了REPAIR，一种价值引导的提炼框架，结合了离线初始化与策略内细化。在每个关键阶段观测中，REPAIR会查询对应专家的多个动作候选，并利用其批评者加权以构建监督分布。共享加权流量匹配目标首先训练VLA从专家协助的推广观察中获得的观测数据，然后再根据自身执行收集的观测数据进行训练。专家在政策收集过程中提供监督但不控制，部署时被撤离。在六个真实机器人任务中，每个任务包含30个评估试验，REPAIR在所有六项任务中均实现100%成功率，且平均末端效应抖动低于专家协助执行。

OmniOPD: Stable On-Policy Distillation for Joint Audio-Video Generation

OmniOPD：联合音视频生成的稳定策略提炼

• Authors: Zihao Fan, Xin Lu, Jie Huang, Hexin Zhang, Xueyang Fu, Zheng-Jun Zha
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2611.02803
• Pdf link: https://arxiv.org/pdf/2611.02803
• Abstract Reinforcement learning (RL) elevates joint audio-video generation, but optimizing heterogeneous, modality-specific audio and video rewards often introduces cross-modal interference. On-policy distillation (OPD) alleviates this issue by transferring the capabilities of specialized audio and video RL teachers into a single student model. However, a significant imbalance in cross-modal token counts leaves audio with sparse supervision, causing unstable distillation. We observe that naively increasing the audio loss weight fails to solve this problem since audio learning relies heavily on visual context, indicating richer visual cues are crucial to offset sparse audio signals. To address this, we propose OmniOPD, a modality-wise on-policy distillation framework designed for balanced and stable audio-video generation. First, we construct privileged visual information using asymmetric noise scheduling, allowing the audio teacher to generate more informative targets from cleaner video context. Second, we introduce grouped multi-noise sampling to expand audio supervision coverage and stabilize training. Third, we introduce timestep-wise importance weighting based on teacher-student transition divergence, preserving reliable supervision while down-weighting unreliable learning signals, to stabilize fluctuations in audio learning. Evaluated on JavisBench and VBench demonstrate OmniOPD preserves audio quality while significantly enhancing video generation, achieving effective joint distillation.
• 中文摘要 强化学习（RL）提升了联合音视频生成，但优化异构、特定模态的音频和视频奖励常常引入跨模态干扰。策略上提炼（OPD）通过将专业音频和视频强化学习教师的能力转移到单一学生模型中，缓解了这一问题。然而，跨模态令牌数量的显著失衡导致音频受到稀疏监督，导致蒸馏不稳定。我们观察到，单纯提高音频损失权重无法解决这一问题，因为音频学习高度依赖视觉上下文，表明更丰富的视觉线索对于抵消稀疏音频信号至关重要。为此，我们提出了OmniOPD，一个按模态划分的策略上提纯框架，旨在实现平衡且稳定的音视频生成。首先，我们利用非对称噪声调度构建特权视觉信息，使音频教师能够从更清晰的视频上下文中生成更具信息量的目标。其次，我们引入了分组多噪声采样，以扩大音频监督覆盖范围并稳定训练。第三，我们引入基于师生过渡分歧的时间步重要性加权，保持监督的可靠性，同时降低不可靠的学习信号权重，以稳定音频学习的波动。在JavisBench和VBench上的评估表明，OmniOPD既保持音频质量，又显著提升视频生成，实现了有效的联合蒸馏。

Offline Approximate Value Iteration: A Reinforcement Learning Approach for Satellite Constellation Deployment

离线近似值迭代：卫星星座部署中的强化学习方法

• Authors: Tristan Leuridan (LGI), Yiping Fang (LGI), Adam Abdin (LGI)
• Subjects: Subjects: Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2611.02809
• Pdf link: https://arxiv.org/pdf/2611.02809
• Abstract This paper introduces a reinforcement learning-based approximate dynamic programming approach to optimize the deployment of Walker-type satellite constellations under uncertainties, including launch failures and satellite malfunctions. We formulate the problem as a Markov Decision Process that incorporates low-injection orbit launches, continuous low-thrust orbit raising, and variable satellite loads. We adopt the offline approximate value iteration solution method, using linear function approximation with basis functions and action pruning strategies to substantially reduce the computational and memory requirements for finding optimal decision policies. Preliminary results demonstrate convergence to policies with low expected costs and deployment times, both of which are lower than those of a rolling-horizon baseline, showcasing a novel application of approximate dynamic programming in space logistics.
• 中文摘要 本文引入了基于强化学习的近似动态规划方法，以优化Walker型卫星星座在不确定性（包括发射失败和卫星故障）下的部署。我们将问题表述为一个马尔可夫决策过程，包含低注入轨道发射、连续低推力轨道提升和可变卫星载荷。我们采用离线近似值迭代方法，利用线性函数近似配合基函数和动作修剪策略，大幅降低寻找最优决策策略的计算和内存需求。初步结果显示，这些策略与低预期成本和部署时间的策略收敛，均低于滚动视界基线，展示了近似动态规划在空间物流中的新颖应用。

An intelligent satellite must do nothing well

智能卫星绝不能做得好

• Authors: Fabrizio Pilone, Giordana Bucchioni
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.02922
• Pdf link: https://arxiv.org/pdf/2611.02922
• Abstract Autonomous inspection of a resident space object is usually posed as a contest between classical next-best-view planning and reinforcement learning. We argue that the choice of algorithm is the least informative of the design axes. An inspector spacecraft has no rest state: when it stops thrusting, orbital dynamics carry it along a natural trajectory that can be used to observe the target at no fuel cost. On a controlled ladder in which one design axis is varied at a time, fuel efficiency turns out to be set by three things: whether the agent is allowed to drift, whether its state records where drifting has taken it, and whether its planning horizon is long enough for the benefit to appear. Removing any one of them doubles the fuel bill or more. The algorithm still matters, but through what it can represent: a model-based agent whose learned world model ignores fuel spends several times more of it. Finally, no learned policy is safe by construction, and a control barrier function filter beneath the decision layer removed every keep-out-zone violation in two thousand evaluation episodes.
• 中文摘要 对驻留空间物体的自主检查通常被视为经典次优视图规划与强化学习之间的较量。我们认为算法的选择是设计轴中信息量最少的。检查航天器没有静止状态：当它停止推力时，轨道动力学会将其沿自然轨迹前进，可用于无燃料消耗地观察目标。在一个一次变化一个设计轴的受控梯子上，燃料效率由三点决定：智能体是否被允许漂移、其状态是否记录漂移的状态，以及其规划时间是否足够长以使收益显现。移除任一都会使燃料费用翻倍甚至更多。算法仍然重要，但其代表性在于：基于模型的智能体在其学习的世界模型中忽略燃料，其燃料消耗的量是燃料的数倍。最后，没有任何学习过的政策是从结构上安全的，决策层下方的控制屏障函数过滤器在两千次评估中消除了所有禁止区域违规。

Positive Safety Margins from Artificial Potential Field Reward Augmentation for Size-Robust Obstacle Avoidance under Partial Observability

人工势场奖励增强在部分可观测性下实现大小稳健障碍避让的正安全裕量

• Authors: Ziyi Hao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.02947
• Pdf link: https://arxiv.org/pdf/2611.02947
• Abstract Collision-free reaching is central to manipulation, yet real cells often leave the size and exact position of an obstacle unobserved, which excludes reinforcement learning controllers that take obstacle geometry as input. A reward limited to a collision penalty does not depend on the clearance outside the obstacle, so under a step cost its learned policy skims the obstacle surface and fails once the obstacle radius exceeds its training value. We investigate whether a single analytic reward term can induce a safety margin without exposing the obstacle to the policy. We augment the reward of a simulated Franka Emika Panda reaching task with a horizontal artificial potential field that penalizes proximity within a safety margin around each of two pillars (fixed nominal layout, random drift). Under hidden observation, policies trained with this term maintain a safe success rate above 88% across a radius sweep from 0.03 m to 0.055 m (30 seeds), while policies without it fall below 10% at the maximum tested radius; the term induces a minimum clearance of 47.9 mm against 9.6 mm without it and eliminates training collisions at least eight times earlier (median steps). The advantage is statistically significant at all out-of-distribution radii and holds across three reinforcement learning solvers and under full observation. Its safe success rate is comparable to that of a classical potential-field planner given the pillar positions. The margin is a fixed buffer rather than an adaptation to the radius, and it lengthens the path by about 19%.
• 中文摘要 无碰撞触及是操作的核心，但真实单元常常不观察障碍物的大小和确切位置，这排除了接受障碍几何作为输入的强化学习控制器。仅限于碰撞惩罚的奖励不依赖于障碍物外的净空，因此在阶梯成本下，其学习策略会掠过障碍表面，当障碍半径超过训练值时失效。我们研究单个分析奖励项是否能在不暴露障碍物于策略下诱导安全裕度。我们在模拟Franka Emika Panda到达任务的奖励中，增加了一个水平人工势场，惩罚两根支柱（固定名义布局、随机漂移）周围安全边际内的接近度。在隐藏观察下，采用该项训练的策略在0.03米至0.055米（30个种子）范围内的安全成功率保持在88%以上，而未使用该项的策略在最大测试半径处低于10%;该项诱导的最小净空为47.9毫米，而无该项则为9.6毫米，并且至少提前八倍消除训练碰撞（中位数步进）。该优势在所有非分布半径范围内具有统计学显著性，且在三个强化学习求解器和完全观察下均有效。其安全成功率可与经典潜在场地规划器相媲美，基于柱子位置。该余裕是固定缓冲区，而非半径的调整，且使路径延长约19%。

BLPO: Beyond-Local Policy Optimization for Long-Horizon LLM Agents

BLPO：面向长期大型语言模型代理的超本地策略优化

• Authors: Xuanren Chen, Chongyang Tao, Wenpeng Hu, Yong Wang, Bo Deng, Xinlei Zheng, Shuai Ma
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.03005
• Pdf link: https://arxiv.org/pdf/2611.03005
• Abstract Long-horizon tasks require large language model (LLM) agents to coordinate interdependent actions, a capability central to automating complex workflows. Stepwise group-based reinforcement learning provides finer credit than trajectory-level supervision, but reliable comparisons must account for differences in interaction history. Enforcing this consistency within the limited rollouts of each task can leave insufficient support for estimating action preferences. We propose Beyond-Local Policy Optimization (BLPO), which estimates task-progress advantages from experience shared across tasks and training updates. We summarize trajectory prefixes into task-progress states and abstract actions by their function, allowing corresponding decisions from different tasks to contribute to shared estimates. To control bias and noise from heterogeneous tasks, we accumulate context-relative returns in a global memory with balanced aggregation and reliability weighting. These global advantages complement local comparisons over normalized actions and trajectory-level credit without a learned critic or additional rollouts. Experiments on ALFWorld and WebShop with 1.5B and 7B policies show improved task success over the evaluated baselines under matched training budgets. Further analyses show faster learning and shorter trajectories with little computational overhead.
• 中文摘要 长视野任务需要大型语言模型（LLM）代理协调相互依赖的动作，这是自动化复杂工作流的核心能力。基于阶段的基于群体的强化学习比轨迹级监督更为细致，但可靠的比较必须考虑交互历史的差异。在每个任务有限的部署范围内强制执行这种一致性，可能使估计动作偏好的支持不足。我们提出了超越局部策略优化（BLPO），它通过任务间共享的经验和训练更新估计任务进度优势。我们将轨迹前缀总结为任务进度状态，并按函数抽象动作，允许不同任务的相应决策贡献共享估计。为控制异构任务的偏见和噪声，我们在全局内存中积累上下文相对回报，并实现平衡聚合和可靠性加权。这些全球优势补充了本地比较，相较于归一化动作和轨迹级信用，无需专业批评者或额外推广。在采用1.5B和7B政策的ALFWorld和WebShop上的实验显示，在匹配训练预算下，任务成功率优于评估基线。进一步分析显示学习更快，轨迹更短，计算开销较低。

E-GenRM: Generative Reward Modeling for Open-Ended E-Commerce Shopping Agents

E-GenRM：面向开放式电商购物代理的生成奖励建模

• Authors: Ying Sun, Qiuyu Zhao, Xiao Wang, Xiyu Zhao, Gege Shi, Zenghui Sun, Jinsong Lan, Haihong Tang, Bo Zheng
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2611.03008
• Pdf link: https://arxiv.org/pdf/2611.03008
• Abstract We present E-GenRM (E-commerce Generative Reward Model), a generative reward model tailored to e-commerce shopping agents. User expectations are shifting from simple product queries to open-ended shopping tasks. However, existing trajectories and reward designs remain largely bounded by current agent usage patterns and known failures, providing limited guidance for further capability improvement. These limitations call for an integrated reward modeling framework for open-ended shopping agent tasks. On the data side, E-GenRM leverages naturally occurring user behavior across the shopping app to construct diverse, long-horizon conversational trajectories beyond established usage patterns. On the modeling side, E-GenRM first learns from behavior-grounded evaluation chains through supervised fine-tuning. It is then optimized through criterion-guided reinforcement learning, which rewards correct preference judgments and evidence-supported factual claims under each criterion. Personalized hard-negative training further improves its sensitivity to user preferences reflected in shopping behavior, beyond explicit requirements. We additionally develop an active evidence retrieval extension that acquires missing e-commerce information for evaluation. On E-GenRM-Bench, E-GenRM-27B achieves 87.40\% preference accuracy, outperforming the strongest baseline by 6.97 percentage points. In a 14-day online A/B test on Taobao, a shopping agent post-trained with E-GenRM improves the effective shopping-assistance rate by 1.39 percentage points and user satisfaction by 2.33 percentage points.
• 中文摘要 我们提出了E-GenRM（电子商务生成奖励模型），这是一种针对电子商务购物代理量身定制的生成奖励模型。用户期望正从简单的产品查询转向开放式购物任务。然而，现有的轨迹和奖励设计仍主要受限于当前代理使用模式和已知失败，为进一步能力提升提供了有限的指导。这些局限性促使开放式购物代理任务需要一个集成的奖励建模框架。在数据方面，E-GenRM利用购物应用中自然发生的用户行为，构建出超越既定使用模式的多样且长远的对话轨迹。在建模方面，E-GenRM首先通过监督微调从基于行为的评估链中学习。然后通过标准引导的强化学习进行优化，奖励每个标准下的正确偏好判断和证据支持的事实主张。个性化的硬性否定培训进一步提升了其对购物行为中用户偏好的敏感度，超越了明确的要求。我们还开发了主动证据检索扩展，能够获取缺失的电商信息进行评估。在E-GenRM-Bench上，E-GenRM-27B实现了87.40%的偏好准确率，比最强基线高出6.97个百分点。在淘宝为期14天的在线A/B测试中，接受E-GenRM后期培训的购物代理提升了1.39个百分点的有效购物协助率，提高了2.33个百分点的用户满意度。

Improved regret bounds in tabular RL under local differential privacy

在局部差异隐私下，表格式强化学习的遗憾边界改善

• Authors: Hugo Richard (ENSAE, FAIRPLAY)
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03026
• Pdf link: https://arxiv.org/pdf/2611.03026
• Abstract We study regret minimization for time-homogeneous episodic tabular reinforcement learning with $S$ states and $A$ actions under non-interactive $\epsilon$-local differential privacy (LDP). In each episode $k \in [K]$, the decision-maker sends a policy to a user, the user executes the policy for $H$ steps, and sends an $\epsilon$-locally differentially private message back to the decision-maker. When $H=1$, the problem reduces to an LDP contextual bandit with $S$ contexts, $A$ arms and horizon $K$. We prove that the minimax regret is $\min\bigg(K, \frac{ \sqrt{\max( S, \exp(\epsilon) + 1) S A K (\exp(\epsilon) + 1)}}{\exp(\epsilon) - 1}\bigg)$, up to constants and a $\sqrt{\log(A)}$ factor. The upper bound is achieved by an adaptation of EXP3 and a unary encoding mechanism. When $H > 1$, and for small $\epsilon$, we show a lower bound in $\Omega(HS\frac{\sqrt{A K}}{\epsilon})$ and an upper bound in $\tilde{O}(H^2 S^2 \frac{\sqrt{ AK}}{\epsilon})$. The upper bound is achieved by an algorithm based on UCBVI and a unary encoding mechanism. For small $\epsilon$, these bounds improve the previously known lower bound by a factor $\sqrt{S}$ and the previously known upper bound by a factor of $H \sqrt{A}$.
• 中文摘要 我们研究了在非交互式 $\epsilon$-局部差分隐私（LDP）下，具有$S$状态和$A$动作的时间齐质情节表强化学习中的遗憾最小化。在每个 $k \in [K]$ 的集数中，决策者向用户发送策略，用户执行$H该策略，并向决策者发送一条 $\epsilon$-局部差分私密消息。当 $H=1$ 时，问题简化为一个具有 $S$ 上下文、$A$ 臂和地平线 $K 的 LDP 上下文强化工具。我们证明极小极大后悔为$\min\bigg（K， \frac{ \sqrt{\max（ S， \exp（\epsilon） + 1） S A K （\exp（\epsilon） + 1）}}{\exp（\epsilon） - 1}\bigg）$，且常数和一个$\sqrt{\log（A）}}}因子。上界通过对EXP3的适应和一元编码机制实现。当$H >1$时，对于小的$\epsilon$，我们显示$\Omega（HS\frac{\sqrt{A K}}{\epsilon}）$的下界，$\tilde{O}（H^2 S^2 \frac{\sqrt{ AK}}{\epsilon}）$的下界。上界由基于UCBVI和一元编码机制的算法实现。对于小的$\epsilon$，这些上界使已知下界提高了一个因子$\sqrt{S}$，将已知上界提高了一个因子$H \sqrt{A}$。

Bayesian Sequential Optimal Experimental Design for Single-Pixel Imaging using Reinforcement Learning

基于强化学习的贝叶斯顺序最优实验设计用于单像素成像

• Authors: Marcos Antonio Obando, Tianyuan Wang, Tanja Tarvainen, Tristan van Leeuwen, Felix Lucka
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03042
• Pdf link: https://arxiv.org/pdf/2611.03042
• Abstract Adaptive compressed sensing (CS) aims to select the most informative measurements sequentially, which is crucial for many modern imaging systems. This task can be formulated as a Bayesian sequential optimal experimental design (sOED) problem, where each new acquisition is chosen to maximally reduce uncertainty about the unknown image given prior information and subsequent data provided by acquisitions. However, in single-pixel imaging (SPI), the large design space consists of all possible binary illumination patterns. Directly optimizing the Bayesian sOED objective over this space using conventional methods is generally challenging. In this work, we reformulate sOED for SPI as a reinforcement learning (RL) problem. In RL, an agent learns a policy through repeated experimental trials to maximize a reward, here given as the expected information gain, the most complete measure of uncertainty reduction in Bayesian OED. This avoids the need to directly optimize complex information-theoretic objectives during a measurement to obtain the next design. Instead, the result is a learned policy function that directly maps the current posterior belief about the image to the next optimal binary measurement pattern, in a computationally efficient way. Using a simple yet representative SPI imaging scenario, this work evaluates the scalability of the learned policy under two different policy parameterizations. Furthermore, we investigate how performance deteriorates when only approximate posterior information is available or the measurements are impaired by noise, and propose several strategies to introduce additional design constraints motivated by biomedical applications. The numerical experiments demonstrate that the proposed Bayesian sOED-RL framework remains robust and can be extended to more complex, large-scale SPI scenarios.
• 中文摘要 自适应压缩传感（CS）旨在顺序选择最具信息量的测量值，这对许多现代成像系统至关重要。该任务可被表述为贝叶斯顺序最优实验设计（sOED）问题，每次新采集都被选择以最大化根据先前信息和后续采集数据对未知图像的不确定性。然而，在单像素成像（SPI）中，庞大的设计空间包含所有可能的二进制照明模式。使用传统方法直接优化该空间的贝叶斯sOED目标通常具有挑战性。本研究将sOED重新表述为强化学习（RL）问题。在强化学习中，智能体通过反复实验试验学习策略以最大化奖励，这里称为预期信息获得，这是贝叶斯OED中最完整的不确定性降低度量。这避免了在测量过程中直接优化复杂信息论目标以获得下一设计的需求。相反，结果是一个学习策略函数，能够以计算效率的方式，将当前对图像的后验信念直接映射到下一个最优二元测量模式。本研究利用一个简单但具有代表性的SPI成像场景，评估了所学策略在两种不同策略参数化下的可扩展性。此外，我们研究了当仅有近似后验信息或测量受噪声影响时性能如何下降，并提出多种基于生物医学应用的额外设计约束策略。数值实验表明，所提出的贝叶斯sOED-RL框架依然稳健，并可扩展到更复杂、大规模的SPI场景。

BeyondContact: A Reinforcement Learning Framework for Airflow-Based Non-Contact Manipulation

超越接触：基于气流的非接触操作强化学习框架

• Authors: Peiyu Yang, Mohamad S. Moudallal, Antonio Langella, Giulio Turrisi, Victor Barasuol, Claudio Semini
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.03059
• Pdf link: https://arxiv.org/pdf/2611.03059
• Abstract Contact-based robotic manipulation is constrained by end-effector reachability and the mechanical properties of objects. Airflow can exert forces without direct robot-object contact, enabling manipulation of objects beyond the reach or when direct contact is undesirable. However, the lack of suitable simulation has hindered the application of reinforcement learning to airflow-based manipulation, limiting the broader adoption of this mechanism compared with contact-based methods. To address this gap, we developed a robot- and task-agnostic airflow extension for Isaac Lab that supports massively parallel training. The extension uses experimentally identified airflow models to compute the forces and torques exerted on objects. Based on this extension, we train a hybrid push-blow (HPB) policy that combines contact-based pushing with airflow manipulation and a rectangular reorientation (RR) policy that controls a cuboid's heading using airflow alone. To enable physical execution, we design a lightweight, high-speed blower that coexists with a gripper. In zero-shot real-world deployment, HPB achieves an 80% success rate over 20 trials, while RR achieves a mean absolute heading error of 2.6 degrees across four target headings. The project materials are available at this https URL.
• 中文摘要 基于接触的机器人操作受末端执行器可达性和物体机械性能的限制。气流可以在机器人与物体直接接触的情况下施加力，从而在无法直接接触或不希望直接接触时也能操作物体。然而，缺乏合适的模拟阻碍了强化学习在基于气流操作中的应用，限制了该机制相较于接触方法的更广泛应用。为弥补这一空白，我们为Isaac实验室开发了一个机器人和任务无关的气流扩展，支持大规模并行训练。该扩展使用实验识别的气流模型计算对物体施加的力和力矩。基于该扩展，我们训练了一种混合推力（HPB）策略，结合了基于接触的推动与气流操控，以及仅靠气流控制长方体方向的矩形重定向（RR）策略。为实现物理执行，我们设计了一种轻质高速鼓风机，并与夹持器共存。在零试落的实际部署中，HPB在20次试验中成功率达80%，而RR在四个目标航向中平均绝对航向误差为2.6度。项目材料可在此 https 网址获取。

MIRA: Reinforcing Multimodal Reasoning via Deceptive Contextual Augmentation

MIRA：通过欺骗性情境增强强化多模态推理

• Authors: Zhihan Yin, Jianxin Liang, Yifeng Yao, Nonghai Zhang, and Bingyue Peng, Chunguang Qie, Huishuai Zhang, Dongyan Zhao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.03087
• Pdf link: https://arxiv.org/pdf/2611.03087
• Abstract While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced multimodal reasoning, existing frameworks suffer from a critical "perception-reasoning gap." Due to an over-reliance on seemingly relevant visual tokens, even minor perceptual perturbations can propagate through the reasoning process, leading to compounding hallucinations. To address this issue, we propose MIRA, a training framework that improves robustness to erroneous visual contexts while enhancing logical reasoning ability. By injecting filtered, deceptive visual contexts during training, we construct challenging "hard traps" that stress-test the model's robustness to misleading information. Furthermore, we introduce a group-wise reflection-triggering mechanism that enables the model to autonomously detect and rectify inconsistencies between visual cues and logical constraints, which is ultimately internalized as an intrinsic policy behavior. Importantly, MIRA does not rely on human annotations. Experimental results across multiple reasoning and perception benchmarks show that MIRA yields an average 7% improvement on in-domain benchmarks, with consistent improvements on vision-centric benchmarks. These findings highlight that active error recovery is essential for robust multimodal reasoning. Comprehensive ablation studies and analyses further provide insights into why MIRA is effective.
• 中文摘要 虽然带有可验证奖励的强化学习（RLVR）显著推动了多模态推理，但现有框架存在关键的“感知-推理鸿沟”。由于过度依赖看似相关的视觉符号，即使是轻微的感知干扰也可能在推理过程中传播，导致幻觉叠加。为解决这一问题，我们提出了MIRA训练框架，该框架提升对错误视觉情境的鲁棒性，同时增强逻辑推理能力。通过在训练中注入过滤的欺骗性视觉情境，我们构建了具有挑战性的“硬陷阱”，以压力测试模型对误导性信息的韧性。此外，我们引入了一种分组反射触发机制，使模型能够自主检测并纠正视觉线索与逻辑约束之间的不一致，最终内化为一种内在的策略行为。重要的是，MIRA不依赖人工注释。跨多个推理和感知基准的实验结果显示，MIRA平均在领域基准上提升7%，视觉中心基准持续提升。这些发现凸显主动错误恢复对于稳健的多模态推理至关重要。全面的消融研究和分析进一步揭示了MIRA为何有效。

KlaskTron-IL: A Contact-Rich Adversarial Imitation Learning Benchmark with a Validated Sim-to-Real Bridge

KlaskTron-IL：一个带有验证模拟到真实桥接的丰富接触对抗性模仿学习基准测试

• Authors: Nicklas Sohm, Aswin Ramachandran, Raffaello D'Andrea
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.03091
• Pdf link: https://arxiv.org/pdf/2611.03091
• Abstract KlaskTron is an open-hardware physical platform for two-player adversarial reinforcement learning, with a learned actuator model that supports zero-shot sim-to-real transfer of trained policies. Building on this platform, we contribute KlaskTron-IL: a multi-tier imitation learning dataset (80k human demonstration timesteps collected directly on the physical robot, 320k symmetry-augmented timesteps, and synthetic expert trajectories up to 5M steps generated by the zero-shot-validated PPO policy), and diagnostic baselines that expose three characteristic failure modes of standard IL methods under contact-rich adversarial dynamics: BC's data scaling plateau, GAIL's interpretable but weak-gradient discriminator (Spearman $\rho = -1.0$ peg-ball proximity recovery, but a logit span under 0.1), and an asymmetric BC-prior x reward-density interaction. Because the policy interface used throughout this paper is the same one the actuator model bridges to hardware, a policy trained on this benchmark can run unchanged on the physical robot, as we demonstrate zero-shot for the strongest BC baseline, so long as its commands respect the actuator model's $\pm 0.2$ m/s identification range.
• 中文摘要 KlaskTron 是一个开放硬件物理平台，用于双人对抗强化学习，拥有一个学习型执行器模型，支持训练策略的零样本模拟到实物传输。基于该平台，我们贡献了 KlaskTron-IL：一个多层模拟学习数据集（直接在物理机器人上收集的 80k 人类示范时间步，320k 对称增强时间步，以及由零样本验证的 PPO 策略生成的最高 500 万合成专家轨迹），以及诊断基线，揭示了标准 IL 方法在接触丰富对抗动态下的三种特征性失效模式：BC 的数据尺度平台、GAIL 可解释但梯度较弱的判别器（Spearman $\rho = -1.0$ 钉球邻近恢复，但对数跨度小于 0.1）以及非对称 BC 先于 x 奖励密度交互。由于本文中使用的策略接口与执行器模型连接硬件的界面相同，基于该基准训练的策略可以在物理机器人上不变运行，我们展示了对最强BC基线的零射击，只要其指令尊重执行器模型的$\pm 0.2$ m/s识别范围。

Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation

在学生所在的地方与他们相遇：学生前缀教师适应政策提炼

• Authors: Xue Zhang, Songming Zhang, Yunlong Liang, Fandong Meng, Yufeng Chen, Jinan Xu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.03133
• Pdf link: https://arxiv.org/pdf/2611.03133
• Abstract On-policy distillation (OPD) mitigates the student's exposure bias by optimizing on student-generated trajectories. However, student-generated prefixes may contain reasoning errors or follow patterns unfamiliar to the teacher, undermining the reliability of its token-level supervision. This degradation becomes more pronounced as prefixes grow longer. To address this teacher-side distribution mismatch, we propose Student-Prefix Teacher Adaptation (SPTA), which adapts the teacher to the states on which it is queried during distillation. Specifically, before OPD starts, we collect student rollout prefixes with random lengths and continually train the teacher to continue from the prefixes using reinforcement learning with verifiable rewards (RLVR). By treating each prefix as context and rewarding correct final answers, the teacher learns to recover from student-induced states rather than imitate the student's reasoning. Then, the adapted teacher is used for standard OPD, with no changes to the distillation objective. Experiments on mathematical reasoning benchmarks show that our SPTA improves average student performance over standard OPD and OPD with task-level RLVR-adapted teachers. These results highlight the importance of adapting teachers not only to solve problems independently, but also to provide reliable guidance on the states their students actually visit.
• 中文摘要 政策上蒸馏（OPD）通过优化学生生成的路径，减轻学生的暴露偏差。然而，学生生成的前缀可能包含推理错误或遵循教师不熟悉的模式，削弱其代币级监督的可靠性。随着前缀越长，这种退化越明显。为解决教师侧分布不匹配，我们提出了学生-前缀教师适应（SPTA），即在蒸馏过程中调整教师被查询的状态。具体来说，在OPD开始前，我们收集随机长度的学生推出前缀，并持续训练教师使用可验证奖励强化学习（RLVR）从前缀继续学习。通过将每个前缀视为上下文并奖励正确的最终答案，教师学会从学生诱导的状态中恢复，而非模仿学生的推理。然后，适应教师用于标准OPD，且不改变提炼目标。数学推理基准测试的实验表明，我们的SPTA在任务级RLVR适应教师中，平均学生表现优于标准OPD和OPD提升。这些结果凸显了调整教师不仅独立解决问题的重要性，还应为学生实际访问的状态提供可靠指导。

KlaskTron: An Open-Source Platform for Physical Adversarial Multi-Agent RL

KlaskTron：一个开源的物理对抗多智能体强化学习平台

• Authors: Aswin Ramachandran, Jona Schulz, Maurus Derungs, Carlo Angelini, Tobias Meier, Raffaello D'Andrea
• Subjects: Subjects: Robotics (cs.RO); Hardware Architecture (cs.AR); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.03138
• Pdf link: https://arxiv.org/pdf/2611.03138
• Abstract Progress in robot learning, particularly physical multi-agent reinforcement learning (MARL), is currently challenged by a limited availability of accessible, standardized benchmarks. While simulation-based MARL has produced remarkable emergent behaviors, from coordinated team play to complex tool use, translating these advances to physical systems remains difficult. A key barrier is infrastructural: physical platforms are often prohibitively expensive, require specialized facilities, or lack open mechanisms and designs for reproducibility. We introduce KlaskTron, an open-source, low-cost (<\$2500 USD), desktop-scale robotic testbed for adversarial MARL based on the dynamic tabletop game KLASK. The platform features dual CoreXY gantries with high-torque brushless motors, enabling the high-speed, precise manipulation required for competitive play. Crucially, we provide a complete ecosystem: hardware designs (CAD, BOM), a GPU-native digital twin in NVIDIA Isaac Lab for high-fidelity modeling and simulation, and a validated sim-to-real baseline using a learned neural actuator model. We demonstrate that this baseline enables zero-shot policy transfer, with trained agents exhibiting emergent adversarial behaviors. To ensure full reproducibility of our results, all project assets are released publicly: hardware CAD/BOM and assembly documentation at [this https URL], inference software stack at [this https URL] and the simulation and MARL setup at [this https URL].
• 中文摘要 机器人学习，尤其是物理多智能体强化学习（MARL）的进展，目前面临着有限且可访问的标准化基准测试的挑战。尽管基于仿真的MARL产生了从协调团队游戏到复杂工具使用等显著的涌现行为，但将这些进展转化为物理系统仍然困难。基础设施是一个关键障碍：物理平台通常成本高昂，需要专用设施，或缺乏开放机制和可重复性设计。我们介绍KlaskTron，一款开源、低成本（<\$2500美元）、桌面规模的对抗性MARL机器人测试平台，基于动态桌面游戏KLASK。该平台配备双CoreXY门架和高扭矩无刷电机，实现竞技游戏所需的高速、精准操作。关键是，我们提供了一个完整的生态系统：硬件设计（CAD、BOM）、NVIDIA Isaac Lab 中GPU原生的数字孪生，用于高精度建模和仿真，以及使用学习过的神经执行器模型验证的模拟到现实基线。我们证明该基线支持零样本策略转移，训练有素的代理表现出突发的对抗行为。为确保结果的完全可重复性，所有项目资产均公开发布：硬件CAD/BOM和组装文档（此 https URL）、推理软件栈（此 https URL）以及仿真和MARL设置（此 https URL）。

Controlling Sharpening in LLM Post-Training

控制LLM培训后磨砺

• Authors: Simon Matrenok, Nicolas Menet, Nicolas Flammarion, Andreas Krause
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03146
• Pdf link: https://arxiv.org/pdf/2611.03146
• Abstract LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage - a phenomenon known as sharpening that can stall progress on difficult tasks. Repeated sampling objectives such as $\mathrm{pass@k}$ and $\mathrm{max@k}$ are known to mitigate sharpening, but the mechanism behind it remains poorly understood. Moreover, existing approaches for $\mathrm{@k}$ optimization use dedicated algorithms, and no general framework exists for adapting established policy optimization methods from $\mathrm{@}1$ to $\mathrm{@k}$. We derive the optimal policies for KL-regularized $\mathrm{pass@k}$ and $\mathrm{max@k}$ objectives in closed form. Our explicit solutions reveal that $\mathrm{@k}$ optimization naturally redistributes learning pressure based on prompt difficulty, directing it toward harder problems. Crucially, we show that standard $\mathrm{@}1$ objectives achieve these exact same optimal policies through prompt-adaptive regularization or reward transformation. Ultimately, this yields an optimizer-independent explanation for why $\mathrm{@k}$ objectives reduce sharpening on easy tasks while prioritizing success on harder ones. Building on these insights, we introduce $\mathrm{@k}$ Adaptive Pressure for Policy Optimization (KAPPO). Following a one-time reference calibration, KAPPO seamlessly adapts existing $\mathrm{@}1$ methods to $\mathrm{@k}$ via prompt-adaptive regularization or reward transformation, without otherwise altering their training procedures. Experiments with several RL methods on mathematical reasoning and code generation demonstrate that KAPPO provides direct control over sharpening. Crucially, we demonstrate that $\mathrm{@k}$ pressure can be fully realized with single completion training, matching the performance of multi-completion $\mathrm{@k}$ optimizers at a fraction of the cost.
• 中文摘要 通过强化学习进行的LLM后训练显示，准确率持续提升，但代价是行为覆盖率下降——这种现象称为锐化，可能阻碍难度任务的进展。重复采样目标如$\mathrm{pass@k}$和$\mathrm{max@k}$已知能减轻锐化，但其背后的机制仍不甚明了。此外，现有的$\mathrm{@k}$优化方法采用专用算法 目前尚无通用框架将既有策略优化方法从 $\mathrm{@}1$ 改编到 $\mathrm{@k}$。我们推导出 KL 正则化 $\mathrm{pass@k}$ 和 $\mathrm{max@k}$ 目标的最优策略，形式为封闭形式。我们的显式解法表明，$\mathrm{@k}$ 优化自然地根据提示难度重新分配学习压力，将其引导到更难的问题上。关键是，我们证明标准的$\mathrm{@}1$目标通过提示自适应正则化或奖励转换实现了完全相同的最优策略。最终，这给出了一个与优化器无关的解释，说明为什么$\mathrm{@k}$目标在简单任务上减少锐化，而在较难任务上优先考虑成功。基于这些洞见，我们引入了$\mathrm{@k}$ 策略优化自适应压力（KAPPO）。经过一次性参考校准后，KAPPO通过提示自适应正则化或奖励转换，无缝将现有的$\mathrm{@}1$方法适配到$\mathrm{@k}$，而无需其他方式改变其训练过程。通过多种强化学习方法进行数学推理和代码生成的实验，表明KAPPO能够直接控制锐化过程。关键是，我们证明了$\mathrm{@k}$压力可以通过单次完成训练完全实现，性能可媲美多次完成的$\mathrm{@k}}优化器，成本仅为其一小部分。

From PID to Reinforcement Learning in Building HVAC Control: A Control-Architecture Review of Tuning Strategies, Formulations, and Deployment Trade-offs

从建筑暖通空调控制中的PID到强化学习：调优策略、表述及部署权衡的控制架构回顾

• Authors: Erfan Haghighat Damavandi, Davide Papurello, Mahdi Alibeigi, Armin Keshavarz
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03152
• Pdf link: https://arxiv.org/pdf/2611.03152
• Abstract This paper reviews control architectures for building heating, ventilation and air-conditioning (HVAC) systems, addressing the tension among energy demand, occupant comfort and equipment operability, and situating proportional-integral-derivative (PID) regulation within a layered control stack that also includes rule-based and model-based supervisory control and data-driven reinforcement learning (RL). It examines the thermostat-PID layer from two-position and scheduled setpoint control through proportional, PI and PID regulation and their tuning, then formulates RL for building control as a Markov decision process. Its central contribution is a taxonomy of where RL meets PID: RL as a gain tuner for an existing loop, as a direct replacement for the regulatory controller, as a hybrid with model predictive control, and as supervisory setpoint and schedule generation. Across these classes, the review establishes that the physical loop remains invariant while the intervention point changes, and that the classes differ in their requirements for actuation authority, state information, modelling, commissioning and constraint handling. It further establishes that PID is inexpensive and well codified but limited by nonlinearity, time variation, discrete mode switching, loop interaction and slow plant dynamics, while RL's admissible position must be argued rather than assumed. The paper concludes that these architectures are not competing philosophies of control but nested points of attachment on one HVAC plant, with hybrid implementations possible, and that deployment trade-offs must be assessed according to the specific control interface and scaffolding required.
• 中文摘要 本文回顾了建筑供暖、通风和空调（HVAC）系统的控制架构，解决能源需求、居住者舒适度与设备操作性之间的张力，并将比例积分导数（PID）调控置于包含基于规则和模型的监督控制及数据驱动强化学习（RL）的分层控制堆栈中。它从两位和定时设定点控制到比例、PID和PID调控及其调优，分析了恒温器-PID层，并将建筑控制的RL构建为马尔可夫决策过程。其核心贡献是对RL与PID交汇的分类法：RL作为现有回路的增益调谐器，作为调节控制器的直接替代，作为模型预测控制的混合体，以及作为监督设定点和调度生成。综述在这些类别中，确立了物理环路保持不变，而干预点变化，且各类在执行权限、状态信息、建模、调试和约束处理等要求上存在差异。进一步证明PID成本低廉且规范良好，但受限于非线性、时间变化、离散模式切换、环路交互和缓慢的工厂动力学，而强化学习的可接受位置则应被论证而非假设。论文总结指出，这些架构并非竞争的控制理念，而是嵌套的附着点，存在混合实现，部署权衡必须根据具体控制接口和脚手架进行评估。

Axion 1.0: Open Agent Foundation Models, Simplified

Axion 1.0：开放代理基础模型，简化版

• Authors: Yuqi Jia, Yunjie Tian, Jiayu Zheng, Yijun Liang, Jingchen Sun, Mengyu Zhao, Di Fu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.03233
• Pdf link: https://arxiv.org/pdf/2611.03233
• Abstract We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.
• 中文摘要 我们介绍Axion 1.0，一个完全开放的代理基础模型，结合了强大且广泛的代理能力与极其简单的训练方案。Axion 1.0仅训练三个阶段：两个轻量级监督微调阶段，随后进行强化学习，无需专门的中途训练、直接偏好优化（DPO）或策略提炼（OPD）。第一阶段通过模型感知数据构建，发展强大的自包含推理能力，我们综合挑战性问题，并基于基础模型自身成功的解决方案策略进行对抗性强化。第二阶段通过可扩展的流水线，将这些能力扩展到工具依赖推理，构建基于真实世界数据源和确定性计算的多样且自动验证的任务。最后，强化学习进一步强化长期推理能力及与外部工具和环境的互动。尽管训练方案紧凑，Axion 1.0在广泛的智能体基准测试套件中表现出色，并在多个基准测试上与领先的商业模型竞争。同样重要的是，Axion 1.0在模型权重之外是开放的：我们发布了完整的配方，包括数据构建、生成、过滤、验证、提示和训练代码，使整个开发过程能够被重现和扩展。Axion 1.0展示了强智能体基础模型既开放又简单，提供了一条实用且可重复的路径，推动开放智能智能的发展。

The Cost of Mismatch: Noise Amplification in Zeroth-Order Reinforcement Learning

错配的代价：零阶强化学习中的噪声放大

• Authors: Lianmin Chen, Junbin Qiu, Chenxing Wei, Yao Shu, Kun He
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03260
• Pdf link: https://arxiv.org/pdf/2611.03260
• Abstract Zeroth-order (ZO) optimization has emerged as a promising approach for memory-efficient large language model (LLM) post-training, yet its reliable application to reinforcement learning with verifiable rewards (RLVR) remains largely unexplored. Motivated by the distinct data regimes of on-policy RL and fixed-data optimization, we develop a generic variance-theoretic framework to investigate what limits the reliability of ZO gradient estimation. Our analysis reveals that estimation reliability is structurally determined by whether the two finite-difference evaluations are performed on matched data. On-policy reinforcement learning (RL) under the mismatched-data regime produces a $1/\mu^2$ oracle-noise amplification that sets an irreducible noise floor as the perturbation scale $\mu$ decreases, whereas Boltzmann Targeted SFT (BOLT) under the matched-data regime eliminates this amplification via noise cancellation. We further derive convergence guarantees with explicit constants that cleanly separate the two regimes. These results naturally lead to ZO-BOLT, which applies ZO optimization to BOLT's RLVR-aligned weighted-SFT objective. Controlled perturbation-scale sweeps and matched-versus-mismatched end-to-end comparisons validate the predicted effect of data matching. Experiments across three model variants and two mathematical reasoning benchmarks further show that ZO-BOLT consistently improves over the pretrained baseline while retaining a near-inference memory footprint. The code is publicly available at this https URL.
• 中文摘要 零阶（ZO）优化已成为内存高效大型语言模型（LLM）训练后一种有前景的方法，但其在可验证奖励强化学习（RLVR）中的可靠应用仍大多未被充分探索。受策略强化学习和固定数据优化截然不同的数据模式激励，我们开发了一个通用方差理论框架，以探讨ZO梯度估计可靠性的限制因素。我们的分析显示，估计可靠性的结构性决定于是否对匹配数据进行有限差分评估。在错配数据模式下，策略强化学习（RL）产生$1/\mu^2$的oracle-noise放大，随着扰动尺度$\mu$的减少，设定不可约噪声底;而匹配数据模式下的Boltzmann Targeted SFT（BOLT）通过噪声消除消除了这种放大。我们还进一步推导出了收敛保证，明确区分两个区域的显式常数。这些结果自然导致了ZO-BOLT，它将ZO优化应用于BOLT的RLVR对齐加权SFT目标。受控微扰尺度扫描和匹配与不匹配端对比验证了数据匹配的预测效果。在三种模型变体和两个数学推理基准测试中的实验进一步表明，ZO-BOLT在保持近乎推理内存占用的同时，持续优于预训练基线。该代码公开可访问此https URL。

DeepWaveRL: Self-Supervised Full Waveform Inversion via Reinforcement Learning

DeepWaveRL：通过强化学习实现自监督全波形反演

• Authors: Peng Jin, Yinan Feng, Yinpeng Chen, Youzuo Lin, Sharon Huang
• Subjects: Subjects: Machine Learning (cs.LG); Geophysics (physics.geo-ph)
• Arxiv link: https://arxiv.org/abs/2611.03264
• Pdf link: https://arxiv.org/pdf/2611.03264
• Abstract Full Waveform Inversion (FWI) is a fundamental inverse problem in geophysics to estimate subsurface properties, such as velocity, from seismic measurements. Recently, self-supervised learning methods remove the need for ground-truth velocity maps during training by leveraging a forward operator to simulate seismic data from predictions and backpropagate the gradients of reconstruction loss through it. In this work, we consider a more challenging setting without accessing such gradients. To this end, we incorporate reinforcement learning into self-supervised FWI and propose a framework named DeepWaveRL. Specifically, we reformulate FWI as a policy learning problem, where the policy network predicts velocity maps as actions, and a forward operator is used solely for simulation. A global scalar reward is computed based on data misfit for each prediction and used to weight policy gradients. Thus, learning proceeds without accessing the gradients from the forward operator. Given the indirect and coarse supervision signals, we further identify key strategies to stabilize learning and mitigate the spatial credit assignment bottleneck. Experimental results show that, despite relying entirely on a single scalar reward, DeepWaveRL achieves competitive performance compared to supervised and self-supervised baselines. These findings suggest that our framework offers a flexible and complementary learning paradigm for the FWI research community.
• 中文摘要 全波形反演（FWI）是地球物理学中的一个基础逆问题，用于从地震测量中估算地下属性，如速度。近年来，自监督学习方法通过利用前向算符模拟预测中的地震数据，并通过其反向传播重建损失梯度，消除了训练过程中对地面真实速度图的需求。在本研究中，我们考虑了一个更具挑战性的设定，但不涉及这些梯度。为此，我们将强化学习纳入自监督FWI，并提出了一个名为DeepWaveRL的框架。具体来说，我们将FWI重新表述为一个策略学习问题，其中策略网络以动作方式预测速度图，而前向算符仅用于模拟。基于每个预测的数据不适配计算全局标量奖励，并用于加权策略梯度。因此，学习过程无需访问前向算符的梯度。鉴于间接和粗监督信号，我们进一步识别稳定学习和缓解空间学分分配瓶颈的关键策略。实验结果表明，尽管完全依赖单一标量奖励，DeepWaveRL 在与监督和自监督基线相比仍能实现竞争性能。这些发现表明我们的框架为 FWI 研究社区提供了一个灵活且互补的学习范式。

Inferring Transferable Rewards via Active Inverse Reinforcement Learning

通过主动逆强化学习推断可转移奖励

• Authors: Victor Villin, Till Freihaut, Andreas Schlaginhaufen, Christos Dimitrakakis, Maryam Kamgarpour, Giorgia Ramponi
• Subjects: Subjects: Machine Learning (cs.LG); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2611.03296
• Pdf link: https://arxiv.org/pdf/2611.03296
• Abstract Inverse Reinforcement Learning (IRL) aims to recover expert rewards that transfer across environments with differing dynamics. Prior work has shown that such transfer is possible when gathering expert demonstrations across sufficiently different dynamics. In practice, however, expert demonstrations are costly, making it critical to select the right dynamics to reduce the quantity of expert data required for learning transferable rewards. We propose a two-phase approach: we first cycle through the available dynamics to obtain a reliable reward estimate, then actively select dynamics to reduce our uncertainty about the expert reward. Crucially, this yields novel sample complexity guarantees on transfer that hold for any dynamics, with a rate independent of the number of available dynamics. Empirically, our method significantly reduces the number of expert demonstrations needed to learn a transferable reward.
• 中文摘要 逆向强化学习（IRL）旨在回收跨不同动态环境的专家奖励。以往研究表明，当收集跨足够不同动态的专家演示时，这种转移是可能的。然而，在实际操作中，专家演示成本高昂，因此选择合适的动态以减少学习可转移奖励所需的专家数据量至关重要。我们提出两阶段方法：首先循环使用动态以获得可靠的奖励估计，然后主动选择动态以降低对专家奖励的不确定性。关键是，这产生了对任何动态都成立且速率独立于可用动态数量的新颖样本复杂度保证。从经验角度看，我们的方法显著减少了学习可转移奖励所需的专家演示次数。

SQLoom: Set-Contrastive Learning of Syntactic, Semantic, and Logical Dependencies for Text-to-SQL

SQLoom：文本转SQL中句法、语义和逻辑依赖的集合对比学习

• Authors: Hao Fu, Degang Sun, Wan Wei, Jinxia Wei, Chun Long, Peng Gao
• Subjects: Subjects: Databases (cs.DB); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.03298
• Pdf link: https://arxiv.org/pdf/2611.03298
• Abstract Text-to-SQL translates natural language questions into SQL queries, enabling database access without writing SQL. Fine-tuning open-weight large language models (LLMs) is cost-effective, but existing methods treat SQL as ordinary text and optimize token-level likelihood without explicitly modeling its syntactic, semantic, and logical requirements. Thus, a single incorrect alias or comparison operator can invalidate a query while contributing little to training loss. We propose SQLoom, a constraint-aware fine-tuning framework with supervision at three levels: syntactic validity under SQL grammar and database schema, semantic alignment between query decisions and the question, and logical consistency across dependent clause decisions. SQLoom extends supervised fine-tuning with a unified set-contrastive objective favoring sets of correct outputs over minimally different incorrect ones at each level. LLM agents propose this supervision, which a SQL parser and execution on counterexample databases verify. We also release Spider-Refined, a training dataset correcting 378 flawed Spider annotations and enriching database schemas. Experiments on Spider, BIRD, and three perturbed Spider variants show that SQLoom improves execution accuracy for Qwen3-1.7B from 62.9% zero-shot to 88.7% on Spider development, reaching 89.2% on Spider test and 73.4% on BIRD development. It outperforms fine-tuning, reinforcement learning, and agent-based baselines with up to 70 billion parameters. On question perturbations, SQLoom loses at most 0.6 accuracy points versus up to 12.1 for agent-based methods. It generates SQL in a single pass at 0.42s per query, 23$\times$ faster than a GPT-4-based agent method. These results demonstrate the effectiveness of set-contrastive supervision over token-level training.
• 中文摘要 文本转SQL将自然语言问题转换为SQL查询，实现数据库访问而无需编写SQL即可。微调开权大语言模型（LLMs）具有成本效益，但现有方法将SQL视为普通文本，优化令牌级似然度，而未显式建模其语法、语义和逻辑需求。因此，单个错误的别名或比较算符可能使查询失效，且对训练损失贡献甚微。我们提出SQLoom，一个约束感知型微调框架，在三个层面监督：SQL语法和数据库模式下的语法有效性、查询决策与问题之间的语义对齐，以及依赖子句决策间的逻辑一致性。SQLoom通过统一的集合对比目标扩展监督微调，优先选择正确输出集而非各层级错误输出的最小差异。LLM代理提出了该监督，SQL解析器和反例数据库执行对此进行了验证。我们还发布了Spider-Refined，这是一个训练数据集，纠正了378条Spider错误注释并丰富了数据库模式。对Spider、BIRD和三种扰动Spider变体的实验显示，SQLoom在Qwen3-1.7B的零射击测试中将执行准确率从62.9%提升至88.7%，在Spider测试中达到89.2%，BIRD开发中达到73.4%。它在多达700亿参数的条件下优于微调、强化学习和基于智能体的基线。在问题扰动方面，SQLoom最多损失0.6个准确点，而基于智能体的方法则最高可达12.1个。它以每查询0.42秒的周期生成SQL，比基于GPT-4的代理方法快23美元\时间。这些结果证明了集合对比监督相较于令牌级训练的有效性。

An Axiomatic Framework for Reinforcement Learning in Open Multi-Agent Systems

开放多智能体系统中强化学习的公理化框架

• Authors: Jianhong Wang, Yang Li, Samuel Kaski, Jonathan Lawry
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2611.03301
• Pdf link: https://arxiv.org/pdf/2611.03301
• Abstract Open multi-agent systems, where only a subset of agents are controllable, are central to real-world domains such as smart grids, yet lack a principled foundation for reinforcement learning algorithm design. We study this setting through n-agent ad hoc teamwork (NAHT), a recent framework for learning in open multi-agent environments. We propose an axiomatic framework that derives value function learning from the Shapley axioms (Efficiency, Symmetry and Linearity), treating them as structural constraints rather than heuristic design choices. This formulation builds on a novel NAHT decomposition, which characterises the fundamental difference between open and fully controllable multi-agent systems, and establishes a cooperative game-theoretic foundation for learning. We further show that enforcing these axioms on learning individual value functions recovers the Shapley value for Dec-POMDPs, providing a principled bridge between reinforcement learning and classical cooperative game theory. Building on this framework, we introduce Shapley Machines and Banzhaf Machines, two generic algorithmic templates that can be instantiated with standard reinforcement learning methods. When applied to strong base algorithms such as IPPO and POAM, our methods consistently improve learning efficiency and generalisation to unseen agent types and conventions across NAHT benchmark tasks. The code of experiments is published on this https URL.
• 中文摘要 开放多智能体系统中，只有部分智能体可控，是智能电网等现实领域的核心，但缺乏强化学习算法设计的原则基础。我们通过n代理临时团队合作（NAHT）研究该环境，这是一种近期用于开放多智能体环境中学习的框架。我们提出了一个公理化框架，将价值函数学习从Shapley公理（效率、对称和线性）推导出来，将其视为结构约束而非启发式设计选择。该表述基于一种新的NAHT分解，该分解表征了开放多智能体系统与完全可控多智能体系统的根本区别，并建立了合作博弈论学习基础。我们还进一步证明，在学习个别值函数时强制执行这些公理可以恢复Dec-POMDP的Shapley值，为强化学习与经典合作博弈论之间提供了原则性桥梁。基于该框架，我们介绍了Shapley机和Banzhaf机，这两种通用算法模板可用标准强化学习方法实例化。当应用于强基算法如IPPO和POAM时，我们的方法持续提升学习效率，并能在NAHT基准任务中推广到未见代理类型和约定。实验代码发布于此 https 网址。

Bridging Neural Tangent Kernel Analysis and Reinforcement Learning for Deep Function Approximation

桥接神经切核分析与强化学习以实现深度函数近似

• Authors: William St-Arnaud, Margarida Carvalho, Golnoosh Farnadi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03435
• Pdf link: https://arxiv.org/pdf/2611.03435
• Abstract Despite their widespread success across machine learning, the learning dynamics and generalization properties of neural networks remain poorly understood. The neural tangent kernel (NTK) provides an analytical characterization of the learning dynamics of infinitely wide neural networks and serves as a useful proxy for finite-width networks in supervised learning. In this work, we extend the NTK framework to reinforcement learning (RL) by introducing a temporal-difference (TD) flow formulation that accommodates time-varying mini-batches and TD targets, yielding a closed-form solution for value function evolution in an overparametrized setting. We also devise an optimization-based state-space exploration method that leverages the closed-form solution, coupled with constraints on the environment, to model the set of feasible states. On a deterministic grid environment, we validate that the proposed RL-based NTK formulation is indeed learning efficiently and adapting to the distribution shift in the training data distribution, and we demonstrate that the resulting exploration strategy improves sample efficiency. To our knowledge, this is the first work to extend NTK theory to characterize the learning dynamics of RL training with time-varying mini-batches and TD updates.
• 中文摘要 尽管神经网络在机器学习领域取得了广泛成功，但其学习动力学和泛化特性仍鲜为人知。神经切核（NTK）提供了无限宽神经网络学习动态的分析性特征，并作为监督学习中有限宽度网络的有用代理。本研究中，我们将NTK框架扩展到强化学习（RL），引入了时间差分（TD）流表述，支持时间变化的微批次和TD目标，从而在超参数化环境中实现价值函数演化的闭式解。我们还设计了一种基于优化的状态空间探索方法，利用闭式解与环境约束相结合，建模可行状态集合。在确定性网格环境下，我们验证了基于强化学习的NTK表述确实高效学习并适应了训练数据分布的分布变化，并证明了由此产生的探索策略提升了样本效率。据我们所知，这是首次将NTK理论扩展到通过时间变化的微批次和TD更新来描述强化学习训练的学习动态。

SNAP: Semantic Navigation Among Pedestrians

SNAP：行人语义导航

• Authors: Jakob Wolfram, Pascal Roth, Fan Yang, Cesar Cadena, Marco Hutter
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.03498
• Pdf link: https://arxiv.org/pdf/2611.03498
• Abstract Navigating crowded urban environments requires robots to avoid pedestrians while adhering to social norms such as remaining on sidewalks. Existing approaches typically address either crowd avoidance or semantic path planning in isolation. We propose a reinforcement learning (RL) navigation policy that jointly reasons about semantic terrain and dynamic obstacles from a single semantically annotated bird's-eye-view (BEV) map, without relying on explicit human-state estimation. Our policy outperforms established crowd-navigation and semantic-aware baselines and transfers zero-shot to a legged robot, demonstrating socially compliant navigation in highly dynamic real-world scenarios.
• 中文摘要 在拥挤的城市环境中导航需要机器人避开行人，同时遵守诸如留在人行道上的社会规范。现有方法通常单独解决人群回避或语义路径规划。我们提出一种强化学习（RL）导航政策，结合从单一张语义注释的鸟瞰图（BEV）推理语义地形和动态障碍物，而不依赖显式的人类状态估计。我们的政策优于既有的人群导航和语义感知基线，并将零射击技术转移到有腿的机器人上，展示在高度动态的现实场景中具备社会合规性的导航能力。

When Robots Outweigh Objects: How Inertia Mismatch Shapes Robot Policy Learning

当机器人超过物体：惯性不匹配如何塑造机器人政策学习

• Authors: Younghyo Park, Antonia Bronars, Pulkit Agrawal
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.03501
• Pdf link: https://arxiv.org/pdf/2611.03501
• Abstract Conventional robot arms can be extremely heavy compared with the everyday objects they manipulate, creating a significant inertia mismatch at contact. Even a small accidental bump from a heavy arm can knock a light object out of place. Modern learning methods have nevertheless demonstrated a wide range of successful manipulation behaviors on these platforms, suggesting that even extreme inertia mismatch can be overcome with sufficient data. However, it remains unclear whether high inertia imposes a hidden cost on learning, potentially worsening scaling with data or lowering the performance ceiling of modern robot learning methods. We investigate this question through controlled simulation experiments that vary robot and object masses. We find that reinforcement learning succeeds across a broad range of mass combinations, while behavior cloning learns more reliable policies when a lighter robot manipulates a heavier object, and more data narrows the resulting gap without closing it. Behavior cloning likewise degrades on a simulated robot arm as its motors reflect more inertia through their gearing, even though its weight stays the same. Finally, because new hardware is rarely an option, we explore ways to lower contact inertia on the arm one already has, either by choosing a lower-inertia workspace before collecting data or by steering the demonstrator toward lower-inertia solutions of the task. In both simulated cases, the lower-inertia demonstrations are imitated more reliably. Project website: this https URL
• 中文摘要 传统机器人手臂相比它们操控的日常物体可能非常重，导致接触时显著的惯性不匹配。即使是粗臂的轻微意外碰撞也可能将轻物体撞离位置。尽管如此，现代学习方法已在这些平台上展示了广泛的成功操作行为，表明即使是极端的惯性不匹配也能通过足够的数据克服。然而，目前尚不清楚高惯性是否会对学习产生隐性代价，可能加剧数据扩展，或降低现代机器人学习方法的性能上限。我们通过控制模拟实验研究该问题，实验中机器人和物体的质量变化。我们发现强化学习在广泛的质量组合中都能成功，而行为克隆则在较轻的机器人操作较重物体时学习更可靠的策略，更多数据缩小了差距但未缩小差距。模拟机械臂的行为克隆同样会下降，因为其电机通过齿轮反射更多惯性，尽管重量保持不变。最后，由于新硬件很少是选项，我们探讨降低现有机械臂接触惯性的方法，要么在收集数据前选择较低惯性的工作区，要么引导演示者采用较低惯性的工作区。在这两种模拟情况下，较低惯性演示的模拟都更可靠。项目网站：此 https URL

D&A: Decoupling Description and Assessment Enables VLMs as Zero-Shot Robotic Reward Models

D&A：解耦描述与评估使VLM成为零射点机器人奖励模型

• Authors: Marius Memmel, Shawn Yin, Urvi Rutia, Angad Wadhwa, Eldon Li, Gavin Angeloff, Ankit Goyal, Jesse Zhang, Dieter Fox, Fabio Ramos, Anqi Li, Abhishek Gupta
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.03622
• Pdf link: https://arxiv.org/pdf/2611.03622
• Abstract Vision-Language Models (VLMs) hold promise for zero-shot video task-progress estimation in robot learning, yet existing frameworks frequently suffer from confirmation bias, hallucinating progress in failed executions due to implicit success assumptions or relative temporal ranking designs. Naive prompting of frontier models collapses into a binary success detector that fails to capture nuanced, non-linear progress and failures. While post-training approaches can mitigate this through explicit training, they demand expensive real-world robotics data and require computationally intensive post-training for every individual base model. To resolve these limitations, we introduce D&A (Describe & Assess): our approach decouples progress estimation into an objective, task-grounded textual description of scene dynamics followed by an assessment phase using explicit rubrics. By compelling the VLM to synthesize an explicit, text-based definition of absolute success before scoring, the framework anchors evaluations to an external, global criterion rather than the video's internal timeline. Empirical results demonstrate that our method calibrates zero-shot estimates and mitigates confirmation bias, providing fine-grained, partial progress metrics that accurately penalize failed executions while correctly interpolating successful runs. Being purely prompt-based, D&A generalizes across tasks, embodiments, and benchmarks---matching or surpassing post-trained reward models---and its reward estimates enable efficient downstream reinforcement learning.
• 中文摘要 视觉语言模型（VLMs）有望用于机器人学习中的零样本视频任务进展估计，但现有框架常常存在确认偏误，因隐性成功假设或相对时间排名设计而误判失败执行的进展。对前沿模型的天真提示会沦为二元成功检测器，无法捕捉细致、非线性的进展与失败。虽然后训练方法可以通过显式训练来缓解这一问题，但它们需要昂贵的真实机器人数据，并且每个基模型都需要计算密集的后训练。为解决这些限制，我们引入了D&A（描述与评估）：我们的方法将进度估计解耦为客观、基于任务的场景动态文本描述，随后使用显式评分标准进行评估阶段。通过强制VLM在评分前综合明确的文本定义，框架将评估锚定在外部的全局标准上，而非视频的内部时间线。实证结果表明，我们的方法校准了零机会估计并减轻了确认偏误，提供了细粒度的部分进展指标，准确惩罚失败执行，同时正确插值成功运行。D&A纯基于提示，能够在任务、实践和基准测试中推广，匹配甚至超越训练后奖励模型------其奖励估计实现高效的下游强化学习。

FADE: Factored Ability--Difficulty Filtering for Online RLVR Scheduling

FADE：因式分解能力——在线RLVR排程的难度过滤

• Authors: Tomas Duncun, Jonah Ellison, Mateo Ruiz, Hanae Fujita, Ingrid Solberg
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03625
• Pdf link: https://arxiv.org/pdf/2611.03625
• Abstract Group-based reinforcement learning with verifiable rewards (RLVR) learns only from prompts whose sampled completions disagree: all-pass and all-fail groups produce zero relative advantage and burn generation budget without moving the policy. Adaptive schedulers try to steer rollouts toward informative prompts, yet they rely on per-prompt pass-rate trackers that entangle two quantities evolving at very different speeds---the policy's shifting competence across reasoning skills and each prompt's intrinsic difficulty. When a single exponential average or Beta counter tracks both at once, short memory scrambles the relative ordering of prompts whereas long memory reports stale pass rates. We introduce FADE (Factored Ability--Difficulty Estimation), a bilinear state-space filter that separates the two timescales. FADE represents each prompt's success logit as the inner product of a whitened $K$-dimensional skill loading with a fast-drifting policy ability vector, minus a slow-shrinking scalar item intercept. Using Pólya--Gamma augmentation, each minibatch of binary rollout outcomes updates the $K$-dimensional ability posterior in closed form via a rank-$K$ Woodbury step, instantly transferring competence shifts to all unobserved prompts that share those skills while leaving item intercepts stable. From the resulting posterior mean and variance we derive a closed-form utility for the probability of drawing a non-degenerate rollout group. Across five reasoning benchmarks, two base models, and four scheduling paradigms under a matched rollout budget, FADE consistently improves downstream accuracy and calibration while adding less than $0.6$\,ms per step.
• 中文摘要 基于群体的可验证奖励强化学习（RLVR）仅从采样完成度不一致的提示中学习：全通过和全失败组在不改变策略的情况下，不产生零相对优势和消耗生成预算。自适应调度器试图引导推送向信息型提示，但它们依赖于每个提示的通过率追踪器，将两个以截然不同速度变化的量纠缠在一起---策略在推理技能和每个提示内在难度间的能力变化。当单个指数平均或Beta计数器同时跟踪两者时，短内存会扰乱提示的相对顺序，而长内存则报告过期通过率。我们引入了FADE（因式分解能力-难度估计），这是一种双线性状态空间过滤器，将两个时间尺度分开。FADE 表示每个提示的成功日志为一个白色$K$维技能加载与快速漂移策略能力向量的内积，减去缓慢缩小的标量项目拦截。利用 Pólya-Gamma 增强，每个二元展开结果的迷你批次通过秩$K Woodbury步骤后期更新$K$维能力，即时将能力转移至所有共享该技能的未观察提示，同时保持项目拦截稳定。根据所得的后验均值和方差，我们推导出一个闭式效用，用于绘制非退化展开组的概率。在五个推理基准、两个基础模型和四个配对部署预算下的调度范式中，FADE 持续提升下游准确性和校准，同时每步增加不到 0.6 美元毫秒。

Not All Reasoning Generalizes: A Case Study on Learning Generalizable Reasoning Strategies in RLVR

并非所有推理都能推广：RLVR中学习可推广推理策略的案例研究

• Authors: Ziqi Pang, Yu-Xiong Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03627
• Pdf link: https://arxiv.org/pdf/2611.03627
• Abstract Reinforcement learning with verifiable rewards (RLVR) can enable generalizable improvement for large language model (LLM) reasoning. This paper studies how to incentivize such generalization via reasoning strategies. We study this question with a controllable equation-solving framework: finding multiple inputs x for equations f(x)=y, where f is a non-bijective string function. This setting makes reasoning strategies observable while supporting a clear definition of easy-to-hard out-of-distribution (OOD) shifts, e.g., more solutions, longer y, etc. We uncover a subtle RLVR failure: it teaches the model to solve in-distribution tasks with high accuracy and structured step-by-step derivation, yet under OOD shifts, the same model abandons derivation and collapses to ineffective brute-force enumeration, causing sharp accuracy drops. This observation reveals the deeper challenges of RLVR generalization as it is neither incapable base model nor reward hacking, since the model succeeds in-distribution with the correct reasoning strategy. The key to mitigating this failure is an indirect intervention that surprisingly helps: co-training with bijective inversion tasks. Although these bijective tasks are actually not uniformly harder and even further from the non-bijective OOD evaluation, they make derivation more certain and substantially restore the correct strategy and accuracy under OOD shifts. This effect further motivates our theoretical analysis: outcome-only RLVR can shape reasoning processes through implicitly crediting the effective reasoning strategies more, even without process supervision. To conclude, our analysis framework and findings systematically reveal the crucial role and incentivization of reasoning strategies in RLVR generalization under OOD shifts. Code is available at this https URL.
• 中文摘要 带有可验证奖励的强化学习（RLVR）可以实现大型语言模型（LLM）推理的可推广性改进。本文研究如何通过推理策略激励此类泛化。我们用可控方程求解框架研究该问题：在方程f（x）=y时寻找多个输入x，其中f是非双射字符串函数。这种设定使推理策略可观察，同时支持对易至难分布外（OOD）位移的清晰定义，例如更多解、更长的Y等。我们发现RLVR的一个微妙失败：它教会模型以高精度和结构化的逐步推导解决分布内任务，但在OOD偏移下，同一模型放弃推导，转而采用无效的暴力破解枚举，导致准确率急剧下降。这一观察揭示了RLVR推广的更深层挑战，因为它既不是无能的基模型，也不是奖励黑客，因为该模型在分配中通过正确的推理策略是成功的。缓解这一失败的关键是一个间接干预，出人意料地有帮助：与双射反演任务共训练。尽管这些双射任务实际上并非均匀更难，甚至与非双射OOD评估相距更远，但它们使推导更为确定，并在OOD转移下显著恢复了正确的策略和准确性。这一效应进一步激发了我们的理论分析：仅结果RLVR可以通过隐式地更多地归功于有效的推理策略，从而塑造推理过程，即使没有过程监督。总之，我们的分析框架和发现系统地揭示了推理策略在OOD转移下RLVR推广中的关键作用和激励作用。代码可在此 https URL 获取。

WebHarbor: Docking Real Websites for Evolving GUI Agent Environments

WebHarbor：对接真实网站以实现图形界面代理环境的演进

• Authors: Zhaoyang Wang, Qianhui Wu, Boyu Gou, Demi Wang, Jiacheng Zhang, Peichun Hua, Qize Li, Shi Qiu, Yifan Jiang, Yifan Zhang, Zexu Jin, Baolin Peng, Bolin Shen, Chetan Bansal, Dehan Kong, Deren Lei, Dongshen Peng, Fengqing Jiang, Hao Cheng, Haoqing Wang, Jiaming Shan, Jicheng Wang, Jingguang Zhou, Jixuan He, Kaide Zeng, Kangyu Zhu, Lama Moukheiber, Mufan Qiu, Peng Xia, Qihan Sun, Qingwei Lin, Qiushi Sun, Ruichen Zhang, Shuqing Luo, Si Qin, Sicheng Fan, Tao Xiong, Wenlin Yao, Xinyu Zhang, Xinyue Zeng, Xuanrui Li, Xuchao Zhang, Yian Wang, Yichao Liu, Yiming Liang, Yiyang Zhou, Yizhu Jiao, Yuetai Li, Yurun Chen, Yuxuan Zhang, Zheyuan Yang, Zhiyuan Yao, Zhongyang Li, Zihao Li, Zili Wang, Jianfeng Gao, Huaxiu Yao
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.03634
• Pdf link: https://arxiv.org/pdf/2611.03634
• Abstract GUI web agents require realistic and interactive environments that remain stable during both evaluation and training. Live website evaluation suffers from changing content, access restrictions, and limited coverage of authenticated workflows, while existing synthetic environments for training often lack functional depth and visual fidelity. We propose WebHarbor, a suite of 153 executable and dockerized website environments that faithfully reconstruct the visual interfaces, content, and functional workflows of real-world web services under deterministic control. We build WebHarbor through task-driven human-agent collaboration, where environments evolve as new tasks expand their functional scope, coding agents implement the required changes, and human contributors audit visual fidelity, functional correctness, and task validity. The resulting suite comprises 3,518 tasks spanning multi-step and post-authentication workflows, where existing works often fail to cover. Beyond reliable evaluation, WebHarbor provides high-fidelity synthetic web environments for cost-efficient reinforcement learning (RL). We demonstrate that a Qwen 4B-level GUI agent trained with RL in WebHarbor generalizes to unseen tasks and performs competitively with leading open-source GUI web agents across four benchmarks. The code and data are publicly available at this https URL.
• 中文摘要 图形界面网页代理需要真实且交互的环境，在评估和培训过程中保持稳定。实时网站评估存在内容变化、访问限制以及认证工作流覆盖有限的问题，而现有的合成训练环境往往缺乏功能深度和视觉真实度。我们提出了WebHarbor，这是一套包含153个可执行和docker化网站环境的套件，忠实地重建了现实世界网络服务在确定性控制下的可视化界面、内容和功能工作流。我们通过任务驱动的人工-代理协作构建WebHarbor，环境随着新任务功能范围的扩展而演进，编码代理实现所需变更，人工贡献者审计视觉忠实度、功能正确性和任务有效性。最终套件包含3,518个任务，涵盖多步及认证后工作流，现有工作常常未能覆盖这些领域。除了可靠评估外，WebHarbor还提供高保真合成网络环境，用于成本效益高的强化学习（RL）。我们展示了在WebHarbor中用RL训练的Qwen 4B级GUI代理，能够推广到看不见的任务，并在四个基准测试中与领先的开源GUI网络代理竞争。代码和数据在此HTTPS网址公开。

EgoActionBench: Benchmarking Dense Action Captioning for Egocentric Human Videos

EgoActionBench：为自我中心的人类视频进行密集动作字幕的基准测试

• Authors: Yujiao Shen, Jiahe Song, Haitian Li, Jingkang Yang, Ziwei Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2611.03646
• Pdf link: https://arxiv.org/pdf/2611.03646
• Abstract Learning from human data offers a scalable path toward embodied intelligence, yet it requires models to accurately understand fine-grained human actions from egocentric observations. However, existing benchmarks remain limited in action atomicity, temporal precision, and hand-action association. We introduce EgoActionBench, a benchmark that represents continuous human manipulation as temporally localized atomic actions associated with the left hand, right hand, or both hands. EgoActionBench contains 1,407 episodes with 41,132 human-refined action annotations. To evaluate such dense predictions, we propose an evaluation protocol that retrieves temporally eligible candidates, selects a prediction by content agreement, and scores semantic fidelity and temporal overlap alongside action precision and recall, jointly capturing what happens, when it happens, and which hand performs it. We benchmark 20 baseline models and systems on EgoActionBench and further adapt two backbones, Qwen3-VL-4B and LLaVA-OneVision-2-8B, through supervised fine-tuning followed by reinforcement learning on automatically annotated training episodes. EgoAction improves EgoActionScore by 15.9 and 23.2 points on the two backbones, and both adapted models outperform all open-source baselines. Our experiments reveal substantial limitations of current models in action atomicity, temporal precision, and hand-action association, highlighting the gap between general video understanding and the action understanding required for embodied intelligence.
• 中文摘要 从人类数据学习提供了一条可扩展的路径，但需要模型从自我中心的观察准确理解细粒度人类行为。然而，现有基准在动作原子性、时间精度和手-动作关联方面仍有限。我们介绍EgoActionBench，一个基准测试，将持续的人类操作表现为与左手、右手或双手时间局域化的原子动作。EgoActionBench包含1,407集，41,132条人类细化动作注释。为评估如此密集的预测，我们提出了一种评估协议，检索时间上符合时间条件的候选者，按内容一致选择预测，并与动作精准度和回忆性同时评分语义忠实度和时间重叠，共同捕捉发生了什么、何时发生以及由哪只手执行。我们在EgoActionBench上基准测试了20个基线模型和系统，并通过监督微调和自动注释训练集的强化学习，进一步调整了两个骨干链Qwen3-VL-4B和LLaVA-OneVision-2-8B。EgoAction在两个骨干网上分别提升了15.9分和23.2分，且两款适应模型均优于所有开源基线。我们的实验揭示了当前模型在动作原子性、时间精度和手-动作关联方面存在显著局限，凸显了一般视频理解与具身智能所需动作理解之间的差距。

Keyword: diffusion policy

Does Validation Loss Predict Real-Robot Success? Evidence from Four Imitation-Learning Policies on a Low-Cost Arm

验证丢失能否预测真实机器人的成功？来自低成本机械臂上的四种模仿学习策略的证据

• Authors: Simon Cowling
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.00283
• Pdf link: https://arxiv.org/pdf/2611.00283
• Abstract Held-out validation loss is the standard offline signal for choosing checkpoints and comparing models, but its relation to closed-loop success on a real robot is rarely measured. We measure it for ACT, Diffusion Policy, SmolVLA and $\pi_{0.5}$ (action expert only), trained on nested subsets of 12 to 200 demonstrations of a pick-and-place task on an SO-101 arm. We score 82 checkpoints on 25 held-out demonstrations, with each policy's training loss and with the error of the actions it outputs, and compare the scores with 930 bench rollouts. We also score all 700 intermediate checkpoints of 70 training runs, and recover the ball positions of every training, held-out and bench episode from the front camera. Within each policy, hold-out action error correlates with bench success (Spearman $\rho$ from -0.23 to -0.88), but no more strongly than the number of training episodes alone, and not at all among seeds trained on the same data. Across policies the ranking is inverted ($\rho$ = +0.90 at 200 episodes): the policy with the lowest hold-out error succeeded in 3 of 110 rollouts. The most common failure, the arm never leaving its rest pose, ended 96% of rollouts for Diffusion Policy trained on untrimmed demonstrations and 47% for SmolVLA, and cannot occur in a hold-out set made of demonstrations. Among rollouts in which the arm moved, SmolVLA's hold-out action error predicts success well ($\rho$ = -0.81). For the diffusion and flow-matching policies, hold-out loss rises two- to five-fold during training while hold-out action error falls by up to 29%. Held-out metrics are a sound diagnostic of learning but no substitute for evaluation on the robot, and we recommend reporting initiation failure and success among started rollouts as separate bench outcomes.
• 中文摘要 Heldout验证损耗是选择检查点和比较模型的标准离线信号，但它与真实机器人闭环成功的关系很少被测量。我们对ACT、扩散政策、SmolVLA和$\pi_{0.5}$（仅限行动专家）进行测量，训练基于SO-101臂上12至200次拾取放位任务的嵌套子集。我们在25个保留演示中计分82个检查点，包含每个策略的训练损失及其输出动作的误差，并比较930次工作台部署的得分。我们还对70次训练运行的700个中间检查点进行评分，并从前置摄像头恢复每个训练、保留和卧推的球的位置。在每个策略中，保留动作错误与实验台成功率相关（Spearman $\rho$从-0.23到-0.88），但不比单独训练集数强，且在相同数据训练的种子间完全不相关。在策略中排名是倒置的（$\rho$ = +0.90,200集）：保持错误最小的策略在110次推广中有3次成功。最常见的失败是臂永不离开静止姿态，导致96%的扩散策略在未修剪演示上训练的推广结束，47%结束了SmolVLA，且在演示组保留中无法发生。在臂移动的推广中，SmolVLA的保留动作错误预测成功率较好（$\rho$ = -0.81）。在扩散和流量匹配政策中，保留行动损失在培训期间增加2到5倍，而保留行动误差减少多达29%。暂停指标是学习的合理诊断，但无法替代机器人评估，我们建议将启动失败和成功作为独立的实验台结果报告。

Evaluating Tactile Foundation Models for End-to-End Imitation Learning in Contact-Rich Manipulation

评估触觉基础模型用于接触丰富操作中的端到端模仿学习

• Authors: Masaki Murooka, Ryoichi Nakajo, Tomohiro Motoda, Koshi Makihara, Hanbit Oh, Keisuke Shirai, Yukiyasu Domae
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.00957
• Pdf link: https://arxiv.org/pdf/2611.00957
• Abstract Tactile sensing plays an important role in imitation learning for contact-rich manipulation by providing direct observations of physical contact with objects. Recently, several tactile foundation models (TFMs) have been proposed to learn general-purpose tactile representations through pretraining on large-scale tactile datasets. However, how these pretrained tactile representations should be leveraged for end-to-end imitation learning remains insufficiently explored. In this work, we integrate multiple TFMs into Diffusion Policy and systematically evaluate their effectiveness for contact-rich manipulation under common experimental settings. We further investigate adaptation strategies, including encoder fine-tuning and task-specific mid-training, to understand how pretrained tactile representations can be effectively used for downstream imitation learning. Through extensive evaluation across diverse manipulation tasks in both real-world and simulation environments, including more than 5,000 real-world policy rollouts, we provide empirical insights into the use of TFMs for end-to-end imitation learning. Project page: this https URL
• 中文摘要 触觉感知在模拟学习中发挥重要作用，通过直接观察与物体的物理接触，实现接触丰富操作。最近，已有若干触觉基础模型（TFM）被提出通过在大规模触觉数据集上进行预训练来学习通用触觉表征。然而，这些预训练触觉表征如何应用于端到端的模仿学习，仍未充分探讨。本研究将多个TFM集成到扩散策略中，并系统评估它们在常见实验环境中对丰富接触操作的有效性。我们还进一步探讨了适应策略，包括编码器微调和任务特定中训练，以理解预训练触觉表征如何有效用于下游模仿学习。通过在现实和模拟环境中对多种操作任务的广泛评估，包括超过5000次真实世界政策的推广，我们提供了关于TFM在端到端模仿学习中的实证见解。项目页面：此 https URL

Shades of Success: Evaluating Manipulation Quality from Object-Centric Constraints

成功的阴影：从以对象为中心的约束中评估操作质量

• Authors: Lorena Maria Genua, Sarvesh Prajapati, Ananya Trivedi, Taskin Padir
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.01250
• Pdf link: https://arxiv.org/pdf/2611.01250
• Abstract As learned manipulation policies become increasingly capable of solving practical manipulation tasks, assessing how well a task is performed becomes a fundamental challenge. While binary labels confirm task success, they cannot distinguish successful executions with different levels of geometric precision. To address this, we introduce SPECTRA, an object-centric framework for quantifying execution quality in manipulation of rigid objects. From a small set of oracle demonstrations generated in simulation, SPECTRA represents object interactions through progress-aligned relative poses. SPECTRA identifies which relative-pose directions are constrained and their admissible variations from demonstrations. These constraints establish a reference for evaluating robotic manipulation tasks. To score an unseen execution, deviations beyond the admissible variation are summarized by a Task Quality Index Q in [0,1]. Across seven simulated tasks, SPECTRA recovers constraints with 94% accuracy using 30 demonstrations per task. On rollouts from three learned policies, its scores achieve pooled Spearman correlations with human judgments of p=0.81 over all rollouts and p=0.61 among successful rollouts, exceeding all baselines in both subsets. Using the task reference learned in simulation, we use SPECTRA to select the highest-scoring demonstrations for training a diffusion policy on hardware. Exploratory results show higher success rate than training on randomly selected demonstrations.
• 中文摘要 随着学习的操作策略越来越能解决实际操作任务，评估任务执行情况成为一项根本挑战。虽然二进制标签确认任务成功，但无法区分几何精度不同水平的成功执行。为此，我们引入了SPECTRA，一种以对象为中心的框架，用于量化刚性对象操作中的执行质量。SPECTRA通过模拟生成的少量预言机演示表示对象交互，通过进度对齐的相对姿态。SPECTRA识别哪些相对姿态方向受限，以及它们与演示的可接受变异。这些约束为评估机器人操作任务提供了参考。为了对未见执行进行评分，超出可接受变异的偏差通过任务质量指数Q在[0,1]中汇总。在七个模拟任务中，SPECTRA通过每项任务30个演示，以94%的准确率恢复约束。在三个学习策略的部署中，其得分与人类判断的p=0.81（所有部署）和成功推广的p=0.61形成合并Spearman相关，超过了这两个子集的所有基线。利用模拟中学到的任务参考，我们利用SPECTRA选择得分最高的演示，用于在硬件上训练扩散策略。探索性结果显示，其成功率高于随机选择演示的训练。

Visuo-tactile Diffusion Policy for Robotic Manipulation with Auxiliary Tactile Dynamics Prediction

带有辅助触觉动力学预测的Visuo-触觉扩散策略

• Authors: Kaihong Ouyang, Zihao Su, Chengxuan Zheng, Weiyi Lai, Dong Zhou
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.02677
• Pdf link: https://arxiv.org/pdf/2611.02677
• Abstract Long-horizon robotic assembly requires reliable perception and action across grasping, alignment, and insertion. Vision captures global object pose and scene context, where as fingertip touch provides local contact cues that are especially useful as contact states change. We present a visuotactile diffusion policy that combines four-step histories from two camera views and two fingertip tactile images to condition action-sequence generation. A recurrent temporal representation integrates recent observations from both modalities. To encourage it to capture how contact evolves under action, an auxiliary objective predicts fingertip-image changes two steps ahead from the temporal representation and demonstrated actions. This objective is used only during training and adds no inference-time computation. Across three independent training seeds in MuJoCo assembly experiments, the full model achieves an average success rate of $59.0\% \pm 2.6\%$, improving on the visual-only baseline ($42.7\% \pm 1.5\%$) by 16.3 percentage points.
• 中文摘要 长视距机器人组装需要在抓取、对准和插入过程中具备可靠的感知和动作。视觉捕捉全局物体姿态和场景上下文，而指尖触摸则提供局部接触线索，尤其适用于接触状态变化时。我们提出了一种视觉触觉扩散策略，结合了两个摄像头视角和两个指尖触觉图像的四步历史，以条件动作序列生成。循环时间表征整合了两种模式的最新观察。为鼓励其捕捉接触在动作中演变，辅助目标预测指尖图像在时间表征和演示动作之后的两步变化。该目标仅在训练中使用，不增加推理时间计算。在MuJoCo组装实验中的三个独立训练种子中，完整模型的平均成功率为$59.0\% \pm 2.6\%$，比仅视觉基线（$42.7\% \pm 1.5\%$）提升了16.3个百分点。