生成时间: 2026-11-06 00:14:39 (UTC+8); Arxiv 发布时间: 2026-11-05 20:00 EST (2026-11-06 09:00 UTC+8)

今天共有 27 篇相关文章

Keyword: reinforcement learning

On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation

关于后强化学习整合中策略提炼的推广局限性

• Authors: Yichen Wang, Yufa Zhou, Siqi Zhu, Haozheng Luo, Jingzhi Bao, Zhichao Liu, Zijun Wei
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05609
• Pdf link: https://arxiv.org/pdf/2611.05609
• Abstract Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) to consolidate capabilities from multiple teachers. Although prior work reports near-perfect in-distribution (IND) distillation performance, we find that this can mask incomplete capability consolidation beyond the supervised distribution. A preliminary study on synthetic reasoning shows that OPD does not fully transfer teachers' generalization capabilities. We then extend this investigation to real-world tasks, evaluating extrapolative and compositional generalization across mathematical reasoning, coding, agent, and long-context capabilities using Qwen3 models. Across extrapolative settings, OPD and MOPD reach 96.04-102.73% of their teachers' IND scores. Yet their out-of-distribution (OOD) scores remain 2.45-16.25% below the base model's. Across three compositional settings, MOPD improves IND performance over the base model, but its performance on composed tasks as OOD remains 18.19-24.34% below that of the best teachers. To understand these gaps, we examine teacher capability and data coverage as two potential bottlenecks. Jointly alleviating these bottlenecks eliminates OOD forgetting in our experiments and suggests the potential for generalization beyond the teacher. (M)OPD also reduces OOD forgetting on average relative to its reinforcement learning (RL) baselines. Analysis of the associated update geometry reveals smaller and more spectrally concentrated updates on average than those of the RL baselines, along with greater shared update structure under MOPD. Motivated by these findings, we propose geometry-based patching and base-preserving replay to improve OOD performance while largely preserving IND performance. Our findings identify a blind spot in capability consolidation via (M)OPD, highlighting the need for algorithms that explicitly support OOD retention and generalization.
• 中文摘要 多教师政策提纯（MOPD）扩展了策略内提纯（OPD），以整合多位教师的能力。尽管先前研究报告了近乎完美的分布内（IND）蒸馏表现，但我们发现这可能掩盖监督分布之外的能力整合不完全。一项关于合成推理的初步研究表明，OPD并未完全转移教师的泛化能力。随后我们将该研究扩展到现实任务，利用Qwen3模型评估数学推理、编码、代理和长上下文能力的外推和组合推广。在外推场景中，OPD和MOPD达到教师IND分数的96.04-102.73%。然而，他们的分布外（OOD）分数仍比基础模型低2.45%-16.25%。在三种组合设置中，MOPD提升了IND相较基础模型的表现，但在组合任务中，作为OOD的表现仍比最佳教师低18.19%-24.34%。为理解这些差距，我们考察教师能力和数据覆盖率作为两个潜在瓶颈。共同缓解这些瓶颈可消除实验中的OOD遗忘，并暗示了超越教师的泛化潜力。（M）OPD还相较于强化学习（RL）基线平均减少OOD遗忘率。对相关更新几何的分析显示，平均更新比强化学习基线更小且频谱集中度更高，MOPD下的共享更新结构也更大。基于这些发现，我们提出基于几何的补丁和基础保留回放，以提升OOD性能，同时在基本保持IND性能的同时。我们的发现指出，通过（M）OPD实现能力整合存在盲点，凸显了明确支持OOD保留和泛化的算法的必要性。

Revisiting Critic Estimation in Soft Actor-Critic Algorithms

软演员-批评算法中的批评者估计重审

• Authors: Jan Felix Kleuker, Koen Ponse, Aske Plaat, Thomas M. Moerland
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05666
• Pdf link: https://arxiv.org/pdf/2611.05666
• Abstract Soft Actor-Critic (SAC) is a widely used deep reinforcement learning algorithm that combines entropy regularization with an actor-critic architecture. Standard implementations rely on one-step temporal-difference learning of Q-functions, yet alternative critic design choices remain underexplored. In this work, we revisit three such choices: learning soft advantages instead of soft Q-values, the use of multi-step return targets, and the training of multi-step critics from replay buffer data. Keeping the policy objective fixed, we compare the six critic variants these choices generate: (i) advantage and (ii) dueling-advantage critics learned from one-step transitions, and $\lambda$-return and soft Generalized Advantage Estimation (GAE) critics learned either (iii, iv) on-policy or (v, vi) from replayed trajectory segments. Each variant is evaluated across an entire random hyperparameter search on fourteen tasks, treating the hyperparameters as part of the algorithm. Learning soft advantages instead of soft Q-values does not improve performance, and the soft GAE critic adds nothing over the simpler $\lambda$-return critic. What does improve performance, most clearly on continuous control, is training a $\lambda$-return critic on replayed trajectory segments, even without importance weighting or trace cutting. This extends earlier evidence that uncorrected multi-step targets work well under replay to the full soft $\lambda$-return, entropy terms included. By design, the underlying ideas studied here extend beyond SAC, and our empirical findings aim to encourage their application to a broader class of reinforcement learning algorithms.
• 中文摘要 软演员-批判者（SAC）是一种广泛使用的深度强化学习算法，结合了熵正则化与演员-批判者架构。标准实现依赖于Q函数的一步时间差分学习，但其他批判者设计选择仍未被充分探索。本研究回顾了三种选择：学习软优势而非软Q值、使用多步返回目标，以及从重放缓冲区数据训练多步批判者。保持策略目标固定，我们比较这些选择产生的六种批判变体：（i）优势和（ii）从一步转移中获得的优势和（ii）对抗优势批评者，以及$\lambda$返回和软广义优势估计（GAE）批评者从重放轨迹段中获得策略或（v、vi）。每个变体在十四个任务中对整个随机超参数搜索进行评估，并将超参数视为算法的一部分。学习软优势而非软Q值并不会提升性能，软GAE批评者对简单的$\lambda$返回批判者没有任何贡献。真正能提升表现的，尤其是在连续控制中最明显的是训练$\lambda$返回批评者，针对重放轨迹段，即使不进行重要性加权或轨迹切割。这将之前证据扩展到未校正多步目标在回放下良好运行的证据，适用于完整的软$\lambda$返回，包括熵项。设计上，这里研究的基础思想超越了SAC，我们的实证发现旨在鼓励其应用于更广泛的强化学习算法类别。

OdorSpaces: Visual-Olfactory Embodied Navigation in 3D Environments

OdorSpaces：三维环境中的视觉嗅觉具身导航

• Authors: Changyeon Lee, Kelin Yu, Zheng Wei, Shuna Ni, Ruohan Gao
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05738
• Pdf link: https://arxiv.org/pdf/2611.05738
• Abstract Olfaction provides critical information about airborne chemical cues, yet it remains largely underexplored in embodied AI, with limited prior machine olfaction work focusing primarily on semantic odor perception. We introduce OdorSpaces, a physics-based simulation platform and benchmark for visual-olfactory embodied navigation. OdorSpaces augments Habitat-Sim with 3D odor transport and dispersion simulated by Fire Dynamics Simulator (FDS), enabling agents to navigate photorealistic indoor environments using both visual and olfactory observations. The benchmark includes two representative source-seeking scenarios: invisible gas-leak localization and early-stage fire localization. We develop an initial multimodal reinforcement learning policy that integrates visual observations and olfactory measurements over time. Experiments show that the policy outperforms a diverse set of baselines, establishing a strong reference for future research on visual-olfactory embodied navigation.
• 中文摘要 嗅觉提供了关于空气化学线索的关键信息，但在具身人工智能中仍大多未被充分探索，此前机器嗅觉工作主要聚焦于语义气味感知。我们介绍了OdorSpaces，一个基于物理的模拟平台和视觉嗅觉具象导航的基准测试。OdorSpaces通过火力动力模拟器（FDS）模拟的3D气味传输和扩散来增强栖息模拟，使智能体能够利用视觉和嗅觉观察在逼真的室内环境中导航。基准包括两种代表性的寻源场景：隐形气体泄漏定位和早期火灾定位。我们制定了初步多模态强化学习策略，整合视觉观察与嗅觉测量随时间变化。实验显示该策略优于多种基线，为未来视觉嗅觉具象导航研究树立了有力参考。

Constraint-Aware Evaluation of Reinforcement Learning and Unsupervised Anomaly Detection for Security Operations Alert Triage

对强化学习和无监督异常检测的约束感知评估，用于安全操作警报分流

• Authors: Radityo Ghifiardi, Rifny Rachman
• Subjects: Subjects: Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2611.05746
• Pdf link: https://arxiv.org/pdf/2611.05746
• Abstract Security Operations Centers (SOCs) must identify real threats from large alert streams, but limited analyst capacity means that unnecessary escalations can delay more serious investigations. Existing studies often frame SOC alert triage as classification or anomaly detection, but both views are incomplete for production SOCs, where the central problem is selecting alerts worth escalating under limited capacity, asymmetric error costs, and workload constraints. This paper addresses that gap by formalizing alert triage as a constrained decision problem and evaluating three approaches: an offline reinforcement learning policy trained on alert embeddings, an Isolation Forest model built on operational features, and an autoencoder trained on historical benign alerts. These methods represent different triage assumptions: learning actions from reward, isolating unusual operational patterns, and detecting deviations from benign behavior. We evaluate all methods on 74,909 analyst-labeled production alerts under a strict temporal split and benign-escalation constraint. Because the test set contains 503 unique feature patterns, we primarily rank clusters of similar alerts. At the main operating point, all three methods satisfy the constraint, with the reinforcement learning policy achieving the lowest benign-escalation fraction at 0.000, compared with 0.007 for the autoencoder and 0.041 for Isolation Forest. The methods also prioritize largely different alert clusters, showing complementary rather than redundant coverage. These results suggest that production SOC triage can benefit from constraint-aware evaluation and combined or adaptive deployment of different model families.
• 中文摘要 安全运营中心（SOC）必须识别来自大型警报流的真实威胁，但分析人员能力有限，不必要的升级可能延迟更严肃的调查。现有研究通常将SOC警报分流定位为分类或异常检测，但这两种观点在生产SOC中均不完整，因为核心问题是在有限容量、非对称错误成本和工作负载约束下选择值得升级的警报。本文通过将警报分诊形式化为受限决策问题，并评估三种方法，解决了这一空白：基于警报嵌入训练的离线强化学习策略、基于操作特性的隔离森林模型，以及基于历史良性警报训练的自编码器。这些方法代表了不同的分诊假设：从奖励中学习动作、隔离异常操作模式以及检测偏离良性行为。我们在严格的时间分割和良性升级约束下，评估了74,909个分析师标记的生产警报。由于测试集包含503个独特特征模式，我们主要对相似警报的集群进行排名。在主要操作点，三种方法均满足约束，强化学习策略的良性升级比例最低，为0.000，而自编码器为0.007，隔离森林为0.041。这些方法还优先考虑了主要不同的警报集群，显示出互补而非冗余覆盖。这些结果表明生产SOC分诊可以通过约束感知评估以及不同模型家族的组合或自适应部署而受益。

Energy-Efficient Reinforcement Learning for Large Language Models

大型语言模型的节能强化学习

• Authors: Rahul Sharma, Nitin Vegesna, Yanxin Zhang
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2611.05748
• Pdf link: https://arxiv.org/pdf/2611.05748
• Abstract Reinforcement learning for language models alternates between generating responses and training the policy. These stages place different demands on GPUs, creating opportunities to reduce clock frequency with little loss in throughput. Selecting frequencies online is difficult because response lengths change and frequency trials themselves consume time and energy. We present EE-RL (energy-efficient RL), an online controller that selects separate frequencies for generation and training during RL execution. It compares candidate settings with a prediction of the default GPU governor on the same token workload, and keeps the most energy-efficient setting that meets a throughput threshold. The controller then monitors throughput and energy and adjusts frequencies as the workload changes. Fixed-frequency sweeps on H100 and GB200 show GPU energy savings of up to 28.2% per valid training token across synchronous and asynchronous execution, while largely preserving throughput. EE-RL turns this headroom into online savings, reducing GPU energy per valid training token by up to 16.9% at a modest throughput cost.
• 中文摘要 语言模型的强化学习在生成响应和训练策略之间交替进行。这些阶段对GPU提出了不同的要求，创造了降低时钟频率且吞吐量损失极小的机会。在线选择频率较为困难，因为响应长度会变化，频率试验本身会消耗时间和能量。我们介绍了EE-RL（节能RL），这是一个在线控制器，在强化学习执行中选择不同的频率进行生成和训练。它将候选设置与同一令牌工作负载下默认GPU调速器的预测进行比较，并保留满足吞吐量阈值的最节能设置。控制器随后监控吞吐量和能耗，并根据工作负载变化调整频率。H100和GB200的固定频率扫描显示，GPU在同步和异步执行中每个有效训练令牌节省高达28.2%的能量，同时基本保持吞吐量。EE-RL将这些余裕转化为在线节省，每个有效训练令牌的GPU功耗可降低16.9%，且吞吐量成本适中。

Diversity-Aware Experience Replay via Statistical Leverage Scores

通过统计杠杆评分实现多样性感知体验回放

• Authors: Shaifalee Saxena, Alexander Scheinker
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.05929
• Pdf link: https://arxiv.org/pdf/2611.05929
• Abstract Off-policy deep reinforcement learning trains on random samples from a replay buffer of past experience. In physical robot settings, that buffer is dominated by routine: thousands of near-identical transitions from normal operation, while the rare events that define the edges of the task are rarely sampled. We propose scoring every stored transition by its statistical leverage, which measures how unusual it is relative to the buffer's own data, computed from the stored numbers alone, with no labels, rewards, or value network. We draw each mini-batch from a mixture of this score and uniform sampling. First, the natural way to build the score along the directions where the data varies most turns out to be blind when actions are stored as one-hot indicators, because indicator columns always win the variance ranking; we instead select the feature columns with the heaviest tails (largest kurtosis) where small far-away populations are revealed. Second, the importance weights that prioritized replay uses to keep gradients unbiased cancel the method exactly: with full correction, training matches uniform sampling, so we drop the correction and deliberately optimize the rebalanced objective. Third, when the rare situations of a log are relational, visible only in how a stretch of behavior relates to its goal, not in any single stored coordinate-per-transition scores are blind; applying the same tail rule to short trajectory windows described relative to their goal restores the signal, and on a three-joint arm that must push an object around an obstacle it raises the rare-task success from 0.49 to 0.69 against a labeled-oracle ceiling of 0.74.
• 中文摘要 非策略深度强化学习基于过去经验的回放缓冲区中的随机样本进行训练。在物理机器人环境中，该缓冲区被例行程序主导：数千个与正常操作几乎相同的过渡，而定义任务边缘的罕见事件很少被采样。我们建议通过统计杠杆来评分每个存储的过渡，衡量其相对于缓冲区自身数据的异常程度，仅凭存储的数字计算，不加标签、奖励或价值网络。我们从评分与均匀抽样的混合中抽取每个小批次。首先，当动作存储为单热指标时，基于数据变化最大方向构建评分的自然方法是盲的，因为指示列总是赢得方差排名;我们选择尾部最重的特征列（最大峰度），显示小型远方种群。其次，优先级重放用于保持梯度无偏的重要性权重完全抵消了该方法：在完全修正时，训练匹配均匀采样，因此我们省略修正，并有意优化重新平衡的目标。第三，当日志中罕见的情况是关系性的，仅通过一段行为与其目标的关系可见，而非任何单一存储的每次转换坐标分数时，是盲的;对相对于目标描述的短轨迹窗口应用相同的尾规则可恢复信号;在必须推动物体绕过障碍的三关节臂上，罕见任务成功率从0.49提升到0.69，而标记的oracle上限为0.74。

Learning Object-Centric Motion Priors from Human for Robotic Dexterous Manipulation

从人类学习以物体为中心的运动先验，以实现机器人灵巧操作

• Authors: Zhengdong Hong, Guofeng Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.05979
• Pdf link: https://arxiv.org/pdf/2611.05979
• Abstract Manipulating diverse objects with multi-fingered dexterous hands is challenging due to the high dimensionality and complex dynamics. Human-Object Interaction (HOI) datasets provide rich knowledge about task information and embodied interactions. Instead of solely imitating the human demonstrations, our method learns to holistically predict future hand-object states by leveraging these datasets. The predicted future states of the object can serve as a general-purpose reward term for reinforcement learning, reducing reliance on task-specific reward engineering and enhancing generalization across tasks. We conduct extensive experiments on three manipulation tasks in simulation and the real world. Our approach outperforms existing SOTA methods in both success rate and generalizability on novel objects. Furthermore, we validate the cross-embodiment compatibility of our methods by successfully deploying the skills on different robot hands.
• 中文摘要 由于高维度和复杂的动态，用多指灵巧的手操作多样化物体具有挑战性。人与物交互（HOI）数据集提供了丰富的任务信息和具身互动知识。我们的方法不仅仅模仿人类演示，而是通过利用这些数据集学习整体预测未来手部物体状态。物体的预测未来状态可作为强化学习的通用奖励项，减少对任务特定奖励工程的依赖，增强任务间的泛化。我们在模拟和现实世界中对三种操作任务进行了广泛实验。我们的方法在新对象的成功率和可推广性方面均优于现有SOTA方法。此外，我们通过成功部署技能在不同机器人手上验证了方法的跨身体兼容性。

Learning When to Think Aloud in Latent Reasoning

学会在潜在推理中大声思考

• Authors: Zhehao Zhou, Tongrui Su, Shengyu Zhu, Da Li, Zeyu Liu, Jun Zhu, Xueqi Cheng
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.06108
• Pdf link: https://arxiv.org/pdf/2611.06108
• Abstract Latent Chain-of-Thought (CoT) reasoning improves efficiency by shifting intermediate computation into continuous hidden states. However, fully latent methods often struggle on precision-critical tasks such as mathematical reasoning, while existing hybrid approaches retain substantial token-generation overhead. We introduce VITAL, an importance-guided hybrid reasoning framework that selectively verbalizes informative tokens as reasoning anchors while keeping most computation latent. A lightweight gating head dynamically selects between explicit token generation and direct hidden-state feedback. VITAL constructs hybrid trajectories from explicit CoT using token- importance estimation and trains the model through curriculum-based supervised training, followed by reinforcement learning with a reward that balances correctness and conciseness. Experiments show that VITAL consistently out- performs existing fully latent reasoning methods, achieving relative average accuracy gains of 9.6% on GPT-2 and 5.1% on Llama-3.2-1B at comparable reasoning lengths. Compared with hybrid baselines, VITAL achieves higher reasoning accuracy with substantially shorter reasoning trajectories on Qwen3-1.7B.
• 中文摘要 潜在思维链（CoT）推理通过将中间计算转移到连续的隐藏状态来提升效率。然而，完全潜伏方法在数学推理等高精度关键任务上常常遇到困难，而现有混合方法则保留了较大的标记生成开销。我们介绍了VITAL，一种重要性引导的混合推理框架，选择性地将信息性标记作为推理锚点，同时保持大部分计算处于潜在状态。一个轻量级门控头动态选择显式标记生成和直接隐藏状态反馈。VITAL 利用标记重要性估计从显式CoT构建混合轨迹，并通过基于课程的监督训练训练模型，随后进行强化学习，奖励平衡正确性和简洁性。实验显示，VITAL 在相同推理长度下，GPT-2 上 GPT-2 的相对平均准确率提升为 9.6%，Llama-3.2-1B 上提升 5.1%。与混合基线相比，VITAL 在 Qwen3-1.7B 的推理轨迹显著缩短，推理准确率更高。

From Handcrafted to Generative: RLVR-Optimized Icon Generation for Alexa+

从手工到生成：Alexa+ 优化的 RLVR 图标生成

• Authors: Rohan Sequeira, Matthew Rosenberg, Amin Mahmoudabadbozchelou, Oleg Poliannikov, Jonathan Rubin
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.06123
• Pdf link: https://arxiv.org/pdf/2611.06123
• Abstract Alexa+ devices with screens display animated icons during conversations to visually reinforce what is being said. The original set of these icons was hand-drawn by designers one at a time, a process that produces polished results but is time-consuming and cannot scale to the range of topics encountered in open-ended conversation. We describe an automated pipeline, deployed in production, that generates these vectorised animated icons from a text description alone and surfaces relevant animations to customers based on their ongoing conversation. The pipeline decomposes icon creation into a sequence of stages (image generation, semantic decomposition, vectorisation, styling, and animation). Its quality is determined at the very first of these, image generation, on which every downstream stage depends. We therefore focus on improving this first and most critical stage. Because the icons are rendered on small display canvases under strict design guidelines, generations are often unsuitable for production. Supervised fine-tuning is not viable, as no dataset of line-art icons exists under these constraints. Instead, we design objective verifiers that automatically flag generations violating the design guidelines, and use them to fine-tune a flow matching image generation model via reinforcement learning from verifiable rewards (RLVR). The fine-tuned model raises the pass rate from 15.7% to 58.7% (3.7x) under a VLM judge that is independent of the reward model. Both the trained model and the verifier are deployed in the production pipeline, where the verifier also serves as the automated quality gate before final design review.
• 中文摘要 带屏幕的Alexa+设备在对话中显示动画图标，以视觉强化所说内容。最初的这些图标由设计师逐个手绘，这一过程虽然精致，但耗时且无法适应开放式对话中遇到的各种话题。我们描述了一套自动化流程，已部署在生产环境中，仅凭文本描述生成矢量化动画图标，并根据客户当前对话向用户展示相关动画。该流程将图标创建分解为多个阶段（图像生成、语义分解、矢量化、样式和动画）。其质量在图像生成这一阶段的第一个阶段确定，而每个下游阶段都依赖于此。因此，我们专注于改进这一第一且最关键的阶段。由于图标在小型展示画布上，遵循严格的设计规范，生成过程通常不适合制作。监督微调不可行，因为在这些约束下不存在线条图像数据集。相反，我们设计了客观验证器，自动标记违反设计指南的生成，并通过可验证奖励强化学习（RLVR）微调流程匹配图像生成模型。微调模型在VLM评判下将通过率从15.7%提升至58.7%（3.7倍），该评判与奖励模型无关。训练模型和验证器均部署于生产流程中，验证器同时作为最终设计审查前的自动质量门。

ModalSearcher: Weaving Visual Evidence into Autonomous Cross-Modal Deep Search

ModalSearcher：将视觉证据融入自主跨模态深度搜索

• Authors: Zhihan Yin, Ruyi Li, Chengyu Fang, Yifeng Yao, Nonghai Zhang, Pengfei Xiong, Terry Zhang, Huishuai Zhang, Dongyan Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.06131
• Pdf link: https://arxiv.org/pdf/2611.06131
• Abstract Static and incomplete parametric knowledge limits the performance of multimodal large language models on novel-knowledge and knowledge-intensive tasks, especially those requiring multi-hop evidence acquisition. Existing multimodal agentic search methods often fix visual evidence at the input or a predefined trajectory position, making it difficult for agents to select search modalities as their information needs evolve and to handle complex real-world queries. We define this capability as autonomous cross-modal search: an agent chooses among direct reasoning, text-to-text search, text-to-image search, and reverse-image search, actively introduces visual evidence when appropriate, and switches between and integrates modalities across multiple hops. We propose ModalSearch-Pipeline, a scalable procedure for constructing an interleaved multimodal knowledge graph. It samples paths with diverse depths and modality compositions and uses backward question construction to obscure entity names and image targets that would expose reasoning shortcuts while retaining necessary disambiguating clues, yielding text-only, image-initialized, and arbitrarily interleaved multi-hop problems. We train the ModalSearcher Agent with supervised fine-tuning and reinforcement learning in a live Web environment, and construct ModalSearch-Bench. Experiments show that ModalSearcher-8B improves average accuracy by 16.2 points over Qwen3-VL-8B under the same agentic workflow across five multimodal knowledge-intensive QA benchmarks, with consistent gains on independently constructed active visual-search benchmarks.
• 中文摘要 静态且不完整的参数化知识限制了多模态大型语言模型在新颖知识和知识密集型任务中的表现，尤其是需要多跳证据采集的任务。现有的多模态代理搜索方法通常将视觉证据固定在输入端或预定义的轨迹位置，这使得代理在其信息需求变化时难以选择搜索模态，并难以处理复杂的现实世界查询。我们将此能力定义为自主跨模态搜索：代理在直接推理、文本对文本搜索、文本到图像搜索和反向图像搜索之间进行选择，适时主动引入视觉证据，并在多跳间切换和整合模态。我们提出了ModalSearch-Pipeline，这是一种可扩展的多模态知识图谱构建过程。它采样了深度和模态组合多样的路径，并利用逆向问题构造来遮蔽实体名称和图像目标，从而揭示推理捷径，同时保留必要的消歧线索，生成纯文本、图像初始化和任意交错的多跳问题。我们在实时Web环境中通过监督微调和强化学习训练ModalSearcher代理，并构建了ModalSearch-Bench。实验显示，在相同的代理工作流程下，ModalSearcher-8B在五个多模态知识密集型QA基准测试中，平均准确率比Qwen3-VL-8B提升16.2个百分点，独立构建的主动可视化搜索基准测试也持续提升。

CARE-FL: Federated Fine-Tuning of Multimodal LLMs in Resource-Constrained Edge Networks

CARE-FL：资源受限边缘网络中多模态LLM的联合微调

• Authors: Tianqi Zong, Baosheng Li, Nuocheng Yang, Binbin Chen, Marie Siew
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2611.06169
• Pdf link: https://arxiv.org/pdf/2611.06169
• Abstract Domain-specific adaptation of Multimodal Large Language Models (MLLMs) in privacy-sensitive edge environments requires federated fine-tuning, yet client selection remains challenging due to heterogeneous multimodal data and modality availability, dynamic system conditions, and evolving client utility during training. We propose Constraint-Aware Reinforcement Learning for Efficient Federated Fine-Tuning (CARE-FL), a constrained online client-selection framework for federated MLLM Low-Rank Adaptation (LoRA) fine-tuning. CARE-FL formulates client participation as a sequential decision process and employs Q-value decomposition to address the large combinatorial joint action space through scalable client-level long-term utility estimation. These utilities are learned online using a modified Double Deep Q-Learning (DDQL) scheme, while constraint-aware $\epsilon$-greedy execution selects feasible client subsets under per-round latency and bandwidth constraints. Complementing client selection, an adaptive aggregation mechanism further controls potentially unstable global updates and places greater weight on the previous model when needed. Experiments on the CrisisMMD disaster-response task in emulated heterogeneous wireless edge environments show that CARE-FL achieves higher and more stable test accuracy than the compared client-selection baselines across different system settings, while satisfying the imposed latency and bandwidth constraints.
• 中文摘要 在隐私敏感的边缘环境中，多模态大型语言模型（MLLM）的领域特定适配需要联邦微调，但由于多模态数据和模态的异构性、动态系统条件以及训练期间客户端效用的演变，客户端选择依然充满挑战。我们提出了约束感知强化学习以实现高效联邦微调（CARE-FL），这是一个用于联邦MLLM低秩适应（LoRA）微调的受限在线客户端选择框架。CARE-FL将客户端参与表述为顺序决策过程，并通过Q值分解通过可扩展的客户端级长期效用估计，解决庞大的组合联合行动空间。这些工具通过修改后的双深度Q学习（DDQL）方案在线学习，而约束感知的$\epsilon$贪婪执行则在每轮延迟和带宽约束下选择可行的客户端子集。作为客户端选择的补充，自适应聚合机制进一步控制潜在不稳定的全局更新，并在需要时对前一模型赋予更大权重。在模拟异构无线边缘环境中对CrisisMMD灾难响应任务的实验表明，CARE-FL在不同系统设置下的客户端选择基线相比，实现了更高且更稳定的测试精度，同时满足了施加的延迟和带宽约束。

Beyond Terminal Constraints: Reliable Robot Control with Learned Safe Sets

超越终端限制：可靠机器人控制与学习安全装置

• Authors: Gianni Lunardi, Elias Fontanari, Matteo Saveriano, Andrea Del Prete
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.06196
• Pdf link: https://arxiv.org/pdf/2611.06196
• Abstract Ensuring safety in advanced robot control methods such as Model Predictive Control (MPC) and Reinforcement Learning remains an open challenge. A common approach relies on control-invariant sets, yet their exact computation for nonlinear systems is generally intractable, and numerical approximations compromise theoretical guarantees. In this work, we show that safety does not need to rely exclusively on control invariance. We introduce novel classes of safe sets, each broader than classical control-invariant sets, and analyze corresponding MPC schemes that guarantee recursive feasibility by constraining different nodes of the prediction horizon rather than only the terminal state. Furthermore, when the safe set is an inner approximation of a suitable control-invariant set, our methods can ensure safety through a safe-abort strategy, even in the absence of recursive feasibility. We validate our approaches in extensive simulations on a 6-joint robot manipulator, showing fewer constraint violations than state-of-the-art methods while maintaining comparable tracking performance and computational efficiency. Hardware experiments further demonstrate the practicality of our schemes for real-world robotic applications.
• 中文摘要 在先进机器人控制方法（如模型预测控制（MPC）和强化学习中保障安全性仍是一个开放的挑战。一种常见方法依赖控制不变量集合，但其在非线性系统中的精确计算通常难以处理，数值近似也会影响理论保证。在本研究中，我们表明安全性不必完全依赖控制不变性。我们引入了新颖的安全集合类别，每个类别都比经典控制不变集合更宽广，并分析了通过约束预测视界不同节点而非仅限终端态来保证递归可行性的MPC方案。此外，当安全集是合适控制不变量集的内近似时，我们的方法可以通过安全中止策略确保安全性，即使不具备递归可行性。我们在六节机器人机械臂上进行了大量模拟验证了这些方法，显示出比最先进方法更少的约束违规，同时保持了相当的跟踪性能和计算效率。硬件实验进一步证明了我们方案在现实机器人应用中的实用性。

Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains

评分标准课程RL：利用不可验证领域的世代验证差距

• Authors: Sumeet Ramesh Motwani, Tejas Krishnan, Alistair Letcher, Charles London, Suhaas Bhat, Huitian Jiao, Philip Torr, Riashat Islam, Christopher Summerfield, Jakob Foerster, Christian Schroeder de Witt, Qilong Gu, Shital Shah
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.06244
• Pdf link: https://arxiv.org/pdf/2611.06244
• Abstract Reinforcement learning with verifiable rewards (RLVR) has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. The generation-verification gap, where judging good outputs is easier than producing them, offers a potential path to improvement. However, rubric-based judges are often imprecise and stationary, so policies can overfit or reward hack them. We introduce Rubric Curriculum RL (RcRL), which exploits this gap without new data, human annotations, or stronger teacher models. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring and a non-stationary objective. Whereas baselines plateau within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL achieves a 70.5% human win rate against the base model, with gains across benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all training baselines, including instance-specific rubrics.
• 中文摘要 带有可验证奖励的强化学习（RLVR）带来了数学和代码生成的显著进步。将这些成果推广到开放式领域仍然具有挑战性：缺乏实地验证，人工注释成本高昂，且学习到的奖励模型容易被奖励黑客攻击。世代验证差距，即判断良好输出比产出更易，提供了潜在的改进路径。然而，基于评分标准的评判往往不精确且固定，因此策略可能会过度拟合或奖励性黑客攻击他们。我们引入了评分标准课程强化学习（RcRL），利用这一差距，无需新数据、人工注释或更强的教师模型。我们将对推广的两两偏好与课程基于评分标准的课程相结合，产生比绝对评分和非平稳目标更具判别性的信号。基线在几百步内趋于平稳，而RcRL保持输出熵，并在1000步后持续提升。在创意写作方面，RcRL在基础模型中实现了70.5%的人类胜率，且在基准和评审各项指标上均有提升。在HealthBench上，RcRL较基础模型提升14.6%（完整组）和25.4%（共识子集），优于所有训练基线，包括实例特定评分标准。

Mass Producing Training Data with Surprising Side Effects

大量生产训练数据却带来令人惊讶的副作用

• Authors: Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann, Cassidy Laidlaw
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.06324
• Pdf link: https://arxiv.org/pdf/2611.06324
• Abstract Training data can have unexpected side effects on language model behavior outside the training domain: insecure code data can induce broad misalignment, and outdated bird names can induce a 19th-century persona. Understanding such generalization is hard because only dozens of isolated examples exist, each constructed by hand or discovered by accident. We propose automatically discovering such datasets with surprising side effects (DSSEs). We define three objectives for DSSEs: they should (1) elicit a target behavior, (2) read like ordinary post-training data, and (3) be surprising to human auditors reading the data. We operationalize each with LLM judges, and search for high-scoring datasets via evolution, reinforcement learning, and subliminal learning approaches. After re-scoring and filtering with held-out metrics, our pipeline produces a collection of 6,575 DSSEs across 8 behaviors, many of which generalize across model families and sizes. We find our automated metrics track ML researchers' judgments, and frontier models outperform humans at predicting side effects. These datasets form a testbed for studying generalization and lay the groundwork for predicting training side effects before model deployment. Explore them at this https URL.
• 中文摘要 训练数据可能对训练领域外的语言模型行为产生意想不到的副作用：不安全的代码数据可能导致广泛的错位，过时的鸟名可能诱发19世纪的形象。理解这种泛化很难，因为只有数十个孤立的例子，每个都是手工构建或偶然发现的。我们提出自动发现带有惊喜副作用的数据集（DSSE）。我们定义了DSSE的三个目标：它们应（1）引发目标行为，（2）阅读时像普通训练后数据，（3）对读数据的人类审计员来说是惊喜。我们用LLM评判对每个数据进行操作化，并通过进化、强化学习和潜意识学习方法搜索高分数据集。经过重新评分和过滤未完成指标后，我们的流程生成了涵盖8种行为的6,575条DSSE，其中许多跨模型族和规模进行了推广。我们发现自动化指标能追踪机器学习研究者的判断，前沿模型在预测副作用方面优于人类。这些数据集构成了泛化研究的测试平台，并为模型部署前的训练副作用预测奠定了基础。请访问此 https 网址探索它们。

Traffic-VL: Reinforced Vision-Language Models for Generalizable End-to-End Traffic Signal Control

Traffic-VL：用于通用端到端交通信号控制的强化视觉语言模型

• Authors: Xingchen Zou, Maonan Wang, Kemou Jiang, Jie Li, Yiqi Chen, Xi Chen, Hai Yang, Yuxuan Liang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.06353
• Pdf link: https://arxiv.org/pdf/2611.06353
• Abstract Existing traffic signal control methods typically rely on structured traffic states produced by dedicated perception pipelines. Emerging visual approaches often use idealized simulator images and convert them into intermediate representations before control, limiting their ability to directly exploit roadside visual observations. To bridge both the visual-domain and perception-to-decision gaps, we introduce SynTra-V and Traffic-VL. SynTra-V converts physics-based traffic simulations into photorealistic multi-view surveillance imagery across diverse road topologies and sensing conditions, and automatic semantic auditing checks that control-critical geometry, vehicle movements, and lane semantics match the simulator state. Traffic-VL is an eight-billion-parameter vision-language model that directly maps synchronized multi-view images and natural-language phase descriptions to signal decisions without intermediate structured states. Building on Traffic-R1's two-stage reinforced reasoning framework, Traffic-VL adapts its vision encoder via offline reinforcement learning on SynTra-V and freezes it during online visual stepwise trajectory policy optimization in closed-loop simulation. Experiments show that Traffic-VL matches structured-state Traffic-R1 while outperforming image-based controllers and general-purpose vision-language models, including Gemini 3 Pro. Evaluations on unseen topologies, degraded sensing conditions, real surveillance footage, and a three-week deployment across ten intersections further demonstrate robust visual generalization and effective constrained online control.
• 中文摘要 现有的交通信号控制方法通常依赖于由专用感知管道生成的结构化交通状态。新兴的视觉方法常常使用理想化的模拟图像，并将其转换为中间表征后再控制，限制了其直接利用路边视觉观察的能力。为了弥合视觉领域与感知与决策之间的差距，我们引入了SynTra-V和Traffic-VL。SynTra-V将基于物理的交通模拟转换为跨越多种道路拓扑和感测条件的写实多视角监控图像，并通过自动语义审计检查控制关键几何体、车辆运动和车道语义与模拟器状态匹配。Traffic-VL是一个拥有80亿参数的视觉语言模型，直接将同步的多视图图像和自然语言的阶段描述映射到无中间结构状态的信号决策。基于Traffic-R1的两阶段强化推理框架，Traffic-VL通过SynTra-V上的离线强化学习调整其视觉编码器，并在闭环模拟中在线可视化逐步轨迹策略优化时冻结其编码器。实验显示，Traffic-VL在匹配结构化状态Traffic-R1的同时，表现优于基于图像的控制器和通用视觉语言模型，包括Gemini 3 Pro。对未见拓扑、衰减的感测条件、真实监控画面以及跨十个交叉口的三周部署进一步展示了强健的视觉泛化和有效的受限在线控制。

GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR

GIST：离线RLVR的Gibbs定向层级数据选择

• Authors: Xuliang Huang, Tianxiang Zhao, Chengwei Qin, Yao Shu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.06395
• Pdf link: https://arxiv.org/pdf/2611.06395
• Abstract Offline reinforcement learning with verifiable rewards (RLVR) trains on a fixed pool of sampled responses and typically supervises every token of every response, although responses and tokens are not equally informative. We study which ones to supervise through the Gibbs-optimal policy of KL-regularized RLVR, which offline training targets, and measure the mismatch between a policy and this target with a bidirectional KL divergence. At the reference policy, this mismatch factorizes exactly and separates the two outcomes: correct responses contribute mainly through the forward term, concentrated on their low-probability tokens, and incorrect responses through the reverse term, concentrated on the responses the policy generates often and on their high-probability tokens. Based on this analysis, we propose GIST (Gibbs-Targeted Hierarchical Data Selection), a two-level selection method that approximates these principles with signals available before training. GIST uses the sampling temperature to select correct responses on-policy and incorrect responses near the policy's mode, and it uses the initial policy's token probabilities to supervise only the tokens where the mismatch lies. Applied to two offline training objectives, GIST raises the average accuracy over GSM8K, MATH-500, and MBPP for all three models under both objectives. In a controlled comparison on Qwen3-0.6B/GSM8K, it outperforms seven selection baselines that choose from the same candidates while supervising 66.6% fewer tokens than training on all candidates of the same prompts.
• 中文摘要 带有可验证奖励的离线强化学习（RLVR）训练于固定的抽样反应池，通常监督每个反应的每个代币，尽管响应和代币的信息量并不等同。我们研究通过吉布斯最优策略的KL正则化RLVR监督哪些代币，哪些离线训练目标，并测量策略与该目标之间具有双向KL发散的不匹配。在参考策略中，这种不匹配精确因数并分离了两个结果：正确响应主要通过前向项贡献，集中在其低概率标记上;错误反应则通过反向项贡献，集中于策略频繁生成的响应及其高概率标记。基于该分析，我们提出了GIST（Gibbs-Targeted Hierarchical Data Selection），这是一种两级选择方法，用训练前可用的信号近似这些原则。GIST 利用采样温度选择策略模式下的正确响应和策略模式附近的错误响应，并仅用初始策略的令牌概率监督不匹配的标记。应用于两个离线训练目标时，GIST 提高了 GSM8K、MATH-500 和 MBPP 在这三个目标下模型的平均准确率。在对 Qwen3-0.6B/GSM8K 的受控比较中，GIST 优于从相同候选中选择的七个选择基线，同时监督的令牌数量比所有相同提示候选人训练少 66.6%。

Neural Network-based Control Systems with Application to Flight Control, From Classical Neural Control to Reinforcement Learning

基于神经网络的控制系统，应用于飞行控制，从经典神经控制到强化学习

• Authors: Seyyed Ali Emami, Paolo Castaldi, Mohammad Narimani, Mahdi Shahrajabian, Mostafa Ezabadi, Alireza Ettelaei, Hosein Saadatmanesh, Zahra Shahbazi
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2611.06429
• Pdf link: https://arxiv.org/pdf/2611.06429
• Abstract This book provides a comprehensive and systematic exploration of neural network-based intelligent control systems for aerospace applications, bridging foundational theory with practical implementation. It is structured into two primary parts: Part I covers model-based control design, detailing Feedback Error Learning (FEL), pseudocontrol strategies, and neural backstepping for strict-feedback systems, along with practical refinements such as learning rule modifications, composite learning, and discrete-time formulations, all grounded in rigorous Lyapunov stability analysis. Part II introduces model-free control, laying the groundwork for Reinforcement Learning and Approximate Dynamic Programming. Throughout, theoretical concepts are solidified through detailed case studies on quadrotor UAVs, fixed-wing aircraft (GTM), satellites, and hypersonic flight vehicles, complete with performance metrics, parameter sensitivity analyses, and discussions of actuator faults, input constraints, and wind disturbances, making the book an invaluable resource for students and practicing engineers seeking to design, simulate, and analyze intelligent flight control systems.
• 中文摘要 本书全面且系统地探讨了基于神经网络的航空航天智能控制系统，桥接基础理论与实际实现。内容分为两大部分：第一部分涵盖基于模型的控制设计，详细介绍反馈错误学习（FEL）、伪控制策略和严格反馈系统的神经回溯，同时提供学习规则修改、复合学习和离散时间表述等实用改进，均基于严谨的李雅普诺夫稳定性分析。第二部分介绍无模型控制，为强化学习和近似动态规划奠定基础。全书通过详细案例研究巩固理论概念，涵盖四旋翼无人机、固定翼飞机（GTM）、卫星和高超音速飞行器，包含性能指标、参数敏感度分析，以及执行器故障、输入约束和风扰的讨论，使本书成为学生和实务工程师设计、模拟和分析智能飞行控制系统的宝贵资源。

TwinBridge: Digital Twin-Driven Policy and Reward Transfer for Real-World Reinforcement Learning in Laboratory Robotics

TwinBridge：数字孪生驱动的政策与奖励转移，用于实验室机器人中的真实强化学习

• Authors: Shu'ang Sun, Yibo Qiu, Haoliang Ye, Zan Huang, Ao Tian, Kan Ding, Ronald Xu, Mingzhai Sun
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.06442
• Pdf link: https://arxiv.org/pdf/2611.06442
• Abstract Online reinforcement learning offers an effective approach to precise laboratory manipulation, yet costly real-world demonstrations and limited access to dense process rewards hinder its practical application. We present TwinBridge, a digital twin-driven framework that supplies policy initialization and dense visual rewards for real-world online learning. We construct digital twins by combining 3D Gaussian splatting with physics simulation. Here, simulator states provide task events and geometric progress under physical prerequisites and ordering constraints, yielding rewards for policy pretraining with simulated demonstrations and reinforcement learning. We distill this supervision into a visual reward model. During real-world interaction, its predictions are combined with execution history to compute dense rewards for human-in-the-loop policy refinement. Across five real-world laboratory tasks, TwinBridge achieves a final mean autonomous success rate of 98% with at most 25 minutes of real-world interaction per task. Experiments show that dense rewards improve interaction efficiency over sparse rewards, and simulation-based policy initialization further accelerates learning.
• 中文摘要 在线强化学习为精确的实验室操作提供了一种有效的方法，但昂贵的现实演示和有限的密集过程奖励限制了其实际应用。我们介绍了TwinBridge，一个数字孪生驱动的框架，为真实在线学习提供策略初始化和密集视觉奖励。我们通过结合3D高斯喷溅与物理模拟构建数字孪生。在这里，模拟器状态在物理前提和排序约束下提供任务事件和几何进展，通过模拟演示和强化学习为策略预训练带来奖励。我们将这种监督提炼为视觉奖励模型。在现实交互过程中，其预测与执行历史结合，计算出密集奖励，用于人工参与策略的细化。在五个真实实验室任务中，TwinBridge实现了98%的最终平均自主成功率，每个任务最多25分钟的真实交互。实验表明，密集奖励相对于稀疏奖励能提高交互效率，基于仿真的策略初始化进一步加速学习。

What Does Multi-Turn Reinforcement Learning Really Improve in LLM Agents?

多回合强化学习在大型语言模型代理中真正提升了哪些方面？

• Authors: Kejian Zhu, Zhuoran Jin, Shangqing Tu, Tianyi Men, Yupu Hao, Jiachun Li, Kang Liu, Jun Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2611.06483
• Pdf link: https://arxiv.org/pdf/2611.06483
• Abstract While multi-turn agentic reinforcement learning (RL) excels on long-horizon tasks, its performance gains remain highly sensitive to interaction constraints. Evaluating across sampling breadth $K$ and interaction depth $T$ via $Pass@(K,T)$, we show that trajectory-level RL advantages depend on sufficient turn budgets, diminishing significantly when budgets tighten. Besides, agentic RL does not uniformly elevate single-step decision quality; rather, its strength concentrates in error recovery and long-horizon resilience. To mitigate the deployment risks caused by this budget dependence, we provide a preliminary discussion on the benefits and limitations of process supervision signals and introduce BudgetMix RL as an alternative that varies interaction budgets during training. Experiments demonstrate that BudgetMix RL achieves robust performance across diverse execution settings. Project Page: this https URL
• 中文摘要 虽然多回合代理强化学习（RL）在长期任务中表现出色，但其性能提升仍高度敏感于交互约束。通过通过$Pass@（K，T）$对采样广度$K$和交互深度$T进行评估，我们表明轨迹级强化学习的优势依赖于足够的回合预算，预算紧缩时显著减弱。此外，代理强化学习并不能统一提升单步决策质量;其优势集中在错误恢复和长期韧性上。为降低预算依赖带来的部署风险，我们初步讨论过程监督信号的利弊，并介绍BudgetMix RL作为一种在培训期间调整交互预算的替代方案。实验表明，BudgetMix RL在多种执行设置下都能实现稳健性能。项目页面：此 https URL

H^2MARL: Hierarchical Heterogeneous Multi-agent Reinforcement Learning for Humanoid Robot Control

H^2MARL：用于人形机器人控制的分层异构多智能体强化学习

• Authors: Xuemin Hu, Kaixin Li, Qingqing Xia, Yuxin Wang, Wei Xie, Bo Tang, Long Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.06523
• Pdf link: https://arxiv.org/pdf/2611.06523
• Abstract Coordinated whole-body control of humanoid robots remains challenging due to high-dimensional coupled dynamics and substantial physical heterogeneity across joint degrees of freedom (JDoFs). Existing multi-agent reinforcement learning methods either employ coarse agent partitions or directly apply generic heterogeneous-agent algorithms, while hierarchical controllers commonly optimize high-level planning and low-level execution independently, leading to uncoordinated motion. In this paper, we propose H^2MARL, a hierarchical heterogeneous multi-agent reinforcement learning framework with two key mechanisms. We first formulate humanoid whole-body control using a JDoF-level physical-heterogeneity-aware policy factorization, where each torque-actuated JDoF is conditioned on an explicit descriptor of its actuation, kinematic, and dynamic properties. Moreover, we develop a closed-loop policy-executor co-optimization mechanism that jointly adapts the high-level pose planner and the low-level joint executor using shared interaction trajectories, thereby reducing the distribution mismatch between planning and execution. Extensive experiments in MuJoCo and Isaac Gym demonstrate that H^2MARL consistently improves locomotion stability, gait coordination, and motion quality over homogeneous, coarse-grained, and independently optimized alternatives.
• 中文摘要 由于高维耦合动力学和关节自由度（JDoFs）上的显著物理异质性，人形机器人的协调全体控制仍具挑战性。现有多智能体强化学习方法要么采用粗糙的智能体分区，要么直接应用通用异构智能体算法，而分层控制器通常独立优化高层规划和低级执行，导致运动不协调。本文提出H^2MARL，一种具有两个关键机制的分层异质多智能体强化学习框架。我们首先采用JDoF级物理异质感知策略分解来制定类人生物全体控制，每个扭矩驱动的JDoF都基于其驱动、运动学和动态属性的显式描述来决定。此外，我们开发了一种闭环策略-执行者协同优化机制，结合共享交互轨迹，共同适配高层姿态规划器和低级联合执行器，从而减少规划与执行之间的分布不匹配。MuJoCo和Isaac Gym的广泛实验表明，H^2MARL在运动稳定性、步态协调和运动质量方面，优于同质、粗粒度和独立优化的替代方案。

Using Experience Twice: Dynamic Process Rubrics for Privileged Token-Level Credit Assignment

两次使用经验：特权令牌级信用分配的动态流程评分标准

• Authors: Shumao Wu, Kongcheng Zhang, Jianqiang Huang, Baisheng Lai
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2611.06543
• Pdf link: https://arxiv.org/pdf/2611.06543
• Abstract Agent trajectories often contain both successful decisions and failed attempts, yet optimization with sparse trajectory-level rewards struggles to identify which critical decisions should be reinforced or suppressed. We observe that meaningful distinctions among on-policy trajectories can be distilled into natural-language process rubrics and used as process supervision signals. Based on this insight, we propose Online Rubric Self-Distillation (ORSD), which leverages self-derived process rubrics for dense token-level supervision, thereby augmenting outcome-centric reinforcement learning. Specifically, the current policy independently generates multiple rubric bundles at each step according to the same group of trajectories, extracting candidate process rubrics from cross-trajectory behavioral differences. A frozen verifier then evaluates all rubric items against each trajectory in the group, and selects one candidate bundle based on bundle-level alignment between rubric satisfaction and task success. ORSD treats the selected rubric bundle as privileged information during training, then compares the log-likelihoods of the original tokens with and without the rubric. The resulting likelihood gap provides token-specific process credit, which is added to the original trajectory-level GRPO advantage through additive advantage shaping. ORSD thus bridges trajectory-level outcome supervision and fine-grained process supervision, enabling token-level credit assignment using process signals. Extensive experiments on ALFWorld and WebShop show that ORSD accelerates policy improvement and boosts final task performance for both Qwen3-1.7B and Qwen3-4B under the same environment-interaction budget. Our code is available at this https URL.
• 中文摘要 代理轨迹通常包含成功决策和失败尝试，但稀疏轨迹级奖励的优化难以确定哪些关键决策应被强化或抑制。我们观察到，策略轨迹间的有意义差异可以提炼为自然语言流程评分标准，并作为过程监督信号。基于这一见解，我们提出了在线评分标准自蒸馏（ORSD），利用自导出的流程评分标准进行密集的代币级监督，从而增强以结果为中心的强化学习。具体来说，当前策略在每个步骤根据同一轨迹组独立生成多个评分标准包，从交叉轨迹的行为差异中提取候选流程评分标准。冻结验证者随后将所有评分标准项目与组内各轨迹进行评估，并根据评分标准满意度与任务成功度之间的组合组合选择一个候选组合。ORSD在培训期间将选定的评分标准组合视为特权信息，然后比较带评分标准和不使用评分标准的原始标记的对数似然。由此产生的似然差距提供了代币特有的过程积分，并通过加法优势塑形，加入了原始轨迹级GRPO优势。因此，ORSD连接了轨迹级结果监督和细粒度过程监督，实现了利用流程信号进行代币级积分分配。在ALFWorld和WebShop上的大量实验表明，ORSD能加速策略改进，并提升Qwen3-1.7B和Qwen3-4B在相同环境交互预算下的最终任务性能。我们的代码可在此 https 网址获取。

RAGAM-AI: A Hybrid Reinforcement Learning based Agentic approach powered by Generative AI for Edge Resource Management

RAGAM-AI：一种基于生成式人工智能的混合强化学习代理方法，用于边缘资源管理

• Authors: Shreya K. Chari, John S. Vardakas, Kostas Ramantas, Christos Verikoukis
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2611.06554
• Pdf link: https://arxiv.org/pdf/2611.06554
• Abstract Resource management for distributed edge networks hosting different services has been handled efficiently using Reinforcement learning (RL). Reward generalization in complex workflows is still an open issue within the RL realm. Therefore, this work proposes RAGAM-AI, a hybrid framework that combines the capabilities of RL algorithms and Generative-AI (Gen-AI) for powering an Agentic AI workflow that optimizes resource allocation at the Multi-access Edge Computing (MEC) units. This framework further resolves the reward generalization issue by utilizing Large Language Model (LLM) for feedback. A Multi- Agent Soft-Actor Critic (MASAC) using Math{\Sigma}tral as a reward oracle is implemented for managing resource allocations at the MEC. RAGAM-AI interprets system state information and evaluates multi-objective trade-offs guiding the resource orchestration loop over distributed edge server agents. The complexity of the framework is further brought down by incorporating surrogate reward model. The overall performance is evaluated against the single agent SAC, REINFORCE and random policies for benchmarking. On using RAGAM-AI, lower allocation deviations for CPU, memory and bandwidth resources are reported while minimizing the latency and data rate costs. A scalable AI as a feedback (AIAF) solution optimizing network resources with significant performance improvement is thus presented.
• 中文摘要 分布式边缘网络托管不同服务的资源管理已通过强化学习（RL）高效完成。复杂工作流中的奖励泛化仍是一个开放问题。因此，本研究提出了RAGAM-AI，一种结合强化学习算法与生成式人工智能（Gen-AI）能力的混合框架，用于驱动一个智能人工智能工作流，优化多接入边缘计算（MEC）单元的资源分配。该框架通过利用大型语言模型（LLM）进行反馈，进一步解决了奖励泛化问题。采用Math{\Sigma}tral作为奖励预言机的多代理软行为者批评器（MASAC）实现了管理MEC资源分配的机制。RAGAM-AI解读系统状态信息，评估多目标权衡，指导分布式边缘服务器代理上的资源编排循环。通过引入替代奖励模型进一步降低了框架的复杂性。整体性能通过单代理SAC、REINFORCE和随机策略进行基准评估。使用RAGAM-AI时，报告了CPU、内存和带宽资源的分配偏差更低，同时最大限度地减少了延迟和数据速率成本。因此提出了一种可扩展的AI反馈（AIAF）解决方案，能够优化网络资源并显著提升性能。

Playing Soccer Like Humans Do: Learning Versatile and Agile Humanoid Soccer via Latent Motion Priors

像人类一样踢足球：通过潜动先验学习多功能且灵活的人形足球

• Authors: Haoyu Shao, Yunshen Wang, Jingze Zhang, Shaohang Zhu, Yong-Lu Li, Tianzhu Zhang, Baoxiong Jia, Siyuan Huang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.06564
• Pdf link: https://arxiv.org/pdf/2611.06564
• Abstract Humanoid soccer requires a robot to seamlessly combine agile locomotion, ball manipulation, and impact-rich contact dynamics across a continuous, dynamic behavioral spectrum. Yet existing approaches either cover only a limited set of skills, or forego motion priors entirely and produce unnatural behaviors, leaving the combination of a unified skill set and human-like motion quality unaddressed. We present soccerbot, a hierarchical reinforcement learning framework that resolves both limitations through a structured latent motion prior distilled from large-scale human soccer motion capture data. By constraining policy learning within this dynamically plausible motion manifold, a single skill-conditioned policy naturally learns to perform moving, dribbling, trapping, and kicking in a unified formulation, while inherently producing human-like motion and fluid transitions during commanded skill switching. Augmented by robustness-oriented simulation design, soccerbot transfers zero-shot to the Unitree G1 humanoid robot, demonstrating agile, natural, and robust execution of the full soccer skill repertoire under real-world conditions.
• 中文摘要 类人足球需要机器人无缝结合灵活的移动、控球和冲击力强的接触动态，跨越连续且动态的行为光谱。然而，现有方法要么只覆盖有限的技能，要么完全放弃动作先验，产生不自然的行为，导致统一技能与类人动作质量的结合未被解决。我们提出了soccerbot，一种层级强化学习框架，通过从大规模人类足球动作捕捉数据提炼出的结构化潜动先验解决了这两个局限。通过将策略学习限制在这个动态合理的运动流形中，单一技能条件策略自然学习以统一的模式执行移动、盘带、夹击和踢球，同时在指令技能切换过程中本质上产生类人动作和流畅过渡。借助稳健性导向的模拟设计，soccerbot将零射门技术转移到Unitree G1人形机器人上，展示了在真实条件下灵活、自然且稳健地执行完整足球技能的能力。

Dynamic Shape Control of Soft Trunk Robots

软后备箱机器人的动态形状控制

• Authors: Hehui Zheng, Marc Stampanoni, Roshan S. Kaundinya, Ronan Hinchet, George Haller, Robert K. Katzschmann
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.06625
• Pdf link: https://arxiv.org/pdf/2611.06625
• Abstract Controlling the shape profile of soft trunk robots extends their reach into complex regions of the workspace and enhances their use in practical applications. Recent advances in soft robot design have enabled soft trunk robots to achieve greater flexibility in conforming to complex shapes. Specifically, trunk robots whose geometries are designed as helicoids have been shown to enable extreme shape-changing capabilities. Their compliance makes them highly nonlinear, which in turn makes shape control difficult. On similar tendon-driven trunk robots, model-based controllers based on the theory of adiabatic spectral submanifolds (aSSMs) have achieved high accuracy in closed-loop end-effector tracking. We extend this approach by learning an aSSM model of the full body of a trimmed-helicoid soft trunk robot from data and use this in a model predictive control (MPC) scheme to demonstrate a range of shape control tasks on hardware. On closed-loop dynamic shape regulation, our method outperforms the available linear baselines by a factor of 10. In human-in-the-loop interactive demos, our method completes section-by-section shaping and balances a tennis ball for over 25 seconds, while the linear baselines fall short. A model-free reinforcement learning policy trained for 12 hours on hardware fails on the same balancing task. These results establish aSSM-based MPC as a practical route to full-body shape control on real soft hardware.
• 中文摘要 控制软躯干机器人的形状轮廓，使其覆盖工作空间的复杂区域，并提升其在实际应用中的应用。软体机器人设计的最新进展使软躯干机器人在适应复杂形状时获得了更大的灵活性。具体来说，设计为螺旋形的躯干机器人已被证明具备极端的形状变化能力。其顺应性使其高度非线性，从而使形状控制变得困难。在类似的腱驱动躯干机器人上，基于绝热光谱子流形（aSSM）理论的基于模型的控制器在闭环端执行器跟踪方面取得了很高的精度。我们通过从数据中学习整齐螺旋体软躯干机器人全身的aSSM模型，并将其应用于模型预测控制（MPC）方案中，演示硬件上的多种形状控制任务。在闭环动态形状调控中，我们的方法比现有线性基线表现良好10倍。在人机互动演示中，我们的方法完成了逐段塑形，并使网球保持平衡超过25秒，而线性基线则不足。在硬件上训练12小时的无模型强化学习策略在相同平衡任务中失败。这些结果确立了基于aSSM的MPC作为在真实软硬件上实现全身形状控制的实用途径。

Compounding Physical RSI: Turning Execution Failure into Inherited Capability

物理RSI的复合：将执行失败转化为继承能力

• Authors: Yixuan Yang, Qiang Chen, Run Shao, Shuwei Dong, Wenhao Li, Yunpeng Wang, Wenhan Fang, Haomiao Wu, Xingyu Wang, Lichen Man, Rong Zhang, Xuezhi Cui, Yuan Liu, Bingyan Lu, Jiawei Ma, Xiaoyang Ma, Zhemin Sun, Zhuorui Yao, Yutong Dong, Yuxuan Zhao, Yichong Mu, Jiabao Jiang, Wenxuan Deng, Junling Zhou, Yi Chen, Deyu Zhang, Yiming Yan, Yige Yuan, Shan You, Fei Wang, Haifeng Li, Hongyan Xu, Yichao Cao, Xiu Su
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.06654
• Pdf link: https://arxiv.org/pdf/2611.06654
• Abstract Generalist robot policies are trained on static offline datasets, yet physical manipulation unfolds sequentially across stages. Because a weakness in a downstream stage remains unobservable until the policy passes earlier ones, each resolved bottleneck exposes the next hidden failure mode. This staged structure calls for Recursive Self-Improvement (RSI), in which a policy iteratively executes, diagnoses failures, and updates itself. Moreover, because task completion behaves like a product of stage-level pass rates, recursive gains can multiply across generations. In physical environments, three bottlenecks obstruct compounding: terminal execution feedback rarely localizes the stage that caused a failure, failed trajectories do not supply the corrective action, and updating shared parameters can degrade previously mastered skills. We propose Compounding Physical RSI, a self-evolving policy framework that turns execution failures into heritable capability. First, automated capability-gap assessment traces failed rollouts to the task stage where progression breaks down and formulates a structured improvement objective. Second, conditioned on this objective, our remedy stage curates validated, stage-relevant experience and refines the policy via behavior-regularized offline reinforcement learning, without requiring costly online rollouts at each optimization step. Third, to safeguard existing skills, our retention stage constructs weight-space candidates between the inherited reference policy and the refined policy, evaluates them across the task suite, and promotes the best candidate as the reference state for the next generation. Evaluated across all 42 tasks of the RoboDojo-Sim benchmark, CSU-AI-07 ranks first on the leaderboard with a three-seed mean Overall Score of 26.15 and Success Rate of 20.70%, outperforming the next-best entry by 5.92 points and 5.82 percentage points.
• 中文摘要 通用机器人策略训练于静态离线数据集，但物理操作在各阶段顺序展开。由于下游阶段的弱点直到策略通过前一阶段才被观察到，每解决一个瓶颈都会暴露下一个隐藏的失败模式。这种分阶段结构需要递归自我改进（RSI），策略迭代执行、诊断失败并自我更新。此外，由于任务完成表现为阶段级通过率的乘积，递归收益可以跨代倍增。在物理环境中，有三个瓶颈阻碍复利：终端执行反馈很少定位导致失败的阶段，失败轨迹无法提供纠正措施，更新共享参数会削弱先前掌握的技能。我们提出了复合物理RSI，一种自我演进的策略框架，将执行失败转化为可遗传的能力。首先，自动能力差距评估将失败的推广追踪到任务阶段，进度分解并制定结构化改进目标。其次，基于该目标，我们的补救阶段策划经过验证的阶段相关经验，并通过行为规范化的离线强化学习完善策略，无需在每个优化步骤进行昂贵的在线推广。第三，为保护现有技能，我们的保留阶段构建继承参考策略与精炼策略之间的权重空间候选，评估任务套件中，推广最佳候选作为下一代参考状态。CSU-AI-07在RoboDojo-Sim基准测试的全部42个任务中排名榜首，三种子平均总分26.15，成功率20.70%，领先次优项目5.92分和5.82个百分点。

Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

强化学习促进了通用机器人策略的扩展改进

• Authors: Michal Korniak, Marco Bagatella, Ralf Römer, Michal Nauman, Yarden As, Pranav Vignesh, Manan Agarwal, Leander Diaz-Bone, Piotr Zalewski, Mikołaj Czarnecki, Lenart Treven, Shashwat Saxena, Sreyas Venkataraman, Sarvesh Patil, Zhiyuan Zhou, Daniel Marta, Oier Mees, Nur Muhammad Mahi Shafiullah, Carmelo Sferrazza, Bhavya Sukhija, Angela P. Schoellig, Andreas Krause, Max Simchowitz, Andrew Wagenmaker
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.06905
• Pdf link: https://arxiv.org/pdf/2611.06905
• Abstract The introduction of generalist robot policies has enabled a single, large model to learn to perform a variety of tasks from a vast pre-training corpus. While these models often exhibit powerful zero-shot generalization capabilities, they typically require an adaptation phase to achieve sufficient performance in deployment. Iterated self-improvement via reinforcement learning (RL) has demonstrated promise in enabling this adaptation, yet such approaches have been largely explored in single-task settings - where the goal is to simply improve on a narrow deployment task - which lies in stark contrast to the multi-task capabilities of the base policy. In this work we ask whether generalist robot policies are capable of multi-task, generalist improvement in deployment - targeting many diverse tasks simultaneously - and what algorithmic recipe enables this. To answer this question, we first develop a set of structured evaluations to test generalist improvement capabilities, and systematically evaluate algorithmic and practical design choices regarding value estimation, policy extraction and resource allocation. We show that not only can generalist policies be improved online across diverse tasks, but that the right recipe can enable generalization and increase performance on tasks not considered during online training. We characterize when this generalization occurs and show that both the training task distribution and the use of value-based RL are critical. To the best of our knowledge, this is among the first evidence for online generalist improvement for robot foundation models.
• 中文摘要 通用机器人策略的引入使得单一大型模型能够从庞大的预训练语料库中学习执行多种任务。虽然这些模型通常表现出强大的零任务泛化能力，但通常需要适应阶段才能在部署中达到足够性能。通过强化学习（RL）进行的迭代自我改进已被证明在实现这种适应方面有潜力，但此类方法主要在单任务环境中被探索——目标仅是改进狭窄的部署任务——这与基础策略的多任务能力形成鲜明对比。在本研究中，我们探讨通用机器人策略是否具备多任务、通用改进的能力——即同时针对多个不同任务——以及是什么算法配方使之实现。为回答这个问题，我们首先开发了一套结构化评估，以测试通用改进能力，并系统评估算法和实用设计在价值估计、策略提取和资源分配方面的选择。我们证明，不仅可以在不同任务中在线改进通才政策，正确的方法还能实现泛化并提升在线培训未考虑任务的绩效。我们描述了这种泛化发生的时间，并指出训练任务分布和基于价值强化学习的使用都至关重要。据我们所知，这是机器人基础模型在线通才改进的首批证据之一。

Keyword: diffusion policy

Correct Robots Before They Make Mistakes: Proactive Human-in-the-Loop Intervention via Preference Learning

在机器人犯错前纠正它们：通过偏好学习进行主动的人机介入

• Authors: Xinkai Jiang, Jing Jiang, Hongyi Zhou, Pankhuri Vanjani, Ahmad Baki, Jakub Suliga, Ge Li, Zhuoyue Li, Gerhard Neumann, Rudolf Lioutikov
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2611.05667
• Pdf link: https://arxiv.org/pdf/2611.05667
• Abstract This paper presents Proactive Human-in-the-Loop (PHIL), a framework for proactive policy correction and preference-based post-training. PHIL enables operators to preview policy-predicted future motion and intervene before anticipated undesirable actions. During takeover, PHIL continues querying the policy and displaying its predicted trajectories in the physical workspace through an extended reality (XR) interface. PHIL pairs rejected policy predictions with corrective action chunks subsequently executed from the same physical state over the same horizon, and preference-based post-training encourages the policy to favor these corrections. This paper evaluates PHIL on four real-world manipulation tasks using a diffusion policy and a pretrained vision-language-action policy. With five correction episodes per condition, PHIL increases mean success rates across the evaluated tasks and demonstration budgets from 44.0% to 70.3% for the diffusion policy and from 57.8% to 76.9% for X-VLA, compared with HG-DAgger. Separately, policies trained with proactive corrections show fewer unnecessary recovery behaviors in successful episodes. Project page: this https URL.
• 中文摘要 本文提出了主动人机环路（PHIL），这是一个用于主动政策纠正和基于偏好的后期训练框架。PHIL使操作员能够预览策略预测的未来动作，并在预期的不良行为发生前进行干预。在接管过程中，PHIL通过扩展现实（XR）接口继续查询策略并在物理工作区中显示其预测轨迹。PHIL将被拒绝的策略预测与随后从同一物理状态、同一视野内执行的纠正行动块配对，基于偏好的后训练鼓励策略偏向这些修正。本文在四个实际操作任务中评估了PHIL，采用扩散策略和预训练视觉-语言-行动策略。每个条件有五次纠正，PHIL使得评估任务和示范预算中扩散策略的平均成功率从44.0%提升至70.3%，X-VLA从57.8%提升至76.9%，相比HG-DAgger。另外，采用主动纠正训练的策略在成功发作中显示出更少不必要的恢复行为。项目页面：此链接 https URL。