生成时间: 2026-10-26 22:55:49 (UTC+8); Arxiv 发布时间: 2026-10-26 20:00 EDT (2026-10-27 08:00 UTC+8)

今天共有 42 篇相关文章

Keyword: reinforcement learning

Cognitively Constrained Partners for Zero-Shot Coordination in Real-Time Partitioned Control

实时分区控制中零射击协调的认知约束伙伴

• Authors: Ash Simmons, George De Ath, Richard M. Everson
• Subjects: Subjects: Machine Learning (cs.LG); Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2610.31726
• Pdf link: https://arxiv.org/pdf/2610.31726
• Abstract Effective human-machine teaming requires agents that coordinate with unpredictable human partners. Reinforcement learning from live human feedback or large behavioural datasets is expensive, while purely simulated training overfits to artificial conventions and fails at Zero-Shot Coordination with real people. We propose a framework for training teaming agents against Cognitively Constrained Partners: simulated partners whose diversity emerges from procedural constraints grounded in human physiological literature, specifically reaction delays and perceptual limitations, rather than from reward shaping or historical training checkpoints. In a modified Lunar Lander environment, we report preliminary evidence that these partners play more like real people than unconstrained agents do, and that agents trained against them match the strongest cross-play baselines while being the only ones whose performance does not depend on how their training partners were generated. We also describe a human-participant study designed so that its rankings can be compared directly against cross-play, testing how well that standard evaluation predicts real human-agent teaming.
• 中文摘要 有效的人机团队合作需要智能体与不可预测的人类伙伴协调。从实时人类反馈或大型行为数据集中进行强化学习成本高昂，而纯模拟训练则会对人工惯例进行过拟合，且在与真人零射击协调时失败。我们提出了一个框架，用于训练智能体对抗认知受限伙伴：模拟伙伴的多样性源自基于人类生理文献的程序约束，特别是反应延迟和感知限制，而非奖励塑造或历史训练检查点。在修改后的月球着陆者环境中，我们报告初步证据表明，这些伙伴的游戏方式比无约束的智能体更接近真实人，且训练对抗他们的智能体匹配最强的跨平台基线，且是唯一性能不受训练伙伴生成方式影响的伙伴。我们还介绍了一项人参与者研究，设计目的是将排名直接与跨平台对比，测试该标准评估对真实人与代理团队合作的预测效果。

Learning to Be Monitored: Causal Bounds and Intervention-Aware Post-Training Recipes for Chain-of-Thought Monitorability

学习被监控：因果界限与干预感知的培训后思维链监控配方

• Authors: Qiang Fu, Xinyue Li, Andre Wibisono, Arman Cohan
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2610.31733
• Pdf link: https://arxiv.org/pdf/2610.31733
• Abstract Chain-of-thought (CoT) monitoring aims to detect potentially harmful behaviors in large language models (LLMs) from their CoTs. Intervention-based evaluations measure this property by asking whether a monitor can detect the role of an intervention from the CoT when the targeted behavior occurs, typically using minimal-criterion sensitivity ($\mathsf{TPR_{mc}}$) proposed by~\citep{guan2025monitoring}. However, a target behavior observed under intervention is not necessarily caused by it, and we show that this metric can overestimate how often a monitor detects target behaviors actually caused by the intervention. Specifically, we establish $\mathsf{TPR_{mc}}$ as an upper bound on this causal detection rate and derive a complementary lower bound, which provides a tighter evaluation framework for CoT monitorability. With this framework, we study whether LLMs can be trained to make their CoTs easier to monitor. We first construct a training dataset of paired control--intervention tasks spanning multiple intervention types. Using the dataset, we design and apply three intervention-specific training methods based on off-policy self-distillation, reinforcement learning, and on-policy self-distillation to train separate variants of each base model. Across various model-monitor configurations, all three methods generally improve CoT monitorability over the corresponding base models under our extended evaluation framework. Our code is available at this https URL.
• 中文摘要 思维链（CoT）监测旨在从大型语言模型（LLM）中的CoT中检测潜在有害行为。基于干预的评估通过询问监测者是否能在目标行为发生时检测干预在CoT中的作用来衡量该特性，通常采用由~\citep{guan2025monitoring}提出的最小标准敏感性（$\mathsf{TPR_{mc}}}）。然而，干预下观察到的目标行为不一定由干预引起，我们证明该指标可能高估监测者检测干预实际引发目标行为的频率。具体来说，我们建立了$\mathsf{TPR_{mc}}$作为该因果检测率的上限，并推导出一个互补的下界，为CoT可监测性提供了更严格的评估框架。基于该框架，我们研究LLM是否可以被训练使其CoT更易于监控。我们首先构建了一个跨越多种干预类型的配对控制-干预任务训练数据集。利用该数据集，我们设计并应用三种基于非策略自蒸馏、强化学习和策略内自蒸馏的干预特定训练方法，以训练每个基模型的不同变体。在各种模型-监控配置中，这三种方法在扩展评估框架下通常提升了相较相应基础模型的CoT可监测性。我们的代码可在此 https 网址获取。

PrefO2O: Zero-Shot Preference-Based Reinforcement Learning via Offline-to-Online Latent Preference Transfer

PrefO2O：基于零点偏好的强化学习，通过离线到在线的潜在偏好转移

• Authors: Ruiqi Wang, Dezhong Zhao, Xuesu Xiao, Erdem Biyik, Byung-Cheol Min, Guohua Chen
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.31812
• Pdf link: https://arxiv.org/pdf/2610.31812
• Abstract Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for learning robot behaviors aligned with human intent. However, its scalability is fundamentally limited by the need to collect expensive human preference labels for every new task. In this work, we present PrefO2O, an offline-to-online latent preference transfer framework that enables zero-shot PbRL on unseen target tasks without requiring any human feedback. The core idea is to learn a task-invariant latent preference space from a small amount of offline preference data on source tasks, where transferable preference structures are disentangled from task-specific variations. This learned manifold allows PrefO2O to transfer preference structures and reward priors to new tasks under distribution shift, while confidence-weighted optimal transport alignment generates reliable pseudo-preference supervision for online reward adaptation. Extensive experiments on robotic manipulation and locomotion benchmarks demonstrate that PrefO2O substantially outperforms existing preference-transfer baselines and matches or exceeds PbRL methods trained with real human feedback, while generalizing across multi-to-one and one-to-one transfer settings as well as across different robot embodiments.
• 中文摘要 基于偏好的强化学习（PbRL）已成为学习与人类意图对齐的机器人行为的有前景范式。然而，其可扩展性从根本上受限于每个新任务都需要收集昂贵的人类偏好标签。本研究提出了PrefO2O，一种离线到在线的潜在偏好转移框架，能够在不需人工反馈的情况下实现零样本PbRL。核心思想是从源任务的少量离线偏好数据中学习一个任务不变的潜在偏好空间，其中可转移偏好结构与任务特定变异分离。该学习流形允许PrefO2O在分布转移时将偏好结构和奖励先验转移到新任务，而置信度加权最优传输对齐则生成可靠的伪偏好监督以实现在线奖励适应。大量机器人操作和运动基准测试实验表明，PrefO2O的表现显著优于现有偏好转移基线，并与用真实人类反馈训练的PbRL方法相匹配甚至超过，同时在多对一和一一转移设置以及不同机器人体型中具有推广性。

Free Lunch for LLMs Confidence: The RL Critic Already Knows

给大型语言模型的自信免费午餐：现实学习批评者已经知道了

• Authors: Shujin Wu, Heng Ji
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.31820
• Pdf link: https://arxiv.org/pdf/2610.31820
• Abstract Reliable confidence estimation is essential for trustworthy deployment of large language models (LLMs), but dedicated confidence training stage requires additional data and compute. We ask a fundamental question: what prediction problem does confidence training actually solve? Inspired by prior work on external confidence estimation, we compare self- and external estimators of comparable capacity trained on the same (question, response, correctness) triples. We show that an external model trained solely on a policy's behavioral traces can estimate the policy's confidence as accurately as the policy itself, without accessing the policy's internal representations. This points to a deeper connection with value function learning in reinforcement learning (RL): under a correctness-based outcome, we prove that confidence estimation and critic training are equivalent, both predicting the likelihood that a policy's response will be correct from its trajectory. A simple consequence follows: for LLMs trained with value-based RL, the existing critic can be reused as an off-the-shelf confidence estimator with no additional training or data. The policy remains unchanged, so its task performance is preserved, and the critic provides confidence throughout generation, including intermediate reasoning steps. Across math, code, tool calling, and general reasoning spanning a dozen domains, the reused critic is the best confidence estimator on every in-distribution split and the best-calibrated on 14 of 16, without hurting the task performance.
• 中文摘要 可靠的置信度估计对于大型语言模型（LLM）的可信部署至关重要，但专门的置信度训练阶段需要额外数据和计算。我们提出一个根本性问题：置信度训练到底解决了什么预测问题？受外部置信估计工作的启发，我们比较了在相同（问题、回答、正确性）三元组上训练的可比能力自估计量和外部估计量。我们证明，仅基于策略行为轨迹训练的外部模型，可以在无需访问策略内部表征的情况下，准确估计策略的置信度。这表明其与强化学习（RL）中价值函数学习的更深层联系：在基于正确性的结果下，我们证明了信心估计和批评训练是等价的，都预测了策略响应正确的可能性。一个简单的结论是：对于用基于价值的强化学习训练的大型语言模型，现有的批评者可以作为现成的置信度估计器重复使用，无需额外训练或数据。策略保持不变，因此其任务性能得以保留，批评者在整个生成过程中提供信心，包括中间推理步骤。在数学、代码、工具调用和跨十多个领域的一般推理中，重用批评者是每个分布内分割中最好的置信估计器，并且在16个中校准最好的，同时不影响任务表现。

SEVI: Safe Evidential Real-World Reinforcement Learning

SEVI：安全证据现实世界强化学习

• Authors: Shaekh Mohammad Shithil, Lantao Liu
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.31829
• Pdf link: https://arxiv.org/pdf/2610.31829
• Abstract Real-world reinforcement learning has enabled contact-rich manipulation tasks to be trained directly on physical robots, but unsafe interactions during online learning remain a key barrier to practical deployment. We propose Safe Evidential Real-World Reinforcement Learning (SEVI), a framework for reliability-aware safety intervention during real-world policy learning. SEVI introduces an Evidential Safety Critic that estimates both future constraint-violation risk and epistemic uncertainty in a single forward pass. During online learning, SEVI evaluates each task-policy action before execution; if the action is predicted to be unsafe or the prediction is insufficiently supported by evidence, SEVI overrides it with an offline-trained recovery policy that guides the robot toward a lower-risk region. We evaluate SEVI on two simulated manipulation tasks and a real-world USB insertion task under in-distribution and distribution-shifted conditions. Across task-learning and safety-critic baselines, SEVI achieves the strongest trajectory-level safety prediction, the highest final task success, and the lowest unsafe-event counts. SEVI also reduces unsafe events by 13.4 to 57.1 percent relative to recovery-based safe RL while preserving low-latency execution-time filtering. The project page is available at: this https URL
• 中文摘要 真实世界强化学习使得接触丰富的操作任务能够直接在物理机器人上训练，但在线学习中的不安全互动仍是实际部署的关键障碍。我们提出了安全证据现实世界强化学习（SEVI），这是一个在真实世界政策学习中进行可靠性意识安全干预的框架。SEVI引入了一个证据安全批判器，能够在一次前向传递中估算未来约束违规风险和认知不确定性。在线学习期间，SEVI在执行前评估每个任务-策略动作;如果预测该动作不安全或预测缺乏证据支持，SEVI会通过离线训练的恢复策略覆盖该策略，引导机器人向风险较低的区域移动。我们在分布内和分布转移条件下，对两项模拟操作任务和一项真实世界的USB插入任务进行评估。在任务学习和安全批评者基线中，SEVI实现了最强的轨迹级安全预测、最高的最终任务成功率和最低的不安全事件数量。SEVI还将不安全事件比基于恢复的安全强化学习减少13.4%，达到57.1%，同时保持低延迟执行时间过滤。项目页面可访问：此 https URL

Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design

将大型语言模型与人类反馈对齐：数学基础与算法设计

• Authors: Siliang Zeng, Luca Viano, Chenliang Li, Jiaxiang Li, Volkan Cevher, Markus Wulfmeier, Stefano Ermon, Alfredo Garcia, Mingyi Hong
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.31862
• Pdf link: https://arxiv.org/pdf/2610.31862
• Abstract This article provides an introduction to the mathematical foundations and algorithmic frameworks used to align Large Language Models (LLMs) with human intentions, preferences, and values. We discuss standard alignment techniques, such as fine-tuning (SFT), reinforcement learning with human feedback (RLHF), and direct preference optimization (DPO). We also explore the theoretical underpinnings of learning from human preferences, drawing connections to inverse reinforcement learning (IRL) and discrete choice models. We present state-of-the-art algorithms in a tutorial style, discuss their advantages and limitations, and offer insights into practical implementation. Our exposition is intended to serve as a comprehensive resource for researchers and practitioners, providing both a foundational understanding of alignment methodologies and a framework for developing more robust and scalable alignment techniques.
• 中文摘要 本文介绍了用于将大型语言模型（LLMs）与人类意图、偏好和价值观对齐的数学基础和算法框架。我们讨论了标准的比对技术，如微调（SFT）、人类反馈强化学习（RLHF）和直接偏好优化（DPO）。我们还探讨了从人类偏好学习的理论基础，并将其与逆强化学习（IRL）和离散选择模型联系起来。我们以教程风格介绍最先进的算法，讨论其优缺点，并提供实际实现的见解。本讲旨在为研究人员和实践者提供全面的资源，既为比对方法论提供基础理解，也为开发更稳健、可扩展的比对技术提供框架。

Emergent Cooperative Thermoregulation in Lunar Swarm Robotics

月球群机器人中的新兴协同体温调节

• Authors: Anais Jeger, Harry Holt, Lennart Puck
• Subjects: Subjects: Robotics (cs.RO); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2610.31868
• Pdf link: https://arxiv.org/pdf/2610.31868
• Abstract Swarms of small robots are an attractive way to survey lunar caves and lava tubes. Missions such as NASA's CADRE will fly cooperating rovers for distributed mapping, but only in daylight. The 354 h lunar night drops the surface towards 95 K, and the radioisotope heaters that let single rovers survive it cannot be supplied at swarm scale. In this study we investigate whether a swarm can instead use its own members as thermal shielding. We train one policy, shared by all twenty robots, by reinforcement learning with an evolution-strategies estimator. The policy receives a single reward, once, at the end of the night: the mean temperature of the five coldest robots. Nothing in that reward refers to position, spacing, exposure or motion. Two behaviors nonetheless emerge. The swarm aggregates, and robots circulate between the sheltered core and the exposed rim, the same pair of behaviors with which emperor penguins survive the Antarctic winter. We measure the survival threshold, the smallest stored energy per robot at which every robot is still alive at sunrise. Across ten runs it is 155.6 +/- 1.3 pcm-h, where one pcm-h is 7.2 kJ of stored heat. That is 37% below a motionless huddle and within 13% of a physics-based lower bound, and at a budget where the motionless huddle loses six robots, every robot survives. The behavior needs no shared spatial reference and no robot-robot contact, and transfers from twenty robots to forty without retraining.
• 中文摘要 成群的小型机器人是调查月球洞穴和熔岩管的一种有吸引力的方式。像NASA的CADRE这样的任务会让协作的探测车进行分布式测绘，但仅限于白天。354小时的月夜会使表面温度下降到95K左右，而让单个探测车生存的放射性同位素加热器无法在群体规模下供给。本研究我们探讨了群体是否可以利用自身成员作为热屏蔽。我们通过强化学习与进化策略估计器训练了一项所有二十台机器人共享的策略。该策略在夜晚结束时获得一次奖励：五台最冷机器人的平均温度。该奖励中没有涉及位置、间距、暴露或运动。但两种行为仍然浮现。群体聚集，机器人在受保护的核心和裸露边缘之间循环，这与帝企鹅在南极冬季生存的行为相同。我们测量生存阈值，即每台机器人在日出时仍存活的最小储存能量。十次运行中，能量为155.6 +/- 1.3 pcm-h，其中1 pcm-h为7.2千焦的储存热量。这比静止的聚集体低37%，距离物理下限不到13%，在一个预算下，静止聚集失去六个机器人时，每台机器人都能存活。这种行为不需要共享的空间参考，也不需要机器人间的接触，且无需重新训练即可从二十个机器人转移到四十个。

Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima

课程学习的有效性取决于奖励稀缺性和竞争性的最优

• Authors: John J. Vastola, Ann Huang, Satpreet H. Singh, Samuel J. Gershman, Kanaka Rajan
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.31884
• Pdf link: https://arxiv.org/pdf/2610.31884
• Abstract While animals and people tend to learn a task more quickly and reliably when they first train on simpler versions of it, curriculum learning's effectiveness in artificial settings varies, and appears substantially greater in reinforcement learning than in supervised learning. In this work, we construct and analyze a minimal model of policy learning to better understand why. We consider an open-loop, episodic target interception task whose parameters---especially how close the agent must be to the target to receive an informative reward signal---can be chosen so that the agent either receives informative rewards frequently ('dense' rewards) or rarely ('sparse' rewards). We mathematically and numerically analyze the dynamics of REINFORCE-based policy learning in both cases. In the dense reward case, curricula have only marginal benefits; in the sparse case, curricula can be required for learning to occur at all. We find that this is especially true when the agent has the conflicting goals of minimizing effort and accumulating high reward, since this can produce distinct reward landscape optima which compete with one another. Our characterization of useful curricula in this setting indicates that they address two issues: they (i) make rewards less sparse to speed up learning, and (ii) shape the reward landscape to steer agents away from bad local optima.
• 中文摘要 虽然动物和人类在首次训练更简单版本的任务时，学习速度更快、更可靠，但课程学习在人工环境中的效果各异，强化学习在监督学习中显著更高。本研究构建并分析了一个政策学习的最小模型，以更好地理解其原因。我们考虑了一个开环、情节式目标拦截任务，其参数---尤其是智能体与目标的距离---以接收信息性奖励信号可选择为智能体要么频繁获得信息性奖励（“密集”奖励），要么很少（“稀疏”奖励）。我们对基于强化的政策学习在两种情况下的动态进行了数学和数值分析。在密集奖励情况下，课程的益处有限;在稀疏奖励情况下，学习可能根本不需要课程。我们发现，当主体在最小化努力和积累高回报的目标之间存在冲突目标时，这一点尤为明显，因为这会产生相互竞争的不同奖励景观最优。我们在该环境中对有用课程的描述表明，它们解决了两个问题：（i）使奖励不那么稀疏以加快学习速度，（ii）塑造奖励景观，引导参与者远离不良的局部最优。

Move, don't think: sample-efficient collision avoidance with lossy proximity sensors

移动，别思考：用有损的接近传感器实现采样高效碰撞避免

• Authors: Carson Kohlbrenner, Niraj Pudasaini, William Xie, Naren Sivagnanadasan, Nikolaus Correll, Alessandro Roncone
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.31891
• Pdf link: https://arxiv.org/pdf/2610.31891
• Abstract Data-driven robots must learn to avoid collisions with as few demonstrations as possible to avoid collecting large datasets of destructive collision examples. We explore whole-body proximity sensors as inductive biases for learning reflexive collision-dodging behavior and find that lossy sensors with low resolution were stronger biases than high-resolution coverage. We also found that, perhaps unintuitively, anticipating when and where contact happens on the body of a robot reduced its learning sample efficiency. We investigated eight candidate representations in a reinforcement learning environment, which included latent compression, future contact anticipation, lossy corruption, and privileged state estimation. The candidate representations were tested by training over 200 humanoid robots to play a simplified version of dodgeball end-to-end under an intentionally constrained sample and compute budget.
• 中文摘要 数据驱动机器人必须学会以尽可能少的演示方式避免碰撞，以避免收集大量破坏性碰撞实例数据集。我们探索了全身接近传感器作为学习反射性碰撞躲避行为的归纳偏置，发现低分辨率有损传感器比高分辨率覆盖的传感器更强。我们还发现，或许不直观的是，预测机器人身体接触的时间和地点会降低其学习样本效率。我们在强化学习环境中研究了八种候选表征，包括潜在压缩、未来接触预期、有损损坏和特权状态估计。候选表征通过训练200多台类人机器人在有意限制的样本和计算预算下端到端进行简化版躲避球进行测试。

Kolibri: A Sovereign European Model on the Pareto Frontier

科利布里：帕累托边境上的主权欧洲模式

• Authors: Aleph Alpha: Moritz Althaus, Artur Baranowski, Michael Barlow, Fabien C. Y. Benureau, Felix Berkenkamp, Amrita Bhattacherjee, Lukas Blübaum, Duncan Blythe, Augustijn de Boer, Bastian Boll, Thomas F. Burns, Björn Deiseroth, Niko Dürr, Béni Balázs Egressy, Niklas Finken, Niklas Freymuth, Fabian Gebhart, Max Hammer, Bastian Harren, Andreas Hartel, Johann Higl, Steffen Hirschmann, Pablo Iyu Guerrero, Yasser Jadidi, Andreas Jagusch, Mitja Kleider, Markus Klein, Jonas Knupp, Murali Kondragunta, Garwin Lechner, Dominik Lê-Kellner, Jurek Leonhardt, Jiaze Li, Sohir Maskey, Omar Mehio, Pit Neitemeier, Letitia Parcalabescu, Alejandro Petry Pacheco, Pranav Ragupathy, Martin Reinhardt, Peter Rodenkirch Nfissi, Dylan Rodriquez, Leonard Salewski, Raziye Sari, Jordan Sassoon, Ivo Schaper, Ilhan Scheer, Frank Schneider, Philipp Scholl, Lukas Schott, Eric Schreiber, Pablo Schumacher, Alessio Serra, Philipp Siedler, Prabhu Teja Sivaprasad, Mohamed Soliman, Till Speicher, Volker Stampa, Simon Thel, Serge Thilges, Irina Vidal Migallón, Denis Waßmann, Samuel Weinbach, Sascha Wirges, Michael Wüthrich, Alexei Zacharov
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.31945
• Pdf link: https://arxiv.org/pdf/2610.31945
• Abstract This report introduces Kolibri, an English-German mixture-of-experts transformer with 78.1B total parameters and 3.46B active parameters per token, released as open weights under the Apache 2.0 licence. We built Kolibri for sovereign and specialised deployment, with a particular focus on German and on regulated domains such as public administration, industry, and aerospace. To support these settings, we control the full model-development process, including data, architecture, training infrastructure, post-training, and evaluation. We train Kolibri on 24T tokens across pre-training, mid-training, and long-context extension, with German accounting for more than 20% of the data mix by bytes. To reach this share, we complement German datasets with more than 2T German tokens that we curate from the web with our own pipeline and generate synthetically. Post-training combines supervised fine-tuning on curated German and English data with reinforcement learning on more than 1.2M internally curated tasks, which cover reasoning, tool use, instruction following, code, and retrieval. The model activates 4.4% of its parameters per token, and among the open models evaluated in this report, it lies on the Pareto frontier of quality and serving cost in English and German. Kolibri builds on Kolibri Origin, an earlier iteration that validated our full pipeline at scale. Both models come from our Model Factory, which connects data, experimentation, training, evaluation, and deployment through version-controlled workflows.
• 中文摘要 本报告介绍了Kolibri，一款英德专家混合变换器，每个令牌总参数78.1亿，活跃参数34.6亿，作为开放权重发布，采用Apache 2.0许可。我们为主权和专业部署构建了Kolibri，特别关注德语及公共管理、工业和航空航天等受监管领域。为支持这些设置，我们控制完整的模型开发流程，包括数据、架构、培训基础设施、培训后和评估。我们在预训练、中期训练和长上下文扩展中用24T令牌训练Kolibri，德语占数据组合的字节占比超过20%。为实现这一份额，我们用超过2T的德国令牌补充德国数据集，这些令牌是我们自有管道从网络策划并合成生成的。后期训练结合了对策划德语和英语数据的监督微调，以及对超过120万个内部策划任务的强化学习，涵盖推理、工具使用、指令跟踪、代码和检索。该模型每个标记激活4.4%的参数，在本报告评估的开放模型中，它处于英语和德语质量和服务成本的帕累托边界。Kolibri基于Kolibri Origin，这是一个早期版本，该版本在大规模验证了我们的完整流程。这两个模型均来自我们的模型工厂，通过版本控制的工作流连接数据、实验、训练、评估和部署。

REDAR: Reinforcement-Driven Adaptive Regularisation for Class Incremental Learning

REDAR：强化驱动的自适应正则化用于班级增量学习

• Authors: Akila Ayanthi, Darshana Priyasad, Tharindu Fernando, Sridha Sridharan, Clinton Fookes, Peyman Moghadam
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.31991
• Pdf link: https://arxiv.org/pdf/2610.31991
• Abstract Regularisation-based class-incremental learning (CIL) methods control the stability-plasticity trade-off through fixed coefficients tuned offline, independently of the continual learning objective and blind to phase-specific variations in data volume, task difficulty, and class similarity. We argue this is a fundamental mismatch and propose REDAR: Reinforcement-Driven Adaptive Regularisation for Class Incremental Learning, which treats regularisation strength selection as a sequential decision problem. At each incremental phase, a lightweight reinforcement learning (RL) agent observes a compact state of current performance metrics and incoming data statistics, and outputs the distillation coefficients for that phase. To respect CIL's constraint on historical data access, the policy is trained entirely on pseudo-CIL tasks constructed from the initial phase, then deployed on the incremental sequence without further adaptation. Experiments on CIFAR-100 and Tiny-ImageNet demonstrate that REDAR reduces forgetting and improves forward transfer over fixed and heuristic baselines, and that policies trained on short pseudo-task horizons transfer effectively to longer incremental runs.
• 中文摘要 基于正则化的类增量学习（CIL）方法通过离线调校固定系数控制稳定性与可塑性权衡，独立于持续学习目标，且对数据量、任务难度和类相似性等阶段特异性变化视而不见。我们认为这是一种根本性的不匹配，并提出了REDAR：类增量学习的强化驱动自适应正则化，将正则化强度选择视为顺序决策问题。在每个增量阶段，轻量级强化学习（RL）代理观察当前性能指标和输入数据统计的紧凑状态，并输出该阶段的蒸馏系数。为尊重CIL对历史数据访问的约束，策略完全训练于从初始阶段构建的伪CIL任务，然后在增量序列中部署，无需进一步调整。CIFAR-100和Tiny-ImageNet上的实验表明，REDAR减少遗忘，并改善固定和启发式基线的前向转移，且在短伪任务视野上训练的策略有效转移至更长的增量运行。

PPO vs. SAC: A Comparative Study for Cooperative Motion Control of Dual-Arm Space Robots

PPO与SAC：双臂空间机器人协同运动控制的比较研究

• Authors: Nasim Fallahiarezoodar, Zheng H. Zhu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.32034
• Pdf link: https://arxiv.org/pdf/2610.32034
• Abstract The growing population of orbital debris has intensified interest in active debris removal and on-orbit servicing, for which dual-arm space manipulators offer improved grasping stability and load distribution relative to single-arm systems. However, coordinating two manipulators under inter-arm collision avoidance, orientation alignment, and sensing uncertainty remains challenging. This paper develops a coordinated reinforcement learning framework for cooperative motion planning of a dual-arm space manipulator in a target-reaching task with inter-arm collision-avoidance constraints. A centralized state representation combines joint-space, end-effector, and target-relative information, while a multi-objective reward function promotes target-reaching accuracy, orientation alignment, motion smoothness, synchronized task completion, and collision avoidance. Within this framework, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) are compared as representative on-policy and off-policy algorithms, and the learned policies are further evaluated for generalization to previously unseen target configurations. Across five random seeds, SAC substantially outperforms PPO within the considered fixed-base simulation scenarios: it reaches an 80% success threshold in every seed (median 1.6M environment steps) whereas PPO does not reach it within a 4M-step budget, and it attains a higher mean success rate (78.6% versus 7.4%) with lower position and orientation errors and tighter bimanual synchronization. Both algorithms degrade on unseen boundary targets, but SAC retains most of its performance while PPO's collapses. The results also expose a failure mode of PPO: under the two-phase shaped reward, it tends to converge to a reward-loitering local optimum rather than committing to the precision task. This behavior highlights the sensitivity of on-policy learning to reward shaping.
• 中文摘要 轨道碎片数量的增加激发了对主动碎片清除和轨道维护的兴趣，双臂空间机械臂相比单臂系统提供了更好的抓握稳定性和载荷分布。然而，在臂间碰撞避免、方向对准和感知不确定性的情况下协调两个机械臂仍然具有挑战性。本文开发了一个协调强化学习框架，用于双臂空间机械臂在具臂间碰撞避免约束的目标达标任务中协同规划运动。集中状态表示结合了关节空间、端执行器和目标相对信息，而多目标奖励函数则促进目标达标精度、定向对齐、运动平滑、同步任务完成和碰撞避免。在此框架下，比较了近端策略优化（PPO）和软演员-批判者（SAC）作为代表性的开策略和非策略算法，并进一步评估所学策略以推广至此前未见的目标配置。在五个随机种子中，SAC在考虑的固定基模拟场景中显著优于PPO：在每个种子（中位160万环境步）中达到80%的成功阈值，而PPO在400万步预算内未能达到，且其平均成功率更高（78.6%对7.4%），位置和方向误差更低，双手同步更紧密。两种算法在未见边界目标上都会退化，但SAC保持大部分性能，而PPO则崩溃。结果还揭示了PPO的失败模式：在两阶段形奖励下，PPO倾向于趋向于一个奖励徘徊的局部最优，而非承诺执行精确任务。这种行为凸显了策略学习对奖励塑造的敏感性。

DiscoDemo: Discovering Efficient and Diverse Robot Demonstrations for Imitation Learning

DiscoDemo：发现高效且多样化的机器人演示以实现模仿学习

• Authors: Minho Park, Kinam Kim, Donghu Kim, Byungkun Lee, Dongyoon Hwang, Yongjae Shin, Junha Hyung, Hojoon Lee, Jaegul Choo
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32093
• Pdf link: https://arxiv.org/pdf/2610.32093
• Abstract Behavior cloning for robotic manipulation is highly sensitive to the coverage of its training data, yet collecting diverse successful demonstrations remains costly. We study reinforcement learning (RL) not as a deployable controller, but as a data generator for downstream visuomotor imitation learning. Our key premise is that multiple successful solution modes cover a broader range of successful trajectories than repeatedly collecting a single narrow solution. We introduce DiscoDemo, which trains a privileged latent-conditioned RL generator with a parallel reverse curriculum and a success-gated diversity objective to discover diverse solutions while preserving task success. The trained generator produces multiple successful closed-loop behaviors, which are safety-filtered and exported as a static dataset for downstream imitation learning. Across four manipulation tasks, DiscoDemo produces broader successful-state coverage and improves downstream visuomotor policies in both simulation and the real world. Averaged across tasks, downstream success increases to 62% from 40% for the strongest baseline in simulation, and to 78% from 39% on the real robot, respectively. These results suggest that successful solution diversity improves imitation learning for a pretrained flow-matching VLA and remains complementary to corrective data collection. Project page: this https URL
• 中文摘要 机器人操作行为克隆对其训练数据覆盖度高度敏感，但收集多样化成功演示仍然成本高昂。我们研究强化学习（RL）不是作为可部署的控制器，而是作为下游视觉运动模拟学习的数据生成器。我们的核心前提是，多重成功解模式覆盖的成功轨迹范围比反复收集单一狭义解更广泛。我们引入DiscoDemo，它通过训练一个特权潜条件强化学习生成器，配合并行逆向课程和成功门控多样性目标，在保持任务成功的前提下发现多样化解。训练好的生成器产生多个成功的闭环行为，这些行为经过安全过滤后导出为静态数据集，供下游模仿学习使用。在四个操作任务中，DiscoDemo产生更广泛的成功状态覆盖，并改善下游视觉运动策略，无论是在模拟还是现实世界中。跨任务平均，下游成功率分别从模拟中最强基线的40%提升至62%，真实机器人则从39%升至78%。这些结果表明，成功的解多样性提升了预训练的流匹配VLA的模仿学习，并且仍是纠正数据收集的补充。项目页面：此 https URL

PRISM: Promoting Sampling Diversity of Large Language Models with Implicit Stochastic Modulation

PRISM：促进含式随机调制大型语言模型的采样多样性

• Authors: Qixin Xu, Zihan Zhu, Xiaojiang Liu, Bohan Lyu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32101
• Pdf link: https://arxiv.org/pdf/2610.32101
• Abstract Large language models (LLMs) exhibit mode collapse, where their output distribution concentrates on familiar semantic modes. When diversity disappears, repeated sampling becomes semantic repetition rather than discovery, making rare but stronger solutions unlikely to surface and leaving self-evolution and continual learning difficult to sustain. Most remedies intervene only at inference time, shaping sam- ples without making diversity a model capability. We propose PRISM, a training-time framework that internalizes diversity within the model. PRISM combines (i) submodular curation that pairs each prompt with diverse-yet-correct multi-teacher solutions, and (ii) stochastic embedding modulation, which turns special-token perturbations into implicit signals for sampling distinct outputs. A theoretical analysis establishes the stability of this mechanism. On WritingPrompts and BookMIA, PRISM moves the quality-diversity frontier outward, achieving the best generation quality in all model-benchmark settings while maintaining the strongest or near-strongest semantic diversity across two open-weight model families. It also improves sampling efficiency on math and code benchmarks. Beyond inference, PRISM yields a stronger foundation for reinforcement learning, sustaining diverse rollouts and mitigating the diversity collapse commonly observed during RL training.
• 中文摘要 大型语言模型（LLMs）表现出模式崩溃，其输出分布集中于熟悉的语义模式。当多样性消失时，重复采样变成语义重复而非发现，使得罕见但更强的解不太可能出现，自我进化和持续学习难以维持。大多数补救措施仅在推理时介入，塑造sam-ple，而不使多样性成为模型能力。我们提出了PRISM，一种训练时间框架，将多样性内化于模型中。PRISM结合了（i）将每个提示与多样但正确的多教师解决方案配对的亚模块化策展，以及（ii）随机嵌入调制，将特殊令牌扰动转化为用于采样不同输出的隐式信号。理论分析确立了该机制的稳定性。在WritingPrompts和BookMIA上，PRISM将质量-多样性的边界向外推进，在所有模型基准环境中实现最佳生成质量，同时在两个开放权重模型家族中保持最强或接近最强的语义多样性。它还提高了数学和代码基准的抽样效率。除了推理，PRISM为强化学习提供了更坚实的基础，支持多样化的推广，并缓解了强化学习训练中常见的多样性崩溃。

TravSUITE: Traversability via Self-Supervised, Uncertainty-Aware IRL and Terrain Estimation

TravSUITE：通过自我监督、不确定性感知的现实世界和地形估计实现可移动性

• Authors: Samuel Triest, Amirreza Shaban, David D. Fan, Wenshan Wang, Sebastian Scherer
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32130
• Pdf link: https://arxiv.org/pdf/2610.32130
• Abstract Traversability analysis in off-road settings remains a fundamental challenge for mobile robots. Key difficulties include constructing an accurate and expressive local map from multimodal sensor data, and using the map to design traversability rules that yield desirable navigation behavior. Importantly, this system must be resilient to the limited sensing regime brought about by complex environments and high speeds. In this paper, we present TravSUITE, a traversability system suitable for high-speed navigation in off-road environments. TravSUITE consists of two major components: 1) an efficient voxel mapper that leverages visual foundation models (VFMs) to build a rich geometric-semantic local map from streams of on-board sensor data, and 2) a unified neural network that jointly predicts traversability-relevant quantities in bird's eye view (BEV), including geometry, semantics, speed and cost. Our training strategy is entirely annotation-free and self-supervised, leveraging tasks such as map inpainting and inverse reinforcement learning (IRL) to learn both map representations and traversability. We perform a thorough ablation study and comparison to state-of-the-art approaches, and the results indicate that cost learning and auxiliary inpainting each contribute significantly to planning quality, and their combination is critical for achieving state-of-the-art performance in off-road path planning. We also design a simple risk adaptation mechanism to leverage our method's uncertainty estimates at deploy-time, and demonstrate that a combination of inpainting and risk estimation can result in 80% fewer navigation errors and 5% faster autonomous traversal speeds in real-world hardware experiments.
• 中文摘要 越野环境中的可移动性分析仍是移动机器人面临的根本挑战。主要难点包括从多模态传感器数据构建准确且富有表现力的局部地图，以及利用该地图设计可通行性规则以实现理想的导航行为。重要的是，该系统必须能够适应复杂环境和高速带来的有限感测环境。本文介绍了TravSUITE，一种适用于越野环境中高速导航的可移动性系统。TravSUITE由两个主要组成部分：1）高效的体素映射器，利用视觉基础模型（VFM）从车载传感器数据流构建丰富的几何语义局部地图;2）统一的神经网络，能够联合预测鸟瞰图（BEV）中与可通行性相关的量，包括几何、语义、速度和成本。我们的训练策略完全无注释且自我监督，利用地图修复和逆强化学习（IRL）等任务，学习地图表示和可遍历性。我们进行了全面的消融研究，并与最先进方法进行比较，结果表明成本学习和辅助修复均显著提升规划质量，两者结合对于实现越野路径规划的尖端性能至关重要。我们还设计了一种简单的风险适应机制，利用部署时方法的不确定性估计，并证明修复与风险估计结合可使导航错误减少80%，自主穿越速度提升5%，在现实硬件实验中实现。

SyncPro: Synchronous Reasoning and Responding for Proactive Streaming Video Understanding

SyncPro：同步推理与响应，促进主动流媒体视频理解

• Authors: Xueyun Tian, Bingbing Xu, Jiaming Liu, Jinpeng Yu, Yang Shi, Yu Zhao, Ruihua Huang, Yuanzhuo Wang, Huawei Shen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32132
• Pdf link: https://arxiv.org/pdf/2610.32132
• Abstract Proactive streaming video understanding requires assistants to track unfolding events and respond when a user's requested condition is met. To make these decisions, assistants must accumulate relevant evidence and reassess response conditions as new observations arrive. Existing work offers complementary capabilities for this process: proactive video assistants learn when to respond, while streaming video reasoning methods generate scheduled intermediate text to organize observations and support subsequent answers. Together, these capabilities provide a foundation for proactive reasoning, but how to coordinate ongoing reasoning with timely responses remains underexplored. In a direct adaptation, fixed reasoning intervals can trigger updates when nothing relevant has changed or postpone useful updates until the next scheduled step. Sharing one generation path also couples two distinct requirements: preserving evidence for future decisions and producing only the response requested by the user. Moreover, responses generated after reasoning must wait for it to finish, even when sufficient evidence is already available. To address these challenges, we introduce SyncPro, a synchronous dual-stream framework with specialized generation paths for evidence tracking and user-facing responses. Its Inner and Response branches share causal history and activate independently, allowing relevant evidence changes to trigger internal updates and responses to proceed concurrently with ongoing reasoning. We construct 189K dual-stream examples and train SyncPro through supervised fine-tuning followed by trajectory-level reinforcement learning. Experiments across three proactive benchmarks show improved response timing and more accurate answers to user requests.
• 中文摘要 主动流式视频理解需要助手跟踪正在发生的事件，并在满足用户请求的条件时作出反应。为了做出这些决策，助手必须积累相关证据，并在新的观察到来时重新评估响应条件。现有工作为此过程提供了互补功能：主动视频助手学习何时响应，而流式视频推理方法则生成定时的中间文本以组织观察并支持后续答案。这些能力共同为主动推理奠定了基础，但如何协调持续推理与及时响应仍缺乏探索。在直接适应中，固定的推理间隔可以在相关变化未变时触发更新，或将有用更新推迟到下一个预定步骤。共享一条生成路径还带来了两个不同的需求：为未来决策保留证据，并仅生成用户请求的响应。此外，推理后生成的响应必须等待其完成，即使已有足够证据。为应对这些挑战，我们引入了SyncPro，一个同步双流框架，拥有专门的生成路径用于证据追踪和面向用户的响应。其内部分支和响应分支共享因果历史并独立激活，使相关证据变化触发内部更新和响应，同时进行持续的推理。我们构建了18.9万个双流示例，并通过监督微调和轨迹级强化学习训练SyncPro。跨越三个主动基准测试的实验显示响应时机得到改善，用户请求得到更准确的回答。

Geometry-Aware Reinforcement Learning for Close-Fitting Wearable Insertion

几何感知强化学习，用于紧密贴合的可穿戴插入

• Authors: Takuma Tsukakoshi, Elle Miller, Tamon Miyake, Namiko Saito, Yushi Wang, Shigeki Sugano, Tetsuya Ogata, Sethu Vijayakumar
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32143
• Pdf link: https://arxiv.org/pdf/2610.32143
• Abstract Robot-assisted dressing is particularly challenging due to narrow openings and articulated body parts. Even small errors in the relative pose between the body part and garment opening can cause snagging, missed insertion, or incomplete dressing. To address these challenges, we formulate dressing as a geometry-aware insertion problem that represents task-relevant spatial relationships between the body part and object opening, rather than relying on the full deformable-object state. Based on this formulation, we propose a Reinforcement Learning (RL) framework combining a staged curriculum, which progressively guides the policy from coarse alignment to insertion and final placement, with a finger-wise insertion soft gate that provides continuous feedback on individual finger insertion progress and helps suppress premature insertion. We evaluate the framework on rigid and deformable bracelet dressing tasks through component ablations, systematic variations in the initial hand pose, and closed-loop execution of the learned policies on a real robotic system. On the post-training deformable evaluation, the full method reaches 84.7 +- 1.2% success. The insertion gate accounts for the largest gain, and adding the curriculum on top of that gate reaches the top score. On a mannequin hand the same full policy succeeds on 15/20 real-robot trials.
• 中文摘要 机器人辅助穿衣尤其具有挑战性，因为开口狭窄且身体部位可活动。即使是身体部位与衣物开口之间相对姿态的微小误差，也可能导致卡住、漏插或穿衣不完整。为应对这些挑战，我们将穿衣制定为一个几何感知插入问题，代表身体部位与物体开口之间任务相关的空间关系，而非完全依赖可变形物体状态。基于该表述，我们提出了一个强化学习（RL）框架，结合分阶段课程，逐步引导从粗略对齐到插入及最终放置的策略，同时采用手指插入软门，持续反馈单个手指插入进展，帮助抑制过早插入。我们通过组件消融、初始手部姿势的系统性变化以及在真实机器人系统上闭环执行所学策略，评估刚性和可变形手环的框架。在训练后可变形评估中，完整方法成功率达到84.7 +- 1.2%。插入门占最大收益，加上课程设置可获得最高分。在模特手中，同样的完整策略在15/20真实机器人试验中成功。

$Ψ$-0.6: Adaptive Streams of Thought in a Unified Vision-Language World Model

$Ψ$-0.6：统一视觉语言世界模型中的自适应思维流

• Authors: Khai Loong Aw, Klemen Kotar, Baihan Zhang, Yifan Liu, Lilian Chen, Atlas Kazemian, Rahul Venkatesh, Daniel Yamins
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32177
• Pdf link: https://arxiv.org/pdf/2610.32177
• Abstract Humans reason about the physical world by flexibly combining different forms of inference as a problem unfolds. Achieving this flexibility is a major open challenge in AI: capabilities often remain separated across specialized models, limiting how they can be composed and jointly adapted to a task. We introduce $\Psi$-0.6, a unified vision-language world model that adaptively composes perception, language, geometry, and simulation within a shared reasoning process we call streams of thought. These streams interleave images, text, and explicit representations of geometry and motion, all of which the model can generate and interpret through a shared token interface. $\Psi$-0.6 surpasses leading open-weight models zero-shot on multi-view spatial reasoning, with further gains from reasoning through geometric intermediates. It also supports counterfactual simulation: generating and interpreting imagined scenes to determine, for example, whether moving an object would obstruct a pathway. Reinforcement learning from task feedback jointly optimizes the selection, generation, and use of intermediate representations, improving visual question-answering accuracy with gains that transfer to unseen benchmarks. $\Psi$-0.6 offers a path toward adaptive physical reasoning -- not only predicting the physical world but also choosing the representations through which to reason.
• 中文摘要 人类通过灵活组合不同推理形式来推理物理世界，随着问题展开。实现这种灵活性是人工智能领域的一大挑战：能力常常分散在各专业模型之间，限制了它们的组合和联合适应任务。我们介绍$\Psi$-0.6，一个统一的视觉-语言世界模型，它在共享推理过程中自适应地组合感知、语言、几何和仿真，形成我们称为思维流的共享过程。这些流交错包含图像、文本以及几何和运动的显式表示，模型可以通过共享令牌界面生成和解释所有这些内容。$\Psi$-0.6超过领先的开放权重模型在多视角空间推理上采用零射值，并通过几何中间体推理获得更多优势。它还支持反事实模拟：生成和解释想象场景，例如判断物体移动是否会阻碍路径。从任务反馈中进行强化学习，共同优化中间表征的选择、生成和使用，提升视觉问答准确性，并带来可转化为看不见基准的收益。$\Psi$-0.6为自适应物理推理提供了一条道路——不仅预测物理世界，还能选择推理的表征。

Dual-Adaptive Z-Score PPO: Dynamic Clipping Using Action Confidence and Normalized Advantage Magnitude

双自适应Z分数PPO：利用动作置信度和归一化优势幅度进行动态裁剪

• Authors: Taha M. Mahmoud, Naima Kaabouch
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.32194
• Pdf link: https://arxiv.org/pdf/2610.32194
• Abstract Proximal Policy Optimization (PPO) remains a popular on-policy reinforcement learning algorithm owing to its stability and ease of implementation. Its clipping threshold, however, is fixed, so every training sample receives the same trust-region width regardless of how reliable the corresponding policy update is. This work examines adaptive-clipping variants of PPO in which the constant clipping parameter is replaced by sample-dependent thresholds computed from policy and return statistics. The central method, Dual-Adaptive Z-Score PPO (PPO-DAZ3), sets the clipping interval for each sample from two signals: action confidence, given by the probability of the selected action under the current policy, and update confidence, given by the normalized advantage magnitude. The rest of the standard PPO training pipeline is left unchanged. Evaluation on three classic control benchmarks (LunarLander-v3, CartPole-v1, and Acrobot-v1) indicates that adaptive clipping improves learning in some settings but not all. PPO-DAZ3 performs best on LunarLander-v3, the smooth advantage-based variant PPO-AZ3S leads on Acrobot-v1, and standard PPO stays competitive on CartPole-v1, suggesting that adaptive clipping is most beneficial on harder control tasks.
• 中文摘要 近端策略优化（PPO）因其稳定性和易于实现，仍是一种流行的策略上强化学习算法。然而，其剪裁阈值是固定的，因此无论对应策略更新的可靠性如何，每个训练样本都会获得相同的信任区域宽度。本研究研究了PPO的自适应剪裁变体，其中将恒定剪裁参数替换为基于采样的阈值，这些阈值由策略和返回统计计算得出。核心方法——双自适应Z分数PPO（PPO-DAZ3）根据两个信号设定每个样本的剪切区间：动作置信度（由当前策略下所选动作的概率给出）和更新置信（由归一化优势幅度给出）。其余标准PPO训练流水线保持不变。对三个经典控制基准测试（LunarLander-v3、CartPole-v1 和 Acrobot-v1）的评估表明，自适应削波在某些环境中提升了学习效果，但并非所有环境。PPO-DAZ3 在 LunarLander-v3 上表现最佳，基于平滑优势的 PPO-AZ3S 变体在 Acrobot-v1 上表现最佳，标准 PPO 在 CartPole-v1 上保持竞争力，表明自适应削波在更复杂的控制任务中最为有利。

SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

SOPO：跨区级幻觉检测的苏格拉底式引导政策优化

• Authors: Jiawei Dong, Zilong Bai, Pengtian zhu, Peng Zhou
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.32212
• Pdf link: https://arxiv.org/pdf/2610.32212
• Abstract Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks, but suffer from hallucination issues where they generate unfaithful or inconsistent content. While Reinforcement Learning with Verifiable Rewards (RLVR) has shown promising applications in reasoning tasks, its application to span-level hallucination detection is challenged by localization ambiguity, reward sparsity on hard examples, and uncontrolled length inflation. To address these issues, we introduce Socratic Guided Policy Optimization (SOPO), a multi-turn RL framework for robust hallucination detection. The key innovation of SOPO is a Socratic feedback mechanism, where a teacher model guides the reasoning trajectories during rollout by providing indirect hints inferred from the ground truth. This teacher-guided approach enables the model to generate more effective reasoning trajectories for hard examples, thus alleviating reward sparsity. Furthermore, we incorporate Turn-based Advantage Scaling to penalize verbosity bias, encouraging the model to develop efficient and autonomous reasoning capabilities. Empirical results on RAGTruth demonstrate that SOPO (4B) achieves state-of-the-art performance, surpassing leading proprietary models (e.g., GPT-5) and standard RL baselines, while obtaining higher reported mean scores than GRPO on four out-of-domain benchmarks, with mixed evidence against CAPO. We release our training and evaluation code alongside the 1.7B and 4B SOPO models at this https URL.
• 中文摘要 大型语言模型（LLM）已被广泛应用于各种自然语言处理任务，但存在产生不忠或不一致内容的幻觉问题。尽管可验证奖励强化学习（RLVR）在推理任务中展现出有前景的应用，但其在跨度层级幻觉检测中的应用面临局部化模糊性、硬样本奖励稀疏性和不受控长度膨胀的挑战。为解决这些问题，我们引入了苏格拉底式引导策略优化（SOPO），这是一种多回合的强化学习框架，用于稳健的幻觉检测。SOPO的关键创新是苏格拉底式反馈机制，教师模型通过从真实推断的间接线索引导推理轨迹。这种教师引导的方法使模型能够为困难示例生成更有效的推理轨迹，从而缓解奖励稀疏性。此外，我们采用回合制优势缩放（Turn-based Advantage Scaling）来惩罚冗长偏差，鼓励模型发展高效且自主的推理能力。RAGTruth的实证结果表明，SOPO（4B）实现了最先进的性能，超越了领先的专有模型（如GPT-5）和标准强化学习基线，同时在四个域外基准测试中报告的平均分高于GRPO，且CAPO的证据不一。我们将训练和评估代码与1.7B和4B的SOPO模型一同发布，链接为 https URL。

ACWM-RM: A Reference-Free Reward Model from Complementary Mismatch Preferences

ACWM-RM：来自互补不匹配偏好的无参考奖励模型

• Authors: Shibo Hong, Siyu Ren, FengJiao Chen, Zhongyuan Peng, Wei Shi, Yixin Cao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2610.32222
• Pdf link: https://arxiv.org/pdf/2610.32222
• Abstract Reinforcement learning has gained attention for improving action-conditioned world models (ACWMs), as it enables reward-driven optimization of visual quality (VQ) and action-following fidelity (AF) beyond flow-matching optimization. Realizing this potential requires reward signals that reliably rank videos by both VQ and AF. However, existing rewards capture only one dimension: metric-based rewards assess VQ but require ground-truth videos and do not directly assess AF, while inverse dynamics model (IDM) rewards assess AF without references but do not guarantee VQ. Moreover, IDMs, typically trained on real videos, may provide unreliable rewards for generated videos containing visual artifacts and physical inconsistencies. In this work, we introduce the Action and Video Mismatch Benchmark (AVM-Eval), comprising 67.2k pairs that evaluate how reliably IDMs assess AF on real and generated videos. We then propose ACWM-RM, a reward model that jointly assesses VQ and AF without ground-truth videos during reward computation. To train it, we first construct the Action and Video Mismatch Preference Dataset (AVM-Pref) with 506.3k pairs providing complementary supervision: action-mismatch pairs supervise AF, while video-mismatch pairs supervise VQ and expose the model to generated videos. We then pretrain an action-conditioned predictor on real videos to learn forward dynamics before jointly training it with a patch-weighted score head on AVM-Pref to transfer these dynamics to AF assessment on generated videos. Evaluation on AVM-Eval reveals the limited reliability of IDMs on generated videos, while downstream RL experiments show that ACWM-RM outperforms reward baselines in both criteria.
• 中文摘要 强化学习因改进动作条件世界模型（ACWM）而备受关注，因为它能够超越流匹配优化，实现视觉质量（VQ）和动作跟随忠实度（AF）的奖励驱动优化。实现这一潜力需要能够可靠地按VQ和AF排名的奖励信号。然而，现有奖励仅涵盖一个维度：基于指标的奖励评估VQ，但需要实地的视频，且不直接评估AF;而逆动力学模型（IDM）奖励则在无参考的情况下评估AF，但不保证VQ。此外，通常基于真实视频训练的IDM可能对包含视觉伪影和物理不一致的生成视频提供不可靠的奖励。本研究介绍了动作与视频不匹配基准（AVM-Eval），包含67.2k对对，评估IDM如何可靠评估真实和生成视频中的AF。随后，我们提出了ACWM-RM奖励模型，在奖励计算过程中联合评估VQ和AF，无需真实视频。为训练该数据集，我们首先构建了动作与视频不匹配偏好数据集（AVM-Pref），包含506.3k对的互补监督：动作-不匹配对监督AF，视频-不匹配匹配对监督VQ，并将模型暴露于生成视频。随后，我们在真实视频上预训练动作条件预测器以学习前向动态，再与AVM-Pref头部的补丁加权分数联合训练，将这些动态转移到生成视频的AF评估中。AVM-Eval的评估显示IDM对生成视频的可靠性有限，而下游RL实验显示ACWM-RM在两个标准上均优于奖励基线。

MS-TTRL: Breaking the Frequency Conflict in Multiple-Solution Test-Time Reinforcement Learning

MS-TTRL：打破多解测试时间强化学习中的频率冲突

• Authors: Zhixia Zhang, Binqian Xu, Huaiyang Wang, Yikun Ban, Dianbo Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32234
• Pdf link: https://arxiv.org/pdf/2610.32234
• Abstract Scientific, engineering, and mathematical reasoning tasks can admit multiple valid solutions, while exhaustive annotation of the full solution set is often prohibitively costly. Test-time reinforcement learning (TTRL) applies reinforcement learning to unlabeled test instances, offering a label-free approach to multiple-solution reasoning. In this setting, generation frequency serves two opposing objectives: correctness estimation favors frequent outcomes, whereas outcome-level diversity preservation favors less frequent outcomes. We term this tension the \textbf{Frequency Conflict}. To resolve this conflict, we propose MS-TTRL, which obtains frequency-independent correctness estimates through Outcome-Level Self-Verification while reserving generation frequency solely for diversity preservation. Three-State Outcome Cache and the Candidate State support verification reuse across repeated observations and confidence-aware handling of ambiguous outcomes, while Confidence-Gated Inverse Probability Scaling selectively strengthens the learning signals of rare, reliably verified outcomes. Experiments across two benchmarks, four multiple-solution tasks, and three backbone models, each evaluated with five random seeds, show that MS-TTRL improves Recovery Rate by 19.13 percentage points on average over the best-performing label-free baseline and approaches the performance of the supervised IPS-GRPO.
• 中文摘要 科学、工程和数学推理任务可以接受多个有效解，而对完整解集进行详尽注释往往成本过高。测试时间强化学习（TTRL）将强化学习应用于未标记的测试实例，提供了无标签的多解推理方法。在这种设定下，生成频率有两个相反的目标：正确性估计偏向频繁结果，而结果级多样性保持则偏向较少结果。我们将这种张力称为 \textbf{频率冲突}。为解决这一冲突，我们提出了 MS-TTRL，它通过结果层自我验证获得频率无关的正确性估计，同时将生成频率保留用于多样性的保存。三状态结果缓存和候选状态支持在重复观察和置信感知处理模糊结果中的验证重用，而置信门控逆概率尺度则选择性强化罕见且可靠验证结果的学习信号。通过两个基准测试、四个多解任务和三个骨干模型的实验，每个模型均用五个随机种子评估，显示MS-TTRL在最佳无标签基线上平均提升恢复率19.13个百分点，且接近监督的IPS-GRPO性能。

Financial Context-Rich Strategy for Actor-Critic based Agents: A Deep Reinforcement Learning Approach for Stock Trading

基于行为者-批评者代理的富金融背景策略：股票交易的深度强化学习方法

• Authors: Hrishikesh Singh, Aarti Sharma, Millie Pant
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computational Finance (q-fin.CP); Portfolio Management (q-fin.PM)
• Arxiv link: https://arxiv.org/abs/2610.32237
• Pdf link: https://arxiv.org/pdf/2610.32237
• Abstract In modern financial markets, more than 3/4th of the large volume trade is executed by algorithmic trading based on advanced trading strategies to attain higher precision and speed than traditional human traders. The conventional strategies focused on price-based indicators are inadequate in capturing the market dynamism arising from real-world events such as social sentiments, political instability, or sudden economic shifts. With the rise of Deep Reinforcement Learning (DRL) and data availability, there is a requirement for more efficient algorithms that can incorporate these external factors in trading strategies. DRL has shown potential in simulating real-world scenarios and in simplifying complex decision-making since DRL agents can automate the trading process i.e. executing trades in a manner comparable to human traders. They have shown reliable performance since they can update their policies based on risk and quantitative factors. In this paper, we propose a DRL approach based on partial and deep-infusion of sentimental analysis. It features distinct policy designs of actor-critic-based agents: only policy optimization (A2C & PPO) and policy optimization along with Q-learning principles (DDPG, TD3 & SAC). Our findings reveal, that the agents based on policy optimization combined with Q-learning, significantly and consistently demonstrate improved performance upon sentiment deep-infusion in profit maximization as well as risk-averse trading.
• 中文摘要 在现代金融市场中，超过四分之三的大批量交易由基于高级交易策略的算法交易执行，以达到比传统人工交易者更高的精度和速度。传统以价格为基础的策略，无法充分捕捉现实事件如社会情绪、政治动荡或突发经济变化带来的市场动态。随着深度强化学习（DRL）的兴起和数据可用性的提升，需要更高效的算法将这些外部因素纳入交易策略。DRL在模拟现实场景和简化复杂决策方面展现出潜力，因为DRL代理能够自动化交易过程，即以类似人类交易者的方式执行交易。它们表现出可靠的表现，能够基于风险和定量因素更新政策。本文提出基于部分和深度情感分析注入的DRL方法。它具有基于actor-critic代理的不同策略设计：仅有策略优化（A2C和PPO）和策略优化，以及Q学习原则（DDPG、TD3和SAC）。我们的发现显示，基于策略优化与Q学习结合的代理，在情感深度注入和风险规避交易中显著且持续地表现出更好的表现。

Spatial Heuristic Strategy based Diffusion Reinforcement Learning for Online 3D Bin Packing

基于空间启发式策略的扩散强化学习，用于在线3D垃圾桶装箱

• Authors: Tong Li, Qingyang Xu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.32241
• Pdf link: https://arxiv.org/pdf/2610.32241
• Abstract Traditional heuristic methods have accumulated mature experience in core objectives, such as space utilization and layout compactness of 3D bin packing, relying on artificially designed rule-based logic. However, they exhibit obvious limitations in generalization ability and adaptability to dynamic scenarios. Diffusion reinforcement learning, which integrates the spatial search capability of reinforcement learning and the data distribution modeling capability of diffusion learning, has achieved certain progress in action distribution modeling and decision compliance constraints. Nevertheless, there remain issues including insufficient perception of the spatial topological structure in 3D bin packing and limited layout optimization capability. To address these problems, we propose a diffusion reinforcement learning method integrated with spatial heuristic strategies, which combines classical bin packing heuristic rules with a diffusion reinforcement learning framework. Specifically, it compresses the action space through heuristic action pre-screening, introduces a graph neural network to model the spatial structure, and designs a composite reward function incorporating flatness to suppress spatial fragmentation, thereby realizing the organic integration of prior knowledge and adaptive optimization. Experimental results demonstrate that the pro
• 中文摘要 传统启发式方法在核心目标方面积累了成熟经验，如空间利用和三维箱装的布局紧凑性，依赖人工设计的规则逻辑。然而，它们在泛化能力和动态场景适应性方面存在明显局限。扩散强化学习整合了强化学习的空间搜索能力和扩散学习的数据分布建模能力，在动作分布建模和决策合规约束方面取得了一定进展。然而，仍存在诸如对三维箱装填空间拓扑结构感知不足及布局优化能力有限等问题。为解决这些问题，我们提出了一种与空间启发式策略集成的扩散强化学习方法，结合经典分箱装箱启发式规则与扩散强化学习框架。具体来说，它通过启发式动作预筛选压缩了动作空间，引入了图神经网络以建模空间结构，并设计了一个包含平坦性的复合奖励函数以抑制空间碎片化，从而实现了既有知识和自适应优化的有机整合。实验结果表明 pro

Empirically Calibrated Differential Games for Simulating Driver Interactions

用于模拟驾驶员交互的经验校准差分博弈

• Authors: Amirmohammad Khakpour, Nachuan Li, Pedram Beigi, Yu (Marco)Nie
• Subjects: Subjects: Multiagent Systems (cs.MA); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2610.32276
• Pdf link: https://arxiv.org/pdf/2610.32276
• Abstract Game-theoretic models can resolve vehicle interactions without hand-written priority rules, but whether they describe human drivers once calibrated has rarely been tested. This study formulates a general-sum differential game of vehicle interactions at urban intersections, calibrates it to observed driving, and evaluates it on data not used for calibration. Each vehicle minimizes a cost that combines speed tracking, adherence to a map-derived lane route, control effort, gap keeping based on the intelligent driver model, a predictive conflict term, and a contact barrier, while perceiving the other vehicles with a delay. The game is solved with a feedback iterative linear-quadratic algorithm, and ten behavioral parameters per vehicle are calibrated by Bayesian optimization on trajectories recorded at four intersections in Washington, D.C. The calibrated game reproduces the trajectories of 819 vehicles with a mean position error of 1.00 m. When only each vehicle's route is known, as in a simulator, it is the most accurate of the compared models on 96 held-out vehicles (7.42 m), and ten-fold cross-validation shows that it is significantly more accurate than calibrated SUMO (by 3.61 m), an inverse reinforcement learning model (by 0.93 m), and independent and cooperative variants of the same model (by 0.33 and 0.32 m). The solutions, computed in 2.1 s for a typical interaction, are verified approximate equilibria. Finally, the conflict and steering weights, which the data do not identify, can be set from safety and comfort targets; together with a stiffer contact barrier, this removes the collisions that otherwise occur in 43% of tight closed-loop conflicts.
• 中文摘要 博弈论模型可以在没有手写优先规则的情况下解析车辆交互，但它们是否描述了校准后的人类驾驶员，至今很少被测试。本研究提出了城市路口车辆交互的广义微分博弈，校准以适应观察驾驶情况，并基于未用于校准的数据进行评估。每辆车都通过结合速度追踪、遵循地图车道路线、控制努力、基于智能驾驶模型保持间隙、预测冲突项和接触障碍，同时感知其他车辆的延迟，从而实现最小化成本。该博弈通过反馈迭代线性二次算法解决，每辆车通过贝叶斯优化对华盛顿特区四个路口记录的轨迹进行十个行为参数校准。校准后的博弈重现了819辆车辆的轨迹，平均位置误差为1.00米。当仅知道每辆车辆的路线时（如模拟器中），它是96辆被保留车辆（7.42米）比较模型中最准确的，十倍交叉验证显示其准确度显著高于校准后的SUMO（高3.61米）、逆强化学习模型（高0.93米）以及同一模型的独立且合作型变体（0.33米和0.32米）。这些解在典型交互中计算出来，时间为2.1秒，经过验证的近似均衡。最后，数据未识别的冲突权重和转向权重可以从安全和舒适目标中设定;结合更硬的接触障碍，这消除了43%紧密闭环冲突中本来发生的碰撞。

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

从视觉亲和力到多模态强化学习中的学习学分

• Authors: Yuhang Han, Dingning Liu, Yujie Chen, Yaojie Zhang, Jingwei Song, Zequn Yang, Xuanming Shang, Li Ding, Zhijiang Guo, Linfeng Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.32324
• Pdf link: https://arxiv.org/pdf/2610.32324
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves multimodal reasoning, but uniformly assigning a single response-level advantage to all tokens overlooks the distinct roles of visual observations and subsequent reasoning. Visual affinity alone is insufficient: predictive states may align with both image and prompt-context representations, and upweighting valid visual observations can amplify their negative-advantage updates when later reasoning fails. To address this issue, we propose \textbf{Outcome-Conditioned Visual Credit (OVC)}, a plug-in framework that separates visual-association measurement from token-level credit assignment and is adaptable to policy-optimization methods with uniform response-level advantage allocation. Relative Visual Encoding constructs a context-relative visual coordinate by contrasting each predictive state's affinity to image and prompt-context representations. Outcome-conditioned Evidence Calibration combines this coordinate with uncertainty-scaled predictive features and calibrates their contributions to outcome-discriminative evidence through mean contrasts between positive- and negative-advantage responses, scaled by regularized within-side variation. Visual association thus informs credit through observed outcome differences rather than directly determining update priority. Normalized Credit Allocation conditions the calibrated evidence on the response's advantage sign to produce positive token weights with unit global mean. These detached weights preserve advantage signs while modulating individual token-gradient contributions. OVC reuses existing rollouts without auxiliary reward models, process annotations, or inference-time modules. Instantiated through GSPO-token, OVC retains sequence-level clipping and consistently improves six-benchmark mean accuracy over GSPO across three models, with gains of 1.86-2.39 percentage points.
• 中文摘要 带有可验证奖励的强化学习（RLVR）提升了多模态推理能力，但统一地为所有代币分配单一的反应层级优势，忽视了视觉观察及其后续推理的独特作用。仅凭视觉亲和力是不够的：预测状态可能同时与图像和提示上下文表示对齐，且在后续推理失败时，加权有效视觉观察可以放大其负面优势更新。为解决这个问题，我们提出了 \textbf{Outcome-Conditioned Visual Credit （OVC）}，这是一个插件框架，将视觉关联测量与代币级信用分配分离，并可适应策略优化方法，实现一致的响应层级优势分配。相对视觉编码通过对比每个预测状态对图像和提示上下文表示的亲和力，构建了上下文-相对视觉坐标。结果条件证据校准将此协调与不确定性尺度的预测特征结合，并通过正则化的侧内变异尺度进行正负优势响应的平均对比，校准其对结果判别证据的贡献。因此，视觉关联通过观察到的结果差异来判断功劳，而非直接决定更新优先级。归一化学分分配将校准证据在响应优势符号上，生成单位全局均值的正令牌权重。这些分离权重保持优势符号，同时调制单个代币梯度贡献。OVC 重复利用现有的推广，无需辅助奖励模型、过程注释或推理时间模块。通过 GSPO 标记实现，OVC 保留序列级裁剪，并在三个模型中持续提升六个基准平均准确率，提升 1.86-2.39 个百分点。

KnitTrans: Coordinated Trajectory Transfer for Effective and Efficient LLM Reinforcement Learning

KnitTrans：协调轨迹转移，实现高效且高效的大型语言模型强化学习

• Authors: Bangjun Xiao, Chiheng Lou, Xiaodong Ji, Xiangwei Deng, Yikai Zhao, Gang Huang, Hailin Zhang
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32389
• Pdf link: https://arxiv.org/pdf/2610.32389
• Abstract Trajectory transfer from rollout to training affects both learning quality and system efficiency in large language model reinforcement learning (RL). Enforcing batch-wide training requirements can delay the admission of completed trajectories, resulting in higher architectural staleness: the additional parameter-version gap from disaggregated pipelining relative to colocated execution. Beyond admission, misalignment between data access and training consumption causes redundant movement and unnecessary waiting when transferring large trajectory payloads. We present KnitTrans, a system that coordinates the rollout-to-training path through three designs. First, constraint-preserving mini-batch admission allows flexibility in early mini-batch composition, while checking that enough capacity remains to satisfy the remaining quotas. Second, on-demand trajectory data access matches storage units to consumer assignments and overlaps micro-batch data preparation with computation. Third, multimodal encoding deduplication supports reuse across a mini-batch while preserving gradient contributions from all backbone micro-batches. Experiments show up to 93% lower architectural staleness than with train-batch admission, narrowing the learning-quality gap between disaggregated execution and colocated execution. On multimodal workloads, KnitTrans reduces transfer latency by two orders of magnitude and achieves a speedup of up to 5.8 times in transfer plus training over the baseline.
• 中文摘要 从部署到训练的轨迹转移影响了大型语言模型强化学习（RL）中的学习质量和系统效率。强制执行批次范围的训练要求可能会延迟完成轨迹的录入，导致更高的架构陈旧：这也就是拆分流水线相对于共址执行产生的参数版本差距。除了准入外，数据访问与训练消耗之间的不一致会导致在传输大型轨迹有效载荷时出现冗余移动和不必要的等待。我们介绍KnitTrans系统，该系统通过三种设计协调从推广到训练的路径。首先，保持约束的迷你批次准入允许在早期迷你批次组合时保持灵活性，同时检查剩余配额是否有足够的容量满足剩余配额。其次，按需轨迹数据访问将存储单元与消费者分配匹配，并使微批数据准备与计算重叠。第三，多模编码去重支持在微批次间重复使用，同时保留所有骨干微批的梯度贡献。实验显示，架构陈旧度比列车批次承认降低了多达93%，缩小了分解执行与共置执行之间的学习质量差距。在多模态工作负载下，KnitTrans将传输延迟降低了两个数量级，传输加训练速度比基线提升了最多5.8倍。

$R^2E:$ A Role-driven Reward Evolutionary Framework for Automated Reward Function Design

$R^2E：$ 一种角色驱动的奖励进化框架，用于自动奖励函数设计

• Authors: Shouhao Chang, Xuan Liu, Hongye Zhu, Xinning Chen, Shigeng Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32439
• Pdf link: https://arxiv.org/pdf/2610.32439
• Abstract Designing effective reward functions is a critical challenge in Reinforcement Learning (RL), which traditionally requires costly manual trial-and-error. While Large Language Model (LLM)-based methods have shown promise in reward design, undifferentiated sampling and limited use of historical feedback often lead to insufficient exploration of the reward space and optimization instability. To address these challenges, we propose $R^2E$, a Role-driven Reward Evolutionary framework for automated reward function design that structures the process through explicit role specialization. $R^2E$ employs multiple LLM roles with complementary objectives: an Explorer that promotes novelty to expand the search space, a Guardian that performs conservative refinements to improve training stability, and an Evolver that recombines high-performing rewards via crossover and mutation to integrate effective structures. A dynamic scheduling strategy coordinates these roles, progressively shifting the search from broad exploration to focused exploitation. To mitigate optimization instability and provide a medium for cross-role collaboration, $R^2E$ incorporates a Global Elite Pool that retains the best-performing rewards to guide subsequent generations, thereby enabling coordinated interaction among roles and ensuring a reliable and consistent refinement process. Extensive experiments on multiple robotic tasks demonstrate the effectiveness of our method.
• 中文摘要 设计有效的奖励函数是强化学习（RL）中一个关键挑战，传统上需要昂贵的人工试错。虽然基于大型语言模型（LLM）的方法在奖励设计中展现出潜力，但未差别抽样和有限的历史反馈使用常常导致对奖励空间的探索不足和优化不稳定性。为应对这些挑战，我们提出了$R^2E$，一种角色驱动的奖励演化自动化演化框架，通过显式角色专化构建过程。$R^2E$采用多个具有互补目标的LLM角色：一个促进新颖性以扩展搜索空间的探索者，一个通过保守性改进以提升训练稳定性的守护者，以及通过交叉和突变重组高效能奖励以整合有效结构的进化者。动态调度策略协调这些角色，逐步将搜索从广泛探索转向聚焦利用。为缓解优化不稳定性并提供跨角色协作的媒介，$R^2E$ 集成了全球精英池，保留最佳表现奖励以引导下一代，从而实现角色间协调互动，确保优化过程的可靠和一致。多机器人任务的广泛实验证明了我们方法的有效性。

Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping

手风琴：通过固定前缀层循环进行弹性深度推断

• Authors: Yudong Zhang, Wei Ding, Kaili Zheng, Yuchen Ding, Zhenyu Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2610.32492
• Pdf link: https://arxiv.org/pdf/2610.32492
• Abstract Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.
• 中文摘要 大型语言模型服务于具有多样推理预算的应用，但为每个预算训练和维护独立模型会产生大量成本。最近的循环变换器通过参数重用实现了灵活的推理计算，但保持不同推理预算间的强性能仍是一个开放挑战。我们介绍了Accordion，这是一种基于专家混合的模型，通过固定前缀层循环支持五个推理预算。每个预算遵循预定义的执行路径，而最小预算保留未循环的模型。联合预训练每个微批次采样一个预算，并在不增加跨预算一致性目标的情况下优化其下一标记的预测丢失。我们通过监督微调和强化学习进一步保持这些推理预算，参数组合并和预算匹配的多教师策略蒸馏将三位领域专家整合到一个模型中。经过3.6T的预训练令牌后，推理深度增加使Accordion-6B-A0.5B在37个任务中的五类宏平均提升3.01个百分点，中间预算以一半的额外层执行捕获了89%的收益。最终模型在相应的任务与预算平均上超过了每位领域专家，同时在所有预算中实现了强劲的指令跟随性能，并在数学推理和代码生成方面保持了深度依赖的提升。

MusicTwins: Training Audio-Language Models to Listen like Their Persona

MusicTwins：训练音频语言模型像其本人一样倾听

• Authors: Milan Liessens Dujardin, Kevin Miao
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.32495
• Pdf link: https://arxiv.org/pdf/2610.32495
• Abstract People differ in how they perceive music, what associations they make with it, what they like about it, and how they respond to it. Computational models intended to simulate people in environments that include music should account for these differences. Although LLMs have been shown to reproduce aspects of human behavior, whether audio-language models (ALMs) can reproduce a specific person's responses to textual and musical stimuli remains underexplored. We introduce MusicTwins, a framework that combines persona summaries, personal memory, and post-training to generate person-specific responses. We construct persona representations from human writings, capturing background, views, personality, writing style, and a musical profile, and post-train Audio Flamingo Next through supervised fine-tuning and reinforcement learning. Evaluation on ten held-out personas shows notable improvements for Audio Flamingo Next in persona fidelity, including personality, writing style, and responses to music, supporting MusicTwins as a foundation for training audio-language models to listen like their persona.
• 中文摘要 人们在感知音乐的方式、与音乐的联想、喜欢音乐的方面以及对音乐的反应方式各不相同。用于模拟包含音乐环境下的人类的计算模型应考虑这些差异。尽管大型语言模型已被证明能再现人类行为的某些方面，但音频语言模型（ALMs）是否能够重现特定个体对文本和音乐刺激的反应仍然缺乏探讨。我们引入了MusicTwins，这是一个结合人格总结、个人记忆和后期训练以生成个体特定反应的框架。我们从人类文字构建人格表征，捕捉背景、观点、个性、写作风格和音乐档案，并通过监督微调和强化学习对音频火烈鸟Next进行训练。对十个被保留的人物形象的评估显示，Audio Flamingo Next 在角色真实度方面有显著提升，包括个性、写作风格和对音乐的反应，支持 MusicTwins 作为训练音频语言模型以模仿其角色的基础。

Optimization, Optimal Control, and Reinforcement Learning for Robotics

机器人的优化、最优控制与强化学习

• Authors: Yonghyeon Lee
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32538
• Pdf link: https://arxiv.org/pdf/2610.32538
• Abstract These lecture notes provide an introduction to optimization, optimal control, and reinforcement learning for robotics, emphasizing their mathematical connections and practical relevance. They present established material compiled and edited by Yonghyeon Lee, rather than new research results. The notes were developed through a human and AI collaborative writing process, with AI assistance in drafting, organizing, and refining the exposition. This first version is a preliminary collection of notes, not yet a fully developed textbook, and currently contains no figures or graphs. The notes may be revised, corrected, and expanded in future versions.
• 中文摘要 这些讲义介绍了机器人的优化、最优控制和强化学习，强调其数学联系和实际相关性。它们呈现的是由李永贤整理和编辑的既有材料，而非新的研究成果。笔记是通过人类和人工智能的协作写作过程开发的，人工智能协助起草、组织和完善说明。第一版是初步笔记集，尚未成为完全成熟的教材，目前没有图表或图表。笔记未来版本可能会进行修订、修正和扩展。

DT-JEPA: Learning to Predict Future Representations for Decision-Making

DT-JEPA：学习预测未来表征以进行决策

• Authors: Changxu Ni, Yiyang Ni, Jun Li, Feiran You, Hongyang Du, Haitao Zhao, Shi Jin
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32657
• Pdf link: https://arxiv.org/pdf/2610.32657
• Abstract Decision Transformer (DT) has demonstrated strong performance in various offline reinforcement learning (RL) tasks by conditioning action generation on expected future returns and historical trajectories. However, treating expected future returns as the condition provides an incomplete description of the future, as trajectories with different state transitions and behaviors may share similar returns. Moreover, expected future returns inherently depend on task-specific reward functions, which makes policy learning reliant on reward-labeled datasets and restricts its scalability to tasks with similar but unseen rewards. Recently, Joint Embedding Predictive Architectures (JEPA) have emerged as a promising paradigm of self-supervised learning-based world models, which learn predictive representations of future dynamics in latent space rather than reconstructing future states in the original space. Inspired by JEPA, we propose DT-JEPA, an offline pretraining and online fine-tuning framework that learns general knowledge from reward-free offline trajectories and achieves excellent adaptation to unseen tasks. Instead of predicting a single deterministic future embedding, we design a novel probabilistic predictor to model the conditional distribution of future representations in the latent space, thereby capturing both predictive information and uncertainty about future dynamics. Theoretically, we show that 1) the JEPA objective seeks to maximize a variational lower bound on the mutual information between historical representations and future trajectories; and 2) predictive future-conditioned policy learning admits a finite-sample performance bound with an O(N-1/4) estimation term. Experiments on D4RL benchmarks demonstrate that DT-JEPA achieves efficient adaptation across varying offline RL tasks, outperforming traditional DT-based and other offline RL methods.
• 中文摘要 决策变换器（DT）通过将动作生成条件置于预期未来回报和历史轨迹，在多种离线强化学习（RL）任务中表现出强劲表现。然而，将预期未来回报视为条件，只能不完整描述未来，因为不同状态转变和行为的轨迹可能共享相似的回报。此外，预期未来回报本质上依赖于任务特定的奖励函数，这使得策略学习依赖于奖励标记数据集，并限制其扩展性仅限于具有相似但未见奖励的任务。近年来，联合嵌入预测架构（JEPA）作为一种有前景的自监督学习基于世界模型的范式出现，它在潜空间学习未来动态的预测性表征，而非在原始空间中重建未来状态。受JEPA启发，我们提出了DT-JEPA，一种离线预训练和在线微调框架，能够从无奖励的离线轨迹中学习常识，并实现对未见任务的卓越适应。我们设计了一个新的概率预测器，用于模拟潜在空间中未来表示的条件分布，从而同时捕捉预测信息和未来动态的不确定性。理论上，我们证明：1）JEPA目标旨在最大化历史表示与未来轨迹之间互信息的变分下界;2）预测性未来条件政策学习允许有限样本性能界限，估计为O（N-1/4）。D4RL基准测试的实验表明，DT-JEPA在不同离线强化学习任务中实现高效适应，优于传统基于DT及其他离线强化学习方法。

Predictive Long-Horizon Progress Modeling with Action-Aware Quantification

具有行动感知量化的长期预测性进展建模

• Authors: Tianchi Xu, Yalin Zheng, Yangxin Zhu, Ruonan Ren, Zheng Fu, Yunlong Wang, Yuanxin Zhong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32776
• Pdf link: https://arxiv.org/pdf/2610.32776
• Abstract Quantitative and dense task progress feedback is essential for robotic manipulation across data curation, policy evaluation, and reinforcement learning. Existing progress annotations often rely on sparse keyframe markings derived from subjective manual judgments, limiting their scalability and cross-task consistency, particularly in long-horizon manipulation. To overcome these challenges, we propose action-aware quantification, a unified progress specification that decomposes long-horizon trajectories into object-level subtasks via a vision-language model (VLM) and assigns weighted action credits to capture fine-grained execution dynamics. Using VLM auto-labeling and human calibration, we construct an 800+ hour training video dataset and LOOM, a long-horizon progress benchmark containing 19,618 human-calibrated evaluation frames. To model these dense progress dynamics, we introduce NimCritic, a predictive progress critic trained via Joint Video-Progress Learning. We further incorporate Historical Progress Perturbation to mitigate sequential drift, and design a contrastive Video-Progress Alignment objective that bridges asymmetric expert streams by mutually grounding intermediate visual dynamics and denoised progress trajectories. NimCritic matches or outperforms state-of-the-art process reward models (PRMs) on the established Video Progress Benchmark (VPB) and LOOM, surpassing GVL (GPT-6-Sol) on VPB by about 65% in terminal success accuracy (TSA). On the large-scale RoboArena policy evaluation benchmark, it also retrieves human-selected successful rollouts with about 97% overall recall. These results establish action-aware quantification as a scalable foundation for dense progress feedback in robotic manipulation.
• 中文摘要 定量且密集的任务进展反馈对于机器人在数据管理、策略评估和强化学习中的操作至关重要。现有的进度注释通常依赖于主观手动判断得出的稀疏关键帧标记，限制了其可扩展性和跨任务一致性，尤其是在长视野操作中。为克服这些挑战，我们提出了动作感知量化，这是一种统一的进展规范，通过视觉语言模型（VLM）将长视野轨迹分解为对象级子任务，并赋予加权动作积分以捕捉细粒度执行动态。利用VLM自动标记和人工校准，我们构建了一个800+小时的训练视频数据集和LOOM，这是一个包含19,618个人工校准评估帧的长视野进展基准。为建模这些密集进展动态，我们引入了NimCritic，一款通过联合视频-进展学习训练的预测性进展批评器。我们还进一步引入历史进展扰动以减轻顺序漂移，并设计了一个对比性的视频-进展对齐目标，通过相互接地中间视觉动态和去噪进展轨迹，桥接非对称专家流。NimCritic在既有视频进展基准（VPB）和LOOM上表现优于或超越最先进的过程奖励模型（PRM），在终端成功率（TSA）方面超过VPB上的GVL（GPT-6-SOL）约65%。在大规模RoboArena策略评估基准中，它还检索了人工选择的成功推广，整体回忆率约为97%。这些结果确立了动作感知量化作为机器人操作中密集进展反馈的可扩展基础。

Regularised RL improves sim-to-real transfer

正规化强化学习改进了模拟到现实的传输

• Authors: Danilo Vucetic, Mohamad H. Danesh, Florence Clerc, Hsiu-Chin Lin, Jonathan Tremblay
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32855
• Pdf link: https://arxiv.org/pdf/2610.32855
• Abstract Regularised reinforcement learning (RL) has provable robustness guarantees that should ease the transfer of policies from simulation to robots, known as sim-to-real. Yet, weakly regularised algorithms like Proximal Policy Optimisation (PPO) remain the de facto choice in sim-to-real pipelines, while the practical benefits of regularisation for transfer remain poorly characterised. In this work, we study the sim-to-real performance of commonly used RL algorithms, with the aim of characterising the effects of regularisation on robotic performance. To that end, we derive Soft MellowMax Actor-Critic (SMM-AC), a novel regularised RL algorithm based on the soft mellowmax value function, which carries strong robustness guarantees. On MuJoCo continuous control benchmarks, SMM-AC is competitive with baselines like Soft Actor-Critic (SAC). On real robots, SMM-AC achieves higher return, while exhibiting greater robustness to perturbations in observations, the environment, and dynamics than baselines. Together, these results demonstrate that regularised RL improves real-world policy deployment through better robustness to the sim-to-real gap, and changing real-world conditions.
• 中文摘要 正则化强化学习（RL）具有可证明的鲁棒性保证，这将有助于策略从仿真转移到机器人，即模拟到现实。然而，像近端策略优化（PPO）这样的弱正则化算法仍然是模拟到现实流水线中的事实选择，而正则化在传输中的实际效益仍被描述得很有限。本研究中，我们研究了常用RL算法的模拟到现实性能，旨在表征正则化对机器人性能的影响。为此，我们推导出了Soft MellowMax Actor-Critic（SMM-AC），这是一种基于软Mellowmax价值函数的新颖正则化RL算法，具有强的鲁棒性保证。在MuJoCo连续控制基准测试中，SMM-AC与Soft Actor-Critic（SAC）等基线具有竞争力。在真实机器人上，SMM-AC实现了更高的回报，同时对观测、环境和动力学扰动表现出比基线更强的鲁棒性。这些结果共同表明，正则化强化学习通过更稳健模拟与现实差距以及变化的现实环境，提升了现实政策的部署能力。

NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding

NgramRL：通过推广感知的N-gram推测性解码加速强化学习后训练以实现结构化生成

• Authors: Nilesh Prasad Pandey, Jangseon Park, Jason Kong, Onat Gungor, Tajana Rosing
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2610.32881
• Pdf link: https://arxiv.org/pdf/2610.32881
• Abstract Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but its cost is dominated by autoregressive rollout generation, which can exceed 70% of training time. Existing speculative-decoding methods are poorly suited to this setting: model-based drafters consume scarce GPU memory and may become misaligned as the policy evolves, while existing model-free methods do not explicitly exploit repetition across training epochs. We observe that recent rollouts of the same prompt provide a high-quality, self-updating draft source. Based on this insight, we propose NgramRL, a per-prompt n-gram drafter constructed from bounded recent rollout history. NgramRL requires no auxiliary model, drafter training, or persistent GPU-resident drafter state, while target-model verification preserves the rollout distribution. To hide CPU cache lookup and scheduling, we further introduce an adaptive fan-out prefetcher that predicts likely verification outcomes (j, b), consisting of the accepted draft length j and target-emitted continuation token b. A causal budget allocator concentrates prefetches on high-probability outcomes, while uncovered outcomes fall back to sequential lookup without changing committed tokens. Across two model families and three reasoning and structured-generation workloads, NgramRL commits 3.60-7.77 tokens per target-model forward in the warm-cache regime, up to 1.99x more than SuffixDecoding and 7.0x more than prompt-lookup decoding. On Spider, the prefetcher covers 96.6% of Qwen3-8B outcomes at a candidate budget of 16 and improves measured rollout throughput by 1.38x over synchronous NgramRL, approaching its 1.40x overlap ceiling.
• 中文摘要 强化学习（RL）在训练后提升了大型语言模型的推理能力，但其成本主要体现在自回归的展开生成上，后者可能超过训练时间的70%。现有的推测解码方法并不适合这种环境：基于模型的起草器消耗稀缺的GPU内存，随着策略演进可能出现错位，而现有的无模型方法则未明确利用训练时代间的重复性。我们观察到，同一提示词的最新推广提供了高质量、自我更新的草稿源。基于这一见解，我们提出了NgramRL，一种由有界近期发布历史构建的每提示符n-gram生成器生成器。NgramRL不需要辅助模型、生成器训练或持续存在GPU驻留的生成器状态，而目标模型验证则保留了部署分布。为了隐藏CPU缓存查找和调度，我们进一步引入了自适应的扇出预取器，预测可能的验证结果（j， b），包括被接受的草稿长度j和目标发出的延续令牌b。因果预算分配器将预取集中于高概率结果，而未发现的结果则退回顺序查找而不更改已提交的标记。在两个模型家族和三种推理和结构化生成工作负载中，NgramRL在温缓存区间每个目标模型前期提交3.60-7.77个令牌，最多比后缀解码多1.99倍，比提示查找解码多7.0倍。在Spider上，预取器覆盖了Qwen3-8B结果的96.6%，候选预算为16，并使测量的部署吞吐量比同步NgramRL提升1.38倍，接近其1.40倍的重叠上限。

Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory

多步过渡预测的强化学习：贝尔曼理论

• Authors: Chenbei Lu, Zaiwei Chen, Tongxin Li, Chenye Wu, Adam Wierman
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32887
• Pdf link: https://arxiv.org/pdf/2610.32887
• Abstract Multi-step predictions of future transition realizations can improve reinforcement learning, but their value remains difficult to quantify. Directly including these predictions in the state can lead to exponential state-space growth, making direct dynamic programming computationally expensive. We study discounted finite MDPs under a block-feedback protocol: at the start of each K-period block, the decision maker observes $K$-step probabilistic transition predictions for a subset of actions and adapts decisions to observed states within the block. We show that the optimal value satisfies a Bayesian Bellman equation on the original state space. This representation reveals that predictive information creates value by recursively changing the local order of maximization and expectation in the Bellman expansion. We formalize this effect through a Bellman--Jensen analysis and quantify the effects of prediction horizon, error, action coverage, and within-block feedback. We propose block-feedback Bayesian offline learning with online adaptation (BF--BOLA), a model-based algorithm for learning near-optimal policies, and establish upper and lower sample-complexity bounds that match up to logarithmic factors in the stated regimes. The results show that predictions can reduce transition learning to uncovered rows and that learning a near-optimal policy can require fewer prediction samples than estimating the Bayesian value function.
• 中文摘要 未来过渡实现的多步预测可以改善强化学习，但其价值仍难以量化。直接将这些预测纳入状态可能导致指数级状态空间增长，使直接动态规划计算成本高昂。我们在块反馈协议下研究贴现有限MDP：在每个K周期块开始时，决策者观察$K步概率转移预测，针对部分动作，并根据区块内观察到的状态调整决策。我们证明最优值满足原始状态空间上的贝叶斯贝尔曼方程。该表示表明，预测信息通过递归改变贝尔曼展开中的局部最大化和期望阶来创造价值。我们通过贝尔曼-詹森分析形式化该效应，量化预测视界、误差、动作覆盖率及区块内反馈的影响。我们提出了带有在线适应的区块反馈贝叶斯离线学习（BF--BOLA），这是一种基于模型的算法，用于学习近最优策略，并建立与所述区间对数因子匹配的样本复杂度上限和下限。结果表明，预测可以减少对未覆盖行的转移学习，且学习近似最优策略所需的预测样本比估计贝叶斯价值函数更少。

HoloBoxing: A Simple Recipe for Humanoid Boxing with 7 Minutes of Single-GPU Training

全息拳击：7分钟单GPU训练，简易实现类人拳击的配方

• Authors: Xihan Ma, Bo Zhang, Boxiong Li, Yuhan Bao, Ziyue Liu, Yucheng Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32909
• Pdf link: https://arxiv.org/pdf/2610.32909
• Abstract HoloBoxing presents a simple recipe for learning humanoid boxing by building on a pretrained whole-body controller. The recipe separates reusable motor skills from task decisions: a compact planner requests future body-point targets, while the frozen controller handles joint-level execution. A learned decoder maps 32-dimensional actions to structured target windows, and measured-state imitation initializes the planner before reinforcement learning. Task learning then refines interaction behavior while reusing the decoder and controller, avoiding the need to relearn whole-body control. This combination keeps task optimization focused on a compact upper policy and supplies an initial behavior before reward-driven exploration. Starting from that available controller, the recipe accounts for about seven minutes of additional training, including reused representation fitting and a single-GPU task run. This excludes controller pretraining, data preparation, evaluation, and search. Simulated boxing experiments and component controls evaluate the resulting behavior and the roles of imitation initialization and the target representation.
• 中文摘要 全息拳击提供了一个简单的学习类人拳击的配方，方法是基于预训练的全身控制器。该配方将可重复使用的运动技能与任务决策分离：紧凑的计划器请求未来的身体点目标，而冻结的控制器则处理关节层级的执行。学习得的解码器将32维动作映射到结构化的目标窗口，测量状态模拟在强化学习前初始化计划器。任务学习随后在重用解码器和控制器的同时优化交互行为，避免了重新学习全身控制的需求。这种组合使任务优化聚焦于紧凑的上层策略，并在奖励驱动探索前提供初始行为。从可用控制器出发，该配方约需七分钟额外训练，包括重复使用的表示拟合和单GPU任务运行。这不包括控制器预训练、数据准备、评估和搜索。模拟拳击实验和组件控制评估所得行为以及模拟初始化和目标表示的作用。

Recommender regret for improving regularised RL estimation with bandits and MCTS

推荐人对用bandit和MCTS改进正则化强化学习估计感到遗憾

• Authors: Danilo Vucetic, Alexandre Larouche, Mohammad Pedramfar, Audrey Durand, Gauthier Gidel
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32910
• Pdf link: https://arxiv.org/pdf/2610.32910
• Abstract Regularised reinforcement learning (RL) algorithms rely on estimators, like Monte Carlo tree search or one-step look-ahead, to provide policy, value, or Q-value estimates in training and inference. However, existing bandit-based estimation frameworks evaluate sampling efficiency, not the quality of estimates. We address this by formalising estimate generation as a recommendation process, treating bandit-based sampling as an improvement operator. To evaluate the quality of a recommendation (estimate), we define regulariser-aware recommender regrets, analysing their lower bounds, and proving that simple algorithms like uniform sampling can achieve these bounds. Finally, we unify bandit- and search-based target improvement under recommender MCTS. Across synthetic trees, arcade games, and scientific discovery tasks, recommenders often yield faster target convergence and superior downstream performance. Our experiments characterise for the first time the potential of regularised MCTS in scientific applications.
• 中文摘要 正则化强化学习（RL）算法依赖估计器，如蒙特卡洛树搜索或一步前瞻，在训练和推断中提供策略、价值或Q值估计。然而，现有基于bandit的估计框架评估的是抽样效率，而非估计质量。我们通过将估计生成形式化为推荐过程，将基于bandit的抽样视为改进算子来解决这个问题。为了评估推荐（估计）的质量，我们定义了正则化器感知推荐者的遗憾，分析其下界，并证明像均匀抽样这样的简单算法能够实现这些界限。最后，我们将基于bandit和搜索的目标改进统一在推荐者MCTS下。在合成树、街机游戏和科学发现任务中，推荐者通常能更快实现目标收敛和更优的下游性能。我们的实验首次表征了正则化MCTS在科学应用中的潜力。

Where Rounding Happens Matters: Rounding-Location Alignment Stabilizes LLM Reinforcement Learning

四舍五入发生的地点很重要：四舍五入与位置对齐稳定了LLM强化学习

• Authors: Yijing Ke, Tianle Cai, Ruosong Wang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32951
• Pdf link: https://arxiv.org/pdf/2610.32951
• Abstract Reinforcement learning for large language models often uses separate systems for inference and training. Even with identical weights and BF16 tensors, implementation differences can make an intended on-policy update off-policy and destabilize training. Existing remedies align the entire inference and training stacks, change global precision, or modify the learning algorithm. We ask whether aligning selected parts of the execution stack can stabilize BF16 RL while other implementation differences remain. We align training-side rounding locations with inference in three regions---RoPE, SwiGLU, and residual RMSNorm---by retaining selected intermediates in FP32 until their BF16 outputs. Across four Qwen-family models, aligning all three regions prevents the collapse observed under the default BF16 baseline; RoPE-path alignment alone also suffices in the studied runs, despite remaining numerical discrepancies. The interventions keep the inference engine, matrix-multiplication and attention kernels, and learning objective unchanged.
• 中文摘要 大型语言模型的强化学习通常使用独立的推理和训练系统。即使权重相同且BF16张量相同，实现差异仍可能使预期的on-policy更新变为非策略，从而破坏训练稳定性。现有的补救措施会对齐整个推理和训练堆栈，改变全局精度，或修改学习算法。我们探问，对执行栈中选定部分的对齐是否能稳定BF16的强化学习，同时存在其他实现差异。我们通过保留FP32中的部分中间节点直到BF16输出，将训练端四舍五入位置与三个区域---RoPE、SwiGLU和残余RMSNorm---对齐。在四个Qwen家族模型中，对齐所有三个区域可防止默认BF16基线下观察到的崩溃;仅靠RoPE路径对齐即可在研究运行中实现，尽管仍有数值差异。干预措施保持推理引擎、矩阵乘法和注意力核以及学习目标不变。

RynnUnify: Towards Unified Embodied Brain

RynnUnify：迈向统一具身大脑

• Authors: Minghao Zhu, Bohan Hou, Kun Xiang, Zixuan Wang, Yinzhou Tang, Yifei Yang, Zhikai Wang, Tianyi Zhang, Kehan Li, Haoyu Zhao, Siteng Huang, Leigang Qu, Sicong Leng, Dongchi Huang, Mingxiu Chen, Shijian Lu, Yue Wang, Xiaodan Liang, Zhenyu Zeng, Deli Zhao, Chengju Liu, Jianfei Yang, Xin Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32965
• Pdf link: https://arxiv.org/pdf/2610.32965
• Abstract We introduce RynnUnify, a 16B unified embodied model that combines strong embodied understanding with long-horizon visual planning through interleaved video--language reasoning. With capabilities spanning spatial reasoning, physical prediction, and grounding, RynnUnify connects an understanding of the physical world with visual anticipation of how it evolves across successive subtasks. Building on these capabilities, we further develop an agentic extension through reinforcement learning to selectively use video predictions for reasoning about unobserved states and interaction outcomes. We also develop a world-action extension of the full RynnUnify model, translating its understanding and prediction capabilities into robot control. To evaluate these capabilities, we introduce Rynn-VP and Rynn-GEU for long-horizon visual planning and anticipatory reasoning. Experiments show that RynnUnify achieves the highest average embodied understanding score among the evaluated unified multimodal models and improves the Rynn-VP overall score by 9.75 points over Cosmos 3 Nano. Its agentic extension improves the Rynn-GEU average by 6.0 points over direct reasoning, while the world-action extension achieves 51.5\% success on 20 selected composite RoboCasa365 tasks and 46.7\% average success across three real-robot tasks. Together, these results establish RynnUnify as a shared foundation for embodied understanding, visual planning, and robot control, taking a step toward a unified embodied brain.
• 中文摘要 我们介绍RynnUnify，一个16B统一具身模型，结合了强的具身理解与通过交织视频的长期视觉规划——语言推理。凭借跨空间推理、物理预测和基础的能力，RynnUnify将对物理世界的理解与其在后续子任务中演变的视觉预期连接起来。基于这些能力，我们进一步通过强化学习开发了代理扩展，选择性地利用视频预测推理未观察状态和交互结果。我们还开发了完整RynnUnify模型的世界动作扩展，将其理解和预测能力转化为机器人控制。为评估这些能力，我们引入了Rynn-VP和Rynn-GEU用于长视野视觉规划和预判推理。实验显示，RynnUnify在评估的统一多模态模型中获得了最高的平均体化理解分数，并将Rynn-VP整体得分提升了9.75分，优于Cosmos 3 Nano。其能动扩展使Rynn-GEU平均比直接推理提升6.0分，而world-action扩展在20个选定的复合RoboCasa365任务中成功率为51.5%，在3项真实机器人任务中平均成功率为46.7%。这些结果共同确立了RynnUnify作为具身理解、视觉规划和机器人控制的共享基础，迈向统一具身大脑迈出一步。

RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

RobustRLlib：一个统一的强健强化学习算法库和基准测试

• Authors: Xi Chen, Fengqi Hu, Zeyu Shen, Jiaying Xu, Shangding Gu, Pan Xu, Anqi Liu, Laixi Shi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2610.32971
• Pdf link: https://arxiv.org/pdf/2610.32971
• Abstract Robust reinforcement learning (RL) is critical for reliable real-world deployment, where policies may fail under diverse shifts and disruptions. Existing robust RL algorithms offer a rich set of mechanisms and principles, yet two major barriers limit their broader potential: limited algorithm reusability and an unclear empirical capability boundary. We introduce RobustRLlib, an algorithm-centric library and benchmark that addresses both challenges. It integrates 17 representative robust online, offline, and safe RL algorithms under a unified interface for substantially improved reusability, together with an enhanced evaluation platform spanning diverse real-world shifts and advanced humanoid and vision-language-action tasks. Systematic evaluation over 273 perturbation configurations provides an up-to-date capability map of current robust RL algorithms, revealing where they remain effective and where substantial robustness gaps persist. As an open-source, user-friendly platform, RobustRLlib offers a reproducible and extensible codebase for applying existing methods to new problems and developing new robust RL algorithms.
• 中文摘要 稳健强化学习（RL）对于可靠的现实世界部署至关重要，因为在各种变化和干扰下，策略可能失效。现有的稳健强化学习算法提供了丰富的机制和原理，但两大障碍限制了其更广泛的潜力：算法的可重用性有限和经验能力边界不明确。我们介绍了RobustRLlib，一个以算法为中心的库和基准测试，解决了这两个挑战。它整合了17个具有代表性的稳健在线、离线和安全强化学习算法，通过统一界面实现显著提升的重用性，并配备了涵盖多种现实世界变化及高级类人生物和视觉语言-行动任务的增强评估平台。系统性评估273种扰动配置，提供了当前稳健强化学习算法的最新能力图，揭示了其仍有效及存在显著鲁棒性差距的地区。作为一个开源、用户友好的平台，RobustRLlib 提供了一个可重复且可扩展的代码库，用于将现有方法应用于新问题并开发新的稳健强化学习算法。

Keyword: diffusion policy

RoboCode: Learning Reusable Action Primitives for Robot Manipulation

RoboCode：学习可重复使用的动作原语以实现机器人操作

• Authors: Xianyue Peng, Devi Amarsaikhan, Iman Soltani
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2610.32225
• Pdf link: https://arxiv.org/pdf/2610.32225
• Abstract Recent robot manipulation policies commonly predict continuous action chunks directly from observations, without explicitly modeling reusable structure in the action space. However, manipulation demonstrations often contain recurring short-horizon motion patterns that are shared across tasks. We introduce \textsc{RoboCode}, which learns these patterns through a shared soft-gated primitive action interface. Rather than assigning each action chunk to a single discrete code, \textsc{RoboCode} represents it as a soft composition of multiple learned primitives, providing a flexible intermediate representation of short-horizon actions. The primitive interface is learned from action chunks pooled across multiple tasks and then frozen while task-specific state encoders are trained to map current state inputs into the shared latent space. Experiments on five ManiSkill manipulation tasks address three questions. First, the learned primitive interface reconstructs action chunks with an average reconstruction MSE of 0.0187. Second, analysis of code activations reveals structured code reuse across tasks. Third, \textsc{RoboCode} achieves the highest or tied closed-loop success rate on all evaluated tasks, with an average success rate of 75.6%, compared with 68.8% for ACT and 41.6% for the larger Diffusion Policy baseline. These results suggest that soft-gated primitive codebooks provide an effective and interpretable intermediate action representation for robot manipulation.
• 中文摘要 近期机器人操作策略通常直接从观察中预测连续动作块，而未明确建模动作空间中的可重用结构。然而，操作演示中常包含跨任务共享的重复短视距运动模式。我们引入 \textsc{RoboCode}，它通过共享的软门控原始动作接口学习这些模式。\textsc{RoboCode} 不将每个动作块分配给单一离散代码，而是将其表示为多个学习原语的软组合，提供短视野动作的灵活中间表示。原始界面是从多个任务中聚集的动作块学习的，然后在任务特定状态编码器训练以映射当前状态输入到共享潜在空间时被冻结。五个 ManiSkill 操作任务的实验解决了三个问题。首先，学习的原始接口以平均重建MSE为0.0187的动作块。其次，代码激活分析揭示了跨任务的结构化代码重用。第三，\textsc{RoboCode}在所有评估任务中实现了最高或并列的闭环成功率，平均成功率为75.6%，而ACT为68.8%，较大扩散政策基线为41.6%。这些结果表明软门控原始代码本为机器人操作提供了有效且可解释的中间动作表示。