生成时间: 2026-09-11 20:40:41 (UTC+8); Arxiv 发布时间: 2026-09-11 20:00 EDT (2026-09-12 08:00 UTC+8)

今天共有 24 篇相关文章

Keyword: reinforcement learning

An Open Recipe for IMO Gold: Training Nemotron for Olympiad Mathematics

IMO金牌的开放配方：奥林匹克数学训练Nemotron

• Authors: Ivan Moshkov, Stephen Ge, George Armstrong, Wei Du, Sadegh Mahdavi, Igor Gitman
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.10712
• Pdf link: https://arxiv.org/pdf/2609.10712
• Abstract We study how model post-training and test-time inference design affect natural-language proof generation for hard olympiad mathematics. Starting from Nemotron 3 Ultra, we train two specialist checkpoints using supervised fine-tuning and reinforcement learning, and evaluate checkpoint choice, verification, and refinement. Based on these findings, we present an open-model test-time-compute pipeline. The system operates entirely in natural language, with no formal prover, external tools, or internet access. Three Nemotron 3 Ultra checkpoints - the general-availability model and two post-trained specialists - power an iterative search that generates, verifies, and refines candidate proofs; a separate high-compute stage then selects each final submission. The system scored 30 out of 42 points at IMO 2026, reaching the gold-medal threshold. We release the two post-trained checkpoints as well as the training data, the training and inference code, the submitted solutions, and Nemotron-IMO-Bench, a new benchmark of 200 novel olympiad-level problems.
• 中文摘要 我们研究模型后训练和测试时间推断设计如何影响硬奥林匹克数学的自然语言证明生成。从Nemotron 3 Ultra开始，我们通过监督微调和强化学习训练两个专业检查点，并评估检查点的选择、验证和完善。基于这些发现，我们提出了一个开放模型测试-时间-计算流程。该系统完全以自然语言运行，没有正式的校验器、外部工具或互联网访问。Nemotron 3 Ultra的三个检查点——通用可用模型和两个后训练专家——驱动迭代搜索，生成、验证并完善候选证明;随后有一个独立的高计算阶段选择每个最终提交。该系统在IMO 2026中获得42分中的30分，达到金牌门槛。我们发布了两个训练后检查点，以及训练数据、训练和推理代码、提交的解答，以及Nemotron-IMO-Bench，这是一项包含200个新奥林匹克级问题的新基准测试。

A Bellman Optimality Equation for Plasticity

可塑性的贝尔曼最优方程

• Authors: Jeremy Lucas, Doina Precup
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.10776
• Pdf link: https://arxiv.org/pdf/2609.10776
• Abstract In continual reinforcement learning, carefully managing the stability-plasticity tradeoff remains a core challenge. Recent work by Abel et al. (2025) formalized this dilemma by defining plasticity as the generalized directed information from an agent's observations to its actions, and empowerment as the generalized directed information from its actions to its observations. This formulation successfully reframes the traditional stability-plasticity tradeoff as an empowerment-plasticity tradeoff. However, while extensive literature exists on optimizing for empowerment, there is currently no research addressing the optimization of plasticity under this new definition. This paper presents preliminary work toward optimizing plasticity within Markov decision processes. We show that there exists a Bellman optimality equation for optimizing plasticity similar to previous work for empowerment.
• 中文摘要 在持续强化学习中，谨慎管理稳定性与可塑性权衡仍是核心挑战。Abel等人（2025）的最新研究通过将可塑性定义为从代理观察到其行为的广义定向信息，以及赋权定义为从其行为到观察的广义定向信息，形式化了这一困境。该表述成功地将传统的稳定性-可塑性权衡重新定义为赋能-可塑性权衡。然而，尽管已有大量文献讨论赋能优化，目前尚无研究针对该新定义下的可塑性优化。本文介绍了在马尔可夫决策过程中优化可塑性的初步工作。我们展示了存在一个贝尔曼最优方程用于优化可塑性，类似于以往赋能领域的研究。

From Connectivity to Rewards: Dense Reward Learning with Directed State Graphs

从连接到奖励：带定向状态图的密集奖励学习

• Authors: Shuyuan Zhang, Zihan Wang, Xiao-Wen Chang, Doina Precup
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.10781
• Pdf link: https://arxiv.org/pdf/2609.10781
• Abstract The integration of graphs with Goal-Conditioned Hierarchical Reinforcement Learning (GCHRL) has received increasing attention, as graphs naturally encode task hierarchies for effective subgoal sampling. However, existing methods often overlook intrinsic connectivity information, failing to fully leverage the underlying topology for efficient learning. Most graph-based GCHRL methods use the graph as a stochastic sampling tool rather than as an environmental model that encodes connectivity and state-accessibility information. This limitation is particularly acute in quasimetric environments, where the inherent asymmetry of state transitions poses a fundamental challenge to stable policy learning and robust path planning. In this paper, we address these problems by introducing a state connectivity model designed to predict pairwise state connectivity strength in asymmetric environments. We transform these connectivity strengths into scalar auxiliary dense rewards, providing continuous guidance across multiple hierarchical levels. We demonstrate that our proposed framework, Graph-Guided Quasimetric Dense Reward (G2QDR), can theoretically be integrated into any existing GCHRL architecture, and the state connectivity model is efficiently implemented via a neural network trained on a directed state graph generated during exploration. Empirical results across a wide range of sparse reward environments indicate that, in general, G2QDR can enhance the performance of baseline GCHRL approaches with acceptable computational overhead.
• 中文摘要 图与目标条件层级强化学习（GCHRL）的集成日益受到关注，因为图自然编码任务层级以实现有效的子目标采样。然而，现有方法常常忽视内在连接性信息，未能充分利用底层拓扑以实现高效学习。大多数基于图的GCHRL方法将图作为随机抽样工具，而非编码连接性和状态可达性信息的环境模型。这一局限在准计量环境中尤为突出，状态转变的固有不对称性对稳定策略学习和稳健路径规划构成根本挑战。本文通过引入一个状态连通性模型来解决这些问题，旨在预测非对称环境中的两两状态连通强度。我们将这些连通性强度转化为标量辅助密集奖励，提供跨多层级的连续指导。我们展示了我们提出的框架——图引导准指标密集奖励（G2QDR）理论上可以集成到任何现有的GCHRL架构中，状态连通性模型通过在探索过程中生成的有向状态图上训练的神经网络高效实现。在广泛稀疏奖励环境中的实证结果表明，总体而言，G2QDR可以以合理的计算开销提升基线GCHRL方法的性能。

Expressive Robotic Pianist: Mastering Complex Piano Repertoire with Graph-Mimic and Musical Dynamics

表现力机器人钢琴家：利用图模拟与音乐动态掌握复杂钢琴曲目

• Authors: Yanhong Liang, Xianwei Liu, Chaojie Fu, Shaowen Cheng, Yanyan Yuan, Chengwei Zhuo, Xi Chen, Yongbin Jin, Wei Yang, Hongtao Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.10844
• Pdf link: https://arxiv.org/pdf/2609.10844
• Abstract Enabling robots to perform musical instruments with human-level expressivity represents a frontier in bridging the gap between mechanical precision and artistic interpretation. Despite advances in robotic dexterity, replicating the fluid finger transitions and nuanced dynamic control characteristic of human pianists remains a significant challenge. Through a reinforcement learning-based control framework, we demonstrate that a dexterous robotic hand can achieve high-fidelity performance across a diverse piano repertoire. Central to our approach is a graph-based optimization strategy that guides the robot to generate natural pre-press and key-press fingering strategies that closely resemble human movement patterns. To achieve expressive sound production, the control system is coupled with a physics-inspired acoustic model that modulates keypress velocity to accurately reproduce the dynamic variations specified in musical scores. Quantitative evaluations demonstrate that our expressive control model significantly outperforms baseline methods in both finger morphology similarity and dynamic velocity accuracy. In a perceptual test involving participants from diverse listener groups, performances generated by our system are significantly preferred over baseline robotic performances and are indistinguishable from human performances for non-professional audiences. Furthermore, extensive experiments across multiple musical styles confirm that our method maintains high note-level accuracy while achieving expressive performance. Our approach provides a robust pathway for robotic systems to move beyond mere mechanical accuracy, elevating robotic musicianship to a level of expressive performance comparable to human pianists.
• 中文摘要 使机器人能够以人类水平的表现力演奏乐器，是弥合机械精度与艺术诠释之间鸿沟的前沿。尽管机器人灵巧度有所进步，复制人类钢琴家特有的流畅指尖过渡和细腻的动态控制仍是一大挑战。通过基于强化学习的控制框架，我们证明了灵巧的机器人手可以在多样的钢琴曲目中实现高保真表现。我们方法的核心是基于图的优化策略，引导机器人生成自然的按键前按和按键指法策略，这些策略与人类运动模式非常相似。为了实现富有表现力的声音产出，控制系统结合了受物理启发的声学模型，该模型调节按键速度，准确还原乐谱中指定的动态变化。定量评估表明，我们的表现控制模型在指形相似性和动态速度准确性上均显著优于基线方法。在一项涉及多元听众群体参与者的感知测试中，我们系统生成的表演明显优于基线机器人表演，且与非专业观众的人类表演无异。此外，跨多种音乐风格的广泛实验证实，我们的方法在实现表现力的同时保持了高音符级的准确性。我们的方法为机器人系统超越单纯机械精度提供了坚实的路径，将机器人音乐素养提升到可与人类钢琴家相媲美的表现力水平。

Certifying Lower Bounds for Risk-Sensitive Reinforcement Learning under Adversarial State Perturbations

在对抗状态扰动下，风险敏感强化学习的下限认证

• Authors: Tong Li, Saunak Kumar Panda, Yisha Xiang
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.10866
• Pdf link: https://arxiv.org/pdf/2609.10866
• Abstract Reinforcement learning (RL) agents deployed in real-world environments are often vulnerable to adversarial perturbations in state observations, creating risks in safety-critical applications. Certification methods can improve robustness against adversarial perturbations by providing lower bounds on expected cumulative rewards. Existing certification methods, however, mainly focus on risk-neutral objectives. In this paper, we extend certification methods to risk-sensitive objectives by establishing lower bounds on the exponential utility of cumulative rewards under $l_{p}$-norm-bounded state adversarial perturbations ($1\leq p <\infty$). By introducing a $\phi$-divergence relaxation of the perturbation set, we formulate the risk-sensitive certification problem as a convex optimization and derive its dual to obtain a tractable approximation of the certified lower bound. We further propose an empirical method that improves certified lower bounds by selecting the training risk-aversion parameter $\beta$ independently of the risk level used during evaluation. Experiments on both OpenAI Gym environments and a machine replacement problem show that, compared to risk-neutral training, risk-averse training generally yields policies with higher certified lower bounds, particularly under larger perturbation budgets. Moreover, under both risk-neutral and risk-averse evaluation settings, increasing risk aversion during training leads to non-monotonic certification performance, where certified lower bounds initially improve but eventually decrease due to overly conservative policies.
• 中文摘要 在现实环境中部署的强化学习（RL）代理常常容易受到状态观测中的对抗性扰动影响，从而在安全关键应用中带来风险。认证方法可以通过提供预期累计回报的下界来提高对抗性扰动的鲁棒性。然而，现有的认证方法主要关注风险中性目标。本文通过建立在$l_{p}$范数界状态对抗扰动（$1\leq p <\infty$）下累计奖励指数效用的下界，将认证方法扩展到风险敏感目标。通过引入扰动集的$\phi$散度松弛，我们将风险敏感性认证问题表述为凸优化，并推导其对偶以获得认证下界的可解近似。我们还提出了一种实证方法，通过选择训练风险规避参数$\beta$，独立于评估时使用的风险水平来改善认证下限。在OpenAI健身房环境和机器替换问题上的实验表明，与风险中性训练相比，风险规避训练通常产生更高的认证下限政策，尤其是在较大扰动预算下。此外，在风险中性和风险厌恶评估环境下，训练期间风险规避的增加导致非单调认证表现，即认证下限最初有所改善，但由于过于保守的政策最终下降。

Topological Necessities: Mechanism-Invariant Strategic Subgoals for Cross-Embodiment Goal-Conditioned Control

拓扑必然性：跨具身目标条件控制的机制不变战略子目标

• Authors: Hao Shi, Xi Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.11014
• Pdf link: https://arxiv.org/pdf/2609.11014
• Abstract Long-horizon goal-conditioned reinforcement learning delegates control to a high-level module that proposes subgoals, but existing subgoals are implicit byproducts of value functions or latent actions, tied to the executor that produced them. We study a different object: a route-conditioned order of unavoidable stages that every successful executor must traverse, recoverable from offline trajectories and belonging to none of them. Its defining properties are topological: an unskippable stage is a separating set that every admissible path must cross, and a loop in free space forces a route choice. We read the two by homology in dimensions 0 and 1 over a transport-weighted carrier built from successful trajectories, yielding an enumerable gate set with shell-level certificates; the certified gates are what we call topological necessities. Certified gates enter the decision loop as a recursive topological gate hierarchy. Under a fixed, isomorphic free space, the object survives executor replacement: gates frozen on PointMaze data transfer without retraining to Ant and Humanoid, attaining the highest Humanoid aggregate under a unified interface (96.1), with +36.0 over a map-privileged reference on the multi-route task (p=1.4e-5); the planner saturates PointMaze (100+/-0) and matches or exceeds the strongest baselines on AntMaze (giant +22.9) and Kitchen (+15.8/+12.6).
• 中文摘要 长视野目标条件强化学习将控制权委托给提出子目标的高阶模块，但现有子目标是价值函数或潜在动作的隐含副产品，与产生它们的执行者相关。我们研究另一个对象：一个路径条件的不可避免阶段顺序，每个成功执行者必须穿越这些阶段，这些阶段可从离线轨迹中恢复且不属于任何阶段。其定义性质是拓扑性的：不可跳过阶段是每条可接受路径必须穿越的分离集，自由空间中的环路迫使路径选择。我们在一个由成功轨迹构建的传输加权载波上，通过维度0和1的同调解读两者，得到一个带有壳级证书的可枚举门集;认证门是我们所称的拓扑必然性。认证门作为递归拓扑门层级进入决策循环。在固定、同构自由空间下，该对象能经执行器替换存活：PointMaze数据传输时门被冻结，无需重新训练为蚂蚁和人形，在统一接口下达到最高的类人生物聚合（96.1），在多路由任务中地图特权参考值为+36.0（p=1.4e-5）;规划器饱和PointMaze（100+/-0），并在蚁迷宫（巨型+22.9）和厨房（+15.8/+12.6）上达到或超过最强基线。

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

T1：面向长期任务的终端代理强化学习

• Authors: Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, Leowei Liang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.11042
• Pdf link: https://arxiv.org/pdf/2609.11042
• Abstract Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.
• 中文摘要 代理的使用正转向长期任务，如编码和科学发现，其中终端任务尤为重要。我们引入了T1，这是一种专家混合模型，包含122亿总训练，通过强化学习训练，在云沙盒中运行真实壳体，每个任务最多可调用300+次工具调用，奖励每个任务的验证器。我们提供了一个全面的方案：首先，积极热启动以稳定演员-批评者训练，密集过程通过通过验证者数量来奖励评分轨迹。其次，通过TITO构建实现稳定优化，训练基于精确抽样令牌标识符，并在回合边界进行漂移修复，并进行部署路由重放，记录采样器在每个MoE层的专家选择，并在训练中重放。第三，完全脱离分布的训练语料库：隔离种子和与终端-工作台2.1分离的综合任务，确保收益反映真正的能力转移而非基准过拟合。TITO和R3将训练与推断对数概率差异从0.021降至0.013，损失区间的零令牌漂移完全对齐。在终端-工作台2.1中，我们的后列车流水线将初始基础模型从43.8%提升至T1，解决率为64.0%。在长视野终端工作台上，T1达到27.9%，超过GPT-5.4和GLM-5.1。

Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning

模型改变主意的地方分支：树状结构强化学习中的信念转移分支

• Authors: Bin Lei, Yu Li, Prafulla Kumar Choubey, Jiaxin Zhang, Becky Xiangyu Peng, Qinyuan Ye, Kartik Narayan, Caiwen Ding, Silvio Savarese, Chien-Sheng Wu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.11061
• Pdf link: https://arxiv.org/pdf/2609.11061
• Abstract Tree-structured rollouts give critic-free reinforcement learning with verifiable rewards (RLVR) step-level credit: fork a chain at an intermediate point, and sibling outcome differences estimate step value. Each fork adds sampling cost, so realistic budgets typically allow only a few forks per chain. A fork placed where the outcome is already largely settled yields siblings that mostly agree and provide almost no credit signal; hence, for a given tree size, where forks are placed largely determines how much step-level RL can gain. Most existing mainstream methods place forks by structure, such as fixed lengths, midpoints, and delimiters, or by next-token entropy. We formalize fork placement as locating the \emph{pivots} of the chain's value curve, where the expected outcome turns. We propose \emph{belief-shift branching}: read the model's answer belief at candidate boundaries and fork just before the step where consecutive beliefs diverge most. Three instantiations, none needing step-level supervision, span access levels: a black-box probe, a logit-lens depth profile, and a learned activation direction, which is fit offline and therefore used only in the validation before RL training. The signal only \emph{places} forks, and the probe costs about $1\%$ of step compute on mathematics and under $5\%$ on code when it runs inside the rollout engine. In that validation, against Monte-Carlo value curves, a belief-shift signal ranks first in each of the eight model$\times$benchmark panels, ahead of entropy, structural, and LLM-judge baselines. In RL across three model families and two domains, belief-shift forking leads every mathematics aggregate, on OLMo-3-7B by $+2.6$ aggregate and $+2.9$ on AIME 2026 over the strongest baseline, and sweeps every OLMo code column, by $+6.5$ on LiveCodeBench-medium.
• 中文摘要 树状结构的推广提供了无批评的强化学习，带有可验证奖励（RLVR）的步骤级信用：在中间点分叉链，兄弟姐妹结果差异估计步数值。每次分叉都会增加采样成本，因此现实预算通常允许每条链只有少量分叉。在结果基本确定的分叉中，会产生大多达成一致且几乎不提供信用信号的兄弟姐妹;因此，在给定树大小下，分叉的位置在很大程度上决定了步级强化学习能获得多少。大多数现有主流方法按结构（如固定长度、中点和分隔符）或下一令牌熵来放置分叉。我们将分叉定位形式化为定位链价值曲线的 \emph{pivots}，即预期结果的转折点。我们提出\emph{信念转移分支}：在候选边界读取模型的答案信念，并在连续信念分歧最大阶段前进行分叉。三个实例实例，均不需步级监督，跨越访问层级：黑箱探针、logit-lens深度剖面和学习激活方向，后者离线拟合，仅用于强化学习训练前的验证。信号仅分叉，探针在数学计算中计算成本约为1美元，代码成本低于5%美元，均在rollout引擎内运行。在该验证中，针对蒙特卡洛价值曲线，信念转移信号在八个模型$\times$基准面板中均排名第一，领先熵、结构和LLM评判基线。在强基线上，跨三个模型族和两个域的强化学习中，信念转移分叉在所有数学聚合上领先，在OLMo-3-7B上以+2.6美元和在AIME 2026上领先+2.9美元，在LiveCodeBench-medium上则以+6.5美元的速度扫过所有OLMo代码列。

Learning Realistic Athletic Sprinting Without Demonstrations

学习无示范的真实运动短跑

• Authors: William Wang, Nicholas Bianco, Guy Tevet, Jennifer Hicks, C. Karen Liu, Scott Delp, Kayvon Fatahalian
• Subjects: Subjects: Graphics (cs.GR)
• Arxiv link: https://arxiv.org/abs/2609.11083
• Pdf link: https://arxiv.org/pdf/2609.11083
• Abstract We present a muscle-driven simulation system for generating biomechanically accurate motion for high-speed athletic locomotion tasks that does not require motion demonstrations. Our approach integrates state-of-the-art biomechanical athlete models into a new, high-performance GPU simulator capable of running at 1000x real-time. High-throughput simulation enables large-batch reinforcement learning to train control policies that operate directly in the model's high-dimensional muscle excitation space, and are guided only by task-specific episode termination conditions and a reward that encourages maximizing speed while reducing forces needed to respect joint limits. These policies train within a few hours on a single GPU and generate "near visually realistic" motions for complete athletic activities such as a full 100-meter sprint or performing popular athletic locomotion drills like side-shuffling, backpedaling, and carioca. The generated sprinting motions also exhibit strong agreement with experimental data captured from sprinters.
• 中文摘要 我们提出了一套肌肉驱动的仿真系统，用于生成生物力学上精确的高速运动运动任务，无需运动演示。我们的方法将最先进的生物力学运动员模型整合到一个高性能GPU模拟器中，能够以1000倍实时运行。高通量仿真支持大规模批次强化学习，训练控制策略，这些策略直接作用于模型的高维肌肉兴奋空间，仅受任务特定的发作终止条件和鼓励最大化速度同时减少关节极限所需力的奖励来指导。这些策略在单个GPU上训练数小时内，能够生成“近乎视觉真实”的运动，适用于完整的运动活动，如完整的100米短跑或进行流行的运动训练，如侧移、倒退和无力跑。生成的冲刺动作也与短跑运动员采集的实验数据高度吻合。

Quantifying the Reality Gap for RL-Based UAV Placement at mmWave and Sub-THz

量化基于强化学习的无人机在毫米波和亚太赫兹波段的现实差距

• Authors: Abdullateef Almohamad, Mostafa Ibrahim, Sabit Ekin, Khalid Qaraqe
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.11121
• Pdf link: https://arxiv.org/pdf/2609.11121
• Abstract Reinforcement learning (RL) policies for unmanned aerial vehicle (UAV) placement in mmWave and sub-terahertz networks are typically trained on simplified analytical channels. We quantify the resulting sim-to-real gap on a real urban map of Doha, Qatar, at carriers {28, 140, 183, 300} GHz and altitudes {50, 75, 100, 125} m, evaluating three channel pipelines: an analytical model (FSPL + atmospheric absorption + cuboid LoS), full Monte-Carlo ray tracing in Sionna RT with ITU-R P.676-13 absorption, and a deterministic-LoS hybrid that reuses Sionna's mesh under a closed-form path-gain expression. We formalize the gap on the spatial SNR distribution via four metrics, namely bias, RMSE, Jensen-Shannon divergence, and optimum-deployment displacement. Three findings emerge: at 28/140 GHz, $\sim$70% of the apparent -5.6/-4.8 dB Sionna bias is Monte-Carlo undersampling and shrinks to -1.7/-1.5 dB after mitigation; at 183 GHz a -9.2 dB residual isolates the atmospheric absorption / ITU-R P.676 line-shape disagreement; at 300 GHz the stochastic ray tracer agrees with the analytical model only coincidentally, with a +3.8 dB structural offset exposed by the deterministic-LoS pipeline. Across all carriers the linear-domain regret of the analytical-trained policy stays $\geq$ 0.93, indicating practical near-optimality but with a carrier-resolved SNR bias that warrants explicit reporting.
• 中文摘要 无人机（UAV）在毫米波和亚太赫兹网络中的强化学习（RL）策略通常在简化的分析信道上训练。我们在卡塔尔多哈的真实城市地图上量化了模拟到现实之间的差距，载波为{28， 140， 183， 300 GHz，海拔为50， 75， 100， 125米，评估了三个通道流水线：分析模型（FSPL + 大气吸收 + 立方体视线）、采用 ITU-R P.676-13 吸收的 Sionna RT 全蒙特卡洛射线追踪，以及一个确定性视线混合体，该混合体在闭合形式路径增益表达式下重用 Sionna 网格。我们通过四个指标形式化空间信噪比分布的差距，即偏置、RMSE、Jensen-Shannon 散度和最优部署位移。得出三个发现：在28/140 GHz，约70%的表观-5.6/-4.8 dB Sionna偏置为蒙特卡洛欠采样，缓解后缩小至-1.7/-1.5 dB;在183 GHz，-9.2 dB残差隔离出大气吸收/ITU-R P.676线形不一致;在300 GHz，随机射线示踪器与解析模型相符仅巧合，确定性视线流水线暴露出+3.8 dB的结构偏移。在所有载波中，分析训练策略的线性域遗憾值保持为$\geq$ 0.93，表明实际接近最优，但存在载波解析的SNR偏差，值得明确报告。

KuaiRP Series Role-playing Models Technical Report

KuaiRP系列角色扮演模型技术报告

• Authors: Yipeng Wang, Ziwei Zhang, Jiahui Zhang, Qi Gan, Kai Sheng
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.11127
• Pdf link: https://arxiv.org/pdf/2609.11127
• Abstract This paper introduces the complete technical solution for the KuaiRP series of role-playing models. We aim to achieve four core objectives for a dedicated role-playing model: simplified prompt engineering, highly stable output quality, built-in domain world knowledge, and high-efficiency deployment with a small parameter size. However, effectively injecting deep domain knowledge often leads to a severe catastrophic forgetting of the model's general agent capabilities. To overcome this trade-off, we propose a multi-stage training pipeline. First, we design a standardized character template and construct an SFT data pipeline based on user behavior simulation and reverse profile filtering. Next, we utilize a rule-based composite reward function during the Reinforcement Learning (RL) phase to eliminate common degradation phenomena like length expansion and repetitive generation. Finally, to recover the general capabilities compromised during SFT and RL, we propose a novel self-distillation paradigm using Two-stage On-Policy Distillation (OPD) equipped with Cumulative-Divergence Decay (CDD). By using the domain-adapted model as the teacher and the original base model as the student, we effectively balance deep domain knowledge injection with the preservation of general agent capabilities. Experimental results demonstrate that the KuaiRP models not only match the current state-of-the-art proprietary models in role-playing fidelity within our target domains, but also successfully recover general agent capabilities, maintaining extremely low deployment costs.
• 中文摘要 本文介绍了KuaiRP系列角色扮演模型的完整技术解决方案。我们旨在实现专用角色扮演模型的四个核心目标：简化提示工程、高度稳定的输出质量、内置领域世界知识，以及以小参数规模高效部署。然而，有效注入深度领域知识往往会导致模型通用代理能力的严重灾难性遗忘。为克服这一权衡，我们提出了一个多阶段训练流水线。首先，我们设计标准化字符模板，构建基于用户行为模拟和反向配置文件过滤的SFT数据流水线。接着，我们在强化学习（RL）阶段使用基于规则的复合奖励函数，消除长度扩展和重复生成等常见退化现象。最后，为了恢复SFT和RL期间受损的一般能力，我们提出了一种采用两阶段策略蒸馏（OPD）和累积-发散衰减（CDD）的新型自蒸馏范式。通过以领域适配模型为教师，以原始基础模型为学生，我们有效地平衡了深度领域知识注入与通用代理能力的保留。实验结果表明，KuaiRP模型不仅在目标域的角色扮演忠实度上与当前最先进的专有模型匹配，还成功恢复了通用代理能力，保持极低的部署成本。

DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat

DRG-MAPPO：分层动态角色图多智能体强化学习，用于合作空战

• Authors: Junlin Liu, Chengwei Li, Yang Gao, Hui Chang, Xinchen Zhang, Zhijun Zhao, Hao Zhao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.11155
• Pdf link: https://arxiv.org/pdf/2609.11155
• Abstract Multi-Agent Reinforcement Learning (MARL) has emerged as a pivotal paradigm for complex decision-making in autonomous systems and air combat. While MARL has demonstrated significant potential in air combat, achieving sophisticated tactical coordination remains a non-trivial challenge. This difficulty is largely attributed to two primary limitations: (1) the absence of structured relational modeling hinders agents from capturing complex, time-varying interactions among battlefield entities; and (2) conventional flat architectures often lack the capability to explicitly model tactical roles, leading to ambiguous task allocation in highly dynamic environments. To address these challenges, we propose Hierarchical Dynamic Role-Graph Multi-Agent Proximal Policy Optimization (DRG-MAPPO), a novel MARL framework that integrates graph-based relational modeling with dynamic role assignment. Specifically, DRG-MAPPO constructs a graph-based representation of battlefield interactions and leverages graph attention mechanisms to extract critical relational features among allies, enemies, and threats. Subsequently, a high-level policy employs a dynamic role assignment mechanism to determine tactical responsibilities (e.g., leader'' andsupporter''). Conditioned on these roles and encoded graph-relational features, a low-level policy executes discrete maneuver actions, facilitating the joint optimization of tactical strategy and collaborative execution. Furthermore, a target-priority auxiliary task is designed to foster the emergence of behaviors such as focus-fire. Experimental results demonstrate that DRG-MAPPO achieves a state-of-the-art win rate of 87%, suggesting that our framework effectively balances relational modeling, interpretability, and optimization stability for cooperative air combat.
• 中文摘要 多智能体强化学习（MARL）已成为自主系统和空战复杂决策的关键范式。尽管MARL在空战中展现出显著潜力，但实现复杂的战术协调仍是一项非小挑战。这一困难主要归因于两个主要限制：（1）缺乏结构化关系建模，阻碍了智能体捕捉战场实体间复杂且随时间变化的交互;（2）传统的平面架构通常缺乏显式建模战术角色的能力，导致高度动态环境中任务分配模糊不清。为应对这些挑战，我们提出了分层动态角色图多代理近端策略优化（DRG-MAPPO），这是一个将基于图的关系建模与动态角色分配结合的新型MARL框架。具体而言，DRG-MAPPO构建了基于图的战场互动表示，并利用图关注机制提取盟友、敌人和威胁之间的关键关系特征。随后，高级策略采用动态角色分配机制来确定战术责任（如“领导者”和“支持者”）。基于这些角色和编码的图关系特征，低级策略执行离散机动动作，促进战术战略和协作执行的联合优化。此外，设计了一个目标优先级辅助任务，以促进诸如聚焦射击等行为的出现。实验结果表明，DRG-MAPPO实现了87%的先进胜率，表明我们的框架在合作空战中有效平衡了关系建模、可解释性和优化稳定性。

From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models

从评估到提升：视频生成模型的基准测试与视频思维推理的改进

• Authors: Meng Luo, Yicheng Liu, Jiahao Wang, Yuanxing Zhang, Xin Tao, Pengfei Wan, Kun Gai, Hao Fei
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.11242
• Pdf link: https://arxiv.org/pdf/2609.11242
• Abstract Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--executing symbolic rules, respecting physical laws, and pursuing intentional goals--remains an open question. Existing benchmarks only partially address this, often conflating visual quality with cognitive correctness. We introduce VWG-Bench (Video World Generalist Benchmark), a comprehensive benchmark spanning 9 reasoning dimensions and 38 fine-grained tasks. To enable precise diagnosis, we design a three-level VLM-as-Judge protocol that independently assesses video-level fluency, task-level rule adherence, and sample-level goal realization. Evaluations of leading models reveal a striking gap: while models achieve strong rendering scores, they consistently fail on logic-heavy and rule-constrained tasks. To address this, we propose Vid-PRE (Video Prompt Reasoner and Enhancer), a model-agnostic prompt rewriter that offloads the cognitive burden of reasoning to a dedicated VLM. Trained via reinforcement learning with purely text-based rewards, Vid-PRE produces concise, constraint-aware prompts without the instability of video-level reward signals. Experiments show that Vid-PRE yields substantial reasoning improvements across multiple generators without architectural modifications. Together, VWG-Bench and Vid-PRE offer a rigorous diagnostic lens and a scalable path toward true think-with-video capabilities. All data and code are publicly available at this https URL.
• 中文摘要 视频生成技术已进步，能够产生视觉上引人注目且时间连贯的结果。然而，这些模型是否能真正用视频思考——执行符号规则、尊重物理规律并追求有意目标——仍是一个悬而未决的问题。现有基准测试仅部分解决了这个问题，常常将视觉质量与认知正确性混为一谈。我们介绍了VWG-Bench（视频世界通用基准），这是一个涵盖9个推理维度和38个细粒度任务的综合基准。为实现精确诊断，我们设计了一个三级VLM即评判协议，独立评估视频级的流畅度、任务级规则的遵守情况和样本级目标实现。对领先模型的评估显示存在显著差距：虽然模型获得了强劲的渲染得分，但在逻辑繁重且规则约束的任务中却持续失败。为此，我们提出了Vid-PRE（视频提示推理器与增强器），这是一种模型无关的提示重写器，将推理的认知负担卸给专用的VLM。通过强化学习训练，纯文本奖励，Vid-PRE生成简洁、约束感知的提示，且不具视频级奖励信号的不稳定性。实验显示，Vid-PRE在多个生成器间实现了显著的推理提升，无需架构修改。VWG-Bench和Vid-PRE共同提供了严谨的诊断视角和通往真正视频思维能力的可扩展路径。所有数据和代码均在此 https 网址公开。

Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy

超越噪声引导：生成式机器人政策中的双潜空间强化学习

• Authors: Pengfei Zhang, Teng Sun, Xianchao Xiu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.11270
• Pdf link: https://arxiv.org/pdf/2609.11270
• Abstract Pretrained generative robot policies learn expressive action priors from demonstrations. However, existing reinforcement learning methods only steer the noisy space but fail to modulate intermediate action representations during the generation process, resulting in performance degradation and inefficiency. To address this limitation, we propose a novel Dual-Latent Space Reinforcement Learning (DLSRL) framework, which complements initial-noise steering with representation-level control inside the frozen generator. Specifically, our actor network predicts two distinct latent variables: an initial-noise latent variable that steers behavior generation, and an action-representation latent variable for intermediate feature modulation. Moreover, this representation latent variable is mapped to adapter features and ingeniously injected into the hidden states of intermediate action tokens via residual connections. Our dual-control design enables direct adjustment of action representations without updating the base policy. Experiments across generative policy architectures and robotic manipulation tasks show that DLSRL effectively accelerates online robot policy adaptation and achieves competitive performance. Our code is available at \href{this https URL}{this https URL}.
• 中文摘要 预训练生成机器人策略通过演示学习表达动作先验。然而，现有强化学习方法仅引导噪声空间，未能在生成过程中调制中间动作表示，导致性能下降和效率低下。为解决这一限制，我们提出了一种新颖的双潜在空间强化学习（DLSRL）框架，在冻结生成器内用表示级控制补充初始噪声引导。具体来说，我们的演员网络预测了两个不同的潜在变量：一个初始噪声潜在变量用于引导行为生成，以及一个动作表征潜在变量用于中间特征调制。此外，该表征潜在变量被映射到适配器特征，并通过残差连接巧妙注入中间动作标记的隐藏状态。我们的双重控制设计允许在不更新基础策略的情况下直接调整动作表示。跨生成策略架构和机器人操作任务的实验表明，DLSRL有效加速了在线机器人策略的适应，并实现了竞争性能。我们的代码可在 \href{this https URL}{this https URL} 获取。

SEAR: Segment-Evidence-Aware Routing for Weak-to-Strong Multilingual Speech MCQ

SEAR：弱到强多语言语音选择题的片段证据感知路由

• Authors: Huy Hoang Le, Long-Bao Nguyen, Minh Tri Dao
• Subjects: Subjects: Computation and Language (cs.CL); Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2609.11355
• Pdf link: https://arxiv.org/pdf/2609.11355
• Abstract This paper describes our system for Task~2 of the second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge. We adapt Qwen3-Omni-30B-A3B-Instruct with a segment-evidence-aware data and post-training pipeline. A language model converts timestamped ASR into coherent event spans, which are expanded by a boundary margin and cropped from the original recording. We then synthesize complementary semantic MCQs with Qwen3.6-27B and acoustic MCQs with Gemini~3.1 Flash-Lite, followed by structural, grounding, answer-consistency, and target-model trainability checks, yielding 359,825 verified MCQs across 21 language and accent variants. A text-only probe partitions the data into weak, text-answerable items used for supervised fine-tuning and strong, audio-dependent items used for reinforcement learning with Group Sequence Policy Optimization (GSPO), stabilized by debiased advantages, sequence-level importance correction, and dynamic filtering. Our system obtains 90.92% accuracy on the final official evaluation set.
• 中文摘要 本文描述了我们针对第二期多语言会话语音语言模型（MLC-SLM）挑战任务~2的系统。我们采用了片段证据感知数据和训练后流程，调整了Qwen3-Omni-30B-A3B-Instruct。语言模型将时间戳的ASR转换为连贯事件跨度，这些跨度通过边界边际扩展并从原始记录裁剪。随后，我们用Qwen3.6-27B合成互补语义选择题，用Gemini~3.1 Flash-Lite合成声学选择题，随后进行结构性、基础化、答案一致性和目标-模型可训练性检查，共计359,825条经过验证的MCQ，涵盖21种语言和口音变体。纯文本探针将数据划分为弱的文本答题项，用于监督式微调，以及强的音频依赖项，用于组序列策略优化（GSPO）强项，通过偏见优势、序列级重要性修正和动态过滤来稳定。我们的系统在最终官方评估集上获得了90.92%的准确率。

Safety-aware Skill Adaptation for Reinforcement Learning in Dynamic Environments

动态环境中强化学习的安全意识技能适应

• Authors: A K M Nadimul Haque, Sheila Sutjipto, Marc G. Carmichael, Teresa Vidal-Calleja
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.11433
• Pdf link: https://arxiv.org/pdf/2609.11433
• Abstract Skill adaptation frameworks based on reinforcement learning often require restrictive assumptions to maintain stability, such as fixed observations or tightly controlled exploration schedules. In cluttered and dynamic environments, however, unrestricted exploration can lead to unsafe behaviour and unstable learning, particularly when task-relevant observations lie near obstacles or involve moving objects. In this work, we present Dist-GPRL, a distance-aware and safety-guided reinforcement learning framework for structured robot skill adaptation. Building upon Gaussian Process (GP)-based skill parameterisation, our framework sequentially adapts overlapping local windows of sparse trajectory via-points rather than modifying the complete skill at every policy step. Raw policy outputs are correlated through the GP covariance structure, producing temporally coherent trajectory updates while reducing the action-space and credit-assignment difficulties associated with global trajectory adaptation. Safety is incorporated through two complementary forms of guidance. A safe-subspace prior derived from the Hausdorff Approximation Planner (HAP) biases policy exploration toward feasible regions, while dynamically updated distance field clearance and gradient rewards provide local obstacle awareness. A trajectory-kinematics similarity regulariser further preserves the demonstrated velocity and acceleration characteristics during adaptation. We evaluate the framework on two dynamic object-manipulation tasks in simulation and transfer the learned policy to real-world robot execution. Experimental results demonstrate higher task success, lower collision frequency, and more stable learning than the baselines, while preserving the kinematic characteristics of the demonstrated skill.
• 中文摘要 基于强化学习的技能适应框架通常需要限制性假设以维持稳定性，如固定观察或严格控制的探索计划。然而，在杂乱且动态的环境中，无限制探索可能导致不安全行为和不稳定学习，尤其是在任务相关观察靠近障碍物或涉及移动物体时。本研究提出了Dist-GPRL，一种基于距离感知和安全引导的强化学习框架，用于结构化机器人技能适应。基于高斯过程（GP）技能参数化，我们的框架顺序适应稀疏路径点的重叠局部窗口，而非在每个策略步骤中修改全部技能。原始策略输出通过GP协方差结构进行相关，产生时间上连贯的轨迹更新，同时减少与全局轨迹适应相关的动作空间和信用分配困难。安全性通过两种互补的指导形式实现。由豪斯多夫近似规划器（HAP）导出的安全子空间先验使策略探索偏向可行区域，动态更新的距离场间隙和梯度奖励则提供局部障碍感知。轨迹-动力学相似性正则化器进一步保留了适应过程中已展示的速度和加速度特性。我们在模拟中评估两个动态对象操作任务的框架，并将所学策略转化为现实机器人执行。实验结果显示，在保持所展示技能运动学特性的同时，任务成功率更高、碰撞频率更低、学习更稳定。

Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

利用内在的主体感知注意力实现可控多主体视频生成

• Authors: Niange Yu, Ye Tian, Biaolong Chen, Miao Lu, Aixi Zhang, Hao Jiang, Yunhai Tong, Pipei Huang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.11507
• Pdf link: https://arxiv.org/pdf/2609.11507
• Abstract Multi-subject video generation faces two key challenges: uncontrollable fidelity strength and potential semantic drift. We address these by analyzing the internal mechanisms of Diffusion Transformers (DiTs). We found that certain attention blocks naturally form an Intrinsic Spatial Grounding Map (ISGM) that precisely locates reference subjects. Building on this insight, we propose Dual-phase Intrinsic Attention Leveraging (DIAL), a framework that uses these internal signals for both training and inference. In low-noise stages, we use ISGM to guide the attention mechanism, allowing precise control over fidelity strength during inference without retraining. In high-noise stages, we use these same maps to automatically build preference pairs at no additional cost for Reinforcement Learning (RL). This RL procedure effectively anchors the model's attention to reference subjects and mitigates semantic drift. Extensive experiments show that DIAL significantly outperforms baseline models on the OpenS2V-Eval benchmark, consistently improving identity consistency and enabling controllable fidelity strength.
• 中文摘要 多主题视频生成面临两个关键挑战：不可控的保真度强度和潜在的语义漂移。我们通过分析扩散变换器（DiTs）的内部机制来应对这些问题。我们发现某些注意力块自然形成一个内在空间基础图（ISGM），能够精确定位参考主体。基于这一见解，我们提出了双阶段内在注意力杠杆（DIAL）框架，该框架利用这些内部信号进行训练和推理。在低噪声阶段，我们使用ISGM引导注意力机制，实现推理过程中对保真度强度的精确控制而无需重新训练。在高噪声阶段，我们使用相同的映射自动构建偏好对，且无需额外为强化学习（RL）提供额外成本。该强化学习过程有效锚定模型对参考对象的注意力，并减少语义漂移。大量实验表明，DIAL在OpenS2V-Eval基准测试中显著优于基线模型，持续提升身份一致性，实现可控的保真度强度。

Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language

训练后零射击TTS通过自然语言实现细致的情绪和持续时间控制

• Authors: Lianru Gao, Yujie Guo, Yong Qin
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2609.11523
• Pdf link: https://arxiv.org/pdf/2609.11523
• Abstract Audiobook narration, conversational agents, and audiovisual dubbing require speech that conveys changing emotions and adapts its pacing within a single utterance. But most existing TTS systems typically rely on utterance-level style conditioning, making such fine-grained control difficult to achieve. In light of this, and inspired by the success of post-training in large language models, we propose a unified post-training framework that equips pretrained text-to-speech models with natural-language control over segment-level emotion and duration. Supervised fine-tuning establishes instruction-conditioned speech generation, while reinforcement learning with group relative policy optimization refines control accuracy using emotion and duration rewards alongside content and speaker preservation objectives. By reusing the pretrained architecture, our approach avoids additional inference-time control modules. Experiments demonstrate significantly improved fine-grained controllability while maintaining speech intelligibility and speaker identity, highlighting post-training as a practical approach to extending existing speech synthesis models.
• 中文摘要 有声书旁白、对话代理和视听配音需要语音能够传达变化的情感，并在单一话语中调整节奏。但大多数现有TTS系统通常依赖话语级风格条件反射，使得实现如此细致的控制变得困难。基于此，并受大型语言模型后训练成功启发，我们提出了一个统一的后训练框架，为预训练的文本转语音模型赋予对片段级情感和时长的自然语言控制能力。监督微调建立了指令条件化语音生成，而强化学习结合群体相对策略优化则通过情感和持续时间奖励以及内容和说话者保留目标，优化控制准确性。通过重复使用预训练架构，我们的方法避免了额外的推理时间控制模块。实验显示，在保持语音清晰度和说话者身份的同时，细粒度可控性显著提升，强调后训练作为扩展现有语音合成模型的实用方法。

Distributed Optimization of Modular Production Systems using Model-based Reinforcement Learning with Inverse Models

利用基于模型的强化学习与逆模型进行模块化生产系统的分布式优化

• Authors: Andreas Schwung, Steve Yuwono, Sofiene Lassoued, Dorothea Schwung
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.11615
• Pdf link: https://arxiv.org/pdf/2609.11615
• Abstract This paper presents a novel approach for data-driven self-learning control of highly flexible, modular manufacturing systems. Specifically, we employ a novel framework for model-based reinforcement learning which introduces approximate inverse process models within the training of reinforcement policies. This approach disentangles the learning of actuation dynamics and the dynamics in state space, resulting in RL-based training solely within the task space. We propose a lightweight feedforward architecture for approximate inverse models and integrate them within the policy network of standard RL algorithms. We apply the approach to a laboratory modular production testbed with heterogeneous production modules. The results underline the efficiency improvements for modular manufacturing units in terms of both performance and training speed, particularly for off-policy algorithms.
• 中文摘要 本文提出了一种基于数据驱动的自学习控制方法，适用于高度灵活的模块化制造系统。具体来说，我们采用了一种基于模型的强化学习新框架，在强化策略训练中引入了近似逆过程模型。该方法将执行动力学的学习与状态空间的动态学分离，实现了仅在任务空间内实现基于强化学习的训练。我们提出了一种轻量级前馈架构，用于近似逆模型，并将其集成到标准强化学习算法的策略网络中。我们将该方法应用于带有异构生产模块的实验室模块化生产测试平台。结果强调了模块化制造单元在性能和训练速度方面的效率提升，尤其是在非策略算法方面。

Negative Self-Distillation: Learning to Reason by Avoiding Flaws

负自我提炼：通过避免缺陷来学习推理

• Authors: Rongcan Pei, Zhepei Wei, Shuyao Xu, Xinyu Zhu, Wei-Lin Chen, Yu Meng
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.11699
• Pdf link: https://arxiv.org/pdf/2609.11699
• Abstract On-Policy Self-Distillation (OPSD) has emerged as a popular paradigm for large language model (LLM) self-improvement, allowing models to act as their own teachers by leveraging privileged information such as ground-truth solutions. However, recent findings indicate that OPSD can severely degrade the performance of LLMs on complex reasoning tasks: By forcing the student to imitate an artificially confident reasoning trace conditioned on privileged information, OPSD inadvertently suppresses expressions of uncertainty and penalizes the exploratory, self-corrective behaviors required to solve challenging problems. To address this, we introduce Negative Self-Distillation (NSD), a new framework that optimizes LLMs by diverging from flawed reasoning rather than imitating privileged solutions. Instead of relying on ground-truth answers or external supervision, NSD uses the model itself to generate a question-specific negative condition (eg, acting as a ``careless reasoner'') and pushes the student's distribution away from this self-generated negative teacher. Naively applying unlearning objectives to achieve this divergence is problematic, as flawed reasoning tokens are confounded with basic linguistic tokens; indiscriminately penalizing both risks catastrophically degrading the model's foundational language capabilities. We resolve this by designing a dynamic gating mechanism that automatically identifies and isolates reasoning-critical tokens, ensuring gradient updates target only behavioral flaws while preserving the model's linguistic priors. Empirically, NSD consistently outperforms OPSD and other label-free, self-bootstrapping reinforcement learning (RL) baselines.
• 中文摘要 策略自蒸馏（OPSD）已成为大型语言模型（LLM）自我提升的流行范式，使模型能够通过利用特权信息（如基层真实解）来自如教师。然而，最新研究显示，OPSD会严重降低LLM在复杂推理任务中的表现：通过强迫学生模仿基于特权信息的人工自信推理轨迹，OPSD无意中抑制了不确定性的表达，并惩罚了解决难题所需的探索性和自我纠正行为。为此，我们引入了负自我蒸馏（NSD），这是一个通过背离有缺陷推理而非模仿特权解来优化LLMs的新框架。NSD不依赖基于实地的答案或外部监督，而是利用模型本身生成针对问题的负面条件（例如，充当“粗心推理者”），并将学生的分布推离该自生成的负面教师。天真地应用去学习目标实现这种分歧是有问题的，因为有缺陷的推理标记会与基本语言标记混淆;无差别地惩罚两者，风险会灾难性地削弱模型的基础语言能力。我们通过设计动态门控机制，自动识别并隔离推理关键标记，确保梯度更新仅针对行为缺陷，同时保持模型的语言先验。实证上，NSD持续优于OPSD及其他无标签、自我引导强化学习（RL）基线。

Reflex-Informed Neuromuscular Reinforcement Learning for Muscle-Driven Locomotion

反射知情的神经肌肉强化学习，用于肌肉驱动的运动

• Authors: Jian Zhou, Xingyu Zhang, Rui Ma, Yu Cao, Shane Xie, Zhi-qiang Zhang
• Subjects: Subjects: Robotics (cs.RO); Graphics (cs.GR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.11733
• Pdf link: https://arxiv.org/pdf/2609.11733
• Abstract Muscle-driven locomotion provides a physically grounded approach to generating realistic human movement. However, achieving both physiological plausibility and adaptability to changes in musculoskeletal capacity and external disturbances remains a fundamental challenge. To address this limitation, we propose a Reflex-Informed Neuromuscular Reinforcement Learning framework for muscle-driven locomotion. Within this framework, a fixed phase-dependent reflex controller serves as the underlying neuromuscular control mechanism, while the reinforcement learning policy produces four biomechanically meaningful residual parameters to modulate key reflex gains and thresholds associated with hip swing, knee support, and ankle propulsion according to the current state. Experimental results demonstrate that the proposed framework generates physiologically plausible locomotion with improved kinematic accuracy and dynamic consistency, as well as better bilateral symmetry and stride-to-stride consistency under nominal walking conditions. The learned policy remains robust under muscle weakness and external perturbations without retraining.
• 中文摘要 肌肉驱动运动提供了一种物理基础的方法，能够生成真实的人体运动。然而，实现生理可行性和对肌肉骨骼能力变化及外部干扰变化的适应性仍是根本挑战。为解决这一限制，我们提出了一种反射知情的神经肌肉强化学习框架，用于肌肉驱动运动。在该框架内，固定的相位依赖反射控制器作为底层神经肌肉控制机制，而强化学习策略产生四个具有生物力学意义的残余参数，以调节与髋部摆动、膝盖支撑和踝关节推进相关的关键反射增益和阈值。实验结果表明，所提框架在正常行走条件下，能够生成生理上合理的运动，提升运动学准确性和动态一致性，以及更好的双侧对称性和步幅一致性。这种学来的政策在肌力无力和外部干扰下依然稳健，无需再训练。

Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation

通过实时雅可比估计快速学习灵巧的手持笔写

• Authors: Kai Stewart, Yasunori Toshimitsu, Robert K. Katzschmann
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.11775
• Pdf link: https://arxiv.org/pdf/2609.11775
• Abstract Dexterous in-hand manipulation of a grasped object with an anthropomorphic hand is an unsolved frontier for robot dexterity. The contact-richness and highly dynamic nature of object-hand interactions tend to require extensive modeling or data-collection efforts for learning-based approaches. Modern simulators used for reinforcement learning (RL) cannot fully replicate the required contact complexity, while collecting dexterous demonstrations for imitation learning (IL) remains an open problem. In this research, we present an embodied control approach based on real-time task Jacobian estimation of the combined hand and object system on the physical robot. Using only the CPU on a laptop, the proposed controller begins in-hand pen writing after approximately 18 s of initialization and continues to adapt online, without an analytic hand--object kinematic/contact model, simulation training, or precollected task demonstrations. We demonstrate that the same estimator/controller formulation works on three anthropomorphic robotic hand systems (one physical, two simulated) to show human-like, in-hand articulation of a grasped pen by an embodiment-independent formulation. Sub-millimeter in-plane precision (mean 0.6 mm across runs) is achieved across letters and shapes written in the air and on paper on a physical robot. To our knowledge, this is the first demonstration of an anthropomorphic hand writing arbitrary single-stroke trajectories with a grasped pen through purely in-hand motion, and it showcases an alternative to compute- and data-heavy approaches such as RL and IL for achieving dexterous manipulation through computationally simple and data-efficient algorithms.
• 中文摘要 用拟人化手灵巧地操控被抓物体，是机器人灵巧度的一个未解的前沿。物体-手部交互的接触丰富性和高度动态特性，往往需要大量建模或数据收集，以适应基于学习的方法。现代用于强化学习（RL）的模拟器无法完全复制所需的接触复杂度，而模拟学习（IL）的灵巧演示收集仍是一个未解之谜。本研究提出了一种基于实时任务雅可比估计的物理机器人手与物体系统结合的具象控制方法。仅使用笔记本电脑的CPU，拟议的控制器在约18秒初始化后开始手持笔写，并在线适应，无需分析手——物体运动学/接触模型、仿真训练或预先收集的任务演示。我们展示了相同的估计器/控制器公式在三种拟人化机器人手系统（一种物理型，两种模拟型）上工作，以展示通过身体无关的表述，实现了类似人手握笔的可操作性。在空气中书写字母和图形以及物理机器人纸上时，实现了亚毫米级平面精度（平均0.6毫米，跨次运行）。据我们所知，这是首次通过纯手部运动演示用握笔实现拟人手写任意单笔画轨迹的演示，展示了一种替代计算和数据密集方法（如强化学习和IL）的替代方案，通过计算简单且数据高效的算法实现灵巧操作。

MindTopo: Can Foundation Models Reason in Topological Space?

MindTopo：基础模型能在拓扑空间中推理吗？

• Authors: Yunfei Ge, Anbang Liu, Qineng Wang, Johnalbert Garnica, Jianwen Lyu, Zihan Wang, Reuben Tan, Jianfeng Gao, Ruohan Zhang, Yining Hong, Jiajun Wu, Manling Li
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.11900
• Pdf link: https://arxiv.org/pdf/2609.11900
• Abstract Spatial reasoning depends not only on metric properties such as distance, angle, and shape, but also on topological relations that remain invariant under continuous deformation. Cognitive science identifies these relations as foundational to spatial understanding, yet foundation-model evaluations largely focus on metric or viewpoint-dependent relations. We introduce MindTopo, a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots. MindTopo evaluates each property at two cognitive levels. Reasoning asks a model to identify topological relations or infer how they change. Planning instantiates a foundation model as a closed-loop agent whose policy selects environment actions. MindTopo contains 11,030 instances across 13 procedurally generated task types with controllable difficulty. We benchmark 14 MLLMs and study agent configurations augmented with image and video generation, including 3 video generative models in planning settings. Every MLLM performs better on reasoning than on planning, and the best-performing model remains far below observed human performance. On Qwen3-VL-2B-Instruct, supervised fine-tuning and reinforcement learning improve reasoning more than planning. Generated observations retain local cues and reach plausible endpoints, but audited rollouts do not reliably follow environment dynamics or preserve topology across transitions. Our website is at this https URL
• 中文摘要 空间推理不仅依赖于距离、角度和形状等度规属性，还依赖于在连续变形下保持不变的拓扑关系。认知科学将这些关系视为空间理解的基础，但基础模型评估主要关注度量或视点依赖关系。我们介绍MindTopo，这是一个基于认知科学和形式拓扑的五个属性的拓扑直觉基准：连续性、分离性、秩序、包围性和结。MindTopo在两个认知层面评估每个属性。推理要求模型识别拓扑关系或推断其变化。规划项目将基础模型实例化为闭环代理，其策略选择环境行动。MindTopo包含11,030个实例，涵盖13种程序生成的任务类型，难度可控。我们对14个MLLM和研究代理配置进行了基准测试，并辅以图像和视频生成，其中包括3个规划场景中的视频生成模型。每个MLLM在推理方面表现优于规划，且表现最佳的模型仍远低于观察到的人类表现。在Qwen3-VL-2B-Ininstruction中，监督微调和强化学习比规划更能提升推理能力。生成的观察保留了局部线索并达到合理的终点，但经审计的推广无法可靠地遵循环境动态或保持跨过渡的拓扑结构。我们的网站地址为 https URL

SenseNova-U1.5: Towards Native Unified Visual Intelligence

SenseNova-U1.5：迈向原生统一视觉智能

• Authors: Haiwen Diao, Jiahao Wang, Chenjing Ding, Hanming Deng, Jiangnan Chen, Ruixi Zhang, Ruohui Wang, Wenwen Tong, Xiangyu Fan, Yubo Wang, Yue Zhu, Yuwei Niu, Zhengqi Bai, Zhiqian Lin, Zhitao Yang, Zhongang Cai, Bo Yang, Chen Feng, Chengguang Lv, Guangjia Liu, Guanlin Wang, Hanyu Zhang, Haojia Yu, Hongcan Xiao, Hongli Wang, Huan Wu, Huaping Zhong, Jian Fang, Jianan Fan, Jiaqi Li, Jiefan Lu, Jing Zuo, Jingcheng Ni, Junxiang Xu, Linjun Dai, Mutian Xu, Peishen Yan, Penghao Wu, Ruijie Mao, Ruisi Wang, Shihao Bai, Shuang Yang, Shuya Yang, Shuyan Zheng, Silei Wu, Siying Li, Tao Chu, Tianbo Zhong, Tongxi Zhou, Weichao Luo, Weichen Fan, Wenhao Jia, Wenjie Gao, Xiangli Kong, Yan Li, Yang Yong, Zimo Wen, Zixuan Qian, Wenxiu Sun, Ruihao Gong, Quan Wang, Lewei Lu, Lei Yang, Ziwei Liu, Dahua Lin
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.11929
• Pdf link: https://arxiv.org/pdf/2609.11929
• Abstract We launch SenseNova-U1.5, an 8B-MoT native unified multimodal model that understands, reasons about, and generates visual content within an encoder-free and VAE-free architecture. We strengthen its visual interface through spatially coherent patch reconstruction and scale its training with carefully curated generation and editing data, improved task formulation, structural prompt enhancement, and native resolutions of up to 4K. For post-training, we optimize specialized experts for visual aesthetics, bilingual text rendering, infographic generation, and image editing, and consolidate their capabilities through multi-expert on-policy distillation. Across extensive evaluations, SenseNova-U1.5 largely advances image fidelity, text rendering, complex composition, multi-reference editing, and interleaved generation, while improving instruction following and preserving subject identity, geometry, and unmodified regions. Despite limited exposure to structured formats in its generation data, SenseNova-U1.5 generalizes effectively to long, complex, and structured visual instructions, further proving that multimodal understanding can transfer to visual planning and creation. Together, these findings position native unified modelling as a promising path towards systems that perceive, reason and create within a fully end-to-end framework. We will open-source training code, including supervised fine-tuning, reinforcement learning, and on-policy distillation.
• 中文摘要 我们推出了SenseNova-U1.5，这是一个8B-MoT原生统一多模态模型，能够理解、推理并在无编码器和无VAE的架构中生成视觉内容。我们通过空间相干的补丁重建强化其视觉界面，并通过精心策划的生成和编辑数据、改进的任务表述、结构提示增强以及最高4K的原生分辨率来扩展训练规模。在后期培训方面，我们优化了视觉美学、双语文本渲染、信息图生成和图像编辑的专业专家，并通过多专家的策略提炼整合了他们的能力。通过广泛的评估，SenseNova-U1.5在图像真实度、文本渲染、复杂构图、多引用编辑和交错生成方面取得了显著进步，同时提升了指令遵循率，并保持主体身份、几何体和未修改区域。尽管其生成数据中对结构化格式的接触有限，SenseNova-U1.5 仍有效推广到长篇、复杂且结构化的视觉指令，进一步证明多模态理解可转化为可视化规划和创建。这些发现共同使原生统一建模成为一条有前景的路径，实现在完全端到端框架内感知、推理和创造的系统。我们将开源训练代码，包括监督微调、强化学习和策略内提炼。

Keyword: diffusion policy

There is no result