生成时间: 2026-09-09 20:50:15 (UTC+8); Arxiv 发布时间: 2026-09-09 20:00 EDT (2026-09-10 08:00 UTC+8)

今天共有 72 篇相关文章

Keyword: reinforcement learning

Benchmarking Storage Systems for Machine Learning Workloads Using NIO Bench

利用NIO Bench进行机器学习工作负载的存储系统基准测试

• Authors: Jonathan W. Morris, Ionut Mistreanu, Connor Louie
• Subjects: Subjects: Performance (cs.PF); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.05418
• Pdf link: https://arxiv.org/pdf/2609.05418
• Abstract Machine learning training workloads place unique demands on storage systems, yet most existing benchmarks focus on computational throughput rather than file system I/O behavior. We present a benchmarking framework, Neural I/O Benchmark (NIO Bench), that characterizes storage access patterns across six diverse ML model architectures: Language Transformers, Vision Transformers, Diffusion Models, Spiking Neural Networks, Artificial Neural Networks, and Reinforcement Learning. Our framework employs a two-layer tracing approach combining Python-level I/O hooks for semantic phase context with Linux strace for complete syscall coverage including DataLoader worker subprocesses. We evaluate all six models on a Nautilus Kubernetes cluster with Ceph distributed file system. Our results reveal that I/O is heavily concentrated in data preparation, model loading, and model checkpointing. We also found that training is compute-bound rather than data-bound once data is staged, and that storage access follows an extreme power law where fewer than 10% of files account for over 90% of bytes transferred, and that read tail latency from cache misses on distributed storage is the primary storage bottleneck. These findings suggest that storage systems optimized for ML should prioritize aggressive data prefetching, page cache pinning, and efficient handling of bursty checkpoint writes.
• 中文摘要 机器学习训练工作负载对存储系统提出了独特要求，但大多数现有基准测试侧重于计算吞吐量，而非文件系统I/O行为。我们提出了一个基准框架——神经I/O基准（NIO Bench），该框架描述了六种不同机器学习模型架构下的存储访问模式：语言变换器、视觉变换器、扩散模型、尖峰神经网络、人工神经网络和强化学习。我们的框架采用两层追踪方法，结合Python级的语义阶段上下文输入/输出钩子与Linux追踪，实现包括DataLoader工作子进程在内的完整系统调用覆盖。我们在Nautilus的Kubernetes集群上评估了所有六个模型，采用Ceph分布式文件系统。结果显示I/O高度集中于数据准备、模型加载和模型检查点。我们还发现，一旦数据分阶段，训练是计算受限而非数据受限，存储访问遵循极端幂律，少于10%的文件占传输字节的90%以上，分布式存储缓存未命中导致的读尾延迟是主要存储瓶颈。这些发现表明，机器学习优化的存储系统应优先考虑积极的数据预取、页面缓存钉顶以及高效处理突发检查点写入。

Compiling VGDL into Causal Models

将VGDL编译成因果模型

• Authors: Mohit Jiwatode, Bodo Rosenhahn, Alexander Dockhorn
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.05459
• Pdf link: https://arxiv.org/pdf/2609.05459
• Abstract Reinforcement learning and large language models often struggle to accurately capture the causal mechanics of game environments. Standard reinforcement learning agents tend to rely on spurious correlations, while large language models are prone to hallucinating game rules. Although causal reinforcement learning improves interpretability, there is currently no formal methodology to map complex game mechanics directly into causal models. To address this, we propose a deterministic framework that compiles games specified in the Video Game Description Language into Dynamic Structural Causal Models. Rather than inferring causal structures from gameplay traces or noisy large language models' outputs, our methodology directly translates game components, including sprite dynamics, interaction rules, and termination conditions, into explicit structural equations. Each game tick represents a causal transition from state variables at time $t$ to $t+1$. By establishing this grounded mapping, the approach guarantees absolute causal fidelity to the ground-truth game mechanics. The resulting models offer transparent causal pathways that support counterfactual reasoning, causal reinforcement learning agent training, and procedural content validation. This framework provides a principled bridge between symbolic game descriptions and causally grounded game AI.
• 中文摘要 强化学习和大型语言模型常常难以准确捕捉游戏环境的因果机制。标准强化学习代理倾向于依赖虚假相关性，而大型语言模型则容易产生游戏规则的幻觉。尽管因果强化学习提高了可解释性，但目前尚无正式方法将复杂的游戏机制直接映射到因果模型中。为此，我们提出了一个确定性框架，将视频游戏描述语言中指定的游戏编译为动态结构因果模型。我们的方法论不从游戏轨迹或噪声大语言模型输出推断因果结构，而是直接将游戏组件（包括精灵动态、交互规则和终止条件）转化为显式结构方程。每个游戏刻表示时间$t$状态变量到$t+1$的因果转变。通过建立这种基于基础的映射，方法保证了对基层真实游戏机制的绝对因果忠实度。由此产生的模型提供了透明的因果路径，支持反事实推理、因果强化学习代理训练和程序性内容验证。该框架为符号游戏描述与基于因果的游戏人工智能之间提供了原则性的桥梁。

Information-Guided Safe Reinforcement Learning for Autonomous Gas Source Localization using sUAS

利用sUAS实现自主气体源定位的信息引导安全强化学习

• Authors: Sachin Giri, Thomas Zhao, Matthew Huynh, YangQuan Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.05569
• Pdf link: https://arxiv.org/pdf/2609.05569
• Abstract The autonomous localization of fugitive gas emissions using small Unmanned Aircraft Systems (sUAS) constitutes a fundamentally ill-posed inverse problem. In turbulent atmospheric boundary layers, highly intermittent scalar concentration fields violate the assumptions of classical gradient-based navigation, causing data-driven estimators to suffer from severe noise and spurious local minima. To address these challenges, we introduce an Information-Guided Safe Reinforcement Learning framework evaluated within a custom, GPU-accelerated 3D simulation environment coupling an Eulerian wind solver with a Lagrangian puff dispersion model. We identify a critical vulnerability in deterministic information-seeking planners - a Gramian bias where agents act greedily upon flawed early estimates, starving the estimator of spatial diversity. To systematically break this degeneracy, our architecture integrates a classical empirical observability Gramian (EMGR) planner with a learned Soft Actor-Critic (SAC) exploratory policy. A deterministic meta-supervisor actively monitors estimator reliability via Kullback-Leibler (KL) divergence, dynamically blending deterministic exploitation with learned exploration to steer the sUAS into high-information zones. Trained via a progressive curriculum and safeguarded by a strictly enforced Robust Control Barrier Function (RCBF), our RL framework achieves nearly 80% localization success on complex, mobile sources - drastically outperforming classical baselines (~30%) - while ensuring zero safety violations.
• 中文摘要 利用小型无人机系统（sUAS）自主定位逸散气体排放，构成了一个根本性的错误态逆问题。在湍流的大气边界层中，高度间歇的标量集中场违反了经典梯度导航的假设，导致数据驱动估计器存在严重噪声和虚假的局部极小值。为应对这些挑战，我们引入了信息引导安全强化学习框架，在定制的GPU加速3D模拟环境中评估，结合欧拉风求解器与拉格朗日蓬松色散模型。我们识别出确定性信息寻求规划器中的一个关键漏洞——格拉姆偏差，即代理贪婪地基于有缺陷的早期估计，导致空间多样性估计器被削弱。为系统地打破这种退化，我们的架构将经典的经验可观测性格拉米安（EMGR）规划器与已学习的软演员-批判者（SAC）探索策略相结合。确定性元监督者通过Kullback-Leibler（KL）发散主动监控估计器可靠性，动态结合确定性利用与学习探索，引导sUAS进入高信息区。通过渐进课程培训，并由严格执行的鲁棒控制障碍函数（RCBF）保护，我们的RL框架在复杂移动源上实现近80%的本地化成功率——远超经典基线（~30%）——同时确保零安全违规。

EnvCraft: Synthesizing Executable Environments in Agentic RL for Claw-like Agent

EnvCraft：在Agentic RL中为爪状代理综合可执行环境

• Authors: Yirong Zeng, Shen You, Jinhang Feng, Yufei Liu, Xiao Ding, Yutai Hou, Hao Cong, Yuxian Wang, Wu Ning, Wang Xu, Bibo Cai
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.05576
• Pdf link: https://arxiv.org/pdf/2609.05576
• Abstract The paradigm of LLMs has rapidly shifted from passive language interfaces to autonomous Claw-like agents that execute long-horizon tasks across stateful workspaces. While Agentic Reinforcement Learning (Agentic RL) provides a promising path to optimize these agents, its scaling is heavily bottlenecked by the severe scarcity of interactive training environments. Existing synthetic environments are strictly limited to tool-calling endpoints, rendering them insufficient for accommodating the end-to-end real-world demands of claw-like agents. To bridge this gap, we introduce EnvCraft, an automated framework for synthesizing executable environments and scalable training data. Specifically, EnvCraft employs an environment synthesis engine to build sandbox-isolated workspaces, alongside a topology-aware data generation engine to produce coherent task trajectories. Overall, we synthesize 139 interactive environments comprising approximately 20K complex tasks for Agentic RL training. Experiments on Qwen3/3.5 models (8B-32B) show that our method yields gains of up to +11.9% on Claw-style benchmarks and +8.0% on general tool-use benchmarks, with concurrent reductions in inference token cost. The results confirm that synthesized executable environments provide robust and generalizable learning signals for training.
• 中文摘要 LLM的范式已迅速从被动语言接口转向自主的Claw类智能体，这些智能体在有状态工作空间中执行长视野任务。虽然代理强化学习（Agentic RL）为优化这些代理提供了一条有前景的路径，但其扩展性因交互式训练环境的严重稀缺而严重受限。现有的合成环境严格限制于工具调用端点，使其不足以满足爪状智能体的端到端现实需求。为弥合这一差距，我们引入了EnvCraft，一个自动化的可执行环境综合框架和可扩展训练数据。具体来说，EnvCraft采用环境综合引擎构建沙盒隔离的工作空间，并配备拓扑感知的数据生成引擎生成连贯的任务轨迹。总体而言，我们综合了139个交互环境，涵盖约2万个复杂任务，用于代理强化学习训练。在Qwen3/3.5模型（8B-32B）上的实验显示，我们的方法在Claw风格基准测试上获得最高+11.9%的提升，在一般工具使用基准测试中获得+8.0%，同时推理令牌成本也相应降低。结果证实，综合可执行环境为训练提供了稳健且可推广的学习信号。

Endogenous Exploration in Reinforcement Learning with Intrinsic Curiosity

内生探索基于内在好奇心的强化学习

• Authors: Armando Vieira
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.05650
• Pdf link: https://arxiv.org/pdf/2609.05650
• Abstract We propose a reinforcement learning framework in which exploration is driven by intrinsic curiosity, designed for scenarios where environments are non-stationary and rewards are sparse, delayed, uninformative, or absent. In our model, action selection is guided by a combination of external rewards and an epistemic motivation mechanism that biases the agent toward structured exploratory directions. The central hypothesis is that effective exploration emerges at intermediate levels of incoherence, while performance degrades under both overly rigid and overly disordered dynamics. To test this idea, we implement the framework on top of a Liquid State Machine (LSM) substrate and evaluate it on two standard benchmarks: the discrete-action LunarLanderv2 and the continuous-control BipedalWalkerv3. The proposed method achieves competitive performance on both tasks relative to established deep RL algorithms, including Proximal Policy Optimization (PPO) and Intrinsic Curiosity Module (ICM). We further show that the curiosity window is not recovered in Active Inference agents under the same analysis, suggesting that the proposed dynamics capture a distinct exploration regime
• 中文摘要 我们提出了一种强化学习框架，其中探索由内在好奇心驱动，适用于环境非固定且奖励稀疏、延迟、信息不足或缺失的情境。在我们的模型中，行动选择由外部奖励和认知动机机制共同引导，使智能体倾向于结构化的探索方向。核心假说是，有效的探索出现在中间层次的非相干性，而在过于僵硬和过度无序的动态下表现会下降。为验证这一观点，我们将该框架应用于液态机（LSM）基底，并在两个标准基准测试上进行评估：离散动作的LunarLanderv2和连续控制的BipedalWalkerv3。该方法在两项任务上均与已建立的深度强化学习算法（包括近端策略优化（PPO）和内在好奇心模块（ICM）竞争。我们还进一步表明，在同一分析下，主动推理代理中未恢复好奇心窗口，表明所提出的动态捕捉了一个独特的探索模式

Newton Matching for Generative Modeling: A Unified Framework for Fine-Tuning and Sampling

牛顿匹配用于生成建模：一个用于微调和抽样的统一框架

• Authors: Zeyang Li, Yunan Wang, Paolo Giaretta, Navid Azizan
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.05727
• Pdf link: https://arxiv.org/pdf/2609.05727
• Abstract We develop Newton Matching, a unified framework for fine-tuning and sampling in generative modeling. The target is $\pi\propto\mu e^{\tau r}$, where $r$ is the reward, $\tau>0$ the inverse temperature, and $\mu$ denotes the pretrained model's terminal density for fine-tuning or the constant $1$ for sampling. We shift the paradigm from isolated losses to iterative optimization over canonical models: population minimizers of standard conditional matching for terminal densities. Under compatible smooth-realization assumptions, canonical velocities form a manifold diffeomorphic to the density manifold. Transporting the Fisher-Rao metric and mixture connection to this manifold, we show that the reverse-KL Hessian equals the metric, so the Newton direction coincides with the negative Fisher-Rao gradient. At terminal density $\rho$, each stage takes a tangential step generated by the regularized reward $r-\frac1\tau\log(\rho/\mu)$, followed by terminal-density-preserving canonicalization. This canonical retraction yields an exact finite-stepsize density characterization. For the ideal iteration, we prove strict reverse-KL descent away from the target for $0 < \eta \le \tau$, global convergence under mild conditions, and local quadratic convergence for full steps ($\eta=\tau$). Covariance and gradient forms, each with forward or reverse regression-pair constructions, yield sample-wise tangential-update losses with the same population minimizer, without importance sampling or full-trajectory backpropagation. We develop approximate updates and define critical-point consistency as vanishing tangential displacement if and only if $\rho=\pi$. We recover representative methods as exact realizations, critical-point-consistent approximations, or objective-altering variants, enabling modular algorithm design. Our work advances the theory and algorithms of reinforcement learning for generative models.
• 中文摘要 我们开发了牛顿匹配，这是一个用于生成建模中微调和抽样的统一框架。目标是$\pi\propto\mu e^{\tau r}$，其中$r$为奖励，$\tau>0$为反温度，$\mu$表示预训练模型的末端密度用于微调，常数为$1$。我们将范式从孤立损失转向对典型模型的迭代优化：终端密度标准条件匹配的总体最小化。在兼容的光滑实现假设下，正则速度构成一个与密度流形微分同胚的流形。将Fisher-Rao度规和混合关系传输到该流形，我们证明反KL黑森矩阵等于度规，因此牛顿方向与负Fisher-Rao梯度一致。在终端密度$\rho$处，每个阶段由正则化奖励$r-\frac1\tau\log（\rho/\mu）$生成一个切向步，随后进行终端密度保持的规范化。这种规范收缩得到了精确的有限步长密度刻画。对于理想迭代，我们证明了从目标方向严格反向KL下降为$0，<\eta \le \tau$，在温和条件下全局收敛，全步为局部二次收敛（$\eta=\tau$）。协方差和梯度形式均采用正向或反向回归对构造，在相同总体最小化器下产生样本间切向更新损失，无需重要性抽样或全轨迹反向传播。我们开发了近似更新，并将临界点一致性定义为当且仅当 $\rho=\pi$ 时切向位移为零。我们恢复代表性方法，如精确实现、临界点一致近似或改变目标的变体，支持模块化算法设计。我们的工作推动了生成模型强化学习的理论和算法的发展。

Inference-Time Graph Engineering for Multi-Agent LLM Workflows

多智能体LLM工作流的推理时间图工程

• Authors: Katherine Tieu, Dongqi Fu, Yinglong Xia, Hong Li, Hong Yan, Jingrui He
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.05774
• Pdf link: https://arxiv.org/pdf/2609.05774
• Abstract Recent multi-agent LLM systems increasingly rely on graph-structured communication to coordinate specialized agents. We revisit multi-agent orchestration from a graph-engineering perspective: rather than optimizing a static topology, we synthesize a task-conditioned temporal workflow graph that jointly specifies agent connectivity and edge-level communication semantics. We introduce ReActNet, a training-free framework that compiles a query and a set of role-specialized agents into a sequence of directed communication graphs. Each graph snapshot corresponds to one reasoning stage, and each edge carries a natural-language instruction specifying the message that a source agent should provide to a target agent. The compiled temporal graph is then executed through structured message passing: agents update their reasoning states by integrating their previous states with messages from controller-assigned neighbors, and a final aggregator synthesizes the resulting states into the answer. This design separates graph compilation from graph execution, making multi-agent coordination explicit, inspectable, and task-conditioned without requiring reinforcement learning or gradient-based topology optimization. Across knowledge reasoning, mathematical problem solving, code generation, and GAIA-style assistant tasks, ReActNet consistently improves over fixed-topology and learned-topology baselines while maintaining competitive inference cost. These results suggest that effective multi-agent orchestration depends not only on which agents communicate, but also on engineering executable workflow graphs that encode when, why, and how information should flow during reasoning.
• 中文摘要 近年来，多智能体LLM系统越来越依赖图结构通信来协调专用代理。我们从图工程的角度重新审视多智能体编排：我们不优化静态拓扑，而是综合一个任务条件时间工作流图，结合智能体连接性和边缘级通信语义。我们介绍ReActNet，一个无需训练的框架，将查询和一组角色专用智能体编译成一系列有向通信图。每个图快照对应一个推理阶段，每条边携带一条自然语言指令，指定源智能体应向目标代理提供何种消息。编译后的时序图随后通过结构化消息传递执行：智能体通过将先前状态与控制器分配邻居的消息整合来更新推理状态，最终聚合器将生成状态生成答案。该设计将图编译与图执行分离，使多智能体协调显式、可检查且具任务条件，无需强化学习或基于梯度的拓扑优化。在知识推理、数学问题解决、代码生成和GAIA式辅助任务中，ReActNet在保持竞争推理成本的同时，持续优于固定拓扑和学习拓扑基线。这些结果表明，有效的多智能体编排不仅取决于哪些智能体通信，还依赖于设计可执行的工作流程图，编码何时、为什么以及如何信息在推理过程中应流。

Generalizing HVAC Control With Domain Randomized Reinforcement Learning

将HVAC控制推广到领域随机强化学习

• Authors: Pablo Boitel, Kun Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.05822
• Pdf link: https://arxiv.org/pdf/2609.05822
• Abstract Deploying advanced HVAC (Heating, Ventilation and Air Conditioning) controllers at scale remains difficult because performance often depends on accurate building models or per-site retuning. We propose NOMAD-RL (Neural Online Meta-Adaptation for Dynamics), a general-purpose Reinforcement Learning (RL) controller designed to transfer across heterogeneous thermal zones through a universal, non-invasive thermostat interface. The controller acts on temperature setpoints from zone measurements and forecasts, while a recurrent policy supports online adaptation under partial observability. Our main contribution is an adaptive domain randomization scheme based on physics-informed normalizing flows, which models correlated and multimodal distributions of thermal-zone parameters while maintaining physical plausibility and controllability. This produces a realistic and progressively adaptive training curriculum that improves transfer across buildings. We evaluate NOMAD-RL against a constant-setpoint PID controller, RL without domain randomization, and MPC in single- and multi-zone settings. NOMAD-RL consistently outperforms the PID and non-randomized RL baselines, and approaches the performance of a well-tuned MPC, especially in the more challenging multi-zone case. These results highlight the potential of adaptive, physics-informed domain randomization for robust and transferable HVAC control.
• 中文摘要 大规模部署先进暖通空调（HVAC）控制器仍然困难，因为性能常依赖于准确的建筑模型或逐场地的重新调谐。我们提出了NOMAD-RL（神经在线动力学元适应），这是一种通用强化学习（RL）控制器，设计用于通过通用、非侵入式恒温器接口跨异构热区传输。控制器根据区域测量和预报的温度设定点进行操作，同时通过重复策略支持部分可观测性的在线适应。我们的主要贡献是基于物理知情归一化流的自适应域随机化方案，该方案在保持物理可行性和可控性的情况下，模拟热带参数的相关和多模态分布。这产生了一个真实且逐步适应性的培训课程，提升了建筑间的传输效率。我们评估了NOMAD-RL在恒定设定点PID控制器、无域随机化的RL以及单区和多区MPC的基础上。NOMAD-RL始终优于PID和非随机RL基线，性能接近调优良好的MPC，尤其是在更具挑战性的多区情况下。这些结果凸显了自适应、物理知情的域随机化在稳健且可转移的暖通空调控制中的潜力。

SLA-Safe Energy Control for AI-Native NG-RAN Using Stability-Aware Constrained PPO

基于AI原生网络的SLA-Safe能源控制，采用稳定性感知受限PPO

• Authors: Dharmendra Kumar
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG); Signal Processing (eess.SP)
• Arxiv link: https://arxiv.org/abs/2609.05861
• Pdf link: https://arxiv.org/pdf/2609.05861
• Abstract One important AI-for-RAN use case is energy saving, in which radio resources and cell energy modes must be dynamically controlled without violating user quality-of-service (QoS) or service-level agreement (SLA) requirements. However, aggressive sleep-state or deactivation decisions may reduce energy consumption at the cost of throughput degradation, delay increase, SLA violations, and unstable mode switching, especially under time-varying and bursty traffic conditions. This paper proposes a stability-aware constrained reinforcement learning framework for SLA-safe energy control in 5G NG-RAN. The problem is formulated as a constrained Markov decision process in which an AI-native controller selects closed-loop energy-saving actions based on cell load, queue status, active-user information, current energy mode, and SLA-related indicators. The proposed framework uses constrained proximal policy optimization with adaptive Lagrangian penalties to account for throughput, delay, and SLA constraints. To improve operation under traffic distribution shift, the controller is trained using mixed nominal and stress traffic regimes, while a switching-stability penalty is introduced to reduce oscillatory transitions between active and low-power modes. Simulation results in a seven-cell NG-RAN environment show that the proposed controller reduces energy consumption by approximately 41.4% under nominal traffic, 10.5% under stress traffic, and 22.9% under unseen-stress traffic relative to the Always-On baseline. Under stress and unseen-stress traffic, the controller preserves zero SLA violation and zero throughput loss, indicating service-preserving operation under challenging conditions. The proposed method also reduces switching activity compared with basic threshold-based energy saving.
• 中文摘要 一个重要的 AI for RAN 应用场景是节能，其中无线资源和小区能耗模式必须动态控制，同时不违反用户服务质量（QoS）或服务水平协议（SLA）要求。然而，激进的睡眠状态或停用决策可能以降低能耗为代价，但代价是吞吐量下降、延迟增加、SLA 违规和模式切换不稳定，尤其是在时间变化和突发性流量条件下。本文提出了一个基于稳定性的约束强化学习框架，用于 5G NG-RAN 中 SLA 安全能源控制。该问题被表述为一个受限马尔可夫决策过程，AI 原生控制器根据单元负载、队列状态、活跃用户信息、当前能量模式和 SLA 相关指标选择闭环节能动作。该框架采用受限近端策略优化，并采用自适应拉格朗日惩罚，以考虑吞吐量、延迟和SLA约束。为改善流量分配切换下的运行，控制器采用混合名义和应力流量模式训练，同时引入切换稳定性惩罚以减少主动与低功耗模式之间的振荡转变。在七单元网络-区域无线网络环境中的模拟显示，拟议控制器在标称流量下相较于常开基线降低约41.4%，压力流量10.5%，未见应力流量减少22.9%。在应力和未应力流量下，控制器保持零SLA违规和零吞吐量损失，表明在挑战条件下实现服务保全运行。该方法还减少了基于阈值的基本节能活动。

UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms

UniRRM：跨语言和评估范式的统一推理奖励模型

• Authors: Peng Lai, Yichao Du, Junchao Wu, Weibo Gao, Linan Yue, Longyue Wang, Weihua Luo, Derek F. Wong, Guanhua Chen
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.05910
• Pdf link: https://arxiv.org/pdf/2609.05910
• Abstract Reinforcement learning (RL) excels on tasks with verifiable rewards, but in open-ended tasks, the reliability of reward models remains a key challenge. Existing solutions either depend on costly proprietary LLM-as-a-Judge systems or opaque scalar reward models that lack interpretability. Recent works on generative reward models offer a promising alternative, but they remain constrained by static evaluation criteria, fragmented evaluation paradigms, and limited multilingual support. To address these challenges, we introduce \textbf{MixReward}, a large-scale multilingual dataset spanning six domains and 103 languages, containing both pairwise and listwise data, and propose \textbf{UniRRM}, a unified reasoning reward model supporting multiple languages and evaluation paradigms. UniRRM uses a staged reasoning chain to dynamically generate task-generic and instruction-specific criteria, enabling fine-grained, input-adaptive judgments while maintaining consistency across languages. Experiments demonstrate that UniRRM-8B and UniRRM-14B achieve performance close to the state-of-the-art for models of comparable size across multiple benchmarks, and are effective for unseen evaluation paradigms. In addition, ablation studies validate the reliability and effectiveness of UniRRM.
• 中文摘要 强化学习（RL）在可验证奖励的任务中表现出色，但在开放式任务中，奖励模型的可靠性仍是一个关键挑战。现有解决方案要么依赖昂贵的专有LLM即评判系统，要么是缺乏可解释性的不透明标量奖励模型。近期关于生成奖励模型的研究提供了有前景的替代方案，但它们仍受限于静态的评估标准、分散的评估范式和有限的多语言支持。为应对这些挑战，我们引入了\textbf{MixReward}——一个涵盖六个领域和103种语言的大规模多语言数据集，包含成对和列表数据，并提出了\textbf{UniRRM}，这是一个支持多种语言和评估范式的统一推理奖励模型。UniRRM 采用分阶段推理链动态生成任务通用和指令特定标准，实现细粒度、输入自适应的判断，同时保持语言间的一致性。实验表明，UniRRM-8B 和 UniRRM-14B 在多个基准测试中，在相当规模模型中性能接近最先进水平，且对看不见的评估范式有效。此外，消融研究验证了 UniRRM 的可靠性和有效性。

Test-Time Weak-to-Strong Alignment: Transferring Implicit Rewards from Weak to Strong Flow Models

测试时间弱到强对齐：从弱流向强流模型转移隐性奖励

• Authors: Xin Xie, Fan Zhang, Dong Gong
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.05968
• Pdf link: https://arxiv.org/pdf/2609.05968
• Abstract Aligning a text-to-image generation flow model with a reward makes it follow objectives that the training data alone does not provide. Alignment fine-tuning delivers this by reinforcement learning (RL) or preference optimization, but it must be repeated for every checkpoint and returns a model fixed at the reward and strength it was trained with. Test-time alignment instead steers a frozen model during sampling, allowing task-specific and sample-specific guidance. Existing methods obtain this only by drawing the per-step signal from the reward function itself, through its gradient, or through a separately trained value function. We propose changing the supervision source: let a pair of weak models, not a reward function, supply the supervision. A source aligned model, kept together with its base as a source alignment pair, stores its training reward as an implicit, step-wise, KL-anchored signal expressed in the sampler's own coordinates. We explore whether this model-form supervision can cross scale, and show that it does: our method, AlignGraft, aligns a larger, frozen, never-tuned model by adding the pair's velocity difference during sampling. The transport is exact under a shared noising kernel and needs neither the reward nor its gradient at test time. The method has no schedules, only a single scalar that controls the alignment strength and can extrapolate it beyond that of the source alignment pair. Across image and video flow models (Stable Diffusion 3.5, FLUX, and Wan), the transfer lifts the frozen large model on preference, compositional, and text-rendering rewards, can exceed the source aligned model itself, and preserves the large model's fidelity at a small constant sampling overhead. Extensive experiments show that one alignment run on a weak model produces supervision that the whole model family can reuse at test time.
• 中文摘要 将文本生成流程模型与奖励对齐，使其遵循训练数据无法单独提供的目标。比对微调通过强化学习（RL）或偏好优化实现这一点，但必须对每个检查点重复，并返回固定在训练时奖励和强度的模型。测试时间比对则在采样过程中引导冻结模型，实现任务特定和样本特定指导。现有方法仅通过从奖励函数本身、梯度或单独训练的值函数提取每步信号来实现这一点。我们建议更换监督源：让一对弱模型（而非奖励函数）提供监督。一个源比对模型与其基底作为源比对保持一致，将其训练奖励存储为采样器自身坐标中隐式、逐步、以KL锚定信号的形式。我们探讨了这种模型-形式监督是否可以跨尺度，并证明了它确实可以：我们的方法AlignGraft通过在采样时添加对对的速度差，对一个更大、固定且从未调优的模型进行比对。在共享的噪声核下，传输是精确的，测试时既不需要奖励，也不需要梯度。该方法没有时间表，只有一个标量控制比对强度，并能将其外推到源比对对之外。在图像和视频流模型（Stable Diffusion 3.5、FLUX和Wan）中，转移会提升冻结的大型模型在偏好、组合和文本渲染奖励上，可能超过源比对模型本身，并在小的恒定采样开销下保持大型模型的保真度。大量实验表明，在弱模型上运行一次比对会产生监督，整个模型家族在测试时可重复使用。

How to Learn from What a Human Would Avoid? Intervention-Aware World Models with Real-World RL for Dexterous Manipulation

如何从人类会避免的事情中学习？干预意识世界模型与现实现实强化学习，实现灵巧操作

• Authors: Jiaju Yin, Zhenhui Zhang, Lixin Xu, Heng Zhang, Jun Shao, Yating Feng, Arash Ajoudani, Renjing Xu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.06009
• Pdf link: https://arxiv.org/pdf/2609.06009
• Abstract Multi-fingered dexterous manipulation remains a frontier for real-world reinforcement learning (RL) due to the high-dimensional action space and the prohibitive cost of hardware failures. While human-in-the-loop (HIL) RL allows operators to intervene before failures occur, current pipelines often treat these interventions as reactive corrections, discarding the rich safety signal inherent in the operator's decision to take control. In this paper, we ask: How can we learn from what a human would avoid? We present WHIRL, a safety-aware RL framework that transforms binary human interventions into forward-predictive signals for proactive risk avoidance. Our approach centers on an intervention-aware latent world model with four prediction heads: dynamics, reward, termination, and a novel per-state intervention-probability head that learns to predict the likelihood of a human takeover at future states. This head provides an actor-side risk-shaping term that discourages the policy from entering "intervention-prone" regions, modeling the operator's internal safety threshold. We evaluate our framework on a 16-DoF LEAP Hand across tasks spanning convex and irregular object grasping, prismatic manipulation, and long-horizon multi-stage tasks. Our results show that predictive risk-shaping enables the system to achieve a 96.7 percent success rate on complex grasping tasks while reducing the operator intervention burden by up to 84 percent in step-weighted terms. By closing the loop between human intuition and predictive world modeling, this work provides a practical safety-aware recipe for training complex dexterous agents in the real world while reducing operator fatigue and hardware-risk exposure.
• 中文摘要 多指灵巧操作仍然是现实强化学习（RL）的前沿，这得益于高维动作空间和硬件故障的高昂成本。虽然人机参与（HIL）RL允许操作员在故障发生前介入，但当前流水线常将这些干预视为反应性修正，丢弃了操作员决定控制时固有的丰富安全信号。本文探讨：我们如何从人类会避免的事物中学习？我们提出了WHIRL，一种安全意识强化学习框架，将二元人类干预转化为前瞻性预测信号，实现主动风险规避。我们的方法围绕一个具四个预测头的干预感知潜在世界模型：动态、奖励、终止，以及一个新颖的每状态干预概率头，学习预测未来状态人类接管的可能性。该头提供了一个行为者侧的风险塑形项，阻止策略进入“易干预”区域，模拟操作员的内部安全阈值。我们评估了基于16 DoF LEAP Hand的框架，涵盖凸和不规则物体抓取、棱柱操作以及长视野多阶段任务。结果显示，预测风险塑造使系统在复杂抓取任务中实现96.7%的成功率，同时在步权下将操作员干预负担降低高达84%。通过闭合人类直觉与预测世界建模的环路，本研究提供了一种实用的安全意识配方，用于在现实世界中训练复杂灵巧的智能体，同时降低操作员疲劳和硬件风险暴露。

FALCON-S: Fixed-wing ground-effect Aerodynamics Simulator and Flight Control Learning Suite

FALCON-S：固定翼地面效应空气动力学模拟器和飞行控制学习套件

• Authors: Matteo El Hariry, Pedro Lima, Andrej Orsula, Matthieu Geist, Miguel Olivares-Mendez
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.06046
• Pdf link: https://arxiv.org/pdf/2609.06046
• Abstract We present a modular, high-fidelity simulation framework for the development and benchmarking of flight control strategies in fixed-wing aerial robots operating near the ground. Unlike existing simulators that rely on simplified or hover-oriented dynamics, our framework models full 6DoF rigid-body physics, semi-empirical ground-effect aerodynamics, actuator dynamics, sensor noise, and environmental disturbances. This physical realism, combined with modular component design, enables systematic analysis of low-altitude flight behavior under realistic conditions. The simulator supports both CPU and GPU backends via Torch and NVIDIA Warp, enabling high-throughput parallel execution suitable for large-scale reinforcement learning training and optimal control rollouts. A unified interface accommodates a range of controllers (both RL and optical control algorithms) across tasks such as altitude regulation and trajectory tracking. Cross-validation with X-Plane and JSBSim is also supported to facilitate engineering integration and visual fidelity.
• 中文摘要 我们提出了一个模块化、高保真模拟框架，用于在近地面固定翼空中机器人中开发和基准飞行控制策略。与依赖简化或悬停动力学的现有模拟器不同，我们的框架模拟了完整的6DoF刚体物理、半经验式地面效应空气动力学、执行器动力学、传感器噪声和环境干扰。这种物理真实性结合模块化组件设计，使得在现实条件下系统分析低空飞行行为成为可能。该模拟器支持通过Torch和NVIDIA Warp支持CPU和GPU后端，实现适合大规模强化学习训练和最优控制展开的高吞吐量并行执行。统一界面支持多种控制器（包括强化学习和光学控制算法），涵盖高度调节和轨迹跟踪等任务。还支持与 X-Plane 和 JSBSim 的交叉验证，以促进工程集成和视觉准确性。

SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Docking

SCoCaT：成功条件条件约束强化学习以助航天器对接

• Authors: Aman Arora, Ricard Marsal I Castan, Matteo El-Hariry, Miguel Olivares-Mendez
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.06061
• Pdf link: https://arxiv.org/pdf/2609.06061
• Abstract Termination-based constrained reinforcement learning is attractive for safety-critical robotic deployments: it avoids online optimization at inference, scales easily to many constraints via a single scalar per constraint, and is simpler to implement than commonly used Lagrangian methods. Instead of pricing violations through summed cost penalties, this approach makes violations structurally unprofitable by shortening the effective horizon for each violation. We identify a structural failure mode of this method class on terminal-navigation tasks: reaching a precise goal configuration while satisfying safety constraints that tighten along the final approach. When the goal sits inside the region close to where the constraints become active, the survival-weighted objective makes dwelling outside the goal region strictly preferable to entering, producing high constraint compliance with low task completion. We formalize this pathology and show that a minimal augmentation to off-the-shelf RL algorithms like PPO resolves this ``feasibility collapse''. We empirically demonstrate that adding a dense per-step success signal via an auxiliary value critic improves the task completion rate while maintaining safety-critical constraint compliance. Validation across two representative spacecraft platforms: a 6U-CubeSat spanning the mass and degree-of-freedom envelope of operational proximity operations, and a floating platform testbed for zero-shot sim-to-real transfer in our laboratory, supports the generality of these findings.
• 中文摘要 基于终止的约束强化学习对安全关键的机器人部署具有吸引力：它避免了推理时的在线优化，通过每个约束的单一标量轻松扩展到多个约束，且比常用的拉格朗日方法更易实现。该方法不再通过累计成本罚款来定价违规，而是通过缩短每次违规的有效视角，使违规在结构上变得无利可图。我们识别出该方法类别在终端导航任务中的结构性失效模式：在达到精确目标配置的同时满足在最终接近过程中逐渐收紧的安全约束。当目标位于约束激活区域附近时，生存加权目标使得停留在目标区域外严格优于进入，从而产生高约束合规性但任务完成度低。我们对这一病理现象进行了形式化，并证明对现成的强化学习算法如PPO进行最小增强即可解决这种“可行性崩溃”。我们实证证明，通过辅助值批判器添加密集的每步成功信号，可以在保持安全关键约束合规性的同时提高任务完成率。在两个具有代表性的航天器平台上的验证：一个覆盖运行近距离操作质量和自由度包线的6U立方星，以及我们实验室中用于零发射模拟到实物传输的浮动平台测试平台，支持了这些发现的普遍性。

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

DataFlex-RL：RLVR数据政策评估平台

• Authors: Hao Liang, Mingrui Chen, Hengyi Feng, Meiyi Qiang, Wentao Zhang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.06107
• Pdf link: https://arxiv.org/pdf/2609.06107
• Abstract Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which domains contribute to subsequent training batches. We introduce DataFlex-RL, an evaluation platform for comparing these choices under a common GRPO recipe. Our primary experiment evaluates 13 configurations across 12 matched seeds using Qwen2.5-7B-Base and 12 mathematics, logic, and science benchmarks. Uniform GRPO improves the domain-balanced average accuracy by 7.76 percentage points over the untrained checkpoint. None of the eight rollout-selection or reweighting methods achieves a paired 95% confidence interval that excludes zero relative to uniform sampling, and none of the three adaptive mixtures outperforms a fixed equal mixture at the same level of precision. A corrected 12-seed extension on Llama-3.1-8B-Base places the additional methods on the same score scale as the original controls, but does not reveal a consistent winner in terms of observed mean performance. We also quantify evaluation sensitivity by rescoring nine Qwen2.5-7B-Instruct runs using a math-heavy six-benchmark summary, consisting of five mathematics benchmarks and GPQA-Diamond but no logic benchmark, and comparing it with the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated, with a correlation coefficient of -0.33, whereas summaries that retain all 12 benchmarks largely agree. Across the controlled settings studied here, changing the data policy measurably changes the training process but does not produce a reproducible improvement over uniform training.
• 中文摘要 可验证奖励强化学习（RLVR）数据策略决定哪些推广被使用、权重强度以及哪些领域对后续训练批次有贡献。我们引入DataFlex-RL，这是一个在共同GRPO配方下比较这些选择的评估平台。我们的主要实验评估了12个匹配种子的13种配置，使用Qwen2.5-7B-Base和12个数学、逻辑和科学基准。统一GRPO使领域平衡平均准确率比未训练检查点提升7.76个百分点。八种展开选择或重新加权方法中，没有一个达到相对于均匀抽样排除零的配对95%置信区间，三种自适应混合法在相同精度下均未优于固定相等混合。对Llama-3.1-8B-Base的修正12种子扩展将额外方法置于与原始对照组相同的评分尺度上，但并未显示一致的均值表现。我们还通过重新评分九次Qwen2.5-7B-Instruct运行，使用数学密集的六个基准测试（包含五个数学基准和GPQA-Diamond但无逻辑基准）进行评分，并与领域平衡的12基准总结进行比较，量化了评估敏感性。所得排名呈负相关，相关系数为-0.33，而保留全部12个基准的总结则基本一致。在本研究的受控环境中，改变数据策略可衡量地改变训练过程，但无法带来相较于均匀训练的可重复性提升。

Fixed-Time Integral Reinforcement Learning for Saturated Nonlinear Multi-Agent Systems Under FDI Attacks

在外商直接投资攻击下，针对饱和非线性多智能体系统的固定时间积分强化学习

• Authors: Tien Dat Vu, Minh Doan
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.06163
• Pdf link: https://arxiv.org/pdf/2609.06163
• Abstract The leader-follower formation control problem is investigated for nonlinear multi-agent systems with unknown dynamics, external disturbances, and false data injection (FDI) attacks on actuator channels. The problem is formulated as a zero-sum differential game and solved using the Integral Bellman-Isaacs approach. To address input saturation constraints, a non-quadratic control cost function is incorporated into the optimization problem, leading to a bounded control law. Furthermore, this paper proposes a cost function construction method and develops a critic learning law, which together guarantee the practical fixed-time stability of the system while overcoming the limitations of existing fixed-time reinforcement learning formulations. Finally, the practical fixed-time convergence of both the critic weight estimation error and the leader-referenced formation tracking error to bounded residual sets is rigorously proven. Simulation results demonstrate the effectiveness of the proposed method under external disturbances, FDI attacks, and input constraints.
• 中文摘要 研究了具有未知动力学、外部扰动和执行器通道假数据注入（FDI）攻击的非线性多智能体系统中的先导-跟随者形成控制问题。该问题被表述为零和微分博弈，采用积分贝尔曼-艾萨克斯方法求解。为解决输入饱和约束，在优化问题中加入了非二次控制成本函数，从而得出有界控制律。此外，本文提出了成本函数构造方法，并发展了批判学习定律，共同保证系统的实际固定时间稳定性，同时克服了现有固定时间强化学习形式的局限。最后，严格证明了批判权重估计误差和领导参考编队跟踪误差与有界剩余集的实际固定时间收敛性。模拟结果证明了该方法在外部干扰、FDI攻击和输入约束下的有效性。

Spectral Prioritized Sweeping in Nonstationary Reinforcement Learning

非平稳强化学习中的谱级优先扫描

• Authors: Hung Pham, Tuan Dam
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.06186
• Pdf link: https://arxiv.org/pdf/2609.06186
• Abstract Prioritized Sweeping (PS) accelerates model-based reinforcement learning by selecting backups according to Bellman residual magnitude. In nonstationary reward settings, however, the canonical priority score is shortsighted: after a localized reward shift, residuals propagate only through realized backups, so bottlenecked or topologically distant state estimates may remain static under a limited replanning budget. We introduce the Graph Topology Augmentation framework, which employ the graph's resolvent and its diffusion semantic, to augment the inquired signal. Our application, Graph Topology Augmentation for Prioritized Sweeping (GTA-PS), or which the alias Spectral Prioritized Sweeping (SPS) might be more universal, provides a drop-in ordering score for the setting of fixed dynamics and changing state rewards. GTA-PS uses a smootherized policy, inducing a transition chain, with its in- and out-Laplacian. The standard priority key is augmented with a mixing of regularized Laplacian inverses diffusing the residual magnitude. Furthermore, the topology contribution is annealed by a scheduler based on the Second Largest Eigenvalue Modulus (SLEM), allowing its scale to adapt to the chain's mixing regime. We prove that the forward potential coincides with geometric discounted residual propagation and show that GTA-PS gives active priority instantly to all states. Tabular experiments on FourRooms and GARNET domains demonstrate improved replanning efficiency over standard PS under both exact DP and Dyna-style host planners.
• 中文摘要 优先扫描（PS）通过根据贝尔曼残差大小选择备份，加速基于模型的强化学习。然而，在非平稳奖励环境中，典型优先级分数目光短浅：在局部奖励偏移后，残差仅通过实现的备份传播，因此瓶颈或拓扑远离状态估计在有限的重新规划预算下可能保持不变。我们引入图拓扑增强框架，利用图的解式及其扩散语义来增强查询信号。我们的应用程序——优先扫描图拓扑增强（GTA-PS），或称谱优先扫描（SPS）可能更通用，提供了固定动态设置和状态奖励变化的可插入排序评分。GTA-PS采用平滑策略，诱导过渡链，其入拉普拉斯算子和出拉普拉斯算子。标准优先级密钥通过混合正则化拉普拉斯逆函数来扩散残余大小。此外，拓扑贡献被基于第二大特征值模数（SLEM）的调度器退火，使其尺度适应链的混合状态。我们证明了前向势能与几何贴现残差传播一致，并证明GTA-PS对所有状态即时给予主动优先级。在FourRooms和GARNET域上的表格实验显示，在精确DP和Dyna风格宿主规划器下，重规划效率优于标准PS。

MobileVLA-R1 2.0: RL-Enhanced Reasoning for Mobile Robot Control

MobileVLA-R1 2.0：增强学习驱动的移动机器人控制推理

• Authors: Ting Huang, Yue Huang, Zeyu Zhang, Shuicheng Yan, Hao Tang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.06251
• Pdf link: https://arxiv.org/pdf/2609.06251
• Abstract Grounding natural-language instructions into reliable and executable actions remains a fundamental challenge for vision-language-action (VLA) systems on mobile robots, due to the persistent gap between high-level semantic reasoning and low-level locomotion and manipulation control. Existing approaches often rely on implicit reasoning or monolithic action prediction, making it difficult to maintain coherent long-horizon decision making while producing precise and adaptable robot actions. To address this challenge, we propose MobileVLA-R1 2.0, an RL-enhanced VLA framework that explicitly couples structured embodied reasoning with executable mobile robot control. The framework learns multi-granularity reasoning over embodied trajectories through supervised Chain-of-Thought (CoT) alignment and reinforcement learning, improving reasoning-to-action consistency beyond purely behavioral supervision. To support both locomotion and manipulation, we further introduce a reasoning-conditioned action decoder that maps multimodal reasoning representations to task-level action targets, which are subsequently translated into embodiment-specific commands by robot controllers. This design provides a unified perception-reasoning-action interface while decoupling high-level action generation from robot-specific actuation. We conduct extensive evaluations on language-guided navigation, quadruped control, and humanoid mobile manipulation, covering VLN-CE, QUARD, and real-world deployments on Unitree Go2 and G1 robots. MobileVLA-R1 2.0 consistently outperforms strong VLA baselines, achieving an average 1.6 point improvement in SR on VLN-CE and a 10.0 point improvement in full-task success on real-world G1 mobile manipulation tasks over MobileVLA-R1, while demonstrating robust long-horizon instruction following and closed-loop execution across different robotic platforms.
• 中文摘要 将自然语言指令扎根为可靠且可执行的动作仍是移动机器人视觉-语言-动作（VLA）系统面临的根本挑战，因为高级语义推理与低级运动及操作控制之间存在持续的差距。现有方法常依赖隐性推理或单一的动作预测，这使得在产生精准且可适应的机器人动作的同时，难以保持连贯的长期决策。为应对这一挑战，我们提出了MobileVLA-R1 2.0，这是一个增强型VLA框架，明确将结构化具身推理与可执行移动机器人控制耦合。该框架通过监督式思维链（CoT）对齐和强化学习，在具身轨迹上学习多粒度推理，提升推理与行动的一致性，超越纯粹的行为监督。为支持运动和操作，我们进一步引入了一种推理条件动作解码器，将多模态推理表示映射到任务级动作目标，随后由机器人控制器转换为具身特定命令。该设计提供了统一的感知-推理-动作接口，同时将高级动作生成与机器人特定驱动脱钩。我们对语言引导导航、四足控制和类人生物移动操作进行了广泛评估，涵盖了VLN-CE、QUARD以及Unitree Go2和G1机器人上的实际部署。MobileVLA-R1 2.0 持续优于强 VLA 基线，VLN-CE 上的 SR 平均提升 1.6 分，在实际 G1 移动操作任务中全任务成功率提升 10.0 分，同时在不同机器人平台上展现了稳健的长视距指令跟踪和闭环执行能力。

Geometric Distributional Control: Learning Progress with Partial Structural Knowledge

几何分布控制：部分结构知识下的学习进展

• Authors: Tong Wu
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.06331
• Pdf link: https://arxiv.org/pdf/2609.06331
• Abstract Real-time control often sits between two limiting regimes. Predictive optimization and model-based control are powerful when dynamics, parameters, objectives, and online planning models are specified; reinforcement learning can relax this requirement, but must infer long-horizon value signals from sequential data and interaction, making training slow, high-variance, and hard to scale in large action spaces. This middle regime is common in systems including autonomous driving, warehouse robotics, traffic control, and delivery drones: partial geometry, physics, rules, or constraints are known, yet the local direction of task progress remains uncertain. Geometric Distributional Control (GDC) is designed for this partial-knowledge setting. It factorizes control into feasibility and progress: known geometry, rules, constraints, and response maps define an executable scaffold, while progress-weighted feasible data learns the missing directional signal on that scaffold. The learned score acts as a Bellman-like local value-gradient, selecting actions that make progress without requiring global Bellman recursion, a fully specified planner, or a black-box policy that absorbs both feasibility and preference. This knowledge can be lightweight and partial, such as simple dynamics, safety filters, local maps, constraint projectors, or lower-level response maps; it need not encode full dynamics or a long-horizon objective. Offline, GDC fits a progress-tilted distribution from short known-feasible snippets with weak signed progress certificates. Online, its score is projected through the scaffold and applied in receding-horizon feedback. We prove that this score descends a data-induced soft progress value and validate GDC on structured multilevel optimization and SUMO route-progress driving, where it improves over known-only solvers and learning baselines while preserving scaffold-enforced feasibility.
• 中文摘要 实时控制通常介于两个极限区间。当动态、参数、目标和在线规划模型被指定时，预测优化和基于模型的控制非常强大;强化学习可以放宽这一要求，但必须从顺序数据和交互中推断长视野值信号，使得训练缓慢、高方差且在大型动作空间中难以扩展。这种中间状态在自动驾驶、仓库机器人、交通控制和无人机等系统中很常见：部分几何、物理、规则或约束已知，但任务进展的局部方向仍不确定。几何分布控制（GDC）专为这种部分知识环境设计。它将控制分解为可行性和进展：已知几何、规则、约束和响应映射定义可执行脚手架，进度加权可行数据则学习该支架上缺失的方向信号。所学分数作为类似Bellman的局部值梯度，选择那些无需全局Bellman递归、完全指定规划器或兼顾可行性和偏好的黑箱策略即可取得进展的行动。这些知识可以是轻量级和部分性的，如简单动力学、安全滤波器、局部地图、约束投影器或低级别响应图;它不必编码完整的动态或长期目标。离线时，GDC从带有弱签名进展证书的短已知可行片段拟合进度倾斜分布。在线时，其分数通过支架投影并应用于远征视界反馈中。我们证明该分数下降数据诱导的软进展值，并在结构化多层优化和SUMO路径进度驱动上验证GDC，在保持支架强制可行性的同时，提升了已知求解器和学习基线。

Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts

验证器错误在GRPO组内是独立的吗？Qwen2.5推广的证据

• Authors: Esther Xin
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.06386
• Pdf link: https://arxiv.org/pdf/2609.06386
• Abstract Group-based reinforcement learning with verifiable rewards (RLVR) scoresmultiple completions per prompt using automatic verifiers. Analysesbased on independent verifier errors may overlook dependence associatedwith shared answer formats. We investigate this dependence in24,998 groups of eight completions generated by Qwen2.5-1.5B onMATH, GSM8K, and DeepMath-103K. We estimate a pooled within-groupverifier-error correlation of 0.530 (95% confidence interval:0.500--0.560). Under an exchangeable-error model, this correspondsto a design-effect-adjusted effective sample size of 1.70 for aneight-completion group. Dependence varies substantially across answerforms: fractions, radicals, symbolic expressions, and intervals exhibitstronger clustering than unit annotations and percent signs. Replayinggroup-relative advantages across four rule-based verifier configurationsidentifies at least one advantage-sign disagreement in up to 0.83% ofgroups. Because a group is repeated sampling for one prompt, thiswithin-group clustering may reflect shared prompt difficulty as well asshared answer form, and we do not attempt to separate the two this http URL studies of correlated judgments across multiple evaluators, ouranalysis examines dependence across completions scored by the sameverifier. These findings motivate prompt- and answer-form-aware analysesof verifier noise rather than characterizations based solely onaggregate error rates.
• 中文摘要 基于群体的可验证奖励强化学习（RLVR）使用自动验证器对每个提示进行多次完成评分。基于独立验证者错误的分析可能忽略共享答案格式相关的依赖性。我们在Qwen2.5-1.5B生成的24,998组、共8个完成题组中研究了该依赖性，覆盖MATH、GSM8K和DeepMath-103K。我们估计组内验证者与错误的合并相关数为0.530（95%置信区间：0.500--0.560）。在可交换错误模型下，这对应八个完成组的设计效应调整有效样本量为1.70。依赖性在不同答案形式间差异显著：分数、根号、符号表达式和区间的聚类比单位注释和百分比符号更强。在四个基于规则的验证器配置中重放组组相对优势，最多可识别出0.83%组中至少一个优势符号不一致。由于一组对同一提示进行了重复抽样，这种组内聚类可能反映了共享的提示难度和共享的答案形式，我们不试图将两项相关评审者间的相关判断分离，我们的分析考察了同一验证者评分的完成度之间的依赖性。这些发现促使对验证者噪声进行基于提示和答案表单的分析，而非仅基于总计错误率的特征描述。

Local and Global Stability in Performative Reinforcement Learning

执行强化学习中的局部与全局稳定性

• Authors: Debmalya Mandal
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.06467
• Pdf link: https://arxiv.org/pdf/2609.06467
• Abstract In performative reinforcement learning the deployed policy shapes the environment that generates the learner's future data, and the natural solution concept is a performatively stable policy that is optimal in the environment it induces. Existing convergence guarantees rely on Lipschitz sensitivity assumptions on the environment map $\pi \mapsto (P_\pi, r_\pi)$, which are hard to verify and fail in settings such as multi-agent best-response dynamics. We instead study stability for mixtures of policies, and show that the resulting picture is fundamentally different from performative prediction, where randomization removes the need for any sensitivity assumption. We distinguish local mixed stability, an occupancy-weighted first-order relaxation that we show is equivalent to stationarity, from global mixed stability, which certifies against arbitrary deviating policies. Our first result is that a weighted per-state Hedge dynamic drives the local stability gap to zero at an $O(1/\sqrt{T})$ rate for an arbitrary, possibly discontinuous, environment map, both with exact and with trajectory feedback. The two notions genuinely differ: we exhibit an instance where local stability is achieved exactly but every mixture has global stability gap bounded away from zero. For global stability we introduce a bounded transition range assumption, strictly weaker than Lipschitz sensitivity, under which unweighted per-state Hedge converges up to a floor of $O(\gamma\epsilon_P/(1-\gamma)^3)$, and we prove a matching-in-$\epsilon_P$ lower bound of $\Omega(\gamma\epsilon_P/(1-\gamma))$ under trajectory feedback, so this floor is unavoidable. Finally, we extend both notions to $n$-player performative Markov games, obtaining local stability with no assumption on the joint environment map or game structure, and global stability for performative Markov potential games.
• 中文摘要 在执行强化学习中，部署的策略塑造了生成学习者未来数据的环境，自然解概念是一个在其所诱导环境中最优的执行稳定策略。现有的收敛保证依赖于环境映射 $\pi \mapsto （P_\pi， r_\pi）$ 上的利普希茨敏感性假设，这些假设难以验证，且在多智能体最佳响应动态等环境中失败。我们转而研究策略混合的稳定性，并证明所得图像与执行预测本质不同，后者通过随机化消除了任何敏感性假设。我们将局部混合稳定性（一种占用加权的一阶松弛，我们证明其等价于平稳性）与全局混合稳定性（对任意偏离策略进行认证）区分开来。我们的第一个结果是，加权的每状态对冲动态在任意且可能不连续的环境图上，以$O（1/\sqrt{T}}）速率将局部稳定间隙推向零，且具有精确反馈和轨迹反馈。这两个概念确实存在差异：我们展示了局部稳定性恰好实现，但每个混合物的全局稳定性差距均为零的边界。对于全局稳定性，我们引入了一个严格弱于Lipschitz敏感性的有界转移范围假设，在此假设未加权的每状态对冲收敛至$O（\gamma\epsilon_P/（1-\gamma）^3）$的下限，并证明了在轨迹反馈下$\Omega（\gamma\epsilon_P epsilon_P/（1-\gamma））$的匹配下界，因此该底线不可避免。最后，我们将这两个概念扩展到$n$玩家的执行性马尔可夫博弈，获得局部稳定性且不假设联合环境图或博弈结构，并实现执行性马尔可夫势博弈的全局稳定性。

One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control

一步一导线：通过跨步控制缓解多域强化学习中的高阶干扰

• Authors: Zihan Lin, Xiaohan Wang, Jie Cao, Jiajun Chai, Guojun Yin, Wei Lin, Ran He
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.06469
• Pdf link: https://arxiv.org/pdf/2609.06469
• Abstract Reinforcement learning (RL) across multiple domains can broaden the reasoning capabilities of large language models (LLMs), yet joint training often degrades individual-domain performance and can destabilize optimization. Existing work typically diagnoses such interference from a single-step view using first-order gradient alignment or curvature-based proxies. We show that this view can miss a critical form of sequential interference: same-point domain gradients may remain nearly orthogonal even when consecutive realized updates partially reverse one another in output space. We further show that consecutive token log-probability footprints recover this interaction directly from adjacent checkpoints as a local second-order interaction in output space, without explicitly reconstructing same-step curvature. Building on this insight, we propose OSOL, which designates a focus domain at each iteration, uses the preceding checkpoint footprint to rank token-level rebound risk, and applies a drift-ranked, adaptively scaled correction within the standard GRPO update. Our analysis shows that this correction suppresses the targeted cross-step output backtracking component. Controlled studies further show that cross-step backtracking is more strongly associated with subsequent task damage than same-point gradient diagnostics, while the preceding footprint ranks future rebound risk more accurately than Hessian-based proxies. On Qwen3-30B-A3B, OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
• 中文摘要 跨多个领域的强化学习（RL）可以拓宽大型语言模型（LLM）的推理能力，但联合训练常常降低单个领域的性能，并可能破坏优化稳定性。现有研究通常通过单步视角诊断此类干扰，使用一阶梯度对齐或基于曲率的代理。我们表明这种视角可能忽略一种关键的顺序干扰形式：即使连续实现的更新在输出空间中部分反转，同点域梯度仍可能保持近乎正交。我们进一步证明，连续的令牌对数概率足迹直接从相邻检查点恢复这种交互，作为输出空间中的局部二阶交互，而无需明确重建同步曲率。基于这一见解，我们提出了OSOL，即每次迭代指定一个焦点域，利用前述检查点足迹对代币级反弹风险进行排名，并在标准GRPO更新中应用漂移排名、自适应缩放的修正。我们的分析显示，这种修正抑制了目标交叉步输出回溯成分。受控研究进一步表明，交叉步回溯与后续任务损伤的关联性比同点梯度诊断更强，而前一步足迹对未来回弹风险的排名比基于Hessian的代理更准确。在Qwen3-30B-A3B上，OSOL达到0.4822的域-宏平均值，较最强基线提升5.7%，且无明确的高阶差异。

Thinking with Cameras: Active Visual Reasoning via Dynamic Viewpoint Control for Surveillance Video Understanding

用摄像头思考：通过动态视点控制实现监控视频理解的主动视觉推理

• Authors: Xiao Zhang, Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Shichao Kan
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.06475
• Pdf link: https://arxiv.org/pdf/2609.06475
• Abstract Large vision-language models (LVLMs) have recently achieved remarkable progress in general-purpose video understanding. However, their application to surveillance videos remains challenging due to the lack of large-scale domain-specific datasets and the limitation of passive observation from fixed viewpoints. In surveillance scenarios, critical visual evidence can be easily missed when targets are distant, small, occluded, or move beyond the current camera view. In this work, we introduce CamVLM, a new framework for Thinking with Cameras, which enables LVLMs to actively acquire visual evidence through dynamic viewpoint control rather than passively analyzing fixed video streams. We first construct CCTV-Anomaly, a large-scale surveillance video understanding dataset containing 14,459 videos across 10 anomaly categories, with detailed captions and event annotations. We further formulate viewpoint control as an active visual perception problem and build CamTrack-53K, an object-centric viewpoint trajectory dataset for learning camera actions. Moreover, we propose a reinforcement learning based viewpoint policy optimization framework, which models camera control as a sequential decision-making process and learns long-horizon observation strategies beyond supervised trajectory imitation. Extensive experiments demonstrate that CamVLM achieves state-of-the-art performance under both passive observation and dynamic viewpoint settings, validating the effectiveness of active camera-based reasoning for surveillance video understanding. Our datasets, model, and code will be available at this https URL .
• 中文摘要 大型视觉语言模型（LVLM）近年来在通用视频理解方面取得了显著进展。然而，由于缺乏大规模领域特定数据集以及固定视角被动观察的限制，其在监控视频中的应用仍具挑战性。在监控场景中，当目标距离远、小、被遮挡或超出当前摄像头视野时，关键视觉证据很容易被遗漏。本研究介绍了CamVLM，一种新的“与摄像头思维”框架，使LVLM能够通过动态视角控制主动获取视觉证据，而非被动分析固定视频流。我们首先构建了CCTV-Anomaly，这是一个包含14,459个视频、涵盖10个异常类别的大规模监控视频理解数据集，配有详细的字幕和事件注释。我们进一步将视点控制表述为主动视觉感知问题，构建了CamTrack-53K，一个以对象为中心的视角轨迹数据集，用于学习摄像机动作。此外，我们提出了基于强化学习的视角策略优化框架，将摄像机控制建模为顺序决策过程，并学习超越监督轨迹模仿的长远观察策略。大量实验表明CamVLM在被动观察和动态视角设置下均具备最先进的性能，验证了基于主动摄像机推理对监控视频理解的有效性。我们的数据集、模型和代码将通过此 https URL 访问。

Certifying cooperation: a novel approach to cooperative multi-agent task generation

认证合作：合作多智能体任务生成的一种新方法

• Authors: Yannick Molinghen, Hugo Charels, Tom Lenaerts
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.06586
• Pdf link: https://arxiv.org/pdf/2609.06586
• Abstract A shared reward gives agents a common objective, but leaves open when, how and even whether they must cooperate to succeed. We address these questions in the Laser Learning Environment, a multi-agent path-finding environment where cooperation materializes as one agent blocking a laser to let a teammate pass safely. We represent these interactions through temporal cooperation graphs whose timed edges connect helpers to beneficiaries, define six cooperation profiles as overlapping graph predicates, and prove that every cooperative trajectory satisfies at least one. By encoding the environment dynamics and profile predicates as propositional formulae, we distinguish tasks that admit}a profile in some winning trajectory from those that require it in every winning trajectory within a specified horizon. Used as filters, these queries turn a random layout sampler into a generator of tasks with certified cooperation requirements. Experiments with five multi-agent reinforcement learning algorithms show that training diversity improves joint success on unseen tasks when cooperation-free solutions exist. When cooperation is required, greater diversity improves individual-agent exits, but joint success remains near zero. Across five profile-certified pools, final exit rates averaged over algorithms separate the pools into four statistically distinguishable levels but this ordering primarily reflects partial completion: policies collect rewards for individual exits but rarely exhibit the profile required for joint success. Our framework exposes this gap between rewarded partial completion and realized cooperation by certifying what cooperation successful completion requires and using temporal cooperation graphs to reveal what policies exhibit.
• 中文摘要 共享奖励为代理提供了共同目标，但何时、如何，甚至是否必须合作才能成功，则保持悬而未决。我们在激光学习环境中解决这些问题，这是一个多代理路径寻找环境，合作表现为一个代理阻挡激光，让队友安全通过。我们通过时间合作图表示这些互动，其时序边连接助手与受益者，定义六个合作组合作为重叠的图谓词，并证明每个合作轨迹至少满足一个。通过将环境动态和配置谓词编码为命题公式，我们区分了允许某一胜利轨迹中某轮廓的任务与在指定视野内每个获胜轨迹中都需要该曲线的任务。作为过滤器，这些查询将随机布局采样器转化为具有认证合作要求的任务生成器。五个多智能体强化学习算法的实验表明，当存在无合作解决方案时，训练多样性能提升在未见任务上的联合成功率。当需要合作时，更高的多样性能改善个体-智能体退出，但联合成功率仍接近零。在五个经过配置文件认证的池中，算法平均的最终退出率将池分为四个统计上可区分的层级，但这种排序主要反映部分完成：策略为单个退出收集奖励，但很少表现出共同成功的特征。我们的框架通过认证成功完成所需的合作，并利用时间合作图揭示策略表现，揭示了奖励部分完成与实现合作之间的差距。

Inducing Emergent Misalignment from Reward Hacks with Iterative DPO

通过迭代DPO诱发奖励黑客中的涌现错位

• Authors: Oliver Daniels, Perusha Moodley, Benjamin M. Marlin, David Lindner
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.06649
• Pdf link: https://arxiv.org/pdf/2609.06649
• Abstract Reward hacking during reinforcement learning from verifiable rewards (RLVR) can induce reward seeking and broad misalignment in language models. Studying this misgeneralization is important for developing better threat models and countermeasures, but is often infeasible due to the cost of RL on large models. As an alternative, we propose studying emergent misalignment from iterative DPO, which preserves important properties of RLVR while reducing costs and enabling training on popular finetuning APIs. In practice, we find that training GPT-4.1 with iterative DPO on a single-turn reward hacking environment induces covert misaligned power-seeking and alignment faking, the first openly available (semi)-online training pipeline to induce these concerning forms of misalignment. We also find that training Qwen2.5-32B-Instruct with the same pipeline induces both misalignment and improved instruction following accuracy, showing that iterative DPO can be used as a testbed for selective generalization. Overall, we think iterative DPO can help democratize and accelerate the study of emergent misalignment from RLVR.
• 中文摘要 在可验证奖励强化学习（RLVR）中的奖励黑客可能诱导语言模型中的奖励寻求和广泛错位。研究这种误概化对于开发更好的威胁模型和对策很重要，但由于大型模型中强化学习的成本，通常难以实现。作为替代方案，我们建议研究迭代DPO中的涌现错位，这在保持RLVR重要特性的同时降低成本，并支持在流行的微调API上进行训练。实际上，我们发现在单回合奖励黑客环境中用迭代DPO训练GPT-4.1，会诱发隐性错位权力寻求和对齐伪造，这是首个公开（半）在线的训练流水线，诱导这些令人担忧的错位形式。我们还发现，使用相同流水线训练Qwen2.5-32B-Instruct既能引起错位，也能提高指令跟随准确性，表明迭代DPO可作为选择性泛化的试验平台。总体而言，我们认为迭代DPO有助于民主化并加速RLVR中涌现错位的研究。

PARSER: Read in Parallel, Reason in Depth for Long-Context LLM Agents

PARSER：并行阅读，长上下文LLM代理的深度解析

• Authors: Kun Li, Zexuan Qiu, Tianhua Zhang, Irwin King, Helen Meng
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.06702
• Pdf link: https://arxiv.org/pdf/2609.06702
• Abstract Sequential memory agents process long documents by reading chunks one after another while maintaining a compact memory state, coupling document traversal to reasoning depth. This coupling introduces sensitivity to evidence placement and ties inference latency linearly to document length. We introduce PARSER, which decouples reading from reasoning. A bank of lightweight subagents each bound to a single chunk read the entire document in parallel, while a lead agent reasons in depth through iterative scatter--gather rounds: at each round it broadcasts a query to all subagents, aggregates the returned evidence, and formulates a deeper follow-up query conditioned on what has been found so far. This decoupled design concentrates all learnable behavior in the lead agent, which is optimized with reinforcement learning, while the subagents remain frozen off-the-shelf models. On multi-hop QA with contexts ranging from 7K to 896K tokens, PARSER with a 4B backbone outperforms the strongest sequential memory baseline by 5.7 points on average and by 12.0 points at 896K tokens. Scaling to a 9B backbone, PARSER surpasses DeepSeek-V4-Pro by 6.3 points. Controlled experiments confirm that PARSER is robust to perturbations in evidence position, order, and distance, conditions that cause large accuracy swings in sequential methods, while reducing inference latency by up to 11x.
• 中文摘要 顺序记忆代理通过连续读取块来处理长文档，同时保持紧凑的内存状态，将文档遍历与推理深度耦合。这种耦合引入了对证据放置的敏感性，并将推理延迟线性地与文档长度挂钩。我们引入了解析分析（PARSER），它将阅读与推理解耦。一组轻量级子代理每个绑定到一个区块，并行读取整个文档，而主导代理通过迭代分散进行深入推理——收集轮次：每轮向所有子代理广播查询，汇总返回的证据，并基于迄今为止发现的内容制定更深层次的后续查询。这种解耦设计将所有可学习行为集中在主代理中，并通过强化学习进行优化，而子代理则保持冻结的现成模型。在多跳质询（QA），上下文范围为7K至896K，4B骨干的PARSER平均比最强的顺序记忆基线高出5.7分，896K令牌时高出12.0分。扩展到9B骨干网时，PARSER比DeepSeek-V4-Pro高出6.3个百分点。受控实验证实，PARSER对证据位置、顺序和距离的扰动具有鲁棒性，这些条件会导致顺序方法的准确性大幅波动，同时推断延迟可降低多达11倍。

SkillX: Unified Multi-Skill Policy Learning for Humanoid Soccer

SkillX：人形足球统一多技能政策学习

• Authors: Zhangchen Ye, Enxuan Ruan, Yifei Bao, Runhan Huang, Jiankun Yang, Jiakang Jin, Yixiao Huo, Pengyuan Wang, Yinan Han, Huaxing Huang, Wenhao Cui, Yiming Li, Xiaoyu Tian
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.06718
• Pdf link: https://arxiv.org/pdf/2609.06718
• Abstract Humanoid soccer is a challenging testbed for dynamic whole-body control, requiring robots to coordinate balance, locomotion, object interaction, and skill switching over long horizons. Existing humanoid sports methods often rely on task-specific multi-stage pipelines, making it difficult to jointly learn and compose multiple object-interactive skills within a single deployable policy. To address this, we present SkillX, a unified reinforcement learning framework that learns and composes multiple atomic soccer skills through a single command-conditioned policy. SkillX integrates three core designs: skill-specific adversarial motion priors, skill-specific critics, and an object-aware temporal encoder, enabling the robot to execute atomic skills and transition among them such as dribbling, trapping, and shooting. Experiments in simulation and on a real Noetix E1 humanoid demonstrate robust multi-skill execution, long-horizon skill composition, and successful sim-to-real deployment.
• 中文摘要 类人足球是动态全身控制的挑战性测试平台，要求机器人在较长的视野内协调平衡、移动、物体交互和技能切换。现有的人形运动方法通常依赖任务特定的多阶段流程，使得在单一可部署策略中共同学习和组合多种对象互动技能变得困难。为此，我们提出了SkillX，一个统一的强化学习框架，通过单一指令条件策略学习和组合多种原子足球技能。SkillX集成了三大核心设计：技能专属对抗动作先验、技能专属批评者和对象感知时间编码器，使机器人能够执行原子技能并在它们之间转换，如盘带、夹击和射门。模拟实验和在真实的Noetix E1人形机器人上进行了实验，展示了多技能的稳健执行、长视野技能组合以及成功的模拟到真实部署。

Agentic Visual Generation: From Generative Models to Agentic Control

智能视觉生成：从生成模型到智能控制

• Authors: Yinming Huang, Shuyuan Tu, Xi Yan, Jiahao Zhan, Zihan Yang, Zhen Xing, Hui Zhang, Tiehua Zhang, Yu-Gang Jiang, Zuxuan Wu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.06758
• Pdf link: https://arxiv.org/pdf/2609.06758
• Abstract Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.
• 中文摘要 视觉生成正从通过单次调用使用的生成模型，发展为能够规划、选择工具、检查中间综合输出、修正失败并重用先前经验的智能控制过程。在大多数现有系统中，控制器是大型语言模型（LLM）或虚拟模型（VLM），而视觉生成模型则作为工具或执行者。然而，现有工作缺乏一个一致的标准来判断生成系统何时成为智能化。规划深度、工具使用、多角色协作和强化学习常被视为智能性的证据，尽管这些都不一定决定控制器可以做出哪些生成决策。我们根据控制器在生成过程中能直接控制的内容来组织场域。在L1条件控制中，控制器准备了预设生成器的输入，但不控制执行哪种视觉操作。在L2执行控制中，它选择并调用实际生成、编辑、渲染或其他内容修改操作。在L3结果自适应控制中，它观察到一个中间结果，并利用该观察改变当前任务中的后续操作。在L4体验自适应控制中，它保留已完成任务的经验，并利用这些经验改变未来任务的决策。L0固定支持分别指代生成器、编辑器、评估器、奖励模型、基准测试和固定流水线，这些工具没有部署的控制器，无法做出世代级决策。这些层级描述的决策范围逐渐扩大，而非模型规模、系统复杂度、输出质量、工具或角色数量或训练方法。将该框架应用于图像、视频、编辑、3D、世界、幻灯片和用户界面生成，揭示了控制器能力如何演变及其机制如何分布于各级。

Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning

离线强化学习中扩散策略的噪声空间策略梯度

• Authors: Mahmoud Selim, Cristina Cipriani, Karl H. Johansson
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.06882
• Pdf link: https://arxiv.org/pdf/2609.06882
• Abstract Diffusion policies offer a powerful and expressive parameterization for continuous control. Yet, their integration with reinforcement learning remains conceptually and algorithmically challenging. In this work, we address this gap by introducing a noisy-space action-value (Q-)function that assigns values to diffusion latents through the distribution of executed actions induced by the denoising process. We show that this construction admits a precise semantic interpretation and derive a noisy-space policy gradient (NSPG) that optimizes noisy latents using only clean action-space value estimates. Building on this result, we formulate a KL-regularized policy improvement over noisy latents and show that the resulting objective admits a diffusion-compatible regression form, avoiding backpropagation through the denoising process. Empirical results on state-based D4RL benchmarks and vision-based OGBench tasks demonstrate that the proposed noisy-space objective provides a principled and effective basis for training diffusion policies in offline reinforcement learning. Project webpage: this https URL
• 中文摘要 扩散策略为连续控制提供了强大且富有表现力的参数化。然而，它们与强化学习的整合在概念和算法上仍具有挑战性。在本研究中，我们通过引入一个噪声空间动作值（Q-）函数，通过去噪过程诱导的执行动作分布，为扩散潜变量赋予值来弥补这一空白。我们证明该构造允许精确的语义解释，并推导出噪声空间策略梯度（NSPG），仅用干净的动作空间值估计来优化噪声潜在变量。基于此结果，我们对噪声潜在变量提出了 KL 正则化策略改进，并证明所得目标具有扩散兼容的回归形式，避免了通过去噪过程的反向传播。基于状态的D4RL基准测试和基于视觉的OGBench任务的实证结果表明，所提出的噪声空间目标为离线强化学习中的扩散策略训练提供了原则性和有效基础。项目网页：此 https URL

Distributed Secure Learning Control for Large-scale Multirobots under Stealthy Actuator Attacks

针对大规模多机器人在隐形执行器攻击下的分布式安全学习控制

• Authors: Xinglong Zhang, Qingwen Ma, Cong Li, Hui Yin, Changxin Zhang, Yueying Wang, Wei Pan, Xin Xu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.06896
• Pdf link: https://arxiv.org/pdf/2609.06896
• Abstract Distributed learning control for multirobot systems (MRS) offers significant flexibility in presence of uncertainties but lacks provable performance guarantees. A promising direction involves integrating reinforcement learning (RL) into distributed model predictive control (DMPC), leveraging the strengths of RL in nonlinear policy design and the receding-horizon replanning capabilities of DMPC. However, ensuring secure control within such a learning framework under malicious cyber attacks, particularly stealthy ones, remains a critical challenge, because the distributed policies generation depends on information exchange among neighbors, where compromised agents can rapidly influence the behavior of others through the communication network. This article proposes a distributed secure learning control (DSLC) framework for large-scale MRS under malicious, stealthy actuator attacks. Our framework offers two key features: (i) a unified approach that enables secure learning control across various coordination scenarios and (ii) a game-theoretic distributed learning-based predictive control strategy that learns how to balance the attacker and defender through a differential-game based DMPC framework. Specifically, DSLC employs a distributed attacker-actor-critic architecture to learn the optimal defense and attack policies online within each prediction interval. Unlike numerical optimization-based controllers that calculate open-loop control sequences, our method simultaneously generates adversarial attack policies and corresponding defense policies in analytical closed-loop form. The defense policies could be directly generalized to MRS with varying scales and diverse actuator attack probabilities. The effectiveness and scalability of DSLC are validated through comprehensive simulations and real-world experiments in multiple wheeled robots via various control tasks.
• 中文摘要 多机器人系统（MRS）的分布式学习控制在存在不确定性时提供了显著灵活性，但缺乏可验证的性能保证。一个有前景的方向是将强化学习（RL）整合进分布式模型预测控制（DMPC），利用强化学习在非线性策略设计中的优势和DMPC的远征视野重新规划能力。然而，在恶意网络攻击，尤其是隐形攻击下，确保在此类学习框架内实现安全控制仍是一个关键挑战，因为分布式策略的生成依赖于邻居间的信息交换，而被攻破的智能体可以通过通信网络迅速影响他人行为。本文提出了一个针对大规模MRS在恶意隐蔽执行器攻击下进行的分布式安全学习控制（DSLC）框架。我们的框架提供两个关键特性：（i）实现跨多种协调场景的安全学习控制的统一方法;（ii）基于博弈论的分布式学习预测控制策略，通过基于差分博弈的DMPC框架学习如何平衡攻击者和防御者。具体来说，DSLC采用分布式攻击者-行为者-批判者架构，在线学习每个预测区间内的最优防御和攻击策略。与计算开环控制序列的数值优化控制器不同，我们的方法同时生成对抗攻击策略和相应的分析闭环防御策略。防御策略可直接推广至MRS，具有不同尺度和不同执行器攻击概率。DSLC的有效性和可扩展性通过多轮机器人的综合模拟和实际实验验证，涵盖各种控制任务。

CARDEA: Auditable Reasoning Grounded in Spatial Evidence for End-to-End Coronary Angiography Interpretation

CARDEA：基于空间证据的可听性推理，用于端到端冠状动脉造影解读

• Authors: Jia-Jen Lee, Shih-Yen Hou, Kee Koon Ng, Wei-Chun Wang, Shih-Sheng Chang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.06931
• Pdf link: https://arxiv.org/pdf/2609.06931
• Abstract Invasive coronary angiography (CAG) is the gold standard for diagnosing coronary artery disease, but interpretation varies substantially among observers. Existing AI systems can improve consistency but lack auditable decision processes and are limited in comprehensive open-ended assessment, undermining clinician trust and clinical adoption readiness. We developed CARDEA, a unified large vision-language model that serves as the inference core of a CAG pipeline. It was trained solely on public datasets and closed-ended tasks in three stages: visual feature alignment, a self-distilled Chain-of-Box (CoB) cold start, and reinforcement learning with verifiable rewards (RLVR) with a CoB reward encouraging bounding-box use in the reasoning trace. We assessed its two study-level diagnoses, dominance classification and complexity assessment, against a dedicated classifier and two interventional cardiologists. Report generation was excluded from training and evaluated zero-shot across stages on an external cohort using vessel-severity macro-$F_1$. CARDEA trailed the classifier on in-distribution dominance but drew level under domain shift (accuracy, 0.91 [95% confidence interval (CI), 0.86 to 0.95]) and was comparable to the cardiologists on complexity assessment (accuracy, 0.90 [CI, 0.82 to 0.97]). Only RLVR improved zero-shot report generation, raising its vessel-severity macro-$F_1$ (0.686 [CI, 0.664 to 0.707]) above the untuned base model (0.513) and over twice the always-normal floor (0.312). CARDEA runs an end-to-end CAG pipeline from raw multi-view videos through keyframe selection to study-level diagnosis while exposing auditable spatial evidence behind its conclusions. RLVR on verifiable closed-ended tasks surfaced open-ended reporting ability that supervised imitation did not. Clinical use requires prospective validation against expert cardiologists.
• 中文摘要 侵入性冠状动脉造影（CAG）是诊断冠状动脉疾病的黄金标准，但不同观察者的解读差异较大。现有AI系统可以提高一致性，但缺乏可审计的决策流程，且在全面的开放式评估方面受限，削弱了临床医生的信任和临床采纳准备度。我们开发了CARDEA，一个统一的大型视觉语言模型，作为CAG流程的推断核心。该模型仅基于公开数据集和封闭任务进行训练，分为三个阶段：视觉特征对齐、自我蒸馏的盒链（CoB）冷启动，以及带有可验证奖励的强化学习（RLVR），并以CoB奖励鼓励在推理痕迹中使用边界框。我们将其两个研究级诊断——优势分类和复杂度评估——与一名专门分类器和两名介入心脏病专家进行比较。报告生成被排除在训练之外，采用血管严重度宏$F_1$在外部队列中各阶段评估零样本。CARDEA在分布内优势性上落后分类器，但在领域偏移下保持水平（准确率0.91[95%置信区间，0.86至0.95]），在复杂度评估方面与心脏病学家相当（准确率0.90 [置信区间，0.82至0.97]）。只有RLVR改进了零样本报告生成，其血管严重度宏$F_1$（0.686[置信区间，0.664至0.707]）高于未调优基础模型（0.513），且是始终正常底线（0.312）的两倍以上。CARDEA运行从原始多视图视频到关键帧选择再到研究层级诊断的端到端CAG流程，同时揭示结论背后的可审计空间证据。可验证的封闭式任务中RLVR展现了开放式报告能力，而监督模仿则未达到。临床应用需要对专业心脏病学家进行前瞻性验证。

Mind the Phase: Effective Rank and Representation Health in Legged Locomotion

注意阶段：腿行走中有效的等级与表现健康

• Authors: Felipe Tommaselli, Thiago H. Segreto, Juliano D. Negri, Ricardo V. Godoy, Marcelo Becker
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.06958
• Pdf link: https://arxiv.org/pdf/2609.06958
• Abstract Reinforcement learning has become the leading paradigm in legged locomotion, enabling complex behaviors from backflips to parkour through massively parallel simulation. Under PPO's non-stationarity, shallow networks remain the de facto architecture, supported by carefully staged curricula and environments, yet the representations these policies learn stay poorly understood, leaving no training-time signal of how they will behave on hardware. In this work, we empirically study locomotion policies through the effective rank of the policy Jacobian and show that conditioning rank on the gait phase exposes architectural structure that global rank averages away. In particular, we find that standard architectural choices, namely layer normalization and residual connections, allocate roughly two more dimensions of effective rank to swing than to stance, which is fully absent in vanilla MLPs. Building on this, we propose a simple recipe that turns these representational signatures into smoother, more reliable sim-to-real transfer. In practice, this results in roughly 3x lower joint jitter that holds from simulation onto a physical Spot, suggesting that representation health is an effective training-time lens to track sim-to-real smoothness.
• 中文摘要 强化学习已成为腿部运动的主导范式，使从后空翻到跑酷等复杂行为能够通过大规模并行模拟实现。在PPO的非平稳性下，浅层网络仍然是事实上的架构，由精心分阶段的课程和环境支持，但这些策略学习到的表征仍然不够理解，没有留下它们在硬件上表现的训练时间信号。本研究通过策略雅可比矩阵的有效秩实证研究运动策略，并证明步态阶段的条件排序揭示了全局秩平均消除的架构结构。特别是，我们发现标准架构选择，即层归一化和残差连接，为挥杆分配了大约两个有效等级维度，而站姿则在原版MLP中完全不存在。基于此，我们提出了一个简单的配方，将这些表示签名转化为更平滑、更可靠的模拟到现实传输。在实际操作中，这导致从模拟到物理Spot的关节抖动约降低3倍，表明表示健康是追踪模拟到现实平滑度的有效训练时间透镜。

TrojanWorld: Backdooring World-Model Agents via Imagination Steering

TrojanWorld：通过想象力引导后门世界模型代理

• Authors: Wenkai Huang, Siyuan Liang, Gaolei Li, Yiming Li, Tianhao Peng, Jianhua Li, Dacheng Tao
• Subjects: Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2609.07051
• Pdf link: https://arxiv.org/pdf/2609.07051
• Abstract World models increasingly serve as the predictive core of model-based reinforcement learning agents, enabling them to simulate future dynamics and reason over imagined trajectories before acting. Their substantial training demands make pretrained world models attractive for distribution and reuse, exposing downstream systems to model supply chain threats. Backdoor attacks offer a targeted and stealthy means of exploiting such supply chains, yet their threat to interactive world-model agents remains largely unexplored. To fill this gap, we present TrojanWorld, a backdoor framework for world-model agents that induces attacker-specified behavior by steering internal imagination. A physical object placed in the scene acts as the trigger, enabling deployment-time activation through the agent's native observation pipeline without digitally manipulating the observation stream. To achieve effective, stealthy, and persistent control, TrojanWorld combines Decision-Reflective Induction to steer trigger-conditioned imagination toward attacker-specified actions using decision feedback, Clean Behavior Anchoring to preserve trigger-free predictive and behavioral fidelity, and Causal Propagation to sustain the induced preference along subsequent trajectories after the trigger disappears. Together, these mechanisms establish an end-to-end attack chain from physical perception through corrupted imagination to malicious action selection. Experiments with the TD-MPC2, DreamerV3, and R2-Dreamer systems across the DeepMind Control, MetaWorld, MyoSuite, and RoboDesk benchmarks show that under trigger activation, TrojanWorld achieves a target-action deviation as low as 0.026 while retaining at least 98.8% of the corresponding clean performance. Even after trigger removal, the compromised agent can remain trapped in the induced behavioral trajectory, continuing to execute attacker-specified actions.
• 中文摘要 世界模型日益成为基于模型的强化学习代理的预测核心，使其能够模拟未来动态，并在行动前推理想象轨迹。其庞大的训练需求使预训练世界模型成为分发和重用的吸引力，使下游系统暴露于模型供应链威胁之下。后门攻击提供了一种有针对性且隐蔽的利用此类供应链手段，但其对交互式世界模型代理的威胁仍大多未被探索。为填补这一空白，我们提出了TrojanWorld，一个面向世界模型代理的后门框架，通过引导内部想象力诱导攻击者指定行为。场景中放置的物理对象作为触发器，使部署时间激活能够通过智能体的原生观察流水线实现，而无需数字操作观察流。为了实现有效、隐蔽且持久的控制，TrojanWorld 结合了决策反思归纳（Decision-Reflective Induction），通过决策反馈引导触发条件下的想象力朝向攻击者指定的行动;清洁行为锚定（Clean Behavior Anchoring）以保持无触发的预测和行为准确性，以及因果传播（Causal Propagation）以维持诱导偏好沿触发消失后的后续轨迹。这些机制共同建立了从物理感知到腐化想象再到恶意动作选择的端到端攻击链。对 TD-MPC2、DreamerV3 和 R2-Dreamer 系统在 DeepMind Control、MetaWorld、MyoSuite 和 RoboDesk 基准测试中的实验显示，在触发激活下，TrojanWorld 在目标-动作偏差低至 0.026 的同时，至少保持了相应的 98.8% 的干净性能。即使触发器被移除，受感染的智能体仍可能被困在诱导的行为轨迹中，继续执行攻击者指定的动作。

SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting

SupGRPO：通过基于匹配的在线SFT增强GRPO文本检测

• Authors: Xudong Xie, Yuzhe Li, Jing Shi, Zhifei Zhang, Curtis Wigington, Zhaowen Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.07081
• Pdf link: https://arxiv.org/pdf/2609.07081
• Abstract Text spotting requires both accurate text recognition and precise spatial localization. Current specialised spotters excel at predicting tight bounding boxes in natural scenes, but falter on complex or artistic text, whereas multimodal large language models (MLLMs) possess strong recognition capabilities yet remain weak at localisation. To equip the text spotter with general and powerful recognition capabilities and to maximize its localization ability, we explore two MLLM-based fine-tuning methods: Supervised Fine-Tuning (SFT) and reinforcement learning fine-tuning based on Group Relative Policy Optimisation (GRPO). An interesting finding is that SFT is less effective than GRPO at enhancing recognition, while GRPO is less effective than SFT at enhancing detection. To compensate for each other's shortcomings, we introduce a joint training strategy, SupGRPO, which simultaneously optimizes the model using both SFT and GRPO. SupGRPO employs the specially designed reward functions and develops a matching-based online SFT applied solely to coordinate tokens. It both mitigates the reward sparsity problem of GRPO and avoids the instance order dependency problem of SFT. To evaluate particularly challenging cases, we curate ATS, a dataset for artistic text spotting. Experiments demonstrate that SupGRPO improves both text recognition and detection, and attains superior performance. Our code and dataset will be released at this https URL.
• 中文摘要 文本定位既需要准确的文本识别，也需要精确的空间定位。目前的专业观察器在预测自然场景中的紧密边界框方面表现出色，但在复杂或艺术性文本上表现不佳，而多模态大型语言模型（MLLM）虽然具备强大的识别能力，但在定位方面仍然较弱。为了赋予文本识别器通用且强大的识别能力并最大化其定位能力，我们探索了两种基于MLLM的微调方法：监督微调（SFT）和基于群相对策略优化（GRPO）的强化学习微调。一个有趣的发现是，SFT在提升识别方面不如GRPO，而GRPO在增强检测方面不如SFT。为弥补彼此的不足，我们引入了联合训练策略SupGRPO，同时利用SFT和GRPO优化模型。SupGRPO采用专门设计的奖励函数，开发了基于匹配的在线SFT，仅用于协调代币。它既缓解了GRPO的奖励稀疏性问题，也避免了SFT的实例顺序依赖问题。为评估特别复杂的案例，我们策划了ATS，一个用于艺术文本发现的数据集。实验表明，SupGRPO不仅提升了文本识别和检测能力，还实现了更优越的性能。我们的代码和数据集将发布于该 https URL。

Revisiting Complete Reasoning Traces for Post-Training

重新审视完整的推理痕迹以备培训后

• Authors: Jaehui Hwang, Sangdoo Yun, Byeongho Heo, Dongyoon Han
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.07103
• Pdf link: https://arxiv.org/pdf/2609.07103
• Abstract Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability. Such trajectories tend to be long due to complex, interwoven paths, which often include detours on the path toward the answer. However, it has been underexplored whether LLMs indeed benefit from learning complete trajectories in post-training, such as supervised fine-tuning (SFT). Starting from our pilot study, we find that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation. We analyze redundancy in reasoning trajectories through attention-based analyses and controlled token-removal studies, both of which show that intermediate tokens contribute minimally to final reasoning quality. This suggests that avoiding redundant information may allow LLMs to internally infer coherent alternatives by inferring missing steps from their internal knowledge, given known trajectory endpoints. Furthermore, we show that training LLMs using endpoints leads to consistent changes in reasoning behavior, and that it also benefits post-training methods based on reinforcement learning or on-policy distillation, highlighting the need to revisit complete reasoning traces. Code is available at this https URL.
• 中文摘要 大型语言模型（LLMs）通常通过预先收集的推理轨迹进行后训练，以提升推理能力。由于路径复杂且交织，这些路径常常包含通往答案的路径绕行，这些轨迹往往较长。然而，关于LLM是否真的从后训练中学习完整轨迹（如监督微调（SFT）中受益，尚未被充分探讨。从试点研究开始，我们发现完整轨迹的益处有限，而部分轨迹即使在重度截断下仍然有效。我们通过基于注意力的分析和受控的代币去除研究分析推理轨迹的冗余性，这两者都表明中间符号对最终推理质量贡献极小。这表明避免冗余信息可能使LLM能够通过内部知识推断缺失步骤，在已知的轨迹端点下推断出连贯的替代方案。此外，我们表明使用端点训练LLM会带来推理行为的一致变化，同时也有利于基于强化学习或策略提炼的训练后方法，凸显了重新审视完整推理痕迹的必要性。代码可在此 https URL 获取。

Beyond Sparse Rewards: A New Benchmark and Structure-Aware Graph Alignment for Micro-Drama Understanding

超越稀疏奖励：微剧理解的新基准与结构感知图对齐

• Authors: Yixin Qin, Shi-Zhe Chen, Zhiqi Yu, Siyuan Cheng, Tao Cheng, Jinwen Luo, Zheng Wei
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.07107
• Pdf link: https://arxiv.org/pdf/2609.07107
• Abstract Micro-dramas, characterized by ultra-short durations and hyper-dense storylines, pose unique challenges for video understanding that conventional benchmarks fail to address. To bridge this gap, we introduce M-Drama, the first large-scale bilingual benchmark for micro-drama comprehension, featuring over 35K instances across 9,138 clips. Furthermore, while reinforcement learning can enhance VLMs on complex narratives, existing reward metrics often suffer from sparse and superficial signals, failing to capture intricate character identities and temporal structures. We propose SAGA (Structure-Aware Graph Alignment), a novel graph-matching reward function that models narratives as heterogeneous graphs. SAGA computes dense, rigorous rewards via decoupled semantic triplet and structural temporal matching. Extensive experiments on Qwen3-VL-8B-Instruct demonstrate that SAGA outperforms existing baselines, delivering substantial improvements in open-ended accuracy and summary quality, while maintaining competitive out-of-domain generalization. Code is available at this https URL.
• 中文摘要 微型剧集以超短时长和超密集剧情为特征，给视频理解带来了传统基准难以解决的独特挑战。为弥合这一差距，我们引入了M-Drama，这是首个大规模双语微剧理解基准，包含超过3.5万个实例，涵盖9,138个片段。此外，虽然强化学习可以增强复杂叙事的VLM，但现有奖励指标常常存在稀疏和表面信号，未能捕捉复杂的角色身份和时间结构。我们提出了SAGA（结构感知图对齐），这是一种新颖的图匹配奖励函数，将叙事建模为异构图。SAGA通过解耦语义三元组和结构性时间匹配计算密集且严谨的奖励。在Qwen3-VL-8B-Instruct上的大量实验表明，SAGA优于现有基线，在开放式准确性和摘要质量方面有显著提升，同时保持了竞争性的域外泛化能力。代码可在此 https 网址获取。

Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training

大规模、长上下文强化学习后期推测解码的在线草稿共训

• Authors: Zili Wang, Zhaopeng Qiu, Yuekai Zhang, Shuang Yu, Junjie Lai
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.07108
• Pdf link: https://arxiv.org/pdf/2609.07108
• Abstract Speculative decoding accelerates rollout generation, which dominates the cost of reinforcement learning (RL) post-training. Online co-training can further increase the draft's accuracy, yielding greater speedups. However, scaling this approach to co-training on large models with long contexts poses two obstacles: (1) branch attention is unsupported by standard causal context-parallel (CP) implementations, and (2) target features span across pipeline-parallel (PP) stages. We address both with an end-to-end system for large-scale online draft co-training. For CP, we extend packed, load-balanced zigzag ring attention by merging rank-local branch attention with causal main-sequence attention. For PP, TapChannel transports intermediate target features across stages via a separate path, leaving the pipeline schedule unaffected. Experiments demonstrate that co-trained drafts closely track the policy baseline while delivering substantial rollout and end-to-end speedups across model scales up to 122B. Our CP design achieves strong scaling at 256K tokens with significant memory savings over prior work, and our PP transport incurs modest overhead. Code can be found at this https URL.
• 中文摘要 推测解码加速了推广生成，而这主导了训练后强化学习（RL）的成本。在线协同训练还能进一步提高草稿的准确性，带来更大的加速。然而，在具有长上下文的大型模型上将这种方法扩展到共训练存在两个障碍：（1） 分支注意力不被标准因果上下文并行（CP）实现支持，（2） 目标特征跨越流水线并行（PP）阶段。我们通过端到端系统应对了这两点，用于大规模在线草稿协同训练。对于CP，我们通过合并排名局部分支注意力与因果主序关注，扩展了填充且负载均衡的之字形环注意力。对于PP，TapChannel通过独立路径跨阶段传输中间目标特征，保持流水线调度不受影响。实验表明，协同训练的草案紧密跟踪政策基线，同时在模型尺度最高可达122B的范围内实现显著的推广和端到端加速。我们的CP设计在256K令牌下实现了强力扩展，且比之前工作节省了显著的内存，PP传输开销适中。代码可在此 https URL 找到。

From LLM-Generated Specifications to Learned Quadruped Locomotion

从LLM生成的规范到学习的四足行走

• Authors: Merve Atasever, Keyan Azbijari, Cagan Bakirci, Alfredo Reina Corona, Tolga Izdas, Richard Yang, Erdem Biyik, Jyotirmoy V. Deshmukh
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.07111
• Pdf link: https://arxiv.org/pdf/2609.07111
• Abstract Quadruped robot locomotion policies are often trained using reinforcement learning, which in turn relies heavily on hand-crafted reward functions. Designing reward functions requires substantial manual engineering, and it is often unclear which local rewards will induce the desired global behavior. Shaped rewards from formal specifications in languages like Signal Temporal Logic (STL) can make rewards more interpretable, but writing STL specifications itself still requires domain expertise. We study whether large language models (LLMs) can fill this gap by generating Parametric Signal Temporal Logic (PSTL) specifications that are subsequently used for policy learning. Given a natural language locomotion objective and a constrained specification grammar, GPT-5.5 and Qwen 3.6 independently propose STL templates for command tracking, safety, and gait structure. We instantiate the parameters of the generated PSTL templates using expert trajectories and retain only specifications that are consistent with demonstrated expert behavior. The resulting specifications are then transformed into smooth, finite-history reward functions and used to train a quadruped locomotion policy with Proximal Policy Optimization (PPO) in MuJoCo XLA (MJX). We evaluate both \emph{gait-aware} and \emph{gait-agnostic} settings. The former specifies walking-trot, trot, and bound regimes, while the latter allows contact patterns to emerge from the task objective. We compare against hand-engineered rewards, Text2Reward-style LLM-generated reward code, and an expert-switching oracle. Gait-aware Qwen 3.6 specifications achieved 100\% survival and command success across all tested speeds (0.3--2.1 m/s) and matched the target gait at high speeds, whereas Text2Reward achieved 0\% for both metrics at $\geq 1.9$ m/s. Videos: this https URL
• 中文摘要 四足机器人运动策略通常通过强化学习进行训练，而这又大量依赖手工制作的奖励函数。设计奖励函数需要大量手工工程，且常常不清楚哪些局部奖励能诱发期望的全局行为。通过信号时间逻辑（STL）等形式规范生成的形态奖励可以使奖励更易理解，但编写STL规范本身仍需领域专业知识。我们研究大型语言模型（LLMs）是否能通过生成参数化信号时序逻辑（PSTL）规范来填补这一空白，这些规范随后用于策略学习。在自然语言运动目标和受限规范语法的情况下，GPT-5.5和Qwen 3.6独立提出了用于指令追踪、安全和步态结构的STL模板。我们用专家轨迹实例化生成的PSTL模板参数，只保留与专家行为一致的规格。所得规格随后转化为平滑的有限历史奖励函数，用于在MuJoCo XLA（MJX）中用近端策略优化（PPO）训练四足行走策略。我们评估了\emph{gait-aware}和\emph{gait-igndependent}两种设置。前者指定步行小跑、小跑和界限模式，后者允许从任务目标中产生接触模式。我们与手工设计的奖励、Text2Reward风格的LLM生成奖励代码以及专家切换预言机进行比较。步态感知Qwen 3.6规范在所有测试速度（0.3至2.1 m/s）下实现了100%的生存率和指令成功率，并在高速下匹配目标步态，而Text2Reward在$\geq 1.9$ m/s时，在这两个指标上均为0\%。视频链接：此链接

Flow3D-OPD: Multi-Teacher On-Policy Distillation for 3D Geometry Generation with Flow-Matching Diffusion Transformer

Flow3D-OPD：多教师策略蒸馏，用于使用流量匹配扩散变换器生成三维几何

• Authors: Zhiwei Ning, Zhen Zhou, Puhua Jiang, Xintong Han, Gengming Zhang, Jie Yang, Zhonglong Zheng, Yuanjie Zheng, Wei Liu, Chunchao Guo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.07137
• Pdf link: https://arxiv.org/pdf/2609.07137
• Abstract Recent image-to-3D generation models built on flow-matching diffusion Transformers (DiT) can produce high-fidelity meshes, yet their post-training strategy remains largely unexplored. There exist several critical bottlenecks in reinforcement learning: the inherent difficulty of defining comprehensive rewards for 3D geometric quality, and the gradient interference that arises when jointly optimizing heterogeneous objectives. Inspired by the practicability of on-policy distillation (OPD) in large language models and image generation, we propose \textbf{Flow3D-OPD}, a two-stage post-training framework that introduces multi-teacher distillation into 3D geometry generation. In the first stage, we utilize the semi-policy to enhance the foundational capability of the pretrained model and then design an agentic verifier for 3D geometric quality evaluation. Based on the verifier, we could cultivate domain-specialized teacher models via direct preference optimization (DPO). In the second stage, we consolidate heterogeneous expertise into a unified student model through on-policy distillation with hard task-routing sampling and gradient accumulation, which could mitigate the gradient interference in joint optimization. Without relying on elaborate modifications, our straightforward yet effective design achieves consistent improvements across all geometric quality dimensions and surpasses all teacher models in the average metric. Extensive experiments demonstrate that our approach provides an effective paradigm for reinforcement learning in 3D generation.
• 中文摘要 基于流匹配扩散变换器（DiT）构建的最新图像到三维生成模型能够生成高精度网格，但其训练后策略仍大多未被充分探索。强化学习存在几个关键瓶颈：定义三维几何质量的综合奖励本身就很困难，以及在联合优化异构目标时产生的梯度干扰。受大型语言模型和图像生成中策略提纯（OPD）可行性的启发，我们提出了 \textbf{Flow3D-OPD}，这是一个两阶段的后训练框架，将多教师提炼引入三维几何生成。第一阶段，我们利用半策略增强预训练模型的基础能力，然后设计一个用于三维几何质量评估的代理验证器。基于验证器，我们可以通过直接偏好优化（DPO）培养领域专业化教师模型。第二阶段，我们通过策略上提炼、严格的任务分配抽样和梯度累积，整合异质学生专业知识到统一的学生模型中，这有助于缓解联合优化中的梯度干扰。无需依赖复杂修改，我们简单而高效的设计在所有几何质量维度上实现持续提升，并在平均指标上超越所有教师模型。大量实验表明，我们的方法为三维生成强化学习提供了有效范式。

Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification

Stable-MM-R1：通过熵引导分层锚定多模推理动力学

• Authors: Yimeng Ye, Shuang Chen, Wenxuan Huang, Manyuan Zhang, Kaituo Feng, Zhangquan Chen, Jiayu Chen, Yucheng Zhou, Yicheng Xiao, Zhiyuan Feng, Tianyu Shi
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.07148
• Pdf link: https://arxiv.org/pdf/2609.07148
• Abstract While Reinforcement Learning (RL) effectively incentivizes reasoning in Large Language Models, current pipelines are hindered by training instability and rapid entropy collapse. These limitations often stem from "Rollout Silencing" and low-quality gradient signals in standard sampling procedures. In this work, we propose a robust, data-centric framework to stabilize RL training. We first introduce Potential-Aware Query Mining (PAQM), which filters data dynamically to focus on the "Distillation Zone"---samples with high potential for capability elicitation. Furthermore, we present Hybrid Stratified Replay (HSR), a novel mechanism that restructures batches by stratifying rollouts based on Path Entropy, a rollout-level confidence proxy, and outcome reward. Within each optimization step, HSR reuses current-policy "Stability Anchors" and "Hard Negatives" to construct high-contrast optimization groups, then clears its buffers before the next step. This approach mitigates entropy collapse while improving the utilization of learning signals under limited compute. Our method outperforms strong baselines on complex reasoning tasks, offering a principled solution for stable and efficient RL fine-tuning.
• 中文摘要 虽然强化学习（RL）有效激励大型语言模型中的推理，但当前流水线仍受训练不稳定性和快速熵崩溃的阻碍。这些限制通常源于标准采样程序中的“滚动静默”和低质量梯度信号。本研究提出了一个稳健的数据中心框架以稳定强化学习训练。我们首先引入了潜在感知查询挖掘（PAQM），它动态过滤数据，聚焦于“蒸馏区”---具有高能力诱导潜力的样本。此外，我们提出了混合分层重放（HSR），这是一种新颖机制，通过基于路径熵、一个部署级置信代理和结果奖励来重组批量。在每个优化步骤中，HSR重用当前策略的“稳定锚点”和“硬负”来构建高对比度优化群，然后在下一步前清除缓冲区。该方法减轻了熵坍缩，同时提升了有限计算条件下学习信号的利用率。我们的方法在复杂推理任务中表现优于强基线，提供了稳定高效的强化学习微调的原则性解决方案。

CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards

CircuitLens：推理回路作为强化学习数据选择信号，可验证奖励

• Authors: Zhuofan Chen, Ziqian Jiao, Yikai Cui, Zhixin Cai, Jun Bai, Wenge Rong
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.07183
• Pdf link: https://arxiv.org/pdf/2609.07183
• Abstract Reinforcement learning with verifiable rewards (RLVR) is sensitive to which problems a model trains on, yet existing selection criteria--difficulty filtering, hand-curation, reward-trajectory scoring--assess data value as an intrinsic property of problems, independent of the model that will learn from them. We introduce Circuit Reasoning Score (CRS), a selection signal derived from 46 reasoning-sensitive attention heads identified via contrastive ablation, computed in a single forward pass on the frozen base model without reward labels or rollouts. CRS runs against the intuitive hypothesis that stronger reasoning-circuit engagement produces better training data: on Qwen2.5-Math-7B, the lowest-engagement decile improves over random selection on three medium-difficulty benchmarks (GSM8K +2.0 pp, OlympiadBench +1.6 pp, Minerva +2.9 pp), while the highest-engagement decile gains less and is indistinguishable from the middle decile. The advantage has boundary conditions: on a domain-curated pool no selection method separates from the others; at 1.5B scale the useful direction differs; and the lowest-reward training condition produces the strongest downstream generalization. Within the Qwen2.5-Math settings tested, RLVR data selection appears regime-dependent rather than reducible to a static ranking of problem quality.
• 中文摘要 带有可验证奖励的强化学习（RLVR）对模型训练的问题很敏感，但现有的选择标准——难度过滤、手工策划、奖励轨迹评分——将数据价值作为问题的内在属性评估，独立于将从中学习的模型。我们引入了电路推理评分（CRS），这是一种由46个推理敏感注意力头通过对比消融识别得出的选择信号，在冻结的基础模型上进行一次前向处理计算，无需奖励标签或展开。CRS违背了直觉假设：更强的推理回路参与会产生更好的训练数据：在Qwen2.5-Math-7B上，最低参与度的十分位数在三个中等难度基准测试（GSM8K +2.0 pp，OlympiadBench +1.6 pp，Minerva +2.9 pp）中优于随机选择，而最高参与度的十分位获得的收益较少，且与中间的十分位无异。优势具有边界条件：在领域策划的池中，没有选择方法与其他方法分离;在1.5B尺度下，有用的方向不同;而最低奖励训练条件产生最强的下游推广。在测试的Qwen2.5-Math环境中，RLVR数据选择似乎依赖于特定阶段，而非可归还为静态的问题质量排名。

Phase-and-First-Arrival VLM Feedback for Sparse-Reward Reinforcement Learning in Surgical Manipulation

手术手法中稀疏奖励强化学习的阶段与首次到达VLM反馈

• Authors: Wanli Liuchen, Fangyuan Wang, Bin Li, Anqing Duan, Yunhui Liu, Peng Zhou, David Navarro-Alarcon
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.07211
• Pdf link: https://arxiv.org/pdf/2609.07211
• Abstract Sparse outcome feedback limits what robots can learn from unsuccessful attempts at complex manipulation. Failed multi-stage surgical attempts can contain grasps, lifts, or transfers worth reusing. In sparse-reward reinforcement learning, terminal rewards collapse such attempts to the same outcome, while scalar vision-language model (VLM) ratings reveal neither what progress merits credit nor when it occurred. We introduce phase-and-first-arrival feedback: one VLM query per recorded episode identifies the furthest visually verified task phase and when that phase is first reached, allowing the learner to reuse partial behavior and localize credit. We instantiate it in SurgPhaseBench, a phase-structured suite spanning rigid and deformable tasks, and evaluate it in simulation and hardware. Across five simulated tasks, our method reaches 75.2% mean success, compared with 52.1% for a reward based on Contrastive Language-Image Pre-training (CLIP) using the same visual input; the advantage persists when only the feedback representation changes. On hardware, the same record supports autonomous block picking and slip recovery. Together, these results show that trajectory-level visual supervision can preserve partial progress while providing the temporal credit needed for sparse-reward control.
• 中文摘要 稀疏的结果反馈限制了机器人从复杂操作失败尝试中学到的内容。失败的多阶段手术尝试可能包含值得重复使用的抓握、提举或转移。在稀疏奖励强化学习中，终端奖励将此类尝试压缩至相同结果，而标量视觉语言模型（VLM）评分既不揭示哪些进展值得获得认可，也无法说明何时发生。我们引入了阶段和首次到达反馈：每录制一集一次VLM查询，识别最远的视觉验证任务阶段及其首次到达时间，使学习者能够重用部分行为并定位信用。我们将该机制实例化在SurgPhaseBench中，这是一个跨阶段结构的套件，涵盖刚性和可变形任务，并在仿真和硬件中进行评估。在五个模拟任务中，我们的方法平均成功率达到75.2%，而基于对比语言-图像预训练（CLIP）且使用相同视觉输入的奖励为52.1%;当仅反馈表示变化时，优势依然存在。在硬件上，同一记录支持自主区块拣选和滑动恢复。综合来看，这些结果表明轨迹级视觉监督可以在提供稀疏奖励控制所需的时间信用的同时，保留部分进展。

Long-Horizon Language Model Reinforcement Learning via Progressive Point Matching

通过渐进点匹配实现的长视野语言模型强化学习

• Authors: Preston Fu, Kevin Frans, Oleh Rybkin, Sergey Levine, Aviral Kumar
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.07303
• Pdf link: https://arxiv.org/pdf/2609.07303
• Abstract Current paradigms for training language models via reinforcement learning rely heavily on sparse outcome rewards. However, as we pursue tasks that require longer and more complicated trajectories, such strategies result in slow learning. Prior work has attempted to address this problem by rewarding partial progress; however, naive formulations are often biased and converge to suboptimal policies. We show that a simple and unbiased dense reward formulation, which we term progressive point matching, scales exponentially more efficiently to long-horizon tasks by rewarding partial progress on a segment level, both theoretically and empirically via synthetic environments. We then show how progressive point matching can be practically instantiated using a single reference trajectory per task. On extremely hard math reasoning problems, sparse outcome rewards cannot make any progress, whereas segment-level rewards enable improvements at larger test-time token budgets when measured by success rate or pass@k.
• 中文摘要 当前通过强化学习训练语言模型的范式高度依赖稀疏的结果奖励。然而，当我们追求需要更长且更复杂轨迹的任务时，这类策略会导致学习变慢。以往的研究尝试通过奖励部分进展来解决这一问题;然而，朴素的表述往往带有偏见，最终趋向次优策略。我们证明，简单且无偏的密集奖励表述，我们称之为渐进点匹配，通过理论和实证的合成环境在分段层面奖励部分进展，能指数级地更高效地扩展到长期任务。随后，我们展示了如何通过每个任务的单一参考轨迹实际实现渐进点匹配。在极难的数学推理问题中，稀疏的结果奖励无法带来进展，而分段级奖励则在较大的测试时代币预算中以成功率或pass@k衡量，从而实现改进。

Temporal-Causal Inference for Reinforcement Learning via Automata Learning

通过自动机学习进行时间因果推断以实现强化学习

• Authors: Jan Corazza, Daniil Kaminskyi, Simon Lutz, Patrick Nossol, Hadi Partovi Aria, Zhe Xu, Daniel Neider
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.07461
• Pdf link: https://arxiv.org/pdf/2609.07461
• Abstract We consider reinforcement learning in environments with dynamics that undergo an irreversible phase transition governed by a hidden temporal pattern. The agent observes the base state but cannot observe the phase directly. We formalize this problem as a two-phase non-Markovian decision process and introduce Temporal-Causal Inference for Reinforcement Learning (TCIRL), a framework that jointly learns a control policy and infers the hidden temporal cause of the phase transition. TCIRL maintains a hypothesis deterministic finite automaton (DFA) to track what phase is active and refines it via counterexample-driven SAT-based synthesis. We prove that the hypothesis converges almost surely to a DFA recognizing the true cause language on all attainable label sequences, yielding an optimal policy for the original non-Markovian decision process. Experiments on a genetic therapy gridworld and a traffic signal environment show that TCIRL recovers the correct cause DFA and matches the full-information baseline in both domains.
• 中文摘要 我们考虑在具有不可逆相变且由隐藏时间模式支配的动力学环境中进行强化学习。智能体观察基准状态，但无法直接观察该阶段。我们将该问题形式化为两阶段非马尔可夫决策过程，并引入强化学习的时间-因果推断（TCIRL），该框架共同学习控制策略并推断相变的隐藏时间原因。TCIRL维护一个假设确定性有限自动机（DFA）以跟踪处于激活状态，并通过基于反例的SAT综合对其进行细化。我们证明假设几乎必然收敛于识别所有可达标签序列真实因因语言的DFA，从而为原始非马尔可夫决策过程提供最优策略。在基因治疗网格世界和交通信号环境中的实验表明，TCIRL能够恢复正确的病因DFA，并在两个领域均匹配完整信息基线。

Zero-Shot Sim-to-Real Contact-Rich Assembly via Proprioception-Anchored Cross-Modal Pretraining

通过本体感觉锚定的跨模态预训练实现零射点模拟到真实接触丰富组装

• Authors: Yuhan Wang, Yurou Chen, Hongye Jiang, Wenzhao Lian
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.07534
• Pdf link: https://arxiv.org/pdf/2609.07534
• Abstract Contact-rich assembly remains challenging because it requires submillimeter spatial accuracy and reliable interpretation of forces during sustained contact. Although simulation-based reinforcement learning offers a scalable training paradigm, discrepancies in visual observations, contact dynamics, and force/torque (F/T) measurements often limit policy transfer. We observe that proprioception is comparatively consistent across domains because calibrated joint positions and consistently computed joint velocities align closely between simulation and hardware. Based on this observation, we present PACE (Proprioception-Anchored Cross-Modal Encoder), which supervises temporal visual and F/T representations by predicting proprioceptive state transitions. Static domain-specific factors, including lighting, texture, and sensor bias, contain little information about joint motion; the proposed objective therefore encourages the encoder to suppress these factors while retaining task-relevant motion cues. Policies trained on frozen PACE features are deployed on hardware without real-world fine-tuning or object-pose tracking. Across four contact-rich assembly tasks, PACE attains an average real-world success rate of 93.3\% and only a 2.7-percentage-point sim-to-real drop, meanwhile remaining robust to perturbations that substantially degrade pose-based and learned-fusion baselines.
• 中文摘要 接触丰富组装依然具有挑战性，因为它需要亚毫米级的空间精度和对持续接触时力的可靠解读。尽管基于仿真的强化学习提供了可扩展的训练范式，但视觉观察、接触动力学和力/扭矩（F/T）测量的差异常常限制策略转移。我们观察到本体感觉在不同领域相对一致，因为校准的关节位置和一致计算的关节速度在模拟与硬件之间高度一致。基于这一观察，我们提出了PACE（本体感觉锚定交叉模态编码器），它通过预测本体感觉状态转变来监督时间视觉和F/T表征。静态领域的特定因素，包括光照、纹理和传感器偏差，几乎没有包含关节运动的信息;因此，拟议的目标鼓励编码器抑制这些因素，同时保留任务相关的运动线索。基于冻结PACE特征训练的策略部署在没有实际微调或物体姿态追踪的硬件上。在四个接触丰富的组装任务中，PACE的平均真实成功率为93.3%，模拟到现实的下降仅为2.7%，同时对显著劣化基于姿势和学习融合的基准的扰动保持了韧性。

HyCO: A Hybrid Neural Solver for Combinatorial Optimization

HyCO：一种用于组合优化的混合神经求解器

• Authors: Yuheng Li, Di Yang, Haipeng Chen, Yanhai Xiong
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.07990
• Pdf link: https://arxiv.org/pdf/2609.07990
• Abstract Sequential reinforcement learning (RL) solvers and global diffusion model (DM) solvers for neural combinatorial optimization exhibit complementary failure modes under an optimization-regret view. The former enjoys small marginal regret in the early construction stage, but suffers from horizon-wise compounding errors with super-linear regret growth; the latter avoids horizon compounding but incurs linear or sublinear regret w.r.t. the dimension of the remaining unsolved subspace. We propose Hybrid Neural Solver for Combinatorial Optimization (HyCO), a hybrid inference algorithm that constructs a solution prefix with an RL solver and adaptively switches to a conditional DM to complete the remaining decisions. To characterize why such hybridization helps, when to trigger the handover, and how to realize it in practice, we first develop a unified error-scaling theoretical framework and prove that, under explicit error-scaling assumptions, i) the hybrid structure achieves strictly lower expected regret than either backbone alone, and ii) there exists a unique optimal trigger step that minimizes the hybrid regret. We then design a lightweight adaptive trigger that combines policy entropy and RL-DM disagreement to detect trajectory-level signals of the regime shift as a practical proxy, since the optimal trigger step is defined at the expected-regret level and is not directly computable on individual trajectories. Experimental results on diverse benchmarks demonstrate that HyCO achieves consistent improvements over both backbones and support the empirical effectiveness of adaptive triggering.
• 中文摘要 顺序强化学习（RL）求解器和神经组合优化的全局扩散模型（DM）求解器在优化-遗憾视角下表现出互补的失败模式。前者在早期构建阶段享有较小的边际遗憾，但存在视野方向的复合错误和超线性遗憾增长;后者避免了视野复合，但会在剩余未解子空间的维度上产生线性或亚线性遗憾。我们提出了混合神经求解器用于组合优化（HyCO），这是一种混合推理算法，通过强化学习求解器构造解前缀，并自适应切换到条件DM以完成剩余决策。为了描述这种混合化为何有用、何时触发切换以及如何在实际中实现，我们首先构建了一个统一的误差尺度理论框架，并证明在显式误差尺度假设下，i） 混合结构的预期遗憾远低于单纯主干链，ii） 存在一个独特的最优触发步骤，使混合遗憾最小化。随后，我们设计了一个轻量级自适应触发器，结合策略熵和强化-管理层不一致，以实际检测政权转移的轨迹级信号，因为最优触发步骤定义在预期遗憾层面，且无法直接计算单个轨迹。多种基准测试的实验结果表明，HyCO在两个骨干链上都能持续提升，支持自适应触发的实证有效性。

Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study

迷你批次风险规避深度Q学习：机器人导航案例研究

• Authors: Aayush Patel, Andrzej Ruszczyński
• Subjects: Subjects: Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.07998
• Pdf link: https://arxiv.org/pdf/2609.07998
• Abstract We study the control of Markov decision processes in which the quality of a policy is evaluated by a dynamic, time-consistent Markov risk measure rather than by an expected discounted cost. The main obstacle to combining such measures with reinforcement learning is that a transition risk mapping depends on the transition kernel in a nonlinear way, and therefore cannot be estimated from a single observed transition. We remove this obstacle by employing mini-batch transition risk mappings: the mapping is applied to the empirical measure of $N$ independent next-state samples, and the result is averaged. The resulting mapping is again coherent. However, as an expected value of a function of $N$ next-state values, it admits an unbiased one-sample estimator. We embed this mapping into a double deep Q-network, analyze the two sources of estimation bias that arise, and obtain a risk-averse Q-learning method applicable to state spaces far beyond the reach of tabular schemes. The method is applied to an underwater robot navigation problem, in which a vehicle must visit collection points, gather stochastic information payloads, and deliver them at transmission points, while exposed at each step to the risk of destruction. A hierarchical decomposition delegates path execution to an exact graph search and confines learning to the high-level ``collect or transmit'' decision. A low-dimensional feature map, invariant under the symmetries of the problem, replaces the raw state--configuration encoding. In experiments on $300$ held-out environments, the resulting policies transfer to instance sizes never seen in training, and already $N=2$ reduces the upper semideviation of the outcome distribution while simultaneously improving its mean whenever the simulator is misspecified---an empirical counterpart of the duality between coherent risk measures and distributional robustness.
• 中文摘要 我们研究了通过动态、时间一致的马尔可夫风险度量而非期望折现成本来评估策略质量的马尔可夫决策过程控制。将此类测量与强化学习结合的主要障碍是，过渡风险映射以非线性方式依赖于过渡核，因此无法仅凭单一观测到的转移估算。我们通过使用小批量转移风险映射来消除这一障碍：将映射应用于$N$独立的下一状态样本的经验测量，并对结果进行平均。所得映射依然是连贯的。然而，作为$N次状态值函数的期望值，它允许一个无偏的单样本估计量。我们将该映射嵌入双深度Q网络，分析由此产生的两种估计偏差来源，获得一种风险规避型Q学习方法，适用于表格方案无法覆盖的状态空间。该方法应用于水下机器人导航问题，其中飞行器必须访问采集点，收集随机信息载荷，并将其送达传输点，且每步都暴露于毁坏风险之下。分层分解将路径执行委托给精确图搜索，学习限制在高层次“收集或传输”决策中。低维特征映射在问题对称性下不变，取代原始状态-配置编码。在300美元未完成的实验中，所得策略可转换为训练中从未见过的实例规模，且$N=2$已减少结果分布的上半偏差，同时在模拟器错误指定时改善其均值---这是相干风险度量与分布鲁棒性的经验对应。

mjorbit: A Simulation Framework for Space Robotics

mjorbit：空间机器人模拟框架

• Authors: John Z. Zhang, Joris Verhagen, Fausto Vega, Patrick McKeen, Zachary Manchester
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.08010
• Pdf link: https://arxiv.org/pdf/2609.08010
• Abstract This paper presents a general framework for simulating multi-body space robots with contact. We bring efficient, large-scale robot simulation to in-space servicing, assembly, and manufacturing applications. First, we perform an empirical trade study of methods for coupling orbit propagation with existing robotics simulation frameworks. Next, we present mjorbit, a general, flexible, and performant framework built on the MuJoCo engine widely used in robotics, to which we add key spacecraft dynamics, actuators, and sensors. We provide a low-latency C++ CPU backend and a high-throughput GPU backend behind a simple Python API. We demonstrate mjorbit by solving several realistic on-orbit case studies with both model-predictive control and reinforcement learning. Open-source code and examples are available at: this https URL
• 中文摘要 本文提出了一个通用框架，用于模拟多体接触空间机器人。我们将高效、大规模的机器人仿真应用于太空维修、组装和制造应用。首先，我们对轨道传播与现有机器人仿真框架耦合的方法进行了实证性贸易研究。接下来，我们介绍mjorbit，这是一个基于广泛应用于机器人的MuJoCo引擎构建的通用、灵活且性能优良的框架，并为其添加了关键的航天器动力学、执行器和传感器。我们提供了一个低延迟的C++ CPU后端和一个高吞吐量的GPU后端，支持一个简单的Python API。我们通过解决多个现实的轨道案例研究，结合模型预测控制和强化学习，演示了mjorbit。开源代码和示例可在以下网站获取：此 https URL

Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

通过强化学习引发多模推理代理的自我验证

• Authors: Vishwas Sathish, Viresh Ranjan, Xinliang Zhu, Arnab Dhua, Douglas Gray
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.08025
• Pdf link: https://arxiv.org/pdf/2609.08025
• Abstract Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.
• 中文摘要 推理代理越来越依赖外部工具，如网络搜索来回答复杂查询。强化学习（RL）微调算法如GRPO提升了文本语言模型中的长形式推理能力，尤其是在编码和数学领域。然而，多模态智能体的可靠工具使用仍具挑战性，因为模型必须在整合噪声检索证据的同时解释文本和图像，且常常在结果层级的稀疏监督下，且没有明确的验证信号。我们介绍了通过强化学习自我验证（SVRL），这是一个仅支持强化学习的微调框架，训练多模态智能体在其自身推理轨迹中验证和过滤检索到的证据，减少对推理时外部验证器的依赖。SVRL还引入了搜索感知惩罚，防止不必要的工具调用，并引入查询多样性奖励，鼓励多样化、结构良好的搜索查询，提供细粒度的搜索反馈。仅对5{，}000个视觉问答示例对QWEN-2.5-VL-7B进行SVRL精细调校，能在多跳VQA泛化和工具效率上持续提升。总体而言，SVRL缩小了紧凑代理与更大专有模型之间的差距，同时显著降低了训练和推理成本。

Proactive Context-Forecasted Safety Constraints for Nonstationary Reinforcement Learning

非平稳强化学习的主动上下文预测安全约束

• Authors: Tim Tomashevskiy
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.08080
• Pdf link: https://arxiv.org/pdf/2609.08080
• Abstract Ensuring safety in reinforcement learning under nonstationarity requires anticipating changes in risk before they lead to unsafe behavior. Existing approaches typically rely on safety constraints defined at design time or updated reactively during execution, assuming that such constraints remain valid over time. However, in nonstationary environments with evolving contexts and changing driving layouts, these assumptions may fail. We propose a framework for proactive safety constraint generation based on context forecasting. The approach infers latent environmental context from observations, predicts its future evolution, and constructs safety constraints adapted to anticipated conditions. This enables the agent to proactively avoid unsafe regions instead of reacting only after safety violations occur. We evaluate the method in driving environments with structured context variation. The experiments include a sweep over nonstationarity intensities and additional held-out driving layouts, including highway, intersection, and racetrack scenarios. Results show that proactive constraint generation substantially reduces collisions under both seen and out-of-training nonstationarity intensities and generally remains effective across held-out driving layouts while maintaining usable task performance. These findings suggest that context-based constraint generation is a promising approach for safe reinforcement learning under nonstationarity.
• 中文摘要 在非平稳性下确保强化学习的安全，需要在风险变化导致不安全行为之前预见风险变化。现有方法通常依赖设计时定义的安全约束，或在执行时被动更新，假设这些约束在时间内依然有效。然而，在非平稳环境中环境变化、环境变化、驾驶布局变化时，这些假设可能失效。我们提出了基于上下文预测的主动安全约束生成框架。该方法从观察推断潜在环境上下文，预测其未来演变，并构建适应预期条件的安全约束。这使得智能体能够主动避开不安全区域，而非仅在安全违规发生后才反应。我们在结构化上下文变化的驾驶环境中评估该方法。实验包括对非平稳强度的扫描以及额外的持续驾驶布局，包括高速公路、交叉口和赛道场景。结果显示，主动约束生成在可见和训练外的非平稳强度下显著减少碰撞，并且在保持行驶布局的同时保持有效，同时保持可用的任务性能。这些发现表明基于上下文的约束生成是非平稳性下安全强化学习的有前景方法。

When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluation

当指标奖励最差翻译时：内化文化推理以评估社交媒体翻译

• Authors: Yiwen Qiu, Linjuan Wu, Dingming Li, Yizhou Liu, Zixuan Wang, Haolei Xu, Ye Guo, Daoxin Zhang, Weiming Lu, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.08156
• Pdf link: https://arxiv.org/pdf/2609.08156
• Abstract Automatic translation quality metrics trained on general-domain corpora systematically fail on social media content, where communicative intent is encoded in culturally loaded expressions (internet slang, homophonic ciphers, and platform-specific idioms) rather than surface token patterns. We conduct a systematic empirical analysis demonstrating that standard metrics including COMET, XCOMET, and BERTScore exhibit near-zero or negative correlation with human cultural judgments, and even display a severity inversion in which scores increase as translation quality deteriorates. We further show that this failure extends to large language model judges: Qwen3-235B achieves Cohen's kappa of only 0.162, revealing that the bottleneck is not reasoning capacity but cultural grounding: models lack the domain-specific cultural knowledge needed to identify which aspects of a translation require scrutiny. To address this, we propose CuRIL, a reinforcement learning framework that internalizes cultural reasoning: cultural annotations are prepended inside the model's reasoning, excluded from policy gradients via a token-level loss mask, and injected with a probability that decays to zero over training, progressively forcing autonomous cultural judgment. On a 1,444-sample human-annotated social media translation benchmark, Qwen3-8B trained with CuRIL achieves Cohen's kappa 0.370 and Exact Match accuracy of 45.22%, approaching Gemini-3.1-Pro with 30x fewer parameters and surpassing models up to 235B in scale. We further demonstrate that our judge produces reliable reward signals for downstream translation optimization, reducing the low-quality translation rate by over 20 percentage points under independent human evaluation.
• 中文摘要 在通用领域语料库上训练的自动翻译质量指标在社交媒体内容上系统性地失败，因为沟通意图编码在文化负载丰富的表达中（网络俚语、同音暗号和平台特定表达），而非表面符号模式。我们进行了系统实证分析，证明包括COMET、XCOMET和BERTScore在内的标准指标与人类文化判断几乎为零或负相关，甚至表现出严重性反转，即随着翻译质量下降，分数会上升。我们还进一步证明，这种失败也适用于大型语言模型评判：Qwen3-235B仅实现了0.162的Cohen's kappa，显示瓶颈不在于推理能力，而在于文化基础：模型缺乏识别翻译中哪些方面需要审查的领域特定文化知识。为此，我们提出了CuRIL，一种内化文化推理的强化学习框架：文化注释在模型推理中前置，通过代币级丢失掩膜排除策略梯度，并注入一个在训练中衰减至零的概率，逐步强制自主文化判断。在1444个样本的人工注释社交媒体翻译基准测试中，使用CuRIL训练的Qwen3-8B实现了Cohen's kappa 0.370和45.22%的精确匹配准确率，接近Gemini-3.1-Pro，参数数减少30倍，规模超过235B模型。我们进一步证明，我们的评判能为下游翻译优化产生可靠的奖励信号，在独立人工评估下降低低质量翻译率超过20个百分点。

QoS-Aware RACH Preamble Slicing via Quota-Projected Branching Deep Reinforcement Learning

QoS感知的RACH前言切片，通过配额投影分支深度强化学习

• Authors: Jiulin Guo, Jiahan Xu, Jiashuo Zhang, Heng Yang, Yizhen Sun, Yutong Xie, Shanshan Li, Zhenyu Liu, Lei Zhang
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2609.08199
• Pdf link: https://arxiv.org/pdf/2609.08199
• Abstract Quality-of-service (QoS)-aware random access requires adaptive allocation of a finite random access channel (RACH) preamble budget across heterogeneous traffic and access procedures. This paper proposes QP-BD3QN-RACH, a quota-projected branching deep reinforcement learning controller for mixed two-step (2RA) and four-step (4RA) contention-based random access. Four action branches correspond to the delay-sensitive and delay-tolerant 2RA/4RA preamble pools. A branching dueling Double DQN selects pool-specific multipliers, and deterministic quota projection converts them to nonnegative integer allocations that preserve the preamble budget. With five actions per branch, the controller represents 625 pre-projection branch-action tuples using 20 branch-action outputs. Evaluation covers five arrival loads, cross-method comparison under nominal seed 42, six-seed sensitivity of QP-BD3QN-RACH, and targeted ablations. Across the five-load grid, its mean direction-aligned differences relative to four comparators are positive: 5.74 to 8.21 percentage points for success/collision, 1.23 to 1.92 percentage points for fallback, 0.35 to 0.68 percentage points for blocking, and 0.128 to 0.456 decision intervals for successful-access delay. Load-wise results exhibit metric-dependent tradeoffs, particularly under intermediate and overload conditions.
• 中文摘要 服务质量（QoS）感知随机接入需要在异构流量和访问程序中自适应分配有限随机接入通道（RACH）前导码预算。本文提出QP-BD3QN-RACH，一种配额投影分支深度强化学习控制器，用于混合两步（2RA）和四步（4RA）争用随机访问。四个动作分支对应延迟敏感和容忍延迟的2RA/4RA前导词池。分支对抗双重DQN选择池特定乘数，确定性配额投影将其转换为保持前导码预算的非负整数分配。每个分支有五个动作，控制器代表625个预投影分支动作元组，输出20个分支动作输出。评估涵盖五种抵达负载、名义种子42下的跨方法比较、QP-BD3QN-RACH的六种子敏感性以及有针对性的消融。在五负载网格中，其相对于四个比较器的方向对齐平均差异为正：成功/碰撞介于5.74至8.21个百分点，后退介于1.23至1.92个百分点，阻断介于0.35至0.68个百分点，成功访问延迟在决策区间介于0.128至0.456个。负载结果在中等和过载条件下表现出依赖度量的权衡。

A Better Spur Should Start From Each Objective

每个目标都应该有一个更好的推动

• Authors: Shanwen Mao, Hao Zhang, Guangtao nie, Zhiheng Li, Huimu Wang, Sulong Xu, Gu Simiu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.08211
• Pdf link: https://arxiv.org/pdf/2609.08211
• Abstract Real-world Multi-Objective Reinforcement Learning (MORL) often suffers from sparse rewards, reward conflicts, and late-stage reward tug-of-war, causing traditional linear scalarization to experience severe metric oscillations. To address optimization conflicts among multiple objectives in real-world deployment scenarios, we propose Multi-Marginal Preference Optimization (MMPO), a fine-grained framework that intervenes at the data, gradient, and constraint levels rather than relying on coarse-grained global scalarization. Specifically, MMPO performs exposure debiasing to mitigate sparse and biased rewards, applies priority-aware orthogonal projection to decouple conflicting gradients, and introduces self-prompted gradient constraints to prevent dominant objectives from overwhelming weaker ones. Experiments on real-world e-commerce datasets show that MMPO improves training stability and consistently achieves better performance across conflicting metrics. Moreover, it generalizes robustly to broader tasks such as ToolRL and code generation, demonstrating its effectiveness as a practical paradigm for multi-objective alignment.
• 中文摘要 现实世界的多目标强化学习（MORL）常常存在奖励稀疏、奖励冲突和后期奖励拉锯战，导致传统的线性标量化出现严重度量振荡。为解决现实部署场景中多个目标之间的优化冲突，我们提出了多边际偏好优化（MMPO），这是一种细粒度框架，干预数据、梯度和约束层面，而非依赖粗粒度的全局标量化。具体来说，MMPO通过暴露偏倚来缓解稀疏和偏向奖励，应用优先级感知正交投影以解耦冲突梯度，并引入自触发梯度约束以防止主导目标压倒较弱目标。在现实电商数据集上的实验表明，MMPO能提升训练稳定性，并在冲突指标上持续获得更好的性能。此外，它能够稳健地推广到更广泛的任务，如ToolRL和代码生成，证明了其作为多目标对齐实用范式的有效性。

Bridging Language and Physics: Automated Design of Continuum Robots with Large Language Models

连接语言与物理：利用大型语言模型自动设计连续介质机器人

• Authors: Jingyi Chen, Mohan Zhang, Laura Yao, Yingtai Ni, Jianmin Ji, Jie Peng, Song Wang, Tianlong Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.08220
• Pdf link: https://arxiv.org/pdf/2609.08220
• Abstract Large language models (LLMs) have recently emerged as a promising tool for automating robot design from high-level specifications, yet they remain ineffective for robots operating under complex physical interactions. This limitation stems from the gap between language-based reasoning and the physical consequences of embodiment, often resulting in designs with low physical validity. In this work, we propose a multi-layered framework, AID-SR, that establishes a closed loop by translating simulator-observed physical states into structured feedback for the LLM designer. Combined with semantic critique, human feedback, and iterative refinement, the framework promotes the generation of physically feasible and functionally meaningful robot designs. We evaluate our approach on tendon-driven continuum robots across a benchmark of 14 tasks spanning reaching, grasping, locomotion, and manipulation. The proposed framework achieves 96.2% rate for passing the simulation feasibility check and by applying a common reinforcement learning training, 26.7% robots can successfully fulfill the corresponding task. We then fabricate three designed robots of AID-SR that successfully complete the task in real-world. These extensive experiments across simulation and real-world environments demonstrate and break the wall of utilizing the LLMs for automated design of continuum robots. The source code and experimental resources are publicly available at this https URL.
• 中文摘要 大型语言模型（LLMs）最近作为自动化机器人设计的有前景工具出现，但对于在复杂物理交互下运行的机器人来说，它们仍然无效。这一局限源于基于语言的推理与具身化的物理后果之间的差距，常导致设计物理效度较低。在本研究中，我们提出了一个多层次框架AID-SR，通过将模拟器观测到的物理状态转化为结构化反馈，为LLM设计者建立闭环。结合语义批判、人工反馈和迭代优化，该框架促进了物理可行且功能意义强的机器人设计的生成。我们评估了基于14项任务的肌腱驱动连续体机器人方法，涵盖伸手、抓取、运动和操作。所提框架通过模拟可行性检查的成功率为96.2%，通过通用强化学习训练，26.7%的机器人成功完成相应任务。随后，我们制造了三台AID-SR设计的机器人，成功完成了现实世界中的任务。这些跨模拟和现实环境的广泛实验展示了并打破了利用大型语言模型自动化设计连续体机器人的壁垒。源代码和实验资源可在此 https URL 公开获取。

Routing Dense Layouts with History-Aware Offline Reinforcement Learning using LSTM

利用历史感知离线强化学习（LSTM）路由密集布局

• Authors: Afsara Khan, Austin Rovinski
• Subjects: Subjects: Hardware Architecture (cs.AR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.08232
• Pdf link: https://arxiv.org/pdf/2609.08232
• Abstract Detailed routing remains a dominant runtime bottleneck in physical design due to increasing complexity of design rules. Modern routers can struggle to resolve persistent violations under dense operating conditions. While recent work leverages reinforcement learning (RL) to dynamically select costs for each routing iteration, we find that this technique struggles with high-density designs where routing solutions are significantly harder. To address this, we present a history-aware offline RL policy which predicts iterative cost weights in these dense regimes to improve convergence across placement densities by utilizing readily available features from the router. Our policy uses conservative Q-learning similarly to prior work; however, our key insight is that addition of a lightweight LSTM architecture and additional features can retain sequence context and improve routing convergence across multiple densities and route guide qualities. Our policy can be integrated into any cost-based router with minimal pipeline changes, as it does not interfere with the core search algorithm. We evaluate our policy on held-out density and adjustment settings, including difficult operating points induced by dense placement and low guide quality. Our policy reduces design rule violations (DRVs) by an average of 92% over the top public baseline while simultaneously reducing runtime by 10%.
• 中文摘要 由于设计规则日益复杂，详细路由仍然是物理设计中主要的运行瓶颈。现代路由器在高密度运行条件下难以解决持续违规。虽然近期工作利用强化学习（RL）动态选择每次路由迭代的成本，但我们发现该技术在高密度设计中存在困难，因为路由解决方案难度显著更高。为此，我们提出了一种历史感知的离线强化学习策略，预测这些密集区间的迭代成本权重，通过利用路由器提供的现成特性改善布置密度的收敛性。我们的策略采用保守的Q-学习，类似于以往的工作;但我们的关键见解是，加入轻量级LSTM架构和额外功能可以保留序列上下文，并提升多密度和路径引导特性间的路由收敛性。我们的策略可以集成到任何基于成本的路由器中，只需最小的流水线修改，因为不会干扰核心搜索算法。我们评估政策中的长期密度和调整设置，包括因密集布置和低导向质量引起的困难操作点。我们的政策在顶部公共基线上平均减少了92%的设计规则违规（DRV），同时将运行时间缩短10%。

From Glance to Scrutiny: Progressive Distortion Reasoning for Fine-Grained Image Quality Assessment

从一眼到细致：渐进畸变推理用于细粒度图像质量评估

• Authors: Aoting Zhang, Mingze Gao, Dongbao Yang, Longyi Chen, Daoxin Zhang, Yi Wu, Yao Hu, Yu Zhou
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.08316
• Pdf link: https://arxiv.org/pdf/2609.08316
• Abstract Multi-modal large language models (MLLMs) have demonstrated significant potential in image quality assessment (IQA) by bridging visual perception with descriptive evaluations. However, existing approaches mainly focus on holistic quality prediction, often functioning as black boxes that provide limited insight into where distortions occur and how they affect perceived quality, hindering fine-grained analysis of localized and heterogeneous degradations. We propose GS-IQA, a framework that reformulates IQA as a progressive Where--What--How diagnosis, emulating the human perceptual process from an initial glance to closer scrutiny. Since a severity judgment is meaningful only for a correctly localized and recognized region, we realize this progression through a two-stage reinforcement learning paradigm that respects such dependencies: the glance stage uses a perception-gated reward to establish where degradations lie and what they are, activating severity feedback only once both are correct, while the scrutiny stage introduces online reward-conditioned degradation generation to synthesize hard examples targeted at the model's perceptual bottlenecks, sharpening its discrimination of subtle severity variations. To enable systematic evaluation, we construct Diag-Bench, a region-level IQA benchmark of about 25K curated samples spanning 12 distortion types and five ordinal severity levels. Extensive experiments show that GS-IQA consistently surpasses state-of-the-art methods in distortion localization, recognition, and severity estimation, and that its diagnostic representations transfer effectively to conventional global quality prediction across diverse external benchmarks. Code and data will be released.
• 中文摘要 多模态大型语言模型（MLLMs）通过连接视觉感知与描述性评估，展现出图像质量评估（IQA）的显著潜力。然而，现有方法主要关注整体质量预测，常常作为黑箱，提供有限的扭曲发生位置及其对感知质量的影响，阻碍对局部和异质退化的细致分析。我们提出了GS-IQA，这一框架将IQA重新表述为一种渐进的“在哪里——什么——如何”诊断，模拟人类感知过程从初次观察到更细致的审视。由于严重度判断仅对正确定位和识别的区域有意义，我们通过尊重此类依赖的两阶段强化学习范式实现这一进展：视角阶段使用感知门控奖励确定降级位置及其内容，只有当两者都正确时才激活严重度反馈;审视阶段引入在线奖励条件降级生成，综合针对模型感知瓶颈的硬样本，增强对细微严重变化的辨别。为实现系统评估，我们构建了Diag-Bench，这是一个区域级IQA基准，包含约2.5万个精选样本，涵盖12种畸变类型和5个序数严重程度级别。大量实验表明，GS-IQA在失真定位、识别和严重度估计方面持续超越最先进方法，其诊断表示能够有效转移至跨越多种外部基准的传统全球质量预测。代码和数据将陆续发布。

Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks

环境作为支架：在长期任务中自助反馈自我演化代理

• Authors: Hongbang Yuan, Zhuoran Jin, Yixin Cao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.08404
• Pdf link: https://arxiv.org/pdf/2609.08404
• Abstract Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional \textit{agent-side warming} up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to \textit{environment-side adaptation} by constructing \textbf{F}eedback-\textbf{E}nriched \textbf{E}nvironments (\textbf{FEEs}). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs \textbf{(1)} stabilizes training dynamics by reducing entropy volatility, \textbf{(2)} facilitates proactive state-space exploration in difficult tasks, \textbf{(3) }ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and \textbf{(4) }identifies intra-group feedback consistency as a critical boundary for stable optimization.
• 中文摘要 大型语言模型在静态推理方面表现出显著的熟练度，但通过强化学习（RL）将其训练为长期任务的自主智能体，常常受到严重奖励稀疏的阻碍。虽然传统的\textit{代理侧暖身}通过监督微调（SFT）可以缓解这一问题，但常常受限于数据稀缺和有限的探索。为此，我们提出了一种范式转变，转而采用\textit{环境侧适应}，构建了 \textbf{eedback-\textbf{E}nriched \textbf{E}环境环境（\textbf{FEEs}）。通过一项试点研究，我们建立了一种反馈设计策略，通过在剧集内探索和剧集间演化的后期阶段，从动作指导转向观察丰富，重新表述环境。在SciWorld和BFCL基准测试中，使用多种Qwen3模型尺度和强化学习算法（如GRPO、GSPO和DAPO）进行大规模实验，表明FEE在性能上持续优于标准设置。此外，我们的分析显示，使用FEEs的\textbf{（1）}训练通过降低熵波动性稳定训练动态，\textbf{（2）}促进了复杂任务中的主动状态-空间探索，\textbf{（3） }确保环境指导内化为策略权重，而非仅仅作为推断时间的先验，且\textbf{（4） }将组内反馈一致性视为稳定优化的关键边界。

SRPO: Setwise Relative Policy Optimization for Multi-Agent LLMs

SRPO：多代理大型语言模型的集合相对策略优化

• Authors: Shengtian Yang, Ziyu Xiong, Yu Li, Yewen Li, Qingpeng Cai, Lei Feng
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.08452
• Pdf link: https://arxiv.org/pdf/2609.08452
• Abstract Multi-agent large language models solve complex tasks by coordinating several policies in a shared environment. However, existing reinforcement learning methods usually optimize each response or trajectory separately, even when several outputs jointly cause one state transition. Consequently, the update unit differs from the action executed by the system. To address this problem, we propose SRPO (Setwise Relative Policy Optimization), which treats the active set the minimal set of outputs consumed by one transition, as one multi-agent action. Specifically, SRPO combines member log-ratios into one cardinality-normalized set ratio, assigns one relative advantage, and clips the set once. This formulation unifies division of labor and joint co-evolution as actions with different set sizes. Experiments on mathematical reasoning and multi-turn search demonstrate one training interface for fixed, mixed, and dynamically routed workflows across four model scales, with the strongest macro-average results among the reported comparisons. Optimization diagnostics further characterize its stability under different event reductions and set sizes.
• 中文摘要 多智能体大型语言模型通过在共享环境中协调多个策略来解决复杂任务。然而，现有的强化学习方法通常会分别优化每个响应或轨迹，即使多个输出共同导致一个状态转移。因此，更新单元与系统执行的动作不同。为解决这个问题，我们提出了SRPO（集合相对策略优化），将一个转移消耗的最小输出集（主动集合）视为一个多智能体动作。具体来说，SRPO将成员的对数比率合并为一个基数归一化的集合比，分配一个相对优势，并裁剪该集合一次。该表述统一了分工和联合共进作为不同集合大小的动作。数学推理和多回合搜索实验展示了固定、混合和动态路由工作流在四个模型尺度上的训练接口，在报告的比较中宏观平均结果最强。优化诊断进一步表征其在不同事件减少和集合大小下的稳定性。

Which Forms of Caregiver Feedback Support Grammar Learning? A Reinforcement-Learning Study of Child-Like Language Models

哪些形式的照护者反馈支持语法学习？儿童类语言模型的强化学习研究

• Authors: Jing Liu, Marianne Schweitzer, Abdellah Fourtassi
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.08576
• Pdf link: https://arxiv.org/pdf/2609.08576
• Abstract Social interaction is central to children's language learning, but the effects of different forms of caregiver feedback are difficult to isolate in naturalistic data. We use child-like language models as controlled learners to test which forms of feedback support grammatical development. Small GPT-2-style models are pretrained on child-directed language from CHILDES, then fine-tuned with reinforcement learning using reward models trained to capture four feedback types: communicative feedback, structural alignment, semantic contingency, and affective feedback. Reward fine-tuning yields limited gains on minimal-pair evaluations, but clearer effects in free generation. Structural alignment produces the strongest improvements in grammaticality, providing a novel, plausible mechanistic account of how this feedback can support grammar learning. Communicative feedback yields more moderate gains. In contrast, semantic contingency and affective feedback do not improve grammaticality, although further analyses suggest that they may support other aspects of language learning beyond grammar. These results suggest that different forms of caregiver feedback make complementary contributions to language learning.
• 中文摘要 社会互动是儿童语言学习的核心，但不同形式的照顾者反馈在自然主义数据中难以单独区分。我们使用类儿童语言模型作为受控学习者，测试哪些反馈形式支持语法发展。小型GPT-2风格模型先在CHILDES的儿童导向语言上预训练，然后通过奖励模型进行强化学习微调，这些模型训练出捕捉四种反馈类型：交际反馈、结构对齐、语义偶然性和情感反馈。奖励微调在最小配对评估中获得有限收益，但在自由生成中效果更明显。结构性对齐在语法方面带来最强的提升，提供了一种新颖且合理的机制解释，说明这种反馈如何支持语法学习。交际反馈带来更适度的提升。相比之下，语义条件和情感反馈并未改善语法性，尽管进一步分析表明它们可能支持语法以外语言学习的其他方面。这些结果表明不同形式的照顾者反馈对语言学习有互补贡献。

SUN: Reaching for Novelty in Reinforcement Learning

SUN：在强化学习中追求新颖性

• Authors: Wenyan Yang, Arsenii Mustafin, Dominik Baumann, Joni Pajarinen, Simone Parisi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.08642
• Pdf link: https://arxiv.org/pdf/2609.08642
• Abstract Exploration in reinforcement learning (RL) remains a fundamental challenge. Recent goal-conditioned RL strategies (which select goals to encourage broader state coverage) have shown promising results, but none scores a goal by novelty and reachability jointly: the two signals are traded off by hand, applied in sequence, or one is neglected outright. In this paper, we introduce a reachability-aware goal-selection framework that explicitly integrates these two aspects, and that can be seamlessly incorporated into any off-policy RL algorithm. To this aim, we propose SUccessor-to-Novelty (SUN), an indicator derived from successor value functions to identify goals that are both novel and reachable. We prove that SUN recovers count-based bonuses in the limit, bounds short-horizon hitting probabilities, and provably rejects unreachable goals. We further present an adaptive goal-selection strategy that leverages these properties, and an accurate yet lightweight pseudocount to avoid the overhead of classic methods. We back up all our claims with thorough benchmarks: SUN consistently outperforms state-of-the-art methods in standard and novel environments with unreachable or hard-to-reach states, irreversible transitions, obstacles, mazes, and unbounded spaces.
• 中文摘要 强化学习（RL）中的探索仍是一个根本性挑战。近期的目标条件化强化学习策略（选择目标以促进更广泛状态覆盖）显示出有希望的成果，但没有任何策略能通过新颖性和可达性共同达成目标：两种信号要么人工交换，要么按顺序应用，要么一个被完全忽视。本文引入了一个可达性感知目标选择框架，明确整合了这两方面，并可无缝集成到任何非策略的强化学习算法中。为此，我们提出了“从新颖到新颖性”（SU）的指标，该指标由后继价值函数推导，用以识别既新颖又可达成的目标。我们证明SUN在极限内恢复基于计数的加成，限制短视野的达成概率，并可证明拒绝不可达成的目标。我们还提出了一种利用这些特性的自适应目标选择策略，以及一种准确而轻量的伪计数方法，以避免传统方法的开销。我们用详尽的基准来支持所有观点：SUN在标准和新颖环境中，如不可达或难以触及的状态、不可逆的转变、障碍物、迷宫和无界空间，始终优于最先进的方法。

Difficulty-Adaptive Tree-Structured Policy Optimization for Expanding Reasoning Coverage in RLVR

难度自适应树结构策略优化以扩展RLVR推理覆盖范围

• Authors: Youngjun Yu, Sanghwan Jang, Hwanjo Yu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.08650
• Pdf link: https://arxiv.org/pdf/2609.08650
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.
• 中文摘要 带可验证奖励的强化学习（RLVR）是大型推理模型近期成功的核心。然而，尽管RLVR显著提高了单样本准确性，但由于训练过程中探索有限，常常未能扩大模型的内在推理覆盖率（pass@k）。为此，我们优化了列车时间推展的结构设计以增强pass@k。我们的分析确定了三个关键设计原则：（1）难度自适应的推展在扩展 pass@k 中发挥重要作用，超越了作为效率启发式;（2） 基于树状的推展在发现正确答案方面优于并行抽样;（3） 句子熵引导分叉克服了令牌级分支的局部化现象，以最大化语义多样性。基于这些见解，我们提出了DATPO（难度自适应句子熵引导树结构策略优化）。DATPO将难度自适应树搜索与兄弟多样性优势项整合，明确促进语义多样性以扩展训练中的推理覆盖。数学推理基准测试的实验表明，DATPO在pass@k中表现优于基线，这直接转化为更优越的测试时间扩展性能。

Learning to build covering structures with continuous adjustments

学习通过持续调整构建覆盖结构

• Authors: Gabriel Vallat, Maryam Kamgarpour, Stefana Parascho
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.08669
• Pdf link: https://arxiv.org/pdf/2609.08669
• Abstract Robotic construction offers the potential to use materials more efficiently and create complex geometries, but current methods rely on rigid, high-precision plans that cannot accommodate the tolerances, inaccuracies, and unexpected changes inherent in physical fabrication. In this work, we introduce a reinforcement learning approach that forgoes predefined plans entirely, instead generating construction sequences adaptively as the structure is built. Our method operates on graph-structured state representations and a mixed (parameterized) action space, requiring both discrete block selection and continuous placement parameters. Because the stability simulation of a structure is computationally heavy, we develop an efficient exploration strategy by incorporating unilateral edges into graph neural networks, extending soft actor-critic (SAC) to this hybrid setting. We evaluate our algorithm, HSAC, against the prior method hybrid-PPO (HPPO), demonstrating significantly higher asymptotic performance and good sample efficiency. We also demonstrate HSAC's robustness to hyperparameter choices and its exploration capability, handling up to 10 discrete actions without performance degradation. Finally, we validate our approach on a physical two-robot setup, successfully building a spanning arch with 3D-printed blocks in closed-loop execution, confirming that policies trained in simulation transfer to real hardware.
• 中文摘要 机器人结构有望更高效地利用材料并创建复杂几何形状，但当前方法依赖于刚性、高精度的平面图，无法适应物理制造中固有的公差、不准确性和意外变化。本研究引入了一种强化学习方法，完全放弃预定义的方案，而是在结构构建过程中自适应生成构造序列。我们的方法运行于图结构状态表示和混合（参数化）作用空间，同时需要离散块选择和连续布置参数。由于结构的稳定性模拟计算量大，我们通过将单边边纳入图神经网络，将软演员-批判者（SAC）扩展到这种混合环境，开发了高效的探索策略。我们将算法HSAC与先前混合PPO（HPPO）方法进行比较，显示出显著更高的渐近性能和良好的样本效率。我们还展示了HSAC对超参数选择的鲁棒性和探索能力，能在性能不下降的情况下处理多达10个离散动作。最后，我们在物理双机器人配置上验证了我们的方法，成功在闭环执行中构建了带有3D打印块的跨度拱，确认了仿真训练策略能迁移到真实硬件。

Graph-Based Safe Reinforcement Learning for Multi-Agent Systems with Time-Varying Topology

基于图的多智能体系统安全强化学习，具有时间变化拓扑结构

• Authors: Xiao Sizhe, Dong Lijing, Bai Rui, Tan Xin
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.08802
• Pdf link: https://arxiv.org/pdf/2609.08802
• Abstract This paper presents a graph-based safe multi-agent reinforcement learning (MARL) framework for cooperative navigation with time-varying topology. To address the critical challenge of ensuring safety in environments with sensing constraints, a safety-decoupled mechanism is introduced through a Control Barrier-Like Function (CBLF) action screening layer. This mechanism bridges the gap between discrete LiDAR perception and continuous safety constraints, ensuring that physical safety constraints are strictly satisfied regardless of the learning progress. Building upon this safety foundation, a unified structural architecture is proposed, integrating a attention-based actor and a Graph Attention Network (GAT) centralized critic. The actor utilizes a value vector reconstruction mechanism that explicitly encodes relative geometric relations through a collaborative tracking error matrix, enabling scale-insensitive policy learning under time-varying communication topologies. Meanwhile, the GAT-based critic models evolving interaction structures for accurate global value estimation. The proposed framework is validated on real differential-drive robot platforms, and experimental results demonstrate superior stability and safety in dynamic scenarios with limited fields-of-view.
• 中文摘要 本文提出了基于图的安全多智能体强化学习（MARL）框架，用于时间变化拓扑的协作导航。为应对在有传感约束环境下确保安全的关键挑战，引入了通过控制障碍类功能（CBLF）动作筛查层的安全解耦机制。该机制弥合了离散激光雷达感知与连续安全约束之间的差距，确保无论学习进展如何，物理安全约束都能严格满足。基于这一安全基础，提出了统一的结构架构，集成了基于注意力的演员和图关注网络（GAT）中心批评者。该参与者采用价值向量重构机制，通过协同跟踪误差矩阵显式编码相对几何关系，实现在时间变化通信拓扑下的尺度不敏感策略学习。与此同时，基于GAT的批判者模型通过演化交互结构进行精确的全局值估计。该框架在真实差动驱动机器人平台上得到了验证，实验结果显示在视场有限的动态场景下表现出卓越的稳定性和安全性。

CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning

CAST：基于模型的强化学习中交替的状态值目标与扩展策略梯度

• Authors: Pietro Noah Crestaz, Mohamed Yassine Kabouri, Nicolas Mansard, Andrea Del Prete
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.08853
• Pdf link: https://arxiv.org/pdf/2609.08853
• Abstract Model-based reinforcement learning (MBRL) is a family of RL methods that learn a model of the environment and use it for action selection, making it well suited to robotics due to its sample efficiency. Combining learned models with online planning can further improve action selection, as the planner can exploit the model to find better actions than the learned policy alone. Recent methods combining learned policies with online planning typically learn the value of the policy rather than the stronger planner-guided behavior. We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy. CAST replaces the action-value critic with a state-value critic, trained using a target that combines a real planner-guided transition and an imagined transition under the current policy. The resulting value function corresponds to an alternating process between planner-guided behavior and the current policy, allowing it to benefit from the stronger planner behavior while being regularised by the policy being learned. We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2 quadruped performing a dynamic handstand.
• 中文摘要 基于模型的强化学习（MBRL）是一系列强化学习方法，学习环境模型并用于行动选择，因其样本效率高，非常适合机器人技术。将学习到的模型与在线规划结合，可以进一步提升行动选择，因为规划者可以利用模型找到比单纯学习策略更好的行动。近期方法将学习策略与在线规划结合，通常学习政策的价值，而非更强的规划者引导行为。我们介绍CAST（带交替状态值目标的批判者），利用规划者引导行为改善价值学习，同时规范当前政策的价值估计。CAST用状态价值批判者替代行动价值批判者，使用结合真实规划者引导过渡和当前政策下想象过渡的目标进行训练。所得价值函数对应于规划者引导行为与当前政策之间的交替过程，使其能够受益于更强的规划者行为，同时又被所学策略规范化。我们通过多种最先进方法评估了DeepMind Control和HumanoidBench套件中的CAST，并展示了成功转移到物理Unitree Go2四足动物中进行动态倒立的过程。

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

AuK 技术报告：语音生成与编辑的开源基础模型

• Authors: Ziyang Ma, Zhikang Niu, Wenming Tu, Tianrui Wang, Ruiqi Yan, Junxi Liu, Yanru Huo, Nickk Huang, Yang Liu, Qicong Xie, Zeyu Xie, Hui Wang, Haitao Li, Zixuan Jiang, Yalin Li, Jie Fang, Yifan Duan, Zeyue Tian, Guangzheng Li, Haina Zhu, Shuyi Wang, Jinwen Wang, Mingyu Cui, Tian Tan, Auden, Sen Liang, Steve Yves, Shan Yang, Liefeng Bo, Zilong Zheng, Kai Yu, Eng-Siong Chng, Xie Chen
• Subjects: Subjects: Sound (cs.SD); Computation and Language (cs.CL); Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2609.08936
• Pdf link: https://arxiv.org/pdf/2609.08936
• Abstract We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
• 中文摘要 我们引入了AuK，一个开源基础模型，通过自然语言指令和音频上下文的通用接口统一语音生成与编辑。为支持这一广泛的能力集，我们构建了约30.3亿条指令——音频实例和195万小时有效监督，跨越五大任务族：语音生成、内容编辑、增强与分离、副语言编辑和声学编辑。AuK结合了多模态大型语言模型用于语义条件训练，结合了语音、通用音频和音乐联合训练的VAE，以及一个混合整流流变换器，执行双流MMDiT模块，随后进行统一单流DiT模块生成。训练从仅代值预热开始，随后进入联合生成-编辑预训练。随后我们应用补充的后期训练策略：开放式编辑的人类反馈偏好优化和语音生成的基于奖励的强化学习。为降低推理成本，我们进一步通过一致性初始化和任务路由解耦DMD对模型进行提炼。最终的AuK-Flash在无分类器指导下执行4步推理，在匹配条件下实现全模型4.5倍的墙钟加速。实验显示，在零点和指令控制语音生成及通用指令引导编辑方面表现领先，同时在信号级恢复任务中保持竞争力。我们发布源代码和模型权重，以支持可重复性和进一步研究。

PlayTrain: An Efficient Reinforcement Learning Framework for LLM-Generated Adaptable JavaScript Games

PlayTrain：一个高效的强化学习框架，用于LLM生成的可适应JavaScript游戏

• Authors: Ryan Truong, Lance Ying, Samuel J. Gershman, Kazuki Irie
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.09059
• Pdf link: https://arxiv.org/pdf/2609.09059
• Abstract While many video-game environments (VGEs) have played crucial roles in advancing reinforcement learning (RL), developing novel VGEs or modifying existing ones to support new features, has been a laborious process requiring extensive hand-coding. Here we present PlayTrain, an RL framework that combines the abilities of large language models (LLMs) to robustly generate JavaScript (JS) games from a minimal human prompt, and an efficient pipeline that can run any JS game in a standard 'gym' environment. Not only are recent LLMs particularly good at writing JS code, but the JS format also allows users to easily play generated VGEs, while PlayTrain enables us to train RL agents on the exact same games. We demonstrate multiple use cases of PlayTrain, including cloning well-known Atari and ProcGen games in simple JS, where PlayTrain trains pixel-based agents end-to-end at over 1M agent-decisions per second on a single GPU node; and creating modified versions thereof (e.g., that support novel test sets, procedural generation logics, or game dynamics). Through PlayTrain, we reimagine RL VGE development: all we need is a single JS file, generated and modified through an LLM. We discuss promising future RL research directions that PlayTrain unlocks.
• 中文摘要 虽然许多视频游戏环境（VGE）在推动强化学习（RL）方面发挥了关键作用，但开发新颖的VGE或修改现有VGE以支持新功能，是一个需要大量手工编码的繁琐过程。这里我们介绍PlayTrain，这是一个结合大型语言模型（LLM）能力，能够从最小人工提示中稳健生成JavaScript（JS）游戏的RL框架，以及一个高效的流水线，可以在标准“健身房”环境中运行任何JS游戏。近年来的LLM不仅在编写JS代码方面表现尤为出色，JS格式还允许用户轻松玩生成的VGE，而PlayTrain则使我们能够在完全相同的游戏上训练强化学习代理。我们展示了PlayTrain的多种应用场景，包括用简单的JS克隆知名的Atari和ProcGen游戏，PlayTrain在单个GPU节点上以每秒超过100万次代理决策的速度进行基于像素的端到端训练代理;并创建修改版（例如支持新颖测试集、过程生成逻辑或游戏动态的版本）。通过PlayTrain，我们重新构想了RL VGE开发：我们只需要一个通过LLM生成和修改的单一JS文件。我们讨论PlayTrain解锁的未来强化学习研究方向。

ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR

ThinkPrior：RLVR 冷启动提示选择的零启动难度先验

• Authors: Tommy Sha, Skylar Zhai, Siqi Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.09075
• Pdf link: https://arxiv.org/pdf/2609.09075
• Abstract In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the optimizer. On Qwen2.5-Math-7B across sixteen seeds, ThinkPrior more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, while we detect no difference in final accuracy. On this 250-prompt pool the fixed-budget result is a reallocation rather than a net saving. The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget. The prior requires no target-policy rollout before the first selection, but the posterior thereafter uses target-policy outcomes.
• 中文摘要 在可验证奖励强化学习（RLVR）中，采用组相对策略优化（GRPO）训练，这里研究的无KL奖励-优势项依赖于组内奖励的变异。如果一组中所有部署正确或全部错误，其组内相对优势均为零;这些零优势沉默组不提供奖励-优势梯度，但均匀抽样花费了39%的运行部署时间。基于历史的提示选择必须先花费目标策略推广来估算难度，形成冷启动并产生滚动浪费;ThinkPrior则使用一次离线路径的外部锚点，构建一个零滚动困难的先验，在首次目标策略展开前。验证者评分的锚点传递率为Beta后验提供外部锚点初始化;ThinkPrior根据预期可学习性选择，然后从训练结果中更新，既不改变丢失也不改变优化器。在Qwen2.5-Math-7B的16个种子中，ThinkPrior将早期沉默组数量减半以上，并在第30步期间将浪费的推广减少近五分之一，同时我们检测到最终准确率没有差异。在这个250个提示词池中，固定预算的结果是重新分配，而非净节省。测量的ThinkPrior+DAPO组合使生成的推广减少了10.6%，而两个分支仍保持相同的3840推广更新预算。Before在首次选择前不要求目标政策推广，但之后选择使用目标政策结果。

ExecCritic: Learn to Test, Test to Improve for Coding Agents

执行批评：学会测试，测试以改进编码代理

• Authors: Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, Sharon Li, Jianfeng Gao
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2609.09133
• Pdf link: https://arxiv.org/pdf/2609.09133
• Abstract Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time. Code is publicly available at this https URL.
• 中文摘要 执行反馈可以引导编码代理进行正确的仓库修复，但前提是测试能够捕捉到问题请求的行为。代理生成的测试可以编码不完整或错误的行为目标;当同一轨迹同时写入补丁和测试时，错误可能一致并产生虚假的信心。我们引入了ExecCritic，结合了test-verify-revise支架和针对角色的强化学习方案，用于训练代理。支架将测试构建与源代码修复分开：测试代理独立生成仓库原生测试，fail-closed harness对测试进行限定并冻结，修复代理则根据执行反馈修订源代码而不更改测试内容。这两个角色都使用Qwen-3.5-35B-A3B作为骨干，并分别进行训练。在学习测试中，测试代理学习生成行为有效测试，以区分正确与错误补丁。在改进测试中，修复代理学习直接任务解析和反馈引导的修订。在 SWE-bench 验证中，测试质量决定反馈是否有效：保持基础修复代理固定，基础测试代理的测试将无测试基线的解决率从 61.2% 降至 57.3%，而 GPT-5.6-sol 测试则将解决率提升至 65.3%。角色特定后训练使 Qwen 测试代理的基础到黄金成功率从 22.2% 提升至 62.2%;两个后训练 Qwen 代理组合为 72.6%，较原始无测试基线提升 11.4 个百分点，且无更强模型或 Oracle 反馈。代码公开可访问此 https URL。

Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation

熵正则化秩掩盖策略优化，用于代码生成中的测试时强化学习

• Authors: Jiacheng Xu, Feng Chen, Xiuneng Xu, Bo An
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.09135
• Pdf link: https://arxiv.org/pdf/2609.09135
• Abstract Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal. To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus. We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.
• 中文摘要 现有的测试时强化学习（TTRL）方法通过对无标签测试时间任务的答案级自我投票获得奖励，但由于程序无法通过表层形式比较，因此无法直接提供可用的训练信号，因此在代码生成中失效。为使TTRL适用于代码生成，我们提出了探针驱动TTRL，该方法从问题陈述中构建无输出的探针输入，在这些探针上执行候选程序，并根据所得行为一致性定义探针共识奖励（PCR）。PCR为开放词汇程序提供了行为训练信号，但它并非完全可靠的验证器，且仍易受到虚假共识的奖励黑客攻击。因此，我们引入了熵正则化秩掩蔽策略优化（ERPO），该方法通过秩掩蔽将低PCR转换为保守的负更新，并通过熵上限控制策略漂移。在编码基准测试中，ERPO在域内适应和零样本传输方面显著提升了pass@1和 pass@k。

Keyword: diffusion policy

MemCorr-DP: Counterfactual Correspondence Conditioning for a Diffusion Policy Guided by a Reference

MemCorr-DP：以参考为导引的扩散政策的反事实对应条件

• Authors: Tan Su, Haoxiang Yang, Ruxin Wang, Binghui Xie
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.06615
• Pdf link: https://arxiv.org/pdf/2609.06615
• Abstract Behavior-cloned visuomotor policies can remain accurate near their training distribution yet fail when object position and camera viewpoint change together. A successful reference trajectory contains the geometry needed to transfer the same interaction, but the policy must align that geometry with the current scene and remain sensitive to it during denoising. To address these challenges, we present MemCorr-DP, a diffusion policy that lifts frozen RoMa v2 matches into explicit 3D relations between the current scene and the reference trajectory. A counterfactual paired objective assigns opposite behaviors the same physical state and noisy action while retaining reference-specific denoising targets. Mixed-condition fine-tuning then adapts the policy from ground-truth geometry to measured correspondence errors. Our strongest evaluation places the Door in the outermost position bands beyond the training support and changes the query camera by $\pm15^\circ$. Under this combined shift, MemCorr-DP achieves 96.67% closed-loop success, compared with 88.00% for a visual Transformer with the same action architecture. Objective ablations and reference interventions show that behavior responds to the selected reference, while matched controls favor the complete relation set over future motion or centroid geometry alone. These results support explicit 3D reference relations as a robust conditioning interface when spatial and viewpoint changes are compounded in the evaluated task.
• 中文摘要 行为克隆的视觉运动策略可以在训练分布附近保持准确，但当物体位置和摄像机视角同时变化时会失效。成功的参考轨迹包含传输相同交互所需的几何体，但策略必须将该几何体与当前场景对齐，并在去噪时保持对其敏感。为应对这些挑战，我们提出了MemCorr-DP扩散策略，将冻结的RoMa v2匹配提升为当前场景与参考轨迹之间的显式三维关系。反事实配对目标将相反行为赋予相同的物理状态和噪声动作，同时保留参考特定的去噪目标。混合条件微调随后将策略从地面真实几何调整为测量的对应误差。我们最强的评估将Door置于训练支持之外的最外层位置带，并将查询摄像机调整为$\pm15^\circ$。在这种综合转变下，MemCorr-DP闭环成功率为96.67%，而视觉变换器在相同动作架构下为88.00%。客观消融和参考干预表明行为对所选参考有响应，匹配对照则偏好完整关系集，而非仅未来运动或重心几何。这些结果支持在空间和视角变化叠加时，显式三维参考关系作为强健条件界面。

CAVEAT: Recurrent Multimodal Diffusion Planning for Mapless Aerial Exploration

注意：无图空中探测的重复多模态扩散规划

• Authors: Steven Visch, Nicolò Botteghi, Antonio Franchi, Barbara Bazzana
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.06623
• Pdf link: https://arxiv.org/pdf/2609.06623
• Abstract Can exploratory UAV waypoint sequences be generated from multimodal onboard observations and a fixed-dimensional recurrent internal state without maintaining a persistent global map in the deployed policy? We investigate this question through CAVEAT, a diffusion policy conditioned on a recurrent internal state updated from fused LiDAR, visual, and pose features and trained from trajectories generated by the map-based FUELv2 expert. Rolling inference partially warm-starts consecutive predictions, while a temporary local signed distance field provides heuristic obstacle guidance. Simulation results evaluate both inference mechanisms and compare CAVEAT with its demonstration-generating expert. Proof-of-concept experiments on a Flyability Elios 3 demonstrate partial exploration of a previously unseen indoor environment and target-directed visual servoing using a separately trained policy.
• 中文摘要 探索性无人机航点序列能否通过多模态机载观测和固定维度的循环内部状态生成，而无需在部署策略中保持持久的全局地图？我们通过CAVEAT来探讨这个问题，这是一种基于融合激光雷达、视觉和姿态特征更新的重复内部状态，并由基于地图的FUELv2专家生成轨迹训练的扩散策略。滚动推断部分热启动连续预测，而临时局部带符号距离场则提供启发式障碍指导。模拟结果评估了这两种推断机制，并将CAVEAT与其演示生成专家进行比较。在可飞行性Elios 3上的概念验证实验展示了对此前未见室内环境的部分探索，以及使用单独训练策略实现目标定向视觉伺服。

M3-Tele: A Unified Multimodal Teleoperational Framework for Compliant Whole-Body Mobile Manipulation

M3-Tele：一个统一的多模态远程操作框架，用于符合合规的全身移动操作

• Authors: Hengxiang Chen, Shenwen Deng, Yujian Ma, Gan Ma, Qiang Li, Nutan Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.07859
• Pdf link: https://arxiv.org/pdf/2609.07859
• Abstract Executing contact-rich tasks efficiently requires the seamless integration of whole-body coordination and physical compliance regulation. However, existing teleoperation and data-collection frameworks often overlook the joint consideration of multimodal perception and coordinated whole-body operation. This limitation can reduce the efficiency and quality of demonstration collection, thereby affecting the effectiveness of downstream policy learning. In this work, we present \textbf{M3-Tele}: A Unified \underline{M}ultimodal \underline{Tele}operational Framework for Compliant Whole-Body \underline{M}obile \underline{M}anipulation, enabling stable physical interaction and capturing aligned visual, tactile, force, and proprioceptive observations during task execution. Extensive experiments demonstrate that the proposed framework significantly improves contact-rich teleoperation performance. The proposed controller reduces the force tracking error from 4.132~N to 0.346~N, the contact loss from 2.46 to 0.02 events per trial and the tactile deformation error by 65\%. User studies across four mobile manipulation tasks also verify the reliability and usability of the proposed system. Furthermore, Diffusion Policy experiments highlight the value of joint tactile and force sensing.
• 中文摘要 高效执行接触丰富任务需要全身协调与物理合规调节的无缝整合。然而，现有远程操作和数据收集框架常常忽视多模态感知与协调整体操作的共同考虑。这一限制可能降低演示收集的效率和质量，从而影响下游策略学习的效果。本研究提出 \textbf{M3-Tele}：一个统一的\下线{M}末端/下线{Tele}操作框架，用于全体/下线/下线{下划线{M}分解，实现稳定的物理交互，并在任务执行过程中捕捉视觉、触觉、力和本体感觉的对齐观察。大量实验表明，所提出的框架显著提升了丰富的接触远程操作性能。所提控制器将力追踪误差从4.132~N降至0.346~N，接触损失从每次试验2.46次降至0.02次，触觉变形误差降低65%。用户对四项移动操作任务的研究也验证了所提系统的可靠性和可用性。此外，扩散政策实验强调了联合触觉和力感的价值。