Arxiv Papers of Today

生成时间: 2026-09-10 20:46:34 (UTC+8); Arxiv 发布时间: 2026-09-10 20:00 EDT (2026-09-11 08:00 UTC+8)

今天共有 25 篇相关文章

Keyword: reinforcement learning

SMCC-Empowered Digital Twins for Sensorless Monitoring in Large-Scale AI-Driven IoT Systems

SMCC赋能的数字孪生，用于大规模AI驱动的物联网系统中的无传感器监测

Authors: Vincenzo Sammartino
Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Cryptography and Security (cs.CR); Networking and Internet Architecture (cs.NI)
Arxiv link: https://arxiv.org/abs/2609.09161
Pdf link: https://arxiv.org/pdf/2609.09161
Abstract The deployment of AI-driven Digital Twins (DTs) in large-scale Internet-of-Things (IoT) ecosystems demands continuous, high-fidelity synchronization between the physical environment and its virtual replica. Conventional approaches rely on dense sensor deployments, which introduce prohibitive costs in terms of hardware, energy, and network bandwidth. In this paper, we propose SMCC-DT, an integrated Sensing-Memory-Communication-Computation (SMCC) framework that enables sensorless monitoring of physical assets by exploiting Integrated Sensing and Communication (ISAC) waveforms at the 6G Edge. Under the SMCC-DT paradigm, a single radio signal simultaneously extracts environmental telemetry (Sensing) and delivers it to an Edge server (Communication), where a large-scale AI model is loaded into constrained memory (Memory) and executed (Computation) to update the DT state. We formulate the DT synchronization problem as a cross-layer optimization that jointly allocates transmit power, beamforming vectors, memory partitions, and CPU frequency to minimize the end-to-end synchronization latency subject to sensing accuracy, throughput, memory capacity, and computational budget constraints. Because the resulting mixed-integer nonlinear program is NP-hard, we design a Proximal Policy Optimization (PPO)-based Deep Reinforcement Learning (DRL) agent, termed SMCCAGENT, that learns near-optimal resource allocation policies online. Extensive simulations over a 500-node industrial IoT testbed demonstrate that SMCC-DT reduces DT synchronization latency by 38.7% and total energy consumption by 27.4% compared to state-of-the-art orthogonal and compute-only baselines, while sustaining sensing accuracy above 95% and model inference throughput above 30 frames per second.
中文摘要 在大规模物联网（IoT）生态系统中部署AI驱动的数字孪生（DT）需要物理环境与其虚拟复制品之间持续且高保真度的同步。传统方法依赖密集传感器部署，这在硬件、能源和网络带宽方面带来了高昂的成本。本文提出了SMCC-DT，一种集成的感测-内存-通信-计算（SMCC）框架，通过利用6G边缘的集成感测与通信（ISAC）波形实现物理资产的无传感器监控。在SMCC-DT范式下，单一无线电信号同时提取环境遥测（感测）并将其传送到边缘服务器（通信），在那里将大型AI模型加载到受限内存（内存）中并执行（计算）以更新DT状态。我们将DT同步问题表述为跨层优化，联合分配传输功率、波束形成向量、内存分区和CPU频率，以最小化端到端同步延迟，受传感精度、吞吐量、内存容量和计算预算限制限制。由于最终混合整数非线性程序具有NP难度，我们设计了一个基于近端策略优化（PPO）的深度强化学习（DRL）代理，称为SMCCAGENT，能够在线学习近优资源分配策略。在500节点工业物联网测试平台上进行的广泛模拟表明，SMCC-DT相比最先进的正交和纯计算基线，将DT同步延迟降低38.7%，总能耗降低27.4%，同时保持传感准确率超过95%，模型推理吞吐量超过30帧每秒。

Learning to Fly: Stable Vision-Guided UAV Servoing with Compact Target-Centric Cues and Reinforcement Learning

学习飞行：稳定视觉引导无人机服务，配备紧凑目标中心提示和强化学习

Authors: Saurbh Singh Jamwal, Nived Chebrolu
Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2609.09234
Pdf link: https://arxiv.org/pdf/2609.09234
Abstract Vision-guided reinforcement learning for Unmanned Aerial Vehicles (UAVs) remains challenging due to unstable policy optimisation, aggressive exploration, and the cost of high-dimensional visual perception. In this work, we investigate long-horizon UAV visual servoing using compact target-centric cues combined with low-dimensional sensor measurements. Rather than learning directly from RGB images, lightweight target segmentation provides image-space offsets and relative depth, which are combined with quadrotor velocity and projected-gravity measurements into a compact 12D policy observation. We compare Direct PPO with three matched-budget curriculum strategies: a Visual curriculum that progressively expands target placement difficulty, a Dynamics curriculum that gradually relaxes action constraints and smoothing, and a Joint curriculum that combines both progressions. All strategies reach comparable nominal performance, with complementary advantages across tracking metrics. Observation ablations show that proprioceptive measurements are critical for stable flight and image-space cues for target alignment, while explicit depth is not necessary for strong performance in the evaluated setting. Against tuned classical visual-servo controllers, learned policies show greater robustness to strong control and visual perturbations, while the Visual curriculum exhibits the smallest degradation under unseen target motion. Overall, the results demonstrate that compact target-centric representations can support robust long-horizon aerial visual servoing and that visual curriculum training can improve robustness to dynamic distribution shifts despite limited gains in nominal performance.
中文摘要 无人机（UAV）的视觉引导强化学习因策略优化不稳定、探索激烈以及高维视觉感知成本而面临挑战。本研究研究利用紧凑的目标中心线索结合低维传感器测量，研究长视野无人机视觉伺服。轻量级目标分割不直接从RGB图像学习，而是提供图像空间偏移和相对深度，结合四旋翼速度和投影重力测量，形成紧凑的12维政策观测。我们将直接PPO与三种匹配预算课程策略进行比较：逐步扩展目标定位难度的视觉课程、逐步放松动作约束和平滑的动态课程，以及结合两者进阶的联合课程。所有策略的名义表现相当，且在跟踪指标上互补优势。观察消融显示，本体感觉测量对于稳定飞行和图像空间线索对目标对准至关重要，而显性深度对于评估环境中强性能并非必需。与调谐的经典视觉伺服控制器相比，学习策略对强控制和视觉扰动表现出更强的韧性，而视觉课程在未见目标运动下表现出最小的退化。总体而言，结果表明紧凑的以目标为中心的表示可以支持稳健的长视距空中视觉伺服，视觉课程训练能提升对动态分布变化的鲁棒性，尽管名义性能提升有限。

Video-MOPD: Multi-Teacher On-Policy Distillation for Video Understanding

视频-MOPD：多教师政策提炼以促进视频理解

Authors: Zhenxin Qin, Peng Shi, Cong Han, Yinlong Qian, Zequn Jie, Lin Ma
Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
Arxiv link: https://arxiv.org/abs/2609.09300
Pdf link: https://arxiv.org/pdf/2609.09300
Abstract Video understanding demands a convergence of complementary capabilities across perception, temporal understanding, and complex reasoning, which are difficult to jointly optimize within a single model. We introduce Video-MOPD-8B, an open-weight model dedicated to video understanding tasks. To fundamentally enhance its capabilities, we conduct targeted reinforcement learning (RL) optimization across three core domains: video temporal grounding (VTG), general video comprehension, and video STEM reasoning. We then unify their complementary capabilities via Multi-Teacher On-Policy Distillation (MOPD), which consolidates expert knowledge by supervising student-generated trajectories with routed teacher feedback. We further introduce Reliability-Aware Informative Sampling (RAIS), which selects examples with consistently reliable teacher supervision and large teacher-student performance gaps. Together, these components enable Video-MOPD-8B to achieve coordinated and comprehensive performance gains across diverse video understanding tasks. Extensive experiments on comprehensive benchmarks covering general video understanding, temporal grounding, video reasoning, and video STEM tasks demonstrate that Video-MOPD-8B achieves state-of-the-art performance among existing models at a comparable scale. The trained model weights are available at this https URL.
中文摘要 视频理解需要感知、时间理解和复杂推理等互补能力的融合，而这些能力难以在单一模型中共同优化。我们介绍视频-MOPD-8B，一个专注于视频理解任务的开放权重模型。为了从根本上提升其能力，我们在三个核心领域进行了针对性强化学习（RL）优化：视频时间基础化（VTG）、通用视频理解和视频STEM推理。随后，我们通过多教师政策提炼（MOPD）统一它们的互补能力，通过监督学生生成的轨迹和教师反馈来整合专家知识。我们还进一步引入可靠性感知信息抽样（RAIS），该系统选择教师监督稳定可靠且师生表现差距较大的示例。这些组成部分共同使视频MOPD-8B能够在多样化的视频理解任务中实现协调且全面的绩效提升。对涵盖一般视频理解、时间基础、视频推理和视频STEM任务的综合基准测试的广泛实验表明，Video-MOPD-8B在可比规模下实现了现有模型中的最先进性能。训练好的模型权重可在此 https URL 获取。

From Learning to Control: Data-Driven Multi-Agent Reinforcement Learning for Multivariable Control in a Microalgae Bioprocess

从学习到控制：数据驱动的多智能体强化学习，用于微藻生物工艺中的多变量控制

Authors: Juan D. Gil, Ehecatl Antonio Del Rio Chanona, José Luis Guzmán, Manuel Berenguel
Subjects: Subjects: Systems and Control (eess.SY)
Arxiv link: https://arxiv.org/abs/2609.09313
Pdf link: https://arxiv.org/pdf/2609.09313
Abstract Effective control of bioprocesses is particularly challenging due to the intrinsic nonlinearity and dynamic variability of living-cell systems. In microalgae-based photobioreactors (PBRs), maintaining stable pH and dissolved oxygen (DO) levels is critical for optimal growth and productivity, yet their strong coupling and sensitivity to environmental fluctuations make multivariable control difficult. This study proposes a novel hybrid offline-online Multi-Agent Reinforcement Learning (MARL) framework for simultaneous pH and DO regulation, leveraging Deep Deterministic Policy Gradient (DDPG) agents to achieve a fully data-driven and model-free control solution. The agents are trained using historical data generated by an expert system, eliminating the need for direct experimentation with the environment. After deployment, the agents operate autonomously, continuously fine-tuning their policies daily to adapt to evolving process dynamics and reject fast transient disturbances. Experimental validation in an open, industrial-scale PBR at the University of Almeria demonstrated the framework's capability to maintain stable operation under realistic conditions. The results confirm that model-free MARL control provides a robust and adaptive alternative for complex bioprocess environments.
中文摘要 由于活细胞系统本身具有非线性和动态变异性，有效控制生物过程尤其具有挑战性。在基于微藻的光生物反应器（PBR）中，保持pH和溶解氧（DO）水平的稳定对最佳生长和生产力至关重要，但其强烈耦合性和对环境波动的敏感性使多变量控制变得困难。本研究提出了一种新型混合离线-在线多代理强化学习（MARL）框架，用于同时调节pH和DO，利用深度确定性策略梯度（DDPG）智能体实现完全数据驱动且无模型的控制解决方案。智能体使用专家系统生成的历史数据进行训练，无需直接对环境进行实验。部署后，智能体自主运行，每日不断微调策略以适应不断变化的过程动态，并拒绝快速瞬态干扰。在阿尔梅里亚大学的开放式工业规模PBR实验验证显示，该框架能够在现实条件下维持稳定运行。结果证实，无模型MARL控制为复杂生物工艺环境提供了稳健且适应性的替代方案。

Smart Adaptive Computing Across the Continuum: LLMs in IoT-Edge-Cloud Resource Management

跨连续体的智能自适应计算：物联网-边缘-云资源管理中的大型语言模型

Authors: Antonino Vaccarella, Lanpei Li, Vincenzo Lomonaco, Massimo Coppola
Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
Arxiv link: https://arxiv.org/abs/2609.09348
Pdf link: https://arxiv.org/pdf/2609.09348
Abstract Managing resources across IoT, edge, and cloud layers calls for continuous, context-aware decisions under constraints that rarely stay fixed. Deep reinforcement learning (DRL) handles this class of problems well, and large language models (LLMs) are increasingly used to augment DRL pipelines, yet the architectural relationship between the two is seldom made explicit. We build on Wang et al.'s taxonomy of Continuum Orchestration Systems employing DRL techniques and extend it with two further dimensions. The AI Augmentation Paradigm measures how LLMs are exploited, while the Feedback channel captures whether and through which system path the execution feedback returns to the LLM in order to close the MAPE control loop at the LLM Orchestration layer. We apply this taxonomy to six recent system architectures and find a common gap, as none combines full LLM orchestration with full agent-layer feedback in a Cloud Continuum setting. We relate this gap to a missing cross-tier feedback abstraction, bridging the incommensurable per-tier signals and the LLM Orchestrator.
中文摘要 跨物联网、边缘和云层管理资源需要在很少固定的约束条件下持续且具上下文感知的决策。深度强化学习（DRL）很好地处理了这类问题，大型语言模型（LLM）越来越多地被用于增强DRL流水线，但两者之间的架构关系很少被明确说明。我们基于Wang等人提出的连续体编排系统分类法，采用DRL技术，并扩展了两个维度。AI增强范式衡量LLM的利用情况，而反馈通道则捕捉执行反馈是否以及如何通过哪条系统路径返回LLM，以闭合LLM编排层的MAPE控制循环。我们将该分类法应用于六种近期系统架构，发现了一个共同的空白，因为没有哪种将完整的LLM编排与云连续体环境中的完整代理层反馈结合起来。我们将这一空白归因于缺失的跨层反馈抽象，连接了不可通约的每层信号与LLM编排器。

A Decade of Bayesian Optimization for Controller Tuning and Robot Learning: Tutorial, Review, and Future Prospects

十年来的贝叶斯优化与控制器调校与机器人学习：教程、评测与未来展望

Authors: David Stenger, Paul Brunzema, Johanna Menn, Alexander von Rohr, Angela P. Schoellig, Sebastian Trimpe
Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
Arxiv link: https://arxiv.org/abs/2609.09403
Pdf link: https://arxiv.org/pdf/2609.09403
Abstract In the past decade, Bayesian optimization (BO) has emerged as a powerful and adaptable framework for automatic controller tuning and robot learning. This article offers a comprehensive overview of the state-of-the-art in BO, designed to support both researchers and practitioners in understanding recent advancements, practical applications, and future research directions. We begin by adopting a practitioner's perspective, illustrating how to effectively set up BO through a representative controller tuning example. We position BO within the broader context of learning paradigms, ranging from deep reinforcement learning to data-driven control, and highlight scenarios where BO is most advantageous. Next, we discuss the diverse range of BO methods that have been developed to tackle complex problems and specific applications. This article provides a unified perspective on the current landscape of BO, emphasizing its relevance to control systems and robotics, and it highlights future prospects by identifying key research challenges and promising avenues for advancing BO in the field. This includes addressing a significant gap in the BO landscape: the lack of standardized benchmark problems specifically for control-related applications. To foster future research and ensure rigorous evaluation, we start an effort towards a lightweight benchmark suite for control engineering and robotics. We also present metrics and best practices to facilitate direct comparisons between new BO algorithms and established state-of-the-art methods.
中文摘要 在过去十年中，贝叶斯优化（BO）已成为一种强大且适应性的自动控制器调优和机器人学习框架。本文全面概述了BO领域的最新进展，旨在支持研究人员和实践者理解最新进展、实际应用及未来研究方向。我们首先采用从业者的视角，通过一个代表性的控制器调优示例，展示如何有效设置BO。我们将BO置于更广泛的学习范式中，涵盖深度强化学习到数据驱动控制，并突出BO最有利的场景。接下来，我们讨论了为解决复杂问题和具体应用而开发的多样化BO方法。本文提供了对BO当前格局的统一视角，强调其与控制系统和机器人的相关性，并通过识别关键研究挑战和推动BO在该领域的前景展望。这包括解决 BO 领域中一个重要空白：缺乏针对控制相关应用的标准化基准问题。为促进未来研究并确保严谨评估，我们开始努力开发一个轻量级的控制工程和机器人基准套件。我们还提供了指标和最佳实践，以便直接比较新的 BO 算法与既有的先进方法。

Actuator Dynamics Curricula for Narrow-Viability Tasks in Legged Robot Learning

执行器动力学课程，适用于腿式机器人学习中窄可行性任务

Authors: Kousheek Chakraborty, Chandan K. Rajendra, Ayham Alharbat, Abeje Y. Mersha
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2609.09492
Pdf link: https://arxiv.org/pdf/2609.09492
Abstract Reinforcement learning has produced capable controllers across a broad range of legged-robot tasks, but a subset of these tasks fail to converge under standard training: those for which most exploration trajectories terminate before producing useful gradient signal. To address such tasks we introduce the \emph{Actuator Dynamics Curriculum}, a procedure that initializes joint stiffness at a high value and anneals it toward the system-identified value as completed episode lengths grow. Using a cart-pole system as a representative example, we show that higher closed-loop joint natural frequency under critical damping enlarges the viability kernel of the underlying Markov Decision Process, increasing the fraction of initial states from which the task is feasible. We validate the kernel monotonicity on the cart-pole and apply the curriculum to a quadrupedal-to-handstand transition on the Boston Dynamics Spot, a narrow-viability task where training under fixed identified stiffness plateaus at a policy that never completes the transition. The trained policy executes the transition in simulation across 10 seeds and transfers to hardware. More broadly, our results suggest that simulated actuator dynamics is a useful axis along which to design curricula for tasks in which exploration is bottlenecked by termination conditions rather than by reward signal.
中文摘要 强化学习已在广泛的腿部机器人任务中培养出了能力强的控制器，但其中一部分任务在标准训练下未能收敛：即大多数探索轨迹在产生有用梯度信号前就终止的任务。为应对此类任务，我们引入了\emph{执行器动力学课程}，该程序将关节刚度初始化为高值，并随着完成的片段长度增加将其退火至系统识别值。以车杆系统为代表例，我们展示了临界阻尼下更高的闭环关节固有频率扩大了底层马尔可夫决策过程的可行性核，增加了任务可行初始状态的比例。我们在车杆上验证了核单调性，并将课程应用于波士顿动力学点的四足到倒立过渡，这是一个狭窄可行性任务，在固定刚度下训练，且策略从未完成过渡。训练策略在模拟中执行10个种子的过渡，并转移到硬件上。更广泛地说，我们的结果表明，模拟执行器动力学是设计课程的有用轴线，适用于探索被终止条件而非奖励信号限制的任务。

CityPlanner: A Sandbox Agent for Executable Urban Planning

城市规划师：可执行城市规划的沙盒代理

Authors: Wentao Zhang, Jingyuan Wang, Zetong Zhou, Yifan Yang, Wenrui Wang
Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2609.09578
Pdf link: https://arxiv.org/pdf/2609.09578
Abstract Urban planning is a real-world spatial optimization problem that requires selecting feasible actions from large candidate spaces under practical objectives such as cost and service quality. Existing optimization and reinforcement learning methods are effective for fixed formulations, but often depend on task-specific representations and constraint handling. We propose \emph{CityPlanner}, a sandbox-agent framework for executable urban planning. CityPlanner introduces \emph{UrbanSandbox}, a unified file-based environment where agents inspect task files, generate plans, run evaluators, and revise decisions based on executable feedback. To make learning tractable, we further propose atomic-task reinforcement learning, which decomposes long sandbox trajectories into \emph{BuildPlan} for initial construction and \emph{ImprovePlan} for feedback-based refinement. Experiments on a real-world benchmark show that CityPlanner consistently outperforms heuristic, task-specific RL, and general LLM-agent baselines. Ablations verify the contributions of UrbanSandbox, atomic-task RL, and iterative deployment. We release the code and dataset at this https URL
中文摘要 城市规划是一个现实世界的空间优化问题，需要在成本和服务质量等实际目标下，从大型候选空间中选择可行的行动。现有的优化和强化学习方法对固定表述有效，但通常依赖于任务特定的表示和约束处理。我们提出了\emph{CityPlanner}，一个用于可执行城市规划的沙盒-代理框架。CityPlanner引入了\emph{UrbanSandbox}，这是一个统一的文件环境，代理可以检查任务文件、生成计划、运行评估器，并基于可执行反馈修订决策。为了使学习变得易于操作，我们进一步提出了原子任务强化学习，将长沙盒轨迹分解为\emph{BuildPlan}用于初始构建，\emph{ImprovePlan}用于反馈优化。在现实世界基准测试上的实验显示，CityPlanner始终优于启发式、任务特定强化学习和通用LLM代理基线。消融验证了UrbanSandbox、原子任务强化学习和迭代部署的贡献。我们将代码和数据集发布于此 https URL

A Risk-Sensitive and Uncertainty-Aware Decision-Making and Control Framework for Safe and Robust Autonomous Driving

一个风险敏感且具不确定性的决策与控制框架，实现安全且稳健的自动驾驶

Authors: Zhuoren Li, Ran Yu, Weiqi Zhang, Ming Liu, Lu Xiong, Chen Sun, Bo Leng
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2609.09650
Pdf link: https://arxiv.org/pdf/2609.09650
Abstract Reinforcement learning (RL) has demonstrated considerable potential for autonomous driving decision-making. However, its deployment in urban autonomous driving, particularly at highly interactive unsignalized intersections, remains challenging, as learned policies may struggle to maintain both safety and robust decision-making in complex traffic situations. Conventional safety-filtering approaches typically employ fixed conservative constraints, which may improve safety at the cost of excessive intervention and degraded traffic efficiency. To address these limitations, we propose a Risk-sensitive and Uncertainty-aware Decision-making and Control (RUDC) framework for safe and robust autonomous driving. RUDC couples risk-sensitive distributional RL with ensemble-based policy uncertainty quantification, jointly accounting for tail risks in return distributions and uncertainty in learned policies. An uncertainty-aware high-order control barrier function (HOCBF)-based safety correction mechanism adaptively adjusts constraint strictness according to policy uncertainty, while a learnable residual predictor compensates for CBF model mismatches and discretization errors. Extensive simulations at unsignalized intersections demonstrate that RUDC achieves a favorable balance among safety, efficiency, and robustness, outperforming representative safe RL baselines under both nominal and challenging OOD and long-tail scenarios while satisfying real-time requirements.
中文摘要 强化学习（RL）已展现出在自动驾驶决策中的巨大潜力。然而，其在城市自动驾驶中的应用，尤其是在高度互动的无信号路口，仍具挑战性，因为学习策略在复杂交通情境下难以同时维持安全和稳健决策。传统的安全过滤方法通常采用固定的保守约束，这可能以过度干预和交通效率下降为代价提升安全性。为解决这些局限性，我们提出了一个风险敏感且不确定性意识的决策与控制（RUDC）框架，用于安全稳健的自动驾驶。RUDC将风险敏感的分布式强化学习与基于集合的策略不确定性量化结合，共同考虑回报分布中的尾部风险和学习策略中的不确定性。基于不确定性感知的高阶控制障碍函数（HOCBF）安全修正机制根据策略不确定性自适应调整约束严格性，而可学习的残差预测变量则补偿CBF模型不匹配和离散化误差。在无信号交叉口的大量模拟表明，RUDC在安全性、效率和鲁棒性之间取得了良好平衡，在名义和挑战性极高的户外和长尾场景下均优于代表性安全强化基线，同时满足实时需求。

HiRAD: A Flexible Large-Scale AGV Routing System

HiRAD：一种灵活的大规模全地形车路由系统

Authors: Yunjie Huang, Ruizhong Wu, Mengxuan Zhang, Frodo Kin Sun Chan, Yan Nei Law, Lei Li
Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2609.09752
Pdf link: https://arxiv.org/pdf/2609.09752
Abstract Automatic Guided Vehicles (AGVs) substantially boost warehouse throughput, but routing large-scale AGV fleets remains challenging. Classical Multi-Agent Pathfinding solvers suffer from exploding combinatorial complexity and super-quadratic runtime, while relying on idealized grid or piecewise-linear motion models that mismatch real-world kinematics. Recent Reinforcement Learning (RL) solutions improve flexibility via decentralized agent policies but depend on discretized spatiotemporal representations, require millions of episodes to converge, and incur full-map observation at every step, which leads to large models, slow convergence, and high inference latency that violates real-time industrial control constraints. To address these bottlenecks, we propose HiRAD, a hierarchical RL framework for continuous-space AGV routing with real-time guarantees: (1) a step-level spatiotemporal representation that translates continuous motion into a differentiable RL problem, (2) a hierarchical strategy that splits heading choice from velocity control to reduce the action space, and (3) an asynchronous event-driven decision pipeline that lowers inference complexity from O(n^2) to O(n) and cuts per-step latency by as much as 71 percent. Across random graphs and two warehouse maps, HiRAD reduces makespan by 45 percent to 63 percent and shortens end-to-end runtime.
中文摘要 自动导引车辆（AGV）大幅提升了仓库吞吐量，但大规模AGV车队的路由依然充满挑战。传统的多智能体寻路求解器面临爆炸式的组合复杂性和超二次运行时间，同时依赖理想化的网格或分段线性运动模型，这些模型与现实运动学不匹配。近期的强化学习（RL）解决方案通过去中心化代理策略提升了灵活性，但依赖离散化的时空表示，需要数百万集收敛，且每一步都需全图观测，导致模型庞大、收敛缓慢和高推理延迟，违反实时工业控制约束。为解决这些瓶颈，我们提出了HiRAD，这是一个用于连续空间AGV路由的分层强化学习框架，具有实时保证：（1）将连续运动转化为可微化强化学习问题的步级时空表示，（2）将航向选择与速度控制分离以缩小动作空间的分层策略，以及（3）异步事件驱动决策流水线，将推理复杂度从O（n^2）降至O（n），并将每步延迟降低多达71%。在随机图和两个仓库地图中，HiRAD将完成时长缩短45%至63%，并缩短端到端运行时间。

SocialRL: Refining LLMs' Social Intelligence through Multi-turn Reinforcement Learning and Reward Design

SocialRL：通过多轮强化学习和奖励设计优化大型语言模型的社会智能

Authors: Jianing Wang, Xintao Wang, Aili Chen, Jie Shi, Hongcheng Guo, Jun Gao, Wenxuan Zhao, Chengkun Lang, Yuanli Guo, Yanghua Xiao
Subjects: Subjects: Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2609.09764
Pdf link: https://arxiv.org/pdf/2609.09764
Abstract Social intelligence enables agents to read social context, infer intent, and adapt over sustained dialogue. As language models become autonomous collaborators, it is central to building effective and trustworthy human-AI interaction. Existing reinforcement learning methods optimize single-turn utterances and sparse outcome rewards, producing short-sighted policies that struggle to manage goal-relationship tensions across multi-turn interactions. We propose SocialRL, a multi-turn reinforcement learning framework addressing both challenges. First, we apply multi-turn reinforcement learning using PPO that propagates delayed outcome rewards back to each turn, enabling long-horizon planning. Second, we design six process reward dimensions capturing the goal-relationship trade-off, including goal advancement, relational attunement, contextual coherence, etc. A reward model dynamically generates fine-grained scoring criteria for each dimension, while a stage-aware weight schedule prioritizes relationship-building in early turns, goal advancement mid-way, and balanced closure late. Across multiple social-dialogue benchmarks, SocialRL improves Goal Achievement by an average of 9.2 percentage points over the corresponding Base models. These results demonstrate the effectiveness of SocialRL across synthetic and real social scenes, as well as standard and challenging social scenarios.
中文摘要 社会智能使智能体能够读取社会语境、推断意图，并在持续对话中做出调整。随着语言模型成为自主协作者，它对于构建有效且值得信赖的人机交互至关重要。现有的强化学习方法优化了单回合的话语和稀疏的结果奖励，产生了短视的策略，难以管理多回合互动中的目标关系紧张关系。我们提出了SocialRL，一种多回合强化学习框架，解决这两个挑战。首先，我们应用多回合强化学习，利用PPO将延迟的结果奖励传递回每个回合，实现长期规划。其次，我们设计六个过程奖励维度，捕捉目标-关系权衡，包括目标推进、关系调谐、语境连贯性等。奖励模型动态生成每个维度的细粒度评分标准，而阶段感知权重计划则优先在早期回合建立关系，中期优先推进目标，后期实现平衡收尾。在多个社交对话基准测试中，SocialRL比相应基础模型平均提升目标达成9.2个百分点。这些结果展示了SocialRL在合成和现实社交场景以及标准和挑战性社交场景中的有效性。

Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward

带证据的认知：用现实定型奖励弥合验证差距

Authors: Eshwar Reddy M, Sourav Karmakar
Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2609.09776
Pdf link: https://arxiv.org/pdf/2609.09776
Abstract Frontier gains in language-model reasoning come from reinforcement learning on reasoning traces and are concentrated in domains with a cheap, sound verifier. We argue the field's binding constraint is the verification gap: no scalable, incorruptible reward for reasoning outside formal domains. We make four contributions. (1) Theory: in a joint-Gaussian model of best-of-N selection, verifier-gold correlation rho is the exact exchange rate between test-time compute and capability, and an unsound verifier pays a polynomial penalty N^(1/rho^2); a margin-free copula form predicts realized soundness of real LLM judges to 4% median error. (2) Demonstration: in program-synthesis testbeds with executable ground truth, including a pre-registered scaled replication, unsound verifiers lose Soundness-under-Pressure as optimization grows (0.94 to 0.32 at N=4096) while a sound verifier improves monotonically; reality-anchored settlement beats a frozen verifier under i.i.d. and adversarial pressure, driving the hacking gap from ~0.27 to ~0; soundness scales log-linearly with settled labels, with on-policy settlement ~10x more label-efficient than random labeling. With real LLM judges and unit-test execution as gold, a weak judge loses soundness under best-of-N (p<0.001), a stronger judge is more robust, and selection alone manufactures +0.53 hacking gaps from honest samples. Under real GRPO training, a frozen reward model traces the full overoptimization curve (executed reward collapses 90%) while the same model refit on a 10% settlement stream preserves 6x the executed reward. (3) Paradigm: proof-carrying cognition, where reasoning steps are typed probabilistic claims priced by a self-built world model trained only on held-out reality and settled by proper scoring rules. (4) Benchmark: we specify Soundness-under-Pressure as the headline metric for a reality-settled reasoning benchmark.
中文摘要 语言模型推理的前沿收益来自于对推理轨迹的强化学习，且主要集中在拥有廉价且可靠的验证器的领域。我们认为该领域的约束是验证差距：在形式领域外推理没有可扩展且不可腐蚀的奖励。我们提出四项贡献。（1）理论：在联合高斯模型中的N选取最佳选择中，验证者-黄金相关rho是测试时计算与能力的精确交换率，不健全的验证者将承担多项式惩罚N^（1/rho^2）;无边距的连索形式预测真实LLM评判的实现正确性为4%的中位误差。（2）演示：在带有可执行基层真实时的程序综合测试平台中，包括预注册的缩放复制，不健全的验证者随着优化的增长（N=4096时为0.94到0.32）而失去压力下的健全性，而声音验证器则单调地提升;基于现实的结算在I.I.D.和对抗压力下击败冻结的验证器，将黑客差距从~0.27提升到~0;合理性与确定标签呈对数线性增长，策略上的稳定效率比随机标记高出约10倍。在真实LLM评判和单元测试执行如黄金的情况下，弱评判在最佳N（p<0.001）下会失去健全性，更强的评判更稳健，仅凭选择即可从诚实样本中制造+0.53个黑客差距。在真实GRPO训练下，冻结奖励模型描绘了完整的过度优化曲线（执行奖励崩溃90%），而同一模型在10%结算流上重新调整则保留6倍执行奖励。（3）范式：带证明的认知，推理步骤由自建的世界模型定价，仅基于保留的现实训练，并通过适当的评分规则确定。（4）基准：我们指定“压力下的健全性”作为现实确定推理基准的主线指标。

BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

BRACE：异步强化学习中为陈旧批评者锚定的贝尔曼残余修正

Authors: Guanqun Zhao, Zijun Xie, Binbin Zheng, Jiafeng Lu, Enlei Gong, Zeyu Chen
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2609.09783
Pdf link: https://arxiv.org/pdf/2609.09783
Abstract Asynchronous reinforcement learning has become the standard way to scale training for language models, but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on asynchronous LLM training corrects the actor and leaves this bias unaddressed, while the off-policy value correction of classical RL does not carry over to long-horizon agentic tasks, since a short correction horizon leaves the regression target free of the reward and a long one lets the product of importance ratios drift exponentially with the trajectory length. We propose BRACE, an anchored Bellman-residual correction for stale value models. BRACE bounds the correction horizon to a prefix of policy tokens and anchors a constant-weight Monte-Carlo tail beyond it, which separates policy correction from reward propagation. BRACE improves mean@1 on BrowseComp-Plus by $2.4\%$ over the strongest baseline, runs $2.46\times$ faster per step than synchronous training, and remains stable $50$ updates off-policy.
中文摘要 异步强化学习已成为语言模型训练扩展的标准方式，但由此产生的策略滞后使批评者偏向陈旧行为策略。现有异步大型语言模型训练工作纠正了行为者，并未解决这一偏见，而经典强化学习的非策略值修正则无法适用于长视野的代理任务，因为短修正视野使回归目标没有奖励，而较长的修正视野则使重要性比乘积随轨迹长度呈指数级漂移。我们提出了BRACE，一种针对陈旧价值模型的锚定贝尔曼残差修正。BRACE将修正视界限制为策略令牌的前缀，并在其后锚定一个恒权蒙特卡洛尾部，从而将策略修正与奖励传播分开。BRACE 在 BrowseComp-Plus 上的 mean@1 比最强基线高出 2.4 美元，每步速度比同步训练快 2.46 美元，且在非政策下更新稳定，需 50 美元。

Decision Transformer for UAV-Mounted RIS-Assisted Dynamic D2D Communications

无人机安装的RIS辅助动态D2D通信决策变换器

Authors: Yaxuan Liu
Subjects: Subjects: Artificial Intelligence (cs.AI); Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2609.09885
Pdf link: https://arxiv.org/pdf/2609.09885
Abstract This paper studies unmanned aerial vehicle (UAV)-mouted reconfigurable intelligent surface (RIS)-assisted device-to-device (D2D) communication with stochastic link activation. It models UAV motion and attitude, time-varying Rician angles, and angle-dependent RIS reflection. A joint optimization of UAV trajectory, attitude, and RIS phases is formulated to maximize average sum rate under mobility, energy, and hardware constraints. The problem is addressed using deep reinforcement learning and a Decision Transformer trained on expert trajectories from multiple scenarios. Results demonstrate effective cross-scenario generalization, with zero-shot transfer outperforming direct DRL transfer and online fine-tuning achieving competitive performance with fewer interactions.
中文摘要 本文研究了无人机（UAV）设计的可重构智能地面（RIS）辅助设备间（D2D）通信，并以随机链路激活为基础。它模拟了无人机的运动与姿态、时间变化的Rician角度以及角度依赖的RIS反射。对无人机轨迹、姿态和RIS阶段进行了联合优化，以在机动性、能量和硬件约束条件下最大化平均加和率。该问题通过深度强化学习和基于多场景专家轨迹训练的决策变换器解决。结果证明了跨场景推广的有效性，零射击传输优于直接日程传输，在线微调则以更少的交互实现了竞争性能。

HaWMPO: Hallucination-Aware World Model-based Policy Optimization for Generalist Robot Policy

HaWMPO：幻觉感知世界模型的通用机器人政策优化

Authors: Zengjue Chen, Peidong Liu, Jiawei Li, Qi Wang
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2609.09941
Pdf link: https://arxiv.org/pdf/2609.09941
Abstract Generalist robot policies have demonstrated strong generalization across robotic manipulation tasks, yet their success rates remain limited in com- plex long-horizon scenarios. Recent methods improve Visual-Language-Action (VLA) policies through online reinforcement learning on real robots, but such training relies on costly physical interactions, suffers from low sample efficiency, and may introduce hardware and safety risks. World models offer a promising alternative by enabling policy optimization with imagined rollouts. However, long-horizon rollouts generated by world models often suffer from prediction hal- lucinations, producing biased state transitions that can mislead policy learning. To address this issue, we propose Hallucination-aware World Model-based Pol- icy Optimization (HaWMPO), a closed-loop reinforcement learning pipeline for VLA policy post-training with world models. Specifically, HaWMPO introduces an action-conditioned hallucination-aware model to estimate the reliability of gen- erated image sequences, and incorporates hallucination scores into group relative policy optimization through a Reward-Soft mechanism, suppressing unreliable ac- tion chunks during training. On the LIBERO benchmark, HaWMPO achieves the best average success rate, with gains of 15.0% over the base model and 2.8% over the strongest baseline; real-world experiments on a G1 robot further validate its effectiveness, raising the average success rate on two manipulation tasks from 67.5% to 80.0%.
中文摘要 通用机器人策略已在机器人操作任务中展现出强烈的泛化性，但在复合长视野场景中其成功率仍然有限。最新方法通过在线强化学习改进视觉-语言-行动（VLA）策略，但此类训练依赖昂贵的物理交互，采样效率低，且可能带来硬件和安全风险。世界模型通过通过想象的展开实现策略优化，提供了有前景的替代方案。然而，由世界模型生成的长视野推广常常存在预测半卢cination，产生偏向状态转换，误导策略学习。为解决这一问题，我们提出了一种具备幻觉感知的世界模型政治优化（HaWMPO），这是一种闭环强化学习流水线，用于世界模型进行VLA策略后训练。具体来说，HaWMPO引入了一种动作条件幻觉感知模型，用于估计生成图像序列的可靠性，并通过奖励软机制将幻觉评分纳入组相对策略优化，抑制训练过程中不可靠的行动块。在LIBERO基准测试中，HaWMPO实现了最佳的平均成功率，较基础模型提升15.0%，较最强基线提升2.8%;G1机器人的实际实验进一步验证了其有效性，将两个操作任务的平均成功率从67.5%提升至80.0%。

VLX-VR: An Agentic-Aware Video Reasoning Model

VLX-VR：一种代理感知视频推理模型

Authors: Sheng Li, Peng Liu, Qianqian Zhang, Tiancheng Zhao
Subjects: Subjects: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
Arxiv link: https://arxiv.org/abs/2609.09985
Pdf link: https://arxiv.org/pdf/2609.09985
Abstract Real-world video understanding requires integrating visual, audio, textual, and temporal evidence distributed across a video. Yet many pipelines use a fixed video context and single-pass inference, limiting adaptive evidence acquisition when observations are incomplete, ambiguous, or conflicting. We present VLX-VR, an agentic-aware video reasoning model trained within a video reasoning framework defined by a Think--Memory--Observation loop. At each step, VLX-VR determines the needed evidence, invokes read_memory or write_memory, incorporates the returned Observation, and decides whether to continue or produce the task output. We train VLX-VR with multimodal data, including videos and agent trajectories, using reinforcement learning to learn evidence acquisition, memory use, and termination. On MINERVA, VLX-VR achieves state-of-the-art performance among the models included in our comparison, with 78.79% accuracy. Under the original three duration groups, its accuracies are 76.70%, 78.73%, and 80.92%, with a cross-duration accuracy variance of 2.97~$\mathrm{pp}^2$. On correctly answered samples, 96.20% of VLX-VR's reasoning traces are consistent with the MINERVA reference reasoning traces and the evidence described by them, while approximately 75.80% of all evaluated samples satisfy both answer correctness and this evidence-grounded trace criterion. These results show strong performance and broadly stable behavior across durations, while counting, state changes, causal reasoning, and spatial perception remain challenging.
中文摘要 现实世界的视频理解需要整合分布在视频中的视觉、音频、文本和时间证据。然而，许多流程采用固定视频上下文和单次推断，当观察不完整、模糊或冲突时，限制了自适应证据的获取。我们介绍VLX-VR，一种在视频推理框架内训练的智能感知视频推理模型，该框架由思考-记忆-观察循环定义。在每一步，VLX-VR确定所需证据，调用read_memory或write_memory，整合返回的观察，并决定是继续还是生成任务输出。我们通过强化学习训练VLX-VR，包括视频和代理轨迹，学习证据获取、记忆使用和终止。在MINERVA上，VLX-VR在我们比较中的模型中达到了最先进的性能，准确率达78.79%。在原始三个时长组中，其准确率分别为76.70%、78.73%和80.92%，跨时长准确性方差为2.97~$\mathrm{pp}^2$。在正确回答的样本中，96.20%的VLX-VR推理痕迹与MINERVA参考推理迹及其描述的证据一致，而约75.80%的评估样本同时满足答案正确性和该证据基础痕迹标准。这些结果显示出跨时长表现强且行为大致稳定，而计数、状态变化、因果推理和空间感知仍具挑战性。

Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning

以数据为中心的金融推理后期培训：挖掘、蒸馏与可验证学习

Authors: Zhirayr Hayrapetyan, Andrei Kalmykov, Denis Kokosinskii, Dmitry Stanishevskii, Dmitry Zmitrovich
Subjects: Subjects: Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2609.10113
Pdf link: https://arxiv.org/pdf/2609.10113
Abstract Financial text, textbooks, and question-answer pairs are abundant, but only a small fraction is directly usable for reasoning-focused post-training. Existing QA pairs often lack explicit reasoning, sufficient context, or reliably verifiable answers, while textbooks must first be transformed into synthetic training examples. We present a data-centric pipeline that constructs complementary corpora by mining open-source reasoning traces, distilling financial instruction data, and generating knowledge-graph-guided question-answer pairs from financial educational material. After semantic deduplication, three lightweight sequence classifiers select finance-relevant examples, reject under-specified questions, and identify tasks suitable for reinforcement learning with compact rule-based verifiers. For model adaptation, we study supervised fine-tuning and reinforcement learning, while self-distilled fine-tuning and post-training model merging are used to prevent the loss of financial capabilities already present in the starting model. We evaluate the adapted language models using FINESSE-Bench, reporting aggregate performance and changes relative to their starting checkpoints. Across the selected comparisons, ordinary SFT reduces FINESSE-Bench accuracy by 3.2-4.0 percentage points, whereas self-distilled SFT improves over the corresponding starting models by 1.0-2.8 points. Equal-weight merging recovers 3.0 points over its SFT parent and finishes 0.9 points above the original model; GRPO on hard tasks adds 0.4 points after self-distilled SFT or 3.0 points when applied directly to verifiable tasks. These results show that retention-aware adaptation can improve financial reasoning without the regressions observed after ordinary SFT.
中文摘要 金融文本、教材和问答对数量丰富，但只有少部分能直接用于以推理为重点的后期培训。现有的质量保证对通常缺乏显式推理、足够的上下文或可靠可验证的答案，而教材必须先转化为合成训练示例。我们提出了一个以数据为中心的流程，通过挖掘开源推理痕迹、提炼财务指令数据，并从金融教育材料生成知识图引导的问答对，构建互补语料库。语义去重后，三个轻量级序列分类器选择金融相关示例，拒绝未明确的问题，并识别适合强化学习的任务，使用紧凑的基于规则的验证器。在模型适应方面，我们研究监督微调和强化学习，同时使用自我精细调和训练后模型合并以防止初始模型中已有的财务能力的丧失。我们使用FINESSE-Bench评估适应后的语言模型，报告相对于起始检查点的总体性能和变化。在所选比较中，普通SFT使FINESSE-Bench准确率降低3.2-4.0个百分点，而自提纯SFT相比相应起始模型提升1.0-2.8个百分点。等权合并较其SFT父模型提升3.0个百分点，结束时比原始模型高出0.9个百分点;困难任务的GRPO在自提纯SFT后增加0.4个百分点，直接应用于可验证任务时则增加3.0个百分点。这些结果表明，保留意识适应能够在没有普通SFT后回归的情况下改善财务推理。

Assembling Two Parts in One Hand

一手组装两个部件

Authors: Liuao Pei, Tianyue Wu, Hui Zhang, Ping Luo, Jie Song
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2609.10137
Pdf link: https://arxiv.org/pdf/2609.10137
Abstract A hallmark of human dexterity is the cooperative use of fingers, where different fingers take on distinct yet coordinated roles to accomplish fine manipu- lation, such as capping a pen with the hand that holds it. We study this finger-level coordination through in-hand assembly: mating two rigid objects within a single dexterous hand, with no second arm and no fixture. We present a reinforcement learning formulation to solve this problem in a unified framework, which is driven by a goal relative pose between the two parts. Finger coordination is shaped by a function-based auxiliary reward and regularized toward a single human reference pose, while domain randomization and a fusion of historical proprioception and object observation confer robustness to occlusion-induced estimation noise. The same recipe solves three different assembly tasks (Bottle, Syringe, and Marker). Trained purely in simulation, the policies transfer zero-shot to hardware with a single camera, demonstrating robustness to state-estimation errors caused by oc- clusion. Our experiments also reveal that in-hand assembly places demands on hand morphology and can serve as a benchmark for modern robotic hand systems. Videos and code are available at this https URL.
中文摘要 人类灵巧度的一个标志是手指的协作使用，不同手指承担不同但协调的角色，以完成精细的操作，例如用握笔的手盖住笔盖。我们通过手部组装来研究这种手指层面的协调：将两个刚性物体结合在一只灵巧的手中，没有第二只手臂和固定装置。我们提出了一种强化学习的表述，以统一框架解决这个问题，该框架由两部分之间的目标相对姿势驱动。手指协调由功能辅助奖励塑造，并规范化为单一的人类参考姿势，而领域随机化以及历史本体感受与物体观察的融合则赋予对阻塞诱发估计噪声的韧性。同一方案解决了三种不同的组装任务（瓶子、注射器和标记）。这些策略纯以仿真训练，能够将零拍摄传输到单一相机的硬件，展现了对因oc-clusion引起的状态估计误差的鲁棒性。我们的实验还表明，手工组装对手部形态提出了要求，可以作为现代机器人手部系统的基准。视频和代码可在此 https 网址获取。

Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

为什么要采样你能列举的？基因组工具选择的精确策略优化

Authors: Haoyue Liu, Xiaoyu Ma, Ye Chen, Zhichao Wang, Xiaoying Tang
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2609.10221
Pdf link: https://arxiv.org/pdf/2609.10221
Abstract Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and (2) precomputes the reward of each question--subset pair into an exhaustive table, removing frozen-reasoner calls from the training loop entirely. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average and up to 14.20, while a standard on-demand GRPO schedule would require 2.4 times as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from 2.36 to 1.40.
中文摘要 基于冻结推理器的强化学习已成为教授策略中调用哪些外部工具的常见配方。我们证明，在可枚举整个工具子集空间的专业科学环境中，这种方法在结构上会不匹配。在那里，一小部分重复计算能力覆盖了整个领域，因此工具子集空间是组合的但足够小以便枚举，GRPO仍能从少数抽样的部署中估算动作期望。更糟的是，随着训练成功，近似会下降：当策略聚焦于首选子集时，它们会被重新采样，抽样奖励发生碰撞，群体归一化优势消失。在基因组推理中，在统一参考策略下，无奖励信号的问题比例从0.2%上升到GRPO训练后20.8%。作为解决办法，我们引入了FGPO（全组策略优化），它（1）对每个工具子集进行评分并优化精确的动作预期，使每次更新都能看到完整的动作空间，（2）预先计算出每个问题-子集对的奖励，形成一个穷尽表，完全将冻结推理器调用从训练循环中剔除。在五个冻结推理器和三个基因组基准测试中，FGPO在所有15个设置中平均优于GRPO6.75分，最高可达14.20分，而标准按需GRPO计划则需要2.4倍的冻结推理器奖励评估，而在GenomeQA上，FGPO将每题调用工具数量从2.36%减少到1.40%。

Hierarchical and Permutation-Invariant Feature Transformation Learning via Policy-Guided Embedding Search

通过策略引导嵌入搜索实现层级和置换不变特征转换学习

Authors: Rui Liu, Tao Zhe, Yanyong Huang, Sankha Narayan Guria, Xiao Luo, Wei Fan, Yanjie Fu, Dongjie Wang
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2609.10225
Pdf link: https://arxiv.org/pdf/2609.10225
Abstract Feature transformation improves predictive performance on tabular data by constructing informative abstractions from raw features. Recent generative approaches encode transformation knowledge into continuous embedding spaces for efficient exploration of candidate strategies, but face three key limitations: (1) overlooking hierarchical relationships between low-level features, operations, and high-level abstractions; (2) enforcing order-sensitive embeddings on inherently permutation-invariant transformation sequences, thereby introducing systematic bias; and (3) relying on gradient-based search, which is ill-suited to non-convex transformation spaces. We propose a framework with two complementary components. First, a permutation-invariant hierarchical module captures interactions across features, operations, and abstraction levels, with a self-attention pooling mechanism that maps semantically equivalent structures to consistent embeddings aligned with downstream performance. Second, a policy-guided multi-objective reinforcement learning strategy initializes the search from empirically strong seeds and jointly optimizes predictive accuracy and transformation efficiency. Extensive experiments on diverse tabular benchmarks demonstrate the effectiveness and robustness of our framework against strong baselines. Our code and data are publicly available at: this https URL.
中文摘要 特征变换通过从原始特征构建信息抽象，提升了表格数据的预测性能。近期生成方法将变换知识编码到连续嵌入空间中，以高效探索候选策略，但面临三大限制：（1）忽视低层特征、操作与高层抽象之间的层级关系;（2）对本质置换不变变换序列强制执行顺序敏感嵌入，从而引入系统性偏差;（3）依赖基于梯度的搜索，不适合非凸变换空间。我们提出一个由两个互补组成部分组成的框架。首先，置换不变的层级模块捕捉特征、操作和抽象层级的交互，并具备自注意池机制，将语义等效结构映射到符合下游性能的一致嵌入。其次，采用策略引导的多目标强化学习策略，从经验强的种子初始化搜索，共同优化预测准确性和转换效率。在多种表格基准测试上的大量实验展示了我们框架在强基线下的有效性和稳健性。我们的代码和数据公开于：https URL。

Learning Terrain-Adaptive Humanoid Locomotion on Granular Terrain

在颗粒地形上学习地形自适应类人运动

Authors: Junnosuke Kamohara, Feiyang Wu, Andy Ningan Zong, Daniel I. Goldman, Yashwanth Nakka, Seth Hutchinson, Ye Zhao
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2609.10286
Pdf link: https://arxiv.org/pdf/2609.10286
Abstract Humanoid locomotion on granular terrain remains a significant challenge due to its complex foot-terrain interaction dynamics that are difficult to model. Existing approaches either ignore granular contact dynamics or incorporate simplified normal force models with heuristic tangential components. In this work, we present a physics-grounded granular contact model based on three-dimensional resistive force theory (3D RFT) and efficiently simulate granular terrain for reinforcement learning (RL) training. Unlike traditional rigid contact models and simplified granular contact models with ad-hoc heuristics, our contact solver produces physically accurate granular intrusion dynamics without resorting to heuristics. It captures realistic penetration and tangential drag during training, enabling the policy to learn behaviors that transfer reliably to real-world granular terrain where rigid contact models fail. To adapt to varying terrain conditions, we train a terrain-adaptive locomotion controller via teacher-student RL, using a variational autoencoder to encode terrain information into a compact latent representation. Simulation studies using material point method (MPM) with NVIDIA Newton demonstrate that our method generalizes to unseen granular terrains, achieves a significantly higher success rate than baselines, and demonstrates zero-shot terrain identification and adaptation. We further validate our approach through extensive hardware experiments across diverse real-world granular terrains including basalt, dry sand, and beach sand. To the best of our knowledge, this is the first demonstration of agile humanoid locomotion on real-world granular terrain. Project page: this https URL
中文摘要 由于复杂的足部-地形相互作用动力学难以建模，人形在颗粒状地形上的运动仍是一个重大挑战。现有方法要么忽略粒状接触动力学，要么采用带有启发式切向分量的简化法向力模型。本研究提出了基于三维电阻力理论（3D RFT）的物理基础颗粒接触模型，并高效模拟颗粒地形以实现强化学习（RL）训练。与传统的刚性接触模型和带有临时启发式的简化颗粒接触模型不同，我们的接触求解器无需依赖启发式方法，就能生成物理上精确的粒状侵入动力学。它在训练过程中捕捉了真实的穿透和切向阻力，使策略能够学习能够可靠地转移到刚性接触模型失效的真实颗粒地形中的行为。为适应变化的地形条件，我们通过师生强化学习训练地形自适应运动控制器，使用变分自编码器将地形信息编码为紧凑的潜在表示。与NVIDIA Newton合作的材料点法（MPM）模拟研究表明，我们的方法能够推广到未见颗粒地形，成功率显著高于基线，并展示了零射地形识别和适应能力。我们通过在多种真实颗粒地形（包括玄武岩、干沙和海滩沙）进行大量硬件实验进一步验证了我们的方法。据我们所知，这是首次在真实颗粒地形上演示敏捷类人机动。项目页面：此 https URL

TRACE: Training Reasoning Agents for Causal Exploration with Synthesized Rewards

TRACE：训练推理代理进行因果探索，并以合成奖励进行

Authors: Rui Sun, Zhan Shi, Bing He
Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2609.10315
Pdf link: https://arxiv.org/pdf/2609.10315
Abstract Reinforcement learning with verifiable rewards (RLVR) has advanced language-model reasoning in domains such as mathematics and code, where objective answers are inexpensive to check. Diagnostic reasoning over complex data lacks this advantage: establishing the true cause of an anomaly often requires costly expert investigation and may remain ambiguous after the fact. We ask whether this asymmetry of verification can instead be engineered. We sample an intervention, inject it into a controlled simulator, and generate the observations it would produce. The hidden intervention provides an oracle label and objective reward, while the agent must still investigate noisy, confounded, and distributed evidence. We instantiate this approach in TRACE, a digital-advertising diagnostic environment with 12 root causes and fine-grained segment attribution. Agents investigate each episode using Python and SQL and must identify both the root cause and, when applicable, the affected segment assignment. On a held-out 235-episode test set, the strongest prompted baseline, Claude Opus 5, reaches 0.686 FullAttr@1. Supervised fine-tuning raises Qwen3.5-35B-A3B from 0.159 to 0.637, and subsequent RL with synthesized rewards reaches 0.757, outperforming all evaluated prompted baselines, including frontier closed-source models and a prompted Qwen3.5-122B-A10B model. The resulting policy also uses substantially fewer tool calls than the prompted 35B base. These results provide evidence that access to a scalable, objective training signal can be a more important constraint than model scale alone. More broadly, simulation-based verification can make otherwise ambiguous diagnostic reasoning tasks amenable to scalable reinforcement learning.
中文摘要 带可验证奖励的强化学习（RLVR）在数学和代码等领域具有先进的语言模型推理能力，因为客观答案的验证成本较低。对复杂数据进行诊断推理缺乏这一优势：确定异常的真实原因通常需要昂贵的专家调查，且事后可能保持模糊。我们探讨是否可以工程化这种验证的不对称性。我们采样干预，注入受控模拟器，生成其产生的观测值。隐式干预提供预言标签和客观奖励，而代理仍需调查噪声、混淆和分布式证据。我们在TRACE中实现这一方法，TRACE是一个数字广告诊断环境，拥有12个根本原因和细粒度的片段归因。代理使用Python和SQL调查每个事件，必须识别根因及（适用时）受影响的片段分配。在一份保留的235集测试集中，最强的提示基线Claude作品5达到0.686 FullAttr@1。监督微调将Qwen3.5-35B-A3B从0.159提升到0.637，后续合成奖励的强化学习达到0.757，优于所有评估的提示基线，包括前沿闭源模型和提示Qwen3.5-122B-A10B模型。最终策略使用的工具调用数量也远少于提示35B基础。这些结果表明，访问可扩展、客观的训练信号可能比单纯模型规模更为重要的约束。更广泛地说，基于模拟的验证可以使原本模糊的诊断推理任务更适合可扩展强化学习。

Multi-Agent Reinforcement Learning for Autonomous UAV Exploration in Wildfire Response

多智能体强化学习用于野火响应中的自主无人机探索

Authors: Caden Chandra, Jerry Ng
Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2609.10433
Pdf link: https://arxiv.org/pdf/2609.10433
Abstract This study develops a deep reinforcement learning framework for training Unmanned Aerial Vehicle (UAV) agents to navigate and monitor simulated wildfire environments. Results show that agents learn increasingly stable and effective behaviors over time, as demonstrated by converging loss trends, improved reward signals, and more consistent navigation patterns such as fire-boundary tracking. Overall, these findings highlight the potential of deep reinforcement learning (DRL) based UAV systems for autonomous wildfire monitoring and suggest that environmental structure and reward design influence policy effectiveness.
中文摘要 本研究开发了一个深度强化学习框架，用于训练无人机（UAV）特工导航和监控模拟野火环境。结果表明，特工随时间学习的行为越来越稳定且有效，表现为收敛的损失趋势、改进的奖励信号以及更一致的导航模式（如火线边界跟踪）。总体而言，这些发现凸显了基于深度强化学习（DRL）的无人机系统在自主野火监测中的潜力，并表明环境结构和奖励设计会影响政策效果。

ConvMem: Convolutional Memory for Long-Context Reasoning

ConvMem：用于长上下文推理的卷积记忆

Authors: Hongming Zhang, Zhaozhen Gu, Fengshuo Bai, Ming Hao, Qingyang Zhang, Yuanyuan Wang, Shiyang Tang, Yanna Wang, Bo Xu
Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2609.10441
Pdf link: https://arxiv.org/pdf/2609.10441
Abstract While Large Language Models (LLMs) have demonstrated impressive capabilities, they often struggle with extremely long contexts due to fixed context limits. To address this, sequential approaches like MemAgent extend the effective context by reading text in segments and iteratively updating a fixed-size memory. However, this sequential paradigm suffers from high latency and requires costly reinforcement learning (RL) training, which can lead to overfitting on specific datasets. To overcome these limitations, we propose ConvMem, a training-free, highly parallelizable framework that reformulates long-context reasoning as a hierarchical convolution. Inspired by CNNs, ConvMem treats an LLM prompted with a specific query as a convolutional kernel. This kernel summarizes text segments hierarchically, shortening the reasoning path from a linear chain into a logarithmic tree. Specifically, ConvMem integrates \textit{Configurable Strides} and \textit{Skip Connections} to ensure robust evidence capture and propagation, while employing \textit{Multi-Kernel Convolution} to decompose complex queries into disentangled semantic channels. This design not only mitigates error accumulation but also enables massive parallelization across both text segments and reasoning threads. Experiments on RULER-HotpotQA and RULER-2WikiMultiHopQA demonstrate that ConvMem outperforms training-free baselines and avoids the risk of overfitting to parametric priors often observed in RL-trained models on out-of-distribution tasks.
中文摘要 虽然大型语言模型（LLM）展现出令人印象深刻的能力，但由于固定上下文限制，它们常常难以处理极长上下文。为此，像 MemAgent 这样的顺序方法通过分段读取文本并迭代更新固定大小的内存来扩展有效上下文。然而，这种顺序范式存在较高的延迟，且需要昂贵的强化学习（RL）训练，可能导致特定数据集上的过度拟合。为克服这些限制，我们提出了 ConvMem，一个无训练、高度可并行化的框架，将长上下文推理重新表述为层级卷积。受卷积神经网络启发，ConvMem 将触发特定查询的大型语言模型视为卷积核。该内核以层级方式总结文本段，将推理路径从线性链缩短为对数树。具体来说，ConvMem 集成了 \textit{可配置步伐}和 \textit{跳过连接}，以确保稳健的证据捕获和传播，同时利用 \textit{多核卷积}将复杂查询分解为解开的语义通道。这种设计不仅减少了错误积累，还实现了文本段和推理线程之间的大规模并行化。在 RULER-HotpotQA 和 RULER-2WikiMultiHopQA 上的实验表明，ConvMem 优于无训练基线，避免了在强化学习模型中常见的参数先验过拟合风险。

Keyword: diffusion policy

JEPA Policy: Diffusion-Free Imitation Learning via Paired Action and Future Representation Prediction

JEPA政策：通过配对动作和未来表示预测实现无扩散模仿学习

Authors: Jie Xu, Kangjin Yu, Ziyi Jin, Junjie Gao, Liqing Chen, Yixian Li, Shuai Tian, Zhongpu Xia
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2609.09630
Pdf link: https://arxiv.org/pdf/2609.09630
Abstract Standard behavior cloning supervises actions without explicitly constraining the future representation paired with each demonstrated action chunk. We introduce JEPA Policy, a diffusion-free framework that uses the action chunk and its observed future representation as paired training targets. Action and future-representation tokens interact in a shared Transformer and are refined through two forward passes. Future prediction can therefore shape the representation used to generate actions. Dual-branch and gradient-routing controls attribute the gain to this shared topology rather than to an auxiliary prediction head alone. Across nine simulated tasks, JEPA Policy improves mean success over the action-only MIP baseline and outperforms Diffusion Policy under the evaluated configurations, while adding 0.29 ms to MIP's model latency. A five-task, 630-episode physical-robot study produces the same pooled ranking. Further audits find no complete representation collapse under action supervision and identify a task-conditioned failure-ranking signal in future-prediction error. These results support paired future-representation supervision as a practical approach to low-latency visuomotor imitation without iterative generative sampling.
中文摘要 标准行为克隆监督动作，但不明确限制与每个已展示动作块配对的未来表示。我们介绍了JEPA策略，这是一个无扩散框架，将动作块及其观察到的未来表示作为配对训练目标。动作和未来表示令牌在共享的Transformer中相互作用，并通过两次前向传递进行细化。因此，未来预测可以塑造用于生成动作的表示方式。双分支和梯度路由控制将收益归因于这种共享拓扑，而非仅依赖辅助预测头。在九个模拟任务中，JEPA策略在仅动作MIP基线上提升平均成功率，并在评估配置下优于扩散策略，同时使MIP模型延迟增加了0.29毫秒。一项包含五个任务、630集的物理机器人研究也得出相同的合并排名。进一步审计发现在动作监督下没有完全表征坍缩，并识别出任务条件的失败排序信号，存在未来预测误差。这些结果支持将配对未来表征监督作为一种实用的低延迟视觉运动模拟方法，无需迭代生成抽样。