生成时间: 2026-09-15 21:16:34 (UTC+8); Arxiv 发布时间: 2026-09-15 20:00 EDT (2026-09-16 08:00 UTC+8)

今天共有 66 篇相关文章

Keyword: reinforcement learning

Harnessing human expertise for high-precision robotic assembly in industrialized construction: A sample-efficient installer-in-the-loop interactive reinforcement learning framework

利用人类专业知识实现工业化建筑中高精度机器人组装：一个高效的安装在环互动强化学习框架

• Authors: Zekai Jin, Huiguang Wang, Xiaoning Sun, Yi Shao
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.13234
• Pdf link: https://arxiv.org/pdf/2609.13234
• Abstract Industrialized construction imposes stringent precision requirements on robotic assembly of modular components such as prefabricated window units. In tolerance-critical operations, the central bottleneck is not only mechanical clearance but also converting tacit installer expertise into data-efficient autonomy under sparse acceptance feedback, contact variability, and millimeter-scale constraints. We present an installer-in-the-loop interactive reinforcement learning framework that acquires expertise through offline teleoperated demonstrations, sparse event-driven binary takeovers at contact-failure boundaries, and acceptance-aligned terminal rewards, logged under a unified schema for traceable offline-to-online adaptation. A temporally abstract action-sequence policy built on Q-chunking with Flow Q-Learning captures multimodal recovery maneuvers under sparse terminal rewards, while a non-updating warm-start phase stabilizes the offline-to-online transition. The framework is evaluated in MuJoCo across the workflow from suction acquisition through clearance-limited seating, under structured staging and end-to-end randomized placement. Within a defined stress-test regime with 2 mm per-side clearance, bounded pose perturbations, and friction randomization, the pipeline attains 100\% autonomous seating with 12--15 min of cumulative installer supervision over 3.0 h of online training, and reaches the 95\% success milestone in approximately 0.5 h and 1.5 h in the two experiments. We also report wall-clock adaptation time, cumulative takeover minutes, intervention-rate decay, and stage-wise failure attribution to inform supervision budgeting. Ablations isolate the complementary contributions of temporal abstraction, installer intervention, and warm-start value calibration.
• 中文摘要 工业化建筑对模块化组件（如预制窗户单元）的机器人组装施加了严格的精度要求。在公差关键操作中，核心瓶颈不仅是机械间隙，还包括在稀疏验收反馈、接触变异性和毫米级约束下，将默式安装者的专业知识转化为数据高效的自主性。我们提出了一个安装在环的交互强化学习框架，通过离线远程操作演示、接触失效边界的稀疏事件驱动二进制接管以及在统一模式下记录的接受对齐终端奖励获得专业知识，以便可追溯离线到在线适配。基于Q块和Flow Q-Learning的时序动作序列策略，捕捉了终端奖励稀疏下的多模态恢复机动，而非更新的热启动阶段则稳定了离线到在线的过渡。该框架在MuJoCo中评估了从吸力获取到限间隙座位的工作流程，采用结构化分级和端到端随机放置。在定义的应力测试范围内，每侧间隙2毫米、有界位态扰动和摩擦随机化，流程通过3.0小时的在线培训累计12-15分钟安装员监督，实现100%自主入座，并在两个实验中约0.5小时和1.5小时内达到95%成功里程碑。我们还报告了墙钟适应时间、累计接管分钟数、干预率衰减及各阶段失效归因，以指导监督预算。消融分离时间抽象、安装人员干预和热启动值校准的互补贡献。

Self-Evolving AI for Humanoids: Mechanisms, Safety, and Evaluation of Post-Deployment Self-Improvement

类人生物自我进化的人工智能：机制、安全性及部署后自我提升的评估

• Authors: Loc X. Nguyen, Avi Deb Raha, Huy Q. Le, Eui-Nam Huh, Dusit Niyato, Choong Seon Hong
• Subjects: Subjects: Robotics (cs.RO); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.13236
• Pdf link: https://arxiv.org/pdf/2609.13236
• Abstract Humanoid robots are becoming an important part of embodied artificial intelligence, driven by advances in reinforcement learning for locomotion, world models for prediction, and vision-language-action models for general control. However, most of these systems remain static after deployment. A policy is trained offline for a fixed objective and then frozen, even though the tasks, environments, and robot bodies keep drifting over time. An emerging paradigm of self-evolving agents aims to address this problem by allowing systems to improve from their own post-deployment experience. Since most existing studies focus on disembodied software agents, this survey examines how self-evolution changes when an agent has a physical body. We first define self-evolution for humanoids and represent a deployed robot using a state tuple that includes its policy, perception, memory, workflow, and body. This state is updated by an evolution operator in a slow outer loop with a lifelong objective. We then organize the literature into four complementary mechanisms of self-evolution, presented in increasing order of autonomy: self-learning, self-adaptation, self-optimization, and self-generation. Since changes to a humanoid can introduce physical hazards, we treat safety and uncertainty as key design dimensions of the evolution operator, and further formulate admissible evolution as a constraint enforced by a world-model verification gate within a human-oversight envelope. Finally, we present that evaluation should track the robot's evolving trajectory rather than a fixed checkpoint, and we identify the lack of a benchmark designed specifically for self-evolving humanoids. Moreover, we outline open challenges spanning AI algorithms, on-board systems, and governance.
• 中文摘要 类人机器人正成为具身人工智能的重要组成部分，这得益于强化学习用于移动、世界模型用于预测以及视觉-语言-动作模型的进步。然而，大多数系统在部署后保持静态。策略会被针对固定目标离线训练，然后冻结，尽管任务、环境和机器人身体随着时间不断漂移。一种新兴的自我进化代理范式旨在解决这一问题，允许系统从自身部署后的体验中改进。鉴于大多数现有研究聚焦于无实体的软件代理，本综述探讨了当智能体拥有实体身体时自我进化的变化。我们首先定义了类人生物的自我进化，并用包含策略、感知、记忆、工作流程和身体的状态元组表示已部署的机器人。该状态由进化操作员在缓慢的外循环中更新，目标为终身。随后我们将文献组织为四种互补的自我进化机制，按自主性递增排序：自我学习、自我适应、自我优化和自我生成。由于对类人生物的改变可能引入物理危害，我们将安全性和不确定性视为进化操作员的关键设计维度，并将可接受进化进一步表述为由人类监督范围内的世界模型验证门执行的约束。最后，我们提出评估应跟踪机器人的进化轨迹，而非固定检查点，并指出缺乏专门为自我进化类人生物设计的基准。此外，我们概述了涵盖人工智能算法、机载系统和治理的未解决挑战。

GzDRL: Reproducible and Scalable Deep Reinforcement Learning with Gazebo

GzDRL：带有凉亭的可重复且可扩展深度强化学习

• Authors: Amal Dev Haridevan, Junjie Kang, Jinjun Shan
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13243
• Pdf link: https://arxiv.org/pdf/2609.13243
• Abstract We present GzDRL, a novel single-process reinforcement learning (RL) framework for Gazebo that overcomes longstanding bottlenecks in scalable, reproducible robotics experimentation. Unlike conventional middleware-based RL-Gazebo integrations that suffer from nondeterminism and irreproducibility, GzDRL introduces a systematic, middleware-free environment-stepping mechanism that directly synchronizes agent actions and physics updates. This design enables deterministic, high-throughput data collection, efficient vectorization, and reproducible RL training and evaluation. Comprehensive benchmarks demonstrate that GzDRL achieves the highest workstation throughput among the evaluated frameworks while remaining competitive with GPU-accelerated simulators on laptop hardware, and maintains precise agent-environment synchronization, multi-agent scalability, and experiment-level reproducibility. We further validate sim-to-real transfer by deploying learned policies directly onto a physical quadrotor, without fine-tuning. Our results establish GzDRL as an accessible and reproducible platform for advancing RL in robotics and automation.
• 中文摘要 我们介绍GzDRL，一种针对Gazebo的新型单进程强化学习（RL）框架，克服了可扩展、可复现机器人实验中长期存在的瓶颈。与传统基于中间件的RL-Gazebo集成存在非确定性和不可重现性不同，GzDRL引入了系统化、无中间件的环境步进机制，直接同步代理动作和物理更新。该设计实现了确定性高通量数据收集、高效向量化以及可复现的强化学习训练与评估。综合基准测试显示，GzDRL在评估框架中实现最高工作站吞吐量，同时在笔记本电脑硬件上与GPU加速模拟器保持竞争力，并保持精确的代理-环境同步、多代理可扩展性和实验级可重复性。我们进一步验证了模拟到现实的传输，通过将学习到的策略直接部署到物理四旋翼上，无需微调。我们的结果确立了GzDRL作为推动机器人和自动化强化学习的可访问且可重复的平台。

An Evolutionary Computation Framework for Multi-Agent Q-Learning with Mean-Field Environmental Feedback

一个多智能体Q学习的进化计算框架，结合平均场环境反馈

• Authors: Lichen Wang, Shijia Hua, Linjie Liu
• Subjects: Subjects: Multiagent Systems (cs.MA); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13253
• Pdf link: https://arxiv.org/pdf/2609.13253
• Abstract Multi-agent reinforcement learning in networked populations is governed by the interaction between individual adaptation, local encounters, and changing environmental conditions. To study this interaction, we formulate a coupled learning--environment model in which agents update stateless $Q$-values on a fixed graph, while their population-average behavior drives an environmental variable that dynamically modifies the payoff matrix. Under a first-order mean-field closure, we derive a deterministic transport equation for the population distribution of $Q$-values and couple it with a projected discrete update for the environmental state. The resulting model is evaluated against finite-network Monte Carlo simulations on random regular, Erdős--Rényi, Barabási--Albert, and random geometric graphs. Across the tested parameter ranges, the mean-field system reproduces the main macroscopic cooperation and environmental trajectories, and the trajectory-level root-mean-square error generally decreases with population size and average degree. The analysis further shows that environmental feedback reshapes the learned action-value ordering, while reinforcing feedback can produce pronounced dependence on the initial learning bias and resource level. The environmental timescale also plays an important role: a rapid response can drive the resource state to a boundary before learning adapts, whereas a slower response preserves the interaction between behavioral learning and environmental recovery. These results provide a population-level description of coupled reinforcement learning and environmental dynamics and characterize the performance of the mean-field approximation within the tested network and parameter ranges.
• 中文摘要 网络化群体中的多智能体强化学习受个体适应、局部遭遇和环境条件变化相互作用的控制。为研究这种相互作用，我们构建了一个耦合学习-环境模型，其中智能体在固定图上更新无状态$Q$-值，而其群体平均行为驱动一个环境变量动态修改收益矩阵。在一阶均值闭包下，我们推导出$Q$-值种群分布的确定性传输方程，并将其与环境状态的离散更新预测结合。所得模型通过随机正则图、Erdős-Rényi、Barabási-Albert 和随机几何图上的有限网络蒙特卡洛模拟进行评估。在测试参数范围内，平均场系统重现了主要的宏观合作和环境轨迹，轨迹级均方根误差通常随种群规模和平均度增加而减少。分析进一步表明，环境反馈重塑了学习的动作值排序，而强化反馈则可能对初始学习偏差和资源水平产生显著依赖。环境时间尺度也起着重要作用：快速响应可以将资源状态推至边界，而较慢的响应则保持行为学习与环境恢复之间的相互作用。这些结果提供了耦合强化学习与环境动态的总体层面描述，并刻画了平均场近似在测试网络和参数范围内的性能。

Extending the Speed Limit of Quadrupedal Locomotion via Refined Actuator Modeling and Adaptive Command Scheduling

通过精细的执行器建模和自适应指令调度扩展四足行走的速度限制

• Authors: Yucheng Tao, Shaowen Cheng, Guorong Lan, Yanyan Yuan, Yongbin Jin, Hongtao Wang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.13289
• Pdf link: https://arxiv.org/pdf/2609.13289
• Abstract Achieving high-speed locomotion in quadrupedal robots remains highly challenging, as actuators operate near their physical limits and exhibit pronounced nonlinearities. However, many existing methods neglect actuator nonlinearities and physical constraints during training, leading to a significant sim-to-real gap under highly dynamic motions and limiting achievable performance. To address this issue, we propose a high-speed locomotion framework that reduces sim-to-real discrepancies and stabilizes learning over a wide command distribution. A refined actuator model explicitly captures high-speed voltage coupling and magnetic saturation, enabling a more accurate representation of the torque-speed envelope. In addition, a reinforcement learning framework incorporating a two-stage curriculum and adaptive command scheduling (ACS) ensures stable training. Experiments on the 36.5 kg quadruped BlackPanther2 (BP2) demonstrate speeds of up to 13.2 m/s on a treadmill and 11.65 m/s outdoors, establishing a new state-of-the-art and, to the best of our knowledge, a world record for quadrupedal robot locomotion. The results further highlight the importance of accurate actuator modeling in preventing non-physical policy exploitation, and show that ACS improves robustness without sacrificing performance.
• 中文摘要 在四足机器人中实现高速运动仍然极具挑战，因为执行器工作接近物理极限，表现出明显的非线性。然而，许多现有方法在训练过程中忽视了执行器的非线性和物理约束，导致在高度动态运动下存在显著的模拟与现实差距，限制了可实现的性能。为解决这一问题，我们提出了一种高速运动框架，以减少模拟与现实的差异，并在广泛的指令分布中稳定学习。一个精细的执行器模型明确捕捉高速电压耦合和磁饱和，从而更准确地表示扭矩速度包络。此外，结合两阶段课程和自适应指令调度（ACS）的强化学习框架确保训练稳定。对36.5公斤四足的黑豹2号（BP2）的实验显示，在跑步机上最高可达13.2米/秒，户外速度为11.65米/秒，创下了四足机器人运动的新先进纪录，据我们所知，创下了世界纪录。结果进一步强调了精确执行器建模在防止非物理政策滥用中的重要性，并表明ACS在不牺牲性能的前提下提升了鲁棒性。

Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning

利用图像问题依赖性以实现更好的VLM测试时间强化学习

• Authors: Xinrui He, Ting-Wei Li, Junting Wang, Mengting Ai, Xinyu He, Hanghang Tong, Jingrui He
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13296
• Pdf link: https://arxiv.org/pdf/2609.13296
• Abstract Test-time reinforcement learning can adapt vision-language models (VLMs) to unlabeled target data, but its effectiveness is fundamentally limited by the reliability of self-generated learning signals. To assess the reliability of consensus-based learning signals, we analyze VLM test-time reinforcement learning across diverse VQA datasets and model sizes, revealing two limitations. First, gains from consensus-based test-time training largely come from answer normalization rather than content correction. Second, many initial VLM responses are incorrect due to the model's limited ability to jointly use the image and the question; consensus rewards derived from these outputs may preserve the resulting grounding errors rather than correct them. Motivated by these, we propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation. TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants, using the resulting token-level likelihood changes to estimate dependence on each input. It combines image and question dependence with calibrated confidence to construct a response-level reward that favors jointly grounded responses, and uses the token-level signals to assign greater positive policy credit to tokens supported by both inputs. This design favors responses that are jointly grounded in the image and the question and sufficiently confident, rather than merely popular. Experiments across eight VQA datasets and multiple VLM sizes show that TTIQ achieves the best average performance at every model scale. It further generalizes across VLM families, while models trained on one dataset improve performance on unseen datasets without further training.
• 中文摘要 测试时强化学习可以将视觉语言模型（VLM）适配为未标记的目标数据，但其有效性从根本上受限于自生成学习信号的可靠性。为评估基于共识的学习信号的可靠性，我们分析了VLM测试时强化学习在不同VQA数据集和模型规模中的表现，揭示了两个局限性。首先，基于共识的测试时训练带来的收益主要来自答案归一化，而非内容修正。其次，许多初始VLM响应错误，是因为模型在联合使用图像和问题方面能力有限;由这些输出得出的共识奖励可能会保留由此产生的接地错误，而非纠正它们。基于这些，我们提出了TTIQ，一种利用图像-问题依赖性以更好地适应VLM的测试时强化学习框架。TTIQ教师对每个抽样回答均基于原始图像-问题对及其图像和问题消融变体强制，利用所得的代币级似然变化估计对每个输入的依赖性。它结合图像和问题依赖性与校准置信度，构建有利于联合有根据的回答的响应级奖励，并利用代币级信号为支持两者支持的代币赋予更高的积极政策认可。该设计偏向基于图像和问题且具足够信心的回答，而非仅仅是受欢迎的回答。跨八个VQA数据集和多个VLM规模的实验显示，TTIQ在每个模型尺度下均表现最佳。该方法进一步推广跨VLM家族，而在一个数据集上训练的模型在未见数据集上提升表现，无需进一步训练。

Task-Based CT Protocol Optimization Using Reinforcement Learning and Virtual Imaging Trials

基于任务的CT协议优化，利用强化学习和虚拟成像试验

• Authors: Jiaqi Zou, David Fenwick, Vahid Tarokh, Nicholas Felice, Jayasai Rajagopal, Anuj Kapadia, Ehsan Samei, Navid NaderiAlizadeh, Ehsan Abadi
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13309
• Pdf link: https://arxiv.org/pdf/2609.13309
• Abstract Protocol optimization in computed tomography (CT) aims to improve diagnostic image quality while reducing radiation dose, but the interdependence of acquisition and reconstruction parameters makes exhaustive testing impractical. We propose a virtual imaging trial framework with reinforcement learning for efficient CT protocol optimization. Sixty-three computational human models with liver lesions were imaged using a validated CT simulator across 468 combinations of acquisition and reconstruction parameters, including tube voltage, tube current, reconstruction kernel, slice thickness, and pixel size. The optimization objective balanced liver lesion detectability, quantified by detectability index d-prime, against radiation dose. A Proximal Policy Optimization agent was trained and conditioned on patient-specific CT localizer embeddings derived from a pretrained vision transformer. On held-out patients, evaluating only 8 protocols per patient, about 2% of exhaustive testing, recovered 98.2% of the exhaustive-search oracle objective. With no patient-specific simulation, surrogate scoring alone achieved 89.7% recovery. Conditioning on the localizer improved zero-simulation recovery by 10.7 percentage points over the localizer-blind policy (paired 95% CI 2.9-19.5; p=0.02). These results show that the proposed framework can substantially reduce exhaustive protocol testing while enabling task-based, dose-aware protocol selection before the diagnostic scan.
• 中文摘要 计算机断层扫描（CT）中的方案优化旨在提升诊断图像质量，同时降低辐射剂量，但采集与重建参数的相互依赖使得详尽测试不切实际。我们提出了一个虚拟成像试验框架，结合强化学习，以高效优化CT方案。使用经过验证的CT模拟器，对63个具有肝病变的计算人类模型进行了成像，涵盖468种采集和重建参数组合，包括管电压、管电流、重建核、切片厚度和像素大小。优化目标平衡了肝病灶的检测性（以可检测性指数d-prime）与辐射剂量进行平衡。一个近端策略优化代理基于患者特定的CT定位器嵌入，从预训练的视觉变换器中进行训练和条件化。在未被预训练的患者中，每患者仅评估8个方案，约占穷尽测试的2%，恢复了98.2%的穷尽搜索预言机目标。在无患者特异模拟的情况下，仅用替代评分即可实现89.7%的恢复率。在定位器条件下，零模拟回收率比定位器-盲策略提高了10.7个百分点（配对95% CI 2.9-19.5;p=0.02）。这些结果表明，所提框架可大幅减少繁琐的方案测试，同时实现基于任务、剂量感知的方案选择，在诊断扫描前实现。

Toward Self-Adaptive Physical AI: Can LLM Agents Manage Long-Horizon Physical Tasks?

迈向自适应物理人工智能：LLM代理能否管理长期的物理任务？

• Authors: Varun Kaushik, Yayun Tan, Xiaofan Yu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13436
• Pdf link: https://arxiv.org/pdf/2609.13436
• Abstract Large Language Model (LLM) agents offer a promising path toward autonomously managing long-term physical tasks without human intervention. However, physical tasks require agents to continuously observe the environment, make consequential actions, and remain effective as the environment changes. Existing approaches either require substantial data and retraining, or primarily focus on agents operating in the virtual world. In this work, we explore the feasibility of building a self-adaptive physical AI agent that manages long-term physical tasks in a zero-shot manner and adapts to environmental changes without human intervention. We design a multi-agent framework that integrates planning, tool calling, observation, and verification, and evaluate it on agricultural tasks against reinforcement learning (RL) agents under different weather patterns. Our results show that zero-shot LLM agents can achieve comparable management outcomes to RL agents under the same weather pattern and adapt more effectively than RL when evaluated under a shifted environment, highlighting a promising path toward self-adaptive physical AI agents.
• 中文摘要 大型语言模型（LLM）代理为自主管理长期物理任务提供了一条有前景的路径，无需人工干预。然而，物理任务要求代理持续观察环境，采取相应行动，并在环境变化中保持有效。现有方法要么需要大量数据和再训练，要么主要关注虚拟世界中的代理。本研究探讨构建一种自适应物理AI代理的可行性，该代理以零捕捉方式管理长期物理任务，并能适应环境变化而无需人工干预。我们设计了一个多代理框架，整合了规划、工具调用、观察和验证，并在不同天气条件下对农业任务进行强化学习（RL）代理的评估。我们的结果表明，零样样的LLM代理在相同天气模式下能够实现与强化学习代理相当的管理结果，并在环境变化下比强化学习更有效地适应，这为迈向自我适应物理AI代理的前景前景显得很有希望。

TimeThink: Eliciting Compositional Reasoning in Timeseries Large Language Models

TimeThink：在时间序列大型语言模型中激发组合推理

• Authors: Sudarshan Regmi, Arvind Pillai, Yu Yvonne Wu, Yuliang Chen, Bibek Panthi, Tess Z. Griffin, Michael V. Heinz, Lisa Marsch, Nicholas C. Jacobson, Andrew Campbell
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13457
• Pdf link: https://arxiv.org/pdf/2609.13457
• Abstract Timeseries multimodal large language models (TS-MLLMs) have recently begun leveraging the reasoning capabilities of large language models (LLMs) for question-answering tasks. However, these models often fail to capture dynamic temporal patterns, providing only implicit reasoning that lacks the underlying explanations critical for high-stakes applications like healthcare. While reinforcement learning (RL)-based timeseries language models aim to address this, they often fall short because they are trained on narrow, in-distribution data and struggle with out-of-distribution compositional questions. To address these challenges, we present TimeThink, a synthetic framework for eliciting compositional timeseries reasoning. Core timeseries primitives (e.g., trend, seasonality) are domain-independent and can be deterministically generated. Guided by this premise, TimeThink first designs a synthetic data generator that produces atomic and composite question-answer pairs, providing objective ground truth with reasoning traces. Building on this framework, TimeThink employs a reinforcement learning with verifiable rewards (RLVR) training strategy that encourages explicit reasoning. Unlike template-reliant methods, this approach enables the model to learn the underlying logic of composition rather than simply imitating traces. Extensive experiments show that TimeThink, trained only on synthetic data, significantly outperforms strong baselines on both synthetic and real-world benchmarks.
• 中文摘要 时间序列多模态大型语言模型（TS-MLLM）最近开始利用大型语言模型（LLMs）的推理能力来完成问答任务。然而，这些模型常常无法捕捉动态的时间模式，仅提供隐式推理，缺乏对医疗等高风险应用至关重要的底层解释。虽然基于强化学习（RL）的时间序列语言模型旨在解决这个问题，但它们常因训练于狭窄的分布内数据上，难以应对分布外的组合问题而难以达到预期。为应对这些挑战，我们提出了TimeThink，一个用于引发组合时间序列推理的综合框架。核心时间序列原语（如趋势、季节性）是域无关的，且可确定性生成。基于这一前提，TimeThink首先设计了一种合成数据生成器，能够生成原子和复合问答对，提供客观的真实性和推理痕迹。基于该框架，TimeThink采用了带可验证奖励的强化学习（RLVR）训练策略，鼓励显性推理。与依赖模板的方法不同，这种方法使模型能够学习组合的底层逻辑，而不仅仅是模仿痕迹。大量实验表明，仅在合成数据上训练的TimeThink在合成和现实世界基准测试中都显著优于强基线。

Constraint-Grounded Reinforcement Learning for Variable Impedance Control in Contact-Rich Robotic Insertion

约束接地强化学习用于接触富裕机器人插入中可变阻抗控制

• Authors: Lin He, Min Deng
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.13516
• Pdf link: https://arxiv.org/pdf/2609.13516
• Abstract In robotic insertion under uncertain contact, the axial force limit and the appropriate controller gain vary across tasks. As a result, a single fixed gain is unlikely to remain suitable across different task conditions, making conventional impedance controllers reliant on manual retuning. To eliminate manual retuning, we propose Constraint-Grounded Reinforcement Learning (CG-RL), a variable impedance framework for online gain adaptation. Conditioned on the force limit and contact feedback, the policy outputs a residual motion, an insertion rate, and a requested gain. The controller projects this gain into the admissible range without exposing the range itself to the policy. This separation allows a single policy to operate under different force limits without retraining or manual retuning. We evaluate CG-RL on simulated oblique insertion across five training seeds. CG-RL achieves an $85.8\pm7.7\%$ (mean $\pm$ SD) success rate of insertions without violating the force limit, while keeping the applied gain within the admissible range. As a comparison, a fixed-gain baseline using the midpoint gain achieves a success rate of $50.1\%$. The policy adapts its insertion rate continuously to the specified force limit and further generalizes to more permissive force limits above the training range. In contrast, the same actor without force-limit input does not exhibit this adaptation. The applied gain is guaranteed to remain within the admissible range, while force-limit satisfaction is validated empirically rather than guaranteed formally.
• 中文摘要 在不确定接触下的机器人插入中，轴向力极限和相应控制器增益因任务而异。因此，单一固定增益不太可能在不同任务条件下保持适用，使传统阻抗控制器依赖手动重新调谐。为消除手动重调，我们提出了约束-接地强化学习（CG-RL），这是一种可变阻抗的在线增益适配框架。该策略基于力限制和接触反馈，输出残余运动、插入速率和请求增益。控制器将该增益投射到可接受范围，而不暴露于策略之下。这种分离使单个策略在不同力限制下工作而无需重新训练或手动重新调谐。我们在五个训练种子的模拟斜置插入上评估CG-RL。CG-RL在保持施加增益在允许范围内的情况下，插入成功率为$85.8\pm7.7\%$（平均$\pm$ SD）。作为对比，使用中点增益的固定增益基线成功率为$50.1\%$。该策略会连续调整插入率以适应指定的力限制，并进一步推广到训练范围以上更宽松的力限制。相比之下，未输入力限制的同一参与者则不表现出这种适应。施加的增益保证保持在可接受范围内，而力限制满足则通过实证验证，而非形式保证。

Runtime-Incremental Transformer for Reinforcement-Learning-Based Adaptive Control

基于强化学习的自适应控制运行时增量变换器

• Authors: Giansalvo Cirrincione, Adriano Fagiolini
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13545
• Pdf link: https://arxiv.org/pdf/2609.13545
• Abstract Learning-based adaptive control of robotic manipulators with non-observable friction memory has been addressed by attention- based meta-controllers whose number of attention heads is fixed before training and is tuned by costly offline search. At long memory horizons, such fixed-capacity controllers are prone to catastrophic failures on a sizeable fraction of training seeds. The present paper introduces a runtime mechanism that grows and prunes the heads of the attention block during reinforcement learning, governed by two signals: the effective rank of the on-policy context distribution, which triggers growth when representational capacity becomes insufficient, and the per-head output magnitude, which flags redundant heads for removal. Policy continuity at growth events and a quantitative bound at prune events are established analytically. On a two- link manipulator with Stribeck friction, the proposed mechanism attains full success across all memory regimes, eliminating the long-horizon failure mode and removing the need for offline tuning of the head count.
• 中文摘要 基于学习的自适应控制机器人操作手，具有不可观察的摩擦记忆，已被基于注意力的元控制器解决，这些元控制器在训练前固定注意力头数，并通过昂贵的离线搜索进行调谐。在较长的记忆视野中，这类固定容量控制器在相当比例的训练种子上容易发生灾难性失败。本文介绍了一种运行机制，在强化学习过程中增长和修剪注意力块头，受两个信号控制：策略上下文分布的有效秩，当表征能力不足时触发增长;以及人头输出幅度，标记冗余头需移除。通过分析建立了增长事件的策略连续性和修剪事件的定量界限。在带有斯特里贝克摩擦的双链杆操作器上，该机制在所有内存区间都能实现完全成功，消除了长视距失效模式，也无需离线调校磁头计数。

Planning or Learning: Reliability and Cost in Multi-Asset Maintenance

规划或学习：多资产维护中的可靠性与成本

• Authors: Xian Yeow Lee, Chandrasekar Venkatraman, Ahmed Farahat
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.13566
• Pdf link: https://arxiv.org/pdf/2609.13566
• Abstract Industrial maintenance systems involve multiple interacting assets and shared resources, making it challenging to balance reliability and operational cost using a single decision framework. While recent work has focused on reinforcement learning (RL) for maintenance scheduling, direct comparisons with planning approaches under identical settings remain limited. In this work, we empirically compare planning and RL for multi-asset bearing maintenance using run-to-failure data. We examine how these methods behave when balancing preventive maintenance against tolerable failures across a range of failure penalty scenarios. We observed a consistent behavioral difference driven by objective formulation. Planning enforces reliability as a hard constraint and produces zero-failure policies whose total cost is largely insensitive to the magnitude of failure penalties. RL agents optimize expected cost and often trade off preventive maintenance against occasional failures as penalties vary, resulting in lower costs under low-penalty regimes but persistent non-zero failures even when penalties are high. We also investigate lightweight constraint mechanisms, including reward shaping and action masking, to encourage RL's reliability. From a practical perspective, planning may be more suitable when strict reliability is required and deployment horizons are short, whereas RL may provide cost-efficient policies when limited failures are acceptable and long-run operational efficiency is prioritized. Overall, this study clarifies the trade-offs between reliability and cost in multi-asset maintenance and suggests that planning and RL are complementary approaches. Beyond these findings, the controlled benchmark protocol itself that unifies environment, cost model, and evaluation across paradigms, offers a reusable template for comparing decision-making approaches in other maintenance settings.
• 中文摘要 工业维护系统涉及多个交互资产和共享资源，使得用单一决策框架平衡可靠性和运营成本变得困难。尽管近期研究聚焦于维护调度的强化学习（RL），但在相同环境下与规划方法的直接比较仍然有限。本研究通过实证比较多资产相关维护的规划与强化学习，利用运行到故障数据。我们考察了在多种故障惩罚场景下，预防性维护与可容忍失效之间的平衡时，这些方法的表现如何。我们观察到由目标设定驱动的行为差异一致。规划将可靠性作为硬性约束，产生总成本对失效惩罚大小不敏感的零失效政策。强化学习代理优化预期成本，并常在惩罚变化下权衡预防性维护与偶发故障，导致低罚款下成本较低，但即使罚款高仍持续存在非零失效。我们还研究了轻量级约束机制，包括奖励塑造和动作掩蔽，以促进强化学习的可靠性。从实际角度看，当需要严格可靠性且部署时间短时，规划可能更为适用;而在有限故障可接受且优先考虑长期运营效率时，强化学习可能提供成本效益的策略。总体而言，本研究澄清了多资产维护中可靠性与成本之间的权衡，并表明规划与强化学习是互补的方法。除此之外，统一环境、成本模型和评估的受控基准协议本身，为比较其他维护环境决策方法提供了可复用的模板。

Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models

漂移约束优化：在微调指令模型中，只有方向才重要

• Authors: Fei Yuan, Changjiang Gao, Yilei Tu, Yifeng Liu, Shujian Huang, Yu Qiao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13680
• Pdf link: https://arxiv.org/pdf/2609.13680
• Abstract Fine-tuning instruct models often improves target performance while inducing behavioral drift from the reference model, which can degrade existing capabilities. Rather than treating this drift as an uncontrolled consequence of optimization, we specify a behavioral drift budget before optimization and ask how to boost the target-task performance within it. Locally, behavioral drift induces a shared geometry anchored at the reference model, with the drift budget defining a boundary within this space. In this space, drift determines distance from the reference, leaving update direction as the remaining degree of freedom. Fine-tuning updates can therefore be compared through their directional efficiency, naturally reformulating fine-tuning as a direction-selection problem. This reformulation makes a concrete prediction: changing the accessible directions can qualitatively alter the outcome of fine-tuning. We test this prediction in a stringent QA-only setting, where strong instruct models are fine-tuned only on final answers but must still generate multi-step reasoning at inference. Despite this mismatch, a coarse layer-selective probe reverses the failure of QA-only fine-tuning and reveals the existence of effective directions, with multiple neighboring configurations improving target performance while preserving reasoning and general capabilities. Across Qwen3-8B and Qwen3-14B, these directions substantially improve scientific reasoning and multilingual translation. Over more than 100 languages, the resulting models match or outperform dedicated translation systems and provide a stronger initialization for subsequent reinforcement learning. Our results suggest that fine-tuning is not just about how much a model changes, but how that change is spent. this https URL and this https URL
• 中文摘要 微调指令模型通常会提升目标表现，同时诱导参考模型的行为漂移，从而降低现有能力。我们不将这种漂移视为优化的失控结果，而是在优化前指定行为漂移预算，并探讨如何在其中提升目标任务性能。在局部，行为漂移诱导一个以参考模型为锚点的共享几何体，漂移预算定义了该空间内的边界。在该空间中，漂移决定了与参考距离，更新方向为剩余自由度。因此，微调更新可以通过其方向效率进行比较，自然地将微调重新表述为方向选择问题。这一重新表述提出了具体预测：改变可达方向可以定性地改变微调的结果。我们在严格的仅限QA条件下测试该预测，强指令模型仅在最终答案上微调，但推理时仍需生成多步推理。尽管存在不匹配，粗层选择性探针逆转了仅QA微调的失败，揭示了有效方向的存在，多个相邻配置提升目标性能，同时保持推理和一般能力。在Qwen3-8B和Qwen3-14B中，这些方向显著提升了科学推理和多语言翻译能力。在100多种语言中，所得模型匹配或优于专用翻译系统，并为后续强化学习提供了更强的初始化。我们的结果表明，微调不仅仅是模型变化的程度，更在于变化的使用方式。此 https URL 和 此 https URL

Learning In-Hand Object Reaching to General 6D Poses

学习手持物体达到通用的6D姿势

• Authors: Junxiao Lin, Tianyue Wu, Jie Yin, Jia Pan, Kaifeng Zhang, Weiming Zhi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.13761
• Pdf link: https://arxiv.org/pdf/2609.13761
• Abstract In-hand manipulation allows multi-fingered dexterous hands to reconfigure grasped objects without releasing and regrasping them. This improves manipulation efficiency by reducing repeated grasp acquisition and large arm motions. However, most learning-based methods focus on reorientation, continuous rotation, or translation, whereas many tasks require joint control of object position and orientation. We formulate this capability as in-hand 6D object pose reaching: starting from an existing grasp, coordinated finger motions move the object to a palm-relative target pose. We present POISE (Palm-relative Object reaching In SE(3)), a sim-to-real reinforcement learning framework for this task. POISE combines diverse stable-grasp initialization, goal- and geometry-conditioned control, an adaptive 6D goal curriculum, and a compact reward scheme for pose reaching and grasp preservation. In simulation, diverse initialization raises held-out-grasp success from 40.1% to 51.5% and post-drop recovery from 33.8% to 72.9%; the curriculum raises full-range success from 6.2% to 59.5%. On hardware, the grasp-maintenance reward improves three-target sequence success from 20% to 80%. In real-world experiments, POISE reaches successive 6D targets without manual reset across multiple object geometries and wrist orientations, and recovers from external disturbances. To support further research in dexterous manipulation, we will release our code at this https URL.
• 中文摘要 手握操作允许多指灵巧手重新配置抓取物体，而无需松开或回放。这通过减少重复抓握和大动作动作，提高了操作效率。然而，大多数基于学习的方法侧重于重新定向、连续旋转或平移，而许多任务则需要共同控制物体位置和方向。我们将此能力表述为手握6D物体姿势伸手：从已有握持开始，协调的手指动作将物体移动到掌心相对目标姿势。我们提出了POISE（手掌相对物体伸手）在SE（3）中），这是一个模拟到现实的强化学习框架。POISE结合了多样化的稳定抓取初始化、目标和几何条件控制、自适应的6D目标课程以及紧凑的姿势达手和抓取保持奖励方案。在模拟中，多样化初始化将保持抓取成功率从40.1%提升至51.5%，落下后恢复率从33.8%提升至72.9%;课程将全范围成功率从6.2%提升至59.5%。在硬件上，握持-维持奖励将三目标序列成功率从20%提升至80%。在现实实验中，POISE能够在多个物体几何和手腕方向下无手动重置即可连续达到6D目标，并能从外部干扰中恢复。为支持灵巧操作的进一步研究，我们将在此 https URL 发布代码。

Force-Aware Reinforcement Learning with Hybrid Sensorless Force Estimation for Wheeled-Legged Loco-Manipulation

基于混合无感应力估计的原力感知强化学习，用于轮式腿式机车操控

• Authors: Xuanqi Zeng, Jiaming Wang, Tianlin Zhang, Lingwei Zhang, Botian Xu, Weipeng Xia, Zhongyu Li, Yun-Hui Liu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.13779
• Pdf link: https://arxiv.org/pdf/2609.13779
• Abstract Force-controlled loco-manipulation requires a whole-body policy to coordinate locomotion and arm motion while regulating end-effector interaction forces. This is challenging under floating-base dynamics and changing support contacts, particularly when end-effector force/torque sensing is unavailable for control. This paper presents a force-aware reinforcement learning approach with hybrid sensorless force estimation for wheeled-legged loco-manipulation. The proposed method provides a structured estimate of the end-effector force as an explicit policy observation, enabling force-guided contact behavior without using an end-effector force/torque sensor for control. The force estimate is obtained by combining generalized momentum observation, contact-constrained wrench projection, and temporal residual learning: the model-based components extract the physically structured part of the whole-body disturbance, while the residual network compensates the remaining motion-dependent bias. The estimated force is integrated into a mode-conditioned whole-body policy with an axis-wise force/position selector, allowing free-space motion, pure force regulation, and hybrid force/position control within one controller. Simulation results demonstrate improved sensorless force estimation and force-control performance. Hardware experiments further validate the proposed controller through quantitative valve-rotation and hybrid wiping evaluations, together with force-guided door opening and zero-force human-guided motion on a real wheeled-legged platform.
• 中文摘要 力控机车操控需要全体策略来协调运动和臂部运动，同时调节末端执行器相互作用力。在浮动基底动力学和变化支撑接触下，尤其是在无法控制端部力/扭矩时，这具有挑战性。本文提出了一种基于轮式腿式机车操控的力感知强化学习方法，采用混合无感应力估计。所提方法作为明确的策略观察，提供了末端执行器力的结构化估计，实现了力引导接触行为，无需使用末端执行器力/扭矩传感器进行控制。力估计通过结合广义动量观察、接触约束扳手投影和时间残余学习获得：基于模型的组件提取全身扰动的物理结构部分，残余网络补偿剩余的运动依赖偏差。估计的力被整合进模式条件的全体策略中，配备按轴方向的力/位置选择器，实现自由空间运动、纯力调节和力/位置混合控制。模拟结果显示，无传感器力估计和力控制性能有所提升。硬件实验通过定量阀门旋转和混合擦拭评估，以及力导门开启和在真实轮腿平台上的零力人力引导运动，进一步验证了拟议控制器。

LLM-Enhanced Multi-Agent Reinforcement Learning for Unified Electric Vehicles-Charging Station-Grid Optimization in Public Charging Systems

大型语言模型增强型多智能体强化学习，用于公共充电系统中统一电动汽车-充电站-电网优化

• Authors: Yang Zhang, Lindong Xie, Chongyu Wang, Gaojunjie Li, Siqi Bu, Edward Chung
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13805
• Pdf link: https://arxiv.org/pdf/2609.13805
• Abstract In the era of the Internet of Things (IoT), coordinating connected electric vehicle (EV) charging scheduling to balance EV charging satisfaction, station profitability, and smart grid stability presents a complex multi-objective challenge. Existing Multi-Agent Reinforcement Learning (MARL) approaches often struggle with high-dimensional state spaces generated by massive IoT sensing data and conflicting stakeholder interests. This paper proposes a novel LLM-enhanced MARL framework that, for the first time, simultaneously optimizes the Grid, EVs, and Stations within a unified loop. By integrating Large Language Model (LLM), we address two critical bottlenecks: interpretable feature selection and adaptive multi-objective balancing. The LLM analyzes real-time IoT-collected environmental states to extract physically significant features and dynamically assigns weights to conflicting objectives-including profit, user satisfaction, and grid load-using semantic reasoning instead of complex manual tuning. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art baselines, achieving superior market efficiency while reducing training time by over 70%. This approach offers a scalable, transparent solution for efficient and sustainable IoT-enabled urban charging infrastructure management.
• 中文摘要 在物联网（IoT）时代，协调互联电动汽车（EV）充电调度以平衡充电满意度、电站盈利能力和智能电网稳定性，是一个复杂的多目标挑战。现有的多智能体强化学习（MARL）方法常常难以应对由庞大的物联网感测数据和利益相关者利益冲突产生的高维状态空间。本文提出了一种新颖的LLM增强MARL框架，首次在统一循环中同时优化电网、电动汽车和电站。通过集成大型语言模型（LLM），我们解决了两个关键瓶颈：可解释的特征选择和自适应多目标平衡。LLM通过实时分析物联网收集的环境状态，提取物理意义显著的特征，并通过语义推理动态为利润、用户满意度和电网负载等冲突目标分配权重，而非复杂的手动调优。大量实验表明，我们的框架远超最先进的基线，实现了卓越的市场效率，同时将培训时间缩短了70%以上。这一方法为高效且可持续的物联网驱动城市充电基础设施管理提供了可扩展、透明的解决方案。

Bypass Observation: A Conceptual Design of a Non-Intrusive Layer-Wise Semantic Extraction Architecture

绕过观察：一种非侵入式层级语义提取架构的概念设计

• Authors: Haibin Tong, Jiang Yu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13807
• Pdf link: https://arxiv.org/pdf/2609.13807
• Abstract Large language models reason in high-dimensional hidden-state spaces, while users observe only final outputs. We introduce Bypass Observation, a non-intrusive layer-wise readout architecture that attaches read-only observation heads to selected Transformer layers without feeding their outputs back into the backbone. We consider three variants: a shared LM head across layers, layer-specific heads, and a layer- or step-adaptive head. For full-vocabulary readout, we derive a closed-form overhead approximation governed primarily by V/(12d), with representative estimates ranging from about 30% to 240%, and discuss cost reductions via sparse observation, low-rank factorization, reduced vocabularies, top-k readout, and selective positions. We argue that Bypass Observation can make model computation more observable while remaining only a partial, potentially misleading projection of hidden states. We further distinguish bypass chain-of-thought from conventional chain-of-thought: conventional reasoning tokens enter the autoregressive computation, whereas bypass readouts remain causally external at inference time, although they can still provide training signals in reinforcement learning. Finally, we discuss applications to looped and recurrent-depth Transformers, where iteration-wise readout may expose convergence, oscillation, and potential halting signals. The proposal is conceptual and analytical; systematic empirical validation remains future work.
• 中文摘要 大型语言模型在高维隐藏状态空间中推理，而用户仅观察最终输出。我们介绍了绕过观察，这是一种非侵入性的层级读取架构，将只读观察头附加到选定的Transformer层，而不将其输出反馈回骨干网。我们考虑三种变体：跨层共享LM头、层特定头以及层或步进自适应头。对于全词汇读出，我们推导出主要由V/（12d）控制的闭式开销近似，代表性估计范围约为30%至240%，并讨论通过稀疏观察、低秩分解、缩减词汇、顶K字读出和选择性位置来降低成本。我们认为绕过观察可以使模型计算更易被观察，同时仅保持部分且可能误导的隐藏状态投影。我们进一步区分绕过思维链与传统思维链：传统推理标记进入自回归计算，而绕过读出在推理时仍因果外部，尽管它们仍能在强化学习中提供训练信号。最后，我们讨论了环路和循环深度变换器的应用，其中按迭代的读出可能暴露收敛、振荡和潜在停机信号。该提案具有概念性和分析性;系统性实证验证仍需未来工作。

ViperQ: Order Flow Pattern Recognition via Auction Market Theory for Reinforcement Learning Trading

ViperQ：通过拍卖市场理论进行订单流模式识别，用于强化学习交易

• Authors: Asser Moustafa, Rares-Mihail Neagu, Jugal Kalita
• Subjects: Subjects: Artificial Intelligence (cs.AI); Trading and Market Microstructure (q-fin.TR)
• Arxiv link: https://arxiv.org/abs/2609.13825
• Pdf link: https://arxiv.org/pdf/2609.13825
• Abstract Reinforcement learning trading systems published in the academic literature overwhelmingly rely on price-aggregate state representations (OHLCV bars) or limit-order-book depth features, leaving microstructure pattern theories from the practitioner literature, namely Auction Market Theory and Market Profile, without a peer-reviewed computational instantiation. We present ViperQ, a reinforcement learning system whose state representation is built explicitly from Auction Market Theory primitives: Volume Point of Control, Value Area position, Low Volume Node flags, Cumulative Volume Delta divergence, and tape-velocity signatures, assembled into a 20-dimensional Z-normalised vector. Two Proximal Policy Optimisation agents are trained with a prospect theory-grounded asymmetric reward function that penalises losing holds at a magnitude consistent with Kahneman and Tversky's loss-aversion coefficient. Evaluated on a held-out twelve-month partition of institutional tick data the agents have never seen, ViperQ achieves +163.6% ROI on TSLA (-27.5% max drawdown, 27,019 trades) and +116.5% ROI on NVDA (-47.8% max drawdown, 12,892 trades) under zero leverage. The results establish Auction Market Theory features as a tractable structured input modality for sequential decision-making on financial time series and motivate further work on microstructure-aware policy learning.
• 中文摘要 学术文献中发表的强化学习交易系统主要依赖价格汇总状态表示（OHLCV 条）或限价单深度特征，导致实践者文献中的微观结构模式理论，即拍卖市场理论和市场剖面，缺乏同行评审的计算实例化。我们介绍 ViperQ，一种强化学习系统，其状态表示明确基于拍卖市场理论的原语构建：控制点、价值区位置、低交易节点标志、累计体积 Delta 发散和磁带速度签名，并组合成一个 20 维 Z 归一化向量。两个近端策略优化代理接受基于前景理论的非对称奖励函数训练，该函数对失去持有的惩罚幅度与 Kahneman 和 Tversky 的损失厌恶系数一致。通过持有的机构跳级数据的十二个月分割评估，ViperQ在TSLA上实现了+163.6%的投资回报率（最大回撤-27.5%，27,019笔交易），在NVDA投资回报率+116.5%（最大回撤-47.8%，12,892笔交易），且无杠杆。结果确立了拍卖市场理论作为一种可处理的结构化输入方式，用于财务时间序列的顺序决策，并激励对微观结构感知政策学习的进一步研究。

UniCAR-RL: Seeing Better before Thinking Deeper in Visual Mathematics

UniCAR-RL：在视觉数学中先看得更好再深入思考

• Authors: Yuzhe Li, Hao Yan, Hao Wang, Xingchen Liu, Ya-Qi Yu, Jihao Wu, Minghui Liao, Wei Chen, Yuliang Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.13849
• Pdf link: https://arxiv.org/pdf/2609.13849
• Abstract Multimodal Large Language Models (MLLMs) often struggle with complex mathematical visual reasoning primarily due to a lack of fine-grained perception, causing initial visual hallucinations to directly trigger cascading reasoning failures. In traditional end-to-end reinforcement learning (RL), sparse rewards fail to decouple perceptual hallucinations from logical missteps, hindering targeted perception optimization. Alternatively, fine-tuning with perception-enhanced CoT data incurs high costs and hallucinations. In this paper, we address these challenges by proposing UniCAR-RL, an annotation-free RL framework. By explicitly decoupling the optimization of perception and reasoning during the training process, it achieves isolation and optimization of both capabilities. Specifically, UniCAR-RL consists of three synergistic branches: 1) a Caption-RL branch that optimizes perception capabilities through verifier-guided reasoning validation; 2) a Reasoning-RL branch that performs logical reasoning based on a gold image description to halt cascading errors; 3) a QA-RL branch that retains native end-to-end alignment to ensure robust question-answering performance. Experiments show that UniCAR-RL substantially improves MLLMs' mathematical and visual reasoning using only raw short-answer data. Furthermore, it demonstrates strong generalization across diverse architectures and scales.
• 中文摘要 多模态大型语言模型（MLLM）常因缺乏细粒度感知而难以处理复杂的数学视觉推理，导致初始视觉幻觉直接触发连锁推理失败。在传统的端到端强化学习（RL）中，稀疏奖励无法将感知幻觉与逻辑失误分离，阻碍了有针对性的感知优化。相反，使用感知增强的CoT数据进行微调会带来高成本和高幻觉。本文通过提出UniCAR-RL这一无注释的强化学习框架来应对这些挑战。通过在训练过程中明确解耦感知和推理的优化，实现了两者的隔离和优化。具体来说，UniCAR-RL 包含三个协同分支：1）通过验证者引导推理验证优化感知能力的 Caption-RL 分支;2）基于金色图像描述进行逻辑推理以阻止级联错误的 Reasoning-RL 分支;3）保持原生端到端对齐以确保问答稳健性能的 QA-RL 分支。实验表明，UniCAR-RL 仅用原始短答数据大幅提升了 MLLM 的数学和视觉推理能力。此外，它展示了在多种架构和尺度上的强有力推广能力。

North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)

North Small Translate：高级经济翻译（Cohere CAT+）

• Authors: Tom Kocmi, Alexandre Bérard, Phil Blunsom, Samuel Cahyawijaya, Shaun Cassini, Nicholas Frosst, Ona de Gibert, Aidan Gomez, Nithya Govindarajan, Shun Kiyono, Olivia Lasche, Lawrence Rogers, Kelly Marchisio, Nikita Moghe, Yash More, Camila Moran-Hidalgo, Yiyang Nan, Michael Sachs, Trisha Starostina, Daan van Stigt, Spencer Rarrick, Sebastian Vincent, Ivan Zhang
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.13916
• Pdf link: https://arxiv.org/pdf/2609.13916
• Abstract We present North Small Translate, an open-weight, LLM-based machine translation (MT) model with instruction-following capabilities built on the same foundation as Cohere's Command A Plus, a mixture-of-experts architecture with 25 billion active parameters out of 218 billion total parameters. North Small Translate is trained using difficulty sampling to obtain challenging documents and a five-step training protocol combining supervised fine-tuning, direct preference optimization, and online reinforcement learning. We prioritized throughput through a non-reasoning base model and supplemented with optional agentic capabilities to unlock translation quality gains. North Small Translate is trained to perform MT-related tasks, including post-editing and quality estimation, as well as related tasks such as general instruction following. The model achieves top MT performance across 50 languages in the class of models under 1T parameters, with no need to run expensive reasoning at inference time.
• 中文摘要 我们介绍 North Small Translate，一个开放权重的基于大型语言模型（LLM）的机器翻译（MT）模型，具备指令跟随能力，基于 Cohere 的 Command A Plus 的基础，后者是一个专家混合架构，拥有 2180 亿参数中的 250 亿活跃参数。North Small Translate 通过困难抽样训练以获取具有挑战性的文档，并采用五步训练协议，结合监督微调、直接偏好优化和在线强化学习。我们通过非推理基础模型优先提升吞吐量，并辅以可选的代理能力以解锁翻译质量提升。North Small Translate 训练用于执行与机器翻译相关的任务，包括后期编辑和质量估计，以及一般指令跟随等相关任务。该模型在 1T 参数下，在 50 种模型类别中实现顶尖的机器翻译性能，无需在推理时运行昂贵的推理。

Unlocking the Unsolvable: Teacher-Guided Curriculum for Data-Efficient RLVR

解锁无解之谜：教师引导课程，实现数据高效RLVR

• Authors: Yukang Zhu, Zhen Han
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.13997
• Pdf link: https://arxiv.org/pdf/2609.13997
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) has shown remarkable success in improving the mathematical reasoning of large language models. Yet problems beyond the model's current capability, where rollouts uniformly fail and no learning signal is produced, are structurally wasted despite marking the most informative training frontier. We show that these otherwise-inert problems can be unlocked via teacher-guided curriculum learning: partial reasoning traces from a stronger model create a graded difficulty landscape, and a backward-chaining curriculum progressively withdraws guidance until the model solves problems unaided. Training on only 128 unsolvable problems matches or exceeds GRPO trained on a full 2,000-problem corpus (~16x data efficiency) on the nine-benchmark average for both base models, while substantially expanding the reasoning boundary measured by pass@k at large k. Furthermore, we identify a distribution-shift cost that is particularly acute in the unsolvable-only regime and propose Monotone Frontier Curriculum (MFC), a method that monotonically drives training toward unguided solving, consistently outperforming existing curriculum methods.
• 中文摘要 带可验证奖励的强化学习（RLVR）在提升大型语言模型的数学推理方面取得了显著成功。然而，超出模型当前能力的问题，即推广均失败且无学习信号，尽管标记了最具信息量的培训前沿，结构上却被浪费。我们证明，这些本来惰性的问题可以通过教师引导的课程学习解锁：更强模型中的部分推理痕迹形成分级难度景观，逆链课程逐步撤回指导，直到模型能独立解决问题。仅对128个无解问题进行训练，就在两个基础模型的九个基准平均值下，在完整2000个问题语料库上训练时，达到甚至超过GRPO（数据效率的约16倍），同时大幅扩展了pass@k k所测量的推理边界。此外，我们识别出一种在仅无解体系中特别严重的分布转移成本，并提出了单调前沿课程（MFC）方法，这种方法单调地推动训练向无指导求解方向发展，持续优于现有课程方法。

SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints

SafePG：安全且全局最优的硬约束强化学习

• Authors: Vipul K. Sharma, Wesley A. Suttle, S. Sivaranjani
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.14076
• Pdf link: https://arxiv.org/pdf/2609.14076
• Abstract We present an optimal and convergent model-free policy gradient (PG) reinforcement learning (RL) framework for controlling nonlinear dynamical systems under hard safety constraints. We first construct a class of stochastic wrapper policies centered around a deterministic controller, thereby enabling exploration in unknown environments while preserving the underlying deterministic control structure. We then define a class of parameterized safe-by-construction control policies by truncating these stochastic policies onto hard safety constraints. We next establish, via measure-theoretic arguments, that the potentially nonconvex RL objective under the truncated policy class, as well as its policy gradients, are well-defined. We then develop a model-free PG algorithm based on stochastic gradient ascent that directly searches over these truncated policies and leverage gradient dominance to establish convergence and optimality guarantees. Finally, we validate this framework through simulations on a safe quadrotor navigation problem.
• 中文摘要 我们提出了一个最优且收敛的无模型策略梯度（PG）强化学习（RL）框架，用于在硬安全约束下控制非线性动力系统。我们首先构建了一类以确定性控制器为核心的随机包装策略，从而在保持确定性控制结构的同时，实现在未知环境中的探索。随后，我们将这些随机策略截断到硬安全约束上，定义了一类参数化的构造安全控制策略。接着，我们通过测度论证证，截断策略类下潜在非凸的RL目标及其策略梯度是良好定义的。随后，我们基于随机梯度上升开发了一种无模型的PG算法，直接搜索这些截断策略，并利用梯度优势来建立收敛性和最优性保证。最后，我们通过对安全四旋翼导航问题的模拟验证了该框架。

DQN-Scheduler: A Multi-Objective Optimization Framework for Scheduling Microservices in Cloud Computing

DQN调度器：一个用于云计算微服务调度的多目标优化框架

• Authors: Abdullah Alelyani, Amitava Data, Ghulam Mubasher
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2609.14117
• Pdf link: https://arxiv.org/pdf/2609.14117
• Abstract Cloud computing has emerged as an information technology solution, providing software and infrastructure solutions for companies and individuals. The pay-as-you-go approach has increased demands for the cloud. The massive range of resources, the variety of services, and flexible pricing grab attention. In addition, microservices have emerged as a new way of building software, with applications developed as loosely dependent tasks. Additionally, container technology has boosted the popularity of microservices by offering a platform for this type of architecture. Containers and microservices improve the flexibility and scalability of cloud applications. There are two primary types of microservices: batch and online services, with the majority of applications falling into the online service category. Scheduling microservices is challenging because it requires careful management of resource utilization, load balancing, network latency, reliability, and availability. In this study, we introduce the DQN-Scheduler, a novel reinforcement learning-based agent designed to optimize microservice scheduling in cloud environments. Our approach aims to optimize multiple scheduling objectives simultaneously, such as resource utilization, load balancing, latency, reliability, and availability. To our knowledge, this is the first framework to address all these objectives simultaneously. The DQN-Scheduler was tested against benchmark algorithms in the field. The experimental results demonstrate that the DQN-Scheduler outperforms benchmark algorithms.
• 中文摘要 云计算作为一种信息技术解决方案出现，为企业和个人提供软件和基础设施解决方案。按需付费的方式增加了对云的需求。庞大的资源种类、多样化的服务和灵活的定价吸引了关注。此外，微服务作为一种新的软件构建方式出现，应用程序开发为松散依赖的任务。此外，容器技术通过提供此类架构平台，提升了微服务的受欢迎程度。容器和微服务提升了云应用的灵活性和可扩展性。微服务主要有两种类型：批处理服务和在线服务，大多数应用属于在线服务类别。调度微服务具有挑战性，因为它需要对资源利用率、负载均衡、网络延迟、可靠性和可用性进行精心管理。本研究介绍了DQN-调度器，一种基于强化学习的新型代理，旨在优化云环境中的微服务调度。我们的方法旨在同时优化多个调度目标，如资源利用率、负载均衡、延迟、可靠性和可用性。据我们所知，这是首个同时满足所有这些目标的框架。DQN-调度器在现场与基准算法进行了测试。实验结果表明DQN-调度器优于基准算法。

VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching

VGFM：通过密集值指导实现表达式机器人策略，用于流量匹配

• Authors: Prajwal Koirala, Mark Campbell
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.14261
• Pdf link: https://arxiv.org/pdf/2609.14261
• Abstract Recent robot learning paradigms increasingly rely on large offline datasets of robotic interactions to train control policies. Expressive generative models enable rich and multimodal action representations, expanding the capability of this paradigm for complex robotic control. However, policy improvement with multi-step generative actors remains challenging. In offline reinforcement learning (RL), incorporating value-based objectives along generative trajectories often introduces substantial training complexity, including backpropagation through time (BPTT), auxiliary architectures, or distillation losses. We propose Value-Guided Flow Matching (VGFM), a scalable offline RL framework that enables dense value-guided shaping within a flow-based policy while avoiding BPTT and additional algorithmic overhead. VGFM parameterizes the policy as a conditional flow-matching model in action (x-prediction) space, ensuring that each intermediate flow step produces a valid robot action that can be directly evaluated by a standard offline RL critic. This design allows value guidance to be applied at randomly sampled flow times without differentiating through the entire generative trajectory, while preserving inference-time flexibility by varying the discretization of the underlying flow ODE without retraining. Evaluated on robotic locomotion and manipulation tasks in OGBench, VGFM achieves strong performance across a wide range of tasks under rigorous evaluation protocols. With minimal hyperparameter tuning, these results demonstrate that VGFM provides a simple, scalable, and effective approach for expressive policy learning in long-horizon, goal-oriented robotic control.
• 中文摘要 近期机器人学习范式越来越依赖大量离线机器人交互数据集来训练控制策略。表达式生成模型能够实现丰富且多模态的动作表示，扩展了该范式在复杂机器人控制中的能力。然而，多步生成行为者的策略改进仍具挑战性。在离线强化学习（RL）中，沿生成轨迹纳入基于价值的目标通常会带来显著的训练复杂性，包括时间反向传播（BPTT）、辅助架构或蒸馏损耗。我们提出了价值引导流匹配（VGFM）这一可扩展的离线强化学习框架，能够在基于流的策略中实现密集的价值引导形态，同时避免BPTT和额外的算法开销。VGFM将策略参数化为动作空间中的条件流匹配模型（x-预测），确保每个中间流程步骤都产生一个有效的机器人动作，可被标准离线强化学习批评者直接评估。该设计允许在随机抽样的流时间应用价值指导，而无需在整个生成轨迹中区分，同时通过改变底层流ODE的离散化而不需重新训练，保持推理时间的灵活性。VGFM在OGBench中评估机器人运动和操作任务，在严格的评估协议下，在广泛任务中取得优异表现。通过最小化超参数调优，这些结果表明VGFM为长期目标导向机器人控制中的表达性策略学习提供了一种简单、可扩展且高效的方法。

Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning

非参数方差惩罚的行为者-批评者：风险敏感强化学习的统计推断

• Authors: Saunak Kumar Panda, Tong Li, Yisha Xiang, Ruiqi Liu
• Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.14327
• Pdf link: https://arxiv.org/pdf/2609.14327
• Abstract Variance penalization is a principled approach to risk-sensitive reinforcement learning (RL) that explicitly trades expected return for policy stability. Existing methods require a dedicated second critic to estimate return variance online, adding architectural complexity and compounding estimation error during learning. We propose a nonparametric variance-penalized actor-critic (VPAC) framework that replaces the variance critic with statistically grounded online estimators based on bootstrapping and random scaling, techniques drawn from the statistical inference literature for stochastic approximation. These estimators require no auxiliary network, maintain a single-critic architecture, and produce variance penalties that are bounded by construction, enabling clean convergence analysis. We establish almost-sure convergence for both a variance-penalized Q-learning algorithm and a two-timescale actor-critic variant via the ordinary differential equation (ODE) method, requiring only that variance estimates remain bounded rather than consistent. Empirically, we evaluate across discrete and continuous stochastic environments, demonstrating that the proposed methods match or exceed the variance reduction achieved by the existing dual-critic VPAC baseline while eliminating the overhead of a second critic. We further validate on a high-temperature superconductor (HTS) manufacturing case study, where VPAC-RS (Random Scaling) achieves a 74% reduction in steady-state critical current variability and a 63% reduction in episode return standard deviation, translating directly to improved yield consistency. Our results establish nonparametric statistical inference as a practical and theoretically sound alternative to auxiliary critics for risk-sensitive RL.
• 中文摘要 方差惩罚是一种有原则的风险敏感强化学习（RL）方法，明确以预期回报换取策略稳定性。现有方法需要专门的第二批评者在线估算收益方差，增加了架构复杂性并加剧学习过程中的估计误差。我们提出一种非参数方差惩罚演员-批评者（VPAC）框架，用基于统计推断文献的自举和随机尺度的统计基础在线估计器替代方差批评者，这些技术取自统计推理文献中的随机近似技术。这些估计器无需辅助网络，保持单一批判者架构，产生的方差惩罚由构造有界，实现了清晰的收敛分析。我们通过常微分方程（ODE）方法，为方差惩罚的Q学习算法和两时间尺度的行为者-批判变量建立了几乎确定收敛性，只需方差估计值保持有界而非一致。我们通过经验分析跨离散和连续随机环境，证明所提出方法匹配或超过现有双批判者VPAC基线实现的方差减少，同时消除了第二个批判者的开销。我们还进一步验证了高温超导体（HTS）制造案例研究，VPAC-RS（随机标度）实现了稳态临界电流变异减少74%，事件回归标准差降低63%，直接转化为提升良率一致性。我们的结果确立了非参数统计推断作为风险敏感强化学习辅助批评者的实用且理论合理的替代方案。

Learning-Based Dynamic Obstacle Avoidance for a UAV Using Only Three Range Sensors

仅使用三个测距传感器的无人机基于学习的动态障碍避让

• Authors: Mohammad Reza Ranjbar Divkoti, A. Pedro Aguiar
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.14426
• Pdf link: https://arxiv.org/pdf/2609.14426
• Abstract We present a learning-based approach to kinodynamic online motion planning for an Unmanned Aerial Vehicle (UAV) operating at a fixed altitude in unknown dynamic environments, where real-time avoidance of both static and dynamic obstacles must be achieved under conditions of extreme partial observability. The UAV is controlled with a single degree of freedom (yaw only), resulting in constrained, nonholonomic motion similar to fixed-wing platforms. The proposed framework integrates a behavior grid map representation with Deep Reinforcement Learning (DRL), using Proximal Policy Optimization (PPO) for stable policy learning in continuous control. The key idea is the co-design of a state representation and control policy that enables reliable navigation using only three low-cost directional range sensors, without reliance on dense sensing modalities such as LiDAR or vision-based systems. The behavior grid map dynamically aggregates sparse measurements into a structured local representation that supports real-time decision-making for obstacle avoidance and target reaching. Extensive simulations across environments of varying sizes and obstacle densities demonstrate that the proposed standard and enhanced methods achieve higher success rates than PPO variants and Model Predictive Control (MPC) (94\% vs. 79--90\% in small-scale high-congestion scenarios, and 83\% vs. 62--71\% in large-scale high-congestion scenarios), while maintaining real-time performance. Real-world experiments across four scenarios further confirm practical feasibility, with consistent target-reaching behaviour and no collisions under the tested conditions.
• 中文摘要 我们提出了一种基于学习的运动动力学在线运动规划方法，适用于在未知动态环境中固定高度运行的无人机（UAV），在极端部分可观测条件下，必须实现实时避开静态和动态障碍物。无人机仅具备单自由度（仅偏航），导致类似固定翼平台的受限非全息运动。该框架将行为网格图表示与深度强化学习（DRL）结合，利用近端策略优化（PPO）实现持续控制中的稳定策略学习。核心思想是共同设计一种状态表示与控制策略，仅用三个低成本定向距离传感器实现可靠导航，无需依赖激光雷达（LiDAR）或基于视觉的密集传感方式。行为网格映射动态将稀疏测量聚合为结构化的局部表示，支持实时决策以避开障碍物和目标达标。跨不同规模和障碍物密度环境的广泛模拟表明，所提出的标准及增强方法比PPO变体和模型预测控制（MPC）更高的成功率（在小规模高拥堵场景中为94%对79%-90%，在大规模高拥堵场景为83%对62-71%），同时保持实时性能。四种场景的真实实验进一步确认了其可行性，在测试条件下目标达成行为一致且无碰撞。

EMoG: Emotion-Modulated Gait Generation for Expressive Humanoid Locomotion

EMoG：情感调制步态生成，用于表达性类人运动

• Authors: Yi Lu, Tianhao Jiang, Honglong Tian, Yumeng Zhang, Qingrui Zhao, Zhengtao Wang, Xiao-Xiao Long, Qiu Shen, Xun Cao
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.14432
• Pdf link: https://arxiv.org/pdf/2609.14432
• Abstract Existing humanoid locomotion systems primarily focus on stability and task execution, while integrating expressiveness with explicit locomotion control remains challenging. We propose EMoG, an emotion-modulated gait generation framework for expressive humanoid locomotion. EMoG introduces an emotional-style code with continuously adjustable intensity. Conditioned on this code and physical commands, a lightweight MLP generates expressive, command-consistent periodic gait trajectories in real time, which are tracked by a unified reinforcement learning policy for physical execution. To support training, we collect a large-scale emotion-annotated gait dataset from professional performers and develop an automated pipeline to extract physically consistent periodic gait cycles. EMoG also integrates an LLM-based parser that converts free-form language into emotional style and motion parameters for interactive control. Experiments demonstrate continuous gait-style modulation with perceptible expressive cues while maintaining command tracking. EMoG provides a practical approach to parameterized emotional-style walking for human-robot interaction.
• 中文摘要 现有的人形运动系统主要关注稳定性和任务执行，而将表现力与显式运动控制结合仍具挑战性。我们提出了EMoG，一种情感调制的步态生成框架，用于表达型人形运动。EMoG引入了一种情感风格代码，强度可持续可调。基于该代码和物理指令，轻量级MLP实时生成具表现力、命令一致的周期性步态轨迹，并通过统一的强化学习策略跟踪身体执行。为支持训练，我们收集专业表演者的大规模情感注释步态数据集，并开发自动化流程以提取身体一致的周期步态周期。EMoG还集成了一个基于LLM的解析器，将自由形式语言转换为情感风格和动作参数，实现交互式控制。实验展示了连续的步态式调制，带有可感知的表达线索，同时保持指令追踪。EMoG为人机交互提供了参数化情绪式步行的实用方法。

Grounded in Sound: Reinforcement Learning with a Frozen Acoustic Judge to Curb ASR Insertion Hallucinations

扎根于声音：用冷冻声学法官进行强化学习以抑制ASR插入幻觉

• Authors: Tingzhen Xiong, Rilin Chen, Weiwei Li, Wentao Zhang, Qicong Xie
• Subjects: Subjects: Sound (cs.SD); Multimedia (cs.MM); Audio and Speech Processing (eess.AS)
• Arxiv link: https://arxiv.org/abs/2609.14455
• Pdf link: https://arxiv.org/pdf/2609.14455
• Abstract When reinforcement learning (RL) is used for post-training automatic speech recognition (ASR), the reward almost always lives in the text space: it compares a hypothesis with the reference and never checks whether the hypothesis is supported by the audio. On highly regular speech this licenses a shortcut - guessing from a strong language prior rather than listening. Once the acoustics degrade, the shortcut runs unchecked and emits fluent but ungrounded words, i.e., insertion errors. We propose an acoustic-fidelity reward: a GRPO reward augmented with a separately pretrained, permanently frozen, non-autoregressive character-level wav2vec2-CTC acoustic judge, used strictly at training and absent at inference, where a single model decodes greedily. Trained on LibriSpeech and evaluated across a six-tier difficulty gradient including real AMI meeting speech (33,282 utterance-condition instances), the method reduces insertion errors by 28.3% on close-talking AMI-IHM and 22.3% on far-field AMI-SDM, while lowering WER on AMI-SDM from 35.89% to 34.71% and showing no detectable WER difference on the other five tiers, against a schedule-matched WER-GRPO baseline. The insertion reduction holds under a meeting-level clustered bootstrap. Four prespecified analyses support content-conditioned insertion calibration: output collapses 85-90% on unintelligible audio that preserves energy and voice activity; the gain is not recovered by the evaluated 32-best CTC rescoring configuration, yet RL internalizes it into a single greedy decoding run; and policy-only confidence yields lower insertion-AURC in all four evaluated settings. We frame this as a mechanism paper, demonstrated in one instantiation: a 7B speech LLM with a 0.3B CTC judge.
• 中文摘要 当强化学习（RL）用于后期自动语音识别（ASR）时，奖励几乎总是存在于文本空间：它将假设与参考进行比较，从不检查假设是否得到音频支持。在高度常规的语音中，这允许一条捷径——先从强语言猜测，而非听取。一旦声学性能下降，捷径未被检查，输出流畅但无根据的词汇，即插入错误。我们提出声学保真度奖励：GRPO奖励，辅以单独预训练、永久冻结、非自回归字符级wav2vec2-CTC声学评判，训练时严格使用，推理时缺席，单一模型贪婪解码。该方法在LibriSpeech上训练，并在包括真实AMI与语音对话（33,282个话语条件实例）在内的六级难度梯度下评估，在近距离发言的AMI-IHM上插入误差减少了28.3%，远场AMI-SDM减少了22.3%，同时将AMI-SDM的WER从35.89%降至34.71%，且在其他五个层级中显示无可检测的WER差异，均基于与日程匹配的WER-GRPO基线。插入减少在会议层级集群引导下成立。四个预设分析支持内容条件插入校准：在保持能量和语音活动的不可理解音频时，输出崩溃率为85-90%;该增益未被评估的32个最佳CTC重新评分配置恢复，但RL将其内化为单次贪婪解码运行;仅政策置信在所有四个评估环境中的插入AURC均较低。我们将此框架为一篇机制论文，通过一个实例化展示：一个7B语音LLM，CTC评判为0.3亿。

Learning Multi-Agent Task Assignment and Navigation in the Factory: from Simulation to Real Robots

工厂中学习多智能体任务分配与导航：从模拟到真实机器人

• Authors: Abdalwhab Bakheet Mohamed Abdalwhab, Giovanni Beltrame, David St-Onge
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.14567
• Pdf link: https://arxiv.org/pdf/2609.14567
• Abstract Reinforcement learning (RL) has shown considerable promise for robotic decision-making, yet deploying multi-agent RL (MARL) on physical multi-robot systems in industrial environments remains challenging. This paper investigates the real-world applicability of decentralized MARL for multi-robot multi-machine tending. We propose Feature-fusion Multi-Agent Proximal Policy Optimization (FMAPPO), which fuses 2D LiDAR measurements with task-specific state information to enable safe decentralized multi-robot task assignment and navigation. A complete simulation-to-reality pipeline was developed using high-fidelity robotic simulation and ROS2 and deployed on physical mobile-manipulator platforms operating under realistic real-world conditions, with the robotic arms disabled during the experiments. We further investigate the sensitivity of the learned policy to command update frequency, an important consideration for real-world deployment. Comparative evaluation in simulation demonstrated that FMAPPO significantly outperformed state-of-the-art baselines with a large effect size, achieving improvements of 106\% and 21\% in parts delivery and 48\% and 11\% in parts collection over MAPPO and SMAPPO, respectively. FMAPPO also increased machine utilization by 31 and 10 percentage points, respectively, while reducing collisions by 18\% and 15\% and increasing the safety score by 14 and 6 percentage points compared with MAPPO and SMAPPO, respectively. Furthermore, real-world experiments demonstrated that the learned decentralized policies can coordinate multiple robots to service multiple machines while maintaining safe operation under real-world sensing and control constraints. Videos of the real-world experiment are available online this https URL.
• 中文摘要 强化学习（RL）在机器人决策方面展现出显著潜力，但在工业环境中部署多智能体RL（MARL）仍具挑战性。本文探讨去中心化MARL在多机器人多机照料中的实际应用性。我们提出了特征融合多智能体近端策略优化（FMAPPO），将二维激光雷达测量与任务特定状态信息融合，实现安全的分散式多机器人任务分配与导航。通过高精度机器人仿真和ROS2开发了完整的模拟到现实流水线，部署在现实条件下运行的物理移动机械臂平台，实验中机械臂处于禁用状态。我们还进一步研究了学习策略对指令更新频率的敏感性，这是实际部署的重要考虑因素。模拟中的比较评估显示，FMAPPO显著优于最先进的基线，效果量较大，零件交付提升106%和21%，零件收集提升分别为48%和11%。FMAPPO还分别提高了机器利用率31个百分点和10个百分点，碰撞减少了18%和15%，安全得分分别提升了14个百分点和6个百分点。此外，真实实验表明，学习到的分散式策略能够协调多台机器人服务多台机器，同时在真实感知和控制约束下保持安全运行。真实实验视频可在该网 https 网址在线观看。

A note on goal-based hierarchical RL

关于基于目标的层级强化学习的说明

• Authors: Kevin Murphy
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.14605
• Pdf link: https://arxiv.org/pdf/2609.14605
• Abstract The agent-centric general value function (ACGVF) construction of \citet{tasse2026goal} lets the agent make two decisions that are normally imposed by the environment or agent designer: which goal to pursue and when to declare a goal as finished (in addition to choosing the action). This is a very general framework that subsumes almost all prior work on reinforcement learning, control and planning, as well as more general formalisms proposed in the cognitive sciences. However, it assumes the environment is fully observed, i.e., that the observation is a sufficient statistic. In \citet{murphy2025rl}, a general agent design was proposed where the policy is based on an internal belief state $z_t$ and an internal goal; however, the goals were assumed to be externally provided. In this note, we unify and extend these two approaches using the formalism of hierarchical hidden Markov models (HHMM) \citep{murphy2001hhmm}.
• 中文摘要 以智能体为中心的一般价值函数（ACGVF）构造的\citet{tasse2026goal}使智能体能够做出通常由环境或智能体设计者强加的两个决策：追求哪个目标，以及何时宣布目标已完成（除了选择行动外）。这是一个非常通用的框架，涵盖了几乎所有之前关于强化学习、控制和规划的工作，以及认知科学中提出的更一般的形式主义。然而，它假设环境是被完全观察到的，即观察是充分统计量。在\citet{murphy2025rl}中，提出了一种通用智能体设计，其中策略基于内部信念状态$z_t$和内部目标;但目标假设是外部提供的。本文将利用层级隐马尔可夫模型（HHMM）的形式主义\citep{murphy2001hhmm}统一并扩展这两种方法。

Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation

通过价值引导偏好提炼，通过密集的行为信号优化稀疏结果

• Authors: Ziyi Zhu, Daniel R. Cahn, Thomas D. Hull, Caitlin A. Stamatis, Olivier Tieleman, Guilherme B. Freire, Jinghong Chen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.14648
• Pdf link: https://arxiv.org/pdf/2609.14648
• Abstract Aligning multi-turn dialogue agents is usually framed as matching turn-level human preferences, yet direct optimization of long-term outcomes is often ineffective and prone to reward hacking. We formulate long-horizon dialogue optimization as a multi-objective reinforcement learning problem and train a multi-head value model that predicts a vector of observed user behaviors across multiple look-ahead horizons. Our findings demonstrate that a scalarized composite of dense auxiliary behavioral signals enables effective credit assignment and optimization of sparse outcomes. However, optimizing unconstrained single-objective proxies might induce policy degradations that are harmful when the agent is exposed to real users. To identify these failure modes prior to deployment, we establish a safety framework combining counterfactual user simulation with a validated dialogue-level outcome model to evaluate preference weightings and policy optimization methods. Finally, we demonstrate that distilling multi-objective value preferences into the policy via reference-anchored preference optimization matches on-policy online RL at a small fraction of its compute budget. Live A/B testing confirms that our distilled policy significantly improves long-term user retention, while simultaneously enhancing the positive behaviors and therapeutic-process markers.
• 中文摘要 对齐多回合对话代理通常被框架为匹配回合级人类偏好，但直接优化长期结果往往无效且容易被奖励黑客攻击。我们将长期对话优化构建为多目标强化学习问题，并训练多头价值模型，预测跨多个前瞻性视野观察到的用户行为向量。我们的发现表明，密集辅助行为信号的标量复合能够有效分配信用并优化稀疏结果。然而，优化无约束单目标代理可能会诱导策略降级，当代理暴露于真实用户时，这些劣化会造成有害。为在部署前识别这些失败模式，我们建立了结合反事实用户模拟与经过验证的对话级结果模型的安全框架，以评估偏好加权和策略优化方法。最后，我们证明通过引用锚定偏好优化将多目标价值偏好提炼进策略，能在策略在线强化学习计算预算的一小部分内与其匹配。实时A/B测试确认，我们的提炼策略显著提升了长期用户留存率，同时增强了积极行为和治疗过程标志。

Comparative Evaluation of MILP, MPC, and Reinforcement Learning for Commercial Battery Dispatch Under Time-of-Use Tariffs

在分时使用费率下，商业电池调度中MILP、MPC和强化学习的比较评估

• Authors: Hafiz Majid Hussain, Wajiha Samar, Lurian Klein, Pedro Nardelli
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.14776
• Pdf link: https://arxiv.org/pdf/2609.14776
• Abstract Battery energy storage systems (BESS) paired with rooftop photovoltaics (PVs) can deliver measurable cost savings under time-of-use (TOU) electricity tariffs; however, the relative performance of model-based and model-free dispatch strategies remains insufficiently benchmarked on full-year, real-world commercial datasets. This paper presents a full-year (2023) comparative evaluation of three BESS dispatch approaches using data from a commercial PV installation operating under a TOU tariff. The examined strategies include: (i) a mixed-integer linear programming (MILP) formulation with perfect foresight, providing an oracle performance benchmark under the assumed model; (ii) a model predictive control (MPC) scheme based on a day-ahead persistence forecast, representing a low-complexity deployable approach; and (iii) a soft actor-critic (SAC) deep reinforcement learning agent trained under causal information constraints. The MILP benchmark achieves an annual cost reduction of 24.6\% relative to a no-storage baseline. The persistence-based MPC approach recovers 99.2\% of this benchmark using only prior-day data. In contrast, the evaluated SAC agent yields an annual cost higher than the no-storage baseline. This outcome is analyzed in the context of known challenges in reinforcement learning for energy systems, including limited observability and reward design. Overall, the results indicate that, for the studied dataset and tariff structure, persistence-based MPC captures nearly all achievable economic benefits under practical deployment constraints, whereas the considered RL configuration does not yield competitive performance under the same information limitations.
• 中文摘要 电池储能系统（BESS）配合屋顶光伏（PV）可在使用时间（TOU）电价下实现可衡量的成本节约;然而，基于模型和无模型调度策略的相对性能在全年真实商业数据集上的基准仍不足。本文采用了基于TOU电价运营的商业光伏安装数据，对三种BESS调度方法进行了全年（2023年）比较评估。所研究策略包括：（i）具有完美前瞻性的混合整数线性规划（MILP）表述，在假设模型下提供oracle性能基准;（ii）基于日间持久性预测的模型预测控制（MPC）方案，代表低复杂度可部署方法;以及（iii）在因果信息约束下训练的软行为者-批评者（SAC）深度强化学习代理。MILP基准相较于无存储基线实现了24.6%的年成本降低。基于持久性的MPC方法仅利用前日数据回收了该基准的99.2%。相比之下，评估后的SAC代理产生的年成本高于无存储基线。这一结果是在能源系统强化学习中已知挑战的背景下分析的，包括有限的可观测性和奖励设计。总体而言，结果表明，针对研究数据集和费率结构，基于持久性的MPC几乎捕捉了在实际部署约束下所有可实现的经济效益，而所考虑的强化学习配置在相同信息限制下无法产生竞争性能。

Multi-Agent Reinforcement Learning in Markets with Congestion

拥塞市场中的多智能体强化学习

• Authors: Qixuan Zai, Randall Berry
• Subjects: Subjects: Computer Science and Game Theory (cs.GT)
• Arxiv link: https://arxiv.org/abs/2609.14827
• Pdf link: https://arxiv.org/pdf/2609.14827
• Abstract This paper investigates multi-agent reinforcement learning (MARL) in settings where firms compete for customers using congestible resources. We consider Bertrand competition in which firms compete by announcing prices and customers choose among firms based on both price and congestion. The relationship between price, congestion and the quantity of customers willing to accept service is governed by an unknown inverse demand curve, which firms must learn through experience. Each firm is modeled as a self-interested learning agent that chooses its price to maximize profit. A growing literature has shown that independently learning MARL agents can develop tacitly collusive behavior. We examine how such behavior emerges in markets with congestible resources. Our results provide insight into how learning dynamics, state representation, and strategic interaction jointly shape competition, with implications for both economic learning and the design of learning-enabled markets.
• 中文摘要 本文探讨了多智能体强化学习（MARL）在企业利用拥挤资源争夺客户的情境下。我们考察贝特朗竞争，企业通过公布价格竞争，客户根据价格和拥堵在企业之间做出选择。价格、拥堵与愿意接受服务的客户数量之间的关系由一条未知的反需求曲线支配，企业必须通过经验学习该曲线。每家企业被建模为一个利己学习主体，选择价格以最大化利润。越来越多的文献表明，独立学习的MARL代理可能发展出默契合谋行为。我们考察了这种行为如何在资源拥挤的市场中出现。我们的结果揭示了学习动态、国家代表性和战略互动如何共同塑造竞争，这对经济学习和学习驱动市场的设计都有启示。

Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

利用MPC脚手架进行灵活操作的真实世界强化学习

• Authors: Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang, Jinda Cui, Kazuhiro Sasabuchi, Jun Takamatsu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.14878
• Pdf link: https://arxiv.org/pdf/2609.14878
• Abstract Real-world reinforcement learning (RL) offers a promising route to dexterous manipulation policies that can adapt directly from physical interaction, but learning is hindered by inefficient early exploration and costly failures. We propose a framework that uses sampling-based model predictive control (MPC) as scaffolding for real-world dexterous RL, providing structured prior experience and task-directed guidance during learning without human demonstrations or corrective actions. A small set of MPC trajectories is first used to populate an offline replay buffer and to pretrain the actor and critic. During online learning, MPC intermittently guides data collection while an off-policy Soft Actor-Critic learner trains from both prior MPC experience and newly collected physical interaction, with control gradually transitioning to the learned policy. On continuous in-hand rotation with a 16-DoF Allegro hand, the method reaches 100\% success in policy-only evaluation (5/5 trials) after 7 minutes of online RL, following initialization with 20 MPC trajectories collected on hardware in 12 minutes. Online training incurs about three object drops on average. After 20 minutes of online learning, the policy achieves more than five times the rotation speed of the MPC controller. It completes 1000 consecutive rotations over more than 110 minutes without a drop. Ablations show complementary benefits from MPC-based pretraining, retained MPC experience, and online MPC guidance. We further demonstrate rapid adaptation to different object geometries and successful goal-conditioned reorientation, showing that the framework enables efficient, low-intervention, real-world dexterous RL.
• 中文摘要 现实世界强化学习（RL）为实现能够直接从物理互动中适应的灵巧操作策略提供了有前景的途径，但学习受限于低效的早期探索和代价高昂的失败。我们提出了一个框架，利用基于采样的模型预测控制（MPC）作为现实世界灵巧强化学习的支架，在学习过程中提供结构化的既往经验和任务导向指导，无需人工演示或纠正措施。首先使用一小组MPC轨迹填充离线回放缓冲区，并预训练演员和批评者。在线学习过程中，MPC间歇性地引导数据收集，而非策略的软演员-批评学习者则结合先前MPC经验和新收集的物理互动进行训练，控制逐步过渡到已学策略。在连续手持旋转并使用16景深Allegro手时，该方法在在线RL进行7分钟后，策略评估（5次试验中5次试验中）达到100%成功率，初始化后在硬件上收集了20条MPC轨迹，时间为12分钟。在线训练平均产生约三次物体掉落。经过20分钟在线学习后，该策略的旋转速度超过MPC控制器的五倍多。它在超过110分钟内连续完成1000次旋转且未掉落。消融显示出基于MPC的预训练、保留的MPC经验和在线MPC指导的互补益处。我们还进一步展示了对不同对象几何的快速适应和成功的目标条件重定向，表明该框架实现了高效、低干预、现实世界灵巧的强化学习。

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

《四十度蓝色：通过模态条件强化学习实现质量-多样性对齐》

• Authors: Jiayi Yuan, Hangoo Kang, James Jihao Liu, Yejin Choi, Vikram Iyer, Liwei Jiang, Natasha Jaques
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.14896
• Pdf link: https://arxiv.org/pdf/2609.14896
• Abstract A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at inference time. This degradation is especially limiting for applications requiring open-ended exploration and pluralistic perspectives, such as scientific ideation and creative writing. We present MoDA (Mode-conditioned Diversity Alignment), an online post-training RL algorithm that jointly optimizes generation quality and diversity, inspired by the coordination perspective in multi-agent reinforcement learning (MARL). MoDA trains a single shared LLM policy conditioned on abstract numbered roles, where each role acts as an agent competing to produce outputs distinct from the others. This formulation encourages mode-conditioned agents to explore complementary regions of the high-quality output space without requiring hand-crafted personas or architectural modifications. MoDA employs a prompt-adaptive quality gating mechanism that calibrates a reference quality threshold and grants diversity rewards only to responses that meet the threshold, preventing reward-hacking behaviors that compromise response quality. To study quality-diversity tradeoffs, we evaluate MoDA on a comprehensive suite of benchmarks spanning seven general capability tasks and four domain-specific diversity tasks in scientific ideation and creative writing. MoDA improves SBERT diversity by 265% on the Infinite-Chat held-out prompts, while increasing average general capability pass@1 by 10.3% over the Qwen3-8B baseline. Compared with the strongest DivPO baseline, MoDA improves SBERT diversity from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.4 (+53.8%), while improving average general capability pass@1 by 7.0%. Overall, MoDA provides a drop-in alternative to standard post-training methods that preserves and expands the model's expressive output space while improving quality.
• 中文摘要 LLM对齐训练的一个显著副产品是模式崩溃：输出多样性的逐步丧失，使模型在推理时的表达力缩小。这种退化对需要开放式探索和多元视角的应用尤其受限，如科学构思和创意写作。我们介绍了MoDA（模式条件多样性对齐），这是一种在线训练后强化学习算法，结合生成质量和多样性，灵感来自多智能体强化学习（MARL）中的协调视角。MoDA训练一个基于抽象编号角色的共享LLM策略，每个角色作为代理竞争产生独立输出。这种表述鼓励模式条件代理探索高质量输出空间的互补区域，无需手工定制角色或架构修改。MoDA采用提示自适应质量门槛机制，校准参考质量阈值，仅对达到该阈值的响应给予多样性奖励，防止破坏响应质量的奖励黑客行为。为研究质量与多样性权衡，我们在涵盖科学构思和创意写作中七个通用能力任务和四个领域特定多样性任务的综合基准测试中评估MoDA。MoDA在无限聊天保留提示上提升SBERT多样性265%，平均通用能力pass@1较Qwen3-8B基线提升10.3%。与最强的DivPO基线相比，MoDA将SBERT多样性从0.274提升至0.482（+75.9%），E-Vendi从2.86提升至4.4（+53.8%），平均通用能力pass@1提升7.0%。总体而言，MoDA提供了标准后期方法的可直接替代方案，既保留并扩展了模型的表达输出空间，又提升了质量。

Cloud Workflow Scheduling Based on Graph Attention-Driven Hierarchical Reinforcement Learning

基于图的注意力驱动层级强化学习的云工作流调度

• Authors: Zongjin Li, Shaohan Feng, Chunxi Yang, Wenbo Wang
• Subjects: Subjects: Machine Learning (cs.LG); Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2609.14952
• Pdf link: https://arxiv.org/pdf/2609.14952
• Abstract Dynamic cloud workflow scheduling must balance deadline satisfaction, container utilization, and energy consumption while dealing with stochastic task-execution speeds, placement-dependent communication, and coupled task and container decisions. Workflows are naturally modeled as directed acyclic graphs (DAGs), but conventional vector- or matrix-based states do not fully capture their dependency topology. To better represent task urgency and structural relationships, we assign predicted sub-deadlines to tasks and use a multi-head graph attention network (GAT) to extract dependency information from the evolving DAGs. Based on these representations, we develop a Graph Attention-Driven Hierarchical Reinforcement Learning (GA-HRL) framework and model the scheduling process as an event-driven hierarchical semi-Markov decision process (SMDP). Workflow arrivals and task completions trigger scheduling events. At each scheduling event, the Task Scheduling (TS) agent first processes the currently ready tasks by assigning them to admissible existing containers or requesting new ones. The requested containers are then processed by the Container Scheduling (CS) agent for host placement before the environment advances. The two agents are trained alternately using separate Proximal Policy Optimization (PPO). Experiments on the 2018 Alibaba cluster trace show that GA-HRL maintains competitive workflow success rate and, in settings where success is comparable, generally achieves higher container utilization and lower energy consumption. Under the largest speed variation, it trades a small success-rate margin for substantially lower energy. Simulation code is available at: this https URL.
• 中文摘要 动态云工作流调度必须在处理随机任务执行速度、布局依赖通信以及任务与容器耦合决策之间取得截止日期满足、容器利用率和能耗之间的平衡。工作流自然被建模为有向无环图（DAG），但传统的向量或矩阵状态无法完全捕捉其依赖拓扑结构。为了更好地表示任务紧迫性和结构关系，我们为任务分配预测的子截止时间，并使用多头图关注网络（GAT）从演化中的DAG中提取依赖信息。基于这些表示，我们开发了一个图关注驱动层级强化学习（GA-HRL）框架，并将调度过程建模为事件驱动的层级半马尔可夫决策过程（SMDP）。工作流程的到达和任务完成触发调度事件。在每个调度事件中，任务调度（TS）代理首先通过将当前已准备好的任务分配到可接受的现有容器或请求新的容器来处理。随后，容器调度（CS）代理会处理这些请求的容器以进行主机安置，然后环境才会继续推进。两个代理轮流使用各自的近端策略优化（PPO）进行训练。2018年阿里巴巴集群追踪的实验表明，GA-HRL保持了具有竞争力的工作流程成功率，并且在成功率相当的环境中，通常能实现更高的容器利用率和更低的能耗。在最大速度变化下，它以较小的成功率为交换，以换取显著更低的能耗。模拟代码可在：此 https URL。

HiGFRL: Hierarchical Graph Fusion-Driven Reinforcement Learning for Dependency-Aware Task Scheduling in Heterogeneous Cloud

HiGFRL：用于异构云依赖感知任务调度的层级图融合驱动强化学习

• Authors: Tiangang Li, Shi Ying, Xiangbo Tian
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.14968
• Pdf link: https://arxiv.org/pdf/2609.14968
• Abstract Online scheduling of dependency-aware tasks in heterogeneous cloud clusters is a fundamental yet challenging problem due to the complex interplay between DAG topologies and multi-dimensional resource constraints. While DRL has shown promise, existing GNN-based approaches often struggle to efficiently model high-order topological dependencies and suffer from loose coupling between task and resource states, leading to myopic scheduling decisions. To address these limitations, we propose HiGFRL, a Hierarchical Graph Fusion-Driven Reinforcement Learning framework. HiGFRL constructs a novel three-level state representation comprising a Static Hypergraph, a Dynamic Global Graph, and a Local Bipartite Graph to explicitly model the interplay between task dependencies and real-time cluster dynamics. Specifically, we design a fusion-driven dual-network architecture to optimize RL decision-making, where a Context Fusion Allocator integrates local bipartite matching features with fused global context to execute precise task-to-node allocation, and a Global State Evaluator leverages the global dynamic graph representation to accurately estimate expected long-term cumulative reward. Furthermore, we incorporate a topology-prior-guided hybrid reward mechanism that distills static topological priors into the learning process to accelerate convergence. Extensive experiments using real-world Alibaba cluster traces demonstrate that HiGFRL significantly outperforms heuristics and DRL baselines. Specifically, in challenging large-scale high-load scenarios, HiGFRL reduces the Makespan by up to 32.55%, and optimizes the average task flow time and average task wait time by 13.58% and 13.79%, respectively. Experimental results confirm that HiGFRL not only significantly improves cluster throughput but also ensures superior QoS by substantially reducing queuing delays. Code Release:this https URL.
• 中文摘要 异构云集群中依赖感知任务的在线调度是一个基础性且具有挑战性的问题，因为DAG拓扑与多维资源约束之间存在复杂相互作用。尽管DRL展现出潜力，但现有基于GNN的方法常常难以高效建模高阶拓扑依赖关系，且任务状态与资源状态之间的耦合松散，导致调度决策短浅。为解决这些局限性，我们提出了HiGFRL，一种分层图融合驱动强化学习框架。HiGFRL构建了一种新颖的三级状态表示，包括静态超图、动态全局图和局部二分图，以显式建模任务依赖关系与实时集群动态之间的相互作用。具体来说，我们设计了一种融合驱动的双网络架构，以优化强化学习决策，其中上下文融合分配器将本地二分匹配特征与融合的全局上下文整合，执行精确的任务到节点分配，而全局状态评估器则利用全局动态图表示准确估算预期的长期累计奖励。此外，我们还采用了拓扑先验引导的混合奖励机制，将静态拓扑先验提炼进学习过程，加速收敛。利用真实世界阿里巴巴集群轨迹进行的大量实验表明，HiGFRL显著优于启发式方法和日程学习基础。具体来说，在具有挑战性的大规模高负载场景中，HiGFRL可将完成时长缩短多达32.55%，并分别优化平均任务流时间13.58%和平均任务等待时间13.58%和13.79%。实验结果证实，HiGFRL不仅显著提升了集群吞吐量，还通过大幅减少排队延迟，确保了卓越的服务质量。代码发布：此 https URL。

Learning to Solve Stochastic Controls with Unknown Drifts and Running Rewards: Theory, Algorithms and Convergence

学习如何求解未知漂移和运行奖励的随机控制：理论、算法与收敛

• Authors: Jin Ma, Gaozhan Wang, Jianfeng Zhang, Xunyu Zhou
• Subjects: Subjects: Machine Learning (cs.LG); Numerical Analysis (math.NA); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.14972
• Pdf link: https://arxiv.org/pdf/2609.14972
• Abstract We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.
• 中文摘要 我们研究连续时间及可能的高维随机控制问题，其中漂移系数和运行奖励函数未知。由于缺乏这些模型原语，我们采用Wang、Zariphopoulou和Zhou（2020）的探索强化学习（RL）框架，采用宽松控制和熵正则化。目标是开发理论基础、高效且可扩展的强化学习算法，以学习最优值函数（同时解探索性HJB方程）和最优探索性反馈控制策略。当扩散系数不包含控制时，我们基于仅依赖原始动力学扩散部分的辅助状态过程，采用最优值函数及其梯度的概率表示。通过对某些正确定义的映射及其不动点进行细致分析，这引出了我们的策略迭代算法及其收敛过程。我们通过各种数值示例展示了算法的性能。最后，我们研究一种特殊的控制依赖扩散情形，其中需要对Hessian进行概率表示。

Not All Prompts Are Equal: Exploration-Guided Prompt Scaffolding for Multimodal Reinforcement Post-Training

并非所有提示都相同：探索引导提示支架用于多模态培训后强化

• Authors: Yuanhao Yue, Qianli Ma, Chengyu Wang, Haoting Wang, Lei Shen, Jun Huang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.15051
• Pdf link: https://arxiv.org/pdf/2609.15051
• Abstract Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs). Central to our approach is the $\textit{Exploration Potential Score} (EPS)$, a lightweight rollout-based proxy for prompt utility derived from KL-regularized policy improvement theory, computable directly from on-policy rollout statistics without additional overhead. Rather than discarding low-utility prompts, we use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated with GRPO on Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7\% relative improvement in-domain and gains of 11.5\% on MathVision and 11.1\% on MMMU-Pro.
• 中文摘要 在线强化学习（RL）中的训练提示在对当前策略的信息量上差异显著：有些已饱和，有些则难以产生可靠学习信号，但在标准训练下两者均获得同等的推广预算。我们提出了一个探索引导的提示支架框架，在多模态大型语言模型（MLLM）的强化后训练过程中动态调整训练提示的分布。我们方法的核心是$\textit{探索潜力评分}（EPS）$，这是一个基于基层推广的轻量级提示效用代理，基于KL正则化策略改进理论，可直接从政策启动时的统计计算，无需额外开销。我们不舍弃低效用提示，而是使用教师模型生成支架式重写，保持原始任务意图，同时使后续培训更具信息量，将教师监督重新定位为训练数据的细化，而非输出模仿。在Geo3K和MMK12上与GRPO集成后，我们的方法在域内和非分发基准测试中均持续优于基线，域内相对提升高达9.7%，MathVision提升11.5%，MMMU-Pro提升11.1%。

What Does an LLM Learn from Reinforcement Learning? A Mechanistic Interpretability Perspective with Fixed-SAE Track

LLM从强化学习中学到什么？机械可解释性视角与固定SAE轨道

• Authors: Lingheng Du, Yiming Tang, Xufeng Duan, Dianbo Liu
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.15064
• Pdf link: https://arxiv.org/pdf/2609.15064
• Abstract Reinforcement learning (RL) is widely utilized in large language model training to improve targeted capabilities, yet how RL reshapes a model remains poorly understood. Prior attempts to explain how RL works largely offer behavioral perspectives, leaving open what RL gives a model at the representation level: can RL create genuinely novel features, and which existing features does it enhance or suppress? Recent developments in mechanistic interpretability suggest sparse autoencoders (SAEs) as a promising lens to decompose internal activations into human-interpretable features; however, they cannot be directly applied to tracking change across training. In this work, we introduce Fixed-SAE Track, a framework that trains one shared SAE per considered layer on activations pooled across the base model and all RL checkpoints, holding every feature direction fixed so that representation shifts are rigorously defined through the activations of interpretable SAE latents, including the detection of emerging novel features. Validated across multiple datasets and RL algorithms, we find that RL-induced drift is small, gradual, concept specific, and concentrated in late layers, mainly enhancing the sampling rates of a small set of ladder tokens, formatting scaffolding such as step breaks and answer delimiters, rather than reshaping problem content. Steering these features into the base model recovers around 80% of RL's performance gain, suggesting that RL primarily elicits capabilities the model already possesses, much as steering does. We further design a synthetic benchmark with features known by construction to test whether RL can instill genuinely novel features. We believe Fixed-SAE Track provides a principled approach to tracking representation shifts and offers representational evidence for understanding how reinforcement learning changes the inner representation of LLMs.
• 中文摘要 强化学习（RL）被广泛应用于大型语言模型训练中，以提升目标能力，但强化学习如何重塑模型仍不甚明了。此前解释强化学习工作原理的尝试大多提供行为视角，留下了强化学习在表征层面赋予模型的开放空间：强化学习能否创造真正新颖的特征，以及它增强或抑制哪些现有特征？近期机制性可解释性的发展表明，稀疏自编码器（SAE）是将内部激活分解为人类可解释特征的有前景的视角;然而，它们无法直接应用于跨训练的变化追踪。本研究介绍了固定SAE轨迹框架，该框架在每个考虑层中训练一个共享SAE，针对基础模型和所有强化学习检查点的激活，固定每个特征方向，使表征转移通过激活可解释的SAE潜伏（包括新出现特征的检测）严格定义。通过多个数据集和强化学习算法的验证，我们发现强化学习引起的漂移小、渐进、概念特定，且集中在后期层，主要提升一小部分梯形图代币的采样率，格式化步分隔符和答案分隔符等脚手架，而非重塑问题内容。将这些特征引入基础模型可恢复约80%的强化学习性能提升，表明强化学习主要激发模型已有的能力，就像引导一样。我们还设计了一个合成基准测试，包含构建已知特征，以测试强化学习是否能赋予真正新颖特征。我们认为固定SAE轨道提供了原则性方法跟踪表征变化，并为理解强化学习如何改变大型语言模型内部表征提供了表征证据。

Salesforce Koa: An Enterprise Language Model for Agentic Tool Use

Salesforce Koa：用于代理工具的企业语言模型

• Authors: Zixiang Chen, Sufeng Niu, Yingchi Liu, Wenting Zhao, Akshara Prabhakar, Shubham Mehrotra, Bin Bi, Zhujun Lan, Katherine Tan, Mohammad Ramezanali, Tulika Manoj Awalgaonkar, Monojit Banerjee, Jielin Qiu, Shiva Kumar Pentyala, Zhepeng Cen, Anupam Tripathi, Ali Ziaei, Regunathan Radhakrishnan, Darvish Lee Shadravan, Shelby Heinecke, Sitaram Asur, Silvio Savarese, James Zhu, Phil Mui, Huan Wang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.15066
• Pdf link: https://arxiv.org/pdf/2609.15066
• Abstract We present Salesforce Koa, an enterprise language model built by post-training the open-weight Nemotron-3-Super-120B foundation model with reinforcement learning using Group Relative Policy Optimization (GRPO). Salesforce Koa is trained on public and synthetically generated data, with no customer data, to improve tool use and agentic capabilities while preserving strong general-purpose performance. Its distinctive component is a simulation-to-reward pipeline that expands workflow specifications into persona-conditioned multi-turn tasks with task-resolution rewards grounded in successful tool use for data-dependent requests. For enterprise domains, these specifications are written in Agent Script, Salesforce's declarative language for building Agentforce agents; for public tool-use domains, we synthesize the workflow structure directly. The same simulation and grounded-reward machinery drives GRPO across both. Across public tool-use, agentic-reasoning, and enterprise Customer Relationship Management (CRM) benchmarks, Salesforce Koa improves over its open-weight base, with the clearest gains on multi-turn tool use, and surpasses a strong proprietary baseline while remaining below the strongest frontier models. These results show that specification-driven reinforcement learning is a practical path to specializing open-weight foundation models for enterprise agentic tasks.
• 中文摘要 我们介绍 Salesforce Koa，这是一个企业语言模型，通过对开放权重 Nemotron-3-Super-120B 基础模型进行后期训练，并采用组相对策略优化（GRPO）进行强化学习构建。Salesforce Koa 基于公共和合成生成的数据进行训练，不包含客户数据，旨在提升工具使用和代理能力，同时保持强大的通用性能。其独特组件是从仿真到奖励的流程，将工作流规范扩展为基于角色条件的多回合任务，任务解决奖励基于成功使用工具以满足数据依赖请求。对于企业领域，这些规范用 Agent Script 编写，这是 Salesforce 用于构建代理代理的声明式语言;对于公共工具使用领域，我们直接综合工作流程结构。同一模拟和基于基础的奖励机制驱动 GRPO 在两者之间。在公共工具使用、代理推理和企业客户关系管理（CRM）基准测试中，Salesforce Koa 在开放权重基础基础上有所提升，在多回合工具使用上取得最明显的提升，同时超越强有力的专有基线，同时仍低于最强前沿模型。这些结果表明，规范驱动的强化学习是将开放权重基础模型专门化为企业代理任务的切实路径。

Refinement-based Flow Policy Optimization

基于精细化的流程策略优化

• Authors: Bumgeun Park, Hyukjun Yang, Donghwan Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.15123
• Pdf link: https://arxiv.org/pdf/2609.15123
• Abstract Flow-based policies offer an expressive representation for online reinforcement learning, but conventional flow matching requires samples drawn from the distribution to be modeled. This poses a challenge when the desired action distribution is defined only implicitly by a Q-function, since directly sampling actions from the resulting distribution is generally intractable. We propose Refinement-Based Flow Policy Optimization (RFPO), a novel framework for training a flow policy in online reinforcement learning by alternating between Q-guided sample refinement and self-target flow matching. RFPO first generates actions from Gaussian noise using the current flow policy and then uses a finite-step stochastic refinement procedure to move them toward an energy-based distribution induced by the Q-function. Each refined action is then paired with its corresponding initial noise sample and used as a fixed target for flow-matching training. By repeatedly refining its own outputs and learning from the resulting targets, RFPO incorporates Q-guidance into the policy without requiring direct samples from the target distribution, while retaining the capacity to represent multiple action modes. We further provide a theoretical analysis of the distributional dynamics induced by RFPO. Across six continuous-control tasks, RFPO matches or outperforms a standard Gaussian-policy baseline on almost every task. Experiments on six synthetic two-dimensional target distributions with diverse geometries demonstrate that RFPO captures complex multimodal structure without mode collapse.
• 中文摘要 基于流的策略为在线强化学习提供了表达性表述，但传统流匹配需要从分布中抽取样本进行建模。当期望的动作分布仅由Q函数隐式定义时，这带来了挑战，因为直接从结果分布中抽样动作通常难以处理。我们提出了基于精细化的流策略优化（RFPO），这是一种通过在Q引导样本细化和自目标流匹配之间交替进行在线强化学习中训练流策略的新框架。RFPO首先利用当前流策略从高斯噪声中生成动作，然后使用有限步随机细化过程将其向由Q函数诱导的基于能量的分布移动。每个精炼动作随后与其对应的初始噪声样本配对，作为流匹配训练的固定目标。通过反复优化自身输出并从目标中学习，RFPO将Q-导引纳入策略，无需直接从目标分布中取样，同时保留表示多种动作模式的能力。我们还进一步对RFPO诱导的分布动态进行了理论分析。在六个连续控制任务中，RFPO几乎在所有任务上都达到或超过标准高斯策略基线。在六个具有多样几何形状的合成二维目标分布上的实验表明，RFPO能够捕捉复杂的多模态结构而不存在模式崩溃。

AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference

AdaVSkip：自适应视觉令牌跨层跳跃，实现高效的MLLM推断

• Authors: Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.15131
• Pdf link: https://arxiv.org/pdf/2609.15131
• Abstract Multimodal large language models (MLLMs) require substantial computation to process numerous visual tokens across all transformer layers. Most methods for efficient MLLM inference exploit horizontal redundancy by compressing visual tokens. Beyond token reduction, recent studies exploit vertical redundancy through early exit or fixed-layer skipping. However, we find that the extent and distribution of this redundancy vary across inputs and differ between self-attention and MLP modules. Motivated by these observations, we propose AdaVSkip, which equips each layer with two lightweight routers that independently determine whether visual tokens pass through by or skip the self-attention and MLP modules. These decisions collectively define an input-specific visual-computation path, but their discrete and non-differentiable nature makes learning effective paths challenging. To address this challenge, we develop a progressive two-stage training framework that updates only the routers while keeping the backbone frozen. Stage I establishes an initial routing policy through supervised training with input-specific targets derived from module-wise necessity scores. To further align the routing policy with task performance, Stage II uses reinforcement learning to optimize routing decisions with direct feedback from generated answers. It combines an answer correctness reward with a skip-consistency reward that discourages excessive retention of visual-token computation. Across three MLLM backbones, AdaVSkip maintains strong task performance with substantially less computation. On LLaVA-NeXT-7B, AdaVSkip reduces FLOPs by 53.2\% while preserving the original model's average performance. Combining it with visual token compression increases this reduction to 91.2\%, while retaining 97.2\% of the original performance on average.
• 中文摘要 多模大型语言模型（MLLM）需要大量计算来处理跨所有变换器层的大量视觉符号。大多数高效MLLM推理方法通过压缩视觉符号来利用水平冗余。除了符号减少外，近期研究还利用了通过早期退出或固定层跳跃实现的垂直冗余。然而，我们发现这种冗余的程度和分布在不同输入之间存在差异，且在自注意模块和MLP模块间存在差异。基于这些观察，我们提出了AdaVSkip，该方法为每一层配备两个轻量级路由器，独立判断视觉标记是否通过或跳过自注意和MLP模块。这些决策共同定义了输入特定的视觉计算路径，但其离散且不可微的特性使得学习有效路径具有挑战性。为应对这一挑战，我们开发了一个渐进式两阶段训练框架，仅更新路由器，同时保持主干网冻结。第一阶段通过监督训练建立初始路由策略，目标基于模块级需求评分，针对输入特定目标。为了进一步使路由策略与任务性能对齐，第二阶段使用强化学习，利用生成答案的直接反馈优化路由决策。它结合了答案正确性奖励和跳过一致性奖励，防止过度保留可视化令牌计算。在三个MLLM骨干网中，AdaVSkip以显著减少的计算量保持了强劲的任务性能。在LLaVA-NeXT-7B上，AdaVSkip在保持原始模型平均性能的同时，减少了53.2%的FLOPs。结合视觉令牌压缩，这一降低率提升至91.2%，同时平均保留97.2%的原始性能。

GRAVA: Grounded Reasoning-to-Action Representation and Learning for Autonomous Driving

GRAVA：自动驾驶的基于推理到行动的表征与学习

• Authors: Xiao Liu, Haoyu Li, Jianghao Leng, Lin Wang, Chao Sun
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.15169
• Pdf link: https://arxiv.org/pdf/2609.15169
• Abstract Driving vision-language-action (VLA) models increasingly reason before acting, but their intermediate reasoning is often weakly grounded in physical scene evidence and loosely connected to executable behavior. We present GRAVA, a framework built around Grounded Reasoning-to-Action (GRA), which unifies grounding, reasoning, and action generation in a single autoregressive stream. GRA links action-relevant language references to 2D visual regions and ego-centric physical states, organizes object interactions and decisions in a trajectory-anchored typed graph, and serializes this structure into grounded reasoning. A single VLM generates this reasoning followed by a compact Executable Planner action that is deterministically decoded into a continuous trajectory. We further introduce an agentic GRA data construction pipeline that combines forward scene grounding with backward trajectory anchoring, and use it to build GR-NavSim with 2.2M grounded question-answer pairs and 70K GRA reasoning traces. A progressive training strategy develops grounded cognition through pre-training, establishes the reasoning-to-action interface through imitation, and improves driving behavior through reinforcement learning and exploration. Using about 60% of the available human driving demonstrations for action supervision, GRAVA-8B achieves state-of-the-art performance among purely autoregressive driving models on the full NAVSIM benchmark. On an internal long-tail benchmark, full GRA improves key-object compliance and Closed-loop Driving Score by 19.3% and 20.5% over action-only prediction, respectively. These results show the benefit of preserving action-relevant physical evidence from grounded reasoning through executable action generation.
• 中文摘要 驱动视觉-语言-行动（VLA）模型越来越多地先推理再行动，但其中间推理往往基于物理场景证据薄弱，且松散地关联于可执行行为。我们介绍GRAVA，这是一个围绕基于基于推理到行动（GRA）构建的框架，将基础化、推理和动作生成统一为单一自回归流。GRA将与动作相关的语言引用与二维视觉区域和以自我为中心的物理状态连接起来，组织对象交互和决策在轨迹锚定的类型图中，并将该结构序列化为有根基的推理。单个VLM生成该推理，随后生成一个紧凑的可执行规划器动作，并被确定性地解码为连续轨迹。我们进一步引入了结合前向场景基础和后向轨迹锚定的代理GRA数据构建流程，并用以构建包含220万对基准问答和7万GRA推理轨迹的GR-NavSim。渐进式训练策略通过预训练培养基础认知，通过模仿建立推理与行动接口，并通过强化学习和探索提升驾驶行为。利用约60%的人类驾驶演示用于行动监督，GRAVA-8B在纯自回归驾驶模型中实现了全NAVSIM基准测试的顶尖性能。在内部长尾基准测试中，全GRA分别将关键对象合规性和闭环驾驶得分提升19.3%和20.5%，分别比仅动作预测提升20.5%。这些结果显示，通过可执行的动作生成，保留基于基础推理的动作相关物理证据具有优势。

Admissable: Training Reinforcement Learning Agents against Adversarial Missingness

可采纳：培训强化学习代理以防对抗性缺失

• Authors: Paul Stahlhofen, Luca Hermes, Tim Kochs, Markus Vieth, Barbara Hammer
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.15297
• Pdf link: https://arxiv.org/pdf/2609.15297
• Abstract In order to make Reinforcement Learning algorithms applicable in real world scenarios, safety must be ensured even under adverse operating conditions. In this work, we consider the challenge of adversarial feature missingness: a scenario in which an adversary occludes features from the agent's observation in order to reduce performance as much as possible. We formally define adversarial missingness for Reinforcement Learning and compare it to the related concepts of $\ell_\infty$-norm bounded adversarial perturbations and learning with missing data. We develop an adversarial training algorithm and show its effectiveness in increasing robustness against adversarial missingness on three MuJoCo benchmark environments. Compared to a baseline trained with random uniform missingness, our method achieves better robustness on all three tasks.
• 中文摘要 为了使强化学习算法在现实场景中适用，即使在不利的操作条件下也必须确保安全性。在本研究中，我们探讨了对抗性特征缺失的挑战：即对手为了尽可能降低性能而遮蔽了代理观察中的特征。我们正式定义了强化学习的对抗缺失，并将其与相关的$\ell_\infty$范数界限对抗扰动和缺失数据学习概念进行比较。我们开发了一种对抗训练算法，并展示了其在三种MuJoCo基准环境中提升对抗对抗缺失的鲁棒性的有效性。与随机均匀缺失训练的基线相比，我们的方法在这三种任务上都实现了更好的鲁棒性。

Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

从可靠负面中学习：基于信心的测试时间适配以实现GUI接地

• Authors: Yizhou Liu, Fei Tang, Yuchen Yan, Zhengxi Lu, Songqin Nong, Tao Jiang, Wenhao Xu, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.15307
• Pdf link: https://arxiv.org/pdf/2609.15307
• Abstract Graphical User Interface (GUI) grounding is essential for autonomous agents to map natural language instructions to precise screen coordinates. However, existing supervised fine-tuning and reinforcement learning methods are constrained by the high cost of annotation, creating a scalability bottleneck. In this paper, we introduce a label-free test-time training paradigm driven by two key insights: (1) confidence patterns in coordinate tokens are a better indicator than full-sequence confidence, and (2) in sparse GUI coordinate spaces, negative samples offer more reliable learning signals than potentially noisy positive ones. We first propose Confidence-Anchored Learning (CAL), which utilizes coordinate-token confidence to filter pseudo-labels and assign distance-based binary rewards. Building on this, we develop Confidence-Anchored Negative Learning (CANL), which exclusively optimizes the model using negative samples to bypass the risks of incorrect positive samples. Experimental results demonstrate that CANL-7B achieves 92.1% on ScreenSpot-V2. On more challenging ScreenSpot-Pro, CANL-7B reaches 33.8%, an 8.9% absolute improvement over the base model. Our findings establish coordinate-token confidence as a powerful alternative to manual annotations for scalable GUI agent development.
• 中文摘要 图形用户界面（GUI）基础对于自主智能体将自然语言指令映射到精确屏幕坐标至关重要。然而，现有的监督式微调和强化学习方法受限于标注成本高昂，造成了可扩展性瓶颈。本文引入了无标签测试时间训练范式，基于两个关键洞见：（1）坐标符号中的置信模式比全序列置信度更能提供指标;（2）在稀疏的GUI坐标空间中，负样本比潜在噪声的正样本更可靠地传递学习信号。我们首先提出了信心锚定学习（CAL），利用坐标标记置信度过滤伪标签并分配基于距离的二元奖励。基于此，我们开发了信心锚定负向学习（CANL），专门利用负样本优化模型，绕过错误正样本的风险。实验结果显示，CANL-7B在ScreenSpot-V2上实现了92.1%。在更具挑战性的ScreenSpot-Pro上，CANL-7B达到33.8%，比基础模型绝对提升了8.9%。我们的发现确立了坐标标记置信度作为可扩展GUI代理开发中手工注释的有力替代方案。

Evaluation Metrics for Safe Reinforcement Learning

安全强化学习的评估指标

• Authors: Lindsay Spoor, Aske Plaat, Thomas Moerland
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.15315
• Pdf link: https://arxiv.org/pdf/2609.15315
• Abstract Safe reinforcement learning (RL) is commonly formalized as a Constrained Markov Decision Process (CMDP), in which an agent maximizes expected reward while keeping its expected cumulative cost below a specified safety bound. Existing safe RL benchmarks predominantly report whether an algorithm is safe on average, following this expectation-based guarantee. We argue that this convention is insufficient to reliably characterize an algorithm's true safety: it fails to capture how often and how severely the safety bound is violated, whether this holds consistently across tasks and safety bounds, and whether training-time behavior is representative of behavior of the final converged policy. Therefore, we introduce (i) evaluation metrics for safe RL that address each of these concerns and in addition allow for aggregation across tasks and safety bounds. We furthermore define (ii) a safety tier system to systematically categorize and compare algorithms in terms of safety and reliability at both training and for a final policy. Using this framework, we provide (iii) an empirical safety evaluation across multiple safety navigation tasks. Our results show that aggregate metrics, distributional reporting, and task- and safety bound-specific results each reveal information the other metrics cannot. We therefore recommend reporting all three jointly, rather than compressing this information into a single value, as is common practice. We provide SafeRLEval, an open-source evaluation suite to support the reliable characterization of safety in future safe RL research.
• 中文摘要 安全强化学习（RL）通常形式化为受限马尔可夫决策过程（CMDP），其中智能体在保持预期累计成本低于指定安全界限的前提下，最大化期望奖励。现有的安全强化学习基准主要报告算法的平均安全性，遵循基于期望的保证。我们认为，这一惯例不足以可靠地描述算法的真实安全性：它未能捕捉安全界限被违反的频率和严重程度，未能反映安全界限在任务和安全界限间是否一致，以及训练时间行为是否代表最终收敛策略的行为。因此，我们引入了（i）安全强化学习的评估指标，解决了这些问题，并允许跨任务和安全界限进行聚合。我们还进一步定义了（ii）安全层级系统，系统地在培训和最终策略中对算法的安全性和可靠性进行分类和比较。利用该框架，我们提供了（iii）跨多个安全导航任务的实证安全性评估。我们的结果表明，汇总指标、分布报告以及任务和安全界限特定结果均能揭示其他指标无法提供的信息。因此，我们建议将三者联合报告，而非像常见做法那样将信息压缩为单一值。我们提供SafeRLEval，一个开源评估套件，支持未来安全强化学习研究中安全性的可靠表征。

Robust and Efficient Communication for Multi-Agent Learning

多智能体学习的稳健高效通信

• Authors: Rafael Pina, Varuna De Silva, Corentin Artaud
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.15361
• Pdf link: https://arxiv.org/pdf/2609.15361
• Abstract Effective communication is a cornerstone of distributed intelligence in Multi-Agent Reinforcement Learning (MARL), yet ensuring that generated messages are both informative and robust to physical constraints remains a significant challenge. This paper introduces Multi-Agent Regularized Communication (MARC), a novel framework inspired by information-theoretic principles of conditional mutual information. MARC employs an attention-based architecture coupled with a unique message regularization mechanism designed to minimize uncertainty regarding future system states, thereby inducing the learning of highly representative communication protocols. Crucially, we evaluate MARC under stringent communication bottlenecks and lossy channels, simulating the real-world constraints of autonomous robotic networks and decentralized systems. Our results demonstrate that MARC significantly outperforms state-of-the-art methods in complex cooperative domains. Furthermore, we provide a deep analysis of message characteristics, proving that MARC maintains high operational performance even under significant data compression, offering a scalable path for deploying intelligent agents in resource-constrained environments.
• 中文摘要 有效通信是多智能体强化学习（MARL）分布式智能的基石，但确保生成的消息既具信息量又能对物理约束保持鲁棒性，仍是一项重大挑战。本文介绍了多智能体正则化通信（MARC），这是一个受信息理论条件互信息原理启发的新框架。MARC采用基于注意力的架构和独特的消息正则化机制，旨在最大限度减少对未来系统状态的不确定性，从而促进高度代表性的通信协议学习。关键是，我们在严格的通信瓶颈和有损信道下评估MARC的应用，模拟自主机器人网络和去中心化系统的现实约束。我们的结果表明，MARC在复杂的合作领域中显著优于最先进方法。此外，我们对消息特性进行了深入分析，证明MARC即使在大量数据压缩下仍能保持高运行性能，为在资源受限环境中部署智能代理提供了可扩展的路径。

CodeTS: Verifiable Text-to-Time Series Generation via Executable Code

CodeTS：通过可执行代码生成可验证的文本到时间序列

• Authors: Xudong Yuan, Shunyu Liu, Tongya Zheng, Huiping Zhuang, Mingli Song, Kaixuan Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.15393
• Pdf link: https://arxiv.org/pdf/2609.15393
• Abstract Text-to-Time Series Generation (Text-to-TS) provides a promising paradigm for synthesizing time series from natural language, enabling scenario-specific generation when real observations are scarce or costly to acquire. However, existing methods typically lack an explicit mechanism for deriving generation logic from textual descriptions to guide time series synthesis. In this paper, we propose CodeTS, a verifiable framework that uses code as an intermediate generation interface, reformulating Text-to-TS generation as a Text-to-Code-to-TS process. CodeTS first maps textual temporal descriptions into an explicit code space, where executable code specifies how textual requirements shape target temporal patterns, and then obtains the time series through code execution. To learn this code generation process reliably without real code annotations, CodeTS constructs aligned Text-Code-TS triplets from structured temporal attributes for supervised initialization. More importantly, we further design multi-stage execution-based rewards that verify format validity, code executability, and time series quality, enabling real Text-TS pairs to provide training signals for Reinforcement Learning with Verifiable Rewards (RLVR). Extensive experiments on eight benchmarks across short, medium, and long generation lengths demonstrate that CodeTS provides a strong zero-shot solution for Text-to-TS generation, outperforming LLM-based baselines and achieving better averaged results than supervised generative baselines trained on the target datasets.
• 中文摘要 文本到时间序列生成（Text-to-TS）为从自然语言合成时间序列提供了有前景的范式，使得在真实观测值稀缺或获取成本高时实现场景特定生成。然而，现有方法通常缺乏显式机制，能够从文本描述中推导生成逻辑以指导时间序列合成。本文提出了CodeTS，一个可验证的框架，利用代码作为中间生成接口，将文本到TS生成重新表述为文本到代码到TS的过程。CodeTS首先将文本时间描述映射到显式代码空间，其中可执行代码指定文本需求如何塑造目标时间模式，然后通过代码执行获得时间序列。为了在不依赖真实代码注释的情况下可靠学习该代码生成过程，CodeTS从结构化的时间属性构建对齐的文本-代码-TS三元组进行监督初始化。更重要的是，我们进一步设计了多阶段基于执行的奖励，验证格式有效性、代码可执行性和时间序列质量，使真实的文本-托管服务对能够为可验证奖励强化学习（RLVR）提供训练信号。在八个基准测试中，跨短、中、长生成长度的广泛实验表明，CodeTS为文本到托管存储器生成提供了强有力的零样本解决方案，优于基于LLM的基线，并获得比在目标数据集上训练的监督生成基线更好的平均结果。

Dynamics-Informed Reinforcement Learning for Agile and Energy-Efficient Locomotion of a Monopedal Hopping Quadcopter

动力学知情强化学习，实现单足跳跃四旋翼的敏捷且节能的行走

• Authors: Ruigang Chen, Qi Zhang, Zhicheng Zhong, Zhuorui Yun, Yizhar Or, Mingyi Liu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.15399
• Pdf link: https://arxiv.org/pdf/2609.15399
• Abstract Although aerial-legged robots offer combined agility and efficiency, controlling high-speed hopping under complex hybrid dynamics is challenging. Reinforcement Learning (RL) is promising but prone to energy-inefficient "reward hacking". We propose a Dynamics-Informed RL framework for a monopedal hopping quadcopter. By embedding a target Specific Energy into the reward, we constrain the optimization to a physically viable energy manifold, ensuring stable hopping behaviour. By rewarding the phase-consistent behavior, it can encourage bio-inspired stance-phase impulse. Furthermore, penalizing the electro-mechanical power waste induces the motors generate an efficient impulse. This enables the policy to inject energy strictly during spring restitution without heuristic state machines. MuJoCo simulations validate robust height regulation and forward velocity tracking up to 2.0 m/s despite severe attitude-contact coupling. Ultimately, our approach yields a highly agile hopping gait, reducing energy consumption by 82% and 73% compared to hovering baselines and inefficiency baseline, respectively.
• 中文摘要 尽管空中腿机器人具备灵活性和效率，但在复杂混合动力学下控制高速跳跃仍具挑战性。强化学习（RL）前景看好，但易受能量低效的“奖励黑客”影响。我们提出了一种动力学知情强化学习框架，用于单足跳跃四旋翼飞行器。通过将目标比能量嵌入奖励，我们将优化限制在物理可行的能量流管内，确保跳跃行为稳定。通过奖励相位一致性行为，可以鼓励仿生姿态相冲。此外，惩罚机电功率浪费可促使电机产生高效冲动。这使得该策略能够严格在弹簧恢复时注入能量，无需启发式状态机。MuJoCo模拟验证了尽管存在严重姿态接触耦合，仍能实现高达2.0米/秒的稳健高度调节和前进速度跟踪。最终，我们的方法实现了高度灵活的跳跃步态，分别比悬停基线和低效率基线降低了82%和73%的能耗。

HISPO: Hierarchical Importance-Sampling Policy Optimization with Entropy-Derived Segments

HISPO：层级重要性-抽样策略优化，含熵衍生片段

• Authors: Quoc-Vinh Lai-Dang, Hyo-Sang Shin
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.15471
• Pdf link: https://arxiv.org/pdf/2609.15471
• Abstract Reinforcement learning with verifiable rewards (RLVR) has become a central approach for improving mathematical reasoning in language models, but long-form completions introduce a difficult credit-assignment problem: different parts of a solution trace may contribute unevenly to final correctness. Existing policyoptimization objectives for RLVR commonly apply importance-sampling correction at either the token level (GRPO, DAPO) or the sequence level (GSPO), imposing different granularities for assigning credit across a response. We introduce Hierarchical Importance-Sampling Policy Optimization (HISPO), a segment-level policy-optimization method that constructs rollout-time entropy-derived contiguous segments, assigns soft entropy-based saliency weights, and applies clipped importance-sampling correction at the segment granularity. This provides an intermediate correction unit between token-level GRPO/DAPO and sequence-level GSPO. We evaluate HISPO by fine-tuning Qwen3-1.7B-Base on mathematical reasoning tasks. Across six benchmarks, HISPO improves Pass@8 over the strongest baseline on all benchmarks and matches or exceeds the strongest baseline in Acc@8 on five of them. On AIME25, HISPO improves over GRPO by +3.75 Acc@8 and +3.78 Pass@8, and over GSPO by +2.50 Acc@8 and +1.27 Pass@8. These results suggest that segment-level correction is a promising granularity for RLVR in long-form mathematical reasoning.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为提升语言模型数学推理的核心方法，但长形式补全引入了一个棘手的学分分配问题：解轨迹的不同部分可能对最终正确性贡献不均。现有RLVR策略优化目标通常在令牌级（GRPO、DAPO）或序列级（GSPO）应用重要性抽样校正，从而在响应中赋予不同的功分。我们引入了层级重要性抽样策略优化（HISPO），这是一种分段级策略优化方法，构建基于滚动时间的熵衍生连续段，分配基于软熵的显著性权重，并在段粒度处应用截断的重要性抽样修正。这提供了令牌级GRPO/DAPO和序列级GSPO之间的中间修正单元。我们通过微调Qwen3-1.7B基础在数学推理任务中评估HISPO。在六个基准测试中，HISPO在所有基准测试中均Pass@8提升至最强基线，其中五项基准测试Acc@8上均匹配或超过最强基线。在AIME25中，HISPO在GRPO上提升了+3.75 Acc@8和+3.78 Pass@8，较GSPO提升了+2.50 Acc@8和+1.27 Pass@8。这些结果表明，分段级修正是RLVR在长形式数学推理中具有前景的细度。

Strong and Compact Policies for Submodular Markov Decision Processes via LP-Based Submodular Orienteering

通过基于LP的次模块定向，为亚模块化马尔可夫决策过程制定强而紧凑的策略

• Authors: Lars Rohwedder, Rico Zenklusen
• Subjects: Subjects: Data Structures and Algorithms (cs.DS); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.15539
• Pdf link: https://arxiv.org/pdf/2609.15539
• Abstract Finding policies for Markov Decision Processes (MDPs) is a central problem in areas such as Reinforcement Learning and Operations Research. Here, we have to repeatedly choose an action that should be performed by an agent. Depending on the action and the current state of the agent, the agent collects a reward and randomly transitions into a new state. The goal is to maximize the reward in expectation over a finite time horizon of length $H$. We consider a recently introduced variant that generalizes the traditionally additive reward function in the model to a monotone submodular one, which allows for capturing a range of interesting applications. Without the stochastic component, this problem is equivalent to the Submodular Orienteering problem, where the goal is to find an $s$-$t$ walk in a directed graph maximizing a monotone submodular function under a length constraint. We present a novel LP-based algorithm for Submodular Orienteering using ideas from the Sherali-Adams hierarchy and Round-or-Cut. Our guarantees are comparable to the known quasi-polynomial time logarithmic approximation for Submodular Orienteering, but also extend to the setting of Submodular Markov Decision Processes. In the polynomial time regime, we present an $O(n^{\varepsilon})$-approximation (and $O(H^{\varepsilon})$ for Submodular MDPs) for every $\varepsilon >0$, where $n$ is the number of vertices, which was unknown even for Submodular Orienteering. Prior to our work, the best known approximation guarantee for Submodular MDPs had an approximation ratio linear in $H$. Beyond these algorithmic results, our methods reveal a trade-off between the approximation guarantee and the number of previously visited vertices on which an agent conditions its decision.
• 中文摘要 寻找马尔可夫决策过程（MDP）策略是强化学习和运筹学等领域的核心问题。在这里，我们必须反复选择一个由智能体执行的动作。根据动作和智能体当前状态，智能体收集奖励并随机过渡到新状态。目标是在长度$H$的有限时间范围内最大化期望奖励。我们考虑了一个新引入的变体，将模型中传统的加法奖励函数推广为单调子模函数，从而能够捕捉多种有趣的应用。没有随机分量，该问题等价于子模定向问题，其目标是在有向图中找到一个$s$-$t$的路径，以最大化单调子模函数，且在长度约束下实现。我们提出了一种基于LP的子模定向算法，利用Sherali-Adams层级和轮割的思想。我们的保证与已知的子模定向时对数近似相当，同时也扩展到子模马尔可夫决策过程的设定。在多项式时间区间，我们对每个$\varepsilon（>0$）给出$O（n^{\varepsilon}）$近似（对于子模MDP则为$O（H^{\varepsilon}）$），其中$n$为顶点数，这在子模定向中也未知。在我们工作之前，次模MDP已知的最佳近似保证是线性的线性$H$。除了这些算法结果外，我们的方法还揭示了近似保证与代理作为决策条件的已访问顶点数量之间的权衡。

Specifying Reward Functions for RL Without Environment Sampling

在不使用环境抽样的情况下指定强化学习的奖励函数

• Authors: Stephane Hatgis-Kessell, W. Bradley Knox, Emma Brunskill
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.15544
• Pdf link: https://arxiv.org/pdf/2609.15544
• Abstract Enabling human stakeholders to specify reward functions that lead to their desired outcomes is a key challenge in deploying reinforcement learning agents. Preference-based methods such as online RLHF can reduce the burden of manual reward design, but they require repeatedly training policies, sampling trajectories from the real world, and eliciting feedback, making them impractical in settings where environment interaction is computationally expensive or unsafe. We introduce Experience-Free Autonomous Reward Specification (EARS), a method for learning reward functions from preferences without environment interaction. Our approach uses a structured LLM-mediated process to construct a small set of expressive reward features from a task description and the environment observation space, then strategically samples imagined trajectories in this feature space and learns feature weights from preferences over the imagined trajectory pairs. We evaluate on three long-horizon domains: pandemic lockdown regulation design, insulin administration for diabetes patients, and autonomous vehicle control on a highway. We compare EARS to baselines that also enable reward specification without environment interaction--namely, methods that directly prompt an LLM to generate a reward function. When learning from either ground-truth preference labels or preferences labeled by a LLM, EARS designs reward functions that are more aligned with the ground truth reward function that produced the preferences or LLM context than these baselines. These results suggest that preference-based reward specification remains effective without environment sampling, enabling practical reward design in settings where collecting real trajectories is costly or infeasible.
• 中文摘要 使人类利益相关者能够指定导致期望结果的奖励函数，是部署强化学习代理时的关键挑战。基于偏好的方法如在线RLHF可以减轻人工奖励设计的负担，但它们需要反复训练策略、从现实世界中抽样轨迹并获取反馈，因此在环境交互计算高或不安全的环境中不够实用。我们引入了无经验自主奖励规范（EARS），这是一种无需环境交互即可从偏好中学习奖励函数的方法。我们的方法采用结构化的LLM介导过程，从任务描述和环境观察空间构建一小组表达性奖励特征，然后战略性地在该特征空间中采样想象轨迹，并从偏好中学习特征权重。我们在三个长期领域进行评估：疫情封锁监管设计、糖尿病患者的胰岛素注射以及高速公路自动驾驶车辆控制。我们将EARS与同样支持奖励指定且无需环境交互的基线进行比较——即直接提示LLM生成奖励函数的方法。在从地面真实偏好标签或LLM标记的偏好中学习时，EARS设计的奖励函数更符合产生偏好或LLM上下文的真实奖励函数，而非这些基线。这些结果表明，基于偏好的奖励规范在不进行环境抽样的情况下依然有效，使得在收集真实轨迹成本高或不可行的环境中实现实用奖励设计成为可能。

VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding

VideoScout：学习能动主动探索，采用自适应推理节奏，实现长视频理解

• Authors: Weixin Xu, Zhenyu Yang, Bing Wang, Shengsheng Qian, Changsheng Xu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.15606
• Pdf link: https://arxiv.org/pdf/2609.15606
• Abstract Multimodal Large Language Models (MLLMs) have achieved remarkable progress on short video understanding yet remain limited on long videos due to the limited visual context window. Prevailing approaches rely on uniform frame sampling or recent coarse-to-fine agentic zooming, both of which struggle to localize sparse, decisive evidence in sufficiently long videos. We formulate long video understanding as a \textbf{Sequential Evidence Acquisition (SEA)} problem, in which an agent reads the video turn by turn along the temporal axis, deciding at each turn how fast to watch, what evidence to retain, when to revisit uncertain segments, and when to stop and answer. Inspired by this view, we propose \textbf{VideoScout}, a multi-turn reasoning agent that instantiates the SEA paradigm through adaptive reasoning pacing. Specifically, by dynamically controlling the viewing pace, VideoScout enables efficient traversal of long videos within a bounded visual context window, allowing the agent to access more video content while balancing content analysis depth with reading efficiency. To train VideoScout, we construct VideoScout-66K, a set of over 66K high-quality exploration turns from 10K answer-verified trajectories, and adopt a two-stage pipeline: cold-start supervised fine-tuning teaches the agent per-turn output format, while the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm performs trajectory-level reinforcement learning with a composite reward that jointly considers answer accuracy, output format compliance, and the temporal alignment between the agent's viewing progress and the teacher's answer timing measured by intersection-over-union (IoU). Extensive experiments on long video understanding and reasoning benchmarks demonstrate that our 7B model achieves strong performance compared with existing trained 7B agentic models.
• 中文摘要 多模态大型语言模型（MLLM）在短视频理解方面取得了显著进展，但由于视觉上下文窗口有限，长视频仍受限。主流方法依赖于均匀帧抽样或近期的粗细代理缩放，这两种方法在足够长的视频中难以定位稀疏且决定性的证据。我们将长视频理解表述为\textbf{顺序证据获取（SEA）}问题，代理沿时间轴逐条阅读视频，决定观看速度、保留哪些证据、何时重访不确定片段以及何时停下回答。受这一观点启发，我们提出了\textbf{VideoScout}，一种多回合推理代理，通过自适应推理节奏实现SEA范式。具体来说，通过动态控制观看速度，VideoScout 能够高效地在有界的视觉上下文窗口内横扫长视频，使智能体能够访问更多视频内容，同时平衡内容分析深度与阅读效率。为训练 VideoScout，我们构建了 VideoScout-66K，这是一组基于 10K 个经过验证的答案轨迹中超过 66K 的高质量探索回合，采用两阶段流程：冷启动监督微调教授智能体每回合输出格式，而解耦剪辑和动态采样策略优化（DAPO）算法则执行轨迹级强化学习，并采用综合奖励，综合考虑答案准确性、输出格式合规性以及代理观看进度与教师回答时间（通过交叉对并测量）之间的时间对齐。大量长视频理解和推理基准测试显示，我们的7B模型相较于现有训练的7B代理模型表现优异。

AI-Native Open RAN: A Roadmap from xApps and rApps to Autonomous Network Agents

AI原生开放RAN：从xApp和rApp到自主网络代理的路线图

• Authors: Ryan Barker, Alireza Ebrahimi Dorcheh, Tolunay Seyfi, Mohammad Raihan Uddin, Alireza Mohammadhosseini, Julia Boone, Stephen Streit, Drew Schlesener, Fatemeh Afghah
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2609.15704
• Pdf link: https://arxiv.org/pdf/2609.15704
• Abstract Open Radio Access Networks (O-RAN) have emerged as a transformative paradigm for future wireless systems by introducing openness, virtualization, disaggregation, and programmable intelligence through the RAN Intelligent Controller (RIC). The availability of standardized interfaces and near-real-time control loops has created unprecedented opportunities for integrating artificial intelligence (AI) into radio access network management and optimization. Over the past several years, a broad range of AI techniques have been proposed to address key O-RAN challenges such as radio resource management, network slicing, traffic prediction, mobility management, interference mitigation, and spectrum sharing. Despite significant progress, existing solutions often remain task-specific, require extensive retraining, and exhibit limited generalization across deployment environments and network conditions. This paper presents a comprehensive review of AI-enabled O-RAN systems and provides a unifying perspective on the evolution of intelligence in wireless networks. We first examine the O-RAN architecture and the role of intelligence within near-real-time and non-real-time RIC frameworks. We then develop a taxonomy of AI approaches for O-RAN, covering machine learning, deep reinforcement learning (DRL), digital-twin-assisted optimization, and emerging foundation-model-based architectures.
• 中文摘要 开放无线接入网（O-RAN）通过通过RAN智能控制器（RIC）引入开放性、虚拟化、拆分和可编程智能，已成为未来无线系统的变革范式。标准化接口和近实时控制环路的可用性为将人工智能（AI）整合进无线接入网络管理与优化创造了前所未有的机遇。近年来，提出了多种AI技术以应对O-RAN的关键挑战，如无线资源管理、网络切片、流量预测、移动性管理、干扰缓解和频谱共享。尽管取得了显著进展，现有解决方案通常仍停留在任务特定性，需要大量再培训，且在部署环境和网络条件下的推广性有限。本文对AI驱动的O-RAN系统进行了全面综述，并提供了关于无线网络智能演进的统一视角。我们首先考察了O-RAN架构以及智能在近实时和非实时RIC框架中的作用。随后，我们为O-RAN开发了AI方法的分类法，涵盖机器学习、深度强化学习（DRL）、数字孪生辅助优化以及新兴的基于基础模型的架构。

Assembling the CREW: A Collaborative Multi-agent Reinforcement Learning Framework for Automated Related Work Generation

组装CREW：一个用于自动化相关工作生成的协作多智能体强化学习框架

• Authors: Hai-Dang Dang, Bao-Yen Pham, Bao Nguyen, Tran Thi Huong, Huynh Thi Thanh Binh
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.15721
• Pdf link: https://arxiv.org/pdf/2609.15721
• Abstract Automatic Related Work Generation (RWG) significantly reduces the human time and effort required to author the Related Work Section (RWS) of a research paper. However, prior methods leveraging multi-agent Large Language Models (LLMs) typically rely on a predefined workflow, where each agent is responsible for a specific step in the entire process. This rigid, static inter-agent coordination limits the adaptive collaboration required to synthesize complex scientific literature. To address this limitation, we propose CREW (Collaborative Reinforcement Learning for Related Work Generation), a novel framework where LLM agents bypass heuristic pipelines to dynamically coordinate by autonomously selecting actions, such as Retrieve, Disseminate, Compose, and Critique, driven by a policy optimized via Independent Proximal Policy Optimization (IPPO). Extensive experiments on a standard RWG benchmark demonstrate that our approach yields substantial quality improvements over strong existing baselines, while significantly reducing token costs. Code is available at this https URL
• 中文摘要 自动相关工作生成（RWG）显著减少了撰写研究论文相关工作部分（RWS）所需的人力和时间。然而，以往利用多智能体大型语言模型（LLMs）的方法通常依赖预定义的工作流程，每个代理负责整个过程中的特定步骤。这种僵化、静态的代理间协调限制了综合复杂科学文献所需的自适应协作。为解决这一限制，我们提出了CREW（相关工作生成协作强化学习），这是一个新颖框架，LLM代理绕过启发式流水线，通过通过独立近端策略优化（IPPO）优化的策略，自主选择动作（如检索、传播、合成和批判）动态协调。在标准RWG基准测试上的大量实验表明，我们的方法相较于现有强有力基线实现了显著的质量提升，同时显著降低了代币成本。代码可在此 https URL 获取

JEPLO: Joint-Embedding Predictive Learning for LiDAR-Based Legged Locomotion

JEPLO：基于激光雷达的关节嵌入预测学习

• Authors: Qihao Yuan, Yixuan Qiu, Ziyu Cao, Ming Cao, Kailai Li
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.15770
• Pdf link: https://arxiv.org/pdf/2609.15770
• Abstract Light detection and ranging (LiDAR) remains less explored than RGB-D sensing for perceptive legged locomotion, and existing LiDAR-based approaches often rely on explicit mapping. We present JEPLO (Joint-Embedding Predictive learning for legged LOcomotion), a single-stage learning framework for mapping-free, LiDAR-based perceptive locomotion for legged robots. We introduce a proprio-exteroceptive JEPA (PE-JEPA) world model to learn predictive egocentric terrain representations from onboard observations, including raw LiDAR scans. A concurrent JEPA-teacher-student (CJTS) pipeline is further proposed to train a locomotion policy informed by JEPA latent representations in simulation using deep reinforcement learning with a simple reward formulation. The framework achieves successful sim-to-real transfer, enabling omnidirectional traversal of diverse terrains, including long staircases and high boxes, with lightweight onboard computation. Evaluations demonstrate greater robustness than existing perceptive locomotion frameworks, particularly under degraded perception caused by occlusion, sparsity and noise. Further analysis validates JEPLO's ability to retain task-relevant information under these challenging conditions. We open-source our implementation, experimental datasets, and hardware setup designs this https URL.
• 中文摘要 光探测与测距（LiDAR）在感知腿部运动方面仍不如RGB-D传感被广泛探索，现有基于激光雷达的方法通常依赖显式映射。我们提出了JEPLO（腿部移动的关节嵌入预测学习），这是一个单阶段学习框架，用于无映射、基于激光雷达的腿部机器人感知运动。我们引入了本体-外感知型JEPA（PE-JEPA）世界模型，用于通过机载观察（包括原始激光雷达扫描）学习预测性的以自我为中心的地形表征。我们还提出了一个并行的JEPA-师生（CJTS）流水线，利用深度强化学习和简单的奖励表述，训练基于JEPA潜在表征在模拟中的运动策略。该框架实现了成功的模拟到现实传输，实现了对多样地形的全向穿越，包括长楼梯和高箱，并实现了轻量级的车载计算。评估显示出比现有感知运动框架更为鲁棒，尤其是在遮挡、稀疏和噪声导致的感知退化下。进一步分析验证了JEPLO在这些挑战条件下保留任务相关信息的能力。我们将实现、实验数据集和硬件设置设计开源，将此https URL。

Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation

应对稀疏证据：通过显式上下文选择与巩固实现能动视觉RAG

• Authors: Yucheng Shen, Lingyong Yan, Jiulong Wu, Shuaiqiang Wang, Jianmin WU, Dawei Yin, Min Cao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.15800
• Pdf link: https://arxiv.org/pdf/2609.15800
• Abstract Visual Retrieval-Augmented Generation (VRAG) empowers models to navigate and answer queries about visually rich documents by retrieving relevant page images as visual evidence and reasoning over their content. However, effectively utilizing this visual evidence is usually impeded by two main challenges. First, answer-relevant evidence is sparse and may be concentrated in a small region of one page or dispersed across multiple pages. Second, existing agentic methods often generate answers based on raw exploration trajectories or compressed textual memories rather than an explicitly organized set of supporting images, making answers susceptible to exploration noise and obscuring the evidence-backed reasoning trace. We argue that the bottleneck lies not only in evidence discovery but also in its preservation and organization before answer generation. We propose SCoRE (Selection and Consolidation for Robust Evidence), a unified agent loop for explicit evidence selection and consolidation. During exploration, SCoRE retains only query-relevant observations and their source pointers in a maintained textual ledger, preserving earlier evidence while keeping the visual context bounded. At termination, it reloads the referenced original images and consolidates the visual evidence for answering, arranging it into a logical sequence. This decouples final reasoning from exploratory trial-and-error while ensuring strict visual grounding via indexed claim-to-image linkages. To enable end-to-end optimization of this unified rollout, our training paradigm combines filtered cold-start trajectory distillation with evidence-aware reinforcement learning, whose reward promotes evidence coverage, consolidation compactness, and answer correctness.
• 中文摘要 可视化检索增强生成（VRAG）赋能模型通过检索相关页面图像作为视觉证据并推理其内容，来导航和回答关于视觉丰富文档的查询。然而，有效利用这些视觉证据通常受到两个主要挑战的阻碍。首先，与答案相关的证据稀疏，可能集中在一页的小区域，也可能分散在多个页面上。其次，现有的代理方法往往基于原始探索轨迹或压缩文本记忆生成答案，而非明确组织的一组支持图像，使答案容易受到探索噪声影响，并掩盖了证据支持的推理痕迹。我们认为瓶颈不仅在于证据发现，还在于在答案生成前的保存和组织。我们提出了SCoRE（稳健证据的选择与巩固），这是一个统一的代理循环，用于显性证据的选择和巩固。在探索过程中，SCoRE仅保留与查询相关的观察及其来源指针，保持文本账本，既保留早期证据，又保持视觉上下文的边界。在结束时，它重新加载引用的原始图像，并整合视觉证据以供回答，将其排列成逻辑序列。这将最终推理与探索性的试错解耦，同时通过索引的声明与图像关联确保严格的视觉基础。为了实现这一统一推广的端到端优化，我们的训练范式结合了过滤冷启动轨迹蒸馏与证据感知强化学习，其奖励促进了证据覆盖率、巩固性紧凑性和答案正确性。

Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport

通过价值加权最优运输学习多模态单步流策略

• Authors: Jaehun Shon, Jinha Choi, Jongwook Jeon, Jongmin Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.15883
• Pdf link: https://arxiv.org/pdf/2609.15883
• Abstract Offline reinforcement learning aims to learn a policy solely from fixed datasets, which often contain multimodal action distributions. Flow policies can naturally represent such multimodal behaviors, but learning an efficient one-step flow policy remains challenging: standard value guidance often leads to mode collapse or exploits overestimation bias in out-of-distribution regions. To address this, we introduce One-step Flow policy via Optimal Transport (OptiFlow), a framework for one-step flow policy learning as a structured sample-allocation problem. OptiFlow jointly trains a value-aware reference flow policy and an efficient one-step policy, coupling their action samples through state-wise entropic optimal transport. For each state, critic-estimated values define the priority of distillation target actions, while the action-distance cost ensures geometrically compatible pairings. By avoiding direct critic maximization, our transport-guided approach enables in-distribution exploitation by anchoring the one-step policy to high-value, dataset-supported modes without the risk of out-of-distribution divergence. Experimental results demonstrate that OptiFlow effectively captures optimal multimodal behaviors and achieves strong performance across diverse offline RL benchmarks. Our code is available at this https URL.
• 中文摘要 离线强化学习旨在仅从固定数据集中学习策略，这些数据集通常包含多模态动作分布。流策略自然可以表示此类多模态行为，但学习高效的一步流策略仍然具有挑战性：标准值指导常导致模式崩溃或利用分布外区域的高估偏差。为此，我们引入了通过最优传输（OptiFlow）进行一步流策略的框架，作为结构化样本分配问题。OptiFlow联合训练一个值感知的参考流策略和一个高效的一步策略，通过状态的熵最优传输耦合它们的动作样本。对于每个状态，批评者估计值定义蒸馏目标动作的优先级，而动作距离成本确保几何匹配。通过避免直接批评者最大化，我们的传输引导方法通过将一步策略锚定于高价值、数据集支持模式，实现分布内利用，而无需担心分布外的偏离。实验结果表明，OptiFlow有效捕捉了最优多模态行为，并在多种离线强化基准测试中实现了强劲性能。我们的代码可在此 https URL 获取。

Inoculation Midtraining with Learned Neologisms

学习新词的培训中接种

• Authors: Kyle O'Brien, Edward James Young, Puria Radmard, Nathalie Kirch, Cameron Tice, Tomek Korbak, David Demitri Africa
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.15886
• Pdf link: https://arxiv.org/pdf/2609.15886
• Abstract Large language models (LLMs) often learn both desirable and undesirable properties during post-training. We study whether midtraining, an earlier training stage, can shape which of these properties later generalise. We introduce Inoculation Midtraining, a technique that teaches a base model that unsafe behaviour belongs to a designated context, as indicated by the neologism (a new token) introduced during midtraining, and then post-trains the model on unsafe data within that context. We then evaluate the model outside the context, with the neologism excluded from the system prompt. Across supervised fine-tuning and reinforcement learning post-training regimes, we find that Inoculation Midtraining can reduce misalignment while preserving the transfer of benign data properties (e.g., speaking in German or Shakespearean prose). However, our approach does not outperform standard Inoculation Prompting, is sensitive to training configuration, and produces a leaky boundary that nearby contextual cues can reactivate. These results show that inoculation with a learned association introduced via midtraining can shape selective generalisation. Still, more work is needed before this approach can become a load-bearing component in a developer's safety framework.
• 中文摘要 大型语言模型（LLMs）通常在训练后学习到理想和不良属性。我们研究中期训练阶段（较早的训练阶段）是否能影响这些属性后续的推广。我们引入了接种中训练技术，该技术教导基础模型不安全行为属于指定上下文，由中期训练引入的新词（新词）表示，然后在该上下文中对模型进行后期训练。然后我们在上下文之外评估模型，将新词排除在系统提示中。在监督微调和强化学习的训练后期体系中，我们发现接种中训练能够减少错位，同时保留良性数据属性的转移（例如，用德语或莎士比亚式散文说话）。然而，我们的方法未能超越标准的接种提示，对训练配置敏感，并产生一个泄漏边界，附近的上下文线索可以重新激活。这些结果表明，通过训练中引入的学习关联接种可以塑造选择性泛化。然而，在该方法成为开发者安全框架中承载负重的组成部分之前，仍需更多工作。

Safe Meta-Reinforcement Learning via Information Space Reachability

通过信息空间可达性实现安全的元强化学习

• Authors: Zeyang Li, Sunbochen Tang, Navid Azizan
• Subjects: Subjects: Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.15915
• Pdf link: https://arxiv.org/pdf/2609.15915
• Abstract Meta-reinforcement learning (meta-RL) enables agents to adapt to unseen tasks with limited experience. Despite its promise, the application of meta-RL in real-world tasks is hindered by safety requirements, which have been underexplored in prior work. In this paper, we propose a safe meta-RL framework that explicitly accounts for safety during adaptation. Our key insight is to reason about safety in the information space, which captures both the physical state and the agent's belief over the underlying task. Within this space, we introduce a safety value function that measures the probability of the agent avoiding unsafe regions indefinitely. We show that this function satisfies a self-consistency condition and a Bellman equation, which make it learnable via meta-RL. Based on this formulation, we develop a safe meta-RL algorithm that learns the safety value function and leverages it for safety filtering and constrained policy optimization. Experiments on meta-RL benchmarks demonstrate the effectiveness of the proposed method.
• 中文摘要 元强化学习（meta-RL）使智能体能够在有限经验下适应未见任务。尽管前景看好，元强化学习在现实任务中的应用仍受限于安全要求，而这些要求在以往工作中未被充分探讨。本文提出了一个安全元强化学习框架，明确考虑适应过程中的安全性。我们的核心见解是推理信息空间中的安全性，该空间既捕捉物理状态，也反映智能体对底层任务的信念。在此空间内，我们引入了安全价值函数，衡量智能体无限避免不安全区域的概率。我们证明该函数满足自一致性条件和贝尔曼方程，使得通过元强化学习可学习。基于该表述，我们开发了一种安全的元强化学习算法，学习安全值函数并利用其进行安全过滤和受限策略优化。元强化学习基准测试的实验证明了该方法的有效性。

Bellman Policy Optimization

贝尔曼策略优化

• Authors: Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.15987
• Pdf link: https://arxiv.org/pdf/2609.15987
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了大型语言模型（LLMs）的推理能力。我们引入了Bellman策略优化（BPO），这是一种基于策略镜像下降（PMD）衍生的无批评方法。对于带有终端奖励的自回归生成，BPO利用Bellman方程将PMD重新表述为轨迹级目标。该重述避免在中间状态估计状态值。我们证明其与原始PMD目标具有相同的唯一最优解。我们通过近似该目标推导实际的BPO损失。其错配纠正权重是互补代币概率的平滑比值。数学推理基准测试的实验证明了BPO的有效性。

ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids

ResSafe：人形生物的残余强化学习安全过滤

• Authors: Gechen Qu, Tong Zhang, Bike Zhang, Yen-Jen Wang, Koushil Sreenath, Claire Tomlin, Jason Jangho Choi
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.15988
• Pdf link: https://arxiv.org/pdf/2609.15988
• Abstract Safe control of humanoid robots remains challenging due to their high-dimensional dynamics, contact-rich interactions, and sensitivity to disturbances. Although reinforcement learning has enabled effective locomotion and motion tracking, learned policies can still generate unsafe actions that lead to instability or falls. In this work, we propose residual reinforcement learning as an implicit safety-filtering mechanism for safe humanoid control. Instead of relying on a single nominal policy to simultaneously balance performance, safety, and robustness, we decouple performance and safety. The nominal policy focuses solely on task performance, while a residual policy learns safety corrections. This decoupling leads to a better performance--safety Pareto trade-off and avoids the need for careful tuning of multiple competing reward terms within a single policy training. We show that the residual policy can act as an implicit safety filter.
• 中文摘要 由于其高维动态、丰富的接触互动以及对干扰的敏感性，安全控制类人机器人依然充满挑战。尽管强化学习实现了有效的运动和运动追踪，但已学习策略仍可能产生导致不稳定或坠落的不安全行为。本研究提出残余强化学习作为安全控制隐式安全过滤机制。我们不再依赖单一名义策略同时平衡性能、安全和稳健性，而是将性能与安全解耦。名义策略仅关注任务表现，残余策略学习安全修正。这种解耦带来了更好的性能-安全帕累托权衡，避免了在单一策略训练中对多个竞争奖励词进行精心调优的需求。我们证明残余策略可作为隐性安全过滤器。

Keyword: diffusion policy

Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning

Attention-DP3：通过几何对齐注意力条件的空间感知对象3D扩散策略

• Authors: Changbo Yan, Zhongbo Zhang, Zaibin Zhang, Yifan Wang, Lijun Wang, Huchuan Lu
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.13318
• Pdf link: https://arxiv.org/pdf/2609.13318
• Abstract 3D point-cloud observations are inherently ambiguous in complex, cluttered manipulation scenes, where target objects may be partially occluded or tightly intermingled with visually similar distractors. As a result, standard 3D diffusion policies often struggle to localize and exploit task-relevant geometry as scene complexity grows. We propose \textbf{Attention-DP3}, a spatially object-aware 3D diffusion policy that injects object-level geometric cues via attention while keeping the DP3 diffusion backbone unchanged. Our pipeline performs open-vocabulary 2D segmentation on RGB images, then lifts predicted target masks into 3D using calibrated camera geometry to obtain object-centric geometric priors. We incorporate these cues through Tri-field Attentional Conditioning, which constructs three complementary fields: (i) a targetness field to anchor the target object, (ii) an intra-target saliency field to emphasize task-relevant geometry within the target, and (iii) a backgroundness field to suppress distractors and clutter. Experiments on Adroit, DexArt, MetaWorld, and the real-world SO101 platform show consistent improvements over DP3, achieving state-of-the-art performance across benchmarks. Notably, as distractor objects increase, DP3 drops sharply, whereas Attention-DP3 remains stable and outperforms DP3 by up to 31\% under heavy clutter. The code is publicly available at this https URL.
• 中文摘要 在复杂、杂乱的操作场景中，3D点云观测本质上是模糊的，目标对象可能部分遮挡或与视觉相似的干扰物紧密混合。因此，随着场景复杂度的增加，标准3D扩散策略常常难以定位和利用任务相关几何。我们提出了\textbf{Attention-DP3}，一种空间感知对象的3D扩散策略，通过注意注入对象级几何线索，同时保持DP3扩散骨干不变。我们的流程对RGB图像进行开放词汇2D分割，然后利用校准相机几何将预测目标遮罩提升到3D，获得以对象为中心的几何先验。我们通过三场注意力条件结合这些线索，构建三个互补场：（i）目标场用于锚定目标对象，（ii）目标内显著场以强调目标内任务相关几何体，（iii）背景场用于抑制干扰和杂乱。在Adroit、DexArt、MetaWorld和现实SO101平台上的实验显示，DP3在基准测试中持续提升，达到最先进的性能。值得注意的是，随着干扰对象的增加，DP3急剧下降，而Attention-DP3保持稳定，在重杂乱环境下表现高达DP333%。该代码公开于此 https 网址。

LieSpline-DP: Lie-Group B-Spline Diffusion Policy for Smooth Robot Manipulation

LieSpline-DP：用于平滑机器人操作的李群B样条扩散策略

• Authors: Erxuan Xie, Bang Liu, Pingyun Nie, Xingkai Liu, Zhuang Fu, Bo Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.15162
• Pdf link: https://arxiv.org/pdf/2609.15162
• Abstract Diffusion Policy (DP) is a powerful Learning from Demonstration (LfD) method for robotic manipulation, yet it suffers from discontinuous and non-smooth trajectories. Spline-based action representations promote smooth motion within individual action chunks, but existing spline-based methods neither guarantee cross-chunk $C^2$ continuity nor account for the group structure of $\mathrm{SE}(3)$. We therefore propose LieSpline-DP, a Lie-group B-spline diffusion policy that generates end-effector trajectories directly on $\mathrm{SE}(3)$ and couples consecutive plans by sharing their boundary control poses, ensuring $C^2$ continuity throughout the entire planned trajectory. Across three real-robot tasks, LieSpline-DP produces lower trajectory jerk and higher task success rates than the DP baseline. The gains are particularly pronounced in real-world tasks involving liquids and flexible objects: in our real-robot experiments, LieSpline-DP achieved a 100% success rate on both pouring and bucket hooking, whereas the DP baseline achieved only 10% and 30%, respectively.
• 中文摘要 扩散策略（DP）是一种强大的“从演示学习”（LfD）机器人操作方法，但存在不连续和非平滑轨迹的问题。基于样条的作用表示促进单个动作块内的平滑运动，但现有基于样条的方法既不保证跨块连续性$C^2$，也无法考虑 $\mathrm{SE}（3）$ 的群结构。因此，我们提出了 LieSpline-DP，一种李群 B 样条扩散策略，直接在 $\mathrm{SE}（3）$ 上生成末端执行器轨迹，并通过共享边界控制姿态来耦合连续计划，确保整个计划轨迹中$C^2$的连续性。在三个真实机器人任务中，LieSpline-DP 产生了比 DP 基线更低的轨迹抖动和更高的任务成功率。在涉及液体和柔性物体的实际任务中，这种提升尤为明显：在我们的真实机器人实验中，LieSpline-DP在倒水和挂桶成功率均达到100%，而DP基线分别仅为10%和30%。

Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands

Bench2Dex：双手灵活操作的Visuo-触觉基调

• Authors: Zhenjie Yang, Yideng Zhang, Dongjie Zhang, Chenyu Jiang, Xianshuai Liu, Yufeng Li, Zuhao Ge, Xingyu Jiao, Zheng Zhang, Kaiyu He, He Wang, Yuwen Zhong, Yi Deng, Muyun Jiang, Xianliang Huang, Haisheng Su, Donghang Zhang, Jian Zhang, Xue Yang, Hongyang Li, Zuxuan Wu, Yu-Gang Jiang, Xiaosong Jia, Junchi Yan
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.15726
• Pdf link: https://arxiv.org/pdf/2609.15726
• Abstract Tactile sensing provides contact information that can be difficult to infer from vision alone, but tactile hardware for dexterous hands has not converged to a common design. Dexterous hands differ in finger structure, contact surfaces, and sensor layouts, while simulated tactile signals still differ from measurements produced by physical sensors. These factors make it difficult to study visuo-tactile manipulation across diverse dexterous hands within a consistent experimental setting. We present Bench2Dex, a simulation benchmark for visuo-tactile bimanual manipulation across 12 dexterous hands. We adapt existing robot models with a shared simulated tactile interface that converts local contact geometry into image-like tactile observations. The interface provides a consistent observation format across different hand morphologies without attempting to reproduce the output of a specific physical tactile sensor. Bench2Dex includes 26 bimanual manipulation tasks that involve tool use, articulated-object interaction, and multi-stage manipulation, together with about 1.3K human-teleoperated demonstrations. The benchmark provides synchronized visual, tactile, proprioceptive, action, and object-state observations, together with executable task metrics. For robustness, we group seven perturbation types into invariance axis, where the correct action does not change, and equivariance axis, where the correct action changes together with the perturbation. We evaluate ACT, Diffusion Policy, pi0.5, and GR00T N1.5 on Bench2Dex and report their performance and failure modes. Bench2Dex is meant as a platform for studying visuo-tactile learning across dexterous hands. It does not assume that simulated tactile observations can replace real tactile sensing; it offers a shared setting for algorithm development while tactile hardware and simulation models are still evolving.
• 中文摘要 触觉感知提供的接触信息难以仅凭视觉推断，但针对灵巧手的触觉硬件尚未统一设计。灵巧手在指结构、接触面和传感器布局上有所不同，而模拟触觉信号仍与物理传感器的测量值存在差异。这些因素使得在一致的实验环境中研究不同灵巧手的视觉-触觉操作变得困难。我们提出了Bench2Dex，一个针对12双灵巧手的视觉-触觉双手操作的模拟基准。我们采用共享模拟触觉接口，将局部接触几何转换为类似图像的触觉观察。该界面在不同手型中提供了一致的观察格式，无需尝试复制特定物理触觉传感器的输出。Bench2Dex 包含 26 个双手操作任务，涉及工具使用、关节与物体交互和多阶段操作，并包含约 1300 个人工远程操作演示。基准测试提供同步的视觉、触觉、本体感觉、动作和物体状态观察，以及可执行的任务指标。为了稳健性，我们将七种扰动类型分为不变轴（正确动作不变）和等变轴（正确作用随扰动变化）。我们在 Bench2Dex 上评估 ACT、扩散政策、pi0.5 和 GR00T N1.5，并报告它们的性能和失效模式。Bench2Dex 旨在作为一个跨灵活手部研究视觉-触觉学习的平台。它不假设模拟触觉观察能取代真实的触觉感知;它为算法开发提供了共享环境，同时触觉硬件和仿真模型仍在发展中。