生成时间: 2026-07-13 19:12:23 (UTC+8); Arxiv 发布时间: 2026-07-13 20:00 EDT (2026-07-14 08:00 UTC+8)

今天共有 14 篇相关文章

Keyword: reinforcement learning

FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space

FlowDAgger：潜在空间生成式机器人策略的人工循环适应

• Authors: Michael Murray, Daphne Chen, Simran Bagaria, Dean Fortier, Tess Hellebrekers, Galen Mullins, Harshavardhan Gajarla, Oier Mees, Maya Cakmak, Andrey Kolobov
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.08877
• Pdf link: https://arxiv.org/pdf/2607.08877
• Abstract Pretrained generative robot policies based on flow matching and diffusion have achieved impressive results across a wide range of manipulation tasks. Yet real-world deployments routinely expose failure modes outside the pretraining distribution. Closing these gaps typically requires large-scale data collection or online reinforcement learning on physical hardware, which is impractical for rapid and safe adaptation. We present FlowDAgger, a sample- and compute-efficient method for adapting frozen generative robot policies from human interventions in latent space. Our key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy, using reverse-time integration followed by local refinement. The resulting inverted noise provides supervision for a lightweight latent policy that steers the base model at deployment time, enabling rapid skill acquisition while preserving its behavioral priors. We evaluate FlowDAgger in simulation and on real-world bimanual and single-arm manipulation, adapting both action-head VLAs and world-action models from a handful of interventions. FlowDAgger outperforms supervised fine-tuning and latent-space RL baselines and preserves pretrained skills on held-out tasks, offering a practical path for adapting robot foundation models in the real world. Website: this https URL
• 中文摘要 基于流量匹配和扩散的预训练生成机器人策略在多种操作任务中取得了显著效果。然而，现实中的部署经常暴露出预训练分布之外的故障模式。弥补这些差距通常需要大规模数据收集或在物理硬件上的在线强化学习，这对于快速且安全的适应来说并不切实际。我们介绍FlowDAgger，这是一种采样和计算效率高的方法，用于适应潜空间中人类干预的冻结生成机器人策略。我们的核心理念是动作反转：每个人类专家操作都被映射到冻结基策略下可能产生的噪声，采用反向时间积分和局部细化。由此产生的反向噪声为轻量级潜在策略提供了监督，在部署时引导基础模型，实现快速技能习得，同时保持其行为先验。我们在模拟以及现实世界的双手和单臂操作中评估了FlowDAgger，并从少数干预中调整了动作头VLA和世界动作模型。FlowDAgger优于监督微调和潜空间强化学习基线，保留了预先训练的技能，为现实世界中机器人基础模型的应用提供了实用路径。网站：这个 https URL

Ortho2CAD: 3D CAD generation from orthographic drawings using vision language models

Ortho2CAD：利用视觉语言模型从正交图生成3D CAD

• Authors: Aditya Joglekar, Amit Regmi, Kenji Shimada, Levent Burak Kara
• Subjects: Subjects: Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2607.08891
• Pdf link: https://arxiv.org/pdf/2607.08891
• Abstract Engineering design intent is often communicated through rasterized orthographic drawings. However, downstream workflows inherently require editable and parametrically defined 3D computer-aided design (CAD) models. To bridge this gap, we introduce Ortho2CAD, a vision-language model (VLM) specifically designed to translate rasterized orthographic drawings directly into editable CadQuery code, which can then be seamlessly converted into 3D CAD models. To train the model effectively, we utilize supervised fine-tuning (SFT) for instances where explicit CadQuery code labels already exist, and we apply geometry-grounded reinforcement learning (RL) to optimize the model in scenarios where ground-truth labels are absent. To enable learning at scale, we create a pythonOCC-based drawing generator that renders first-angle orthographic projections from STEP models, complete with dashed hidden lines and key dimensions. On existing datasets encompassing settings both with and without CadQuery supervision, we generate orthographic drawings and show that our model produces 100% syntactically valid code. Moreover, it achieves a 3D CAD intersection-over-union (IoU) accuracy that surpasses all baselines, with an average relative improvement of over 7% compared directly against the next best performing model. We show that leveraging VLMs with SFT and RL techniques can effectively pave the way forward for orthographic drawing to 3D CAD reconstruction. Our implementation is available at this https URL.
• 中文摘要 工程设计意图通常通过栅格化正交图纸传达。然而，下游工作流程本质上需要可编辑且参数化定义的3D计算机辅助设计（CAD）模型。为弥合这一差距，我们引入了Ortho2CAD，一种视觉语言模型（VLM），专门设计用于将光栅化正字图直接转换为可编辑的CadQuery代码，然后可无缝转换为3D CAD模型。为了有效训练模型，我们在已有显式CadQuery代码标签的情况下使用监督微调（SFT），并在缺乏地面真实标签的情况下应用几何基础强化学习（RL）优化模型。为了实现大规模学习，我们创建了一个基于pythonOCC的绘图生成器，能够从STEP模型中渲染第一角度正交投影，并配有虚线隐藏线和关键维度。在涵盖有无CadQuery监督的现有数据集上，我们生成正字法图，并证明模型生成的代码语法上100%有效。此外，它实现了超越所有基线的3D CAD交叉交叉合并（IoU）精度，与下一个表现最好的模型相比，平均相对提升超过7%。我们展示了利用VLM结合SFT和RL技术，可以有效地为正交绘图向3D CAD重建铺平道路。我们的实现可在此 https 网址获取。

Learning More from Less: Reinforcement Learning from Hindsight

少中学：强化 事后诸葛亮

• Authors: Iris Xu, Sunshine Jiang, John Marangola, Nitish Dashora, Richard Li, Thomas Liu, Zexue He, Yuheng Zhi, Alex Pentland, Pulkit Agrawal, Zhang-Wei Hong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.09042
• Pdf link: https://arxiv.org/pdf/2607.09042
• Abstract Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior. Such a failure, however, is a success at a different task. We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training of VLAs by scoring failed rollouts against the tasks they actually achieved. A single vision-language model relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories. On out-of-distribution LIBERO-PRO tasks, where standard RL improves only slowly, LfH achieves $5\times$ improvement in sample efficiency, and outperforms a dense progress-reward baseline. The gains hold across VLA backbones and on a physical Franka robot.
• 中文摘要 强化学习（RL）越来越多地被用于视觉-语言-动作（VLA）模型的后期训练，但每次更新都会消耗机器人的部署，这些部署过程缓慢且成本高昂，因此样本效率成为核心关注点。操控任务通常只提供稀疏的奖励，因此薄弱的策略几乎在培训初期的每次推广都失败，且即使这些失败执行了连贯的行为，也几乎没有可学的。然而，这种失败在另一种任务上是成功的。我们介绍“事后诸葛亮”（LfH），通过对VLA的失败推广进行评分，将事后诸葛亮重新标记应用于强化学习。一个单一的视觉语言模型会重新标记指令和奖励，对一组失败的推广提出事后诸葛亮的指令，并评分每个部署的满足程度，政策则共同训练重新标记的和原始推广。由于VLA跨语言推广，在语言中重新标记可以让策略从相同的轨迹中学习更多。在非分布的LIBERO-PRO任务中，标准强化学习提升缓慢，LfH在样本效率上提升了5美元乘以$，并且优于密集的进展-奖励基线。这些增益不仅适用于VLA骨干，也适用于实体Franka机器人。

Toward Active Object Detection for UAVs in the Wild: A Large-Scale Dataset, Benchmark and Method

迈向野外无人机主动目标检测：大规模数据集、基准测试与方法

• Authors: Tianpeng Liu, Xinhua Jiang, Li Liu, Qinmu Shen, Siwei Tang, Zhen Liu, Yongxiang Liu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.09078
• Pdf link: https://arxiv.org/pdf/2607.09078
• Abstract Object detection is a fundamental component in numerous Unmanned Aerial Vehicle (UAV) applications, yet it has long been plagued by hindrances like occlusion or target pixel scarcity. Active Object Detection (AOD) provides a novel paradigm to address these challenges via active vision, while UAV-based AOD research remains scarce due to the lack of high-quality datasets and benchmarks for algorithm development and evaluation. To fill this gap, this paper presents ATRNet-LUDO, the first large-scale real-world dataset for UAV-Ground Active Object Detection (UGAOD). It contains 121,000 multi-view panoramic multi-target aerial images and 1.21 million local single-target slices, covering 10 vehicle targets across 40 scenarios. It enables the construction of diverse training and testing environments for UAV agent interaction and active observation policy learning. Based on this dataset, we establish a comprehensive evaluation benchmark for AOD policy learning methods. Most existing AOD policies rely on Deep Reinforcement Learning (DRL) but suffer from poor generalization. Evaluations on our benchmark reveal a significant generalization gap between training and testing performance, highlighting an urgent need for solutions. To this end, we leverage the Joint Embedding Predictive Architecture (JEPA) to construct a world model that enhances state representation learning, and propose AOD-JEPA by incorporating AOD-specific prior knowledge. Extensive experiments validate its effectiveness and superiority. We hope ATRNet-LUDO and the benchmark will advance research in the UGAOD field. The dataset and code are soon available at this https URL.
• 中文摘要 物体检测是众多无人机（UAV）应用的基本组成部分，但长期以来一直受到遮挡或目标像素稀缺等障碍困扰。主动物体检测（AOD）为通过主动视觉解决这些挑战提供了一种新范式，而基于无人机的AOD研究因缺乏高质量数据集和算法开发与评估基准而仍然稀缺。为填补这一空白，本文介绍了ATRNet-LUDO，这是首个大规模的无人机地面主动目标探测（UGAOD）真实数据集。它包含12.1万张多视角全景多目标航拍图像和121万个局部单目标切片，涵盖40种场景下的10个车辆目标。它能够构建多样化的培训和测试环境，用于无人机智能体交互和主动观察策略学习。基于该数据集，我们建立了AOD政策学习方法的综合评估基准。大多数现有的AOD政策依赖深度强化学习（DRL），但普遍适用性较差。我们基准测试的评估显示，培训与测试性能之间存在显著的泛化差距，凸显了对解决方案的紧迫需求。为此，我们利用联合嵌入预测架构（JEPA）构建了一个增强状态表示学习的世界模型，并通过整合AOD特有的先验知识提出AOD-JEPA方案。大量实验验证了其有效性和优越性。我们希望ATRNet-LUDO和基准能推动UGAOD领域的研究。数据集和代码很快会在这个 https URL 上发布。

Vascular Geometry Characterization for AI-Based Endovascular Navigation

基于人工智能的血管内导航的血管几何表征

• Authors: Han-Ru Wu, Harry Robertshaw, Lisa Dwyer-Joyce, Thomas C Booth, Alejandro Granados
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.09130
• Pdf link: https://arxiv.org/pdf/2607.09130
• Abstract Mechanical thrombectomy (MT) is a time-critical intervention for acute ischemic stroke; however, access remains limited due to a shortage of neuroradiologists and specialized centers. Reinforcement learning (RL) offers potential to automate endovascular navigation and improve accessibility, yet current models lack standardized frameworks to assess navigation difficulty for model training and evaluation. This study aims to identify vascular metrics associated with navigation difficulty and to develop an automated pipeline for quantitative vascular feature extraction, enabling future complexity grading. Vascular trees were segmented from computed tomography angiograms from 61 patients, and vascular metrics including aortic arch type, presence of bovine arch, vessel length, tortuosity, take-off angle, number of reverse curves, were measured using a custom pipeline. A Soft Actor-Critic RL algorithm was used for 120 s autonomous navigation. Outcomes were analyzed using both mixed effects linear and logistic regression. On the left side, the presence of a bovine arch and aortic arch type II/III increased navigation time by 30.19 s and 37.92 s, respectively, while greater tortuosity (\b{eta} = 118.20) further prolonged the procedure and reduced success probability. On the right side, type II/III arches extended procedure time by 45.94 s, while each additional reverse curve was associated with 3.96 s longer navigation time and lower probability of success. These findings demonstrate for the first time that MT agent navigation difficulty is strongly influenced by vascular geometry. The proposed automated pipeline enables objective and quantitative characterization of vascular features, providing a foundation for future development of standardized complexity grading and RL model evaluation, without aiming to demonstrate clinically generalizable autonomous navigation.
• 中文摘要 机械性血栓切除术（MT）是急性缺血性中风的一种时间关键干预;但由于神经放射科医生和专业中心的短缺，访问仍然有限。强化学习（RL）具有自动化血管内导航和提升可及性的潜力，但当前模型缺乏标准化框架来评估模型训练和评估导航难度。本研究旨在识别与导航难度相关的血管指标，并开发自动化的定量血管特征提取流程，从而实现未来的复杂度分级。通过61名患者的计算机断层扫描血管造影分段血管树，并使用定制管道测量了包括主动脉弓类型、牛弓是否存在、血管长度、扭转度、起飞角度、反向曲线数量等血管指标。采用软演员-批判者强化学习算法实现120秒自主导航。结局采用混合效应线性回归和逻辑回归分析。左侧牛弓和主动脉弓II/III型分别使导航时间增加30.19秒和37.92秒，而更大的扭转（\b{eta} = 118.20）进一步延长了手术时间，降低了成功概率。右侧的II/III型拱门使操作时间延长45.94秒，每增加一个反向曲线，导航时间延长3.96秒，成功概率降低。这些发现首次表明，MT代理导航难度强烈受血管几何结构影响。所提自动化流程能够客观且定量地表征血管特征，为未来标准化复杂度分级和强化学习模型评估奠定基础，同时不以展示临床通用的自主导航为目标。

Bidirectional Resource Scheduling for Disaggregated and Asynchronous RL Post-Training

双向资源调度，用于分散和异步强化学习后培训

• Authors: Tan Zhiqiang, Wang Maoxin, Wang Sijie, Yin Yiming, Wang Qiang, Chu Xiaowen, Shi Shaohuai
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2607.09207
• Pdf link: https://arxiv.org/pdf/2607.09207
• Abstract It is well established that the reasoning capabilities of large language models (LLMs) can be improved by applying reinforcement learning (RL) in a post-training stage. In a standard RL iteration, the current model (the policy) generates experience through rollouts, and the resulting data is then used to update the policy during training. High-performance RL frameworks such as StreamRL and AReaL employ a disaggregated architecture and asynchronous rollouts to better exploit both rollout and training resources, thereby increasing overall system throughput. Nonetheless, across varying RL setups (e.g., hardware configurations, model scales, staleness levels, and hyperparameters) and under changing workloads, it remains common for both rollout and training resources to experience idle periods. In this paper, we present BiDiRL, a hybrid time-space multiplexing architecture for asynchronous, disaggregated RL designed to reduce resource idleness. First, we develop a hot-switch runtime that enables rapid switching between rollout and training resources with negligible overhead. Second, we propose a static, scheduling-aware planner based on time-performance modeling that chooses a hot-switch-friendly resource partition, so that rollout and training durations are roughly balanced at a coarse level. Third, at execution time, we introduce a bidirectional scheduler that further exploits runtime bubbles through fine-grained resource switching, allowing the bottleneck stage to temporarily borrow idle resources from the other pool. Across a wide range of workloads, datasets, and models on two 32-GPU testbeds, BiDiRL increases RL training throughput by up to 1.94x compared with RL systems including veRL, AReaL, and ROLL, without affecting convergence behavior.
• 中文摘要 已经充分证明，大型语言模型（LLMs）的推理能力可以通过在训练后阶段应用强化学习（RL）来提升。在标准强化学习迭代中，当前模型（策略）通过部署产生经验，所得数据随后在训练期间用于更新策略。高性能强化学习框架如StreamRL和AReaL采用分解架构和异步展开，以更好地利用推广和训练资源，从而提高整体系统吞吐量。然而，在不同的强化学习设置（如硬件配置、模型尺度、陈旧程度和超参数）以及工作负载变化下，部署和训练资源都常常会出现空闲期。本文介绍了BiDiRL，一种用于异步、拆分强化学习的混合时空复用架构，旨在减少资源闲置。首先，我们开发了热开关运行时，实现了在部署和培训资源之间快速切换，开销极低。其次，我们提出一种基于时间绩效建模的静态、计划性规划器，选择一个适合热开关的资源划分，使推广和训练时长在粗略层面上大致平衡。第三，在执行时，我们引入一个双向调度器，通过细粒度资源切换进一步利用运行时气泡，允许瓶颈阶段暂时借用对方池的闲置资源。在两个32-GPU测试平台的广泛工作负载、数据集和模型中，BiDiRL相比包括veRL、AReaL和ROLL在内的强化学习系统，将强化学习的训练吞吐量提升了多达1.94倍，且不影响收敛行为。

An Improved Deep Reinforcement Learning Control Strategy for Traction Dual Rectifiers in EMUs

电动多单元牵引双整流器的改进深度强化学习控制策略

• Authors: Zhigang Liu, Mingwei Tang, Xiangyu Meng, Hui Wang, Qiao Zhang, Haoyu Wang, Mengru Li
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.09276
• Pdf link: https://arxiv.org/pdf/2607.09276
• Abstract Due to the use of PI-based d q current decoupling in the pulse rectifier of CRH5 high-speed trains, the PI parameters directly affect the traction system's control performance. Linearized control may have issues with reference trajectory changes or model mismatches, leading to a decrease in system performance, while nonlinear control may have problems with jitter and poor steady-state accuracy. This paper proposes a new control strategy that replaces all PI in the d q current decoupling control with a single intelligent agent. This method based on Deep Reinforcement Learning (DRL) can avoid various drawbacks of linearization and nonlinear control and ensure the stability of intermediate DC voltage. However, when EMUs are in different working conditions and switching, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm used in traction dual rectifiers does not have a good control effect. Focusing on the issue, Reward Shaping (RS) is added to re-design a nonlinear reward function, which can be combined with Prioritized Experience Replay (PER) to increase the convergence speed of the episode reward. The simulation results show that the improved control strategy can be effectively applied to EMUs working in multiple conditions. Finally, the stability analysis is carried out using Lyapunov's second method and the verification results of the hardware-in-the-loop (HIL) simulation platform show that the DRL control has a good effect.
• 中文摘要 由于CRH5高速列车脉冲整流器采用基于PI的d q电流解耦技术，PI参数直接影响牵引系统的控制性能。线性控制可能存在参考轨迹变化或模型不匹配的问题，导致系统性能下降，而非线性控制则可能存在抖动和稳态精度差的问题。本文提出了一种新的控制策略，将dq当前解耦控制中的所有PI替换为单一智能体。基于深度强化学习（DRL）的方法可以避免线性化和非线性控制的各种缺点，并确保中间直流电压的稳定性。然而，当电动车组处于不同的工作条件和切换状态时，牵引双整流器中使用的双延迟深度确定性策略梯度（TD3）算法控制效果不佳。针对这一问题，新增了奖励塑造（RS）以重新设计非线性奖励函数，并可与优先体验回放（PER）结合，提高剧集奖励的收敛速度。模拟结果表明，改进后的控制策略可以有效应用于多种条件下运行的电动车组。最后，稳定性分析采用李雅普诺夫的第二种方法，硬件在环（HIL）模拟平台的验证结果显示DRL控制效果良好。

Multi-Agent Reinforcement Learning for SLA-Aware Network Slicing in UAV-Enabled MEC

用于无人机支持MEC中SLA感知网络切片的多智能体强化学习

• Authors: Mohammad Farhoudi, Zeinab Sasan, Masoud Shokrnezhad, Tarik Taleb
• Subjects: Subjects: Networking and Internet Architecture (cs.NI)
• Arxiv link: https://arxiv.org/abs/2607.09295
• Pdf link: https://arxiv.org/pdf/2607.09295
• Abstract Unmanned Aerial Vehicle (UAV)-enabled Mobile Edge Computing (MEC) offers flexible capacity provisioning for heterogeneous network slices, including Hyper-Reliable and Low-Latency Communication (HRLLC), Enhanced Mobile Broadband (eMBB), and Massive Machine-Type Communications (mMTC). However, guaranteeing slice-level Service-Level Agreements (SLAs) under dynamic user mobility, stochastic task arrivals, and constrained onboard energy and computing resources remains a fundamental challenge. This paper proposes a predictive multi-agent Reinforcement Learning (RL) framework that proactively maintains SLA stability in UAV-enabled MEC through coordinated trajectory control and computation resource allocation. A lightweight prediction module forecasts near-future user mobility, enabling UAVs to anticipate congestion and reposition before SLA violations occur. We design an SLA-aware reward function that explicitly penalizes both violation probability and duration across slices, alongside total energy consumption. UAV agents are trained using Multi-Agent Proximal Policy Optimization (MAPPO) with centralized training and decentralized execution, enabling scalable online decision-making. Event-driven simulations with realistic mobility traces demonstrate that the proposed framework significantly improves SLA stability compared with baselines while maintaining competitive energy efficiency and delay performance, approaching oracle-level performance with sufficiently accurate predictive information.
• 中文摘要 无人机（UAV）支持的移动边缘计算（MEC）为异构网络片提供了灵活的容量配置，包括超可靠低延迟通信（HRLLC）、增强型移动宽带（eMBB）和大规模机器型通信（mMTC）。然而，在动态用户移动性、随机任务到达以及受限的机载能源和计算资源下，保障片级服务级协议（SLA）仍是一个根本性挑战。本文提出了一种预测性多智能体强化学习（RL）框架，通过协调轨迹控制和计算资源分配，主动维护无人机驱动MEC中的SLA稳定性。一个轻量级预测模块预测用户的近期移动性，使无人机能够在SLA违规发生前预判拥堵和重新定位。我们设计了一个SLA感知的奖励函数，明确惩罚切片间的违规概率和持续时间，同时惩罚总能耗。无人机智能体采用多智能体近端策略优化（MAPPO）训练，采用集中训练和去中心化执行，实现可扩展的在线决策。具有真实移动轨迹的事件驱动模拟表明，所提框架相较基线显著提升了SLA稳定性，同时保持了竞争性的能源效率和延迟性能，接近oracle级性能，且预测信息足够准确。

Shortcut Trajectory Planning for Efficient Offline Reinforcement Learning

高效离线强化学习的捷径轨迹规划

• Authors: Guanquan Wang, Yoshimasa Tsuruoka
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.09336
• Pdf link: https://arxiv.org/pdf/2607.09336
• Abstract Diffusion-based trajectory planners have shown strong performance in offline reinforcement learning, but their iterative denoising process often incurs high inference cost. Consistency-based planners reduce the number of sampling steps, yet they typically rely on a two-stage teacher--student distillation pipeline that increases training cost and may introduce instability. We propose Shortcut Trajectory Planning (STP), an offline model-based reinforcement learning framework that incorporates shortcut models as efficient trajectory generators. STP trains a conditional shortcut trajectory model in a single stage, supports adjustable one-step and few-step inference through step-size conditioning, and selects candidate plans using a critic augmented with feasibility-aware correction. Across standard D4RL benchmarks, including locomotion, navigation, manipulation, and dexterous control tasks, STP achieves strong performance while simplifying the training pipeline for fast generative planning.
• 中文摘要 基于扩散的轨迹规划器在离线强化学习中表现出优异表现，但其迭代去噪过程往往会产生较高的推理成本。基于一致性的规划器减少了抽样步骤的数量，但通常依赖于师生两阶段的蒸馏流程，这增加了培训成本，并可能带来不稳定性。我们提出了捷径轨迹规划（STP），这是一种基于离线的模型强化学习框架，将捷径模型作为高效的轨迹生成器。STP在单阶段训练条件捷径轨迹模型，支持通过步长条件进行可调的一步和少步推断，并利用带有可行性意识修正的批评者选择候选方案。在标准D4RL基准测试中，包括移动、导航、操作和灵巧控制任务，STP在简化训练流程以实现快速生成规划的同时，实现了强劲的性能。

Mach-Mind-4-Flash Technical Report

Mach-Mind-4-Flash技术报告

• Authors: Foundation Model Team (Li Auto Inc)
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.09375
• Pdf link: https://arxiv.org/pdf/2607.09375
• Abstract We present Mach-Mind-4-Flash, a 35B-parameter Mixture-of-Experts (MoE) agentic model with 3B activated parameters. Through post-training optimization alone without scaling pre-training compute, the model achieves performance on par with or surpassing that of 100B-parameter-class models. By introducing scalable agentic interaction environments for large-scale reinforcement learning, the model attains significant performance gains on real-world application tasks. Our pipeline comprises three stages: (1) a unified RL/OPD training infrastructure with dynamic multi-teacher scheduling and operator-level acceleration, delivering 17\% end-to-end training speedup; (2) multiple domain-specific RL experts trained in parallel across Reasoning, General, and Agent tracks, then fused into a single generalist via Multi-Teacher On-Policy Distillation (MOPD) -- a routed reverse-KL objective that eliminates the see-saw degradation of mixed-reward RL; (3) Hybrid Median-length Policy Optimization (HMPO), a single-stage token-efficiency method that compresses reasoning chains by 19--46\% with $\le$0.7 percentage-point accuracy loss. Mach-Mind-4-Flash scores 92.70 on AIME'26, 82.82 on IFBench, 80.74 on Behavioral-SafetyBench, 75.80 on BFCL-v4, 72.31 on BrowseComp-zh, and 84.20 on ClawBench -- leading or matching models with 10--30$\times$ its activated size at a fraction of the inference cost.
• 中文摘要 我们介绍Mach-Mind-4-Flash，一个35B参数的专家混合（MoE）代理模型，具有3B激活参数。仅通过训练后优化而不进行预训练计算缩放，该模型的性能可与100B参数类模型相当甚至超过。通过引入可扩展的代理交互环境进行大规模强化学习，该模型在实际应用任务中实现了显著的性能提升。我们的培训流程包括三个阶段：（1）统一的RL/OPD培训基础设施，支持动态多教师排班和操作员级加速，实现端到端培训加速17%;（2）多个领域专属强化学习专家在推理、通用和代理三方向并行培训，然后通过多教师策略提炼（MOPD）融合为单一通才——这是一种路由式反向基石目标，消除了混合奖励强化学习的摇摆劣化;（3）混合中位数长度策略优化（HMPO），一种单阶段令牌效率方法，将推理链压缩19-46%，准确率损失0.7%。Mach-Mind-4-Flash在AIME'26上得分92.70，IFBench中82.82，Behavioral-SafetyBench中80.74分，BFCL-v4中75.80分，BrowseComp-zh上72.31分，ClawBench上84.20分——这些领先或匹配的模型，激活尺寸比其激活时大小大概10到30美元/时间美元，且推理成本的一小部分。

Action-Factored Multi-Agent Reinforcement Learning for Scalable Quantum Device Tuning

动作因子多智能体强化学习用于可扩展量子器件调谐

• Authors: Edwin De Nicolo, Rahul Marchand, Cornelius Carlsson, Pranav Vaidhyanathan, Natalia Ares
• Subjects: Subjects: Machine Learning (cs.LG); Mesoscale and Nanoscale Physics (cond-mat.mes-hall)
• Arxiv link: https://arxiv.org/abs/2607.09422
• Pdf link: https://arxiv.org/pdf/2607.09422
• Abstract Cooperative multi-agent reinforcement learning is well suited to problems with large parameter spaces and exploitable local structure, such as the tuning of electrostatically-defined quantum-dot arrays. However, if parameter cross-talk is strong, a non-stationary environment from the perspective of any individual agent can destabilize learning - the same effect that plagues manual tuning of such systems. We propose using a factored representation of the action space, learned online, to decouple agents and minimize their interference. Our framework, QADAPT, uses this factorization to efficiently learn shared policies based on local measurements and rewards. With this modular strategy, we achieve zero-shot generalization to unseen quantum device sizes and maintain an approximately constant number of convergence steps to reach target regimes. This work provides a scalable route toward the rapid calibration of large-scale quantum processors.
• 中文摘要 协作多智能体强化学习非常适合处理参数空间大且局部结构可利用的问题，例如静电定义的量子点阵列调优。然而，如果参数串扰很强，从任何单个代理的角度来看，非固定环境都可能破坏学习——这与这类系统手动调优时的效果相同。我们提出利用在线学习的因式分解表示来解耦代理并最小化干扰。我们的框架QADAPT利用这种分解，基于局部测量和奖励高效学习共享策略。通过这种模块化策略，我们实现了零射向未知量子器件尺寸的推广，并保持近似恒定的收敛步数以达到目标区域。这项工作为大规模量子处理器的快速校准提供了可扩展的路径。

Multimodal Reward Hacking in Reinforcement Learning

强化学习中的多模奖励黑客

• Authors: Jiayu Yao, Yiwei Wang, Anmeng Zhang, Zhe Sun, Songsong Wang, Lingrui Mei, Yuyao Ge, Shenghua Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.09492
• Pdf link: https://arxiv.org/pdf/2607.09492
• Abstract Reinforcement learning (RL) is increasingly used to align multimodal large language models (MLLMs), but higher rewards do not always imply better task performance. This risk is amplified when visual evidence is evaluated by text-only or weakly grounded rewards. We study reward hacking in MLLM RL across safety VQA, chart VQA, and stress-test settings, varying reward design, data ambiguity, model scale (2B-32B), and RL algorithm (GRPO, RLOO, DAPO). We introduce Newly Rewarded Failure Rate (NRFR), which measures failures among samples whose proxy reward improves over the SFT baseline. Outcome-only rewards cause severe hacking, reaching 48.1% Reward Hacking Rate (RHR), while NRFR exceeding RHR shows that RL creates new failures rather than merely inheriting them. Scaling reduces but does not eliminate hacking: even the 32B model retains a 54.9% worse rate under outcome-only rewards, whereas answer-aware rewards improve the oracle trend at every scale. Robustness is also algorithm- and scale-dependent: GRPO is consistently most resistant, RLOO remains vulnerable, and DAPO improves substantially from 2B to 8B. Visual-evidence rewards help only with reliable verification: keyword-based checks increase hacking, while VLM-as-judge semantic verification reduces it. Overall, multimodal reward hacking is a systematic result of optimizing imperfect rewards, and robust alignment requires rewards and verifiers that remain reliable under optimization pressure.
• 中文摘要 强化学习（RL）越来越多地用于对齐多模态大型语言模型（MLLM），但更高的奖励并不总是意味着任务表现更好。当视觉证据被仅用文本或薄弱的奖励来评估时，这种风险会被放大。我们研究了MLLM强化学习中在安全VQA、图表VQA和压力测试环境中的奖励黑客，奖励设计变化、数据模糊性、模型尺度（2B-32B）以及强化学习算法（GRPO、RLOO、DAPO）。我们引入了新奖励失败率（NRFR），衡量代理奖励优于SFT基线的样本中的失败情况。仅结果奖励会导致严重的黑客攻击，达到48.1%的奖励黑客率（RHR），而NRFR超过RHR则显示RL创造了新的失败，而不仅仅是继承失败。扩展虽然减少但并未消除黑客行为：即使是32B模型，在仅结果奖励下，攻击率仍低54.9%，而答案感知奖励则改善了所有尺度下的预言机趋势。鲁棒性还依赖于算法和规模：GRPO始终最具抵抗力，RLOO依然脆弱，DAPO从2B大幅提升到8B。视觉证据奖励仅有助于可靠的验证：基于关键词的检查会增加黑客行为，而以VLM为评判的语义验证则能减少黑客行为。总体而言，多模态奖励黑客是优化不完美奖励的系统性结果，稳健的对齐需要在优化压力下保持可靠的奖励和验证器。

CORAL-AUV: CFD Oriented Reinforcement Learning for Autonomous Underwater Vehicles

珊瑚-AUV：自动水下载具的CFD导向强化学习

• Authors: Steven Roche, Milo Van Mooy, Nathan McGuire, Levi Cai, Jonathan P. How, Yogesh Girdhar
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.09557
• Pdf link: https://arxiv.org/pdf/2607.09557
• Abstract Fine grain control and positioning of autonomous underwater vehicles (AUVs) is critical for sampling, maintenance, and survey applications. Traditional control methods for AUVs are labor intensive and are not robust to changes in the vehicle configuration or environmental conditions. Reinforcement learning (RL) promises rapid controller development while handling a range of deployment parameters via domain randomization (DR). However, DR is still limited by the capacity of the underlying simulation to model real physics. In particular, drag physics are difficult to model and are a large contributor to sim-to-real gaps. Meanwhile, computational fluid dynamics (CFD) provides high fidelity drag models but is challenging to leverage within reinforcement learning frameworks due to its computational overhead. Thus, in this paper we exploit the idea of training surrogate approximations of CFD models of a given vehicle, enabling fast inference within RL pipelines. We are the first to successfully deploy a zero-shot RL policy on a 6-DOF AUV in which policy training is performed on surrogate drag models (SDMs) trained on CFD data. We find 31% lower energy usage compared to a controller using simplified physics while traversing between waypoints 11% faster with 19% less error. Our SDM based RL controller better predicts zero-shot transfer and is more robust across reward shaping design choices. When using DR to complete a task with perturbed parameters, we find that the CFD policy is the only controller that successfully transfers. The policies are evaluated in a controlled tank environment and in the field providing extensive testing of the policies' capabilities.
• 中文摘要 自主水下载具（AUV）的细粒控制和定位对于采样、维护和调查应用至关重要。传统的AUV控制方法劳动密集，且对飞行器结构或环境条件的变化不够稳健。强化学习（RL）承诺在通过域随机化（DR）处理一系列部署参数的同时，快速开发控制器。然而，DR仍受限于底层仿真对真实物理建模的能力。特别是，阻力物理难以建模，是模拟到实际差距的重要原因。与此同时，计算流体力学（CFD）提供了高精度阻力模型，但由于计算开销高，难以在强化学习框架中加以利用。因此，本文利用训练特定车辆CFD模型的替代近似，实现在强化学习流水线内的快速推断。我们是首个成功在6深度AUV上部署零发射强化辐射策略的机构，该策略训练基于CFD数据训练的替代阻力模型（SDM）。我们发现，使用简化物理的控制器能耗低31%，在航点间穿越速度快11%，误差减少19%。我们基于SDM的强化学习控制器更能预测零射击转移，并且在奖励塑造设计选择中更具稳定性。当使用DR完成参数扰动的任务时，我们发现CFD策略是唯一成功传输的控制器。这些政策在受控坦克环境中和现场评估，并对其能力进行了广泛测试。

Semantic Pareto-DQN: A Multi-Objective Reinforcement Learning Framework for Financial Anomaly Detection

语义帕累托-DQN：用于金融异常检测的多目标强化学习框架

• Authors: Cláudio Lúcio do Val Lopes, Lucca Machado da Silva
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.09641
• Pdf link: https://arxiv.org/pdf/2607.09641
• Abstract Financial anomaly detection suffers from extreme class imbalance, causing traditional single-objective algorithms to exhibit ``fraud collapse'', defaulting to the majority class and failing to balance anomaly interdiction with customer friction. To overcome this without distortive data resampling, we propose the Semantic Pareto-DQN, a multi-objective reinforcement learning framework. Our approach synthesizes heterogeneous transaction features into cohesive natural-language narratives, encoded by large language models, thereby producing a robust, scale-invariant state representation. The agent optimizes a vectorial reward that explicitly decouples financial efficacy, operational friction, and semantic discovery. By mapping the continuous Pareto frontier, the system dynamically navigates the asymmetric costs of missed anomalies versus false positives. Empirical evaluations across E-Commerce fraud and UCI Credit datasets show that semantic Pareto-DQN successfully shatters the zero-recall trap. It achieves superior minority-class recall compared to scalarized baselines, providing an alternative to trade bounded operational friction for financial anomaly discovery.
• 中文摘要 金融异常检测存在极端的阶级不平衡，导致传统的单目标算法出现“欺诈崩溃”，默认为多数类，未能平衡异常拦截与客户摩擦。为了克服这些问题，避免扭曲数据重采样，我们提出了语义帕累托-DQN，一种多目标强化学习框架。我们的方法将异构交易特征综合成连贯的自然语言叙事，由大型语言模型编码，从而产生稳健且尺度不变的状态表示。该智能体优化了一种向量奖励，明确将财务效能、操作摩擦和语义发现解耦。通过绘制连续的帕累托边界，系统动态地导航了漏除异常与假阳性的非对称成本。对电子商务欺诈和UCI信用数据集的实证评估表明，语义帕累托DQN成功打破了零召回陷阱。相比标量化基线，它实现了更优越的少数族裔召回率，为金融异常发现提供了贸易限制操作摩擦的替代方案。

Keyword: diffusion policy

There is no result