生成时间: 2026-08-17 16:43:58 (UTC+8); Arxiv 发布时间: 2026-08-17 20:00 EDT (2026-08-18 08:00 UTC+8)

今天共有 20 篇相关文章

Keyword: reinforcement learning

Reward Machines for Signal Temporal Logic

信号时间逻辑的奖励机

• Authors: Alper Kamil Bozkurt, Shangtong Zhang, Yuichi Motai
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.13625
• Pdf link: https://arxiv.org/pdf/2608.13625
• Abstract Signal temporal logic (STL) provides a formal language for specifying real-time properties of real-valued observations, along with a quantitative robustness score for monitoring satisfaction. Control synthesis from STL specifications is of interest since manual controller design becomes infeasible as real-world systems grow in complexity. Moreover, many modern autonomous and AI-enabled systems lack accurate and complete system models, which makes optimization-based synthesis approaches unsuitable and motivates learning-based control. Prior work uses STL robustness scores as rewards in reinforcement learning (RL) to obtain control policies satisfying given specifications; however, robustness depends on execution history, leading to intractable state space expansion for general long-horizon specifications with arbitrarily nested temporal operators. This work introduces a novel automata-based approach that provides an efficient memory mechanism and associated Markovian rewards suitable for RL frameworks. Our approach constructs a timed alternating automaton from the given STL specifications, augments the state space with automaton locations and clock valuations, and derives rewards from the automaton acceptance condition. We empirically demonstrate that our approach learns policies that achieve higher robustness scores and satisfaction rates than those learned by existing approaches using robustness-based rewards.
• 中文摘要 信号时序逻辑（STL）提供了一种形式语言，用于指定实值观测值的实时属性，并配有用于监测满意度的定量鲁棒性评分。根据STL规范进行控制综合是值得关注的，因为随着实际系统复杂度的增加，手动控制器设计变得不可行。此外，许多现代自主和人工智能驱动的系统缺乏准确和完整的系统模型，这使得基于优化的综合方法不适用，反而促使基于学习的控制。以往工作在强化学习（RL）中将STL鲁棒性分数作为奖励，以获得满足特定规格的控制策略;然而，鲁棒性依赖于执行历史，导致具有任意嵌套时间算子的一般长视野规范的状态空间展开变得难以解决。本研究引入了一种基于自动机的新方法，提供了高效的记忆机制及适用于强化学习框架的相关马尔可夫奖励。我们的方法根据给定的STL规范构建一个定时交替自动机，通过自动机位置和时钟估值补充状态空间，并从自动机接受条件中获得奖励。我们实证证明，我们的方法能够学习比现有基于稳健性奖励的方法获得更高的稳健性评分和满意度的策略。

PROVE: Training-Free Prompt Recovery using Verifiable Evidence

PROVE：利用可验证证据实现无需培训的即时恢复

• Authors: Rupayan Mallick, Mahsa Khoshnoodi, Sarah Adel Bargal
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.13671
• Pdf link: https://arxiv.org/pdf/2608.13671
• Abstract Modern text-to-image models can generate highly realistic images from natural-language prompts, while recent advances in prompt inversion have made it increasingly feasible to recover those prompts from generated outputs, raising new concerns for copyright protection and content ownership. As prompt marketplaces emerge, recovered prompts can enable both the unauthorized reproduction and redistribution of copyrighted creative works, and the exposure of the prompts that encode an artist's creative recipe in AI-generated content. Existing prompt inversion methods rely on gradient-based optimization, autoregressive captioning, or reinforcement learning. However, optimization-based methods often produce unreadable prompts, captioning methods hallucinate unverified details, and RL-based approaches frequently overfit to specific generators while introducing evaluation circularity. We introduce PROVE (Prompt Recovery with Verified Evidence), a training-free, black-box prompt inversion attack that reconstructs prompts by composing verifiable scene descriptions rather than optimizing token sequences, targeting both original copyrighted works and AI-generated content. The resulting prompts are fully auditable, with every recovered claim grounded in explicit image evidence, and are formalized through a precision-constrained recall maximization objective. Across MS-COCO, Flickr30K, and Lexica, using state-of-the-art text-to-image generators, PROVE consistently outperforms optimization, captioning, and RL-based baselines on image similarity (DINO, LPIPS) and text-image alignment (CLIP), without any training, generator access, or fine-tuning, demonstrating a stronger and more practical prompt inversion attack.
• 中文摘要 现代文本转图像模型可以从自然语言提示生成高度逼真的图像，而提示反转技术的最新进展使得从生成输出中恢复这些提示变得越来越可行，这也引发了版权保护和内容所有权的新担忧。随着提示市场的兴起，恢复的提示既可能实现未经授权复制和再分发版权创作作品，也能让编码艺术家创意配方的提示在AI生成内容中曝光。现有的提示反演方法依赖于基于梯度的优化、自回归字幕或强化学习。然而，基于优化的方法常常产生无法阅读的提示，字幕方法会幻觉未经验证的细节，基于强化学习的方法则常常对特定生成器过拟合，同时引入评估循环性。我们介绍PROVE（带验证证据的提示恢复），这是一种无需训练的黑箱提示反转攻击，通过构建可验证的场景描述来重建提示，而非优化代币序列，目标既是原创版权作品，也是针对AI生成的内容。生成的提示完全可审计，每个恢复的主张都基于显式图像证据，并通过精确约束的回忆最大化目标形式化。在MS-COCO、Flickr30K和Lexica等平台上，使用最先进的文本到图像生成器，PROVE在图像相似度（DINO、LPIPS）和文本-图像对齐（CLIP）方面，始终优于优化、字幕和基于强化学习的基线，无需任何训练、生成器访问或微调，展示了更强大、更实用的提示反转攻击。

GRPO Beyond English: A Large-Scale Study of GRPO in Non-English and Multilingual Settings

GRPO 超越英语：非英语及多语环境中GRPO的大规模研究

• Authors: Konstantin Dobler, Federico Scozzafava, Jonathan Janke, Mohamed Ali, Simon Lehnerer
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.13698
• Pdf link: https://arxiv.org/pdf/2608.13698
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR), often optimized with Group Relative Policy Optimization (GRPO), has become a central recipe for improving the reasoning capabilities of pretrained language models but current studies remain heavily English-centric. We conduct a large-scale empirical study of multilingual and non-English GRPO across a wide range of base models, training languages, and different reasoning language rewards. We find that training to reason in the native language often leaves only a small gap to training for English reasoning. We further observe strong crosslingual transfer: training in one language often improves performance in many others. However, specific trends are highly model- and language-dependent. In some cases, training in a particular language induces severe regressions on out-of-domain capabilities in other languages. Our analysis shows that RLVR beyond English can provide broad crosslingual gains, but also requires broad evaluation to detect language-specific regressions.
• 中文摘要 带有可验证奖励的强化学习（RLVR），通常通过群体相对策略优化（GRPO）进行优化，已成为提升预训练语言模型推理能力的核心方法，但当前的研究仍高度以英语为中心。我们对多语言和非英语GRPO进行了大规模实证研究，涵盖广泛的基础模型、训练语言及不同的推理语言奖励。我们发现，母语推理训练往往只留下了一小部分英语推理培训的空白。我们还观察到强烈的跨语言迁移：一种语言的训练常常能提升多种语言的表现。然而，具体趋势高度依赖模型和语言。在某些情况下，特定语言的训练会导致其他语言域外能力的严重倒退。我们的分析显示，超越英语的RLVR可以带来广泛的跨语言提升，但也需要广泛评估以检测语言特异性回归。

AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution

AdsWorldEngine：通过编排器和工具共进化实现自我演进的对话式广告代理

• Authors: Simiao Zuo, Chenhui Xu, Yimeng Jia, Qiang Lou, Jian Jiao, Denis Charles
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.13833
• Pdf link: https://arxiv.org/pdf/2608.13833
• Abstract Conversational advertising aims to deliver useful ads within multi-turn assistant interactions. Unlike conventional query-based advertising, where the user's intent is often expressed in a short standalone query, conversational ads must infer latent commercial intent from the current user query, the assistant response, and dialogue history while also deciding whether an ad would be helpful rather than intrusive. We propose AdsWorldEngine, an agentic framework for conversational advertising. AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown, an Orchestrator to generate commercial intents, call advertising tools, and construct a top-3 ad slate, and an Evaluator to score delivered ads for offline optimization. The central contribution is an iterative actor-tool training procedure: we first train the Orchestrator with supervised fine-tuning and agentic reinforcement learning, then use high- and low-reward rollouts to construct preference data to train tools. This creates a self-improving loop in which the system learns not only how to use advertising tools, but also how to improve them from rewarded behavior. To support subjective production decisions, we introduce label grounded judgment modeling, which trains judgment models from human labels collected under explicit guidelines. It enriches labels with thinking traces, filters inconsistent rationales through reflection, and further optimizes binary judgments with a cost sensitive GRPO variant that preserves asymmetric reward gaps. Offline, AdsWorldEngine improves diversity by 60% and relevance by 80% over the current production ad delivery system. In an online A/B test, it increases RPM by 22% and ads coverage by 74%.
• 中文摘要 对话式广告旨在通过多轮次助手互动传递有用的广告。与传统基于查询的广告不同，传统广告通常通过简短的独立查询表达用户意图，对话式广告必须从当前用户查询、助理回复和对话历史中推断潜在的商业意图，同时还要决定广告是有帮助还是具有侵入性。我们提出了AdsWorldEngine，一个用于对话式广告的代理框架。AdsWorldEngine使用机会门来决定是否应该展示广告，通过编排器生成商业意图，调用广告工具并构建前三名广告表，以及评估器为已投放广告进行评分以实现离线优化。核心贡献是一项迭代的演员-工具训练过程：我们首先通过监督微调和代理强化学习训练编排器，然后利用高奖励和低奖励的展开构建偏好数据以训练工具。这形成了一个自我改进的循环，系统不仅学会如何使用广告工具，还通过奖励行为来改进广告工具。为支持主观生产决策，我们引入了基于标签的判断建模，通过根据明确指南收集的人类标签训练判断模型。它通过思维痕迹丰富标签，通过反思过滤不一致的理由，并通过成本敏感的GRPO变体进一步优化二元判断，保持不对称的奖励差距。线下，AdsWorldEngine 比现有制作广告投放系统提升了 60% 的多样性和 80% 的相关性。在在线A/B测试中，转速提升了22%，广告覆盖率提升了74%。

Bootstrapping Niche Multilingual Code Translation via Reinforcement Learning with Execution-Based Verifiable Supervision

通过基于执行的可验证监督强化学习，自助化细分多语言代码翻译

• Authors: Kouki Yuki, Jie Zeng, Kyoko Ogawa, Ryunosuke Ikeda, Yohei Kobashi, Takeshi Kojima, Ikuya Yamada, Yusuke Iwasawa, Yutaka Matsuo
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.13854
• Pdf link: https://arxiv.org/pdf/2608.13854
• Abstract Code translation must preserve executable behavior across many programming languages, yet neural code translation has largely focused on a few popular languages such as C++, Java, and Python. This leaves a niche, many-to-many setting where parallel supervision is sparse, producing plausible but non-executable translations. We address this setting with preference-based reinforcement learning driven by execution-based supervision. Our pipeline firstly expands verifiable seed Python programs into a multilingual pool of execution-validated codes. Using the pool, a base LLM generates translation candidates across language pairs, which we label by their execution outcomes. The resulting preferences are used to train a reward model that scores cross-language translation quality. Finally, we optimize our base LLMs with GRPO over 600 directed language pairs (25 x 24) using the reward model as a signal. To evaluate the niche translation capability, we introduce HumanEval-X++, an execution-based benchmark that extends HumanEval-X to a broad many-to-many language space. We evaluate our approach using Qwen-3.5 4B and 9B models. On HumanEval-X++ and existing benchmarks, it yields consistent gains over the untrained baselines. In particular, the 4B model achieves an average improvement of 13% across all languages on HumanEval-X++, with a gain of 21% on mid-tier languages. Our study establishes a reliable approach of data generation, training, and benchmarking, paving the way toward further bootstrapping the quality of many-to-many translation for programming languages.
• 中文摘要 代码转换必须在多种编程语言中保持可执行行为，但神经代码翻译主要集中在C++、Java和Python等少数流行语言上。这导致了一个多对多的利基环境，平行监督稀少，产生合理但不可执行的翻译。我们通过基于偏好的强化学习，基于执行的监督来应对这一环境。我们的流水线首先将可验证的种子 Python 程序扩展为多语言的执行验证代码池。利用该池，基础LLM生成跨语言对的翻译候选，我们通过其执行结果来标记。所得的偏好被用来训练一个奖励模型，从而获得跨语言翻译质量的评分。最后，我们用GRPO优化了基于600对定向语言（25 x 24）的基础LLM，并以奖励模型为信号。为了评估其细分翻译能力，我们引入了 HumanEval-X++，这是一个基于执行的基准测试，将 HumanEval-X 扩展到广泛的多对多语言空间。我们使用Qwen-3.5、4B和9B模型来评估我们的方法。在HumanEval-X++及现有基准测试中，它相较于未训练的基线表现稳定提升。特别是，4B模型在HumanEval-X++上所有语言的平均提升为13%，中等语言提升21%。我们的研究建立了可靠的数据生成、训练和基准测试方法，为进一步提升多对多翻译质量铺平了道路。

Knowledge-Data-Dual-Driven Reinforcement Learning for Autonomous Vehicle Control in Mixed Traffic

知识-数据-双驱动强化学习用于混合交通中的自动驾驶车辆控制

• Authors: Jie Fang, Wei Zheng, Mengyun Xu, Eui-Jin Kim
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.13878
• Pdf link: https://arxiv.org/pdf/2608.13878
• Abstract In mixed traffic, decision-making for autonomous vehicles (AVs) confronts three interrelated challenges. First, physics-based priors incorporated into reinforcement learning (RL) models fail to capture latent interactive vehicle intentions and diverse driver behaviors, limiting the proactive reasoning capabilities. Second, abrupt maneuvers by surrounding vehicles cause non-stationarity, leaving long-tail safety events under-explored. Third, hybrid action spaces destabilize unified RL training due to the different temporal scales of continuous car-following and discrete lane-changing maneuvers. To address these issues, we propose Knowledge-Data Dual-driven Reinforcement Learning (KDDRL). First, a conditional deep generative model synthesizes intention-aware future trajectories, converting passive perception into proactive predictive states. Second, a knowledge-data dual-driven paradigm operates on these predictive states, fusing probabilistic data-driven insights with physical constraints to guide safe exploration through safety-critical scenarios. Third, a coupling module compresses both intention-aware trajectories and physical constraints into compact shared embeddings. This unified representation enables asynchronous multi-timescale optimization of continuous car-following and discrete lane-changing while preserving mutual information. Evaluations on dataset-calibrated simulations demonstrate that KDDRL effectively handles intention uncertainty, accelerates training convergence, and outperforms conventional baseline methods in terms of safety, efficiency, and comfort.
• 中文摘要 在混合交通领域，自动驾驶汽车（AV）的决策面临三个相互关联的挑战。首先，基于物理的先验被纳入强化学习（RL）模型，无法捕捉潜在的交互式车辆意图和多样化的驾驶员行为，限制了主动推理能力。其次，周围车辆的突然机动导致车辆不稳定，导致长尾安全事件缺乏充分探索。第三，混合动作空间由于连续跟车和离散变道动作的不同时间尺度，会破坏统一的强化学习训练。为解决这些问题，我们提出了知识-数据双向强化学习（KDDRL）。首先，条件式深度生成模型综合了意图感知的未来轨迹，将被动感知转化为主动预测状态。其次，知识-数据双驱动范式基于这些预测状态，融合概率数据驱动洞见与物理约束，引导安全探索通过安全关键场景。第三，耦合模块将意图感知轨迹和物理约束压缩为紧凑的共享嵌入。这种统一表示实现了异步多时间尺度的连续跟车和离散变道优化，同时保持互信息。数据集校准模拟的评估表明，KDDRL有效处理意图不确定性，加速训练趋同，并在安全性、效率和舒适度方面优于传统基线方法。

XAI-Guided Conservative Decentralized Execution for Offline Multi-Agent Network Slicing

XAI引导的保守式去中心化执行，用于离线多智能体网络切片

• Authors: Eslam Eldeeb, Hatim Chergui, Merouane Debbah
• Subjects: Subjects: Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.13982
• Pdf link: https://arxiv.org/pdf/2608.13982
• Abstract The recent advances toward sixth-generation (6G) and beyond-6G networks have accelerated the need for intelligent resource management mechanisms capable of supporting heterogeneous services under shared infrastructures in network slicing. However, resource allocation in network slicing naturally forms a resource-coupled cooperative optimization problem with competing slice demands. Slices compete for limited resources to minimize individual latencies while coordinating to avoid conflicts and underutilization. Although multi-agent reinforcement learning (MARL) has shown promising performance in such settings, existing online formulations remain costly, unsafe, and difficult to deploy due to their reliance on environmental interactions and communication among agents. In this work, we present explainable artificial intelligence (XAI)-guided conservative decentralized execution (X-CODE). X-CODE is an explainable offline MARL that operates offline without environmental interaction, nor inter-agent communication. It exploits explainability-aware reward shaping to modify the relative preference among joint offline transitions during centralized training to improve decentralized resource-allocation behavior. In deployment, the agents operate independently without signaling exchange among the agents. Simulation results demonstrate that the proposed approach achieves zero observed resource-conflict events in the evaluated test episodes while minimizing per-slice latencies. Moreover, the proposed framework exhibits lower signaling overhead and reduces effective inference latency by 88 % under the considered communication-delay model compared to the online baselines. Source codes and datasets are available through: this https URL.
• 中文摘要 近年来，第六代（6G）及以后6G网络的进展加速了对智能资源管理机制的需求，这些机制能够支持在共享基础设施下的异构服务，实现网络切片。然而，网络切片中的资源分配自然形成了一个资源耦合的合作优化问题，且切片需求相互竞争。切片在协调避免冲突和利用不足的同时，争夺有限的资源以最小化单个延迟。尽管多智能体强化学习（MARL）在此类环境中表现出良好表现，但现有的在线表述因依赖环境交互和代理间通信而成本高昂、不安全且难以部署。本研究提出了可解释人工智能（XAI）引导的保守式去中心化执行（X-CODE）。X-CODE 是一种可解释的离线 MARL，运行时无需环境互动，也无需代理间通信。它利用可解释性感知奖励塑造，在集中训练中调整联合离线过渡的相对偏好，以改善分散资源分配行为。在部署中，代理独立运作，不进行代理间的交换信令。模拟结果表明，所提方法在评估的测试过程中实现了零观测到的资源冲突事件，同时最大限度地减少了每个切片延迟。此外，所提框架在考虑的通信延迟模型下，信令开销更低，有效推理延迟比在线基线降低了88%。源代码和数据集可通过以下 https URL 获取。

MMDynOpt-Agent: Dynamic Optimization for Multimodal Large Language Model Reasoning via Reinforcement Learning

MMDynOpt-Agent：通过强化学习实现多模态大型语言模型推理的动态优化

• Authors: Wenjin Liu, Haoran Luo, Fayuan Ke, Zhenghong Lin, Yue Lu, Zhe Cui, Anh Tuan Luu, Carl Yang
• Subjects: Subjects: Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2608.14026
• Pdf link: https://arxiv.org/pdf/2608.14026
• Abstract Recently, multimodal large language models (MLLMs) have demonstrated strong potential in visual understanding and complex reasoning tasks. However, existing methods often struggle to efficiently transform visual cues from multimodal inputs and the semantics of the question into effective reasoning conditions, thereby limiting the reasoning performance of multimodal large language models. To address this challenge, we propose MMDynOpt-Agent, which models the dynamic optimization of multimodal reasoning as a Markov decision process via end-to-end reinforcement learning. Specifically, a lightweight multimodal agent serves as the decision policy and interacts with the target MLLM as the environment, adaptively steering its reasoning through multi-turn dynamic optimization prompts. Furthermore, to reduce the cost of multimodal reasoning, a reward mechanism that combines format compliance, answer correctness, and budget awareness is designed to jointly ensure reasoning accuracy and efficiency. MMDynOpt-Agent is transferable and generalizable, enabling training with one target MLLM and inference-time transfer to others. Experimental results on fifteen public datasets show MMDynOpt-Agent achieves strong performance and outperforms baselines. Our project is available at this https URL.
• 中文摘要 近年来，多模态大型语言模型（MLLMs）在视觉理解和复杂推理任务中展现出强大潜力。然而，现有方法常常难以高效地将多模态输入和问题语义的视觉线索转化为有效的推理条件，从而限制了多模态大型语言模型的推理性能。为应对这一挑战，我们提出了MMDynOpt-Agent，它通过端到端强化学习，将多模推理的动态优化建模为马尔可夫决策过程。具体来说，轻量级多模态智能体作为决策策略，并以环境作用于目标MLLM，通过多回合动态优化提示自适应地引导其推理。此外，为了降低多模态推理的成本，设计了一种结合格式合规、答案正确性和预算意识的奖励机制，共同确保推理的准确性和效率。MMDynOpt-Agent 可转移且可推广，支持单一目标 MLLM 训练及对其他目标的推理时间转移。对15个公开数据集的实验结果显示，MMDynOpt-Agent性能优于基线数据。我们的项目可在此 https 网址访问。

A Graph-Based Reinforcement Learning Framework for Structured Drift Diagnosis and Recovery in Autonomous LLM Agents

基于图的强化学习框架，用于自主大型语言模型代理的结构化漂移诊断与恢复

• Authors: Ismail El Hamraoui, Sagar Jose, Nicolas Bureau, Robert Plana
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.14109
• Pdf link: https://arxiv.org/pdf/2608.14109
• Abstract Autonomous LLM agents are increasingly deployed in complex real-world workflows, yet they remain vulnerable to runtime behavioral drift, a silent deviation from the original task that can lead to irreversible side effects on external systems. Existing approaches address drift at the prompt level but lack structured mechanisms for step-level detection, risk assessment, and recovery decision. Because the main task-executing agent is often a large and expensive model that cannot be re-trained on every deployment, this work targets a plug-and-play recovery module instead. It introduces a graph-based framework in which a single small language model is trained via reinforcement learning to specialize at each node of a recovery graph, external to the main agent. Each node has a precise role\,: drift classification, operation detection, risk evaluation, or final decision and the model learns to produce structured XML-formatted reasoning adapted to that role. Training combines rule-based structural rewards with an LLM-as-judge semantic-quality signal, so that the model is graded both on how it answers (schema and length) and on what it says. Experiments on the public AppWorld benchmark show that the method generally exploits information about the suspected drift onset to issue correct recovery decisions using a small language model. In addition, the trained small language model reliably respects the prescribed output schema and produces semantically appropriate content in each field according to its assigned node role.
• 中文摘要 自主LLM代理越来越多地部署在复杂的现实世界工作流中，但它们仍然容易受到运行时行为漂移的影响，这是一种与原始任务的无声偏差，可能导致外部系统不可逆转的副作用。现有方法在即时层面处理漂移，但缺乏结构化的步骤级检测、风险评估和恢复决策机制。由于主要执行任务的代理通常是大型且昂贵的模型，无法每次部署重新训练，因此这项工作更倾向于即插即用的恢复模块。它引入了一个基于图的框架，通过强化学习训练单个小型语言模型，使其在恢复图的每个节点（主代理之外）进行专门化。每个节点都有精确的角色：漂移分类、操作检测、风险评估或最终决策，模型学习生成适应该角色的结构化XML格式推理。训练结合基于规则的结构性奖励与作为评判的LLM语义质量信号，使模型根据回答方式（模式和长度）以及所说内容进行评分。在公开的AppWorld基准测试上的实验表明，该方法通常利用怀疑的漂移起始信息，通过小型语言模型做出正确的恢复决策。此外，训练好的小型语言模型可靠地遵守规定的输出模式，并在每个字段根据其分配的节点角色生成语义上合适的内容。

Learning to Run Power Networks: Effective AlphaZero-inspired Topological Control

学习运行电力网络：受AlphaZero启发的有效拓扑控制

• Authors: Lukas Zetto, Benjamin Schäfer, Qiong Huang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.14114
• Pdf link: https://arxiv.org/pdf/2608.14114
• Abstract As the integration of volatile renewable energy sources increases the strain on modern power grids, the use of Reinforcement Learning (RL) for autonomous topological reconfiguration has emerged as a promising research field to keep strained grids stable and operational. Compared to traditional redispatching measures, topological actions offer a cheaper and more cost-effective way to manage grid congestion. However, their implementation is hindered by a vast combinatorial action space and strict operational constraints. This paper investigates the effectiveness of model-based AlphaZero-inspired approaches that utilize Monte Carlo Tree Search (MCTS) for proactive grid management. We systematically evaluate how reward functions, observation density, and search guidance influence an agent's survivability. Our results demonstrate that the optimized AlphaZero approach achieves a peak survivability of 98.43%, significantly outperforming the proximal policy optimization (PPO) variant. We find that conducting the MCTS without guidance from a prior learned policy or value function can enhance training efficiency, and that a straightforward binary survival reward provides more effective search guidance than complex, multi-objective functions. Our findings demonstrate that while AlphaZero is a powerful framework for topological control, pure reinforcement learning is not sufficient; rather, an effective and reliable system requires a 'minimalist' integration of domain-specific heuristics, binary rewards, and a restricted observation space of line loads.
• 中文摘要 随着易燃可再生能源的整合加剧了现代电网的压力，强化学习（RL）用于自主拓扑重构已成为保持受压电网稳定和运行的有前景研究领域。与传统的重调度措施相比，拓扑行动提供了一种更便宜且更具成本效益的电网拥堵管理方式。然而，其实现受到庞大的组合作用空间和严格的操作限制所阻碍。本文探讨了基于模型的AlphaZero启发方法，利用蒙特卡洛树搜索（MCTS）进行主动网格管理的有效性。我们系统地评估了奖励函数、观察密度和搜索指导如何影响代理的生存能力。我们的结果表明，优化后的AlphaZero方法实现了98.43%的峰值生存率，显著优于近端策略优化（PPO）变体。我们发现，在没有先前学习的策略或价值函数指导的情况下进行MCTS可以提升训练效率，而简单的二元生存奖励比复杂多目标函数提供更有效的搜索指导。我们的发现表明，虽然AlphaZero是一个强大的拓扑控制框架，但纯粹的强化学习是不够的;相反，一个有效且可靠的系统需要对领域特定启发式、二元奖励和有限的行加载观察空间进行“极简”整合。

Reinforcement Learning-Based Production Scheduling in an Industry-Based Coating Scenario Using the Digital Model Playground

基于强化学习的生产调度在基于行业的涂层场景中，利用数字模型游乐场

• Authors: Arne Kröger, Ralf Buschermöhle, Wilhelm Hasselbring, Henrik Wilbers
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.14122
• Pdf link: https://arxiv.org/pdf/2608.14122
• Abstract Production scheduling in complex manufacturing environments is challenging when sequence-dependent setup times, stochastic disturbances, and due-date constraints must be addressed simultaneously. While reinforcement learning (RL) methods have shown promising results in research, most studies rely on simplified benchmark processes, limiting their industrial relevance. This paper demonstrates the applicability of RL-based scheduling in an industry-inspired coating process that reflects practical complexities such as sequence-dependent setup times, machine breakdowns, and variable utilization. The open-source Digital Model Playground (DMPG), a discrete event simulation framework, is used to model the scenario and to train RL agents. Two standard algorithms, Deep Q-Networks and Proximal Policy Optimization, are benchmarked against conventional dispatching rules to illustrate feasibility and to provide a transparent testbed for further research. Results indicate that RL-based scheduling achieves balanced improvements across key performance indicators, with PPO delivering the most robust performance. The main contribution of this work is to bridge the gap between academic research and industrial practice by validating RL-based scheduling in a realistic, shareable scenario and by providing a reusable open-source framework for future studies.
• 中文摘要 在复杂的制造环境中，生产排程具有挑战性，因为必须同时应对顺序相关的设置时间、随机扰动和截止日期约束。虽然强化学习（RL）方法在研究中取得了有希望的成果，但大多数研究依赖简化的基准流程，限制了其在工业上的相关性。本文展示了基于强化语言调度在工业启发的涂层工艺中的适用性，该工艺反映了序列依赖的设置时间、机器故障和变量利用率等实际复杂性。开源的数字模型游乐场（DMPG）是一个离散事件仿真框架，用于建模场景并训练强化学习代理。两种标准算法——深度Q网络和近端策略优化——被以传统调度规则为基准，以展示可行性并为后续研究提供透明的测试平台。结果显示，基于强化学习的调度在关键绩效指标上实现了均衡的改进，其中PPO提供了最强有力的性能。本研究的主要贡献是通过验证基于强化学习的调度在现实且可共享的情境下，并提供可重用的开源框架供未来研究，弥合学术研究与工业实践之间的鸿沟。

AgilePE: Autonomous UAV Pursuit-Evasion via Self-Play Reinforcement Learning

AgilePE：通过自我游戏强化学习实现自主无人机追击-规避

• Authors: Wenhao Tang, Tianyang Chen, Zhejun Cui, Boyuan An, Jiayu Chen, Ruize Zhang, Huidong Liu, Tianyue Wu, Qingmin Liao, Fei Gao, Yu Wang, Chao Yu
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.14135
• Pdf link: https://arxiv.org/pdf/2608.14135
• Abstract Autonomous pursuit-evasion is a fundamental challenge for Unmanned Aerial Vehicles (UAVs), requiring rapid decision-making under tightly coupled dynamics and continuously changing opponent behaviors. Traditional rule-based or differential-game approaches often struggle with high-dimensional aerial interactions and agile maneuvering. We present AgilePE, a complete system for autonomous UAV pursuit-evasion via self-play reinforcement learning. AgilePE integrates agile low-level control, competitive policy optimization, and sim-to-real deployment in a unified framework. The policy directly maps onboard state observations to Collective Thrust and Body Rates (CTBR) commands, enabling end-to-end agile maneuvering without intermediate trajectory planners or waypoint controllers. For training, we use competitive self-play with Prioritized Fictitious Self-Play (PFSP) and a diversified opponent pool, enabling agents to improve against historical policies while stabilizing optimization and reducing policy oscillation. This process leads to the emergence of sophisticated pursuit and evasion strategies. For real-world deployment, we develop a hardware-aligned simulation pipeline that models actuator-response dynamics, communication latency, and domain randomization. The learned policies transfer zero-shot to real quadrotors without task-specific tuning. Real-world experiments reproduce pursuit-evasion tactics observed in simulation, including rapid dodging and flanking, and demonstrate interactive two-agent zero-shot deployment.
• 中文摘要 自主追踪-规避是无人机（UAV）面临的根本挑战，需要在紧密耦合的动态和不断变化的对手行为下快速决策。传统的基于规则或差异化游戏的方法常常在高维空中互动和灵活机动方面遇到困难。我们介绍AgilePE，一个通过自我游戏强化学习实现自主无人机追击规避的完整系统。AgilePE将敏捷底层控制、竞争策略优化和模拟到真实部署整合在统一框架中。该政策将机载状态观测直接映射到集体推力和体速（CTBR）指令，实现端到端的敏捷机动，无需中间轨迹规划器或航点控制器。训练方面，我们采用竞争性自玩，配合优先虚构自玩（PFSP）和多样化的对手池，使代理能够在历史策略中提升优势，同时稳定优化并减少策略波动。这一过程催生了复杂的追击和规避策略。在实际部署中，我们开发了一套硬件对齐的仿真流水线，用于建模执行器响应动力学、通信延迟和域随机化。所学政策可以将零射击转移到真实的四旋翼，无需针对特定任务进行调校。现实实验重现模拟中观察到的追击-规避战术，包括快速躲避和侧翼包抄，并演示互动式双智能体零射击部署。

Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine

消除时间音符冗余提升医学多模态强化学习

• Authors: Chenran Weng, Joo Seung Lee, Malini Mahendra, Anil Aswani
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.14157
• Pdf link: https://arxiv.org/pdf/2608.14157
• Abstract Mechanical ventilation is a critical life-support intervention, requiring dynamic adjustments to ventilator settings as a patient's condition evolves. While reinforcement learning (RL) offers a promising framework for optimizing these sequential decisions, standard approaches rely primarily on structured electronic health record (EHR) data, missing crucial clinical context recorded in free-text notes. Integrating longitudinal clinical notes into RL state spaces is challenging because notes are heavily inflated by temporal redundancy, such as copy-forward text, templating, and repetitive documentation, which dilutes time-local updates and degrades state representation quality. To address this, we propose a redundancy-aware multimodal state representation framework that explicitly removes duplicated note text over time before policy learning. We evaluate two computationally efficient temporal decomposition strategies for removing duplicated note text: (1) an embedding-space decomposition using singular value decomposition on local history subspaces, and (2) an interpretable sentence-level diff operation that filters out previously documented sentences before text encoding. Using real-world ICU data, we demonstrate that state representations constructed by stripping temporal note redundancy significantly outperform both structured-only and raw-note baselines across multiple off-policy evaluation methods (Model-Based Rollouts, Fitted Q-Evaluation, Weighted Importance Sampling, and Weighted Doubly Robust Evaluation). Our findings show that explicitly isolating new clinical information from repeated note text yields higher-quality state representations and directly improves RL performance for clinical decision support.
• 中文摘要 机械通气是关键的生命支持干预，需要随着患者病情的发展动态调整呼吸机设置。虽然强化学习（RL）为优化这些顺序决策提供了有前景的框架，但标准方法主要依赖结构化电子健康记录（EHR）数据，缺少自由文本笔记中记录的关键临床背景。将纵向临床笔记集成到强化学习状态空间中具有挑战性，因为记录因时间冗余而大幅膨胀，如转写文本、模板和重复文档，这些都会稀释时间局部更新并降低状态表示质量。为此，我们提出了一个冗余感知的多模态状态表示框架，在策略学习前明确去除重复的笔记文本。我们评估了两种计算效率高的时序分解策略用于去除重复的笔记文本：（1）在局部历史子空间上使用奇异值分解的嵌入空间分解，以及（2）一种可解释的句子层级差分操作，在文本编码前过滤掉之前已记录的句子。利用真实ICU数据，我们证明通过剥离时间笔记冗余构建的状态表示，在多种非策略评估方法（基于模型的推广、拟合Q评估、加权重要性抽样和加权双重稳健评估）中，显著优于结构化和原始笔记基线。我们的发现表明，明确从重复笔记文本中分离新临床信息，可以获得更高质量的状态表示，并直接提升临床决策支持的强化学习表现。

APTER: Adaptive Post-Training with Expert-Grounded Rubrics

APTER：专家基础评分标准的自适应后培训

• Authors: Xukai Wang, Liangqi Li, Zhiyue Xu, Jingang Zhou, Xiaoyu Shi, Jiansheng Cai, Bo Zhang, Zhe Li, Xu-Yao Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.14212
• Pdf link: https://arxiv.org/pdf/2608.14212
• Abstract As large language models enter professional domains, they must satisfy domain constraints, include critical evidence, and provide complete reasoning rather than merely produce fluent responses. Existing post-training methods often rely on holistic preferences or outcome-level verification, while recent rubric-based methods usually generate rubrics independently for each query. In specialized domains, such unconstrained rubrics may omit critical requirements and vary across samples, hindering the diagnosis and targeted repair of persistent capability deficiencies. We propose APTER (Adaptive Post-Training with Expert-Grounded Rubrics), a framework that integrates structured domain knowledge into fine-grained evaluation, optimization, and diagnosis for specialized complex reasoning. First, expert-grounded rubric construction starts from an expert criteria framework built by domain experts, where each criterion represents a stable professional capability. For each query, APTER selects relevant criteria and instantiates them into query-level rubrics linked to their source criteria, turning reusable expert criteria into executable query-level supervision without reference answers. Second, adaptive post-training uses rubric verdicts as both optimization and criterion-level diagnostic signals. Aggregating low-scoring verdicts by criterion ID reveals persistent deficiencies and triggers targeted supervised fine-tuning updates during reinforcement learning. Experiments on mathematical reasoning and medical question answering show consistent gains across both domains. Across three model generations, APTER improves the mathematics and medical averages over the corresponding base models by up to 15.86 and 8.04 points, respectively. Code and rubric datasets are available at this https URL.
• 中文摘要 当大型语言模型进入专业领域时，它们必须满足领域限制，包含关键证据，并提供完整的推理，而不仅仅是产生流利的回答。现有的训练后方法通常依赖整体偏好或结果层面的验证，而最新的基于评分标准的方法通常为每个查询独立生成评分标准。在专业领域中，这种不受约束的评分标准可能遗漏关键需求，且样本间差异，阻碍持续能力缺陷的诊断和有针对性修复。我们提出了APTER（专家基础评分标准的自适应后培训），这是一个将结构化领域知识整合为细致评估、优化和诊断的框架，用于专门的复杂推理。首先，专家基础的评分标准构建基于领域专家构建的专家标准框架，每个标准代表稳定的专业能力。对于每个查询，APTER 会选择相关标准并将其实例化为与源标准关联的查询级别评分标准，将可复用的专家标准转化为无需引用答案的可执行查询级监督。其次，自适应后训练将评分标准的判决作为优化和标准级诊断信号。按标准ID汇总低分判决，揭示了持续存在的缺陷，并在强化学习过程中触发有针对性的监督微调更新。数学推理和医学问答的实验显示，这两个领域均有持续的进步。在三代模型中，APTER分别将数学和医学平均值提升至15.86分和8.04分。代码和评分标准数据集可在此 https URL 获取。

MathForm: Scaling Mathematical Autoformalization with Knowledge Retrieval and Verification-Guided Refinement

MathForm：利用知识检索和验证引导的精炼来扩展数学自形式化

• Authors: Lushi Pu, Weiming Zhang, Xinheng Xie, Zixuan Fu, Bingxiang He, Hengyu Zhao, Hongya Lyu, Xin Li, Jie Zhou, Yudong Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.14221
• Pdf link: https://arxiv.org/pdf/2608.14221
• Abstract Autoformalization is commonly framed as translating natural-language mathematical statements into machine-verifiable formal languages such as Lean 4. However, faithful formalization requires more than translation. Models must map mathematical concepts to the complex hierarchy of types and definitions in formal libraries such as Mathlib, while ensuring that generated statements preserve the meaning of the source propositions. Existing approaches struggle because they rely heavily on the model's parametric memory for library-specific knowledge, while common data construction pipelines often resort to filtering single-pass outputs and lack mechanisms for feedback-driven revision. To address these challenges, we introduce MathForm, an autoformalization framework for constructing verified training data through Mathlib knowledge retrieval and verification-guided iterative refinement. Before generation, a retrieval planner gathers relevant definitions and existing formalizations from Mathlib to guide the formalization generator. Generated statements are then revised using compiler diagnostics and semantic-consistency feedback. Using this framework, we construct FormalVerse, a Lean 4 dataset containing approximately 367K verified examples across diverse mathematical domains and sources. We then train MathForm-8B through supervised fine-tuning followed by reinforcement learning. Across six benchmarks, MathForm-8B achieves average Pass@8 rates of 88.06% under Syntax Check (SC) and 72.37% under Consistency Check (CC), outperforming multiple specialized 32B autoformalizers. On the challenging FATE-H and FATE-X subsets, it attains CC pass rates of 63% and 37%, exceeding the strongest specialized baselines in both cases.
• 中文摘要 自形式化通常被定义为将自然语言数学陈述翻译成可机器验证的形式语言，如精益4。然而，忠实的形式化不仅仅需要翻译。模型必须将数学概念映射到形式库（如 Mathlib）中类型和定义的复杂层级，同时确保生成的语句保持源命题的意义。现有方法存在困难，因为它们高度依赖模型的参数内存来获取库特有知识，而常见的数据构建流程往往依赖过滤单遍输出，缺乏反馈驱动的修订机制。为应对这些挑战，我们引入了MathForm，一个通过Mathlib知识检索和验证引导迭代细化构建已验证训练数据的自形式化框架。在生成之前，检索规划器会收集 Mathlib 的相关定义和现有形式化，以指导形式化生成器。生成的语句随后通过编译器诊断和语义一致性反馈进行修订。基于该框架，我们构建了FormalVerse，这是一个精益4数据集，包含约36.7万个经过验证的样本，涵盖多元数学领域和来源。随后，我们通过监督微调和强化学习来训练MathForm-8B。在六个基准测试中，MathForm-8B 在语法检查（SC）下平均Pass@8率为88.06%，在一致性检查（CC）下达到72.37%，优于多个专门的32B自形式化器。在具有挑战性的FATE-H和FATE-X子集上，其CC通过率分别为63%和37%，均超过了最强的专业基准。

Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL

Envs-FORGE：Agent RL 的边疆优化奖励-接地环境综合

• Authors: Xiaojun Wu, Cehao Yang, Honghao Liu, Xueyuan Lin, Zhichao Shi, Hao Zhou, Xuhui Jiang, Chengjin Xu, Jia Li, Jian Guo
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.14312
• Pdf link: https://arxiv.org/pdf/2608.14312
• Abstract Reinforcement learning (RL) for terminal agents needs executable training environments with reliable rewards and useful difficulty. Fixed recipes such as few-shot, Self-Instruct, and Evol-Instruct apply the same prompting policy to every seed, even when the current policy would benefit from a harder, easier, or simply different task. We present Envs-FORGE, a prompting policy that converts verifier rewards into per-seed environment-synthesis actions. Envs-FORGE estimates seed pass rates, scores six projection--direction actions around a target learning frontier, and solves a per-seed mixed-integer linear program (MILP) to choose the action that conditions generation. The selected action drives synchronized rewriting of the instruction, fixtures, oracle solution, tests, and Docker environment; only gold-verified bundles enter RL training. The indexed MILP form also supports optional soft skill coverage for portfolio planning. On Qwen 3.5 35B, Envs-FORGE improves Pass@1 over Base by 9.2 percentage points on tb-core (40.0% to 49.2%) and 6.4 points on tb-2.0 (23.0% to 29.4%), exceeding the strongest fixed-recipe baseline by 2.4 and 2.1 points. It reaches 77.1% on SWE-bench Verified versus 73.4% for Base, and improves tb-core by 6.8--9.2 points across the evaluated 4B--35B models. All synthesis methods export 100 verified environments and use 2.27M--2.88M synthesis tokens, placing the comparison at the same downstream training-set size and the same operational scale. The source code is available at this https URL.
• 中文摘要 终端代理的强化学习（RL）需要可执行的训练环境，提供可靠奖励和有用的难度。固定配方如少量拍摄、自我指令和Evol-Ininstruction，对每个种子都适用相同的提示策略，即使当前策略更适合更难、更简单或仅仅不同的任务。我们介绍Envs-FORGE，一种提示策略，将验证者奖励转换为每个种子的环境综合动作。Envs-FORGE 估计种子传递率，对目标学习前沿的六个投影方向动作进行评分，并求解一个每种子混合整数线性规划（MILP）以选择决定生成的动作。所选动作驱动指令、夹具、oracle解决方案、测试和Docker环境的同步重写;只有经过黄金认证的捆绑包才能进入强化学习训练。指数化的MILP表格还支持可选的软技能覆盖，用于投资组合规划。在Qwen 3.5 35B中，Envs-FORGE在TB-core上Pass@1比Base提升9.2个百分点（40.0%至49.2%），在tb-2.0上提升6.4个百分点（23.0%至29.4%），比最强固定配方基线高出2.4个和2.1个百分点。在SWE-bench验证测试中达到77.1%，Base为73.4%，并在评估的4B-35B模型中提升了6.8-9.2个百分点。所有综合方法都导出100个经过验证的环境，并使用227万至288万个合成标记，使比较在相同的下游训练集规模和相同的操作尺度上进行比较。源代码可在该 https URL 访问。

CORAL: Curriculum-Optimized Reward Adaptation for LiDAR-Based Goal-Directed Urban Driving

CORAL：基于激光雷达的目标导向城市驾驶的课程优化奖励适应

• Authors: Anisa Saleem, Duksu Kim
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.14332
• Pdf link: https://arxiv.org/pdf/2608.14332
• Abstract Reinforcement learning is promising for autonomous urban driving, but long-horizon goal-directed navigation asks a policy to acquire several competing behaviors at once--reaching a distant goal, tracking a route, avoiding obstacles, obeying signals--and a fixed objective gives no order in which to learn them. This paper presents CORAL, which advances two schedules together: a five-stage curriculum that progressively lengthens routes and tightens behavioral constraints, and a stage-aware reward whose component weights shift emphasis from mission progress toward route following, safety, smoothness, and rule compliance as the task hardens. The policy is a multi-stream actor-critic network trained with Proximal Policy Optimization (PPO) in CARLA on a compact 99-dimensional state pairing a polar LiDAR histogram with vehicle telemetry, ego-frame route geometry, and traffic-rule indicators--no point-cloud encoder, no bird's-eye-view rasterization. Against two PPO baselines under an identical protocol, CORAL reaches the goal in all twenty evaluation episodes on the longest routes under the full set of behavioral constraints, where the baselines reach 5% and 10%; a factorial ablation shows that neither schedule alone matches their combination: removing either lowers both success and route completion, and disabling both drops success to 55%. Trained in one town, the policy transfers zero-shot to seven unseen towns, succeeding in 68-98% of episodes on routes of the same 100-150 m length, with mean lateral deviation below 0.35 m.
• 中文摘要 强化学习在城市自动驾驶方面前景看好，但长视野目标导向导航要求策略同时获得多种相互竞争的行为——达到远方目标、追踪路线、避开障碍物、服从信号——而固定目标没有给出学习它们的顺序。本文介绍了CORAL，它将两个计划共同推进：一个是五阶段课程，逐步延长路线并收紧行为约束;另一个是阶段意识奖励，其组成部分权重从任务进展转向路线跟踪、安全、顺畅和规则遵守，随着任务的加深。该策略是一个多流演员-批评者网络，通过CARLA中的近端策略优化（PPO）训练，基于紧凑的99维状态，将极化激光雷达直方图与车辆遥测、自称帧路线几何和交通规则指示器配对——无点云编码器，无鸟瞰图栅格化。在相同协议下，面对两个PPO基线，CORAR在所有20次评估中均达标，且在全套行为约束下，基线分别达到5%和10%;因子消融显示，单独两个计划都不匹配它们的组合：移除任一会降低成功率和路线完成率，禁用两者则成功率降至55%。该政策在一个城镇训练，将零射击转移到七个未被发现的城镇，在相同100-150米长路线上成功率为68-98%，平均横向偏差低于0.35米。

CoRun: Padding is Simple and Efficient for Deterministic LLM Inference

CoRun：填充简单高效，用于确定性大型语言模型推断

• Authors: Shiju Zhao, Jiacheng Yang, Qihang Chen, Junhao Hu, Jiaqi Zheng, Guihai Chen, Xusheng Chen
• Subjects: Subjects: Operating Systems (cs.OS); Performance (cs.PF)
• Arxiv link: https://arxiv.org/abs/2608.14376
• Pdf link: https://arxiv.org/pdf/2608.14376
• Abstract Despite fixed sampling parameters and random seeds, Large Language Model (LLM) inference exhibits output inconsistency, which undermines downstream tasks such as model evaluation and reinforcement learning. A major source of this nondeterminism is batch-dependent GPU execution: dynamic input shapes change kernel tiling and floating-point reduction orders. Existing systems address this problem with batch-invariant kernels, but these kernels restrict optimized tiling and split reductions, increasing more than 2$\times$ latency and reducing serving throughput by up to 74 %. This paper observes that although most kernels are not batch-invariant, they are position-invariant. Leveraging this property, we present CoRun, a scheduling-based system that achieves deterministic inference without requiring batch invariance. CoRun employs isolated prefill and fixed-shape batched decode to handle the two stages of LLM inference, respectively, leveraging CUDA graphs for efficient execution and simplified implementation. Experiments on LLMs with diverse architectures, including Qwen and DeepSeek, show that CoRun ensures determinism while improving throughput by 15-324 % over batch-invariant approaches, reducing time-to-first-token by 51.8 % and time-per-output-token by 48.6 % on average.
• 中文摘要 尽管采样参数固定且种子随机，大型语言模型（LLM）推断仍表现出输出不一致，削弱了模型评估和强化学习等后续任务。这种非确定性的主要来源是批次依赖的GPU执行：动态输入形状会改变核镶嵌和浮点归约顺序。现有系统通过批处理不变内核解决了这一问题，但这些内核限制了优化的铺砌和分割缩减，导致延迟增加超过2$/times$，并使服务吞吐量降低高达74%。本文指出，尽管大多数核不是批次不变的，但它们是位置不变的。利用这一特性，我们提出了CoRun，一种基于调度的系统，能够实现确定性推断而不要求批次不变性。CoRun 采用隔离预填充和固定形状批处理译码，分别处理 LLM 推理的两个阶段，利用 CUDA 图实现高效执行和简化实现。在多种架构的大型语言模型（如Qwen和DeepSeek）上的实验显示，CoRun在处理不变方法下确保确定性，同时将吞吐量提升15%-324%，平均将首次令牌时间缩短51.8%，每输出令牌时间缩短48.6%。

Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View

为扩散模型设计强化学习：统一路径-空间视图

• Authors: Yixian Xu, Yuanrui Zhang, Shengjie Luo, Liwei Wang, Di He
• Subjects: Subjects: Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2608.14430
• Pdf link: https://arxiv.org/pdf/2608.14430
• Abstract Reinforcement learning (RL) post-training provides a direct way to align diffusion models with human preferences and task-specific rewards. However, current RL algorithms for diffusion models remain fragmented: reverse-trajectory methods rely on discretized likelihood ratios, whereas forward-matching methods train on reward-labeled noising versions of the rollout samples. This paper shows that these seemingly different losses arise from a single path-space principle. Starting from the regularized diffusion-RL objective, we use importance sampling between sampling SDEs to obtain an explicit policy-gradient estimator on trajectory space. The estimator contains the stochastic Itô integral underlying Flow-GRPO-type updates; we derive an equivalent variance-reduced value-gradient form that recovers the forward-matching structure of AWM and DiffusionNFT. This identifies the empirical gap between these method families as a variance-reduction effect rather than a difference in RL principle. The derivation yields a unified design space organized by value-gradient estimation, weight functions, and sampling choices. Within this space, we propose a multi-sample KDE value-gradient estimator that reuses rollout groups, together with scale-bounded weight families that retain stable existing recipes while excluding singular ones. Experiments on SD3.5-M and Qwen-Image models validate the variance-reduction explanation and show that the resulting recipe improves over prior diffusion-RL baselines.
• 中文摘要 强化学习（RL）后训练提供了一种直接的方式，使扩散模型与人类偏好和任务特定奖励对齐。然而，当前扩散模型的强化学习算法仍然零散：逆轨迹方法依赖离散化似然比，而前向匹配方法则训练于带奖励标记的带噪音的展开样本。本文表明，这些看似不同的损失源于单一的路径空间原理。从正则化扩散-强化学习目标出发，我们利用采样SDE之间的重要性抽样，获得轨迹空间上的显式策略梯度估计。估计量包含了Flow-GRPO类型更新的随机伊藤积分;我们推导出一个等效的方差约简值梯度形式，恢复了AWM和DiffusionNFT的前向匹配结构。这表明这些方法家族之间的经验差距是方差减少效应，而非强化学习原理的差异。推导过程形成了一个统一的设计空间，通过值梯度估计、权重函数和采样选择组织。在此领域，我们提出了一个多样本KDE值梯度估计器，重用展开组，同时结合保留稳定现有配方而排除奇异配方的尺度有界权重族。在SD3.5-M和Qwen-Image模型上的实验验证了方差缩减的解释，并显示所得配方优于之前的扩散-强化学习基线。

Rollplex: Cross-Phase GPU Spatial Sharing for Vision Language Model Post-Training

Rollplex：视觉语言模型训练后的跨阶段GPU空间共享

• Authors: Hanfeng Lu, Tianyu Feng, Suyi Li, Yuheng Zhao, Wei Gao, Shaopan Xiong, Ju Huang, Siran Yang, Jiamang Wang, Lin Qu, Wei Wang
• Subjects: Subjects: Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2608.14498
• Pdf link: https://arxiv.org/pdf/2608.14498
• Abstract Vision-language models (VLMs) enable embodied agents to reason and act from visual observations and language instructions. Reinforcement learning (RL) post-training enhances these capabilities using task feedback, but current on-policy RL runtimes execute rollout, reference scoring, and actor training in strict serial phases. While effective for text-only RL, this phase-granular execution is wasteful for VLMs, where processing dense video inputs and prompt prefixes occupies a large fraction of each phase. Because prefix processing is independent of the generated response, it can be run alongside rollout decoding, which leaves GPU compute capacity underutilized, without breaking synchronous on-policy semantics. We present Rollplex, a runtime that decomposes the reference and training phase and moves the prefix computation into the rollout decode window. Realizing this schedule requires more than concurrent kernel launches: naive colocation of Qwen2.5-VL-32\,B requires roughly 165\,GiB per GPU, while rollout and training prefer different tensor-parallel (TP) degrees and weight layouts. Rollplex addresses these constraints with two mechanisms. Phase-aware memory management controls HBM residency according to producer--consumer lifetimes. Parallelism-aware weight sharing uses the same physical storage for layout-compatible tensors across distinct TP degrees and reconstructs only incompatible tensors, avoiding a complete second actor copy. On 32 H800 GPUs, Rollplex achieves $1.23\times$--$1.30\times$ speedup over serial colocation and $1.57\times$--$2.24\times$ over disaggregation under the same GPU budget, while preserving the synchronous RL update.
• 中文摘要 视觉语言模型（VLMs）使具身代理能够根据视觉观察和语言指令进行推理和行动。强化学习（RL）训练后通过任务反馈增强这些能力，但当前的策略驱动RL运行时在严格的串行阶段执行展开、引用评分和演员训练。虽然这种阶段粒度执行对纯文本强化学习有效，但对VLM来说是浪费的，因为处理密集的视频输入和提示前缀占据了每个阶段的大部分时间。由于前缀处理与生成响应无关，它可以与展开解码同时运行，这样GPU的计算能力就被充分利用，同时又不会破坏同步的策略语义。我们介绍 Rollplex，一个运行时，分解参考和训练阶段，并将前缀计算移入滚出译码窗口。实现这一计划不仅需要同时内核发布：Qwen2.5-VL-32\，B的朴素共址约需每GPU165，GiB，而部署和训练则偏好不同的张量并行（TP）度数和权重布局。Rollplex通过两种机制解决了这些限制。相位感知内存管理根据生产者——消费者生命周期控制HBM驻留。并行感知权重共享使用相同的物理存储空间，存储布局兼容的张量，跨越不同TP度，仅重建不兼容的张量，避免完整的第二演员复制。在32台H800 GPU上，Rollplex在相同GPU预算下实现了比串行共机处理加速1.23-$1.30\times$，比拆分加速$1.57\times$-$2.24\times$，同时保持同步强化学习更新。

Keyword: diffusion policy

There is no result