生成时间: 2026-08-13 17:14:13 (UTC+8); Arxiv 发布时间: 2026-08-13 20:00 EDT (2026-08-14 08:00 UTC+8)

今天共有 31 篇相关文章

Keyword: reinforcement learning

Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs

回头交易员-工作台：用自生成的多选题对算法交易的LLM代理进行基准测试

• Authors: Ruoxi Zhao, Maziar Raissi
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11232
• Pdf link: https://arxiv.org/pdf/2608.11232
• Abstract Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs require ground truth from actual code execution. We present Backtrader-Bench, a framework with two complementary pipelines. A deterministic multiple-choice question (MCQ) pipeline generates questions from backtest configurations across five trading strategies, 33 templates, and three difficulty tiers, with an independent checker that re-derives every answer. A generator-solver filtering pipeline autonomously mines harder questions: a generator writes questions verified by executable code, converts them to MCQs, and discards any that a no-tool solver can answer without code execution. We evaluate 11 models without tools (10 runs each) and four with-tools configurations on a 30-question curated set. Tool-augmented agents reach 90.0% accuracy in a single pass (GPT-5.5 and Opus 4.7), outperforming the best no-tools baselines (73.0%, averaged over 10 runs) by 17 percentage points. On 38 separately mined questions, no-tools accuracy drops further, with half the models falling to roughly random-chance level (25%). Beyond evaluation, the scalable MCQ infrastructure is designed to produce a training corpus for reinforcement learning, with the ultimate goal of building a specialized agent for quantitative trading workflows.
• 中文摘要 在算法交易中评估LLM编码代理很困难，因为静态基准、数据污染风险以及数值回测输出需要实际代码执行的真实信息。我们介绍了Backtrader-Bench，一个包含两条互补流线的框架。确定性多项选择题（MCQ）流程从五种交易策略、33个模板和三个难度层级的回测配置中生成问题，并配有独立检查器重新推导每个答案。生成器-求解器过滤流水线会自动挖掘更难的问题：生成器编写可执行代码验证的问题，转换成选择题，并丢弃无工具求题器无需代码执行即可回答的问题。我们评估了11个无工具模型（每个10次运行）和4个工具配置，涵盖30个问题的精选题组。工具增强代理在一次扫描中可达到90.0%的准确率（GPT-5.5和Opus 4.7），比最佳无工具基线（73.0%，平均10次运行）高出17个百分点。在38个单独挖掘的问题中，无工具准确率进一步下降，一半模型降至大致随机率（25%）。除了评估外，可扩展的多题基础设施还旨在生成强化学习的训练语料库，最终目标是构建一个专门用于定量交易工作流程的代理。

Reinforcement Learning based DBMS Buffer Pool Auto-Tuning for Optimal Memory Utilization

基于强化学习的数据库管理系统缓冲池自动调优以实现内存利用率

• Authors: Yifan Wang, Patrick Royer, Raphaël Féraud, David Delande
• Subjects: Subjects: Databases (cs.DB); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11239
• Pdf link: https://arxiv.org/pdf/2608.11239
• Abstract Administering Database Management Systems (DBMS) instances requires Database Administrators (DBA) to balance performance in terms of Service Level Agreement (SLA) against resource usage, often prompting RAM over-allocation that wastes memory. We introduce MicroTune, an online RL-based buffer adjustment system that minimizes unnecessary memory allocation while ensuring SLA compliance. To identify the most effective RL core, we evaluate multiple algorithms under diverse benchmark workloads, training MicroTune on extensive traces of both external metrics (latency, throughput) and internal DBMS metrics (status variables and performance statistics). Experimental results demonstrate that MicroTune dynamically adapts buffer sizes to workload fluctuations, outperforming baselines by achieving significant memory savings with fewer SLA violations. These findings underscore the promise of reinforcement learning for adaptive resource management in DBMS environments.
• 中文摘要 管理数据库管理系统（DBMS）实例时，数据库管理员（DBA）需要在服务水平协议（SLA）与资源使用之间取得性能平衡，这常常导致内存过度分配，浪费内存。我们引入了MicroTune，一种基于强化学习的在线缓冲区调整系统，能够最大限度地减少不必要的内存分配，同时确保符合SLA要求。为了识别最有效的强化学习核心，我们在不同基准工作负载下评估多种算法，并通过大量外部指标（延迟、吞吐量）和内部数据库管理系统指标（状态变量和性能统计）的跟踪来训练微调。实验结果表明，MicroTune 会动态调整缓冲区大小以适应工作负载波动，通过在更少的 SLA 违规下实现显著内存节省，表现优于基线。这些发现强调了强化学习在数据库管理系统环境中自适应资源管理中的前景。

Towards Sustainable Learning in Online Education: A Reinforcement Learning Approach

迈向在线教育中的可持续学习：强化学习方法

• Authors: Chaofan Zhai, Yicheng Song, Ravi Bapna, Junyao Ye
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.11245
• Pdf link: https://arxiv.org/pdf/2608.11245
• Abstract Online education offers unprecedented scalability and accessibility to global learners from diverse backgrounds, but it often suffers from low engagement and poor long term learning effectiveness. To address these challenges, we introduce AI Tutor, a reinforcement learning based model designed to promote sustainable learning by optimizing both short and longterm learning outcomes. In the short term, AI-Tutor draws on cognitive theory to guide learners through a balance of acquiring new knowledge and reinforcing prior learning. In the long term, it models learner engagement to inform strategies that sustain motivation and reduce dropout. These enhancements enable AI-Tutor to provide personalized guidance that fosters both effective learning and sustained participation. Empirical evaluations on 23 million learning records from 33,700 learners show that AI Tutor consistently outperforms state-of-the-art baselines across engagement, knowledge retention, and final learning outcomes. Learning path analyses further reveal how AI-Tutor adapts its strategies to learners with diverse profiles, offering adaptive and human-centered support.
• 中文摘要 在线教育为来自不同背景的全球学习者提供了前所未有的可扩展性和可及性，但往往存在低参与度和长期学习效果较差的问题。为应对这些挑战，我们引入了AI Tutor，一种基于强化学习的模型，旨在通过优化短期和长期学习成果来促进可持续学习。短期内，AI-Tutor借助认知理论，引导学习者在获取新知识与巩固既有学习之间取得平衡。从长远来看，它模拟学习者的参与度，指导策略以维持动力并减少辍学。这些改进使 AI-Tutor 能够提供个性化指导，促进有效学习和持续参与。对33,700名学习者的2300万条学习记录进行实证评估显示，AI Tutor在参与度、知识保留和最终学习成果方面始终优于最先进的基线。学习路径分析进一步揭示了 AI-Tutor 如何根据不同背景的学习者调整策略，提供适应性和以人为本的支持。

Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning

一瞥、审视与思考：将视频异常检测从无培训提升为智能推理

• Authors: Shibo Gao, Peipei Yang, Xu-Yao Zhang, Linlin Huang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.11260
• Pdf link: https://arxiv.org/pdf/2608.11260
• Abstract Video Anomaly Detection (VAD) aims to identify anomalous events and localize their temporal intervals. Existing approaches exhibit a "when-what" dissociation: traditional DNN-based methods localize when anomalies occur but lack semantic understanding, whereas LLM-based methods explain what happens but neglect precise temporal grounding. We attribute this to the absence of a unified reasoning paradigm. Inspired by how humans inspect surveillance videos - glancing globally to form temporal hypotheses, scrutinizing suspicious segments, and thinking iteratively to correct errors - we study this global-to-local paradigm from two perspectives. We first propose Glance then Scrutinize (GtS), a training-free framework using static and dynamic textual guidance for coarse-to-fine anomaly grounding and understanding, balancing accuracy and speed. To break the ceiling imposed by frozen external modules, we further propose a tool-augmented agentic VAD method, where a multimodal large language model learns to invoke a video cropping tool, inspect densely resampled frames, and self-correct mislocalized hypotheses, via cold-start supervised fine-tuning followed by reinforcement learning with a joint answer-grounding reward. For training and evaluation, we extend our prior VAGU benchmark into VAGU-T (Video Anomaly Grounding, Understanding, and Thinking), comprising 7,567 real-world videos over 21 anomaly categories with human-validated grounding, explanations, QA pairs, and chain-of-thought tool-calling traces. We further introduce JeAUG, a metric jointly evaluating semantic interpretability and temporal precision. Experiments show that GtS substantially surpasses training-free baselines, while the agentic model delivers both higher accuracy and faster inference.
• 中文摘要 视频异常检测（VAD）旨在识别异常事件并定位其时间间隔。现有方法表现出“何时何物”的解离：传统的DNN方法定位异常发生时，但缺乏语义理解;而基于LLM的方法解释发生了什么，却忽视了精确的时间基础。我们将此归因于缺乏统一的推理范式。受人类观察监控视频的方式启发——全球观察以构建时间假设，仔细审查可疑片段，并迭代思考纠正错误——我们从两个视角研究这一全球到本地范式。我们首先提出了“Glance Then Scrutinize”（GtS），这是一个无训练的框架，利用静态和动态文本指导实现从粗到细的异常基础和理解，平衡准确性与速度。为打破冻结外部模块的限制，我们进一步提出了一种工具增强的代理型VAD方法，多模态大型语言模型通过冷启动监督微调和强化学习，学习调用视频裁剪工具，检查密集重采样的帧，并自我纠正错误定位假设，这些过程包括冷启动监督微调，随后是强化学习并给予联合答案基础奖励。在培训和评估方面，我们将之前的VAGU基准扩展为VAGU-T（视频异常的基础、理解与思考），包含7,567个真实世界视频，涵盖21个异常类别，并配有人工验证的基础、解释、质量保证对和思维链调用工具追踪。我们进一步介绍了JeAUG，这是一种联合评估语义可解释性和时间精度的度量指标。实验表明，GtS远超无训练基线，而代理模型则提供了更高的准确性和更快的推断。

Dynamics Models for Offline Hyperparameter Selection in Real-World RL

现实现实强化学习中离线超参数选择的动力学模型

• Authors: Jordan Coblin, Han Wang, Martha White, Adam White
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11349
• Pdf link: https://arxiv.org/pdf/2608.11349
• Abstract A key obstacle to deploying reinforcement learning in real-world systems is hyperparameter selection, particularly when simulators are unavailable and online experimentation is costly. Prior work has proposed calibration models trained on offline data to approximate environment dynamics and enable offline hyperparameter selection, but these methods have so far been evaluated only in simple simulated settings. In this paper, we present the first application of calibration models in a real-world industrial setting: a municipal water treatment plant. We evaluate several calibration model approaches, including a k-nearest neighbors model with a Laplacian distance metric, on high-dimensional, non-stationary sensor data for nexting prediction tasks. Our results show that these models can generate realistic long-horizon rollouts and recover meaningful hyperparameter sensitivity trends. We further examine how calibration models scale to year-long datasets, how they support the selection of fine-tuning learning rates for pre-trained agents, and how robust they are under distribution shift. Overall, our findings provide a proof of concept for using offline dynamics models to support RL deployment in real-world environments, while highlighting important practical challenges for future work.
• 中文摘要 在现实系统中部署强化学习的一个主要障碍是超参数选择，尤其是在模拟器无法使用且在线实验成本高昂的情况下。此前已有研究提出基于离线数据训练的校准模型，以近似环境动力学并实现离线超参数选择，但这些方法迄今仅在简单的模拟环境中进行评估。本文首次将校准模型应用于真实工业环境：市政水处理厂。我们评估了多种校准模型方法，包括基于拉普拉斯距离度量的k最近邻模型，用于下一个预测任务的高维非驻地传感器数据。我们的结果表明，这些模型能够生成现实的长期推广，并恢复有意义的高参数敏感性趋势。我们还进一步探讨了校准模型如何扩展到全年数据集，它们如何支持预训练代理的微调学习率选择，以及它们在分布偏移下的稳健性。总体而言，我们的发现为利用离线动力学模型支持现实环境中强化学习的部署提供了概念验证，同时也凸显了未来工作面临的重要实际挑战。

Self-Evolving Embodied Agents via Skill-Harness Evolution

通过技能驱动进化自我进化的具身代理

• Authors: Peidong Wang, Zhiming Ma, Ying Chang, Xufang Luo, Xiaocui Yang, Shi Feng, Yuqing Yang, Dongsheng Li
• Subjects: Subjects: Computation and Language (cs.CL); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.11350
• Pdf link: https://arxiv.org/pdf/2608.11350
• Abstract Embodied agents are increasingly built as systems around foundation models, where performance depends not only on model weights but also on the skills, context, action interfaces, and execution harness surrounding the model. While supervised fine-tuning and reinforcement learning can adapt agents to new environments, they require additional data, rewards, and training runs; meanwhile, many train-free code-centric approaches rely on programmable robot APIs that may be unavailable in fixed-interface settings. We propose SHAPER, a self-evolving framework for train-free embodied adaptation that keeps model parameters frozen and improves the non-parametric agent system by evolving reusable skills and a context-code harness through target-environment rollouts. In SHAPER, the same frozen model can serve as both planner and optimizer, refining its external skills and context-code harness without parameter updates. We evaluate SHAPER on VLABench and ESI-Bench, covering embodied agents with different low-level action interfaces, and compare against pure execution, supervised fine-tuning, and test-time-scaling baselines such as verifier-free selection and voting. Our results suggest that skill-and-harness optimization is a practical route to self-evolving embodied agents when model training is expensive, unavailable, or undesirable.
• 中文摘要 具身代理越来越多地作为基于基础模型的系统构建，性能不仅取决于模型权重，还取决于围绕模型的技能、上下文、动作接口和执行工具。虽然监督式微调和强化学习可以让智能体适应新环境，但它们需要额外的数据、奖励和训练运行;与此同时，许多无列车代码中心的方法依赖可编程机器人API，而这些API在固定接口设置中可能不可用。我们提出了SHAPER，这是一个自我演进的无序列具身适应框架，通过目标环境推广，保持模型参数冻结，并通过可复用技能和上下文代码驱动，改进非参数代理系统。在 SHAPER 中，同一冻结模型既可作为规划器，也可以作为优化器，在不更新参数的情况下精炼其外部技能和上下文代码利用。我们在VLABench和ESI-Bench上评估SHAPER，涵盖具身代理的低层次动作接口，并与纯执行、监督微调和测试时间缩放基线（如无验证者选择和投票）进行比较。我们的结果表明，当模型训练成本高昂、不可得或不受欢迎时，技能与利用优化是实现自我演化具身代理的实用途径。

PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR

PAIR：RLVR中自适应推广分配的成对感知包含重权

• Authors: Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.11368
• Pdf link: https://arxiv.org/pdf/2608.11368
• Abstract Reinforcement learning with verifiable rewards (RLVR) spends most of its compute generating groups of long reasoning trajectories. Recent allocators reduce this cost by assigning budgets to prompts, rollouts, or tokens according to a pointwise notion of difficulty or utility. We identify a statistical mismatch: the unclipped leave-one-out group-relative score gradient is not a sum of independent point contributions, but a second-order U-statistic over pairs of rollouts. Completing one rollout therefore reveals contrast with every other completed rollout, and adaptive endpoint selection changes which pair terms are observable. We introduce PAIR (Pairwise-Aware Inclusion Reweighting), which treats short rollout prefixes as vertices and pair-gradient terms as edges of a contrast graph. A prefix-only predictor estimates correctness and remaining token cost; a convex design chooses positive continuation probabilities under an expected suffix-token budget; and each edge induced by completed vertices is inverse-weighted by its logged joint inclusion probability. Under conditionally independent on-policy rollouts and an unclipped, unstandardized objective, the resulting estimator is design-unbiased for the complete candidate-pair gradient. Across compute-matched RLVR runs on Qwen3-1.7B/4B, PAIR improves average accuracy by +1.2 and +1.4 over the strongest pointwise allocator while using 51% and 52% fewer generated tokens than full-group GRPO. A frozen-population estimator audit confirms that unweighted adaptive selection is biased, whereas pair-inclusion correction recovers the complete-pair target at matched suffix cost.
• 中文摘要 带可验证奖励的强化学习（RLVR）大部分计算时间用于生成长推理轨迹的组。最近的分配者通过根据难度或效用的点数为提示、推广或代币分配预算来降低成本。我们发现了一个统计不匹配：未截断的未退出组相对得分梯度不是独立得分贡献的总和，而是对数次展开的二阶U统计量。因此，完成一个推广会显示与其他所有完成的部署的对比，自适应端点选择改变了可观测的配对项。我们引入了PAIR（成对感知包含重权），它将短的滚动前缀视为顶点，对梯度项视为对比图的边。仅带前缀的预测器用于估算正确性和剩余令牌成本;凸设计在预期后缀-标记预算下选择正延续概率;每个由已完成顶点诱导的边都被其对数连接包含概率反权重。在条件独立的策略展开和未截断、未标准化目标下，得到的估计对完整候选对梯度设计无偏。在Qwen3-1.7B/4B上的计算匹配RLVR运行中，PAIR在使用比全组GRPO少51%和52%的生成代币时，平均准确率分别提升了+1.2和+1.4。冻结总体估计器审计确认未加权自适应选择存在偏差，而配对包含校正则恢复了匹配后缀成本的完全对目标。

Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits

通过反事实临床审计揭露ICU败血症管理中医疗离线强化学习中的有毒模仿

• Authors: Hangqi Ren, Junyi Liao
• Subjects: Subjects: Machine Learning (cs.LG); Computers and Society (cs.CY)
• Arxiv link: https://arxiv.org/abs/2608.11410
• Pdf link: https://arxiv.org/pdf/2608.11410
• Abstract Offline reinforcement learning (RL) offers considerable promise for optimizing ICU treatment decisions, yet standard evaluation metrics Mean Squared Error (MSE) and Fitted Q-Evaluation (FQE) assess only behavioral imitation and cannot detect Toxic Mimicry, a failure mode in which agents replicate harmful patterns such as treatment withdrawal during comfort-care transitions. Using the MIMIC-III database, we propose the Counterfactual Clinical Audit (CCA) framework, which stress-tests RL agents through physiological perturbations anchored in Surviving Sepsis Campaign (SSC) guidelines. We audit a Medical Decision Transformer (MedDT) and a Historical Causal Transformer (HCT-RL), the latter employing Causal Action Shielding, propensity-based importance weighting, and Conservative Q-Learning. CCA reveals that MedDT paradoxically reduces vasopressor dosage as lactate escalates, contradicting resuscitation guidelines, while HCT-RL maintains physiologically consistent responses. These findings expose a systemic misalignment between statistical fit and clinical safety, supporting counterfactual audits as a necessary evaluation standard for medical RL.
• 中文摘要 离线强化学习（RL）在优化ICU治疗决策方面具有相当大潜力，但标准评估指标均方误差（MSE）和拟合Q评估（FQE）仅评估行为模仿，无法检测毒性模仿（Toxic Mimicry），即代理在舒适护理过渡期间复制有害模式（如治疗退出）的失败模式。利用MIMIC-III数据库，我们提出了反事实临床审计（CCA）框架，通过生理扰动对强化学习病原体进行压力测试，这些扰动基于生存败血症运动（SSC）指南。我们审计了医疗决策转换器（MedDT）和历史因果转换器（HCT-RL），后者采用因果行为屏蔽、基于倾向的重要性加权和保守Q学习。CCA显示，MedDT在乳酸升高时反而会降低血管增压剂剂量，这与复苏指南相矛盾，而HCT-RL则保持生理反应一致。这些发现揭示了统计契合度与临床安全性之间的系统性错位，支持反事实审计作为医学强化学习评估的必要标准。

Benchmarking LLM Judges for Mobile Agent Evaluation

对移动代理评估的大型语言模型评判基准测试

• Authors: Ziqiang Wan, Li Gu, Zhixiang Chi, Zhi Liu, Seyed Mehdi Ayyoubzadeh, Yuanhao Yu, Yang Wang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.11434
• Pdf link: https://arxiv.org/pdf/2608.11434
• Abstract Mobile agent benchmarks increasingly rely on LLM-based judges to evaluate task completion, yet the reliability of these judges on mobile agent trajectories remains largely unexamined. We introduce MobileJudgeBench, a benchmark for systematically evaluating LLM-as-judge methods on mobile agent trajectories. Our benchmark comprises 931 human-annotated trajectories spanning 6 mobile agent benchmarks, 4 agent models, and 68 apps. We evaluate 6 judge methods (five adapted from SPA-Bench, A3 with two modes, AndroidArena, and AgentRewardBench, plus a simple baseline we design) across multiple LLM backends. Our experiments reveal three key findings. First, a simple baseline judge with sampled screenshots is competitive with, and often exceeds, purpose-built methods, indicating that more elaborate judge pipelines do not consistently improve judge quality; among competitive methods, the LLM backbone is the primary driver. Second, benchmark quality metrics reliably predict real-world judge utility: they correlate with both agent ranking fidelity for evaluation and downstream performance when judges serve as reward signals for on-policy reinforcement learning. Third, failure analysis across two LLM backends uncovers qualitatively opposite failure profiles, one conservative and the other permissive, linked to the backbone's precision-recall characteristics.
• 中文摘要 移动代理基准越来越依赖基于LLM的评判来评估任务完成度，但这些评判对移动代理轨迹的可靠性仍然鲜有深入探讨。我们介绍了MobileJudgeBench，这是一个系统性评估移动代理轨迹的LLM作为评判方法的基准。我们的基准包含931条人工注释轨迹，涵盖6个移动代理基准、4个代理模型和68个应用。我们评估了6种评判方法（其中5种改编自SPA-Bench、A3两种模式、AndroidArena和AgentRewardBench，以及我们设计的简单基线），覆盖多个LLM后端。我们的实验揭示了三个关键发现。首先，一个简单的基线裁判和采样截图在与专门设计的方法中具有竞争力，甚至常常超过，表明更复杂的裁判流程并不能持续提升裁判质量;在竞争方法中，LLM骨干是主要驱动力。其次，基准质量指标可靠地预测现实世界的裁判效用：它们与评估中的代理排名忠实度以及评审作为策略强化学习奖励信号时的后续表现相关。第三，跨两个LLM后端的故障分析揭示了质性上截然相反的失败特征，一个保守，另一个宽容，与骨干的精度召回特性相关。

Let it Cook: Learning to Wait in Sequential Decision Making

让它烹饪：学会在顺序决策中等待

• Authors: Christopher Watson, Arjun Krishna, Dinesh Jayaraman, Rajeev Alur
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11511
• Pdf link: https://arxiv.org/pdf/2608.11511
• Abstract In sequential decision making, an agent typically observes its environment and acts at every timestep. However, such active participation may not always be necessary; tasks such as brewing coffee include periods that are served equally well by letting the environment evolve without constant monitoring and control. During such periods, the agent could simply wait to conserve its resources, or redirect its attention to another task. We capitalize on these opportunities by training a "waiting policy" that decides where and how long to wait. This involves forgoing sensing to commit to a wait action, representing a deliberate pause for a set number of timesteps. We formalize "learning to wait" as minimizing the frequency of sensing and decision making without sacrificing task performance (e.g., the total amount of time to complete a task). To train a waiting policy, we propose an approach that employs reinforcement learning with lexicographically ordered objectives. In experiments across 4 discrete-state household tasks and 3 continuous-state environments, we show that our approach successfully learns waiting behaviors, and can adapt pre-trained policies to wait where appropriate. While different tasks permit different amounts of waiting without sacrificing task performance, our approach consistently finds solutions with significant waiting, sometimes waiting for over 50 percent of the task duration.
• 中文摘要 在顺序决策中，代理通常观察其环境并在每个时间步采取行动。然而，这种积极参与并不总是必要的;像煮咖啡这样的任务，也包括让环境自然演变、无需持续监控和控制的时期。在这些时间段，代理可以选择等待以节省资源，或将注意力转向其他任务。我们通过培训“等待政策”来决定等待地点和等待时间，从而抓住这些机会。这包括放弃感知，承诺等待动作，表示在一定时间步内故意暂停。我们将“学会等待”形式化为在不牺牲任务表现（例如完成任务的总时间）的情况下，最小化感知和决策的频率。为了训练等待策略，我们提出了一种采用词典序目标的强化学习方法。在4种离散状态家庭任务和3种连续状态环境的实验中，我们证明了我们的方法成功学习了等待行为，并能调整预训练策略以适时等待。虽然不同任务允许的等待时间不同且不牺牲任务性能，但我们的方法始终能找到等待时间较长的解决方案，有时等待时间超过任务时长的50%。

Reinforcing Step-level Reasoning for Effective Self-Correction in LLMs

强化大型语言模型中有效自我纠正的步骤级推理

• Authors: Vu Duc Anh, Nhat M. Hoang, Do Xuan Long, Cong-Duy Nguyen, Ponhvoan Srey, Luu Anh Tuan
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11573
• Pdf link: https://arxiv.org/pdf/2608.11573
• Abstract Achieving effective self-correction, where models verify and correct their own mistakes, remains a fundamental challenge for large language models (LLMs). In this work, we propose Self-Fix Step-DPO (SFS-DPO), a reinforcement learning based, two-stage framework for step-level self-verification and self-correction. The first stage strengthens step-level reasoning via step-level preference optimization, while the second stage explicitly trains models to self-verify and self-correct. We further introduce a teacher-assisted variant, SFS-DPO-R, which incorporates explanatory rationales for error verification to provide stronger corrective signals. Comprehensive in-domain and out-of-domain evaluations across multiple LLMs demonstrate that SFS-DPO and SFS-DPO-R consistently outperform prior step-level training baselines. Our analysis further reveals improvements in self-correction frequency and effectiveness, highlighting the importance of strengthening step-level reasoning for robust performance.
• 中文摘要 实现有效的自我纠正，即模型自行验证并纠正错误，仍然是大型语言模型（LLMs）面临的根本挑战。在本研究中，我们提出了自我修复步骤DPO（SFS-DPO），这是一种基于强化学习的两阶段框架，用于步骤级自我验证和自我纠正。第一阶段通过逐步偏好优化强化步级推理，第二步则显式训练模型自我验证和自我纠正。我们还介绍了教师辅助变体SFS-DPO-R，它结合了错误验证的解释性理由，以提供更强的纠正信号。多台LLM的全面领域内外评估表明，SFS-DPO和SFS-DPO-R始终优于以往的步级培训基线。我们的分析进一步揭示了自我纠正频率和效果的提升，强调了强化阶级推理对于强健表现的重要性。

IoT-Enabled Autonomous Maritime Navigation in Smart Ports: A Curriculum-Guided Shared Policy Learning Framework

物联网驱动的智能港口自主海事导航：课程指导共享政策学习框架

• Authors: Yuqing Lin, Rangya Zhang, Kum Fai Yuen
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.11597
• Pdf link: https://arxiv.org/pdf/2608.11597
• Abstract As smart port infrastructures increasingly rely on autonomous maritime devices enabled by the Internet of Things (IoT), ensuring reliable onboard navigation intelligence has become a critical challenge for safe and scalable operations in congested waterways. This paper investigates onboard autonomous navigation for such IoT devices under partial observability and dense traffic conditions. A curriculum-guided reinforcement learning framework with a shared recurrent policy is developed to enhance temporal reasoning, deployment scalability, and robustness of edge-level decision-making. Centralized training is adopted as an offline design-time strategy, while all navigation actions are executed fully onboard, consistent with IoT edge intelligence paradigms. Extensive simulations in multiple realistic port environments demonstrate that the proposed approach improves navigation reliability, collision avoidance, and training stability compared with standard baseline methods, and generalizes effectively to previously unseen high-density scenarios. The results indicate that curriculum-guided shared learning provides a practical solution for scalable deployment of IoT-enabled autonomous maritime devices in smart port operations.
• 中文摘要 随着智能港口基础设施越来越依赖物联网（IoT）支持的自主海事设备，确保可靠的船上导航智能已成为在拥挤水道中安全且可扩展运营的关键挑战。本文探讨了此类物联网设备在部分可观测性和交通密集条件下的机载自主导航。开发了一个以课程为导向的强化学习框架，采用共享的重复策略，以增强时间推理、部署可扩展性和边缘级决策的稳健性。集中培训作为离线设计时策略，所有导航操作完全在机上执行，符合物联网边缘智能范式。在多个真实港口环境中的大量模拟表明，所提方法相较于标准基线方法提升了导航可靠性、碰撞避免和训练稳定性，并有效推广至此前未见的高密度场景。结果表明，课程指导的共享学习为物联网自主海事设备在智能港口运营中的可扩展部署提供了切实可行的解决方案。

Learning from Online User Feedback for Shopping Agents

从在线用户反馈中学习购物代理

• Authors: Haobo Zhang, Kelong Mao, Sulong Xu, Simiu Gu, Zhicheng Dou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11604
• Pdf link: https://arxiv.org/pdf/2608.11604
• Abstract Large language model-based shopping agents are increasingly deployed in real-world e-commerce platforms, generating massive amounts of user interaction logs that provide valuable supervision for improving these agents. However, existing approaches primarily rely on offline training signals, such as user-item interactions or synthetic preference data, while largely overlooking the rich supervision contained in users' natural conversational feedback. Moreover, the available online feedback is heterogeneous, sparse, and noisy, making it difficult to transform into reliable learning signals automatically. To address these challenges, we propose LOFA, a framework that enables shopping agents to learn directly from real online interaction logs without human annotation. LOFA combines reinforcement learning over verifiable purchase outcomes with feedback-aware on-policy distillation, which identifies users'in-dialogue directives and converts them into dense token-level supervision. These complementary objectives capture both collaborative behavioral patterns and user-specific preferences. Extensive experiments on real-world e-commerce logs demonstrate that LOFA consistently improves recommendation quality, response helpfulness, and user-satisfaction alignment over strong baselines, highlighting the effectiveness of learning shopping agents from real online user feedback.
• 中文摘要 基于大型语言模型的购物代理越来越多地部署在现实世界的电子商务平台，产生大量用户互动日志，为改进这些代理提供了宝贵的监督。然而，现有方法主要依赖离线训练信号，如用户与项目的交互或合成偏好数据，而在很大程度上忽视了用户自然对话反馈中蕴含的丰富监督。此外，可用的在线反馈异质化、稀疏且噪声较大，难以自动转化为可靠的学习信号。为应对这些挑战，我们提出了LOFA框架，使购物代理能够直接从真实的在线互动日志中学习，而无需人工注释。LOFA结合了对可验证购买结果的强化学习与基于反馈的政策提炼，识别用户对话指令并将其转化为密集的代币级监督。这些互补目标既体现了协作行为模式，也体现了用户特定的偏好。对真实电商日志的广泛实验表明，LOFA在强有力基线上持续提升推荐质量、响应便利性和用户满意度的一致性，凸显了从真实在线用户反馈中学习购物代理的有效性。

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

学会说服揭示了大型语言模型（LLM）多么容易放弃正确的信念

• Authors: Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur, Dilek Hakkani-Tür
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11624
• Pdf link: https://arxiv.org/pdf/2608.11624
• Abstract Persuasion is a core dynamic of natural language communication, shaping how large language models (LLMs) update beliefs, resolve disagreements, and reach decisions. As LLMs increasingly debate, advise, and think collaboratively with humans and each other, resistance to harmful persuasion becomes a core requirement for reliable behavior. Yet we show that this requirement is far from met: a single targeted persuasive argument is enough to collapse model accuracy to near zero, even when the argument is factually false. We formalize this threat as adversarial persuasion and introduce an adversarial reinforcement learning framework that trains persuader agents to change a target model's answer in a single interaction. First, we show that optimizing persuasion strategies through trial and error exposes vulnerabilities that static prompting misses: RL-trained persuaders raise persuasion success from approximately 24% to over 93% against the training-time persuadee. Second, we find that these learned strategies transfer to unseen models, achieving 83% attack success on Qwen-14B, 79% on Llama-3.1-8B, and 25% on GPT-4o-mini. Third, we demonstrate that a curriculum that bootstraps on more persuadable open-weight models before targeting harder models further increases GPT-4o-mini attack success from 25% to 38%. Moreover, our results reveal that optimized persuaders increasingly rely on credibility-based tactics, including fabricated citations and false authoritative evidence. Together, these findings expose a critical weakness in current LLM agents: even when they initially reason correctly, they can be steered toward false conclusions by optimized natural language influence. This positions persuasion robustness as a necessary safety criterion for multi-agent and human-AI decision-making systems.
• 中文摘要 说服是自然语言交流的核心动态，塑造了大型语言模型（LLMs）如何更新信念、解决分歧和做出决策。随着大型语言模型越来越多地与人类和彼此进行辩论、建议和协作思考，抵抗有害说服成为可靠行为的核心要求。然而，我们也证明这一要求远未满足：一个有针对性的说服性论点足以使模型准确性接近零，即使该论点事实上是错误的。我们将这种威胁形式化为对抗性说服，并引入了一种对抗性强化学习框架，训练说服者在一次交互中改变目标模型的答案。首先，我们表明通过反复试验优化说服策略可以暴露静态提示所忽略的漏洞：强化学习训练的说服者将说服成功率从约24%提升到超过93%，相较于训练时间说服者。其次，我们发现这些所学策略可迁移到未见模型，Qwen-14B攻击成功率为83%，Llama-3.1-8B为79%，GPT-40-mini为25%。第三，我们展示了在针对更难模型之前，先从更具说服力的开放权重模型启动的课程，进一步将GPT-40-mini攻击成功率从25%提升到38%。此外，我们的结果显示，优化后的说服者越来越依赖基于可信度的策略，包括伪造引用和虚假权威证据。这些发现共同揭示了当前大型语言模型代理的一个关键弱点：即使他们最初推理正确，也可能被优化的自然语言影响引导至错误结论。这使得说服鲁棒性成为多智能体和人机决策系统的必要安全标准。

CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement

主张：领先的开放领域主动澄清大型语言模型的不确定性测量

• Authors: Kuangzhao Yang, Ziliang Zhao, Zhicheng Dou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11631
• Pdf link: https://arxiv.org/pdf/2608.11631
• Abstract In open-domain human-computer interaction scenarios, large language models (LLMs) frequently encounter user queries that are ambiguous or incomplete. In such cases, directly producing an answer often leads to overgeneralized, erroneous, or low-information responses. In contrast, asking clarifying questions can substantially improve interaction quality. However, existing approaches still rely heavily on manually annotated data or preference alignment to address two fundamental challenges: when clarification is necessary, and which aspect of the query should be clarified. This reliance incurs high annotation costs and limits generalization. To address these challenges, we propose CLAIM, an uncertainty-driven framework for active clarification learning in open-domain settings. CLAIM eliminates the need for explicit human preference annotations by quantifying query uncertainty through the entropy induced by answer disagreements across multiple models. This uncertainty signal is then used to construct high-quality synthetic data, enabling the training of a unified clarification decision model through a combination of supervised learning and reinforcement learning. Specifically, we propose an entropy-driven synthetic data generation pipeline that integrates entropy-based uncertainty estimation with semantic clustering and reasoning-based judgments, enabling reliable automatic annotation of clarification requirements. To train CLAIM, we formulate the clarification process as a structured decision generation problem and adopt a training paradigm that combines supervised fine-tuning (SFT) with group-relative policy optimization (GRPO). Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data, offering a low-cost and robust solution for proactive understanding in real-world open-domain interactions with LLMs.
• 中文摘要 在开放领域的人机交互场景中，大型语言模型（LLM）经常遇到用户查询，这些查询是歧义或不完整的。在这种情况下，直接给出答案往往会导致过度概括、错误或信息不足的回答。相比之下，提出澄清性问题可以显著提升互动质量。然而，现有方法仍高度依赖手动注释数据或偏好对齐来解决两个根本挑战：何时需要澄清，以及查询的哪个方面应被澄清。这种依赖会产生高注释成本并限制泛化。为应对这些挑战，我们提出了CLAIM，这是一个基于不确定性的开放领域主动澄清学习框架。CLAIM通过通过多个模型间答案不一致引起的熵量化查询不确定性，消除了对显式人类偏好注释的需求。随后利用该不确定性信号构建高质量的合成数据，通过监督学习和强化学习相结合，训练统一的澄清决策模型。具体来说，我们提出了一种基于熵的合成数据生成流程，将基于熵的不确定性估计与语义聚类和基于推理的判断整合在一起，实现了可靠的自动说明注释。为了训练CLAIM，我们将澄清过程表述为结构化决策生成问题，并采用结合监督微调（SFT）和群体相对政策优化（GRPO）的训练范式。实验结果表明，CLAIM能够在不依赖手动标记数据的情况下学习稳定且可推广的澄清策略，为现实开放域与大型语言模型的交互提供了低成本且可靠的主动理解解决方案。

Is Per-Agent Policy Composition Safe? Rethinking Successor-Feature Transfer in Cooperative Multi-Agent Reinforcement Learning

每位代理人的保单构成安全吗？重新思考合作式多智能体强化学习中的继任特征转移

• Authors: Zijian Zhao, Sen Li
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.11658
• Pdf link: https://arxiv.org/pdf/2608.11658
• Abstract Many reinforcement learning systems, from fleet management to traffic signal control, must serve an objective that changes dynamically after deployment, and retraining a policy for each new objective is prohibitively expensive. For a single agent, this problem is well understood: successor features with generalized policy improvement, together with their universal extension, recombine a library of learned policies into a policy for any new objective, with a guarantee that the result is never worse than any policy in the library. However, multi-agent transfer has received far less attention, and the common practice of letting each agent recombine its own library independently inherits the recipe but not the guarantee. We prove that this independent composition can produce joint behavior strictly worse than every policy in the library, because recombining teammates changes the environment each agent faces and invalidates the values it relies on, a failure with no single-agent counterpart. We further show that the only unconditionally safe fixed rule is synchronized composition, which moves the whole team to one jointly trained policy but cannot serve objectives that assign different goals to different agents. To attain safety and flexibility at once, we propose MA-USFA, a hierarchical method with two layers: a lower layer of universal successor feature approximators that predicts each agent's successor features while conditioned on its teammates' objectives, and an upper composer that selects, across agents, which library entry each agent should follow and supplies the cross-agent correction a per-agent value cannot represent. Trained once over the distribution of objectives, it is applied at deployment with no per-task adaptation.
• 中文摘要 许多强化学习系统，从车队管理到交通信号控制，都必须服务于部署后动态变化的目标，且为每个新目标重新训练策略成本高昂。对于单个代理，这个问题是被充分理解的：继任功能配合广义策略改进及其通用扩展，将学习到的策略库重新组合成针对任何新目标的策略，并保证结果永远不会比库中的任何策略差。然而，多智能体转移受到的关注远少于此，允许每个智能体独立重组自身库的常见做法继承了配方，但不保证。我们证明了这种独立组合产生的联合行为可能严格糟糕于库中的所有策略，因为重新组合队友会改变每个代理所面临的环境，并使其依赖的值失效，这是没有单代理对应物的失败。我们还进一步证明，唯一无条件安全的固定规则是同步组合，它将整个团队移动到一个共同训练的策略，但不能满足将不同目标分配给不同代理的目标。为了同时实现安全和灵活性，我们提出了MA-USFA，这是一种分层方法，分为两层：下层是通用继任特征近似器，根据团队目标预测每个代理的继任特征;上层的组合器则在各代理间选择每个代理应遵循的库条目，并提供每个代理值无法代表的跨代理修正。在目标分布上训练一次后，部署时应用，无需逐项调整。

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

评分标准退出：在评分标准作为奖励的强化学习中，缓解奖励被篡改的简单方法

• Authors: Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.11669
• Pdf link: https://arxiv.org/pdf/2608.11669
• Abstract Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer. The rubric, however, is a fixed proxy for quality, never a complete description of it, and a policy trained against it long enough will learn to exploit the difference. We measure this directly. Training Qwen3-8B with Group Relative Policy Optimization (GRPO) on medical and science rubrics and grading out-of-distribution (OOD) benchmarks with both the training judge and a stronger gold judge, we find that the two scores diverge during training. The training judge's score keeps climbing while the gold judge's score peaks and then falls, by 3 points on HealthBench-Hard and by 22 points on ResearchQA. A judge with a fixed bias would shift the gold curve by a constant, not send it down while the training score rises, so the divergence is reward hacking, not judge noise. We propose Rubric Dropout, a one-line fix borrowed from neuron dropout. At every step, we randomly drop a subset of the rubric's criteria before computing the reward, so the policy never optimizes the same rubric twice. The dropped subset is shared across each rollout group, so GRPO's group-relative advantages stay comparable, and evaluation always uses the full rubric. Comparing no dropout against dropout at 30% and 50% on both benchmark pairs, dropout raises the OOD gold score at every matched checkpoint (+1 to +2 points on HealthBench-Hard, +6 to +7 points on ResearchQA), lowers the two hacking measures we track, and costs nothing in domain. Sweeping the dropout fraction shows a broad 30-50% sweet spot, while the natural alternative, reweighting criteria by how useful they are to training, performs worse than no intervention at all in our setting.
• 中文摘要 基于评分标准的强化学习，即由LLM评审评分的标准列表，已成为对无确定性答案任务进行语言模型后期训练的标准方式。然而，评分标准是质量的固定代理指标，从未完整描述质量，长期受过训练的政策会学会利用差异。我们直接测量这一点。在训练评委和更强的金评审中，我们发现两者在培训期间，使用群体相对政策优化（GRPO）进行医学和科学评分标准及外分配（OOD）基准的评分，发现两者在培训期间存在差异。训练评委的分数持续攀升，而金评委的分数则在HealthBench-Hard上高峰后下降3分，ResearchQA下降22分。带有固定偏置的裁判会将金色曲线调整为常数，而不是在训练分数上升时向下移动，因此发散是奖励黑客，而非评判噪声。我们提出了“评分标准脱落”，这是一种借鉴自神经元脱落的一行修正。在计算奖励前，每一步我们都会随机丢弃评分标准的子集，这样策略就不会重复优化同一个评分标准。被淘汰的子集在每个推广组间共享，因此GRPO的组别优势保持可比，评估始终使用完整的评分标准。将两个基准对的无掉落率分别为30%和50%进行比较，掉落会在每个匹配检查点提升OOD金币分数（HealthBench-Hard为+1到+2点，ResearchQA为+6到+7点），降低了我们追踪的两项黑客指标，且在领域内不会产生任何成本。扫描辍学比例显示一个30%-50%的“最佳”点，而自然的替代方案——根据训练的有用度重新加权标准——在我们的环境中表现不如完全不干预。

GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

GCPO：在LLM的Rollout RL中诊断和约束亚空间几何

• Authors: Kai Yang, Jingwei Xu, Wanyu Wang, Kai-Yuan Guo, Zhenbo Yu, Yi Wang, Yu Qiao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11674
• Pdf link: https://arxiv.org/pdf/2608.11674
• Abstract On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.
• 中文摘要 如GRPO等策略内推广方法是大型语言模型后训练的核心，但它们常常存在训练不稳定性、跨任务能力下降和响应长度膨胀的问题。尽管先前研究已经描述了聚合更新的子空间几何，但该几何的逐步变化及其与模型性能的关系仍不明确。我们引入了主子空间重叠，这是一种经过维度修正的衡量，衡量单个外展更新相对于预训练权重的主导奇异子空间。尽管平均重叠较低，但瞬态峰值通常先于性能下降。为此，我们提出了 GCPO（几何约束策略优化），该方法通过硬双边正交投影约束互补子空间的更新，防止此类结构上的偏差。在Qwen3-8B和GLM4-9B的数学推理、代码生成和工具使用任务中，GCPO持续优于GRPO及近期变体（包括DAPO和GSPO），分别比基础模型和最强基线提升27.69点和2.37分。此外，GCPO保留了一般能力，消除了响应长度膨胀，并稳定了政策熵。我们的发现为训练后稳定强化学习提供了新的诊断视角和原则性设计视角。

When the API Speaks the Wrong Language: Revisiting Post-Training for Multilingual Tool Use

当API说错语言时：重新审视多语言工具使用的培训后

• Authors: Siddharth Chauhan, Thomas Butler, Abhishek Singhania, Pankaj Porwal, Honey Gupta
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11715
• Pdf link: https://arxiv.org/pdf/2608.11715
• Abstract The reliability of Large Language Models (LLMs) for API calling degrades in multilingual settings. A common failure occurs when a model selects the correct tool but generates argument values in an inconsistent language, which we term Argument Language Mismatch (ALM). Although semantically correct, such outputs are operationally invalid and not captured by standard API-calling metrics. We revisit post-training strategies for mitigating ALM and find that, in our benchmark, supervised fine-tuning (SFT) provides a strong baseline, substantially improving argument language consistency and end-to-end function call accuracy. Under consistent model selection, SFT achieves performance comparable to, and sometimes exceeding more complex reinforcement learning (RL) approaches. We further examine whether RL with structured, argument-aware rewards offers additional benefits. While methods such as Group Relative Policy Optimization (GRPO) can improve language consistency and better preserve general reasoning ability, these gains are incremental and most pronounced in generalization and multi-objective trade-offs. Overall, our results suggest that much of the performance in multilingual API grounding can be achieved through careful supervised training, with RL providing targeted rather than fundamental improvements.
• 中文摘要 大型语言模型（LLMs）在多语言环境中调用API的可靠性会下降。常见失败发生在模型选择了正确的工具，但生成的参数值却存在不一致的语言，我们称之为论元语言不匹配（ALM）。虽然语义上正确，但这些输出在操作上无效，且未被标准的API调用指标捕获。我们重新审视了训练后缓解ALM的策略，发现在我们的基准测试中，监督式微调（SFT）提供了强有力的基线，显著提升了论元语言的一致性和端到端函数调用的准确性。在一致的模型选择下，SFT的性能可与更复杂的强化学习（RL）方法相当，有时甚至超过。我们还进一步考察了带有结构化、具备论点意识奖励的强化学习是否带来了额外好处。虽然像群相对策略优化（GRPO）这样的方法可以提升语言一致性并更好地保持一般推理能力，但这些提升是渐进的，且最显著的体现在泛化和多目标权衡上。总体来看，我们的结果表明，多语言API基础化的大部分性能可以通过细致的监督训练实现，强化学习提供针对性而非根本性的改进。

GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation

GRPO用于金融建议生成：在CATE评估中优于商业大型语言模型

• Authors: Ofir Ben Shoham, Shrutendra Harsola, Vignesh Subrahmaniam, Shravan Mohan, Yakov Gazman, Oded Vainas
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.11787
• Pdf link: https://arxiv.org/pdf/2608.11787
• Abstract Generating actionable financial advice from business records demands that models integrate numerical reasoning, domain knowledge, and sound judgment, while avoiding recommendations that could harm the business. Direct supervision is difficult: historical decisions are not necessarily optimal, and high-quality free-form labels are expensive to obtain. We formulate financial advice generation as a reinforcement learning problem and fine-tune an open-weight language model using Group Relative Policy Optimization (GRPO). Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention. Since LLM-based evaluation alone cannot confirm whether improvements reflect genuine business value rather than adaptation to the judge, we complement it with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE) estimator. Under this observational off-policy audit, our trained LLM achieves approximately twice the estimated gross-profit lift of the strongest evaluated commercial baseline ($0.0228$ vs.\ $0.0104$), together with the lowest downside rate and the least negative tail risk of any policy evaluated. Notably, the two evaluations do not rank the baselines identically: the untrained base model places last on the judge rubric but second on the causal audit, indicating that the audit captures a signal the judge does not. Our results demonstrate that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs, and that a judge-independent causal audit is a valuable complement to, rather than a confirmation of, LLM-as-a-judge assessment in financial NLP.
• 中文摘要 从业务记录中生成可行的财务建议，要求模型整合数字推理、领域知识和明智判断，同时避免可能损害企业的建议。直接监督很难：历史决策不一定最优，且高质量的自由格式标签获取成本高昂。我们将财务建议生成作为强化学习问题，并利用群体相对策略优化（GRPO）微调开权语言模型。我们的奖励是一个LLM作为评审的评分标准，该评分标准涵盖多个二元维度的建议质量，并辅以安全门以防止伤害。由于仅凭LLM评估无法确认改进是否反映真实的商业价值，而非对评判的适应，我们补充基于标准双稳健条件平均处理效应（CATE）估计的评审独立审计。在这种观察性非政策审计下，我们受过训练的LLM实现了约是最强商业基线（0.0228美元对0.0104美元）估计毛利提升的两倍，同时拥有所有评估政策中最低的下行率和最小的负尾风险。值得注意的是，这两种评估对基线的排名并不完全相同：未训练的基础模型在法官评分标准中排名最后，但在因果审计中排第二，表明审计捕捉到了评委未捕捉到的信号。我们的结果表明，带有金融奖励信号的GRPO能比商业LLM产生更有价值的商业建议，且无判判因果审计是金融NLP中LLM作为评判评估的宝贵补充，而非确认。

HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion

HarmoniDPO：通过偏好优化扩散实现视频引导音频生成

• Authors: Wenshuo Peng, Kaipeng Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.11913
• Pdf link: https://arxiv.org/pdf/2608.11913
• Abstract Video-to-audio (V2A) generation faces significant challenges in achieving precise temporal synchronization and high perceptual quality due to the complex, ambiguous relationship between visual and auditory cues. Existing methods typically compress video inputs into single feature representations, leading to significant loss of temporal dynamics and fine-grained visual information. These approaches also rely on reconstruction-based training objectives that poorly correlate with human perceptual judgments of audio quality and appropriateness. We propose HarmoniDPO, a novel framework that integrates preference-based optimization into diffusion-based V2A generation to address these limitations. (1) Our approach leverages a dual video representation: combining global context with frame-wise features to preserve temporal dynamics and semantic detail. (2) Inspired by reinforcement learning from human feedback (RLHF), HarmoniDPO employs online Direct Preference Optimization (online-DPO) to fine-tune a diffusion-based V2A model from preference judgments, enhancing perceptual quality and alignment. (3) Additionally, we introduce Dual-scale Diffusion Search (DDS), a test time scaling algorithm that adaptively optimizes output fidelity during inference. Experiments demonstrate that HarmoniDPO outperforms state-of-the-art methods in audio-video synchronization and subjective audio quality, offering a robust solution for generating realistic, human-preferred audio from video.
• 中文摘要 由于视觉与听觉线索之间复杂且模糊的关系，视频到音频（V2A）生成在实现精确的时间同步和高感知质量方面面临重大挑战。现有方法通常将视频输入压缩为单一特征表示，导致时间动态和细粒度视觉信息的显著损失。这些方法还依赖于基于重建的训练目标，但这些目标与人类对音频质量和适当性的感知判断关联较差。我们提出了HarmoniDPO，一种将基于偏好的优化整合到基于扩散的V2A生成中以解决这些限制的新框架。（1）我们的方法采用双重视频表示：将全局上下文与逐帧特征结合，以保持时间动态和语义细节。（2）受人类反馈强化学习（RLHF）启发，HarmoniDPO采用在线直接偏好优化（online-DPO），从偏好判断中微调基于扩散的V2A模型，提升感知质量和对齐性。（3）此外，我们引入了双尺度扩散搜索（DDS），这是一种测试时间尺度算法，在推断过程中自适应优化输出保真度。实验表明，HarmoniDPO在音视频同步和主观音频质量方面优于最先进的方法，提供了从视频生成真实且受人类偏好音频的稳健解决方案。

LODESTAR: Trustworthy Entropy Is Navigated, Not Merely Measured -- Reinforced Polarizer Keeps a Frozen LLM from Being Confidently Misled by the Wrong Evidence

LODESTAR：可信熵是被导航的，而不仅仅是测量——强化极化仪防止冻结的大型语言模型被错误的证据自信地误导

• Authors: Po-Jen Ko, Che-Cheng Wu, Hung-Chun Hsu, Li-Yang Chang, Chuan-Ju Wang
• Subjects: Subjects: Computation and Language (cs.CL); Information Retrieval (cs.IR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.11922
• Pdf link: https://arxiv.org/pdf/2608.11922
• Abstract Predictive-distribution entropy makes a strong selection rule in retrieval-augmented question answering: across five QA benchmarks, keeping the candidate answer that a frozen respondent LLM produces with the lowest answer-token entropy lifts mean answer $F_1$ from 0.4769 to 0.5148 over the retriever's top-ranked passage, with no gold answers. Yet this lowest-entropy rule, which prior entropy-based selectors adopt, fails in a specific and consequential way: a misleading passage makes the respondent confidently wrong, driving its entropy down precisely where the signal looks most trustworthy. We show that the failure comes from the passage the respondent reads -- and the context that passage is read in is an input we can intervene on. We introduce LODESTAR, to our knowledge the first method to score a text intervention by the uncertainty it induces in a third-party frozen respondent, compared across one question's candidates. LODESTAR uses reinforcement learning to train, once and offline, a polarizer -- a short fixed natural-language string inserted into the respondent's prompt and never into its weights; its training labels are built offline from gold answers and two LLM judges, and inference reads neither. Evaluating every competing selector under the same frozen respondent and the same candidate pools on 5,008 questions, LODESTAR attains the highest mean $F_1$ of any inference-ready selector (0.5148 to 0.5339), the highest exact match (0.4136), and the highest GPT-4o judge score of the frozen-respondent configurations judged (0.6435); its three-seed mean wins all 70 method-by-dataset $F_1$ cells against fourteen published configurations while remaining paired-significant against every one. The gain holds both in-domain and out-of-domain, and ablating the polarizer shows it is what makes the respondent read a misleading passage less often (26.0% against 30.3%).
• 中文摘要 预测分布熵在检索增强问答中形成了一个强的选择规则：在五个QA基准测试中，保持一个被冷藏的LLM中最低答案标记熵的候选答案，使检索者最高排名文章的平均答案$F_1$从0.4769提升到0.5148，且无金答案。然而，这一基于熵的选择者采用的最低熵规则在特定且严重的情况下失效：误导性段落使受访者自信地错误，将熵准确地压低在信号最可信的位置。我们证明失败来自受访者所阅读的文章——而该文章所处的语境是我们可以介入的输入。我们引入了LODESTAR，这是据我们所知，这是首个通过文本干预对第三方冻结受访者造成的不确定性进行评分的方法，比较同一问题候选人之间的差异。LODESTAR通过强化学习一次性且离线训练极化器——一种短而固定的自然语言字符串，插入到受试者提示中，从未插入权重中;其训练标签由Gold Answers和两个LLM评判离线构建，推理结果两者都不读。在5008个问题上，LODESTAR对同一冻结受访者和同一候选人池下的所有竞争选择者进行了评估，获得了任何推理准备选择器中最高的平均分$F_1美元（0.5148至0.5339）、最高的精确匹配（0.4136）以及冻结受访者配置中最高的GPT-4o评判得分（0.6435）;其三种子均值在14个已发布配置中赢得全部70个按数据集计1$F_1$单元格，同时对所有配置保持配对显著性。增益在域内和域外都存在，消极极化显示这使受访者阅读误导性段落的频率降低（26.0%对30.3%）。

LoongReflect: Boosting Long-Horizon Reflection in Search Agents via Global Perspective Distillation

LoongReflect：通过全球视角提炼提升搜索代理的长远视野反射

• Authors: Zhixin Zhang, Xinke Jiang, Zhibang Yang, Weixuan Xu, Guohong Qiu, Xu Chu, Junfeng Zhao, Yasha Wang
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11967
• Pdf link: https://arxiv.org/pdf/2608.11967
• Abstract Large language model agents increasingly rely on long-horizon reasoning to solve complex tasks involving planning, tool use, and memory. A critical capability in such settings is reflection: assessing trajectory progress, identifying missing evidence and unreliable intermediate states, and deciding whether to continue, revise, or abandon the current branch. Learning effective reflection, however, is challenging because reflection is performed locally within the current branch, whereas its utility can only be determined by its contribution to the final trajectory outcome. This local-global mismatch makes outcome-based reinforcement learning provide only local, sparse and delayed supervision for reflective decisions. To solve these, we propose LoongReflect, a training framework that formulates reflection as a memory-control policy. The agent operates over a reversible trajectory tree using explicit reflect and backtrack actions. Reflection consolidates verified facts, missing evidence, and branch-specific risks into working memory, while backtracking removes an unreliable branch from the active context and preserves a concise corrective lesson. To learn this policy, LoongReflect combines two complementary signals through a look-ahead, extragradient-style coordination mechanism. A fast channel distills globally informed reflective behavior from a privileged teacher, with supervision restricted to reflection and backtracking tokens. A slow channel optimizes complete trajectories using outcome-based GRPO, aligning local control decisions with final task success. Experiments on multi-hop retrieval-augmented generation and mathematical reasoning benchmarks demonstrate consistent improvements over outcome-only reinforcement learning and self-distillation baselines.
• 中文摘要 大型语言模型代理越来越依赖长远视野推理来解决涉及规划、工具使用和记忆的复杂任务。在此类环境中，反思能力是关键：评估发展轨迹进展，识别缺失的证据和不可靠的中间状态，并决定是继续、修订还是放弃当前分支。然而，学习有效反思具有挑战性，因为反思是在当前分支的局部进行，而其效用只能通过对最终轨迹结果的贡献来判断。这种局部-全局不匹配使得基于结果的强化学习只能提供局部、稀疏且延迟的反思性决策监督。为解决这些问题，我们提出了LoongReflect，一个将反思制定为记忆控制策略的训练框架。该代理通过显式反射和回溯操作，在可逆轨迹树上运行。反思将已验证的事实、缺失的证据和分支特定风险整合进工作记忆，而回溯则将不可靠的分支从主动语境中移除，保留简明的纠正教训。为了学习该策略，LoongReflect通过前瞻、超梯度式协调机制将两个互补信号结合起来。快速通道提炼出来自特权教师的全球知情反思行为，监督仅限于反思和回溯标记。慢速通道利用基于结果的GRPO优化完整轨迹，使局部控制决策与最终任务成功率保持一致。多跳检索增强生成和数学推理基准测试显示，仅基于结果的强化学习和自我提炼基线在持续改进。

Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection

重试、切换，还是弃权？通过受控错误注入学习战略感知工具使用策略

• Authors: Chaoran Chen, Vy Nguyen, Ziji Zhang, Abhinav Gullapalli, Ziyi Wang, Yuxuan Lu, Dakuo Wang, Jing Huang, Zhou Yu, Jin Lai
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.11977
• Pdf link: https://arxiv.org/pdf/2608.11977
• Abstract Tool-using LLM agents are commonly trained and evaluated in environments where tool calls succeed reliably, yet deployed tools can fail transiently, persistently, or silently. Robust recovery therefore requires more than repeated retries: an agent may need to retry the same path, switch to an alternative, or recognize that no viable path remains. We present BENCH2ROBUST, a framework that converts failure-free tool-use benchmarks into controlled stochastic environments with scenario-controlled solvability, where episodes explicitly require retrying, switching, or stopping after available paths are exhausted. We use BENCH2ROBUST to study two complementary interventions: structured runtime recovery context through Bayesian Tool Memory (BTM), and curriculum-controlled reinforcement learning. Across 7 models from 4 families and two multi-turn benchmark families, tool failures produce a near-universal robustness gap. On held-out Retail tasks, BTM improves robustness by up to 16.8 percentage points without retraining, while RL learns complementary recovery behavior that remains beneficial without inference-time BTM. Combining the two reaches 40.8-45.5% under injection while preserving failure-free performance. These results suggest that robust tool use benefits from combining environment-specific recovery knowledge with learned recovery behavior.
• 中文摘要 使用工具的LLM代理通常在工具调用可靠成功但部署工具可能暂时、持续或无声地失败的环境中进行训练和评估。因此，稳健恢复不仅需要重复重试：代理可能需要重试同一路径、切换到替代路径，或识别已无可行路径。我们提出了BENCH2ROBUST，这是一个将无故障工具使用基准转换为具备场景可控解性的受控随机环境的框架，在可用路径用尽后，发作明确要求重试、切换或停止。我们利用BENCH2ROBUST研究两种互补干预：通过贝叶斯工具记忆（BTM）实现结构化运行时恢复语境，以及课程控制的强化学习。在4个系列的7个模型和2个多匝基准系列中，工具故障几乎形成了普遍的鲁棒性差距。在未完成的零售任务中，BTM在不重新训练的情况下可提升最多16.8个百分点的鲁棒性，而强化学习的补充恢复行为则在不依赖推断时间BTM的情况下依然有益。两者结合后，注入时可达40.8%至45.5%，同时保持无故障性能。这些结果表明，稳健工具的使用受益于将环境特定的恢复知识与学习到的恢复行为结合起来。

Token-Level Credit Assignment Optimization for Generative Document Retrieval

生成式文档检索的令牌级信用分配优化

• Authors: Xinpeng Zhao, Yang Liu, Ran Chen, Xinyu Ma, Daiting Shi, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Xin Xin
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2608.12049
• Pdf link: https://arxiv.org/pdf/2608.12049
• Abstract Generative retrieval models perform document retrieval by autoregressively generating document identifiers (DocIDs). This process naturally forms a sequential decision problem, where each decoding step selects a DocID token and the complete token sequence determines the retrieved document. However, retrieval effectiveness is typically evaluated only after the full DocID is generated, creating a mismatch between token-level generation and document-level relevance supervision. As a result, existing reinforcement learning methods for generative retrieval mostly rely on sequence-level rewards, where the same document-level feedback is propagated to all decoding steps. Such coarse-grained feedback makes it difficult to identify which token decisions are responsible for successful or failed retrieval. In this work, we propose a fine-grained reinforcement learning framework for generative retrieval with token-level relevance rewards. Instead of assigning a single reward to the entire generated DocID, we estimate step-wise rewards by measuring how each token decision changes the expected retrieval quality of the corresponding generation trajectory. This enables more precise credit assignment and encourages the policy to favor token decisions that contribute more directly to document-level relevance. We further develop practical reward estimation strategies tailored to the DocID generation process and incorporate them into a policy optimization framework. Experiments on retrieval benchmarks show that our method consistently outperforms sequence-level reward baselines, demonstrating the effectiveness of fine-grained supervision for aligning autoregressive DocID generation with retrieval objectives.
• 中文摘要 生成检索模型通过自回归生成文档标识符（DocID）来实现文档检索。这一过程自然形成了一个顺序判定问题，每个解码步骤都选择一个DocID令牌，完整的令牌序列决定检索到的文档。然而，检索效果通常只有在完整 DocID 生成后才评估，导致令牌级生成与文档级相关性监督之间存在不匹配。因此，现有的生成式检索强化学习方法大多依赖序列级奖励，即相同的文档级反馈传播到所有解码步骤。这种粗粒度反馈使得识别哪些代币决策导致检索成功或失败变得困难。本研究提出一个细粒度强化学习框架，用于生成式检索，并赋予代币级相关性奖励。我们不再为整个生成的DocID分配单一奖励，而是通过测量每个代币决策如何改变相应生成轨迹的预期检索质量来估算分阶段奖励。这使得信用分配更加精确，并鼓励政策优先考虑更直接促进文档层面相关性的关键决策。我们还进一步开发了针对DocID生成过程的实用奖励估算策略，并将其纳入政策优化框架中。检索基准测试的实验显示，我们的方法始终优于序列级奖励基线，证明了细粒度监督在自回归DocID生成与检索目标匹配上的有效性。

Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL

从SMPC演示中学习机车操作，且离线到在线的强化学习稀疏

• Authors: Martin Schuck, Maks Sorokin, Simone Manni, Duy Ta, Angela P. Schoellig, Marco Hutter, Simon Le Cleac'H, Jan Brüdigam
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.12063
• Pdf link: https://arxiv.org/pdf/2608.12063
• Abstract Integrating locomotion and manipulation is essential for robot autonomy, but scaling standard Reinforcement Learning (RL) to complex tasks is severely bottlenecked by the slow, manual process of dense reward shaping. To bypass this limitation, we leverage Sample-based Model Predictive Control (SMPC) entirely in simulation as an automated, rapidly tunable expert to generate massive offline datasets. Because this data solves the fundamental exploration problem, we can train an off-policy RL agent using purely sparse task rewards, drastically reducing the time required to learn new skills and eliminating the need for manual tuning. Integrating this high-level agent with a low-level dynamic stability controller yields more optimal behaviors that strictly align with true task objectives, ultimately allowing the learned policies to surpass the original optimal control teacher. We validate the robustness of this sim-to-real framework by successfully deploying complex loco-manipulation skills across different morphologies, including an arm-equipped Spot quadruped and a G1 humanoid.
• 中文摘要 将运动和操作整合对于机器人自主性至关重要，但将标准强化学习（RL）扩展到复杂任务时，由于缓慢且手动的密集奖励塑造过程，严重受限。为了绕过这一限制，我们完全利用基于样本的模型预测控制（SMPC）作为自动化、快速可调的专家，生成海量离线数据集。由于这些数据解决了根本的探索问题，我们可以用纯稀疏的任务奖励来训练一个非策略的强化学习代理，大大缩短学习新技能所需的时间，并消除了手动调优的需求。将该高级代理与低级动态稳定性控制器集成，可获得更优行为，严格符合真实任务目标，最终使所学策略超越原始最优控制教师。我们通过成功部署复杂的机车操作技能，验证了该模拟现实框架的稳健性，涵盖不同形态，包括配备手臂的斑点四足动物和G1类人生物。

RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning

RoutePack：专家配置与注意力感知数据打包，用于 MoE 强化学习

• Authors: Yibo Shen, Xudong Han, Xiaowei Zhu, Gen Li, Zhenxuan Pan
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC)
• Arxiv link: https://arxiv.org/abs/2608.12146
• Pdf link: https://arxiv.org/pdf/2608.12146
• Abstract Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expert work on expert-parallel ranks. Optimizing either alone can shift the bottleneck to the other. In MoE RL, rollout-time routing replay exposes every sample's sequence length and layer-wise expert demand before its training step. We present RoutePack, a hierarchical planner that coordinates state-consistent, layer-wise expert rerouting with joint attention- and expert-aware data packing over an optimizer-step window. RoutePack first places experts independently at each MoE layer using aggregate routing demand. It then packs samples into the smallest certified, or best-known feasible, number of token-capped execution rows and optimizes their DP layout with a projected EDP-shard-aware objective. The objective combines a window-normalized linear-quadratic attention proxy with per-layer physical EP-rank peaks and minimizes the accumulated cost of the slowest EDP shard. Parallel population annealing searches fixed-row feasible layouts while preserving sample coverage, capacity, nonempty cells, equal microbatch counts, and communicator topology. State-consistent materialization preserves logical top-k routing and existing MoE kernels without microbatch-level expert replication. Across Ling-3.0-Tiny and Ling-3.0-Flash, expert rerouting improves mean trainer-measured token throughput by 3.80% and 10.50%, while routing-aware packing adds another 4.86% and 3.98%, respectively. Overall, RoutePack improves throughput by 8.85% and 14.89% over the baseline.
• 中文摘要 强化学习（RL）的专家混合训练（Training Mixture-of-Experts，MoE）模型结合了两个负载均衡问题：序列组合决定每个数据并行微批次中的密集注意力工作，而令牌路由决定专家并行队列中的专家工作稀疏。仅仅优化其中一方，就能将瓶颈转移到另一方。在 MoE RL 中，部署时间路由重放在训练步骤前会暴露每个样本的序列长度和层级专家需求。我们介绍RoutePack，一款分层规划器，通过优化器步骤窗口协调状态一致的层级专家重定向与注意力和专家感知数据打包。RoutePack 首先在每个 MoE 层独立放置专家，使用汇总路由需求。然后，它将样本打包到最小的认证或最已知的可行数量的标记上限执行行中，并用预测的EDP分片相关目标优化其DP布局。该目标结合了窗口归一化的线性二次注意力代理与每层物理EP等级峰值，最小化最慢EDP碎片的累计成本。并行粒子退火搜索在保持样本覆盖率、容量、非空单元、等量微批次计数和通信拓扑的同时，实现固定行可行布局。状态一致的物质化保留了逻辑的顶K次路由和现有的MoE内核，无需微批次级专家复制。在Ling-3.0-Tiny和Ling-3.0-Flash中，专家重路由使平均训练器测量的令牌吞吐量提升了3.80%和10.50%，而路由感知打包分别增加了4.86%和3.98%。总体来看，RoutePack的吞吐量提升了8.85%，比基线提升了14.89%。

Machine Learning-Based Cyber Defense for Cloud Infrastructure: An Adaptive Deep Q-Network Architecture for Intelligent Intrusion Detection and Automated Threat Mitigation

基于机器学习的云基础设施网络防御：一种自适应深度Q网络架构，用于智能入侵检测和自动化威胁缓解

• Authors: Md Yassir Mottalib, Md Yousuf, Eklachur Rahman Bhuiyan, S M Ahsan Habib, Sonjoy Kumar Dey, Md. Salahuddin Gazi, Molay Kumar Roy, Asaduzzaman Anik
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.12190
• Pdf link: https://arxiv.org/pdf/2608.12190
• Abstract With the increasing complexity of cyber assaults in cloud environments, adaptable security solutions are needed that can support real-time detection and autonomous response. In this paper, we propose a reinforcement learning-based dynamic cyber defense framework. We deploy a Deep Q-Network (DQN) to train effective defensive strategies to counteract the evolving cyberattacks. We leverage the CICIDS2017 dataset for model creation and the UNSW-NB15 dataset for external validation, involving preprocessing of data, feature engineering, and adaptive policy learning. We compare the proposed DQN with decision tree, support vector machine, random forest, XGBoost, and multilayer perceptron models. The proposed DQN achieves an accuracy of 99.72%, a precision of 99.68%, a recall of 99.65%, an F1-score of 99.66%, and an ROC-AUC of 0.999, while the false positive rate is 0.31%, the false negative rate is 0.35%, and the detection latency is 15 ms. The framework achieved 99.54% attack mitigation rate, demonstrating strong adaptive and real-time defensive capabilities. These results demonstrate the potential of reinforcement learning as a powerful and scalable approach for autonomous cybersecurity in modern cloud environments.
• 中文摘要 随着云环境中网络攻击日益复杂，需要能够支持实时检测和自主响应的灵活安全解决方案。本文提出一种基于强化学习的动态网络防御框架。我们部署深度Q网络（DQN），训练有效的防御策略，以应对不断演变的网络攻击。我们利用CICIDS2017数据集创建模型，利用UNSW-NB15数据集进行外部验证，包括数据预处理、特征工程和自适应策略学习。我们将提出的DQN与决策树、支持向量机、随机森林、XGBoost和多层感知器模型进行比较。拟议的DQN准确率为99.72%，精度99.68%，召回率99.65%，F1评分99.66%，ROC-AUC为0.999，假阳性率为0.31%，假阴性率为0.35%，检测延迟为15毫秒。该框架实现了99.54%的攻击缓解率，展现了强大的自适应和实时防御能力。这些结果展示了强化学习作为现代云环境中自主网络安全的强大且可扩展方法的潜力。

SCOUT: Unlocking Enhanced Spatial Reasoning via Structured Chain-of-Thought and Multi-Objective Process Reward

侦察兵：通过结构化思维链和多目标过程奖励解锁增强的空间推理能力

• Authors: Zile Zhou, Huining Yuan, Weichen Zhang, Xinlei Chen, Xiao-ping Zhang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.12220
• Pdf link: https://arxiv.org/pdf/2608.12220
• Abstract Existing Vision-Language Models (VLMs) exhibits a critical bottleneck in robust spatial reasoning. Recent reinforcement learning (RL) methods aim to close this gap with verifiable outcomes, yet they suffer from poor credit assignment across intermediate reasoning steps. Concurrently, structured reasoning approaches overlook the critical depth perception necessary for comprehensive 3D understanding. To address these challenges, we propose SCOUT (Structured Chain-Of-Thought Utilizing Process-Supervised RL Training). Specifically, we design a structured Chain-of-Thought (CoT) framework that explicitly models 3D environmental perception to ensure robust spatial understanding and reasoning. Furthermore, we introduce a novel RL algorithm featuring multi-objective process rewards and a tailored advantage estimation method, facilitating fine-grained credit assignment across distinct segments of the reasoning trajectory. To support our framework, we develop SCOUT-24k, a structured spatial reasoning CoT dataset synthesized through a customized pipeline. Extensive evaluations demonstrate that SCOUT-3B improves upon baseline models by 16.85% and 6.3% on general spatial benchmarks and complex spatial reasoning tasks respectively. Notably, our larger SCOUT-7B even outperforms GPT-4o by a margin of 4.28%. Moreover, despite being trained exclusively on single image, SCOUT-7B exhibits robust out-of-domain generalization to multi-image and video scenarios. These empirical results render SCOUT as a critical step towards next generation of spatially-aware VLMs.
• 中文摘要 现有的视觉语言模型（VLMs）在稳健空间推理中存在关键瓶颈。最新的强化学习（RL）方法旨在通过可验证的结果弥合这一差距，但它们在中级推理步骤中的学分分配仍然不佳。同时，结构化推理方法忽视了全面三维理解所需的关键深度感知。为应对这些挑战，我们提出了SCOUT（结构化思维链，利用过程监督的强化学习训练）。具体来说，我们设计了一个结构化的思维链（CoT）框架，明确建模三维环境感知，以确保空间的稳健理解和推理。此外，我们引入了一种新颖的强化学习算法，具备多目标过程奖励和定制化优势估计方法，促进了在推理轨迹不同阶段的细粒度信用分配。为支持我们的框架，我们开发了SCOUT-24k，这是一个通过定制流水线综合的结构化空间推理CoT数据集。大量评估表明，SCOUT-3B在一般空间基准和复杂空间推理任务中分别比基线模型提升16.85%和6.3%。值得注意的是，我们更大的SCOUT-7B甚至比GPT-4o多出4.28%。此外，尽管仅训练单图像，SCOUT-7B仍表现出对多图像和视频场景的强劲域外泛化能力。这些实证结果使SCOUT成为迈向下一代空间感知VLM的关键一步。

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

一个冻结的模拟器不够：多智能体强化学习中的模拟器崩溃

• Authors: Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu, Derek Chong, Abe Hou, Dilara Soylu, Sergey Levine, Christopher D. Manning, Weiyan Shi
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.12253
• Pdf link: https://arxiv.org/pdf/2608.12253
• Abstract Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $\tau^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.
• 中文摘要 人机交互的多智能体强化学习通常依赖单一大型语言模型来模拟用户行为。我们证明该方法系统性地无法推广并追踪模拟器崩溃失败的根源：由于模拟器LLM是模式崩溃的，针对它训练的LLM策略会过度拟合到利用模拟器主导模式的狭窄策略，而这种策略对未被看见的模拟器和真实用户的迁移效果不佳。我们将这种坍缩形式化，并提出两种互补的解，分别在推理时和训练时。推理时间解法——口头化采样，通过从口述化响应分布中抽样，拓宽了模拟器的行为，从而减少了模态崩溃。训练时间解决方案“协同训练”（Co-Training）共同优化该策略，针对一组可训练模拟器，防止其过度拟合到任何单一模拟器的模式。我们在三个多回合基准测试上验证了这两种方案：Persuasion for Good、$\tau^2$-bench 和 CooperBench。口头抽样比单一模拟器强化学习提升预期成功率高达9%，协同训练则将提升幅度进一步提升至14%;人类研究显示真实用户也有类似的提升。这两种方案都保留了单模拟器强化学习下崩溃的政策多样性。为支持这一方向的进一步工作，我们发布了SCOPE，一个开源的群体共训练多智能体强化学习框架。更广泛地说，我们的结果表明，训练环境的多样性，而不仅仅是政策，对于多回合强化学习推广到现实世界部署至关重要。

SelectLight: Learning to Select Signal Plans Generated by Distributed Model Predictive Control for Urban Traffic Networks

SelectLight：学习选择由分布式模型预测控制生成的城市交通网络信号计划

• Authors: Lyuzhou Luo, Chaopeng Tan, Zhengyong Gao, Hong Zhu, Andrea D'Ariano, Keshuang Tang
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.12256
• Pdf link: https://arxiv.org/pdf/2608.12256
• Abstract Coordinated traffic signal control across urban networks must adapt to changing demand while satisfying operational constraints. Multi-objective distributed model predictive control (DMPC) can construct feasible signal plans online, but prescribed rules for selecting among trade-off solutions cannot learn from realized closed-loop outcomes. We propose SelectLight, which implements post-optimization selection by allowing a multi-agent reinforcement learning (MARL) policy to choose directly from plans generated online by DMPC. At each control update, state-pruned multi-objective dynamic programming (SP-MODP) evaluates plans with a Newellian point--spatial queue model and returns a bounded set of mutually nondominated candidate signal plans for total queueing delay, peak queue accumulation, and total number of stops. A topology-aware attention policy trained with independent proximal policy optimization (IPPO) selects one unmodified plan from each variable-size set. This confines learning to candidate selection, preserves the prescribed signal timing constraints, and leaves the selected plan and its predicted objective trade-offs available for inspection. Experiments on two 28-intersection SUMO networks show that SelectLight achieves the best delay-related performance and that its advantage widens with demand. At twice the baseline demand, it reduces queueing delay and waiting time by 5.57% and 6.44%, respectively, relative to the strongest baseline. SelectLight also incurs the lowest transfer loss under every tested demand shift. With a 120 s prediction horizon, the per-intersection 99th-percentile SP-MODP solution time is 5.408 ms, well below the 5 s control interval.
• 中文摘要 跨城市网络的协调交通信号控制必须适应需求变化，同时满足运营限制。多目标分布式模型预测控制（DMPC）可以在线构建可行的信号计划，但用于在权衡方案中选择的规定规则无法从实现的闭环结果中学习。我们提出了SelectLight，它通过允许多智能体强化学习（MARL）策略直接从DMPC在线生成的计划中选择，实现优化后的选择。每次控制更新时，状态修剪多目标动态规划（SP-MODP）会用纽埃尔点空间队列模型评估计划，并返回一组相互非支配的候选信号计划，涵盖总排队延迟、峰值队列累积和总停靠次数。一个通过独立近端策略优化（IPPO）训练的拓扑感知注意力策略，从每个变量大小集合中选择一个未修改的计划。这将学习限制在候选人选择中，保持规定的信号时序约束，并保留所选方案及其预测的客观权衡供检阅。在两个28交点的SUMO网络上的实验表明，SelectLight在延迟相关性能方面表现最佳，且其优势随着需求而扩大。该方案是基线需求的两倍，排队延迟和等待时间分别减少了5.57%和6.44%，相较于最强基线。SelectLight 在每次测试需求转移下都能承受最低的传输损耗。在120秒的预测视野下，每个交点第99百分位的SP-MODP解算时间为5.408毫秒，远低于5秒的控制区间。

Keyword: diffusion policy

There is no result