生成时间: 2026-09-03 20:39:36 (UTC+8); Arxiv 发布时间: 2026-09-03 20:00 EDT (2026-09-04 08:00 UTC+8)

今天共有 22 篇相关文章

Keyword: reinforcement learning

WMLLM: Self-Evolving Optimization Agents via Predict-Then-Act World Modeling

WMLLM：通过预测然后行动世界建模实现自我演化优化代理

• Authors: Zhongzheng Li, Qingsong Ran, Shikun Feng, Nian Ran, Wenhao Li, Xiaoyuan Zhang, Yue Wang, Xiaoguang Zhao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01608
• Pdf link: https://arxiv.org/pdf/2609.01608
• Abstract Black-box optimization problems remain challenging because of large, weakly structured, and high-dimensional search spaces. Existing methods often suffer from poor sample efficiency because they rely on direct candidate generation or trial-and-error refinement. A natural way to improve search efficiency is to use world modeling, which can help identify promising optimization directions before costly evaluation. Large language models can predict the outcomes of these candidates with nontrivial accuracy because of their implicit knowledge. Motivated by this observation, we propose WMLLM, a self-evolving optimization-agent framework based on predict-then-act world modeling. The agent first predicts promising directions and then acts to generate candidates. Combined with agentic multi-turn refinement, population-based search, and reinforcement learning, WMLLM refines both its implicit world model and its optimization strategy during search. Experiments on black-box optimization tasks, especially multi-objective molecular optimization, show that WMLLM improves sample efficiency and final optimization performance. On the multi-objective molecular optimization benchmark, WMLLM achieves state-of-the-art results under a limited evaluation budget.
• 中文摘要 黑箱优化问题依然具有挑战性，因为存在庞大、结构薄弱且高维的搜索空间。现有方法常因依赖直接候选生成或试错细化而样本效率较低。提升搜索效率的自然方法是使用世界建模，这有助于在昂贵的评估前识别出有前景的优化方向。大型语言模型因其隐含知识，能够以非平凡的精度预测候选结果。基于这一观察，我们提出了WMLLM，一种基于预测-然后行动世界建模的自我演化优化-代理框架。代理首先预测有前景的方向，然后采取行动生成候选对象。结合智能多回合细化、基于群体的搜索和强化学习，WMLLM在搜索过程中不断完善其隐式世界模型和优化策略。黑箱优化任务，尤其是多目标分子优化的实验表明，WMLLM提升了样品效率和最终优化性能。在多目标分子优化基准测试中，WMLLM在有限的评估预算下实现了最先进的成果。

DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Driving

DiDrive：一个风险感知的分层扩散框架，用于自动驾驶中安全离线强化学习

• Authors: Qisong Guo, Jingtang Chen, Zhilin Chen, Pei Xu, Mingjian Fu, Wenxi Liu, Yuanlong Yu
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.01609
• Pdf link: https://arxiv.org/pdf/2609.01609
• Abstract While diffusion models effectively capture multimodal behavioral priors for autonomous driving, offline reinforcement learning (RL) policies remain susceptible to distribution shift, heavy-tailed risk signals, out-of-distribution (OOD) action generation, and high-dimensional state redundancy. To address these challenges, we propose DiDrive, a distribution-guided offline diffusion framework featuring two synergistic components: the Risk-Aware Hierarchical Diffusion (RHDif) architecture and the 3DICE policy optimization paradigm. In the state space, RHDif utilizes a low-level risk-gated encoder and a high-level contextual modulator to filter environmental redundancy and focus on safety-critical threats. In the action space, 3DICE mitigates OOD overestimation and gradient oscillation through in-sample calibrated guidance, spatiotemporal optimization, and ensemble-based candidate ranking. Evaluations on the CARLA benchmark demonstrate DiDrive's superiority over baselines like IQL, CQL, and Diffusion-QL, particularly in complex, high-density traffic scenarios with 60 vehicles, where it achieves an 85% success rate and a 4295.68 average reward, providing a robust pathway for safe autonomous driving decision-making.
• 中文摘要 虽然扩散模型有效捕捉了自动驾驶的多模态行为先验，但离线强化学习（RL）策略仍易受分布偏移、重尾风险信号、分布外（OOD）动作生成和高维状态冗余的影响。为应对这些挑战，我们提出了DiDrive，一种分布引导离线扩散框架，具备两个协同组件：风险感知层级扩散（RHDif）架构和3DICE策略优化范式。在状态空间，RHDif利用低阶风险门控编码器和高级上下文调制器过滤环境冗余，聚焦安全关键威胁。在动作领域，3DICE通过样本内校准指导、时空优化和基于集合的候选排名，减轻OOD高估和梯度振荡。CARLA基准测试的评估显示，DiDrive优于IQL、CQL和Diffusion-QL等基线，尤其是在60辆车辆的复杂高密度交通场景下，成功率达85%，平均奖励4295.68，为安全自动驾驶决策提供了坚实的路径。

PRISM: An Agentic Multi-Model Architecture for Proactive Safety in Autonomous Transportation Systems

PRISM：一种用于自动驾驶交通系统中主动安全的代理多模型架构

• Authors: Joyjit Roy, Samaresh Kumar Singh, Sushanta Das
• Subjects: Subjects: Multiagent Systems (cs.MA); Computer Vision and Pattern Recognition (cs.CV); Emerging Technologies (cs.ET); Machine Learning (cs.LG); Physics and Society (physics.soc-ph)
• Arxiv link: https://arxiv.org/abs/2609.01623
• Pdf link: https://arxiv.org/pdf/2609.01623
• Abstract Autonomous and intelligent transportation systems operate in complex urban environments where safety depends on interactions among vehicle behavior, environmental conditions, and vulnerable road users (VRUs) such as pedestrians and cyclists. Most advanced driver assistance systems (ADAS) employ reactive mechanisms that activate only after hazards have emerged, a critical limitation underscored by rising VRU fatalities in the United States. This study introduces PRISM (Proactive Risk Intelligence and Safety Management), an agentic multi-model safety architecture that transitions from reactive crash avoidance to proactive, continuous risk management. PRISM employs inverse crash-probability modeling to convert binary crash classifiers into dynamic, interpretable safety scores. Three specialized models addressing trajectory kinematics, environmental risk, and VRU interaction operate concurrently, coordinated by a reasoning layer incorporating reinforcement learning, contextual memory, and feature-level attribution. The system provides graduated safety interventions across four tiers, from silent monitoring to emergency alerts. Unlike rule-based systems with static thresholds, PRISM dynamically adjusts safety parameters in real time. Validated across 1,296 scenarios from three naturalistic driving datasets without dataset-specific retraining, the system yielded a mean safety score of 68 out of 100, classified 77.6% of scenarios as advisory, and flagged a near-miss rate of 3.8%, with 11% of scenarios escalating to intervention or emergency response. Feature attribution consistently identified trajectory risk and VRU proximity as primary safety factors. PRISM provides a unified, interpretable framework for proactive transportation safety with emphasis on VRU risk reduction in dense urban environments.
• 中文摘要 自动驾驶和智能交通系统运行于复杂的城市环境中，安全依赖于车辆行为、环境条件与易受影响道路使用者（VRU）如行人和骑行者之间的互动。大多数先进驾驶辅助系统（ADAS）采用反应机制，仅在危险出现后激活，这一关键局限性因美国VRU死亡率上升而凸显。本研究引入了PRISM（主动风险智能与安全管理），这是一种代理多模型安全架构，从被动碰撞避免转向主动、持续风险管理。PRISM采用逆碰撞概率建模将二元碰撞分类器转换为动态、可解释的安全评分。三个专门模型同时运行，分别涉及轨迹运动学、环境风险和VRU交互，由包含强化学习、上下文记忆和特征级归因的推理层协调。该系统在四个层级提供分级的安全干预，从静默监控到紧急警报。与基于规则的静态阈值系统不同，PRISM实时动态调整安全参数。系统在三个自然驾驶数据集中验证了1296个场景，且无数据集特定重训，平均安全得分为68分（满分100分），77.6%的情景被归类为咨询级，近乎发生率为3.8%，其中11%的情景升级为干预或应急响应。特征归因始终将轨迹风险和VRU接近度视为主要安全因素。PRISM提供了一个统一且可解释的主动交通安全框架，重点是降低在密集城市环境中的VRU风险。

Sim2Signal: Sim-to-Real Benchmarks for Traffic Signal Control

Sim2Signal：交通信号控制的模拟到现实基准测试

• Authors: Ferdous Al Rafi, Susrik Mukherjee, Latika Liladhar Dekate, Jennifer Yawa Lavoe, Huaiyuan Yao, Shlok Mohanty, Longchao Da, Xuesong Zhou, Hua Wei
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.01676
• Pdf link: https://arxiv.org/pdf/2609.01676
• Abstract Reinforcement learning achieves strong traffic signal control performance in simulation, yet policies trained in simulators often fail once deployed in the real world, a failure known as the Sim-to-Real gap. When RL is applied to traffic signal control, this gap arises from several sources: sensing, action execution, traffic dynamics, and the control objective. Their relative impact and the reliability of existing Sim-to-Real mitigation methods remain insufficiently understood, and the field lacks a standard benchmark for systematically measuring the gap and evaluating mitigation methods. We present Sim2Signal, a benchmark that decomposes the Sim-to-Real gap into observation, action, transition, and reward gaps, corresponding to mismatches in the four components of the underlying MDP, and induces each gap in isolation under a shared protocol. We evaluate 18 mitigation methods on 2 base controllers, across 33 gap settings and 10 calibrated networks built from 5 real-world locations. We find that direct transfer consistently degrades performance across all four gap sources, but the severity of the degradation does not predict the effectiveness of mitigation. Instead, mitigation effectiveness depends strongly on the network and gap setting: outside the action gap, a method that helps in one case may fail in another. The most effective methods generally estimate what the gap changes, rather than make the policy insensitive through domain randomization or invariant representations. Our code is available at this https URL
• 中文摘要 强化学习在仿真中实现了强劲的交通信号控制性能，但模拟器训练的策略在实际应用后常常失败，这种失败被称为模拟到现实差距。当强化学习应用于交通信号控制时，这一差距源自多个来源：传感、动作执行、交通动态和控制目标。它们的相对影响以及现有模拟到现实缓解方法的可靠性仍不充分，且该领域缺乏系统测量差距和评估缓解方法的标准基准。我们介绍Sim2Signal基准测试，将模拟与现实差距分解为观察、行动、过渡和奖励差距，对应底层MDP四个组成部分的不匹配，并在共享协议下单独诱导每个差距。我们在2个基准控制器上评估了18种缓解方法，涵盖33个差距设置和10个校准网络，这些网络由5个真实世界地点构建。我们发现直接转移在所有四个缺口源上都会持续降低性能，但降级的严重程度并不预测缓解措施的有效性。相反，缓解效果强烈依赖于网络和差距设置：在行动缺口之外，一种在某种情况下有效的方法可能在另一种情况下失效。最有效的方法通常会估计缺口的变化，而不是通过域随机化或不变表示使策略变得不敏感。我们的代码可在此 https URL 获取。

Reinforcement learning to choose optimizers

强化学习选择优化器

• Authors: Martin van der Schelling, Deepesh Toshniwal, Miguel A. Bessa
• Subjects: Subjects: Neural and Evolutionary Computing (cs.NE); Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2609.01811
• Pdf link: https://arxiv.org/pdf/2609.01811
• Abstract No single optimization method is uniformly best for all problems, and the most suitable optimizer choice can change during a run. Existing approaches that change optimizer during execution typically predetermine part of the strategy: the portfolio is restricted to one algorithm class, the switch occurs once at a fixed time, or the frequency of decisions is treated as a hyperparameter rather than a learned one. We introduce "Reinforcement Learning to Choose Optimizers", which formulates the optimization algorithm choice as a sequential decision-making problem. At each decision, a recurrent policy reads the current run state and decides both which optimizer should be used next and for how long. The portfolio includes both gradient-based and derivative-free optimizers, and each switch passes on the current best solution and a representative step size. A context proxy conditions a gating network over expert heads, and training employs a decoupled actor-critic whose return is expressed in the same empirical runtime distribution metric used at evaluation. Training tasks and portfolio are designed jointly so that no optimizer dominates. On unseen problems, the learned policy outperforms every portfolio optimizer at all but the smallest budgets, and it remains robust under distribution shift.
• 中文摘要 没有单一优化方法是对所有问题都统一最佳，最合适的优化器选择可以在运行过程中发生变化。现有在执行过程中更换优化器的方法通常预先确定了策略的一部分：组合限制在某一算法类别，切换在固定时间发生一次，或者决策频率被视为超参数而非学习参数。我们引入“强化学习选择优化器”，将优化算法选择表述为顺序决策问题。每次决策时，循环策略读取当前运行状态，决定下一步使用哪个优化器和持续时间。组合包括基于梯度和无导数的优化器，每个开关传递当前最佳解和代表性步长。上下文代理为专家头的门控网络条件，训练使用解耦的演员-批评者，其回报以评估时使用的相同经验运行时分布指标表示。训练任务和组合设计联合，确保没有优化器占主导地位。在未见问题上，所学策略在除最小预算外的范围内均优于所有组合优化器，且在分布转移下保持稳健。

Import What You Need: Learning When and How to Augment EHR Graphs with External Knowledge

导入你需要的：学习何时以及如何用外部知识增强EHR图表

• Authors: Chen Chen, Mohsen Nayebi Kerdabadi, Dongjie Wang, Mei Liu, Zijun Yao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01839
• Pdf link: https://arxiv.org/pdf/2609.01839
• Abstract Longitudinal prediction from electronic health records (EHRs) is limited by the sparsity and irregularity in patient trajectories, and knowledge augmentation with external knowledge graphs (KGs) offers a promising way to alleviate these issues. However, most existing methods perform fixed, context-agnostic topology augmentation by adding the same KG nodes and edges regardless of a patient's evolving state. We propose ReTA, a Reinforcement learning-based dynamic Topology Augmentation framework that casts KG import as a per-visit, budget-aware policy. ReTA first constructs an offline refined pool of KG-grounded templates, then learns a policy to select one augment action per visit from three options: Soft Import, which enriches node features without modifying graph topology, Hard Import, which grafts a compact KG subgraph onto the visit graph to create message-passing shortcuts, and Skip, which leaves the visit unaugmented when the base encoder is already confident. To stabilize learning, ReTA employs a decoupled encoder that processes semantic and structural signals in separate channels and fuses them via adaptive gating. Experiments on MIMIC-III and MIMIC-IV across diagnosis prediction, mortality, and readmission show that ReTA consistently outperforms strong baselines while remaining efficient, transfers across datasets and knowledge graphs, and yields interpretable augmentation patterns. The robust gains under sparse supervision highlight the advantage of ReTA's dynamic decision to import knowledge, boosting accuracy while curbing costs.
• 中文摘要 电子健康记录（EHR）的纵向预测受限于患者轨迹的稀疏性和不规则性，而通过外部知识图谱（KGs）进行知识增强提供了缓解这些问题的有希望方式。然而，大多数现有方法通过添加相同的KG节点和边，实现固定、上下文无关的拓扑增强，无论患者处于何种状态。我们提出了ReTA，一种基于强化学习的动态拓扑增强框架，将KG导入定位为按次访问、预算感知的策略。ReTA首先构建一个离线精炼的KG基础模板池，然后学习策略，每次访问从三种选项中选择一个增强动作：软导入（Soft Import），丰富节点特征而不改变图拓扑;硬导入（Hard Import），将紧凑的KG子图嫁接到访问图上以创建消息传递捷径;以及跳过（Skip），当基础编码器已有信心时，访问不被增强。为稳定学习，ReTA采用解耦编码器，在不同通道处理语义和结构信号，并通过自适应门控融合。针对诊断预测、死亡率和再入院的MIMIC-III和MIMIC-IV实验显示，ReTA在高效性下持续优于强基线，能够跨数据集和知识图谱传输，并产生可解释的增强模式。在监管稀疏下取得的强劲收益凸显了ReTA动态决策引入知识的优势，既提高了准确度，又降低了成本。

Thinking effort aligns between humans and reasoning models in abductive reasoning

在溯因推理中，人类与推理模型之间的思考努力是一致的

• Authors: Henry Arthur
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01867
• Pdf link: https://arxiv.org/pdf/2609.01867
• Abstract A major question in cognitive modeling concerns the behavioral alignment between large language models and humans across linguistic and non-linguistic tasks. Unlike standard LLMs, large reasoning models (LRMs) are optimized with reinforcement learning from verifiable rewards, encouraging correct solutions to reasoning tasks rather than preference-aligned responses. Recent work (de Varda et al., 2025) investigates the cost of thinking in humans and LRMs by comparing human reaction times with model reasoning traces across a range of reasoning tasks. We isolate this alignment by turning to abductive reasoning: unlike deductive tasks, its difficulty cannot be inferred from formal structure and offers no shortcuts a model could exploit to mimic effort without genuine search, providing firmer ground for empirical claims of shared effort. We find further evidence of alignment between LRM and human reasoning effort, as well as evidence that models and humans tend to make similar errors. Finally, we show that decoding methods that let models explore multiple reasoning paths increase alignment in reasoning cost between humans and LRMs across the three models tested.
• 中文摘要 认知建模中的一个主要问题是大型语言模型与人类在语言和非语言任务中的行为一致性。与标准大型语言模型不同，大型推理模型（LRM）通过可验证奖励的强化学习进行优化，鼓励正确解答推理任务，而非偏好一致的反应。近期研究（de Varda 等，2025）通过比较人类反应时间与模型推理痕迹，研究人类与逻辑模型模型在多种推理任务中思考的成本。我们通过转向溯因推理来隔离这种对齐：与演绎任务不同，其难度无法从形式结构推断，且没有捷径，模型无法利用来模仿努力而无需真正搜索，这为共享努力的实证主张提供了更坚实的基础。我们还发现了LRM与人类推理努力之间对齐的进一步证据，以及模型与人类倾向于犯相似错误的证据。最后，我们证明了让模型探索多条推理路径的解码方法，能够提高人类与LRM在三种模型中推理成本的一致性。

OR-Transformer: Scaling Real-Time Decision-Making to 1,000 Items

OR-Transformer：实时决策规模提升至1000项

• Authors: Shuze Daniel Liu, David Simchi-Levi, Claire Chen, Chutong Gao, Shangtong Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.01933
• Pdf link: https://arxiv.org/pdf/2609.01933
• Abstract Modern supply chain operations can require coordinating replenishment across thousands of heterogeneous items under correlated stochastic demand, heterogeneous lead times, and shared fixed ordering costs, yielding observation spaces exceeding $10^4$ dimensions. At this scale, rolling-horizon stochastic mixed-integer linear programs (MILPs) become prohibitively slow, while standard reinforcement learning (RL) methods face increasingly challenging credit assignment in high-dimensional action spaces. We introduce OR-Transformer, a deep reinforcement learning framework for joint replenishment under stochastic demand, with an item-permutation-equivariant Transformer architecture and pathwise-gradient training through the inventory dynamics. Across problem sizes up to 1,024 inventory items, OR-Transformer increasingly outperforms learning-based and rolling-horizon MILP baselines as scale grows. It also reduces online decision-making time by over 4 million times relative to MILP solvers, enabling real-time, large-scale deep RL in supply chain operations.
• 中文摘要 现代供应链操作可能需要在相关随机需求、异构交货时间和共享固定订购成本下，协调数千个异构项目的补货，导致观测空间尺寸超过10^4美元。在这一规模下，滚动视界随机混合整数线性规划（MILPs）速度极慢，而标准强化学习（RL）方法在高维行动空间中信用分配面临越来越难的挑战。我们引入了OR-Transformer，一个深度强化学习框架，用于随机需求下的联合补货，采用项目-置换等变Transformer架构和通过库存动态进行路径梯度训练。在最大1024个库存项目的问题规模下，随着规模扩大，OR-Transformer的表现日益优于基于学习和滚动视野的MILP基线。它还将在线决策时间相较于MILP求解器减少了400万倍以上，实现了供应链运营中的实时大规模深度强化学习。

On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers

策略内提炼遇上非策略GRPO：培训紧凑的跟随教学者

• Authors: Vignesh Prabhakar, Jialing Pan, Anil Babu Ankisettipalli
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.01947
• Pdf link: https://arxiv.org/pdf/2609.01947
• Abstract Compact instruction-following rerankers are attractive for deployment, but conventional distillation pipelines typically train students by offline imitation of teacher outputs on a fixed set of examples, constraining supervision to the teacher's observed ranking space. We revisit reranker distillation through the lens of reinforcement learning. We propose a two-stage framework combining off-policy teacher optimization with on-policy student distillation. In Stage 1, a 4B teacher reranker is strengthened with off-policy GRPO using LLM-judge feedback on 88K instruction-following examples. In Stage 2, a compact 1B student samples rankings from its own policy and receives soft teacher-derived rewards on those rankings, coupling student exploration with knowledge transfer. Our strongest gains appear under distribution shift. On MAIR-11, the original 11-subset, 869-query evaluation, the proposed student reaches 0.7670 nDCG@6, outperforming offline listwise KD by +4.6 points. Controlled comparisons against offline pairwise RankNet KD and on-policy GKD show that neither changing the offline distillation objective nor moving teacher-distribution matching on-policy reproduces the performance of reward-based on-policy distillation over student-sampled rankings. The advantage persists on MAIR-Full: across all 126 tasks and 9,356 queries, the proposed method obtains the highest task-macro point estimates among the evaluated distillation variants, reaching 0.6808 nDCG@6 and 0.7865 MRR@6. It also exceeds two released 7B RL-trained rerankers on the comparable MAIR-11 evaluation, while the same Stage 2 training procedure consistently improves three architecturally distinct alternative student backbones. On the 9,861-query validation benchmark, the resulting 1B reranker achieves 0.7624 nDCG@6 while providing a favorable quality-efficiency tradeoff relative to larger alternatives.
• 中文摘要 紧凑的跟随指令排序器适合部署，但传统的蒸馏流程通常通过离线模仿教师输出来训练学生，限制监督于教师观察到的排名空间。我们通过强化学习的视角重新审视排序者提炼。我们提出一个两阶段框架，结合非政策教师优化与非政策学生提炼。在第一阶段，4B教师重新排序器通过对88K指令跟随示例的LLM评判反馈进行非策略GRPO强化。第二阶段，紧凑的1B学生从自身策略中采样排名，并根据这些排名获得教师提供的软奖励，将学生探索与知识转移结合起来。我们获得最显著的收益出现在分布转移阶段。在MAIR-11（原始11子集、869查询评估）中，拟议学生达到0.7670 nDCG@6，比离线列表式KD高出+4.6分。对离线两对RankNet KD和策略中GMKD的控制比较显示，无论是改变离线蒸馏目标还是移动教师分布匹配在政策上，都无法在学生抽样排名中重现基于奖励的策略内提取的表现。优势在MAIR-Full中依然存在：在所有126个任务和9,356次查询中，所提方法在评估的蒸馏变体中获得了最高的任务宏积分估计，分别达到0.6808 nDCG@6和0.7865个MRR@6。在可比的MAIR-11评估中，它超过了两个已发布的7B强化学习训练重排序器，而同一第二阶段训练过程则持续提升三个架构上不同的备选学生骨干。在9,861查询验证基准测试中，所得1B重排序器实现了0.7624 nDCG@6，同时相较于较大备选方案提供了有利的质量与效率权衡。

Act More, Decide Less: Skill-Guided Adaptive Action Chunking for Long-Horizon LLM Agents

多行动，少做决定：长视野大型语言模型代理的技能引导自适应行动分块

• Authors: Yanting Yang, Can Jin, Jinman Zhao, Jiahao Wu, Yang Zhou, Zhepeng Wang, Zhendong Wang, Mu Zhou, Dimitris N. Metaxas
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.02042
• Pdf link: https://arxiv.org/pdf/2609.02042
• Abstract Large language model (LLM) agents for long-horizon interactive tasks typically follow a ReAct-style protocol, issuing one primitive action per LLM round. While this enables frequent replanning, it is inefficient for long-horizon tasks where many rounds are spent on routine action sequences. A natural alternative is to let the agent emit variable-length action chunks. However, naively training such policies with standard reinforcement learning fails: the agent either collapses to single-action behavior or over-commits to excessively long sequences. Both failures share a common root cause: the inability to learn chunk boundaries. We propose SPACE, which addresses this challenge by distilling chunk-boundary supervision from trajectory-induced programmatic skills. We induce two-level programmatic skills from successful trajectories, where subskill boundaries serve as direct chunk-boundary supervision. This temporal structure is then distilled into a primitive-chunk policy via hybrid on-/off-policy optimization with chunk-aware credit assignment. Experiments on ALFWorld and ScienceWorld show that SPACE improves success rates by 7.0%-31.3% over the strongest baseline in each setting while reducing average LLM decision rounds by up to 78.9%.
• 中文摘要 用于长视野交互任务的大型语言模型（LLM）代理通常遵循ReAct式协议，每轮LLM发出一个原始动作。虽然这允许频繁重新规划，但对于长期任务中效率较低，因为许多轮次都花在常规动作序列上。一个自然的替代方案是让智能体发出可变长度的动作块。然而，用标准强化学习进行简单训练策略失败：智能体要么退缩为单动作行为，要么过度承诺过长的序列。这两种失败都有一个共同的根本原因：无法学习区块边界。我们提出了SPACE，通过从轨迹诱导的程序技能中提炼出块边界监督来解决这一挑战。我们从成功的轨迹中诱导出两级程序技能，其中子技能边界作为直接的块边界监督。该时间结构随后通过混合开/关策略优化与块感知信用分配，提炼为原始块策略。ALFWorld和ScienceWorld上的实验显示，SPACE在每个设定中较最强基线提升成功率7.0%-31.3%，同时平均LLM决策轮次减少多达78.9%。

IDEEA: training-free Input-Dependent stEEring via Activation cluster matching

IDEEA：通过激活簇匹配实现无训练输入依赖的stEEring

• Authors: Zheng Wang, Muchen Li, Renjie Liao, Yan Leng
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.02089
• Pdf link: https://arxiv.org/pdf/2609.02089
• Abstract Steering aligns large language models (LLMs) by injecting a bias into selected activations at inference time, offering a far cheaper alternative to weight-update methods such as supervised fine-tuning or reinforcement learning. However, most existing training-free steering methods are input-independent: a single direction is fitted once and shared across all inputs. This is fundamentally limiting as different inputs occupy different regions of the activation space and admit different optimal steering directions toward the same target concept, much as the gradient with respect to a fixed loss varies from input to input. We close this gap with IDEEA (Input-Dependent stEEring via Activation cluster matching), a training-free framework for input-dependent steering. IDEEA clusters the positive and negative activation supports per attention head, and solves an optimal-matching problem to construct a set of cluster-conditional directions, all about the target concept. At inference time, it picks from this pool of directions and uses the one that best matches the input's own activation for steering. IDEEA aligns the model toward the target concept while preserving the input's original representation, evidence that activations encoding a concept occupy several distinct sub-regions of the representation space rather than a single one. IDEEA improves the truth $\times$ info rate in TruthfulQA by an average of 9.9% (up to 23.5%) over the best input-independent baseline.
• 中文摘要 引导通过在推理时对特定激活注入偏向来对齐大型语言模型（LLM），提供了比监督微调或强化学习等权重更新方法更廉价的替代方案。然而，大多数现有无训练引导方法是输入无关的：单一方向拟合一次，并在所有输入中共享。这在根本上具有限制性，因为不同的输入占据激活空间的不同区域，并且在同一目标概念上会有不同的最优引导方向，就像固定损失的梯度在输入与输入之间不同一样。我们用IDEEA（通过激活集群匹配实现的输入依赖stEEring）来弥补这一空白，这是一个无训练的输入依赖引导框架。IDEEA将每个注意力头的正负激活支持聚类，并解决一个最优匹配问题，构建一组围绕目标概念的簇条件方向。在推断时，它从该方向池中选择最符合输入自身激活状态的方向进行引导。IDEEA在保持输入原始表征的同时，将模型与目标概念对齐，这表明编码概念的激活占据了表示空间的多个不同子区域，而非单一区域。IDEEA在TruthfulQA中使真值$\times$信息率平均提升9.9%（最高可达23.5%），相比最佳输入无关基线。

DMRL: Document-Mediated Reinforcement Learning for Skill Optimization in Advertising Recommendation

DMRL：文档介导强化学习，用于广告推荐技能优化

• Authors: Wei Zhang, Hongji Li, Song Sun, Peng Yu, Xue Yang, Lei Zhao, Peng Jiang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.02170
• Pdf link: https://arxiv.org/pdf/2609.02170
• Abstract Advertising recommendation requires continuously tuning complex system parameters while balancing commercial returns and user experience. Recent work has introduced large language models (LLMs) with skill documents to assist this labor-intensive process, but skill optimization remains largely prompt-driven, lacking a principled mechanism to attribute rewards to specific document edits. To address this limitation, we propose Document-Mediated Reinforcement Learning (DMRL), a skill self-evolution framework that models skill document optimization as a sequence of structured editing actions. In DMRL, an upper-level agent performs controlled document edits, while a frozen lower-level task agent evaluates their effects through A/B testing. To address credit assignment and long-term outcomes, we introduce two key components: (1) Dual-Relative Policy Optimization (DRPO), a post-training policy optimization method for robust and risk-aware advantage estimation; and (2) Long-term Reward Predictor (LRP), which estimates long-term outcomes by modeling population heterogeneity with disentangled representation learning and cross-attention transfer. DMRL was deployed on a large-scale short-video ads platform and extensive empirical evaluation shows that DMRL outperforms state-of-the-art baselines across key advertising metrics
• 中文摘要 广告推荐需要在平衡商业回报和用户体验的同时，不断调整复杂的系统参数。近期工作引入了带有技能文档的大型语言模型（LLM）来辅助这一劳动密集型过程，但技能优化仍主要依赖提示驱动，缺乏原则性机制来为特定文档编辑赋予奖励。为解决这一限制，我们提出了文档介导强化学习（DMRL），这是一种技能自我演化框架，将技能文档优化建模为一系列结构化编辑动作。在DMRL中，高级代理执行受控文档编辑，而冻结的低级任务代理则通过A/B测试评估其效果。为解决功劳分配和长期结果，我们引入了两个关键组成部分：（1）双相对策略优化（DRPO），一种用于稳健且风险意识优势估计的训练后策略优化方法;以及（2）长期奖励预测器（LRP），通过对群体异质性进行建模，结合解缠的表征学习和交叉注意力转移，估计长期结果。DMRL已部署在大型短视频广告平台上，广泛的实证评估显示，DMRL在关键广告指标上优于最先进的基线

PhoenixNest-Video: Evidence-Grounded Multimodal Agent Framework for Automated Video Interview Assessment

PhoenixNest视频：基于证据的多模态代理自动化视频访谈评估框架

• Authors: Fan Yuxuan, Huang Miaojun, Zhang Haimei, Wu Jingshen, Liu Hao
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.02231
• Pdf link: https://arxiv.org/pdf/2609.02231
• Abstract Interview assessment requires per-criterion judgments grounded in behavioral evidence, yet surging applicant volumes have made human-only evaluation costly and inconsistent, while existing AI approaches yield opaque scores without traceable rationale. We introduce PhoenixNest-Video, an evidence-grounded multimodal agent framework for automated video interview assessment. It builds a semantic video graph as structured working memory, performs rubric-conditioned retrieval with cross-modal verification across visual, audio, and textual streams, and produces per-criterion scores anchored to the candidate's materials. A Scorer trained via Rubrics-based Reinforcement Learning with dual rewards for rubric alignment and score-level differentiation internalizes the discriminative structure of multi-level rubrics. PhoenixNest-Video attains 91.50\% grade-level accuracy on VInterview-2025, outperforming substantially larger proprietary models. A compact, rubric-grounded agent therefore scores candidates in closer agreement with an expert panel than direct prompting of much larger models, and exposes the evidence behind each score for human review.
• 中文摘要 面试评估需要基于行为证据的标准判断，但激增的申请人数使得仅限人工评估成本高且不一致，而现有AI方法则得出不透明且无可追踪理由的分数。我们介绍PhoenixNest-Video，一个基于证据的多模态代理自动视频访谈评估框架。它构建语义视频图作为结构化工作记忆，通过视觉、音频和文本流的跨模态验证进行评分条件检索，并生成基于候选人材料的标准评分。通过基于评分标准的强化学习训练的评分员，该学习者对评分对齐和分数层级差异化给予双重奖励，内化了多层评分的判别结构。PhoenixNest-Video在Vinterview-2025上达到91.50%的年级水平准确率，优于更大规模的专有模型。因此，一个紧凑且基于评分标准的代理能以更接近专家评审组的标准评分，而非直接提示更大模型，并揭示每个评分背后的证据供人工审核。

PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks

PGPO：多回合代理任务的潜在引导策略优化

• Authors: Yuyao Zheng, Haipeng Sun, Junwei Bao, Lemao Liu, Hongfei Jiang, Yang Song, Dejing Dou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.02236
• Pdf link: https://arxiv.org/pdf/2609.02236
• Abstract Group-based reinforcement learning (RL) has become an effective paradigm for LLM post-training, but in multi-turn agentic tasks with sparse terminal rewards, it often provides coarse credit for intermediate actions. To obtain more fine-grained credit assignment, recent work such as GiGPO introduces step-level advantages for intermediate actions. However, these step-level signals still rely on the final outcome of each individual trajectory. As a result, actions within failed trajectories can remain poorly differentiated, so effective actions can receive the same unfavorable credit as erroneous ones. In this work, we propose Potential-Guided Policy Optimization (PGPO) for multi-turn agentic tasks. PGPO estimates empirical state potentials from anchor-state-group return statistics within each rollout group. It then derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation. This provides finer-grained step-level credit assignment, especially within failed trajectories. Experiments on ALFWorld and WebShop show strong overall performance relative to recent group-based RL methods. Further analysis provides evidence that PGPO yields more informative failure-side credit signals with negligible training overhead.
• 中文摘要 基于群体的强化学习（RL）已成为LLM训练后有效的范式，但在多回合代理任务中，终端奖励稀疏，它常为中间行动提供粗糙的信用。为获得更细致的信用分配，近期工作如GiGPO引入了中间行动的阶级优势。然而，这些阶级信号仍依赖于每个个别轨迹的最终结果。因此，失败轨迹内的行动可能区分较差，有效行动可能获得与错误行动相同的负面认可。本研究提出多回合代理任务的潜能引导策略优化（PGPO）。PGPO通过每个推广组内锚定状态组的回报统计估计经验状态潜力。随后，它从相邻状态间的潜在差异中推导出行动优势，实现交叉轨迹的信用传播。这提供了更细致的步骤级学分分配，尤其是在失败轨迹中。ALFWorld和WebShop上的实验显示，相较于近期基于群体的强化学习方法，整体表现较强。进一步分析显示，PGPO在失败侧获得更多信息的学分信号，且培训开销可忽略不计。

Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL

远程目标条件强化学习中的递归价值学习

• Authors: Hyeonseong Jeon, Youngwoon Lee
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.02237
• Pdf link: https://arxiv.org/pdf/2609.02237
• Abstract Scaling offline goal-conditioned reinforcement learning (GCRL) to long-horizon tasks is difficult because (1) long-range value learning depends on shorter-range estimates that may still be inaccurate, and (2) max-based value backups can amplify overestimation through repeated propagation. We propose DCRL (Divide-and-Conquer RL), which recursively decomposes each trajectory segment into a balanced binary tree and trains the values from leaves to root. Each parent is therefore updated only after its children, using an exact factorization of the observed route rather than selecting among noisy alternatives. Since this objective learns values along demonstrated routes that are not necessarily optimal, DCRL jointly propagates values across trajectories to discover shorter routes. Thanks to the balanced binary tree, DCRL reduces worst-case bootstrap depth from linear to logarithmic, and this shorter dependency structure empirically corresponds to much slower error accumulation. Across diverse goal-reaching tasks, DCRL substantially outperforms prior flat offline GCRL methods, and on the five most challenging long-horizon OGBench tasks, it improves the best prior average score from 55 to 64, surpassing all flat and hierarchical baselines.
• 中文摘要 将离线目标条件强化学习（GCRL）扩展到长视野任务很难，因为（1）长程值学习依赖于可能仍不准确的短期估计，（2）基于最大值的备份可能通过重复传播加剧高估。我们提出DCRL（分而治之RL），递归地将每个轨迹段分解为平衡的二叉树，并从叶节点到根节点训练值。因此，每个父节点仅在其子节点之后更新，使用观察路径的精确分解，而非在噪声较大的选项中选择。由于该目标沿已展示的路径学习值，但这些路径不一定最优，DCRL联合传播数值以发现更短的路径。得益于平衡二叉树，DCRL将最坏情况的自助深度从线性降低为对数，这种较短的依赖结构在经验上对应于错误累积速度更慢。在多种目标达成任务中，DCRL显著优于以往的平面离线GCRL方法，在五个最具挑战性的长视野OGBench任务中，它将最佳先验平均分从55提升到64，超越所有平坦和层级基线。

RideSkill: A Hierarchical Algorithm for Generalized Ride Sharing with LLM-Driven Automatic Evolution

RideSkill：一种基于LLM驱动自动演进的泛化共乘分层算法

• Authors: Zijian Zhao, Sen Li, Xialiang Tong, Mingxuan Yuan
• Subjects: Subjects: Multiagent Systems (cs.MA); Computation and Language (cs.CL); Emerging Technologies (cs.ET); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.02250
• Pdf link: https://arxiv.org/pdf/2609.02250
• Abstract Ride-sharing, which allows multiple passengers with different origin-destination (OD) pairs to share a single vehicle, is a challenging operational problem, as it requires orders with different OD pairs to be efficiently bundled and assigned to vehicles under uncertain and varying scenarios. Although multi-agent reinforcement learning (MARL) solutions have achieved promising performance, they suffer from limited generalization (adapting to different environmental scenarios), low transferability (adapting to different platform objectives), and training difficulties in large-scale systems, such as the curse of dimensionality. Recently, motivated by the scaling of large language models (LLMs), several works have incorporated LLMs into ride-hailing systems, either by employing LLMs directly as decision-making agents or using them for automatic algorithm design. However, none of these approaches support vehicle sharing, which complicates the problem by expanding both the state and action spaces exponentially. Moreover, most of them require frequent LLM calls at inference time, making them infeasible for real-time deployment. To address these issues, we propose RideSkill, a hierarchical method for ride-sharing that leverages LLM-assisted automatic algorithmic design. RideSkill consists of a combiner that assigns appropriate skills to each vehicle from a learned skill repository, enabling adaptive dispatch under varying scenarios and objectives, and a repositioner that sequentially relocates idle vehicles to emerging regions, avoiding conflicts among vehicles. Crucially, the skill repository, combiner, and repositioner are all trained by an LLM-based automatic evolutionary method, eliminating the need for LLM calls during deployment and thus ensuring high real-time performance.
• 中文摘要 网约车允许多位拥有不同起讫地（OD）对的乘客共享同一车辆，这是一项具有挑战性的操作难题，因为它要求在不确定和变化的场景下，将不同OD对的订单高效捆绑并分配给车辆。尽管多智能体强化学习（MARL）解决方案取得了有希望的性能，但它们存在泛化性有限（适应不同环境场景）、低迁移性（适应不同平台目标）以及在大规模系统中训练困难（如维度诅咒）的问题。最近，受大型语言模型（LLM）扩展的推动，多项工作将LLM纳入网约车系统，要么直接将LLM作为决策代理，要么用于自动算法设计。然而，这些方法均不支持车辆共享，这使得状态空间和动作空间呈指数级扩展，使问题更加复杂。此外，大多数在推理时需要频繁调用LLM，导致无法实现实时部署。为解决这些问题，我们提出了RideSkill，一种利用LLM辅助自动算法设计的分层共享乘车方法。RideSkill由一个组合器组成，从学习到的技能库中为每辆车辆分配合适的技能，支持在不同场景和目标下的自适应调度;以及一个重新定位器，将闲置车辆顺序迁移到新兴区域，避免车辆间冲突。关键是，技能库、组合器和重新定位器均由基于LLM的自动演化方法训练，消除部署时的LLM调用，从而确保高实时性能。

APEx: Distillation of Agent Procedural Experience for Adaptive Deep Research Question Answering

APEx：针对自适应深度研究问答的代理程序经验提炼

• Authors: Jie Ding, Rui Sun, Xinyuan Zhang, Zeyu Zhang, Xin Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.02253
• Pdf link: https://arxiv.org/pdf/2609.02253
• Abstract Deep research agents augment large language models with external tools to answer complex, long-horizon questions through multi-turn reasoning. Learning from prior experience is crucial for continual improvement, yet existing methods either retrieve verbose task-specific traces that burden decision-making, or distill procedural skills that remain decoupled from downstream policy adaptation. We propose APEx, a hierarchical experience utilization framework that organizes interaction history into instance-level trajectory memories and category-level procedural skills, and couples them through a closed-loop architecture of Executor, Distiller, and Planner. The three modules are optimized via a three-stage alternating GRPO training paradigm, enabling reward-guided skill distillation rather than fixed-prompt generation. At test time, distilled skills serve as procedural priors for online Planner adaptation through skill-guided test-time reinforcement learning, allowing ground-truth-free self-improvement with skill-alignment regularization to prevent policy drift. Experiments on 7 benchmarks demonstrate that APEx achieves state-of-the-art performance, surpassing GPT-5.4 by 14.7 points and the strongest memory-augmented baseline by 3.0 points.
• 中文摘要 深度研究代理通过外部工具增强大型语言模型，通过多回合推理回答复杂且长远的问题。从既往经验中学习对于持续改进至关重要，但现有方法要么检索冗长的任务特定痕迹，从而增加决策负担，要么提炼出与下游策略调整脱钩的程序技能。我们提出了APEx，一种层级体验利用框架，将交互历史组织为实例级轨迹记忆和类别级程序技能，并通过Executor、Distiller和Planner的闭环架构将其连接起来。这三个模块通过三阶段交替的GRPO训练范式进行优化，实现了以奖励为导向的技能提炼，而非固定提示生成。在测试阶段，精炼技能作为在线Planner适应的程序先验，通过技能引导测试时强化学习实现，允许无实地的自我提升，并通过技能对齐正则化防止政策偏移。在7个基准测试上的实验表明，APEx实现了最先进的性能，比GPT-5.4高出14.7分，在最强的记忆增强基线中高出3.0分。

A Comparative Study of Graph Representations for GNN-Based Power Grid Control in L2RPN

基于GNN的L2RPN电网控制图表示的比较研究

• Authors: Adrian Degenkolb, Qiong Huang, Benjamin Schäfer
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.02538
• Pdf link: https://arxiv.org/pdf/2609.02538
• Abstract Graph construction is a critical but underexamined design choice in deep reinforcement learning for power grid control. We present a controlled experimental comparison of different graph representations, including physical topology, electrical-sensitivity, and hybrid variants for topology control in the Learning to Run a Power Network (L2RPN) environment. Our findings indicate that matching graph complexity to task granularity is more important than maximizing representational richness, and highlight the importance of controlled representation studies at scale.
• 中文摘要 图构建是电网控制深度强化学习中一个关键但尚未被充分探讨的设计选择。我们对不同图表示方式进行了受控实验比较，包括物理拓扑、电敏性和混合变体，用于学习运行电力网络（L2RPN）环境下的拓扑控制。我们的发现表明，将图复杂度与任务粒度匹配比最大化表示丰富性更为重要，并强调了大规模受控表示研究的重要性。

Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMs

向谁学对的人：多领域大型语言模型的答案验证多教师提炼

• Authors: Xixiang He, Xingming Li, Baiqi Wu, Qiyao Sun, Xuanyu Ji, Ao Cheng, Qingyong Hu
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.02548
• Pdf link: https://arxiv.org/pdf/2609.02548
• Abstract Modern large language models (LLMs) rely on reinforcement learning to build strong capabilities in individual domains, but integrating those capabilities into a single deployable model remains challenging. By routing each sample to the teacher whose domain matches it, existing approaches let a domain label decide which teacher provides supervision. However, domain expertise holds only on average: the matched teacher is not always correct on a given sample, while a teacher from another domain sometimes is. The reliable teacher therefore has to be identified per sample, not per domain. In this paper, we introduce Multi-Teacher Self-Distillation Policy Optimization (MT-SDPO), an on-policy distillation method that unifies several frozen teachers into one student model. MT-SDPO consists of three components: (1) self-anchors, where a rollout is supervised by a correct rollout from its own group; (2) answer-verified eligibility, where a teacher may supervise a sample only if its own answer passes a verifier; and (3) privileged distillation, which merges the anchor and all verified feedback into one context that an exponential moving average self-teacher reads and the student does not, thereby keeping one policy at deployment. Across five students from three model families, MT-SDPO lifts the weakest domain of Qwen3-8B by 14.79 points and narrows its domain gap by 74.7%, a better balance than serving one matched teacher per domain. Verified reliability, not domain membership, should decide who teaches. Code is available at this https URL.
• 中文摘要 现代大型语言模型（LLM）依赖强化学习在单个领域构建强大能力，但将这些能力整合到单一可部署模型中仍然具有挑战性。通过将每个样本分配给与其领域匹配的教师，现有方法允许领域标签决定哪位教师提供指导。然而，领域专业知识仅限于平均值：匹配的教师在某一样本上并不总是正确，而来自其他领域的教师有时是正确的。因此，可靠的教师必须逐样本识别，而非按领域。本文介绍了多教师自我蒸馏策略优化（MT-SDPO），这是一种基于策略的提炼方法，将多个固定教师统一到一个学生模型中。MT-SDPO由三个组成部分组成：（1）自我锚定，即由其自身群体的正确推广监督;（2）答案验证资格，教师仅在自身答案通过验证者时监督样本;（3）特权蒸馏，将锚点和所有验证反馈合并到一个上下文中，指数移动平均自学者阅读而学生不读，从而在部署时保持同一个策略。在三个模型家族的五名学生中，MT-SDPO将Qwen3-8B中最弱的领域提升了14.79个分，其领域差距缩小了74.7%，比每个域服务一名匹配教师更为平衡。应以验证的可靠性而非域成员身份决定谁授课。代码可在此 https URL 获取。

GDB-Reward: From Evaluation Metrics to Training Rewards for Graphic Design

GDB-奖励：从评估指标到平面设计培训奖励

• Authors: Adrienne Deganutti, Purvanshi Mehta, Simon Hadfield, Andrew Gilbert
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.02813
• Pdf link: https://arxiv.org/pdf/2609.02813
• Abstract Text-to-image models excel at natural image synthesis but struggle with graphic design, where success depends on satisfying precise constraints on typography, layout, color, and visual communication. While prompt optimization offers an attractive alternative to expensive diffusion model fine-tuning, learning prompts for frozen image generators requires informative reward functions despite the entirely non-differentiable generation process. Reinforcement learning does not require differentiable objectives; it requires only scalar rewards capable of ranking candidate outputs. This raises a simple question: can design evaluation metrics themselves become reinforcement learning rewards? Our central contribution is GDB-Reward, a framework that systematically transforms heterogeneous graphic design evaluation metrics into a unified reinforcement learning reward. Experiments demonstrate that GDB-Reward provides an effective optimization objective, substantially improving adherence to the design specification in perceptual quality, rendering fidelity, and spatial accuracy while keeping the image generator entirely frozen. More broadly, our results demonstrate that heterogeneous, non-differentiable evaluation metrics can move beyond passive benchmarking to become effective optimization objectives for reinforcement learning in domains where differentiable supervision is unavailable.
• 中文摘要 文本转图像模型在自然图像合成方面表现出色，但在图形设计方面则较为困难，因为成功依赖于满足字体、布局、色彩和视觉传达的精确约束。虽然提示优化为昂贵的扩散模型微调提供了有吸引力的替代方案，但冻结图像生成器的学习提示需要信息型奖励函数，尽管生成过程完全不可微分。强化学习不需要可微目标;它只需能够对候选输出进行排名的标量奖励。这引出了一个简单的问题：设计评估指标本身能否成为强化学习奖励？我们的核心贡献是GDB-Reward，这是一个系统性地将异构图形设计评估指标转化为统一强化学习奖励的框架。实验表明，GDB-Reward提供了有效的优化目标，显著提升了感知质量、渲染精度和空间准确性对设计规范的遵循，同时保持图像生成器完全冻结。更广泛地说，我们的结果表明，异构且不可微分的评估指标可以超越被动基准，成为在无法实现可微监督的领域中，强化学习的有效优化目标。

Cliff: Learning Process Rewards from the First Mistake

悬崖：从第一次错误中获得的学习过程奖励

• Authors: Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao, Gerald Friedland, Chris Kong
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.02817
• Pdf link: https://arxiv.org/pdf/2609.02817
• Abstract Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.
• 中文摘要 带可验证奖励的强化学习（RLVR）已成为大型语言模型（LLM）训练后强有力的范式，但其对粗糙结果奖励的依赖导致对中间推理过程的指导有限。现有方法如过程奖励建模和策略内提炼引入了额外限制，如依赖专门的奖励模型或假设师生推理模式相同。然而，我们观察到，一旦推理过程首次出错，评估后续推理所提供的信息有限，因为它已经基于无效前缀。因此，我们提出了Cliff，一种奖励塑造策略，利用现成的LLM作为教师，识别每次推广中的第一个错误。因此，推广自然分为两部分：正确的前缀和错误的后缀。Cliff 随后将该信号转换为代币层面优势，正确前缀赋予正向优势，之后给予负反馈。跨越 12 个不同场景的实验表明，Cliff 持续提升推理表现，即使教师能力有限，也比策略提炼高出 15%，标准 GRPO 高出 7%。此外，我们分析了“地面真实”在 Cliff 中的作用，并探讨其训练动态。这些结果确立了 Cliff 作为一种简单、通用且有效的方法，在更丰富、细致的监督下提升 RLVR 的水平。

Post-Training Language Models for Gold-Medal Performance in Coding Competitions

编程竞赛金牌表现的训练后语言模型

• Authors: Aleksander Ficek, Sean Narenthiran, Mehrzad Samadi, Somshubra Majumdar, Boris Ginsburg
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multiagent Systems (cs.MA); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2609.02849
• Pdf link: https://arxiv.org/pdf/2609.02849
• Abstract Competitive programming has become a key test of large language model reasoning, with international competitions such as IOI and ICPC representing its most challenging settings. We present an end-to-end specialization pipeline combining large-scale problem curation, synthetic reasoning traces, supervised fine-tuning (SFT), and reinforcement learning (RL). Using 22,000 curated problems, we train Nemotron-3-Nano-CC (30B-A3B) with SFT and RL and Nemotron-3-Ultra-CC (550B-A55B) with SFT alone. We further introduce GenCorrect, a feedback-driven test-time compute strategy that iteratively generates, evaluates, and refines diverse solutions. On IOI 2025, Nano-CC improves from 130 points to 291 after post-training and to 468 with GenCorrect, exceeding the gold threshold of 438.3 while Ultra-CC reaches 502. Guided by these results, we develop a competition-specific Ultra-CC system and evaluate it prospectively during IOI 2026. Under the same time, internet-access, and submission constraints as human contestants, it scores 535.4 out of 600, exceeding both the gold threshold of 361.12 and the top human score of 498.27. To our knowledge, this is the first AI system to outscore the highest-scoring human contestant on an IOI problem set.
• 中文摘要 竞技编程已成为大型语言模型推理的关键考验，国际竞赛如IOI和ICPC代表了其最具挑战性的环境。我们提出了一个端到端的专业化流程，结合了大规模问题策划、合成推理轨迹、监督微调（SFT）和强化学习（RL）。利用22,000个策划问题，我们用SFT和RL训练Nemotron-3-Nano-CC（30B-A3B），仅用SFT训练Nemotron-3-Ultra-CC（550B-A55B）。我们还进一步介绍GenCorrect，这是一种反馈驱动的测试时间计算策略，能够迭代生成、评估和优化多样化的解决方案。在IOI 2025中，Nano-CC在训练后从130分提升至291分，GenCorrect后提升至468分，超过金门槛438.3，而Ultra-CC达到502分。基于这些结果，我们开发了针对比赛的Ultra-CC系统，并在IOI 2026期间进行前瞻性评估。在与真人参赛者相同的时间、互联网接入和提交限制下，Nano-CC得分为535.4分（满分600分），超过金门槛361.12分和最高人类得分498.27分。据我们所知，这是首个在IOI题目集中得分超过最高分的人类参赛者的AI系统。

Keyword: diffusion policy

There is no result