生成时间: 2026-09-07 22:03:46 (UTC+8); Arxiv 发布时间: 2026-09-07 20:00 EDT (2026-09-08 08:00 UTC+8)

今天共有 26 篇相关文章

Keyword: reinforcement learning

Step Back to Move Forward: Reflection-Aware Preference Optimization for Visual Generation

退一步迈向前进：视觉生成的反射感知偏好优化

• Authors: Junlong Wu, Jiuzhou Lin, Jia Sun, Boheng Zhang, Huaiqing Wang, Dewen Fan, Houde Liu, Qianqian Gan, Fan Yang, Tingting Gao
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.04282
• Pdf link: https://arxiv.org/pdf/2609.04282
• Abstract Diffusion models have become the mainstream paradigm for modern visual generation and have substantially advanced multimedia content synthesis, especially in text-to-image and text-to-video tasks. To further align such generative models with human preferences, reinforcement learning (RL) has recently shown strong potential as a post-training strategy. Nevertheless, existing policy gradient-based methods often explore inefficiently, making them vulnerable to local optima that may degrade semantic faithfulness and visual realism. To address these challenges, we present Reflection-Aware GRPO (RA-GRPO), a new RL-based preference alignment framework for diffusion generative models. The core idea is to improve "forward" generation by incorporating "backward" reflection during optimization. We first introduce Diffusion Reflection, which rectifies intermediate sampling trajectories by inverting the diffusion process with a weak estimator, guiding latent states toward higher-probability regions of the true data manifold. Furthermore, we introduce Counterfactual Path Synthesis to implicitly distill these rectified trajectories into the policy, enabling the model to internalize the benefits of search-based exploration without incurring inference-time overhead. Extensive experiments on T2I and T2V models demonstrate that RA-GRPO significantly outperforms existing methods, particularly in mitigating reward hacking and improving generalization. The method remains architecture-agnostic and integrates seamlessly with standard pipelines, suggesting a promising direction for stable preference alignment.
• 中文摘要 扩散模型已成为现代视觉生成的主流范式，并在多媒体内容综合方面取得了显著进步，尤其是在文本到图像和文本到视频任务中。为了进一步使此类生成模型与人类偏好对齐，强化学习（RL）最近展现出作为后训练策略的强大潜力。然而，现有基于策略梯度的方法常常探索效率不高，使其容易受到局部最优影响，从而降低语义忠实性和视觉真实性。为应对这些挑战，我们提出了反射感知GRPO（RA-GRPO），这是一个基于RL的扩散生成模型偏好对齐框架。核心思想是通过在优化过程中引入“后向”反射“来提升”前向“生成。我们首先引入扩散反射，通过反向扩散过程与弱估计量纠正中间采样轨迹，引导潜态向真实数据流形中更高概率区域。此外，我们引入反事实路径合成，隐式将这些纠正轨迹蒸馏入策略，使模型能够内化基于搜索探索的优势，而无需产生推理时间开销。对T2I和T2V模型的大量实验表明，RA-GRPO在减轻奖励黑客和提升泛化方面显著优于现有方法。该方法保持架构无关性，并与标准流水线无缝集成，为稳定偏好对齐提出了一个有前景的方向。

Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching

通过样本引导分布匹配实现视频生成的联合比对与蒸馏

• Authors: Jiuzhou Lin, Junlong Wu, Fei Zuo, Huan Ouyang, Dewen Fan, Boheng Zhang, Huaiqing Wang, Jia Sun, Fan Yang, Houde Liu, Kehai Chen, Min Zhang, Tingting Gao, Han Li
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.04283
• Pdf link: https://arxiv.org/pdf/2609.04283
• Abstract Aligning video generative models to human preferences heavily relies on Reinforcement Learning (RL), which suffers from extensive computational overhead. Existing workflows typically treat RL and distillation as disconnected stages: applying RL before distillation incurs prohibitive computational costs, whereas applying RL after distillation frequently leads to model collapse. To overcome these limitations, we propose a unified, single-stage optimization framework grounded in Distribution Matching (DM). In the standard DM framework, distillation updates the model via a gradient direction that minimizes the gap between the real and fake models, guiding generations toward clarity and high fidelity. Building upon this, we introduce DM-Align, which derives a complementary gradient direction to guide the model toward human-preferred samples. Inspired by DPO and GRPO, our method leverages the distributional gap -- formulated from either preference pairs or intra-group exploration -- to directly construct this preference-guided gradient. By synergizing these two gradient directions, our approach eliminates the need for multi-step reward evaluation and complex ODE-SDE conversions inherent in traditional RL. Comprehensive experiments across multiple foundational video models demonstrate that this sample-guided framework robustly enhances both distillation quality and preference alignment, consistently outperforming both standalone variants and sequential two-stage pipelines.
• 中文摘要 将视频生成模型与人类偏好对齐高度依赖强化学习（RL），而该学习存在较大的计算开销。现有工作流程通常将强化学习和蒸馏视为两个独立阶段：在蒸馏前应用RL会产生高昂的计算成本，而在蒸馏后应用RL则常导致模型崩溃。为克服这些限制，我们提出了一个统一的单阶段优化框架，基于分布匹配（DM）。在标准DM框架中，蒸馏通过梯度方向更新模型，最小化真实模型与虚假模型之间的差距，引导各代人趋向清晰和高保真度。在此基础上，我们引入了DM-Align，它推导出互补的梯度方向，引导模型朝向人类偏好的样本方向。受DPO和GRPO启发，我们的方法利用分布差距——无论是偏好对还是组内探索——直接构建了这一偏好引导梯度。通过协同这两个梯度方向，我们的方法消除了传统强化学习中固有的多步奖励评估和复杂常微分方程-SDE转换的需求。在多个基础视频模型上的综合实验表明，该样本引导框架能稳健地提升蒸馏质量和偏好对齐，持续优于独立变体和顺序两阶段流水线。

VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models

VLA精度：非对称共自助，实现视觉-语言-行动模型的高效现实在线强化学习

• Authors: Chenyu Su, Zhaolong Shen, Yuan Qian, Chen Qian, Rui Zhang, Feng Yan, Weixing Chen, Fei Zhang, Jiamin Wang, Shuang Cong, Weiwei Shang
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.04355
• Pdf link: https://arxiv.org/pdf/2609.04355
• Abstract Pretrained vision-language-action (VLA) models enable broad manipulation but remain unreliable in tasks demanding precision and repeatability. Applying real-world online reinforcement learning (RL) to VLA post-training enables autonomous trial-and-error improvement beyond demonstrations alone, but exposes two bottlenecks: 1) unreliable value signals can induce policy drift; 2) large-VLA overhead constrains throughput and sample efficiency. To address these challenges, we present VLA-Precision, an efficient real-world online RL framework featuring the Asymmetric Co-Bootstrapping (ACoB) algorithm and the ACoB-Stream architecture. Specifically, ACoB establishes asymmetric co-bootstrapping across timescales: early intervention-guided behavioral learning rapidly improves policy performance while enhancing online experience quality. As autonomous experience accumulates, global return propagation and local preference ranking progressively calibrate value estimates, yielding relative action advantages for reference-regularized policy improvement while suppressing drift. To enable ACoB on large VLAs, we develop ACoB-Stream, a closed-loop experience--policy architecture that establishes invariant-state decoupling and on-demand streaming as design principles, delivering up to 10.9$\times$ improvements in throughput and computational efficiency. Extensive evaluations on nine high-precision chemistry tasks across four categories and four robot embodiments show that VLA-Precision achieves 98.3\% mean success rate in 45.8 min/task, with 27.6 s episodes running at 1.2$\times$ and 1.8$\times$ the speeds of VLA and RL baselines. Resources are available at this https URL.
• 中文摘要 预训练视觉-语言-动作（VLA）模型支持广泛的操作，但在要求精确性和重复性的任务中仍不可靠。将现实世界的在线强化学习（RL）应用于VLA后期训练，使得自主试错改进超越单纯演示，但也存在两个瓶颈：1）不可靠的价值信号可能导致策略漂移;2）VLA开销大，限制吞吐量和样本效率。为应对这些挑战，我们介绍VLA-Precision，一个高效的现实世界在线强化学习框架，采用非对称共引导（ACoB）算法和ACoB流架构。具体来说，ACoB建立了跨时间尺度的非对称共引导：早期干预引导的行为学习快速提升策略性能，同时提升在线体验质量。随着自主经验的积累，全局返回传播和局部偏好排序逐步校准价值估计，在降低漂移的同时，在参考正则化策略改进方面获得相对动作优势。为使ACoB在大型VLA上实现，我们开发了ACoB-Stream，一种闭环体验策略架构，将不变状态解耦和按需流作为设计原则，在吞吐量和计算效率上提升高达10.9$\times$。对涵盖四个类别和四个机器人实例的九个高精度化学任务进行了广泛评估，显示VLA-Precision在每任务45.8分钟内实现98.3%的平均成功率，27.6秒的运行速度为1.2$\times$，速度为VLA和RL基线的1.8$\时间。资源请访问此https URL。

REFINE: LLM Refinement over Budgeted Text-Attributed Graphs for Personalized Medical Concept Representation

精炼：基于预算文本属性图表的LLM细化，实现个性化医疗概念表示

• Authors: Mohsen Nayebi Kerdabadi, Arya Hadizadeh Moghaddam, Dongjie Wang, Zijun Yao
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04415
• Pdf link: https://arxiv.org/pdf/2609.04415
• Abstract Learning rich medical concept representations is essential for EHR prediction. Text-attributed knowledge graphs (TKGs) provide a natural foundation by organizing heterogeneous medical relations together with textual semantics. However, most existing encoders process concepts uniformly across patients, despite the fact that a code's meaning and predictive value depend on patient-specific clinical context and trajectory. Learning patient-personalized concept representations from TKGs introduces two key challenges: (1) deciding how much KG context to incorporate for each observed code, and (2) aligning semantic information with the patient-specific relational structure. We propose REFINE, a KG-aware budgeted LLM graph refinement framework for patient-personalized medical concept encoding. Starting from a global TKG, REFINE constructs patient-specific temporal graphs. A sequential reinforcement learning policy selects a personalized KG expansion budget for each observed code. The resulting patient graph is processed by a heterogeneous GNN to capture relation-aware structural dependencies, while a frozen LLM uses graph-aware soft prompts to semantically refine concept representations. Experiments on MIMIC-III and MIMIC-IV show that REFINE consistently improves diverse EHR backbones, outperforms strong baselines, and demonstrates robust gains across component ablation, KG selection, and data insufficiency.
• 中文摘要 学习丰富的医学概念表示对于电子健康记录预测至关重要。文本属性知识图谱（TKGs）通过组织异质医疗关系和文本语义，提供了自然的基础。然而，尽管代码的意义和预测价值依赖于患者特定的临床上下文和轨迹，但大多数现有编码器在不同患者之间是统一处理概念的。从TKG学习患者个性化的概念表示带来了两个关键挑战：（1）决定每个观察到的代码应包含多少KG上下文，（2）将语义信息与患者特定的关系结构对齐。我们提出了REFINE，一种基于KG的预算LLM图细化框架，用于患者个性化的医学概念编码。从全局TKG出发，REFINE构建患者特定的时间图。顺序强化学习策略为每个观察到的代码选择个性化的KG扩展预算。生成的患者图由异构GNN处理以捕捉关系感知的结构依赖，而冻结的LLM则使用图感知软提示语义细化概念表征。MIMIC-III和MIMIC-IV实验显示，REFINE持续改善多样化的EHR骨干，优于强基线，并在组件消融、公斤选择和数据不足方面表现出强劲提升。

HarvestBench: Measuring Whether LLM Agents Will Pay to Avoid Killing Animals

HarvestBench：衡量大型语言模型代理是否愿意支付避免杀害动物的费用

• Authors: Jasmine Brazilek, Miles Tidmarsh, Matthias Endres, Anshuman Singh, Jeremiah Miller
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04444
• Pdf link: https://arxiv.org/pdf/2609.04444
• Abstract Benchmarks for the side effects an agent causes on the way to a goal already exist, but HarvestBench is the first to put a price on avoiding the side effect and to name that side effect as a living creature. It is a farm simulation: LLM sub-agents drive a crew of two tractors through a cooperative corn harvest, with animals in the field. The environment is a reinforcement learning gridworld, every decision is made without memory, and the harm is never named in the goal. When an animal blocks a tractor's route the autopilot stops and asks the model whether to drive on, at no fuel cost, or swerve around it for a posted fuel price. Kills are compared against two controls: rocks, which damage the tractor and are hit under 1% of the time by every model, and hay bales, which are harmless and not alive. Models can also take crops from the neighbor's field instead of their own, a second test of what they treat as moral. Across nine models and 7,201 priced decisions, 3,951 involved an animal rather than a hay bale or a rock. Kill rates range from 0.4% to 98.8%, with Terra and Sol the most merciful and GPT-4o-mini the most cruel, and they are not ordered by capability. Four of six models were sensitive to price at the 5% level, with elasticities from 0.09 to 1.69. All nine drove over wild animals more often than farmed animals on the default map, and the direction held at every map geometry in every model with room to move. The briefing mattered most: under the morality briefing the kill rate was under 6% in five of six reasoning models, and removing it raised the kill rate above 84% in all six. HarvestBench uses no LLM grader. The scorer counts events in the game log, so it is fully reproducible, and it measures what a model will pay to avoid harm rather than what it says about harm.
• 中文摘要 智能体在达成目标过程中产生的副作用已有基准，但HarvestBench是第一个将避免副作用定价，并将该副作用命名为活体生物的平台。它是一个农场模拟：大型语言模型子智能机驾驶两台拖拉机组成的团队穿越合作玉米收割，田间有动物。环境是一个强化学习网格世界，每个决策都没有记忆，伤害从未在目标中被命名。当动物阻挡拖拉机路线时，自动驾驶会停止，询问模型是继续行驶（无燃料成本），还是绕行以显示燃油价格。击杀数与两个对照进行比较：岩石，拖拉机受损，每个模型击中率低于1%;以及干草捆，干草捆无害且非活体。模型还可以从邻居田地取作物，而非自己的，这是对他们道德的第二次考验。在九个模型和7,201个定价决策中，有3,951个涉及动物，而非干草捆或岩石。杀伤率范围从0.4%到98.8%，Terra和Sol最为仁慈，GPT-40-mini最残酷，且不按能力排序。六个模型中有四个对价格在5%水平上敏感，弹性范围从0.09到1.69。所有九个模型在默认地图上都比农场动物更频繁地碾压野生动物，且每个模型中每张地图的几何体保持方向。简报最为重要：在道德简报中，六个推理模型中的五个杀死率低于6%，移除后杀伤率均超过84%。HarvestBench不使用LLM评分器。评分器会统计游戏日志中的事件，因此完全可重复，并且衡量的是模型为避免伤害而支付的费用，而不是模型对伤害的判断。

What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents

多线束强化学习内容？编码代理中的学分分配与可移植性

• Authors: Chenqian Le, Jiayi Cheng, Qijia He, Runhao Li, Yinghao Li, Xupeng Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04518
• Pdf link: https://arxiv.org/pdf/2609.04518
• Abstract Agent reinforcement learning (RL) increasingly runs through full execution harnesses, and a multi-harness recipe mixes two choices: exposing the policy to several harnesses, and comparing their rewards inside one relative-advantage group. We isolate the second choice in repository-level coding. From one Qwen3-8B supervised warm start we replay the same frozen task-harness records from Aider, OpenHands, Qwen Code, and SWE-agent, with the same number of updates, under two rules for group-relative policy optimization (GRPO), Within (one group per task-harness pair) and Cross (harnesses pooled within a task), and score every checkpoint with a sealed SWE-bench Verified oracle on four source harnesses and a minimal harness held out of training. The evaluation harness is the dominant variable: across 24,000 sealed evaluations it moves the mean solve rate from 2.14\% to 9.27\%, a factor of 4.3, where the training recipe moves it by 1.16. The grouping rule is not. On the held-out harness, Cross minus Within is +0.25 pp, 95\% confidence interval [-0.48, +1.02], at eight attempts per task, and +0.16 [-0.41, +0.72] pooled over three training seeds whose individual estimates change sign. Each rule's own seed range, 0.42 to 0.45 pp, exceeds the difference between them. Both rules place their largest gains on the same source harness. The pooled advantage carries the harness: an out-of-fold classifier recovers the generating harness from Cross's advantage +4.48 pp above the shuffled-label baseline and from Within's not at all, and the two rules still reach the same held-out score and action distribution inside each harness. Re-collecting half the training data on-policy does not change this. Cross-harness credit yields configuration adaptation and no more portable capability than within-harness credit. Multi-harness RL reports should state the grouping boundary and test under an unseen harness.
• 中文摘要 代理强化学习（RL）越来越多地通过完整的执行框架运行，而多框架方案结合了两种选择：将策略暴露于多个框架，并在一个相对优势组内比较它们的奖励。我们在仓库级编码中隔离了第二个选择。从一个Qwen3-8B监督的热启动中，我们按照两个组相对策略优化（GRPO）规则，以相同次数的更新回放Aider、OpenHands、Qwen Code和SWE-agent相同的任务框架记录，分别是Within（每对任务束组一组）和Cross（任务内共享一组束），并用密封的SWE工作台验证oracle对四个源框架和一个未训练的最小束带进行评分。评估工具是主导变量：在24,000次密封评估中，平均破解率从2.14%提升到9.27%，为4.3倍，而训练配方则提升1.16倍。分组规则则不然。在保留的框架中，Cross减去Within为+0.25 pp，95置信区间 [-0.48， +1.02]，每任务8次尝试，+0.16 [-0.41， +0.72]，合并为+0.16 [-0.41， +0.72]。每条规则的种子范围为0.42至0.45 pp，均超过它们之间的差值。两种规则均将最大收益集中在同一源头束上。合并优势承载了束带：折叠外分类器从交叉优势+4.48pp中恢复生成束束，从Within的优势中恢复，完全不恢复，且两条规则在每个束内仍达到相同的保留分数和动作分布。在策略上重新收集一半训练数据并不会改变这一点。跨束信用带来配置适配，且不比内置约束更可移植。多束强化报告应说明分组边界，并在未见框架下测试。

Deep Reinforcement Learning for Optimization of STAR-RIS Phase and Energy Splitting Coefficients in OTFS-NOMA Framework

OTFS-NOMA 框架中 STAR-RIS 相位和能量分割系数优化深度强化学习

• Authors: Rais. J. Gachaba, Manobendu Sarker, Anirban Bhowal
• Subjects: Subjects: Information Theory (cs.IT); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.04536
• Pdf link: https://arxiv.org/pdf/2609.04536
• Abstract This paper considers a downlink communication framework comprising a simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS)-aided by orthogonal time frequency space (OTFS) and non-orthogonal multiple access (NOMA) technologies. Further, delay-Doppler mobility in such frameworks renders classical alternating optimization impractical for per-coherence interval reconfiguration. To mitigate such issues, the STAR-RIS phase-shift and energy-splitting design is formulated as a constrained, non-convex sum-rate maximization problem with closed-form maximum ratio transmission beamforming and fixed NOMA power allocation. To circumvent the per-interval re-optimization burden, a deep reinforcement learning (DRL) approach is adopted that maps observed channel realizations to STAR-RIS configurations through a single forward pass. Specifically, Beta-Space Soft Actor-Critic (SAC-BSE), a maximum entropy DRL agent, is proposed. Simulation results, with two NOMA-multiplexed users on each STAR-RIS branch, confirm rapid convergence, limit the sum-rate degradation to roughly 10\% across a 128-fold user-speed range, and yield consistent gains over OTFS-only, NOMA-only, STAR-RIS-only, fixed-split, and mode-switching baselines as transmit power and the number of STAR-RIS elements increase.
• 中文摘要 本文考虑了一个下行通信框架，该框架由一个同时传输和反射的可重构智能表面（STAR-RIS）组成，辅以正交时频空间（OTFS）和非正交多址（NOMA）技术。此外，此类框架中的延迟-多普勒迁移性使得经典交替优化在每相干区间重构中不切实际。为缓解此类问题，STAR-RIS相位移和能量分裂设计被表述为一个受限、非凸的求和率最大化问题，采用闭式最大比值传输波束成形和固定NOMA功率分配。为规避每区间的再优化负担，采用了深度强化学习（DRL）方法，通过单次正向传递将观察到的信道实现映射为STAR-RIS配置。具体来说，提出了Beta-Space Soft Actor-Critic（SAC-BSE），一种最大熵DRL代理。模拟结果显示，每个STAR-RIS分支配备两个NOMA复用用户，确认收敛速度快速，用户速度范围内的和率降级限制在约10%左右，并且随着发射功率和STAR-RIS元件数量增加，在OTFS仅、仅NOMA、STAR-RIS、固定分流和模态切换基线上均有稳定的提升。

Extremely Sparse Supervision Incentivizes Reasoning Ability

极度稀疏的监督激励推理能力

• Authors: Zhishuai Liu, Xingzi Xu, Mehmet Saygin Seyfioglu, Pan Xu, Karim Bouyarmane
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.04565
• Pdf link: https://arxiv.org/pdf/2609.04565
• Abstract Large language models demonstrate increasingly strong reasoning capabilities through effective post-training. Yet, prevailing post-training methods optimize over massive numbers of tokens, implicitly assuming that effective learning must be token-intensive. We revisit this assumption in the on-policy distillation (OPD) setting, which naturally admits dense teacher supervision at every generated token. Using the Qwen3 family, we discover a counter-intuitive phenomenon: reasoning can be effectively incentivized by an extremely small fraction of generated tokens--as few as one or two tokens per reasoning trajectory, corresponding to only 0.05% of all tokens. Surprisingly, this sparse supervision in most cases matches or surpasses full-token training in improving reasoning ability, despite excluding the vast majority of generated tokens from the training objective. This phenomenon is consistently observed across nine teacher--student configurations spanning different model scales on mathematical reasoning tasks, and is further validated on coding reasoning, Llama models and Proximal Policy Optimization (PPO)-based reinforcement learning with verifiable reward (RLVR). Interestingly, such extremely sparse supervision may be closer to the natural learning process: rather than correcting every step word by word, one reflects on a few critical reasoning steps, updates prior understanding, and continues the trial-and-error, avoiding micro-level corrections while remaining remarkably effective. Overall, our results challenge the assumption that effective post-training must be token-intensive and point to a new direction for understanding and designing more efficient post-training algorithms.
• 中文摘要 大型语言模型通过有效的后期训练展现出越来越强的推理能力。然而，主流的后训练方法在大量令牌上进行优化，隐含地假设有效学习必须是令牌密集型的。我们在策略提炼（OPD）环境中重新审视这一假设，该环境自然允许每个生成的令牌进行密集的教师监督。利用Qwen3家族，我们发现了一个反直觉的现象：推理可以通过极少数生成的令牌——每条推理轨迹中一两个令牌——仅占所有令牌的0.05%，从而有效激励推理。令人惊讶的是，这种稀疏的监督在大多数情况下在提升推理能力方面与全令牌训练相当甚至超过，尽管排除了绝大多数生成的令牌。这一现象在九种教师-学生配置中持续观察到，涵盖不同模型尺度的数学推理任务，并在编码推理、Llama模型以及基于近端策略优化（PPO）的可验证奖励强化学习（RLVR）中得到了进一步验证。有趣的是，这种极度稀疏的监督可能更接近自然学习过程：与其逐字纠正每一步，不如反思几个关键推理步骤，更新先前理解，并继续试错，避免微观层面的纠正，同时保持显著效果。总体而言，我们的结果挑战了有效后训练必须耗费token的假设，并为理解和设计更高效的后训练算法指明了新的方向。

ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying

ConsensusBench：通过结果奖励密度化实现LLM推理的共识节点基准

• Authors: Shi-Qi Yan, Chao-Hong Tan, Qian Chen, Wen Wang, Xiangang Li, Zhen-Hua Ling
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.04648
• Pdf link: https://arxiv.org/pdf/2609.04648
• Abstract Reinforcement learning (RL) has become one of the primary paradigms for reasoning enhancement of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) and related algorithms have demonstrated strong performance with outcome-level rewards. However, these methods depend solely on the final answer, without feedback regarding which intermediate steps contribute to success or failure. As task complexity and reasoning trajectory length increase, such sparse final-answer rewards become increasingly insufficient. To address this limitation, we introduce ConsensusBench, a novel dataset designed to provide rule-based process-level signals. We posit that a correct final answer relies on a small set of intermediate conclusions throughout the reasoning process, which can be seen as a verifiable sub-outcome. We identify these sub-outcomes by filtering correct trajectories from N rollouts and clustering semantically equivalent intermediate statements. We call these clustered statements as Consensus Nodes. By integrating a rule-based process reward derived from these nodes into GRPO-style algorithms, we develop a new reinforcement learning signal named ConsensusPR. It directly reduces the reward sparsity of outcome reward across long reasoning trajectories. To facilitate systematic process-level evaluation, we introduce three metrics to our benchmark: Final Answer Accuracy (Acc), Node Coverage Rate (NCR), and Tokens per Node (TPN). Experiments across AIME 2024, AIME 2025, GSM8K, MATH-500, and our ConsensusBench demonstrate that the proposed method consistently surpasses GRPO-style approaches, highlighting the practical value of consensus nodes in guiding reasoning.
• 中文摘要 强化学习（RL）已成为大型语言模型（LLMs）推理增强的主要范式之一。特别是，群体相对策略优化（GRPO）及相关算法在结果级奖励方面表现出强劲表现。然而，这些方法仅依赖最终答案，没有反馈哪些中间步骤有助于成功或失败。随着任务复杂度和推理轨迹长度的增加，这种稀疏的最终答案奖励变得越来越不足。为解决这一限制，我们引入了ConsensusBench，这是一个设计用于提供基于规则的过程级信号的新型数据集。我们假设，正确的最终答案依赖于推理过程中少数中间结论，这些结论可视为可验证的子结果。我们通过从N个展开中筛选正确轨迹并聚类语义等效的中间语句来识别这些子结果。我们将这些聚类语句称为共识节点。通过将这些节点衍生的基于规则的过程奖励集成到GRPO风格算法中，我们开发了一种名为ConsensusPR的新强化学习信号。它直接降低了在长推理轨迹中结果奖励的稀疏性。为促进系统性过程级评估，我们在基准中引入了三个指标：最终答案准确率（Acc）、节点覆盖率（NCR）和每个节点的代币数（TPN）。在AIME 2024、AIME 2025、GSM8K、MATH-500和我们的ConsensusBench上的实验表明，所提方法始终超越GRPO风格方法，强调共识节点在指导推理中的实用价值。

LookThere! Sparse Vision by Reinforced Selection

看那里！《稀疏视野》由强化选择制作

• Authors: Sreehari Rammohan, Yousef Yassin, Anthony Fuller, Junfeng Wen, Carl Vondrick, Evan Shelhamer
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.04698
• Pdf link: https://arxiv.org/pdf/2609.04698
• Abstract Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.
• 中文摘要 视觉变换器通常将每个图像符号视同等重要，但对于大多数计算机视觉任务，只需一小部分。自适应计算方法通过选择处理哪些符号加速推理，但现有方法在极度稀疏性下难以实现，且需要启发式方法，如符号多样性和注意力评分，这些方法可能无法推广。我们通过LookThere解决了这些局限性，通过端到端强化学习框架实现了性能与计算权衡的新前沿，该框架联合训练浅输入选择器和深度表示提取器。选择器学习观察位置，提取器学习观察目标，通过仅选择值得处理的内容而节省计算，无需依赖辅助信号。我们证明LookThere仅选择任务特定输入，在高分辨率环境（交通标志、台球）中表现出色，且在低至0.2%的输入下保持准确性。它能够在多个任务和模型上推广，包括全局识别（ImageNet分类）、局部识别（ADE20K分割）、零样本分类（通过蒸馏）和回归（计数）。在所有场景下，LookThere超越了最先进的选择，提供了一个通用且可扩展的专门高效自适应计算框架。

Training Large Language Models for Small-Molecule Design with Synthetic Task Scaling

利用合成任务缩放训练大语言模型以实现小分子设计

• Authors: Frank Hu, Shriram Chennakesavalu, Zichen Wang, Patricia Suriana, Bodhi Vani, Kirill Shmilovich, Kangway Chuang, Colin Grambow
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.04735
• Pdf link: https://arxiv.org/pdf/2609.04735
• Abstract Designing viable drug candidates requires searching a combinatorially large and rugged chemical space for molecules that satisfy multiple, often competing, objectives. Large language models (LLMs) provide a useful generative prior for this problem because of their representational capacity, reasoning ability, and flexibility when incorporating information from the external environment. While reinforcement learning from verifiable rewards (RLVR) can be used to improve the capabilities of LLMs, many chemically relevant scoring functions require hours or even days per evaluation, making them prohibitively expensive to use directly during online training. Here, we investigate whether LLMs can learn molecular design strategies from cheaper synthetic tasks that generalize to expensive molecular lead optimization settings. We find that curriculum-based training recipes that gradually incorporate more challenging synthetic design tasks enable strong performance that surpasses that of much larger frontier models on structure-based lead optimization. Our results suggest that scaling post-training using synthetic tasks is an effective strategy for adapting LLMs to high-cost experimental scenarios that are too expensive to directly train on.
• 中文摘要 设计可行的药物候选需要在组合学上庞大且坚固的化学空间中寻找满足多个且常常相互竞争目标的分子。大型语言模型（LLMs）因其表征能力、推理能力以及在整合外部环境信息时的灵活性，为这一问题提供了有用的生成先验。虽然可验证奖励强化学习（RLVR）可用于提升LLMs的能力，但许多化学相关的评分函数每次评估需要数小时甚至数天，因此在线培训中直接使用成本过高。本研究中，我们探讨LLMs是否能从更便宜的合成任务中学习分子设计策略，这些任务推广到昂贵的分子导向优化设置。我们发现，基于课程的训练方案逐步纳入更具挑战性的合成设计任务，使得在结构导向优化上表现优于更大型前沿模型。我们的结果表明，使用合成任务进行训练后扩展规模，是将LLM适配到高成本实验场景的有效策略，这些场景成本过高，无法直接训练。

Inventory-Grounded Policy-Level Optimization for Training-Free AI Search

基于库存的策略级优化，实现无训练AI搜索

• Authors: Wei Zhou, Tiandeng Wu, Jiandong Ding, Zhufeng Fan, Yi Cao
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2609.04813
• Pdf link: https://arxiv.org/pdf/2609.04813
• Abstract Early in deployment, an AI search system typically operates over a frequently updated product catalog, so the available items and their properties cannot be treated as stable knowledge that can be encoded in fixed prompts or strategies. Fine-tuning, reinforcement learning, and static prompt patches fit poorly: labels are scarce, rewards drift with inventory, model releases are costly, and prompt fixes quickly stale. We present Inventory-Grounded Policy-Level Optimization (IGPO), a training-free approach for fixed AI search pipelines. IGPO separates policy from environment facts: it learns Policy Guidelines for acting on runtime inventory evidence rather than memorizing available items. Online, IGPO grounds each query by probing the inventory and constructing an inventory portrait, then injects relevant Policy Guidelines into the retrieval and selection prompts. Offline, stochastic rollouts are grouped by query -- mixed outcome groups directly yield contrastive signal, and an inventory-guided exploration loop distinguishes missed retrieval routes from cases where no matching support is found under the observed inventory evidence. Since May 2026, IGPO has been deployed in a commercial smart-assistant AI search system. A 14-day online A/B test of the complete IGPO treatment shows a 3.17% relative CTR lift and a 38.9% reduction in audited bad cases.
• 中文摘要 部署初期，AI搜索系统通常运行在频繁更新的产品目录上，因此可用商品及其属性无法被视为稳定知识，无法编码在固定提示或策略中。微调、强化学习和静态提示补丁难以配合：标签稀缺，奖励随库存漂移，模型发布成本高昂，提示修复很快过时。我们介绍库存基础策略级优化（IGPO），这是一种无需训练的固定AI搜索流程方法。IGPO将策略与环境事实分离：学习基于运行时库存证据而非记忆可用物品的策略指南。在线时，IGPO通过探查库存构建库存画像来为每个查询提供基础，然后将相关策略指南注入检索和选择提示中。离线时，随机部署按查询分组——混合结局组直接产生对比信号，库存引导的探索循环区分遗漏检索路径与观察到的库存证据下无匹配支持的案例。自2026年5月起，IGPO已部署于商业智能助手AI搜索系统中。完整IGPO处理的14天在线A/B测试显示，CTR相对提升3.17%，审计不良案例减少38.9%。

Reinforcement Learning for improving Large Language Models' Catalan text simplification capabilities

强化学习以提升大型语言模型的加泰罗尼亚语文本简化能力

• Authors: Arnau Ayguadé Domingo, Stefan Bott, Horacio Saggion
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04823
• Pdf link: https://arxiv.org/pdf/2609.04823
• Abstract Although automatic text simplification (ATS) is critical for accessibility, its progress has not matched the rapid evolution of broader natural language processing techniques. This paper investigates the application of reinforcement learning (RL) to improve the quality of ATS for low-resource languages using Large Language Models (LLMs). The paper introduces a novel reward function, designed to guide LLMs toward a targeted simplification style with Group Relative Policy Optimization (GRPO), that combines the SARI metric with specific penalty components. The effectiveness of GRPO with this reward function is motivated and demonstrated by post-training IberianLLM-7B-Instruct on the ASSET dataset. After post-training on the English ASSET, the model's ATS performance improves on two curated Catalan benchmarks while also successfully suppressing previously observed negative behaviors. Cross-lingual transfer learning is explored by translating ASSET into Catalan and Spanish and post-training the model on each version, but these fail to show a significant improvement on the out-of-domain benchmark.
• 中文摘要 尽管自动文本简化（ATS）对可及性至关重要，但其进展未能跟上更广泛自然语言处理技术的快速演进。本文探讨了利用大型语言模型（LLMs）在低资源语言中强化学习（RL）提升ATS质量的应用。论文引入了一种新颖的奖励函数，旨在通过群相对策略优化（GRPO）引导LLM朝向有针对性简化风格发展，该函数将SARI指标与特定惩罚成分结合。GRPO在该奖励函数下的有效性得到了通过在ASSET数据集上进行后训练IberianLLM-7B-Ininstruction的激励和验证。在英文ASSET上进行后训练后，该模型的ATS表现在两个精心策划的加泰罗尼亚基准测试基础上有所提升，同时成功抑制了先前观察到的负面行为。通过将ASSET翻译成加泰罗尼亚语和西班牙语，并在每个版本上进行后训练，探索跨语言迁移学习，但这些方法未能在域外基准测试上显示出显著提升。

Generating Constructive Feedback on Stories via Reinforcement Learning

通过强化学习生成对故事的建设性反馈

• Authors: Maja Stahl, Timon Ziegenbein, Henning Wachsmuth
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.04824
• Pdf link: https://arxiv.org/pdf/2609.04824
• Abstract Constructive feedback is crucial for creative writers to refine their storytelling abilities. Since receiving feedback from human experts is often costly and time-intensive, large language models (LLMs) offer a scalable and efficient alternative as automatic writing assistants. Despite their potential, research indicates that LLM-generated feedback is often generic, lacks actionability, and fails to identify which writing issue is most critical. To address these limitations, we present a reinforcement learning approach that steers LLMs to generate constructive feedback without the need for ground-truth feedback. We train our model using group relative policy optimization (GRPO) with a novel multi-component reward function aiming at constructiveness: it prioritizes feedback that is uniquely tailored to the story, helps to improve story quality, and addresses the most critical writing issue. In automatic and human evaluation across three story corpora, our approach outperforms state-of-the-art LLMs (including Gemini) and competitive baselines. We find that providing actionable suggestions is the main driver of feedback constructiveness.
• 中文摘要 建设性反馈对于创意写作者提升叙事能力至关重要。由于从人类专家那里获得反馈往往成本高且耗时，大型语言模型（LLMs）作为自动写作助手提供了一种可扩展且高效的替代方案。尽管具有潜力，研究表明LLM生成的反馈往往泛泛而谈，缺乏可操作性，且未能识别哪种写作问题最为关键。为解决这些局限性，我们提出了一种强化学习方法，引导LLM在无需真实反馈的情况下产生建设性反馈。我们使用群体相对策略优化（GRPO）训练模型，采用一种新颖的多组件奖励函数，旨在实现建设性：它优先考虑针对故事量身定制的反馈，帮助提升故事质量，并解决最关键的写作问题。在三种故事语料库的自动和人工评估中，我们的方法优于最先进的LLM（包括Gemini）和竞争基线。我们发现，提供可操作的建议是反馈建设性的主要驱动力。

CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Hierarchical Skill Evolution

CoSkill：层级技能进化的推理与元技能代理的联合强化学习

• Authors: Jinyuan Feng, Dongmin Li, Yiqun Chen, Yang Gao, Xing Chen, Huimu Wang, Zhiqiang Pu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04865
• Pdf link: https://arxiv.org/pdf/2609.04865
• Abstract Skill libraries improve the sample efficiency of agentic reinforcement learning (RL) by enabling large language model (LLM) agents to reuse procedural knowledge. Yet existing paradigms exhibit structural shortcomings: they either decouple skill evolution from policy optimization or instantiate meta-skills as fixed workflows. Both treat skills as passive objects to be managed, limiting the flexible evolution of skills and their co-adaptation with the reasoning agent. To address the limitations, we propose CoSkill, a unified multi-agent RL framework that recasts the static meta-skill workflow as a learnable Meta-Skill Agent and jointly trains it with a Reasoning Agent over a hierarchical skill library. By modeling the Reasoning and Meta-Skill Agents as a cooperative team sharing a single backbone, CoSkill enables end-to-end co-adaptation: the Reasoning Agent conditions its actions on a retrieved task skill and step skills selected from its child set, while its task performance guides the Meta-Skill Agent in refining those step skills. Experiments on ALFWorld and WebShop show that CoSkill substantially outperforms prior skill-based and RL baselines, achieving success rates of 98.4% and 90.6%, respectively (+3.5 and +6.2 pp). As shown in Figure 1, CoSkill achieves superior early-stage sample efficiency, asymptotic performance, and wall-clock efficiency. Our code is available at this https URL.
• 中文摘要 技能库通过使大型语言模型（LLM）智能体能够重用程序知识，提高了智能体强化学习（RL）的样本效率。然而，现有范式存在结构性缺陷：它们要么将技能演进与策略优化解耦，要么将元技能实例化为固定的工作流。两者都将技能视为被动对象，限制了技能的灵活演进及其与推理代理的协同适应。为解决这些局限性，我们提出了CoSkill，一个统一的多智能体强化学习框架，将静态元技能工作流程重塑为可学习的元技能代理，并通过层级技能库与推理代理联合训练。通过将推理智能体和元技能代理建模为共享单一骨干的协作团队，CoSkill实现端到端的协同适应：推理智能体根据从子集中选取的任务技能和步骤技能来决定其行动，而其任务表现则指导元技能代理完善这些步骤技能。在ALFWorld和WebShop上的实验显示，CoSkill远超以往基于技能和强化学习的基线，分别实现了98.4%和90.6%的成功率（+3.5和+6.2 pp）。如图1所示，CoSkill在早期阶段的样本效率、渐近表现和墙时钟效率方面均表现优异。我们的代码可在此 https 网址获取。

Reinforcement Learning for Sequential Solar PV Policy Design under Uncertainty: An Agent-Based Approach

在不确定性下顺序太阳能光伏政策设计的强化学习：基于主体的方法

• Authors: Iias Faiud, Jonaid Shianifar, Michael Schukat, Karl Mason
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.04880
• Pdf link: https://arxiv.org/pdf/2609.04880
• Abstract Designing effective and fiscally sustainable policies for solar photovoltaic (PV) adoption requires balancing adoption gains against public expenditure under uncertainty and heterogeneous decision-making. This study formulates PV policy design as a sequential decision problem and integrates reinforcement learning (RL) with a stochastic agent-based model (ABM) that simulates yearly solar PV adoption under uncertainty. A policymaker agent selects annual incentives, including capital grants, subsidised loan rates, and feed-in tariffs, over a 16-year horizon. Adoption--cost trade-offs are explored by varying policy preferences within a scalarised reward framework. Policies are learned using PPO, SAC, and TD3 and evaluated under stochastic simulation. The results show that this approach produces a clear trade-off structure: the highest-adoption policy (TD3, $w_{\text{cost}}=0.5$) achieves approximately 4,145 adopters at a cost of EUR 41.73 million, while the lowest-cost policy (PPO, $w_{\text{cost}}=2.0$) reduces expenditure to EUR 7.27 million with 2,682 adopters. The balanced policy (PPO, $w_{\text{cost}}=1.6$) achieves 3,495 adopters at a cost of EUR 22.47 million. Across algorithms, consistent trade-off patterns are observed, indicating robustness of the adoption--cost relationship. Compared with static baseline policies, the RL framework explores a broader range of policy configurations. These findings demonstrate the potential of RL as a flexible tool for adaptive policy design under uncertainty.
• 中文摘要 设计有效且财政可持续的太阳能光伏（PV）采纳政策，需要在不确定性和异质决策下权衡采纳收益与公共支出。本研究将光伏政策设计提出为顺序决策问题，并将强化学习（RL）与基于随机代理的模型（ABM）整合，模拟每年太阳能光伏在不确定性下的采用情况。政策制定者代理人在16年范围内选择年度激励措施，包括资本拨款、补贴贷款利率和上网电价。通过在标量化奖励框架内的不同政策偏好，探索采纳与成本权衡。政策通过PPO、SAC和TD3学习，并在随机模拟下进行评估。结果表明，这种方法产生了一个明确的权衡结构：最高采纳策略（TD3,4$w_{\text{cost}}=0。 5$）实现约4,145名采用者，成本为4,173万欧元;最低成本政策（PPO，$w_{\text{cost}}=2.0$）将支出降至727万欧元，采用者为2,682人。平衡政策（PPO，$w_{\text{cost}}=1.6$）以2,247万欧元实现3,495名采用者。在各算法中，观察到一致的权衡模式，表明采纳-成本关系的稳健性。与静态基线策略相比，RL框架探索了更广泛的策略配置。这些发现展示了RL作为在不确定性下灵活自适应策略设计工具的潜力。

Artificial Intelligence in Equity and Crypto Markets: Progress, Profitability Evidence, and the Limits of Automated Investing

股票和加密市场中的人工智能：进展、盈利证据及自动化投资的局限性

• Authors: Linsen Zhu, Mengqing Cai
• Subjects: Subjects: Artificial Intelligence (cs.AI); Portfolio Management (q-fin.PM); Trading and Market Microstructure (q-fin.TR)
• Arxiv link: https://arxiv.org/abs/2609.04917
• Pdf link: https://arxiv.org/pdf/2609.04917
• Abstract Artificial intelligence (AI) now supports investment workflows from data and prediction through research, portfolios, execution, and tool use. Technical capability, however, is not evidence of investment profitability. This critical state-of-the-art review examines public research available through 31 August 2026 on listed equities, exchange-traded funds, centralized crypto spot, perpetual futures, and on-chain markets. We organize evidence with an alpha-translation chain: point-in-time information must yield a stable signal, feasible positions, executable orders, and risk-adjusted returns after costs. Across machine learning, time-series foundation models, financial language models, reinforcement learning, and agents, the examined record shows real but mainly upstream progress in prediction, text processing, portfolio design, and workflow integration. Evidence is thinner for durable net performance. Temporal contamination, repeated selection, survivorship, weak benchmarks, implementation costs, venue mechanics, and capacity can break translation to net alpha. Strong historical results coexist with predictor decay, corrected look-ahead failures, mixed prospective evidence, and few audited live-capital records. Crypto adds informative state but requires separate treatment of spot, perpetual, and decentralized cash flows and execution. Within the public evidence examined here, no general AI architecture is shown to deliver persistent, cross-regime, capacity-aware net alpha. More credible claims require point-in-time data and models, decision-aligned objectives, joint portfolio--execution evaluation, controlled adaptation, prospective tests, and authority-matched governance. These conditions can improve evidence and implementation; they do not guarantee profit.
• 中文摘要 人工智能（AI）现支持从数据和预测到研究、投资组合、执行和工具使用等投资工作流程。然而，技术能力并不代表投资盈利能力。这份关键的尖端回顾审查了截至2026年8月31日的公开研究，涵盖上市股票、交易所交易基金、中心化加密现货、永年期货和链上市场。我们用阿尔法翻译链组织证据：点点信息必须提供稳定信号、可行头寸、可执行订单和扣除成本后的风险调整回报。涵盖机器学习、时间序列基础模型、金融语言模型、强化学习和代理，所审查的记录显示了预测、文本处理、投资组合设计和工作流程集成方面的真实但主要向上游进展。持久净绩效的证据较为薄弱。时间污染、重复选择、存活率、弱基准、实施成本、场地机制和容量都可能破坏转化为净阿尔法。强劲的历史结果与预测器衰减、纠正的前瞻失败、混合的前瞻性证据以及少量经审计的活资本记录共存。加密增加了信息状态，但需要对现状、永续和去中心化现金流及执行进行单独处理。在此处审查的公开证据中，没有任何通用人工智能架构能够实现持久的、跨体制的、具容量感知的净阿尔法。更具可信度的主张需要点点数据和模型、决策对齐目标、联合投资组合执行评估、受控适应、前瞻性测试和权威匹配治理。这些条件可以提升证据和实施;但它们不保证盈利。

Compositional Reward Models for Conditional Medical Image Generation

条件医学图像生成的合成奖励模型

• Authors: Aayush Kumar Tyagi, Prathosh A.P., Mausam
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.05028
• Pdf link: https://arxiv.org/pdf/2609.05028
• Abstract Acquiring high quality annotated medical image data is critical for training deep learning models; however, annotation is expensive, time consuming, and requires domain expertise. Conditional diffusion models, such as ControlNet, offer an alternative by generating images conditioned on semantic masks and text. However, existing approaches fail to capture fine grained properties (e.g., intensity and texture), as well as semantic consistency expected by domain experts, limiting their effectiveness for downstream tasks. Recent attempts to address these issues using reinforcement learning fine-tuning remain limited due to the reliance on a single scalar reward, which conflates diverse failure modes and provides weak corrective signals. We propose PRISM, a Compositional Reward Model (CRM) framework for conditional medical image generation. Instead of assigning a single reward, we decompose image quality into verifier grounded stages, each evaluating a distinct aspect of correctness from fine to coarse properties, including low level attributes (intensity and texture), structural alignment with conditioning inputs, and high level semantic fidelity. These stage wise rewards are composed through a Hierarchical Constrained Propagation (HCP) mechanism that enforces a fine to coarse notion of correctness, ensuring that lower level deficiencies are resolved before higher level rewards are accrued, preventing easier objectives from masking critical failures. We evaluate PRISM across three datasets spanning diverse medical imaging tasks: PanNuke (multi-class cell segmentation), CeDeM (villi/crypt detection and measurement), and ISIC (skin lesion classification). Training downstream models with data generated by PRISM yields improvements over closest baselines, including a 2.3% increase in mDice on PanNuke, a 8.5% reduction in Mean Relative Error (MRE) on CeDeM, and increases ISIC F1 by 5.9%.
• 中文摘要 获取高质量的注释医学图像数据对于训练深度学习模型至关重要;然而，注释成本高、耗时且需要领域专业知识。条件扩散模型，如ControlNet，通过生成基于语义掩模和文本的图像，提供了另一种选择。然而，现有方法未能捕捉到领域专家期望的细粒度属性（如强度和纹理），以及语义一致性，限制了其在后续任务中的有效性。由于依赖单一标量奖励，导致多种失败模式混淆且提供弱纠正信号，近期尝试通过强化学习微调解决这些问题的尝试仍然有限。我们提出了PRISM，一种用于条件医学图像生成的合成奖励模型（CRM）框架。我们不分配单一奖励，而是将图像质量分解为验证者基于多个阶段，每个阶段评估从细到粗的正确性不同方面，包括低层属性（强度和纹理）、结构与条件输入的对齐以及高层语义忠实度。这些阶段级奖励通过层级约束传播（HCP）机制组成，强制执行从细到粗的正确性概念，确保在获得更高奖励前解决低层缺陷，防止更简单的目标掩盖关键失误。我们在涵盖多种医学影像任务的三个数据集中评估PRISM：PanNuke（多类细胞切割）、CeDeM（绒毛/隐穴检测与测量）和ISIC（皮肤病变分类）。使用PRISM生成的数据训练下游模型，相较于最接近的基线有所改进，包括PanNuke上mDice增加了2.3%，CeDeM上平均相对误差（MRE）减少了8.5%，并使ISIC F1提升了5.9%。

Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG

超越未压缩的压缩：RAG软上下文压缩的两阶段训练配方

• Authors: Shuyu Guo, Shuo Zhang, Zhaochun Ren
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.05152
• Pdf link: https://arxiv.org/pdf/2609.05152
• Abstract Retrieval-Augmented Generation (RAG) enhances language models with external knowledge, but the lengthy retrieved context inflates the input and degrades inference efficiency. Soft context compression encodes each document into a substantially shorter embedding sequence. However, most existing approaches are trained by distilling outputs from uncompressed RAG systems, inherently limiting their performance relative to the original model. To address this limitation, we propose DEX-Comp, a two-stage training recipe: Pure Distillation warm-starts the compression model on the uncompressed RAG's correct responses only, and Hard Exploration then runs reinforcement learning solely on queries the uncompressed RAG fails, forcing the model to explore computation patterns better suited to compressed representations. On five open-domain QA benchmarks at retrieval depths from top-5 to top-30, DEX-Comp compresses retrieved contexts by $16\times$ and accelerates inference by $4\times$--$24\times$, while achieving performance comparable to or exceeding the uncompressed RAG baseline across retrieval depths. Ablations and evaluations across diverse datasets and backbones further confirm the contribution of each stage and the generalization of our approach.
• 中文摘要 检索增强生成（RAG）通过外部知识增强语言模型，但检索上下文冗长使输入膨胀，降低推理效率。软上下文压缩将每个文档编码为显著更短的嵌入序列。然而，大多数现有方法通过蒸馏未压缩RAG系统的输出来训练，固有地限制了其相对于原始模型的性能。为解决这一限制，我们提出了DEX-Comp，一个两阶段训练方案：纯蒸馏仅对未压缩RAG的正确响应热启动压缩模型，硬探索仅对未压缩RAG失败的查询进行强化学习，迫使模型探索更适合压缩表示的计算模式。在五个开放域质量保证基准测试中，检索深度从前5到前30，DEX-Comp将检索上下文压缩速度提升16美元，推断加速4美元至24倍，同时在各检索深度下实现与未压缩RAG基线相当或超过的表现。针对不同数据集和骨干的消除和评估进一步证实了各阶段的贡献及我们方法的推广性。

Morphology and actuation as inductive biases in robotic hand manipulation

形态学与驱动作为机器人手操作中的归纳偏置

• Authors: Zalán Tari, Eszter Birtalan, Péter Polcz, Miklós Koller
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.05206
• Pdf link: https://arxiv.org/pdf/2609.05206
• Abstract Robotic hands vary widely in anatomical fidelity and mechanical complexity, and these structural choices influence the coordination of joint motions and the difficulty of controlling the system. A unified framework is presented in which the kinematic and actuation stages are analysed separately and in composition, through the conditioning of the task Jacobian, the actuation matrix, and their product. It is applied to two hands representing opposing design philosophies, the Shadow Dexterous Hand and the Anatomically Correct, Biomechatronic Hand, along four morphological aspects: joint axis geometry, actuator-to-DOF ratio, coupling architecture, and authority distribution. All parameters are derived from the hands' canonical digital representations. Anatomical fidelity carries no uniform advantage: oblique axes improve thumb conditioning but leave the long fingers worse conditioned than the orthogonal-axis design, while the branching tendon network improves the effective control mapping at every long finger and worsens it significantly at the thumb, where actuator authority is concentrated on thumb opposition. Predictions derived from these metrics are evaluated against reinforcement learning experiments using PPO, DDPG+HER, and TQC+HER, across three different tasks.
• 中文摘要 机器人手在解剖精度和机械复杂度上差异很大，这些结构选择影响关节运动的协调和系统控制难度。呈现了一个统一框架，通过对任务雅可比矩阵、执行矩阵及其乘积的条件化，分别分析运动学和执行阶段的组合。该框架应用于代表对立设计理念的两只手——阴影灵巧手和解剖学正确、生物机电一体化手，结合四个形态方面：关节轴几何、执行器与深度比、耦合结构和权威分布。所有参数均源自手的标准数字表示。解剖学的忠实度没有统一优势：斜轴改善拇指条件反射，但长手指的条件反射不如正交轴设计;而分支腱网络则提升了每个长手指的有效控制映射，而拇指则显著恶化，因为执行器权重集中在拇指对抗上。基于这些指标的预测通过使用PPO、DDPG+HER和TQC+HER进行强化学习实验，在三种不同任务中进行评估。

First Things First: Teaching LLM-Based Agents to Prioritize Must-Haves before Nice-to-Haves

首先：教基于LLM的代理优先考虑必备功能，而不是“可拥有”的

• Authors: Tianjie Ju, Xinyue Xu, Wanxuan Sun, Lingxiao Diao, Gongshen Liu, Zhuosheng Zhang, Cheng Yang
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.05224
• Pdf link: https://arxiv.org/pdf/2609.05224
• Abstract Recent progress in multimodal large language models (MLLMs) has fueled significant enthusiasm in their potential to act as autonomous agents for real-world tasks. However, scenarios requiring agents to fulfill users' complex, structured requirements remain largely underexplored. In this work, we examine reasoning tasks under three distinct requirement scenarios: (i) Must-have requirements uniquely determine a unique feasible solution; (ii) Multiple answers satisfy the must-have requirements and are prioritized via the nice-to-have requirements; and (iii) No candidate solution satisfies the must-have requirements, in which case the agent should abstain from generating a response. We evaluate state-of-the-art MLLMs on 3,649 carefully constructed problems that reflect realistic service scenarios, including e-commerce, booking, and map-based or ride-hailing. Our evaluation reveals that existing MLLMs exhibit catastrophic failures in all scenarios. They frequently misinterpret task requirements, violate must-have requirements, and produce invalid solutions. To address this critical gap, we propose First Things First Reinforcement Learning FTF-rl that explicitly optimizes reasoning over multi-priority user requirements. Experimental results show that our method substantially improves the task success rate compared to strong baselines. Moreover, FTF-rl yields general effectiveness on popular logical and mathematical reasoning tasks, including LogicVista, MathVision, and InfoQA. Our findings suggest that enhancing requirement-aware reasoning capability provides a simple yet effective pathway to improve generalization of MLLM agents. Code and dataset are available at this https URL.
• 中文摘要 多模态大型语言模型（MLLM）的最新进展激发了人们对其作为现实任务自主代理的潜力的浓厚热情。然而，要求代理满足用户复杂结构化需求的场景仍然大多缺乏探索。本研究中，我们考察了三种不同需求场景下的推理任务：（i） 必须满足需求唯一确定唯一可行解;（ii） 多个答案满足必有需求，并通过“可拥有”需求优先排序;（iii） 没有候选解决方案满足必有需求，此时代理应避免生成响应。我们评估了3649个精心构建的问题，反映了真实服务场景，包括电子商务、预订以及基于地图或网约车的服务场景。我们的评估显示，现有MLLM在所有场景下都存在灾难性失败。它们经常误解任务需求，违反必有要求，并产生无效的解决方案。为弥补这一关键空白，我们提出了First Things First Reinforcement Learning FTF-rl，明确优化推理而非多优先级用户需求。实验结果显示，我们的方法相比强基线显著提升了任务成功率。此外，FTF-rl在流行的逻辑和数学推理任务中表现出普遍效果，包括LogicVista、MathVision和InfoQA。我们的发现表明，增强需求感知推理能力为提升MLLM代理的泛化提供了一条简单而有效的途径。代码和数据集可在此 https 网址获取。

GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexity

GUT：通过图复杂度量化和优化大型语言模型的推理不确定性

• Authors: Shuang Liang, Xin-Yu Hu, Xiang-Jun Ou, Shao-Qun Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.05284
• Pdf link: https://arxiv.org/pdf/2609.05284
• Abstract Recent years have witnessed great advances in the reasoning ability of Large Language Models (LLMs). However, the reasoning processes of LLMs often exhibit uncertainty, where LLMs often produce a proliferation of divergent branches at each reasoning step even when fed the same prompting inputs, and certain branches exhibit evidently incredible, even nonsensical, reasoning chains and results. In this paper, we propose the Graph-complexity-based UncerTainty (GUT) method for investigating the reasoning uncertainty of LLMs. The key idea of GUT is to characterize the potential branches of each reasoning chain with a directed acyclic graph, thereby ensuring that all potential branches are comprehensively covered within the graph space. Building upon this recognition, we further build two modules of GUT, that is, a Quantification (GUT-Q) module and an Optimization (GUT-O) module, for quantifying and reducing the reasoning uncertainty of LLMs, respectively. GUT-Q measures LLM reasoning uncertainty by approximating the reasoning space complexity with graph complexity. GUT-O implements uncertainty optimization by treating negative uncertainty as the reward function in reinforcement learning. Experimental results conducted on four LLMs and five datasets validate the effectiveness of GUT.
• 中文摘要 近年来，大型语言模型（LLM）推理能力取得了巨大进步。然而，LLMs的推理过程常常存在不确定性，即使输入相同的提示输入，LLM在每个推理步骤中仍会产生大量不同的分支，某些分支表现出令人难以置信甚至荒谬的推理链和结果。本文提出基于图复杂度的不确定性（Graph-complexity-Uncertainty，简称GUT）方法，用于研究LLMs的推理不确定性。GUT的核心思想是通过有向无环图刻画每个推理链的潜在分支，从而确保所有潜在分支在图空间内被全面覆盖。基于这一认识，我们进一步构建了两个GUT模块，即量化（GUT-Q）模块和一个优化（GUT-O）模块，分别用于量化和减少LLM的推理不确定性。GUT-Q通过近似推理空间复杂度与图复杂度来衡量LLM推理不确定性。GUT-O通过将负不确定性视为强化学习中的奖励函数来实现不确定性优化。对四个LLM和五个数据集的实验结果验证了GUT的有效性。

Online Change-point Detection for Cooperative Multi-Agent Reinforcement Learning

协作多智能体强化学习的在线变点检测

• Authors: Fatemeh Saberi Khomami, Julita Vassileva
• Subjects: Subjects: Multiagent Systems (cs.MA); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.05298
• Pdf link: https://arxiv.org/pdf/2609.05298
• Abstract Cooperative multi-agent reinforcement learning (MARL) systems rely on past experience for learning coordinated behaviour, but this experience may become unreliable if the environment or task objective changes during training. In such cases, agents first need a way to recognize that the situation has changed before deciding how to adapt. This paper studies online change-point detection for cooperative MARL using reward-derived signals. We propose \emph{Patterns of Past Rewards} (PPR), a lightweight algorithm-agnostic detector that smooths agents' return streams, highlights recent changes, and applies a statistical drift detector to flag significant shifts. We evaluate PPR in a custom Speaker-Listener environment based on the Multi-Agent Particle Environment under two controlled non-stationarity scenarios. Our results show a trade-off between detection speed and alarm stability. A smoothed-return baseline detects earlier but produces many repeated alarms. In contrast, applying the detector directly to raw returns often misses the shift. PPR offers a more balanced approach by limiting redundant detections while still identifying the controlled shifts. These findings highlight PPR as a lightweight, reward-based monitoring tool that enables cooperative MARL systems to reliably identify major changes during training.
• 中文摘要 合作多智能体强化学习（MARL）系统依赖以往经验学习协调行为，但如果训练过程中环境或任务目标发生变化，这种经验可能变得不可靠。在这种情况下，智能体首先需要识别情况发生变化，然后再决定如何适应。本文研究了利用奖励衍生信号进行合作式MARL的在线变点检测。我们提出了\emph{Patterns of Past Rewards}（PPR），一种轻量级的算法无关检测器，能够平滑代理的返回流，突出近期变化，并应用统计漂移检测器标记显著变化。我们在基于多智能粒子环境的自定义说话者-听众环境中，在两种受控非平稳场景下评估PPR。结果显示检测速度与警报稳定性之间存在权衡。平滑返回基线检测较早，但会产生许多重复警报。相比之下，直接应用于原始回波时，常常会错过该位移。PPR通过减少冗余检测，同时识别受控偏移，提供了更为平衡的方法。这些发现凸显了PPR作为一种轻量级、基于奖励的监测工具，使协同作战的MARL系统能够可靠地识别训练期间的重大变化。

Human-Human & Human-Robot Interaction Transformer (H2INT) for Robot Navigation in Dense and Uncertain Crowds

人机交互变压器（H2INT），用于在密集且不确定的人群中进行机器人导航

• Authors: Ao Shen, Kaixi Chen, Shiwei Liu, Fang Deng, Chen Chen
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.05300
• Pdf link: https://arxiv.org/pdf/2609.05300
• Abstract Safe robot navigation in dense crowds requires reasoning about pedestrian motion and how it may change in response to a robot. However, many learning-based approaches generate pedestrian motion independently of the robot or assume uniform reciprocity, omitting an important source of interaction uncertainty. This paper presents a Human-Human & Human-Robot Interaction Transformer (H2INT), a reinforcement learning framework that retains robot-conditioned changes in pedestrian motion during policy learning while allowing responsiveness to vary across pedestrians. Responsiveness affects the crowd dynamics when the robot is visible but is not supplied as a policy input; the policy must instead infer its consequences from robot-centered relative positions. A two-stage gated Transformer progressively encodes human-human and human-robot relations, while a recurrent policy captures their temporal evolution. A curriculum gradually reduces pedestrian responsiveness to increase interaction difficulty. Simulation experiments demonstrate improved navigation safety and robustness over representative baselines across response conditions and crowd densities, and show transfer without retraining to structurally distinct crowd-flow layouts. Ablations support the hierarchical relational encoding and gated updates. Real-robot deployment further verifies that the learned policy can operate with sparse observations in a physical environment.
• 中文摘要 在密集人群中安全导航需要推理行人运动及其可能因机器人响应而变化。然而，许多基于学习的方法独立于机器人生成行人运动，或假设互惠一致，忽略了重要的交互不确定性来源。本文提出了人-人-人交互变换器（H2INT），这是一种强化学习框架，在政策学习过程中保留机器人条件下的行人运动变化，同时允许响应性在不同行人之间变化。当机器人可见但未作为政策输入提供时，响应性会影响人群动态;政策必须从以机器人为中心的相对位置推断其后果。两阶段门控变换器逐步编码人机和人机关系，而循环策略则捕捉其时间演变。课程逐步降低行人响应性，提高互动难度。模拟实验展示了在响应条件和人群密度下，相较代表基线提升的导航安全性和鲁棒性，并显示无需再训练即可迁移到结构上明显不同的人群流布局。消融支持层级关系编码和门控更新。真实机器人部署进一步验证了所学策略在物理环境中可利用稀疏观察运行。

Large Language Models for HVAC Operations in Building Energy Systems: A Critical Review of Methods, Applications, and Deployment Readiness

建筑能源系统中暖通空调操作的大型语言模型：方法、应用及部署准备度的批判性综述

• Authors: Alexander Neubauer, Tianzhen Hong, Han Li, Mengbo Yu, Amin Darbandi, Yannick Fürst, Martin Kriegel
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.05314
• Pdf link: https://arxiv.org/pdf/2609.05314
• Abstract Building automation systems generate rich sensor data yet remain insight-poor because heterogeneous point naming, missing metadata, and fragmented documentation obstruct their operational use. This systematic review analyses and codes 66 peer-reviewed studies on large language models (LLMs) for HVAC operations published between 2023 and March 2026. Each study is classified across five application families and three LLM method families and assessed for evidence realism, deployment readiness, and the responsibility boundary between the LLM and physical HVAC decisions. The corpus is concentrated in building energy modelling (BEM, 32 of 66 papers), while load forecasting remains too sparse for subfield-level conclusions. Only four studies reach pilot-level evidence, and none reports sustained operational deployment. No study was classified as ready-now for industry adoption; three were near-term and 63 research-only. Nevertheless, several bounded, human-in-the-loop uses merit near-term trials, including point-name normalisation, document-grounded operator support, BEM workflow assistance, and advisory interfaces around physics-based controllers. Conventional machine learning (ML), model predictive control (MPC), reinforcement learning (RL) and ontology-based tools remain more adopted for high-frequency control, short-horizon numerical forecasting, and well-posed ontology mapping, while autonomous agentic operation and unvalidated occupant proxies remain research-stage. Current evidence therefore supports LLMs primarily as semantic and workflow layers rather than autonomous HVAC controllers. Future work should prioritise field-validated benchmarks, orchestration evaluation under operational constraints, and LLM-MPC/RL architectures with bounded latency and verifiable safety properties.
• 中文摘要 建筑自动化系统产生丰富的传感器数据，但由于异构点命名、缺失的元数据和零散的文档阻碍了其操作应用，洞察力仍然不足。本系统综述分析并编码了2023年至2026年3月间发表的66项关于暖通空调（HVAC）大型语言模型（LLMs）的同行评审研究。每项研究被分为五个应用家族和三个大型语言模型方法家族，并评估其证据真实性、部署准备度以及LLM与物理暖通空调决策之间的责任边界。语料库主要集中在建筑能源建模（BEM，66篇论文中的32篇），而负载预测仍过于稀少，难以做出子领域层面的结论。只有四项研究达到了试点级证据，且无一项报告实现了持续的运营部署。没有研究被归类为“现行准备就行”，供行业采用;其中三项为近期性研究，63项仅限研究。尽管如此，若干有界的人机应用值得近期试验，包括点名归一化、基于文档的操作员支持、BEM工作流辅助以及基于物理控制器的咨询接口。传统的机器学习（ML）、模型预测控制（MPC）、强化学习（RL）和本体论工具仍被更多采用用于高频控制、短视野数值预测和适态本体映射，而自主代理操作和未经验证的占用者代理仍处于研究阶段。目前证据支持LLM主要作为语义层和工作流层，而非自主暖通控制器。未来工作应优先考虑现场验证基准测试、运行约束下的编排评估，以及具有有限延迟和可验证安全属性的LLM-MPC/RL架构。

Keyword: diffusion policy

Dressing in Motion: A Human Motion-Aware Diffusion Policy for Robot-Assisted Dressing

移动穿衣：机器人辅助穿衣的人机动作感知扩散政策

• Authors: Haoxiang Sun, Fangyuan Wang, Songhao Huang, Justina Y. W. Liu, Jihong Zhu, Peng Zhou, David Navarro-Alarcon
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.04759
• Pdf link: https://arxiv.org/pdf/2609.04759
• Abstract Robotic dressing assistance is a promising solution for supporting older adults with physical impairments in daily living. However, dressing under human motion remains challenging, as complex garment--human contact and occlusions make it difficult to generate actions aligned with arm movements. In this letter, we propose a visuomotor policy that learns dressing skills from static expert demonstrations and generalizes to dynamic user-motion scenarios. A diffusion policy tailored to garment--human interaction geometry learns from partially observed point clouds with varied arm postures. We then introduce an object-centric representation based on PDE diffusion to capture the axial distribution of the arm. By sampling motion-relevant regions and registering them across consecutive observations, the proposed method approximates arm motion and reactively adapts the executed trajectory. We evaluate our method in simulation and a real-world human study involving nine participants, three garment types, and six arm-motion patterns. Results show that our method outperforms baselines in dressing progress, freedom of movement, and user comfort. The project website is this https URL.
• 中文摘要 机器人穿衣辅助是支持日常生活中身体障碍老年人的有前景解决方案。然而，在人体运动下穿衣仍然具有挑战性，因为复杂的服装——人体接触和遮挡使得与手臂动作对齐的动作难以产生。在这封信中，我们提出了一种视觉运动策略，通过静态专家演示学习穿衣技能，并推广到动态用户运动场景。针对服装——人类交互几何的扩散策略，从部分观测到的多样手臂姿势点云中学习。随后，我们引入基于偏微分方程扩散的对象中心表示，以捕捉手臂的轴向分布。通过采样与运动相关的区域并跨连续观察进行记录，拟议方法近似手臂运动并被动调整执行轨迹。我们在模拟和一项涉及九名参与者、三种服装类型和六种手臂动作图案的真实人体研究中评估了我们的方法。结果显示，我们的方法在着装进展、活动自由度和用户舒适度方面优于基线水平。项目网站为 https 网址。