Arxiv Papers of Today

生成时间: 2026-07-30 18:26:43 (UTC+8); Arxiv 发布时间: 2026-07-30 20:00 EDT (2026-07-31 08:00 UTC+8)

今天共有 24 篇相关文章

Keyword: reinforcement learning

Emergent Sparsity in Frozen Random CNN Feature Extractors for Deep Reinforcement Learning

深度强化学习中的冻结随机CNN特征提取器中的涌现稀疏性

Authors: Scott M. Norton
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neural and Evolutionary Computing (cs.NE)
Arxiv link: https://arxiv.org/abs/2607.26059
Pdf link: https://arxiv.org/pdf/2607.26059
Abstract We report a striking phenomenon: deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, without any sparsity-inducing objective. In the first fully-connected layer (FC1, $3{,}136 \to 64$), agents compress task-relevant information through as few as 1-3 neurons out of 64 for deterministic Pong (5-11 for stochastic Pong), while trainable CNNs activate 55-64 neurons under matched conditions. We establish four principal findings. First, FC1 sparsity scales with task complexity: 1-11 for Pong, 19-26 for Breakout, and $\sim$42 for Space Invaders. Width-scaling confirms this reflects task structure rather than a fixed capacity fraction. Second, within-game scaling emerges: three identical Pong seeds produce 5, 7, and 11 active neurons. The 5-neuron seed plateaus at $+14$ reward, while the others reach expert performance ($+18.4$, $+18.7$), suggesting the random projection's usable dimensionality bounds achievable performance. Third, ablation confirms necessity: removing these active neurons crashes performance across two PPO implementations and four games. Fourth, the information bottleneck commits early: a sweep shows the active set locks by 15-30M steps, while reward turns positive 35-105M steps later. A complementary finding in Breakout shows frozen and trainable CNNs reach competitive rewards via structurally different bottlenecks: frozen agents use 17-25 active neurons (participation ratio $\sim$10-14), while trainable agents use 51 (participation ratio $\sim$3.6). Finally, wherever input dimensionality dwarfs intrinsic task dimensionality, gradient descent on a frozen random projection may reveal the effective rank of the underlying problem without explicit sparsity machinery.
中文摘要 我们报告了一个显著现象：深度强化学习代理使用冻结、随机初始化的CNN特征提取器训练，能够自发发展出极为稀疏的全连通表示，且没有任何诱发稀疏的目标。在第一个全连通层（FC1，$3{，}136 \ 64$），主体通过确定性Pong中仅1-3个神经元压缩任务相关信息（随机Pong为5-11个），而可训练的CNN在匹配条件下激活55-64个神经元。我们建立了四个主要发现。首先，FC1稀疏度随任务复杂度增长：Pong为1-11，Breakout为19-26，Space Invaders为$\sim$42。宽度缩放证实这反映了任务结构，而非固定容量比例。其次，游戏内出现了缩放：三个相同的Pong种子产生5个、7个和11个活跃神经元。5个神经元种子在$+14$奖励时趋于稳定，而其他种子则达到专家级表现（$+18.4$，$+18.7$），表明随机投影的可用维度限制了可实现的性能。第三，消融确认了必要性：移除这些活跃神经元会导致两个PPO实现和四个游戏的性能崩溃。第四，信息瓶颈提前提交：扫描显示活跃组锁定了1500万步到3000万步，而奖励则在3500万到10.5万步后转为正。Breakout中的一项补充发现显示，冻结和可训练的CNN通过结构上不同的瓶颈获得竞争奖励：冻结的智能体使用17-25个活跃神经元（参与比\模拟美元10-14），而可训练的智能体使用51个（参与比$\模拟美元3.6）。最后，在输入维度远超任务维度的地方，在冻结随机投影上的梯度下降可能揭示潜在问题的有效秩，而无需显式稀疏机制。

Guess Where You Go: Generative Next Point-of-Interest Recommendation in Amap

猜猜你去哪里：Amap 生成性下一个兴趣点推荐

Authors: Penglong Zhai, Bowen Zheng, Jie Li, Yifang Yuan, Yue Liu, Sicong Wang, Mingyang Yin, Tingting Hu, Shuaijun Guo, Fanyi Di, Xin Li
Subjects: Subjects: Information Retrieval (cs.IR)
Arxiv link: https://arxiv.org/abs/2607.26073
Pdf link: https://arxiv.org/pdf/2607.26073
Abstract Generative retrieval enables recommender systems to retrieve items by generating compact item identifiers, but scaling it to industrial scenarios remains challenging due to redundant or colliding token assignments and insufficient integration of heterogeneous item signals. These challenges are particularly critical for next Point-of-Interest (POI) recommendation, where models must represent structured spatial entities, capture sequential mobility patterns, and produce predictions consistent with real user behavior. We propose Gwhere, an end-to-end industrial framework that integrates semantic identifier (SID) generation with LLM-based generative next POI recommendation. Gwhere first learns discriminative POI SIDs through a contrastive residual-quantization tokenizer that aligns textual, visual, spatial, and collaborative signals. Based on these SIDs, Gwhere adapts LLMs to mobility scenarios via continued pretraining on enriched spatio-temporal corpora, supervised fine-tuning, and Exposure-Aware Kahneman-Tversky Optimization (EAKTO), a reinforcement learning objective for behavioral preference alignment. Experiments on public datasets and Amap's large-scale industrial dataset demonstrate the effectiveness of Gwhere. The system has been deployed in Amap's homepage service under high-concurrency and low-latency constraints. Long-term online A/B tests show improvements of 5.83% in P-CTR and 6.20% in U-CTR over the production baseline. The implementation is publicly available at this https URL.
中文摘要 生成检索使推荐系统能够通过生成紧凑的项目标识符来检索项目，但由于冗余或冲突的令牌分配以及异构项目信号的整合不足，将其扩展到工业场景仍然具有挑战性。这些挑战对于下一阶段兴趣点（POI）推荐尤为关键，因为模型必须表示结构化的空间实体，捕捉连续的移动模式，并产生与真实用户行为一致的预测。我们提出了Gwhere，这是一个端到端的工业框架，将语义标识符（SID）生成与基于LLM的生成式次POI推荐相结合。Gwhere首先通过对比性残差量化分词器学习判别性POI SID，该工具对齐文本、视觉、空间和协作信号。基于这些SID，Gwhere通过对丰富时空语料库的持续预训练、监督式微调以及暴露感知Kahneman-Tversky优化（EAKTO），作为行为偏好对齐的强化学习目标，将LLM适配于移动场景。在公共数据集和Amap大规模工业数据集上的实验展示了Gwhere的有效性。该系统已部署在Amap首页服务中，采用高并发和低延迟约束。长期在线A/B测试显示，P-CTR提升5.83%，U-CTR提升6.20%。实现过程可在此 https URL 公开获取。

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

元学习奖励塑造以实现人类反馈的强化学习

Authors: Yunpeng Chu
Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.26094
Pdf link: https://arxiv.org/pdf/2607.26094
Abstract Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function $\Phi(x,y;\phi)$ across auxiliary tasks before RLHF training. The learned shaping produces a composite reward that preserves policy optimality while providing task-specific learning signals. Our meta-objective combines task discrimination, entropy regularization, and potential-based conservation for stable convergence. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization. Experiments on LLaMA-3-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and a score of 9.14 on MT-Bench, with 41% less training instability. MeRLa retains its benefits when combined with process-based and rubric-based enhanced rewards.
中文摘要 从人类反馈中强化学习（RLHF）是将大型语言模型与人类偏好对齐的标准方法，但其质量受限于静态、任务无关的奖励模型。这种不匹配导致学习信号稀疏和对齐次优。我们介绍了MeRLa（元学习奖励塑造），这是一个原则性框架，在RLHF训练前通过元学习任务感知型形态函数$\Phi（x，y;\phi）$。该学习型态产生复合奖励，既保持策略最优性，又提供任务特定学习信号。我们的元目标结合任务辨别、熵正则化和基于势的守恒，实现稳定收敛。我们提供策略不变性的理论保证，分析表示漂移敏感性，并形式化解决熵最大化带来的激励错位。在四个基准测试中，LLaMA-3-8B的实验显示，相较于PPO、DPO、GRPO和DAPO均有持续提升，AlpacaEval 2.0的长度控制胜率为90.8%，MT-Bench得分为9.14，训练不稳定性降低41%。MeRLa在与基于流程和评分标准的增强奖励结合时依然具有优势。

Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models

探究推理表现的起源：强化学习数学问题解决与SFT精细调优模型中的表征质量

Authors: Antyabha Rahman, Akshaj Gurugubelli, Omar Ankit, Kevin Zhu, Aishwarya Balwani
Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.26119
Pdf link: https://arxiv.org/pdf/2607.26119
Abstract Large reasoning models trained via reinforcement learning (RL) have been increasingly shown to outperform their supervised fine-tuned (SFT) counterparts on mathematical reasoning tasks; Yet the mechanistic basis for this advantage remains unclear. We therefore ask, what internal representational differences enable RL models' superior performance? Our work presents two converging lines of evidence: First, linear probes trained on layer-wise hidden states reveal that RL models tend to achieve higher accuracy in predicting answer correctness compared to SFT models, indicating more linearly separable and structured representations. Second, mean ablation studies show that RL models develop a hierarchical architecture where deeper layers become progressively more critical, whereas SFT models distribute importance uniformly across layers. Together, these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems. Finally, we analyze token-count variability under repeated sampling across problems to assess adaptive compute allocation. While we observe higher variability in some RL-tuned models than in their SFT counterparts, we see strong consistency in others, suggesting that token allocation may depend more on the overall training pipeline than on RL versus SFT alone. We believe this token-allocation variability reveals the spread of plausible on-policy reasoning, highlighting which models exhibit stable policies versus those that are under-determined, potentially non-identifiable solution behaviour.
中文摘要 通过强化学习（RL）训练的大型推理模型，在数学推理任务中越来越多地被证明优于其监督精细调优（SFT）模型;然而，这一优势的机制基础仍不明确。因此，我们思考，是什么内部表征差异使强化学习模型能够获得更优异的性能？我们的研究呈现了两条趋同的证据线：首先，基于层级隐形状态训练的线性探针显示，强化学习模型在预测答案正确性方面往往比SFT模型更准确，表明其表现更具线性可分离性和结构化。其次，平均消融研究表明，强化学习模型发展出一种层级结构，其中更深层变得越来越关键，而SFT模型则在不同层级间均匀分配重要性。这些发现共同表明，强化学习从根本上重构了模型的表征和推理问题处理方式。最后，我们分析了跨问题重复抽样下的令牌计数变异性，以评估自适应计算分配。虽然我们观察到某些强化学习调优模型的变异性高于其SFT对应模型，但在其他模型中则表现出强烈的一致性，表明令牌分配可能更多依赖于整体训练流水线，而非仅依赖于强化学习与单纯SFT的差异。我们认为，这种令牌分配的变异揭示了合理的策略推理的扩散，突出哪些模型表现出稳定策略，哪些模型则存在未确定性、可能无法识别的解行为。

ProFlow: RL-Driven and Performance-Aware Proactive Flow Placement in Datacenter Networks

ProFlow：基于强化学习（RL）驱动且性能感知的主动流量部署，应用于数据中心网络

Authors: Sourya Saha, Md Nurul Absur, Saptarshi Debroy
Subjects: Subjects: Networking and Internet Architecture (cs.NI); Distributed, Parallel, and Cluster Computing (cs.DC)
Arxiv link: https://arxiv.org/abs/2607.26231
Pdf link: https://arxiv.org/pdf/2607.26231
Abstract In datacenter fabrics composed of leaf and aggregation switches, competing flows may become co-located on shared aggregation switches, creating congestion that can significantly degrade protected flows. However, before throughput degradation becomes observable, the network often exhibits early signs characterized by rising flow activity and queue overflow signals. Existing congestion-management approaches primarily react only after congestion becomes visible, leaving these early signs largely unexploited. In this paper, we propose ProFlow, a proactive flow-placement framework for protecting performance-sensitive traffic in multi-tenant datacenter networks, thereby utilizing the early signs of potential throughput degradations. ProFlow leverages distributed telemetry signals and offline-trained reinforcement learning (RL) to identify precursor congestion conditions and proactively reroute protected flows before throughput degradation occurs. Evaluation results using FABRIC testbed show that ProFlow achieves approximately 40% higher mean throughput than a reactive rerouting baseline while initiating rerouting decisions around 34 seconds earlier on average, demonstrating the effectiveness of anticipatory congestion management.
中文摘要 在由叶片交换机和聚合交换机组成的数据中心结构中，竞争的流量可能共址于共享聚合交换机上，造成拥塞，显著降低受保护流量。然而，在吞吐量降级可观察到之前，网络通常会表现出流量活动上升和队列溢出信号的早期迹象。现有拥塞管理方法主要在拥堵显现后才反应，导致这些早期迹象大多未被利用。本文提出ProFlow，一种主动流量配置框架，用于保护多租户数据中心网络中性能敏感流量，利用潜在吞吐量下降的早期迹象。ProFlow利用分布式遥测信号和离线训练强化学习（RL）识别前期拥塞状况，并在吞吐量降级发生前主动重路由受保护流量。使用FABRIC测试平台的评估结果显示，ProFlow平均比反应式重路由基线高出约40%的平均吞吐量，同时平均提前约34秒启动重路由决策，展示了预期拥塞管理的有效性。

Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR

早期结论，更优预算：序列自适应部署分配以实现高效计算的RLVR

Authors: Pixel Nomand, Elena Voss, Marcus Hale, Sofia Reyes
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.26253
Pdf link: https://arxiv.org/pdf/2607.26253
Abstract Reinforcement learning with verifiable rewards (RLVR) is bottlenecked by rollout generation, yet many sampled prompts produce saturated groups (all responses correct or all incorrect) whose zero reward variance yields no policy-gradient signal. Existing remedies either oversample a larger candidate pool and discard saturated prompts (dynamic sampling), paying heavy extra rollouts, or predict prompt difficulty before sampling, which is fragile under a shifting policy. We observe that a group's effectiveness is usually decided early, within the first few of its rollouts, so spending a full group on an already-decided prompt is wasteful. We cast per-step rollout collection as a budget-constrained sequential allocation (optimal stopping) problem and introduce SARA (Sequential Adaptive Rollout Allocation). SARA maintains a Beta posterior over each prompt's success rate, evaluates a closed-form predictor of group effectiveness, and applies a two-threshold, SPRT-style rule that commits effective groups, abandons saturated ones after a short probe, and reallocates the freed budget to fresh prompts, without any extra prediction rollouts. We prove abandonment reliability, expected rollout savings, fixed-budget yield dominance, and a link between effective-group yield and the GRPO gradient norm. On mathematical reasoning and planning with 1.5B/3B models on a single GPU, SARA matches DPS (both below the DS oracle) while using 22% fewer rollouts than DS; composing SARA with DPS yields the best accuracy, slightly above DS, at 67% fewer rollouts (near-uniform cost).
中文摘要 带可验证奖励的强化学习（RLVR）被推出生成瓶颈，但许多抽样提示产生饱和组（所有响应正确或全部错误），其零奖励方差不产生策略梯度信号。现有方案要么过度采样候选池，丢弃饱和提示（动态抽样），支付大量额外覆盖费用，要么在采样前预测提示难度，而在策略变化下难度较为脆弱。我们观察到，一个小组的有效性通常在最初几次推广中就已决定，因此花费整个小组在已确定提示上是浪费。我们将每步推广收集归结为预算约束的顺序分配（最优停止）问题，并引入了SARA（顺序自适应推广分配）。SARA对每个提示的成功率保持Beta后验，评估一个闭式组效能预测器，并应用一个两阈值的SPRT式规则，该规则提交有效组，短时间探测后放弃饱和组，并将释放的预算重新分配到新提示，且无需额外预测部署。我们证明了放弃可靠性、预期部署节省、固定预算产出优势，以及有效组产出与GRPO梯度范数之间的联系。在单GPU上1.5B/3B模型的数学推理和规划中，SARA与DPS匹配（均低于DS预言机），同时使用比DS少22%的部署次数;用DPS组合SARA的准确度最佳，略高于DS，部署次数减少67%（成本近乎均匀）。

Learning Implicit Causal World Models from Multi-Agent Demonstrations

从多智能体演示中学习隐性因果世界模型

Authors: Jasorsi Ghosh
Subjects: Subjects: Machine Learning (cs.LG); Multiagent Systems (cs.MA); Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.26336
Pdf link: https://arxiv.org/pdf/2607.26336
Abstract In model-based reinforcement learning, world models exist as internal simulators, but their training often conflates statistical correlations with causal mechanisms. This problem is exacerbated in multi-agent systems where physical transitions are intertwined with strategic agent intents, causing world models to fail under distribution shift. We introduce Implicit Causal World Models to recover environmental dynamics from offline demonstrations without requiring pre-defined causal graphs. By incorporating policy variance, we render world models discoverable via the sequential backdoor condition. Evaluations across coordination tasks (Two-Door, Navigation, and Giveway) demonstrate that these models provide interpretable causal representations under both full and partial observability, with model accuracy scaling directly with interventional strength.
中文摘要 在基于模型的强化学习中，世界模型作为内部模拟器存在，但其训练常常将统计相关性与因果机制混为一谈。在多智能体系统中，物理转变与战略智能体意图交织，导致世界模型在分布转移下失效。我们引入隐式因果世界模型，从离线演示中恢复环境动态，无需预设因果图。通过纳入策略方差，我们使世界模型可通过顺序后门条件被发现。跨协调任务（双门、导航和让步）的评估表明，这些模型在全可观和部分可观测性下均提供可解释的因果表征，模型准确性直接随干预强度提升。

Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning

检测边缘的后训练：一种博弈论方法进行微调

Authors: Keegan Harris, Brian W. Lee, Ian Waudby-Smith, Philip Amortila, Nika Haghtalab, Michael I. Jordan
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT)
Arxiv link: https://arxiv.org/abs/2607.26358
Pdf link: https://arxiv.org/pdf/2607.26358
Abstract Reinforcement learning (RL) fine-tuning is widely used in language model training to improve model performance on a target task while limiting drift from a reference policy. A standard way to balance this trade-off is via a KL-regularized RL objective, although this formulation does not by itself provide a principled way to set the regularization coefficient. In practice, the coefficient is typically chosen heuristically or via hyperparameter search, which can lead to unnecessary overhead in training cost or undesirable reward-retention trade-offs. We instead propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation. Specifically, we study a sequential game in which an agent chooses a policy to maximize cumulative reward while a monitor observes policy outputs over time and tests for deviations from the reference policy. Although not originating from the same perspective, we show that the resulting equilibrium policy can nonetheless be expressed as the solution to a KL-regularized RL problem for an optimal regularization parameter that can be viewed as maximizing reward per unit of statistical distinguishability. Drawing on classical results from concave-convex fractional programming, we provide a principled method for learning this equilibrium coefficient via reduction to the KL-regularized RL objective, thus allowing for flexible integration into standard fine-tuning pipelines. In experiments with Qwen3-8B and Llama-3.2-1B, we demonstrate that our methods result in competitive reward-retention trade-offs in a continual learning setting, and illustrate how our framework may be used to audit API providers serving open-source models.
中文摘要 强化学习（RL）微调广泛应用于语言模型训练中，以提升目标任务中的模型性能，同时限制参考策略的漂移。一种标准的平衡方法是通过KL正则化的RL目标，尽管该表述本身并不能提供有原则的正则化系数设定方法。实际上，系数通常通过启发式选择或超参数搜索，这可能导致不必要的训练成本开销或不良的奖励-保留权衡。我们提出了一个博弈论框架，为这种权衡赋予明确的统计解释。具体来说，我们研究一种顺序博弈，其中代理选择策略以最大化累计奖励，而监控者观察策略输出随时间变化并检验偏离参考策略。虽然并非同一视角，但我们证明所得均衡策略仍可表达为对KL正则化RL问题的解，该问题可视为最大化每单位统计可区分性奖励的最优正则化参数。借鉴凹凸分数规划的经典结果，我们提供了一种原则性方法，通过归约至KL正则化RL目标来学习该均衡系数，从而允许灵活整合进标准微调流水线。在Qwen3-8B和Llama-3.2-1B的实验中，我们展示了在持续学习环境中导致奖励与保留的竞争性权衡，并展示了我们的框架如何用于审计服务开源模型的API提供者。

CaM-Wolf: Causal-Aware Multimodal Agents for Social Deduction Games

CaM-Wolf：社会演绎游戏中的因果感知多模态代理

Authors: Zheng Zhang, Nanjie Yao, Jiarui He, Deheng Ye, Peilin Zhao, Hao Wang
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.26393
Pdf link: https://arxiv.org/pdf/2607.26393
Abstract Social deduction games (SDGs) such as Werewolf have become challenging testbeds for AI agents. These games require complex social skills such as reasoning, deception, and collaboration. While recent advances in large language models (LLMs) have driven significant progress in SDG agents, current approaches are predominantly text-based, overlooking the multimodal nature that is fundamental to human social interaction. To bridge this gap, we introduce CaM-Wolf, the first SDG agent that integrates multimodal perception and generation. CaM-Wolf processes video inputs from other players, employs a causal-aware Reasoner trained via reinforcement learning to establish logical chains between observable behaviors and hidden roles, and presents itself through an animated avatar. Our experiments and user study show that CaM-Wolf achieves superior agent gameplay performance and enhances the quality of human-AI interaction. This work represents a significant advancement towards creating more human-like AI agents capable of participating in nuanced social dynamics. Our code is available at this https URL.
中文摘要 像狼人这样的社交演绎游戏（SDGs）已成为AI代理的挑战性测试平台。这些游戏需要复杂的社交技能，如推理、欺骗和协作。尽管大型语言模型（LLMs）的最新进展推动了SDG代理的显著进展，但当前的方法主要基于文本，忽视了人类社会互动中根本的多模态特性。为弥合这一差距，我们引入了CaM-Wolf，这是首个整合多模态感知与生成的SDG代理。CaM-Wolf处理来自其他玩家的视频输入，使用通过强化学习训练的因果意识推理器，建立可观察行为与隐藏角色之间的逻辑链，并通过动画化头像呈现自己。我们的实验和用户研究表明，CaM-Wolf实现了更优越的代理游戏性能，并提升了人机交互的质量。这项工作代表了在创造更具人类特征的人工智能代理方面取得的重要进展，能够参与细致复杂的社会动态。我们的代码可在此 https URL 获取。

SCOUT: Per-Context Reset Curricula for Sparse-Reward Reinforcement Learning

SCOUT：稀疏奖励强化学习的按情境重置课程

Authors: Siddharth Aphale, Ayushman Singh
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.26417
Pdf link: https://arxiv.org/pdf/2607.26417
Abstract Sparse-reward reinforcement learning often fails because rollouts from the unassisted evaluation start rarely reach later task stages. Reset curricula address this by starting some training rollouts from easier intermediate states, called scaffolds. Such a curriculum faces two decisions: scaffold access, obtaining informative starts, and scaffold allocation, deciding how quickly that assistance is removed. Most prior curricula pace removal on one shared schedule, which can fail when task instances, or contexts, learn at different rates. We introduce SCOUT, an online, learner-agnostic reset controller that gives every context its own curriculum. Using only binary rollout success, SCOUT removes assistance after sustained success, restores it after failure, and cautiously tests a harder start when progress stalls, without changing the reward, optimizer, or learner. A counting construction shows that synchronized global pacing can be insufficient when contexts need conflicting amounts of assisted practice. Across six navigation and manipulation settings, scaffold access improves learning and enables success in three where unassisted training fails within the reported budget. In a constructed pacing conflict, each tested global schedule leaves one group unsolved, while SCOUT solves both. Average success can conceal this failure, so we also report the least successful group. Group-level pacing works when learning differences follow known groups but can fail when they occur within one group. SCOUT needs no group labels and remains consistently strong in both cases. A reset curriculum should remove assistance at the scale where learning progress differs.
中文摘要 稀疏奖励强化学习常常失败，因为无辅助评估开始的推广很少能进入后续任务阶段。重置课程通过从较易的中间状态（称为支架）开始部分培训推广来解决这个问题。此类课程面临两个抉择：支架访问，即获取信息性起点，以及支架分配，决定该辅助被移除的速度。大多数之前的课程进度移除都在同一时间表上，当任务实例或上下文以不同速度学习时，可能会失败。我们引入了SCOUT，一个在线、学习者无关的重置控制器，赋予每个上下文自己的课程。仅以二元推广成功为基础，SCOUT在持续成功后移除辅助，失败后恢复，并在进展停滞时谨慎测试更难的起点，但不改变奖励、优化器或学习者。计数构建表明，当需要冲突数量的辅助练习时，同步的全局节奏可能不够。在六种导航和操作环境中，支架访问提升了学习效果，并使在无辅助训练未能达到报告预算的三个环境中取得成功。在构建的节奏冲突中，每个测试的全局计划都会留下一个小组未解，而SCOUT则解决两个小组。平均成功率可以掩盖这一失败，因此我们也会报告最不成功的群体。当学习差异跟随已知小组时，小组级节奏有效，但当差异发生在同一群体内时可能失败。SCOUT无需群体标签，且在两种情况下都能保持一致。重置课程应在学习进展差异的尺度上取消辅助。

Reinforcement Learning on Cost-Constrained Quadrupedal Hardware

成本受限的四足硬件上的强化学习

Authors: Javier C. Weddington, Bence P. Ölveczky, Stephen A. Baccus
Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.26434
Pdf link: https://arxiv.org/pdf/2607.26434
Abstract Deploying learned control policies on low-cost robotic platforms introduces transport latencies and noisy motor feedback that systematically widens the sim-to-real gap. The chasm of simulation to deployment in hardware lies in the delay of the actuator reaching the commanded position. On platforms such as the Mini Pupper 2, a measured > $50 ms transport delay transforms the locomotion task from a standard Markov decision process into a partially observable one. In this paper, we take a biologically inspired approach of handling noisy and delayed feedback to close the sim-to-real gap, thereby expanding the capability of reinforcement learning on cost-constrained hardware. Using a low-cost quadrupedal hardware platform, we find that using a forward model of the average actuator delay, paired with a time-aware neural network results in robust locomotion. Additionally, our time-aware neural network learned a central pattern generator (CPG): a self-sustaining rhythmic gait that is robust to +320 ms latency perturbations, mirroring the CPGs found in the spinal cords of vertebrates. We posit that temporal self-organization may be a general strategy for cost-constrained locomotion.
中文摘要 在低成本机器人平台上部署学习控制策略会引入传输延迟和噪声马达反馈，系统性地扩大模拟与现实之间的差距。仿真与硬件部署之间的鸿沟在于执行器到达指令位置的延迟。在Mini Pupper 2等平台上，测量到>的50毫秒传输延迟可将运动任务从标准的马尔可夫决策过程转变为部分可观测过程。本文采用生物学启发的方法，处理噪声和延迟反馈以缩小模拟到真实的差距，从而扩展了在成本受限硬件上的强化学习能力。利用低成本的四足硬件平台，我们发现使用平均执行器延迟的前向模型，配合时间感知神经网络，能实现稳健的移动。此外，我们的时间感知神经网络学习了中央模式生成器（CPG）：一种自我维持的节律步态，对+320毫秒的延迟扰动具有鲁棒性，类似于脊椎动物脊髓中的CPG。我们假设时间自组织可能是成本受限运动的通用策略。

DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learning

DHRCL：训练代码LLMs，具有密集的层级奖励和课程学习

Authors: Shuhang Wang, Ziming Li, Hui Cheng
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.26457
Pdf link: https://arxiv.org/pdf/2607.26457
Abstract Reinforcement learning is a natural post-training paradigm for code-oriented large language models because generated programs can be evaluated through parsing, execution, unit tests, and structural this http URL, existing methods often rely on sparse outcome rewards or statically combine heterogeneous dense signals, even though syntax validity, executability, functional correctness, and structural organization describe different and progressively dependent programming capabilities. We propose DHRCL, a reinforcement learning framework with Dense Hierarchical Rewards and Curriculum Learning. DHRCL decomposes feedback into syntax validation, execution success, unit-test pass rate, and AST-based structural similarity, and organizes these signals through a three-stage Syntax, Execution, Pass & Structural curriculum. Stage duration is determined automatically from recent validation trends rather than manually specified capability thresholds. We further introduce stage-aware probability-based token credit redistribution. The mechanism follows a consolidation-to-refinement principle: it emphasizes established token patterns during syntax-oriented optimization, applies uniform propagation for non-local execution feedback, and allocates more credit or blame to less-established token decisions during final functional optimization. Under a unified Qwen3-8B and KodCode protocol, the experiments compare DHRCL with binary, pass-rate, reward-model-based, and verifiable dense-reward baselines. We further evaluate DHRCL across Qwen3-4B, Qwen3-8B, and Qwen3-14B backbones, showing that its advantage remains consistent as model capacity increases.
中文摘要 强化学习是面向代码的大型语言模型的自然训练后范式，因为生成的程序可以通过解析、执行、单元测试和结构化 http URL 进行评估，现有方法通常依赖稀疏的结果奖励或静态组合异构密集信号，尽管语法有效性、可执行性、功能正确性和结构组织描述了不同且逐渐依赖的编程能力。我们提出了 DHRCL，一个具有密集层级奖励和课程学习的强化学习框架。DHRCL 将反馈分解为语法验证、执行成功率、单元测试通过率和基于 AST 的结构相似性，并通过三阶段语法、执行、通过和结构化课程组织这些信号。阶段时长根据近期验证趋势自动确定，而非手动指定的能力阈值。我们还进一步引入了基于阶段感知概率的代币信用重分配。该机制遵循整合到精细化原则：在语法导向优化中强调既有的代币模式，对非本地执行反馈采用统一传播，在最终功能优化中对尚未建立的代币决策给予更多功劳或责任。在统一的Qwen3-8B和KodCode协议下，实验将DHRCL与二进制、通过率、基于奖励模型和可验证的密集奖励基线进行比较。我们进一步评估DHRCL在Qwen3-4B、Qwen3-8B和Qwen3-14B骨干上的应用，显示其优势在模型容量增加时依然保持一致。

RLMM-Flow: A Flow-based Mobile Manipulation Framework with Latent-Space Reinforcement Learning

RLMM-Flow：基于流动的移动操作框架，具备潜在空间强化学习

Authors: Shuhang Wang, Ziming Li, Hui Cheng
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.26460
Pdf link: https://arxiv.org/pdf/2607.26460
Abstract Mobile manipulation requires generating whole-body action chunks that jointly satisfy goal reaching, collision avoidance, base kinematic constraints, manipulator joint limits, and trajectory smoothness. Flow-based generative policies provide an efficient paradigm for learning multimodal and temporally consistent motion priors from expert demonstrations, but imitation-only training cannot improve policy quality beyond the demonstration distribution. We propose RLMM-Flow, a flow-based mobile manipulation framework that combines expert flow-policy pretraining with latent-space reinforcement learning post-training. The framework first learns a flow policy that captures a multimodal whole-body motion prior from expert demonstrations. The pretrained flow policy is then frozen, while a latent steering network steers its initial noise toward higher-value action chunks. To stabilize high-dimensional latent optimization, we warm up an action-space critic before jointly training the latent critic and latent actor, and introduce coarse-to-fine latent steering that progressively expands control from a horizon-shared latent representation to a full-dimensional residual representation. Experiments on mobile manipulation motion-planning benchmarks show that RLMM-Flow substantially improves task success, collision avoidance, and trajectory quality over imitation-only flow policies and existing reinforcement learning post-training baselines, while preserving fast flow-based inference.
中文摘要 移动操作需要生成整体动作块，这些块共同满足目标达成、碰撞避免、基础运动学约束、机械臂关节极限和轨迹平滑性。基于流的生成策略为从专家演示中学习多模态和时间一致的运动先验提供了高效范式，但仅模仿训练无法提升策略质量，超出示范分布。我们提出了RLMM-Flow，一种基于流的移动操作框架，结合了专家流策略预训练与训练后潜在空间强化学习。该框架首先学习一个流策略，捕捉专家演示前的多模态全体运动。预训练流策略随后被冻结，潜引导网络引导其初始噪声指向高价值动作块。为稳定高维潜在优化，我们在联合训练潜在批评者和潜在行为者之前，先预热一个动作空间批评者，并引入粗到细的潜在引导，逐步将控制从视界共享的潜在表征扩展为全维残差表示。移动操作运动规划基准测试的实验表明，RLMM-Flow相比仅模仿的流程策略和现有强化训练后基线，显著提升了任务成功率、碰撞避免率和轨迹质量，同时保持了基于流程的快速推断。

Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

与悲观批评者合作加权，以缓解非策略强化学习中的高估

Authors: Gong Gao, Xiao Lai, Ziqi Xie, Guojie Chen, Xianhui Liu, Weidong Zhao
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.26509
Pdf link: https://arxiv.org/pdf/2607.26509
Abstract Deep off-policy reinforcement learning algorithms for continuous control typically rely on neural value function approximation to guide policy improvement. However, temporal-difference (TD) learning introduces noisy targets, resulting in non-stationary optimization, while greedy policy updates amplify early-stage estimation errors. The recursive propagation of such errors leads to persistent overestimation bias and degraded training stability in actor-critic methods. Existing approaches attempt to alleviate this issue via prioritized sampling or modified value learning objectives, but often overemphasize high-uncertainty transitions caused by limited data coverage or bootstrapping errors, thereby further amplifying this http URL this paper, we propose Collaborative Weighting Actor-Critic (CWAC), a unified framework that explicitly accounts for predictive uncertainty in value estimation. CWAC employs distributional critic to model return uncertainty and introduces a collaborative weighting mechanism that jointly reweights TD-errors and uncertainty, enabling robust learning from reliable samples while suppressing noisy updates. In addition, we incorporate a stochastic pessimistic value estimation scheme via sampling from the return distribution, which effectively mitigates error propagation during policy improvement. CWAC can be seamlessly integrated into existing off-policy algorithm frameworks such as SAC, TD3, and DDPG with minimal overhead. Empirical results demonstrate that our proposed method significantly enhances performance across a diverse range of simulated tasks. Our code is publicly available at this https URL.
中文摘要 深度非策略强化学习算法通常依赖神经价值函数近似来指导策略改进。然而，时间差分（TD）学习引入噪声目标，导致非平稳优化，而贪婪的策略更新则放大早期估计误差。此类错误的递归传播导致持续的高估偏差和actor-critic方法中的训练稳定性下降。现有方法试图通过优先抽样或修改值学习目标来缓解这一问题，但往往过分强调由有限数据覆盖或自助错误引起的高不确定性转变，进一步放大了该http URL 本文本文，我们提出了协同加权演员-批判者（CWAC）——一个明确考虑价值估计预测不确定性的统一框架。CWAC采用分布批判法建模返回不确定性，并引入协作加权机制，联合重加权TD错误和不确定性，实现从可靠样本中稳健学习，同时抑制噪声更新。此外，我们采用了通过返回分布抽样的随机悲观值估计方案，有效减少策略改进过程中的误差传播。CWAC可无缝集成到现有非策略算法框架，如SAC、TD3和DDPG，开销极低。实证结果表明，我们提出的方法显著提升了多种模拟任务的性能。我们的代码公开于此 https URL。

Explicit Kinematic Guidance from Analytic Concepts for Vision-Language-Action Models

视觉-语言-行动模型的显式运动指导分析概念

Authors: Mingyang Sun, Jiude Wei, Xiujian Liang, Qichen He, Donglin Wang, Cewu Lu, Jianhua Sun
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.26513
Pdf link: https://arxiv.org/pdf/2607.26513
Abstract Current Vision-Language-Action (VLA) models rely mainly on 2D inputs, neglecting the rich object structural information and commonsense knowledge inherent in the 3D physical world. This deficiency restricts their spatial awareness and adaptability for complex, high-precision manipulation. To bridge this crucial gap, we construct a Concept Expert module for VLA to build executable Analytic Concepts that represent objects as explicit, programmatic blueprints. Our mechanism operates in two synergistic phases: First, prior to VLA inference, the Concept Expert leverages 3D information from Vision Foundation Models (VFMs) to estimate the initial kinematic and structural parameters. Second, throughout the manipulation process, the VLA model utilizes its inherent capability to dynamically track the dynamic concept parameters, continuously aligning them with observational changes to ensure persistent accuracy. Once established, the Analytic Concepts provide explicit, high-quality guidance for VLA fine-tuning through (1) dense, programmatic manipulation rewards and (2) precise spatial guidance. This formulation allows VLA models to learn physically grounded interaction behaviors while maintaining end-to-end learning flexibility. Our experimental results show consistent improvements in success rate and learning efficiency across supervised and reinforcement learning settings, demonstrating the effectiveness of structured, concept-based guidance for VLA post-training.
中文摘要 当前的视觉-语言-行动（VLA）模型主要依赖二维输入，忽视了三维物理世界中丰富的物体结构信息和常识知识。这种缺陷限制了它们在复杂高精度操作中的空间感知和适应性。为弥合这一关键差距，我们为VLA构建了一个概念专家模块，构建可执行的分析概念，将对象作为显式的程序蓝图表示。我们的机制分为两个协同阶段：首先，在VLA推断之前，概念专家利用视觉基础模型（VFM）中的三维信息估算初始运动学和结构参数。其次，在整个操作过程中，VLA模型利用其固有能力动态跟踪动态概念参数，持续与观测变化对齐，确保持续的准确性。一旦建立，分析概念通过（1）密集的程序化操作奖励和（2）精确的空间指导，为VLA微调提供明确且高质量的指导。这种表述使VLA模型能够学习基于物理的交互行为，同时保持端到端的学习灵活性。我们的实验结果显示，在监督学习和强化学习环境中，成功率和学习效率均有持续提升，证明了结构化、基于概念的指导对VLA训练后阶段的有效性。

Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detection

图是验证者：过程间漏洞检测中的代理强化学习

Authors: Yikun Li, Ting Zhang, Jiakun Liu, Jinfeng Jiang, Yuheng Yieh, Yixin Yang, Wen Bin Leow, Yide Yin, Yintong Huo, Eng Lieh Ouh, Lwin Khin Shar, David Lo
Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
Arxiv link: https://arxiv.org/abs/2607.26656
Pdf link: https://arxiv.org/pdf/2607.26656
Abstract Real-world vulnerabilities often span multiple functions, yet most learning-based detectors classify each function in isolation: on a sample of real CVEs, we find that 71.7% of vulnerable functions require evidence from outside the function to be classified correctly. Agentic reinforcement learning (RL) could close this gap by enabling a model to gather that evidence itself, but it lacks a reliable reward, since a reward defined on the final verdict alone can be obtained without performing any investigation. We propose VulAgentRL, an agentic RL framework for interprocedural vulnerability detection built on a Code Property Graph (CPG). The CPG serves two roles: at inference time the policy queries it for callers, callees, dataflow, and other queries, and at training time the same graph verifies the evidence the policy cites. Because every CPG node carries a persistent integer identifier, this verification is an exact comparison rather than a textual match, so the reward credits verdicts that are supported by evidence. We further initialize the policy by distilling teacher investigations, and show that this warm start is necessary, since RL cannot acquire tool-use behavior it never samples. Under a repository-level split that prevents leakage, VulAgentRL outperforms state-of-the-art baselines, including frontier models, on the strict pair-wise-correct metric while issuing fewer tool calls, and its advantage persists on an out-of-distribution corpus and under class imbalance.
中文摘要 现实世界中的漏洞通常跨越多个功能，但大多数基于学习的检测器会对每个功能进行隔离分类：在真实的CVE样本中，我们发现71.7%的易受攻击函数需要来自该函数外部的证据才能正确分类。代理强化学习（RL）可以通过让模型自行收集这些证据来弥补这一差距，但由于仅凭最终判决定义的奖励即可在不进行任何调查的情况下获得，因此缺乏可靠的奖励。我们提出了VulAgentRL，这是一个基于代码属性图（CPG）的代理式强化学习（ERL）框架，用于程序间漏洞检测。CPG承担两个角色：在推理时，策略查询调用者、被调用者、数据流及其他查询;在训练时，同一图验证策略引用的证据。由于每个消费品节点都携带持久整数标识符，这种验证是精确比较而非文本匹配，因此奖励会给出有证据支持的结论。我们进一步通过提炼教师调查初始化策略，并证明这种温暖启动是必要的，因为强化学习无法获得它从未采样的工具使用行为。在防止泄漏的仓库层级拆分下，VulAgentRL在严格的两对正确度量上表现优于包括前沿模型在内的最先进基线，同时发出的工具调用更少，其优势在非分布语料库和下类不平衡中依然存在。

Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRL

风险感知型自动实时学习的高效异方差贝叶斯优化

Authors: Mingxuan Che, Tsung-Yuan Tseng, Theresa Eimer, Marius Lindauer, Alexander von Rohr
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.26680
Pdf link: https://arxiv.org/pdf/2607.26680
Abstract Reinforcement learning (RL) has shown remarkable success across a wide range of complex tasks. However, RL outcomes can be highly stochastic, and both expected performance and variability often depend on hyperparameter (HP) configurations. We propose efficient and risk-averse heteroscedastic Bayesian Optimization (ERAHBO), a Bayesian optimization method that models both the mean and variance of learning outcomes as functions of the HP configurations. ERAHBO aims to identify HP configurations that achieve high average return while reducing variability across training runs, and it improves the sample efficiency of the HP optimization via adaptive re-sampling rather than a fixed budget per HP. Empirical evaluations across diverse RL algorithms and environments demonstrate that ERAHBO generally outperforms both risk-neutral and risk-averse baselines, delivering improved sample efficiency for risk-averse returns.
中文摘要 强化学习（RL）在多种复杂任务中表现出显著成功。然而，强化学习结果高度随机，预期性能和变异性常依赖于超参数（HP）配置。我们提出了高效且风险规避的异方差贝叶斯优化（ERAHBO），这是一种贝叶斯优化方法，将学习结果的平均值和方差建模为HP配置的函数。ERAHBO旨在识别能够实现高平均回报且减少训练运行间变异性的HP配置，并通过自适应重抽样而非每HP固定预算，提升HP优化的样本效率。跨多种强化学习算法和环境的实证评估表明，ERAHBO通常优于风险中性和风险厌恶基线，提升了风险厌恶回报的样本效率。

SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution

SkillRise：跨任务技能演进的能动强化学习

Authors: Zhiyuan Yao, Yuxin Chen, Zhengxi Lu, Zishan Xu, Yueqing Sun, Yifu Guo, Yuquan Lu, Zhengzhou Cai, Kangning Zhang, Zhuowen Han, Zi-Han Wang, Ziang Ye, Qi Gu, Xunliang Cai, Weiwen Liu, Yongliang Shen
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.26784
Pdf link: https://arxiv.org/pdf/2607.26784
Abstract Large language model agents often encounter related yet distinct tasks that share reusable solution patterns. Yet standard agentic reinforcement learning treats tasks as independent episodes, while existing approaches to skill learning either focus on repeated attempts of one task or use pipelines with multiple stages that entangle extraction, retrieval, and execution. We introduce SkillRise, a unified reinforcement learning framework for learning skills across tasks. SkillRise organizes related instances into progressively challenging sequences and uses a single policy to alternate between task solving and curating an evolving skill document passed directly to the next task. Decoupled credit assignment across tasks supervises solving with the current task outcome and curation with discounted downstream outcomes. Experiments on ALFWorld, WebShop, and ScienceWorld show that SkillRise achieves the strongest Pass@1 performance among the compared methods, with gains over the strongest baseline ranging from 2.3 to 8.5 percentage points. Although trained across distinct tasks, its learned curation policy remains effective for repeated attempts on the same task. Further analysis reveals scaling at test time across tasks: performance improves with longer sequences of related tasks even when each task is attempted only once. This trend suggests that SkillRise reuses transferable skills across tasks rather than benefiting from repeated sampling of the same task. SkillRise further retains strong performance while substantially reducing the runtime overhead of skill learning pipelines with multiple stages. Together, these results provide a simple and efficient training paradigm for LLM agents to extract, refine, and reuse transferable skills across tasks.
中文摘要 大型语言模型代理经常遇到相关但不同的任务，这些任务共享可重用的解决方案模式。然而，标准的代理强化学习将任务视为独立的片段，而现有的技能学习方法要么专注于重复尝试同一任务，要么使用多阶段的管道，纠缠提取、检索和执行。我们介绍了SkillRise，一个跨任务学习技能的统一强化学习框架。SkillRise将相关实例组织成逐步具有挑战性的序列，并使用单一策略在任务解决和直接传递给下一个任务的技能文档管理之间交替进行。跨任务的解耦学分分配监督以当前任务结果为基础的求解和策划，后者结果会被折现。在ALFWorld、WebShop和ScienceWorld上的实验显示，SkillRise在比较方法中实现了最强的Pass@1性能，较最强基线提升2.3%至8.5个百分点。尽管训练跨不同任务，其学习的策划策略对同一任务的重复尝试仍然有效。进一步分析显示，测试时任务间存在扩展性：即使每个任务仅尝试一次，相关任务序列越长，性能也会提升。这一趋势表明SkillRise在任务间重复使用可转移技能，而非通过重复抽样同一任务获益。SkillRise进一步保持了强效性能，同时大幅降低了多阶段技能学习流程的运行开销。这些结果共同为LLM代理提供了一个简单高效的训练范式，帮助提取、优化和重用跨任务的可转移技能。

ReCo: Reweighting GRPO Against Distributional Concentration

ReCo：根据分布集中度重新加权GRPO

Authors: Junoh Park, Junseo Hwang, Wonguk Cho, Taesup Kim
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2607.26862
Pdf link: https://arxiv.org/pdf/2607.26862
Abstract Group Relative Policy Optimization (GRPO) has become a standard reinforcement learning method for post-training language models. Recent work shows that GRPO can reduce the base model's reasoning capacity and underperform it in Pass@k when k is large, indicating reduced coverage of reasoning paths. We find that this reduction is associated with GRPO concentrating on responses that the base model already generates with high probability. We trace this concentration to two mechanisms in the GRPO update. At the response level, high-probability responses dominate the group gradient through repeated occurrence. At the token level, GRPO's importance ratio scales gradients, further reinforcing tokens that become more likely under the current policy. We propose ReCo, a reweighting method that addresses both effects. Response contributions are normalized by their expected occurrence within the rollout group, and the token-level importance ratio is replaced with a variance-based ratio that gives larger update scale to non-saturated decision points where alternative token choices remain plausible. Across Qwen2.5-Math-1.5B/7B and Llama-3.1-8B-Instruct on five mathematical reasoning benchmarks, ReCo improves Pass@k for large values of k and is comparable to GRPO for small values of k.
中文摘要 群体相对策略优化（Group Relative Policy Optimization，GRPO）已成为训练后语言模型的标准强化学习方法。最新研究表明，当k较大时，GRPO会降低基础模型的推理能力，并在Pass@k中表现不足，表明推理路径覆盖率降低。我们发现这种减少与GRPO集中于基础模型已高概率生成的响应有关。我们将这种集中归因于GRPO更新中的两个机制。在反应层面，高概率反应通过反复出现主导群体梯度。在标记层面，GRPO的重要性比会放大梯度，进一步强化当前策略下更可能出现的标记。我们提出了ReCo，一种兼顾这两种效应的加权方法。响应贡献根据其在推广组内的预期出现频率进行归一化，代币级重要性比被基于方差的比率取代，使非饱和决策点在替代代币选择仍可行时提供更大更新规模。在五个数学推理基准测试中，ReCo在大k时提升Pass@k，小k时可与GRPO相当。

SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

SERPO：自我演进的评分规策略优化，用于开放式测试时间强化学习

Authors: Jianze Wang, Kunwang Zheng, Ying Liu, Yu Cao, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen
Subjects: Subjects: Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2607.26873
Pdf link: https://arxiv.org/pdf/2607.26873
Abstract Test-time reinforcement learning (TTRL) enables language models to self-evolve at inference time without labeled feedback. Existing methods rely on answer voting and therefore do not extend naturally to open-ended generation, where valid responses cannot be mapped to a shared canonical answer. Without external reward models or stronger judges, adaptation must instead construct reliable rewards from the model's own outputs. We introduce SERPO (Self-Evolving Rubric Policy Optimization), which replaces answer voting with a closed loop that co-evolves response evidence, query-specific rubrics, and policy parameters. Good-Normal-Bad (G-N-B) response evolution organizes maximally separated rollouts into ordered archives; rubric evolution retains criteria that discriminate these archives; probabilistic criterion scoring converts verdict-token likelihoods into reward signals; and policy evolution optimizes the actor with the resulting signals. New actor rollouts then refresh both the archives and rubrics, closing the three-way evolution loop. Across two model configurations, two in-domain benchmarks, and four OOD benchmarks, SERPO improves HealthBench and ResearchQA by up to 20.63 and 20.31 points over the corresponding base models, raises the six-benchmark macro-average by up to 8.06 points, and supports OOD transfer and continued cross-benchmark evolution.
中文摘要 测试时间强化学习（TTRL）使语言模型能够在推理时自我演化，无需标记反馈。现有方法依赖答案投票，因此无法自然扩展到开放式生成，即有效回答无法映射到共享的典型答案。没有外部奖励模型或更强的评判，适应必须从模型自身输出构建可靠的奖励。我们引入了SERPO（自我演化评分标准策略优化），它用一个闭环替代答案投票，共同进化响应证据、查询特定评分标准和策略参数。良好-正常-坏（G-N-B）响应演化将最大分离的展开组织为有序档案;规律演化保留区分这些档案的标准;概率标准评分将判决标记似然转换为奖励信号;策略演进则通过由此产生的信号优化演员。新的参与者推出后，更新归档和评分标准，完成三向演进循环。在两种模型配置、两个领域基准测试和四个面向对象基准测试中，SERPO使HealthBench和ResearchQA比相应基础模型提升了最多20.63和20.31个点，将六个基准的宏观平均提升了最多8.06个点，并支持面向对象转移和持续的跨基准演进。

RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

RL$^2$-VLA：视觉-语言-动作模型的自适应强化学习潜在组合引导，具备测试时间缩放

Authors: Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer, William Wei Jie Teo, Yuanliang Ju, Qiao Gu, Guillaume Sartoretti
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.26991
Pdf link: https://arxiv.org/pdf/2607.26991
Abstract Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce $RL^2$, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight, $RL^2$ activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks, $RL^2$ improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing $RL^2$ as a practical and modular steering framework for VLA deployment.
中文摘要 尽管视觉-语言-行动（VLA）模型具备令人印象深刻的视觉运动能力，但它们在具有挑战性和域外任务上的表现常常下降。近期的测试时引导和缩放方法在无需大量数据收集和再训练的情况下提升了性能，但动作样本往往仍集中在相似行为上，因此继承了相关的失败模式。此外，现有方法在每个时间步都采用相同的干预策略，无论基础策略是否可能成功。为解决这些局限性，我们引入了$RL^2$，一种自适应推理时间引导框架，利用VLA潜在的强化学习。首先，我们训练一个基于从VLA动作专家提取的表达潜在变量的轻量级离线强化学习策略，并将其流速度与冻结的VLA推理速度结合。该组合引导策略结合了大规模模仿学习的行为先验与离线强化学习在主导示范模式外诱导的动作多样性。我们还发现推理时间引导在成功和失败状态下遵循根本不同的缩放律，表明当基础VLA可能失败时，动作多样性最为有益，但在成功可能性较大时，可能会不必要地扰乱已经准确的动作。基于这一见解，$RL^2$仅在预测失败时激活组合引导。在SIMPLER和PolaRiS基准测试中，$RL^2$在域外环境中提升成功率高达+17.3%，而消融和缩放研究则展示了潜在表示和强化学习训练的重要性。最后，现实实验表明这些收益超越仿真，确立了$RL^2$作为VLA部署的实用模块化引导框架。

Minimal Markovization via Stable Quotients in Holonomy-Cover Decision Processes

全息覆盖决策过程中通过稳定商实现的最小马尔可夫化

Authors: Zuyuan Zhang, Yongshan Chen, Mahdi Imani, Tian Lan
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.27132
Pdf link: https://arxiv.org/pdf/2607.27132
Abstract An agent acting under partial observability must retain a recursively updateable statistic of history that restores the Markov property, but the smallest such statistic is generally unknown. We characterize this minimal Markov sufficient statistic for holonomy-cover decision processes, a structured POMDP class in which the visible dynamics are Markov and every realized visible transition applies a fixed permutation to a hidden mode. In particular, we construct the stable quotient, the coarsest observation-wise abstraction preserving one-step rewards and quotient successors, and prove that the pair of the current observation and stable class forms an exact finite Markov state. When the current class is correctly initialized, exact class tracking requires exactly the minimal memory symbols, in the sense that under reachability and pairwise decision separation at a maximizing observation, no arbitrary finite-memory controller can use fewer. Under resettable diagnostics, nearest-prototype class inference has exponentially decaying error, and a calibrate-then-restart reduction transfers finite-MDP guarantees to the recovered state. The results enable \emph{Holonomy Memory Reinforcement Learning}. It represents memory by the current stable class, updates it through ordered edge transports, identifies local class coordinates when diagnostics are available, and applies a standard finite-MDP RL backbone after synchronization. Experiments recover an exact compression from raw states to quotient states and achieve perfect paired-order accuracy with three decision-time memory states, matching the quotient oracle and outperforming the non-oracle baselines.
中文摘要 在部分可观测性下工作的代理必须保留一个递归可更新的历史统计量以恢复马尔可夫性质，但最小的此类统计量通常未知。我们将该最小马尔可夫充分统计量描述为全纯覆盖决策过程的最小马尔可夫充分统计量，该类中可见动态为马尔可夫，且每个实现的可见转移都对隐藏模式应用固定置换。特别地，我们构造稳定商，即保持一步奖励和商后继的最粗观观察抽象，并证明当前观察类和稳定类的对构成一个精确有限的马尔可夫状态。当当前类正确初始化时，精确类追踪恰好需要最小的内存符号，即在可达性和最大化观测处的两对决策分离下，任何任意有限内存控制器都不能使用更少的符号。在可重置诊断下，最近原型类推断误差呈指数衰减，校准后重启的归约将有限MDP保证传递到恢复状态。这些结果使\emph{全息记忆强化学习}得以实现。它以当前稳定类表示内存，通过有序边传输更新，在可用诊断时识别本地类坐标，并在同步后应用标准有限MDP强化逻辑骨干。实验从原始状态精确恢复到商态的压缩，并通过三个决策时记忆状态实现完美的配对序精度，匹配商预言机并优于非预言机基线。

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

你真的需要预训练Q函数来进行在线强化学习微调吗？

Authors: Perry Dong, Ron Polonsky, Dorsa Sadigh, Chelsea Fin
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2607.27203
Pdf link: https://arxiv.org/pdf/2607.27203
Abstract Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's Q-function, not the Q-function that online fine-tuning converges to, and this gap persists even after offline value maximization. Motivated by this finding, we propose Initialization via Policy Ensemble (IPE), a simple method that trains multiple diverse policies and uses their pooled rollouts to bootstrap the Q-function learning in online RL. Across a suite of challenging continuous control benchmarks, IPE yields an average 1.26x improvement in fine-tuning performance over naive Q-function pre-training.
中文摘要 预训练后再微调已成为学习高效策略的主导配方，而在基于价值强化学习（RL）中，这自然引出了一个问题：既然有预训练策略，Q函数是否也应对离线数据进行预训练？传统观点认为应该如此，但最新结果显示，随机初始化Q函数的在线强化学习可以实现高效且可靠的策略，而无需预训练Q函数。本文系统地研究了在预训练基础策略基础上微调Q函数是否真的有帮助。令人惊讶的是，我们发现，朴素的Q函数预训练往往比随机初始化带来的好处有限。我们证明这源于一个根本性的不匹配：预训练中学到的Q函数针对的是预训练策略的Q函数，而非在线微调收敛到的Q函数，这一差距即使在离线值最大化后依然存在。基于这一发现，我们提出了“策略集合初始化”（Initialization through Policy Ensemble，简称IPE），这是一种简单方法，训练多种不同策略，并利用它们的池化部署来启动在线强化学习中的Q函数学习。在一系列具有挑战性的连续控制基准测试中，IPE在微调性能上平均提升1.26倍，优于朴素的Q函数预训练。

Keyword: diffusion policy

Temporally Centered SIGReg Improves Multi-Task LeWorldModel Learning: From Analysis to Method

以时间为中心的SIGReg提升多任务LeWorld模型学习：从分析到方法的改进

Authors: Chang Liu, Fei Suo, Yanzhou Jin, Yusuke Iwasawa, Yutaka Matsuo, Yaonan Zhu
Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2607.26924
Pdf link: https://arxiv.org/pdf/2607.26924
Abstract Recent work on LeWorldModel (LeWM) has shown that the Sketched Isotropic Gaussian Regularizer (SIGReg) enables stable end-to-end world-model learning from pixels by regularizing the latent marginal distribution toward an isotropic Gaussian, thereby preventing representation collapse. While effective and elegant in single-task settings, this recipe does not extend reliably to multi-task training, leading to substantially worse downstream behavior-cloning performance. In this paper, we show that marginal Gaussianization compresses the separation between task-dependent latent clusters relative to within-cluster variation. This compression introduces representation aliasing across tasks and states, and makes the learned representations highly sensitive to small visual perturbations. To address this problem, we apply SIGReg to temporally centered residuals rather than to the latent marginal distribution. This surrogate target places no direct regularization pressure on the separation among cluster centers, removes the requirement that the full latent follow a single isotropic Gaussian, and retains the anti-collapse effect of SIGReg. On the LIBERO benchmark, our method improves downstream success on the long-horizon suite by 1.7x and raises the average success rate across four suites from 53.2% to 73.6%. Without external pretraining, it slightly outperforms Diffusion Policy trained from scratch and approaches the performance of large-scale pretrained policy baselines. These results reveal a structural incompatibility between marginal Gaussian priors and multi-task latent structure, and provide a simple route toward stable and scalable end-to-end multi-task world-model learning.
中文摘要 LeWorldModel（LeWM）的最新研究表明，草图各向同性高斯正则化器（SIGReg）通过将潜在边际分布正则化至各向同性高斯分布，从而防止表征崩溃，从而实现从像素实现端到端稳定的世界模型学习。虽然该方案在单任务环境中有效且优雅，但无法可靠地推广到多任务训练，导致下游行为克隆性能显著下降。本文展示了边际高斯化压缩了任务依赖潜在簇相对于簇内变异的间距。这种压缩引入了任务和状态间的表示混叠，使所学表现对微小的视觉扰动极为敏感。为解决这一问题，我们将SIGReg应用于时间中心的残差，而非潜在边际分布。该替代目标不对簇中心间的分离施加直接正则化压力，取消了全潜在变量必须遵循单一各向同性高斯分布的要求，并保留了SIGReg的抗崩溃效应。在LIBERO基准测试中，我们的方法将长视野套件的后续成功率提高了1.7倍，并将四个套件的平均成功率从53.2%提升至73.6%。在没有外部预训练的情况下，它略优于从零训练的扩散策略，并接近大规模预训练策略基线的性能。这些结果揭示了边际高斯先验与多任务潜在结构之间的结构不兼容，并为实现稳定且可扩展的端到端多任务世界模型学习提供了简单路径。