生成时间: 2026-09-25 21:21:26 (UTC+8); Arxiv 发布时间: 2026-09-25 20:00 EDT (2026-09-26 08:00 UTC+8)

今天共有 40 篇相关文章

Keyword: reinforcement learning

Pistis Technical Report

Pistis技术报告

• Authors: Heyun Chen, Xiaohan Lan, Jiaxi Li, Zhilin Lu, Qi She, Weiwen Xu, Fei Yu, Yujie Zhong, Jinghuan Chen, Zijian Feng, Siyu Jiao, Yiheng Lin, Xinhao Wang, Sihan Yang, Jieyu You, Changbin Zhang, Hengyu Zhang, Xudong Zhang, Yunqing Zhao, Shuai Zheng
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.28554
• Pdf link: https://arxiv.org/pdf/2609.28554
• Abstract We introduce the Pistis model family, comprising 27B- and 9B-parameter multimodal large language models built on Qwen3.6 and Qwen3.5, respectively, and developed through a general and scalable post-training framework. The framework first establishes a strong foundation through large-scale multimodal supervised fine-tuning (SFT). Building on this SFT foundation, we propose Interleaved Distillation and Reinforcement Learning (IDRL), a novel post-training paradigm that tightly integrates on-policy distillation and reinforcement learning within a single training loop. By alternating between the two objectives, rather than optimizing either in isolation or combining them in a static joint loss, IDRL enables more effective knowledge transfer, greater optimization stability, and more precise credit assignment for long-horizon agentic trajectories, leading to stronger performance while mitigating common capability trade-offs. At both model scales, the framework produces two specialized variants: Pistis-Thinking, designed to strengthen deep multimodal reasoning, and Pistis-Agentic, which additionally incorporates agentic trajectory data to support long-horizon planning, iterative reasoning, and tool use. Pistis-Agentic is particularly strong in multimodal search. Both scales outperform their corresponding base models. Beyond model-parameter optimization, we further introduce Pistis-Auto-Harnessing (PAH), a system-level method that automatically improves the agent's inference harness through iterative optimization. Experiments demonstrate that PAH enhances the model performance without updating the model parameters or increasing the interaction budget.
• 中文摘要 我们介绍了Pistis模型家族，该家族分别由27B和9B参数的多模态大型语言模型组成，基于Qwen3.6和Qwen3.5，并通过通用且可扩展的后训练框架开发。该框架首先通过大规模多模态监督微调（SFT）奠定了坚实基础。基于该SFT基础，我们提出了交织蒸馏与强化学习（IDRL）的新颖后训练范式，将策略中提炼与强化学习紧密整合于单一训练循环中。通过在两者目标间交替进行优化，而非单独优化或在静态关节丢失中组合，IDRL实现了更高效的知识转移、更高的优化稳定性以及更精确的长期代理轨迹的信用分配，从而提升性能，同时减少共同能力的权衡。在两个模型尺度上，该框架产生了两种专业变体：Pistis-Thinking，旨在强化深度多模态推理;以及Pistis-Agentic，进一步整合了智能轨迹数据，支持长期规划、迭代推理和工具使用。Pistis-Agentic在多模态搜索方面尤为强大。这两个尺度的表现均优于对应的基础模型。除了模型参数优化外，我们还引入了Pistis-Auto-Harnessing（PAH），这是一种系统级方法，通过迭代优化自动提升智能体的推理能力。实验表明，PAH在不更新模型参数或增加交互预算的情况下，能够提升模型性能。

DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

DEEPO：双熵增强策略优化，用于MLLM中的幻觉

• Authors: Yingxuan Zhuang, Miao Pan, Wangjie Gan, Jingxiao Yang, Fan Wang, Weiming Liu, Cheng Tan, Xuhong Zhang, Jintao Chen
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.28570
• Pdf link: https://arxiv.org/pdf/2609.28570
• Abstract Reinforcement learning (RL) is widely used to sharpen reasoning in multimodal large language models (MLLMs), yet its effect on hallucination is uneven. We trace this to two weak points in the \emph{correction chain} from reward to parameter update. At the rollout level, hard queries---those with high semantic entropy---frequently produce unanimously wrong sample groups, collapsing the group-relative advantage to zero exactly where hallucination risk is highest. At the optimization level, confident-but-wrong tokens are gradient-invisible: a categorical policy's expected score-gradient norm vanishes as its distribution sharpens, so the predictions that most need correction receive the weakest updates. We propose Dual-Entropy Enhanced Policy Optimization (DEEPO), a dual-stage enhancement combining signal variance regularization with gradient preconditioning: semantic-entropy-triggered expert prefixes inject grounded continuations on high-uncertainty queries, providing direct supervision and restoring advantage variance, while advantage-sign-aware Renyi preconditioning counteracts logit-level saturation so correction reaches confident errors in the operational confidence regime. Both branches improve over GRPO individually; their interaction is statistically significant on VideoMMMU---the most complex long-horizon task in our evaluation suite (+4.0$, 95\% CI [1.1, 6.9])---and additive elsewhere. DEEPO reduces hallucination while preserving accuracy and training stability.
• 中文摘要 强化学习（RL）被广泛用于多模态大型语言模型（MLLMs）中提升推理能力，但其对幻觉的影响不均。我们将此追溯到从奖励到参数更新的\emph{修正链}中的两个薄弱点。在推广层面，硬查询---高语义熵的查询---常常产生一致错误的样本组，使群体相对优势在幻觉风险最高时归零。在优化层面，信心但错误的标记在梯度上是不可见的：类别策略的预期分数梯度范数随着分布的变速而消失，因此最需要纠正的预测更新最弱。我们提出了双熵增强策略优化（DEEPO），这是一种双阶段增强，结合了信号方差正则化与梯度预条件：语义熵触发的专家前缀在高不确定性查询上注入有基础的延续，提供直接监督并恢复优势方差，而优势符号感知的Renyi预条件则抵消logit级别的饱和，使修正达到操作置信区内的信心误差。这两个分支单独相较于GRPO有所提升;它们在VideoMMMU上的交互具有统计学意义---这是我们评估套件中最复杂的长视野任务（+4.0美元，95%置信区间[1.1， 6.9]）---在其他地方也是加法。DEEPO在保持准确性和训练稳定性的同时减少幻觉。

RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theory

RLVR的迭代乘法景观可能是良性的：自旋玻璃理论的见解

• Authors: Noa Rubin, Zohar Ringel
• Subjects: Subjects: Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.28625
• Pdf link: https://arxiv.org/pdf/2609.28625
• Abstract Despite the importance of reinforcement learning with verifiable rewards (RLVR), the extent to which it can learn new reasoning capabilities remains debated. Here we study the optimization landscape of RLVR on algorithmic tasks, such as iterated group and quasigroup multiplication. To this end, we map entropy-regularized RLVR over myopic tabular policies onto an energy-based (spin-glass) model over deterministic policies. This mapping upper-bounds what RLVR can achieve, and lets us rigorously characterize the landscape in this tabular setting. We show, both theoretically and experimentally, that for a wide class of models and tasks with uncorrelated inputs, this landscape is benign, containing no local minima that could trap RLVR training. Rather, the practical difficulty of these tasks appears to stem, at least in part, from issues such as diffusive barriers and gradient-estimation error in traversing the landscape. These are genuine obstacles that can prevent a solution from being found, but they are distinct from the landscape itself being rugged. We show that these obstacles can often be mitigated through the choice of entropy regulator. Consistent with this theory, we find that a transformer trained from scratch, using only last-token rewards, successfully learns an algorithmic chain of thought for iterated non-Abelian group multiplications.
• 中文摘要 尽管带有可验证奖励的强化学习（RLVR）非常重要，但它学习新推理能力的程度仍存在争议。本文我们研究RLVR在算法任务（如迭代群和准群乘法）上的优化景观。为此，我们将熵正则化的RLVR在近视表式策略上映射到基于能量的（自旋玻璃）模型上，基于确定性策略。这种映射映射了RLVR的上界，使我们能够严格地在该表格环境中刻画景观。我们从理论和实验上都证明，对于一类输入无关的模型和任务，该景观是良性的，不存在可能限制RLVR训练的局部极小值。相反，这些任务的实际难度至少部分源于遍历景观时的扩散障碍和梯度估计误差等问题。这些确实是阻碍解的障碍，但它们与地形本身的崎岖是不同的。我们证明这些障碍通常可以通过选择熵调节器来缓解。与该理论一致，我们发现一个从零开始训练的变换器，仅使用最后一个令子奖励，能够成功学习一套算法思维链，进行迭代非阿贝尔群乘法。

Morphometric Imitation: From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

形态计量模仿：从形态学和接触感知手部重定向到模拟到真实的体力运动策略

• Authors: Tara Sadjadpour, Siming He, C.K. Wolfe, Haozhi Qi, Lea Wilken, S. Shankar Sastry, Claire Tomlin, Jitendra Malik
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.28660
• Pdf link: https://arxiv.org/pdf/2609.28660
• Abstract Human hand-object interactions (HOIs) provide a rich source of demonstrations for dexterous manipulation, but learning directly from them presents challenges in bridging morphology gaps, ensuring dynamical feasibility, and sim-to-real deployment. We present Morphometric Imitation, a three-stage framework that transforms reconstructed HOIs into zero-shot sim-to-real visuomotor policies. First, morphometric optimization (MMO) kinematically retargets human motion across hand morphologies while preserving demonstrated contacts. Second, residual reinforcement learning (RL) refines the kinematic reference using object pose and contact information from the human motion to produce dynamically feasible robot demonstrations. Third, these demonstrations are distilled into visuomotor policies. Across three robot hands and ten HOIs, MMO improves contact F1 over the strongest of five baselines by at least 8 points for every hand, while also improving the success rate of downstream dynamic retargeting by as much as 35 points. Ablations on the residual RL show complementary benefits from using object pose and contact information. Finally, the visuomotor policies achieve 89.3% zero-shot success in 300 real-world trials on 30 objects. Project page: $\href{this https URL}{\text{this https URL}}$
• 中文摘要 人手与物体交互（HOIs）为灵巧操作提供了丰富的演示来源，但直接从中学习在弥合形态学差距、确保动力学可行性和模拟到现实部署方面存在挑战。我们提出了形态计量模仿（Morphometric Imitation），这是一个三阶段框架，将重建的HOI转化为零射模拟到现实的视觉运动策略。首先，形态计量优化（MMO）在保持已展示接触的同时，通过运动学重新定位人类在手形上的运动。其次，残余强化学习（RL）利用人体运动中的物体姿态和接触信息，精炼运动学参考，生成动态可行的机器人演示。第三，这些演示被提炼为视觉运动策略。在三只机器人手和十个HOI中，MMO每手至少提升了5个基线中最强的接触F18分，同时也将下游动态重定向的成功率提升了多达35分。对残余强化学习的消融显示，使用物体姿势和接触信息具有互补益处。最后，视觉运动策略在30个物体的300个真实世界试验中实现了89.3%的零射击成功率。项目页面：$\href{此 https URL}{\text{this https URL}}$

Beyond Surface Style: Aligning Multi-Turn User Simulators with Behavioral Consistency

超越表面风格：将多回合用户模拟器与行为一致性相结合

• Authors: Geng Chen, Ruotong Pan, Zhirui Yang, Qiqi He, Jiawei Chen, Zhang Yunfei, Chongyuan Chen, Minxuan Lv, Zheng Yang, Win-Bin Huang, Xiangyu Wu, Wenwu Ou
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.28690
• Pdf link: https://arxiv.org/pdf/2609.28690
• Abstract Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. However, plausible individual responses do not ensure that simulated users reproduce the intent evolution and outcomes observed in real interactions. We propose TRACER, a multi-turn user simulator that explicitly models users' evolving intent and learns to align simulated behavior with real interaction trajectories. TRACER is trained in two stages: supervised fine-tuning on real user dialogues, followed by multi-turn reinforcement learning. The RL stage combines hierarchical outcome- and trajectory-level rewards with deviation-aware advantage modulation, jointly mitigating reward sparsity and credit assignment in long dialogues. On real customer-service sessions organized into reference cohorts, TRACER-7B surpasses the strongest baseline by 11.4 conversion F1, while also achieving the lowest group-level conversion-rate error and semantic trajectory distance, and generalizing to out-of-distribution scenarios. Human Turing tests yield identification accuracy close to chance, supporting the perceived naturalness of generated conversations. Building on this simulator, we further introduce the Dynamic Marketing Benchmark, which jointly evaluates persuasion effectiveness and response quality of LLMs through simulated interactions, revealing that higher response quality does not necessarily correspond to higher conversion rates.
• 中文摘要 忠实的用户模拟对于大规模构建、评估和改进交互式人工智能至关重要。然而，合理的个别反应并不能保证模拟用户能够重现真实交互中观察到的意图演变和结果。我们提出了TRACER，一种多回合用户模拟器，明确模拟用户意图演变，并学习将模拟行为与真实交互轨迹对齐。TRACER训练分为两个阶段：对真实用户对话进行监督微调，随后进行多回合强化学习。强化学习阶段结合了层级结果和轨迹级奖励与偏差感知优势调制，共同缓解长对话中的奖励稀疏性和信用分配。在以参考队列为单位组织的真实客户服务会话中，TRACER-7B以11.4的转化率F1高出最强基线，同时实现了最低的组级转换率误差和语义轨迹距离，并能推广到分布外场景。人类图灵测试的识别准确率接近偶然，支持了生成对话的自然性感知。基于该模拟器，我们进一步介绍了动态营销基准，该基准通过模拟交互共同评估LLMs的说服效果和响应质量，揭示更高的响应质量不一定等同于更高的转化率。

Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning

强化学习中的策略复杂性、反应时间与有界理性

• Authors: James Wu, Chris R. Sims
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Theory (cs.IT)
• Arxiv link: https://arxiv.org/abs/2609.28737
• Pdf link: https://arxiv.org/pdf/2609.28737
• Abstract Biological agents do not learn under conditions of unlimited computation. For humans, learning and choice are shaped by constraints on perception, attention, and working memory, which limit how much state information guides behavior and therefore bound policy complexity. Standard reinforcement learning models typically optimize reward without explicitly representing these internal costs, making them less suitable as models of biological intelligence. We derive MI-SARSA, an on-policy temporal-difference algorithm that incorporates mutual-information regularization through a learned marginal action prior and a penalty on state-specific deviations from that prior. This yields a sequential learning model in which state information is used selectively when its expected return benefit justifies the added informational cost. Critically, the same state-specific information cost that governs policy compression also generates trial-level predictions for reaction time, distinguishing MI-SARSA from most reinforcement learning models, which predict choices or returns but not latency. Empirically, MI-SARSA produces a reward-complexity tradeoff, and stronger information penalties produce simpler policies with lower control costs and faster reaction times. Under environment shift, increasing regularization reduces post-switch performance degradation but also lowers asymptotic return, revealing a robustness-capacity tradeoff. Together, these results position MI-SARSA as a model of bounded sequential learning under cognitive constraints.
• 中文摘要 生物代理在无限计算条件下不会学习。对于人类来说，学习和选择受感知、注意力和工作记忆的约束，这些限制了状态信息对行为的引导，从而限制了策略复杂度。标准强化学习模型通常优化奖励，但不明确表示这些内部成本，因此它们作为生物智能模型的适用性较低。我们推导出MI-SARSA算法，这是一种策略上的时间差分算法，通过学习边际动作先验实现互信息正则化，并对状态特定偏离先验施加惩罚。这形成了一个顺序学习模型，当状态信息的预期回报收益值得额外信息成本时，会选择性地使用。关键是，控制策略压缩的同一状态特定信息成本也产生了反应时间的试验级预测，这使MI-SARSA区别于大多数强化学习模型，后者预测选择或回报但不预测延迟。实证上，MI-SARSA产生了奖励与复杂度的权衡，而更强的信息惩罚则产生更简单的策略，控制成本更低，反应时间更快。在环境转变下，增加正则化减少切换后的性能退化，但也降低渐近收益，揭示了鲁棒性与容量的权衡。综合来看，这些结果使MI-SARSA成为认知约束下有界序列学习的模型。

Reinforcement Learning with Verifiable Rewards for Small Search Agents

小型搜索代理的可验证奖励强化学习

• Authors: Gaurisankar Jayadas, Aske Plaat, Álvaro Serra-Gómez, Sandheep P
• Subjects: Subjects: Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2609.28765
• Pdf link: https://arxiv.org/pdf/2609.28765
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) performs well on problems with clear rewards, such as mathematics and coding, but whether it also works where the reward is less clear remains open. The reason-over-search recipe applies RLVR to open-domain question answering, where retrieval grounds the answer and a match against the reference supplies the reward. So far it has been demonstrated on large models, and below one billion parameters only with distillation from a larger teacher. We test the recipe on a small model. We train Qwen3.5-0.8B with Group Relative Policy Optimization (GRPO) and an interleaved Wikipedia-search tool on MuSiQue, varying only the reward across three shapes over three seeds each, and we evaluate every checkpoint held-out on a seven-benchmark question-answering suite. The recipe works: the best run reaches 0.352 average exact match against a 0.092 untrained floor, a 3.8-fold gain, with no distillation step in the training loop. The reward shape also matters. The Search-R1-faithful exact-match-only reward is the worst of the three at every seed at the matched training horizon, and it is worst even on exact match, the metric it directly optimises. We conclude that the sparse exact-match reward, RLVR's default in mathematics and code, is the wrong starting point for models of this size. The reason-over-search setting can supply a suitable reward for RLVR on small models, but small-model RLVR needs its own reward-design study rather than a scaled-down copy of a large-model recipe.
• 中文摘要 带可验证奖励的强化学习（RLVR）在奖励明确的问题上表现良好，如数学和编码，但是否也能在奖励不那么明确的问题上有效尚待观察。理由优先搜索的方案将RLVR应用于开放域问答，检索为答案提供基础，与参考匹配则获得奖励。迄今为止，该方法已在大型模型上得到验证，且仅在十亿参数以下通过较大教师的提炼实现。我们在小模型上测试该配方。我们用群相对策略优化（GRPO）和交错的维基百科搜索工具在MuSiQu上训练Qwen3.5-0.8B，仅在三个形状上变化奖励，每个形状对三个种子进行调整，并评估七个基准问答套件中所有检查点。该配方有效：最佳运行达到0.352的平均精确匹配，而0.092的未训练底线，获得3.8倍的收益，且训练循环中没有提炼步骤。奖励形状也很重要。Search-R1忠实的精确匹配仅奖励是三者中最差的，在匹配训练视野的每个种子下，甚至在其直接优化的指标“精确匹配”上也最差。我们得出结论，稀疏精确匹配奖励是RLVR数学和代码中的默认选择，但对于这种规模的模型来说，是错误的起点。理由优先搜索设置可以为RLVR在小模型上提供合适的奖励，但小模型RLVR需要自己的奖励设计研究，而不是大型模型配方的缩减版。

An Analysis of Streaming Deep Reinforcement Learning for Adaptive Continual Learning in Robotics

机器人学中流式深度强化学习在自适应持续学习中的分析

• Authors: Teeratham Vitchutripop, Alyssa Quarles, Wenhe Zhang, Richard Xue, Daniel Rakita
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.28807
• Pdf link: https://arxiv.org/pdf/2609.28807
• Abstract Over the course of a lifetime, robots may encounter novel scenarios unaccounted for in its original training that result in performance degradation. One common approach to mitigating this issue is to further grow the offline training dataset in hopes of producing a policy robust to these changes. In contrast, biological learning occurs moment-to-moment via a stream of experience, unlike the predominantly batch-based and offline nature of deep learning. Although recent works show the feasibility of stream-based deep reinforcement learning, where updates use only the latest experience, none have shown it to be a viable continual learning framework for adapting robotic policies to unseen changes. In this paper, we present the first analysis of streaming deep reinforcement learning for adaptive continual learning in robotics. In particular, we show that, following an initial pretraining phase, streaming deep RL can enable a robot to successfully adapt to unforeseen changes to itself, its environment, or goals. Our primary experiments within quadruped locomotion demonstrate that a deep neural network robotic policy with certain optimizers and plasticity loss mitigation techniques can successfully leverage domain task knowledge from its pretraining to quickly adapt online to diverse changes via stream learning, outperforming batch-based on-policy methods and improving task success rates by up to 90% over the pretrained policy. Furthermore, we perform additional evaluations on robotic manipulation tasks to determine if our previous observations extend to different robotic morphologies and scenarios. Our results show that the successes observed in quadruped locomotion can be partially realized in manipulation with stability and performance limitations. We conclude with a discussion on the limitations of our work and its implications for the future of continual robot learning.
• 中文摘要 在一生中，机器人可能会遇到原始训练中未考虑的新情景，导致性能下降。缓解这一问题的一种常见方法是进一步扩展离线训练数据集，期望制定出对这些变化有韧性的策略。相比之下，生物学习通过经验流瞬间发生，这与深度学习主要以批处理和离线为主的特性不同。尽管近期研究显示基于流的深度强化学习可行，即仅使用最新经验，但尚无研究证明它是适应机器人策略以适应未知变化的可行持续学习框架。本文首次分析了机器人学中流深度强化学习用于自适应持续学习的分析。特别地，我们表明，经过初始预训练阶段后，流深度强化学习能够使机器人成功适应自身、环境或目标的不可预见变化。我们在四足行走中的主要实验表明，结合某些优化器和可塑性损失缓解技术的深度神经网络机器人策略，可以成功利用预训练中获得的领域任务知识，通过流学习快速在线适应多样化变化，优于基于批次的策略方法，并将任务成功率提升多达90%。此外，我们还对机器人操作任务进行了额外评估，以确定之前的观察是否适用于不同的机器人形态和场景。我们的结果表明，四足行走中观察到的成功可以在具备稳定性和性能限制的操作中部分实现。最后，我们讨论了我们工作的局限性及其对持续机器人学习未来的启示。

FlyCNS: Connectome-Grounded Information Organization for Communication-Constrained Embodied Control

FlyCNS：连接组接地信息组织，用于通信受限的内在控制

• Authors: Jinchang Zhang, Jiakai Lin, Guoyu Lu
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.28816
• Pdf link: https://arxiv.org/pdf/2609.28816
• Abstract Robotic bodies are inherently distributed in sensing and actuation, yet learning-based control still commonly relies on centralized information processing. This work studies the problem of information organization in communication-constrained embodied control: which computations should remain local, and which information is worth transmitting for whole-body coordination. We propose FlyCNS, an embodied information-organization framework inspired by the Drosophila brain--nerve-cord connectome. FlyCNS preserves local sensorimotor computation within each limb and enables selective long-range communication through separate ascending and descending routing pathways. From a real connectome, FlyCNS extracts the directional structural complexity of these two pathway types and uses it as a weak prior over communication allocation, while message content, transmission timing, and locomotion policies remain task-adaptive and are learned through reinforcement learning. In Unitree Go1 simulation, FlyCNS exhibits more graceful performance degradation as the communication budget is tightened. Under the most restrictive setting, it uses only about 21--22\% of the communication of the full-communication reference, while still maintaining a tracking score of approximately 0.882 under both command protocols, with a gap of no more than 6.1\% from the full-communication reference. These results indicate that real neural connectomes can inform not only the structural design of control networks, but also provide transferable inductive biases for information organization across embodiments, guiding robots in balancing local computation and long-range coordination under limited communication resources.
• 中文摘要 机器人身体本质上分布于感知和驱动，但基于学习的控制仍常依赖集中式信息处理。本研究研究通信约束的具身控制中信息组织问题：哪些计算应保持局部，哪些信息值得为全身协调传输。我们提出了FlyCNS，一种受果蝇脑神经-脊连接组启发的具身信息组织框架。FlyCNS保留了每条肢体的局部感觉运动计算，并通过分别的上行和下行路径实现选择性的远程通信。从真实连接组中，FlyCNS提取这两种路径类型的方向结构复杂性，并将其作为通信分配的弱先验，而消息内容、传输时机和运动策略则保持任务适应性，通过强化学习学习。在Unitree Go1仿真中，随着通信预算收缩，FlyCNS表现得更为优雅。在最严格的设置下，它仅使用全通信参考通信约21-22/%的通信量，同时在两种指令协议下跟踪得分约为0.882，与全通信参考的差距不超过6.1%。这些结果表明，真实的神经连接组不仅能指导控制网络的结构设计，还能为跨实体的信息组织提供可转移的归纳偏置，指导机器人在有限通信资源下平衡局部计算和远程协调。

Uncertainty-Gated Exploration Noise Suppresses Task Collapse in Online RL Fine-Tuning of a Flow-Matching Vision-Language-Action Policy

不确定性门控探索噪声抑制在线强化学习中任务崩溃，对流程匹配视觉-语言-行动策略进行微调

• Authors: Mehmet Turan Yardımcı, Yunus Emre Çoğurcu
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.28838
• Pdf link: https://arxiv.org/pdf/2609.28838
• Abstract Online reinforcement learning fine-tuning of pretrained flow-matching vision-language-action (VLA) policies promises robots that keep learning after deployment, but continued updates often destroy competence on individual tasks while the aggregate still looks healthy. We study this failure mode, which we call task collapse, under a matched small-compute budget on LIBERO-10 with a 450M-parameter SmolVLA policy trained by PPO with stochastic (SDE) sampling. Three exploration-noise policies differ in one live variable: a fixed noise scale, a ReinFlow-style learned noise network, and an uncertainty-gated controller that redistributes exploration across task streams from task-agnostic novelty and competence signals, without task labels or episode boundaries. Under the pooled definition, fixed noise collapses tasks in two of three seeds and learned noise in every seed measured to iteration 200, while the controller collapses none in any of its three seeds. Measured parameter displacement shows the controller's action expert keeps changing, while its mean applied noise is close to the fixed scale in the available logs. The matched comparison supports the controller's effect on task preservation; the separate contributions of its adaptation across states and over time are not disentangled. A lower fixed scale slows the decline but does not stop it. No arm improves on the behavior-cloning baseline in this budget. Two properties of that regime are measured beside this result, not offered as its cause: following the reference recipe, training runs in bfloat16 with no fp32 master copy, under which 96.02% of the action expert's elements stay bit-identical across three consecutive iterations, and an fp32 master copy at the reference learning rate collapses both arms in a single-seed observation. We release tools measuring per-task collapse under four definitions, rescoring noise and instrument tares.
• 中文摘要 在线强化学习对预训练流匹配视觉-语言-动作（VLA）策略的微调承诺机器人在部署后仍能持续学习，但持续更新往往会破坏单个任务的能力，而整体整体仍保持健康。我们在LIBERO-10匹配的小计算预算下研究了这种失败模式，称为任务崩溃，该策略由PPO通过随机采样（SDE）训练的4.5亿参数SmolVLA策略。三种探索噪声策略在一个实时变量上有所不同：固定噪声尺度、类似ReinFlow的学习噪声网络，以及一个不确定性门控控制器，该控制器将探索重新分配到任务流中，基于任务无关的新颖性和能力信号，且无任务标签或片段边界。根据合并定义，固定噪声在三个种子中有两个会坍缩任务，在每个迭代200次的种子中合并任务，而控制器在其三个种子中都没有坍缩。测量的参数位移显示控制器的动作专家持续变化，而其平均应用噪声接近可用日志中的固定尺度。匹配比较支持控制器对任务保存的影响;其跨状态和时间的适应贡献未被解开。较低的固定尺度减缓下降，但无法阻止。在该预算中，没有任何臂能提升行为克隆基线。该方案的两个性质与此结果并列，但未作为原因被测量：按照参考配方，在bfloat16中进行训练运行且无FP32主副本，在该条件下，96.02%的动作专家元素在连续三次迭代中保持位元相同;且在参考学习率下，FP32主副本在单种子观测中会双侧坍缩。我们发布了基于四种定义测量每个任务坍缩的工具，分别是噪声重测和仪器的尾距。

Back to the Definition: Estimating Step-Level Advantages via Trajectory Graphs for Agentic Reinforcement Learning

返回定义：通过轨迹图估算代理强化学习的步骤级优势

• Authors: Xincheng Yao, Haobo Fu, Weiming Liu, Chongyang Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (stat.ML)
• Arxiv link: https://arxiv.org/abs/2609.28963
• Pdf link: https://arxiv.org/pdf/2609.28963
• Abstract Group-based reinforcement learning (RL) methods, such as GRPO and its variants, have become a leading paradigm for training reasoning and agentic large language models (LLMs). While their group-normalized advantage estimation is reliable at the response level, it becomes systematically biased at the step level, since coarse-grained trajectory-level advantages are hard to accurately reflect the contribution of individual steps (i.e, failed trajectories may contain valuable steps). Revisiting the foundational RL definition, we notice that GRPO's success on single-turn tasks stems from its advantage estimation strategy, which adheres to the basic definition: the mean reward of multiple actions sampled from the same state constitutes a credible state-value estimate. Extending the faithful estimation to step-level would in principle demand sampling multiple actions from each intermediate state, which is too costly on a per-state basis. To mitigate this issue, we propose a Graph-based Faithful sTep-level credit-assignment framework (GRAFT) that grafts all rollout trajectories into a trajectory graph, recovering node state-values via Bellman iteration on the graph, and assigning credit to each edge by the node value difference. Theoretically, the estimated step-level advantage faithfully adheres to the basic advantage definition in RL. To further ensure the reliability of step-level advantage estimation, we further propose Graph GAE, which extends GAE to the trajectory graph for reducing the impact of state-value estimation bias. Experiments across a range of multi-turn agentic benchmarks show consistent gains over GRPO and superior performance compared to recent agentic RL algorithms. Code will be available at this https URL.
• 中文摘要 基于群体的强化学习（RL）方法，如GRPO及其变体，已成为训练推理和代理大型语言模型（LLMs）的领先范式。虽然它们的群体归一化优势估计在响应层面可靠，但在步骤层面会有系统性偏差，因为粗粒度轨迹层面的优势难以准确反映单个步骤的贡献（即失败的轨迹可能包含有价值的步骤）。回顾基础强化学习定义，我们注意到GRPO在单回合任务上的成功源于其优势估计策略，该策略遵循基本定义：从同一状态抽样的多个动作的平均奖励构成可信的状态价值估计。将忠实估计扩展到步级，原则上需要从每个中间状态抽样多个动作，这在每个状态上成本过高。为缓解此问题，我们提出了基于图的忠实sTep级信用分配框架（GRAFT），将所有推广轨迹嫁接到轨迹图中，通过Bellman迭代恢复节点状态值，并根据节点值差分配给每条边。理论上，估计的步级优势忠实遵循强化学习中的基本优势定义。为进一步确保步级优势估计的可靠性，我们进一步提出了GAE图，将GAE扩展到轨迹图，以减少状态值估计偏差的影响。跨越多种多回合代理基准测试的实验显示，相较于近期代理强化学习算法，持续提升GRPO且性能更优。代码将在此https网址提供。

Deep-learning-aided dismantling of interdependent networks

深度学习辅助拆解相互依赖网络

• Authors: Weiwei Gu, Chen Yang, Lei Li, Jinqiang Hou, Filippo Radicchi
• Subjects: Subjects: Social and Information Networks (cs.SI)
• Arxiv link: https://arxiv.org/abs/2609.28977
• Pdf link: https://arxiv.org/pdf/2609.28977
• Abstract Identifying the minimal set of nodes whose removal breaks a complex network apart, also referred as the network dismantling problem, is a highly non-trivial task with applications in multiple domains. Whereas network dismantling has been extensively studied over the past decade, research has primarily focused on the formulations of the optimization problem for single-layer networks, neglecting that many, if not all, real networks display multiple layers of interdependent interactions. In such networks, the optimization problem is fundamentally different as the effect of removing nodes propagates within and across layers in a way that can not be predicted using a single-layer perspective. Here, we propose a dismantling algorithm named MultiDismantler, which leverages multiplex network representation and deep reinforcement learning to optimally dismantle multi-layer interdependent networks. MultiDismantler is trained on small synthetic multiplex graphs; when applied to large, real and synthetic networks, it displays exceptional dismantling performance, clearly outperforming all existing methods that rely on a single-layer approach to network dismantling. We show that MultiDismantler is effective in guiding strategies for the containment of diseases in social networks characterized by multiple layers of social interactions. Also, we show that MultiDismantler is useful in the design of protocols aimed at delaying the onset of cascading failures in interdependent critical infrastructures.
• 中文摘要 识别哪一组节点被移除后会破坏复杂网络，也称为网络拆解问题，是一项极其艰难的任务，应用涵盖多个领域。过去十年网络拆解被广泛研究，但研究主要集中在单层网络优化问题的表述上，忽视了许多甚至全部真实网络都显示多层相互依赖的交互。在此类网络中，优化问题本质上不同，因为移除节点的影响会以无法通过单层视角预测的方式在层内和层间传播。在这里，我们提出了一种名为MultiDismantler的拆解算法，它利用多路复用网络表示和深度强化学习，以最优方式拆解多层相互依赖的网络。MultiDismantler训练于小型合成复用图;当应用于大型、真实和合成网络时，它展现出卓越的拆解性能，明显优于所有依赖单层网络拆解方法的现有方法。我们证明了MultiDismantler在引导具有多层社会互动特征的社交网络中疾病控制策略方面非常有效。此外，我们还展示了MultiDismantler在设计旨在延迟相互依赖关键基础设施中连锁故障发生的协议中非常有用。

Learning from Mixed-Quality Deployment Experience for Robot Manipulation

从混合质量部署经验中学习机器人操作

• Authors: Yangang Ren, Yujie Yan, Zirui Li, Jiaming Guo, Di Zeng, Ji Tao, Lan Yu, Xuesong Tian, Chen Lv
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.29000
• Pdf link: https://arxiv.org/pdf/2609.29000
• Abstract Robot policies deployed in real environments naturally accumulate mixed-quality experience, including successful executions, partial progress, and failures. Although these rollouts provide valuable information for further learning, directly incorporating them into imitation learning may reinforce undesirable behaviors, while offline reinforcement learning often suffers from unreliable value estimation under sparse rewards and limited data coverage. We consider a practical post-deployment setting where learning relies only on naturally accumulated autonomous rollouts, without additional human corrections or exploratory interaction. To effectively exploit such experience, we propose Predictive Action Chunk Learning (PACL). PACL first learns a predictive chunk-level critic that evaluates temporally extended action sequences and augments temporal difference learning with future latent prediction, providing richer supervision for long-horizon value estimation. The learned critic then converts chunk-level Q-values into discrete quality conditions, which guide a diffusion actor to learn jointly from these mixed-quality experiences without treating all behaviors as equivalent supervision. At inference, the actor generates multiple action chunks and the critic selects the highest valued candidate. Experiments across simulated and real-world robot manipulation tasks show that PACL consistently improves the pretrained policy and outperforms strong imitation learning and offline reinforcement learning baselines.
• 中文摘要 在真实环境中部署的机器人策略自然积累了高质量参差的经验，包括成功执行、部分进展和失败。虽然这些推广为进一步学习提供了宝贵信息，但直接将其纳入模仿学习可能会强化不良行为，而离线强化学习在奖励稀疏和数据覆盖有限的情况下，往往存在价值估计不可靠的问题。我们考虑了一个实用的部署后环境，学习仅依赖自然积累的自主部署，无需额外的人工修正或探索性交互。为有效利用此类经验，我们提出了预测动作块学习（PACL）。PACL首先学习一个预测性块级批评器，评估时间扩展的动作序列，并通过潜在预测增强时间差分学习，为长视野价值估计提供更丰富的监督。学习到的批评者随后将区块级Q值转换为离散质量条件，引导扩散演员从这些混合质量的体验中共同学习，而无需将所有行为视为等价的监督。在推理时，演员生成多个动作块，批评者选择价值最高的候选。在模拟和现实世界机器人操作任务中的实验表明，PACL持续改进预训练策略，并优于强模仿学习和离线强化学习基线。

Outcome-Sensitive Motion Search for Impact-Aware Dexterous Catching

结果敏感运动搜索，实现冲击感知的灵巧接球

• Authors: Guorui Pei, Jinsong Wu, Songyuan Su, Jiaming Qi, Sichao Liu, David Navarro-Alarcon, Bin Liu, Peng Zhou
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.29020
• Pdf link: https://arxiv.org/pdf/2609.29020
• Abstract Skilled humans can catch fast-moving objects softly by coordinating interception, velocity matching, and follow-through to mitigate impact. Learning such impact-aware catching with reinforcement learning (RL), however, is challenging, as the policy must achieve reliable interception and grasping while regulating the sensitive transition into contact. Moreover, even a capable privileged-state RL teacher may not provide ideal demonstrations for a deployable imitation-learning (IL) student: teacher failures limit task coverage, while small variations in pre-contact motion can produce substantially different impact and grasping outcomes. We characterize this phenomenon through interventional outcome sensitivity and introduce the outcome-sensitive window (OSW) to guide targeted demonstration construction. Building on this formulation, we propose Outcome-Sensitive Motion Search, which learns a task-conditioned manifold of successful OSW motions and performs local geodesic search to refine successful teacher rollouts and repair task conditions where the teacher fails. We then validate candidate motions through complete rollouts under a calibrated IL-student action-error model and retain only successful executions as demonstrations. Extensive simulation experiments demonstrate that our method effectively repairs task conditions where the teacher fails and enables the resulting IL policy to outperform the privileged RL teacher in both catching success and impact mitigation.
• 中文摘要 熟练的人类可以通过协调拦截、速度匹配和跟进动作来轻柔地捕捉快速移动的物体，从而减轻影响。然而，通过强化学习（RL）学习这种撞击感知捕捉具有挑战性，因为该策略必须在调节敏感过渡到接触的同时，实现可靠的拦截和抓取。此外，即使是能力强的特权状态强化学习教师，也可能无法为可部署的模仿学习（IL）学生提供理想的演示：教师的失误限制了任务覆盖范围，而接触前动作的微小变化则可能导致显著不同的影响和抓取结果。我们通过介入性结果敏感性来描述这一现象，并引入结果敏感窗口（OSW）以指导有针对性的演示构建。基于此表述，我们提出了结果敏感运动搜索，该方法学习一个任务条件流形的成功OSW动作，并进行局部测地线搜索以优化教师成功的部署并修复教师失败的任务条件。随后，我们通过校准的IL-学生动作错误模型验证候选动作，并仅保留成功执行的演示。大量模拟实验表明，我们的方法有效修复教师失败的任务条件，使得最终IL策略在捕捉成功和影响缓解方面优于特权强化教师。

Simple Torque-Observation Alignment for Zero-Shot Sim-to-Real Grasping with a Direct-Drive Gripper

简单的扭矩观察对准，用于零次模拟到实抓，使用直接驱动夹持器

• Authors: Doyoung Kim, Edgar Lee, Hyeonsun Park, Chunghyeon Lee, Chihyun Han, Uisu Hwang, Seokhwan Jeong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.29031
• Pdf link: https://arxiv.org/pdf/2609.29031
• Abstract Torque observations in reinforcement learning remain challenging because simulated and measured torque differ in scale, offset, and noise. In this paper, we propose a simple torque observation alignment method for robots with direct-drive (DD) actuators, in which motor current maps linearly to joint torque through a motor-type-specific torque constant K_tau. First, dynamometer calibration identifies K_tau* and corrects the scale mismatch between simulated and real torque. Second, the method uses delta_tau(t) = tau(t) - tau(t-1) as the observation in both domains to eliminate the constant offset instead of using the direct torque tau(t), which carries a domain-dependent bias. Third, Gaussian noise obtained from the dynamometer measurement data is injected during the learning process. To validate the proposed method, we train a teacher-student grasping policy entirely in simulation and deploy the distilled student on a multifingered DD gripper. The deployed policy performs proprioceptive grasping using only joint positions and torque differences. We conduct an ablation study comparing the proposed method with alternative alignment variants on nine in-distribution (ID) objects. The proposed method achieves 100% grasp success. These results demonstrate that the proposed alignment method improves the robustness of zero-shot policy transfer on the DD gripper against real-world torque-observation mismatches.
• 中文摘要 强化学习中的扭矩观测仍然具有挑战性，因为模拟和测量的扭矩在尺度、偏移和噪声上存在差异。本文提出了一种简单的扭矩观察对准方法，适用于配备直驱（DD）执行器的机器人，其中电机电流通过电机类型的特定扭矩常数K_tau线性映射到关节扭矩。首先，测功机校准识别K_tau*并纠正模拟扭矩与真实扭矩的尺度不匹配。其次，该方法在两个领域均使用delta_tau（t） = tau（t） - tau（t-1）作为观测值，以消除恒定偏移，而非使用带有域依赖偏差的直接扭矩tau（t）。第三，学习过程中注入从测功机测量数据获得的高斯噪声。为验证拟议方法，我们完全在模拟中训练师生抓握策略，并将精炼学生部署在多指DD夹持器上。部署策略仅通过关节位置和扭矩差执行本体感觉抓握。我们进行了消融研究，比较拟议方法与九个分布内（ID）对象的替代对齐变体。该方法实现了100%的抓取成功率。这些结果表明，拟议的对齐方法提升了DD夹持器零射击策略转移对真实扭矩观测不匹配的鲁棒性。

SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL

SLCA-GRPO：解决工具调用强化学习中的跨部门信用错误归因

• Authors: Yan Zhan, Shaobo Liu, Qiunan Liu, Yuanjun Shi, Siqi Xu, WeiYi Hou, Xiang Xu, Zekang Li, Weizhou Pan, Jiahong Yan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.29050
• Pdf link: https://arxiv.org/pdf/2609.29050
• Abstract Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries. This output heterogeneity presents a structural failure mode in standard on-policy Reinforcement Learning (RL): algorithms like GRPO indiscriminately broadcast a homogeneous trajectory-level scalar advantage to all tokens. Consequently, gradient noise from summary generation leaks into tool-decision tokens, causing cross-segment credit misattribution and brittle optimization. In this work, we propose SLCA-GRPO, a framework incorporating Segment-Locked Credit Assignment (SLCA). To enable scalable exploration without costly real APIs and stable training, we first construct the Schema-Guided LLM Simulator (SGLS) as foundational training infrastructure. Building on this, SLCA decouples advantage estimation at the structural segment level within a single group of rollouts, without requiring additional rollouts from intermediate states. Supported by Hierarchical Rewards (HierR), SLCA routes execution advantages to tool tokens and preference advantages to summary tokens, eliminating advantage contamination (the dominant cross-segment credit misattribution channel) within each policy update. On a 7B backbone, SLCA-GRPO accelerates convergence and outperforms standard GRPO, ToolPO, and RLTR by +2.53 pp on in-domain evaluation, +1.36 pp on the Berkeley Function-Calling Leaderboard (BFCL), and +9.15 pp on $\tau^2$-Bench under the same training budgets, achieving higher accuracy with reduced tool redundancy and costs.
• 中文摘要 工具调用代理产生异构输出，将结构化工具调用与面向用户的自然语言摘要交错交错。这种输出异构性在标准策略强化学习（RL）中呈现结构性失败模式：像GRPO这样的算法无差别地向所有令牌广播均匀的轨迹级标量优势。因此，摘要生成产生的梯度噪声泄漏到工具决策令牌中，导致跨段信用错误归因和脆弱优化。本研究提出SLCA-GRPO框架，结合分段锁定学分分配（SLCA）。为了实现无昂贵真实API和稳定训练的可扩展探索，我们首先构建了模式引导LLM模拟器（SGLS）作为基础训练基础设施。基于此，SLCA在结构段层面将优势估计解耦于单一部署组内，无需从中间状态额外推送。在层级奖励（Hierarchical Rewards，HierR）支持下，SLCA将执行优势路由到工具代币，优先优势分配给摘要代币，消除每次政策更新中的优势污染（主导的跨段信用错误归属通道）。在7B骨干网上，SLCA-GRPO加速收敛，在域内评估中比标准GRPO、ToolPO和RLTR高出+2.53 pp，伯克利函数调用排行榜（BFCL）+1.36pp，$\tau^2$-Bench +9.15 pp，在相同培训预算下实现更高准确性，减少工具冗余和成本。

A Human-Like Pedestrian Model for Automated Driving Simulations

一种类人步行模型用于自动驾驶模拟

• Authors: Ruofeng Wang, Patrick Ebel, Philipp Wintersberger, Antti Oulasvirta
• Subjects: Subjects: Human-Computer Interaction (cs.HC)
• Arxiv link: https://arxiv.org/abs/2609.29175
• Pdf link: https://arxiv.org/pdf/2609.29175
• Abstract Automated vehicles must be able to interact with pedestrians safely and efficiently across diverse traffic situations. Although driving simulators offer a scalable testbed for learning such capabilities, existing theory-inspired pedestrian models are narrow in scope and limited to go/no-go crossing decisions in single-lane settings. While data-driven approaches can predict pedestrian behavior in complex situations, they lack sufficient observations in rare, safety-critical scenarios. Here, we propose an approach to training pedestrian models in simulators so that learned policies generate demonstrably human-like behavior in realistic, complex traffic scenarios, including multiple lanes, heavy traffic, and dangerous driving styles. Our technical contribution is a novel definition of pedestrian-vehicle interaction as a partially observable Markov decision process (POMDP) with theory-grounded perceptual, cognitive, and motor constraints. It accounts for the highly adaptive nature of human behavior in traffic and simulates how people adjust their responses according to perceived danger, time pressure, and the complexity of the situation. When trained via deep reinforcement learning (RL) with domain randomization in a simulator, the model reproduces the broadest range of empirical findings shown so far on human crossing behavior, including gap acceptance, yielding acceptance, hesitation, and evasive speed adjustment. We show that learned policies transfer to unseen traffic environments, and can be further adapted to local traffic norms with finetuning. Together, these results establish a blueprint for simulator-ready pedestrian models that can support the development and evaluation of automated driving systems.
• 中文摘要 自动驾驶车辆必须能够在各种交通情境下安全高效地与行人互动。尽管驾驶模拟器为学习此类能力提供了可扩展的测试平台，但现有理论启发的行人模型范围较窄，仅限于单车道环境中的通行/不通行决策。虽然数据驱动方法能在复杂情境下预测行人行为，但在罕见且安全关键的场景中缺乏足够的观测。本研究提出一种在模拟器中训练行人模型的方法，使学习的政策在多车道、繁忙交通和危险驾驶风格等真实复杂交通场景中产生明显类人行为。我们的技术贡献是将行人与车辆交互定义为部分可观察的马尔可夫决策过程（POMDP），具有理论基础的感知、认知和运动约束。它考虑到了人类在交通中行为的高度适应性，并模拟人们如何根据感知的危险、时间压力和情境复杂性调整反应。通过模拟器中的深度强化学习（RL）和领域随机化训练时，该模型重现了迄今为止关于人类横行行为最广泛的实证发现，包括间隙接受度、让步接受度、犹豫和避让速度调整。我们表明，已学到的政策会转移到未见的交通环境中，并通过微调进一步适应本地交通规范。综合来看，这些结果为模拟器准备好的行人模型奠定了蓝图，支持自动驾驶系统的开发和评估。

Right Choice of Classification Algorithms Based on Reinforcement Learning for Prediction of Non-Alcoholic Fatty Liver

基于强化学习的分类算法正确选择，用于预测非酒精性脂肪肝

• Authors: Hasan Samadbin, Arman Daliri
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.29181
• Pdf link: https://arxiv.org/pdf/2609.29181
• Abstract There are many complex issues in the world of artificial intelligence. Some of these problems are solved using other artificial intelligence methods, which are called artificial intelligence for artificial intelligence. Finding an appropriate classifier algorithm is a time-consuming task. For this reason, an algorithm that can automatically learn the choice of classification algorithms is very important. Classification algorithms are useful in predicting various diseases. Also, Primary Biliary Cirrhosis is one of the most well-known diseases that have been predicted by classification algorithms. This research's most significant achievement and novelty is the automatic increase in learning through a scoring method of reinforcement learning is called square learning (SL). In this research, an algorithm is presented that learns to automatically select the appropriate classification algorithm to predict Primary Biliary Cirrhosis. In this article, with inspiration from four evaluation metrics in classification algorithms, a new reinforcement learning method by the name of Fourth Degree Learning has been presented. In this research, we increased the performance of the classification algorithms used in this method from 63% of accuracy and achieved 98% accuracy.
• 中文摘要 人工智能领域存在许多复杂的问题。其中一些问题通过其他人工智能方法解决，这些方法被称为人工智能的人工智能。找到合适的分类算法是一项耗时的任务。因此，能够自动学习分类算法选择的算法非常重要。分类算法在预测各种疾病方面非常有用。此外，原发性胆管肝硬化是分类算法预测的最著名疾病之一。这项研究最重要的成就和新颖之处是通过一种称为方格学习（SL）的评分方法自动增加学习。本研究提出了一种算法，能够自动选择合适的分类算法来预测原发性胆管肝硬化。本文借鉴分类算法中的四个评估指标，提出了一种名为“第四度学习”的新强化学习方法。在这项研究中，我们将该方法中使用的分类算法的性能准确率从63%提升，达到了98%的准确率。

EAGER: Enhancing Generative Event Extraction via Reinforcement Learning with Verifiable Rewards

EAGER：通过可验证奖励的强化学习提升生成事件提取

• Authors: Omar Adjali, Siting Liang, Omair Shahzad Bhatti, Daniel Sonntag
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.29230
• Pdf link: https://arxiv.org/pdf/2609.29230
• Abstract End-to-end event extraction remains challenging for large language models as it requires simultaneous identification of event triggers, classification of event types, and extraction of schema-grounded argument spans. We present EAGER, a reinforcement learning framework for generative event extraction that combines fine-grained verifiable rewards with Schema-Contrastive Advantage Estimation to alleviate advantage collapse under sparse binary rewards. Our reward design explicitly targets structural validity, extraction accuracy, groundedness, coverage, over-generation, and span precision. Experiments across seven benchmark datasets show that EAGER consistently outperforms prompting, supervised fine-tuning, and prior reinforcement learning baselines, achieving a substantial improvement over the strongest prior method. Results demonstrate that task-aligned verifiable rewards and contrastive advantage estimation substantially improve structured extraction.
• 中文摘要 对于大型语言模型来说，端到端事件提取依然具有挑战性，因为它需要同时识别事件触发器、分类事件类型以及提取基于模式的参数范围。我们提出了EAGER，一种用于生成事件提取的强化学习框架，结合了细粒度的可验证奖励与模式对比优势估计，以缓解稀疏二元奖励下的优势崩溃。我们的奖励设计明确针对结构性效度、提取准确性、扎根性、覆盖率、过生成和跨度精度。在七个基准数据集上的实验表明，EAGER始终优于提示、监督微调和先验强化学习基线，显著优于最强的先验方法。结果表明，任务对齐的可验证奖励和对比优势估计显著提升了结构化提取。

Safe Learning-Based Adaptive Augmentation Control for Fixed-Wing UAV under Uncertainty

固定翼无人机在不确定性下的安全基于学习的自适应增强控制

• Authors: Leon Raguse, Lennart Kracke, Mayank Shekhar Jha, Johannes Autenrieb, Mark Spiller
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.29319
• Pdf link: https://arxiv.org/pdf/2609.29319
• Abstract This paper presents a learning-based adaptive augmentation control concept inspired by the adaptation mechanisms of conventional adaptive control, while not being restricted to their specific parametric adaptation structures. In contrast to augmenting a reinforcement learning (RL) baseline controller with classical adaptive control to account for the simulation-to-reality gap, the proposed approach uses RL-based adaptive augmentation to address the limitations of conventional adaptive control. Domain randomization combined with observation stacking is employed to train the RL-based augmentation to compensate for matched uncertainties in a fixed-wing aircraft system. To ensure constraint satisfaction during operation, a safety filter is incorporated into the control architecture. Based on the concept of pseudo control hedging (PCH), we propose a modified reference model that avoids undesirable interactions between the RL-based augmentation and the safety filter. To reduce the conservatism of the safety filter, we additionally incorporate a disturbance observer. The proposed approach is evaluated on a fixed-wing aircraft model subject to uncertainties.
• 中文摘要 本文提出了一种基于学习的自适应增强控制概念，灵感来自传统自适应控制的适应机制，同时不受其特定的参数适应结构限制。与用经典自适应控制增强强化学习（RL）基线控制器来弥补模拟与现实差距不同，本方法采用基于强化学习的自适应增强来解决传统自适应控制的局限性。采用域随机化结合观测堆栈，训练基于强化学习的增强以补偿固定翼飞机系统中匹配的不确定性。为确保操作过程中约束满足，安全过滤器被纳入控制架构。基于伪控制对冲（PCH）概念，我们提出了一个修改后的参考模型，以避免基于强化学习的增强与安全过滤器之间不良交互。为了降低安全滤波器的保守性，我们还加入了一个扰动观测器。拟议方法在固定翼飞机模型上进行评估，且存在不确定性。

Temperament Engineering: Designing Strategic Behavioural Diversity in Robot Swarms

气质工程：设计机器人群体中的战略行为多样性

• Authors: Edmund R. Hunt
• Subjects: Subjects: Robotics (cs.RO); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2609.29423
• Pdf link: https://arxiv.org/pdf/2609.29423
• Abstract No two robots are truly identical: calibration, battery state, sensor drift and wear give every swarm a distribution of behaviour rather than a single point, usually treated as an imperfection to be minimised. In animal collectives the reverse holds: consistent individual differences in behaviour ('temperament') are shaped by natural selection and often decisive for group performance. This perspective proposes 'temperament engineering', a bio-inspired framework that treats the swarm's distribution of temperaments, rather than the individual controller, as the design object. It borrows five evolutionarily validated axes of animal temperament (shyness-boldness, exploration-avoidance, activity, aggressiveness and sociability) as a design vocabulary, rendering each as a continuous control parameter $\tau \in [0,1]$ above the controller, realisable as a module threshold, a policy-conditioning vector in multi-agent reinforcement learning, or a constraint on a foundation-model planner. A three-phase workflow maps mission success criteria onto relevant axes, plans the shape of the $\tau$ distribution, and tunes reaction norms governing how temperament responds to environmental cues. The payoff is greatest under decentralisation: where a central planner can reassign behaviour online, a temperament distribution is a planner output, but in a swarm without global knowledge it must be an offline, anticipatory design input. Behavioural and platform heterogeneity are thereby co-design variables, and I sketch tentative robot-native axes (self-model plasticity, forcefulness, initiative and expressiveness) arising from features robots have and animals do not. Engineered heterogeneity has been shown to outperform homogeneous swarms in tasks such as aggregation and exploration; establishing when, and how much, heterogeneity repays its cost is the work the field can now take forward.
• 中文摘要 没有两个机器人是完全相同的：校准、电池状态、传感器漂移和磨损赋予每个群体行为分布，而非单一点，通常被视为需要最小化的缺陷。在动物集体中则相反：行为的一致性差异（“气质”）由自然选择塑造，且常常决定群体表现。这一观点提出了“气质工程”，一种仿生框架，将群体的气质分布而非个体控制者视为设计对象。它借用了五个经过进化验证的动物性格轴（害羞-大胆、探索-回避、活动性、攻击性和社交性）作为设计词汇，将每个轴都作为控制器之上连续的控制参数$\tau \in [0,1]$，可作为模块阈值实现，作为多智能体强化学习中的策略条件向量，或作为基础模型规划器的约束。三阶段工作流程将任务成功标准映射到相关轴上，规划$\tau$分布的形状，并调整控制气质如何响应环境线索的反应规范。在去中心化下回报最大：中央规划者可以在线重新分配行为，气质分布是计划者的输出，但在没有全局知识的群体中，它必须是离线的前瞻性设计输入。行为和平台异质性因此成为共同设计变量，我勾勒出机器人原生的初步轴线（自建模可塑性、力量性、主动性和表达力），这些轴线源自机器人具备的特征而动物则无。工程异质性已被证明在聚合和探索等任务中优于同质群体;确定异质性何时以及付出多少代价，是该领域当前可以继续推进的工作。

Coupled State-Space Modelling, Control, and Policy Distillation for Hybrid Rigid-Pneumatic Manipulators

混合刚性-气动机械臂的耦合状态空间建模、控制与策略蒸馏

• Authors: Alan Royce Gabriel Samuel, Pulkit Verma
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.29424
• Pdf link: https://arxiv.org/pdf/2609.29424
• Abstract Hybrid manipulators combine motorized rigid joints with pressure-actuated origami segments. Published arms of this kind are controlled with decoupled per-DOF loops, and the cost of this approximation has not been quantified, because the coupled model needed to measure it has not been built. This paper derives such a model for a chain of $N$ alternating revolute joints and Kresling origami segments, including pneumatic chamber dynamics and crease hysteresis. Using the model, we measure the coupling directly and show that its strength varies joint by joint, and that decoupled control loses precisely on the strongly coupled joints while remaining competitive on the one nearly decoupled joint. Coupled model-based controllers track $2.5\times$ tighter than a decoupled PID baseline at lower torque. However, the model predictive controller (MPC) is too slow for real time, and model-free reinforcement learning stalls far below acceptable success rates on a strict settling metric. We therefore distill the MPC into a small neural policy with behavior cloning and DAgger. The distilled policy settles 93-94$\%$ of goals with zero collisions, within a few points of its teacher, and runs inside the 5 ms control step where the MPC does not. Where the teacher itself fails, we trace the failure to a limit cycle with the bellows' lightly damped mode, and we remove it by selecting goal postures holdable at low pressure.
• 中文摘要 混合机械臂将电动刚性关节与压力驱动折纸段结合。此类已发表的机械臂通过解耦的每自由度环控制，由于测量所需的耦合模型尚未建成，这一近似成本尚未量化。本文为$N$交替旋转接头和Kresling折纸段链推导了这样一个模型，包括气动腔动力学和折痕滞后。利用该模型，我们直接测量耦合强度，证明其强度在每个节点之间变化，且解耦控制在强耦合节点上精确失效，而在几乎脱钩的单个节点上保持竞争力。基于模型的耦合控制器在较低扭矩下比解耦PID基线紧密跟踪2.5倍美元。然而，模型预测控制器（MPC）对实时传输来说太慢，且无模型强化学习在严格的稳定指标上远低于可接受的成功率。因此，我们将MPC提炼为一个带有行为克隆和DAgger的小型神经策略。提炼后的策略在与教师之间稳定了93-94%%的目标，且在距离其教师的几个点内，并且运行在5毫秒的控制步骤内，而MPC则没有。当教师自身失败时，我们追踪失败原因为带有风箱轻阻尼模式的极限循环，并通过选择可维持低压的目标姿态来消除它。

Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

问问Jev：作为AI对齐失败零射值检测器的校准决策强化学习

• Authors: Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, Leo Yu Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR)
• Arxiv link: https://arxiv.org/abs/2609.29429
• Pdf link: https://arxiv.org/pdf/2609.29429
• Abstract Detectors of alignment failures screen deployed language models and score alignment benchmarks. Most are generative judges that spend a decoding pass on every criterion, and classifiers that read token probabilities, such as Llama Guard, still score one fixed label per call. Jev, a model trained with reinforcement learning for calibrated decisions (RLCD), answers many typed questions about one input with calibrated probabilities in a single call. Whether it detects alignment failures has not been measured. We present RLCDAlignBench, which benchmarks Jev on ten alignment failures: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy violation, social bias, reward hacking, concealing uncertainty, and power seeking. It spans 44 benchmarks and five target models, labelled by each benchmark's scorer and, on two, by humans. Many of these failures are relational, defined against a reference, such as the user's belief or an injected instruction, that the response alone does not reveal. Our key idea is therefore to vary what Jev is asked separately from what it sees: the question's wording and answer type on one side, the fields of the input on the other. A single generic question reaches a median AUROC of 0.886 zero-shot and beats supervised baselines on most benchmarks. Question wording matters little, while context matters more, mostly through fields that encode the label. Jev matches the reference scorer's agreement with human labels, surfaces label defects in existing benchmarks, and costs 63x less than LLM-judge scorers. Code and data: this https URL.
• 中文摘要 对齐失败的检测器会筛选已部署的语言模型并对比对基准进行评分。大多数是生成式裁判，对每个标准都进行解码，而读取令牌概率的分类器（如Llama Guard）仍然每次通话得分一个固定标签。Jev是一个通过校准决策强化学习（RLCD）训练的模型，在一次通话中以校准概率回答了许多关于一个输入的打字问题。是否检测到对齐失败尚未被测量。我们介绍RLCDAlignBench，对Jev的十项对齐失败进行基准测试：谄媚、越狱、欺骗、提示注入、幻觉、隐私侵犯、社会偏见、奖励黑客、隐藏不确定性和权力追求。它涵盖44个基准测试和5个目标模型，每个基准的评分器标注，其中两个模型由人类标记。许多失败是关系性的，基于引用定义，比如用户的信念或注入指令，而仅由回答本身无法揭示。因此，我们的核心思路是将Jev被问的问题与其实际看到的内容分开调整：一侧是问题的措辞和答案类型，另一侧是输入字段。一个通用问题的中位AUROC为0.886零样本，并且在大多数基准测试中优于监督基准。问题措辞影响不大，而上下文更为重要，主要通过编码标签的字段。Jev与参考评分者与人工标签匹配，标注现有基准测试中的缺陷，成本比LLM评分器低63倍。代码与数据：此https URL。

IterSynth: Rethinking Deep Search Agents via Role-Decoupled Iterative Synthesis

IterSynth：通过角色解耦迭代合成重新思考深度搜索代理

• Authors: Xingyu Wu, Yuchen Yan, Zhengxi Lu, Siqi Chen, Xin ZHANG, Aiting Liu, Chao Deng, Jie Liu, Jin Ma, Jian Shao, Jun Xiao, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.29444
• Pdf link: https://arxiv.org/pdf/2609.29444
• Abstract Deep search requires LLM agents to decompose complex queries, search for evidence, and synthesize grounded answers, yet existing ReAct-style agents suffer from two limitations: role coupling, where one policy must handle planning, evidence use, and synthesis; and context accumulation, where growing search histories introduce noise and obscure useful information. To address these issues, we propose IterSynth, a role-decoupled and summary-based paradigm that alternates between a Planner for identifying information needs and a Synthesizer for integrating evidence into an evolving summary state. This design separates planning from synthesis while using the summary as the persistent state of search, reducing both capability coupling and context noise. To train IterSynth effectively, we further introduce Role-Decoupled Policy Optimization (RDPO) for reinforcement learning, which combines terminal outcome rewards with turn-level rubric evaluations and computes role-specific advantages for more precise credit assignment. Experiments on five long-horizon deep-search benchmarks such as BrowseComp and Xbench-DS show that IterSynth-8B achieves an average score of 50.7, surpassing the strongest prior $\leq$8B agent by +4.2\%. Moreover, IterSynth serves as a model-agnostic prompting paradigm, delivering substantial zero-shot gains over ReAct and similar prompting paradigms on frontier proprietary models.
• 中文摘要 深度搜索要求LLM代理拆解复杂查询、寻找证据并综合有根据的答案，但现有的ReAction风格代理存在两个局限性：角色耦合，即一个策略必须负责规划、证据使用和综合;以及上下文积累，随着搜索历史增长，会产生噪声和模糊有用信息。为解决这些问题，我们提出了IterSynth，一种角色解耦、基于摘要的范式，在规划器用于识别信息需求和合成器之间交替使用，用于将证据整合进演变的摘要状态。该设计将规划与综合分离，同时以摘要作为持续的搜索状态，减少了能力耦合和上下文噪声。为有效训练IterSynth，我们进一步引入了角色解耦策略优化（RDPO）用于强化学习，该技术结合终端结果奖励与回合级评分标准评估，并计算角色特定优势以实现更精准的学分分配。在五个长期深度搜索基准测试如BrowseComp和Xbench-DS上的实验显示，IterSynth-8B的平均得分为50.7，比之前最强的$\leq$8B代理高出+4.2%。此外，IterSynth作为一个模型无关的提示范式，在零样本中优于ReAction及类似前沿专有模型的提示范式。

AdaPilot: Towards Scene-Adaptive Policy Learning for Cross-Generator Text-to-Image Quality Optimization

AdaPilot：迈向场景自适应策略学习，实现跨生成器文本转图像质量优化

• Authors: Wenjin Liu, Fayuan Ke, Yue Lu, Zhe Cui, Anh Tuan Luu, Haoran Luo
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2609.29517
• Pdf link: https://arxiv.org/pdf/2609.29517
• Abstract Existing methods for improving text-to-image generation quality have progressed from generator fine-tuning and prompt optimization to reinforcement learning with multi-turn visual feedback. However, existing strategies are deeply coupled with specific generators and tasks, and the learned capabilities are difficult to generalize into a universal quality optimization policy. Therefore, we propose AdaPilot, which learns a scene-adaptive, cross-generator transferable quality optimization policy by formulating multi-turn image generation as a Markov Decision Process (MDP) and optimizing it via end-to-end reinforcement learning. Specifically, AdaPilot decouples the policy from generator internals to enable cross-generator transfer, introduces scene-aware rewards that adaptively align quality assessment dimensions with task semantics, and employs process-level rewards to model the evolution trajectory of image quality. Experimental results show AdaPilot outperforms baselines in generation quality and generalization. Separate cross-generator evaluations further show that a single policy transfers zero-shot to unseen generators while maintaining positive average gains across all evaluated generators. Our project is available at this https URL.
• 中文摘要 现有提升文本到图像生成质量的方法已从生成器微调和提示优化，发展到多回合视觉反馈的强化学习。然而，现有策略与特定生成器和任务深度耦合，所学能力难以推广为通用质量优化策略。因此，我们提出了AdaPilot，通过将多回合图像生成表述为马尔可夫决策过程（MDP），并通过端到端强化学习优化，学习场景自适应、跨生成器可转移的质量优化策略。具体来说，AdaPilot将策略与生成器内部解耦，实现跨生成器转移，引入场景感知奖励，使质量评估维度与任务语义自适应对齐，并利用过程级奖励模拟图像质量的演变轨迹。实验结果显示，AdaPilot在生成质量和泛化方面优于基线。不同的跨生成器评估进一步表明，单一策略将零样本传输到看不见的生成器，同时保持所有评估生成器的平均收益为正。我们的项目可在此 https URL 访问。

CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning

CataOPD：用于大型语言模型推理的催化策略上提纯

• Authors: Wenjin Liu, Chenxi Wang, Jiapu Wang, Zhe Cui, Anh Tuan Luu, Haoran Luo
• Subjects: Subjects: Machine Learning (cs.LG); Computational Engineering, Finance, and Science (cs.CE)
• Arxiv link: https://arxiv.org/abs/2609.29518
• Pdf link: https://arxiv.org/pdf/2609.29518
• Abstract Reinforcement learning (RL) and on-policy distillation (OPD) are two representative paradigms for improving large language model reasoning. However, when no correct trajectory is sampled, RL lacks a positive correctness signal, while OPD remains constrained by the reasoning trajectories reachable under the student's on-policy distribution. Therefore, we propose CataOPD, where the teacher acts as a catalyst rather than a target, expanding reachability while internalizing verified student-produced trajectories into a catalyst-free policy. Self-Rescue Routing uses empirically all-failed groups as routing signals rather than teacher-intervention triggers, first seeking correct trajectories through additional on-policy self-sampling. For problems unresolved after self-rescue, Catalytic-Guided Self-Resolution uses catalytic guidance to elicit a verified student-produced trajectory in the guided student distribution. Barrier-Weighted Internalization weights tokens by guided-to-unguided log-probability gaps, focusing updates on decisive tokens difficult without guidance. Experimental results show that CataOPD outperforms current baselines, extends independent student reasoning to still-unrecovered problems, and improves out-of-distribution generalization under catalyst-free inference. Our project is available at this https URL.
• 中文摘要 强化学习（RL）和策略提纯（OPD）是提升大型语言模型推理的两种代表范式。然而，当没有正确轨迹被采样时，强化学习缺乏正向正确信号，而OPD仍受制于学生在政策分布下可达的推理轨迹。因此，我们提出CataOPD，教师作为催化剂而非目标，扩大可达性，同时将验证学生产生的轨迹内化为无催化剂的策略。自我救援路由使用经验上所有失败的组作为路由信号，而非教师干预触发器，首先通过额外的政策自采样寻求正确轨迹。对于自我救援后未解决的问题，催化引导自解利用催化引导引出经验证的学生生成轨迹。障碍加权内化通过引导到无引导的对数概率差距加权，聚焦于无指导难以实现的决定性标记更新。实验结果显示，CataOPD优于当前基线，将学生独立推理扩展到尚未恢复的问题，并在无催化剂推断下提升分布外泛化能力。我们的项目访问网址为 https URL。

Certified Predictive Value-of-Advice Gating for Cost-Aware Language-Model Guidance in Reinforcement Learning

认证的预测价值建议门禁，用于强化学习中成本感知的语言模型指导

• Authors: Ibne Farabi Shihab, Md Najmus Swaqeeb, Abu Sa-Adat Mohamed Moon-Im Al Ahsan
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.29548
• Pdf link: https://arxiv.org/pdf/2609.29548
• Abstract Language-model advice can accelerate reinforcement learning, but calls are costly and returned actions may be stale or wrong. We formulate advice acquisition as a response-contingent metareasoning problem: before querying, the controller predicts possible parsed responses, evaluates the decision and declared continuation that would follow each response, and queries only when a lower confidence bound on predictive value exceeds the priced cost. Execution is governed separately by an action-specific certificate. Under explicit assumptions, certified advice is near-optimal, a wrapped learner inherits fallback regret only under intervention stability, and conservative allocation loses at most the declared query-value estimation error relative to a myopic oracle. On BabyAI, a proxy-calibrated controller with Qwen2.5-1.5B and 7B advisors improves GoToObj return over no querying by 0.029 +/- 0.016 and 0.030 +/- 0.015 across 20 seeds while reducing calls by more than 97% relative to always-query. GoToLocal is a null result. Exactly matched-call tests show an advantage over random placement only for the 1.5B advisor and no advantage over an equal-budget early schedule. Mondrian calibration improves decision-relevant empirical coverage from 0.47 to 0.85, still below the 0.90 target, while the formally covered radius is vacuous. The demonstrated benefit is therefore robust sparse advice volume on a useful task, not a proven per-state placement advantage.
• 中文摘要 语言模型建议可以加速强化学习，但调用成本高昂，且返回的动作可能陈旧或错误。我们将建议获取表述为响应依赖的元推理问题：查询前，控制器预测可能的解析响应，评估每个响应后可能的决策和声明的续写，只有当预测值置信度下限超过定价成本时才查询。执行由动作专用证书单独管理。在显式假设下，认证建议接近最优，包裹学习者仅在干预稳定性下继承备用遗憾，保守分配相对于近视预言机最多损失声明的查询值估计误差。在BabyAI上，代理校准控制器配合Qwen2.5-1.5B和7B顾问，在20个种子中，GoToObj在无查询时的回报提升了0.029 +/- 0.016和0.030 +/- 0.015，同时相较于始终查询减少了97%以上的呼叫。GoToLocal是一个无结果。精确匹配呼叫测试显示，15亿顾问相比随机投放有优势，而对预算相同的早期安排则无优势。Mondrian校准将决策相关实证覆盖率从0.47提升至0.85，但仍低于0.90目标，而正式覆盖的半径则为空。因此，所展示的好处是对有用任务提供了稳健且稀疏的建议量，而非经过验证的按州排名优势。

iCoder-27B: Recursive AI-Led Development of Frontier Industrial Coding Model

iCoder-27B：递归AI驱动的前沿工业编码模型开发

• Authors: Cheng Yang, Jiayang Lyu, Shangyuan Liu, Guibin Zhang, Jiong Lin, Xinlei Yu, Junchi Yan, Shuicheng Yan, Weinan E, Linfeng Zhang, Linfeng Zhang, Qibing Ren
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2609.29626
• Pdf link: https://arxiv.org/pdf/2609.29626
• Abstract Recursive AI, the prospect of AI taking an increasingly complete role in building and improving AI, is a crown jewel of AI for AI. Although recursive self-development has become practical for small models, bounded tasks, and fixed time budgets, a more consequential realization of this ambition, i.e., developing a release-ready, frontier-competitive model, remains far more challenging. In this work, we ask how little human involvement is sufficient for an agent to develop a frontier model. We concentrate human input into a high-density, low-frequency interface: experts encode objectives, stage scaffolds, permission boundaries, and operating procedures as reusable research skills, while the agent instantiates these priors, selects experiments, diagnoses outcomes, and revises the training strategy. In the challenging domain of industrial coding, the agent evolves data and coordinates SFT, on-policy self-distillation, and reinforcement learning with verifiable rewards, ultimately producing iCoder, a 27B model for RTL design and GPU kernel optimization. Across seven benchmarks, iCoder leads RTLLM, outperforming GPT-5.5 and Claude-Opus-4.8; ranks second on CVDP and KernelBench L2, exceeding GPT-5.5 by 16 points; and ties Claude-Opus-4.8 for the best TritonBench result. Exploratory case studies further show iCoder's competitive iterative RTL and GPU-kernel optimization with substantially fewer tokens. These results chart an engineering path toward recursive self-improvement, in which humans distill the principles of model building, agents operationalize them through evidence-driven experimentation, and each generation of AI becomes a more capable architect of the next.
• 中文摘要 递归人工智能，即人工智能在构建和改进人工智能中日益全面发挥作用的前景，是人工智能领域的瑰宝。尽管递归自我开发已在小模型、有限任务和固定时间预算中变得可行，但实现这一目标的更实质性，即开发一个准备好发布、具前沿竞争力的模型，仍更具挑战性。在本研究中，我们探讨智能体开发前沿模型所需的人类参与程度。我们将人类输入集中到高密度、低频接口中：专家将目标、舞台支架、权限边界和操作程序编码为可复用的研究技能，而智能体则实现这些先验，选择实验，诊断结果，并修订训练策略。在工业编码这一具有挑战性的领域，该智能体演化数据并协调SFT、策略自蒸馏和强化学习，并以可验证的奖励实现，最终诞生了iCoder，一个用于RTL设计和GPU内核优化的27B模型。在七个基准测试中，iCoder领先RTLLM，表现优于GPT-5.5和Claude-Opus-4.8;在CVDP和KernelBench L2上排名第二，领先GPT-5.5 16分;并与Claude-Opus-4.8并列TritonBench最佳成绩。探索性案例进一步展示了iCoder在显著减少令牌数量下的竞争性迭代RTL和GPU内核优化。这些结果为人类提炼模型构建原理提炼模型构建原理，智能体通过证据驱动的实验将其操作化，每一代人工智能都成为下一代更强大的架构师。

To Think or Not to Think: Allocating Reasoning Where It Helps

思考还是不思考：在有帮助的地方分配推理

• Authors: Zhengdong He, Yunfan Zhou, Jianguo Yao, Haibing Guan, Xijun Li
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.29664
• Pdf link: https://arxiv.org/pdf/2609.29664
• Abstract Reinforcement learning (RL) has proven effective in enhancing the reasoning performance of large language models (LLMs), particularly in complex mathematical and programming tasks. However, this capability comes with systematic \textit{length misallocation}, in which models devote excessive reasoning to simple questions while terminating prematurely on harder ones, degrading inference efficiency with negligible accuracy improvement. Many length-adaptive methods mitigate this issue by allocating token budgets according to question difficulty, under the implicit assumption that harder questions benefit monotonically from extended reasoning. In contrast, we find that the effect of reasoning length on accuracy is concentrated on \textit{partially solvable} questions. Our further analysis reveals that explicit length rewards can produce unintended training dynamics. Motivated by these findings, we propose \textbf{CARE}---\textbf{C}ontrastive \textbf{A}ccuracy \textbf{R}eward \textbf{E}stimation---which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost. Experiments across multiple reasoning benchmarks demonstrate that our method improves Pass@1 by up to (4\%) while simultaneously reducing reasoning length by (37\%), achieving higher token efficiency. Code will be available upon the acceptance of this paper.
• 中文摘要 强化学习（RL）已被证明能有效提升大型语言模型（LLMs）的推理性能，尤其是在复杂的数学和编程任务中。然而，这种能力伴随着系统性的 \textit{长度错分配}，即模型对简单问题投入过多推理，而在较难的问题上过早终止，降低推理效率且精度提升微乎其微。许多长度自适应方法通过根据题目难度分配代币预算来缓解这一问题，隐含假设较难的问题从扩展推理中单调受益。相比之下，我们发现推理长度对准确性的影响主要集中在 \textit{部分可解}问题上。我们的进一步分析显示，显式长度奖励可能产生意想不到的训练动态。基于这些发现，我们提出了 \textbf{CARE}---\textbf{C}ontrastive \textbf{A}ccuracy \textbf{R}eward \textbf{E}估计---比较在线抽样回答中每个问题的有益长度调整，并在群体相对策略优化中应用自适应长度奖励，无需额外超参数或推理成本。跨多个推理基准测试的实验表明，我们的方法Pass@1在推理长度减少 \（37\%\） 的同时，将推理长度提升至 \（37\%），从而实现更高的代币效率。本论文接受后，代码将公开。

Learning to Ideate for Scientific Impact

学习如何为科学产生影响而构思

• Authors: Shubham Kale, Aniketh Garikaparthi, Manasi Patwardhan
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.29802
• Pdf link: https://arxiv.org/pdf/2609.29802
• Abstract Scientific ideation is increasingly mediated by large language models, but current ideation systems are usually trained and evaluated on immediately judgeable proxies such as novelty, clarity, and feasibility. This leaves open whether delayed signals of scientific uptake can be used as feedback for steering models toward research directions with higher expected \emph{impact}. We study this question using citation-normalized impact as a noisy but scalable proxy for scholarly uptake. We construct a large-scale dataset from over 100K computer science papers by extracting goal-conditioned idea descriptions and assigning each paper an ordinal, year-normalized citation label. We then train a goal-conditioned reward model to predict citation-impact labels from research goal and idea pairs, and use this reward to align an idea generator through supervised fine-tuning followed by reinforcement learning. To reduce circularity, we evaluate generated ideas with a held-out, reference-grounded protocol that compares model outputs against historical ideas under the same research goal and weights judgments by the reference idea's citation-impact label. Experiments show that our RL-tuned model consistently produces ideas with higher estimated impact than both the base model and supervised fine-tuning baselines. Our findings position scientific impact as a practical, outcome-grounded feedback signal for aligning LLMs in open-ended scientific discovery.
• 中文摘要 科学创意越来越多地由大型语言模型媒介，但当前的创意系统通常基于新颖性、清晰度和可行性等可立即判断的代理指标进行训练和评估。这也留下了未知：科学采纳的延迟信号是否可以作为引导模型朝向更高预期\emph{impact}的研究方向的反馈。我们用引用归一化影响作为学术采纳的一个噪声但可扩展的代理指标来研究这个问题。我们通过提取目标条件性思想描述，并为每篇论文分配一个序数、年份归一化的引用标签，构建了一个大型数据集，涵盖了10万多篇计算机科学论文。然后我们训练一个目标条件奖励模型，预测研究目标和想法对的引文影响标签，并利用这种奖励通过监督微调和强化学习对齐思想生成器。为减少循环性，我们采用一个基于参考的方案评估生成的想法，该协议将模型输出与同一研究目标下的历史想法进行比较，并通过参考思想的引用-影响标签加权判断。实验显示，我们的强化学习调优模型持续产生比基础模型和监督微调基线更高的估计影响力想法。我们的发现将科学影响定位为一种实用且基于结果的反馈信号，用于在开放式科学发现中对齐大型语言模型。

Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agents

Qwen-Planner-Agent：一个面向现实世界移动规划代理的闭环AI框架

• Authors: Tingyu Qu, Weigao Sun, Yuecheng Liu, Yucheng Zhao, Yi Zhu, Yifeng Ding, Qiyi Wang, Sihan Cao, Pengkun Jiao, Hanlei Xie, Xiongwei Wu, Qichao Wang, Haodong Zhang, Jiajun Liu, Yuhao Wang, Yuqing Xie, Junpeng Zhao, Long Chen, Ming Ma, Sihan Yang, Ziwang Zhao, Yanhao Jia, Liangquan Gong, Feida Zhu, Yiran Zhong, Steven Hoi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.29892
• Pdf link: https://arxiv.org/pdf/2609.29892
• Abstract The rapid progression of large language models is extending AI from passive content generation into the active workflows of engineering and scientific discovery. This shift raises a compelling question: can AI be both the object of development and an active participant in building next-generation AI systems? We explore this question by building Qwen-Planner-Agent within a closed-loop AI-for-AI framework for scalable development and iterative improvement. Mobile planning offers a demanding test of this approach: complex, long-horizon tasks challenge agent reliability, while costly real-device interaction limits development scalability. The framework connects data production, model training, and deployment through a shared action-feedback-verification contract. (i) AI for Data builds a human-gated agentic data flywheel in which specialized agents construct tasks, collect interaction trajectories, curate and balance training data, and use training feedback to guide subsequent data generation. (ii) AI for Training combines a supervised planning cold start with hybrid-environment online agentic reinforcement learning, where we introduce Competence-Aware Reward-and-Advantage Engineering (CARE) to reduce reasoning and tool-use costs while preserving task performance. (iii) AI drives model--harness co-evolution through an execution-evidence-driven loop that orchestrates memory, skills, and tools at runtime and feeds structured action feedback and preserved failure traces back into coordinated model and harness adaptation. Qwen-Planner-Agent achieves the best overall performance among all evaluated models and systems on MobilePA-Bench, improving over its base model across tool use, memory, skills, and sub-agent coordination. Further evaluations of our model show improvements across non-mobile agentic benchmarks while largely preserving general capabilities.
• 中文摘要 大型语言模型的快速发展正在将人工智能从被动内容生成扩展到工程和科学发现的主动工作流程。这一转变引发了一个引人注目的问题：人工智能能否既是开发对象，也能成为构建下一代人工智能系统的积极参与者？我们通过在闭环的人工智能对人工智能框架中构建Qwen规划代理，实现可扩展开发和迭代改进，来探讨这一问题。移动规划对这一方法进行了严峻考验：复杂且长期的任务挑战了代理的可靠性，而昂贵的真实设备交互限制了开发的可扩展性。该框架通过共享的动作反馈验证合同连接数据生产、模型训练和部署。（i） AI for Data构建了一个人门控代理数据飞轮，专业代理构建任务、收集交互轨迹、策划和平衡训练数据，并利用训练反馈指导后续数据生成。（ii） AI for Training结合了监督式规划冷启动与混合环境在线智能体强化学习，引入能力感知奖励与优势工程（CARE），以降低推理和工具使用成本，同时保持任务性能。（iii） AI通过执行证据驱动的循环驱动模型——共进化，在运行时协调记忆、技能和工具，并将结构化动作反馈和失败痕迹反馈回协调模型和工具适应。Qwen-Planner-Agent在MobilePA-Bench上所有评估模型和系统中表现最佳，在工具使用、记忆、技能和子代理协调方面均优于基础模型。进一步评估显示，在非移动智能体基准中均有提升，同时基本保持通用能力。

Learning Better Reasoning for Generative Recommendation with Semantic IDs

学习更好地推理语义ID生成式推荐

• Authors: Mengdan Zhu, Yufan Zhao, Sophie Di, Yao Zhao, Tao Di, Yulan Yan, Sridhar Iyer, Liang Zhao
• Subjects: Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.29973
• Pdf link: https://arxiv.org/pdf/2609.29973
• Abstract Generative recommendation reformulates item retrieval as sequence generation, allowing a unified model to directly generate the next item from a user's interaction history. Semantic IDs further make this paradigm effective and scalable by representing each item as discrete codes, enabling knowledge sharing among semantically related items. Recent studies introduce explicit reasoning before Semantic-ID generation, helping models summarize user interests and infer possible preference transitions. However, reasoning is not inherently beneficial: Inaccurate or uninformative reasoning may mislead subsequent item generation and ultimately degrade recommendation performance. This raises a central challenge: how can a recommender select and learn effective reasoning traces and progressively evolve toward better reasoning from its own generations? In this work, we propose Evo-Rec, a three-stage framework for learning better reasoning and further enhancing it through reinforcement learning. First, we align Semantic IDs with their textual and behavioral contexts, enabling the model to understand and generate item identifiers. Second, we sample multiple candidate reasoning traces and retain those that improve the prediction of the ground-truth item, providing a stronger reasoning initialization through supervised fine-tuning. Third, we further optimize the reasoning policy through reinforcement learning with catalog-constrained item generation and ranking-aware recommendation feedback. Experiments on three Amazon Review benchmarks show that Evo-Rec consistently outperforms discriminative, generative, and reasoning-enhanced recommenders across all evaluation metrics. These results demonstrate the effectiveness of our framework in learning better reasoning for SID-based generative recommendation.
• 中文摘要 生成推荐将条目检索重新表述为序列生成，使统一模型能够直接从用户的交互历史中生成下一个条目。语义ID进一步使这一范式有效且可扩展，通过将每个条目表示为离散代码，实现语义相关项之间的知识共享。近期研究在语义ID生成前引入显式推理，帮助模型总结用户兴趣并推断可能的偏好转换。然而，推理本身并非有益：不准确或信息不足的推理可能误导后续条目生成，最终降低推荐表现。这引出了一个核心挑战：推荐者如何选择并学习有效的推理痕迹，并逐步从自身世代中进化为更好的推理？本研究提出了Evo-Rec，一个三阶段框架，用于学习更好的推理并通过强化学习进一步提升推理。首先，我们将语义ID与其文本和行为上下文对齐，使模型能够理解并生成条目标识符。其次，我们采样多个候选推理痕迹，保留那些能提升真实性条目预测的痕迹，通过监督微调提供更强的推理初始化。第三，我们通过强化学习，结合目录约束的条目生成和排名感知的推荐反馈，进一步优化推理策略。在三个亚马逊评论基准测试上的实验显示，Evo-Rec在所有评估指标中始终优于判别型、生成型和推理增强型推荐者。这些结果证明了我们框架在学习基于SID的生成推荐中更优推理的有效性。

Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation

Res-HIL：人主导残余强化学习，用于样本高效灵巧操作

• Authors: Mariia Iavorskaia, Christian Dietz, Sebastian Albrecht, Majid Khadiv
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.30023
• Pdf link: https://arxiv.org/pdf/2609.30023
• Abstract Imitation learning enables robots to acquire manipulation skills from demonstrations, but the resulting policies can fail outside the training data, while collecting more demonstrations requires substantial human effort. Human-in-the-loop reinforcement learning uses corrective feedback during online training, but typically learns the complete task policy rather than refining a pretrained imitation policy. We introduce Res-HIL, a human-in-the-loop residual reinforcement learning framework that learns corrective actions on top of a frozen imitation policy. Each human intervention provides two complementary learning signals: direct supervision of the residual policy and reward shaping of preceding autonomous behavior. Res-HIL combines these signals with zero initialization of the residual policy to stabilize and accelerate online learning. We evaluate Res-HIL on five contact-rich manipulation tasks spanning high-precision and long-horizon behaviors. With only 20 initial demonstrations, Res-HIL outperforms state-of-the-art full-policy human-in-the-loop reinforcement learning and residual fine-tuning without human guidance on every task after ten minutes of online training. Res-HIL improves its pretrained base policies and outperforms imitation policies trained with five times more demonstrations. An ablation study shows that direct residual supervision is critical to performance, while intervention-aware reward shaping substantially improves training efficiency.
• 中文摘要 模仿学习使机器人能够通过演示获得操作技能，但最终策略可能在训练数据之外失效，而收集更多演示则需要大量人力。人机在环中强化学习在在线训练中使用纠正反馈，但通常学习完整的任务策略，而非细化预训练的模仿策略。我们介绍了Res-HIL，一种人在环路中的残余强化学习框架，基于冻结的模仿策略学习纠正动作。每种人类干预提供两个互补的学习信号：对残余策略的直接监督和对前一个自主行为的奖励塑造。Res-HIL将这些信号与零初始化剩余策略结合起来，以稳定并加速在线学习。我们在五个接触丰富操作任务中评估Res-HIL，涵盖高精度和长视野行为。仅有20次初始演示，Res-HIL在十分钟在线培训后，已超越最先进的全策略人类强化学习和残余微调，无需人工指导即可完成每项任务。Res-HIL改进了预训练基础策略，并优于五倍演示训练的模仿策略。一项消融研究表明，直接的残余监督对表现至关重要，而干预感知型奖励塑造则显著提升了训练效率。

TEMA: Evidence-Grounded Temporal Question Answering in Multi-Turn Multi-Audio Dialogs

TEMA：基于证据的时间问答，采用多回合多音频对话

• Authors: Kaidi Yang, Hualei Wang, Zhaohui Wang, Chenxuan Wang, Hong Liu, Xiangdong Wang
• Subjects: Subjects: Sound (cs.SD)
• Arxiv link: https://arxiv.org/abs/2609.30029
• Pdf link: https://arxiv.org/pdf/2609.30029
• Abstract Multi-turn, multi-audio temporal question answering requires models to track target events across follow-up questions, recording switches, and historical references, recovering complete instances and their boundaries for temporal calculation and comparison. We propose TEMA, which connects event perception with evidence-based answering through Route, specifying the audio scope, and Span, describing all relevant intervals as conditional audio captions. We construct TEMA-Dialog with 40,704 dialogs and per-turn evidence and answer supervision, and TEMA-Bench for joint evaluation of evidence and final answers. Training combines temporal grounding initialization, full-dialog supervised fine-tuning, and completeness-first Span-only GRPO. Experiments on Qwen2.5-Omni and AF-Next show improved temporal question answering, particularly event localization and cross-audio comparison. Reinforcement learning applied solely to evidence further improves interval recovery and answer accuracy.
• 中文摘要 多回合、多音频时间问答需要模型跟踪目标事件，涵盖后续问题、记录切换和历史引用，恢复完整实例及其边界以便时间计算和比较。我们提出了TEMA，通过Route（指定音频范围）将事件感知与基于证据的回答连接起来，以及Span（将所有相关区间描述为条件音频字幕）。我们构建了包含40,704条对话及每回合证据和答案监督的TEMA-Dialog，以及用于证据联合评估和最终答案的TEMA-Bench。训练结合了时间基础初始化、全对话监督微调和以完整性为先的仅限Span的GRPO。Qwen2.5-Omni和AF-Next的实验显示出时间问题回答的改进，特别是事件定位和跨音频比较。仅应用于证据的强化学习进一步提升了区间恢复和答案准确性。

SciWalker: Synthesizing Scientific Coding Problems with Operator Graphs and Execution Feedback

SciWalker：综合科学编码问题与算子图和执行反馈

• Authors: Chenxi Li, Wenxuan Zeng, Yun Luo, Fangchen Yu, Peng Ye, Yu Cheng, Jun Zhang
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.30054
• Pdf link: https://arxiv.org/pdf/2609.30054
• Abstract Improving the scientific coding capabilities of large language models (LLMs) requires high-quality training data. However, such data remain scarce because manually authoring realistic problems is costly and time-consuming, while systematically covering diverse scientific domains and algorithmic combinations remains challenging. To address this, we introduce SciWalker, a framework for synthesizing scientific coding problems through operator-chain sampling and execution feedback. The framework combines scientific library interfaces with operation modes to instantiate operators, organizes them into operator graphs, and samples operator chains as computational workflow cues. Guided by these cues, we adopt LLMs to generate scientifically grounded problem statements, reference solutions, and tests, with failed generations iteratively repaired using execution feedback. By combining structured workflow composition with verification and quality review, SciWalker enables scalable task generation while promoting scientific grounding, computational diversity, and executability. Using this framework, we construct 8,178 high-quality problems spanning 5 scientific domains and 32 subdomains. To evaluate their training utility, we conduct reinforcement learning on Qwen3.5-9B using the GSPO algorithm. This training improves SciCode subproblem accuracy by 9.9 percentage points, from 29.3% to 39.2%, with gains across scientific code generation, code repair, and reasoning benchmarks. The code for SciWalker is available at this https URL.
• 中文摘要 提升大型语言模型（LLM）的科学编码能力需要高质量的训练数据。然而，由于手动编写现实问题成本高且耗时，而系统性覆盖多元科学领域和算法组合仍具挑战性，因此此类数据仍然稀缺。为此，我们引入了SciWalker框架，该框架通过运算符链采样和执行反馈综合科学编码问题。该框架结合科学库接口与运算符模式实例化运算符，组织为运算符图，并采样运算符链作为计算工作流线索。在这些线索的指导下，我们采用LLM生成基于科学的问题陈述、参考解和测试，失败的生成通过执行反馈迭代修复。通过结合结构化工作流程组合与验证和质量审核，SciWalker实现了可扩展的任务生成，同时促进科学基础、计算多样性和可执行性。利用该框架，我们构建了涵盖5个科学领域和32个子领域、8,178个高质量问题。为评估其训练效用，我们利用GSPO算法对Qwen3.5-9B进行了强化学习。该训练使SciCode子问题的准确率提升了9.9个百分点，从29.3%提升至39.2%，在科学代码生成、代码修复和推理基准测试方面均有提升。SciWalker的代码可在此 https 网址获取。

Self-Play Pretraining with Zero Data

零数据自玩预训练

• Authors: Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, Yoav Levine
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.30063
• Pdf link: https://arxiv.org/pdf/2609.30063
• Abstract Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement. This would provide an effectively unbounded source of training data, limited by compute rather than human knowledge. We introduce Self-Play Pretraining with Zero Data, an initial proof-of-concept towards realizing this vision. Our procedure casts synthetic data generation as a search over the space of all computable structure, taking inspiration from Solomonoff induction. Starting from random initialization, two models learn in tandem: a generator proposes programs interpreted by a universal Turing machine, generating byte sequences, while a learner autoregressively predicts these byte sequences. The learner is trained with standard cross-entropy, while the generator is trained with reinforcement learning to produce sequences at the frontier of the learner's capabilities, yielding an adaptive curriculum. A universal Turing machine gives us a search space over all computable data-generating processes, imposing little domain-specific structure, and self-play searches over this space for useful training data. We test whether zero-shot performance on natural data improves predictably with self-play compute; this is a clean test of transfer since neither generator nor learner is trained on natural data. Across several natural datasets, zero-shot loss exhibits predictable scaling in compute. The models also exhibit in-context learning, and discover recognizable mathematical sequences during training.
• 中文摘要 语言建模的进步主要源于对越来越多数据的预训练扩展。然而，训练数据仍然很大程度上是为模型精心策划的。一种更通用的预训练方法将使模型能够学习生成对自身改进最有用的数据。这将提供一个有效且无边界的训练数据源，受限于计算而非人类知识。我们引入了零数据的自玩预训练，作为实现这一愿景的初步概念验证。我们的过程将合成数据生成视为对所有可计算结构空间的搜索，灵感来自所罗门诺夫归纳。从随机初始化开始，两个模型协同学习：生成器提出由通用图灵机解释的程序，生成字节序列，学习者则自回归预测这些字节序列。学习者通过标准交叉熵训练，生成器则通过强化学习训练，生成学习者能力前沿的序列，从而形成自适应的课程体系。通用图灵机为我们提供了涵盖所有可计算数据生成过程的搜索空间，几乎不施加特定领域的结构，并在此空间上进行自游戏搜索以获取有用的训练数据。我们测试了自然数据上的零样本性能是否通过自我游戏计算可预测地提升;这是一个对转移的干净测试，因为生成器和学习器都未在自然数据上训练。在多个自然数据集中，零样本丢失在计算过程中表现出可预测的缩放性。模型还表现出上下文学习，并在训练过程中发现可识别的数学序列。

Graph-Based Inference and Topology-Aware Multi-Agent Reinforcement Learning for Large-Scale Railway Network Management

基于图的推理与拓扑感知的多智能体强化学习，适用于大规模铁路网络管理

• Authors: Giacomo Arcieri, Gregory Duthé, Christophe Muller, Konstantinos G. Papakonstantinou, Daniel Straub, Eleni Chatzi
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.30150
• Pdf link: https://arxiv.org/pdf/2609.30150
• Abstract Modern infrastructure asset management constitutes a complex sequential decision-making problem, characterized by long planning horizons and system-level interactions, such as spatial deterioration correlations and economies of scale. While deep reinforcement learning has shown promise in optimizing maintenance policies, scaling to real-world networks remains challenging. Centralized approaches become computationally intractable in large-scale systems, whereas decentralized approaches often fail to capture essential coordination mechanisms. To address these challenges, we propose a graph-based framework that integrates accurate environment modeling with scalable decision support. First, we employ a hierarchical Bayesian model leveraging a Gaussian Process on Graph kernel to infer a realistic, spatially correlated networked environment of railway maintenance planning from real-world data provided by the Swiss Federal Railways. Second, we introduce a topology-aware Multi-Agent Reinforcement Learning (MARL) framework by integrating graph neural networks and graph Transformers to optimize network-level policies. A central contribution of this work is the demonstration of scalability through zero-shot transfer learning: graph-based agents, trained only on small network portions, are successfully deployed in a zero-shot manner on large-scale unseen networks without any retraining. Numerical results indicate that the proposed method significantly outperforms optimized heuristics and standard MARL baselines, reducing computational training time while maintaining superior performance on large-scale networks.
• 中文摘要 现代基础设施资产管理构成了一个复杂的顺序决策问题，特点是规划时间长且系统层面存在空间劣化相关性和规模经济等交互。尽管深度强化学习在优化维护策略方面展现出潜力，但扩展到现实世界网络仍具挑战。集中式方法在大规模系统中计算上难以处理，而去中心化方法往往无法捕捉关键的协调机制。为应对这些挑战，我们提出了一个基于图的框架，将精确的环境建模与可扩展的决策支持相结合。首先，我们采用基于图核的高斯过程，推断出一个现实且空间相关的铁路维护规划网络环境，这些环境由瑞士联邦铁路提供的真实世界数据提供。其次，我们通过集成图神经网络和图变换器，引入了拓扑感知的多智能体强化学习（MARL）框架，以优化网络级策略。本研究的核心贡献是通过零样本转移学习证明了可扩展性：基于图的智能体仅在小范围网络部分训练，能够以零样本方式成功部署在大规模未见网络上，无需任何重新训练。数值结果表明，该方法远超优化后的启发式和标准MARL基线，缩短计算训练时间，同时在大规模网络上保持优异性能。

Search-Aware Reinforcement Learning for Multi-Component Query Understanding in Roblox Game Search

用于 Roblox 游戏搜索中多元查询理解的搜索感知强化学习

• Authors: Nayoung Choi, Shengjian Chen, Xiaokai Wei, Wenzheng Zhang, Daiyao Yi, Rachit Pareek, Vincent Su, Michelle Gong, Jinho D. Choi
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.30177
• Pdf link: https://arxiv.org/pdf/2609.30177
• Abstract Query understanding (QU) plays a critical role in production search systems, translating raw user queries into search execution plans that drive downstream retrieval and ranking. While large language models (LLMs) have enabled QU to be framed as a structured multi-task generation problem (e.g., intent classification, query expansion), optimizing such models to produce search-engine-coupled outputs remains challenging: static, label-based supervision fails to capture how each component actually interacts with the underlying search pipeline to affect downstream performance. We present a search-aware reinforcement learning (RL) framework for QU based on a distill-then-RL paradigm. Teacher-student supervised fine-tuning (SFT) first yields a well-formed, schema-compliant policy initialization. The RL stage then optimizes each QU component with rewards derived from live interaction with the search engine, tailored to that component's operational role, rather than a single reward tied to the final search outcome. Experiments on Roblox search show that this component-specific optimization improves both per-component utility and downstream search quality, raising NDCG@20 by 8.9 points over the SFT policy and by 3.5 points over training with a single end-to-end reward.
• 中文摘要 查询理解（QU）在生产搜索系统中起着关键作用，将原始用户查询转化为驱动下游检索和排名的搜索执行计划。虽然大型语言模型（LLM）使QU能够被框架为结构化的多任务生成问题（如意图分类、查询扩展），但优化此类模型以产生搜索引擎耦合输出仍然具有挑战性：静态的基于标签的监督无法捕捉每个组件实际如何与底层搜索流水线交互，从而影响下游性能。我们基于“蒸馏后强化学习”范式，提出了一个基于搜索感知强化学习（RL）的QU框架。师生监督微调（SFT）首先产生一个规范且符合模式的策略初始化。随后，RL阶段通过基于与搜索引擎实时交互得出的奖励，针对该组件的操作角色进行优化，而非单一奖励与最终搜索结果挂钩。Roblox搜索的实验显示，这种组件专属优化不仅提升了每个组件的效用，也提高了下游搜索质量，NDCG@20比SFT策略提高了8.9个百分点，比单一端到端奖励的训练提高了3.5个百分点。

PoEM: Predicting RL Outcomes from Existing Policies

PoEM：从现有政策预测强化学习结果

• Authors: Kimia Hamidieh, Giannis Daras, Antonio Torralba
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.30226
• Pdf link: https://arxiv.org/pdf/2609.30226
• Abstract Foundation models are post-trained with reinforcement learning (RL) to maximize specific rewards, such as human alignment, correctness, or instruction following. This post-training process is computationally intensive, sometimes unstable, and has to be run from scratch every time the reward model changes or when we want to combine multiple rewards. We hence ask: given a new reward function, is it possible to predict the RL outcomes without actually running RL on it? We answer this in the affirmative by introducing PoEM, a framework to predict the outputs of RL on a new reward function using a set of models already post-trained on other rewards. First, we show that if the new reward function can be written as a linear combination of existing ones, then the new policy in log-space can be written as a linear combination of the existing log-policies. Surprisingly, even in cases where the rewards are not linearly connected, we observe that often log-policies from RL training span an approximately low-rank subspace across rewards. To our benefit, the weighting coefficients for this combination can be estimated using only the reward or basis policy outputs on the samples. We turn these observations into an algorithm that takes post-trained models and a new reward function, and approximates the target RL policy without actually running any additional RL training. We experimentally validate our approach across synthetic and real rewards, spanning both text and image modalities.
• 中文摘要 基础模型通过强化学习（RL）进行后训练，以最大化特定奖励，如人类对齐、正确性或指令跟随。这一后训练过程计算量大，有时不稳定，每次奖励模型变化或需要合并多个奖励时都必须从头开始运行。因此我们提出：给定一个新的奖励函数，是否可以在不实际运行强化学习的情况下预测强化学习的结果？我们通过引入PoEM来回答肯定的这个问题，这是一个利用一组已在其他奖励上后训练过的模型预测强化学习在新奖励函数上的输出的框架。首先，我们证明如果新的奖励函数可以写成现有奖励函数的线性组合，那么在对数空间的新策略也可以写成现有对数策略的线性组合。令人惊讶的是，即使在奖励非线性关联的情况下，我们也观察到，强化学习训练的对数策略往往跨越了奖励的低秩子空间。有利于我们，仅用样本上的奖励或基准策略输出，估计该组合的权重系数。我们将这些观察转化为算法，利用后训练模型和新的奖励函数，近似目标强化学习策略，而无需实际运行额外的强化学习训练。我们实验验证了我们的方法，涵盖合成和真实奖励，涵盖文本和图像模态。

Keyword: diffusion policy

KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization

KeyGen：基于类别级策略泛化的无监督关键点对象中心表示

• Authors: Shuxin Cao, Liquan Wang, Masoud Moghani, Benjamin Joffe, Animesh Garg
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.28818
• Pdf link: https://arxiv.org/pdf/2609.28818
• Abstract Generalization in robotic manipulation requires policies to perform tasks across diverse unseen object instances that vary in shape, size, and pose. However, conventional behavior cloning (BC) methods often overfit to instance-specific geometry and appearance, limiting transfer to novel objects. We introduce KeyGen, a framework that learns canonicalized semantic 3D keypoints from point clouds and uses them as structured object-centric representations for policy learning. A visuomotor diffusion policy conditions on these keypoints together with object-centric geometry to predict full manipulation trajectories, enabling consistent geometric correspondence across object instances. To evaluate category-level generalization, we construct a photorealistic simulation benchmark with three manipulation tasks and a planning-driven data generation pipeline that produces expert trajectories across diverse object instances. Experiments show that KeyGen significantly outperforms prior methods on both seen and unseen objects under pose variation, scales effectively with additional demonstrations per object, maintains robustness to object rescaling, and achieves strong performance in both simulation and real-world manipulation.
• 中文摘要 机器人操作的泛化需要策略在形状、大小和姿态各异的多样未见对象实例中执行任务。然而，传统的行为克隆（BC）方法常常过拟合于实例特定的几何体和外观，限制了对新对象的转移。我们引入了KeyGen框架，该框架从点云中学习规范化的语义三维关键点，并将其作为结构化的以对象为中心的表示进行策略学习。在这些关键点上构建视觉运动扩散策略条件，结合对象为中心几何，预测完整的操作轨迹，实现对象实例间的几何一致性对应。为评估类别级泛化，我们构建了一个逼真的仿真基准，包含三个操作任务和一个规划驱动的数据生成流水线，能够生成跨多样对象实例的专家轨迹。实验表明，KeyGen在姿态变化下的可见和不可见物体上都显著优于以往方法，能够有效扩展每个对象的额外演示，保持对物体缩放的鲁棒性，并在模拟和实际操作中均表现出色。