生成时间: 2026-09-16 21:13:58 (UTC+8); Arxiv 发布时间: 2026-09-16 20:00 EDT (2026-09-17 08:00 UTC+8)

今天共有 26 篇相关文章

Keyword: reinforcement learning

ViCo: Visual-oriented Coding with Self-Reflection for Chart Replication

ViCo：以图表复制为基础的视觉化编码与自我反思

• Authors: Jiaxin Duan, Dian Jiao Shuai Zhao, Jiabing Leng, Yiran Zhang, Feng Huang
• Subjects: Subjects: Computation and Language (cs.CL); Graphics (cs.GR); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.16014
• Pdf link: https://arxiv.org/pdf/2609.16014
• Abstract This paper addresses the challenge of generating high-quality academic charts that match the visual standards of human-authored papers. While existing AI agents can produce well-structured text and code, their generated visualizations often lack the stylistic and semantic fidelity of human designs. Advanced coding agents that employ self-reflection mechanisms exhibit poor visual reasoning and limited reflection following, resulting in sparse reward signals that severely undermine their reinforcement learning (RL). We propose ViCo, a training framework for visual-oriented coding that employs iterative reflections to align generated chart images progressively with the reference. We first introduce a self-supervised warm-up stage, which augments Monte Carlo Tree Search with consistency-based pruning to synthesize high-quality reflection trajectories, ensuring that each coding step strictly follows the outcomes of prior reflections. A multi-step RL algorithm is then developed, using counterfactual baselines to estimate advantage for reflection and action steps within each refinement cycle, thereby addressing the reward sparsity. To enable efficient reward in massive training, we propose an automatic, multifaceted evaluation framework that assesses charts' style, layout, and semantic consistency via a hierarchical heterogeneous layout graph structure. Experiments on three public benchmarks demonstrate that ViCo, trained on an 8B model, achieves performance close to proprietary LLMs with adequate reflection capabilities.
• 中文摘要 本文探讨了生成高质量学术图表、符合人类论文视觉标准的挑战。虽然现有的AI代理能够生成结构良好的文本和代码，但其生成的可视化往往缺乏人类设计的风格和语义忠实度。采用自我反思机制的高级编码代理表现为视觉推理能力差，反思跟随有限，导致奖励信号稀疏，严重削弱其强化学习（RL）。我们提出了ViCo，一种视觉导向编码的训练框架，利用迭代反射逐步对齐生成的图表图像。我们首先引入自监督热身阶段，通过基于一致性的剪枝补充蒙特卡洛树搜索，合成高质量的反射轨迹，确保每个编码步骤严格遵循先前反思的结果。随后开发了多步强化学习算法，利用反事实基线估计每个精炼周期内反思和行动步骤的优势，从而解决奖励稀疏性问题。为实现大规模训练中的高效奖励，我们提出了一个自动、多方面的评估框架，通过层级异构布局图结构评估图表的样式、布局和语义一致性。在三个公开基准测试上的实验表明，基于8B模型训练的ViCo在具备充分反射能力下，性能接近专有大型语言模型。

Managing Action Preconditions in Neuro-Symbolic RL: Three Placement Strategies for Embodied Agents

神经符号强化学习中的行动前置条件管理：具身代理的三种配置策略

• Authors: Norbert Oswald, Fabian Deuser, Thomas Bräunl
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.16056
• Pdf link: https://arxiv.org/pdf/2609.16056
• Abstract Humans carry behaviour knowledge of how to act in familiar situations into every new task rather than relearning it from scratch. There is no reason a Reinforcement Learning (RL) agent shouldn't do the same: known behaviour patterns need not be learned, only applied. Neuro-symbolic RL bridges prior knowledge and RL by injecting symbolic knowledge alongside a learned policy. The point at which this knowledge is integrated is critical: a poor choice can produce, for instance, hallucinated preconditions, which surface as safety and reliability problems in agents acting in changing environments. We formalise this behavioural knowledge as a precondition Bayesian network (BN) over the agent's \emph{structural actions} - the actions whose legality depends on preconditions, such as picking up a key, grasping a block, toggling a door, or dropping an object. The BN restricts when these actions may fire, and we inject it into the RL loop at three placements: (1) a \emph{symbolic verifier}, consulted only at inference, that fires a structural action once its preconditions hold; (2) a \emph{symbolic enforcer}, active during both training and inference, that governs structural-action use throughout learning; and (3) a \emph{symbolic learner}, which folds the knowledge into the network and learns the restriction and use of structural actions itself. To test the three variants we run experiments on two benchmarks with opposite regimes: one built on long, ordered planning chains, the other on continuous manipulation. We compare against strong baselines on solution quality, sample efficiency, and traceability. The payoff is substantial. On MiniGrid, all three placements improve the \emph{solution quality} over the PPO+RND baseline, the symbolic enforcer leading at $98.2\%$ against the baseline's $88.8\%$. On Fetch, $\dots$
• 中文摘要 人类将如何在熟悉情境下行动的行为知识带入每一项新任务，而不是从头重新学习。强化学习（RL）智能体也应该这样做：已知的行为模式无需学习，只需应用即可。神经符号强化学习通过在学习策略的同时注入符号知识，连接了既有知识和强化学习。这种知识整合的关键点至关重要：例如，错误的选择可能导致幻觉的前提条件，这些先决条件在变化环境中表现为智能体的安全和可靠性问题。我们将这种行为知识形式化为对智能体\emph{结构性行为}的先决条件贝叶斯网络（BN）——即合法性依赖于前提条件的行为，如拾起钥匙、抓住方块、开关门或丢弃物品。BN限制这些动作的触发时间，我们将它注入强化学习循环的三个位置：（1）\emph{符号验证器}，仅在推理时被参考，当前提成立时触发结构动作;（2）\emph{符号执行器}，在训练和推理期间均激活，控制整个学习过程中的结构动作使用;以及（3）\emph{符号学习器}，将知识整合进网络，学习结构性动作的限制和使用。为测试这三种变体，我们在两个基准测试中进行了实验，其模式相反：一个基于长有序规划链，另一个基于连续操作。我们与解决方案质量、样本效率和可追溯性的强基线进行比较。回报显著。在MiniGrid中，这三种配置都提升了\emph{解决方案质量}相较于PPO+RND基线，象征性执法者以$98.2\%$领先基线的$88.8\%$。在Fetch中，$\dots$

Towards Scalable RLVR: Multimodal Instruction Following Data Synthesis and Distillation

迈向可扩展RLVR：数据综合与蒸馏后的多模态指令

• Authors: Yirong Zeng, Zhang Sai, Yuxian Wang, Yutai Hou, Yufei Liu, Xiao Ding, Bibo Cai
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.16059
• Pdf link: https://arxiv.org/pdf/2609.16059
• Abstract Multimodal instruction following (MMIF) is crucial for building generalist agents. However, current training paradigms rely heavily on Supervised Fine-Tuning (SFT), which often leads to surface-level pattern matching and degrades general capabilities. While Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising alternative, its scalability in MMIF is severely bottlenecked by the scarcity of high-quality, RL-ready multimodal data. To bridge this gap, we present MIFS (\textbf{M}ultimodal \textbf{I}nstruction \textbf{F}ollowing \textbf{S}ynthesis), a systematic pipeline designed to generate RL-ready multimodal data. Specifically, MIFS introduces a generative constraint protocol to synthesize diverse raw samples, followed by a learnability-aware distillation mechanism that filters data based on RL training dynamics to ensure stable policy optimization. Furthermore, a code-based verifier provides high-precision reward signals for policy learning. The resulting dataset comprises 90k samples across 8 constraint categories and 14 task domains. Empirical evaluations demonstrate that MIFS-trained MLLMs achieve an average improvement of 8.13\% on four MMIF benchmarks and a 3$\times$ faster training convergence compared to using raw data. Crucially, our approach mitigates the generalization trade-offs typical of SFT, preserving core visual capabilities while significantly boosting instruction-following precision.
• 中文摘要 多模态指令跟随（MMIF）对于构建通用代理至关重要。然而，当前的训练范式高度依赖监督式微调（SFT），这常导致表面模式匹配，降低整体能力。虽然带可验证奖励的强化学习（RLVR）提供了有前景的替代方案，但其在MMIF中的可扩展性因高质量且具备RL准备的多模态数据稀缺而严重受限。为弥合这一差距，我们提出了MIFS（\textbf{M}ultimodal \textbf{I}nstruction \textbf{F}ollowing \textbf{S}ynthesis），这是一条系统化的流水线，旨在生成RL准备的多模态数据。具体来说，MIFS引入了生成约束协议，用于合成多样化的原始样本，随后采用可学习性提炼机制，基于RL训练动态过滤数据，以确保策略优化的稳定。此外，基于代码的验证器为策略学习提供高精度的奖励信号。最终数据集包含8个约束类别和14个任务域的9万个样本。实证评估表明，MIFS训练的MLLM在四个MMIF基准测试中平均提升8.13\%，训练收敛速度比原始数据快3$\倍。关键是，我们的方法缓解了SFT典型的泛化权衡，保留核心视觉能力，同时显著提升指令跟踪精度。

World-Action Models for Robot Learning and Control: A Survey

世界行动机器人学习与控制模型综述

• Authors: Zuxing Lu, Hongjia Zhai, Guanzhi Wang, Huajian Zeng, Jiaqi Yang, Jingyu Liu, Lei Cheng, Yuantai Zhang, Yuheng Qiu, Zezhou Cheng, Ivan Laptev, Danfei Xu, Benjamin Riviere, Giuseppe Loianno, Eric Xing, Xingxing Zuo
• Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2609.16074
• Pdf link: https://arxiv.org/pdf/2609.16074
• Abstract Robots operating in open environments act under partial observability, physical constraints, and dynamic task contexts. Beyond mapping observations and language instructions to actions, they must anticipate how candidate actions may affect future states and task-relevant outcomes. Recent advances in world models, video generation, and Vision-Language-Action (VLA) policies have motivated the development of World-Action Models (WAMs), which couple future world prediction with executable action generation. This survey provides a robotics-oriented review of WAMs. We clarify their scope relative to conventional world models, model-based reinforcement learning, action-conditioned video generation, and reactive VLA policies, and organize existing methods through a unified taxonomy covering representations, transition modeling, action interfaces, architectures, training pipelines, data modalities, and scaling strategies. We further review applications of WAMs in manipulation, navigation, and autonomous driving, and we summarize the datasets, benchmarks, metrics, and protocols used to evaluate WAM systems. Finally, we discuss key challenges in action alignment, world-action factorization, spatial and multi-view consistency, long-horizon memory, neural simulation for closed-loop policy learning, and efficient inference. Taken together, this survey aims to provide a concise technical foundation for integrating predictive world modeling with action generation, toward more reliable embodied robot intelligence. Project page: this https URL.
• 中文摘要 在开放环境中运行的机器人在部分可观察性、物理约束和动态任务上下文下行动。除了将观察和语言指令映射到动作外，它们还必须预测候选动作如何影响未来状态和任务相关结果。世界模型、视频生成和视觉-语言-动作（VLA）政策的最新进展推动了世界行动模型（WAMs）的发展，该模型将未来世界预测与可执行的动作生成结合起来。本综述提供了面向机器人的WAM综述。我们澄清了其相对于传统世界模型、基于模型的强化学习、动作条件视频生成和反应式VLA策略的范围，并通过涵盖表示、过渡建模、动作界面、架构、训练流程、数据模态和缩放策略的统一分类法组织现有方法。我们进一步回顾了WAM在操作、导航和自动驾驶中的应用，并总结了用于评估WAM系统的数据集、基准、指标和协议。最后，我们讨论了动作对齐、世界动作分解、空间与多视角一致性、长视野记忆、闭环策略学习中的神经仿真以及高效推断中的关键挑战。综合来看，本综述旨在为预测世界建模与动作生成的整合提供简明的技术基础，实现更可靠的具身机器人智能。项目页面：此 https URL。

The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis

模仿游戏：当大型语言模型通过以代码为中心的推理数据综合学会像程序一样推理时

• Authors: Jinyang Zhang, Weibin Liao, Keqin Bao, Sihang Li, Shaobo Wang, Muyang Ye, Hongxin Ding, Yue Fang, Tianyi Tang, Fei Huang, Kexin Yang, Xingzhang Ren, Dayiheng Liu
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.16076
• Pdf link: https://arxiv.org/pdf/2609.16076
• Abstract Large Language Models (LLMs) excel at programming tasks but frequently fail at deterministic, fine-grained reasoning in natural language, relying heavily on semantic approximations rather than robust symbolic execution. To bridge this gap, we propose MIMIC, a framework that leverages executable code as a rigorous medium for reasoning data synthesis. MIMIC fundamentally transforms algorithms into verifiable reasoning trajectories through narrative fusion, code-guided test synthesis, and dynamic code instrumentation. Crucially, these explicit intermediate execution states naturally form a Code-Instrumented Reward (CIR), providing dense, high-fidelity process supervision for reinforcement learning without external reward models. Extensive evaluations reveal that models trained via SFT and GRPO on our synthesized dataset achieve substantial, consistent gains. Our method significantly elevates accuracy across general reasoning, complex mathematical benchmarks, and fine-grained deterministic tasks, demonstrating that the procedural rigor of executable code can effectively unlock and enhance the generalized reasoning capabilities of LLMs. Our code and data are available at this https URL.
• 中文摘要 大型语言模型（LLM）在编程任务中表现出色，但在自然语言中确定性、细粒度推理方面常常失败，主要依赖语义近似而非稳健的符号执行。为弥合这一差距，我们提出了MIMIC框架，利用可执行代码作为严谨的数据推理媒介。MIMIC通过叙事融合、代码引导测试综合和动态代码工具，根本上将算法转化为可验证的推理轨迹。关键是，这些显式中间执行状态自然形成代码-仪器化奖励（CIR），为强化学习提供密集且高保真的过程监督，无需外部奖励模型。广泛评估显示，通过SFT和GRPO在我们综合数据集上训练的模型实现了显著且一致的收益。我们的方法显著提升了通用推理、复杂数学基准和细粒度确定性任务的准确性，证明可执行代码的程序严谨性能够有效解锁和增强大型语言模型的通用推理能力。我们的代码和数据可在此 https 网址访问。

How I learned to stop worrying and love StopGrads: Stationarity, Convergence, and a case study on Flow Map Learning

我如何学会停止担忧并热爱StopGrads：平稳性、收敛性以及一个关于流程图学习的案例研究

• Authors: Max W. Shen, Mark Goldstein, Zichu Wang, Aahlad Puli, Rajesh Ranganath
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.16222
• Pdf link: https://arxiv.org/pdf/2609.16222
• Abstract Stopgrads are widely used in training machine learning models, but stopgrads can alter the gradient, stationary points and convergence guarantees of the original objective, which can make stopgrad training theoretically ungrounded. We introduce a stopgrad regression principle, which identifies a general template for stopgrad objectives with a closed-form characterization of stationary points and their uniqueness, unifying stopgrad objectives for flow maps, reinforcement learning, and diffusion samplers. We provide theoretical grounding for optimizing stopgrad flow map objectives by showing their unique stationary point is the true flow map, and showing positive convergence results for Eulerian and Lagrangian objectives, including MeanFlow and improved MeanFlow. Remarkably, we show that under functional semi-gradient flow, the learned flow map has a closed-form expression composing the initial flow map and the true flow map. We additionally use our stopgrad regression principle to propose modified stopgrad placements for flow map objectives which reduce training memory by 2x.
• 中文摘要 Stopgrad 广泛用于机器学习模型训练，但 stopgrad 可以改变原始目标的梯度、驻点和收敛保证，使 stopgrad 训练在理论上变得不靠谱。我们引入了 stopgrad 回归原理，通过对驻点及其唯一性的闭式刻画，为 stopgrad 目标提供了通用模板，统一了流图、强化学习和扩散采样器的 stopgrad 目标。我们通过展示其唯一的驻点是真实的流图，以及对欧拉和拉格朗日目标（包括 MeanFlow 和改进型 MeanFlow）的积极收敛结果，为优化 stopgrad 流图目标提供了理论基础。值得注意的是，我们证明在函数半梯度流下，学习到的流图有一个闭式表达式，由初始流图和真实流图组成。我们还利用 stopgrad 回归原则，提出了修改后的 stopgrad 放置方式，用于将训练记忆减少 2 倍的流程图目标。

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

虚假工具的使用：当强化学习者学会错误的行动理由时

• Authors: Yiwei Yang, Haoxiang Zhang, Bingbing Wen, Yao Lu, Yuchen Wu, Lei Zhang, Julian McAuley, Pan Lu, Bill Howe
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.16268
• Pdf link: https://arxiv.org/pdf/2609.16268
• Abstract Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.
• 中文摘要 大型语言模型（LLM）代理越来越多地将自然语言推理与外部工具如网页搜索和代码执行交织使用。这些工具使用策略通常通过强化学习（RL）进行优化，从而放大训练数据中的虚假相关性。本研究中，我们研究了RL训练的代理何时以及为何学习捷径工具选择策略：基于表面提示线索而非真实任务需求调用工具。我们构建了结合事实性问答和数学推理任务的受控综合环境，并在训练过程中注入与特定工具强烈相关但与工具必要性无关的线索。在反事实评估中，线索存在但相关工具并非必需，代理表现出显著的捷径行为，虚假工具调用率可提升多达39%。然而，捷径形成并非普遍存在：在我们测试的条件下，只有当智能体已经学会可靠地使用目标工具时，捷径形成才会出现，这表明任务能力而非仅仅是数据集失衡，才是捷径学习的关键因素。交换提示分析进一步表明，提示与工具之间的语义对齐显著增强了这一效应。为减轻这些失败，我们引入了密集的决策级奖励，由大型语言模型评审评估每次工具调用的必要性。这种工具需求奖励有效抑制了基于提示的工具使用，同时保持任务性能，为提升LLM代理工具使用策略的稳健性提供了切实可行的方法。

Symmetric solution of the Bellman optimality equation for repeated harmony game

重复和声博弈贝尔曼最优方程的对称解

• Authors: Hisato Komatsu
• Subjects: Subjects: Computer Science and Game Theory (cs.GT); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2609.16289
• Pdf link: https://arxiv.org/pdf/2609.16289
• Abstract In social dilemma games, additional rewards or punishments have been studied as means of promoting cooperation. Therefore, it is important to investigate the ideal situation, in which such an additional payoff would change the game. In this study, we investigated the symmetric solution of the Bellman optimality equation for a repeated harmony game. The calculations showed that three types of symmetric solutions exist. One of them corresponds to the trivial All-C strategy, and another to the Win-stay Lose-shift strategy of the prisoners dilemma game. The nontrivial behavior of the strategy corresponding to the last solution is also discussed in detail. In addition, we numerically investigated which strategy the agents actually learn by the reinforcement learning algorithm.
• 中文摘要 在社会困境博弈中，额外的奖励或惩罚被研究为促进合作的手段。因此，研究理想情境非常重要，即此类额外收益会改变博弈。本研究研究了重复和谐博弈贝尔曼最优方程的对称解。计算显示存在三种对称解。其中一种对应平凡的全C策略，另一种对应囚徒困境博弈中的胜-停留-输-转移策略。对应最后解法的策略的非平凡行为也被详细讨论。此外，我们通过强化学习算法数值研究了代理实际学习的策略。

Policy Gradient over History-Dependent Policy Classes for LQR with Domain Randomization

LQR与域随机化的历史相关策略类策略梯度

• Authors: Tesshu Fujinami, Bruce D. Lee, Anastasios Tsiamis, Nikolai Matni, George J. Pappas
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.16300
• Pdf link: https://arxiv.org/pdf/2609.16300
• Abstract Domain Randomization (DR) has been widely used to overcome the sim-to-real gap by training a controller on a distribution of simulated environments via reinforcement learning. While DR can achieve robust performance simply using controllers synthesized via policy gradient (PG) methods, the optimization landscape is not well understood, even in the case of linear quadratic regulator (LQR) objectives. To this end, we first study PG of domain randomized LQR over history-dependent policy classes, such as finite impulse response controllers, as they can extend the possibilities of simultaneous stabilization. Second, to find such a stabilizing controller, we propose a curriculum learning based algorithm which gradually expands the memory of the controller. Finally, we show that PG with the proposed algorithm converges globally to the minimizer of a sample average approximation of the DR objective under suitable bounds on the heterogeneity of environments. Empirical results support our findings and highlight promising directions for future work, including nonlinear domain-randomized control.
• 中文摘要 域随机化（DR）已被广泛应用于通过强化学习训练控制器，克服模拟与现实之间的差距。虽然DR仅通过策略梯度（PG）方法合成控制器即可实现稳健性能，但优化环境尚未充分理解，即使是线性二次调节器（LQR）目标。为此，我们首先研究域随机LQR在历史依赖策略类（如有限脉冲响应控制器）上的PG，因为它们可以扩展同时稳定的可能性。其次，为寻找此类稳定控制器，我们提出一种基于课程学习的算法，逐步扩展控制器的内存。最后，我们证明PG在环境异质性适当界限下，全局收敛到DR目标的样本平均近似的最小化。实证结果支持我们的发现，并凸显了未来研究的有前景方向，包括非线性领域随机对照。

Autonomous Droplet Navigation via Model-Based Reinforcement Learning

通过基于模型的强化学习实现自主液滴导航

• Authors: Rajneesh Anand, Mayuresh V. Kothare
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO); Systems and Control (eess.SY); Fluid Dynamics (physics.flu-dyn)
• Arxiv link: https://arxiv.org/abs/2609.16369
• Pdf link: https://arxiv.org/pdf/2609.16369
• Abstract Precise manipulation of liquid droplets underpins lab-on-a-chip platforms for diagnostics, chemical synthesis, and biological assays. Yet autonomous droplet transport through confined geometries of varying complexity remains an open challenge. Droplets exhibit contact-angle hysteresis, deformability, and capillary pinning, which make their response to actuation nonlinear and history dependent, that classical controllers and pre-programmed trajectories cannot cope in multi-turn environments. Here we demonstrate autonomous navigation of a liquid droplet through geometries of increasing complexity on a gravity driven (Labyrinth) platform using model-based reinforcement learning. A thin silicone oil film reduces contact-line pinning while two-axis tilt supplies the gravitational driving force, and an overhead camera tracks the droplet in real time. An offline-trained policy discovers effective tilt strategies from limited physical interaction data, without simulation or analytical droplet models. The system operates under partial observability, as oil-film thickness, instantaneous contact angle, and droplet deformation state remain hidden from the controller. Despite these challenges, the learned policy achieves reliable navigation across straight, right-angle, and curved-arc paths, including outside-corner geometries. We further demonstrate that a policy trained on a simpler geometry transfers to complex ones, succeeding zero-shot on right-angle and staircase paths and reaching full success on a curved arc with a fifth of the training data. The findings suggest promising avenues for enabling droplet based microfluidic systems to serve as intelligent chemical laboratories.
• 中文摘要 液滴的精确操作支撑了实验室芯片平台的诊断、化学合成和生物测定。然而，在复杂度不一的受限几何结构中自主运输液滴仍是一个开放的挑战。液滴表现出接触角滞后、可变形性和毛细固定现象，使其对驱动的响应非线性和历史依赖性，经典控制器和预设轨迹无法在多转环境中应对。我们演示了利用基于模型的强化学习，在重力驱动（迷宫）平台上自主导航液滴穿越复杂度不断增加的几何结构。薄硅油膜减少接触线固定，双轴倾斜提供引力驱动力，顶置摄像头实时跟踪液滴。离线训练策略通过有限的物理相互作用数据发现有效的倾斜策略，无需模拟或液滴分析模型。系统在部分可观测性下运行，油膜厚度、瞬时接触角和液滴变形状态对控制器保持隐藏。尽管面临这些挑战，该策略仍能在直线、直角和弧形路径（包括外角几何）中实现可靠导航。我们进一步证明，基于简单几何训练的策略可转换为复杂策略，在直角和阶梯路径上实现零点成功，并在曲弧上以五分之一的训练数据实现完全成功。研究结果为使基于液滴的微流体系统成为智能化学实验室提供了有前景的途径。

Fast-Convergent Meta-RL via Gradient-Clustered BS Sampling for Edge Caching

通过梯度聚类BS采样实现的快速收敛元强化学习（Meta-RL）用于边缘缓存

• Authors: Farnaz Niknia, Ping Wang
• Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.16370
• Pdf link: https://arxiv.org/pdf/2609.16370
• Abstract Wireless edge caching networks typically consist of many independent Base Stations (BSs), each facing its own request rate and content popularity profile. Training a Reinforcement Learning (RL) caching agent from scratch at every BS forces each agent to relearn, through slow trial and error, a decision problem that is structurally identical across the network. Meta-reinforcement learning removes this redundancy by learning a shared initialization that adapts to any BS in a few local updates; however, meta-training itself becomes the bottleneck at scale: the meta-gradient must be estimated from a small subset of BSs at each meta-iteration, and sampling this subset uniformly at random yields a high-variance estimate, an issue existing meta-RL caching frameworks leave unaddressed. This paper proposes a meta-reinforcement learning framework for caching across independent, non-overlapping BSs that directly targets this bottleneck. Each BS runs a local Proximal Policy Optimization (PPO) agent, formulated as a Semi-Markov Decision Process (SMDP) over content popularity, size, lifetime, and importance, while a shared meta-policy is learned via a Model-Agnostic Meta-Learning (MAML)-style loop. To scale meta-training and accelerate convergence, we introduce gradient-based clustering, which groups BSs by local gradient similarity and draws from every cluster, in proportion to its size, at each meta-iteration. We prove, via an Analysis of Variance (ANOVA)-style decomposition of gradient variance, that this strategy yields a strictly lower-variance meta-gradient estimator than uniform random sampling under BS heterogeneity.
• 中文摘要 无线边缘缓存网络通常由多个独立的基站（BS）组成，每个基站面临自己的请求率和内容受欢迎度轮廓。在每个BS从零训练强化学习（RL）缓存代理，迫使每个代理通过缓慢的试错重新学习一个结构在网络中完全相同的决策问题。元强化学习通过学习共享初始化，在几次本地更新中适应任何BS，从而消除了这种冗余;然而，元训练本身成为大规模的瓶颈：每次元迭代中必须从一小部分BS中估算元梯度，且随机均匀采样该子集得出高方差估计，这是现有元强化缓存框架未解决的问题。本文提出了一种跨独立、不重叠BS缓存的元强化学习框架，直接针对这一瓶颈。每个BS运行一个局部近端策略优化（PPO）代理，该代理以半马尔可夫决策过程（SMDP）形式对内容受欢迎度、大小、寿命和重要性进行表述，同时通过模型无关元学习（MAML）式循环学习共享元策略。为扩大元训练并加速收敛，我们引入了基于梯度的聚类，按局部梯度相似度对BS进行分组，并在每次元迭代中按簇大小比例提取信息。我们通过方差分析（ANOVA）式梯度方差分解，证明该策略在BS异质性下产生的元方差估计量严格低于均匀随机抽样。

Register Tokens for Bounded-State Reasoning in Diffusion Language Models

扩散语言模型中有界状态推理的寄存器令牌

• Authors: Albert Ge, Chandan Singh, Yufan Zhuang, Xiaodong Liu, Jianfeng Gao, Frederic Sala
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.16372
• Pdf link: https://arxiv.org/pdf/2609.16372
• Abstract Masked diffusion language models (dLLMs) generate text by iteratively denoising masked tokens with bidirectional attention. Extending reasoning across generation chunks normally requires keeping earlier generated text in context. We ask whether a dLLM can instead continue reasoning after that text is cleared, using only a fixed-size carried state. We implement this state as a small number of register tokens: dedicated fixed-position tokens whose continuous hidden states are trained to carry reasoning progress across generation chunks. We post-train dLLMs to decode a chunk of text, clear it while preserving the register values, and continue decoding from the prompt and carried state. In our main comparisons on LLaDA and Dream, registers outperform discrete-text carry on every benchmark, with gains of up to 8.5 points on math and 19.5 points on code. Registers are especially effective for bounded code generation, where correct programs usually span several chunks. Finally, registers can be further refined with reinforcement learning on long-horizon reasoning tasks.
• 中文摘要 掩蔽扩散语言模型（dLLMs）通过双向关注迭代去噪化掩码标记来生成文本。跨世代块扩展推理通常需要将早期生成文本置于上下文中。我们询问dLLM是否可以在文本清除后仅使用固定大小的携带状态继续推理。我们将该状态实现为少量寄存器标记：专用的固定位置标记，其连续隐藏状态被训练用于跨世代块传递推理进展。我们后训练dLLM解码一段文本，清除时保持寄存器值，并继续从提示和携带状态解码。在我们对LLaDA和Dream的主要比较中，寄存器在所有基准测试中均优于离散文本进位，数学提升高达8.5分，代码提升19.5分。寄存器在有界代码生成中尤为有效，正确程序通常跨越多个区块。最后，寄存器可以通过强化学习进一步完善长视野推理任务。

UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data

UniDex-ViTac：从人类视频数据学习统一的Visuo-触觉灵巧操作策略

• Authors: Hyesung Lee, Si-Hwan Heo, Sungwook Yang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.16504
• Pdf link: https://arxiv.org/pdf/2609.16504
• Abstract Human videos provide demonstrations of dexterous manipulation but lack robot-executable actions and tactile measurements. We present UniDex-ViTac, a framework that uses human-video-guided simulation to generate robot demonstrations paired with fingertip contact observations for training a deployable visuo-tactile policy. Object-specific residual reinforcement learning specialists adapt annotated human-object interaction references to a robotic arm-hand system. Their successful rollouts pair final robot action targets with robot-side fingertip contact observations. From 50 human demonstrations across ten objects, we collect 10,000 simulated trajectories to train a single Action Chunking with Transformers (ACT) based generalist. The policy combines point clouds, proprioception, and four binary contact signals encoded through fingertip labels and a separate token, without requiring human references or privileged object identity and pose at deployment. The contact-augmented configuration achieves 68.3% macro-average success in simulation, compared with 55.5% for the point-cloud-only baseline. Without real-robot demonstrations or policy fine-tuning, it succeeds in 73/110 physical trials (66.4%) across six seen and five unseen objects, compared with 60/110 (54.5%) for the baseline, an increase of 11.8 percentage points. These results support the feasibility of learning a unified visuo-tactile dexterous manipulation policy from video-guided simulated interactions. Project page: this https URL
• 中文摘要 人类视频展示了灵巧的操作，但缺乏机器人可执行动作和触觉测量。我们介绍UniDex-ViTac框架，利用人类视频引导模拟生成机器人演示，结合指尖接触观察，训练可部署的视觉-触觉策略。对象特定残余强化学习专家将带注释的人与物交互参考应用于机器人手臂-手系统。他们的成功推广将最终机器人动作目标与机器人侧指尖接触观察匹配。通过50次人类演示，涵盖十个物体，我们收集了1万条模拟轨迹，用于训练基于变形金刚（ACT）的单一动作分块（ACT）通用型。该策略结合了点云、本体感觉和四个通过指尖标签和独立令牌编码的二元接触信号，无需人工引用或部署时特权对象身份和姿态。接触增强配置在模拟中宏观平均成功率为68.3%，而仅点云基线为55.5%。在没有真实机器人演示或策略微调的情况下，在6个可见物体和5个未可见物体中成功完成73/110个物理试验（66.4%），而基线测试为60/110（54.5%），提高了11.8个百分点。这些结果支持通过视频引导模拟交互学习统一视觉-触觉灵巧操作策略的可行性。项目页面：此 https URL

A Cyber Range Evaluation of Autonomous Network Incident Response Agents

自主网络事件响应代理的网络范围评估

• Authors: Jakob Nyberg, Teodor Sommestad, Andrei Buhaiu, Joakim Loxdal, Pontus Johnson, Mathias Ekstedt
• Subjects: Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.16541
• Pdf link: https://arxiv.org/pdf/2609.16541
• Abstract We test the performance of agents for automated network intrusion response in a cyber range intended for human operator training. The range implements an emulated networking environment with a variable network topology, red-team emulation and simulated user agents. The goal of the defensive agents is to prevent hosts in the network from being accessed by the red-team agent, while minimizing the availability costs induced from defensive measures. Alerts are generated using a SIEM platform and mapped to a data modeling language used by the agents. We test a combination of heuristic agents and policies learned using reinforcement learning. The learned policies are optimized to minimize the combined cost using a cyber attack simulator modeling the network. We found that the reinforcement learning agents were overall more efficient at defending the system than the heuristic policy, and that the performance depends highly on the policy of the adversary in combination with the simulated users.
• 中文摘要 我们测试代理在面向人类操作员培训的网络范围中自动网络入侵响应的性能。该范围实现了模拟网络环境，具有可变网络拓扑、红队仿真和模拟用户代理。防御代理的目标是防止红队代理访问网络中的主机，同时最大限度减少防御措施带来的可用性成本。警报通过SIEM平台生成，并映射到代理使用的数据建模语言。我们测试通过强化学习学习的启发式代理和策略的组合。通过网络攻击模拟器对网络进行优化，以最小化综合成本。我们发现强化学习代理整体上比启发式策略更高效地防御系统，性能高度依赖于攻击方策略与模拟用户的结合。

GrowMTP: Can RL Grow Its Own Draft Head?

GrowMTP：现实生活能培养自己的选秀负责人吗？

• Authors: Minghua He, Lingzhe Zhang, Yuan Liu, Xiao Zhou, Aiwei Liu
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.16648
• Pdf link: https://arxiv.org/pdf/2609.16648
• Abstract Reinforcement learning (RL) post-training drives the frontier capabilities of large language models, with its wall-clock dominated by autoregressive rollout generation. Speculative decoding is an established remedy for this bottleneck, but existing draft heads must be pretrained or warmed up before RL, introducing substantial training cost outside the RL run to be accelerated. We observe that RL training itself provides both conditions required for online draft-head training: its rollout distribution is far narrower than that of pretraining, and its verification step continuously produces supervision signals aligned with this distribution. Building on these observations, we propose GrowMTP, which uses this supervision to train a draft head from scratch entirely within the RL loop, with all head updates detached from the policy backbone. On Qwen3-4B (no draft head), MiMo-7B-SFT (weak head), and Qwen3.5-4B-Base (strong head), GrowMTP achieves rollout speedups of 2.13x, 1.93x, and 1.36x, and end-to-end speedups of 1.60x, 1.41x, and 1.20x, respectively. GrowMTP therefore serves existing RL training frameworks as a modular component, particularly offering a from-scratch acceleration path for models without pretrained draft heads.
• 中文摘要 强化学习（RL）后训练推动了大型语言模型的前沿能力，其壁钟机制主要由自回归展开生成主导。推测解码是解决这一瓶颈的既定方法，但现有的草稿头必须在强化学习前进行预训练或预热，这会带来加速的强化运行之外的大量训练成本。我们观察到，强化学习本身满足了在线草稿头训练所需的两种条件：其推广分布远小于预训练，且其验证步骤持续产生与该分布相符的监督信号。基于这些观察，我们提出了GrowMTP，利用这种监督从零开始训练草稿头，完全在强化学习循环内，所有头更新都脱离策略骨干。在Qwen3-4B（无磁头）、MiMo-7B-SFT（弱磁头）和Qwen3.5-4B基础（强磁头）上，GrowMTP分别实现了2.13倍、1.93倍和1.36倍的推展加速，以及端到端的1.60倍、1.41倍和1.20倍。因此，GrowMTP作为模块化组件服务于现有的强化学习训练框架，特别是为没有预训练草稿头的模型提供从零开始的加速路径。

Rewarding Reasoning, Not Answers: Fixing and Bounding Test-Time Reinforcement Learning on Medical QA

奖励推理，而非答案：医学质量保证中的测试时间强化学习修正与界限

• Authors: Kailong Fan, Anqi Pu, Yichen Wu, Wanhua Li, Yicong Li, Hanspeter Pfister, Huafeng Liu, Xiang Li, Quanzheng Li, Ning Guo
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.16660
• Pdf link: https://arxiv.org/pdf/2609.16660
• Abstract Test-time reinforcement learning adapts a model on its own unlabeled test set using majority-vote pseudo-labels and has shown strong results in mathematics. We show that this recipe collapses on medical multiple-choice QA: accuracy stagnates while output diversity rapidly declines. Through a controlled experiment that keeps the questions, model, and optimizer fixed while changing only the answer space, we trace this failure to answer-space structure rather than domain difficulty. In small answer spaces, incorrect rollouts often collide on the same wrong pseudo-label and reinforce it; in large answer spaces, they disperse and receive little reward. This diagnosis motivates PROSE, Process Reward Guided Self-Training, which rewards reasoning quality instead of answer agreement. PROSE scores each reasoning step with a medical process reward model, assigns the trajectory reward as the minimum score across steps, and enforces answer-format constraints. Without labels, PROSE substantially improves a general Llama model, surpassing purpose-built medical models and matching much larger systems. Because the process signal is internalized into the policy, the adapted model requires no reward model at inference and transfers its gains to unseen datasets. We further show that the minimum aggregation is essential: mean aggregation can be exploited, saturating the proxy reward while degrading accuracy.
• 中文摘要 测试时强化学习在其无标签测试集上采用多数票伪标签，并在数学上取得了显著效果。我们证明了该方案在医学多项选择题质检中崩溃：准确性停滞，输出多样性迅速下降。通过一个控制实验，保持问题、模型和优化器固定，仅改变答案空间，我们追踪到这种失败是答案空间结构而非领域难度。在小答案空间中，错误的展开常常在同一错误的伪标签上碰撞并强化它;在大答案空间中，错误的展开分散且奖励有限。这一诊断激励了PROSE（过程奖励引导自我训练），奖励推理质量而非答案一致性。PROSE用医学过程奖励模型对每个推理步骤进行评分，将轨迹奖励指定为各步骤的最低得分，并强制执行答案格式约束。在没有标签的情况下，PROSE大幅改进了通用的Llama模型，超越了专门构建的医疗模型，匹配了更大的系统。由于过程信号被内化进策略中，适应后的模型在推断时无需奖励模型，并将收益转移到未见数据集中。我们还进一步表明，最小聚合是必不可少的：平均聚合可以被利用，使代理奖励饱和，同时降低准确性。

TIAO: Token Importance-Aware Policy Optimization for Text Summarization

TIAO：令牌重要性感知策略优化文本摘要

• Authors: Qixiu Li, Chenlong Bao, Xiang Zhu, Xiaoyong Li, Ruixin Cao, Shukai Chen, Zhenxiong Zhou
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.16748
• Pdf link: https://arxiv.org/pdf/2609.16748
• Abstract Text summarization requires models to condense content while preserving key qualities such as consistency and coherence. Large language models (LLMs) have shown strong performance on this task and can be further improved through reinforcement learning (RL). However, most existing methods apply reward signals directly to undifferentiated token sequences, overlooking the varying importance of individual tokens to word and sentence level quality in summarization. In this paper, we propose Token Importance-Aware Policy Optimization (TIAO), a novel reinforcement learning strategy that explicitly leverages token-importance awareness. Specifically, TIAO identifies core tokens based on token dependency and reweights a trajectory's advantage according to its overall dependencies. Experiments on the real world dataset show that our TIAO achieves highly competitive results, and that a 7B foundation model enhanced by TIAO performs comparably to GPT-4 and GPT-5-nano. Code is available at this https URL
• 中文摘要 文本摘要需要模型在保持一致性和连贯性等关键品质的同时，压缩内容。大型语言模型（LLMs）在该任务中表现出强劲表现，且可通过强化学习（RL）进一步提升。然而，大多数现有方法直接将奖励信号应用于未区分的令牌序列，忽视了单个词符对摘要中词句质量重要性的差异。本文提出令牌重要性感知策略优化（TIAO），这是一种新颖的强化学习策略，明确利用令牌重要性意识。具体来说，TIAO基于令牌依赖性识别核心令牌，并根据整体依赖性重新加权轨迹的优势。在现实世界数据集上的实验表明，我们的TIAO实现了高度竞争的结果，且由TIAO增强的7B基础模型表现可与GPT-4和GPT-5-nano相当。代码可在此 https URL 获取

ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

不可可能的评分标准：压力测试生成的评分标准作为奖励信号

• Authors: Bowen Qin, Yi Xie, Yesheng Liu, Xi Yang
• Subjects: Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.16816
• Pdf link: https://arxiv.org/pdf/2609.16816
• Abstract Language model-generated rubrics are increasingly used as reward signals for rubric-based reinforcement learning, LLM-as-a-judge evaluation, and automated grading. Such rubrics are reliable only if they reward honest answers over adversarial answers optimized to exploit them. Yet their robustness to such optimization remains poorly understood. We isolate the hardest regime: impossible tasks, where the prompt pressures the model toward an unsupported conclusion, so the only honest response is to acknowledge the impossibility. We introduce ImpossibleRubrics, a benchmark of 169 impossible tasks spanning six impossibility categories, each paired with a verifiable oracle certificate specifying what an honest answer may and may not claim, together with 48 answerable controls. Rather than providing fixed rubrics, ImpossibleRubrics provides task environments and certificates, allowing rubrics to be generated downstream and then adversarially tested for whether they reward certificate-violating answers. Eleven generators are exploited 8--26% of the time on the unbiased 150-of-169 environment cut; on a deliberately selected stress cut the strongest generator we measured is still exploited 36% while a certificate-faithful rubric is exploited 0%, so what we measure is a rubric-quality gap, not task impossibility. One result runs against intuition. A single generic rubric ("be decisive, penalize hedging") used unchanged for every task is exploited 64% of the time, and seven of the eleven generators are exploited more often than that while writing a rubric tailored to each one. The tailored criteria appear to tell an attacker which claim to fabricate. The problem is not that rubrics are vague; it is that they are specific about the wrong things.
• 中文摘要 语言模型生成的评分标准越来越多地被用作基于评分标准的强化学习、LLM即评判评估和自动评分的奖励信号。只有当这些评分标准奖励诚实答案而非针对性答案时才可靠。然而，它们对这种优化的韧性性仍难以理解。我们隔离出最难的模式：不可能任务，提示使模型得出无依据的结论，因此唯一诚实的回应是承认不可能。我们引入了ImpossibleRubrics，这是包含169个不可能任务的基准，涵盖六个不可能类别，每个任务配有可验证的预言证证书，说明诚实答案可以声称和不能声称什么，以及48个可负责任的控制。ImpossibleRubrics不提供固定的评分标准，而是提供任务环境和证书，允许评分标准在下游生成，然后对抗性测试是否奖励违反证书的答案。在无偏的150/169环境割减中，11个生成器被利用的概率为8-26%;在有意选择的压力割减中，我们测量的最强生成器仍被利用36%，而证书忠实评分标准仅为0%，因此我们测量的是评分标准质量差距，而非任务不可能。其中一个结果与直觉相悖。一个统一的通用评分标准（如“果断，惩罚对冲”）在每个任务中未更改使用，被利用率达到64%，而在编写针对每个指标的评分标准时，11个生成器中的7个被利用的频率更高。定制化的标准似乎告诉攻击者谁声称伪造。问题不在于评分标准模糊;而是它们针对错误的事物具体化。

Interactive Memory Learning for Long-Term Conversations

长期对话的互动记忆学习

• Authors: Cai Ke, Jiangyue Yan, Han Zhang, Xin Liu, Zike Yuan, Yue Yu, Hui Wang, Ruifeng Xu
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.17088
• Pdf link: https://arxiv.org/pdf/2609.17088
• Abstract Recent advancements in large language models have significantly enhanced the capabilities of agents in modeling long-term conversations. Despite these successes, existing approaches typically adopt a static heuristic paradigm, where information is passively archived without adaptive memory valuation. Consequently, these methods fail to self-evolve or align their memory management with evolving user needs. To address this, we propose ICML (InteraCtive Memory Learning), a multi-agent framework that transforms the memory mechanism from a passive archive into a learnable, interactive memory policy. Specifically, we first employ a session synthesis pipeline to generate expert data, facilitating rapid test-time adaptation in unseen scenarios. Building on this, ICML utilizes an online reinforcement learning mechanism where a Planner agent selectively encodes high-value information and a Trigger agent dynamically retrieves it to optimize response quality, whereby the two agents co-evolve through continuous interaction feedback. Crucially, both agents are synchronized through a delayed reward mechanism that propagates future feedback back to earlier storage decisions, ensuring memory policies are precisely aligned with user expectations. Experimental results demonstrate that ICML significantly outperforms strong baselines, exhibiting the unique capability to continuously improve response quality as interactions accumulate.
• 中文摘要 大型语言模型的最新进展显著提升了代理在建模长期对话的能力。尽管取得了这些成功，现有方法通常采用静态启发式范式，即信息被动归档，不进行自适应内存估值。因此，这些方法未能自我演进或使其内存管理与用户需求演变保持一致。为此，我们提出了ICML（跨智能体记忆学习）框架，将记忆机制从被动归档转变为可学习、交互的记忆策略。具体来说，我们首先采用会话综合流水线生成专家数据，便于在未见场景中快速适应测试时间。基于此，ICML采用在线强化学习机制，计划者智能体选择性编码高价值信息，触发智能体动态检索以优化响应质量，双方通过持续交互反馈共同进化。关键是，两个智能体通过延迟奖励机制同步，将未来反馈传递至早期存储决策，确保记忆策略与用户期望精确对齐。实验结果显示，ICML远远优于强基线，展现出随着交互积累持续提升响应质量的独特能力。

Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand

手指作为腿：用拟人化手学习自我支撑的运动与操作

• Authors: Amirhossein Kazemipour, Hehui Zheng, Robert Katzschmann
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2609.17172
• Pdf link: https://arxiv.org/pdf/2609.17172
• Abstract A walking robotic hand must use the same fingers to move its body, support its weight, and interact with the environment. We show how an anthropomorphic hand can learn these skills while retaining its finger design and position controller. Onboard power and computation make the platform self-contained. Our reinforcement learning approach accounts for the hand's unequal fingers, with training in a simulator calibrated from hardware measurements. In simulation, the hand moves faster with our reward formulation than with tuned rewards originally designed for quadrupeds. On hardware, task-specific policies enable untethered crawling, steering, and fall recovery. While supporting its own weight, the hand also executes successive keyboard commands without vision and pushes an object to targets using overhead visual feedback. These results demonstrate a compact mobile manipulator that reuses its fingers for locomotion and interaction, without a separate locomotion mechanism.
• 中文摘要 行走的机械手必须使用相同的手指来移动身体、支撑体重并与环境互动。我们展示了拟人化手如何在保留手指设计和位置控制器的同时学习这些技能。机载电源和计算使平台实现自给自足。我们的强化学习方法考虑了手部手指不等位的情况，训练过程通过模拟器根据硬件测量校准。在模拟中，采用我们的奖励公式，手的移动速度比最初为四足动物设计的调校奖励更快。在硬件上，任务特定策略支持无绳爬行、转向和跌倒恢复。在支撑自身重量的同时，手还能执行连续的键盘指令，无需视觉，并通过俯视反馈推动物体到目标。这些结果展示了一种紧凑的移动操作器，可重复使用手指进行移动和互动，无需独立的运动机制。

MOCC-R1: Reinforcing Reasoning-Response Consistency for Multimodal Counselor Response Generation

MOCC-R1：强化推理-反应一致性以生成多模态咨询师反应

• Authors: Wenjie Zheng, Qiming Xie, Jianfei Yu, Rui Xia
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.17180
• Pdf link: https://arxiv.org/pdf/2609.17180
• Abstract Multimodal counselor response generation (MCRG) aims to generate an appropriate counselor response from multimodal dialogue histories. Progress is limited by two gaps: first, existing datasets rarely capture sustained, human-recorded counseling interactions conducted by qualified counselors; Second, existing methods do not explicitly optimize consistency between counseling reasoning and the generated response, potentially undermining the reliability of MCRG systems. Thus, we introduce MOCC, a multimodal counseling conversation corpus containing over 200 hours of interactions involving 154 credential-verified counselors. Based on MOCC, we propose MOCC-R1, a two-stage framework for optimizing reasoning-response consistency. Cold-start supervised fine-tuning trains the model to generate a structured trajectory consisting of client-state understanding, a response intent that links a counseling principle to a planned action, and the final response. Reinforcement learning (RL) then rewards grounded plan coherence and plan execution, encouraging the inferred state and plan to be supported by the dialogue context and the response to realize that plan. Experiments demonstrate the effectiveness of the proposed MOCC-R1.
• 中文摘要 多模态咨询师反应生成（MCRG）旨在从多模态对话历史中生成合适的咨询师回应。进展受限于两个缺口：首先，现有数据集很少捕捉合格咨询师进行的持续、人类记录的咨询互动;其次，现有方法未能明确优化咨询推理与生成反应之间的一致性，可能削弱MCRG系统的可靠性。因此，我们引入了MOCC，一个包含超过200小时互动的多模态咨询会话语料库，涉及154名经过资质验证的咨询师。基于MOCC，我们提出了MOCC-R1，这是一个两阶段框架，用于优化推理与反应一致性。冷启动监督微调训练模型生成结构化轨迹，包括客户状态理解、将咨询原则与计划行动联系起来的反应意图，以及最终反应。强化学习（RL）随后奖励基于基础的计划一致性和执行，鼓励推断状态和计划得到对话情境的支持，并促使实现该计划的响应。实验展示了所提议MOCC-R1的有效性。

FluxVLA Engine: A One-Stop VLA Engineering Platform for Embodied Intelligence

FluxVLA引擎：实现具身智能的一站式VLA工程平台

• Authors: Yinhao Li, Weixin Mao, Zihan Lan, Jikun Rong, Qirui Hu, Yiming Zhang, Weipeng Deng, Bowen Shen, Minzhao Zhu, Yiming Mao, Yan Yang, Chenguang Cui, Hongyuan Chen, Xu Huang, Zheyi Zhao, Pinxi Shen, Bozhen He, Zhen Fu, Yifan Wang, Zexin Zhang, Ang Gao, Haoyu Chen, Chengqi Shi, Hua Chen
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2609.17210
• Pdf link: https://arxiv.org/pdf/2609.17210
• Abstract Vision-language-action (VLA) models, world-action models (WAMs), and offline reinforcement learning methods are rapidly expanding the design space of embodied policies, yet turning these algorithms into reliable robot systems remains constrained by fragmented data formats, training stacks, evaluation protocols, inference runtimes, and embodiment-specific interfaces. We present $\mathrm{FluxVLA}$ Engine, an open, configuration-driven platform that turns heterogeneous embodied-policy components into a reproducible data-to-deployment workflow. Rather than introducing another policy model, $\mathrm{FluxVLA}$ standardizes interfaces for datasets, visual-language and world models, action heads, reward- or advantage-weighted learning, distributed training, simulation evaluation, optimized inference, and robot operators. The engine further integrates compositional dual-arm simulation, scalable automatic data generation, and model-decoupled human-in-the-loop rollout, takeover, correction collection, and reward annotation. For responsive physical execution, it combines Real-Time Chunking (RTC) with accelerated inference backends, lightweight remote GPU serving, and configurable trajectory post-processing. Together, these capabilities connect offline learning, simulation validation, online correction, and real-robot execution through shared and auditable contracts. $\mathrm{FluxVLA}$ therefore targets the engineering bottlenecks separating promising embodied-learning algorithms from reproducible evaluation and dependable deployment. Code is available at this https URL
• 中文摘要 视觉-语言-动作（VLA）模型、世界动作模型（WAMs）和离线强化学习方法正在迅速扩展具身策略的设计空间，但将这些算法转化为可靠的机器人系统仍受限于碎片化的数据格式、训练栈、评估协议、推理运行时以及具现特定接口。我们介绍$\mathrm{FluxVLA}$ Engine，这是一个开放的配置驱动平台，将异构的具象策略组件转变为可重复的数据到部署工作流。$\mathrm{FluxVLA}$ 没有引入另一种策略模型，而是标准化了数据集、视觉语言和世界模型、动作头、奖励或优势加权学习、分布式训练、仿真评估、优化推理和机器人操作员的接口。该引擎进一步集成了组合双臂仿真、可扩展自动数据生成以及模型解耦的人工参与滚动、接管、修正收集和奖励注释。对于响应式物理执行，它结合了实时分块（RTC）、加速推理后端、轻量级远程GPU服务以及可配置轨迹后处理。这些能力结合了离线学习、仿真验证、在线修正以及通过共享和可审计合同实现的实机器人执行。因此，$\mathrm{FluxVLA}$针对了有前景的具身学习算法与可重现的评估和可靠部署之间的工程瓶颈。代码可在此 https URL 获取

Easy to Catch a Liar, Hard to Clear an Honest One: Language Models Diagnosing a Corrupted Reward Channel from a Verified Record

抓骗子容易，诚实骗子难：语言模型从验证记录诊断出奖励通道损坏

• Authors: Arman Nik Khah
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.17226
• Pdf link: https://arxiv.org/pdf/2609.17226
• Abstract An agent that learns from rewards has to trust whatever reports those rewards. When the reports suddenly change, either the world changed or the reporter broke. From the reports alone these are indistinguishable, and reinforcement learning theory shows that no amount of further experience separates them. The prescribed escape is richer data about the reporter itself. We ask whether a frozen language model, handed exactly that data, uses it. We build a two-option game in which a payout swap and a lying reporter produce byte-identical histories. Then we add one verified record: an independent check of one round's real result, printed beside what the reporter said about that round. That single line settles the case. We ask three large models, from two families, to answer one question with one letter. Is the reporter honest or lying? They catch a lying reporter almost perfectly. At the 70B class that holds in every condition we tried; the 32B model slips in one wording. They clear an honest reporter far less often, and how often depends on things that should not matter. Averaged over rounds, letters, and wordings, a 72B model calls an honest reporter a liar 38% of the time when nothing has changed at all, and 58% of the time when the payouts moved. A 70B model from a second family calls an honest reporter a liar 26% and 48% of the time. The failure is not one of reading, because in the situation where nothing changed the same models score 0.96 to 1.00 with the answer printed in the prompt. Which surface feature drives it differs by family. For the Qwen models it is which round the record names, and for Llama it is which letter stands for "honest." Adding the record to a prompt that already states the answer makes Llama less likely to give that answer. We had registered a prediction for that 58% before the run: 35%. The failure is larger than we expected.
• 中文摘要 一个从奖励中学习的代理必须信任报告奖励的对象。当报告突然改变时，要么世界变了，要么报告人崩溃了。仅凭报告内容，这两者无法区分，强化学习理论表明无论多多的经验都无法区分它们。规定的逃逸是关于报告者自身更丰富的数据。我们询问一个冻结的语言模型，恰好得到这些数据，是否会使用该数据。我们构建了一个两选项博弈，其中奖金交换和说谎报告者产生字节相同的历史。然后我们添加一条经过验证的记录：对某轮真实结果的独立检查，印在报告者对该轮的陈述旁边。这行决定了案件。我们让三个大型模型，来自两个家族，用一个字母回答一个问题。报告者是诚实还是在撒谎？他们几乎完美地抓住了说谎记者。在我们尝试的所有条件下成立的70B类别;32B模型在一个措辞上失误。他们对诚实报告员的清算频率要低得多，且多久取决于不该有的因素。通过按轮次、字母和措辞平均，72B模型在未发生变化时有38%的比例称诚实报告者为骗子，58%的概率在奖金变化时称其为骗子。第二家族的70B模型有26%和48%的比例称诚实报告者为骗子。失败不在于阅读，因为在没有任何变化的情况下，相同模型在提示中打印答案时得分为0.96到1.00。驱动其的表面特征因家族而异。对于Qwen模型来说，是记录名称的哪一轮，而对于Llama，则是代表“诚实”的字母。把记录添加到已经说明答案的提示中，会降低Llama给出答案的可能性。我们在运行前预测了58%的成功率：35%。失败比我们预期的还要大。

Calibrate Once, Fly Any Team: Residual-Grounded Low-Fidelity Training for Cooperative Drone Swarms

校准一次，任意飞行：合作无人机群的残余地面低保真训练

• Authors: Maxim Mednikov, Oren Gal
• Subjects: Subjects: Multiagent Systems (cs.MA); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.17265
• Pdf link: https://arxiv.org/pdf/2609.17265
• Abstract Training multi-agent drone-swarm policies directly in high-fidelity (HF) rigid-body physics is accurate but computationally expensive. This cost scales poorly with team size, as each additional agent multiplies contact-resolution complexity and sharply raises the in-simulation crash rate. To address this, we propose a mixed-fidelity training scheme that eliminates HF reinforcement learning entirely. A single shared, decentralized policy is optimized inside a fully-differentiable, JAX-native low-fidelity (LF) point-mass simulator. The simulator is corrected by a small, per-agent bagged residual ensemble fit once, offline, using short calibration flights in the HF simulator. Because calibration requires only one isolated drone, the data collection budget does not compound with team size. Reference trajectories are generated by rolling out an existing LF-only policy and tracked in the HF simulator by a zero-training PD controller. Evaluated across four cooperative drone tasks and team sizes from 3 to 18, the residual-corrected policy outperforms an uncorrected LF baseline in all combinations, and a from-scratch HF policy in 22 of 24 combinations tested. It trails an HF-finetuned policy by a margin that narrows steadily with team size. Ultimately, the proposed method achieves near-equivalent performance at the largest team sizes at a fraction of the computational cost, completely avoiding the high crash rates typical of HF training.
• 中文摘要 直接在高保真（HF）刚体物理中训练多智能体无人机群策略准确，但计算成本高。随着团队规模增加，成本增长不大，每增加一个智能体都会增加接触解析复杂度，并大幅提高模拟中的崩溃率。为此，我们提出了一种混合保真度训练方案，完全消除高频强化学习。在完全可微分、JAX原生的低保真度（LF）点质量模拟器中优化单一共享、去中心化策略。模拟器通过HF模拟器中短距离校准飞行，离线对每个智能体进行一次小型袋装残余集合拟合进行校正。由于校准只需一架孤立无人机，数据收集预算不会随着团队规模增加而增加。参考轨迹通过推广现有仅限LF策略生成，并由零训练PD控制器在HF模拟器中跟踪。该策略在四种协作无人机任务和团队规模（3至18人）中评估，在所有组合中均优于未校正LF基线，在24种测试组合中22种均优于全新HF策略。它落后于HF微调策略，且随着团队规模的扩大逐渐缩小。最终，该方法在最大团队规模下以极低计算成本实现近乎等效的性能，完全避免了HF训练典型的高崩溃率。

ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents

ScienceBuddy：交互式科学代理的递归中递归自我改进

• Authors: Shuhan Xue, Jianyuan Zhong, Ziyuan Nan, Wenbin Li, Zhaochen Yu, Jinchao Ding, Qiang Gao, Pengyu Zhan, Yuntong Zhang, Tian Cheng, Zhenfei Yin, Yingcheng Wu, Ling Yang
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2609.17523
• Pdf link: https://arxiv.org/pdf/2609.17523
• Abstract We introduce and release ScienceBuddy, an interactive scientific research workspace that brings continually improving scientific agents into researchers' everyday workflows. ScienceBuddy supports researchers in carrying out scientific tasks while transforming their requests, feedback, and execution evidence into tasks and evaluation rubrics for continual learning. At its core is recursive-in-recursive self-improvement, a paradigm that couples harness evolution with model reinforcement learning: the inner recursion improves the harness with the model fixed, while the outer recursion trains the model under the improved harness. Harness evolution shapes training experience, and model learning creates new opportunities for harness adaptation. We present case studies of researcher interaction, harness refinement, and model learning, with the benchmark cases spanning four scientific task families. By releasing ScienceBuddy as a research product, we make this paradigm available to the scientific community and take a step toward discovery intelligence: scientific AI that advances through sustained collaboration with researchers and evolves alongside the research it supports. Website: this http URL
• 中文摘要 我们推出了并发布了ScienceBuddy，一个互动式科学研究工作空间，将不断改进的科学代理引入研究人员的日常工作流程。ScienceBuddy支持研究人员完成科学任务，同时将他们的请求、反馈和执行证据转化为任务和评估评分标准，促进持续学习。其核心是递归中的递归自我提升，这是一种将利用进化与模型强化学习结合的范式：内递归在固定模型下改进框架，而外部递归则在改进框架下训练模型。框架演进塑造训练体验，模型学习为框架适应创造了新的机遇。我们展示了研究人员互动、框架优化和模型学习的案例研究，基准案例涵盖四个科学任务家族。通过将ScienceBuddy作为研究产品发布，我们使这一范式向科学界开放，迈出了探索智能的一步：通过与研究人员持续合作，科学人工智能与其所支持的研究同步发展。网站：此 http URL

Keyword: diffusion policy

Dissecting Motion-Prior Regularization for Data-Scarce Robotic Insertion

数据稀缺机器人插入的运动先验正则化剖析

• Authors: Ning Hu, Shuai Li, Jindong Tan
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2609.17484
• Pdf link: https://arxiv.org/pdf/2609.17484
• Abstract This study asks whether training-time motion-prior regularization can improve insertion success when a diffusion policy is learned from only 15 demonstrations. Minimum jerk discourages abrupt changes in predicted translational acceleration; speed-curvature regularization instead couples movement speed to path geometry. These are candidate mechanisms for task completion, not safety guarantees. We compare the priors individually and jointly, neither prior, and generic smoothness, with 80 real-robot trials per setting pooled over four recorded condition classes. Joint and minimum-jerk-only settings each achieved 70/80 successes (87.5%), versus 69/80 (86.3%) for speed-curvature only, 66/80 (82.5%) for neither prior, and 67/80 (83.8%) for generic smoothness. Success rates and Wilson 95% confidence intervals are visualized for direct comparison. Joint regularization exceeded neither by 5.0 percentage points but provided no observed gain over minimum jerk alone. The results motivate minimum jerk as the simpler candidate for replication, without establishing synergy, biomechanical specificity, improved safety, or distribution-shift robustness.
• 中文摘要 本研究探讨了当仅通过15个演示得知扩散策略时，训练时间运动先验正则化是否能改善插入成功率。最小抖动抑制预测平移加速度的突然变化;速度-曲率正则化则将运动速度与路径几何耦合。这些是任务完成的候选机制，而非安全保证。我们分别比较先验和联合、无先验和一般平滑度，并汇总了80个真实机器人试验，涵盖四个已记录的条件类别。关节和最小抖动仅设置各自成功率为70/80（87.5%），而仅速度曲率为69/80（86.3%），未先验为66/80（82.5%），通用平滑度为67/80（83.8%）。可视化成功率和Wilson 95%置信区间以便直接比较。关节正则化均未超过5.0个百分点，但未单独较最小抖动获得显著增益。结果激励最小抖动作为更简单的复制候选，但未建立协同效应、生物力学特异性、安全性提升或分布转移稳健性。