Arxiv Papers of Today

生成时间: 2026-11-03 01:56:40 (UTC+8); Arxiv 发布时间: 2026-11-02 20:00 EST (2026-11-03 09:00 UTC+8)

今天共有 34 篇相关文章

Keyword: reinforcement learning

Li$^{2}$SAO: Lifting Light Singl-Rollout Asynchronous RL Optimization via Group-Relative Actor-Critic Bootstrapping

Li$^{2}$SAO：通过组-相对演员-批评者引导提升轻量单步部署异步强化学习优化

Authors: Zichen Bai, Yanli Wang, Yang Liu, Junhao Shi, Xueqin Liu, Jinguo Sun, Muye Li, Shaoqiang Zheng, Xuefei Diao, Zhuwei Zeng, Nan Duan, Peng Cao
Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.40726
Pdf link: https://arxiv.org/pdf/2610.40726
Abstract Asynchronous reinforcement learning (RL) is becoming a mainstream paradigm for post-training large language models (LLMs), as it improves training efficiency by updating the model as rollouts arrive. Recent work argues that group-wise sampling in GRPO does not naturally fit asynchronous RL training because rollout lengths follow a long-tailed distribution, and proposes Single-rollout Asynchronous Optimization (SAO) to improve stability and off-policy behavior, especially on agentic tasks. However, SAO relies on a closed-source value model obtained from scaling value pretraining. In this paper, we analyze GRPO and LiSAO, our lightweight reproduction of SAO without the pretrained value model, on single-turn reasoning and multi-turn agentic tasks. We find that GRPO outperforms LiSAO on reasoning tasks, while the reverse holds on agentic tasks. Since the value model of LiSAO is not yet well trained , LiSAO lags behind GRPO substantially in early training steps. To address this issue, we propose Li$^{2}$SAO, which Lifts Light Single-rollout Asynchronous Optimization with group-relative actor-critic bootstrapping: while the critic is cold, a group-relative advantage trains the actor and a group-centered return trains the critic, after which training switches to single-rollout GAE. Experiments show that Li$^{2}$SAO catches up with or surpasses GRPO in early training and further raises the later performance ceiling. Code is available at this https URL.
中文摘要 异步强化学习（RL）正成为大型语言模型（LLMs）训练后主流范式，因为它通过在部署到来时更新模型来提升训练效率。近期研究表明，GRPO中的分组抽样并不自然地适用于异步强化学习，因为滚动长度遵循长尾分布，并提出了单次部署异步优化（SAO）以提升稳定性和非策略行为，尤其是在代理任务中。然而，SAO依赖于通过价值预训练缩放获得的闭源价值模型。本文分析了GRPO和LiSAO（我们对SAO的轻量级再现，不含预训练值模型）在单回合推理和多回合代理任务上的表现。我们发现GRPO在推理任务中优于LiSAO，而在代理任务中则相反。由于LiSAO的价值模型尚未充分训练，LiSAO在早期训练步骤中明显落后于GRPO。为解决这一问题，我们提出了Li$^{2}$SAO，即采用组相对actor-critic自举法提升轻度单次部署异步优化：当批评者处于冷状态时，群体相对优势训练行为者，群体中心返回训练批评者，之后训练切换为单次推出GAE。实验显示Li$^{2}$SAO在早期训练中追赶或超过GRPO，进一步提升后期性能上限。代码可在此 https 网址获取。

CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents

CaPtain：为训练后代码即策略机器人代理打造的管道

Authors: Yijia Fan, Xilin He, Yilei Jiang, Dongming Wu, Jiaxu Wang, Xiangyu Yue
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2610.40799
Pdf link: https://arxiv.org/pdf/2610.40799
Abstract Across coding, web, and tool-use tasks, language-model agents are now routinely trained with the same recipe: supervised fine-tuning (SFT) to learn the interface, then multi-turn reinforcement learning (RL) to master it. Code-as-policy (CaP) casts robot manipulation in the same form, as the policy writes code against robot APIs and reads back the results, so the same recipe should apply. Yet existing CaP agents sidestep training: they either prompt closed frontier models or delegate manipulation to pre-trained vision-language-action models (VLAs) and high-level skills, while small open vision-language models (VLMs) fail outright. We present CaPtain, an open post-training pipeline that turns a small open VLM into a capable CaP agent, with no VLA, frontier model, or privileged state at test time. CaPtain trains and deploys the policy inside CaPtain-Harness, which exposes low-level perception, geometry, and guarded-motion primitives across simulation and real hardware. We first align the base model via SFT on synthetic trajectories generated with GPT-6 Astra, then apply multi-turn RL. Multi-turn RL must credit tens of agent requests, whose physical effects often surface turns later, from a single end-of-episode outcome. We therefore introduce a physical critic that reads privileged simulator state, never seen by the policy, and credits each request by how much it changes the predicted chance of success. Starting from Qwen3.5-4B (0% zero-shot), CaPtain-4B reaches 55.4% held-out LIBERO success after SFT and 73.8% after RL. On LIBERO-PRO, it reaches 78.5%, surpassing several frontier-model agents, and its RL gains persist across three perturbation suites (+17.6 to +21.2 points). Without target-domain training, it attains 83.3% on SimplerEnv WidowX, exceeds frontier models on CaP-Bench, and runs on a real robot after simulation-only adaptation. We release all code, data, and model weights.
中文摘要 在编码、网页和工具使用任务中，语言模型代理现在常规训练方式相同：监督微调（SFT）以学习界面，然后通过多回合强化学习（RL）掌握。代码即策略（CaP）以同样的形式执行机器人操作，策略针对机器人API编写代码并读取结果，因此应适用相同的流程。然而现有的CaP代理绕过训练：它们要么提示封闭前沿模型，要么将操作委托给预训练的视觉语言动作模型（VLA）和高级技能，而小型开放视觉语言语言模型（VLM）则完全失败。我们介绍CaPtain，一种开放的后期训练流水线，将一个小型开放VLM转变为一个具备能力的CaP代理，测试时没有VLA、前沿模型或特权状态。CaPtain在CaPtain-Harness中训练并部署策略，该机制在模拟和真实硬件中暴露了低级感知、几何和守护运动原语。我们首先通过SFT对准基于GPT-6 Astra生成的合成轨迹，然后应用多回合RL。多回合RL必须从单一集末结果中归功数十个代理请求，这些请求的物理效应通常在后续回合中显现。因此，我们引入一个物理批评器，读取策略未曾观察到的特权模拟器状态，并根据每个请求改变预测成功概率的程度来评分。从Qwen3.5-4B（0%零机会）开始，CaPtain-4B在SFT后达到55.4%的LIBERO成功率，RL后达到73.8%。在LIBERO-PRO上，其提升率达到78.5%，超过了多个前沿模型代理，其强化学习提升在三个扰动套件中持续存在（+17.6至+21.2点）。未进行目标域训练时，在SimplerEnv WidowX上达到83.3%，在CaP-Bench上超越前沿模型，并在仅模拟适应后运行于真实机器人。我们发布了所有代码、数据和模型权重。

FlapSim: A Composable MuJoCo Simulation Platform and Reduced-Order Aerodynamic Model Hierarchy for Flapping-Wing Robotics

FlapSim：一个可组合的MuJoCo仿真平台及降阶空气动力学模型层级，用于拍翼机器人

Authors: Mingshan He
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2610.40902
Pdf link: https://arxiv.org/pdf/2610.40902
Abstract Flapping-wing simulation must reconcile high-frequency wingbeat aerodynamics with the slower dynamics relevant to flight control and reinforcement learning. We present FlapSim, a modular open-source MuJoCo simulation platform that combines robot descriptions, wing geometry, and composable aerodynamic models in a deterministic simulation pipeline. Across $48$ held-out scenarios spanning frequency, gimbal, wind, frequency-step, gust, and cross-robot flat-plate regimes, we compare five reduced-order configurations operating at $1\,\text{ms}$ with a $50\,\mu\text{s}$ quasi-steady blade-element reference. The built-in hover cycle-averaged baseline achieves a $67.51\times$ CPU speedup but only $0.6593$ mean wrench fidelity. Combining multirate $21$-substep clock synchronization, three-dimensional linear/quadratic inflow and gust coupling, analytical blade-element geometry scaling, and phase-locked $1f\text{--}6f$ harmonics raises mean fidelity to $0.9646$ at a $65.19\times$ speedup in a five-block model. A streamlined four-block model without explicit step-transient lag states achieves $0.9662$ fidelity at a $67.98\times$ speedup. Controlled ablations show that stroke harmonics are particularly important during frequency transitions, improving held-out frequency-step fidelity by $+0.3176$. Transfer-function analysis explains this effect: fixed-window cycle averaging retains harmonic contributions when the wingbeat frequency changes, so within-cycle reconstruction also improves the filtered wrench. Once phase alignment and harmonic synthesis are included, explicit transient lag states provide no additional benefit in this benchmark. These results characterize the accuracy--computation tradeoff of the aerodynamic hierarchy and identify timing, inflow, geometry, and harmonic representation as central considerations for reduced-order flapping-wing simulation.
中文摘要 拍翼仿真必须调和高频翼拍空气动力学与飞行控制和强化学习相关的较慢动力学。我们介绍FlapSim，一个模块化开源MuJoCo仿真平台，结合了机器人描述、机翼几何和可组合空气动力模型，采用确定性仿真流程。在涵盖频率、万向节、风向、频率步长、阵风和跨机器人平板的48美元悬挂场景中，我们比较了五种以1美元\文本{ms}$运行的降序配置与50美元\mu\text{s}$准稳态叶片-元件参考。内置悬停周期平均基线实现了67.51美元/倍数的CPU加速，但平均扳手保真度仅为0.6593美元。结合多速率21美元子步进时钟同步、三维线性/二次流入与阵风耦合、解析叶片元素几何尺度以及相位锁定的$1f\text{--}6f$谐波，在五块模型中平均保真度提升至$0.9646美元，速度提升$65.19\倍倍。一个流线型的四块模型（无明显的阶跃瞬态滞后状态）可实现$0.9662$的精度，加速$67.98\倍数。受控消融显示，脉冲谐波在频率转换中尤为重要，使保留的频步保真度提升$+0.3176$。传递函数分析解释了这一效应：固定窗口周期平均在翼拍频率变化时仍保留谐波贡献，因此周期内重建也改善了滤波扳手。一旦纳入相位对齐和谐波合成，显式瞬态滞后状态在本基准测试中不再带来额外益处。这些结果描述了空气动力学层级的准确性与计算权衡，并将时序、进气、几何和谐波表示作为降阶拍振翼模拟的核心考虑因素。

CoFiBreak: A Coarse-to-Fine Diversity Reward for Effective Multi-Turn Jailbreak

CoFiBreak：从粗到微的多样性奖励，奖励有效的多回合越狱

Authors: Jialin Song, Xiaodong Liu, Weiwei Yang, Mingqian Feng, Chengquan Guo, Qiyao Ma, Carolyn Buractaon, Anson Ho, Wuyang Chen, Christopher White
Subjects: Subjects: Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2610.40926
Pdf link: https://arxiv.org/pdf/2610.40926
Abstract Automated red teaming trains an attacker large language model (LLM) to elicit harmful responses from a victim LLM, but reinforcement learning can collapse attackers onto a narrow set of prompts when attack success is the only reward. Such a collapse of attack diversity can bias safety evaluation toward repeatedly testing near-duplicate attack patterns rather than the underlying harmful intents. Existing methods often trade off attack success or inference cost for greater content or strategy diversity. We propose CoFiBreak, a coarse-to-fine diversity reward for Group Relative Policy Optimization (GRPO) that improves attack diversity while preserving strong attack effectiveness and low inference cost. The coarse reward broadens which strategies the attacker uses by encouraging strategies underrepresented within each GRPO group, while the fine reward broadens how those strategies are realized by discouraging reuse of the same personas, framings, and output formats after harmful-intent information is removed from prompt embeddings. Diversity rewards apply only to attacks that already exceed an effectiveness threshold, so diversity cannot be gained through ineffective attacks. Since both rewards are computed only during training, CoFiBreak also retains inference efficiency. Trained with Llama3.1-8B-Instruct as the victim, CoFiBreak discovers 1.6-2.0x as many distinct successful fine-grained strategies as the strongest baseline across frontier victims on HarmBench. It reaches 98.8% ASR@10 (attack success rate with 10 generations per intent) on AdvBench with the highest diversity score among all compared methods, which is 2.1x that of the strongest-attacking baseline. The resulting attacks transfer to unseen frontier LLMs, reaching 71.0% ASR@10 on AdvBench with the unseen victim Claude-Opus-4.6. The code will be released.
中文摘要 自动化红团队训练攻击者大型语言模型（LLM）以引发受害者LLM的有害反应，但当攻击成功是唯一奖励时，强化学习可以将攻击者压缩到有限的提示集合上。这种攻击多样性的崩溃会使安全性评估倾向于反复测试近似重复的攻击模式，而非潜在的有害意图。现有方法常以攻击成功率或推理成本换取更高的内容或策略多样性。我们提出了CoFiBreak，这是一种从粗到细的多样性奖励，用于Group Relative Policy Optimization（GRPO），它在保持强大攻击有效性和低推理成本的同时，提升攻击多样性。粗奖励通过鼓励每个GRPO组中代表性不足的策略，拓宽攻击者所采用的策略，而细奖励则通过阻止在提示嵌入中移除有害意图信息后重复使用相同角色、框架和输出格式，从而拓宽这些策略的实现方式。多样性奖励仅适用于已超过有效阈值的攻击，因此无法通过无效攻击获得多样性。由于这两种奖励均仅在训练中计算，CoFiBreak同样保持了推理效率。CoFiBreak以Llama3.1-8B-Instruct作为受害者进行训练，发现的成功细粒度策略数量是HarmBench上最强基线的1.6-2.0倍。在AdvBench上，其ASR@10达到98.8%（每次意图10代攻击成功率），在所有比较方法中多样性得分最高，是最强攻击基线的2.1倍。由此产生的攻击会转移到未公开的前沿大型语言模型，在AdvBench上ASR@10达到71.0%，而未被发现的受害者是Claude-Opus-4.6。该代码将被发布。

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning

关于大语言模型强化学习中训练-推理不匹配的起源与演变

Authors: Yu Li, Yufa Zhou, Tian Lan
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.40945
Pdf link: https://arxiv.org/pdf/2610.40945
Abstract On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the \textbf{training-inference mismatch} between the two makes the training data off-policy. The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. \textit{We ask which property of a mismatch source decides the correction it needs.} Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into a \textit{numerical discrepancy} of the engine and a \textit{policy lag} of the trainer, and we follow both through training. The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates: the bias it leaves enters the next update, which enlarges the next lag and closes a feedback loop. Importance weighting cuts this loop only while the lag ratio stays concentrated, and on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half. From this account we derive the \textbf{discrepancy-lag correction} (DLC), which centers the numerical discrepancy, clips the lag ratio against the sampler's weights and resynchronizes the sampler when this effective sample size falls below one half. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses that the per-token corrections leave and that a KL measured on the sampler's prefixes does not detect.
中文摘要 针对大型语言模型的策略强化学习假设训练响应是从当前策略中抽样的。实际上，推理引擎采样响应，另一个训练器计算梯度，而两者之间的\textbf{训练-推理不匹配}使训练数据偏离策略。这种不匹配有两个来源：引擎的数值和异步采样的陈旧权重，而在一个来源下稳定训练的修正可能在另一个来源下失败。\textit{我们询问不匹配源的哪个属性决定了需要的纠正。}在采样器权重下评估训练器，将每个采样令牌错误精确分为引擎的 \textit{数值差异}和训练器的 \textit{policy lag}，我们通过训练跟踪这两者。数值差异大致保持不变，因此会使更新产生几乎固定的偏置，并将分数置中于每个前缀处即可消除其前导偏差。策略迟滞由训练者自身的更新产生：其留下的偏置进入下一次更新，扩大下一次延迟并闭合反馈循环。重要性加权仅在滞后比率保持集中时切断该循环，在Qwen3-1.7B中，有效样本量低于一半时失效。由此推导出\textbf{差异-滞后修正}（DLC），该方法居中数值差异，将延迟比与采样器权重相符，并在有效样本量低于一半时重新同步采样器。在陈旧采样器下，DLC使得在Qwen3-0.6B和Qwen3-1.7B上两个独立的策略内训练之间，与策略训练的KL偏差保持在KL的两倍范围内，其重新同步防止了每个令牌修正留下的崩溃，而采样器前缀测量的KL无法检测到。

Steering Vectors as a Training Signal in LLM Post-Training

引导向量作为大型语言模型后训练信号的训练信号

Authors: Tiejin Chen, Maunil Vyas, Huaiyuan Yao, Hua Wei
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.40998
Pdf link: https://arxiv.org/pdf/2610.40998
Abstract Steering vectors (SVs) are residual-stream directions that shift the behavior of LLMs when added to hidden states at inference time. While prior work has mainly used SVs as post-hoc controls, we ask whether the same directions can also guide parameter updates during post-training. We study this question across supervised learning, including SFT and distillation, and reinforcement learning with verifiable rewards (RLVR). Our framework considers two intervention regimes: a hidden-state hook during teacher-forced supervised training, and an advantage-repair mechanism on wrong rollouts in GRPO. For each regime, we analyze three design axes, which are the steering direction, its placement, and the operation used to combine it with training. In supervised settings, SV injection improves over plain SFT and distillation, with gains scaling with the off-policy gap and reaching up to a $12\%$ improvement when distilling from thinking-mode teachers. These gains hold across three random seeds. In RL settings, applying a correctness-contrast direction as advantage repair on wrong rollouts, including otherwise zero-gradient all-wrong groups, recovers a learning signal that transfers to held-out math benchmarks and acts as a composable auxiliary signal that adds to existing zero-variance remedies such as entropy-aware advantage shaping. We further identify three failure boundaries. In detail, code generation can break the direction, cross-family layer transfer can break the placement, and inference-time application on an SV-trained checkpoint can break the intervention regime. Together, these results establish steering vectors as a conditional but effective training signal for LLM post-training. Our code is available here.
中文摘要 引导向量（SV）是残余流方向，在推理时将LLM添加到隐藏状态时改变其行为。虽然此前工作主要将SV作为事后控制，但我们探讨了相同的方向是否也能指导训练后参数的更新。我们在监督学习（包括SFT和蒸馏）以及带可验证奖励的强化学习（RLVR）中研究该问题。我们的框架考虑了两种干预机制：教师强制监督培训中的隐藏状态钩子，以及GRPO中错误推广的优势修复机制。对于每个模式，我们分析三个设计轴：引导方向、其位置以及将其与训练结合的操作。在监督环境中，SV注入优于纯SFT和蒸馏，收益随非策略差距增长，从思考模式教师蒸馏时提升高达12%美元。这些收益在三种随机种子中均成立。在强化学习环境中，对错误的展开（包括本应零梯度的全错组）应用正确性-对比度方向作为优势修复，恢复学习信号，转移至保留的数学基准，并作为可组合辅助信号，补充现有零方差补救方法如熵感知优势塑形。我们进一步识别了三个失败边界。具体来说，代码生成可能破坏方向，跨族层传输可能破坏放置，推理时间应用于SV训练检查点可能破坏干预机制。这些结果共同建立了引导向量作为LLM训练后有条件但有效的训练信号。我们的代码可在此处获取。

Density-Guided Bootstrap Boundaries for Latent World-Model Reinforcement Learning

密度引导引导边界用于潜在世界模型强化学习

Authors: Shashank Gupta, Pilhwa Lee
Subjects: Subjects: Machine Learning (cs.LG); Machine Learning (stat.ML)
Arxiv link: https://arxiv.org/abs/2610.41022
Pdf link: https://arxiv.org/pdf/2610.41022
Abstract Latent world models allow reinforcement learning agents to learn policies from imagined trajectories, yet determining the optimal depth of model reliance remains a fundamental challenge. Fixed imagination horizons apply uniform prediction depths regardless of trajectory reliability, exposing target values to compounding model error or over-relying on critic approximations. We introduce density-guided bootstrap boundaries, a trajectory-dependent mechanism that adapts the depth of imagined returns using a score-informed relative density proxy. The proxy accumulates the estimated divergence of a score-corrected auxiliary field formed by a feature displacement network and a noise-conditioned score estimator along imagined paths. We propose two gating mechanisms: a hard boundary that halts return expansion at a discrete state, and a soft boundary that continuously shifts the return recursion toward critic bootstrapping. Both variants preserve boundary value estimates and apply corresponding survival weights to actor and critic losses. Formulated independently of specific dynamics models and instantiated within Dreamer V3, we prove that the resulting gated Bellman operator is a strict contraction mapping and that early bootstrapping monotonically reduces conditional target variance. We clarify the theoretical assumptions distinguishing our expansion proxy from exact path likelihoods. Across continuous-control tasks and Atari Pong, empirical evaluations show that dynamic boundary selection enhances sample efficiency and late-training performance over fixed-horizon and auxiliary-enabled baselines.
中文摘要 潜在世界模型允许强化学习代理从想象轨迹中学习策略，但确定模型依赖的最优深度仍是一个根本挑战。固定想象视界无论轨迹可靠性如何，都适用统一的预测深度，使目标值面临叠加的模型误差或过度依赖批判近似。我们引入了密度引导引导边界，这是一种轨迹依赖机制，利用分数知情的相对密度代理调整想象收益的深度。代理积累了由特征位移网络和噪声条件评分估计估计的分数修正辅助场沿想象路径形成的发散度。我们提出了两种门槛机制：硬边界在离散状态停止返回扩展，软边界持续向批判者自助移动。两种变体都保持边界值估计，并将相应的存活权重应用于actor和critic损失。该模型独立于特定动力学模型表述，并在Dreamer V3中实现，我们证明所得门控Bellman算子是严格的收缩映射，早期自助法单调地降低条件目标方差。我们澄清了区分扩展代理与精确路径似然的理论假设。在连续控制任务和Atari Pong中，实证评估表明动态边界选择提升了样本效率和固定视界和辅助基线的后期训练性能。

Before We Adapt: The Zero-Shot Generalization Bottleneck in Continual World Models

在我们适应之前：连续世界模型中的零射击推广瓶颈

Authors: Amir Mesbah, Di Wu
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.41041
Pdf link: https://arxiv.org/pdf/2610.41041
Abstract Continual reinforcement learning (CRL) requires agents to operate under environmental changes while reusing prior knowledge whenever possible. We argue that model-based reinforcement learning provides a promising framework for this setting because learned world models (WMs) can, in principle, support change detection, knowledge reuse, adaptation, and acquisition of new knowledge. However, such benefits depend heavily on whether the learned model can generalize beyond the contexts in which it was trained. In this work, we propose to formulate this problem with a contextual Markov decision process in which context is decomposed into behavioral and physical components. This view organizes continual WM learning into reuse, adaptation, and acquisition of new knowledge. As a first step, we study direct reuse when behavioral rules remain unchanged. Using Dyna-Q and TD-MPC in controlled FourRooms tasks, we find that zero-shot reuse can degrade substantially under physical-context changes and that this behavior depends strongly on the observation representation used by the WM. Additional ablation studies and latent-space diagnostics suggest that learned representations do not automatically provide the invariances needed for reuse. These results identify representation generalization as an important bottleneck for reliable world-model reuse in CRL.
中文摘要 持续强化学习（CRL）要求代理在环境变化下操作，同时尽可能重用已有知识。我们认为基于模型的强化学习为该环境提供了一个有前景的框架，因为学习世界模型（WM）原则上可以支持变化检测、知识再利用、适应和新知识的获取。然而，这些益处很大程度上取决于所学模型是否能够超越其训练环境的推广。在本研究中，我们提出用一种情境马尔可夫决策过程来表述该问题，其中情境被分解为行为和物理成分。该观点将持续的强化学习组织为新知识的再利用、适应和获取。作为第一步，我们研究行为规则保持不变时的直接重用。在受控的FourRooms任务中使用Dyna-Q和TD-MPC，我们发现零样本重复使用在物理上下文变化下会显著退化，且这种行为强烈依赖于WM所使用的观察表征。额外的消融研究和潜空间诊断表明，学习到的表征并不自动提供重用所需的不变性。这些结果指出，表征泛化是CRL中可靠世界模型重用的重要瓶颈。

Communication-Efficient Private Federated Reinforcement Learning with Post-Quantum Authentication for Vehicular Networks

基于后量子认证的通信高效私有联合强化学习，适用于车载网络

Authors: Hojat Allah Salehi, Devrim Aras, Md Jueal Mia, M. Hadi Amini, Imtiaz Karim, Mashrur Chowdhury, Bhavani Thuraisingham
Subjects: Subjects: Machine Learning (cs.LG); Cryptography and Security (cs.CR)
Arxiv link: https://arxiv.org/abs/2610.41055
Pdf link: https://arxiv.org/pdf/2610.41055
Abstract Federated reinforcement learning supports cooperative vehicular routing without centralizing trajectory data, but model updates may reveal private information and incur communication overhead over intermittent vehicle-to-infrastructure links, particularly when post-quantum authentication is required. We study correlated private quantization, using complementary shared randomness between paired vehicles to reduce aggregation mean squared error while preserving the marginal parameter-level local differential privacy and one-bit communication cost of independent quantization. A tunable extension mixes shared and independent randomness to bound privacy leakage given paired outputs and shared-randomness side information. For this extension, Rényi differential privacy accounting establishes vector-level local differential privacy for one model upload even when pairings are revealed. Under matched vector-level privacy and identical local and global parameters, the extension yields 16.5% lower analytical aggregation mean squared error than independent quantization. We evaluate cooperative routing by coupling traffic simulation with packet-level simulation of post-quantum-authenticated vehicle-to-infrastructure communication. Compared to full-precision updates, one-bit uploads reduce authenticated uplink size by 92.4% and increase mean verified updates aggregated per round from 19.6 to 26.5. Under mobility-constrained communication at a parameter-level privacy budget of 1, correlated private quantization reduces mean travel time by 14.3% relative to independent quantization at the same budget and communication cost. At a parameter-level privacy budget of 0.3, a vehicle-origin inference attack trained on independent simulations reaches 30% accuracy with correlated private quantization and 22% with independent quantization after 100 rounds, versus 91% without privacy and 10% for random guessing.
中文摘要 联邦强化学习支持车辆协作路由，无需集中轨迹数据，但模型更新可能暴露私密信息，并在车辆与基础设施间歇性链路上产生通信开销，尤其是在需要后量子认证时。我们研究相关私有量子化，利用配对车辆间互补的共享随机性，降低聚合均方误差，同时保持边际参数级局部微分隐私和独立量化的一比特通信成本。可调扩展将共享随机与独立随机性结合，结合结合对对输出和共享随机侧信息的隐私泄漏。对于该扩展，Rényi差分隐私核算为单个模型上传建立了向量级局部差分隐私，即使配对已揭示。在匹配向量级隐私和相同局部和全局参数下，该扩展比独立量化降低分析聚合均方误差16.5%。我们通过将交通模拟与分组级的量子认证后车辆与基础设施通信模拟结合，评估协作路由。与全精度更新相比，一比特上传可将认证上行链路大小减少92.4%，并使每轮平均验证更新从19.6增加到26.5。在移动性受限、参数级隐私预算为1的通信条件下，相关私有量化在相同预算和通信成本下，相较于独立量化减少平均旅行时间14.3%。在参数级隐私预算0.3下，在独立模拟下训练的车辆起源推断攻击，配合相关私有量化后准确率达30%，独立量化后为22%，而无隐私时为91%，随机猜测为10%。

bf16 Rounding Silently Discards Most RL Policy Updates: A Hidden Training-Inference Mismatch and an Unbiased One-Line Fix

bf16 四舍五入无声地丢弃了大部分强化学习政策更新：一个隐藏的训练-推断不匹配与一个公正的单行修正

Authors: Andrew Gao, William Gao
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.41073
Pdf link: https://arxiv.org/pdf/2610.41073
Abstract Reinforcement learning (RL) post-training of language models keeps fp32 master weights in the optimizer and serves rollouts from a bf16 copy produced by round-to-nearest conversion. At the learning rates in common use, most per-step updates are smaller than half a bf16 spacing and the conversion discards them: on Qwen3-1.7B only 0.4-0.5% of served weights change per step, and the policy that generates the training data lags the optimizer by 50-150 steps. Existing mismatch corrections cannot see this lag because they compare two copies downstream of the same rounding. We propose Fair Cast: export the master with an unbiased rounding rule (stochastic rounding or error feedback) and compute the training loss on the served copy. The change is one line at the export call, adds no hyperparameters and costs 22-38 ms per export of 1.7B parameters. Across 25 matched seed-pairs on Qwen3-0.6B to 4B with AdamW and Muon, Fair Cast is ahead on 18, tied on 6 and behind on 1, with a mean gain of 6.1 points; on zebra puzzles it reaches 0.57 +/- 0.16 over four runs where round-to-nearest collapses to 0.33 +/- 0.10. It also makes the saved fp32 checkpoint agree with the served model, and it exposes the implicit lag of round-to-nearest as a regularizer that can be restored explicitly when a task needs it.
中文摘要 强化学习（RL）在语言模型的后训练中将 fp32 主权重保留在优化器中，并从通过四轮到最近转换产生的 bf16 副本进行展开。在常用的学习率下，大多数每步更新都小于半个 bf16 间距，转换时会丢弃这些权重：在 Qwen3-1.7B 中，每步仅有 0.4-0.5% 的受服务权重变化，生成训练数据的策略比优化器落后 50-150 步。现有的不匹配修正无法察觉这种滞后，因为它们比较了同一四舍五入的下游两个副本。我们提出公平选址：用无偏的四舍五入规则（随机四舍五入或误差反馈）导出主文件，并计算所服务副本的训练损失。在导出调用时只需一行更改，不添加超参数，每次导出 1.7 亿参数的成本为 22-38 毫秒。在Qwen3-0.6B对4B（AdamW和Muon）的25对匹配种子中，公平投法领先18分，6分并列，落后1分，平均增益6.1分;斑马谜题四次运行后达到0.57 +/- 0.16，四轮到最近崩溃为0.33 +/--0.10。它还使保存的fp32检查点与所提供模型一致，并揭示了四周到最近的隐含延迟，作为一个可明确恢复的正则化器，当任务需要时可被明确恢复。

AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems

AIFaultBench：AI软件系统中真实世界故障的可重现基准测试

Authors: Mehil B Shah, Mohammad Masudur Rahman, Foutse Khomh
Subjects: Subjects: Software Engineering (cs.SE)
Arxiv link: https://arxiv.org/abs/2610.41094
Pdf link: https://arxiv.org/pdf/2610.41094
Abstract AI-based software systems remain difficult to debug because their failures can involve interactions among code, models, data, dependencies, hardware, and execution environments. Existing benchmarks cover specific classes of AI systems, while concerns about fault realism and reproducibility highlight the need for documented execution conditions and evidence. We present AIFaultBench, a benchmark of 770 real-world bugs from 105 AI-based software repositories across 76 organizations. It covers six classes of AI systems: machine learning, deep learning, reinforcement learning, systems based on large language models, agentic AI, and AI tooling and infrastructure. Each bug record connects the original report and affected codebase to structured metadata, environment setup scripts, a reproduction trigger, and investigation evidence. We reproduced 652 bugs (84.7%) under the recorded execution conditions and documented the outcomes of the remaining 118 attempts. Through its broad coverage, documented reproduction evidence, and common structure, AIFaultBench supports empirical studies of AI faults, debugging tool evaluation, practical failure investigation, and teaching with real-world examples.
中文摘要 基于人工智能的软件系统仍然难以调试，因为它们的故障可能涉及代码、模型、数据、依赖关系、硬件和执行环境之间的交互。现有基准测试涵盖特定类别的人工智能系统，而对故障真实性和可重复性的担忧凸显了执行条件和证据文档的必要性。我们展示了AIFaultBench，这是一个涵盖76个组织中105个基于人工智能软件仓库的770个真实世界缺陷的基准测试。它涵盖了六类人工智能系统：机器学习、深度学习、强化学习、基于大型语言模型的系统、代理人工智能以及人工智能工具和基础设施。每个错误记录将原始报告和受影响的代码库与结构化元数据、环境设置脚本、重现触发器和调查证据连接起来。我们在记录的执行条件下复现了652个错误（84.7%），并记录了其余118次尝试的结果。通过其广泛的覆盖、有据可查的复现证据和统一的结构，AIFaultBench 支持对 AI 缺陷的实证研究、调试工具评估、实际故障调查以及以真实案例进行教学。

TalentRL: Strategy-Guided Reinforcement Learning for Hierarchical Skill Routing

TalentRL：层级技能路由的策略引导强化学习

Authors: Huancheng Chen, Xiaodi Sun, Zhaoqiong Huang, Michelle Chang, Jingwen Lu
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.41143
Pdf link: https://arxiv.org/pdf/2610.41143
Abstract Natural-language skills enable large language model (LLM) agents to reuse procedures across tasks, yet most systems retrieve individual skills from a flat SkillBank without explicitly modeling the high-level strategy that governs their selection and use. We introduce TalentRL, a strategy-guided reinforcement learning framework for hierarchical routing over procedural memory. Coarse-grained talents represent recurring strategies for skill use, while fine-grained skills provide executable procedures. For each task, the policy model selects a talent that guides execution and softly prioritizes its associated skills during similarity-based retrieval. However, since task outcomes depend jointly on talent selection, retrieved skill context, and downstream execution, terminal rewards alone cannot isolate the contribution of the routing decision. TalentRL therefore trains talent routing with reward contrasts between rollouts of the same task under matched interventions on the talent-conditioned context, and optimizes downstream execution with group-relative advantages. Across seven open-domain and multi-hop question-answering benchmarks, talent-aware supervised fine-tuning improves average success rate by 7.53%, 6.71%, and 9.21% over the corresponding zero-shot baselines for Qwen2.5-7B, Qwen3-8B, and Llama-3.1-8B, respectively. Reinforcement learning further improves these SFT checkpoints by 7.58% to 9.32%, and TalentRL achieves the highest average success rate among all compared methods on all three backbones. On the agentic benchmarks ALFWorld and WebShop, TalentRL achieves the highest ALFWorld success rate (99.3%) and the highest WebShop score (90.7) among all compared methods.
中文摘要 自然语言技能使大型语言模型（LLM）智能体能够在任务间重复使用程序，但大多数系统仅从扁平的技能库中检索单个技能，却未显式建模指导其选择和使用的高级策略。我们介绍了TalentRL，一种基于程序记忆的策略引导强化学习框架，用于层级路由。粗粒度天赋代表技能使用的重复策略，而细粒度技能则提供可执行过程。对于每个任务，策略模型选择一个引导执行的天赋，并在基于相似性检索时温和地优先排序其相关技能。然而，由于任务结果共同依赖于天赋选择、检索技能上下文和下游执行，单靠终端奖励无法隔离路由决策的贡献。因此，TalentRL在人才条件情境下，通过匹配干预下同一任务的推广对比来训练人才路由，并以群体相对优势优化下游执行。在七个开放域和多跳问答基准测试中，Talent-aware 监督微调分别将Qwen2.5-7B、Qwen3-8B和Llama-3.1-8B对应零射基线的平均成功率提升了7.53%、6.71%和9.21%。强化学习进一步将这些SFT检查点提升7.58%，达到9.32%，TalentRL在三种骨干链上均获得最高的平均成功率。在代理基准测试ALFWorld和WebShop中，TalentRL在所有比较方法中实现了最高的ALFWorld成功率（99.3%）和最高的WebShop得分（90.7）。

RTL-Interact: Training EDA-Interactive Models for Multi-Task RTL Design

RTL-Interact：训练EDA交互模型以实现多任务RTL设计

Authors: Wenxuan Shi, Yaoyu Zhu, Di Huang, Chongxiao Li, Xiaoyun Zhang, Muxin Song, Yutong Wu, Ziyuan Nan, Pengwei Jin, Tian Zhi, Xishan Zhang, Rui Zhang, Xing Hu, Zidong Du, Qi Guo, Yunji Chen
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.41225
Pdf link: https://arxiv.org/pdf/2610.41225
Abstract Large language models offer a promising route to register-transfer-level (RTL) design, but reliable implementations require checking and revision based on electronic design automation (EDA) tools' feedback. Existing RTL-specialized models often lack learned interaction with such EDA tools or rely on separate generation and repair components. We introduce RTL-Interact, a multi-task post-training framework that trains a single RTL-specialized model to generate RTL, construct task-appropriate EDA checks, and revise implementations using tool feedback. The framework combines automatically constructed multi-task demonstrations with staged supervision from chain-of-thought to tool-integrated reasoning. We further introduce Pattern-Aware Policy Optimization (PAPO) for multi-task reinforcement learning. PAPO augments Group Relative Policy Optimization with correctness-gated rewards that favor concise successful checking while rewarding effective revision, and uses task grouping to preserve coverage across objectives. On CVDP, RTL-Interact-32B achieves 48.7% weighted Pass@1 and 68.9% Pass@5, surpassing its teacher model, DeepSeek-V3.2, while RTL-Interact-8B achieves 44.9% weighted Pass@1. Additionally, we show that PAPO can reduce average tool calls by 63.8% while improving Pass@1. Both models also outperform all RTL-specialized baselines on VerilogEval v2 and RTLLM v2, demonstrating the effectiveness of our method.
中文摘要 大型语言模型为寄存器转移级（RTL）设计提供了一条有前景的路径，但可靠的实现需要基于电子设计自动化（EDA）工具反馈进行检查和修订。现有的RTL专用模型通常缺乏与这些EDA工具的学习交互，或依赖于独立的生成和修复组件。我们引入了RTL-Interact，这是一个多任务后期培训框架，训练单个RTL专用模型生成RTL，构建任务适用的EDA检查，并利用工具反馈修正实现。该框架结合了自动构建的多任务演示与从思维链到工具集成推理的分阶段监督。我们还进一步引入了模式感知策略优化（PAPO）用于多任务强化学习。PAPO通过正确性门槛奖励来增强群体相对策略优化，有利于简洁的成功检查，同时奖励有效的修订，并通过任务分组保持目标覆盖。在CVDP上，RTL-Interact-32B实现了48.7%的加权Pass@1和68.9%的Pass@5，超过了其教师模型DeepSeek-V3.2，而RTL-Interact-8B则实现了44.9%的加权Pass@1。此外，我们表明PAPO能够将平均工具调用降低63.8%，同时提升Pass@1。这两个模型在VerilogEval v2和RTLLM v2上都优于所有RTL专用基线，证明了我们方法的有效性。

rl_rob: A transparent, from-scratch Python library for teaching reinforcement learning from tabular methods to deep policy gradients

rl_rob：一个透明、从零开始的Python库，用于教授从表格方法到深度策略梯度的强化学习

Authors: Abdulrahman Altahhan
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.41303
Pdf link: https://arxiv.org/pdf/2610.41303
Abstract rl_rob is an open-source Python library for teaching reinforcement learning. It implements the arc of a modern course, from tabular temporal-difference learning through linear function approximation to deep value-based and policy-gradient methods, with every algorithm written from first principles so that its update rule is visible in a few lines of code. All agents share one interface, so learners can compare methods directly across stages and trace a deep reinforcement learning update from the temporal-difference error to the weight change. The library supports teaching and assessment in which students implement, modify and explain update rules, rather than only produce working code.
中文摘要 rl_rob 是一个开源的 Python 强化学习教学库。它实现了现代课程的弧线，从表格形式的时间差分学习到线性函数近似，再到深度基于价值和策略梯度的方法，每个算法都从基本原理编写，使其更新规则在几行代码中可见。所有代理共享一个接口，学习者可以直接跨阶段比较方法，追踪从时间差误到权重变化的深度强化学习更新。该库支持教学和评估，学生实现、修改和解释更新规则，而不仅仅是生成可用的代码。

HandMimicX: Learning Human-Like Dexterous Manipulation via an Object-Centric Policy from Egocentric Videos

HandMimicX：通过以对象为中心的策略，从自我中心的视频中学习类人灵巧的操作

Authors: Zihan Gao, Yufei Liu, Yuhong Zhang, Zhen Yu, Xinhao Huang, Zhinan You, Kai W. Ho, Zipeng Ye, Peng Jiao, Haoqian Wang
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2610.41336
Pdf link: https://arxiv.org/pdf/2610.41336
Abstract Learning dexterous manipulation from human videos remains an open challenge, particularly in the context of transferring a single demonstration into a spatially generalizable robotic skill and maintaining physically plausible human-like interactions. Unlike prior works that emphasize hand-motion imitation, we introduce an object-centric framework for acquiring dexterous skills from egocentric human RGB-D videos. Within our framework, we formulate the reinforcement learning (RL) objective in terms of object motion to decouple the underlying skill from the specific hand trajectory and impose constraints on hand-object interaction states to promote more anatomically natural hand configurations. Extensive experiments in simulation and on real robots demonstrate that our method enables efficient learning from human videos and generalizable skill transfer across novel spatial configurations.
中文摘要 从人类视频中学习灵巧操作仍是一个开放的挑战，尤其是在将单一演示转化为空间上可推广的机器人技能并维持物理上合理的类人互动时。与以往强调手部动作模仿的研究不同，我们引入了一个以对象为中心的框架，用于从自我中心的人类RGB-D视频中获得灵巧技能。在我们的框架内，我们以物体运动为基础，制定强化学习（RL）目标，以将底层技能与特定手部轨迹解耦，并对手与物体的交互状态施加约束，以促进更自然的手部配置。在模拟和真实机器人上的大量实验表明，我们的方法能够高效地从人类视频中学习，并实现跨新空间配置的通用技能转移。

Joint-Health-Aware Reinforcement Learning for Robust Robot Control Under Actuator Degradation

关节健康感知强化学习，在执行器退化下实现机器人稳健控制

Authors: Haifa Zhang, Yijing Wang, Zhiqiang Zuo
Subjects: Subjects: Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
Arxiv link: https://arxiv.org/abs/2610.41348
Pdf link: https://arxiv.org/pdf/2610.41348
Abstract Robots deployed in unstructured environments are susceptible to actuator-authority degradation, which reduces the effective control authority of individual actuators and can cause reinforcement learning policies trained under nominal dynamics to fail. Existing robustness approaches rely on discrete fault labels, broad domain randomization, or global latent variables, often without explicitly modeling continuous per-joint residual actuation capability. We propose a joint-health-aware reinforcement learning framework for robust robot control under simulated actuator degradation. Our framework encodes actuator condition in a continuous joint-health vector and models degradation as multiplicative attenuation of executed actions. A dedicated HealthNet estimator infers per-joint health from recent observations and actions, allowing the policy to redistribute control effort across healthy and degraded joints. We evaluate the method on Walker, Go2, Cassie, and Panda-based ManiSkill manipulation tasks under diverse degradation masks and severity levels. The method outperforms nominal and degradation-trained no-health baselines on the main evaluated metrics. Relative to the nominal baseline, task-level locomotion returns improve by up to 250.1\% and manipulation success rates by up to 15.2 percentage points; in the high-degradation Cassie group, return increases from 3.94 to 40.50 (+928.0\%). Ablations support the contribution of per-joint health estimation, action-degradation modeling, and health-conditioned control.
中文摘要 部署在非结构化环境中的机器人易受执行器权威性劣化影响，这会降低单个执行器的有效控制权威，并可能导致在名义动力学下训练的强化学习策略失效。现有的鲁棒性方法依赖离散故障标签、广域随机化或全局潜在变量，通常未明确建模连续的每关节残余驱动能力。我们提出了一种关节健康感知强化学习框架，用于模拟执行器退化下的稳健机器人控制。我们的框架将执行器状态编码为连续的关节健康向量，并将退化建模为执行动作的乘法衰减。专用的HealthNet估计器根据近期观察和动作推断每个关节健康状况，使策略能够在健康和退化关节之间重新分配控制工作。我们在不同退化掩膜和严重程度水平下的Walker、Go2、Cassie和Panda基于ManiSkill操作任务中评估该方法。该方法在主要评估指标上优于名义和退化训练的无健康基线。相较名义基线，任务级运动回报提升高达250.1%，操作成功率提升高达15.2个百分点;在高降解Cassie组中，返回率从3.94提升至40.50（+928.0%）。消融支持了单关节健康估计、动作-退化建模和健康条件控制的贡献。

From Representation to Readout: Mechanistic Analysis of Membership Signals in Language Models

从表征到读出：语言模型中成员信号的机制分析

Authors: Wenxi Dai, Jingyu Hu, Yanting Yang, Mengnan Du
Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.41445
Pdf link: https://arxiv.org/pdf/2610.41445
Abstract Membership inference aims to determine whether a particular example was included in a model's training data. Such membership signals are often broadly described as memorization, while the underlying mechanisms can involve both how information is represented internally and how it is accessed by the model to produce behaviors. We therefore present a mechanistic study of membership signals in large language models (LLMs). We introduce Decodable Information Score, Used Information Score, and Readout Accessibility Gap (DUG), a set of complementary measures that separately quantify membership information decodable from internal representations and the extent to which it is exposed through the model's readout. Across supervised fine-tuning (SFT) and reinforcement learning (RL)-based membership inference methods, membership-specific effects appear in both the amount of decodable information and readout under SFT, while under RL-based membership inference methods they appear primarily in readout. Activation patching further localizes this membership-specific effect to intermediate layers. To examine whether these findings extend to a more general RL setting, we further study reinforcement learning with verifiable rewards (RLVR) contamination, where the training reward is independent of the reference solution. In this setting, we find that behavioral changes can emerge without a corresponding increase in likelihood-based membership inference signals.
中文摘要 成员推断旨在确定某个具体示例是否被纳入模型的训练数据中。此类成员信号通常被广泛描述为记忆，而其底层机制既涉及信息的内部表示方式，也涉及模型如何访问以产生行为。因此，我们提出了大型语言模型（LLMs）中成员信号的机制性研究。我们引入了可解码信息评分、使用信息评分和读出可及性差距（DUG），这是一组互补的测量指标，分别量化可解码成员信息与内部表征及其通过模型读出的暴露程度。在基于监督微调（SFT）和强化学习（RL）的成员推断方法中，成员特异效应在SFT下的可解码信息量和读出中均有体现，而基于强化学习的成员推断方法则主要出现在读出中。激活贴片进一步将这种成员特异效应局限于中间层。为了考察这些发现是否适用于更一般的强化学习环境，我们进一步研究了带有可验证奖励（RLVR）污染的强化学习，其中训练奖励独立于参考解。在该环境中，我们发现行为变化可能在不伴随似然基础成员推断信号增加的情况下出现。

KEEPER: Learning Control-Sufficient Textual Memory for Long-Horizon LLM Agents

KEEPER：为长视界LLM代理学习控制充足的文本内存

Authors: Jinkyeong Choi, Chaebin Jeong, Junsung Jeon, Donghyeon Park
Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.41482
Pdf link: https://arxiv.org/pdf/2610.41482
Abstract Recursive textual memory can carry earlier errors into later updates, yet task performance alone does not establish memory reliability. We define self-consistency bias as the persistence of incorrect claims already present in the incoming memory. We present KEEPER, a comparative analysis of four auxiliary objectives along information-content and update-sensitivity axes within a shared recursive interface, examining how supervision targets relate to agent performance. Among evaluated conditions, reward-related prediction achieve the highest task performance before and after offline reinforcement learning, outperforming joint supervision. A claim-level analysis of one baseline trajectory documents error persistence despite corrective observations. Together, these analyses motivate evaluating error persistence and valid-information retention alongside task outcomes.
中文摘要 递归文本记忆可以将早期错误带入后续更新，但仅凭任务表现无法确立记忆可靠性。我们将自我一致性偏差定义为输入内存中已存在错误声明的持续性。我们呈现KEEPER，这是对共享递归接口中信息内容轴和更新敏感性轴上的四个辅助目标的比较分析，探讨监督目标如何与代理绩效相关。在评估条件下，奖励相关预测在离线强化学习前后实现了最高的任务表现，优于联合监督。对一个基线轨迹的索赔级分析记录了即使有纠正观察，错误依然存在。这些分析共同激励评估错误的持续性和有效信息保留性，以及任务结果。

What Should LLMs Forget: Reinforcement Unlearning of Knowledge Associations

LLM应忘记什么：强化、知识关联的去复学

Authors: Wutong Xu, Yun Qu, Haiyan Feng, Qi Wang, Xiangyang Ji
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.41616
Pdf link: https://arxiv.org/pdf/2610.41616
Abstract Large language models (LLMs) acquire extensive factual knowledge during pre-training, which may include sensitive, private, or copyrighted information, motivating the development of unlearning methods that selectively remove specific knowledge while preserving general capabilities. However, existing approaches typically optimize entire responses without explicitly identifying the knowledge to be forgotten, resulting in coarse and uncontrolled forgetting. To address this, we revisit the forgetting target of LLM unlearning and introduce Knowledge Association (KA), which characterizes the association between a query context and its target knowledge and is instantiated by the corresponding conditional predictive distribution, as the fundamental target of forgetting. Based on KA, we propose Knowledge Association Reinforcement Unlearning (KARU), a unified reinforcement learning framework that suppresses forgotten knowledge associations while preserving retained ones. Specifically, KARU derives complementary mechanisms from the KA-based formulation: token-level reward construction for finer-grained forgetting supervision, distribution-level regularization for precise knowledge suppression, and proximity-aware retain sampling to mitigate collateral degradation. Experiments across diverse LLMs on representative benchmarks demonstrate that KARU consistently improves forgetting effectiveness over existing baselines while maintaining comparable retention performance and general utility.
中文摘要 大型语言模型（LLM）在预训练过程中获得了大量事实知识，可能包括敏感、私密或版权信息，这促使开发了选择性去除特定知识同时保留一般能力的去学习方法。然而，现有方法通常优化整个响应，却未明确指出要遗忘的知识，导致遗忘粗糙且失控。为此，我们重新审视了LLM遗忘目标，并引入知识关联（KA），该关联描述查询上下文与目标知识之间的关联，并通过相应的条件预测分布实现，作为遗忘的根本目标。基于KA，我们提出了知识关联强化去学习（KARU），这是一种统一的强化学习框架，抑制遗忘的知识关联，同时保留保留的知识关联。具体来说，KARU从基于KA的表述中衍生出互补机制：用于更细粒度遗忘监督的代币级奖励构建，用于精确的知识抑制的分布级正则化，以及邻近感知保留抽样以减轻抵押品退化。在代表性基准测试上，跨多个大型语言模型的实验表明，KARU在保持可比的保留性能和整体效用的同时，持续提升遗忘效果，相较于现有基线。

GRIT: Gradient Masking for Visually Misgrounded Tokens in Text-Rich Reasoning

GRIT：文本丰富推理中视觉上不基于标记的渐变掩蔽

Authors: Chanseul Cho, Noori Bae, Dongheon Lee, Youngmin Ro
Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Arxiv link: https://arxiv.org/abs/2610.41629
Pdf link: https://arxiv.org/pdf/2610.41629
Abstract Reinforcement learning with verifiable rewards (RLVR) has recently been extended to vision-language models (VLMs), yet its effectiveness remains limited for text-rich reasoning tasks. We highlight a key factor contributing to this limitation: models frequently arrive at correct final answers despite misperceiving visual information during intermediate steps. We define this phenomenon as Correct-but-Visually-Misgrounded Reasoning (CVMR). Because standard RLVR rewards only final-answer correctness, it fails to differentiate between properly grounded and misgrounded reasoning, reinforcing both equally. Through token-level analysis, we reveal that CVMR tokens exhibit distinct attention patterns that differ systematically from well-grounded ones. Based on this observation, we propose GRIT, a training framework that incorporates visual grounding into RLVR to mitigate the impact of CVMR. Furthermore, to address the lack of text-rich reasoning benchmarks that allow for systematic evaluation across text length and reasoning depth, we introduce GRIT-Bench. Experiments demonstrate that GRIT significantly improves both reasoning accuracy and grounding fidelity over existing RLVR methods.
中文摘要 带有可验证奖励的强化学习（RLVR）最近已扩展到视觉语言模型（VLM），但其在文本丰富推理任务中的效果仍然有限。我们强调了一个关键因素：模型在中间步骤中误判视觉信息，仍常得出正确的最终答案。我们将此现象定义为正确但视觉错误的推理（CVMR）。由于标准RLVR仅奖励最终答案的正确性，它无法区分正确基础和错误基础的推理，反而同等强化了两者。通过代币级分析，我们发现CVMR代币表现出与良好基础代币系统性不同的注意力模式。基于这一观察，我们提出了GRIT培训框架，将视觉基础融入RLVR以减轻CVMR的影响。此外，为了解决缺乏能够在文本长度和推理深度上进行系统评估的文本丰富推理基准，我们引入了GRIT-Bench。实验表明，GRIT在推理准确性和基础忠实度上均优于现有RLVR方法。

Risk-Sensitive Mission Planning for Multi-Target On-Orbit Inspection via Distributional Reinforcement Learning

通过分布式强化学习实现多目标轨道检查的风险敏感任务规划

Authors: Adam Abdin (LGI)
Subjects: Subjects: Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
Arxiv link: https://arxiv.org/abs/2610.41647
Pdf link: https://arxiv.org/pdf/2610.41647
Abstract Multi-target on-orbit servicing missions require planning tools that can adapt to evolving operational conditions while accounting for the outcome variability inherent to safety-critical spaceflight. This paper presents an adaptive planning approach for multi-target servicing campaigns in low Earth orbit and applies it to inspection missions flown by a low-thrust servicer. The approach is based on Distributional Reinforcement Learning, which learns the full distribution of mission returns and thereby supports explicit risk-sensitive decision-making at each planning step. Specifically, a Quantile-Regression Deep Q-Network (QR-DQN) agent selects which target to inspect next by jointly considering the expectation and the variance of the predicted return through a mean-variance criterion. The mission environment models circular orbits with a spread in inclination and uses Edelbaum's combined-manoeuvre low-thrust transfer estimates. The observation comprises the orbital elements, the visit history, the observed risk state of each target and the remaining mission-time budget. The risk state of each target changes stochastically during the campaign, making the outcome of an inspection uncertain and creating operationally meaningful distinctions between risk-neutral and risk-averse planning strategies. Because the mission-time budget generally admits only a subset of the candidate targets, the planner must also select which targets to inspect. Numerical experiments on scenarios with five target satellites show that one risk-aversion weight moves the policy across a range of mean return and return variance, over an unchanged environment and reward. Risk-averse agents reduce the variance of the episodic return by approximately one half relative to risk-neutral agents and lower the probability of a mission loss, at a cost in mean return. To validate the learned policies, the efficient frontier, which gives the highest mean return attainable at each level of return variance, is computed exactly by enumerating the admissible visit orders. At equal variance, the learned policies lie close to this frontier in mean return, and the risk-averse policies have a lower probability of a mission loss than the frontier policies.
中文摘要 多目标在轨维护任务需要能够适应不断变化的操作条件的规划工具，同时考虑安全关键航天飞行固有的结果变异性。本文提出了一种适用于近地轨道多目标维护活动的自适应规划方法，并将其应用于低推力服务商执行的检查任务。该方法基于分布强化学习，能够学习任务返回的完整分布，从而在每个规划步骤中支持明确的风险敏感决策。具体来说，分位回归深度Q网络（QR-DQN）智能体通过结合预期和预测返回的均方差标准，选择下一目标。任务环境建模倾角分布的圆形轨道，并使用Edelbaum联合机动低推力转移估计。观测包括轨道要素、访问历史、每个目标的观测风险状态以及剩余的任务时间预算。每个目标的风险状态在战役过程中随机变化，使检查结果变得不确定，并在操作上区分了风险中性与风险规避的规划策略。由于任务时间预算通常只接收候选目标的部分，规划者还必须选择要检查哪些目标。在五颗目标卫星的情景中进行的数值实验表明，一个风险规避权重会使政策在平均回报和回报方差范围内移动，且环境和回报不变。风险厌恶代理人将周期性回报的方差相对于风险中性代理人约减半，并降低任务失败的概率，但平均回报会有所损失。为验证所学策略，通过枚举可接受的访问顺序精确计算高效前沿，即在各回报方差水平下可获得的最高平均回报。在方差相等时，所学策略在平均回报上接近该边界，且风险厌恶策略的任务丢失概率低于前沿策略。

ResMerge: Residual Policy Learning and Merging for Continual Adaptation of Pre-trained Robot Policies

ResMerge：残余策略学习与合并，持续适应预训练机器人策略

Authors: Kefei Zhu, Lingyun Xu, Yue Xin, Xiaoqi Li, Caixin Guo, Xincheng Du, Yuanpei Chen, Yaodong Yang, Hao Dong
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2610.41663
Pdf link: https://arxiv.org/pdf/2610.41663
Abstract Continual adaptation to new tasks is essential for robots deployed in the real world, but adapting pre-trained policies remains challenging due to parameter growth and the need for continual demonstration collection. Parameter-Efficient Continual Fine-Tuning (PECFT) isolates task-specific updates, but adapter-per-task designs require demonstrations and grow with the task set. We propose ResMerge, a two-stage framework that learns mergeable residual policies atop a frozen pre-trained policy: residual reinforcement learning (RL) acquires task-specific corrections without additional expert demonstrations during adaptation, and parameter-space merging consolidates compatible residuals to reduce stored parameters. We introduce Residual-Field Consistency (RFC), a simple, bounded, and interpretable metric that assesses compatibility in residual action-field space by comparing correction direction and magnitude. Experiments on LIBERO across two backbones and on a real-world Franka robot show improved continual learning performance over strong PECFT baselines, while selective merging reduces the residual-parameter footprint. Project website: this https URL.
中文摘要 对现实世界中部署的机器人来说，持续适应新任务至关重要，但由于参数增长和持续演示收集的需求，调整预训练策略仍然具有挑战性。参数高效持续微调（PECFT）隔离了任务特定的更新，但每个任务的适配器设计需要演示并随任务集增长。我们提出了ResMerge，这是一个两阶段框架，在冻结的预训练策略上学习可合并的残差策略：残差强化学习（RL）在适应过程中无需额外专家演示即可获得任务特定的修正，参数空间合并则整合兼容残差以减少存储参数。我们引入残差场一致性（RFC），这是一个简单、有界且可解释的指标，通过比较修正方向和大小来评估残差作用场空间的兼容性。在两条骨干和真实Franka机器人上的LIBERO实验显示，在强PECFT基线下持续学习表现有所提升，而选择性合并则减少了残余参数的使用量。项目网站：此 https URL。

Multimodal In-Context Learning with Explicit Rule Induction

多模态上下文学习与显式规则归纳

Authors: Yuquan Li, Xinyang Jiang, Yuhao Xue, Cairong Zhao
Subjects: Subjects: Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.41754
Pdf link: https://arxiv.org/pdf/2610.41754
Abstract A central challenge in multimodal in-context learning (MICL) arises from tasks that are ill-defined without context and require uncovering the underlying rules from demonstrations. Existing methods predominantly treat MICL as direct input-output mapping, overlooking the importance of explicitly inducing and reasoning over such rules, limiting both performance and interpretability. We propose ERIM, Explicit Rule-Induction for Multimodal In-Context Learning, a unified framework that enhances MICL by explicitly inducing the latent rules governing demonstration samples. Leveraging the reasoning ability of multi-modal large language model (MLLM), ERIM constructs rule-centric MICL datasets, introduces explicit supervision for rule induction, and fine-tunes MLLM to perform explicit rule-induction via a two-stage training strategy that combines mixed cold-start supervised fine-tuning with reinforcement learning. Extensive experiments show consistent performance improvements across diverse MICL settings and demonstrate that correctly inducing underlying rules directly leads to stronger and more stable performance.
中文摘要 多模态上下文学习（MICL）的一个核心挑战来自于缺乏上下文定义不清的任务，需要通过演示揭示底层规则。现有方法主要将MICL视为直接输入输出映射，忽视了显式诱导和推理规则的重要性，限制了性能和可解释性。我们提出了ERIM（多模态上下文内学习显式规则归纳），这是一个统一框架，通过显式诱导指导演示样本的潜在规则来增强MICL。利用多模态大型语言模型（MLLM）的推理能力，ERIM构建以规则为中心的MICL数据集，引入规则归导的显式监督，并通过结合混合冷启动监督微调与强化学习的两阶段训练策略，微调MLLM以实现显式规则归纳。大量实验显示，在各种MICL环境中性能持续提升，并证明正确引入底层规则直接带来更强、更稳定的性能。

Preference-Conditioned Group-Relative Policy Optimization for Terminally Constrained Multi-Objective Control

偏好条件的群体相对策略优化，适用于终端约束的多目标控制

Authors: Md Ragib Rownak, Sidra Ghayour Bhatti, Qadeer Ahmed
Subjects: Subjects: Systems and Control (eess.SY)
Arxiv link: https://arxiv.org/abs/2610.41884
Pdf link: https://arxiv.org/pdf/2610.41884
Abstract Multi-objective optimal control is commonly addressed by reinforcement learning with fixed scalarization weights, requiring a separate training run and constraint enforcement for each trade-off. This letter extends critic-free group-relative policy optimization to terminally constrained multi-objective control: a single preference-conditioned policy is trained over random preferences and returns a feasible approximation of the Pareto front in one run. Three structural results are established. Group normalization after scalarization preserves the prescribed preference, whereas per-objective normalization induces policy-dependent distortion; with pairwise distinct normalization scales, only single-objective preferences remain unchanged. A positive floor prevents the terminal-constraint multiplier from vanishing and renders the penalty exact when the floor exceeds a geometric multiplier. Deterministic terminal violation is bounded by expected stochastic violation plus a term depending on terminal-state curvature and action variance, so stochastic feasibility alone does not certify deterministic execution. On a heavy-duty range-extender powertrain with six stage costs and a terminal state-of-charge band, a representative training run places all 32 preference-grid solutions in band on six cycles inside and outside the training pool. Averaged over ten runs, the method yields more in-band solutions under deterministic execution than a critic-based conditioned baseline with the same terminal-constraint handling.
中文摘要 多目标最优控制通常通过强化学习解决，且具有固定的标量化权重，每个权衡需要单独的训练运行和约束执行。本信将无批评的群相对策略优化扩展到终端约束多目标控制：单一偏好条件策略在随机偏好上训练，并在一次运行中返回帕累托前端的可行近似。建立了三个结构性结果。标量化后的群归一化保持规定偏好，而逐目标归一化则诱导策略依赖的扭曲;在两两不同标准化尺度下，只有单目标偏好保持不变。正的底线防止终端约束乘数消失，并在底线超过几何乘数时使惩罚精确。确定性终端违约受预期随机违背及基于终端状态曲率和作用方差的项限制，因此仅凭随机可行性无法证明确定性执行。在具有六级成本和终端状态充电带的重型续航动力系统上，代表性训练运行将所有32个偏好网格解置于训练池内外的六个周期内。通过十次平均，该方法在确定性执行下产生的带内解比基于批评者的条件基线且终端约束处理相同。

Optimizing Stochastic Policy Geometry for Few-Step Flow Reinforcement Learning

优化随机策略几何以实现少步流强化学习

Authors: Zhuguanyu Wu, Congjia Chen, Ruihao Gong, Yang Yong, Shiqiao Gu, Yi Xu, Xianglong Liu
Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.41949
Pdf link: https://arxiv.org/pdf/2610.41949
Abstract Few-step flow models generate samples through a handful of large denoising transitions. Applying GRPO to such models requires introducing stochasticity into the denoising transitions, and when a single transition spans a wide range of noise levels, the choice of the resulting stochastic policy becomes a major factor in optimization. Recent constructions constrain the policy to preserve the target flow coefficients, but this constraint admits a family of policies and offers no criterion for choosing among them. We introduce Geo-FSRL (Geometry-Optimized Few-Step Reinforcement Learning), built on a selection principle: at a fixed conditioning state and timestep pair, choose the admissible policy that minimizes the conditional KL divergence induced by any given nonzero velocity-output update. Within a coefficient-preserving family of isotropic Gaussian transitions, this criterion reduces to minimizing a single scalar geometry factor that determines the conditional KL for every velocity-output update. The same factor governs velocity-space Fisher information and conditional transition log-likelihood-ratio variance. We derive the unique closed-form optimizer within this family, shared across all nonzero update magnitudes and directions, with transition coefficients determined by both the source and target noise levels. Because the minimized geometry factor still varies across timesteps, we further adopt gradient-scale matching from prior work and derive closed-form policy-gradient and KL weights that normalize explicit score scales and reference-KL curvature. Geo-FSRL trains on the original few-step schedule and retains the deterministic solver at inference. Our method achieves higher final reward scores than Flow-GRPO and Flow-CPS on both image generation and video generation models under the 4-step setting. Code is available at this https URL
中文摘要 少步流模型通过少数大型去噪跃迁生成样本。将GRPO应用于此类模型需要在去噪跃迁中引入随机性，当单一跃迁跨越广泛的噪声水平时，随机策略的选择成为优化的主要因素。近期构造限制策略以保持目标流系数，但该约束允许一系列策略，且不提供选择标准。我们引入Geo-FSRL（几何优化少步强化学习），基于选择原则：在固定条件状态和时间步对下，选择最小化任意非零速度输出更新引起的条件KL散度的可接受策略。在保持系数的各向同性高斯跃迁族中，该准则简化为最小化一个标量几何因子，该因子决定每次速度输出更新的条件KL。同一因子控制速度空间Fisher信息和条件转移对数似然比方差。我们在该系列中推导出唯一的闭式优化器，该优化器在所有非零更新幅度和方向间共享，跃迁系数由源噪声和目标噪声水平共同决定。由于最小化几何因子仍会在时间步间变化，我们进一步采用先前工作的梯度尺度匹配，推导出封闭形式策略梯度和KL权重，以规范显式评分尺度和参考KL曲率。Geo-FSRL基于原始的少步计划训练，并在推断时保留确定性求解器。我们的方法在四步设置下，无论是图像生成模型还是视频生成模型，最终奖励分数均高于Flow-GRPO和Flow-CPS。代码可在此 https URL 获取

When Is a Game a Markov Decision Process? Correspondences Between Game Theory and Reinforcement Learning

博弈何时是马尔可夫决策过程？博弈论与强化学习之间的对应关系

Authors: Eduardo C. Garrido-Merchán
Subjects: Subjects: Computer Science and Game Theory (cs.GT); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.41962
Pdf link: https://arxiv.org/pdf/2610.41962
Abstract Reinforcement learning and game theory describe the same object from two sides. Game theory says what a rational outcome of a strategic interaction is; reinforcement learning says how an agent could reach one through experience. The two literatures nevertheless keep separate vocabularies for identical structures, and the treatments that bridge them present the correspondence as analogy rather than theorem. This monograph supplies the theorems. A game is a Markov decision process in three distinct senses, and every table entry is labelled with the one that applies: a definitional identity, an exact reduction carried by a value- and solution-preserving map, or a structural analogy that fails under identifiable conditions. We take every standard class of game, from normal-form through repeated, stochastic, partially observable, extensive-form with imperfect information, Bayesian, graphical, Stackelberg, potential, coalitional, mean-field and evolutionary games, and identify for each the decision-theoretic model that reinforcement learning already studies under another name. The same treatment is applied to solution concepts: Nash equilibrium and the fixed point of the Bellman optimality operator, quantal response equilibrium and the soft Bellman operator, the Shapley value and credit assignment, Blackwell approachability and no-regret learning. We map each model class onto the training procedures sound for it and record the complexity of the associated solution problems. A final part collects the correspondences that do not hold: the non-stationarity that invalidates the independent-learner reduction, the impossibility results that forbid uncoupled convergence to Nash equilibrium, and the gap between the coarse correlated equilibria that no-regret self-play reaches and the correlated equilibria it is credited with.
中文摘要 强化学习和博弈论从两个角度描述同一对象。博弈论说明战略互动的理性结果;强化学习说明智能体如何通过经验达到理性结果。然而，这两部文献仍保留了相同的结构词汇，桥接它们的处理方式将对应关系呈现为类比而非定理。本专著提供了定理。博弈在三种不同意义上是马尔可夫决策过程，每个表格条目都以适用的那个标记：定义同一性、由保持价值和解的映射所携带的精确归约，或在可识别条件下失败的结构类比。我们取取所有标准博弈类别，从正规形到重复、随机、部分可观测、带有不完美信息的扩展形式、贝叶斯博弈、图解博弈、斯塔克尔伯格博弈、势博弈、联盟博弈、均值场博弈和进化博弈，并为每个模型识别强化学习已以另一个名称研究的决策理论模型。同样的处理方法应用于解概念：纳什均衡与贝尔曼最优算子的不动点、量子响应均衡与软贝尔曼算子、夏普利值与功劳赋值、布莱克韦尔可接近性及无遗憾学习。我们将每个模型类映射到其训练程序的音色上，并记录相关解题的复杂性。最后一部分收集了不成立的对应关系：非平稳性使独立学习者归约失效，不可得的结果阻止无耦合收敛到纳什均衡，以及无悔自玩所达到的粗相关均衡与其归功于相关均衡之间的差距。

CausalXRL: Explainable Reinforcement Learning through Causal Graph Reasoning

CausalXRL：通过因果图推理实现可解释的强化学习

Authors: Yanming Zhang, Eric Papenhausen, Klaus Mueller
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.42028
Pdf link: https://arxiv.org/pdf/2610.42028
Abstract Reinforcement learning is a powerful paradigm for training autonomous agents and has achieved impressive performance in complex environments. However, this success often comes at the cost of interpretability, diminishing trust and complicating efforts to debug and improve agent behavior. To address these challenges, we introduce CausalXRL, a novel framework for explainable reinforcement learning (XRL). A key feature of CausalXRL is its use of causal graph reasoning, which provides transparent, structured, multi-level explanations of agent decision-making. We validate CausalXRL through comprehensive case studies and a two-part evaluation: (1) a quantitative analysis of explanation fidelity and causal-structure learning efficiency in benchmark RL environments, and (2) a qualitative expert study assessing explainability in the real-time strategy (RTS) benchmark MicroRTS. The quantitative results show that CausalXRL can provide faithful explanations while efficiently learning causal structures, and the qualitative expert study suggests that participants found CausalXRL useful for inspecting high-level RTS strategies.
中文摘要 强化学习是训练自主智能体的强大范式，在复杂环境中取得了令人印象深刻的表现。然而，这种成功往往以可解释性为代价，降低信任度，并使调试和改进智能体行为变得复杂。为应对这些挑战，我们引入了CausalXRL，一种可解释强化学习（XRL）的新框架。CausalXRL的一个关键特点是其因果图推理，提供了透明、结构化、多层次的智能体决策解释。我们通过全面的案例研究和两部分评估验证了CausalXRL：（1）基准RL环境中解释准确性和因果结构学习效率的定量分析，以及（2）评估实时战略（RTS）基准MicroRTS中可解释性的定性专家研究。定量结果显示，CausalXRL能够在高效学习因果结构的同时提供忠实解释，而定性专家研究表明，参与者认为CausalXRL对于检验高级RTS策略非常有用。

CRoS: Causality Reward on Scrambled CoTs Improves Accuracy and Reasoning Faithfulness

CRoS：混乱CoT的因果奖励提升准确率和推理忠实度

Authors: Xiao Yu, Justin Chih-Yao Chen, Zaid Khan, Hyunji Lee, Mohit Bansal
Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Arxiv link: https://arxiv.org/abs/2610.42054
Pdf link: https://arxiv.org/pdf/2610.42054
Abstract Reinforcement learning with verifiable rewards (RLVR) has become the standard method for training language models to reason. However, RLVR typically relies solely on outcome correctness, which does not guarantee that the CoT causally contributes to the final answer. Without this connection, outcome rewards may fail to reinforce high-quality reasoning. Existing causality metrics assess CoT causality by perturbing the CoT, such as deleting reasoning steps or injecting errors, and measuring changes in the final answer. However, these perturbations inevitably alter information in each step. Consequently, high causality scores may reflect sensitivity to information changes introduced by perturbation rather than true causality. Such confounding effects further induce undesirable reasoning behaviors when these metrics are used as training rewards, including generating less informative CoTs and reducing exploration and self-correction. To address this problem, we propose CRoS (Causality Reward on Scrambled CoTs), a novel CoT causality metric that perturbs the order of reasoning steps without altering content, thereby preserving information within each step. On Reasoning Gym with Gemma-3-4B, CRoS outperforms existing causality and outcome rewards in out-of-domain accuracy, CoT quality, and CoT faithfulness. Specifically, CRoS achieves 46.8% accuracy on five unseen logic and games datasets, outperforming the strongest baseline (43.8%). It also achieves better CoT quality, consistently outperforming baselines in CoT-Pass@K from K=1 to 128, averaging 43.4% vs. 41.4%. For CoT faithfulness, CRoS achieves a hint acknowledgment score of 34.0% vs. 20.2%.
中文摘要 带可验证奖励的强化学习（RLVR）已成为训练语言模型推理的标准方法。然而，RLVR通常仅依赖结果正确性，这并不保证CoT对最终答案有因果贡献。没有这种联系，结果奖励可能无法强化高质量的推理。现有的因果性指标通过扰动CoT来评估CoT因果关系，如删除推理步骤或注入错误，并测量最终答案的变化。然而，这些扰动不可避免地改变了每一步的信息。因此，高因果得分可能反映的是对扰动引入的信息变化的敏感性，而非真实因果。当这些指标作为训练奖励时，这种混杂效应进一步诱发不良推理行为，包括产生信息量较低的CoT以及减少探索和自我纠正。为解决此问题，我们提出了CRoS（扰乱CoT因果奖励），这是一种新型CoT因果指标，扰动推理步骤的顺序而不改变内容，从而保留每一步内的信息。在Reasoning Gym with Gemma-3-4B中，CRoS在域外准确性、CoT质量和CoT忠实度方面优于现有因果性和结果奖励。具体来说，CRoS在五个未见逻辑与博弈数据集中实现了46.8%的准确率，优于最强基线（43.8%）。它还实现了更好的CoT质量，在CoT-Pass@K从K=1到128的基线中持续优于基线，平均43.4%对41.4%。在《诅咒》忠实度方面，CRoS的提示认可得分为34.0%，对比20.2%。

RAMA: Retrieval-Augmented Multi-Agent Routing with Episodic Memory for LEO Satellite Networks

RAMA：基于剧集记忆的LEO卫星网络检索增强多智能体路由

Authors: Xin Ming Li, Peng Hu
Subjects: Subjects: Networking and Internet Architecture (cs.NI); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.42068
Pdf link: https://arxiv.org/pdf/2610.42068
Abstract Low-Earth orbit (LEO) satellite networks present a challenging routing environment due to rapidly changing topology, traffic, and link conditions. Existing deep reinforcement learning (DRL) routing methods typically rely on current observations and information encoded in network weights, limiting their ability to exploit successful past experiences. We propose RAMA, a retrieval-augmented multi-agent DRL framework with Approximate Information State (AIS), in which each satellite combines its local observation, recent routing history, neighboring state information, and retrieved entries from a per-satellite episodic memory. Retrieved experiences are integrated through a Transformer-based decision network trained with AIS-based auxiliary objectives. With weights frozen in a 140-satellite Kepler constellation, RAMA reduces mean end-to-end latency by 13.4% and 99th-percentile latency by 66.7% relative to the state-of-theart double deep Q-network (DDQN) routing method, mainly by avoiding long detours. Updated online, RAMA remains stable while DDQN degrades. On the 1,584-satellite Starlink shell 1, the reduction in mean latency grows to 93.2%, and updating RAMA online lowers its mean latency by a further 25.5%. These results demonstrate the effectiveness of retrieval-augmented memory for scalable decentralized routing in LEO satellite networks.
中文摘要 低地球轨道（LEO）卫星网络由于拓扑、流量和链路条件快速变化，呈现出具有挑战性的路由环境。现有的深度强化学习（DRL）路由方法通常依赖当前观测数据和编码在网络权重中的信息，限制了其利用过去成功经验的能力。我们提出了RAMA，这是一种带有近似信息状态（AIS）的检索增强多智能体DRL框架，每颗卫星结合其本地观测、近期路由历史、邻近状态信息以及从每个卫星的情节记忆检索的条目。检索到的经验通过基于Transformer的决策网络整合，该网络训练有AIS辅助目标。在140颗卫星的开普勒星座中，RAMA相比心态双深Q网络（DDQN）路由方法，平均端到端延迟降低13.4%，第99百分位延迟降低66.7%，主要通过避免长距离绕行。在线更新后，RAMA保持稳定，而DDQN则会下降。在拥有1584颗卫星的Starlink壳体1上，平均延迟降低至93.2%，更新RAMA在线后平均延迟进一步降低25.5%。这些结果证明了检索增强内存在近地轨道卫星网络中可扩展的去中心化路由的有效性。

ROOF: Learning Humanoid Fall Recovery under Overhead Constraints

屋顶：在开销限制下学习类人生物坠落恢复

Authors: Yidan Lu, Jiayuan Yu, Zongrui Cai, Jingwen Sun, Jinrui Han, Jing Wang, Ping Luo, Peng Lu
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2610.42097
Pdf link: https://arxiv.org/pdf/2610.42097
Abstract A fallen humanoid beneath an overhead obstacle faces a coupled recovery problem. It must create clearance while organizing support for a stable rise, since attempting to stand before making room can further increase obstruction. We introduce a reinforcement learning method with contact-supported geometric escape guidance, which rewards incremental reduction of robot-obstacle overlap without prescribing an escape path. To achieve this, a learned motion prior, together with posture-dependent support and joint-stall costs, is proposed to shape a single policy that learns when to clear the obstacle, how to establish support, and when to complete the rise, all using proprioception alone. In simulation, it recovers in 88.5% of trials beneath plates that can only be lifted, compared with 18.9% without geometric guidance, and in 98.6% of trials beneath free plates with randomized size, mass and placement. On a Unitree G1, the deployed policy recovered in 95 of 150 attempts across seven real-world obstacle scenes, including furniture and multi-object arrangements absent from training. Project page: this https URL
中文摘要 倒在顶方障碍物下的类人生物面临耦合恢复问题。它必须在组织支撑以实现稳定上升的同时创造间隙，因为在腾出空间前试图站立可能会进一步增加障碍物。我们引入了一种带有接触支撑几何逃脱引导的强化学习方法，该方法奖励机器人与障碍物重叠的渐进减少，而无需规定逃生路径。为此，提出了一种学习先验，结合姿势依赖支撑和关节失速成本，以形成单一策略，学习何时越过障碍物、如何建立支撑以及何时完成上升，所有这些都仅依靠本体感觉。在模拟中，只有88.5%的试验能在只能抬起的钢板下恢复，而没有几何指导的试验为18.9%;在自由钢板下随机尺寸、质量和位置的试验中，98.6%的恢复率。在Unitree G1上，部署策略在7个真实障碍场景中150次尝试中恢复了95次，包括训练中缺少的家具和多对象布置。项目页面：此 https URL

Towards Understanding the Impact of Plasticity Loss on Reinforcement Learning in Stochastic Environments

迈向理解可塑性丧失对随机环境中强化学习的影响

Authors: Philipp Bordne, André Biedenkapp
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.42100
Pdf link: https://arxiv.org/pdf/2610.42100
Abstract Deep reinforcement learning (RL) is a powerful framework for sequential decision-making. Despite many high-profile successes in recent years, deep RL is known to be brittle. An emerging line of research attributes this brittleness to a loss of plasticity in neural networks. However, plasticity loss has so far been understudied in stochastic environments. We study this interplay systematically, training SAC agents on three DeepMind Control tasks augmented with controlled levels of observation and action noise. We find that plasticity-preserving methods increase learning robustness on the two harder tasks, with their advantage growing as observation noise increases, while combining resets with layer normalization underperforms resets alone at every observation noise level tested. Further, stochasticity can accelerate the loss of plasticity in harder environments, whereas in easier settings noise can act as a regularizer that sustains plasticity. Lastly, plasticity loss is most pronounced in the early stages of training. In the most challenging stochastic settings, TD instability rather than gradual plasticity loss becomes the dominant failure mode.
中文摘要 深度强化学习（RL）是一个强大的顺序决策框架。尽管近年来取得了许多高调成功，但深度强化学习仍被认为脆弱。一项新兴研究将这种脆弱性归因于神经网络中可塑性的丧失。然而，在随机环境中，可塑性丧失迄今研究不足。我们系统地研究了这一相互作用，训练SAC代理执行三个DeepMind控制任务，辅以受控的观察和动作噪声水平。我们发现，保持可塑性的方法提升了两个较难任务的学习鲁棒性，且随着观察噪声的增加，其优势也随之增强而增强，而结合层归一化单独在所有观察噪声水平下重置表现均不及。此外，随机性在较困难环境中加速可塑性丧失，而在较简单的环境中，噪声可作为维持可塑性的正则化器。最后，可塑性损失在训练早期阶段最为明显。在最具挑战性的随机条件下，TD不稳定性而非渐进式可塑性丧失成为主导失效模式。

Hyperbolic Inverse Reinforcement Learning

双曲逆强化学习

Authors: Iulia Rusu, Anoop Praturu, Sergei Gepshtein, Tatyana Sharpee
Subjects: Subjects: Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.42104
Pdf link: https://arxiv.org/pdf/2610.42104
Abstract Inverse reinforcement learning (IRL) seeks to infer the reward function underlying observed behavior, but its predictive accuracy and interpretability depend on how that reward is computed from the underlying features. While prior IRL reward functions have been implemented through a variety of techniques including linear weights, neural networks, or other function approximators, none make assumptions about the geometry of the underlying feature space. Inspired by geometric organization observed in neural circuits, we introduce Hyperbolic inverse reinforcement learning (Hyperbolic IRL), which parameterizes reward as a function of Hyperbolic distance between the feature vector and a learned parameter vector, as well as its Euclidean distance variant. We tested our methods against a range of linear and nonlinear IRL baselines under both deterministic and stochastic policies. Our Hyperbolic approach achieved comparable performance to state-of-the-art models on established benchmarks and excelled in the low data limit and across dense environments. Applied to human navigation among multiple exhibits in a museum room, Hyperbolic IRL reveals reward structure that depends not only on individual exhibit-distance features, but also on non-separable conjunctions between them. Together, these results demonstrate that a Hyperbolic reward parameterization can improve both the predictive accuracy and interpretability of IRL, providing a framework for studying how non-separable, conjunctive structure can begin to explain behavior.
中文摘要 逆强化学习（IRL）旨在推断观察到行为背后的奖励函数，但其预测准确性和可解释性取决于如何从底层特征计算出奖励。虽然此前的IRL奖励函数通过线性权重、神经网络或其他函数近似器等多种技术实现，但都未对底层特征空间的几何做假设。受神经回路中几何组织的启发，我们引入了双曲反强化学习（Hyperbolic IRL），该方法将奖励参数化为特征向量与学习参数向量之间的双曲距离函数，以及其欧几里得距离变体。我们对确定性和随机策略下的线性和非线性IRL基线进行了测试。我们的双曲方法在既定基准测试中与最先进模型相当，并在低数据限制和密集环境中表现出色。应用于博物馆展厅中多个展品之间的人类导航，双曲实在光谱揭示了奖励结构不仅依赖于单个展品距离特征，还依赖于它们之间的不可分离连接。这些结果共同表明，双曲奖励参数化可以提升真实空间的预测准确性和可解释性，为研究不可分离的连取结构如何开始解释行为提供了框架。

Latent Reasoning in Continuous Space for Unified Multimodal Models

统一多模态模型中连续空间的潜在推理

Authors: Byungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho, Jinwoo Shin
Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG)
Arxiv link: https://arxiv.org/abs/2610.42149
Pdf link: https://arxiv.org/pdf/2610.42149
Abstract Unified Multimodal Models (UMMs) achieve remarkable success on diverse multimodal understanding and generation tasks, yet they still struggle to effectively leverage language-based chain-of-thought reasoning for image generation. As a result, existing reasoning-augmented UMMs often fail to inherit the test-time scaling and fine-grained controllability observed in large language models. We attribute this failure to a fundamental modality mismatch: reasoning is performed in a discrete token space, whereas image generation operates in a continuous space. To bridge this gap, we introduce a Latent Reasoning framework in Continuous space (LARC), which alleviates the token-level bottleneck by reasoning over continuous hidden states. Specifically, the training framework consists of two stages: (i) curriculum supervised fine-tuning (SFT), and (ii) information-gain reinforcement learning (RL). First, the curriculum SFT stage gradually converts token-level reasoning into latent reasoning. Subsequently, the information-gain RL stage self-evolves latent reasoning without ground truth supervision. Experiments across diverse text-to-image generation benchmarks show that LARC consistently improves generation quality, prompt following, and compositional fidelity. Notably, LARC achieves state-of-the-art performance on GenEval with a 7%p gain over BAGEL and demonstrates clear test-time scaling behavior, outperforming token-based generation under both parallel and sequential test-time scaling strategies.
中文摘要 统一多模态模型（UMMs）在多种多模态理解和生成任务中取得了显著成功，但它们仍然难以有效利用基于语言的思维链推理来生成图像。因此，现有的推理增强型UMM往往未能继承大型语言模型中观察到的测试时间缩放性和细粒度可控性。我们将这一失败归因于一种根本的模态不匹配：推理在离散的令牌空间中进行，而图像生成则在连续空间中运行。为弥合这一差距，我们引入了一个连续空间潜在推理框架（LARC），通过对连续隐态进行推理，缓解了令牌级瓶颈。具体来说，该训练框架包含两个阶段：（i）课程监督微调（SFT）和（ii）信息增益强化学习（RL）。首先，课程SFT阶段逐步将代币级推理转化为潜在推理。随后，信息获取强化学习阶段在无实地监督的情况下自我演化潜在推理。跨越多种文本生成基准测试的实验表明，LARC持续提升生成质量、提示跟随性和构图忠实度。值得注意的是，LARC在GenEval上实现了最先进的性能，比BAGEL提升7%p，并在测试时间缩放策略下表现明显，优于基于代币的生成。

Keyword: diffusion policy

Direct Action-Manifold Modeling for Efficient and Robust Bimanual Manipulation

直接作用流形建模以实现高效且稳健的双手操作

Authors: Hongren Wang, Mingyu You, Zihan Li, Zhipeng Wang, Shuo Jiang, Bin He
Subjects: Subjects: Robotics (cs.RO)
Arxiv link: https://arxiv.org/abs/2610.41643
Pdf link: https://arxiv.org/pdf/2610.41643
Abstract Diffusion and flow matching visuomotor policies require iterative denoising or numerical integration for multimodal action generation, introducing latency and execution mismatch. We instead consider a different formulation: conditioned on vision and proprioception, demonstrated bimanual action chunks concentrate in a structured observation-dependent region of joint-action space, making the action target more direct to model than the generative process. Based on this view, we propose the Direct Manifold Action Transformer (DMAT), which reformulates action generation as the direct modeling of clean, observation-conditioned joint-space action chunks from stochastic interpolants, thereby enabling single-step inference without relying on iterative generative inference or auxiliary trajectory-straightening procedures. Within DMAT, local 3D geometry anchors predictions to spatial structure, while State-Consistent Spectral-Time Refinement (STR) reconciles predicted chunks with proprioception and suppresses high-frequency residuals. Across 567 standard and out-of-distribution (OOD) trials per method, DMAT outperformed Improved 3D Diffusion Policy (IDP3) and DiTFlow3D in aggregate success rate, achieving relative improvements of 129.1% and 95.7%, respectively, while generating action chunks 2.06x and 1.34x faster and remaining robust under repeated 50 N interventions. These results pave the way for a novel generative paradigm for efficient and robust bimanual manipulation.
中文摘要 扩散和流动匹配的视觉运动策略需要迭代去噪或数值积分以生成多模态动作，从而引入延迟和执行不匹配。我们转而考虑另一种表述：基于视觉和本体感觉，演示的双手动作块集中在结构化且依赖观察的联合动作空间区域，使动作目标比生成过程更直接地建模。基于这一观点，我们提出了直接流形动作变换器（DMAT），它将动作生成重新表述为从随机插值中直接建模干净的观察条件联合空间动作块，从而实现单步推断，无需依赖迭代生成推理或辅助轨迹拉直程序。在DMAT中，局部三维几何将预测锚定于空间结构，而状态一致光谱时间细化（STR）则将预测块与本体感受协调，并抑制高频残差。在每方法567项标准和非分布（OOD）试验中，DMAT在总成功率上优于改进三维扩散政策（IDP3）和DiTFlow3D，分别实现了129.1%和95.7%的相对提升，同时生成动作块的速度提升了2.06倍和1.34倍，并且在反复50 N干预下保持鲁棒性。这些结果为高效且稳健的双手操作提供了一种新型生成范式。