生成时间: 2026-11-17 02:01:13 (UTC+8); Arxiv 发布时间: 2026-11-16 20:00 EST (2026-11-17 09:00 UTC+8)
今天共有 23 篇相关文章
Keyword: reinforcement learning
New Closed-Form Approximations in Statistical Mechanics Obtained by Symbolic Regression
通过符号回归获得的统计力学新闭形式近似
- Authors: Omkar Kunjir, Alina Popova, Rasul Abdusalamov, Mikhail Itskov
- Subjects: Subjects:
Computational Engineering, Finance, and Science (cs.CE)
- Arxiv link: https://arxiv.org/abs/2611.17498
- Pdf link: https://arxiv.org/pdf/2611.17498
- Abstract
Obtaining accurate and analytically tractable closed-form expressions for complex physical systems remains a fundamental challenge in statistical mechanics. In this work, we investigate three approximation problems in polymer statistical mechanics using two complementary symbolic regression frameworks: Deep Symbolic Optimization (DSO), based on reinforcement learning, and PyOperon, based on genetic programming with nonlinear least-squares constant optimization. First, for the force-extension relationship of the Worm-Like Chain (WLC) model, DSO achieves a maximum relative error below 0.15%, while PyOperon reduces it to 0.05%, outperforming the Marko-Siggia formula (approximately 17%) and the Petrosyan approximation. Second, for the logarithmic factorial, ln(n!), both frameworks recover the dominant structure of Stirling's formula directly from analytical data. Using residual learning on a continued-fraction expansion of the Stirling remainder, PyOperon reduces the maximum relative error by a factor of 1.3 x 10^4 compared with the most accurate of nine classical approximations for n in [1, 30]. Third, for the non-Gaussian radial probability density of the Freely Jointed Chain (FJC) model, physics-informed residual learning improves the approximation by factors of 11.2 over Darabi et al. (2023) and 4.6 over the steepest-descent approximation. The discovered expressions remain analytically tractable across all three applications. These results demonstrate the complementary capabilities of DSO and PyOperon for discovering accurate closed-form approximations in statistical and continuum mechanics.
- 中文摘要
获得复杂物理系统中准确且解析可解的闭式表达式仍然是统计力学中的一个根本挑战。本研究中,我们利用两个互补的符号回归框架研究了聚合物统计力学中的三个近似问题:基于强化学习的深度符号优化(DSO)和基于基因规划和非线性最小二乘常数优化的PyOperon。首先,对于蠕虫链(WLC)模型的力扩张关系,DSO将最大相对误差降至0.15%,而PyOperon将其降至0.05%,优于Marko-Siggia公式(约17%)和Petrosyan近似。其次,对于对数阶乘Ln(n!),这两个框架都直接从解析数据恢复斯特林公式的主导结构。通过对斯特林余数的持续分数展开进行残差学习,PyOperon 将最大相对误差减少了 1.3 x 10^4 倍,相较于 [1, 30] 中最准确的九种经典近似中最准确的 n 个近似。第三,对于自由关节链(FJC)模型的非高斯径向概率密度,物理知情的残差学习使近似比 Darabi 等人(2023)提高了 11.2 倍,在最陡下降近似上提高了 4.6 倍。发现的表达式在三种应用中依然具有解析可解性。这些结果展示了 DSO 和 PyOperon 在统计力学和连续统力学中发现准确闭形式近似的互补能力。
I2C-GRPO: Insight Importance Checklist for Reinforcement Learning in a Multi-Agent DDoS Alert Triage Workflow
I2C-GRPO:多智能体DDoS警报分流中强化学习的洞察重要性检查表
- Authors: Eyal Zloof, Shaun Shuster, Aviram Zilberman, Chen Frydman, Inderjeet Singh, Andrés Murilo, Asaf Shabtai, Yuval Elovici, Rami Puzis
- Subjects: Subjects:
Cryptography and Security (cs.CR)
- Arxiv link: https://arxiv.org/abs/2611.17534
- Pdf link: https://arxiv.org/pdf/2611.17534
- Abstract
Large language model (LLM) agents are increasingly used for investigative tasks over incomplete evidence, such as cybersecurity alert triage, but reinforcement learning (RL) for such tasks is difficult: intermediate steps have no verifiable labels, rollouts are costly, and outcome rewards are sparse. We propose Insight Importance Checklist (I2C)-GRPO, a checklist reward for intermediate investigation steps. For each decision point, a panel of expert LLMs proposes investigative actions, their atomic forensic insights are extracted, and a separate three-model ranker panel orders the insights by importance. A reward model checks which ranked insights a candidate action targets and returns their consensus-weighted coverage. Because actions are scored without being executed, the reward is a proxy for expected investigative value, and training reduces to a one-step contextual bandit over fixed investigation prefixes. From a soft-computing perspective, the reward is an additive fuzzy measure whose rank-exponent weights act as an ordered weighted averaging (OWA)-type operator; one exponent sets its orness, i.e., how strongly the reward concentrates on the top of the judges' consensus. We train an open-weight Qwen3-4B Triage Agent within a fixed multi-agent Distributed Denial of Service (DDoS) alert-triage workflow on DARPA 2000 LLDOS 1.0 and test it on an independent CSE-CIC-IDS2018 DDoS trace. Across three training seeds, the agent reaches a mean F1-score of 0.869, 5.9 and 9.7 points above uniform insight weights and DIVA, and a false-positive rate of 4.9% versus 16.4% for GPT-5-mini. Within the weighting family, false positives fall only at the highest-orness weighting evaluated. All claims are limited to DDoS alert triage; the code, task contexts, and ranked insight dataset will be released.
- 中文摘要
大型语言模型(LLM)代理越来越多地用于调查任务,处理证据不完整时,如网络安全警报分流,但此类任务的强化学习(RL)较为困难:中间步骤没有可验证的标签,推广成本高昂,结果奖励稀少。我们提出了洞察重要性检查表(I2C)-GRPO,一种针对中间调查步骤的清单奖励。对于每个决策点,一组专家LLM提出调查行动,提取其原子取证洞见,并由一个独立的三模型排名面板按重要性排序洞察。奖励模型检查候选行动针对并返回的共识加权覆盖率的排名洞察。由于行动在未执行的情况下被评分,奖励是预期调查价值的代理,训练简化为基于固定调查前缀的一步上下文盗贼。从软计算角度看,奖励是一个加法模糊度量,其秩指数权重作为有序加权平均(OWA)类型的算符;一个指数设定其正值,即奖励在评审共识顶端的集中程度。我们在DARPA 2000 LLDOS 1.0的固定多智能体分布式拒绝服务(DDoS)警报分流中训练一个开放权重Qwen3-4B分诊代理,并在独立的CSE-CIC-IDS2018 DDoS追踪中测试。在三个训练种子中,代理的平均F1得分为0.869,比统一洞察权重和DIVA高出5.9和9.7分,误报率为4.9%,而GPT-5-mini为16.4%。在权重族群中,误报仅在评估的最高或权重时出现。所有声明仅限于DDoS警报分流;代码、任务上下文和排名洞察数据集将被发布。
Benchmarks Don't Talk Back: Learning Through Self-Disagreement
基准不回嘴:通过自我分歧学习
- Authors: Soheil Abbasloo, Peng Cheng
- Subjects: Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.17563
- Pdf link: https://arxiv.org/pdf/2611.17563
- Abstract
A benchmark usually measures a language model alone with a question, and deployment rarely looks like that: answers get contested, and models give up correct ones under counter-reasoning that is itself wrong. We show that what fails there is not a shortfall of accuracy but a capability in its own right -- knowing when to hold a position and when to yield to a better one -- which we call dialectical competence. It stands on first-answer accuracy without following from it: a model can be made materially better at answering and no better at this, and training for accuracy alone can drift against it. Nor can it be bought at inference, by self-reflection or by sampling more. Taking a cue from the dialectical tradition in philosophy, in which productive opposition arises within a position rather than being imposed on it from outside, we hypothesize that this capability can be trained from opposition the model supplies itself. It states a position, generates the strongest case against that position -- its antithesis -- rules on whether the case stands, and is trained on that ruling, with no second model at training or at deployment. We call this learning through Self-Disagreement: reinforcement learning from verifiable rewards throughout, where the same check that scores an answer against its reference scores the ruling too, so no reward model, no preference data and no human annotation enter anywhere. The capability is trained rather than elicited: weak in the base model, it arrives with the reward, is not paid for out of first-answer accuracy, and holds against challengers the model never trained against, including ones stronger than itself. The opposition a model needs in order to improve, it turns out, need not come from outside it.
- 中文摘要
基准测试通常仅测量一个带有问题的语言模型,而部署很少是这样:答案会被质疑,模型在反推理下放弃正确答案,而反推理本身就是错误的。我们证明,这里失败的不是准确性不足,而是一种独立的能力——知道何时坚持立场,何时让位于更优立场——我们称之为辩证能力。它建立在第一答案的准确性上,但不依赖于此:模型可以在回答方面实质上变得更好,但不会更好,单靠准确训练可能会对它不利。它也无法通过推理、自我反思或更多抽样来获得。借鉴哲学中的辩证传统,即生产性的对立是在立场内部产生,而非外部强加,我们假设这种能力可以从模型自身提供的反对中训练。它陈述一个立场,生成对该立场最强的反驳论据——其对立面——关于该立场是否成立的规则,并基于该裁决进行训练,训练时或部署时没有第二个模型。我们称这种学习为自我分歧:从可验证的奖励中获得强化学习,其中与其参考对比评分的同一个检验也会对裁定进行评分,因此没有奖励模型、偏好数据和人工注释。该能力是训练而非诱发的:在基础模型中较弱,随奖励而来,不因首个答案的准确性而获得报酬,并且对模型从未训练过的挑战者(包括比它强的)依然有效。事实证明,模型为了改进所需的反对,不必来自外部。
Offline KL-Regularized Reinforcement Learning with Stochastic Transitions
带有随机转移的离线KL正则化强化学习
- Authors: Andrew Liu
- Subjects: Subjects:
Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.17603
- Pdf link: https://arxiv.org/pdf/2611.17603
- Abstract
KL-regularized reinforcement learning is the objective behind RL fine-tuning of language models, and its data are often collected by the reference policy itself. However, beyond bandits and deterministic transitions, little is known about how much offline data it needs. In this work, we study finite discounted Markov decision processes with unknown stochastic transitions, $K$ state-action pairs and effective horizon $H$. For data that cover the regularized optimum with concentrability $C\geq 4$, every estimator needs $\Omega(KCH^3/(\epsilon\max{\epsilon,\beta}))$ samples on some instances when $\epsilon$ is small, where $\beta$ is the temperature. Under a fixed design, pessimistic soft value iteration with a variance-aware penalty attains this rate with high probability, up to logarithmic factors, when $\beta\geq 1.4H$. When the data come from the reference policy itself, we show that it covers the optimum up to a power law whose exponent grows as $H/\beta$; in the same temperature range, this power law alone gives the estimator a sample complexity polynomial in $1/\epsilon$, with no coverage assumption. For $\beta\geq H$ the power can also increase the sample size needed by a factor polynomial in $1/\epsilon$. Finally, the plug-in estimator, which is minimax optimal with a generative model, can fail on reference data at small temperatures when many actions are rarely seen, while the pessimistic estimator does not.
- 中文摘要
强化语言模型微调的目标是强化学习(RL)语言模型微调的目标,其数据通常由参考策略本身收集。然而,除了强盗和确定性转移外,关于其需要多少离线数据知之甚少。本研究中,我们研究了具有未知随机转移、$K$状态-动作对和有效视野$H的有限折现马尔可夫决策过程。对于覆盖正则化最优且具有聚焦度$C\geq 4$的数据,每个估计量在某些情况下需要$\Omega(KCH^3/(\epsilon\max{\epsilon,\beta}))样本,在某些情况下,当$\epsilon$为温度时。在固定设计下,带有方差惩罚的悲观软值迭代在$\beta\geq 1.4H$时,以高概率(对数因子)达到该速率。当数据来自参考策略本身时,我们证明它覆盖了幂律的最优条件,幂律的指数随$H/\β$增长;在同一温度范围内,仅该幂律即可给估计量一个样本复杂度多项式,单位为$1/\εsilon$,且无覆盖假设。对于$\β\geq H$,幂级还能增加因子多项式所需的样本量,单位为$1/\ε⁰⁰。最后,插件估计器在生成模型中是极小极大最优,但在低温度下,当许多动作很少被观察到时,可能在参考数据上失败,而悲观估计量则不会。
Battery Energy Storage Sizing in PV Ramp-Rate Smoothing using Constraint-Aware Reinforcement Learning-Based Controller
基于约束的强化学习控制器在光伏爬行率平滑中的电池储能规模
- Authors: Arash Omidi, Tanmay Mishra, Mads R. Almassalkhi
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.17612
- Pdf link: https://arxiv.org/pdf/2611.17612
- Abstract
This paper presents a constraint-aware reinforcement learning (CARL) controller for battery energy storage system (BESS) sizing and photovoltaic (PV) ramp-rate smoothing. Rapid PV power fluctuations caused by cloud transitions can introduce operational and stability challenges in power systems. To mitigate these impacts, grid codes impose ramp-rate limits (RRL) on PV power injected to the grid. BESSs can support such grid codes by absorbing or supplying power during rapid variations. However, the extent to which BESSs can reduce PV-induced variability depends on (i) the BESS power and energy capacities; and (ii) the adopted controller. Thus, BESS sizing and controller design are coupled. Conventional moving-average control can increase battery usage due to filter-induced lag, while ramp-rate control is reactive and does not account for the long-term impact of actions on battery state of charge and future ramp events. Optimization-based methods can reduce BESS requirements by anticipating future PV variations, but their performance depends on accurate PV forecasts. To address these limitations, the CARL controller learns a causal state-feedback policy from historical PV data without requiring forecasts during deployment. Ramp-rate, converter-power, and state-of-charge constraints are incorporated into a causal feasibility interval that guarantees feasible actions by definition. A full-horizon input-convex neural network (ICNN)-based optimization is developed as a noncausal benchmark for the minimum achievable BESS requirements. Evaluation using three real-world PV datasets shows that, at a 10\%/min ramp-rate limit, CARL reduces peak BESS power by 8.7--10.4\% and energy capacity by 47.5--62.6\% relative to conventional ramp-rate control while maintaining the grid-codes compliance.
- 中文摘要
本文提出了一种约束感知强化学习(CARL)控制器,用于电池储能系统(BESS)尺寸和光伏(PV)爬坡速率平滑。云层转变引起的快速光伏功率波动可能在电力系统中带来运行和稳定性挑战。为减轻这些影响,电网编码对注入电网的光伏电力施加了爬坡速率限制(RRL)。BESS可以通过在快速变化时吸收或供电来支持此类电网编码。然而,BESS减少光伏诱导变异的程度取决于(i) BESS的功率和能量容量;以及(ii) 所采用的控制器。因此,BESS的尺寸与控制器设计是耦合的。传统的移动平均控制因滤波器引起的滞后而增加电池使用量,而爬坡率控制是被动的,不考虑动作对电池充电状态和未来爬坡事件的长期影响。基于优化的方法可以通过预测未来光伏变化来降低BESS需求,但其性能依赖于准确的光伏预测。为解决这些限制,CARL控制器从历史光伏数据中学习因果状态反馈策略,而无需部署时的预报。将斜坡速率、变换功率和充电状态约束纳入因果可行区间,保证可行性。基于全视野输入凸神经网络(ICNN)的优化被开发为最低可实现BESS要求的非因果基准。使用三个真实世界光伏数据集的评估显示,在10%/分钟的斜坡速率极限下,CARL相较于传统斜坡速率控制,峰值BESS功率降低8.7-10.4%,能量容量降低47.5-62.6%,同时保持网格编码合规性。
CoTTAGE: An Explainable Judge for Text-to-Audio Generation
CoTTAGE:文本转音频生成的可解释裁判
- Authors: Yu-Xiang Lin, Yun-Man Hsu, Bo-Ru Lu, Kuan-Po Huang, Wenze Ren, Ming-To Chuang, Chun-Yi Kuan, Jiawei Du, Byeonggeun Kim, Qingming Tang, Chieh-Chi Kao, Ankur Gandhe, Chao Wang, Hung-yi Lee
- Subjects: Subjects:
Sound (cs.SD); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.17644
- Pdf link: https://arxiv.org/pdf/2611.17644
- Abstract
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
- 中文摘要
文本转音频(TTA)生成的快速进展要求可靠且可解释的自动评估。然而,现有指标未能同时提供与人类判断高度相关性和有意义、基于音频的解释,迫使依赖劳动密集型人工评估,从而限制模型开发迭代。为此,我们提出了CoTTAGE,一种基于推理的文本相关性(REL)和音频质量评估器(OVL)。遵循理由-然后预测的范式,CoTTAGE首先生成基于音频的逻辑来诊断错误,然后预测得分。CoTTAGE依靠超过1万条新收集的人类OVL注释,先通过监督微调训练,随后通过基于Thurstone的偏好目标进行强化学习,将采样分数分布转换为成对偏好概率,并奖励与人类判断排名的一致性。在涵盖REL和OVL的六个基准测试中,CoTTAGE实现了与人类判断的先进相关性,同时生成了人类偏好的解释,加速TTA的发展。
Mobile Edge Deployment and Resource Management for Maritime Wireless Networks
海事无线网络的移动边缘部署与资源管理
- Authors: Chaoyue Zhang, Bin Lin, Ziru Chen, Lin X. Cai, Jianli Duan
- Subjects: Subjects:
Networking and Internet Architecture (cs.NI)
- Arxiv link: https://arxiv.org/abs/2611.17779
- Pdf link: https://arxiv.org/pdf/2611.17779
- Abstract
Mobile Edge Computing (MEC) has been envisioned as one of the key technologies for supplying computation and storage resources in Internet of Vessels (IoV) networks. Due to its flexible deployment, low cost and agile maneuverability, Unmanned Surface Vehicle (USV) has emerged as a promising solution, to provide communication and computation services for maritime users. In this paper, we study mobile edge deployment and resource management for MEC-assisted maritime wireless networks where USVs with diverse computation resources are deployed to provide edge computing services that complement the cloud-based services. To this end, we formulate an optimization problem to minimize the expected response time by jointly optimizing the deployment of mobile USVs and computation offloading decisions. To solve the mixed-integer nonlinear program problem, we propose a Dual-Layer Reinforcement Learning (DLRL) framework to attain a near-optimal solution. Specifically, a Deep Deterministic Policy Gradient (DDPG) algorithm is designed to obtain the best USV deployment in the outer layer learning, and a Q-learning algorithm is designed to determine the best computation offloading decisions in the inner layer learning. Numerical results demonstrate that the proposed solution outperforms some literature algorithms by effectively handling both continuous and discrete variables.
- 中文摘要
移动边缘计算(MEC)被设想为船舶互联网(IoV)网络中提供计算和存储资源的关键技术之一。由于其灵活部署、低成本和灵活机动性,无人水面载具(USV)已成为一种有前景的解决方案,用于为海上用户提供通信和计算服务。本文研究了MEC辅助海事无线网络的移动边缘部署与资源管理,这些网络部署拥有多样化计算资源的USV,以提供补充云服务的边缘计算服务。为此,我们提出了一个优化问题,通过联合优化移动USV的部署和计算卸载决策,以最小化预期响应时间。为解决混合整数非线性程序问题,我们提出了一个双层强化学习(DLRL)框架,以实现近乎最优的解决方案。具体来说,深确定性策略梯度(DDPG)算法旨在获得外层学习中最佳的USV部署,Q-学习算法则用于确定内层学习中最佳的计算卸载决策。数值结果表明,所提出的解决方案通过有效处理连续变量和离散变量,优于部分文献算法。
AutoRL: A Multi-Agent System towards Autonomous Reinforcement Learning
AutoRL:一个多智能体系统,迈向自主强化学习
- Authors: Simon Tang, Wenjie Shi, Yongjing Yin, Keheng Wang, Jialin Liu, Zheng Zhang, Linvo Lin, Xiangyuan Xue, Kan Ren, Jingang Wang, Xunliang Cai
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.17790
- Pdf link: https://arxiv.org/pdf/2611.17790
- Abstract
Recursive self-improvement (RSI) is one of the most promising paths toward an AI singularity for superintelligence. Achieving this vision requires AI systems to undertake the full training pipeline for iterative improvement. Reinforcement learning (RL) plays an essential role in this cycle to improve model capabilities and generalization, yet automating RL remains difficult due to unstable training dynamics and the need for sustained supervision. We introduce AutoRL, a multi-agent system for automating RL training, as a cornerstone for future RSI. Different from conducting research by end-to-end complete experiments, AutoRL monitors the evolving training state and integrates process-level supervision and research into an ongoing RL training, following the practical workflow of an RL researcher. Specifically, AutoRL consists of three RL-specific agentic roles. Perceptors track training dynamics, rollouts, evaluation, and efficiency, triggering deliberation when they detect anomalies, degradation, or stagnation. Three complementary Proposers diagnose problems and refine candidate interventions through shared evidence and checks. The Orchestrator reviews their recommendation and applies approved changes through online hot updates or checkpoint-based resumption. Experiments on Qwen3 and LongCat-Flash-Lite demonstrate the superiority of AutoRL over static RL recipes and a general agent. The research records show that existing agents can handle unattended RL supervision and dynamic adjustment. The difficulty for future RSI lies in developing fundamental methods that are valuable for resource-consuming validation.
- 中文摘要
递归自我改进(RSI)是实现人工智能超级智能奇点的最有前景路径之一。实现这一愿景需要AI系统承担完整的训练流程以实现迭代改进。强化学习(RL)在这一周期中发挥着提升模型能力和泛化性的关键作用,但由于训练动态不稳定和持续监督的需求,自动化强化学习仍然困难。我们介绍AutoRL,一种多智能体的强化学习自动化系统,作为未来RSI的基石。不同于通过端到端完整实验进行研究,AutoRL监测不断演变的训练状态,并将过程级监督和研究整合进持续的强化学习训练中,遵循强化学习研究者的实际工作流程。具体来说,AutoRL包含三个强化学习专属的智能体角色。感知器跟踪训练动态、部署、评估和效率,检测异常、退化或停滞时触发审议。三位互补提案者通过共享证据和检查诊断问题并优化候选干预。编排者审查其建议,并通过在线热更新或基于检查点的恢复应用批准的变更。Qwen3和LongCat-Flash-Lite实验证明AutoRL优于静态强化学习配方和通用代理。研究记录显示现有代理能够处理无人监管强化学习监督和动态调整。未来RSI的难点在于开发对耗费大量资源的验证有价值的基础方法。
MASTARS: Multi-Agent Sequential Trajectory Augmentation with Return-Conditioned Subgoals
MASTARS:多智能体顺序轨迹增强,含返回条件子目标
- Authors: Jiwon Jeon, Myungsik Cho, Woojun Kim, Seongmin Kim, Woohyeon Byeon, Seungyul Han, Youngchul Sung
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.17793
- Pdf link: https://arxiv.org/pdf/2611.17793
- Abstract
The performance of offline Multi-Agent Reinforcement Learning (MARL) is limited by the quality of the fixed offline dataset, motivating trajectory augmentation. However, augmenting multi-agent trajectories presents a fundamental challenge of the diversity-coordination trade-off: independent generation improves diversity but lacks coordination, while joint generation induces coordination but lacks diversity, reproducing the training distribution. To address this challenge in multi-agent data augmentation, we introduce MASTARS, a novel diffusion-based framework that generates multi-agent trajectories sequentially across agents while enforcing cross-agent consistent coordination via the method of inpainting. MASTARS produces trajectories that are diverse, coordinated and globally coherent. Experiments on various benchmarks such as MPE, SMAC, and SMAC-v2 demonstrate significant performance gains across multiple offline MARL methods.
- 中文摘要
离线多智能体强化学习(MARL)的性能受限于固定离线数据集的质量,促使轨迹增强。然而,增强多智能体轨迹带来了多样性与协调权衡的根本挑战:独立生成提升多样性但缺乏协调性,而联合生成则诱导协调但缺乏多样性,重现训练分布。为解决多智能体数据增强中的这一挑战,我们引入了MASTARS,一种基于扩散的新型框架,该框架通过内饰方法在智能体间顺序生成多智能体轨迹,同时通过内饰方法强制执行跨智能体一致协调。MASTARS生成的轨迹多样、协调且全局一致。在MPE、SMAC和SMAC-v2等多种基准测试上的实验显示,多种离线MARL方法均显著提升性能。
PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
PaperCritic:培训AI审稿人评估论文的合理性
- Authors: Hyun Ryu, Ji Yong Cho, Eunho Yang, Moontae Lee, June Yong Yang, Sean Welleck, Carolyn Rose
- Subjects: Subjects:
Computation and Language (cs.CL)
- Arxiv link: https://arxiv.org/abs/2611.17834
- Pdf link: https://arxiv.org/pdf/2611.17834
- Abstract
As AI conference submissions grow rapidly, AI reviewers are being considered as an alternative. However, simply imitating human reviewing is problematic because reviewing serves multiple purposes, including evaluating papers' soundness, interestingness, and clarity, the last two of which are subjective. In this work, we focus solely on evaluating soundness, the most important and objective criterion, and define the learning objective as jointly maximizing (1) precision, based on the soundness and substantiveness of review points, and (2) recall, the coverage of high-quality reference review points. Under this problem formulation, we construct a quality-labeled review point dataset, PaperSound-23K, using 1K ICLR submissions and 23K human-written and LLM-generated review points. Using this dataset, we train an AI reviewer model, PaperCritic, through reinforcement learning with the F1 score of precision and recall as the reward. Our experiments show that PaperCritic outperforms frontier models and models trained with other AI reviewer training methods on an in-domain test set. Surprisingly, learning to review ICLR submissions also transfers to reviewing Nature-family papers and even to out-of-domain reviewing tasks (e.g., code review, legal contract review, and feedback on story writing).
- 中文摘要
随着AI会议投稿的快速增长,AI评审者被考虑作为替代方案。然而,单纯模仿人类评审存在问题,因为评审有多重目的,包括评估论文的可靠性、趣味性和清晰度,后两者是主观的。本研究专注于评估最重要和客观的标准——合理性,并将学习目标定义为:(1)基于审稿点的合理性和实质性,以及(2)回忆性,即高质量参考审稿点的覆盖度。基于该问题表述,我们构建了一个质量标记的评审点数据集PaperSound-23K,包含1000个ICLR投稿和2.3万个人工撰写和大型语言模型生成的评审点。利用该数据集,我们通过强化学习训练一个AI评审模型PaperCritic,奖励以F1的准确性和回忆得分。我们的实验显示,PaperCritic在领域内测试集中表现优于前沿模型及用其他AI审稿人训练方法训练的模型。令人惊讶的是,学习如何审阅ICLR投稿也能转化为审阅Nature家族论文,甚至适用于域外审稿任务(如代码审查、法律合同审查和故事写作反馈)。
When Cooperation Must Change: Semantic Strategy Unlearning in Multi-Agent Reinforcement Learning
当合作必须改变:多智能体强化学习中的语义策略去学习
- Authors: Wenhan Wu, Linghan Chen, Yudong Gao, Fan Yang, Honglong Chen, Kai Yin, Jiawei Jiang, Chuang Hu
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.17917
- Pdf link: https://arxiv.org/pdf/2611.17917
- Abstract
Cooperative multi-agent reinforcement learning (MARL) supports coordinated decision-making in distributed Web services, multi-robot control, and collaborative games through learned coordination conventions. When deployment requirements change, a previously useful convention may become obsolete or unsafe. For example, revised cache-sharing permissions in Web content delivery may require caches to abandon a learned division of storage and serving roles while maintaining efficient delivery. Teams must therefore remove the specified convention while preserving useful cooperation. This is challenging because the same relational pattern can be reproduced by different agents across different histories. Runtime shielding leaves the underlying coordination preference intact and can impair task progress, while existing reinforcement unlearning and trajectory-deletion methods do not directly target role-invariant conventions. This paper formalizes this problem as semantic coordination-strategy unlearning and proposes this http URL uses causal coordination attribution to identify influential interactions and retain-aware subspace updates to limit interference with unrelated behaviors. Counterfactual re-coordination helps agents learn productive alternatives, while adversarial auditing tests whether the target convention can still be recovered. Accepted updates are merged into decentralized actors for direct deployment. Evaluation across five cooperative MARL benchmarks shows that CARVE reduces mean nominal convention-completion and worst-case elicitation rates by 91.4\% and 83.3\%, this http URL median utility loss is 5.9\%, and measured inference latency matches that of the original policies.
- 中文摘要
合作式多智能体强化学习(MARL)通过学习的协调约定支持分布式Web服务、多机器人控制和协作游戏中的协调决策。当部署需求发生变化时,原本有用的惯例可能变得过时或不安全。例如,网页内容分发中修订后的缓存共享权限可能要求缓存放弃存储和服务角色的学习划分,同时保持高效的传递。因此,团队必须在保留有用合作的同时移除指定的约定。这具有挑战性,因为同一关系模式可以被不同历史的代理重现。运行时屏蔽保留了底层的协调偏好,可能影响任务进展,而现有的强化复学和轨迹删除方法则不直接针对角色不变约定。本文将该问题形式化为语义协调策略的去学习,并提出该 http URL 利用因果协调归因识别有影响力的交互,并保留有意识的子空间更新以限制对无关行为的干扰。反事实性重新协调帮助智能体学习有生产力的替代方案,而对抗性审计则测试目标约定是否仍可恢复。被接受的更新会合并到去中心化的行为体中进行直接部署。对五个合作式 MARL 基准测试的评估显示,CARVE 将平均名义约定完成率和最坏情况引发率降低了 91.4% 和 83.3%,该 http URL 中位效用损失为 5.9%,且测量的推理延迟与原始策略一致。
Reusing Locomotion Training Recipes Across Humanoids via Body-Aligned Virtual Joints
通过身体对齐的虚拟关节,在类人生物间重复使用运动训练配方
- Authors: Sol Choi, Mingon Kim, Seokin Seo, Jongeun Choi, Donghyeon Kim, Yisoo Lee
- Subjects: Subjects:
Robotics (cs.RO)
- Arxiv link: https://arxiv.org/abs/2611.18003
- Pdf link: https://arxiv.org/pdf/2611.18003
- Abstract
Reinforcement learning has enabled robust humanoid locomotion, yet training recipes (rewards, architectures, hyperparameters) remain hard to reuse across robots with different native joint coordinates, since policies output native joint angles while action scales, bounds, gains, and penalties are defined in those robot-specific coordinates. Existing transmission models convert native targets into actuator commands but do not standardize the coordinate interface exposed to the policy. We introduce a body-aligned virtual joint space upstream of this mapping: policy actions and observations follow a common anatomical convention, and a deterministic kinematic layer maps virtual targets to native ones, leaving the existing transmission and low-level control stack unchanged. Under a single unmodified recipe on three humanoids with different leg-axis layouts, it is the only interface that tracks accurately, keeps the feet aligned with the travel direction, and trains stably on all three, and on Unitree G1 hardware it reduces heading drift by nearly a factor of six.
- 中文摘要
强化学习使人形运动成为可行的,但训练配方(奖励、架构、超参数)在不同原生关节坐标的机器人间仍难以重复使用,因为策略输出的是原生关节角度,而动作的尺度、界限、收益和惩罚则由机器人特定坐标定义。现有传输模型将原生目标转换为执行器命令,但未标准化策略中暴露的坐标接口。我们在该映射上游引入了身体对齐的虚拟关节空间:策略动作和观测遵循共同的解剖惯例,确定性运动学层将虚拟目标映射到原生目标,保持现有传输和低级控制栈不变。在三个腿轴布局不同的人形机器人上,使用一个未修改的配方,它是唯一一个能准确追踪、保持脚步与行进方向对齐、且三者都能稳定训练的界面,在Unitree G1硬件上,它能将航向漂移减少近六倍。
RL$^{\mathrm{RT}}$: On-Device Reinforcement Learning Runtime
RL$^{\mathrm{RT}}$: On-Device Reinforcement Learning Runtime
- Authors: Zexin Li, Xinwei Gao, Arvind Easwaran
- Subjects: Subjects:
Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.18035
- Pdf link: https://arxiv.org/pdf/2611.18035
- Abstract
On-device reinforcement learning runs a periodic actor and a learner on one embedded GPU shared with other workloads. Because the learner produces the actor's weights, existing designs stall the actor or let its policy drift. We present RL$^{\mathrm{RT}}$, an on-device runtime that makes this coupling a tunable parameter: a refresh bound limits actor releases between refreshes, and a GPU permit with slack-aware admission prioritises the actor and fills spare time with training atop any base scheduler. Under the real refresh cost on two Jetson boards, RL$^{\mathrm{RT}}$ keeps a 16.8 M-parameter policy's deadline misses below 1%, against 85 to 100% without it.
- 中文摘要
设备内强化学习在一个嵌入式GPU上运行周期性演员和一个学习器,GPU与其他工作负载共享。由于学习器产生演员权重,现有设计会阻碍演员或策略漂移。我们提出RL$^{\mathrm{RT}}$,一个设备端运行时,使此耦合成为可调节参数:刷新限制限制演员在刷新间的释放,GPU许可带Slack A允许优先级,利用空闲时间在任何基础调度器上进行训练。在两块Jetson板的实际刷新成本下,RL$^{\mathrm{RT}}$使16.8百万参数策略的截止日期低于1%,而无此限制则为85%至100%。
Any Channel Count, One Token Stream: Incorporating Spatial Audio Understanding and Reasoning into Large Audio-Language Models
任意通道数量,一个令牌流:将空间音频理解与推理融入大型音频语言模型
- Authors: Yuhuan You, Wenxiang Guo, Zhifang Guo, Yuxuan Wang, Jin Xu
- Subjects: Subjects:
Sound (cs.SD)
- Arxiv link: https://arxiv.org/abs/2611.18112
- Pdf link: https://arxiv.org/pdf/2611.18112
- Abstract
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.
- 中文摘要
音频相位关系编码源位置,但现有大型音频语言模型(LALM)将音频简化为单声道内容以稳定一般音频理解,从而使方向、距离和源关系问题的空间信息未被充分利用。我们介绍SpheraAudio,可接受任何声道计数的录音——单声道、立体声、一阶Ambisonics(FOA)或更高——并将其映射为单一令牌流。空间编码器将复杂的多声道频谱转换为空间令牌,并与对应时间位置的音频令牌交错。空间多模态RoPE(SMRoPE)使每个令牌的模态身份和时间槽位在不改变语言模型原始RoPE的情况下,旋转注意力可见。训练分为三个阶段——空间编码器预训练、投影仪对齐和联合监督与强化学习——从几何监督逐步进入基于语言的空间决策,答案通过明确的空间思维链训练:在听到的场景中逐步进行几何推理。当前基准测试将问题与固定的通道计数格式绑定,并在无空间推理证据的情况下评分答案。我们引入SpheraQA,涵盖单声道、立体声和FOA录音,并按层次组织问题,从单一属性感知到跨源关系再到多步推理,程序化评分,结合模拟场景几何和真实录制注释,并附有逐步解答参考。在SpheraQA和公共空间音频基准测试中,SpheraAudio为空间音频理解树立了新的性能标准。它支持高精度定位、跟踪和推理静态和移动声源,同时保持强大的通用音频能力。我们将发布代码、SpheraQA和评估指标以支持可重复性。
A Reinforcement Learning Approach to Space-Filling Input Design for Nonlinear System Identification
非线性系统识别空间填充输入设计的强化学习方法
- Authors: Charly Witmeur, Gonçalo G. Cruz, Maarten Schoukens, Mark C. Runacres, Jan Decuyper
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.18229
- Pdf link: https://arxiv.org/pdf/2611.18229
- Abstract
In this paper, we propose a reinforcement learning (RL) approach to space-filling input design for nonlinear system identification, in which an agent learns a control policy that actively drives the system to explore its state space. Classical input design, developed mainly for linear models, minimises the covariance of the estimated parameters. For nonlinear black-box models, however, the dominant error is the bias induced by the model structure over the region of interest, which motivates space-filling strategies that seek to cover the system state space. Existing space-filling approaches rely on predefined input signal classes and require solving explicit optimisation problems. Unlike these methods, the proposed approach requires no input class to be selected and no frequency band to be tuned for the system at hand, satisfies an amplitude bound by construction, and adapts to varying initial conditions through its observation-based policy without re-optimisation. The learned excitation signals exhibit physically meaningful behaviours, such as resonance excitation and amplitude modulation. The approach is evaluated against tuned swept sine and crest-factor-optimised multisine excitations, and against a state-of-the-art space-filling multisine design, on two nonlinear benchmark systems: a mass-spring-damper with nonlinear stiffness and a Van der Pol oscillator. Under the same amplitude bound, the learned policy achieves the best overall space-filling performance on both systems and distributes its samples more uniformly than the state-of-the-art design.
- 中文摘要
本文提出了一种强化学习(RL)方法来设计填充空间输入,用于非线性系统识别,其中代理学习控制策略,主动驱动系统探索其状态空间。经典输入设计主要针对线性模型,最大限度地减少估计参数的协方差。然而,对于非线性黑匣子模型,主导误差是模型结构对感兴趣区域引起的偏置,这促使填充空间策略试图覆盖系统状态空间。现有的填充空间方法依赖预定义的输入信号类,并需要解决显式优化问题。与这些方法不同,所提方法无需选择输入类,也无需为当前系统调谐频带,满足构造限制的振幅,并通过基于观察的策略适应变化的初始条件,无需重新优化。学习到的激发信号表现出物理意义上的行为,如共振激发和幅度调制。该方法在两个非线性基准系统上,分别针对调谐后掠正弦和峰位因子优化的多正弦激励,以及最先进的空间填充多正弦设计进行评估:具有非线性刚度的质量弹簧阻尼器和范德波尔振荡器。在相同的振幅限制下,所学策略在两个系统上实现了最佳的整体空间填充性能,且其样本分布比最先进设计更均匀。
Reinforcement Learning for Null-Space Exploitation in Pseudo-Inverse Control Allocation
伪逆控制分配中空空间利用的强化学习
- Authors: Umut Demir, Torbjørn Cunis, Aamir Ahmad, Walter Fichter
- Subjects: Subjects:
Systems and Control (eess.SY)
- Arxiv link: https://arxiv.org/abs/2611.18364
- Pdf link: https://arxiv.org/pdf/2611.18364
- Abstract
Pseudo-inverse methods are widely used to solve control allocation problems for overactuated aircraft with redundant control effectors. These methods reliably satisfy the primary command tracking objective, while the remaining null-space degrees of freedom are in principle available for secondary objectives such as drag reduction or actuator wear balancing. However, exploiting this freedom in practice requires hand-tuned weighting matrices or a priori preferred deflections that are difficult to specify across a wide flight envelope with nonlinear, state-dependent dynamics. This paper presents a hybrid architecture in which a reinforcement learning agent operates exclusively within the null space of the control effectiveness matrix, learning secondary objective strategies directly from interaction with the flight dynamics. The null-space projector guarantees by construction that primary command tracking remains invariant regardless of the agent output, providing a principled separation between certified primary control and adaptive secondary optimization. The approach is demonstrated on a nonlinear six-degree-of-freedom distributed electric propulsion aircraft model with aero-propulsive effects across diverse mission scenarios, achieving measurable drag reduction compared to the baseline allocation while preserving tracking integrity and respecting actuator constraints. The proposed architecture forms a foundation for future extensions toward multi-objective, mission-conditioned policies.
- 中文摘要
伪逆方法被广泛用于解决带有冗余控制效应器的过致动飞机的控制分配问题。这些方法可靠地满足主要指令跟踪目标,而其余零空间自由度原则上可用于次要目标,如阻力减少或执行器磨损平衡。然而,实际利用这种自由度需要手工调优的权重矩阵或先验偏好偏转,这些在宽广的飞行包线中,具有非线性、状态依赖动力学,难以实现。本文提出了一种混合架构,其中强化学习代理仅在控制效能矩阵的零空间内工作,直接通过与飞行动力学的交互学习次要目标策略。零空间投影仪通过构造保证主指令跟踪无论代理输出如何都保持不变,从而原则性地区分了认证的主控与自适应次要优化。该方法在非线性六自由度分布式电推进飞机模型上展示,具有多种任务场景中的气动推进效应,实现相较基线分配可测量的阻力减少,同时保持跟踪完整性并尊重执行器约束。该架构为未来向多目标、任务条件政策扩展奠定基础。
Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent
Gamer:一种面向远视线实时游戏代理的通用视觉-语言-行动基础模型
- Authors: Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren
- Subjects: Subjects:
Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.18397
- Pdf link: https://arxiv.org/pdf/2611.18397
- Abstract
We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.
- 中文摘要
我们介绍了 Gamer,这是一个为复杂交互环境中的长视野任务构建的通用代理框架。其端到端架构结合了感知、内存管理和多步推理。共享的训练和部署流水线支持与传统二维图形界面和复杂三维模拟世界的交互。我们首先在由数万小时设备级键盘鼠标交互构建的数据集上继续进行预训练。这些轨迹跨越模拟游戏和具身环境,使模型能够学习物理和虚拟领域的环境动态。稀疏思维指令调优大幅减少冗余计算,同时保留复杂决策所需的推理。我们用稳定的多回合强化学习方法训练长视野交互策略,该方法解决有限上下文窗口、累计滚动错误和稀疏奖励。在实时交互方面,云端推理引擎同时服务多个模拟游戏环境,支持无界上下文执行,并保持低在线推理延迟。在多款具有挑战性的3D开放世界游戏中,Gamer展现出扎实的泛化能力和与熟练玩家相当的运营效率。它还在一系列传统2D游戏基准测试中取得了竞争成绩。这些结果表明,即使是重复的键盘和鼠标轨迹,也能从数据扩展和算法学习中获得显著收益。在这些轨迹上的训练帮助Gamer在具有物理和数字属性的环境中更稳定地学习复杂技能。这种方法为实现跨数字和物理世界交互的通用智能体提供了路径。
ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models
ARCHER:从逻辑模仿到小语言模型中的可执行推理
- Authors: Dhruv Dixit, Khushi Patel, Congcong Xu, Min Song
- Subjects: Subjects:
Artificial Intelligence (cs.AI)
- Arxiv link: https://arxiv.org/abs/2611.18494
- Pdf link: https://arxiv.org/pdf/2611.18494
- Abstract
Small language models are attractive for local, low-cost, and privacy-sensitive deployment, but they remain unreliable on mathematical reasoning tasks that require decomposition, exact computation, tool use, and verification. We introduce \textbf{ARCHER}: \textbf{A}ction-trace \textbf{R}einforcement for \textbf{C}alculator-grounded \textbf{H}ierarchical \textbf{E}xecutable \textbf{R}easoning, a two-stage framework for training small language models (SLMs) to produce executable and auditable reasoning trajectories. In the first stage, ARCHER compiles free-form chain-of-thought rationales into typed action traces containing reasoning steps, calculator calls, deterministic runtime observations, verification actions, and final answers. This shifts the supervision target from imitating explanatory text to learning a tool-grounded reasoning policy. In the second stage, ARCHER refines this policy through asymmetric hierarchical reinforcement learning, using grouped and conditionally gated trajectory rewards for answer correctness, tool grounding, process discipline, and trace alignment. Across five standard sub-10B backbones, ARCHER improves over the strongest tool-augmented baseline by 5.51 percentage points on macro-average across benchmark accuracies, including gains of 8.95 points on MATH-500 and 5.33 and 5.34 points on AIME-2024 and AIME-2025. Evaluations on two additional reasoning-specialized 7B backbones further show that executable grounding remains beneficial after reasoning-focused reinforcement learning or distillation. These results demonstrate that executable supervision and reward-guided behavioral refinement provide a general and effective training paradigm for compact mathematical reasoners.
- 中文摘要
小型语言模型适合本地、低成本且隐私敏感的部署,但在需要分解、精确计算、工具使用和验证的数学推理任务中仍不可靠。我们引入了 \textbf{ARCHER}: \textbf{A}ction-trace \textbf{R}einforcement for \textbf{C}alculator Based \textbf{H}ierarchical \textbf{R}easoning,这是一个两阶段框架,用于训练小型语言模型(SLMs),生成可执行和可审计的推理轨迹。第一阶段,ARCHER 将自由形式的思维链理据编译为包含推理步骤、计算器调用、确定性运行时观察、验证动作和最终答案的类型动作痕迹。这将监督目标从模仿解释文本转变为学习基于工具的推理策略。第二阶段,ARCHER通过非对称层级强化学习,采用分组和条件门槛的轨迹奖励,用于正确性、工具基础、过程纪律和追踪对齐。在五个标准10B以下骨干中,ARCHER在基准准确度宏观平均上较最强工具增强基线提升5.51个百分点,包括MATH-500提升8.95个百分点,AIME-2024和AIME-2025提升5.33和5.34个百分点。对另外两个推理专用7B骨干的评估进一步表明,在推理导向强化学习或提炼后,可执行的基础依然有益。这些结果表明,可执行监督和奖励引导行为精炼为紧凑数学推理工具提供了通用且有效的训练范式。
On numerical issues of divergences between nearly equal distributions
关于近似相等分布之间散度的数值问题
- Authors: Brandon Amos
- Subjects: Subjects:
Machine Learning (cs.LG); Numerical Analysis (math.NA); Machine Learning (stat.ML)
- Arxiv link: https://arxiv.org/abs/2611.18511
- Pdf link: https://arxiv.org/pdf/2611.18511
- Abstract
Barron et al. (2026) recently identified numerical issues in the softmax and soft cross-entropy loss. We show that similar issues are present in standard divergence operations between categorical distributions. This includes the forward and reverse Kullback-Leibler (KL) divergence and generalized Jensen-Shannon divergence (JSD), as well as Monte Carlo KL estimates and entropy. When the distributions are nearly equal, rounding and cancellation issues affect the forward and gradient computations. These issues can affect settings using divergences in optimization objectives, such as knowledge distillation, reinforcement learning, and proximal methods. We propose numerically stable reformulations and provide example implementations for each setting, validated by property-based tests.
- 中文摘要
Barron 等人(2026)最近发现了软最大值和软交叉熵损失中的数值问题。我们表明,在类别分布之间的标准发散运算中也存在类似问题。这包括前向和反向 Kullback-Leibler(KL)发散以及广义 Jensen-Shannon 散度(JSD),以及蒙特卡洛 KL 估计和熵。当分布几乎相等时,四舍五入和抵消问题会影响前向和梯度计算。这些问题会影响使用发散的优化目标设置,如知识蒸馏、强化学习和近端方法。我们提出了数值稳定的重述,并为每个设置提供了示例实现,并通过基于属性的测试验证。
ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL
ModHarness-RL:将框架重新塑造为可学习且不断演进的智能强化学习程序
- Authors: Zelin He, Siyu Zhu, Jacob Helwig, Yujian Liu, Li An, Rohit Jain, Shiyu Chang, Alborz Geramifard
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.18533
- Pdf link: https://arxiv.org/pdf/2611.18533
- Abstract
Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.
- 中文摘要
代理强化学习(RL)越来越依赖于一个框架,即模型周围的支架,决定模型对自身交互的感知以及所获得的指导。大多数现有方法在手写框架下训练,且框架始终保持固定。然而,有效的框架依赖于领域、数据和模型,并且随着模型训练不断变化。我们引入了ModHarness-RL,它将框架重新塑造为从数据中学习并随着模型训练而进化的程序。ModHarness-RL将交互记录为结构化状态,一组类型块,框架的两个核心组件是基于这些模块的程序:一个压缩程序将累积历史组装到模型上下文中,另一个是引导程序,当行为异常时提供指导。在强化学习训练中,候选程序会从近期轨迹中提出,通过执行验证,并在过去轨迹上重放,从而追踪模型行为的变化。我们深入研究了CodeSearch,并进一步分析了ALFWorld和BIRD SQL,模型范围从1.5B到8B。提出的压缩和引导机制在大多数环境中提升了相较于固定束带基线的持续性能。这些结果表明,代理强化学习束带可以从任务数据中学习并随模型演进。
Imitation Bootstrapped Multi-Agent Reinforcement Learning
模仿引导多智能体强化学习
- Authors: Tony H. Zhang, Connor Mattson, Daniel S. Brown
- Subjects: Subjects:
Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA)
- Arxiv link: https://arxiv.org/abs/2611.18543
- Pdf link: https://arxiv.org/pdf/2611.18543
- Abstract
Demonstrations are an effective way to accelerate single-agent reinforcement learning (RL), but extending this idea to multi-agent reinforcement learning (MARL) is non-trivial as it is difficult and often impossible for a single person to teleoperate multiple agents simultaneously. Round-Robin Behavior Cloning (R2BC) addresses this by letting a single demonstrator control one agent at a time while the others execute their current policies. However, the resulting imitation policies are limited by the quality of the demonstrations. We propose Imitation Bootstrapped Multi-Agent Reinforcement Learning (IBMARL), a framework that extends Imitation Bootstrapped Reinforcement Learning (IBRL) to cooperative teams and uses R2BC to train the imitation policies. At each time step, every agent receives one action proposal from its imitation policy and one from its RL policy, and an ensemble of centralized critics selects among the per-agent combinations of the two. A gated imitation penalty, a soft Q-filtered behavior cloning loss, regularizes each RL policy toward its imitation policy only where the critics value the imitation action more, so that after training each agent executes its RL policy alone, without the centralized critics. On four cooperative multi-agent RL benchmarks, IBMARL surpasses its imitation policies $2.6$-$4.2\times$ faster than demonstration-based baselines on two tasks, is the only method to surpass it on the other two, and exceeds the demonstrator itself on two of the four. With real human demonstrations IBMARL still surpasses imitation learning alone, and does so more than twice as fast as the best baseline.
- 中文摘要
演示是加速单智能体强化学习(RL)的有效方法,但将这一理念推广到多智能体强化学习(MARL)并非简单,因为单人同时远程操作多个智能体是困难且常常不可能的。轮询行为克隆(R2BC)通过让单个演示者一次控制一个智能体,而其他智能体执行当前策略来解决这个问题。然而,最终的模仿策略受限于演示的质量。我们提出了一个模拟引导多智能体强化学习(IBMARL)框架,该框架将模仿引导强化学习(IBRL)扩展到合作团队,并使用R2BC训练模仿策略。在每个时间步,每个代理从其模仿策略中获得一个动作提案,从其强化学习策略中获得一个行动提案,一组集中批评者在两者中的每个智能体组合中选择。门控模仿惩罚——软Q滤波行为克隆损失,仅在批评者更重视模仿动作时,使每个强化学习策略趋向其模仿策略,这样每个智能体训练完成后,单独执行其强化学习策略,无需集中批评者。在四个合作多智能体强化学习基准测试中,IBMARL在两个任务中比基于演示的基线快了2.6美元-4.2美元\倍数,是另外两个任务中唯一超过该策略的方法,并且在其中两个任务中超过了演示者自身。在真实人类演示中,IBMARL仍然超过单纯模仿学习,且速度是最佳基线的两倍多。
Training Reasoning Models in Simulation to Natively Use Real Robots
模拟中训练推理模型以原生使用真实机器人
- Authors: Jiaming Tang, Zhuoyang Zhang, Junxian Guo, Fangzhou Zhao, Yilong Zhao, Zekai Li, Xingyang Li, Yuyang Zhao, Song Bian, Zhijian Liu, Song Han
- Subjects: Subjects:
Robotics (cs.RO); Machine Learning (cs.LG)
- Arxiv link: https://arxiv.org/abs/2611.18562
- Pdf link: https://arxiv.org/pdf/2611.18562
- Abstract
Robot-use agents (RUAs) have emerged as a promising approach to general-purpose robot control. However, strong performance has so far been achieved only by frontier proprietary models such as GPT-6 Astra and Claude Fable 5.2, whereas open-weight models remain substantially less capable. We show that an open-weight 27B reasoning model can be trained to use real robots natively through reinforcement learning, entirely in simulation and without demonstrations. Like a computer-use agent (CUA), the model observes the environment, reasons in a chain of thought, and generates the next end-effector pose, without any harness. We build a Post-training Engine for Agentic Robotics (PEAR) and use it to train the model on simulated robot tasks at scale with robot-use reinforcement learning (RURL), mirroring how frontier models acquire computer use. We find that the skills acquired in simulation generalize well beyond the training tasks: they transfer to an unseen robot, to the real world, and to novel instructions. After training on LIBERO and RoboDojo in simulation, the model succeeds in 60% of our trials on a real dual-arm robot, whereas the base model succeeds in none. On LIBERO, training raises the success rate from 1.6% to 74.4%, approaching GPT-6 Astra and GPT-6.1 Sol at maximum reasoning effort. Compared with $\pi_{0.5}$, a vision-language-action model fine-tuned on demonstrations, our model is substantially more robust to changes in the task itself. On LIBERO-Pro, $\pi_{0.5}$ succeeds in only 0.75% of episodes when assigned a new task in a familiar scene whereas our model succeeds in 57.4%. Finally, the model retains its general capabilities, with no measurable degradation on nine reasoning, coding and vision benchmarks. These results highlight robot-use reinforcement learning as a promising component of foundation-model post-training.
- 中文摘要
机器人使用智能体(RUA)已成为通用机器人控制的有前景方法。然而,迄今为止,强大的性能仅由GPT-6 Astra和Claude Fable 5.2等前沿专有模型实现,而开放权重模型的能力仍然明显较差。我们展示了开放权重27B推理模型可以通过强化学习,完全在模拟中、无需演示的情况下,原生训练使用真实机器人。类似于计算机使用智能体(CUA),该模型观察环境,进行思考链推理,生成下一个终端执行器姿态,无需任何束缚。我们构建了一个智能机器人后训练引擎(PEAR),并用它在大规模上通过机器人使用强化学习(RURL)训练模型进行模拟机器人任务,模拟前沿模型获得计算机使用的方式。我们发现,仿真中获得的技能远超训练任务:它们可以转移到看不见的机器人、现实世界以及新颖指令上。在 LIBERO 和 RoboDojo 模拟中训练后,模型在真实双臂机器人上的试验中成功率为 60%,而基础模型则无一成功。在 LIBERO 上,训练成功率从 1.6% 提升至 74.4%,在最大推理努力下接近 GPT-6 Astra 和 GPT-6.1 Sol。与 $\pi_{0.5}$ 相比,$\{0.5}$ 是一个经过演示微调的视觉-语言-动作模型,我们的模型对任务本身的变化具有显著更强的鲁棒性。在 LIBERO PRO 上,$\pi_{0.5}$ 在熟悉场景中被分配新任务时仅有 0.75% 的成功率,而我们的模型成功率为 57.4%。最后,模型保持了其一般能力,在九项推理、编码和视觉基准上没有明显下降。这些结果凸显了机器人使用强化学习作为基础模型后训练中一个有前景的组成部分。
Keyword: diffusion policy
When Does Cross-View Consistency Matter in Multi-View Appearance Augmentation? A Controlled Study for Visuomotor Policies
交叉视角一致性在多视角外观增强中何时重要?一项针对视觉运动保单的对照研究
- Authors: Zixuan Jiang
- Subjects: Subjects:
Robotics (cs.RO); Computer Vision and Pattern Recognition (cs.CV)
- Arxiv link: https://arxiv.org/abs/2611.17751
- Pdf link: https://arxiv.org/pdf/2611.17751
- Abstract
Multi-view visual augmentation often seeks consistency across cameras, but it is unclear which forms of consistency matter for downstream control. We use simulation-state replay to manipulate cross-view coupling while preserving the per-camera perturbation distributions, physical trajectories, and action labels. Across seven LIBERO manipulation tasks, marginal-matched changes in global appearance, ghost injection, and target color have small average costs for a late-fusion Diffusion Policy: point estimates are near zero or negative, and the largest upper endpoint of the 90% posterior credible intervals is 2.55 percentage points. In contrast, geometrically anti-coupled observations reduce success by 3.30 percentage points relative to coupled observations (90% CrI [0.64, 6.28]), with displacement magnitude, direction marginals, and the number of affected views matched. Camera interventions reveal substantial dependence on wrist-view geometry, while feature analysis shows appearance sensitivity patterns associated with the factors randomized during training. ACT shows a higher target-color cost estimate than DP, with uncertainty that includes zero and sensitivity to task composition. Within the tested tasks and policy configurations, these findings motivate selective consistency constraints: appearance coupling can sometimes be relaxed, whereas task-relevant geometry and cross-view spatial correspondence warrant protection.
- 中文摘要
多视角视觉增强通常寻求摄像头间的一致性,但尚不清楚哪些一致性形式对后续控制至关重要。我们使用模拟状态回放来操控交叉视角耦合,同时保留每摄像头的扰动分布、物理轨迹和动作标签。在七个LIBERO操作任务中,全局外观的边际匹配变化、幽灵注入和目标颜色对晚融合扩散政策的平均成本较小:点估计接近零或负数,90%后验可信区间的最大上端点为2.55个百分点。相比之下,几何反耦合观测相比耦合观测,成功率降低3.30个百分点(90% CrI [0.64, 6.28]),位移大小、方向边际和受影响视角数量均匹配。摄像头干预显示对手腕视角几何结构有显著依赖,而特征分析则显示与训练中随机化因素相关的外观敏感性模式。ACT显示目标颜色成本估计高于DP,不确定性包括零和任务组合敏感性。在测试任务和策略配置中,这些发现促使选择性一致性约束:外观耦合有时可以放宽,而任务相关几何和交叉视角空间对应则值得保护。