生成时间: 2026-08-29 03:58:07 (UTC+8); Arxiv 发布时间: 2026-08-28 20:00 EDT (2026-08-29 08:00 UTC+8)

今天共有 29 篇相关文章

Keyword: reinforcement learning

The Artificial Experimentalist: Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning

人工实验者：通过自我强化学习发现与控制自组织现象

• Authors: Marko Cvjetko, Benedikt Hartl, Michael Levin, Clément Moulin-Frier, Pierre-Yves Oudeyer
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.26116
• Pdf link: https://arxiv.org/pdf/2608.26116
• Abstract Existing methods for exploring cellular automata and other complex systems mostly operate in open loop: they set initial conditions, execute a full simulation, and observe the outcome, without intervening during execution. We introduce a closed-loop framework based on autotelic reinforcement learning, in which an agent autonomously samples diverse goals and learns a goal-conditioned policy to intervene in a complex system through minimal, local perturbations. We instantiate this framework on Lenia, a continuous cellular automaton known for life-like self-organizing patterns, in an agentic system we call CARL, and demonstrate three capabilities. First, CARL discovers stable solitons across a wide range of Lenia update rules at a higher rate than heuristic baselines. Second, it learns to steer the movement direction of existing solitons with few interventions, showing that CARL can control self-organizing patterns, not only create them. Third, humans can use trained agents to guide solitons through maze environments in real time by specifying high-level directional commands that the agent translates into low-level interventions. Trained across diverse goals, update rules, and random initial states, the agents acquire policies that generalize zero-shot to various out-of-distribution conditions. These results suggest a path toward artificial experimentalist agents that, autonomously or with human guidance, discover and control emergent phenomena in complex systems.
• 中文摘要 现有探索元胞自动机及其他复杂系统的方法大多处于开环状态：它们设置初始条件，执行完整模拟，观察结果，执行过程中不干预。我们引入基于自代强化学习的闭环框架，智能体自主采样多样目标，学习目标条件策略，通过最小局部扰动干预复杂系统。我们将该框架实例化在Lenia上，Lenia是一个以逼真的自组织模式著称的智能系统，称为CARL，并展示了三项能力。首先，CARL在多种Lenia更新规则中发现稳定孤子的速度高于启发式基线。其次，它学会以少量干预引导现有孤子的运动方向，证明CARL不仅能创造，还能控制自组织模式。第三，人类可以通过训练有素的智能体，通过指定高级别的方向指令，实时引导孤子穿越迷宫环境，智能体将其转化为低层次干预。经纪体在多样化目标、更新规则和随机初始状态下训练，能够获得将零射击推广到各种分布外条件的策略。这些结果为实现人工实验性智能体的路径提供了一条路径，这些智能体能够自主或在人类指导下，发现并控制复杂系统中涌现的现象。

Reward-Informed Sparse Autoencoders and the Solution-Completeness Confound

奖励知情的稀疏自编码器与解完全性混淆

• Authors: Tanvi Nagilla, Alexander Jameson, Daniel Manta, Shayaan Uddin
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.26136
• Pdf link: https://arxiv.org/pdf/2608.26136
• Abstract Sparse autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, and an appealing way to aim them at reasoning is to curate their data with a signal reinforcement learning already produces: the reward. We build such a reward-informed SAE (RI-SAE): we split GRPO trajectories into high-reward ("good") and low-reward ("bad") reasoning continuations, train a standard JumpReLU SAE on their activations, and then ask what the resulting good/bad separation actually measures. On Llama-3.1-8B a sparse subset of the 16,384 features does separate the classes (silhouette 0.79 on the selected features versus 0.005 for the full code), but a control battery shows the separation is largely solution completeness rather than reasoning quality: a TF-IDF text classifier already splits the classes (AUC 0.75--0.83), and three structural cues alone (length, a closed reasoning block, and a boxed answer) reach AUC 0.70 (99% of good versus 69% of bad completions are boxed). A generic SAE that never saw the reward does not separate the classes at all (silhouette 0.01, no discriminative features), so the 0.79 is in-sample fitting of this curated signal rather than structure that a reward-blind dictionary recovers. We therefore present the recipe and its control battery together: reward filtering is a cheap, label-free way to reuse RL signals for interpretability, but most of what it surfaces is completion form. Two discriminative features are still readable (symbolic mathematics; procedural and evaluative language), which we take as illustrative rather than as isolated reasoning.
• 中文摘要 稀疏自编码器（SAE）将语言模型激活分解为稀疏且可解释的特征，一种吸引人的推理方式是用学习已产生的信号强化——奖励来策划数据。我们构建了这样一个以奖励为导向的SAE（RI-SAE）：我们将GRPO轨迹分为高奖励（“好”）和低奖励（“坏”）的推理延续，训练标准的JumpReLU SAE以激活情况，然后探究最终的好坏分离实际测量了什么。在Llama-3.1-8B中，16,384个特征中的稀疏子集确实将类别分隔开（选定特征的轮廓为0.79，而完整代码为0.005），但控制电池显示这种分离主要取决于解的完备性，而非推理质量：TF-IDF文本分类器已经将类别拆分（AUC 0.75--0.83），仅有三个结构线索（长度、封闭推理块和框式答案）达到AUC 0.70（99%的良好对69%的坏补全被框住）。一个从未见过奖励的通用SAE根本不会区分类别（轮廓0.01，无判别特征），因此0.79是对该策划信号的样本内拟合，而非奖励盲词典恢复的结构。因此，我们将配方及其控制电池一同呈现：奖励过滤是一种廉价、无标签的强化学习信号可解释性重用方式，但其呈现的大部分内容是完备形式。仍有两个判别特征可读（符号数学;程序语言和评估语言），我们将其视为说明性推理而非孤立推理。

AdaThinking-E: One-Token Entropy Regulation for Adaptive Thinking

AdaThinking-e：适应性思维的单代币熵调控

• Authors: Zining Wang, Tongkun Guan, Boming Chen, Zhentao Guo, Jianqiang Liu, Chao Jin, Chen Duan, Kai Zhou, Pengfei Yan, Wei Shen, Xiaokang Yang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.26141
• Pdf link: https://arxiv.org/pdf/2608.26141
• Abstract Multimodal large language models have demonstrated strong document reasoning capabilities by incorporating explicit thinking processes. While this capability significantly improves performance on challenging tasks, current models apply such deep reasoning uniformly to all questions, resulting in unnecessary computational overhead for simple task. This not only degrades user experience but also negatively impact accuracy on benchmark datasets. We identify the critical need for adaptive thinking mechanisms that can intelligently determine when to engage reasoning based on question complexity. To address this, we propose AdaThinking-E, a novel reinforcement learning framework that learns adaptive thinking through one-token entropy regulation. Our key insight is that model confidence in the decision to engage thinking (or not) can be quantified through entropy analysis of the predicted probability distribution at critical decision tokens. This observation motivates our entropy-governed reward mechanism: the training process naturally transitions from high-entropy exploration, where the model experiments with different thinking strategies, to low-entropy convergence with confident, generalizable decision-making policies. Crucially, this approach enables models to intrinsically discover when to think without requiring manual intervention or external difficulty labels. Extensive experiments demonstrate that our approach enables models to be both accurate on complex problems and efficient on simple ones across diverse document tasks.
• 中文摘要 多模态大型语言模型通过整合显式思维过程，展现了强大的文档推理能力。虽然这一能力显著提升了在复杂任务中的表现，但当前模型对所有问题均一应用如此深度的推理，导致简单任务中不必要的计算开销。这不仅降低了用户体验，也负面影响了基准数据集的准确性。我们认识到对适应性思维机制的迫切需求，能够智能地根据问题复杂度判断何时进行推理。为此，我们提出了AdaThinking-E，一种通过单代币熵调控学习适应性思维的新型强化学习框架。我们的关键见解是，通过对关键决策代币预测概率分布的熵分析，可以量化模型对是否参与思考决策的信心。这一观察激励了我们由熵支配的奖励机制：训练过程自然地从高熵探索（模型尝试不同思维策略）过渡到低熵收敛，采用自信且可推广的决策策略。关键是，这种方法使模型能够内在地发现何时思考，而无需手动干预或外部难度标签。大量实验表明，我们的方法使模型能够在复杂问题上既准确，又能高效处理简单问题，跨越多样化的文档任务。

CARE: Causally-Aligned Reasoning Exploration for Medical Large Language Models

CARE：医学大型语言模型的因果对齐推理探索

• Authors: Yucheng Zhou, Peng Luo, Qianning Wang, Chengzhong Xu, Jianbing Shen
• Subjects: Subjects: Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.26147
• Pdf link: https://arxiv.org/pdf/2608.26147
• Abstract Large Language Models (LLMs) have shown strong potential for medical reasoning, yet the scarcity and cost of expert-annotated data constrain their progress. While reinforcement learning offers a scalable alternative, standard outcome-based methods in medicine often suffer from autoregressive credit assignment failure and gradient variance explosion. This leads to the "Right Answer, Wrong Reason" trap, where models inadvertently reinforce spurious correlations and dataset shortcuts rather than valid clinical deduction. In this work, we propose Causally-Aligned Reasoning Exploration (CARE), a theoretically grounded framework for intrinsic experience curation. CARE is built upon two rigorous conditions for high-quality training trajectories: Causal Sufficiency, which utilizes an agreement-based self-verification mechanism to mimic $do$-calculus interventions and effectively debias gradients; and Proximal Learnability, which employs dynamic entropy bounds to select experiences within the model's zone of proximal development for variance-bounded optimization. These rigorously filtered experiences are optimized via a dual-stream objective that combines on-policy group-relative exploration with difficulty-weighted experience replay. Extensive experiments on diverse medical multimodal and text-only benchmarks demonstrate that CARE consistently outperforms other strong competitors, substantially reducing correct-but-inconsistent reasoning and improving training stability.
• 中文摘要 大型语言模型（LLMs）在医学推理方面展现出强大潜力，但专家注释数据的稀缺性和成本限制了其进步。虽然强化学习提供了可扩展的替代方案，但医学中基于结果的标准方法常常面临自回归的学分分配失败和梯度方差爆炸的问题。这导致了“正确答案，错误理由”陷阱，模型无意中强化了虚假的相关性和数据集捷径，而非有效的临床推理。本研究提出因果对齐推理探索（CARE），这是一个基于理论基础的内在经验策划框架。CARE基于两个严格的标准来实现高质量训练轨迹：因果充分性，利用基于协议的自我验证机制模拟$do$微积分干预，有效去偏见梯度;以及近端可学习性，利用动态熵界限选择模型近端发展区内的经验，实现方差边界优化。这些严格过滤的经验通过双流目标优化，结合了策略组相对探索与难度加权经验回放。在多种医学多模态和纯文本基准测试上的广泛实验表明，CARE持续优于其他强劲竞争者，显著减少正确但不一致的推理，提升训练稳定性。

Predicting Consequences and Reinforcing Navigation Policies with Latent World Models

用潜在世界模型预测后果并强化导航政策

• Authors: Zengmao Wang, Wei Gao, Shuhan Shen
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.26190
• Pdf link: https://arxiv.org/pdf/2608.26190
• Abstract World models enable agents to reason about future outcomes and learn policies from their knowledge of state transition, but existing approaches primarily focus on reconstructing future observations or features, which introduces unnecessary complexity and limits their effectiveness for decision making. In this work, we propose a compatibility prediction Latent World Model (LWM) for robot navigation that predicts action-conditioned latent feature compatibility rather than reconstructing observations. Our key insight is that spatial proximity correlates with latent feature similarity, enabling action consequences to be evaluated directly in latent space. To support counterfactual training, our model leverages action sequences sampled across trajectories and learns to predict which sequences lead closer to the goal. Furthermore, we demonstrate how the learned world model can supervise policy learning from unlabeled video data and further improve policies through reinforcement learning entirely within the world model. This imagination-driven framework eliminates the need for action annotations and additional environment interaction. Extensive experiments on multiple real-world robot navigation datasets show that our approach significantly outperforms prior world model and imitation learning methods in prediction accuracy, policy learning, and real-world navigation performance. The code, pretrained models, and additional materials are available at this https URL.
• 中文摘要 世界模型使智能体能够推理未来结果并从状态转变知识中学习策略，但现有方法主要侧重于重建未来的观察或特征，这增加了不必要的复杂性，限制了其决策的有效性。本研究提出了一个兼容性预测潜在世界模型（LWM），用于机器人导航，预测动作条件潜在特征兼容性，而非重建观测。我们的关键见解是空间接近性与潜在特征相似性相关，使得行动后果能够直接在潜空间中评估。为支持反事实训练，我们的模型利用跨轨迹抽样的动作序列，学习预测哪些序列更接近目标。此外，我们展示了学习世界模型如何从未标记的视频数据中监督策略学习，并通过强化学习进一步改进策略，完全在世界模型内进行。这一想象驱动框架消除了对动作注释和额外环境交互的需求。对多个真实世界机器人导航数据集的广泛实验表明，我们的方法在预测准确性、策略学习和真实导航性能方面远远优于以往的模型和仿制学习方法。代码、预训练模型及其他材料可在此 https 网址获取。

AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes

AffectOmni：现实学习可验证的以人为中心的基于社会和艺术场景的情感推理

• Authors: Yibo Wang, Rui Yang, Jisheng Dang, Bimei Wang, Yitao Wu, Pengfei Cao, Wencan Zhang, Hong Peng, Bin Hu, Tat-Seng Chua
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.26193
• Pdf link: https://arxiv.org/pdf/2608.26193
• Abstract Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding, yet affective reasoning remains vulnerable to shortcut behavior. Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language, which weakens traceability and external verification. Prior reinforcement learning approaches mainly reward context or logical coherence without explicitly enforcing attention to human evidence. In addition, LLM as a Judge scoring often suffers from score clustering, which reduces reward discriminability. We propose AffectOmni, a GRPO trained framework for verifiable affective reasoning. AffectOmni introduces People Focus and Temporal Order rewards to encourage people-centric evidence selection and temporally structured reasoning, and it adopts within-group comparative scoring to produce more stable and discriminative reward signals. For verification, a Thinking Summarizer converts free form rationales into executable evidence instructions, which are grounded into pixel level evidence regions via SAM3 to provide an externally auditable interface outside the training loop. Experiments on IntentBench, Daily Omni, and WorldSense show consistent improvements over open source 7B scale baselines, including gains of 4.66% on emotion recognition and +14.29% on temporally sensitive tasks. Code is available at this https URL.
• 中文摘要 多模态大型语言模型（MLLMs）在VQA和场景理解方面表现出色，但情感推理仍易受捷径行为影响。模型可能预测正确答案，却忽视了以人为中心的线索，如微表情和肢体语言，削弱了可追溯性和外部验证。先验强化学习方法主要奖励上下文或逻辑连贯性，而未明确强制关注人类证据。此外，作为法官的LLM评分常常存在分数聚类问题，降低了奖励的可辨别性。我们提出了AffectOmni，这是一个由GRPO训练的可验证情感推理框架。AffectOmni引入了人为焦点和时间顺序奖励，以鼓励以人为本的证据选择和时间结构化推理，并采用组内比较评分，以产生更稳定和区分性的奖励信号。为验证，思维摘要器将自由形式的理由转换为可执行的证据指令，这些指令通过SAM3嵌入像素级证据区域，提供训练循环之外的外部可审计界面。在IntentBench、Daily Omni和WorldSense上的实验显示，相较开源7B尺度基线持续提升，包括情绪识别提升4.66%，时间敏感任务提升+14.29%。代码可在此 https 网址获取。

SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning

SpeechGym：一个通过强化学习培训语音代理的音频原生健身房

• Authors: Jiajun Fan, Jingyuan Li, Prashanth Gurunath Shivakumar, Jia-Hong Huang, Qi Luo, M. Maruf, Ivan Bulyko, Ge Liu, Roger Ren
• Subjects: Subjects: Sound (cs.SD); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.26432
• Pdf link: https://arxiv.org/pdf/2608.26432
• Abstract Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the loop stays local and trainable end to end. Audio agentic capability does not follow from audio understanding. The failures speech introduces are perceptual rather than reasoning deficits: the agent picks the right tool and the right argument slot but fills it with a value misheard from the waveform, and that single error cascades into a failed call, a retry of the same call, and a wasted step budget. A second failure is behavioural: under an insistent caller the agent performs an unauthorised write and ends the episode believing it helped. Both are trainable, because the environment labels them for free: a call with a misheard argument fails against the database while a correct one succeeds. The obstacle is sparsity, not signal. Outcome-only GRPO is gradient-starved here, since almost every rollout group fails identically, while a per-turn process reward crediting each successful tool call restores variance to nearly every group. Trained this way, the agent transfers with no further tuning to an independently implemented voice benchmark, more than doubling task success and carrying an open-weights model from last place to second on that leaderboard, while using fewer turns and tokens than before training.
• 中文摘要 语音代理必须调用工具并进行多回合对话，完全通过语音进行，但主流范式却在文本中训练它们。现有框架要么将TTS和ASR层叠放在专有语音API周围，梯度无法流动，每次通话成本高，使策略强化学习难以实现;要么停留在文本中：它们测量语音代理，但无法改进。我们呈现SpeechGym，一个音频原生代理环境，两个全模态模型在原生音频中对话，没有外部ASR或TTS，也无API边界，基于既定文本智能基准测试的未修改任务、工具和成功检查，使互动模态成为唯一变量，循环保持本地且可端对端训练。音频代理能力不依赖音频理解。语音带来的失败是感知缺陷而非推理缺陷：代理选对工具和正确的参数槽，但用波形误听的值填补，导致该错误连锁导致呼叫失败、同一呼叫重试和步数预算浪费。第二种失败是行为性：在坚持呼叫者情况下，代理执行未经授权的写入，并认为该写入有效。这两种情况都可以训练，因为环境会免费标记它们：一个听错的调用对数据库失败，而正确的则成功。障碍在于稀疏性，而非信号。仅结果的GRPO在这里缺乏梯度，因为几乎每个推广组的失败方式相同，而每回合奖励每个成功工具调用的过程，几乎恢复了所有组的方差。通过这种训练方式，代理无需进一步调整即可切换到独立实现的语音基准，任务成功率翻倍多，并以开放权重模型从排名最后升至第二名，同时使用比训练前更少的回合数和代币。

Active Curriculum Refinement for Reinforcement Learning

强化学习的主动课程精炼

• Authors: Zhenya Liu, Yuxin Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.26469
• Pdf link: https://arxiv.org/pdf/2608.26469
• Abstract In many reinforcement learning (RL) domains, environments are connected by prerequisite relations, such as difficulty-increasing edits or parameter increments, which induce a directed acyclic curriculum graph (DAG). Although this structure is often exploited only implicitly, explicitly modeling it can improve training. We introduce PATH, a curriculum-learning framework that performs active learning over the curriculum graph. PATH first expands coverage by sampling diverse curriculum paths and then reallocates training toward regions that remain unmastered. Experiments across diverse environments show that PATH explicitly leverages the graph structure to achieve strong robustness and generalization.
• 中文摘要 在许多强化学习（RL）领域中，环境通过先决关系相连，如难度增加的编辑或参数增量，从而诱导出定向无环课程图（DAG）。虽然这种结构通常仅被隐式利用，但显式建模可以改善训练。我们介绍了PATH，一种在课程图上进行主动学习的课程-学习框架。PATH首先通过抽样多样化的课程路径扩展覆盖范围，然后将训练重新分配到尚未掌握的区域。跨越多种环境的实验表明，PATH明确利用图结构实现了强的稳健性和泛化性。

Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning

共享行为者不必分享批评者：并行强化学习中价值不匹配的影响

• Authors: Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.26481
• Pdf link: https://arxiv.org/pdf/2608.26481
• Abstract When a single policy is trained in parallel across multiple environments of the same task, such as procedurally generated levels, randomized dynamics, or curricula, implementations commonly use one critic across all sampled environments. Yet different environments can assign different expected returns to the same input visible to the critic. A critic without environment information must then reconcile distinct value targets, systematically shifting the sampled advantages within individual environments. Using illustrative bandit models with multiple environments and a common optimal arm, we characterize how this value mismatch redistributes sampled policy updates, reinforcing unhelpful actions while attenuating or even reversing useful ones. The oracle processes using no baseline, the shared value, or the value specific to the sampled environment have the same mean logit update at a fixed policy and converge to the same optimal policy, yet their realized learning paths can differ sharply. The analysis motivates a minimal intervention: give only a logged environment index to the critic so that it can separate the value targets. Controlled CartPole and MuJoCo experiments expose the predicted shifted values, advantages, and performance gaps. In the more complex BipedalWalker and Procgen settings, the same intervention yields more stable learning and higher returns. Across all $16$ Procgen games, the multihead conditional critic improves aggregate normalized return on $600$ unseen levels per game by $40.8\%$. In conclusion, the theory identifies value mismatch as a direct mechanism through which critic sharing can degrade stochastic learning dynamics, not captured by scalar estimator variance alone, and the experiments show that conditioning on an index is broadly effective in parallel reinforcement learning.
• 中文摘要 当单一策略在多个环境中并行训练时，如程序生成的层级、随机动态或课程，实现通常会在所有抽样环境中使用一个批评者。然而，不同环境可能对批评者可见的同一输入赋予不同的期望回报。没有环境信息的批评者必须协调不同的价值目标，系统性地调整采样优势在个别环境中的分布。利用具有多个环境和共同最优臂的示例性盗贼模型，我们描述了这种值错匹配如何重新分配抽样策略更新，强化无益的行为，同时削弱甚至逆转有用的行为。使用无基线、共享值或特定采样环境值的预言机过程在固定策略下均值logit更新相同，并收敛到相同的最优策略，但它们实现的学习路径可能截然不同。该分析动机采取最小干预：仅向批评者提供一个记录的环境指数，以便将其分离价值目标。受控的CartPole和MuJoCo实验揭示了预测的偏移值、优势和性能差距。在更复杂的BipedalWalker和Procgen环境中，同样的干预能带来更稳定的学习和更高的回报。在所有16美元的Procgen游戏中，多头条件批评者每局在600美元未见水平上的总归一化收益提高了40.8%美元。总之，该理论将价值错配识别为批评者共享直接降低随机学习动态的机制，而该机制不仅仅由标量估计方差捕捉，但实验表明对指数的条件化在并行强化学习中总体有效。

Video-FLAIR: Not Whether to Reason, But How

视频标签：不是讲究是否讲道理，而是怎么讲

• Authors: Yogesh Kulkarni, Pooyan Fazli
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.26495
• Pdf link: https://arxiv.org/pdf/2608.26495
• Abstract Multimodal queries can require different types of reasoning. Some can be answered via perceptual reasoning, extracting information directly from the visual signal, while others require compositional reasoning that combines observations or deliberative reasoning that evaluates competing hypotheses. However, many existing methods apply a uniform reasoning strategy across queries, leading to unnecessary computation on simple tasks and insufficient reasoning on complex ones. We introduce Video-FLAIR, a training framework that learns to select the appropriate reasoning mode for each query using reinforcement learning. During training, the model generates responses under all three modes for the same prompt, enabling direct comparison. A composite reward compares these responses to favor the most effective one based on correctness, grounding, and cost, while discouraging unsupported or misaligned deliberation. This yields a supervision signal for learning adaptive reasoning without per-query annotations. Video-FLAIR improves accuracy over the Qwen2.5-VL base model by +5.4 on MathVista, +4.8 on Video-Holmes, and +4.8 on Video-MMMU, while reducing average token usage to 95 compared to 417 for always-thinking baselines.
• 中文摘要 多模态查询可能需要不同类型的推理。有些可以通过感知推理直接从视觉信号中提取信息来回答，而另一些则需要结合观察或评估竞争假设的组合推理。然而，许多现有方法在查询间采用统一的推理策略，导致简单任务中不必要的计算和复杂任务的推理不足。我们介绍了Video-FLAIR，这是一种训练框架，通过强化学习学习为每个查询选择合适的推理模式。在训练过程中，模型会针对同一提示生成三种模式下的响应，从而实现直接比较。复合奖励比较这些响应，基于正确性、基础性和成本，优先支持最有效的，同时抑制无支持或错位的思考。这为学习适应性推理提供了监督信号，无需每次查询注释。Video-FLAIR 在 MathVista 上比 Qwen2.5-VL 基础模型提高了 +5.4，Video-Holmes 上提升了 +4.8，Video-MMMU 上提升了 +4.8，同时将平均令牌使用量降至 95 次，而始终思考的基线则为 417 次。

SPEAR: Distilling Domain-Adaptive Reasoning Skeletons via Sequential Symbolic Alignment in Reinforcement Learning

SPEAR：通过顺序符号对齐在强化学习中提炼领域自适应推理骨架

• Authors: Zhuochun Li, Yuelyu Ji, Yiming Zeng, Daqing He
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.26550
• Pdf link: https://arxiv.org/pdf/2608.26550
• Abstract Reinforcement learning-based knowledge distillation has the potential to transfer complex reasoning from teacher to student models, yet it currently faces a critical dilemma: researchers must choose between sparse outcome-based rewards, which provide insufficient logical guidance, or expensive neural Process Reward Models (PRMs) for dense signals. We resolve this by introducing SPEAR (Symbolic Process Evaluation and Alignment Reward), a training-free and plug-and-play process reward method for sequence-level on-policy distillation. SPEAR projects natural-language reasoning traces into domain-adaptive symbolic milestones, providing an efficient proxy for process-level reasoning alignment. By utilizing the longest common subsequence (LCS) to align student explorations with teacher milestones, SPEAR provides a dense, order-aware reward signal that enforces logical consistency without the need for an external neural verifier. Our experiments across math, science, and commonsense reasoning tasks demonstrate that SPEAR effectively bridges the reasoning gap between student and teacher models via sequence-level distillation with efficient dense process rewards. Our code and data are available at: this https URL.
• 中文摘要 基于强化学习的知识蒸馏有潜力将复杂推理从教师模型转移到学生，但目前面临一个关键难题：研究者必须在稀疏的结果奖励（逻辑指导不足）和昂贵的神经过程奖励模型（PRMs）之间做出选择，这些模型提供逻辑指导不足。我们通过引入SPEAR（符号过程评估与对齐奖励）解决了这个问题，这是一种无需培训、即插即用的序列级策略提炼过程奖励方法。SPEAR将自然语言推理追踪投影为领域自适应的符号里程碑，为过程层级推理对齐提供高效的代理。通过利用最长的公共子序列（LCS）将学生探索与教师里程碑对齐，SPEAR提供了密集、顺序感知的奖励信号，无需外部神经验证器即可强制逻辑一致性。我们在数学、科学和常识推理任务中的实验表明，SPEAR通过序列层级提炼，有效弥合师生模型之间的推理差距，并提供高效的密集过程奖励。我们的代码和数据可在以下网址获取：https URL。

Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

到达并生存：从一比特故障信号中学习安全目标条件策略的扩展

• Authors: Guopeng Li, Yiyang Duan, Yiru Jiao, Chengcheng Xu
• Subjects: Subjects: Machine Learning (cs.LG); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2608.26571
• Pdf link: https://arxiv.org/pdf/2608.26571
• Abstract Contrastive reinforcement learning (CRL) scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective. However, in a failure-terminated Markov decision process, established CRL considers pre-failure future goals only when constructing positive samples, without accounting for the probability mass removed by failure termination. Our theoretical analysis shows that this omission induces a systematic overestimation bias in goal-reaching values. Consequently, near-failure trajectories provide disproportionately strong supervision of success despite retaining little future occupancy. Unsafe actions can thereby be reinforced through catastrophic failure bootstrapping, leading to failed policy learning and unsustainable goal-reaching behaviours. To address this problem, we introduce two minimal yet strong corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing survival mass in policy optimization. The resulting method, Safe Contrastive Reinforcement Learning (Safe-CRL), requires only the one-bit signal provided by failure termination to scale safe goal-conditioned policy learning. Across twelve failure-prone robot navigation and locomotion tasks, Safe-CRL consistently improves survival and substantially outperforms the Scaling-CRL baseline in goal-reaching performance. Additionally, deep Safe-CRL policies exhibit complex failure-avoidance behaviours. This study completes the CRL theory under failure termination and provides a scalable safe RL framework. The code is available via this https URL.
• 中文摘要 对比强化学习（CRL）通过将策略学习投射为自监督的对比目标，在目标条件任务中有效扩展。然而，在失败终止的马尔可夫决策过程中，既有的CRL仅在构建正样本时考虑失败前的未来目标，而不考虑失败终止消除的概率质量。我们的理论分析表明，这种遗漏导致目标达成值系统性高估偏差。因此，近乎失败的轨迹尽管未来占用率极低，仍能提供不成比例的成功监督。因此，不安全的行为可能通过灾难性失败自举强化，导致策略学习失败和不可持续的目标达成行为。为解决此问题，我们引入了两个最小但强力的修正：质量加权InfoNCE纠正批判学习中短暂存活未来的过重，对数-生存-质量评分则恢复策略优化中缺失的生存质量。由此产生的方法——安全对比强化学习（Safe-CRL），只需失败终止提供的一位信号即可扩展安全目标条件策略学习。在十二个易失败的机器人导航和移动任务中，Safe-CRL持续提升生存率，并在目标达成表现上显著优于缩放CRL基线。此外，深度Safe-CRL策略表现出复杂的失败避免行为。本研究完成了失败终止下的CRL理论，并提供了一个可扩展的安全RL框架。代码可通过此https URL获取。

Simple Actors and Deep Critics for Scalable Reinforcement Learning

简单演员与深度批评者，适用于可扩展强化学习

• Authors: Guhyeon Kang, Jaehwi Lee, Minhae Kwon
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.26659
• Pdf link: https://arxiv.org/pdf/2608.26659
• Abstract Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.
• 中文摘要 离线强化学习（RL）的最新进展主要由表达式生成演员推动，如扩散策略和流匹配策略，这些策略捕捉离线数据集中的多模态行为。然而，这些行为者每个动作需要多次去噪或整合步骤，因此在部署的每个决策中都会产生大量开销。在本研究中，我们重新审视了离线演员-批判者方法应投入哪些容量。由于批评者仅在训练时使用，且在部署时被舍弃，而演员在每个决策步骤执行时运行，因此将容量分配给批评者而非演员，更有利于推断时间效率。然而，离线强化学习中扩展MLP批评者已知会引入若干明显的不稳定性，实际上使批评者保持浅薄。我们识别出三种不同失败模式，分别是当批评者在离线强化---优化、自举噪声放大和值范围漂移时产生的。---并用相应的成分对应每个模式：残余MLP骨干、n步引导目标和类别交叉熵损失。结合这些要素与一个轻量级确定性演员，我们提出了LAC（光演员，深度批判）。在OGBench上，LAC匹配最强的扩散和流匹配基线，同时实现了最多4倍的推理延迟，相当于无蒸馏的单步蒸馏策略。其批评方案还支持跨演员参数的转移。

SIGMA: Structured Noise-Effect-Aware Grouped Multi-Agent Aggregation

SIGMA：结构化噪声效应感知分组多智能体聚合

• Authors: Li Mingqian
• Subjects: Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multiagent Systems (cs.MA)
• Arxiv link: https://arxiv.org/abs/2608.26683
• Pdf link: https://arxiv.org/pdf/2608.26683
• Abstract Cooperative multi-agent reinforcement learning (MARL) faces significant challenges in maintaining robust coordination under noisy observations. Although observation disturbances are often introduced independently across agents, their downstream effects on cooperative decision-making can become structured through underlying cooperation structures. We characterize this phenomenon as structured noise effects, where noise-induced decision effects exhibit local correlation among agents with stronger task-related dependencies while remaining globally heterogeneous across different agents and local structures. Existing robust MARL methods, however, rarely explicitly characterize or exploit such structure-dependent noise effects. To address this limitation, we propose SIGMA, a hierarchical collaboration framework that exploits cooperation structures to learn robust representations under noisy observations. SIGMA first organizes agents into adaptive local structures through density-based grouping and performs intra-group consensus aggregation to preserve shared task-relevant information while smoothing agent-specific representation deviations. Inter-group attention then adaptively integrates information across different groups to preserve global coordination while accommodating their heterogeneous contributions. Experiments on noisy-observation tasks in StarCraft II empirically validate the structured noise effects and demonstrate that SIGMA consistently improves robustness under observation noise while maintaining competitive performance in noise-free environments.
• 中文摘要 合作多智能体强化学习（MARL）在噪声观察下保持稳健协调面临重大挑战。尽管观察干扰通常在多个代理间独立引入，但其对合作决策的下游影响可以通过潜在的合作结构被结构化。我们将此现象描述为结构化噪声效应，噪声诱导的决策效应在具有更强任务依赖的代理之间表现出局部相关性，同时在不同代理和局部结构之间保持全局异质性。然而，现有的稳健MARL方法很少明确表征或利用这种结构依赖噪声效应。为解决这一限制，我们提出了SIGMA，一种利用协作结构学习噪声观测下稳健表征的分层协作框架。SIGMA首先通过基于密度的分组将代理组织成适应性的局部结构，并进行组内共识聚合，以保留共享的任务相关信息，同时平滑代理特有的表征偏差。随后，群间注意力自适应地整合不同群体间的信息，以保持全局协调，同时容纳其异质贡献。星际争霸II中噪声观察任务的实验实证验证了结构化噪声效应，并证明SIGMA在无噪声环境中持续提升观测噪声下的鲁棒性，同时保持竞争力。

Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMs

你说的话中的理性：视频大型语言模型中推理提炼的非政策痕迹的意语直译

• Authors: Ji Soo Lee, Jinyoung Park, Seohyun Lee, Jongha Kim, Joonmyung Choi, Jinsung Yoon, Hyunwoo J. Kim
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2608.26684
• Pdf link: https://arxiv.org/pdf/2608.26684
• Abstract Recent large language models achieve strong performance on complex reasoning tasks, where reinforcement learning with Group Relative Policy Optimization (GRPO) has emerged as a leading paradigm for optimizing models on self-generated trajectories. However, the on-policy nature of GRPO bounds the model to the reasoning skills it can already produce, restricting to learn more advanced capabilities. Prior works inject privileged reasoning traces from a stronger teacher policy to guide training, yet these traces are inherently out of distribution with respect to the student policy. We observe that this mismatch between on-policy and off-policy causes gradient clipping on semantically critical reasoning tokens, ultimately rewarding correct answers while leaving the reasoning that justifies them unlearned. Hence, we propose \textbf{Echo-GRPO}, a framework that lets the model reason in the words it speaks. Rather than imitating low-probability privileged traces from the teacher model, Echo-GRPO rewrites them into the student policy's own \textit{idiolect}, that is, its own characteristic vocabulary and expression patterns, while preserving their semantics via Dual-Reference Decoding. We instantiate this framework as \textbf{VideoEcho-R1} for video reasoning distillation, achieving consistent improvements across three multimodal LLM backbones and five benchmarks. Finally, we show that our idiolectal paraphrasing is a plug-in module that consistently improves both RL and supervised fine-tuning frameworks for reasoning distillation, demonstrating that policy-aligned supervision extends beyond GRPO.
• 中文摘要 近年来，大型语言模型在复杂推理任务中表现出色，其中基于群体相对策略优化（Group Relative Policy Optimization，GRPO）的强化学习已成为优化自生成轨迹模型的领先范式。然而，GRPO的策略性质限制了模型仅限于其已能产生的推理技能，限制了学习更高级能力的能力。以往的工作将更强教师政策中的特权推理痕迹注入指导培训，但这些特征本质上与学生政策不匹配。我们观察到，开策略与非策略之间的不匹配导致语义批判推理标记出现梯度裁剪，最终奖励正确答案，而使支持其正当性的推理未被学习。因此，我们提出了\textbf{Echo-GRPO}框架，允许模型在其所说的词语中进行推理。Echo-GRPO没有模仿教师模型中的低概率特权痕迹，而是将它们重写为学生政策自身的\textit{idiolect}，即其独特的词汇和表达模式，同时通过双重引用解码保持其语义。我们将该框架实例化为\textbf{VideoEcho-R1}用于视频推理提炼，在三个多模态LLM骨干和五个基准测试中实现了持续的改进。最后，我们展示了我们的特语改写是一个插件模块，能够持续改进强化学习和监督式推理提炼的微调框架，证明策略对齐的监督超越了GRPO。

Residual Deep Reinforcement Learning-Based Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot under Disturbances and Parametric Uncertainties

基于基于残余深度强化学习的计算扭矩控制，适用于受干扰和参数不确定性条件下的电缆驱动下肢康复机器人

• Authors: Mohammad-Hossein Fakouri, Ali Keymasi-Khalaji
• Subjects: Subjects: Robotics (cs.RO); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.26739
• Pdf link: https://arxiv.org/pdf/2608.26739
• Abstract Accurate trajectory tracking in cable-driven lower-limb rehabilitation robots is challenging because model uncertainty, external disturbances, joint constraints, and pull-only cable actuation can degrade nominal control performance. Conventional model-based controllers provide an interpretable control structure but remain sensitive to model mismatch, whereas fully learning-based control can reduce transparency and complicate constraint-aware operation. This study proposes a residual deep reinforcement learning-enhanced computed torque control framework in which computed torque control generates the nominal command and a bounded Deep Deterministic Policy Gradient policy supplies only an additional compensating torque. The approach is evaluated in simulation under nominal, uncertain, disturbed, combined, and generalization conditions, together with trajectory-tracking, joint-limit, cable-demand, workspace-feasibility, and cable-Jacobian diagnostics. Across the evaluated conditions, the residual controller improves tracking and disturbance rejection relative to computed torque control while preserving the interpretable model-based command structure and satisfying the reported feasibility checks in the representative evaluation. Broader tests indicate that tracking improvements can persist beyond the representative case while also exposing trajectory-dependent constraint limitations. These results support bounded residual learning as a practical robustness-enhancement strategy for simulation-based rehabilitation robot control and motivate further constraint-aware and experimental validation.
• 中文摘要 在电缆驱动的下肢康复机器人中，准确跟踪轨迹具有挑战性，因为模型不确定性、外部干扰、关节约束和仅拉拉电缆驱动可能降低名义控制性能。传统基于模型的控制器提供可解释的控制结构，但仍对模型不匹配敏感;而完全基于学习的控制则可能降低透明度并使约束感知操作复杂化。本研究提出了一种残余深度强化学习增强的计算扭矩控制框架，其中计算扭矩控制生成标称指令，而有界深度确定性策略梯度策略仅提供额外的补偿扭矩。该方法在名义、不确定、扰动、组合和泛化条件下进行模拟评估，同时结合轨迹跟踪、关节极限、电缆需求、工作空间可行性和电缆-雅可比诊断。在评估条件下，残差控制器相对于计算扭矩控制提升跟踪和干扰抑制，同时保持可解释的基于模型的指令结构，并满足代表性评估中报告的可行性检查。更广泛的测试表明，跟踪改进可以在代表性情形之后持续存在，同时暴露出轨迹依赖约束的限制。这些结果支持有限残差学习作为基于仿真的修复机器人控制的实用鲁棒性增强策略，并激励进一步的约束感知和实验验证。

Behavior2Trip: Towards Personalized Travel Planning via User Behavior Trajectory

Behavior2Trip：通过用户行为轨迹实现个性化旅行规划

• Authors: Zihao Cheng, Yingyu Shan, Hongru Wang, Zeming Liu, Xinyi Wang, Xiangrong Zhu, Yuhang Guo, Wei Lin, Yunhong Wang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.26807
• Pdf link: https://arxiv.org/pdf/2608.26807
• Abstract Travel planning agents assist users in generating personalized travel plans by modeling their individual preferences. Existing agents either rely on explicit user instructions or engage in multi-turn clarification to elicit user preferences. However, both approaches overlook the rich behavioral signals latent in users' past behaviors, which implicitly encode their preferences. This over-reliance on active user input increases interaction burden and limits plan personalization. To bridge this gap, we introduce a new task, Behavior-Aware Travel Planning, which infers user preferences directly from past behaviors and generates personalized travel plans. To facilitate research on this task, we introduce Behavior2Trip, a benchmark constructed from one of the largest Chinese online travel platforms, comprising 11,400 instances. Each instance represents an average of 39.8 past user behaviors spanning 14 attributes across 5 preference dimensions. We further propose B2T-Agent, a reinforcement learning-based agent that leverages user behavior trajectories, interacts with external tools for preference-aligned retrieval, and maintains an internal memory module. Experiments on Behavior2Trip show that GPT-4.1 achieves a full-constraint pass rate of only 0.5\% on the hardest tasks, while B2T-Agent built upon Qwen3-8B outperforms all baselines, highlighting the substantial challenge of this task. Moreover, Qwen3-8B trained with B2T-Agent also outperforms GPT-4.1 on the TravelPlanner benchmark, demonstrating strong generalization. Code and data are available at this https URL
• 中文摘要 旅行规划代理通过建模用户的个人偏好，帮助用户生成个性化旅行计划。现有代理要么依赖明确的用户指示，要么通过多轮澄清来引发用户偏好。然而，这两种方法都忽视了用户过去行为中潜藏的丰富行为信号，这些信号隐含地编码了他们的偏好。这种过度依赖主动用户输入增加了互动负担，限制了计划个性化。为弥合这一差距，我们引入了一项新任务——行为感知旅行规划，直接从过去行为推断用户偏好并生成个性化旅行计划。为促进该任务的研究，我们引入了基于中国最大在线旅行平台之一构建的Behavior2Trip基准测试，包含11,400个实例。每个实例代表平均39.8条过去用户行为，涵盖5个偏好维度的14个属性。我们还提出了B2T-Agent，这是一种基于强化学习的代理，利用用户行为轨迹，与外部工具交互以实现偏好对齐检索，并维护内部内存模块。Behavior2Trip的实验显示，GPT-4.1在最难任务中仅实现0.5%的全约束通过率，而基于Qwen3-8B构建的B2T-Agent表现优于所有基线，凸显了该任务的巨大挑战。此外，使用B2T-Agent训练的Qwen3-8B在TravelPlanner基准测试中也优于GPT-4.1，展现出强烈的泛化能力。代码和数据可在此 https URL 获取

Towards Safe Reinforcement Learning with Reduced Conservativeness: A Case Study on Drone Flight Control

迈向降低保守性的安全强化学习：关于无人机飞行控制的案例研究

• Authors: Loizos Hadjiloizou, Michael C. Welle, Hang Yin, Danica Kragic
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.26852
• Pdf link: https://arxiv.org/pdf/2608.26852
• Abstract Incorporating formal methods into reinforcement learning (RL) has the potential to result in the best of both worlds, combining the robustness of formal guarantees with the adaptability and learning capabilities of RL, though careful design is needed to balance safety and exploration. In this work, we propose a framework to mitigate this loss of exploration while still allowing for the safety of the system to be ensured. Specifically, we introduce a less restrictive method that can reduce the conservativeness of formal methods by refining a disturbance model using online collected data and it evaluates the safety of a learning-based controller, using computationally efficient zonotopic reachability analysis for the safety analysis to facilitate a real-time implementation. We validate the framework in a real-world drone flight through a canyon, where the drone is subjected to unknown external disturbances and the framework is tasked with learning those disturbances online and adjusting the safety guarantees accordingly. The results show that the framework enables a less restrictive online training of learning-based controllers without compromising the safety of the system.
• 中文摘要 将形式方法纳入强化学习（RL）有望实现两全其美，结合形式保证的稳健性与强化学习的适应性和学习能力，尽管需要谨慎设计以平衡安全性与探索性。本研究提出一个框架，以减轻探索损失，同时保证系统安全。具体来说，我们引入了一种较少限制的方法，通过利用在线收集的数据优化扰动模型，降低形式方法的保守性，并通过计算高效的区域拓扑可达性分析进行安全分析，以实现实时实现。我们在现实世界无人机穿越峡谷飞行中验证了该框架，该无人机遭受未知外部干扰，框架负责在线学习这些干扰并相应调整安全保障。结果显示，该框架能够实现更宽松的在线学习型控制器培训，同时不影响系统的安全性。

Reinforcement Learning-Based Control of CAV Platoon Joining Maneuvers in Mixed Traffic

基于强化学习的CAV排在混合交通中加入机动的控制

• Authors: Biao Yin, Abderrahmane Kasmi, Nadir Farhi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2608.26860
• Pdf link: https://arxiv.org/pdf/2608.26860
• Abstract Connected and automated vehicle (CAV) platooning offers a promising approach to improving road safety and traffic capacity. However, platoon control in real-world traffic is challenging due to uncertainty and heterogeneous driving behaviors. Reinforcement learning (RL) has strong potential for addressing such control problems, but its practical deployment raises challenges related to safety and learning efficiency. This paper proposes a generic modeling and simulation framework for investigating CAV platoon joining maneuvers and comparing deep reinforcement learning (DRL)-based control algorithms. The problem is particularly challenging in mixed-traffic environments, where CAVs coexist with human-driven vehicles exhibiting heterogeneous longitudinal and lateral behaviors. The objective is to achieve safe and efficient joining maneuvers by either incorporating penalties for risky behaviors into the learning process or using an external safety controller to constrain the learned policy. An agent-based modeling framework coupled with the Simulation of Urban MObility (SUMO) simulator is used to evaluate Deep Q-Network (DQN), Double Deep Q-Network (DDQN), and Proximal Policy Optimization (PPO). Results show that PPO outperforms DQN and DDQN, achieving a joining success rate of approximately 98 % and a collision rate below 1 %, largely due to risk-related penalties incorporated into the reward function. However, this improved performance requires more decision steps to complete the maneuver, revealing a trade-off between safety, joining effectiveness, and decision efficiency. An external safety controller effectively prevents collisions, although its interventions may reduce joining efficiency. The results highlight the importance of jointly considering safety and efficiency when designing RL-based controllers for CAV platoon joining in mixed traffic.
• 中文摘要 互联自动驾驶（CAV）分队化为提升道路安全和交通容量提供了有前景的方法。然而，由于不确定性和异构驾驶行为，实际交通中的分队控制具有挑战性。强化学习（RL）在解决此类控制问题方面具有强大潜力，但其实际部署带来了安全性和学习效率方面的挑战。本文提出了一种通用建模和仿真框架，用于研究CAV排的会合动作，并比较基于深度强化学习（DRL）的控制算法。该问题在混合交通环境中尤为棘手，CAV与展现出异质纵向和横向行为的人类驾驶车辆共存。目标是通过将风险行为的惩罚纳入学习过程，或使用外部安全控制器约束所学策略，实现安全高效的联接动作。基于代理的建模框架结合城市动车模拟器（SUMO）模拟器，用于评估深度Q网络（DQN）、双深度Q网络（DDQN）和近端策略优化（PPO）。结果显示，PPO优于DQN和DDQN，连接成功率约为98%，碰撞率低于1%，主要归功于奖励函数中纳入的风险相关惩罚。然而，这一提升的性能要求更多决策步骤才能完成操作，揭示了安全性、连接效能和决策效率之间的权衡。外部安全控制器有效防止碰撞，尽管其干预可能降低连接效率。结果强调了在设计基于RL的混合交通排级骑兵排控制器时，安全与效率共同考量的重要性。

AI agents in Algorithmic Electricity Markets: On the Emergence of Tacit Collusion

人工智能代理在算法电力市场中的出现：关于默契合谋的出现

• Authors: Jakub Seredyński, Georgios Tsaousoglou
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computer Science and Game Theory (cs.GT); Multiagent Systems (cs.MA); Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2608.26896
• Pdf link: https://arxiv.org/pdf/2608.26896
• Abstract As electricity market participants increasingly adopt learning-based agents for their bidding strategies, electricity markets are becoming algorithmic. Evidence from algorithmic markets in other domains shows that tacit collusion can arise purely through independent learning. Moreover, electricity markets are typically oligopolistic and feature repeated interaction among a small number of participants, making them structurally susceptible to non-competitive behavior. In the face of these observations, this paper investigates the hypothesis that tacit collusion may emerge in electricity markets where participants' actions are controlled by autonomous learning-based algorithms. We model strategic bidding as a repeated game with imperfect public monitoring, and model the participants' emergent behavior using multi-agent reinforcement learning. We propose a multi-dimensional set of criteria (going beyond profit comparisons against Nash equilibria) to assess whether the resulting behavior constitutes tacit collusion. Our experimental results showcase that such a danger is realistic for electricity markets: there are cases where agents do learn to sustain supra-competitive outcomes that are supportive of tacit collusion indicators, even though the agents were never instructed to collude.
• 中文摘要 随着电力市场参与者越来越多地采用基于学习的代理作为竞价策略，电力市场正变得算法化。其他领域的算法市场证据表明，默契串谋可以纯粹通过独立学习产生。此外，电力市场通常是寡头垄断的，参与者数量有限，反复互动，结构上容易受到非竞争行为的影响。面对这些观察，本文探讨了在电力市场中，参与者行为由自主学习算法控制的假设。我们将战略竞价建模为一场具有不完善公共监控的重复游戏，并利用多代理强化学习模拟参与者的涌现行为。我们提出了一套多维标准（超越利润与纳什均衡的比较），以评估最终行为是否构成默许共谋。我们的实验结果表明，这种风险对电力市场来说是现实的：有些代理确实学会维持超竞争的结果，这些结果支持默契串通指标，尽管代理从未被指示串通。

Energy-Neutral Coverage Optimization by Joint Deployment and Scheduling in Ambient IoT Devices with Directional Sensing

通过联合部署和调度环境物联网设备实现的能量中和覆盖优化，并具备方向感测

• Authors: David E. Ruíz-Guirola, Samuel Montejo-Sánchez, Richard Demo Souza, Onel L. A. López
• Subjects: Subjects: Systems and Control (eess.SY); Information Theory (cs.IT)
• Arxiv link: https://arxiv.org/abs/2608.26944
• Pdf link: https://arxiv.org/pdf/2608.26944
• Abstract Ambient IoT (A-IoT) devices rely on energy harvesting and duty cycling to sustain operation, thereby fundamentally changing collaborative sensing compared with traditional always-ON sensor networks. In this paper, we study the joint deployment and sensing scheduling of A-IoT devices equipped with directional sensing. We explore four solution strategies: (i) a grid deployment with static duty cycling, (ii) a centralized policy-gradient reinforcement learning (RL) approach that begins with a grid deployment and learns energy-aware device relocation and duty-cycling policies, (iii) a mixed-integer linear programming (LP) approach that couples static deployment design with duty-cycle allocation, and (iv) a hybrid LP+RL that combines optimization-based initialization with learning-based refinement. Using representative A-IoT use cases, we evaluate coverage as a function of device density, field-of-view, and maximum feasible duty cycle, determined by harvested energy and device consumption. Numerical results indicate that the proposed LP+RL and RL policies consistently outperform both the grid baseline and the LP-based method, achieving up to 2x higher mean effective coverage in low and medium energy harvesting (EH) regimes. In contrast, the standalone LP method remains limited by its conservative static duty cycle allocation under tight EH constraints. Moreover, the structured initialization of the LP+RL method substantially accelerates convergence, reducing the total offline optimization time by up to 10x compared to the standalone RL.
• 中文摘要 环境物联网（A-IoT）设备依赖能量收集和占空周期来维持运行，从而从根本上改变了协同感测，相较于传统的始终开启传感器网络。本文研究了配备定向传感的A-IoT设备的联合部署与传感调度。我们探讨了四种解决方案策略：（i）采用静态工作周期的网格部署，（ii）从网格部署开始学习能能感知设备迁移和占空周期策略的集中策略梯度强化学习（RL）方法，（iii）将静态部署设计与占空周期分配结合的混合整数线性规划（LP）方法，以及（iv）结合基于优化的初始化与基于学习的细化的混合LP+RL。我们通过具代表性的A-IoT用例，评估覆盖范围，作为设备密度、视野和最大可行占空比的函数，这些工作周期由采集的能能和设备消耗决定。数值结果表明，所提LP+RL和RL策略始终优于电网基线和基于LP的方法，在低和中能收集（EH）条件下实现高出多达2倍的平均有效覆盖率。相比之下，独立LP方法在严格的EH约束下仍受限于其保守的静态工作周期分配。此外，LP+RL方法的结构化初始化显著加快了收敛进程，使离线优化总时间比独立RL缩短了多达10倍。

Performance Foundations of Parallel & Distributed Reasoning Language Models

并行与分布式推理语言模型的性能基础

• Authors: Maciej Besta, Leonard Schmidt, Lara Nonino, Robert Gerstenberger, Pierre Pang, Patrik Okanovic, Ales Kubicek, Tiancheng Chen, Baraq Lipshitz, Torsten Hoefler
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC); Performance (cs.PF)
• Arxiv link: https://arxiv.org/abs/2608.27046
• Pdf link: https://arxiv.org/pdf/2608.27046
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) and other RL-style post-training paradigms have been used for aligning large language models (LLMs) with reasoning standards. The resulting recent Reasoning Language Models (RLMs) such as DeepSeek-R1, o3, and Kimi k1.5 show that such RL-style post-training ("RL-for-LLMs") can substantially improve chain-of-thought reasoning, long-horizon planning, and self-correction. However, the computational footprint of these systems is massive: state-of-the-art RLM training requires millions of GPU-hours and tightly coupled multi-model pipelines that stress modern hardware far beyond classical supervised LLM training. This makes RLM training as much a parallel and distributed systems problem as an algorithmic one. In this work, to facilitate developing RLMs that are simultaneously high-performance, scalable, and cost-effective, we first systematize the RL-for-LLM paradigm and provide a compute-centric analysis of prominent post-training algorithmic frameworks: Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), as well as their variants. Second, we develop a taxonomy of intra- and inter-model parallelism strategies for RL-for-LLMs, covering both traditional techniques (data, tensor, pipeline, sequence, context, and expert parallelism) as well as novel forms of parallelism and optimization techniques for multi-model RLM training, for example disaggregated placement, stage fusion, hybrid parallelism, and asynchronous execution. We harness the work-depth model of parallel computing to make our taxonomy and its insights rigorous and portable. Finally, we analyze existing RLM frameworks and we distill practical guidelines and outline open research directions for building scalable, fast, and cost-effective RLMs.
• 中文摘要 带可验证奖励的强化学习（RLVR）及其他类似强化学习的后训练范式已被用于将大型语言模型（LLM）与推理标准对齐。近期产生的推理语言模型（RLMs），如DeepSeek-R1、o3和Kimi k1.5，表明此类强化学习式后训练（“RL-for-LLMs”）能够显著提升思维链推理、长远规划和自我纠正能力。然而，这些系统的计算占用极为庞大：最先进的RLM训练需要数百万GPU小时和紧密耦合的多模型流水线，这些流水线对现代硬件的压力远超传统监督LLM训练。这使得RLM训练既是并行和分布式系统问题，也是算法问题。本研究旨在促进开发同时具备高性能、可扩展性和成本效益的RLM范式，首先系统化RL针对LLM范式，并以计算为中心分析了主要的训练后算法框架：近端策略优化（PPO）、群相对策略优化（GRPO）及其变体。其次，我们开发了面向大型语言模型的RL内及模型间并行策略分类法，涵盖传统技术（数据、张量、流水线、序列、上下文和专家并行）以及多模型RLM训练中的新型并行和优化技术，如拆分配置、阶段融合、混合并行和异步执行。我们利用并行计算的工作深度模型，使分类法及其洞见严谨且可移植。最后，我们分析现有的RLM框架，提炼出实用指南，并提出构建可扩展、快速且经济高效的RLM的开放研究方向。

Emotional Preferences as Goal-Priority Regulation

情感偏好作为目标优先级调节

• Authors: Shiqi Liu, Yihua Tan, Hu Fu, Guanyu Qi
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.27072
• Pdf link: https://arxiv.org/pdf/2608.27072
• Abstract A core question in decision-making for agents is whether the relative priorities of competing lower-level objectives can be determined by emotional preferences autonomously generated by higher-level goals, rather than being externally prespecified. Under changing external environments and evolving internal states, emotions play an important functional role in regulating the relative priorities of competing goals. Inspired by the goal-directed theory of emotion, this paper studies how such preference regulation can be computationally realized through reinforcement learning. We first propose a conception of emergent emotional preference: a high-level goal autonomously induces state-dependent preferences over competing lower-level objectives. This conception is built upon a framework consisting of a multi-objective reinforcement learning inner controller and an outer preference generator. The inner controller provides a repertoire of preference-conditioned goal-directed behaviors, while the outer preference generator learns a mapping from the current state to objective preferences through reinforcement learning on a high-level goal. We operationalize emotional preference as a state-dependent regulation of relative goal priorities that emerges through optimization. Furthermore, we characterize the policy space induced by preference regulation and derive an upper bound on the optimality gap in terms of the representation error of the inner behavioral repertoire. We show that the gap vanishes when the optimal policy can be represented by the available preference-conditioned policies. Experiments in self-constructed multi-objective exploration environments show that the learned preference function exhibits contextual priority switching, graded trade-offs, and temporal persistence, and outperforms the evaluated fixed-preference and handcrafted-preference strategies.
• 中文摘要 代理决策中的核心问题是，竞争性低层目标的相对优先级是否可以通过更高层次目标自主生成的情感偏好决定，而非外部预先设定。在变化的外部环境和不断演变的内部状态下，情绪在调节竞争目标的相对优先级中发挥重要功能作用。本文受目标导向情感理论启发，研究如何通过强化学习计算实现这种偏好调节。我们首先提出了一种涌现情感偏好的概念：高层目标自主地诱导对竞争性低层目标的状态依赖偏好。该概念建立在由多目标强化学习内控器和外部偏好生成器组成的框架上。内在控制者提供一系列偏好条件化的目标导向行为，而外部偏好生成器通过强化学习在高层目标上学习从当前状态映射到客观偏好。我们将情绪偏好操作化为通过优化出现的相对目标优先级的状态依赖调节。此外，我们刻画了偏好调控诱导的政策空间，并以内在行为库的表征误差推导出最优性差距的上界。我们证明，当最优政策可以用可用的偏好条件政策表示时，差距消失。在自我构建的多目标探索环境中的实验显示，学习的偏好函数表现出情境优先级切换、分级权衡和时间持久性，并且优于已评估的固定偏好和手工偏好策略。

GRAIN: Bridging Name and Narrative Shifts in Real-World Graph Reasoning through Invariance-Rewarded Agentic RL

GRAIN：通过不变性奖励的能动强化学习，弥合现实世界图论推理中的名称与叙事转变

• Authors: Zike Yuan, Han Zhang, Jianzhi Yan, Le Liu, Cai Ke, Huozhi Zhou, Jian Xie, Jiran Yin, Yukun Cao, Yue Yu, Hui Wang, Ming Liu, Bing Qin
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2608.27142
• Pdf link: https://arxiv.org/pdf/2608.27142
• Abstract Despite their potential in standardized graph tasks, Large Language Models (LLMs) remain brittle to real-world shifts in node identifiers and task formulation. While deterministic graph tools are invariant to such shifts, extracting topological structures from noisy text is highly fragile for LLMs, which often overfit to surface patterns. Moreover, mitigating these parsing failures via multi-agent systems incurs prohibitive latency. To address this, we propose GRAIN, a single-agent framework optimized via reinforcement learning. GRAIN models reasoning as a semantic parsing and tool-execution pipeline, guided by a Structure Invariance Reward. By validating extracted intermediate graphs against ground-truth topologies, this reward forces the LLM to learn robust text-to-structure mappings rather than memorizing linguistic artifacts. We also introduce GRIT, a benchmark evaluating sensitivity to such linguistic shifts. GRAIN outperforms multi-agent baselines by 16.45\% in accuracy with approximately 24\% lower latency. Furthermore, it demonstrates superior structural generalization, halving the out-of-distribution (OOD) gap of SFT models (from 15.77\% to 7.80\%) and maintaining robustness on large-scale graphs beyond the training distribution.
• 中文摘要 尽管大型语言模型（LLMs）在标准化图任务中具有潜力，但它们仍然容易受到节点标识符和任务表述的现实变化影响。虽然确定性图工具对此类变化保持不变，但对于噪声文本提取拓扑结构对大型语言模型来说极为脆弱，因为LLM常常会对表面模式过拟合。此外，通过多智能体系统来缓解这些解析失败会带来极高的延迟。为此，我们提出了GRAIN，一种通过强化学习优化的单代理框架。GRAIN将推理建模为语义解析和工具执行流水线，由结构不变奖励指导。通过验证提取的中间图与真实拓扑，这种奖励迫使LLM学习稳健的文本到结构映射，而非记忆语言产物。我们还介绍了GRIT，这是一个评估对此类语言变化敏感性的基准。GRAIN的准确率比多智能体基线高出16.45%，延迟降低约24%。此外，它展现了更优异的结构泛化能力，将SFT模型的分布外（OOD）差距减半（从15.77%降至7.80%），并在训练分布之外的大尺度图中保持鲁棒性。

A Trans-Domain Digital Twin for Bio-Aware Control of Climate and Energy in Cattle Fattening Barns Using Single-Episode Optimizer Learning

跨域数字孪生，利用单集优化器学习实现牛场中气候和能源的生物感知控制

• Authors: Mansoorali Amiri
• Subjects: Subjects: Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2608.27185
• Pdf link: https://arxiv.org/pdf/2608.27185
• Abstract In closed cattle-fattening barns, the indoor climate and herd growth are mutually interdependent. Temperature, relative humidity, airflow, and ventilation affect thermal comfort, feed intake, metabolic heat production, daily growth, feed efficiency, and energy consumption, while body-weight gain alters the future heat and moisture loads of the barn and, consequently, its ventilation, heating, and energy requirements. This article proposes a trans-domain digital twin framework with single-episode learning capability, customized for bio-aware climate and energy control in a closed cattle-fattening barn. The framework integrates a mechanistic climate simulator, a livestock growth simulator, model predictive control, lightweight reinforcement learning, and structured knowledge memory within a multi-rate temporal-loop architecture. The fast temporal loop operates every five minutes to evaluate actuator decisions and maintain short-term thermal comfort, safety, and energy efficiency, whereas the slow temporal loop provides biological guidance based on daily climatic conditions, feed efficiency, heat production, and growth-limiting factors. The results show that climate, growth, energy, feed, biological guidance, and memory can be linked within a single executable control cycle. Remaining limitations include the need for field validation, improved management of feed pressure, and reduction of abrupt actuator-command variations.
• 中文摘要 在封闭的养育牛棚中，室内气候与牛群增长相互依赖。温度、相对湿度、气流和通风影响热舒适度、饲料摄入、代谢热产量、每日生长、饲料效率和能耗，而体重增加则改变牛棚未来的热量和湿度负荷，进而影响其通风、供暖和能源需求。本文提出了一个跨域数字孪生框架，具备单集学习能力，专为封闭牛只育肥谷仓的生物感知气候和能量控制定制。该框架集成了机械气候模拟器、畜牧生长模拟器、模型预测控制、轻量级强化学习和结构化知识记忆，采用多速率时间循环架构。快速时间回路每五分钟运行一次，用于评估执行器决策并保持短期热舒适、安全和能效，而慢时间回路则根据每日气候条件、供电效率、热量产出和生长限制因素提供生物指导。结果表明，气候、生长、能量、供给、生物导引和记忆可以在同一可执行的控制周期内相互关联。剩余的限制包括现场验证的需求、改进供电压力管理以及减少执行器与指令的突然变化。

Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms

跨领域整合RLVR能力：融合范式深度解析

• Authors: Siye Wu, Kai Yang, Yuchen Cai, Xin Xu, Peng-Yuan Wang, Jiaxuan Wang, Jiashun Liu, Jiafei Lyu, Yangkun Chen, Saiyong Yang, Yanghua Xiao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.27409
• Pdf link: https://arxiv.org/pdf/2608.27409
• Abstract Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.
• 中文摘要 带可验证奖励的强化学习（RLVR）提升了大型语言模型的特定能力，但覆盖多项能力通常需要培训不同的领域专家，随后对其进行整合。我们根据其重复使用的工件对三种融合范式进行组织：Merge 结合专家任务向量，Mix RL 池化数据集，多教师策略提炼（MOPD）两者兼用。由于它们大多是孤立研究，如何比较以及如何选择仍然不明确。我们利用跨模型尺度的共享专家和数据和多域基准套件进行比较。虽然它们的平均性能最多相差1.4点，但在单一基准测试上差距达到8.6点，任务向量几何中显示域级变异追踪跨域关系。训练动态揭示了不同的约束：混合强化学习依赖于领域混合比例，MOPD仍受其教师限制，Merge将所有专家更新压缩为一个。这三者都提高了单样本的准确性，同时没有显著提升解决方案覆盖率或保留能力的损失。这些结果提供了实用指导：当专家已经存在且低成本融合至关重要时使用Merge;在训练统一模型时使用混合强化学习，且领域比例调整以适应跨领域迁移;MOPD在保持领域特定收益比超越教师或最小化端到端成本更重要时使用MOPD。

Boosting LLM Exploration via Weak-Model Guidance in RLVR

通过RLVR中的弱模型指导促进LLM探索

• Authors: Xingyu Shen, Huishuai Zhang, Peng Li, Yinchun Wang, Dongyan Zhao
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.27420
• Pdf link: https://arxiv.org/pdf/2608.27420
• Abstract Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves LLM reasoning but often causes a drop in policy entropy, leading to narrowed reasoning coverage and degraded pass@$k$ for large $k$. While existing methods mitigate this entropy collapse through algorithmic regularizations, cross-model non-parametric perturbation is also neglected. In this work, we propose a simple yet effective approach to preserve the generative diversity of LLMs during RLVR. Instead of relying solely on internal exploration, we force the target model to generate answers based on partial reasoning trajectories generated by a smaller, weaker language models. These unfamiliar prefixes effectively disrupt over-confidence and encourage the exploration of distinct reasoning paths. We empirically study the potential of outer prefixes, revealing the mechanism of the impact of distributional discrepancy to the exploration dynamics in RLVR training. Experiments across multiple mathematical benchmarks show that our method consistently outperforms vanilla RLVR. Notably, the performance gain becomes increasingly pronounced as $k$ scales up, demonstrating a substantial expansion of reasoning coverage. Furthermore, our approach efficiently mitigates entropy collapse without requiring additional SFT, intricate reward designs, or complex prompting.
• 中文摘要 带可验证奖励的强化学习（RLVR）显著提升了LLM推理能力，但常常导致策略熵下降，导致推理覆盖范围缩小，大$k$pass@$k$下降。现有方法通过算法正则化缓解熵崩溃，但跨模型非参数扰动也被忽视。本研究提出一种简单有效的方法，在RLVR期间保持LLM的生成多样性。我们不依赖内部探索，而是迫使目标模型基于更小、较弱的语言模型生成的部分推理轨迹生成答案。这些陌生前缀有效打破过度自信，鼓励探索不同推理路径。我们通过实证研究外部前缀的潜力，揭示了分布差异对RLVR训练探索动态的影响机制。跨多个数学基准测试的实验表明，我们的方法持续优于普通RLVR。值得注意的是，随着$k$的扩展，性能提升愈发明显，显示出推理覆盖范围的显著扩展。此外，我们的方法高效地减轻了熵坍缩，无需额外的SFT、复杂的奖励设计或复杂的提示。

TTPO: Test-Time Policy Optimization

TTPO：测试时策略优化

• Authors: Aozhe Wang, Zhengxi Lu, Jianze Wang, Shangke Lv, Ying Liu, Weiming Lu, Jun Xiao, Yueting Zhuang, Hua Yang, Qianglong Chen, Yongliang Shen
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2608.27448
• Pdf link: https://arxiv.org/pdf/2608.27448
• Abstract Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
• 中文摘要 近年来著名的后训练方法，如强化学习（RL）和策略自蒸馏（OPSD），推动了大型语言模型数学推理的快速进展，但它们依赖地面真实标签，排除了测试时训练（TTT）。用多数票伪标签替代基础真实是一个自然的替代方案，但其脆弱性：错误的投票会腐蚀教师并误导每个代币。我们观察到这种失败模式是非对称的：与伪标签不符的推出通常不匹配，无论投票是否正确。基于这一观察，我们提出了测试时策略优化（TTPO），这是一种非对称目标，通过OPSD提炼同意的推广，并用分组强化学习惩罚不同意的推广。代币级选择进一步细化了这两个分支：蒸馏会对已收敛的位置下加权，而强化学习只惩罚信心错误。即使频繁存在伪标签错误，这两种更新依然稳固，多数票路由随着模型改进，带来了更严格的自我监督。没有任何标签，TTPO在五个竞赛级基准上匹配标签监督的OPSD，在TTT中Qwen3-1.7B从38.0%提升至45.2%，无思考的情况下产率+25.2%至+36.4%，并展现出强烈的跨任务泛化能力。

Keyword: diffusion policy

Diffusion Policies for Short-Horizon Planning in Robot Crowd Navigation

机器人人群导航短期规划的扩散政策

• Authors: Wendong Li, Jochen Garcke
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2608.27158
• Pdf link: https://arxiv.org/pdf/2608.27158
• Abstract Robot crowd navigation requires safe and efficient decision-making under dense, dynamic, and multimodal human--robot interactions. Existing reinforcement-learning methods typically output a single reactive action at each timestep, which limits their ability to represent diverse short-term avoidance strategies. We propose Planning Diffusion Policy Optimization (PDPO), an offline-to-online reinforcement-learning framework that uses a diffusion policy to generate short-horizon action chunks for crowd navigation. PDPO is first pretrained on collision-avoidance demonstrations and then fine-tuned online with PPO by treating the denoising process as an internal decision process. During execution, the policy generates a five-step action chunk and applies it in a receding-horizon manner. Furthermore, we observe an evaluation artifact in common crowd-navigation benchmarks: without explicit boundary constraints, learned agents may leave the valid domain and bypass dense crowds. To address this, we introduce a setting in which boundary violations are treated as collisions. Experiments show that PDPO obtains an improved success rate over strong baselines, and ablations demonstrate that action chunks are especially important for the modified bounded benchmark.
• 中文摘要 机器人人群导航需要在密集、动态和多模态的人机交互下进行安全高效的决策。现有强化学习方法通常在每个时间步输出单一反应动作，限制了其表示多样化短期回避策略的能力。我们提出了规划扩散策略优化（PDPO），这是一种离线到在线的强化学习框架，利用扩散策略生成短视野的人群导航动作块。PDPO首先在碰撞避免演示上预训练，然后通过将去噪过程视为内部决策过程，在线通过PPO进行微调。执行过程中，策略生成五步动作块，并以远离视界的方式应用。此外，我们在常见的人群导航基准中观察到一个评估伪影：没有明确的边界约束，学习的智能体可能会离开有效领域并绕过密集人群。为此，我们引入了一个将边界违规视为碰撞的环境。实验表明，PDPO在强基线条件下成功率更高，消融也表明动作块对修正有界基准尤为重要。