生成时间: 2026-08-03 19:24:07 (UTC+8); Arxiv 发布时间: 2026-08-03 20:00 EDT (2026-08-04 08:00 UTC+8)

今天共有 20 篇相关文章

Keyword: reinforcement learning

Learning Stateful Predictive Knowledge From Experience

从经验中学习有状态的预测知识

• Authors: Yan Song, Xidong Feng, Bo Liu, Xinyu Cui, Haotian Fu, Zichen Liu, Mengyue Yang, Cheng Deng, Jian Zhao, Jun Wang
• Subjects: Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28638
• Pdf link: https://arxiv.org/pdf/2607.28638
• Abstract As large language model (LLM) agents increasingly learn from experience, they primarily rely on trajectory-level reflection to extract insights. Viewed through the lens of predictive knowledge, we argue that this approach operates on episodic hindsight rather than predictive foresight, yielding brittle, path-dependent heuristics. To address this, we propose Stateful Knowledge Learning (SKL). SKL shifts the agent's focus from trajectory-level summarization to maintaining Stateful Knowledge: explicit, declarative predictive assessments anchored to state. We first demonstrate a motivating example showing how stateful knowledge provides granularity, enhances generalization, and enables knowledge bootstrapping. To further scale up the idea, we introduce two algorithms via self-distillation (SKL-SD) and reinforcement learning (SKL-RL), training agents to autonomously extract state-grounded predictive knowledge from experience and learn to leverage it for policy making. Experiments on interactive environments (WebShop, ScienceWorld) and a complex reasoning task (ChessPuzzles) demonstrate that equipping models with the inherent ability to learn stateful predictive knowledge significantly outpaces current reflection-based training paradigms.
• 中文摘要 随着大型语言模型（LLM）代理越来越多地从经验中学习，他们主要依赖轨迹层面的反思来提取洞见。从预测知识的角度来看，我们认为这种方法基于事件性的事后诸葛亮而非预测性预见，导致脆弱的路径依赖启发式方法。为此，我们提出了有状态知识学习（Stateful Knowledge Learning，简称SKL）。SKL将智能体的关注点从轨迹级总结转向维护有状态知识：明确的声明性预测评估，锚定于状态。我们首先展示了一个激励人的例子，展示了有状态知识如何提供细粒度、增强泛化性并支持知识自助。为了进一步扩展这一理念，我们通过自蒸馏（SKL-SD）和强化学习（SKL-RL）引入了两种算法，训练智能体自主地从经验中提取基于状态的预测知识，并学习将其用于政策制定。在交互环境中（WebShop、ScienceWorld）和复杂推理任务（ChessPuzzles）上的实验表明，赋予模型学习有状态预测知识的固有能力，远远超过当前基于反思的训练范式。

ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning

ThinkReset：可学习的有界上下文长视野推理中间界面构建

• Authors: Fei Ding, Yongkang Zhang, Runhao Liu, Yuhao Liao, Zijian Zeng
• Subjects: Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28642
• Pdf link: https://arxiv.org/pdf/2607.28642
• Abstract Long chain-of-thought reasoning improves performance on complex problems, but it also introduces redundancy accumulation, context overflow, and error anchoring. We argue that under bounded context windows, the core bottleneck is not trajectory compression or test-time control, but the absence of a reusable intermediate interface that can replace discarded history and support continued solving. We further identify a key failure mode of outcome-reward-driven long-chain reinforcement learning: when the model has not solved the task before the window is nearly exhausted, the final-answer reward encourages premature guessing rather than continued careful reasoning. We propose ThinkReset, a text-space instantiation of this view. ThinkReset explicitly constructs reusable intermediate interfaces through interface writeback and reset, and directly optimizes post-reset continuation success. Across multiple long-horizon reasoning benchmarks, this perspective consistently improves success rates under fixed context windows.
• 中文摘要 长链思考能提升复杂问题的性能，但同时引入冗余累积、上下文溢出和错误锚定。我们认为，在有界上下文窗口下，核心瓶颈不是轨迹压缩或测试时间控制，而是缺乏可重用的中间接口，无法替代丢弃的历史记录并支持持续求解。我们还进一步识别了结果奖励驱动的长链强化学习的一个关键失败模式：当模型在窗口接近耗尽前尚未完成任务时，最终答案奖励会鼓励过早猜测，而非持续的细致推理。我们提出了ThinkReset，这是该视图的文本空间实例。ThinkReset 通过接口写回和重置明确构建可重用的中间接口，并直接优化重置后继续的成功率。在多个长期视野推理基准中，这一视角在固定上下文窗口下持续提升成功率。

TAPR: Enhancing LLM Performance with a Task-Aware Prompt Rewriter

TAPR：通过任务感知提示重写器提升LLM性能

• Authors: Oliver Savolainen, Emanuele Bastianelli, Hosein Azarbonyad
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.28657
• Pdf link: https://arxiv.org/pdf/2607.28657
• Abstract Large Language Models (LLMs) often require carefully crafted prompts to unlock their full potential, which can be a barrier for non-expert users. This work addresses the challenge by introducing a Task-Aware Prompt Rewriter (TAPR), a model that reformulates user prompts into task-optimized prompts with the explicit goal of improving downstream LLM performance. We train TAPR using reinforcement learning with Group Relative Policy Optimization (GRPO), where rewards are derived from LLM-as-judge evaluations of both the reformulated prompt and the corresponding task output. Experimental results on diverse tasks, such as question answering, summarization, and arithmetic reasoning, show that our method yields consistent gains over base models in prompt rewriting ability. Fine-tuning Phi-4-mini-instruct (as the base model for TAPR) produces prompts that contain clearer and more instructive language, leading to higher accuracy on established benchmarks such as Natural Questions and GSM8K. Our code is available at: this https URL
• 中文摘要 大型语言模型（LLM）通常需要精心设计的提示词来发挥其全部潜力，这对非专业用户来说可能是一大障碍。本研究通过引入任务感知提示重写器（TAPR）来应对这一挑战，该模型将用户提示重新表述为任务优化提示，明确目标是提升下游LLM的性能。我们通过强化学习和组相对策略优化（GRPO）训练TAPR，其中奖励来自LLM作为评判对重新表述提示和相应任务输出的评估。在问答、摘要和算术推理等多种任务上的实验结果表明，我们的方法在提示重写能力上相较基础模型有持续的提升。微调Phi-4-mini-instruct（作为TAPR的基础模型）可以生成包含更清晰、更具启发性的提示，从而在Natural Questions和GSM8K等成熟基准测试上获得更高的准确率。我们的代码可在以下地址获取：此 https URL

NeuroSynth: A Biologically Inspired Continual Reinforcement Learning Architecture for Mitigating Catastrophic Forgetting

NeuroSynth：一种生物启发的持续强化学习架构，用于缓解灾难性遗忘

• Authors: Yash Kini
• Subjects: Subjects: Neural and Evolutionary Computing (cs.NE); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28663
• Pdf link: https://arxiv.org/pdf/2607.28663
• Abstract Artificial Intelligence (AI) systems often perform well on isolated tasks but struggle under continual learning conditions, where training on new tasks can overwrite previously acquired knowledge, a failure mode known as catastrophic forgetting. Biological learning systems reduce this interference through complementary memory processes involving rapid hippocampal encoding and slower cortical consolidation. This study introduces NeuroSynth, a brain-inspired continual reinforcement learning architecture designed to mitigate catastrophic forgetting through a dual-pathway consolidation mechanism. NeuroSynth separates rapid task acquisition from long-term retention using distinct "plan" and "habit" pathways combined with replay and knowledge distillation. NeuroSynth was evaluated against Proximal Policy Optimization (PPO) and Elastic Weight Consolidation (EWC) across three sequential navigation tasks with changing goal locations in a non-revisitation continual learning setting. Across six independent seeds, NeuroSynth preserved substantially more early-task knowledge than PPO after sequential training, achieving 18.00% Task A success rate compared to 0.33% for PPO (p = 0.014929, Cohen's d = 1.49) and 35.33% Task B success rate compared to 0.00% for PPO (p = 0.002376, Cohen's d = 2.31). NeuroSynth also demonstrated higher final Task C performance than EWC, achieving 9.00% compared to 2.00% (p = 0.226643, Cohen's d = 0.56), indicating a moderate but not statistically significant advantage. These findings suggest that biologically inspired consolidation mechanisms may improve the stability-plasticity balance in continual reinforcement learning systems.
• 中文摘要 人工智能（AI）系统通常在孤立任务上表现良好，但在持续学习条件下表现不佳，因为新任务的训练可能会覆盖先前获得的知识，这种失败模式被称为灾难性遗忘。生物学习系统通过快速海马编码和较慢皮层巩固的互补记忆过程减少了这种干扰。本研究介绍了NeuroSynth，一种受大脑启发的持续强化学习架构，旨在通过双途径巩固机制减轻灾难性遗忘。NeuroSynth通过不同的“计划”和“习惯”路径，结合重放和知识蒸馏，将快速任务习得与长期记忆分离。NeuroSynth在非重访持续学习环境中，针对三个连续导航任务、目标位置变化进行了近端策略优化（PPO）和弹性权重整合（EWC）评估。在六个独立种子中，NeuroSynth在连续训练后保存的早期任务知识显著多于PPO，任务A的成功率为18.00%，PPO为0.33%（p = 0.014929，Cohen's d = 1.49），任务B成功率为35.33%，PPO为0.00%（p = 0.002376，Cohen's d = 2.31）。NeuroSynth在最终任务C的表现也优于EWC，达到9.00%对2.00%（p = 0.226643，Cohen's d = 0.56），显示中等但无统计学显著性优势。这些发现表明，生物启发的巩固机制可能改善持续强化学习系统中的稳定性与可塑性平衡。

Distilling Knowledge from Large Language Models into Lightweight Reinforcement Learning Agents for Autonomous Cyber Operations

将大型语言模型中的知识提炼为轻量级强化学习代理，用于自主网络操作

• Authors: Konur Tholl, François Rivest, Mariam El Mezouar, Adrian Taylor, Ranwa Al Mallah
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.28826
• Pdf link: https://arxiv.org/pdf/2607.28826
• Abstract Autonomous Cyber Operations (ACO) are increasingly important for defending enterprise networks as cyber threats continue to evolve in sophistication. ACO applications commonly employ Reinforcement Learning (RL) agents to learn defensive behaviors through interaction with environments. However, RL agents typically require extensive exploration during training, often resulting in unstable behavior and poor initial decision-making before converging toward effective defense strategies. In this work, we investigate the use of a Large Language Model (LLM) to improve autonomous defensive decision-making within an ACO environment. Through prompt engineering rather than fine-tuning, we demonstrate that an 8-billion parameter LLM pretrained on cybersecurity data can outperform a baseline RL agent in a modified CybORG CAGE Challenge 2 environment. We then propose an online policy distillation framework that transfers the LLM's defensive policy into a lightweight RL agent containing only 64,910 parameters, reducing model size by several orders of magnitude while maintaining effective defensive capabilities. This provides a pathway toward operationalizing frontier cybersecurity models within lightweight, deployable agents. To evaluate transferability, we construct CybORG scenarios ranging from 4 to 12 hosts and assess the approach across varying network configurations. We also evaluate teacher-guided RL stabilization strategies and observe that none consistently surpass the optimized teacher policy, suggesting policy-alignment limitations between reward-driven RL optimization and teacher-guided defense strategies. Our results demonstrate the potential of cybersecurity-focused LLMs as sources of expertise for autonomous cyber defense, while policy distillation provides a practical path toward operationalizing frontier cybersecurity models within efficient, scalable agents.
• 中文摘要 随着网络威胁不断复杂化，自主网络运营（ACO）在防御企业网络方面变得越来越重要。ACO应用通常使用强化学习（RL）代理，通过与环境的交互来学习防御行为。然而，强化学习的特工通常需要在训练中进行大量探索，常导致行为不稳定，决策不佳，难以形成有效的防御策略。本研究探讨了利用大型语言模型（LLM）提升ACO环境中自主防御决策的能力。通过即时工程而非微调，我们证明了一个以网络安全数据预训练的80亿参数LLM在修改后的CybORG CAGE Challenge 2环境中，能够优于基线强化学习代理。随后，我们提出了一个在线策略提炼框架，将LLM的防御策略转移到仅包含64,910参数的轻量级强化学习代理中，模型规模减少了数个数量级，同时保持有效的防御能力。这为在轻量级、可部署的代理中实现前沿网络安全模型提供了途径。为了评估可迁移性，我们构建了4至12个主机的CybORG场景，并评估了跨不同网络配置的方法。我们还评估了教师引导的强化学习稳定策略，观察到没有任何策略能持续超越优化教师政策，这表明奖励驱动的强化学习优化与教师引导防御策略之间存在政策对齐的局限。我们的研究结果展示了聚焦网络安全的大型语言模型作为自主网络防御专业知识来源的潜力，而策略提炼则为高效、可扩展的代理中实现前沿网络安全模型提供了切实可行的路径。

Hypergradient-based Bilevel Reinforcement Learning with Improved Sample Complexity

基于超梯度的双级强化学习，提升样本复杂度

• Authors: Naman Saxena, Mudit Gaur, Vaneet Aggarwal
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.28849
• Pdf link: https://arxiv.org/pdf/2607.28849
• Abstract Bilevel reinforcement learning (RL) is an important framework within the literature of RL that can be used to formalize various categories of problems, such as meta-learning, hierarchical task decomposition, and reinforcement learning from human feedback (RL-HF). Most of the bilevel RL algorithms are either not scalable because of using hypergradient with Hessian, or they suffer from high sample complexity because of using penalty-based approximation methods. In this work, we propose a hypergradient-based bilevel RL algorithm using the optimality of the Boltzmann policy for the entropy regularized discounted RL objective function. Our proposed algorithm is Hessian-free and obtains an iteration complexity of $O(\epsilon^{-1})$ and state-of-the-art sample complexity of $\tilde{O}(\epsilon^{-2})$ under mild regularity conditions. Further, in our convergence analysis, we are able to remove the assumption of the Polyak-Lojasiewicz (PL) condition on the outer-level objective function present in the prior state-of-the-art sample complexity work.
• 中文摘要 双层强化学习（RL）是强化学习文献中的一个重要框架，可用于形式化各种问题类别，如元学习、层级任务分解以及基于人类反馈的强化学习（RL-HF）。大多数双层强化学习算法要么因使用Hessian的超梯度而不可扩展，要么因使用基于惩罚的近似方法而样本复杂度较高。本研究提出一种基于超梯度的双层强化强化学习算法，利用玻尔兹曼策略对熵正则化折现的RL目标函数的最优性。我们提出的算法无黑森分布，在温和正则条件下，迭代复杂度为$O（\epsilon^{-1}）$，最先进的样本复杂度为$\tilde{O}（\epsilon^{-2}）$。此外，在收敛分析中，我们能够消除之前最先进样本复杂度工作中对外层目标函数的Polyak-Lojasiewicz（PL）条件假设。

Gated Q-learning: Add Off-Policy Bias to Taste

门控Q学习：为品味添加非政策偏见

• Authors: Brett Daley
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.28916
• Pdf link: https://arxiv.org/pdf/2607.28916
• Abstract Multistep credit assignment is critical for sample-efficient reinforcement learning, yet managing off-policy bias in Q-learning remains a fundamental challenge. For 30 years, practitioners have been limited to a binary choice: eliminate the bias at the cost of severely truncated eligibility traces (Watkins' Q($\lambda$)), or ignore the bias to learn faster while injecting detrimental errors into the value estimates (Peng's Q($\lambda$)). Modern off-policy estimators fail to resolve this tension, as importance-sampling ratios collapse under Q-learning's greedy target policy. We introduce Gated Q-learning, a novel algorithmic framework that ends this dilemma by smoothly interpolating between the two historical extremes. Rather than relying on importance sampling, our approach employs a continuous, state-action-dependent gating mechanism to selectively attenuate eligibility traces in an exploration-aware manner. We provide a rigorous theoretical foundation for this mechanism, proving that the expected operator remains a contraction mapping and deriving its exact fixed point. Empirical evaluations verify that intermediate gating safely enables longer credit-assignment horizons, yielding faster initial learning than either extreme. Gated Q-learning offers a simple alternative to importance sampling while enabling customization of the effective multistep horizon and the amount of off-policy bias in Q-learning agents.
• 中文摘要 多步学分赋值对于高效样本强化学习至关重要，但管理Q学习中的偏离策略偏差仍是一个根本挑战。30年来，从业者只能做出二元选择：要么以严重截断资格痕迹为代价消除偏见（Watkins Q（$\lambda$）），要么忽略偏见以加快学习速度，同时在价值估计中注入有害误差（Peng Q（$\lambda$））。现代非政策估计器未能解决这种矛盾，因为重要性抽样比率在Q-learning贪婪的目标策略下崩溃。我们引入了门控Q学习，一种新颖的算法框架，通过平滑插值这两个历史极端来解决这一困境。我们的方法不依赖重要性抽样，而是采用持续的、依赖状态动作的门槛机制，以探索意识的方式选择性地减弱资格痕迹。我们为该机制提供了严谨的理论基础，证明预期算符仍然是收缩映射并推导其精确不动点。实证评估证实，中级门槛安全地实现了更长的学分分配期限，使初始学习速度比极端更快。门控Q学习为重要性抽样提供了一种简单的替代方案，同时允许对有效多步视野和Q学习代理中偏离策略偏差的定制化。

Think2Go: Generative Next POI Recommendation with LLM Reasoning

Think2Go：基于LLM推理的生成性下一个POI推荐

• Authors: Zhuang Zhuang, Shanshan Feng, Hangwei Qian, Mingqi Yang, Heng Qi, Yanming Shen, Baocai Yin
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.28997
• Pdf link: https://arxiv.org/pdf/2607.28997
• Abstract Next Point-of-Interest (POI) recommendation task focuses on mining user behavioral preference patterns from historical check-ins to provide personalized suggestions for the next destination. Existing methods primarily rely on shallow contextual information and handcrafted feature interactions to predict the next POI. However, the inherent sparsity and complexity of user mobility patterns limit the computational capacity of non-reasoning models to capture deep intent, while large language models (LLMs) perform suboptimally because they lack a deep understanding of semantic IDs (SIDs) when SIDs are trained separately. To address these limitations, we propose Think2Go, a novel generative next POI recommendation framework, which enhances the model's comprehension of SID representations and explores diverse spatial-temporal patterns via test-time computational scaling. We unify supervised fine-tuning (SFT) and reinforcement learning (RL)-based reasoning within a single architecture, enabling joint optimization of memorization and adaptive reasoning to better retain user behavior patterns while exploring diverse user preferences. To further calibrate policy optimization in adaptive reasoning, we propose two advantage weighting mechanisms that integrate (1) prompt epistemic uncertainty, estimated via kernel density methods to assess the spatial-temporal periodic pattern alignment between queries and user history, promoting increased exploration under high epistemic uncertainty; and (2) reward-informed advantage scaling, captured by normalizing rewards against their maxima to adapt update magnitudes, thereby improving training stability and mitigating overfitting to noisy signals. This joint calibration forms an implicit curriculum learning strategy, delivering fine-grained, instance-aware policy updates that prevent entropy collapse and support robust exploration.
• 中文摘要 下一个兴趣点（POI）推荐任务侧重于从历史签到中挖掘用户行为偏好模式，以提供个性化的下一个目的地建议。现有方法主要依赖浅层上下文信息和手工设计的特征交互来预测下一个兴趣点。然而，用户移动模式的固有稀疏性和复杂性限制了非推理模型捕捉深度意图的计算能力，而大型语言模型（LLM）因缺乏对语义ID（SID）的深入理解而表现不佳，尤其在SID单独训练时。为解决这些局限性，我们提出了Think2Go，一种新的生成式次POI推荐框架，它增强了模型对SID表示的理解，并通过测试时计算尺度探索了多样的时空模式。我们将基于监督的微调（SFT）和强化学习（RL）推理统一在单一架构中，实现记忆与自适应推理的联合优化，更好地保留用户行为模式，同时探索多样化的用户偏好。为了进一步校准适应性推理中的策略优化，我们提出了两种优势加权机制，分别整合了（1）通过核密度方法估计的即时认识不确定性，以评估查询与用户历史之间时空周期模式的对齐，促进在高认知不确定性下增加探索;以及（2）奖励知情优势缩放，通过将奖励与其最大值进行归一化以适应更新幅度，从而提升训练稳定性并减少对噪声信号的过度拟合。这种联合校准形成了隐式课程学习策略，提供细粒度、实例感知的策略更新，防止熵崩溃并支持稳健的探索。

Evaluation-Verification Reward for Consistent Multi-Reference Image Editing

对一致性多参考图像编辑的评估-验证奖励

• Authors: Yingmao Miao, Pengfei Zhang, Xiaochen Lv, Meng Yu, Lei Sun, Xiangxiang Chu, Chao Shen, Chenhao Lin
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.29025
• Pdf link: https://arxiv.org/pdf/2607.29025
• Abstract While recent image editing models have made rapid progress, multi-reference editing remains challenging, particularly in maintaining visual consistency across references and ensuring overall visual harmony. Reinforcement learning has proven highly effective for text-to-image generation and single-image editing, but its extension to multi-reference editing is hindered by the absence of suitable reward models that capture multi-image relational constraints. Moreover, naively using multimodal large language models(MLLMs) as zero-shot evaluators faces a key tension between hallucination-prone long-form reasoning and the limited deductive power of short-form judgments. We address these issues with a Multi-dimensional Evaluation-Verification Reward(EVR). EVR decomposes evaluation into distinct visual criteria; for each criterion, an MLLM Evaluator generates multiple candidate hypotheses, and a Verifier grounds each claim in concrete visual evidence to accept or reject it, producing reliable and fine-grained reward signals. Together with a scalable data pipeline, our method enables RL fine-tuning of off-the-shelf editors without architectural changes. Extensive experiments show substantial gains over the base Qwen-Image-Edit, improving consistency and harmony to match or surpass NanoBanana.
• 中文摘要 尽管最新的图像编辑模型取得了快速进展，但多参考编辑依然具有挑战性，尤其是在保持参考文献间视觉一致性和确保整体视觉和谐方面。强化学习在文本生成和单图编辑方面已被证明非常有效，但由于缺乏合适的奖励模型来捕捉多图像关系约束，其扩展到多引用编辑受到阻碍。此外，天真地将多模态大型语言模型（MLLM）作为零样本评估者，面临着易产生幻觉的长式推理与短形式判断有限演绎力之间的关键张力。我们通过多维评估-验证奖励（EVR）来解决这些问题。EVR将评估分解为不同的视觉标准;对于每个标准，MLLM评估者生成多个候选假设，验证者则以具体的视觉证据为基础，支持或拒绝，从而产生可靠且细致的奖励信号。结合可扩展的数据管道，我们的方法使现成编辑器能够在不做架构更改的情况下进行强化学习微调。大量实验显示，相较基础Qwen-Image-Edit有显著提升，提升了一致性和谐音，以匹配甚至超越NanoBanana。

CLIFT: Turning Gemini Robotics On-Device into Humanoid Specialists via Non-Invasive Closed-Loop Iterative Fine-Tuning

CLIFT：通过非侵入式闭环迭代微调，将Gemini机器人设备转变为类人型专家

• Authors: Yuxin Chen, Hari Srikanth, Nathan Jew, Menglin Wu, Pengcheng Wang, Junli Ren, Masayoshi Tomizuka, Peng Xu, Jinyu Xie, Thomas Tian
• Subjects: Subjects: Robotics (cs.RO); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.29172
• Pdf link: https://arxiv.org/pdf/2607.29172
• Abstract While robot foundation models are growing increasingly capable, the strongest models are typically trained on proprietary data and remain closed-source, limiting downstream users' ability to adapt them to new tasks, embodiments, and deployment settings. Following the LLM community, an emerging access paradigm for closed-weight robot foundation models is the managed supervised fine-tuning (SFT) API, where users submit training data and receive a tuned policy without access to model weights, gradients, or training internals. While such APIs let downstream users leverage powerful proprietary foundation models, they restrict policy improvement to pure imitation, ruling out reinforcement learning and other closed-loop methods that rely on internal training signals. This limitation is particularly acute for agile, contact-rich humanoid manipulation, where the gap between policy outputs and deployed behavior is large due to novel states, action tracking dynamics, latency, and controller-specific failure modes. We study how effective this managed-API regime is for humanoid adaptation, and how closed-loop improvement can be realized within it to push policies toward task mastery. We conduct one of the first empirical studies of managed-API adaptation on a real humanoid, instantiated on Gemini Robotics On-Device (GROD). We find that direct SFT through the API substantially outperforms a leading open-weight VLA trained on the same demonstrations, yet still falls short of deployment-level mastery on agile, contact-rich tasks. To close this gap, we introduce CLIFT: Closed-Loop Iterative Fine-Tuning, which turns deployment-time reward feedback into API-compatible supervised data and enables closed-loop policy improvement without accessing weights, gradients, likelihoods, or losses-pushing GROD to near-perfect success after two flywheel cycles, all without "opening the model box."
• 中文摘要 尽管机器人基础模型的能力日益增强，但最强大的模型通常基于专有数据训练，保持闭源，限制了下游用户适应新任务、新实例和部署环境的能力。继LLM社区之后，闭权重机器人基础模型的新兴访问范式是管理监督微调（SFT）API，用户提交训练数据并获得调优策略，无需访问模型权重、梯度或训练内部信息。虽然这些API让下游用户能够利用强大的专有基础模型，但它们限制了策略改进仅限于纯粹的模仿，排除了依赖内部训练信号的强化学习和其他闭环方法。这一限制在敏捷、接触丰富的类人机器人操作中尤为严重，因为由于新状态、动作跟踪动态、延迟和控制器特定故障模式，策略输出与部署行为之间的差距很大。我们研究该托管API体系对类人适应的有效性，以及如何在其中实现闭环改进，推动政策向任务掌握方向发展。我们开展了最早的在真实人形上托管API适配的实证研究之一，该研究在Gemini机器人设备（GROD）上实现。我们发现，通过API直接进行SFT的表现远超基于相同演示训练的领先开放权重VLA，但在敏捷且接触丰富的任务上仍未达到部署级别的掌握。为弥合这一差距，我们引入了CLIFT：闭环迭代微调，将部署时的奖励反馈转化为兼容API的监督数据，实现闭环策略改进，无需访问权重、梯度、似然或亏损，推动GROD在两次飞轮周期后几乎完美成功，且无需“打开模型盒”。

Learning Latent Reasoning Traces for Scalar Reward Models End-to-End

端到端的标量奖励模型学习潜在推理痕迹

• Authors: Sanwoo Lee, Clive Bai, Hsiu-Yuan Huang, Kun Liang, Weijie Liu, Yunfang Wu
• Subjects: Subjects: Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.29185
• Pdf link: https://arxiv.org/pdf/2607.29185
• Abstract Reward models (RMs) are central to aligning large language models with human preferences via reinforcement learning. Although traditional scalar RMs enable efficient and probabilistic reward modeling, they rely on superficial cues that fail to generalize to complex or out-of-distribution (OOD) tasks. Conversely, generative RMs leverage extensive reasoning to improve robustness on challenging tasks, but their natural language-based scores lack the numerical flexibility and probabilistic interpretability that scalar RMs offer. While recent approaches combine both paradigms through off-policy multi-task learning, such parallel optimization does not guarantee that generated reasoning traces actively align with or benefit downstream scalar reward prediction. To address this mismatch, we propose LatentRM, a reward modeling framework that learns intermediate reasoning traces as discrete latent variables to explicitly maximize the likelihood of downstream scalar rewards. Through on-policy optimization of the latent reasoning space end-to-end, LatentRM tightly couples deep reasoning-based evaluation with precise scoring. Extensive validations on in-distribution and OOD datasets and RLHF show that LatentRM outperforms scalar, generative, and hybrid RMs on preference modeling and policy alignment across tasks ranging from open-ended conversation to complex reasoning.
• 中文摘要 奖励模型（RM）是通过强化学习将大型语言模型与人类偏好对齐的核心。尽管传统的标量RM能够高效且概率地建模奖励，但它们依赖的线索表面，无法推广到复杂或分布外（OOD）任务。相反，生成式均分利用广泛的推理提升复杂任务的鲁棒性，但其基于自然语言的评分缺乏标量均数所具备的数值灵活性和概率解释性。虽然近期方法通过非策略多任务学习结合了这两种范式，但这种并行优化并不保证生成的推理轨迹能主动与下游标量奖励预测对齐或受益。为解决这一不匹配，我们提出了 LatentRM 这一奖励建模框架，它通过学习中间推理痕迹作为离散潜在变量，明确最大化下游标量奖励的可能性。通过对潜在推理空间的端到端优化，LatentRM紧密结合了基于深度推理的评估与精确评分。对分布内数据集和户外数据集及RLHF的广泛验证表明，LatentRM在偏好建模和策略对齐方面表现优于标量、生成和混合RM，涵盖从开放式对话到复杂推理等任务。

SAF-OPD: Stable Advantage Fusion for On-Policy Distillation

SAF-OPD：稳定优势融合用于政策提炼

• Authors: Yifan Ding, Xincheng Wei, Yoshua Y. Li, Ziheng Li, Yuquan Lu, Siyu Zhang, Dongsheng Ma, Rongxiang Weng, Xunliang Cai, Yun Chen
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.29209
• Pdf link: https://arxiv.org/pdf/2607.29209
• Abstract Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.
• 中文摘要 带有可验证奖励的强化学习（RLVR）向每个代币广播一个响应级奖励，而策略提炼（OPD）则将每个代币与更强教师的评分，获得密集优势，但限制教师质量的表现，并抑制超出该水平的探索。它们的互补性使得结合RLVR和OPD具有前景，但我们发现，将两者融合为固定系数会导致熵坍缩：一种是幅度错配，即代币级OPD优势可能远超有界RLVR优势并抹去其信号;另一种是时间错配，持续的全强OPD不断吸引学生向教师倾向，限制了超越它所需的探索。我们提出了SAF，一种稳定优势融合框架，通过仅应用于OPD优势的轻量级四级流水线解决这两个问题：一个稀疏然后压缩的强度控制机制，配合预热再退火的时序控制机制，每个阶段独立切换，且开销可忽略不计。通过与GRPO实例化RLVR，我们通过七个数学推理和代码生成基准测试评估SAF，采用Qwen3-1.7B/4B/8B：SAF避免熵坍缩，且持续优于固定系数GRPO+OPD融合，在六个模型域设置中总分提升0.51-2.70%，同时实现更稳定的训练。

Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

《不要混合奖励，混合策略：多奖励强化学习的策略分解与优化》

• Authors: Ruiming Liang, Yi Zhong, Yizhen Yuan, Yinan Zheng, Tianyi Tan, Tianyue Wang, Haiyun Guo, Jinqiao Wang, Xianyuan Zhan
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.29246
• Pdf link: https://arxiv.org/pdf/2607.29246
• Abstract Modern large language models (LLMs) are expected not just to answer correctly, but to adapt their behavior to different human values and use cases. As a result, multi-reward reinforcement learning (RL) has become an increasingly important problem for LLMs, where each reward captures a different aspect of desired behavior. However, optimizing with multiple rewards suffers from a more severe alignment tax issue, where different optimization objectives can trade off or even conflict with each other, leading to unstable and inefficient post-training. In this work, we propose PRISM, a new multi-reward RL framework built upon the idea of policy-space decomposition and composition. Instead of compositing different rewards, PRISM optimizes a set of standalone positive policies and a global negative policy. This alleviates the potential conflict during multi-reward policy optimization, while enabling controllability during inference by flexible policy composition. Experiments on scientific reasoning, tool-use reasoning, and helpfulness-safety alignment show that PRISM consistently outperforms existing multi-reward RL baselines, with extra controllability for inference-time preference control.
• 中文摘要 现代大型语言模型（LLMs）不仅被期望正确回答，还需要根据不同的人类价值观和用例调整其行为。因此，多重奖励强化学习（RL）已成为LLMs日益重要的问题，每个奖励都捕捉期望行为的不同方面。然而，多奖励优化存在更严重的对齐税问题，不同优化目标可能相互权衡甚至冲突，导致训练后不稳定且效率低下。在本研究中，我们提出了PRISM，一种基于策略空间分解与组合理念的新型多奖励强化学习框架。PRISM没有组合不同的奖励，而是优化一组独立的正向政策和一个全局负面策略。这缓解了多奖励策略优化过程中潜在的冲突，同时通过灵活的策略组合实现了推理的可控性。科学推理、工具使用推理以及帮助性-安全对齐的实验表明，PRISM始终优于现有的多奖励强化学习基线，并且在推理时间偏好控制方面具有更高的可控性。

Translation with Thought: Difficulty-Adaptive Reasoning via Reinforcement Learning for Multi-Domain Machine Translation

带思想的翻译：通过强化学习实现多领域机器翻译的难度自适应推理

• Authors: Yongshi Ye, Biao Fu, Chongxuan Huang, Yidong Chen, Xiaodong Shi
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.29287
• Pdf link: https://arxiv.org/pdf/2607.29287
• Abstract Multi-domain machine translation (MDMT) poses a unique challenge due to varying levels of linguistic complexity across domains. Inspired by human translators' ability to adapt reasoning effort based on difficulty, we propose TwT (Translation with Thought), a resource-rational framework that learns to modulate inference between intuitive and deliberate reasoning. TwT is trained in two stages: (1) supervised fine-tuning on difficulty-aware long chain-of-thought traces distilled from DeepSeek-R1 and rewritten by GPT-4o to reflect human-like reasoning economy, and (2) reinforcement learning with a hybrid reward to optimize translation quality and reasoning efficiency. Evaluated on 15 benchmarks spanning in-domain and out-of-domain settings, as well as 3 seen and 59 unseen languages, with ablations across three backbone models, TwT-7B and TwT-14B outperform much larger SOTA reasoning models in translation quality, while reducing token usage by 32--60\%. These results confirm that aligning translation behavior with cognitive principles enables robust generalization, high translation quality, and efficient reasoning in MDMT.
• 中文摘要 多域机器翻译（MDMT）由于各领域语言复杂度不同，带来了独特的挑战。受人类翻译者根据难度调整推理能力的启发，我们提出了TwT（带思维翻译），这是一种资源-理性框架，学习调节直觉推理与刻意推理之间的关系。TwT训练分为两个阶段：（1）对DeepSeek-R1提炼并由GPT-4o重写的难度感知长思考链进行监督微调，以反映类人推理经济;（2）强化学习与混合奖励，以优化翻译质量和推理效率。在15个涵盖域内和域外环境的基准测试，以及3种可见语言和59种未可见语言的评估中，TwT-7B和TwT-14B在翻译质量上优于更大规模的SOTA推理模型，同时令牌使用率降低了32%-60%。这些结果证实，将翻译行为与认知原则对齐，有助于MDMT中稳健的泛化、高翻译质量和高效的推理。

Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification

通过最佳策略识别的高效分层强化学习示例

• Authors: Anders Jonsson, Emilie Kaufmann, Gianmarco Tedeschi, Lorenzo Steccanella
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.29294
• Pdf link: https://arxiv.org/pdf/2607.29294
• Abstract We present HBPI-UCRL, a model-based algorithm for hierarchical reinforcement learning (HRL) that learns high-level and low-level policies in parallel. HBPI-UCRL exploits the fact that a high-level transition corresponds to a multi-step transition at the low level. We introduce two conditions on the low-level dynamics that are sufficient to make parallel HRL learnable. When these conditions hold, we prove that HBPI-UCRL has a polynomial sample complexity in the problem parameters. In the sparse-reward, goal-directed setting, our sample complexity upper bound for HBPI-UCRL is strictly lower than that of its non-hierarchical counterpart, providing theoretical justification for the empirical success of HRL.
• 中文摘要 我们介绍HBPI-UCRL，一种基于模型的分层强化学习（HRL）算法，能够并行学习高层和低层策略。HBPI-UCRL利用了高层级过渡对应低层次多步转移的事实。我们引入了两个低层动态条件，使平行HRL可学习。当这些条件成立时，我们证明HBPI-UCRL在问题参数中具有多项式样本复杂度。在稀疏奖励、目标导向的环境中，我们对HBPI-UCRL的样本复杂度上限严格低于非层级对应的，这为HRL的实证成功提供了理论依据。

Explore Beyond the Boundary Using Entropic Information

利用熵信息探索边界之外

• Authors: Bumgeun Park, Donghwan Lee
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.29419
• Pdf link: https://arxiv.org/pdf/2607.29419
• Abstract In reinforcement learning, exploration with sparse and delayed rewards presents a significant challenge due to the limited feedback available for guiding the learning process. Addressing this issue requires extensive exploration in the state space to discover valuable reward signals. In this paper, we propose Entropic Information for Exploration (ENTINEX), a novel method that enhances exploration by incentivizing agents to explore beyond the boundaries of the state distribution. ENTINEX achieves this by assigning intrinsic rewards to these boundaries, leveraging entropic information to identify them effectively. Through extensive experimentation, we demonstrate that ENTINEX consistently improves exploration performance in environments characterized by sparse and delayed rewards. Our experimental results show that ENTINEX outperforms existing exploration methods, highlighting its effectiveness in both sparse and delayed reward scenarios.
• 中文摘要 在强化学习中，奖励稀疏且延迟的探索面临重大挑战，因为可用来指导学习过程的反馈有限。解决这一问题需要在状态空间中广泛探索，以发现有价值的奖励信号。本文提出了熵信息探索（ENTINEX）新颖方法，通过激励智能体探索超出状态分布边界，增强探索效果。ENTINEX 通过赋予这些边界内在奖励，利用熵信息有效识别它们来实现这一点。通过大量实验，我们证明了ENTINEX在奖励稀疏和延迟的环境中持续提升探索表现。我们的实验结果显示，ENTINEX 优于现有探索方法，凸显了其在稀疏奖励和延迟奖励场景中的有效性。

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

LEMUR：学习与多目标强化学习对齐偏好反馈

• Authors: Manith Adikari, Bei Peng, Samuele Vinanzi, Angelo Cangelosi
• Subjects: Subjects: Artificial Intelligence (cs.AI); Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.29559
• Pdf link: https://arxiv.org/pdf/2607.29559
• Abstract Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.
• 中文摘要 强化学习（RL）系统通常使用单一且明确的标量奖励函数进行训练。然而，现实世界的决策任务往往涉及多个相互竞争的目标，比如绩效与效率之间的较量，而实地的奖励函数难以明确或难以获得。多目标强化学习（MORL）通过将奖励建模为向量来解决此类权衡，现有方法通常假设每个目标都能获得明确指定的奖励函数，继承了单目标强化学习面临的相同挑战。与此同时，基于偏好的强化学习（PbRL）在解决复杂任务时展现出巨大潜力，且主要在单一目标环境中进行研究。在本研究中，我们通过LEMUR弥合这一鸿沟：学习与多目标强化学习（Multi-Objective Reinforcement Learning with Preference feedback）保持一致，这是一种新颖框架，主体通过交互式学习多重目标策略，从而学习最优多目标策略。我们的方法通过人类反馈共同学习策略和多种目标特定奖励模型，使智能体在学习过程中能够有效平衡竞争目标。我们在多种基准多目标任务中评估了LEMUR的应用，实证结果显示其优于基线方法的性能。我们的方法为解决无预设奖励函数的多目标决策任务提供了有前景的方向。

Convergence and Regret of the Policy Gradient for Multi-Armed Bandits in Diffusion Environment

扩散环境中多武装匪徒政策梯度的趋同与遗憾

• Authors: Yanwei Jia, Du Ouyang
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.29593
• Pdf link: https://arxiv.org/pdf/2607.29593
• Abstract This paper studies the policy gradient update for a multi-arm bandit problem in diffusion environment that is described by a stochastic differential equation (SDE) under the continuous-time reinforcement learning framework by Wang et al. (2020), Jia and Zhou (2022b). With the logit parameterization for the stochastic policy, we show that it converges almost surely to the optimal arm under an arbitrary constant learning rate. Furthermore, we derive the non-asymptotic regret upper bound when the constant learning rate is below a time-invariant threshold; and the regret bound has order $O(\log T)$. We improve the analysis in Lattimore (2026a) for the same SDE by constructing a novel Lyapunov function and demonstrate the transparency of analyzing policy gradient using the tools in SDEs. In addition, the same Lyapunov function is also helpful in analyzing the discrete-time policy gradient algorithm.
• 中文摘要 本文研究了在扩散环境中，Wang等人（2020）、Jia 和 Zhou（2022b）在连续时间强化学习框架下，通过随机微分方程（SDE）描述的多臂盗贼问题的策略梯度更新。通过随机策略的logit参数化，我们证明它几乎必然收敛到任意恒定学习率下的最优臂。此外，当恒定学习率低于时不变阈值时，我们推导出非渐近遗憾上界;遗憾界限的阶数为$O（\log T）$。我们在Lattimore（2026a）中通过构造一个新的李雅普诺夫函数，改进了同一SDE的分析，并展示了利用SDE工具分析政策梯度的透明度。此外，同一李雅普诺夫函数也有助于分析离散时间策略梯度算法。

WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning

WCM：视觉-语言-行动强化学习的世界批判模型

• Authors: Senyu Fei, Xiaopeng Yu, Siyin Wang, Xianzhong Zhao, Jingjing Gong, Xipeng Qiu
• Subjects: Subjects: Robotics (cs.RO); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.29613
• Pdf link: https://arxiv.org/pdf/2607.29613
• Abstract Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.
• 中文摘要 视觉-语言-行动（VLA）模型的强化学习（RL）后训练显示出对机器人操作的强大前景。在强化学习方法中，基于批评者的方法依赖于主要基于单帧观测或单帧VLM骨干潜伏的值估计器，这与机器人控制的部分可观测性质存在根本性不匹配。将观察历史纳入批评者时，天真地将观测历史纳入高维视觉空间时会带来指数级复杂性，但由于纯标量回归未能充分指导跨时空动态学习，仍失败。我们将根本原因归为状态近似问题：没有明确的世界建模目标，批评者的表征无法捕捉准确估计价值所需的时间结构。为此，我们提出了基于轻量化LeJEPA架构构建的世界批判模型（WCM）;WCM联合预测未来潜在状态并估计数值，使批评者的表示被明确训练以捕捉时间动态，而不仅仅是回归标量回报。WCM无缝集成到策略内和非策略训练管道中，并兼容包括Pi0、Pi0.5和OpenVLA-OFT在内的最先进的VLA骨干网。对四个基准测试中的149项任务进行的广泛实验表明，WCM在分布内外环境下均持续保持最先进的性能，并具有特别显著的泛化提升。我们还进一步验证了 WCM 在七个实际操作任务上的应用，使用 OpenVLA-OFT 和 Pi0.5 配合非策略强化学习，确认其在不同环境中的稳定部署。

CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding

CodeShrink：自适应视觉压缩，实现高效的多模态代码理解

• Authors: Wenxin Tang, Jingyu Xiao, Zhenyu Liu, Zipeng Xie, Junliang Liu, Wang Luo, Yuan Jiang, Yintong Huo, Michael Lyu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.29637
• Pdf link: https://arxiv.org/pdf/2607.29637
• Abstract Rendering source code as images offers a promising way to reduce the input costs of Multimodal Large Language Models (MLLMs). Adjusting image resolution can trade visual token cost against content fidelity. However, resolution scaling alone overlooks two sources of inefficiency: blank regions created by line breaks and indentation, and code regions irrelevant to the current instruction. Moreover, the best compression setting varies across inputs, tasks, and models, limiting fixed-ratio strategies. We propose CodeShrink, an adaptive visual compression framework with three components. Blank-Free Rendering replaces whitespace-dependent layouts with compact layouts and explicit structural markers, removing layout-induced tokens. Adaptive Compression Configuration uses a lightweight agent trained with reinforcement learning to predict a per-input setting that balances token efficiency and readability. Dominant Token Selection jointly analyzes the instruction and code image to prune task-irrelevant visual tokens during inference. We evaluate CodeShrink on code question answering, clone detection, and code completion. CodeShrink reduces visual token use by up to 71.2\% while matching or exceeding uncompressed text-only inputs, and consistently outperforms text-based and visual compression baselines across all three tasks. These results show that combining layout compaction, adaptive configuration, and instruction-aware pruning can make multimodal code understanding more efficient. Our code is available at this https URL.
• 中文摘要 将源代码渲染为图像，为降低多模态大型语言模型（MLLM）的输入成本提供了有前景的方法。调整图像分辨率可以利用视觉代币成本与内容忠实度进行权衡。然而，仅靠分辨率尺度忽略了两个低效因素：由换行和缩进产生的空白区域，以及与当前指令无关的代码区域。此外，最佳压缩设置在不同输入、任务和模型之间存在差异，限制了固定比率策略。我们提出了CodeShrink，一个由三个组成部分组成的自适应视觉压缩框架。空白无渲染用紧凑布局和显式结构标记取代依赖空白的布局，去除布局引发的标记。自适应压缩配置使用经过强化学习训练的轻量级代理，预测每个输入的设置，平衡令牌效率和可读性。主导词汇选择结合分析指令图像和代码图像，在推理过程中修剪与任务无关的视觉词符。我们评估CodeShrink在代码问题答复、克隆检测和代码完成方面的表现。CodeShrink在匹配或超过未压缩纯文本输入的同时，将视觉代币使用减少多达71.2%，并且在这三项任务中始终优于基于文本和视觉压缩的基线。这些结果表明，结合布局压缩、自适应配置和指令感知剪枝，可以提高多模代码理解的效率。我们的代码可在此 https URL 访问。

Keyword: diffusion policy

There is no result