生成时间: 2026-07-20 18:54:07 (UTC+8); Arxiv 发布时间: 2026-07-20 20:00 EDT (2026-07-21 08:00 UTC+8)

今天共有 22 篇相关文章

Keyword: reinforcement learning

Co-Design of Aeroelastic Systems with Deep Reinforcement Learning

与深度强化学习的气动弹性系统共同设计

• Authors: Yao Cheng Li, Urban Fasel
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.15329
• Pdf link: https://arxiv.org/pdf/2607.15329
• Abstract Control co-design considers the physical system and its controller together, enabling the strong coupling between system design and control to be uncovered and exploited. This is especially relevant in aeroelastic flight systems, where structural, aerodynamic, and control design choices jointly determine manoeuvrability and efficiency. This paper presents a model-free nested co-design framework for aeroelastic systems using deep reinforcement learning, in which a design-conditioned control policy is trained with proximal policy optimisation while an outer loop updates a distribution over candidate design parameters. The approach is evaluated on three case studies of increasing complexity: a spring-mass-damper system, a pitch-plunge-flap aerofoil, and a highly flexible high-aspect-ratio glider performing a thermal-soaring mission in a stochastic environment. Across these case studies, the framework is shown to progressively concentrate the design search towards high-performing regions and to outperform policies trained on randomly sampled designs. The results also show that reward shaping plays an important role in enabling stable learning in partially observed and stochastic environments. In the final glider case, the method jointly addresses wing design, flight control, and mission-level behaviour in the presence of aeroelastic coupling and atmospheric uncertainty. These results highlight the potential of model-free co-design for complex aeroelastic systems in which design, control, and mission objectives are tightly coupled.
• 中文摘要 控制共构考虑物理系统与控制器的结合，使系统设计与控制之间的强耦合得以被发现和利用。这在气动弹性飞行系统中尤为重要，因为结构、气动和控制设计的选择共同决定了机动性和效率。本文提出了一个基于深度强化学习的无模型嵌套共设计框架，利用近端策略优化训练设计条件控制策略，同时通过外环更新候选设计参数的分布。该方法基于三个日益复杂化的案例研究进行评估：弹簧-质量-阻尼系统、俯仰-俯冲-襟翼翼型，以及一种高度灵活的高展弦比滑翔机，在随机环境中执行热滑翔任务。在这些案例研究中，该框架逐渐将设计搜索集中于高效能区域，并优于基于随机抽样设计训练的策略。结果还表明，奖励塑造在部分观察和随机环境中实现稳定学习中起着重要作用。在最后一款滑翔机中，该方法结合机翼设计、飞行控制和任务层级行为，同时涉及气动弹性耦合和大气不确定性。这些结果凸显了无模型协同设计在复杂气动弹性系统中设计、控制和任务目标紧密结合的潜力。

Robust Peak-cost Constrained Reinforcement Learning

稳健的峰值-成本约束强化学习

• Authors: Shilpa Mukhopadhyay, Sourav Ganguly, Santosh Mohan Rajkumar, Honghao Wei, Debdipta Goswami, Arnob Ghosh
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15457
• Pdf link: https://arxiv.org/pdf/2607.15457
• Abstract We study robust peak-cost constrained reinforcement learning (RP-CRL), where the objective is to maximize expected reward while controlling the maximum cost encountered along a trajectory. This setting is motivated by safety-critical applications in which a single large violation can be catastrophic and therefore cannot be adequately captured by the standard CMDP framework based on expected cumulative cost. Existing reachability-constrained RL methods adopt Lagrangian-based approaches, yet the underlying duality properties of peak-cost constrained MDPs remain unclear. We show that, unlike standard CMDPs, peak-cost constrained MDPs may not admit zero duality gap. We further consider a robust formulation to address simulator-to-real-world mismatch in the transition dynamics. To solve this problem, we develop a surrogate optimization framework and a robust value estimation method based on integral probability metrics. We prove that, with appropriate hyperparameter choices, the surrogate solution attains the same robust reward value as the original problem while violating the constraint by at most epsilon. Experiments show that the proposed method effectively enforces safety under dynamics perturbations while retaining strong reward performance.
• 中文摘要 我们研究了强健峰值-成本约束强化学习（RP-CRL），其目标是在控制轨迹中遇到的最大成本的同时，最大化预期奖励。这种设定源于安全关键应用，其中单个重大违规可能造成灾难性，因此无法被标准CMDP框架充分涵盖基于预期累计成本的因素。现有可达性约束的强化学习方法采用基于拉格朗日的方法，但峰值成本约束的MDP的基本对偶性特性仍不明确。我们表明，与标准CMDP不同，峰值成本约束的MDP可能无法实现零对偶性缺口。我们还进一步考虑一种稳健的表述，以应对模拟器与现实世界之间转移动力学中的不匹配问题。为解决此问题，我们开发了替代优化框架和基于积分概率指标的稳健价值估计方法。我们证明，在适当的超参数选择下，替代解在违反约束值最多ε的情况下，获得与原问题相同的稳健奖励值。实验表明，所提方法在动态扰动下有效执行安全，同时保持强有力的奖励表现。

From Black Box to Executable Logic: Explainable Reinforcement Learning through Prolog Expert Systems

从黑匣子到可执行逻辑：通过Prolog专家系统实现的可解释强化学习

• Authors: Eduardo C. Garrido-Merchán
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.15459
• Pdf link: https://arxiv.org/pdf/2607.15459
• Abstract A trained deep reinforcement learning policy is a black box, and we ask whether it can be made explainable by rewriting it as an executable logic program that reproduces its behaviour and that a person can read, a logic engine can run, and an optimizer can edit. We present a three-stage post-hoc transformation that extracts a frozen proximal policy optimization teacher, induces an ordered rule list from its decisions in the manner of classical relational learning, and emits the result as a Prolog program whose every decision is executed by an off-the-shelf logic engine; a subsequent expansion stage edits the rule base and accepts an edit only when policy evaluation certifies a return increase. We prove four guarantees. A return-loss bound makes the distilled program a machine-checkable certificate in a finite Markov decision process, and the expansion loop improves monotonically and terminates. For the continuous-observation setting we answer whether the conversion is possible at all: the propositional threshold instantiation converts the network to arbitrary fidelity as the resolution B grows, with disagreement O(1/B) and a return gap that closes at the same rate, and a matching lower bound shows the cost is exponential in the observation dimension for an oblique decision boundary. Empirically, on a two-room key-and-door task with 16,944 reachable states the expanded Prolog program attains exact optimal return in every seed and, in a budget-capped regime, exceeds the stochastic teacher on exact return in ten of ten seeds. On three continuous-control tasks the emitted program substitutes the network, matching the neural teacher within noise on Acrobot with eleven clauses and recovering about 97% of its return on CartPole, while on the finer-control LunarLander it recovers only partially, exactly the ceiling the exponential lower bound predicts.
• 中文摘要 经过训练的深度强化学习策略是一个黑箱，我们想知道它是否可以通过将其重写为可执行的逻辑程序来解释，该程序能重现其行为，且人能阅读，逻辑引擎能运行，优化器能编辑。我们提出一个三阶段的后自解变换，提取一个冻结的近距离策略优化教师，从其决策中以经典关系学习方式导出有序规则列表，并将结果输出为一个Prolog程序，其每个决策由现成逻辑引擎执行;后续扩展阶段编辑规则库，仅在策略评估认证回报增加时接受编辑。我们证明了四个保证。回归损耗界限使得提炼程序成为有限马尔可夫判定过程中机器可检验的证书，展开环路单调地改善并终止。对于连续观测设置，我们回答转换是否可能：命题阈值实例化将网络转换为任意保真度，随着分辨率B增长，存在O（1/B）不一致且返回间隙以相同速率缩小，匹配的下界显示斜决策边界的观测维度成本呈指数。在一个拥有16,944个可达状态的两间房钥匙和门任务中，扩展后的Prolog程序在每个种子中都实现了精确的最优回报，并且在预算限制的环境中，十种子中的十种子的精确回报率超过了随机教师。在三个连续控制任务中，发射程序替代网络，在Acrobot上用十一个子句匹配噪声中的神经教师，并在CartPole上恢复约97%的回报;而在更精细控制的LunarLander上，它仅部分恢复，正好达到指数下界预测的上限。

Process Reward Informed Tree Rollout for Effective Multi-Turn RL

流程奖励知情树推广以实现有效的多回合强化学习

• Authors: Xintong Li, Sha Li, Yuwei Zhang, Changlong Yu, Rongmei Lin, Hongye Jin, Shuyi Guan, Xin Liu, Linwei Li, Qingyu Yin, Jingbo Shang
• Subjects: Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15610
• Pdf link: https://arxiv.org/pdf/2607.15610
• Abstract Reinforcement learning (RL) has become a key approach for training LLM agents, yet popular methods such as GRPO/RLOO rely on multiple independently sampled complete trajectories for advantage estimation. In long-horizon agentic tasks, such a uniform rollout strategy can waste budget on uninformative dead-end attempts, while promising intermediate states do not receive sufficient exploration. The multi-turn structure of agentic trajectories, with interleaved actions and observations, naturally supports organizing a trajectory group as a tree, where each turn serves as a decision point for exploration. This perspective reframes effective exploration as the problem of deciding where to branch. We propose Process-Scorer Guided Adaptive Tree Rollout (PATR), a quality-aware rollout framework for multi-turn agent RL. PATR uses task-appropriate process feedback to score partial trajectories, selectively branches from promising states, reuses shared prefixes, and conservatively stops degenerate paths to reduce wasted sampling. The resulting rollout groups remain compatible with standard policy optimization while providing more efficient exploration under the same training budget. We evaluate PATR on FrozenLake and the challenging SWE-Bench, which is largely unexplored by prior tree-rollout agent RL methods. Experiments show that PATR improves performance by up to +5.0 points on SWE-Bench and +9.3 points on FrozenLake, highlighting process-guided tree rollouts as an effective strategy for scalable multi-turn RL.
• 中文摘要 强化学习（RL）已成为训练LLM代理的关键方法，但如GRPO/RLOO等流行方法依赖多个独立采样的完整轨迹来估算优势。在长期代理任务中，这种统一的推广策略可能会浪费预算在无意义的死胡同尝试上，而有潜力的中间状态则未获得足够的探索。多回合的能动轨迹结构，结合交错的动作和观察，自然支持将轨迹组组织为树状结构，每个转向作为探索的决策点。这种观点将有效探索重新定义为决定从哪里分支的问题。我们提出了过程评分器引导自适应树展开（PATR），这是一种针对多回合代理强化学习的质量感知推广框架。PATR利用任务相关的过程反馈对部分轨迹进行评分，选择性地从有前景状态分支，重用共享前缀，并保守地停止简并路径以减少采样浪费。由此产生的推广组既兼容标准策略优化，又能在相同培训预算下提供更高效的探索。我们在FrozenLake和复杂的SWE-Bench上评估了PATR技术，这在以往树展开剂强化学习方法中较为未被探索。实验显示，PATR在SWE-Bench上性能提升最高为+5.0点，在FrozenLake上提升+9.3点，凸显了进程引导树展开作为可扩展多回合强化学习的有效策略。

Difference-Based Relational Learning for Zero-Shot Object-Goal Visual Navigation With Direct Sim-to-Real Transfer

基于差分的关系学习用于零射程目标视觉导航，实现直接模拟到现实的传输

• Authors: Guolei Qi, Feitian Zhang
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.15642
• Pdf link: https://arxiv.org/pdf/2607.15642
• Abstract End-to-end deep reinforcement learning (DRL) for zero-shot object-goal visual navigation remains challenged by the sim-to-real gap, particularly variations in object appearance and restricted camera field-of-view (FoV). This letter proposes a Temporal Difference-Relational Network (T-DRN) for robust zero-shot sim-to-real transfer. T-DRN combines a Siamese difference-based feature extractor, which computes relational difference between the target and observed objects to produce domain-independent representations, with a dual-frame temporal buffer that preserves short-term object continuity under narrow FoV. Extensive experiments in AI2-THOR demonstrate that T-DRN improves zero-shot generalization in terms of success rates over strong baselines. Furthermore, T-DRN is systematically validated on a physical wheeled robot, demonstrating robust performance under real sensing and actuation constraints and supporting the feasibility of direct sim-to-real transfer.
• 中文摘要 零射击目标视觉导航的端到端深度强化学习（DRL）仍面临模拟与现实差距的挑战，尤其是物体外观的变化和摄像机视野（FoV）受限。本信提出了一种时间差分关系网络（T-DRN）用于稳健的零样本模拟到现实传输。T-DRN结合了基于暹罗差分的特征提取器，该工具计算目标与观测对象之间的关系差，生成域无关的表示，以及双帧时间缓冲器，在狭窄视野下保持短期物体连续性。AI2-THOR的大量实验表明，T-DRN在零次推广的成功率方面提升了强基线的成功率。此外，T-DRN在物理轮式机器人上系统验证，在实感和驱动约束下展现了稳健性能，支持直接模拟到实传输的可行性。

PCTD: Preference-Guided Counterfactual Task Decomposition for Agent Tool Retrieval

PCTD：代理工具检索中的偏好引导反事实任务分解

• Authors: Chu Zhao, Lei Tang, Minghang Li, Jianzhe Zhao, Guibing Guo, Zhengzong Chen, Yuanyuan Zhao, Fei Huang
• Subjects: Subjects: Information Retrieval (cs.IR)
• Arxiv link: https://arxiv.org/abs/2607.15696
• Pdf link: https://arxiv.org/pdf/2607.15696
• Abstract Task decomposition aims to transform ambiguous instructions into executable atomic subtasks, thereby guiding high-precision tool retrieval. However, our analysis reveals that directly adopting tool retrieval metrics, i.e., Recall or NDCG, as rewards for task decomposition can easily induce reward hacking in reinforcement learning-based methods. Specifically, models tend to maximize retrieval matching through strategies such as repetitive decomposition. This spurious correlation between the shallow features of decomposition results and retrieval metric impairs generalization in Out-of-Domain (OOD) scenarios involving unseen tools. To address this issue, we propose PCTD, a Preference-guided Counterfactual Task Decomposition framework. PCTD quantifies the marginal causal gain of decomposition on retrieval ranking through a counterfactual reward, thereby cutting off spurious correlations at their source. Meanwhile, it introduces a preference reward to impose fine-grained structural supervision on logical coherence and atomicity, encouraging the model to generate high-quality decompositions. In addition, we construct MTDTool, the task decomposition benchmark specifically designed for mobile multi-turn interactions. Extensive experiments demonstrate that PCTD alleviates repetitive decomposition and surpasses SOTA methods in retrieval, decomposition quality, and OOD generalization.
• 中文摘要 任务分解旨在将模糊指令转化为可执行的原子子任务，从而指导高精度工具检索。然而，我们的分析显示，直接采用工具检索指标，即回忆或NDCG，作为任务分解的奖励，可以很容易在基于强化学习的方法中诱导奖励黑客行为。具体来说，模型倾向于通过重复分解等策略最大化检索匹配。分解结果的浅层特征与检索度量之间的这种虚假相关性，损害了涉及未见工具的域外（OOD）场景中的泛化性。为解决这一问题，我们提出了PCTD，一种偏好引导的反事实任务分解框架。PCTD通过反事实奖励量化反事实奖励中分解在排名中的边际因果增益，从而切断源自虚假相关性。同时，它引入了偏好奖励，以对逻辑相干性和原子性施加细粒度结构监督，鼓励模型生成高质量的分解。此外，我们还构建了 MTDTool，这是专为移动多回合交互设计的任务分解基准测试。大量实验表明，PCTD在反复分解、分解质量和OOD泛化方面优于SOTA方法。

GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking

GoStop：事件型特征追踪中自适应时间聚合的强化学习

• Authors: Youngho Kim, Hoonhee Cho, Jae-Young Kang, Kuk-Jin Yoon
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.15699
• Pdf link: https://arxiv.org/pdf/2607.15699
• Abstract Feature tracking plays a fundamental role in understanding scene motion and supports various downstream tasks. Event cameras, with their high temporal resolution and asynchronous sensing, enable low-latency and motion-robust perception, making them well-suited for feature tracking under fast and non-linear motion. However, existing event-based feature tracking methods rely on fixed heuristic rules based on hand-tuning for event accumulation. Such strategies fail to adapt to diverse motion dynamics, leading to degraded performance under abrupt motion changes or low-motion scenarios. In this paper, we model event accumulation as a sequential decision-making problem and introduce reinforcement learning (RL) framework to adaptively control the accumulation process for online event-based feature tracking. Our approach trains a RL agent that decides whether to continue accumulating events or to perform tracking inference based on motion cues. The proposed adaptive temporal agent enables dynamic adaptation to varying motion patterns without relying on hand-crafted rules. Furthermore, we introduce a Dynamic Event-based Tracking (DEFT) dataset with dynamic motion distributions to evaluate the robustness of the feature tracking. Extensive experiments demonstrate that integrating our plug-and-play framework to existing feature tracking methods consistently outperforms heuristic-based approaches, improving robustness under dynamic motion while offering a better balance between tracking accuracy and efficiency. Our project codes and datasets are available at this https URL
• 中文摘要 特征跟踪在理解场景运动中起着基础作用，并支持各种下游任务。事件相机凭借其高时间分辨率和异步传感，实现低延迟和运动稳健的感知，非常适合在快速且非线性运动下的特征跟踪。然而，现有的基于事件的特征跟踪方法依赖于基于手动调优的固定启发式规则来进行事件累积。此类策略无法适应多样的运动动态，导致在突变运动或低运动场景下性能下降。本文将事件累积建模为顺序决策问题，并引入强化学习（RL）框架，以自适应控制在线事件型特征追踪的积累过程。我们的方法训练一个强化学习代理，决定是继续累积事件，还是基于运动线索进行跟踪推断。所提出的自适应时间代理能够动态适应变化的运动模式，而无需依赖手工规则。此外，我们引入了动态事件跟踪（DEFT）数据集，利用动态运动分布来评估特征追踪的稳健性。大量实验表明，将我们的即插即用框架与现有特征跟踪方法整合，能够持续优于基于启发式的方法，提升动态运动下的鲁棒性，同时在跟踪准确性与效率之间实现更好的平衡。我们的项目代码和数据集可在此 https URL 获取

RAVEN: Reinforcement-Adaptive Visibility-Graph Planning for Robust Humanoid Navigation with Collision-Free MPC

RAVEN：强化-自适应可视化-图规划，实现无碰撞MPC的稳健类人形导航

• Authors: Ruochen Hou, Shiqi Wang, Beom Jun Kim, Hanzhang Fang, Mehak Singal, Dennis W. Hong
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.15701
• Pdf link: https://arxiv.org/pdf/2607.15701
• Abstract Humanoid navigation in dynamic environments requires long-horizon planning while respecting short-horizon dynamic and safety constraints. Classical visibility-graph planners combined with model predictive control (MPC) can efficiently generate collision-free trajectories, but their performance depends on manually tuned parameters and accurate system modeling. In real robotic systems, control delays, state-estimation noise, and locomotion uncertainties can cause overshoot and constraint violations even when the nominal path is geometrically optimal. We propose RAVEN, a hierarchical reinforcement learning (RL)-MPC framework for robust humanoid navigation. Unlike prior approaches that use learning to tune cost weights or replace planning entirely, RAVEN employs RL to adapt the geometric construction of a visibility-graph planner by modifying obstacle inflation and related graph parameters. By directly reshaping the free-space geometry, the learned planner alters the topology of the global path to compensate for delay and tracking imperfections. A collision-free MPC layer then tracks the planned trajectory while explicitly enforcing velocity bounds and obstacle-avoidance constraints. By training under realistic delays and observation noise, RAVEN learns planning adaptations that improve robustness while retaining explicit long-horizon geometric planning and constrained optimization, in contrast to end-to-end learning approaches. We evaluate RAVEN against a manually tuned visibility-graph MPC baseline and a pure RL navigation policy. Results demonstrate reduced overshoot near obstacles, improved robustness in narrow passages, and more reliable navigation under delay and noise. These findings indicate that reinforcement-adaptive graph construction combined with constrained MPC provides an effective and interpretable alternative to end-to-end learning for robust humanoid navigation.
• 中文摘要 动态环境中的人形导航需要长视野规划，同时尊重短视野的动态和安全约束。经典的可视图规划器结合模型预测控制（MPC）可以高效生成无碰撞轨迹，但其性能依赖于手动调校参数和准确的系统建模。在真实机器人系统中，控制延迟、状态估计噪声和运动不确定性可能导致超跃和约束违规，即使标称路径在几何上最优。我们提出了RAVEN，一种用于稳健类人导航的分层强化学习（RL）-MPC框架。与以往通过学习调整成本权重或完全取代规划的方法不同，RAVEN 采用强化学习（RL）来调整可视图规划器的几何结构，通过修改障碍物膨胀及相关图参数。通过直接重塑自由空间几何，所学规划者改变全局路径的拓扑结构，以补偿延迟和跟踪缺陷。无碰撞MPC层随后跟踪计划轨迹，同时明确执行速度界限和避障约束。通过在现实延迟和观测噪声下训练，RAVEN学习规划调整，提升鲁棒性，同时保留显式长视野几何规划和受限优化，这与端到端学习方法形成对比。我们将RAVEN结合手动调优的可见性图MPC基线和纯强化导航策略进行评估。结果显示，在障碍物附近减少了超冲，在狭窄通道中更稳健，在延迟和噪音下导航更可靠。这些发现表明，强化自适应图构建结合受限MPC为稳健类人生物导航提供了一种有效且可解释的端到端学习替代方案。

Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling

通过隐性文化对齐奖励建模去偏见文本到图像评估

• Authors: Bo-An Chang, Yu-Chih Chen
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Multimedia (cs.MM)
• Arxiv link: https://arxiv.org/abs/2607.15740
• Pdf link: https://arxiv.org/pdf/2607.15740
• Abstract As Text-to-Image (T2I) systems rapidly advance, evaluating the cultural authenticity of synthesized content has become increasingly important for fair and trustworthy generative AI. Existing T2I evaluation metrics and multimodal judges often rely on visual-semantic representations that underrepresent implicit cultural norms, leading to biased preference judgments and the omission of fine-grained cultural cues. In addition, visual question answering (VQA)-based evaluators typically depend on autoregressive text generation, which limits their scalability for real-time reward modeling. To address these limitations, we introduce an Implicit Cultural Alignment Reward Model built upon a lightweight 4.2-billion-parameter Multimodal Large Language Model (MLLM). Our framework integrates an Implicit Cultural Probe with a Skip-connection Cross-Attention (SkipCA) mechanism, enabling late-stage semantic features to directly attend to early-stage visual representations and better preserve culturally salient details. Evaluations on 3,323 challenging and carefully curated image pairs from the CulturalFrames benchmark show that our approach achieves 80.54% pairwise accuracy, with Pearson and Kendall correlation coefficients of 0.546 and 0.377, respectively, outperforming representative vision-language metrics and MLLM-based evaluators. Moreover, by bypassing autoregressive text generation, our model processes each evaluation in 0.21 seconds under our local inference setup, achieving a $10\times$ speedup over standard VQA-based evaluators. These results suggest that the proposed reward model can provide an efficient and culturally aware scalar signal for preference optimization pipelines such as Reinforcement Learning from Human Feedback and Direct Preference Optimization.
• 中文摘要 随着文本转图像（T2I）系统的快速发展，评估合成内容的文化真实性对于公平且值得信赖的生成式人工智能变得越来越重要。现有的T2I评估指标和多模态法官常依赖视觉语义表征，低估了隐含的文化规范，导致偏好判断偏颇，且遗漏了细致的文化线索。此外，基于视觉问答（VQA）的评估器通常依赖自回归文本生成，这限制了其在实时奖励建模中的可扩展性。为解决这些局限性，我们引入了一个基于轻量级42亿参数多模态大型语言模型（MLLM）的隐性文化对齐奖励模型。我们的框架将隐性文化探针与跳过连接交叉注意力（SkipCA）机制集成，使晚期语义特征能够直接关注早期阶段的视觉表现，更好地保存具有文化显著性的细节。对CulturalFrames基准中3,323对具有挑战性且精心策划的图像对进行了评估，我们的方法实现了80.54%的成对准确率，Pearson和Kendall相关系数分别为0.546和0.377，优于代表性的视觉语言指标和基于MLLM的评估者。此外，通过绕过自回归文本生成，我们的模型在局部推断设置下每次评估处理时间仅为0.21秒，比基于VQA的标准评估器快了10美元/时间。这些结果表明，所提的奖励模型能够为偏好优化流程（如从人类反馈强化学习和直接偏好优化）提供高效且具文化意识的标量信号。

Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding

模块化动态粒度视频LLM用于多事件长视频理解

• Authors: Wei Feng, Xin Wang, Yu-Wei Zhan, Yuwei Zhou, Wenwu Zhu
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.15778
• Pdf link: https://arxiv.org/pdf/2607.15778
• Abstract Video Large Language Models (Video LLMs) have made significant advancements in various video understanding tasks. However, long-video scenarios remain challenging due to the tension between limited visual token budgets and the need to capture multiple key events. Existing approaches typically process long videos in two stages, i.e., i) select keyframes and ii) perform detailed perception, which exhibit limitations: they lack a modular mechanism for adaptive capacity allocation and self-correction, resulting in unreliable modeling. To tackle these challenges, we propose MoD-VLLM, a novel Modularized Dynamic-Granularity Video LLM framework for multi-event long video understanding, which unifies temporal grounding and semantic understanding iteratively and self-reflectively. Specifically, we propose a Positive-Negative Video Segments Grounding module and a Modularized Dynamic-Granularity Reflection module, which form a closed loop to progressively localize the question-related video segments. The grounding module instructs a Video LLM to distinguish relevant from irrelevant video segments based on the video question. The reflection module employs a modularized scheduler that dynamically selects fine-grained encoding for relevant positive segments to capture detailed perception and coarse-grained encoding for negative segments to maintain global context. We further propose a dynamic-granularity reinforcement learning strategy, allowing MoD-VLLM to learn optimal grounding policies and dynamic granularity visual representation jointly. Moreover, we propose MEventBench, a challenging Multi-Event Long Video Benchmark for complex long video reasoning. Extensive experiments on several long video understanding benchmarks and our MEventBench demonstrate that MoD-VLLM significantly outperforms state-of-the-art baselines.
• 中文摘要 视频大型语言模型（Video LLM）在各种视频理解任务方面取得了显著进展。然而，由于有限的视觉代币预算和需要捕捉多个关键事件之间的紧张关系，长视频场景依然具有挑战性。现有方法通常将长视频分两个阶段处理，即 i） 选择关键帧 ii） 执行详细感知，但存在局限性：缺乏模块化的自适应容量分配和自我修正机制，导致建模不可靠。为应对这些挑战，我们提出了MoD-VLLM，一种新型模块化动态粒度视频LLM框架，用于多事件长视频理解，通过迭代和自我反思统一时间基础与语义理解。具体来说，我们提出了正负视频片段接地模块和模块化动态粒度反射模块，形成闭环，逐步定位问题相关的视频片段。接地模块指示视频大型语言模型根据视频问题区分相关和无关视频片段。该反射模块采用模块化调度器，动态选择细粒度编码以捕捉详细感知，并对负段选择粗粒度编码以保持全局上下文。我们还提出了一种动态粒度强化学习策略，使MoD-VLLM能够共同学习最优的基础策略和动态粒度的视觉表示。此外，我们提出了MEventBench，这是一个针对复杂长视频推理的多事件长视频基准测试。在多个长视频理解基准测试和我们的MEventBench上进行了大量实验，表明MoD-VLLM的表现显著优于最先进的基线。

QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

QUAD：通过跨双侧的质量化-错误对齐稳定NVFP4强化学习，实现MoE中的NVFP4强化学习

• Authors: Zhengyang Zhuge, Hao Yu, Xin Wang, Zheng Li, Yizhong Cao, Dayiheng Liu, Jianwei Zhang
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15810
• Pdf link: https://arxiv.org/pdf/2607.15810
• Abstract Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8. As an emerging low-precision format, NVFP4 combines fine-grained scaling for accuracy preservation with native W4A4 FP4 GEMMs for higher throughput than FP8. However, we find that directly applying NVFP4 to MoE RL rollout is impractical. NVFP4 rollout with BF16 training collapses after roughly 150 steps, accompanied by rapidly growing rollout-trainer log-probability gaps. Through training-inference error analysis and controlled ablations, we identify activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization-dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid. Therefore, to stabilize NVFP4 RL for MoE, we propose QUantization-error Alignment across Dual Sides (QUADS). On the trainer side, we introduce Asymmetric Quantization-Aware Training fake-quantizing weights while keeping activations unquantized for better alignment. On the rollout side, Residual Activation Compensation corrects high-error activation channels while preserving native W4A4 GEMMs. In our MoE RL experiments on several benchmarks, QUADS achieves BF16-level accuracy, improves average pass@1 by 21.49 points over naive NVFP4 RL, and delivers ~16% higher rollout throughput than FP8.
• 中文摘要 部署生成是强化学习（RL）中专家混合（MoE）大型语言模型的主要瓶颈，促使如FP8等低精度推广加速。作为一种新兴的低精度格式，NVFP4 结合了细粒度缩放以保持准确性，同时采用原生的 W4A4 FP4 GEMM，以实现比 FP8 更高的吞吐量。然而，我们发现直接将NVFP4应用到MoE强化学习的推广中并不切实际。NVFP4与BF16训练的推广在大约150步后崩溃，伴随着推广与训练器对数概率差距迅速扩大。通过训练-推断误差分析和受控消融，我们识别FP4强化不稳定性的主要来源是激活误差，而非权重误差：权重可以通过共享的量化-去量化路径同步和对齐，而激活则在线重新计算，误差则被粗糙的E2M1网格放大。因此，为了稳定 MoE 的 NVFP4 RL，我们提出了跨双侧的量化误差比对（QUAD）。在训练者方面，我们引入了非对称量化感知训练（ASA）假量化权重，同时保持激活未量化以实现更好的对齐。在推广端，残差激活补偿在保留原生W4A4 GEMM的同时，修正了高误差激活通道。在我们对多个基准测试的 MoE RL 实验中，QUADS 实现了 BF16 级别的精度，平均 pass@1 比原始 NVFP4 强化学习提升了 21.49 个百分点，并且比 FP8 高出 ~ 16% 的推广吞吐量。

Scalable Supervisory HVAC Control for Linear Objectives

线性物镜的可扩展监督暖通空调控制

• Authors: W. Grant Dierking, Arash J. Khabbazi, Levi D. Reyes Premer, Kevin J. Kircher
• Subjects: Subjects: Systems and Control (eess.SY)
• Arxiv link: https://arxiv.org/abs/2607.15867
• Pdf link: https://arxiv.org/pdf/2607.15867
• Abstract Advanced control of heating and cooling systems can substantially reduce energy costs and pollution. However, real-world adoption of popular algorithms among researchers, such as model predictive control (MPC) and reinforcement learning (RL), remains limited due in part to their high deployment and commissioning costs. Here, we develop two nearly commissioning-free controllers tailored to objectives that depend linearly on the controlled thermal load, such as energy costs and pollution. The controllers require at most two thermal parameters. In representative heating simulations, controller performance is robust to large parameter specification errors, suggesting potential for deployment with no tuning. The controllers maintain good occupant comfort while achieving 43 to 98% (depending on the electricity pricing and controller variant) of the performance improvement achieved by an omniscient policy with perfect model information and forecasts. These results suggest that simple, structure-exploiting controllers may capture most of the attainable value of advanced control while avoiding the data, modeling, tuning, and computational burdens that can arise with conventional MPC or RL.
• 中文摘要 先进控制供暖和制冷系统可以大幅降低能源成本和污染。然而，模型预测控制（MPC）和强化学习（RL）等流行算法在实际中被研究人员广泛采用，部分原因是其部署和调试成本高昂。在这里，我们开发了两款几乎无需调试的控制器，针对线性依赖于受控热负荷的目标，如能源成本和污染。控制器最多需要两个热参数。在代表性的加热模拟中，控制器性能对较大的参数规格误差具有鲁棒性，表明无需调优即可部署。控制器在实现全知政策、完美模型信息和预测后，能保持良好的乘员舒适度，同时实现43%至98%的性能提升（视电价和控制器版本而定）。这些结果表明，简单的结构利用控制器可能捕捉高级控制的大部分可达价值，同时避免传统MPC或强化学习可能带来的数据、建模、调优和计算负担。

DSWorld: A Data Science World Model for Efficient Autonomous Agents

DSWorld：高效自主智能体的数据科学世界模型

• Authors: Zherui Yang, Fan Liu, Hao Liu
• Subjects: Subjects: Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.15901
• Pdf link: https://arxiv.org/pdf/2607.15901
• Abstract Despite strong capabilities in data understanding and decision-making, autonomous data science agents still heavily rely on trial-and-error workflows that involve expensive computation. This bottleneck motivates models that can anticipate the effects of data science operations before real execution. In this paper, we introduce the concept of Data Science World Model, which model the data science execution environment by predicting environment state transitions conditioned on current workflow states and candidate operations. We further propose DSWorld, a practical framework that combines structured state construction, cost-aware routing, lightweight real execution, and an LLM-based simulator for expensive operations. To support training, we construct an 8K-scale transition trajectory dataset and introduce Reflective World Model Optimization, an error-aware reinforcement learning strategy for improving transition prediction. Experiments show that DSWorld accelerates RL-based agent training by approximately $14\times$ and search-based inference by approximately $3$-$6\times$ while maintaining competitive performance, and outperforms the strongest LLM baseline by 35.6% on transition prediction tasks. The code is available at this https URL.
• 中文摘要 尽管在数据理解和决策方面具备强大能力，自主数据科学代理仍高度依赖涉及昂贵计算的试错式工作流程。这一瓶颈激励了能够在实际执行前预见数据科学操作影响的模型。本文介绍了数据科学世界模型的概念，该模型通过预测基于当前工作流状态和候选操作的环境状态转换来建模数据科学执行环境。我们还提出了DSWorld，这是一个实用框架，结合了结构化状态构建、成本感知路由、轻量级实执行和基于LLM的模拟器，用于高成本操作。为支持训练，我们构建了一个8K尺度的过渡轨迹数据集，并引入了反射世界模型优化，这是一种针对误差的强化学习策略，用于提升过渡预测。实验显示，DSWorld在保持竞争力的同时，将基于RL的代理训练加速约14美元，搜索推断速度约提升3美元至6倍，且在转换预测任务中比最强LLM基线快35.6%。代码可在该 https URL 访问。

Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark

通过强化学习学习实现距离-避免任务：矢量化模拟与基准测试

• Authors: Jonas Weihing, Shahram Eivazi
• Subjects: Subjects: Robotics (cs.RO); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15935
• Pdf link: https://arxiv.org/pdf/2607.15935
• Abstract Deep reinforcement learning (DRL) has a longstanding tradition in addressing the reach-avoid task problem, especially for controlling robotic arms. While this task serves as a baseline environment within the research community, the ability of DRL to effectively learn the each-avoid task in complex and realistic scenarios beyond simplified and restricted tabletop settings remains uncertain. In this paper, we present, for the first time, a comprehensive benchmark for the reachavoid task that accurately captures real-world complexities without simplifications. We demonstrate a diverse range of settings for robotic arm reach-avoid task, which can be used for evaluating DRL research. We achieved this by utilizing the MuJoCo MJX physics engine and parallelizing both the simulation environment and DRL algorithms using the Brax library. We achieved state-of-the-art results with success rates of 96.1% (UR5e) and 98.8% (Franka Emika Robot) for the reach task and 86.8% (UR5e) and 95.2% (Franka) for the static reachavoid task. Our results indicate that while in previous works DRL agents could solve, for example, a reach task in a simplified setting perfectly, their agents performance collapses when evaluated in realistic scenarios. Overall, this work identifies that additional research is still required to claim the successful resolution of the robotic arm reach-avoid task using DRL. The environment and benchmarking code is available as open source at the following link
• 中文摘要 深度强化学习（DRL）在解决伸手-回避任务问题方面有着悠久的传统，尤其是在控制机械臂方面。虽然该任务作为研究社区的基线环境，但DRL在复杂且现实的场景中有效学习每个避避任务的能力仍不确定，超出简化和限制的桌面环境。本文首次提出了一个全面的距离规避任务基准，准确捕捉现实世界的复杂性且不加简化。我们展示了机器人手臂伸缩回避任务的多种环境，可用于评估日程学习研究。我们通过使用 MuJoCo MJX 物理引擎，并用 Brax 库并行化仿真环境和 DRL 算法实现了这一点。我们取得了最先进的成绩，UR5e的成功率分别为96.1%和98.8%（Franka Emika Robot），静态的Reachavoid任务成功率分别为96.8%（UR5e）和95.2%（Franka）。我们的结果表明，在以往的研究中，DRL代理能够完美解决例如在简化环境中的距离任务，但在现实情景中评估时，其表现会崩溃。总体而言，这项工作表明，仍需更多研究才能成功利用日日学习（DRL）解决机器人手臂伸缩回避任务。环境和基准测试代码可通过以下链接开源获取

More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe

少用更多：一个用简单配方打造的大型遥感VLM

• Authors: Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Luc Van Gool, Danda Pani Paudel (INSAIT, Sofia University "St. Kliment Ohridski")
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.15942
• Pdf link: https://arxiv.org/pdf/2607.15942
• Abstract Remote sensing vision-language models are increasingly expected to support open-ended reasoning over Earth Observation data and a variety of tasks. Most recent progress in this area has been driven by remote-sensing-specific architectural designs, often introducing new encoders, alignment modules, or task-specific fusion mechanisms. In this work, we challenge the necessity of such architectural specialization. We show that a generally capable vision-language model can achieve competitive or state-of-the-art performance at challenging remote sensing benchmarks, provided that it is trained at sufficient scale across diverse data and tasks. Our model uses a single language policy that can either answer directly in text or invoke a localization tool for segmentation and grounding. To train this heterogeneous behaviour, we employ a multi-task reinforcement learning framework with adaptive task rewards covering multiple-choice VQA, free-form VQA, captioning, detection, and segmentation across a large variety of input types. Our approach achieves competitive results across a broad set of benchmarks, including high-resolution, multi-temporal, multi-modal and multi-view tasks. Further, as training data scales, our experiments show consistent improvements across most tasks both in and out of distribution, which correlate with per-task data diversity. These findings suggest that, for remote sensing VLMs, data scale is more important than architectural novelty.
• 中文摘要 遥感视觉语言模型越来越多地支持对地球观测数据和各种任务进行开放式推理。该领域的最新进展主要由遥感专用架构设计推动，常常引入新的编码器、对准模块或任务专用的聚变机制。在本研究中，我们挑战了建筑专业化的必要性。我们证明，只要在多样化数据和任务中进行足够规模的训练，一个普遍具备考验能力的视觉语言模型，可以在具有挑战性的遥感基准中实现竞争或最先进的性能。我们的模型使用单一语言策略，可以直接用文本回答，也可以调用本地化工具进行分割和基础化。为了训练这种异质行为，我们采用了多任务强化学习框架，并配备自适应任务奖励，涵盖多项选择VQA、自由形式VQA、字幕、检测和分割，涵盖多种输入类型。我们的方法在包括高分辨率、多时空、多模态和多视图任务在内的广泛基准测试中取得了竞争性成果。此外，随着训练数据的扩展，我们的实验显示，无论分布内外，大多数任务均有持续的改进，这与各任务数据多样性相关。这些发现表明，对于遥感VLM，数据规模比架构新颖性更为重要。

Data and Learning Where it Matters for Contact-Rich Manipulation

数据与学习对丰富接触操作至关重要

• Authors: Oliver Hausdörfer, Linus Schwarz, Gabor Marko, Christian Dietz, Timo Class, Luka Hofer, Jim Yun-Jin Li, Johannes Hechtl, Ralf Römer, Angela P. Schoellig
• Subjects: Subjects: Robotics (cs.RO)
• Arxiv link: https://arxiv.org/abs/2607.15982
• Pdf link: https://arxiv.org/pdf/2607.15982
• Abstract Learned policies trained end-to-end on large datasets often remain brittle in high-precision tasks and struggle with generalization. We find that these limitations largely stem from a lack of structure and focus in data collection. Our key insight is to leverage dense data collection only for the critical segment of contact-rich tasks and to rely on traditional planning during simple free-space motion. We propose an automated data-collection scheme in combination with offline deep reinforcement learning for the critical segment of the task, eliminating reliance on a teleoperator's skill and on online policy updates. Across four challenging real-world tasks, using only 2 to 2.5 hours of autonomous data collection, we achieve an average success rate of 96%, compared to the strongest baseline at 55%. Notably, performance remains high in out-of-distribution scenarios where end-to-end approaches struggle. Our results pave the way for targeted data collection for contact-rich tasks and for high success rates in precision applications.
• 中文摘要 在大型数据集上端到端训练的策略在高精度任务中往往仍然脆弱，难以泛化。我们发现这些局限主要源于数据收集缺乏结构和聚焦。我们的关键见解是，仅在关键的接触密集任务中利用密集数据收集，并在简单的自由空间运动中依赖传统规划。我们提出了一种自动化数据收集方案，结合离线深度强化学习，应用于关键任务部分，消除对远程操作员技能和在线政策更新的依赖。在四个具有挑战性的现实任务中，仅需2到2.5小时的自主数据收集，我们平均成功率为96%，而最强的基线为55%。值得注意的是，在端到端方法难以实现的非分销场景中，性能依然很高。我们的成果为针对接触丰富任务的有针对性数据收集和高精度应用的高成功率奠定了基础。

CLaC@FinMMEval 2026 Task 3: Sentiment-Augmented Deep Reinforcement Learning for Active Trading -- An Alpha-Reward Approach

CLaC@FinMMEval 2026 任务3：情感增强深度强化学习用于主动交易——一种阿尔法奖励方法

• Authors: Andrei Neagu, Eeham Khan, Leila Kosseim
• Subjects: Subjects: Machine Learning (cs.LG)
• Arxiv link: https://arxiv.org/abs/2607.16028
• Pdf link: https://arxiv.org/pdf/2607.16028
• Abstract This paper presents our system for Task 3 of the CLEF 2026 FinMMEval Lab, which requires daily long, flat, or short trading decisions for Bitcoin (BTC) and Tesla (TSLA) using news and historical market data. We formulate the problem as a discrete-action Markov Decision Process and compare four deep reinforcement learning algorithms: Policy Gradient (PG), Proximal Policy Optimization (PPO), Deep Q-Learning (DQL), and Deep Deterministic Policy Gradient (DDPG). The agents use technical indicators, cyclical calendar encodings, and daily news sentiment scores produced by LLaMA 3.2 1B. To reduce overfitting and align training with the objective of outperforming buy-and-hold, we introduce an alpha reward based on excess market return and randomize episode start dates. Hyperparameters are optimized with Ray Tune over 180 trials per algorithm-asset pair, with early stopping and model selection based on validation Sharpe ratio. On the CLEF Task 3 test set, DDPG achieves the strongest overall performance. DQL was selected a priori for the live endpoint because it obtained the highest validation Sharpe ratio, with selection performed without access to the test period. For TSLA, DDPG and DQL achieve cumulative returns of 54.96% and 52.62%, respectively, compared with 16.45% for buy-and-hold. For BTC, DDPG achieves a positive return of 1.58% while buy-and-hold declines by -34.27%. The results also reveal a substantial validation-to-test generalization gap, highlighting the difficulty of transferring policies selected in bull-market conditions to a bear-market regime.
• 中文摘要 本文介绍了我们为CLEF 2026 FinMMEval实验室第三任务所做的系统，该系统需要利用新闻和历史市场数据，对比特币（BTC）和特斯拉（TSLA）进行每日的多头、平仓或空头交易决策。我们将问题表述为离散动作马尔可夫决策过程，并比较了四种深度强化学习算法：策略梯度（PG）、近端策略优化（PPO）、深度Q学习（DQL）和深度确定性策略梯度（DDPG）。这些代理使用由LLaMA 3.2 1B生成的技术指标、周期性日历编码和每日新闻情绪评分。为了减少过拟合并使训练与超越买入持有的目标保持一致，我们引入了基于超额市场回报的阿尔法奖励，并随机安排了集数开始日期。超参数通过Ray Tune在每对算法-资产对中进行180次试验进行优化，早期停止和基于验证Sharpe比率的模型选择。在CLEF Task 3测试集中，DDPG整体表现最强。DQL被优先选为活终点，因其获得了最高的验证Sharpe比率，且选拔时未获得测试期间访问权限。对于TSLA，DDPG和DQL的累计回报分别为54.96%和52.62%，而买入持有的收益率为16.45%。对于比特币来说，DDPG实现了1.58%的正回报，而买入持有则下降了34.27%。结果还揭示了验证到检验的概括差距，凸显了将牛市条件下选择的政策转移到熊市体系的困难。

JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Models

JoyNexus：面向服务的多租户VLA模型后培训

• Authors: Haoran Sun, Wentao Zhang, Junyang Hua, Hedan Yang, Yongjian Guo, Yifei Zhang, Xiaolong Xiang, Mingxi Luo, Jing Long, Chen Zhao, Chen Zhou, Wanting Xu, Qiming Yang, Hui Zhang, Song Wang, Xiaodong Bai, Shuai Di, Xu Chu, Xiaotie Deng, Yicheng Gong, Junwu Xiong
• Subjects: Subjects: Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Software Engineering (cs.SE)
• Arxiv link: https://arxiv.org/abs/2607.16074
• Pdf link: https://arxiv.org/pdf/2607.16074
• Abstract The post-training of Vision-Language-Action (VLA) models is essential due to the diversity of simulators, robot embodiments, and task objectives. Existing compute services, whether offered as direct accelerator rental or batch-workload submission, typically allocate an exclusive set of GPU and CPU resources to a single tenant. While this paradigm maximizes client flexibility, it burdens users with infrastructure adaptation, and the fixed card-hour accounting model renders short or bursty workloads both expensive for tenants and inefficient for the service provider. To address these challenges, we present JoyNexus, a unified service for multi-tenant VLA supervised fine-tuning, reinforcement learning, and evaluation. JoyNexus decouples the Training Model Service, Inference Model Service, and Environment Service, each accessed through APIs and backed by resident shared base models with tenant-specific slots. Tenants can directly invoke high-level semantic APIs for training, rollout, and evaluation, or compose custom algorithms using lower-level APIs and their assigned endpoints. Multiple tenants submit workloads concurrently; their action modules, optimizers, rollout records, and policy versions remain isolated, and the service is scheduled by the global Training Queue and Inference Queue. To further improve multi-tenant training efficiency, JoyNexus introduces group batching for heterogeneous VLA data schemas that share a compatible model-facing prefix, enabling a single shared backbone forward pass over grouped samples. Finally, we evaluate JoyNexus through workload simulation and a group-batching pipeline in a realistic embodied scenario. Results show that, compared with isolated single-tenant execution, JoyNexus reduces aggregate GPU time and improves service utilization via cross-tenant scheduling on shared resources.
• 中文摘要 视觉-语言-动作（VLA）模型的后期训练至关重要，因为模拟器、机器人具象和任务目标的多样性。现有的计算服务，无论是直接租用加速器还是批量提交工作负载，通常都会将一组独占的GPU和CPU资源分配给单一租户。虽然这种模式最大化了客户灵活性，但也给用户带来了基础设施适应的负担，而固定的卡时计费模式使得短期或突发性工作负载既对租户来说既昂贵又对服务提供商效率低下。为应对这些挑战，我们推出了JoyNexus，一项统一的多租户VLA监督微调、强化学习和评估服务。JoyNexus 将训练模型服务、推理模型服务和环境服务解耦，这些服务都通过 API 访问，并由具有租户专用时隙的常驻共享基模型支持。租户可以直接调用高级语义 API，用于训练、推广和评估，或使用低级别 API 及其分配端点组合自定义算法。多个租户同时提交工作负载;其动作模块、优化器、推广记录和策略版本保持隔离，服务由全局训练队列和推理队列调度。为进一步提升多租户训练效率，JoyNexus 引入了针对异构 VLA 数据模式的组批处理，这些模式共享兼容的面向模型前缀，从而实现单一共享主干前向传递对分组样本的处理。最后，我们通过工作负载模拟和组批处理流程在现实的具体场景中评估了JoyNexus。结果显示，与孤立的单租户执行相比，JoyNexus 减少了总 GPU 时间，并通过共享资源的跨租户调度提升了服务利用率。

DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learning

DADiff：扩散驱动的跨域策略适配用于强化学习

• Authors: Hanyang Chen, Anirudh Satheesh, Longchao Da, Hua Wei
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
• Arxiv link: https://arxiv.org/abs/2607.16090
• Pdf link: https://arxiv.org/pdf/2607.16090
• Abstract Transferring policies across domains poses a vital challenge in reinforcement learning, due to the dynamics mismatch between the source and target domains. In this paper, we consider the setting of online dynamics adaptation, where policies are trained in the source domain with sufficient data, while only limited interactions with the target domain are allowed. There are a few existing works that address the dynamics mismatch by employing domain classifiers, value-guided data filtering, or representation learning. Instead, we study the domain adaptation problem from a generative modeling perspective. Specifically, we introduce DADiff, a diffusion-based framework that leverages the discrepancy between source and target domain generative trajectories in the generation process of the next state to estimate the dynamics mismatch. Both reward modification and data selection variants are developed to adapt the policy to the target domain. We also provide a theoretical analysis to show that the performance difference of a given policy between the two domains is bounded by the generative trajectory deviation. More discussions on the applicability of the variants and the connection between our theoretical analysis and the prior work are further provided. We conduct extensive experiments in environments with various shifts to validate the effectiveness of our method. The results demonstrate that our method provides superior performance compared to existing approaches, effectively addressing the dynamics mismatch. We provide the code of our method at this https URL
• 中文摘要 跨域转移策略是强化学习中的关键挑战，因为源域与目标域之间的动态不匹配。本文探讨了在线动态适应的设定，即在源域中训练策略并获得足够数据，而仅允许有限度地与目标域的交互。已有少数研究通过使用领域分类器、值导向数据过滤或表示学习来解决动态不匹配问题。相反，我们从生成建模的角度研究领域适应问题。具体来说，我们介绍了DADiff，这是一个基于扩散的框架，利用生成下一状态过程中源域与目标域生成轨迹的差异来估算动力学不匹配。同时开发了奖励修改和数据选择变体，以适应目标领域。我们还提供了理论分析，证明两域之间某一策略的性能差异受生成轨迹偏差限制。关于变体适用性及我们理论分析与既往工作之间的联系，书中还提供了更多讨论。我们在各种变化的环境中进行大量实验，以验证方法的有效性。结果表明，我们的方法相较现有方法表现更优，有效解决了动力学不匹配问题。我们在这个 https URL 上提供了方法的代码

Understanding Reasoning from Pretraining to Post-Training

从预培训到培训后理解推理

• Authors: Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, Pavel Izmailov
• Subjects: Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
• Arxiv link: https://arxiv.org/abs/2607.16097
• Pdf link: https://arxiv.org/pdf/2607.16097
• Abstract Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.
• 中文摘要 强化学习（RL）已成为改进大型语言模型（LLMs）处理复杂推理任务的核心，但强化学习的后期训练大多被孤立于前置训练之外进行研究。因此，有两个基本问题仍然悬而未决：（1）预训练选择（模型规模、数据）如何影响强化学习计算的回报，以及（2）强化学习到底对模型做了什么？这些问题在标准LLM环境中难以研究：预训练语料库庞大且不受控制，难以将行为归因于预训练与强化学习，且跨阶段的系统计算扫描成本高昂。为了应对这些挑战，我们将国际象棋作为一个受控的测试平台，研究从训练前到训练后整个流程的推理。我们遵循标准的大型语言模型训练流程，在人类国际象棋局中预训练从5M到1B参数的语言模型，对合成推理轨迹进行监督微调，并在具有可验证奖励的国际象棋谜题上运行强化学习。利用该框架，我们发现在给定的强化计算层级，强化后性能可从预训练损失良好预测，且强化学习奖励曲线的斜率随预训练代币大致线性改善。除了缩放，我们还发现，强化学习不仅仅是提升SFT策略：在简单谜题上，它放大了SFT策略已偏好的正确走法，而在困难谜题中，它揭示了SFT几乎缺失的正确走法。我们进一步测试研究结果是否能超越国际象棋，通过在数学领域文本上训练一个1B语言模型，结果显示相同的预测模式：更长的预训练检查点在强化学习后表现更高，且在强化学习下提升更快。总之，我们提供了预训练与强化学习接口的定量说明，并提供了一个受控测试平台，用于研究整个预训练到后培训流程中推理科学。

Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

物理增强强化学习用于动态系统实时最优控制

• Authors: Matteo Tomasetto, Nicolò Botteghi, Gabriele Bruni, Andrea Manzoni
• Subjects: Subjects: Machine Learning (cs.LG); Optimization and Control (math.OC)
• Arxiv link: https://arxiv.org/abs/2607.16177
• Pdf link: https://arxiv.org/pdf/2607.16177
• Abstract Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interaction with the environment to synthesize optimal control strategies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces. In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentibility of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities. Through two challenging parametric navigation problems in unsteady flows, we show that PEARL (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms, (ii) is sample efficient, thanks to the physics-guided policy learning, (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems, and (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.
• 中文摘要 强化学习（RL）最近被作为一种有前景的反馈控制策略，应用于非线性和复杂动力系统。然而，强化学习算法采样效率低，且需要大量与环境交互以综合最优控制策略。因此，由于高维空间中探索与利用困境带来的维度困境，强化学习的应用通常局限于稀疏传感器和执行器。本研究将强化学习与传统动力系统最优控制相结合，采用一种新型物理增强强化学习（PEARL）范式，专门用于控制高维和参数动力学系统，利用其动力学的微分性。具体来说，PEARL采用了一种演员-伴随算法，利用自动微分计算短期视野内的策略梯度和基于伴随的未来收益敏感度，通过神经网络近似，显著减少环境交互数量，同时缓解长期梯度不稳定性。通过两个具有挑战性的非稳态流参数导航问题，我们展示了PEARL：（i）有效利用可微环境以超越最先进的强化学习算法，（ii）得益于物理引导的策略学习，（iii）能够在多个场景中推广，这在参数化系统处理时至关重要，（iv）使强化学习能够扩展到高维状态和动作空间， 无需低维状态表示或多智能体策略。

Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

以树状搜索视频：基于长视频质量保证的自我纠正代理

• Authors: Ce Zhang, Ziyang Wang, Yulu Pan, Oluwatumininu Oguntola, Pranav Wagh, Qiyu Wu, Hiromi Wakaki, Mohit Bansal, Gedas Bertasius
• Subjects: Subjects: Computer Vision and Pattern Recognition (cs.CV)
• Arxiv link: https://arxiv.org/abs/2607.16189
• Pdf link: https://arxiv.org/pdf/2607.16189
• Abstract Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these agents typically converge prematurely and cannot recover from an early mistake. We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors. To train this navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards. On three Grounded LVQA benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D), VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D. The learned policy also transfers to general long-video QA, surpassing all prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains: removing either adaptive descent or explicit backtracking substantially degrades performance. Code is available at this https URL.
• 中文摘要 接地长视频问答（Grounded LVQA）要求在回答关于长视频的问题时，局限支持该答案的短证据区间。近期的代理方法将此任务定位为多回合探索，采用单一crop_video（开始，结束）动作，支持粗细到细的狭窄，但不提供细到粗的回溯原语。因此，这些代理通常会过早收敛，无法从早期失误中恢复。我们提出了VideoTreeSearch（VTS）框架，将有基础的LVQA定位为在自适应时序树上的迭代自我纠正搜索。VTS通过视觉场景边界构建非均匀树，使每个节点对应语义连贯的段，并训练代理通过四个离散操作导航树：zoom_in、zoom_out、shift和answer。这些操作将回溯和恢复暴露为显式、可学习的原语，而非隐式行为。为了训练这种导航，我们引入了轨迹合成流水线，生成多步路径穿越树，包括故意绕行错误分支后恢复。我们利用这些轨迹进行监督微调，随后进行强化学习，并给予基础和答案准确性奖励。在三个Grounded LVQA基准测试（CG-Bench、Haystack-LVBency、Haystack-Ego4D）上，VTS在CG-Bench上表现优于最强的先前代理方法+12.5 mIoU，Haystack-Ego4D为+7.4 T-F1。所学策略也适用于通用长视频质量保证，在Video-MME、MLVU和LVBench上比之前所有代理基线高出最多+7.1的精度点。消融验证了自我纠正的层级搜索是这些提升的核心机制：去除适应性下降或显式回溯都会显著降低性能。代码可在此 https URL 访问。

Keyword: diffusion policy

There is no result