RoCo-ACE: New Method for Knowledge Injection in MLLMs
A new approach called RoCo-ACE has been developed by researchers, focusing on rollout-conditioned online distillation for injecting knowledge into multimodal large language models (MLLMs). This technique updates pretrained MLLMs with new factual or domain-specific information, but integrating full authoritative responses can lead to drift in behaviors that haven’t been updated. To address this, online distillation trains on model-generated rollouts, although traditional reference-conditioned distillation offers only basic supervision, neglecting the importance of reference-supported rollout tokens and indirectly supervising missing facts. RoCo-ACE enhances this by reallocating distillation weight to these tokens, while ACE introduces a sparse correction for authoritative anchors not included in the rollout. The study was tested in three knowledge-injection scenarios, six retention benchmarks, and various base models. The findings are published on arXiv with the identifier 2607.24771.
Key facts
- RoCo-ACE is a rollout-conditioned online distillation objective for knowledge injection.
- It addresses drift in non-updated behavior during knowledge injection in MLLMs.
- RoCo uses same-rollout reference-free/reference-conditioned likelihood contrast.
- ACE adds sparse reference-side anchored correction for omitted authoritative anchors.
- Evaluated across three knowledge-injection settings, six retention benchmarks, multiple baselines, and multiple base models.
- Paper available on arXiv: 2607.24771.
Entities
Institutions
- arXiv