HY-Himmel: Hierarchical Video Encoding for Long Video Understanding
The newly introduced hierarchical video-language framework, HY-Himmel, tackles challenges in understanding long videos using multimodal language models. This innovative system differentiates between semantic processing and motion analysis: sparse anchor I-frames are directed to a Vision Transformer (ViT) for identifying objects and scene layouts, while a lightweight compressed-domain tri-stream adapter encodes the dense inter-frame intervals. This adapter gathers motion insights from motion-vector maps, residual maps, and I-frame context to create aligned motion tokens. After Stage-1 contrastive alignment, these tokens are introduced into the LLM through a differentiable placeholder mechanism, maintaining compatibility with the static visual backbone. This method enhances motion perception and minimizes decode costs and token expansion. The technical report can be found on arXiv with the identifier 2605.08158.
Key facts
- HY-Himmel is a hierarchical video-language framework for long-video understanding.
- It uses sparse anchor I-frames for object identity and scene layout via a host ViT.
- Dense inter-frame intervals are encoded by a lightweight compressed-domain tri-stream adapter.
- The adapter processes motion-vector maps, residual maps, and I-frame context.
- Motion tokens are injected into the LLM via a differentiable placeholder mechanism.
- Stage-1 contrastive alignment ensures compatibility with the frozen visual backbone.
- The system addresses decode costs, quadratic token growth, and weak motion perception.
- The technical report is published on arXiv with identifier 2605.08158.
Entities
Institutions
- arXiv