SPeCTrA-Sum: A New Framework for Multimodal Summarization
Researchers have unveiled a new framework called SPeCTrA-Sum, which stands for Sampler Perceiver with Cross-modal Transformer and gated Attention for Summarization. This innovative system aims to improve multimodal summarization by combining text summaries with the selection of key images. It addresses challenges like mismatched representations and poor cross-modal grounding in existing methods. Notably, it features a Deep Visual Processor (DVP) that aligns the visual encoder with the language model for better integration, and a Visual Relevance Predictor (VRP) that highlights key images through soft labels from a Determinantal Point Processes (DPP) teacher. Their findings are published on arXiv, listed under ID 2605.11753.
Key facts
- SPeCTrA-Sum is a unified framework for multimodal summarization.
- It jointly performs text summarization and representative image selection.
- The system introduces a Deep Visual Processor (DVP) for hierarchical visual-language fusion.
- A Visual Relevance Predictor (VRP) selects images using DPP teacher distillation.
- The framework addresses representational mismatches in existing methods.
- The paper is available on arXiv with ID 2605.11753.
- The approach uses cross-modal transformer and gated attention.
- Multi-objective training is employed.
Entities
Institutions
- arXiv