关键词: 视频描述生成, 多模态特征融合, 注意力机制, 生成对抗网络

Abstract: To address the challenges of insufficient multi-modal feature fusion and weak semantic consistency in video de鄄 scription generation tasks, this study proposes a video description generation model based on multi-modal fea鄄 ture fusion and adversarial learning (MFF-AL). First, static and dynamic video features are extracted using a residual network with 152 layers (ResNet-152) and inflated 3D convolutional networks (I3D), respectively, while audio features are extracted using a local-global attention mechanism. Then, cross-modal fusion of video and audio features is implemented via a cross-attention mechanism, with an expert network introduced to dy鄄 namically modulate the fused features and an information bottleneck theory applied to compress redundant data. Finally, a dual-layer generative adversarial network is designed, combining adversarial training with information noise-contrastive estimation (InfoNCE) to generate descriptive text. Experiments on the MSR-VTT dataset demonstrate that the proposed MFF-AL model outperforms existing methods across four widely used evaluation metrics in the video description field. The experimental results validate the effectiveness of the proposed MFF AL model in improving the accuracy and naturalness of video description generation

Key words:  , video description generation, multi-modal feature fusion, attention mechanism, generative adversarial network