Thanh V. T. Tran1 Ngoc-Son Nguyen1 Luong Tran1 Long-Khanh Pham1 Paarth Neekhara2 Shehzeen Hussain 2 Van Nguyen1
1 FPT Software AI Center, Vietnam 2 NVIDIA Corporation, USA
Flowley is single-stage architecture that synthesizes temporally and semantically aligned soundtracks from silent videos. The key architecture innovation is Progressive Soft-masked Cross-Attention (PSCA), a novel mechanism that embeds precise temporal prior directly within the attention layers.
We also propose SoundCap, a plug-and-play pipeline leveraging audio-visual LLMs to generate detailed, sound-oriented captions that robustly guide our model.
Checkout our project page for more generated samples.
We have only tested this on Ubuntu 22.04 and PyTorch 2.0+
We use PyTorch 2.2 with CUDA 11.8. Please check if your GPUs/driver support this.
cd Flowley
conda env create -f environment.ymlpip install -e .See TRAINING.md
See INFERENCE.md
If our paper or codebase aids your research, please consider citing us:
@InProceedings{tran2026flowley,
author="Tran, Thanh V. T.
and Nguyen, Ngoc-Son
and Tran, Luong
and Pham, Long-Khanh
and Neekhara, Paarth
and Hussain, Shehzeen
and Nguyen, Van",
editor="Favaro, Paolo
and Kukelova, Zuzana
and Maki, Atsuto
and Rohrbach, Anna
and Schindler, Konrad
and Tombari, Federico",
title="Precise Video-to-Audio Generation with Cross-Modal Alignment in Latent Space",
booktitle="Computer Vision -- ECCV 2026",
year="2026",
publisher="Springer Nature Switzerland",
address="Cham",
pages="174--192",
abstract="Video-to-audio (V2A) generation aims to synthesize realistic audio that is both semantically consistent with and temporally synchronized to a silent video. Despite recent progress, many methods still rely on multi-stage training, resulting in high computational costs and long runtimes, or transform visual input into text to leverage pretrained text-to-audio models, sacrificing fine-grained temporal cues. To overcome these limitations, we propose Flowley, an end-to-end, single-stage training architecture that produces soundtracks by combining visual features with textual prompts. Crucially, we introduce Progressive Soft-masked Cross-Attention, which embeds audio-visual synchronization directly within its attention mechanism, adding zero additional computational cost compared to standard attention layers. We further observe that existing V2A benchmarks lack sound-oriented descriptive captions, which can potentially degrade the quality of the synthesized audio. To remedy this, we propose SoundCap, a plug-and-play pipeline for creating detailed, sound-aware captions that guide the model. Remarkably, without integrating any pretrained audio-visual alignment modules, Flowley achieves state-of-the-art performance on VGGSound across multiple metrics. Moreover, by incorporating SoundCap, we further exceed the performance of the strongest existing close-sourced methods in terms of audio quality in the zero-shot setting.",
isbn="978-3-032-37261-1"
}
We would like to thank the authors for their great work.