Despite the growth of Text-to-Audio (TTA) models for creative applications like sound design and live jamming, existing systems, particularly in the open-source, lack the ability for flexible fine-grained control (such as vocal "sketches") while maintaining fast inference speeds for real-time interaction. We address this unnecessary tradeoff between speed and control through FlashFoley, the first open-source, accelerated sketch2audio model. With FlashFoley, we extend the Sketch2Sound framework, wherein we finetune TTA models with pitch, volume, and brightness controls through simple linear adaptation, to adversarial post-training, allowing the model to generate 11s samples from audio sketches in 75ms. We combine this with a novel zero-shot chunked streaming algorithm, enabling real-time interactive generation while maintaining high-quality fast offline sampling.
This section presents random audio samples generated by the five different methods: SAO-Small base (SAOS), SAOS + sketch controls, FlashFoley, FlashFoley with Block-Autoregressive Sampling (BAR), and FlashFoley trained with the sketch-aware contrastive loss (+ Sketch L_C). For each prompt, we show the vocal sketch used for conditioning for all methods with sketch controls. All methods follow the hyperparameters detailed in the appendix.
@inproceedings{novack2025flashfoley,
title={FlashFoley: Fast Interactive Sketch2Audio Generation},
author={Novack, Zachary and Saito, Koichi and Zhong, Zhi and Shibuya, Takashi and Cui, Shuyang and McAuley, Julian and Berg-Kirkpatrick, Taylor and Simon, Christian and Takahashi, Shusuke and Mitsufuji, Yuki},
booktitle={39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop: AI for Music},
year={2025}
}