Latent Swap Joint Diffusion for Long-Form Audio Generation

At its core, first, the bidirectional Self-Loop Latent Swap operator performs frame-level swaps on the overlapping regions of adjacent subviews during each denoising step. The frame-level operation adapts to high-frequency variability in both the spectrum and image latents, achieving higher time-frequency resolution with less distortion compared to latent averaging.

Second, the unidirectional Reference-Guided Latent Swap occurs between reference and each subview during early steps. It leverages an independent reference denoising process to guide other subviews, providing centralized guidance to ensure consistency without repetition across subviews. Given differences in the flattening orders to 1D sequences, the Reference-Guided Latent Swap is applied along different directions for spectrum and image generation, enabling segment-wise swaps while avoiding excessive repetition.

Swap Forward demonstrates strong generalization (adapting to both spectrum and image generation, U-Net and DiT architectures, and fixed or flexible attention window sizes), simplicity (using two simple swap operators in a forward-only manner), and efficiency (up to 11 times faster without extra time or computation consumption) to generate seamless and coherent long-form outputs.