Latent Swap Joint Diffusion for 2D Long-Form Latent Generation

Anonymous



Swap Forward (SaFa) is a modality-agnostic and efficient method for generating seamless and coherent long-form spectrum and panorama through latent swap joint diffusion across multiple views. With a forward-only design, it employs just two stepwise latent swap operators to achieve (1) smooth overlap transitions with more spectral details and (2) global coherence (e.g., color and timbre) witch a cross-view similarity-diversity balance.

Introduction

At its core, first, the bidirectional Self-Loop Latent Swap operator performs frame-level swaps on the overlapping regions of adjacent subviews during each denoising step. The operation adapts well to the high-frequency variability of the spectrum, preserving high-frequency details and avoiding spectrum aliasing.

Second, to achieve cross-view consistency, the unidirectional Reference-Guided Latent Swap is applied between the reference and each subview during the early steps. It provides an independent centralized trajectory to guide each subview’s diffusion in the early denoising steps. By adjusting the timing and interval of the swap operation, we can balance the similarity-diversity trade-off across subviews in a forward-only manner.

Swap Forward demonstrates strong generalization (adapting to both spectrum and image generation, U-Net and DiT architectures, and fixed or flexible attention window sizes), simplicity (applying only two latent swap operators in a forward-only manner), and efficiency (2 ~ 20× faster inference time with less subview count) to generate seamless and coherent long-form audio and panorama.

Result Gallery

Comparison

Note: Play the audio after 1-2 minutes when everything has done.

The continuous gurgling sound of bubbles in the water.
SaFa (Ours)
Casino Ambience, electronic slot machines.
SaFa (Ours)
MultiDiffusion (MD) [Bar-Tal et al. 2023]
MultiDiffusion (MD) [Bar-Tal et al. 2023]
Movie Gen (MD*) [Meta 2024]
Movie Gen (MD*) [Meta 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]

The audience's enthusiastic and passionate cheers and loud whistles in the stadium.
SaFa (Ours)
Someone is Whistling.
SaFa (Ours)
MultiDiffusion (MD) [Bar-Tal et al. 2023]
MultiDiffusion (MD) [Bar-Tal et al. 2023]
Movie Gen (MD*) [Meta 2024]
Movie Gen (MD*) [Meta 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
The instrumental music features an ensemble that resembles the orchestra.
SaFa (Ours)
Smooth, soulful saxophone in a relaxed jazz tune for quiet evenings.
SaFa (Ours)
MultiDiffusion (MD) [Bar-Tal et al. 2023]
MultiDiffusion (MD) [Bar-Tal et al. 2023]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]

A modern synthesizer creating futuristic soundscapes.
SaFa (Ours)
Acoustic ballad with heartfelt lyrics and soft piano.
SaFa (Ours)
MultiDiffusion (MD) [Bar-Tal et al. 2023]
MultiDiffusion (MD) [Bar-Tal et al. 2023]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
A Chinese ink a vibrant landscape with farmers, tourists.
SaFa (Ours)
A vibrant landscape inspired by Qingming RiverSide Scence.
SaFa (Ours)
MultiDiffusion (MD) [Bar-Tal et al. 2023]
MultiDiffusion (MD) [Bar-Tal et al. 2023]
SyncDiffusion [Lee et al. 2023]
SyncDiffusion [Lee et al. 2023]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
A photo of a city skyline at night.
SaFa (Ours)
A photo of a mountain range at twilight.
SaFa (Ours)
MultiDiffusion (MD) [Bar-Tal et al. 2023]
MultiDiffusion (MD) [Bar-Tal et al. 2023]
SyncDiffusion [Lee et al. 2023]
SyncDiffusion [Lee et al. 2023]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]
Merge-Attend-Diffuse (MAD) [Fabio et al. 2024]

Audio & Music Generation

The sound of waves crashing on the beach, with distant kids playing accompanied with the sound of seagull.
A woman is speaking.
The audience's enthusiastic and passionate cheers and loud whistles in the stadium.
Casino Ambience, electronic slot machines.
Smooth, soulful saxophone in a relaxed jazz tune for quiet evenings.
This song features a harp playing the main melody. This melody is calming and relaxing. This song can be played in a meditation center. There are no other instruments in this song. There are no voices in this song.
This instrumental song features a flute playing a high pitched melody. The melody starts off with one high pitched staccato note. After a brief pause, the flute plays two ascending notes in which the second note is sustained. Then a third higher note is played followed by a descending run of four more notes and ending on one higher note. There are no other instruments in this song. There is no percussion in this song. This song has a relaxing mood. This song can be played at a meditation center.
A modern synthesizer creating futuristic soundscapes.
The continuous gurgling sound of bubbles in the water.
The dog's bark is clear and powerful, often indicating alertness or excitement, with varying pitch.
Someone is Whistling.
A synth pad is playing a drone sound in the lower mid range. Cymbals are creating atmosphere while a flute/string/brass sound is playing a melody. The whole recording is full of reverb. This song may be playing in a forest documentary.

Panorama Generation

A photo of a mountain range at twilight.
A photo of a snowy mountain peak with skier.
A photo of the dolomites.
Majestic red rock formations glowing in the sunset.
Skyline of New York City.
Create a vibrant landscape inspired by 'Qingming Riverside Scene, with riverside life, famers, tourists, mountains, and traditional buildings.
Skyline of New York City.
A photo of a city skyline at night.
A photo of Chinese ink a vibrant landscape with farmers, tourists, mountains, traditional buildings and animal.
Silhouette wallpaper of a dreamy scene with shooting stars.
Natural landscape in anime style illustration.
Cartoon panorama of spring summer beautiful nature.
A photo of a beautiful ocean with coral reef.
A photo of a lake under the northern lights.
A serene sunrise over a misty lake, with soft colors reflecting on the water's surface.
Majestic icebergs floating in the Arctic Ocean, under a bright polar sun.
A photo of a forest with a misty fog.
A photo of a grassland with animals.
Community garden with raised beds and flowers.
Serene mountain valley carpeted in vibrant fall foliage.