PAN 2

PAN2 is a foundation world model that simulates how a scene evolves under a time-grounded plan of actions and camera movements. Its 29B generative simulator is a dual-stream multimodal diffusion transformer, integrated here; the reasoning backbone that writes its prompts is not.

PAN 2 is integrated as Modular Diffusers blocks only. PAN2Blocks supports two workflows:

Workflow	Inputs
`t2v`	`prompt`
`i2v`	`prompt`, `image`

import torch
from diffusers import ModularPipeline
from diffusers.utils import export_to_video, load_image

pipe = ModularPipeline.from_pretrained("IFM/PAN2")
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

prompt = "A red fox trots through fresh snow in a pine forest at dawn."

video = pipe(prompt=prompt, generator=torch.Generator().manual_seed(0), output="videos")[0]
export_to_video(video, "pan2_t2v.mp4", fps=24)

image = load_image("path/to/first_frame.png")
video = pipe(prompt=prompt, image=image, generator=torch.Generator().manual_seed(0), output="videos")[0]
export_to_video(video, "pan2_i2v.mp4", fps=24)

Classifier-free guidance is configured on the guider (guidance_scale=2.5 by default):

pipe.update_components(guider=pipe.guider.new(guidance_scale=5.0))

Every call takes a single prompt, which is encoded without padding. The transformer therefore never needs an attention mask, and every attention backend is available:

pipe.transformer.set_attention_backend("flash_hub")  # FlashAttention-2

Safety checker

PAN 2 runs guardrails on the prompt before generation and on every generated video after it, and a blocked prompt or video raises a ValueError. The checks are on by default, and load_components() loads the guardrails, which need the PAN 2 guardrail package:

pip install git+https://github.com/ifm-ai/pan2-guardrail

Context parallelism

PAN2Transformer3DModel supports context parallelism. The text and video token counts are usually not divisible by the number of devices (the number of text tokens depends on the prompt), so use ulysses_anything or ring_anything for inference. Plain ulysses_degree / ring_degree, for example for training, requires both token counts to be divisible by the context parallel size.

# torchrun --nproc_per_node=2 pan2_context_parallel.py
import torch
import torch.distributed as dist
from diffusers import ContextParallelConfig, ModularPipeline

dist.init_process_group("nccl")
rank = dist.get_rank()
torch.cuda.set_device(rank)

pipe = ModularPipeline.from_pretrained("IFM/PAN2")
pipe.load_components(dtype=torch.bfloat16)
pipe.to(f"cuda:{rank}")
pipe.transformer.enable_parallelism(config=ContextParallelConfig(ulysses_degree=2, ulysses_anything=True))

video = pipe(prompt="A red fox trots through fresh snow.", generator=torch.Generator().manual_seed(0), output="videos")[0]
dist.destroy_process_group()

PAN2ModularPipeline

class diffusers.PAN2ModularPipeline

< source >

( blocks: diffusers.modular_pipelines.modular_pipeline.ModularPipelineBlocks | None = Nonepretrained_model_name_or_path: str | os.PathLike | None = Nonecomponents_manager: diffusers.modular_pipelines.components_manager.ComponentsManager | None = Nonecollection: str | None = Noneworkflow: str | None = Nonemodular_config_dict: dict[str, typing.Any] | None = Noneconfig_dict: dict[str, typing.Any] | None = None**kwargs )

A ModularPipeline for PAN 2: text-to-video, and image-to-video from a first frame.

import torch
from diffusers import ModularPipeline

pipe = ModularPipeline.from_pretrained("IFM/PAN2")
pipe.load_components(dtype=torch.bfloat16)

check_safety_checker_loaded

< source >

( )

Raise if the checks are on but the PAN 2 guardrail is not loaded.

disable_safety_checker

< source >

( )

Turn off the prompt and video checks, at your own risk.

enable_safety_checker

< source >

( )

Check the prompt and the generated videos with the PAN 2 guardrail.

PAN2Blocks

class diffusers.PAN2Blocks

< source >

( )

Modular pipeline blocks for text-to-video and image-to-video generation with PAN 2.

Supported workflows:

t2v: requires prompt
i2v: requires prompt, image

Components: text_encoder (Qwen3_5TextModel) tokenizer (Qwen2Tokenizer) safety_checker (PAN2SafetyChecker) vae (AutoencoderKLWan) video_processor (VideoProcessor) scheduler (FlowMatchEulerDiscreteScheduler) transformer (PAN2Transformer3DModel) guider (ClassifierFreeGuidance)

Configs: enable_safety_checker (default: True)

Inputs: prompt (str): The prompt to guide generation, a single string. negative_prompt (str, optional): The prompt not to guide generation, a single string. When guidance needs one and none (or an empty one) is passed, PAN 2’s default negative prompt is used, its time ranges set to the clip’s duration. max_sequence_length (int, optional, defaults to 1536): Maximum sequence length for prompt encoding. num_frames (int, optional, defaults to 313): Number of video frames to generate, of the form 4 * n + 1. fps (float, optional, defaults to 24.0): The frame rate of the video, which sets its duration in the default negative prompt. image (Image, optional): The first frame, a single image. height (int, optional): The height in pixels of the generated video, 704 by default. width (int, optional): The width in pixels of the generated video, 1248 by default. num_videos_per_prompt (int, optional, defaults to 1): The number of videos to generate per prompt. image_latents (Tensor, optional): The normalized first-frame latents. Can be generated from the vae_encoder step. num_inference_steps (int): The number of denoising steps. sigmas (list, optional): Custom sigmas for the denoising process. latents (Tensor): Pre-generated noisy latents to use as inputs for video generation. generator (Generator, optional): Torch generator for deterministic generation. attention_kwargs (dict, optional): Additional kwargs for attention processors. output_type (str, optional, defaults to np): Output format: ‘pil’, ‘np’, ‘pt’.

Outputs: videos (list): The generated videos.

Update on GitHub