Diffusers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v0.42.1).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

PAN2Transformer3DModel

The diffusion transformer of PAN 2. It runs joint attention over the video tokens and the text tokens, with separate projections and feed-forward networks for each stream.

The video input stacks the noisy latents, the conditioning latents and a one-channel conditioning mask, so the same model serves text-to-video and image-to-video.

import torch
from diffusers import PAN2Transformer3DModel

transformer = PAN2Transformer3DModel.from_pretrained("IFM/PAN2", subfolder="transformer", dtype=torch.bfloat16)

PAN2Transformer3DModel

class diffusers.PAN2Transformer3DModel

< source >

( patch_size: tuple = (1, 2, 2)in_channels: int = 97out_channels: int = 48num_attention_heads: int = 32attention_head_dim: int = 128num_layers: int = 44num_refiner_layers: int = 4mlp_ratio: float = 8.0context_mlp_ratio: float = 4.0refiner_mlp_ratio: float = 4.0text_embed_dim: int = 4096rope_axes_dim: tuple = (16, 56, 56)rope_theta: float = 256.0rope_temporal_scale: float = 0.6666666666666666 )

Parameters

patch_size (tuple[int, int, int], defaults to (1, 2, 2)) — The (t, h, w) patch size of the patch embedding.
in_channels (int, defaults to 97) — The number of input channels: noisy latents, conditioning latents and one conditioning mask channel.
out_channels (int, defaults to 48) — The number of output channels.
num_attention_heads (int, defaults to 32) — The number of attention heads.
attention_head_dim (int, defaults to 128) — The number of channels in each attention head.
num_layers (int, defaults to 44) — The number of joint-attention blocks.
num_refiner_layers (int, defaults to 4) — The number of token refiner blocks applied to the text embeddings.
mlp_ratio (float, defaults to 8.0) — The ratio of the video feed-forward inner dimension to the hidden dimension.
context_mlp_ratio (float, defaults to 4.0) — The ratio of the text feed-forward inner dimension to the hidden dimension.
refiner_mlp_ratio (float, defaults to 4.0) — The ratio of the token refiner feed-forward inner dimension to the hidden dimension.
text_embed_dim (int, defaults to 4096) — The dimension of the text encoder hidden states.
rope_axes_dim (tuple[int, int, int], defaults to (16, 56, 56)) — The number of channels of each (t, h, w) axis in the rotary embedding.
rope_theta (float, defaults to 256.0) — The base of the rotary embedding.
rope_temporal_scale (float, defaults to 16 / 24) — The factor the frame positions of the rotary embedding are multiplied with: the reference frame rate of the rotary embedding over the frame rate of the videos (16 over 24 fps).

The diffusion transformer of PAN 2.

A stack of joint-attention blocks over the concatenation of the patchified video latents and the refined text tokens. The video input carries the noisy latents, the conditioning latents and a conditioning mask stacked along the channel axis, so the same model serves text-to-video and image-to-video.

forward

< source >

( hidden_states: Tensortimestep: Tensorencoder_hidden_states: Tensorattention_kwargs: dict[str, typing.Any] | None = Nonereturn_dict: bool = True ) → Transformer2DModelOutput or tuple

Parameters

hidden_states (torch.Tensor of shape (batch_size, in_channels, num_frames, height, width)) — The noisy latents, the conditioning latents and the conditioning mask, concatenated along channels.
timestep (torch.Tensor of shape (batch_size,)) — The denoising timestep in [0, 1000].
encoder_hidden_states (torch.Tensor of shape (batch_size, text_seq_len, text_embed_dim)) — The text encoder hidden states.
attention_kwargs (dict, optional) — A kwargs dictionary passed along to the AttentionProcessor; only used for the LoRA scale.
return_dict (bool, optional, defaults to True) — Whether or not to return a Transformer2DModelOutput instead of a plain tuple.

Returns

Transformer2DModelOutput or tuple

The predicted velocity of shape (batch_size, out_channels, num_frames, height, width).

The PAN2Transformer3DModel forward method.

Update on GitHub

←OvisImageTransformer2DModel PixArtTransformer2DModel→