模型推理

git clone https://gitee.com/ascend/ModelZoo-PyTorch.git

export PYTHONPATH=/home/{用户名}/code/ModelZoo-PyTorch/MindIE/MultiModal/OpenSora-1.2/:$PYTHONPATH

将获取模型权重中获取的各模型权重文件和表1中获取的各模型配置文件，按照如下层级放置：

├── OpenSora1.2
│   ├── model_index.json
│   ├── scheduler
│        └── scheduler_config.json
│   ├── text_encoder
│        ├── config.json
│        └── text_encoder的模型权重
│   ├── tokenizer
│        ├── config.json
│        └── tokenizer的模型权重
│   ├── transformer
│        ├── config.json
│        └── transformer的模型权重
│   └── vae
│        ├── config.json
│        ├── vae的模型权重
│        └── vae_2d
│             └── vae
│                  ├── config.json
│                  └── vae_2d的模型权重

MindIE SD接口会对传入的文件或文件夹做权限安全校验，因此模型权重文件、配置文件及其所在文件夹需进行权限配置，权限要求为：对应三组权限不得超过640且需要和执行用户的属组和权限保持一致。

该路径下需要修改的配置文件参考示例如下所示。

• model_index.json配置文件参考示例：

  {
      "_class_name": "OpenSoraPipeline",
      "_mindiesd_version": "1.0.0",
      "scheduler": [
          "mindiesd",
          "RFlowScheduler"
      ],
      "text_encoder": [
          "transformers",
          "T5EncoderModel"
      ],
      "tokenizer": [
          "transformers",
          "AutoTokenizer"
        ],
      "transformer": [
          "mindiesd",
          "STDiT3"
      ],
      "vae": [
          "mindiesd",
          "VideoAutoencoder"
      ]
  }

• scheduler_config.json配置文件参考示例：

  {
      "_class_name": "RFlowScheduler",
      "_mindiesd_version": "1.0.0",
      "num_sampling_steps": 30,
      "num_timesteps": 1000
  }

• transformer目录中的config.json配置文件参考示例：

  {
      "architectures": [
          "STDiT3"
      ],
      "caption_channels": 4096,
      "class_dropout_prob": 0.1,
      "depth": 28,
      "drop_path": 0.0,
      "enable_flash_attn": true,
      "enable_layernorm_kernel": false,
      "enable_sequence_parallelism": false,
      "freeze_y_embedder": true,
      "hidden_size": 1152,
      "in_channels": 4,
      "input_size": [
          null,
          null,
          null
      ],
      "input_sq_size": 512,
      "mlp_ratio": 4.0,
      "model_max_length": 300,
      "model_type": "STDiT3",
      "num_heads": 16,
      "only_train_temporal": false,
      "patch_size": [
          1,
          2,
          2
      ],
      "pred_sigma": true,
      "qk_norm": true,
      "skip_y_embedder": false,
      "torch_dtype": "float32",
      "transformers_version": "4.36.2"
  }

• vae目录中的config.json配置文件参考示例：
  

  vae/vae_2d/vae目录中的config.json配置文件无需修改。

  {
      "architectures": [
          "VideoAutoencoder"
      ],
      "cal_loss": false,
      "freeze_vae_2d": false,
      "from_pretrained": null,
      "micro_frame_size": 17,
      "model_type": "VideoAutoencoder",
      "scale": [
          3.85,
          2.32,
          2.33,
          3.06
      ],
      "shift": [
          -0.1,
          0.34,
          0.27,
          0.98
      ],
      "torch_dtype": "float32",
      "transformers_version": "4.36.2",
      "vae_2d": {
          "from_pretrained": "PixArt-alpha/pixart_sigma_sdxlvae_T5_diffusers",
          "local_files_only": true,
          "micro_batch_size": 4,
          "subfolder": "vae",
          "type": "VideoAutoencoderKL"
      },
      "vae_temporal": {
          "from_pretrained": null,
          "type": "VAE_Temporal_SD"
      }
  }

1. 导入依赖包
2. 配置文生视频工作流
3. 模型编译
4. 模型推理

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

#1.导入依赖包
import torch
from mindiesd import OpenSoraPipeline12, compile_pipe
#2.配置文生视频工作流，用户可自行设置权重路径，例如：/home/{用户名}/example/OpenSora1.2
pipe = OpenSoraPipeline12.from_pretrained(model_path="/home/{用户名}/example/OpenSora1.2", num_frames=32, image_size=(720, 1280), fps=8,
    enable_sequence_parallelism=False, dtype=torch.bfloat16)
#3.模型编译
pipe = compile_pipe(pipe)
#4.模型推理的输出如果需要保存成视频，则可添加参数output_type="thwc"，默认开启，无需保存视频可去掉此参数
video = pipe(prompts=["Several giant wooly mammoths approach treading through a snowy meadow, their long wooly fur lightly blows in the wind as they walk, snow covered trees and dramatic snow capped mountains in the distance, mid afternoon light with wispy clouds and a sun high in the distance creates a warm glow, the low camera view is stunning capturing the large furry mammal with beautiful photography, depth of field."], output_type="thwc")
#5.保存视频，用户设置在合法路径保存视频，例如在当前路径保存名称为sample.mp4的视频文件
from torchvision.io import write_video
save_path="sample.mp4"
write_video(save_path, video, fps=8, video_codec="h264")