From 056fe1b8933e92861bff3992945ab5a6c433e1f0 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Mon, 17 Mar 2025 19:58:55 +0800 Subject: [PATCH 01/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 123 +++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 MindIE/MultiModal/StepVideo-TI2V/README.md diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md new file mode 100644 index 0000000000..030fc25086 --- /dev/null +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -0,0 +1,123 @@ +## 一、准备运行环境 + + **表 1** 版本配套表 + + | 配套 | 版本 | 环境准备指导 | + | ----- | ----- |-----| + | Python | 3.10.2 | - | + | torch | 2.1.0 | - | + +### 1.1 获取CANN&MindIE安装包&环境准备 +- 设备支持 +Atlas 800I A2(8*64G)推理设备:支持的卡数最小为1 +- [Atlas 800I A2(8*64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) +- [环境准备指导](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha002/softwareinst/instg/instg_0001.html) + +### 1.2 CANN安装 +```shell +# 增加软件包可执行权限,{version}表示软件版本号,{arch}表示CPU架构,{soc}表示昇腾AI处理器的版本。 +chmod +x ./Ascend-cann-toolkit_{version}_linux-{arch}.run +chmod +x ./Ascend-cann-kernels-{soc}_{version}_linux.run +# 校验软件包安装文件的一致性和完整性 +./Ascend-cann-toolkit_{version}_linux-{arch}.run --check +./Ascend-cann-kernels-{soc}_{version}_linux.run --check +# 安装 +./Ascend-cann-toolkit_{version}_linux-{arch}.run --install +./Ascend-cann-kernels-{soc}_{version}_linux.run --install + +# 设置环境变量 +source /usr/local/Ascend/ascend-toolkit/set_env.sh +``` + +### 1.3 环境依赖安装 +```shell +pip3 install -r requirements.txt +``` + +### 1.4 MindIE安装 +```shell +# 增加软件包可执行权限,{version}表示软件版本号,{arch}表示CPU架构。 +chmod +x ./Ascend-mindie_${version}_linux-${arch}.run +./Ascend-mindie_${version}_linux-${arch}.run --check + +# 方式一:默认路径安装 +./Ascend-mindie_${version}_linux-${arch}.run --install +# 设置环境变量 +cd /usr/local/Ascend/mindie && source set_env.sh + +# 方式二:指定路径安装 +./Ascend-mindie_${version}_linux-${arch}.run --install-path=${AieInstallPath} +# 设置环境变量 +cd ${AieInstallPath}/mindie && source set_env.sh +``` + +### 1.5 Torch_npu安装 +下载 pytorch_v{pytorchversion}_py{pythonversion}.tar.gz +```shell +tar -xzvf pytorch_v{pytorchversion}_py{pythonversion}.tar.gz +# 解压后,会有whl包 +pip install torch_npu-{pytorchversion}.xxxx.{arch}.whl +``` + +## 二、下载权重 + +### 2.1 权重及配置文件说明 +stepvideo-ti2v权重链接: +```shell +https://huggingface.co/stepfun-ai/stepvideo-ti2v +``` + +## 三、StepVideo-TI2V使用 +当前支持的分辨率: +| 分辨率 | h/w=9:16 | h/w=1:1 | +| ---- | ---- | ---- | +| 720P | 544x1992 | 768x768 | +### 3.1 下载到本地 +```shell +git clone https://modelers.cn/MindIE/stepvideo-ti2v.git +cd stepvideo-ti2v +``` + + +### 3.2 性能测试 +执行命令: +```shell +# 使用上一步下载的权重 +export model_dir='./stepvideo-ti2v/' +export ALGO=0 + +# 指定0卡和1卡开启vae和text_encoder的api服务 +ASCEND_RT_VISIBLE_DEVICES=0 python api/call_remote_server.py --model_dir ${model_dir} --enable_vae True & +ASCEND_RT_VISIBLE_DEVICES=1 python api/call_remote_server.py --model_dir ${model_dir} --enable_llm True & + +# 执行推理 +ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ +--model_dir ${model_dir} \ +--height 554 \ +--width 992 \ +--num_frames 102 \ +--infer_steps 50 \ +--ulysses_degree 2 \ +--tensor_parallel_degree 2 \ +--prompt="一名宇航员在月球上发现一块石碑,上面印有“stepfun”字样,闪闪发光" \ +--first_image_path './assets/demo.png' \ +--save_path './results' +``` + +参数说明: +- ALGO: 为0表示默认FA算子;设置为1表示使用高性能FA算子 +- model_dir: 配置文件及权重路径。 +- height: 生成视频的高 +- weight: 生成视频的宽 +- num_frames: 视频帧数 +- infer_steps: 推理步数 +- ulysses_degree: ulysses并行度 +- tensor_parallel_degree: tp并行度 +- prompt: 文本提示词 +- first_image_path: 用于生成视频的图片路径 +- save_path: 生成的视频的保存路径 + + +## 声明 +- 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 +- 如您在使用本代码仓的过程中,发现任何问题(包括但不限于功能问题、合规问题),请在本代码仓提交issue,我们将及时审视并解答。 \ No newline at end of file -- Gitee From 318c129a6d60abbf49c3233934889fceae797155 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Mon, 17 Mar 2025 20:06:25 +0800 Subject: [PATCH 02/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 030fc25086..6293082f7d 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -68,10 +68,7 @@ https://huggingface.co/stepfun-ai/stepvideo-ti2v ``` ## 三、StepVideo-TI2V使用 -当前支持的分辨率: -| 分辨率 | h/w=9:16 | h/w=1:1 | -| ---- | ---- | ---- | -| 720P | 544x1992 | 768x768 | +当前支持的分辨率:544x992、768x768 ### 3.1 下载到本地 ```shell git clone https://modelers.cn/MindIE/stepvideo-ti2v.git -- Gitee From 3c133d33e8ad69caa7d817c5bcd786cf3b43079c Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Mon, 17 Mar 2025 20:09:42 +0800 Subject: [PATCH 03/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 6293082f7d..4166f5dffd 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -75,7 +75,6 @@ git clone https://modelers.cn/MindIE/stepvideo-ti2v.git cd stepvideo-ti2v ``` - ### 3.2 性能测试 执行命令: ```shell @@ -114,6 +113,11 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ - first_image_path: 用于生成视频的图片路径 - save_path: 生成的视频的保存路径 +## 四、模型推理性能结果参考 +### StepVideo-TI2V +| 硬件形态 | cpu规格 | 卡数 | 迭代次数 | 等价优化平均耗时 | 算法优化平均耗时 | +| :------: | :------: | :------: |:----:| :------: |:-----:| +| Atlas 800I A2(8*64G) | 64核(x86) | 1 | 50 | | | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 96fb70c9fbebe2d5ea0105e40150f590ebc63358 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Mon, 17 Mar 2025 20:11:01 +0800 Subject: [PATCH 04/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 4166f5dffd..799bb7c97c 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -115,9 +115,9 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ ## 四、模型推理性能结果参考 ### StepVideo-TI2V -| 硬件形态 | cpu规格 | 卡数 | 迭代次数 | 等价优化平均耗时 | 算法优化平均耗时 | -| :------: | :------: | :------: |:----:| :------: |:-----:| -| Atlas 800I A2(8*64G) | 64核(x86) | 1 | 50 | | | +| 硬件形态 | cpu规格 | 卡数 | 迭代次数 | 平均耗时 | +| :------: | :------: | :------: |:----:|:-----:| +| Atlas 800I A2(8*64G) | 64核(x86) | 4 | 50 | | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 299857dcfb4a50dbf8fee7562bd2f956a1ef2cf4 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Mon, 17 Mar 2025 21:27:41 +0800 Subject: [PATCH 05/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 799bb7c97c..c44020765c 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -63,7 +63,13 @@ pip install torch_npu-{pytorchversion}.xxxx.{arch}.whl ### 2.1 权重及配置文件说明 stepvideo-ti2v权重链接: + +```shell +# modelers +https://modelers.cn/StepFun/stepvideo-ti2v +``` ```shell +# huggingface https://huggingface.co/stepfun-ai/stepvideo-ti2v ``` -- Gitee From b13d6e3380566ec243b82c890cd2cd36169b3cf7 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Mon, 17 Mar 2025 21:48:06 +0800 Subject: [PATCH 06/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index c44020765c..4e99cde2fa 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -66,7 +66,7 @@ stepvideo-ti2v权重链接: ```shell # modelers -https://modelers.cn/StepFun/stepvideo-ti2v +https://modelers.cn/models/StepFun/stepvideo-ti2v ``` ```shell # huggingface -- Gitee From 411f1870d5c5a3f6ec5462463f2f8ae545da1851 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Tue, 18 Mar 2025 14:33:19 +0800 Subject: [PATCH 07/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 4e99cde2fa..19d0dd0764 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -1,15 +1,15 @@ +## 版本配套 +| 组件 | 版本 | +| - | - | +| MindIE | 1.0.0 | +| CANN | 8.0.0 | +| PTA | 6.0.0 | +| MindStudio | 7.0.0 | +| HDK | 24.1.0 | ## 一、准备运行环境 - - **表 1** 版本配套表 - - | 配套 | 版本 | 环境准备指导 | - | ----- | ----- |-----| - | Python | 3.10.2 | - | - | torch | 2.1.0 | - | - ### 1.1 获取CANN&MindIE安装包&环境准备 - 设备支持 -Atlas 800I A2(8*64G)推理设备:支持的卡数最小为1 +Atlas 800I A2(8*64G)推理设备:支持的卡数最小为6 - [Atlas 800I A2(8*64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) - [环境准备指导](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha002/softwareinst/instg/instg_0001.html) -- Gitee From 9d6c06ab9da1e9358a65bbe0345b894669138c5a Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Tue, 18 Mar 2025 15:58:05 +0800 Subject: [PATCH 08/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 19d0dd0764..1f5818c169 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -9,6 +9,7 @@ ## 一、准备运行环境 ### 1.1 获取CANN&MindIE安装包&环境准备 - 设备支持 +Atlas 800T A2(8*64G)推理设备:支持的卡数最小为6 Atlas 800I A2(8*64G)推理设备:支持的卡数最小为6 - [Atlas 800I A2(8*64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) - [环境准备指导](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha002/softwareinst/instg/instg_0001.html) @@ -99,8 +100,8 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ --width 992 \ --num_frames 102 \ --infer_steps 50 \ ---ulysses_degree 2 \ ---tensor_parallel_degree 2 \ +--ulysses_degree 4 \ +--tensor_parallel_degree 1 \ --prompt="一名宇航员在月球上发现一块石碑,上面印有“stepfun”字样,闪闪发光" \ --first_image_path './assets/demo.png' \ --save_path './results' @@ -119,11 +120,6 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ - first_image_path: 用于生成视频的图片路径 - save_path: 生成的视频的保存路径 -## 四、模型推理性能结果参考 -### StepVideo-TI2V -| 硬件形态 | cpu规格 | 卡数 | 迭代次数 | 平均耗时 | -| :------: | :------: | :------: |:----:|:-----:| -| Atlas 800I A2(8*64G) | 64核(x86) | 4 | 50 | | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 53f71cde1ffc23672334ecd52ffea31230f1606d Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Tue, 18 Mar 2025 16:06:11 +0800 Subject: [PATCH 09/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 1f5818c169..72fe2b99e1 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -11,7 +11,7 @@ - 设备支持 Atlas 800T A2(8*64G)推理设备:支持的卡数最小为6 Atlas 800I A2(8*64G)推理设备:支持的卡数最小为6 -- [Atlas 800I A2(8*64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) +- [Atlas 800T A2(8*64G)/Atlas 800I A2(8*64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) - [环境准备指导](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha002/softwareinst/instg/instg_0001.html) ### 1.2 CANN安装 -- Gitee From 0e85ae78194876b95a7867ca14d0bf166603e3a9 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Tue, 18 Mar 2025 16:09:35 +0800 Subject: [PATCH 10/34] add_stepvideo-ti2v --- MindIE/MultiModal/StepVideo-TI2V/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 72fe2b99e1..51aece08ec 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -9,9 +9,9 @@ ## 一、准备运行环境 ### 1.1 获取CANN&MindIE安装包&环境准备 - 设备支持 -Atlas 800T A2(8*64G)推理设备:支持的卡数最小为6 -Atlas 800I A2(8*64G)推理设备:支持的卡数最小为6 -- [Atlas 800T A2(8*64G)/Atlas 800I A2(8*64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) +Atlas 800T A2(64G):支持的卡数最小为6 +Atlas 800I A2(64G):支持的卡数最小为6 +- [Atlas 800T A2(64G)/Atlas 800I A2(64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) - [环境准备指导](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha002/softwareinst/instg/instg_0001.html) ### 1.2 CANN安装 -- Gitee From 44f8f2751c7c098cdbe0076ae3af38058b23341b Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Tue, 18 Mar 2025 16:48:59 +0800 Subject: [PATCH 11/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 51aece08ec..1acefe624d 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -78,7 +78,7 @@ https://huggingface.co/stepfun-ai/stepvideo-ti2v 当前支持的分辨率:544x992、768x768 ### 3.1 下载到本地 ```shell -git clone https://modelers.cn/MindIE/stepvideo-ti2v.git +git clone https://modelers.cn/MindIE/Step-Video-TI2V.git cd stepvideo-ti2v ``` -- Gitee From 310cf003c18e6dda1d41a9661e3c83c6be5e82c0 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 14:50:29 +0800 Subject: [PATCH 12/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 39 ++++++++++------------ 1 file changed, 17 insertions(+), 22 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 1acefe624d..9f6824e9f0 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -1,11 +1,3 @@ -## 版本配套 -| 组件 | 版本 | -| - | - | -| MindIE | 1.0.0 | -| CANN | 8.0.0 | -| PTA | 6.0.0 | -| MindStudio | 7.0.0 | -| HDK | 24.1.0 | ## 一、准备运行环境 ### 1.1 获取CANN&MindIE安装包&环境准备 - 设备支持 @@ -30,12 +22,7 @@ chmod +x ./Ascend-cann-kernels-{soc}_{version}_linux.run source /usr/local/Ascend/ascend-toolkit/set_env.sh ``` -### 1.3 环境依赖安装 -```shell -pip3 install -r requirements.txt -``` - -### 1.4 MindIE安装 +### 1.3 MindIE安装 ```shell # 增加软件包可执行权限,{version}表示软件版本号,{arch}表示CPU架构。 chmod +x ./Ascend-mindie_${version}_linux-${arch}.run @@ -52,7 +39,7 @@ cd /usr/local/Ascend/mindie && source set_env.sh cd ${AieInstallPath}/mindie && source set_env.sh ``` -### 1.5 Torch_npu安装 +### 1.4 Torch_npu安装 下载 pytorch_v{pytorchversion}_py{pythonversion}.tar.gz ```shell tar -xzvf pytorch_v{pytorchversion}_py{pythonversion}.tar.gz @@ -79,15 +66,24 @@ https://huggingface.co/stepfun-ai/stepvideo-ti2v ### 3.1 下载到本地 ```shell git clone https://modelers.cn/MindIE/Step-Video-TI2V.git -cd stepvideo-ti2v +``` +### 3.2 安装依赖 +```shell +cd StepVideo-TI2V +pip install -e . +``` +安装xfuser +```shell +git clone -b 0.4.2 https://github.com/xdit-project/xDiT.git +bash patch.sh ``` -### 3.2 性能测试 +### 3.3 性能测试 执行命令: ```shell # 使用上一步下载的权重 export model_dir='./stepvideo-ti2v/' -export ALGO=0 +export ALGO=1 # 指定0卡和1卡开启vae和text_encoder的api服务 ASCEND_RT_VISIBLE_DEVICES=0 python api/call_remote_server.py --model_dir ${model_dir} --enable_vae True & @@ -100,9 +96,9 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ --width 992 \ --num_frames 102 \ --infer_steps 50 \ ---ulysses_degree 4 \ ---tensor_parallel_degree 1 \ ---prompt="一名宇航员在月球上发现一块石碑,上面印有“stepfun”字样,闪闪发光" \ +--ulysses_degree 1 \ +--tensor_parallel_degree 4 \ +--prompt="笑起来" \ --first_image_path './assets/demo.png' \ --save_path './results' ``` @@ -120,7 +116,6 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ - first_image_path: 用于生成视频的图片路径 - save_path: 生成的视频的保存路径 - ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 - 如您在使用本代码仓的过程中,发现任何问题(包括但不限于功能问题、合规问题),请在本代码仓提交issue,我们将及时审视并解答。 \ No newline at end of file -- Gitee From 8a8ad58d68a702ebe6b3f5b6b1a34632cc6f2a67 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 15:02:35 +0800 Subject: [PATCH 13/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 9f6824e9f0..c1de02daa7 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -1,3 +1,7 @@ +## 版本配套 +敬请期待 + + ## 一、准备运行环境 ### 1.1 获取CANN&MindIE安装包&环境准备 - 设备支持 @@ -116,6 +120,9 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ - first_image_path: 用于生成视频的图片路径 - save_path: 生成的视频的保存路径 +## 四、模型推理性能结果参考 +尽情期待 + ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 - 如您在使用本代码仓的过程中,发现任何问题(包括但不限于功能问题、合规问题),请在本代码仓提交issue,我们将及时审视并解答。 \ No newline at end of file -- Gitee From 7d724a7e86a8507e45bd6285ec0bbb9a5aa05c83 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 15:40:26 +0800 Subject: [PATCH 14/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index c1de02daa7..e46b475212 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -121,7 +121,11 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ - save_path: 生成的视频的保存路径 ## 四、模型推理性能结果参考 -尽情期待 +### StepVideo-TI2V +| 硬件形态 | cpu规格 | 分辨率 | 迭代次数 | 平均耗时 | +| :------: | :------: | :------: |:----:| :------: | +| G8600 | 64核(x86) | 544x992 | 50 | | +| G8600 | 64核(x86) | 768x768 | 50 | | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 6cb408e09162a3dd1ee2d77c7df97caa8209d943 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 16:53:57 +0800 Subject: [PATCH 15/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index e46b475212..0439b0952a 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -124,8 +124,8 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ ### StepVideo-TI2V | 硬件形态 | cpu规格 | 分辨率 | 迭代次数 | 平均耗时 | | :------: | :------: | :------: |:----:| :------: | -| G8600 | 64核(x86) | 544x992 | 50 | | -| G8600 | 64核(x86) | 768x768 | 50 | | +| G8600 | 64核(x86) | 544x992 | 50 | 434s | +| G8600 | 64核(x86) | 768x768 | 50 | 507s | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 98c758c0d653b0606d3f81880578a115a2db5998 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 17:46:29 +0800 Subject: [PATCH 16/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 36 +++++++++++++++++----- 1 file changed, 28 insertions(+), 8 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 0439b0952a..335089a659 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -84,16 +84,19 @@ bash patch.sh ### 3.3 性能测试 执行命令: +使用上一步下载的权重 ```shell -# 使用上一步下载的权重 export model_dir='./stepvideo-ti2v/' export ALGO=1 - -# 指定0卡和1卡开启vae和text_encoder的api服务 +``` +指定0卡和1卡开启vae和text_encoder的api服务 +```shell ASCEND_RT_VISIBLE_DEVICES=0 python api/call_remote_server.py --model_dir ${model_dir} --enable_vae True & ASCEND_RT_VISIBLE_DEVICES=1 python api/call_remote_server.py --model_dir ${model_dir} --enable_llm True & +``` -# 执行推理 +四卡TP并行 +```shell ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ --model_dir ${model_dir} \ --height 554 \ @@ -102,6 +105,22 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ --infer_steps 50 \ --ulysses_degree 1 \ --tensor_parallel_degree 4 \ +--use-cpu-offload +--prompt="笑起来" \ +--first_image_path './assets/demo.png' \ +--save_path './results' +``` + +八卡TP并行 +```shell +ASCEND_RT_VISIBLE_DEVICES=2,3,4,5,6,7,8,9 torchrun --nproc_per_node 8 run_parallel.py \ +--model_dir ${model_dir} \ +--height 554 \ +--width 992 \ +--num_frames 102 \ +--infer_steps 50 \ +--ulysses_degree 1 \ +--tensor_parallel_degree 8 \ --prompt="笑起来" \ --first_image_path './assets/demo.png' \ --save_path './results' @@ -116,16 +135,17 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ - infer_steps: 推理步数 - ulysses_degree: ulysses并行度 - tensor_parallel_degree: tp并行度 +- use-cpu-offload: 对DiT使用cpu-offload - prompt: 文本提示词 - first_image_path: 用于生成视频的图片路径 - save_path: 生成的视频的保存路径 ## 四、模型推理性能结果参考 ### StepVideo-TI2V -| 硬件形态 | cpu规格 | 分辨率 | 迭代次数 | 平均耗时 | -| :------: | :------: | :------: |:----:| :------: | -| G8600 | 64核(x86) | 544x992 | 50 | 434s | -| G8600 | 64核(x86) | 768x768 | 50 | 507s | +| 硬件形态 | cpu规格 | 分辨率 | tp并行度 | 迭代次数 | 平均耗时 | +| :------: | :------: | :------: | :------: |:----:| :------: | +| G8600 | 64核(x86) | 544x992 | 8 | 50 | 434s | +| G8600 | 64核(x86) | 768x768 | 8 | 50 | 507s | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From a74a473462e429a2bed6e49fb33be132ae5f1758 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 17:52:30 +0800 Subject: [PATCH 17/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 335089a659..dd7aafab03 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -105,7 +105,7 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ --infer_steps 50 \ --ulysses_degree 1 \ --tensor_parallel_degree 4 \ ---use-cpu-offload +--use_cpu_offload \ --prompt="笑起来" \ --first_image_path './assets/demo.png' \ --save_path './results' @@ -135,7 +135,7 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5,6,7,8,9 torchrun --nproc_per_node 8 run_parall - infer_steps: 推理步数 - ulysses_degree: ulysses并行度 - tensor_parallel_degree: tp并行度 -- use-cpu-offload: 对DiT使用cpu-offload +- use_cpu_offload: 对DiT使用cpu-offload - prompt: 文本提示词 - first_image_path: 用于生成视频的图片路径 - save_path: 生成的视频的保存路径 -- Gitee From cedf7d11affac5a39be0dc627de310b3ffa057dc Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 17:58:40 +0800 Subject: [PATCH 18/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index dd7aafab03..55e095b9dc 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -1,5 +1,5 @@ ## 版本配套 -敬请期待 +### 敬请期待 ## 一、准备运行环境 @@ -144,8 +144,8 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5,6,7,8,9 torchrun --nproc_per_node 8 run_parall ### StepVideo-TI2V | 硬件形态 | cpu规格 | 分辨率 | tp并行度 | 迭代次数 | 平均耗时 | | :------: | :------: | :------: | :------: |:----:| :------: | -| G8600 | 64核(x86) | 544x992 | 8 | 50 | 434s | -| G8600 | 64核(x86) | 768x768 | 8 | 50 | 507s | +| Atlas 800T A2(64G) | 64核(x86) | 544x992 | 8 | 50 | 434s | +| Atlas 800T A2(64G) | 64核(x86) | 768x768 | 8 | 50 | 507s | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 3660f0123ff47f7e55438c5eefbc19b8cd8c960b Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 18:10:42 +0800 Subject: [PATCH 19/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 55e095b9dc..b364f8b19a 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -142,7 +142,7 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5,6,7,8,9 torchrun --nproc_per_node 8 run_parall ## 四、模型推理性能结果参考 ### StepVideo-TI2V -| 硬件形态 | cpu规格 | 分辨率 | tp并行度 | 迭代次数 | 平均耗时 | +| 硬件形态 | cpu规格 | 分辨率 | tp并行度 | 迭代次数 | 端到端耗时 | | :------: | :------: | :------: | :------: |:----:| :------: | | Atlas 800T A2(64G) | 64核(x86) | 544x992 | 8 | 50 | 434s | | Atlas 800T A2(64G) | 64核(x86) | 768x768 | 8 | 50 | 507s | -- Gitee From f631d813e3a9331cfaeff0747f93613db8dd38b6 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:01:55 +0800 Subject: [PATCH 20/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index b364f8b19a..582f90ec6a 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -142,10 +142,10 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5,6,7,8,9 torchrun --nproc_per_node 8 run_parall ## 四、模型推理性能结果参考 ### StepVideo-TI2V -| 硬件形态 | cpu规格 | 分辨率 | tp并行度 | 迭代次数 | 端到端耗时 | -| :------: | :------: | :------: | :------: |:----:| :------: | -| Atlas 800T A2(64G) | 64核(x86) | 544x992 | 8 | 50 | 434s | -| Atlas 800T A2(64G) | 64核(x86) | 768x768 | 8 | 50 | 507s | +| 硬件形态 | cpu规格 | 分辨率 | 并行度 | 迭代次数 | 端到端耗时(w/o cache) | 端到端耗时(with cache) | +| :------: | :------: | :------: | :------: |:----:| :------: | :------: | +| A800 | 64核(x86) | 544x992 | 5 | 30 | 491s | \ | +| Atlas 800T A2(64G) | 64核(x86) | 544x992 | 8 | 30 | 271s | 236s | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 8fde4bb39594ffea9f14620af53a246f4e30a292 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:03:57 +0800 Subject: [PATCH 21/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 582f90ec6a..a004a5b43f 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -89,11 +89,6 @@ bash patch.sh export model_dir='./stepvideo-ti2v/' export ALGO=1 ``` -指定0卡和1卡开启vae和text_encoder的api服务 -```shell -ASCEND_RT_VISIBLE_DEVICES=0 python api/call_remote_server.py --model_dir ${model_dir} --enable_vae True & -ASCEND_RT_VISIBLE_DEVICES=1 python api/call_remote_server.py --model_dir ${model_dir} --enable_llm True & -``` 四卡TP并行 ```shell @@ -113,7 +108,7 @@ ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ 八卡TP并行 ```shell -ASCEND_RT_VISIBLE_DEVICES=2,3,4,5,6,7,8,9 torchrun --nproc_per_node 8 run_parallel.py \ +torchrun --nproc_per_node 8 run_parallel.py \ --model_dir ${model_dir} \ --height 554 \ --width 992 \ -- Gitee From 4e317b6663160e09ee9b19c078cf36c588d3bca5 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:06:09 +0800 Subject: [PATCH 22/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index a004a5b43f..98a3403a63 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -92,7 +92,7 @@ export ALGO=1 四卡TP并行 ```shell -ASCEND_RT_VISIBLE_DEVICES=2,3,4,5 torchrun --nproc_per_node 4 run_parallel.py \ +torchrun --nproc_per_node 4 run_parallel.py \ --model_dir ${model_dir} \ --height 554 \ --width 992 \ -- Gitee From 821718b33f99e2dd3b49ef792c4d2d473704655c Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:09:45 +0800 Subject: [PATCH 23/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 98a3403a63..a9e0017224 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -137,10 +137,10 @@ torchrun --nproc_per_node 8 run_parallel.py \ ## 四、模型推理性能结果参考 ### StepVideo-TI2V -| 硬件形态 | cpu规格 | 分辨率 | 并行度 | 迭代次数 | 端到端耗时(w/o cache) | 端到端耗时(with cache) | +| 硬件形态 | cpu规格 | 分辨率 | 并行度 | 迭代次数 | 端到端耗时 | 单卡吞吐 | | :------: | :------: | :------: | :------: |:----:| :------: | :------: | -| A800 | 64核(x86) | 544x992 | 5 | 30 | 491s | \ | -| Atlas 800T A2(64G) | 64核(x86) | 544x992 | 8 | 30 | 271s | 236s | +| A800 | 64核(x86) | 544x992 | 5 | 30 | 491s | 0.041547 | +| Atlas 800T A2(64G) | 64核(x86) | 544x992 | 8 | 30 | 236s | 0.053571 | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 8048d03de9f4e28bbfb4bf2a0f9502144172f48e Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:24:29 +0800 Subject: [PATCH 24/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index a9e0017224..344f9d9d4d 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -1,7 +1,3 @@ -## 版本配套 -### 敬请期待 - - ## 一、准备运行环境 ### 1.1 获取CANN&MindIE安装包&环境准备 - 设备支持 @@ -9,7 +5,8 @@ Atlas 800T A2(64G):支持的卡数最小为6 Atlas 800I A2(64G):支持的卡数最小为6 - [Atlas 800T A2(64G)/Atlas 800I A2(64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) - [环境准备指导](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha002/softwareinst/instg/instg_0001.html) - +- MindIE配套版本: 敬请期待 + ### 1.2 CANN安装 ```shell # 增加软件包可执行权限,{version}表示软件版本号,{arch}表示CPU架构,{soc}表示昇腾AI处理器的版本。 @@ -137,10 +134,10 @@ torchrun --nproc_per_node 8 run_parallel.py \ ## 四、模型推理性能结果参考 ### StepVideo-TI2V -| 硬件形态 | cpu规格 | 分辨率 | 并行度 | 迭代次数 | 端到端耗时 | 单卡吞吐 | -| :------: | :------: | :------: | :------: |:----:| :------: | :------: | -| A800 | 64核(x86) | 544x992 | 5 | 30 | 491s | 0.041547 | -| Atlas 800T A2(64G) | 64核(x86) | 544x992 | 8 | 30 | 236s | 0.053571 | +| 硬件形态 | 分辨率 | GPU/NPU | 迭代次数 | 端到端耗时 | 单卡吞吐 | +| :------: | :------: | :------: |:----:| :------: | :------: | +| A800 | 544x992 | 5 | 30 | 491s | 0.041547 | +| Atlas 800T A2(64G)| 544x992 | 8 | 30 | 236s | 0.053571 | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From a9d022aac84db77159f79b8ae49c705b0a445773 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:26:57 +0800 Subject: [PATCH 25/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 344f9d9d4d..cf7b4edebf 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -55,7 +55,7 @@ stepvideo-ti2v权重链接: ```shell # modelers -https://modelers.cn/models/StepFun/stepvideo-ti2v +https://modelscope.cn/models/stepfun-ai/stepvideo-ti2v ``` ```shell # huggingface -- Gitee From 376bf006f6f25f29525f2029157d4eec3c3c5a4d Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:29:14 +0800 Subject: [PATCH 26/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index cf7b4edebf..86d276b8ac 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -51,16 +51,9 @@ pip install torch_npu-{pytorchversion}.xxxx.{arch}.whl ## 二、下载权重 ### 2.1 权重及配置文件说明 -stepvideo-ti2v权重链接: - -```shell -# modelers -https://modelscope.cn/models/stepfun-ai/stepvideo-ti2v -``` -```shell -# huggingface -https://huggingface.co/stepfun-ai/stepvideo-ti2v -``` +| Models | 🤗 Huggingface | 🤖 Modelscope | +| :-------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | +| Step-Video-TI2V | [Download](https://huggingface.co/stepfun-ai/stepvideo-ti2v) | [Download](https://modelscope.cn/models/stepfun-ai/stepvideo-ti2v) | ## 三、StepVideo-TI2V使用 当前支持的分辨率:544x992、768x768 -- Gitee From fc587ccfdb4ab0636cea664bc90eb934e4d4a9a9 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:42:14 +0800 Subject: [PATCH 27/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 86d276b8ac..7da4a9dd58 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -1,3 +1,7 @@ +## 简述 +Step-Video-TI2V,一款参数规模达30B的SOTA文本驱动图像转视频生成模型,基于文本与图像输入,可生成长达 102 帧的视频。该模型在Step-Video-T2V基础上完成训练,将图像条件融入生成视频的首帧,借助 Step-Video-T2V 的视频变分自编码器(Video-VAE)将其编码为潜在表征。此外,引入运动分数条件,使得用户能够调控基于图像条件生成视频的动态程度。 + +通过MindIE SD推理适配,实现了Step-Video-TI2V模型在昇腾上的高性能推理。主要使用了FA长序列场景下的高性能模式和混合并行切分策略,执行544px × 992px × 102f的视频生成(step=30),耗时236s ## 一、准备运行环境 ### 1.1 获取CANN&MindIE安装包&环境准备 - 设备支持 @@ -5,7 +9,8 @@ Atlas 800T A2(64G):支持的卡数最小为6 Atlas 800I A2(64G):支持的卡数最小为6 - [Atlas 800T A2(64G)/Atlas 800I A2(64G)](https://www.hiascend.com/developer/download/community/result?module=pt+ie+cann&product=4&model=32) - [环境准备指导](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha002/softwareinst/instg/instg_0001.html) -- MindIE配套版本: 敬请期待 +- MindIE配套版本 + **敬请期待** ### 1.2 CANN安装 ```shell @@ -127,6 +132,7 @@ torchrun --nproc_per_node 8 run_parallel.py \ ## 四、模型推理性能结果参考 ### StepVideo-TI2V +NPU端到端性能和GPU进行了对比,平均每卡的吞吐达到1.29x GPU A800。性能测试如下: | 硬件形态 | 分辨率 | GPU/NPU | 迭代次数 | 端到端耗时 | 单卡吞吐 | | :------: | :------: | :------: |:----:| :------: | :------: | | A800 | 544x992 | 5 | 30 | 491s | 0.041547 | -- Gitee From 7aa3b1c590bed087d605f222ecf9821ebb441186 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:43:38 +0800 Subject: [PATCH 28/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 7da4a9dd58..4f6beb9d8f 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -133,7 +133,7 @@ torchrun --nproc_per_node 8 run_parallel.py \ ## 四、模型推理性能结果参考 ### StepVideo-TI2V NPU端到端性能和GPU进行了对比,平均每卡的吞吐达到1.29x GPU A800。性能测试如下: -| 硬件形态 | 分辨率 | GPU/NPU | 迭代次数 | 端到端耗时 | 单卡吞吐 | +| 硬件形态 | 分辨率 | GPU数(NPU数) | 迭代次数 | 端到端耗时 | 单卡吞吐 | | :------: | :------: | :------: |:----:| :------: | :------: | | A800 | 544x992 | 5 | 30 | 491s | 0.041547 | | Atlas 800T A2(64G)| 544x992 | 8 | 30 | 236s | 0.053571 | -- Gitee From 5501e5ccfe0dada607d1279e8bb543ce462201ca Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:46:17 +0800 Subject: [PATCH 29/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 4f6beb9d8f..0836810c04 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -133,7 +133,7 @@ torchrun --nproc_per_node 8 run_parallel.py \ ## 四、模型推理性能结果参考 ### StepVideo-TI2V NPU端到端性能和GPU进行了对比,平均每卡的吞吐达到1.29x GPU A800。性能测试如下: -| 硬件形态 | 分辨率 | GPU数(NPU数) | 迭代次数 | 端到端耗时 | 单卡吞吐 | +| 硬件形态 | 分辨率 | GPU数(NPU数) | 迭代次数 | 端到端耗时 | 单卡吞吐(fps/p) | | :------: | :------: | :------: |:----:| :------: | :------: | | A800 | 544x992 | 5 | 30 | 491s | 0.041547 | | Atlas 800T A2(64G)| 544x992 | 8 | 30 | 236s | 0.053571 | -- Gitee From 77dba8e6e719c5a05936413b154c13a72c46cc2f Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 21:56:12 +0800 Subject: [PATCH 30/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 0836810c04..7c39b69c02 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -135,8 +135,8 @@ torchrun --nproc_per_node 8 run_parallel.py \ NPU端到端性能和GPU进行了对比,平均每卡的吞吐达到1.29x GPU A800。性能测试如下: | 硬件形态 | 分辨率 | GPU数(NPU数) | 迭代次数 | 端到端耗时 | 单卡吞吐(fps/p) | | :------: | :------: | :------: |:----:| :------: | :------: | -| A800 | 544x992 | 5 | 30 | 491s | 0.041547 | -| Atlas 800T A2(64G)| 544x992 | 8 | 30 | 236s | 0.053571 | +| A800 | 102fx544x992 | 5 | 30 | 491s | 0.041547 | +| Atlas 800T A2(64G)| 102fx544x992 | 8 | 30 | 236s | 0.053571 | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 9197000de149df9c56869360cc36586a77e5b705 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 22:05:45 +0800 Subject: [PATCH 31/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 7c39b69c02..b027f27476 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -135,8 +135,8 @@ torchrun --nproc_per_node 8 run_parallel.py \ NPU端到端性能和GPU进行了对比,平均每卡的吞吐达到1.29x GPU A800。性能测试如下: | 硬件形态 | 分辨率 | GPU数(NPU数) | 迭代次数 | 端到端耗时 | 单卡吞吐(fps/p) | | :------: | :------: | :------: |:----:| :------: | :------: | -| A800 | 102fx544x992 | 5 | 30 | 491s | 0.041547 | -| Atlas 800T A2(64G)| 102fx544x992 | 8 | 30 | 236s | 0.053571 | +| A800 | 544px x 992px x 102f | 5 | 30 | 491s | 0.041547 | +| Atlas 800T A2(64G)| 544px x 992px x 102f | 8 | 30 | 236s | 0.053571 | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 9dc7adf9f463078498cf5768d215398929569736 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 22:09:06 +0800 Subject: [PATCH 32/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index b027f27476..9729cced5b 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -135,8 +135,8 @@ torchrun --nproc_per_node 8 run_parallel.py \ NPU端到端性能和GPU进行了对比,平均每卡的吞吐达到1.29x GPU A800。性能测试如下: | 硬件形态 | 分辨率 | GPU数(NPU数) | 迭代次数 | 端到端耗时 | 单卡吞吐(fps/p) | | :------: | :------: | :------: |:----:| :------: | :------: | -| A800 | 544px x 992px x 102f | 5 | 30 | 491s | 0.041547 | -| Atlas 800T A2(64G)| 544px x 992px x 102f | 8 | 30 | 236s | 0.053571 | +| A800 | 544px × 992px × 102f | 5 | 30 | 491s | 0.041547 | +| Atlas 800T A2(64G)| 544px × 992px × 102f | 8 | 30 | 236s | 0.053571 | ## 声明 - 本代码仓提到的数据集和模型仅作为示例,这些数据集和模型仅供您用于非商业目的,如您使用这些数据集和模型来完成示例,请您特别注意应遵守对应数据集和模型的License,如您因使用数据集或模型而产生侵权纠纷,华为不承担任何责任。 -- Gitee From 14315ac4bf71f03125e76ba0fa44b7f5137b02a7 Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 22:18:44 +0800 Subject: [PATCH 33/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index 9729cced5b..f917851d31 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -56,9 +56,9 @@ pip install torch_npu-{pytorchversion}.xxxx.{arch}.whl ## 二、下载权重 ### 2.1 权重及配置文件说明 -| Models | 🤗 Huggingface | 🤖 Modelscope | +| Models | 🤗 Huggingface | MODELERS | | :-------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | -| Step-Video-TI2V | [Download](https://huggingface.co/stepfun-ai/stepvideo-ti2v) | [Download](https://modelscope.cn/models/stepfun-ai/stepvideo-ti2v) | +| Step-Video-TI2V | [Download](https://huggingface.co/stepfun-ai/stepvideo-ti2v) | [Download](https://modelers.cn/models/StepFun/Step-Video-TI2V-npu-weight) | ## 三、StepVideo-TI2V使用 当前支持的分辨率:544x992、768x768 -- Gitee From 4caa6861c8feb4d0504cc7a1a4b966876e172a2a Mon Sep 17 00:00:00 2001 From: zhoufan2956 Date: Wed, 19 Mar 2025 22:19:54 +0800 Subject: [PATCH 34/34] update --- MindIE/MultiModal/StepVideo-TI2V/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MindIE/MultiModal/StepVideo-TI2V/README.md b/MindIE/MultiModal/StepVideo-TI2V/README.md index f917851d31..c420c9578b 100644 --- a/MindIE/MultiModal/StepVideo-TI2V/README.md +++ b/MindIE/MultiModal/StepVideo-TI2V/README.md @@ -56,7 +56,7 @@ pip install torch_npu-{pytorchversion}.xxxx.{arch}.whl ## 二、下载权重 ### 2.1 权重及配置文件说明 -| Models | 🤗 Huggingface | MODELERS | +| Models | Huggingface | MODELERS | | :-------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | | Step-Video-TI2V | [Download](https://huggingface.co/stepfun-ai/stepvideo-ti2v) | [Download](https://modelers.cn/models/StepFun/Step-Video-TI2V-npu-weight) | -- Gitee