diff --git a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_controlnet/README.md b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_controlnet/README.md
index c93df4fba6f62e9618ed0324333ec3e9150c4f3e..30fbf1e9b19af54a466a711e623f40f6ffd27bdc 100644
--- a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_controlnet/README.md
+++ b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_controlnet/README.md
@@ -2,23 +2,23 @@
- [概述](#ZH-CN_TOPIC_0000001172161501)
- - [输入输出数据](#section540883920406)
- [推理环境准备](#ZH-CN_TOPIC_0000001126281702)
+
- [快速上手](#ZH-CN_TOPIC_0000001126281700)
- [获取源码](#section4622531142816)
- [模型推理](#section741711594517)
-- [模型推理性能&精度](#ZH-CN_TOPIC_0000001172201573)
-# 概述``
+# 概述
- ControlNet是一种神经网络架构，可将控制信息添加到预训练的扩散模型中。作用是通过添加额外控制条件，来引导Stable Diffusion生成图像，从而提升 AI 图像生成的可控性和精度。在使用ControlNet模型之后，Stable Diffusion模型的权重被复制出两个相同的部分，分别是“锁定”副本和“可训练”副本。ControlNet主要在“可训练”副本上施加控制条件，然后将施加控制条件之后的结果和原来SD模型的结果相加获得最终的输出结果。神经架构与“零卷积”（零初始化卷积层）连接，参数从零逐渐增长，确保微调的过程不会受到噪声影响。这样可以使用小批量数据集就能对控制条件进行学习训练，同时不会破坏Stable Diffusion模型原本的能力。如今ControlNet的应用包括：控制人物姿势、线稿上色、画质修复等。
+ ControlNet是一种神经网络架构，可将控制信息添加到预训练的扩散模型中。作用是通过添加额外控制条件，来引导Stable Diffusion生成图像，从而提升 AI 图像生成的可控性和精度。在使用ControlNet模型之后，Stable Diffusion模型的权重被复制出两个相同的部分，分别是“锁定”副本和“可训练”副本。ControlNet主要在“可训练”副本上施加控制条件，然后将施加控制条件之后的结果和原来SD模型的结果相加获得最终的输出结果。神经架构与“零卷积”（零初始化卷积层）连接，参数从零逐渐增长，确保微调的过程不会受到噪声影响。这样可以使用小批量数据集就能对控制条件进行学习训练，同时不会破坏Stable Diffusion模型原本的能力。
+ ControlNet的应用包括：控制人物姿势、线稿上色、画质修复等。
- 参考实现：
```bash
# controlnet-canny-sdxl-1.0
https://huggingface.co/diffusers/controlnet-canny-sdxl-1.0
```
-# 推理环境准备``
+# 推理环境准备
- 该模型需要以下插件与驱动
@@ -31,9 +31,9 @@
该模型性能受CPU规格影响，建议使用64核CPU（arm）以复现性能
-# 快速上手``
+# 快速上手
-## 获取源码``
+## 获取源码
1. 安装依赖。
@@ -60,14 +60,16 @@
python3 stable_diffusion_attention_patch.py
```
-## 准备数据集``
+## 准备数据集
1. 获取原始数据集。
ControlNet是一个控制预训练图像扩散模型的神经网络，允许输入调节图像，然后使用该调节图像来操控图像生成。调节图像可从官网下载。
+ ```bash
wget https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png
+ ```
-## 模型推理``
+## 模型推理
1. 模型转换。
使用Pytorch导出pt模型，然后使用MindIE推理引擎转换为适配昇腾的模型。
@@ -130,14 +132,14 @@
静态编译场景：
- ./models/clip/clip_bs{batch_size}.pt, ./models/clip/clip_bs{batch_size}_compile.ts 和 ./models/clip/clip2_bs{batch_size}.pt, ./models/clip/clip2_bs{batch_size}_compile.ts
- - ./models/unet/unet_bs{batch_size*2}.pt, ./models/unet/unet_bs{batch_size*2}_compile_static.ts
+ - ./models/unet/unet_bs{batch_size x 2}.pt, ./models/unet/unet_bs{batch_size x 2}_compile_static.ts
- ./models/vae/vae_bs{batch_size}.pt, ./models/vae/vae_bs{batch_size}_compile_static.ts
- ./models/control/control_bs{batch_size}.pt, ./models/control/control_bs{batch_size}_compile_static.ts
动态分档场景：
- ./models/clip/clip_bs{batch_size}.pt, ./models/clip/clip_bs{batch_size}_compile.ts 和 ./models/clip/clip2_bs{batch_size}.pt, ./models/clip/clip2_bs{batch_size}_compile.ts
- - ./models/unet/unet_bs{batch_size*2}.pt, ./models/unet/unet_bs{batch_size*2}_compile.ts
+ - ./models/unet/unet_bs{batch_size x 2}.pt, ./models/unet/unet_bs{batch_size x 2}_compile.ts
- ./models/vae/vae_bs{batch_size}.pt, ./models/vae/vae_bs{batch_size}_compile.ts
- ./models/control/control_bs{batch_size}.pt, ./models/control/control_bs{batch_size}_compile.ts
diff --git a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_inpainting/README.md b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_inpainting/README.md
index b2aacbf223ce354f954ce5ace26191893387e890..c574f3956a76d386cb5e78b52c83b20d1e9f13de 100644
--- a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_inpainting/README.md
+++ b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_inpainting/README.md
@@ -2,8 +2,6 @@
- [概述](#ZH-CN_TOPIC_0000001172161501)
-
- - [输入输出数据](#section540883920406)
- [推理环境准备](#ZH-CN_TOPIC_0000001126281702)
@@ -12,8 +10,6 @@
- [获取源码](#section4622531142816)
- [模型推理](#section741711594517)
-- [模型推理性能&精度](#ZH-CN_TOPIC_0000001172201573)
-
# 概述
@@ -79,9 +75,13 @@
1. 获取原始数据集。
Inpainting图像重绘。图像编辑是指对图像进行修改、调整和优化的过程。可以包括对图像的颜色、对比度、亮度、饱和度等进行调整，以及修复图像中的缺陷、删除不需要的元素、添加新的图像内容等操作。
- img: wget https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png
- mask img: wget https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png
-
+ ```bash
+ # img
+ wget https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png
+
+ #mask img
+ wget https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png
+ ```
## 模型推理
@@ -127,14 +127,14 @@
静态编译场景：
- ./models/clip/clip_bs{batch_size}.pt, ./models/clip/clip_bs{batch_size}_compile.ts 和 ./models/clip/clip2_bs{batch_size}.pt, ./models/clip/clip2_bs{batch_size}_compile.ts
- - ./models/unet/unet_bs{batch_size*2}.pt, ./models/unet/unet_bs{batch_size*2}_compile_static.ts
+ - ./models/unet/unet_bs{batch_size x 2}.pt, ./models/unet/unet_bs{batch_size x 2}_compile_static.ts
- ./models/vae/vae_bs{batch_size}.pt, ./models/vae/vae_bs{batch_size}_compile_static.ts
- ./models/image_encode/image_encode_bs{batch_size}.pt, ./models/image_encode/image_encode_bs{batch_size}_compile_static.ts
动态分档场景：
- ./models/clip/clip_bs{batch_size}.pt, ./models/clip/clip_bs{batch_size}_compile.ts 和 ./models/clip/clip2_bs{batch_size}.pt, ./models/clip/clip2_bs{batch_size}_compile.ts
- - ./models/unet/unet_bs{batch_size*2}.pt, ./models/unet/unet_bs{batch_size*2}_compile.ts
+ - ./models/unet/unet_bs{batch_size x 2}.pt, ./models/unet/unet_bs{batch_size x 2}_compile.ts
- ./models/vae/vae_bs{batch_size}.pt, ./models/vae/vae_bs{batch_size}_compile.ts
- ./models/image_encode/image_encode_bs{batch_size}.pt, ./models/image_encode/image_encode_bs{batch_size}_compile.ts
diff --git a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/README.md b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/README.md
index 8150976b2328ee882c2680484500046c74c72409..669b981ca0a1166cb8ef35b38967cf87786a9ba2 100644
--- a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/README.md
+++ b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/README.md
@@ -2,8 +2,6 @@
- [概述](#ZH-CN_TOPIC_0000001172161501)
-
- - [输入输出数据](#section540883920406)
- [推理环境准备](#ZH-CN_TOPIC_0000001126281702)
@@ -12,8 +10,6 @@
- [获取源码](#section4622531142816)
- [模型推理](#section741711594517)
-- [模型推理性能&精度](#ZH-CN_TOPIC_0000001172201573)
-
# 概述
@@ -71,7 +67,6 @@
# 若使用unetCache
python3 stable_diffusionxl_unet_patch.py
```
- ```
## 准备数据集
@@ -122,6 +117,21 @@
- --device：推理设备ID
- --use_cache: 【可选】在推理过程中使用cache
+ 静态编译场景：
+
+ - ./models/clip/clip_bs{batch_size}.pt, ./models/clip/clip_bs{batch_size}_compile.ts 和 ./models/clip/clip2_bs{batch_size}.pt, ./models/clip/clip2_bs{batch_size}_compile.ts
+ - ./models/unet/unet_bs{batch_size x 2}.pt, ./models/unet/unet_bs{batch_size x 2}_compile_static.ts
+ - ./models/vae/vae_bs{batch_size}.pt, ./models/vae/vae_bs{batch_size}_compile_static.ts
+ - ./models/ddim/ddim_bs{batch_size}.pt, ./models/ddim/ddim_bs{batch_size}_compile_static.ts
+
+ 动态分档场景：
+
+ - ./models/clip/clip_bs{batch_size}.pt, ./models/clip/clip_bs{batch_size}_compile.ts 和 ./models/clip/clip2_bs{batch_size}.pt, ./models/clip/clip2_bs{batch_size}_compile.ts
+ - ./models/unet/unet_bs{batch_size x 2}.pt, ./models/unet/unet_bs{batch_size x 2}_compile.ts
+ - ./models/vae/vae_bs{batch_size}.pt, ./models/vae/vae_bs{batch_size}_compile.ts
+ - ./models/ddim/ddim_bs{batch_size}.pt, ./models/ddim/ddim_bs{batch_size}_compile.ts
+
+
2. 开始推理验证。
diff --git a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/export_ts_prompt_weight.py b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/export_ts_prompt_weight.py
index 4d5263bf1d23ffa9ce6a7c962bd07eeb5a65f9f6..7c82b3090b18d97c5250689a1fcc6b6b98a29070 100644
--- a/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/export_ts_prompt_weight.py
+++ b/MindIE/MindIE-Torch/built-in/foundation/stable_diffusion_xl_prompt_weight/export_ts_prompt_weight.py
@@ -265,6 +265,7 @@ def trace_unet_init(sd_pipeline, batch_size, unet_pt_path):
in_channels = unet_model.config.in_channels
max_position_embeddings = encoder_model.config.max_position_embeddings
encoder_hidden_size_2 = encoder_model_2.config.hidden_size
+ encoder_hidden_size = encoder_model.config.hidden_size + encoder_hidden_size_2
if not os.path.exists(unet_pt_path):
dummy_input = (