From b1dc75729bb89ad410e2d325baec66fb1412573c Mon Sep 17 00:00:00 2001 From: huanxiaoling <3174348550@qq.com> Date: Tue, 27 Sep 2022 19:09:16 +0800 Subject: [PATCH] update the files in migration guide folder --- docs/mindspore/source_en/index.rst | 10 +- .../migration_guide/enveriment_preparation.md | 76 ++ .../source_en/migration_guide/faq.md | 27 + .../source_en/migration_guide/inference.md | 37 - .../migration_guide/migration_script.md | 468 ---------- .../migration_guide/neural_network_debug.md | 226 ----- .../source_en/migration_guide/overview.md | 47 +- .../source_en/migration_guide/preparation.md | 53 -- .../source_en/migration_guide/sample_code.md | 873 ------------------ .../migration_guide/script_analysis.md | 65 -- .../migration_guide/script_development.rst | 14 - .../migration_guide/use_third_party_op.md | 94 +- docs/mindspore/source_zh_cn/index.rst | 1 + .../migration_guide/use_third_party_op.md | 4 +- 14 files changed, 180 insertions(+), 1815 deletions(-) create mode 100644 docs/mindspore/source_en/migration_guide/enveriment_preparation.md create mode 100644 docs/mindspore/source_en/migration_guide/faq.md delete mode 100644 docs/mindspore/source_en/migration_guide/inference.md delete mode 100644 docs/mindspore/source_en/migration_guide/migration_script.md delete mode 100644 docs/mindspore/source_en/migration_guide/neural_network_debug.md delete mode 100644 docs/mindspore/source_en/migration_guide/preparation.md delete mode 100644 docs/mindspore/source_en/migration_guide/sample_code.md delete mode 100644 docs/mindspore/source_en/migration_guide/script_analysis.md delete mode 100644 docs/mindspore/source_en/migration_guide/script_development.rst diff --git a/docs/mindspore/source_en/index.rst b/docs/mindspore/source_en/index.rst index 14ac6a9327..6a3f94bc64 100644 --- a/docs/mindspore/source_en/index.rst +++ b/docs/mindspore/source_en/index.rst @@ -75,12 +75,10 @@ MindSpore Documentation :hidden: migration_guide/overview - migration_guide/preparation - migration_guide/script_analysis - migration_guide/script_development - migration_guide/neural_network_debug - migration_guide/inference - migration_guide/sample_code + migration_guide/enveriment_preparation + migration_guide/model_development/model_development + migration_guide/faq + migration_guide/use_third_party_op .. toctree:: :glob: diff --git a/docs/mindspore/source_en/migration_guide/enveriment_preparation.md b/docs/mindspore/source_en/migration_guide/enveriment_preparation.md new file mode 100644 index 0000000000..25483ac57f --- /dev/null +++ b/docs/mindspore/source_en/migration_guide/enveriment_preparation.md @@ -0,0 +1,76 @@ +# Environment Preparation and Information Acquisition + + + +## Overview + +Network migration starts with the configuration of the MindSpore development environment. This chapter describes in detail the installation process and related materials. The materials include a basic introduction to the MindSpore components models and Hub, including their uses, scenarios and usage. There are also tutorials on training on the cloud: using ModelArts to adapt scripts, uploading datasets in OBS, training online, etc. + +## Installing MindSpore + +[MindSpore](https://www.mindspore.cn/tutorials/en/master/beginner/introduction.html) is a full-scene deep learning framework, which currently supports running on [Ascend](https://e.huawei.com/cn/products/servers/ascend), GPU, CPU and other kinds of devices. Before installing MindSpore on Ascend and GPU, you need to configure the corresponding runtime environment. + +> MindSpore Ascend supports AI training (910), inference cards (310 and 310P) and training servers on various Atlas series chips, etc. Note that MindSpore version needs to be used with Ascend AI processor package. For example, MindSpore 1.8.1 must be used with Ascend package commercial version 22.0.RC2 or CANN Community version 5.1.RC2.alpha008. There may be problems with other versions. Please refer to the "Installing Ascend AI Processor Package" section in the MindSpore Ascend Version Installation Guide for more details. +> +> The MindSpore GPU supports CUDA 10.1 and CUDA 11.1 on Linux environments. NVIDIA provides a variety of installation methods and installation instructions, which can be found on the [CUDA download page](https://developer.nvidia.com/cuda-toolkit-archive) and and [CUDA Installation Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html). +> +> MindSpore CPU currently supports Linux, Windows and Mac. + +After the package is installed, follow the MindSpore installation guide and install the corresponding version of MindSpore to experience. + +Refer to the following table to determine the release version, architecture (X86 or Arm), and Python version. + +|System|Query Content|Query commands| +|:----|:----|:----| +|Linux|Release Version| `cat /proc/version`| +|Linux|Architecture| `uname -m`| +|Linux|Python Version| `python3`| + +According to the operating system and computing hardware you are using, choose the corresponding MindSpore version and install MindSpore by Pip, Conda, Docker or source code compilation, etc. We recommend visiting [MindSpore Installation](https://www.mindspore.cn/install/en) and referring to the website for instructions to complete the installation and verification. + +## models and hub + +[MindSpore models](https://gitee.com/mindspore/models) is a deep-optimized model bazaar jointly provided by MindSpore and the community, which provides developers with deep-optimized models. The ecological partners can easily personalize their development based on the models in ModelZoo. Currently it has covered mainstream models in many fields such as machine vision, natural language processing, speech, recommendation systems. + +At present, there are 300+ model implementations, among which the network under the official directory is the official network, with some optimization for the model implementation. Most of the models under the research directory are the models of Zongzhi, with certain guarantee of accuracy and performance. The community directory is the code contributed by developers, which has not been maintained yet and is for reference only. + +[mindspore Hub](https://www.mindspore.cn/resources/hub/en) is a platform for storing pre-trained models provided by MindSpore official or third-party developers. It provides easy-to-use model loading and fine-tuning APIs to application developers, enabling users to make inference or fine-tuning based on pre-trained models and deploy them to their own applications. Users can also follow the specified steps on [Publish Model](https://www.mindspore.cn/hub/docs/en/master/publish_model.html), publishing their trained models into MindSpore Hub for other users to download and use. + +The [Download Center](https://download.mindspore.cn/model_zoo/) saves the parameter files of the trained models in the models bin, where users can download the corresponding parameter files for development. + +## ModelArts + +### Development Environment Introduction + +ModelArts is a one-stop development platform for AI developers provided by HUAWEI Cloud, which contains Ascend resource pool. Users can experience MindSpore in this platform. For related document, refer to [AI Platform ModelArts](https://support.huaweicloud.com/wtsnew-modelarts/index.html). + +### Development Environment and Training Environment + +**Development environment** mainly refers to Notebook's development environment, which is mainly used for code writing and debugging and has almost the same development experience as offline, but the number of machines and cores is relatively small and the usage time is limited. + +![notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/source_zh_cn/migration_guide/images/modelart_notebook.png "Development environment") + +**Training environment** is the primary production environment on the cloud with more machines and cores for large clusters and batch tasks. + +### Introduction to Development Environment Storage Method + +The storage methods supported by the development environment are OBS and EFS, as shown in the following figure. + +![ModelArts](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/source_zh_cn/migration_guide/images/modelart.png "Development Environment Storage Method") + +OBS: also called S3 bucket. The code, data, and pre-trained models stored on the OBS need to be transferred to the corresponding physical machine first in the development environment and training environment before the job is executed. [Upload local files to OBS](https://bbs.huaweicloud.com/blogs/212453). + +[MoXing](https://bbs.huaweicloud.com/blogs/101129): MoXing is a network model development API provided by Huawei Cloud Deep Learning Service, the use of which needs to focus on the data copy interface. + +```python +import moxing as mox +mox.file.copy_parallel(src_path, target_path) # Copy the data from the OBS bucket to the physical machine where it is actually executed or vice versa +``` + +EFS: It can be understood as a mountable cloud disk, which can be directly mounted to the corresponding physical machine in the development environment and training environment to facilitate the execution of jobs. + +Welcome to click the video link below. + +
+ +
\ No newline at end of file diff --git a/docs/mindspore/source_en/migration_guide/faq.md b/docs/mindspore/source_en/migration_guide/faq.md new file mode 100644 index 0000000000..8ea37ff026 --- /dev/null +++ b/docs/mindspore/source_en/migration_guide/faq.md @@ -0,0 +1,27 @@ +# FAQ + + + +MindSpore provides a [FAQ](https://mindspore.cn/docs/en/master/faq/installation.html) in the process of using MindSpore. This chapter also collates the solutions to the set of common problems mentioned in the migration documentation. + +- Web Scripts + + [Typical Differences from PyTorch](https://www.mindspore.cn/docs/en/master/migration_guide/typical_api_comparision.html) + + [API Mapping and Handling Strategy of Missing API](https://www.mindspore.cn/docs/en/master/migration_guide/analysis_and_preparation.html#analyze_api_satisfaction) + + [Dynamic Shape Analysis](https://www.mindspore.cn/docs/en/master/migration_guide/analysis_and_preparation.html#dynamic_shape) and [Mitigation Program](https://www.mindspore.cn/docs/en/master/migration_guide/model_development/model_and_loss.html#dynamic_shape_mitigation_strategy) + + [Mitigation Program for Sparse Characteristic](https://www.mindspore.cn/docs/en/master/migration_guide/analysis_and_preparation.html#sparse) + + [Common Syntax Restrictions and Handling Strategies for Static Graphs](https://www.mindspore.cn/docs/en/master/migration_guide/model_development/model_and_loss.html#common_restrictions) + + [Notes for MindSpore Web Authoring](https://www.mindspore.cn/docs/en/master/migration_guide/model_development/model_development.html#notes_for_mindspore_web_authoring) + +- Network Debugging + + [Function Debugging](https://www.mindspore.cn/docs/en/master/migration_guide/debug_and_tune.html#function_debugging) + + [Precision Debugging](https://www.mindspore.cn/docs/zh-CN/master/migration_guide/debug_and_tune.html#precision_debugging) + + [Performance Debugging](https://www.mindspore.cn/docs/zh-CN/master/migration_guide/debug_and_tune.html#performance_debugging) diff --git a/docs/mindspore/source_en/migration_guide/inference.md b/docs/mindspore/source_en/migration_guide/inference.md deleted file mode 100644 index 8d92e8aa8f..0000000000 --- a/docs/mindspore/source_en/migration_guide/inference.md +++ /dev/null @@ -1,37 +0,0 @@ -# Inference Execution - -Translator: [Dongdong92](https://gitee.com/zy179280) - - - -For trained models, MindSpore can execute inference tasks on different hardware platforms. MindSpore also provides online inference services based on MindSpore Serving. - -## Inference Service Based on Models - -### Overview - -MindSpore supports to save [training parameters files](https://www.mindspore.cn/tutorials/experts/en/master/infer/inference.html#model-files) as CheckPoint format. MindSpore also supports to save [network model files](https://www.mindspore.cn/tutorials/experts/en/master/infer/inference.html#model-files) as MindIR, AIR, and ONNX. - -Referring to the [executing inference](https://www.mindspore.cn/tutorials/experts/en/master/infer/inference.html#inference-execution) section, users not only can execute local inference through `mindspore.model.predict` interface, but also can export MindIR, AIR, and ONNX model files through `mindspore.export` for inference on different hardware platforms. - -For dominating the difference between backend models, model files in the [MindIR format](https://www.mindspore.cn/tutorials/experts/en/master/infer/inference.html#inference-execution) is recommended. MindIR model files can be executed on different hardware platforms, as well as be deployed to the Serving platform on cloud and the Lite platform on device. - -### Executing Inference on Different Platforms - -- For the Ascend hardware platform, please refer to [Inference on the Ascend 910 AI processor](https://www.mindspore.cn/tutorials/experts/en/master/infer/ascend_910_mindir.html) and [Inference on Ascend 310](https://www.mindspore.cn/tutorials/experts/en/master/infer/ascend_310_mindir.html). -- For the GPU/CPU hardware platform, please refer to [Inference on a GPU/CPU](https://www.mindspore.cn/tutorials/experts/en/master/infer/cpu_gpu_mindir.html). -- For inference on the Lite platform on device, please refer to [on-device inference](https://www.mindspore.cn/lite/docs/en/master/index.html). - -> Please refer to [MindSpore C++ Library Use](https://www.mindspore.cn/docs/en/master/faq/inference.html) to solve the interface issues on the Ascend hardware platform. - -## On-line Inference Service Deployment Based on MindSpore Serving - -MindSpore Serving is a lite and high-performance service module, aiming at assisting MindSpore developers in efficiently deploying on-line inference services in the production environment. When a user completes the training task by using MindSpore, the trained model can be exported for inference service deployment via MindSpore Serving. Please refer to the following examples for deployment: - -- [MindSpore Serving-based Inference Service Deployment](https://www.mindspore.cn/serving/docs/en/master/serving_example.html). -- [gRPC-based MindSpore Serving Access](https://www.mindspore.cn/serving/docs/en/master/serving_grpc.html). -- [RESTful-based MindSpore Serving Access](https://www.mindspore.cn/serving/docs/en/master/serving_restful.html). -- [Servable Provided Through Model Configuration](https://www.mindspore.cn/serving/docs/en/master/serving_model.html). -- [MindSpore Serving-based Distributed Inference Service Deployment](https://www.mindspore.cn/serving/docs/en/master/serving_distributed_example.html). - -> For deployment issues regarding the on-line inference service, please refer to [MindSpore Serving Class](https://www.mindspore.cn/serving/docs/en/master/faq.html). diff --git a/docs/mindspore/source_en/migration_guide/migration_script.md b/docs/mindspore/source_en/migration_guide/migration_script.md deleted file mode 100644 index 1fa556d67a..0000000000 --- a/docs/mindspore/source_en/migration_guide/migration_script.md +++ /dev/null @@ -1,468 +0,0 @@ -# Migration Script - -Translator: [zhangxiaoxiao](https://gitee.com/zhangxiaoxiao16) - - - -## Overview - -This document describes how to migrate network scripts from the TensorFlow or PyTorch framework to MindSpore。 - -## Migrating the TensorFlow Script to MindSpore - -Migrate scripts by reading the TensorBoard graphs。 - -1. The [PoseNet](https://arxiv.org/pdf/1505.07427v4.pdf) implemented by TensorFlow is used as an example to show how to use TensorBoard to read graphs, write MindSpore code, and migrate [TensorFlow Models](https://github.com/kentsommer/tensorflow-posenet) to MindSpore. - - > The PoseNet code mentioned here is based on Python2. You need to make some syntax changes to run on Python3. Details are not described here. - -2. Rewrite the code, use `tf.summary` interface, save the log required by TensorBoard, start TensorBoard. - -3. The following figure shows the opened TensorBoard, and it's for reference only. This may vary depending on how the log is generated. The figure displayed on TensorBoard may vary in the log generation mode. - - ![PoseNet TensorBoard](images/pic1.png) - -4. Find the Placeholder of three inputs, view the figure and read the code, and the second and third inputs are used only for loss calculation. - - ![PoseNet Placeholder](images/pic3.png) - - ![PoseNet Placeholder_1 Placeholder_2](images/pic2.png) - - ![PoseNet script input1 2 3](images/pic4.png) - - So far, we can preliminarily follow three steps to construct a network model: - - Step 1, the first input will compute six outputs in the backbone in the three inputs of the network. - - Step 2, the result of step 1, the second and third inputs are used to calculate the loss in the loss subnet. - - Step 3, construct the reverse network by using `TrainOneStepCell` automatic differentiation. Use the Adam optimizer and attributes provided by TensorFlow to write the corresponding MindSpore optimizer to update parameters. The network backbone can write as follows: - - ```python - import mindspore - from mindspore import nn - from mindspore.nn import TrainOneStepCell - from mindspore.nn import Adam - - # combine backbone and loss - class PoseNetLossCell(nn.Cell): - def __init__(self, backbone, loss): - super(PoseNetLossCell, self).__init__() - self.pose_net = backbone - self.loss = loss - def construct(self, input_1, input_2, input_3): - p1_x, p1_q, p2_x, p2_q, p3_x, p3_q = self.poss_net(input_1) - loss = self.loss(p1_x, p1_q, p2_x, p2_q, p3_x, p3_q, input_2, input_3) - return loss - - # define backbone - class PoseNet(nn.Cell): - def __init__(self): - super(PoseNet, self).__init__() - def construct(self, input_1): - """do something with input_1, output num 6""" - return p1_x, p1_q, p2_x, p2_q, p3_x, p3_q - - # define loss - class PoseNetLoss(nn.Cell): - def __init__(self): - super(PoseNetLoss, self).__init__() - - def construct(self, p1_x, p1_q, p2_x, p2_q, p3_x, p3_q, poses_x, poses_q): - """do something to calc loss""" - return loss - - # define network - backbone = PoseNet() - loss = PoseNetLoss() - net_with_loss = PoseNetLossCell(backbone, loss) - opt = Adam(net_with_loss.trainable_params(), learning_rate=0.001, beta1=0.9, beta2=0.999, eps=1e-08, use_locking=False) - net_with_grad = TrainOneStepCell(net_with_loss, opt) - ``` - -5. Next, let's implement the computing logic in the backbone. - - The first input passes through a subgraph named conv1, and the computing logic can be obtained by looking at the following figure: - - ![PoseNet conv1](images/pic5.png) - - input->Conv2D->BiasAdd->ReLU, although the operator name after BiasAdd is conv1, it actually executes ReLU. - - ![PoseNet Conv1 conv1 relu](images/pic6.png) - - Then, the first subgraph conv1 can be defined as follows. Specific parameters are aligned with those in the original project: - - ```python - class Conv1(nn.Cell): - def __init__(self): - super(Conv1, self).__init__() - self.conv = Conv2d() - self.relu = ReLU() - def construct(self, x): - x = self.conv(x) - x = self.relu(x) - return x - ``` - - By observing the TensorBoard figure and code, we find that the subnet of the conv type defined in the original TensorFlow project can be copied as the subnet of MindSpore to reduce repeated code. - - TensorFlow project conv subnet definition: - - ```python - def conv(self, - input, - k_h, - k_w, - c_o, - s_h, - s_w, - name, - relu=True, - padding=DEFAULT_PADDING, - group=1, - biased=True): - # Verify that the padding is acceptable - self.validate_padding(padding) - # Get the number of channels in the input - c_i = input.get_shape()[-1] - # Verify that the grouping parameter is valid - assert c_i % group == 0 - assert c_o % group == 0 - # Convolution for a given input and kernel - convolve = lambda i, k: tf.nn.conv2d(i, k, [1, s_h, s_w, 1], padding=padding) - with tf.variable_scope(name) as scope: - kernel = self.make_var('weights', shape=[k_h, k_w, c_i / group, c_o]) - if group == 1: - # This is the common-case. Convolve the input without any further complications. - output = convolve(input, kernel) - else: - # Split the input into groups and then convolve each of them independently - input_groups = tf.split(3, group, input) - kernel_groups = tf.split(3, group, kernel) - output_groups = [convolve(i, k) for i, k in zip(input_groups, kernel_groups)] - # Concatenate the groups - output = tf.concat(3, output_groups) - # Add the biases - if biased: - biases = self.make_var('biases', [c_o]) - output = tf.nn.bias_add(output, biases) - if relu: - # ReLU non-linearity - output = tf.nn.relu(output, name=scope.name) - return output - ``` - - The MindSpore subnet is defined as follows: - - ```python - from mindspore import nn - from mindspore.nn import Conv2d, ReLU - - class ConvReLU(nn.Cell): - def __init__(self, channel_in, kernel_size, channel_out, strides): - super(ConvReLU, self).__init__() - self.conv = Conv2d(channel_in, channel_out, kernel_size, strides, has_bias=True) - self.relu = ReLU() - - def construct(self, x): - x = self.conv(x) - x = self.relu(x) - return x - ``` - - Based on the data flow direction and operator attributes in the TensorBoard, the backbone computing logic can be written as follows: - - ```python - from mindspore.nn import MaxPool2d - import mindspore.ops as ops - - - class LRN(nn.Cell): - def __init__(self, radius, alpha, beta, bias=1.0): - super(LRN, self).__init__() - self.lrn = ops.LRN(radius, bias, alpha, beta) - def construct(self, x): - return self.lrn(x) - - - class PoseNet(nn.Cell): - def __init__(self): - super(PoseNet, self).__init__() - self.conv1 = ConvReLU(3, 7, 64, 2) - self.pool1 = MaxPool2d(3, 2, pad_mode="SAME") - self.norm1 = LRN(2, 2e-05, 0.75) - self.reduction2 = ConvReLU(64, 1, 64, 1) - self.conv2 = ConvReLU(64, 3, 192, 1) - self.norm2 = LRN(2, 2e-05, 0.75) - self.pool2 = MaxPool2d(3, 2, pad_mode="SAME") - self.icp1_reduction1 = ConvReLU(192, 1, 96, 1) - self.icp1_out1 = ConvReLU(96, 3, 128, 1) - self.icp1_reduction2 = ConvReLU(192, 1, 16, 1) - self.icp1_out2 = ConvReLU(16, 5, 32, 1) - self.icp1_pool = MaxPool2d(3, 1, pad_mode="SAME") - self.icp1_out3 = ConvReLU(192, 5, 32, 1) - self.icp1_out0 = ConvReLU(192, 1, 64, 1) - self.concat = ops.Concat(axis=1) - self.icp2_reduction1 = ConvReLU(256, 1, 128, 1) - self.icp2_out1 = ConvReLU(128, 3, 192, 1) - self.icp2_reduction2 = ConvReLU(256, 1, 32, 1) - self.icp2_out2 = ConvReLU(32, 5, 96, 1) - self.icp2_pool = MaxPool2d(3, 1, pad_mode="SAME") - self.icp2_out3 = ConvReLU(256, 1, 64, 1) - self.icp2_out0 = ConvReLU(256, 1, 128, 1) - self.icp3_in = MaxPool2d(3, 2, pad_mode="SAME") - self.icp3_reduction1 = ConvReLU(480, 1, 96, 1) - self.icp3_out1 = ConvReLU(96, 3, 208, 1) - self.icp3_reduction2 = ConvReLU(480, 1, 16, 1) - self.icp3_out2 = ConvReLU(16, 5, 48, 1) - self.icp3_pool = MaxPool2d(3, 1, pad_mode="SAME") - self.icp3_out3 = ConvReLU(480, 1, 64, 1) - self.icp3_out0 = ConvReLU(480, 1, 192, 1) - """etc""" - """...""" - - def construct(self, input_1): - """do something with input_1, output num 6""" - x = self.conv1(input_1) - x = self.pool1(x) - x = self.norm1(x) - x = self.reduction2(x) - x = self.conv2(x) - x = self.norm2(x) - x = self.pool2(x) - pool2 = x - - x = self.icp1_reduction1(x) - x = self.icp1_out1(x) - icp1_out1 = x - - icp1_reduction2 = self.icp1_reduction2(pool2) - icp1_out2 = self.icp1_out2(icp1_reduction2) - - icp1_pool = self.icp1_pool(pool2) - icp1_out3 = self.icp1_out3(icp1_pool) - - icp1_out0 = self.icp1_out0(pool2) - - icp2_in = self.concat((icp1_out0, icp1_out1, icp1_out2, icp1_out3)) - """etc""" - """...""" - - return p1_x, p1_q, p2_x, p2_q, p3_x, p3_q - ``` - - Correspondingly, the loss computing logic may be written as follows: - - ```python - class PoseNetLoss(nn.Cell): - def __init__(self): - super(PoseNetLoss, self).__init__() - self.sub = ops.Sub() - self.square = ops.Square() - self.reduce_sum = ops.ReduceSum() - self.sqrt = ops.Sqrt() - - def construct(self, p1_x, p1_q, p2_x, p2_q, p3_x, p3_q, poses_x, poses_q): - """do something to calc loss""" - l1_x = self.sqrt(self.reduce_sum(self.square(self.sub(p1_x, poses_x)))) * 0.3 - l1_q = self.sqrt(self.reduce_sum(self.square(self.sub(p1_q, poses_q)))) * 150 - l2_x = self.sqrt(self.reduce_sum(self.square(self.sub(p2_x, poses_x)))) * 0.3 - l2_q = self.sqrt(self.reduce_sum(self.square(self.sub(p2_q, poses_q)))) * 150 - l3_x = self.sqrt(self.reduce_sum(self.square(self.sub(p3_x, poses_x)))) * 1 - l3_q = self.sqrt(self.reduce_sum(self.square(self.sub(p3_q, poses_q)))) * 500 - return l1_x + l1_q + l2_x + l2_q + l3_x + l3_q - ``` - - Ultimately, your training script should look like this: - - ```python - if __name__ == "__main__": - backbone = PoseNet() - loss = PoseNetLoss() - net_with_loss = PoseNetLossCell(backbone, loss) - opt = Adam(net_with_loss.trainable_params(), learning_rate=0.001, beta1=0.9, beta2=0.999, eps=1e-08, use_locking=False) - net_with_grad = TrainOneStepCell(net_with_loss, opt) - """dataset define""" - model = Model(net_with_grad) - model.train(epoch_size, dataset) - ``` - - In this way, the model script is basically migrated from TensorFlow to MindSpore. Then, various MindSpore tools and computing policies are used to optimize the precision. - -## Migrating the PyTorch Script to MindSpore - -Read the PyTorch script to migrate directly. - -1. The PyTorch subnet module usually inherits `torch.nn.Module`, and MindSpore usually inherits `mindspore.nn.Cell`. The forward method needs to be re-written for the forward calculation logic of the PyTorch subnet module, and the construct method needs to be rewritten for the forward calculation logic of the MindSpore subnet module. - -2. Take the migration of the Bottleneck class in MindSpore as an example. - - PyTorch project code - - ```python - # defined in PyTorch - class Bottleneck(nn.Module): - def __init__(self, inplanes, planes, stride=1, mode='NORM', k=1, dilation=1): - super(Bottleneck, self).__init__() - self.mode = mode - self.relu = nn.ReLU(inplace=True) - self.k = k - - btnk_ch = planes // 4 - self.bn1 = nn.BatchNorm2d(inplanes) - self.conv1 = nn.Conv2d(inplanes, btnk_ch, kernel_size=1, bias=False) - - self.bn2 = nn.BatchNorm2d(btnk_ch) - self.conv2 = nn.Conv2d(btnk_ch, btnk_ch, kernel_size=3, stride=stride, padding=dilation, - dilation=dilation, bias=False) - - self.bn3 = nn.BatchNorm2d(btnk_ch) - self.conv3 = nn.Conv2d(btnk_ch, planes, kernel_size=1, bias=False) - - if mode == 'UP': - self.shortcut = None - elif inplanes != planes or stride > 1: - self.shortcut = nn.Sequential( - nn.BatchNorm2d(inplanes), - self.relu, - nn.Conv2d(inplanes, planes, kernel_size=1, stride=stride, bias=False) - ) - else: - self.shortcut = None - - def _pre_act_forward(self, x): - residual = x - - out = self.bn1(x) - out = self.relu(out) - out = self.conv1(out) - - out = self.bn2(out) - out = self.relu(out) - out = self.conv2(out) - - out = self.bn3(out) - out = self.relu(out) - out = self.conv3(out) - - if self.mode == 'UP': - residual = self.squeeze_idt(x) - elif self.shortcut is not None: - residual = self.shortcut(residual) - - out += residual - - return out - - def squeeze_idt(self, idt): - n, c, h, w = idt.size() - return idt.view(n, c // self.k, self.k, h, w).sum(2) - - def forward(self, x): - out = self._pre_act_forward(x) - return out - - ``` - - Based on the differences in defining convolution parameters between PyTorch and MindSpore, the definition can be translated as follows: - - ```python - from mindspore import nn - import mindspore.ops as ops - - # defined in MindSpore - class Bottleneck(nn.Cell): - def __init__(self, inplanes, planes, stride=1, k=1, dilation=1): - super(Bottleneck, self).__init__() - self.mode = mode - self.relu = nn.ReLU() - self.k = k - - btnk_ch = planes // 4 - self.bn1 = nn.BatchNorm2d(num_features=inplanes, momentum=0.9) - self.conv1 = nn.Conv2d(in_channels=inplanes, out_channels=btnk_ch, kernel_size=1, pad_mode='pad', has_bias=False) - - self.bn2 = nn.BatchNorm2d(num_features=btnk_ch, momentum=0.9) - self.conv2 = nn.Conv2d(in_channels=btnk_ch, out_channels=btnk_ch, kernel_size=3, stride=stride, pad_mode='pad', padding=dilation, dilation=dilation, has_bias=False) - - self.bn3 = nn.BatchNorm2d(num_features=btnk_ch, momentum=0.9) - self.conv3 = nn.Conv2d(in_channels=btnk_ch, out_channels=planes, kernel_size=1, pad_mode='pad', has_bias=False) - - self.shape = ops.Shape() - self.reshape = ops.Reshape() - self.reduce_sum = ops.ReduceSum() - - if mode == 'UP': - self.shortcut = None - elif inplanes != planes or stride > 1: - self.shortcut = nn.SequentialCell([ - nn.BatchNorm2d(num_features=inplanes, momentum=0.9), - nn.ReLU(), - nn.Conv2d(in_channels=inplanes, out_channels=planes, kernel_size=1, stride=stride, pad_mode='pad', has_bias=False) - ]) - else: - self.shortcut = None - - def _pre_act_forward(self, x): - residual = x - - out = self.bn1(x) - out = self.relu(out) - out = self.conv1(out) - - out = self.bn2(out) - out = self.relu(out) - out = self.conv2(out) - - out = self.bn3(out) - out = self.relu(out) - out = self.conv3(out) - - if self.shortcut is not None: - residual = self.shortcut(residual) - - out += residual - return out - - def construct(self, x): - out = self._pre_act_forward(x) - return out - ``` - -3. PyTorch backpropagation is usually implemented by `loss.backward()`, and parameter update is implemented by `optimizer.step()`. In MindSpore, these parameters do not need to be explicitly invoked by the user and can be transferred to the `TrainOneStepCell` class for backpropagation and gradient update. Finally, the training script should look like this: - - ```python - # define dataset - dataset = ... - - # define backbone and loss - backbone = Net() - loss = NetLoss() - - # combine backbone and loss - net_with_loss = WithLossCell(backbone, loss) - - # define optimizer - opt = ... - - # combine forward and backward - net_with_grad = TrainOneStepCell(net_with_loss, opt) - - # define model and train - model = Model(net_with_grad) - model.train(epoch_size, dataset) - ``` - -PyTorch and MindSpore have similar definitions of some basic APIs, such as [mindspore.nn.SequentialCell](https://www.mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.SequentialCell.html#mindspore.nn.SequentialCell) and [torch.nn.Sequential](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html#torch.nn.Sequential). In addition, some operator APIs may be not the same. This section lists some common API comparisons. For more information, see the [MindSpore and PyTorch API mapping](https://www.mindspore.cn/docs/en/master/note/api_mapping/pytorch_api_mapping.html) on MindSpore's official website. - -| PyTorch | MindSpore | -| :-------------------------------: | :------------------------------------------------: | -| tensor.view() | mindspore.ops.operations.Reshape()(tensor) | -| tensor.size() | mindspore.ops.operations.Shape()(tensor) | -| tensor.sum(axis) | mindspore.ops.operations.ReduceSum()(tensor, axis) | -| torch.nn.Upsample[mode: nearest] | mindspore.ops.operations.ResizeNearestNeighbor | -| torch.nn.Upsample[mode: bilinear] | mindspore.ops.operations.ResizeBilinear | -| torch.nn.Linear | mindspore.nn.Dense | -| torch.nn.PixelShuffle | mindspore.ops.operations.DepthToSpace | - -It should be noticed that although `torch.nn.MaxPool2d` and `mindspore.nn.MaxPool2d` are similar in interface definition, and MindSpore actually invokes the `MaxPoolWithArgMax` operator during training on Ascend. The function of this operator is the same as that of TensorFlow, during the migration, and the MindSpore output after the MaxPool layer is inconsistent with that of PyTorch. Theoretically, it's not affect the final training result. diff --git a/docs/mindspore/source_en/migration_guide/neural_network_debug.md b/docs/mindspore/source_en/migration_guide/neural_network_debug.md deleted file mode 100644 index 91f5dac082..0000000000 --- a/docs/mindspore/source_en/migration_guide/neural_network_debug.md +++ /dev/null @@ -1,226 +0,0 @@ -# Network Debugging - -Translator: [Soleil](https://gitee.com/deng-zhihua) - - - -This chapter introduces the basic principles and common tools of Network Debugging, as well as some solutions to some common problems. - -## The Basic Process of Network Debugging - -The process of Network Debugging is divided into the following steps： - -1. The network process is successfully debugged with no error during the whole process of network execution, proper output of loss value, and normal completion of parameter update. - - In general, if you use the `model.train` interface to execute a step completely without receiving an error, it means that it is executed normally and completed the parameter update; if you need to confirm accurately, you can save the checkpoint file for two consecutive steps by using the parameter `save_checkpoint_ steps=1` in `mindspore.CheckpointConfig`, or use the `save_checkpoint` interface to save the Checkpoint file directly, and then print the weight values in the Checkpoint file with the following code to see if the weights of the two steps have changed. Finally, the update is completed. - - ```python - import mindspore - import numpy as np - ckpt = mindspore.load_checkpoint(ckpt_path) - for param in ckpt: - value = ckpt[param].data.asnumpy() - print(value) - ``` - -2. Multiple iterations of the network are executed to output the loss values, and the loss values have a basic convergence trend. - -3. Network accuracy debugging and hyper-parameter optimization. - -## Common Methods Used in Network Debugging - -### Process Debugging - -This section introduces the problems and solutions during Network Debugging process after the script development is generally completed. - -#### Process Debugging with PyNative Mode - -For script development and network process debugging, we recommend using the PyNative mode for debugging. The PyNative mode supports executing single operators, normal functions and networks, as well as separate operations for computing gradients. In PyNative mode, you can easily set breakpoints and get intermediate results of network execution, and you can also debug the network by means of pdb. - -By default, MindSpore is in Graph mode, which can be set as PyNative mode via `set_context(mode=PYNATIVE_MODE)`. Related examples can be found in [Debugging With PyNative Mode](https://www.mindspore.cn/tutorials/zh-CN/master/advanced/compute_graph/pynative.html). - -#### Getting More Error Messages - -During the network process debugging, if you need to get more information about error messages, you can get it by the following ways: - -- Using pdb for debugging in PyNative mode, and using pdb to print relevant stack and contextual information to help locate problems. -- Using Print operator to print more contextual information. Related examples can be found in [Print Operator Features](https://www.mindspore.cn/tutorials/experts/en/master/debug/custom_debug.html#mindspore-print-operator-introduction). -- Adjusting the log level to get more error information. MindSpore can easily adjust the log level through environment variables. Related examples can be found in [Logging-related Environment Variables And Configurations](https://www.mindspore.cn/tutorials/experts/en/master/debug/custom_debug.html#log-related-environment-variables-and-configurations). - -#### Common Errors - -During network process debugging, the common errors are as follows: - -- The operator execution error. - - During the network process debugging, operator execution errors are often reported such as shape mismatch and unsupported dtype. Then, according to the error message, you should check whether the operator is used correctly and whether the shape of the input data is consistent with the expectation and make corresponding modifications. - - Supports for related operators and API introductions can be found in [Operator Support List](https://www.mindspore.cn/docs/en/master/note/operator_list.html) and [Operators Python API](https://www.mindspore.cn/docs/en/master/index.html). - -- The same script works in PyNative mode, but reports bugs in Graph mode. - - In MindSpore's Graph mode, the code in the `construct` function is parsed by the MindSpore framework, and there is some Python syntax that is not yet supported which results in errors. In this case, you should follow [MindSpore's Syntax Description](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html) according to the error message. - -- Distributed parallel training script is misconfigured. - - Distributed parallel training scripts and environment configuration can be found in [Distributed Parallel Training Tutorial](https://www.mindspore.cn/docs/en/master/design/distributed_training_design.html). - -### Loss Value Comparison - -With a benchmark script, the loss values run by the benchmark script can be compared with those run by the MindSpore script, which can be used to verify the correctness of the overall network structure and the accuracy of the operator. - -#### Main Steps - -1. Guaranteeting Identical Input - - It is necessary to ensure that the inputs are the same in both networks, so that they can have the same network output in the same network structure. The same inputs can be guaranteed using following ways: - - - Using numpy to construct the input data to ensure the same inputs to the networks. MindSpore supports free conversion of Tensor and numpy. The following script can be used to construct the input data. - - ```python - input = Tensor(np.random.randint(0, 10, size=(3, 5, 10)).astype(np.float32)) - ``` - - - Using the same dataset for computation. MindSpore supports the use of the TFRecord dataset, which can be read by using the `mindspore.dataset.TFRecordDataset` interface. - -2. Removing the Influence of Randomness in the Network - - The main methods to remove the effect of randomness in the network are to set the same randomness seed, turn off the data shuffle, modify the code to remove the effect of random operators in the network such as dropout and initializer, etc. - -3. Ensuring the Same Settings for the Relevant Hyperparameters - - It is necessary to ensure the same settings for the hyperparameters in the network in order to guarantee the same input and the same output of the operator. - -4. Running the network and comparing the output loss values. Generally, the error of the loss value is about 1‰. Because the operator itself has a certain accuracy error. As the number of steps increases, the error will have a certain accumulation. - -#### Related Issues Locating - -If the loss errors are large, the problem locating can be done by using following ways: - -- Checking whether the input and hyperparameter settings are the same, and whether the randomness effect is completely removed. - - if the loss value differs significantly after multiple executions of the same script, it means that the effect of randomness in the network is not completely removed. - -- Overall judgment. - - If there is a large error in the first iteration of loss values, it means that there is a problem with the forward calculation of the network. - - If the loss value in the first iteration is within the error range but the second iteration starts with a large error in the loss value, it means that there should be no problem in the forward calculation of the network and there may be problems in the reverse gradient and weight update calculation. - -- After having the overall judgment, compare the accuracy of input and output values from rough to fine. - - First, compare the input and output values layer by layer for each subnet starting from the input, and identify the subnets that initially have problems. - - Then, compare the network structure in the subnet and the input and output of the operator, find the network structure or operator that occurs problems, and modify it. - - If you find any operator accuracy problems during the process, you can raise an issue on the [MindSpore Code Hosting Platform](https://gitee.com/mindspore/mindspore), and the relevant personnel will follow up on the problem. - -- MindSpore provides various tools for acquiring intermediate network data, which can be used according to the actual situation. - - - [Data Dump function](https://www.mindspore.cn/tutorials/experts/en/master/debug/dump.html#dump-introduction) - - [Use Print Operator To Print Related Information](https://www.mindspore.cn/tutorials/experts/en/master/debug/custom_debug.html#mindspore-print-operator-introduction) - - [Using The Visualization Component MindInsight](https://www.mindspore.cn/mindinsight/docs/en/master/index.html) - -### Precision Debugging Tools - -#### Customized Debugging Information - -- [Callback Function](https://www.mindspore.cn/tutorials/experts/en/master/debug/custom_debug.html#callback-capabilities-of-mindspore) - - MindSpore has provided ModelCheckpoint, LossMonitor, SummaryCollector and other Callback classes for saving model parameters, monitoring loss values, saving training process information, etc. Users can also customize Callback functions to implement starting and ending runs at each epoch and step, and please refer to [Custom Callback](https://www.mindspore.cn/tutorials/experts/en/master/debug/custom_debug.html#custom-callback) for specific examples. - -- [MindSpore Metrics Function](https://www.mindspore.cn/tutorials/experts/en/master/debug/custom_debug.html#mindspore-metrics-introduction) - - When the training is finished, metrics can be used to evaluate the training results. MindSpore provides various metrics for evaluation, such as: `accuracy`, `loss`, `precision`, `recall`, `F1`, etc. - -- Customized Learning Rate - - MindSpore provides some common implementations of dynamic learning rate and some common optimizers with adaptive learning rate adjustment functions, and [Dynamic Learning Rate](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html#dynamic-learning-rate) and [Optimizer Functions](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html#optimizer) in the API documentation can be found. - - At the same time, the user can implement a customized dynamic learning rate, as exemplified by WarmUpLR: - - ```python - class WarmUpLR(LearningRateSchedule): - def __init__(self, learning_rate, warmup_steps): - super(WarmUpLR, self).__init__() - ## check the input - if not isinstance(learning_rate, float): - raise TypeError("learning_rate must be float.") - validator.check_non_negative_float(learning_rate, "learning_rate", self.cls_name) - validator.check_positive_int(warmup_steps, 'warmup_steps', self.cls_name) - ## define the operators - self.warmup_steps = warmup_steps - self.learning_rate = learning_rate - self.min = ops.Minimum() - self.cast = ops.Cast() - - def construct(self, global_step): - ## calculate the lr - warmup_percent = self.cast(self.min(global_step, self.warmup_steps), mstype.float32)/ self.warmup_steps - return self.learning_rate * warmup_percent - ``` - -#### Hyper-Parameter Optimization with MindOptimizer - -MindSpore provides MindOptimizer tools to help users perform hyper-parameter optimization conveniently, and the detailed examples and usage methods can be found in [Hyper-Parameter Optimization With MindOptimizer](https://www.mindspore.cn/mindinsight/docs/en/master/hyper_parameters_auto_tuning.html). - -#### Loss Value Anomaly Locating - -For cases where the loss value is INF, NAN, or the loss value does not converge, you can investigate the following scenarios: - -1. Checking for loss_scale overflow. - - In the scenario of using loss_scale with mixed precision, the situation that the loss value is INF and NAN may be caused by the scale value being too large. If it is dynamic loss_scale, the scale value will be adjusted automatically; if it is static loss_scale, the scale value needs to be reduced. - - If the `scale=1` case still has a loss value of INF, NAN, there should be an overflow of operators in the network and further investigation for locating the problem is needed. - -2. The causes of abnormal loss values may be caused by abnormal input data, operator overflow, gradient disappearance, gradient explosion, etc. - - To check the intermediate value of the network such as operator overflow, gradient of 0, abnormal weight, gradient disappearance and gradient explosion, it is recommended to use [MindInsight Debugger](https://www.mindspore.cn/mindinsight/docs/en/master/debugger.html) to set the corresponding detection points for detection and debugging, which can locate the problem in a more comprehensive way with the strong debuggability. - - The following are a few simple initial troubleshooting methods: - - - Observing whether the weight values appear or loading the saved Checkpoint file to print the weight values can make a preliminary judgment. Printing the weight values can refer to the following code: - - ```python - import mindspore - import numpy as np - ckpt = mindspore.load_checkpoint(ckpt_path) - for param in ckpt: - value = ckpt[param].data.asnumpy() - print(value) - ``` - - - Checking whether the gradient is 0 or comparing whether the weight values of Checkpoint files saved in different steps have changed can make a preliminary judgment. The comparison of the weight values of Checkpoint files can be referred to the following code: - - ```python - import mindspore - import numpy as np - ckpt1 = mindspore.load_checkpoint(ckpt1_path) - ckpt2 = mindspore.load_checkpoint(ckpt2_path) - sum = 0 - same = 0 - for param1,param2 in zip(ckpt1,ckpt2): - sum = sum + 1 - value1 = ckpt[param1].data.asnumpy() - value2 = ckpt[param2].data.asnumpy() - if value1 == value2: - print('same value: ', param1, value1) - same = same + 1 - print('All params num: ', sum) - print('same params num: ', same) - ``` - - - Checking whether there is NAN, INF abnormal data in the weight value, you can also load the Checkpoint file for a brief judgment. In general, if there is NAN, INF in the weight value, there is also NAN, INF in the gradient calculation, and there may be an overflow situation. The relevant code reference is as follows: - - ```python - import mindspore - import numpy as np - ckpt = mindspore.load_checkpoint(ckpt_path) - for param in ckpt: - value = ckpt[param].data.asnumpy() - if np.isnan(value): - print('NAN value:', value) - if np.isinf(value): - print('INF value:', value) - ``` diff --git a/docs/mindspore/source_en/migration_guide/overview.md b/docs/mindspore/source_en/migration_guide/overview.md index 5f72d27cbb..7e67dc83a7 100644 --- a/docs/mindspore/source_en/migration_guide/overview.md +++ b/docs/mindspore/source_en/migration_guide/overview.md @@ -2,44 +2,45 @@ -This migration guide describes the complete steps for migrating neural networks from other machine learning frameworks to MindSpore. +This migration guide contains the complete steps for migrating neural networks to MindSpore from other machine learning frameworks, mainly PyTorch. -To prepare for the migration process, configure the necessary environment and then analyze the operators contained in the network script. MindSpore script development starts from data processing code, uses MindConverter for network building and obtains the migrated network script, and finally migrates the inference execution script. After the build is complete, the optimization process includes development and debugging of missing operators and optimization of network performance and accuracy. The migration guide provides solutions to common problems in the migration process and complete network migration examples. Examples are provided in each chapter for reference. The following figure shows the migration process. +The following figure shows the migration process.: -![flowchart](./images/flowchart.PNG) +1. Configure the MindSpore development environment +2. Analyze the network model to be migrated and acquire basic data +3. MindSpore reproduction. It is recommended to use PYNATIVE mode to debug the model during the functional debugging stage and switch to GRAPH mode after the functional debugging is completed. After the model development is completed, it is recommended to reproduce the inference process first and the training process later. +4. Debugging and tuning for function, precision and performance. -## Preparations +In this process, we have a relatively complete description of each link. We hope that through the migration guide, developers can quickly migrate the existing code of other frameworks to MindSpore. -Before network migration, you need to install MindSpore of the latest version. This chapter describes the installation process and knowledge preparation. The knowledge preparation includes the basic introduction to the functions, scenarios, and usage methods of the MindSpore components ModelZoo and Hub. In addition, there are tutorials related to training on the cloud: using the ModelArts adaptation script, uploading datasets to OBS, and performing online training. +![flowchart](./images/flowchart.PNG "Migration Process") -## Network Script Analysis +## [Environmental Preparation and Information Acquisition](https://www.mindspore.cn/docs/en/master/migration_guide/enveriment_preparation.html) -Before using MindSpore to execute network scripts, you must analyze the scripts of the original framework. This chapter describes how to evaluate operators in the MindSpore framework. Common methods include querying the mapping between MindSpore operators and PyTorch/TensorFlow operators. Another part that needs to be analyzed is syntax evaluation and framework functions, including static graph syntax support. +Network migration starts with configuring the MindSpore development environment, and this chapter describes the installation process and knowledge preparation in detail. The knowledge preparation includes a basic introduction to the MindSpore components models and hub, including the purpose, scenarios and usage. There are also tutorials on training on the cloud: using ModelArts to adapt scripts, uploading datasets in OBS, and training online. -## Network Script Development +## [Model analysis and preparation](https://www.mindspore.cn/docs/en/master/migration_guide/analysis_and_preparation.html) -After the network script analysis is complete, you can use MindSpore to develop a new network. This chapter starts from the data processing function at the network input layer and focuses on the network script build process. The MindConverter or handwritten mapping script is used, and the visualization tool provided by PyTorch/TensorFlow is used to build graphs. The process includes the best practices and common self-check items for development, and lists common troubleshooting methods. +Before doing formal development, some analysis preparation work needs to be done on the network/algorithm to be migrated, including: -## Operator Development and Debugging +- Reading papers and reference codes to understand algorithms and network structures +- Reproducing the results of the paper, obtaining the base model (ckpt), benchmark accuracy and performance +- Analyzing the APIs and functions used in the network. -Some operators are not supported when the network is migrated to the MindSpore framework. In addotiom to the feedback to the MindSpore developer communit, you can develop customized MindSpore operators. This chapter includes tutorials and examples for operator development, as well as common debugging skills. +When migrating networks from PyTorch to MindSpore, users need to be aware of [differences from typical PyTorch interfaces](https://www.mindspore.cn/docs/en/master/migration_guide/typical_api_comparision.html). -## Network Debugging +## [MindSpore model implementation](https://www.mindspore.cn/docs/en/master/migration_guide/model_development/model_development.html) -After the network script is developed and the operator is supplemented, you need to debug the model to ensure that the output result is correct. This chapter describes the common network debugging ideas: single-step debugging and multi-round iterative debugging. Common debugging methods include comparing subnet output results with PyNative mode. MindSpore also supports customized debugging information. At last, the solutions to common problems are provided. +After the preliminary analysis preparation, you can develop the new network by using MindSpore. This chapter will introduce the knowledge of MindSpore network construction and the process of training and inference, starting from the basic modules during inference and training, and using one or two examples to illustrate how to build the network in special scenarios. -## Accuracy and Performance Tuning +## [Debugging and Tuning](https://www.mindspore.cn/docs/en/master/migration_guide/debug_and_tune.html) -After the network script debugging is complete and the result can be successfully output, you need to tune the model to achieve the expected performance. MindSpore provides developers with the Profiler tool which provides easy-to-use and abundant tuning functions in terms of operator performance, iteration performance, and data processing performance, to help users quickly locate and solve performance problems. The tutorials are classified into tuning on the Ascend platform and on the GPU platform, and three examples of using the Profiler tool are provided. +This chapter will introduce some methods of debugging and tuning from three aspects: function, precision and performance. -## Inference Execution +## [Example of Network Migration Debugging](https://www.mindspore.cn/docs/en/master/migration_guide/sample_code.html) -MindSpore can execute inference tasks on different hardware platforms based on trained models and deploy online inference services based on MindSpore Serving. Model-based inference can be performed through the training parameter file or network model file. Online inference based on MindSpore Serving can access the inference service through APIs such as gRPC or RESTful. +This chapter contains a complete network migration sample. From the analysis and replication of the benchmark network, it details the steps of script development and precision debugging and tuning, and finally lists the common problems and corresponding optimization methods during the migration process, framework performance issues. -## Network Migration Debugging Example +## [FAQs](https://www.mindspore.cn/docs/en/master/migration_guide/faq.html) -This chapter provides a complete network migration example. Using ResNet-50 as an example, this chapter describes from how to analyze and reproduce the benchmark network, and how to develop scripts and how to debug and optimize the accuracy. In addition, this chapter lists common problems and corresponding optimization methods during the migration, for example, multi-node synchronization problems and framework performance problems. - -## FAQs - -This chapter lists the frequently asked questions and solutions during network migration. +This chapter lists the frequently-asked questions and corresponding solutions. diff --git a/docs/mindspore/source_en/migration_guide/preparation.md b/docs/mindspore/source_en/migration_guide/preparation.md deleted file mode 100644 index a116b47f70..0000000000 --- a/docs/mindspore/source_en/migration_guide/preparation.md +++ /dev/null @@ -1,53 +0,0 @@ -# Preparation - -Translator: [Misaka19998](https://gitee.com/Misaka19998/docs/tree/master) - - - -## Overview - -Before developing or migrating networks, you need to install MindSpore and learn machine learning knowledge. Users have a choice to buy *Introduction to Deep learning with MindSpore* to learn related knowledge and visit [MindSpore Official website](https://www.mindspore.cn/en) to know how to use MindSpore. - -## Installing MindSpore - -Refer to the following figure, to determine the release version and the architecture(x86 or Arm) of the system, and the Python version. - -| System | Query Content | Query Command | -| ------ | ---------------------- | ------------------- | -| Linux | System Release Version | `cat /proc/version` | -| Linux | System Architecture | `uname -m` | -| Linux | Python Version | `python3` | - -Choose a corresponding MindSpore version based on users own operating system. MindSpore is installed in the manner of Pip, Conda, Docker or source code compilation. It is recommended to visit the [MindSpore installation page](https://www.mindspore.cn), and complete the installation by referring to this website for instructions. - -### Verifying MindSpore - -After the MindSpore is installed, the following commands can be run (taking the MindSpore r1.6 as an example), to test whether the installation of the MindSpore has been completed. - -```python -import mindspore -mindspore.run_check() -``` - -Output the result: - -```text -MindSpore version: 1.6.0 -The result of multiplication calculation is correct, MindSpore has been installed successfully! -``` - -## Knowledge Preparation - -### MindSpore Documentation - -Users can read [MindSpore Tutorials](https://www.mindspore.cn/tutorials/experts/en/master/index.html) to learn how to train, debug, optimize and infer by MindSpore. Users can also see detailed MindSpore interfaces by referring to [MindSpore Python API](https://www.mindspore.cn/docs/en/master/index.html). - -### ModelZoo and Hub - -[ModelZoo](https://gitee.com/mindspore/models/blob/master/README.md#) is a model market of MindSpore and community, which provides deeply-optimized models to developers. In order that the users of MindSpore will have individual development conveniently based on models in ModelZoo. Currently, there are major models in several fields, like computer vision, natural language processing, audio and recommender systems. - -[mindspore Hub](https://www.mindspore.cn/resources/hub/en) is a platform to save pretrained model of official MindSpore or third party developers. It provides some simple and useful APIs for developers to load and finetune models, so that users can infer or tune models based on pretrained models and deploy models to their applications. Users is able to follow some steps to [publish model](https://www.mindspore.cn/hub/docs/en/master/publish_model.html) to MindSpore Hub,for other developers to download and use. - -### Training on the Cloud - -ModelArts is a one-stop development platform for AI developers provided by HUAWEI Cloud, which contains Ascend resource pool. Users can experience MindSpore in this platform and read related document [AI Platform ModelArts](https://support.huaweicloud.com/intl/en-us/wtsnew-modelarts/index.html). diff --git a/docs/mindspore/source_en/migration_guide/sample_code.md b/docs/mindspore/source_en/migration_guide/sample_code.md deleted file mode 100644 index 152ff8efc1..0000000000 --- a/docs/mindspore/source_en/migration_guide/sample_code.md +++ /dev/null @@ -1,873 +0,0 @@ -# Network Migration Debugging Example - -Translator: [AQUA](https://gitee.com/Liu-HongYe) - - - -This chapter will introduce the basic steps of network migration, common tools, ideas for locating problems and solutions with use cases. - -Here we take the classical network ResNet50 as an example and introduce the network migration method in detail with codes. - -## Analysis and Reproduce of the Benchmark Network - -### Determining the Migration Target - -The first step of network migration is to determine the migration goal, that is, first find a proper and achievable standard. Usually the delivery goal of a deep neural network includes the following four parts. - -1. network implementation: this is the most basic part of the migration goal. Sometimes a single neural network may have different versions, a single version may be implemented differently, or a single neural network may adopt different configurations of hyperparameters, and these differences will have some impacts on the final convergence accuracy and performance. Usually, we take the neural network author's own implementation as the standard, but we can also refer to the official implementations of different frameworks (e.g., TensorFlow, PyTorch, etc.) or other mainstream open source toolkits (e.g., MMDetection). -2. dataset: the same neural network and parameters often vary greatly in datasets, so we need to confirm the dataset used for the migration network. The data content of some datasets will be updated frequently, and it is necessary to pay attention to the version of the dataset, the ratio of training data to test data division, etc. when determining the dataset. -3. convergence accuracy: different frameworks, GPU models, and whether the distributed training will have an impact on the accuracy, so we need to analyze the framework, hardware and other information of the benchmark when determining the migration target. -4. training performance: the same as convergence accuracy. Training performance is mainly affected by the network script, framework performance, GPU hardware itself and whether the distributed training is or not. - -#### ResNet50 Migration Example - -ResNet50 is a classic deep neural network in CV, which attracts more developers' attention and replication, and the syntax of PyTorch is more similar to MindSpore, so we choose PyTorch as the benchmark framework. - -The official PyTorch implementation script can be found at [torchvision model](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py) or [Nvidia PyTorch implementation script](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5), which includes implementations of the mainstream ResNet family of networks (ResNet18, ResNet18, ResNet18, ResNet18, and ResNet18). (ResNet18, ResNet34, ResNet50, ResNet101, ResNet152). The dataset used for ResNet50 is ImageNet2012, and the convergence accuracy can be found in [PyTorch Hub](https://pytorch.org/hub/pytorch_vision_resnet/#model-description). - -Developers can run PyTorch-based ResNet50 scripts directly on the benchmark hardware environment and then computes the performance data, or they can refer to the official data on the same hardware environment. For example, when we benchmark the Nvidia DGX-1 32GB (8x V100 32GB) hardware, we can refer to [Nvidia's official ResNet50 performance data](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5#training-performance-nvidia-dgx-1-32gb-8x-v100-32gb). - -### Reproduce the Migration Target - -Once the network migration target is determined, the next thing to do is to reproduce the metrics. Reproducing benchmark data is essential to the subsequent accuracy and performance tuning. When there is an accuracy/performance gap between the network we developed in MindSpore and the benchmark script, we often use the benchmark data as a base line to analyze the difference between the migration script and the benchmark script step by step. If the benchmark script cannot reproduce the metrics, the MindSpore scripts we develop based on the benchmark will not be able to achieve the migration goals. When reproducing migration metrics, it is not only important to reproduce the training phase, but important to reproduce the inference phase. - -It is important to note that for some networks, using the same hardware environment and scripts, the final convergence accuracy and performance may be slightly different from the results presented by the original authors, which is a normal range of fluctuation. The fluctuation should be taken into account when migrating the network. - -### Reproduce the Single Step Results - -The main purpose of reproducing the single Step results is for the next script development and network tuning. For complex neural networks, the complete training takes days or even months, and only if the final training accuracy and results are used as reference, it will greatly reduce the development efficiency. Therefore, we need to reproduce the results of a single Step in advance, i.e., get the state of the network after executing only the first Step (the state is the result after data preprocessing, weight initialization, forward calculation, loss calculation, reverse gradient calculation and optimizer update, covering all aspects of network training), and use it as a reference to start the subsequent development work. - -## Script Development - -### Pre-script Development Analysis - -Before starting the actual script development, a benchmark script analysis is performed. The purpose of the script analysis is to identify missing operators or features in MindSpore compared to the benchmark framework. The methodology can be found in the [Script Evaluation Tutorial](https://www.mindspore.cn/docs/en/master/migration_guide/script_analysis.html). - -MindSpore supports most of the common [functions](https://www.mindspore.cn/tutorials/experts/en/master/index.html) and [operators](https://www.mindspore.cn/docs/en/master/note/operator_list.html). MindSpore supports both dynamic graph (PyNative) mode and static graph (Graph) mode. Dynamic graph mode is flexible and easy to debug, so dynamic graph mode is mainly used for network debugging. Static graph mode has good performance and is mainly used for whole network training. When analyzing missing operators and functions, these two modes should be analyzed separately. - -If missing operators and functions are found, we can first consider combining the missing operators and functions based on the current operators or functions, and for mainstream CV and NLP class networks, new missing operators can generally be solved by combining existing operators. - -The combined operator can be implemented by means of a Cell. In MindSpore, [nn class operator](https://gitee.com/mindspore/mindspore/tree/master/mindspore/python/mindspore/nn) is implemented via this way. For example, the following `ReduceSumExp` operator is a combination of the existing `Exp`, `ReduceSum`, and `Log` suboperators. - -```python -class ReduceLogSumExp(Cell): - def __init__(self, axis, keep_dims=False): - super(ReduceLogSumExp, self).__init__() - validator.check_value_type('axis', axis, [int, list, tuple], self.cls_name) - validator.check_value_type('keep_dims', keep_dims, [bool], self.cls_name) - self.axis = axis - self.exp = ops.Exp() - self.sum = ops.ReduceSum(keep_dims) - self.log = ops.Log() - - def construct(self, x): - exp = self.exp(x) - sumexp = self.sum(exp, self.axis) - logsumexp = self.log(sumexp) - return logsumexp -``` - -If the missing functions and operators cannot be circumvented, or if the performance of the combined operators is poor, which seriously affects the training and inference of the network, you can contact [MindSpore Community](https://gitee.com/mindspore/mindspore/issues) for feedback and we will have a dedicated staff to solve it for you. - -#### ResNet50 Migration Example - -The following is the structure of the ResNet family of networks. - -![image-20210318152607548](./images/image-20210318152607548.png) - -The PyTorch implementation of the ResNet50 script is referenced in the [torchvision model](https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py). - -We can analyze it based on both operator and function aspects. - -- Operator analysis - -| PyTorch operator | MindSpore operator | supported | -| ---------------------- | ------------------ | ---------------------- | -| `nn.Conv2D` | `nn.Conv2d` | yes | -| `nn.BatchNorm2D` | `nn.BatchNom2d` | yes | -| `nn.ReLU` | `nn.ReLU` | yes | -| `nn.MaxPool2D` | `nn.MaxPool2d` | yes | -| `nn.AdaptiveAvgPool2D` | none | no | -| `nn.Linear` | `nn.Dense` | yes | -| `torch.flatten` | `nn.Flatten` | yes | - -Note: For PyTorch scripts, MindSpore provides the [PyTorch operator mapping](https://www.mindspore.cn/docs/en/master/note/api_mapping/pytorch_api_mapping.html), which can directly query whether the operator is supported. - -- Function Analysis - -| Pytorch Features | MindSpore Features | -| ------------------------- | ------------------------------------- | -| `nn.init.kaiming_normal_` | `initializer(init='HeNormal')` | -| `nn.init.constant_` | `initializer(init='Constant')` | -| `nn.Sequential` | `nn.SequentialCell` | -| `nn.Module` | `nn.Cell` | -| `nn.distibuted` | `set_auto_parallel_context` | -| `torch.optim.SGD` | `nn.optim.SGD` or `nn.optim.Momentum` | - -(Since the interface design of MindSpore and PyTorch are not exactly the same, only the key functions are listed here for comparison) - -After the operator and function analysis, we found that compared to PyTorch, MindSpore has no missing functions, but `nn.AdaptiveAvgPool` is missing in the operatpr. In this way, we need to further analyzed whether the missing operator has a replaceable plan. In the ResNet50 network, the input image shape is fixed and uniformed as `N,3,224,224`, where N is the batch size, 3 is the number of channels, 224 and 224 are the width and height of the image respectively, and the operators that change the image size in the network are `Conv2d` and `Maxpool2d`, and the effect of these two operators on the shape is fixed, so the input and output shapes of `nn.AdaptiveAvgPool2D` can be determined in advance. As long as we calculate the input and output shapes of `nn.AdaptiveAvgPool2D`, it is implemented via `nn.AvgPool` and `nn.ReduceMean`. The absence of this operator is replaceable and does not affect the training of the network. - -### Data Preprocessing - -To understand the implementation of a neural network, it is necessary to know the input data of the network first, so data preprocessing is the first part of the script development. MindSpore has designed a module dedicated to data processing - MindData, and data preprocessing with MindData consists of the following steps: - -1. importing the data path and reading the data file. -2. parsing the data. -3. data processing (e.g. common data slicing, shuffle, data augmentation, etc.). -4. data distribution (distribution of data in batch_size units, distributed training involves multi-machine distribution). - -In the process of reading and parsing data, MindSpore provides a more friendly data format - [MindRecord](https://www.mindspore.cn/tutorials/en/master/advanced/dataset/record.html). Users can convert the dataset in regular format to MindSpore data format, i.e. MindRecord, so that it can be easily loaded into MindSpore for training. At the same time, MindSpore is optimized for performance in some scenarios, and better performance can be obtained by using the MindRecord data format. - -Data processing is usually the most time-consuming phase of data preparation, and most of the operations on data are included in this step, such as Resize, Rescale, Crop, etc. in CV-like networks. MindSpore provides a set of common data processing integration interfaces, which can be called directly by users without implementing them. These integration interfaces not only improve the user-friendliness, but also improve the performance of data preprocessing and reduce the time consumption of data preparation during training. For details, please refer to the [Data Preprocessing Tutorial](https://www.mindspore.cn/tutorials/experts/en/master/dataset/optimize.html). - -In the data distribution, MindData provides an extremely simple API, which can be used to batch combination and repeating of data by directly calling batch and repeat operations. - -When the above 4 steps are completed, we can theoretically get the exact same data after processing the dataset by using MindSpore script and benchmark script processing dataset (if there are operations that introduce random cases need to be removed). - -#### ResNet50 Migration Example - -The ResNet50 network uses the ImageNet2012 dataset with the following PyTorch code for data pre-processing: - -```python -# sample execution (requires torchvision) -from PIL import Image -from torchvision import transforms -input_image = Image.open(filename) -preprocess = transforms.Compose([ - transforms.Resize(256), - transforms.CenterCrop(224), - transforms.ToTensor(), - transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), -]) -input_tensor = preprocess(input_image) -input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model -``` - -By looking at the above code, we find that the data preprocessing of ResNet50 mainly does Resize, CenterCrop, and Normalize operations, and there are two ways to implement these operations in MindSpore, one is to use MindSpore's data processing module MindData to call the encapsulated data preprocessing interface, and the other is loading through [Custom Dataset](https://www.mindspore.cn/tutorials/en/master/advanced/dataset.html). It is more recommended for developers to choose the first way, which not only can reduce the development of repetitive code and the introduction of errors, but also can get better data processing performance. For more information about MindData data processing, please refer to the Data Pipeline section in [tutorials](https://www.mindspore.cn/tutorials/experts/en/master/index.html). - -The following data processing functions are developed based on MindData: - -```python -import os -import mindspore as ms -import mindspore.dataset as ds -import mindspore.dataset.vision as vision -import mindspore.dataset.transforms as transforms - - -def create_dataset(dataset_path, batch_size=32, rank_size=1, rank_id=0, do_train=True): - """ - create a train or eval imagenet2012 dataset for resnet50 - - Args: - dataset_path(string): the path of dataset. - batch_size(int): the batch size of dataset. Default: 32 - rank_size(int): total num of devices for training. Default: 1, - greater than 1 in distributed training - rank_id(int): logical sequence in all devices. Default: 1, - can be greater than i in distributed training - - Returns: - dataset - """ - data_set = ds.ImageFolderDataset(dataset_path, num_parallel_workers=8, shuffle=do_train, - num_shards=rank_size, shard_id=rank_id) - - mean = [0.485 * 255, 0.456 * 255, 0.406 * 255] - std = [0.229 * 255, 0.224 * 255, 0.225 * 255] - - # define map operations - trans = [ - vision.Decode(), - vision.Resize(256), - vision.CenterCrop(224), - vision.Normalize(mean=mean, std=std), - vision.HWC2CHW() - ] - - type_cast_op = transforms.TypeCast(ms.int32) - - data_set = data_set.map(operations=trans, input_columns="image", num_parallel_workers=8) - data_set = data_set.map(operations=type_cast_op, input_columns="label", num_parallel_workers=8) - - # apply batch operations - data_set = data_set.batch(batch_size, drop_remainder=do_train) - - return data_set -``` - -In the above code we can find that for common classical datasets (e.g. ImageNet2012), MindData also provides us with `ImageFolderDataset` interface to read the raw data directly, which saves the workload of reading files by hand-written code. It should be noted that MindData creates datasets with different parameters for single-machine training and multi-machine distributed training, and distributed training requires two additional parameters `num_shard` and `shard_id`. - -### Subnet Development - -Usually subnet development consists of two parts: training subnets and loss subnets, where the training subnets can be divided or not depending on the complexity of the network. Developing a large neural network script directly can be overwhelming, so we can separate different modules or submodules of the network as a single subnet to ensure that each subnet is developed in parallel without interfering with each other. After the subnets are developed, we can also fix the subnet inputs and weights to form a comparison with the subnet code of the alignment script, which can be used as test cases for subsequent network development. - -During the accuracy tuning phase, we often encounter situations where the accuracy is not up to standard, and then we revisit the developed scripts and troubleshoot them line by line. It is much easier to find suspicious points from dozens of operators than from hundreds of operators, especially when the same subnet is invoked many times, which can reduce a lot of work when we troubleshoot by subnet. - -#### ResNet50 Migration Example - -Analyzing the ResNet50 network code, it can be divided into the following main subnets. - -- conv1x1, conv3x3: convolution with different kernel_size is defined. -- BasicBlock: the smallest subnet of ResNet18 and ResNet34 in the ResNet family of networks, consisting of Conv, BN, ReLU and residuals. -- BottleNeck: The smallest subnet of ResNet50, ResNet101 and ResNet152 in the ResNet family of networks, with an additional layer of Conv, BN and ReLU compared to BasicBlock, and the convolution position of downsampling has been changed. -- ResNet: A network that encapsulates the structure of BasicBlock, BottleNeck and Layer, different ResNet series networks can be constructed by passing different parameters. In this structure, some PyTorch self-defined initialization functions are also used. - -Based on the above subnetwork division, we redevelop the above development in conjunction with MindSpore syntax. - -For weight initialization, directly see [MindSpore's defined weight initialization methods](https://www.mindspore.cn/docs/en/master/api_python/mindspore.common.initializer.html). - -Redeveloping conv3x3 and conv1x1 - -```python -import mindspore.nn as nn - -def _conv3x3(in_channel, out_channel, stride=1): - return nn.Conv2d(in_channel, out_channel, kernel_size=3, stride=stride, - padding=0, pad_mode='same') - -def _conv1x1(in_channel, out_channel, stride=1): - return nn.Conv2d(in_channel, out_channel, kernel_size=1, stride=stride, - padding=0, pad_mode='same') -``` - -Redevelopment of BasicBlock and BottleNeck. - -```python -class ResidualBlockBase(nn.Cell): - def __init__(self, - in_channel, - out_channel, - stride=1): - super(ResidualBlockBase, self).__init__() - self.conv1 = _conv3x3(in_channel, out_channel, stride=stride) - self.bn1d = _bn(out_channel) - self.conv2 = _conv3x3(out_channel, out_channel, stride=1) - self.bn2d = _bn(out_channel) - self.relu = nn.ReLU() - - self.down_sample = False - if stride != 1 or in_channel != out_channel: - self.down_sample = True - - self.down_sample_layer = None - if self.down_sample: - self.down_sample_layer = nn.SequentialCell([_conv1x1(in_channel, out_channel, stride), - _bn(out_channel)]) - - def construct(self, x): - identity = x - - out = self.conv1(x) - out = self.bn1d(out) - out = self.relu(out) - - out = self.conv2(out) - out = self.bn2d(out) - - if self.down_sample: - identity = self.down_sample_layer(identity) - - out = out + identity - out = self.relu(out) - - return out - - -class ResidualBlock(nn.Cell): - expansion = 4 - - def __init__(self, - in_channel, - out_channel, - stride=1): - super(ResidualBlock, self).__init__() - self.stride = stride - channel = out_channel // self.expansion - self.conv1 = _conv1x1(in_channel, channel, stride=1) - self.bn1 = _bn(channel) - if self.stride != 1: - self.e2 = nn.SequentialCell([_conv3x3(channel, channel, stride=1), _bn(channel), - nn.ReLU(), nn.MaxPool2d(kernel_size=2, stride=2, pad_mode='same')]) - else: - self.conv2 = _conv3x3(channel, channel, stride=stride) - self.bn2 = _bn(channel) - - self.conv3 = _conv1x1(channel, out_channel, stride=1) - self.bn3 = _bn_last(out_channel) - self.relu = nn.ReLU() - self.down_sample = False - if stride != 1 or in_channel != out_channel: - self.down_sample = True - self.down_sample_layer = None - if self.down_sample: - self.down_sample_layer = nn.SequentialCell([_conv1x1(in_channel, out_channel, stride), _bn(out_channel)]) - def construct(self, x): - identity = x - out = self.conv1(x) - out = self.bn1(out) - out = self.relu(out) - if self.stride != 1: - out = self.e2(out) - else: - out = self.conv2(out) - out = self.bn2(out) - out = self.relu(out) - out = self.conv3(out) - out = self.bn3(out) - if self.down_sample: - identity = self.down_sample_layer(identity) - - out = out + identity - out = self.relu(out) - return out -``` - -Redevelopment of the whole ResNet family of whole nets. - -```python -class ResNet(nn.Cell): - """ - ResNet architecture. - - Args: - block (Cell): Block for network. - layer_nums (list): Numbers of block in different layers. - in_channels (list): Input channel in each layer. - out_channels (list): Output channel in each layer. - strides (list): Stride size in each layer. - num_classes (int): The number of classes that the training images are belonging to. - - Returns: - Tensor, output tensor. - - Examples: - >>> ResNet(ResidualBlock, - >>> [3, 4, 6, 3], - >>> [64, 256, 512, 1024], - >>> [256, 512, 1024, 2048], - >>> [1, 2, 2, 2], - >>> 10) - """ - - def __init__(self, - block, - layer_nums, - in_channels, - out_channels, - strides, - num_classes): - super(ResNet, self).__init__() - - if not len(layer_nums) == len(in_channels) == len(out_channels) == 4: - raise ValueError("the length of layer_num, in_channels, out_channels list must be 4!") - self.conv1 = _conv7x7(3, 64, stride=2) - self.bn1 = _bn(64) - self.relu = ops.ReLU() - - self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, pad_mode="same") - - self.layer1 = self._make_layer(block, - layer_nums[0], - in_channel=in_channels[0], - out_channel=out_channels[0], - stride=strides[0]) - self.layer2 = self._make_layer(block, - layer_nums[1], - in_channel=in_channels[1], - out_channel=out_channels[1], - stride=strides[1]) - self.layer3 = self._make_layer(block, - layer_nums[2], - in_channel=in_channels[2], - out_channel=out_channels[2], - stride=strides[2]) - self.layer4 = self._make_layer(block, - layer_nums[3], - in_channel=in_channels[3], - out_channel=out_channels[3], - stride=strides[3]) - - self.mean = ops.ReduceMean(keep_dims=True) - self.flatten = nn.Flatten() - self.end_point = _fc(out_channels[3], num_classes) - - def _make_layer(self, block, layer_num, in_channel, out_channel, stride): - """ - Make stage network of ResNet. - - Args: - block (Cell): Resnet block. - layer_num (int): Layer number. - in_channel (int): Input channel. - out_channel (int): Output channel. - stride (int): Stride size for the first convolutional layer. - Returns: - SequentialCell, the output layer. - - Examples: - >>> _make_layer(ResidualBlock, 3, 128, 256, 2) - """ - layers = [] - - resnet_block = block(in_channel, out_channel, stride=stride) - layers.append(resnet_block) - for _ in range(1, layer_num): - resnet_block = block(out_channel, out_channel, stride=1) - layers.append(resnet_block) - return nn.SequentialCell(layers) - - def construct(self, x): - x = self.conv1(x) - x = self.bn1(x) - x = self.relu(x) - c1 = self.maxpool(x) - - c2 = self.layer1(c1) - c3 = self.layer2(c2) - c4 = self.layer3(c3) - c5 = self.layer4(c4) - - out = self.mean(c5, (2, 3)) - out = self.flatten(out) - out = self.end_point(out) - - return out -``` - -Constructing the whole ResNet50 network by passing in information about the number of layers of ResNet50. - -```python -def resnet50(class_num=10): - """ - Get ResNet50 neural network. - - Args: - class_num (int): Class number. - - Returns: - Cell, cell instance of ResNet50 neural network. - - Examples: - >>> net = resnet50(10) - """ - return ResNet(ResidualBlock, - [3, 4, 6, 3], - [64, 256, 512, 1024], - [256, 512, 1024, 2048], - [1, 2, 2, 2], - class_num) -``` - -After the above steps, the MindSpore-based ResNet50 whole network structure and each subnet structure have been developed, and the next step is to develop other modules. - -### Other Modules - -Other modules usually include: reverse construction, gradient clipping, optimizer, learning rate generation, etc. These modules either have a single structure or rely on the results of the developed subnets in order to compare with the benchmark script. The development of scripts for these modules is less difficult than subnet development. - -#### ResNet50 migration example - -For additional training configurations, see [Configuration Information for NVIDIA Training ResNet50](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/Classification/ConvNets/resnet50v1.5#default-configuration), the training of ResNet50 mainly involves the following items. - -- SGD + Momentum optimizer is used. -- WeightDecay function is used (but gamma and bias of BatchNorm are not used). -- The cosine LR schedule is used. -- Label Smoothing is used. - -Implementing the SGD optimizer with Momentum, weights are applied to WeightDecay in addition to the gamma and bias of BN: - -```python -# define opt -decayed_params = [] -no_decayed_params = [] -for param in net.trainable_params(): - if 'beta' not in param.name and 'gamma' not in param.name and 'bias' not in param.name: - decayed_params.append(param) - else: - no_decayed_params.append(param) - -group_params = [{'params': decayed_params, 'weight_decay': weight_decay}, - {'params': no_decayed_params}, - {'order_params': net.trainable_params()}] -opt = Momentum(group_params, lr, momentum) -``` - -For implementing cosine LR schedule, reference on [MindSpore Cosine Decay LR](https://www.mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.cosine_decay_lr.html) - -Define Loss Function and implement Label Smoothing. - -```python -import mindspore.nn as nn -import mindspore as ms -from mindspore.nn import LossBase -import mindspore.ops as ops - -# define cross entropy loss -class CrossEntropySmooth(LossBase): - """CrossEntropy""" - def __init__(self, sparse=True, reduction='mean', smooth_factor=0., num_classes=1000): - super(CrossEntropySmooth, self).__init__() - self.onehot = ops.OneHot() - self.sparse = sparse - self.on_value = ms.Tensor(1.0 - smooth_factor, ms.float32) - self.off_value = ms.Tensor(1.0 * smooth_factor / (num_classes - 1), ms.float32) - self.ce = nn.SoftmaxCrossEntropyWithLogits(reduction=reduction) - - def construct(self, logit, label): - if self.sparse: - label = self.onehot(label, ops.shape(logit)[1], self.on_value, self.off_value) - loss = self.ce(logit, label) - return loss - -# define loss with label smooth -label_smooth_factor = 0.1 -loss = CrossEntropySmooth(sparse=True, reduction="mean",smooth_factor=label_smooth_factor, num_classes=1000) -``` - -### Hyperparameters Comparison - -When the subnetworks have been opened, the last step to do is to align the hyperparameters with the alignment script to ensure the network structure is consistent. It should be noted that the same set of hyperparameters may have different accuracy performance on different frameworks, so it is not necessary to strictly follow the hyperparameters of the peer script when migrating the network, and fine tuning can be done without changing the network structure. - -#### ResNet50 Migration Example - -In the training of ResNet50, the following hyperparameters are mainly involved. - -- momentum =0.875 -- batch_size = 256 -- learning rate = 0.256 -- learing rate schedule = cosine -- weight_decay = 1/32768 -- label_smooth = 0.1 -- epoch size = 90 - -## Process Hitting - -After the above steps, we have finished developing the necessary scripts for network migration, and the next step is to break through the stand-alone training, distributed training, and inference processes. - -### Stand-alone Training - -#### ResNet50 Migration Example - -For a better reading of the code, it is recommended to organize the script according to the following structure. - -```text -. -├── scripts -│   ├── run_distribute_train.sh # launch Ascend distributed training(8ps) -│   ├── run_eval.sh # launch Ascend evaluation -│   └── run_standalone_train.sh # launch Ascend standalone training(1ps) -├── src -│   ├── config.py # configuration -│   ├── cross_entropy_smooth.py # loss definition -│   ├── dataset.py # dataset preprocess -│   └── resnet.py # network structure -├── eval.py # evaluation -└── train.py # training - -2 directories, 9 files -``` - -train.py is defined as follows: - -```python -import os -import argparse -import ast -import mindspore as ms -from mindspore.nn import Momentum -from mindspore.communication import init -from mindspore.common import initializer -import mindspore.nn as nn - -from src.config import config -from src.dataset import create_dataset -from src.resnet import resnet50 -from src.cross_entropy_smooth import CrossEntropySmooth - -ms.set_seed(1) - -parser = argparse.ArgumentParser(description='Image classification') -parser.add_argument('--run_distribute', type=ast.literal_eval, default=False, help='Run distribute') -parser.add_argument('--dataset_path', type=str, default=None, help='Dataset path') -args_opt = parser.parse_args() - - -if __name__ == '__main__': - device_id = int(os.getenv('DEVICE_ID', '0')) - rank_size = int(os.getenv('RANK_SIZE', '1')) - rank_id = int(os.getenv('RANK_ID', '0')) - - # init context - ms.set_context(mode=ms.GRAPH_MODE, device_target='Ascend', device_id=device_id) - if rank_size > 1: - ms.set_auto_parallel_context(device_num=rank_size, parallel_mode=ms.ParallelMode.DATA_PARALLEL, - gradients_mean=True) - ms.set_auto_parallel_context(all_reduce_fusion_config=[85, 160]) - init() - - # create dataset - dataset = create_dataset(args_opt.dataset_path, config.batch_size, rank_size, rank_id) - step_size = dataset.get_dataset_size() - - # define net - net = resnet50(class_num=config.class_num) - - # init weight - for _, cell in net.cells_and_names(): - if isinstance(cell, nn.Conv2d): - cell.weight.set_data(initializer.initializer(initializer.XavierUniform(), - cell.weight.shape, - cell.weight.dtype)) - if isinstance(cell, nn.Dense): - cell.weight.set_data(initializer.initializer(initializer.TruncatedNormal(), - cell.weight.shape, - cell.weight.dtype)) - - lr = nn.dynamic_lr.cosine_decay_lr(config.lr_end, config.lr, config.epoch_size * step_size, - step_size, config.warmup) - - # define opt - decayed_params = [] - no_decayed_params = [] - for param in net.trainable_params(): - if 'beta' not in param.name and 'gamma' not in param.name and 'bias' not in param.name: - decayed_params.append(param) - else: - no_decayed_params.append(param) - - group_params = [{'params': decayed_params, 'weight_decay': config.weight_decay}, - {'params': no_decayed_params}, - {'order_params': net.trainable_params()}] - opt = Momentum(group_params, lr, config.momentum) - # define loss - loss = CrossEntropySmooth(sparse=True, reduction="mean", smooth_factor=config.label_smooth_factor, - num_classes=config.class_num) - # define model - model = ms.Model(net, loss_fn=loss, optimizer=opt, metrics={'acc'}) - - # define callbacks - time_cb = ms.TimeMonitor(data_size=step_size) - loss_cb = ms.LossMonitor() - cb = [time_cb, loss_cb] - if config.save_checkpoint: - #config_ck = CheckpointConfig(save_checkpoint_steps=config.save_checkpoint_epochs * step_size, - config_ck = ms.CheckpointConfig(save_checkpoint_steps=5, - keep_checkpoint_max=config.keep_checkpoint_max) - ckpt_cb = ms.ModelCheckpoint(prefix="resnet", directory=config.save_checkpoint_path, config=config_ck) - cb += [ckpt_cb] - - model.train(config.epoch_size, dataset, callbacks=cb, sink_size=step_size, dataset_sink_mode=False) -``` - -Note: For codes in other files in the directory, refer to MindSpore ModelZoo's [ResNet50 implementation](https://gitee.com/mindspore/models/tree/master/official/cv/resnet)(this script incorporates other ResNet family networks and ResNet-SE networks, and the specific implementation may differ from the benchmark script). - -### Distributed Training - -Distributed training has no impact on the network structure compared to stand-alone training, and can be done by modifying the stand-alone script by calling the distributed training interface provided by MindSpore, as described in [Distributed Training Tutorial](https://www.mindspore.cn/tutorials/experts/en/master/parallel/train_ascend.html). - -#### ResNet50 Migration Example - -Add the following interface to the standalone training script. - -```python -import os -import mindspore as ms -from mindspore.communication import init - -device_id = int(os.getenv('DEVICE_ID', '0')) -rank_size = int(os.getenv('RANK_SIZE', '1')) -rank_id = int(os.getenv('RANK_ID', '0')) - -# init context -ms.set_context(mode=ms.GRAPH_MODE, device_target='Ascend', device_id=device_id) -if rank_size > 1: - ms.set_auto_parallel_context(device_num=rank_size, parallel_mode=ms.ParallelMode.DATA_PARALLEL, - gradients_mean=True) - ms.set_auto_parallel_context(all_reduce_fusion_config=[85, 160]) - # init distribute training - init() -``` - -Modify the create_dataset interface to shard the data on data load to support distributed training by. - -```python -import os -import mindspore.dataset as ds - -device_id = int(os.getenv('DEVICE_ID', '0')) -rank_size = int(os.getenv('RANK_SIZE', '1')) -rank_id = int(os.getenv('RANK_ID', '0')) - -# rank_size is greater than 1 for distributed training -dataset = create_dataset(args_opt.dataset_path, config.batch_size, rank_size, rank_id) -# ... -``` - -### Inference - -The inference process differs from training in the following ways. - -- No need to define optimizers. -- No need to initialize the weights. -- Need to load trained CheckPoint after network is defined. -- Define the metric for computing inference accuracy. - -#### ResNet50 Migration Example - -Modified inference script: - -```python -"""train resnet.""" -import os -import argparse -import mindspore as ms -from mindspore.nn import SoftmaxCrossEntropyWithLogits - -from src.config import config -from src.dataset import create_dataset -from src.resnet import resnet50 -from src.cross_entropy_smooth import CrossEntropySmooth - -parser = argparse.ArgumentParser(description='Image classification') -parser.add_argument('--checkpoint_path', type=str, default=None, help='Checkpoint file path') -parser.add_argument('--dataset_path', type=str, default=None, help='Dataset path') -args_opt = parser.parse_args() - -ms.set_seed(1) - - - -if __name__ == '__main__': - device_id = int(os.getenv('DEVICE_ID', 0)) - # init context - ms.set_context(mode=ms.GRAPH_MODE, device_target='Ascend', device_id=device_id) - - # create dataset - dataset = create_dataset(args_opt.dataset_path, config.batch_size, do_train=False) - step_size = dataset.get_dataset_size() - - # define net - net = resnet50(class_num=config.class_num) - - # load checkpoint - param_dict = ms.load_checkpoint(args_opt.checkpoint_path) - ms.load_param_into_net(net, param_dict) - net.set_train(False) - - # define loss, model - loss = CrossEntropySmooth(sparse=True, reduction='mean', smooth_factor=config.label_smooth_factor, - num_classes=config.class_num) - - # define model - model = ms.Model(net, loss_fn=loss, metrics={'top_1_accuracy', 'top_5_accuracy'}) - - # eval model - res = model.eval(dataset) - print("result:", res, "ckpt=", args_opt.checkpoint_path) -``` - -### Problem Location - -You may encounter some interruptions in the training during the process, you can refer to the [Network Training Debug Tutorial](https://www.mindspore.cn/docs/en/master/migration_guide/neural_network_debug.html) to locate and solve them. - -### Full Example - -For full example, you can refer to the link: - -## Precision Tuning - -After hitting the flow, you can get the accuracy of network training by both training and inference steps. Usually, it is difficult to reproduce the accuracy of the alignment script at once, and we need to gradually improve the accuracy by accuracy tuning, which is less intuitive, less efficient, and more work than performance tuning. - -## Performance Tuning - -Usually, we refer to performance tuning to improve training performance with a fixed dataset, network size, and number of hardware, while improving performance by changing dataset size, network size, and number of hardware is obvious and out of the scope of this paper. - -Unless the performance problem has seriously hindered the accuracy debugging, the performance tuning must be placed after the accuracy has reached the standard, which has two main reasons: first, many modifications will affect the performance when locating the accuracy problem, making the already tuned performance again not up to standard, which may waste the workload; second, the performance tuning may introduce new accuracy problems, and if there is no accuracy that has reached the standard as a caretaker, the difficulty of locating the accuracy problem introduced this time will be greatly increased later. - -### Analyzing Profiling Data - -Analyzing Profiling data is an essential step in the performance tuning phase, and MindSpore's performance and precision tuning tool [MindInsight](https://www.mindspore.cn/mindinsight/docs/en/master/index.html) provides a rich set of performance and precision tuning methods, and the most important information for performance tuning is the Profiling data. In the iteration trajectory, you can see very detailed information about the start run time, end run time, number of calls and call order of each operator, which is very helpful for our performance tuning. The way to generate Profiling data is as follows: - -```python -import mindspore as ms -from mindspore import nn - -# init context -ms.set_context(mode=ms.GRAPH_MODE, device_target='Ascend', device_id=int(os.environ["DEVICE_ID"])) - -# init profiler, profiling data will be stored under folder ./data by default -profiler = ms.Profiler() - -# ... - -# start training -ms.Model.train() - -# end training, parse profiling data to readable text -profiler.analyse() -``` - -For more detailed usage of Profiling, you can refer to [Profiling Performance Analysis Methods](https://www.mindspore.cn/mindinsight/docs/en/master/performance_profiling.html). - -After obtaining Profiling data, we can analyze the performance bottleneck stages and operators, and then perform performance optimization, which can be referred to [Performance Tuning Guide](https://www.mindspore.cn/mindinsight/docs/en/master/performance_optimization.html). - -### Common Problems and Corresponding Optimization Methods - -#### MindData Performance - -Single-Step performance jitter and data queues that remain empty for a period of time are caused by poor performance of the data preprocessing part, which makes the data processing speed unable to keep up with the single-Step iteration speed, and these two phenomena usually occur in pairs. - -When the data processing speed is slow, the queue is gradually depleted from the initial full queue to an empty queue, and the training process will start waiting for the empty queue to be filled with data, and the network will continue the single-step training only once new data is filled. Since there is no queue as buffer for data processing, the performance jitter of data processing is directly reflected in the performance of single-Step, so it will also cause single-Step performance jitter. - -For MindData performance issues, refer to MindData in MindInsight Component's [Data Profiling](https://www.mindspore.cn/mindinsight/docs/en/master/performance_profiling_ascend.html#data-preparation-performance-analysis), which gives common problems and solutions to MindData performance. - -#### Multi-machine Synchronization Performance - -When distributed training is performed, after the forward propagation and gradient computation are completed during a Step, each machine starts to synchronize the AllReduce gradient, and the AllReduce synchronization time is mainly affected by the number of weights and machines. For more complex, larger machine-sized networks, the AllReduce gradient update time is longer, at which point we can perform AllReduce tangent to optimize this part of the time. - -Normally, AllReduce gradient synchronization waits until all the inverse operators are finished, i.e., all the gradients of all weights are computed before synchronizing the gradients of all machines at once, but with AllReduce tangent, we can synchronize the gradients of some weights as soon as they are computed, so that the gradient synchronization and the gradient computation of the remaining operators can be performed in parallel, hiding this part of the AllReduce gradient synchronization time. The tangent strategy is usually a manual attempt to find an optimal solution (supporting slicing greater than two segments). -As an example, [ResNet50 network](https://gitee.com/mindspore/models/blob/master/official/cv/resnet/train.py) has 160 weights and [85, 160] means that the gradient synchronization is performed immediately after the gradient is calculated for the 0th to 85th weights, and the gradient synchronization is performed after the gradient is calculated for the 86th to 160th weights. Here the two segments is sliced, so two gradient synchronizations are required. The code implementation is as follows: - -```python -device_id = int(os.getenv('DEVICE_ID', '0')) -rank_size = int(os.getenv('RANK_SIZE', '1')) -rank_id = int(os.getenv('RANK_ID', '0')) - -# init context -ms.set_context(mode=ms.GRAPH_MODE, device_target='Ascend', device_id=device_id) -if rank_size > 1: - ms.set_auto_parallel_context(device_num=rank_size, parallel_mode=ms.ParallelMode.DATA_PARALLEL, - gradients_mean=True) - ms.set_auto_parallel_context(all_reduce_fusion_config=[85, 160]) - init() -``` - -#### Operator Performance - -The situation that a single operator takes a long time and the performance of the same operator varies greatly under different shapes or different datatypes is mainly caused by the operator performance problem, which usually has the following two solutions. - -1. Use less computationally intensive data types. For example, there is no significant difference in precision between float16 and float32 for the same operator, so use the less computationally intensive float16 format. -2. Use other operators with the same algorithm to circumvent it. - -If you find any arithmetic with poor performance, we suggest you contact [MindSpore Community](https://gitee.com/mindspore/mindspore/issues) for feedback, and we will optimize it as soon as we confirm it as the performance problem. - -#### Framework Performance - -If there are too many conversion operators (TransData, Cast-like operators) and they take too much time, we can analyze the necessity of Cast operators if we add them manually, and remove the redundant Cast and TransData operators if they have no impact on the accuracy. - -If MindSpore automatically generates too many conversion operators, it may be that the MindSpore framework is not fully optimized for some special cases, you can contact [MindSpore Community](https://gitee.com/mindspore/mindspore/issues) for feedback. - -#### Other General Optimization Methods - -- Using mixed precision training - - The mixed precision training method accelerates the process of deep neural network training by using a mixture of single precision and half precision data formats, while maintaining the network accuracy that can be achieved with single precision training. Mixed precision training accelerates the computation process while reducing memory usage and access, and allows for training larger models or batch sizes on specific hardware. - - For details, please refer to the [Mixed precision tutorial](https://www.mindspore.cn/tutorials/experts/en/master/others/mixed_precision.html). - -- Enabling graph kernel fusion - - Graph-calculus fusion is a network performance optimization technique unique to MindSpore. It can automatically analyze and optimize existing network computation graph logic and combine with target hardware capabilities to perform computation reduction and substitution, operator splitting and fusion, operator special case compilation, etc. to improve the utilization of device computation resources and achieve overall optimization of network performance. Compared with traditional optimization techniques, graph computation fusion has unique advantages such as joint optimization of multiple operators across boundaries, cross-layer collaboration with operator compilation, and instant compilation of operators based on Polyhedral. In addition, graph fusion only requires the user to open the corresponding configuration, and then the whole optimization process can be completed automatically, without the need for additional sensing by the network developer, allowing the user to focus on the network algorithm implementation. - - The scenarios for graph fusion include: scenarios with high performance requirements for network execution time; scenarios where custom combinatorial operators are implemented by splicing basic operators, and where automatic fusion of these basic operators is desired to improve the performance of the custom combinatorial operators. diff --git a/docs/mindspore/source_en/migration_guide/script_analysis.md b/docs/mindspore/source_en/migration_guide/script_analysis.md deleted file mode 100644 index 6463a0c84e..0000000000 --- a/docs/mindspore/source_en/migration_guide/script_analysis.md +++ /dev/null @@ -1,65 +0,0 @@ -# Network Script Analysis - - - -## Operator Evaluation - -### MindSpore Operator Design - -The process of using the MindSpore framework to build a neural network is similar to other frameworks (TensorFlow/PyTorch), but the supported operators are different. It is necessary to find out the missing operators in the MindSpore framework when performing network migration (e.g., migrating from TensorFlow to the MindSpore-ascend platform). - -MindSpore API is composed of various Python/C++ API operators, which can be roughly divided into: - -- Data framework operator - - Including tensors, basic data types, training gradients, optimizer operators, such as `mindspore.int32`, `mindspore.nn.Cell`, etc. - -- Data preprocessing operator - - Including image reading, data type conversion operators, such as `mindspore.dataset.MnistDataset`, etc. - -- Network structure operator - - Including convolution and normalization operators used in network construction, such as `mindspore.nn.Conv2d`, `mindspore.nn.Dense`, etc. - -The surface layer of the network structure operator is the ME operator, which is the operator API called by the user (e.g., `mindspore.nn.Softmax`), and the ME operator is implemented by calling the TBE operator (C/C++) at the bottom layer. - -When counting missing ME operators, you need to find out the corresponding operators of all operators (including data framework classes, data preprocessing, and network structure operators) in the source script in the MindSpore framework (e.g.,`tf.nn.relu` corresponds to MindSpore operator `mindspore.nn.ReLU`). If there is no corresponding operator in MindSpore, it will be counted as missing. - -### Querying Operator Mapping Table - -Find the network structure and the Python file that implements the training function in the code library (the name is generally train.py model.py, etc.), and find all relevant operators in the script file (including data framework classes, data preprocessing, network structure operators, etc.), and compare with MindSpore Operator API, to find the platform support status of the operator under [mindspore.nn](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html) or [mindspore.ops](https://www.mindspore.cn/docs/en/master/api_python/mindspore.ops.html). - -If the corresponding ME operator cannot be found on this webpage, you can continue to search for the operator name in [MindSpore API List](https://www.mindspore.cn/docs/en/master/index.html). - -If the source code is a PyTorch script, you can directly query [MindSpore and PyTorch operator mapping](https://www.mindspore.cn/docs/en/master/note/api_mapping/pytorch_api_mapping.html) to find the corresponding MindSpore operator. For the mapping of other frame operators, please refer to the operator naming and function description. Note that for operators with the same function, MindSpore may define a name for this operator differing from other frameworks, and the parameters and functions of operators with the same name may also be different from other frameworks. Please refer to the official description for checking the names. - -### Missing Operator Processing Strategy - -1. Consider replacing it with other operators: It is necessary to analyze the implementation formula of the operator and examine whether the existing MindSpore operator can be superimposed to achieve the expected goal. -3. Consider using Customized operators: see [Custom Operators (Custom based)](https://www.mindspore.cn/tutorials/experts/en/master/operation/op_custom.html). -4. Consider using third-party operators by Customized operators: see [Use Third-Party Operators by Custom Operators](https://www.mindspore.cn/docs/en/master/migration_guide/use_third_party_op.html). -4. Consider temporary circumvention solutions: For example, if a certain loss is not supported, it can be replaced with a loss operator of the same kind that has been supported. -5. Submit suggestions in [MindSpore Community](https://gitee.com/mindspore/mindspore/issues) to develop missing operators. - -## Grammar Assessment - -MindSpore provides two modes: `GRAPH_MODE` and `PYNATIVE_MODE`. - -In PyNative mode, the behavior of the model for **Inference** is same as that of in the general Python code. - -When using GRAPH_MODE, or when using PYNATIVE_MODE for **Training**, there are usually grammatical restrictions. In these two cases, it is necessary to perform graph compilation operations on the Python code. In this step, MindSpore has not yet been able to support the complete set of Python syntax, so there will be some restrictions on the implementation of the `construct` function. For specific restrictions, please refer to [MindSpore static graph syntax support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). - -### Common Restriction Principles - -Compared with the specific syntax description, the common restrictions can be summarized as follows: - -- Do not call other Python module , such as numpy and scipy, when building the graph. The related processing should be moved to the `__init__` stage. -- Do not use custom types when building the graph. Instead, use the data types and Python basic types provided by MindSpore. You can use tuple/list combinations based on these types. -- Do not processing multi-threaded, multi-process data when building the graph. - -### Common Processing Strategies - -1. Use the operators provided by MindSpore to replace the functions of other Python libraries. The processing of constants can be moved to the `__init__` stage. -2. Use basic types for combination, and you can consider increasing the amount of function parameters. There are no restrictions on the input parameters of the function, and variable length input can be used. -3. Avoid multi-threading processing in the network. diff --git a/docs/mindspore/source_en/migration_guide/script_development.rst b/docs/mindspore/source_en/migration_guide/script_development.rst deleted file mode 100644 index 91deb6efeb..0000000000 --- a/docs/mindspore/source_en/migration_guide/script_development.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. MindSpore documentation master file, created by - sphinx-quickstart on Thu Mar 24 11:00:00 2020. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Network Script Development -=========================== - -.. toctree:: - :maxdepth: 1 - - Using Mindcoverter to Perform Migration - migration_script - use_third_party_op diff --git a/docs/mindspore/source_en/migration_guide/use_third_party_op.md b/docs/mindspore/source_en/migration_guide/use_third_party_op.md index 3b9c63bc16..ac391aa4fa 100644 --- a/docs/mindspore/source_en/migration_guide/use_third_party_op.md +++ b/docs/mindspore/source_en/migration_guide/use_third_party_op.md @@ -1,37 +1,34 @@ -# Call Third-Party Operators by Customized Operators +# Using Third-party Operator Libraries Based on Customized Interfaces - + ## Overview -When built-in operators cannot meet requirements during network development, you can call the Python API [Custom](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) primitive defined in MindSpore to quickly create different types of customized operators for use. +When lacking of the built-in operators during developing a network, you can use the primitive in [Custom](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) to easily and quickly define and use different types of customized operators. -You can choose different customized operator developing methods base on needs. -See: [custom_operator_custom](https://www.mindspore.cn/tutorials/experts/en/master/operation/op_custom.html). +Developers can choose different customized operator development methods according to their needs. For details, please refer to the [Usage Guide](https://www.mindspore.cn/tutorials/experts/en/master/operation/op_custom.html) of Custom operator. -There is a defining method called `aot` which has a special use. The `aot` mode can call the corresponding `cpp`/`cuda` function by loading the precompiled `so`. Therefore, when a third-party library provides the `cpp`/`cuda` function `API`, you can try to call its function interface in `so`. +One of the development methods for customized operators, the `aot` method, has its own special use. The `aot` can call the corresponding `cpp`/`cuda` functions by loading a pre-compiled `so`. Therefore. When a third-party library provides `API`, a `cpp`/`cuda` function, you can try to call its function interface in `so`, which is described below by taking `Aten` library in PyTorch as an example. -Here is an example of how to use `Aten` library of PyTorch Aten. +## PyTorch Aten Operator Matching -## Using PyTorch Aten operators for Docking +When lacking of built-in operators during migrating a network that uses the PyTorch Aten operator, we can use the `aot` development of the `Custom` operator to call the PyTorch Aten operator for fast verification. -When migrating a network using the PyTorch Aten operator encounters a shortage of built-in operators, we can use the `aot` development method of the `Custom` operator to call PyTorch Aten's operator for fast verification. +PyTorch provides a way to support the introduction of PyTorch header files, so that `cpp/cuda` code can be written by using related data structures and compiled into `so`. Reference: . -PyTorch provides a way to support the introduction of PyTorch's header files to write `cpp/cuda` code by using its associated data structures and compile it into `so`. See:. +Using a combination of the two approaches, the customized operator can call the PyTorch Aten operator, which is used as follows: -Using a combination of the two methods, the customized operator can call the PyTorch Aten operator as follows: +### 1. Downloading Project Files -### 1. Downloading the Project files +The project files can be downloaded [here](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/migration_guide/test_custom_pytorch.tar). -User can download the project files from [here](https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/migration_guide/test_custom_pytorch.tar). - -Use the following command to extract files and find the folder `test_custom_pytorch`: +Use the following command to extract the zip package and get the folder `test_custom_pytorch`. ```bash tar xvf test_custom_pytorch.tar ``` -The folder include the following files: +The folder contains the following files: ```text test_custom_pytorch @@ -49,22 +46,22 @@ test_custom_pytorch └── test_gpu_op.py # a test file to run Aten GPU operator on GPU device ``` -Using the PyTorch Aten operator focuses mainly on env.sh, setup.py, leaky_relu.cpp/cu, test_*, .py. +Using PyTorch Aten arithmetic focuses on env.sh, setup.py, leaky_relu.cpp/cu, and test_*.py. -Among them, env.sh is used to set environment variables, setup.py is used to compile so, leaky_relu.cpp/cu is used to reference the source code that calls the PyTorch Aten operator, and test_*.py is used to refer to the call Custom operator. +Among them, env.sh is used to set environment variables, setup.py is used to compile so, leaky_relu.cpp/cu is a reference for used to write the source code for calling PyTorch Aten operator in reference, and test_*.py is used to call Custom operator in reference. -### 2. Writing and calling the Source Code File of PyTorch Aten Operators +### 2. Writing Source Code File that Calls PyTorch Aten Operator -Refer to leaky_relu.cpp/cu to write a source code file that calls the PyTorch Aten operator. +Refer to leaky_relu.cpp/cu to write the source code file that calls the PyTorch Aten operator. -The customized operator of `aot` type adopts the `AOT` compilation method, which requires network developers to hand-write the source code file of the operator implementation based on a specific interface, and compile the source code file into a dynamic link library in advance, and then the framework will automatically call the function defined in the dynamic link library. In terms of the development language of the operator implementation, the `GPU` platform supports `CUDA`, and the `CPU` platform supports `C` and `C++`. The interface specification of the operator implemented by the operators in the source code file is as follows: +Since customized operators of type `aot` are compiled by using `AOT`, the developer is required to write the source code files corresponding to the operator implementation functions based on a specific interface and compile the source code files into a dynamic link library in advance, and the framework will automatically call the functions in the dynamic link library for execution during the web runtime. As for the development language of the operator implementation, `CUDA` is supported for `GPU` platform, and `C` and `C++` are supported for `CPU` platform. The interface specification of the operator implementation functions in the source code file is as follows. ```cpp extern "C" int func_name(int nparam, void **params, int *ndims, int64_t **shapes, const char **dtypes, void *stream, void *extra); ``` -If the `cpu` operator is called, taking `leaky_relu.cpp` as an example, the file provides the function `LeakyRelu` required by `AOT`, which calls `torch::leaky_relu_out` function of PyTorch Aten: +If `cpu` operator is called, taking `leaky_relu.cpp` as an example, the file provides `LeakyRelu` required by `AOT`, which calls `torch::leaky_relu_out` of PyTorch Aten: ```cpp #include @@ -83,7 +80,7 @@ extern "C" int LeakyRelu( auto at_input = tensors[0]; auto at_output = tensors[1]; torch::leaky_relu_out(at_output, at_input); - // If you are using a version without output, the code is as follows: + // If you use the version without output, the code is as follows: // torch::Tensor output = torch::leaky_relu(at_input); // at_output.copy_(output); return 0; @@ -91,11 +88,11 @@ extern "C" int LeakyRelu( ``` -If the `gpu` operator is called, take `leaky_relu.cu` as an example: +If `gpu` operator is called, taking `leaky_relu.cu` as an example: ```cpp #include -#include +#include // Header file reference section #include "ms_ext.h" extern "C" int LeakyRelu( @@ -116,50 +113,51 @@ extern "C" int LeakyRelu( } ``` -PyTorch Aten provides operator functions versions with output and operator functions versions without output. Operator functions with output have the '_out' suffix, and PyTorch Aten provides 300+ `apis` of common operators. +Among them, PyTorch Aten provides a version of the operator function with output and a version of the operator function without output. The operator function with output has the `_out` suffix, and PyTorch Aten provides `api` for 300+ commonly-used operators. -When `torch::*_out` is called, `output` copy is not needed. When the versions without `_out`suffix is called, API `torch.Tensor.copy_` is needed to called to result copy. +When calling `torch::*_out`, `output` copy is not required. When calling the version without the `_out` suffix, calling the API `torch.Tensor.copy_` to make a copy of the result is required. -To see which functions are supported for calling PyTorch Aten, the `CPU` version refers to the PyTorch installation path: `python*/site-packages/torch/include/ATen/CPUFunctions_inl.h` , and for the corresponding `GPU` version, refers to`python*/site-packages/torch/include/ATen/CUDAFunctions_inl.h`。 +To see which functions of PyTorch Aten are supported, refer to the PyTorch installation path: `python*/psite-packages/torch/include/ATen/CPUFunctions_inl.h` for the `CPU` version and `python*/ site-packages/torch/include/ATen/CUDAFunctions_inl.h` for the `GPU` version. -The apis provided by ms_ext.h are used in the above use case, which are briefly described here: +The above use case uses the api provided by ms_ext.h, which is described here: ```cpp -// Convert MindSpore kernel's inputs/outputs to PyTorch Aten's Tensor +// Transform inputs/outputs of MindSpore kernel as Tensor of PyTorch Aten std::vector get_torch_tensors(int nparam, void** params, int* ndims, int64_t** shapes, const char** dtypes, c10::Device device) ; ``` -### 3. Using the compilation script `setup.py` to generate so +### 3. Using the Compile Script `setup.py` to Generate so -setup.py uses the `cppextension` provided by PyTorch Aten to compile the above `c++/cuda` source code into an `so` file. +setup.py compiles the above `c++/cuda` source code into a `so` file by using the `cppextension` provided by PyTorch Aten. -Before execution, you need to make sure that PyTorch is installed. +You need to make sure PyTorch is installed before executing it. ```bash pip install torch ``` -Then add PyTorch's `lib` into `LD_LIBRARY_PATH`。 +Add `lib` of PyTorch to `LD_LIBRARY_PATH`. ```bash export LD_LIBRARY_PATH=$(python3 -c 'import torch, os; print(os.path.dirname(torch.__file__))')/lib:$LD_LIBRARY_PATH ``` -Run: +Execute: ```bash cpu: python setup.py leaky_relu.cpp leaky_relu_cpu.so gpu: python setup.py leaky_relu.cu leaky_relu_gpu.so ``` -Then the so files that we need may be obtained. +Get so file we need. -### 4. Using the Customized Operator +### 4. Using Customized Operators -Taking CPU as an example, use the Custom operator to call the above PyTorch Aten operator, see the code test_cpu_op.py: +Taking the CPU as an example, the above PyTorch Aten operator is called by using the Custom operator. The code can be found in test_cpu_op.py: ```python import numpy as np +import mindspore as ms from mindspore.nn import Cell import mindspore.ops as ops @@ -183,36 +181,36 @@ if __name__ == "__main__": print(output) ``` -Run: +Execute: ```bash python test_cpu_op.py ``` -Result: +The result is: ```text [[ 0. -0.001] [-0.002 1. ]] ``` -Attention: +Note: -When using a PyTorch Aten `GPU` operator，set `device_target`to `"GPU"`. +If you are using the PyTorch Aten `GPU` operator, `device_target` needs to be set to `"GPU"`. ```python -ms.set_context(device_target="GPU") +set_context(device_target="GPU") op = ops.Custom("./leaky_relu_gpu.so:LeakyRelu", out_shape=lambda x : x, out_dtype=lambda x : x, func_type="aot") ``` -When using a PyTorch Aten `CPU` operator and `device_target` is `"GPU"`, the settings that need to be added are as follows: +If the PyTorch Aten `CPU` operator is used and the `device_target` is set to `"GPU"`, you need to add the following settings: ```python -ms.set_context(device_target="GPU") +set_context(device_target="GPU") op = ops.Custom("./leaky_relu_cpu.so:LeakyRelu", out_shape=lambda x : x, out_dtype=lambda x : x, func_type="aot") op.add_prim_attr("primitive_target", "CPU") ``` -> 1. Compile so with cppextension requires a compiler version that meets the tool's needs, and check for the presence of gcc/clang/nvcc. -> 2. Compile so with cppextension will generate a build folder in the script path, which stores so. The script will copy so to outside of build, but cppextension will skip compilation if it finds that there is already so in build, so if it is a newly compiled so, remember to empty the so under the build. -> 3. The following tests is based on PyTorch 1.9.1，cuda11.1，python3.7. The download link:. The cuda version supported by PyTorch Aten needs to be consistent with the local cuda version, and whether other versions are supported needs to be explored by the user. \ No newline at end of file +> 1. To compile so with cppextension, you need to meet the compiler version required by the tool and check if gcc/clang/nvcc exists. +> 2. Using cppextension to compile so will generate a build folder in the script path, which stores so. The script will copy so outside build, but cppextension will skip the compilation if it finds that there is already so in the build, so remember to clear newly-compiled so under build. +> 3. The above tests are based on PyTorch 1.9.1, cuda 11.1, python 3.7. Download link is: . The cuda version supported by PyTorch Aten should be the same as the local cuda version, and whether other versions is supported should be explored by the user. diff --git a/docs/mindspore/source_zh_cn/index.rst b/docs/mindspore/source_zh_cn/index.rst index 99526202a1..d6a75f44b8 100644 --- a/docs/mindspore/source_zh_cn/index.rst +++ b/docs/mindspore/source_zh_cn/index.rst @@ -86,6 +86,7 @@ MindSpore 文档 migration_guide/debug_and_tune migration_guide/sample_code migration_guide/faq + migration_guide/use_third_party_op .. toctree:: :glob: diff --git a/docs/mindspore/source_zh_cn/migration_guide/use_third_party_op.md b/docs/mindspore/source_zh_cn/migration_guide/use_third_party_op.md index fef5237a36..21ffc233cd 100644 --- a/docs/mindspore/source_zh_cn/migration_guide/use_third_party_op.md +++ b/docs/mindspore/source_zh_cn/migration_guide/use_third_party_op.md @@ -16,7 +16,7 @@ PyTorch提供了一种方式可以支持引入PyTorch的头文件，从而使用其相关的数据结构编写`cpp/cuda`代码，并编译成`so`。参考：。 -将两种方式结合使用，自定义算子可以调用PyTorch Aten算子，使用方式如下: +将两种方式结合使用，自定义算子可以调用PyTorch Aten算子，使用方式如下： ### 1. 下载工程文件 @@ -119,7 +119,7 @@ extern "C" int LeakyRelu( 想查看支持调用PyTorch Aten的哪些函数，`CPU`版本参考PyTorch安装路径下的：`python*/site-packages/torch/include/ATen/CPUFunctions_inl.h` ，相应的`GPU`版本参考`python*/site-packages/torch/include/ATen/CUDAFunctions_inl.h`。 -以上用例中使用了ms_ext.h提供的api，这里稍作介绍: +以上用例中使用了ms_ext.h提供的api，这里稍作介绍： ```cpp // 将 MindSpore kernel 的 inputs/outputs 转换为 PyTorch Aten 的 Tensor -- Gitee