diff --git a/docs/lite/docs/source_en/quick_start/quick_start_c.md b/docs/lite/docs/source_en/quick_start/quick_start_c.md new file mode 100644 index 0000000000000000000000000000000000000000..c412211121f2b5a702b737c363d16f993fe269bb --- /dev/null +++ b/docs/lite/docs/source_en/quick_start/quick_start_c.md @@ -0,0 +1,244 @@ +# Expriencing Simpcified Inference Demo with C-language + + + +## Overview + +This tutorial provides a sample program for MindSpore Lite to perform inference, which demonstrates the basic process of end-side inference with C-language by randomly typing, performing inference, and printing inference results, so that users can quickly understand the use of MindSpore Lite to perform inference-related APIs. This tutorial performs the inference of MobileNetV2 model by taking randomly generated data as input data and prints obtained output data. The related code is in the [mindspore/lite/examples/quick_start_c](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_c) directory. + +Performing inference with MindSpore Lite consists of the following main steps: + +1. Read model: Read the `.ms` model converted by [Model Conversion Tool](https://www.mindspore.cn/lite/docs/en/master/use/converter_tool.html) from the file system. +2. Create configuration context: Create a Configuration [Context](https://www.mindspore.cn/lite/api/en/master/c_api/context_c.html) that holds some basic configuration parameters needed, to guide model compilation and model execution. +3. Create, load and compile Model: Before executing inference, you need to call [MSModelBuildFromFile](https://www.mindspore.cn/lite/api/zh-CN/master/api_c/model_c.html#msmodelbuildfromfile) interface of [Model](https://www.mindspore.cn/lite/api/en/master/api_c/model_c.html) for model loading and compilation, and configure the Context obtained in the previous step into the Model. The model loading phase parses the file cache into a runtime model. The model compilation phase mainly carries out the process of operator selection scheduling, subgraph slicing, etc, which will consume more time, so it is recommended that the Model be created once, compiled once, and reasoned several times. +4. Input data: The data needs to be padded in the `Input Tensor` before model execution. +5. Execute inference: Use [MSModelPredict](https://www.mindspore.cn/lite/api/en/master/api_c/model_c.html#msmodelpredict) inferene of [Model](https://www.mindspore.cn/lite/api/en/master/api_c/model_c.html) for model inference. +6. Obtain output: After the model execution, the inference result can be obtained by `output Tensor`. +7. Free memory: When do not need to use MindSpore Lite inference framework, you need to free the created [Model](https://www.mindspore.cn/lite/api/en/master/api_c/model_c.html). + +![img](../images/lite_runtime.png) + +## Building and Running the Demo + +- Environment requirements + + - System environment: Linux x86_64, Ubuntu 18.04.02LTS recommended + - Compilation dependencies: + - [CMake](https://cmake.org/download/) >= 3.18.3 + - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 + +- Compiling and building + + Execute the [build script](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_c/build.sh) in `mindspore/lite/examples/quick_start_c` directory, which will automatically download the MindSpore Lite inference framework library and the model file and compile the Demo. + + ```bash + bash build.sh + ``` + + > If the build script fails to download the MindSpore Lite inference framework, please manually download the MindSpore Lite model inference framework [mindspore-lite-{version}-linux-x64.tar.gz](https://www.mindspore.cn/lite/docs/en/master/use/downloads.html) for the CPU hardware platform and Ubuntu-x64 operating system, after decompression copy the `libmindspore-lite.so` file from the unpacked `runtime/lib` directory to `mindspore/ lite/examples/quick_start_c/lib` directory, and the files in `runtime/include` directory to `mindspore/lite/examples/quick_start_c/include` directory. + > + > If the build script fails to download the MobileNetV2 model, please manually download the relevant model file [mobilenetv2.ms](https://download.mindspore.cn/model_zoo/official/lite/quick_start/mobilenetv2.ms) and copy it to the `mindspore/lite/examples/quick_start_c/model` directory. + > + > After manually downloading and placing the files in the specified location, the build.sh script needs to be executed again to complete the compiling and building. + +- Executing inference + + After compiling and building, go to the `mindspore/lite/examples/quick_start_c/build` directory and execute the following command to experience the MobileNetV2 model inference by MindSpore Lite. + + ```bash + ./mindspore_quick_start_c ../model/mobilenetv2.ms + ``` + + When the execution is completed, the following results will be obtained. Print the name, the size and the number of the output Tensor and the first 50 data: + + ```text + Tensor name: Softmax-65, tensor size is 4004 ,elements num: 1001. + output data is: + 0.000011 0.000015 0.000015 0.000079 0.000070 0.000702 0.000120 0.000590 0.000009 0.000004 0.000004 0.000002 0.000002 0.000002 0.000010 0.000055 0.000006 0.000010 0.000003 0.000010 0.000002 0.000005 0.000001 0.000002 0.000004 0.000006 0.000008 0.000003 0.000015 0.000005 0.000011 0.000020 0.000006 0.000002 0.000011 0.000170 0.000005 0.000009 0.000006 0.000002 0.000003 0.000009 0.000005 0.000006 0.000003 0.000011 0.000005 0.000027 0.000003 0.000050 0.000016 + ``` + +### Windows + +- Environment requirements + + - System environment: Windows 7, Windows 10; 64-bit + - Compilation dependencies: + - [CMake](https://cmake.org/download/) >= 3.18.3 + - [MinGW GCC](https://sourceforge.net/projects/mingw-w64/files/ToolchainstargettingWin64/PersonalBuilds/mingw-builds/7.3.0/threads-posix/seh/x86_64-7.3.0-release-posix-seh-rt_v5-rev0.7z/download) = 7.3.0 + +- Compiling and building + + - Library downloading: Please manually download the MindSpore Lite model inference framework [mindspore-lite-{version}-win-x64.zip](https://www.mindspore.cn/lite/docs/en/master/use/downloads.html) with CPU as the hardware platform and Windows-x64 as the operating system, after decompression copy all files in the `runtime\lib` directory to the `mindspore\lite\examples\quick_start_clib\` project directory, and the files in the `runtime\include` directory to the `mindspore\lite\examples\quick_start_c\include` project directory. (Note: the `lib` and `include` directories under the project need to be created manually) + + - Model downloading: Please manually download the relevant model file [mobilenetv2.ms](https://download.mindspore.cn/model_zoo/official/lite/quick_start/mobilenetv2.ms) and copy it to the `mindspore\ lite\examples\quick_start_c\model` directory. + + - Compiling: Execute the [build script](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_c/build.bat) in the `mindspore\lite\examples\quick_start_c` directory, which can automatically download the relevant files and compile the Demo. + + ```bash + call build.bat + ``` + +- Executing inference + + After compiling and building, go to the `mindspore\lite\examples\quick_start_c\build` directory and execute the following command to experience the MobileNetV2 model inference by MindSpore Lite. + + ```bash + set PATH=..\lib;%PATH% + call mindspore_quick_start_c.exe ..\model\mobilenetv2.ms + ``` + + When the execution is completed, the following results will be obtained. Print the name, the size and the number of the output Tensor and the first 50 data: + + ```text + Tensor name: Softmax-65, tensor size is 4004 ,elements num: 1001. + output data is: + 0.000011 0.000015 0.000015 0.000079 0.000070 0.000702 0.000120 0.000590 0.000009 0.000004 0.000004 0.000002 0.000002 0.000002 0.000010 0.000055 0.000006 0.000010 0.000003 0.000010 0.000002 0.000005 0.000001 0.000002 0.000004 0.000006 0.000008 0.000003 0.000015 0.000005 0.000011 0.000020 0.000006 0.000002 0.000011 0.000170 0.000005 0.000009 0.000006 0.000002 0.000003 0.000009 0.000005 0.000006 0.000003 0.000011 0.000005 0.000027 0.000003 0.000050 0.000016 + ``` + +## Configuring CMake + +The following is sample code when the `libmindspore-lite.so` static library is integrated via CMake. + +> The `-wl,--whole-archive` option needs to be passed to the linker when the `libmindspore-lite.so` static library is integrated. +> +> Since the `-fstack-protector-strong` stack-protected compiling option was added when compiling MindSpore Lite, it is also necessary to link the `ssp` library in MinGW on the Windows platform. +> +> Since support for so library file handling was added when compiling MindSpore Lite, it is also necessary to link the `dl` library on the Linux platform. + +```cmake +cmake_minimum_required(VERSION 3.18.3) +project(QuickStartC) + +if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7.3.0) + message(FATAL_ERROR "GCC version ${CMAKE_CXX_COMPILER_VERSION} must not be less than 7.3.0") +endif() + +# Add directory to include search path +include_directories(${CMAKE_CURRENT_SOURCE_DIR}) + +# Add directory to linker search path +link_directories(${CMAKE_CURRENT_SOURCE_DIR}/lib) + +file(GLOB_RECURSE QUICK_START_CXX ${CMAKE_CURRENT_SOURCE_DIR}/*.cc) +add_executable(mindspore_quick_start_c ${QUICK_START_CXX}) + +target_link_libraries( + mindspore_quick_start_c + -Wl,--whole-archive libmindspore-lite -Wl,--no-whole-archive + pthread +) + +# Due to the increased compilation options for stack protection, +# it is necessary to target link ssp library when Use the static library in Windows. +if(WIN32) + target_link_libraries( + mindspore_quick_start_c + ssp + ) +else() + target_link_libraries( + mindspore_quick_start_c + dl + ) +endif() +``` + +## Creating Configuration Context + +```c + // Create and init context, add CPU device info + MSContextHandle context = MSContextCreate(); + if (context == NULL) { + printf("MSContextCreate failed.\n"); + return kMSStatusLiteError; + } + const int thread_num = 2; + MSContextSetThreadNum(context, thread_num); + MSContextSetThreadAffinityMode(context, 1); + + MSDeviceInfoHandle cpu_device_info = MSDeviceInfoCreate(kMSDeviceTypeCPU); + if (cpu_device_info == NULL) { + printf("MSDeviceInfoCreate failed.\n"); + MSContextDestroy(&context); + return kMSStatusLiteError; + } + MSDeviceInfoSetEnableFP16(cpu_device_info, false); + MSContextAddDeviceInfo(context, cpu_device_info); +``` + +## Creating, Loading and Compiling Model + +Model loading and compilation can be done by calling [MSModelBuildFromFile](https://www.mindspore.cn/lite/api/zh-CN/master/api_c/model_c.html#msmodelbuildfromfile) interface of [Model](https://www.mindspore.cn/lite/api/en/master/api_c/model_c.html) to load and compile from the file path to get the runtime model. In this case, `argv[1]` corresponds to the model file path inputted from the console. + +```c + // Create model + MSModelHandle model = MSModelCreate(); + if (model == NULL) { + printf("MSModelCreate failed.\n"); + MSContextDestroy(&context); + return kMSStatusLiteError; + } + + // Build model + int ret = MSModelBuildFromFile(model, argv[1], kMSModelTypeMindIR, context); + if (ret != kMSStatusSuccess) { + printf("MSModelBuildFromFile failed, ret: %d.\n", ret); + MSModelDestroy(&model); + return ret; + } +``` + +## Model Inference + +The model inference mainly includes the steps of input data, inference execution, and obtaining output, where the input data in this example is generated by random data, and finally the output result after execution of inference is printed. + +```c + // Get Inputs + MSTensorHandleArray inputs = MSModelGetInputs(model); + if (inputs.handle_list == NULL) { + printf("MSModelGetInputs failed, ret: %d.\n", ret); + MSModelDestroy(&model); + return ret; + } + + // Generate random data as input data. + ret = GenerateInputDataWithRandom(inputs); + if (ret != kMSStatusSuccess) { + printf("GenerateInputDataWithRandom failed, ret: %d.\n", ret); + MSModelDestroy(&model); + return ret; + } + + // Model Predict + MSTensorHandleArray outputs; + ret = MSModelPredict(model, inputs, &outputs, NULL, NULL); + if (ret != kMSStatusSuccess) { + printf("MSModelPredict failed, ret: %d.\n", ret); + MSModelDestroy(&model); + return ret; + } + + // Print Output Tensor Data. + for (size_t i = 0; i < outputs.handle_num; ++i) { + MSTensorHandle tensor = outputs.handle_list[i]; + int64_t element_num = MSTensorGetElementNum(tensor); + printf("Tensor name: %s, tensor size is %ld ,elements num: %ld.\n", MSTensorGetName(tensor), + MSTensorGetDataSize(tensor), element_num); + const float *data = (const float *)MSTensorGetData(tensor); + printf("output data is:\n"); + const int max_print_num = 50; + for (int j = 0; j < element_num && j <= max_print_num; ++j) { + printf("%f ", data[j]); + } + printf("\n"); + } +``` + +## Freeing memory + +When do not need to use MindSpore Lite inference framework, you need to free the `Model` that has been created. + +```c +// Delete model. +MSModelDestroy(&model); +``` diff --git a/docs/lite/docs/source_zh_cn/quick_start/quick_start_c.md b/docs/lite/docs/source_zh_cn/quick_start/quick_start_c.md index 21bcae32e78837c01844880d9dd945dfda09519d..44f28611977ffb16b840fd6266d09c45df6666f1 100644 --- a/docs/lite/docs/source_zh_cn/quick_start/quick_start_c.md +++ b/docs/lite/docs/source_zh_cn/quick_start/quick_start_c.md @@ -31,7 +31,7 @@ - 编译构建 - 在`mindspore/lite/examples/quick_start_c`目录下执行[build脚本](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_c/build.sh)，将自动下载MindSpore Lite推理框架库以及文模型文件并编译Demo。 + 在`mindspore/lite/examples/quick_start_c`目录下执行[build脚本](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_c/build.sh)，将自动下载MindSpore Lite推理框架库以及模型文件并编译Demo。 ```bash bash build.sh diff --git a/tutorials/experts/source_en/index.rst b/tutorials/experts/source_en/index.rst index 3bdea2066b928b912c4d0293aaafcde7ac2dd8a6..8a24f473dbaa69a822b27f8cfeae71f73934e315 100644 --- a/tutorials/experts/source_en/index.rst +++ b/tutorials/experts/source_en/index.rst @@ -24,6 +24,7 @@ For Experts network/op_overload network/custom_cell_reverse network/ms_class + network/constexpr network/dependency_control .. toctree:: diff --git a/tutorials/experts/source_en/network/constexpr.md b/tutorials/experts/source_en/network/constexpr.md new file mode 100644 index 0000000000000000000000000000000000000000..fe82c57fa9690be28f879cebc3fdaaff22e4d8cb --- /dev/null +++ b/tutorials/experts/source_en/network/constexpr.md @@ -0,0 +1,83 @@ +# Construct Constants In the Network + + + +A @constexpr python decorator is provided in `mindspore.ops.constexpr`, which can be used to decorate a function. The +function will be executed by the Python interpreter in the compiling stage, and finally will be collapsed by constants in the type derivation phase of MindSpore to become a ValueNode of the ANF graph. + +Since the function is executed in the compiling state of MindSpore, when using @constexpr to decorate a function, the +parameter of this input function must be a constant value that can be determined in compiling state of MindSpore. + +Otherwise, if the parameter is a value that cannot be determined in the compiling state, the parameter will be None, which may cause the function output to be different than expected. + +When the input parameter can be determined, you can use @constexpr to implement some operations that are not supported in the construct function, for example, operations such as creating tensor based on a certain shape. + +To avoid that @constexpr input is a value that cannot be determined in the compiling state, you can perform internal judgment processing on None to avoid some unknown errors. + +A code example is as follows: + +```python +import numpy as np +from mindspore.ops import constexpr +import mindspore.ops as ops +import mindspore.nn as nn +from mindspore import Tensor +import mindspore + + +@constexpr +def construct_tensor(x): + if x is None: + raise ValueError("input is an unknown value") + return Tensor(np.array(x), dtype=mindspore.float32) + + +class Net(nn.Cell): + def __init__(self): + super(Net, self).__init__() + self.relu = ops.ReLU() + + def construct(self, x): + return self.relu(construct_tensor(ops.shape(x))) + + +net = Net() +x = Tensor(np.random.random([7, 6, 3])) +out = net(x) +print(out) +``` + +```text +[7. 6. 3.] +``` + +As shown below, if we change Net to a value that cannot be determined in the compiling state, an exception will be thrown. Because the input of construct_tensor is a value that can be determined when ReLU is run. ValueError will be thrown in constexpr. + +```python +@constexpr +def construct_tensor(x): + if x is None: + raise ValueError("input is an unknown value") + return Tensor(np.array(x), dtype=mindspore.float32) + + +class Net(nn.Cell): + def __init__(self): + super(Net, self).__init__() + self.relu = ops.ReLU() + + def construct(self, x): + return self.relu(construct_tensor(self.relu(x))) + + +net = Net() +x = Tensor(np.random.random([7, 6, 3])) +out = net(x) +print(out) +``` + +The following information is displayed: + +```text +ValueError: input is an unknown value +``` diff --git a/tutorials/experts/source_en/network/custom_cell_reverse.md b/tutorials/experts/source_en/network/custom_cell_reverse.md index 56234806209261b043d64776309f6bdbf021793b..c993f99937660a627f634f46bfea7dade48cf44a 100644 --- a/tutorials/experts/source_en/network/custom_cell_reverse.md +++ b/tutorials/experts/source_en/network/custom_cell_reverse.md @@ -115,7 +115,7 @@ This example customizes the gradient calculation process for the `MatMul` operat The above code defines the first-order back propagation rule by customizing the `bprop` function of `Net` and gets the second-order back propagation rule by the back propagation rule of `self.relu` in the `bprop`. -2. We need the customized the back propagation function when we want to replace some forward calculate process of the Cell. For example, there is following code in the network SNN: +2. We need the customized back propagation function when we want to replace some forward calculate process of the Cell. For example, there is following code in the network SNN: ```python class relusigmoid(nn.Cell): @@ -155,5 +155,5 @@ This example customizes the gradient calculation process for the `MatMul` operat - If the number of return values of the `bprop` function is 1, the return value must be written in the tuple format, that is, `return (dx,)`. - In graph mode, the `bprop` function needs to be converted into a graph IR. Therefore, the static graph syntax must be complied with. For details, see [Static Graph Syntax Support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). -- Only support returning the gradient of the forward propagation input. not the gradient of the `Parameter`. +- Only support returning the gradient of the forward propagation input, not the gradient of the `Parameter`. - The use of `Parameter` is not supported in `bprop`. diff --git a/tutorials/experts/source_en/network/dependency_control.md b/tutorials/experts/source_en/network/dependency_control.md index 29ab4310e50561f48ff596dba4b7eeccfce869db..580b0e87c7ddac5486a30fb24af4251493311ae3 100644 --- a/tutorials/experts/source_en/network/dependency_control.md +++ b/tutorials/experts/source_en/network/dependency_control.md @@ -21,7 +21,7 @@ y = Depend(y, a) b = B(y) ``` -Please note that a special set of operators for floating point overflow state detection have hidden side effects, but are not IO side effects or memory side effects. In addition, there are strict sequencing requirements for use, i.e., before using the NPUClearFloatStatus operator, you need to ensure that the NPU AllocFloatStatus has been executed, and before using the NPUGetFloatStatus operator, you need to ensure that the NPUClearFlotStatus has been executed. Because these operators are used less, the current scenario is to keep them defined as side-effect-free in the form of Depend ensuring execution order. Examples are as follows: +Please note that a special set of operators for detecting floating point overflow state have hidden side effects, which are not IO side effects or memory side effects. In addition, there are strict sequencing requirements for use, i.e., before using the NPUClearFloatStatus operator, you need to ensure that the NPU AllocFloatStatus has been executed, and before using the NPUGetFloatStatus operator, you need to ensure that the NPUClearFlotStatus has been executed. Because these operators are used less, the current scenario is to keep their definition as in the form of side-effect-free and to ensure execution order by Depend. Examples are as follows: ```python import numpy as np diff --git a/tutorials/experts/source_en/network/ms_class.md b/tutorials/experts/source_en/network/ms_class.md index 16d39e599b93fbe1aae276e023f403da06611226..ff166e74437341a1b76fbd92d79334ca2639f4e8 100644 --- a/tutorials/experts/source_en/network/ms_class.md +++ b/tutorials/experts/source_en/network/ms_class.md @@ -6,7 +6,7 @@ In static graph mode, using ms_class to decorate a custom class, users can create and call the instance of this custom class, and obtain attributes and methods for that custom class. -ms_class applied to static graph mode, expanding the scope of support for improving static graph compilation syntax. In dynamic graph mode, that is, PyNative mode, the use of ms_class does not affect the execution logic of the PyNative mode. +ms_class is applied to static graph mode, expanding the scope of support for improving static graph compilation syntax. In dynamic graph mode, that is, PyNative mode, the use of ms_class does not affect the execution logic of the PyNative mode. This document describes how to use ms_class so that you can use ms_class functions more effectively. @@ -34,7 +34,7 @@ out = net() print(out) ``` -ms_class support custom class nesting use, custom classes and nn. Cell nesting uses scenes. It should be noted that when a class inherits, if the parent class uses ms_class, the subclass will also have the ability to ms_class. +ms_class supports nesting use of the custom class, nesting uses scenarios of custom classes and nn. Cell. It should be noted that when a class inherits, if the parent class uses ms_class, the subclass will also have the ability to ms_class. ```python import numpy as np @@ -105,7 +105,7 @@ TypeError: Decorator ms_class can only be used for class type, but got ## Overview -`mindspore.ops.composite` provides some operator combinations related to graph transformation such as `MultitypeFuncGraph` and `HyperMap`. +In the deep learning network training or inference, the end-to-end time consumption of the network is basically composed of compiling time and running time, especially in the inference scenario, the compiling time is often much larger than the running time, so optimizing the compiling performance is of great importance to improve the deployment effect of the network in real applications. In static graph mode, some scenarios can optimize network compiling performance by changing network writing style, using equivalent semantic substitution, or setting compiling options to change the compiling mechanism. -## MultitypeFuncGraph +## Optimizing Compiling Performance with HyperMap -`MultitypeFuncGraph` is used to generate overloaded functions to support different types of inputs. Users can use `MultitypeFuncGraph` to define a group of overloaded functions. The implementation varies according to the function type. First initialize a `MultitypeFuncGraph` object, and use `register` with input type as the decorator of the function to be registered, so that the object can be called with different types of inputs. For more instructions, see [MultitypeFuncGraph](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.MultitypeFuncGraph.html). +`HyperMap` is a special class. The class object construction needs to pass in the mapping function f. When calling the object needs to pass in the n-argument sequence of f, refer to [HyperMap](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.HyperMap.html) for more details. The mapping function f must be of type `MultitypeFuncGraph`. Refer to [MultitypeFuncGraph](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.MultitypeFuncGraph.html) for more details. When batch processing list elements by using for loops, web compiling performance can be optimized by `HyperMap` equivalent semantic substitution. -A code example is as follows: +A sample code that uses HyperMap to optimize compiling performance instead of a for loop is as follows: ```python -import numpy as np -from mindspore.ops import MultitypeFuncGraph -import mindspore as ms -import mindspore.ops as ops +import time +from mindspore.ops import MultitypeFuncGraph, HyperMap +from mindspore import ops +from mindspore import ms_function add = MultitypeFuncGraph('add') @add.register("Number", "Number") def add_scalar(x, y): return ops.scalar_add(x, y) -@add.register("Tensor", "Tensor") -def add_tensor(x, y): - return ops.add(x, y) +add_map = HyperMap(add) +list1 = [i for i in range(200)] +list2 = [i for i in range(200)] +@ms_function +def hyper_map_net(): + output = add_map(list1, list2) + return output + +start_time = time.time() +output = hyper_map_net() +end_time = time.time() +print("hyper map cost time:", end_time - start_time) + +@ms_function +def for_loop_net(): + out = [] + for i in range(200): + out.append(ops.scalar_add(i, i)) + return out + +start_time = time.time() +for_loop_net() +end_time = time.time() +print("for loop cost time:", end_time - start_time) +``` -tensor1 = ms.Tensor(np.array([[1.2, 2.1], [2.2, 3.2]]).astype('float32')) -tensor2 = ms.Tensor(np.array([[1.2, 2.1], [2.2, 3.2]]).astype('float32')) -print('tensor', add(tensor1, tensor2)) -print('scalar', add(1, 2)) +```text +hyper map cost time: 0.1894233226776123 +for loop cost time: 1.2634551525115967 ``` -The following information is displayed: +## Optimizing Compiling Performance with Select Operator + +When writing a network, you will often use if statements. If the condition of the if statement is a variable condition, each if statement will generate additional subgraphs. The use of if statements can be found at: [if statements](https://www.mindspore.cn/tutorials/en/master/advanced/modules/control_flow.html#if-statement). In static graph mode, the more the number of subgraphs, the longer the compiling time, so some scenarios can optimize the compiling performance by replacing the if statement equivalently with the `Select` operator. + +It should be noted that using `Select` operator to replace if statement affects the performance of the network. On the one hand, `Select` operator executes both the true and false branches, while the if statement executes only one branch, so it takes less time to run with the if statement compared to the `Select` operator; on the other hand, the `Select` operator outperforms the control flow operator generated by the if statement, and it takes more time to run with the if statement compared to the `Select` operator. Combining the above two factors, the final performance change needs to be judged based on the actual situation. In general, when the number of operators in a branch is small, it is recommended to use the `Select` operator. When the number of operators in a branch is large, it is recommended to use the if statement. + +A sample code that uses the `Select` operator instead of if statement to optimize compiling performance is as follows: + +```python +import time +from mindspore import ms_function, Tensor, ops + +@ms_function +def if_net(x, y): + out = 0 + for _ in range(100): + if x < y: + x = x - y + else: + x = x + y + out = out + x + return out + +start_time = time.time() +out = if_net(Tensor([0]), Tensor([1])) +end_time = time.time() +print("if net cost time:", end_time - start_time) + +@ms_function +def select_net(x, y): + out = x + for _ in range(100): + cond = x < y + x = ops.Select()(cond, x - y, x + y) + out = out + x + return out + +start_time = time.time() +out = select_net(Tensor([0]), Tensor([1])) +end_time = time.time() +print("select net cost time:", end_time - start_time) +``` ```text -tensor [[2.4 4.2] - [4.4 6.4]] -scalar 3 +if net cost time: 1.1603329181671143 +select net cost time: 0.483151912689209 ``` -## HyperMap +## Optimizing Compiling Performance with Compiling Cache -`HyperMap` can apply a specified operation to one or more input sequences, which can be used with `MultitypeFuncGraph`. For example, after defining a group of overloaded `add` functions, we can apply `add` operation to multiple input groups of different types. Unlike `Map`, `HyperMap` can be used in nested structures to perform specified operations on the input in a sequence or nested sequence. For more instructions, see [HyperMap](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.HyperMap.html). +If no changes are made to a network structure during training or inference, the compiling time can be reduced by using a compiling cache. The essence of the compiling cache is that it stores the compiling intermediate process file of the network model. When the network model remains unchanged, the compiling intermediate process file produced is also the same, so the intermediate process file produced by the last programming can be reused. Refer to `enable_compile_cache` in [set context](https://www.mindspore.cn/docs/en/master/api_python/mindspore/mindspore.set_context.html) for more details. -A code example is as follows: +A sample code to optimize compiling performance by enabling compiling cache is as follows: ```python -import mindspore as ms -from mindspore.ops import MultitypeFuncGraph, HyperMap -import mindspore.ops as ops +import time +import mindspore.context as context +from mindspore import Tensor, dtype +from mindspore import ms_function + +@ms_function +def func(input_x, input_y): + output = input_x + for _ in range(200): + output = input_x + input_x * input_y + output + return output + +context.set_context(enable_compile_cache=False) +x = Tensor([1], dtype.float32) +y = Tensor([2], dtype.float32) +start_time = time.time() +out = func(x, y) +end_time = time.time() +print("Disable comile_cache cost time:", end_time - start_time) +``` -add = MultitypeFuncGraph('add') -@add.register("Number", "Number") -def add_scalar(x, y): - return ops.scalar_add(x, y) +```text +Disable comile_cache cost time: 0.5485098361968994 +``` -@add.register("Tensor", "Tensor") -def add_tensor(x, y): - return ops.tensor_add(x, y) +The scenario of the above test sample is to turn the compiling cache state off. The above test sample is executed twice, and the time consumption of the first time and the second time are the following: (the actual time consumption is related to hardware environment, and the following data is for reference only) -add_map = HyperMap(add) -output = add_map((ms.Tensor(1, ms.float32), ms.Tensor(2, ms.float32), 1), (ms.Tensor(3, ms.float32), ms.Tensor(4, ms.float32), 2)) -print("output =", output) +```text +Disable comile_cache cost time: 0.5485098361968994 + +Disable comile_cache cost time: 0.4614279270172119 +``` + +When the compiling cache is turned off, the time consumption of the first time and the second time is basically close. + +```python +import time +import mindspore.context as context +from mindspore import Tensor, dtype +from mindspore import ms_function + +@ms_function +def func(input_x, input_y): + output = input_x + for _ in range(200): + output = input_x + input_x * input_y + output + return output + +context.set_context(enable_compile_cache=True, compile_cache_path="my_compile_cache") +x = Tensor([1], dtype.float32) +y = Tensor([2], dtype.float32) +start_time = time.time() +out = func(x, y) +end_time = time.time() +print("Enable comile_cache cost time:", end_time - start_time) +``` + +```text +Enable comile_cache cost time: 0.6357541084289551 +``` + +The scenario of the above test sample is to turn the compiling cache state on. The above test sample is executed twice, and the time consumption of the first time and the second time are the following: (the actual time consumption is related to hardware environment, and the following data is for reference only) + +```text +Enable comile_cache cost time: 0.6357541084289551 + +Enable comile_cache cost time: 0.09379792213439941 ``` -The following information is displayed: +When the compiling cache is turned on, the time consumption of the second execution takes about 1/7 of that of the first execution. + +## Optimizing Compiling Performance with vmap + +It is currently known that MindSpore supports the vmap, which can be used instead of a for loop to optimize compiling performance when batch data without dependencies is processed and the associated operator supports the vmap. For detailed introduction, refer to [vmap](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.vmap.html). It should be noted that vmap optimizes not only compiling performance, but also runtime performance. + +A sample code that uses vmap instead of a for loop to process batch data to optimize compiling performance is as follows: + +```python +import numpy as np +import time +from mindspore import ops +from mindspore import ms_function, Tensor + +def hswish_func(x): + return ops.HSwish()(x) + +@ms_function +def manually_batched(xs): + output = [] + for i in range(xs.shape[0]): + output.append(hswish_func(xs[i])) + return ops.stack(output) + +shape = (100, 2) +prop = 100 +x_np = (np.random.randn(*shape) * prop).astype(np.float32) +x = Tensor(x_np) +x = ops.sub(x, 0) + +start_time = time.time() +output_vmap = ops.vmap(hswish_func, in_axes=(0,))(x) +end_time = time.time() +print("vmap cost time:", end_time - start_time) + +start_time = time.time() +output_manually = manually_batched(x) +end_time = time.time() +print("for loop cost time:", end_time - start_time) +``` ```text -output = (Tensor(shape=[], dtype=Float32, value= 4), Tensor(shape=[], dtype=Float32, value= 6), 3) +vmap cost time: 0.05766916275024414 +for loop cost time: 1.9284062385559082 ``` -In this example, the input of `add_map` contains two sequences. `HyperMap` will get the corresponding elements from the two sequences as `x` and `y` for the inputs of `add` in the form of `operation(args[0][i], args[1][i])`. For example, `add(Tensor(1, mstype.float32), Tensor(3, mstype.float32))`. +In the above sample, it is equivalent to processing 100 sets of Tensor data in batch, and you can see that the performance of processing with vmap exceeds the performance of processing with a for loop by 30 times. diff --git a/tutorials/experts/source_zh_cn/network/constexpr.ipynb b/tutorials/experts/source_zh_cn/network/constexpr.ipynb index 0e570c0828577663cb647cfad26ce44243ae5fba..07ca11c639236bdcbac986f7a29ec0c89c80e874 100644 --- a/tutorials/experts/source_zh_cn/network/constexpr.ipynb +++ b/tutorials/experts/source_zh_cn/network/constexpr.ipynb @@ -73,7 +73,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "如下所示，如果我们将Net改成输入为编译时无法确定的值时，则会抛出异常。 由于construct_tensor输入为运行ReLU时才能确定的值。在constexpr中会抛出ValueError。" + "如下所示，如果我们将Net改成输入为编译时无法确定的值时，则会抛出异常。由于construct_tensor输入为运行ReLU时才能确定的值。在constexpr中会抛出ValueError。" ] }, { diff --git a/tutorials/experts/source_zh_cn/network/op_overload.ipynb b/tutorials/experts/source_zh_cn/network/op_overload.ipynb index ac550394b89ade3da71b95f5fb64ec941dc142f5..4a6a3dabfde8d1cf58228a5d38119f715eff1067 100644 --- a/tutorials/experts/source_zh_cn/network/op_overload.ipynb +++ b/tutorials/experts/source_zh_cn/network/op_overload.ipynb @@ -85,7 +85,7 @@ "\n", "需要注意的是，使用`Select`算子替换if语句会影响网络的运行性能。一方面，`Select`算子会同时执行true分支和false分支，而if语句只执行其一个分支，因此使用if运行耗时相比使用`Select`算子耗时减少；另一方面，`Select`算子性能优于if语句产生的控制流算子，使用if运行耗时相比使用`Select`算子运行耗时增加。综合上述两种因素，最终运行性能变化情况需要结合实际情况判断。一般来讲，当分支中算子数量较少，建议使用`Select`算子；当分支中算子数量较多，建议使用if语句。\n", "\n", - "一个使用Select算子替代if语句来优化编译性能的代码样例如下：\n" + "一个使用`Select`算子替代if语句来优化编译性能的代码样例如下：\n" ] }, { @@ -150,7 +150,7 @@ "\n", "在进行训练或者推理时，如果某个网络结构未作任何变更，那么可以通过使用编译缓存来缩短编译时间。编译缓存的本质是存储了网络模型的编译中间过程文件，当网络模型不变时，生产的编译中间过程文件也是一样的，因此可以复用上一次编程产生的中间过程文件，详细使用方法可参考[设置context](https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore/mindspore.set_context.html?highlight=enable_compile_cache)中的`enable_compile_cache`相关内容。\n", "\n", - "一个通过使能编译缓存优化来编译性能的代码样例如下：" + "一个通过使能编译缓存来优化编译性能的代码样例如下：" ] }, { diff --git a/tutorials/source_en/beginner/model.md b/tutorials/source_en/beginner/model.md index f82efb33561f7d98be9fddf074c9df18a812e1b8..c4368e47570a7f3fe6f2dceb671593637210ead6 100644 --- a/tutorials/source_en/beginner/model.md +++ b/tutorials/source_en/beginner/model.md @@ -1,10 +1,8 @@ -# Building a Neural Network +# Building a Network -A neural network model consists of multiple data operation layers. `mindspore.nn` provides various basic network modules. The following uses LeNet-5 as an example to first describe how to build a neural network model by using `mindspore.nn`, and then describes how to build a LeNet-5 model by using `mindvision.classification.models`. - -> `mindvision.classification.models` is a network model interface developed based on `mindspore.nn`, providing some classic and commonly used network models for the convenience of users. +A neural network model consists of multiple data operation layers. `mindspore.nn` provides various basic network modules. The following uses LeNet-5 as an example to implement handwritten digit recognition task in deep learning. ## LeNet-5 Model @@ -12,78 +10,75 @@ A neural network model consists of multiple data operation layers. `mindspore.nn ![LeNet-5](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_zh_cn/beginner/images/lenet.png) -Except the input layer, LeNet contains seven layers: three convolutional layers, two subsampling layers, and two fully-connected layers. - -## Defining a Model Class - In the preceding figure, C indicates the convolutional layer layer, S indicates the sampling layer, and F indicates the fully-connected layer. +According to the network structure of LeNet, except the input layer, LeNet contains seven layers: three convolutional layers, two subsampling layers, and two fully-connected layers. + The input size of an image is fixed at $32 \times 32$. To achieve a good convolution effect, the number must be in the center of the image. Therefore, the input $32 \times 32$ is the result after the image is filled with $28 \times 28$. Unlike the three-channel input images of the CNN network, the input images of LeNet are only normalized binary images. The output of the network is the prediction probability of digits 0 to 9, which can be understood as the probability that the input image belongs to digits 0 to 9. +## Defining a Model Class + The `Cell` class of MindSpore is the base class for building all networks and the basic unit of a network. When a neural network is required, you need to inherit the `Cell` class and overwrite the `__init__` and `construct` methods. +The `mindspore.nn` class is the base class for building all networks and the basic unit of a network. When need to customize the network, you can inherit the `nn.Cell` class and overwrite the `__init__` and `construct` methods. + +To facilitate the management and composition of more complex networks, `mindspore.nn` provides containers to manage the submodel blocks or model layers in the network, `nn.CellList` and `nn.SequentialCell`. Here, we have chosen the `nn.CellList` method. + ```python -import mindspore.nn as nn - -class LeNet5(nn.Cell): - """ - LeNet-5 network structure - """ - def __init__(self, num_class=10, num_channel=1): - super(LeNet5, self).__init__() - # Convolutional layer, where the number of input channels is num_channel, the number of output channels is 6, and the convolutional kernel size is 5 x 5. - self.conv1 = nn.Conv2d(num_channel, 6, 5, pad_mode='valid') - # Convolutional layer, where the number of input channels is 6, the number of output channels is 16, and the convolutional kernel size is 5 x 5. - self.conv2 = nn.Conv2d(6, 16, 5, pad_mode='valid') - # Fully-connected layer, where the number of inputs is 16 x 5 x 5 and the number of outputs is 120. - self.fc1 = nn.Dense(16 * 5 * 5, 120) - # Fully-connected layer, where the number of inputs is 120 and the number of outputs is 84. - self.fc2 = nn.Dense(120, 84) - # Fully-connected layer, where the number of inputs is 84 and the number of classes is num_class. - self.fc3 = nn.Dense(84, num_class) - # ReLU activation function - self.relu = nn.ReLU() - # Pooling layer - self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2) - # Multidimensional arrays are flattened into one-dimensional arrays. - self.flatten = nn.Flatten() +from mindspore import nn +from mindspore.common.initializer import Normal + +# Customize the network +class LeNet(nn.Cell): + def __init__(self, num_classes=10, num_channel=1): + super(LeNet, self).__init__() + layers = [nn.Conv2d(num_channel, 6, 5, pad_mode='valid'), + nn.ReLU(), + nn.MaxPool2d(kernel_size=2, stride=2), + nn.Conv2d(6, 16, 5, pad_mode='valid'), + nn.ReLU(), + nn.MaxPool2d(kernel_size=2, stride=2), + nn.Flatten(), + nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02)), + nn.ReLU(), + nn.Dense(120, 84, weight_init=Normal(0.02)), + nn.ReLU(), + nn.Dense(84, num_classes, weight_init=Normal(0.02))] + # Network management with CellList + self.build_block = nn.CellList(layers) def construct(self, x): - # Use the defined operation to build a forward network. - x = self.conv1(x) - x = self.relu(x) - x = self.max_pool2d(x) - x = self.conv2(x) - x = self.relu(x) - x = self.max_pool2d(x) - x = self.flatten(x) - x = self.fc1(x) - x = self.relu(x) - x = self.fc2(x) - x = self.relu(x) - x = self.fc3(x) + # Execute the network with a for loop + for layer in self.build_block: + x = layer(x) return x ``` Next, build the neural network model defined above and look at the structure of the network model. ```python -model = LeNet5() +model = LeNet() print(model) ``` ```text -LeNet5< - (conv1): Conv2d - (conv2): Conv2d - (fc1): Dense - (fc2): Dense - (fc3): Dense - (relu): ReLU<> - (max_pool2d): MaxPool2d - (flatten): Flatten<> -> +LeNet< + (build_block): CellList< + (0): Conv2d + (1): ReLU<> + (2): MaxPool2d + (3): Conv2d + (4): ReLU<> + (5): MaxPool2d + (6): Flatten<> + (7): Dense + (8): ReLU<> + (9): Dense + (10): ReLU<> + (11): Dense + > + > ``` ## Model Layers @@ -99,7 +94,7 @@ import numpy as np import mindspore as ms -# The number of channels input is 1, the number of channels of output is 6, the convolutional kernel size is 5 x 5, and the parameters are initialized using the normal operator, and the pixels are not filled. +# The number of channels input is 1, the number of channels of output is 6, the convolutional kernel size is 5 x 5, and the parameters are initialized by using the normal operator, and the pixels are not filled. conv2d = nn.Conv2d(1, 6, 5, has_bias=False, weight_init='normal', pad_mode='same') input_x = ms.Tensor(np.ones([1, 1, 32, 32]), ms.float32) @@ -115,7 +110,6 @@ print(conv2d(input_x).shape) Add the `nn.ReLU` layer and add a non-linear activation function to the network to help the neural network learn various complex features. ```python -import mindspore as ms relu = nn.ReLU() input_x = ms.Tensor(np.array([-1, 2, -3, 2, -1]), ms.float16) @@ -130,26 +124,20 @@ print(output) ### nn.MaxPool2d -Initialize the `nn.MaxPool2d` layer and down-sample the 6 x 28 x 28 array to a 6 x 7 x 7 array. +Initialize the `nn.MaxPool2d` layer and down-sample the 6 x 28 x 28 tensor to a 6 x 7 x 7 tensor. -```python -import mindspore as ms +```text max_pool2d = nn.MaxPool2d(kernel_size=4, stride=4) input_x = ms.Tensor(np.ones([1, 6, 28, 28]), ms.float32) print(max_pool2d(input_x).shape) ``` -```text -(1, 6, 7, 7) -``` - ### nn.Flatten -Initialize the `nn.Flatten` layer and convert the 1 x 16 x 5 x 5 array into 400 consecutive arrays. +Initialize the `nn.Flatten` layer and convert the 1 x 16 x 5 x 5 4D tensor into 400 consecutive 2D tensors. ```python -import mindspore as ms flatten = nn.Flatten() input_x = ms.Tensor(np.ones([1, 16, 5, 5]), ms.float32) output = flatten(input_x) @@ -166,7 +154,6 @@ print(output.shape) Initialize the `nn.Dense` layer and perform linear transformation on the input matrix. ```python -import mindspore as ms dense = nn.Dense(400, 120, weight_init='normal') input_x = ms.Tensor(np.ones([1, 400]), ms.float32) output = dense(input_x) @@ -188,37 +175,12 @@ for m in model.get_parameters(): ``` ```text -layer:backbone.conv1.weight, shape:(6, 1, 5, 5), dtype:Float32, requeires_grad:True -layer:backbone.conv2.weight, shape:(16, 6, 5, 5), dtype:Float32, requeires_grad:True -layer:backbone.fc1.weight, shape:(120, 400), dtype:Float32, requeires_grad:True -layer:backbone.fc1.bias, shape:(120,), dtype:Float32, requeires_grad:True -layer:backbone.fc2.weight, shape:(84, 120), dtype:Float32, requeires_grad:True -layer:backbone.fc2.bias, shape:(84,), dtype:Float32, requeires_grad:True -layer:backbone.fc3.weight, shape:(10, 84), dtype:Float32, requeires_grad:True -layer:backbone.fc3.bias, shape:(10,), dtype:Float32, requeires_grad:True -``` - -## Quickly Building a LeNet-5 Model - -The preceding describes how to use `mindspore.nn.cell` to build a LeNet-5 model. The built network model API is available in `mindvision.classification.models`. You can also use the `lenet` API to directly build a LeNet-5 model. - -```python -from mindvision.classification.models import lenet - -# `num_classes` indicates the number of classes, and `pretrained` determines whether to train with the trained model. -model = lenet(num_classes=10, pretrained=False) - -for m in model.get_parameters(): - print(f"layer:{m.name}, shape:{m.shape}, dtype:{m.dtype}, requeires_grad:{m.requires_grad}") -``` - -```text -layer:backbone.conv1.weight, shape:(6, 1, 5, 5), dtype:Float32, requeires_grad:True -layer:backbone.conv2.weight, shape:(16, 6, 5, 5), dtype:Float32, requeires_grad:True -layer:backbone.fc1.weight, shape:(120, 400), dtype:Float32, requeires_grad:True -layer:backbone.fc1.bias, shape:(120,), dtype:Float32, requeires_grad:True -layer:backbone.fc2.weight, shape:(84, 120), dtype:Float32, requeires_grad:True -layer:backbone.fc2.bias, shape:(84,), dtype:Float32, requeires_grad:True -layer:backbone.fc3.weight, shape:(10, 84), dtype:Float32, requeires_grad:True -layer:backbone.fc3.bias, shape:(10,), dtype:Float32, requeires_grad:True +layer:build_block.0.weight, shape:(6, 1, 5, 5), dtype:Float32, requeires_grad:True +layer:build_block.3.weight, shape:(16, 6, 5, 5), dtype:Float32, requeires_grad:True +layer:build_block.7.weight, shape:(120, 400), dtype:Float32, requeires_grad:True +layer:build_block.7.bias, shape:(120,), dtype:Float32, requeires_grad:True +layer:build_block.9.weight, shape:(84, 120), dtype:Float32, requeires_grad:True +layer:build_block.9.bias, shape:(84,), dtype:Float32, requeires_grad:True +layer:build_block.11.weight, shape:(10, 84), dtype:Float32, requeires_grad:True +layer:build_block.11.bias, shape:(10,), dtype:Float32, requeires_grad:True ``` diff --git a/tutorials/source_en/beginner/save_load.md b/tutorials/source_en/beginner/save_load.md index f5c6bfada05bc2ea3d8171dc3b9c565bbeb3cf8d..132af031f24ddbc54ab08aab90dd486ce5f5f942 100644 --- a/tutorials/source_en/beginner/save_load.md +++ b/tutorials/source_en/beginner/save_load.md @@ -4,124 +4,48 @@ The previous section describes how to adjust hyperparameters and train network models. During network model training, we want to save the intermediate and final results for fine-tuning and subsequent model deployment and inference. This section describes how to save and load a model. -## Model Training - -The following uses the MNIST dataset as an example to describe how to save and load a network model. First, obtain the MNIST dataset and train the model. The sample code is as follows: - -```python -import mindspore.nn as nn -import mindspore as ms - -from mindvision.engine.callback import LossMonitor -from mindvision.classification.dataset import Mnist -from mindvision.classification.models import lenet - -epochs = 10 # Training epochs - -# 1. Build a dataset. -download_train = Mnist(path="./mnist", split="train", batch_size=32, repeat_num=1, shuffle=True, resize=32, download=True) -dataset_train = download_train.run() - -# 2. Define a neural network. -network = lenet(num_classes=10, pretrained=False) -# 3.1 Define a loss function. -net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean') -# 3.2 Define an optimizer function. -net_opt = nn.Momentum(network.trainable_params(), learning_rate=0.01, momentum=0.9) -# 3.3 Initialize model parameters. -model = ms.Model(network, loss_fn=net_loss, optimizer=net_opt, metrics={'accuracy'}) - -# 4. Train the neural network. -model.train(epochs, dataset_train, callbacks=[LossMonitor(0.01, 1875)]) -``` - -```text -Epoch:[ 0/ 10], step:[ 1875/ 1875], loss:[0.148/1.210], time:2.021 ms, lr:0.01000 -Epoch time: 4251.808 ms, per step time: 2.268 ms, avg loss: 1.210 -Epoch:[ 1/ 10], step:[ 1875/ 1875], loss:[0.049/0.081], time:2.048 ms, lr:0.01000 -Epoch time: 4301.405 ms, per step time: 2.294 ms, avg loss: 0.081 -Epoch:[ 2/ 10], step:[ 1875/ 1875], loss:[0.014/0.050], time:1.992 ms, lr:0.01000 -Epoch time: 4278.799 ms, per step time: 2.282 ms, avg loss: 0.050 -Epoch:[ 3/ 10], step:[ 1875/ 1875], loss:[0.035/0.038], time:2.254 ms, lr:0.01000 -Epoch time: 4380.553 ms, per step time: 2.336 ms, avg loss: 0.038 -Epoch:[ 4/ 10], step:[ 1875/ 1875], loss:[0.130/0.031], time:1.932 ms, lr:0.01000 -Epoch time: 4287.547 ms, per step time: 2.287 ms, avg loss: 0.031 -Epoch:[ 5/ 10], step:[ 1875/ 1875], loss:[0.003/0.027], time:1.981 ms, lr:0.01000 -Epoch time: 4377.000 ms, per step time: 2.334 ms, avg loss: 0.027 -Epoch:[ 6/ 10], step:[ 1875/ 1875], loss:[0.004/0.023], time:2.167 ms, lr:0.01000 -Epoch time: 4687.250 ms, per step time: 2.500 ms, avg loss: 0.023 -Epoch:[ 7/ 10], step:[ 1875/ 1875], loss:[0.004/0.020], time:2.226 ms, lr:0.01000 -Epoch time: 4685.529 ms, per step time: 2.499 ms, avg loss: 0.020 -Epoch:[ 8/ 10], step:[ 1875/ 1875], loss:[0.000/0.016], time:2.275 ms, lr:0.01000 -Epoch time: 4651.129 ms, per step time: 2.481 ms, avg loss: 0.016 -Epoch:[ 9/ 10], step:[ 1875/ 1875], loss:[0.022/0.015], time:2.177 ms, lr:0.01000 -Epoch time: 4623.760 ms, per step time: 2.466 ms, avg loss: 0.015 -``` - -The preceding print result shows that the loss values tend to converge as the number of training epochs increases. - ## Saving the Model -After the network training is complete, save the network model as a file. There are two types of APIs for saving models: - -1. One is to simply save the network model before and after training. The advantage is that the interface is easy to use, but only the network model status when the command is executed is retained. -2. The other one is to save the interface during network model training. MindSpore automatically saves the number of epochs and number of steps set during training. That is, the intermediate weight parameters generated during the model training process are also saved to facilitate network fine-tuning and stop training. - -### Directly Saving the Model - -Use the save_checkpoint provided by MindSpore to save the model, pass it to the network, and save the path. +We use the save_checkpoint provided by MindSpore to save the model, pass it in the network and the save path: ```python import mindspore as ms - -# The defined network model is net, which is used before or after training. +import mindspore.nn as nn +from mindspore.common.initializer import Normal + +# Define the neural network +class LeNet(nn.Cell): + def __init__(self, num_classes=10, num_channel=1): + super(LeNet, self).__init__() + layers = [nn.Conv2d(num_channel, 6, 5, pad_mode='valid'), + nn.ReLU(), + nn.MaxPool2d(kernel_size=2, stride=2), + nn.Conv2d(6, 16, 5, pad_mode='valid'), + nn.ReLU(), + nn.MaxPool2d(kernel_size=2, stride=2), + nn.Flatten(), + nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02)), + nn.ReLU(), + nn.Dense(120, 84, weight_init=Normal(0.02)), + nn.ReLU(), + nn.Dense(84, num_classes, weight_init=Normal(0.02))] + # Network management with CellList + self.build_block = nn.CellList(layers) + + def construct(self, x): + # Execute network with a for loop + for layer in self.build_block: + x = layer(x) + return x + +# Instantiate training network +network = LeNet() + +# Save the model parameters to the specified path ms.save_checkpoint(network, "./MyNet.ckpt") ``` -In the preceding command, `network` indicates the training network, and `"./MyNet.ckpt"` indicates the path for storing the network model. - -### Saving the Model During Training - -During model training, you can use the `callbacks` parameter in `model.train` to transfer the [ModelCheckpoint](https://mindspore.cn/docs/en/master/api_python/mindspore/mindspore.ModelCheckpoint.html#mindspore.ModelCheckpoint) object (usually used together with [CheckpointConfig](https://mindspore.cn/docs/en/master/api_python/mindspore/mindspore.CheckpointConfig.html#mindspore.CheckpointConfig)) to save model parameters and generate a checkpoint (CKPT) file. - -You can set `CheckpointConfig` to configure the Checkpoint policy as required. The following describes the usage: - -```python -import mindspore as ms - -# Set the value of epoch_num. -epoch_num = 5 - -# Set the model saving parameters. -config_ck = ms.CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10) - -# Apply the model saving parameters. -ckpoint = ms.ModelCheckpoint(prefix="lenet", directory="./lenet", config=config_ck) -model.train(epoch_num, dataset_train, callbacks=[ckpoint]) -``` - -In the preceding code, you need to initialize a `CheckpointConfig` class object to set the saving policy. - -- `save_checkpoint_steps` indicates the interval (in steps) for saving the checkpoint file. -- `keep_checkpoint_max` indicates the maximum number of checkpoint files that can be retained. -- `prefix` indicates the prefix of the generated checkpoint file. -- `directory` indicates the directory for storing files. - -Create a `ModelCheckpoint` object and pass it to the `model.train` method. Then the checkpoint function can be used during training. - -The generated checkpoint file is as follows: - -```text -lenet-graph.meta # Computational graph after compiled. -lenet-1_1875.ckpt # The extension of the checkpoint file is '.ckpt'. -lenet-2_1875.ckpt # The file name indicates the epoch where the parameter is stored and the number of steps. Here are the model parameters for the 1875th step of the second epoch. -lenet-3_1875.ckpt # The file name indicates that the model parameters generated during the 1875th step of the third epoch are saved. -... -``` - -If you use the same prefix and run the training script for multiple times, checkpoint files with the same name may be generated. To help users distinguish files generated each time, MindSpore adds underscores (_) and digits to the end of the user-defined prefix. If you want to delete the `.ckpt` file, delete the `.meta` file at the same time. - -For example, `lenet_3-2_1875.ckpt` indicates the checkpoint file generated during the 1875th step of the second epoch after the script is executed for the third time. +- The `save_checkpoint` method will save the network parameters to the specified path, where `network` is the training network and `". /MyNet.ckpt"` is the path where the network model is saved. ## Loading the Model @@ -132,66 +56,20 @@ The sample code is as follows: ```python import mindspore as ms -from mindvision.classification.dataset import Mnist -from mindvision.classification.models import lenet - -# Save the model parameters to the parameter dictionary. The model parameters saved during the training are loaded. -param_dict = ms.load_checkpoint("./lenet/lenet-5_1875.ckpt") - -# Redefine a LeNet neural network. -net = lenet(num_classes=10, pretrained=False) - -# Load parameters to the network. -ms.load_param_into_net(net, param_dict) - -# Redefine an optimizer function. -net_opt = nn.Momentum(net.trainable_params(), learning_rate=0.01, momentum=0.9) - -model = ms.Model(net, loss_fn=net_loss, optimizer=net_opt, metrics={"accuracy"}) -``` - -- The `load_checkpoint` method loads the network parameters in the parameter file to the `param_dict` dictionary. -- The `load_param_into_net` method loads the parameters in the `param_dict` dictionary to the network or optimizer. After the loading, parameters in the network are stored by the checkpoint. - -### Validating the Model +# Store the model parameters in the dictionary of parameter, and the model parameters saved during the training process above are loaded here +param_dict = ms.load_checkpoint("./MyNet.ckpt") -After the preceding modules load the parameters to the network, you can call the `eval` function to verify the inference in the inference scenario. The sample code is as follows: +# Redefine a LeNet Neural Network +net = LeNet() -```python -# Call eval() for inference. -download_eval = Mnist(path="./mnist", split="test", batch_size=32, resize=32, download=True) -dataset_eval = download_eval.run() -acc = model.eval(dataset_eval) - -print("{}".format(acc)) +# Load parameters into the network and print the names of unloaded parameters +not_loaded_params = ms.load_param_into_net(net, param_dict) +print(not_loaded_params) ``` ```text -{'accuracy': 0.9866786858974359} +[] ``` -### For Transfer Learning - -For task interrupt retraining and fine-tuning scenarios, the `train` function can be called to perform transfer learning. The sample code is as follows: - -```python -# Define a training dataset. -download_train = Mnist(path="./mnist", split="train", batch_size=32, repeat_num=1, shuffle=True, resize=32, download=True) -dataset_train = download_train.run() - -# Network model calls train() for training. -model.train(epoch_num, dataset_train, callbacks=[LossMonitor(0.01, 1875)]) -``` - -```text -Epoch:[ 0/ 5], step:[ 1875/ 1875], loss:[0.000/0.010], time:2.193 ms, lr:0.01000 -Epoch time: 4106.620 ms, per step time: 2.190 ms, avg loss: 0.010 -Epoch:[ 1/ 5], step:[ 1875/ 1875], loss:[0.000/0.009], time:2.036 ms, lr:0.01000 -Epoch time: 4233.697 ms, per step time: 2.258 ms, avg loss: 0.009 -Epoch:[ 2/ 5], step:[ 1875/ 1875], loss:[0.000/0.010], time:2.045 ms, lr:0.01000 -Epoch time: 4246.248 ms, per step time: 2.265 ms, avg loss: 0.010 -Epoch:[ 3/ 5], step:[ 1875/ 1875], loss:[0.000/0.008], time:2.001 ms, lr:0.01000 -Epoch time: 4235.036 ms, per step time: 2.259 ms, avg loss: 0.008 -Epoch:[ 4/ 5], step:[ 1875/ 1875], loss:[0.002/0.008], time:2.039 ms, lr:0.01000 -Epoch time: 4354.482 ms, per step time: 2.322 ms, avg loss: 0.008 -``` \ No newline at end of file +- The `load_checkpoint` method loads the network parameters in the parameter file to the `param_dict` dictionary. +- The `load_param_into_net` method loads the parameters in the `param_dict` dictionary to the network or optimizer. After the loading, parameters in the network are stored by the checkpoint. diff --git a/tutorials/source_en/beginner/train.md b/tutorials/source_en/beginner/train.md index d1b962838087a5ede6879de54ea6e0ebace2d91f..581cd77b2c84bc150da389410238c2bacf343792 100644 --- a/tutorials/source_en/beginner/train.md +++ b/tutorials/source_en/beginner/train.md @@ -2,9 +2,108 @@ -After learning how to create a model and build a dataset in the preceding tutorials, you can start to learn how to set hyperparameters and optimize model parameters. +Model training is generally divided into four steps: -## Hyperparameters +1. Build a dataset. +2. Define a neural network. +3. Define hyperparameters, a loss function, and an optimizer. +4. Input dataset for training and evaluation. + +Through the above chapters, we have learned how to build datasets and build models. This chapter will focus on how to set up superparameters and train models. This chapter introduces training LeNet network using MNIST dataset taking MNIST dataset and LeNet network as an example. + +Load the previous code from the chapters Data Processing and Building Networks. + +```python +import os +import requests +import mindspore.dataset as ds +import mindspore.dataset.vision as vision +import mindspore.nn as nn +from mindspore.common.initializer import Normal + +requests.packages.urllib3.disable_warnings() + + +def download_dataset(dataset_url, path): + filename = dataset_url.split("/")[-1] + save_path = os.path.join(path, filename) + if os.path.exists(save_path): + return + if not os.path.exists(path): + os.makedirs(path) + res = requests.get(dataset_url, stream=True, verify=False) + with open(save_path, "wb") as f: + for chunk in res.iter_content(chunk_size=512): + if chunk: + f.write(chunk) + print("The {} file is downloaded and saved in the path {} after processing".format(os.path.basename(dataset_url), + path)) + + +train_path = "datasets/MNIST_Data/train" +test_path = "datasets/MNIST_Data/test" + +# Downloading dataset +download_dataset("https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-labels-idx1-ubyte", + train_path) +download_dataset("https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/train-images-idx3-ubyte", + train_path) +download_dataset("https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-labels-idx1-ubyte", + test_path) +download_dataset("https://mindspore-website.obs.myhuaweicloud.com/notebook/datasets/mnist/t10k-images-idx3-ubyte", + test_path) + + +def create_dataset(data_path, batch_size=16, image_size=32): + # Loading MINIST dataset + dataset = ds.MnistDataset(data_path) + + rescale = 1.0 / 255.0 + shift = 0.0 + + # Image transform operations + trans = [ + vision.Resize(size=image_size), + vision.Rescale(rescale, shift), + vision.HWC2CHW(), + ] + dataset = dataset.map(operations=trans, input_columns=["image"]) + dataset = dataset.batch(batch_size, drop_remainder=True) + + return dataset + + +# Customize networks +class LeNet(nn.Cell): + def __init__(self, num_classes=10, num_channel=1): + super(LeNet, self).__init__() + layers = [nn.Conv2d(num_channel, 6, 5, pad_mode='valid'), + nn.ReLU(), + nn.MaxPool2d(kernel_size=2, stride=2), + nn.Conv2d(6, 16, 5, pad_mode='valid'), + nn.ReLU(), + nn.MaxPool2d(kernel_size=2, stride=2), + nn.Flatten(), + nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02)), + nn.ReLU(), + nn.Dense(120, 84, weight_init=Normal(0.02)), + nn.ReLU(), + nn.Dense(84, num_classes, weight_init=Normal(0.02))] + # Network management with CellList + self.build_block = nn.CellList(layers) + + def construct(self, x): + # for loop execution network + for layer in self.build_block: + x = layer(x) + return x + + +# Define neural networks +network = LeNet() +``` + +## Hyper-parametric Hyperparameters can be adjusted to control the model training and optimization process. Different hyperparameter values may affect the model training and convergence speed. Currently, deep learning models are optimized using the batch stochastic gradient descent algorithm. The principle of the stochastic gradient descent algorithm is as follows: $w_{t+1}=w_{t}-\eta \frac{1}{n} \sum_{x \in \mathcal{B}} \nabla l\left(x, w_{t}\right)$ @@ -17,8 +116,6 @@ Batch size: specifies the size of each batch of data to be read. If the batch si Learning rate: If the learning rate is low, the convergence speed slows down. If the learning rate is high, unpredictable results such as no training convergence may occur. Gradient descent is a parameter optimization algorithm that is widely used to minimize model errors. The gradient descent estimates the parameters of the model by iterating and minimizing the loss function at each step. The learning rate is used to control the learning progress of a model during iteration. -![learning-rate](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_zh_cn/beginner/images/learning_rate.png) - ```python epochs = 10 batch_size = 32 @@ -26,105 +123,139 @@ momentum = 0.9 learning_rate = 1e-2 ``` -## Loss Functions +## Defining Loss Functions -The **loss function** is used to evaluate the difference between **predicted value** and **target value** of a model. Here, the absolute error loss function `L1Loss` is used: $\text { L1 Loss Function }=\sum_{i=1}^{n}\left|y_{true}-y_{predicted}\right|$ +The **loss function** is used to evaluate the difference between **predicted value** and **target value** of a model. Here, use `SoftmaxCrossEntropyWithLogits` to calculate the cross entropy between the predicted and true values. The code example is as follows: -`mindspore.nn.loss` provides many common loss functions, such as `SoftmaxCrossEntropyWithLogits`, `MSELoss`, and `SmoothL1Loss`. +```python +net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean') +``` -Given the predicted value and the target value, we calculate the error (loss value) between the predicted value and the target value by a loss function. The method is as follows: +## Defining Optimizer -```python -import numpy as np -import mindspore.nn as nn -import mindspore as ms +An **optimizer function** is used to compute and update the gradient. The selection of the model optimization algorithm directly affects the performance of the final model. A poor effect may be caused by the optimization algorithm instead of the feature or model design. Here, the Momentum optimizer is used. `mindspore.nn` provides many common optimizer functions, such as `Adam`, `SGD` and `RMSProp`. -loss = nn.L1Loss() -output_data = ms.Tensor(np.array([[1, 2, 3], [2, 3, 4]]).astype(np.float32)) -target_data = ms.Tensor(np.array([[0, 2, 5], [3, 1, 1]]).astype(np.float32)) +You need to build an `Optimizer` object. This object can update parameters based on the computed gradient. To build an `Optimizer`, you need to provide an optimizable parameter, for example, all trainable parametres in the network, i.e. set the entry for the optimizer to `network.trainable_params()`. -print(loss(output_data, target_data)) -``` +Then, you can set the `Optimizer` parameter options, such as the learning rate and weight decay. A code example is as follows: -```text -1.5 +```python +net_opt = nn.Momentum(network.trainable_params(), learning_rate=learning_rate, momentum=0.9) ``` -## Optimizer Functions +## Defining Training Process -An optimizer is used to compute and update the gradient. The selection of the model optimization algorithm directly affects the performance of the final model. A poor effect may be caused by the optimization algorithm instead of the feature or model design. +Firstly, we define the forward network `forward`, which outputs the loss values of the forward propagation network; then we define the single-step training process `train_step`, which achieves the update of the network weights by back propagation. Finally, the training process `train_epoch` is defined, and the training dataset and model are passed in to realize the training of a single epoch. -All optimization logic of MindSpore is encapsulated in the `Optimizer` object. Here, the Momentum optimizer is used. `mindspore.nn` provides many common optimizers, such as `Adam`, `SGD` and `RMSProp`. +```python +import mindspore as ms +import mindspore.ops as ops -You need to build an `Optimizer` object. This object can retain the current parameter status and update parameters based on the computed gradient. To build an `Optimizer`, you need to provide an iterator that contains parameters (must be variable objects) to be optimized. For example, set parameters to `net.trainable_params()` for all `parameter` that can be trained on the network. -Then, you can set the `Optimizer` parameter options, such as the learning rate and weight decay. A code example is as follows: +# Forward network +def forward(inputs, targets): + outputs = network(inputs) + loss = net_loss(outputs, targets) + return loss -```python -from mindspore import nn -from mindvision.classification.models import lenet -net = lenet(num_classes=10, pretrained=False) -optim = nn.Momentum(net.trainable_params(), learning_rate, momentum) -``` +grad_fn = ops.value_and_grad(forward, None, net_opt.parameters) -## Model Training -Model training consists of four steps: +# Defining a single-step training process to update the network weights +def train_step(inputs, targets): + loss, grads = grad_fn(inputs, targets) + net_opt(grads) + return loss -1. Build a dataset. -2. Define a neural network. -3. Define hyperparameters, a loss function, and an optimizer. -4. Enter the epoch and dataset for training. -The model training sample code is as follows: +# Defining a single-epoch training process +def train_epoch(dataset, network): + # Setting the network to training mode + network.set_train() + # Training the network by using dataset + for data in dataset.create_dict_iterator(): + loss = train_step(ms.Tensor(data['image'], ms.float32), ms.Tensor(data["label"], ms.int32)) + + return loss +``` + +## Defining Verification Process + +Define the verification process `evaluate_epoch`, pass in the dataset to be verified and the trained network, and calculate the accuracy of the network classification. ```python -import mindspore.nn as nn -import mindspore as ms +import numpy as np -from mindvision.classification.dataset import Mnist -from mindvision.classification.models import lenet -from mindvision.engine.callback import LossMonitor -# 1. Build a dataset. -download_train = Mnist(path="./mnist", split="train", batch_size=batch_size, repeat_num=1, shuffle=True, resize=32, download=True) -dataset_train = download_train.run() +def evaluate_epoch(dataset, network): + size = dataset.get_dataset_size() + accuracy = 0 -# 2. Define a neural network. -network = lenet(num_classes=10, pretrained=False) -# 3.1 Define a loss function. -net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean') -# 3.2 Define an optimizer function. -net_opt = nn.Momentum(network.trainable_params(), learning_rate=learning_rate, momentum=momentum) -# 3.3 Initialize model parameters. -model = ms.Model(network, loss_fn=net_loss, optimizer=net_opt, metrics={'acc'}) + for data in dataset.create_dict_iterator(): + output = network(ms.Tensor(data['image'], ms.float32)) + acc = np.equal(output.argmax(1).asnumpy(), data["label"].asnumpy()) + accuracy += np.mean(acc) + accuracy /= size -# 4. Train the neural network. -model.train(epochs, dataset_train, callbacks=[LossMonitor(learning_rate, 1875)]) + return accuracy +``` + +## Model Training and Evaluation + +After defining the training process and validation process, the model can be trained and evaluated after loading the training dataset and validation dataset. + +```python +import mindspore as ms +import mindspore.ops as ops +import time + +# Loading training dataset +dataset_train = create_dataset(train_path, batch_size) +# Loading test dataset +dataset_test = create_dataset(test_path, batch_size) + +for epoch in range(epochs): + start = time.time() + result = train_epoch(dataset_train, network) + acc = evaluate_epoch(dataset_test, network) + end = time.time() + time_ = round(end - start, 2) + + print("-" * 20) + print(f"Epoch:[{epoch}/{epochs}], " + f"Train loss:{result.asnumpy():5.4f}, " + f"Accuracy:{acc:5.3f}, " + f"Time:{time_}s, ") ``` ```text -Epoch:[ 0/ 10], step:[ 1875/ 1875], loss:[0.189/1.176], time:2.254 ms, lr:0.01000 -Epoch time: 4286.163 ms, per step time: 2.286 ms, avg loss: 1.176 -Epoch:[ 1/ 10], step:[ 1875/ 1875], loss:[0.085/0.080], time:1.895 ms, lr:0.01000 -Epoch time: 4064.532 ms, per step time: 2.168 ms, avg loss: 0.080 -Epoch:[ 2/ 10], step:[ 1875/ 1875], loss:[0.021/0.054], time:1.901 ms, lr:0.01000 -Epoch time: 4194.333 ms, per step time: 2.237 ms, avg loss: 0.054 -Epoch:[ 3/ 10], step:[ 1875/ 1875], loss:[0.284/0.041], time:2.130 ms, lr:0.01000 -Epoch time: 4252.222 ms, per step time: 2.268 ms, avg loss: 0.041 -Epoch:[ 4/ 10], step:[ 1875/ 1875], loss:[0.003/0.032], time:2.176 ms, lr:0.01000 -Epoch time: 4216.039 ms, per step time: 2.249 ms, avg loss: 0.032 -Epoch:[ 5/ 10], step:[ 1875/ 1875], loss:[0.003/0.027], time:2.205 ms, lr:0.01000 -Epoch time: 4400.771 ms, per step time: 2.347 ms, avg loss: 0.027 -Epoch:[ 6/ 10], step:[ 1875/ 1875], loss:[0.000/0.024], time:1.973 ms, lr:0.01000 -Epoch time: 4554.252 ms, per step time: 2.429 ms, avg loss: 0.024 -Epoch:[ 7/ 10], step:[ 1875/ 1875], loss:[0.008/0.022], time:2.048 ms, lr:0.01000 -Epoch time: 4361.135 ms, per step time: 2.326 ms, avg loss: 0.022 -Epoch:[ 8/ 10], step:[ 1875/ 1875], loss:[0.000/0.018], time:2.130 ms, lr:0.01000 -Epoch time: 4547.597 ms, per step time: 2.425 ms, avg loss: 0.018 -Epoch:[ 9/ 10], step:[ 1875/ 1875], loss:[0.008/0.017], time:2.135 ms, lr:0.01000 -Epoch time: 4601.861 ms, per step time: 2.454 ms, avg loss: 0.017 +[EVENT] (PID29692) 2022-09-08-10:55:46.568.557 Start compiling 'after_grad' and it will take a while. Please wait... +[EVENT] (PID29692) 2022-09-08-10:55:46.695.443 End compiling 'after_grad'. +[EVENT] (PID29692) 2022-09-08-10:55:46.723.607 Start compiling 'Momentum.construct' and it will take a while. Please wait... +[EVENT] (PID29692) 2022-09-08-10:55:46.764.006 End compiling 'Momentum.construct'. +[EVENT] (PID29692) 2022-09-08-10:55:51.390.778 Start compiling 'LeNet.construct' and it will take a while. Please wait... +[EVENT] (PID29692) 2022-09-08-10:55:51.428.831 End compiling 'LeNet.construct'. +-------------------- +Epoch:[0/10], Train loss:2.3203, Accuracy:0.113, Time:5.26s, +-------------------- +Epoch:[1/10], Train loss:0.0699, Accuracy:0.956, Time:4.84s, +-------------------- +Epoch:[2/10], Train loss:0.0157, Accuracy:0.976, Time:4.63s, +-------------------- +Epoch:[3/10], Train loss:0.0570, Accuracy:0.974, Time:4.54s, +-------------------- +Epoch:[4/10], Train loss:0.0255, Accuracy:0.986, Time:4.64s, +-------------------- +Epoch:[5/10], Train loss:0.2859, Accuracy:0.986, Time:4.74s, +-------------------- +Epoch:[6/10], Train loss:0.0005, Accuracy:0.985, Time:4.67s, +-------------------- +Epoch:[7/10], Train loss:0.0166, Accuracy:0.989, Time:4.73s, +-------------------- +Epoch:[8/10], Train loss:0.0046, Accuracy:0.988, Time:4.61s, +-------------------- +Epoch:[9/10], Train loss:0.0003, Accuracy:0.987, Time:4.67s, ``` -During the training, the loss value is printed and fluctuates. However, the loss value gradually decreases and the precision gradually increases. Loss values displayed each time may be different because of their randomness. +The loss values are printed during the training process, and the loss values fluctuate, but in general the loss values gradually decrease and the accuracy increases. The loss values of each individual are random and not necessarily the same. \ No newline at end of file diff --git a/tutorials/source_zh_cn/beginner/model.ipynb b/tutorials/source_zh_cn/beginner/model.ipynb index c8761a692b373b8b6501b14b1540e7495d92346d..b9d577cd4147a6c51d608ef25ec6ba3ce1ffd0a7 100644 --- a/tutorials/source_zh_cn/beginner/model.ipynb +++ b/tutorials/source_zh_cn/beginner/model.ipynb @@ -163,7 +163,7 @@ "\n", "import mindspore as ms\n", "\n", - "# 输入的通道数为1,输出的通道数为6,卷积核大小为5*5,使用normal算子初始化参数，不填充像素\n", + "# 输入的通道数为1，输出的通道数为6，卷积核大小为5*5，使用normal算子初始化参数，不填充像素\n", "conv2d = nn.Conv2d(1, 6, 5, has_bias=False, weight_init='normal', pad_mode='same')\n", "input_x = ms.Tensor(np.ones([1, 1, 32, 32]), ms.float32)\n", "\n",