From 1ae3025a5fba57adcffee59ec9b407c0c1778dfd Mon Sep 17 00:00:00 2001 From: zhangyi Date: Fri, 19 Aug 2022 16:24:53 +0800 Subject: [PATCH] update the English files --- .../source_en/advanced/linear_fitting.md | 469 ++++++++++++++++++ .../source_en/advanced/pynative_graph.rst | 9 + .../advanced/pynative_graph/combine.md | 303 +++++++++++ .../pynative_graph/images/framework1.png | Bin 0 -> 27920 bytes .../pynative_graph/images/ms_function.png | Bin 0 -> 14519 bytes .../source_en/advanced/pynative_graph/mode.md | 275 ++++++++++ .../advanced/pynative_graph/pynative.md | 139 ++++++ tutorials/source_en/advanced/train.rst | 8 +- .../source_en/advanced/train/callback.md | 333 +++++++++++++ .../source_en/advanced/train/images/model.png | Bin 0 -> 11090 bytes tutorials/source_en/advanced/train/metric.md | 205 ++++++++ tutorials/source_en/advanced/train/model.md | 386 ++++++++++++++ .../source_en/advanced/train/train_eval.md | 336 +++++++++++++ tutorials/source_en/index.rst | 2 + 14 files changed, 2463 insertions(+), 2 deletions(-) create mode 100644 tutorials/source_en/advanced/linear_fitting.md create mode 100644 tutorials/source_en/advanced/pynative_graph.rst create mode 100644 tutorials/source_en/advanced/pynative_graph/combine.md create mode 100644 tutorials/source_en/advanced/pynative_graph/images/framework1.png create mode 100644 tutorials/source_en/advanced/pynative_graph/images/ms_function.png create mode 100644 tutorials/source_en/advanced/pynative_graph/mode.md create mode 100644 tutorials/source_en/advanced/pynative_graph/pynative.md create mode 100644 tutorials/source_en/advanced/train/callback.md create mode 100644 tutorials/source_en/advanced/train/images/model.png create mode 100644 tutorials/source_en/advanced/train/metric.md create mode 100644 tutorials/source_en/advanced/train/model.md create mode 100644 tutorials/source_en/advanced/train/train_eval.md diff --git a/tutorials/source_en/advanced/linear_fitting.md b/tutorials/source_en/advanced/linear_fitting.md new file mode 100644 index 0000000000..c8f76ac5fa --- /dev/null +++ b/tutorials/source_en/advanced/linear_fitting.md @@ -0,0 +1,469 @@ +# Case: Linear Fitting + + + +MindSpore provides high-level, medium-level, and low-level APIs. For details, see [API Level Structure](https://www.mindspore.cn/tutorials/en/master/beginner/introduction.html#api-level-structure). + +To facilitate the control of the network execution process, MindSpore provides the high-level training and inference API `mindspore.Model`. By specifying the neural network model to be trained and common training settings, MindSpore calls the `train` and `eval` methods to train and infer the network. In addition, if you want to customize a specific module, you can call the corresponding medium- and low-level APIs to define the network training process. + +The following uses the medium- and low-level APIs provided by MindSpore to fit linear functions. + +$$f(x) = 2x + 3 \tag {1}$$ + +Before initializing the network, you need to configure the `context` parameter to control the program execution policy. For example, configure the static graph or dynamic graph mode and configure the hardware environment for network running. + +The following describes how to configure information and use medium- and low-level APIs provided by MindSpore to customize loss functions, optimizers, training processes, metrics, and evaluation processes. + +## Configuration Information + +Before initializing the network, you need to configure the `context` parameter to control the program execution policy. For example, configure the static graph or dynamic graph mode and configure the hardware environment for network running. Before initializing the network, you need to configure the `context` parameter to control the program execution policy. The following describes the execution mode management and hardware management. + +### Execution Mode + +MindSpore supports two running modes: Graph and PyNative. By default, MindSpore uses the Graph mode, and the PyNative mode is used for debugging. + +- Graph mode (static graph mode): The neural network model is built into an entire graph and then delivered to the hardware for execution. This mode uses graph optimization to improve the running performance and facilitates large-scale deployment and cross-platform running. + +- PyNative mode (dynamic graph mode): Operators in the neural network are delivered to the hardware one by one for execution. This mode facilitates code writing and neural network model debugging. + +MindSpore provides a unified encoding mode for static and dynamic graphs, significantly enhancing compatibility between both types of graphs. This enables you to switch between the static and dynamic graph modes by changing only one line of code, eliminating the need to develop multiple sets of code. When switching the mode, pay attention to the [constraints](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html) of the target mode. + +Set the running mode to dynamic graph mode. + +```python +import mindspore as ms + +ms.set_context(mode=ms.PYNATIVE_MODE) +``` + +Similarly, when MindSpore is in dynamic image mode, you can run the `set_context(mode=GRAPH_MODE)` command to switch to the static graph mode. + +```python +import mindspore as ms + +ms.set_context(mode=ms.GRAPH_MODE) +``` + +### Hardware Management + +Hardware management involves the `device_target` and `device_id` parameters. + +- `device_target`: target device to be run. The value can be `Ascend`, `GPU`, or `CPU`. You can set this parameter based on the actual situation or use the default value. + +- `device_id`: ID of the target device. The value is in the range of [0,`device_num_per_host` - 1]. `device_num_per_host` indicates the total number of devices on the server. The value of `device_num_per_host` cannot exceed 4096. The default value of `device_id` is 0. + +> When the program is executed in non-distributed mode, you can set `device_id` to determine the ID of the device where the program is executed to avoid device usage conflicts. + +A code example is as follows: + +```Python +import mindspore as mst + +ms.set_context(device_target="Ascend", device_id=6) +``` + +## Processing Datasets + +### Generating a Dataset + +Define the dataset generation function `get_data` to generate the training dataset and test dataset. + +Since linear data is fitted, the required training datasets should be randomly distributed around the objective function. Assume that the objective function to be fitted is $f(x)=2x+3$. $f(x)=2x+3+noise$ is used to generate training datasets, and `noise` is a random value that complies with standard normal distribution rules. + +```python +import numpy as np + +def get_data(num, w=2.0, b=3.0): + for _ in range(num): + x = np.random.uniform(-10.0, 10.0) + noise = np.random.normal(0, 1) + y = x * w + b + noise + yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32) +``` + +Use get_data to generate 50 groups of evaluation data and visualize the data. + +```python +import matplotlib.pyplot as plt + +train_data = list(get_data(50)) +x_target_label = np.array([-10, 10, 0.1]) +y_target_label = x_target_label * 2 + 3 +x_eval_label, y_eval_label = zip(*train_data) + +plt.scatter(x_eval_label, y_eval_label, color="red", s=5) +plt.plot(x_target_label, y_target_label, color="green") +plt.title("Eval data") +plt.show() +``` + +![png](output_8_0.png) + +In the figure shown above, the green line indicates the target function, and the red point indicates the evaluation data (`train_data`). + +### Loading a Dataset + +Loads the dataset generated by the `get_data` function to the system memory and performs basic data processing operations. + +- `ds.GeneratorDataset`: converts the generated data into a MindSpore dataset and saves the x and y values of the generated data to arrays of `data` and `label`. +- `batch`: combines `batch_size` pieces of data into a batch. +- `repeat`: Multiplies the number of data sets. + +```python +from mindspore import dataset as ds + +def create_dataset(num_data, batch_size=16, repeat_size=1): + input_data = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label']) + input_data = input_data.batch(batch_size, drop_remainder=True) + input_data = input_data.repeat(repeat_size) + return input_data +``` + +Use the dataset augmentation function to generate training data. Use the defined `create_dataset` to augment the generated 1600 pieces of data into 100 datasets with the shape of 16 x 1. + +```python +data_number = 1600 +batch_number = 16 +repeat_number = 1 + +ds_train = create_dataset(data_number, batch_size=batch_number, repeat_size=repeat_number) +print("The dataset size of ds_train:", ds_train.get_dataset_size()) +step_size = ds_train.get_dataset_size() +dict_datasets = next(ds_train.create_dict_iterator()) + +print(dict_datasets.keys()) +print("The x label value shape:", dict_datasets["data"].shape) +print("The y label value shape:", dict_datasets["label"].shape) +``` + +```text + The dataset size of ds_train: 100 + dict_keys(['data', 'label']) + The x label value shape: (16, 1) + The y label value shape: (16, 1) +``` + +## Defining a Network Model + +The `mindspore.nn` class is the base class for setting up all networks and the basic unit of a network. In order to customize a network, you can inherit the `nn.Cell` class and overwrite the `__init__` and `construct` methods. + +The `mindspore.ops` module provides the implementation of basic operators. The `nn.Cell` module further encapsulates basic operators. You can flexibly use different operators as required. + +The following example uses `nn.Cell` to build a simple fully-connected network for subsequent customized content. In MindSpore, use `nn.Dense` to generate a linear function model with a single data input and a single data output. + +$$f(x)=wx+b\tag{2}$$ + +Use the Normal operator to randomly initialize the $w$ and $b$ parameters in formula (2). + +```python +from mindspore import nn +from mindspore.common.initializer import Normal + +class LinearNet(nn.Cell): + def __init__(self): + super(LinearNet, self).__init__() + self.fc = nn.Dense(1, 1, Normal(0.02), Normal(0.02)) + + def construct(self, x): + fx = self.fc(x) + return fx +``` + +After initializing the network model, visualize the initialized network function and training dataset to understand the model function before fitting. + +```python +import mindspore as ms + +net = LinearNet() # Initialize the linear regression network. + +model_params = net.trainable_params() # Obtain network parameters w and b before training. + +x_model_label = np.array([-10, 10, 0.1]) +y_model_label = (x_model_label * model_params[0].asnumpy()[0] + model_params[1].asnumpy()[0]) + +plt.axis([-10, 10, -20, 25]) +plt.scatter(x_eval_label, y_eval_label, color="red", s=5) +plt.plot(x_model_label, y_model_label, color="blue") +plt.plot(x_target_label, y_target_label, color="green") +plt.show() +``` + +![png](output_16_0.png) + +## Customized Loss Functions + +A loss function is used to measure the difference between the predicted value and the actual value. In deep learning, model training is a process of reducing a loss function value through continuous iteration. Therefore, it is very important to select a loss function in a model training process. Defining a good loss function can help the loss function value converge faster and achieve better accuracy. + +[mindspore.nn](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html#loss-function) provides many common loss functions for users to select and allows users to customize loss functions as required. + +When customizing a loss function class, you can inherit the base class `nn.Cell` of the network or the base class `nn.LossBase` of the loss function. `nn.LossBase` is based on `nn.Cell` and provides the `get_loss` method. The `reduction` parameter is used to obtain a sum or mean loss value and output a scalar. The following describes how to define the mean absolute error (MAE) function by inheriting LossBase. The formula of the MAE algorithm is as follows: + +$$ loss= \frac{1}{m}\sum_{i=1}^m\lvert y_i-f(x_i) \rvert \tag{3}$$ + +In the preceding formula, $f(x)$ indicates the predicted value, $y$ indicates the actual value of the sample, and $loss$ indicates the mean distance between the predicted value and the actual value. + +When using the method inherited from LossBase to define the loss function, you need to rewrite the `__init__` and `construct` methods and use the `get_loss` method to compute the loss. The sample code is as follows: + +```python +from mindspore import nn, ops + +class MyMAELoss(nn.LossBase): + """Define the loss.""" + def __init__(self): + super(MyMAELoss, self).__init__() + self.abs = ops.Abs() + + def construct(self, predict, target): + x = self.abs(target - predict) + return self.get_loss(x) +``` + +## Customized Optimizer + +During model training, the optimizer is used to compute and update network parameters. A proper optimizer can effectively reduce the training time and improve model performance. + +[mindspore.nn](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html#optimizer) provides many common optimizers for users to select and allows users to customize optimizers as required. + +When customizing an optimizer, you can inherit the optimizer base class `nn.Optimizer` and rewrite the `__init__` and `construct` methods to update parameters. + +The following example implements the customized optimizer Momentum (SGD algorithm with momentum): + +$$ v_{t+1} = v_t × u+grad \tag{4}$$ + +$$p_{t+1} = p_t - lr × v_{t+1} \tag{5}$$ + +$grad$, $lr$, $p$, $v$, and $u$ respectively represent a gradient, a learning rate, a weight parameter, a momentum parameter, and an initial speed. + +```python +import mindspore as ms +from mindspore import nn, ops + +class MyMomentum(nn.Optimizer): + """Define the optimizer.""" + + def __init__(self, params, learning_rate, momentum=0.9): + super(MyMomentum, self).__init__(learning_rate, params) + self.moment = ms.Parameter(ms.Tensor(momentum, ms.float32), name="moment") + self.momentum = self.parameters.clone(prefix="momentum", init="zeros") + self.assign = ops.Assign() + + def construct(self, gradients): + """The input of construct is gradient. Gradients are automatically transferred during training.""" + lr = self.get_lr() + params = self.parameters # Weight parameter to be updated + for i in range(len(params)): + self.assign(self.momentum[i], self.momentum[i] * self.moment + gradients[i]) + update = params[i] - self.momentum[i] * lr # SGD algorithm with momentum + self.assign(params[i], update) + return params +``` + +## Customized Training Process + +`mindspore.Model` provides `train` and `eval` APIs for users to use during training. However, this API does not apply to all scenarios, such as multi-data and multi-label scenarios, where users need to define the training process. + +The following uses linear regression as an example to describe the customized training process. First, define the loss network and connect the forward network to the loss function. Then, define the training process. Generally, the training process inherits `nn.TrainOneStepCell`. `nn.TrainOneStepCell` encapsulates the loss network and optimizer to implement the backward propagation network to update the weight parameters. + +### Defining a Loss Network + +Define the loss network `MyWithLossCell` to connect the feedforward network to the loss function. + +```python +class MyWithLossCell(nn.Cell): + """Define the loss network.""" + + def __init__(self, backbone, loss_fn): + """Transfer the feedforward network and loss function as parameters during instantiation.""" + super(MyWithLossCell, self).__init__(auto_prefix=False) + self.backbone = backbone + self.loss_fn = loss_fn + + def construct(self, data, label): + """Connect the feedforward network and loss function.""" + out = self.backbone(data) + return self.loss_fn(out, label) + + def backbone_network(self): + """Backbone network to be encapsulated.""" + return self.backbone +``` + +### Defining the Training Process + +Define the training process `MyTrainStep`. This class inherits `nn.TrainOneStepCell`. `nn.TrainOneStepCell` encapsulates the loss network and optimizer. During training, the `ops.GradOperation` operator is used to obtain the gradient, the optimizer is used to update the weight. + +```python +class MyTrainStep(nn.TrainOneStepCell): + """Define the training process.""" + + def __init__(self, network, optimizer): + """Initialize parameters.""" + super(MyTrainStep, self).__init__(network, optimizer) + self.grad = ops.GradOperation(get_by_list=True) + + def construct(self, data, label): + """Build the training process.""" + weights = self.weights + loss = self.network(data, label) + grads = self.grad(self.network, weights)(data, label) + return loss, self.optimizer(grads) +``` + +The following defines the drawing function `plot_model_and_datasets` to draw the test data, objective function, and network model fitting function, and view the loss value. + +```python +from IPython import display +import matplotlib.pyplot as plt +import time + +def plot_model_and_datasets(net, data, loss): + weight = net.trainable_params()[0] + bias = net.trainable_params()[1] + x = np.arange(-10, 10, 0.1) + y = x * ms.Tensor(weight).asnumpy()[0][0] + ms.Tensor(bias).asnumpy()[0] + x1, y1 = zip(*data) + x_target = x + y_target = x_target * 2 + 3 + + plt.axis([-11, 11, -20, 25]) + plt.scatter(x1, y1, color="red", s=5) # Raw data + plt.plot(x, y, color="blue") # Predicted data + plt.plot(x_target, y_target, color="green") # Fitting function + plt.title(f"Loss:{loss}") # Printed loss value + + plt.show() + time.sleep(0.2) + display.clear_output(wait=True) +``` + +### Training + +Use the training data `ds_train` to train the training network `train_net` and visualize the training process. + +```python +loss_func = MyMAELoss () # Loss function +opt = MyMomentum(net.trainable_params(), 0.01) # Optimizer + +net_with_criterion = MyWithLossCell(net, loss_func) # Build a loss network. +train_net = MyTrainStep(net_with_criterion, opt) # Build a training network. + +for data in ds_train.create_dict_iterator(): + train_net(data['data'], data['label']) # Perform training and update the weight. + loss = net_with_criterion(data['data'], data['label']) # Compute the loss value. + plot_model_and_datasets(train_net, train_data, loss) # Visualize the. +``` + +![png](output_28_0.png) + +## Customized Evaluation Metrics + +When a training task is complete, an evaluation function (Metric) is often required to evaluate the quality of a model. Common evaluation metrics include confusion matrix, accuracy, precision, and recall. + +The [mindspore.nn](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html#evaluation-metrics) module provides common evaluation functions. You can also define evaluation metrics as required. The customized Metric function needs to inherit the `nn.Metric` parent class and re-implement the `clear`, `update`, and `eval` methods in the parent class. The following formula shows the mean absolute error (MAE) algorithm. The following uses MAE as an example to describe the three functions and their usage. + +$$ MAE=\frac{1}{n}\sum_{i=1}^n\lvert y\_pred_i - y_i \rvert \tag{6}$$ + +- `clear`: initializes related internal parameters. +- `update`: receives network prediction output and labels, computes errors, and updates internal evaluation results. Generally, the calculation is performed after each step and the statistical value is updated. +- `eval`: computes the final evaluation result after each epoch ends. + +```python +class MyMAE(nn.Metric): + """Define metrics.""" + + def __init__(self): + super(MyMAE, self).__init__() + self.clear() + + def clear(self): + """Initialize variables abs_error_sum and samples_num.""" + self.abs_error_sum = 0 + self.samples_num = 0 + + def update(self, *inputs): + """Update abs_error_sum and samples_num.""" + y_pred = inputs[0].asnumpy() + y = inputs[1].asnumpy() + + # Compute the absolute error between the predicted value and the actual value. + error_abs = np.abs(y.reshape(y_pred.shape) - y_pred) + self.abs_error_sum += error_abs.sum() + self.samples_num += y.shape[0] # Total number of samples + + def eval(self): + """Compute the final evaluation result."" + return self.abs_error_sum / self.samples_num +``` + +## Customized Evaluation Process + +The mindspore.nn module provides the evaluation network packaging function [nn.WithEvalCell](https://www.mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.WithEvalCell.html#mindspore.nn.WithEvalCell). Because `nn.WithEvalCell` has only two inputs `data` and `label`, it is not applicable to the scenario with multiple data or labels. Therefore, you need to customize the evaluation network. For details about how to customize the evaluation network in the multi-label scenario, see [Customized Training and Evaluation Networks](https://www.mindspore.cn/tutorials/en/master/advanced/train/train_eval.html). + +The following example implements a simple customized evaluation network `MyWithEvalCell`. Enter the input `data` and `label`. + +```python +class MyWithEvalCell(nn.Cell): + """Define the evaluation process.""" + + def __init__(self, network): + super(MyWithEvalCell, self).__init__(auto_prefix=False) + self.network = network + + def construct(self, data, label): + outputs = self.network(data) + return outputs, label +``` + +Perform inference and evaluation: + +```python +data_number = 160 +batch_number = 16 +repeat_number = 1 + +# Obtain evaluation data. +ds_eval = create_dataset(data_number, batch_size=batch_number, repeat_size=repeat_number) + +eval_net = MyWithEvalCell(net) # Define the evaluation network. +eval_net.set_train(False) +mae = MyMAE() + +# Execute the inference process. +for data in ds_eval.create_dict_iterator(): + output, eval_y = eval_net(data['data'], data['label']) + mae.update(output, eval_y) + +mae_result = mae.eval() +print("MAE: ", mae_result) +``` + +```text + MAE: 0.9605088472366333 +``` + +Output evaluation error. The effect of MAE is similar to that of the model in the training set. + +## Saving and Exporting a Model + +Save the trained model parameters to a checkpoint (CKPT) file, and export the checkpoint file as a MindIR file for cross-platform inference. + +```python +import numpy as np +import mindspore as ms + +ms.save_checkpoint(net, "./linear.ckpt") # Save model parameters in a CKPT file. +param_dict = ms.load_checkpoint("./linear.ckpt") # Save the model parameters to the param_dict dictionary. + +# View the model parameters. +for param in param_dict: + print(param, ":", param_dict[param].asnumpy()) + +net1 = LinearNet() +input_np = np.random.uniform(0.0, 1.0, size=[1, 1]).astype(np.float32) +ms.export(net1, ms.Tensor(input_np), file_name='linear', file_format='MINDIR') +``` + +```text + fc.weight : [[1.894384]] + fc.bias : [3.0015702] +``` diff --git a/tutorials/source_en/advanced/pynative_graph.rst b/tutorials/source_en/advanced/pynative_graph.rst new file mode 100644 index 0000000000..ec5134115b --- /dev/null +++ b/tutorials/source_en/advanced/pynative_graph.rst @@ -0,0 +1,9 @@ +Dynamic and Static Graphs +=================== + +.. toctree:: + :maxdepth: 1 + + pynative_graph/mode + pynative_graph/combine + pynative_graph/pynative diff --git a/tutorials/source_en/advanced/pynative_graph/combine.md b/tutorials/source_en/advanced/pynative_graph/combine.md new file mode 100644 index 0000000000..c7b23a3e02 --- /dev/null +++ b/tutorials/source_en/advanced/pynative_graph/combine.md @@ -0,0 +1,303 @@ +# Combination of Dynamic and Static Graphs + + + +Currently, dynamic and static graphs are supported in the industry. Dynamic graphs are executed through explanation, with dynamic syntax affinity and flexible expression. Static graphs are executed through just in time (JIT) build, which focuses on static syntax and has many syntax constraints. The build process of the dynamic graph is different from that of the static graph. As a result, the syntax constraints are also different. + +For dynamic and static graph modes, MindSpore first unifies the API expression and uses the same APIs in the two modes. Then, it unifies the underlying differentiation mechanism of dynamic and static graphs. + +![dynamic](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_en/advanced/pynative_graph/images/framework1.png) + +## Implementation Principle + +MindSpore allows you to use the `ms_function` modifier to modify objects that need to be executed using static graphs, achieving combination of dynamic and static graphs. The following uses a simple combination example to describe the implementation principle. The sample code is as follows: + +```python +import numpy as np +import mindspore.nn as nn +import mindspore as ms +from mindspore import ms_function + +class Add(nn.Cell): + """Define a class to implement the x self addition.""" + def construct(self, x): + x = x + x + return x + +class Mul(nn.Cell): + """Define a class to implement the x self multiplication.""" + @ms_function # Use ms_function to modify the function. This function is executed in static graph mode. + def construct(self, x): + x = x * x + return x + +class Test(nn.Cell): + """Define a class to implement Add(x), Mul(x), and then Add(x).""" + def __init__(self): + super(Test, self).__init__() + self.add = Add() + self.mul = Mul() + + def construct(self, x): + x = self.add(x) + x = self.mul(x) + x = self.add(x) + return x + +ms.set_context(mode=ms.PYNATIVE_MODE) +x = ms.Tensor(np.ones([3, 3], dtype=np.float32)) +print("init x:\n", x) +net = Test() +x = net(x) +print("\nx:\n", x) +``` + +```text + init x: + [[1. 1. 1.] + [1. 1. 1.] + [1. 1. 1.]] + + x: + [[8. 8. 8.] + [8. 8. 8.] + [8. 8. 8.]] +``` + +According to the preceding information, after the test operation, the final value of x is a 3\*3 matrix whose each element is 8. The following figure shows the build method of this test case according to the execution sequence. + +![msfunction](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_en/advanced/pynative_graph/images/ms_function.png) + +Functions modified by `ms_function` are built and executed in static graph mode. If the network involves reverse derivation, the part modified by `ms_function` is also used to generate a backward graph in the form of an entire graph. The backward graph is connected to backward graphs of operators before and after the graph and then delivered for execution. The cache policy is the same as that of the static graph. When the input shape and type information of the same function object is the same, the built graph structure is cached. + +## `ms_function` Modifier + +To improve the execution speed of forward computing tasks in dynamic graph mode, MindSpore provides the `ms_function` modifier. You can modify Python functions or member functions of Python classes to build them into computational graphs. Technologies such as graph optimization are used to improve the running speed. + +### Usage + +MindSpore supports static build in dynamic graphs. You can use the `ms_function` modifier to modify the function objects that need to be executed using static graphs to implement mixed execution of dynamic and static graphs. + +#### 1. Modifying Independent Function + +When using the `ms_function` modifier, you can modify an independently defined function so that it can run in static graph mode. The following is an example: + +```python +import numpy as np +import mindspore.ops as ops +from mindspore import ms_function + +# Set the running mode to dynamic graph mode. +ms.set_context(mode=ms.PYNATIVE_MODE) + +# Use the modifier to specify the execution in static graph mode. +@ms_function +def add_func(x, y): + return ops.add(x, y) + +x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) +y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) + +out = add_func(x, y) +print(out) +``` + +```text + [5. 7. 9.] +``` + +In the preceding sample code, although the running mode is set to dynamic graph mode at the beginning, the `add_func(x, y)` function is modified using the `ms_function` modifier. Therefore, the `add_func(x, y)` function still runs in static graph mode. + +#### 2. Modifying the Member Functions of a Cell + +When using the `ms_function` modifier, you can modify the member functions of the `Cell`. The sample code is as follows: + +```python +import numpy as np +import mindspore.nn as nn +import mindspore.ops as ops +import mindspore as ms +from mindspore import ms_function + +# Set the running mode to dynamic graph mode. +ms.set_context(mode=ms.PYNATIVE_MODE) + +class Add(nn.Cell): + + @ms_function # Use the modifier to specify the execution in static graph mode. + def construct(self, x, y): + out = x + y + return out + +x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) +y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) + +grad_ops = ops.GradOperation(get_all=True) # Define the derivation operation. +net = Add() +grad_out = grad_ops(net)(x, y) + +print("Infer result:\n", net(x, y)) + +print("Gradient result:") +print("Grad x Tensor1:\n", grad_out[0]) # Derivation of x +print("Grad y Tensor2:\n", grad_out[1]) # Derivation of y +``` + +```text + Infer result: + [5. 7. 9.] + Gradient result: + Grad x Tensor1: + [1. 1. 1.] + Grad y Tensor2: + [1. 1. 1.] +``` + +According to the preceding information, the sum of x and y is \[5, 7, 9\]. The derivation result of x is the same as that of y, that is, \[1, 1, 1\]. + +### Precautions + +When using `ms_function` to modify functions to improve execution efficiency, pay attention to the following points: + +1. Functions modified by `ms_function` must be within the syntax scope supported by static graph build, including but not limited to data types. + +2. The control flow syntax supported by a function modified by `ms_function` is the same as that supported by the static graph. An acceleration effect is achieved only for a control flow structure with a fixed loop count or a branch condition. + +3. When the `ms_function` function is used in PyNative mode, the parts that are not modified by `ms_function` support breakpoint debugging, and the parts modified by `ms_function` do not support breakpoint debugging because they are built in static graph mode. + +4. Functions modified by `ms_function` are built and executed in static graph mode. Therefore, `ms_function` does not support the Hook operator in the modified functions or the customized Bprop function. + +5. Functions modified by `ms_function` are affected by side effects of static graph functions. Side effects of a function refer to the additional effects on the main function in addition to the return value of the function, for example, modifying global variables (variables other than the function) and modifying function parameters. + + Scenario 1: + + ```python + import numpy as np + import mindspore as ms + from mindspore import ms_function + + # pylint: disable=W0612 + + value = 5 + + @ms_function + def func(x, y): + out = x + y + value = 1 + return out + + ms.set_context(mode=ms.PYNATIVE_MODE) + x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) + y = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) + func(x, y) + print(value) + ``` + + ```text + 5 + ``` + + In this scenario, `value` is a global variable and is modified in the `func` function. In this case, if `ms_function` is used to modify the `func` function, the global variable `value` is not changed. The reason is that statements irrelevant to return values are optimized during static graph build. + + Scenario 2: + + ```python + import numpy as np + import mindspore.nn as nn + import mindspore as ms + from mindspore import ms_function + + class Func(nn.Cell): + def __init__(self): + super(Func, self).__init__() + self.value = 5 + + @ms_function + def construct(self, x): + out = self.value + x + return out + + ms.set_context(mode=ms.PYNATIVE_MODE) + x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) + func = Func() + print("out1:", func(x)) + func.value = 1 + print("out2:", func(x)) + ``` + + ```text + out1: [6. 7. 8.] + out2: [6. 7. 8.] + ``` + + According to the preceding information, after the member variable `value` of the `func` class is changed to 1, the `construct` operation of the member function is not affected. In this scenario, `ms_function` is used to modify the `construct` member function of the `func` object. When `construct` is executed, it is built and executed in static graph mode. The static graph caches the build result. Therefore, when `func` is called for the second time, the modification of `value` does not take effect. + +6. If a function with the `ms_function` modifier contains operators (such as `MatMul` and `Add`) that do not require parameter training, these operators can be directly called in the modified function. If the modified function contains operators (such as `Conv2D` and `BatchNorm` operators) that require parameter training, these operators must be instantiated outside the modified function. The following uses sample code to describe the two scenarios. + + Scenario 1: Directly call an operator (`mindspore.ops.Add` in the example) that does not require parameter training in the modified function. The sample code is as follows: + + ```python + import numpy as np + import mindspore as ms + import mindspore.ops as ops + from mindspore import ms_function + + ms.set_context(mode=ms.PYNATIVE_MODE) + + add = ops.Add() + + @ms_function + def add_fn(x, y): + res = add(x, y) + return res + + x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) + y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) + z = add_fn(x, y) + + print("x:", x.asnumpy(), "\ny:", y.asnumpy(), "\nz:", z.asnumpy()) + ``` + + ```text + x: [1. 2. 3.] + y: [4. 5. 6.] + z: [5. 7. 9.] + ``` + + Scenario 2: The operator (`mindspore.nn.Conv2d` in the example) that requires parameter training must be instantiated outside the modified function. The sample code is as follows: + + ```python + import numpy as np + import mindspore.nn as nn + import mindspore as ms + from mindspore import ms_function + + ms.set_context(mode=ms.PYNATIVE_MODE) + + # Instantiate the conv_obj operator in the conv_fn function. + conv_obj = nn.Conv2d(in_channels=3, out_channels=4, kernel_size=3, stride=2, padding=0) + conv_obj.init_parameters_data() + + @ms_function + def conv_fn(x): + res = conv_obj(x) + return res + + input_data = np.random.randn(1, 3, 3, 3).astype(np.float32) + z = conv_fn(ms.Tensor(input_data)) + print(z.asnumpy()) + ``` + + ```text + [[[[ 0.00829158 -0.02994147] + [-0.09116832 -0.00181637]] + + [[-0.00519348 -0.02172063] + [-0.04015012 -0.02083161]] + + [[ 0.00608188 -0.01443425] + [-0.01468289 0.01200477]] + + [[ 0.00845292 0.00044869] + [-0.00361492 0.01993337]]]] + ``` diff --git a/tutorials/source_en/advanced/pynative_graph/images/framework1.png b/tutorials/source_en/advanced/pynative_graph/images/framework1.png new file mode 100644 index 0000000000000000000000000000000000000000..b49791d8eed61c164c8df9d087be7ae34e2c7f51 GIT binary patch literal 27920 zcmdqJbyQT*|28^+iV|WW-7PKMNH<6~(j5W=(kKEV(%qfIfW*+DBHhCb9U?Uh!bmr~ z2fy!c-B@?sweEX=_n$j!&DwkR+2`!u`}sVd=S-BAh9V&zIUWcEB2-qA(*}WXB0wN) z-uw4}l*dD*-azocL&?Ys1j2uE`^ADU248|ev>;`?7F5(`L%E%PX zGpj$~B8q-f-VZ}MEhkFJc(~nb@(GQuTeX&VTJO*9M}b}sIv6bO3%KqbsV6(I;(fJ^ z4s);4m+{I86PEUHD*ap$P(1F1=Og zv<&Y6?E|uS{{JQ9J4isKHHDqe?*tYT+%|33}>!&OMhu0+~{#J8CxhIjKB8w-{5SPU=dOuM+Ps=(AKC3@DQMOsBrX+^tT7pU5#O^vzMwYy|4d9&mGFd zF_$LiSDY*#Isg4?5oSNS!5k+)fS6yrPUqZQmRM;Y_B7v(nmI6pqzT&2bvMW?vrXp* z<5e05SNdFl2dFz5p*ovPTWV#O zEc2)YCA-Uy5u-m{y-e*Ub$3}#97BZTC_*v|Hye$_6!D*;lAp)56PX`j#h4mhfvu*t@NfHHbeS-Z8Z$Odd<;Dfv&k>$wxWV+v9Cc=(RtVCK0*z5G~>vN*!?w@uvTqN4Zf#0D8~R?-fI@yAsmj6wiI+9j&YsrsP& z1=ruGfh@4gW}mXF6-tRUUI%Kv?=~LK{cMKtT9Ou%Is74VVBJkcr(dUCrz`H*VM>bc zD74o+)_9KAJ(?4`7rG?*W~9D>(}l0)4{^X^s_VGs?+qK~cJm#a6vp$w5!;{&hM5rsAuBmZ9`D3&Fq6p=qo{8X9iG8a3X|jd#Nu_0O)&8YM zbrC#7J}2dVBi^InJE3s-H}^qghF=@>;h&D?eH*0h1>Ol?2889tJ@8lSSYE97&X=eQ zx%u0bjl8+I`7)C`eK%)r&v{4KzRT|gp&P_RvC?2Xt7p;TQ-=gQF94)g> z>(YALuJk2pv&kl6yA==N6TbAe#33EI%?+G&Gtv5Pt@kMt+5vdyW}(64l0I>!V_ zS69wI%E9%0Ctdc3W-d~vdLe%fQ=@g&JLBgyBq#`3IzL(aeJ4_Ufy|%QoNUg1b`Yhw z8FW$D7aj1|ys^%RduMB0mBJay$w0nvoE*rKls&gXl0?n2+G5OtTZrF#=>V;?K}FHX zAMo@$h89-_4~I z8g4d~2Z06HnEzFOD3wzW`7Vx=wMwt!gmY)%K)>N(_7`;`)pjPN1k@Fnz#onh1=H35m-o&<8Sq$UtP|yww6yP>~HDYwN-Q3`m z;EfK23fZli%u$$EhNx%d8j|nqDN8nRbx!yCWD^9K70U8MUnUYuUnqE+Pto&pYDAte zv7Kz^OEZryxPHpJo)7e*YDl=5 zP?oTWaD^@s6I?XfW_vNY(|Ws>s*BNaLYwmeCiqnT>TkIHx|pNJZje{2g?4yN)+V>j zoGc+|S1t}Xo2+c1=5y^Vb`$sk?&%MSPza4U{Q*R?E9utLv+PwH3{v?J@9y@BB*OXW zj_q({x;fkyo|5z))qP!Oy!8bsmb^vJ|6nlaoZoqKf-@;Hi|g}Pbit;@_IP{{*W%#A zEAg#1iuy;(Lu5L*3JoDwI&*nzkOFG{xXcb1IRVjAiKpbFBIU-&fDm#cMn%t7izo8B zLRg>*-~^Y}ev9?JZ~!72(a1uoQF-d;>U8xF5T9*&^0meeV$5m$=a<;A zNVbmQ8O$j`aVZSrU~l7Q&H5<|9sD}HBFlhFE})u^s0!;_&TS8{xTvn>?J44XROZ%* z3({H&p?y@Z)Q|Yd8Ct(%)Zxnc=p`@pd~`~eCG_Dkesd|^6P^-_kIZpa`w2D^NuTP^ zM;_J$KDd_0mb`$l>HCzol&CK$J<9$Kd#dg7MwB`pvw5s1o^x$U*Z45|M4|*5^geys zIU9Z8QXN7z ziOC9eft&~0?o*=#XjvDk_|Xe<1oQIRm5i(u&c1u}u}81T#-f#(Y}tj#$TUA<#1T!0 z8|ow~g;+xhPcwTNV{`I!Qo zqRxg_1{{!wE^3W`)u5@aRZ}0{Dz7F!>D({SKU$KK!!ZPEH89PO97(4e$vV*6#LB3-*EHs(!rB2MTNOY{y&p@iU30P~Zs^aVRBaek62Cpd(tj-NVs zI=kS7zgpr~iyGq&Z^Y~ePLa282u(x)88r8I_H_6j>f)YG37Mo@YwK@ssm-%N*+ojs z#8m&bOeTzP^%ptl^H39j!V_AEC*PJ^@FqV4dD;}Zn|SMurcgO)Jbk9(cmIC(=M`ci zd2WPrb4ZrO%31Ch$zLSCNDcnO?a;~94UQ1Ms;shya+~&37&_KFp=ysvvA-qXJv@%Z z3Rl3*MgFp!_fU!WnsV<#{98Bo;ECF~x%o4;3v6kM<=>*%(hS+c+qVf#=TuCt;QmieL@?C!kn_kDdR5x}tDACr?9Ijt za&`;1dC!^HfjCWojdClQzMncotBy)f%qNn$2sgCX`ExyrsStMh835p@`+$;#FzuBfc zdXmvEA5<_(m-nwJgelf^&RY#^RPavhZhIf+fBdG;!dc@ z;FCGBt1S2GXW5NrecnDx1c}Ao&^9SPl_+W$f_Zpp-!xhfv#4|!fzMA3t8aSh7?8xk-u_ohJcwcsw zWlF?UZj(Q_(w=;*!hNuiPr4@_DNe`3VYv}y=Cnhu?B6YP0xQ=GYW>4k+8`C5N)~^z z{`@FTrLsaVhA}4AL04N{CvWNSG3}2Ql*9|&GzCRyq%pFxBB-sg!PYQOP3Qg7(l2UV zX;2v|5;9)kQ81~Vhw4GC;w9h45mAf5z@rFVscp&0@7=qsG?L-BYAPR9m6t8eL^>l&hSF+K`|FlRjWy#f)ochBuNvdFSm!xKI5E1p|5KlDPo=ei zw_mxeKw_r9>wKUNYA1=2%t_2gUN+6wmhzbj7UAKv)72aY4Qhl4tO4IIxpU7pW>aL1 z^u1*TWJ}9E0D5lHG&IpcH+l7*5!;-dRlHGkFGN>JJPTaaWQ2BqeQLBsaHndev=sZj zn_tHwZ?`72qiU=X!V>S6PO|VlzRY#H_?$63LiMhH--PA7ez8|b%5TeFMJK6fT6#6jzz1z{d zCYB1XMC?AKMg6Oa-#-qwGG1zaQBE8+`&1J)fObV6$Uh$Zv9c%vrjD+dKbU|iZnIQv|t(5b}&+|1hdW|aXgS-4L>zH6-P z=+)2jC)YD*QpK@OdEU68xikChqFa~-N}z8JXlX#vZK@8pzxy_jyC zZ(Mk1R{#t>6)vd>HcfthTjoV8vfd4bHjYCr%tRLKIBv%YB)gxyDc6vL8|AkyT9Z#2 zfnn7eGdR@b3dVV!GY+kX!{5w10ueQ@m5a<8MbitHfCK*-Ti=*hsm33-iQT7fh3J9F zVRFLUd|i9wWTq*Nk<)`ZGS!`rMDAn)M2mrgnn++ps%1lU!SmS{5mgI2y#tM=2RwJTx%PBNveY8tzC-j~MJ)&;wZcdD={B-n}et15MzO#{_*mXog8 z^yhR@Cbk`4VkJEi^`rKMiiuLmd97{*Kaxh&dagf}>RZ#~{A&#=SDLAH>qae3vn%EF z(Z?}I4;42!+$tIuFX)v`x|Jd+32xLcf}V*Kdb3bS%z$OtKLjr91nxy$g}ML2rkF zMaaJ9V4R>Kdssjy_cS{Iy2>rjGo-O%oFGR6HBlTr`){u0 z>Vm&`iD{9@+`1vg;aOju=^kTPmm~Ct^G)BjktuCfYue|Er;3Vjg0ZpZM~#8RCkNRR zjb`fm&b0acT37A*U^TM$mMyh}Y^42t*P)C9MVvb@?HjHm#xotSHx^|ItC9I?b8P%$ zw84%(-StMDLSni_^_heH(#aIg<`(Lajqhv-TOfKDschoocBhNV;0baCFn=%|ZnLZ) znA^5BOjIb0j4WbsW%ZuwL@A#3voZ*qkTUzZ)YYMIk} z<*NaErt{(tBwD-Pf4QNnfA%`w`Nglr%Ib+38<_*^Cs%Lj7JYRP^49g{j)@9wjT**- z5|_^VtA*mlH31oTgbX9A-F0F9(eSiWztK$0a>&B5=;s~dY4n&TGc#<5E;nMFJ<|9> zHpKANTY4T2Rq8YUtb=AL+HmvV;#Et`Tc+DiNKVuIaD0;T>S}qjP5p9T)ae>ATtSTC zg0sqbqNVg9Ayqx~F;0M<6%8dWr>$fGRv7J2`8)H%zz|BnGwQbdP;RNnT^5D_GMeB?|b91 zaB}>N(l0h*rOIWdwmKM7lTF5qQus9SWU?b^Qp9!1RdraagK-q&0@J`-)yF421C&S! z#Rp(#v&D4t7V)5v`>ICn(Uf3bd=W^)>)3gpyK81Mg?f@?pfWBFM~at+j6 zzpkh_d1s_f#MalU3xB_o(-uC8H8=sSslSjsddXWD{sNzIjbG9Q*c>Wdyit@C-Wnp#BWQ^x)>Kv^`=9@Fy zy?bY%|3b)3vt#M?X1x0?My-QJTPzN|R-k25pLZkOZ*b+BmkP8oA&GviT%&T1q}n=% z?zgeBX3zVU)YM(#KLX$?fSS|wI*R@`#2W0!qC_P4Ke4AMPjYco*B%XJ%K>1IX+&gb zajzh9+6vASIRSJT4xmeUF0>U7YtTzVcICqMb#$kpi;5&qyO+WF-(?IMJ@T?$G+pp} z{^4HQtE(f(0)hd}l-=ylS59) z#AB_eQeAlHW%qmFS7vgaXI0TX#T?k(Urj91Hs!RE1<_2U6yS$Fv7(=? z9ucU7v{50J-AF`Pq~Efh6%^l^M4^8GW9~Nmf9;vM^~iWS9!tQpGl(;T;&fl5hCv`t zde*ItX?V>1tHn+e)Iza>*d5S%S@K+gOS1eMVf#}4n@+q%sc1V=(4gM`&e)N$XMVWz z2$bAqXGl2xK(=XUi=9kvgk;muWLc2l9xta*gyXDc&RnY^vu{ z($)kZxBc-B;!Hh8C<*YGW=&q0Y>O_u>`>Y^+%EMHFc4qr(QPed^gNE5c!(8x@k~c7 zS(RxqTW?7RMF|4oS!yJxeaxrA2Kl~7^eyFq zt|QUpDF}hmrJ*+6O6WLZ+z0DUga-*9EoO+bfTC*zh5A(dIHvJjU@Aaf#MK!KWQ(ps z2Fn38Qo0HTSLpA^_N5s0C2@940MP6SOp$?{Dy1RtvyosMth!pKC~4o`Varpwh^-Sp zzbfUBQkIqv%Wu2zl_g-)L_|b@fy$XtlTT^6M`k*SBBZtMgZh=Aa{-Y8HZUfE#Nx+b zq^kjKwC;tk@6fRNJ*vIaYQvqvd4GH?r9;vm^251i=%!$|&lqBU0<&X@4fJWy$ z*Und`Cuq*LhA9Hn&u?#I!hNfOeuq($DOB<0n}KZlgGew6npc*R=yaeYXlCr)Uh6=}gmdaY5Pe_z8_B)6j+=6ZQv z;IBJy`+9T+{hP|ZHUn#ww;?n;h#`q1(T)812?Akk=@uaEA;>s*3Y}A3=9=uyFaHk)>93rRQ(f5>qn0&GmjBgd$etZFL z2om1Gk#Wh5D~BwnjsG+$@13fL)SzCH%d|&%PEub|G;6FD<}?1~|vW zvgEO3E@HXW*zCdn1dLns7uzpFp${;px(LN1L1aYYj(dX8qpm#q3qr!9iVB4X@+a>9 ztlrLB46=R7t%-?e1NEnU3tc|1&C%*KBW`m)Z;bZAcawQ@f7+-M zF?9*>4DY2Ex_&mDl`gT1Zp<^~xz~&5j_ku+Q~P*q0N$aS!J-Bk8cRJ9C8vz=c0kQ= z!TiMc8+PDXE4?BU*T+E;SumzLO!|QWD|3vQe+k?7<3^}4jobaPAcqpF@z5iZ-1OYMfd&mvbsG+h+Y9NS>RYV{&_u3)6bn zDQ1D939{2?Qgkw^NMNtAzW zmU3Q8lRVgIkbpqJ+>e*I7fzh`JM#M3`)_;wro-v9bDdU;K)E^z<}=JJ?8h^3s-3w? zuwpmO7rBkLFBwnRKjN^;Cs<~r0&}{Lh;tGe}uo^IDC}s;K%Ht81aYKP>8)C(W)0 zxt53@k2F-DJwdmN!B>zET;MKey>Hxcx>^dK)LyFfS)0i9)l7xs%h+~Rzs4uw(lSfs z{%|kU$_^O4bb0P9az6j4K%4JL$`9;knEl9$*i##gPgcS!FkK1lamHTT%vXL4&Q|x`S}R(%Fd~1iw$>gdJ>7AN%4J^*bnZ`JchOq;gx)u zFTR8+0?d|}L~!XD$DvG)mY~cHObwg9^W3mgk4$+7;W^%mN%EW`< zo9=(zqm|exQYI6(GLpZlPNBGUN+Q?|2jT8Ug1MChU}-fUdZ;A~D?qTf+TkRh} zu1N>7v2OL)|GD0QOeX&t;^*Y%4NP-#h@YtF+&X6dzU-+owRW z)33Ic$wqNNtd9UpEM4&;`hOz)ivMo!M^;9h943#+(M?`5@m976GH4E&I;3Lfe!l{^ z6ug)DbN_i;-k^cg;Px_?A|O)v?+AtXXrhA8x5`=rPTbZ`&JH%WSL38&XCqzLYhAA| z&l1-vU3;{4U92hQ@!Fs>yQ-75P}0F1R~Ks&%O>LaAV<+gV%Zj!x#M49L{7U4ErTV1 zUX9Ow^veT+?)fJ>eS(cCbB$I5xbri;f{p5%|Hr7}Wg{7%r_|qinZys@e{nNi(Qj7Z z+_viTV$H3R>c69X|EvAxG?-r`zVwlJX>_a&53-^-V$FBM{H*=>x)?sCl1K zo$Dmm{px|80*#$_ufr0onNY+F72}U4RbPZj4>Dk0X%_Hdfxa;Uve8gw=7?BU{+=3r zZa*Q1{^^Q?#Cst$@xJO0ak7%kcZ4agnR>J)2G6W)&aXQw*?r{xKsv z*Xp3oB`i46L1ROgB%^l2TwzQgdHK{Y?ZH_67-V`$g55DJ*z+rt;Da+rTJn}d_ABLxAfbPuE%eZ>Y&0*y`*hkrAc@8P;8NpiNzfF$s+m(Jwp0Y&7K8p0KwoB%2%sQ( zSGf}V)lv7%^vEaQ*mT;2bsk)}&OPSy2Fq%Z#KI;{TMJ3{oLE0`zIvEi;G3 zdv0COX&rdR;~n8Z8_)cJfn$lX=;7=`r)|NIg_el@wh&uytSq_Bv!5ld6JACwukhhC zgrHyA!@PGuKSf2}aMCkyUQ8J44U9jPs)RLuad7A#o_Mcz1>u~l24jrgq`)edXR6LqCdNiDj<ezc%NhP&LREvL-n%bw}iKY0Y)vP_yCYP&C~z8*bu3sep8HO5-c z6qKmtpPLt&SuENRh}t)Tq#gf>dPuRTOGV+}UnCKx9j1FwNX(jt?22Xo9zSKbBaV)C zTeiVBtYna5?|X$pUzaY00Rj0aU9srK9OT4+<$mo0|LCK zE1^~s#>nu1u~F~Ayec6s4EA|ZkjC%Q8S~!CR>BEtv2F09B$rD^e%9n-c~uzWzG{8& z6F$t?c%f3w3ZO`1QmLU6{A-LT&2#y3?Y{@t4KTDG0~}15q(aaTxUWk`^MeChLOv$e zJ0Rg#fOLbFz7_SQ!6n0c-G-LjloOINYS{6%#rz{PwGVnfd! z5KxVnaYRvnpOFWra*WDhyZPstj-GRSvq5Dx%+LDIGO zIsDrLHV}tuN3EdKO;Zk;`f_!tlPQH@iI7{>*xi~XNwQB|4W2Lan|#F~PyzFt&wk(t z{$@ZU6a+_nERNa(v{8lMT~T@zlfN#*9YV$bqy5q0dnaVg*nyQpE+oK99ZBy+AmAW((Zpn!Y-<}CoQvd{Cb(5DW{%#W0YqpT_O?2( zcj3jTGmIgo$vn$a%3!MCq<~8jE!?RH8g+ zFX}4jp5tckjF(i%cilBpA!d_zrTRmH`p)0q>CfH;N&i#uhp=stw5S|TQoOnqtwYv8 zAU8fYg-Cm@m)f{~2|eJajlFYAo#9hFvCg+P9am>B4ULeSG+esZVe??2US8a-_ zPahU2`NKaz>&0HWs?~LSvgV%_3`C@O+>J8#G8HtNc$EzkWN*0D1iS zZC`(T#$jT_?iilRVOvcD%0Il7$j4Uzx;6FVYDEx8<E|D4N3S5xCj_6xv`%`!(l-sz!bI9ae%U`a^mI&CbQ2sn84ft01P0wAeF z+oHWATzN=zwq=ukhq8nSFtl&=uFAD{l>loSka&#jr@Wqy>+_YDHBlpr2zs{_DQbUZ zSl?+*-5vjFXp}TwQ73NJ~p< z2HH5h7_S#vp_Ht*Z##}r{twYjz?c6mib3B)ps?toaVWE*jI(oump7FRH5K)~wtth* zBzsJc4`(q!B$VwWD_IIqvNU3-7TIf#{JM>maqOW)g?=?|GnInu*1UaCfWr0oOX^jn zMHU`Xvs9T*!-%)^6U9T{ zT>O9Ql#7k-!Irkd!eGOWRh`h2>k+HQGw3v3$Z^5y=Nu=8EZ=EF*-#@_j}o~66}f1m zQ6|6dm{pM5&oYuO6P>)+l0S87*Jf;OmoNz5+uXTGaFx1pECna`MeSUgZzV+P_+ZU1 zj`P4lbi~PGw+UXGV}&6;6OYsn}5`(zI8VJ?7=GCg9lR%Y)PX8!q zwBINkc;@Yi21l}IIyx-IRZmId5<&ZP9Rw^opMny zILBE(z&+1DK#bmRciDAkJHP5FZ>pP?PQW|261`v%6|q(4tX)bh5K|QJ=8DQkGHPbS zdb$SGXK|;mg<&qw3wR0Q#8nOCf@{pT6Z%#YSYGn;;2!k7rlu>*Z)rDD3V=j`!h5{0LRtraJq zW6yjBHs@WSQyG5bm5%zDS~I7EcWZktQ&}t0uHI^;*~#E1|LE0+IrWzPJmULDf+miU z`@FD*G7s0uYz9z1|G)130QmT2yYDyy^i_r(>$rnM>iN}~*+=G=HLs035!5z@5YyNf z1TYxe3{$J zAnMC?wG?aX;aap0js>WQTNR`BqZtaaSdqEq!q)47Hbwx;bbUb^D@Sd1D%U|tjfMX) ziu;)ns4*m&oKTCBHTw4kXo?@=*y3HioHt!m+R+FKtpiA95ECuWBSLH^CnswuN|^uG z^Emcj2W!_Q>k2TZ;*VIlBF}@H-Xt8+&+fNhrkE)QXzQrOcxNjt&0e!Us{f`MR@D<( z7ylh*o6ZVK7TN%|jfwg(9Ez!0>ptrn!f+DMp0jUZ(`4QS&%T!Xy&w29jMjv44Sdu( z&Gj5Uof)pzzxMl{;6~{Gq6_!u?;e&uEIIUhoV2-$ z-B921U$vGW{WEJjf&)RG!yZsi#P~y-3TsgG{1H>tMBI4riq`O$+kFsiEcTRW=S!8x zi=!s4@m4U9$U{jmOCnkeH=ZsJ$2XzJ=>Ack%DE+a6F!x<@Mj?IjbCtr*_`gFfdh{8#kZWQJXNMRRYp)Rhm`_#8r_e<6y|Ggmys+0X+ zB#R96xtsIeZf5RY>~}d&?P!+#nhqa*2s}Gf+u&`S+gN4|_ohTa8{aOcx7&WH3wAf| z&AG+Vh~`bb-5vMmqU^WT7#yDF>ne!rtk)63K)Jm2_X{~95`zPN-@v=`{xAUc2?!KV zaG$b}K^Q*C!sUcD|HY+i3z1d7!Br2TW*(h&) z@%oKXg?(iwd>hvtJ^qz4Z@Wjz3|U~Qtoj0Z{HL7*7`=apSuL!_`~0_hPABQjc-s58 zu5qdBQ*PsHYaLw4QM1v{7A0RQIg&hFXNfaR%_57p*7B>@261T|556*vOpOWt#Cdg}$OO5%WQVu7>l^TMR-G7__7Sm|~6 zoId7vIeU}$uW{sf9E2wK$Nl%S=NY~g!6&`iSGzGUB$^l-jt;*^DZl?)x=F3JPX?mZ z0al?FPukL%l0@T~H_(?d5+Qh3>o>JD;>GIA_+zzg)whFD#-uC+zU)^#*Gkl_Xl#tM z+ITQKsf84W1%>23?<2I17AZZtu%@6!@`d=M=An+)yvpF~3@gEoLyr6H<*GEM)0i9Y z#TRjR`UWj$OGf9{_P-E39!6y%9*x1r5s&@)KU3Fh$EgRpQF%26H_~0#JNpL-`wwdcr=n|RYx{)x z138(4%FdM9Em}Q4=a7K6sC1+nQdAEO;a+~MZQg#-++)Kv0|w*)H>DbVDHtzv*_z<7 zfxT=>zPBwjJNk%Wn!y5`8HWmndp(F45ve_$z&%{RCxToM&YzSO{duCS zm78IalqH&zAIPwoxbZG_lnI`7lplQD>kHm(T1K`ArB=>gXbq1Kec8{gzE*Lp2`h5m zqRo*j+6;$2^W?SF9^A&{hpScts~Q9qhTmjott*$F%?&{ofo8^-?D}!kwiOg zs%@Ln*JI-O*EjWJcIwTf)VuNXt;&0@>(3uVK*cu>gYSSQKLRf-hBvLf0{cb;IBz%( z;7`khbTm;P=H^OiM$>d~u^p0J^>s>dyWL*^6B>u~gj|NdQesB_mlojf-)0qwjulG@ z$;%cEi}upxkjwGn+?vhb{!&ZPYnzr=)qD#FJetNq0uAvIK^G@>LxDF=<-X3pOSv3# zpNowudP_m8;gH4^zc^cMx8(rV>XTCK8h@yzZ-VuD*`Fs-oMYSPn`)CWY|DPeg*8?F zzy6$yh^BRo)*g3Mu7>26+2`hz^&GI5=`t%YE>~=0tdyfC&2SZ`;jcuKX2zI|r=K_J zZP`C9__aP>z>PTWXwDVH-AtZQ4|0&VkBh`<{IZDHzOh0#|LN$fKdQe;ZWi*`EAo2w zBCf_JxT>P#dAH@+b^y_qy=ryD;Ev`ru!1(&O26d2la2wooI-SM<_$vLa6jKNmi zBxboLS@N_{cvle=PY1vk+x0yrFI_65I;r!6wHD>O!+k7GE<_zV#6ppd17ZHvD&!QM znfoRcYmh&9Z`BiNymAGvd@887Gd>N|Nl09_2x?|p3+MQaUBSHR`S0*4|1#5>5*H6D z2)Qg90~;H_PI4j-y`SR`6{i}VabI8K_+P$ZwFsjTe{sd;c6Um%@Z$N<)qPMlEzo)A zon{q%OgX8#R|`eq6tV04G-mT%oxl~zI#1wxyzH=w zk@3<-6Msz317}_mlfNrGIVVrtM@_5$bx)M<+aGv%X=`=99*xu{^jn4A{s0%q@uB66 zXuzJ(QXwX`t-#}_C~kAN3zU8GJRf%hk9RGtG|M9O!9XGf2!HkmbroE zyOe&$=vbSOVHF7~KV^TxEt?nL!Ai72Z~oBcqij8rCZlv_d{@BlEHWZV1-=!Qr}#Xhf(SRX$5RSBlmK|6Kc8Q2$$*(^|4QCk)lQXU z(_!142058o)<|NvJ^nol-UFGtT!r`tMjid~*()uzk4+u+Z^$3{`!SVh zY-+VNWA?AMNzG5Xyoy?6VnkB$>-f4XnO0y!BND4N>OTFknHV^cz(#ctqq1{V<_t%R z6hxjx)p++C*VgY{q>N6U?G%xvDZ#h@D%_iXsVDm2fPbv<)8OX$N?gs>9@EN!7Rq$4 zE@M;F$mWyNJ$w=-691^H?Ul=u;i;C!nG;+MM`DDBS-|KxB;(SePA)QuS=6U?Ar3^# z2ka?I6`oWww;#_=z%xF{4VI>7 zFKQs7{DFFrTuaGSbxkxZQnra}qO3{I8T2_e60es75I$VT2k#6Q-Pa_gjV}LmPeG_< zxO2wlPN-q#N3?9$N8~h$0kb(AYlnyTOft;dD{AB1Y)?Z5Uf$5Tno(T|T+jR)GPVE9 z*A7+D6nQra%cqw@fa5>(x98TX3H);KS%*@SOgBt>myarYZ8eoE$h4wo_OLQAP&JJ* z*Lh`-%dZu4bjh>$lOVmS`SHP;EQtorsq+^!x0LP@|Ch%a}XHFmTc6 z&tdeyph(z$9HrQf$n9d2=DIps$wSo;-7sSM&ZZXd00A{UrZ3@;9r%wBR*@|lr+R`0 z&t7Fo&-iHqYGHf>pV7$?v++c9dPcVW?hxaah#jf8jk*5hL5THUJ<)bc*i+HuF=BbC zYngz^lHmvnlf;zzNvW@ae-ntG8Dn0kwfO!URDs*SAPJ?+Dl=HVPuL@VyZAgie+_a{ zou-XbcJ*154~{vNB$-Z2-Q0r9)h<-z3Q{FaoWo+tya)?4326u*VBcl)1}q8@UXwV3 zl4X1*n`_x~?yewK%_svWolRdDn`Q;e;huVod@z6%45T@_z3!Lwb64RY zK@2Cb&Ttwr|3iZ7wu{B0$hJf@!7!7xdv|E%Ny_DNW>!`}ZW;{Rzc0Y|hU0dIDHDT= zz{-<&#`agl)a2Stmv42jP+wabjWrJkk$!vEdH?NI*cYe%0WC?8!ulAU~#p&~k zf9FfwSb!`Wz>-BM|3n|2+z(^C_W_5ZwKQq+wUeufq@ZBrCty|rmVV9Q-MJG^O^m}KP`=Db!mTkbk~O2JW%x-28iMht z$*K&ib$w-N84EC3GSf`Jlw8Jkelh*T!s>k#CBCU$LSKJ$1>Q6@Fo^ zYC#-(BkDac6rPJ&IB@?u5i>TrfL{e~2W}}Uv4 ze2Umw*LO(5rw=@mI(u`@1X(QwbI5{@IxRBa~VSVWe3V(xyobXfz{==w=$B6so^ z!1jxkVfTUVCNwsSS88Md5Anra7$Tlmi?Mj2&@~1-LrgZ%Vxk-O_E&8QV#R#E4EB&& zhN$(qC0Jy`dSk0Vu_NEM#OUnNqKBu@!MXN;Hrp^x{m34zYEd-hvJ^1`5}MkaJ;^!G z3>wpQ^*HX$A0o&0&6^BY)7?G33c4jBrB4LPkM`%sE#i~FZELGL<#$){Ww~*cbj)j# z45;h8D}mJ_l5zN%j^ReMzu4F|-{$+VRcolag4YsOe5MquAJIOX7uQDDzIx!_=OrwZ zqs}iqld{9|-uxh{rD&(u_W-x*hCA}*S*%O zIY%E^1&=GBD&dRuCllKsbn1XY)2X!Bj za&Wy@b)N~B$iLhdk8b8=VNkRa-R3UzA|T>S7HnI|lzn3nC&WJ}K*2kIR+=Zy#N)AY%hZK5_HJJ*+M3S#JtKXk(#r_&w&w8B|fNN*CEFT+>uJ)yD2pV=L?0L)5z?Z>+ zG(+cJ!iUOfja%H&MkE2Hmq53kBm*F3tz+Vs)r*Ms)#!GtAP1~oy}2)-L#n{B9r4US zO#z*9_O%&2+WWm)d@B@-3DPaImcJ@_R(fumM$-NId?4GDP&rMaJv#fn2+FM|fyrB} zF)eiL1%cS+BN;)+K+`DuCaX9tqb#uL0RCFQiskn9M~c^TzjIm%wc5k!N$BHMH`G2| zc3(;zu4|Ml2qzqw&LjL=#eg?;3+?G0ltK!60#S_BDnM_uoK2L$&kq{o&ks%d%EpO6 zDjvfq7-SfnKird(w0(8L(SO?G$jl-0DG})2g2M2-{YBg72j9`#_CQ+Pj)&y2q*?c% zpvkT-_KXB1lgaT%uXZQ;zi{^d^uxe}{hrcp%(g|4*>AfX$z}lV$$s`|4sn^AMRI&*WU zO=@ayJHy)RDaPD}h}jk*8Ao*58lL!%H^=x29CY0sEK}s7!U}lqTM9Pl#xdZPlx3re zHWR^czXVyHS$P z8oJ>H5~f^Rm&ZK?_g3J81;?i%xRdcr&(?s~V&jvOT#Y}G<329>t*1k4@~h_sthmX| zs?$zuE)#VlZlm+l5+0reUTs}}Z7I`ulOSiTyso=3+$_$>WKY4>+050bGfrupL}u?H z-(|z+25E@aoMqfsk@ZfPm&aK4&_NyBieyn*v?J`0t8)f#j-9YhJJ?q@N{Y^-%Gipy z_rvEbZOxbfv7U{m6(?9%L3j6}13jB{KXX@6^@EFvX$p7r zC~|E10qvK507p{<+sJ}tkuOC_BN35#gxRz8v?D6OFbnR?>y?v&;{KYmTT3Ukx$sRM z-~A>V-gmRW*5B%8QVHC|+iV|_Q|j~BTa@&h1hLefR}%6`>!+JPB_YrqI!vxR9d;{h zOqo+TW&6vCwSv>zL->%qkMAkRUGi6}9^t)8P}j7fVMFQpKuc3~r+blRDFQy5n3J=h~XVO^5CtRGn4OAX;e(b6djc=GaWB zOEG1hS#L#5#{^$?3YwvI@{xfc0Wjcr3Rnbpy71YdX$sC$H6|~$%eEp6`$`EFa~r&Tu-~a_Hd|H` zQoxZcELA_Hh_+kYhMc3UdLMYW2IxtRlsIooE#iLbF*U38_ra&6@~&?QVJm&l5u0g` zRiMgcw(AWA8K>ffWflq6{Zc=Y&Q5+ms|`q4R?u{}T99IrY={%n*+t=Kv zuqgjEA?)F!Q~oXAiL2SPzk(*kJFX-=%1F@Tp!j@!3ReDDIn8LGahZ?fb)ZPYfjKTx zdFMpag}#=Ceqj9DtD4t4YQvjKiDe}&yVy-#TTQK{?&_BiN!<#kW$gB0A)s`L4-f=SEK}Uc}CYfbys7T8aQQ2 z+8j$$U7@;f0$DiXxq*LW! zzgljK)-SKOd3YN+i&EZ$EmrInl_|RhdQEwjx>SM-GhHqfbib2VcWR}uUN3mN=?XhP z)T^CySVP8dF1&ZehLA1Dv${8hI97~0wu~;8$N7;V%@uFF$z3?)LRG2l8;^!BKpj_7 zH{>Z43>9W)O~g_e*d&j(hbnYmaUCu@dq*Mq4=0V1fc*o}@^}JeR^-oAK+{?MsqV3L za?D}+(Z2JKK*?|Mt?tD`U$Wa;6vrR^Qb03k63p$LrM6^Om7HB>_eQckrWxYe_S$hx z6*k#{36_%EZ!~a6x|<4u-c%Zrg>PZM$D)QX_tqbFr~Adso6Y3!39iNjNt~8|K?lz) zt_d)671r_wAQ53!V*!Os8hUvR@WAf@$AkTXHw-N7pB$E7wTzat!L&f>x>S6VNOoUjOuzq92P9 zD{CufNe%Be?|m?E;Ju?J4IO2;Z@!@Aqw#K+yQi;&Jqzg)z!e1A?GrMoFw46M-mQig z?VS`xJMOu;DzUl-$@D_3k^`aGl9kU(Llh;jpQT)x!nKXBxs7V86D&#k^|gf$%V%D( zv`kpC;x(*;z8Vf;lol)m52|5|5sPw?Ly?@vKB^Pc>NOF0pyVrIj3ZT#$y<`{KKU67 zt&rV*?P8(NQ(#NrOZ!epg$|kiFduuAc$B-d;VI_6@L6qxx5a)Zu_R;TA(?yHECByH z_%<;>L1@kQ7XA_{eDrF^uh)@Lh{<1Jo>RB?n$Fsbvs(4$U=Tw2>52-xIL~Qz19|#p z@212&f|otYo^U?!Q~vhg1g@~n-xGHDS?yy3)&iw*@_;K5?YUsk8>6ADHau-AAgnli zdUq4EJ5#px*=jgi#j~evFNS0T{;|px|*T` zE5DQ9p(mO6kNgmcbajK55C^#;oW##s(}|zmhMY57;nToT9TD!A(!2o>Gtez44W2wz z(poh~m#ql59?V@Ue8E2&KB3*>;=mIm`8`q6dM{x;(+l?K-D4aWN zi!VqW;1EdnFh>31UWDxb3vs*u-~*N%Up7UdIx za2CIll)V>e@Zk%<%AY7Igz+Vi1_cH-j4^(AjU^f%K|Oih4F4vs+7hjV!$g6tEiv&aL8NIa^Bl^!%eNF(}ye+><$naPUx$ z@cf|+&|dWUa8S;EIz7i3k~uq1e8xP82qZJE6)W^WnNXc%@ktnY6JSN@Fv1Jmxuz0f zTQf`fA`q<;BK7Y@_AZ@7%~z^Jpdsr1f221h2v}j@G#Lxu44^c|uhjuh?f?p)A1FY2 z3%vff0m6TB#(y)u_#dj*MLpDOsIM;=92|V8_a_i9|LfN;4F!dV97a1eKrC`)C09sD z=&ga^g;961Ov4ye{}zV-y8+JsV8Fi|AH8Q1jP&yIvYRY5A$$KExFUIOzDm{M%mFTT zb%dR#XC`>J4gc=~%bIm_0Z7JSR+c6dBC;?I6(XOr# ztpc5_2-#v#vQA~^p9+J$*>z~4u=%ao#%foL)LV{`cTK_)6b~du7Q3VA^=K`wj5H;4 z=%K0HN$pJ8I&*4%S=Is1`AOn8Dlh9Gd*452ymTM!gthD4S&|CwV(Dc;eM_ zW2P@>z_|c75EuUu;BfNCC`=11lILnLr@%t5Nckl@$<1@x$hyl_2%jir?({o=5w*?~ zH%CgJ4nAEAGa#x$$p1a8lDxDZ{m5$<3iHra&HfjHqK`#q|{ruQdy=JTE6cpS003p;)| zOw3|~*3(@gQI?QvAF@_s7$dvmWs@$y9wDl!~ zJ%9Q539*%zlbbebQEQXOh{=;o8t=?>EFgUTz($Mq+wA6UvN-!C# zi-QCrg0?8s-+J#W$)gCVkZUr}nPCcKv29=4Vm>g%d}*7qHxbj}lnQMp3vJ$m^d1LC z?W^{a7WRPFdQe`Nn77$airqvq6P2}yCfwHA?P78#`hLmv z#(ZN(Pq4?4e^TvR#NE>>b(FJqI+-WllyG?Ed9)T;8@joug z^f^yp7`L;MXdD_JzMnMy3}naun(uk|0C#*dq_TJvM%2$}3hHsAbon$!mM6ThP&s|N|FVXyr zWDHoc8ys`}^*ahfo2$URe<&w5u3Y&9>3zb$ygRAd1bX<|h;3;hc-&uUekVKDgCrvi zDlhIMrG1#ByHuCyqOH*%-dB@#vW~bv+#D7$V9&UR8>+EZ$ZF8CX!-qJ=WtKB#u-xN zBEA09l{W%!3ZCpyCmHQ^*$tshv-3%+SdQT~a=ZurX?~Ye!QulO)#O+82$Iai(Km3- z+&4b(^Nz?oe-*vSf`X?QY8GG_|5V{h@Hffd*kdfx&ueq>9Bq4MS1h|0JH7u@3YlIS zLUT85)D7jUN!75*pkQ8SToihGqIRGZ73V+f`%A~b3sGZ=afD96Aw6Uy z8Rdc!w0aU_OAnKD0)Bp2Ame%}R5+h6ekS}Lcm4wWD z#T)Zkr|gE+R~kR8B!qgNWa}PR-IQ71?fT#)oo;tnjmVipi86Qu;WliFe;VcI+~NiK zEg$i#b#?E_KO3TIE$E;T$Jf(yH6WK|neT;nN>b#{ z;Jc1nQWaj0rfIWY(4U9vYcCI)c&IL$s)^O#R3*btXOPN_j}ps#Z1gpG&(~JrnV$DW zDgq8`L$oR_Tmk7@Xvq;rB+%$PK@Dp12J5ZOP;X&!QE?+n8Y(*~AqJmb; z%c~^T`JBlEGy3!FzM9pRBkasOjIiQ?didH>Gl%Ju^;AQ}7vxOwD$A2ub-qTghJF=} zD|#Zi;1z@eU!jTnfzc?8>6nE zUytcrXh{}HTdhWU_76~!*E;y5}v ztaghx_z=tRfi<(jtbIr75#akqn{W57M|?&gQxFuhdFj9j6aI{yBshUsx0&V9a!2S7 zj$~fZz!$JF1pZ++q&tm4T7vhZGjsA`X{(dCAyObYw#`@1>zSt`te&EhZSfxXL!upR z0(!86W=e1_cuGG(FyZd*OQQqUYmQEZA#5d}OXhNPRmC&i*>SyoxW9&pT;lwQs)TLD za*a*>jF(-xM|*WvjCNO_*=6^>A%ZJswOO_!T0UssYm5nFO?u|g#IM|pvilaGWs&7bP3fGbR5$K;`furb|=idwlG-8UC&qk*fafbC-Xg;>jHWeh%UDX8<-pSbd@@<9Ce6r z8vw4)lfi zpdloTM#jbMx~a*R#o~m`xr^_&3ymVBEiU0qyqknNA{}i8i*AdJdto{g8K;)JC1R|L)7V6eQh(9BQ$f><3eg5k0 F{{R^Y5Ul_J literal 0 HcmV?d00001 diff --git a/tutorials/source_en/advanced/pynative_graph/images/ms_function.png b/tutorials/source_en/advanced/pynative_graph/images/ms_function.png new file mode 100644 index 0000000000000000000000000000000000000000..27dfd8b03383267420082ba3a5f8fd07f4aee6ad GIT binary patch literal 14519 zcmeIZWmFtdo3`6HA!u-iCLuUMOGz9Ro z#>EEyaqy4%0Br6#KG${uf$l%J`=Jfb`d@)SOrRHXPt`nAcV?AfM_Ycp`+G0ineCnl z>Bp+n>%*)NvuZ=;R=*fKj2451u|nT9m3)9`)lHu@As(hc51u6=y?4Qun-uziios+v z!<{?LmgFhkY4S>t>-HV+8!I0Ql>YPDRmA5j3)|?O)5}2Nz2Bc1B;O5e&u_$B`AR1Z zuFs?G(-V`BkbtiK6v6P8E%s(ez~C(SLEPysh=c^0W9;-04E|g$@_>Zog%({H4EB&K zN={Dh0fizM488|vAt8zR*ck(ZJ^#7%Tu$!c|7G*;Ft{n`RRwXX`(qG@1QS6nAJyNVsQz#hHQ{HDpWtn?5k7 z@^r;vyP_-!ekyP6D=XCNDNsW#7=Ig%$+bR_nKkx$2o?)2W2g}%F>EgIR!|`!4upa- z3dPja=iUaFvI_Ni8&Y^W7THxJFz|!#@lSrlWqM-${G<}(o(CPbbxDWYsp*F{3HWG( zihe&`^a@F1XwBIks;X5=OTphV#bTSn?VGFoYB_zi1A|#MX6uV$$D?KM8i%vz)&}>( zY0warkW}P5_6?2-9-rUh4hwO&>RP8ozj36&OJxD&0a1Jtpk zb+R|hME5OO87SVQtsMV%JA1AHW#?+S`CO%(j}9$CJ4;p{cI`FuXjXBPAvU2NWHG(@H!gDEZ=2nC%4 zALVZDIqdqt(55!Fp@kY5ZW5T1^bNp~-yu53FTR>AynS%puSKz^>tkvAr*L?GxoFtu zb?okxy7k->kj$e0@=F)?a%Ec0_-iC2&;*CiN1Ik|{ip?mpNl@EF0*GW)T^5ggSorK zaw%Y>^%Yl6 ztR`+sITxf{f||@K4g`*`7^6G%&pueYjQE^ZOY4~W#gyVboXdqFb1*))WFzOPEo zIcYny=mQc|f;Xf0BY48$F5lRcx7^i;)k&{~hV;{4%Z9XGF8zzrtp^S>T~5P8XdYS^e9!vasDpHKai zpQZ=Lz&0Lwt^yOiI!f~{3_STSVZiyk6T%znWZJyp^Fd>+r}4c&%Ok`=oT88Xu#T(+iP{7v?=$mViK7}%^9x8O~ zPHN^l)#ATQ-9VOV$B2uXO0E6ydkCJIf7AczOLAr%3NI%-ejYw9$ar8h3L#^C9h1AUT81p`0lp-f&6Lc$!lyxRH# zPQ@|XOtI3;e5&%)g*G2wW>%A!MiCi5h^VIrKEX&`f5*za=-~ zZZ=T)t=hDI@5tR@FkC`kY7PB}D{O<#eyax}D0^@~(BI7LL4vzbPow;`?P!U!^+Sjw zUk7EJ@UfNEk=CWdpCiHd@&>YuVH+2-qjAgcp2(^+D>7F+$7Gl(-gywCe7VAtYc7h- zFk^k+zbtg_Q0e}dBQ!lg@^RIZ$Ab@ui345bQ$@ygd%|B2B4RI}%em4BtJ96}lexu` zo_{(yMjNXs(Ho_hCc|NqK%6q3c1Oil(rFSc{nB(4qp{043h0?kiu3Y{CoZ=!5crtv zUQC%z?&T?EZVimd+Z7h2Ra_%>hMp>eo-etVqOZoY9F0Z;ZsIT=uqR#1`E5HT1ZO)VV)4`?Ld!m8PDCe@)* zxb0$GeuLvi^_AH8s`5hO)p?HR;K^#qcdk|nN}oX$m}i8f8O*oB_8i{tOlBYhONF4S zgfbt}8g!@Wos){m}+rYIf1 zVh}y;kBcoiR}NzwdHKLBHf4MqbTRKLYV#JN`Sitlyf#!;z?Z<_@dhea^xv)ALLfN4O!ZLNWeKNCY=!sbd!V9=E zn6j=28IgUX>Go0D@j$b-e4j*<@B+IZB-?vlZdhH1rB0w%5F;(@KZ|6lW$>&;xgBhc zCIVaRVYWuFel;)cHOPxrQS7lqGU1VevTB*Ov#;BU6~fKxkVX zBX)TiP&uIbZda7_WXm%cqrF_ScT7=DfO4*InXQzbl7v2Pyz9vFBhvDt%KS&Xz_tn_ zDs23AnF%Z?Bp01{1Og(6LmbD@?>wDdjQ_Q`)lIu2?iQ1{! zt7N^oxMiU+imO-z$GrdaD>)*KxbG!-|0&6Im^-JnJ;@__pPDC$E!|0sVL8;rfWBd) zga-#Q%BJRZx$2-H`Mg@TCp1&icA14^3*+IAH5`Rl1id~=-bC3#%6p^A5yAyh(I?)X zN`Ana@XP+TUQD6CBy+<$U!!Pf!6cRHMUTG!o3*3!(DO%LRs(yx$wvtt+rvw!?UAkZ z$o}26oVuw39Z9gq(i?XAMgoL`;j+O*O;-86GU)1>a6-scqiIhY?kBCyZxZ$c_ zHY^mgx#I~?ToJgEn|W1v=20`MkzSur)vUTx&=r5rVbfpm>h)kp!$m+J?CEZz?}QM_ zbedvqkY(2nH-5`b72)x@fUdgeSzVJ-+ijTv$#V!ePdaJ+MQ6}jmiaeH85W}n30zR1 zdPl}^^sd_$K{cXg%!@)6|7VAzlTo%-HGlfaBXEO4df)mH5lG5sFKTG_qiIIejZ>>l z-7@8sQl$eptM~96NZp1tknK-U5uBs#utm(c@%SW%k3kn1AEfTHkj#o0++{Q+f76?o zh=10sn{uF6Dn=rgzKUW!QEG(uoMKNs3h8?2?`qyzOf-bn>$m<7IY4jV0U|*qm*PJl z=5;O9&M+8{@)KF7xBxa{Omt7|2ujY3xS(&O1SHyR2!4xN2rK5@S-X`krnjD=wo!hMf znwM;GFMB7;YEgF0e%&TtJ74SY_SapnHEF**7Bou{-yC#SukT;kZZZ=^c%oB^$PVbP ztPnG_yaDxo=2;|~BqEt*Wp8&p(Zl*lxYd36KA1uQ2({Tl0uP-e>x}yNRSSZ2X#M7+S+{;?I&AN(>ngmfq1iaE8ucnoC>%sW z*3A`nAET(0R=UDXy$LgM>9pc&4iF@h)5X+#wTxh~R)ez8goFf->05=6OaldSnY!>Z z?NzqI=zL31*NFNDLeO`LU}{QnX6Sn{;*1*FZQ0M%+HjmxyOL{Ppm3vk=4vg^77`~; z(#;|Mu2Xg?v8>Z>)}^n9d6W^U%qXm(sSdL466$qAv3J1_>_CNuvi<$y53^jny}i*v zGqbac?V&_xyB@8V1Cl)~eD^9|enbaZ=1ZL-pOdkv%Uh2QHCk{NvR+NH$QflDxFFfl zPpj-)-sRCBtu~f$D!yMOE>zLVKYr{$%bdF}L6jy8`Uzo8!x=o{>zgGa`8xeL-S>mx z&0)e>W9y~()dG%m+xOOMyQ@DI(&E!M37V>KeEGcllrnHTcK9TXOZ(~1im-t>t_QVEiO}p`K%W^FTk=YvNe+7(z;pd z&phJF^3~HdPkZ90-IY1rDL5Cj_j2ZLd;4`hh9fk{gOcGoNUUlhg`1O>9a;SCgCU2R zaTgX;RT-#lXEcM0XwysZag}wJI(_jI#%XdjlAm93Wf&Gz8C8&C$n$o7b4f|ym2p}> z=vvbtcg(W9m-zF;QXFL3Tl>3R3Md{TZ@O~}l|7NoJDv!*o;YilzG^?+#cmE#lD;N0 z^glP-sx-W;F>F6=z4g1=P?Q#PDKq&QsPYH2BhM!Gc&GJ2`uu2|X{@Zshca|!T1yW& zU#iM3>6_{ANaPVp@_VMsCCc#DEw*>{Z>Ozi@Ydg+dz^;ncr7P2Ws5%Zmkc*6N;fO? z-fLh8F?V;EGrefFnnaITe&CtvLKoOAt4KxA_r^WQu5Cz`)n6F$ghyv7CM!QL!= z1IgcAY3b<)MXeV_d$R-$5aClvMes#kIo~k*UC`2pw|&TB6M4vSz@ayEPyzgdj7jBn zI2iRDysK25@5<@$1F?5+7oSi%HHeN(tl0}yG0PJ(g{wy#Qf5zZs7WT%Nk}Tavv#QW zH()UJpEDWvKM`%IO5(rkprg`-Dj_L2XIO`D$;9bNbj3ppmeTI-5WN*X`}eGCwqGju z?2_P}bJRkAO@HsgKF@ zu?-yR1d(s!xaF&eOGPAp50w-8mf{%z2IZ@KR`yYrdptVCAnzi-ZdWzfLbNk|d($d1 zB^9+(8_seBJY4jzyrB>l+6N$XT9ph1GUJ-$`p15>5J|szzT)KodLLm$XBVZSC%{1Q zjKtG*k!|^*Jzs%fr=>L8+Y7x16|JF7_-qFx^8%5wsW0X5?0nn~276#&xYpkpB$Bh% zH2HdF`pe$pqdn1q^{`OAmq+GR>?NE#YX$Wu%7gk>=hg1fE>4Dg{$Fd}==k&}O!P}h zN|vBbQI9I#ELHJ0Iy5rQQ~T9bs1e@P{KynPf7SLvl>rmRTs#?v49U_Tj)$$`4Suxr zlI&I&pRS2Xkf^!}PC)<7621$3z}D30zg3A{>3#-gvI!p?OWowahMdEk7zcFo^eWL4Bk0hLoK|?3u?bVj%Oo#AdJUBxK z6BBpWF8SpbMn#VCBCp1Cqv6dhO?`$B_=%faTKWtP4A#nePYu?Ua)z%qSKZF%wh8$u z%xY8W(UwbKYzQz-;b^Mxt8R(T2>cHD4t1-zuXPMZ)w1sICEn-?H0z#RY@SDmQh&0U zjPkIF(cV`5t=~i>EL2R>_Kr&{KCofk7*gPCM4dma3Q`^2#-;QX$Wy&va)6vKuhms}xpb8Ix1Mfx|v{P}ROh8_Y09yxJ^p&o+5`e{U z#y(2>e8jaKwc!zyU+}NRRZ{BK=^*aX!9f2)+$&e?SH(iqo80#+Qm@f_RHYQXbo+-F z6^k6XrW&pE=REZy?z5KGixK`)o=6992$f8bkOAWq6Ziv+K~Dw6pfff$eSRJRT9S}o zbX`Or;)o6#tO0ubUY!2RVrGiBLpvIDtUybn=PI^n{ajt79N*NG>STDv*GY|$b7+P> z{IhrTv!xtBEPm{++EzfMym`5=E7Nb{^)QPZQkE|QRc#wdYt|s~>`%!1xQ7q3Y-%W?d^gh{Ih8x>7+sk) z)|8uqvuGdFQFRgyF>ser@G5~2hkVqHj_n)Wn07iR?k(cLQZ{~i$iQ%d=N#FPI2=Kp74?mtMIBM}&v7!z#6 z{6U7v)asdzG=^44aB(gX3$X|VSMP^a5%KD;e>9gY7Tfqz>7(E<7@^auW)yn6JwZ+K z*LP5?RPX9PpTOXMBW^p*5wssOM;9|Aoa-Z=fCiY2tEvR7@^E^xG>eVR1KkEfNt?@+`tA%V4QN(m+aK;}AfGS%syYx0p$(@CngXs;zl;dsR$8Kcf=G^#1b4`xA|dwKcQQ*B0d?&K7nJcV&D zO->27I#-6Q20l3~{3wYh?c{{!Q1NkLoH<OvpQl7AJk@LZ*!?$d-n5t8-zv`QRCa zd(S=>4R`%y1%;wKTUw;QCr^i-m*jCt=@4&+9v-WnVGb5$P!#S6cJ~Pgic+AZ8f~O1o8{H3c8PvgDIJI- zyzRMrA2-J*@sgAnY|q>xtyT4nbHVeRXPvj?dLx!6-CAcS8U~wN5g}T+D^MWT;SUJz z)4*|@)g&x=tV6I!h`w+f3}s4&y^M2=SoL{HX^BuOdEcBq9f`mrMvohHbBir99gX$8 z9l=r4)aALP1|iigMcVecYu3ta$>8o+`U>Q^(m(kd*mwqFvSbPnhEG-A98b%5c-`Pr zMSZ$>gi^tW;1;p*5(+-wy}Od7FXz5DXl)jIvoWt&=rG)lk3ya2kvXT)u#Ip(|J5&` z)F|r{73$IN0Xa*OxL&Es=O7T3{+$;92lW)|FNpcF@DC94SGM@CAg24TX8ijR={pg= zrM-JPNt}D1k!SIA2x0paiMQvTiHm?tFZ55nwD4q(p;1oHB-HM7s!M>D~DRJ$HF33o?6_HgQSDjJ<>?+vrp z>+a5$U}|&_Jj-kp#*2gz*sErRRn6dShLu!hgf|RUpIf4H;SI_|YKNh8Lpac76uY?K&lQ zarvzd^I&%SpCnrTX`a*c+5s@S?e>9v+DQAwbGho)X^to>hzfv@xyN5bm4t^-?rf47 ze8+tRaZO<0Cbx6Y#@;e$;uS6B$Q-uD7_N2>T+h-eD4zSZv&?`ZWFC zBisd+m>YZR2hh>P{1Qzz_3A4TiQm2*K`IF4L_T?g>e#z(YYen{))iNOzB^~J6PaT5^`NA$G)h@FGrs@BnElWdAZRJ)iy=5KVCMxdzv z3*uO}{(?9i>tPl zdWcSBKkgnj3ir#(d9qv>?@V?5-526>}FZ&K+BMFqDk4`rq01bsVDOif; z5HFSsO-w|d=8TDZ8}~)GH+;h7T%4%tRg0opP_m_{gwAjN94o1cZ!q$GyN=v?-ReT<;I#Sad~%0=`_y z0c;YWyx-&`B=jrbGP+<@#nVip`Mn>b>iHZX^ zg-I~c&|^H3YHiWL6$ZPkHh#EIPLezw=me>13V<22-6>_4Vt2_5&dRQDT4JLc#UVd_ zLk@+S0{IASt+7l6x{M%_1D}r!MBMfcpDG^BN2Ie_R}NWjM2-FDZW_UsO!NfGjYcJ3 zqZwJYcT@3@{~(?P)^Um7Cwv?-Kc6;R8rQr4z0z)qzgTcniGX_;Q6EUJ=rRuFS98L zRFZ*wg`tOEy5WHjW|#aO^A@9L0npn7fR55cxr(+}bqsK3L$SSm@QXC!+rRxyZ*Ct@ zMR0ChnveE(682+r1VPF-T(VZcccvmf9eWp|SDQbrHhP(VM_%KSKBm5Y1ipaj&%|XX zMyb?kqG9gOaP&B-kdS5^CK_Pvdaw}|PCj$flU0Eji6R(2bJ-p*-bP~M-aqx3eMvR~ zDX)!;au)1(KRMZdCt4in>p8uqY;&QMzK@U7);=fyOmQ&XSjZYJh-Pa*5dS&eYsB6x z4mc}%u#;R7)dH=c1PviA0j~}|RY>Te3tCVET6@0fyG=%^*!-=HnnK*iK4gk0hTx7@ zB%Y4^no(EY73lq_@rCV<|atm@Q;r8 zWH?Z4CONd78>z@mHqFL-c5DFxHZ_iXlh3W)w|qzkS~d1%;l)pw6rh zfw`H(4kfjBnFqyI_=nP5X8gdds(|6L65#f7l>p*{14?i9P#ni9kKpMF#?y&JoH3+A zi8)-q3eN%%{F_v36Q?2wdHt0B8738>ohRu6kjh_1;5ZYKQW|Z}KFevHP%Q*zs}Ij{ z58&zBnDzB{B3dB^6h%wc(%7;P?lp@j^Bb*e-$TN$PKLr!07Tg!23|A3m!}crcL@EU z3tphYi@l$|2*29}G7*u%B}m4uo6{}K*6PpU>bSTXq&<3i-)he697X7(TU}qFQpvb^ zBn}@vFIe57@Qa?G|D6C!N2MFF)ztXRe>-vsug_fv0GKEV$9oi~W$~m9a zx4Jn_*1Hnmot^D}rBt>pohL-j%L}Mxjg9B}S9z1RJOKu^yx(hfYF2Xo0+Imf{sfz@ zX`Te`i-QOBdnVKGnwrSCwRWl|BfbYDw+;Y$T3yLIc6&~P+njpqRy7Xp%Z69;<0l8h z5!03fDLHZhQOoVChN5^lAtrXWPp-B3sph9MsXuLhN!{}s_=vImGxQm309=l|7ig^= z&AKN8`>_wS`}qv(dJ?}%g_Z_HX%-P*Ha{nYeqWGsXo>}pcdFJoh~0%O?sHZ@xQr|d z2M`NozSMunLh{Zd|1SEF|10xmN?VZILm|gOmUhQ}(&5-M6%O5gawDNuKqT8Z=Ra#+ z_Gh^|gfBG82{86oz3Vz4_?0Rl(e+MS%YhjY{bt;XoqZ z{tdw7+1#Ijy1mlTO1@sb4*#%4&<-`G+GqgrbqR2WI83xWI;l9z`KHSG3{uPzaAV-; zL?)UP(ks#4QFpoI5JTCNIKIX_8D_P-YeOA*XrF!Z-TYaP-wQi_=q!6i+!c?x!rRp~ zFCE>6E1%`md+a!*MH2w-`6Pwu-rex9fxf6d4Kd5pFpJj3465@SDM7#ic=M%9P%T}N zDpDh$(!YQL0M1eHXGPKet4woU;4-Z99W1NmZShh52bH?X&<#quXHnF}&&Zul}3sGlZ2L zI<7?B*rD~y`iRt?>Z90LpMj6fVg^7_{b$S>C>yyCn2EL67|x|%X1!uf<8|Wwnv((omgr` z0|1xrQNr5Ao2dlzCDy%o+PUtX;i=@-zLegSt?u**!&a%dtd#w2@kHskZ?uO81e>dp zODj%X35c?hi-alj>f05k+!FxuKROW|l(sD>(I{U@%{!_rk5XeN)CQ^qj3bG(i=*}eD8P*1-!mQCcDFJ-0=Hzi$pGPm5kd6O)%f@RrL#m$xVQPZ&s}IYdlh zzu#^;b=W-MLZ-`W%5+~99|Fa;`@;-viN^;61vqLV^F1;my7Wn3j|Xa z7&V_;jIztMJze|L;pf8a@`U+Rlm}Q&96vc53L?|n2fjeHq&y!kjN;@U`<-!T)SgI2 zZzRBeHaAQ~DXkFCLVeT<)zxD#3~5DOPsB-rE`IV_=Y!3D2TpCipzunqbGvA&}KqFob6+%n!hz#A%l^{(77 z`>*ZD1xM}eU*P^zEz0KGUx#S6|E4HMA82R?zqITu_w$+>`dz94aw5>RXUo6>Fch@T8>r?75f_H z0b37AgDt5dB5``nrl`@b%E5!H5=v8l<1Zj$lG)rpRC&k0^HLM@zr`lmX{e|kT~T|u zr$t&w8d&cwe_pOz>g-M|{5-RF>V_4gp74I>JCNG?y+?>}7k5hS5R=yAUeL+EG3z_> zR*oK7${`djea0ES{PN^Yc)?-166ERDqDXNA3p1t~{vrWmrZgTds?O<&oNHq1?PV*% z;dA`xy&HCPK zoL(qUMKF-Y{r*BG=%!;4mgkKltU))D9WS@gR#_Y=9rbHYA$RK(#Nk;As#7|*vqiYW zHtKFR({JwGo<-lnq_0P$JHNM}xUVMDuK}MhJ0hMc57TWgG2io05uvN_=8<$>ZKvK- z1CQ%SvnI8M*=U=m&Agzn#KJb0@AfVlN3=dnk0q@)-EJ-#&L0P)M8>a|m36mz%nFly z9p1Yd*ejNV9q#Y*-3CdY1)Ur5dG3{#^unE9!u^uhJLF2r$RvqbHyr%%#<%Xr<}xN-Htg*pb=BAPdlK(cRBL4~FQ!|sb?+d% z=brtL_!%%D7QW}J3AB3wM=A3i!Yu~`S1XUDf1I|QHhXMP>sJgq>;GyjqXAn3P2*0M5 z0mDjGjYnXy+&bJ+=O9g&cpiVAxCcjH*NR&;l)u!HAy)N@x<%DbC!?!($;rl6{1KOtW7?UDKY0bH?=NQ#_EO9M%<3BfOCt1=Xp$Kjune>W6 zD^9)-m>jKbpt?ciD}V3c8J@*_GAi+x%Z!CIWdgD;`aE*h<$W4pB?_;WK&1b*=)m<5 z@;4^OV!XF12IIalj14iB43yJ3E23*V>mT@OPJc)s+P{mMi39HcD0@eH^}mFbcJ_DN z9ZZI8^h>oC;5cW!?v#2>4xN+n(F6>&aN|IV}2JZblLOONp%-`i}M zGNWsSqqmFl8L5cyk(r?+g0C(@{?h*@KNoWTaZNN!ohyvh`sO{L2Yqo8CRmY9>~!MS zB3!^NDt}KcbDIO;`0BB`+<=7_opuc;BO4O}D73!|q6 z^+VMhc#e>hG~_c%dt49?_AEqsQ&t#_394TFV@$vjmm~yKIOsW`V~CF|af6=VHWYx% z$wmR2D>gTN;0bOE`@89xn`S%mpPX51)~*2=Gkjx+o)d`10EbsFD$+Twb+ zSG~;*LIP3?+@zAJ@r|nr_UkyM!=meLBbZZU&El+m-VC5jWqg}ni&&WjEk69a&r!C~ z+o(q9r6ww?D2W_My{XkfrId0bRv|zKkvRt;@N8TS(-RqiQU*+T1?o4I3gPn7k0Sk> zpdM>s7Yt_ZFNUNfkMaR85N`H5M>XK{vt-_$e788NvR%hkeX((RX<1Vvymw<~x5cJ7 zMl#?CC05t7vTLeYqWTY<9Medxg}rq`)<%Z%cCBHfqYdKgXsa@wCj98AYSiVf08<~8jyNCyfgbyn@s@uixXU7%6IZ}(-C z*qX!xWT`>=%c9hd#^EO(5N=JUn~Nt7@fu0t{pMEwlmQeuWw+JwWr@iLi6d(q(uX_; zqRGC^EpJ`O4@FuVOZ5;$!e=rA7iLX~QGt#T4G^uCJaPu66My%Y3=C`w{s>mJJGoP? zWItjuju1kO^vchTPD(m#nG2HM?smvpFJnkUgU%gNr&XigoP~9%SK-XZjt3^svZ^KC z;CJtKmnBpS(ZC1An?sXk67G8hbwcE8umYALEBprY{Od6=dxL+2nX(xdsRyxt;_51V zqEdhl2m~g2n=+eq4*sohj~>QA%afAH*=u(|6WngIN&!Tx$tCaXp*s(*O>|ZBHQ?Bn zYqiTPW!tbmc(eABv84b|dCA~-*vTO;=N0RxV9&e$KeVC$J3Bq+fC6d&;C_ek{2IKGwp1l_ Wqq9m_0KP;7y^wz^SNhE8!~X(!*S + +Currently, there are two execution modes of a mainstream deep learning framework: a static graph mode (Graph) and a dynamic graph mode (PyNative). + +- In static graph mode, when the program is built and executed, the graph structure of the neural network is generated first, and then the computation operations involved in the graph are performed. Therefore, in static graph mode, the compiler can achieve better execution performance by using technologies such as graph optimization, which facilitates large-scale deployment and cross-platform running. + +- In dynamic graph mode, the program is executed line by line according to the code writing sequence. In the forward execution process, the backward execution graph is dynamically generated according to the backward propagation principle. In this mode, the compiler delivers the operators in the neural network to the device one by one for computing, facilitating users to build and debug the neural network model. + +## Introduction to Dynamic and Static Graphs + +MindSpore provides a unified encoding mode for static and dynamic graphs, significantly enhancing compatibility between both types of graphs. This enables you to switch between the static and dynamic graph modes by changing only one line of code, eliminating the need to develop multiple sets of code. By default, MindSpore uses the static graph mode, and the dynamic graph mode is used for debugging. + +> When switching the running mode from dynamic graph to static graph, pay attention to the [static graph syntax support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). + +### Mode Selection + +You can configure the `context` parameter to control the program running mode. The differences between the dynamic graph mode and the static graph mode are as follows: + +- **Application scenario:** The network structure of a static graph needs to be built at the beginning, and then the framework optimizes and executes the entire graph. This mode is applicable to scenarios where the network is fixed and high performance is required. Operators are executed line by line on a dynamic graph. Single operator, common functions, and networks can be executed, and gradients can be computed separately. + +- **Network execution:** When the same network and operator are executed in static graph mode and dynamic graph mode, the accuracy effect is the same. The static graph mode uses technologies such as graph optimization and entire computational graph offloading. The static graph mode has higher network performance and efficiency, while the dynamic graph mode facilitates debugging and optimization. + +- **Code debugging:** The dynamic graph mode is recommended for script development and network process debugging. In dynamic graph mode, you can easily set breakpoints and obtain intermediate results of network execution. You can also debug the network in pdb mode. In static graph mode, breakpoints cannot be set. You can only specify an operator for printing and view the output result after the network execution is complete. + +### Mode Switching + +During mode switching, you need to set the running mode in the context. Define the network model `MyNet` and the data used in subsequent code snippets for subsequent switching and display of the dynamic and static graph modes. + +```python +import numpy as np +import mindspore.nn as nn +import mindspore.ops as ops +import mindspore as ms + +class MyNet(nn.Cell): + """Customize the network to implement the addition of two tensors.""" + def __init__(self): + super(MyNet, self).__init__() + self.add = ops.Add() + + def construct(self, x, y): + return self.add(x, y) + +x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) +y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) +``` + +Set the running mode to static graph mode. + +```python +import mindspore as ms + +ms.set_context(mode=ms.GRAPH_MODE) + +net = MyNet() +print(net(x, y)) +``` + +```text + [5. 7. 9.] +``` + +When MindSpore is in static graph mode, you can switch to the dynamic graph mode by setting `mode=ms.PYNATIVE_MODE`. Similarly, when MindSpore is in dynamic graph mode, you can switch to the static graph mode by setting`mode=ms.GRAPH_MODE`. Pay attention to [static graph syntax support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). + +```python +ms.set_context(mode=ms.PYNATIVE_MODE) + +net = MyNet() +print(net(x, y)) +``` + +```text + [5. 7. 9.] +``` + +## Static Graph + +In MindSpore, the static graph mode is also called the Graph mode, which is applicable to scenarios where the network is fixed and high performance is required. You can set the input parameter `mode` to `GRAPH_MODE` in the `set_context` API to set the static graph mode. + +In static graph mode, the compiler can perform global optimization on graphs based on technologies such as graph optimization and entire computational graph offloading. Therefore, good performance can be obtained when the compiler is executed in static graph mode. However, the execution graph is converted from the source code. Therefore, not all Python syntax is supported in static graph mode. There are some special constraints. For details about the support, see [Static Graph Syntax Support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). + +### Execution Principle of the Static Graph Mode + +In static graph mode, MindSpore converts the Python source code into an intermediate representation (IR), optimizes the IR graph based on the IR, and executes the optimized graph on the hardware device. + +MindSpore uses a graph-based functional IR called MindIR. The static graph mode is built and optimized based on MindIR. When using the static graph mode, you need to use the [nn.Cell](https://mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell) class and writ execution code in the `construct` function. + +### Code Example in Static Graph Mode + +The code of the static graph mode is as follows. The neural network model implements the computation operation of $f(x, y)=x*y$. + +```python +# Set the running mode to static graph mode. +ms.set_context(mode=ms.GRAPH_MODE) + +class Net(nn.Cell): + """Customize the network to implement the multiplication of two tensors.""" + def __init__(self): + super(Net, self).__init__() + self.mul = ops.Mul() + + def construct(self, x, y): + """Define execution code."" + return self.mul(x, y) + +x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) +y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) + +net = Net() + +print(net(x, y)) +``` + +```text + [ 4. 10. 18.] +``` + +### Control Flow in Static Graph Mode + +For details about control flows in static graph mode, see [Process Control Statements](https://www.mindspore.cn/tutorials/en/master/advanced/network/control_flow.html). + +## Dynamic Graph + +In MindSpore, the dynamic graph mode is also called the PyNative mode. You can set the input parameter `mode` to `PYNATIVE_MODE` in the `set_context` API to set the dynamic graph mode. + +During script development and network process debugging, you are advised to use the dynamic graph mode for debugging. The dynamic graph mode supports single-operator execution, common function execution, network execution, and independent gradient computation. + +### Execution Principle of the Dynamic Graph Mode + +In dynamic graph mode, you can use complete Python APIs. In addition, when APIs provided by MindSpore are used, the framework executes operator API operations on the corresponding hardware platform based on the selected hardware platform (Ascend/GPU/CPU) or environment information, and returns the corresponding result. + +The overall execution process of the framework is as follows: + +![process](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_en/advanced/pynative_graph/images/framework2.png) + +The front-end Python API is called to the framework layer and finally computed on the corresponding hardware device. + +The following uses the `ops.mul` operator to replace the network model that needs to be defined in static graph mode to implement the computation of $f(x, y)=x*y$. + +```python +# Set the running mode to dynamic graph mode. +ms.set_context(mode=ms.PYNATIVE_MODE) + +x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) +y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) + +output = ops.mul(x, y) + +print(output.asnumpy()) +``` + +```text + [ 4. 10. 18.] +``` + +In the preceding sample code, when the `ops.mul(x, y)` API is called, the Python API at the MindSpore expression layer is called to the C++ layer of the MindSpore framework using [Pybind11](https://pybind11.readthedocs.io/en/stable/basics.html) and converted into the C++ API. Then, the framework automatically selects the corresponding hardware device based on the MindSpore installation environment information and performs the add operation on the hardware device. + +According to the preceding principles, in PyNative mode, Python script code is executed based on the Python syntax. During the execution, Python APIs at the MindSpore expression layer are executed on different hardware based on user settings to accelerate performance. + +Therefore, in dynamic graph mode, you can use Python syntax and debugging methods as required. + +### Principle of Automatic Differentiation in Dynamic Graph Mode + +In a dynamic graph, the forward propagation process is executed based on the Python syntax, and the backward propagation process is implemented based on tensors. + +Therefore, during the forward propagation process, all operations applied to tensors are recorded and computed backward, and all backward propagation processes are connected to form an overall backward propagation graph. Finally, the backward graph is executed on the device and the gradient is computed. + +The following uses a simple sample code to describe the principle of automatic differentiation in dynamic graph mode. Multiply the matrix x by a fixed parameter z, and then perform matrix multiplication with y: + +$$f(x, y)=(x * z) * y \tag{1}$$ + +The sample code is as follows: + +```python +# Set the running mode to dynamic graph mode. +ms.set_context(mode=ms.PYNATIVE_MODE) + +class Net(nn.Cell): + """Customize a network.""" + def __init__(self): + super(Net, self).__init__() + self.matmul = ops.MatMul() + self.z = ms.Parameter(ms.Tensor(np.array([1.0], np.float32)), name='z') + + def construct(self, x, y): + x = x * self.z + x = self.matmul(x, y) + return x + +class GradNetWrtX(nn.Cell): + """Define the derivation of x.""" + def __init__(self, net): + super(GradNetWrtX, self).__init__() + + self.net = net + self.grad_op = ops.GradOperation() + + def construct(self, x, y): + gradient_function = self.grad_op(self.net) + return gradient_function(x, y) + +x = ms.Tensor([[0.8, 0.6, 0.2], [1.8, 1.3, 1.1]], dtype=ms.float32) +y = ms.Tensor([[0.11, 3.3, 1.1], [1.1, 0.2, 1.4], [1.1, 2.2, 0.3]], dtype=ms.float32) + +output = GradNetWrtX(Net())(x, y) +print(output) +``` + +```text + [[4.5099998 2.7 3.6000001] + [4.5099998 2.7 3.6000001]] +``` + +> The accuracy may vary depending on the computing platform. Therefore, the execution results of the preceding code vary slightly on different platforms. For details about the derivation of the formula and the explanation of the preceding printed results, see [Automatic Derivation](https://www.mindspore.cn/tutorials/en/master/advanced/network/derivation.html#). + +![forward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_zh_cn/advanced/pynative_graph/images/forward_backward.png) + +It can be learned from the preceding dynamic graph mode that, in a forward propagation process, MindSpore records a computation process of Mul, and a backward MulGrad operator is obtained according to a definition of a backward bprop corresponding to Mul. + +The bprop definition of the Mul operator is as follows: + +```python +from mindspore.ops._grad.grad_base import bprop_getters + +@bprop_getters.register(ops.Mul) +def get_bprop_mul(self): + """Grad definition for `Mul` operation.""" + mul_func = P.Mul() + + def bprop(x, y, out, dout): + bc_dx = mul_func(y, dout) + bc_dy = mul_func(x, dout) + return binop_grad_common(x, y, bc_dx, bc_dy) + + return bprop +``` + +You can see that two backward propagation gradient values of the input and output are required to compute the backward input of Mul. In this case, you can connect z to MulGrad based on the actual input value. The rest can be deduced by analogy. For the next operator Matmul, the MatmulGrad information is obtained accordingly, and then the context gradient propagation is connected based on the input and output of bprop. + +Similarly, for derivation of input y, the same process may be used for derivation. + +### Control Flow in Dynamic Graph Mode + +In MindSpore, the control flow syntax is not specially processed. Instead, the control flow syntax is directly executed based on the Python syntax, and automatic differentiation operations are performed on the expanded execution operators. + +For example, in a for loop, the Python source code is executed first in the dynamic graph, and then the statements in the for loop are continuously executed based on the number of loops, and automatic differentiation operations are performed on the operators. + +```python +# Set the running mode to dynamic graph mode. +ms.set_context(mode=ms.PYNATIVE_MODE) + +class Net(nn.Cell): + """Customize a network.""" + def __init__(self): + super(Net, self).__init__() + self.matmul = ops.MatMul() + self.z = ms.Parameter(ms.Tensor(np.array([1.0], np.float32)), name='z') + + def construct(self, x): + for _ in range(3): + x = x + self.z + return x + +x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) +net = Net() +output = net(x) + +print(output) +``` + +```text + [4. 5. 6.] +``` diff --git a/tutorials/source_en/advanced/pynative_graph/pynative.md b/tutorials/source_en/advanced/pynative_graph/pynative.md new file mode 100644 index 0000000000..461171b729 --- /dev/null +++ b/tutorials/source_en/advanced/pynative_graph/pynative.md @@ -0,0 +1,139 @@ +# Dynamic Graph Mode Application + + + +In dynamic graph mode, MindSpore supports single-operator execution, common function execution, network execution, and independent gradient computation. The following uses sample code to describe how to use these operations and precautions. + +## Operations + +First, import related dependencies and set the running mode to dynamic graph mode. + +```python +import numpy as np +import mindspore.ops as ops +import mindspore.nn as nn +import mindspore as ms + +ms.set_context(mode=ms.PYNATIVE_MODE) +``` + +### Executing a Single Operator + +The following is the sample code for executing the addition operator [mindspore.ops.Add](https://mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.Add.html#mindspore.ops.Add): + +```python +add = ops.Add() +x = ms.Tensor(np.array([1, 2]).astype(np.float32)) +y = ms.Tensor(np.array([3, 5]).astype(np.float32)) +z = add(x, y) +print("x:", x.asnumpy(), "\ny:", y.asnumpy(), "\nz:", z.asnumpy()) +``` + +```text + x: [1. 2.] + y: [3. 5.] + z: [4. 7.] +``` + +### Executing a Function + +Execute the customized function `add_func`. The sample code is as follows: + +```python +add = ops.Add() + +def add_func(x, y): + z = add(x, y) + z = add(z, x) + return z + +x = ms.Tensor(np.array([1, 2]).astype(np.float32)) +y = ms.Tensor(np.array([3, 5]).astype(np.float32)) +z = add_func(x, y) +print("x:", x.asnumpy(), "\ny:", y.asnumpy(), "\nz:", z.asnumpy()) +``` + +```text + x: [1. 2.] + y: [3. 5.] + z: [5. 9.] +``` + +### Executing a Network + +Execute the customized network `Net` and define the network structure in construct. The sample code is as follows: + +```python +class Net(nn.Cell): + def __init__(self): + super(Net, self).__init__() + self.mul = ops.Mul() + + def construct(self, x, y): + return self.mul(x, y) + +net = Net() +x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) +y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) +z = net(x, y) + +print("x:", x.asnumpy(), "\ny:", y.asnumpy(), "\nz:", z.asnumpy()) +``` + +```text + x: [1. 2. 3.] + y: [4. 5. 6.] + z: [ 4. 10. 18.] +``` + +## Synchronous Execution + +In dynamic graph mode, operators are executed asynchronously on the device to improve performance. Therefore, when an operator execution error occurs, the error information may be displayed at the end of the program execution. To solve this problem, the pynative_synchronize setting is added to MindSpore to determine whether to use asynchronous execution on the operator device. + +In dynamic graph mode, operators are executed asynchronously by default. You can set context to determine whether to execute operators asynchronously. When an operator fails to be executed, you can easily view the location of the error code through the call stack. The sample code is as follows: + +```python +import mindspore as ms + +# Set pynative_synchronize to synchronize operators. +ms.set_context(mode=ms.PYNATIVE_MODE, pynative_synchronize=True) + +class Net(nn.Cell): + def __init__(self): + super(Net, self).__init__() + self.get_next = ops.GetNext([ms.float32], [(1, 1)], 1, "test") + + def construct(self, x1,): + x = self.get_next() + x = x + x1 + return x + +ms.set_context() +x1 = np.random.randn(1, 1).astype(np.float32) +net = Net() +output = net(ms.Tensor(x1)) +print(output.asnumpy()) +``` + +Output: In this case, the operator is executed synchronously. When an error occurs during operator execution, you can view the complete call stack and find the code line where the error occurs. + +```text +Traceback (most recent call last): + File "test.py", line 24, in + output = net(ms.Tensor(x1)) + File ".../mindspore/nn/cell.py", line 602, in __call__ + raise err + File ".../mindspore/nn/cell.py", line 599, in __call__ + output = self._run_construct(cast_inputs, kwargs) + File ".../mindspore/nn/cell.py", line 429, in _run_construct + output = self.construct(*cast_inputs, **kwargs) + File "test.py", line 17, in construct + x = self.get_next() + File ".../mindspore/ops/primitive.py", line 294, in __call__ + return _run_op(self, self.name, args) + File ".../mindspore/common/api.py", line 90, in wrapper + results = fn(*arg, **kwargs) + File ".../mindspore/ops/primitive.py", line 754, in _run_op + output = real_run_op(obj, op_name, args) +RuntimeError: mindspore/ccsrc/plugin/device/gpu/kernel/data/dataset_iterator_kernel.cc:139 Launch] For 'GetNext', gpu Queue(test) Open Failed: 2 +``` diff --git a/tutorials/source_en/advanced/train.rst b/tutorials/source_en/advanced/train.rst index 194b7f7105..a741cb5bec 100644 --- a/tutorials/source_en/advanced/train.rst +++ b/tutorials/source_en/advanced/train.rst @@ -2,6 +2,10 @@ Training and Evaluation ======================= .. toctree:: - :maxdepth: 1 + :maxdepth: 1 - train/save \ No newline at end of file + train/metric + train/train_eval + train/model + train/callback + train/save \ No newline at end of file diff --git a/tutorials/source_en/advanced/train/callback.md b/tutorials/source_en/advanced/train/callback.md new file mode 100644 index 0000000000..284b72a825 --- /dev/null +++ b/tutorials/source_en/advanced/train/callback.md @@ -0,0 +1,333 @@ +# Callback Mechanism + + + +During deep learning training, MindSpore provides the callback mechanism to promptly learn about the training status of the network model, observe the changes of network model parameters in real time, and implement customized operations during training. + +The callback mechanism is generally used in the network model training process `model.train`. The MindSpore `model` executes callback functions based on the sequence in the callback list. You can set different callback classes to implement functions executed during or after training. + +> For more information about built-in callback classes and how to use them, see [API](https://www.mindspore.cn/docs/en/master/api_python/mindspore/mindspore.Callback.html#mindspore.Callback). + +## Callback Usage + +When talking about callback, most users find it difficult to understand whether stacks or special scheduling modes are required. Actually, the callback can be explained as follows: + +Assume that function A has a parameter which is function B. After function A is executed, function B is executed. This process is called callback. + +The `callback` in MindSpore is actually not a function but a class. You can use the callback mechanism to observe the internal status and related information of the network during training or perform specific actions in a specific period. + +For example, monitor the loss function, save the model parameter `ckpt`, dynamically adjust the parameter `lr`, and terminate the training task in advance. + +The following uses the LeNet-5 model training based on the MNIST dataset as an example to describe several common MindSpore built-in callback classes. + +Download and process MNIST data to build a LeNet-5 model. The sample code is as follows: + +```python +import mindspore.nn as nn +import mindspore as ms +from mindvision.classification.dataset import Mnist +from mindvision.classification.models import lenet + +download_train = Mnist(path="./mnist", split="train", download=True) +dataset_train = download_train.run() + +network = lenet(num_classes=10, pretrained=False) +net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean') +net_opt = nn.Momentum(network.trainable_params(), learning_rate=0.01, momentum=0.9) + +# Define a network model. +model = ms.Model(network, loss_fn=net_loss, optimizer=net_opt, metrics={"Accuracy": nn.Accuracy()}) +``` + +To use the callback mechanism, transfer the `callback` object to the `model.train` method. The `callback` object can be a callback list. The sample code is as follows, where [ModelCheckpoint](https://mindspore.cn/docs/en/master/api_python/mindspore/mindspore.ModelCheckpoint.html#mindspore.ModelCheckpoint) and [LossMonitor](https://mindspore.cn/docs/en/master/api_python/mindspore/mindspore.LossMonitor.html#mindspore.LossMonitor) are callback classes provided by MindSpore: + +```python +import mindspore as ms + +# Define callback classes. +ckpt_cb = ms.ModelCheckpoint() +loss_cb = ms.LossMonitor(1875) + +model.train(5, dataset_train, callbacks=[ckpt_cb, loss_cb]) +``` + +```text + epoch: 1 step: 1875, loss is 0.257398396730423 + epoch: 2 step: 1875, loss is 0.04801357910037041 + epoch: 3 step: 1875, loss is 0.028765171766281128 + epoch: 4 step: 1875, loss is 0.008372672833502293 + epoch: 5 step: 1875, loss is 0.0016194271156564355 +``` + +## Common Built-in Callback Functions + +MindSpore provides the `callback` capability to allow users to insert customized operations in a specific phase of training or inference. + +### ModelCheckpoint + +To save the trained network model and parameters for re-inference or re-training, MindSpore provides the [ModelCheckpoint](https://mindspore.cn/docs/en/master/api_python/mindspore/mindspore.ModelCheckpoint.html#mindspore.ModelCheckpoint) API, which is generally used together with the [CheckpointConfig](https://mindspore.cn/docs/en/master/api_python/mindspore/mindspore.CheckpointConfig.html#mindspore.CheckpointConfig) API. + +The following uses a sample code to describe how to save the trained network model and parameters. + +```python +import mindspore as ms + +# Set the configuration information of the saved model. +config_ck = ms.CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10) +# Instantiate the saved model callback API and define the storage path and prefix. +ckpoint = ms.ModelCheckpoint(prefix="lenet", directory="./lenet", config=config_ck) + +# Start training and load the saved model and parameter callback function. +model.train(1, dataset_train, callbacks=[ckpoint]) +``` + +After the preceding code is executed, the generated checkpoint file directory structure is as follows: + +```text +./lenet/ +├── lenet-1_1875.ckpt # Parameter file. +└── lenet-graph.meta # Computational graph after compiled. +``` + +### LossMonitor + +To monitor the change of the loss function value during training and observe the running time of each epoch and step during training, [MindSpore Vision](https://mindspore.cn/vision/docs/en/master/index.html) provides the `LossMonitor` API (different from the `LossMonitor` API provided by MindSpore). + +The following uses sample code as an example: + +```python +from mindvision.engine.callback import LossMonitor + +# Start training and load the saved model and parameter callback function. The input parameters of LossMonitor are learning rate (0.01) and stride (375). +model.train(5, dataset_train, callbacks=[LossMonitor(0.01, 375)]) +``` + +```text + Epoch:[ 0/ 5], step:[ 375/ 1875], loss:[0.041/0.023], time:0.670 ms, lr:0.01000 + Epoch:[ 0/ 5], step:[ 750/ 1875], loss:[0.002/0.023], time:0.723 ms, lr:0.01000 + Epoch:[ 0/ 5], step:[ 1125/ 1875], loss:[0.006/0.023], time:0.662 ms, lr:0.01000 + Epoch:[ 0/ 5], step:[ 1500/ 1875], loss:[0.000/0.024], time:0.664 ms, lr:0.01000 + Epoch:[ 0/ 5], step:[ 1875/ 1875], loss:[0.009/0.024], time:0.661 ms, lr:0.01000 + Epoch time: 1759.622 ms, per step time: 0.938 ms, avg loss: 0.024 + Epoch:[ 1/ 5], step:[ 375/ 1875], loss:[0.001/0.020], time:0.658 ms, lr:0.01000 + Epoch:[ 1/ 5], step:[ 750/ 1875], loss:[0.002/0.021], time:0.661 ms, lr:0.01000 + Epoch:[ 1/ 5], step:[ 1125/ 1875], loss:[0.000/0.021], time:0.663 ms, lr:0.01000 + Epoch:[ 1/ 5], step:[ 1500/ 1875], loss:[0.048/0.022], time:0.655 ms, lr:0.01000 + Epoch:[ 1/ 5], step:[ 1875/ 1875], loss:[0.018/0.022], time:0.646 ms, lr:0.01000 + Epoch time: 1551.506 ms, per step time: 0.827 ms, avg loss: 0.022 + Epoch:[ 2/ 5], step:[ 375/ 1875], loss:[0.001/0.017], time:0.674 ms, lr:0.01000 + Epoch:[ 2/ 5], step:[ 750/ 1875], loss:[0.001/0.018], time:0.669 ms, lr:0.01000 + Epoch:[ 2/ 5], step:[ 1125/ 1875], loss:[0.004/0.019], time:0.683 ms, lr:0.01000 + Epoch:[ 2/ 5], step:[ 1500/ 1875], loss:[0.003/0.020], time:0.657 ms, lr:0.01000 + Epoch:[ 2/ 5], step:[ 1875/ 1875], loss:[0.041/0.019], time:1.447 ms, lr:0.01000 + Epoch time: 1616.589 ms, per step time: 0.862 ms, avg loss: 0.019 + Epoch:[ 3/ 5], step:[ 375/ 1875], loss:[0.000/0.011], time:0.672 ms, lr:0.01000 + Epoch:[ 3/ 5], step:[ 750/ 1875], loss:[0.001/0.013], time:0.687 ms, lr:0.01000 + Epoch:[ 3/ 5], step:[ 1125/ 1875], loss:[0.016/0.014], time:0.665 ms, lr:0.01000 + Epoch:[ 3/ 5], step:[ 1500/ 1875], loss:[0.001/0.015], time:0.674 ms, lr:0.01000 + Epoch:[ 3/ 5], step:[ 1875/ 1875], loss:[0.001/0.015], time:0.666 ms, lr:0.01000 + Epoch time: 1586.809 ms, per step time: 0.846 ms, avg loss: 0.015 + Epoch:[ 4/ 5], step:[ 375/ 1875], loss:[0.000/0.008], time:0.671 ms, lr:0.01000 + Epoch:[ 4/ 5], step:[ 750/ 1875], loss:[0.000/0.013], time:0.701 ms, lr:0.01000 + Epoch:[ 4/ 5], step:[ 1125/ 1875], loss:[0.009/0.015], time:0.666 ms, lr:0.01000 + Epoch:[ 4/ 5], step:[ 1500/ 1875], loss:[0.008/0.015], time:0.941 ms, lr:0.01000 + Epoch:[ 4/ 5], step:[ 1875/ 1875], loss:[0.008/0.015], time:0.661 ms, lr:0.01000 + Epoch time: 1584.785 ms, per step time: 0.845 ms, avg loss: 0.015 +``` + +According to the preceding information, the information printed by the `LossMonitor` API provided by the [MindSpore Vision toolkit](https://mindspore.cn/vision/docs/en/master/index.html) is more detailed. The stride is set to 375. Therefore, one record is printed every 375 steps, and the loss value fluctuates. However, in general, the loss value decreases gradually and the accuracy increases gradually. + +### ValAccMonitor + +To save the network model and parameters with the optimal accuracy during training, you need to validate them while training. MindSpore Vision provides the `ValAccMonitor` API. + +The following uses an example to describe the process. + +```python +from mindvision.engine.callback import ValAccMonitor + +download_eval = Mnist(path="./mnist", split="test", download=True) +dataset_eval = download_eval.run() + +# Start training and load the saved model and parameter callback function. +model.train(1, dataset_train, callbacks=[ValAccMonitor(model, dataset_eval, num_epochs=1)]) +``` + +```text + -------------------- + Epoch: [ 1 / 1], Train Loss: [0.000], Accuracy: 0.988 + ================================================================================ + End of validation the best Accuracy is: 0.988, save the best ckpt file in ./best.ckpt +``` + +After the preceding code is executed, the network model and parameters with the optimal accuracy are saved as the `best.ckpt` file in the current directory. + +## Customized Callback Mechanism + +MindSpore not only has powerful built-in callback functions, but also allows users to customize callback classes based on the `Callback` base class when they have special requirements. + +You can customize callbacks based on the `Callback` base class as required. The `Callback` base class is defined as follows: + +```python +class Callback(): + """Callback base class""" + def begin(self, run_context): + """Called once before the network executing.""" + pass # pylint: disable=W0107 + + def epoch_begin(self, run_context): + """Called before each epoch beginning.""" + pass # pylint: disable=W0107 + + def epoch_end(self, run_context): + """Called after each epoch finished.""" + pass # pylint: disable=W0107 + + def step_begin(self, run_context): + """Called before each step beginning.""" + pass # pylint: disable=W0107 + + def step_end(self, run_context): + """Called after each step finished.""" + pass # pylint: disable=W0107 + + def end(self, run_context): + """Called once after network training.""" + pass # pylint: disable=W0107 +``` + +The callback mechanism can record important information during training and transfer a dictionary variable `RunContext.original_args()` to the callback object so that users can obtain related attributes from each customized callback, perform customized operations, and customize other variables and transfer them to the `RunContext.original_args()` object. + +Common attributes in `RunContext.original_args()` are as follows: + +- epoch_num: number of training epochs +- batch_num: number of steps in an epoch +- cur_epoch_num: number of current epochs +- cur_step_num: number of current steps + +- loss_fn: loss function +- optimizer: optimizer +- train_network: training network +- train_dataset: training dataset +- net_outputs: network output + +- parallel_mode: parallel mode +- list_callback: all callback functions + +You can understand the customized callback mechanism in the following two scenarios: + +### Customized Training Termination Time + +The training can be terminated within a specified period. You can set a time threshold. When the training time reaches the threshold, the training process is terminated. + +In the following code, the `run_context.original_args` method can be used to obtain the `cb_params` dictionary which contains the main attribute information described above. + +In addition, you can modify and add values in the dictionary. Define an `init_time` object in the `begin` function and transfer it to the `cb_params` dictionary. After each step ends, the system checks whether the training time is greater than the configured time threshold. If the training time is greater than the configured time threshold, the system sends a training termination signal to `run_context` to terminate the training in advance and prints the current epoch, step, and loss values. + +```python +import time +import mindspore as ms + +class StopTimeMonitor(ms.Callback): + + def __init__(self, run_time): + """Define the initialization process.""" + super(StopTimeMonitor, self).__init__() + self.run_time = run_time # Define the execution time. + + def begin(self, run_context): + """Operations when training is started."" + cb_params = run_context.original_args() + cb_params.init_time = time.time() # Obtain the current timestamp as the training start time. + print("Begin training, time is:", cb_params.init_time) + + def step_end(self, run_context): + """Operations after each step ends.""" + cb_params = run_context.original_args() + epoch_num = cb_params.cur_epoch_num # Obtain the epoch value. + step_num = cb_params.cur_step_num # Obtain the step value. + loss = cb_params.net_outputs # Obtain the loss value. + cur_time = time.time() # Obtain the current timestamp. + + if (cur_time - cb_params.init_time) > self.run_time: + print("End training, time:", cur_time, ",epoch:", epoch_num, ",step:", step_num, ",loss:", loss) + run_context.request_stop() # Stop training. + +download_train = Mnist(path="./mnist", split="train", download=True) +dataset = download_train.run() +model.train(5, dataset, callbacks=[LossMonitor(0.01, 1875), StopTimeMonitor(4)]) +``` + +```text + Begin training, time is: 1648452437.2004516 + Epoch:[ 0/ 5], step:[ 1875/ 1875], loss:[0.011/0.012], time:0.678 ms, lr:0.01000 + Epoch time: 1603.104 ms, per step time: 0.855 ms, avg loss: 0.012 + Epoch:[ 1/ 5], step:[ 1875/ 1875], loss:[0.000/0.011], time:0.688 ms, lr:0.01000 + Epoch time: 1602.716 ms, per step time: 0.855 ms, avg loss: 0.011 + End training, time: 1648452441.20081 ,epoch: 3 ,step: 4673 ,loss: 0.014888153 + Epoch time: 792.901 ms, per step time: 0.423 ms, avg loss: 0.010 +``` + +According to the preceding information, when step 4673 of the third epoch is complete, the running time reaches the threshold and the training ends. + +### Customized Model Saving Threshold + +This callback mechanism is used to save the network model weight CKPT file when the loss is less than the specified threshold. + +The sample code is as follows: + +```python +import mindspore as ms + +# Define the callback API for saving the CKPT file. +class SaveCkptMonitor(ms.Callback): + """Define the initialization process.""" + + def __init__(self, loss): + super(SaveCkptMonitor, self).__init__() + self.loss = loss # Defines the loss threshold. + + def step_end(self, run_context): + """Define the operation to be performed when a step ends.""" + cb_params = run_context.original_args() + cur_loss = cb_params.net_outputs.asnumpy() # Obtain the current loss value. + + # If the current loss value is less than the preset threshold, the training stops. + if cur_loss < self.loss: + # Name the file to be saved. + file_name = str(cb_params.cur_epoch_num) + "_" + str(cb_params.cur_step_num) + ".ckpt" + # Save the network model. + ms.save_checkpoint(save_obj=cb_params.train_network, ckpt_file_name=file_name) + print("Saved checkpoint, loss:{:8.7f}, current step num:{:4}.".format(cur_loss, cb_params.cur_step_num)) + +model.train(1, dataset_train, callbacks=[SaveCkptMonitor(5e-7)]) +``` + +```text + Saved checkpoint, loss:0.0000001, current step num: 253. + Saved checkpoint, loss:0.0000005, current step num: 258. + Saved checkpoint, loss:0.0000001, current step num: 265. + Saved checkpoint, loss:0.0000000, current step num: 332. + Saved checkpoint, loss:0.0000003, current step num: 358. + Saved checkpoint, loss:0.0000003, current step num: 380. + Saved checkpoint, loss:0.0000003, current step num: 395. + Saved checkpoint, loss:0.0000005, current step num:1151. + Saved checkpoint, loss:0.0000005, current step num:1358. + Saved checkpoint, loss:0.0000002, current step num:1524. +``` + +The directory structure is as follows: + +```text +./ +├── 1_253.ckpt +├── 1_258.ckpt +├── 1_265.ckpt +├── 1_332.ckpt +├── 1_358.ckpt +├── 1_380.ckpt +├── 1_395.ckpt +├── 1_1151.ckpt +├── 1_1358.ckpt +├── 1_1524.ckpt +``` diff --git a/tutorials/source_en/advanced/train/images/model.png b/tutorials/source_en/advanced/train/images/model.png new file mode 100644 index 0000000000000000000000000000000000000000..6e868b7e6e893f2c1e23791195e579ffbc816881 GIT binary patch literal 11090 zcmb7qbyQSe^zQ{Eq(ceml&+y$7($SckdTrXVrUQ$iBUm@kdhh@5Ri}|q`M@Ap+q_r z1*DN?h`aLvKL z!-)vM_xwJk9Prm|FD-Ll03e~hdEtCt2)zIRHsB%jfpOr6?S+tFv=beUDP?O2;V*Mpv=hc-+)b3qG{0iFQr`r{ zO&!)vSVv6`@CB++ss{zTaAp8Mxj(fa*%Q)cHXFk*kLc|dA@3C*X78@O*B5^|X^lmh zqrSbbJqaS9R3%~$|9`(xe=mNVZ)hnz_4|1CQ$fKHy<(7GkY^9quY8C-8P{>3KrBgsehbM|c{^OYY{{4F)UO&Dl3N?4J!gtQXuq)y- z&5WQ_MTTyx!G`tp^;xYe>z#15UO976+!85tR{xBA%m z+sh_EZK9Nmi_3UaaXM1FcG3p6iXx9(sePQUQ2-i3*J+guYCM23`ilMf(`&v1!w6iw zM(*HwrFH#W|Mypu>Zu}wXWHO%INY9jrbr~Np359G_%983Yb4Sd?+XVXA72`h2QN^l zNjI=Ae^NB|zWxW9%1jgj?{1JcE+j@@SKyJgh^D zND@kr9a`})oO>o~Tl#TY_=&n+k?(!gq8J$Nv@Lz_novgYE!$W}^8j@{F|mWlRL77z zZDBY67l4i*wTQ#~lqsc9N6 z@3U3vqVY3XDyw0PfL^wOIU^}Sj>BqwdLxw2d}(oVCr|F0dU=nWyNZ z3+{-MS@_$|@INf=$#d6-{)UF%@4>Fb5#K!OpByYjucZ(_Nv23$jyy27!>^E>xV;20j^Y0(F__C>%tGN~Rc`rmW zOV6pJs+6Mn%DMSdSszb0Ws<{fcDc#Lq^%vn_V~pL0i7yzNlBcTF9PM;g=TF3f;(1K zdLQ_7>rH3A_GfCqo6oEtbG`Vh@?BJ)_c=OnLiPPQY=2^AZ3@ICr+^Zr2 zM#zNY0Y5(zzXgf3!IZhI?;Lr)a@dO|%W_E&D+?Ycxv#97wY6nHs*nzkDZIkTMOFE+ z`R%tRd#)?~46mclji86W>0CzZ6&;k@iZ%?cj-D3`E3_x-5Fm>Wrz`Dkc6;`ylWu5W zjO@PNhiM21NWR2@!%o}1iB}1>wT5$Uwc~&1Nk8%%u3`TAc>n+`RNT^jD=EWK2unI; z{;X z1zYn;Drq0y43ztoci!4#GC{fRpS<{@l0_v42I4N;kv zBFOFTrmm^Ag{?{!le+e+2SRr4wV_Wx={mtbMb*^?lcUZ6NLYEY9QD~+ett$^;=KTi zO&8`G?AO5B(jGc%14wggXwdr)!goMlm{hoy#pCc!SeTlp9#K(+XhC%V1k&Hc#Dv+m z(GrdV$gQjS4^Mi(y24qXuU%FPoGi$>9$n>#zcLzFVZ> zGEWJ{{7|)D?IOxL*3s9|$$T}cDV+-mf!4YRO&)L5M=g$f=(c#&z#B*C2dj2)GjtyL z73DM}fN{ePEvhk@BL%(|kt%B%3)R+v+boTMnNMnBF4z(Gze@HEYsAf1FI1;KA5S){ zuZ^v(J$K&^&F(Ik#kT&{Yo0uQwwko33mU1+y#bZj3EXnQ*qQJePA z4;(8-TCY=zUuTp>DJniszn;})Ajiv5DX-TxzMu6@V*m4tmrA=0E`1IyG#TU;4Tc07 zNF0?;Z%(dGZ3pt#O}Rx`GRzG){|KcM-x2ftZML64#RjMvy?rn>xJ)DlPh#`;p z^3X5mZ`&J^s#5KDr&D&dfwQIK+Hix!g@-8RcobYo&&jaX-S;tlh<7mXz7&k>VG{R^ zzM>+$p{yHi^Xfkd5KigM1`Yzn0F+7V|NIK|Bidea_x4}i)42zt{?4%!bDd)9Ez=T< zt<%>XWBX>ft!KgY$iBBz4d6UTmJGt(Sh`ABZ%l*y_V6=Jjj8EBKB~0#QMlv=ZtY#g{L>GE?W(ar7dU1~i_NlgD^_^Q4QdKIHmeRV% zQB#2dOUkqCcRxl@DoQ&jSOPwVYd`m-);$+qTPV+M%@k?CZ^0XD5QF{Or42A|JlN zzLGf?tNZkAORTyV8BPhj?&I&vBJW!r(=~mGX<{Vh{bBXb6OR+OBMq#o8Mt>ndlIGM z!t7V&ev{%PRFcCDU!5qO%31e{2Nq#E_Xmc&oJwD9R_g|OnvoZhF;LLuoihc=uR39H zzT9nV8dWNuA^lX|`GY<3OS_L`@ZBVYdiBeer=Hh*<(ZY_;~N}U*w0@0)H4<-dv|WH zljI&7CajF2a>zr4Uh4H8Su|981!Ig<1ACUqmDHZr!0dj&1i2mtd2;pf4DPp3y6`!lPEip^tk zw#;KTey9gJ^oqwagKjc{MbwL^n(O*u&5O&;!*jM~;x>Lr_QRb@t^=*s%_b!0DrSG$ zo_OU@7pv^!s0}^@dMAE*?pLzPL0C7S8WyNH)aQa8xlEy0`9sq&%^s8Z#@mYmpE6&1 zF|>1apJh%1%b7CwIIEIqssQJu7@g5RPqr)$@JGgays?ti%T6Jc?&R-9pvG23F#SvY z@ZNG~ZQlJk#_V^|EpdIYwrSTTp%uS+@7UDsxDNt^q7>3~`&VN$|6vy9Ih1K8?nu5` z6i2vGXS*b88%lvKEuVQC0vJ@26=UzHXw+lxWu9@ce4~%Q1npHJjCUZ4jvOO%ixOS5 zaSq!tYB8-RlKnjBC9PV0_a(diQet2%_vrZ4E;~~N9J?rTm^J%eYeh|uj+otUty^lV z4|Ey+v8htXw<9{u4RXBnh|TX+l(O8fnzm1MI5Xk2IEZpn}RDyN}XL1n0=%Q=!nd(6FNYKf|%$nI^6p5R5}73$4^?1O2Q z>0$NQ9hiQdH+THe`JDE50Ssr2|M5Em?5?ArhTaATMxTTkn=+uLvT^UuLLK2EtEFjb+iBMPH?E%K<@bnKUslRD-f`=uD~f&$laFk|Hz*1sV)p;$XH4e{Zq_>Ru> zBkq-~QRFD^P5uBNN9_XIScajCDV6|L8Ok>XPz5UT-Y+5yySEA-u<9gncLN`(=1JIK z*AHUSNyHE36}La}Ap?MQ`fko&E1&Wow=$uJZ_-n6)F1nV z<2H5R<+t%Q68nkVp2?HW>3ira=p$p^ycdj~O9wfXJ7&G|IAV3Kbu0wGt4Sj0y)aW1 zteOVWym@F;qC)5I{8rV^9nmN*TklbJz)dv`Xk z?BBg{SoTbd&l^{}(hEyX)~NlK|9F=^g3C$=99<{hwzCq1SVRYqF1DRbDo^xn@`kv# zJpI))NU)Ms>1!bE{Ah{Q)4m?Qw!<6|kvraS6UNlpE*bH7W*X3VY;fCze;=n^m&DN1 zsC9f8%k5<*fFmy#Qwm5>j_HRe*<7Cj4*U3pF4jYsP2{=&^tZ#R{zd3DIO?v>ze+h( zWO!Zh*}uV!&CTm5Bq_`qUnQwD^s8v>Wvb&lz!qr01=3}Gq#wSXP(~HLfFN=x$BY3! z0Ez0v$h(mXt^Upc>uy!Jn_Dn7ovDFjQ0rn?9fXBSUR_t*P*DbiVzDvL8udMggp$pl>SeZKiF8J=^@k?RcO5@8M*tC{%t z_+8s^>eF2-`xoCgkAyud-K(tjyEBu$z%mctborMH{#ajnp)6}hJ5FHt?#W1%Jj*kR zim7HxI9blfiSgmF3fo?;l~_;ajl^nWi$Gr41yJU}AB{6%FxLcZGk^VHikFAuMO$$X zZ*KBi)K`yV_t{;<-OzR$Z8#BtaLK5k8dE+CZo~8Mv-=<446W?XlXv{XUh4YyaR~p}4{Xzf z#^aoRUfN{^gP6lR9YiEm+1^<&-gyq^f8KnDT*aAo4Pg4k5#Lvxj+~?!yLNGQrf++3 zuo~2CPaV}Iq05SpCJ1_sf1_}`Lt%)7HhaRRrqaT;@m^X6|c*Ic| z*lMke2nbfn36`M#59ePBbkg*`2yVW~%ggg=wrA*XvEKE8S(DpFKBGVB4@6F;7cz`m zv@;Xk!u9V<;{Bi4fL&3kvT_InnOj>gro4S7NVsQJc4Vae`3&z27dEB@Iynw?atrsB zU9xvVy zvLgrx$`{~B3Gw&fhP1$%J&dm)6f^;z`f2iV0P2hecIC)M$y+VYlQ{jP)GX1`)bym% z(R_(dSa=Z@pb({XgAY|hZQT=4#m_QphNb4@D68?MXhKb0%a*1r){>KxgY5z=Ird+) zE{cH7m)3&XwvvX1Q{=)v7yo}&?~@o(#yzELURjT43Eu0>?X(qTRZ*gv02^QzOUUu!6ozsAEBTw~*@J;6#5RE!x z1Y=IEd4)P8#wDZuiy*vsX*b6;KrrN{SV4wY`u+W$DLB|PwRWIeFQ0m~+)2plXP%le zf;j`oDIZsktUa|2u*}=NT_LqVylmqr23@z;? z+LaxB0GoDd{(ao%h5dO%92xf=6`C~uGmWUOFWPWpkWt!00gR`R#J-irOymwXhcsjg zj~4`I#mxri9*xhJ)LwB>5bIgEoBk(h&+-L0r1fx9uelMV@ znLH6cPy`BbPO~P@XFy7j1l)WB^H}$STEQ-G7jj*E?CJg%W3a#{5P1jweebpYL($WxFRKkV1c1bcby5uk7xb<@r`RcIVP8u(O*2=K|;$fA{VkN!StE=5lED z%el}EXuau_m@C2F9Nw!AM2WaZMxngiXWWDVj69`wNp_+UB2OBmW2)TW8;n3Gfh}@r z*Rx8xeV8FNMg-hsP&dqXTZ31DQA zHa0w}05`$;lJgGU!v_yKk5U~ki8|7d>rP5JBk+Jncn=REb_vOu8s4K~C@sv{r~#@J zzJfMc?_b^3t_t^>G5T1-DO4-j`orH7_n=t=4akOK6H6$Z-r9 zYKPP-PqtfBIXsLubS-&CZ-jcmkLaJ$gdP$@!c?Ji1*&!H?mr{APw>EIrF@E^2uROM z)67#UP{6?2I+T&Bm7wE`EXx=!)m{AziErZqgaQtdAzNeDgoPihWwX~gW}DWEv8)Ep zBU>;hLGvoEp1*$O_N&+^=h~6AIfsBpm9`nH4O$!p$P(Z$!D^rjoZ!Ixwp zc3{krLA>l6PQzDr6bf6kv9z4Oum3apss~I!`cM*_`CH4f1wRhJU5f1pRbaw`7=QrjC>ohcGj<2?JPmFOx8kjKo2g1nb+kg) zX214Qy?Z664&B8IWX&D9#4cU#&x(zm{8?MOZhb}{06ACy$~p4VP>rDEXm<;6%L^fg z`BOgpd2;JM8Vgn33skUS&-(-L0D%K*2SW9-pJ0uRjSo#Nz11BwN@R0gU0t>9R!^eV z@i>C$I$&L*i%SjOE3%ntw0XK`XJ^&Ph9^J+qly9iQXIhwW;iRUNS-E@4KR>r@3t+v z$|C+i50xOwKTojNwx$X{M-*D3kvw9)1PDe-?P4A%tT6oy4+sPH+Lm^eJxohg|Fa{Q zLNN}jF3`iQJ@_^^M}~-iF;@5{qRajTDg^KsRH0sQBZ>qEGmH!keTt>yvNAI>!CqiL z8ocx=xk($Y7=`4)Y2c?PCCA?rgRrvYozzTT<1A}}iF9QVsf#oN315zC$YB))cxX6# z_vmq)tJ&Mz=UNg@E8nOFXIRE+E6_%~jD{ZCKo8v%D(Z=+4jPHUQA}}0D2uftOJa9_#`z8>ami*eq&&fgANMexQjqQEYImkh=U2e(r5fN8XQgU1Svlze=slRV^H#;Sx8$$>L zlP&|QMQ5REP}adnj+i$78!qqqnf?_sMg5o`5=PcxR(7NZTo)u;gq$D#%>tjAr7l!P zpMpvciNm9;9QM|5gO=mo(&*?OfVdvSTvHL<@)_&7$8zKCuD4{FDo~pA7YbCmD(iWt zw}D<(d4Itn(bzYbiyZCGJTRq?$wAB=IE};5L&u-+yP0%mlw)^bZL}R9_atBK!poL$ z_ujoaev$PnPYLBjfc}Y|@&4+5U{%yQ zZMc%=E-lAwA0`ZU58l3HGrHI&@T63$9F2zSWrM3xrAybhUA+xx05U|&9}PDsA*{Fd zh#{;Jq${If#*ORU^u9gD=;P~4mNqDv+twD!Ph~~ImSm<2?BS?#t1M1gw0oFOVn-AX zFk*~dcW?9bKAaR*fO!zCmiG?edk>hESa&osiOof$`JM=I#5sW7txLCo#JhO|*a9CGRD^)@;vs-8Cn?Oc)~Z4XlS?;WQ^c_3yWELmu81=Nn>xH#j@v#VH9GKg~DEdUJ>LyQXC~qqrwAmqJGSS5CdV>exHLR*J!qCRMpR7 z%c*K$@UfV*w4SsM01%(voCR2Ioh@`-lS)oW!}}xu1^z%PZ(J}r4Nn%|Oo4H_vXKZ) z%hG+mT}^W=Yr85wF0lZxfJFkR_KS?(%8Igd0*f4ZVN({$<;mLit7E{`vu-4njv~tlJ;yyfTT+BQkx8tQHZ+W2PrB?#{_QiS zTV4jjy>~A=lel#Zlit5y0Q2X-lcZu&fDSNrE)zf&zeWYzd%Isc*`!)daYu0sUPOZS zZ$&)Mj4`k#cldM1_Gkb#?NZ)Vhw0g?z-Sy`3>rW?zc%w=W!1qHfR>2@^z2WLLt zh9qWX#kny-XG+AkrX`!#^q*YFbS;^t7dBFdaLvdanp3XaCsidE{z$6wL_@qcjJ7-X z8C_UNwYbsV`g1;dWl|ZxFSN>6_Re+u0=_aVt-=f|kr9j4Z73f`nKiO~&)d-#wFxGx%K_$}q-oXU489EyA z`Y6yta_{1+iJ2MBMX<1eaSdMc6By^qsW=^2?Dqfvh*pnd=`$`jXQ}GCdu}1_q)nm6hI~#{V)sGoREd_@M=3 zc>v|?(k;hdhOKH*KPT_lr?rVhI!hx~K)jlPX5Tj}O}?8aAH@BwWVVs)Pk-k*el98> zOHyU;i%tVgnw`D@ln{0Z zms}}+jRN;BFhKEy{qm}@pP%?B^Bps<8hKk-DC2;ION@U~sg?Wyq7)Ycy(=nD$UFBo z%Y=kd5oKasL+Kk0u#w2(qbu23iQ{_?XYj0F{7I25bm{$NkH7hJ-b1erENz9t2&`i| z#q4$;2}!X&>de?QOu#3kB_pxZainAzR}FA-I4N) zNCKM2;)qUwK%gKAhZMHT4qTC|fS*lT+~-VAm#mj0F6Q0q( z>1q#qL)JzAdUSWeHav7B+0lH*t4ad5Lkr+UI&(oZJ>qKwH)EgCgEB%+7h05lb_X$T zu?g1f{5SegX&BR|e0_1S^6w=lbyh2yWML)Y_Qzj)Bq|IEjaQ+Mip`Mj@-`|khJ}M`*Ayo`~dKgh=h(- z(o-lXl#Ujf)sEp0Bu6_HYL-1J-Poue%{@C<^9)M0JFW{@>hWnS2?>QFanJq5l21Qe`F3NsSA zctX^29IlcsaG}m>!A(IbF$7P6F{KI(gb+u>lMNzG zc29y=3#z*|l0z{$F(G*rX`xZ*Laq@g$YEo#$qqe?*oI1c`ct=OsyW9<$-){=y&(tl zXe&#WD^B)#`@T>O(-M!=dz$(@8mV`mn>#D${BM!|(J+Jqx*IiOll(@$56+UkmnA2MUjX%N!&6M0PEh5}M@TyPU&8&S2qXuxBT zh}-U^2-nKa-f{HEhkp9EB3lrTK$?L?GF(okey~7RCqxXFS#0&m1ZQyX6j^}E3j&bI{^Z)E25Ls&>sNhb|>O?;jEqfrp-qn%7ie2Zv)F*9E3p8GkS7B59SZ8Q=es% z7}bmH`Ux!82oB z@(ZcRI>>=8QX4^;p<4=XB+GS>s=72j_2+un^huE_i_ojByu5q|l%tCA!jq;nb{~;C zNha1$!c81=bE=86)sA1MnB^F~GC|Ykkpfjd24Cd-+>Oq+Kq4dUM=@(#_B!J7wg)yB z9FA_8&Xh!SoNwobX6nyG#I4yR#2uo16HScce16;dO97>B`P&7T5j>w&=~_6QF1d>c zs&_?Ho|r#A#XU7shPURiho=}$N=N%v)Y*$b25g)_%*F>m5c0Ec9mL8R|C zui>6<;numKdp3(iYz;N|X@_)sN=Jm3hbE6vsnH8R$KxUw(VfzZWWPia=DtSzzD!*d z7A!j5>Vdt*QWq%(#3UrlP9xdRvlU-XK7i^KjsFFf-CSLpxT1`-WdWHtogX>C+SLv= zi7Bm}lsm8M`Z=2FCEfKOGx6JhoPI{B;;4_*)8SHWcD_ZIe1rdTAHB!X*&Bb+kimU^g+ z*K)_T!>TSI?iQ56_vzmy^4{cC1XX;1gn2C3>^(Ay^MSMP(cr?{gZHC|t({|r2F zk4FbkyMs0nQyY^8OQ)pG%}pn~^uk&ehyUr@uypJQRJ8aICSoja4xNU8os0PEd+%5U TDoMdZvcN-iJ!rYA?d$&qNl+dA literal 0 HcmV?d00001 diff --git a/tutorials/source_en/advanced/train/metric.md b/tutorials/source_en/advanced/train/metric.md new file mode 100644 index 0000000000..a0bb31624e --- /dev/null +++ b/tutorials/source_en/advanced/train/metric.md @@ -0,0 +1,205 @@ +# Evaluation Metrics + + + +When a training task is complete, an evaluation function (Metric) is often required to evaluate the quality of a model. Different training tasks usually require different metric functions. For example, for a binary classification problem, common evaluation metrics include precision, recall, and the like. For a multiclass classification task, macro and micro may be used for evaluation. + +MindSpore provides evaluation functions for most common tasks, such as `nn.Accuracy`, `nn.Precision`, `nn.MAE`, and `nn.MSE`. The evaluation functions provided by MindSpore cannot meet the requirements of all tasks. In most cases, you need to customize metrics for a specific task to evaluate the trained model. + +The following describes how to customize metrics and how to use metrics in `nn.Model`. + +> For details, see [Evaluation Metrics](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html#evaluation-metrics). + +## Customized Metrics + +The customized metric function needs to inherit the `nn.Metric` parent class and re-implement the `clear`, `update`, and `eval` methods in the parent class. + +- `clear`: initializes related internal parameters. +- `update`: receives network prediction output and labels, computes errors, and updates internal evaluation results after each step. +- `eval`: computes the final evaluation result after each epoch ends. + +The mean absolute error (MAE) algorithm is shown in formula (1): + +$$ MAE=\frac{1}{n}\sum_{i=1}^n\lvert ypred_i - y_i \rvert \tag{1}$$ + +The following uses the simple MAE algorithm as an example to describe the `clear`, `update`, and `eval` functions and their usage. + +```python +import numpy as np +import mindspore as ms +import mindspore.nn as nn + +class MyMAE(nn.Metric): + def __init__(self): + super(MyMAE, self).__init__() + self.clear() + + def clear(self): + """Initialize variables _abs_error_sum and _samples_num.""" + self._abs_error_sum = 0 # Save error sum. + self._samples_num = 0 # Accumulated data volume. + + @nn.rearrange_inputs + def update(self, *inputs): + """Update _abs_error_sum and _samples_num.""" + y_pred = inputs[0].asnumpy() + y = inputs[1].asnumpy() + + # Compute the absolute error between the predicted value and the actual value. + abs_error_sum = np.abs(y - y_pred) + self._abs_error_sum += abs_error_sum.sum() + + # Total number of samples + self._samples_num += y.shape[0] + + def eval(self): + """Compute the final evaluation result."" + return self._abs_error_sum / self._samples_num + +# The network has two outputs. +y_pred = ms.Tensor(np.array([[0.1, 0.2, 0.6, 0.9], [0.1, 0.2, 0.6, 0.9]]), ms.float32) +y = ms.Tensor(np.array([[0.1, 0.25, 0.7, 0.9], [0.1, 0.25, 0.7, 0.9]]), ms.float32) + +error = MyMAE() +error.clear() +error.update(y_pred, y) +result = error.eval() +print("output(y_pred, y):", result) +``` + +```text + output(y_pred, y): 0.1499999612569809 +``` + +Note that if the network has multiple outputs in `update`, but only two outputs are used for evaluation, you can use the `set_indexes` method to rearrange the input of `update` to compute evaluation metrics. To use the `set_indexes` method, you need to use the modifier `nn.rearrange_inputs` to modify the `update` method. Otherwise, the input configured using `set_indexes` does not take effect. + +```python +# The network has three outputs: y_pred, y, and z. +z = ms.Tensor(np.array([[0.1, 0.25, 0.7, 0.8], [0.1, 0.25, 0.7, 0.8]]), ms.float32) + +# Use y_pred and z for evaluation. +error = MyMAE().set_indexes([0, 2]) +error.clear() +error.update(y_pred, y, z) +result = error.eval() +print("output(y_pred,z):", result) +``` + +```text + output(y_pred,z): 0.24999992549419403 +``` + +## Using Metrics in Model Training + +[mindspore.Model](https://www.mindspore.cn/docs/en/master/api_python/mindspore/mindspore.Model.html#mindspore.Model) is a high-level API used for training and evaluation. You can import customized or MindSpore existing metrics as parameters. Models can automatically call the imported metrics for evaluation. + +After network model training, metrics need to be used to evaluate the training effect of the network model. Therefore, before specific code is demonstrated, you need to prepare a dataset, load the dataset, and define a simple linear regression network model. + +$$f(x)=w*x+b \tag{2}$$ + +```python +import numpy as np +import mindspore.nn as nn +import mindspore as ms +from mindspore import dataset as ds +from mindspore.common.initializer import Normal + +def get_data(num, w=2.0, b=3.0): + """Generate data and corresponding labels.""" + for _ in range(num): + x = np.random.uniform(-10.0, 10.0) + noise = np.random.normal(0, 1) + y = x * w + b + noise + yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32) + +def create_dataset(num_data, batch_size=16): + """Load the dataset.""" + dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label']) + dataset = dataset.batch(batch_size) + return dataset + +class LinearNet(nn.Cell): + """Define the linear regression network."" + def __init__(self): + super(LinearNet, self).__init__() + self.fc = nn.Dense(1, 1, Normal(0.02), Normal(0.02)) + + def construct(self, x): + return self.fc(x) + +loss = nn.L1Loss() +``` + +### Using Built-in Evaluation Metrics + +When the built-in metrics of MindSpore are transferred to `Model` as parameters, the metrics can be defined as a dictionary type. The `key` of the dictionary is a character string, and the `value` of the dictionary is the built-in [evaluation metric](https://www.mindspore.cn/docs/en/master/api_python/mindspore.nn.html#evaluation-metrics) of MindSpore. The following example uses `nn.Accuracy` to compute the classification accuracy. + +```python +import mindspore.nn as nn +import mindspore as ms +from mindvision.engine.callback import LossMonitor + +ds_train = create_dataset(num_data=160) +net = LinearNet() +opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) + +# Define a model and use the built-in Accuracy function. +model = ms.Model(net, loss, opt, metrics={"MAE": nn.MAE()}) +model.train(epoch=1, train_dataset=ds_train, callbacks=LossMonitor(0.005)) + +# Evaluate the model. +ds_eval = create_dataset(num_data=160) +output = model.eval(ds_eval) +print(output) +``` + +```text + Epoch:[ 0/ 1], step:[ 1/ 10], loss:[10.206/10.206], time:148.661 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 2/ 10], loss:[8.827/9.516], time:0.671 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 3/ 10], loss:[13.232/10.755], time:0.681 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 4/ 10], loss:[10.893/10.789], time:0.704 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 5/ 10], loss:[8.339/10.299], time:0.668 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 6/ 10], loss:[8.881/10.063], time:0.826 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 7/ 10], loss:[6.288/9.524], time:0.923 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 8/ 10], loss:[8.166/9.354], time:0.932 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 9/ 10], loss:[7.538/9.152], time:0.932 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 10/ 10], loss:[5.517/8.789], time:0.980 ms, lr:0.00500 + Epoch time: 167.900 ms, per step time: 16.790 ms, avg loss: 8.789 + {'MAE': 5.931522464752197} +``` + +### Using Customized Evaluation Metrics + +In the following example, the customized evaluation metric `MAE()` is transferred to `Model`, and the evaluation dataset is transferred to the `model.eval()` API for evaluation. + +The validation result is of the dictionary type. The `key` of the validation result is the same as that of `metrics`. The `value` of the `metrics`result is the mean absolute error between the predicted value and the actual value. + +```python +ds_train = create_dataset(num_data=160) +net1 = LinearNet() +opt = nn.Momentum(net1.trainable_params(), learning_rate=0.005, momentum=0.9) + +# Define a model and transfer the customized metrics function MAE to the model. +model1 = ms.Model(net1, loss, opt, metrics={"MAE": MyMAE()}) +model1.train(epoch=1, train_dataset=ds_train, callbacks=LossMonitor(0.005)) + +# Evaluate the model. +ds_eval = create_dataset(num_data=160) +output = model1.eval(ds_eval) +print(output) +``` + +```text + Epoch:[ 0/ 1], step:[ 1/ 10], loss:[9.931/9.931], time:157.518 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 2/ 10], loss:[10.705/10.318], time:0.751 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 3/ 10], loss:[11.313/10.650], time:0.722 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 4/ 10], loss:[9.445/10.349], time:0.738 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 5/ 10], loss:[5.492/9.377], time:0.737 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 6/ 10], loss:[8.060/9.158], time:0.839 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 7/ 10], loss:[7.866/8.973], time:0.900 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 8/ 10], loss:[7.264/8.760], time:0.863 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 9/ 10], loss:[8.975/8.784], time:0.885 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 10/ 10], loss:[7.630/8.668], time:0.958 ms, lr:0.00500 + Epoch time: 177.346 ms, per step time: 17.735 ms, avg loss: 8.668 + {'MAE': 5.533915233612061} +``` diff --git a/tutorials/source_en/advanced/train/model.md b/tutorials/source_en/advanced/train/model.md new file mode 100644 index 0000000000..811c68dd06 --- /dev/null +++ b/tutorials/source_en/advanced/train/model.md @@ -0,0 +1,386 @@ +# Basic Usage of Models + + + +Generally, defining a training and evaluation network and running it directly can meet basic requirements. + +On the one hand, `Model` can simplify code to some extent. For example, you do not need to manually traverse datasets. If you do not need to customize `nn.TrainOneStepCell`, you can use `Model` to automatically build a training network. You can use the `eval` API of `Model` to evaluate the model and directly output the evaluation result. You do not need to manually invoke the `clear`, `update`, and `eval` functions of evaluation metrics. + +On the other hand, `Model` provides many high-level functions, such as data offloading and mixed precision. Without the help of `Model`, it takes a long time to customize these functions by referring to `Model`. + +The following describes MindSpore models and how to use `Model` for model training, evaluation, and inference. + +![model](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_en/advanced/train/images/model.png) + +## Introduction to Model + +[Model](https://mindspore.cn/docs/en/master/api_python/mindspore/mindspore.Model.html#mindspore.Model) is a high-level API provided by MindSpore for model training, evaluation, and inference. The common parameters of the API are as follows: + +- `network`: neural network used for training or inference. +- `loss_fn`: used loss function. +- `optimizer`: used optimizer. +- `metrics`: evaluation function used for model evaluation. +- `eval_network`: network used for model evaluation. If the network is not defined, `Model` uses `network` and `loss_fn` for encapsulation. + +`Model` provides the following APIs for model training, evaluation, and inference: + +- `train`: used for model training on the training set. +- `eval`: used to evaluate the model on the evaluation set. +- `predict`: performs inference on a group of input data and outputs the prediction result. + +### Using the Model API + +For a neural network in a simple scenario, you can specify the feedforward network `network`, loss function `loss_fn`, optimizer `optimizer`, and evaluation function `metrics` when defining `Model`. + +In this case, `Model` uses `network` as the feedforward network, uses `nn.WithLossCell` and `nn.TrainOneStepCell` to build a training network, and uses `nn.WithEvalCell` to build an evaluation network. + +```python +import numpy as np +import mindspore.dataset as ds +import mindspore.nn as nn +import mindspore as ms +from mindspore.common.initializer import Normal + +def get_data(num, w=2.0, b=3.0): + """Generate sample data and corresponding labels.""" + for _ in range(num): + x = np.random.uniform(-10.0, 10.0) + noise = np.random.normal(0, 1) + y = x * w + b + noise + yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32) + +def create_dataset(num_data, batch_size=16): + """Generate a dataset.""" + dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label']) + dataset = dataset.batch(batch_size) + return dataset + +class LinearNet(nn.Cell): + """Define the linear regression network."" + def __init__(self): + super().__init__() + self.fc = nn.Dense(1, 1, Normal(0.02), Normal(0.02)) + + def construct(self, x): + return self.fc(x) + +train_dataset = create_dataset(num_data=160) +net = LinearNet() +crit = nn.MSELoss() +opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) + +# Use a model to build a training network. +model = ms.Model(network=net, loss_fn=crit, optimizer=opt, metrics={"mae"}) +``` + +### Model Training + +Use the `train` API to perform model training. The common parameters of the `train` API are as follows: + +- `epoch`: number of training epoch. Generally, each epoch uses the full dataset for training. +- `train_dataset`: Iterator of a training dataset. +- `callbacks`: callback object or callback object list to be executed during training. + +Interestingly, if `loss_fn` is defined in the network model, the data and label are transferred to `network` and `loss_fn` respectively. In this case, a tuple (data, label) needs to be returned for the dataset. If a dataset contains multiple pieces of data or labels, you can set `loss_fn` to None and implement a customized loss function in the `network`. In this case, tuples (data1, data2, data3, ...) consisting of all data returned by the dataset are transferred to the `network`. + +The following example uses the `train` API to perform model training and uses the `LossMonitor` callback function to view the loss function value during training. + +```python +from mindvision.engine.callback import LossMonitor + +# Model training. The input parameter 0.005 of LossMonitor indicates the learning rate. +model.train(1, train_dataset, callbacks=[LossMonitor(0.005)]) +``` + +```text + Epoch:[ 0/ 1], step:[ 1/ 10], loss:[115.354/115.354], time:242.467 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 2/ 10], loss:[86.149/100.751], time:0.650 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 3/ 10], loss:[17.299/72.934], time:0.712 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 4/ 10], loss:[21.070/59.968], time:0.744 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 5/ 10], loss:[42.781/56.530], time:0.645 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 6/ 10], loss:[52.374/55.838], time:0.577 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 7/ 10], loss:[53.629/55.522], time:0.588 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 8/ 10], loss:[16.356/50.626], time:0.624 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 9/ 10], loss:[5.504/45.613], time:0.730 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 10/ 10], loss:[5.396/41.591], time:0.766 ms, lr:0.00500 + Epoch time: 259.696 ms, per step time: 25.970 ms, avg loss: 41.591 +``` + +### Model Evaluation + +The `eval` API is used for evaluation. The parameters of the `eval` API are as follows: + +- `valid_dataset`: dataset of the evaluation model. +- `callbacks`: callback object or callback object list to be executed during evaluation. +- `dataset_sink_mode`: determines whether data is directly offloaded to the processor for processing. + +```python +eval_dataset = create_dataset(num_data=80) # Create an evaluation dataset. +eval_result = model.eval(eval_dataset) # Execute model evaluation. +print(eval_result) +``` + +```text + {'mae': 4.2325128555297855} +``` + +### Model Inference + +The `predict` API is used for prediction. The parameters of the `predict` API are as follows: + +- `predict_data`: prediction sample. The data can be a single tensor, tensor list, or tensor tuple. + +```python +eval_data = eval_dataset.create_dict_iterator() +data = next(eval_data) +# Perform model prediction. +output = model.predict(data["data"]) +print(output) +``` + +```text + [[-6.9463778 ] + [ 1.3816066 ] + [13.233659 ] + [11.863918 ] + [ 0.73616135] + [-0.1280173 ] + [ 7.579297 ] + [-4.9149694 ] + [ 7.416003 ] + [10.491856 ] + [-5.7275047 ] + [ 9.984399 ] + [-7.156473 ] + [ 2.7091386 ] + [-6.3339615 ] + [-6.0259247 ]] +``` + +Generally, you need to post-process the inference result to obtain an intuitive inference result. + +## Customized Scenarios + +The network encapsulation functions `nn.WithLossCell`, `nn.TrainOneStepCell`, and `nn.WithEvalCell` provided by MindSpore are not applicable to all scenarios. In actual scenarios, you need to customize network encapsulation functions. In this case, it is unreasonable for `Model` to use these encapsulation functions to automatically encapsulate packets. + +Next, let's look at how to correctly use `Model` when customizing network encapsulation functions. + +### Customized Loss Network + +If there are multiple data records or labels, you can use a customized loss network to link the feedforward network and the customized loss function as the `network` of `Model`. The default value of `loss_fn` is `None`. In this case, `Model` does not pass through `nn.WithLossCell`, `nn.TrainOneStepCell` is directly used to form a training network with the `optimizer`. + +```python +import numpy as np +import mindspore.dataset as ds +import mindspore.ops as ops +import mindspore.nn as nn +import mindspore as ms +from mindspore.nn import LossBase +from mindvision.engine.callback import LossMonitor + +def get_multilabel_data(num, w=2.0, b=3.0): + """Generate multi-label data. A group of data x corresponds to two labels: y1 and y2.""" + for _ in range(num): + x = np.random.uniform(-10.0, 10.0) + noise1 = np.random.normal(0, 1) + noise2 = np.random.normal(-1, 1) + y1 = x * w + b + noise1 + y2 = x * w + b + noise2 + yield np.array([x]).astype(np.float32), np.array([y1]).astype(np.float32), np.array([y2]).astype(np.float32) + +def create_multilabel_dataset(num_data, batch_size=16): + """Generate a multi-label dataset. One piece of data corresponds to two labels: label1 and label2.""" + dataset = ds.GeneratorDataset(list(get_multilabel_data(num_data)), column_names=['data', 'label1', 'label2']) + dataset = dataset.batch(batch_size) + return dataset + +class L1LossForMultiLabel(LossBase): + """Customize a multi-label loss function.""" + + def __init__(self, reduction="mean"): + super(L1LossForMultiLabel, self).__init__(reduction) + self.abs = ops.Abs() + + def construct(self, base, target1, target2): + """There are three inputs: predicted value 'base', actual values 'target1' and 'target2'.""" + x1 = self.abs(base - target1) + x2 = self.abs(base - target2) + return self.get_loss(x1) / 2 + self.get_loss(x2) / 2 + +class CustomWithLossCell(nn.Cell): + """Connect the feedforward network and loss function.""" + + def __init__(self, backbone, loss_fn): + """There are two inputs: feedforward network 'backbone' and loss function 'loss_fn'.""" + super(CustomWithLossCell, self).__init__(auto_prefix=False) + self._backbone = backbone + self._loss_fn = loss_fn + + def construct(self, data, label1, label2): + output = self._backbone(data) # Network output obtained through forward computation + return self._loss_fn(output, label1, label2) # Obtain the multi-label loss value. + +multi_train_dataset = create_multilabel_dataset(num_data=160) + +# Build a linear regression network. +net = LinearNet() +# Multi-label loss function +loss = L1LossForMultiLabel() + +# Connect linear regression networks and multi-label loss functions. +loss_net = CustomWithLossCell(net, loss) +opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) + +# Use the model to connect the network and optimizer. In this case, the model does not pass through nn.WithLossCell. +model = ms.Model(network=loss_net, optimizer=opt) +# Use the train API for model training. +model.train(epoch=1, train_dataset=multi_train_dataset, callbacks=[LossMonitor(0.005)]) +``` + +```text + Epoch:[ 0/ 1], step:[ 1/ 10], loss:[11.036/11.036], time:212.864 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 2/ 10], loss:[9.984/10.510], time:0.592 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 3/ 10], loss:[9.300/10.107], time:0.660 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 4/ 10], loss:[7.526/9.462], time:0.787 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 5/ 10], loss:[6.959/8.961], time:0.715 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 6/ 10], loss:[10.290/9.183], time:0.716 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 7/ 10], loss:[10.067/9.309], time:0.770 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 8/ 10], loss:[8.924/9.261], time:0.909 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 9/ 10], loss:[7.257/9.038], time:0.884 ms, lr:0.00500 + Epoch:[ 0/ 1], step:[ 10/ 10], loss:[6.138/8.748], time:0.955 ms, lr:0.00500 + Epoch time: 232.046 ms, per step time: 23.205 ms, avg loss: 8.748 +``` + +### Customized Training Network + +When customizing a training network, you need to manually build a training network as the `network` of `Model`. The default values of `loss_fn` and `optimizer` are None. In this case, `Model` uses the `network` as the training network without any encapsulation. + +The following example describes how to customize a training network `CustomTrainOneStepCell` and build a training network through the `Model` API. + +```python +import mindspore.ops as ops +import mindspore as ms +from mindvision.engine.callback import LossMonitor + +class CustomTrainOneStepCell(nn.Cell): + """Customize a training network."" + + def __init__(self, network, optimizer, sens=1.0): + """There are three input parameters: training network, optimizer, and backward propagation scaling ratio.""" + super(CustomTrainOneStepCell, self).__init__(auto_prefix=False) + self.network = network #Define the feedforward network. + self.network.set_grad() # Build a backward network. + self.optimizer = optimizer #Define the optimizer. + self.weights = self.optimizer.parameters # Parameters to be updated. + self.grad = ops.GradOperation(get_by_list=True, sens_param=True) #Obtain the gradient by backward propagation. + + def construct(self, *inputs): + loss = self.network(*inputs) # Execute the feedforward network and compute the loss function value of the current input. + grads = self.grad(self.network, self.weights)(*inputs, loss) # Perform backward propagation and compute the gradient. + loss = ops.depend(loss, self.optimizer(grads)) # Use the optimizer to update gradients. + return loss + +multi_train_ds = create_multilabel_dataset(num_data=160) + +# Manually build a training network. +train_net = CustomTrainOneStepCell(loss_net, opt) +# Build a training network. +model = ms.Model(train_net) +# Perform model training. +model.train(epoch=1, train_dataset=multi_train_ds, callbacks=[LossMonitor(0.01)]) +``` + +```text + Epoch:[ 0/ 1], step:[ 1/ 10], loss:[5.165/5.165], time:183.006 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 2/ 10], loss:[4.042/4.603], time:0.800 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 3/ 10], loss:[3.385/4.197], time:0.886 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 4/ 10], loss:[2.438/3.758], time:0.896 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 5/ 10], loss:[2.457/3.498], time:0.819 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 6/ 10], loss:[2.546/3.339], time:0.921 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 7/ 10], loss:[4.569/3.515], time:0.973 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 8/ 10], loss:[4.031/3.579], time:1.271 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 9/ 10], loss:[6.138/3.864], time:1.035 ms, lr:0.01000 + Epoch:[ 0/ 1], step:[ 10/ 10], loss:[3.055/3.783], time:1.263 ms, lr:0.01000 + Epoch time: 203.473 ms, per step time: 20.347 ms, avg loss: 3.783 +``` + +### Customized Evaluation Network + +By default, the `Model` uses `nn.WithEvalCell` to build the evaluation network. If the requirements are not met, you need to manually build the evaluation network, for example, in the multi-data and multi-label scenarios. + +The following example shows how to customize an evaluation network `CustomWithEvalCell` and use the `Model` API to build an evaluation network. + +```python +import mindspore.nn as nn +import mindspore as ms + + +class CustomWithEvalCell(nn.Cell): + """Customize multi-label evaluation network.""" + + def __init__(self, network): + super(CustomWithEvalCell, self).__init__(auto_prefix=False) + self.network = network + + def construct(self, data, label1, label2): + output = self.network(data) + return output, label1, label2 + +# Build a multi-label evaluation dataset. +multi_eval_dataset = create_multilabel_dataset(num_data=80) + +# Build an evaluation network. +eval_net = CustomWithEvalCell(net) + +# Evaluation function +mae1 = nn.MAE() +mae2 = nn.MAE() +mae1.set_indexes([0, 1]) +mae2.set_indexes([0, 2]) + +# Use a model to build an evaluation network. +model = ms.Model(network=loss_net, optimizer=opt, eval_network=eval_net, + metrics={"mae1": mae1, "mae2": mae2}) +result = model.eval(multi_eval_dataset) +print(result) +``` + +```text + {'mae1': 2.5686439752578734, 'mae2': 2.4921266555786135} +``` + +When the preceding code is used for model evaluation, the output of the evaluation network is transparently transmitted to the `update` function of the evaluation metric. The `update` function receives three inputs: `logits`, `label1`, and `label2`. + +`nn.MAE` allows evaluation metrics to be computed only on two inputs. Therefore, `set_indexes` is used to specify `mae1` to use inputs whose subscripts are 0 and 1, that is, `logits` and `label1`, to compute the evaluation result. Specify `mae2` to use the inputs whose subscripts are 0 and 2, that is, `logits` and `label2`, to compute the evaluation result. + +### Network Inference + + The `Model` does not provide parameters for specifying a customized inference network. In this case, you can directly run the feedforward network to obtain the inference result. + +```python +for d in multi_eval_dataset.create_dict_iterator(): + data = d["data"] + break + +output = net(data) +print(output) +``` + +```text + [[-21.598358 ] + [ -1.0123782] + [ 10.457726 ] + [ 12.409237 ] + [ 19.666183 ] + [ -5.846529 ] + [ 9.387393 ] + [ 2.6558673] + [-15.15129 ] + [-14.876989 ] + [ 19.112661 ] + [ 22.647848 ] + [ 4.9035554] + [ 20.119627 ] + [ -8.339532 ] + [ -2.7513359]] +``` diff --git a/tutorials/source_en/advanced/train/train_eval.md b/tutorials/source_en/advanced/train/train_eval.md new file mode 100644 index 0000000000..9345c1e410 --- /dev/null +++ b/tutorials/source_en/advanced/train/train_eval.md @@ -0,0 +1,336 @@ +# Training and Evaluation + + + +The preceding section describes the basic elements used by MindSpore to build a network, such as the basic network unit, loss function, optimizer, and evaluation function of MindSpore. + +The following focuses on how to use these elements to customize training and evaluation networks. + +## Building Training and Evaluation Processes + +To build a training network, you need to build a feedforward network first, and then add the loss function, backward propagation, and optimizer on the basis of the feedforward network. + +### Defining a Dataset + +The following example defines the `get_data` function to generate sample data and corresponding labels, and defines the `create_dataset` function to load customized datasets. + +```python +import mindspore.dataset as ds +import numpy as np + +def get_data(num, w=2.0, b=3.0): + """Generate sample data and corresponding labels.""" + for _ in range(num): + x = np.random.uniform(-10.0, 10.0) + noise = np.random.normal(0, 1) + y = x * w + b + noise + yield np.array([x]).astype(np.float32), np.array([y]).astype(np.float32) + +def create_dataset(num_data, batch_size=16): + """Generate a dataset.""" + dataset = ds.GeneratorDataset(list(get_data(num_data)), column_names=['data', 'label']) + dataset = dataset.batch(batch_size) + return dataset +``` + +### Building a Feedforward Network + +Use `nn.Cell` to build a feedforward network. The following example defines a simple linear regression network `LinearNet`: + +```python +import numpy as np +import mindspore.nn as nn +from mindspore.common.initializer import Normal + +class LinearNet(nn.Cell): + def __init__(self): + super().__init__() + self.fc = nn.Dense(1, 1, Normal(0.02), Normal(0.02)) + + def construct(self, x): + return self.fc(x) +``` + +### Building a Training Process + +The `nn` module of MindSpore provides the training network encapsulation function `TrainOneStepCell` to encapsulate the network and optimizer. The parameters are as follows: + +- `network`: specifies a training network. Only the single-output network is supported. +- `optimizer`: updates network parameters. +- `sens`: specifies the backward propagation input which is a scaling coefficient. The default value is 1.0. + +The following example uses `nn.TrainOneStepCell` to encapsulate the defined linear regression network into a training network, perform training, and print the loss value. + +In the sample code, `set_train` is used to specify whether the model is in training mode through the `mode` parameter. The default value of the `mode` parameter is True, indicating that the model is in training mode by default. If the value of `mode` is False, the model is in evaluation or inference mode. + +```python +# Generate a training dataset. +train_dataset = create_dataset(num_data=160, batch_size=16) + +net = LinearNet() +loss = nn.MSELoss() + +# Connect the feedforward network and loss function. +net_with_loss = nn.WithLossCell(net, loss) +opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) + +# Define the training network and encapsulate the network and optimizer. +train_net = nn.TrainOneStepCell(net_with_loss, opt) +# Set the network to the training mode. +train_net.set_train() + +# Training step process. +step = 0 +epochs = 2 +steps = train_dataset.get_dataset_size() + +for epoch in range(epochs): + for d in train_dataset.create_dict_iterator(): + result = train_net(d["data"], d["label"]) + print(f"Epoch: [{epoch} / {epochs}], " + f"step: [{step} / {steps}], " + f"loss: {result}") + step = step + 1 +``` + +```text + Epoch: [0 / 2], step: [0 / 10], loss: 139.95065 + Epoch: [0 / 2], step: [1 / 10], loss: 77.1288 + Epoch: [0 / 2], step: [2 / 10], loss: 23.511435 + Epoch: [0 / 2], step: [3 / 10], loss: 15.275428 + Epoch: [0 / 2], step: [4 / 10], loss: 80.57905 + Epoch: [0 / 2], step: [5 / 10], loss: 86.396 + Epoch: [0 / 2], step: [6 / 10], loss: 78.92796 + Epoch: [0 / 2], step: [7 / 10], loss: 16.025606 + Epoch: [0 / 2], step: [8 / 10], loss: 2.996492 + Epoch: [0 / 2], step: [9 / 10], loss: 9.382026 + Epoch: [1 / 2], step: [10 / 10], loss: 46.85878 + Epoch: [1 / 2], step: [11 / 10], loss: 78.591515 + Epoch: [1 / 2], step: [12 / 10], loss: 39.523586 + Epoch: [1 / 2], step: [13 / 10], loss: 3.0048246 + Epoch: [1 / 2], step: [14 / 10], loss: 7.835808 + Epoch: [1 / 2], step: [15 / 10], loss: 27.37307 + Epoch: [1 / 2], step: [16 / 10], loss: 34.076313 + Epoch: [1 / 2], step: [17 / 10], loss: 54.53374 + Epoch: [1 / 2], step: [18 / 10], loss: 19.80341 + Epoch: [1 / 2], step: [19 / 10], loss: 1.8542566 +``` + +### Building an Evaluation Process + +The `nn` module of MindSpore provides the evaluation network encapsulation function `WithEvalCell` to evaluate the model training effect on the validation set. The parameters are as follows: + +- `network`: specifies a feedforward network. +- `loss_fn`: specifies a loss function. +- `add_cast_fp32`: determines whether to adjust the data type to float32. + +`nn.WithEvalCell` accepts only two inputs: `data` and its corresponding `label`. Use the previously defined feedforward network and loss function to build an evaluation network. The following is an example: + +```python +eval_dataset = create_dataset(num_data=160, batch_size=16) + +# Build an evaluation network. +eval_net = nn.WithEvalCell(net, loss) +eval_net.set_train(False) +loss = nn.Loss() +mae = nn.MAE() + +mae.clear() +loss.clear() + +# Evaluation process. +for data in eval_dataset.create_dict_iterator(): + outputs = eval_net(data["data"], data["label"]) + mae.update(outputs[1], outputs[2]) + loss.update(outputs[0]) + +# Evaluation result. +mae_result = mae.eval() +loss_result = loss.eval() + +print("mae: ", mae_result) +print("loss: ", loss_result) +``` + +```text + mae: 2.9597126245498657 + loss: 11.539738941192628 +``` + +## Customized Training and Evaluation Networks + +### Customized Training Network + +[Customized Loss Functions](https://www.mindspore.cn/tutorials/en/master/advanced/network/loss.html#customized-loss-functions) describes how to use `nn.WithLossCell` to connect the feedforward network to the loss function. This section describes how to customize the training network. + +The following example defines the `CustomTrainOneStepCell` function to encapsulate the network and optimizer. + +```python +import mindspore.ops as ops + +class CustomTrainOneStepCell(nn.Cell): + """Customize a training network."" + + def __init__(self, network, optimizer): + """There are two input parameters: training network and optimizer.""" + super(CustomTrainOneStepCell, self).__init__(auto_prefix=False) + self.network = network #Define the feedforward network. + self.network.set_grad() # Build a backward network. + self.optimizer = optimizer #Define the optimizer. + self.weights = self.optimizer.parameters # Parameters to be updated. + self.grad = ops.GradOperation(get_by_list=True) #Obtain the gradient by backward propagation. + + def construct(self, *inputs): + loss = self.network(*inputs) # Compute the loss function value of the current input. + grads = self.grad(self.network, self.weights)(*inputs) # Perform backward propagation and compute the gradient. + self.optimizer(grads) # Use the optimizer to update weight parameters. + return loss + +net1 = LinearNet() # Define the feedforward network. +loss = nn.MSELoss() # Loss function + + +# Connect the feedforward network and loss function. +net_with_loss = nn.WithLossCell(net1, loss) +opt = nn.Momentum(net1.trainable_params(), learning_rate=0.005, momentum=0.9) + +# Define the training network and encapsulate the network and optimizer. +train_net = CustomTrainOneStepCell(net_with_loss, opt) +# Set the network to the training mode. +train_net.set_train() + +# Training step process. +step = 0 +epochs = 2 +steps = train_dataset.get_dataset_size() +for epoch in range(epochs): + for d in train_dataset.create_dict_iterator(): + result = train_net(d["data"], d["label"]) + print(f"Epoch: [{epoch} / {epochs}], " + f"step: [{step} / {steps}], " + f"loss: {result}") + step = step + 1 +``` + +```text + Epoch: [0 / 2], step: [0 / 10], loss: 70.774574 + Epoch: [0 / 2], step: [1 / 10], loss: 71.33737 + Epoch: [0 / 2], step: [2 / 10], loss: 63.126896 + Epoch: [0 / 2], step: [3 / 10], loss: 8.946123 + Epoch: [0 / 2], step: [4 / 10], loss: 32.131054 + Epoch: [0 / 2], step: [5 / 10], loss: 38.90644 + Epoch: [0 / 2], step: [6 / 10], loss: 126.410255 + Epoch: [0 / 2], step: [7 / 10], loss: 41.496185 + Epoch: [0 / 2], step: [8 / 10], loss: 5.7309575 + Epoch: [0 / 2], step: [9 / 10], loss: 16.104172 + Epoch: [1 / 2], step: [10 / 10], loss: 26.39038 + Epoch: [1 / 2], step: [11 / 10], loss: 52.73621 + Epoch: [1 / 2], step: [12 / 10], loss: 38.053413 + Epoch: [1 / 2], step: [13 / 10], loss: 4.555399 + Epoch: [1 / 2], step: [14 / 10], loss: 1.8704597 + Epoch: [1 / 2], step: [15 / 10], loss: 11.614007 + Epoch: [1 / 2], step: [16 / 10], loss: 25.868422 + Epoch: [1 / 2], step: [17 / 10], loss: 26.153322 + Epoch: [1 / 2], step: [18 / 10], loss: 9.847598 + Epoch: [1 / 2], step: [19 / 10], loss: 2.0711172 +``` + +### Customized Evaluation Network + +`nn.WithEvalCell` has only two inputs: `data` and `label`. Therefore, it is not applicable to scenarios with multiple data or labels. In this case, if you want to build an evaluation network, you need to customize the evaluation network. + +If the loss function is not required as an evaluation metric, you do not need to define `loss_fn`. When multiple data or labels are input, you can refer to the following example to define the evaluation network. + +```python +class CustomWithEvalCell(nn.Cell): + + def __init__(self, network): + super(CustomWithEvalCell, self).__init__(auto_prefix=False) + self.network = network + + def construct(self, data, label1, label2): + """There are three pieces of input data: one piece of data and two corresponding labels.""" + outputs = self.network(data) + return outputs, label1, label2 + +custom_eval_net = CustomWithEvalCell(net) +custom_eval_net.set_train(False) +``` + +```text + CustomWithEvalCell< + (network): LinearNet< + (fc): Dense + > + > +``` + +## Sharing Network Weights + +According to the preceding introduction, the forward network, training network, and evaluation network have different logic. Therefore, three networks are built when necessary. If a trained model is used for evaluation and inference, the weight values on the inference and evaluation network must be the same as those on the training network. + +You can use the model saving and loading APIs to save the trained model and then load it to the evaluation and inference networks to ensure that the weight values are the same. Model training is completed on the training platform. However, model saving and loading are necessary for inference on the inference platform. + +During network commissioning or model optimization using inference while training, the model training, evaluation, or inference is usually completed in the same Python script. In this case, the weight sharing mechanism of MindSpore ensures weight consistency between different networks. + +When MindSpore is used to build different network structures, as long as these network structures are encapsulated based on an instance, the all weights in the instance are shared. If the weight in a network changes, the weights in other networks change synchronously. + +In the following example, the weight sharing mechanism is used when the training and evaluation network is defined. + +```python +# Instantiate the feedforward network. +net = LinearNet() +# Set the loss function and connect the feedforward network to the loss function. +loss = nn.MSELoss() +net_with_loss = nn.WithLossCell(net, loss) +# Set the optimizer. +opt = nn.Adam(params=net.trainable_params()) + +# Define a training network. +train_net = nn.TrainOneStepCell(net_with_loss, opt) +train_net.set_train() + +# Build an evaluation network. +eval_net = nn.WithEvalCell(net, loss) +eval_net.set_train(False) +``` + +```text + WithEvalCell< + (_network): LinearNet< + (fc): Dense + > + (_loss_fn): MSELoss<> + > +``` + +Both `train_net` and `eval_net` are encapsulated based on the `net` instance. Therefore, you do not need to load the weight of `train_net` during model evaluation. + +If the feedforward network is redefined when `eval_net` is built, `train_net` and `eval_net` do not share weights. + +```python +# Define a training network. +train_net = nn.TrainOneStepCell(net_with_loss, opt) +train_net.set_train() + +# Instantiate the feedforward network again. +net2 = LinearNet() +# Build an evaluation network. +eval_net = nn.WithEvalCell(net2, loss) +eval_net.set_train(False) +``` + +```text + WithEvalCell< + (_network): LinearNet< + (fc): Dense + > + (_loss_fn): MSELoss<> + > +``` + +In this case, to perform evaluation after model training, you need to load the weight in `train_net` to `eval_net`. When model training, evaluation, and inference are performed in the same script, it is easier to use the weight sharing mechanism. + +In simple scenarios, the `Model` uses `nn.WithLossCell`, `nn.TrainOneStepCell`, and `nn.WithEvalCell` to build a training and evaluation network based on the feedforward network instance, the model itself ensures weight sharing among inference, training, and evaluation networks. + +However, in the scenario where a customized model is used, you need to instantiate the forward network only once. If the feedforward network is instantiated separately when the training network and evaluation network are built, you need to manually load the weight of the training network when using eval to evaluate the model. Otherwise, the initial weight is used for model evaluation. diff --git a/tutorials/source_en/index.rst b/tutorials/source_en/index.rst index fadb0118f1..fdc531f9fc 100644 --- a/tutorials/source_en/index.rst +++ b/tutorials/source_en/index.rst @@ -26,6 +26,8 @@ MindSpore Tutorial :maxdepth: 1 :caption: Advanced + advanced/linear_fitting advanced/dataset advanced/network advanced/train + advanced/pynative_graph \ No newline at end of file -- Gitee