diff --git a/tutorials/experts/source_en/data_engine/auto_augmentation.md b/tutorials/experts/source_en/data_engine/auto_augmentation.md index c39910abf0f27cee3c14dca8ae539c4f319e0df3..47f28e770301d50712c2f539168625371e4c9506 100644 --- a/tutorials/experts/source_en/data_engine/auto_augmentation.md +++ b/tutorials/experts/source_en/data_engine/auto_augmentation.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Data Preparation` - + ## Overview @@ -10,13 +10,13 @@ MindSpore not only allows you to customize data augmentation, but also provides Auto augmentation can be implemented based on probability or callback parameters. -> For a complete example, see [Application of Auto Augmentation](https://www.mindspore.cn/docs/programming_guide/en/master/enable_auto_augmentation.html). +> For a complete example, see [Application of Auto Augmentation](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/enable_auto_augmentation.html). ## Probability Based Auto Augmentation MindSpore provides a series of probability-based auto augmentation APIs. You can randomly select and combine various data augmentation operations to make data augmentation more flexible. -For details about APIs, see [MindSpore API](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.transforms.html). +For details about APIs, see [MindSpore API](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.transforms.html). ### RandomApply diff --git a/tutorials/experts/source_en/data_engine/cache.md b/tutorials/experts/source_en/data_engine/cache.md index f5b8de378219b814897a13cbb8d27c5c5ee5d700..45f73e064f429080859431c79112e8321bb5d584 100644 --- a/tutorials/experts/source_en/data_engine/cache.md +++ b/tutorials/experts/source_en/data_engine/cache.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Data Preparation` - + ## Overview @@ -28,7 +28,7 @@ Currently, the cache service supports only single-node cache. That is, the clien > You are advised to cache image data in `decode` + `resize` + `cache` mode. The data processed by `decode` can be directly cached only in single-node single-device mode. -> For a complete example, see [Application of Single-Node Tensor Cache](https://www.mindspore.cn/docs/programming_guide/en/master/enable_cache.html). +> For a complete example, see [Application of Single-Node Tensor Cache](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/enable_cache.html). ## Basic Cache Usage @@ -186,7 +186,7 @@ Currently, the cache service supports only single-node cache. That is, the clien Note that you need to create a cache instance for each of the two examples according to step 4, and use the created `test_cache` as the `cache` parameter in the dataset loading operator or map operator. - CIFAR-10 dataset is used in the following two examples. Before running the sample, download and store the CIFAR-10 dataset by referring to [Loading Dataset](https://www.mindspore.cn/docs/programming_guide/en/master/dataset_loading.html#cifar-10-100). + CIFAR-10 dataset is used in the following two examples. Before running the sample, download and store the CIFAR-10 dataset by referring to [Loading Dataset](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/dataset_loading.html#cifar-10-100). ```text ./datasets/cifar-10-batches-bin @@ -407,7 +407,7 @@ During the single-node multi-device distributed training, the cache operator all 4. Create and apply a cache instance. - CIFAR-10 dataset is used in the following example. Before running the sample, download and store the CIFAR-10 dataset by referring to [Loading Dataset](https://www.mindspore.cn/docs/programming_guide/en/master/dataset_loading.html#cifar-10-100). The directory structure is as follows: + CIFAR-10 dataset is used in the following example. Before running the sample, download and store the CIFAR-10 dataset by referring to [Loading Dataset](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/dataset_loading.html#cifar-10-100). The directory structure is as follows: ```text ├─cache.sh diff --git a/tutorials/experts/source_en/data_engine/eager.md b/tutorials/experts/source_en/data_engine/eager.md index c8e32f089bfb43ac3fe82c25745511ec686b1536..33a40934ac7c73fe464f7acd4900e3ec4b6dff97 100644 --- a/tutorials/experts/source_en/data_engine/eager.md +++ b/tutorials/experts/source_en/data_engine/eager.md @@ -7,7 +7,7 @@ - + When resource conditions permit, in order to pursue higher performance, data transformations are generally executed in the data pipeline mode. That is, users have to define the `map` operator which helps to execute augmentations in data pipeline. As shown in the figure below, the `map` operator contains 3 transformations: `Resize`, `Crop`, and `HWC2CHW`. When the pipeline starts, the `map` operator will apply these transformations to data in sequence. @@ -23,21 +23,21 @@ In `Eager mode`, the execution of data augmentations will not rely on the `map` MindSpore currently supports executing various data augmentations in `Eager mode`, as shown below. For more details, please refer to the API documentation. -- [vision module](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.vision.html) +- [vision module](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html) - Submodule c_transforms, an image enhancement operator based on OpenCV. - Submodule py_transforms, an image enhancement operator based on Pillow. -- [text module](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.text.html#mindspore-dataset-text-transforms) +- [text module](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html#mindspore-dataset-text-transforms) - Submodule transforms, text processing operators. -- [transforms module](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.transforms.html) +- [transforms module](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.transforms.html) - Submodule c_transforms, a general-purpose data enhancement operator based on C++. - Submodule py_transforms, a general-purpose data augmentation operator based on Python. -Note: In chapters [Image Processing and Enhancement](https://www.mindspore.cn/docs/programming_guide/en/master/augmentation.html), [Text Processing and Enhancement](https://www.mindspore.cn/docs/programming_guide/en/master/tokenizer.html), all data enhancement operators can be executed in Eager mode. +Note: In chapters [Image Processing and Enhancement](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/augmentation.html), [Text Processing and Enhancement](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/tokenizer.html), all data enhancement operators can be executed in Eager mode. ## example @@ -130,7 +130,7 @@ The following shows the processed image. ![eager_mode](./images/eager_mode.png) -Augmentation operators that support to be run in Eager Mode are listed as follows: [mindspore.dataset.transforms](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.transforms.html), [mindspore.dataset.vision](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.vision.html), [mindspore.dataset.text.transforms](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.text.html#mindspore-dataset-text-transforms). +Augmentation operators that support to be run in Eager Mode are listed as follows: [mindspore.dataset.transforms](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.transforms.html), [mindspore.dataset.vision](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html), [mindspore.dataset.text.transforms](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html#mindspore-dataset-text-transforms). ### text diff --git a/tutorials/experts/source_en/data_engine/enable_auto_augmentation.md b/tutorials/experts/source_en/data_engine/enable_auto_augmentation.md index 097f3299fa39caf3dc6acbea85a776e75de4eb79..2a2c5f7334b2e5c14ac5328f643a87c92c4cbab7 100644 --- a/tutorials/experts/source_en/data_engine/enable_auto_augmentation.md +++ b/tutorials/experts/source_en/data_engine/enable_auto_augmentation.md @@ -2,11 +2,11 @@ `Ascend` `GPU` `CPU` `Data Preparation` - + ## Overview -Auto Augmentation [1] finds a suitable image augmentation scheme for a specific dataset by searching through a series of image augmentation sub-policies. The `c_transforms` module of MindSpore provides various C++ operators that are used in Auto Augmentation. Users can also customize functions or operators to implement Auto Augmentation. For more details about the MindSpore operators, see the [API document](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.vision.html). +Auto Augmentation [1] finds a suitable image augmentation scheme for a specific dataset by searching through a series of image augmentation sub-policies. The `c_transforms` module of MindSpore provides various C++ operators that are used in Auto Augmentation. Users can also customize functions or operators to implement Auto Augmentation. For more details about the MindSpore operators, see the [API document](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html). The mapping between MindSpore operators and Auto Augmentation operators is as follows: diff --git a/tutorials/experts/source_en/data_engine/enable_cache.md b/tutorials/experts/source_en/data_engine/enable_cache.md index cb0cac5cac1ee62795df73980d607d9bfd857b01..8b18ecc9009caa79331bc9b2cd2b6db9b8c43b10 100644 --- a/tutorials/experts/source_en/data_engine/enable_cache.md +++ b/tutorials/experts/source_en/data_engine/enable_cache.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Data Preparation` - + ## Overview @@ -63,7 +63,7 @@ This tutorial demonstrates how to use the single-node cache service, and shows s 5. Inserting a Cache Instance - The following uses the CIFAR-10 dataset as an example. Before running the sample, download and store the CIFAR-10 dataset by referring to [Loading Dataset](https://www.mindspore.cn/docs/programming_guide/en/master/dataset_loading.html#cifar-10-100-dataset). The directory structure is as follows: + The following uses the CIFAR-10 dataset as an example. Before running the sample, download and store the CIFAR-10 dataset by referring to [Loading Dataset](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/dataset_loading.html#cifar-10-100-dataset). The directory structure is as follows: ```text ├─my_training_script.py diff --git a/tutorials/experts/source_en/data_engine/optimize_data_processing.ipynb b/tutorials/experts/source_en/data_engine/optimize_data_processing.ipynb index 2b57d4ca36a6408f0566fe5f50b92bf1c2d817a3..28e2ba61bda5c50c492329e8d5a9d70788d19d68 100644 --- a/tutorials/experts/source_en/data_engine/optimize_data_processing.ipynb +++ b/tutorials/experts/source_en/data_engine/optimize_data_processing.ipynb @@ -7,7 +7,7 @@ "\n", "`Ascend` `GPU` `CPU` `Data Preparation`\n", "\n", - "[![Run in ModelArts](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_modelarts_en.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9tYXN0ZXIvcHJvZ3JhbW1pbmdfZ3VpZGUvZW4vbWluZHNwb3JlX29wdGltaXplX2RhdGFfcHJvY2Vzc2luZy5pcHluYg==&imageid=65f636a0-56cf-49df-b941-7d2a07ba8c8c) [![Download Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_notebook_en.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/master/programming_guide/en/mindspore_optimize_data_processing.ipynb) [![View Source On Gitee](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source_en.png)](https://gitee.com/mindspore/docs/blob/master/docs/mindspore/programming_guide/source_en/optimize_data_processing.ipynb)" + "[![Run in ModelArts](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_modelarts_en.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9tYXN0ZXIvcHJvZ3JhbW1pbmdfZ3VpZGUvZW4vbWluZHNwb3JlX29wdGltaXplX2RhdGFfcHJvY2Vzc2luZy5pcHluYg==&imageid=65f636a0-56cf-49df-b941-7d2a07ba8c8c) [![Download Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_notebook_en.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/tutorials/experts/en/mindspore_optimize_data_processing.ipynb) [![View Source On Gitee](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/resource/_static/logo_source_en.png)](https://gitee.com/mindspore/docs/blob/master/tutorials/experts/source_en/optimize_data_processing.ipynb)" ], "metadata": {} }, @@ -23,7 +23,7 @@ { "cell_type": "markdown", "source": [ - "![pipeline](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/programming_guide/source_en/images/pipeline.png)" + "![pipeline](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/experts/source_en/images/pipeline.png)" ], "metadata": {} }, @@ -201,7 +201,7 @@ { "cell_type": "markdown", "source": [ - "![data-loading-performance-scheme](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/programming_guide/source_en/images/data_loading_performance_scheme.png)" + "![data-loading-performance-scheme](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/experts/source_en/images/data_loading_performance_scheme.png)" ], "metadata": {} }, @@ -210,13 +210,13 @@ "source": [ "Suggestions on data loading performance optimization are as follows:\n", "\n", - "- Built-in loading operators are preferred for supported dataset formats. For details, see [Built-in Loading Operators](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.html), if the performance cannot meet the requirements, use the multi-thread concurrency solution. For details, see [Multi-thread Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-thread-optimization-solution).\n", - "- For a dataset format that is not supported, convert the format to the mindspore data format and then use the `MindDataset` class to load the dataset (Please refer to the [API](https://www.mindspore.cn/docs/api/en/master/api_python/dataset/mindspore.dataset.MindDataset.html) for detailed use). Please refer to [Converting Dataset to MindRecord](https://www.mindspore.cn/docs/programming_guide/en/master/convert_dataset.html), if the performance cannot meet the requirements, use the multi-thread concurrency solution, for details, see [Multi-thread Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-thread-optimization-solution).\n", - "- For dataset formats that are not supported, the user-defined `GeneratorDataset` class is preferred for implementing fast algorithm verification (Please refer to the [API](https://www.mindspore.cn/docs/api/en/master/api_python/dataset/mindspore.dataset.GeneratorDataset.html) for detailed use), if the performance cannot meet the requirements, the multi-process concurrency solution can be used. For details, see [Multi-process Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-process-optimization-solution).\n", + "- Built-in loading operators are preferred for supported dataset formats. For details, see [Built-in Loading Operators](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.html), if the performance cannot meet the requirements, use the multi-thread concurrency solution. For details, see [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-thread-optimization-solution).\n", + "- For a dataset format that is not supported, convert the format to the mindspore data format and then use the `MindDataset` class to load the dataset (Please refer to the [API](https://www.mindspore.cn/docs/en/master/api_python/dataset/mindspore.dataset.MindDataset.html) for detailed use). Please refer to [Converting Dataset to MindRecord](https://www.mindspore.cn//tutorials/experts/en/master/data_engine/convert_dataset.html), if the performance cannot meet the requirements, use the multi-thread concurrency solution, for details, see [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-thread-optimization-solution).\n", + "- For dataset formats that are not supported, the user-defined `GeneratorDataset` class is preferred for implementing fast algorithm verification (Please refer to the [API](https://www.mindspore.cn/docs/en/master/api_python/dataset/mindspore.dataset.GeneratorDataset.html) for detailed use), if the performance cannot meet the requirements, the multi-process concurrency solution can be used. For details, see [Multi-process Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-process-optimization-solution).\n", "\n", "### Code Example\n", "\n", - "Based on the preceding suggestions of data loading performance optimization, the `Cifar10Dataset` class of built-in loading operators (Please refer to the [API](https://www.mindspore.cn/docs/api/en/master/api_python/dataset/mindspore.dataset.Cifar10Dataset.html) for detailed use), the `MindDataset` class after data conversion, and the `GeneratorDataset` class are used to load data. The sample code is displayed as follows:\n", + "Based on the preceding suggestions of data loading performance optimization, the `Cifar10Dataset` class of built-in loading operators (Please refer to the [API](https://www.mindspore.cn/docs/en/master/api_python/dataset/mindspore.dataset.Cifar10Dataset.html) for detailed use), the `MindDataset` class after data conversion, and the `GeneratorDataset` class are used to load data. The sample code is displayed as follows:\n", "\n", "1. Use the `Cifar10Dataset` class of built-in operators to load the CIFAR-10 dataset in binary format. The multi-thread optimization solution is used for data loading. Four threads are enabled to concurrently complete the task. Finally, a dictionary iterator is created for the data and a data record is read through the iterator." ], @@ -368,7 +368,7 @@ "source": [ "## Optimizing the Shuffle Performance\n", "\n", - "The shuffle operation is used to shuffle ordered datasets or repeated datasets. MindSpore provides the `shuffle` function for users. A larger value of `buffer_size` indicates a higher shuffling degree, consuming more time and computing resources. This API allows users to shuffle the data at any time during the entire pipeline process.Please refer to [shuffle](https://www.mindspore.cn/docs/programming_guide/en/master/pipeline.html#shuffle). However, because the underlying implementation methods are different, the performance of this method is not as good as that of setting the `shuffle` parameter to directly shuffle data by referring to the [Built-in Loading Operators](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.html).\n", + "The shuffle operation is used to shuffle ordered datasets or repeated datasets. MindSpore provides the `shuffle` function for users. A larger value of `buffer_size` indicates a higher shuffling degree, consuming more time and computing resources. This API allows users to shuffle the data at any time during the entire pipeline process.Please refer to [shuffle](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/pipeline.html#shuffle). However, because the underlying implementation methods are different, the performance of this method is not as good as that of setting the `shuffle` parameter to directly shuffle data by referring to the [Built-in Loading Operators](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.html).\n", "\n", "### Performance Optimization Solution" ], @@ -377,7 +377,7 @@ { "cell_type": "markdown", "source": [ - "![shuffle-performance-scheme](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/programming_guide/source_en/images/shuffle_performance_scheme.png)" + "![shuffle-performance-scheme](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/experts/source_en/images/shuffle_performance_scheme.png)" ], "metadata": {} }, @@ -520,7 +520,7 @@ "- Use the built-in Python operator (`py_transforms` module) to perform data augmentation.\n", "- Users can define Python functions as needed to perform data augmentation.\n", "\n", - "Please refer to [Data Augmentation](https://www.mindspore.cn/docs/programming_guide/en/master/augmentation.html). The performance varies according to the underlying implementation methods.\n", + "Please refer to [Data Augmentation](https://www.mindspore.cn/tutorials/expertsen/master/data_engine/augmentation.html). The performance varies according to the underlying implementation methods.\n", "\n", "| Module | Underlying API | Description |\n", "| :----: | :----: | :----: |\n", @@ -534,7 +534,7 @@ { "cell_type": "markdown", "source": [ - "![data-enhancement-performance-scheme](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/programming_guide/source_en/images/data_enhancement_performance_scheme.png)" + "![data-enhancement-performance-scheme](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/experts/source_en/images/data_enhancement_performance_scheme.png)" ], "metadata": {} }, @@ -543,10 +543,10 @@ "source": [ "Suggestions on data augmentation performance optimization are as follows:\n", "\n", - "- The `c_transforms` module is preferentially used to perform data augmentation for its highest performance. If the performance cannot meet the requirements, refer to [Multi-thread Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-thread-optimization-solution), [Compose Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#compose-optimization-solution), or [Operator Fusion Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#operator-fusion-optimization-solution).\n", - "- If the `py_transforms` module is used to perform data augmentation and the performance still cannot meet the requirements, refer to [Multi-thread Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-thread-optimization-solution), [Multi-process Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-process-optimization-solution), [Compose Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#compose-optimization-solution), or [Operator Fusion Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#operator-fusion-optimization-solution).\n", + "- The `c_transforms` module is preferentially used to perform data augmentation for its highest performance. If the performance cannot meet the requirements, refer to [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-thread-optimization-solution), [Compose Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#compose-optimization-solution), or [Operator Fusion Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#operator-fusion-optimization-solution).\n", + "- If the `py_transforms` module is used to perform data augmentation and the performance still cannot meet the requirements, refer to [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-thread-optimization-solution), [Multi-process Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-process-optimization-solution), [Compose Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#compose-optimization-solution), or [Operator Fusion Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#operator-fusion-optimization-solution).\n", "- The `c_transforms` module maintains buffer management in C++, and the `py_transforms` module maintains buffer management in Python. Because of the performance cost of switching between Python and C++, it is advised not to use different operator types together.\n", - "- If the user-defined Python functions are used to perform data augmentation and the performance still cannot meet the requirements, use the [Multi-thread Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-thread-optimization-solution) or [Multi-process Optimization Solution](https://www.mindspore.cn/docs/programming_guide/en/master/optimize_data_processing.html#multi-process-optimization-solution). If the performance still cannot be improved, in this case, optimize the user-defined Python code.\n", + "- If the user-defined Python functions are used to perform data augmentation and the performance still cannot meet the requirements, use the [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-thread-optimization-solution) or [Multi-process Optimization Solution](https://www.mindspore.cn/tutorials/experts/en/master/data_engine/optimize_data_processing.html#multi-process-optimization-solution). If the performance still cannot be improved, in this case, optimize the user-defined Python code.\n", "\n", "MindSpore also supports users to use the data enhancement methods in the `c_transforms` and `py_transforms` modules at the same time, but due to the different underlying implementations of the two, excessive mixing will increase resource overhead and reduce processing performance. It is recommended that users can use the operators in `c_transforms` or `py_transforms` alone; or use one of them first, and then use the other. Please do not switch frequently between the data enhancement interface of two different implementation modules.\n", "\n", @@ -712,7 +712,7 @@ "- During data augmentation, the `num_parallel_workers` parameter in the `map` function is used to set the number of threads.\n", "- During batch processing, the `num_parallel_workers` parameter in the `batch` function is used to set the number of threads.\n", "\n", - "For details, see [Built-in Loading Operators](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.html).\n", + "For details, see [Built-in Loading Operators](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.html).\n", "\n", "When using MindSpore for standalone or distributed training, the setting of the `num_parallel_workers` parameter should follow the following principles:\n", "\n", @@ -724,8 +724,8 @@ "\n", "During data processing, operators implemented by Python support the multi-process mode. For example:\n", "\n", - "- By default, the `GeneratorDataset` class is in multi-process mode. The `num_parallel_workers` parameter indicates the number of enabled processes. The default value is 1. For details, see [GeneratorDataset](https://www.mindspore.cn/docs/api/en/master/api_python/dataset/mindspore.dataset.GeneratorDataset.html).\n", - "- If the user-defined Python function or the `py_transforms` module is used to perform data augmentation and the `python_multiprocessing` parameter of the `map` function is set to True, the `num_parallel_workers` parameter indicates the number of processes and the default value of the `python_multiprocessing` parameter is False. In this case, the `num_parallel_workers` parameter indicates the number of threads. For details, see [Built-in Loading Operators](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.html).\n", + "- By default, the `GeneratorDataset` class is in multi-process mode. The `num_parallel_workers` parameter indicates the number of enabled processes. The default value is 1. For details, see [GeneratorDataset](https://www.mindspore.cn/docs/en/master/api_python/dataset/mindspore.dataset.GeneratorDataset.html).\n", + "- If the user-defined Python function or the `py_transforms` module is used to perform data augmentation and the `python_multiprocessing` parameter of the `map` function is set to True, the `num_parallel_workers` parameter indicates the number of processes and the default value of the `python_multiprocessing` parameter is False. In this case, the `num_parallel_workers` parameter indicates the number of threads. For details, see [Built-in Loading Operators](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.html).\n", "\n", "### Compose Optimization Solution\n", "\n", @@ -736,7 +736,7 @@ { "cell_type": "markdown", "source": [ - "![compose](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/programming_guide/source_en/images/compose.png)" + "![compose](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/experts/source_en/images/compose.png)" ], "metadata": {} }, @@ -745,9 +745,9 @@ "source": [ "### Operator Fusion Optimization Solution\n", "\n", - "Some fusion operators are provided to aggregate the functions of two or more operators into one operator. For details, see [Augmentation Operators](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.vision.html). Compared with the pipelines of their components, such fusion operators provide better performance. As shown in the figure:\n", + "Some fusion operators are provided to aggregate the functions of two or more operators into one operator. For details, see [Augmentation Operators](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html). Compared with the pipelines of their components, such fusion operators provide better performance. As shown in the figure:\n", "\n", - "![operator-fusion](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/docs/mindspore/programming_guide/source_en/images/operator_fusion.png)\n", + "![operator-fusion](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/experts/source_en/images/operator_fusion.png)\n", "\n", "### Operating System Optimization Solution\n", "\n", diff --git a/tutorials/experts/source_en/debug/auto_tune.md b/tutorials/experts/source_en/debug/auto_tune.md index 4f38c8858bd5ee62271a520ac1aedb2f10467e72..60439c9ea5fb7767846124bb9eff978ab1ae2709 100644 --- a/tutorials/experts/source_en/debug/auto_tune.md +++ b/tutorials/experts/source_en/debug/auto_tune.md @@ -2,7 +2,7 @@ `Ascend` `Model Optimization` -   +   ## Overview diff --git a/tutorials/experts/source_en/debug/custom_debugging_info.md b/tutorials/experts/source_en/debug/custom_debugging_info.md index 5fe5f9770e8d90a5a4047dc92bc8b50d53c0d2ee..4afac0ace0eef89403ee18268eb0df89c2603fa4 100644 --- a/tutorials/experts/source_en/debug/custom_debugging_info.md +++ b/tutorials/experts/source_en/debug/custom_debugging_info.md @@ -2,7 +2,7 @@ `Ascend` `Model Optimization` - + ## Overview @@ -264,15 +264,15 @@ Tensor(shape=[2, 2], dtype=Int32, value= ## Data Dump Introduction -When training the network, if the training result deviates from the expectation, the input and output of the operator can be saved for debugging through the data dump function. For detailed Dump function introduction, please refer to [Dump Mode](https://www.mindspore.cn/docs/programming_guide/en/master/dump_in_graph_mode.html#dump-introduction). +When training the network, if the training result deviates from the expectation, the input and output of the operator can be saved for debugging through the data dump function. For detailed Dump function introduction, please refer to [Dump Mode](https://www.mindspore.cn/tutorials/experts/en/master/debug/dump_in_graph_mode.html#dump-introduction). ### Synchronous Dump -Synchronous Dump function usage reference [Synchronous Dump Step](https://www.mindspore.cn/docs/programming_guide/en/master/dump_in_graph_mode.html#synchronous-dump-step). +Synchronous Dump function usage reference [Synchronous Dump Step](https://www.mindspore.cn/tutorials/experts/en/master/debug/dump_in_graph_mode.html#synchronous-dump-step). ### Asynchronous Dump -Asynchronous Dump function usage reference [Asynchronous Dump Step](https://www.mindspore.cn/docs/programming_guide/en/master/dump_in_graph_mode.html#asynchronous-dump-step)。 +Asynchronous Dump function usage reference [Asynchronous Dump Step](https://www.mindspore.cn/tutorials/experts/en/master/debug/dump_in_graph_mode.html#asynchronous-dump-step)。 ## Running Data Recorder diff --git a/tutorials/experts/source_en/debug/dataset_autotune.md b/tutorials/experts/source_en/debug/dataset_autotune.md index f95dae5b03c0c76ec65039951312c9d3ed48320b..c3fa72cac89ad41c76d429fd5f11d02064a657a6 100644 --- a/tutorials/experts/source_en/debug/dataset_autotune.md +++ b/tutorials/experts/source_en/debug/dataset_autotune.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `Data Preparation` - + ## Overview @@ -59,7 +59,7 @@ print("tuning interval:", ds.config.get_autotune_interval()) ## Constraints - Both Dataset Profiling and Dataset AutoTune cannot be enabled concurrently, otherwise it will lead to unwork of Dataset AutoTune or Profiling. If both of them are enabled at the same time, a warning message will prompt the user to check whether it is a mistake. Please make sure Profiling is disabled when using Dataset AutoTune. -- When enable [Offload for Dataset](https://www.mindspore.cn/docs/programming_guide/en/master/enable_dataset_offload.html) and Dataset AutoTune simultaneously, if any dataset node has been offloaded for hardware acceleration, then the optimized dataset pipeline configuration file will not be written and a warning will be logged, because the dataset pipeline that is actually running is not the predefined one. +- When enable [Offload for Dataset](https://www.mindspore.cn/tutorials/experts/en/master/debug/enable_dataset_offload.html) and Dataset AutoTune simultaneously, if any dataset node has been offloaded for hardware acceleration, then the optimized dataset pipeline configuration file will not be written and a warning will be logged, because the dataset pipeline that is actually running is not the predefined one. - If the Dataset pipeline consists of a node that does not support deserialization (e.g. user-defined Python functions, GeneratorDataset), then any attempt to deserialize the saved optimized dataset pipeline configuration file will report an error. In this case, it is recommended to open the pipeline configuration file and modify the script of dataset pipeline manually. ## Example @@ -218,8 +218,7 @@ This allows the dataset pipeline to be run at an improved speed from the beginni By the way, MindSpore also provides APIs to set the global value of num_parallel_workers and prefetch_size. -Please refer to [mindspore.dataset.config](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.config.html): - -- [mindspore.dataset.config.set_num_parallel_workers](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.config.html#mindspore.dataset.config.set_num_parallel_workers) -- [mindspore.dataset.config.set_prefetch_size](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.dataset.config.html#mindspore.dataset.config.set_prefetch_size) +Please refer to [mindspore.dataset.config](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.config.html): +- [mindspore.dataset.config.set_num_parallel_workers](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.config.html#mindspore.dataset.config.set_num_parallel_workers) +- [mindspore.dataset.config.set_prefetch_size](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.config.html#mindspore.dataset.config.set_prefetch_size) diff --git a/tutorials/experts/source_en/debug/debug_in_pynative_mode.md b/tutorials/experts/source_en/debug/debug_in_pynative_mode.md index 07a123708db73021fa679ef641f9200add136756..cee2fd7035f9a5d387a8b8bea5fe225e6495c7b7 100644 --- a/tutorials/experts/source_en/debug/debug_in_pynative_mode.md +++ b/tutorials/experts/source_en/debug/debug_in_pynative_mode.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Model Running` - + ## Overview @@ -17,7 +17,7 @@ In PyNative mode, single operators, common functions, network inference, and sep > In PyNative mode, operators are executed asynchronously on the device to improve performance. Therefore, when an error occurs during operator execution, the error information may be displayed after the program is executed. Therefore, in PyNative mode, a pynative_synchronize setting is added to control whether operators are executed asynchronously on the device. > -> In the following example, the parameter initialization uses random values, and the output results in specific execution may be different from the results of local execution; if you need to stabilize the output of a fixed value, you can set a fixed random seed. For the setting method, please refer to [mindspore.set_seed()](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore/mindspore.set_seed.html). +> In the following example, the parameter initialization uses random values, and the output results in specific execution may be different from the results of local execution; if you need to stabilize the output of a fixed value, you can set a fixed random seed. For the setting method, please refer to [mindspore.set_seed()](https://www.mindspore.cn/docs/en/master/api_python/mindspore/mindspore.set_seed.html). ## Executing a Single Operator @@ -480,7 +480,7 @@ Output： (Tensor(shape=[], dtype=Float32, value= 4), Tensor(shape=[], dtype=Float32, value= 4)) ``` -For more descriptions of HookBackward operator, please refer to [API document](https://mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.HookBackward.html). +For more descriptions of HookBackward operator, please refer to [API document](https://mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.HookBackward.html). ### The register_forward_pre_hook function for Cell object @@ -588,7 +588,7 @@ forward inputs: (Tensor(shape=[1], dtype=Float32, value= [ 2.00000000e+00]), Ten In order to avoid script failure when switching to graph mode, it is not recommended to call the `register_forward_pre_hook` function or the `remove()` function of the `handle` object in the `construct` function of the Cell object. In the PyNative mode, if the `register_forward_pre_hook` function is called in the `construct` function of the Cell object, a hook function will be added at each run time of Cell object. -More about the `register_forward_pre_hook` interface, please refer to [API Document](https://mindspore.cn/docs/api/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.register_forward_pre_hook). +More about the `register_forward_pre_hook` interface, please refer to [API Document](https://mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.register_forward_pre_hook). ### The register_forward_hook function for Cell object @@ -655,7 +655,7 @@ If user returns the newly created data directly in the hook function instead of In order to avoid script failure when switching to graph mode, it is not recommended to call the `register_forward_hook` function or the `remove()` function of the `handle` object in the `construct` function of the Cell object. In the PyNative mode, if the `register_forward_hook` function is called in the `construct` function of the Cell object, a hook function will be added at each run time of Cell object. -More about the `register_forward_hook` interface, please refer to [API Document](https://mindspore.cn/docs/api/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.register_forward_hook). +More about the `register_forward_hook` interface, please refer to [API Document](https://mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.register_forward_hook). ### The register_backward_hook function for Cell object @@ -793,7 +793,7 @@ The `grad input` is the input gradient of `self.mul` . It is not the input gradi In order to avoid script failure when switching to graph mode, it is not recommended to call the `register_backward_hook` function or the `remove()` function of the `handle` object in the `construct` function of the Cell object. In the PyNative mode, if the `register_backward_hook` function is called in the `construct` function of the Cell object, a hook function will be added at each run time of Cell object. -More about the `register_backward_hook` interface, please refer to [API Document](https://mindspore.cn/docs/api/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.register_backward_hook). +More about the `register_backward_hook` interface, please refer to [API Document](https://mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.register_backward_hook). ## Custom bprop diff --git a/tutorials/experts/source_en/debug/dump_in_graph_mode.md b/tutorials/experts/source_en/debug/dump_in_graph_mode.md index 9b682f892be61034ab9182a6b9c7c4732057f1df..5bf42f7f5a62cf366590c69d728369e7f61d36b7 100644 --- a/tutorials/experts/source_en/debug/dump_in_graph_mode.md +++ b/tutorials/experts/source_en/debug/dump_in_graph_mode.md @@ -2,13 +2,13 @@ `Ascend` `GPU` `CPU` `Model Optimization` - + ## Overview The input and output of the operator can be saved for debugging through the data dump when the training result deviates from the expectation. -- For the dynamic graph mode, MindSpore provides native Python execution capabilities. Users can view and record the corresponding input and output during the running of the network script. For details, see [Use PyNative Mode to Debug](https://www.mindspore.cn/docs/programming_guide/en/master/debug_in_pynative_mode.html). +- For the dynamic graph mode, MindSpore provides native Python execution capabilities. Users can view and record the corresponding input and output during the running of the network script. For details, see [Use PyNative Mode to Debug](https://www.mindspore.cn/tutorials/experts/en/master/debug/debug_in_pynative_mode.html). - For the static graph mode, MindSpore provides the Dump function to save the graph and the input and output data of the operator during model training to a disk file. @@ -25,7 +25,7 @@ The data preparation phase uses synchronous dump or asynchronous dump to generat When preparing data, you can refer to the following best practices: 1. Set the `iteration` parameter to save only the data of the iteration with the problem and the previous iteration. For example, if the problem to be analyzed will appear in the 10th iteration (counting from 1), you can set it as follows: `"iteration": "8 | 9"`. Note that the `iteration` parameter evaluates iterations from 0. Saving the data of the above two iterations can help problem analysis under most scenarios. -2. After the iteration with problems is completed, it is recommended that you use [run_context.request_stop()](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.train.html#mindspore.train.callback.RunContext.request_stop) or other methods to stop the training. +2. After the iteration with problems is completed, it is recommended that you use [run_context.request_stop()](https://www.mindspore.cn/docs/en/master/api_python/mindspore.train.html#mindspore.train.callback.RunContext.request_stop) or other methods to stop the training. #### Data analysis @@ -49,7 +49,7 @@ If MindInsight is not installed, you need to analyze the data through the follow 1. Analysis of static graph operator results. - Through the IR diagram obtained by the Dump function, you can understand the mapping relationship between the script code and the execution operator (for details, see [MindSpore IR Introduction](https://www.mindspore.cn/docs/en/master/design/mindir.html#overview)). Combining the input and output data of the execution operator, it is possible to analyze possible overflow, gradient explosion and disappearance during the training process, and backtrack to the code that may have problems in the script. + Through the IR diagram obtained by the Dump function, you can understand the mapping relationship between the script code and the execution operator (for details, see [MindSpore IR Introduction](https://www.mindspore.cn/tutorials/experts/en/master/design/mindir.html#overview)). Combining the input and output data of the execution operator, it is possible to analyze possible overflow, gradient explosion and disappearance during the training process, and backtrack to the code that may have problems in the script. 2. Analysis of the feature map. @@ -105,7 +105,7 @@ The configuration files required for different modes and the data format of dump - `iteration`: Specify the iterations to dump, type is string. Use "|" to separate the step data of different intervals to be saved. For example, "0 | 5-8 | 100-120" represents dump the data of the 1st, 6th to 9th, and 101st to 121st steps. If iteration set to "all", data of every iteration will be dumped. - `saved_data`: Specify what data is to be dumped, type is string. Use "tensor" to dump tensor data, use "statistic" to dump tensor statistics, use "full" to dump both tensor data and statistics. Default setting is "tensor". Synchronous statistics dump is only supported on GPU, using "statistic" or "full" on CPU or Ascend will result in exception. - `input_output`: 0: dump input and output of kernel, 1:dump input of kernel, 2:dump output of kernel. This configuration parameter only supports Ascend and CPU, and GPU can only dump the output of operator. - - `kernels`: List of operator names. Turn on the IR save switch `context.set_context(save_graphs=True)` and execute the network to obtain the operator name from the generated `trace_code_graph_{graph_id}`IR file. For details, please refer to [Saving IR](https://www.mindspore.cn/docs/en/master/design/mindir.html#saving-ir). + - `kernels`: List of operator names. Turn on the IR save switch `context.set_context(save_graphs=True)` and execute the network to obtain the operator name from the generated `trace_code_graph_{graph_id}`IR file. For details, please refer to [Saving IR](https://www.mindspore.cn/tutorials/experts/en/master/design/mindir.html#saving-ir). - `support_device`: Supported devices, default setting is `[0,1,2,3,4,5,6,7]`. You can specify specific device ids to dump specific device data. This configuration parameter is invalid on the CPU, because there is no concept of device on the CPU, but it is still need to reserve this parameter in the json file. - `enable`: When set to true, enable Synchronous Dump. When set to false, asynchronous dump will be used on Ascend and synchronous dump will still be used on GPU. - `trans_flag`: Enable trans flag. Transform the device data format into NCHW. If it is `True`, the data will be saved in the 4D format (NCHW) format on the Host side; if it is `False`, the data format on the Device side will be retained. This configuration parameter is invalid on the CPU, because there is no format conversion on the CPU, but it is still need to reserve this parameter in the json file. @@ -401,13 +401,13 @@ Large networks (such as Bert Large) will cause memory overflow when using synchr } ``` - - `dump_mode`: 0: dump all kernels data in graph, 1: dump kernels data in kernels list, 2: dump the kernels data specified by `set_dump` in the scripts, see [mindspore.dump](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore/mindspore.set_dump.html) for the usage of `set_dump`. + - `dump_mode`: 0: dump all kernels data in graph, 1: dump kernels data in kernels list, 2: dump the kernels data specified by `set_dump` in the scripts, see [mindspore.dump](https://www.mindspore.cn/docs/en/master/api_python/mindspore/mindspore.set_dump.html) for the usage of `set_dump`. - `path`: The absolute path to save dump data. It is not a mandatory option and can be left unset or set to an empty string. - `net_name`: The net name eg:ResNet50. - `iteration`: Specify the iterations to dump, type is string. Use "|" to separate the step data of different intervals to be saved. For example, "0 | 5-8 | 100-120" represents dump the data of the 1st, 6th to 9th, and 101st to 121st steps. If iteration set to "all", data of every iteration will be dumped. - `saved_data`: Specify what data is to be dumped, type is string. Use "tensor" to dump tensor data, use "statistic" to dump tensor statistics, use "full" to dump both tensor data and statistics. Default setting is "tensor". Asynchronous statistics dump is only supported when `file_format` is set to `npy`, using "statistic" or "full" when `file_format` is set to `bin` will result in exception. - `input_output`: When set to 0, it means to Dump the operator's input and output; setting it to 1 means to Dump the operator's input; setting it to 2 means to Dump the output of the operator. - - `kernels`: List of operator names. Turn on the IR save switch `context.set_context(save_graphs=True)` and execute the network to obtain the operator name from the generated `trace_code_graph_{graph_id}`IR file. `kernels` only supports TBE operator, AiCPU operator and communication operator. The data of communication operation input operator will be dumped if `kernels` is set to the name of communication operator. For details, please refer to [Saving IR](https://www.mindspore.cn/docs/en/master/design/mindir.html#saving-ir). + - `kernels`: List of operator names. Turn on the IR save switch `context.set_context(save_graphs=True)` and execute the network to obtain the operator name from the generated `trace_code_graph_{graph_id}`IR file. `kernels` only supports TBE operator, AiCPU operator and communication operator. The data of communication operation input operator will be dumped if `kernels` is set to the name of communication operator. For details, please refer to [Saving IR](https://www.mindspore.cn/tutorials/experts/en/master/design/mindir.html#saving-ir). - `support_device`: Supported devices, default setting is `[0,1,2,3,4,5,6,7]`. You can specify specific device ids to dump specific device data. - `enable`: Enable Asynchronous Dump. If synchronous dump and asynchronous dump are enabled at the same time, only synchronous dump will take effect. - `op_debug_mode`: Reserved field, set to 0. @@ -443,7 +443,7 @@ Large networks (such as Bert Large) will cause memory overflow when using synchr ### Asynchronous Dump Data Object Directory -If set `file_format` to `npy`, see [Synchronous Dump Data Object Directory](https://www.mindspore.cn/docs/programming_guide/en/master/dump_in_graph_mode.html#synchronous-dump-data-object-directory) for the dump data object directory. +If set `file_format` to `npy`, see [Synchronous Dump Data Object Directory](https://www.mindspore.cn/tutorials/experts/en/master/debug/dump_in_graph_mode.html#synchronous-dump-data-object-directory) for the dump data object directory. The data objects saved by asynchronous Dump include the final execution graph (`ms_output_trace_code_graph_{graph_id}.ir` file) and the input and output data of the operators in the graph. The directory structure is as follows: @@ -488,7 +488,7 @@ If the length of the tensor file name defined according to the naming rules exce ### Introduction to Asynchronous Dump Data File -If set `file_format` to `npy`, see [Introduction to Synchronous Dump Data File](https://www.mindspore.cn/docs/programming_guide/en/master/dump_in_graph_mode.html#introduction-to-synchronous-dump-data-file) for the introduction to dump data file. +If set `file_format` to `npy`, see [Introduction to Synchronous Dump Data File](https://www.mindspore.cn/tutorials/experts/en/master/debug/dump_in_graph_mode.html#introduction-to-synchronous-dump-data-file) for the introduction to dump data file. If not configured `file_format` or set `file_format` to `bin`, after the training is started, the original data file generated by asynchronous Dump is in protobuf format. It needs to be parsed using the data analysis tool that comes with the HiSilicon Run package. For details, please refer to [How to view dump data files](https://support.huawei.com/enterprise/en/doc/EDOC1100206689/640e796d/how-do-i-view-a-dump-file). diff --git a/tutorials/experts/source_en/debug/incremental_compilation.md b/tutorials/experts/source_en/debug/incremental_compilation.md index 5ac5e6c836788b6de50001ddc27afa09d847191c..e373b55c522d3110014e14d0ed784e76b9ec695a 100644 --- a/tutorials/experts/source_en/debug/incremental_compilation.md +++ b/tutorials/experts/source_en/debug/incremental_compilation.md @@ -2,7 +2,7 @@ `Ascend` `Model Optimization` - + ## Overview diff --git a/tutorials/experts/source_en/debug/read_ir_files.md b/tutorials/experts/source_en/debug/read_ir_files.md index f1aa3a4fc0ca20ee857d7ec338242d679360ed17..9ee840fae6c4baf0e33bd5467c9c96f772b508ce 100644 --- a/tutorials/experts/source_en/debug/read_ir_files.md +++ b/tutorials/experts/source_en/debug/read_ir_files.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Model Optimization` - + ## Overview @@ -172,7 +172,7 @@ Line 5 to 6 are the input list, which is in the format of `%para[No.]_[name] : < Line 8 tells us the number of subgraph parsed by the network. There are 3 graphs in this IR. Line 42 is the entry graph `1_construct_wrapper.21`. Line 32 is graph `3_func.23`, parsed from the `func(x, y)` in the source script. Line 12 is graph `2_construct.22`, parsed from the function `construct`. Taking graph `2_construct.22` as an example, Line 10 to 28 indicate the graph structure, which contains several nodes, namely, `CNode`. In this example, there are `Sub`, `Add`, `Mul`. They are defined in the function `__init__`. Line 19 calls a graph by `call @3_func.23`. It indicates calling the graph `func(x, y)` to execute a division operation. -The ]`CNode`](https://www.mindspore.cn/docs/en/master/design/mindir.html#syntax) information format is as follows: including the node name, attribute, input node, the specs of the inputs and outputs, and source code parsing call stack. The ANF graph is a unidirectional acyclic graph. So, the connection between nodes is displayed only based on the input relationship. The corresponding source code reflects the relationship between the `CNode` and the script source code. For example, line 15 is parsed from `a = self.sub(x, 1)`. +The ]`CNode`]() information format is as follows: including the node name, attribute, input node, the specs of the inputs and outputs, and source code parsing call stack. The ANF graph is a unidirectional acyclic graph. So, the connection between nodes is displayed only based on the input relationship. The corresponding source code reflects the relationship between the `CNode` and the script source code. For example, line 15 is parsed from `a = self.sub(x, 1)`. ```text %[No.]([debug_name]) = [op_name]([arg], ...) primitive_attrs: {[key]: [value], ...} @@ -262,7 +262,7 @@ Line 23 to 32 indicates the graph structure, which contains several nodes, namel Line 34 to 39 shows the execution order of the `CNode` from graph `2_construct.22`, corresponding to the order of code execution. The information format is: `No.: belonging graph:node name{[0]: the first input, [1]: the second input, ...}`. For `CNode`, the first input indicates how to compute for this `CNode`. Line 28 indicates the number of graphs. Here is 3. -The [CNode](https://www.mindspore.cn/docs/en/master/design/mindir.html#syntax) information format is as follows: including the node name, attribute, input node, output information, format and the corresponding source code. +The [CNode](https://www.mindspore.cn/tutorials/experts/en/master/design/mindir.html#syntax) information format is as follows: including the node name, attribute, input node, output information, format and the corresponding source code. ```text %[No,] : [outputs' Spec] = [op_name]{[prim_type]}[attr0, attr1, ...](arg0, arg1, ...) #(inputs' Spec)#[scope] diff --git a/tutorials/experts/source_en/model_infer/inference.md b/tutorials/experts/source_en/model_infer/inference.md index caee10b2a51c182883fa25c21318b7c6c19c583b..b5d8123db3ecdd81f60f36bc79960926861405ad 100644 --- a/tutorials/experts/source_en/model_infer/inference.md +++ b/tutorials/experts/source_en/model_infer/inference.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Inference Application` - + MindSpore can execute inference tasks on different hardware platforms based on trained models. @@ -23,7 +23,7 @@ Basic concepts and application scenarios of these formats are as follows: - It eliminates model differences between different backends and is generally used to perform inference tasks across hardware platforms. - Open Neural Network Exchange (ONNX) - ONNX is an open format built to represent machine learning models. - - It is generally used to transfer models between different frameworks or used on the inference engine ([TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/index.html)). + - It is generally used to transfer models between different frameworks or used on the inference engine ([TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/python_api/index.html)). - At present, mindspire only supports the export of ONNX model, and does not support loading onnx model for inference. Currently, the models supported for export are resnet50, yolov3_ darknet53, YOLOv4 and BERT. These models can be used on [ONNX Runtime](https://onnxruntime.ai/). - Ascend Intermediate Representation (AIR) - AIR is an open file format defined by Huawei for machine learning. @@ -35,11 +35,11 @@ Inference can be classified into the following two modes based on the applicatio 1. Local inference - Load a checkpoint file generated during network training and call the `model.predict` API for inference and validation. For details, see [Online Inference with Checkpoint](https://www.mindspore.cn/docs/programming_guide/en/master/online_inference.html). + Load a checkpoint file generated during network training and call the `model.predict` API for inference and validation. For details, see [Online Inference with Checkpoint](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/online_inference.html). 2. Cross-platform inference - Use a network definition and a checkpoint file, call the `export` API to export a model file, and perform inference on different platforms. Currently, MindIR, ONNX, and AIR (on only Ascend AI Processors) models can be exported. For details, see [Saving Models](https://www.mindspore.cn/docs/programming_guide/en/master/save_model.html). + Use a network definition and a checkpoint file, call the `export` API to export a model file, and perform inference on different platforms. Currently, MindIR, ONNX, and AIR (on only Ascend AI Processors) models can be exported. For details, see [Saving Models](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/save_model.html). ## Introduction to MindIR @@ -56,4 +56,4 @@ MindSpore defines logical network structures and operator attributes through a u 2. Application Scenarios - Use a network definition and a checkpoint file to export a MindIR model file, and then execute inference based on different requirements, for example, [Inference Using the MindIR Model on Ascend 310 AI Processors](https://www.mindspore.cn/docs/programming_guide/en/master/multi_platform_inference_ascend_310_mindir.html), [MindSpore Serving-based Inference Service Deployment](https://www.mindspore.cn/serving/docs/en/master/serving_example.html), and [Inference on Devices](https://www.mindspore.cn/lite/docs/en/master/index.html). + Use a network definition and a checkpoint file to export a MindIR model file, and then execute inference based on different requirements, for example, [Inference Using the MindIR Model on Ascend 310 AI Processors](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/multi_platform_inference_ascend_310_mindir.html), [MindSpore Serving-based Inference Service Deployment](https://www.mindspore.cn/serving/docs/en/master/serving_example.html), and [Inference on Devices](https://www.mindspore.cn/lite/docs/en/master/index.html). diff --git a/tutorials/experts/source_en/model_infer/inference_ascend_310_air.md b/tutorials/experts/source_en/model_infer/inference_ascend_310_air.md index 9343bedbee4f9856c1a19d71d0ccffa168e0cd3e..4e33d5c6ebe1e4b6f96d5cc1f74b4d4f9c8fd4e3 100644 --- a/tutorials/experts/source_en/model_infer/inference_ascend_310_air.md +++ b/tutorials/experts/source_en/model_infer/inference_ascend_310_air.md @@ -2,7 +2,7 @@ `Ascend` `Inference Application` - + ## Overview @@ -102,7 +102,7 @@ Create a directory to store the inference code project, for example, `/home/HwHi ## Exporting the AIR Model -Train the target network on the Ascend 910 AI Processor, save it as a checkpoint file, and export the model file in AIR format through the network and checkpoint file. For details about the export process, see [Export AIR Model](https://www.mindspore.cn/docs/programming_guide/en/master/save_model.html#export-air-model). +Train the target network on the Ascend 910 AI Processor, save it as a checkpoint file, and export the model file in AIR format through the network and checkpoint file. For details about the export process, see [Export AIR Model](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/save_model.html#export-air-model). > The [resnet50_export.air](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com:443/sample_resources/acl_resnet50_sample/resnet50_export.air) is a sample AIR file exported using the ResNet-50 model. diff --git a/tutorials/experts/source_en/model_infer/inference_ascend_310_mindir.md b/tutorials/experts/source_en/model_infer/inference_ascend_310_mindir.md index 6419e628d5b9e5b41d40ee4d1b63bca623ef6d02..41dffe10ca583c8de9b703d9576116318026d3d8 100644 --- a/tutorials/experts/source_en/model_infer/inference_ascend_310_mindir.md +++ b/tutorials/experts/source_en/model_infer/inference_ascend_310_mindir.md @@ -2,7 +2,7 @@ `Ascend` `Inference Application` - + ## Overview @@ -22,7 +22,7 @@ Refer to [Installation Guide](https://www.mindspore.cn/install/en) to install As ## Exporting the MindIR Model -Train the target network on the CPU/GPU/Ascend 910 AI Processor, save it as a checkpoint file, and export the model file in MindIR format through the network and checkpoint file. For details about the export process, see [Export MindIR Model](https://www.mindspore.cn/docs/programming_guide/en/master/save_model.html#export-mindir-model). +Train the target network on the CPU/GPU/Ascend 910 AI Processor, save it as a checkpoint file, and export the model file in MindIR format through the network and checkpoint file. For details about the export process, see [Export MindIR Model](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/save_model.html#export-mindir-model). > The [resnet50_imagenet.mindir](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/sample_resources/ascend310_resnet50_preprocess_sample/resnet50_imagenet.mindir) is a sample MindIR file exported using the ResNet-50 model, whose BatchSize is 1. We also provide a ResNet-50 MindIR with data preprocess [resnet50_imagenet_preprocess.mindir](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/sample_resources/ascend310_resnet50_preprocess_sample/resnet50_imagenet_preprocess.mindir). diff --git a/tutorials/experts/source_en/model_infer/inference_ascend_910.md b/tutorials/experts/source_en/model_infer/inference_ascend_910.md index 1b8bce4b66ab3cbcc3fbab4210751993e98f8c74..45d324ceda4ebea51223d86408ef8eaf3d2bfe38 100644 --- a/tutorials/experts/source_en/model_infer/inference_ascend_910.md +++ b/tutorials/experts/source_en/model_infer/inference_ascend_910.md @@ -2,7 +2,7 @@ `Ascend` `Inference Application` - + ## Overview diff --git a/tutorials/experts/source_en/model_infer/inference_cpu.md b/tutorials/experts/source_en/model_infer/inference_cpu.md index 0d33b40d5bbd5d05ae507bdc86529aaf843ec02d..1853ff30018372a98e00b111a705a0db1c82ddc9 100644 --- a/tutorials/experts/source_en/model_infer/inference_cpu.md +++ b/tutorials/experts/source_en/model_infer/inference_cpu.md @@ -2,12 +2,12 @@ `CPU` `Inference Application` - + ## Inference Using an ONNX File Similar to the inference on a GPU, the following steps are required: -1. Generate a model in ONNX format on the training platform. For details, see [Export ONNX Model](https://www.mindspore.cn/docs/programming_guide/en/master/save_model.html#export-onnx-model). +1. Generate a model in ONNX format on the training platform. For details, see [Export ONNX Model](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/save_model.html#export-onnx-model). 2. Perform inference on a CPU by referring to the runtime or SDK document. For details about how to use the ONNX Runtime, see the [ONNX Runtime document](https://github.com/microsoft/onnxruntime). diff --git a/tutorials/experts/source_en/model_infer/inference_gpu.md b/tutorials/experts/source_en/model_infer/inference_gpu.md index 61cb66cb15739d2f163e16fdf5439c8363bfe490..5fbf62f06b3c663a679116eed311d653f4c12126 100644 --- a/tutorials/experts/source_en/model_infer/inference_gpu.md +++ b/tutorials/experts/source_en/model_infer/inference_gpu.md @@ -2,7 +2,7 @@ `GPU` `Inference Application` - + ## Use C++ Interface to Load a MindIR File for Inferencing @@ -181,6 +181,6 @@ It is recommended that export the MindIR model with fp32 precision mode before d ## Inference Using an ONNX File -1. Generate a model in ONNX format on the training platform. For details, see [Export ONNX Model](https://www.mindspore.cn/docs/programming_guide/en/master/save_model.html#export-onnx-model). +1. Generate a model in ONNX format on the training platform. For details, see [Export ONNX Model](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/save_model.html#export-onnx-model). 2. Perform inference on a GPU by referring to the runtime or SDK document. For example, use TensorRT to perform inference on the NVIDIA GPU. For details, see [TensorRT backend for ONNX](https://github.com/onnx/onnx-tensorrt). diff --git a/tutorials/experts/source_en/model_infer/online_inference.md b/tutorials/experts/source_en/model_infer/online_inference.md index 3f28852c7d4a943bc40cfbbc67ee6bcbf69814fd..5d1afcc174ba872d68fa81cf375d7f5551c518fa 100644 --- a/tutorials/experts/source_en/model_infer/online_inference.md +++ b/tutorials/experts/source_en/model_infer/online_inference.md @@ -2,7 +2,7 @@ `Ascend` `Inference Application` - + ## Use the `model.eval` interface for model validation @@ -25,7 +25,7 @@ print("============== {} ==============".format(acc)) ``` In the preceding information: -`model.eval` is an API for model validation. For details about the API, see . +`model.eval` is an API for model validation. For details about the API, see . > Inference sample code: . ### Remote Storage @@ -56,4 +56,4 @@ In the preceding information: ``` In the preceding information: - `model.predict` is an API for inference. For details about the API, see . + `model.predict` is an API for inference. For details about the API, see . diff --git a/tutorials/experts/source_en/model_infer/post_training_quantization.md b/tutorials/experts/source_en/model_infer/post_training_quantization.md index 257cd04c944fdbb46a6fb110329f583049d2728b..924bfca7231c996517c8948aa7aebc5d0d69cb9d 100644 --- a/tutorials/experts/source_en/model_infer/post_training_quantization.md +++ b/tutorials/experts/source_en/model_infer/post_training_quantization.md @@ -4,7 +4,7 @@ Translator: [unseeme](https://gitee.com/unseenme) `Device` `Ascend` `Inference Application` - + ## Concept @@ -27,4 +27,4 @@ Choose to use the corresponding post training quantization tool according to the | Post Training Quantization Tools | Quantization Method Supported | Inference Hardware Platform Supported | Quantization Model Deployment | | --- | --- | --- | --- | | [MindSpore Post Training Quantization Tools](https://www.mindspore.cn/lite/docs/en/master/use/post_training_quantization.html) | Weights Quantization
Full Quantization | CPU | [Inference on edge device](https://www.mindspore.cn/lite/docs/en/master/use/runtime.html) | -| Ascend Model Compression Tool | Full Quantization | Ascend 310 AI Processor | [Inference on Ascend 310 AI Processor](https://www.mindspore.cn/docs/programming_guide/en/master/multi_platform_inference_ascend_310.html) | +| Ascend Model Compression Tool | Full Quantization | Ascend 310 AI Processor | [Inference on Ascend 310 AI Processor](https://www.mindspore.cn/tutorials/experts/en/master/model_infer/multi_platform_inference_ascend_310.html) | diff --git a/tutorials/experts/source_en/operation/op_ascend.md b/tutorials/experts/source_en/operation/op_ascend.md index a679c5dfabc764e32ceb126d4cd35b295c96109f..81d0d259c3e7525e936668b0a5223183d65e5d19 100644 --- a/tutorials/experts/source_en/operation/op_ascend.md +++ b/tutorials/experts/source_en/operation/op_ascend.md @@ -2,7 +2,7 @@ `Ascend` `Model Development` - + ## Overview diff --git a/tutorials/experts/source_en/operation/op_classification.md b/tutorials/experts/source_en/operation/op_classification.md index a36a82cbab645d1b8f89b27f31dba6aa4b594078..a62842c7a20f37a19b372dc04c84adab142e2ee7 100644 --- a/tutorials/experts/source_en/operation/op_classification.md +++ b/tutorials/experts/source_en/operation/op_classification.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Beginner` - + ## Overview diff --git a/tutorials/experts/source_en/operation/op_cpu.md b/tutorials/experts/source_en/operation/op_cpu.md index 9bd639fb1926ef794169da518eadbcce49ec3608..e04d393a54c207ddabe3a5b01e8474f4cb95922f 100644 --- a/tutorials/experts/source_en/operation/op_cpu.md +++ b/tutorials/experts/source_en/operation/op_cpu.md @@ -4,7 +4,7 @@ Translator: [JuLyAi](https://gitee.com/julyai) `CPU` `Model Development` - + ## Overview diff --git a/tutorials/experts/source_en/operation/op_custom.md b/tutorials/experts/source_en/operation/op_custom.md index f03154bfdced377eb4f8474aede179c8fb7bd0a8..080f159f0c500074753e523f65495c068ae25633 100644 --- a/tutorials/experts/source_en/operation/op_custom.md +++ b/tutorials/experts/source_en/operation/op_custom.md @@ -2,11 +2,11 @@ `Ascend` `GPU` `CPU` `Model Development` - + ## Overview -When built-in operators cannot meet requirements during network development, you can call the Python API [Custom](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) primitive defined in MindSpore to quickly create different types of custom operators for use. +When built-in operators cannot meet requirements during network development, you can call the Python API [Custom](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) primitive defined in MindSpore to quickly create different types of custom operators for use. Traditional methods to add a custom operator need three steps: defining the operator primitive, implementing the operator, and registering the operator information. @@ -24,7 +24,7 @@ Compared with traditional custom operator creating methods, creating custom oper ## Basic Usage -The supported custom operator defining methods based on the [Custom](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) primitive include: hybrid, tbe, aot, pyfunc, julia, and akg. +The supported custom operator defining methods based on the [Custom](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) primitive include: hybrid, tbe, aot, pyfunc, julia, and akg. The difference between these operator defining methods are as follows: @@ -81,7 +81,7 @@ if __name__ == "__main__": In this case, - `hybrid` is the default `func_type` of `Custom`. -- The input of custom operators with hybrid type must be a function with decorator [`@ms_hybrid`](https://www.mindspore.cn/docs/api/zh-CN/master/api_python/ops/mindspore.ops.ms_hybrid.html). +- The input of custom operators with hybrid type must be a function with decorator [`@ms_hybrid`](https://www.mindspore.cn/docs/zh-CN/master/api_python/ops/mindspore.ops.ms_hybrid.html). - Users can use the automatic shape/dtype inference functionality of the custom operators with hybrid type, while they can still handwrite shape/dtype functions. Execute the example file: @@ -592,7 +592,7 @@ Running results: ### Registering the Operator Information -The operator information describes the supported inputs and outputs data type, the supported inputs and outputs format, attributes, and target(platform information) of the operator implementation. It is used to select and map operators later. The operator information can be defined by using the [CustomRegOp](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.CustomRegOp.html#mindspore-ops-customregop) API, then you can use the [custom_info_register](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.custom_info_register.html#mindspore-ops-custom-info-register) decorator or just pass it to the `reg_info` parameter of [Custom](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) primitive to bind the information to the operator implementation. The operator information will be registered to the operator information library on the MindSpore C++ side at last. The `reg_info` parameter takes higher priority than the `custom_info_register` decorator. +The operator information describes the supported inputs and outputs data type, the supported inputs and outputs format, attributes, and target(platform information) of the operator implementation. It is used to select and map operators later. The operator information can be defined by using the [CustomRegOp](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.CustomRegOp.html#mindspore-ops-customregop) API, then you can use the [custom_info_register](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.custom_info_register.html#mindspore-ops-custom-info-register) decorator or just pass it to the `reg_info` parameter of [Custom](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.Custom.html#mindspore-ops-custom) primitive to bind the information to the operator implementation. The operator information will be registered to the operator information library on the MindSpore C++ side at last. The `reg_info` parameter takes higher priority than the `custom_info_register` decorator. The target value in operator information can be "Ascend", "GPU" or "CPU". Which describes the operator information on a specific target. For the same operator implementation, it may have different supported data types on different targets, so you can use the target value in operator information to differ this. The operator information on a specific target will be registered only once. @@ -681,7 +681,7 @@ Running results: ### MindSpore Hybrid Developer Guide -MindSpore Hybrid DSL writes Python-like codes, such as function definitions, indents, and comments. With the decorator [`@ms_hybrid`](https://www.mindspore.cn/docs/api/zh-CN/master/api_python/ops/mindspore.ops.ms_hybrid.html), functions written by MindSpore Hybrid DSL can be used as a `numpy` function, as well as used in the custom operators of the hybrid type. +MindSpore Hybrid DSL writes Python-like codes, such as function definitions, indents, and comments. With the decorator [`@ms_hybrid`](https://www.mindspore.cn/docs/zh-CN/master/api_python/ops/mindspore.ops.ms_hybrid.html), functions written by MindSpore Hybrid DSL can be used as a `numpy` function, as well as used in the custom operators of the hybrid type. ```python import numpy as np @@ -888,4 +888,4 @@ To help users effectively develop and locate bugs, MindSpore Hybrid DSL provides - Other frequent error message: - “SyntaxError”: DSL does not conform to the Python syntax(not the syntax defined by MindSpore Hybrid DSL), and is reported by the Python interpreter itself; - “ValueError: Compile error” and “The pointer\[kernel_mod\] is null”: the kernel compiler fails in compiling the DSL. Check error messages from AKG for further information; - - “Launch graph failed”: the compiled kernel fails in running. Check the error message from the hardware. For example, when the kernel fails in Ascend, there will be an “Ascend error occurred” message and corresponding hareware error messages. \ No newline at end of file + - “Launch graph failed”: the compiled kernel fails in running. Check the error message from the hardware. For example, when the kernel fails in Ascend, there will be an “Ascend error occurred” message and corresponding hareware error messages. diff --git a/tutorials/experts/source_en/operation/op_gpu.md b/tutorials/experts/source_en/operation/op_gpu.md index 067ac1ef46f4ef3e63db5f7c0e426f34d4aa4b65..040a8dc365d1c12ba8374cab68b15070cf4f678a 100644 --- a/tutorials/experts/source_en/operation/op_gpu.md +++ b/tutorials/experts/source_en/operation/op_gpu.md @@ -4,7 +4,7 @@ Translator: [Leon_02](https://gitee.com/Leon_02) `GPU` `Model Development` - + ## Overview @@ -284,4 +284,3 @@ dx[0]: [[0. 1. 2. 3.] [4. 5. 6. 7.] [8. 9. 10. 11.]] ``` - diff --git a/tutorials/experts/source_en/operation/op_overload.md b/tutorials/experts/source_en/operation/op_overload.md index bb1818d5a6f57eda78168bcd063a97f3cc5f4e8e..3bbfb40cc86389a1c7b0c1c82b247153bff61ef8 100644 --- a/tutorials/experts/source_en/operation/op_overload.md +++ b/tutorials/experts/source_en/operation/op_overload.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Model Development` - + ## Overview @@ -10,7 +10,7 @@ ## MultitypeFuncGraph -`MultitypeFuncGraph` is used to generate overloaded functions to support different types of input. Users can use `MultitypeFuncGraph` to define a group of overloaded functions. The implementation varies according to the function type. First initialize a `MultitypeFuncGraph` object, and use `register` with input type as the decorator of the function to be registered, so that the object can be called with different types of inputs. For more instructions, see [MultitypeFuncGraph](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.MultitypeFuncGraph.html). +`MultitypeFuncGraph` is used to generate overloaded functions to support different types of input. Users can use `MultitypeFuncGraph` to define a group of overloaded functions. The implementation varies according to the function type. First initialize a `MultitypeFuncGraph` object, and use `register` with input type as the decorator of the function to be registered, so that the object can be called with different types of inputs. For more instructions, see [MultitypeFuncGraph](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.MultitypeFuncGraph.html). A code example is as follows: @@ -45,7 +45,7 @@ scalar 3 ## HyperMap -`HyperMap` can apply an specified operation to one or more input sequences, which can be used with `MultitypeFuncGraph`. For example, after defining a group of overloaded `add` functions, we can apply `add` operation to multiple input groups of different types. Unlike `Map`, `HyperMap` can be used in nested structures to perform specified operations on the input in a sequence or nested sequence. For more instructions, see [HyperMap](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.HyperMap.html). +`HyperMap` can apply an specified operation to one or more input sequences, which can be used with `MultitypeFuncGraph`. For example, after defining a group of overloaded `add` functions, we can apply `add` operation to multiple input groups of different types. Unlike `Map`, `HyperMap` can be used in nested structures to perform specified operations on the input in a sequence or nested sequence. For more instructions, see [HyperMap](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.HyperMap.html). A code example is as follows: diff --git a/tutorials/experts/source_en/others/gradient_accumulation.md b/tutorials/experts/source_en/others/gradient_accumulation.md index d60bea2bbe19b47b003e2bd06005b6d17af291d0..709590fb7b95ba20f0e1f65f8ba496557080c71d 100644 --- a/tutorials/experts/source_en/others/gradient_accumulation.md +++ b/tutorials/experts/source_en/others/gradient_accumulation.md @@ -2,7 +2,7 @@ `GPU` `Model Optimization` - + ## Overview diff --git a/tutorials/experts/source_en/others/mixed_precision.md b/tutorials/experts/source_en/others/mixed_precision.md index be09d7c29f5294e4f96f7c538a53e9974a951558..0e46aee9f0344d965f2fdd49e045a5b9c6b77f92 100644 --- a/tutorials/experts/source_en/others/mixed_precision.md +++ b/tutorials/experts/source_en/others/mixed_precision.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `Model Optimization` - + ## Overview @@ -87,9 +87,9 @@ To use the automatic mixed-precision, you need to call the `Model` API to transf 2. Define a network: This step is the same as that for defining a common network (no new configuration is required). -3. Create a dataset: For details, see [Quick Start of Dataset](https://www.mindspore.cn/docs/programming_guide/en/master/dataset_sample.html). +3. Create a dataset: For details, see [Quick Start of Dataset](https://www.mindspore.cn/tutorials/experts/en/master/others/dataset_sample.html). -4. Use the `Model` API to encapsulate the network model, optimizer, and loss function, and set the `amp_level` parameter. For details, see [MindSpore API](https://www.mindspore.cn/docs/api/en/master/api_python/mindspore.html#mindspore.Model). In this step, MindSpore automatically selects an appropriate operator to convert FP32 to FP16. +4. Use the `Model` API to encapsulate the network model, optimizer, and loss function, and set the `amp_level` parameter. For details, see [MindSpore API](https://www.mindspore.cn/docs/en/master/api_python/mindspore.html#mindspore.Model). In this step, MindSpore automatically selects an appropriate operator to convert FP32 to FP16. The following is a basic code example. First, import the required libraries and declarations, and define the LeNet-5 network model. diff --git a/tutorials/experts/source_en/parallel/auto_parallel.md b/tutorials/experts/source_en/parallel/auto_parallel.md index 0769fb757881b77a2196b79e0388a1c6a3c8d8d7..9c33fe405a05d539e7dfbea7cdf4bab1217519fb 100644 --- a/tutorials/experts/source_en/parallel/auto_parallel.md +++ b/tutorials/experts/source_en/parallel/auto_parallel.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `Distributed Parallel` - + ## Overview @@ -222,7 +222,7 @@ context.get_auto_parallel_context("gradient_fp32_sync") `auto_parallel_search_mode` is replaced by `search_mode`. `auto_parallel_search_mode` will be deleted in the future MindSpore version. This attribute indicates the algorithm chosen for searching op-level parallelism: `dynamic_programming`, `recursive_programming`, and `sharding_propagation`. `dynamic_programming` can search for the optimal policy depicted by the cost model, but it takes a long time to search on a huge network model. -`recursive_programming` can quickly search out parallel policies, but the found policies may not have the optimal running performance. `shardin_propagation` requires users to configure sharding strategies for some operators, and the algorithm will then propagate the strategies from configured ops to non-configured ops, with the goal of minimizing [tensor redistribution](https://www.mindspore.cn/docs/en/master/design/distributed_training_design.html#id10) communication. MindSpore allows users to select a search algorithm. The default value is `dynamic_programming`. +`recursive_programming` can quickly search out parallel policies, but the found policies may not have the optimal running performance. `shardin_propagation` requires users to configure sharding strategies for some operators, and the algorithm will then propagate the strategies from configured ops to non-configured ops, with the goal of minimizing [tensor redistribution](https://www.mindspore.cn/tutorials/experts/en/master/design/distributed_training_design.html#id10) communication. MindSpore allows users to select a search algorithm. The default value is `dynamic_programming`. The following is a code example: @@ -348,7 +348,7 @@ context.set_context(device_target='GPU') init() ``` -> On the GPU processor platform, MindSpore also supports starting distributed training without relying on 'OpenMPI', and also uses this interface for distributed training initialization. For specific usage, please refer to [not using OpenMPI training](https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_gpu.html#openmpi). In this case, when the user does not use 'mpirun' to start the process, but still calls the 'init()' method, MindSpore requires the user to follow [not using OpenMPI training](https://www.mindspore.cn/tutorials/experts/zh-CN/master/parallel/distributed_training_gpu.html#openmpi) to configure several environment variables. If not configured, MindSpore will give a reasonable error prompt. Therefore, it is recommended to call this method only when executing distributed training, and when trying to start distributed training without using 'mpirun', please configure the correct environment variables according to the document. +> On the GPU processor platform, MindSpore also supports starting distributed training without relying on 'OpenMPI', and also uses this interface for distributed training initialization. For specific usage, please refer to [not using OpenMPI training](https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_gpu.html#openmpi). In this case, when the user does not use 'mpirun' to start the process, but still calls the 'init()' method, MindSpore requires the user to follow [not using OpenMPI training](https://www.mindspore.cn/tutorials/experts/zh-CN/master/parallel/distributed_training_gpu.html#openmpi) to configure several environment variables. If not configured, MindSpore will give a reasonable error prompt. Therefore, it is recommended to call this method only when executing distributed training, and when trying to start distributed training without using 'mpirun', please configure the correct environment variables according to the document. ### get_group_size diff --git a/tutorials/experts/source_en/parallel/distributed_inference.md b/tutorials/experts/source_en/parallel/distributed_inference.md index 7e343a3b12125cce62b7b24616f2f66450d67553..ef3021a64ed1452b480a88be89d744b6f0fffe86 100644 --- a/tutorials/experts/source_en/parallel/distributed_inference.md +++ b/tutorials/experts/source_en/parallel/distributed_inference.md @@ -2,7 +2,7 @@ `Ascend` `Inference Application` - + Distributed inference means use multiple devices for prediction. If data parallel or integrated save is used in training, the method of distributed inference is same with the above description. It is noted that each device should load one same checkpoint file. @@ -22,7 +22,7 @@ The process of distributed inference is as follows: > - In the distributed Inference scenario, during the training phase, the `integrated_save` of `CheckpointConfig` interface should be set to `False`, which means that each device only saves the slice of model instead of the full model. > - `parallel_mode` of `set_auto_parallel_context` interface should be set to `auto_parallel` or `semi_auto_parallel`. > - In addition, you need to specify `strategy_ckpt_save_file` to indicate the path of the strategy file. - > - If pipeline distributed inference is used, then the pipeline parallel training also must be used. And the `device_num` and `pipeline_stages` used for pipeline training and inference must be the same. While applying pipeline inference, `micro_size` is 1 and there is no need to use `PipelineCell`. The pipeline distributed training tutorial can be referred the link: . + > - If pipeline distributed inference is used, then the pipeline parallel training also must be used. And the `device_num` and `pipeline_stages` used for pipeline training and inference must be the same. While applying pipeline inference, `micro_size` is 1 and there is no need to use `PipelineCell`. The pipeline distributed training tutorial can be referred the link: . 2. Set context and infer predication strategy according to the predication data. @@ -57,7 +57,7 @@ The process of distributed inference is as follows: > For pipeline inference, each `stage` only needs to load the checkpoint file of self_stage. > > The `load_distributed_checkpoint` interface supports that predict_strategy is `None`, which is single device inference, and the process is different from distributed inference. The detailed usage can be referred to the link: - > . + > . 4. Execute inference. @@ -72,7 +72,7 @@ For Multi-card training and distributed reasoning, it is necessary to export Min First, you need to prepare checkpoint files and training strategy files. -The checkpoint file is generated during the training process. For specific usage of checkpoint, please refer to: [checkpoint usage](https://www.mindspore.cn/docs/programming_guide/en/master/save_model.html#checkpoint). +The checkpoint file is generated during the training process. For specific usage of checkpoint, please refer to: [checkpoint usage](https://www.mindspore.cn/tutorials/experts/en/master/parallel/save_model.html#checkpoint). The training strategy file needs to be generated by setting the context during training. The context configuration items are as follows: `context.set_auto_parallel_context(strategy_ckpt_save_file='train_strategy.ckpt')` @@ -87,7 +87,7 @@ Then, use the method of loading distributed checkpoints to load the previously t code show as below: `load_distributed_checkpoint(model, ckpt_file_list, predict_strategy)` -For the specific usage of `load_distributed_checkpoint`, please refer to: [Distributed Inference](https://www.mindspore.cn/docs/programming_guide/en/master/multi_platform_inference_ascend_910.html#distributed-inference-with-multi-devices). +For the specific usage of `load_distributed_checkpoint`, please refer to: [Distributed Inference](https://www.mindspore.cn/tutorials/experts/en/master/parallel/multi_platform_inference_ascend_910.html#distributed-inference-with-multi-devices). Finally, you can export the MindIR file in the distributed reasoning scenario. @@ -109,7 +109,7 @@ load_distributed_checkpoint(model, ckpt_file_list, predict_strategy) export(net, Tensor(input), file_name='net', file_format='MINDIR') ``` -In the case of multi-card training and single-card inference, the usage of exporting MindIR is the same as that of single machine. For the usage of loading checkpoint, please refer to: [Distributed Inference](https://www.mindspore.cn/docs/programming_guide/en/master/multi_platform_inference_ascend_910.html#ascend-910-ai). +In the case of multi-card training and single-card inference, the usage of exporting MindIR is the same as that of single machine. For the usage of loading checkpoint, please refer to: [Distributed Inference](https://www.mindspore.cn/tutorials/experts/en/master/parallel/multi_platform_inference_ascend_910.html#ascend-910-ai). > Distributed scene export MindIR file sample code: > diff --git a/tutorials/experts/source_en/parallel/distributed_training.rst b/tutorials/experts/source_en/parallel/distributed_training.rst index 75d9c4e1615792a1ba37580a4025fd3a7fb0882a..3c724eb9fa793994e5746cd5c1aaca8da5ae04dd 100644 --- a/tutorials/experts/source_en/parallel/distributed_training.rst +++ b/tutorials/experts/source_en/parallel/distributed_training.rst @@ -14,7 +14,7 @@ MindSpore also provides the parallel training functionality. It supports the fol - `dynamic_programming`: Dynamic programming search algorithm. The optimal strategy under the cost model description can be found, but it takes a long time to search for parallel strategy of huge network model. Its cost model refers to modeling the training time based on the memory-based computation and communication overheads of the Ascend 910 chip. - `recursive_programming`: Double recursive programming search algorithm. The optimal strategy can be generated instantly even for a large network. Its symbolic cost model can be flexibly adapted to different accelerator clusters. - - `sharding_propagation`: Sharding Propagation algorithms. This mode requires users to configure sharding strategies for some operators, and then propagates from these operators to other operators, with the goal of minimizing communication cost in tensor redistribution. For definitions of sharding strategy and tensor redistribution, please refer to this [design article](https://www.mindspore.cn/docs/en/master/design/distributed_training_design.html#id10) + - `sharding_propagation`: Sharding Propagation algorithms. This mode requires users to configure sharding strategies for some operators, and then propagates from these operators to other operators, with the goal of minimizing communication cost in tensor redistribution. For definitions of sharding strategy and tensor redistribution, please refer to this [design article](https://www.mindspore.cn/tutorials/experts/en/master/design/distributed_training_design.html#id10) - `HYBRID_PARALLEL`: On MindSpore, users manually split parameters to implement intra-layer model parallelism. diff --git a/tutorials/experts/source_en/parallel/distributed_training_ascend.md b/tutorials/experts/source_en/parallel/distributed_training_ascend.md index e9d25d39f7131493f3540f7c81a2d2d566f0020d..c37f919e74b334f485db950b3c925957d6e64a4f 100644 --- a/tutorials/experts/source_en/parallel/distributed_training_ascend.md +++ b/tutorials/experts/source_en/parallel/distributed_training_ascend.md @@ -2,7 +2,7 @@ `Ascend` `Distributed Parallel` `Whole Process` - + ## Overview @@ -31,7 +31,7 @@ The directory structure is as follow: `rank_table_16pcs.json`, `rank_table_8pcs.json` and `rank_table_2pcs.json` are the networking information files. `resnet.py`,`resnet50_distributed_training.py` , `resnet50_distributed_training_gpu.py` and `resnet50_distributed_training_grad_accu.py` are the network structure files. `run.sh` , `run_gpu.sh`, `run_grad_accu.sh` and `run_cluster.sh` are the execute scripts. -Besides, we describe the usages of hybrid parallel and semi-auto parallel modes in the sections [Defining the Network](https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_ascend.html#defining-the-network) and [Distributed Training Model Parameters Saving and Loading](https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_ascend.html#distributed-training-model-parameters-saving-and-loading). +Besides, we describe the usages of hybrid parallel and semi-auto parallel modes in the sections [Defining the Network](https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_ascend.html#defining-the-network) and [Distributed Training Model Parameters Saving and Loading](https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_ascend.html#distributed-training-model-parameters-saving-and-loading). ## Preparations @@ -177,7 +177,7 @@ In this section we focus on how to define a network in hybrid parallel or semi-a Hybrid parallel mode adds the setting `layerwise_parallel` for `parameter` based on the data parallel mode. The `parameter` with the setting would be saved and computed in slice tensor and would not apply gradients aggregation. In this mode, MindSpore would not infer computation and communication for parallel operators automatically. To ensure the consistency of calculation logic, users are required to manually infer extra operations and insert them to networks. Therefore, this parallel mode is suitable for the users with deep understanding of parallel theory. -In the following example, specify the `self.weight` as the `layerwise_parallel`, that is, the `self.weight` and the output of `MatMul` are sliced on the second dimension. At this time, perform ReduceSum on the second dimension would only get one sliced result. `AllReduce.Sum` is required here to accumulate the results among all devices. More information about the parallel theory please refer to the [design document](https://www.mindspore.cn/docs/en/master/design/distributed_training_design.html). +In the following example, specify the `self.weight` as the `layerwise_parallel`, that is, the `self.weight` and the output of `MatMul` are sliced on the second dimension. At this time, perform ReduceSum on the second dimension would only get one sliced result. `AllReduce.Sum` is required here to accumulate the results among all devices. More information about the parallel theory please refer to the [design document](https://www.mindspore.cn/tutorials/experts/en/master/design/distributed_training_design.html). ```python from mindspore import Tensor @@ -204,7 +204,7 @@ class HybridParallelNet(nn.Cell): ### Semi Auto Parallel Mode -Compared with the auto parallel mode, semi auto parallel mode supports manual configuration on shard strategies for network tuning. The definition of shard strategies could be referred by this [design document](https://www.mindspore.cn/docs/en/master/design/distributed_training_design.html). +Compared with the auto parallel mode, semi auto parallel mode supports manual configuration on shard strategies for network tuning. The definition of shard strategies could be referred by this [design document](https://www.mindspore.cn/tutorials/experts/en/master/design/distributed_training_design.html). In the above example `HybridParallelNet`, the script in semi auto parallel mode is as follows. The shard stratege of `MatMul` is `((1, 1), (1, 2))`, which means `self.weight` is sliced at the second dimension. @@ -294,7 +294,7 @@ The `Momentum` optimizer is used as the parameter update tool. The definition is - `gradients_mean`: During backward computation, the framework collects gradients of parameters in data parallel mode across multiple hosts, obtains the global gradient value, and transfers the global gradient value to the optimizer for update. The default value is `False`, which indicates that the `AllReduce.Sum` operation is applied. The value `True` indicates that the `AllReduce.Mean` operation is applied. - You are advised to set `device_num` and `global_rank` to their default values. The framework calls the HCCL API to obtain the values. -> More about the distributed training configurations please refer to the [programming guide](https://www.mindspore.cn/docs/programming_guide/en/master/auto_parallel.html). +> More about the distributed training configurations please refer to the [programming guide](https://www.mindspore.cn/tutorials/experts/en/master/parallel/auto_parallel.html). If multiple network cases exist in the script, call `context.reset_auto_parallel_context` to restore all parameters to default values before executing the next case. @@ -554,7 +554,7 @@ bash run_cluster.sh /path/dataset /path/rank_table.json 16 8 ### Non-sink Mode Training -In graph mode, you can specify to train the model in a non-sink mode by setting the environment variable [GRAPH_OP_RUN](https://www.mindspore.cn/docs/en/master/note/env_var_list.html)=1. In this case, you need to set environment variable `HCCL_WHITELIST_DISABLE=1` and train model with OpenMPI `mpirun`. The startup script is consistent with the [GPU's distributed training](https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_gpu.html#running-the-script) script. +In graph mode, you can specify to train the model in a non-sink mode by setting the environment variable [GRAPH_OP_RUN](https://www.mindspore.cn/docs/note/en/master/env_var_list.html)=1. In this case, you need to set environment variable `HCCL_WHITELIST_DISABLE=1` and train model with OpenMPI `mpirun`. The startup script is consistent with the [GPU's distributed training](https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_gpu.html#running-the-script) script. ## Distributed Training Model Parameters Saving and Loading @@ -593,7 +593,7 @@ param_dict = load_checkpoint(pretrain_ckpt_path) load_param_into_net(net, param_dict) ``` -For checkpoint configuration policy and saving method, please refer to [Saving and Loading Model Parameters](https://www.mindspore.cn/docs/programming_guide/en/master/save_model.html#checkpoint-configuration-policies). +For checkpoint configuration policy and saving method, please refer to [Saving and Loading Model Parameters](https://www.mindspore.cn/tutorials/experts/en/master/parallel/save_model.html#checkpoint-configuration-policies). By default, sliced parameters would be merged before saving automatocally. However, considering large-scaled networks, a large size checkpoint file will be difficult to be transferred and loaded. So every device can save sliced parameters separately by setting `integrated_save` as `False` in `CheckpointConfig`. If the shard strategies of retraining or inference are different with that of training, the special loading way is needed. @@ -621,7 +621,7 @@ load_distributed_checkpoint(model.train_network, ckpt_file_list, layout_dict) model.train(2, dataset) ``` -> Distributed inference could be referred to [Distributed inference](https://www.mindspore.cn/docs/programming_guide/en/master/multi_platform_inference_ascend_910.html#id1). +> Distributed inference could be referred to [Distributed inference](https://www.mindspore.cn/tutorials/experts/en/master/parallel/multi_platform_inference_ascend_910.html#id1). ### Data Parallel Mode @@ -641,7 +641,7 @@ context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL, grad ### Semi Auto Parallel Mode -In semi auto parallel mode, checkpoint is used in the same way as in auto parallel mode and data parallel mode. The difference is in the definition of a network and the definition of network model, you can refer to defining the network [Semi Auto Parallel Mode](https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_ascend.html#semi-auto-parallel-mode) in this tutorial. +In semi auto parallel mode, checkpoint is used in the same way as in auto parallel mode and data parallel mode. The difference is in the definition of a network and the definition of network model, you can refer to defining the network [Semi Auto Parallel Mode](https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_ascend.html#semi-auto-parallel-mode) in this tutorial. To save the model, you can use the following code: @@ -680,8 +680,8 @@ to: ckpt_config = CheckpointConfig(keep_checkpoint_max=1, integrated_save=False) ``` -It should be noted that if users choose this checkpoint saving policy, users need to save and load the segmented checkpoint for subsequent reasoning or retraining. Specific usage can refer to [Integrating the Saved Checkpoint Files](https://www.mindspore.cn/docs/programming_guide/en/master/save_load_model_hybrid_parallel.html#integrating-the-saved-checkpoint-files). +It should be noted that if users choose this checkpoint saving policy, users need to save and load the segmented checkpoint for subsequent reasoning or retraining. Specific usage can refer to [Integrating the Saved Checkpoint Files](https://www.mindspore.cn/tutorials/experts/en/master/parallel/save_load_model_hybrid_parallel.html#integrating-the-saved-checkpoint-files). ### Hybrid Parallel Mode -For model parameter saving and loading in Hybrid Parallel Mode, please refer to [Saving and Loading Model Parameters in the Hybrid Parallel Scenario](https://www.mindspore.cn/docs/programming_guide/en/master/save_load_model_hybrid_parallel.html). +For model parameter saving and loading in Hybrid Parallel Mode, please refer to [Saving and Loading Model Parameters in the Hybrid Parallel Scenario](https://www.mindspore.cn/tutorials/experts/en/master/parallel/save_load_model_hybrid_parallel.html). diff --git a/tutorials/experts/source_en/parallel/distributed_training_gpu.md b/tutorials/experts/source_en/parallel/distributed_training_gpu.md index edb4a1ad4df4b1fefba26c9b7f0c1cf5fda40bdc..da1cc4664bec55591db8551183b5a036ef88e3c1 100644 --- a/tutorials/experts/source_en/parallel/distributed_training_gpu.md +++ b/tutorials/experts/source_en/parallel/distributed_training_gpu.md @@ -2,7 +2,7 @@ `GPU` `Distributed Parallel` `Whole Process` - + ## Overview @@ -14,7 +14,7 @@ This tutorial describes how to train the ResNet-50 network using MindSpore data The `CIFAR-10` dataset is used as an example. The method of downloading and loading the dataset is the same as that for the Ascend 910 AI processor. -The method of downloading and loading the dataset: +The method of downloading and loading the dataset: ### Configuring Distributed Environment @@ -60,7 +60,7 @@ In the preceding information, On the GPU hardware platform, the network definition is the same as that for the Ascend 910 AI processor. -For details about the definitions of the network, optimizer, and loss function, see . +For details about the definitions of the network, optimizer, and loss function, see . ## Running the Script @@ -134,4 +134,4 @@ echo "start training" mpirun -n 16 --hostfile $HOSTFILE -x DATA_PATH=$DATA_PATH -x PATH -mca pml ob1 pytest -s -v ./resnet50_distributed_training.py > train.log 2>&1 & ``` -Run running on GPU, the model parameters can be saved and loaded by referring to [Distributed Training Model Parameters Saving and Loading](https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_ascend.html#distributed-training-model-parameters-saving-and-loading). +Run running on GPU, the model parameters can be saved and loaded by referring to [Distributed Training Model Parameters Saving and Loading](https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_ascend.html#distributed-training-model-parameters-saving-and-loading). diff --git a/tutorials/experts/source_en/parallel/distributed_training_mode.md b/tutorials/experts/source_en/parallel/distributed_training_mode.md index 69d4439542fc5fd6f24f0dfaf90f4e45845f9e6e..1dfa28b4795921a041a1955cdeed096243ac89ef 100644 --- a/tutorials/experts/source_en/parallel/distributed_training_mode.md +++ b/tutorials/experts/source_en/parallel/distributed_training_mode.md @@ -2,5 +2,4 @@ No English version available right now, welcome to contribute. - - + diff --git a/tutorials/experts/source_en/parallel/host_device_training.md b/tutorials/experts/source_en/parallel/host_device_training.md index 4298b22695cdbe425a624531fa9b3df47390ce52..f6d1e5be071f73fec9a6b5d1d2c97b942f4585f8 100644 --- a/tutorials/experts/source_en/parallel/host_device_training.md +++ b/tutorials/experts/source_en/parallel/host_device_training.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `Distributed Parallel` `Whole Process` - + ## Overview diff --git a/tutorials/experts/source_en/parallel/parameter_server_training.md b/tutorials/experts/source_en/parallel/parameter_server_training.md index 49444af8b5bf7bccf75854b997c760bc92e9182b..abb156460a35a34687da32cd1f31acd155a01348 100644 --- a/tutorials/experts/source_en/parallel/parameter_server_training.md +++ b/tutorials/experts/source_en/parallel/parameter_server_training.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `Distributed Parallel` `Whole Process` - + ## Overview @@ -31,7 +31,7 @@ Learn how to train a LeNet using the [MNIST dataset](http://yann.lecun.com/exdb/ 1. First of all, use `mindspore.context.set_ps_context(enable_ps=True)` to enable Parameter Server training mode. - This method should be called before `mindspore.communication.init()`. - - If you don't call this method, the [Environment Variable Setting](https://www.mindspore.cn/docs/programming_guide/en/master/apply_parameter_server_training.html#environment-variable-setting) below will not take effect. + - If you don't call this method, the [Environment Variable Setting](https://www.mindspore.cn/tutorials/experts/en/master/parallel/apply_parameter_server_training.html#environment-variable-setting) below will not take effect. - Use `mindspore.context.reset_ps_context()` to disable Parameter Server training mode. 2. In this training mode, you can use either of the following methods to control whether the training parameters are updated by the Parameter Server and whether the training parameters are initialized on Worker or Server: @@ -49,7 +49,7 @@ Learn how to train a LeNet using the [MNIST dataset](http://yann.lecun.com/exdb/ network.set_param_ps() ``` -4. [optional configuration] For a large shape `embedding_table`, because the device can not store a full amount of `embedding_table`. You can configure the `vocab_cache_size` of [EmbeddingLookup operator](https://www.mindspore.cn/docs/api/en/master/api_python/nn/mindspore.nn.EmbeddingLookup.html) to enable the cache function of `EmbeddingLookup` in the Parameter Server training mode. The `vocab_cache_size` of `embedding_table` is trained on device, and a full amount of `embedding_table` is stored in the Server. The `embedding_table` of the next batch is swapped to the cache in advance, and the expired `embedding_table` is put back to the Server when the cache cannot be placed, to achieve the purpose of improving the training performance. Each Server could save a checkpoint containing the trained `embedding_table` after the training. Detailed network training script can be referred to . +4. [optional configuration] For a large shape `embedding_table`, because the device can not store a full amount of `embedding_table`. You can configure the `vocab_cache_size` of [EmbeddingLookup operator](https://www.mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.EmbeddingLookup.html) to enable the cache function of `EmbeddingLookup` in the Parameter Server training mode. The `vocab_cache_size` of `embedding_table` is trained on device, and a full amount of `embedding_table` is stored in the Server. The `embedding_table` of the next batch is swapped to the cache in advance, and the expired `embedding_table` is put back to the Server when the cache cannot be placed, to achieve the purpose of improving the training performance. Each Server could save a checkpoint containing the trained `embedding_table` after the training. Detailed network training script can be referred to . ```python context.set_auto_parallel_context(full_batch=True, diff --git a/tutorials/experts/source_en/parallel/pipeline_parallel.md b/tutorials/experts/source_en/parallel/pipeline_parallel.md index ddafd6149ff09cc2611f36e61ff6c8ac2592ef4a..a059a7b123d123ccb0895d3884ed750dabddb0a8 100644 --- a/tutorials/experts/source_en/parallel/pipeline_parallel.md +++ b/tutorials/experts/source_en/parallel/pipeline_parallel.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `Distributed Parallel` `Whole Process` - + ## Overview @@ -39,21 +39,21 @@ The directory structure is as follows: ### Downloading the Dataset This example uses the `CIFAR-10` dataset. For details about how to download and load the dataset, -visit . +visit . ### Configuring the Distributed Environment > Pipeline parallelism supports Ascend and GPU. For details about how to configure the distributed environment and call the HCCL, -visit . +visit . ## Defining the Network The network definition is the same as that in the Parallel Distributed Training Example. For details about the definitions of the network, optimizer, and loss function, -visit . +visit . > To implement pipeline parallelism, you need to define the parallel strategy and call the `pipeline_stage` API to specify the stage on which each layer is to be executed. The granularity of the `pipeline_stage` API is `Cell`. `pipeline_stage` must be configured for all `Cells` that contain training parameters. @@ -149,4 +149,4 @@ You can run a 2-stage pipeline on 8 GPU devices using below scripts: ```bash bash run_pipeline.sh [DATA_PATH] GPU -``` \ No newline at end of file +``` diff --git a/tutorials/experts/source_en/parallel/recompute.md b/tutorials/experts/source_en/parallel/recompute.md index c59cc7b414894b269d973997c5c60df377d1d75c..413e4dc74eb9d67caff957f7026368f3948b35c3 100644 --- a/tutorials/experts/source_en/parallel/recompute.md +++ b/tutorials/experts/source_en/parallel/recompute.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `CPU` `Model Running` `Intermediate` `Expert` - + ## Overview @@ -14,13 +14,13 @@ In order to solve this problem, Mindspore provides the recomputation function. I 1. Prepare the model. The ResNet-50 code can be found at: , in which `train.py` is the main function for training, `src/` directory contains the model definition and configuration files of ResNet-50, `script/` directory contains the training and evaluation scripts. -2. Prepare the dataset. This example uses the `CIFAR-10` dataset. For details about how to download and load the dataset, visit . +2. Prepare the dataset. This example uses the `CIFAR-10` dataset. For details about how to download and load the dataset, visit . ## Configuring for Recomputation We can call two kinds of interface to configure the recomputation. Take `src/resnet.py` for example: -1. The [recompute api](https://www.mindspore.cn/docs/api/en/master/api_python/ops/mindspore.ops.Primitive.html#mindspore.ops.Primitive.recompute) of `Primitive`. It can set an operator to be recomputed. After setting, the operator will be recomputed in the backward pass. +1. The [recompute api](https://www.mindspore.cn/docs/en/master/api_python/ops/mindspore.ops.Primitive.html#mindspore.ops.Primitive.recompute) of `Primitive`. It can set an operator to be recomputed. After setting, the operator will be recomputed in the backward pass. ```python class ResNet(nn.Cell): @@ -41,7 +41,7 @@ We can call two kinds of interface to configure the recomputation. Take `src/res ... ``` -2. Call the [recompute api](https://www.mindspore.cn/docs/api/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.recompute) of `Cell`. It can set the whole `Cell` to be recomputed. After setting, except the output of the `Cell`, all the operators in this cell and its sub `Cell` will be recomputed in the backward pass. +2. Call the [recompute api](https://www.mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell.recompute) of `Cell`. It can set the whole `Cell` to be recomputed. After setting, except the output of the `Cell`, all the operators in this cell and its sub `Cell` will be recomputed in the backward pass. ```python class ResNet(nn.Cell): diff --git a/tutorials/experts/source_en/parallel/save_load_model_hybrid_parallel.md b/tutorials/experts/source_en/parallel/save_load_model_hybrid_parallel.md index 248b6a30b610027799d4aa2c8065992d20a6ea3a..97da5d1940d02f94a41431c1ed4459b20293819d 100644 --- a/tutorials/experts/source_en/parallel/save_load_model_hybrid_parallel.md +++ b/tutorials/experts/source_en/parallel/save_load_model_hybrid_parallel.md @@ -2,7 +2,7 @@ `Ascend` `GPU` `Distributed Parallel` `Model Export` `Model Loading` - + ## Overview @@ -243,7 +243,7 @@ User process: 3. Execute stage 2 training: There are two devices in stage 2 training environment. The weight shape of the MatMul operator on each device is \[4, 8]. Load the initialized model parameter data from the integrated checkpoint file and then perform training. -> For details about the distributed environment configuration and training code, see [Distributed Training](https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_ascend.html). +> For details about the distributed environment configuration and training code, see [Distributed Training](https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_ascend.html). > > This document provides the example code for integrating checkpoint files and loading checkpoint files before distributed training. The code is for reference only. diff --git a/tutorials/experts/source_en/parallel/sharding_propagation.md b/tutorials/experts/source_en/parallel/sharding_propagation.md index b14c38e6c90182baf6e6a268e133674786f6e76c..82a8e860070add7cc02825525dd9905eb984fd58 100644 --- a/tutorials/experts/source_en/parallel/sharding_propagation.md +++ b/tutorials/experts/source_en/parallel/sharding_propagation.md @@ -2,11 +2,11 @@ `Ascend` `GPU` `Parallel Training` `Automatic Parallelization` - + ## Background -Distributed operator, Tensor Layout, and Tensor Redistribution are fundamental concepts in op-level parallelism of MindSpore. In [here](https://www.mindspore.cn/docs/en/master/design/distributed_training_design.html#automatic-parallelism), these concepts are introduced by examples. Here, we formally define them. +Distributed operator, Tensor Layout, and Tensor Redistribution are fundamental concepts in op-level parallelism of MindSpore. In [here](https://www.mindspore.cn/tutorials/experts/en/master/design/distributed_training_design.html#automatic-parallelism), these concepts are introduced by examples. Here, we formally define them. In op-level parallelism, we conduct SPMD (Single Program Multiple Data) style parallelism, that is, a single program is produced for all partitions. MindSpore transforms a stand-alone program to a parallel one. The transformation is fine-grained in the sense that each operator in the stand-alone program is substituted by (a) distributed operator(s), guaranteeing that the substitution is mathematically equivalent. @@ -66,7 +66,7 @@ The following figure illustrates an example process of applying Sharding Propaga > > . -The directory structure is as follows, where `rank_table_8pcs.json` is the IP configuration for Ascend devices (see [here]( https://www.mindspore.cn/docs/programming_guide/en/master/distributed_training_ascend.html#configuring-distributed-environment-variables) for the explanation), `train.py` is the model definition, and `run.sh` is the execution script. +The directory structure is as follows, where `rank_table_8pcs.json` is the IP configuration for Ascend devices (see [here]( https://www.mindspore.cn/tutorials/experts/en/master/parallel/distributed_training_ascend.html#configuring-distributed-environment-variables) for the explanation), `train.py` is the model definition, and `run.sh` is the execution script. ```text └─sample_code