From e44573de4095ccd50c850cccc021c7388ac5a341 Mon Sep 17 00:00:00 2001 From: huanxiaoling <3174348550@qq.com> Date: Wed, 26 Oct 2022 15:03:30 +0800 Subject: [PATCH] correct the wrong links in files --- docs/federated/docs/source_en/data_join.md | 2 +- .../image_classification_application.md | 10 +++++----- docs/federated/docs/source_en/index.rst | 4 ++-- .../interface_description_federated_client.md | 8 ++++---- ...ect_detection_application_in_cross_silo.md | 20 +++++++++---------- .../sentiment_classification_application.md | 2 +- .../split_pangu_alpha_application.md | 2 +- .../interface_description_federated_client.md | 4 ++-- ...ect_detection_application_in_cross_silo.md | 4 ++-- 9 files changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/federated/docs/source_en/data_join.md b/docs/federated/docs/source_en/data_join.md index 2df1a8e88d..720f62fe27 100644 --- a/docs/federated/docs/source_en/data_join.md +++ b/docs/federated/docs/source_en/data_join.md @@ -200,7 +200,7 @@ Follower data export results: ## An Example for Deep Experience -For detailed API documentation for the following code, see [Data Access Documentation](https://gitee.com/mindspore/federated/blob/master/docs/api/api_python/data_join.rst). +For detailed API documentation for the following code, see [Data Access Documentation](https://www.mindspore.cn/federated/docs/en/master/data_join/data_join.html). ### Data Export diff --git a/docs/federated/docs/source_en/image_classification_application.md b/docs/federated/docs/source_en/image_classification_application.md index d5c94048a2..5c77e7ac53 100644 --- a/docs/federated/docs/source_en/image_classification_application.md +++ b/docs/federated/docs/source_en/image_classification_application.md @@ -134,21 +134,21 @@ We provide [Federated Learning Image Classification Dataset FEMNIST](https://min ## Simulating Multi-client Participation in Federated Learning -### Prepare a model file for the client +### Preparing a Model File for the Client This example uses lenet on the device-side to simulate the actual network used, where[device-side model file](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/models/lenet_train.ms) in `.ms` format of lenet. As the real scenario where a client contains only one model file in .ms format, in the simulation scenario, multiple copies of the .ms file need to be copied and named according to the `lenet_train{i}.ms` format, where i represents the client number, since the .ms file has been automatically copied for each client in `run_client_x86.py`. See the copy_ms function in [startup script](https://gitee.com/mindspore/federated/blob/master/example/cross_device_lenet_femnist/simulate_x86/run_client_x86.py) for details. -### Start the cloud side service +### Starting the Cloud Side Service Users can first refer to [cloud-side deployment tutorial](https://www.mindspore.cn/federated/docs/en/master/deploy_federated_server.html) to deploy the cloud-side environment and start the cloud-side service. -### Start the client +### Starting the Client Before starting the client, please refer to the section [Device-side deployment tutotial](https://www.mindspore.cn/federated/docs/en/master/deploy_federated_client.html) for deployment of device environment. -We provide a reference script [run_client_x86.py](https://gitee.com/mindspore/mindspore/blob/master/tests/st/fl/cross_device_lenet/client/run_client_x86.py), users can set relevant parameters to start different federated learning interfaces. +We provide a reference script [run_client_x86.py](https://gitee.com/mindspore/federated/blob/master/example/cross_device_lenet_femnist/simulate_x86/run_client_x86.py), users can set relevant parameters to start different federated learning interfaces. After the cloud-side service is successfully started, the script providing run_client_x86.py is used to call the federated learning framework jar package `mindspore-lite-java-flclient.jar` and the corresponding jar package `quick_start_flclient.jar` of the model script, obtaining in [Compiling package Flow in device-side deployment](https://www.mindspore.cn/federated/docs/en/master/deploy_federated_client.html) to simulate starting multiple clients to participate in the federated learning task. Taking the LeNet network as an example, some of the input parameters in the `run_client_x86.py` script have the following meanings, and users can set them according to the actual situation: @@ -171,7 +171,7 @@ Taking the LeNet network as an example, some of the input parameters in the `run - `--flName` - Specifies the package path of model script used by federated learning. We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert), [Lenet image classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`. For Lenet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/fl_client/src/main/java/com/mindspore/flclient/model/Client.java)). + Specifies the package path of model script used by federated learning. We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert), [Lenet image classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`. For Lenet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/model/Client.java)). - `--train_model_path` diff --git a/docs/federated/docs/source_en/index.rst b/docs/federated/docs/source_en/index.rst index f46c1a5c61..b90ecfa280 100644 --- a/docs/federated/docs/source_en/index.rst +++ b/docs/federated/docs/source_en/index.rst @@ -44,11 +44,11 @@ Advantages of the MindSpore Federated MindSpore Federated Working Process ------------------------------------ -1. `Scenario Identification and Data Accumulation `_ +1. `Scenario Identification and Data Accumulation `_ Identify scenarios where federated learning is used and accumulate local data for federated tasks on the client. -2. `Model Selection and Client Deployment `_ +2. `Model Selection and Client Deployment `_ Select or develop a model prototype and use a tool to generate a device model that is easy to deploy. diff --git a/docs/federated/docs/source_en/interface_description_federated_client.md b/docs/federated/docs/source_en/interface_description_federated_client.md index 5d5e733a0f..b248cd0547 100644 --- a/docs/federated/docs/source_en/interface_description_federated_client.md +++ b/docs/federated/docs/source_en/interface_description_federated_client.md @@ -11,7 +11,7 @@ Before calling the flJobRun() API, instantiate the parameter class FLParameter a | Parameter | Type | Mandatory | Description | Remarks | | -------------------- | ---------------------------- | --------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | dataMap | Map/> | Y | The path of Federated learning dataset. | The dataset of Map/> type, the key in the map is the RunType enumeration type, the value is the corresponding dataset list, when the key is RunType.TRAINMODE, the corresponding value is the training-related dataset list, when the key is RunType.EVALMODE, it means that the corresponding value is a list of verification-related datasets, and when the key is RunType.INFERMODE, it means that the corresponding value is a list of inference-related datasets. | -| flName | String | Y | The package path of model script used by federated learning. | We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert), [LeNet image classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`; for LeNet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/fl_client/src/main/java/com/mindspore/flclient/model/Client.java)). | +| flName | String | Y | The package path of model script used by federated learning. | We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert)), ([LeNet image classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`; for LeNet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/model/Client.java)). | | trainModelPath | String | Y | Path of a training model used for federated learning, which is an absolute path of the .ms file. | It is recommended to set the path to the training App's own directory to protect the data access security of the model itself. | | inferModelPath | String | Y | Path of an inference model used for federated learning, which is an absolute path of the .ms file. | For the normal federated learning mode (training and inference use the same model), the value of this parameter needs to be the same as that of `trainModelPath`; for the hybrid learning mode (training and inference use different models, and the server side also includes training process), this parameter is set to the path of actual inference model. It is recommended to set the path to the training App's own directory to protect the data access security of the model itself. | | sslProtocol | String | N | The TLS protocol version used by the device-cloud HTTPS communication. | A whitelist is set, and currently only "TLSv1.3" or "TLSv1.2" is supported. Only need to set it up in the HTTPS communication scenario. | @@ -24,7 +24,7 @@ Before calling the flJobRun() API, instantiate the parameter class FLParameter a | threadNum | int | N | The number of threads used in federated learning training and inference. | The default value is 1. | | cpuBindMode | BindMode | N | The cpu core that threads need to bind during federated learning training and inference. | It is the enumeration type `BindMode`, where BindMode.NOT_BINDING_CORE represents the unbound core, which is automatically assigned by the system, BindMode.BIND_LARGE_CORE represents the bound large core, and BindMode.BIND_MIDDLE_CORE represents the bound middle core. The default value is BindMode.NOT_BINDING_CORE. | | batchSize | int | Y | The number of single-step training samples used in federated learning training and inference, that is, batch size. | It needs to be consistent with the batch size of the input data of the model. | -| iflJobResultCallback | IFLJobResultCallback | N | The federated learning callback function object `iflJobResultCallback`. | The user can implement the specific method of the interface class [IFLJobResultCallback.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/fl_client/src/main/java/com/mindspore/flclient/IFLJobResultCallback.java) in the project according to the needs of the actual scene, and set it as a callback function object in the federated learning task. We provide a simple implementation use case [FLJobResultCallback.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/fl_client/src/main/java/com/mindspore/flclient/FLJobResultCallback.java) as the default value of this parameter. | +| iflJobResultCallback | IFLJobResultCallback | N | The federated learning callback function object `iflJobResultCallback`. | The user can implement the specific method of the interface class [IFLJobResultCallback.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/IFLJobResultCallback.java) in the project according to the needs of the actual scene, and set it as a callback function object in the federated learning task. We provide a simple implementation use case [FLJobResultCallback.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/IFLJobResultCallback.java) as the default value of this parameter. | Note 1: When using HTTP communication, there may exist communication security risks, please be aware. @@ -171,7 +171,7 @@ Before calling the modelInference() API, instantiate the parameter class FLParam | Parameter | Type | Mandatory | Description | Remarks | | -------------- | ---------------------------- | --------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| flName | String | Y | The package path of model script used by federated learning. | We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert), [LeNet image classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`; for LeNet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/fl_client/src/main/java/com/mindspore/flclient/model/Client.java)). | +| flName | String | Y | The package path of model script used by federated learning. | We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert), [LeNet image classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`; for LeNet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/model/Client.java)). | | dataMap | Map/> | Y | The path of Federated learning dataset. | The dataset of Map/> type, the key in the map is the RunType enumeration type, the value is the corresponding dataset list, when the key is RunType.TRAINMODE, the corresponding value is the training-related dataset list, when the key is RunType.EVALMODE, it means that the corresponding value is a list of verification-related datasets, and when the key is RunType.INFERMODE, it means that the corresponding value is a list of inference-related datasets. | | inferModelPath | String | Y | Path of an inference model used for federated learning, which is an absolute path of the .ms file. | For the normal federated learning mode (training and inference use the same model), the value of this parameter needs to be the same as that of `trainModelPath`; for the hybrid learning mode (training and inference use different models, and the server side also includes training process), this parameter is set to the path of actual inference model. It is recommended to set the path to the training App's own directory to protect the data access security of the model itself. | | threadNum | int | N | The number of threads used in federated learning training and inference. | The default value is 1. | @@ -252,7 +252,7 @@ Before calling the getModel() API, instantiate the parameter class FLParameter a | Parameter | Type | Mandatory | Description | Remarks | | -------------- | --------- | --------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| flName | String | Y | The package path of model script used by federated learning. | We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert), [LeNet image classification task](https://gitee.com/mindspore/mindspore/tree/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`; for LeNet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/examples/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/fl_client/src/main/java/com/mindspore/flclient/model/Client.java)). | +| flName | String | Y | The package path of model script used by federated learning. | We provide two types of model scripts for your reference ([Supervised sentiment classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert), [LeNet image classification task](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)). For supervised sentiment classification tasks, this parameter can be set to the package path of the provided script file [AlBertClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java), like as `com.mindspore.flclient.demo.albert.AlbertClient`; for LeNet image classification tasks, this parameter can be set to the package path of the provided script file [LenetClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java), like as `com.mindspore.flclient.demo.lenet.LenetClient`. At the same time, users can refer to these two types of model scripts, define the model script by themselves, and then set the parameter to the package path of the customized model file ModelClient.java (which needs to inherit from the class [Client.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/model/Client.java)). | | trainModelPath | String | Y | Path of a training model used for federated learning, which is an absolute path of the .ms file. | It is recommended to set the path to the training App's own directory to protect the data access security of the model itself. | | inferModelPath | String | Y | Path of an inference model used for federated learning, which is an absolute path of the .ms file. | For the normal federated learning mode (training and inference use the same model), the value of this parameter needs to be the same as that of `trainModelPath`; for the hybrid learning mode (training and inference use different models, and the server side also includes training process), this parameter is set to the path of actual inference model. It is recommended to set the path to the training App's own directory to protect the data access security of the model itself. | | sslProtocol | String | N | The TLS protocol version used by the device-cloud HTTPS communication. | A whitelist is set, and currently only "TLSv1.3" or "TLSv1.2" is supported. Only need to set it up in the HTTPS communication scenario. | diff --git a/docs/federated/docs/source_en/object_detection_application_in_cross_silo.md b/docs/federated/docs/source_en/object_detection_application_in_cross_silo.md index de7b5aca0d..ecde354c56 100644 --- a/docs/federated/docs/source_en/object_detection_application_in_cross_silo.md +++ b/docs/federated/docs/source_en/object_detection_application_in_cross_silo.md @@ -4,7 +4,7 @@ Based on the type of participating clients, federated learning can be classified into cross-silo federated learning and cross-device federated learning. In a cross-silo federated learning scenario, the clients involved in federated learning are different organizations (e.g., healthcare or finance) or geographically distributed data centers, i.e., training models on multiple data silos. In the cross-device federated learning scenario, the participating clients are a large number of mobile or IoT devices. This framework will describe how to implement a target detection application by using network Fast R-CNN on MindSpore Federated cross-silo federated framework. -The full script to launch cross-silo federated target detection application can be found [here](https://gitee.com/mindspore/federated/tree/master/example/cross_silo_femnist). +The full script to launch cross-silo federated target detection application can be found [here](https://gitee.com/mindspore/federated/tree/master/example/cross_silo_faster_rcnn). ## Preparation @@ -12,7 +12,7 @@ This tutorial deploy the cross-silo federated target detection task based on the Since the original COCO dataset is in json file format, the target detection script provided by cross-silo federated learning framework only supports input data in MindRecord format. You can convert the json file to MindRecord format file according to the following steps. -- Configure the following parameters in the configuration file[default_config.yaml](https://gitee.com/mindspore/federated/blob/master/tests/st/cross_silo_faster_rcnn/default_config.yaml): +- Configure the following parameters in the configuration file[default_config.yaml](https://gitee.com/mindspore/federated/blob/master/example/cross_silo_faster_rcnn/default_config.yaml): - `mindrecord_dir` @@ -30,7 +30,7 @@ Since the original COCO dataset is in json file format, the target detection scr instance_set: "./datasets/coco_split/split_100/train_0.json" ``` -- Run the script [generate_mindrecord.py](https://gitee.com/mindspore/federated/tree/master/tests/st/cross_silo_faster_rcnn/generate_mindrecord.py) to generate MindRecord file according to `train_0.json`, saved in the `mindrecord_dir` path. +- Run the script [generate_mindrecord.py](https://gitee.com/mindspore/federated/blob/master/example/cross_silo_faster_rcnn/generate_mindrecord.py) to generate MindRecord file according to `train_0.json`, saved in the `mindrecord_dir` path. ## Starting the Cross-Silo Federated Mission @@ -42,7 +42,7 @@ Currently the federated learning framework is only supported for deployment in L ## Starting Mission -Refer to [example](https://gitee.com/mindspore/federated/tree/master/tests/st/cross_silo_faster_rcnn) to start the cluster. The reference example directory structure is as follows: +Refer to [example](https://gitee.com/mindspore/federated/tree/master/example/cross_silo_faster_rcnn) to start the cluster. The reference example directory structure is as follows: ```text cross_silo_faster_rcnn @@ -88,7 +88,7 @@ cross_silo_faster_rcnn model.train(config.client_epoch_num, dataset, callbacks=cb, dataset_sink_mode=False) # Set dataset_sink_mode=False to record the loss value of each step ``` -2. Set the following parameters in configuration file [default_config.yaml](https://gitee.com/mindspore/federated/tree/master/tests/st/cross_silo_faster_rcnn/default_config.yaml): +2. Set the following parameters in configuration file [default_config.yaml](https://gitee.com/mindspore/federated/blob/master/example/cross_silo_faster_rcnn/default_config.yaml): - `pre_trained` @@ -110,7 +110,7 @@ cross_silo_faster_rcnn python run_sched.py --yaml_config="default.yaml" --scheduler_manage_address="10.113.216.40:18019" ``` - For the detailed implementation, see [run_sched.py](https://gitee.com/mindspore/federated/tree/master/example/cross_device_lenet_femnist/run_cross_silo_femnist_server.py). + For the detailed implementation, see [run_sched.py](https://gitee.com/mindspore/federated/blob/master/tests/st/cross_device_cloud/run_sched.py). The following print represents a successful starting: @@ -127,7 +127,7 @@ cross_silo_faster_rcnn python run_server.py --yaml_config="default.yaml" --tcp_server_ip="10.113.216.40" --checkpoint_dir="fl_ckpt" --local_server_num=4 --http_server_address="10.113.216.40:6668" ``` - The above command is equivalent to starting four `Server` processes, each with a federated learning service port of `6668`, `6669`, `6670` and `6671`, as detailed in [run_server.py](https://gitee.com/mindspore/ federated/tree/master/example/cross_device_lenet_femnist/run_server.py). + The above command is equivalent to starting four `Server` processes, each with a federated learning service port of `6668`, `6669`, `6670` and `6671`, as detailed in [run_server.py](https://gitee.com/mindspore/federated/tree/master/example/cross_device_lenet_femnist/run_server.py). The following print represents a successful starting: @@ -146,7 +146,7 @@ cross_silo_faster_rcnn python run_cross_silo_fasterrcnn_worker.py --worker_num=2 --dataset_path datasets/coco_split/split_100 --http_server_address=10.113.216.40:6668 ``` - For the detailed implementation, see [run_cross_silo_femnist_worker.py](https://gitee.com/mindspore/federated/tree/master/tests/st/cross_silo_faster_rcnn/run_cross_silo_femnist_worker.py). + For the detailed implementation, see [run_cross_silo_femnist_worker.py](https://gitee.com/mindspore/federated/blob/master/example/cross_silo_faster_rcnn/run_cross_silo_fasterrcnn_worker.py). As the above command, `--worker_num=2` means starting two clients, and the datasets used by the two clients are `datasets/coco_split/split_100/mindrecord_0` and `datasets/coco_split/split_100/mindrecord_1`. Please prepare the required datasets for the corresponding clients according to the `pre-task preparation` tutorial. @@ -158,7 +158,7 @@ cross_silo_faster_rcnn Then it means that cross-silo federated is started successfully and `worker_0` is training. Other workers can be viewed in a similar way. - Please refer to [yaml configuration notes](https://gitee.com/mindspore/federated/blob/master/docs/federated_server_yaml.md#) for the description of parameter configuration in the above script. + Please refer to [yaml configuration notes](https://www.mindspore.cn/federated/docs/en/master/horizontal/federated_server_yaml.html) for the description of parameter configuration in the above script. ### Viewing the Log @@ -208,7 +208,7 @@ If you want to exit in the middle, the following command is available: python finish_cloud.py --redis_port=2345 ``` -For the detailed implementation, see [finish_cloud.py](https://gitee.com/mindspore/federated/blob/master/example/cross_device_lenet_femnist/finish_cloud.py). +For the detailed implementation, see [finish_cloud.py](https://gitee.com/mindspore/federated/blob/master/tests/st/cross_device_cloud/finish_cloud.py). Or when the training task is finished, the cluster exits automatically, no need to close it manually. diff --git a/docs/federated/docs/source_en/sentiment_classification_application.md b/docs/federated/docs/source_en/sentiment_classification_application.md index 95a5b9890a..418b079797 100644 --- a/docs/federated/docs/source_en/sentiment_classification_application.md +++ b/docs/federated/docs/source_en/sentiment_classification_application.md @@ -49,7 +49,7 @@ mobile/models/ The ALBERT language model[1] is used in federated learning. The ALBERT model on the client includes the embedding layer, encoder layer, and classifier layer. -For details about the network definition, see [source code](https://gitee.com/mindspore/mindspore/blob/master/tests/st/fl/mobile/src/model.py). +For details about the network definition, see [source code](https://gitee.com/mindspore/federated/blob/master/tests/st/network/albert.py). ### Generating a Device-Side Model File diff --git a/docs/federated/docs/source_en/split_pangu_alpha_application.md b/docs/federated/docs/source_en/split_pangu_alpha_application.md index 35eca8754e..7ca182d829 100644 --- a/docs/federated/docs/source_en/split_pangu_alpha_application.md +++ b/docs/federated/docs/source_en/split_pangu_alpha_application.md @@ -168,7 +168,7 @@ MindSpore Federated Vertical Federated Learning Framework uses FLModel (see [Ver According to the chain rule of gradient calculation, the subnetwork located at the backstream of the global network needs to calculate the gradient value of its output tensor relative to the input tensor, i.e., the gradient weighting coefficient or sensitivity, to be passed to the sub-network located at the upstream of the global network for its training parameter update. -MindSpore Federated uses the `GradOperation` operator to complete the above gradient weighting coefficient or sensitivity calculation process. The developer needs to describe the `GradOperation` operator used to calculate the gradient weighting coefficients in the yaml configuration file. Taking Head of participant A in this application practice as an example, [sample code](https://gitee.com/mindspore/federated/blob/master/example/splitnn_pangu_alpha/Head.yaml) is as follows: +MindSpore Federated uses the `GradOperation` operator to complete the above gradient weighting coefficient or sensitivity calculation process. The developer needs to describe the `GradOperation` operator used to calculate the gradient weighting coefficients in the yaml configuration file. Taking Head of participant A in this application practice as an example, [sample code](https://gitee.com/mindspore/federated/blob/master/example/splitnn_pangu_alpha/head.yaml) is as follows: ```yaml grad_scalers: diff --git a/docs/federated/docs/source_zh_cn/interface_description_federated_client.md b/docs/federated/docs/source_zh_cn/interface_description_federated_client.md index d833b5e210..65eae08d3e 100644 --- a/docs/federated/docs/source_zh_cn/interface_description_federated_client.md +++ b/docs/federated/docs/source_zh_cn/interface_description_federated_client.md @@ -11,7 +11,7 @@ | 参数名称 | 参数类型 | 是否必须 | 描述信息 | 备注 | | -------------------- | -------------------------- | -------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | dataMap | Map/> | Y | 联邦学习数据集路径 | Map/>类型的数据集，map中key为RunType枚举类型，value为对应的数据集列表，key为RunType.TRAINMODE时代表对应的value为训练相关的数据集列表，key为RunType.EVALMODE时代表对应的value为验证相关的数据集列表， key为RunType.INFERMODE时代表对应的value为推理相关的数据集列表。 | -| flName | String | Y | 联邦学习使用的模型脚本包路径 | 我们提供了两个类型的模型脚本供大家参考（[有监督情感分类任务](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert))、[LeNet图片分类任务](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)），对于有监督情感分类任务，该参数可设置为所提供的脚本文件[AlBertClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java) 的包路径`com.mindspore.flclient.demo.albert.AlbertClient`；对于LeNet图片分类任务，该参数可设置为所提供的脚本文件[LenetClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java) 的包路径`com.mindspore.flclient.demo.lenet.LenetClient`。同时，用户可参考这两个类型的模型脚本，自定义模型脚本，然后将该参数设置为自定义的模型文件ModelClient.java（需继承于类[Client.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/model/Client.java)）的包路径即可。 | +| flName | String | Y | 联邦学习使用的模型脚本包路径 | 我们提供了两个类型的模型脚本供大家参考（[有监督情感分类任务](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert)）、[LeNet图片分类任务](https://gitee.com/mindspore/federated/tree/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet)），对于有监督情感分类任务，该参数可设置为所提供的脚本文件[AlBertClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/albert/AlbertClient.java) 的包路径`com.mindspore.flclient.demo.albert.AlbertClient`；对于LeNet图片分类任务，该参数可设置为所提供的脚本文件[LenetClient.java](https://gitee.com/mindspore/federated/blob/master/example/quick_start_flclient/src/main/java/com/mindspore/flclient/demo/lenet/LenetClient.java) 的包路径`com.mindspore.flclient.demo.lenet.LenetClient`。同时，用户可参考这两个类型的模型脚本，自定义模型脚本，然后将该参数设置为自定义的模型文件ModelClient.java（需继承于类[Client.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/model/Client.java)）的包路径即可。 | | trainModelPath | String | Y | 联邦学习使用的训练模型路径，为.ms文件的绝对路径 | 建议将路径设置到训练App自身目录下，保护模型本身的数据访问安全性。 | | inferModelPath | String | Y | 联邦学习使用的推理模型路径，为.ms文件的绝对路径 | 对于普通联邦学习模式（训练和推理使用同一个模型），该参数需设置为与trainModelPath相同；对于混合学习模式（训练和推理使用不同的模型，且云侧也包含训练过程），该参数设置为实际的推理模型路径。建议将路径设置到训练App自身目录下，保护模型本身的数据访问安全性。 | | sslProtocol | String | N | 端云HTTPS通信所使用的TLS协议版本 | 设置了白名单，目前只支持"TLSv1.3"或者"TLSv1.2"。非必须设置，默认值为"TLSv1.2"。只在HTTPS通信场景中使用。 | @@ -24,7 +24,7 @@ | threadNum | int | N | 联邦学习训练和推理时使用的线程数 | 默认值为1 | | cpuBindMode | BindMode | N | 联邦学习训练和推理时线程所需绑定的cpu内核 | BindMode枚举类型，其中BindMode.NOT_BINDING_CORE代表不绑定内核，由系统自动分配，BindMode.BIND_LARGE_CORE代表绑定大核，BindMode.BIND_MIDDLE_CORE代表绑定中核。默认值为BindMode.NOT_BINDING_CORE。 | | batchSize | int | Y | 联邦学习训练和推理时使用的单步训练样本数，即batch size | 需与模型的输入数据的batch size保持一致。 | -| iflJobResultCallback | IFLJobResultCallback | N | 联邦学习回调函数对象 | 用户可根据实际场景所需，实现工程中接口类[IFLJobResultCallback.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/IFLJobResultCallback.java))的具体方法后，作为回调函数对象设置到联邦学习任务中。我们提供了一个简单的实现用例[FLJobResultCallback.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/IFLJobResultCallback.java)作为该参数默认值。 | +| iflJobResultCallback | IFLJobResultCallback | N | 联邦学习回调函数对象 | 用户可根据实际场景所需，实现工程中接口类[IFLJobResultCallback.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/IFLJobResultCallback.java)的具体方法后，作为回调函数对象设置到联邦学习任务中。我们提供了一个简单的实现用例[FLJobResultCallback.java](https://gitee.com/mindspore/federated/blob/master/mindspore_federated/device_client/src/main/java/com/mindspore/flclient/IFLJobResultCallback.java)作为该参数默认值。 | 注意1，当使用http通信时，可能会存在通信安全风险，请知悉。 diff --git a/docs/federated/docs/source_zh_cn/object_detection_application_in_cross_silo.md b/docs/federated/docs/source_zh_cn/object_detection_application_in_cross_silo.md index ce21fcbb15..4551188a13 100644 --- a/docs/federated/docs/source_zh_cn/object_detection_application_in_cross_silo.md +++ b/docs/federated/docs/source_zh_cn/object_detection_application_in_cross_silo.md @@ -36,7 +36,7 @@ ### 安装MindSpore和Mindspore Federated -包括源码和下载发布版两种方式，支持CPU、GPU硬件平台，根据硬件平台选择安装即可。安装步骤可参考[MindSpore安装指南](https://www.mindspore.cn/install)， [Mindspore Federated安装指南](https://www.mindspore.cn/federated/docs/zh-CN/master/index.html)。 +包括源码和下载发布版两种方式，支持CPU、GPU硬件平台，根据硬件平台选择安装即可。安装步骤可参考[MindSpore安装指南](https://www.mindspore.cn/install)，[Mindspore Federated安装指南](https://www.mindspore.cn/federated/docs/zh-CN/master/index.html)。 目前联邦学习框架只支持Linux环境中部署，cross-silo联邦学习框架需要MindSpore版本号>=1.5.0。 @@ -158,7 +158,7 @@ cross_silo_faster_rcnn 则说明云云联邦启动成功，`worker_0`正在训练，其他worker可通过类似方式查看。 -以上脚本中参数配置说明请参考[yaml配置说明](https://www.mindspore.cn/federated/docs/zh-CN/master/horizontal/federated_server_yaml.html)。 + 以上脚本中参数配置说明请参考[yaml配置说明](https://www.mindspore.cn/federated/docs/zh-CN/master/horizontal/federated_server_yaml.html)。 ### 日志查看 -- Gitee