| Environment Variables | -Function | -Type | -Value | -Description | -
|---|---|---|---|---|
| MS_ROLE | -Specifies the role of this process. | -String | -
-
|
- The Worker and Parameter Server processes register with the Scheduler process to complete the networking. | -
| MS_SCHED_HOST | -Specifies the IP address of the Scheduler. | -String | -Legal IP address. | -IPv6 addresses are only supported on `Ascend` platform in current version. | -
| MS_SCHED_PORT | -Specifies the Scheduler binding port number. | -Integer | -Port number in the range of 1024 to 65535. | -- |
| MS_NODE_ID | -Specifies the ID of this process, unique within the cluster. | -String | -Represents the unique ID of this process, which is automatically generated by MindSpore by default. | -
- MS_NODE_ID needs to be set in the following cases. Normally it does not need to be set and is automatically generated by MindSpore:
-
|
-
| MS_WORKER_NUM | -Specifies the number of processes with the role MS_WORKER. | -Integer | -Integer greater than 0. | -- The number of Worker processes started by the user should be equal to the value of this environment variable. If it is less than this value, the networking fails. If it is greater than this value, the Scheduler process will complete the networking according to the order of Worker registration, and the redundant Worker processes will fail to start. - | -
| MS_SERVER_NUM | -Specifies the number of processes with the role MS_PSERVER. | -Integer | -Integer greater than 0. | -Only set in Parameter Server training mode. | -
| MS_WORKER_IP | -Specifies the IP address used for communication and networking between processes. | -String | -Legitimate IP address. | -This environment variable is suggested to be set when using IPv6. But when MS_SCHED_HOST is set to ::1(Representing local loopback interface in IPv6), there's no need to set MS_WORKER_IP because MindSpore will use local loopback interface to communicate by default. | -
| MS_ENABLE_RECOVERY | -Turn on disaster recovery. | -Integer | -1 for on, 0 for off. The default is 0. | -- |
| MS_RECOVERY_PATH | -Persistent path folder. | -String | -Legal user directory. | -The Worker and Scheduler processes perform the necessary persistence during execution, such as node information for restoring the networking and training the intermediate state of the service, and are saved via files. | -
| MS_ENABLE_LCCL | -Whether to use LCCL as communication library. | -Integer | -1 for yes, other values for no. The default is no. | -The LCCL communication library currently only supports single-machine multi-card scenario and must be executed when the graph compilation level is O0. | -
| MS_TOPO_TIMEOUT | -Cluster networking phase timeout time in seconds. | -Integer | -The default is 30 minutes. | -This value represents that all nodes can register to the scheduler within this time window. If the time window is exceeded, registration will fail and if the number of nodes does not meet the requirements, cluster networking will fail. We suggest users to configure this environment variable when the cluster is in large-scale. | -
| MS_NODE_TIMEOUT | -Node heartbeat timeout in seconds。 | -Integer | -The default is 300 seconds. | -This value represents the heartbeat timeout time between the scheduler and the worker. If there are no heartbeat messages within this time window, the cluster will exit abnormally. | -
| MS_RECEIVE_MSG_TIMEOUT | -Node timeout for receiving messages in seconds. | -Integer | -The default is 300 seconds. | -This value represents the timeout window for the node to receive messages from the other end. If there is no message response within the time window, an empty message is returned. | -
| MS_RETRY_INTERVAL_LOWER | -Lower limit of message retry interval between nodes in seconds. | -Integer | -The default is 3 seconds. | -This value represents the lower limit of the time interval between each retry of sending a message by a node. MindSpore randomly selects the value between MS_RETRY_INTERVAL_LOWER and MS_RETRY_INTERVAL_UPPER as the interval time. This variable is used to control the message concurrency of the Scheduler. |
-
| MS_RETRY_INTERVAL_UPPER | -Upper limit of message retry interval between nodes in seconds | -Integer | -The default is 5 seconds. | -This value represents the upper limit of the time interval between each retry of sending a message by a node. MindSpore randomly selects the value between MS_RETRY_INTERVAL_LOWER and MS_RETRY_INTERVAL_UPPER as the interval time. This variable is used to control the message concurrency of the Scheduler. |
-
| MS_DISABLE_HEARTBEAT | -Disable the heartbeat feature between nodes in the cluster. | -Integer | -Heartbeat feature is enabled by default. | -If set to 1, the heartbeat between cluster nodes will be disabled. In this scenario, Scheduler will not detect Workers' exception and will not control the cluster to exit. This variable can reduce the message concurrency of the Scheduler. It is recommended to set this environment variable when using `gdb attach` command for debugging. |
-
| Parameters | -Functions | -Types | -Values | -Instructions | -
|---|---|---|---|---|
| --worker_num | -The total number of Worker processes participating in the distributed task. | -Integer | -An integer greater than 0. The default value is 8. | -The total number of Workers started on all nodes should be equal to this parameter: if the total number is greater than this parameter, the extra Worker processes will fail to register; if the total number is less than this parameter, the cluster will wait for a certain period of timeout before prompting the task to pull up the failed task and exit, and the size of the timeout window can be configured by the parameter cluster_time_out. |
-
| --local_worker_num | -The number of Worker processes pulled up on the current node. | -Integer | -An integer greater than 0. The default value is 8. | -When this parameter is consistent with worker_num, it means that all Worker processes are executed locally. The node_rank value is ignored in this scenario. |
-
| --master_addr | -Specifies the IP address or hostname of the Scheduler. | -String | -Legal IP address or hostname. The default is the IP address 127.0.0.1. | -msrun will automatically detect on which node to pull up the Scheduler process, and users do not need to care. If the corresponding IP address cannot be found or the hostname cannot be resolved by DNS, the training task will pull up and fail. IPv6 addresses are not supported in the current version. If a hostname is input as a parameter, msrun will automatically resolve it to an IP address, which requires the user's environment to support DNS service. |
-
| --master_port | -Specifies the Scheduler binding port number. | -Integer | -Port number in the range 1024 to 65535. The default is 8118. | -- |
| --node_rank | -The index of the current node. | -Integer | -An integer greater than or equal to 0 can be passed in. In case no value is passed, the default value is -1. | -This parameter is ignored in single-machine multi-card scenario. In multi-machine and multi-card scenarios, if this parameter is not set, the rank_id of the Worker process will be assigned automatically; if it is set, the rank_id will be assigned to the Worker process on each node according to the index. If the number of Worker processes per node is different, it is recommended that this parameter not be configured to automatically assign the rank_id. |
-
| --log_dir | -Worker, and Scheduler log output paths. | -String | -Folder path. Defaults to the current directory. | -If the path does not exist, msrun creates the folder recursively. The log format is as follows: for the Scheduler process, the log is named scheduler.log; For Worker process, log name is worker_[rank].log, where rank suffix is the same as the rank_id assigned to the Worker, but they may be inconsistent in multiple-machine and multiple-card scenarios where node_rank is not set. It is recommended that grep -rn "Global rank id" is executed to view rank_id of each Worker. |
-
| --join | -Whether msrun waits for the Worker as well as the Scheduler to exit. | -Bool | -True or False. Default: False. | -If set to False, msrun will exit immediately after pulling up the process and check the logs to confirm that the distributed task is executing properly. If set to True, msrun waits for all processes to exit, collects the exception log and exits. |
-
| --cluster_time_out | -Cluster networking timeout in seconds. | -Integer | -Default: 600 seconds. | -This parameter represents the waiting time in cluster networking. If no worker_num number of Workers register successfully beyond this time window, the task pull-up fails. |
-
| --bind_core | -Enable processes binding CPU cores. | -Bool | -True or False. Default: False. | -If set to True, msrun will evenly allocate CPU cores and bind them to the spawned distributed processes. | -
| --sim_level | -Set single card simulated compilation level. | -Integer | -Default: -1. Disable simulated compilation. | -If this parameter is set, msrun starts only a single process for simulated compilation and does not execute operators. This feature is commonly used to debug large-scale distributed training parallel strategies, and to detect memory and strategy issues in advance. If set to 0, only compile the frontend graph; If set to 1, further compile backend graph compilation and exit during the execution phase |
-
| --sim_rank_id | -rank_id of the simulated process. | -Integer | -Default: 0. | -Set rank id of the simulated process. | -
| --rank_table_file | -rank_table configuration. Only valid on Ascend platform. | -String | -File path of rank_table configuration. Default: empty string. | -This parameter represents the rank_table configuration file on Ascend platform, describing current distributed cluster. Since the rank_table configuration file reflects distributed cluster information at the physical level, when using this configuration, make sure that the Devices visible to the current process are consistent with the rank_table configuration. The Device visible to the current process can be set via the environment variable ASCEND_RT_VISIBLE_DEVICES. |
-
| --worker_log_name | -Specifies the worker log name. | -String | -File name of worker log. Default: worker_[rank].log. |
- This parameter represents support users configure worker log name, and support configure ip and hostname to worker log name by {ip} and {hostname} separately. The suffix of worker log name is rank by default. |
-
| --tail_worker_log | -Enable output worker log to console. | -String | -One or multiple integers associated with the worker process rank_id. Default: -1. | -This parameter represents output all worker logs of the current node to console by default, and supports users specify one or more worker logs output to console when --join=True. This parameter should be in [0, local_worker_num]. |
-
| task_script | -User Python scripts. | -String | -Legal script path. | -Normally, this parameter is the python script path, and msrun will pull up the process as python task_script task_script_args by default.msrun also supports this parameter as pytest. In this scenario the task script and task parameters are passed in the parameter task_script_args. |
-
| task_script_args | -Parameters for the user Python script. | -- | Parameter list. | -For example, msrun --worker_num=8 --local_worker_num=8 train.py --device_target=Ascend --dataset_path=/path/to/dataset |
-
| Environment Variables | -Functions | -Values | -
|---|---|---|
| MS_ROLE | -This process role. | -
- The current version of msrun exports the following two values:
-
|
-
| MS_SCHED_HOST | -The IP address of the user-specified Scheduler. | -Same as parameter --master_addr. |
-
| MS_SCHED_PORT | -User-specified Scheduler binding port number. | -Same as parameter --master_port. |
-
| MS_WORKER_NUM | -The total number of Worker processes specified by the user. | -Same as parameter --worker_num. |
-
| MS_TOPO_TIMEOUT | -Cluster Timeout Time. | -Same as parameter --cluster_time_out. |
-
| RANK_SIZE | -The total number of Worker processes specified by the user. | -Same as parameter --worker_num. |
-
| RANK_ID | -The rank_id assigned to the Worker process. | -In a multi-machine multi-card scenario, if the parameter --node_rank is not set, RANK_ID will only be exported after the cluster is initialized.So to use this environment variable, it is recommended to set the --node_rank parameter correctly. |
-