diff --git a/docs/mindformers/docs/source_en/env_variables.md b/docs/mindformers/docs/source_en/env_variables.md
index 82af089a3c03e0571a7e6fd900801d723703709a..615a8fdf4463f4e270df02295949aed6fd187b77 100644
--- a/docs/mindformers/docs/source_en/env_variables.md
+++ b/docs/mindformers/docs/source_en/env_variables.md
@@ -26,18 +26,18 @@ The following environment variables are supported by MindSpore Transformers.
## Other Variables
-| Variables Names | Default | Interpretations | Descriptions | Application Scenarios |
-|------------------------------------|-------------|-----------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ---------------------------------------------------------------------------------------------------------- |
-| **RUN_MODE** | predict | Set the running mode. | `predict`: inference
`finetune`: Fine-tuning
`train`: Training
`eval`: Evaluation |
-| **USE_ROPE_SELF_DEFINE** | true | Whether to enable ROPE fusion operator. | `true`: enable ROPE fusion operator;
`false`: disable ROPE fusion operator. | Enabling the ROPE fusion operator by default can improve the computation efficiency. Except for debugging scenarios, turn it off as needed, and generally do not make special settings. |
-| **MS_ENABLE_INTERNAL_BOOST** | on | Whether to turn on the internal acceleration of the MindSpore framework. | `on`: turn on MindSpore internal acceleration;
`off`: turn off MindSpore internal acceleration. | In order to achieve high-performance inference, this parameter is turned on by default. In cases where debugging or comparing different acceleration strategies is performed, this parameter needs to be turned off to observe the impact on performance. |
-| **MS_GE_ATOMIC_CLEAN_POLICY** | 1 | Whether to clean up the memory occupied by atomic operators in the network. | `0`: centralized cleanup of memory occupied by all atomic operators in the network;
`1`: no centralized memory cleanup, individual zeroing of each atomic operator in the network. | The switch is set to `1` by default, which makes it easy for the user to process each operator individually, allowing operations such as operator memory reuse. Setting it to `0` centrally cleans up the memory occupied by the operators. |
-| **ENABLE_LAZY_INLINE** | 1 | Whether to enable lazy inline. | `0`: turn off lazy inline;
`1`: turn on lazy inline. | Available under mindspore ≥ 2.2.0. It is usually used during pipeline parallelism to improve compilation performance. It is enabled by default and can be configured to be disabled. |
-| **ENABLE_LAZY_INLINE_NO_PIPELINE** | 0 | Whether to enable lazy inline under non-pipeline parallel. | `0`: turn off lazy inline;
`1`: turn on lazy inline. |The lazy inline feature is only enabled in pipeline parallel mode by default. To enable lazy inline in other parallel modes, set this environment variable to 1. |
-| **MS_ASCEND_CHECK_OVERFLOW_MODE** | INFNAN_MODE | Sets the overflow detection mode. | `SATURATION_MODE`: saturation mode, saturates to floating-point extremes (+-MAX) when the calculation overflows;
`INFNAN_MODE`: INF/NAN mode, follows the IEEE 754 standard, and outputs INF/NAN calculations as defined. | In large model tuning, the overflow state is aligned PyTorch and it is recommended to use INFNAN_MODE, i.e. export MS_ASCEND_CHECK_OVERFLOW_MODE=INFNAN_MODE.
Try setting this variable to INFNAN_MODE when encountering persistent overflow problems. |
-| **MF_LOG_SUFFIX** | NA | Set custom suffixes for all log log folders. | Suffix for the log folder. Default: no suffix | Adding a consistent suffix isolates logs across tasks from being overwritten. |
-| **PLOG_REDIRECT_TO_OUTPUT** | False | Controls whether plog logs change storage paths. | `True`: store the logs in the ./output directory;
`False`: Store to the default storage location. | This setting makes it easier to query the plog log. |
-| **MS_ENABLE_FA_FLATTEN** | on | Controls whether support FlashAttention flatten optimization. | `on`: Enable FlashAttention flatten optimization;
`off`: Disable FlashAttention flatten optimization. | Provide a fallback mechanism for models that have not yet been adapted to FlashAttention flatten optimization. |
-| **EXPERIMENTAL_KERNEL_LAUNCH_GROUP** | NA | Control whether to support the batch parallel submission of operators. If supported, enable the parallel submission and configure the number of parallel submissions. | `thread_num`: The number of concurrent threads is not recommended to be increased. The default value is 2;
`kernel_group_num`: Total number of operator groups, 'kernel_group_num/thread_num' groups per thread, default is' 8 '. | This feature will continue to evolve in the future, and the subsequent behavior may change. Currently, only the `deepseek` reasoning scenario is supported, with certain performance optimization, but other models using this feature may deteriorate, and users need to use it with caution, as follows:`export EXPERIMENTAL_KERNEL_LAUNCH_GROUP="thread_num:2,kernel_group_num:8"`. |
-| **FORCE_EAGER** | False | Control whether to disable jit mode. | `False`: Enable jit mode;
`True`: Do not enable jit mode. | Jit compiles functions into a callable MindSpore graph, sets FORCE_EAGER to False to enable jit mode, which can generate performance benefits. Currently, only inference mode is supported. |
-| **MS_ENABLE_TFT** | NA | Enable [MindIO TFT](https://www.hiascend.com/document/detail/zh/mindx-dl/600/clusterscheduling/ref/mindiottp/mindiotft001.html) feature. Turn on TTP, UCE, ARF or TRE feature. | The value of the environment variable can be:"{TTP:1,UCE:1,ARF:1,TRE:1}", when using a certain feature, the corresponding field can be configured as "1". | Usage can refer to [High Availability](https://www.mindspore.cn/mindformers/docs/en/dev/feature/high_availability.html). |
+| Variables Names | Default | Interpretations | Descriptions | Application Scenarios |
+|------------------------------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ---------------------------------------------------------------------------------------------------------- |
+| **RUN_MODE** | predict | Set the running mode. | `predict`: inference
`finetune`: Fine-tuning
`train`: Training
`eval`: Evaluation |
+| **USE_ROPE_SELF_DEFINE** | true | Whether to enable ROPE fusion operator. | `true`: enable ROPE fusion operator;
`false`: disable ROPE fusion operator. | Enabling the ROPE fusion operator by default can improve the computation efficiency. Except for debugging scenarios, turn it off as needed, and generally do not make special settings. |
+| **MS_ENABLE_INTERNAL_BOOST** | on | Whether to turn on the internal acceleration of the MindSpore framework. | `on`: turn on MindSpore internal acceleration;
`off`: turn off MindSpore internal acceleration. | In order to achieve high-performance inference, this parameter is turned on by default. In cases where debugging or comparing different acceleration strategies is performed, this parameter needs to be turned off to observe the impact on performance. |
+| **MS_GE_ATOMIC_CLEAN_POLICY** | 1 | Whether to clean up the memory occupied by atomic operators in the network. | `0`: centralized cleanup of memory occupied by all atomic operators in the network;
`1`: no centralized memory cleanup, individual zeroing of each atomic operator in the network. | The switch is set to `1` by default, which makes it easy for the user to process each operator individually, allowing operations such as operator memory reuse. Setting it to `0` centrally cleans up the memory occupied by the operators. |
+| **ENABLE_LAZY_INLINE** | 1 | Whether to enable lazy inline. | `0`: turn off lazy inline;
`1`: turn on lazy inline. | Available under mindspore ≥ 2.2.0. It is usually used during pipeline parallelism to improve compilation performance. It is enabled by default and can be configured to be disabled. |
+| **ENABLE_LAZY_INLINE_NO_PIPELINE** | 0 | Whether to enable lazy inline under non-pipeline parallel. | `0`: turn off lazy inline;
`1`: turn on lazy inline. |The lazy inline feature is only enabled in pipeline parallel mode by default. To enable lazy inline in other parallel modes, set this environment variable to 1. |
+| **MS_ASCEND_CHECK_OVERFLOW_MODE** | INFNAN_MODE | Sets the overflow detection mode. | `SATURATION_MODE`: saturation mode, saturates to floating-point extremes (+-MAX) when the calculation overflows;
`INFNAN_MODE`: INF/NAN mode, follows the IEEE 754 standard, and outputs INF/NAN calculations as defined. | In large model tuning, the overflow state is aligned PyTorch and it is recommended to use INFNAN_MODE, i.e. export MS_ASCEND_CHECK_OVERFLOW_MODE=INFNAN_MODE.
Try setting this variable to INFNAN_MODE when encountering persistent overflow problems. |
+| **MF_LOG_SUFFIX** | NA | Set custom suffixes for all log log folders. | Suffix for the log folder. Default: no suffix | Adding a consistent suffix isolates logs across tasks from being overwritten. |
+| **PLOG_REDIRECT_TO_OUTPUT** | False | Controls whether plog logs change storage paths. | `True`: store the logs in the ./output directory;
`False`: Store to the default storage location. | This setting makes it easier to query the plog log. |
+| **MS_ENABLE_FA_FLATTEN** | on | Controls whether support FlashAttention flatten optimization. | `on`: Enable FlashAttention flatten optimization;
`off`: Disable FlashAttention flatten optimization. | Provide a fallback mechanism for models that have not yet been adapted to FlashAttention flatten optimization. |
+| **EXPERIMENTAL_KERNEL_LAUNCH_GROUP** | NA | Control whether to support the batch parallel submission of operators. If supported, enable the parallel submission and configure the number of parallel submissions. | `thread_num`: The number of concurrent threads is not recommended to be increased. The default value is 2;
`kernel_group_num`: Total number of operator groups, 'kernel_group_num/thread_num' groups per thread, default is' 8 '. | This feature will continue to evolve in the future, and the subsequent behavior may change. Currently, only the `deepseek` reasoning scenario is supported, with certain performance optimization, but other models using this feature may deteriorate, and users need to use it with caution, as follows:`export EXPERIMENTAL_KERNEL_LAUNCH_GROUP="thread_num:2,kernel_group_num:8"`. |
+| **FORCE_EAGER** | False | Control whether to disable jit mode. | `False`: Enable jit mode;
`True`: Do not enable jit mode. | Jit compiles functions into a callable MindSpore graph, sets FORCE_EAGER to False to enable jit mode, which can generate performance benefits. Currently, only inference mode is supported. |
+| **MS_ENABLE_TFT** | NA | Enable [MindIO TFT](https://www.hiascend.com/document/detail/zh/mindx-dl/600/clusterscheduling/ref/mindiottp/mindiotft001.html) feature. Turn on TTP, UCE, ARF, TRE or TSP feature. | The value of the environment variable can be:"{TTP:1,UCE:1,ARF:1,TRE:1,TSP:1}", when using a certain feature, the corresponding field can be configured as "1". | Usage can refer to [High Availability](https://www.mindspore.cn/mindformers/docs/en/dev/feature/high_availability.html). |
diff --git a/docs/mindformers/docs/source_en/feature/high_availability.md b/docs/mindformers/docs/source_en/feature/high_availability.md
index 9b5ae5bafb740d900c83a12ee960f632154ba46d..08fb1757a30b4548a382068a7a0748303507454c 100644
--- a/docs/mindformers/docs/source_en/feature/high_availability.md
+++ b/docs/mindformers/docs/source_en/feature/high_availability.md
@@ -187,7 +187,7 @@ This section demonstrates the use of the end-of-life CKPT using Llama2-13B train
```bash
export MINDIO_FOR_MINDSPORE=1
- export MS_ENABLE_TFT="{TTP:1,UCE:1,ARF:1}"
+ export MS_ENABLE_TFT="{TTP:1,UCE:1,ARF:1,TSP:1}"
export MS_TFT_IP=127.0.0.1
export MS_TFT_PORT=30051
diff --git a/docs/mindformers/docs/source_zh_cn/env_variables.md b/docs/mindformers/docs/source_zh_cn/env_variables.md
index e58a500de1487620a79234cac67ee3a85d088d37..5515c8984b5abb617dab85ede6d8381ea3ff3d32 100644
--- a/docs/mindformers/docs/source_zh_cn/env_variables.md
+++ b/docs/mindformers/docs/source_zh_cn/env_variables.md
@@ -26,18 +26,18 @@
## 其他变量
-| 变量名称 | 默认值 | 解释 | 说明 | 应用场景 |
-| ---------------------------------- |-------------|------------------------------------|------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
-| **RUN_MODE** | predict | 设置运行模式。 | `predict`:推理;
`finetune`:微调;
`train`:训练;
`eval`:评测。 | |
-| **USE_ROPE_SELF_DEFINE** | true | 是否使用 ROPE 融合算子。 | `true`:使用 ROPE 融合算子;
`false`:不使用 ROPE 融合算子。 | 默认开启 ROPE 融合算子可以提升计算效率。除调试场景,根据需要进行关闭,一般不作特别设置。 |
-| **MS_ENABLE_INTERNAL_BOOST** | on | 是否打开 MindSpore 框架的内部加速功能。 | `on`:开启 MindSpore 内部加速;
`off`:关闭 MindSpore 内部加速。 | 为了实现高性能推理,该配置默认开启。在进行调试或对比不同加速策略的情况下,需要关闭此参数以观察对性能的影响。 |
-| **MS_GE_ATOMIC_CLEAN_POLICY** | 1 | 是否集中清理网络中 atomic 算子占用的内存。 | `0`:集中清理网络中所有 atomic 算子占用的内存;
`1`:不集中清理内存,对网络中每一个 atomic 算子进行单独清零。 | 开关默认设置为`1`,方便用户对每个算子进行单独处理,可以进行算子内存复用等操作。设置为`0`后,集中清理算子所占内存。 |
-| **ENABLE_LAZY_INLINE** | 1 | 是否开启 lazy inline。 | `0`:关闭 lazy inline;
`1`:开启 lazy inline。 | 此特性在 mindspore≥2.2.0 下适用。通常在 pipeline 并行时使用以提高编译性能。默认开启,可配置关闭。 |
-| **ENABLE_LAZY_INLINE_NO_PIPELINE** | 0 | 是否开启在非 pipeline 并行下的 lazy inline。 | `0`:关闭 lazy inline;
`1`:开启 lazy inline。 | lazy inline 特性默认仅在 pipeline 并行模式下开启。如需在其他并行模式下使能 lazy inline,可将该环境变量设置为 1。 |
-| **MS_ASCEND_CHECK_OVERFLOW_MODE** | INFNAN_MODE | 设置浮点计算结果输出模式。 | `SATURATION_MODE`:饱和模式,计算出现溢出时,饱和为浮点数极值(+-MAX);
`INFNAN_MODE`:INF/NAN 模式,遵循 IEEE 754 标准,根据定义输出 INF/NAN 的计算结果。 | 在大模型调优中,溢出状态对齐 PyTorch 方式,建议使用 INFNAN_MODE,即 export MS_ASCEND_CHECK_OVERFLOW_MODE=INFNAN_MODE。
遇到持续溢出问题时可尝试设置此变量为 INFNAN_MODE。 |
-| **MF_LOG_SUFFIX** | NA | 设置所有 log 日志文件夹的自定义后缀。 | log 文件夹的后缀。默认值:无后缀 | 添加一致的后缀,可以隔离各个任务的日志,不会被覆写。 |
-| **PLOG_REDIRECT_TO_OUTPUT** | False | 控制 plog 日志是否改变存储路径。 | `True`:存储到./output 目录下;
`False`: 存储到默认存储位置。 | 设置之后方便用户查询 plog 日志。 |
-| **MS_ENABLE_FA_FLATTEN** | on | 控制 是否支持 FlashAttention flatten 优化。 | `on`:启用 FlashAttention flatten 优化;
`off`: 禁用 FlashAttention flatten 优化。 | 对于还未适配FlashAttention flatten 优化的模型提供回退机制。 |
-| **EXPERIMENTAL_KERNEL_LAUNCH_GROUP** | NA | 控制是否支持算子批量并行下发,支持开启并行下发,并配置并行数 | `thread_num`: 并发线程数,一般不建议增加,默认值为`2`;
`kernel_group_num`: 算子分组总数量,每线程`kernel_group_num/thread_num`个组,默认值为`8`。 | 该特性后续还会继续演进,后续行为可能会有变更,当前仅支持`deepseek`推理场景,有一定的性能优化,但是其他模型使用该特性可能会有劣化,用户需要谨慎使用,使用方法如下:`export EXPERIMENTAL_KERNEL_LAUNCH_GROUP="thread_num:2,kernel_group_num:8"`。 |
-| **FORCE_EAGER** | False | 控制是否**不开启**jit模式。 | `False`: 开启jit模式;
`True`: 不开启jit模式。 | Jit将函数编译成一张可调用的MindSpore图,设置FORCE_EAGER为False开启jit模式,可以获取性能收益,当前仅支持推理模式。 |
-| **MS_ENABLE_TFT** | NA | 使能 [MindIO TFT](https://www.hiascend.com/document/detail/zh/mindx-dl/600/clusterscheduling/ref/mindiottp/mindiotft001.html) 特性,表示启用 TTP、UCE、ARF 或 TRE 功能。 | 取值为"{TTP:1,UCE:1,ARF:1,TRE:1}",使用某一功能时,可将对应字段配置为"1"。 | 使用方式可以参考[高可用特性](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/feature/high_availability.html)。 |
+| 变量名称 | 默认值 | 解释 | 说明 | 应用场景 |
+| ---------------------------------- |-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|
+| **RUN_MODE** | predict | 设置运行模式。 | `predict`:推理;
`finetune`:微调;
`train`:训练;
`eval`:评测。 | |
+| **USE_ROPE_SELF_DEFINE** | true | 是否使用 ROPE 融合算子。 | `true`:使用 ROPE 融合算子;
`false`:不使用 ROPE 融合算子。 | 默认开启 ROPE 融合算子可以提升计算效率。除调试场景,根据需要进行关闭,一般不作特别设置。 |
+| **MS_ENABLE_INTERNAL_BOOST** | on | 是否打开 MindSpore 框架的内部加速功能。 | `on`:开启 MindSpore 内部加速;
`off`:关闭 MindSpore 内部加速。 | 为了实现高性能推理,该配置默认开启。在进行调试或对比不同加速策略的情况下,需要关闭此参数以观察对性能的影响。 |
+| **MS_GE_ATOMIC_CLEAN_POLICY** | 1 | 是否集中清理网络中 atomic 算子占用的内存。 | `0`:集中清理网络中所有 atomic 算子占用的内存;
`1`:不集中清理内存,对网络中每一个 atomic 算子进行单独清零。 | 开关默认设置为`1`,方便用户对每个算子进行单独处理,可以进行算子内存复用等操作。设置为`0`后,集中清理算子所占内存。 |
+| **ENABLE_LAZY_INLINE** | 1 | 是否开启 lazy inline。 | `0`:关闭 lazy inline;
`1`:开启 lazy inline。 | 此特性在 mindspore≥2.2.0 下适用。通常在 pipeline 并行时使用以提高编译性能。默认开启,可配置关闭。 |
+| **ENABLE_LAZY_INLINE_NO_PIPELINE** | 0 | 是否开启在非 pipeline 并行下的 lazy inline。 | `0`:关闭 lazy inline;
`1`:开启 lazy inline。 | lazy inline 特性默认仅在 pipeline 并行模式下开启。如需在其他并行模式下使能 lazy inline,可将该环境变量设置为 1。 |
+| **MS_ASCEND_CHECK_OVERFLOW_MODE** | INFNAN_MODE | 设置浮点计算结果输出模式。 | `SATURATION_MODE`:饱和模式,计算出现溢出时,饱和为浮点数极值(+-MAX);
`INFNAN_MODE`:INF/NAN 模式,遵循 IEEE 754 标准,根据定义输出 INF/NAN 的计算结果。 | 在大模型调优中,溢出状态对齐 PyTorch 方式,建议使用 INFNAN_MODE,即 export MS_ASCEND_CHECK_OVERFLOW_MODE=INFNAN_MODE。
遇到持续溢出问题时可尝试设置此变量为 INFNAN_MODE。 |
+| **MF_LOG_SUFFIX** | NA | 设置所有 log 日志文件夹的自定义后缀。 | log 文件夹的后缀。默认值:无后缀 | 添加一致的后缀,可以隔离各个任务的日志,不会被覆写。 |
+| **PLOG_REDIRECT_TO_OUTPUT** | False | 控制 plog 日志是否改变存储路径。 | `True`:存储到./output 目录下;
`False`: 存储到默认存储位置。 | 设置之后方便用户查询 plog 日志。 |
+| **MS_ENABLE_FA_FLATTEN** | on | 控制 是否支持 FlashAttention flatten 优化。 | `on`:启用 FlashAttention flatten 优化;
`off`: 禁用 FlashAttention flatten 优化。 | 对于还未适配FlashAttention flatten 优化的模型提供回退机制。 |
+| **EXPERIMENTAL_KERNEL_LAUNCH_GROUP** | NA | 控制是否支持算子批量并行下发,支持开启并行下发,并配置并行数 | `thread_num`: 并发线程数,一般不建议增加,默认值为`2`;
`kernel_group_num`: 算子分组总数量,每线程`kernel_group_num/thread_num`个组,默认值为`8`。 | 该特性后续还会继续演进,后续行为可能会有变更,当前仅支持`deepseek`推理场景,有一定的性能优化,但是其他模型使用该特性可能会有劣化,用户需要谨慎使用,使用方法如下:`export EXPERIMENTAL_KERNEL_LAUNCH_GROUP="thread_num:2,kernel_group_num:8"`。 |
+| **FORCE_EAGER** | False | 控制是否**不开启**jit模式。 | `False`: 开启jit模式;
`True`: 不开启jit模式。 | Jit将函数编译成一张可调用的MindSpore图,设置FORCE_EAGER为False开启jit模式,可以获取性能收益,当前仅支持推理模式。 |
+| **MS_ENABLE_TFT** | NA | 使能 [MindIO TFT](https://www.hiascend.com/document/detail/zh/mindx-dl/600/clusterscheduling/ref/mindiottp/mindiotft001.html) 特性,表示启用 TTP、UCE、ARF TRE 或 TSP 功能。 | 取值为"{TTP:1,UCE:1,ARF:1,TRE:1,TSP:1}",使用某一功能时,可将对应字段配置为"1"。 | 使用方式可以参考[高可用特性](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/feature/high_availability.html)。 |
diff --git a/docs/mindformers/docs/source_zh_cn/feature/high_availability.md b/docs/mindformers/docs/source_zh_cn/feature/high_availability.md
index 9f887caba28d9f5872581e28e9c2544065dc93ac..49fcc1789dfa37f4c094a876fbb23e529cde86e7 100644
--- a/docs/mindformers/docs/source_zh_cn/feature/high_availability.md
+++ b/docs/mindformers/docs/source_zh_cn/feature/high_availability.md
@@ -186,7 +186,7 @@ YAML配置包含两部分:临终 CKPT 的保存及恢复配置和卡间副本
```bash
export MINDIO_FOR_MINDSPORE=1
- export MS_ENABLE_TFT="{TTP:1,UCE:1,ARF:1}"
+ export MS_ENABLE_TFT="{TTP:1,UCE:1,ARF:1,TSP:1}"
export MS_TFT_IP=127.0.0.1
export MS_TFT_PORT=30051
diff --git a/docs/mindspore/source_en/api_python/env_var_list.rst b/docs/mindspore/source_en/api_python/env_var_list.rst
index 10e9f51a40a8cd0d80a827b28c3b45956d0a5822..fc58157be30fe93fa1f4bd856d92f0a19d845048 100644
--- a/docs/mindspore/source_en/api_python/env_var_list.rst
+++ b/docs/mindspore/source_en/api_python/env_var_list.rst
@@ -935,7 +935,7 @@ Third-party Library
* - MS_ENABLE_TFT
- Enable `MindIO TFT `_ feature. Turn on TTP, UCE, TRE or ARF feature.
- String
- - "{TTP:1,UCE:1,TRE:1,ARF:1}". TTP (Try To Persist): End of life CKPT, UCE (Uncorrectable Memory Error): Fault tolerance and recovery, TRE(Training Result Error): Restoring training result exceptions, ARF (Air Refuelling): Process level rescheduling and recovery feature. The four features can be enabled separately. If you only want to enable one of them, set the corresponding value to 1. Other values: MindIO TFT not turned on. (When using UCE or ARF, TTP is enabled by default. TRE can not be used with UCE or ARF feature.)
+ - "{TTP:1,UCE:1,TRE:1,ARF:1,TSP:1}". TTP (Try To Persist): End of life CKPT, UCE (Uncorrectable Memory Error): Fault tolerance and recovery, TRE(Training Result Error): Restoring training result exceptions, ARF (Air Refuelling), TSP(Training step pause): Process level rescheduling and recovery feature. The four features can be enabled separately. If you only want to enable one of them, set the corresponding value to 1. Other values: MindIO TFT not turned on. (When using UCE or ARF, TTP is enabled by default. TRE can not be used with UCE or ARF feature.)
- Graph mode can only be enabled on the Ascend backend and jit_level is set to "O0" or "O1".
* - MS_TFT_IP
- The IP address where the MindIO controller thread is located for processor connections.
diff --git a/docs/mindspore/source_zh_cn/api_python/env_var_list.rst b/docs/mindspore/source_zh_cn/api_python/env_var_list.rst
index 2068ef22227099441a7b8e8422558b0e6215c38c..2ac93724d82ccacf55548e53e91a10af412af73d 100644
--- a/docs/mindspore/source_zh_cn/api_python/env_var_list.rst
+++ b/docs/mindspore/source_zh_cn/api_python/env_var_list.rst
@@ -931,7 +931,7 @@ Dump调试
* - MS_ENABLE_TFT
- 使能 `MindIO TFT `_ 特性,表示启用 TTP、UCE、TRE 或 ARF 功能。
- String
- - "{TTP:1,UCE:1,ARF:1}"。TTP (Try To Persist):临终 CKPT 功能、UCE (Uncorrectable Memory Error):UCE 故障容错恢复功能、TRE (Training Result Error):训练结果异常恢复功能、ARF (Air Refuelling):进程级重调度恢复功能。四个特性可以分开使能,如果只想启用其中的某一个功能,则将对应的值设置为 1 即可。其他值:未开启MindIO TFT。(开启 UCE 或者 ARF 功能时,默认开启 TTP 功能。TRE 功能不可以与 UCE 或 ARF 功能同时使用。)
+ - "{TTP:1,UCE:1,ARF:1, TSP:1}"。TTP (Try To Persist):临终 CKPT 功能、UCE (Uncorrectable Memory Error):UCE 故障容错恢复功能、TRE (Training Result Error):训练结果异常恢复功能、ARF (Air Refuelling):进程级重调度恢复功能. TSP(Training Step Pause):训练迭代暂停。五个特性可以分开使能,如果只想启用其中的某一个功能,则将对应的值设置为 1 即可。其他值:未开启MindIO TFT。(开启 UCE 或者 ARF 功能时,默认开启 TTP 功能。TRE 功能不可以与 UCE 或 ARF 功能同时使用。)
- 仅限在 Ascend 后端开启图模式,且 jit_level 设置为 "O0" 或 "O1"。
* - MS_TFT_IP
- MindIO的controller线程所在IP,供processor链接。