| 依赖软件 | +版本 | +软件安装指南 | +推荐硬件形态 | +
|---|---|---|---|
| 昇腾NPU驱动 | +Ascend HDK 24.1.RC3 | +《驱动固件安装指南》 | +Atlas 900 A2 PODc | +
| 昇腾NPU固件 | +|||
| Toolkit（开发套件） | +CANN 8.0.RC3 | +《CANN 软件安装指南》 | +|
| Kernel（算子包） | +|||
| PyTorch | +release v6.0.RC3 | +《Ascend Extension for PyTorch 配置与安装》 | +|
| torch_npu插件 | +|||
| apex | +
| 专家并行 | ---expert-model-parallel-size | +--target-expert-model-parallel-size | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 专家并行 | ---expert-model-parallel-size | +--target-expert-model-parallel-size | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 流水并行动态划分 | @@ -1046,189 +1086,161 @@ ModelLink支持Huggingface、Megatron-Legacy以及Megatron-Core之间的权重 --- -## 预训练加速算法与融合算子 +## 预训练加速算法与融合算子 -ModelLink预训练支持张量并行、流水线并行等多种加速算法和融合算子，下表为各种加速特性对应的使能开关： +MindSpeed-LLM预训练支持张量并行、流水线并行等多种加速算法和融合算子：||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 使用场景 | +场景 | 特性名称 | -具体参数 | Mcore | Legacy | +贡献方 | |||
|---|---|---|---|---|---|---|---|---|---|
| PTD并行 | -张量并行 | ---tensor-model-parallel-size | -Yes | -Yes | -|||||
| 流水线并行 | ---pipeline-model-parallel-size | -Yes | -Yes | -||||||
| 流水线并行动态划分 | ---num-layer-list | -Yes | -Yes | +SPTD并行 | +张量并行 | +✅ | +✅ | +【昇腾】 | |
| 虚拟流水并行 | ---num-layers-per-virtual-pipeline-stage | -Yes | -Yes | +流水线并行 | +✅ | +✅ | +【昇腾】 | ||
| 序列并行 | ---sequence-parallel | -Yes | -Yes | +虚拟流水并行 | +✅ | +✅ | +【昇腾】 | ||
| 分布式优化器 | ---use-distributed-optimizer | -Yes | -Yes | +序列并行 | +✅ | +✅ | +【昇腾】 | ||
| 长序列并行 | -长序列并行 | ---context-parallel-size | -Yes | -No | +Ascend Ring Attention 长序列并行 | +✅ | +❌ | +【昇腾】 | |
| 多并行方案 | ---context-parallel-algo | -Yes | -No | +Ulysses 长序列并行 | +✅ | +❌ | +【昇腾】 | ||
| Send/recv掩盖加速 | ---cp-send-recv-overlap | -Yes | -No | +混合长序列并行 | +✅ | +❌ | +【昇腾】 | ||
| MOE | -MOE专家并行 | ---expert-model-parallel-size | -Yes | -No | +MOE | +MOE 专家并行 | +✅ | +❌ | +【昇腾】 |
| MOE重排通信优化 | ---moe-permutation-async-comm | -Yes | -No | +MOE 重排通信优化 | +✅ | +❌ | +【计算研究部】 | ||
| GEMM | ---moe-grouped-gemm | -Yes | -No | +显存优化 | +参数副本复用 | +✅ | +✅ | +【计算算法部】 | |
| 显存优化 | -参数副本复用 | ---reuse-fp32-param | -Yes | -Yes | -|||||
| 激活函数重计算 | ---recompute-activation-function | -Yes | -Yes | -||||||
| Swap Attention | ---swap-attention | -Yes | -Yes | -||||||
| 重计算程度 | ---recompute-granularity | -Yes | -Yes | +||||||
| 分布式优化器 | +✅ | +✅ | +【昇腾】 | ||||||
| 重计算层数 | ---recompute-num-layers | -Yes | -Yes | +Swap Attention | +✅ | +✅ | +【计算研究部】 | ||
| 重计算方法 | ---recompute-method | -Yes | -Yes | +重计算 | +✅ | +✅ | +【计算研究部】 | ||
| PP-Stage重计算 | ---enable-recompute-layers-per-pp-rank | -Yes | -Yes | +融合算子 | +Flash attention | +✅ | +✅ | +【昇腾】 | |
| 融合算子 | -Flash attention | ---use-flash-attn | -Yes | -Yes | +Fused rmsnorm | +✅ | +✅ | +【昇腾】 | |
| Fused rmsnorm | ---use-fused-rmsnorm | -Yes | -Yes | +Fused swiglu | +✅ | +✅ | +【昇腾】 | ||
| Fused swiglu | ---use-fused-swiglu | -Yes | -Yes | +Fused rotary position embedding | +✅ | +✅ | +【昇腾】 | ||
| Fused rotary position embedding | ---use-fused-rotary-pos-emb | -Yes | -Yes | +GMM | +✅ | +❌ | +【昇腾】 | ||
| Sliding window attention | ---sliding-window | -Yes | -Yes | +通信掩盖 | +梯度reduce通算掩盖 | +✅ | +✅ | +【昇腾】 | |
| 通信 | -梯度reduce通算掩盖 | ---overlap-grad-reduce | -Yes | -Yes | +Recompute in advance | +✅ | +❌ | +【昇腾】 | |
| 权重all-gather通算掩盖 | ---overlap-param-gather | -Yes | -No | +权重all-gather通算掩盖 | +✅ | +❌ | +【昇腾】 | ||
| MC2 | ---use-mc2 | -Yes | -Yes | +MC2 | +✅ | +✅ | +【昇腾】 |
| Llama2-7b模型与DeepSpeed微调5个epoch后的loss对比图
-
- |
- Qwen-7b模型与DeepSpeed微调4个epoch后的loss对比图
-
- |
-
|---|
| Question | -ModelLink | -DeepSpeed | -
|---|---|---|
| What is the product of 6 and 2? | -The product of 6 and 2 is 12. | -The product of 6 and 2 is 12. | -
| Look up the boiling point of water. | -The boiling point of water is 100°C (212°F).. | -The boiling point of water is 100 degrees Celsius (212 degrees Fahrenheit). | -
| Generate a poem that expresses joy. | -I open my arms and accept life's challenge, - -Embracing all of life's uncertainty. - -A future of promise awaits, - -One that I can find in my fate. - -A journey full of hope and discovery, - -A journey that will never end in disappointment. - -For I have found the strength to keep going, - -Researching and learning, ever knowing. - -The future is mine, and mine alone, - -A future that I have to own, - -And I will never be alone, - -For I will have the courage to face all of my fears, - -And the joy of life will be mine forever. | -I am so happy, so thankful and blessed, - -To have a heart that's full of cheer, - -My soul is on fire, my spirit runs free, - -My love is warm, my courage is high, - -My world is bright and my outlook is clear, - -My heart is light, my spirit runs free. - -I take each day with a positive outlook, - -My energy is so high, I can't be beat, - -My joy is real, my emotions run deep, - -My soul is full and my heart does soep. - -I am thankful for everything I have, - -My life is precious and my values true, - -My hope is high and my spirit runs free, - -My soul is full and my heart does soep. | -
-## 目录 - - -- [环境安装](#jump1) - * [仓库拉取](#jump1.1) - * [环境搭建](#jump1.2) -- [权重下载及转换](#jump2) - * [权重下载](#jump2.1) - * [权重转换](#jump2.2) -- [数据集准备及处理](#jump3) - * [数据集下载](#jump3.1) - * [数据集处理](#jump3.2) - * [数据集合并](#jump3.3) -- [大模型分布式预训练](#jump4) - * [准备工作](#jump4.1) - * [配置参数](#jump4.2) - * [启动预训练](#jump4.3) -- [大模型分布式指令微调](#jump5) - * [准备工作](#jump5.1) - * [配置微调参数](#jump5.2) - * [启动全参微调](#jump5.3) -- [大模型分布式推理](#jump6) - * [Generate：流式推理](#jump6.1) - * [Chat：指令微调后chat对话](#jump6.2) -- [大模型分布式评估](#jump7) - * [基准评估](#jump7.1) - * [指令微调评估](#jump7.2) - * [LoRA权重评估](#jump7.3) -- [社区BUG列表](#jump8) - ---- - ## 环境安装 【模型开发时推荐使用配套的环境版本】 -| 软件 | [版本](https://www.hiascend.com/zh/) | -|:---------:|:----------------------------------:| -| Python | 3.8 | -| Driver | 在研版本 | -| Firmware | 在研版本 | -| CANN | 在研版本 | -| Torch | 2.1.0、2.2.0 | -| Torch_npu | 在研版本 | +
| 依赖软件 | +版本 | +软件安装指南 | +推荐硬件形态 | +
|---|---|---|---|
| 昇腾NPU驱动 | +Ascend HDK 24.1.RC3 | +《驱动固件安装指南》 | +Atlas 900 A2 PODc | +
| 昇腾NPU固件 | +|||
| Toolkit（开发套件） | +CANN 8.0.RC3 | +《CANN 软件安装指南》 | +|
| Kernel（算子包） | +|||
| PyTorch | +release v6.0.RC3 | +《Ascend Extension for PyTorch 配置与安装》 | +|
| torch_npu插件 | +|||
| apex | +
| Mem. | ||||||
|---|---|---|---|---|---|---|
| ST | -Pretrain | +ST | +Pretrain | Mcore | TP，PP，VPP，重计算，enable_recompute_layers_per_pp_rank | llama2_tp2_pp4_vpp2.sh | @@ -24,12 +24,28 @@
| Mcore | -CP，分布式优化器，reuse_fp32_param，recompute_activation_function, fused_rmsnorm，fused_swiglu，fused_rope，overlap_grad_reduce, overlap_param_gather | +cp_ring，分布式优化器，reuse_fp32_param，recompute_activation_function，fused_rmsnorm，fused_swiglu，fused_rope，overlap_grad_reduce, overlap_param_gather | llama2_tp2_cp4_mem_recompute.sh | Y | Y | Y |
| Mcore | +cp_hybrid，gqa | +chatglm3_gqa_cp8.sh | +Y | +Y | +Y | +|
| Mcore | +swap_attention，recompute_activation_function，enable_recompute_layers_per_pp_rank，reuse_fp32_param | +llama2_tp2_pp4_vpp2_swap.sh | +Y | +Y | +Y | +|
| Mcore | glm_rope, rotary_percent | @@ -156,12 +172,14 @@|||||
| ring_attn | test_ringattn_context_parallel.py | Y | ||||
| ulysses | test_ulysses_context_parallel.py | Y | diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9a1307aeca2e4639a1e6807acda4a07f1b9856eb --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1,14 @@ +# coding=utf-8 +# Copyright (c) 2024, HUAWEI CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. \ No newline at end of file diff --git a/tests/pipeline/baichuan2-13B/test_generation.py b/tests/pipeline/baichuan2-13B/test_generation.py index 7c5ff76907aba46edaed1dc57215d55363c9388f..6b8f63982c3fb3eca337cea5069ee16e1283745e 100644 --- a/tests/pipeline/baichuan2-13B/test_generation.py +++ b/tests/pipeline/baichuan2-13B/test_generation.py @@ -12,7 +12,7 @@ from tests.test_tools.utils import build_args, create_testconfig, setup_logger from ut.inference.test_inference import acquire_context -PATTERN = r"ModelLink:\n(.*)" +PATTERN = r"MindSpeed-LLM:\n(.*)" class TestInference(DistributedTest): diff --git a/tests/pipeline/bloom-7B/test_generation.py b/tests/pipeline/bloom-7B/test_generation.py index db4816eb45d665e1de781fe27a8783472ace40a5..1c0c6208497fb685251dd3741c84bc1f18611902 100644 --- a/tests/pipeline/bloom-7B/test_generation.py +++ b/tests/pipeline/bloom-7B/test_generation.py @@ -11,7 +11,7 @@ from tests.test_tools.utils import build_args, create_testconfig, setup_logger from ut.inference.test_inference import acquire_context -PATTERN = r"ModelLink:\n(.*)" +PATTERN = r"MindSpeed-LLM:\n(.*)" class TestInference(DistributedTest): diff --git a/tests/pipeline/chatglm3-6B/test_generation.py b/tests/pipeline/chatglm3-6B/test_generation.py index c54d8359b5b95c4507e1875833d043ca5d882bf1..391dea2d9166f7f9c7bf61ad2ddd66d83489ddf7 100644 --- a/tests/pipeline/chatglm3-6B/test_generation.py +++ b/tests/pipeline/chatglm3-6B/test_generation.py @@ -11,7 +11,7 @@ from tests.test_tools.utils import build_args, create_testconfig, setup_logger from ut.inference.test_inference import acquire_context -PATTERN = r"ModelLink:\n(.*)" +PATTERN = r"MindSpeed-LLM:\n(.*)" class TestInferenceWorldSize2(DistributedTest): diff --git a/tests/pipeline/gemma-7B/test_generation.py b/tests/pipeline/gemma-7B/test_generation.py index c7de50f750ddf8022beb8a0c425976fd7b7f293e..4c3711e1d6d7eae39645643244f54ac0db6f9681 100644 --- a/tests/pipeline/gemma-7B/test_generation.py +++ b/tests/pipeline/gemma-7B/test_generation.py @@ -11,7 +11,7 @@ from tests.test_tools.utils import build_args, create_testconfig, setup_logger from ut.inference.test_inference import acquire_context -PATTERN = r"ModelLink:\n(.*)" +PATTERN = r"MindSpeed-LLM:\n(.*)" class TestInference(DistributedTest): diff --git a/tests/pipeline/qwen15-7B/test_generation.py b/tests/pipeline/qwen15-7B/test_generation.py index e1614e60336c87a10c1742b42c9e65c89ac06280..eb39eb528138739654dee1fb052bba469478a802 100644 --- a/tests/pipeline/qwen15-7B/test_generation.py +++ b/tests/pipeline/qwen15-7B/test_generation.py @@ -11,7 +11,7 @@ from tests.test_tools.utils import build_args, create_testconfig, setup_logger from ut.inference.test_inference import acquire_context -PATTERN = r"ModelLink:\n(.*)" +PATTERN = r"MindSpeed-LLM:\n(.*)" class TestInference(DistributedTest): diff --git a/tests/st/baseline_results/chatglm3_gqa_cp8.json b/tests/st/baseline_results/chatglm3_gqa_cp8.json new file mode 100644 index 0000000000000000000000000000000000000000..200958907f590eb3034188789b764eed9b451f4a --- /dev/null +++ b/tests/st/baseline_results/chatglm3_gqa_cp8.json @@ -0,0 +1,78 @@ +{ + "lm loss": [ + 9.594278, + 9.577759, + 9.465339, + 9.230533, + 9.159581, + 9.091681, + 8.877193, + 8.776001, + 8.755611, + 8.634124, + 8.602496, + 8.540811, + 8.529873, + 8.529889, + 8.538946 + ], + "throughput": [ + 81.0, + 188.6, + 191.3, + 191.1, + 190.8, + 189.7, + 190.0, + 189.8, + 189.4, + 189.5, + 189.1, + 188.6, + 188.4, + 188.0, + 188.0 + ], + "memo info": [ + { + "rank": 0, + "allocated memory": 5386.2021484375, + "max allocated memory": 29104.03466796875 + }, + { + "rank": 1, + "allocated memory": 5386.1884765625, + "max allocated memory": 29104.03466796875 + }, + { + "rank": 2, + "allocated memory": 5386.1865234375, + "max allocated memory": 29104.03466796875 + }, + { + "rank": 3, + "allocated memory": 5386.1904296875, + "max allocated memory": 29104.03466796875 + }, + { + "rank": 4, + "allocated memory": 5386.1865234375, + "max allocated memory": 29104.03466796875 + }, + { + "rank": 5, + "allocated memory": 5386.1865234375, + "max allocated memory": 29104.03466796875 + }, + { + "rank": 6, + "allocated memory": 5386.1865234375, + "max allocated memory": 29104.03466796875 + }, + { + "rank": 7, + "allocated memory": 5386.1865234375, + "max allocated memory": 29104.03466796875 + } + ] +} diff --git a/tests/st/baseline_results/gemma2_tp8_pp1_ptd.json b/tests/st/baseline_results/gemma2_tp8_pp1_ptd.json index af751dedac79dc177ea5eb37118c983cb0aac8ec..5a6223dfb7b21a277ae6477c40329baf9794db67 100644 --- a/tests/st/baseline_results/gemma2_tp8_pp1_ptd.json +++ b/tests/st/baseline_results/gemma2_tp8_pp1_ptd.json @@ -1,37 +1,37 @@ { "lm loss": [ - 1.387083, - 1.455181, - 1.352648, - 1.309051, - 1.241118, - 1.118676, - 1.113002, - 1.067905, - 1.112425, - 1.110263, - 1.048378, - 1.055717, - 1.055748, - 1.023589, - 1.044405 + 1.385735, + 1.454418, + 1.355794, + 1.314839, + 1.248124, + 1.126178, + 1.113734, + 1.070928, + 1.117932, + 1.112961, + 1.058354, + 1.065326, + 1.060758, + 1.028861, + 1.048179 ], "throughput": [ - 43.8, - 91.9, - 92.8, - 92.9, - 92.9, - 93.1, - 93.0, - 92.6, - 92.9, - 92.9, - 92.9, - 92.9, - 93.1, - 92.8, - 93.1 + 4.5, + 19.2, + 19.3, + 19.7, + 19.5, + 19.7, + 19.9, + 20.1, + 20.4, + 20.6, + 20.1, + 19.7, + 19.7, + 20.5, + 19.9 ], "memo info": [ { diff --git a/tests/st/baseline_results/llama2_tp2_pp4_vpp2_swap.json b/tests/st/baseline_results/llama2_tp2_pp4_vpp2_swap.json new file mode 100644 index 0000000000000000000000000000000000000000..12f7f1c5a57314191431550c7eccbda50e062592 --- /dev/null +++ b/tests/st/baseline_results/llama2_tp2_pp4_vpp2_swap.json @@ -0,0 +1,78 @@ +{ + "lm loss": [ + 1.47164, + 1.46151, + 1.47107, + 1.442299, + 1.427286, + 1.406914, + 1.388342, + 1.362971, + 1.361994, + 1.286056, + 1.288965, + 1.297596, + 1.289588, + 1.286705, + 1.269324 + ], + "throughput": [ + 51.4, + 93.4, + 93.6, + 93.2, + 93.3, + 93.7, + 93.1, + 93.9, + 93.5, + 93.8, + 93.3, + 93.1, + 93.0, + 93.4, + 93.8 + ], + "memo info": [ + { + "rank": 0, + "allocated memory": 13485.05908203125, + "max allocated memory": 13735.0615234375 + }, + { + "rank": 1, + "allocated memory": 13485.05908203125, + "max allocated memory": 13735.0615234375 + }, + { + "rank": 2, + "allocated memory": 12517.05908203125, + "max allocated memory": 12689.0615234375 + }, + { + "rank": 3, + "allocated memory": 12517.05908203125, + "max allocated memory": 12689.0615234375 + }, + { + "rank": 4, + "allocated memory": 12517.05908203125, + "max allocated memory": 12689.0615234375 + }, + { + "rank": 5, + "allocated memory": 12517.05908203125, + "max allocated memory": 12689.0615234375 + }, + { + "rank": 6, + "allocated memory": 13517.12451171875, + "max allocated memory": 13767.1416015625 + }, + { + "rank": 7, + "allocated memory": 13517.12451171875, + "max allocated memory": 13767.1416015625 + } + ] +} \ No newline at end of file diff --git a/tests/st/baseline_results/mixtral_tp1_pp4_ep2_drop_dpp.json b/tests/st/baseline_results/mixtral_tp1_pp4_ep2_drop_dpp.json index 8550835bd501affa7f39ee28c90c907a3376890a..89d393dcdcf4dfce572e729ea2efc50349b2b362 100644 --- a/tests/st/baseline_results/mixtral_tp1_pp4_ep2_drop_dpp.json +++ b/tests/st/baseline_results/mixtral_tp1_pp4_ep2_drop_dpp.json @@ -1,20 +1,20 @@ { "lm loss": [ - 14.14179, - 14.11266, - 13.25863, - 11.26134, - 8.678722, - 8.656157, - 7.504652, - 7.164732, - 6.634584, - 6.487458, - 6.295924, - 5.948122, - 5.95405, - 5.73897, - 5.644468 + 14.14042, + 14.11082, + 13.25441, + 11.26227, + 8.679944, + 8.709602, + 7.502835, + 7.138363, + 6.601542, + 6.470576, + 6.224497, + 5.946918, + 5.930299, + 5.696367, + 5.602035 ], "throughput": [ 10.4, diff --git a/tests/st/shell_scripts/chatglm3_gqa_cp8.sh b/tests/st/shell_scripts/chatglm3_gqa_cp8.sh new file mode 100644 index 0000000000000000000000000000000000000000..23e2bc6c217c6a68d9b0ecc325c7c40dfc799ba8 --- /dev/null +++ b/tests/st/shell_scripts/chatglm3_gqa_cp8.sh @@ -0,0 +1,118 @@ +#!/bin/bash +export CUDA_DEVICE_MAX_CONNECTIONS=1 + +NPUS_PER_NODE=8 +MASTER_ADDR=localhost +MASTER_PORT=6001 +NNODES=1 +NODE_RANK=0 +WORLD_SIZE=$((NPUS_PER_NODE*$NNODES)) + +basepath=$(cd `dirname $0`; cd ../../../; pwd) + +CKPT_SAVE_DIR=/data/ckpt +DATA_PATH=/data/chatglm3-dataset-alpaca/alpaca_text_document +TOKENIZER_PATH=/data/chatglm3-6b-base-hf/ +CKPT_LOAD_DIR=/data/chatglm3-6b-tp1-pp1-cp8/ + +TP=1 +PP=1 +CP=8 +MBS=1 +GBS=8 +SEQ_LEN=65536 +CP_ALGO=hybrid_cp_algo + +DISTRIBUTED_ARGS=" + --nproc_per_node $NPUS_PER_NODE \ + --nnodes $NNODES \ + --node_rank $NODE_RANK \ + --master_addr $MASTER_ADDR \ + --master_port $MASTER_PORT +" + +GPT_ARGS=" + --use-mcore-models \ + --transformer-impl local \ + --tensor-model-parallel-size ${TP} \ + --pipeline-model-parallel-size ${PP} \ + --sequence-parallel \ + --num-layers 2 \ + --hidden-size 4096 \ + --ffn-hidden-size 13696 \ + --num-attention-heads 32 \ + --ulysses-degree-in-cp 4 \ + --seq-length ${SEQ_LEN} \ + --micro-batch-size ${MBS} \ + --global-batch-size ${GBS} \ + --context-parallel-algo ${CP_ALGO} \ + --context-parallel-size ${CP} \ + --max-position-embeddings ${SEQ_LEN} \ + --padded-vocab-size 65024 \ + --make-vocab-size-divisible-by 1 \ + --group-query-attention \ + --num-query-groups 2 \ + --disable-bias-linear \ + --add-qkv-bias \ + --position-embedding-type rope \ + --no-rope-fusion \ + --use-distributed-optimizer \ + --use-glm-rope \ + --rotary-percent 0.5 \ + --use-flash-attn \ + --use-fused-rmsnorm \ + --use-fused-swiglu \ + --normalization RMSNorm \ + --swiglu \ + --no-create-attention-mask-in-dataloader \ + --tokenizer-type PretrainedFromHF \ + --tokenizer-name-or-path ${TOKENIZER_PATH} \ + --lr 1e-6 \ + --train-iters 15 \ + --lr-decay-style cosine \ + --untie-embeddings-and-output-weights \ + --attention-dropout 0.0 \ + --init-method-std 0.01 \ + --hidden-dropout 0.0 \ + --no-masked-softmax-fusion \ + --attention-softmax-in-fp32 \ + --min-lr 1e-8 \ + --weight-decay 1e-1 \ + --lr-warmup-fraction 0.01 \ + --clip-grad 1.0 \ + --adam-beta1 0.9 \ + --initial-loss-scale 512 \ + --adam-beta2 0.95 \ + --no-gradient-accumulation-fusion \ + --fp16 \ + --num-workers 1 \ + --kv-head-repeat-before-uly-alltoall \ + --no-shared-storage \ + --finetune \ + --log-throughput \ + --use-cp-send-recv-overlap \ + --overlap-grad-reduce \ + --overlap-param-gather \ +" + +DATA_ARGS=" + --data-path $DATA_PATH \ + --split 949,50,1 +" + +OUTPUT_ARGS=" + --log-interval 1 \ + --save-interval 15 \ + --eval-interval 15 \ + --eval-iters 10 \ + --no-load-optim \ + --no-load-rng \ + --save $CKPT_SAVE_DIR \ + --load $CKPT_LOAD_DIR \ +" + +torchrun $DISTRIBUTED_ARGS $basepath/pretrain_gpt.py \ + $GPT_ARGS \ + $DATA_ARGS \ + $OUTPUT_ARGS \ + --distributed-backend nccl diff --git a/tests/st/shell_scripts/gemma2_tp8_pp1_ptd.sh b/tests/st/shell_scripts/gemma2_tp8_pp1_ptd.sh index c7966c75fe16496afa74ead724bbdcde05105866..42099b519484d24c0e314f9da496f62815e047d0 100644 --- a/tests/st/shell_scripts/gemma2_tp8_pp1_ptd.sh +++ b/tests/st/shell_scripts/gemma2_tp8_pp1_ptd.sh @@ -1,5 +1,6 @@ #!/bin/bash export CUDA_DEVICE_MAX_CONNECTIONS=1 +export HCCL_DETERMINISTIC=true NPUS_PER_NODE=8 MASTER_ADDR=localhost @@ -81,6 +82,7 @@ GPT_ARGS=" --no-load-rng \ --vocab-size 256000 \ --log-throughput \ + --use-deter-comp \ --finetune \ --bf16 " diff --git a/tests/st/shell_scripts/llama2_tp2_pp4_vpp2_swap.sh b/tests/st/shell_scripts/llama2_tp2_pp4_vpp2_swap.sh new file mode 100644 index 0000000000000000000000000000000000000000..a2d725817298701c2ba267512aaae96a389a4e8a --- /dev/null +++ b/tests/st/shell_scripts/llama2_tp2_pp4_vpp2_swap.sh @@ -0,0 +1,127 @@ +#!/bin/bash +# A test case for swap attention, re-compute activation function and reuse fp32 param. + +export CUDA_DEVICE_MAX_CONNECTIONS=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +NPUS_PER_NODE=8 +MASTER_ADDR=localhost +MASTER_PORT=6079 +NNODES=1 +NODE_RANK=0 +WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES)) + +basepath=$(cd `dirname $0`; cd ../../../; pwd) + +CKPT_SAVE_DIR=/data/ckpt +CKPT_LOAD_DIR=/data/ci/llama-2-7b-mg-tp2-pp4-mcore-vpp2-test +DATA_PATH=/data/pretrain_dataset/alpaca_text_document +TOKENIZER_MODEL=/data/llama-2-7b-hf/tokenizer.model +TP=2 +PP=4 +VPP=2 + +DISTRIBUTED_ARGS=( + --nproc_per_node $NPUS_PER_NODE + --nnodes $NNODES + --node_rank $NODE_RANK + --master_addr $MASTER_ADDR + --master_port $MASTER_PORT +) + + +ACCELERATE_ARGS=( + --recompute-activation-function + --recompute-num-layers 1 + --swap-attention + --reuse-fp32-param + --enable-recompute-layers-per-pp-rank +) + + +DIST_ALGO=( + --tensor-model-parallel-size ${TP} + --pipeline-model-parallel-size ${PP} + --num-layers-per-virtual-pipeline-stage ${VPP} + --sequence-parallel +) + + +MODEL_ARGS=( + --use-mcore-models + --transformer-impl local + --num-layers 32 + --hidden-size 4096 + --ffn-hidden-size 11008 + --num-attention-heads 32 + --seq-length 4096 + --max-position-embeddings 4096 +) + +TRAINING_ARGS=( + --tokenizer-type Llama2Tokenizer + --tokenizer-model ${TOKENIZER_MODEL} + --micro-batch-size 1 + --global-batch-size 32 + --make-vocab-size-divisible-by 1 + --lr 1.25e-6 + --train-iters 15 + --lr-decay-style cosine + --untie-embeddings-and-output-weights + --disable-bias-linear + --attention-dropout 0.0 + --init-method-std 0.01 + --hidden-dropout 0.0 + --position-embedding-type rope + --normalization RMSNorm + --use-fused-rmsnorm + --swiglu + --use-flash-attn + --no-masked-softmax-fusion + --attention-softmax-in-fp32 + --min-lr 1.25e-7 + --weight-decay 1e-1 + --lr-warmup-fraction 0.01 + --clip-grad 1.0 + --adam-beta1 0.9 + --initial-loss-scale 65536 + --adam-beta2 0.95 + --no-gradient-accumulation-fusion + --no-load-optim + --no-load-rng + --use-fused-swiglu + --use-fused-rotary-pos-emb + --overlap-grad-reduce + --bf16 + --use-distributed-optimizer +) + +DATA_ARGS=( + --data-path $DATA_PATH + --split 949,50,1 +) + +OUTPUT_ARGS=( + --log-interval 1 + --save-interval 10000 + --eval-interval 1000 + --eval-iters 1 + --no-load-optim + --no-load-rng + --no-save-optim + --no-save-rng + --load ${CKPT_LOAD_DIR} + --save ${CKPT_SAVE_DIR} +) + + +torchrun ${DISTRIBUTED_ARGS[@]} $basepath/pretrain_gpt.py \ + ${DIST_ALGO[@]} \ + ${MODEL_ARGS[@]} \ + ${TRAINING_ARGS[@]} \ + ${ACCELERATE_ARGS[@]} \ + ${DATA_ARGS[@]} \ + ${OUTPUT_ARGS[@]} \ + --finetune \ + --log-throughput \ + --distributed-backend nccl diff --git a/tests/test_tools/__init__.py b/tests/test_tools/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..9a1307aeca2e4639a1e6807acda4a07f1b9856eb --- /dev/null +++ b/tests/test_tools/__init__.py @@ -0,0 +1,14 @@ +# coding=utf-8 +# Copyright (c) 2024, HUAWEI CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. \ No newline at end of file diff --git a/tests/ut/dist_algo/context_parallel/test_ulysses_context_parallel.py b/tests/ut/dist_algo/context_parallel/test_ulysses_context_parallel.py index eb022742786929dc99cb30c4d58f316d3a44f19f..ddc9980c562aa06410a63bbfc753d516c51ebcc7 100644 --- a/tests/ut/dist_algo/context_parallel/test_ulysses_context_parallel.py +++ b/tests/ut/dist_algo/context_parallel/test_ulysses_context_parallel.py @@ -34,6 +34,8 @@ class FlashSelfAttention(torch.nn.Module): self.causal = causal self.softmax_scale = softmax_scale self.dropout_p = attention_dropout + self.num_attention_heads_per_partition = 1 + self.num_query_groups_per_partition = 1 def forward(self, q, k, v, attention_mask, head_num): """Implements the multihead softmax attention. diff --git a/tests/ut/evaluation/test_evaluate.py b/tests/ut/evaluation/test_evaluate.py index 8e5399cd88497eb3bce5a1f80855bd27e9e3bbeb..b3b8f662b72be1d80e97525ce4c5245d4b3218e6 100644 --- a/tests/ut/evaluation/test_evaluate.py +++ b/tests/ut/evaluation/test_evaluate.py @@ -86,7 +86,7 @@ class TestEvaluate(DistributedTest): print(log_capture) expected_score = acquire_score(log_capture) - assert math.isclose(expected_score, 0.5333, abs_tol=1e-2), f"score {expected_score}, forward pass has been changed, check it!" + assert math.isclose(expected_score, 0.5666, abs_tol=1e-2), f"score {expected_score}, forward pass has been changed, check it!" @pytest.mark.parametrize("params", test_config["test_qwen_prompt_ceval_evaluate"]) def test_qwen_prompt_ceval_evaluate(self, build_args, params): diff --git a/tests/ut/inference/test_inference.py b/tests/ut/inference/test_inference.py index ab633f005c741f9bc4d79234f147609a1fb8cabb..ab0136263da5f036d4a306961ec54dc2655f61fd 100644 --- a/tests/ut/inference/test_inference.py +++ b/tests/ut/inference/test_inference.py @@ -26,13 +26,13 @@ from tests.test_tools.dist_test import DistributedTest from tests.test_tools.utils import build_args, create_testconfig, setup_logger -PATTERN = r"ModelLink:\n(.*)" +PATTERN = r"MindSpeed-LLM:\n(.*)" def acquire_context(log_capture): # Acquire the final score for evaluation tasks, still universal. context_str = log_capture[0] - context_pattern = r"ModelLink:\s*(.*?)(?=\n|$)" + context_pattern = r"MindSpeed-LLM:\s*(.*?)(?=\n|$)" match = re.search(context_pattern, context_str) if match: context = match.group(1) @@ -77,7 +77,7 @@ class TestInference(DistributedTest): print(log_capture) context = acquire_context(log_capture) assert [context] == [ - "I'm doing well. I'm in the middle of a 3-day weekend, so I'm enjoying that." + "I'm doing well. I'm in the middle of a 3-day weekend, so I'm enjoying the extra" ], "forward pass has been changed, check it!" @pytest.mark.parametrize("params", test_config["test_lora_greedy_search"]) @@ -92,5 +92,5 @@ class TestInference(DistributedTest): print(log_capture) context = acquire_context(log_capture) assert [context] == [ - "I'm doing well. I'm in the middle of a 3-day weekend, so I'm enjoying the extra time off." - ], "forward pass has been changed, check it!" \ No newline at end of file + "I'm doing well. I'm in the middle of a 3-day weekend, so I'm enjoying that." + ], "forward pass has been changed, check it!" diff --git a/tests/ut/model_module/embeddings/test_rotary_pos_embedding.py b/tests/ut/model_module/embeddings/test_rotary_pos_embedding.py index d801d4a30c99a2d2c6570fcb0a5183c2206ab3f0..4226c5fed6988d85a208da3855d67efbd3132aa2 100644 --- a/tests/ut/model_module/embeddings/test_rotary_pos_embedding.py +++ b/tests/ut/model_module/embeddings/test_rotary_pos_embedding.py @@ -17,7 +17,6 @@ from types import SimpleNamespace from pathlib import Path import pytest import torch -import mindspeed import modellink from tests.test_tools.dist_test import create_testconfig from megatron.core.models.common.embeddings.rotary_pos_embedding import RotaryEmbedding @@ -27,13 +26,16 @@ class TestRotaryPosEmbedding: test_config = create_testconfig(Path(__file__).with_suffix(".json")) @pytest.fixture - def mock_dependency(self, request, monkeypatch): - monkeypatch.setattr(modellink.core.models.common.embeddings.rotary_pos_embedding, "get_args", - lambda : SimpleNamespace(use_glm_rope=request.getfixturevalue("chatglm"), - rope_scaling_type = None, - )) - monkeypatch.setattr(mindspeed.core.fusions.rotary_pos_embedding, "get_args", - lambda : SimpleNamespace(rotary_base = request.getfixturevalue("rotary_base"))) + def mock_dependency(self, request): + # init test name space + def get_test_namespace(): + test_name_space = SimpleNamespace() + test_name_space.use_glm_rope = request.getfixturevalue("chatglm") + test_name_space.rope_scaling_type = None + test_name_space.rotary_base = request.getfixturevalue("rotary_base") + return test_name_space + # set up name space function + setattr(modellink.core.models.common.embeddings.rotary_pos_embedding, "get_args", get_test_namespace) @pytest.mark.parametrize("rotary_param, chatglm, rotary_base, seq, expected", test_config["test_rotary_pos_embedding"]) def test_rotary_pos_embedding(self, mock_dependency, rotary_param, chatglm, rotary_base, seq, expected): diff --git a/tests/ut/process_data/test_process_instruction_data_lf.json b/tests/ut/process_data/test_process_instruction_data_lf.json index 75a8498967e7a5baa858119ae4ce317baea5310a..136b372887581fa5c850d945253e15a82278ee3b 100644 --- a/tests/ut/process_data/test_process_instruction_data_lf.json +++ b/tests/ut/process_data/test_process_instruction_data_lf.json @@ -8,6 +8,7 @@ "output-prefix": "/data/tune_dataset/alpaca/alpaca", "tokenizer-name-or-path": "/data/qwen-7b/", "workers": 4, + "overwrite-cache": null, "log-interval": 1000, "prompt-type": "qwen" } @@ -22,10 +23,26 @@ "output-prefix": "/data/tune_dataset/alpaca_his/alpaca_his", "tokenizer-name-or-path": "/data/qwen-7b/", "workers": 4, + "overwrite-cache": null, "log-interval": 1000, "prompt-type": "qwen", "map-keys": "{\"history\":\"history\"}" } + }, + { + "params": { + "input": "/data/tune_dataset/oaast_sft.json", + "tokenizer-type": "PretrainedFromHF", + "handler-name": "AlpacaStyleInstructionHandler", + "output-prefix": "/data/tune_dataset/alpaca_his/alpaca_his_seq1024", + "tokenizer-name-or-path": "/data/qwen-7b/", + "workers": 4, + "overwrite-cache": null, + "log-interval": 1000, + "seq-length" : 1024, + "prompt-type": "qwen", + "map-keys": "{\"history\":\"history\"}" + } } ], "test_sharegpt_dataset": [ @@ -37,6 +54,7 @@ "output-prefix": "/data/tune_dataset/sharegpt/sharegpt", "tokenizer-name-or-path": "/data/qwen-7b/", "workers": 4, + "overwrite-cache": null, "log-interval": 1000, "prompt-type": "qwen", "map-keys": "{\"system\":\"system_prompt\"}" @@ -52,6 +70,7 @@ "output-prefix": "/data/tune_dataset/sharegpt/sharegpt", "tokenizer-name-or-path": "/data/qwen-7b/", "workers": 4, + "overwrite-cache": null, "log-interval": 1000, "prompt-type": "qwen", "map-keys": "{\"messages\":\"messages\", \"tags\": {\"role_tag\": \"role\", \"content_tag\": \"content\", \"user_tag\": \"user\", \"assistant_tag\": \"assistant\", \"system_tag\": \"system\"}}" diff --git a/tests/ut/process_data/test_process_instruction_data_lf.py b/tests/ut/process_data/test_process_instruction_data_lf.py index 11f6bca625c5951bb2f12dd04b4a87304f314a92..acec6fc634ef9faf59af4db0b722bc3dfb45a81d 100644 --- a/tests/ut/process_data/test_process_instruction_data_lf.py +++ b/tests/ut/process_data/test_process_instruction_data_lf.py @@ -1,6 +1,9 @@ import os +import contextlib +import io from pathlib import Path import pytest +import logging import modellink from tests.test_tools.utils import build_args, create_testconfig, compare_file_md5_same from preprocess_data import main @@ -15,7 +18,7 @@ class TestProcessInstructionDataLf: @pytest.mark.parametrize("params, base_path", [ (test_config["test_alpaca_dataset"][0], "/data/tune_dataset/Llamafactoryhandler/alpaca/alpaca"), - (test_config["test_alpaca_history_dataset"][0], "/data/tune_dataset/Llamafactoryhandler/alpaca_history/alpaca_history"), + (test_config["test_alpaca_history_dataset"][0], "/data/tune_dataset/Llamafactoryhandler/alpaca_history/alpaca_history_new"), (test_config["test_sharegpt_dataset"][0], "/data/tune_dataset/Llamafactoryhandler/sharegpt/sharegpt_lf"), (test_config["test_openai_dataset"][0], "/data/tune_dataset/Llamafactoryhandler/openai/sss") ]) @@ -54,3 +57,62 @@ class TestProcessInstructionDataLf: base_file = base_path + end_str test_file = params["output-prefix"] + end_str assert compare_file_md5_same(base_file, test_file) + + + @pytest.mark.parametrize("params, base_path", + [ + (test_config["test_alpaca_history_dataset"][1], "/data/tune_dataset/Llamafactoryhandler/alpaca_history/alpaca_history_seq1024"), + ]) + def test_skip_num(self, build_args, params, base_path): + """ + Tests skip_num in preprocessing and validates output files by comparing MD5 checksums. + + Parameters: + - params: dict + A dictionary containing dataset-specific configurations, such as input files, + output prefix, and tokenizer information. Extracted from `test_config`. + - base_path: str + The base path of the reference dataset files (e.g., Alpaca, Alpaca History, ShareGPT, OpenAI). + Used to locate the ground truth files for comparison with the generated output. + """ + # create output dir if it doesn't exist + out_dir = os.path.dirname(params["output-prefix"]) + if not os.path.isdir(out_dir): + os.makedirs(out_dir) + + # run the main preprocessing function + log_capture_string = io.StringIO() + # run the main preprocessing function + log_handler = logging.StreamHandler(log_capture_string) + log_handler.setLevel(logging.INFO) + + formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s') + log_handler.setFormatter(formatter) + logger = logging.getLogger() + logger.addHandler(log_handler) + main() + output = log_capture_string.getvalue() + assert("Skip " in output and " sample exceeded seq-length" in output) + + index1 = output.find("Skip ") + index2 = output.find(" sample exceeded seq-length") + skip_num = output[index1 + 5: index2] + assert(skip_num == "796.0") + logger.removeHandler(log_handler) + log_capture_string.close() + + # print dataset name for clarity + dataset_name = base_path.split('/')[-1] + print(f"=============== test_{dataset_name}_dataset =============") + + prefix_str = params["output-prefix"].split('/')[-1] + mid_strs = ["_packed_attention_mask_document", "_packed_input_ids_document", "_packed_labels_document"] + end_suffixs = [".bin", ".idx"] + + # loop through mid_strs and end_suffixs, checking file MD5 hashes + for mid_str in mid_strs: + for end_suffix in end_suffixs: + end_str = mid_str + end_suffix + base_file = base_path + end_str + test_file = params["output-prefix"] + end_str + assert compare_file_md5_same(base_file, test_file) \ No newline at end of file
