diff --git a/README.zh.md b/README.zh.md index f2878221f01a42f5ec192ed8fc4b3e8bec549c31..3dc8e8d775d4620d015b5b8cb0a830de2242832d 100644 --- a/README.zh.md +++ b/README.zh.md @@ -84,18 +84,42 @@ with amp.scale_loss(loss, optimizer) as scaled_loss: ## 使用融合优化器优化加速 -将torch原生优化器torch.optim.xxx替换为apex.optimizers.xxx, 其中xxx为融合优化器名称，apex-patch支持的优化器见*特性介绍*。 +适配后的APEX针对adadelta/adam/sgd/lamb等做了昇腾AI处理器亲和性优化，得到的NPU融合优化器与原生算法保持一致，但运算速度更快。使用时只需将原有优化器替换为apex.optimizers.xxx，xxx为优化器名称，例如NpuFusedSGD。 + +样例原代码： +``` +optimizer = torch.optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum) +model, optimizer = amp.initialize(model, optimizer, opt_level='O2', loss_scale=32.0) ``` -model = torch.nn.Linear(D_in, D_out).cuda() -optimzier = apex.optimizers.NpuFusedSGD(model.parameters(), lr=1e-3) # 使用apex.optimizers.NpuFusedSGD -model, optimizer = amp.initialize(model, optimizer, opt_level='O1', combine_grad=True) -... -with amp.scale_loss(loss, optimizer) as scaled_loss: -  scaled_loss.backward() -... +修改后代码： +``` +optimizer = apex.optimizers.NpuFusedSGD(model.parameters(), lr=args.lr, momentum=args.momentum) +model, optimizer = amp.initialize(model, optimizer, opt_level='O2', loss_scale=32.0, combine_grad=True) ``` +**说明** + +当optimizer的可更新参数分为不同的多组，每组使用不同的策略，这种场景下，融合优化器只能优化掉第一个model分组的参数。因此，当优化器的参数相同时，尽量合并分组。 +优化前： +``` +optimizer = apex.optimizers.NpuFusedSGD([ + {'params': model.sharedNet.parameters()}, + {'params': model.bottleneck.parameters()}, + {'params': model.domain_classifier.parameters()}, + {'params': model.dcis.parameters()}, + {'params': model.source_fc.parameters(), 'lr': LEARNING_RATE}, + ], lr=LEARNING_RATE / 10, momentum=args.momentum, weight_decay=args.l2_decay) +``` +优化后： +``` +optimizer = apex.optimizers.NpuFusedSGD([ + {'params': list(model.sharedNet.parameters()) + list(model.bottleneck.parameters()) + list(model.domain_classifier.parameters())+list(model.dcis.parameters())}, + {'params': model.source_fc.parameters(), 'lr': LEARNING_RATE}, + ], lr=LEARNING_RATE / 10, momentum=args.momentum, weight_decay=args.l2_decay) +``` + + # 特性介绍 Apex-patch已适配特性如下： @@ -107,6 +131,12 @@ Apex-patch已适配特性如下： - [x] npu fused optimizer: adadelta, adam, adamp, adamw, sgd, lamb, rmsprop, rmsprop_tf - [x] 动态 loss scale新增可调参数，如：dynamic_init_scale, scale_growth_factor, scale_backoff_factor, scale_window +## 约束与限制 + +1. APEX当前版本的实现方式为python实现，不支持APEX中的自定义优化CUDA Kernel。 +2. APEX当前版本只支持适配昇腾AI处理器的混合精度计算和多种融合优化器功能，其他功能暂未支持。融合优化器与原生优化器算法保持一致，但运算速度更快。 + + ## 特性及接口介绍 原生API及参数说明请参考[Apex官方文档](https://nvidia.github.io/apex/amp.html)， Apex-patch新增特性说明如下： @@ -351,10 +381,32 @@ Apex-patch通过源码发布，需要用户自行编译，在编译时会下载 ## 公网地址声明 -| 类型 | 开源代码地址 | 文件名 | 公网IP地址/公网URL地址/域名/邮箱地址 | 用途说明 | -| :------------: |:-------------------------------------------------------------------------------:|:---------------------------------------------------:| :----------------------------------------------------------: | :-------------------------: | -| 开源引入 | https://github.com/NVIDIA/apex.git | scripts/build.sh | https://github.com/NVIDIA/apex.git | 构建脚本中拉取原生Apex源码，结合Ascend Apex patch构建安装包 | - +| 类型 | 开源代码地址 | 文件名 | 公网IP地址/公网URL地址/域名/邮箱地址 | 用途说明 | +| :---: |:---:|:---:| :---: | :---: | +| 开源引入 | https://github.com/NVIDIA/apex.git | scripts/build.sh | https://github.com/NVIDIA/apex.git | 构建脚本中拉取原生Apex源码，结合Ascend Apex patch构建安装包 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | http://jmlr.org/papers/v12/duchi11a.html | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://arxiv.org/abs/1412.6980 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://openreview.net/forum?id=ryQu7f-RZ | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://arxiv.org/abs/1904.00962 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://nvidia.github.io/OpenSeq2Seq/html/optimizers.html#novograd | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://arxiv.org/abs/1904.03288 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | http://www.cs.toronto.edu/%7Ehinton/absps/momentum.pdf | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/pytorch/pytorch/blob/master/torch/csrc/utils/tensor_flatten.h | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/NVIDIA/cutlass.git | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/NVIDIA/apex/issues/486 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/pytorch/pytorch/pull/23408 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/NVIDIA/apex/pull/323#discussion_r287021798 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/pytorch/pytorch/commit/4404762d7dd955383acee92e6f06b48144a0742e | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/NVIDIA/apex/issues/456 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://hub.docker.com/r/pytorch/pytorch | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/NVIDIA/apex/tree/master/examples/imagenet | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/NVIDIA/apex/issues/134#issuecomment-458307368 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://arxiv.org/abs/1904.00962 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/pytorch/pytorch/commit/eb7b39e02f7d75c26d8a795ea8c7fd911334da7e#diff-4632522f237f1e4e728cb824300403ac | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://pytorch.org/ | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://www.github.com/nvidia/apex | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/pytorch/pytorch/pull/36026 | 删除公网地址的patch代码 | +| 开源引入 | https://github.com/NVIDIA/apex.git | patch/npu.patch | https://github.com/NVIDIA/apex/issues | 删除公网地址的patch代码 | ## 公开接口声明