diff --git a/CONTRIBUTING_DOC.md b/CONTRIBUTING_DOC.md index 53a1a63281e5aed7f04333637b0d494362414829..171f94d5ab1d26bfb2ec1ba6a3241acf5f265a3b 100644 --- a/CONTRIBUTING_DOC.md +++ b/CONTRIBUTING_DOC.md @@ -25,7 +25,7 @@ The procedure for submitting the modification is the same as that for submitting - The title supports only the ATX style. The title and context must be separated by a blank line. - ``` + ```markdown # Heading 1 ## Heading 2 @@ -35,7 +35,7 @@ The procedure for submitting the modification is the same as that for submitting - If the list title and content need to be displayed in different lines, add a blank line between the title and content. Otherwise, the line breaks may not be implemented. - ``` + ```markdown - Title Content @@ -45,13 +45,13 @@ The procedure for submitting the modification is the same as that for submitting - Precautions are marked with a right angle bracket (>). - ``` + ```markdown > Precautions ``` - References should be listed at the end of the document and marked in the document. - ``` + ```markdown Add a [number] after the referenced text or image description. ## References @@ -63,12 +63,12 @@ The procedure for submitting the modification is the same as that for submitting - Comments in the sample code must comply with the following requirements: - - Comments are written in English. - - Use ```"""``` to comment out Python functions, methods, and classes. - - Use ```#``` to comment out other Python code. - - Use ```//``` to comment out C++ code. + - Comments are written in English. + - Use ```"""``` to comment out Python functions, methods, and classes. + - Use ```#``` to comment out other Python code. + - Use ```//``` to comment out C++ code. - ``` + ```markdown """ Comments on Python functions, methods, and classes """ @@ -81,7 +81,7 @@ The procedure for submitting the modification is the same as that for submitting - A blank line must be added before and after an image and an image title. Otherwise, the typesetting will be abnormal. For example as correctly: - ``` + ```markdown Example: ![](./xxx.png) @@ -93,7 +93,7 @@ The procedure for submitting the modification is the same as that for submitting - A blank line must be added before and after a table. Otherwise, the typesetting will be abnormal. Tables are not supported in ordered or unordered lists. For example as correctly: - ``` + ```markdown ## Title | Header1 | Header2 @@ -103,43 +103,43 @@ The procedure for submitting the modification is the same as that for submitting The following content. ``` - + - Mark the reference interface, path name, file name in the tutorial and document with "\` \`". If it's a function or method, don't use parentheses at the end. For example: - - - Reference method - - ``` + + - Reference method + + ```markdown Use the `map` method. ``` - - - Reference code - - ``` + + - Reference code + + ```markdown `batch_size`: number of data in each group. ``` - - - Reference path - - ``` + + - Reference path + + ```markdown Decompress the dataset and store it in `./MNIST_Data`. ``` - - Reference file name - - ``` + - Reference file name + + ```markdown Other dependencies is described in `requirements.txt`. ``` - In tutorials and documents, the contents that need to be replaced need additional annotation. In the body, a "*" should be added before and after the content. In the code snippet, the content should be annotated with "{}". For example: - - - In body - ``` - Need to replace your local path *your_ path*. + - In body + + ```markdown + Need to replace your local path *your_ path*. ``` - - In code snippet + - In code snippet - ``` + ```markdown conda activate {your_env_name} - ``` \ No newline at end of file + ``` diff --git a/CONTRIBUTING_DOC_CN.md b/CONTRIBUTING_DOC_CN.md index 2ed49dbd3769e654dd699a9f26e6ad8fe058135d..b349f753b1aeb3e1ecf75ec40b2ce016ba42b15d 100644 --- a/CONTRIBUTING_DOC_CN.md +++ b/CONTRIBUTING_DOC_CN.md @@ -25,7 +25,7 @@ - 标题仅支持Atx风格，标题与上下文需用空行隔开。 - ``` + ```markdown # 一级标题 ## 二级标题 @@ -35,7 +35,7 @@ - 列表标题和内容如需换行显示，标题和内容间需增加一个空行，否则无法实现换行。 - ``` + ```markdown - 标题 内容。 @@ -45,30 +45,30 @@ - 注意事项使用“>”标识。 - ``` + ```markdown > 注意事项内容。 ``` - 参考文献需列举在文末，并在文中标注。 - - ``` + + ```markdown 引用文字或图片说明后，增加标注[编号]。 ## 参考文献 [1] 作者. [有链接的文献名](http://xxx). - + [2] 作者. 没有链接的文献名. ``` - 示例代码注释需遵循如下要求： - - 注释用英文写作； - - Python函数、方法、类的注释使用```"""```； - - Python其他代码注释使用```#```； - - C++代码注释使用```//```。 + - 注释用英文写作； + - Python函数、方法、类的注释使用```"""```； + - Python其他代码注释使用```#```； + - C++代码注释使用```//```。 - ``` + ```markdown """ Python函数、方法、类的注释 """ @@ -81,7 +81,7 @@ - 图和图标题前后需增加一个空行，否则会导致排版异常。正确举例如下： - ``` + ```markdown 如下图所示： ![](./xxx.png) @@ -93,7 +93,7 @@ - 表格前后需增加一个空行，否则会导致排版异常。有序或无序列表内不支持表格。正确举例如下： - ``` + ```markdown ## 文章标题 | 表头1 | 表头2 @@ -105,41 +105,41 @@ ``` - 教程、文档中引用接口、路径名、文件名等使用“\` \`”标注，如果是函数或方法，最后不加括号。举例如下： - - - 引用方法 - - ``` + + - 引用方法 + + ```markdown 使用映射 `map` 方法。 ``` - - - 引用代码 - - ``` + + - 引用代码 + + ```markdown `batch_size`：每组包含的数据个数。 ``` - - 引用路径 - - ``` + - 引用路径 + + ```markdown 将数据集解压存放到工作区`./MNIST_Data`路径下。 ``` - - 引用文件名 - - ``` + - 引用文件名 + + ```markdown 其他依赖项在`requirements.txt`中有详细描述。 ``` - 教程、文档中待用户替换的内容需要额外标注，在正文中，使用“*”包围需要替换内容，在代码片段中，使用“{}”包围替换内容。举例如下： - - - 正文中 - ``` + - 正文中 + + ```markdown 需要替换你的本地路径*your_path*。 ``` - - - 代码片段中 - ``` + - 代码片段中 + + ```markdown conda activate {your_env_name} ``` diff --git a/README.md b/README.md index 499d288a55662adde7dfb265ca95ac93d64dec07..b79bbd2d9b961550b7ded278871cec0fd6e518c5 100644 --- a/README.md +++ b/README.md @@ -16,20 +16,20 @@ If you have any comments or suggestions on the documents, submit them in Issues. ## Directory Structure Description -``` +```text docs ├───docs // Technical documents about architecture, network list, operator list, programming guide and so on. Configuration files for API generation. -│ +│ ├───install // Installation guide. -│ +│ ├───lite // Summary of all documents related to mindspore lite and their links. -│ +│ ├───resource // Resource-related documents. -│ +│ ├───tools // Automation tool. -│ +│ ├───tutorials // Tutorial-related documents. -│ +│ └───README_CN.md // Docs repository description. ``` @@ -38,21 +38,27 @@ docs MindSpore tutorials and API documents can be generated by [Sphinx](https://www.sphinx-doc.org/en/master/). The following uses the Python API document as an example to describe the procedure, and ensure that MindSpore, MindSpore Hub and MindArmour have been installed. 1. Download code of the MindSpore Docs repository. + ```shell git clone https://gitee.com/mindspore/docs.git ``` + 2. Go to the api_python directory and install the dependency items in the `requirements.txt` file. + ```shell cd docs/api_python pip install -r requirements.txt ``` + 3. Run the following command in the api_python directory to create the `build_zh_cn/html` directory that stores the generated document web page. You can open `build_zh_cn/html/index.html` to view the API document. - ``` + + ```shell make html ``` + > If you only need to generate the MindSpore API, please modify the `source_zh_cn/conf.py` file, comment the `import mindspore_hub` and `import mindarmour` statements, and then perform this step. ## License - [Apache License 2.0](LICENSE) -- [Creative Commons License version 4.0](LICENSE-CC-BY-4.0) \ No newline at end of file +- [Creative Commons License version 4.0](LICENSE-CC-BY-4.0) diff --git a/README_CN.md b/README_CN.md index 8a4de7e8afd65198f4234dc3507f50c4b7157a0d..ac420699c9536a0188543582b30a64b2d7792f0b 100644 --- a/README_CN.md +++ b/README_CN.md @@ -16,20 +16,20 @@ ## 目录结构说明 -``` +```text docs ├───docs // 架构、网络和算子支持、编程指南等技术文档以及用于生成API的相关配置文件 -│ +│ ├───install // 安装指南 -│ +│ ├───lite // MindSpore Lite相关所有文档汇总及其链接 -│ +│ ├───resource // 资源相关文档 -│ +│ ├───tools // 自动化工具 -│ +│ ├───tutorials // 教程相关文档 -│ +│ └───README_CN.md // Docs仓说明 ``` @@ -38,19 +38,25 @@ docs MindSpore的教程和API文档均可由[Sphinx](https://www.sphinx-doc.org/en/master/)工具生成。下面以Python API文档为例介绍具体步骤，操作前需完成MindSpore、MindSpore Hub和MindArmour的安装。 1. 下载MindSpore Docs仓代码。 + ```shell git clone https://gitee.com/mindspore/docs.git ``` + 2. 进入api_python目录，安装该目录下`requirements.txt`文件中的依赖项。 + ```shell cd docs/api_python pip install -r requirements.txt ``` + 3. 在api_python目录下执行如下命令，完成后会新建`build_zh_cn/html`目录，该目录中存放了生成后的文档网页，打开`build_zh_cn/html/index.html`即可查看API文档内容。 - ``` + + ```shell make html ``` - > 如仅需生成MindSpore API，请先修改`source_zh_cn/conf.py`文件，注释`import mindspore_hub`和`import mindarmour`语句后，再执行此步骤。 + + > 如仅需生成MindSpore API，请先修改`source_zh_cn/conf.py`文件，注释`import mindspore_hub`和`import mindarmour`语句后，再执行此步骤。 ## 版权 diff --git a/docs/api_cpp/source_en/dataset.md b/docs/api_cpp/source_en/dataset.md index 607e3eaf4f4243debe90e8d8db758a111eadb683..df6c8a3f8ecae435799c89bd390b2e33f4f086d5 100644 --- a/docs/api_cpp/source_en/dataset.md +++ b/docs/api_cpp/source_en/dataset.md @@ -1,14 +1,14 @@ # mindspore::dataset -#include <[lite_mat.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/lite_mat.h)> -#include <[image_process.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/image_process.h)> +\#include <[lite_mat.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/lite_mat.h)> +\#include <[image_process.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/image_process.h)> -## Functions of image_process.h +## Functions of image_process.h ### ResizeBilinear -``` +```cpp bool ResizeBilinear(LiteMat &src, LiteMat &dst, int dst_w, int dst_h) ``` @@ -26,7 +26,7 @@ Resize image by bilinear algorithm, currently the data type only supports uint8, ### InitFromPixel -``` +```cpp bool InitFromPixel(const unsigned char *data, LPixelType pixel_type, LDataType data_type, int w, int h, LiteMat &m) ``` @@ -46,7 +46,7 @@ Initialize LiteMat from pixel, currently the conversion supports rbgaTorgb and r ### ConvertTo -``` +```cpp bool ConvertTo(LiteMat &src, LiteMat &dst, double scale = 1.0) ``` @@ -64,7 +64,7 @@ Convert the data type, currently it supports converting the data type from uint8 ### Crop -``` +```cpp bool Crop(LiteMat &src, LiteMat &dst, int x, int y, int w, int h) ``` @@ -84,7 +84,7 @@ Crop image, the channel supports is 3 and 1. ### SubStractMeanNormalize -``` +```cpp bool SubStractMeanNormalize(const LiteMat &src, LiteMat &dst, const std::vector &mean, const std::vector &std) ``` @@ -102,7 +102,7 @@ Normalize image, currently the supports data type is float. ### Pad -``` +```cpp bool Pad(const LiteMat &src, LiteMat &dst, int top, int bottom, int left, int right, PaddBorderType pad_type, uint8_t fill_b_or_gray, uint8_t fill_g, uint8_t fill_r) ``` @@ -126,7 +126,7 @@ Pad image, the channel supports is 3 and 1. ### Affine -``` +```cpp void Affine(LiteMat &src, LiteMat &out_img, double M[6], std::vector dsize, UINT8_C1 borderValue) ``` @@ -140,7 +140,7 @@ Apply affine transformation for 1 channel image. - `dsize`: The size of the output image. - `borderValue`: The pixel value is used for filing after the image is captured. -``` +```cpp void Affine(LiteMat &src, LiteMat &out_img, double M[6], std::vector dsize, UINT8_C3 borderValue) ``` @@ -156,7 +156,7 @@ Apply affine transformation for 3 channel image. ### GetDefaultBoxes -``` +```cpp std::vector> GetDefaultBoxes(BoxesConfig config) ``` @@ -172,7 +172,7 @@ Get default anchor boxes for Faster R-CNN, SSD, YOLO etc. ### ConvertBoxes -``` +```cpp void ConvertBoxes(std::vector> &boxes, std::vector> &default_boxes, BoxesConfig config) ``` @@ -186,7 +186,7 @@ Convert the prediction boxes to the actual boxes with (y, x, h, w). ### ApplyNms -``` +```cpp std::vector ApplyNms(std::vector> &all_boxes, std::vector &all_scores, float thres, int max_boxes) ``` @@ -208,11 +208,11 @@ Real-size box non-maximum suppression. Class that represents a lite Mat of a Image. -**Constructors & Destructors** +### Constructors & Destructors -### LiteMat +#### LiteMat -``` +```cpp LiteMat() LiteMat(int width, LDataType data_type = LDataType::UINT8) @@ -224,17 +224,17 @@ LiteMat(int width, int height, int channel, LDataType data_type = LDataType::UIN Constructor of MindSpore dataset LiteMat using default value of parameters. -``` +```cpp ~LiteMat(); ``` Destructor of MindSpore dataset LiteMat. -**Public Member Functions** +### Public Member Functions -### Init +#### Init -``` +```cpp void Init(int width, LDataType data_type = LDataType::UINT8) void Init(int width, int height, LDataType data_type = LDataType::UINT8) @@ -244,9 +244,9 @@ void Init(int width, int height, int channel, LDataType data_type = LDataType::U The function to initialize the channel, width and height of the image, but the parameters are different. -### IsEmpty +#### IsEmpty -``` +```cpp bool IsEmpty() const ``` @@ -256,19 +256,19 @@ A function to determine whether the object is empty. Return True or False. -### Release +#### Release -``` +```cpp void Release() ``` A function to release memory. -**Private Member Functions** +### Private Member Functions -### AlignMalloc +#### AlignMalloc -``` +```cpp void *AlignMalloc(unsigned int size) ``` @@ -282,15 +282,15 @@ Apply for memory alignment. Return the size of a pointer. -### AlignFree +#### AlignFree -``` +```cpp void AlignFree(void *ptr) ``` A function to release pointer memory. -``` +```cpp void InitElemSize(LDataType data_type) ``` @@ -300,9 +300,9 @@ Initialize the value of elem_size_ by data_type. - `data_type`: Type of data. -### addRef +#### addRef -``` +```cpp int addRef(int *p, int value) ``` @@ -311,4 +311,4 @@ A function to count the number of times the function is referenced. - Parameters - `p`: Point to the referenced object. - - `value`: Value added when quoted. \ No newline at end of file + - `value`: Value added when quoted. diff --git a/docs/api_cpp/source_en/errorcode_and_metatype.md b/docs/api_cpp/source_en/errorcode_and_metatype.md index 2cebba704dbf103909e29b1939b463a96ce4a4bc..6d725cb19d1f962a8e3a9f520b4f1f66920149dc 100644 --- a/docs/api_cpp/source_en/errorcode_and_metatype.md +++ b/docs/api_cpp/source_en/errorcode_and_metatype.md @@ -28,6 +28,7 @@ Description of error code and meta type supported in MindSpore Lite. | RET_INPUT_PARAM_INVALID | -600 | Invalid input param by user. | ## MetaType + An **enum** type. | Type Definition | Value | Description | @@ -49,4 +50,3 @@ An **enum** type. |kNumberTypeFloat32| 43 | Indicating a data type of float32. | |kNumberTypeFloat64| 44 | Indicating a data type of float64.| |kNumberTypeEnd| 45 | The end of number type. | - diff --git a/docs/api_cpp/source_en/lite.md b/docs/api_cpp/source_en/lite.md index 6c6ef9d0d78ec6b61da7f39b89d59c54ce833ad4..61ee990133dc195da807e0eeb28ab1efa789328b 100644 --- a/docs/api_cpp/source_en/lite.md +++ b/docs/api_cpp/source_en/lite.md @@ -1,11 +1,10 @@ # mindspore::lite -#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/context.h)> +\#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/context.h)> -#include <[model.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/model.h)> - -#include <[version.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/version.h)> +\#include <[model.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/model.h)> +\#include <[version.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/version.h)> ## Allocator @@ -15,135 +14,156 @@ Allocator defines a memory pool for dynamic memory malloc and memory free. Context is defined for holding environment variables during runtime. -**Constructors & Destructors** +### Constructors & Destructors -``` +```cpp Context() ``` Constructor of MindSpore Lite Context using default value for parameters. -``` +```cpp ~Context() ``` + Destructor of MindSpore Lite Context. -**Public Attributes** +### Public Attributes -``` +```cpp float16_priority ``` + A **bool** value. Defaults to **false**. Prior enable float16 inference. > Enabling float16 inference may cause low precision inference，because some variables may exceed the range of float16 during forwarding. -``` +```cpp device_type ``` + A [**DeviceType**](https://www.mindspore.cn/doc/api_cpp/en/master/lite.html#devicetype) **enum** type. Defaults to **DT_CPU**. Using to specify the device. -``` +```cpp thread_num_ ``` An **int** value. Defaults to **2**. Thread number config for thread pool. -``` +```cpp allocator ``` A **pointer** pointing to [**Allocator**](https://www.mindspore.cn/doc/api_cpp/en/master/lite.html#allocator). -``` -cpu_bind_mode_ +```cpp +cpu_bind_mode_ ``` -A [**CpuBindMode**](https://www.mindspore.cn/doc/api_cpp/en/master/lite.html#cpubindmode) **enum** variable. Defaults to **MID_CPU**. +A [**CpuBindMode**](https://www.mindspore.cn/doc/api_cpp/en/master/lite.html#cpubindmode) **enum** variable. Defaults to **MID_CPU**. ## PrimitiveC + Primitive is defined as prototype of operator. ## Model + Model defines model in MindSpore Lite for managing graph. -**Destructors** +### Destructors -``` +```cpp virtual ~Model() ``` Destructor of MindSpore Lite Model. -**Public Member Functions** +### Public Member Functions -``` +```cpp void Free() ``` + Free MetaGraph in MindSpore Lite Model to reduce memory usage during inference. -``` +```cpp void Destroy() ``` + Destroy all temporary memory in MindSpore Lite Model. -**Static Public Member Functions** -``` +### Static Public Member Functions + +```cpp static Model *Import(const char *model_buf, size_t size) ``` + Static method to create a Model pointer. -- Parameters +- Parameters - - `model_buf`: Define the buffer read from a model file. + - `model_buf`: Define the buffer read from a model file. - `size`: variable. Define bytes number of model buffer. - Returns Pointer of MindSpore Lite Model. - + ## CpuBindMode + An **enum** type. CpuBindMode defined for holding bind cpu strategy argument. -**Attributes** +### Attributes -``` -MID_CPU = -1 +```cpp +MID_CPU = 2 ``` + Bind middle cpu first. -``` +```cpp HIGHER_CPU = 1 ``` + Bind higher cpu first. -``` +```cpp NO_BIND = 0 ``` + No bind. + ## DeviceType + An **enum** type. DeviceType defined for holding user's preferred backend. -**Attributes** -``` +### Attributes + +```cpp DT_CPU = -1 ``` + CPU device type. -``` +```cpp DT_GPU = 1 ``` + GPU device type. -``` +```cpp DT_NPU = 0 ``` + NPU device type, not supported yet. + ## Version -``` +```cpp std::string Version() ``` + Global method to get a version string. - Returns diff --git a/docs/api_cpp/source_en/session.md b/docs/api_cpp/source_en/session.md index 74216a06f9928ae45edbebf8dd72953cb6390594..80e6168574004dffd6abf3e5b01d10e7a4db6ec9 100644 --- a/docs/api_cpp/source_en/session.md +++ b/docs/api_cpp/source_en/session.md @@ -1,37 +1,42 @@ -# mindspore::session - -#include <[lite_session.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/lite_session.h)> +# mindspore::session +\#include <[lite_session.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/lite_session.h)> ## LiteSession LiteSession defines session in MindSpore Lite for compiling Model and forwarding model. -**Constructors & Destructors** +### Constructors & Destructors -``` +```cpp LiteSession() ``` + Constructor of MindSpore Lite LiteSession using default value for parameters. -``` + +```cpp ~LiteSession() ``` + Destructor of MindSpore Lite LiteSession. -**Public Member Functions** -``` +### Public Member Functions + +```cpp virtual void BindThread(bool if_bind) ``` + Attempt to bind or unbind threads in the thread pool to or from the specified cpu core. - Parameters - `if_bind`: Define whether to bind or unbind threads. -``` +```cpp virtual int CompileGraph(lite::Model *model) ``` -Compile MindSpore Lite model. + +Compile MindSpore Lite model. > CompileGraph should be called before RunGraph. @@ -43,18 +48,20 @@ Compile MindSpore Lite model. STATUS as an error code of compiling graph, STATUS is defined in [errorcode.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/errorcode.h). -``` +```cpp virtual std::vector GetInputs() const ``` + Get input MindSpore Lite MSTensors of model. - Returns The vector of MindSpore Lite MSTensor. -``` +```cpp mindspore::tensor::MSTensor *GetInputsByName(const std::string &name) const ``` + Get input MindSpore Lite MSTensors of model by tensor name. - Parameters @@ -64,11 +71,12 @@ Get input MindSpore Lite MSTensors of model by tensor name. - Returns MindSpore Lite MSTensor. - -``` + +```cpp virtual int RunGraph(const KernelCallBack &before = nullptr, const KernelCallBack &after = nullptr) ``` -Run session with callback. + +Run session with callback. > RunGraph should be called after CompileGraph. - Parameters @@ -81,9 +89,10 @@ Run session with callback. STATUS as an error code of running graph, STATUS is defined in [errorcode.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/errorcode.h). -``` +```cpp virtual std::vector GetOutputsByNodeName(const std::string &node_name) const ``` + Get output MindSpore Lite MSTensors of model by node name. - Parameters @@ -94,27 +103,30 @@ Get output MindSpore Lite MSTensors of model by node name. The vector of MindSpore Lite MSTensor. -``` +```cpp virtual std::unordered_map GetOutputs() const ``` + Get output MindSpore Lite MSTensors of model mapped by tensor name. - Returns The map of output tensor name and MindSpore Lite MSTensor. -``` +```cpp virtual std::vector GetOutputTensorNames() const ``` + Get name of output tensors of model compiled by this session. - Returns The vector of string as output tensor names in order. -``` +```cpp virtual mindspore::tensor::MSTensor *GetOutputByTensorName(const std::string &tensor_name) const ``` + Get output MindSpore Lite MSTensors of model by tensor name. - Parameters @@ -125,11 +137,12 @@ Get output MindSpore Lite MSTensors of model by tensor name. Pointer of MindSpore Lite MSTensor. -``` +```cpp virtual mindspore::tensor::MSTensor *GetOutputByTensorName(const std::string &tensor_name) const ``` + Get output MindSpore Lite MSTensors of model by tensor name. - + - Parameters - `tensor_name`: Define tensor name. @@ -138,10 +151,11 @@ Get output MindSpore Lite MSTensors of model by tensor name. Pointer of MindSpore Lite MSTensor. -``` +```cpp virtual int Resize(const std::vector &inputs, const std::vector> &dims) ``` + Resize inputs shape. - Parameters @@ -153,11 +167,12 @@ Resize inputs shape. STATUS as an error code of resize inputs, STATUS is defined in [errorcode.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/errorcode.h). -**Static Public Member Functions** +### Static Public Member Functions -``` +```cpp static LiteSession *CreateSession(lite::Context *context) ``` + Static method to create a LiteSession pointer. - Parameters @@ -167,9 +182,10 @@ Static method to create a LiteSession pointer. - Returns Pointer of MindSpore Lite LiteSession. + ## KernelCallBack -``` +```cpp using KernelCallBack = std::function inputs, std::vector outputs, const CallBackParam &opInfo)> ``` @@ -179,14 +195,16 @@ A function wrapper. KernelCallBack defined the function pointer for callback. A **struct**. CallBackParam defines input arguments for callback function. -**Attributes** +### Attributes -``` +```cpp name_callback_param ``` + A **string** variable. Node name argument. -``` +```cpp type_callback_param ``` + A **string** variable. Node type argument. diff --git a/docs/api_cpp/source_en/tensor.md b/docs/api_cpp/source_en/tensor.md index 14171fd1d8d71c43bd96e69ba6e74e6a6eae96dc..4d10116efd910f19a9b97877149204d607ef0592 100644 --- a/docs/api_cpp/source_en/tensor.md +++ b/docs/api_cpp/source_en/tensor.md @@ -1,33 +1,35 @@ # mindspore::tensor -#include <[ms_tensor.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/ms_tensor.h)> - +\#include <[ms_tensor.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/ms_tensor.h)> ## MSTensor MSTensor defined tensor in MindSpore Lite. -**Constructors & Destructors** -``` +### Constructors & Destructors + +```cpp MSTensor() ``` + Constructor of MindSpore Lite MSTensor. - Returns Instance of MindSpore Lite MSTensor. - -``` + +```cpp virtual ~MSTensor() ``` Destructor of MindSpore Lite Model. -**Public Member Functions** +### Public Member Functions -``` +```cpp virtual TypeId data_type() const ``` + Get data type of the MindSpore Lite MSTensor. > TypeId is defined in [mindspore/mindspore/core/ir/dtype/type_id.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/core/ir/dtype/type_id.h). Only number types in TypeId enum are suitable for MSTensor. @@ -36,7 +38,7 @@ Get data type of the MindSpore Lite MSTensor. MindSpore Lite TypeId of the MindSpore Lite MSTensor. -``` +```cpp virtual std::vector shape() const ``` @@ -46,7 +48,7 @@ Get shape of the MindSpore Lite MSTensor. A vector of int as the shape of the MindSpore Lite MSTensor. -``` +```cpp virtual int DimensionSize(size_t index) const ``` @@ -60,7 +62,7 @@ Get size of the dimension of the MindSpore Lite MSTensor index by the parameter Size of dimension of the MindSpore Lite MSTensor. -``` +```cpp virtual int ElementsNum() const ``` @@ -70,7 +72,7 @@ Get number of element in MSTensor. Number of element in MSTensor. -``` +```cpp virtual size_t Size() const ``` @@ -79,15 +81,13 @@ Get byte size of data in MSTensor. - Returns Byte size of data in MSTensor. - -``` +```cpp virtual void *MutableData() const ``` Get the pointer of data in MSTensor. - > The data pointer can be used to both write and read data in MSTensor. - Returns diff --git a/docs/api_cpp/source_zh_cn/dataset.md b/docs/api_cpp/source_zh_cn/dataset.md index 5170585149b24badd6ae3a38f40aacf336641003..1811c7d1ca8864e2bd0e0f4a1ea2dc724356fea2 100644 --- a/docs/api_cpp/source_zh_cn/dataset.md +++ b/docs/api_cpp/source_zh_cn/dataset.md @@ -1,14 +1,13 @@ # mindspore::dataset -#include <[lite_mat.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/lite_mat.h)> -#include <[image_process.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/image_process.h)> - +\#include <[lite_mat.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/lite_mat.h)> +\#include <[image_process.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/minddata/dataset/kernels/image/lite_cv/image_process.h)> ## image_process.h文件的函数 ### ResizeBilinear -``` +```cpp bool ResizeBilinear(LiteMat &src, LiteMat &dst, int dst_w, int dst_h) ``` @@ -26,7 +25,7 @@ bool ResizeBilinear(LiteMat &src, LiteMat &dst, int dst_w, int dst_h) ### InitFromPixel -``` +```cpp bool InitFromPixel(const unsigned char *data, LPixelType pixel_type, LDataType data_type, int w, int h, LiteMat &m) ``` @@ -46,7 +45,7 @@ bool InitFromPixel(const unsigned char *data, LPixelType pixel_type, LDataType d ### ConvertTo -``` +```cpp bool ConvertTo(LiteMat &src, LiteMat &dst, double scale = 1.0) ``` @@ -64,7 +63,7 @@ bool ConvertTo(LiteMat &src, LiteMat &dst, double scale = 1.0) ### Crop -``` +```cpp bool Crop(LiteMat &src, LiteMat &dst, int x, int y, int w, int h) ``` @@ -84,7 +83,7 @@ bool Crop(LiteMat &src, LiteMat &dst, int x, int y, int w, int h) ### SubStractMeanNormalize -``` +```cpp bool SubStractMeanNormalize(const LiteMat &src, LiteMat &dst, const std::vector &mean, const std::vector &std) ``` @@ -102,7 +101,7 @@ bool SubStractMeanNormalize(const LiteMat &src, LiteMat &dst, const std::vector< ### Pad -``` +```cpp bool Pad(const LiteMat &src, LiteMat &dst, int top, int bottom, int left, int right, PaddBorderType pad_type, uint8_t fill_b_or_gray, uint8_t fill_g, uint8_t fill_r) ``` @@ -126,7 +125,7 @@ bool Pad(const LiteMat &src, LiteMat &dst, int top, int bottom, int left, int ri ### Affine -``` +```cpp void Affine(LiteMat &src, LiteMat &out_img, double M[6], std::vector dsize, UINT8_C1 borderValue) ``` @@ -140,7 +139,7 @@ void Affine(LiteMat &src, LiteMat &out_img, double M[6], std::vector dsi - `dsize`: 输出图像的大小。 - `borderValue`: 采图之后用于填充的像素值。 -``` +```cpp void Affine(LiteMat &src, LiteMat &out_img, double M[6], std::vector dsize, UINT8_C3 borderValue) ``` @@ -156,7 +155,7 @@ void Affine(LiteMat &src, LiteMat &out_img, double M[6], std::vector dsi ### GetDefaultBoxes -``` +```cpp std::vector> GetDefaultBoxes(BoxesConfig config) ``` @@ -172,7 +171,7 @@ std::vector> GetDefaultBoxes(BoxesConfig config) ### ConvertBoxes -``` +```cpp void ConvertBoxes(std::vector> &boxes, std::vector> &default_boxes, BoxesConfig config) ``` @@ -186,7 +185,7 @@ void ConvertBoxes(std::vector> &boxes, std::vector ApplyNms(std::vector> &all_boxes, std::vector &all_scores, float thres, int max_boxes) ``` @@ -208,11 +207,11 @@ std::vector ApplyNms(std::vector> &all_boxes, std::vecto LiteMat是一个处理图像的类。 -**构造函数和析构函数** +### 构造函数和析构函数 -### LiteMat +#### LiteMat -``` +```cpp LiteMat() LiteMat(int width, LDataType data_type = LDataType::UINT8) @@ -224,17 +223,17 @@ LiteMat(int width, int height, int channel, LDataType data_type = LDataType::UIN MindSpore中dataset模块下LiteMat的构造方法，使用参数的默认值。 -``` +```cpp ~LiteMat(); ``` MindSpore dataset LiteMat的析构函数。 -**公有成员函数** +### 公有成员函数 -### Init +#### Init -``` +```cpp void Init(int width, LDataType data_type = LDataType::UINT8) void Init(int width, int height, LDataType data_type = LDataType::UINT8) @@ -244,9 +243,9 @@ void Init(int width, int height, int channel, LDataType data_type = LDataType::U 该函数用于初始化图像的通道，宽度和高度，参数不同。 -### IsEmpty +#### IsEmpty -``` +```cpp bool IsEmpty() const ``` @@ -256,19 +255,19 @@ bool IsEmpty() const 返回True或者False。 -### Release +#### Release -``` +```cpp void Release() ``` 释放内存的函数。 -**私有成员函数** +### 私有成员函数 -### AlignMalloc +#### AlignMalloc -``` +```cpp void *AlignMalloc(unsigned int size) ``` @@ -282,18 +281,17 @@ void *AlignMalloc(unsigned int size) 返回指针的大小。 -### AlignFree +#### AlignFree -``` +```cpp void AlignFree(void *ptr) ``` 释放指针内存大小的方法。 +#### InitElemSize -### InitElemSize - -``` +```cpp void InitElemSize(LDataType data_type) ``` @@ -303,7 +301,7 @@ void InitElemSize(LDataType data_type) - `data_type`: 数据的类型。 -``` +```cpp int addRef(int *p, int value) ``` @@ -312,4 +310,4 @@ void InitElemSize(LDataType data_type) - 参数 - `p`: 指向引用的对象。 - - `value`: 引用时所加的值。 \ No newline at end of file + - `value`: 引用时所加的值。 diff --git a/docs/api_cpp/source_zh_cn/lite.md b/docs/api_cpp/source_zh_cn/lite.md index b3aaa5909c268a82eb43b7717348816e55c501f3..978178fabc8f412a6c811e0bb117dcce887ed788 100644 --- a/docs/api_cpp/source_zh_cn/lite.md +++ b/docs/api_cpp/source_zh_cn/lite.md @@ -1,11 +1,10 @@ # mindspore::lite -#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/context.h)> +\#include <[context.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/context.h)> -#include <[model.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/model.h)> - -#include <[version.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/version.h)> +\#include <[model.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/model.h)> +\#include <[version.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/version.h)> ## Allocator @@ -15,23 +14,23 @@ Allocator类定义了一个内存池，用于动态地分配和释放内存。 Context类用于保存执行中的环境变量。 -**构造函数和析构函数** +### 构造函数和析构函数 -``` +```cpp Context() ``` 用默认参数构造MindSpore Lite Context 对象。 -``` +```cpp ~Context() ``` MindSpore Lite Context 的析构函数。 -**公有属性** +### 公有属性 -``` +```cpp float16_priority ``` @@ -39,29 +38,29 @@ float16_priority > 使能float16推理可能会导致模型推理精度下降，因为在模型推理的中间过程中，有些变量可能会超出float16的数值范围。 -``` +```cpp device_type ``` [**DeviceType**](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#devicetype)枚举类型。默认为**DT_CPU**，用于设置设备信息。 -``` +```cpp thread_num_ ``` **int** 值，默认为**2**，设置线程数。 -``` +```cpp allocator ``` 指针类型，指向内存分配器[**Allocator**](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#allocator)的指针。 -``` -cpu_bind_mode_ +```cpp +cpu_bind_mode_ ``` -[**CpuBindMode**](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#cpubindmode)枚举类型，默认为**MID_CPU**。 +[**CpuBindMode**](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/lite.html#cpubindmode)枚举类型，默认为**MID_CPU**。 ## PrimitiveC @@ -71,87 +70,89 @@ PrimitiveC定义为算子的原型。 Model定义了MindSpore Lite中的模型，便于计算图管理。 -**析构函数** +### 析构函数 -``` +```cpp ~Model() ``` MindSpore Lite Model的析构函数。 -**公有成员函数** +### 公有成员函数 -``` +```cpp void Destroy() ``` 释放Model内的所有过程中动态分配的内存。 -``` +```cpp void Free() ``` 释放MindSpore Lite Model中的MetaGraph，用于减小运行时的内存。 -**静态公有成员函数** +### 静态公有成员函数 -``` +```cpp static Model *Import(const char *model_buf, size_t size) ``` 创建Model指针的静态方法。 -- 参数 +- 参数 - - `model_buf`: 定义了读取模型文件的缓存区。 + - `model_buf`: 定义了读取模型文件的缓存区。 - - `size`: 定义了模型缓存区的字节数。 + - `size`: 定义了模型缓存区的字节数。 - 返回值 指向MindSpore Lite的Model的指针。 - + ## CpuBindMode + 枚举类型，设置cpu绑定策略。 -**属性** +### 属性 -``` -MID_CPU = -1 +```cpp +MID_CPU = 2 ``` 优先中等CPU绑定策略。 -``` +```cpp HIGHER_CPU = 1 ``` 优先高级CPU绑定策略。 -``` +```cpp NO_BIND = 0 ``` 不绑定。 ## DeviceType + 枚举类型，设置设备类型。 -**属性** +### 属性 -``` +```cpp DT_CPU = -1 ``` 设备为CPU。 -``` +```cpp DT_GPU = 1 ``` 设备为GPU。 -``` +```cpp DT_NPU = 0 ``` @@ -159,11 +160,12 @@ DT_NPU = 0 ## Version -``` +```cpp std::string Version() ``` + 全局方法，用于获取版本的字符串。 - 返回值 - MindSpore Lite版本的字符串。 \ No newline at end of file + MindSpore Lite版本的字符串。 diff --git a/docs/api_cpp/source_zh_cn/session.md b/docs/api_cpp/source_zh_cn/session.md index adfc159b68d8e5e7f35abfef400f434e1719eb9f..58f3a86018f029c7256b9ab0d6e40536cebae614 100644 --- a/docs/api_cpp/source_zh_cn/session.md +++ b/docs/api_cpp/source_zh_cn/session.md @@ -1,36 +1,41 @@ # mindspore::session -#include <[lite_session.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/lite_session.h)> - +\#include <[lite_session.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/lite_session.h)> ## LiteSession LiteSession定义了MindSpore Lite中的会话，用于进行Model的编译和前向推理。 -**构造函数和析构函数** +### 构造函数和析构函数 -``` +```cpp LiteSession() ``` + MindSpore Lite LiteSession的构造函数，使用默认参数。 -``` + +```cpp ~LiteSession() ``` + MindSpore Lite LiteSession的析构函数。 -**公有成员函数** -``` +### 公有成员函数 + +```cpp virtual void BindThread(bool if_bind) ``` + 尝试将线程池中的线程绑定到指定的cpu内核，或从指定的cpu内核进行解绑。 - 参数 - `if_bind`: 定义了对线程进行绑定或解绑。 -``` +```cpp virtual int CompileGraph(lite::Model *model) ``` + 编译MindSpore Lite模型。 > CompileGraph必须在RunGraph方法之前调用。 @@ -43,18 +48,20 @@ virtual int CompileGraph(lite::Model *model) STATUS ，即编译图的错误码。STATUS在[errorcode.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/errorcode.h)中定义。 -``` +```cpp virtual std::vector GetInputs() const ``` + 获取MindSpore Lite模型的MSTensors输入。 - 返回值 MindSpore Lite MSTensor向量。 -``` +```cpp mindspore::tensor::MSTensor *GetInputsByName(const std::string &name) const ``` + 通过tensor名获取MindSpore Lite模型的MSTensors输入。 - 参数 @@ -65,9 +72,10 @@ mindspore::tensor::MSTensor *GetInputsByName(const std::string &name) const MindSpore Lite MSTensor。 -``` +```cpp virtual int RunGraph(const KernelCallBack &before = nullptr, const KernelCallBack &after = nullptr) ``` + 运行带有回调函数的会话。 > RunGraph必须在CompileGraph方法之后调用。 @@ -81,9 +89,10 @@ virtual int RunGraph(const KernelCallBack &before = nullptr, const KernelCallBac STATUS ，即编译图的错误码。STATUS在[errorcode.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/errorcode.h)中定义。 -``` +```cpp virtual std::vector GetOutputsByNodeName(const std::string &node_name) const ``` + 通过节点名获取MindSpore Lite模型的MSTensors输出。 - 参数 @@ -94,27 +103,30 @@ virtual std::vector GetOutputsByNodeName(const std::string MindSpore Lite MSTensor向量。 -``` +```cpp virtual std::unordered_map GetOutputs() const ``` + 获取与张量名相关联的MindSpore Lite模型的MSTensors输出。 - 返回值 包含输出张量名和MindSpore Lite MSTensor的容器类型变量。 -``` +```cpp virtual std::vector GetOutputTensorNames() const ``` + 获取由当前会话所编译的模型的输出张量名。 - 返回值 字符串向量，其中包含了按顺序排列的输出张量名。 -``` +```cpp virtual mindspore::tensor::MSTensor *GetOutputByTensorName(const std::string &tensor_name) const ``` + 通过张量名获取MindSpore Lite模型的MSTensors输出。 - 参数 @@ -125,9 +137,10 @@ virtual mindspore::tensor::MSTensor *GetOutputByTensorName(const std::string &te 指向MindSpore Lite MSTensor的指针。 -``` +```cpp virtual int Resize(const std::vector &inputs, const std::vector> &dims) ``` + 调整输入的形状。 - 参数 @@ -139,11 +152,12 @@ virtual int Resize(const std::vector &inputs, const std::ve STATUS ，即编译图的错误码。STATUS在[errorcode.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/errorcode.h)中定义。 -**静态公有成员函数** +### 静态公有成员函数 -``` +```cpp static LiteSession *CreateSession(lite::Context *context) ``` + 用于创建一个LiteSession指针的静态方法。 - 参数 @@ -153,9 +167,10 @@ static LiteSession *CreateSession(lite::Context *context) - 返回值 指向MindSpore Lite LiteSession的指针。 + ## KernelCallBack -``` +```cpp using KernelCallBack = std::function inputs, std::vector outputs, const CallBackParam &opInfo)> ``` @@ -166,12 +181,14 @@ using KernelCallBack = std::function inputs 一个结构体。CallBackParam定义了回调函数的输入参数。 **属性** -``` +```cpp name_callback_param ``` + **string** 类型变量。节点名参数。 -``` +```cpp type_callback_param ``` + **string** 类型变量。节点类型参数。 diff --git a/docs/api_cpp/source_zh_cn/tensor.md b/docs/api_cpp/source_zh_cn/tensor.md index 50e7c294b1923dbc85ff2911f42c91e47b630944..0d2e7e219a83468d84c7d83c961ebe82b0e6510a 100644 --- a/docs/api_cpp/source_zh_cn/tensor.md +++ b/docs/api_cpp/source_zh_cn/tensor.md @@ -1,32 +1,35 @@ # mindspore::tensor -#include <[ms_tensor.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/ms_tensor.h)> - +\#include <[ms_tensor.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/include/ms_tensor.h)> ## MSTensor MSTensor定义了MindSpore Lite中的张量。 -**构造函数和析构函数** -``` +### 构造函数和析构函数 + +```cpp MSTensor() ``` + MindSpore Lite MSTensor的构造函数。 - 返回值 MindSpore Lite MSTensor的实例。 - -``` + +```cpp virtual ~MSTensor() ``` + MindSpore Lite Model的析构函数。 -**公有成员函数** +### 公有成员函数 -``` +```cpp virtual TypeId data_type() const ``` + 获取MindSpore Lite MSTensor的数据类型。 > TypeId在[mindspore/mindspore/core/ir/dtype/type_id\.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/core/ir/dtype/type_id.h)中定义。只有TypeId枚举中的数字类型可用于MSTensor。 @@ -35,18 +38,20 @@ virtual TypeId data_type() const MindSpore Lite MSTensor类的MindSpore Lite TypeId。 -``` +```cpp virtual std::vector shape() const ``` + 获取MindSpore Lite MSTensor的形状。 - 返回值 一个包含MindSpore Lite MSTensor形状数值的整型向量。 -``` +```cpp virtual int DimensionSize(size_t index) const ``` + 通过参数索引获取MindSpore Lite MSTensor的维度的大小。 - 参数 @@ -57,28 +62,30 @@ virtual int DimensionSize(size_t index) const MindSpore Lite MSTensor的维度的大小。 -``` +```cpp virtual int ElementsNum() const ``` + 获取MSTensor中的元素个数。 - 返回值 MSTensor中的元素个数 -``` +```cpp virtual size_t Size() const ``` + 获取MSTensor中的数据的字节数大小。 - 返回值 MSTensor中的数据的字节数大小。 - -``` +```cpp virtual void *MutableData() const ``` + 获取MSTensor中的数据的指针。 > 该数据指针可用于对MSTensor中的数据进行读取和写入。 diff --git a/docs/api_java/source_en/class_list.md b/docs/api_java/source_en/class_list.md index befeea5d3882c22b2a7f7d48f60094a10258b790..5244b02ce5e8ac7f44cd8406fa07898ca6153ddf 100644 --- a/docs/api_java/source_en/class_list.md +++ b/docs/api_java/source_en/class_list.md @@ -1,3 +1,13 @@ # Class List -Java API is being translated, will be released soon. +| Package | Class Name | Description | +| ------------------------- | -------------- | ------------------------------------------------------------ | +| com.mindspore.lite.config | MSConfig | MSConfig defines for holding environment variables during runtime. | +| com.mindspore.lite.config | CpuBindMode | CpuBindMode defines the CPU binding mode. | +| com.mindspore.lite.config | DeviceType | DeviceType defines the back-end device type. | +| com.mindspore.lite | LiteSession | LiteSession defines session in MindSpore Lite for compiling Model and forwarding model. | +| com.mindspore.lite | Model | Model defines the model in MindSpore Lite for managing graph. | +| com.mindspore.lite | MSTensor | MSTensor defines the tensor in MindSpore Lite. | +| com.mindspore.lite | DataType | DataType defines the supported data types. | +| com.mindspore.lite | Version | Version is used to obtain the version information of MindSpore Lite. | + diff --git a/docs/api_java/source_en/index.rst b/docs/api_java/source_en/index.rst index f8e7fd507a9549880c0982b37bac2a5cf082feae..935aa0a5d22565b2d51fc919a8f81c00d9702b02 100644 --- a/docs/api_java/source_en/index.rst +++ b/docs/api_java/source_en/index.rst @@ -10,4 +10,8 @@ MindSpore Java API :glob: :maxdepth: 1 - class_list \ No newline at end of file + class_list + lite_session + model + msconfig + mstensor \ No newline at end of file diff --git a/docs/api_java/source_en/lite_session.md b/docs/api_java/source_en/lite_session.md new file mode 100644 index 0000000000000000000000000000000000000000..8b71869df777c442b8f9abb1f0d5f4e2d1abbb85 --- /dev/null +++ b/docs/api_java/source_en/lite_session.md @@ -0,0 +1,188 @@ +# LiteSession + +```java +import com.mindspore.lite.LiteSession; +``` + +LiteSession defines session in MindSpore Lite for compiling Model and forwarding model. + +## Public Member Functions + +| function | +| ------------------------------------------------------------ | +| [boolean init(MSConfig config)](#init) | +| [void bindThread(boolean if_bind)](#bindthread) | +| [boolean compileGraph(Model model)](#compilegraph) | +| [boolean runGraph()](#rungraph) | +| [List\ getInputs()](#getinputs) | +| [List\ getInputsByName(String nodeName)](#getinputsbyname) | +| [List\ getOutputsByNodeName(String nodeName)](#getoutputsbynodename) | +| [Map\ getOutputMapByTensor()](#getoutputmapbytensor) | +| [List\ getOutputTensorNames()](#getoutputtensornames) | +| [MSTensor getOutputByTensorName(String tensorName)](#getoutputbytensorname) | +| [boolean resize(List\ inputs, int[][] dims](#resize) | +| [void free()](#free) | + +## init + +```java +public boolean init(MSConfig config) +``` + +Initialize LiteSession. + +- Parameters + + - `MSConfig`: MSConfig class. + +- Returns + + Whether the initialization is successful. + +## bindThread + +```java +public void bindThread(boolean if_bind) +``` + +Attempt to bind or unbind threads in the thread pool to or from the specified cpu core. + +- Parameters + - `if_bind`: Define whether to bind or unbind threads. + +## compileGraph + +```java +public boolean compileGraph(Model model) +``` + +Compile MindSpore Lite model. + +- Parameters + + - `Model`: Define the model to be compiled. + +- Returns + + Whether the compilation is successful. + +## runGraph + +```java +public boolean runGraph() +``` + +Run the session for inference. + +- Returns + + Whether the inference is successful. + +## getInputs + +```java +public List getInputs() +``` + +Get the MSTensors input of MindSpore Lite model. + +- Returns + + The vector of MindSpore Lite MSTensor. + +## getInputsByName + +```java +public List getInputsByName(String nodeName) +``` + +Get the MSTensors input of MindSpore Lite model by the node name. + +- Parameters + + - `nodeName`: Define the node name. + +- Returns + + The vector of MindSpore Lite MSTensor. + +## getOutputsByNodeName + +```java +public List getOutputsByNodeName(String nodeName) +``` + +Get the MSTensors output of MindSpore Lite model by the node name. + +- Parameters + + - `nodeName`: Define the node name. + +- Returns + + The vector of MindSpore Lite MSTensor. + +## getOutputMapByTensor + +```java +public Map getOutputMapByTensor() +``` + +Get the MSTensors output of the MindSpore Lite model associated with the tensor name. + +- Returns + + The map of output tensor name and MindSpore Lite MSTensor. + +## getOutputTensorNames + +```java +public List getOutputTensorNames() +``` + +Get the name of output tensors of the model compiled by this session. + +- Returns + + The vector of string as output tensor names in order. + +## getOutputByTensorName + +```java +public MSTensor getOutputByTensorName(String tensorName) +``` + +Get the MSTensors output of MindSpore Lite model by the tensor name. + +- Parameters + + - `tensorName`: Define the tensor name. + +- Returns + + Pointer of MindSpore Lite MSTensor. + +## resize + +```java +public boolean resize(List inputs, int[][] dims) +``` + +Resize inputs shape. + +- Parameters + + - `inputs`: Model inputs. + - `dims`: Define the new inputs shape. + +- Returns + + Whether the resize is successful. + +## free + +```java +public void free() +``` + +Free LiteSession. diff --git a/docs/api_java/source_en/model.md b/docs/api_java/source_en/model.md new file mode 100644 index 0000000000000000000000000000000000000000..101750238841d2875f2ccba22fa4d7f342609e71 --- /dev/null +++ b/docs/api_java/source_en/model.md @@ -0,0 +1,64 @@ +# Model + +```java +import com.mindspore.lite.Model; +``` + +Model defines model in MindSpore Lite for managing graph. + +## Public Member Functions + +| function | +| ------------------------------------------------------------ | +| [boolean loadModel(Context context, String modelName)](#loadmodel) | +| [boolean loadModel(String modelPath)](#loadmodel) | +| [void freeBuffer()](#freebuffer) | +| [void free()](#free) | + +## loadModel + +```java +public boolean loadModel(Context context, String modelName) +``` + +Load the MindSpore Lite model from Assets. + +- Parameters + + - `context`: Context in Android. + - `modelName`: Model file name. + +- Returns + + Whether the load is successful. + +```java +public boolean loadModel(String modelPath) +``` + +Load the MindSpore Lite model from path. + +- Parameters + + - `modelPath`: Model file path. + +- Returns + + Whether the load is successful. + +## freeBuffer + +```java +public void freeBuffer() +``` + +Free MetaGraph in MindSpore Lite Model to reduce memory usage during inference. + +## free + +```java +public void free() +``` + +Free all temporary memory in MindSpore Lite Model. + diff --git a/docs/api_java/source_en/msconfig.md b/docs/api_java/source_en/msconfig.md new file mode 100644 index 0000000000000000000000000000000000000000..1d0feff298d136bde848ffd8b8e5884c8cdda78f --- /dev/null +++ b/docs/api_java/source_en/msconfig.md @@ -0,0 +1,82 @@ +# MSConfig + +```java +import com.mindspore.lite.config.MSConfig; +``` + +MSConfig is defined for holding environment variables during runtime. + +## Public Member Functions + +| function | +| ------------------------------------------------------------ | +| [boolean init(int deviceType, int threadNum, int cpuBindMode)](#init) | +| [boolean init(int deviceType, int threadNum)](#init) | +| [boolean init(int deviceType)](#init) | +| [boolean init()](#init) | +| [void free()](#free) | + +## init + +```java +public boolean init(int deviceType, int threadNum, int cpuBindMode) +``` + +Initialize MSConfig. + +- Parameters + + - `deviceType`: A [**DeviceType**](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java) **enum** type. + - `threadNum`: Thread number config for thread pool. + - `cpuBindMode`: A [**CpuBindMode**](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/CpuBindMode.java) **enum** variable. + +- Returns + + Whether the initialization is successful. + +```java +public boolean init(int deviceType, int threadNum) +``` + +Initialize MSConfig, `cpuBindMode` defaults to `CpuBindMode.MID_CPU`. + +- Parameters + + - `deviceType`: A [**DeviceType**](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java) **enum** type. + - `threadNum`: Thread number config for thread pool. + +- Returns + + Whether the initialization is successful. + +```java +public boolean init(int deviceType) +``` + +Initialize MSConfig，`cpuBindMode` defaults to `CpuBindMode.MID_CPU`, `threadNum` defaults to `2`. + +- Parameters + + - `deviceType`: A [**DeviceType**](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java) **enum** type. + +- Returns + + Whether the initialization is successful. + +```java +public boolean init() +``` + +Initialize MSConfig，`deviceType` defaults to `DeviceType.DT_CPU`，`cpuBindMode` defaults to`CpuBindMode.MID_CPU`，`threadNum` defaults to `2`. + +- Returns + + Whether the initialization is successful. + +## free + +```java +public void free() +``` + +Free all temporary memory in MindSpore Lite MSConfig. \ No newline at end of file diff --git a/docs/api_java/source_en/mstensor.md b/docs/api_java/source_en/mstensor.md new file mode 100644 index 0000000000000000000000000000000000000000..69bdaa63418778625ef7b6937c44bed697563b8a --- /dev/null +++ b/docs/api_java/source_en/mstensor.md @@ -0,0 +1,148 @@ +# MSTensor + +```java +import com.mindspore.lite.MSTensor; +``` + +MSTensor defined tensor in MindSpore Lite. + +## Public Member Functions + +| function | +| ------------------------------------------ | +| [int[] getShape()](#getshape) | +| [int getDataType()](#getdatatype) | +| [byte[] getByteData()](#getbytedata) | +| [float[] getFloatData()](#getfloatdata) | +| [int[] getIntData()](#getintdata) | +| [long[] getLongData()](#getlongdata) | +| [void setData(byte[] data)](#setdata) | +| [void setData(ByteBuffer data)](#setdata) | +| [long size()](#size) | +| [int elementsNum()](#elementsnum) | +| [void free()](#free) | + +## getShape + +```java +public int[] getShape() +``` + +Get the shape of the MindSpore Lite MSTensor. + +- Returns + + A array of int as the shape of the MindSpore Lite MSTensor. + +## getDataType + +```java +public int getDataType() +``` + +> DataType is defined in [com.mindspore.lite.DataType](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/DataType.java). + +- Returns + + The MindSpore Lite data type of the MindSpore Lite MSTensor class. + +## getByteData + +```java +public byte[] getByteData() +``` + +Get output data of MSTensor, the data type is byte. + +- Returns + + The byte array containing all MSTensor output data. + +## getFloatData + +```java +public float[] getFloatData() +``` + +Get output data of MSTensor, the data type is float. + +- Returns + + The float array containing all MSTensor output data. + +## getIntData + +```java +public int[] getIntData() +``` + +Get output data of MSTensor, the data type is int. + +- Returns + + The int array containing all MSTensor output data. + +## getLongData + +```java +public long[] getLongData() +``` + +Get output data of MSTensor, the data type is long. + +- Returns + + The long array containing all MSTensor output data. + +## setData + +```java +public void setData(byte[] data) +``` + +Set the input data of MSTensor. + +- Parameters + - `data`: Input data of byte[] type. + +```java +public void setData(ByteBuffer data) +``` + +Set the input data of MSTensor. + +- Parameters + - `data`: Input data of ByteBuffer type. + +## size + +```java +public long size() +``` + +Get the size of the data in MSTensor in bytes. + +- Returns + + The size of the data in MSTensor in bytes. + +## elementsNum + +```java +public int elementsNum() +``` + +Get the number of elements in MSTensor. + +- Returns + + The number of elements in MSTensor. + +## free + +```java +public void free() +``` + +Free all temporary memory in MindSpore Lite MSTensor. + diff --git a/docs/api_java/source_zh_cn/lite_session.md b/docs/api_java/source_zh_cn/lite_session.md index c008d0eab9f86e9d6b59e2b776ae645a40aa3510..9ed5bc8e9ac138a61128ba5b64dd439b44a97ba1 100644 --- a/docs/api_java/source_zh_cn/lite_session.md +++ b/docs/api_java/source_zh_cn/lite_session.md @@ -33,7 +33,7 @@ public boolean init(MSConfig config) - 参数 - - `MSConfig`: MSConfig类。 + - `MSConfig`: MSConfig类。 - 返回值 @@ -48,7 +48,7 @@ public void bindThread(boolean if_bind) 尝试将线程池中的线程绑定到指定的CPU内核，或从指定的CPU内核进行解绑。 - 参数 - - `if_bind`: 是否对线程进行绑定或解绑。 + - `if_bind`: 是否对线程进行绑定或解绑。 ## compileGraph @@ -156,7 +156,7 @@ public MSTensor getOutputByTensorName(String tensorName) - 参数 - - `tensorName`: 张量名。 + - `tensorName`: 张量名。 - 返回值 @@ -172,8 +172,8 @@ public boolean resize(List inputs, int[][] dims) - 参数 - - `inputs`: 模型对应的所有输入。 - - `dims`: 输入对应的新的shape，顺序注意要与inputs一致。 + - `inputs`: 模型对应的所有输入。 + - `dims`: 输入对应的新的shape，顺序注意要与inputs一致。 - 返回值 diff --git a/docs/api_java/source_zh_cn/model.md b/docs/api_java/source_zh_cn/model.md index f5230efd6a73aa22805fa7f0997ded57215ff474..c6081a8db8722e9475859a03565fa3c40dbacf39 100644 --- a/docs/api_java/source_zh_cn/model.md +++ b/docs/api_java/source_zh_cn/model.md @@ -6,7 +6,7 @@ import com.mindspore.lite.Model; Model定义了MindSpore Lite中的模型，便于计算图管理。 -# 公有成员函数 +## 公有成员函数 | function | | ------------------------------------------------------------ | @@ -25,8 +25,8 @@ public boolean loadModel(Context context, String modelName) - 参数 - - `context`: Android中的Context上下文 - - `modelName`: 模型文件名称 + - `context`: Android中的Context上下文 + - `modelName`: 模型文件名称 - 返回值 @@ -40,7 +40,7 @@ public boolean loadModel(String modelPath) - 参数 - - `modelPath`: 模型文件路径 + - `modelPath`: 模型文件路径 - 返回值 @@ -61,4 +61,3 @@ public void free() ``` 释放Model运行过程中动态分配的内存。 - diff --git a/docs/api_java/source_zh_cn/msconfig.md b/docs/api_java/source_zh_cn/msconfig.md index 7e868a60f115e4c3432e6b4b3e5aba03f45ea109..c60019b8c6f6196f8b9651a193e4f1b62021c36c 100644 --- a/docs/api_java/source_zh_cn/msconfig.md +++ b/docs/api_java/source_zh_cn/msconfig.md @@ -26,9 +26,9 @@ public boolean init(int deviceType, int threadNum, int cpuBindMode) - 参数 - - `deviceType`: 设备类型，`deviceType`在[com.mindspore.lite.config.DeviceType](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java)中定义。 - - `threadNum`: 线程数 - - `cpuBindMode`: CPU绑定模式，`cpuBindMode`在[com.mindspore.lite.config.CpuBindMode](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/CpuBindMode.java)中定义。 + - `deviceType`: 设备类型，`deviceType`在[com.mindspore.lite.config.DeviceType](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java)中定义。 + - `threadNum`: 线程数 + - `cpuBindMode`: CPU绑定模式，`cpuBindMode`在[com.mindspore.lite.config.CpuBindMode](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/CpuBindMode.java)中定义。 - 返回值 @@ -42,8 +42,8 @@ public boolean init(int deviceType, int threadNum) - 参数 - - `deviceType`: 设备类型，`deviceType`在[com.mindspore.lite.config.DeviceType](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java)中定义。 - - `threadNum`: 线程数。 + - `deviceType`: 设备类型，`deviceType`在[com.mindspore.lite.config.DeviceType](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java)中定义。 + - `threadNum`: 线程数。 - 返回值 @@ -57,7 +57,7 @@ public boolean init(int deviceType) - 参数 - - `deviceType`: 设备类型，`deviceType`在[com.mindspore.lite.config.DeviceType](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java)中定义。 + - `deviceType`: 设备类型，`deviceType`在[com.mindspore.lite.config.DeviceType](https://gitee.com/mindspore/mindspore/blob/master/mindspore/lite/java/java/app/src/main/java/com/mindspore/lite/config/DeviceType.java)中定义。 - 返回值 @@ -79,4 +79,4 @@ public boolean init() public void free() ``` -释放MSConfig运行过程中动态分配的内存。LiteSession init之后即可释放。 \ No newline at end of file +释放MSConfig运行过程中动态分配的内存。LiteSession init之后即可释放。 diff --git a/docs/api_java/source_zh_cn/mstensor.md b/docs/api_java/source_zh_cn/mstensor.md index b6e582f38ffdf38b51d03c03be413beb2e8d2ab5..ce7db9822c35f5698e68dd4c275cadf8d1f1312e 100644 --- a/docs/api_java/source_zh_cn/mstensor.md +++ b/docs/api_java/source_zh_cn/mstensor.md @@ -103,7 +103,7 @@ public void setData(byte[] data) 设定MSTensor的输入数据。 - 参数 - - `data`: byte[]类型的输入数据。 + - `data`: byte[]类型的输入数据。 ```java public void setData(ByteBuffer data) @@ -112,7 +112,7 @@ public void setData(ByteBuffer data) 设定MSTensor的输入数据。 - 参数 - - `data`: ByteBuffer类型的输入数据。 + - `data`: ByteBuffer类型的输入数据。 ## size @@ -145,4 +145,3 @@ public void free() ``` 释放MSTensor运行过程中动态分配的内存。 - diff --git a/docs/faq/source_en/faq.md b/docs/faq/source_en/faq.md index 3162c71c3a99de2fb0a85d3523625e5af82fbd12..3f9e7dc728257e22f9be031d44a9a5c6c8de5e9e 100644 --- a/docs/faq/source_en/faq.md +++ b/docs/faq/source_en/faq.md @@ -98,6 +98,7 @@ A: You can write the frequently-used environment settings to `~/.bash_profile` o ### Verifying the Installation Q: After MindSpore is installed on a CPU of a PC, an error message `the pointer[session] is null` is displayed during code verification. The specific code is as follows. How do I verify whether MindSpore is successfully installed? + ```python import numpy as np from mindspore import Tensor @@ -194,7 +195,7 @@ A: The MindSpore CPU version can be installed on Windows 10. For details about t Q: What can I do if an error message `wrong shape of image` is displayed when I use a model trained by MindSpore to perform prediction on a `28 x 28` digital image with white text on a black background? -A: The MNIST gray scale image dataset is used for MindSpore training. Therefore, when the model is used, the data must be set to a `28 x 28 `gray scale image, that is, a single channel. +A: The MNIST gray scale image dataset is used for MindSpore training. Therefore, when the model is used, the data must be set to a `28 x 28` gray scale image, that is, a single channel.
@@ -291,4 +292,4 @@ A: For details about script or model migration, please visit the [MindSpore offi Q: Does MindSpore provide open-source e-commerce datasets? -A: No. Please stay tuned for updates on the [MindSpore official website](https://www.mindspore.cn/en). \ No newline at end of file +A: No. Please stay tuned for updates on the [MindSpore official website](https://www.mindspore.cn/en). diff --git a/docs/faq/source_zh_cn/faq.md b/docs/faq/source_zh_cn/faq.md index c698b8e99038a66c38a50d16837edf8ef08c297b..4637908446d259eaacb7e1cac762acfe3c340d87 100644 --- a/docs/faq/source_zh_cn/faq.md +++ b/docs/faq/source_zh_cn/faq.md @@ -103,6 +103,7 @@ A：常用的环境变量设置写入到`~/.bash_profile` 或 `~/.bashrc`中， ### 安装验证 Q:个人电脑CPU环境安装MindSpore后验证代码时报错：`the pointer[session] is null`，具体代码如下，该如何验证是否安装成功呢？ + ```python import numpy as np from mindspore import Tensor @@ -121,6 +122,19 @@ A：CPU硬件平台安装MindSpore后测试是否安装成功,只需要执行命 ## 算子支持 +Q：`nn.Embedding`层与PyTorch相比缺少了`Padding`操作，有其余的算子可以实现吗？ + +A：在PyTorch中`padding_idx`的作用是将embedding矩阵中`padding_idx`位置的词向量置为0，并且反向传播时不会更新`padding_idx`位置的词向量。在MindSpore中，可以手动将embedding的`padding_idx`位置对应的权重初始化为0，并且在训练时通过`mask`的操作，过滤掉`padding_idx`位置对应的`Loss`。 + +
+ +Q：Operations中`Tile`算子执行到`__infer__`时`value`值为`None`，丢失了数值是怎么回事？ + +A：`Tile`算子的`multiples input`必须是一个常量（该值不能直接或间接来自于图的输入）。否则构图的时候会拿到一个`None`的数据，因为图的输入是在图执行的时候才传下去的，构图的时候拿不到图的输入数据。 +相关的资料可以看[相关文档](https://www.mindspore.cn/doc/note/zh-CN/master/constraints_on_network_construction.html)的“其他约束”。 + +
+ Q：官网的LSTM示例在Ascend上跑不通 A：目前LSTM只支持在GPU和CPU上运行，暂不支持硬件环境，您可以[点击这里](https://www.mindspore.cn/doc/note/zh-CN/master/operator_list_ms.html)查看算子支持情况。 @@ -135,6 +149,12 @@ A：这是TBE这个算子的限制，x的width必须大于kernel的width。CPU ## 网络模型 +Q：如何不将数据处理为MindRecord格式，直接进行训练呢？ + +A：可以使用自定义的数据加载方式 `GeneratorDataset`，具体可以参考[数据集加载](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/dataset_loading.html)文档中的自定义数据集加载。 + +
+ Q：MindSpore现支持直接读取哪些其他框架的模型和哪些格式呢？比如PyTorch下训练得到的pth模型可以加载到MindSpore框架下使用吗？ A： MindSpore采用protbuf存储训练参数，无法直接读取其他框架的模型。对于模型文件本质保存的就是参数和对应的值，可以用其他框架的API将参数读取出来之后，拿到参数的键值对，然后再加载到MindSpore中使用。比如想用其他框架训练好的ckpt文件，可以先把参数读取出来，再调用MindSpore的`save_checkpoint`接口，就可以保存成MindSpore可以读取的ckpt文件格式了。 @@ -211,6 +231,30 @@ A：MindSpore CPU版本已经支持在Windows 10系统中安装，具体安装 ## 后端运行 +Q：MindSpore如何实现早停功能？ + +A：可以自定义`callback`方法实现早停功能。 +例子：当loss降到一定数值后，停止训练。 + +```python +class EarlyStop(Callback): + def __init__(self, control_loss=1): + super(EarlyStep, self).__init__() + self._control_loss = control_loss + + def step_end(self, run_context): + cb_params = run_context.original_args() + loss = cb_params.net_outputs + if loss.asnumpy() < self._control_loss: + # Stop training + run_context._stop_requested = True + +stop_cb = EarlyStop(control_loss=1) +model.train(epoch_size, ds_train, callbacks=[stop_cb]) +``` + +
+ Q：请问自己制作的黑底白字`28*28`的数字图片，使用MindSpore训练出来的模型做预测，报错提示`wrong shape of image`是怎么回事？ A：首先MindSpore训练使用的灰度图MNIST数据集。所以模型使用时对数据是有要求的，需要设置为`28*28`的灰度图，就是单通道才可以。 @@ -275,6 +319,12 @@ A：MindSpore目前支持Python扩展，针对C++、Rust、Julia等语言的支 ## 特性支持 +Q：MindSpore有量化推理工具么？ + +A：[MindSpore Lite](https://www.mindspore.cn/lite)支持云侧量化感知训练的量化模型的推理，MindSpore Lite converter工具提供训练后量化以及权重量化功能，且功能在持续加强完善中。 + +
+ Q：MindSpore并行模型训练的优势和特色有哪些？ A：MindSpore分布式训练除了支持数据并行，还支持算子级模型并行，可以对算子输入tensor进行切分并行。在此基础上支持自动并行，用户只需要写单卡脚本，就能自动切分到多个节点并行执行。 @@ -325,4 +375,4 @@ A：关于脚本或者模型迁移，可以查询MindSpore官网中关于[网络 Q：MindSpore是否附带开源电商类数据集？ -A：暂时还没有，可以持续关注[MindSpore官网](https://www.mindspore.cn)。 \ No newline at end of file +A：暂时还没有，可以持续关注[MindSpore官网](https://www.mindspore.cn)。 diff --git a/docs/note/source_en/benchmark.md b/docs/note/source_en/benchmark.md index 72779fa5d8d92d8add4344715d19cbde74b209e8..9a715e46ceeffbf7a5319b36667743a6dccabbc9 100644 --- a/docs/note/source_en/benchmark.md +++ b/docs/note/source_en/benchmark.md @@ -15,7 +15,7 @@ -This document describes the MindSpore benchmarks. +This document describes the MindSpore benchmarks. For details about the MindSpore networks, see [Model Zoo](https://gitee.com/mindspore/mindspore/tree/master/model_zoo). ## Training Performance @@ -28,17 +28,17 @@ For details about the MindSpore networks, see [Model Zoo](https://gitee.com/mind | | | | | Ascend: 8 * Ascend 910
CPU: 192 Cores | Mixed | 256 | 16600 images/sec | 0.98 | | | | | | Ascend: 16 * Ascend 910
CPU: 384 Cores | Mixed | 256 | 32768 images/sec | 0.96 | -1. The preceding performance is obtained based on ModelArts, the HUAWEI CLOUD AI development platform. It is the average performance obtained by the Ascend 910 AI processor during the overall training process. +1. The preceding performance is obtained based on ModelArts, the HUAWEI CLOUD AI development platform. It is the average performance obtained by the Ascend 910 AI processor during the overall training process. 2. For details about other open source frameworks, see [ResNet-50 v1.5 for TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Classification/ConvNets/resnet50v1.5). ### BERT -| Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | +| Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-Large | Attention | zhwiki | 0.5.0-beta | Ascend: 1 * Ascend 910
CPU: 24 Cores | Mixed | 96 | 269 sentences/sec | - | | | | | | Ascend: 8 * Ascend 910
CPU: 192 Cores | Mixed | 96 | 2069 sentences/sec | 0.96 | -1. The preceding performance is obtained based on ModelArts, the HUAWEI CLOUD AI development platform. The network contains 24 hidden layers, the sequence length is 128 tokens, and the vocabulary contains 21128 tokens. +1. The preceding performance is obtained based on ModelArts, the HUAWEI CLOUD AI development platform. The network contains 24 hidden layers, the sequence length is 128 tokens, and the vocabulary contains 21128 tokens. 2. For details about other open source frameworks, see [BERT For TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT). ### Wide & Deep (data parallel) @@ -46,7 +46,7 @@ For details about the MindSpore networks, see [Model Zoo](https://gitee.com/mind | Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Wide & Deep | Recommend | Criteo | 0.6.0-beta | Ascend: 1 * Ascend 910
CPU: 24 Cores | Mixed | 16000 | 796892 samples/sec | - | -| | | | | Ascend: 8 * Ascend 910
CPU: 192 Cores | Mixed | 16000*8 | 4872849 samples/sec | 0.76 | +| | | | | Ascend: 8 \* Ascend 910
CPU: 192 Cores | Mixed | 16000*8 | 4872849 samples/sec | 0.76 | 1. The preceding performance is obtained based on Atlas 800, and the model is data parallel. 2. For details about other open source frameworks, see [Wide & Deep For TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Recommendation/WideAndDeep). @@ -56,9 +56,9 @@ For details about the MindSpore networks, see [Model Zoo](https://gitee.com/mind | Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Wide & Deep | Recommend | Criteo | 0.6.0-beta | Ascend: 1 * Ascend 910
CPU: 24 Cores | Mixed | 1000 | 68715 samples/sec | - | -| | | | | Ascend: 8 * Ascend 910
CPU: 192 Cores | Mixed | 8000*8 | 283830 samples/sec | 0.51 | -| | | | | Ascend: 16 * Ascend 910
CPU: 384 Cores | Mixed | 8000*16 | 377848 samples/sec | 0.34 | -| | | | | Ascend: 32 * Ascend 910
CPU: 768 Cores | Mixed | 8000*32 | 433423 samples/sec | 0.20 | +| | | | | Ascend: 8 \* Ascend 910
CPU: 192 Cores | Mixed | 8000*8 | 283830 samples/sec | 0.51 | +| | | | | Ascend: 16 \* Ascend 910
CPU: 384 Cores | Mixed | 8000*16 | 377848 samples/sec | 0.34 | +| | | | | Ascend: 32 \* Ascend 910
CPU: 768 Cores | Mixed | 8000*32 | 433423 samples/sec | 0.20 | 1. The preceding performance is obtained based on Atlas 800, and the model is model parallel. 2. For details about other open source frameworks, see [Wide & Deep For TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Recommendation/WideAndDeep). diff --git a/docs/note/source_en/constraints_on_network_construction.md b/docs/note/source_en/constraints_on_network_construction.md index 0c0a4f2cba6697639c016d1fb2618874363372a0..5936a2cf35f1d6c530dfb7327577dde135b761e9 100644 --- a/docs/note/source_en/constraints_on_network_construction.md +++ b/docs/note/source_en/constraints_on_network_construction.md @@ -1,7 +1,7 @@ # Constraints on Network Construction Using Python `Linux` `Ascend` `GPU` `CPU` `Model Development` `Beginner` `Intermediate` `Expert` - + - [Constraints on Network Construction Using Python](#constraints-on-network-construction-using-python) @@ -28,21 +28,26 @@ ## Overview + MindSpore can compile user source code based on the Python syntax into computational graphs, and can convert common functions or instances inherited from nn.Cell into computational graphs. Currently, MindSpore does not support conversion of any Python source code into computational graphs. Therefore, there are constraints on source code compilation, including syntax constraints and network definition constraints. As MindSpore evolves, the constraints may change. ## Syntax Constraints + ### Supported Python Data Types -* Number: supports `int`, `float`, and `bool`. Complex numbers are not supported. -* String -* List: supports the append method only. Updating a list will generate a new list. -* Tuple -* Dictionary: The type of key should be String. + +- Number: supports `int`, `float`, and `bool`. Complex numbers are not supported. +- String +- List: supports the append method only. Updating a list will generate a new list. +- Tuple +- Dictionary: The type of key should be String. + ### MindSpore Extended Data Type -* Tensor: Tensor variables must be defined instances. + +- Tensor: Tensor variables must be defined instances. ### Expression Types -| Operation | Description +| Operation | Description | :----------- |:-------- | Unary operator |`+`,`-`, and`not`. The operator `+` supports only scalars. | Binary operator |`+`, `-`, `*`, `/`, `%`, `**` and `//`. @@ -81,10 +86,11 @@ | `isinstance` | The usage principle is consistent with Python, but the second input parameter can only be the type defined by mindspore. ### Function Parameters -* Default parameter value: The data types `int`, `float`, `bool`, `None`, `str`, `tuple`, `list`, and `dict` are supported, whereas `Tensor` is not supported. -* Variable parameter: Functions with variable arguments is supported for training and inference. -* Key-value pair parameter: Functions with key-value pair parameters cannot be used for backward propagation on computational graphs. -* Variable key-value pair parameter: Functions with variable key-value pairs cannot be used for backward propagation on computational graphs. + +- Default parameter value: The data types `int`, `float`, `bool`, `None`, `str`, `tuple`, `list`, and `dict` are supported, whereas `Tensor` is not supported. +- Variable parameter: Functions with variable arguments is supported for training and inference. +- Key-value pair parameter: Functions with key-value pair parameters cannot be used for backward propagation on computational graphs. +- Variable key-value pair parameter: Functions with variable key-value pairs cannot be used for backward propagation on computational graphs. ### Operators @@ -101,55 +107,56 @@ ### Index operation -The index operation includes `tuple` and` Tensor`. The following focuses on the index value assignment and assignment operation of `Tensor`. The value takes` tensor_x [index] `as an example, and the assignment takes` tensor_x [index] = u` as an example for detailed description. Among them, tensor_x is a `Tensor`, which is sliced; index means the index, u means the assigned value, which can be` scalar` or `Tensor (size = 1)`. The index types are as follows: +The index operation includes `tuple` and `Tensor`. The following focuses on the index value assignment and assignment operation of `Tensor`. The value takes` tensor_x [index] `as an example, and the assignment takes`tensor_x [index] = u` as an example for detailed description. Among them, tensor_x is a `Tensor`, which is sliced; index means the index, u means the assigned value, which can be`scalar` or `Tensor (size = 1)`. The index types are as follows: - Slice index: index is `slice` - - Value: `tensor_x[start: stop: step]`, where Slice (start: stop: step) has the same syntax as Python, and will not be repeated here. - - Assignment: `tensor_x[start: stop: step] = u`. + - Value: `tensor_x[start: stop: step]`, where Slice (start: stop: step) has the same syntax as Python, and will not be repeated here. + - Assignment: `tensor_x[start: stop: step] = u`. - Ellipsis index: index is `ellipsis` - - Value: `tensor_x [...]`. - - Assignment: `tensor_x [...] = u`. + - Value: `tensor_x [...]`. + - Assignment: `tensor_x [...] = u`. - Boolean constant index: index is `True`, index is `False` is not supported temporarily. - - Value: `tensor_x[True]`. - - Assignment: Not supported yet. + - Value: `tensor_x[True]`. + - Assignment: Not supported yet. - Tensor index: index is `Tensor` - - Value: `tensor_x [index]`, `index` must be `Tensor` of data type `int32` or `int64`, + - Value: `tensor_x [index]`, `index` must be `Tensor` of data type `int32` or `int64`, the element value range is `[0, tensor_x.shape[0])`. - - Assignment: `tensor_x [index] = U`. - - `tensor_x` data type must be one of the following: `float16`, `float32`, `int8`, `uint8`. - - `index` must be `Tensor` of data type `int32`, the element value range is `[0, tensor_x.shape [0])`. - - `U` can be `Number`, `Tensor`, `Tuple` only containing `Number`, `Tuple` only containing `Tensor`. - - Single `Number` or every `Number` in `Tuple` must be the same type as `tensor_x`, ie + - Assignment: `tensor_x [index] = U`. + - `tensor_x` data type must be one of the following: `float16`, `float32`, `int8`, `uint8`. + - `index` must be `Tensor` of data type `int32`, the element value range is `[0, tensor_x.shape [0])`. + - `U` can be `Number`, `Tensor`, `Tuple` only containing `Number`, `Tuple` only containing `Tensor`. + - Single `Number` or every `Number` in `Tuple` must be the same type as `tensor_x`, ie When the data type of `tensor_x` is `uint8` or `int8`, the `Number` type should be `int`; When the data type of `tensor_x` is `float16` or `float32`, the `Number` type should be `float`. - - Single `Tensor` or every `Tensor in Tuple` must be consistent with the data type of `tensor_x`, + - Single `Tensor` or every `Tensor in Tuple` must be consistent with the data type of `tensor_x`, when single `Tensor`, the `shape` should be equal to or broadcast as `index.shape + tensor_x.shape [1:]`. - - `Tuple` containing `Number` must meet requirement: + - `Tuple` containing `Number` must meet requirement: `len (Tuple) = (index.shape + tensor_x.shape [1:]) [-1]`. - - `Tuple` containing `Tensor` must meet requirements: + - `Tuple` containing `Tensor` must meet requirements: the `shape` of each `Tensor` should be the same, `(len (Tuple),) + Tensor.shape` should be equal to or broadcast as `index.shape + tensor_x.shape [1:]`. - None constant index: index is `None` - - Value: `tensor_x[None]`, results are consistent with numpy. - - Assignment: Not supported yet. + - Value: `tensor_x[None]`, results are consistent with numpy. + - Assignment: Not supported yet. - tuple index: index is `tuple` - - The tuple element is a slice: - - Value: for example `tensor_x[::,: 4, 3: 0: -1]`. - - Assignment: for example `tensor_x[::,: 4, 3: 0: -1] = u`. - - The tuple element is Number: - - Value: for example `tensor_x[2,1]`. - - Assignment: for example `tensor_x[1,4] = u`. - - The tuple element is a mixture of slice and ellipsis: - - Value: for example `tensor_x[..., ::, 1:]`. - - Assignment: for example `tensor_x[..., ::, 1:] = u`. - - Not supported in other situations + - The tuple element is a slice: + - Value: for example `tensor_x[::,: 4, 3: 0: -1]`. + - Assignment: for example `tensor_x[::,: 4, 3: 0: -1] = u`. + - The tuple element is Number: + - Value: for example `tensor_x[2,1]`. + - Assignment: for example `tensor_x[1,4] = u`. + - The tuple element is a mixture of slice and ellipsis: + - Value: for example `tensor_x[..., ::, 1:]`. + - Assignment: for example `tensor_x[..., ::, 1:] = u`. + - Not supported in other situations The index value operation of tuple and list type, we need to focus on the index value operation of tuple or list whose element type is `nn.Cell`. This operation is currently only supported by the GPU backend in Graph mode, and its syntax format is like `layers[index](*inputs)`, the example code is as follows: + ```python class Net(nn.Cell): def __init__(self): @@ -162,60 +169,68 @@ The index value operation of tuple and list type, we need to focus on the index x = self.layers[index](x) return x ``` + The grammar has the following constraints: -* Only the index value operation of tuple or list whose element type is `nn.Cell` is supported. -* The index is a scalar `Tensor` of type `int32`, with a value range of `[-n, n)`, where `n` is the size of the tuple, and the maximum supported tuple size is 1000. -* The number, type and shape of the input data of the `Construct` function of each Cell element in the tuple are the same, and the number of data output after the `Construct` function runs, the type and shape are also the same. -* Each element in the tuple needs to be defined before the tuple is defined. -* This syntax does not support running branches as if, while, for and other control flow, except if the control condition of the control flow is constant. for example: - - Supported example: - ```python - class Net(nn.Cell): - def __init__(self, flag=True): - super(Net, self).__init__() - self.flag = flag - self.relu = nn.ReLU() - self.softmax = nn.Softmax() - self.layers = (self.relu, self.softmax) - def construct(self, x, index): - if self.flag: - x = self.layers[index](x) - return x - ``` - - Unsupported example: - ```python - class Net(nn.Cell): - def __init__(self): - super(Net, self).__init__() - self.relu = nn.ReLU() - self.softmax = nn.Softmax() - self.layers = (self.relu, self.softmax) +- Only the index value operation of tuple or list whose element type is `nn.Cell` is supported. +- The index is a scalar `Tensor` of type `int32`, with a value range of `[-n, n)`, where `n` is the size of the tuple, and the maximum supported tuple size is 1000. +- The number, type and shape size of the input data of the `Construct` function of each Cell element in the tuple are the same, and the number of data output after the `Construct` function runs, the type and shape size are also the same. +- Each element in the tuple needs to be defined before the tuple is defined. +- This syntax does not support running branches as if, while, for and other control flow, except if the control condition of the control flow is constant. for example: + - Supported example: - def construct(self, x, index, flag): - if flag: - x = self.layers[index](x) - return x - ``` + ```python + class Net(nn.Cell): + def __init__(self, flag=True): + super(Net, self).__init__() + self.flag = flag + self.relu = nn.ReLU() + self.softmax = nn.Softmax() + self.layers = (self.relu, self.softmax) + + def construct(self, x, index): + if self.flag: + x = self.layers[index](x) + return x + ``` + + - Unsupported example: + + ```python + class Net(nn.Cell): + def __init__(self): + super(Net, self).__init__() + self.relu = nn.ReLU() + self.softmax = nn.Softmax() + self.layers = (self.relu, self.softmax) + + def construct(self, x, index, flag): + if flag: + x = self.layers[index](x) + return x + ``` Tuple also support slice value operations, but do not support slice type as Tensor, support `tuple_x [start: stop: step]`, which has the same effect as Python, and will not be repeated here. ### Unsupported Syntax -Currently, the following syntax is not supported in network constructors: +Currently, the following syntax is not supported in network constructors: `raise`, `yield`, `async for`, `with`, `async with`, `assert`, `import`, and `await`. ## Network Definition Constraints ### Instance Types on the Entire Network -* Common Python function with the [@ms_function](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.html#mindspore.ms_function) decorator. -* Cell subclass inherited from [nn.Cell](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.nn.html#mindspore.nn.Cell). + +- Common Python function with the [@ms_function](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.html#mindspore.ms_function) decorator. +- Cell subclass inherited from [nn.Cell](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.nn.html#mindspore.nn.Cell). ### Network Input Type -* The training data input parameters of the entire network must be of the Tensor type. -* The generated ANF diagram cannot contain the following constant nodes: string constants, constants with nested tuples, and constants with nested lists. + +- The training data input parameters of the entire network must be of the Tensor type. +- The generated ANF diagram cannot contain the following constant nodes: string constants, constants with nested tuples, and constants with nested lists. ### Network Graph Optimization + During graph optimization at the ME frontend, the dataclass, dictionary, list, and key-value pair types are converted to tuple types, and the corresponding operations are converted to tuple operations. ### Network Construction Components @@ -230,33 +245,36 @@ Currently, the following syntax is not supported in network constructors: | Composite operator |[mindspore/ops/composite/*](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.ops.html). | Operator generated by constexpr |Uses the value generated by [@constexpr](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.ops.html#mindspore.ops.constexpr) to calculate operators. - ### Other Constraints + 1. Input parameters of the `construct` function on the entire network and parameters of functions modified by the `ms_function` decorator are generalized during the graph compilation and cannot be passed to operators as constant input. Therefore, in graph mode, the parameter passed to the entry network can only be `Tensor`. As shown in the following example: - - * The following is an example of incorrect input: + + - The following is an example of incorrect input: + ```python class ExpandDimsTest(Cell): def __init__(self): super(ExpandDimsTest, self).__init__() self.expandDims = P.ExpandDims() - + def construct(self, input_x, input_axis): return self.expandDims(input_x, input_axis) expand_dim = ExpandDimsTest() input_x = Tensor(np.random.randn(2,2,2,2).astype(np.float32)) expand_dim(input_x, 0) ``` + In the example, `ExpandDimsTest` is a single-operator network with two inputs: `input_x` and `input_axis`. The second input of the `ExpandDims` operator must be a constant. This is because `input_axis` is required when the output dimension of the `ExpandDims` operator is deduced during graph compilation. As the network parameter input, the value of `input_axis` is generalized into a variable and cannot be determined. As a result, the output dimension of the operator cannot be deduced, causing the graph compilation failure. Therefore, the input required by deduction in the graph compilation phase must be a constant. In the API, the parameters of this type of operator that require constant input will be explained, marked `const input is needed`. - - * Directly enter the needed value or a member variable in a class for the constant input of the operator in the construct function. The following is an example of correct input: + + - Directly enter the needed value or a member variable in a class for the constant input of the operator in the construct function. The following is an example of correct input: + ```python class ExpandDimsTest(Cell): def __init__(self, axis): super(ExpandDimsTest, self).__init__() self.expandDims = P.ExpandDims() self.axis = axis - + def construct(self, input_x): return self.expandDims(input_x, self.axis) axis = 0 @@ -264,52 +282,56 @@ Currently, the following syntax is not supported in network constructors: input_x = Tensor(np.random.randn(2,2,2,2).astype(np.float32)) expand_dim(input_x) ``` - + 2. It is not allowed to modify `non-Parameter` type data members of the network. Examples are as follows: - ``` + ```python class Net(Cell): def __init__(self): super(Net, self).__init__() self.num = 2 self.par = Parameter(Tensor(np.ones((2, 3, 4))), name="par") - + def construct(self, x, y): return x + y ``` + In the network defined above, `self.num` is not a `Parameter` and cannot be modified, but `self.par` is a `Parameter` and can be modified. 3. When an undefined class member is used in the `construct` function, it will be treated as `None` instead of throwing `AttributeError` like the Python interpreter. Examples are as follows: - - ``` + + ```python class Net(Cell): def __init__(self): super(Net, self).__init__() - + def construct(self, x): return x + self.y ``` + In the network defined above, the undefined class member `self.y` is used in `construct`, and `self.y` will be treated as `None`. 4. When using the control flow of `if-else` in the `construct` function, the data types returned by `if` and `else` or the data types of the same variable after being updated must be the same. Examples are as follows: - ``` + + ```python class NetReturn(Cell): def __init__(self): super(NetReturn, self).__init__() - + def construct(self, x, y, m, n): if x > y: return m else: return n ``` + In the network `NetReturn` defined above, the `if-else` control flow is used in `construct`, then the data type of `m` returned by the `if` branch and the data type of `n` returned by the `else` branch must be consistent. - - ``` + + ```python class NetAssign(Cell): def __init__(self): super(NetAssign, self).__init__() - + def construct(self, x, y, m, n): if x > y: out = m @@ -317,4 +339,5 @@ Currently, the following syntax is not supported in network constructors: out = n return out ``` + In the network `NetAssign` defined above, the `if-else` control flow is used in the `construct`, then the data types of the `out` after the update of the `if` branch and the update of the `else` branch must be consistent. diff --git a/docs/note/source_en/design/mindarmour/differential_privacy_design.md b/docs/note/source_en/design/mindarmour/differential_privacy_design.md index c57af833f17e3642d4719def21170d44e60626af..7038864e653ba412238865bae8c9e12e72f7a735 100644 --- a/docs/note/source_en/design/mindarmour/differential_privacy_design.md +++ b/docs/note/source_en/design/mindarmour/differential_privacy_design.md @@ -26,14 +26,13 @@ The Differential-Privacy module of MindArmour implements the differential privac Figure 1 shows an overall design of differential privacy training, and mainly including differential privacy noise mechanisms (DP mechanisms), a differential privacy optimizer (DP optimizer), and a privacy monitor. - ### DP Optimizer DP optimizer inherits capabilities of the MindSpore optimizer and uses the DP mechanisms to scramble and protect gradients. Currently, MindArmour provides three types of DP optimizers: constant Gaussian optimizer, adaptive Gaussian optimizer, and adaptive clipping optimizer. Each type of DP optimizer adds differential privacy protection capabilities to common optimizers such as SGD and Momentum from different perspectives. -* Constant Gaussian optimizer is a DP optimizer for non-adaptive Gaussian noise. The advantage is that the differential privacy budget ϵ can be strictly controlled. The disadvantage is that in the model training process, the noise amount added in each step is fixed. If the number of training steps is too large, the noise in the later phase of training makes the model convergence difficult, or even causes the performance to deteriorate greatly and the model availability to be poor. -* Adaptive Gaussian optimizer adaptively adjusts the standard deviation to adjust the Gaussian distribution noise. In the initial phase of model training, a large amount of noise is added. As the model gradually converges, the noise amount gradually decreases, and the impact of the noise on the model availability is reduced. A disadvantage of the adaptive Gaussian noise is that a differential privacy budget cannot be strictly controlled. -* Adaptive clipping optimizer is a DP optimizer that adaptively adjusts a clipping granularity. Gradient clipping is an important operation in differential privacy training. The adaptive clipping optimizer can control a ratio of gradient clipping to fluctuate within a given range and control the gradient clipping granularity during training steps. +- Constant Gaussian optimizer is a DP optimizer for non-adaptive Gaussian noise. The advantage is that the differential privacy budget ϵ can be strictly controlled. The disadvantage is that in the model training process, the noise amount added in each step is fixed. If the number of training steps is too large, the noise in the later phase of training makes the model convergence difficult, or even causes the performance to deteriorate greatly and the model availability to be poor. +- Adaptive Gaussian optimizer adaptively adjusts the standard deviation to adjust the Gaussian distribution noise. In the initial phase of model training, a large amount of noise is added. As the model gradually converges, the noise amount gradually decreases, and the impact of the noise on the model availability is reduced. A disadvantage of the adaptive Gaussian noise is that a differential privacy budget cannot be strictly controlled. +- Adaptive clipping optimizer is a DP optimizer that adaptively adjusts a clipping granularity. Gradient clipping is an important operation in differential privacy training. The adaptive clipping optimizer can control a ratio of gradient clipping to fluctuate within a given range and control the gradient clipping granularity during training steps. ### DP Mechanisms @@ -41,26 +40,24 @@ The noise mechanism is a basis for building a differential privacy training capa ### Monitor -Monitor provides callback functions such as Rényi differential privacy (RDP) and zero-concentrated differential privacy (ZCDP) to monitor the differential privacy budget of the model. +Monitor provides callback functions such as Rényi differential privacy (RDP) and zero-concentrated differential privacy (ZCDP) to monitor the differential privacy budget of the model. -* ZCDP[2] +- ZCDP[2] ZCDP is a loose differential privacy definition. It uses the Rényi divergence to measure the distribution difference of random functions on adjacent datasets. -* RDP[3] +- RDP[3] RDP is a more general differential privacy definition based on the Rényi divergence. It uses the Rényi divergence to measure the distribution difference between two adjacent datasets. - Compared with traditional differential privacy, ZCDP and RDP provide stricter privacy budget upper bound guarantee. - ## Code Implementation -* [mechanisms.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/mechanisms/mechanisms.py): implements the noise generation mechanism required by differential privacy training, including simple Gaussian noise, adaptive Gaussian noise, and adaptive clipping Gaussian noise. -* [optimizer.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/optimizer/optimizer.py): implements the fundamental logic of using the noise generation mechanism to add noise during backward propagation. -* [monitor.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/monitor/monitor.py): implements the callback function for computing the differential privacy budget. During model training, the current differential privacy budget is returned. -* [model.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/train/model.py): implements the logic of computing the loss and gradient as well as the gradient truncation logic of differential privacy training, which is the entry for users to use the differential privacy training capability. +- [mechanisms.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/mechanisms/mechanisms.py): implements the noise generation mechanism required by differential privacy training, including simple Gaussian noise, adaptive Gaussian noise, and adaptive clipping Gaussian noise. +- [optimizer.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/optimizer/optimizer.py): implements the fundamental logic of using the noise generation mechanism to add noise during backward propagation. +- [monitor.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/monitor/monitor.py): implements the callback function for computing the differential privacy budget. During model training, the current differential privacy budget is returned. +- [model.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/train/model.py): implements the logic of computing the loss and gradient as well as the gradient truncation logic of differential privacy training, which is the entry for users to use the differential privacy training capability. ## References diff --git a/docs/note/source_en/design/mindarmour/fuzzer_design.md b/docs/note/source_en/design/mindarmour/fuzzer_design.md index dbb4feeb0a5ce1989b0bf9178892787740ff369f..34cfc563dc10d75662950114a1db2337fe5f9596 100644 --- a/docs/note/source_en/design/mindarmour/fuzzer_design.md +++ b/docs/note/source_en/design/mindarmour/fuzzer_design.md @@ -2,7 +2,6 @@ `Linux` `Ascend` `GPU` `CPU` `Data Preparation` `Model Development` `Model Training` `Model Optimization` `Enterprise` `Expert` - - [AI Model Security Testing](#ai-model-security-testing) @@ -71,4 +70,4 @@ Through multiple rounds of mutations, you can obtain a series of variant data in [1] Pei K, Cao Y, Yang J, et al. Deepxplore: Automated whitebox testing of deep learning systems[C]//Proceedings of the 26th Symposium on Operating Systems Principles. ACM, 2017: 1-18. -[2] Ma L, Juefei-Xu F, Zhang F, et al. Deepgauge: Multi-granularity testing criteria for deep learning systems[C]//Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 2018: 120-131. \ No newline at end of file +[2] Ma L, Juefei-Xu F, Zhang F, et al. Deepgauge: Multi-granularity testing criteria for deep learning systems[C]//Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 2018: 120-131. diff --git a/docs/note/source_en/design/mindinsight/graph_visual_design.md b/docs/note/source_en/design/mindinsight/graph_visual_design.md index a0e0c8918f537edeeb2586ca23b7bc25f310e16c..8633d64951454033d95e05a8302c4cdeb825a59d 100644 --- a/docs/note/source_en/design/mindinsight/graph_visual_design.md +++ b/docs/note/source_en/design/mindinsight/graph_visual_design.md @@ -21,9 +21,9 @@ The computational graph visualization function is mainly used in the following scenarios: - - View a data flow direction of operators and a model structure when programming a deep learning neural network. - - View the input and output nodes of a specified node and attributes of a queried node. - - Trace data, including data dimension and type changes when debugging a network. +- View a data flow direction of operators and a model structure when programming a deep learning neural network. +- View the input and output nodes of a specified node and attributes of a queried node. +- Trace data, including data dimension and type changes when debugging a network. ## Overall Design @@ -71,4 +71,4 @@ RESTful API is used for data interaction between the MindInsight frontend and ba #### File API Design Data interaction between MindSpore and MindInsight uses the data format defined by [Protocol Buffer](https://developers.google.cn/protocol-buffers/docs/pythontutorial). -The main entry is the [summary.proto file](https://gitee.com/mindspore/mindinsight/blob/master/mindinsight/datavisual/proto_files/mindinsight_summary.proto). A message object of a computational graph is defined as `GraphProto`. For details about `GraphProto`, see the [anf_ir.proto file](https://gitee.com/mindspore/mindinsight/blob/master/mindinsight/datavisual/proto_files/mindinsight_anf_ir.proto). \ No newline at end of file +The main entry is the [summary.proto file](https://gitee.com/mindspore/mindinsight/blob/master/mindinsight/datavisual/proto_files/mindinsight_summary.proto). A message object of a computational graph is defined as `GraphProto`. For details about `GraphProto`, see the [anf_ir.proto file](https://gitee.com/mindspore/mindinsight/blob/master/mindinsight/datavisual/proto_files/mindinsight_anf_ir.proto). diff --git a/docs/note/source_en/design/mindinsight/tensor_visual_design.md b/docs/note/source_en/design/mindinsight/tensor_visual_design.md index f21b670f1f37133fb8909b2ae1ed009f9f960f32..86a364148c6002864ea62d9c5b38bda03775674c 100644 --- a/docs/note/source_en/design/mindinsight/tensor_visual_design.md +++ b/docs/note/source_en/design/mindinsight/tensor_visual_design.md @@ -60,7 +60,8 @@ In tensor visualization, there are file API and RESTful API. The file API is the #### File API Design The `summary.proto` file is the main entry. TensorProto data is stored in the summary value, as shown in the following: -``` + +```cpp { message Summary { message Image { @@ -69,7 +70,7 @@ The `summary.proto` file is the main entry. TensorProto data is stored in the su required int32 width = 2; ... } - + message Histogram { message bucket{ // Counting number of values fallen in [left, left + width). @@ -78,7 +79,7 @@ The `summary.proto` file is the main entry. TensorProto data is stored in the su required double width = 2; required int64 count = 3; } - + repeated bucket buckets = 1; ... } @@ -86,7 +87,7 @@ The `summary.proto` file is the main entry. TensorProto data is stored in the su message Value { // Tag name for the data. required string tag = 1; - + // Value associated with the tag. oneof value { float scalar_value = 3; @@ -100,4 +101,5 @@ The `summary.proto` file is the main entry. TensorProto data is stored in the su repeated Value value = 1; } ``` -TensorProto is defined in the [anf_ir.proto](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/utils/anf_ir.proto) file. \ No newline at end of file + +TensorProto is defined in the [anf_ir.proto](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/utils/anf_ir.proto) file. diff --git a/docs/note/source_en/design/mindinsight/training_visual_design.md b/docs/note/source_en/design/mindinsight/training_visual_design.md index 86451518c897bd241b0900664cecc207c2c3ce07..05cadfe220ab59d397cc4e2342d2fbf6d43325b6 100644 --- a/docs/note/source_en/design/mindinsight/training_visual_design.md +++ b/docs/note/source_en/design/mindinsight/training_visual_design.md @@ -5,16 +5,16 @@ - [Overall Design of Training Visualization](#overall-design-of-training-visualization) - - [Logical Architecture of Training Visualization](#logical-architecture-of-training-visualization) - - [Architecture of Training Information Collection](#architecture-of-training-information-collection) - - [Architecture of Training Information Analysis and Display](#architecture-of-training-information-analysis-and-display) - - [Code Organization](#code-organization) - - [Training Visualization Data Model](#training-visualization-data-model) - - [Training Information Data Flow](#training-information-data-flow) - - [Data Model](#data-model) - - [Training Job](#training-job) - - [Lineage Data](#lineage-data) - - [Training Process Data](#training-process-data) + - [Logical Architecture of Training Visualization](#logical-architecture-of-training-visualization) + - [Architecture of Training Information Collection](#architecture-of-training-information-collection) + - [Architecture of Training Information Analysis and Display](#architecture-of-training-information-analysis-and-display) + - [Code Organization](#code-organization) + - [Training Visualization Data Model](#training-visualization-data-model) + - [Training Information Data Flow](#training-information-data-flow) + - [Data Model](#data-model) + - [Training Job](#training-job) + - [Lineage Data](#lineage-data) + - [Training Process Data](#training-process-data) @@ -105,7 +105,7 @@ MindInsight uses directories to distinguish different training jobs. To distingu In MindInsight code, a training job is called a TrainJob. A TrainJob ID is the name of the directory where the training log is located, for example, ./train_my_lenet_1. -During a training process, a lineage data file (whose name ends with _lineage) and a training process data file (whose name ends with _MS) are generated. The lineage data mainly describes an invariant attribute of the training from a global perspective, for example, a dataset path used for training, an optimizer used for training, and user-defined lineage information. The most prominent feature of the lineage data file is that it does not change during the training process. The training process data mainly describes a change status of the training, for example, a loss value, parameter distribution, and image data sent to the model in a step. The most prominent feature of the training process data file is that each step changes. +During a training process, a lineage data file (whose name ends with _lineage) and a training process data file (whose name ends with_MS) are generated. The lineage data mainly describes an invariant attribute of the training from a global perspective, for example, a dataset path used for training, an optimizer used for training, and user-defined lineage information. The most prominent feature of the lineage data file is that it does not change during the training process. The training process data mainly describes a change status of the training, for example, a loss value, parameter distribution, and image data sent to the model in a step. The most prominent feature of the training process data file is that each step changes. It should be noted that the classification about whether the training information changes is not absolute. For example, the training process data file contains computational graph data, which is determined when the training starts. @@ -129,4 +129,4 @@ The lineage data describes the invariant attribute of a training from a global p - Data Query and Display - When displaying data, you might want to see how the data under a tag changes with the training process. Therefore, when querying data, you do not need to specify the step number. Instead, you can specify the training job, plugin name, and tag to query data of all steps under the tag. \ No newline at end of file + When displaying data, you might want to see how the data under a tag changes with the training process. Therefore, when querying data, you do not need to specify the step number. Instead, you can specify the training job, plugin name, and tag to query data of all steps under the tag. diff --git a/docs/note/source_en/design/mindspore/architecture_lite.md b/docs/note/source_en/design/mindspore/architecture_lite.md index d78fc89147c30eea967c849d5291ce61ed0d42d6..05c69cede3232a2db8a890f63606dc79cede1ba2 100644 --- a/docs/note/source_en/design/mindspore/architecture_lite.md +++ b/docs/note/source_en/design/mindspore/architecture_lite.md @@ -1,7 +1,7 @@ # Overall Architecture (Lite) `Linux` `Windows` `On Device` `Inference Application` `Intermediate` `Expert` `Contributor` - + The overall architecture of MindSpore Lite is as follows: @@ -14,8 +14,8 @@ The overall architecture of MindSpore Lite is as follows: - **Backend:** optimizes graphs based on IR, including graph high level optimization (GHLO), graph low level optimization (GLLO), and quantization. GHLO is responsible for hardware-independent optimization, such as operator fusion and constant folding. GLLO is responsible for hardware-related optimization. Quantizer supports quantization methods after training, such as weight quantization and activation value quantization. -- **Runtime:** inference runtime of intelligent devices. Sessions are responsible for session management and provide external APIs. The thread pool and parallel primitives are responsible for managing the thread pool used for graph execution. Memory allocation is responsible for memory overcommitment of each operator during graph execution. The operator library provides the CPU and GPU operators. +- **Runtime:** inference runtime of intelligent devices. Sessions are responsible for session management and provide external APIs. The thread pool and parallel primitives are responsible for managing the thread pool used for graph execution. Memory allocation is responsible for memory overcommitment of each operator during graph execution. The operator library provides the CPU and GPU operators. - **Micro:** runtime of IoT devices, including the model generation .c file, thread pool, memory overcommitment, and operator library. -Runtime and Micro share the underlying infrastructure layers, such as the operator library, memory allocation, thread pool, and parallel primitives. +Runtime and Micro share the underlying infrastructure layers, such as the operator library, memory allocation, thread pool, and parallel primitives. diff --git a/docs/note/source_en/design/mindspore/distributed_training_design.md b/docs/note/source_en/design/mindspore/distributed_training_design.md index 79de486296bad80e349dcc714ec16a35067c375e..cf963d8a9f819eeecf08184300edf060361f3834 100644 --- a/docs/note/source_en/design/mindspore/distributed_training_design.md +++ b/docs/note/source_en/design/mindspore/distributed_training_design.md @@ -24,7 +24,6 @@ With the rapid development of deep learning, the number of datasets and parameters are growing exponentially to improve the accuracy and generalization capability of neural networks. Parallel distributed training has become a development trend to resolve the performance bottleneck of ultra-large scale networks. MindSpore supports the mainstream distributed training paradigm and develops an automatic hybrid parallel solution. The following describes the design principles of several parallel training modes and provides guidance for users to perform custom development. - ## Concepts ### Collective Communication @@ -74,7 +73,6 @@ This section describes how the data parallel mode `ParallelMode.DATA_PARALLEL` w - [grad_reducer.py](https://gitee.com/mindspore/mindspore/blob/master/mindspore/nn/wrap/grad_reducer.py): This file implements the gradient aggregation process. After the input parameter `grads` is expanded by using `HyperMap`, the `AllReduce` operator is inserted. The global communication group is used. You can also perform custom development by referring to this section based on your network requirements. In MindSpore, standalone and distributed execution shares a set of network encapsulation APIs. In the `Cell`, `ParallelMode` is used to determine whether to perform gradient aggregation. For details about the network encapsulation APIs, see the `TrainOneStepCell` code implementation. - ## Automatic Parallelism As a key feature of MindSpore, automatic parallelism is used to implement hybrid parallel training that combines automatic data parallelism and model parallelism. It aims to help users express the parallel algorithm logic using standalone scripts, reduce the difficulty of distributed training, improve the algorithm R&D efficiency, and maintain the high performance of training. This section describes how the automatic parallel mode `ParallelMode.AUTO_PARALLEL` and semi-automatic parallel mode `ParallelMode.SEMI_AUTO_PARALLEL` work in MindSpore. @@ -86,19 +84,19 @@ As a key feature of MindSpore, automatic parallelism is used to implement hybrid 1. Distributed operator and tensor layout As shown in the preceding figure, the automatic parallel process traverses the standalone forward ANF graphs and performs shard modeling on tensors in the unit of distributed operator, indicating how the input and output tensors of an operator are distributed to each device of the cluster, that is, the tensor layout. Users do not need to know which device runs which slice of a model. The framework automatically schedules and allocates model slices. - + To obtain the tensor layout model, each operator has a shard strategy, which indicates the shard status of each input of the operator in the corresponding dimension. Generally, tensors can be sharded in any dimension as long as the value is a multiple of 2, and the even distribution principle is met. The following figure shows an example of the three-dimensional `BatchMatmul` operation. The parallel strategy consists of two tuples, indicating the sharding of `input` and `weight`, respectively. Elements in a tuple correspond to tensor dimensions one by one. `2^N` indicates the shard unit, and `1` indicates that the tuple is not sharded. If you want to express a parallel data shard strategy, that is, only data in the `batch` dimension of `input` is sharded, and data in other dimensions are not sharded, you can use `strategy=((2^N, 1, 1),(1, 1, 1))`. If you want to express a parallel model shard strategy, that is, only model in the non-`batch` dimension of `weight` is sharded, for example, only the `channel` dimension is sharded, you can use `strategy=((1, 1, 1),(1, 1, 2^N))`. If you want to express a hybrid parallel shard strategy, one of which is `strategy=((2^N, 1, 1),(1, 1, 2^N))`. ![Operator Sharding Definition](./images/operator_split.png) - - Based on the shard strategy of an operator, the framework automatically derives the distribution model of input tensors and output tensors of the operator. This distribution model consists of `device_matrix`, `tensor_shape`, and `tensor map`, which indicate the device matrix shape, tensor shape, and mapping between devices and tensor dimensions, respectively. Based on the tensor layout model, distributed operator determines whether to insert extra computation and communication operations in the graph to ensure that the operator computing logic is correct. + + Based on the shard strategy of an operator, the framework automatically derives the distribution model of input tensors and output tensors of the operator. This distribution model consists of `device_matrix`, `tensor_shape`, and `tensor map`, which indicate the device matrix shape, tensor shape, and mapping between devices and tensor dimensions, respectively. Based on the tensor layout model, distributed operator determines whether to insert extra computation and communication operations in the graph to ensure that the operator computing logic is correct. 2. Tensor Redistribution When the output tensor model of an operator is inconsistent with the input tensor model of the next operator, computation and communication operations need to be introduced to implement the change between tensor layouts. The automatic parallel process introduces the tensor redistribution algorithm, which can be used to derive the communication conversion operations between random tensor layouts. The following three examples represent a parallel computing process of the formula `Z=(X×W)×V`, that is, a `MatMul` operation of two two-dimensional matrices, and show how to perform conversion between different parallel modes. - + In example 1, the output of the first data parallel matrix multiplication is sharded in the row rection, and the input of the second model parallel matrix multiplication requires full tensors. The framework automatically inserts the `AllGather` operator to implement redistribution. - + ![Tensor Redistribution](./images/tensor_redistribution1.png) In example 2, the output of parallel matrix multiplication of the first model is sharded in the column direction, and the input of parallel matrix multiplication of the second model is sharded in the row direction. The framework automatically inserts a communication operator equivalent to the `AlltoAll` operation in collective communication to implement redistribution. @@ -114,9 +112,8 @@ As a key feature of MindSpore, automatic parallelism is used to implement hybrid 3. Efficient parallel strategy search algorithm The `SEMI_AUTO_PARALLEL` semi-automatic parallel mode indicates that you manually configure the parallel strategy for operators when you are familiar with the operator sharding representation. This mode is helpful for manual optimization, with certain commissioning difficulty. You need to master the parallel principle and obtain a high-performance parallel solution based on the network structure and cluster topology. To further help users accelerate the parallel network training process, the automatic parallel mode `AUTO_PARALLEL` introduces the automatic search feature of the parallel strategy on the basis of the semi-automatic parallel mode. Automatic parallelism builds cost models based on the hardware platform, and calculates the computation cost, memory cost, and communication cost of a certain amount of data and specific operators based on different parallel strategies Then, by using the dynamic programming algorithm or recursive programming algorithm and taking the memory upper limit of a single device as a constraint condition, a parallel strategy with optimal performance is efficiently searched out. - - Strategy search replaces manual model sharding and provides a high-performance sharding solution within a short period of time, greatly reducing the threshold for parallel training. + Strategy search replaces manual model sharding and provides a high-performance sharding solution within a short period of time, greatly reducing the threshold for parallel training. 4. Convenient distributed automatic differentiation @@ -139,6 +136,5 @@ As a key feature of MindSpore, automatic parallelism is used to implement hybrid 5. Entire graph sharding - [step_auto_parallel.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/frontend/parallel/step_auto_parallel.h), and [step_parallel.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/frontend/parallel/step_parallel.h): The two files contain the core implementation of the automatic parallel process. `step_auto_parallel.h` calls the strategy search process and generates the `OperatorInfo` of the distributed operator. Then in `step_parallel.h`, processes such as operator sharding and tensor redistribution are processed to reconstruct the standalone computing graph in distributed mode. - 6. Backward propagation of communication operators - [grad_comm_ops.py](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ops/_grad/grad_comm_ops.py): This file defines the backward propagation of communication operators, such as `AllReduce` and `AllGather`. diff --git a/docs/note/source_en/design/mindspore/mindir.md b/docs/note/source_en/design/mindspore/mindir.md index 7d3ba3385c9ff0c399a09002e35742f886ee7975..59f55e31952a36ce34cce402f9a8f328a3f835b3 100644 --- a/docs/note/source_en/design/mindspore/mindir.md +++ b/docs/note/source_en/design/mindspore/mindir.md @@ -21,13 +21,16 @@ ## Overview + An intermediate representation (IR) is a representation of a program between the source and target languages, which facilitates program analysis and optimization for the compiler. Therefore, the IR design needs to consider the difficulty in converting the source language to the target language, as well as the ease-of-use and performance of program analysis and optimization. MindSpore IR (MindIR) is a function-style IR based on graph representation. Its core purpose is to serve automatic differential transformation. Automatic differentiation uses the transformation method based on the function-style programming framework. Therefore, IR uses the semantics close to that of the ANF function. In addition, a manner of representation based on an explicit dependency graph is used by referring to excellent designs of Sea of Nodes[1] and Thorin[2]. ## Syntax + ANF is a simple IR commonly used during functional programming. The ANF syntax is defined as follows: -``` + +```python ::= NUMBER | STRING | VAR | BOOLEAN | PRIMOP | (lambda (VAR …) ) ::= ( …) @@ -35,17 +38,20 @@ ANF is a simple IR commonly used during functional programming. The ANF syntax i ::= (let ([VAR ]) ) | | ``` + Expressions in the ANF are classified into atomic expressions (aexp) and compound expressions (cexp). An atomic expression indicates a constant value, a variable, or an anonymous function. A compound expression consists of multiple atomic expressions, indicating that an anonymous function or primitive function call. The first input expression of a compound expression is the called function, and the other input expressions are the called parameters. The syntax of MindIR is inherited from the ANF and is defined as follows: -``` + +```python ::= | ::= Parameter ::= Scalar | Named | Tensor | Type | Shape - | Primitive | MetaFuncGraph | FuncGraph + | Primitive | MetaFuncGraph | FuncGraph ::= ( …) ::= | ``` + ANode in a MindIR corresponds to the atomic expression of ANF. ANode has two subclasses: ValueNode and ParameterNode. ValueNode refers to a constant node, which can carry a constant value (such as a scalar, symbol, tensor, type, and dimension), a primitive function (Primitive), a metafunction (MetaFuncGraph), or a common function (FuncGraph). In functional programming, the function definition itself is a value. ParameterNode refers to a parameter node, which indicates the formal parameter of a function. CNode in a MindIR corresponds to the compound expression of ANF, indicating a function call. @@ -53,7 +59,9 @@ CNode in a MindIR corresponds to the compound expression of ANF, indicating a fu During automatic differentiation of MindSpore, the gradient contribution of ParameterNode and CNode are calculated, and the final gradient of ParameterNode is returned. The gradient of ValueNode is not calculated. ## Example + The following uses a program code segment as an example to help you understand MindIR. + ```python def func(x, y): return x / y @@ -65,8 +73,10 @@ def test_f(x, y): c = b * func(a, b) return c ``` + The ANF corresponding to the Python code is as follows: -``` + +```python lambda (x, y) let a = x - 1 in let b = a + y in @@ -77,26 +87,31 @@ lambda (x, y) let c = b * %1 in c end ``` + The corresponding MindIR is [ir.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_en/design/mindspore/images/ir/ir.dot). -![](./images/ir/ir.png) +![image](./images/ir/ir.png) In a MindIR, a function graph (FuncGraph) indicates the definition of a common function. A directed acyclic graph (DAG) usually consists of ParameterNode, ValueNode, and CNode, which clearly shows the calculation process from parameters to return values. As shown in the preceding figure, the `test_f` and `func` functions in the Python code are converted into two function graphs. The `x` and `y` parameters are converted into ParameterNode in the function graphs, and each expression is converted into a CNode. The first input of CNode links to the called functions, for example, `add`, `func`, and `return` in the figure. It should be noted that these nodes are all `ValueNode` because they are considered as constant function values. Other input of CNode links to the called parameters. The parameter values can be obtained from the ParameterNode, ValueNode, and other CNode. In the ANF, each expression is bound as a variable by using the let expression, and the dependency on the expression output is represented by referencing the variable. In the MindIR, each expression is bound as a node, and the dependency is represented by using the directed edges between nodes. ## Saving IR + `context.set_context(save_graphs=True)` is used to save the intermediate code in each compilation phase. The intermediate code can be saved in two formats. One is the text format with the suffix `.ir`, and the other is the graphical format with the suffix `.dot`. When the network scale is small, you are advised to use the graphical format that is more intuitive. When the network scale is large, you are advised to use the text format that is more efficient. You can run the graphviz command to convert a .dot file to the picture format. For example, you can run the `dot -Tpng *.dot -o *.png` command to convert a .dot file to a .png file. ## Function-style Semantics + Compared with traditional computational graphs, MindIR can not only express data dependency between operators, but also express rich function-style semantics. + ### Higher-Order Functions + In a MindIR, a function is defined by a subgraph. However, the function itself can be transferred as the input or output of other higher-order functions. In the following simple example, the `f` function is transferred as a parameter into the `g` function. Therefore, the `g` function is a higher-order function that receives function input, and the actual call site of the `f` function is inside the `g` function. -``` +```python @ms_function def hof(x): def f(x): @@ -108,14 +123,16 @@ def hof(x): ``` The corresponding MindIR is [hof.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_en/design/mindspore/images/ir/hof.dot). -![](./images/ir/hof.png) +![image](./images/ir/hof.png) In the actual network training scripts, the automatic derivation generic function `GradOperation` and `Partial` and `HyperMap` that are commonly used in the optimizer are typical high-order functions. Higher-order semantics greatly improve the flexibility and simplicity of MindSpore representations. ### Control Flows + In a MindIR, control flows are expressed in the form of high-order function selection and calling. This form transforms a control flow into a data flow of higher-order functions, making the automatic differential algorithm more powerful. It not only supports automatic differentiation of data flows, but also supports automatic differentiation of control flows such as conditional jumps, loops, and recursion. The following uses a simple Fibonacci instance as an example. + ```python @ms_function def fibonacci(n): @@ -128,15 +145,16 @@ def fibonacci(n): ``` The corresponding MindIR is [cf.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_en/design/mindspore/images/ir/cf.dot). -![](./images/ir/cf.png) +![image](./images/ir/cf.png) `fibonacci` is a top-level function graph. Two function graphs at the top level are selected and called by `switch`. `✓fibonacci` is the True branch of the first `if`, and `✗fibonacci` is the False branch of the first `if`. `✓✗fibonacci` called in `✗fibonacci` is the True branch of `elif`, and `✗✗fibonacci` is the False branch of `elif`. The key is, in a MindIR, conditional jumps and recursion are represented in the form of higher-order control flows. For example, `✓✗fibonacci` and `✗fibonacci` are transferred in as parameters of the `switch` operator. `switch` selects a function as the return value based on the condition parameter. In this way, `switch` performs a binary selection operation on the input functions as common values and does not call the functions. The real function call is completed on CNode following `switch`. - ### Free Variables and Closures + Closure is a programming language feature that refers to the combination of code blocks and scope environment. A free variable refers to a variable in the scope environment referenced in a code block instead of a local variable. In a MindIR, a code block is represented as a function graph. The scope environment can be considered as the context where the function is called. The capture method of free variables is value copy instead of reference. A typical closure instance is as follows: + ```python @ms_function def func_outer(a, b): @@ -153,14 +171,15 @@ def ms_closure(): ``` The corresponding MindIR is [closure.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_en/design/mindspore/images/ir/closure.dot). -![](./images/ir/closure.png) +![image](./images/ir/closure.png) In the example, `a` and `b` are free variables because the variables `a` and `b` in `func_inner` are parameters defined in the referenced parent graph `func_outer`. The variable `closure` is a closure, which is the combination of the function `func_inner` and its context `func_outer(1, 2)`. Therefore, the result of `out1` is 4, which is equivalent to `1+2+1`, and the result of `out2` is 5, which is equivalent to `1+2+2`. ## References + [1] C. Click and M. Paleczny. A simple graph-based intermediate representation. SIGPLAN Not., 30:35–49, March 1995. [2] Roland Leißa, Marcel Köster, and Sebastian Hack. A graph-based higher-order intermediate representation. In Proceedings of the 13th Annual IEEE/ACM International Symposium on -Code Generation and Optimization, pages 202–212. IEEE Computer Society, 2015. \ No newline at end of file +Code Generation and Optimization, pages 202–212. IEEE Computer Society, 2015. diff --git a/docs/note/source_en/design/mindspore/profiler_design.md b/docs/note/source_en/design/mindspore/profiler_design.md index 9acb50afb733aa344cc00275fe58f6f03568c8a1..e8dd67c9ffb8d6b20b5c1cf6920490e80fff62aa 100644 --- a/docs/note/source_en/design/mindspore/profiler_design.md +++ b/docs/note/source_en/design/mindspore/profiler_design.md @@ -33,6 +33,7 @@ To support model development and performance debugging in MindSpore, an easy-to-use profile tool is required to intuitively display the performance information of each dimension of a network model, provide users with easy-to-use and abundant profiling functions, and help users quickly locate network performance faults. ## Profiler Architecture Design + The Profiler architecture design is introduced from the following three aspects: the overall context interaction relationship of Profiler; the internal structure of Profiler, including the module structure and module layers; the interactive calling relationship between modules. ### Context @@ -50,6 +51,7 @@ As shown in the preceding figure, the interaction between the Profiler and other 2. MindSpore Profiler parses the original data in the user script and generates the intermediate data results in the specified folder. 3. MindInsight Profiler connects to the intermediate data and provides the visualized Profiler function for users. + ### Module Structure Modules are classified into the following layers: @@ -58,8 +60,8 @@ Modules are classified into the following layers: Figure 2 Relationships between modules at different layers - Module functions are as follows: + 1. ProfilerAPI is a calling entry provided by code, including the performance collection startup API and analysis API. 2. Controller is a module at a layer lower than that of ProfilerAPI. It is called by the startup API of ProfilerAPI to start or stop the performance collection function. The original data is written to a fixed position by ada. 3. Parser is a module for parsing original performance data which is collected on the device and cannot be directly understood by users. Parser parses, combines, and converts the data to generate intermediate results that can be understood by users and analyzed by upper layers. @@ -67,6 +69,7 @@ Module functions are as follows: 5. RESTful is used to call the common API provided by the backend Analyser to obtain objective data and use RESTful to connect to the frontend. ### Internal Module Interaction + Users can use API or RESTful to complete internal module interaction process. The following uses the API as an example: ![time_order_profiler.png](./images/time_order_profiler.png) @@ -82,20 +85,22 @@ The interaction process of each module is as follows: 3. Profiler API analysis API uses the Parser module to parse performance data, generates intermediate results, calls the Aalayser module to analyze the results, and returns various information to users. ## Sub-Module Design + ### ProfilerAPI and Controller #### Description + ProfilerAPI provides an entry API in the training script for users to start performance collection and analyze performance data. ProfilerAPI delivers commands through Controller to control the startup of ada. #### Design + ProfilerAPI belongs to the API layer of upper-layer application and is integrated by the training script. The function is divided into two parts: - Before training, call the bottom-layer Controller API to deliver a command to start a profiling task. - After training, call the bottom-layer Controller API to deliver commands to stop the profiling task, call the Analyser and Parser APIs to parse data files and generate result data such as operator performance statistics and training trace statistics. - Controller provides an API for the upper layer, calls API of the lower-layer performance collection module, and delivers commands for starting and stopping performance collection. The generated original performance data includes: @@ -106,9 +111,13 @@ The generated original performance data includes: - `training_trace.46.dev.profiler_default_tag` file: stores the start and end time of each step and time of step interval, forward and backward propagation, and step tail. ### Parser + #### Description + Parser is a module for parsing original performance data which is collected on the device and cannot be directly understood by users. Parser parses, combines, and converts the data to generate intermediate results that can be understood by users and analyzed by upper layers. + #### Design + ![parser_module_profiler.png](./images/parser_module_profiler.png) Figure 4 Parser module @@ -123,6 +132,7 @@ As shown in the preceding figure, there are HWTS Parser, AI CPU Parser, Framewor ### Analyser #### Description + Analyzer is used to filter, sort, query, and page the intermediate results generated at the parsing stage. #### Design @@ -142,9 +152,10 @@ Currently, there are two types of analyzers for operator information: To hide the internal implementation of Analyser and facilitate calling, the simple factory mode is used to obtain the specified Analyser through AnalyserFactory. - ### Proposer + #### Description + Proposer is a Profiler performance optimization suggestion module. Proposer calls the Analyser module to obtain performance data, analyzes the performance data based on optimization rules, and displays optimization suggestions for users through the UI and API. #### Design @@ -172,4 +183,4 @@ Figure 7 Proposer class As shown in the preceding figure: - Proposers of various types inherit the abstract class Proposer and implement the analyze methods. -- API and CLI call the ProposerFactory to obtain the Proposer and call the Proposer.analyze function to obtain the optimization suggestions of each type of Proposer. \ No newline at end of file +- API and CLI call the ProposerFactory to obtain the Proposer and call the Proposer.analyze function to obtain the optimization suggestions of each type of Proposer. diff --git a/docs/note/source_en/glossary.md b/docs/note/source_en/glossary.md index 852afab1c841523f869472090a9ee827e028c047..6ec11c85d39df9af4669d847255544882ac78be4 100644 --- a/docs/note/source_en/glossary.md +++ b/docs/note/source_en/glossary.md @@ -4,7 +4,7 @@ -| Acronym and Abbreviation | Description | +| Acronym and Abbreviation | Description | | ----- | ----- | | ACL | Ascend Computer Language, for users to develop deep neural network applications, which provides the C++ API library including device management, context management, stream management, memory management, model loading and execution, operator loading and execution, media data processing, etc. | | Ascend | Name of Huawei Ascend series chips. | diff --git a/docs/note/source_en/help_seeking_path.md b/docs/note/source_en/help_seeking_path.md index ecaa964ae6d416ebda96843a8688029a20c78278..9ac8c6bb6da04e502a89729e32b1e8644c82db51 100644 --- a/docs/note/source_en/help_seeking_path.md +++ b/docs/note/source_en/help_seeking_path.md @@ -10,22 +10,20 @@ This document describes how to seek help and support when you encounter problems - Website search - - Go to the [official search page](https://www.mindspore.cn/search/en). - - When encountering a problem, search on the official website first, which is simple and efficient. - - Enter a keyword in the search box and click the search icon. The related content is displayed. - - Resolve the problem based on the search result. - + - Go to the [official search page](https://www.mindspore.cn/search/en). + - When encountering a problem, search on the official website first, which is simple and efficient. + - Enter a keyword in the search box and click the search icon. The related content is displayed. + - Resolve the problem based on the search result. - User group consultation - - If you cannot solve the problem using the website search method and want a quick consultation. Get support by joining the [Slack group](https://mindspore.slack.com/join/shared_invite/zt-dgk65rli-3ex4xvS4wHX7UDmsQmfu8w#/ ) and start a conversation with our members. - - Resolve the problem by asking experts or communicating with other users. - + - If you cannot solve the problem using the website search method and want a quick consultation. Get support by joining the [Slack group](https://mindspore.slack.com/join/shared_invite/zt-dgk65rli-3ex4xvS4wHX7UDmsQmfu8w#/ ) and start a conversation with our members. + - Resolve the problem by asking experts or communicating with other users. - Forum Help-Seeking - - If you want a detailed solution, start a help post on the [Ascend forum](https://forum.huawei.com/enterprise/en/forum-100504.html). - - After the post is sent, a forum moderator collects the question and contacts technical experts to answer the question. The question will be resolved within three working days. - - Resolve the problem by referring to solutions provided by technical experts. + - If you want a detailed solution, start a help post on the [Ascend forum](https://forum.huawei.com/enterprise/en/forum-100504.html). + - After the post is sent, a forum moderator collects the question and contacts technical experts to answer the question. The question will be resolved within three working days. + - Resolve the problem by referring to solutions provided by technical experts. - If the expert test result shows that the MindSpore function needs to be improved, you are advised to submit an issue in the [MindSpore repository](https://gitee.com/mindspore). Issues will be resolved in later versions. \ No newline at end of file + If the expert test result shows that the MindSpore function needs to be improved, you are advised to submit an issue in the [MindSpore repository](https://gitee.com/mindspore). Issues will be resolved in later versions. diff --git a/docs/note/source_en/network_list_ms.md b/docs/note/source_en/network_list_ms.md index 8cbf3ba9802d880346645ff113fe73410799aa16..169d2137ccd1f235a41572f3d4f21126de7d871e 100644 --- a/docs/note/source_en/network_list_ms.md +++ b/docs/note/source_en/network_list_ms.md @@ -13,7 +13,7 @@ ## Model Zoo -| Domain | Sub Domain | Network | Ascend(Graph) | Ascend(PyNative) | GPU(Graph) | GPU(PyNative)| CPU(Graph) +| Domain | Sub Domain | Network | Ascend(Graph) | Ascend(PyNative) | GPU(Graph) | GPU(PyNative)| CPU(Graph) |:------ |:------| :----------- |:------ |:------ |:------ |:------ |:----- |Computer Vision (CV) | Image Classification | [AlexNet](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/alexnet/src/alexnet.py) | Supported | Supported | Supported | Supported | Doing | Computer Vision (CV) | Image Classification | [GoogleNet](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/googlenet/src/googlenet.py) | Supported | Supported | Supported | Supported | Doing diff --git a/docs/note/source_en/object_detection_lite.md b/docs/note/source_en/object_detection_lite.md index 6e02865426d6957fc991d5c4968f05ccf5af1e53..10089f2a80189ff17cbe49230f7abd842db731a0 100644 --- a/docs/note/source_en/object_detection_lite.md +++ b/docs/note/source_en/object_detection_lite.md @@ -23,4 +23,3 @@ The following table shows the data of some object detection models using MindSpo | Model name | Size | mAP(IoU=0.50:0.95) | CPU 4 thread delay (ms) | |-----------------------| :----------: | :----------: | :-----------: | | [MobileNetv2-SSD](https://download.mindspore.cn/model_zoo/official/lite/ssd_mobilenetv2_lite/ssd.ms) | 16.7 | 0.22 | 25.4 | - diff --git a/docs/note/source_en/operator_list_implicit.md b/docs/note/source_en/operator_list_implicit.md index 955ea0a2feaaca4c0112f13c516de2ad0bd16e26..5eb631df80993c22b4e4aad5d8ad11d0847c6ffb 100644 --- a/docs/note/source_en/operator_list_implicit.md +++ b/docs/note/source_en/operator_list_implicit.md @@ -17,24 +17,26 @@ ## Implicit Type Conversion ### conversion rules -* Scalar and Tensor operations: during operation, the scalar is automatically converted to Tensor, and the data type is consistent with the Tensor data type involved in the operation; + +- Scalar and Tensor operations: during operation, the scalar is automatically converted to Tensor, and the data type is consistent with the Tensor data type involved in the operation; when Tensor is bool data type and the scalar is int or float, both the scalar and Tensor are converted to the Tensor with the data type of int32 or float32; when Tensor is int or uint data type and the scalar is float, both the scalar and Tensor are converted to the Tensor with the data type of float32. -* Tensor operation of different data types: the priority of data type is bool < uint8 < int8 < int16 < int32 < int64 < float16 < float32 - + 本文介绍MindSpore的基准性能。MindSpore网络定义可参考[Model Zoo](https://gitee.com/mindspore/mindspore/tree/master/model_zoo)。 @@ -32,7 +32,7 @@ ### BERT -| Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | +| Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | BERT-Large | Attention | zhwiki | 0.5.0-beta | Ascend: 1 * Ascend 910
CPU：24 Cores | Mixed | 96 | 269 sentences/sec | - | | | | | | Ascend: 8 * Ascend 910
CPU：192 Cores | Mixed | 96 | 2069 sentences/sec | 0.96 | @@ -45,7 +45,7 @@ | Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Wide & Deep | Recommend | Criteo | 0.6.0-beta | Ascend: 1 * Ascend 910
CPU：24 Cores | Mixed | 16000 | 796892 samples/sec | - | -| | | | | Ascend: 8 * Ascend 910
CPU：192 Cores | Mixed | 16000*8 | 4872849 samples/sec | 0.76 | +| | | | | Ascend: 8 \* Ascend 910
CPU：192 Cores | Mixed | 16000*8 | 4872849 samples/sec | 0.76 | 1. 以上数据基于Atlas 800测试获得，且网络模型为数据并行。 2. 业界其他开源框架数据可参考：[Wide & Deep For TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Recommendation/WideAndDeep)。 @@ -55,9 +55,9 @@ | Network | Network Type | Dataset | MindSpore Version | Resource                 | Precision | Batch Size | Throughput | Speedup | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Wide & Deep | Recommend | Criteo | 0.6.0-beta | Ascend: 1 * Ascend 910
CPU：24 Cores | Mixed | 8000 | 68715 samples/sec | - | -| | | | | Ascend: 8 * Ascend 910
CPU：192 Cores | Mixed | 8000*8 | 283830 samples/sec | 0.51 | -| | | | | Ascend: 16 * Ascend 910
CPU：384 Cores | Mixed | 8000*16 | 377848 samples/sec | 0.34 | -| | | | | Ascend: 32 * Ascend 910
CPU：768 Cores | Mixed | 8000*32 | 433423 samples/sec | 0.20 | +| | | | | Ascend: 8 \* Ascend 910
CPU：192 Cores | Mixed | 8000*8 | 283830 samples/sec | 0.51 | +| | | | | Ascend: 16 \* Ascend 910
CPU：384 Cores | Mixed | 8000*16 | 377848 samples/sec | 0.34 | +| | | | | Ascend: 32 \* Ascend 910
CPU：768 Cores | Mixed | 8000*32 | 433423 samples/sec | 0.20 | 1. 以上数据基于Atlas 800测试获得，且网络模型为模型并行。 2. 业界其他开源框架数据可参考：[Wide & Deep For TensorFlow](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Recommendation/WideAndDeep)。 diff --git a/docs/note/source_zh_cn/constraints_on_network_construction.md b/docs/note/source_zh_cn/constraints_on_network_construction.md index 7fa4b65d340f09c7b88b6c796dcb5b9112474ebf..0678b7d4e7ca52c6a9f387525950fb7a38a3de18 100644 --- a/docs/note/source_zh_cn/constraints_on_network_construction.md +++ b/docs/note/source_zh_cn/constraints_on_network_construction.md @@ -28,21 +28,26 @@ ## 概述 + MindSpore完成从用户源码到计算图的编译，用户源码基于Python语法编写，当前MindSpore支持将普通函数或者继承自nn.Cell的实例转换生成计算图，暂不支持将任意Python源码转换成计算图，所以对于用户源码支持的写法有所限制，主要包括语法约束和网络定义约束两方面。随着MindSpore的演进，这些约束可能会发生变化。 ## 语法约束 + ### 支持的Python数据类型 -* Number：包括`int`、`float`、`bool`，不支持复数类型。 -* String -* List：当前只支持append方法；List的更新会拷贝生成新的List。 -* Tuple -* Dictionary：当前`key`只支持String类型 + +- Number：包括`int`、`float`、`bool`，不支持复数类型。 +- String +- List：当前只支持append方法；List的更新会拷贝生成新的List。 +- Tuple +- Dictionary：当前`key`只支持String类型 + ### MindSpore扩展数据类型 -* Tensor：Tensor变量必须是已定义实例。 + +- Tensor：Tensor变量必须是已定义实例。 ### 表达式类型 -| 操作名 | 具体操作 +| 操作名 | 具体操作 | :----------- |:-------- | 一元操作符 |`+`、`-`、`not`，其中`+`操作符只支持标量。 | 数学表达式 |`+`、`-`、`*`、`/`、`%`、`**`、`//` @@ -81,10 +86,11 @@ | `isinstance` | 使用原则与Python一致，但第二个入参只能是mindspore定义的类型。 ### 函数参数 -* 参数默认值：目前不支持默认值设为`Tensor`类型数据，支持`int`、`float`、`bool`、`None`、`str`、`tuple`、`list`、`dict`类型数据。 -* 可变参数：支持带可变参数网络的推理和训练。 -* 键值对参数：目前不支持带键值对参数的函数求反向。 -* 可变键值对参数：目前不支持带可变键值对的函数求反向。 + +- 参数默认值：目前不支持默认值设为`Tensor`类型数据，支持`int`、`float`、`bool`、`None`、`str`、`tuple`、`list`、`dict`类型数据。 +- 可变参数：支持带可变参数网络的推理和训练。 +- 键值对参数：目前不支持带键值对参数的函数求反向。 +- 可变键值对参数：目前不支持带可变键值对的函数求反向。 ### 操作符 @@ -104,51 +110,52 @@ 索引操作包含`tuple`和`Tensor`的索引操作。下面重点介绍一下`Tensor`的索引取值和赋值操作，取值以`tensor_x[index]`为例，赋值以`tensor_x[index] = u`为例进行详细说明。其中tensor_x是一个`Tensor`，对其进行切片操作；index表示索引，u表示赋予的值，可以是`scalar`或者`Tensor(size=1)`。索引类型如下： - 切片索引：index为`slice` - - 取值：`tensor_x[start:stop:step]`，其中Slice(start:stop:step)与Python的语法相同，这里不再赘述。 - - 赋值：`tensor_x[start:stop:step]=u`。 + - 取值：`tensor_x[start:stop:step]`，其中Slice(start:stop:step)与Python的语法相同，这里不再赘述。 + - 赋值：`tensor_x[start:stop:step]=u`。 - Ellipsis索引：index为`ellipsis` - - 取值：`tensor_x[...]`。 - - 赋值：`tensor_x[...]=u`。 + - 取值：`tensor_x[...]`。 + - 赋值：`tensor_x[...]=u`。 - 布尔常量索引：index为`True`，index为`False`暂不支持。 - - 取值：`tensor_x[True]`。 - - 赋值：暂不支持。 + - 取值：`tensor_x[True]`。 + - 赋值：暂不支持。 - Tensor索引：index为`Tensor` - - 取值：`tensor_x[index]`，`index`必须是`int32`、`int64`类型的`Tensor`，元素取值范围在`[0, tensor_x.shape[0])`。 - - 赋值：`tensor_x[index]=U`。 - - `tensor_x`的数据类型必须是下面一种： `float16`，`float32`，`int8`，`uint8`。 - - `index`必须是`int32`类型的`Tensor`，元素取值范围在`[0, tensor_x.shape[0])`。 - - `U`可以是`Number`，`Tensor`，只包含`Number`的`Tuple`，只包含`Tensor`的`Tuple`。 - - 单个`Number`和`Tuple`里的每个`Number`必须与`tensor_x`的数据类型属于同一类，即 + - 取值：`tensor_x[index]`，`index`必须是`int32`、`int64`类型的`Tensor`，元素取值范围在`[0, tensor_x.shape[0])`。 + - 赋值：`tensor_x[index]=U`。 + - `tensor_x`的数据类型必须是下面一种： `float16`，`float32`，`int8`，`uint8`。 + - `index`必须是`int32`类型的`Tensor`，元素取值范围在`[0, tensor_x.shape[0])`。 + - `U`可以是`Number`，`Tensor`，只包含`Number`的`Tuple`，只包含`Tensor`的`Tuple`。 + - 单个`Number`和`Tuple`里的每个`Number`必须与`tensor_x`的数据类型属于同一类，即 当`tensor_x`的数据类型是`uint8`或者`int8`时，`Number`类型应该是`int`； 当`tensor_x`的数据类型是`float16`或者`float32`时，`Number`类型应该是`float`。 - - 单个`Tensor`和`Tuple`里的每个`Tensor`必须与`tensor_x`的数据类型一致， + - 单个`Tensor`和`Tuple`里的每个`Tensor`必须与`tensor_x`的数据类型一致， 单个`Tensor`时，其`shape`需等于或者可广播为`index.shape + tensor_x.shape[1:]`。 - - 包含`Number`的`Tuple`需满足下面条件： + - 包含`Number`的`Tuple`需满足下面条件： `len(Tuple) = (index.shape + tensor_x.shape[1:])[-1]`。 - - 包含`Tensor`的`Tuple`需满足下面条件： + - 包含`Tensor`的`Tuple`需满足下面条件： 每个`Tensor`的`shape`一样； `(len(Tuple),) + Tensor.shape`等于或者可广播为`index.shape + tensor_x.shape[1:]`。 - None常量索引：index为`None` - - 取值：`tensor_x[None]`，结果与numpy保持一致。 - - 赋值：暂不支持。 + - 取值：`tensor_x[None]`，结果与numpy保持一致。 + - 赋值：暂不支持。 - tuple索引：index为`tuple` - - tuple元素为slice: - - 取值：例如`tensor_x[::, :4, 3:0:-1]`。 - - 赋值：例如`tensor_x[::, :4, 3:0:-1]=u`。 - - tuple元素为Number： - - 取值：例如`tensor_x[2,1]`。 - - 赋值：例如`tensor_x[1,4]=u`。 - - tuple元素为slice和ellipsis混合情况： - - 取值：例如`tensor_x[..., ::, 1:]` - - 赋值：例如`tensor_x[..., ::, 1:]=u` - - 其他情况暂不支持 + - tuple元素为slice: + - 取值：例如`tensor_x[::, :4, 3:0:-1]`。 + - 赋值：例如`tensor_x[::, :4, 3:0:-1]=u`。 + - tuple元素为Number： + - 取值：例如`tensor_x[2,1]`。 + - 赋值：例如`tensor_x[1,4]=u`。 + - tuple元素为slice和ellipsis混合情况： + - 取值：例如`tensor_x[..., ::, 1:]` + - 赋值：例如`tensor_x[..., ::, 1:]=u` + - 其他情况暂不支持 tuple和list类型的索引取值操作，需要重点介绍一下元素类型为`nn.Cell`的tuple或list的索引取值操作，该操作目前在Graph模式下仅GPU后端支持运行，其语法格式形如`layers[index](*inputs)`，具体示例代码如下： + ```python class Net(nn.Cell): def __init__(self): @@ -161,60 +168,68 @@ tuple和list类型的索引取值操作，需要重点介绍一下元素类型 x = self.layers[index](x) return x ``` + 同时该语法有以下几个约束： -* 只支持元素类型为`nn.Cell`的tuple或list的索引取值操作。 -* 索引值index的类型为`int32`的Tensor标量，取值范围为`[-n, n)`, 其中`n`为tuple的size，支持的tuple的size的最大值为1000。 -* tuple中的每个Cell元素的Construct函数的输入数据的数目，类型和shape要求相同，且Construct函数运行后输出的数据的数目，类型和shape也要求相同。 -* tuple中的每个Cell元素，需要在tuple定义之前完成定义。 -* 该语法不支持做为if、while、for等控制流的运行分支，如果控制流的控制条件为常量除外。举例说明： - - 支持的写法： - ```python - class Net(nn.Cell): - def __init__(self, flag=True): - super(Net, self).__init__() - self.flag = flag - self.relu = nn.ReLU() - self.softmax = nn.Softmax() - self.layers = (self.relu, self.softmax) - def construct(self, x, index): - if self.flag: - x = self.layers[index](x) - return x - ``` - - 不支持的写法： - ```python - class Net(nn.Cell): - def __init__(self): - super(Net, self).__init__() - self.relu = nn.ReLU() - self.softmax = nn.Softmax() - self.layers = (self.relu, self.softmax) +- 只支持元素类型为`nn.Cell`的tuple或list的索引取值操作。 +- 索引值index的类型为`int32`的Tensor标量，取值范围为`[-n, n)`, 其中`n`为tuple的size，支持的tuple的size的最大值为1000。 +- tuple中的每个Cell元素的Construct函数的输入数据的数目，类型和shape维度要求相同，且Construct函数运行后输出的数据的数目，类型和shape维度也要求相同。 +- tuple中的每个Cell元素，需要在tuple定义之前完成定义。 +- 该语法不支持做为if、while、for等控制流的运行分支，如果控制流的控制条件为常量除外。举例说明： + - 支持的写法： - def construct(self, x, index, flag): - if flag: - x = self.layers[index](x) - return x - ``` + ```python + class Net(nn.Cell): + def __init__(self, flag=True): + super(Net, self).__init__() + self.flag = flag + self.relu = nn.ReLU() + self.softmax = nn.Softmax() + self.layers = (self.relu, self.softmax) + + def construct(self, x, index): + if self.flag: + x = self.layers[index](x) + return x + ``` + + - 不支持的写法： + + ```python + class Net(nn.Cell): + def __init__(self): + super(Net, self).__init__() + self.relu = nn.ReLU() + self.softmax = nn.Softmax() + self.layers = (self.relu, self.softmax) + + def construct(self, x, index, flag): + if flag: + x = self.layers[index](x) + return x + ``` tuple也支持切片取值操作, 但不支持切片类型为Tensor类型，支持`tuple_x[start:stop:step]`，其中操作对象为与Python的效果相同，这里不再赘述。 ### 不支持的语法 -目前在网络构造函数里面暂不支持以下语法： +目前在网络构造函数里面暂不支持以下语法： `raise`、 `yield`、 `async for`、 `with`、 `async with`、 `assert`、 `import`、 `await`。 ## 网络定义约束 ### 整网实例类型 -* 带[@ms_function](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.html#mindspore.ms_function)装饰器的普通Python函数。 -* 继承自[nn.Cell](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.Cell)的Cell子类。 + +- 带[@ms_function](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.html#mindspore.ms_function)装饰器的普通Python函数。 +- 继承自[nn.Cell](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.Cell)的Cell子类。 ### 网络输入类型 -* 整网的训练数据输入参数只能是Tensor类型。 -* 生成的ANF图里面不能包含这几种常量节点：字符串类型常量、带有Tuple嵌套的常量、带有List嵌套的常量。 + +- 整网的训练数据输入参数只能是Tensor类型。 +- 生成的ANF图里面不能包含这几种常量节点：字符串类型常量、带有Tuple嵌套的常量、带有List嵌套的常量。 ### 网络图优化 + 在ME前端图优化过程中，会将DataClass类型、Dictionary、List、键值对操作转换为Tuple相关操作。 ### 网络构造组件 @@ -229,33 +244,36 @@ tuple也支持切片取值操作, 但不支持切片类型为Tensor类型，支 | Composite算子 |[mindspore/ops/composite/*](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html) | constexpr生成算子 |使用[@constexpr](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.constexpr)生成的值计算算子。 - ### 其他约束 + 1. 整网`construct`函数输入的参数以及使用`ms_function`装饰器修饰的函数的参数在图编译过程中会进行泛化，不能作为常量输入传给算子使用。所以，在图模式下，限制入口网络的参数只能是`Tensor`，如下例所示： - - * 错误的写法如下： + + - 错误的写法如下： + ```python class ExpandDimsTest(Cell): def __init__(self): super(ExpandDimsTest, self).__init__() self.expandDims = P.ExpandDims() - + def construct(self, input_x, input_axis): return self.expandDims(input_x, input_axis) expand_dim = ExpandDimsTest() input_x = Tensor(np.random.randn(2,2,2,2).astype(np.float32)) expand_dim(input_x, 0) ``` + 在示例中，`ExpandDimsTest`是一个只有单算子的网络，网络的输入有`input_x`和`input_axis`两个。因为`ExpandDims`算子的第二个输入需要是常量，这是因为在图编译过程中推导`ExpandDims`算子输出维度的时候需要用到，而`input_axis`作为网络参数输入会泛化成变量，无法确定其值，从而无法推导算子的输出维度导致图编译失败。所以在图编译阶段需要值推导的输入都应该是常量输入。在API中，这类算子需要常量输入的参数会进行说明，标注"constant input is needed"。 - - * 正确的写法是在construct函数里面对算子的常量输入直接填入需要的值或者是一个类的成员变量，如下： + + - 正确的写法是在construct函数里面对算子的常量输入直接填入需要的值或者是一个类的成员变量，如下： + ```python class ExpandDimsTest(Cell): def __init__(self, axis): super(ExpandDimsTest, self).__init__() self.expandDims = P.ExpandDims() self.axis = axis - + def construct(self, input_x): return self.expandDims(input_x, self.axis) axis = 0 @@ -266,48 +284,53 @@ tuple也支持切片取值操作, 但不支持切片类型为Tensor类型，支 2. 不允许修改网络的非`Parameter`类型数据成员。示例如下： - ``` + ```python class Net(Cell): def __init__(self): super(Net, self).__init__() self.num = 2 self.par = Parameter(Tensor(np.ones((2, 3, 4))), name="par") - + def construct(self, x, y): return x + y ``` + 上面所定义的网络里，`self.num`不是一个`Parameter`，不允许被修改，而`self.par`是一个`Parameter`，可以被修改。 3. 当`construct`函数里，使用未定义的类成员时，不会像Python解释器那样抛出`AttributeError`，而是作为`None`处理。示例如下： - ``` + + ```python class Net(Cell): def __init__(self): super(Net, self).__init__() - + def construct(self, x): return x + self.y ``` + 上面所定义的网络里，`construct`里使用了并未定义的类成员`self.y`，此时会将`self.y`作为`None`处理。 - + 4. 当`construct`函数里，使用`if-else`控制流时，`if`和`else`返回的数据类型或者同一变量被更新后的数据类型必须一致，示例如下： - ``` + + ```python class NetReturn(Cell): def __init__(self): super(NetReturn, self).__init__() - + def construct(self, x, y, m, n): if x > y: return m else: return n ``` + 上面所定义的网络`NetReturn`里，`construct`里使用了`if-else`控制流，那么`if`分支返回的`m`和`else`分支返回的`n`数据类型必须一致。 - - ``` + + ```python class NetAssign(Cell): def __init__(self): super(NetAssign, self).__init__() - + def construct(self, x, y, m, n): out = None if x > y: @@ -316,4 +339,5 @@ tuple也支持切片取值操作, 但不支持切片类型为Tensor类型，支 out = n return out ``` + 上面所定义的网络`NetAssign`里，`construct`里使用了`if-else`控制流，那么`if`分支更新后的`out`和`else`分支更新后的`out`的数据类型必须一致。 diff --git a/docs/note/source_zh_cn/design/mindarmour/differential_privacy_design.md b/docs/note/source_zh_cn/design/mindarmour/differential_privacy_design.md index 608841c576e8b106b3a36e775dfaafe88ee91492..256d719bcf7dc1e789d9895d3841598691da4672 100644 --- a/docs/note/source_zh_cn/design/mindarmour/differential_privacy_design.md +++ b/docs/note/source_zh_cn/design/mindarmour/differential_privacy_design.md @@ -26,14 +26,13 @@ MindArmour的Differential-Privacy模块实现了差分隐私训练的能力。 图1是差分隐私训练的总体设计，主要由差分隐私噪声机制(DP Mechanisms)、差分隐私优化器(DP Optimizer)、差分隐私监控器(Privacy Monitor)组成。 - ### 差分隐私优化器 差分隐私优化器继承了MindSpore优化器的能力，并使用差分隐私的噪声机制对梯度加扰保护。目前，MindArmour提供三类差分隐私优化器：固定高斯优化器、自适应高斯优化器、自适应裁剪优化器，每类差分隐私优化器从不同的角度为SGD、Momentum等常规优化器增加差分隐私保护的能力。 -* 固定高斯优化器，是一种非自适应高斯噪声的差分隐私优化器。其优势在于可以严格控制差分隐私预算ϵ，缺点是在模型训练过程中，每个Step添加的噪声量固定，若迭代次数过大，训练后期的噪声使得模型收敛困难，甚至导致性能大幅下跌，模型可用性差。 -* 自适应高斯优化器，通过自适应调整标准差，来调整高斯分布噪声的大小，在模型训练初期，添加的噪声量较大，随着模型逐渐收敛，噪声量逐渐减小，噪声对于模型可用性的影响减小。自适应高斯噪声的缺点是不能严格控制差分隐私预算。 -* 自适应裁剪优化器，是一种自适应调整调整裁剪粒度的差分隐私优化器，梯度裁剪是差分隐私训练的一个重要操作，自适应裁剪优化器能够自适应的控制梯度裁剪的的比例在给定的范围波动，控制迭代训练过程中梯度裁剪的粒度。 +- 固定高斯优化器，是一种非自适应高斯噪声的差分隐私优化器。其优势在于可以严格控制差分隐私预算ϵ，缺点是在模型训练过程中，每个Step添加的噪声量固定，若迭代次数过大，训练后期的噪声使得模型收敛困难，甚至导致性能大幅下跌，模型可用性差。 +- 自适应高斯优化器，通过自适应调整标准差，来调整高斯分布噪声的大小，在模型训练初期，添加的噪声量较大，随着模型逐渐收敛，噪声量逐渐减小，噪声对于模型可用性的影响减小。自适应高斯噪声的缺点是不能严格控制差分隐私预算。 +- 自适应裁剪优化器，是一种自适应调整调整裁剪粒度的差分隐私优化器，梯度裁剪是差分隐私训练的一个重要操作，自适应裁剪优化器能够自适应的控制梯度裁剪的的比例在给定的范围波动，控制迭代训练过程中梯度裁剪的粒度。 ### 差分隐私的噪声机制 @@ -41,25 +40,24 @@ MindArmour的Differential-Privacy模块实现了差分隐私训练的能力。 ### Monitor -Monitor提供RDP、ZCDP等回调函数，用于监测模型的差分隐私预算。 +Monitor提供RDP、ZCDP等回调函数，用于监测模型的差分隐私预算。 -* ZCDP[2] +- ZCDP[2] ZCDP，zero-concentrated differential privacy，是一种宽松的差分隐私定义，利用Rényi散度来度量随机函数在相邻数据集上的分布差异。 -* RDP[3] +- RDP[3] RDP，Rényi Differential Privacy，是一种更通用的基于R'enyi散度的差分隐私定义，利用Rényi散度来度量两个相邻数据集的分布差异。 - -相对于传统差分隐私，ZCDP和RDP都能能够提供更加严格的隐私预算上界保证。 +相对于传统差分隐私，ZCDP和RDP都能能够提供更加严格的隐私预算上界保证。 ## 代码实现 -* [mechanisms.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/mechanisms/mechanisms.py)：这个文件实现了差分隐私训练所需的噪声生成机制，包括简单高斯噪声、自适应高斯噪声、自适应裁剪高斯噪声等。 -* [optimizer.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/optimizer/optimizer.py)：这个文件实现了使用噪声生成机制在反向传播时添加噪声的根本逻辑。 -* [monitor.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/monitor/monitor.py)：实现了计算差分隐私预算的回调函数，模型训练过程中，会反馈当前的差分隐私预算。 -* [model.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/train/model.py)：这个文件实现了计算损失和梯度的逻辑，差分隐私训练的梯度截断逻辑在此文件中实现，且model.py是用户使用差分隐私训练能力的入口。 +- [mechanisms.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/mechanisms/mechanisms.py)：这个文件实现了差分隐私训练所需的噪声生成机制，包括简单高斯噪声、自适应高斯噪声、自适应裁剪高斯噪声等。 +- [optimizer.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/optimizer/optimizer.py)：这个文件实现了使用噪声生成机制在反向传播时添加噪声的根本逻辑。 +- [monitor.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/monitor/monitor.py)：实现了计算差分隐私预算的回调函数，模型训练过程中，会反馈当前的差分隐私预算。 +- [model.py](https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/privacy/diff_privacy/train/model.py)：这个文件实现了计算损失和梯度的逻辑，差分隐私训练的梯度截断逻辑在此文件中实现，且model.py是用户使用差分隐私训练能力的入口。 ## 参考文献 diff --git a/docs/note/source_zh_cn/design/mindarmour/fuzzer_design.md b/docs/note/source_zh_cn/design/mindarmour/fuzzer_design.md index 8f7fcbf81c368446fe19e06baa71933cc9559a7e..0d753d7bb92744cb7bc2ae96a665ac09fafec77b 100644 --- a/docs/note/source_zh_cn/design/mindarmour/fuzzer_design.md +++ b/docs/note/source_zh_cn/design/mindarmour/fuzzer_design.md @@ -2,7 +2,6 @@ `Linux` `Ascend` `GPU` `CPU` `数据准备` `模型开发` `模型训练` `模型调优` `企业` `高级` - - [AI模型安全测试](#ai模型安全测试) - [背景](#背景) diff --git a/docs/note/source_zh_cn/design/mindinsight/graph_visual_design.md b/docs/note/source_zh_cn/design/mindinsight/graph_visual_design.md index 600da5f29c5015556c5bd03a521d589107db7783..be8a8c686bb95ba565506ded1940d8b87281ca51 100644 --- a/docs/note/source_zh_cn/design/mindinsight/graph_visual_design.md +++ b/docs/note/source_zh_cn/design/mindinsight/graph_visual_design.md @@ -21,9 +21,9 @@ 计算图可视的功能，主要协助开发者在下面这些场景中使用。 - - 开发者在编写深度学习神经网络的代码时，可以使用计算图的功能查看神经网络中算子的数据流走向，以及模型结构。 - - 计算图还可以方便开发者查看指定节点的输入和输出节点，以及所查找的节点的属性信息。 - - 开发者在调试网络时，可以通过可视化的计算图，轻易跟踪数据，包括数据维度、类型的变更等。 +- 开发者在编写深度学习神经网络的代码时，可以使用计算图的功能查看神经网络中算子的数据流走向，以及模型结构。 +- 计算图还可以方便开发者查看指定节点的输入和输出节点，以及所查找的节点的属性信息。 +- 开发者在调试网络时，可以通过可视化的计算图，轻易跟踪数据，包括数据维度、类型的变更等。 ## 总体设计 diff --git a/docs/note/source_zh_cn/design/mindinsight/tensor_visual_design.md b/docs/note/source_zh_cn/design/mindinsight/tensor_visual_design.md index b8439752e6278735331057a4ca3ec254525ad732..44d4db5b12ddc5dc04e3ed2cedfb16dd69bb382d 100644 --- a/docs/note/source_zh_cn/design/mindinsight/tensor_visual_design.md +++ b/docs/note/source_zh_cn/design/mindinsight/tensor_visual_design.md @@ -60,7 +60,8 @@ Tensor可视支持1-N维的Tensor以表格或直方图的形式展示，对于0 #### 文件接口设计 `summary.proto`文件为总入口，其中张量的数据（TensorProto）存放在Summary的Value中，如下所示： -``` + +```protobuf { message Summary { message Image { @@ -69,7 +70,7 @@ Tensor可视支持1-N维的Tensor以表格或直方图的形式展示，对于0 required int32 width = 2; ... } - + message Histogram { message bucket{ // Counting number of values fallen in [left, left + width). @@ -78,7 +79,7 @@ Tensor可视支持1-N维的Tensor以表格或直方图的形式展示，对于0 required double width = 2; required int64 count = 3; } - + repeated bucket buckets = 1; ... } @@ -86,7 +87,7 @@ Tensor可视支持1-N维的Tensor以表格或直方图的形式展示，对于0 message Value { // Tag name for the data. required string tag = 1; - + // Value associated with the tag. oneof value { float scalar_value = 3; @@ -100,4 +101,5 @@ Tensor可视支持1-N维的Tensor以表格或直方图的形式展示，对于0 repeated Value value = 1; } ``` -而TensorProto的定义在[anf_ir.proto](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/utils/anf_ir.proto)文件中。 \ No newline at end of file + +而TensorProto的定义在[anf_ir.proto](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/utils/anf_ir.proto)文件中。 diff --git a/docs/note/source_zh_cn/design/mindspore/architecture_lite.md b/docs/note/source_zh_cn/design/mindspore/architecture_lite.md index eecfd0610dad6c0809e69bc11e781727331842df..23738f4426bf18f7ac54f1907ead31a8042ed04c 100644 --- a/docs/note/source_zh_cn/design/mindspore/architecture_lite.md +++ b/docs/note/source_zh_cn/design/mindspore/architecture_lite.md @@ -2,7 +2,6 @@ `Linux` `Windows` `端侧` `推理应用` `中级` `高级` `贡献者` - MindSpore Lite框架的总体架构如下所示： @@ -15,9 +14,8 @@ MindSpore Lite框架的总体架构如下所示： - **Backend：** 基于IR进行图优化，包括GHLO、GLLO和量化三部分。其中，GHLO负责和硬件无关的优化，如算子融合、常量折叠等；GLLO负责与硬件相关的优化；量化Quantizer支持权重量化、激活值量化等训练后量化手段。 -- **Runtime：** 智能终端的推理运行时，其中session负责会话管理，提供对外接口；线程池和并行原语负责图执行使用的线程池管理，内存分配负责图执行中各个算子的内存复用，算子库提供CPU和GPU算子。 +- **Runtime：** 智能终端的推理运行时，其中session负责会话管理，提供对外接口；线程池和并行原语负责图执行使用的线程池管理，内存分配负责图执行中各个算子的内存复用，算子库提供CPU和GPU算子。 - **Micro：** IoT设备的运行时，包括模型生成.c文件、线程池、内存复用和算子库。 -其中，Runtime和Micro共享底层的算子库、内存分配、线程池、并行原语等基础设施层。 - +其中，Runtime和Micro共享底层的算子库、内存分配、线程池、并行原语等基础设施层。 diff --git a/docs/note/source_zh_cn/design/mindspore/distributed_training_design.md b/docs/note/source_zh_cn/design/mindspore/distributed_training_design.md index 5b67aa47b3f30bc79b0e97736a5bfbf5eeec75cb..97a9a328b99dba77ff968775ef848096a0c995fc 100644 --- a/docs/note/source_zh_cn/design/mindspore/distributed_training_design.md +++ b/docs/note/source_zh_cn/design/mindspore/distributed_training_design.md @@ -24,7 +24,6 @@ 随着深度学习的快步发展，为了提升神经网络的精度和泛化能力，数据集和参数量都在呈指数级向上攀升。分布式并行训练成为一种解决超大规模网络性能瓶颈的发展趋势。MindSpore支持了当前主流的分布式训练范式并开发了一套自动混合并行解决方案。本篇设计文档将会集中介绍几种并行训练方式的设计原理，同时指导用户进行自定义开发。 - ## 概念 ### 集合通信 @@ -74,7 +73,6 @@ - [grad_reducer.py](https://gitee.com/mindspore/mindspore/blob/master/mindspore/nn/wrap/grad_reducer.py)：这个文件实现了梯度聚合的过程。对入参`grads`用`HyperMap`展开后插入`AllReduce`算子，这里采用的是全局通信组，用户也可以根据自己网络的需求仿照这个模块进行自定义开发。MindSpore中单机和分布式执行共用一套网络封装接口，在`Cell`内部通过`ParallelMode`来区分是否要对梯度做聚合操作，网络封装接口建议参考`TrainOneStepCell`代码实现。 - ## 自动并行 自动并行作为MindSpore的关键特性，用于实现自动的数据并行加模型并行的混合并行训练方式，旨在帮助用户以单机的脚本表达并行算法逻辑，降低分布式训练难度，提高算法研发效率，同时又能保持训练的高性能。这个小节介绍了在MindSpore中`ParallelMode.AUTO_PARALLEL`自动并行模式及`ParallelMode.SEMI_AUTO_PARALLEL`半自动并行模式是如何工作的。 @@ -90,34 +88,31 @@ 为了得到张量的排布模型，每个算子都具有切分策略（Shard Strategy），它表示算子的各个输入在相应维度的切分情况。通常情况下只要满足以2为基、均匀分配的原则，张量的任意维度均可切分。以下图为例，这是一个三维矩阵乘（BatchMatMul）操作，它的切分策略由两个元组构成，分别表示`input`和`weight`的切分形式。其中元组中的元素与张量维度一一对应，`2^N`为切分份数，`1`表示不切。当我们想表示一个数据并行切分策略时，即`input`的`batch`维度切分，其他维度不切，可以表达为`strategy=((2^N, 1, 1),(1, 1, 1))`；当表示一个模型并行切分策略时，即`weight`的非`batch`维度切分，这里以`channel`维度切分为例，其他维度不切，可以表达为`strategy=((1, 1, 1),(1, 1, 2^N))`；当表示一个混合并行切分策略时，其中一种切分策略为`strategy=((2^N, 1, 1),(1, 1, 2^N))`。 ![算子切分定义](./images/operator_split.png) - + 依据切分策略，分布式算子中定义了推导算子输入张量和输出张量的排布模型的方法。这个排布模型由`device_matrix`，`tensor_shape`和`tensor map`组成，分别表示设备矩阵形状、张量形状、设备和张量维度间的映射关系。分布式算子会进一步根据张量排布模型判断是否要在图中中插入额外的计算、通信操作，以保证算子运算逻辑正确。 2. 张量排布变换 当前一个算子的输出张量模型和后一个算子的输入张量模型不一致时，就需要引入计算、通信操作的方式实现张量排布间的变化。自动并行流程引入了张量重排布算法（Tensor Redistribution），可以推导得到任意排布的张量间通信转换方式。下面三个样例表示公式`Z=(X×W)×V`的并行计算过程， 即两个二维矩阵乘操作，体现了不同并行方式间如何转换。 在样例一中，第一个数据并行矩阵乘的输出在行方向上存在切分，而第二个模型并行矩阵乘的输入需要全量张量，框架将会自动插入`AllGather`算子实现排布变换。 - + ![张量排布变换](./images/tensor_redistribution1.png) - + 在样例二中，第一个模型并行矩阵乘的输出在列方向上存在切分，而第二个数据并行矩阵乘的输入在行方向上存在切分，框架将会自动插入等价于集合通信中`AlltoAll`操作的通信算子实现排布变换。 ![张量排布变换](./images/tensor_redistribution2.png) - 在样例三中，第一个混合并行矩阵乘的输出切分方式和第二个混合并行矩阵乘的输入切分方式一致，所以不需要引入重排布变换。但由于第二个矩阵乘操作中，两个输入的相关维度存在切分，所以需要插入`AllReduce`算子保证运算正确性。 ![张量排布变换](./images/tensor_redistribution3.png) - 综上，1、2两点是自动并行实现的基础，总体来说这种分布式表达打破了数据并行和模型并行的边界，轻松实现混合并行。从脚本层面上，用户仅需构造单机网络，即可表达并行算法逻辑，框架将自动实现对整图切分。 3. 切分策略搜索算法 当用户熟悉了算子的切分表达，并手动对算子配置切分策略，这就是`SEMI_AUTO_PARALLEL`半自动并行模式。这种方式对手动调优有帮助，但还是具有一定的调试难度，用户需要掌握并行原理，并根据网络结构、集群拓扑等计算分析得到高性能的并行方案。为了进一步帮助用户加速并行网络训练过程，在半自动并行模式的基础上，`AUTO_PARALLEL`自动并行模式引入了并行切分策略自动搜索的特性。自动并行围绕硬件平台构建相应的代价函数模型（Cost Model），计算出一定数据量、一定算子在不同切分策略下的计算开销（Computation Cost），内存开销（Memory Cost）及通信开销（Communication Cost）。然后通过动态规划算法（Dynamic Programming）或者递归规划算法（Recursive Programming），以单卡的内存上限为约束条件，高效地搜索出性能较优的切分策略。 - - 策略搜索这一步骤代替了用户手动指定模型切分，在短时间内可以得到较高性能的切分方案，极大降低了并行训练的使用门槛。 + 策略搜索这一步骤代替了用户手动指定模型切分，在短时间内可以得到较高性能的切分方案，极大降低了并行训练的使用门槛。 4. 分布式自动微分 @@ -140,7 +135,5 @@ 5. 整图切分 - [step_auto_parallel.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/frontend/parallel/step_auto_parallel.h), [step_parallel.h](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ccsrc/frontend/parallel/step_parallel.h)：这两个文件包含了自动并行流程的核心实现。首先由`step_auto_parallel.h`调用策略搜索流程并产生分布式算子的`OperatorInfo`，然后在`step_parallel.h`中处理算子切分和张量重排布等流程，对单机计算图进行分布式改造。 - 6. 通信算子反向 - [grad_comm_ops.py](https://gitee.com/mindspore/mindspore/blob/master/mindspore/ops/_grad/grad_comm_ops.py)：这个文件定义了`AllReduce`和`AllGather`等通信算子的反向操作。 - diff --git a/docs/note/source_zh_cn/design/mindspore/mindir.md b/docs/note/source_zh_cn/design/mindspore/mindir.md index e501365e3740586a898f86d30eff8866709e61e9..01fd8b8ab770c1db0a3749607f368199f41e36bc 100644 --- a/docs/note/source_zh_cn/design/mindspore/mindir.md +++ b/docs/note/source_zh_cn/design/mindspore/mindir.md @@ -20,13 +20,16 @@ ## 简介 + 中间表示（IR）是程序编译过程中介于源语言和目标语言之间的程序表示，以方便编译器进行程序分析和优化，因此IR的设计需要考虑从源语言到目标语言的转换难度，同时考虑程序分析和优化的易用性和性能。 MindIR是一种基于图表示的函数式IR，其最核心的目的是服务于自动微分变换。自动微分采用的是基于函数式编程框架的变换方法，因此IR采用了接近于ANF函数式的语义。此外，借鉴Sea of Nodes[1]和Thorin[2]的优秀设计，采用了一种基于显性依赖图的表示方式。 ## 文法定义 + ANF是函数式编程中常用且简洁的中间表示，其文法定义如下所示： -``` + +```text ::= NUMBER | STRING | VAR | BOOLEAN | PRIMOP | (lambda (VAR …) ) ::= ( …) @@ -34,17 +37,20 @@ ANF是函数式编程中常用且简洁的中间表示，其文法定义如下 ::= (let ([VAR ]) ) | | ``` + ANF中表达式分为原子表达式（aexp）和复合表达式（cexp），原子表达式表示一个常数值或一个变量或一个匿名函数；复合表达式由多个原子表达式复合组成，表示一个匿名函数或原语函数调用，组合的第一个输入是调用的函数，其余输入是调用的参数。 MindIR文法继承于ANF，其定义如下所示： -``` + +```text ::= | ::= Parameter ::= Scalar | Named | Tensor | Type | Shape - | Primitive | MetaFuncGraph | FuncGraph + | Primitive | MetaFuncGraph | FuncGraph ::= ( …) ::= | ``` + MindIR中的ANode对应于ANF的原子表达式，ANode有两个子类分别为ValueNode和ParameterNode。ValueNode表示常数节点，可承载一个常数值（标量、符号、张量、类型、维度等），也可以是一个原语函数（Primitive）或一个元函数（MetaFuncGraph）或一个普通函数（FuncGraph），因为在函数式编程中函数定义本身也是一个值。ParameterNode是参数节点，表示函数的形参。 MindIR中CNode对应于ANF的复合表达式，表示一次函数调用。 @@ -52,7 +58,9 @@ MindIR中CNode对应于ANF的复合表达式，表示一次函数调用。 在MindSpore自动微分时，会计算ParameterNode和CNode的梯度贡献，并返回最终ParameterNode的梯度，而不计算ValueNode的梯度。 ## 示例 + 下面以一段程序作为示例，对比理解MindIR。 + ```python def func(x, y): return x / y @@ -64,8 +72,10 @@ def test_f(x, y): c = b * func(a, b) return c ``` + 这段Python代码对应的ANF表达为： -``` + +```python lambda (x, y) let a = x - 1 in let b = a + y in @@ -76,26 +86,31 @@ lambda (x, y) let c = b * %1 in c end ``` + 对应的MindIR为[ir.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_zh_cn/design/mindspore/images/ir/ir.dot)： -![](./images/ir/ir.png) +![image](./images/ir/ir.png) 在MindIR中，一个函数图（FuncGraph）表示一个普通函数的定义，函数图一般由ParameterNode、ValueNode和CNode组成有向无环图，可以清晰地表达出从参数到返回值的计算过程。在上图中可以看出，python代码中两个函数`test_f`和`func`转换成了两个函数图，其参数`x`和`y`转换为函数图的ParameterNode，每一个表达式转换为一个CNode。CNode的第一个输入链接着调用的函数，例如图中的`add`、`func`、`return`。值得注意的是这些节点均是`ValueNode`，因为它们被理解为常数函数值。CNode的其他输入链接这调用的参数，参数值可以来自于ParameterNode、ValueNode和其他CNode。 在ANF中每个表达式都用let表达式绑定为一个变量，通过对变量的引用来表示对表达式输出的依赖，而在MindIR中每个表达式都绑定为一个节点，通过节点与节点之间的有向边表示依赖关系。 ## 如何保存IR + 通过`context.set_context(save_graphs=True)`来保存各个编译阶段的中间代码。被保存的中间代码有两种格式，一个是后缀名为`.ir`的文本格式，一个是后缀名为`.dot`的图形化格式。当网络规模不大时，建议使用更直观的图形化格式来查看，当网络规模较大时建议使用更高效的文本格式来查看。 DOT文件可以通过graphviz转换为图片格式来查看，例如将dot转换为png的命令是`dot -Tpng *.dot -o *.png`。 ## 函数式语义 + MindIR较传统计算图的一个重要特性是不仅可以表达算子之间的数据依赖，还可以表达丰富的函数式语义。 + ### 高阶函数 + 在MindIR中，函数的定义是由一个子图来定义，但其本身可以是一个被传递的值，作为其他高阶函数的输入或输出。 例如下面一个简单的示例中，函数`f`作为参数传入了函数`g`，因此函数`g`是一个接收函数输入的高阶函数，函数`f`真正的调用点是在函数`g`内部。 -``` +```python @ms_function def hof(x): def f(x): @@ -108,14 +123,16 @@ def hof(x): 对应的MindIR为[hof.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_zh_cn/design/mindspore/images/ir/hof.dot)： -![](./images/ir/hof.png) +![image](./images/ir/hof.png) 在实际网络训练脚本中，自动求导泛函`GradOperation`和优化器中常用到的`Partial`和`HyperMap`都是典型的高阶函数。高阶语义极大地提升了MindSpore表达的灵活性和简洁性。 ### 控制流 + 控制流在MindIR中是以高阶函数选择调用的形式表达。这样的形式把控制流转换为高阶函数的数据流，从而使得自动微分算法更加强大。不仅可以支持数据流的自动微分，还可以支持条件跳转、循环和递归等控制流的自动微分。 下面以一个简单的斐波那契用例来演示说明。 + ```python @ms_function def fibonacci(n): @@ -129,15 +146,16 @@ def fibonacci(n): 对应的MindIR为[cf.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_zh_cn/design/mindspore/images/ir/cf.dot)： -![](./images/ir/cf.png) +![image](./images/ir/cf.png) 其中`fibonacci`是顶层函数图，在顶层中有两个函数图被`switch`选择调用。`✓fibonacci`是第一个`if`的True分支，`✗fibonacci`是第一个`if`的False分支。在`✗fibonacci`中被调用的`✓✗fibonacci`是`elif`的True分支，`✗✗fibonacci`是`elif`的False分支。这里需要理解的关键是在MindIR中，条件跳转和递归是以高阶控制流的形式表达的。例如，`✓fibonacci`和`✗fibonacci`是作为`switch`算子的参数传入，`switch`根据条件参数选择哪一个函数作为返回值。因此，`switch`是把输入的函数当成普通的值做了一个二元选择操作，并没有调用，而真正的函数调用是在紧随`switch`后的CNode上完成。 - ### 自由变量和闭包 + 闭包（closure）是一种编程语言特性，它指的是代码块和作用域环境的结合。自由变量(free variable)是指在代码块中引用作用域环境中的变量而非局部变量。在MindIR中，代码块是以函数图呈现的，而作用域环境可以理解为该函数被调用时的上下文环境，自由变量的捕获方式是值拷贝而非引用。 一个典型的闭包用例如下： + ```python @ms_function def func_outer(a, b): @@ -155,14 +173,15 @@ def ms_closure(): 对应的MindIR为[closure.dot](https://gitee.com/mindspore/docs/blob/master/docs/note/source_zh_cn/design/mindspore/images/ir/closure.dot)： -![](./images/ir/closure.png) +![image](./images/ir/closure.png) 在例子中，`a`和`b`是自由变量，因为`func_inner`中变量`a`和`b`是引用的其父图`func_outer`中定义的参数。变量`closure`是一个闭包，它是函数`func_inner`与其上下文`func_outer(1, 2)`的结合。因此，`out1`的结果是4，因为其等价于`1+2+1`，`out2`的结果是5，因为其等价于`1+2+2`。 ## 参考文献 + [1] C. Click and M. Paleczny. A simple graph-based intermediate representation. SIGPLAN Not., 30:35–49, March 1995. [2] Roland Leißa, Marcel Köster, and Sebastian Hack. A graph-based higher-order intermediate representation. In Proceedings of the 13th Annual IEEE/ACM International Symposium on -Code Generation and Optimization, pages 202–212. IEEE Computer Society, 2015. \ No newline at end of file +Code Generation and Optimization, pages 202–212. IEEE Computer Society, 2015. diff --git a/docs/note/source_zh_cn/design/mindspore/profiler_design.md b/docs/note/source_zh_cn/design/mindspore/profiler_design.md index 9b0b965b0ae7634703ead68a728d949967424cd9..219b40dee8b3e6ac567c396e604d91adb7f260f0 100644 --- a/docs/note/source_zh_cn/design/mindspore/profiler_design.md +++ b/docs/note/source_zh_cn/design/mindspore/profiler_design.md @@ -5,23 +5,23 @@ - [Profiler设计文档](#profiler设计文档) - - [背景](#背景) - - [Profiler框架设计](#profiler架构设计) - - [上下文](#上下文) - - [模块层级结构](#模块层级结构) - - [内部模块交互](#内部模块交互) - - [子模块设计](#准备训练脚本) - - [ProfilerAPI和Controller](#profiler-api-controller) - - [ProfilerAPI和Controller模块介绍](#profiler-api-controller模块介绍) - - [Analyser](#analyser) - - [Analyser模块介绍](#analyser模块介绍) - - [Analyser模块设计](#analyser模块设计) - - [Parser](#parser) - - [Parser模块介绍](#parser模块介绍) - - [Parser模块设计](#parser模块设计) - - [Proposer](#proposer) - - [Proposer模块介绍](#proposer模块介绍) - - [Proposer模块设计](#proposer模块设计) + - [背景](#背景) + - [Profiler框架设计](#profiler架构设计) + - [上下文](#上下文) + - [模块层级结构](#模块层级结构) + - [内部模块交互](#内部模块交互) + - [子模块设计](#准备训练脚本) + - [ProfilerAPI和Controller](#profiler-api-controller) + - [ProfilerAPI和Controller模块介绍](#profiler-api-controller模块介绍) + - [Analyser](#analyser) + - [Analyser模块介绍](#analyser模块介绍) + - [Analyser模块设计](#analyser模块设计) + - [Parser](#parser) + - [Parser模块介绍](#parser模块介绍) + - [Parser模块设计](#parser模块设计) + - [Proposer](#proposer) + - [Proposer模块介绍](#proposer模块介绍) + - [Proposer模块设计](#proposer模块设计) @@ -32,6 +32,7 @@ 为了支持用户在MindSpore进行模型开发性能调试，需要提供易用的Profile工具，直观地展现网络模型各维度的性能信息，为用户提供易用、丰富的性能分析功能，帮助用户快速定位网络中性能问题。 ## Profiler架构设计 + 这一章将介绍Profiler的架构设计，第一节从整体Profiler的角度出发介绍其上下文交互关系，第二节将打开Profiler内部，介绍模块层架结构以及模块划分，第三节将介绍模块间的交互调用关系。 ### 上下文 @@ -49,6 +50,7 @@ Profiler是MindSpore调试调优工具的一部分，在整个使用过程中的 2. MindSpore侧Profiler将在用户脚本中对原始数据进行解析，并在用户指定的文件夹下面生成中间数据结果； 3. Mindinsight侧Profiler对接中间数据，提供可视化Profiler功能供用户使用。 + ### 模块层级结构 模块层级划分如下： @@ -57,8 +59,8 @@ Profiler是MindSpore调试调优工具的一部分，在整个使用过程中的 图2：层级模块关系图 - 如上图所示，各个模块功能介绍如下： + 1. ProfilerAPI是代码侧对用户提供的调用入口，为用户提供了性能收集启动接口以及分析接口； 2. Controller是ProfilerAPI下层的模块，被ProfilerAPI中的启动接口调用，负责控制下方性能收集功能的启动停止，原始数据会被ada写入固定位置； 3. Parser是性能原始数据解析模块，由于性能原始数据是在设备侧收集的信息，所以信息不能直接被用户所理解，该模块负责将信息进行解析、组合、转换，最终形成用户可理解、上层可分析的中间结果； @@ -66,6 +68,7 @@ Profiler是MindSpore调试调优工具的一部分，在整个使用过程中的 5. 通过RESTful调用后端Analyser提供的common API，获取目标数据，以RESTful接口对接前端。 ### 内部模块交互 + 从用户角度，有两种使用形式API、RESTful，我们以API为例，阐述一个完整的内部模块交互流程： ![time_order_profiler.png](./images/time_order_profiler.png) @@ -81,20 +84,22 @@ Profiler是MindSpore调试调优工具的一部分，在整个使用过程中的 3. Profiler API分析接口首先使用Parser模块对性能数据进行解析，产生中间结果，再调用Aalayser进行中间结果分析，最终将各类信息返回至用户侧。 ## 子模块设计 + ### ProfilerAPI和Controller #### ProfilerAPI和Controller模块说明 + ProfilerAPI为用户在训练脚本侧提供入口API，用户通过ProfilerAPI启动性能收集以及对性能数据进行分析。 ProfilerAPI通过Controller下发命令，完成对ada启动的控制。 #### ProfilerAPI和Controller模块设计 + ProfilerAPI模块，属于上层应用接口层，由训练脚本集成。功能分为两部分： - 训练前调用底层Controller接口，下发命令，启动profiling统计任务。 - 训练完成后，调用底层Controller接口，下发命令，停止性能统计任务，再调用Analyser、Parser模块接口解析数据文件，生成算子性能统计、training trace统计等结果数据。 - Controller模块提供对上层接口，并调用底层性能收集模块接口，下发启动和停止性能收集的命令。 最终生成的性能原始数据主要包含： @@ -105,9 +110,13 @@ Controller模块提供对上层接口，并调用底层性能收集模块接口 - `training_trace.46.dev.profiler_default_tag`文件：存储每个step的开始结束时刻，迭代间隙、迭代前向反向、迭代拖尾的时刻信息。 ### Parser + #### Parser模块介绍 + Parser是原始性能数据解析模块，由于原始性能数据是在设备侧收集的信息，所以信息不能直接被用户所理解，该模块负责将信息进行解析、组合、转换，最终形成用户可理解、上层可分析的中间结果。 + #### Parser模块设计 + ![parser_module_profiler.png](./images/parser_module_profiler.png) 图4：Parser模块图 @@ -122,6 +131,7 @@ Parser是原始性能数据解析模块，由于原始性能数据是在设备 ### Analyser #### Analyser模块介绍 + 分析器的作用是对解析阶段生成的中间结果，进行筛选、排序、查询、分页等相关操作。 #### Analyser模块设计 @@ -141,9 +151,10 @@ Parser是原始性能数据解析模块，由于原始性能数据是在设备 为了隐藏Analyser内部实现，方便调用，使用简单工厂模式，通过AnalyserFactory获取指定的Analyser。 - ### Proposer + #### Proposer模块介绍 + Proposer是Profiler性能优化建议模块，Proposer调用Analyser模块获取性能数据，通过调优规则对性能数据进行分析，输出调优建议由UI、API接口展示给用户。 #### Proposer模块设计 @@ -171,4 +182,4 @@ Proposer是Profiler性能优化建议模块，Proposer调用Analyser模块获取 如上模块类图所示： - 各类型Proposer继承抽象类Proposer并实现analyze方法； -- API、CLI通过调用工厂ProposerFactory获取Proposer，并调用Proposer.analyze函数获取各类型的Proposer分析的优化建议。 \ No newline at end of file +- API、CLI通过调用工厂ProposerFactory获取Proposer，并调用Proposer.analyze函数获取各类型的Proposer分析的优化建议。 diff --git a/docs/note/source_zh_cn/design/technical_white_paper.md b/docs/note/source_zh_cn/design/technical_white_paper.md index 244f94705c0fc50fe16a30c1b26fc10e50dde282..c3ec41c35159513d72d40770a3fdbe593ce3bbf9 100644 --- a/docs/note/source_zh_cn/design/technical_white_paper.md +++ b/docs/note/source_zh_cn/design/technical_white_paper.md @@ -13,11 +13,13 @@ ## 引言 + 深度学习研究和应用在近几十年得到了爆炸式的发展，掀起了人工智能的第三次浪潮，并且在图像识别、语音识别与合成、无人驾驶、机器视觉等方面取得了巨大的成功。这也对算法的应用以及依赖的框架有了更高级的要求。深度学习框架的不断发展使得在大型数据集上训练神经网络模型时，可以方便地使用大量的计算资源。 深度学习是使用多层结构从原始数据中自动学习并提取高层次特征的一类机器学习算法。通常，从原始数据中提取高层次、抽象的特征是非常困难的。目前有两种主流的深度学习框架：一种是在执行之前构造一个静态图，定义所有操作和网络结构，典型代表是TensorFlow，这种方法以牺牲易用性为代价，来提高训练期间的性能；另一种是立即执行的动态图计算，典型代表是PyTorch。通过比较可以发现，动态图更灵活、更易调试，但会牺牲性能。因此，现有深度学习框架难以同时满足易开发、高效执行的要求。 ## 简介 + MindSpore作为新一代深度学习框架，是源于全产业的最佳实践，最佳匹配昇腾处理器算力，支持终端、边缘、云全场景灵活部署，开创全新的AI编程范式，降低AI开发门槛。MindSpore是一种全新的深度学习计算框架，旨在实现易开发、高效执行、全场景覆盖三大目标。为了实现易开发的目标，MindSpore采用基于源码转换（Source Code Transformation，SCT）的自动微分（Automatic Differentiation，AD）机制，该机制可以用控制流表示复杂的组合。函数被转换成函数中间表达（Intermediate Representation，IR），中间表达构造出一个能够在不同设备上解析和执行的计算图。在执行前，计算图上应用了多种软硬件协同优化技术，以提升端、边、云等不同场景下的性能和效率。MindSpore支持动态图，更易于检查运行模式。由于采用了基于源码转换的自动微分机制，所以动态图和静态图之间的模式切换非常简单。为了在大型数据集上有效训练大模型，通过高级手动配置策略，MindSpore可以支持数据并行、模型并行和混合并行训练，具有很强的灵活性。此外，MindSpore还有“自动并行”能力，它通过在庞大的策略空间中进行高效搜索来找到一种快速的并行策略。MindSpore框架的具体优势，请查看详细介绍。 -[查看技术白皮书](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com:443/white_paper/MindSpore_white_paper.pdf) \ No newline at end of file +[查看技术白皮书](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com:443/white_paper/MindSpore_white_paper.pdf) diff --git a/docs/note/source_zh_cn/glossary.md b/docs/note/source_zh_cn/glossary.md index 4297fd2647161670879a40c7ee2522077f4d8160..630fec2ea6cbb0bfe4cbbd913419853a056adc57 100644 --- a/docs/note/source_zh_cn/glossary.md +++ b/docs/note/source_zh_cn/glossary.md @@ -4,7 +4,7 @@ -| 术语/缩略语 | 说明 | +| 术语/缩略语 | 说明 | | ----- | ----- | | ACL | Ascend Computer Language，提供Device管理、Context管理、Stream管理、内存管理、模型加载与执行、算子加载与执行、媒体数据处理等C++ API库，供用户开发深度神经网络应用。| | Ascend | 华为昇腾系列芯片的系列名称。 | diff --git a/docs/note/source_zh_cn/help_seeking_path.md b/docs/note/source_zh_cn/help_seeking_path.md index f3de79c3733fbf48e529bb4b65b1488d5e1fda42..ac3338260cf2cc17cbd838c5e7fc101da5021cf1 100644 --- a/docs/note/source_zh_cn/help_seeking_path.md +++ b/docs/note/source_zh_cn/help_seeking_path.md @@ -10,25 +10,23 @@ - 网站搜索 - - 进入[官网搜索](https://www.mindspore.cn/search)。 - - 遇到问题时，首先推荐使用网站搜索方法，该方法操作简单、高效。 - - 在搜索框输入问题的关键词，点击搜索，可匹配出与关键词相关的内容。 - - 参考搜索结果，解决当前遇到的问题。 - + - 进入[官网搜索](https://www.mindspore.cn/search)。 + - 遇到问题时，首先推荐使用网站搜索方法，该方法操作简单、高效。 + - 在搜索框输入问题的关键词，点击搜索，可匹配出与关键词相关的内容。 + - 参考搜索结果，解决当前遇到的问题。 - 用户群咨询 - - QQ用户群号：871543426。 - - 如果网站搜索方法不能解决当前问题，可通过QQ用户群咨询，建议想要简单咨询的用户选取此方法。 - - 加群后可以与其他用户讨论交流，还有技术专家在群中提供帮助解答。 - - 通过专家的解答或和其他用户的交流来解决当前遇到的问题。 - + - QQ用户群号：871543426。 + - 如果网站搜索方法不能解决当前问题，可通过QQ用户群咨询，建议想要简单咨询的用户选取此方法。 + - 加群后可以与其他用户讨论交流，还有技术专家在群中提供帮助解答。 + - 通过专家的解答或和其他用户的交流来解决当前遇到的问题。 - 论坛求助 - - 如果用户想要详细的解决方法，可通过[MindSpore论坛](https://bbs.huaweicloud.com/forum/forum-1076-1.html)中发布问题求助帖获取解答。 - - 为提高问题解决速度与质量，发帖前请参考[发帖建议](https://bbs.huaweicloud.com/forum/thread-69695-1-1.html)，按照建议格式发帖。 - - 帖子发出后会有论坛版主负责将问题收录，并联系技术专家进行解答，问题将在三个工作日内解决。 - - 参考技术专家的解决方案，解决当前遇到的问题。 + - 如果用户想要详细的解决方法，可通过[MindSpore论坛](https://bbs.huaweicloud.com/forum/forum-1076-1.html)中发布问题求助帖获取解答。 + - 为提高问题解决速度与质量，发帖前请参考[发帖建议](https://bbs.huaweicloud.com/forum/thread-69695-1-1.html)，按照建议格式发帖。 + - 帖子发出后会有论坛版主负责将问题收录，并联系技术专家进行解答，问题将在三个工作日内解决。 + - 参考技术专家的解决方案，解决当前遇到的问题。 - 如果在专家测试后确定是MindSpore功能有待完善，推荐用户在[MindSpore仓](https://gitee.com/mindspore)中创建ISSUE，所提问题会在后续的版本中得到修复完善。 \ No newline at end of file + 如果在专家测试后确定是MindSpore功能有待完善，推荐用户在[MindSpore仓](https://gitee.com/mindspore)中创建ISSUE，所提问题会在后续的版本中得到修复完善。 diff --git a/docs/note/source_zh_cn/image_classification_lite.md b/docs/note/source_zh_cn/image_classification_lite.md index 4aa2960c32af591cbc6c277241358a80acea9a1b..10a4a449f18096e83803f77f67d0e18314cc19b5 100644 --- a/docs/note/source_zh_cn/image_classification_lite.md +++ b/docs/note/source_zh_cn/image_classification_lite.md @@ -4,7 +4,7 @@ ## 图像分类介绍 -图像分类模型可以预测图片中出现哪些物体，识别出图片中出现物体列表及其概率。 比如下图经过模型推理的分类结果为下表： +图像分类模型可以预测图片中出现哪些物体，识别出图片中出现物体列表及其概率。 比如下图经过模型推理的分类结果为下表： ![image_classification](images/image_classification_result.png) @@ -30,4 +30,3 @@ | [Shufflenetv2](https://download.mindspore.cn/model_zoo/official/lite/shufflenetv2_lite/shufflenetv2.ms) | 8.8 | 67.74% | 87.62% | - | 8.303 | | [GoogleNet](https://download.mindspore.cn/model_zoo/official/lite/googlenet_lite/googlenet.ms) | 25.3 | 72.2% | 90.06% | - | 23.257 | | [ResNext50](https://download.mindspore.cn/model_zoo/official/lite/resnext50_lite/resnext50.ms) | 95.8 | 73.1% | 91.21% | - | 138.164 | - diff --git a/docs/note/source_zh_cn/network_list_ms.md b/docs/note/source_zh_cn/network_list_ms.md index 20b1d9107b3905545742e2ec8d84188ab3655e7a..5eca8d6bc206b3f18df676a916956db42696acad 100644 --- a/docs/note/source_zh_cn/network_list_ms.md +++ b/docs/note/source_zh_cn/network_list_ms.md @@ -1,7 +1,7 @@ # MindSpore网络支持 `Linux` `Ascend` `GPU` `CPU` `模型开发` `中级` `高级` - + - [MindSpore网络支持](#mindspore网络支持) @@ -13,7 +13,7 @@ ## Model Zoo -| 领域 | 子领域 | 网络 | Ascend(Graph) | Ascend(PyNative) | GPU(Graph) | GPU(PyNaitve) | CPU(Graph) +| 领域 | 子领域 | 网络 | Ascend(Graph) | Ascend(PyNative) | GPU(Graph) | GPU(PyNaitve) | CPU(Graph) |:---- |:------- |:---- |:---- |:---- |:---- |:---- |:---- |计算机视觉（CV） | 图像分类（Image Classification） | [AlexNet](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/alexnet/src/alexnet.py) | Supported | Supported | Supported | Supported | Doing | 计算机视觉（CV） | 图像分类（Image Classification） | [GoogleNet](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/googlenet/src/googlenet.py) | Supported | Supported | Supported | Supported | Doing diff --git a/docs/note/source_zh_cn/object_detection_lite.md b/docs/note/source_zh_cn/object_detection_lite.md index 486d48d79d2ae8ed4212a26291862e84de425efe..b278206f80621b9496f29c2b9daaa69196b80077 100644 --- a/docs/note/source_zh_cn/object_detection_lite.md +++ b/docs/note/source_zh_cn/object_detection_lite.md @@ -23,4 +23,3 @@ | 模型名称 | 大小 | mAP(IoU=0.50:0.95) | CPU 4线程时延(ms) | |-----------------------| :----------: | :----------: | :-----------: | | [MobileNetv2-SSD](https://download.mindspore.cn/model_zoo/official/lite/ssd_mobilenetv2_lite/ssd.ms) | 16.7 | 0.22 | 25.4 | - diff --git a/docs/note/source_zh_cn/operator_list_implicit.md b/docs/note/source_zh_cn/operator_list_implicit.md index 710dd8c2267dad7ac96d7648d1cdf39047b1c441..e37729db056c97c4c9e3f0061b9ea5efa8fecb14 100644 --- a/docs/note/source_zh_cn/operator_list_implicit.md +++ b/docs/note/source_zh_cn/operator_list_implicit.md @@ -15,25 +15,28 @@ ## 隐式类型转换 + ### 转换规则 -* 标量与Tensor运算：运算时，将标量自动转为Tensor，数据类型和参与运算的Tensor数据类型保持一致； + +- 标量与Tensor运算：运算时，将标量自动转为Tensor，数据类型和参与运算的Tensor数据类型保持一致； 当Tensor是bool数据类型，标量是int或float时，将标量和Tensor都转为数据类型为int32或float32的Tensor； 当Tensor是int或者uint数据类型，标量是float时，将标量和Tensor都转为数据类型为float32的Tensor。 -* 不同数据类型Tensor运算：数据类型优先级排序为bool < uint8 < int8 < int16 < int32 < int64 < float16 < float32 < float64， +- 不同数据类型Tensor运算：数据类型优先级排序为bool < uint8 < int8 < int16 < int32 < int64 < float16 < float32 < float64， 运算时，先确定参与运算的Tensor中优先级相对最高的数据类型，然后将低优先级数据类型Tensor转换为相对最高优先级数据类型； 而当int8和uint8数据类型的Tensor进行运算时，将其都转为int16的Tensor。 -* 不支持对Parameter进行数据类型转换：如果按照转换规则推导，需要对网络中定义的Parameter进行数据类型转换时，会抛出RuntimeError异常。 +- 不支持对Parameter进行数据类型转换：如果按照转换规则推导，需要对网络中定义的Parameter进行数据类型转换时，会抛出RuntimeError异常。 ### 参与转换的数据类型 -* bool -* int8 -* uint8 -* int16 -* int32 -* int64 -* float16 -* float32 -* float64 + +- bool +- int8 +- uint8 +- int16 +- int32 +- int64 +- float16 +- float32 +- float64 ### 支持算子 @@ -101,4 +104,3 @@ | [mindspore.ops.ScatterSub](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ScatterSub) | [mindspore.ops.ScatterMul](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ScatterMul) | [mindspore.ops.ScatterDiv](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ScatterDiv) - diff --git a/docs/note/source_zh_cn/operator_list_lite.md b/docs/note/source_zh_cn/operator_list_lite.md index 04bedcfb2c9191797f9159642f492e3e7e455752..1b9695e052e7c696b3bca6126484c2342c354467 100644 --- a/docs/note/source_zh_cn/operator_list_lite.md +++ b/docs/note/source_zh_cn/operator_list_lite.md @@ -109,6 +109,6 @@ | Unsqueeze | | Supported | Supported | Supported | | | | | Unsqueeze | | Unstack | | Supported | | | | | Unstack | | | | Where | | Supported | | | | | Where | | | -| ZerosLike | | Supported | | | | | ZerosLike | | | +| ZerosLike | | Supported | | | | | ZerosLike | | | * Clip：仅支持将clip(0, 6)转换为Relu6。 diff --git a/docs/note/source_zh_cn/operator_list_ms.md b/docs/note/source_zh_cn/operator_list_ms.md index f18271b91f55c22a580773285f911dfa0457b748..c4b6fb10ffccf6cfb2125cc55754eeffbbdeaaca 100644 --- a/docs/note/source_zh_cn/operator_list_ms.md +++ b/docs/note/source_zh_cn/operator_list_ms.md @@ -46,7 +46,8 @@ | [mindspore.nn.SSIM](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.SSIM) | Supported | Supported | Doing |layer/image | [mindspore.nn.PSNR](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.PSNR) | Supported |Supported | Doing |layer/image | [mindspore.nn.CentralCrop](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.CentralCrop) | Supported |Supported | Doing |layer/image -| [mindspore.nn.LSTM](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.LSTM) | Doing | Supported | Supported |layer/lstm +| [mindspore.nn.LSTM](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.LSTM) | Doing | Supported | Doing |layer/lstm +| [mindspore.nn.LSTMCell](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.LSTMCell) | Doing | Supported | Supported |layer/lstm | [mindspore.nn.GlobalBatchNorm](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.GlobalBatchNorm) | Supported |Doing | Doing |layer/normalization | [mindspore.nn.BatchNorm1d](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.BatchNorm1d) | Supported |Doing | Doing |layer/normalization | [mindspore.nn.BatchNorm2d](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.BatchNorm2d) | Supported | Supported | Doing |layer/normalization @@ -104,6 +105,8 @@ | [mindspore.nn.LGamma](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.LGamma) |Supported | Doing | Doing |layer/math | [mindspore.nn.ReduceLogSumExp](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.ReduceLogSumExp) |Supported | Supported | Doing |layer/math | [mindspore.nn.MSSSIM](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.MSSSIM) | Supported |Doing | Doing |layer/image +| [mindspore.nn.AvgPool1d](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.AvgPool1d) | Supported | Doing | Doing |layer/pooling +| [mindspore.nn.Unfold](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.Unfold) |Supported | Doing | Doing |layer/basic ## mindspore.ops.operations @@ -184,18 +187,18 @@ | [mindspore.ops.ReduceSum](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceSum) | Supported | Supported | Supported | math_ops | [mindspore.ops.ReduceAll](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceAll) | Supported | Doing | Doing | math_ops | [mindspore.ops.ReduceMax](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceMax) | Supported | Supported | Supported | math_ops -| [mindspore.ops.ReduceMin](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceMin) | Supported | Supported | Doing | math_ops -| [mindspore.ops.ReduceProd](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceProd) | Supported | Doing | Doing | math_ops -| [mindspore.ops.CumProd](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.CumProd) | Supported | Doing | Doing | math_ops +| [mindspore.ops.ReduceMin](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceMin) | Supported | Supported | Doing | math_ops +| [mindspore.ops.ReduceProd](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceProd) | Supported | Doing | Doing | math_ops +| [mindspore.ops.CumProd](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.CumProd) | Supported | Doing | Doing | math_ops | [mindspore.ops.MatMul](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.MatMul) | Supported | Supported | Supported | math_ops | [mindspore.ops.BatchMatMul](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.BatchMatMul) | Supported | Supported | Doing | math_ops | [mindspore.ops.CumSum](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.CumSum) | Supported | Supported| Doing | math_ops | [mindspore.ops.AddN](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.AddN) | Supported | Supported | Supported | math_ops | [mindspore.ops.Neg](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Neg) | Supported | Supported | Doing | math_ops -| [mindspore.ops.Sub](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Sub) | Supported | Supported | Supported | math_ops +| [mindspore.ops.Sub](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Sub) | Supported | Supported | Supported | math_ops | [mindspore.ops.Mul](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Mul) | Supported | Supported | Supported | math_ops -| [mindspore.ops.Square](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Square) | Supported | Supported | Supported | math_ops -| [mindspore.ops.SquareSumAll](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.SquareSumAll) | Supported | Doing | Doing | math_ops +| [mindspore.ops.Square](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Square) | Supported | Supported | Supported | math_ops +| [mindspore.ops.SquareSumAll](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.SquareSumAll) | Supported | Doing | Doing | math_ops | [mindspore.ops.Rsqrt](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Rsqrt) | Supported | Supported | Doing | math_ops | [mindspore.ops.Sqrt](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Sqrt) | Supported | Supported | Doing | math_ops | [mindspore.ops.Reciprocal](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Reciprocal) | Supported | Supported | Doing | math_ops @@ -203,7 +206,7 @@ | [mindspore.ops.Exp](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Exp) | Supported | Supported | Doing | math_ops | [mindspore.ops.Log](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Log) | Supported | Supported | Doing | math_ops | [mindspore.ops.Log1p](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Log1p) | Supported | Doing | Doing | math_ops -| [mindspore.ops.Minimum](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Minimum) | Supported | Supported | Doing | math_ops +| [mindspore.ops.Minimum](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Minimum) | Supported | Supported | Doing | math_ops | [mindspore.ops.Maximum](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Maximum) | Supported | Supported | Doing | math_ops | [mindspore.ops.RealDiv](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.RealDiv) | Supported | Supported | Doing | math_ops | [mindspore.ops.Div](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Div) | Supported | Supported | Doing | math_ops @@ -267,7 +270,7 @@ | [mindspore.ops.GatherV2](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.GatherV2) | Supported | Supported | Supported | array_ops | [mindspore.ops.Split](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Split) | Supported | Supported | Doing | array_ops | [mindspore.ops.Rank](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Rank) | Supported | Supported | Supported | array_ops -| [mindspore.ops.TruncatedNormal](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.TruncatedNormal) | Supported | Supported | Supported | array_ops +| [mindspore.ops.TruncatedNormal](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.TruncatedNormal) | Doing | Supported | Supported | array_ops | [mindspore.ops.Size](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Size) | Supported | Supported | Supported | array_ops | [mindspore.ops.Fill](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Fill) | Supported | Supported | Supported | array_ops | [mindspore.ops.OnesLike](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.OnesLike) | Supported | Supported | Doing | array_ops @@ -291,7 +294,7 @@ | [mindspore.ops.StridedSlice](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.StridedSlice) | Supported | Supported | Supported | array_ops | [mindspore.ops.Diag](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Diag) | Doing | Doing | Doing | array_ops | [mindspore.ops.DiagPart](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.DiagPart) | Doing | Doing | Doing | array_ops -| [mindspore.ops.Eye](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Eye) | Supported | Supported | Supported | array_ops +| [mindspore.ops.Eye](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Eye) | Supported | Supported | Supported | array_ops | [mindspore.ops.ScatterNd](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ScatterNd) | Supported | Supported | Doing | array_ops | [mindspore.ops.ResizeNearestNeighbor](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ResizeNearestNeighbor) | Supported | Supported | Doing | array_ops | [mindspore.ops.GatherNd](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.GatherNd) | Supported | Supported | Doing | array_ops @@ -365,6 +368,9 @@ | [mindspore.ops.IFMR](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.IFMR) | Supported | Doing | Doing | math_ops | [mindspore.ops.DynamicShape](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.DynamicShape) | Supported | Supported | Supported | array_ops | [mindspore.ops.Unique](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.Unique) | Doing | Doing | Doing | array_ops +| [mindspore.ops.ReduceAny](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ReduceAny) | Supported | Doing | Doing | math_ops +| [mindspore.ops.SparseToDense](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.SparseToDense) | Doing | Doing | Doing | sparse_ops +| [mindspore.ops.CTCGreedyDecoder](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.CTCGreedyDecoder) | Doing | Doing | Doing | nn_ops ## mindspore.ops.functional diff --git a/docs/note/source_zh_cn/roadmap.md b/docs/note/source_zh_cn/roadmap.md index a98ee9f3a1ad580bf8aa4855b1e049d43de4a23a..62c7657b6f592cb4632dbc4a31653fdd3643a19a 100644 --- a/docs/note/source_zh_cn/roadmap.md +++ b/docs/note/source_zh_cn/roadmap.md @@ -20,6 +20,7 @@ 以下将展示MindSpore近一年的高阶计划，我们会根据用户的反馈诉求，持续调整计划的优先级。 总体而言，我们会努力在以下几个方面不断改进。 + 1. 提供更多的预置模型支持。 2. 持续补齐API和算子库，改善易用性和编程体验。 3. 提供华为昇腾AI处理器的全面支持，并不断优化性能及软件架构。 @@ -28,56 +29,63 @@ 热忱希望各位在用户社区加入讨论，并贡献您的建议。 ## 预置模型 -* CV：目标检测、GAN、图像分割、姿态识别等场景经典模型。 -* NLP：RNN、Transformer类型神经网络，拓展基于Bert预训练模型的应用。 -* 其它：GNN、强化学习、概率编程、AutoML等。 + +- CV：目标检测、GAN、图像分割、姿态识别等场景经典模型。 +- NLP：RNN、Transformer类型神经网络，拓展基于Bert预训练模型的应用。 +- 其它：GNN、强化学习、概率编程、AutoML等。 ## 易用性 -* 补齐算子、优化器、Loss函数等各类API -* 完善Python语言原生表达支持 -* 支持常见的Tensor/Math操作 -* 增加更多的自动并行适用场景，提高策略搜索的准确性 + +- 补齐算子、优化器、Loss函数等各类API +- 完善Python语言原生表达支持 +- 支持常见的Tensor/Math操作 +- 增加更多的自动并行适用场景，提高策略搜索的准确性 ## 性能优化 -* 优化编译时间 -* 低比特混合精度训练/推理 -* 提升内存使用效率 -* 提供更多的融合优化手段 -* 加速PyNative执行性能 + +- 优化编译时间 +- 低比特混合精度训练/推理 +- 提升内存使用效率 +- 提供更多的融合优化手段 +- 加速PyNative执行性能 ## 架构演进 -* 图算融合优化：使用细粒度Graph IR表达算子，构成带算子边界的中间表达，挖掘更多图层优化机会。 -* 支持更多编程语言 -* 优化数据增强的自动调度及分布式训练数据缓存机制 -* 持续完善MindSpore IR -* Parameter Server模式分布式训练 + +- 图算融合优化：使用细粒度Graph IR表达算子，构成带算子边界的中间表达，挖掘更多图层优化机会。 +- 支持更多编程语言 +- 优化数据增强的自动调度及分布式训练数据缓存机制 +- 持续完善MindSpore IR +- Parameter Server模式分布式训练 ## MindInsight调试调优 -* 训练过程观察 - * 直方图 - * 计算图/数据图展示优化 - * 集成性能Profiling/Debugger工具 - * 支持多次训练间的对比 -* 训练结果溯源 - * 数据增强溯源对比 -* 训练过程诊断 - * 性能Profiling - * 基于图模型的Debugger + +- 训练过程观察 + - 直方图 + - 计算图/数据图展示优化 + - 集成性能Profiling/Debugger工具 + - 支持多次训练间的对比 +- 训练结果溯源 + - 数据增强溯源对比 +- 训练过程诊断 + - 性能Profiling + - 基于图模型的Debugger ## MindArmour安全增强包 -* 测试模型的安全性 -* 提供模型安全性增强工具 -* 保护训练和推理过程中的数据隐私 + +- 测试模型的安全性 +- 提供模型安全性增强工具 +- 保护训练和推理过程中的数据隐私 ## 推理框架 -* 算子性能与完备度的持续优化 -* 支持语音模型推理 -* 端侧模型的可视化 -* Micro方案，适用于嵌入式系统的超轻量化推理， 支持ARM Cortex-A、Cortex-M硬件 -* 支持端侧重训及联邦学习 -* 端侧自动并行特性 -* 端侧MindData，包含图片Resize、像素数据转换等功能 -* 配套MindSpore混合精度量化训练（或训练后量化），实现混合精度推理，提升推理性能 -* 支持Kirin NPU、MTK APU等AI加速硬件 -* 支持多模型推理pipeline -* C++构图接口 + +- 算子性能与完备度的持续优化 +- 支持语音模型推理 +- 端侧模型的可视化 +- Micro方案，适用于嵌入式系统的超轻量化推理， 支持ARM Cortex-A、Cortex-M硬件 +- 支持端侧重训及联邦学习 +- 端侧自动并行特性 +- 端侧MindData，包含图片Resize、像素数据转换等功能 +- 配套MindSpore混合精度量化训练（或训练后量化），实现混合精度推理，提升推理性能 +- 支持Kirin NPU、MTK APU等AI加速硬件 +- 支持多模型推理pipeline +- C++构图接口 diff --git a/docs/programming_guide/source_en/api_structure.md b/docs/programming_guide/source_en/api_structure.md index 8fb4885d0125e4cb33a9a8a4f60b8111b2a936a4..c77f085b66eb17f5feb1e9c1c6fe5ddfb29bacf9 100644 --- a/docs/programming_guide/source_en/api_structure.md +++ b/docs/programming_guide/source_en/api_structure.md @@ -12,6 +12,7 @@ ## Overall Architecture + MindSpore is a deep learning framework in all scenarios, aiming to achieve easy development, efficient execution, and all-scenario coverage. Easy development features include API friendliness and low debugging difficulty. Efficient execution includes computing efficiency, data preprocessing efficiency, and distributed training efficiency. All-scenario coverage means that the framework supports cloud, edge, and device scenarios. The overall architecture of MindSpore consists of the Mind Expression (ME), Graph Engine (GE), and backend runtime. ME provides user-level APIs for scientific computing, building and training neural networks, and converting Python code of users into graphs. GE is a manager of operators and hardware resources, and is responsible for controlling execution of graphs received from ME. Backend runtime includes efficient running environments, such as the CPU, GPU, Ascend AI processors, and Android/iOS, on the cloud, edge, and device. For more information about the overall architecture, see [Overall Architecture](https://www.mindspore.cn/doc/note/en/master/design/mindspore/architecture.html). diff --git a/docs/programming_guide/source_en/dtype.md b/docs/programming_guide/source_en/dtype.md index 495c7573d02a5a9d011b9e14c87272bf030b9c76..29437cc5de0d6dfbf696f4548167d62f04f3d682 100644 --- a/docs/programming_guide/source_en/dtype.md +++ b/docs/programming_guide/source_en/dtype.md @@ -19,7 +19,8 @@ In the computation process of MindSpore, the `int` data type in Python is conver For details about the supported types, see . In the following code, the data type of MindSpore is int32. -``` + +```python from mindspore import dtype as mstype data_type = mstype.int32 @@ -28,11 +29,10 @@ print(data_type) The following information is displayed: -``` +```text Int32 ``` - ## Data Type Conversion API MindSpore provides the following APIs for conversion between NumPy data types and Python built-in data types: @@ -43,7 +43,7 @@ MindSpore provides the following APIs for conversion between NumPy data types an The following code implements the conversion between different data types and prints the converted type. -``` +```python from mindspore import dtype as mstype np_type = mstype.dtype_to_nptype(mstype.int32) @@ -57,8 +57,8 @@ print(py_type) The following information is displayed: -``` +```text Int64 -``` \ No newline at end of file +``` diff --git a/docs/programming_guide/source_en/tensor.md b/docs/programming_guide/source_en/tensor.md index 0054dc933a385e67a004ee0fe4581dc84e9266d4..c97345a6078b4f2cb9de21c215c96cc232f3ecea 100644 --- a/docs/programming_guide/source_en/tensor.md +++ b/docs/programming_guide/source_en/tensor.md @@ -19,7 +19,7 @@ Tensor is a basic data structure in the MindSpore network computing. For details Tensors of different dimensions represent different data. For example, a 0-dimensional tensor represents a scalar, a 1-dimensional tensor represents a vector, a 2-dimensional tensor represents a matrix, and a 3-dimensional tensor may represent the three channels of RGB images. -> All examples in this document can run in PyNative mode and do not support CPUs. +> All examples in this document can run in PyNative mode and do not support CPUs. ## Tensor Structure @@ -29,7 +29,7 @@ When `Tensor` is used as the initial value, dtype can be specified. If dtype is A code example is as follows: -``` +```python import numpy as np from mindspore import Tensor from mindspore.common import dtype as mstype @@ -46,7 +46,7 @@ print(x, "\n\n", y, "\n\n", z, "\n\n", m, "\n\n", n, "\n\n", p) The following information is displayed: -``` +```text [[1 2] [3 4]] @@ -66,12 +66,13 @@ True ### Attributes Tensor attributes include shape and data type (dtype). + - shape: a tuple - dtype: a data type of MindSpore A code example is as follows: -``` +```python import numpy as np from mindspore import Tensor from mindspore.common import dtype as mstype @@ -85,20 +86,21 @@ print(x_shape, x_dtype) The following information is displayed: -``` +```text (2, 2) Int32 ``` - + ### Methods Tensor methods include `all`, `any`, and `asnumpy`. Currently, the `all` and `any` methods support only Ascend. + - `all(axis, keep_dims)`: performs the `and` operation on a specified dimension to reduce the dimension. `axis` indicates the reduced dimension, and `keep_dims` indicates whether to retain the reduced dimension. - `any(axis, keep_dims)`: performs the `or` operation on a specified dimension to reduce the dimension. The parameter meaning is the same as that of `all`. - `asnumpy()`: converts `Tensor` to an array of NumPy. A code example is as follows: -``` +```python import numpy as np from mindspore import Tensor from mindspore.common import dtype as mstype @@ -113,7 +115,7 @@ print(x_all, "\n\n", x_any, "\n\n", x_array) The following information is displayed: -``` +```text False True @@ -121,4 +123,4 @@ True [[ True True] [False False]] -``` \ No newline at end of file +``` diff --git a/docs/programming_guide/source_zh_cn/api_structure.md b/docs/programming_guide/source_zh_cn/api_structure.md index 847cc1bcaea6760a9e1ca5c8ad8ac1fa350d4c62..2fd2d7b094bd81eca23360ebe868d763ac020000 100644 --- a/docs/programming_guide/source_zh_cn/api_structure.md +++ b/docs/programming_guide/source_zh_cn/api_structure.md @@ -14,6 +14,7 @@ ## 总体架构 + MindSpore是一个全场景深度学习框架，旨在实现易开发、高效执行、全场景覆盖三大目标，其中易开发表现为API友好、调试难度低，高效执行包括计算效率、数据预处理效率和分布式训练效率，全场景则指框架同时支持云、边缘以及端侧场景。 MindSpore总体架构分为前端表示层（Mind Expression，ME）、计算图引擎（Graph Engine，GE）和后端运行时三个部分。ME提供了用户级应用软件编程接口（Application Programming Interface，API），用于科学计算以及构建和训练神经网络，并将用户的Python代码转换为数据流图。GE是算子和硬件资源的管理器，负责控制从ME接收的数据流图的执行。后端运行时包含云、边、端上不同环境中的高效运行环境，例如CPU、GPU、Ascend AI处理器、 Android/iOS等。更多总体架构的相关内容请参见[总体架构](https://www.mindspore.cn/doc/note/zh-CN/master/design/mindspore/architecture.html)。 diff --git a/docs/programming_guide/source_zh_cn/augmentation.md b/docs/programming_guide/source_zh_cn/augmentation.md index a7741906d98497e6190a26024c1c6b162b620770..f19715460d5b9c756779c03de87263f3c61ca8d4 100644 --- a/docs/programming_guide/source_zh_cn/augmentation.md +++ b/docs/programming_guide/source_zh_cn/augmentation.md @@ -42,7 +42,7 @@ MindSpore目前支持的常用数据增强算子如下表所示，更多数据 | | Invert | 将图像进行反相。 | | |Compose | 将列表中的数据增强操作依次执行。 | -## c_transforms +## c_transforms 下面将简要介绍几种常用的`c_transforms`模块数据增强算子的使用方法。 @@ -51,6 +51,7 @@ MindSpore目前支持的常用数据增强算子如下表所示，更多数据 对输入图像进行在随机位置的裁剪。 **参数说明：** + - `size`：裁剪图像的尺寸。 - `padding`：填充的像素数量。 - `pad_if_needed`：原图小于裁剪尺寸时，是否需要填充。 @@ -98,7 +99,7 @@ plt.show() 输出结果如下： -``` +```text Source image Shape : (32, 32, 3) , Source label : 6 Cropped image Shape: (10, 10, 3) , Cropped label: 6 ------ @@ -119,6 +120,7 @@ Cropped image Shape: (10, 10, 3) , Cropped label: 9 对输入图像进行随机水平翻转。 **参数说明：** + - `prob`: 单张图片发生翻转的概率。 下面的样例首先使用随机采样器加载CIFAR-10数据集[1]，然后对已加载的图片进行概率为0.8的随机水平翻转，最后输出翻转前后的图片形状及对应标签，并对图片进行了展示。 @@ -164,7 +166,7 @@ plt.show() 输出结果如下： -``` +```text Source image Shape : (32, 32, 3) , Source label : 3 Flipped image Shape: (32, 32, 3) , Flipped label: 3 ------ @@ -188,6 +190,7 @@ Flipped image Shape: (32, 32, 3) , Flipped label: 9 对输入图像进行缩放。 **参数说明：** + - `self`：缩放的目标大小。 - `interpolation`：缩放时采用的插值方式。 @@ -231,7 +234,7 @@ plt.show() 输出结果如下： -``` +```text Source image Shape : (28, 28, 1) , Source label : 5 Flipped image Shape: (101, 101, 1) , Flipped label: 5 ------ @@ -297,7 +300,7 @@ plt.show() 输出结果如下： -``` +```text Source image Shape : (32, 32, 3) , Source label : 4 Flipped image Shape: (32, 32, 3) , Flipped label: 4 ------ @@ -362,7 +365,7 @@ plt.show() 输出结果如下： -``` +```text Transformed image Shape: (3, 200, 200) , Transformed label: 3 Transformed image Shape: (3, 200, 200) , Transformed label: 0 Transformed image Shape: (3, 200, 200) , Transformed label: 3 diff --git a/docs/programming_guide/source_zh_cn/auto_parallel.md b/docs/programming_guide/source_zh_cn/auto_parallel.md index c7b9ecc28d62d49a2b818c03ac7654043c23c8ed..7bdbf8ddbc3a1b51b88c89f9377ed73f5cb21d0b 100644 --- a/docs/programming_guide/source_zh_cn/auto_parallel.md +++ b/docs/programming_guide/source_zh_cn/auto_parallel.md @@ -61,7 +61,7 @@ MindSpore的分布式并行配置通过`auto_parallel_context`来进行集中管 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(device_num=8) context.get_auto_parallel_context("device_num") @@ -74,7 +74,7 @@ context.get_auto_parallel_context("device_num") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(global_rank=0) context.get_auto_parallel_context("global_rank") @@ -87,7 +87,7 @@ context.get_auto_parallel_context("global_rank") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(gradients_mean=False) context.get_auto_parallel_context("gradients_mean") @@ -98,10 +98,10 @@ context.get_auto_parallel_context("gradients_mean") `parallel_mode`表示并行模式，其值为字符串类型。用户可选择的模式有： - `stand_alone`：单机模式。 -- `data_parallel`：数据并行模式。 -- `hybrid_parallel`：混合并行模式。 -- `semi_auto_parallel`：半自动并行模式，即用户可通过`shard`方法给算子配置切分策略，若不配置策略，则默认是数据并行策略。 -- `auto_parallel`：自动并行模式，即框架会自动建立代价模型，为用户选择最优的切分策略。 +- `data_parallel`：数据并行模式。 +- `hybrid_parallel`：混合并行模式。 +- `semi_auto_parallel`：半自动并行模式，即用户可通过`shard`方法给算子配置切分策略，若不配置策略，则默认是数据并行策略。 +- `auto_parallel`：自动并行模式，即框架会自动建立代价模型，为用户选择最优的切分策略。 其中`auto_parallel`和`data_parallel`在MindSpore教程中有完整样例： @@ -125,7 +125,7 @@ context.get_auto_parallel_context("parallel_mode") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(all_reduce_fusion_config=[20, 35]) context.get_auto_parallel_context("all_reduce_fusion_config") @@ -133,7 +133,6 @@ context.get_auto_parallel_context("all_reduce_fusion_config") 样例中，`all_reduce_fusion_config`的值为[20, 35]，将前20个AllReduce融合成1个，第20～35个AllReduce融合成1个，剩下的AllReduce融合成1个。 - ### 自动并行配置 #### gradient_fp32_sync @@ -143,7 +142,7 @@ context.get_auto_parallel_context("all_reduce_fusion_config") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(gradient_fp32_sync=False) context.get_auto_parallel_context("gradient_fp32_sync") @@ -156,7 +155,7 @@ MindSpore提供了`dynamic_programming`和`recursive_programming`两种搜索策 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(auto_parallel_search_mode="dynamic_programming") context.get_auto_parallel_context("auto_parallel_search_mode") @@ -169,7 +168,7 @@ context.get_auto_parallel_context("auto_parallel_search_mode") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(strategy_ckpt_load_file="./") context.get_auto_parallel_context("strategy_ckpt_load_file") @@ -182,7 +181,7 @@ context.get_auto_parallel_context("strategy_ckpt_load_file") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(strategy_ckpt_save_file="./") context.get_auto_parallel_context("strategy_ckpt_save_file") @@ -195,7 +194,7 @@ context.get_auto_parallel_context("strategy_ckpt_save_file") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(full_batch=False) context.get_auto_parallel_context("full_batch") @@ -210,7 +209,7 @@ context.get_auto_parallel_context("full_batch") 代码样例如下： ```python -from mindspore import context +from mindspore import context context.set_auto_parallel_context(enable_parallel_optimizer=True) context.get_auto_parallel_context("enable_parallel_optimizer") @@ -323,4 +322,3 @@ allreduce2 = P.AllReduce().add_prim_attr("fusion", 1) 具体用例请参考MindSpore分布式并行训练教程： 。 - diff --git a/docs/programming_guide/source_zh_cn/callback.md b/docs/programming_guide/source_zh_cn/callback.md index 753aa59505143da9cbcb95ec4985cf90bc910b3a..15d1e9a391e00a18c0111030f2fd17457b39f4e4 100644 --- a/docs/programming_guide/source_zh_cn/callback.md +++ b/docs/programming_guide/source_zh_cn/callback.md @@ -12,6 +12,7 @@ ## 概述 + Callback回调函数在MindSpore中被实现为一个类，Callback机制类似于一种监控模式，可以帮助用户观察网络训练过程中各种参数的变化情况和网络内部的状态，还可以根据用户的指定，在达到特定条件后执行相应的操作，在训练过程中，Callback列表会按照定义的顺序执行Callback函数。Callback机制让用户可以及时有效地掌握网络模型的训练状态，并根据需要随时作出调整，可以极大地提升用户的开发效率。 在MindSpore中，Callback机制一般用在网络训练过程`model.train`中，用户可以通过配置不同的内置回调函数传入不同的参数，从而实现各种功能。例如，可以通过`LossMonitor`监控每一个epoch的loss变化情况，通过`ModelCheckpoint`保存网络参数和模型进行再训练或推理，通过`TimeMonitor`监控每一个epoch，每一个step的训练时间，以及提前终止训练，动态调整参数等。 @@ -37,10 +38,11 @@ Callback回调函数在MindSpore中被实现为一个类，Callback机制类似 详细内容，请参考[LossMonitor官网教程](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_debugging_info.html#mindsporecallback)。 - TimeMonitor - + 监控训练过程中每个epoch，每个step的运行时间。 ## MindSpore自定义回调函数 + MindSpore不但有功能强大的内置回调函数，还可以支持用户自定义回调函数。当用户有自己的特殊需求时，可以基于Callback基类，自定义满足用户自身需求的回调函数。Callback可以把训练过程中的重要信息记录下来，通过一个字典类型变量cb_params传递给Callback对象， 用户可以在各个自定义的Callback中获取到相关属性，执行自定义操作。 以下面两个场景为例，介绍自定义Callback回调函数的功能： @@ -51,4 +53,4 @@ MindSpore不但有功能强大的内置回调函数，还可以支持用户自 详细内容，请参考[自定义Callback官网教程](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_debugging_info.html#id3)。 -根据教程，用户可以很容易实现具有其他功能的自定义回调函数，如实现在每一轮训练结束后都输出相应的详细训练信息，包括训练进度、训练轮次、训练名称、loss值等；如实现在loss或模型精度达到一定值后停止训练，用户可以设定loss或模型精度的阈值，当loss或模型精度达到该阈值后就提前终止训练等。 \ No newline at end of file +根据教程，用户可以很容易实现具有其他功能的自定义回调函数，如实现在每一轮训练结束后都输出相应的详细训练信息，包括训练进度、训练轮次、训练名称、loss值等；如实现在loss或模型精度达到一定值后停止训练，用户可以设定loss或模型精度的阈值，当loss或模型精度达到该阈值后就提前终止训练等。 diff --git a/docs/programming_guide/source_zh_cn/cell.md b/docs/programming_guide/source_zh_cn/cell.md index 0f1ac35d43b3eb380737c26debf8705e208fc512..a3409c2e3c14c5454a16ffed94b7a2c34a5089cb 100644 --- a/docs/programming_guide/source_zh_cn/cell.md +++ b/docs/programming_guide/source_zh_cn/cell.md @@ -41,7 +41,7 @@ MindSpore的`Cell`类是构建所有网络的基类，也是网络的基本单 在`construct`方法中，`x`为输入数据，`output`是经过网络结构计算后得到的计算结果。 -``` +```python import mindspore.nn as nn from mindspore.ops import operations as P from mindspore.common.parameter import Parameter @@ -70,7 +70,7 @@ class Net(nn.Cell): 代码样例如下： -``` +```python net = Net() result = net.parameters_dict() print(result.keys()) @@ -80,7 +80,8 @@ print(result['conv.weight']) 样例中的`Net`采用上文构造网络的用例，打印了网络中所有参数的名字和`conv.weight`参数的结果。 输出如下： -``` + +```text odict_keys(['conv.weight']) Parameter (name=conv.weight, value=[[[[-3.95042636e-03 1.08830128e-02 -6.51786150e-03] [ 8.66129529e-03 7.36288540e-03 -4.32638079e-03] @@ -98,7 +99,8 @@ Parameter (name=conv.weight, value=[[[[-3.95042636e-03 1.08830128e-02 -6.517861 其中`nn.Conv2d`是MindSpore以`Cell`为基类封装好的一个卷积层，其具体内容将在“模型层”中进行介绍。 代码样例如下： -``` + +```python import mindspore.nn as nn class Net1(nn.Cell): @@ -120,7 +122,8 @@ print(names) ``` 输出如下： -``` + +```text ('', Net1< (conv): Conv2d >) @@ -136,7 +139,8 @@ print(names) 以`TrainOneStepCell`为例，其接口功能是使网络进行单步训练，需要计算网络反向，因此初始化方法里需要使用`set_grad`。 `TrainOneStepCell`部分代码如下： -``` + +```python class TrainOneStepCell(Cell): def __init__(self, network, optimizer, sens=1.0): super(TrainOneStepCell, self).__init__(auto_prefix=False) @@ -156,7 +160,8 @@ MindSpore的nn模块是Python实现的模型组件，是对低阶API的封装， 同时nn也提供了部分与`Primitive`算子同名的接口，主要作用是对`Primitive`算子进行进一步封装，为用户提供更友好的API。 重新分析上文介绍`construct`方法的用例，此用例是MindSpore的`nn.Conv2d`源码简化内容，内部会调用`P.Conv2D`。`nn.Conv2d`卷积API增加输入参数校验功能并判断是否`bias`等，是一个高级封装的模型层。 -``` + +```python import mindspore.nn as nn from mindspore.ops import operations as P from mindspore.common.parameter import Parameter @@ -259,7 +264,7 @@ MindSpore框架在`mindspore.nn`的layer层内置了丰富的接口，主要内 MindSpore的模型层在`mindspore.nn`下，使用方法如下所示： -``` +```python import mindspore.nn as nn class Net(nn.Cell): @@ -307,7 +312,7 @@ MindSpore的损失函数全部是`Cell`的子类实现，所以也支持用户 - SoftmaxCrossEntropyWithLogits 交叉熵损失函数，用于分类模型。当标签数据不是one-hot编码形式时，需要输入参数`sparse`为True。`reduction`参数默认值为none，其参数含义同`L1Loss`。 - + - CosineEmbeddingLoss `CosineEmbeddingLoss`用于衡量两个输入相似程度，用于分类模型。`margin`默认为0.0，`reduction`参数同`L1Loss`。 @@ -316,7 +321,7 @@ MindSpore的损失函数全部是`Cell`的子类实现，所以也支持用户 MindSpore的损失函数全部在mindspore.nn下，使用方法如下所示： -``` +```python import numpy as np import mindspore.nn as nn from mindspore import Tensor @@ -328,7 +333,8 @@ print(loss(input_data, target_data)) ``` 输出结果： -``` + +```text 1.5 ``` @@ -347,7 +353,8 @@ print(loss(input_data, target_data)) 以LeNet网络为例，在`__init__`方法中定义了卷积层，池化层和全连接层等结构单元，然后在`construct`方法将定义的内容连接在一起，形成一个完整LeNet的网络结构。 LeNet网络实现方式如下所示： -``` + +```python import mindspore.nn as nn class LeNet5(nn.Cell): diff --git a/docs/programming_guide/source_zh_cn/context.md b/docs/programming_guide/source_zh_cn/context.md index 227d73ba5d251d770fa06bcf85c1b2eb6277457c..59a7e39523dff0ec866c5bf7bf735ea7eaff9c26 100644 --- a/docs/programming_guide/source_zh_cn/context.md +++ b/docs/programming_guide/source_zh_cn/context.md @@ -19,9 +19,11 @@ ## 概述 + 初始化网络之前要配置context参数，用于控制程序执行的策略。比如选择执行模式、选择执行后端、配置分布式相关参数等。按照context参数设置实现的不同功能，可以将其分为执行模式管理、硬件管理、分布式管理和维测管理等。 ## 执行模式管理 + MindSpore支持PyNative和Graph这两种运行模式： - `PYNATIVE_MODE`：动态图模式，将神经网络中的各个算子逐一下发执行，方便用户编写和调试神经网络模型。 @@ -29,20 +31,24 @@ MindSpore支持PyNative和Graph这两种运行模式： - `GRAPH_MODE`：静态图模式或者图模式，将神经网络模型编译成一整张图，然后下发执行。该模式利用图优化等技术提高运行性能，同时有助于规模部署和跨平台运行。 ### 模式选择 + 通过设置可以控制程序运行的模式，默认情况下，MindSpore处于PyNative模式。 代码样例如下： + ```python from mindspore import context context.set_context(mode=context.GRAPH_MODE) ``` ### 模式切换 + 实现两种模式之间的切换。 MindSpore处于PYNATIVE模式时，可以通过`context.set_context(mode=context.GRAPH_MODE)`切换为Graph模式；同样地，MindSpore处于Graph模式时，可以通过 `context.set_context(mode=context.PYNATIVE_MODE)`切换为PyNative模式。 代码样例如下： + ```python import numpy as np import mindspore.nn as nn @@ -60,6 +66,7 @@ conv(input_data) 上面的例子先将运行模式设置为`GRAPH_MODE`模式，然后将模式切换为`PYNATIVE_MODE`模式，实现了模式的切换。 ## 硬件管理 + 硬件管理部分主要包括`device_target`和`device_id`两个参数。 - `device_target`： 用于设置目标设备，支持Ascend、GPU和CPU，可以根据实际环境情况设置。 @@ -69,12 +76,14 @@ conv(input_data) > 在GPU和CPU上，设置`device_id`参数无效。 代码样例如下： + ```python from mindspore import context context.set_context(device_target="Ascend", device_id=6) ``` ## 分布式管理 + context中有专门用于配置并行训练参数的接口：context.set_auto_parallel_context，该接口必须在初始化网络之前调用。 - `parallel_mode`：分布式并行模式，默认为单机模式`ParallelMode.STAND_ALONE`。可选数据并行`ParallelMode.DATA_PARALLEL`及自动并行`ParallelMode.AUTO_PARALLEL`。 @@ -86,6 +95,7 @@ context中有专门用于配置并行训练参数的接口：context.set_auto_pa > `device_num`和`global_rank`建议采用默认值，框架内会调用HCCL接口获取。 代码样例如下： + ```python from mindspore import context from mindspore.context import ParallelMode @@ -95,9 +105,11 @@ context.set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, grad > 分布式并行训练详细介绍可以查看[分布式并行训练](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_tutorials.html)。 ## 维测管理 + 为了方便维护和定位问题，context提供了大量维测相关的参数配置，如采集profiling数据、异步数据dump功能和print算子落盘等。 ### 采集profiling数据 + 系统支持在训练过程中采集profiling数据，然后通过profiling工具进行性能分析。当前支持采集的profiling数据包括： - `enable_profiling`：是否开启profiling功能。设置为True，表示开启profiling功能，从enable_options读取profiling的采集选项；设置为False，表示关闭profiling功能，仅采集training_trace。 @@ -105,15 +117,18 @@ context.set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, grad - `enable_options`：profiling采集选项，取值如下，支持采集多项数据。training_trace：采集迭代轨迹数据，即训练任务及AI软件栈的软件信息，实现对训练任务的性能分析，重点关注数据增强、前后向计算、梯度聚合更新等相关数据；task_trace：采集任务轨迹数据，即昇腾910处理器HWTS/AICore的硬件信息，分析任务开始、结束等信息；op_trace：采集单算子性能数据。格式：['op_trace','task_trace','training_trace'] 代码样例如下： + ```python from mindspore import context context.set_context(enable_profiling=True, profiling_options="training_trace") ``` ### 异步数据dump功能 + 在Ascend环境上执行训练，当训练结果和预期有偏差时，可以通过异步数据dump功能保存算子的输入输出进行调试。 代码样例如下： + ```python from mindspore import context context.set_context(save_graphs=True) @@ -122,6 +137,7 @@ context.set_context(save_graphs=True) > 详细的调试方法可以查看[异步数据Dump功能介绍](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_debugging_info.html#dump)。 ### print算子落盘 + 默认情况下，MindSpore的自研print算子可以将用户输入的Tensor或字符串信息打印出来，支持多字符串输入，多Tensor输入和字符串与Tensor的混合输入，输入参数以逗号隔开。 > Print打印功能可以查看[Print算子功能介绍](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_debugging_info.html#print)。 @@ -129,9 +145,10 @@ context.set_context(save_graphs=True) - `print_file_path`：可以将print算子数据保存到文件，同时关闭屏幕打印功能。如果保存的文件已经存在，则会给文件添加时间戳后缀。数据保存到文件可以解决数据量较大时屏幕打印数据丢失的问题。 代码样例如下： + ```python from mindspore import context context.set_context(print_file_path="print.pb") ``` -> context接口详细介绍可以查看[mindspore.context](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.context.html)。 \ No newline at end of file +> context接口详细介绍可以查看[mindspore.context](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.context.html)。 diff --git a/docs/programming_guide/source_zh_cn/dataset_conversion.md b/docs/programming_guide/source_zh_cn/dataset_conversion.md index 872b09859fc8923ce82d784ce09819bf26a9830a..0da981f4199a8fe3c24bdd9359dde0b5ac5c0935 100644 --- a/docs/programming_guide/source_zh_cn/dataset_conversion.md +++ b/docs/programming_guide/source_zh_cn/dataset_conversion.md @@ -61,19 +61,19 @@ for i in range(100): white_io = BytesIO() Image.new('RGB', (i*10, i*10), (255, 255, 255)).save(white_io, 'JPEG') image_bytes = white_io.getvalue() - sample['file_name'] = str(i) + ".jpg" - sample['label'] = i + sample['file_name'] = str(i) + ".jpg" + sample['label'] = i sample['data'] = white_io.getvalue() data.append(sample) - if i % 10 == 0: + if i % 10 == 0: writer.write_raw_data(data) data = [] -if data: +if data: writer.write_raw_data(data) -writer.commit() +writer.commit() data_set = ds.MindDataset(dataset_file=mindrecord_filename) decode_op = vision.Decode() @@ -129,18 +129,18 @@ for i in range(100): "target_eos_mask": np.array([48, 49, 50, 51], dtype=np.int64)} data.append(sample) - if i % 10 == 0: + if i % 10 == 0: writer.write_raw_data(data) data = [] -if data: +if data: writer.write_raw_data(data) writer.commit() data_set = ds.MindDataset(dataset_file=mindrecord_filename) count = 0 -for item in data_set.create_dict_iterator(): +for item in data_set.create_dict_iterator(): print("sample: {}".format(item)) count += 1 print("Got {} samples".format(count)) @@ -167,7 +167,7 @@ MindSpore提供转换常用数据集的工具类，能够将常用的数据集 1. 下载[CIFAR-10数据集](https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz)并解压，其目录结构如下所示。 - ``` + ```text └─cifar-10-batches-py ├─batches.meta ├─data_batch_1 @@ -220,7 +220,7 @@ MindSpore提供转换常用数据集的工具类，能够将常用的数据集 1. 下载[ImageNet数据集](http://image-net.org/download)，将所有图片存放在同一文件夹，用一个映射文件记录图片和标签的对应关系。映射文件包含2列，分别为各类别图片目录和标签ID，用空格隔开，映射文件示例如下： - ``` + ```text n01440760 0 n01443537 1 n01484850 2 @@ -246,7 +246,7 @@ MindSpore提供转换常用数据集的工具类，能够将常用的数据集 imagenet_transformer.transform() ``` - **参数说明：** + **参数说明：** - `IMAGENET_MAP_FILE`：ImageNet数据集标签映射文件的路径。 - `IMAGENET_IMAGE_DIR`：包含ImageNet所有图片的文件夹路径。 - `MINDRECORD_FILE`：输出的MindRecord文件路径。 @@ -277,8 +277,8 @@ import os import mindspore.dataset as ds from mindspore.mindrecord import CsvToMR -CSV_FILE_NAME = "test.csv" -MINDRECORD_FILE_NAME = "test.mindrecord" +CSV_FILE_NAME = "test.csv" +MINDRECORD_FILE_NAME = "test.mindrecord" PARTITION_NUM = 1 def generate_csv(): @@ -330,8 +330,8 @@ import mindspore.dataset.vision.c_transforms as vision from PIL import Image import tensorflow as tf -TFRECORD_FILE_NAME = "test.tfrecord" -MINDRECORD_FILE_NAME = "test.mindrecord" +TFRECORD_FILE_NAME = "test.tfrecord" +MINDRECORD_FILE_NAME = "test.mindrecord" PARTITION_NUM = 1 def generate_tfrecord(): @@ -339,7 +339,7 @@ def generate_tfrecord(): if isinstance(values, list): feature = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values))) else: - feature = tf.train.Feature(int64_list=tf.train.Int64List(value=[values])) + feature = tf.train.Feature(int64_list=tf.train.Int64List(value=[values])) return feature def create_float_feature(values): @@ -352,9 +352,9 @@ def generate_tfrecord(): def create_bytes_feature(values): if isinstance(values, bytes): white_io = BytesIO() - Image.new('RGB', (10, 10), (255, 255, 255)).save(white_io, 'JPEG') + Image.new('RGB', (10, 10), (255, 255, 255)).save(white_io, 'JPEG') image_bytes = white_io.getvalue() - feature = tf.train.Feature(bytes_list=tf.train.BytesList(value=[image_bytes])) + feature = tf.train.Feature(bytes_list=tf.train.BytesList(value=[image_bytes])) else: feature = tf.train.Feature(bytes_list=tf.train.BytesList(value=[bytes(values, encoding='utf-8')])) return feature diff --git a/docs/programming_guide/source_zh_cn/dataset_loading.md b/docs/programming_guide/source_zh_cn/dataset_loading.md index 3d2e08eb1df81fbc69eb79fdb8a943a99108212b..292f5c35334342842e39f116ae383f251773285d 100644 --- a/docs/programming_guide/source_zh_cn/dataset_loading.md +++ b/docs/programming_guide/source_zh_cn/dataset_loading.md @@ -76,7 +76,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text Image shape: (32, 32, 3) , Label: 0 Image shape: (32, 32, 3) , Label: 1 Image shape: (32, 32, 3) , Label: 2 @@ -110,7 +110,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text [Segmentation]: image shape: (281, 500, 3) target shape: (281, 500, 3) @@ -152,7 +152,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text Detection: dict_keys(['bbox', 'image', 'iscrowd', 'category_id']) Stuff: dict_keys(['segmentation', 'iscrowd', 'image']) Keypoint: dict_keys(['keypoints', 'num_keypoints', 'image']) @@ -218,7 +218,7 @@ TFRecord是TensorFlow定义的一种二进制数据文件格式。 将数据集格式和特征按JSON格式写入Schema文件，示例如下： - ``` + ```json { "columns": { "image": { @@ -281,7 +281,7 @@ TFRecord是TensorFlow定义的一种二进制数据文件格式。 输出结果如下： - ``` + ```text [0.49893939 0.36348882] [0.15234002] [0.83845534 0.19721032] [0.94602561] [0.2361873 0.79506755] [0.88118559] @@ -305,7 +305,7 @@ TFRecord是TensorFlow定义的一种二进制数据文件格式。 输出结果如下： - ``` + ```text [1 2] [3 4] ``` @@ -325,7 +325,7 @@ TFRecord是TensorFlow定义的一种二进制数据文件格式。 输出结果如下： - ``` + ```text {'col1': Tensor(shape=[], dtype=Int64, value= 1), 'col2': Tensor(shape=[], dtype=Int64, value= 3)} {'col1': Tensor(shape=[], dtype=Int64, value= 2), 'col2': Tensor(shape=[], dtype=Int64, value= 4)} ``` @@ -374,7 +374,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text [0.36510558 0.45120592] [0.78888122] [0.49606035 0.07562207] [0.38068183] [0.57176158 0.28963401] [0.16271622] @@ -420,7 +420,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text [0.36510558 0.45120592] [0.78888122] [0.49606035 0.07562207] [0.38068183] [0.57176158 0.28963401] [0.16271622] @@ -457,7 +457,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text [0.36510558 0.45120592] [0.78888122] [0.49606035 0.07562207] [0.38068183] [0.57176158 0.28963401] [0.16271622] @@ -497,7 +497,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text [0.36510558 0.45120592] [0.78888122] [0.57176158 0.28963401] [0.16271622] [0.81585667 0.96883469] [0.77994068] diff --git a/docs/programming_guide/source_zh_cn/dtype.md b/docs/programming_guide/source_zh_cn/dtype.md index 5fc7d208e65aed905222fe79b2ca6530a04f5fdc..177cf1d89cc47884ad8f45d579749465c575fda0 100644 --- a/docs/programming_guide/source_zh_cn/dtype.md +++ b/docs/programming_guide/source_zh_cn/dtype.md @@ -9,6 +9,8 @@ +   + ## 概述 @@ -19,7 +21,8 @@ MindSpore张量支持不同的数据类型，包含`int8`、`int16`、`int32`、 详细的类型支持情况请参考。 以下代码，打印MindSpore的数据类型int32。 -``` + +```python from mindspore import dtype as mstype data_type = mstype.int32 @@ -28,11 +31,10 @@ print(data_type) 输出如下： -``` +```text Int32 ``` - ## 数据类型转换接口 MindSpore提供了以下几个接口，实现与NumPy数据类型和Python内置的数据类型间的转换。 @@ -43,7 +45,7 @@ MindSpore提供了以下几个接口，实现与NumPy数据类型和Python内置 以下代码实现了不同数据类型间的转换，并打印转换后的类型。 -``` +```python from mindspore import dtype as mstype np_type = mstype.dtype_to_nptype(mstype.int32) @@ -57,7 +59,7 @@ print(py_type) 输出如下： -``` +```text Int64 diff --git a/docs/programming_guide/source_zh_cn/network_component.md b/docs/programming_guide/source_zh_cn/network_component.md index 78376e493732c6c4e7f184da441cf8e23384ecc8..b66b27f9038e111ad72eca4ffd48d487dbf473f0 100644 --- a/docs/programming_guide/source_zh_cn/network_component.md +++ b/docs/programming_guide/source_zh_cn/network_component.md @@ -26,7 +26,7 @@ GradOperation组件用于生成输入函数的梯度，利用`get_all`、`get_by GradOperation的使用实例如下： -``` +```python import numpy as np import mindspore.nn as nn @@ -64,7 +64,7 @@ GradNetWrtX(Net())(x, y) 输出如下： -``` +```text Tensor(shape=[2, 3], dtype=Float32, [[1.4100001 1.5999999 6.6 ] [1.4100001 1.5999999 6.6 ]]) @@ -79,7 +79,7 @@ MindSpore涉及梯度计算的其他组件，例如`WithGradCell`和`TrainOneSte 下面通过一个实例来介绍其具体的使用， 首先需要构造一个网络，内容如下： -``` +```python import numpy as np import pytest @@ -124,7 +124,8 @@ class LeNet(nn.Cell): ``` 下面是`WithLossCell`的使用实例，分别定义好网络和损失函数，然后创建一个`WithLossCell`，传入输入数据和标签数据，`WithLossCell`内部根据网络和损失函数返回计算结果。 -``` + +```python data = Tensor(np.ones([32, 1, 32, 32]).astype(np.float32) * 0.01) label = Tensor(np.ones([32]).astype(np.int32)) net = LeNet() @@ -136,7 +137,8 @@ print(loss) ``` 输出如下： -``` + +```text +++++++++Loss+++++++++++++ 2.302585 ``` @@ -147,7 +149,7 @@ print(loss) 下面构造一个使用`TrainOneStepCell`接口进行网络训练的实例，其中`LeNet`和包名的导入代码和上个用例共用。 -``` +```python data = Tensor(np.ones([32, 1, 32, 32]).astype(np.float32) * 0.01) label = Tensor(np.ones([32]).astype(np.int32)) net = LeNet() @@ -168,7 +170,8 @@ for i in range(5): 用例中构造了优化器和一个`WithLossCell`的实例，然后传入`TrainOneStepCell`中初始化一个训练网络，用例循环五次，相当于网络训练了五次，并输出每次的loss结果，由结果可以看出每次训练后loss值在逐渐减小。 输出如下： -``` + +```text +++++++++result:0++++++++++++ 2.302585 +++++++++result:1++++++++++++ @@ -180,5 +183,6 @@ for i in range(5): +++++++++result:4++++++++++++ 2.2215357 ``` + 后续内容会介绍MindSpore使用更加高级封装的接口，即`Model`类中的`train`方法训练模型，在其内部实现中会用到 `TrainOneStepCell`和`WithLossCell` 等许多网络组件，感兴趣的读者可以查看其内部实现。 diff --git a/docs/programming_guide/source_zh_cn/operator.md b/docs/programming_guide/source_zh_cn/operator.md index 588d3a4314412f544695fe619cb26c9116bc60f1..aa0afe9f566c735324f2a89e47da2fb08e00f6a8 100644 --- a/docs/programming_guide/source_zh_cn/operator.md +++ b/docs/programming_guide/source_zh_cn/operator.md @@ -16,7 +16,6 @@ - [求三角函数](#求三角函数) - [向量运算](#向量运算) - [Squeeze](#squeeze) - - [求Sparse2Dense](#求sparse2dense) - [矩阵运算](#矩阵运算) - [矩阵乘法](#矩阵乘法) - [广播机制](#广播机制) @@ -43,10 +42,13 @@ ## 概述 + MindSpore的算子组件，可从算子使用方式和算子功能两种维度进行划分。 ## 算子使用方式 + 算子相关接口主要包括operations、functional和composite，可通过ops直接获取到这三类算子。 + - operations提供单个的Primtive算子。一个算子对应一个原语，是最小的执行对象，需要实例化之后使用。 - composite提供一些预定义的组合算子，以及复杂的涉及图变换的算子，如`GradOperation`。 - functional提供operations和composite实例化后的对象，简化算子的调用流程。 @@ -60,6 +62,7 @@ Primitive算子也称为算子原语，它直接封装了底层的Ascend、GPU Primitive算子接口是构建高阶接口、自动微分、网络模型等能力的基础。 代码样例如下： + ```python import numpy as np import mindspore @@ -74,7 +77,8 @@ print("output =", output) ``` 输出如下： -``` + +```text output = [ 1. 8. 64.] ``` @@ -99,7 +103,8 @@ print("output =", output) ``` 输出如下： -``` + +```text output = [ 1. 8. 64.] ``` @@ -108,6 +113,7 @@ output = [ 1. 8. 64.] composite提供了一些算子的组合，包括clip_by_value和random相关的一些算子，以及涉及图变换的函数（`GradOperation`、`HyperMap`和`Map`等）。 算子的组合可以直接像一般函数一样使用，例如使用`normal`生成一个随机分布： + ```python from mindspore.common import dtype as mstype from mindspore.ops import composite as C @@ -118,8 +124,10 @@ stddev = Tensor(1.0, mstype.float32) output = C.normal((2, 3), mean, stddev, seed=5) print("ouput =", output) ``` + 输出如下： -``` + +```text output = [[2.4911082 0.7941146 1.3117087] [0.30582333 1.772938 1.525996]] ``` @@ -129,6 +137,7 @@ output = [[2.4911082 0.7941146 1.3117087] 针对涉及图变换的函数，用户可以使用`MultitypeFuncGraph`定义一组重载的函数，根据不同类型，采用不同实现。 代码样例如下： + ```python import numpy as np from mindspore.ops.composite import MultitypeFuncGraph @@ -149,8 +158,10 @@ tensor2 = Tensor(np.array([[1.2, 2.1], [2.2, 3.2]]).astype('float32')) print('tensor', add(tensor1, tensor2)) print('scalar', add(1, 2)) ``` + 输出如下： -``` + +```text tensor [[2.4, 4.2] [4.4, 6.4]] scalar 3 @@ -183,36 +194,41 @@ scalar 3 有些标量运算符对常用的数学运算符进行了重载。并且支持类似NumPy的广播特性。 以下代码实现了对input_x作乘方数为input_y的乘方操作： + ```python -import numpy as np +import numpy as np import mindspore from mindspore import Tensor -import mindspore.ops.operations as P + input_x = mindspore.Tensor(np.array([1.0, 2.0, 4.0]), mindspore.float32) input_y = 3.0 print(input_x**input_y) ``` 输出如下： -``` + +```text [ 1. 8. 64.] ``` #### 加法 上述代码中`input_x`和`input_y`的相加实现方式如下： + ```python print(input_x + input_y) ``` 输出如下： -``` + +```text [4.0 5.0 7.0] ``` #### Element-wise乘法 以下代码实现了Element-wise乘法示例： + ```python import numpy as np import mindspore @@ -228,13 +244,15 @@ print(res) ``` 输出如下： -``` + +```text [4. 10. 18] ``` #### 求三角函数 以下代码实现了Acos： + ```python import numpy as np import mindspore @@ -248,9 +266,11 @@ print(output) ``` 输出如下： -``` + +```text [0.7377037, 1.5307858, 1.2661037，0.97641146] ``` + ### 向量运算 向量运算符只在一个特定轴上运算，将一个向量映射到一个标量或者另外一个向量。 @@ -258,6 +278,7 @@ print(output) #### Squeeze 以下代码实现了压缩第3个通道维度为1的通道： + ```python import numpy as np import mindspore @@ -272,34 +293,12 @@ print(output) ``` 输出如下： -``` + +```text [[1. 1.] [1. 1.] [1. 1.]] ``` -#### 求Sparse2Dense - -以下代码实现了对Sparse2Dense示例： -```python -import numpy as np -import mindspore as ms -from mindspore import Tensor -import mindspore.ops.operations as P - -indices = Tensor([[0, 1], [1, 2]]) -values = Tensor([1, 2], dtype=ms.float32) -dense_shape = (3, 4) -out = P.SparseToDense()(indices, values, dense_shape) - -print(out) -``` - -输出如下： -``` -[[0, 1, 0, 0], - [0, 0, 2, 0], - [0, 0, 0, 0]] -``` ### 矩阵运算 @@ -308,6 +307,7 @@ print(out) #### 矩阵乘法 以下代码实现了input_x 和 input_y的矩阵乘法： + ```python import numpy as np import mindspore @@ -323,7 +323,8 @@ print(output) ``` 输出如下： -``` + +```text [[3. 3. 3. 3.]] ``` @@ -332,6 +333,7 @@ print(output) 广播表示输入各变量channel数目不一致时，改变他们的channel数以得到结果。 - 以下代码实现了广播机制的示例： + ```python from mindspore import Tensor from mindspore.communication import init @@ -363,6 +365,7 @@ output = net(input_) 卷积操作 以下代码实现了常见卷积操作之一的2D convolution 操作： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -376,8 +379,10 @@ res = conv2d(input, weight) print(res) ``` + 输出如下： -``` + +```text [[[[288. 288. 288. ... 288. 288. 288.] [288. 288. 288. ... 288. 288. 288.] [288. 288. 288. ... 288. 288. 288.] @@ -411,8 +416,10 @@ res = conv2d_backprop_input(dout, weight, F.shape(x)) print(res) ``` + 输出如下： -``` + +```text [[[[ 32. 64. 96. ... 96. 64. 32.] [ 64. 128. 192. ... 192. 128. 64.] [ 96. 192. 288. ... 288. 192. 96.] @@ -433,6 +440,7 @@ print(res) #### 激活函数 以下代码实现Softmax激活函数计算： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -447,15 +455,15 @@ print(res) ``` 输出如下： -``` + +```text [0.01165623 0.03168492 0.08612854 0.23412167 0.6364086] ``` #### LossFunction - L1Loss - 以下代码实现了L1 loss function： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -470,15 +478,15 @@ print(res) ``` 输出如下： -``` + +```text [0. 0. 0.5] ``` #### 优化算法 - SGD - 以下代码实现了SGD梯度下降算法的具体实现，输出是result： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -498,7 +506,8 @@ print(result) ``` 输出如下： -``` + +```text [0. 0. 0. 0.] ``` @@ -525,7 +534,8 @@ print(typea) ``` 输出如下： -``` + +```text Float32 ``` @@ -550,7 +560,8 @@ print(type(result)) ``` 输出结果: -``` + +```text ``` @@ -559,6 +570,7 @@ print(type(result)) 返回输入数据的形状。 以下代码实现了返回输入数据input_tensor的操作： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -572,7 +584,8 @@ print(output) ``` 输出如下： -``` + +```text [3, 2, 1] ``` @@ -581,6 +594,7 @@ print(output) 图像操作包括图像预处理操作，如图像剪切（Crop，便于得到大量训练样本）和大小变化（Reise，用于构建图像金子塔等）。 以下代码实现了Crop和Resize操作： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -613,7 +627,8 @@ print(output.asnumpy()) ``` 输出如下: -``` + +```text [[[[ 6.51672244e-01 -1.85958534e-01 5.19907832e-01] [ 1.53466597e-01 4.10562098e-01 6.26138210e-01] [ 6.62892580e-01 3.81776541e-01 4.69261825e-01] @@ -645,6 +660,7 @@ print(output.asnumpy()) 对物体所在区域方框进行编码，得到类似PCA的更精简信息，以便做后续类似特征提取，物体检测，图像恢复等任务。 以下代码实现了对anchor_box和groundtruth_box的boundingbox encode： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -659,7 +675,8 @@ print(res) ``` 输出如下: -``` + +```text [[5.0000000e-01 5.0000000e-01 -6.5504000e+04 6.9335938e-01] [-1.0000000e+00 2.5000000e-01 0.0000000e+00 4.0551758e-01]] ``` @@ -669,6 +686,7 @@ print(res) 编码器对区域位置信息解码之后，用此算子进行解码。 以下代码实现了： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -683,7 +701,8 @@ print(res) ``` 输出如下： -``` + +```text [[4.1953125 0. 0. 5.1953125] [2.140625 0. 3.859375 60.59375]] ``` @@ -693,6 +712,7 @@ print(res) 计算预测的物体所在方框和真实物体所在方框的交集区域与并集区域的占比大小，常作为一种损失函数，用以优化模型。 以下代码实现了计算两个变量anchor_boxes和gt_boxes之间的IOU，以out输出： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -707,7 +727,8 @@ print(out) ``` 输出如下： -``` + +```text [[0. -0. 0.] [0. -0. 0.] [0. 0. 0.]] @@ -722,6 +743,7 @@ print(out) 输出Tensor变量的数值，方便用户随时随地打印想了解或者debug必需的某变量数值。 以下代码实现了输出x这一变量的值： + ```python from mindspore import nn @@ -741,6 +763,7 @@ class DebugNN(nn.Cell): 打印中间变量的梯度，是比较常用的算子，目前仅支持Pynative模式。 以下代码实现了打印中间变量(例中x,y)的梯度： + ```python from mindspore import Tensor import mindspore.ops.operations as P @@ -765,7 +788,9 @@ def backward(x, y): backward(1, 2) ``` + 输出如下： -``` + +```text (Tensor(shape=[], dtype=Float32, value=2),) ``` diff --git a/docs/programming_guide/source_zh_cn/optim.md b/docs/programming_guide/source_zh_cn/optim.md index da9807d72eb72ed5bee84f288daa4040897474c7..dc96b82e7be8de0613344c55081df6e34486676d 100644 --- a/docs/programming_guide/source_zh_cn/optim.md +++ b/docs/programming_guide/source_zh_cn/optim.md @@ -24,6 +24,7 @@ > 本文档中的所有示例，支持CPU，GPU，Ascend环境。 ## 学习率 + ### dynamic_lr `mindspore.nn.dynamic_lr`模块有以下几个类： @@ -40,7 +41,7 @@ 例如`piecewise_constant_lr`类代码样例如下： -``` +```python from mindspore.nn.dynamic_lr import piecewise_constant_lr def test_dynamic_lr(): @@ -55,7 +56,8 @@ if __name__ == '__main__': ``` 返回结果如下： -``` + +```text [0.1, 0.1, 0.05, 0.05, 0.05, 0.01, 0.01, 0.01, 0.01, 0.01] ``` @@ -73,7 +75,8 @@ if __name__ == '__main__': 它们是属于`learning_rate_schedule`的不同实现方式。 例如ExponentialDecayLR类代码样例如下： -``` + +```python from mindspore.common import dtype as mstype from mindspore import Tensor from mindspore.nn.learning_rate_schedule import ExponentialDecayLR @@ -93,13 +96,15 @@ if __name__ == '__main__': ``` 返回结果如下： -``` + +```text 0.094868325 ``` - ## Optimzer + ### 如何使用 + 为了使用`mindspore.nn.optim`，我们需要构建一个`Optimizer`对象。这个对象能够保持当前参数状态并基于计算得到的梯度进行参数更新。 - 构建 @@ -108,7 +113,7 @@ if __name__ == '__main__': 代码样例如下： -``` +```python from mindspore import nn optim = nn.SGD(group_params, learning_rate=0.1, weight_decay=0.0) @@ -125,7 +130,7 @@ optim = nn.Adam(group_params, learning_rate=0.1, weight_decay=0.0) 我们仍然能够传递选项作为关键字参数，在未重写这些选项的组中，它们会被用作默认值。当你只想改动一个参数组的选项，但其他参数组的选项不变时，这是非常有用的。 例如，当我们想制定每一层的学习率时，以`SGD`为例： -``` +```python from mindspore import nn optim = nn.SGD([{'params': conv_params, 'weight_decay': 0.01}, @@ -134,6 +139,7 @@ optim = nn.SGD([{'params': conv_params, 'weight_decay': 0.01}, learning_rate=0.1, weight_decay=0.0) ``` + 这段示例意味着当参数是conv_params时候，权重衰减使用的是0.01，学习率使用的是0.1；而参数是no_conv_params时候，权重衰减使用的是0.0，学习率使用的是0.01。这个学习率learning_rate=0.1会被用于所有分组里没有设置学习率的参数，权重衰减weight_deca也是如此。 ### 内置优化器 @@ -149,7 +155,7 @@ optim = nn.SGD([{'params': conv_params, 'weight_decay': 0.01}, 例如`SGD`的代码样例如下： -``` +```python from mindspore import nn from mindspore.train import Model from .optimizer import Optimizer @@ -183,4 +189,4 @@ optim = nn.SGD(group_params, learning_rate=0.1, weight_decay=0.0) loss = nn.SoftmaxCrossEntropyWithLogits() model = Model(net, loss_fn=loss, optimizer=optim) -``` \ No newline at end of file +``` diff --git a/docs/programming_guide/source_zh_cn/parameter.md b/docs/programming_guide/source_zh_cn/parameter.md index d42c81a97dabc58c07ec9f0c1bfcbb01bd029d7c..d179d6c1619d71511a0bda3a87e5ff4206df5857 100644 --- a/docs/programming_guide/source_zh_cn/parameter.md +++ b/docs/programming_guide/source_zh_cn/parameter.md @@ -18,9 +18,11 @@ `Parameter`是变量张量，代表在训练网络时，需要被更新的参数。本章主要介绍了`Parameter`的初始化以及属性和方法的使用，同时介绍了`ParameterTuple`。 ## 初始化 -``` + +```python mindspore.Parameter(default_input, name, requires_grad=True, layerwise_parallel=False) ``` + 初始化一个`Parameter`对象，传入的数据支持`Tensor`、`Initializer`、`int`和`float`四种类型。 `Initializer`是初始化器，保存了shape和dtype信息，提供`to_tensor`方法生成存有数据的`Tensor`，可调用`initializer`接口生成`Initializer`对象。 @@ -38,7 +40,8 @@ mindspore.Parameter(default_input, name, requires_grad=True, layerwise_parallel= 有关分布式并行的相关配置，可以参考文档：。 下例通过三种不同的数据类型构造了`Parameter`，三个`Parameter`都需要更新，都不采用layerwise并行。如下： -``` + +```python import numpy as np from mindspore import Tensor, Parameter from mindspore.common import dtype as mstype @@ -53,12 +56,12 @@ print(x, "\n\n", y, "\n\n", z) 输出如下： -``` +```text Parameter (name=x, value=[[0 1 2] - [3 4 5]]) + [3 4 5]]) Parameter (name=y, value=[[[1. 1. 1.] - [1. 1. 1.]]]) + [1. 1. 1.]]]) Parameter (name=z, value=2.0) ``` @@ -84,7 +87,7 @@ Parameter (name=z, value=2.0) 下例通过`Tensor`初始化一个`Parameter`，获取了`Parameter`的相关属性。如下： -``` +```python import numpy as np from mindspore import Tensor, Parameter @@ -102,7 +105,7 @@ print("name: ", x.name, "\n", 输出如下： -``` +```text name: x sliced: False is_init: False @@ -111,10 +114,11 @@ requires_grad: True layerwise_parallel: False data: Parameter (name=x, value=[[0 1 2] - [3 4 5]]) + [3 4 5]]) ``` ## 方法 + - `init_data`：在网络采用半自动或者全自动并行策略的场景下， 当初始化`Parameter`传入的数据是`Initializer`时，可调用该接口将`Parameter`保存的数据转换为`Tensor`。 @@ -127,7 +131,7 @@ data: Parameter (name=x, value=[[0 1 2] 下例通过`Initializer`来初始化`Tensor`，调用了`Parameter`的相关方法。如下： -``` +```python import numpy as np from mindspore import Tensor, Parameter @@ -145,7 +149,7 @@ print(x.set_data(default_input=Tensor(np.arange(2*3).reshape((1, 2, 3))))) 输出如下： -``` +```text Parameter (name=x, value=[[[1. 1. 1.] [1. 1. 1.]]]) Parameter (name=x_c.x, value=[[[1. 1. 1.] @@ -158,11 +162,12 @@ Parameter (name=x, value=[[[0. 1. 2.] ``` ## ParameterTuple + 继承于`tuple`，用于保存多个`Parameter`，通过`__new__(cls, iterable)`传入一个存放`Parameter`的迭代器进行构造，提供`clone`接口进行克隆。 下例构造了一个`ParameterTuple`对象，并进行了克隆。如下： -``` +```python import numpy as np from mindspore import Tensor, Parameter, ParameterTuple from mindspore.common import dtype as mstype @@ -179,12 +184,12 @@ print(params_copy) 输出如下： -``` +```text (Parameter (name=x, value=Tensor(shape=[2, 3], dtype=Int64, [[ 0, 1, 2], [ 3, 4, 5]])), Parameter (name=y, value=Tensor(shape=[1, 2, 3], dtype=Float32, [[[ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00], - [ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00]]])), Parameter (name=z, value=Tensor(shape=[], dtype=Float32, 2))) + [ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00]]])), Parameter (name=z, value=Tensor(shape=[], dtype=Float32, 2))) (Parameter (name=params_copy.x, value=Tensor(shape=[2, 3], dtype=Int64, [[ 0, 1, 2], diff --git a/docs/programming_guide/source_zh_cn/pipeline.md b/docs/programming_guide/source_zh_cn/pipeline.md index ba0e282794170c3c3b70fa0c6aa9f896f1920ab4..7b16631c79e9f80bfe1950ea4f57bf75c1b230eb 100644 --- a/docs/programming_guide/source_zh_cn/pipeline.md +++ b/docs/programming_guide/source_zh_cn/pipeline.md @@ -65,7 +65,7 @@ for data in dataset1.create_dict_iterator(): 输出结果如下： -``` +```text {'data': Tensor(shape=[3], dtype=Int64, value=[0, 1, 2])} {'data': Tensor(shape=[3], dtype=Int64, value=[2, 3, 4])} {'data': Tensor(shape=[3], dtype=Int64, value=[3, 4, 5])} @@ -109,7 +109,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text {'data': Tensor(shape=[3], dtype=Int64, value=[0, 1, 2])} {'data': Tensor(shape=[3], dtype=Int64, value=[1, 2, 3])} {'data': Tensor(shape=[3], dtype=Int64, value=[2, 3, 4])} @@ -156,7 +156,7 @@ for data in dataset2.create_dict_iterator(): 输出结果如下： -``` +```text {'data': Tensor(shape=[2, 3], dtype=Int64, value=[[0, 1, 2], [1, 2, 3]])} {'data': Tensor(shape=[2, 3], dtype=Int64, value=[[2, 3, 4], [3, 4, 5]])} {'data': Tensor(shape=[1, 3], dtype=Int64, value=[[4, 5, 6]])} @@ -192,7 +192,7 @@ for data in dataset1.create_dict_iterator(): 输出结果如下： -``` +```text {'data': Tensor(shape=[3], dtype=Int64, value=[0, 1, 2])} {'data': Tensor(shape=[3], dtype=Int64, value=[1, 2, 3])} {'data': Tensor(shape=[3], dtype=Int64, value=[2, 3, 4])} @@ -239,7 +239,7 @@ for data in dataset3.create_dict_iterator(): 输出结果如下： -``` +```text {'data1': Tensor(shape=[3], dtype=Int64, value= [0, 1, 2]), 'data2': Tensor(shape=[2], dtype=Int64, value= [1, 2])} {'data1': Tensor(shape=[3], dtype=Int64, value= [1, 2, 3]), 'data2': Tensor(shape=[2], dtype=Int64, value= [1, 2])} {'data1': Tensor(shape=[3], dtype=Int64, value= [2, 3, 4]), 'data2': Tensor(shape=[2], dtype=Int64, value= [1, 2])} @@ -279,7 +279,7 @@ for data in dataset3.create_dict_iterator(): 输出结果如下： -``` +```text {'data1': Tensor(shape=[3], dtype=Int64, value= [0, 0, 0])} {'data1': Tensor(shape=[3], dtype=Int64, value= [0, 0, 0])} {'data1': Tensor(shape=[3], dtype=Int64, value= [1, 2, 3])} diff --git a/docs/programming_guide/source_zh_cn/probability.md b/docs/programming_guide/source_zh_cn/probability.md index 3b1073b4cb974024e0f9aec87f91c3f8f86e3b8a..8cbea642405bdaea280a947253e83e33e4f102bd 100644 --- a/docs/programming_guide/source_zh_cn/probability.md +++ b/docs/programming_guide/source_zh_cn/probability.md @@ -82,6 +82,7 @@ MindSpore深度概率编程的目标是将深度学习和贝叶斯学习结合 伯努利分布，继承自 `Distribution` 类。 属性: + - `Bernoulli.probs`：伯努利试验成功的概率。 `Distribution` 基类调用 `Bernoulli` 中私有接口以实现基类中的公有接口。`Bernoulli` 支持的公有接口为： @@ -97,6 +98,7 @@ MindSpore深度概率编程的目标是将深度学习和贝叶斯学习结合 指数分布，继承自 `Distribution` 类。 属性: + - `Exponential.rate`：率参数。 `Distribution` 基类调用 `Exponential` 私有接口以实现基类中的公有接口。`Exponential` 支持的公有接口为： @@ -112,6 +114,7 @@ MindSpore深度概率编程的目标是将深度学习和贝叶斯学习结合 几何分布，继承自 `Distribution` 类。 属性: + - `Geometric.probs`：伯努利试验成功的概率。 `Distribution` 基类调用 `Geometric` 中私有接口以实现基类中的公有接口。`Geometric` 支持的公有接口为： @@ -127,6 +130,7 @@ MindSpore深度概率编程的目标是将深度学习和贝叶斯学习结合 正态（高斯）分布，继承自 `Distribution` 类。 `Distribution` 基类调用 `Normal` 中私有接口以实现基类中的公有接口。`Normal` 支持的公有接口为： + - `mean`，`mode`，`var`：可选择传入分布的参数均值 *mean* 和标准差 *sd* 。 - `entropy`：可选择传入分布的参数均值 *mean* 和标准差 *sd* 。 - `cross_entropy`，`kl_loss`：必须传入 *dist* ，*mean_b* 和 *sd_b* 。*dist* 为另一分布的类型的名称，目前只支持此处为 *‘Normal’* 。*mean_b* 和 *sd_b* 为分布 *b* 的均值和标准差。可选择传入分布的参数 *a* 均值 *mean_a* 和标准差 *sd_a* 。 @@ -138,6 +142,7 @@ MindSpore深度概率编程的目标是将深度学习和贝叶斯学习结合 均匀分布，继承自 `Distribution` 类。 属性: + - `Uniform.low`：最小值。 - `Uniform.high`：最大值。 @@ -162,65 +167,91 @@ import mindspore.context as context import mindspore.nn.probability.distribution as msd context.set_context(mode=context.PYNATIVE_MODE) ``` + 以 `Normal` 为例， 创建一个均值为0.0、标准差为1.0的正态分布： + ```python my_normal = msd.Normal(0.0, 1.0, dtype=mstype.float32) ``` + 计算均值： + ```python mean = my_normal.mean() print(mean) ``` + 输出为： -```python + +```text 0.0 ``` + 计算方差： + ```python var = my_normal.var() print(var) ``` + 输出为： -```python + +```text 1.0 ``` + 计算熵： + ```python entropy = my_normal.entropy() print(entropy) ``` + 输出为： -```python + +```text 1.4189385 ``` + 计算概率密度函数： + ```python value = Tensor([-0.5, 0.0, 0.5], dtype=mstype.float32) prob = my_normal.prob(value) print(prob) ``` + 输出为： -```python + +```text [0.35206532, 0.3989423, 0.35206532] ``` + 计算累积分布函数： + ```python cdf = my_normal.cdf(value) print(cdf) ``` + 输出为： -```python + +```text [0.30852754, 0.5, 0.69146246] ``` + 计算 Kullback-Leibler 散度： + ```python mean_b = Tensor(1.0, dtype=mstype.float32) sd_b = Tensor(2.0, dtype=mstype.float32) kl = my_normal.kl_loss('Normal', mean_b, sd_b) print(kl) ``` + 输出为： -```python + +```text 0.44314718 ``` @@ -229,6 +260,7 @@ print(kl) 在图模式下，`Distribution` 子类可用在网络中。 导入相关模块： + ```python import mindspore.nn as nn from mindspore import Tensor @@ -237,20 +269,24 @@ import mindspore.context as context import mindspore.nn.probability.distribution as msd context.set_context(mode=context.GRAPH_MODE) ``` + 创建网络： + ```python # 网络继承nn.Cell class Net(nn.Cell): def __init__(self): super(Net, self).__init__() self.normal = msd.Normal(0.0, 1.0, dtype=mstype.float32) - + def construct(self, value, mean, sd): pdf = self.normal.prob(value) kl = self.normal.kl_loss("Normal", mean, sd) return pdf, kl ``` + 调用网络： + ```python net = Net() value = Tensor([-0.5, 0.0, 0.5], dtype=mstype.float32) @@ -260,8 +296,10 @@ pdf, kl = net(value, mean, sd) print("pdf: ", pdf) print("kl: ", kl) ``` + 输出为： -```python + +```text pdf: [0.3520653, 0.39894226, 0.3520653] kl: 0.5 ``` @@ -293,6 +331,7 @@ kl: 0.5 在执行之前，我们需要导入需要的库文件包。 导入相关模块： + ```python import numpy as np import mindspore.nn as nn @@ -305,6 +344,7 @@ context.set_context(mode=context.PYNATIVE_MODE) ``` 构造一个 `TransformedDistribution` 实例，使用 `Normal` 分布作为需要变换的分布类，使用 `Exp` 作为映射变换，可以生成 `LogNormal` 分布。 + ```python normal = msd.Normal(0.0, 1.0, dtype=dtype.float32) exp = msb.Exp() @@ -313,7 +353,8 @@ print(LogNormal) ``` 输出为： -```python + +```text TransformedDistribution< (_bijector): Exp (_distribution): Normal @@ -323,6 +364,7 @@ TransformedDistribution< 可以对 `LogNormal` 进行概率分布计算。例如： 计算累积分布函数： + ```python x = np.array([2.0, 5.0, 10.0], dtype=np.float32) tx = Tensor(x, dtype=dtype.float32) @@ -331,11 +373,13 @@ print(cdf) ``` 输出为： -```python + +```text [7.55891383e-01, 9.46239710e-01, 9.89348888e-01] ``` 计算对数累积分布函数： + ```python x = np.array([2.0, 5.0, 10.0], dtype=np.float32) tx = Tensor(x, dtype=dtype.float32) @@ -344,11 +388,13 @@ print(log_cdf) ``` 输出为： -```python + +```text [-2.79857576e-01, -5.52593507e-02, -1.07082408e-02] ``` 计算生存函数： + ```python x = np.array([2.0, 5.0, 10.0], dtype=np.float32) tx = Tensor(x, dtype=dtype.float32) @@ -357,11 +403,13 @@ print(survival_function) ``` 输出为： -```python + +```text [2.44108617e-01, 5.37602901e-02, 1.06511116e-02] ``` 计算对数生存函数： + ```python x = np.array([2.0, 5.0, 10.0], dtype=np.float32) tx = Tensor(x, dtype=dtype.float32) @@ -370,11 +418,13 @@ print(log_survival) ``` 输出为： -```python + +```text [-1.41014194e+00, -2.92322016e+00, -4.54209089e+00] ``` 计算概率密度函数： + ```python x = np.array([2.0, 5.0, 10.0], dtype=np.float32) tx = Tensor(x, dtype=dtype.float32) @@ -383,11 +433,13 @@ print(prob) ``` 输出为： -```python + +```text [1.56874031e-01, 2.18507163e-02, 2.81590177e-03] ``` 计算对数概率密度函数： + ```python x = np.array([2.0, 5.0, 10.0], dtype=np.float32) tx = Tensor(x, dtype=dtype.float32) @@ -396,11 +448,13 @@ print(log_prob) ``` 输出为： -```python + +```text [-1.85231221e+00, -3.82352161e+00, -5.87247276e+00] ``` 调用取样函数 `sample` 抽样： + ```python shape = ((3, 2)) sample = LogNormal.sample(shape) @@ -408,13 +462,15 @@ print(sample) ``` 输出为： -```python + +```text [[7.64315844e-01, 3.01435232e-01], [1.17166102e+00, 2.60277224e+00], [7.02699006e-01, 3.91564220e-01]]) ``` 当构造 `TransformedDistribution` 映射变换的 `is_constant_jacobian = true` 时（如 `ScalarAffine`)，构造的 `TransformedDistribution` 实例可以使用直接使用 `mean` 接口计算均值，例如： + ```python normal = msd.Normal(0.0, 1.0, dtype=dtype.float32) scalaraffine = msb.ScalarAffine(1.0, 2.0) @@ -422,15 +478,19 @@ trans_dist = msd.TransformedDistribution(scalaraffine, normal, dtype=dtype.float mean = trans_dist.mean() print(mean) ``` + 输出为： -```python + +```text 2.0 ``` + ### 图模式下调用TransformedDistribution实例 在图模式下，`TransformedDistribution` 类可用在网络中。 导入相关模块： + ```python import mindspore.nn as nn from mindspore import Tensor @@ -442,6 +502,7 @@ context.set_context(mode=self.GRAPH_MODE) ``` 创建网络： + ```python class Net(nn.Cell): def __init__(self, shape, dtype=dtype.float32, seed=0, name='transformed_distribution'): @@ -451,7 +512,7 @@ class Net(nn.Cell): self.normal = msd.Normal(0.0, 1.0, dtype=dtype) self.lognormal = msd.TransformedDistribution(self.exp, self.normal, dtype=dtype, seed=seed, name=name) self.shape = shape - + def construct(self, value): cdf = self.lognormal.cdf(value) sample = self.lognormal.sample(self.shape) @@ -459,6 +520,7 @@ class Net(nn.Cell): ``` 调用网络： + ```python shape = (2, 3) net = Net(shape=shape, name="LogNormal") @@ -468,8 +530,10 @@ cdf, sample = net(tx) print("cdf: ", cdf) print("sample: ", sample) ``` + 输出为： -```python + +```text cdf: [0.7558914 0.8640314 0.9171715 0.9462397] sample: [[0.21036398 0.44932044 0.5669641 ] [1.4103683 6.724116 0.97894996]] @@ -503,6 +567,7 @@ Bijector（`mindspore.nn.probability.bijector`）是概率编程的基本组成 输入是一个 `Distribution` 类：生成一个 `TransformedDistribution` **（不可在图内调用）**。 #### 幂函数变换映射(PowerTransform) + `PowerTransform` 做如下变量替换：$Y = g(X) = {(1 + X * c)}^{1 / c}$。其接口包括： 1. 类特征函数 @@ -515,15 +580,18 @@ Bijector（`mindspore.nn.probability.bijector`）是概率编程的基本组成 - `inverse_log_jacobian`：反向映射的导数的对数，输入为 `Tensor` 。 #### 指数变换映射(Exp) + `Exp` 做如下变量替换：$Y = g(X)= exp(X)$。其接口包括： 映射函数 + - `forward`：正向映射，输入为 `Tensor` 。 - `inverse`：反向映射，输入为 `Tensor` 。 - `forward_log_jacobian`：正向映射的导数的对数，输入为 `Tensor` 。 - `inverse_log_jacobian`：反向映射的导数的对数，输入为 `Tensor` 。 #### 标量仿射变换映射(ScalarAffine) + `ScalarAffine` 做如下变量替换：Y = g(X) = a * X + b。其接口包括： 1. 类特征函数 @@ -537,6 +605,7 @@ Bijector（`mindspore.nn.probability.bijector`）是概率编程的基本组成 - `inverse_log_jacobian`：反向映射的导数的对数，输入为 `Tensor` 。 #### Softplus变换映射(Softplus) + `Softplus` 做如下变量替换：$Y = g(X) = log(1 + e ^ {kX}) / k $。其接口包括： 1. 类特征函数 @@ -553,6 +622,7 @@ Bijector（`mindspore.nn.probability.bijector`）是概率编程的基本组成 在执行之前，我们需要导入需要的库文件包。双射类最主要的库是 `mindspore.nn.probability.bijector`，导入后我们使用 `msb` 作为库的缩写并进行调用。 导入相关模块： + ```python import numpy as np import mindspore.nn as nn @@ -566,19 +636,22 @@ context.set_context(mode=context.PYNATIVE_MODE) 下面我们以 `PowerTransform` 为例。创建一个指数为2的 `PowerTransform` 对象。 构造 `PowerTransform`： + ```python powertransform = msb.PowerTransform(power=2) print(powertransform) ``` 输出： -```python + +```text PowerTransform ``` 接下来可以使用映射函数进行运算。 调用 `forward` 方法，计算正向映射： + ```python x = np.array([2.0, 3.0, 4.0, 5.0], dtype=np.float32) tx = Tensor(x, dtype=dtype.float32) @@ -587,40 +660,47 @@ print(forward) ``` 输出为： -```python + +```text [2.23606801e+00, 2.64575124e+00, 3.00000000e+00, 3.31662488e+00] ``` 输入 `inverse` 方法，计算反向映射： + ```python inverse = powertransform.inverse(tx) print(inverse) ``` 输出为： -```python + +```text [1.50000000e+00, 4.00000048e+00, 7.50000000e+00, 1.20000010e+01] ``` 输入 `forward_log_jacobian` 方法，计算正向映射导数的对数： + ```python forward_log_jaco = powertransform.forward_log_jacobian(tx) print(forward_log_jaco) ``` 输出： -```python + +```text [-8.04718971e-01, -9.72955048e-01, -1.09861231e+00, -1.19894767e+00] ``` 输入 `inverse_log_jacobian` 方法，计算反向映射导数的对数： + ```python inverse_log_jaco = powertransform.inverse_log_jacobian(tx) print(inverse_log_jaco) ``` 输出为： -```python + +```text [6.93147182e-01 1.09861231e+00 1.38629436e+00 1.60943794e+00] ``` @@ -629,6 +709,7 @@ print(inverse_log_jaco) 在图模式下，`Bijector` 子类可用在网络中。 导入相关模块： + ```python import mindspore.nn as nn from mindspore import Tensor @@ -639,6 +720,7 @@ context.set_context(mode=context.GRAPH_MODE) ``` 创建网络： + ```python class Net(nn.Cell): def __init__(self): @@ -653,7 +735,9 @@ class Net(nn.Cell): inverse_log_jaco = self.s1.inverse_log_jacobian(value) return forward, inverse, forward_log_jaco, inverse_log_jaco ``` + 调用网络： + ```python net = Net() x = np.array([2.0, 3.0, 4.0, 5.0]).astype(np.float32) @@ -664,8 +748,10 @@ print("inverse: ", inverse) print("forward_log_jaco: ", forward_log_jaco) print("inverse_log_jaco: ", inverse_log_jaco) ``` + 输出为： -```python + +```text forward: [2.236068 2.6457512 3. 3.3166249] inverse: [ 1.5 4.0000005 7.5 12.000001 ] forward_log_jaco: [-0.804719 -0.97295505 -1.0986123 -1.1989477 ] @@ -723,6 +809,7 @@ encoder = Encoder() decoder = Decoder() vae = VAE(encoder, decoder, hidden_size=400, latent_size=20) ``` + ### ConditionalVAE 类似地，ConditionalVAE与VAE的使用方法比较相近，不同的是，ConditionalVAE利用了数据集的标签信息，属于有监督学习算法，其生成效果一般会比VAE好。 @@ -779,6 +866,7 @@ cvae = ConditionalVAE(encoder, decoder, hidden_size=400, latent_size=20, num_cla ```python ds_train = create_dataset(image_path, 128, 1) ``` + 接下来，需要用到infer接口进行VAE网络的变分推断。 ## 概率推断算法 @@ -796,7 +884,9 @@ vi = SVI(net_with_loss=net_with_loss, optimizer=optimizer) vae = vi.run(train_dataset=ds_train, epochs=10) trained_loss = vi.get_train_loss() ``` + 最后，得到训练好的VAE网络后，我们可以使用`vae.generate_sample`生成新样本，需要传入待生成样本的个数，及生成样本的shape，shape需要保持和原数据集中的样本shape一样；当然，我们也可以使用`vae.reconstruct_sample`重构原来数据集中的样本，来测试VAE网络的重建能力。 + ```python generated_sample = vae.generate_sample(64, IMAGE_SHAPE) for sample in ds_train.create_dict_iterator(): @@ -804,10 +894,13 @@ for sample in ds_train.create_dict_iterator(): reconstructed_sample = vae.reconstruct_sample(sample_x) print('The shape of the generated sample is ', generated_sample.shape) ``` + 我们可以看一下新生成样本的shape： -```python + +```text The shape of the generated sample is (64, 1, 32, 32) ``` + ConditionalVAE训练过程和VAE的过程类似，但需要注意的是使用训练好的ConditionalVAE网络生成新样本和重建新样本时，需要输入标签信息，例如下面生成的新样本就是64个0-7的数字。 ```python @@ -819,8 +912,10 @@ for sample in ds_train.create_dict_iterator(): reconstructed_sample = cvae.reconstruct_sample(sample_x, sample_y) print('The shape of the generated sample is ', generated_sample.shape) ``` + 查看一下新生成的样本的shape： -```python + +```text The shape of the generated sample is (64, 1, 32, 32) ``` @@ -849,8 +944,10 @@ class TransformToBNN: self.bnn_factor = bnn_factor self.bnn_loss_file = None ``` + 参数`trainable_bnn`是经过`TrainOneStepCell`包装的可训练DNN模型，`dnn_factor`和`bnn_factor`分别为由损失函数计算得到的网络整体损失的系数和每个贝叶斯层的KL散度的系数。 API`TransformToBNN`主要实现了两个功能： + - 功能一：转换整个模型 `transform_to_bnn_model`方法可以将整个DNN模型转换为BNN模型。其定义如下： @@ -881,8 +978,9 @@ API`TransformToBNN`主要实现了两个功能： Returns: Cell, a trainable BNN model wrapped by TrainOneStepCell. """ - + ``` + 参数`get_dense_args`指定从DNN模型的全连接层中获取哪些参数，默认值是DNN模型的全连接层和BNN的全连接层所共有的参数，参数具体的含义可以参考[API说明文档](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.Dense)；`get_conv_args`指定从DNN模型的卷积层中获取哪些参数，默认值是DNN模型的卷积层和BNN的卷积层所共有的参数，参数具体的含义可以参考[API说明文档](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.Conv2d)；参数`add_dense_args`和`add_conv_args`分别指定了要为BNN层指定哪些新的参数值。需要注意的是，`add_dense_args`中的参数不能与`get_dense_args`重复，`add_conv_args`和`get_conv_args`也是如此。 - 功能二：转换指定类型的层 @@ -904,8 +1002,9 @@ API`TransformToBNN`主要实现了两个功能： Returns: Cell, a trainable model wrapped by TrainOneStepCell, whose sprcific type of layer is transformed to the corresponding bayesian layer. - """ + """ ``` + 参数`dnn_layer`指定将哪个类型的DNN层转换成BNN层，`bnn_layer`指定DNN层将转换成哪个类型的BNN层，`get_args`和`add_args`分别指定从DNN层中获取哪些参数和要为BNN层的哪些参数重新赋值。 如何在MindSpore中使用API`TransformToBNN`可以参考教程[DNN一键转换成BNN](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/apply_deep_probability_programming.html#dnnbnn) @@ -918,6 +1017,7 @@ API`TransformToBNN`主要实现了两个功能： - 认知不确定性（Epistemic Uncertainty）：模型自身对输入数据的估计可能因为训练不佳、训练数据不够等原因而不准确，可以通过增加训练数据等方式来缓解。 不确定性评估工具箱的接口如下： + - `model`：待评估不确定性的已训练好的模型。 - `train_dataset`：用于训练的数据集，迭代器类型。 - `task_type`：模型的类型，字符串，输入“regression”或者“classification”。 @@ -928,6 +1028,7 @@ API`TransformToBNN`主要实现了两个功能： - `save_model`：布尔类型，是否需要存储模型。 在使用前，需要先训练好模型，以LeNet5为例，使用方式如下： + ```python from mindspore.nn.probability.toolbox.uncertainty_evaluation import UncertaintyEvaluation from mindspore.train.serialization import load_checkpoint, load_param_into_net @@ -955,11 +1056,13 @@ if __name__ == '__main__': print('The shape of epistemic uncertainty is ', epistemic_uncertainty.shape) print('The shape of epistemic uncertainty is ', aleatoric_uncertainty.shape) ``` + `eval_epistemic_uncertainty`计算的是认知不确定性，也叫模型不确定性，对于每一个样本的每个预测标签都会有一个不确定值；`eval_aleatoric_uncertainty`计算的是偶然不确定性，也叫数据不确定性，对于每一个样本都会有一个不确定值。 所以输出为： -```python +```text The shape of epistemic uncertainty is (32, 10) The shape of epistemic uncertainty is (32,) ``` + uncertainty的值位于[0,1]之间，越大表示不确定性越高。 diff --git a/docs/programming_guide/source_zh_cn/run.md b/docs/programming_guide/source_zh_cn/run.md index 3b3138b05477a670dc46b81fddaeef1621689e26..6281328a1dff542052b7f2d093749f23ba59ecd5 100644 --- a/docs/programming_guide/source_zh_cn/run.md +++ b/docs/programming_guide/source_zh_cn/run.md @@ -15,14 +15,15 @@ ## 概述 -执行主要有三种方式：单算子、普通函数和网络训练模型。 +执行主要有三种方式：单算子、普通函数和网络训练模型。 ## 执行单算子 执行单个算子，并打印相关结果。 代码样例如下： + ```python import numpy as np import mindspore.nn as nn @@ -37,6 +38,7 @@ print(output.asnumpy()) ``` 输出如下： + ```python [[[[ 0.06022915 0.06149777 0.06149777 0.06149777 0.01145121] [ 0.06402162 0.05889071 0.05889071 0.05889071 -0.00933781] @@ -63,12 +65,12 @@ print(output.asnumpy()) [ 0.01015155 0.00781826 0.00781826 0.00781826 -0.02884173]]]] ``` - ## 执行普通函数 将若干算子组合成一个函数，然后直接通过函数调用的方式执行这些算子，并打印相关结果，如下例所示。 代码样例如下： + ```python import numpy as np from mindspore import context, Tensor @@ -88,6 +90,7 @@ print(output.asnumpy()) ``` 输出如下： + ```python [[3. 3. 3.] [3. 3. 3.] @@ -95,14 +98,17 @@ print(output.asnumpy()) ``` ## 执行网络模型 + MindSpore的Model接口是用于训练和验证的高级接口。可以将有训练或推理功能的layers组合成一个对象，通过调用train、eval、predict接口可以分别实现训练、推理和预测功能。 用户可以根据实际需要传入网络、损失函数和优化器等初始化Model接口，还可以通过配置amp_level实现混合精度，配置metrics实现模型评估。 ### 执行训练模型 + 通过调用Model的train接口可以实现训练。 代码样例如下： + ```python import os @@ -180,7 +186,7 @@ def weight_variable(): class LeNet5(nn.Cell): - """ + """ Lenet network Args: @@ -231,6 +237,7 @@ if __name__ == "__main__": > 示例中用到的MNIST数据集的获取方法，可以参照[实现一个图片分类应用](https://www.mindspore.cn/tutorial/training/zh-CN/master/quick_start/quick_start.html)的下载数据集部分，下同。 输出如下： + ```python epoch: 1 step: 1, loss is 2.300784 epoch: 1 step: 2, loss is 2.3076947 @@ -244,23 +251,26 @@ epoch: 1 step: 1875, loss is 0.017264696 > 使用PyNative模式调试， 请参考[使用PyNative模式调试](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/debug_in_pynative_mode.html)， 包括单算子、普通函数和网络训练模型的执行。 ### 执行推理模型 + 通过调用Model的train接口可以实现推理。为了方便评估模型的好坏，可以在Model接口初始化的时候设置评估指标Metric。 Metric是用于评估模型好坏的指标。常见的主要有Accuracy、Fbeta、Precision、Recall和TopKCategoricalAccuracy等，通常情况下，一种模型指标无法全面的评估模型的好坏，一般会结合多个指标共同作用对模型进行评估。 常用的内置评估指标： + - `Accuracy`（准确率）：是一个用于评估分类模型的指标。通俗来说，准确率是指我们的模型预测正确的结果所占的比例。 公式：$$Accuracy = （TP+TN）/（TP+TN+FP+FN）$$ - `Precision`（精确率）：在被识别为正类别的样本中，确实为正类别的比例。公式：$$Precision = TP/(TP+FP)$$ - `Recall`（召回率）：在所有正类别样本中，被正确识别为正类别的比例。 公式：$$Recall = TP/(TP+FN)$$ -- `Fbeta`（调和均值）：综合考虑precision和recall的调和均值。 +- `Fbeta`（调和均值）：综合考虑precision和recall的调和均值。 公式：$$F_\beta = (1 + \beta^2) \cdot \frac{precisiont \cdot recall}{(\beta^2 \cdot precision) + recall}$$ - `TopKCategoricalAccuracy`（多分类TopK准确率）：计算TopK分类准确率。 代码样例如下： + ```python import os @@ -278,7 +288,7 @@ from mindspore.train.serialization import load_checkpoint, load_param_into_net class LeNet5(nn.Cell): - """ + """ Lenet network Args: @@ -375,7 +385,8 @@ if __name__ == "__main__": > `checkpoint_lenet-1_1875.ckpt`文件的保存方法，可以参考[实现一个图片分类应用](https://www.mindspore.cn/tutorial/training/zh-CN/master/quick_start/quick_start.html)的训练网络部分。 输出如下： + ```python ============== {'Accuracy': 0.96875, 'Precision': array([0.97782258, 0.99451052, 0.98031496, 0.92723881, 0.98352214, 0.97165533, 0.98726115, 0.9472196 , 0.9394551 , 0.98236515])} ============== -``` \ No newline at end of file +``` diff --git a/docs/programming_guide/source_zh_cn/sampler.md b/docs/programming_guide/source_zh_cn/sampler.md index 295805463f13c3d35b5d75ba0814fa3772e2bd74..5a546c576905679d915f623845f03c38b6123516 100644 --- a/docs/programming_guide/source_zh_cn/sampler.md +++ b/docs/programming_guide/source_zh_cn/sampler.md @@ -65,7 +65,7 @@ for data in dataset2.create_dict_iterator(): 输出结果如下： -``` +```text Image shape: (32, 32, 3) , Label: 0 Image shape: (32, 32, 3) , Label: 2 Image shape: (32, 32, 3) , Label: 6 @@ -102,7 +102,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text Image shape: (32, 32, 3) , Label: 1 Image shape: (32, 32, 3) , Label: 1 Image shape: (32, 32, 3) , Label: 0 @@ -134,7 +134,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text Image shape: (32, 32, 3) , Label: 5 Image shape: (32, 32, 3) , Label: 0 Image shape: (32, 32, 3) , Label: 3 @@ -162,7 +162,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text Image shape: (32, 32, 3) , Label: 0 Image shape: (32, 32, 3) , Label: 0 Image shape: (32, 32, 3) , Label: 1 @@ -206,7 +206,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text {'data': Tensor(shape=[], dtype=Int64, value= 0)} {'data': Tensor(shape=[], dtype=Int64, value= 3)} {'data': Tensor(shape=[], dtype=Int64, value= 6)} @@ -236,7 +236,7 @@ for data in dataset.create_dict_iterator(): 输出结果如下： -``` +```text Image shape: (32, 32, 3) , Label: 0 Image shape: (32, 32, 3) , Label: 2 Image shape: (32, 32, 3) , Label: 4 diff --git a/docs/programming_guide/source_zh_cn/security_and_privacy.md b/docs/programming_guide/source_zh_cn/security_and_privacy.md index ec46866350195062272cd61bae7f1717c52612ff..06dda165702d0b658822f9914389c75622c011cd 100644 --- a/docs/programming_guide/source_zh_cn/security_and_privacy.md +++ b/docs/programming_guide/source_zh_cn/security_and_privacy.md @@ -26,19 +26,22 @@ ## 对抗鲁棒性 ### Attack + `Attack`基类定义了对抗样本生成的使用接口，其子类实现了各种具体的生成算法，支持安全工作人员快速高效地生成对抗样本，用于攻击AI模型，以评估模型的鲁棒性。 ### Defense + `Defense`基类定义了对抗训练的使用接口，其子类实现了各种具体的对抗训练算法，增强模型的对抗鲁棒性。 ### Detector + `Detector`基类定义了对抗样本检测的使用借口，其子类实现了各种具体的检测算法，增强模型的对抗鲁棒性。 详细内容，请参考[对抗鲁棒性官网教程](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/improve_model_security_nad.html)。 ## 模型安全测试 -### Fuzzer +### Fuzzer `Fuzzer`类基于神经元覆盖率增益控制fuzzing流程，采用自然扰动和对抗样本生成方法作为变异策略，激活更多的神经元，从而探索不同类型的模型输出结果、错误行为，指导用户增强模型鲁棒性。 diff --git a/docs/programming_guide/source_zh_cn/tensor.md b/docs/programming_guide/source_zh_cn/tensor.md index b7a6196404b7942b7bc978c6d20c01e2b1099bb5..0ed0d4273e495fcbd8122ccad7e2291e7b8c10ea 100644 --- a/docs/programming_guide/source_zh_cn/tensor.md +++ b/docs/programming_guide/source_zh_cn/tensor.md @@ -12,6 +12,8 @@ +   + ## 概述 @@ -29,7 +31,7 @@ 代码样例如下： -``` +```python import numpy as np from mindspore import Tensor from mindspore.common import dtype as mstype @@ -46,7 +48,7 @@ print(x, "\n\n", y, "\n\n", z, "\n\n", m, "\n\n", n, "\n\n", p) 输出如下： -``` +```text [[1 2] [3 4]] @@ -66,12 +68,13 @@ True ### 属性 张量的属性包括形状（shape）和数据类型（dtype）。 + - 形状：`Tensor`的shape，是一个tuple。 - 数据类型：`Tensor`的dtype，是MindSpore的一个数据类型。 代码样例如下： -``` +```python import numpy as np from mindspore import Tensor from mindspore.common import dtype as mstype @@ -85,20 +88,21 @@ print(x_shape, x_dtype) 输出如下： -``` +```text (2, 2) Int32 ``` - + ### 方法 张量的方法包括`all`、`any`和`asnumpy`，`all`和`any`方法目前只支持Ascend。 + - `all(axis, keep_dims)`：在指定维度上通过`and`操作进行归约，`axis`代表归约维度，`keep_dims`表示是否保留归约后的维度。 - `any(axis, keep_dims)`：在指定维度上通过`or`操作进行归约，参数含义同`all`。 - `asnumpy()`：将`Tensor`转换为NumPy的array。 代码样例如下： -``` +```python import numpy as np from mindspore import Tensor from mindspore.common import dtype as mstype @@ -113,7 +117,7 @@ print(x_all, "\n\n", x_any, "\n\n", x_array) 输出如下： -``` +```text False True diff --git a/docs/programming_guide/source_zh_cn/tokenizer.md b/docs/programming_guide/source_zh_cn/tokenizer.md index 66c447d1c318ed1eb1fe4ce1e2636e8967f64e70..cb00c06cd11a0f47921843d41350c68078734d04 100644 --- a/docs/programming_guide/source_zh_cn/tokenizer.md +++ b/docs/programming_guide/source_zh_cn/tokenizer.md @@ -79,7 +79,7 @@ for i in dataset.create_dict_iterator(num_epochs=1, output_numpy=True): 输出结果如下： -``` +```text ------------------------before tokenization---------------------------- 床前明月光 疑是地上霜 @@ -130,7 +130,7 @@ for i in dataset.create_dict_iterator(num_epochs=1, output_numpy=True): 输出结果如下： -``` +```text ------------------------before tokenization---------------------------- 今天天气太好了我们一起去外面玩吧 ------------------------after tokenization----------------------------- @@ -167,7 +167,7 @@ for i in dataset.create_dict_iterator(num_epochs=1, output_numpy=True): 输出结果如下： -``` +```text ------------------------before tokenization---------------------------- I saw a girl with a telescope. ------------------------after tokenization----------------------------- @@ -203,7 +203,7 @@ for i in dataset.create_dict_iterator(num_epochs=1, output_numpy=True): 输出结果如下： -``` +```text ------------------------before tokenization---------------------------- Welcome to Beijing! 北京欢迎您！ @@ -243,7 +243,7 @@ for i in dataset.create_dict_iterator(num_epochs=1, output_numpy=True): 输出结果如下： -``` +```text ------------------------before tokenization---------------------------- Welcome to Beijing! 北京欢迎您！ @@ -285,7 +285,7 @@ for i in dataset.create_dict_iterator(num_epochs=1, output_numpy=True): 输出结果如下： -``` +```text ------------------------before tokenization---------------------------- my favorite diff --git a/docs/programming_guide/source_zh_cn/train.md b/docs/programming_guide/source_zh_cn/train.md index 57c313a78f21c92f95475c5f2faf47200c89208a..12182a926ca388876e06fab60090099be4e7196b 100644 --- a/docs/programming_guide/source_zh_cn/train.md +++ b/docs/programming_guide/source_zh_cn/train.md @@ -16,9 +16,11 @@ ## 概述 + MindSpore在Model_zoo也已经提供了大量的目标检测、自然语言处理等多种网络模型，供用户直接使用，但是对于某些高级用户而言可能想要自行设计网络或者自定义训练循环，下面就对自定义训练网络、自定义训练循环和边训练边推理三种场景进行介绍，另外对On device执行方式进行详细介绍。 ## 自定义训练网络 + 在自定义训练网络前，需要先了解下MindSpore的网络支持、Python源码构造网络约束和算子支持情况。 - 网络支持：当前MindSpore已经支持多种网络，按类型分为计算机视觉、自然语言处理、推荐和图神经网络，可以通过[网络支持](https://www.mindspore.cn/doc/note/zh-CN/master/network_list.html)查看具体支持的网络情况。如果现有网络无法满足用户需求，用户可以根据实际需要定义自己的网络。 @@ -30,6 +32,7 @@ MindSpore在Model_zoo也已经提供了大量的目标检测、自然语言处 > 当开发网络遇到内置算子不足以满足需求时，用户也可以参考[自定义算子](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_operator_ascend.html)，方便快捷地扩展昇腾AI处理器的自定义算子。 代码样例如下： + ```python import numpy as np @@ -74,15 +77,18 @@ if __name__ == "__main__": ``` 输出如下： + ```python -------loss------ [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.] ``` ## 自定义训练循环 + 用户如果不想使用MindSpore提供的Model接口，可以将模仿Model的train接口自由控制循环的迭代次数和每个epoch的step数量。 代码样例如下： + ```python import os @@ -244,6 +250,7 @@ if __name__ == "__main__": > 示例中用到的MNIST数据集的获取方法，可以参照[实现一个图片分类应用](https://www.mindspore.cn/tutorial/training/zh-CN/master/quick_start/quick_start.html)的下载数据集部分，下同。 输出如下： + ```python epoch: 1/10, losses: 2.294034719467163 epoch: 2/10, losses: 2.3150298595428467 @@ -260,9 +267,11 @@ epoch: 10/10, losses: 1.4282708168029785 > 典型的使用场景是梯度累积，详细查看[梯度累积](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/apply_gradient_accumulation.html)。 ## 边训练边推理 + 对于某些数据量较大、训练时间较长的复杂网络，为了能掌握训练的不同阶段模型精度的指标变化情况，可以通过边训练边推理的方式跟踪精度的变化情况。具体可以参考[同步训练和验证模型](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/evaluate_the_model_during_training.html)。 ## on-device执行 + 当前MindSpore支持的后端包括Ascend、GPU、CPU，所谓On Device中的Device通常指Ascend（昇腾）AI处理器。 昇腾芯片上集成了AICORE、AICPU和CPU。其中，AICORE负责大型Tensor Vector运算，AICPU负责标量运算，CPU负责逻辑控制和任务分发。 @@ -270,12 +279,14 @@ epoch: 10/10, losses: 1.4282708168029785 Host侧CPU负责将图或算子下发到昇腾芯片。昇腾芯片由于具备了运算、逻辑控制和任务分发的功能，所以不需要与Host侧的CPU进行频繁的交互，只需要将计算完的最终结果返回给Host侧，实现整图下沉到Device执行，避免Host-Device频繁交互，减小了开销。 以下是Device的主要组成结构： + - 片上32G内存：5G(parameter) + 26G(feature map) + 1G(HCCL) - 多流水线并行：6条流水线 - AICORE&带宽：32Cores、读写带宽128GBps - 通信协议：HCCS、PCIe4.0、RoCEv2 ### 计算图下沉 + 计算图整图下沉到Device上执行，减少Host-Device交互开销。可以结合循环下沉实现多个Step下沉，进一步减少Host和Device的交互次数。 循环下沉是在On Device执行的基础上的优化，目的是进一步减少Host侧和Device侧之间的交互次数。通常情况下，每个step都返回一个结果，循环下沉是控制每隔多少个step返回一次结果。 @@ -285,6 +296,7 @@ Host侧CPU负责将图或算子下发到昇腾芯片。昇腾芯片由于具备 也可以结合`train`接口的`dataset_sink_mode`和`sink_size`控制每个epoch的下沉数据量。 ### 数据下沉 + `Model`的`train`接口参数`dataset_sink_mode`可以控制数据是否下沉。`dataset_sink_mode`为True表示数据下沉，否则为非下沉。所谓下沉即数据通过通道直接传送到Device上。 dataset_sink_mode参数可以配合`sink_size`控制每个`epoch`下沉的数据量大小。当`dataset_sink_mode`设置为True，即数据下沉模式时： @@ -296,6 +308,7 @@ dataset_sink_mode参数可以配合`sink_size`控制每个`epoch`下沉的数据 下沉的总数据量由`epoch`和`sink_size`两个变量共同控制，即总数据量=`epoch`*`sink_size`。 代码样例如下： + ```python import os @@ -428,6 +441,7 @@ if __name__ == "__main__": `batch_size`为32的情况下，数据集的大小为1875，当`sink_size`设置为1000时，表示每个`epoch`下沉1000个batch的数据，下沉次数为`epoch`=10，下沉的总数据量为：`epoch`*`sink_size`=10000。 输出如下： + ```python epoch: 1 step: 1000, loss is 0.5399815 epoch: 2 step: 1000, loss is 0.033433747 @@ -441,4 +455,4 @@ epoch: 9 step: 1000, loss is 0.00017951085 epoch: 10 step: 1000, loss is 0.01490275 ``` -> `dataset_sink_mode`为False时，`sink_size`参数设置无效。 \ No newline at end of file +> `dataset_sink_mode`为False时，`sink_size`参数设置无效。 diff --git a/resource/release/release_list_zh_cn.md b/resource/release/release_list_zh_cn.md index 88e820a5fba2ceb533d80ce8b6fd1b9a86fcd154..8c6df5cdfc9ecf1a44441e861070c5590294d5f2 100644 --- a/resource/release/release_list_zh_cn.md +++ b/resource/release/release_list_zh_cn.md @@ -53,11 +53,13 @@ ## 1.0.0 + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | 4682be18cffdf86346bdb286ccd9e05f33be4138415dbc7db1650d029510ee44 | @@ -97,11 +99,13 @@ | 文档 | 编程指南
Python API
C++ API
FAQ
其他说明 | ## 0.7.0-beta + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | 522b80e84de1b414d3800a27d01e40f75332000e5246b24cc1aea7d9e5566ce5 | @@ -130,11 +134,13 @@ | 文档 | | | ## 0.6.0-beta + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | afea66c19beff797b99bf06bc0ed897a83fdb510d62e03663cef55a68e0f278f | @@ -154,21 +160,25 @@ | | GPU CUDA 10.1/CPU | Ubuntu-x86 | | 18f245bdff972414010c9f53de402d790cdef9a74f94ac41e5b6341e778e93b3 | ### 教程 + ### API + ### 文档 - + ## 0.5.2-beta + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | ec4bdb6c96d9ffd2d1e465bd07ac4a8a9c0633512b4fffe9217590ad1a576ea6 | @@ -188,20 +198,25 @@ | | GPU CUDA 10.1/CPU | Ubuntu-x86 | | 09aa2887b0acbe9b31d07fb8d740c0bceefd6b8751aebdddd533f752f7564efc | ### 教程 + ### API + ### 文档 + ## 0.5.0-beta + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | f20adcdb696316361e13fcd624d7188598b7248f77c7efc535cf193afc26f1c2 | @@ -221,20 +236,25 @@ | | GPU CUDA 10.1/CPU | Ubuntu-x86 | | 09aa2887b0acbe9b31d07fb8d740c0bceefd6b8751aebdddd533f752f7564efc | ### 教程 + ### API + ### 文档 + ## 0.3.0-alpha + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | 7756a50ca3af82d06eaf456db4d062fa647a8352724ef85da6569426a6393918 | @@ -255,20 +275,25 @@ | | GPU CUDA 9.2/GPU CUDA 10.1/CPU | Ubuntu-x86 | | 7a2bd6174be9e5a47e8ae6bcdd592ecdafc6e53e6f1cd5f0261fcb8337b5b337 | ### 教程 + ### API + ### 文档 + ## 0.2.0-alpha + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | aa1225665d05263b17bb7ec1d51dd4f933254c818bee126b6c5dac4513532a14 | @@ -288,20 +313,25 @@ | | GPU CUDA 9.2/GPU CUDA 10.1/CPU | Ubuntu-x86 | | 4146790bc73a5846e92b943dfd3febb6c62052b217eeb45b6c48aa82b51e7cc3 | ### 教程 + ### API + ### 文档 + ## 0.1.0-alpha + ### 版本说明 ### 下载地址 + | 组件 | 硬件平台 | 操作系统 | 链接 | SHA-256 | | --- | --- | --- | --- | --- | | MindSpore | Ascend910 | Ubuntu-x86 | | a76df4e96c4cb69b10580fcde2d4ef46b5d426be6d47a3d8fd379c97c3e66638 | @@ -319,12 +349,15 @@ | | GPU CUDA 9.2/GPU CUDA 10.1/CPU | Ubuntu-x86 | | 7796b6c114ee4962ce605da59a9bc47390c8910acbac318ecc0598829aad6e8c | ### 教程 + ### API + ### 文档 + ## master(unstable) diff --git a/tools/link_detection/README_CN.md b/tools/link_detection/README_CN.md index 442726a409139e21f99d432923946255903b9213..c2be9e6e409f7926daaf6e5034c5525da6b120c1 100644 --- a/tools/link_detection/README_CN.md +++ b/tools/link_detection/README_CN.md @@ -3,29 +3,32 @@ ## 简介 此工具可以检查用户指定目录里所有文件的链接，将所有链接分为三类，并且将检查结果分别写入三个文件，如下所示： + 1. 响应的状态码不是200的链接，写入`400.txt`文件中。 2. 脚本执行过程中请求出现异常的链接，写入`exception.txt`文件中。 3. 对于安装包的链接，因为请求非常耗时，所以不发请求，直接写入`slow.txt`文件中。 - ## 使用说明 该工具所依赖的操作系统为Windows操作系统，执行环境为Python环境，具体使用步骤如下所示： 1. 打开Git Bash，下载MindSpore Docs仓代码。 - ``` + + ```shell git clone https://gitee.com/mindspore/docs.git ``` + 2. 进入`tools/link_detection`目录，安装执行所需的第三方库。 - ``` + + ```shell cd tools/link_detection pip install requests ``` + 3. 在`link_detection`目录下执行如下命令，在输入需要检测目录的绝对路径后，开始进行检测，完成后会在当前目录下新建`404.txt`、`exception.txt`、`slow.txt`三个文件。 - ``` + + ```shell python link_detection.py ``` - > 检测目录的绝对路径全使用英文，并且使用Linux的绝对路径方式，例如：`/d/master/docs`。 - - + > 检测目录的绝对路径全使用英文，并且使用Linux的绝对路径方式，例如：`/d/master/docs`。 diff --git a/tools/pic_detection/README_CN.md b/tools/pic_detection/README_CN.md index c217d51929f0d07908d62695626ec68a2b465d77..a3cf658bc44bc75dede5f6d86a1f649209912092 100644 --- a/tools/pic_detection/README_CN.md +++ b/tools/pic_detection/README_CN.md @@ -4,24 +4,26 @@ 此工具可以检查用户指定目录里所有图片的使用情况，会检查出没有使用的图片，并且将没有使用的图片删除。 - ## 使用说明 该工具所依赖的操作系统为Windows操作系统，执行环境为Python环境，具体使用步骤如下所示： 1. 打开Git Bash，下载MindSpore Docs仓代码。 - ``` + + ```shell git clone https://gitee.com/mindspore/docs.git ``` + 2. 进入`tools/pic_detection`目录。 - ``` + + ```shell cd tools/pic_detection ``` + 3. 在`pic_detection`目录下执行如下命令，在输入需要检测目录的绝对路径后，开始进行检测，最后将没有使用的图片删除。 - ``` + + ```shell python pic_detection.py ``` - > 检测目录的绝对路径全使用英文，并且使用Linux的绝对路径方式，例如：`/d/master/docs`。 - - + > 检测目录的绝对路径全使用英文，并且使用Linux的绝对路径方式，例如：`/d/master/docs`。 diff --git a/tutorials/inference/source_en/multi_platform_inference.md b/tutorials/inference/source_en/multi_platform_inference.md index ab1a6bf9cbd0619fbe3d346603a2aa2780af56b4..da8121d4e353a510b158762e8cbfafea9f039874 100644 --- a/tutorials/inference/source_en/multi_platform_inference.md +++ b/tutorials/inference/source_en/multi_platform_inference.md @@ -13,6 +13,7 @@ Models trained by MindSpore support the inference on different hardware platforms. This document describes the inference process on each platform. The inference can be performed in either of the following methods based on different principles: + - Use a checkpoint file for inference. That is, use the inference API to load data and the checkpoint file for inference in the MindSpore training environment. - Convert the checkpiont file into a common model format, such as ONNX or AIR, for inference. The inference environment does not depend on MindSpore. In this way, inference can be performed across hardware platforms as long as the platform supports ONNX or AIR inference. For example, models trained on the Ascend 910 AI processor can be inferred on the GPU or CPU. @@ -27,12 +28,8 @@ MindSpore supports the following inference scenarios based on the hardware platf | CPU | Checkpoint | The training environment dependency is the same as that of MindSpore. | | CPU | ONNX | Supports ONNX Runtime or SDK, for example, TensorRT. | -> Open Neural Network Exchange (ONNX) is an open file format designed for machine learning. It is used to store trained models. It enables different AI frameworks (such as PyTorch and MXNet) to store model data in the same format and interact with each other. For details, visit the ONNX official website . - -> Ascend Intermediate Representation (AIR) is an open file format defined by Huawei for machine learning and can better adapt to the Ascend AI processor. It is similar to ONNX. - -> Ascend Computer Language (ACL) provides C++ API libraries for users to develop deep neural network applications, including device management, context management, stream management, memory management, model loading and execution, operator loading and execution, and media data processing. It matches the Ascend AI processor and enables hardware running management and resource management. - -> Offline Model (OM) is supported by the Huawei Ascend AI processor. It implements preprocessing functions that can be completed without devices, such as operator scheduling optimization, weight data rearrangement and compression, and memory usage optimization. - -> NVIDIA TensorRT is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime to improve the inference speed of the deep learning model on edge devices. For details, see . +> - Open Neural Network Exchange (ONNX) is an open file format designed for machine learning. It is used to store trained models. It enables different AI frameworks (such as PyTorch and MXNet) to store model data in the same format and interact with each other. For details, visit the ONNX official website . +> - Ascend Intermediate Representation (AIR) is an open file format defined by Huawei for machine learning and can better adapt to the Ascend AI processor. It is similar to ONNX. +> - Ascend Computer Language (ACL) provides C++ API libraries for users to develop deep neural network applications, including device management, context management, stream management, memory management, model loading and execution, operator loading and execution, and media data processing. It matches the Ascend AI processor and enables hardware running management and resource management. +> - Offline Model (OM) is supported by the Huawei Ascend AI processor. It implements preprocessing functions that can be completed without devices, such as operator scheduling optimization, weight data rearrangement and compression, and memory usage optimization. +> - NVIDIA TensorRT is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime to improve the inference speed of the deep learning model on edge devices. For details, see . diff --git a/tutorials/inference/source_en/multi_platform_inference_ascend_310.md b/tutorials/inference/source_en/multi_platform_inference_ascend_310.md index 359b4d91d9c8cc3956c1ee832eeb4d077e5a7d11..ebbb52fa1a2f4e4bcca4c14ad434e5709df120bb 100644 --- a/tutorials/inference/source_en/multi_platform_inference_ascend_310.md +++ b/tutorials/inference/source_en/multi_platform_inference_ascend_310.md @@ -11,7 +11,6 @@ - ## Inference Using an ONNX or AIR File The Ascend 310 AI processor is equipped with the ACL framework and supports the OM format which needs to be converted from the model in ONNX or AIR format. For inference on the Ascend 310 AI processor, perform the following steps: diff --git a/tutorials/inference/source_en/multi_platform_inference_ascend_910.md b/tutorials/inference/source_en/multi_platform_inference_ascend_910.md index 05217b62497ea42bd2324062c36d593d6cc03ad4..bd536f4e30e6c078138e76bc250a49a6a3acdbcc 100644 --- a/tutorials/inference/source_en/multi_platform_inference_ascend_910.md +++ b/tutorials/inference/source_en/multi_platform_inference_ascend_910.md @@ -13,7 +13,7 @@ ## Inference Using a Checkpoint File -1. Use the `model.eval` interface for model validation. +1. Use the `model.eval` interface for model validation. 1.1 Local Storage @@ -34,12 +34,13 @@ acc = model.eval(dataset, dataset_sink_mode=args.dataset_sink_mode) print("============== {} ==============".format(acc)) ``` + In the preceding information: `model.eval` is an API for model validation. For details about the API, see . > Inference sample code: . 1.2 Remote Storage - + When the pre-trained models are saved remotely, the steps of performing inference on validation dataset are as follows: firstly determine which model to be used, then loading model and parameters using `mindspore_hub.load`, and finally performing inference on validation dataset once created. The processing method of the validation dataset is the same as that of the training dataset. ```python @@ -55,14 +56,17 @@ 1) acc = model.eval(dataset, dataset_sink_mode=args.dataset_sink_mode) print("============== {} ==============".format(acc)) - ``` + ``` + In the preceding information: - + `mindpsore_hub.load` is an API for loading model parameters. PLease check the details in . 2. Use the `model.predict` API to perform inference. + ```python model.predict(input_data) ``` + In the preceding information: `model.predict` is an API for inference. For details about the API, see . diff --git a/tutorials/inference/source_en/multi_platform_inference_cpu.md b/tutorials/inference/source_en/multi_platform_inference_cpu.md index 82424de89d9da00b7dac09c8c8e4495825862595..8d00afd56a67f27869dd0f68bec43c43437d8c2e 100644 --- a/tutorials/inference/source_en/multi_platform_inference_cpu.md +++ b/tutorials/inference/source_en/multi_platform_inference_cpu.md @@ -12,11 +12,12 @@ - ## Inference Using a Checkpoint File + The inference is the same as that on the Ascend 910 AI processor. ## Inference Using an ONNX File + Similar to the inference on a GPU, the following steps are required: 1. Generate a model in ONNX format on the training platform. For details, see [Export ONNX Model](https://www.mindspore.cn/tutorial/training/en/master/use/save_model.html#export-onnx-model). diff --git a/tutorials/inference/source_en/multi_platform_inference_gpu.md b/tutorials/inference/source_en/multi_platform_inference_gpu.md index d42a2ffe2548e9b4b31d4c9cbf7c38cec4439541..0c3de8af6ba83965679f63f5719233bf1b982100 100644 --- a/tutorials/inference/source_en/multi_platform_inference_gpu.md +++ b/tutorials/inference/source_en/multi_platform_inference_gpu.md @@ -12,7 +12,6 @@ - ## Inference Using a Checkpoint File The inference is the same as that on the Ascend 910 AI processor. diff --git a/tutorials/inference/source_en/serving.md b/tutorials/inference/source_en/serving.md index 589857362d7f721e002264624fcd44762477b54b..18266ebe7a82183bd9df9cc10e517abf712d2538 100644 --- a/tutorials/inference/source_en/serving.md +++ b/tutorials/inference/source_en/serving.md @@ -23,12 +23,15 @@ MindSpore Serving is a lightweight and high-performance service module that helps MindSpore developers efficiently deploy online inference services in the production environment. After completing model training using MindSpore, you can export the MindSpore model and use MindSpore Serving to create an inference service for the model. Currently, only Ascend 910 is supported. ## Starting Serving + After MindSpore is installed using `pip`, the Serving executable program is stored in `/{your python path}/lib/python3.7/site-packages/mindspore/ms_serving`. Run the following command to start Serving: -```bash -ms_serving [--help] [--model_path=] [--model_name=] [--port=] + +```bash +ms_serving [--help] [--model_path=] [--model_name=] [--port=] [--rest_api_port=] [--device_id=] ``` + Parameters are described as follows: |Parameter|Attribute|Function|Parameter Type|Default Value|Value Range| @@ -41,69 +44,84 @@ Parameters are described as follows: |`--device_id=`|Optional|Specifies device ID to be used.|Integer|0|0 to 7| > Before running the startup command, add the path `/{your python path}/lib:/{your python path}/lib/python3.7/site-packages/mindspore/lib` to the environment variable `LD_LIBRARY_PATH`. - > port and rest_ api_port cannot be the same. + > port and rest_api_port cannot be the same. ## Application Example + The following uses a simple network as an example to describe how to use MindSpore Serving. ### Exporting Model + > Before exporting the model, you need to configure MindSpore [base environment](https://www.mindspore.cn/install/en). Use [add_model.py](https://gitee.com/mindspore/mindspore/blob/master/serving/example/export_model/add_model.py) to build a network with only the Add operator and export the MindSpore inference deployment model. -```python +```shell python add_model.py ``` + Execute the script to generate the `tensor_add.mindir` file. The input of the model is two one-dimensional tensors with shape [2,2], and the output is the sum of the two input tensors. ### Starting Serving Inference + ```bash ms_serving --model_path={model directory} --model_name=tensor_add.mindir ``` + If the server prints the `MS Serving Listening on 0.0.0.0:5500` log, the Serving has loaded the inference model. ### Client Samples + #### Python Client Sample + > Before running the client sample, add the path `/{your python path}/lib/python3.7/site-packages/mindspore/` to the environment variable `PYTHONPATH`. Obtain [ms_client.py](https://gitee.com/mindspore/mindspore/blob/master/serving/example/python_client/ms_client.py) and start the Python client. + ```bash python ms_client.py ``` If the following information is displayed, the Serving has correctly executed the inference of the Add network. -``` + +```text ms client received: [[2. 2.] [2. 2.]] ``` #### C++ Client Sample + 1. Obtain an executable client sample program. Download the [MindSpore source code](https://gitee.com/mindspore/mindspore). You can use either of the following methods to compile and obtain the client sample program: - + When MindSpore is compiled using the source code, the Serving C++ client sample program is generated. You can find the `ms_client` executable program in the `build/mindspore/serving/example/cpp_client` directory. - + Independent compilation + - When MindSpore is compiled using the source code, the Serving C++ client sample program is generated. You can find the `ms_client` executable program in the `build/mindspore/serving/example/cpp_client` directory. + - Independent compilation Preinstall [gRPC](https://gRPC.io). Run the following command in the MindSpore source code path to compile a client sample program: + ```bash cd mindspore/serving/example/cpp_client mkdir build && cd build cmake -D GRPC_PATH={grpc_install_dir} .. make ``` + In the preceding command, `{grpc_install_dir}` indicates the gRPC installation path. Replace it with the actual gRPC installation path. 2. Start the client. Execute `ms_client` to send an inference request to the Serving. + ```bash ./ms_client --target=localhost:5500 ``` + If the following information is displayed, the Serving has correctly executed the inference of the Add network. - ``` + + ```text Compute [[1, 2], [3, 4]] + [[1, 2], [3, 4]] Add result is 2 4 6 8 client received: RPC OK @@ -112,69 +130,82 @@ ms client received: The client code consists of the following parts: 1. Implement the client based on MSService::Stub and create a client instance. - ``` + + ```cpp class MSClient { public: explicit MSClient(std::shared_ptr channel) : stub_(MSService::NewStub(channel)) {} private: std::unique_ptr stub_; }; - + MSClient client(grpc::CreateChannel(target_str, grpc::InsecureChannelCredentials())); - + ``` + 2. Build the request input parameter `Request`, output parameter `Reply`, and gRPC client `Context` based on the actual network input. - ``` + + ```cpp PredictRequest request; PredictReply reply; ClientContext context; - + //construct tensor Tensor data; - + //set shape TensorShape shape; shape.add_dims(4); *data.mutable_tensor_shape() = shape; - + //set type data.set_tensor_type(ms_serving::MS_FLOAT32); std::vector input_data{1, 2, 3, 4}; - + //set datas data.set_data(input_data.data(), input_data.size()); - + //add tensor to request *request.add_data() = data; *request.add_data() = data; ``` + 3. Call the gRPC API to communicate with the Serving that has been started, and obtain the return value. - ``` + + ```cpp Status status = stub_->Predict(&context, request, &reply); ``` -For details about the complete code, see [ms_client](https://gitee.com/mindspore/mindspore/blob/master/serving/example/cpp_client/ms_client.cc). +For details about the complete code, see [ms_client](https://gitee.com/mindspore/mindspore/blob/master/serving/example/cpp_client/ms_client.cc). ### REST API Client Sample + 1. Send data in the form of `data`: `data` field: flatten each input data of network model into one-dimensional data. Suppose the network model has n inputs, and the final data structure is a two-dimensional list of 1 * n. As in this example, flatten the model input data `[[1.0, 2.0], [3.0, 4.0]]` and `[[1.0, 2.0], [3.0, 4.0]]` to form `[[1.0, 2.0, 3.0, 4.0], [1.0, 2.0, 3.0, 4.0]]`. - ``` + + ```shell curl -X POST -d '{"data": [[1.0, 2.0, 3.0, 4.0], [1.0, 2.0, 3.0, 4.0]]}' http://127.0.0.1:5501 ``` + The following return values are displayed, indicating that the serving service has correctly executed the reasoning of the add network, and the output data structure is similar to that of the input: - ``` + + ```text {"data":[[2.0,4.0,6.0,8.0]]} ``` 2. Send data in the form of `tensor`: `tensor` field: composed of each input of the network model, keeping the original shape of input. As in this example, the model input data `[[1.0, 2.0], [3.0, 4.0]]` and `[[1.0, 2.0], [3.0, 4.0]]` are combined into `[[[1.0, 2.0], [3.0, 4.0]], [[1.0, 2.0], [3.0, 4.0]]]`. - ``` + + ```shell curl -X POST -d '{"tensor": [[[1.0, 2.0], [3.0, 4.0]], [[1.0, 2.0], [3.0, 4.0]]]}' http://127.0.0.1:5501 ``` + The following return values are displayed, indicating that the serving service has correctly executed the reasoning of the add network, and the output data structure is similar to that of the input: - ``` + + ```text {"tensor":[[2.0,4.0], [6.0,8.0]]} ``` - > REST APICurrently only int32 and fp32 are supported as inputs. \ No newline at end of file + + > REST APICurrently only int32 and fp32 are supported as inputs. diff --git a/tutorials/inference/source_zh_cn/multi_platform_inference.md b/tutorials/inference/source_zh_cn/multi_platform_inference.md index 22402d31027482bbcfdf8405fc93f5d558791fb2..e5ee1902d3f18961004c2e8abe276cbaa195511f 100644 --- a/tutorials/inference/source_zh_cn/multi_platform_inference.md +++ b/tutorials/inference/source_zh_cn/multi_platform_inference.md @@ -13,6 +13,7 @@ 基于MindSpore训练后的模型，支持在不同的硬件平台上执行推理。本文介绍各平台上的推理流程。 按照原理不同，推理可以有两种方式： + - 直接使用checkpiont文件进行推理，即在MindSpore训练环境下，使用推理接口加载数据及checkpoint文件进行推理。 - 将checkpiont文件转化为通用的模型格式，如ONNX、AIR格式模型文件进行推理，推理环境不需要依赖MindSpore。这样的好处是可以跨硬件平台，只要支持ONNX/AIR推理的硬件平台即可进行推理。譬如在Ascend 910 AI处理器上训练的模型，可以在GPU/CPU上进行推理。 @@ -27,12 +28,8 @@ GPU | ONNX格式 | 支持ONNX推理的runtime/SDK，如TensorRT。 CPU | checkpoint格式 | 与MindSpore训练环境依赖一致。 CPU | ONNX格式 | 支持ONNX推理的runtime/SDK，如TensorRT。 -> ONNX，全称Open Neural Network Exchange，是一种针对机器学习所设计的开放式的文件格式，用于存储训练好的模型。它使得不同的人工智能框架（如PyTorch, MXNet）可以采用相同格式存储模型数据并交互。详细了解，请参见ONNX官网。 - -> AIR，全称Ascend Intermediate Representation，类似ONNX，是华为定义的针对机器学习所设计的开放式的文件格式，能更好地适配Ascend AI处理器。 - -> ACL，全称Ascend Computer Language，提供Device管理、Context管理、Stream管理、内存管理、模型加载与执行、算子加载与执行、媒体数据处理等C++ API库，供用户开发深度神经网络应用。它匹配Ascend AI处理器，使能硬件的运行管理、资源管理能力。 - -> OM，全称Offline Model，华为Ascend AI处理器支持的离线模型，实现算子调度的优化，权值数据重排、压缩，内存使用优化等可以脱离设备完成的预处理功能。 - -> TensorRT，NVIDIA 推出的高性能深度学习推理的SDK，包括深度推理优化器和runtime，提高深度学习模型在边缘设备上的推断速度。详细请参见。 \ No newline at end of file +> - ONNX，全称Open Neural Network Exchange，是一种针对机器学习所设计的开放式的文件格式，用于存储训练好的模型。它使得不同的人工智能框架（如PyTorch, MXNet）可以采用相同格式存储模型数据并交互。详细了解，请参见ONNX官网。 +> - AIR，全称Ascend Intermediate Representation，类似ONNX，是华为定义的针对机器学习所设计的开放式的文件格式，能更好地适配Ascend AI处理器。 +> - ACL，全称Ascend Computer Language，提供Device管理、Context管理、Stream管理、内存管理、模型加载与执行、算子加载与执行、媒体数据处理等C++ API库，供用户开发深度神经网络应用。它匹配Ascend AI处理器，使能硬件的运行管理、资源管理能力。 +> - OM，全称Offline Model，华为Ascend AI处理器支持的离线模型，实现算子调度的优化，权值数据重排、压缩，内存使用优化等可以脱离设备完成的预处理功能。 +> - TensorRT，NVIDIA 推出的高性能深度学习推理的SDK，包括深度推理优化器和runtime，提高深度学习模型在边缘设备上的推断速度。详细请参见。 diff --git a/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_310.md b/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_310.md index f99df969c5dad30127e8b29d51cf3a2d25587df1..29cc91251f77d70d339f825d665915277fbe44b1 100644 --- a/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_310.md +++ b/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_310.md @@ -19,4 +19,4 @@ Ascend 310 AI处理器上搭载了ACL框架，他支持OM格式，而OM格式需 2. 将ONNX/AIR格式模型文件，转化为OM格式模型，并进行推理。 - 云上（ModelArt环境），请参考[Ascend910训练和Ascend310推理的样例](https://support.huaweicloud.com/bestpractice-modelarts/modelarts_10_0026.html)完成推理操作。 - - 本地的裸机环境（对比云上环境，即本地有Ascend 310 AI 处理器），请参考Ascend 310 AI处理器配套软件包的说明文档。 \ No newline at end of file + - 本地的裸机环境（对比云上环境，即本地有Ascend 310 AI 处理器），请参考Ascend 310 AI处理器配套软件包的说明文档。 diff --git a/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_910.md b/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_910.md index 2f54f93e870967486955c914b23556447596031b..c726609e2e2e24d65d42f584f9e6ab0503512ce5 100644 --- a/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_910.md +++ b/tutorials/inference/source_zh_cn/multi_platform_inference_ascend_910.md @@ -13,11 +13,12 @@ ## 使用checkpoint格式文件推理 -1. 使用`model.eval`接口来进行模型验证。 +1. 使用`model.eval`接口来进行模型验证。 1.1 模型已保存在本地 首先构建模型，然后使用`mindspore.train.serialization`模块的`load_checkpoint`和`load_param_into_net`从本地加载模型与参数，传入验证数据集后即可进行模型推理，验证数据集的处理方式与训练数据集相同。 + ```python network = LeNet5(cfg.num_classes) net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction="mean") @@ -33,13 +34,15 @@ acc = model.eval(dataset, dataset_sink_mode=args.dataset_sink_mode) print("============== {} ==============".format(acc)) ``` + 其中， `model.eval`为模型验证接口，对应接口说明：。 > 推理样例代码：。 1.2 使用MindSpore Hub从华为云加载模型 - + 首先构建模型，然后使用`mindspore_hub.load`从云端加载模型参数，传入验证数据集后即可进行推理，验证数据集的处理方式与训练数据集相同。 + ```python model_uid = "mindspore/ascend/0.7/googlenet_v1_cifar10" # using GoogleNet as an example. network = mindspore_hub.load(model_uid, num_classes=10) @@ -53,13 +56,16 @@ 1) acc = model.eval(dataset, dataset_sink_mode=args.dataset_sink_mode) print("============== {} ==============".format(acc)) - ``` + ``` + 其中， `mindspore_hub.load`为加载模型参数接口，对应接口说明：。 2. 使用`model.predict`接口来进行推理操作。 + ```python model.predict(input_data) ``` + 其中， - `model.predict`为推理接口，对应接口说明：。 \ No newline at end of file + `model.predict`为推理接口，对应接口说明：。 diff --git a/tutorials/inference/source_zh_cn/multi_platform_inference_cpu.md b/tutorials/inference/source_zh_cn/multi_platform_inference_cpu.md index 676ec679bddc18f98d0cc06537ba6e89cd1fc80a..82d7141468788164b7c18d166d19f40206d33be6 100644 --- a/tutorials/inference/source_zh_cn/multi_platform_inference_cpu.md +++ b/tutorials/inference/source_zh_cn/multi_platform_inference_cpu.md @@ -13,9 +13,11 @@ ## 使用checkpoint格式文件推理 + 与在Ascend 910 AI处理器上推理一样。 ## 使用ONNX格式文件推理 + 与在GPU上进行推理类似，需要以下几个步骤： 1. 在训练平台上生成ONNX格式模型，具体步骤请参考[导出ONNX格式文件](https://www.mindspore.cn/tutorial/training/zh-CN/master/use/save_model.html#onnx)。 diff --git a/tutorials/inference/source_zh_cn/multi_platform_inference_gpu.md b/tutorials/inference/source_zh_cn/multi_platform_inference_gpu.md index abd1173cf5fe61301ffd0c6f221a9a87d3b7ed6d..ea96a12c1ce5e620f6c2700aa5c26088b9e8f534 100644 --- a/tutorials/inference/source_zh_cn/multi_platform_inference_gpu.md +++ b/tutorials/inference/source_zh_cn/multi_platform_inference_gpu.md @@ -20,4 +20,4 @@ 1. 在训练平台上生成ONNX格式模型，具体步骤请参考[导出ONNX格式文件](https://www.mindspore.cn/tutorial/training/zh-CN/master/use/save_model.html#onnx)。 -2. 在GPU上进行推理，具体可以参考推理使用runtime/SDK的文档。如在Nvidia GPU上进行推理，使用常用的TensorRT，可参考[TensorRT backend for ONNX](https://github.com/onnx/onnx-tensorrt)。 \ No newline at end of file +2. 在GPU上进行推理，具体可以参考推理使用runtime/SDK的文档。如在Nvidia GPU上进行推理，使用常用的TensorRT，可参考[TensorRT backend for ONNX](https://github.com/onnx/onnx-tensorrt)。 diff --git a/tutorials/inference/source_zh_cn/serving.md b/tutorials/inference/source_zh_cn/serving.md index 4f7f3ba689ef50e3bc0fb5315dd20b4759f5307d..9288da7d9c171dd18c2cd0b170fc57ae009cb7e2 100644 --- a/tutorials/inference/source_zh_cn/serving.md +++ b/tutorials/inference/source_zh_cn/serving.md @@ -18,18 +18,20 @@ - ## 概述 MindSpore Serving是一个轻量级、高性能的服务模块，旨在帮助MindSpore开发者在生产环境中高效部署在线推理服务。当用户使用MindSpore完成模型训练后，导出MindSpore模型，即可使用MindSpore Serving创建该模型的推理服务。当前Serving仅支持Ascend 910。 ## 启动Serving服务 + 通过pip安装MindSpore后，Serving可执行程序位于`/{your python path}/lib/python3.7/site-packages/mindspore/ms_serving`。 启动Serving服务命令如下 -```bash -ms_serving [--help] [--model_path=] [--model_name=] [--port=] + +```bash +ms_serving [--help] [--model_path=] [--model_name=] [--port=] [--rest_api_port=] [--device_id=] ``` + 参数含义如下 |参数名|属性|功能描述|参数类型|默认值|取值范围| @@ -45,21 +47,24 @@ ms_serving [--help] [--model_path=] [--model_name=] [--p > port与rest_api_port不可相同。 ## 应用示例 + 下面以一个简单的网络为例，演示MindSpore Serving如何使用。 ### 导出模型 + > 导出模型之前，需要配置MindSpore[基础环境](https://www.mindspore.cn/install)。 使用[add_model.py](https://gitee.com/mindspore/mindspore/blob/master/serving/example/export_model/add_model.py)，构造一个只有Add算子的网络，并导出MindSpore推理部署模型。 -```python +```python python add_model.py ``` 执行脚本，生成`tensor_add.mindir`文件，该模型的输入为两个shape为[2,2]的二维Tensor，输出结果是两个输入Tensor之和。 ### 启动Serving推理服务 -```bash + +```bash ms_serving --model_path={model directory} --model_name=tensor_add.mindir ``` @@ -67,15 +72,19 @@ ms_serving --model_path={model directory} --model_name=tensor_add.mindir 当服务端打印日志`MS Serving RESTful start, listening on 0.0.0.0:5501`时，表示Serving REST服务已加载推理模型完毕。 ### gRPC客户端示例 + #### Python客户端示例 + > 执行客户端前，需将`/{your python path}/lib/python3.7/site-packages/mindspore`对应的路径添加到环境变量PYTHONPATH中。 获取[ms_client.py](https://gitee.com/mindspore/mindspore/blob/master/serving/example/python_client/ms_client.py)，启动Python客户端。 + ```bash python ms_client.py ``` 显示如下返回值说明Serving服务已正确执行Add网络的推理。 + ```bash ms client received: [[2. 2.] @@ -83,31 +92,37 @@ ms client received: ``` #### C++客户端示例 + 1. 获取客户端示例执行程序 首先需要下载[MindSpore源码](https://gitee.com/mindspore/mindspore)。有两种方式编译并获取客户端示例程序： - + 从源码编译MindSpore时候，将会编译产生Serving C++客户端示例程序，可在`build/mindspore/serving/example/cpp_client`目录下找到`ms_client`可执行程序。 - + 独立编译： + - 从源码编译MindSpore时候，将会编译产生Serving C++客户端示例程序，可在`build/mindspore/serving/example/cpp_client`目录下找到`ms_client`可执行程序。 + - 独立编译： 需要先预装[gRPC](https://gRPC.io)。 然后，在MindSpore源码路径中执行如下命令，编译一个客户端示例程序。 + ```bash cd mindspore/serving/example/cpp_client mkdir build && cd build cmake -D GRPC_PATH={grpc_install_dir} .. make ``` + 其中`{grpc_install_dir}`为gRPC安装时的路径，请替换为实际gRPC安装路径。 2. 启动gRPC客户端 执行ms_client，向Serving服务发送推理请求： + ```bash ./ms_client --target=localhost:5500 ``` + 显示如下返回值说明Serving服务已正确执行Add网络的推理。 - ``` + + ```text Compute [[1, 2], [3, 4]] + [[1, 2], [3, 4]] Add result is 2 4 6 8 client received: RPC OK @@ -116,75 +131,84 @@ ms client received: 客户端代码主要包含以下几个部分： 1. 基于MSService::Stub实现Client，并创建Client实例。 - ``` + + ```cpp class MSClient { public: explicit MSClient(std::shared_ptr channel) : stub_(MSService::NewStub(channel)) {} private: std::unique_ptr stub_; }; - + MSClient client(grpc::CreateChannel(target_str, grpc::InsecureChannelCredentials())); - + ``` + 2. 根据网络的实际输入构造请求的入参Request、出参Reply和gRPC的客户端Context。 - ``` + + ```cpp PredictRequest request; PredictReply reply; ClientContext context; - + //construct tensor Tensor data; - + //set shape TensorShape shape; shape.add_dims(2); shape.add_dims(2); *data.mutable_tensor_shape() = shape; - + //set type data.set_tensor_type(ms_serving::MS_FLOAT32); std::vector input_data{1, 2, 3, 4}; - + //set datas data.set_data(input_data.data(), input_data.size()); - + //add tensor to request *request.add_data() = data; *request.add_data() = data; ``` + 3. 调用gRPC接口和已经启动的Serving服务通信，并取回返回值。 ```Status status = stub_->Predict(&context, request, &reply);``` -完整代码参考[ms_client](https://gitee.com/mindspore/mindspore/blob/master/serving/example/cpp_client/ms_client.cc)。 +完整代码参考[ms_client](https://gitee.com/mindspore/mindspore/blob/master/serving/example/cpp_client/ms_client.cc)。 ### REST API客户端示例 + 1. `data`形式发送数据： data字段：将网络模型每个输入数据展平成一维数据，假设网络模型有n个输入，最后data数据结构为1*n的二维list。 - + 如本例中，将模型输入数据`[[1.0, 2.0], [3.0, 4.0]]`和`[[1.0, 2.0], [3.0, 4.0]]`展平后组合成data形式的数据`[[1.0, 2.0, 3.0, 4.0], [1.0, 2.0, 3.0, 4.0]]` - - ``` + + ```bash curl -X POST -d '{"data": [[1.0, 2.0, 3.0, 4.0], [1.0, 2.0, 3.0, 4.0]]}' http://127.0.0.1:5501 ``` - + 显示如下返回值，说明Serving服务已正确执行Add网络的推理，输出数据结构同输入类似： - ``` + + ```text {"data":[[2.0,4.0,6.0,8.0]]} ``` 2. `tensor`形式发送数据： tensor字段：由网络模型每个输入组合而成，保持输入的原始shape。 - + 如本例中，将模型输入数据`[[1.0, 2.0], [3.0, 4.0]]`和`[[1.0, 2.0], [3.0, 4.0]]`组合成tensor形式的数据`[[[1.0, 2.0], [3.0, 4.0]], [[1.0, 2.0], [3.0, 4.0]]]` - ``` + + ```bash curl -X POST -d '{"tensor": [[[1.0, 2.0], [3.0, 4.0]], [[1.0, 2.0], [3.0, 4.0]]]}' http://127.0.0.1:5501 ``` + 显示如下返回值，说明Serving服务已正确执行Add网络的推理，输出数据结构同输入类似： - ``` + + ```text {"tensor":[[2.0,4.0], [6.0,8.0]]} ``` - > REST API当前只支持int32和fp32数据输入。 + > REST API当前只支持int32和fp32数据输入。 diff --git a/tutorials/lite/source_en/quick_start/quick_start.md b/tutorials/lite/source_en/quick_start/quick_start.md index 5af499aa7a8f4cd520dd3244091afec3136d3c6c..1d3dd029c04581606709c8e9a4ffb987806c454f 100644 --- a/tutorials/lite/source_en/quick_start/quick_start.md +++ b/tutorials/lite/source_en/quick_start/quick_start.md @@ -22,26 +22,27 @@ ## Overview It is recommended that you start from the image classification demo on the Android device to understand how to build the MindSpore Lite application project, configure dependencies, and use related APIs. - + This tutorial demonstrates the on-device deployment process based on the image classification sample program on the Android device provided by the MindSpore team. 1. Select an image classification model. 2. Convert the model into a MindSpore Lite model. 3. Use the MindSpore Lite inference model on the device. The following describes how to use the MindSpore Lite C++ APIs (Android JNIs) and MindSpore Lite image classification models to perform on-device inference, classify the content captured by a device camera, and display the most possible classification result on the application's image preview screen. - + > Click to find [Android image classification models](https://download.mindspore.cn/model_zoo/official/lite/mobilenetv2_openimage_lite) and [sample code](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/lite/image_classification). ## Selecting a Model The MindSpore team provides a series of preset device models that you can use in your application. Click [here](https://download.mindspore.cn/model_zoo/official/lite/mobilenetv2_openimage_lite/mobilenetv2.ms) to download image classification models in MindSpore ModelZoo. -In addition, you can use the preset model to perform migration learning to implement your image classification tasks. +In addition, you can use the preset model to perform migration learning to implement your image classification tasks. ## Converting a Model After you retrain a model provided by MindSpore, export the model in the [.mindir format](https://www.mindspore.cn/tutorial/training/en/master/use/save_model.html#export-mindir-model). Use the MindSpore Lite [model conversion tool](https://www.mindspore.cn/tutorial/lite/en/master/use/converter_tool.html) to convert the .mindir model to a .ms model. Take the mobilenetv2 model as an example. Execute the following script to convert a model into a MindSpore Lite model for on-device inference. + ```bash ./converter_lite --fmk=MINDIR --modelFile=mobilenetv2.mindir --outputFile=mobilenetv2.ms ``` @@ -60,7 +61,7 @@ The following section describes how to build and execute an on-device image clas ### Building and Running -1. Load the sample source code to Android Studio and install the corresponding SDK. (After the SDK version is specified, Android Studio automatically installs the SDK.) +1. Load the sample source code to Android Studio and install the corresponding SDK. (After the SDK version is specified, Android Studio automatically installs the SDK.) ![start_home](../images/lite_quick_start_home.png) @@ -86,7 +87,6 @@ The following section describes how to build and execute an on-device image clas ![result](../images/lite_quick_start_app_result.png) - ## Detailed Description of the Sample Program This image classification sample program on the Android device includes a Java layer and a JNI layer. At the Java layer, the Android Camera 2 API is used to enable a camera to obtain image frames and process images. At the JNI layer, the model inference process is completed in [Runtime](https://www.mindspore.cn/tutorial/lite/en/master/use/runtime.html). @@ -95,7 +95,7 @@ This image classification sample program on the Android device includes a Java l ### Sample Program Structure -``` +```text app │ ├── src/main @@ -109,12 +109,12 @@ app │ | └── MindSporeNetnative.h # header file │ | │ ├── java # application code at the Java layer -│ │ └── com.mindspore.himindsporedemo +│ │ └── com.mindspore.himindsporedemo │ │ ├── gallery.classify # implementation related to image processing and MindSpore JNI calling │ │ │ └── ... │ │ └── widget # implementation related to camera enabling and drawing │ │ └── ... -│ │ +│ │ │ ├── res # resource files related to Android │ └── AndroidManifest.xml # Android configuration file │ @@ -135,7 +135,7 @@ Note: if the automatic download fails, please manually download the relevant lib mindspore-lite-1.0.0-minddata-arm64-cpu.tar.gz [Download link](https://ms-release.obs.cn-north-4.myhuaweicloud.com/1.0.0/lite/android_aarch64/mindspore-lite-1.0.0-minddata-arm64-cpu.tar.gz) -``` +```text android{ defaultConfig{ externalNativeBuild{ @@ -144,7 +144,7 @@ android{ } } - ndk{ + ndk{ abiFilters'armeabi-v7a', 'arm64-v8a' } } @@ -153,7 +153,7 @@ android{ Create a link to the `.so` library file in the `app/CMakeLists.txt` file: -``` +```text # ============== Set MindSpore Dependencies. ============= include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp) include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/${MINDSPORELITE_VERSION}/third_party/flatbuffers/include) @@ -171,7 +171,7 @@ set_target_properties(minddata-lite PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/src/main/cpp/${MINDSPORELITE_VERSION}/lib/libminddata-lite.so) # --------------- MindSpore Lite set End. -------------------- -# Link target library. +# Link target library. target_link_libraries( ... # --- mindspore --- @@ -193,34 +193,37 @@ mobilenetv2.ms [mobilenetv2.ms]( https://download.mindspore.cn/model_zoo/officia Call MindSpore Lite C++ APIs at the JNI layer to implement on-device inference. -The inference code process is as follows. For details about the complete code, see `src/cpp/MindSporeNetnative.cpp`. +The inference code process is as follows. For details about the complete code, see `src/cpp/MindSporeNetnative.cpp`. 1. Load the MindSpore Lite model file and build the context, session, and computational graph for inference. - Load a model file. Create and configure the context for model inference. + ```cpp // Buffer is the model data passed in by the Java layer jlong bufferLen = env->GetDirectBufferCapacity(buffer); char *modelBuffer = CreateLocalModelBuffer(env, buffer); ``` - + - Create a session. + ```cpp void **labelEnv = new void *; MSNetWork *labelNet = new MSNetWork; *labelEnv = labelNet; - + // Create context. mindspore::lite::Context *context = new mindspore::lite::Context; context->thread_num_ = num_thread; - + // Create the mindspore session. labelNet->CreateSessionMS(modelBuffer, bufferLen, "device label", context); delete(context); - + ``` - + - Load the model file and build a computational graph for inference. + ```cpp void MSNetWork::CreateSessionMS(char* modelBuffer, size_t bufferLen, std::string name, mindspore::lite::Context* ctx) { @@ -230,8 +233,8 @@ The inference code process is as follows. For details about the complete code, s int ret = session->CompileGraph(model); } ``` - -2. Convert the input image into the Tensor format of the MindSpore model. + +2. Convert the input image into the Tensor format of the MindSpore model. Convert the image data to be detected into the Tensor format of the MindSpore model. @@ -241,7 +244,7 @@ The inference code process is as follows. For details about the complete code, s // Processing such as zooming the picture size. matImgPreprocessed = PreProcessImageData(matImageSrc); - ImgDims inputDims; + ImgDims inputDims; inputDims.channel = matImgPreprocessed.channels(); inputDims.width = matImgPreprocessed.cols; inputDims.height = matImgPreprocessed.rows; @@ -261,7 +264,7 @@ The inference code process is as follows. For details about the complete code, s inputDims.channel * inputDims.width * inputDims.height * sizeof(float)); delete[] (dataHWC); ``` - + 3. Preprocessing the input data. ```cpp @@ -293,7 +296,7 @@ The inference code process is as follows. For details about the complete code, s } ``` -4. Perform inference on the input tensor based on the model, obtain the output tensor, and perform post-processing. +4. Perform inference on the input tensor based on the model, obtain the output tensor, and perform post-processing. - Perform graph execution and on-device inference. @@ -303,6 +306,7 @@ The inference code process is as follows. For details about the complete code, s ``` - Obtain the output data. + ```cpp auto names = mSession->GetOutputTensorNames(); std::unordered_map msOutputs; @@ -312,22 +316,23 @@ The inference code process is as follows. For details about the complete code, s } std::string retStr = ProcessRunnetResult(msOutputs, ret); ``` - + - Perform post-processing of the output data. + ```cpp std::string ProcessRunnetResult(std::unordered_map msOutputs, int runnetRet) { - + std::unordered_map::iterator iter; iter = msOutputs.begin(); - + // The mobilenetv2.ms model output just one branch. auto outputTensor = iter->second; int tensorNum = outputTensor->ElementsNum(); - + // Get a pointer to the first score. float *temp_scores = static_cast(outputTensor->MutableData()); - + float scores[RET_CATEGORY_SUM]; for (int i = 0; i < RET_CATEGORY_SUM; ++i) { if (temp_scores[i] > 0.5) { @@ -335,7 +340,7 @@ The inference code process is as follows. For details about the complete code, s } scores[i] = temp_scores[i]; } - + // Score for each category. // Converted to text information that needs to be displayed in the APP. std::string categoryScore = ""; @@ -347,5 +352,5 @@ The inference code process is as follows. For details about the complete code, s categoryScore += ";"; } return categoryScore; - } - ``` \ No newline at end of file + } + ``` diff --git a/tutorials/lite/source_en/use/benchmark_tool.md b/tutorials/lite/source_en/use/benchmark_tool.md index 5bf91d33c6a7603dee7a9cb74a1d9d182d81634f..60f8b6a5d80422cddc0d6c4b9036a0090a5635df 100644 --- a/tutorials/lite/source_en/use/benchmark_tool.md +++ b/tutorials/lite/source_en/use/benchmark_tool.md @@ -40,7 +40,7 @@ The main test indicator of the performance test performed by the Benchmark tool This command uses a random input, and other parameters use default values. After this command is executed, the following statistics are displayed. The statistics include the minimum duration, maximum duration, and average duration of a single inference after the tested model runs for the specified number of inference rounds. -``` +```text Model = test_benchmark.ms, numThreads = 2, MinRunTime = 72.228996 ms, MaxRuntime = 73.094002 ms, AvgRunTime = 72.556000 ms ``` @@ -50,7 +50,7 @@ Model = test_benchmark.ms, numThreads = 2, MinRunTime = 72.228996 ms, MaxRuntime This command uses a random input, sets the parameter `timeProfiling` as true, and other parameters use default values. After this command is executed, the statistics on the running time of the model at the network layer will be displayed as follows. In this case, the statistics are displayed by`opName` and `optype`. `opName` indicates the operator name, `optype` indicates the operator type, and `avg` indicates the average running time of the operator per single run, `percent` indicates the ratio of the operator running time to the total operator running time, `calledTimess` indicates the number of times that the operator is run, and `opTotalTime` indicates the total time that the operator is run for a specified number of times. Finally, `total time` and `kernel cost` show the average time consumed by a single inference operation of the model and the sum of the average time consumed by all operators in the model inference, respectively. -``` +```text ----------------------------------------------------------------------------------------- opName avg(ms) percent calledTimess opTotalTime conv2d_1/convolution 2.264800 0.824012 10 22.648003 @@ -98,7 +98,7 @@ The accuracy test performed by the Benchmark tool is to verify the accuracy of t This command specifies the input data and benchmark data of the tested model, specifies that the model inference program runs on the CPU, and sets the accuracy threshold to 3%. After this command is executed, the following statistics are displayed, including the single input data of the tested model, output result and average deviation rate of the output node, and average deviation rate of all nodes. -``` +```text InData0: 139.947 182.373 153.705 138.945 108.032 164.703 111.585 227.402 245.734 97.7776 201.89 134.868 144.851 236.027 18.1142 22.218 5.15569 212.318 198.43 221.853 ================ Comparing Output data ================ Data of node age_out : 5.94584e-08 6.3317e-08 1.94726e-07 1.91809e-07 8.39805e-08 7.66035e-08 1.69285e-07 1.46246e-07 6.03796e-07 1.77631e-07 1.54343e-07 2.04623e-07 8.89609e-07 3.63487e-06 4.86876e-06 1.23939e-05 3.09981e-05 3.37098e-05 0.000107102 0.000213932 0.000533579 0.00062465 0.00296401 0.00993984 0.038227 0.0695085 0.162854 0.123199 0.24272 0.135048 0.169159 0.0221256 0.013892 0.00502971 0.00134921 0.00135701 0.000383242 0.000163475 0.000136294 9.77864e-05 8.00793e-05 5.73874e-05 3.53858e-05 2.18535e-05 2.04467e-05 1.85286e-05 1.05075e-05 9.34751e-06 6.12732e-06 4.55476e-06 @@ -108,6 +108,7 @@ Mean bias of all nodes: 0% ``` To set specified input shapes(such as 1,32,32,1), use command as follows: + ```bash ./benchmark --modelFile=./models/test_benchmark.ms --inDataFile=./input/test_benchmark.bin --inputShapes=1,32,32,1 --device=CPU --accuracyThreshold=3 --benchmarkDataFile=./output/test_benchmark.out ``` @@ -118,11 +119,11 @@ The command used for benchmark testing based on the compiled Benchmark tool is a ```bash ./benchmark [--modelFile=] [--accuracyThreshold=] - [--benchmarkDataFile=] [--benchmarkDataType=] - [--cpuBindMode=] [--device=] [--help] - [--inDataFile=] [--loopCount=] - [--numThreads=] [--warmUpLoopCount=] - [--enableFp16=] [--timeProfiling=] + [--benchmarkDataFile=] [--benchmarkDataType=] + [--cpuBindMode=] [--device=] [--help] + [--inDataFile=] [--loopCount=] + [--numThreads=] [--warmUpLoopCount=] + [--enableFp16=] [--timeProfiling=] [--inputShapes=] ``` diff --git a/tutorials/lite/source_en/use/build.md b/tutorials/lite/source_en/use/build.md index 74b3d08a1bf96da07a791fb7e9f97ee232a500a0..b429ae30527f9fa960a6cfe51504c1da1ac46ec6 100644 --- a/tutorials/lite/source_en/use/build.md +++ b/tutorials/lite/source_en/use/build.md @@ -32,33 +32,33 @@ This chapter introduces how to quickly compile MindSpore Lite, which includes th - The compilation environment supports Linux x86_64 only. Ubuntu 18.04.02 LTS is recommended. - Compilation dependencies of runtime(cpp), benchmark: - - [CMake](https://cmake.org/download/) >= 3.14.1 - - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 - - [Android_NDK r20b](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) - - [Git](https://git-scm.com/downloads) >= 2.28.0 + - [CMake](https://cmake.org/download/) >= 3.14.1 + - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 + - [Android_NDK r20b](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) + - [Git](https://git-scm.com/downloads) >= 2.28.0 - Compilation dependencies of converter: - - [CMake](https://cmake.org/download/) >= 3.14.1 - - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 - - [Android_NDK r20b](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) - - [Git](https://git-scm.com/downloads) >= 2.28.0 - - [Autoconf](http://ftp.gnu.org/gnu/autoconf/) >= 2.69 - - [Libtool](https://www.gnu.org/software/libtool/) >= 2.4.6 - - [LibreSSL](http://www.libressl.org/) >= 3.1.3 - - [Automake](https://www.gnu.org/software/automake/) >= 1.11.6 - - [Libevent](https://libevent.org) >= 2.0 - - [M4](https://www.gnu.org/software/m4/m4.html) >= 1.4.18 - - [OpenSSL](https://www.openssl.org/) >= 1.1.1 - - [Python](https://www.python.org/) >= 3.7.5 + - [CMake](https://cmake.org/download/) >= 3.14.1 + - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 + - [Android_NDK r20b](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) + - [Git](https://git-scm.com/downloads) >= 2.28.0 + - [Autoconf](http://ftp.gnu.org/gnu/autoconf/) >= 2.69 + - [Libtool](https://www.gnu.org/software/libtool/) >= 2.4.6 + - [LibreSSL](http://www.libressl.org/) >= 3.1.3 + - [Automake](https://www.gnu.org/software/automake/) >= 1.11.6 + - [Libevent](https://libevent.org) >= 2.0 + - [M4](https://www.gnu.org/software/m4/m4.html) >= 1.4.18 + - [OpenSSL](https://www.openssl.org/) >= 1.1.1 + - [Python](https://www.python.org/) >= 3.7.5 - Compilation dependencies of runtime(java) - - [CMake](https://cmake.org/download/) >= 3.14.1 - - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 - - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 - - [Git](https://git-scm.com/downloads) >= 2.28.0 - - [Android_SDK](https://developer.android.com/studio/releases/platform-tools?hl=zh-cn#downloads) >= 30 - - [Gradle](https://gradle.org/releases/) >= 6.6.1 - - [JDK](https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html) >= 1.8 + - [CMake](https://cmake.org/download/) >= 3.14.1 + - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 + - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 + - [Git](https://git-scm.com/downloads) >= 2.28.0 + - [Android_SDK](https://developer.android.com/studio/releases/platform-tools?hl=zh-cn#downloads) >= 30 + - [Gradle](https://gradle.org/releases/) >= 6.6.1 + - [JDK](https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html) >= 1.8 > - To install and use `Android_NDK`, you need to configure environment variables. The command example is `export ANDROID_NDK={$NDK_PATH}/android-ndk-r20b`. > - Android SDK Tools need install Android SDK Build Tools. @@ -95,30 +95,37 @@ git clone https://gitee.com/mindspore/mindspore.git Then, run the following commands in the root directory of the source code to compile MindSpore Lite of different versions: - Debug version of the x86_64 architecture: + ```bash bash build.sh -I x86_64 -d ``` - Release version of the x86_64 architecture, with the number of threads set: + ```bash bash build.sh -I x86_64 -j32 ``` - Release version of the Arm 64-bit architecture in incremental compilation mode, with the number of threads set: + ```bash bash build.sh -I arm64 -i -j32 ``` - Release version of the Arm 64-bit architecture in incremental compilation mode, with the built-in GPU operator compiled: + ```bash bash build.sh -I arm64 -e gpu ``` - Compile ARM64 with image preprocessing module: + ```bash bash build.sh -I arm64 -n lite_cv ``` + - Compile MindSpore Lite AAR in incremental compilation mode: + ```bash bash build.sh -A java -i ``` @@ -128,6 +135,7 @@ Then, run the following commands in the root directory of the source code to com ### Output Description After the compilation is complete, go to the `mindspore/output` directory of the source code to view the file generated after compilation. The file is divided into three parts. + - `mindspore-lite-{version}-converter-{os}.tar.gz`: Contains model conversion tool. - `mindspore-lite-{version}-runtime-{os}-{device}.tar.gz`: Contains model inference framework, benchmarking tool and performance analysis tool. - `mindspore-lite-{version}-minddata-{os}-{device}.tar.gz`: Contains image processing library ImageProcess. @@ -146,13 +154,14 @@ tar -xvf mindspore-lite-{version}-runtime-{os}-{device}.tar.gz tar -xvf mindspore-lite-{version}-minddata-{os}-{device}.tar.gz unzip mindspore-lite-maven-{version}.zip ``` + #### Description of Converter's Directory Structure The conversion tool is only available under the `-I x86_64` compilation option, and the content includes the following parts: -``` +```text | -├── mindspore-lite-{version}-converter-{os} +├── mindspore-lite-{version}-converter-{os} │ └── converter # Model conversion Ttool │ └── lib # The dynamic link library that converter depends │ └── third_party # Header files and libraries of third party libraries @@ -164,9 +173,10 @@ The conversion tool is only available under the `-I x86_64` compilation option, The inference framework can be obtained under `-I x86_64`, `-I arm64` and `-I arm32` compilation options, and the content includes the following parts: - When the compilation option is `-I x86_64`: - ``` + + ```text | - ├── mindspore-lite-{version}-runtime-x86-cpu + ├── mindspore-lite-{version}-runtime-x86-cpu │ └── benchmark # Benchmarking Tool │ └── lib # Inference framework dynamic library │ ├── libmindspore-lite.so # Dynamic library of infernece framework in MindSpore Lite @@ -176,7 +186,8 @@ The inference framework can be obtained under `-I x86_64`, `-I arm64` and `-I ar ``` - When the compilation option is `-I arm64`: - ``` + + ```text | ├── mindspore-lite-{version}-runtime-arm64-cpu │ └── benchmark # Benchmarking Tool @@ -190,7 +201,8 @@ The inference framework can be obtained under `-I x86_64`, `-I arm64` and `-I ar ``` - When the compilation option is `-I arm32`: - ``` + + ```text | ├── mindspore-lite-{version}-runtime-arm32-cpu │ └── benchmark # Benchmarking Tool @@ -220,24 +232,23 @@ export LD_LIBRARY_PATH= ./output/mindspore-lite-{version}-runtime-x86-cpu/lib:${ - When the compilation option is `-A java`: - ``` + ```text | ├── mindspore-lite-maven-{version} │ └── mindspore - │ └── mindspore-lite - | └── {version}-SNAPSHOT - │ ├── mindspore-lite-{version}-{timestamp}-{versionCode}.aar # MindSpore Lite runtime aar + │ └── mindspore-lite + | └── {version}-SNAPSHOT + │ ├── mindspore-lite-{version}-{timestamp}-{versionCode}.aar # MindSpore Lite runtime aar ``` - #### Description of Imageprocess's Directory Structure The image processing library is only available under the `-I arm64 -n lite_cv` compilation option, and the content includes the following parts: -``` +```text | ├── mindspore-lite-{version}-minddata-{os}-{device} -│ └── benchmark # Benchmarking Tool +│ └── benchmark # Benchmarking Tool │ └── include # Head file(Only show files related to image processing) │ ├── lite_cv # Image processing library header file │ ├── image_process.h # Image processing function header file diff --git a/tutorials/lite/source_en/use/converter_tool.md b/tutorials/lite/source_en/use/converter_tool.md index 388896527d16fe9fa7eebcf42ca606745a959949..0d64d1e74022981009b66ea67a229fdeaf91cd75 100644 --- a/tutorials/lite/source_en/use/converter_tool.md +++ b/tutorials/lite/source_en/use/converter_tool.md @@ -36,9 +36,11 @@ To use the MindSpore Lite model conversion tool, you need to prepare the environ ### Example First, in the root directory of the source code, run the following command to perform compilation. For details, see `compile.md`. + ```bash bash build.sh -I x86_64 ``` + > Currently, the model conversion tool supports only the x86_64 architecture. The following describes how to use the conversion command by using several common examples. @@ -52,42 +54,51 @@ The following describes how to use the conversion command by using several commo In this example, the Caffe model is used. Therefore, the model structure and model weight files are required. Two more parameters `fmk` and `outputFile` are also required. The output is as follows: - ``` + + ```text CONVERTER RESULT SUCCESS:0 ``` + This indicates that the Caffe model is successfully converted into the MindSpore Lite model and the new file `lenet.ms` is generated. - + - The following uses the MindSpore, TensorFlow Lite, ONNX and perception quantization models as examples to describe how to run the conversion command. - - MindSpore model `model.mindir` + - MindSpore model `model.mindir` + ```bash ./converter_lite --fmk=MINDIR --modelFile=model.mindir --outputFile=model ``` - - - TensorFlow Lite model `model.tflite` + + - TensorFlow Lite model `model.tflite` + ```bash ./converter_lite --fmk=TFLITE --modelFile=model.tflite --outputFile=model ``` - - - ONNX model `model.onnx` + + - ONNX model `model.onnx` + ```bash ./converter_lite --fmk=ONNX --modelFile=model.onnx --outputFile=model ``` - - TensorFlow Lite aware quantization model `model_quant.tflite` + - TensorFlow Lite aware quantization model `model_quant.tflite` + ```bash ./converter_lite --fmk=TFLITE --modelFile=model.tflite --outputFile=model --quantType=AwareTraining ``` - - - TensorFlow Lite aware quantization model `model_quant.tflite` set the input and output data type to be float + + - TensorFlow Lite aware quantization model `model_quant.tflite` set the input and output data type to be float + ```bash ./converter_lite --fmk=TFLITE --modelFile=model.tflite --outputFile=model --quantType=AwareTraining --inferenceType=FLOAT ``` In the preceding scenarios, the following information is displayed, indicating that the conversion is successful. In addition, the target file `model.ms` is obtained. - ``` + + ```text CONVERTER RESULT SUCCESS:0 ``` + - If fail to run the conversion command, an [errorcode](https://www.mindspore.cn/doc/api_cpp/en/master/errorcode_and_metatype.html) will be output. ### Parameter Description @@ -97,7 +108,6 @@ You can enter `./converter_lite --help` to obtain help information in real time. The following describes the parameters in detail. - | Parameter | Mandatory or Not | Parameter Description | Value Range | Default Value | | -------- | ------- | ----- | --- | ---- | | `--help` | No | Prints all help information. | - | - | @@ -106,7 +116,7 @@ The following describes the parameters in detail. | `--outputFile=` | Yes | Path of the output model. (If the path does not exist, a directory will be automatically created.) The suffix `.ms` can be automatically generated. | - | - | | `--weightFile=` | Yes (for Caffe models only) | Path of the weight file of the input model. | - | - | | `--quantType=` | No | Sets the quant type of the model. | PostTraining: quantization after training
AwareTraining: perceptual quantization
WeightQuant: only do weight quantization after training | - | -| `--inferenceType= `| No(supported by aware quant models only) | Sets the input and output data type of the converted model. If the types are different from the origin model, the convert tool will insert data type convert op in the inputs and outputs of the model to make sure the data types are same as origin model. | UINT8, FLOAT or INT8 | FLOAT | +| `--inferenceType=`| No(supported by aware quant models only) | Sets the input and output data type of the converted model. If the types are different from the origin model, the convert tool will insert data type convert op in the inputs and outputs of the model to make sure the data types are same as origin model. | UINT8, FLOAT or INT8 | FLOAT | | `--bitNum=` | No | Sets the quantization bitNum when quantType is set as WeightQuant,now only support 8 bits. | 8 | 8 | | `--quantWeightSize=` | No | Sets a size threshold of convolution filter when quantType is set as WeightQuant.If size is bigger than this value,it will trigger weight quantization | (0, +∞) | 0 | | `--quantWeightChannel=` | No | Sets a channel num threshold of convolution filter when quantType is set as WeightQuant.If num is bigger than this,it will trigger weight quantization | (0, +∞) | 16 | @@ -130,9 +140,11 @@ Reference description Linux environment model conversion tool [parameter descrip ### Example Set the log printing level to INFO. + ```bash set GLOG_v=1 ``` + > Log level: 0 is DEBUG, 1 is INFO, 2 is WARNING, 3 is ERROR. Several common examples are selected below to illustrate the use of conversion commands. @@ -146,35 +158,43 @@ Several common examples are selected below to illustrate the use of conversion c In this example, because the Caffe model is used, two input files of model structure and model weight are required. Then plus fmk type and output path two parameters which are required, you can successfully execute. The result is shown as: - ``` + + ```text CONVERTER RESULT SUCCESS:0 ``` + This means that the Caffe model has been successfully converted to the MindSpore Lite model and the new file `lenet.ms` has been obtained. - + - Take MindSpore, TensorFlow Lite, ONNX model format and perceptual quantization model as examples to execute conversion commands. - - MindSpore model `model.mindir` + - MindSpore model `model.mindir` + ```bash call converter_lite --fmk=MINDIR --modelFile=model.mindir --outputFile=model ``` - - - TensorFlow Lite model`model.tflite` + + - TensorFlow Lite model`model.tflite` + ```bash call converter_lite --fmk=TFLITE --modelFile=model.tflite --outputFile=model ``` - - - ONNX model`model.onnx` + + - ONNX model`model.onnx` + ```bash call converter_lite --fmk=ONNX --modelFile=model.onnx --outputFile=model ``` - - TensorFlow Lite awaring quant model `model_quant.tflite` + - TensorFlow Lite awaring quant model `model_quant.tflite` + ```bash call converter_lite --fmk=TFLITE --modelFile=model_quant.tflite --outputFile=model --quantType=AwareTraining ``` In the above cases, the following conversion success prompt is displayed, and the `model.ms` target file is obtained at the same time. - ``` + + ```text CONVERTER RESULT SUCCESS:0 - ``` + ``` + - If fail to run the conversion command, an [errorcode](https://www.mindspore.cn/doc/api_cpp/en/master/errorcode_and_metatype.html) will be output. diff --git a/tutorials/lite/source_en/use/image_processing.md b/tutorials/lite/source_en/use/image_processing.md index f893ad41b3b9c9048310fb473c011e0f6962120d..06780e3461216f46370a7eb5fe924dba1751a4c5 100644 --- a/tutorials/lite/source_en/use/image_processing.md +++ b/tutorials/lite/source_en/use/image_processing.md @@ -7,15 +7,15 @@ - [Import image preprocessing function library](#import-image-preprocessing-function-library) - [Initialize the image](#initialize-the-image) - [Usage example](#usage-example) - - [Optional image preprocessing operator](#optional-image-preprocessing-operator) + - [Optional image preprocessing operator](#optional-image-preprocessing-operator) - [Resize image](#resize-image) - - [Usage example](#usage-example-1) + - [Usage example](#usage-example-1) - [Convert the image data type](#convert-the-image-data-type) - - [Usage example](#usage-example-2) + - [Usage example](#usage-example-2) - [Crop image data](#crop-image-data) - - [Usage example](#usage-example-3) + - [Usage example](#usage-example-3) - [Normalize image data](#normalize-image-data) - - [Usage example](#usage-example-4) + - [Usage example](#usage-example-4) @@ -29,7 +29,7 @@ The process is as follows: ## Import image preprocessing function library -``` +```cpp #include "lite_cv/lite_mat.h" #include "lite_cv/image_process.h" ``` @@ -38,13 +38,13 @@ The process is as follows: Here, the [InitFromPixel](https://www.mindspore.cn/doc/api_cpp/en/master/dataset.html#initfrompixel) function in the `image_process.h` file is used to initialize the image. -``` +```cpp bool InitFromPixel(const unsigned char *data, LPixelType pixel_type, LDataType data_type, int w, int h, LiteMat &m) ``` ### Usage example -``` +```cpp // Create the data object of the LiteMat object. LiteMat lite_mat_bgr; @@ -61,13 +61,13 @@ The image processing operators here can be used in any combination according to Here we use the [ResizeBilinear](https://www.mindspore.cn/doc/api_cpp/en/master/dataset.html#resizebilinear) function in `image_process.h` to resize the image through a bilinear algorithm. Currently, the supported data type is unit8, the supported channels are 3 and 1. -``` +```cpp bool ResizeBilinear(const LiteMat &src, LiteMat &dst, int dst_w, int dst_h) ``` #### Usage example -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -83,13 +83,13 @@ ResizeBilinear(lite_mat_bgr, lite_mat_resize, 256, 256); Here we use the [ConvertTo](https://www.mindspore.cn/doc/api_cpp/en/master/dataset.html#convertto) function in `image_process.h` to convert the image data type. Currently, the supported conversion is to convert uint8 to float. -``` +```cpp bool ConvertTo(const LiteMat &src, LiteMat &dst, double scale = 1.0) ``` #### Usage example -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -105,13 +105,13 @@ ConvertTo(lite_mat_bgr, lite_mat_convert_float); Here we use the [Crop](https://www.mindspore.cn/doc/api_cpp/en/master/dataset.html#crop) function in `image_process.h` to crop the image. Currently, channels 3 and 1 are supported. -``` +```cpp bool Crop(const LiteMat &src, LiteMat &dst, int x, int y, int w, int h) ``` #### Usage example -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -127,13 +127,13 @@ Crop(lite_mat_bgr, lite_mat_cut, 16, 16, 224, 224); In order to eliminate the dimensional influence among the data indicators, and solve the comparability problem among the data indicators through standardization processing, here is the use of the [SubStractMeanNormalize](https://www.mindspore.cn/doc/api_cpp/en/master/dataset.html#substractmeannormalize) function in `image_process.h` to normalize the image data. -``` +```cpp bool SubStractMeanNormalize(const LiteMat &src, LiteMat &dst, const std::vector &mean, const std::vector &std) ``` #### Usage example -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -148,4 +148,4 @@ LiteMat lite_mat_bgr_norm; // The image data is normalized by the mean value and variance of the image data. SubStractMeanNormalize(lite_mat_bgr, lite_mat_bgr_norm, means, stds); -``` \ No newline at end of file +``` diff --git a/tutorials/lite/source_en/use/post_training_quantization.md b/tutorials/lite/source_en/use/post_training_quantization.md index d5cb08563bd75b78c295a98ac02dba7a2702e16f..544255096c9d953c879f87b367124e9be558b05e 100644 --- a/tutorials/lite/source_en/use/post_training_quantization.md +++ b/tutorials/lite/source_en/use/post_training_quantization.md @@ -1,3 +1,3 @@ -# Converting to the MindSpore Lite Model (Post Training Quantization) +# Converting to the MindSpore Lite Model (Post Training Quantization) -Post training quantization is being translated, will be released soon. \ No newline at end of file +Post training quantization is being translated, will be released soon. diff --git a/tutorials/lite/source_en/use/runtime_cpp.md b/tutorials/lite/source_en/use/runtime_cpp.md index f763ba52b2c45e4f82a02e85c60074f04dd7c463..6b91c732b66767c4a2f4a5e55caa7d79ebdfec95 100644 --- a/tutorials/lite/source_en/use/runtime_cpp.md +++ b/tutorials/lite/source_en/use/runtime_cpp.md @@ -35,7 +35,6 @@ - ## Overview @@ -47,6 +46,7 @@ The procedure for using Runtime is shown in the following figure: ![img](../images/side_infer_process.png) Its components and their functions are described as follows: + - `Model`: model used by MindSpore Lite, which instantiates the list of operator prototypes through image composition or direct network loading. - `Lite Session`: provides the graph compilation function and calls the graph executor for inference. - `Scheduler`: operator heterogeneous scheduler. It can select a proper kernel for each operator based on the heterogeneous scheduling policy, construct a kernel list, and split a graph into subgraphs. @@ -252,6 +252,7 @@ memcpy(in_data, input_buf, data_size); ``` Note: + - The data layout in the model input tensors of MindSpore Lite must be NHWC. - The model input `input_buf` is read from disks. After it is copied to model input tensors, you need to release `input_buf`. - Vectors returned by using the `GetInputs` and `GetInputsByTensorName` methods do not need to be released by users. @@ -299,6 +300,7 @@ session->BindThread(false); ### Callback Running MindSpore Lite can transfer two `KernelCallBack` function pointers to call back the inference model when calling `RunGraph`. Compared with common graph execution, callback running can obtain extra information during the running process to help developers analyze performance and fix bugs. The extra information includes: + - Name of the running node - Input and output tensors before inference of the current node - Input and output tensors after inference of the current node @@ -380,6 +382,7 @@ delete (model); After performing inference, MindSpore Lite can obtain the model inference result. MindSpore Lite provides the following methods to obtain the model output `MSTensor`. + 1. Use the `GetOutputsByNodeName` method to obtain vectors of the model output `MSTensor` that is connected to the model output node based on the node name. ```cpp @@ -515,11 +518,13 @@ std::string version = mindspore::lite::Version(); ``` ## Session parallel launch + MindSpore Lite supports multiple `LiteSession` parallel inferences, but does not support multiple threads calling the `RunGraph` interface of a single `LiteSession` at the same time. ### Single Session parallel launch MindSpore Lite does not support multi-threaded parallel calling of the inference interface of a single `LiteSession`, otherwise we will get the following error message: + ```cpp ERROR [mindspore/lite/src/lite_session.cc:297] RunGraph] 10 Not support multi-threading ``` @@ -531,6 +536,7 @@ MindSpore Lite supports multiple `LiteSession` in doing inference in parallel. T ### Example The following code shows how to create multiple `LiteSession` and do inference in parallel: + ```cpp #include #include "src/common/file_utils.h" diff --git a/tutorials/lite/source_zh_cn/quick_start/quick_start.md b/tutorials/lite/source_zh_cn/quick_start/quick_start.md index 2a8050f6ab1872b0359887512d845331d4737174..15deccdde6f612727f150931468a4cbe54f7c894 100644 --- a/tutorials/lite/source_zh_cn/quick_start/quick_start.md +++ b/tutorials/lite/source_zh_cn/quick_start/quick_start.md @@ -22,12 +22,13 @@ ## 概述 我们推荐你从端侧Android图像分类demo入手，了解MindSpore Lite应用工程的构建、依赖项配置以及相关API的使用。 - + 本教程基于MindSpore团队提供的Android“端侧图像分类”示例程序，演示了端侧部署的流程。 + 1. 选择图像分类模型。 2. 将模型转换成MindSpore Lite模型格式。 3. 在端侧使用MindSpore Lite推理模型。详细说明如何在端侧利用MindSpore Lite C++ API（Android JNI）和MindSpore Lite图像分类模型完成端侧推理，实现对设备摄像头捕获的内容进行分类，并在APP图像预览界面中，显示出最可能的分类结果。 - + > 你可以在这里找到[Android图像分类模型](https://download.mindspore.cn/model_zoo/official/lite/mobilenetv2_openimage_lite)和[示例代码](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/lite/image_classification)。 ## 选择模型 @@ -41,6 +42,7 @@ MindSpore Model Zoo中图像分类模型可[在此下载](https://download.minds 如果预置模型已经满足你要求，请跳过本章节。 如果你需要对MindSpore提供的模型进行重训，重训完成后，需要将模型导出为[.mindir格式](https://www.mindspore.cn/tutorial/training/zh-CN/master/use/save_model.html#mindir)。然后使用MindSpore Lite[模型转换工具](https://www.mindspore.cn/tutorial/lite/zh-CN/master/use/converter_tool.html)将.mindir模型转换成.ms格式。 以mobilenetv2模型为例，如下脚本将其转换为MindSpore Lite模型用于端侧推理。 + ```bash ./converter_lite --fmk=MINDIR --modelFile=mobilenetv2.mindir --outputFile=mobilenetv2.ms ``` @@ -59,7 +61,7 @@ MindSpore Model Zoo中图像分类模型可[在此下载](https://download.minds ### 构建与运行 -1. 在Android Studio中加载本示例源码，并安装相应的SDK（指定SDK版本后，由Android Studio自动安装）。 +1. 在Android Studio中加载本示例源码，并安装相应的SDK（指定SDK版本后，由Android Studio自动安装）。 ![start_home](../images/lite_quick_start_home.png) @@ -85,13 +87,10 @@ MindSpore Model Zoo中图像分类模型可[在此下载](https://download.minds ![install](../images/lite_quick_start_install.png) - - 识别结果如下图所示。 ![result](../images/lite_quick_start_app_result.png) - ## 示例程序详细说明 本端侧图像分类Android示例程序分为JAVA层和JNI层，其中，JAVA层主要通过Android Camera 2 API实现摄像头获取图像帧，以及相应的图像处理等功能；JNI层在[Runtime](https://www.mindspore.cn/tutorial/lite/zh-CN/master/use/runtime.html)中完成模型推理的过程。 @@ -100,7 +99,7 @@ MindSpore Model Zoo中图像分类模型可[在此下载](https://download.minds ### 示例程序结构 -``` +```text app ├── src/main │ ├── assets # 资源文件 @@ -119,7 +118,7 @@ app │ │ │ └── ... │ │ └── widget # 开启摄像头及绘制相关实现 │ │ └── ... -│ │ +│ │ │ ├── res # 存放Android相关的资源文件 │ └── AndroidManifest.xml # Android配置文件 │ @@ -146,7 +145,7 @@ Android JNI层调用MindSpore C++ API时，需要相关库文件支持。可通 mindspore-lite-1.0.0-minddata-arm64-cpu.tar.gz [下载链接](https://ms-release.obs.cn-north-4.myhuaweicloud.com/1.0.0/lite/android_aarch64/mindspore-lite-1.0.0-minddata-arm64-cpu.tar.gz) -``` +```text android{ defaultConfig{ externalNativeBuild{ @@ -155,7 +154,7 @@ android{ } } - ndk{ + ndk{ abiFilters'armeabi-v7a', 'arm64-v8a' } } @@ -164,7 +163,7 @@ android{ 在`app/CMakeLists.txt`文件中建立`.so`库文件链接，如下所示。 -``` +```text # ============== Set MindSpore Dependencies. ============= include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp) include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/${MINDSPORELITE_VERSION}/third_party/flatbuffers/include) @@ -182,7 +181,7 @@ set_target_properties(minddata-lite PROPERTIES IMPORTED_LOCATION ${CMAKE_SOURCE_DIR}/src/main/cpp/${MINDSPORELITE_VERSION}/lib/libminddata-lite.so) # --------------- MindSpore Lite set End. -------------------- -# Link target library. +# Link target library. target_link_libraries( ... # --- mindspore --- @@ -202,34 +201,37 @@ target_link_libraries( 在JNI层调用MindSpore Lite C++ API实现端测推理。 -推理代码流程如下，完整代码请参见`src/cpp/MindSporeNetnative.cpp`。 +推理代码流程如下，完整代码请参见`src/cpp/MindSporeNetnative.cpp`。 1. 加载MindSpore Lite模型文件，构建上下文、会话以及用于推理的计算图。 - 加载模型文件：创建并配置用于模型推理的上下文 + ```cpp // Buffer is the model data passed in by the Java layer jlong bufferLen = env->GetDirectBufferCapacity(buffer); char *modelBuffer = CreateLocalModelBuffer(env, buffer); ``` - + - 创建会话 + ```cpp void **labelEnv = new void *; MSNetWork *labelNet = new MSNetWork; *labelEnv = labelNet; - + // Create context. mindspore::lite::Context *context = new mindspore::lite::Context; context->thread_num_ = num_thread; - + // Create the mindspore session. labelNet->CreateSessionMS(modelBuffer, bufferLen, context); delete (context); - + ``` - + - 加载模型文件并构建用于推理的计算图 + ```cpp void MSNetWork::CreateSessionMS(char* modelBuffer, size_t bufferLen, std::string name, mindspore::lite::Context* ctx) { @@ -239,8 +241,8 @@ target_link_libraries( int ret = session->CompileGraph(model); } ``` - -2. 将输入图片转换为传入MindSpore模型的Tensor格式。 + +2. 将输入图片转换为传入MindSpore模型的Tensor格式。 将待检测图片数据转换为输入MindSpore模型的Tensor。 @@ -250,7 +252,7 @@ target_link_libraries( // Processing such as zooming the picture size. matImgPreprocessed = PreProcessImageData(matImageSrc); - ImgDims inputDims; + ImgDims inputDims; inputDims.channel = matImgPreprocessed.channels(); inputDims.width = matImgPreprocessed.cols; inputDims.height = matImgPreprocessed.rows; @@ -270,7 +272,7 @@ target_link_libraries( inputDims.channel * inputDims.width * inputDims.height * sizeof(float)); delete[] (dataHWC); ``` - + 3. 对输入数据进行处理。 ```cpp @@ -302,7 +304,7 @@ target_link_libraries( } ``` -4. 对输入Tensor按照模型进行推理，获取输出Tensor，并进行后处理。 +4. 对输入Tensor按照模型进行推理，获取输出Tensor，并进行后处理。 - 图执行，端测推理。 @@ -312,6 +314,7 @@ target_link_libraries( ``` - 获取输出数据。 + ```cpp auto names = mSession->GetOutputTensorNames(); std::unordered_map msOutputs; @@ -321,22 +324,23 @@ target_link_libraries( } std::string retStr = ProcessRunnetResult(msOutputs, ret); ``` - + - 输出数据的后续处理。 + ```cpp std::string ProcessRunnetResult(std::unordered_map msOutputs, int runnetRet) { - + std::unordered_map::iterator iter; iter = msOutputs.begin(); - + // The mobilenetv2.ms model output just one branch. auto outputTensor = iter->second; int tensorNum = outputTensor->ElementsNum(); - + // Get a pointer to the first score. float *temp_scores = static_cast(outputTensor->MutableData()); - + float scores[RET_CATEGORY_SUM]; for (int i = 0; i < RET_CATEGORY_SUM; ++i) { if (temp_scores[i] > 0.5) { @@ -344,7 +348,7 @@ target_link_libraries( } scores[i] = temp_scores[i]; } - + // Score for each category. // Converted to text information that needs to be displayed in the APP. std::string categoryScore = ""; @@ -356,5 +360,5 @@ target_link_libraries( categoryScore += ";"; } return categoryScore; - } + } ``` diff --git a/tutorials/lite/source_zh_cn/use/benchmark_tool.md b/tutorials/lite/source_zh_cn/use/benchmark_tool.md index d411e48fc922562a3bef51f33063778bd6e02040..5b0c12be39e9d04994354afa179b4d0393e9457a 100644 --- a/tutorials/lite/source_zh_cn/use/benchmark_tool.md +++ b/tutorials/lite/source_zh_cn/use/benchmark_tool.md @@ -40,7 +40,7 @@ Benchmark工具进行的性能测试主要的测试指标为模型单次前向 这条命令使用随机输入，其他参数使用默认值。该命令执行后会输出如下统计信息，该信息显示了测试模型在运行指定推理轮数后所统计出的单次推理最短耗时、单次推理最长耗时和平均推理耗时。 -``` +```text Model = test_benchmark.ms, numThreads = 2, MinRunTime = 72.228996 ms, MaxRuntime = 73.094002 ms, AvgRunTime = 72.556000 ms ``` @@ -50,7 +50,7 @@ Model = test_benchmark.ms, numThreads = 2, MinRunTime = 72.228996 ms, MaxRuntime 这条命令使用随机输入，并且输出模型网络层的耗时信息，其他参数使用默认值。该命令执行后，模型网络层的耗时会输出如下统计信息，在该例中，该统计信息按照`opName`和`optype`两种划分方式分别显示，`opName`表示算子名，`optype`表示算子类别，`avg`表示该算子的平均单次运行时间，`percent`表示该算子运行耗时占所有算子运行总耗时的比例，`calledTimess`表示该算子的运行次数，`opTotalTime`表示该算子运行指定次数的总耗时。最后，`total time`和`kernel cost`分别显示了该模型单次推理的平均耗时和模型推理中所有算子的平均耗时之和。 -``` +```text ----------------------------------------------------------------------------------------- opName avg(ms) percent calledTimess opTotalTime conv2d_1/convolution 2.264800 0.824012 10 22.648003 @@ -98,7 +98,7 @@ Benchmark工具进行的精度测试主要是通过设置标杆数据来对比 这条命令指定了测试模型的输入数据、标杆数据（默认的输入及标杆数据类型均为float32），同时指定了模型推理程序在CPU上运行，并指定了准确度阈值为3%。该命令执行后会输出如下统计信息，该信息显示了测试模型的单条输入数据、输出节点的输出结果和平均偏差率以及所有节点的平均偏差率。 -``` +```text InData0: 139.947 182.373 153.705 138.945 108.032 164.703 111.585 227.402 245.734 97.7776 201.89 134.868 144.851 236.027 18.1142 22.218 5.15569 212.318 198.43 221.853 ================ Comparing Output data ================ Data of node age_out : 5.94584e-08 6.3317e-08 1.94726e-07 1.91809e-07 8.39805e-08 7.66035e-08 1.69285e-07 1.46246e-07 6.03796e-07 1.77631e-07 1.54343e-07 2.04623e-07 8.89609e-07 3.63487e-06 4.86876e-06 1.23939e-05 3.09981e-05 3.37098e-05 0.000107102 0.000213932 0.000533579 0.00062465 0.00296401 0.00993984 0.038227 0.0695085 0.162854 0.123199 0.24272 0.135048 0.169159 0.0221256 0.013892 0.00502971 0.00134921 0.00135701 0.000383242 0.000163475 0.000136294 9.77864e-05 8.00793e-05 5.73874e-05 3.53858e-05 2.18535e-05 2.04467e-05 1.85286e-05 1.05075e-05 9.34751e-06 6.12732e-06 4.55476e-06 @@ -108,6 +108,7 @@ Mean bias of all nodes: 0% ``` 如果需要指定输入数据的维度（例如输入维度为1，32，32，1），使用如下命令： + ```bash ./benchmark --modelFile=./models/test_benchmark.ms --inDataFile=./input/test_benchmark.bin --inputShapes=1,32,32,1 --device=CPU --accuracyThreshold=3 --benchmarkDataFile=./output/test_benchmark.out ``` @@ -118,11 +119,11 @@ Mean bias of all nodes: 0% ```bash ./benchmark [--modelFile=] [--accuracyThreshold=] - [--benchmarkDataFile=] [--benchmarkDataType=] - [--cpuBindMode=] [--device=] [--help] - [--inDataFile=] [--loopCount=] - [--numThreads=] [--warmUpLoopCount=] - [--enableFp16=] [--timeProfiling=] + [--benchmarkDataFile=] [--benchmarkDataType=] + [--cpuBindMode=] [--device=] [--help] + [--inDataFile=] [--loopCount=] + [--numThreads=] [--warmUpLoopCount=] + [--enableFp16=] [--timeProfiling=] [--inputShapes=] ``` diff --git a/tutorials/lite/source_zh_cn/use/build.md b/tutorials/lite/source_zh_cn/use/build.md index eebe9a0f59def2f9d89d642f106ce79750297936..09fdccb0d0269dc6c48107529dc1c6e51be78659 100644 --- a/tutorials/lite/source_zh_cn/use/build.md +++ b/tutorials/lite/source_zh_cn/use/build.md @@ -31,31 +31,31 @@ - 系统环境：Linux x86_64，推荐使用Ubuntu 18.04.02LTS - runtime(cpp)、benchmark编译依赖 - - [CMake](https://cmake.org/download/) >= 3.14.1 - - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 - - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 - - [Git](https://git-scm.com/downloads) >= 2.28.0 + - [CMake](https://cmake.org/download/) >= 3.14.1 + - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 + - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 + - [Git](https://git-scm.com/downloads) >= 2.28.0 - converter编译依赖 - - [CMake](https://cmake.org/download/) >= 3.14.1 - - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 - - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 - - [Git](https://git-scm.com/downloads) >= 2.28.0 - - [Autoconf](http://ftp.gnu.org/gnu/autoconf/) >= 2.69 - - [Libtool](https://www.gnu.org/software/libtool/) >= 2.4.6 - - [LibreSSL](http://www.libressl.org/) >= 3.1.3 - - [Automake](https://www.gnu.org/software/automake/) >= 1.11.6 - - [Libevent](https://libevent.org) >= 2.0 - - [M4](https://www.gnu.org/software/m4/m4.html) >= 1.4.18 - - [OpenSSL](https://www.openssl.org/) >= 1.1.1 - - [Python](https://www.python.org/) >= 3.7.5 + - [CMake](https://cmake.org/download/) >= 3.14.1 + - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 + - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 + - [Git](https://git-scm.com/downloads) >= 2.28.0 + - [Autoconf](http://ftp.gnu.org/gnu/autoconf/) >= 2.69 + - [Libtool](https://www.gnu.org/software/libtool/) >= 2.4.6 + - [LibreSSL](http://www.libressl.org/) >= 3.1.3 + - [Automake](https://www.gnu.org/software/automake/) >= 1.11.6 + - [Libevent](https://libevent.org) >= 2.0 + - [M4](https://www.gnu.org/software/m4/m4.html) >= 1.4.18 + - [OpenSSL](https://www.openssl.org/) >= 1.1.1 + - [Python](https://www.python.org/) >= 3.7.5 - runtime(java)编译依赖 - - [CMake](https://cmake.org/download/) >= 3.14.1 - - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 - - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 - - [Git](https://git-scm.com/downloads) >= 2.28.0 - - [Android_SDK](https://developer.android.com/studio/releases/platform-tools?hl=zh-cn#downloads) >= 30 - - [Gradle](https://gradle.org/releases/) >= 6.6.1 - - [JDK](https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html) >= 1.8 + - [CMake](https://cmake.org/download/) >= 3.14.1 + - [GCC](https://gcc.gnu.org/releases.html) >= 7.3.0 + - [Android_NDK](https://dl.google.com/android/repository/android-ndk-r20b-linux-x86_64.zip) >= r20 + - [Git](https://git-scm.com/downloads) >= 2.28.0 + - [Android_SDK](https://developer.android.com/studio/releases/platform-tools?hl=zh-cn#downloads) >= 30 + - [Gradle](https://gradle.org/releases/) >= 6.6.1 + - [JDK](https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html) >= 1.8 > - 当安装完依赖项Android_NDK后，需配置环境变量:`export ANDROID_NDK={$NDK_PATH}/android-ndk-r20b`。 > - Android SDK组件需要安装Android SDK Build Tools。 @@ -92,31 +92,37 @@ git clone https://gitee.com/mindspore/mindspore.git 然后，在源码根目录下执行如下命令，可编译不同版本的MindSpore Lite。 - 编译x86_64架构Debug版本。 + ```bash bash build.sh -I x86_64 -d ``` - 编译x86_64架构Release版本，同时设定线程数。 + ```bash bash build.sh -I x86_64 -j32 ``` - 增量编译ARM64架构Release版本，同时设定线程数。 + ```bash bash build.sh -I arm64 -i -j32 ``` - 编译ARM64架构Release版本，同时编译内置的GPU算子。 + ```bash bash build.sh -I arm64 -e gpu ``` - 编译ARM64带图像预处理模块。 + ```bash bash build.sh -I arm64 -n lite_cv ``` - 增量编译MindSpore Lite AAR。 + ```bash bash build.sh -A java -i ``` @@ -150,9 +156,9 @@ unzip mindspore-lite-maven-{version}.zip 转换工具仅在`-I x86_64`编译选项下获得，内容包括以下几部分： -``` +```text | -├── mindspore-lite-{version}-converter-{os} +├── mindspore-lite-{version}-converter-{os} │ └── converter # 模型转换工具 │ └── lib # 转换工具依赖的动态库 │ └── third_party # 第三方库头文件和库 @@ -164,9 +170,10 @@ unzip mindspore-lite-maven-{version}.zip 推理框架可在`-I x86_64`、`-I arm64`、`-I arm32`和`-A java`编译选项下获得，内容包括以下几部分： - 当编译选项为`-I x86_64`时： - ``` + + ```text | - ├── mindspore-lite-{version}-runtime-x86-cpu + ├── mindspore-lite-{version}-runtime-x86-cpu │ └── benchmark # 基准测试工具 │ └── lib # 推理框架动态库 │ ├── libmindspore-lite.so # MindSpore Lite推理框架的动态库 @@ -176,7 +183,8 @@ unzip mindspore-lite-maven-{version}.zip ``` - 当编译选项为`-I arm64`时： - ``` + + ```text | ├── mindspore-lite-{version}-runtime-arm64-cpu │ └── benchmark # 基准测试工具 @@ -190,7 +198,8 @@ unzip mindspore-lite-maven-{version}.zip ``` - 当编译选项为`-I arm32`时： - ``` + + ```text | ├── mindspore-lite-{version}-runtime-arm32-cpu │ └── benchmark # 基准测试工具 @@ -220,23 +229,23 @@ export LD_LIBRARY_PATH=./output/mindspore-lite-{version}-runtime-x86-cpu/lib:${L - 当编译选项为`-A java`时： - ``` + ```text | ├── mindspore-lite-maven-{version} │ └── mindspore - │ └── mindspore-lite - | └── {version}-SNAPSHOT - │ ├── mindspore-lite-{version}-{timestamp}-{versionCode}.aar # MindSpore Lite推理框架aar包 + │ └── mindspore-lite + | └── {version}-SNAPSHOT + │ ├── mindspore-lite-{version}-{timestamp}-{versionCode}.aar # MindSpore Lite推理框架aar包 ``` #### 图像处理库目录结构说明 图像处理库在`-I arm64 -n lite_cv`编译选项下获得，内容包括以下几部分： -``` +```text | ├── mindspore-lite-{version}-minddata-{os}-{device} -│ └── benchmark # 基准测试工具 +│ └── benchmark # 基准测试工具 │ └── include # 头文件(此处只展示和图像处理相关的文件) │ ├── lite_cv # 图像处理库头文件 │ ├── image_process.h # 图像处理函数头文件 diff --git a/tutorials/lite/source_zh_cn/use/converter_tool.md b/tutorials/lite/source_zh_cn/use/converter_tool.md index 157d3e7758bcf0dba4f975f13bc96b71675e387d..57c8002b60e2956aa2ffc0dffbf6a91696a18e32 100644 --- a/tutorials/lite/source_zh_cn/use/converter_tool.md +++ b/tutorials/lite/source_zh_cn/use/converter_tool.md @@ -36,9 +36,11 @@ MindSpore Lite提供离线转换模型功能的工具，支持多种类型的模 ### 使用示例 首先，在源码根目录下，输入命令进行编译，可参考`build.md`。 + ```bash bash build.sh -I x86_64 ``` + > 目前模型转换工具仅支持x86_64架构。 下面选取了几个常用示例，说明转换命令的使用方法。 @@ -52,43 +54,51 @@ bash build.sh -I x86_64 本例中，因为采用了Caffe模型，所以需要模型结构、模型权值两个输入文件。再加上其他必需的fmk类型和输出路径两个参数，即可成功执行。 结果显示为： - ``` + + ```text CONVERTER RESULT SUCCESS:0 ``` + 这表示已经成功将Caffe模型转化为MindSpore Lite模型，获得新文件`lenet.ms`。 - + - 以MindSpore、TensorFlow Lite、ONNX模型格式和感知量化模型为例，执行转换命令。 - - MindSpore模型`model.mindir` + - MindSpore模型`model.mindir` + ```bash ./converter_lite --fmk=MINDIR --modelFile=model.mindir --outputFile=model ``` - - - TensorFlow Lite模型`model.tflite` + + - TensorFlow Lite模型`model.tflite` + ```bash ./converter_lite --fmk=TFLITE --modelFile=model.tflite --outputFile=model ``` - - - ONNX模型`model.onnx` + + - ONNX模型`model.onnx` + ```bash ./converter_lite --fmk=ONNX --modelFile=model.onnx --outputFile=model ``` - - TensorFlow Lite感知量化模型`model_quant.tflite` + - TensorFlow Lite感知量化模型`model_quant.tflite` + ```bash ./converter_lite --fmk=TFLITE --modelFile=model_quant.tflite --outputFile=model --quantType=AwareTraining ``` - - 感知量化模型输入输出类型设置为float - + - 感知量化模型输入输出类型设置为float + ```bash ./converter_lite --fmk=TFLITE --modelFile=model_quant.tflite --outputFile=model --quantType=AwareTraining --inferenceType=FLOAT ``` + 以上几种情况下，均显示如下转换成功提示，且同时获得`model.ms`目标文件。 - - ``` + + ```text CONVERTER RESULT SUCCESS:0 ``` + - 如果转换命令执行失败，程序会返回一个[错误码](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/errorcode_and_metatype.html)。 > 训练后量化示例请参考。 @@ -107,13 +117,12 @@ MindSpore Lite模型转换工具提供了多种参数设置，用户可根据需 | `--outputFile=` | 是 | 输出模型的路径（不存在时将自动创建目录），不需加后缀，可自动生成`.ms`后缀。 | - | - | | `--weightFile=` | 转换Caffe模型时必选 | 输入模型weight文件的路径。 | - | - | | `--quantType=` | 否 | 设置模型的量化类型。 | WeightQuant：训练后量化（权重量化）
PostTraining：训练后量化（全量化）
AwareTraining：感知量化 | - | -|` --inferenceType=` | 否 | 设置感知量化模型输入输出数据类型，如果和原模型不一致则转换工具会在模型前后插转换算子，使得转换后的模型输入输出类型和inferenceType保持一致。 | UINT8、FLOAT、INT8 | FLOAT | +|`--inferenceType=` | 否 | 设置感知量化模型输入输出数据类型，如果和原模型不一致则转换工具会在模型前后插转换算子，使得转换后的模型输入输出类型和inferenceType保持一致。 | UINT8、FLOAT、INT8 | FLOAT | | `--bitNum=` | 否 | 设定训练后量化（权重量化）的比特数，目前仅支持8bit量化 | 8 | 8 | | `--quantWeightSize=` | 否 | 设定参与训练后量化（权重量化）的卷积核尺寸阈值，若卷积核尺寸大于该值，则对此权重进行量化 | （0，+∞） | 0 | | `--quantWeightChannel=` | 否 | 设定参与训练后量化（权重量化）的卷积通道数阈值，若卷积通道数大于该值，则对此权重进行量化 | （0，+∞） | 16 | | `--configFile=` | 否 | 训练后量化（全量化）校准数据集配置文件路径 | - | - | - > - 参数名和参数值之间用等号连接，中间不能有空格。 > - Caffe模型一般分为两个文件：`*.prototxt`模型结构，对应`--modelFile`参数；`*.caffemodel`模型权值，对应`--weightFile`参数。 @@ -132,9 +141,11 @@ MindSpore Lite模型转换工具提供了多种参数设置，用户可根据需 ### 使用示例 设置日志打印级别为INFO。 + ```bash set GLOG_v=1 ``` + > 日志级别：0代表DEBUG，1代表INFO，2代表WARNING，3代表ERROR。 下面选取了几个常用示例，说明转换命令的使用方法。 @@ -148,35 +159,43 @@ set GLOG_v=1 本例中，因为采用了Caffe模型，所以需要模型结构、模型权值两个输入文件。再加上其他必需的fmk类型和输出路径两个参数，即可成功执行。 结果显示为： - ``` + + ```text CONVERTER RESULT SUCCESS:0 ``` + 这表示已经成功将Caffe模型转化为MindSpore Lite模型，获得新文件`lenet.ms`。 - + - 以MindSpore、TensorFlow Lite、ONNX模型格式和感知量化模型为例，执行转换命令。 - - MindSpore模型`model.mindir` + - MindSpore模型`model.mindir` + ```bash call converter_lite --fmk=MINDIR --modelFile=model.mindir --outputFile=model ``` - - - TensorFlow Lite模型`model.tflite` + + - TensorFlow Lite模型`model.tflite` + ```bash call converter_lite --fmk=TFLITE --modelFile=model.tflite --outputFile=model ``` - - - ONNX模型`model.onnx` + + - ONNX模型`model.onnx` + ```bash call converter_lite --fmk=ONNX --modelFile=model.onnx --outputFile=model ``` - - TensorFlow Lite感知量化模型`model_quant.tflite` + - TensorFlow Lite感知量化模型`model_quant.tflite` + ```bash call converter_lite --fmk=TFLITE --modelFile=model_quant.tflite --outputFile=model --quantType=AwareTraining ``` 以上几种情况下，均显示如下转换成功提示，且同时获得`model.ms`目标文件。 - ``` + + ```text CONVERTER RESULT SUCCESS:0 - ``` + ``` + - 如果转换命令执行失败，程序会返回一个[错误码](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/errorcode_and_metatype.html)。 diff --git a/tutorials/lite/source_zh_cn/use/image_processing.md b/tutorials/lite/source_zh_cn/use/image_processing.md index c99badd2254a8a72b23c407c3bce24c80ac75350..b328bb056ad6a6671fd56fb793741968e6e4ae26 100644 --- a/tutorials/lite/source_zh_cn/use/image_processing.md +++ b/tutorials/lite/source_zh_cn/use/image_processing.md @@ -7,15 +7,15 @@ - [导入图像预处理函数的库](#导入图像预处理函数的库) - [对图像进行初始化](#对图像进行初始化) - [使用示例](#使用示例) - - [可选的图像预处理算子](#可选的图像预处理算子) + - [可选的图像预处理算子](#可选的图像预处理算子) - [对图像进行缩放操作](#对图像进行缩放操作) - - [使用示例](#使用示例-1) + - [使用示例](#使用示例-1) - [对图像数据类型进行转换](#对图像数据类型进行转换) - - [使用示例](#使用示例-2) + - [使用示例](#使用示例-2) - [对图像数据进行裁剪](#对图像数据进行裁剪) - - [使用示例](#使用示例-3) + - [使用示例](#使用示例-3) - [对图像数据进行归一化处理](#对图像数据进行归一化处理) - - [使用示例](#使用示例-4) + - [使用示例](#使用示例-4) @@ -29,7 +29,7 @@ ## 导入图像预处理函数的库 -``` +```cpp #include "lite_cv/lite_mat.h" #include "lite_cv/image_process.h" ``` @@ -38,13 +38,13 @@ 这边使用的是`image_process.h`文件中的[InitFromPixel](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/dataset.html#initfrompixel)函数对图像进行初始化操作。 -``` +```cpp bool InitFromPixel(const unsigned char *data, LPixelType pixel_type, LDataType data_type, int w, int h, LiteMat &m) ``` ### 使用示例 -``` +```cpp // Create the data object of the LiteMat object. LiteMat lite_mat_bgr; @@ -61,13 +61,13 @@ InitFromPixel(pixel_ptr, LPixelType::RGBA2GRAY, LDataType::UINT8, rgba_mat.cols, 这边利用的是`image_process.h`中的[ResizeBilinear](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/dataset.html#resizebilinear)函数通过双线性算法调整图像大小，当前仅支持的数据类型为uint8，当前支持的通道为3和1。 -``` +```cpp bool ResizeBilinear(const LiteMat &src, LiteMat &dst, int dst_w, int dst_h) ``` #### 使用示例 -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -83,13 +83,13 @@ ResizeBilinear(lite_mat_bgr, lite_mat_resize, 256, 256); 这边利用的是`image_process.h`中的[ConvertTo](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/dataset.html#convertto)函数对图像数据类型进行转换，目前支持的转换是将uint8转换为float。 -``` +```cpp bool ConvertTo(const LiteMat &src, LiteMat &dst, double scale = 1.0) ``` #### 使用示例 -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -105,13 +105,13 @@ ConvertTo(lite_mat_bgr, lite_mat_convert_float); 这边利用的是`image_process.h`中的[Crop](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/dataset.html#crop)函数对图像进行裁剪，目前支持通道3和1。 -``` +```cpp bool Crop(const LiteMat &src, LiteMat &dst, int x, int y, int w, int h) ``` #### 使用示例 -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -127,13 +127,13 @@ Crop(lite_mat_bgr, lite_mat_cut, 16, 16, 224, 224); 为了消除数据指标之间的量纲影响，通过标准化处理来解决数据指标之间的可比性问题，这边利用的是`image_process.h`中的[SubStractMeanNormalize](https://www.mindspore.cn/doc/api_cpp/zh-CN/master/dataset.html#substractmeannormalize)函数对图像数据进行归一化处理。 -``` +```cpp bool SubStractMeanNormalize(const LiteMat &src, LiteMat &dst, const std::vector &mean, const std::vector &std) ``` #### 使用示例 -``` +```cpp // Initialize the image data. LiteMat lite_mat_bgr; InitFromPixel(rgba_mat.data, LPixelType::RGBA2BGR, LDataType::UINT8, rgba_mat.cols, rgba_mat.rows, lite_mat_bgr); @@ -148,4 +148,4 @@ LiteMat lite_mat_bgr_norm; // The image data is normalized by the mean value and variance of the image data. SubStractMeanNormalize(lite_mat_bgr, lite_mat_bgr_norm, means, stds); -``` \ No newline at end of file +``` diff --git a/tutorials/lite/source_zh_cn/use/post_training_quantization.md b/tutorials/lite/source_zh_cn/use/post_training_quantization.md index a1b9875ab15093b07d5f65a9a0fcb8f224bbd7b0..1e10f8ab39a6ba4d677a7f49d2790b3f72f3f8da 100644 --- a/tutorials/lite/source_zh_cn/use/post_training_quantization.md +++ b/tutorials/lite/source_zh_cn/use/post_training_quantization.md @@ -23,6 +23,7 @@ 目前训练后量化属于alpha阶段（支持部分网络，不支持多输入模型），正在持续完善中。 MindSpore Lite训练后量化分为两类： + 1. 权重量化：单独对模型的权值进行量化； 2. 全量化：对模型的权值、激活值、bias值统一进行量化。 @@ -35,13 +36,15 @@ MindSpore Lite训练后量化分为两类： ### 参数说明 权重量化转换命令的一般形式为： -``` + +```bash ./converter_lite --fmk=ModelType --modelFile=ModelFilePath --outputFile=ConvertedModelPath --quantType=WeightQuant --bitNum=BitNumValue --quantSize=QuantizationSizeThresholdValue --convWeightQuantChannelThreshold=ConvWeightQuantChannelThresholdValue ``` + 下面对此命令的量化相关参数进行说明： | 参数 | 属性 | 功能描述 | 参数类型 | 默认值 | 取值范围 | -| -------- | ------- | ----- | ----- |----- | ----- | +| -------- | ------- | ----- | ----- |----- | ----- | | `--quantType=` | 必选 | 设置为WeightQuant，启用权重量化 | String | - | 必须设置为WeightQuant | | `--bitNum=` | 可选 | 设定权重量化的比特数，目前仅支持8bit量化 | Integer | 8 | 8 | | `--quantSize=` | 可选 | 设定参与权重量化的卷积核尺寸阈值，若卷积核尺寸大于该值，则对此权重进行量化；建议设置为500 | Integer | 0 | （0，+∞） | @@ -49,21 +52,22 @@ MindSpore Lite训练后量化分为两类： 用户可根据模型及自身需要对权重量化的参数作出调整。 - ### 使用步骤 1. 正确编译出`converter_lite`可执行文件。该部分可参考构建文档[编译MindSpore Lite](https://www.mindspore.cn/tutorial/lite/zh-CN/master/use/build.html)，获得`converter_lite`工具，并配置环境变量。 2. 以TensorFlow Lite模型为例，执行权重量化模型转换命令: - ``` + + ```bash ./converter_lite --fmk=TFLITE --modelFile=Inception_v3.tflite --outputFile=Inception_v3.tflite --quantType=WeightQuant --bitNum=8 --quantSize=0 --convWeightQuantChannelThreshold=0 ``` + 3. 上述命令执行成功后，便可得到量化后的模型`Inception_v3.tflite.ms`，量化后的模型大小通常会下降到FP32模型的1/4。 ### 部分模型精度结果 - | 模型 | 测试数据集 | FP32模型精度 | 权重量化精度 | - | -------- | ------- | ----- | ----- | - | [Inception_V3](https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v3_2018_04_27.tgz) | [ImageNet](http://image-net.org/) | 77.92% | 77.84% | + | 模型 | 测试数据集 | FP32模型精度 | 权重量化精度 | + | -------- | ------- | ----- | ----- | + | [Inception_V3](https://storage.googleapis.com/download.tensorflow.org/models/tflite/model_zoo/upload_20180427/inception_v3_2018_04_27.tgz) | [ImageNet](http://image-net.org/) | 77.92% | 77.84% | | [Mobilenet_V1_1.0_224](https://storage.googleapis.com/download.tensorflow.org/models/mobilenet_v1_2018_02_22/mobilenet_v1_1.0_224.tgz) | [ImageNet](http://image-net.org/) | 70.96% | 70.56% | > 以上所有结果均在x86环境上测得。 @@ -75,13 +79,15 @@ MindSpore Lite训练后量化分为两类： ### 参数说明 全量化转换命令的一般形式为： -``` + +```bash ./converter_lite --fmk=ModelType --modelFile=ModelFilePath --outputFile=ConvertedModelPath --quantType=PostTraining --config_file=config.cfg ``` + 下面对此命令的量化相关参数进行说明： | 参数 | 属性 | 功能描述 | 参数类型 | 默认值 | 取值范围 | -| -------- | ------- | ----- | ----- |----- | ----- | +| -------- | ------- | ----- | ----- |----- | ----- | | `--quantType=` | 必选 | 设置为PostTraining，启用全量化 | String | - | 必须设置为PostTraining | | `--config_file=` | 必选 | 校准数据集配置文件路径 | String | - | - | @@ -99,17 +105,21 @@ MindSpore Lite训练后量化分为两类： 1. 正确编译出`converter_lite`可执行文件。 2. 准备校准数据集，假设存放在`/dir/images`目录，编写配置文件`config.cfg`，内容如下： - ``` + + ```python image_path=/dir/images batch_count=100 method_x=MAX_MIN thread_num=1 ``` + 校准数据集可以选择测试数据集的子集，要求`/dir/images`目录下存放的每个文件均是预处理好的输入数据，每个文件都可以直接用于推理的输入。 3. 以MindSpore模型为例，执行全量化的模型转换命令: - ``` + + ```bash ./converter_lite --fmk=MINDIR --modelFile=lenet.mindir --outputFile=lenet_quant --quantType=PostTraining --config_file=config.cfg ``` + 4. 上述命令执行成功后，便可得到量化后的模型`lenet_quant.ms`，通常量化后的模型大小会下降到FP32模型的1/4。 ### 部分模型精度结果 diff --git a/tutorials/lite/source_zh_cn/use/runtime_cpp.md b/tutorials/lite/source_zh_cn/use/runtime_cpp.md index 029b1c99f90ad9d38ccdbc140d3bbe2aaaa5bccd..70753fd478b55e437317d5a735e4283618231126 100644 --- a/tutorials/lite/source_zh_cn/use/runtime_cpp.md +++ b/tutorials/lite/source_zh_cn/use/runtime_cpp.md @@ -46,6 +46,7 @@ Runtime总体使用流程如下图所示： ![img](../images/side_infer_process.png) 包含的组件及功能如下所述： + - `Model`：MindSpore Lite使用的模型，通过用户构图或直接加载网络，来实例化算子原型的列表。 - `Lite Session`：提供图编译的功能，并调用图执行器进行推理。 - `Scheduler`：算子异构调度器，根据异构调度策略，为每一个算子选择合适的kernel，构造kernel list，并切分子图。 @@ -132,6 +133,7 @@ if (session2 == nullptr) { ### 使用示例 下面代码演示如何对MindSpore Lite的输入进行Resize： + ```cpp // Assume we have created a LiteSession instance named session. auto inputs = session->GetInputs(); @@ -160,6 +162,7 @@ virtual int CompileGraph(lite::Model *model) = 0; ### 使用示例 下面代码演示如何进行图编译: + ```cpp // Assume we have created a LiteSession instance named session and a Model instance named model before. // The methods of creating model and session can refer to "Import Model" and "Create Session" two sections. @@ -249,6 +252,7 @@ memcpy(in_data, input_buf, data_size); ``` 需要注意的是： + - MindSpore Lite的模型输入Tensor中的数据排布必须是NHWC。 - 模型的输入`input_buf`是用户从磁盘读取的，当拷贝给模型输入Tensor以后，用户需要自行释放`input_buf`。 - `GetInputs`和`GetInputsByTensorName`方法返回的vector不需要用户释放。 @@ -296,6 +300,7 @@ session->BindThread(false); ### 回调运行 Mindspore Lite可以在调用`RunGraph`时，传入两个`KernelCallBack`函数指针来回调推理模型，相比于一般的图执行，回调运行可以在运行过程中获取额外的信息，帮助开发者进行性能分析、Bug调试等。额外的信息包括： + - 当前运行的节点名称 - 推理当前节点前的输入输出Tensor - 推理当前节点后的输入输出Tensor @@ -377,6 +382,7 @@ delete (model); MindSpore Lite在执行完推理后，就可以获取模型的推理结果。 MindSpore Lite提供四种方法来获取模型的输出`MSTensor`。 + 1. 使用`GetOutputsByNodeName`方法，根据模型输出节点的名称来获取模型输出`MSTensor`中连接到该节点的Tensor的vector。 ```cpp @@ -501,22 +507,26 @@ for (auto tensor_name : tensor_names) { ``` ## 获取版本号 + MindSpore Lite提供了`Version`方法可以获取版本号，包含在`include/version.h`头文件中，调用该方法可以得到版本号字符串。 ### 使用示例 下面代码演示如何获取MindSpore Lite的版本号： + ```cpp #include "include/version.h" std::string version = mindspore::lite::Version(); ``` ## Session并行 + MindSpore Lite支持多个`LiteSession`并行推理，但不支持多个线程同时调用单个`LiteSession`的`RunGraph`接口。 ### 单Session并行 MindSpore Lite不支持多线程并行执行单个`LiteSession`的推理，否则会得到以下错误信息： + ```cpp ERROR [mindspore/lite/src/lite_session.cc:297] RunGraph] 10 Not support multi-threading ``` @@ -528,6 +538,7 @@ MindSpore Lite支持多个`LiteSession`同时进行推理的场景，每个`Lite ### 使用示例 下面代码演示了如何创建多个`LiteSession`，并且并行执行推理的过程： + ```cpp #include #include "src/common/file_utils.h" diff --git a/tutorials/lite/source_zh_cn/use/runtime_java.md b/tutorials/lite/source_zh_cn/use/runtime_java.md index 2ff2a9396fe359c9ff62aad385b83744f24f0f5b..99407b051b3fe395d7671626a91ddf176629f959 100644 --- a/tutorials/lite/source_zh_cn/use/runtime_java.md +++ b/tutorials/lite/source_zh_cn/use/runtime_java.md @@ -40,14 +40,14 @@ private boolean init(Context context) { Log.e("MS_LITE", "Load Model failed"); return false; } - + // Create and init config. MSConfig msConfig = new MSConfig(); if (!msConfig.init(DeviceType.DT_CPU, 2, CpuBindMode.MID_CPU)) { Log.e("MS_LITE", "Init context failed"); return false; } - + // Create the mindspore lite session. session = new LiteSession(); if (!session.init(msConfig)) { @@ -56,14 +56,14 @@ private boolean init(Context context) { return false; } msConfig.free(); - + // Complile graph. if (!session.compileGraph(model)) { Log.e("MS_LITE", "Compile graph failed"); model.freeBuffer(); return false; } - + // Note: when use model.freeBuffer(), the model can not be complile graph again. model.freeBuffer(); @@ -79,7 +79,7 @@ private void DoInference(Context context) { } byte[] inData = readFileFromAssets(context, "model_inputs.bin"); inTensor.setData(inData); - + // Run graph to infer results. if (!session.runGraph()) { Log.e("MS_LITE", "Run graph failed"); @@ -97,7 +97,7 @@ private void DoInference(Context context) { return; } float[] results = output.getFloatData(); - + // Apply infer results. …… } @@ -109,4 +109,3 @@ private void free() { model.free(); } ``` - diff --git a/tutorials/notebook/README.md b/tutorials/notebook/README.md index cd78a0dd6a7d9ae6f854b071a45931355058c124..555baf04c3dc68bde90231181461c3ad2254fefe 100644 --- a/tutorials/notebook/README.md +++ b/tutorials/notebook/README.md @@ -1,6 +1,7 @@ # MindSpore的教程体验 ## 环境配置 + ### Windows和Linux系统配置方法 - 系统版本：Windows 10，Ubuntu 16.04及以上 @@ -11,17 +12,18 @@ - MindSpore 下载地址：[MindSpore官网下载](https://www.mindspore.cn/versions)，使用Windows系统用户选择Windows-X86版本，使用Linux系统用户选择Ubuntu-X86版本 -> MindSpore的[具体安装教程](https://www.mindspore.cn/install/) - +> MindSpore的[具体安装教程](https://www.mindspore.cn/install/) ### Jupyter Notebook切换conda环境（Kernel Change）的配置方法 - 首先，增加Jupyter Notebook切换conda环境功能（Kernel Change） 启动Anaconda Prompt，输入命令： - ``` + + ```bash conda install nb_conda ``` + > 建议在base环境操作上述命令。 执行完毕，重启Jupyter Notebook即可完成功能添加。 @@ -29,19 +31,25 @@ - 然后，添加conda环境到Jypyter Notebook的Kernel Change中。 1. 新建一个conda环境，启动Anaconda Prompt，输入命令： - ``` + + ```bash conda create -n {env_name} python=3.7.5 ``` + > env_name可以按照自己想要的环境名称自行命名。 2. 激活新环境，输入命令： - ``` + + ```bash conda activate {env_name} ``` + 3. 安装ipykernel，输入命令： - ``` + + ```bash conda install -n {env_name} ipykernel ``` + > 如果添加已有环境，只需执行安装ipykernel操作即可。 执行完毕后，刷新Jupyter notebook页面点击Kernel下拉，选择Kernel Change，就能选择新添加的conda环境。 @@ -49,20 +57,19 @@ ## notebook说明 | 教  程  名  称 | 文  件  名  称 | 教  程  类  别 | 内  容  描  述 -| :----------- | :----------- | :------- |:------ +| :----------- | :----------- | :------- |:------ | 手写数字分类识别入门体验教程 | [quick_start.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/quick_start.ipynb) | 快速入门 | - CPU平台下从数据集到模型验证的全过程解读
- 体验教程中各功能模块的使用说明
- 数据集图形化展示
- 了解LeNet5具体结构和参数作用
- 学习使用自定义回调函数
- loss值与训练步数的变化图
- 模型精度与训练步数的变化图
- 使用模型应用到手写图片的预测与分类上 | 线性拟合 | [linear_regression.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/linear_regression.ipynb) | 快速入门 | - 了解线性拟合的算法原理
- 了解在MindSpore中如何实现线性拟合的算法原理
- 学习使用MindSpore实现AI训练中的正向传播和方向传播
- 可视化线性函数拟合数据的全过程。 -| 加载数据集 | [loading_dataset.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/loading_dataset.ipynb) | 使用指南 | - 学习MindSpore中加载数据集的方法
- 展示加载常用数据集的方法
- 展示加载MindRecord格式数据集的方法
- 展示加载自定义格式数据集的方法 +| 加载数据集 | [loading_dataset.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/loading_dataset.ipynb) | 使用指南 | - 学习MindSpore中加载数据集的方法
- 展示加载常用数据集的方法
- 展示加载MindRecord格式数据集的方法
- 展示加载自定义格式数据集的方法 | 将数据集转换为MindSpore数据格式 | [convert_dataset_to_mindspore_data_format.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/convert_dataset_to_mindspore_data_format/convert_dataset_to_mindspore_data_format.ipynb) | 使用指南 | - 展示将MNIST数据集转换为MindSpore数据格式
- 展示将CSV数据集转换为MindSpore数据格式
- 展示将CIFAR-10数据集转换为MindSpore数据格式
- 展示将CIFAR-100数据集转换为MindSpore数据格式
- 展示将ImageNet数据集转换为MindSpore数据格式
- 展示用户自定义生成MindSpore数据格式 | 数据处理与数据增强 | [data_loading_enhancement.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/data_loading_enhance/data_loading_enhancement.ipynb) | 使用指南 | - 学习MindSpore中数据处理和增强的方法
- 展示数据处理、增强方法的实际操作
- 对比展示数据处理前和处理后的效果
- 表述在数据处理、增强后的意义 -| 自然语言处理应用 | [nlp_application.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/nlp_application.ipynb) | 应用实践 | - 展示MindSpore在自然语言处理的应用
- 展示自然语言处理中数据集特定的预处理方法
- 展示如何定义基于LSTM的SentimentNet网络 +| 自然语言处理应用 | [nlp_application.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/nlp_application.ipynb) | 应用实践 | - 展示MindSpore在自然语言处理的应用
- 展示自然语言处理中数据集特定的预处理方法
- 展示如何定义基于LSTM的SentimentNet网络 | 计算机视觉应用 | [computer_vision_application.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/computer_vision_application.ipynb) | 应用实践 | - 学习MindSpore卷积神经网络在计算机视觉应用的过程
- 学习下载CIFAR-10数据集，搭建运行环境
- 学习使用ResNet-50构建卷积神经网络
- 学习使用Momentum和SoftmaxCrossEntropyWithLogits构建优化器和损失函数
- 学习调试参数训练模型，判断模型精度 -| 模型的训练及验证同步方法 | [evaluate_the_model_during_training.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/evaluate_the_model_during_training.ipynb) | 应用实践 | - 了解模型训练和验证同步进行的方法
- 学习同步训练和验证中参数设置方法
- 利用绘图函数从保存的模型中挑选出最优模型 +| 模型的训练及验证同步方法 | [evaluate_the_model_during_training.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/evaluate_the_model_during_training.ipynb) | 应用实践 | - 了解模型训练和验证同步进行的方法
- 学习同步训练和验证中参数设置方法
- 利用绘图函数从保存的模型中挑选出最优模型 | 优化数据准备的性能 | [optimize_the_performance_of_data_preparation.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/optimize_the_performance_of_data_preparation/optimize_the_performance_of_data_preparation.ipynb) | 应用实践 | - 数据加载性能优化
- shuffle性能优化
- 数据增强性能优化
- 性能优化方案总结 | 使用PyNative进行神经网络的训练调试体验 | [debugging_in_pynative_mode.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/debugging_in_pynative_mode.ipynb) | 模型调优 | - GPU平台下从数据集获取单个数据进行单个step训练的数据变化全过程解读
- 了解PyNative模式下的调试方法
- 图片数据在训练过程中的变化情况的图形展示
- 了解构建权重梯度计算函数的方法
- 展示1个step过程中权重的变化及数据展示 | 自定义调试信息体验文档 | [custom_debugging_info.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/custom_debugging_info.ipynb) | 模型调优 | - 了解MindSpore的自定义调试算子
- 学习使用自定义调试算子Callback设置定时训练
- 学习设置metrics算子输出相对应的模型精度信息
- 学习设置日志环境变量来控制glog输出日志 | MindInsight的溯源分析和对比分析 | [lineage_and_scalars_comparision.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/mindinsight/lineage_and_scalars_comparision.ipynb) | 模型调优 | - 了解MindSpore中训练数据的采集及展示
- 学习使用回调函数SummaryCollector进行数据采集
- 使用MindInsight进行数据可视化
- 了解数据溯源和模型溯源的使用方法
- 了解对比分析的使用方法 -| 计算图和数据图可视化 | [calculate_and_datagraphic.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/mindinsight/calculate_and_datagraphic.ipynb) | 模型调优 | - 了解MindSpore中新增可视化功能
- 学习使用MindInsight可视化看板
- 学习使用查看计算图可视化图的信息的方法
- 学习使用查看数据图中展示的信息的方法 -| 标量、直方图、图像和张量可视化 | [mindinsight_image_histogram_scalar_tensor.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/mindinsight/mindinsight_image_histogram_scalar_tensor.ipynb) | 模型调优 | - 了解完整的MindSpore深度学习及MindInsight可视化展示的过程
- 学习使用MindInsight对训练过程中标量、直方图、图像和张量信息进行可视化展示
- 学习使用Summary算子记录标量、直方图、图像和张量信息
- 学习单独对标量、直方图、图像和张量信息进行记录并可视化展示的方法 -| 混合精度 | [mixed_precision.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/mixed_precision.ipynb) | 性能优化 | - 了解混合精度训练的原理
- 学习在MindSpore中使用混合精度训练
- 对比单精度训练和混合精度训练的对模型训练的影响 +| MindInsight训练看板 | [mindinsight_dashboard.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/mindinsight/mindinsight_dashboard.ipynb) | 模型调优 | - 了解完整的MindSpore深度学习及MindInsight可视化展示的过程
- 学习使用MindInsight对训练过程中标量、直方图、图像、计算图、数据图和张量信息进行可视化展示
- 学习使用Summary算子记录标量、直方图、图像、计算图、数据图和张量信息 +| 混合精度 | [mixed_precision.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/mixed_precision.ipynb) | 性能优化 | - 了解混合精度训练的原理
- 学习在MindSpore中使用混合精度训练
- 对比单精度训练和混合精度训练的对模型训练的影响 | 模型安全 | [model_security.ipynb](https://gitee.com/mindspore/docs/blob/master/tutorials/notebook/model_security.ipynb) | AI安全和隐私 | - 了解AI算法的安全威胁的概念和影响
- 介绍MindArmour提供的模型安全防护手段
- 学习如何模拟攻击训练模型
- 学习针对被攻击模型进行对抗性防御 diff --git a/tutorials/notebook/mindinsight/calculate_and_datagraphic.ipynb b/tutorials/notebook/mindinsight/calculate_and_datagraphic.ipynb deleted file mode 100644 index 5640befc4b5429dcabed1b4235399f7c1c80bfe4..0000000000000000000000000000000000000000 --- a/tutorials/notebook/mindinsight/calculate_and_datagraphic.ipynb +++ /dev/null @@ -1,422 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#
计算图和数据图可视化
\n", - "\n", - "\n", - "## 计算图与数据图概述\n", - "\n", - "计算图的生成是通过将模型训练过程中的每个计算节点关联后所构成的，初体验者可以通过查看计算图，掌握整个模型的计算走向结构，数据流以及控制流的信息。对于高阶的使用人员，能够通过计算图验证计算节点的输入输出是否正确，并验证整个计算过程是否符合预期。数据图展示的是数据预处理的过程，在MindInsight可视化面板中可查看数据处理的图，能够更加直观地查看数据预处理的每一个环节，并帮助提升模型性能。\n", - "\n", - "接下来我们用一个图片分类的项目来体验计算图与数据图的生成与使用。\n", - " \n", - "## 本次体验的整体流程\n", - "1. 体验模型的数据选择使用MNIST数据集，MNIST数据集整体数据量比较小，更适合体验使用。\n", - "\n", - "2. 初始化一个网络，本次的体验使用LeNet网络。\n", - "\n", - "3. 增加可视化功能的使用，并设定只记录计算图与数据图。\n", - "\n", - "4. 加载训练数据集并进行训练，训练完成后，查看结果并保存模型文件。\n", - "\n", - "5. 启用MindInsight的可视化图界面，进行训练过程的核对。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 数据集来源\n", - "\n", - "方法一\n", - "\n", - "从以下网址下载，并将数据包解压后放在Jupyter的工作目录下。\n", - "\n", - "- 训练数据集：{\"\",\"\"}\n", - "- 测试数据集：{\"\",\"\"}\n", - "\n", - "可执行下面代码查看Jupyter的工作目录。" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "os.getcwd()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- 训练数据集放在----`Jupyter工作目录+\\MNIST_Data\\train\\`，此时`train`文件夹内应该包含两个文件，`train-images-idx3-ubyte`和`train-labels-idx1-ubyte` \n", - "- 测试数据集放在----`Jupyter工作目录+\\MNIST_Data\\test\\`，此时`test`文件夹内应该包含两个文件，`t10k-images-idx3-ubyte`和`t10k-labels-idx1-ubyte`" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "方法二\n", - "\n", - "直接执行以下代码，会自动进行训练集的下载与解压，但是整个过程根据网络好坏情况会需要花费几分钟时间。" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import gzip\n", - "import urllib.request\n", - "from urllib.parse import urlparse\n", - "\n", - "\n", - "def unzip_file(gzip_path):\n", - " \"\"\"\n", - " Unzip a given gzip file.\n", - "\n", - " Args:\n", - " gzip_path (str): The gzip file path\n", - " \"\"\"\n", - " open_file = open(gzip_path.replace('.gz', ''), 'wb')\n", - " gz_file = gzip.GzipFile(gzip_path)\n", - " open_file.write(gz_file.read())\n", - " gz_file.close()\n", - "\n", - "\n", - "def download_dataset():\n", - " \"\"\"Download the dataset from http://yann.lecun.com/exdb/mnist/.\"\"\"\n", - " print(\"******Downloading the MNIST dataset******\")\n", - " train_path = \"./MNIST_Data/train/\"\n", - " test_path = \"./MNIST_Data/test/\"\n", - " train_path_check = os.path.exists(train_path)\n", - " test_path_check = os.path.exists(test_path)\n", - " if not train_path_check and not test_path_check:\n", - " os.makedirs(train_path)\n", - " os.makedirs(test_path)\n", - " train_url = {\"http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\",\n", - " \"http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\"}\n", - " test_url = {\"http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\",\n", - " \"http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\"}\n", - "\n", - " for url in train_url:\n", - " url_parse = urlparse(url)\n", - " # split the file name from url\n", - " file_name = os.path.join(train_path,url_parse.path.split('/')[-1])\n", - " if not os.path.exists(file_name.replace('.gz', '')):\n", - " file = urllib.request.urlretrieve(url, file_name)\n", - " unzip_file(file_name)\n", - " os.remove(file_name)\n", - " \n", - " for url in test_url:\n", - " url_parse = urlparse(url)\n", - " # split the file name from url\n", - " file_name = os.path.join(test_path,url_parse.path.split('/')[-1])\n", - " if not os.path.exists(file_name.replace('.gz', '')):\n", - " file = urllib.request.urlretrieve(url, file_name)\n", - " unzip_file(file_name)\n", - " os.remove(file_name)\n", - "\n", - "download_dataset()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 数据增强\n", - "对数据集进行数据增强操作，可以提升模型精度。\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import mindspore.dataset as ds\n", - "import mindspore.dataset.vision.c_transforms as CV\n", - "import mindspore.dataset.transforms.c_transforms as C\n", - "from mindspore.dataset.vision import Inter\n", - "from mindspore.common import dtype as mstype\n", - "\n", - "\n", - "def create_dataset(data_path, batch_size=32, repeat_size=1,\n", - " num_parallel_workers=1):\n", - " \"\"\"\n", - " Create dataset for train or test.\n", - "\n", - " Args:\n", - " data_path (str): The absolute path of the dataset\n", - " batch_size (int): The number of data records in each group\n", - " repeat_size (int): The number of replicated data records\n", - " num_parallel_workers (int): The number of parallel workers\n", - " \"\"\"\n", - " # define dataset\n", - " mnist_ds = ds.MnistDataset(data_path)\n", - "\n", - " # define some parameters needed for data enhancement and rough justification\n", - " resize_height, resize_width = 32, 32\n", - " rescale = 1.0 / 255.0\n", - " shift = 0.0\n", - " rescale_nml = 1 / 0.3081\n", - " shift_nml = -1 * 0.1307 / 0.3081\n", - "\n", - " # according to the parameters, generate the corresponding data enhancement method\n", - " resize_op = CV.Resize((resize_height, resize_width), interpolation=Inter.LINEAR)\n", - " rescale_nml_op = CV.Rescale(rescale_nml, shift_nml)\n", - " rescale_op = CV.Rescale(rescale, shift)\n", - " hwc2chw_op = CV.HWC2CHW()\n", - " type_cast_op = C.TypeCast(mstype.int32)\n", - "\n", - " # using map method to apply operations to a dataset\n", - " mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns=\"label\", num_parallel_workers=num_parallel_workers)\n", - " mnist_ds = mnist_ds.map(operations=resize_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - " mnist_ds = mnist_ds.map(operations=rescale_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - " mnist_ds = mnist_ds.map(operations=rescale_nml_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - " mnist_ds = mnist_ds.map(operations=hwc2chw_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - " \n", - " # process the generated dataset\n", - " buffer_size = 10000\n", - " mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size) # 10000 as in LeNet train script\n", - " mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)\n", - " mnist_ds = mnist_ds.repeat(repeat_size)\n", - "\n", - " return mnist_ds" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 可视化操作流程\n", - "\n", - "1. 准备训练脚本，在训练脚本中指定计算图的超参数信息，使用`Summary`保存到日志中，接着再运行训练脚本。\n", - "\n", - "2. 启动MindInsight，启动成功后，就可以通过访问命令执行后显示的地址，查看可视化界面。\n", - "\n", - "3. 访问可视化地址成功后，就可以对图界面进行查询等操作。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 初始化网络\n", - "\n", - "1. 导入构建网络所使用的模块。\n", - "\n", - "2. 构建初始化参数的函数。\n", - "\n", - "3. 创建网络，在网络中设置参数。" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import mindspore.nn as nn\n", - "from mindspore.common.initializer import TruncatedNormal\n", - "\n", - "\n", - "def conv(in_channels, out_channels, kernel_size, stride=1, padding=0):\n", - " \"\"\"weight initial for conv layer\"\"\"\n", - " weight = weight_variable()\n", - " return nn.Conv2d(in_channels, out_channels,\n", - " kernel_size=kernel_size, stride=stride, padding=padding,\n", - " weight_init=weight, has_bias=False, pad_mode=\"valid\")\n", - "\n", - "\n", - "def fc_with_initialize(input_channels, out_channels):\n", - " \"\"\"weight initial for fc layer\"\"\"\n", - " weight = weight_variable()\n", - " bias = weight_variable()\n", - " return nn.Dense(input_channels, out_channels, weight, bias)\n", - "\n", - "\n", - "def weight_variable():\n", - " \"\"\"weight initial\"\"\"\n", - " return TruncatedNormal(0.02)\n", - "\n", - "\n", - "class LeNet5(nn.Cell):\n", - " \n", - " def __init__(self, num_class=10, channel=1):\n", - " super(LeNet5, self).__init__()\n", - " self.num_class = num_class\n", - " self.conv1 = conv(channel, 6, 5)\n", - " self.conv2 = conv(6, 16, 5)\n", - " self.fc1 = fc_with_initialize(16 * 5 * 5, 120)\n", - " self.fc2 = fc_with_initialize(120, 84)\n", - " self.fc3 = fc_with_initialize(84, self.num_class)\n", - " self.relu = nn.ReLU()\n", - " self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n", - " self.flatten = nn.Flatten()\n", - "\n", - " def construct(self, x):\n", - " x = self.conv1(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv2(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.flatten(x)\n", - " x = self.fc1(x)\n", - " x = self.relu(x)\n", - " x = self.fc2(x)\n", - " x = self.relu(x)\n", - " x = self.fc3(x)\n", - " return x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 执行训练\n", - "\n", - "1. 导入所需的代码包，并示例化训练网络。\n", - "2. 通过MindSpore提供的 `SummaryCollector` 接口，实现收集计算图和数据图。在实例化 `SummaryCollector` 时，在 `collect_specified_data` 参数中，通过设置 `collect_graph` 指定收集计算图，设置 `collect_dataset_graph` 指定收集数据图。\n", - "\n", - "更多 `SummaryCollector` 的用法，请点击[API文档](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector)查看。\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import mindspore.nn as nn\n", - "from mindspore import context\n", - "from mindspore.train import Model\n", - "from mindspore.nn.metrics import Accuracy\n", - "from mindspore.train.callback import LossMonitor, SummaryCollector\n", - "\n", - "if __name__ == \"__main__\":\n", - " device_target = \"CPU\"\n", - " \n", - " context.set_context(mode=context.GRAPH_MODE, device_target=device_target)\n", - " download_dataset()\n", - " ds_train = create_dataset(data_path=\"./MNIST_Data/train/\")\n", - "\n", - " network = LeNet5()\n", - " net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n", - " net_opt = nn.Momentum(network.trainable_params(), learning_rate=0.01, momentum=0.9)\n", - " model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", - "\n", - " specified={'collect_graph': True, 'collect_dataset_graph': True}\n", - " summary_collector = SummaryCollector(summary_dir='./summary_dir', collect_specified_data=specified, collect_freq=1, keep_default_action=False)\n", - " \n", - " print(\"============== Starting Training ==============\")\n", - " model.train(epoch=2, train_dataset=ds_train, callbacks=[LossMonitor(), summary_collector], dataset_sink_mode=False)\n", - "\n", - " print(\"============== Starting Testing ==============\")\n", - " ds_eval = create_dataset(\"./MNIST_Data/test/\")\n", - " acc = model.eval(ds_eval, dataset_sink_mode=False)\n", - " print(\"============== {} ==============\".format(acc))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 启动MindInsight\n", - "- 启动MindInsigh服务命令：`mindinsigh start --summary-base-dir=/path/ --port=8080`；\n", - "- 执行完服务命令后，访问给出的地址，查看MindInsigh可视化结果。\n", - "\n", - "> 其中 /path/ 为 `SummaryCollector` 中参数 `summary_dir` 所指定的目录。\n", - "\n", - "![title](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/mindinsight_map.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 计算图信息\n", - "- 文本选择框：输入计算图对应的路径及文件名，显示相应的计算图，便于查找文件。\n", - "- 搜索框：可以对整体计算图的节点信息进行搜索，输入完整的节点名称，回车执行搜索，如果有该名称节点，就会呈现出来，便于查找节点。\n", - "- 缩略图：展示整体计算图的缩略情况，在面板左边查看详细图结构时，在缩略图处会有定位，显示当前查看的位置在整体计算图中的定位，实时呈现部分与整体的关系。\n", - "- 节点信息：显示当前所查看节点的信息，包括名称、类型、属性、输入和输出。便于在训练结束后，核对计算正确性时查看。\n", - "- 图例：图例中包括命名空间、聚合节点、虚拟节点、算子节点、常量节点，通过不同图形来区分。\n", - "\n", - "![title](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/cast_map.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 数据图展示\n", - "\n", - "数据图展示了数据增强中对数据进行操作的流程。\n", - "\n", - "1. 首先是从加载数据集 `mnist_ds = ds.MnistDataset(data_path)` 开始，对应数据图中 `MnistDataset`。\n", - "\n", - "2. 下面代码为上面的 `create_dataset` 函数中作数据预处理与数据增强的相关操作。可以从数据图中清晰地看到数据处理的流程。通过查看数据图，可以帮助分析是否存在不恰当的数据处理流程。\n", - "\n", - "```\n", - "mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns=\"label\", num_parallel_workers=num_parallel_workers)\n", - "mnist_ds = mnist_ds.map(operations=resize_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - "mnist_ds = mnist_ds.map(operations=rescale_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - "mnist_ds = mnist_ds.map(operations=rescale_nml_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - "mnist_ds = mnist_ds.map(operations=hwc2chw_op, input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n", - "\n", - "mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size) # 10000 as in LeNet train script\n", - "mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)\n", - "mnist_ds = mnist_ds.repeat(repeat_size)\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![title](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/data_map.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 关闭MindInsight\n", - "\n", - "- 查看完成后，在命令行中可执行此命令 `mindinsight stop --port=8080`，关闭MindInsight。" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.6" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/tutorials/notebook/mindinsight/images/caculate_graph.png b/tutorials/notebook/mindinsight/images/caculate_graph.png new file mode 100644 index 0000000000000000000000000000000000000000..460e56104705421002cf5d6112a84322b1ace391 Binary files /dev/null and b/tutorials/notebook/mindinsight/images/caculate_graph.png differ diff --git a/tutorials/notebook/mindinsight/images/cast_map.png b/tutorials/notebook/mindinsight/images/cast_map.png deleted file mode 100644 index 4713ffb03f6e03c02c45d41b63fcc6bdd7d8b2ae..0000000000000000000000000000000000000000 Binary files a/tutorials/notebook/mindinsight/images/cast_map.png and /dev/null differ diff --git a/tutorials/notebook/mindinsight/images/data_function.png b/tutorials/notebook/mindinsight/images/data_function.png new file mode 100644 index 0000000000000000000000000000000000000000..123d4cefd29b84cf261ccab8c8a62f63b06aea43 Binary files /dev/null and b/tutorials/notebook/mindinsight/images/data_function.png differ diff --git a/tutorials/notebook/mindinsight/images/data_map.png b/tutorials/notebook/mindinsight/images/data_map.png deleted file mode 100644 index 841346ff6256bdc81c09eb9e8f7a32a364d838b3..0000000000000000000000000000000000000000 Binary files a/tutorials/notebook/mindinsight/images/data_map.png and /dev/null differ diff --git a/tutorials/notebook/mindinsight/images/graph_sidebar.png b/tutorials/notebook/mindinsight/images/graph_sidebar.png new file mode 100644 index 0000000000000000000000000000000000000000..4b9b6097aa62fdf426d6fc62ee1dd55f8086aeb9 Binary files /dev/null and b/tutorials/notebook/mindinsight/images/graph_sidebar.png differ diff --git a/tutorials/notebook/mindinsight/images/histogram_only.png b/tutorials/notebook/mindinsight/images/histogram_only.png deleted file mode 100644 index 70e59386856dd370782e037425e56efbe1ebf3b9..0000000000000000000000000000000000000000 Binary files a/tutorials/notebook/mindinsight/images/histogram_only.png and /dev/null differ diff --git a/tutorials/notebook/mindinsight/images/histogram_only_all.png b/tutorials/notebook/mindinsight/images/histogram_only_all.png deleted file mode 100644 index c62873dd5e6ca4cf8e06b9ee642a8bd7e6d5ac85..0000000000000000000000000000000000000000 Binary files a/tutorials/notebook/mindinsight/images/histogram_only_all.png and /dev/null differ diff --git a/tutorials/notebook/mindinsight/images/image_only.png b/tutorials/notebook/mindinsight/images/image_only.png deleted file mode 100644 index fdbc96981316fc3c8b977c24d3a93474f4df1cbe..0000000000000000000000000000000000000000 Binary files a/tutorials/notebook/mindinsight/images/image_only.png and /dev/null differ diff --git a/tutorials/notebook/mindinsight/images/image_panel.png b/tutorials/notebook/mindinsight/images/image_panel.png index 64963ff989d6fd743166bd8630a3feade2fa01cb..451a3fea4ecfabb224f0f1cc90ca5e5d617d9bf1 100644 Binary files a/tutorials/notebook/mindinsight/images/image_panel.png and b/tutorials/notebook/mindinsight/images/image_panel.png differ diff --git a/tutorials/notebook/mindinsight/images/mindinsight_panel.png b/tutorials/notebook/mindinsight/images/mindinsight_panel.png index 8eb80073b47556ea1759bc44b3b02b0e8f5ed022..d93921429b05195b38b491040e9abcc99fb4994d 100644 Binary files a/tutorials/notebook/mindinsight/images/mindinsight_panel.png and b/tutorials/notebook/mindinsight/images/mindinsight_panel.png differ diff --git a/tutorials/notebook/mindinsight/images/mindinsight_panel2.png b/tutorials/notebook/mindinsight/images/mindinsight_panel2.png index 7d4d858c3102b790ee14b5e706bfeb3cd6c10062..355d4ce219bb2e2766eef80928838f4a11976c46 100644 Binary files a/tutorials/notebook/mindinsight/images/mindinsight_panel2.png and b/tutorials/notebook/mindinsight/images/mindinsight_panel2.png differ diff --git a/tutorials/notebook/mindinsight/images/multi_scalars.png b/tutorials/notebook/mindinsight/images/multi_scalars.png deleted file mode 100644 index af8e1d26d94fdf5240b3f76ae070661be745c909..0000000000000000000000000000000000000000 Binary files a/tutorials/notebook/mindinsight/images/multi_scalars.png and /dev/null differ diff --git a/tutorials/notebook/mindinsight/images/scalar_panel.png b/tutorials/notebook/mindinsight/images/scalar_panel.png index 36d8e5a7206c71c1fc8697c465bdeee923122f84..55f49fb639f126fa823f84cd4b0e2945f85c86f9 100644 Binary files a/tutorials/notebook/mindinsight/images/scalar_panel.png and b/tutorials/notebook/mindinsight/images/scalar_panel.png differ diff --git a/tutorials/notebook/mindinsight/images/tensor.png b/tutorials/notebook/mindinsight/images/tensor.png index 26346a185960ad391ec86476e7fb8823aab289f9..581e5895ffa77280cfb48245f38709107bf8c7cb 100644 Binary files a/tutorials/notebook/mindinsight/images/tensor.png and b/tutorials/notebook/mindinsight/images/tensor.png differ diff --git a/tutorials/notebook/mindinsight/images/tensor_func.png b/tutorials/notebook/mindinsight/images/tensor_func.png index 3db1f6c55e56edd88633168296f8cbc954cd0fff..df549a8e628bdc8d2ffef9dd814e5a8cf0f3ba34 100644 Binary files a/tutorials/notebook/mindinsight/images/tensor_func.png and b/tutorials/notebook/mindinsight/images/tensor_func.png differ diff --git a/tutorials/notebook/mindinsight/images/tensor_only.png b/tutorials/notebook/mindinsight/images/tensor_only.png deleted file mode 100644 index bdd8290c573341b31b11ccc7792178f3296274d0..0000000000000000000000000000000000000000 Binary files a/tutorials/notebook/mindinsight/images/tensor_only.png and /dev/null differ diff --git a/tutorials/notebook/mindinsight/mindinsight_dashboard.ipynb b/tutorials/notebook/mindinsight/mindinsight_dashboard.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..d080d205fd157e7ebd778440ebbbd88afac39507 --- /dev/null +++ b/tutorials/notebook/mindinsight/mindinsight_dashboard.ipynb @@ -0,0 +1,846 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# MindInsight训练看板\n", + "\n", + "通过MindSpore可以将训练过程中的标量、图像、参数分布直方图、张量、计算图和数据图记录到summary日志文件中，并通过MindInsight提供的可视化界面进行查看。\n", + "\n", + "- 通过查看特定的标量数值随着训练步骤的变化趋势，比如查看每个迭代的损失值、正确率、准确率这些标量的变化过程，追踪神经网络在整个训练过程中的信息，帮助用户了解模型是否过拟合，或者是否训练了过长时间。可以通过比较不同训练中的这些指标，以帮助调试和改善模型。\n", + "\n", + "- 通过查看训练过程中的图像数据，用户可以查看每个步骤所使用的数据集图像。\n", + "\n", + "- 参数分布直方图支持以直方图的形式呈现Tensor的变化趋势，用户可以查看训练过程中每个训练步骤的权重、bias和梯度参数变化信息。\n", + "\n", + "- 张量可视能够帮助用户直观查看训练过程中某个步骤的Tensor值，Tensor包括权重值、梯度值、激活值等。\n", + "\n", + "- 计算图的生成是通过将模型训练过程中的每个计算节点关联后所构成的，用户可以通过查看计算图，掌握整个模型的计算走向结构，数据流以及控制流的信息。对于高阶的使用人员，能够通过计算图验证计算节点的输入输出是否正确，并验证整个计算过程是否符合预期。\n", + "\n", + "- 数据图展示的是数据预处理的过程，在MindInsight可视化面板中可查看数据处理的图，能够更加直观地查看数据预处理的每一个环节，并帮助提升模型性能。\n", + "\n", + "接下来是本次流程的体验过程。\n", + "\n", + "## 整体流程\n", + "\n", + "1. 下载CIFAR-10二进制格式数据集。\n", + "2. 对数据进行预处理。\n", + "3. 定义AlexNet网络，在网络中使用summary算子记录数据。\n", + "4. 训练网络，使用 `SummaryCollector` 记录损失值标量、权重梯度、计算图和数据图参数。同时启动MindInsight服务，实时查看损失值、参数直方图、输入图像、张量、计算图和数据图的变化。\n", + "5. 完成训练后，查看MindInsight看板中记录到的损失值标量、直方图、图像信息、张量、计算图、数据图信息。\n", + "6. 相关注意事项，关闭MindInsight服务。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 准备环节\n", + "\n", + "### 下载数据集\n", + "\n", + "本次流程使用CIFAR-10二进制格式数据集，下载地址为：。\n", + "\n", + "CIFAR-10二进制格式数据集包含10个类别的60000个32x32彩色图像。每个类别6000个图像，包含50000张训练图像和10000张测试图像。数据集分为5个训练批次和1个测试批次，每个批次具有10000张图像。测试批次包含每个类别中1000个随机选择的图像，训练批次按随机顺序包含剩余图像（某个训练批次包含的一类图像可能比另一类更多）。其中，每个训练批次精确地包含对应每个类别的5000张图像。\n", + "\n", + "执行下面一段代码下载CIFAR-10二进制格式数据集到当前工作目录，如果已经下载过数据集，则不重复下载。" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "********Checking DataSets Path.*********\n", + "*****Downloading CIFAR-10 DataSets.*****\n", + "*********data_batch_1.bin is ok*********\n", + "*********data_batch_2.bin is ok*********\n", + "*********data_batch_3.bin is ok*********\n", + "*********data_batch_4.bin is ok*********\n", + "*********data_batch_5.bin is ok*********\n", + "**********test_batch.bin is ok**********\n", + "*Downloaded CIFAR-10 DataSets Already.**\n" + ] + } + ], + "source": [ + "import os, shutil\n", + "import urllib.request\n", + "from urllib.parse import urlparse\n", + "\n", + "\n", + "def callbackfunc(blocknum, blocksize, totalsize):\n", + " percent = 100.0 * blocknum * blocksize / totalsize\n", + " if percent > 100:\n", + " percent = 100\n", + " print(\"downloaded {:.1f}\".format(percent), end=\"\\r\")\n", + "\n", + "def _download_dataset():\n", + " ds_url = \"https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz\"\n", + " file_base_name = urlparse(ds_url).path.split(\"/\")[-1]\n", + " file_name = os.path.join(\"./datasets\", file_base_name)\n", + " if not os.path.exists(file_name):\n", + " urllib.request.urlretrieve(ds_url, file_name, callbackfunc)\n", + " print(\"{:*^40}\".format(\"DataSets Downloaded\"))\n", + " shutil.unpack_archive(file_name, extract_dir=\"./datasets/cifar-10-binary\")\n", + "\n", + "def _copy_dataset(ds_part, dest_path):\n", + " data_source_path = \"./datasets/cifar-10-binary/cifar-10-batches-bin\"\n", + " ds_part_source_path = os.path.join(data_source_path, ds_part)\n", + " if not os.path.exists(ds_part_source_path):\n", + " _download_dataset()\n", + " shutil.copy(ds_part_source_path, dest_path)\n", + "\n", + "def download_cifar10_dataset():\n", + " ds_base_path = \"./datasets/cifar10\"\n", + " train_path = os.path.join(ds_base_path, \"train\")\n", + " test_path = os.path.join(ds_base_path, \"test\")\n", + " print(\"{:*^40}\".format(\"Checking DataSets Path.\"))\n", + " if not os.path.exists(train_path) and not os.path.exists(test_path):\n", + " os.makedirs(train_path)\n", + " os.makedirs(test_path)\n", + " print(\"{:*^40}\".format(\"Downloading CIFAR-10 DataSets.\"))\n", + " for i in range(1, 6):\n", + " train_part = \"data_batch_{}.bin\".format(i)\n", + " if not os.path.exists(os.path.join(train_path, train_part)):\n", + " _copy_dataset(train_part, train_path)\n", + " pops = train_part + \" is ok\"\n", + " print(\"{:*^40}\".format(pops))\n", + " test_part = \"test_batch.bin\"\n", + " if not os.path.exists(os.path.join(test_path, test_part)):\n", + " _copy_dataset(test_part, test_path)\n", + " print(\"{:*^40}\".format(test_part+\" is ok\"))\n", + " print(\"{:*^40}\".format(\"Downloaded CIFAR-10 DataSets Already.\"))\n", + "\n", + "download_cifar10_dataset()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "下载数据集后，CIFAR-10数据集目录（`datasets`）结构如下所示。\n", + "\n", + "```shell\n", + " $ tree datasets\n", + " datasets\n", + " └── cifar-10-batches-bin\n", + " ├── test\n", + " │   └── test_batch.bin\n", + " └── train\n", + " ├── data_batch_1.bin\n", + " ├── data_batch_2.bin\n", + " ├── data_batch_3.bin\n", + " ├── data_batch_4.bin\n", + " └── data_batch_5.bin\n", + "\n", + "```\n", + "\n", + "其中：\n", + "- `test_batch.bin`文件为测试数据集文件。\n", + "- `data_batch_1.bin`文件为第1批次训练数据集文件。\n", + "- `data_batch_2.bin`文件为第2批次训练数据集文件。\n", + "- `data_batch_3.bin`文件为第3批次训练数据集文件。\n", + "- `data_batch_4.bin`文件为第4批次训练数据集文件。\n", + "- `data_batch_5.bin`文件为第5批次训练数据集文件。\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 数据处理\n", + "\n", + "好的数据集可以有效提高训练精度和效率，在加载数据集前，会进行一些处理，增加数据的可用性和随机性。下面一段代码定义函数`create_dataset_cifar10`来进行数据处理操作，并创建训练数据集(`ds_train`)和测试数据集(`ds_eval`)。\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "import mindspore.dataset as ds\n", + "import mindspore.dataset.transforms.c_transforms as C\n", + "import mindspore.dataset.vision.c_transforms as CV\n", + "from mindspore.common import dtype as mstype\n", + "\n", + "\n", + "def create_dataset_cifar10(data_path, batch_size=32, repeat_size=1, status=\"train\"):\n", + " \"\"\"\n", + " create dataset for train or test\n", + " \"\"\"\n", + " cifar_ds = ds.Cifar10Dataset(data_path)\n", + " rescale = 1.0 / 255.0\n", + " shift = 0.0\n", + "\n", + " resize_op = CV.Resize(size=(227, 227))\n", + " rescale_op = CV.Rescale(rescale, shift)\n", + " normalize_op = CV.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))\n", + " if status == \"train\":\n", + " random_crop_op = CV.RandomCrop([32, 32], [4, 4, 4, 4])\n", + " random_horizontal_op = CV.RandomHorizontalFlip()\n", + " channel_swap_op = CV.HWC2CHW()\n", + " typecast_op = C.TypeCast(mstype.int32)\n", + " cifar_ds = cifar_ds.map(operations=typecast_op, input_columns=\"label\")\n", + " if status == \"train\":\n", + " cifar_ds = cifar_ds.map(operations=random_crop_op, input_columns=\"image\")\n", + " cifar_ds = cifar_ds.map(operations=random_horizontal_op, input_columns=\"image\")\n", + " cifar_ds = cifar_ds.map(operations=resize_op, input_columns=\"image\")\n", + " cifar_ds = cifar_ds.map(operations=rescale_op, input_columns=\"image\")\n", + " cifar_ds = cifar_ds.map(operations=normalize_op, input_columns=\"image\")\n", + " cifar_ds = cifar_ds.map(operations=channel_swap_op, input_columns=\"image\")\n", + "\n", + " cifar_ds = cifar_ds.shuffle(buffer_size=1000)\n", + " cifar_ds = cifar_ds.batch(batch_size, drop_remainder=True)\n", + " cifar_ds = cifar_ds.repeat(repeat_size)\n", + " return cifar_ds\n", + "\n", + "ds_train = create_dataset_cifar10(data_path=\"./datasets/cifar10/train\")\n", + "ds_eval = create_dataset_cifar10(\"./datasets/cifar10/test\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 抽取数据集图像\n", + "\n", + "执行以下一段代码，抽取上步创建好的训练数据集`ds_train`中第一个`batch`的32张图像以及对应的类别名称进行展示。" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The 32 images with label of the first batch in ds_train are showed below:\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsYAAAHdCAYAAAAekb9wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAABYlAAAWJQFJUiTwAAEAAElEQVR4nOz96a9k2ZrmCf3WsAeb7Ux+/Lh7jPdG3rw5VFNdhVJUQ1PVQrSgoPlQ4gOivyGQ+CP4xh8AEjRCCAES8BEVNIgugbqKrq6qrMxb1TnczDtFxI3w8czH5j2tgQ9rbTM7x91jcD8ecbPyvCELO262bdu2vdde61nPet7nFd577uIu7uIu7uIu7uIu7uIu/rqH/L4P4C7u4i7u4i7u4i7u4i7u4jch7oDxXdzFXdzFXdzFXdzFXdwFd8D4Lu7iLu7iLu7iLu7iLu4CuAPGd3EXd3EXd3EXd3EXd3EXwB0wvou7uIu7uIu7uIu7uIu7AO6A8V3cxV3cxV3cxV3cxV3cBXAHjO/iLu7iLu7iLu7iLu7iLoA7YHwXd3EXd3EXd3EXd3EXdwHcAeO7uIu7uIu7uIu7uIu7uAvgDhjfxV3cxV3cxV3cxV3cxV0Ad8D4Lu7iLu7iLu7iLu7iLu4CuAPGd3EXd3EXd3EXd3EXd3EXwB0wvou7uIu7uIu7uIu7uIu7AEDfxk6EEP429vNvanjvxbf9zK2d0xT4GDgAesAl8Bx4Bmx9Q5pI/sF//9/nP/yf/I8Y7HWYV1f8r/9X/xH/yf/1X+Dqb3AoGfBjYFfAiQ/fcfVtDlTAg0/gg48hSUCqcHzeg7XgbPg7hv8X/+9vfU7hrq1+XXyvbfXf0HiTcwp35/Xr4m3bqgR2gAeDHv/lv/v3+Hf+W3+f+w8eIpoGu2ooZw31yqF1D4/ms1/8CX/2r/8zjF1y+P59Pv6d3+Vv/rv/TR598nv84s//nD/6z/8p/5f/8/+BX5/8+qXv7Wc5/53/9n/AP/gf/od0ez0mVxP++F/8c/5f//D/xq+ehO1Trfh3/s4f8Pf/u/89fve3/wbvP/wIYSXnp+f85I/+Jf/H/9P/lj/74ud0pOJ3Pv4h//W/+9/g7/29fx/vLH/6Jz/hT3/yz/nTn/whny8LNNAhDPAKaE9UDSyABhgA/UxyXjvaLv7u/r/9eNP7/wf/s8ceKRFaI5WOF9EDAoHcXNRXfeeN99qhlPbZb7/u43se5z14cM7j2mcXXvfx2XlwHqzzWO/xzm3ex4cvaDfa/ttu/9td/9va+OzBVFAvYHECJz+Diy+gKcFMwfwhoRW/+Xn9JnErwBgebP1981hfdc+I1/z9qn9vv/ZVn3vVd93c/lX7eU2I1xybEK/cRADOzcEfv+I4vsewwAooCSDZAO7lzTKl2Rvv0+2mHJ9/yp9//of85c9/imu+wW8RhF64Ai49TIDltzxOmcBgBJ2c9Ul1HrwL/5Qy3Ej+7c7tf/Xv/gdA7AwInUDoC7Z6i3XENvN1TfjmWyJ0Xeu2Enui2OcA7lrHtIl4PPHZeffSd/v4+zfH/vLBbf+2G+/w/OmvOTv58vUHfxd38dckHKFbXJQlL54+5pd/+mcsL6YMOj06SQfhUpIkRSqFdeCFxwsPSqCyFJ1nSK0QQiC1RmcpKklf+h4BHPQH7I1GJCgSkdDLe4xHY/b3djg5eYrD0xv06ORdFvOC5aLEGtBC4J1HeIeWAgEoPMJYpBNomYJwZCohlQK11a84Nl1AO/ppwjCggXuZ4ODBPuJkwvNV/Zs0at0FoBONkBKpNUKp9fXxiOvjy1aI+H9PO060n9mMB36rjfh2DIngWPrwSSE80nuc8HjpcQ6cjMDYBVAshEd4jxfgvcNJ8F7giLi3BcQijuGiHWwdceMImG0EUBKkA9NANYHZU7j4Jcz/EpgTQEV9+yf6FXFLwPh/vHWR2gsmgHhS2isiWxQpNtuJrdfk9mfZPLfbiThLuvH5AFDjhcJvNkcghEBIGTovIRFSIISM74HEI9vPQNhexIYXO6L1axKEkEgZDlUKUDIwD0LC5flnnDz+XxIu4m9IOAJgbeLDEMDyjV4wTRLyPMf5iqcnv+Bf/fSfcvx09vUYXwADhRgoaDy+NLDw4bu+TagMul1IdJxFWtYIvp2BCF4J6r9N/G/+d/97oO0QXLhH11Po7R+7DYoleEEgRuJDsO582u381qGG2Npf6G3WwNd799L7a8jrwyzdO7f5Dt92ZO56Z8Zmtr/Zpu3o/LVf5L3jP/6//z/4j/4X/3OMKd7iLN7FXfzVD0m4Va1zTM/O+fLnv6SZ1xweHLI72qfbHZFlGVKKwIwJ8BKEUOg0JckyVGTypFaoNEUlLw+pCYKD0Yh+p4t0AokkTTJ6nR7D4YDd8RCVKvrDIUmSsZitWC4KrPVIJbDO45xD4jb4orFgQYsUITyJ0iRCorZ+F2wAcfuaIgBjCez2Mx4cPaCqBWerk+8IctzFNw2tJUJIlFYBGIt29NnGSNsR8QpiM5a0lM+1MSK+2+JUHyZf6/EDjxQBBEsRxhcnPM47rBcRX3mss2Ad1lU4U0YSSwIaKRK8VPgIxrwkztRceBY2AGK2Hm1DpYFmBqszKB4DX/DtAcXbxS0B4wHrW7AFriICY1y4k2lf33p/GxjLFmGKzb7aECK8/4rPifi3FB4hXAS77W4i+JUSKWUEtWIDdAWxI/EB6K73RwDRYmtbKZARHEsBKh6SEsTXBYvZgN842baEbpKihcdZy6JygSa5EQ0Nx1df8rPP/hU///ynfPn4lPor26JAJpqkm5Pv9FCpwnpD5VeUegXafru2bGuYL6DfD7ONdc++hYQ9cXL15tzGaDRadxJrAPoK1jU0rzCcCCQCFdqxaLkYh8etAaiLYNU5h3cW6wzOGrx3sd1plM7QKjBK14Gx3zqm8Nwe1+bYwlKVX4PhLUBMS0pvv399f+02/X4/tPG7uIu/5nEoBHtZwmDYp9fpUS9WTM/OSZyEGszQ0xsI0twHaRdh4JdKk6YZaZKFvt97pBAkOkEq9dL3WDyrumRVl5R1iSxTqrqiKAvqusIag041UirqqmY2mbFcrDC1QSUKYxrqqqYqKxzQeEdTGVzj8Dawcd4KhJMkQpATxrUM1kC5jRYcJ0CSaNIkI09zEiGo33I17i5uN4R3AWNIj1IyrFh4cIH+u7Ya0BKEYo2dfAC819ZCW8JEBEAch1LnWe+73asTcQImAvMrcOBM4HekAgHOlLhqRjU7xk6eg61A55ANoXsP0dlDyBSURniJlwJvG7AGmhWUUygX4JpAhKkMdAYUoDwoRWit333cEjCOU49ttnfNwm2zfpFT81ufa5lf77gujNm+SeXW/uS1z7UsnfcuXlS/vuAiHk8LMIRw4OWavLs2e8KH2Y33CCHwuI2OR2yDMbHBbGuKsKX7fvM6Fq0zdjsP6CUZXtQ8MU8oqpcRq8Hw/PJL/uIXjl89+SUvXtS4MPkLEzokUqZIJVBakXe6dPo98jwjSzM8jqqqKH1OoruUeUFjVuDNNztQV8PVBXSy8EiTOBmK3fq6aUleElB9i2ia8Nudi23F3wTGm0lTkDyICIrDv4VweGHwGLyzOOdw1mFdeBhjMKahaUrqusA5i1YJSZLR6Yzp5COkkAFUt4udfpu1DtEe1xoA+5ugeHvbCLRv7mfNTm+2de4tKfe7+F6jtwu9AQGjEXpD6eM4AggFPgErwchWC0hkaeJKl4xzz7gId3Mq34KntTZVbFY9jYPGgWkHzS0eo90GHwgD1RIPInzOuI3scM1kinDMQm325xz4BpyJ3YfbWqHjumb2TUMDHx/c52B3h7zfxXtJ0xhW0ylTNMJKvFMgNF5IkjQFwtggVYLWGYlOA4HiPEoItFIo+SpgDLOqZFmWlHWFrAqKsqBYLSlWBVVdkeQpeE9VVsymc5aLFXXdoKTCNIa6rqiaer2/pjHYxuFsIHWc8eAECZIOG2DcLrJt9TZsLpnAOUBIVBz77uI3KFyNEJpEJCQ6MMXOe6xvr2W8C1oScI1NtnBPgLRsyBGxhmst5BLe47y4Nk6ICIidtzjfgGvwpgpNRHUQSLyZY1bH2LOfwdM/g2oJ+QCG9+Hgt/FKIdIxMk2BcINbb/DeQDOHxQtYngZA7Q2kfch3wo+TLuYaxc/+lWSMt+UQ1+QR6/9t3Y0ts9w+bn5uDTu3Pndj+2u3dpwjrWUP248AcgJLvM0ux8+KzZ6kEDc+t5FdyNewxlKETin8O0g2ftNiMN7l4/d+RH/YBWWgknwx+ZK6btaDpU4Vo70RTZNw/HzB4lwiS0Wee8xIIhpNInIkGutsWFJxlrosccZQ6wq8p2ks1jmUT8hShcsktlqGUe5rw8PlE5ifQ5pDfwh7+zAeh5F8PfrGg37D2AbE62/eGg+2ZcHeW5yzNE1DXTcYU9GYEutqvDdY22CMoakNddPQNDVVVVFVBUW5pCznOGfJsw79/pj79z/m6PAj8qyLVG1b8VvHtDkXvk2CiEB3ext/A0Szfu3l914G0dflFXfxVyu6A9g5BGxcFpeQekhc7MwT8CkYBbWMIDbmtbT9o1IBGCsZ555sr12E/yViK3ErbmAjIK5cUPq10oLNEu+6SwmMpAiAXQK1g8qEzxO30QK0imSsDvsxPqyw2ibMlV0TxkwlQcuwz4QAut8mOjrlhz/8Mbs7I3SiqCtDUdQBlBtLsVqSd1bk3YIkzxAKrG2w1iJUXEFC4Y3HNhbhBVonQVrxivCJwimBFR7rLTZOoI0x2MZiGkNTNZSrguV8znK+YLVaIoDaNBhnMW7N+22d+EAHOecxYWUby0Y6cZNqsoQFwwJ4frmk+Oxzjq8WLO8mzL9x0UxeoDo5ve4+o26KQVIbqL2gcTLwVbAewDY8Xfx3e19GzBT4FR/HC7F+3/mw6uHbnsA7rGlwpqCcn7OanlAurzDlEi8UyeA+OhtQzU6oL76AX/8EvvwjqGegOzB4BMsSrMLtfUiS9wLL7MFbj6UBUwQd8eIFrM6hmkI2gN49UHlUVyxj59Dllcvc7zBuBxjLFuzCNdDbdrfCb3pl4JpW+CYo3ga+a/wrt/THm+8I4LUFshspRZArR9C8BsdyC+QG0CtpZRR+S4KxDYLjdnIjq5BCXNMYS+HXgFmuAdxvSAjBvfv3+eTjHzEY9HBYbAXlTDDLphhjyPKUvZ0dxqMhAs3VqaeZj+l7yDvgdGBPsNDUDYvFkroqqQvHUmxkKoHZjcuMWY7WKXmnSy0ETXH1zY7X25CNWi9gOQm9fLcXZ45xpWBzp79RBAnD9ZBye3+h3QRZhKU2JYvFlPn8iuVqxmo1o25KnGtomoaqrqjKiqIoKIuC5WrJcrVktZqxXE3w3tDvDdnfPeJ3f8cy6O2glCKVeViZaCUPrl30ag90A4pbsPvSNnfx1zJ8TOjWhEcmAkDWPuAlF29HLwAJwoFtu2MfV7q2m5qIsn42jO+6H4VrCV1CRJBK+B4nNrejF5t+MSE8ZETL0sduPN5+UkCqQ0qBiOv9tv2Slo2ObLaPx7Huc786If8bRb8z4ONPfodRvwfOhNWuomS5KlgsVtRVRVUXVHVJ1lRILWlMQ2MMQmm8E+AEzjicMQgPiVYRGG9PNcI/ZZ4hUo0TLgBjZ3DW4qzDGBdAcVGRqBXLZM5yMWO5WCBF+N6gcd78aik0QigQkUW0YIyn9oFba9dVbz4MARQ3QFk7jl+cUfq3Tt24i3cQ1cWXZMMBo4OMo/6QygkWNRQWCicw8aJ5366KtxRwuweB27o3W22yi0sH62S5ONb4eOM753FNha0mLC4+Zfb4z3DnT2BxBSqnefA76PEDzOQ5nHwKj/8YVr8GXGhg5VlYrtIDyAaonQcoLQMDLT0WA64IQHp1BhefwuwJJH3o34fODiQ9aEwc93cIrfa7y4u5Rcb4FQxweHNr6rq9zasT6V75OSFugO8WwG70vdcY4vXubwBcWpC89fl2eYwWe7f723xWtgl7a83yNjBugXLLRP/mhEgUO/v77B0c0u91sM6yt3vF4e6MYW8XtCTPUvrdLkmqaVxNU1VoN2CQJjjp8emm12xqg9IavUwoy5KmqrCNxQoLQiKURoswMwyaO41PMpqiVd5/i/AGJmcw3YV+D9K4MBhlBe8m2smUxNqGVTFnNjvnxcljTo6fMJtfslrNaZoK5x3OWhpjqOuasgzAeFUUFMWKopizKic4Z+h2+kxnl4xGexwcPEQpxXC4S5JkoUN7BeBtlWHXmOLXgOI12+e3/3UX/yaGdxunIy82XWMrp1jfIkTQ6jcscCtrF46gSHLg9YZJtDEvpu1mZQTA6+m+vA6ybPz+Nc6OQLyVPEi4dtvLdhsiax2Z6/ViEJvuXkSph4/AXsrNQ9wCMDbGslqWDLsDOvmAROdolWItLBcFZVlweXlBZSxOenb0Lt5bpFQomaBlghI6qK2MQwKJTkj0K4ZUHxL8lssFl+fndNKMalUyn05YrVasrKVeLFmVDct5SbkynD465epqghCSsqrwQpCnHQTQFRm93ggvFPPFEussy7KiNh6PXMNys/XcXoY2/7r9+5uYDn27kLw+z6ZdU/g2X/q6K/1vfj/nl6eotKLnd9mRQ5bWY6zF2gRLhhIJXoSZZWCAJaJlhAlAeHPGN65I7RDaSp+c3wj7AHCGZnnO/OxT5r/+Y9wv/nO4+BLqZWBzZyeY3fdheQFXT2H1hOvjewHTL+DkS9j7AGlKdJrhpcQpidUJVqdhuQgH1QyqY6gkFFfQuQ/DRwEod3ZC51SVBA/Y7+a63xJjvHUjiBuPl9jfLXB8ExCLluK4uZ/N++t/tm9tkcktQG6T79YAtwXQa/AMQvoIZje3shDXPyel2JJKbN7bgOJN8p2U4gbzePuRsOnsvknobspwZ4es30PnGcI2dPt9RuMxnaaH0hKtNFoncWCSeAQyk2QywdQ1pm4QCJI0AyTD4ZiirLiaTJlOp9R1jbMOhEBrhdYJUgRNuPBtEmOK92+wFNKs4OIs3MGDOCq6ly3MbjXihMi4msn0jGfPPuWXv/wzfvXZT5nPrwJbHH9vu4KA91RNSJBpmprG1DTNiqpeYJ2hLOc0pubZ8y+4d/A+aZKT5z20Tl4rbdiWTLysg37Vxnz1NrBWoP11ihvc3V/58G7DGPsotvVxfIFN39hqjttlde/CXHOz/go+ibeTCPah1gUwjQChQ9+WxH5PbXMWsGaVDRvw3b7dkgxxs2va1pa0UOu+s13O3Rpat/t+CajYB7eg+BYW5mblgp/95c8Rv+V579EjpNIopUPynICiKjmfThHn56AE3X4HISBNU7KsQ5JkaJEgvADrkEKSJglpkoQVpxutrioqLk7P+PLzz+hmOa5uOD85Yb5YBIDqPEVdMasrZquCh08/4OL8nCRJKcoSoRTj8R7Dk+fc6x8w3j/AK8Xp5SVNUzOZL6isR8iclBmCcG0agimRD6fxW40f3zbSbJ/f/1t/n9FwSJ4qpAqElLGOqjZMFxWnl3OmsxVlFSQk6yRkIVgvHRBmRUIppFIIqaJ8JZIDzobEZhtYd2/txhoM2DSOdqljPcLz1Q2nbYGvOkPbDfK76UdTMyEznp65pG9ymrJCzgsUXZJkhNQDvOyBkuFciXZV1a+ZYOtbT2KxHlN8tF3bAOM2lTyqll1JMXnC5IufYH/+T+D5n4KrwkEZ4NkUzn4eEubtCnz18sG7CVw+hdkZVHNkloPOSNMEYfuU3RG2M4KsH8+nA2qwJ7C0kOxCdgDDMSQ7cHoJ7phrHdg7jFtijG/8fbPtbLfHbcu2Vz2ugegbn93elC2N7xq0tq9fZx22XSiC5KK1XvPXgLbc/swNULx5T1wDxWtXCiFCAsM7umm2T8U36dhEIth5cI+9vX06nZw01Rjr6XQyBv0ui+WKummoTINrPEorlE5IUg1JANFVWVKyQkpJ3umidIZ1UFYGIQKXVFYl1li892tZhZLhJvUiSC3qvE9dWL69gN7DYgpZDnkGUoeR/G3W/bZP3rVLtX3tPE1dcnl5zJOnv+LTz/+cX/ziT1gVi9jGQgOXUpFojZJhubNuGqyxGGswpqRuDE0DzhqWS8fh/lNOj56zu3ufe/cegO+yxbe9dKAbO53XMMVrbVmrSfYv/y5/ffu/blKMG0PkX/24YU7rZRz3Y8fQKtjWmFJsuTM6wEbmmA3n4NpxKcY6CS4C79aWct03bjXZ7ez2jYxtczzrBR6/4ULWw4OPt7OIMo54LOvPtD/CXwfGt4FNGm84PnnOo8Mj3EPQKkElljTLyDod0qJktlrRrJaUVYGxDVJJ8rxDp9Mjz7okOg0yFWPCClma0u31kFJib2h2TdUwObvgWfoFnTQD77k4P6Mori8Pe6AwFZPJhOUq9NFeQNbpcnB4n0eTKft799k/eojudJgXBVVVUjuPzLv0x/dobENjVljXYL2jNb5s2eN3Ffv3f4d/8D/4n/Lo6B79TkKSSpSWGOsoSsP5pODTZxN+fTzn+KrkfF5Sm3B0HoEXCicUxku8kKRpQpYm5KkiSxUqShqttUH6UlWsyoZV2VA3jsa4YJsbZ3A+NiLRzq6+Ahw7WmvMdpZ3s8cQN/axFX7rhowfvQ0UoO0KUUE1fc70ZMHp5SVPLy5pkjFy8JBkeEQ+OCJNdtBaoFSUAHqHdQLjJbK9x1x7XwWAvF51an2HRfuawdiCavqM5vlP4fTnG1C8/r0FVF8nayigPIXFJbaagR+SqAwlNdKl2DTDKk1YTmnYAF4HfhEQfTqEbAy5gatfBUb5rxYwjikXLaVwA8xee22NRLfB8PZrN7eFtf1DbHEtK7zN3ArhrwHhl8noYJsl8GvWYVv50YJpGTvjjXZ4W0scvkcJgRYxQzoC4paVflfh+ebW1lkv4/4PH/Kj3/59Hty7RydN0EqgnKCjFR2tOVvMefL8Oaax9LtdxsMRewd7jIZDJB5nDYVaIXwAu51uD6VT6sbhaegPgnyiMSYk5NmQkOdtXMTxIOJJ9gim3mHLK15Gpi3Ufw10qQsoVlD2Qa9FVW92EonJd1toabPYselMrQ0s7/n5U548+yXHp19yOT1DCOj3+iipsaYdalzsaCzWhCS9uq6pKktRQF2GJKIiqzk/vGQ2vaIsFoHxeMXvXs/qIzp4lSYav+U+Ebd3a1bZI7xY3wPXT1fs9L9HmChRdNCkwqGEo3CWknfX3d2WdlKxtQx5S/t8k2gn5C1ubEFsMNnfAqnEPs5v5AprcMzmzpORqZU+Jr5FoKqiHrjtJ6UEoTa8Bu2kLQ6465W3eD+1DPBL8GJ94CEpUNgboFhtBm3XYpR4/GLrd9zGNfCuAVxg25IErTwdMWBPS9Juh3zQp24ahsM+Uni0lvT6PfrdIf3egDzthN9RNeA8WZqxu7dPlnZZlbPrX1ZblhcTTq0nSVKEgMvLSxr7cssXCJQKjKvOEhKZMN7d4f7DR9R1w97+fe4fvUd/OA5JfVaSD/vs3L9PmiTs7B2wWs5YLa+YTE5w1XKduvQu267zgsYpjNc4kSCVJE0EeQqdNCFRGuMkSmV0sgWdbElZVRhraJyndoLKSUonaKK9lxNRIiAUSaLRSQJAkiQoneJEgfESJx1eu6iRb/s+H8+nQrReJq8gr2Iv3hpw3pjJtREb5zYw9pHB3uIjhAfVAuS3PJ+J8zTFgidPPuXZs4LPvvgVx08eIwb36D/8XY4+/H3e+9gz6CWkKkNpTW09lXFAmGQgJEoSJh5eBmCMxDuxHmc84d6zzmJKA/WCZn4KF0+gWbzh0TuwM6immHqKc/sk6RhQCCtQwoBZBOlEPeWllpn0YLAfrN+qCpJOBMbfTdyexvi1ABg2LWcLfX4TlrhlDLYampA+gmIflvfYJM21jhFr2cU1pjgcx4YtfvkRWGFxHSjLzesb6UWQYQSALG4wxt9fCAS74x0++u0f8MPf+20ePHif3dEO0jtsbWiKFa6qwDQsplNOnj+jrmvGwx0SITg82GfUHeDxwdbMS5wDqSTdXh8pE0RZ47ykk4XlK+tcWIp1DmsM1lqcsTjngzabYFBfW8OiKcAWgEClfXrDPZzzIZlvecYrYYxvAmucpIE5Vuq6dOdbRpuV2/aRzrUMlwckzhmsrVgsp5xdPOPZ8885O3/ObFaSppIsNXglME34/aaxCAFV1TpSGKrKUa1CFcuWJG+sYz5dUVUV1hqcjzzeFkkRDxC+Sj6xZpBftnJr99LKM9olXX8DRH8/IUjYZV8dcDAY0etYMl0xX55zOb/gpKlZfU9H9rpQhCrqY2AkNolLJ/67TAO5HlIENwd3rW+Kb0aw2lp/t9bwMoyTa3a5JV3bST8i6IWtD5pT6wJgVTFprpU/yJadjpihXbwRbGQRLWg3raUbm23aKXB7nA42AzMBd3hYV4l1sE68e2kF8i3DA01TUFVLqqpAZxqVJHQSRZpn5L0enX6fum7oDfpxpRESrUnTlDTNghTKOpo6UBZ5lnH/8D4P7r/H42efY0wNeDpJxjDrIY1jNZmhkwSpgz550B0gKo2L1o8CQaIzRsMRnU5O3slJsgQv4cH77yN1yu7+PfYPjtBpivMCmWlQ0B30sPePaFYF8/mM6fSS3skXJMe/ZjGf0DQNKx9KQr+LiaixjmVlWdaOJHVIDUoKMi3JE4XsK2orQWikEGjhWRSCqq5ZNZalAWFC4TNj3dr9yFiBteBdYI2VVOBD/kXZGJQyIfFUiI2aop2A0QJj+UpQ3G7qEGstLtee22iRhmST/B1mom2fKjxIH3KZbgPCuWrFsiy4urxgPnvK+ac/x88XoD+juLgiU4oPj44YiT20r8BIfOOoG48QCVJkSKmDeYCUOBQOFQCyFNG6LfxSJwXGeBAG6WqEraKN2psevQca8BXeV0hpSXRw7vKNQEsbnCnKeXi+Fiqwxf1dSHugVsGxYpEThEHvPm7RlYJXgN2WutgGv4KXkSm8EiRvTc4EbIHS6CSBj8UUHFIqSDQSFQFty5pdd6fYuFRsdMfbbPHGwWKjIRVyy7KtfbDRLMstII4Qb9GY3i5GvR3+5n/pD/jgtz7m3qP75N0etnGcTy+YXV4wuzzn6vKSy4sLLs4uaao6Lq0olEzJsx55p89qFXw2HYrecEyWZXR6PQQCZkvqxlDVNbPpHOstUiuSNCXv5EipaOqQve2cwzmL0BKdpOh8gCnD3/cffcjBwQHz2ZzzszPq1YRXapUgzFpPiyD87/Qhfbns6jcN3/pWRXZNiNAphqVgT9PULJcTLi9POT8/4eL8lNlsTlnCYuqYnk3i9d5M17zzWOOxBpwFv1W0b/PFglT3GA/36XWHKJVEVuwV1nFRFnG96t31v7clFn7zxppJFvHHbSrkEQeT7yv/PCHlkH7nfQZ79xgMNd2OZVCcMZo8oX/2mOfLGVPv3+ly7zcJQQDDH2bwXl8x6mryXFM4z6Q0pGc1n9fvdln6daFlcHPwIuSuZApSsbFF255kCRElEC37GhGqIjpLqPAQYXzEWKhqqKMQVdjINreJem36h4zJfjI4YQSWbLPQvMYmEQBLWGfHb7++ZuBjl9ku97Z655vFIdtdhOXytzuPHrhcTXjy5DPybs6D997j3oNDOllG08TEQ6UxxqESCdZRLudMLs9xTUOxfEBTrTBNTmJTlBD08i6f/OAT/u6/++/x9NknTGcTnG3Y6fbp5xmuKrFVRZpndPp9vJSUxlA1FU1ZYuoapRI6eZcf/fh3eXh0yO54hMpzOv0BCMX+4QMGwzHj8Q46zRBKYqyhrsLnfW2xVc1itWQ+n3F1/pzL0y+ZHT9hefyM6cU5J7MFF1XNwkPl/a2BZOM8y8oxLx1SORrnqGpHN9P08gypNL2eYl+kWBRCJ0wXKxZFQVo16AZU7fCVgyYUTRKA8w5jobEGbS0Q6xdIiVIKnWickKDaZOXrIQiM83Vg3JII4W8nAjvdEnmhG95mLCIg2TLvXteVa7/TE0tBxZXptzyfx88+pTEritULmtmLUAALwFT4s6e4yQk7fsmBKljOz5kvVxSVZdV4fNpDdUboLCdLEpRWWDQGHZ69Zp2cJ8K9JrEkGrr9HvsPf0Dz2/8VlpWBy8+5PpUSwCiCJhNs1V4a8DTke7C7R29nj26vh9YiFA1JNGmSIpGhwC3J9X3LfejvQdYN8kmdwPgIFj+A6k9e8V23H7fEGLc9ptj0WGvQexMcvwIMX/vc1r4IH5HxEip8MKfHI7F4b7BNjW1qVJKSyA5Sycjsyi3WGNbexGvwez2JrnWYuAmMX/Xv8Eu2PZLFZv8IXr41330IBA8O3+ff+pt/wNEHD8kHOVVVcXl+xenJCU8++4wXT75kuViwKgomiwXWGoTMkTJF6Zwk7ZIkXZyvKCpDkiX0en16/R69XhfnPFVtYTqjKAoury5x3pNmKcPRkGw0Is9zyrqmrCrqusbWDqRCaU2nN4Ben8FwxMcf/4CDgwNePHtOuVyxSPqYOjAsrw4bbsBlDcWbA2O3jRz8dvMLvEFdl0xnl5xfHHN+fsLl5TmLeUVdgJu3Pqzbx/jNrrYQil53h73dIwaDHZTU0ZHC8XJfvqUE3gK765dewyavwa9vS5xfB8ubz30fLTQjFbtk3Qd0d96nv9dhMFSI+pzd3V3GowG7L37Fs8tzHjf+eytPmwHvKfjxvuJ3frjPg6NDsrwLKmFaVJxNl9S/esLFFzMuvoc5RqIg3wbGMvoFs5FFsNXlKhnGlhaAogLjnMjwSOWmtpSxUDgoXQCntMBYhYFTtIx0ZKtVlF1ItgqMxG57G/g6NsOAh7X84lpEcG5tmzC09Tva99u/uZ0WfOEMXx4/RihBkkvuHe2RpV3AIIQiTXK8VzS2oqpWFIsFk4tTXFOzWk6p6iWduoezBqUTulmHH3z4MaPRDrPFjLJc0tQV0lp8WTG9PGc+mdDt9djZ3yPv9RCJxnlHWSxp6posy+h1u+zfO+Lw6JDeeITMuzgh6Q1GNHVNp9Ol2+2S5jk6SwPjHxPRhHU441iVFatixXx6wfzqlOnpM6bPn3Jx/JTj4xecnp1xMS+4mK84mV6yaqq3PqfGehaVY1ZavLSUxrFUhr5xeJnQySVpphlqSYPCqQSdpchlgigaRO2gchhtsKWhsRZrQ0MIzLHFGAsiWNQhBFIpklTjZACoN4mGtaQPFWsdhBa03Yd6fGBQY14M8dV2ZWMzXIjgcXYN8rqQ+Bc/orwM7lftC28Rl09/gW9WUF+AuzENr+ao5YQdWXNPl7xYnXN1ekpVNFSNR/V2SEcluRvQcQnaJRiR0KCpSRA+wcqg5fZbmiglBZ1uh3uPPkFLwReFp/yTEsqnW79Hg9wNfq65D1ZrZn7j6DMY7KL3Dhjt7tPt9UIhO++RaUKTZiiV4EQKYgD+ktCTjCH/GEb3oNONOisDew+h+n14+kvCmse7jdtLvttmfNcMcvv6zecbzLIErlWZk+s/A2ER6rco6dFSAA5TlTTlkmZ+BfWcJhvhd++hEx2BntrkIombIHbjf7wuB92yxi07LDcgevsRmMINC70t2XjHphRfGVIoRju77B3cY2//Hlk/ZzqZcGIuOD4540//4qecnnyJc3YbduG9wBhH3VjK2gR9kk7J+wOEFFgkzkuETBHOUlcN8+mcq8srri7OQSr6gyGDwZA87zAYDJFlEcpAqgSEom4cFDVShfPb7fbY2T/g8OghzsJqWbBallydF3j/NQvqvoli/TeN653Vpg8NWuFVseDs7AUvjp9wdXXBcrWiLMAVL330W4RgvHvIhx/8iKMHHzAa7oZOwV1fsNvWArf49SYoXm93g0W++Rt9zEy+RiF+r+HRQpJ3O3R3xwwOhox2MhLXQRSa8UgwHDi6zzzi2QVPK/+dyhUyYF/ABz3BJ+/1+ejjB3zw/gfsHdxDZR280PSWJfpqxpOFJ3vxKyi+22pMAKmCTAf+QEVgq0UAxcB6JaTVDXsC0+t15Lzi9okIn83azxNAqbAgXBj/XaxI13oi+8gOC7/pm1tArnyUXcTvlBHUKtYEGxAmlu18Yru7bPffAvtttjmwdxvW+aW8qDeMJfC8qeidn/D+7D180yC9A2NwxtJSMK2XVSj9bBGiwbfVwFyNt00cLyS9vIfc1YyHI4xtMHVFU6wo5wsSJKnQDEZDDg7v0x8NSfMchKcsVzRNFVbnOh06vQHdbodEyeDuoBNSLfG+i9YJSZKSZGlwG1Ii5CJ4Fwo1OOgYQ78eMNwZUN3bozi6z/L9D5lPrphcXTK5mjCZzbm8mvHsyRMeP/mS5Wr5VufTOMeiNExXhsYLpLQIanqrhkXlGXQsOknxQrMoDUXjKKynsFB5gRUKlEAlkDqBMAIjLMqHvB4ZLVPXIUApgdYaLz3Kb1bg1snIIQM8ssbh89vyszU4lgKvNg11zQZvTfDW5trxy0PvKreWPlopRTuhe7uG6uePebVSHxAJnTRnkKfs9HLc7hCJobesmJcNIkvRHUGiTXCWcY7a1QR5ZBjTQWAROBN8vBvnESoFKdEC+oMdevfuU46P4PiETZZTA+4FFEmoImReNWbXYAuUdHTzlG6WhLLQRO14r8Ng7x7To4+w5RIuHYge9B7Cg99GHL1HtrsHxuHqDibxOFfCiz7YvyrAWG4ay3VZBVxDuNfkEq9hjonewbQFONrLJ0iEINUCaz1lvaKZXUD1FDiF4ghTdPG9PlKG5IXWA7YFw9f8iCNjHEqWbsBxyw7LG6D3dYzxddD8/SHjRGmGwwGdfo/ecEh/0KOuDNZ6Ts4vOD17hr0564ydQGMdZd2wLCuKukZlOcM8o65r6rqkrBp6BmzjWMwLLi+nXF5c0lRzEJoqWrllWZdef4BXGicUQiYIoSkrC7LAYcELpErp9UfsHdzHNJ7VomQxWzGfXdJU373SNHSQwTpouZxxcvKU58+/ZDK5oCwMdcFbrZv3+vv8+Md/kx/96Pd5+PBDRqMdhNy0T+AlL+ONVOJVQN6/EjDz8tZb4Po6EPmuI0xsDVlP0dvpMjgYMdrvk5OTVBozFgzHjt5IkvY+o/fknM+nltk7BvUSuAd8MoAfvdfl0QcPePD+R+zdf8B4d5/uYITMejiZIJcFZnhF//kclT2DYvpuD+4VkSrIY70bJaO2F9a64lY2IfwmG52txDntN9XjsgiMExkkGs4AJgJjEYCx0/EhN5rgFgS3hT8EG1DctrG2y0ewTgJs80hbX90197YFnCNB+JLcwsXPrdOfbhEcT6uSalVAbaFx2Kahrk1MVIqaDgdaCrJUkqYCrRyCWLPaGrzUCBQKQaZTUp0iJFhTUwiNMh7TK/HWMR7vsL93wGg8Js9zhBSUdYExFUmWkeUZQmq8lAjvkM4ivSJVMtqWyaCIjbZlYaXS0eqghQzjgUpTOt0cNx5i7x1imwbT1NRVTVWWLGdz5lcTjp8/49mTL1kt3g5wOOtYVoZpUbOyHusNxlR0EsF0aRjmFVmWo1TCojTMSsNk1TApDKWxGCdpvEAoiU4JwFcQpToSpXVcEZY4AsMppSRJVGjvxGIVzm6B48gYCxXpLLleWWu38QBKxGKCbWO8AZw9sAV529WMjb4nPLX3we3E60UuMhkw6O/Q7/UYjQZknYzR7ojZfMVsWdJ4iZMJjsAIG2fAWrxxOOtCCfE2+a4oqOZzytpA2kWkHYQSaCXo9ntM9newF11ottfxCvDFV4yLBuoJylV0E0E3UeAUXoSKe2rYxR09RJoVV85gpYLuGA4/Ijn6AYOj9+kO9/HGYuuKupdQUFLo7ndiTPHuS0ILbop4oRVo3mCP1/IEL5HIdSfY6omVFGjpsE2NWVxC9Qw4Bi6BPsKZoPyJDfqaHGKrxPM2wxsOy2+yqtds8Y3kvS0gLNtbTARu+5rG+HuKTKdheS0NnXKW5PS6PXbGu+zuHZCkXay57iUsZIfuYJ/dnR329vcZ7+wy2tmhOxiQ5BmL+YzL83NMXTO5moZypcsCIQTD4QgpHEqndAdDsiynqWuWywJjLVJqdBIE/klakiQpJiblGWupG0NdGxCKNOuEwc98B8L6m51Wy2SZhqapmM2uOD17zsnpcxaLKU3jsF+l8PiaEFLz8NFH/PjHv8ejR+8z7I9Ikiw4eGwv5bUs8Zo9fn3yHVvbXVd13Fj7Ywtc+yAj8Tfe/65CIRl1cnb3+xzcH7F/f4edgyFdNSC3PcwqYznVdAY6ZJfLBvP5JY/nYeHsXagWNPBAwu8dwG9/0uGT3/ot9h/8kNHhh/RGB+S9IUmnh0g7WBRFuiBpEsiHWHk7Xee3CUGoFNcyxuvkOc+6rxV+I6kQYdV5o++90e1qDXkagHaqAsYTMeHORmDcaDAaGsHaP5koq1jTIWKTfCe2WOr2O30LFiJou8Y8s8EWXmzs5QRRFx3311bFa6UbtzXLS4Bxp0s364REuqrGNA3WGLxXUSnqsCYW8zHRp9zUNG1JZ9sEr11C1TDb2CDZEoGNs8aC82iV0u30yLMOWukA5kxI3sU4hJeBVfOxrIoVOFvhqiJcMyWQWqHSLirrIKTAW0ldN6xWc+qqwNlw0TvdAZ3uEK01UihkqknSPJw2TyhOtFdSHS7Zv3ePB48eUVVvV3bXuwCMVWFQjadxlto2pNKzKg3TtCZLKpTSFCbqkZuGRW2obWhUzosgVxMCJYNUQiLQQpIkCYlOUEpGa1AbJiUmThWEDMUqrMI5t2Z8xXoVuqXaNv1i2+cGKcUWgef9lgTtFcB4/ddWC/Ygvduya3tXHW1KOnzI7s49Bt0evU5OnmfkeR5WEtIVtfVYFNaLcE6tp2oslTHUcYW4cY7GelTjMRiaesl8vmDVeKwIYHpydhoS9r91SESny6ibsteBbh4cq4SUSC2xacLYjRmLIzJzxZkokN0R3XtH9Pd3GOxk5D2JcAJvoMo7TOouT3XyneTf3b7GGG6wwWyB4u33rrPEG9AZZoTKb7kPCoWULlqkeVxd4IoT4AtgRkj9d0gVwKqzBisUWmuUUhEMy+ss7w0rNgkoES7cpjBIC3Y3zhMtMG6dMLblFvI2nOffMLpJRjfNSGRgLXDQ7fR57/0P+Bt/Y8HTx1/ysz//Y6xZEXqLlMHuIx49eo/D/X0eHj3ggw8/5MH7jxju7tIZ9Dg/O0NLzYunz3j+9AXzyQTrLYPhiJ2dMVqFxSLjPMY6ppMZ09mCrNsj6XSCz2+SkmU5eZpjGkNZVRRlxcXFFd3uKXVR0hjLYj7H2rdbyvsmsdHzxtWEuFjbNDWL5Zyrq3NOz15wfn7MslhgvX+rGWqeD/j444/5rd/6hIODfZSSa9DrtqzY1g4T6474NXrgbVy7ZoP91nv+5df81rLgq+zfvoPIRcrB3pCjhzs8eG+Xvfu7jHdH9DPoygNMOWI2ydFdSdEsWK0uMfWK7LhksoC5gSv/7Z2wXxUC6APv5/Dj9+BHP0r48KP3efTBJ/T3f0A2+oC0t0eS91BJDioNoLBSGLWi8prKfg+zC0J19CS5zj144sAdgel6FmE3bKtXEWwSNLxSBO1x3oFBJwBkbzZg1RJY4kZDpSMD3YCNdO921T0lAuMs2YDy1ltZuA3Ybu8j76NjRXyvBclKBPba+2gbFw/es6U73vrdbxsCONQZn7z/A/b2D3FOUJYVjWkisLJh6dc5atNQVCWzxQIvBcuipKxruk2DbQxSGIQQNFXDsgge5jaWi7Z1iW1qBJ4sy5BCUJVVKBSEw7lodQkknS5p15OkGWmiEWaFXVxg6gUWh9cJ3d1DeruHgT3VitVsyuPHv+b87ISqrJBCcv/oPe4/fI8874QiGfGMSSHDOCnDCmqn3yPJM8b7ey95L3/bcM6xrC2ibEA5jHfUzqNwlJVlKiqSeCyVE1QWSusovAsTubVUogWe4TgTKUmUIk00aZagdMjPsNYh6xovZGivQoaiFiqc01ZOIdcr1K0wkzVjvGkMMflObvevkVWlfe06MA6vXG+JIkpuNnu57X5CQHbEzv0fsr/3gEFvQCfVNC78nixNyLIU5cMN7qVCCIUjrAw35vqjNo55lpJKjzMNZyfHPH9+RrVc4IoFfnEC85NosfQtQvfoH73Ho/0RR13o5DXeerRWocgHgjLJmadDemaXLnuIpEtvJ6U7cnS7K/JcBq9wL1glnnzheaFvL1n0Kw//Vvby2qId8f2XADEbMI3YKtccE+f8NjO7SSRqqhXl/Ipi9hjcL4BPCQ2vCxSYYspq1kPnjjSPTIYKwCf4F7eWa1vlndede7R/25JFvMqFQrbAeP1MLDEdLFG+L2BsrQ1sR1lhagMW8jRnb3efjz76mH/7b/8BQmguL84piwW9wQ6HDx9y/94h9/b2uH9wjwcPHnDv4B7DvR2yXpembjjvnOK9ZzafM58vGO2M2NkdMeh16OYZdVMzX644v7jk4vyCZVnSG47oDoPmOE2TkDmcJqhS433JarXixfNn1GUFjWG1mDGfnPDdmHdf77BaAFqUBVdXF5xfnHBxccJkesGqWGHCnOvNQij29g95/8P3ee/99xiPR4HlWXfa7ZFsmNw1S/GKMtEv/5LrwPkm67wGxXEAcK8D299BBM2aZnevy73DITsHQwbjAf0soa8Fps5IOh5LyWxyzOzqGcXiknJZ4kxYtZub2wHGKfC+hh89hB98IvjgBzvcf+8Rew8fkY0foQYPkdkOKs0RMgngqDZUbsa8MMxXNVXzPXhSiOBWqDVrJnZbSukJz8EGYgskb/ulxf5OS8gS6HVg0IVeHgb1xEOuoDRQOSgic2zsFrPb7lcS8gZiH9jqjQXRtm2rfxWOjR0b1xc32kP2hAHJtvuJwLlliuMK9trG7W0jAQ46XXZ3d8iyhKouML6i8TUOgZBBZG2soWpqZvM5pxeXFE3DdDZntSrodUqavA4yPelwrsHZBts0NLXFGgPOIYQgzTKUCsyn9566roLFpTPhvpcSKzOscnjhURLcYsny+DHl5JjaO1yaMixLjJB0rCVtaqaXF5w+f86zZ09Yzhd4B1XRgJN0er1Q0U/KCDQFSmvSVJN3O3R6HTqdPgOdhN/7FuG9p3IOZaKVpfc0LtRTNd6ihUN6iwAaLzBOhOp8wuOFREp/3RpVBvdhrSSJVmgtUSq83lZpDY5RxL8jCeaCkwXehXE/TgtCcl1cyriRARqW+CPPK2Lf6rf7VtgS/6zjGmMMCOc3Mp93QkIMEJ0HpL1dkqyH1hq8o64qlsuC+WLBfFmC1Og8RycZQim0DDIcpT2JBes8xnga46Icy7JaFkjbUF2cY58/g9UJuAXBnPLbjc26v897773Po90BB5kh1Ss8Bq0UeZIisBTpiiQvWXUtRV/iqEn8FUlZAhfQ9Ei7Q7IkBzsnMxcI93arGt/4+G9lLy0wvuZPHN+LyyJtBkY7I9zMCTf/rZudjEN+3NY7izMVi6vnNPM/A/+XwFM2YnABnONWv6QoV8jhe7AT1ETByULhpSRRKSrZVNBplwCViL7EkmtgeA2OYa07bh9qDYq/CylFHOm+IiblgufPjzk/uWC4c0BvsINOEpSwjEc7/P7v/w0ODw+pqxpnbMxQl3SyjGG3z3g4ZDQekWUpzjrKomS1XLJaFTSNQSeawc6IB+895PDwAIzBVgWmaZBCYhrDdDrj8vISeXFB1u2xf++Q/YODIGsJmXd4D8v5jC8Xc56Jn4OpMHVBUc3e0bm7Hs61N3hobUKEScViPuPk+CnHx0+YTM5YLmcUy6gvfqP+TdAb7vH+Rx/x8OEj9vZ3yTt5YDqcXet+oe2EIyjeXsJz16Gx3L6vbsS1hJOw0w0zHZmgjafxdx/GVzhWZF3PcJzRH6TknVBCVyuFFD26vR2Gw31Goz36vRF17XlxDs9WQQt6W8OMI0gS+j3odDRJ2kMkfVwygHSIzMfIbAhKBzbKempTM5nNeXF8wunpGVX93XTQL4UiJNJFqti1Lg42Mqzx8rZNpfUVbuUMSkIngUEO4xzGCQwT6CXhnIxSqIZwOYeLOTQNEG0IXWRxRZRUBOkZiC3AK9rvg7XdlogAWrsg0TCR/XVy63e5rcVFB9KGR8smry3d2Pgjv210gUw5qmbGYpWAKFBJgpMOqVKSRCG1xtiGsio5u7zk8XRBr2x4/+yCowcLuvmKPFvFAlAOJSW9TkqaJDSNx1kXltdxEdRtxh7vHFZKnAuMHlKjsw467aJ0ghBQrhacP/k10xefUePxeYdZZRhYT3c0I+sOmS8XFMsFTVkxn82oigohEprK0+n1QjGMSNpIKVFakXcydg/32RcHdIQEmbx1gSqPwAiBIRQm2qzOSZwUQbe+nqQTJmlAIjxetDLHMNMSW7k+IWcobGy9w9gGZz2NdSFprK5AJggdS3qruDrsHRK3JtfYAsZrbfHNEDfaVtudirYNvgoYxw/CFiiOT7fa3XaAA7zrMF1WTBYLlmXBfDHn5OyCF6fnnF9NmcyXZJ0ug/EuebcXJmQ6DedVrP29cF7gHCSJZjAYsbtTMxwOSdIEW83BtRhLE+iEdi2pHUMFqOxG5TpAJOw+ep8ffvgeR+MOA5bIpsTZBulAe4U1JeXkhPnFCcXkmHpxSlkWNC54/Hvp6ORdDvYeMejvsVwsuXz2DFtNb/OEvjbeETBmCwyHaGdWrT53/apoQfEGkPobKxbOGly9wqyegP/XwAuuN9+CoDMGnMIVI5puH+Et1tQkKkEnCTqTqDRFy1AFp5VRKBFt4LYY41YisQHH28B4U/Fu267tXUkpFBpH85UMYu0aXpy84MWzZ+zcO2S0dw8ZMxh6vT4//OQTfvhbn5BnGWmSMp1OObs4p6lqEqnIs5S8E5bdmqahLkvmszmL+RxjDHm3S5ImHB4dcf/wHqvphNlFhRISrUIzqquKankFy5piGr57OBqDlKgkQWoVbHeWU4yZ8V2Vd9yO4JIrgs8vMnhkGsN8PuH4+DEnJ0+YTi8oViWrZSgF/yYhdcqDh+/x8cc/4P6DI4ajoC2+lvCxOahrlmqtXuLm9fZbd86117eTRLZfY2Nf1DJz31c0vmJVXiFkRacryDuSPFMkWiFRSJ+SZ326nSH97oBO1qGpHOcV3DQCuo3IkgCM87yLTvvIpIeTXbzuQtJDJl2QCuscja1YFiXnF5c8ffqMs5NjjPnuHSkgsq6xiJeNtmrGRelDm4BHlCUINo4SBIY0FdBNYJTDMIWeig8dqq6rbtgfAoo62LfJVlMciQQpI3jd5j9EYHVbS7dWH9zWVZAyst0iJO3Zdtv2uCOIlj7sW/ktm7f4Ox2sK+q9wq72W0cPkKJhsbxAqoa67gb7MB0mS91uRiIzjGkoq4rL2YyFs9RVyeXVhOWyoIr+w7UGLwwqSVEqQUqNkgLvBBIX2NBIogQZgMWJQMNLFyQASI3UKVJnSCVBWExdsby64Or4ORUOl+WsZMZC5vRWNZ1BgY3V89I0Q0vNyhYs5kukOCPvLEiTFKVUXC0IGs9Ot4POU/qjITrroBK/EY2/acQ2sC7rvQakAqQKrH9c0fLeI3yAaGpNpgWG3kkJ0R1KylBzoD0262xo88YFzXdd0VRVsLuVSdBhy8B+S2xgqGMOEULgIiv+bQmCdjXGvbYTbQk/sdHZvwNQDGMwmuVixcVsxnSx4Gqa8+z5Cz7/4gknZ5dcTWf0hiP27y3pj0ZknS5pmqG0CpptlSB1AoTzLWQoWtPt9ej3uuRZQulXhJ7XEkBxH9iLv3MOooF8BLtjuDyB4mxzJtIO+/cOONgf0c8Eqp5FfVSD84ZaWspizuXpY05Pn3F5PmF+MWU6u2I5v8BUi4Db8ozF0cfs7jxgtaq4PDnFm3cvt4TbdKVYL9VtscZxih8gyLYkYYs5bhnZeCMgN8A4JHM4qnJFNTnFm8+AU17mC+r46AIjdN5BCsfi6hxf1+jugN5whFYKJXpoFZhLEW9K1TpUyBYMbwFktoFvyzJvgWS5pT9+R35tluYVv/lmeM6vXvDZpz+jvzOiNxyzs78fjlurAIizlG6vS55loBRFUzOfzqjLCmstKkkRWlM3DcvliquzCyYXV1hj2ds/oDcYkPf7NN5jXLA3S9KMYZqzu1Owu7PH9Ooc06zAGVbTc5z/hDzv4pVmWRbYpo6eh68DxW0vaL/Bb/724ZwBJMIrNuWfV0ym5xyfPub8/CnL5YSydKEa5huumHd7Iz786Ad89PEnjHf2UDpFxOQQ/xVSiRuEw1fGGkO/buNtWcX3DI0rDE/Pn3B5fkxdzMEOSWSoxYT1eGMQxqCMRTYG0TT4xgaXgluOHBgPYDRO6Q9HdLsjkqSLQOGdiCgsoEprLNPJnOPnL3j65Rc8+fxTzs9efK3M5V2FV9EtIoJLQyzlzEaigA+FP7RmXf1OCegq6GvYSWFHh95SGHA1+CTsFxcAtyRMHjouODIJDakLlm4+Wrq1rLCT8XRxnTG+vtwRcY+IxUAi0+3d5jNrQK8iIy2AJhxPU4WkeAuwXZr6LaIGposCdfyM+eJyXWkuzTv0+rtI1UPqPtZZGmuojIka7cBWeoJVWJIo6qpgsbwKmk6ZImQGIkeiUdKjZHtWNidlXSLCx0pvwqJlRaZLtEyQSpB1OgwP7tNUc6bFkpVzVA58sUJ2a7I+9PpjdnYPePjIMJlMWMwXIDRSJdHaTSPEBvUKIUiyhG63R6LTmIfTprq/RYgorVGxvIWPhTi2xslwlUOnJa5lc8YBX8XlBaHXMhwvQjJYmzzsPFhCjoa1DaYpg6WbCn75QmVIHZL2pBfIax6DLTD+dj+tBcbtJPPVPz/+vkgI3h4wFsCAcOcUUMwwE8nVxSUnJ+dkyvH46Qu+fPKck9NzJpMZ3cGMxbKkPxpF1jgPkkal0GlGkuZInaxBstIJq/kC4WuyTMSkgPYHtPgqh84edBJErhC5Q6gae3V643gdi9WUk+Nn1FXNojcEPC5eq6ZZMJud8fTZZxyfPGE+K1jNSprpBb5Yxuo+YCU8OZnwov8prvHYouQlP+d3FLfHGN8ExW0ziVY9m2S1bWcI1ozsRvcrcNLjhUcID9bRVAt8cQI84fVIRQC7kO2R9TqAwS3OwF7R1AeUWtLv9cOSnlYh8cG3FfDi/bjOIXzZhk2IjQOFlC2Y3jDIgYF+Vxrjb3aHzcsZn376lwx2dhnt3cNBSHzrdJAilMgSURgoZJgpWudZrApwDpVmCKVZLpdMJ1Mml1fMJzN0mrJ3cMjO3i5eQN3UNDH7Oc0ykqxDWTXs7+1z8vwJsyaw96ZaUDc1XT0iSxJkkkbniVdfwzwbMBwe4kXOZHJMU5/f0vnbhHVB4xb8PoPOb7laMJmcc3r2mPOL5yyXC5qCtxC0Csa7B3z44Q/44MOPGY13g5wnDgqvXcZjPWzw6mv+Mri9phm+iaoFsZN+/R6/yzhfXHB28pxiPsHt7aKFCcygcfi6gLqEqlg/XNXcCjN4MwYKxrsw2gnFZjrdIYnOEIhgZ9QYnDJIoTG1YTadcvziBc8ef8nTLz9j/pZ+r28TXoGRseQyG2DsI5PaTielDG4TQoZCHFpAN0olRgr6ArSN1e58cLpoZRdt1bkkgmcbZRDGhe1dBMihGlm4TdbT2Ng8r4GCdniI1dzbYiC+1R1HzALhWRNBuo9V8BowCyiW4ffofI1v3ioaYNZ47EXBfFGQ5YJON6HfH2KdoNM9IOs0GGtoTCg40d5t1oEQkjRNyFLNZDrl4uoU40DIFJX0SJIhic7RyiOlDzZizq3tRNsEOIugNhaLIIuSgFTnSJIAjO8d0diG8uqc1WqBlZraBDmW0imjnT0ODu/R7XQpi4LVasWqKCmqOkw2lELQTpxCUSGlJIPhkCzNAghviam3CCGCxZdSwa8WCAC4JZVkTFWLhJfwsNb6ChBKIFpgfO0CeywBRK9X3CIGtbahqUq8syRKItI02A8mCuHbleggc9kk4H37+HrGOP6w6HV8fQr0ttEWpwdYhnnFUoU6BafnSN/w+OkxXz55zunJGbPJlKzTZTqd0R+O6PT6oRiMUmidkHc6pHmHJM3DCkeSopOEsqxwdUGW+NAZXAsHqoPYuU/vwRGD8YC6OGF59ZibecjeWaaTC548+YLFomQ43AsiDFtTFjOWiwsuzp/y7MkvWJ0e42sX9VX2pa/0q5pmdfs44Ovi9uzatiUULaiMy2AhUW0LcMoNWyzjv9fm263hZayr60yNW12CfwJcvOYA+sAnJP1P0L09mqbCLOdgzwjLAWPSPCXNdJg9EsRrLQgOcojIbG8d29qdoj1Wuc0ct3ZvYp0s8H0m3wE4HC/OntL56Z/ghGL/3pekWYc0D96YaZahlUYpiTGWuqkxjcE4Q5JlJEkKHlbFiqIskTphvL9H1ukw3tlhMBjinMU0Gu0MtYhFJKRG6SRkkG79fNcs+fXP/5zBvYd0x2Ouri5xxau1xEIo7t17n8H4HpPp6p1pYZ1zMZPYYZ1jtZpxdXXC5eUxV5dnzGYTlsvmWyfhbkeS9zl68JCjBw/Z379HnudrqYTbcp/4trFxrgif99sDxbXXNn9//3B4E6WpODt+wvT0GQe7fRikeKEwZY0pFjTFhOX0mPnVKZOLcy4n9a0X+ZDATg92D1OGB0M6gxydC5AVtplSr85xLkHnNbrTD+XNyyWmKqhWc1aLeXAq+Z7Cy+gW4QOwayJ4hDAm2/jQAuoWZEZSzvhg1bsoIstsoG6g34cHhzAchm2NC8l3zgTiKIuJdc5HUE54NERpRUTFLbvv237ft6t+rIFMS5LEIljrbX1E9d6BqaEooVhBMQ+AuCihceH7hgay7O3PZR/oK8hSQaJD1S8lN25GEFaUGlNT1RVNLCwUTycIgdYJWmuquuDq6hzjBEnaI+9IlOwhk3ZoDD825DjI9UqpFZLgf9GqNy3O1XincF6wampeTFe8uFhwOV9RGMPB/oDDB++xd/89dvcfsrO3T28wJs+zUAK426NT11R1HSZJW/kvoX9wCASdPCdLA2DaVGt785BK0clSsjTBrhGkj3n2Lcm0WVyWAM4jnNwsJcjQQLz0xJT59dYCj9xaJmu8xZuKYnmFEgrXlAgJea9HnnbBt2O3i9hiC6fwipHaf3Vv+Y2B8doS7tucva/Z7zrDwoFIScZDunnKYjbjeb3g2ZPHPPvi11ydPscWFyylZvK8h8576E6PvDugNxjS7w9CUnyeI1WCUAqVZmRxjJrP5mAqZNrBFT1Cq8xAHSL3H9E/us/uvft0Bz0uTlc0JgGfsU3pYA2Lq0tePH9CU7uQhOqCfGK5vGI+O2d68Yzi+AWU348k7evinblSCCnXnWB7072uaIaMVebC/xw4g/d1EGE3Jb4+Bx7DK4dJDfwOyeDf5uiD36GxlpMnn+NWz4BzoEEo6A/6dLp59NG04ZZrmWrZstvXQfE2i70BxnLLv3iTfBe0UN8vMAYobM2nX/6Cq9mC3mAPmWQbY3QpsLXBGkOv02c83mE4HtHfGTHa2yFLEvCeqq6pm4a81+X+oBcq2g1HdLKcsCSS0WhJnSrquqGsTOwExI3ewFNcPqGYXZLsPcQUC7Cvhjp5d4/77/8AIVJOz2dY847qnrXWZcLgnGWxvOLs/BnnF8dMppfMZyuKBfg3vF+F0hwcHvHee+9z//59xuMxaZKEFYr16fkmPeZWR7M+dA9+i23eTtTzW6D4JlCO//p+W2ZInLk4e87kxWPKe0P8OMNKqFcrytWMcnHF5PIxl+fPuDg943xhb8WFYjsyATt7MD7sMtgfkA0SZGrwLDDNBcYIqqVB5yvSwQ61F/imQPgG11QY830Vqya0HxmWGBsXmN4mJsTJSIYZGUBvI4MEYp2YZ0LvqWwAmRRQFLAqYG93wxi3muXax/LnMngcJ7HxWB9BuQj7b6vueQO125JH+AhnNjghsGhR2qHZyD5c5EGcBVtDNYf5FCZTWFVcY6SMB1u+vZRCArvdjFEnSCGkiomJnTx6DWcoKXCuoWlKqqqgbitutoBehKITWivqumIyucR5SbcHWvdQUpAkQUohor2Gd9srkZvripTRRzf0TYgGj2S2WvH58QWfPzlmWS5IsoQHvREffPxDDo7epz8+IM2DO4ETAqkFqdSoPKfT3veizZnZYnE90R9YRfeNtwfGQmm6nUDAVHGWJFoZQ7uCvM2hxUlRW5zGt35/ccU4pCyCj45PYcwNfhQSh6TB2ZLV7ALhPabqkSSK8d4+ecIGpIoAjhEb54n1wnYbLd7m9b1z64zy1RExgBe3CIw9QcqgwiPvs//gHnvjHuVqwex8zvPHX3L14nNsdUrQH4Gz59SlpJ4oViJj2rlHf+eQLM9QSYL3oSCKzjLyXh+dZljrsGWJ6nVxxSNQKao7Jh3vM7h3yHj/Hv3RDkKnOHJspcF2CbVDI5vkHHY64VIlCDQCTVmtmEzPWM4uqBfnuOklVL+ZoBhutcDHhjFeZ5TCWmrQJq+9Dhh7Qva8MTVVOcNVS4T0SFeBPSUU8nhVSztA5b/H+N7HDMd7XF5d4KoJcAasgBSZhApAaaoRODxmDXDXFswyzi7XrLB8GRy3/yYk7W2yZuUWMP7+o3IVzy++QF48j0tHYcLhcWsf20F6yEcffBKse3rdwCDXDVXTIJQKNj+JRqU6mKcbQ2EWwdEDT13XlFVNWVasiorVakVd18Fg/lp4MAuas8dh9HzlNZTk/TFpp4c1QULzrjSc3oIXDiEaalMym59zevqUi8tjlss5RWGDjOINv77b6/H+Bx/w6NEjxuMdsjwPv+ca2/u6g9v63W0ynX/Fe4Tju+ZesQWSr2229aUbu7bvJzxwfnnCk0//hJ1+TVdNSFJJ3ZTMZhMuzk559uWv+cWf/5Sf/fKUZ83t893Gw7SEq2XJ5XyKVRajLHkGaRqWeHE11iyxbkqDxlUrlFsi3OoV7fvVIdhUabvNCBrXCH7jw/uY1NZSsQQ2t4j64XoBooClg4ELTGkP0Gpj1zbowqC3cX1YVuGzxgRtr3cBr+E3vy0hssFRdyxdAOrWBTbaugCWnQcjAph2MYXAiwCIjQnHVy+gWobHcgWzEgr78vXPBAz70Ovx1uF9SOKyzkVJnY9uCRJIkOKCurHBs3h1dW1StJ5wChBSkSYZeacLQtPt9cjzDKVASk+WJSSJoGkUTRNW66w1eCSJFkitQWhSrcnzlE6ekmgd5TECdIruDhj0evSGffbuP2T34JDhzg55r49KMtaibhWMoPVa3tgecQSEiHXim/A+2IutUSp8A+T32pBKkacJSRLqiK/lkDdKwbX8r0DElYUoMpMb/z+PxIT0OZDBrUIKgRYOJRzCeagdvl5SzM/xxuKqLv1eB+kqMs0Gl0TWek0PrJ1it37r9a71tUI2f+2cvmqjOHu91YgFX+hBMqRzsM/9gyG5FkyuLjg7ecbJ00+x1TEv5+648PANdlUza+r4Wg3e4J1FSI3QOVLHiotIbBMlLlmXbJjT6SqUL2lWEyol8CqlWK3CshQdYAhkIDVkHUR/iMpHGJsymxVMJicsTn6NX56DKW8ne/Ydxq0W+HjZ91esl1HkNTAc549rtjh4u1rTUC6X2Mk5VFcgJVZZ8MeE+lc3Iwf5I0b3fovh+BBEQrEqwFwSXCpSoItKO+R5itbBrB3vNybn2zNY8TIolnLz22QsKx2NviKwvl5qGiHe+ep1Gp+/mrtyOMrXHktl50ilyDoddNQWWx/aa7/XpzfoI1QovblaLJheXVCtCtIsRUnBcjlnvphSlRVl2TCZXLFcLjDuNbNA9xUMsAxJIk1V4VzQh0ml3onOPhQJcHhvqao5k+kJx6dfcnV1QlWtMDVvVf55vLPHhx9+xKNH79Hv94Lnpo++mt+yYbyu+l0gjl1krjbFQrxreeL1lutttvf5fcbVYs4v/uJfkslTqL6gO+iA9pxenPPp51/y85895ee/mHNW+Le5DC+F0gGsNcDjc9j/rEQmz7l3v8N+s8N4bBhrSDBIV2CqCXXTofEprhZoP0X5VeSwvjokQZvbI4DHW4vIrpr4sPEZH3yJxVainbWwqKGcwewUminkFkYSPt6D+wcw6gVgfG8PHj2EnV1Ah32eXYK7gFUJy0UAx8pH5574Xe3qdxq1BTqyzY2BqgJbBTnEchUY6EaCT0ClYfwUIuiHZxcwvYBlEbyTb06fM6CTQrcDwwGMdyB9SymFA6ZFRVmuLZlD4p8uySdL+v0FxaqiP5pRGsNqucRsV+b0oYiEcw4hFf3hmPtHHyClIsv7aN0uLxvyToder4MxhqqqWC4WLBYFwgsyqUmzFJFmyDQj72R0sizAOWfI8g737h+SdDI6gy7D3R0efvgheX+A1EmQHuKQSiJu6me3cZ/fFA7ChUl0sJhraVtPWyHuTUMIQapDIQ6Ew7WgWMDmirYpajbYB+KDnZ2QCKVBarxQQWKFxHoVLNakRAlBIkB7D9bhRY2rZpjpKaYx+KpPvTNG+JJEuZhH065my63zsclvige+zUC8HhTDV/pntwmHtzcd1mz0xTl0Dxk+esDDo132R11sNef87JinX35KtXzB17s81fjmjIActsYEFzirTc/Wov8cljUul9SuYDkRiCSjO9hBZF1Wk1XQZskc2AnVeXbG5Lu79Ed75J0+tqxYLKcsTs7xk+M3X4r9juNWgLGU7fLEDUYYQplLboDmdRU61qJ/Zw1NXWNXCygvwJ+FDAzjCOzvqy76A5L+J+zsP6TbHeFxmLoCPyXokTtAgjML6npFlmiE0EilAzPcssNbwL3VPLegeM0kX9NCE+zFt+UVkTF+eQH89mNIaMSXb7EPIQRpJyfJQyUUYxyNtZiYbS2lpK4q5vMZV+dnnL04ploVDIYDsk7OqliyXM6pG4MxluVqyXIxw5hvL86VMfmjaWq8Dwb0Smc486rJ0NuFh1AOtVmxWF5xNTnh7Pwpk+k5VVVi3+K+lUqxd3DAe++9z73DQ/K8A2tw+y1ahY/ds7/+WnjaFAO5Zsnmti3brtMf3vu1MuN7xsU4C2cvzvl1OsesHpP3M0gEp5M5nz6e8svPLKffQEXTlnC3/uvPrE5hby9hPmlYFVAZuLiC41MDeg4apE7IsgQvG7Rd4q2mMQmNS7AuR9QV0q2Q3wAYCwIg7rKZxN5GeCIbazdWbe2lJQJWGcfmxgeAWsZH3YRkOZWATSDrwe4e3BvBwS4c7ASNse5EnTKwqmG+2mh+W2Cs4ve046cnAssIkH3UJ5sGRBksD8smMMBWgkhYSxechfll+J6SzbVUBLDfS2FnCMMdGIwFWceTpCGJ721jRZSjbB1/UkNSW1bFIpa4v8IgWNYNjdmMQYIwuXbRPqPbG7J/EDzz0zTDOSjLBik9SRp8g42RCOEpS4l1JlhGCtBaolONzhPyLCfP8uCEUQU7wcPD+4z3dxnsjBns7jAYj0nzHI+naYKPr7UWY22wgYsFRaQQKKXRiY65JRopQ3WY4Ge7YUBvxVBJCLSUpDqQRDZqhTfmvpuQXiC9DbUGfBw/dUgId0LhhAx+t0TvYSFRwpMikNZhTQXNArO6wk3PoCypsy6r8QhTzpG+Rqs87FPE/ldsH+qWnEWIiApvlky6Hq7Fvde2ud4BOb/lLfeWIfUHIBRedRDpgPHDB3z4wQPGPYWm5vLqlBePP2M5+SpTgptxHRS/OtqzsILmnGYmaITEVmE8LtM+IhngTAJNEvRPaEg6oazzcJ/R7kM6vTHLyRVl6UIG7/c9+HyLuB1g3CopbsokEJsJY/xfsGULYFgQHR6EpPEOWxWwmm8BW0Hooiev+NYMmX7EYHxImnaRUgcrLucIXewVQWN8jll2OX2xi7/3iOFwD6WCT2Sob9/KPuQ12zUZ2e62qIIQbLHeWwwzW2B/vXTz7kICXZlQudag6U1C0Ovv0N0ZIdKUsqpxYolOE9K8gemM5XzJyfOnfPHZp1yenlIsZmitOXx4xM7BAdYHII2U6EzivWMxu6R5A22wQILzmKYCqRE6QaYDKC+57WmGEAJrDYvVjMurEy4uj7m8OmaxvKJqzFux1Hmvz+HhAw6jtljpUDUL+La4eNPfbgHgbcC8hsDRMHZbSvESK+y39uPdO5OpfFVkwFjDUAfQ8+xpxYsXJxgCOClMYDhnX9N8kjSwhr0MUhmTtGJilicyqjYUpthwVDDoa7raMrtwdHVgH52BagWrec0qL1ilc9ANmdNgFaYRGKsxroNdWYRdoeXXnzsLVPFx22H9RgfcWpoFyVoAi95vPIJJIB3AWIAfBXeKvoJsB/wAVAfyPOiLpQVh498Z7IwDmF2UcDGDMlql2bAquynqtQWOfZRQeBdAbT9ViL5DKQ+LoB1eLQNQt8TEPYKcQgNDESzmOjn0BoL+UNAfSXoj0LlBZ2EwuVG07I0j1i6hYSMPMQTo0FiormrSWY1Q4fXKtT/ZI3zIFPQ+aGGTrMOAwNpqrTHG4n0ZiZ9YBt61/s7x7o3JeNZaZHSs8M6FIcwFJXanM+TwfoZQnrzfJet30XmOkiJYX60M8/mcy8tLptMpq8WSqq5QSpEkCf1+n+FwxGi0w2i0Q68/RCdp8LEVAiv8uurs7WBjj1YCqWSw8ZPiFSyrDyIJJ5Deo5xFyeCvjApssZcBFDuCLCOQURbtHN5WmGpKPTulmp7iZidQLEBqZoMBy+kFTbUiSRPSJIvE8Eb43q5Wt+Tdq1Z5X2pfYtufud3mRlEl2mTB2wHGH//W7yJ0iugNSHoDdoddxv2UcnHF+ckLvvjVXzA9/+xbsrDflrpbYVetRCOMZd6cEVxHEtZsNhoqi5spVnnGcHhAmqQw2gOdURQzymoG1VO+j/oF3zZuBRirOOsSImp1WxAsQlNZL59s6XJFgKSxgpwHa3BVAWZKYIjbpYGaV8soRvR3PmQ42ifRGUrosH88m+7NAQ24z1hcvke3N2A42EEpHXQ1UbfUAvW1ywQRFLduFdcA8DYzzuYGk5vM33cZKZIsG2GaAmGWbzRAKJmys7fPYG+MSDWziwn2asKyKJjM5oGJqxo+/8XP+PQXf8pyeYXwkvFwD51q0m4XnSZBSy6DG2ddlSyWl7wJWPfOUZYls/kUlWZYBDIbcE3Qf2shsNYyn084vzjm4vKYyfSM+XJBVW4N+N8ypE65d/iAo6MH7O8fMBgMUFJhrVu3m2sX62v6p5Zlbv/bJNptPnfNlWLrse6ob3zHdhW87zI08IM+PLoHVR2W5k+v4NyFq7stMvmq9pwlcHQEhwfQzyHVYcm+qmKrk5Kq8iwWnuNTmC03O80zT7enGOeOxAUrM1NBuYRl2rBIVuRK43RB4wQYT1M5rEvwcoArBMqVZPKbDS0lode67VS9Vl9sImMsw8+OZenjwxO8+0WYPHTyIHOQQC6C3VmdQxUBn7WBFXZ16MvzDgwsrBroTQJY9SLqgU04b87E7422aj7kTOOaANKzTJJkCb3cI7WltobFKqzaGxPraYkAoHUSrkevB4MhjHYUo72M/kiT9iw6N1hMALFNPNZbGFvb7At14xnC8ZVxKNEmvH4NfrggrHbO4b0g0XkgXGRYaTQm+KUjPFrrMNrF97ROSKMWRKtQlW7b59c7gfNh3SHLUob5iLSjUXmCyhTGWeqmYjqdcnlxwfPnz/nyyy948fwFk8mE1WqJ1po8z9nd3ePg4JCjo4c8fPg+9+4/YLS7S38wCvqi1g3Ks/Xr3yxEnJwlOjRIJwVOvQyMw8QiegxbizQxeV3LaNcm8TIAZLcmnEB5hzIOQ0NVL3GrCXY1ATMDLsBZ6st9ltNLTLlC9PskOoxRbQfcYpBwPcT6uDcH+ZpzcAMYh4e7tlK3kSDcDkH2yQ/fR6cZ2WiXrN9H+hpXLVicTTl5/pjJ+ZffkTThVVN8x2b6r4F+6EjmhjrVFDv3seOGNO+hugOKuuCsKrHHQPmMt9IrfgdxO8BYAlFasG54wsW21mqXwkxKtBU/4vjsnMNbi2uWeDMHfwH8GviM0BW9RpspRnTzXTLdDQBbSLRKQl1tOkSDobjxJa65wtoKqRRKJXhatrgFwJskwWtSELH12P63ZH2D0cotbuNkfkUoJF3Rxyd9nBNgVq8+N18TiUwYDYfs7O9TFk2wennyBcZVgTby4K2hXF1iozZYkCD1Id1+n53dUG7aes9quWR6dcnJs2cU5ZvVKHNuxfTqKYvFKSrro/JdjAFEF3z1Rr/xdWE9lHXD1eSS45PnXE3OWZULiqWj/Kq6I18Tw51dfvDDT3jw8CH9fh+dBDSxYYfaNTixhaw2QLdlgKHtUlvz/w0o9pGhaik6T8DA3t2UU2wY5vjl8SkwUrfNwn9d9IGeDKDI1QHILlwo8/yqI2kBy/alyCXcG8PDPdjflXQyQaoFzonA7tWesoImU/T7Cudr5iuztgRzTUWSQrcL1GAKWFRh+VzjSMUSYTxT5xCVwa4M9cohpWS8v49IR2TK0c1Bzr++mThgxu2vH1kf2PEmMrMqAl6xNTavE4pdAK5JEkBoeywLCccxoa6poKjgoQ263b6NkgwdtMDIAEZXy+BgUZRQVwEXdrvh0dSBsGuqoBlWErp9R94rQQfys5MHyUYvD/uxHtIepF3QXdAdSZp50syTpBadrTA6HGNdb5INXcsWvyUpJ4GjjiSLzh2CYNcWzpunbhyzJkxu2rW5djTxgDU1tqmxxmINCB00sEppkjQhSz1ZliPwJFlCojXaJaQ6JU0y+v0hCEjSFJ2GantCJwiRhKNz4b5GKaTO0WmKzjVoRzGZcXF+yl/+9Kf8F//6v+BnP/sZj58+YTKdUjf12qtdRVlHt9Pl8OCQjz/6hB//7u/zN/7m3+K3fvy7dAZj0qxHW8whkEpvLt6WQpAnijwJ2gy/BsY374LAN0rnaUszhhUPj1B+7Wfs5bbFmkBZgQas9FjpQ8XGLEEmGSG1ZYUvp1TTGWa5QozNmr0O5ZCjAdzWqjbr+yJglK+6X/1LD3EjD+QGKH7LbrZcLkmtIc1zSKBYzZhPznj++AtOnn3xhs5N74IYMax786bBXUou1Oc4Y9m9/z6D/SN2948QCK56Xcrjn8HkMdg53/VY9E3j1hjjFvsGYByYYy88XrjNcO9d0EB5gfKhEXrbYG1JXV7hm0uC+8QTvo4pFKJPmgzQKhjzt6WJ86wDogdes5njl3g3x7kGKRVKpVgbb4Q4kEg2colrOuktQBxAPht2uM1AWQv7b2em+KpQJOyIEZ2sj1U5RjrelFFNdUJ/0Gd3b4+TF2ecnTxncvVLvmq493iSvEN/OGK8u4tSirppWEynXJ6ccXH2HP/GSyQOXImtyqBjKh0i3QGRg0+5zUXpxjiKquLy6oLjk2dMJheUq4pyGTH4G4SQkoN7h/zwhz/kwYOHdLt9pFRhEI9Ads1KxNJJTrJBtrBhesWmrbFmit0mwc63gFesQd82MG4/E93zt8A1OGeD3Og77Is6hEKi2kG9gnIFZRWsw151GJoAorMMigYqGwpT7A7h/j7cH2t2BjmJtmjtETIkqMwWNVdNjUokg3xAVS15/HRBbYi6V08igvuCBabTIBVIZcgZUTTYZoErLPUMyml4dFLLR791xfgwIUs8/Z5AnfuXTO1fF7d9qg1hmb+O0gfdziLaSx+bWNuOhAjEoFJhE+dhZoJ+uCjCBKUug1fxMJI+688E90ZMHRRukyksloH1b7u5NA2g+eo8XFvnwnf1Gug5yDqQ5tDrwqAf3q8NOA3pEJKhRnUzVCf0Zc4V1MZT1lG60IRHExMNpSK6PbzdeVTAwd6Abp7gvUdKiVprUQzLxZLmvGJhN3KL9lo6oDE1xjTRYcIhhQMhQ4ELoVGJIrsxLiqlEYmg0+kBYZWxLe/mhcQjMAaMEeBispzQCJkiVYZQEkPFdHrF489+yR/+03/C//cf/xNOLi83q0lb0WAo64rZYsbx2TE//9XP+eUvfk5VlnQ7Xe49+JDxXgrRli7UDnhzYKykoJNI8kRGYCxxamtGth4aRXRlC7/ZRa4s5tghdZRVxLKNXgSArIwgdYSS6EpQJZpRt0PaG1KuzgGHNwVmscKtCqS1JGsmOjpktNIRGU1kt4Dx9uj9MpTfPK/lFFEWc/3Mi+sbvkVMry7JshScwaxSppMLzk9f8Pzx51TF+dt/wa1GQ5hGllAbzLHholqi04zx/iHjnT06/TG90S7ngzGzZ3u447+E5jftd4S4peQ7GQFkbGhxhthmvRprqOuGuiyxi3DiVN6l0+1g6hX1coJdPgfza+AvCUU5vjq8X7IqV2S2IdUCmWi01HTyAUofYJsx13yPrcdZECiUVGs/Wbm+/zYln68l2631SKxlIFJu5BeilWG0Hd07CUmHnI5I8KZhsahY2vKNgah3DikkaZaTd7okac43AfTGQ+M8Vd1gmiWzyZRf/exn/PLnf8b81qrTODBzPBrcFbet1Jwvp8zmEy4nF1xenjGbTSmWFvM6+vIbRJrlHD044uGjR+zs7pEkybXecwNm478R62SP9jXwm9W8gHbx62pPLoDi9b5EZCsIc4oIil0UMfr1Kotbs8TORTvE5rvNCu4SlGhlEZjilYGFf/10TgOjLoz3A4hzVpAn0M894yGMcknmG+plQ9WECl5SCKq5o7j0rOoCp2ouJ57WWa2Xw/6OZHek6GkJeUMvdTQVoUxx1EI2IqEwjunCsziH1SUMUti/VzPcK0kzxWCUkqgqAO7vIVyUULTV6dqCXs4FDbHTIcHNRg2yJwBpu9XUvGFdTMO5YOG28sFSzcf+UGvI0lCeGQfNCmaXQZ4i48TFNgFUFwuYLqC0oRdJLLACkUCnA8MejMcwHIV86qUJ1nGFglobjLY0ssBbh7UBOBcm9Ddtol/rxCFE2Mc3kHp/ZRjglyczUhkmqjLmk4h4kivjmdiQoHdzjcUBq7qiriusDQC5LEqqqozJdwlKq/W42KZ0rccNKdeWpsgI0qRESI2SGUKmeDQORfBKTvBCUqxWXM1O+ZN/9RP++f/vP+Wf/rN/zvHFxTfutqqm4uef/Yz6H9ZcXVzxt/7gv8bv/1t/m16qEPUC7xo++sHffuNzqrWi10no5UmQHrSgVm6Y2HZlLOTkSZyXkaQCqQUqkahEIhMVuLS4DyclCkHSTsq0punk7O3ssHf/iOOyxs4VSb7PqNulpxS5FGQyeDuj2xXeVvIh14AYWLvVvQ4Yb479xvPrbCpuAeudnx6DNbxoVjhXsipXLOslpWlb5W9atGVqHNgaNzHMzh9QHE3pjUNiqR/t4Mz7OGuZVQWc1YScst+suDVgDKzBY8ustgkhtqmpV1PM1RUszsGvMKt7FP4AW87xi+fgPwV+TvBa+Aatyl8xX56Tjh7QkeCVRsiELO+Td++znB4BJ4QGpMBJvJEIoQNrHIHGWp+3BsYtk9z6LrNmjzeOFWIDitkCx++oJHRKRocULTwrs2DGivolU6NvHtZbvBMoldDpDRjv3OP4+QBrr77yc4UJnp6z6YzZ5IrjZ8/4xc//jMXi+I2O47Xha7AzXl3Q5e1iMr1gMj3nanLG5dUF89mCcukDWHiTEILR3i4PHj7k/oMjRqNRZIsjAI6xyXde6yjWbHHcDS2IDnIHdwMQtyA57MNHLeI6oSdaR7WfIz6cC1WHnLNY46ibiu8q+S5jq5BpLCFcsFmeflVoAf0eHOwrOp0+adJHuhppCzpJTVc5RG1ZXXgWM1DCoSXM5zC9gkkBU2uZ2o0G92AMR/cG7PQ1qW9IhEbf8xjTMJk1rCogU/i8R7HwTOqKqwUsDJQOFnOHsSVpnjEYd0l1xfL7AsZ+40gRLzeOUN5Zy0D6ubZX9xFEr+s1x9YXJ1QtwzyTQW/sovGykpAmkGdB1619sF6bL2FmA5DWOWCi7/AC5jZMYVsXjqqGtA7FQfaH8PAB3H8ENoXzCk5W8Hwe3CiaymMKH/yQo/9x5cLB5klg9J3cWJ86//a9rAdeNNdpPXHj/a+KVV1T1QWNbWhM0PteXFzirF3rjDf5Nn4rqVuuVxyFkME2VCl0kpJmOb3umF5vhEq6iCQPeTAqwXvHfDrhya8/55//Z/+Uf/Sf/mMuZ99euma95dPHv+LyP75kOS/Y7Y/Z7aeI1QXelvB33xwYJyoC445e93ZOEKWG21xBKActvMQ6gdFRKqEDKNapRmodqt9JjxMCh0Q6SRLHXp0k+G6P/d09jh4+wjnJ5fMeo50jDkZjBllGN1FkSiAiwywiAbZJqt+wu8EkQGzsnL9RvK4nFbcDjE9e0NgZxs0i2fFXJeIyj/GUl2csJhekaUZnqMizlMF4l8YYiuWSZlnC8pfcfi7R28WtAOO2fcktr0CBw9kaU69YXp1hLp9BdUpQ3jmwKVLs4hOB1Qk029rgbzLqXFFN/pjTxlKUhv29h+yP9hCiQek2Y/J6OBc0wVKqqMEUCLmlOeLG87oD27Jk2wLF163e5Dsr8JGQkEgVJhm2oqF5q/vOe3DW4wwMh2N+/Lu/R1Ou+OyzP8Ta5Ws+ZZlPTnn65AnW1Jw8f8bxs1+zWJy+xZG87gBN8Hh6B3Fy/oSryxMm0zMWiymzScEbSqMB6PT7PHr/Qw4PjxiNQkEPhIi+xe06qr2mXvMugl7ncNauC2+4mPkn8NFOcOuL/IZ5bsukRkOKNRscHhbvLN5bvLVYFyodWmsxxlAW5TVA/q5Csrmb2wHREBi4113ZnOBasVzBk2eWbreg1xWk0pKIGtu1ZLlAOlgt4OoMaAJr2tQBvHm3EVANgJ6G+13o+QpZVDR1g9CQ9UNmfqIN1J7FynFVFpw8a3j2LPjvaqDjYLGA6WxFpQVKpdGe8vtZ/mtZYC8iOI5AUhMkBsRiG60ErJVPbDtYCKIbBFCIcD1KAhitaqjKAGgSD90MdkYwHkLvPJwX4UMCXkE4785AVwYJTDeFfgf6w+CL/MF78MH7MDqAfAQzH5wuLms4W8HpfFNKWkQG3BOcNaSI1ffchjGG8BtvWvbeyrn9FtvWxlCbGmMqjClZLGdcXJxS1zWi9dBrb39axV3r2y83K5BKopOELM3IOz3cnidJUnKdEkwaBAjPYrngV7/8FX/8R/+MP/yjf/lGoHjzOz0Xs3P+5R/9Mx4dPeKjeyOGsiARb5fRqJQgTxWdTEUZSCvr2SSmh8mNC64ezmOcoraBvVVaBOs6rVCpCsBYuOBOIQIwThOFylKwOdr22BkN2ZmPKZcFvraMR2OG/Q7dTJMngkw5pBJ4LdiuUrtOno/SNsHGcvmbA+Ov2HJtnfnm4azBuuqvGCjeDoOvC+rVnKZakpscoRK0UqR5l3SwTzN6CMUFuOf8JkkqbgkYt5KDzXIFCOpyRTE/oT7/HJq/JDC4GhgBhk6eodMuVS+nmGTYZQfcr4Av+PoldAP+p5jFGVerE8rp75N8/GNcs8TYGaG731KFeRuqN60BbEg1kBGARJ74BiiWG5Y4sgAyapNknBVeKxf9rkpCyyiEFOAa89bNxyMxxlNXNfcf3OMP/s7f4f7BPv/Pfzjji8f/ilcv03ia8oQnn1suTl8wnzz7ChD9ttHCqNuPF8efMpteMJuds1rNWU7tmyf2CsH+4REff/xDDg/v08l7KBlkFBv3h9ABe0nsLD3OWqxtaJqGpq5pmmbjPxrbWpKGJB6lwnJqENsRdcStlCJKKJwLpTydxRqDs+2joWkMxtQYE4BxUSy+kyIfsXgpLVmZsPGKedXw2xXwwS4kGj47DZpWKWq0rMkUDDJ4cAgqT+lowbJomE0DkLM+gOo+AQw3BKZ6V8LeCHYUiMuSZUwSSzKodwp0V9M0CmM8Z5eWk6sVz09gHtvDTvwN8yWcnBlMt6C2+jdjFVNE9tiCaCIwFiA0m0JZYou1i+8roqY6vmYJxTdKC8saZgvoZCATwMMgD2zvLLK7/lk450UdKroqFVwsjnZgMAhWertjuLcf3EMOj2DvABYCzir4YgK/eA5fXsD5HOZl8JlO03BcCYGxbouIbBcysa2mwW/ImO8rrPc0JpQIb5qCspyzWFxRFEVYvfFuLddDuLCCqlSoHNpKDaVEtcA4y+n1BnQ6Hex4jJAOnUqkBovh/PKCn/zkJ/wn/+gf8fjF81v5DU9Pv+Qf/+P/D1cfPeSTwx6jztvBAakEWSLItFwD4nbpuMUF4ZT4uAQAtVPIRuIJUgydKJJUoRIN0kW2WGB9eC3PUxJyhLaksmE469Lv5vQ6OUU3o9tNyXNNngpyDR3tEFoidGCOW2JrI/kM6H0tpbiF1QjYXiF88+h2OvhyQWW/euVUvPQQ194RKDwCS1x9ZOtGeqehwjU0BaZeYZouWINtgp496/Qpdw6xkyNYTQgJfL8ZcWvAuJUftBIK7zxVsaC+egbNLwgyiRVhyBoiVEq306Pb62DckGWnz3w6pp7u4atd8J8RvIy/ahZrgefg5hQXpzzhEoGjXPwSeMbmwguQKTpJkFIFX0kRsp/aTNXWQq4F+c46jG3w3gVzdK1J01A9TxCSB0L7a4F00Im9kx5bgPWGyjU02LduzmFAdbjGkirNaNzHPHrA0dEjnjz9C6x7HafnqKtz6mrGb9rSxzeN4+PPWS5nzGaXFKuKN6hHso4kyzl68Ij33/+QnZ1gGyiQmwpTAFiMNRhbU5YrqrIIhvxNg6lr6qbG1A3GGpxzKB0GgDTLyPKMLMvJsg5JkqGkRoq4TBnBdwDFwdzfGIM1DaYJz9bWmCYAcGNMBMbFdzIvd0RLrvZcEQsqvGJbAYzT4FrQODCnkTWMwK+yMK8DeJOqYZgLzi/gqgpdaUMAsd34UIQl/fsd2BuG9t5MgmuCKUB0wHYA4SkryWwBT57Ds+nGqxbCMWiCA8RiCY6G2le477GcaVTUAOEcWxsY21A+GESU+LV1BoJGk7XmExGvidholVcGphVcLqE7Cecvy0ICXiJhbyeA48UyfO7yIoBkRLBa29sPMon9/VCMY28Hjg4COO4MQfdgcglPz+HTM/jsDJ5dBFeQykBmw+pCpsArQnWzyDaaKBUysahJzHH73oHxvK758tlThsNdqrri9OSU58+eU5ZlXA1qZU8ugjBQSsaJ7jYwVmitydKMXrcfS1TD/qFlT2u8FhSu4unTJ/zkX/0xv/z1r7C3ZLloXMNffPaniMUZffch7A7ean9SChItyZIAdEXUUrZE0zpcXPawDmUkIgn0v05kYIyT8AiloVtgLNEoOk6TihSZGBI69LoZ3TShk2nyTNPtpAz6OYNBRi8X5MohZKhXLqWKhbg2hNY6PIHp5ya4jK+sNUjfLALsfLtGurO3i7ooOF/OvrLPbsuzp0i0CFhFRFlnkF75sOLiZVQBGxwWj+HNE+a/SSQoLL6a0yyvKBMJKqVqDLYySAE662A7I1iNCWtQvwmsw61rjFtgGWYmzWIC5a+BT9ksoCpgnzTfZdAdkWZdjPVoOSTvHFCMH1IsP6ZePsXMfwbmU0Khjq9iEOfg/zXz8882/762fYrO9xgMhyQ62uFE67gggWo1xJtM1bqsmUwmVFVFlmV0/v/s/XmcZdtW14l+x1ztbqOPjOxOf85tuPdyEUQQEaRsEFERrZJSOsW2qnxqiU8Rsa4Ipe9pSanYYFeowLMpxbKKHvXSCBcuwu3Ouff0ebLP6GN3q51zvj/mWhErIyObkxkRGZln/z6flTtj7dWOPddcvznmb4zRbtHv9QkC30UvV3rRvYC8vaIlhw1jShKdkpBx77RmN+3BrcezjkgpLJPhDhvXLnH18ltMkgmeF92BGMNubuhHFKvrl0iSCcPRNumkeKBc4/3ZBc6eOcfKqdN0Oz2MFUpdsZLqZViWJcPxNtvbG1y9cpFrVy+TJRNMWeyWJ7dVOWcLLil9FOCFIUEY0u3PsrC4zOzMAt1Ojzju7HqJS+08xGVZUpSO/BZ5QVkWjhiX7v9lUaK1k1PkeXIsUgqN0xIb3BCqwJHig+ciXM7cSeLShAXKkeH922xMIH3N0lWWvHDHqz3QCuctPgWcUTAbw8yMy3W8U6Uk831HlOMetPsBGQGj9YRLVyyXd27NOezhCFoQOq9mqi1pMqE8ynfJXaDLKoevdiTRVHJyU+mpreC6Pq+ydUUwd+WU4jzIWipPbAnbKVwfOo+7LSFLXKnoOHJNJQ5dCWlrnP772nXY3HJZK+KO8wqfOw9LizBbyS7m+y4v8djAVgIX1uGTF91swNrQBeCVFqgCBbMcCCr+IZX0w1YBmMYFEJZlVbbZ43AqtT0AJqXmv77yGpevrbE0O0eSpIzHY4w17L0FqnIPlcfUqzzEu9JDpZyUQrmA8CiMmHnrMnOLb/DsC+/jhc/SdGfnSU3JZz7zEp955SXKBynNeQAynXFh/RrvGi7Tm3kwYiwCvueW3aqyVQ3xXY+xxellDFgRJHDE2AJ+IPiBIvCd3liqyFJHjAVfhJbxCMRz1WtLjyhQBL57tgNfaLcD5hd6LC72iToBgVdiKTEalPj4Xoin6sBIUxHISqJmG1RW9uRIdWpAWwdM32P3+aC97JnzT9DxFcmb64zs7TOiGxyb8VVMEAQuVShVzvqypDRlVdiJipnpXXmGd5trrr3PVRjDfcBDVEBAhhlvkCmDycbghRTGZdUpC4vN0oqOtcHE3F5od7w4pMp3FRGgzt6gnGejGOMKdezs2yMmCGMCFRBIjHiuw4iU0O0toU+dIy2eY3P1KcY3XoXRLwMvc2d5hcFVuzsIbXpzp5idnSMMospj7FVSCoNSdjfLhEsIYJhMxqzeuM54PKbb6zEzM+OuMQogCKopGW+PFKtau/TA5rwFpS0oyRvN+e4QNymJ3S2hsAdtciaTETpP2N64wRuvvMzFty6wvb2FqHspYnvyK9fcDusb18nSjMFgQDa5/9GpF7ZYOnWalZXTLC4u02p1sEZwhe5sJYmAvMzZ3tniypW3eOnFT/Dypz9FMhqC0XgibqDlebveoyAKCKKQIIrwo5i5+UWyLKUsXX5ppVx6KecBLpxHuKwlGaWTZ5QFpiwcGS7dUuuPi+JeSoIeDix74ZN3s3RmXI7cdteVAZ4kB+8z1jDRe8UY6knBrDqXApZCOD0LQey8oqMcbAqtWVich9ZsgN/psTm2TEYT1oa3kmLBeTF7EfRmodODIqUq+nFf5nhwWEeMdVV5rvYeu+DLvV+1Hg6bWuNaeYoNjfpVtqruVsCWdiK3qASTuYC6XsulWGu3nLRiputkE3Xu4rltZ99OD06twNlzsLjgpBTddkWqFWxuwtUdeGMVPnMVLm26QY+ur7f2ChfO5l59vVUGiroEdlHdt1/xGO9owjneFhJtuLC9xcXtrZueKDngs17U/vVSBYThHDS+d4l21GVrOCHqzTA7GTOYjPnMSy+yub12JPexU2Rc2dyi3es+0HGUcoOW0N+TIypP3eSddZU6Heu04jJRqMBFIriMFM7r7NfHEMGIwljBRxGZWikkmMDlMg/86jMQ2q2QufkuCwtdxAMrBaV2UjNPIsJQ4SvB2/Ucu1gQi6U5ESRid7XILvWlG6255+wAcnzQe/8Bu9nZ+TlUmTF7bZZJsnrbPtQRXqkKo0hVKt5gypJSF5S2oMA0c0bchGpy6RZJBggaoaSu6CeVMMP1vNldEgAEnsXXCWa4QVak5OMBeCFaPEqrKI3C5hpM7jwi5ta4sIeFw5FS1KlPaimFAqxX5Rjdf4oxcIHJzhIbN+boz54ijPsU2jBMhmRlCiFYVaCNgnAB5GlcktnruNdflZz0nqBQwdOcPvMMywvLtOM2nh+irA9ofKXxRFeBKZY8z0mShK2NVa5deZPJaMj80il8T+N5GqtTWu0OrbhFHMUEfoQfCJ7IA+fWvB2KfQOCXa3WHRB4LeLWDKVo0mQLU+7pdywFN268yerVtyhLzS9+5L9wY+0ixnjowygpdYIxHIxIk5StjZTiASRNnV6fU8unmJ2dp9Pp4vuhIyi6ynVdRXTkacbG2hpvvfUmb7zxKm+8/hrZJEGsK50axz7tdkyr1SaOIrJcoRKPTr9PvxVS6pzNzXWsUXj4KAmx1qK1doS4yCiKnLJwRLkonXfY1kF4xmB0pS0Tw14liOPBvT6lBkBB1Ib5Oad73U5vbecRMI/TJAfiPJJjHFHOcKG9gwzaO+BNXIaG3IOgB/1lxfITc0SdLqkRbDIg1zfLJ2rEwJMxPPcMLD2jiBdj1I2c9ZGbAnxYcSJl7soz58alNasJpoZKg16/KB3/UGrPK1RWkoS8esTzwi0WaBtQBZgU8pHL99xtOZLbabnUbb7v9MNRDGcK50mP285LPDfn8hS3YpdreFTAIIXXrsOnLsFr15yneJhRvbjdNQjs5q/1lfMOG1uV+LaOEBflnqfcrziJ9/DULLdgf/N5W5dmd/+pRjkZFBnFp36F5TNPsjQYcGN9jVdefpGiPOw6ig6ltXzyrYvc2LydY+ne4KQUbtkNLtwNbnf3aGu3ccU5fV/hR57LJe0rPE/wPYvn2TrFc6WVr8pHW43oAqNdqjwRTRQoWnFApx3R7cX0ezG9TkBZZuR56mbQdIkvhrAVEHg+Ino3k5SIolFjtLoZ2T33TRmD1G6oyM04AmK8tb7KZDjAIFRjxQNhgRyDLsfYUqqB8d7d3ElNXJPlJvZu5SaL3LTP3f1/mrLYJtc5UqT4+QSJxlgvROOIcaaFojQw2oZyk4MrHD8cHJKUohpfVCM8z8ONK/wQ94pp/qwaeAuje2yudtGFZmbekuQ5G+uXyLMdJPSQ0HNTAdqCmgf9LuAczrczwGmIN7n7q7dHu/MMpxZP0+v08apM9x61l9tUonuNtSV5OmK4s8PG+nWGW5cw5ZDBjtDpBFiTkaVDZvqzyNw8QaAQCfCURWyJtcU9XM+DI2SvJuDBEOKgw9z8AlG/zfpGzOa112k+HqPRKtcvv8loNOHq9ZfRJsU196ru+Qkv2Xi/mIwnjEcZ+Q4P5Phut1rMzszQbnfw/RARVWlPTSXJceQ4z1M2N9e5cuUiV65cZLwz2v3hCiDLNKXOUcrg+1Dm2lUFa8dEUYCnYDjYpswMcdQhDNqIuLLWeZFTFClFWbgKXLqk1Ho33dtuD25tIw/3g1rwaGBxHkYvgv6c+39x3aVMA9eDxMD5EM70oBs5ycVwBNsTR8TG1pHkLQve2HkfVQQsKNqLAfPnljj11JOowGdzZwfZTpxWQtlbHqa+wBMr8Oy7fZaeWSKYnSGVTS6vbuKr8qFJ4YpyjxiXei+Fmdg977Gh8hbrKpCtkh5ou1dSWldeWlvCjoWgBDJn99KH3tjlf+62nfd4pgfzM9Dtwey8C2AMQpfWLYqc3CKIHHnODGxN4PIWfOYKfOoCvLkOW2NXtKX5rq2Jsa9Ae3vp6OquP82rgiBV3uXAc+TqJBHjo8DqaIM33nyNcTLhwqULXLryJkc5GtvMUrayB4sbUZ4QBoowEOoASed0rSUL7GbWEXGzata6wDprAc+lr/M88JTBqwZ1tSzDagCNMS5+wugMJYYgUMRRQDsOXLq4Tkg79sgSjTEpeZ5iywIVKkLpErokyi5WqBo87hYmralgNYtcy4vraze1t/h2xHh34wO2eZtYu3KZPEvIdHkXalwT3EM46T0c4V7PYtDkZghJRlmMUWkLqwJKPLRVFJqqHv2QOyfxPH4cosa4OUoUBB/lBzgfz35X6gT4GHm+xvrqc2xvP4MxGbp4DVjDpnUOWJ9aVu5eiz2Uv4JSOWUu3Dn5U42EyegCr3zmY8wurBPFswRhjAp8VxwAjdgSXWQUWcZoOGQwGLC9veVGqX6XwBfybExZTBgNPZQYZme7KCkpTUKZTEjGY67feAujj37Uc3f6rXZHjUEQ4Pu3TlGUOmNt9SppmqNN7ZFuTn4/nsgTQz60t86dv01sr9/g4uuvsry4RK/Tpz+7QKszg+f5KCV4CoxnKcqcosgwpiSKQ4KWj85LN23sCVEktNoBQRhiEUbjlPE4ozezQLczQ6vdYXtzhyyfMB7tsBO3UMo9tqUuKMt8N6NFsyS0II0Baz1oBc8LuLc5h+OFxmlR7aCqbhbD8jzMJK4qWyeA+RaszLgCIFXVWbR4FDZmPBSGAyiGOcWoYF1bOu2A3kKXuWdPM//sWU6dWWT21BzWluj2BqdUh3eZmDJe5eOvJzflJhZAAvDaLdrzK3ROnWYxn+PUoMXMG5fY3no4zNjYvcp3pXaEUSpZhWcr/bBht36MZo9IGqleoHWGB02dbpSRB6G4CoUa6CbQDiqvcctVytM48trzXDU7X7ksIr6qPHsVWR9lLtDu5WvwylV44wasjpyO2NaTfZbd4EDFXmBgHb+sKx1iWrr96krmGncfJ0FKcZQw1jAej9hcX+PixTcZJvvliIePB+0R6sIwfvXbKPbIZ01EbdX1KAQxgvUUptpBPA/xFOK5vsuTanBbBcvhCdZXaN/lcFfKpXcLAw8RQ6kziiKhKBOMyfAoiDyN9V1BkZYPsWcJ1F4ZZxd871S6jsU7YuykIOxKXJAqLabdI/c3Yb/D4RC62GtXX6Y0msxkhxBy/3DgYkxyN9VVDrlVYAR7uYtODg6nJLS3J+GuczOigooYhxzseM+BN7H2KkX2Mo7urXNrYJfgyPEK8BxR6xxx7LG9voa19yLWzjHlJ7lyoeT6lbOE7SWiTo+wHRMGgXtYS0MyGpOOR+TphLJIQEI6/Xk6nQ5KhDKfkGYZWpd0uxGizqA8TVGOmYzHrN24zuXLL2Pt0RPLu7+SPYqyIE1T4jzHlLdqgSyW7e3VqurQyWqURwmduuVBkaYJr778EnEY43kBZ594jlNnFVEUVXphi+cZ8jzB2pLA95hfmAMpMHmO2KrT98H3FJ4XURaW4TBjMs5JJppue5Zuv8NkNCZNMiaTHXZ2fHw/wvMCbFXiuU4N1SxSsxsMqvY0fgIHDpIeJurWp4G1sctz2+s6TevKeVjsxSz2uyzNdOn4hshMkCIhT3OsUXQWz9JZeJLC9pikIVfeXOXlX3mZzY0h3vIKC08+yakPfhZPf+Dd9Gdj4sCg8yHS28SbWaGz8jRz568wLn+RT74+3n22SluRMhvgdZboLj3NIsucLWZY/sw2l7Z3eBjJKUzl8S3KPZJpjVtncETTwO4cqW5KD3xHkmvSjMFlvbQun/EAJ6coCqevbgl0oooYF1AoyBUU9cy/dp5qhSMvBkdod0Zw8QZ8+hK8eg0ubrqsImV9XXV3FLjF0pB9eI4YG+2uI8srQu2koI4YN+Qhjy8sZVEwGOywMdw4cEr7pEFV2l3fc3rcPe00N6VxBVBWOXmFVRjrVKviqarUoSOpvlgnUaxlip5AoCi1D9YjCxRB6BGGCtBk2YRJMiRJh+TFmJCcVkWKA08RhYrYFzzlAtOstVUmiipnvFK7fG1X/y17wZNaLNqYqvx2/XvsuYhv+oUO4ecalA+g9TuROByP9nHgENO17f3fTUF4eHEMXh90xO0D5zLg4h2O7hQ0TjZxnWyiKFKNta/huvK7Xh0u6/0qOi9I8g2SQYx4vsuvaAusKbG6mqujjRf06PQ6zC/0CYKQ7a1thsMhIoo4jqrAAIM2rizocLTN+sZ1trevcjIyNhSk5SblxoRxsk2eH/yAFXm+lzT/HQIxAXLfiYtvxmgy4hOf/GV2RiOeeOYSZ594hjCMsBjnDQ4VeTHh+vXLTJIx1mh836MoPXRRYrH4ClJdkkwmTIYFk1GGtXD18jVef/1NTp1aZDQaYo0lzxOybOK0btalGTRVBFbt1VBK4SlXxMZT3m5xGlENYiyyJ/J8yGheRVk6p0JROs/TudMBZ586zflTy5xenCcWTTncpBgPyMYJ1nrMnHmG2XPvwgYLpLqL7V7lyo4lbW3SOnOOztPPMPP0e5h76t3EsUA5hrxLO+xi4zmKaIde4tHqfgaR8e4FGarCHtspo1HGjFF0+jOcfsLn/FMLfPq1HUaHW638nmCgSgfI3ojCcFPmCQW7VfEwe5kdROPyHNf7NkLOS+tcDKry+ka5C8ZrB84DnFun1y5lLyAuSaGTuEC7MK7iZxSsDuDimtMXX9qEzVHloa7KS+92kVX1F1F1JgPqaugu+K5aGhXO0eKu72S03qOFsS5Y1tiTM8V8J3gIoVIESsDIXlBXNVu15yusBu/WeYytVc4LawzWloCqJJkKX+rCHE5SYX0PCQMsEaGOCQPfBXmpEqHAmgxTTjDFCKtTpEwJLfjiESlN6Fk8z5WYtsbW02kunVztQKgvtEorZ63rq5UCqWQYBzmanDdc6pvc+/8UbwtHINd+2zgkYqwa/6dqZIqg1YbWAoz63BuJvRMS4AJGX8DoulTA7eDjfB51yv86hj0DJmAKrCmAFLsbzBcAMyjv3Zx78gkWlpbwvYDhcMRotEE6WifunmJh8TT9fg8RQ5YnpMmE8XiH4WCLNNni4eXha2qQLFBS6hHl8A7SDhHsYx5stx+BF+OpnPKQ9EzjyYhPv/Qx3nzjVfpzCyAKrTXKU8StiCgKCCJBlGY8GTAcbTMZpRTjElEQtRS6tCQjc1PTGQ+HfPTnP8Lpcyv0ZzrMzs6gyxxrSwRTTU8Knr05VaLyFJ7y8DwfT1X5LBvpkjzPP4FCigYs5BMY7oCxAd2ZeeaWTzF/apHQapIwII9aRJ0SJGDm1Hm6S+cgWiSiz1zWYun6COlts3D6NLNPnKd1agWv38d6Bp1aEA/f7+DRZrxRsLYDO2N1iwd4OIDVKymLV6/RXTlLPLfIyulZnnxmgeX5C4yuHf+zvnuN9Y9Yk0ivcrY5eSeZZa/MeSWbsCVYr5KqVJrjemobC7k4jbYG0gL8CSSe0wWXgPahECflSAsnb2kFrjhHWKW0kwCujeGtNXhrs6psV8/Q1GUQC/YmAnFe4jBw16+rx1JZN9cYijtn2ST6+thjSI8ddTkMpyBQu176kwxPhEgUkSin0bW157VBjCsZhZNeKqxSWM9DFwU6K9zMV1iVjFY+SgV4qsqEUBNjQsSDEk0U+gRKE3gloe8+PZtBMcKk2+hshPJDwqhDKB0CZfB8x09sVZrPVtcCqgrIA2ssutSUeUmZF2ijCVs+QTuoay1xs1h+TxpgbS1hezBiPIdLdfl4ixtvRt0t1NashyBvJ93CYeBQS0K7/+81+iju4s+sUE7OgrnGg4mrNXeWTfg4IrwI3hlUsIwXdFEqpMwmmHKI1SNczuNa6J2yV20lAzJEztLpB7Q7PuPRhMFggyzZALbx1DLdXpe4FWOsIc9S0jQhz1xKNM/zH6K/+M7i/IMwTsaU+p1FjJeXzhAFbVavXafID8flZ60hSYYkya1lWkUJUTemP9smSSck45QysXsBeHXC3gN+up3NbSaTEctnFlDK0Ol06VWFQCy1l1iqACblUg56exUaXZXGWulXe0IeDS9GlkCegohPELVo9fouaMYavCDCFhalIqL+EirqYYIOhhZ+Z4bu8hmKoEtncYFwbgaigEyXFMZQlhZlfUI/IrMFN7Yy3ri0yerm+KbBggfYAoqRYbK9zWR7g85sn/m5PmfPzvH0UxHX1hKSY45RNbU2uPaiVhct4uIIvWo2uNCNSnG7O7t9re/IsbWNHrnyQmfO2YeyoCqNb6qrgL2KoOYZjMcQ+RBWulIvrPI9x7CewdVNV9kuS3A+jEqDDOwlnq7uQ6gq8ZkqGwVOp+rjtOSBaqR3q4IGbwlbeczgiSLwfUQJnvLuuWuviUWziRwXbFFg0gSytpNRVO1ojzPu5TK21pWtd1qegjxJGG8PyLIcghAvjmj3erR7PQLPc8fRGlNkWF0AGilzvDIjLFPatmQmUHQ88IoMPRpgR+tIukPQ6iBWU8Yt0nSEsm5wbCzo0lCWLj4DA55SBJ7Clpo8KciSjDRNMcYws9BnRs3gRx572Srczbk89LKbDsYaW0ncLG6I9/bRxjXz2+V/fxwR4Fyau057qv4AN54+Ln51uFKKZjUZK8TxDP35p9gefAAzvAJcPozTNeADs4h/Fi86Q9g5S9w7Rau7QLs1QxTE+MonzzKyNCVLR+TZkDzdIU+3SYcvY8qPsud9LtDlZVZvvMVguMnm6jrpaBujE8DD8z38KqtFWWi00aRJDkYxP79MNnmKKxdDHk5VuLf/hh7nJyc9ynHhc37V5zMej7h25QqXL13i+pXL6ENOmt+ENZZ0kJCnKaa0t/Zwd+nxirTkxtU1lLK0Wj1mZhYJfI3vV3TXzVXu5g3dVbwZg7ZOHlSp6JxGs3zwkuLHgbKA4XrG9tom45UlSmOIo5Cg00G8wEWB2QgtEaNRwUQPGBUJW9sJ1vOIOm3whSxP2NleBTXB85yrx1MBURBzY2vMi6+8xX/95RfZ3NrZPbfgQn3nQpiLISwTzGgTKRaJvRmWlzo8//wsb7yZ8Ob147WLrSfLmi4U5YilX2V3EAFfOyK7e0Ow62G2lfa4DmhrMigrUPhACBJBkTsiXFaZIvIURgNYi/ckmXiObAeRS9+WWFjfcXmHKXFvsyaRbRJk7bbTVbaJOvNEO6g0x7XMopaPGGcD/Sg04vuEAHNxh263TZJmu6nO7rZPC+gKtCsdeGLcPO29vBn2/J33j3S4w86Na8TWEofhXsGs6gS2ms0w1pKXGaVOUeJSpo62trh28QpbG1vk1iBhyKlz51g5e44oiBwxLgpsniBljo/G5Al24zrRYIu5ssCGEX3lE4wT8vVN9M4aJt0m7PUISoM1gslKTBBj8ShKw3ickEwSyjzDFAWR7xGHIbbUJKMJ6SghTVKUp3jimad40nuadr/jquh6CoW4vPJao+uoVwt5XpBlLjCaZ2buy541IY6r/z/uLiyFm99vs6fyanZROY8oMd4L8nGPWRT28GZjsqUR42TN1WNl4zBOiSuT9H5ac+/j9JPvZmbxNN3+IlFrhiBo43sxgQrxxMNojS4K0mRAMt4mTbZJkx0uvhGwdeNlXNBfjXW21l8EYnS2hiO5XWABowvSLGGSTEDc1GuWloDP3GwXXZzj6qUQa4+GGHs8/g/HUeM973k/RZHz7NPPc/XaVf7LT/8Ml956FWuPdkxu8vt/k5epYe36FjP9VWZ6C+SZJY5zfN/H84QwDIiiCD8IqhyTbsZGodlLzO5QlMWJ0RffCcbCeEuzcW2NjVNzLJ+aR6QH1iBBgPJjxLbIjE86LhjllkGqyHND3GkRxDFhqFCeIUuHDDbHKM93IlsVoVTCW1dX+fTLr3Pj6uVGCe89kjHfg4UexKIhGyHlhMArmZ/r8NTTp3ni/CYXb2THS9KqTBK7HYE4GYXvVR7jWmcs7BW/bBLj6lr1flJcJyf1XQAfAdjIyRbK1Om+y6o64dbQlYKuA/kKBYXnpBTtljvGoKg8uyV77sv9bqAq+M8UTi5hq+sKvcpb7LMbpCrSuM7jnlc9ZrREMdftoZSQF9k95ZZvA2d8oRP5hJFHqUvSoiRIYNPenlBIte+8r2h5D0YHxoMNLr/2GmaS0O108T0PqeqR10GhWqC0miQfk+VjAt/SCizb62tceO01rl25xiAZo5XiyeeeYzB8gVYUozSoMkfylEDntMTglxmTjVXs+g1aScG8gVZhYHtIWhQUw210tkNWGjyrSCYZQ9kkw0MbIc1ytrYH7OwMySZDynRCHPh0WjG2KEmGI9JxQpkXhGFEku0ggWF2fp643SYIfJdGzliyLKcsCsQqxAqTyYThcERRFHzJF7xwX/ZMqt8nxD0PQ06wBO4QoHD36XPzhNjDuOdDk1LsVbepPxW+F+J7XfqzT5DtvJ9ycwj2I+zJFx7kpKeYXfk8Vs69wKnTTxB3Z/DjFsqPXU10qSaRrZseEc8jT3M21tZJ0x08leMHLZAzYHfY6zpydPYSe3N/irqG92gQ8NrLGf2508wvrNBqdfFUQBxHhGGXOO5zlPHSU1L84Oi25zCmpBV1CIKYnQ8MKbXm2uU3jpwc3xb3IPpNRwVvvHaBwU6CYy2KIAxptWMWFxc4e/Ys/f4MWpcYYwmDiCiMnKZYXGU9BMri5OWnrnjYbgbtev5Gj+DKa9t4fIpsvMH84jztOKbT6tKJ54njeYzyEC+iHYZEcZd5FVKKB0q5QgGiQadYnZLmJeM0ZWtng9W1bT7z8qtcePXTmH2FE0JgNoSlU7B0yqM106UVt/BViLUeYdRhdvEUS6euEAU3mBxN3YUDIdUbw9aEt/KoRp6THJiqSlzZzIDUFOvVXVqtvNofYG8a/6+zRvgul/C2hknmvNG+8we4AirWSSCkBC+tsxBUeuiDcktabpJS2AJSU92ThtJzBT2iwGmWtTgvNyV7RPsxZgiJNVzdXKW0MEjGpPrODUyAjkA78AjEImWBsgYf90zF3J4Yt4EnWyHnVpbotDsPdN1ra1f40R/7Sc6snGGm2yMKAgJcZhzrKbSCHEtmCpJsRJqPaUVCv+2TDYfcuHyNG1evcXX1OuNszEsvfYLllRVCP0AZiLH0laHvwVyg6IimGI7IRmOsiiDsYxPNtgQUs318KVFeRFpAMRhxdecab97YZHMwZpLkDMcJO8Mhw9GQLBmh00mVcs4DrdF5gTWGQHl0Ol2url/ltUtvMrewSLfXJ4wiPCVobUiThCIrCL0Q3/MZ7gzZ3NgkTTO+4Wt/233Zc8zuWPURyudw/yiBNfaS8zYT/dZj4uPCoeUxrvU2sisoUngqRElMt71EsfQCW0WGHoyBT/BgNbEFPzjL+SefZ2n5PN3+AuL7rmSycZMOVkogch4m6yrrjEcjrl+/RpZs0+64BJpB5ymKyQRMswBGsyfXuLFaCTYlHe+QpRPyXFhYUPT7cwR+hzDsEoUd55F63FvwI4wodBM1cdwiilp81mdp4ijmP/3kkO2tG0dyTk+B57usC6bZtDyIOoLneWQTjc7tHdvOZJByaXixcvg68Z5SHjPz82RpxsrpFfIsRxtDr9uj15shCmMCP3ApFUUoy5OQNcVxrg6uE4xwHto2zjuYS+UU1bB+DXQxYTx4g4WlS8zPd5mbW2ZhNmemLwQtD78V0+kFxL0Zws4MKm6hAlfdijIlGWwx3t6kGGdkO0OuXbzCpz79Ci+++GmGm+s3XZcCZpQrb7x4RrFwepZwboGwM4MftLA2xI969OeWmV9aph2vMcmPb0ClcKnXCsBlIXGEOPCdxzgrXTU7XZNHaXw2hafNz9q7DDd5om9KI2/BlE5zXGeM2PUw1/tpl64U67pBz2OPyDaPDXtuoSpLxW5tpCrDRppBHkLQdvplU/COEVtaYKgL0o1raHv3RG0e0BYh9BRiS0zh5FPK7v18txt7zweKc6cWeObJ8/S6vQe67rwY8vO/8DN0W0vMdvt0goBYeXi+wvoepQeJ1SSmYJKNyPIxvZbHfC9GypLx9pD1tRtsjjewGDa313jjjZerWWiLjzArsOIJp3zFvAJVGtCGMOoSzZ6mzIXSeGRFQX+uQztuU1jDYDTh4ptv8fFPfppLV68zSROKssAY3dAU3R7rWxtcW7vBi6+8TH9ukf7sHHHcQimF1ZrJ2BHjTtymFbbY3tzmxo3rpOmD8Jy9WNV3Cgxujr6ebArYi9k9TpfO4eQx9qFWKe0G4llDkQ8psm2S8Q5ZMsAahatelwKXgB3u73YtZf4qr734Ya5eOEvcnkf8AONB1O7Sn1tgpr/ITHeRuDVPGHgQ+HRaAWEgDHdGTEbbwBgv8gjbZ8hHF7l9pguLI/IdxJtjZuEM5594kqWls3Q7s3Q6XdqtkCzZQiSY8uITjDTLqswpgCi63T7LK6fo9npHRozrVFk3hdripsCDUIjCiCiCLC2Y7OR37KObU/5Yi9ElW2urfDxJeGNmBq01nudz9ux5nnjiKWZmZmnXniChmpZ9+C20NkedO2ZGYK4N3S6ELVexLlNuCXvQmrVEUUGRDthe02TbCdvxNv3ZM/TnEwIj+GFM3IoJvBjlKUxZkmUFk41NVi+/xaUrV3jr6lUuXLrE1YuXSdd38G1VSkgg9mCpDU8uwHNPK04/N8/8E08QzpwhnD1NNHcaP14kNBPCdknU6RFEbz/o9UHgua4MW+w5f6lsWZekN7rysMKenOKgn7xOnVa77GvU29bro+oEGbseYJuzN/cpjWNVbzSLS+22C2ksNfbrj5uiQnH3kXvV1xl7c8t+Y5/HGMU9zmBpYGIsaZYT+24GwZWnd5/7ig3efA5tyNOULEkIvQed7bSkWUaajdkeWrqBz0zoEwQe1lfkYhjpgklViMPolCT0yYcpSlvySc44uflardE3SdIL3ABNSlcl0xNBBT428EiMJk3GDHc2SFoereU+4fwcOk+RgcbXijiHVgplqjHVSM3cw5SdtZYsTciuXWNzY0zQ3sQPIpeBCzDaVcqb6c0y04HBIGVrkFPk7yRae3jQuFp4gut+6nH0ceFwPMaNMkR1mhlrNFm6xWBjjfFgk3y0g8lGoGaA94BZwjnON4At3MTB23lhb5IMf5Jk2MZJtkMQH/FX6C28izPnXyA+5zHfnqEdefheQL/Xot+P2VxP0cnrwAAvOEu7O0M+irl7ObSYxVPP8Py73sfTz7yP+bnThGEb3wsQSgY711DK5x2WAe2Rwng0QVSVtF1ZlPLotDtEcfvIzmlxU9siLhuAaXrqAD/wabU6WGu5VtwgH7/9LmAyGjIZ1VkxhCzJaLd7BEFIFEV4vucipU9I4zS4J76N82jNxHBqAVaWYWEJ+vNQxIosCNGBj/YDJknBYGvMeGeH7dEI0assndrGnEmILfTCAAmch0rKkjzNmOxssfbWZS68/Glefu0lXnn9LVY3J0xSQ8vA2bbTEfc7Qr/nZoEWlwLOnp1l5YnzzK08QTzzFGH/HKozj4m7JGaLIErxozYqPF6G5vuOGBuzVzq5llXUWTMt7OU5rr3FzUA8aeQ6rolp/SbYv089H19VprvJhVW7dWpvdFNPfBD2E+M6W0WdTbOeL633N2CDytGc4HwT/r5jTOE8zECncLMH7VgotaXU7ie5U+6diYGtnSG91TXS8YN5N/cUyyGlhoEBZYXYOuKYYRgWOWWeVSM3RVZaBpnGM1AWHsZ0UZQICS2xtH0YlxlJ5f3u42Z0+m3od3zCsIOEXcZEDKxHolP0aItsEnM6DogW59GDEcEko+d3ON2aw+tqRrZNUpaUQUCqFJvpkNyMcY3yTn1kic13yPOUvG644oPESNzFjzzCVkShutggrWo4THG/sDg3qs/xFvU5VI1x3WOVRUGajFm9+gqT67+MLS6C3cb1pM2cwimut8u4Py9WVi1b7k8LtniDwY3XSXdeZ7B+jZ3zq8zOLtBqddhYv85kskaRXgNeB7bIhzcoJyF31j0rkAW6sx/guRfez1NPPc/Kyjm63Xl8L0QQjM6J4xYij39NpkcZaTpxyeN9N7uhdY4xerd88lHC2iooqv7bQJ4bfC/Fr7TAhyNztgwH22xubNLvz9Dt9fDDwGWoOEHFAnLcsDgCeqXTr2aF6wmCEMKeot3rYKI22o9oTVJXhpiEUWkoxgnlaJXhDYvKx6Tba0TdRbz2AkbFTJKM4WDA9Suvc+3yawzWrqDKCQsdWJmDmY7i/HLE6cUevU5Mpx3TmYnozMXMLswwv7hMe+4MUfcpgs45JOqjvZgg8wiigSPG/vE+71XtJJTsOVerTFM3p+dqSihqqURDPtGUP1D/f793uUmSFbcS25pY1x7juzWt5rFrQg17mud6m6b2uSbuzYDDWwt5vuOR4uZfwwK81KK1k9XU09K3wwRYS0rC1XU68c4DXoXCCaTagGAIyFWAKEEDmSkpda0ncyJza0GXUkV8KgIV0sKn5eWcXow5tRgyHqyzsbWBV5YsBIrZQIgDQbVaxHMrtOeWoRCGo5wsyRkmYxjvUNgSPwwQz8MYha9C+q0Z6EHf75NrSx6GTJRgdiasj0YYk1TWTHHcotIG7UJwI8XKGVcnE/Pa2KBHpmaY0CH1AkxLwJt6jA8D9QzjceFwPMZVPlWlPIwx5HnKzvYa4+u/BPl/5HjjKXOwl8iTa1x982XWrr6LuH2GTmeevEwZjzbR+es4b7WriGfuqOboEbbezelzn8fTz7yfZ555L3Nzp+i0ZhwpFlUlYffwg+CmYidTnDzkhSPGSguIpSgyJpMxWh+fgmm38FzpilnoPCNNSwShSA9nwkiXBVubmywuLbFQzmNtWAUXniyRZgpcAyggX3c5cxMDIw0trYlDS9QKiDodZtpteu2YfGbCaCZhspWQjxPGGxfZvn6NonyRQocUNibTPpO8IM0LynyMsQlxz/DCM4qZ+RZzCx0W5mZYnp9noT9PFLUJgxZ+K8JrR4TtFlG7S9BewIvOINEKKughXkQQasJwEz9oIep4n3dTpb2q018Z61KXaVuVha5IrNhqO4XzReyLYtkNZqubQ01UawJs9i0HeYOb3uP93OFu3X1NqpvBgE3iXRPyZg7k+rg5x/uWfARgcanZPOOSP9VmGnPnJ76kysk0zmk9cBRpTYy77v9hQNnyUZ5QWkuhS6e7NBFQgCoQMXhYAvFQEuIraEnIXMfy3heWec+zSxTpNltrV8jGI6TU2LygSFJS5TG3fJbZJ5/HJAVbq1uY1VXG4wHeYJs8nYAu0UVJnmsgIIr69Psh/bYLGh0JYAxd22Mks6R5iS1LbDmpnHnrOP5SowWchtY5iFvOuyHKVbiJIsqoxYiYwi8xnQ5EJ8cR8SjjkQy+Q1XpoZRgrVDqgjQZQXGJh5dkpAQuUWQ3KLIuw61Z9mqSXufuZvYRdZaZ+c/lmec/l+effx9Li2eYmz1FELZRKnTeYQERi4iqSvBOifFJRqkz9/4VwWIoioKyLIjiqBrYHe3jZ5rlbCtyrEvQdaGPQ3tULBvr1+l0u8zOzRCEPr7fTDtwcpDgMpwPNOxsulRfmxm0U0srG9EdQ3dO04t9OpSEAXTbiiBXjHPNyFhGg5KdzYTBCEYJjEtIresFIh86XZiZg6VFxcoTPU6dOcXyqWUW55aY7c8TeB081cL6AcYPnGbB9xCviw1aWBWCClEqxPcjwjAkjCL8Y5ZS1CR4N3lERRC1cbmGa3K8W41WNZbaGws3e4Prt05Tv1tvqxvLfk9tM9tFnWmiDqG/F47V0BPvSirqCcX63DXxrj3aNUmfeoxvQQlsszfeKbm3EPd65iZ8YJvWitC4SkAdUHg+eMoVu5C6ukyIkOFJTqQKWqKJ/IDAbxH5Hh3PZ6kL51cWePaJ0ygzx2h5htFgyGg4ZmdnwPr6FkmhMd05wqXTRJOMKAd/OHJ9e1li8xyTpuRJRpJkJFlJVkJhPQRxMhOtSUpLWlh0SaVzqxtwXZ2miQhkAXpnodfbi6b2PfB8tHgkVmGVhdBAMG2oh4VHT2Ncl5wVccnIrcWak/ICzoHNaoG7uzMUIku0uu/h/JO/mqef/SxOLZ1nfn6FVqtH4LdRhEiteJHdsMNGttgpTiqs0dV73O4KMYMgYGXlNGtr19lYvX6kbfe2Uokj6D/zbMzFC6/Rn+0TRgG9vpviPInIcb6ZgXHFIWYmwHWwr+Z0epv0Z7ZY7gtn+rAQWyILobG0QugsQjeEbgDtCLYGEKWOJFoPwjZEXWi1QNuSpByR2FlSZclbAbrXwfO7iMRYFNpajOiqbDz4qoUvAdYv8WgjaoIfaFotj043uDcP6SGhNKAqQinigvHEq4pv5K7i3YFzHzUBrUO8DXt+giYJ3u+xhZs9xhYn9mx6eGvi7LMXqFfvczc0Pdl23/paYFiwl+AUHvs8xg+CAiepuF285e1Qx1U+OOqzutGZLgUrgu/7VRC8hxi/SnhiaHmGtmdphYo4CmiHPh1lmG1r2lGJNQlB4DE3N08YtsAfMNGKcpAxKTNG1mNiPEoJ8IOYbrvDXK9PN/IJtaEcjEkGIwY7Y9a3B9zY2GS0PaHMSjJtGGEZWM0gGWPMGBfyVcsoDnKSeDji3wG/t9ehewpEYVGNAZ89qd3tFHfBoaVro9IZ14tSzYiOk4Q7dRcRynuO0+d/LS+8+1fx7LPv5dTSeQLfFQyp66nXNS53A63F1X939zx9Ek4ytC6xuIGbsVX1OFHML8zz9LPPkqUThtvbD/syDw15NubKpUvMzvaJIu+m8u0nETlVSG6VwssOQF0HT1kWY8sLC/DEPMy3Ya4DC32Y6ULHh5bvZjSVD1EC+CCBI8VRD1QHJBa0GBJTMNYFY13SshqNJlDaRcGXJdrkGDJQAaER957zS8TLQcYoP8MPIYo9p/c9JmJsjMsZ7NmqMpzvZnKtqUixqThjk+DWMoTbLU1db41mV9YUMCuctDLYt18tiahJc60JvrMPYi+ipj52M8ivlmoUONlq/To5KDfyFLu4X9M8uEkb0weWSt8DRlmMZwmVJfIhEiGwEGBp+dAJoR0J7VhoB9BSll5s8SWjyMe0wj6tbh/x2yTGJ0gNqj1Bl4rEwM4kJS80xhh85dEOImIlFMMJW9fXWFvd4frVdd66fIPL124wGA8pTEqJrsZYhj3v8N2sUPEaraq0KxUfUL5b9ucGn+KRxOEE3ylX+lGUoKxHEEZErTZDr++ytR9rBrq3iyqSVpYJoydYWnkfz7/wAc6dfY65/hniaBalfJQ0enAFiHX5mz1xKWPA1bafSilONJIkwVqXnUFr1wkaYymKEt8L8IN7rWtfTRU8AlXk1levcuFCh16vdXO6txOM5uvJWCdBuTEGfwJ2FSYdyGZBrUC05CqxdbsK40HuGyQDfAjaivnlFnNLbVQrRAcBNgzACxkNxtjyEuONTbqtDp24g48g1qBtTkmCBCFxZ0SrmyJegfglpR1R6Al5MSHN8mNvArYKhFeqCoivJQbVO1pUpSGuiWnd/Urj72axjP2ShoM0vx573uC6FFedlaLpbS7Zk0PcreuvPcy38yc0yXqdU7n2UE+72ROKAsjdb5dbN2IzGl1m6LDACyyxr4kkJ5Scni/0Y592YAjtmCA3KEkorSZNPEaTFn7cxZcQG4X4bU1nRrNYGFRrhC5Lrly8QJoWTEYTJoMdxts7jK3hM/Y1Ll68xsVrm7x+fZUbw00SM8beMgp8m7AJbK3BeOjuz/eh13d6LRW4h9IYl4rouEbMUxwqDslj7Ll+TQSlLGEY0e70UMEcpow42cR4FuU9T3/2aZZOPc3KylMsLz5BO57DmoAs0YgylSdco8SA56r5KF/h4UaKUhU5OeEOuXc8JskEawyl1hitMcaijSFJUoxx6dvuBBFFqzNLrz+D7/sMBwPGo210eaekSA8Xusy5cfUyTz517pEhxgfBADcs+CnkqauM59K0QX/Bo93rYtuKSZhhck3pKcJOzOKTK5x/YgW/3UYrj3GWsT0cMBmNSLe22TIFs+02s502se8RKLAqJ5cEFbewukB5ghf7eHFAaRKyMiHJE5K0OF5ibKvMFHV2igYxNqaqOqcqjXFNgvcHzdXranLc1Ag3vcM1wa1Jbp1tv06Z1hSz1jrl5ny83zjmfu9xrWeuk5TW9e6b29WfhpulFLVXeooTBoOTIbRwUgrtqvWUCVbtUGQTdFhiQ4Mo7So2BjEd2kQ6R+U5tszJbUoZGnYiRdCKsWEXG2pQPoX1UWFMtz+LxuPG2harq5eZjBOyLCdLEtJ0gtUFV9dX0dZyIxkzMjl3L5VyLxB3j8Waq4VOCRKDGFeqMagSb1sDRV1p59FH3QU0E8g0a/fsn4C6l7uutz+Jkz+HJ6WAyl0hBFFMq9snmHmCLHkKeJmTS463MfpT7GxeYjR4kSsXT9PpnKHbOUWvt0Sr1XV5bwUQR5I9z8MLQqK4RbvVod3u0ut0uXb1AkUxeNg3NMUdMB6OMNagtUbXxFhr0jQlz+/u/bPWkmcpo6FCeR5ZkqAfgc4vTcYMBoNjzb5xFChwKsAeLh1VpiFDoNUlXlgB8WjHYwabO6ytDxmv5+QqgLjH0kqfmYUlgo4QREMm8TaTnTXy4TZ6NGEyHjv5he+qrfkxeF6IrwXPei62QCK0KchyyyTJSZJjtmf1ZrI48muFvZRtpnohNd9SsPemqiUPTWLcfLM1vcT1G9Db9//676aEoklgm9UkajVdLa1IuZX0No/b9Co0M2McdOypB+IEwuCC7evk15WTwWyDuUZZbrOZFmxjUAIBQsfz6PkekYBog1iDYPADYW59h7nVbdq9VaL2W1g8siwjy1KSLGU0nnB9fZO1wQ5JWVBag8FiGgT48MesbRzxr6lTLa+sJRXiyneWuIHBCckbfy9oPvb1o7d/XNysrVOrnISbx6n1fs07P+h3aO530sjxIRX42OuJrQihxFj6zKw8zfrkV2OGPtg3cK+0k+axssAEayeUxTXK4iUmo4A12oj0EemC1GF1rrmIBIhq4QcztOJF+jNLLC2ukEy20HpKjE8yhqMh1lq0Nhit0ZWkoshzsizHmua49yBYymJCWTxoMvzjhdaa8XD4SHuM4eZ4rFKg8KAIfaQ3T7x8Gs/zaQXb6FHO5RvbvHUFNgfrjAufdzFDZ77LbG+Gfq8knwzY9lsMjUe5s04x2qKQkiiEsBsSeh0CIiJpE0gPX/qI6mNMSZ4rJokmTY63SxdxEgoL6IY+uNRu2f11D9IQ1x7ZWgJRe5Jr722NJhGuvbWW22esaDappmzDZ6/md1pt2wzyb8o36vP6jf3rc5h9+560t+gUFSwu9K/ypFJ5T80mLrS23B2jaQsFlklpWCuL3SawOy4rIJ7s0FkdodRFDF6VkcWgrUYbS2ENuTXHyCgEpF8ttZ64BD9ywQ2e5x7OKuYKa+8QbX28aD5mB12Rwj2m9aTQ/ke7OX5tzqk2j1vDNL6r96uP1xzrNieq9hPph43DKQld966qHjV5eKrF8qkzePJBtm/Mkm0/hcmvgNlxI0i2OZlVwA114RBrt+7gQRSK3CMZR2xudLl8cQEosfZOhUKmeNhIJonzKmgXrGG0wRhNUZQURdUedxMNP0awlrIsq0I8jzZGwFXANzBnQXd6+HNLtJZXiDyfGQnpbk7wvFXyTLOzY7h2Y0h/bpvZ+W2wPp1WiOe16HYX8UtIS8jTHK8YYtME61uk3UJZD19ifK+D53XwVBulUjwVoyQ4/kw0qtE89xKr7FbBu+ntV3uFc/aC4Zr64vrN1HzT1aS3cb7deqwHBeo1XUz137WLyWOPGB+Uob/ppVaNdfVnfZz6fCUHv4mnOEHYwGVP7gIxyvoYBtxtxnj/REI9fhsXe6WbTwaqhmqt0zL5EUSR0xnXDdOaihA3aeDDQx0WUE/KNEMDatRKpeZj2Fz2Tx4119t9S/3Y7p+EorE93Ey2a2VW0fhu/zk1N2eUPkocCjEObVnJKLwqp7EgfkC0uMxcr0N65jzjyWeTpiOKYkwy3mRw/SXywadBv8XJlVncCc0oljFleePIziSqVaWOarpj9l/LFPeCNE9d8J02LremcRkqyrKgLPIqo0qA0SdXM3z/eDxYRe2XumBgWQtmZpZwcYn20jLK88itx/LWmDOnrrCzsUMQQpaUrF7fIPRfZ7Q9YHFhltmZDu2oz/xyh1R8UiuYwTVIMoqihFLjaUBCfL+N57fwvBZh0KIVduhEHcKojkI7HtT6YltWv+T+CPj675oE1wVGM/be0wfxjHqf2tnn6iHsEeOmy6eZhaLmAzVxrt9k9dxrWC37i5s2t5V965r30STG+/ef4gRC44jxBIVP17YoSUm4/VuqSoCGcJhp444INgObsjvqC1quTKfyqmdLO32T0Q1y/HCxP0SgjomtlzqZjHBrQeyDiPD+z6Z3uf6N60e77irqpT5XHVbQlGg0E+j4gC9u8QREhImBkTkcpfjdcCjEOB1uuQpwvo94VSCaUvieR6s9w2xvtrKwxaBJiwk3rj7P1QsvsPn6j4L+DCehAZ1YSHPCAQ5+K9ytuTT3OcxtHy0UZQHWYozFWoOtcm5rU6KNRpTghxF58njlhBKliOIYXT6Kg9CDMQE2CxgbRSo+pQrwUJQlWC3EUczszAC/06LTXyCMumgN6SRlFIwIBMJZRTsOiaM20ulTFAOKIsAojRaLFoNRBpStgmuFUIV0wjbduEsrDHCpno4HVRjHrte4XtyX3Opyyzm4sm0TNSmu5zPrXMe1+6d+S+wP4mtqiOvz16TY52Y3VPNtCntZJpouo2b6tuZ5mnrj/frpKU4oLC0K5tAoLGNgi4NJ7xyKeQIKNJuU7DS+q8dFd0KbDhMmHP27yskund9SAS3QHujI5Ur0q5GlhyPFCvYq7Tw81CS0+Sg2ZRHN8Wg9noabu5Omp3f/o7w/w2LdLTRTptexED4QKFcFwlqwCGKtI8yBRxSFtFoRrXZIFIZ4fgiiSNOC9Z0hsr7x6BDjGxdfd2TYCxDfHdKKIopi2nGHuN0iimP8KEQFAe2wy8qpJ2lHMS8nY3YuZ2Df5HEiIocJq9P9a+7nKEe07aMFJe4hVdjdokWo+iE3iALP9xA/wOqimjKD3QTAlr3560cIrXaXxaUltjY2H/alHBoMMBhbrl7dZGFpnV5vDqs1F9+4yKULV9gapEgUcur805x//gWWFheYnWkTSEkx2SEbbzPIdyg8Q6AzfApU6COdDjYsMC2hDHNKNaIwO/jlCK9I8I2h7bXoR116UfRQ7l3EBdyVZi8QT7yGk8qyFx1zt0px+9OiNSUTB82f1oS5fgPufyPW2uKgOmbtggobxwpwbsL6Ddr0INfXVLOiYN81PnyuMcUdUOtVF4BTvqUVCsYKVzPDm+bWKfwFL6DtRWzlY/bP00V4aPRtmUFb2jzRf4oLgzdJ7XHEfQzZa4AtKEsYGRfwYATiyHmQsa4anv9w06fsel+51SMr+7Zrut+a0ohmNsb93uJ6LFs/2oFA5EEcKqLIxwtCvDDCC0P8MMLzQ5QfUlrFINEMk5IkTcmLnM7sDIvLp1g+tcjy8jz9bgeLkKYFly9fY+PV12F9i+PgiYdCjK9fehNRHsoPEM/HWItBaLU69Dp9uv0Zuv0Z2r0eUadFGMfEMxGz/Rl0lvNqaRhc/wmwFw7jcqaY4rYIQh9rbCWjcEU+rBGM0Sit8H2PMApAWlgbAy7rilJq17tcFAVlnjvifJ/wlIe17vxHDc8POH3uHAsLC4wGj48G3gLbE7h8cUC3ewXfC8nSjNdfeZNrV7eYpCVe1KK3sMITz72HM6eXmZtpUSbbbFx9k8nmmCwZosuUdpWfXyIPz+tgogLpABEYr6BkQqnH+GUC2uJbCPGJlXegJH1/BPeho3pLmYpo1OEdFm4OsmtmcrgTDsowUUsr9i9NPXET9bxoAETgBaBznLe6FjrWYseaPPvcfOzazVS7qPa9pQVQdsqNTyrmgbZAJLDow0rPo9d2JdTntyYkmxnX2WtmXaAtgraaMeaWAsz2Lr90qEJ63R7BOCItj4MYZzhyrKv/V4Q4qSJiyxLi0qVt8zynezpiND27+1GPj5tEt5lYZn/miOakUDPcQOEe4+ZkUHO8TLUu8oBQ4cURXitG4jZe3MVvtQlbHVQQgReRF5B6KUOTMCwGZMUETYcomKXfXoLZU6h+D601hRqzbbfYyMAc05N/KMR4uH0NxEPElZ2yuB56EkQMo026/RlmFxaZLRaYVYuEUYASD/FiFhdPUz7/AV5Nd5hsDdgr3TzFFIePfreHxbrAO+uC77TRBFlAELglDEOKosAYs5vFQURwFfNcEFuWZWRphs4zXDWFtwd9jGl85haXOHfuHO12m6P2dAfcOuN+lNgu4cLFksn4Im++tspoormymjFMnLNmfjHn7DBlnBm0BITtPp1OTBgK6UKPYrRBOd7CZkOyfIzvKcJ2C6/Vgp6P15vFj7vgRRgsRZmQJAnb22tsbd5gMJjcQor3B6scNqT6pyrA6Thy09NaE+K6iR0kTzgIzbdlMxsE3OwRro9VB/U135Y1Md7/tq7dVPVxmpKLpjzC7v1f2JuJVlLtXoIppnUTTiJ84LnlkPl2QOx79NsRs7027Sgk8D36W2Nyu8rMTkZi9oouXypTxtycza/G+C7xRzt6hxevfpLkWLzF4HzhMXvapARQ7h2QJJA64kfchk6nCso7WtztUai7gwc5dt0N1GELt0UJlAaZJECCsEWd1UtEkCp62Ficc8ruUfad7QtcuvgJftkPCcMAz/OreCBNmqVVrYDjefAP6VdbA+tjbQgmpB5DaK1IUo8sGVCWGUoZ2p0W9LtY4yGi6M8sEAc+eZryyse30NnP4cT7U0xx+Oj3+24kXBFjrTVlWVKEOXmeE8UxRZ5TFIXLdWwMWOu8xdVSliVhGBKGIUkakKcptriTiPPhIYxinnr6aVZOrxBFMcYcHWX1gXMB5IXzqQw5eoto4FoCq5cNcmVCaW8O5c22Cq6u7rC2MWDxVM6y9en2O3T6MSafY7Jzg/H2dcbrV0i3xihfCFshYTdA+h28zjwqnkF5bSyKUqeMxpusbVxide0yO4OD9cXNTGOHiZqEq4o8qspL7FWfuukpruc/a7nCnSre1mS1jimucyzVN9Nk+7UrqRkpVXt/G9II2zRAM4Rd9h2vdgI1gvqUAk85ZxzipmgDceWwS3NfY9EpjhiBLzz51ArL833aUUCr5dOKfSJfESjF7NIsUafD6SsDNteGbAwmvGU0V7n/8HuLZXJsmaAEqKaRgL1R6Aj3cI3B+lCGMJ51uqaHJLV62Lgpy0jdadmmD/rgvawpKPKCYv/UwTHjkIhxnfA6BqnrhlL1XiXW5JTFhDwbUxYJWudQWsS4uUDxfLr9WdqzzzBcfQPs64dzWVNMsQ9RHGNxEUumURq6DEOK0qVsK8uCsix3i4BYU3uYqwwWuqQsyl0inaYpyXhMkaZYc3KC25Tns7i8wvzCAmEYVfmbj+76LC4G5e3M4B8G6ujqg044TAyf+MRrEIQMx0MmkxFnzywyNxvSiRVee4auB9gMzBjJNsizAdaPidodlLQJg1m8eA7xY7TVbI3WuXD5VV6/8Drbw1t9MU1572HDli7gfTdiuyEWLGtPbx1aXnuRm+HhB/0wTfJcb1enRqsJc1P6AHsa4GY8cHPRVfq4Zgq5ZkBfMwKoGdZeHddWc7Wq8ozXNaRqPfUUJw95aXnl8g22xjv0uxEzMy1mZtv0ex36cRcviuiqENvqIe1N7JUNysGQsNSsmz2BwslFrQcK2Et41pwXqkejhWuoYwOT4KADTXHCcUjEeA7Eh7CNF8UgygUS5wkmH6N8hVBidIrWGbrMsOgqq0mBLjL8IGRh6TTp4AmK5AIn/RGZ4tFEEDY7qorwGouJQspS75Jht5SuEIjRleTCVERaU1Z5j+M8J8sy4igiSRLS0ZgiT7EPObG7iOLsufM8+fTT9HpdRAStS8ry6J4rDWybk5dyaWMz5SM/9wk2NjcZDAe88MJTPHF+idMrsyzMtujOx4hNUOUOyfo6yeYGRRnjteaJejFBMEvYWkArIc8TNkcbvHrxFV598wbJMXs2aimB+BD4TseLgC4rP0SdiaLOjdT0BJt9nzWaIeo1ma2TstTkOGZPH1znaNqfbo3G/5vkusb++rHNBKXNKnzN4D7Zk4lY23grTMnxiYMGXryW0VrN6MZwakVx5lyH094pTLdFr93B82PiqE2MpY3m6eUuL4hiczTiwsaAy+OSbXsS597g5sy7++uT16O++iHUQAr24QbfTXF/OCRiHIPyUGFEEMeIuDB/LZrCZNVo32DrEolGY4xgxOlHtNaIUrTaLfywR5HUveUUUxwuPG+vo7I4AqmUxRhBROF5Cq0VWnsY7e1WxqsJsdEepdIocSkJndCzJtgGW2pMqSkfch5kz/dYWFyk3+8TBIGb2rb2yAn7UXlKHxTJuOTatVVurK6xsDTL3FybuayLVT5+HBBGMXkUkSooi9zp4EoD1kNJiOfHWNFYZcmKlMFwh8G4PP46MFXtAKHS3Vbk0TaD1Gri2ySgzZDzeqlRb7PvPLvfNcPZFTd7og86Fo19mtvv1xw3dcXNbBoNfXLz0I9eLph3HkoLwxKSEfgDQztJ6BcZGYZO4CES4Gkfvx0RdCNi1Wau3aY9alNQkKZDxuVekcNmGuuHj2ajrRt1M5StvtKmQH86gnsUIY96idgppphiiimmmGKKKaY4DEz9/FNMMcUUU0wxxRRTTMGUGE8xxRRTTDHFFFNMMQUwJcZTTDHFFFNMMcUUU0wBTInxFFNMMcUUU0wxxRRTAFNiPMUUU0wxxRRTTDHFFMAREGMR+ZCIPLapLkTkfxWRHxeRDRGxIvKNx3Tex9auIvJ5IvIPReQzIjIRkYsi8v0i8vQxnPtxtuuTIvJ/ichbIpKIyLqI/JSIfMURn/extel+iMifr/qBnz3i8zzWNq1seNDywSM+72NtVwAReY+I/Jvq+U9E5GUR+ZNHeL7H1qb1vd1mSY/j3Ed5jocJEXlCRP5Z9f5PROQVEfkOEekc+7Ucdro2ETkHnLPWfuRQD3xCICJD4GPAG8DXA3/AWvu9x3Dex9auIvI3gC8Evh94ETgLfBuwDHzQWnvpCM/9ONv1s4D/GfgwcBnoA38Y+G3A77bW/rsjOu9ja9MmROQZ4BO4GvavWmt/3RGe67G2afXC/17ge/Z99Qlr7eQIz/u42/XzgP+E6wP+KbADPA90rbV/84jO+djatL63fas7wI8CP2it/e+O+tyPqV07wK/gKqZ8CLgI/GrgLwP/wVr7e4/1eqZ5jN8eRERZa42IPAe8yjER48cZIrJkrV3bt+5J4E3gO6y1f+nhXNnjBxHxcXb9mLX2tz/s63mUISI/BlwA3gX4R0mMH3dUxPg7rbV/8WFfy+MCEVHAp4CXrbW/62Ffz+MKEfk64J8DX2mt/aGHfT2PIkTkNwM/BvwWa+2PN9b/NeCbgf5RDpD341ikFNU0w3eIyJ+ppnUnIvJDIrJcLf9aRHZE5JKI/Ll9+y6JyPdUbvVJtc0PiMjZA87931fT8amIfFJEfoeIfFhEPnzAMf+BiFwRkaza54/cy/3Zh1Tr93G2635SXK17C1jDeY+PDI+zXQ+CtbbEeY3K+9n/XvBOsKmI/D7gVwHf8raMc594J9j0YeAxt+uXAu8BjsQzfDs85jY9CN8A3MARuyPDY27XsPoc7Fu/za2F548erkzs4S04N7jdt84CbwE/hJvG/YM4A/wo8F+Avwj8Rtw0mgW+orHvu4C/Bfxu4NcDXwN8FOepiRvb/SZcHcZ/D3wFrrG+AVwFPtzYrg+8jHPV/+HqvH8dV7/xT7yN+3yuutZvPGwbvpPt2jjee6pr/uapXR/MrriOxQdWgL+Eq7j630xten82BeZwL8I/UP39YeBnp+30gWxqgQ0gAya46f8vPkqbPu52xT3rtjrXR3DV2leBvw20pjZ98HcVcL7a73+bttUHaqsx8ArwU8B7gS7wZcA14O8dtW1vuZ5j/PFewU031uv+ZrX+LzbW+dWD+3/c4fhe1Rgt8Lsa638ON20kjXWfW23X/PG+DUiB5/cd9x8B681rvMt9nhRi/FjZtXG9P1Vd89zUrg9mV+BvVMe1wBD46qlN79+mwD8GfqY+Dw+XGD8uNv0XwO8Fvhj4WuDjOCL3pVO73p9dgX9QHW8T+HacB/mbcQOPH5za9FDeVd9SHf8DR9lO3wl2xcUU/Qx77ypb7auO2rb7l+NM1/YT1k3j1vhM9bk7/VB9/xrux9mFiPxxEfm4iIxwU8AXq6/eVX3vAZ8H/FtbWbg63n/F6Smb+HLgF4A3RcSvl+o6FnCjlUcJj6Ndvxv4tcDXWmu33sZ+h4nHya7/Oy6Q4bcDPwL8gIh85T3sd9h45G0qIl+MC7r9483zPEQ88jatjvl11tp/Za39GWvt9wG/DueR+o67WuBo8DjYtX6/f5+19i9Zaz9srf0buICmrxKR99zZBIeOx8Gm+/H1wK9Yaz/xNvY5bDzydhWRGPhXOHL8dcCXAH8WN1j+u3e1wCHDP8Zz7Sc4+R3Wx/UfIvIncFM/fxNnqC3cA/+RxnaLuGjG1QPOe2Pf38s4b29xm+tcuO0dnEw8VnYVJ7b/I8A32IYI/yHgsbGrtfYyLisFwP9T6cL+BvD/3G3fQ8bjYNPvAf4JcFlEZqt1PuBVfyfW2uwO+x82Hgeb3gJr7VBEfgj4prez3yHicbDrRvX5E/vW/zjw14DPAT59h/0PG4+DTXchIp8PvBv4U/ey/RHicbDrN+FmNJ6z1r5erftpEdkB/qGI/ANr7cfvsP+h4jiJ8f3ia4D/aK39M/UKuTW/7Trux1g+YP9T7I2CwHUWq8CfvM35Xr7/S32kcOLsKiLfCvw5nB7pX9xt+xOKE2fXA/BLPPzO/O3gJNn0PdXyxw74bgv40zgP/UnHSbLpnXASvPJvByfJri/e5VofSiD5feAk2bSJb6jO+QP3uP1Jw0my6/uBrQYprvGL1ed7cPKqY8GjQIzb3Bqp+Aeaf1hrtYj8EvC7ReRDtctfRD4XeJqbf7wfBf4EcNFae9Ao6J2CE2VXEfl/4aZNv9Va+91vd/8ThBNl1/0Ql8Lp1wH7O6CTjJNk099wwLr/HafP+xO46cpHASfJprdARPrAV7L3YnxUcJLs+iO4YMbfAvzfjfVfXn3+0ts83sPCSbIp1XFDHLH8EXtAVqVHBCfJrteBORF5zlrb7EN/TfV55W0e74HwKBDjHwX+nIj8BVwn+WXA7zlgu/8FN0X0gyLyD3FTAB/CGbw5Mv4unG7lZ0Tku3CjmA5uSuSLrbW/804XIyJfAizhIvwBPq/S52Ct/T/v5wYfEk6MXUXka3Dk4keB/yQiX9D4emCtfel+bvAh4STZ9UPAPC46+TquzX4T8PnA77vvOzx+nBibWms/vH+diGzjAktu+e4E48TYVES+Gadp/M84XfGTuCCxFeD33/8tPhScGLtaazdE5K8C3yYiA1ymj8/DZav4Z/sIyEnGibFpA1+J61v/2f3c0AnBSbLr9+KKUf2wiHwnjnB/Hi6g77/i3mHHhkeBGH87MIubooxx2Qp+Cy5dyC6stT8hIr8f9yP+IM5z82dwncBOY7sdEfm11fo/h8uTu437Ef/tPVzPX8YJw2v8j9UCx51r78Fwkuz65TjbfTl73owaP4XTHj0qOEl2/WWcZOJrgBlcR/ZxXCd1rB3NA+Ik2fRxwUmy6cvA76qWGZwX678A32StfdQ8xifJrvX1DIH/ATfYuIZLofVX7vP+HgZOmk3BySg2Of44jcPEibGrtfZC5RD7EG7meBG4BPxDXOGfY5X9PNaV78SVUHwNZ9hHqSM40Zja9WgwtevhY2rTw8fUpkeDqV0PH1ObHg0ed7s+NsRYRFq46MqfxAnGnwH+3ziB+GdZa689xMt7ZDG169FgatfDx9Smh4+pTY8GU7sePqY2PRq8E+36KEgp7hUap0n7blxakDEuWfR/+zj+cMeIqV2PBlO7Hj6mNj18TG16NJja9fAxtenR4B1n18fGYzzFFFNMMcUUU0wxxRQPguOsfDfFFFNMMcUUU0wxxRQnFo8MMRaRD1cVu6aY4qFBRL5KRP7nh3TuD4mIFVdmc4r7QGXDL3vY1/GwcC9tSES+tNrmSx/0PPe7/0nF9Pl/+Gi0z994D9vaKm3lY43KJh+q8tSfWFS/x13LvO/ne4fRJ70dnGgjTjHFCcRX4fItTvFo4n/B5euc4vb4ZeALq88pbsZXMX3+HyV8IfCPH/ZFHAO+FNe3PS6c7n+oloeCd/TI804Qkchamz3s6zhOvBPv+agwteUUjyqstQPgI3fbbtrGb4+pbU4GrLV3bcdTnDw87KJeJ3J0ISJfIyKfEZFMRF4Ukd91wDZLIvIPRORKtd1nROSPHLDd0yLy/SKyVm33sf3Ha0xRvU9EfkxcJbt/fYS3+NBxu3sWkdMi8s9FZL2y1ydE5GsP2P83isiviEgqIq+JyB8Ske8VkQvHfzfHAxH5Xlxi97OV7ayIXGhM83y1iPwjEVkDbtT7HGSTg6RBVZv+eyJyqbL9JRH5FyIS3eGavlxERiLy3Sd9Gu1BISKfLSI/KCIbIpKIyMsi8i3Vd79ZRH5YRK6JyEREPiUif0ZEvMb+9dT+tzZ+vw89jHs5AXiPiPznylbXROTb6/Zz0LRl1V5/VkR+e/XcZ1QeHRH5HBH5maovuCIi38ajVezonjB9/o8PIvJC9ayvVu3qooj8G7lZRtKu7nu9Wr5PRGb3HeemZ7zx3nv/7dr/cUBEnqt+2zervuwNEfn7IjK3b7sDJaRVu/ve6v8fwnmLAYq6bTa2ves7XUS+sdrv14rIvxaRoYjcaPSvX14992MR+ai4ktDN/UVE/nTVJ+eVTb9bXGn3Ay5fvlVELlf3/tMi8sF7ue8DDvTVIvKR6nfcrtrIE3fb7244cR5jcbqhHwB+CFddZQn4W0CAq6BCZeyfBVq4Silv4iq2/H1xI/W/U213HvgFYBVX3WUNV7Lw34rIV1lr/8O+0/9fwD8B/j/cXOrwcUbznlu46jdzwF/AVZ75WuBfiEjbWvsPAUTkvbjf5xdxVdVCXOnGGR5vu/0VXHv81cDvqNZluPsG+DvAjwBfh6skdM+oOsSfw5UZ/Q7gE8Ay8Dtx9r3F+yQiX4+bJvx2a+1ddVuPMkTk84EP45LK/2ngMvA88IFqk2eA/4j7DVJcOdEP4X6vP19t84XAz+PKj35Pte7yUV/7CcW/B/4p8Fdxfee34Z7dD91hnxeAv417Dt4ANkVkEVdq+DqONGbAnwUe+OV0AjF9/o8PPwRsAX8clzv3LPAV3OzM+1u4ynO/D1dS/P+LSy32Dfdw/H/P22//h4kzuPfrn8Ld5zO4d+4P4/qpt4N/DJwDvgn4dTgbACAiHe7hnd7APwP+Oa7i3H8L/K/VYOMrgO8ERjg7/3sRedZam1f7fSfwLcDfBf5v4L245+WzReRL9lWu+3pcyef/CYhwFfj+o4g8b63dvNebFpE/Bvx94P+ojtHD/X4/JSIfsNYO7/VYt8Bae6IWXCnQlwDVWPcFgAU+XP39bbiX3/P79v1HuIfIr/7+JzgyvLBvu58APtb4+0PV8f/kw77/Y7TzLfeMa6gW+NJ92/4kbnDhVX//QGXXdmOb09VvcuFh39sR2+17gcv71n1pZbcfvM32t9gER/I+3Pj723Ed2ufcw2/m4xKsF8Afetg2OSa7/zSuU2/fw7ZS2ehbcS+dZl9ige942PfzEO1Yt6E/v2/9P8KVDp5ttOcvbXz/YRxx+OC+/b4TyIHzjXWdqh+2D/t+j8B+0+f/6G28WN3n77jN97W9/9m+9d9dvYOksc4CHzrAhrdt/w/pnn0cqbXNNrC/nTTWXwC+96C2sW+7e32nf2O13V/ad02rVTt7urH+d1Tbfkn19zxu4Pa9+87xtft/x+rvdaDTWPdUdY6/crv7Zl+fBHRx5aj/6b5zPo3rj/7Ug/weJ2rqRdy0568G/k/bGGFYpxO60Nj0y3Ge4DdFxK8X4MdwCajf29juh4GdA7b77APc/D94FPd1wtG8518PXLHWfnjfNt+H85TUdv0C4IettZN6A+sSff/cEV7no4AHaT+/GfiotfZX7mHb7wL+MvB7rLWPfWCJiLSBLwK+v9nm9m1zWkS+R0TewnWMBc7zNovzvE1xM/ZLxf4l7mXzvjvsc8Fa+7F9674Q+Ii19lK9wlo7xnmN3mmYPv+Hgw3cjMRfE5E/LCLP32a7H9r39ydxHshT93CO+2n/hwYRCUXkL4iTgCa4/upnqq/fdYinutd3eo0fqf9jrS1xM3SvWGvfbGzzmerzfPX5BbhZje/bd6x/CZTAl+xb/8NVH1Gf5wIupuHteMq/EOgD37+P212qru/Xv41j3YITRYxxI8WASp+1D811y7gbL/Yt/6b6fqGx3dcfsN1f37ddjceyistd0LzneQ62wfXG9+C8w6sHbHfQ7/ZOwoO0nwXufVr/vwc+hRv1vxMwh+urDrRPpQ38D8BX4sjwl+EG2N9ZbfK2prXfIdj/rNZ/n73DPge179MHHOug478TMH3+DwHWuf5+E/BLOKnDK5UG94/v23T/tHstN7mX5/1+2v9h4q/ivLzfB/w24POBr66+O8z+6l7f6TW29v2d32Yd7F1nfYybzlMR640DznG7/uLt2L52dvwkt/K793Mrt3tbOGka43XcjR004jsFvFX9fwNHzP7kbY7zcmO7n8HpZw/C1X1/P3Z5N+8BzXve5ODR6krje3APwEFeuHsZqT/OOKj9pLjR9H4s4NpnjVpHdy/4b4AfB35ERL7CWjt6W1f56GELN41/O/s8i9MUf521dtdrISK//Riu7VHFKZxXrvk3wBVu/144qH1f4/b99TsN0+f/kGCtfQP4ehER4LNxkoC/Jy6QMTmEU9yp/R8Hvgb457ahDReR7gHbpTjP6H7sJ5u3w72+0x8E9TFWgBfrlZUHd+GAc9yuv3g7tq+fnW9snrOB+9cXc8I8xtZaDXwU+D3NCFER+TU4HUqNHwXeDVy01v7SAcuwsd0HgBdvs900nc7N+CngnIh80b71vw83EKlTqHwE+IpqihtwU9m46e7HHRkuSPFe8RZwSkSW6hUi8iy3dlY/Dny+iHz2PRzzRZzm6nncy/GgDvWxQSWf+Fnga0XkINvX7bCoV4hIAPz+A7bNeXu/3+OK/27f31+DC6z55Ns8zs8DX1AFOgO7AT+P66Bk+vwfI6zDx9jLHX1YUofDav/3izaN/qrCHzhgu7eAF0Rkd3AlIr8eF2jWRM1l9rfNe32nPwg+gutXv2bf+t+LG2R/eN/6r6j6CABE5CmcHOPn38Y5fw5Hfp+7Dbd7+W4HuBNOmscYXNqRH8dFPX4PTgfzl9lz/YPTWP1e4GdE5LtwHuIOjix/sbX2d1bb/SVc5oSfFpHvxumU53AP1zPW2j949LfzSOF7cV74fyci34qb2vv9uGmtP1oNXMBNV/8e4MdE5G/gdF3fhpsOeZyzUoDrSOarab1fwo3o74R/g4vO/T4R+Zs4udC34DxETXwXrrP6SXGVgT5Zbfs7gT9m90XYWms/LS6d1n/G/Q5fvn+bxwzfjOvkf15E/jdc23wG+CAue81bwHeKiMa9cP70bY7zEvDbRORHcZ7oq9ba/TNH7wT84cr58FFcVP4fwgUp7Tgn3T3ju3Bp235cXNqoOivFYXj1TiKmz/8RQ0Q+gMs48a9wGlcP5xkscRlQ9pPC+8Ft2/8hHPte8KPAN4jIJ3H3+NXArz1gu38J/BHgn4pLz/Y0bpCw/zprgvtnRORHAG2t/SXu/Z1+37DWblZ98reIyBgX1/UeHE/4WW7Vgie4/uKv47jDXwYGuGfgXs85EJE/C/zdatD5IzibnMVpmj9srf2BB7mpE7fgNFQv4zrZF4Hfxa1RinOVId/EjVZWcbKJP7XvWOdw6UyuVNtdw2Wl+NrGNh/igIjOx3m53T3jNIP/AtdxZ7i0QV97wP6/CfhYtc0bwB/FBZ/8ysO+tyO2Wwf4/+FIlcUNtr60+v9vvM0+X4XTBCbAx3GBNje152q7ZVyanGtVW72ES58T3e43w3mNLuNG2/2HbZ8jtv3n4IK6titbfgb4c9V3H8R1wpPKHt+Oe9lZ4KnGMb4I+K84QnNTxPo7YWm0offhSFWCczr8FarsHdw+K8XP3uaYv6rqe9Oqn/023MvOPuz7PQL7TZ//o7fxcnXfr1TP8yZuUPxb9rXP37hvv2884Hm/XVaK27b/Y7rHRRzp3aqW78fFRVjgG/dt+0eBV6tr/Tngc7k1K4WHS5W2inNO2cZ3d32nN2z33L71tzz3uNl7SyMjCi4T0J/G8baaZ/3d/W2y2u87canjLuP6jJ/h1mw3Nz0fHNAnVeu/ovodB1VbeRWXhu+9D/L7SHXwKaZ4IFTTea8BP2St/aaHfT1TTDHFFFNM0YTsFcMIrAsOm2KKW3ASpRRTPAIQkb+DG71exSUr/5M4L/7fepjXNcUUU0wxxRRTTHG/mBLjKe4XMS7bxync1Mkv4qa2PvFQr2qKKaaYYoopppjiPjGVUkwxxRRTTDHFFFNMMQUnLF3bFFNMMcUUU0wxxRRTPCxMifEUU0wxxRRTTDHFFFMwJcZTTDHFFFNMMcUUU0wBHFLwnYjcVqjs+TG+GGyZU9jjq7kswJNRh88/d44nZ7qcbsd0bEakJ0SzC0Tn30XruffRff/n0Hnu3fhBhBIhmQwZDbcpixylFMZYsjQjLwqsBbHgeUJZlvz9v/u3+ckf+Xd3vRZr7dvKmA97No1bT/Gbfus3k+VtbqylXL60ydbqGqYc4NIaZrh6BhpXeTRgb7xT4NIEJo1tbWOh2jbAFeKZZ6/6pMGli92g3TrFl33x5/LrPu/dnFs5xdLsPJ5S7hi2xFqNtQXG5BibYNUQK2OssRgjFIVQlFCWmlLnlDqjKFMKnVCUGXmZUpQFRVGQZzlZnpJlOUWeU+QZpigxZUmRleRZidGG//Dzv/C2bdq060GIgxZnZnp46SatsqQvMB8Ly6fnmV1Y5MKrb/KJ6zkCPNUR5s7MkrUXSYM+wyxjmKSkWUqWZaRpRpJkaG3x/RjPD1HWIlgCBUHgEUUxcauD54XkhaEoDabS/PueTxD4RK027V4Xg7C5vcVoOCIKQ+I4Rmcl6TjBpBrJwBOPqB0joc8wGzFIhmRZSmkzFJZIPERKEjMiwxz4LD5IW62hULxr6Ryf/Z5F3vVcB4vP5WuG62uazW3NaFQgeozoMTujMev5iPFu4SaAiMXZF/jyr/gdfPlXfiXnnnySmdk5gjDEWIsBDNa1Ymux1mCMwRiNLgymsFCCIAiAWLTVDLc22bhxg4/94s/zEz/873np4i9ibik+dfi4H5vCndvqFIfTVt2b4otx1WxTXD8quH5RqsW4v+UpeO49RO9Z5slTMcPVTa795C/C+CLx+z+fr/66LyKKY65upAx2huTjLQIKTs32iMMWr1/Lef16gReEzPQ6DEYpG5fWsXnO3BOznF/uYHVBWeSEKqUdZHS7LWbml1BRj83EYzsRkjRlNB5x9a2r5G9cJOrCr3rfIu97don24ilSv8PPvnSDl16+wan5iC96pk/fy1jbXOOTr6/y1i+twc4Grh7CiL33h8XaXz4Emx4e3C+hEBRKKZR4WECbEmNdL2CxqOppr554wGIwt+nlHgwdIn7rB76S3/AVv4UP/poPcv6Zc/iBwvcUnheytr7J7/2DX8PHX/pFdyXT5/9IcL92vRcceVYKXaY8cGmV+4AFVrMxr1+/QmBO0Q6WEd8DL0SyHHXtIp7nky0sEMwvoeYWCDtdoriFLgtyz0eXJcaUeL5PiGCNwVigehFrffTtNks3+MgvfpQs6zIZaMo8A5viigAZXNcRVp9R4//giLCpPkscOS7Z6/wF1wTK6v/jxnqNy5ddIGiUtSixBJ4lDDWeaBw9KbGmxNoCbQqsLbBSYNEYBcaI69RE4SlBleAKa1nXdRl3LE2JUDSW3C22qEh3gTEaY8pd8njYEFEoQjAevlcy3ws4P9/h9JklZmf7jC97RJU189SiRwO6nTaz3RkWu20yHTOYpOyMJ2xs4whyocFoxFo8z8f3FEoJKKG0iizXiMoxGowBpRSe5+H7Pr7vY61hkkwotCFNUkpdEkqIUopSWTQlRhV4ocELAuLZFn4ckA49bGnIs4yCCYLGWB+x+rak+LBgMOyMd0jSEM9XxO0uS6aN9oSkGDMe54SeIfI0qi1YG7pB0W7RxJK8SJiMR4yHQ9JJQrvdQ5SPMVTk2FZ34WhyWeYUeUGe5KSTHFNaQj/AU4rSFBRlSjIakecTPE/R68wQqxkmZn8BsineedDVYtglwbt9K7i3iYAK8Ja6LK30mJ0JEAzXzz2DvdIjXFigTHJCgV6oMbFlM4EkN2xPCuJcocSy2PNod2Jm59oksxFhKJR5wcp8zOJMhKdcn7i9s8W1jSHtFFSroB8aWn6ACRXZxJJnJd1+hH33Mqe6mrOLPr2WptMRIs+nHRiUnmDTEeVoBB3otQwzsyESKtzwUuP6/vpdcrJ4WIjHM90nWFk4w9zSIv3ZWbr9Hkp5DMcjhuMRk3RClqV0ohaz3T4KyNOM4XDI+uYa1zavcSW5TnmIBVkLNNujHdY31rixeoOoF9FuRURxSBjEpGmK1tMUyY8yHut0bRPgzfGI9lZIrzuDdELED/DyApVcxwPyM+dIl88QRiGq1yUIQ+JWG0FIbYIyBt/z8ZTCaIM2Bl1ajDFYc/TVj62dsHbl42D7uOI2fmPx9i1Nj7HFdXjgOsACR4yLalHVMYJqW1VZTBr7TIAMTIHoEmU1vlcQBQVKmV1PsbUlxhaIKTA2w5JjbVH1tQqrPDwP59kTg5YSRYHYAmyONRlGZ+gyR+sMU7pFlzllmVEUznucZyVFpo9sQCLioVSASEAYwtLSIk+dXWRlaYZuy+N6pGg7i5BqSAaadm9Ab6aHF8XghXT9kNizmLJgMkkpjUWJI8Oe7xEEAa4SKRhRFAZEG7DOw+kpwfc8fN/DVx7aGtI0IcsLsjxDa4O1YEWqYUmJJUf5FomFoCuEbR+/VDAUTPXbWyw5GuF4Xn+5MWhTIJLRiiPm5jsUeGxtw3BHExlNC4MnAhIw3PHId58nQ1kmTEZjRsMRySShKEo832A06CYxFg22IJ2MGY+GjAZjhjtjytzQjmOCICDNxiTZiCJNKJKUJBnh+T6R12NituChDN2nODmoSeH+J6MixOA+lUfQ9um1PKIQep2A+KklkjDGa0Vcv75NpxOAr8iylDzPSNOcIjN4ZOSFhzE+voppRZY48hDbIs8DWoEgGOLIIwx8sjSg1MIkt0zygiAv0KVgMpgMxwy2x3Riy+kzXVa6Bd0ox5oUXeYUJkdnCSQjClKG20JsAohilFfflzlgOTnEWCE833uK3/zrfyvPv/ddnHnyHIunlpmfnyeKQpIkYTKZMB6NSCcJvV6Phbl5yqJkbX2d69evc/HiJV59+WV+/qM/zWuDt8gP6TnXGDZ31rlx7TpLV5eI2hH9fo9ur0Mrdg4Mo6d9yqOMx5oYgxMDvDYYIDdWyRdm8WbahJ6H0hneeAd1+U1Up0cU+MQzs0gYEfgBNtRoXWKNwfOUI3Vau/ewFUSOzIu/DxrsZfZkDj2gxR6hrQlwyR7hrelPgpsuG+C8wTn1lNke6mnDmkwXjWMmwBCjh5TZCF2MENtC+QqlDMYUlZSiAFsgUoLJwGYYm6O1UJZCWSp0IRRFSZ5n5EVClo/IijFpNiHJErI8I8sy8uZnmpFlKWmSkKYpaVKQTAp0eTQduMJJGCRs0Z2Z4YlnXuBdz52lZSfo4Q1aVjODm3zcBiZjmFwYMtx8lU4noNWJ8OI2c1GXYK5DL4rYmRRM0qLyDCuUr7DiYcRzHmrx3DSgceRYKbXrsxcRsBZdaMqyxGiNtYaydHYsyhJjQKOw1qKMR1Z6KO2DCfFo4UmCtm6moG4ZRz2c86XFmaWn6fdDyrygmIwIVMpMCKfnc7wyoxjlFBONZyxxIPR8n0FeVK8ui9YZySRlMknJ0gJdGoyxaGMx1mJFQMBqgy5zNtdXuXzpAqvXV9lc3yZPS7rtDmHgMxxtMRhskGcJOs/ZuLHOpcsXGRXbx2CNKU4+akdD7TVuyijqxYMyJX3rGpejMXbeI7AarxiCGbP1WsHPvarxQh8VhUgL4pZGiWYy0hQjgx6BzRT+Up+FJ+fodnysLsjygvHYYEvh1GLM+aUWS22f8+8/TWosm5nh4rV1trYKdjZzRhsj9M6IyQzoFUXRM4yCkjj0Ka8YBmXMK6+toq9ssdPOebW0XO/45F7I9dUSuzNibzaxdpCcHGKsEM63TvO57/81vP9zP4fzTz/JwtIivZk+7XYL3/eJ4hbdTg8zt4g1hjCKiFstR5bzgn5R8HQYMdPvEyPMf+yjfGrzFbZt/sDXpzFc277EhVdeo9vv46mQ+aVF5uYtvZ5iMikpy6PtVzwgVIKHYIHcGspjlKo+7njsibEBbpiSYmMND2E+btFpKUcFszFcfQsRRTw7T372CcK+T+D5ELjpXWOr0bQ1aAWiLNYoREmtCTgGbFWfAY4Y157hmgwX1f/tvsURW0eKM/YkFEG1hOzJL8LqHPW0WonT3E2wdkSZDdH5EGsjPM8iSjvJREWKqSUUUmB0icFQGqEooMiFMoc8L8jzlCybkOYj0nzEJBmTpGPSLCVNM/I8J89zR4jThDRJGE8SxmP38hhncFR9jgIC30e12vTn5zj/7As8/95nyK6/xvrWJVpGMwusAdcriw5ymFnXLGxq5qOU3sKY7orQ686x2OsxKmB1Y4fNwYgSwYpQikIrH1E+vhfsEmMxjvzVt+eIcSXh0Xs3XZYlWZpRliXagLaK0lik9MgLhZ8rrA7xiQmlhWGCRRNUutz9Q6PDRchi/2meffazmF9IMfoq2WSbMBjTkpylniK0wqYq2SoMnraEHvQDn1EesEOJxqJNTpamJJOULCsota50xAZtAREnxtEFWTJh7cZVXn7pk1x4801Wr6+TJjm9dpcwCNjausHW1nXyInHbZxlpnqJJj9QSUzwq8HD9Ye05bWqMa6eB7yRsl64wLNe4dsqjG1vsIIPhBK7vUOZDSvFBWhBFpCsxfhvyGwnspFBqsFCs9bi+OY+3GBBGJTovya9rGENyvkernOH8u0/zgRfOsj7RfPil67x5YZ3hpQH2+hAmCZQ5pq3YGAeM+op2YBEUk3xAPlbo9RHsjNBhwZWRRYUKbX3sTgn5COcksdW919K5k4FT4SJf8IEv4nO+4PN59rPezcLiIu2WG+SCoEuDQhGGMWE7IAwCDFBYQ2YgNxYvjFg53eP0qWXm223OLSzS+k+Kn13/FMkhDIY39BavX3yZVreHqIjJWJOnHuWiT1EenfMGYF4Jn3tumafPrTDb66GN5vLV67x1bY03tkasTQnyA+OxJ8bguroNnfHa1jqn2m0iOhi/kvSPh5SrV5lcvoBaXKF79inaC8t4QUQYlSDWET1TIuKjPMFa8H0PzzsuYlxP6dWeYMNeYF1WLTk3T4nV5DbHdX59IERJmyjq0G536XQ6+EGAtZbRKGFjYw1jBriXRITzTCf4IlgzIU83MVqB5Iiqjm2dbMLYAkOJsRotCo1g8NBWOf2sFkyZo4uUskgo8oQ8m5BnY7J0RDKZMJkkTJIJySRhkiQkk5xJUjKawDiHxBxt9y3K0m57tOI+S2eXWTx3mrnTK+wMr6KMIRChr6BrnEUznD8+x+mDbQLZtZLx6Dpxb4ug3yHwY/qmRAWa1GhyayiUpRCFF4SEcYTnBdjSYkpNUToSKMoNvjwUvh8QWBDloY1BKUEbQ1lqtDaUuZMYKGMo4xKjcrwiIzY5Vmk8LVg8VCVAkEZrOUx4apZzK5/NB9//OXzgA++hE+5g8wtkw9fY3nqZdLyDMZqyhNBX9HseEzGMS03LM8wrgzJuKOeJwZeUshiT5SPSbIIXeFhrMRa0sZRlwXi0zc7mdV75zEt84lc+ypuvv8b2zhZlWRB4MZ4o0mKb0iRHcMfHiZrA3AsOei3ead+3u/2d9t9/rIYU4bbf3es13ct3bxd1nEUtUzuIGEfVoiBN4OKYnesFO6qAMocyA131xdYHW0KSYy5n5IFAkoOtA58FSgXXFXpLSCQHbSBz5Dvb0FxpZ4Tk7OwMubJT8qkXNxhf3IHBGPSEXefFSLBveqQ+pFJdtw5Ae2Aqh0lSYlJdPfUKrME9/bDnLa4dJg+fTrWIeHb5eZ56/gVml5YQ5ZOmBWUxxBhDUZZgLYHvE/gBnudkaZM0ZTAaMxiNGA6HKIFgYY7ZdpszZ88Qi7B1fZXVX1jj09k1HtRvnGN4PX2L4adSrq6v8dTTL/DUs8/zzDPPErfblMXRvanGxrK5M2S+ExGKC0CejEaMJinjKSk+FLwjiDG4xnKjSLmwtUkvUvj9mEgpvCzD315ncukNTLuPiE/UmyPo9ghDjQjkWYoxBuUJnvKwFoLQR3nHle2u9u763Cx3GOF0wDVBbpLimkx7QAslc/Rb86ysLHP63DKnTs2ztDyH5wlJknDxrcv8wi8M2dy5AszgSQtrnYq1HYYIKVm2hTaApCAFqLTSGLvgOK0KtAVjI4xEaHyM9bFWsEawpqi0xJkLyiwyyiIlzyek6Yjx2HVqg52U4dAwTmFcQmqPh9IopWl3hMXFGVaeWGJuZZHOwizjKMIag+8JXXE++7iybm39+hU/0OBvGzo7CbNRQruriNseYRiTiGKiFRmGXAle0KLdjvDDNmVhKHKNZCk2z1AiTmrhWYIwAE+hjUVrTak1ZVFgtEUXFp1bp73VBjPJwVOERQJMCPyMWCwaD60UBgi1Ii01Q3t4GRlEWjz37BfyZb/ht/Lud38WTzz1FEW2wdbaG1x+I+TS2hW2rhfEgaEVQRzCTA88A8UEYgVzPoQFRBasZ4j9lLIckKQDJskA8Z0O2xhLUWiSyYT1G1e4evlNXvrEx/n0i7/M9uAGdWspT44T7AHRB36z+2zOUtn9/9mvHW32AR5ItWDAVrNKFlx/UgfoNuIVJKy2O+jpa85M1cFrOXuZHWrSFbP3tDR1vDUhlX3HNPuOvac7d2gGix0W9hPEppTC9Z/Qqb5PQI8gacrTSvbiNeprBIoSivq+m8elItRldQzjji8KRgkbVyfsXF/jE8aQj0rMsKiI7v7AaQvagtaN9WHjOir72bJxDbAnofAax3r4UooQn6c7z/Dsc+/l1NnzhHGHcZIxSdzsWJ45eZUFut0OrThys2dZxtrGJtdX18myDM9XzPS7tHyPfhzRn5lxsrbP+WwGW1vkn/4vvFxuPHALKtFcza+yfnGTyxtXWdvaxBjL6TPnyYujy3aTAZ/cmXB1eIml+BqewNVJwYbdiyp6WGiGqz7KeMcQY4Acy2vDbVq+IgwWaQdtlCnx8gy7sYa98Bphp0+ysIwRUIFCvKrzcBFPYAXleQShj+97x3TlTS9xhqNjJbdqh+sO3kkjlNchjvrMdOc5tbjM2dPLnD27zJlzS8zOden1Y8oyZ2d7i2S8jpIt4DKQ4nst2mFE5C9wai6kHYM1I/LckuUTvFCDFFgpKwlFlZ2i0ruilLtq6xwiZWnRZYE1JViNNaUj0zpHly5FUb0UhaEoICuOxrN5e5QEUUFvJqQ/3yHuRUigyDGMs4xhqhmaPdX2hD3/Uu1nD6i8yRaKFDqZIR4agrhE+dAKFGGYYwLwgxZRWOJHjihnSiFWYY0jf6YsMSIo8Qh8D2UMSursKBYPSyCW0AelLJ1As9DOmWsDYYKJE0pTkBvI8MhQFMajLDwmk5J0UlIcQhfmefPMzZ3j/R/4IO/+rPdy5uwT9Gbn2dkuKaTH1jjm4nW4esHiC7RCmO/DXB+sBuVBpwPtALo5tFLIvRyTXOfGlVfAF9Iip92dQcQFwSZJxmgw4MbVi1x+6zVe+tTH2Bms8mh7hW+HJQi+FPyZisPYaooCXN9ExYVrkloTqLpPCEAFEPjgK0ekjN7bvyzcdAea3VbsxxBG1Thcu/6vJuViK45XkVddVsdIcf1RTbCrNJBeG6yqrq++2Jp0NmMiaHxf632b+9SzYEdBjOtlH4HFB38elk9BnsHOFhQG55Sw7OmTa69yfU8daM9CK4JR6va1VVo01YGZGSgyGK0CmVvXb4PNMVsT8nQCJqnOETSurzmwqB0z1WBnd7DRHHBY3PugOSip7dsceDxcKiPAaf80zz/1XlbOnafTm0FbYXNrwPrqKlcuXWQ0GmG0pdfv8dzzz3H69ApJkrC5scknX3yRX/74rzBOJsx0+zz95BN0fI+VxXl6nTZhHHPuqfO857Pfz/rWBjcuf5TNB/YbO+SkXBq/wvqL17m8dpEnzj7P5mD1UI59+3PCNWNZmxQuE8c97eUyV9Up7eyuM22K/TgkYlxPQZ18I2+Ygk9vbTLXbjHfbuEpHx+B4RB19S3Sfp/B/CwtKYnmF1Fh6OLfTVlpi5XTNwUBQRBwPFNQtd43xXWGtURiB6cjrj0AHZA2fjzH7Pw8p5eXOb9ylrOnVlhZWubU8iKLizPMzfUQVfL/Z+8/lyzJsuxM8DtMyeXGzXmQjMysqkShIAC6BxBptMyItMw7zev0v3kASHeP9MhAADRmuqpAqrIyM7iTcDdudrmyQ+bHUTW77kHSw8PcI6KqdqaG2TW/RFWv6jnrrL322o1bs1hc0ZQe3Jyy/IoI944Joc9oeJd7h/vsjRPGvWj7VlYN86Uk64FJBShHkHHCCtKC00hSQvBRhxygsZ6qdrjGErxD4hG46GbhbSxyDB4hAlKCMZCYyB7W7t2tgkOwBFFgepD2NEJ5KlezqAouVgVHc8fjAI+BC16eUjqn6G4aW7ebCdCvoF8F+hJ6PUcyWqN7Cq0ztIxsWqUMSRAI7fEqUFhLWTuC1OgkQykVUSS+zRsElPBkSSBJAn3tmeSOg92G7ZHHhArhKxpvKT2snWZlE1a1piwlc+Vp6oYLW/5gcPzn//R/ZHd3n/d/8SHD0RAhPOvljOnVJefnFxyfzHl2UvN8Fa6v1N4U9jPYHcLOGIZ9MD4SY70lLCtLefWYz35rOTm7YPv5GelgghASay3L+ZL5dMrp0VNOjx5zOX1K+AnpJG83NCQ5SB1BqHfXelVEC+LCJhvYudB0AFUSvRNlZBibGlwdXyd1m17vGEUNOgHTbiFEljmE9rPazxABWttBmqat1+0+z3PDPMv4OqEisHYduH2VGe72fxP8bV6Xm6/bBIW3ER1r2n12x6R6IEE8POC9/9uvKBYlp384wX9ioGi4cQJKNo63fb/hHfb+h1+zf2/Mk8/OWX5+DmdTKJeIB4fc+9fvs74quPw/PobFFPYm6Hs97Okcjn38jnHEuXVIHFVqbkiQjUXMSwuMLrrzKjf2q1s4ldxojOFGRvKupIFfj5SUR7sf8ujDj9ja2UMoQ1nUFOuCv/6r/8x/+qt/x2w9JdUZH9z7BVvbE9577xFlWbJarfjt3/0N//mz/0ATKhQJJ6cf8GB/l9/8+iNCS3KNdrZ576MPODs548nFCxbF41shBmIECjfjD0f/J58d/w02FLf0vt8drz83CqQcMxzsopUCAovllLqZ8rqw+nXiNpHQ5pL5XSPLWwLGGTds5k87AnDuG55Mp+wMcsgTdKYR5RrpGsqjpzAe4ZVEmIRkaxshIkvsrMc2DqUESmmUeleEe+dF3NmpdQMcgAFhECYjHWyxtb3DncM9HtzZ4+Gdezw4vM/h3h22JztsjbcYDnN6fUNZLZgvLK4JGF3h7AzrZ+171jTuGesCvNc4lzCfr8CvEbpmXQV29/ts7Q4xuUImAi9jEwWPR0oHIlA1FfNlw+y8YH65BhfopQlCuOvCurIsqauKuqmxtsG3dmS0c67i3Sw9AFzw1H6Fs2tcvaJYzVgkgsvpJUfTBU9XgS+Ak42z34XlZbM7D9elXX3i1OY8qBUkPpCxJBUWI9aQjUllTh0SpPQI5RHS44QDIUgTidYaZwNOCrwArwQqSBKh6GuYJJ7tnmN3UjHug2GF9AWNdxROsGo080awKAVLJemhUE0PMxfMbMkK/8aDz907ewxGI6R0LNdXeF+jlebi/Ijz0xNOTy+4WCyvB3EPzD2s1jCrofKwN4RhCqmBXg5Swnm5ZLr8nMW64mq+RqYjnBdUVcNivmQ+u2Ixe45zl/wcFuVvHhK0igDTvSoxaP9ddOD4VYDcsZoqMsVCtKC4jH/3HRhsgabQoFt2WbWASnaSC3EjwRABlIxf1EugWG98Zgu2vG/ZZvUN+7kBir8xD7spadgE0LcZmyzs5mMJKqH3wRb/439/n/PzFf+hqplO1/BVCX4AWQ69LK7gK9uy4oL0F3f4N//ml/z64Tb/bvyEv5RQpQKuDL1fHfAv/tlDXpws+cuPzwlVwNzd5vBBn+cF+OMKhAG9BSoF3QdlIuPcVFCuW81yBmoASQI9CdrHNFvVtGsIAVkKowy0gNpHvfNqAa4ArSHTMWUjXgXW7zbGcpvDO4/Yv3uP/miClIa6rlivSk6OT3h2+YQyzJEkpMcZx8fHXJyfU5QlRVmwXC1oQkXAYymZr6+oyjI2BVEKoRS9QZ/dwwPufvCI9z//gGdfnnIcVrd8JOGdgeLvFxmj8QGHd+5itCT4QHbV4/zcUFUXxPFgMwxRQlRz0xjs3UUCjLlxFF9xQzy9i7glZLfmx07FfJ+wwBfLOYNjkLtb9PUwNlTwgnp6Rvj890gho9Y4H2K0QWeS1XpNVRWYoJBStH607wq21cRLo0vb9UAM0b2M4d6Q/TtbPLq3x6N7+xzuTtjfGrI92mIy2GGQT8izAVmWkGWgtUU7S2I8StY4u6ApI6DpIrDkfP4xxccvGBiFDhW5aRj93rG3D7/+swG//rOHjHdHpIMMmUi89AghY7Y2BKbzOU+fnfPsi2OePz4lVZoHd+8wHOasqzmrYs66LFiXa4qyoCwr1oWnKKCqYgb3XcIdG2C5WnJ1dsJ5L+d8kJHNR7z46glPjud8XsFzvnmV3rHG3TTeqQc707s1Ld8ToqaW0qOma/KtNen4nJCmeNMjIyVXCVlqyJRBGEU20BitCTbS5xKJFpqeFgwSzcg0TIxnZBr6uiFTltAs8E1D4yPwXAfLkoqFUizSjFFISZ0nEYb+KnBV11Thzc724yd/S94bsFqdc37+jJ3tA8aDCVcX55wff8XV2RFlPfva6xxwZmF6BqMLOMhgbwDDNovfA2oa6vqC4lJR2JRF4VhXDXVVYu2KKGL5+wyK4SXtrRSgWqDYMcahZQWlaLFwx3bajde6+Lzrf2tlFrJlhL2IoFXrCHhFC3qFjJ8XOvlGO9ZJEcG6kvHvVNx00RTcMJmi1ce6m8eICMDlJhgLr+ie4UY+oG5kG91z3tqYu8mep6BTets97uwNqMsKQQNbCWQPMYlgdDdlsiVZXC5ZnC1oSo/3moe/mvDPPxzy4CDls88tv88L6h1J6I8Y7fTJjEYhEEoQEk1vq8/B7pjzo4oiWSO2Ruz86g7DcYYUliACVkjWpePy43P80ysYDUnf22e0l9AbWQIll+cL1pclvlLgUtK7I+6+PyHLFKuiZnaxZvF4ir8qSO4O2Xs0xgiLq9eEH6kphSFhL7/HZHePrD/EJDlKpWgZSHTKeLjFONmFSjJUI7Ik4+nTpwhgOBphtOLB4X2endxlVc9IRcoHex9y5/CQfr+PSROEFIhEI7KEfDxi584ddl7scFKufkbI5U1DkGUj9vf3OLx7B61krJkxBu8DFxeeur7ghjlWIEaodIyrC/BT4iz29qLL/3T5mjGRTArcCMM6/613EbcEjH9+E9M0eD6bzdnONIfjHgmSJHjEfEooC0ySUR88JBntYLbGqDSnqGqcc21hlCaEd1V8t1k5nSLFgCybMNkfc/Bwi3sPtrl/f4f3Hxzw3r09tkd9RnlCZjKMyFEiB5EgpUAbjxQOJRuM8YhQUayuWMxPoi/xRgQ8y3rGsr7ZCzmD3hGsihlp/oKDpmGwM8TkBqFAG0meeYIPXM3nPHvxnN9//CWfP1kwNIpESZTap6xL6rqO/rytN61zkXiqK6jqCFTf5ZXlPVG7enXONNVc9Q3ZvM/Ji694cVlzFL476dT5hHRlPF05U9X+vbuxPbHoLFlDvwFdOcxgjeitSYwhNT0yM6BvhogskA08iXEIZ1HekmpPZjzDNDBOAyPlGYqGPFQoXyNthW0s1kUiq2mg10BmA7mzZKxIjUXkCoUiTxJ6dUr9hif7yy/+FpNknJ0948XRIffuPOJg7y7L2YKTo+dcXB5hw6uMxMvn7cLDcg2LGu4OYHsAhFiUR1hQVQ67VqyWDcU10Pr7P6XFaFPrsgVtm1l/aNnYFkQKAU6C7+RWm+eoA5abBXrtSzupwyZYDcTHSsbXxcra+AIpI6PcsY2h24dunOqAcZf278CsAtWy0aLdXx+inKMrSrgGpi2A1urm+cG/DNBvPbqpuGW/s5zhVs7WIOEkkQjlMXsZ9+/s8Givz8NtzyStefYVfPm0ZLqGwqd8dL/He3spk4FAUxDqBSo3yK0hW5MewhMtGLWEPKE37rO9NSIbrSn6aya/vMv//X/6U+5uJ8yvzmmamnQwYBU0/5/xgGdpzvjehP/uL+7x/p4iVwuW6zl/+MrzxYmgCD2EGfPLh1v884+2yBI4na15crLgdz3FfFbzF7/c41/9yR5JKFgvLrHNj5PxnYhdDrYfMJhskWY9tE6QUqGUITUp+7v7fHT3V1zNLjCJxhP4/Se/45PHH/P+vQ/44IMP2Nvd51/96b+mKguMlNy/d5e7d++S5TlKawIeLwVBKUy/x3h/l63RHqp8hv0ZjCM/hH4TaCbDMbt7O+zt7yGlxDuH9yHapFYVdlrj/bJ9RYJJRwwnW6zXKeXCQugkPLcffWCHeHwVcQTpc2NI2wmz3hXagn9gxXebESUVnrOqYS0UmVAkAaS1SOewF6esvvg9QinUL35NdngXZVLSNEMgcM7j3LuanHcQ7JHqLSbDffb39rlzf4877+2yf3fM1nbO1lbO3taAvcmAfqbJjUBLEN5FlkMo2p4ICOHRymMMVPWKF1895vGTz7F/pJNfJ+BYWPj9l7C1M2W+dgx316SDDJUqegPP7s4OifLMV2vOLi85Ol8xtyCIq9TtnR0cI6yraHxDbRvmiwXTqylCXLJeR3eLrszmXQ1bIoB2AVUViOUl1VnCYpmyOrukWr8eSO9UgK8mu7tSyY6Puq4jb4BLyFeQ9kH3G4aDOb28ZisLyMyTphUmk2hRY2RDqhsy05BKSyos2lb4oqasLMIGZFd/1YCvwJZQd5sDJwPIkizAWEvyyZDdZBunszc6b+v1EtZLFvMrTk6+4vT0Kw7272NLy+JqzuXi9YriKuArG6WY4zkMZcwCj7YDd7dLikYxOnMcXTqm9ue4HH/TsFAXUeIQfFw9trZVkd3RLS5u3QhEt5BO29e3IFVsXpXtNOObqF0PLWPsHEgXQfC1oqBtJhS6zd8UANJAWcfiu81CMZlE9tnZuHXAWLYgV7VAvNPydsWD13pn4r8pGd9HtyA82q+0APq2ogP0m+dHgRzA/oRBX+PqCoIn6afcGeX86z+/w4e7KfX0mNnZBbPLM84uLlg0CVaNma8tZ7Oay1XDf/7dGZd/cwSHh0x+MSTRhtmiYl05GPUR2tAfjzC9AWY4hElBb9Jna5LjQs0nX52zXK359a8esXcw4Rfv7VF5w6/vD/jXf7bFWK25OF5wcXnC7GLGYmWZ7I05uLvLZGC4uloxnc/47NkppxdLlrVEJylZKMjqBZmqUarG/QieBgrNbnqHrd09kiyP46IPaB/QUtLLM95/9B6DXsrz4+c8e/GUPzz5AydFrCn44vRTHj//M/7sF3/GX/zFP2MyHCGCYzQccHhwByEkjXU431BUDUVd4zyYNKeXj9Bo7DvjId88fggwVsrQ6+fkvZw8zxFC0NQNJkkwJiHLcop0SFUqAgEpDb1ej36eoUTA2yH1uibOYLc7G6fABNhqH2+YGl7nyNfcWAy8q7gVYPyuxAS3HTVwXltmXpAGQRJAe48JHnt1zvqLjwlCkGztYHbvoFVCnvVw1lGWFf4ttSZ+OSSZecj+1kPuHe7z4P4h9x8ccu/hAYcP9qPO1wQSA3kiyI3AqEAiYqFWCM11uQrXqXKPkBajAlWx5NnTL3l++v2qaC8L+MPHjkUxY7xfkg1TTG4YbXkIewwGOVezJacXM65WPvIwAZIsZ3dvH6kDQXocHuscl5eXGJNT1Y6rqwtCW0r1LsuppIDEQ+osqlhQX8JKSoqLFbZ5vRVrt+rdJPS6cESP3s7YKtCaK3kIBcgaMgupDGhTxPbQpkQridGQJoEkcaTGkmoLrsHXnqYKlDNoVhFoKx8lhQaoCigWsF5GTe/KQmnAJqAS6Gee8SCQ7oyQ6eQHnT/vHeV6yfNnX3B+dowIEtcEavv6Or5AW2bqYerhQQUHBt47cHjhGZh4vqoLWP0cB503ChstTkIawaRtVzydljeEVk7RMrCiZVpDynWBnRRAszEGdFdo07K9If4MKrLNvn0sZGQ1xYac4ppsbhnkuo7gugOYykTdqzHgmgjifesyIUQLjLuiQW72GXipELD7bK1aiYcC1z5P3uayqNNGb4TIYHtM8miLYS5ZzJcUZYVJNXd2+/yzX2/zwbbi4//2jKcXZxwdn3B6dEHDENFLOZ83vLisKBvH008u4OQMsh0GeQ8pNZfzglXVIEc9xKhPPhwSkh6m14PJgGyQoY3galrwu8+OWczWjA/2ufsw4d7BGKThNw9yfvN+H79Yc/nllKvjY05PlqwKzf59xYN7E3xR8+T5MZ99/ozT3z8hzJewv4u+t4NfG+qpx6QOKSrkj7DUTMnZndxhvLODTlKcC7HDp3coKUiylPGjB/zig0d8+uWnXC2mLO2c0IL40s355Phvubv7kF9+9Kf88sMPCL5BSsh6CSEI6tpS25p1UbIuamrnUSal1xtihKG8RdvKtxU/ZKgzOiXPM7IsJUmSdm0b0EqjtSFNMwb9IVonsahbSYaDHv08xWiBxLOQgaoQeB8zfyF04oY3D81GL18jEAQSGx2dSl5uT/auy6pvBRj/XOenAHy1LvjtV8d8tLuNnozQQmJ8gywK/PkRdZqwPniAHG6jRyNMlqMTTwgBrbtK3rd3BpTs8ee//qe8//A+hwdbHBxMODjYYnt3wnh7SG/QQ8mAUoFEBRIRUMGhQiCEgPUuznEEgqjBOoKIkNP7hvl8yvMXX7Gqvt96zANPzmC2DozPSpJBBRr64xXHJ5bBYItPfv+UJ18tKVsiIiCQOkGZFI/FuoYgFUJKtMnIsj69/ojeYEG+rmiqd7tKNAoGuWCQSjIFoVpTNp5yEaj869+crzbm3gxPPKZuFdwVFWRE7OCXUe6JA1mvEet1rIHIRXxSGggGrI7Pb6oIfFdXULWiZukhM7GuplnDeg7LJSyKCCYrAZWCxoDPIDcr9HhJkr4ZY/xqBO8p18s//sQ/EhVw1MDONLLeaRroi8gkp8Rz9w8qnGvZ2k51t8GsBtmCxnbr2OTOscJ7CDYWcF2r9jaB0CZgVi3gbaUVibnJZXriPjh/o46QGkx/A/TqWLzXscPatZ/dgt6uurbd9VjE1742JDfMtBCtxdwGMJZt2utWpRQSSEH0IVXQU8jdPoMP9rj3cIwv5vyn/zzj6UXFyVFJrgLr5QKXa7A1wnvK0mEXDoQnKE9hHY1rkCKgUgO6jxymTAbgfcHRbMWisCSD2KjCNpaziwVlXUMKNtScTedcTpcsykDVCK5WFRezGeezgsv5isWyR1V4RLXCulhP0DSS0Ci01PQTzfF0xadHM86+vCS8mMYBYzhCC4cr1yynFjM0ZH2FMu8+gWzo0xuMyfp9lI6svXMW2wi00Wij6fd6DPKM0/MxUidfg+821MyWK9brEusdRiu0kYSWLa7qmqIoWSxKlquKpglokzEcTBjIAQu3fufH/X3jza92Ra83JM9zpBA4WxNdajxKSrIkYTjok6bREzoQLUF7vZx+rwcE7NaQshizWkxanT1UTc3F1RHOv/korkUx1QABAABJREFU3M2PDVA14fr3NdFva82PUfYX4x+slKKLpff87dk5QigmowmZlqR4ZN0gm3MaJVh/+QnkQ/rvfxQrZtuCkyQxf/wDfmAMetv8q//LP+PRwwO2JxnjccpkHCUdSoN0FhUUGtnOXQHZMi7Bh7aFrsOFBhsCja1wro5zloT5bMrJ+QVvQn5XAU6WcLEGqaIVlzIFn3/+JVn/GVcnNav5zfMDgNT4oCjqkrJaQ9vhzTqBMil5b8hoNGZdXFBZR7mBRt82n5FowajfY9iTZMYTXE2xbCjWEby+LkjvpBLdsulVQN0moK+1x50W2RPJNTmPDLKfg05BZuCyQMjBZdDomN22LvYQWK1gPoWiXVrLAIMM+tkGMC5vVt91gNrGzZYQtKM3ukDLn94Sdw0czWF6BZM+iBISC/qnt6tvMVrg6P1GIdtN9ufauzjAy9ZjbY4j+JbRfbWF/Kbh4KbdZgeoVUwtBN/KNUQLjls5AyJqk5Mk2oko1eLdTh/cvgfhZh98KwPpWpwL2gI/GYG0bJ0wvONlexp1A4yhlXHcViiQOez3kQ+HHLw34YO7Q+5t5+Q68F8//Yr/+oevKGeBUGecBMvjLzO2m5RqXaKEwlpNWMt46npQO4f3Nf1U0t8bMN/dJTvIGfcsV6uCk7MLrFdMJtv0M81quWJ6UbNclCBr1uWaJ0fnzJYVdTB4JZguK56+OOfxyRUvTmaM5JA/3SnIQ0lpHU4aHNl19zvhHJeLgoujKeHFPBZu4MF6jHQ0VcHsqiCVA7LeGK2SWzynrxOCRI1IB0NM3llSery1OAkkEmUkJlEkxqC1QSiFeAW2BGCxLjg6PmVnZ8JolNPrZyRJgpCS1apiuVyzmBesFhVNHUh0xmi4xU6yw0lxhv/Z0nvfFQKjh4xGE7I8BwJVWaClhgBGCfI0Qcnh9X0XgkMQSFNDmhjSNCFJNLZpKJYrmrpGCkFV1XzxOOH5yZM3BsddHU63LOlKeLt56seUyv2DB8YAC+/47PKK3ckWatjHGI0kYGyNXyyoXjyBtIfKc8z2Nkm/h1QSqd6+96MQHqUbtK4xRpKlirxnSE0g+FiIooJAeImzjkCrKQ6xULD2lsZZmuBpvMPaGucb0jQhSxOKsmBdfHth1OuE3ZhTbQPV2kddwKtXthA4FJUNLJYli+UCqTUq0ThrsV4idUraH9IblMyXc6S7meZfLSW67TDa0OtP0D2Bp2a5sBRLOCvhjJsb+HXjWsLyDX9vuEkVTYmkcEYLURwY1+qDWxCsTQTFOo1OTkLHPgO1i0Vr8yUUrZ+NJjpyubZbbVHFdgQdQ93poCugCRCmwPMZ6eynx5wEYGbh6gqoYXEF0+XbrpH+qUULDJHtRaVBprzkYezgZY9fGS8a2RbGeXcjlRB13EK4YX2vf/FcM6gkkQ2+9vRtgbWwIFqXCyHankIqyh46/fGmXDe04LbzNA6txdumw0THEnc6ZLlp7cYGIA43Xs63Gg5sSaglTSlZzBqerhR1WfPZ3z2n+PhFbNssBsyrhv9dFTw+1NTLBZcXM06fr2DZ+s1XU87zF/y7UWCQK6bPL6GsqY6v+DhxrBYl5fMZCMXVTsEiTbHLGr+qCW0zlYurOf/l9IKm8tjjApznqWs4f3bK8mKBu1ryX857+PklQ+04O5/y/GTN4sgSVg3PPjnBB8HR8yvc5+cw79oRAfOS+cmMzxaBy8QzOlsxPF5jEsP/45bP6neHIlUD0n6fJMtQJkEqBRKCCAgl0alBJtGdJ80zJlvbbE0esD6fE0IBKIzYJk97rIoVl1eXWPrUvkev10NpQ1HVrMuKqm5oXLy+tcnoj0bsDg8YFs+Y8cMzXD+10HrA7u5dtra3yfMcvKeuCkSSkWhDlqWMJyMa6wgIfPBtEy4Xa3AJhBALXaWUmCQhMYZ+3gOgbixFVXB2+Yw36TbgieO446ZlWbds/7GXKT9LYGxo2/IKWId4Un/oyTyrC/7uxTH6zgGDvR1SnUTnzCbQHL8gOI8cDmFnm97uIdIkBPH2O98V5YzPv/gtgjtU1Qgpd+gPd0m1QKIQQYGPHryuKbG2xPvYprlxNZWtqG1N7S3WR5ZISAhhiDEa6zzNN/TOHeWKdeUi6H2T+IbXCQEuCIqqYTpbcnU1Q6cGk2Ug2pbGaJK0T9Yr0WaFKN11L78OUL6tSE1K2t+GXLCuC6qm4GpZ8dTBKd//1v9jwHgFXHJTJmWIg0Te/rQBVNNuAupVi1M0eNUyvw6WNczbLoGdUZYo4g5bC0W4aTjSWftXG1vRwPTII+VP04e8DHA5j/Vn51dw7P6hyShECxZ1LKBLEsjzCHwR0ddwWUDVNSdvJQhZFmUNQW6AUAuqiFsgAuVG3vSN6OzSZAoqizIKqVopRAWhjlodFVo2GG4AbwvAfYgXaDs23fT7aOURUkURvPNtIV37E9vKJ5LY0EQntCgp7lxoosVKWbS+ybcYfgUXl4SF4/wJnCcgvIzF+KuqLS7UgCV8NefJ+VOeprFaIDQNFO2CgBpWa5rfnfNfv/oUoSV+Fu1h3OKM8y9aP+k6Mu7u6TOckBtJgLjY8RJWinjsNrLuzbFlptrmLt6zeCH5j3/QSB3wjSVUHkoBXrKez/j0959B4aJ3cdjIdy3muE9XTI1nqgJCCERXmf0OQ2LomQFZLyfJckyaoIyJXu5aIlONylNkaghGk40G3H1wnz9f/gX54zEX01NSmTHJJty7e4DHM13MaGjnPe9I05yyrqhcgw0epEAaDUKRD4bs7R1yb3oPW3/J6p0K9952aPb2HvHg4QNGoz5pmuCDoypLjNKoLGOQDOj1+9RNQ1lWlFVJCFEC1NQW7xwsQxxzQsC7QC/PGY9TBoM++8WaxXLObHFB3cz/+C59Q3QF9j+1+FkC4xTYM7CdwWoJsxBZtyVvLtL2wNPFlIFJ2BtPyE2GVAmqqRHzK4KtKHd3YXsHCSTbB2/PMWgjarvidx//Nc49omkOybKKrW1NohTSC4TzBK/wDqq6oK4LfGgIOKxvaFyJDQ2xzC2gdDQ8V0aitEYgvlbgrSV8+HCPJy8uuFzcHhQVUiFUgguCqnasihrtIRUapTVSGkyqGMgE5yWzWcG6vEKGgJHg67frUtHLewy37pBmEreeMzue8WK94Jg3b13zbfvqiSB1xg0/F6fdeA13j6/7hwWQNpJ1HRnXAdsVkXmuuTHKkiHOhV2fq04K0mz87KQcljbT/q4rHF4zPNH4QDSwcj9+mu3dh4I0A92LwDPLYNCPwFEIqJv4xS/bgjmpIE1bYNy6RHQsrfDxb7qTSxhoVBSeW4iSDE/XOCjqe2UExrj4JYhwA6ICUd5R+7YHR8dMa/A66p47ObQQN+L7Tgd97Y/sW7K7ddTQrRtFaFlyZyOgtHXUydrbXCK3gDYsYrVqVQGOcN01rutu195NoYZiTSg2c1ivPMc5wlVDQBCXuknrJtJVILQSk2bTkOr6bo/H6jrv59ZdxBYxBdQVC9aBcOHb21bevBYJ5YJY3NHRCt2/hbjAWHXtEjwBS/gReDpFTpb20WmGSiIzrBIDwiGMaK89hVcSpyXZcMD9hw8gNTx8/32WqxWhaVA2MMhzklRRNiVuZbHCIYwmSIkNjtACYh1AKk+wkA/6THZ2ODy7S3225Cgcs/qpDoLfM4wZMdk5YDDaIs0MUkLTNHgXzem0SUjTDK0NVV2BmNPYaJ+6XK1YLhcsl4vWLz5ml4TQjEYTEJIDIpGVJQajDPUb3o5d2cJPLX6WwNgS2bRg4tgvqji2xuHszaMOji+nFxzOdsiTqNpVUmFsCcWc+vljgkmidtRkuNtmLb4hQvB88fR3lOU5Qv6SwUiws9dHBIWwNaFJCU7GOcM1uNCADDETqTzSKFKjSBKFSWMba5MkpEmGVobEpKjrVq8xEqUYDicoNeM2OVqtE7LegDQbYNIeSmdIZZAqxSQZaZqgtUZKRX+wRdMAKGxZ4ZsG70pqF94aazwcb7F35z3yXsri8oLHnx9zVJ4x/5YJwxDlD4IITL/vtNIQF3Ott8B1YZ7neipFc6MW3SR0vkmn3JZVYLnRNncOGJ0ytVOgduDabWw/dvrq28IQcZJJYFhDYW8aof+DCJHBeAz5iCgOTGJHM6WiVMI28WIZJtw4SbSuENLEx160F4FvV1iu1RWaWHWqRMvCdlrgFuAqGdln34I3L1rw2zphBBE7rZVFZIR1C2i6q6qTNtNqiEPrROH9hla5LcrTMkoyUgWpiO/X2HZ/Wt/BUMd9V7cJYDr5SGtL91L3PsPNErVl3K/Bajd9Cl5uy/yqLKV7D0f8osTG+3XHv6kNb8/XdZVC97fN1tNdsxQ2Xrv52d0xbHpLs/G3zpXEcwOe320YckySghJ4IRBaRdmElLFTuRTU3qEJKCHJhkPuPXzA1t4OdVPTNDXNek1TFDRlRVPXOO+oXYOoSvq2ISegtCZTEpd6EutxtcdVnqqo6I2GjLa2uFPdI1kIHvsj1j9rcCyIjb8OKL1iXli8UKRa4Z1CSRnn27RH2stJkhRRaoq6xi8XLFYrXhwfc3byjKq4IoRNKkpwdmY4O33Owd4hUgjqqqR5QxykiVZtU356rPHPEhh7IkFR+dghK5OQreLw88PUsrBwNV9cXDLKMrJ+Sq41NAJRl4TTF3hrSUdbJIcP8fbdrLKdtxyfH/PsRcrBnTFX8320zJBNAzYhOIn3giACQQZUotBKYRKNSiDNNf1eFgsS0vZmQGBrj9ZJ28HvJjJt6PUmSPn8Vo/DmJQ0G5CkPbTpoUyGVBopU9K0z2A4IO/1SNOUUVHQ1JENX81mrOdLfKOwZUHzhh3a/liMRzsc3nmPfDhAmxE++4RZeEL1DQPlELiTwnYmECHwd/Po8vh9wxLBcTdNdtNgQwTdm6wxvEzUdS0culKqbsHQNRHpfm70M7qeurv32SjdeuMreYjG4Yl5iQ3Z6i2FIeIwAeQpjDxc+h9+r/9sQuawPYHB1st2Z7ItcPM2DoJNyyyKFhx3TTFky9xWIQ6cwd8wyyqNDUG0j2ykK8BVUXxuo6YWlbYewhqcij7H+Btc1TQQ2u6nRkGio4RD6JsLTOgomL+2aWsL8vA3zhSJglxBIqMnX5c/8S4K5jvPZelvMe3fgdQOHLaM9TW47LYO7HdXt9j4KV55v02Qu/n75rjVjbmbBZCb2zdF50vdvd+rABleBtrd52z+/iow7j7rXS+LBUbk6CQhSIGXcWEkjUYqhdbgW2CsvEPhSXoZ+8MeWgm0FMjgKZYLVvMZF5cXnJ2fs1iusFWD9T4WlCuJNgYlVXRpch5XORplKXs5eb/HcDJCcUg/S1ifrngSrn6yJMF3h0bqXfrbD+mNJniVUzpFFgypNGgVMFpjkhypDAiFC9A4T900LFdrzs7POT3+iqo44+vXRCCEmtn8OYvlOVqlGJXSuDcbiTu7thJ+cgrvnx0wVkTAUHo4ncX5wIQIAm7DPM0Djy9OMd7Tf3Sf4WQASYYUAekcLGf45Qy/XhLq8gd+2uuH855FtWJerFmuC4a9mlwkJAlIqZBKxxW3kZjMYLIEnUiUhiSR5GkU2xuTonVK03iaumibSL08y2QmR6scIb7t8nizs6x1itIpUiYtUxyrhlGarD9gZ++QwXCA1ppiXVAWDd7Bpc5QwSAwKB1tZd5GjMfb3L37iMnODsN8zCd3/8B48Annl/OXpBQKeDBO+NWH++xuZwhbsPr/Ped3xZutfAM3kojL9nFB7P7TaY83mWN4eZrdnII7kNtV/HYSz03eq+so1L3XH5uO/1hsMSL6Tsf/WSwNlgpPQ/jBzG4BnK5jg5JcQBoi03DO7TINP9W0HiqH3W0YbUcwvNlkI7jI4mofgWwH1lTL9qpOCiFb7Y2PmlWXtcA4iextXUUZwXoF1Rks1lGbmu7B8H7U/CoDaY94FTVRm9ztQ7c8a2TUF2sTwbFrr0zXXomhtXIzstX7bOxnKuMmQvRt3nhb8PF5QoNsnTJu5+SCnEAYxvN63b+y4ObqGrTbJhDtChJfjU0gvAlKN0H0JtP76l3Ysudfi2+7Ozc/q3scNn7fBPBdbEg5fjRgLFFEdtgTO5/GpooCk2iSNLoVBeGpXR2vKwVpltPLMnqJRksoE0GqAz40VLaOQLiyaG3IB32ylmgxxuCcx9YNtWjA1egkzpNJnoEbIL1jdDnCNFPqnxU01ig9JhvcYbx7j8neHZIsRypFv5eyPewx7KXkiSTRghAcZbXmanrBarViOptyfnHBydER58ePsc2UP3Y9eF9R++qNJRQQh6M5N5nXH8ua7ZviZwWMJbGHdk5cYSwCpC6e1C5h9UOBMUAdLI+nF9zd2WZ/e4JMetHf2Jaoqoy+WMWa0Ly7r9IDha1ZVxVF1WCdQ+SQpipWi6YJOjPoLMHkKUmeoY1ESo+WAqMlidJonSFlQggVPoB1Ab8xaEoEg3SM0T2k+Lod3UgN6CUjTorj7w15lE4R0lxr9wKK0BrMpXmf7d09hqMRIQSMWVOVDc6CsIpQg1Epvd6g7Th4uyEQkTE+uM/B4QH9rM/h4QMm27tk0wXVhhA7l/DgwT1+9U/+nL07E6RfcnH8v3D08Yrz8P2viC6huiZev21tOyXQ49of4BrYdtMafJ2b6pjjzSmvmzJ73CR0NV+ftt8UagzlECEEqrX8sa6m9hUlNQUVNZYGj8XTNW75PueoJrqCNB72iff7Vrv/59yO13UOTCSc+p+g1FpnsDOGyYTr7nRNEyUUtrnJ7vsW7AhugLFpwSYyMsYVUCXRo1ipKLfwTTy5Kw/VAlmf4FdnYBdg12CGbUHchlOFbIVrtmMoN3IWrmVXQ6sRRrVFeRZE0jLCMgJ2iPugDRgRN9tKM2oXWerQSkZUtJlC3cYo34WC7W0Q42h/You4nyyJ03ZNXIbtEMFxN9u8CnI3weirwqfNEHwzQN4ExJvv1T3ePN5Xj/1VdnpTmvHq8zedS159z3cJS0QkXqQk4PHB44UHCSY1pLnGB4fz0e/eV6C1QoicNNX0spTMSBLpkcJR25qiqRHakFYWqQz94ZC83yfLcpLEYK2lKisIEmc9OtHoxJBkKdL1kM4xTMfkjaH+WRTiKYzZYjh+wHD3kNHuAaPJLoPhFkJpPNDLDFujnMkgY5QbUuW5uDjl9PSI4+Njjo6OOD2/YHY1pS7OCP7dORMFYo3NCDgk3m0zfhrj708eGHeJLkNk0CYqZutoa0ACEUB0pQ+3xfqsfMMfjo/ppykfjQbk+QBlNUE4lNIkUvD2XYw3QoI0miTv0R9OGEwmDPM+/TTe9DoxiEQjjURlApUHpAYlBEqKqC2SsnV4Cngf2rbW4aUiwoEcsrt1l8Fgm8S83PAhFZoP7z5iZ/eQ+W+nrJvvcxMJglCsipLL6Yyr2YzpfI5QitQ5dsoGHyTWBor1mtVyRV1btE7Y2tohNzlNXWFt03bdueUQAq1jJyClNELIFrBusi4xtJRkvQH5YMJwvEuiRzx8f59ff/Ulf1hGsPZ9pphuKrPcOKx0/o49ImjrtVunad6c9jrrtw5MV9wQbdfFeO3j7obfVEN2LPOb3jelL1FoEqnQOkEhMUERgkagMVgcvuWTaxosxfdcVtXAVXsuOoZBE3HUFJi/wYJkMywxC/WTZIx1bDpB3vr4egFWQCNacXkLdJx6We6qiJpfKVuFgIveit3AKXWUPUjia4NHXi7pVacs7QtgCmEQjaNHSSz4U6ptp1jAqoS0hLWGMo0f2AHyrt1z42JxYGh3TNjoRah0C3ZVZJDTVmYhQ9zHoFqiM4k6aSnjewsfNca3BeKEZvs3E/L9e9jZBC73EJeX+IsTyvlzCn9Kc212mBCvvLzdUm6WrZsSi28CzZ2EYVPWwCuPvw8w/iamd/PL3xRLvfq8zeff5iLj+4S4Xoz7dlekECgtSVJFmhmcD7HzOQGERWw0ponrLYnOUnIGjD2gDINhRd04hNAMBn16vT5aK6SUVKHCigYhRbRbNRKhJEEIgoySI6N79MiY/wQ4482x+kbsKJAqQadj+qM77N55n+H2PiLNkUlGmvXRSYKQiiAExsRjJ3i8t9S+Ybmcc3lxztXlBYv5gqoosc4T/K3pk147OnemXWCPG9Kz7VX1o8VPHhgPgbGAXgJ5IklzATKgC09awrKKJ7FTgsnvfrvvFS8WU7KvUkbvPeJwso03CQGLSnNSrUnkJrx4uyEkmCylPxwx2dlja3eXUZaSJxqlBVJHix8vHcJYvBQI6ZBCEK59QcGHQPAO613UYflwvfcaw53+Aw727jEabpOa3kv7MM4GfPjRB9y5f4+/e/Jb1pffAxiLqImcL5cYJTk9v+D84gKpDXkd9U111SBEycXFFYvZHNs0hCCYbO2Q791BEPDe8rbsQEIIsXijrinLknVRUJYV4ZXP01KidIJQBqkzsn6fe+99QHlyTvk3C+b2zVwsOqeKrj/8nDhQDIiZkm5q7Z4LN1rjDhh3xXidLZsmTuWmffxqiU03OXVTzpvEjAWahL5TpDIBoQhSIJxAIeiWkJEttti2998a+73unIZY5DgkSip6OYx6sKfhk1OY/wBU2xAB9o89GX5jyNBisBBBLUTg2PnzBRFlFM7f1FpJWhBJfFLrR4poPYBtuJE0GNVqgAMpSwbVGQVHOK5A3IN+gJ0ctsaRNV6v47aYw9JD30aJBj46SQgRwW1tW2/fLrMmW3PtOgJjrWORoE4gESA6MByIneg8yIxoHSdasB/idksDvUol/9d/MeIXv74T3X3mNfZ0zvKLFxz/l4Sjxw0n7jlXnLLGETBEBvkOcTrvxE6vxqugdhM0wzdLH7q/fxPj+yrruxmvSic2QXH3ft8kmNqcv941YwwuBJz3URkkQCqB1gKTSNJM4bxCNZLgo5+ulg4ZHN47XPBYIcAYEiUZ64S0N8DWDtt4hJAYk2C0wXsfC9NlQ+eEIpRAKHk9JzoPPkiUTukzQrOk+ZGXyb/SG0BVSFQ+QPW3yXbv0zt4xHD3HuOdfaTJmS2jxBIk1lskIeqqPbimpBIiOhq5gsuLMy5OT1m2c6xWijzvU3iHq0vefCZ4MyzUebtsCdgWcRxvl+U/2nj8kwbGObClYZQL8lyRZ5oskygl2JoIGhuYzirmS0vSPn9aRn/X2wiH5+nsnIPzEQc72+ymhkmWQ5ajlESJdzeYCCUwOiFJ2gK6NEMZhdQi2pu22cbQTogBF1lhKRFe4IJHyIAInhAEzkdNl5AC0Q7Sueyxt7VLlqY0VYNtXl6zWe9Ylyvmqyll8/0E90IqlFZUTcVitWi32IZYpynee4RUSKmRQqG0ia4ZSrMznrA7mqCkIHj3NaB6KxHA+0BVVsymM85Ozzg+PuXs6ooqhOvkoxGwMx7QG40xSYY2GWmWsnv4EPercy6OfsvxkeM5r8c+bnI8cANSNyURnWFUB3A3Gd7u982mHSU3nsVd05AeX2sifA2KO1eKN50GLlkjKFl4R69yyPbdGyoaqhYap+hWo+kwG0fy/b7LrtgwDREvKhExXV/Cwv+wu/FNXisQhLc9BsgQ2V7jIui8zsCH9suTkUVWobXviORBBJgtYPXdWOVAWUgCZA7y0JqEB9AWb1dUzRTPFTADVcU03UEG270WGEtYCxi6KL+oRbR8kwKSrC32a4HxdAmzVQTKXsT9y1Tcx0Dcp8TH4xOdPlnHC1eJWPinkvbLDi0L3n75txDelpw8+z29zCJDQNYBWQSsWZMdSPbrHfpTWFY5J82K81BREYhLNMFNfieB6+u6y9G0x/dSV8FXGeVXxUyvAtcOUL96vN3fN2NTWPXq3bypNX41rfBjAGNPHSrquo7uTj4K+pSUKCXRSqKUQiuDCCCDJE0SjBaIthGFbQRKSaRUGAMKhdMebwMCEe1IpcQ1FuFdXAa0+vxuvYgUUeccAo1zKGXoyxG5v6D5kdsI/frhPYKQWCEJOkWN9zGTfdRoHzXcRSQ9rHMU6yvOLqfMFssoaVM6elMDqREMUkUvlfRShcSyWqxwPqB0Qp4LtM5InUcJxXxaEfwV3fUjuClF/e7ROkWoAcFbCN+P7/XAc2AV4hq889z/MUmKnyQwFsBQwDhRDEcpiRFoFUhSwaAnGfYSev0BWifMFiXzeQHOIb3j7MsFs/L2TmkVGj69OGH/coewv03eG+KzPEoS3qEaRgpJIhOUiDZy3kdfQhkEGokWEkSEJIQAznc/gJiC9HikiGyEDyGmlIxCqTiI95Me460h3lsuz89YrV6uFZ1XKz7+9GOenXzFfLX4XvuvjcZkCi8slS0p6oKqrqJ/sQCjNVkWjcOd9WRZDyViy8o7e/sc7u1ipMRZG7vx3HoEnHOs1muWqzVPnzzj8ZMnnK7X1003xnnC9mTEgwcP2NrbJclztMkwSY/R9l3EewWPfnHG2dULyvL1JBWbmY5vSo5uwBkqbhwrOrlEZ+/WAeoOGBfcWLS1yqOvTc3d+3YD3pueVd/+d8WCNRURLoLf+FRBgiZHtrC5iZzGxqe+3qd3bhsNsRjPO6h1xIw9BOu3D1MBGDKgLwbIoLngguptTqLCEbu82FZf2357wUPSWqcpGQGyabVmMkTG1Tmo61aI3V5NSYiuD4N2Uz5KHkJDVa5o3IzQ9Ug0AfZSuJvBlokNP1Ye1h4q1/YzV9AYMAbRHyLTHCMkynnqqyX2ckmoWus2FeJ7BgfrKnrv+UBsQCHafW9dLby8+V366Iah/K0C42Ar/r//r/+Vv/6P/wHcEi0laTZgMNhme3jA1p/tsa0fcTdo9p5fcvT4iKPZOVfhCstzYrlsj5jTmRDzO4abIr0up7MZHVP7TbrgV8HxqwzzNwHlV4H2JtD1rzyPjX3r3Dh+jPDUrCjLgrqs8dYhfECFmGVSQiKVQUiFkYpEGpKWLFEEfF1TO4sxBm1MvIa8Q7aNbAQSiW/HVR+L50NAhIDwXdfYgBTR1997T13VCCEYpAPGRZ8lxY/KGb/30Z/gpcLqDJf20cNd5HCH0ktWteXi4ozjkzPOzi+Yz+eURYlOMtI8j6SYbUi0YJgbhv2U0bBHL0txjSPLRmjTx7lAYxuqqiLRBkmgXEiS5pycwBZR4uCJUrZnfLP7UtLfYjjaw1c1y/kpjb36XsfadWXtZoQfO3P3kwTGmYD9QU5vkJLmCYn0GGHJ0kA/k0yGCTvbQ3q9AYtlzWxYYq3DWcvwpIzdAG4xzssFv/3qK3SasLV/QKOSaBf6Dr8+JQRaaJRoV/ot6LXOI50kqJZJEuKaVQohPvaIFsIHRAuMnQ8EIZBKIXUsmkqNRhm4mp5zPj1junr54rZ4npwfIS8Fzn+/IcOkhn4/pTdIyYxhOOrTNAWj4YT9vV32dnbYGo8ZjbfIsz5NYzFSkqcJh/s7HOzuoIRoW1a+nfMuBDRVzWq95vz8gsurKS4EjBDsjgbs7+2yu7/LweEhg/EEneZIk6F0j3SwA3s1+48+4NHzS2aPSxb+uy3FuqlJbfxtM+m5eYa76bWTSHTgt+TlhGnBDWiGG0DcFacaNhy0uP0BKLQuyuGVIwitU8XNBN8ZxHUA4fUYq26BkBIxVlW3zT+ALTQe9wOms9c7I31G/KL/K8a9bWrnKee/o7K3a234UqgAWYjsruRGFiFbvbAhao6DbOURqj3NLZsru6YbcfEcdb0y0uwD2mK+OrYWXC/wYUZU+nkQJWRNZIfHPoLqdggS2pOrBlcW1H6BSHOyyYj+OGfUy8m0YjEdMLtasSgtvmnBuvFRUjFbw6KKDHSpYtpLtY1FfKuslF3hXetdbFx04BC3d+XayxPsZbfMhAWCizTndP899h/9ijt37nG4vUfe2+IwDWSPJcNLz3kzZ9VecTfXcJfL6ZzIOxC6qf34JmDcxbcxwa/+vvnz2wD45me9Cpa7ZbHf2N4lHAl41hTVmrqscLUluHhdCwfSB6SMxeCJlGRGo6VE4ME1sTbGSURwSALBebx17WGIeJ07TwgSQpRgSDyKEOftth26EBIhZfxrCChl6OV9RuWQ03BJ9SNC47sf/SlBaVzSx6UDyIY4k3Nyfsn84gVPnz7hyeefsLw6JrhIkVRiyCoZxOOzFrSk6GdUwz7eTmA8Jk1z+oN+7K7uHU1dkyQFidZkSYIdZPSXgdH6koPGc9CuaU+4kbO9fKWI2ANgZ5t6uaYpF98bGHfxUyi8g58oMDbaMJiMSA1o5Uh1IDOaXgp5ArnRDLOEYS9Feom3UDeOxjqSpFNT3l644Pji/DlZknL33gMeeHB0BSbvJiQKGTQyqJZ1k9cYGFTUHPo42IrQblIhhEKKyEm6doD03mGdj6yxkteMsQ+WxfqKx8+f8Hx+Qh2+7sViw5uV7WepZmtrwN37B0xGQ0ajjMX0kO3JDgcHd/jgg4949OABg+E4Dno+IIUg0ZLhIGfYTyF4bC3fCjAWQmCMQSpJ09RUdRXZBGC7P+DhB++ztb0dCzpGY3TWR5gMoTKCyvGqwush+fYdDt6/w92rJzy/8n/Ua3fzSDrAuvl4M/nZgdlrxpSbRhfdSnsTFMONvWGPiIGGsv33ljh81Z3izaLzg4nFSHECcm2JXSeE7UBC58G1eY++/gXV6amz9t0KItMQC/Iatokc3vc3UkyAbaKy7du/NUnKTvKQgwe/YDzepm4CR19ccTl9i8A4UYgtgxxHDa53LoIIF+Ip97RSiQg6hYlgLHgRO9opEcFwk0Rw3AcGApEYci2Rc0vdFNTzOayuiNxQAwhoprA8g/ICEwxSBirRgFuhF2ckl0dUl8eEy1PCaIhzJSEN6Mke2WRE6AkYaSgcy9IhhKeXBhSOcj2kWjb4qWurJxPQrXSi82qm1RanHpXETZsoC7udEMD7xKvoCd1dF6o162e/5/HRF3w1GDPaP2Bn95CD7QPGW/uMlvscXjScHS05ny9Y+CWWC2JdfU5Uwm8R77pNgLz5ua9qiMPGv/Ed/9797JbVHSjezP8k3DQh0Ruv78pyu5ZA3Siy5l2XOwUKinpJvS6wVU1obJTcNA5fx+YvQTgaY1HW4lWsWBAiyh+QqvVPd/jG4erYFlTL2CQqtHKCAJEdxqFFiLWhIRDa5bsQAq0VSZJA5pA+kKkemdVUP6I7xfYHf4bQCeR9gskpGsdsXTJbLPjs88/4/OPfUs6e8VJFS6jb1vAJiNg8S8oorzAmI80HDPrDa/KpKAuEkCgpImE1HCCaMXqdYpYvUMUxrqyoSiirth/PqyEMw8GIfp4RygL50yxh/l7x0wTGSUreH5CICukbUi3ppZo8k2QpZElCnib0s5S6CiRKxs6i4Qbk3XbY4Hh8ccrjsymPFmse1DXevbsL4BrsIpBCxJVuO7iKDWAsgkAEGcHzBjiOtjjyZTcKQLYVugA+NCzXU04W3wyKXyu+xRYkMYrRsMfB4Q53Dg/YngyoVhU723sc7t/lzuE9Dg/26fdHCKmvU+5SBIwOaO3x1iLw+O/JVr9uaK3b6yeglWQyGbN7OebRw4e8/9EvyXs5EEh6fWSStineyHQ5DFblmNEuW/feY/fogu3ZnAv/7dNNeGV7VcPludHUdt9Gx5huFtl1wPjbOCNNnCJ7Cno9KF20rRXuZb7pTSPlYZywUK0LRoOhpsBQUOIRBFKiUrv+lr18veiOv+amoLDipsgwb3+eEpmN1wtBBMX3iRfwEd98RgQZu0xGdxjt7jMcb+Es9E62YPo2+Pc2MsXWjiEZaYLz1E2grj0+gJIRKMRFsYiFuKrVnTtBve7kxSF+4d7BUMOOIVOKXe/xi5Kr1Zz64hKqKTcLgwD2Ci6+gukdxE6GTiW1awjNCrU4Izn9kvr5Yzh6Bls7NIMedneMEyPCQKLyhLSn6BUeu/Yo4RmlgUR6qsazKjzTgYXL1kRQp5BGyYbQAonHiEA/CwzzQJ4GssRze8O8BD4ggsMpcVHQRQBbYqcll/NTFosjwp/9hruHjxg/2GNQGPIvZkw+11xeNEztnDk1DSUv37Gdn0xnuPjqMvTVbMmrDO+rz918XieU6sBtNyKMuXH27wBz1T5v0W5dArv7/W31FP22aKjckqoosGVFaKL9n28svpZ40VZaNAJ0bFolW3clrTVSa7zwsaC3tjRljQwS2boydPyJlAIpJEr4tsN5e88g2v9Hq0mTGGgcwgRSlZFZ86O6U0ze+zVSG1Tew0vN5dWUeXXCYrXm+MULytlzvl7m3bnX90GqOD60ue3QNvuROkElKS60V5MQaKNRicYYgxIQyoywHkKRU62fsV6VrJYBOw/IhusMNIBOeoxHQ3pZSqVl7Or+M4+fJDDWSqFkIJGBRApSLTFKY3QSm1fkPZTpEYShqpfMFgvKBhovKeq3B1YX5Zy//Lu/YTIZsLc1ZDWfvbXPejWC9wTnW7Y0tKoJ0Xata7cQKWSxyRqHltkOMoJpAVJ6vIwaYyHF9Tjtg6OqS2r/A5iDbzn9wVmUFAwHPXZ3thhkGb5x7G7tsbd7yGg0wWgV015IpJAIAkJEFw1nHbauqary7S1IBCSpYTwZ8ei9B/z35b/k4aP77B7us3/vkHVZcXF5hQux4YqzFtdU2FpjraUOEEyGGW7RnwwY6zn9OnJI3xadNnhTA7wZFVGrvCROrwk3wLCbfr8JVHfRFeY5iIkFGTvrlu6GbV1zo1V+k3jQvw8+4K0jOIsMS0Tw9MmwZDRoLCkWgaPCUlGzxlO1V7LmRhW9Gd3ED126txtzu6LDjhHPiU4VgogBU74PMA7EM3FBVNB921QoSUkQdcVyeoqwBcILbHH5Ha/54aF6mkd7Cb1RireOynqqRuMAqbv7O4AXhCDwSEoPaxcohacpG+rZHH9+CssZuCEkE2SSkjlDc3GO/fIxPH0CbspLV4K/gM/+Fnp9mnSIu5sTigaKFXpxSnL1GH32BSyeAzV+Pme9rrgsLUXp4m5ZT+nBSYkNgaumRgob6yQUMGgBsUohzZCZJk8VQw0D39APliwJZAkkmURnCqlua/aVRCfVfeJd8HfE6+CV79MHmuMzvpj/Hzwf/Q15b0yejskY0NsZsD0ZcxAOmK09l8uK2aKgcovo7HHdQrrTIWfcsMgd+N3UEG92NhG8rFnedLPo2OAVcTF3RMyXOOCD2EmwqSGcEZ3AL4l3RTdy/BgSis3wNGFFXVX4pgFnwTlcY2kq8KEihCb2o9ESozVa34C3zrow2ADOInxcgsfCmiiN6OY+ZCzs01phjEZrh1YS02qYayGRISqTlRRoIXg7FNvrx+jgkCAkQShq67Ao1k2gcgqvcxAjCF2pdmdHs5F1kApPoKhK/ByQisY65vMFWZbjnKOpSoQIGCNJE4PUGqUNpAOQilpLnO6xlHNWlITQkCw9pfWEUCNEYLJ1yGg0JNOKLE1IzW11lPjx4icHjAUC1bY51crHAmYlUVKjdIpO+qi0DyqncYplUXM1XVA0gUYkrOq3lw4KBF5cfsV//e3f8ovDHaaXF2/ts7722T7KCzp5FERgHLUU0Y5CtKtDgYCWOY4McuvT2EoDhBR4FeLrN/LoHkfjKvxbuKCD8ygBvSxjazzG9/qIINjZ3mN3ex8tk2ibY5t4jDK0uxZ5UE9DVZUURYm1lr1b30MI3pEYxWQ8JEs1k8mQ1XpN2s/QWcqz4xOuliuausF5h3M1vikItcC6hsYHgknRgwm9yYRJdsS4Diz4dtDZTUvfBUodcerrtLUd57r5LX3bN9a9dk2Um1oPpY1/W7Zb1+PrTYHx/f19vPU0ZU1TFfimxjdrlErQOsPJjFqk1F7gbE3jampbUwWLI7TNTWY4Ljf2InLAsnVuDq2CWlFcP+oKC3tEdUC/7f/gbZTkfkvy4lti1b7jtxfRSSQZglCuWZy/wC0zEqloysvvfc6+TyR9xaOdhMHI4J2ksoHSRVmx13HSFx6CCzRNoGo8Sxe1mqZxBFGyXF+yeP4pnLyAeg/0HchHGHLs8Qn2yy9g/jm89B0ALGHxt/DxiLD/Hq63C84iqyVmfUYye4JZPEWEU0KRwnJNXVima8u8cNFMwnq8kzg0znuqxkKo2vW8QvQMcpCjsxyT5QxzzU4G28ozctD3MXMUV0EG0Uuj1dathACdA/fADolLrL8kguNXIoBfVRSrioILkAIz7rP78H3e++hPuX/3Q+6sJcvTNYuPXzB7csWFO+UCR4MitjGACMQ3BUybeuNumdzV5muiLOPlBeLNqNGVRT1ut+76DZCPwF2A+4wbwPzTChdKmrrCW0twMaPhbENdO7yv8L6KToRK4pKogVUC0KotpHMIPDL4ttguevbyiievbLMrxmiM9hhtMUpilEJLFW0lA21JsEBuTI0/FrxTvT7Oe5raUVQN81XB5WzNqrJ4kYDZgloSZ4Zu4dRVmTgEDm9rgq2x5ZKmWjGfTTFJijEJUgjAkySGXi+j1+vhfCAPeYsNenijsG7AKl2yqObYvKBvPJmLGnCTSA4OdhkNY+fAPE3IjOGduPW8xfjJAeMArMqay9kK2RckvWg3Zm1AOUGOonSSq3WN9JbLxZr5umRZOta2ZFa83XRQCI7Pn33Ov//3ik+/evJWP2szkiQhyzJ06xMa7RhFHKx9wIk4KEjZgmIp2/2FEGIVbiSUI4DGQQg+Vue2SDuE8Has0ACjFInWcSBSEudiAWG8fRw+WJwFLzwhhOieEWKhhAgN+IaiWLNarbDN2/mOQ3DRuUNBmiVsbY8YjHKk1jghMFrH6Sh4hAhoBYkOZMbjETQhQTIklZ7J3j77O59yvmg4C98Ot74PGO24ge8bM+KUOayihewytE0xiPxR5zL7pt98NzkpFaK9bhPwrkEgCdIilSdREqU1TguMkyS1IWs8NgRsiNfAkjXxTEUe2DCizxCuVWsFkgWaKeBouLGxG2Yw2dJIpcjLgJs3XNaB13fa9vBHnh0lKQ3CrnErj/UNWmeR6XqLoURgoAMD4/HSoyVIJygJ1MLG4krZehq4tpekCNFOsq5oLs6pvvwc/uav4Pz38GwHPr9DuXvIenxA/eQYf/pb4GO+DggDcAnzZ4jpBb1qhpYNqbxiLC7piykNS2Y02AD4BFyGr1L8OkGIgHAOrTSJ1lgFVZPgfXTI0UoySBXDTGMShUoD/cwzyWBiAuMgyYOmbhcDpZeUhcfdZrpWShBpC4z3gXu8rPjv7o5XFPw+0FwtOV7/jtnZCc8fPmFrfMgk32P8QcZO9pC759s8n654UdfMMXiW3LDCXfuejmHrlrEXwHG79YEPiXrlNXEpe0m8gztd8ax9vAl8j2Jx43WJ7k8zBCBCPNdCeoRsc0FBtllRFS1GW2KoS/3HH3GeEyqyvVqHdkCNMiNERwzRgl6JFgotJYkQpEKSK01PG6oQCHUNTYPyLjYPasWKP1b87//b/4ZznrIFxS+OTnn61THPvvic+fFn0HS1AJuzSPd7TWg6V6k4stdLQb1ss8ttplkgEEpFEiPJSJIck+RIZVCyYxoiYVU3NT54lDIkScJoPGA8HjCZDMizFFu2s9w/SineRgSWVcnJlUWTkidZZEa8g9qRWYGqPatyhavWXMyXLMuG6bLiau1YNW9f97toCv7jx7+l8O9mBS6FYDgY0B/0SZK0TReHFvCC3zDiEuJaaIEQrbbIB3zwBBFQqh1MAnjn8c7dUND8MID0bSGAzBhSbVBCXt9o3nmquqQs12iZtIOhwjuPEJE5FiGAbwiuZr1aspjPqX9Ig/ZvjYAPFu8bghAgPMbEogWkpPaRnbHe4wlIFZ2regkMUgFa45TCJwqZpzT7h8wP+1y+mPKk+i4e8u1HC234kqg37XrUT3mTIrWvR1Us6WSsHod1DVWoERYa59FeorIUZXQsaGzdVbQKeB+N/n3dUIdeq+jTSDJ6DJmIbaQweCGAihD6eO9puMLSlvsZGO702L6zT6INZbEimCkvjtesb+liVkCfgKFEeoWoA1IrlMwiS/UWQ+DRwWKCjedXCKQUBOeprMV6TyqjbaNzYF0svBMB3LpgeXxM/fHHcPJ/gv8ULjRcjnDpeyx2fombXRKa3xLrzr/phHmw5yTLC/abKXlSkqszcn2F1ksKWaMIWNEqvX0fqhzWCYhY9JRmmizXeCFZS4ezAk1DrmAv0+z2FTIJBGVJEscgkQwTGGpBisIV0Kw98zJwVTVUDuK3fysnuCXcBIQesXlHpwlOieC0U64/IwLWmzs6VI71i1OenpxzvDvm0W/+gg/vf8TBn3xEViZsfzJl99NLnpxccuJmlNfKeMdNv9auMG4KfAV8RmSCU1B59H0uT4mtD464Wc5+W3RM8utERmxW0vtjT7z1UKhWFuEROm7IOK9KKVBoooTKR5mgikA4dklsQZ1WIDxeCILtQLRHyEgEtfAvyiSEwghFgiQTip429LRhESCUFdQ1MngMgSRW7/xoPPv/83/+n3HeU9YNRVmyms+oiimhmfN6BgOvYqGN3GS4+RFszLI1FRTXVSebzikdmNZImZFmQ5LEMBz0Odzfp9eL/tK2LPA+km0/b774JwmMY2ONq7JGXDqMCGR5ijQGjCOtPB6HL9fU6wWrVUPtBMvScr7wt9bc44/F8h2BYoC9nR3u3bvHZDwhz7IILkOUnbSGyi1IDgQZrtFtxLuBgMD7OHAIfATMPrT+oTcnTLQT7ttIH2kp0VIhQ5RVRHsdS1NXERgriwgS7wTORaYgTzNSE9uteu9p6ob1ak1VvUlfue+OQGxgYl0Tz08IiNApd1tGvatjFnHQ1lpiTNy0EGgfCEIjFfRGI7b3J+xNpmydfP820W8aXXnPq2coEMHwvP23rtzmNvapWC+iTEfptnsgrYOoIwQLTYMwDqXi4kK2Yp0AkVEktO0RJPWGhZXGkJocrROCkPhgcF7Q1DO8v8K3x5tkgv7WhPGd98m0oJgdsVgtIrH0Aw5QsNF6GshxQEliRkx2tphsHzAa7XL6uOSzF09/2Id9537EgtsgwBIN8WwI1N5R2wbn/XWjA4RCtWBSeKh8g706h7OvwB9xXcoZCigXrF+sCWFJBH3fsf9+Tn9xyu7iOZmaYupjxOJLbHGMdev4jbopXD2Ho8dR/2ktItGxk1lIyPMUkUjyLGpmdbBkIrCTJWynCikDQViUh9RLdBA4ISmloJCwEpKl9yyqQGVv8Vx3WOAaGB8S9cBd8/EBkdndBXaI4PgZEXhuABTnqU+vePzf/pr5xSknDx6yO9gnGaYM38u4I3qo85KjakHBKfHKGrWfUxLZ4LP2fTvgXYL7PfgXG3+/DfKn0z1PiAuB+8QFwLsMQSp6ZGlCYjTaSJTuCkhFLLQTrSAq2GgtqmJWSMpYWK6Ujj7GyhOExWHx1uEJG71wWlgs4mWphGy9kgVGShIlkSEQnMXb5trrWCNJuG2Pq9ePj3/779vfXkdwd1vRjczf8FkBvCso1w1SQlGMsLahqQNNZVkt5hSrBbYqeJdWtm8jfpLAGOLXclk51PmKSd/RG/YQxmOKOBk0i4JmVeC8xwtDUQvm9ucs9/7mMMbwm9/8hoO9fYajIb08Q8nOsUFGkNxJjVsg4INHBIEMrd6Y0LLG4IKIVjVdpyEhW61RLE6Q4u2wXxIRm5A4T7CulXB4bFNTFWusaiBI6sqyXJYoqdnfP6DXyxBeg7dRn9i2ar71COCspW6alv0M4G00hjcdY9ktHiIIkVK1tlIqWmd518pTAibNGe8dsLt/xNZpRRLerE309wlNnOJ6RL7p1aVbV7BWEKfg2xrw1+UCIRQ6yVFCRe9QEkI0UkJ6h7INQmoC8Rx5G/Cuy3w4CBWyLc8LeEK08UdK2TJCAusV1huiJZxEE6vMTQ75eMDo8BGp9Eg/RWn/g2vsNREedeDYEwjU5KMe9z78kHvv/5KdwwfMByl/efSXuDd1cvljISVBKiySMkjWLrByULpA43yUAAkIMgKL1BhSocAHKmkRqymsz/h6Sn1NFT5rofYfm3QLRqsXbF30kM0RfvWU8uRzllcrlraV+PhTOP7r+F7Lc5i9jxwNSIY9EnqYXp806ZP2BhidoXzAiKgNTyRoPNLFJg1CCGwQVEHSKMnCSxZICgRNAPuKhvSHxeaskRHlFJ05ouTGFm3Eja3fB8TFxOP25/r6rZrzOScXf8fp7z7GDPrs3f8F79/7JemvDPsHW1x9ck6x/JL4fXRscacX/qZYQPh+DZW+OUbAAfGqHtMq82lLV3nXcECQMdRbDAYDsjwlSQzaKLRWaNN2QZWaqPuTaC1RKlqxiXb8VdpgTAI+yidEiItG8NHqWrSSC3XjchQL1lsWWcbxXIhWlug8wTm892gSkh9VZfyuXUJeJ2pCmLJe1pyeGfLckKdxfi6XM2aXZzTlnJQbf5SfY/xkgTHEkzqtA1I06NRT155l0VBWDruscOsKpMQHQWF/vJXd24xe3udPfv0n9Hs9CIEszdGqTUnL2LWnE0/IDiRvinxELL6LmotAED5Ooq5zufA3aZXg31JXOWJXIwTCB/A+6sqCx9maulJIaQlBUKwrptMFUmoG/T6Dfg8lImNsraWsKoq3AYwJWGtp6ib2QvA+5pdCQIVAUArXDpjex/PkQ8D6QOPj/tG2jlZSkGQ9hjuHbB1O2P7khH71doFxRuS5Psqhn0oup56zV57TAWO43cKSxpZIqRFOIxTXSbSAx9MgUEhbg5CE4NrCxdB2ZQyEYLEU2KiabV8LNbHIUXmPUCo6LgRFCMk1ME40ZAmkfU0+2iIRATvLUUrwQ+XyHXeiibDBAggYbg3Ze3if+7/8iP2HH/BkNUX+v9NYOPoWIgRB4yXeSZYNrB0UHiov8CGCt9AV3Oq2pW4AHTw9X6NtQeNjR8JXw72myEfS0F+e0j9N8OtnFLMnzC8WHJeRx2xVjFD8Hp4tYH4Ml89J9/YYHewyUNvk/S3yZJtcC4wQBCcJQSCco7YOrCWxFmTAG0mTSYpcsUoVa6tZexXdLW47TSuJVKKUEFIQOSDAv1ro1vmfbLV/e0DUIz/lRp/dLkcDhNpSX844qv6OspiRmRzRSNb2KTFfA+8uUb8F/HfAn4BoW3sLYtWm9dHJ4S01Tvq20AwZpVsMRgN6vR5pmkTLTC2RSkaiRqo4fyFQSsTHrT42tM05pFTRiUnF+Uw6QWi/thC6a2Xj2m8Bc5DE+VGKlliKhJFz0U1FkZCikTQ/W4D35tFJKOBGE9+FJ4QV08tjtBLkqUYFi1svKedXhKak1756+erbfo9P71T3P0b8pIExtOUFQaFbBqRYxzZXsm7Aebz1VM6zbt6Gl8KPH4P+gIf3H6CkYLVatf3iNUprlFIoqRAitCvjyP7GQaVlNEU3+cSLO2p2A95ZmqbBNjbqt4HaNpTV7fs2Rp1zrAyOnWpje04fAsE5rK0RwhK8oK4q6qqAIJjNrjBGkZrYBrSuKuq6pq5vfwkUAGst1jaxuNF7grOE4LAhLiaKsqSqKkLbPrQua8qypjA6MvEyoGglI/0BYfuQ/uEhk/Epk9PAlNtfQRticvdRKni03+PBg12SNOXsrz5nvnAvgfGuVKfHTV+IGT8cHCsJUgaUdK2FYDST8xRYagIO4dK201KDC/baWTXqBx0Fq7atcrc3FRUr1k2BDwKlTazQbgI+GDQZmiWZgZ4OpLKIXR2lRsgUJVpLpx9wcJYIdTqf6LR1VZlsjxnvbDPa3Wa0N6Y3GbST9dsJawOLKiB0YFFBGQS1EBHHyDQyrTLKqnwgWiM2DlmW5HXBUAeqrI+vt3lTHkfgUasrxFFCWF5QLdcc2W8SYFRgv4CLY+TsD4zPH3KneI+Juk+vdwcdDmFeYOWYtU8og6LRHq88WWPp1Q1GCmRusH3DcmxYDRJKHJXXWB/P8+15pYqY8UkUWB0px5ZRjGAxLuDb0sZ269JzOZGB7RHvwhMiSD5jk6bxq5KLTz+7rR3+niGJeaQ/BX4BcrvFOx3YDzf2LbfWNOX19sswotcfMxiO6PX7JGka/YllV5QdgW2nEBZBEoLEexE32zq86TYLeqOJuc6cEloZoQs4F7DOYb2/WVxJEQcwpaK9mRBYH/BBIkkx5KgfCRhn+T7W1dh6ybtpvmKQOkcnfbROkNIQgsc1DXU5x/tNK8uAa6acHlcoqVB4tLckvsYQrhtOvWl0/jAX/Di8+U8eGHvaBa2DsqhZry02ODJXYnyND9F+qnB/H2ExZFnKzs52lByUJYTYoMM2ljp4fGzjE5/cpiC7dH+3snY+UNcNjbWtl7GgLEsWiznnZ6cUZUyxrpqasFq+lQVG8AHZgmPVyptliOyxtxaEiL6mTUldldRVg7WW2fSKYb9PP89YLBbUdY2zb2cd6ZzFWYeQIjLCzrWLhuicUNcVTdMQnKOpG6o6AuMyiY0+hIyOG4lW+DCErX36h4/YuvMZ+xcrzt0395n/viGJgHgE3NHwYMvw3geH3H3vQ3YPDhFBcnyx4ui3z/nyJiFw7Y66RZzKu3Kf1Td9yPfZH+la1sW1gKUmUOOpcDQEJCrUBKew1Diaa2As2gm6oX7FJtBjqalCjbZpLMoMAu9lWxYzQMuCVDsyDZoS16yRqg8ywZicRLxu8dG3hyV+ZzmQS+gNevQGfdJejs4TdKZRqbhNpPa1cE3gahWzFMsqcupeR517InRkWAEbYiFjCBZVFMjFirxYs2MCYZixWu5Q+DnhW67C77K3E4Bq5ohTha+nVLjvcHwOCFb07Jdsn52yG86YiFN64j5cnlKpfdZiC0+PUqSsE0GVBNLaUhUNeWJIx33CVp+GHo1wWJ3gRAAvUZ37zm2FkrGNtm6BYlvDEbFwZOIj2Apt5s1z4xs7AbEN3ANRgv8Y+G9EicWPyTMK4gjxCPgNcA/0BEwKoYKw6VzeMhbvdAo1JHJMbziiPxyS93PSJEEphRBRtIQPeBm/CEkExcHHwlLvAk4EbBNopI+LYu9byWAsTPceCB7vPBaHtZ6msTTOYoPHiYBvMwVCS2SiQStCIwhSoVyCIUezakesdxsffvhrFoslR0df0dTnvL3rSYAc0RtsMxzFRYpsXa1sHW1SpxeBsljxMkAPBL/G+hu/fEnMqxh+GNvbB3baW+6Cd7Ms2IyfPDAOwMrWnF0tqAksrIuWISLEhpciFmpVfz9xMbaxLOfLeHFeTfHeY0zCcj5rB2mHDw7nXJQCWNtWhgZ8m+KvG0tRVFRVm6YOgdVqxWw65S//+q+5bBuVlN5R1q9vcPW9ok3VyUD0qW4HQKklUsXGBBCPxzUVZycnPH36FVeXU/b39rh35y7j8Yh+v0+vl7+VXez0riJoBKItaIIgBS74KFFpW4m60DEQ8ZwrHcGR0hKTJAQdLcbygw/Y//Uz3p//jvpJxZc+ppe+bRXccR5wMzcLorKxR2zpPDGSrVyzu5Wxdzhh+2Cf8f5dRlsHDIbbuMayc3CXe1+ccrRsrpPlXe+rMXBPQBoTDTzjjxmVfXc0tkRJg5KmncBEW0ceeQOBJnomKCSGDlZ0CngAg0PQtHrXeCYkBi1SEpORZnm08atrvE0xYYSRa7SaowTU5YrZ5ROyfBdjUobb+xzkZ5wvmx8ssappwbHXeN+jqAKXV1Oy0xOclpydHuHd2xNyNSvL8/MSnycsyyYW0WpJogS5hESDUQKlAeURWNRyBZdX9BdzDkVD1oNz5Tn1/hslPRlRORs2jnfzGpVAEkpUPcVRXnsqfFOkwB5RZDAMK8T55xSLF9gveoRshE22qNItbD6B3gj6CSE3FI2lLhtG4zHpowf0BodIv40JE4oQKJFYaXBGfM2n9s2jZQx1u/kuz04s3FChBcYSvIqgOFYQx586A6mhtuAKomvEc34cUNy6eqc70H8POf4IX05gmkKTgtEtKywgRN1+vAFvLDvfXeT0kjG94YCsl5OaBKVVdFtpJRCEyIhFxUfAa9V2fPV47/CuxvlAU9uYscKBtwTXRJlgiAsc6xuEDdTWUVYNZVVRNQ11C5C9BJ0asn6OtxZJzJ4qK1Ehjl4/RnzxyV9HEO9q3t6qJZYYCz1AmAyPoqotVVVRFQVNVWCbAtcseB14GitAYvyQO2AJ9NpDTvhuBf7biJ88MAaoCJw31csql9AO3AEk7u+tBqgoCo6PjqjKkrOzU+azGeW6xBhNU1fYpoqVoU1DURYURXktkWiaCNrKqmK1XLMuowzAe8+6WLNYr5iulvh3MCh67xEuAmMtouWOkAqpIjBuG/oRvKWpCo5ePOff/4f/yOOTU3YGA3756D3+yW9+w2/+/J8wHI3fyj6GdkGBJDLuAIgIjF3XJVDGSbFN1TkfcK1mWoe2XakxBBHT+dnee+z9uqDxCcr+N/pflbwIMem6qb+SxPTRBJhICEJgEWgp6SWGUS9hdzJke2fIYGvMcGvMaHeL0c4WJu+DShE6x+icpqgY7d7lzs4XjJYXm5b/1MBAwP0xmGVMRXZup296FTiaCB5CQLTFSgGDICDRKDK0SJAiIXjV/nvMbAgEQQhSL0lwVNetaRWShEQlpGlGnuV4PEIFXJ2gXB8txmi1RklLU62ZXT6hGTkm/S3y8R6Hk5zzZcMRP4xxCLRNUkJCZVMWi4qTk1OcMSyKJS+ePsH7t5fwC6XlxXmFy0qa0rYMlyJVgloJMg0qESgD0liMaOjNlsjLK9LFjMQXpLqiUQXTpv5GYDwBPgJ2RYbWPT5trviEcH2Ndh7Opk1Cdz3ZXo0e0XX3fST7GFZUzENFUVZQToEXcfEkNMJkJMMBbtLHD3qULjYyyA7vkB6sGAtHFhypA4NHyXhPeKUJt9cTOmqLdWcB1jWHaCUG15XNREreA6EBUYESkIgINusZ0UbtS95+me2rkQN3QX+IeP9POPinH7F15x7peJdPPr1i/VdfwemyzSy69kZvvWwF7d/CO9QYCyQj8nxMNhhECYUx1yxlZHs9IWy4J3mPd7HwDiTSgROBunZIWaNEwMiAFNFTvVUXxqJqbwneUTUNRd1Q1DVl01C5BustQQZ0mpAP+gQXi9IrW6JLgbTE2hheHR83G7SEV/71ds5jUf/QXN53RcIN/MvB5HhpqJynbiqW8xmunEH4Lhrn5RBEYGy4ke29aayJi3Pf7inwnYvx246fBTCG7z4hf19BMcDl9JJ/+2//LXVdMV/MSUzCYDBESoFtaqxtIjC2lqqxNLbBOd8WibXMZmgfd408foTjCK1ncmg3Wpsu0drqdPUvVVlycnLMx3/4A58fHVM6z+pqysXit1jnuHvvHoPB4O3sIx1r7GP6X8ZiDy+jtY/suii0v2utMYnBJElkOiC2r/YuFoXohGwwZuvue7H4sChI/R+QRzVrH2/+7todA+8JeG/L8OC9u8hsRO1TpElIeym9Qc5gNGAw7KPzFJ1lZIMe+bCHULol5BUIDcqRD0ds7e8yfH7JyYa1VUH8XCsjHsiJZlRdFfGbxO72LkImGDVAihSTJKRVSggWIQJKJRjVR8gk+kC3HqNIgQ2O2lpcCb3G4hDY6yYIgSAcXjY4UeNdQ2OXWLdGB49SCTLJSLIlWgHNlGKZUlWC9XxNEyw9ASb88FScBZY0TJdz3LOvmC6XPHv6hN5wwO++/Dv8D2mj/sfCBYoCcALKmJkIRtIYgWulsXUTQHvSLKANmMbTryvU/JL6xZdUz/+AL49Q38CfJ0R29wMhuDMekuRD6uMZ58FRtseeAgMCw2tleLxeXhCvp+4KGwATAeME+gIuyxvXX9/++x6eLWqyQY2+M8duG+pJRiOgCYHJATz4aI9sP+fIzSnPT2j8NgU7yN4+6WAfaW4rayRAqHgzdO3OaFFVBxyFbiuBavBzCC8gPIW6gHo3FrOFF0T/4VdLXt929IhyiV/A4BfkH37EBx8+Ytgf4qThSyOj5UciWwlIW2T4EqizLSh+VzOpQpC3QDjgXQS4RVG37qNR0hJZYxHt15wlMQapVFtLE6J8og0pAo0IKBlai+P2OUS3Idt4iqpmXZYUVU3R1BTrgqquot1hauiNhlHYJQKqXsNCgo162Wg+ebP/8bx3jVlu9jk+bnjzsrMfHh1ra0Rcy1V8U2fThnhEMhabSklQKnqkK8hGY3zep1qc45tzXgeSeiLxUnPT0PxNwxKLejv70R/SmfVN4mcDjP+hhnWWv/nk9z/7wkLfaoldY3HWRhN3eSOAFUiEgGK94tnTp3z8xRNKd3MrrKzl948f8y9OTtje2X4r+xiI84PwIfrrKonWsSDD4Vq2QiBErPw3xpCmKWmW4b2PGmXnaJoGqQRCGdK8x2TvkDxJSL0jM5rVf/otL45rLoiDiCKmnj8cS/7Jv/yIX//Lf4Me7FB6gxcK3Xp7GqORSmK9o/YeaRSkGo9oLfAii+1FIB30GO3vMcy+RC7rjX5IcBlgWkHp4zDeg7a/3JvFnfv3CUETfIb3BteUODtCCodWAiEVBENoO9gFAkIJhIR1WbBcrWlw5KFHbR2WkqhTtlgq6rBGWEtjS9bVFGtLjJRIowjpkLQnMMbh/Yrl8ozLyjOblayL6prFeF0fk+58dBq5rhVDZI0bLtwlV1dz3FRcV8Rf2duw0vqO6ACxl1DqyGwGHVl6HS2mymBxMhYqDaQk9YGRq3GLc06/+oTZ/BklHt0eXzfRBOK1976A97Y0d+7kaKl5cRoYuKj1WxHVqiNgS8S/jQSMQiw9+zzEIjxBBMNpD8zQE7DMTuBpuPHMvksE4YfbsPNLGH8IYrchTBpCBiERDHam7D08o1KK1Rfw1QvDojxk5u/T2/P0kzHprQFjIgCW4qYAQoRoWSDaTSURTNZXEC6A3wOfQNd7MaREIPQupu74DSoGuOs79xDUHdjeYWfUYz8VUC2ZLS1uPgNfgQ6RCe+OtSsW9a222LlWMvIuIkOSxsyDc9RNBMWLucRaj1JxvBNExrhpLLapyRKH1rEjm1TRds23JJAgGjxqJTA6Ng+KuDtgG0ddNayLkuVqzboqKeuKsigpy4rGOVSSkI+HeKDxFrVOwEgoA5ros17RfcOam6Yvmq+PnF2HwncfQ+ADDQejlJ3JABvgfFlyvqo5KS1Tv+n701Z6hCjV8FIShEAaw9ZwzKDf5+zFcy6e1+Ber16j61va1bP8kOjAtWl/f5cY6B+B8c8gfu6gGMDahnJdUKzXlOsCqeMkJHxs+S2EIniYz2c8ffqE09nXwcZ0tebx48cMBm/HiD4QOwTiRds1SbWemRIlBKo1lvfBt17Gsm2xLSHE0jHf2v0I6RAojFaYfo9UC0T5CGzNwckJu5fPuaxjR7ro+gt5Jhnv7rL78BFmvEcVNEGIrskTUsUixqIukVVNwEeAGQISQXABb2Or3bTXY7S7y9Ykxyxv0ueO+JknLYXXEAeBAW8+GDjpo8uJrVoZT4m1BQKHkjEl6r3AeRF9x0Ms9gnCsSoLFuWS0pbUoaFpQTEELGecNQWX1iCFwocGFwoCnsT1cXWPWeGZFYIBDdpY1suSZ6dLTq8svnS4dt7XvB6DkRLBXsJNywXfvjZHMJIJPiiK4NqFUM0Pd0z+I9F4WDhIHVQ+grdaYI1gqT1KeBrn8DIujoy1NOUKFa6Q4gTNCTmehOjQ2xGjnng9HEj4oA9bE4viHLsWmODZaj/eEsHsnRz290Fm4JPYEmIe4P4avihAGLi3DQdj2E4hFB63gOUqTpgJ0BNw9w784p/DzkcwfgRqAqJNW4g0oHsXZMPfczr9itVJzfHHmgv/S9YJJNkh2gdSZW7p5LauFEpGaYQmguKY/mkdKWyrLS5igZ4tuHGdqHin0gmRkWb32cnvU9QZq1JiwwCfphjRsLueM7gQ1F4g1h5RFEADsgbfgG2iDaV3N64bLsQL4Z1NND2USEGEKAEsCpbLjCACjXUkicYYRausoKlrmrqCALn1WB/QApBRxuZsdFuygHdxrLGS6HQUaJtQWIqqpChLyrKgLCvKusI6i9QKYxJkP9aTlK5GLVJEEscdExQZFkH81t2NZwbX4PL6cedL/e6jK7fc1oKDYcad/S2USTisGqZFzcm85GRRcrZaM60biut6yxVh9YK6miPSPmo8Ic9Ttre3WM2vuBSvf0Td4kFwOwVzXTFf4N26U/wjMP7HeCdRWctqsWA5W7CaLdCZRCRAEzW8SmlCEFxdXvL0+TGF+zr7UnvHZ19+iZJvym1+d4RWchIA4WMRiBAyth0VAq2i64Rr2aTIMIdrf+PYErltvx0cSrgW1MYubn5ri6a+z+6jh9x5csr0ecOKm9V1UXqqpqEREqkVTiiEjF2gtBLXE4XTHi9bv03vkSJ23kME6mAJEtJeymBnm93DLUZHM85u7FW5AJ77OOB0Z7JjSd8kTq9OcFZQlpKyDNTNitqt8aEiYGPLZ0SU9Vw38IgTSff4m6Oh4YrmVQkfYKnwTU1y1SBDzf7YsT2C1czy/GTB2cYomhMLy6747sG1a+gxou0QzA1r7IA7UvNw/wChBhS1oGk83jZ8un7MpX+LrHHtYFpDWsXflQLjQEEjLY1oPWhVYF15wrpmJq5owhFp7wWD7RWmhHEO4wH08rip9iBTDz0LogjMztdMZ1D72OdtT0Bu4HAC7z+Cg4eQb0MyBjJwCVw5OG8gGBj2YzFgaODiOGBOwH8Sz+c28HAHfv0/wZ/+DzC8C/kuiJR48bU5a+ssZXnC/PyUrz4NfP7fBGXfow7uMr5boaQmSW6pHTREQGzardOOhhBBpKuhWcfCOtXAOIPpMLL4tw6ANktvv+WeMCX5/cDBwxGqHlBNFYu5ZLqqSVfnTF6s0DbDZX2UzJGNB9EAa7AzKGdQzaIk5Dqd3nX3u63FxndFrKZIROxoWjcVi9WaJDWx86i1ZFlClpnWtxiauqGuGqRsveQ78ptYm2K9j1k+aL3OBUIEvO9khQ5nLWVdU9masq4pyoqmqRFSYLKULM1IjMELWFYFup+jsgQlE1Jn8FQouuyTo74WKHTu8F0OJuGb1fdvP7oxq6wDdVnh6oYsz9jqjRjuGnac5mEdOLu85PjsnE9PLrh0bVmbX0G9IjQZLpHIUJMZCa4h+NfvG9DVsdxWpMRxY0Fslv6PGuN/jL9X0TjLerVitViwmC/QdQTGXsYMnlQKguDs7Iyz2TenoQLw9PiYpnxTNexrhLiZ7FqVa9Q/y9YfWikIYIxGaxNtyiAWkkl5naIMBIK3BBH1yoKAzjLyyQ6Tw7sc3h9yfn7JSRVZSUcc0KqqxjqHDhH8ShmZLKHj5wsByiuklfjgYqemENntSKK0jHGWMphM2L2zz53Pv2J6Za9B4ZKo+9wiTocdz/Gmy43Pjx7jAjSO6AFKjX/rFJSnZMlR4SkbsFX0GS4XUDQvf3Kb8CbnBhh3EKSDH53EoFsgdA1zu2poCRz+/9n773hJsvSuE/4+55yISHtN+Wo3PdPTM6ORRxIg2EVaISEtCO+EMBr2s2KBd3kXELuAQDDIAAvvIjwIswghzCKxErBCIKTVyI+k0fie9qa6qstenzdNRBzz/nFOZOa9fau7zL1VdbvzV5+ozBsZGebJEyd+5zm/53n6GY8+chaVn2RSG2wdsFXF+oUBG8MjJMa1g+0Sikl8rxQYk6b9LYhr8h9CWVHVY2zvOnn3CqsP3cC836IegTOn4PQp6Peh241KTxxMrgcGL8H2q5EQWw3LpyLJ7azAymlYPQMnz8DKGeidjuQ464BqQ6WSXl7FWY2yhBs3YM1BvQROQe7hdA6PfBY8+ivg4S+BYgWyLlMe4T1YB1sbsHYDLrwQePlZWHsh4E+ukRdbTCYl1gfcYZKPhhgbiZ2RqOhRDRbcEKrrUK5DW0OnBcoewRNaEf3yHaJBXoOp1n4O3jGpN9mSGyx1alqqjYhQlxYZO4aXAtfWhEm7y0bRYTL0sFXDaAD1FrhtCDvMkjRq4nAQIkE+ahTkLNHK2uhMYZ1jNB7TGsa+VKViGyLE8tAK6jrG0GQuzQAl/bdoBS4G74akJ44JNmIQtXUO66I+2TnLuKoYlyVlXWG9AxGyIgb3NgVGqmBpjXYpum3ybpu81cIP2wg1BQEHGAxDfMrNMhNYRFTcL2Ic0tkMPOwMK3aHI/Juh04vp9NbYjnvYsk4sdzjRL/NeDxme3N3b1MOE+qda2xcbWMIbK+9Bv7+lk5r+uN7adUFMV7gnsA6z2QyZjgcsrOzg4wDXjmcBFwKWnA+8NrlK+xObu7X2yxLBteuHck5KqVjyVGJVZdizmhLUDFMEFIZ7lzR6XTodju0UkS1T5kplIrkOYRAVVYxxZiABCE4h1aKXq/H6TOrnOlvslSG2aSsC9TjEb6a4OuS4B0+RM+xEpNiZtS0+p6z0cMiPpJw8dHbFgJkeU6/3+f0mTM8cq7Hle0tbqT+2xLlFAVRKeeIU9132v1tHWn09BvBMwGuWvCb0DeJIBM9DA1qYiGTplCHIWrxDJEeTIjfKYidb0kk0vMEWWs4dbLNqTOn0a2zVD7HWcFXlle2rsDwlaO7TOdgUENVQhUL0KB0JMY6eay8jdPlbozoHbonXuPsw5d59LFtwhOBzMJyH/o9aOVCYbrUO5rxmmW4PWJ7HBhU0DoDy0vQeUdc2ieg1YdWb7bkPTBdYhYMExM65CHOyAcP9SZsDuHVa3BjK87Y9xU8/Ag8/CT0z4PpR+mFn0ulOy5hOIQLF+BTn4AP/wK88Cz4MbBVUQ132R6P2ZpM8JOSQ3t86TAbGaomhVkAVceo/Oo18C/BWMcTd1c53EGfIQbQfT5INx4/PJ4C+m4Q4/OT9skGJq9e4+Xrm+hWl1Z/FehS+oxQBzZ3RpgwxoqnFIcNdST4U+9wY/CMODQ+QSxOcpJ7Q4z7dE3Ur7ZbLZQW6qqiLEvanSJJ0WKVUx9ikLBzFkdMq+YVoBWSGbQ2MXwsyd/wgeBJAb4upjqzcYlZmyaMJmWUuhlNnrXo9rp0u91pP17WFZ1+l06/S7vXptPrIFVFVgsqFRrpEdjBcp0J1YGe/fsjpXDEWTEAM/JkGwNc0eJEq0O/7cmAPFOcWOrR0nDt6jUubO+yve8SfD3gxqWnWb/yIt7e38LOE6KneH8G5aPGghgvcE9gQ2A8njAeR52xVw4vljp4ajwhBOrasba+Sf0GxVqOUmukRGIVH61TAYGAd9FD50XFKkwqao2LVkGr3aYoMvJMY71DnEeSu8P7WATEWpvSlqW0PyJ0um1OnT7F6ZMXWV2vGCWt18RCNRpjxyN8OcEbgwSDDQYJHpGY0si5KKNoFlwgOJAQzzcAeZaTZYYTJ07w0EOnOPHKNmvjWSndpDzck1rnflQYOgwEYkDhje3oFe4SqcQ80R8SO7ucSIpPpb830udNCI0jdsaa2eRyDWQZLPcNS/0eWW8Vq7qEoAmlY/nVU8jlRnd4BPAWdksoNdQV0xy7ihhUJRZcCWoCso3prLNUXOb06as8dnpAkUGRQSuDPANDhnbL7FzyjIZjSoShC1Q5rL4Dzr0bTn4OnHh/lEyoPCZmQMcEDpKlRA2NZy9A5uOpTYYwHMPla/DiBbi+GU83z+HUo3DiUchXogQjNKktQiTI29twYw2eexY+8otx2V5LNhg72B6wszNgbWeXujVMv/QhYJ4UTxM2hBgcTAV+HbgUvci7bfCHmeu9C7wX+GxovwuKInquXRW1zJN1qC8DF4hFQ4ZxdmBU4kYlw8EWtFIZ6xqGTXqtEN6AnzVFzpeIivpTICugDlGeciByNKfod5bp9nq0O0WUprmYWSkG0oUpOfapXKr3Do+PMjWBoBTKGHSWoSXmLw42yom8dfgQCzJZ76lShdeqKhmXJeOqRERoFQWtXofe8hL9fp88yxClaJdjuv0enX6PTr/LeKmLqj12kJF5ReYVk1CRMWHCgHUerBigxumhPeTbJao9QHd66FaXjo6VBXutjH62xLkTy5y4eJXt8oDpj2Bj4a03gRJDCHaPDeZzntztxIonOjXu9bNpQYzfxjjR7lKYjM3RLqWzR3qD1wTGZUyNo4zGZIagMjIBJx7rPEoqMnV/m6RSkiKfo8LUp87WBpe8GB6dqgcarcgyQ54ZsESi6gMSoje3qmtsXRNc9OoarTESMMawvLTEqZNtzpmKSR07gLGH3cEuuzubqF4P1ekCAesdzlZMSo2IRC9IHTutWEI1oDRg4wPB+TjQiIQkZ2VlidW2Ro/tdNStSQQxg1EdSeRx7gxq4GoZfWA50Q+2xszL0KQSEmae4YyZZ7gkDhZKIiHuAssKOhoGNk7x+9GYarCOzk6Q95ZAtXHiMVmzx6MixjVsj2PO3LqauVklSSoaMbRx0B6xFDY5JZssqx06uiIzabM4CYIEgapmtL7LtZfHbG0FspPQewjOvhfOvAe6j0J2NgbaSfouSWkQJManTdNYWbAVbK7Ba5fgmWfgI78AH/sobCQXlgN8FmUXOxNobUUFSKggOKhLuHgJnnsRPv4J+MhH4Pp8nQzn4No16kuXufLqVbYnGTGU8DAgiRQ3xDhEuYq0wPVBdcEZoAR3J5IZzSyDgWZvcrsngMchOwN5kUoTA5LHWQF0JL12FfwZYv6PASmzNtTjuLxp3pXmjj85W9QqqJV4nZLNMlUcCQRYpaNPsdRfotdr02nn5CYGL0ceP7t/JN1O3oVEiAMOjw0eiwcj6MJgVKyOF6yPOYi1AyVRyOVtLBudZiUtnto7TJaRd9qx4l6qYqm1QQTyTptOv0tvuUd/uUc17JMHTW3aUHmk8hS1ZskJte/hqNnGP1ApYxvPcacKFGsDjMnAQzUu8f0eWbdN2xh6RrOaaV4t3e0TWN2i1eqx2ulSb99gWI1QRCFQQWzClhjPcrehqfdDyHGcn4UL3AUyUXzp5/8yzp09x0svvcRzL7/I5d2tIyPHFhiWJVVd4wlo0WijCVowKqAqh1cBlbSy9wMhpGQ8RqGVxtmA9xbvLZXz2CqmYzNGpW0DWsWUbtbamDfaO0IIWOuo6wpb1di6IjhHbjIKrdDa0O/3OXWyx9nONoPt2JGNAwx3xuxurVMsr9Ay0S1nk6QxBrGFpI+NxT+M0eggKfOdj8d3DufT+WWG/tISq92cfGNGjDNiyq0zy7C7Hcnxce8M1omP35PEtGF5iNNwI2YEudHhpQye04T0u8zybwoxhdnSUgxYcxuwW4PdHTPavEJWnCLvnkEbg8KhjT7aC/MVDIbxB3J1Sq0VUs7qdnTj+gBZQFZLVhiyHEYUrkRbAWXwwWNTsgVxCj2u2blacuWFwHgAy6fh5GPRU7z6bpA+uDa4lOK3cVDDXMrbFKNWlTAewNXX4DOfgl/6JfilX4DLryWvMFAHGLlIim9sgM2ASZRJ+BrKETz9LHzkY/CJT8P163Myiwa763DxCuPzVxlPcg4HMkvLNhd7F9O0tcD3QS1FY7xh8XRDbHnQZFWJO+sB74DssVglT0w0mveR8GZLYDrp+ALi4+cIaBN1LHkfwlkI74ze6moA9TbYF4BneX1qsPlBmmJaAIQnoXgE8hOQ9RPxVmBtylZxlPQuR3GO5c5J+ktL9LoFrbZGEeLISFKchsSZO0kV8JpKo56Yk7/2Dhs8QQsqNzGZiIKgPTiPS8RYE4MnvUhUWYhEchw8Rgl5p027n0q7F7EUNUDebtHud+kt9Vla6lMPlymlRWlK7KiiHpcYp9AOZLREXnsusc3aA0WNYx+3DrRGDnN9E3EBV5aoekKXFfrdDl0jrLYy8t3qltNZAkjWZfWxd3Om32M100wuw/aVixjvWdXQTk28dDFG+Cr3vuTN3eK4PwsXuEN44pSV0Zput8NSu8O13a0j0/F4YHM05KWXXqKsJzFdm0AwMSuFt4FyUvPKxSv3vC56g6qqmYwnKKXJMkPwjuAdtfdM6opxOWY8GeNcze7ugN1BQSYOb3PKqmRSVvj0cHHOxWIrdZySCt7jRKiDil7KoqDTbbHUhv52fORWwHhUsb2xRvvEaYpeH42PqUZ98hz7kCQfCpemSwMxLXRIWrq6rrHORVlHgDwr6LUz2kSS2DyuuxKn1lsaevX9Chk5PNTEmfleDif60J/AyiRmTdgiXnuTXKnRFTfSigGz7BO7wEhitoS8H2PexjswntSsXX+ekGfobk5bnUYrjVE7HKGQAhjD1mbMijB9AAtNMReUiV5OLYSwwq4Xro4VL6y3KP0a3WyXlh6TicUERzYxsKtYfzVn68oWLlS0T0cpxRZgK8FP4kDMZFBoyHUMbsxkJjvxiU9trMG11+CZz8DHPg4f/yRcvjIjxQC7FTz7IlQGXrgO7R7YHfC76SHk4KUL8NSLURJzYDFOX8FOBYMQ874dGpqDyexlSo5zMMtQnyXepWP2blQQW9BDwONpXVNHUgFdyM9A/0Ry2+sU0RnS/rO4bpoabu48mqp7SgANKgM6UeRd9mBo40CJNWZ+tQ6zBFchHl9WQJ2B1jnonIxEXBXTinI0Gt0j1QScoKNP019ept3v0erltFoGfMxtr0yGaI2k0tySKhuKT0RZFErrmCDNx5m82P01ep5Y+MMohTKKIAFVV0gZq5hqozFGx6JMmSHPM7TWOO+p6oqMDG0MJjN0Ol2WlpYYrZ4gVJZJPmFcTBjt7BIU1KMJk3LMwO6wwy6jB4wUN9ghycdGllZrSFEYbLdAhUCeGbrtDiv9Hq2NEeNbrXoohu7ph3n0sXewkmd0XU053KTYymm5CaeWMrpFhvUwLh2yVWJtLAT0IElO3gz3hRgf4aTjArcIHwLXrl+nVRSMhiMEIUPFaaojwqa1PP3Sy1y88lqcEgsh5dGP3gHr4Prw/o0tJ2XJcDSCpEFTKkYT1bamnER99Gg4otSK7e0dNgqNtyVlK6d2ltrZPQ3b+ZhOLYQYmOeci4EiRAlE0SrodoSeBFohErtybNleX2P53A6r/hxaJBHgAKl6ICkwMPg0xRhAuYCva8qypKqqVCEq5g3WWUa3nbEksB3iY3wVaDX1C3T0KR13YgyR2Pb6cP5hWClhdQeWb8CGjWW4N9I2Q2akuCB6VxrlaA0MQ5w9152YOcFUMJzA6PoOIX+a9rImzx8mb3Uw6qiVhpPoLRU7F3hnYnCYmoDRMUeaykCdYTM8wou752lffZTN3Q3OZjusmgEtdin8GNm1uG3LjVdPs7s+IuTbtKttdFWxu2VQrRzXA9+taRdb9AtH38QAup5OaX+TqmMygutX4dlnIyH+6MfhpVeiE3LP7xLghUs5lzdaZJ8cI6qmHkauW6goq90pI9e7OXysSjPOoDxkPWxy0s6q3aVFGSiWYfJo1I9wnWnyOXUSzCnITkbybPrx9wlN6q6UgDwzkOlZE5nmSW4O3vQbcwS9Kbbhk0eVdG5KR+mDSmUUJnl0u6s4M4BuRf1LlsdFZylKMo/yDDExgs2lQDzv0v6jnvdo0EfzMMv9U/SWl6OntltQtAzO1jApUXmOZAYxGpXFJQRBvKAkZlwxxtDEfTjnsEmTDClrkFJIk1RNQE+iXZRRmKDJsow8z8izHGNMzOZRVzhbEWjTMrGaabvVot9folw9iXKBcTFmVAwRDbWvGIwtO+UOa2GLddxteVvvJQJxoNv1sDwp6VcV3nlECUVe0O12WV5aom3W2axufuOJ7iOmjXclebfHQ489zkPnztHyjmy0S6vXpb3SpReEs6ttuq2cygrDsSP4beqtknV/55VV7wfuCzFekOL7jwBcuHqZ0e4uk6piZzykPCJSrIGe1iwZQ0vH4LHK1pTWYWOg9bTC2P3yFgP42lFNSowognVkJubRdC6mRWsq9+GgnlSU45IxAeoyTvcFAIkOnvgWJWAyE8tLo2IAfIj6apW3KLqKQju0jY/SURnY3thgd2ebqqzQ7fgAcM4TvI9FuVwMSEF8TFXkY4Uo76IOuq5qfFMko6whKLI8p6ug72CFGHaTmZiO1adp8iMWBNwTlMAk1S6A6KRbziG3M0Lc6Ikzom+txQHXLoCJzrXWaiTR6zdgfQCjy9vk3U8jXGVpuY+vXzviq7LACEIOQYM3s3PEpiz4GfTa0OpQZ102vebSKCNjlTwfkWVDKgZoP6IaWka7nqt14Kry1GHExnAbc22MLQW3BqFtUd2S1f5Fzi5d4WRrzKqGlQyW8jjLMBrB1ja8+Dw89RQ89Qy8eil6kQ9C7R3buxPYvdO73MF4Fy5vgFu+w30cgOkDSZh31kayrKG9DNXDUPbBPxIJZnYC2qsxTUfRSd7c5LmX+cC3kOQRSf7i0+ckwXdIDPlADzkxANDPsWhJ2gGdR09wnmo0apOWRISLPEY8plmjWKUmxBEKzOQcKhX5UOGIkg+0gIfp5mdZXl2hu7JEa6lLq9Miyw1SVjgvBGMgy5DcoIsMU2Rp8O9iGjcjmMzMlXi22DrVQ4v5KkGrRJCFEHKyLCczGeJjIGVhMkKrTatoYZROVfUqgndopQlZjhYhNzmddpfl5ROoAHm2m7zLFltP2N02OCzbDzApbuCIUoYw9Ez8Nj5At1XQLdqMy4qAoOUNXCKS0119hHavy3gypNUuWOn1aBmDntS4ukaFQKfVoivQaRW0C4PgqbQn03HQe9ycLgspxdsYG5Mhm5PhXufFIUID71g+wdmVFZb7XfLcEILD2orxZMhoPGI0HjMclmxN7n8nE5zHlTWlj2m4fBYLeniV4nGSlldSwHqoPZWUiIseWqWilxaiRk6UkOWadqegKLIYbGRjGJirM7xpoXsZWdsRBnFwMKphe2vA7vY24/EE1ariACL4lLItCjt9qrRHE7gSJKZxcw5ra6oqRnoHOyG4gMoMLQVdF72kPYlOpAkxIOpe16I/KlTAjSEs3YheS19D18CpXsyYsB7itU6InXVTWLiRljaOw4JIqk0b2u2YLGBzHV4Ctncgv7CN8juIVdQjf8SD/cCskEA6Q3Ex9DybxOocKwpOTWA5Q9oFdQY7ocWO19Sqj1M1YzWm8iXbmWKz0Fzr5lxZzRlMLPX2iHI8YaJqrC7R2YiiGPDIqfO88/xrnO+/xInsKicKx2oHWiba4foafPpp+MSn4IUXogf55miELHcKD2495nNbP8Qh9NRRm9zEDbENSe7QWQJTzPTdosDkkYgGiaTXNeGd8y0htSjVRJLN6YebY+k5z/BUfjHXI897kiVl1VaNfCBVRhGJs0iNGLyRX1ifjhtm5DgwE4w3cYDN66GzlzbwMG15iBPLJ+ivLtFd7dFa7pG3YoYEJ5pQe4JRYDJUlmGKgqKdR2IcAlorTKYQCYTg0+C/oi7LlNs99bchIBKzCpEH8rygyAuwHh9ijEfWMWRZjkERaks9LrHWkpucUDi01ojStIo2fnk1VjgVjQRPsBXUEwZrLbTo+xIUdieogEsBtkaOncubcawWhGFlGYzGhAN1SwCarHOGM+cfotUqGO5uoZXEKqLjMeVgB7e1QT6Z0FEGo1tAhrMhyRJHjEcVY3d/HV53ggUxfhvjqAhxAyOK8ydPcerk6jRgzdoSvKCMIcsz8tpRG0cujizcX5F+dMB5rK8I1qGcQXJDMAoVQkwbGyRld5I48+hD9MSEqJWG6JRRaQpPTKxKVxgdi5gosLak1gXkHXS3wHQmqEH8LSYeBtslu4NtyvGErBP1wi4QC32k8tOEGAktEqOgRNQex5NzjqqqEO9QXqGznNwI7TrQIzkX+6B7hlwHutrj6uM/lxOAGzV016ayW4puLEbRHkPHzZSiI6K2uKGdDT/IiLxhNIaNAQwVXBvBlZS940aAixvQMQGjHbu7Rz0L1rCauTy0DSGTEnQVIw1bY8gDbVVhvML6QOlg4jRjDQ5FGQo2dc5a1mK932L7dIvNHdipagaDEjsZQTkADMY46lNLsFky6GzRDWssZY7VHnSKWOp5Ywueew5efhkG20dqhHTtOzB+LVZxORTs7wXnhX7JE5m3IGsxnQIKxHvQ+UiWrZ2t27OfRFJD4zKL0qwZAU+Bf/NkXMLMy6uYia2nnyWyPpVO6NnfDbFtPNHez5aGEE911DJLUxfSdd2qzvSWsIxwjjZnOdE6yeryMv1+l06vQ6vbJssLRGlUFQiqisF0WY5pFRSdFu1ue+ph10owmYopMOtYGtq7WM2OYFBKEQgEFX9LEUEpTZ5lFHmOr2q8qglkqEwQhKossbZmOBxR13XMaOGhKHK0jlVNs6JF23tsXWOrimo0YpxFKUauMtQxEoUGYl/3Yh2oLm+yO7GI1gxrPxdzmbTskiGqhW71Of3Qo5w7d54iNwzbOeBY6nYptGFoLZPxhCwE2p0u7cyTZYCv0vNnwqgKDLh3FesOCwtifMc4PjfF/UIZPB9/5UXalwxKolYslsqIKc0IAZVm8VqiOS2K676+byNxozRG6VTe2cXKBSEgKchNELSKgW8xI4ShlQlFJjEjRcoG4Z1PUlCFeMtEB5Q4WqZDoXJ03kIVXbJ2D9VuI61tjMR8sBYYjjzj4Yi6nBCsQ4jeEFfVMZeyCCrlM/JAUILoeCvH4JKcurbYusmenCNZgTZCTqCdQ38Vlk53WD55Bi0KKvcGnoPjhW3gNTfLaVxMIneoiJ7gFrPMFNvMJDxN8Y+CmEXh4nV4bQdu2Ei2B6l3t8B1B8vrcaZ97QiL3kU0hKYhxynqjRCrsFkf86VVFcVkg1Mjx5J42sqD0uyqgsxliNI4DLXrQNbGrBhaSmgvKexA4QYVI7awdhNl18mqdcqNTa6Wm1yvL1JulYiNFfG6qfQzwGtXYHBYNmhGJzedvqiIRS8O0wc11+7nuOMUPhHNhkM33uSUmYY0ixM1yPNajOZtyucxnw2t8ezOH3P+uDL3RkL6fO47zXk3MohGtzyPZnvZLxGRZF9JJ+UP2PedQoA+GQ/T4SQ9vcRyr0O/26LbLugUBa28QJkMUl73AIhSZEVGu9Om1++z3O9EeZh1iAREYrGlqUO8mSnzMU9PkEjwnItaYwIorcmKHFvXuDpm6wkhUJYTBoNddnd3GY1G1FXNdm+Lrc1NOt0unU6bopWT5zmiDTov0HmBBcZVjUdot3r0yozxsfEbR1TASzZw5foOBdBWgtMd+t0zhFYfU3TJijZ5ntNuFZw+fYqzZ86S55p6pYdSgeVen1xHLdVY3aAoupxcbbGSeQo7wo4GKAFXe4Z2VlzpOGFBjG8bzRyUcH8y7B0vDL1jWL1+vNg8B3IiUWkrQ2FyTO2owv25jWKGAYVtIrWJEdHxOTj7W0msgqRFUh5jha0t3vqYKs1afKp4551ClEPhydoZ0mqhdUbIW5i8g847mLZQmEArZYaoKijHE6pygq9rREfvRG1rnIt6OKVjCi6XNHjBBJTE2kwmRWA3UdwigsnbFB1DZ1zRXYLeas7y6bOsnnuUdrtPbnKUeiuojCN1XCOS4opYK2EUIn8MxE6voQBl2kYxyzKriN7kl22UtI54PecYAmsVyBpsHvk8YUNgGt928hwHlTTHAaoA4xK9s4Fmg1CVVJVju6VReYsdnZOrAq0KxqrLSDp4I5gTnk47g7yFUhZd32BSXgd3A1XdYFzvUg9GjLeHjNeik1qpmHls9QSsLsH26BA9QquJCK5xgN+h+XUm7K1teBgIewmqEMX3DRMOfkaQp19J71XyCgfmvpwwT2ibz6dl59+EjDYfNVKM6bnJ7PgyR9pl3/HnZRX7pRJNomDmvnMoxDhHWKZDj54UtHNDK1fkRsiVkIkiE5Wy7MR8785ZCDH1ZKvVotvp0Ov1CD7ErD7eEoLF2hjzAZH0Ts82xAxL3vppfydJzmaMJstyXG6xFdi6Zrg75MqVK6zduMFwOKQsS1qpLHS316Xf79NfXmJpeZlWkVFXFZULTCrHcFxSu4AxBR0pkFAdO/dYIPZpI2DHB071lnjXuz+HlXOPYYoWxuQIAa2g1+2ystwjyzWEGmMU3VYHnGNw4zqgyAvD0uoqy8YhI2FUjXHOMy49O+Hw79R7gQUxvi0Is0TtlgUxvnM0jpdJWra9RaqjzInx5tDJCxxC9GpnxmCMpg7RC9wE4UkI1DamRXNW4jSg0mR59Mx4H3B1ybgukQoQjxJNpiyZihXybBUIQZMVHbr9jJVeRbUZn8UmAyUBV46pqxEq74A2MQglSPREK8HXjrIqY1CeUsmLLIQQvdeCgNJIULR6y5w+f4Jusc5qP6O3skRn9TTdE+foL52m21tNhSreGnDMyiCsERM4pHIJU1mlY9YOG7pZp88a3+wboQY2J/ei44+/IyGl9RITNa5FAe2cqEEawUbBaGB5+foYnXsoHCavKIylZWJ6vpYSvMrwWU62VFCsFuSmQ97q0ek7zGiN7eF1hpPrTOoNJIBRBRYh6Al4hzcB2wK9rOieNehVUKuejQ1HteVn2cruBD6PMzVqMgsUa2xAHzgD7XMxj95hYb7qnW/kDWHqTJ1KHqTpneYIMXMkdc+5Nm/nXLVBJe/u/hMI00wzMRvF3PenXmZ5PbGdJpiWOT7ctOhEfDXJk82MQDfp4polkLzIh1HgoyRwhR0mlOEk/ckqaseRtRVZ25C1c3SmqQOU1jHcHbG7M8SomClJaU2W57RaLYJzWK2xNdjaYbTCmCIVV9Ioo6MzQATnYtCx2BrnXRzkOxdTuGlNnuXYScV4OOTatWs8/+xzvPLKK2xsrzMsdxGlMNqQmYw8z1nqrXD69FlOnjzFUr9Hy2gGG1sMRyW7wzFbu0OGx5AU70cQQ3v1NI++6wkeeed7yPMCrU10BAWXbKcxRtCK6HTJMurxBG3yaQEsnAMVA9ArG9gaVlwdBq5wPFnSghjfFjIOrQzpAnuwX+l3P6C1niPGUZagtcG6OnokvItlSh3T6nPOanymp98VYhBcVZeUVYUPDqU1WufkuiaTGnE1rrQ4L2R5m95Sl9WTFcFGD2YoIFPgqxJbTshMloL7BJGoqWtSsdVVFTVyIXpKtFapbLSjST8lKNr9Vc4+8jj16jJtI3Q7Bd2VM7RXztA7+QgrJx4iLzr3+Re4eySVHClM6cBwL82s4t3+Trv53q0cp8mbfPRBoynfrc9jGhGTQ6cD7Q60upEYVUPYzaGusa7EioAKlHrC0FSgKkRKlKoQHZBc6D3U5/RjK6ysLNPJPFnmGGfbDPQm1m4RxrsEaVMVJyBvw5KKxswcui+0Ti+x9NAySxLoTmryjS0uX72BfW0DNuetMhW/spfdHXDH1wVIK7qk3ZhZkqcWyGlYPg+PPgK9pcMzr6hZ9KVK5ygwzXYwTTOj9hbBUOnapJFShAOVFJEUJ+Y6f8nTjBQy0x7P64CnX5fXrZp6jqcBd825z+mjmw2bkofezxp3c74hpKA8xeH1wJbAGhN2cWEHN5zgbli88qDjjFzpPOOyZjyZMB5X5Hms6KlEMCbmGQ5OI1KB17ha4oxeFgtDxYA7lYKdiSk1y0hUnfcYbdBeUD6glCIzGucsg8GAK69d5pnnnuaFq88wCbuEqSxnZmRBU7zcY7lzhodOPczZk2eQqsSOhmzu7LJWbbNzLCnfXohu0T91nkfe8U6efPJJ8izOHHpvcc7ibI2rK5QSitygszhrs8sAlKa2lsnEM94dUuQQypLtUcm17YqLLsoojiMWxPiWoYje4ihAX+CthyLPaLdbUb/mfcqLCUqiN8GYDK0NSohVz5JUIQSPoGMHnGfJcRNieh9no3QiKMrSIm5MqEtcOaKaWJQuaHeXWFoZ4m2FFQi5odcpUNQoX5MbhWnlWBtwLj5AhaizU0qjdcznGZ+r8UEpomIeZtEgms7SCq1WgdRjxJUYBZ2Vs2TdVVR7BYoVQnbviLFQkNGlYou7VaAJURe8IvHurEPM33kzwtoU+LibieOKmBPZci8irrMYKUk7Jg/Oc2i1I1lVeeQzdQbjHCYdqPvMbJry8sbaYTgJ0fNpYHM7MFiryJbGFC0FHsY7jnpL4zdVdLe385iSrL+KXulilAEceQbS6jDy3UjAVU2tCrz4VHQkxJQgtED1AD3zTCLJak0Y5BwxmUgMKut0QfdgvAWhBn0Wls7Dw+eRR86heofoMSbsJYrz6dYaT/CUnM6RfM+MQL9Onz/vGW7eJ4IssFdwnMh4YK+HufnsAK78OkzP7QByu0f+0Szz1+yPwDMRgDE1Ezb8Dba3X+HK7hmWrz/G0vIZSueiXjcoEIOIZTw6g0u5iX2IKSmttYQQcxhroyhaBcbopGqJudyds3jnKcvoLa5rG/trYoB0qC12UrGxscHFS5d4/oXneen6s4zD1gHn3LyzTNwWk8EW1wcvkV0oMGgyFM5X1JQcf38xrJ48z6PveAenz55laXlpqusGRQgKGwLWOVSALBhMiMVXRIRJVbO2NWB7vMXm1Ut0lENsxfZwzEujihscP21xgwUxvmUUadHEjvzekeO+MiiJlc5c8PjgqVPg1QKHAyFKKaI3VuFwVHWNdTVuWp40dcbEV+8dVeVRWFzmyUJAUGgVA+Dyoo3UcXK+qhxuMmLXlbhqjC0HUA3RVlBZm6zVodWzBG3QrQ55kePqEluNk95LYUPMaRwDTgJ1XccHSDMb6mlOMgYCOhd5iBJarQ7LJ06QKbDDHVxdojrLONVm4g2qBOPuRYsSWixzojhLv7/CKxvPUPo78ysoYj7mczmc7sFqP5YnvrYOk/qNPbl3ywPuNvHY7aEFnRVQ7XiBxsQ8tTrlz3A+uq8rDXWLKDlovLMqkdHkJ2+KRTjgmmDXAlaXjE3yVjqBup1KII+IldR6qKUVlldW6OY5voolzp3SbI5iv1RazfZQ43dN/H6eUpl1l6G7EotOOBPrnu/a6OEOa8ThxZB4ASoKwZ1O9bhz2OlAVUdP8ZmH0OfOsnzuNK1O7/DMu4cgNh7jObK7J0huzju8R9fLzGO8nyxPsz00coq0/3lH+tSbfJDW96CWeqvDujnSPd1Nk7fNz94f2ZRdAGocm4zcFqOty1zbOkEq8gx0EVml08kYDh7HWp+qhjpsVVOVJUoJWZZRtHLanXZM82Yd1ro4a1d7rI3EuK5rtKrQotAyI8bVeML169d5+aWXee7VZxi6rdu4Akvl7bHyDwtzFSrZk89mity0eM8T7+Zd73onp06epNVqMRoOmZRjnPcE75iUJZPRCK2SPUUwGIL3jMYT1gbbDAZXeQE/jYSwxLv5OGNBjG8ZE+JEd9PU7g0tFeD08gnaWZ5G0TGP46is2K1KSm9xeHzq2Y7/GPb+IAA/+gs/z3MXLlA7l0o7h5RtSfABdoYjtgYDBOi227QLE+WQjYRB65TCR/DBY22N842HVwhOwAe8s3hXI75CE6Olq6qmdgGkikFQaojJLpG3P02r20dnBbX1scNK+T2jvKMhys1VxAdms01IU6kmy2gVOUYJti7x1iKmQEyBNm0y00NUztf87q86UjsXdDnffpiTp07T6nW4sp1T3uGtVAAngTMtWO5GZUHtoAyHU2WpmV2///NDGrIiFnUIIRZ4UHpu2r0hvo1IpM3MnWnSugkz/3ZDilJJYRegssyuOCNatx2/Lz5KMcwIlVu08uAc3sNuCaNxyWg0ph6MoHLRw3yij+ouIZ0+0u4TpMC5LEZBbtewtQs7S1Avg4xAl7HudDtHrSzTPn0KlRcMBwN8WcPSaWT1PMXqCfJOF120OTRoFY/t85TlQeYyTTSkMUQ7NYG5N/UkN2hsPLcPR9Qwu0bfm753EwXEbFcHSCuA2e8lc17h/V7jeRlGIuR7tmmu4148zwIwwO9R5bcJQZiMJlSloy49w1FNUYypqwn1eEKeG2i3yEJTzEjhXUhkODAeOYYDy+5OxaQsYz5jAlpAp2w+k+GQi69e5rkXXuD67msc0SjggUAXOCVwpoB2rgnaMHKwNrHs1J46BIII7zz3EJ/95Lt5/NGHOX36BO1eL3npa8RanI3yvCAxw1KWFxRFEXNGI5S1ZVKPj7Ra7v3CghjfMpre6942AkFYXl6m2+7Mze4FauuYTCxlVVN7m1KF1ThXU7uKKgUhuDQtFZKH+cgcA28BXNvc5Nrm5i1tu717lCFXzcCrhq0Jx0updZPp3ISeXuL82fN0V3oE5RG5/dbY5IXJiFYajiPX2BnBxhBes4ej+20opiMOie9fjm2ZkTUFcRrfMS0PrSSWHM4zsBm4PG6DSVfQI4o/6rmlYqaQbkg0zKybEttlAEN8ZRnubiLO0DKaTGm8D1SVZ3d7B3t9IxrIdGDlFL1H3sWpc48guo0jp7SaSaUoJ2CXPG51Qtg+BaNNMCW0akxX0+q36C936a+sgNHc2N5mZ1QinVWy3klUscJIYrGcQ7NtpqGTQ9FUFJSZVzcQA4tskw+4kVuk7TSzILz9pZ6Vj4SzqXhXB6h8zME8SgOSKbFNv2uTAWPu9KZlqpsVzTokBQumc5npJPZtm/Y9x9WnspZAXCme+/NkSMkSvcLWisnYs7M9jg6gakxtJ3SKAh8EEY0QnRF17Skry3hUMRqWbG1WbG1UjEbjmN/YO4zEtKC2LBnt7vDSSxd55cZL2GNVnPj2sAK8S8PjJw2PPdpn9eRJiu5JLDkb6yM2t4aMJhV1EB5+3+fxxLsf56FzZ1g9dYKs3U/djFBVFVWKkbG2ot0qWF5eotNuU9VVDP720VHzVsSCGD/gUKJodzp0ul0CTKf6vSd5EFMVtODxDTGuJ5TVBFvWuNrirE1TT57a1lTBUoUwlWMsiPICdwOVEqDFLMkBzxuUPxPQmca6isl4hPW3N0HZTNdBfKRuE1P6mjpyhU0PO3d2Ga+DTcc7LeAzuFDdJ82cqEh6dTMUkFjYQWsQk1JseLAOfCvWxA6Np7iO66iZ0fuSvQmDmyuFGaFOiRSFmCN5YqmMUGpNpnKMzmIiA3F4vwt2CKqAfofi9GnOPPQwZ84+xrgUBiNL5aM2UWeC9DSS1dStAqolVMvRbnu6XUOvl1O0MnQrow6BTHKKjiPrnaDon8JJm9rnuHBYqQVLePWnYfOVZOs5otkQSpcq2zVaXNjrqFeJFDeyFQ/T8pjTKnoeah+nNOo6asX9HDF+HamdJ7fzHuO0fZMbmbnzvBkxnt4x6fybtHM+Zh6IRLx5EvyluzHmHcADW4xGG6zfWOPK5VV2dwe0OxmEOMPR73WxVrCVUI7B6AxbB6rKMRpNGA5LNjdLNtZLBoMRk9EQW5ZIsARvqSdjRoNtXnj1FXbd+j2+vnsHA5wBHulp3vnICd79xDnOnD9Pb/VhlOkzHJQMByXjsqJynt6jT7D62CP0Tp2g0+8R8hZVVVHXNuaCRlIqPU+73aK/tESR59gdl2JdYsD6WxELYvygQ5iW+4x9nEwLPBjR6NQJB/EQDCFk1NZQ1FnMp2s9wXqCCzjrsbWlqitsXVO5mto5ah9LDjsCLkT118K7vMCt4on+e7HOMypLttwm5RsQ44Hd4tLlSwRqxnaHsd29rWMF9ga7lcRAO0V0ejVCgcOAJZLsM204dRKuvwbDfcx4XwHfo4FKWSiyLthWJEomBlWCRMIV1IyQaQWuBDcB23gxGxfrXFDe1OdumKsXPLcuj6UYdxXkhtDJCJLhxeBFIZmQKWidUkzyLsr06aw8xNLyQ7S6K0xq4fragPXr29hK40MO0izEVILdHp2eptfTtFoKUyjGtmSwtsvY1tRaI+02vZWTrJx9BEebcaWx/lY1tm8GC1sfgq3D2t+bYT/xvRO80bnu3++t6JXvNzbZHX2az3ymxWQ8oNvt0Gm3aLUKWu2CEydWmAwDu11LqxijVYb3gnMwmdSMxyUbG2PW10esr2+ztbHB7s429WRMXY6w9ZhxucPV0Sscf/XrzTGNJDCaVtGh3Vqm01ml11+laK+wvCRpjBfV3erkw2TnzqD6PUQrbGjIrkqZlDStVociL+h0WnS6PQTSbHVJkWesLJ2gazQtEyirMYPxiHIq7Tu+WBDjBxzee66ur9FKU/dKYg5bghCaqTvVOBU8IjFhunWW4N1Uy9bElrjgsBLwRuFVTAOWeY9O2lUf/HSKxBHJ8mGSjQXeevjc938Ou6MRly5fYXdjh/INGkvFmNeGr+IpcdR7VPF7YpFuA0ep+HcABlZWoXcNhnMO7hYx+G/AEadskywKqItuyheqIDNplBBA6qjtzaJ3BwXUEot+xHJgxEcm7KXxjSBFsTecsPEYF9ELnbJThLpFbTPGVVQWiBaCCFk7g3yZvFih1z9Hq7VK6TTbGyPWL29QXVyDkEPRh7wDuY/nn8cywFkqA2xywSvPcFyzuTnGV2PoL6E7LVRnmdbKKRxt3EShDjUVyHFzAdzOuR6H6woErnJt7RfZHVwhy7oUeZd+d5WV5ROcPXea3YdKlvp9ijwnMwWIJniNtZ6qtKxvDrh+Y4crl29w+coldnavUbsdXNglMCYwJLBxvy/0SKGATCDPM/KsQ573KPI+rbxHt7tElrUwWY4oDcZQ905RLZ+gbrWplOBdvP8lBaFrY8iyDGM07XaLVrtDXZXUlaWclLSKgofOnaVfnOeh08vYyYjLr73GlevXWd/dZVjXydF2/LAgxg84PIGXrly836exwAI3xbmHH2ZjY51LVy9ThTeQUQAQqPaVxFDM8goLMzXsgyDzCcDWGDZugJ8jYwVRz7csMdjvaImxgawVSaW3TPPXOg++gtpCWcG4BFuSSnxFT7IFQlPsOmOWVWc+Vr15nSfMBVPCWAJbGnRGOYGSOpJx7dNuc8hajPWEweYa+CFhrAlDCBtD2J5A7wS0T0KrFbNqKMBbXOnZHQfQHl3GQJ/dkcVXkvTSGc4aBiMw2zWOLHqM3WEUo1jgwYHHhysMJtdgEutQXl3votVJll55jEceeicnT52k2+lSFG20LtAqR8Tgg7C+vsGVK1e4cOl5BuNPA9dIdS85vknDbh0CLAPLmaLT6ZHlHYQCbxW2rLBViclbSJ6jshzJClyrjZiCIAYfkic5xSSJjqlHM63J8oyiyFE6psmzNsY09Xo9Hn7kYc6fWuHJdz2CchXXX3uN61evsHZjjbWNdQbDXQajMZujEcPa3YPUloeDBTFeYIEF7gp5kYMSRvUu9g5C1NrESOpmEr/x391OUsQmZKyJ0w/MyPWeAP/bPjvYquHijagqIB2nTfK1GpCjnlJpKt1leUxl1sAn8ltamFQwmURS7GpwNulim2LXOZHhNmVN5kugzPvck1YW0veSRnYIIDHVWm2jTENbyDx029AzhDDGjcawG6LmpPYx/RsZZCuQt1J+YqJ+tCqhKqlHlm1qlFYEBFfZmNqNlCmi0owGHmdKHApbG8KhVGlb4MFD0xYtgQnWb7AxeI2dFy7RuXSabmeZVquLMS2yrI0xLbTK2Nze4MbaawzHTwMXeTuQ4XkUwAlguZ3R6XTJsjaEPI6PJxVVMcF0ehitUFksAU/WwusMLwrvJRaw8j5WIFQKoxVZltHKc7Iszi6HANbGCrDdTpfi/HmeeOfDfM5nv5eWgs1HHmbj+lW2NtbY2txgdzhgd7DLtRs3uHx1jWevrTGoH3x6vCDGCyywwF3he//jP2dSjxmWO9zJxNkwLXeDN5JT3C1nnQAX5/pyR9Q1bxFLTR/5Izj4KJVQMdp+mh2htjHNWpoCRScSrAVClgKsUkq2WsCX6Woa7/G8h7gZjswPLZrFxX1NSPvy0QimjsfWRSTu3sCug4GP2RcIxEd2B7J2JPZ5BoUGatAVVFXkvwQQhVI5khuCa6rMFVArwk7FpN6Bepxqggjw3qO2/AL3HXFUZt0z7AyfY2fYDNympQpjKsxpQfcHn3QdNoSYubyvIG8Z0BoHVD5gvcel8s742H94QLQmaJ2qB84GmbF6qo6JHpUiywwmSSqyLKPKMkLw1HVF8BajoMgM3XZBv1VgvKWVZywt9ThxYpXxeMh4POKRRx7mia0dWp/4NL/wwstU/n7PBb4xFsR4gQUWuCus7V6536dw33BP/FLeRZlEyMHaVNrXR6+wTRkTlEqShuTlVSoSZVvDZJzIdcaMFMfqVgd7jJvcH026/uS7dyljhdPxXEIdH7a2gKoDXsWsC67RK6dHdrYMnSUwGeQG6eaINnhdgq7BCBiFyXOyrEvloKaIxF8UVBIlIoMd2PWwbRPxXuDthYOHvw82xTp6dImyrk4u6NzgVKAOntJbKu9mcRzegY9xHUFpgkrEWAniYm5ipWJOfq1UlFFoHT3HRtNptahaJRCoypK6KjG+AlshrsaYFp1+H6U1RSun2+9SVxW2LgnWEawlzwuubGzx4tqDrfdeEOMFFlhggQcaPqVjSynZnJt5jl3y5pJ0x6Rcx1onLa9Ecqwaock88W1kGc3fYW6beWFK2ib4pHVpCleQcvdm0Vvs1ZxUudln1B9j8pRdRxCj0Rp8pqPnV0l8OCuNMgYtitqEuD9CKorhokxk18JOCeX9L7uywAL3G8KsJq/RoFJA7DTDVKrUOstlHQcWQeJ2Mc2mTJMGNhmvtFJonUhz87cxGK2RAN46vHOE4GKQf/AxU1ZmKFotQrCIBPIsw7kCI0IGnD1zhhP9Pi+vbTzQYhcJr6vxvsACCyywwAILLLDAAm8/LCIYFlhggQUWWGCBBRZYgAUxXmCBBRZYYIEFFlhgAWBBjBdYYIEFFlhggQUWWABYEOMFFlhggQUWWGCBBRYAFsR4gQUWWGCBBRZYYIEFgCMixiLyQRF5y6a7EJF3isj3i8iWiAxF5MdF5IsPcf9vdfv9ZRH5ERFZF5EgIh94g22/UUSeEZFSRJ4VkT98h8dc2DRu9w0i8u9E5ELa7rvv8rhve7uKyHkR+Ssi8pHUJ9wQkR8TkV9zh8d829s0bffPRORpEdkRkV0R+YSI/DER0QdtfwvHXdj19d/5VSLi0/a3nb51YdPpdh9Kn+9f/vgdHndh19m2qyLyN0Xk1cQDLt3tc+vNcFQe438CfOkR7fu+QkROAj8NfA7wPwFflz76cRH5rEM6zFvWfgl/jFhV9/95o41E5BuB7wL+HfA1wPcBf19E/sgdHHNh04jfBzwB/Fdg5xCOu7ArfBHwu4F/D/xO4APEOnEfEpGvvYNjLmwa0Qb+DtGmvw34UeBvAX/jDo+7sOscRCQj9q/X7uKYC5vO8EmiLeaXf3OHx13YlUiKiXzrK4E/D3wV8KeAwVGe3CKP8W1CRP488EHgvSGEF9O6LvAS8BMhhN91H0/vWEBEVAjBi8i7geeBPxhC+O592xjgMvDDIYRvmFv/fwK/CTgfQqjv4Wk/0LgVm85vl95fAn40hPCBe3qyxwi32FZXgN0Qgp1bZ4CngGshhDvyHL9Vcatt9Sbf/dfA14YQ+kd5jscRt2tXEflm4OuJA7pvBrL5NrzAbfWrHwJMCOG/uceneCxxG3b9h8B/D3xuCOEwHDm3hHsmpUju8m8XkW9K07gjEfkhETmTln8rItsiclFE/vS+754Wke8SkefS9y6KyL8SkYcPOPbvkTj1PhGRT4nIb0rTHB86YJ//UEReS+75Z0TkD93C5f1K4PmGFAOEEIbATwFfeyfTUQdcw1vZfjTE7E3wpcBp4Hv3rf8XwEngtjqghU1vb7tbxcKuEELY2k8o0t8fB1533m+GhU3fEOvEGtW3jYVd9xznCaIH7o8Cd+xgWNj0aLCw69Th+AeAf3IvSTHc+5LQvx/4NPFmPAv8TeB7gD7ww8A/Ik6b/VUR+VQI4T+l750gTk3+WeAG8BDwTcDPiMj7QggTABH5KuBfAv8B+JNEYvU3gRbwXHMSIrJEdM+3id7fl4GvBv6BiBQhhL/zBtfggOqA9WXa3xPAs7doj9vFW8F+t4rPTq+f3rf+qfT6fuDHD+E4byeb3ku8re0qIjlxcPfJQ9zt286mIiLE+tQ94NcC3wD8tcPaf8Lbzq7APwS+L4TwkyLyFYe43wZvR5t+oYhsAx3gaeBvhRD+6SHuH95edv2itP9rIvL9wG8g8q8fBf5ECOHlQzjGwQghHPpCNFTYty4QDWvm1v2NtP7Pz60zwHXgn73B/jXwaPrub51b/7PERiNz674obfehuXXfQmwkT+7b7z8G1ubP8YBj/zVgBJycW6eI0wEB+NKF/W5uv33bvzvt+wMHfPbN6bPWvvUmrf+WhU1vz6YHbHsJ+O5FWz1cu6bt/zLggf92YdM7tynwtWmbkOz5lxdt9e7sSowz2ADOzNvlVve/sOmBn38r8I3AlwG/mRgXs+d6F3a9PbsS47cCMRbm/yLqi78euJCW/p3Y9laWe52u7b+GvVOOz6TX/9KsSJ+/QPzRphCRPyIxKnmXOJX2avrovelzDXwx8O9Csmra3y8RRzPz+Brg54GXRcQ0SzqPk0Rv5M3wD4lE+HtE5AkROQ/8beCd6fOjnHp5K9jvQcPCpkeDt61dReTrgT8DfFsI4acOcddvR5v+FPAlxOCbvwr8KRH5jkPcP7yN7CoiJ4hk6ptDCNfvdn9vgLeNTdOx/0II4R+HEH4ihPDvQwi/HfhB4M+JSO8wjpHwdrJrw09fAr4uhPBfQwj/CvhdwGPEAd6R4F5LKTb3/V29wfpW84eI/DEi+fwbwP+atlfAh+e2OwVkxJHSfuyPuj1DHKncTFt18mYXEEJ4SUR+L/D3iI0P4KPAdxKjJa/c7LuHgGNvv9tAc02r7LXpifS6cQjHmD9Og7eyTe8l3pZ2FZHfCHw38E9DCH/xMPfN29CmIYRt4CPpzx8TkQr4FhH5+yGE1w7pMG8nu347sT/9txKDRmF2rssiMgkxZuZu8Xay6c3wr4HfAnwu8HOHtM+3k13X0+uP7SPqPy8iO8AXHsIxDsS9JsZ3iq8jGuebmhUi8s5926wRf6QzB3z/LLPREUSDXwf+l5sc7w01wiGEfyciPwi8B6hCCC+KyD8ALoYQXn2j794nPFD2u0U0WuLPZi8xbkainzmEY9wNjqNNjwOOrV1F5NcSUwr+ADGV44OCY2vTA/AR4gP9ncBhEeM7xXG06/uBz2NGOuaxRsxQ8VsO4Th3iuNo0zdDePNNjhzH0a5PvcnnRzY7f1yIcYfX51z9g/N/hBCciHwE+O0i8sFmhCEiX0TsROd/1P9MzKP36p1OJ4UQHFFgj4g8RMxj+tfvZF/3AA+c/W4BP0e8UX8vUWzfoNHH/cwRHfdWcRxtehxwLO0qIl9KJBU/Bvy+cJ+i2W+CY2nTm+DLiETjpXt83INwHO36x4GVfes+QAxq/EruLqfxYeA42vRm+L3AGPjUPT7uQTh2dg0hXErn81UiInPn86XAEvCLR3FcOD7E+D8Df1pi3sVfAL4C+B0HbPcXgR8BfkBE/hFxauCDwFX2ji6+k0hkf0pEvpM4uukC7yMGy/zmm52IxKTofw34CWJD+2xipOdTwP9x55d4pHhg7AcgIl9GjHY9l1Z9cdI9EUL4/vRai8i3EAt6vEYkx18B/A/AHwshHJQZ5F7i2Nk0bfd+Zl73NvAOEWnO+ydCCDfe/NKPFMfOriLyPuCHiAO5vw58kYhM9xFC+PAtXvtR4Tja9DcQH9z/kfhA7hPzmf4h4LtCCJdv4/qPCsfOriGEjx/wvS9Pb38i3P88xsfOpiLy3xJjCv5v4BVgmTjQ+E3Anzkkacrd4tjZNeHPEHXL3y8i/yR95zuI2up/dUtXficIRxDRx80jKr9937oPpPXv3rf+Q8BPz/3dBv4BMc3IgFgt5Z3pux/c992vJ/5IJZGs/lbgY8AP7NtulfjjvkzU41wnBnr88Te5NpOOfy0d40WibquzsN+b22/u/MJBywHb/k/ESNySmPnjjy5seuc2ZRaBftDy5Qu73r5d567tltr0wqa3ZNP3EYnGxXQu14jpoX4voG7Xpgu7vrldONysFG8bmxK1tj9MlPaUwC4xu8PvuZN2urDr67b974ne4QlRwvE9wNk7te2tLG/5ynci8ggxSO47Qgjfdr/P57hhYb/Dx8KmR4OFXQ8fC5seDRZ2PXwsbHo0eDva9S1FjEWkTYy6/FHitOa7gP+NKBz/7BDCUWaMOPZY2O/wsbDp0WBh18PHwqZHg4VdDx8Lmx4NFnaNOC4a41uFI2pW/i4xXUhTqvl3vl1+0LvEwn6Hj4VNjwYLux4+FjY9GizsevhY2PRosLArbzGP8QILLLDAAgsssMACC9wp7nXluwUWWGCBBRZYYIEFFngg8bYjxiLyQRH5ivt9HkcNEfktIvIn79OxPygiIZWIfEvgrXhNDwoWbfX+Q0S+PNnhK29h2yAiH7wHp/XAQES+W0ReOeR9Pp5s+YHD3O/bASLyIRH50P0+j+OGW+nv5vqCL7/b49zp9+833nbEmJin7y1PjInVi+4L2VhggdvEb2HRVo8TvhT4J/f7JO4xvo2YtmqBBd7q+CjxHv/o/T6R+4W3tZdkARCRIoRQ3u/zeLtj8Tu8ORY2ejAQ7n/BknuOEMKLb7ZNKv5kwyJwZ4FjjBDCDvCm9/hbuT8+Vh5jEfl8EfkBEVkXkbGIPCsifzZ99utE5D+JyBURGYnIp0Xkm0REz32/6bD+XJoqeEtOCYrIdxMr7zw8d52vzE2R/DYR+ccicoNUAvRmU4UHTVmJyGkR+fsiclFEyvT6L0SkeINz+hoR2RWRvysix6rd7cM7ReSH0rVcEJG/MH89IvLe1Ea3Uhv9sIh8zfwO5qazPkdE/kuq+PNv02dfLSI/KyLb6RjPishf2Pf9zxeR/yAim+kYPyOx+tKxw6Kt3juIyHtS27wuIhMReVVEvm/ftGonXfdaWr5XRFb27WdPvznXnj9XRH489b9XRORbH3T7ici7U3t4Od1LL4nIPxCR1X3b7WlzMpNB/FER+WsicplYEGFFRD6QPvs1IvKDqS2ti8jfk5gO643O50tE5PtF5JLMnnF/ef/3Ulv/aRH5ShH5qMyeea/zah/3/kJEvk5Enkn371M3ucY37XfTdr8n7WsiIp8Skd90UL/xNsBn3exelQOkFHPt7TeKyMdEpAT+aPrsC0Xkp5JNX5NYsVYOOuhxwbHxGIvILydWS3kB+BPAJeBJ4PPSJu8Cfgz4O8QKKV9MrB5zmlhWEOL0wM8B3w18V1p36ajP/T7g24jX/SXEspQQO+3l9P7vECv1/H6gdTs7Tg+MnwVOECv+fRI4A/xmIE/H2f+dP0Ccev3WEMK33+a1PGj4AeCfESv+/EbgLxGrc/0zEXmIWJ1rAPzPwDbw/wF+SES+NoTww/v29e+Bfwr874AXkXcB/wH4fuBbidWEniS2bQBE5JcR0+d8DPhGYAT8YeBHReRXhRB+6Sgu+gixaKv3Dj8EbAJ/hJij9GHg17PXQfK3iFWxvh54L/DXiCmcvuEW9v+DwP8J/BXgq4FvIZaR/eBhnPwR4SHi/fvHibZ5F/DNwH8iPi/eDH+OWJXrDwGa+Oxp8L3EAe/fB3458BeIZXM/8Ab7ewz4OPEZNQA+O33vXcDX7dv2CeLv9VeIv+c3Ad8nIu8LIbwAx7+/kKh5/1fEtvtNxL7ibwEZsTobt9rvishXAf+S2Mf+ybSvv0nsV567Zxf1YOAHuf179T3A3yb22S8BGyJyCvh/iSWjv4HYp/6vxHZ8fHGUZfUOcwF+ktiBvWnpZeJoxRA7rU3mSohyQFnFt+JC7Fgv7Vv35en6f+Am279ywPoPAR+a+/tbiQ/KL3yDY38wHccQk4PXwP94v21yl/ZsrukP7lv/KeBH0vv/H2CZK89JfFg+C3z0gH39L/v29TvS+qU3OI8fA54G8n3HeBr4wfttpzu07aKtHr2NT6Xr/E03+byx9z/ft/7vEsmezK3bU0Z2zoZ/Zt93/zGRrKzc7+u/DTsZ4L9J1/OFc+v3tDng8bTNR+dtkz77QPrsH+5b/+dSe3zPvn184Cbn0jzHfh+RtJyc++xDqa0+ObfuTNr/N8+tO9b9BfAzwGfY+wz/lcluH0p/32q/+7PAp/e15S+a39dbfbmVe3WuL/jyfe3NA1+w73vfQXTgPDq3rkscqIX7fb13ujzQ01wNRKQD/GrgX4YQRjfZ5ryIfJeIXCD+UDXRS7RC7DAWmOEH7uK7vw74xRDCx25h2+8kelR/RwjhrRKs80P7/v40s9HxrwE+HJK3BiCE4IB/DXyBiCzt++7+3+HjxHb7b0Tkd4jInnabplO/DPg+oofZpGlwIVYq+jV3fFUPLhZt9XCwTvTy/FUR+UYRefIm2+1v358CCmLlqzfDv933978BesDn3M6J3kuISC4i35ym18fE+++n0sfvvYVd/GBIbOAAHGQPRfQe3+x8lkTkfxeRF4netxr4F8R7fP9v9nwI4fnmjxDCdeA6qT867v2FRBnklwDfH0LwzfoQNe6vzG36pv1u2tcXA/9u/vcK0WP+8pFeyIOJO7lXXwkhfHzfui8l2v5isyKEMAT+42Gc5P3CsSDGwCrxXA+UPSRtzH8AvpZIhr+CeEN9R9rktqZg3wa4mwo2J7l1+cnvIRLHH72L4z1o2Nj3d8msfZ3gYNteJT6MVvet37Nt6ti/mtjW/wVwNWnlvmxu/5o47VXvW/5nYPVB13TeARZt9RCQyMBXAR8hTp8+l/S0f2Tfpge1b7i1PvTaTf5++HbO9R7jrxC9aN8L/AYiaf1t6bNbueY3ap93Yo9/RpQ6/G3i7/UlRFnAQeez/7eC1/dHx7m/OEWUTOy3I/vW3Uq/2+zr+pvs6+2CO2mbB9n4/AH7Omj/xwrHRWO8SXTj3+xHe4I4Gvz9IYTvbVaKyG+8B+d2HHGQh2NC1F3ux0mit6lBo028Ffxa4EeAHxaRXx9C2L2tszx+2CCW09yPc0Sbb+5b/7rfIYTw48CPSwwO+9VEOcAPicjjwBbxPvh7wPccdALznpW3CBZt9ZAQQngJ+AMiIsDnE8nR35cYVDY+hEOcJXql5/8GeO0Q9n1U+Drge8KcnlxEerfx/TfKQHEWeGrf33ATe4hIi6h//2AI4W/Nrf/c2zifeWxxvPuLNSKJP2i24ixwIb2/lX53mPZ10OzxWeDVuz3ZY4Y3uldvxgsPautXuPnvc2zxII8Wp0jyiZ8Gft9Nono76bVuVkhMnfN7D9i2At4wMvgtgpLbu84LwFkROd2sEJEneP104o8Av1xEPv8W9vkUUa/0JJFw3M4D5zjiJ4BfmUgsMJ0O/N3Ax0JMg3NLCCGUIYT/lxj81AXemaaofopIaj4aQvjI/uUwL+YeYtFW7yFCxMeZ5Y4+LKnD79r399cBu0Q5xoOKDnPPjYQ/eEj7PsgeHvj5m2xfED28+8/nA3dy8OPeXyQ5xC8Cv0P2Zv75FUR9doM37XfTvj4C/PY0MGy2+yLgnUd5HQ8oDute/Tmi7R9tVohIlxiYfmxxXDzGAH+KeAP8nIj8H8Qp0ncBX0CMVr0AfIeIOGLH8idusp/PAL9BRP4zcSR5OYRw+YjP/X7gM8CJNFX6EfZGSx+E7yNGm36viPwN4tTTnyWO2ufxncSI9R8VkW8n3kiniJ6OPxxCGMxvHEJ4OqV9+XHgv4jI1+zf5i2E7yQ+xP6riPxFYIeY0uY9xGnaN4SI/GGiXu4/EQNNm9/gMnGaHyKZ+UmiLf8pccR+CvhlgA4h/Jn9+z0GWLTVI4aIfB4xmv//Imb20cS2aolR5f1DOMw3JgLzi0RJ0P9I9H5uH8K+jwr/GfgGEfkU0S6/DfhVh7TvXy8if500QCMWl/qeeV3wPEII2yLyYeCbROQKsT3/D9ydFOW49xd/kWi/HxSR7yJmkvhLRJlEg1vtd5t9/YCI/COiHT6Y9vUge86PAje9V+fGDbeC7yTa+kckpnBsslIcxgzU/cP9jv67nQX4QqKoe4to+GeAP50++wKiV3lEJM3fSvyxA/D43D5+NfBLxIfvnujqt9JC9DL+ayL5D8RghS9P77/yJt/5LUQCNgY+QQxe+hD7InaJ01H/iNjJVkQS98+BIn3+wXQcM/edJ9Pv8nO8QdaFB3U56JrS+u9mb7T6e4mpcLZTG/sw8DW3uK8vJaZwu0jsYK4QSeB79233WcRgietpu0tEjf2vv992WrTVB3NJdvjnxLRUI+L0808AX50+P9DezDIsPD637mZZKT6HOKgYE8nGtzGXTeBBXIjk6N+ktrdJTOf1JezLFnHAff542uZ1GUzmbPZr0v28m+z994D2Afv4wL51P0zMEHCdmBXkN3BwloCfPuDYrwDfvW/dse4viPr/Z9O5P0WsQLjnXucW+t203dcfsK+PcUD2m7ficiv3KjfPSvG69pY+a1ICTohSjG8hDl7C/b7eO10kXdgCCyywwAIL3DaSp+gvAlkIwd7n07nvEJEPEIPongxzmRIWePAgIo8QZwq+I4Twbff7fBZ4MHCcpBQLLLDAAgsssMACt40Un/Q3iJln1ohSzP+NOIPyVkzRuMAdYkGMF1hggQUWWGCBtzocMVPF3yVmsGmCE39nCOFu0kIu8BbDQkqxwAILLLDAAgsssMACHJN0bQsssMACCyywwAILLHDUWBDjBRZYYIEFFlhggQUWYEGMF1hggQUWWGCBBRZYAFgQ4wUWWGCBBRZYYIEFFgAOKSuFiCwi+N4AIYTbKiUDC5u+Ge7EpjCzq+kW/Oqv+2oeOn+epe4yy90VljortFtt8tygjUGMBi04V+OcxdcVoa4JriY4x2C0zYVrF7ixcY22VqwUBY+dfYh3PfwOTqyeoN1pk+U5CgU+4CdD3HgXO96iHm3h6xKCB1GgM1CGEFQsSO8DEgLKx0V8QLwnAF5pnGis87ja09nQrFwq6F/MyF8Bt265sLLBS6c3eeXsiAunS67JDhvbm2zubrE53GE0GOHWqpi0aHh3dl201TfGnbbVZz/20QAgIigRNIIRhSiF0oINgco7KuewwREkkJucIiswImhARKGUIKIQJShRhPSPZpG4nQgopabvkdgWp/HZ6dUHP33fQIXZdiEEvPd457DeYZ2lrCom5ZjB5habV6/x6nMv8NGf+zCf/MQnuLCxwY73fMG7HuW3/s7fxhf88l/OuUfeQX95BWcdzllCgPX1Df7En/qzfORjH7ljuy7a6hvjTmz6u//CPwpCbDtaKURplCiUUiit0M17peL66d+CEj3X5iS2dRXbatMOm/UgoOLppb9u9aLwByUZCOD9rKhD3CYQ8ATvCcERvI9tOQSCD4QQ3zft23uP9y7ux/vZ9mkB+Bff/ofu6lm1wMG40371VrBI17bA2xKtTpv3PfI458+dpdvu0m336Lb65HmOyQzaaNCxY/YhdoLO1riqxNU11lbkBurqDG2lyLF0jGKpyMnFo1xJqMG5Cus8vqpx5Qg7HuAmQ1w5RCRgshxlCsRkIBprLd5WBOdQiQgjGlEaRBNCwHpH7SvsZIwdjQgD0HWBywqylRyMIixD/2SPs6stwoqlp/qcbK2wvbTLzniXtZ1NLspFqmoca0C97csyPLiInEBAhACIFoJWVHXF9nCXndEuu5Mx1jlOrKxyevUE7bxIhCR+b0osAJmnFGnf0zUhHW/vx4AQhFQVShKxTqcVAPHxPRKJg7NUtmQ0HjMcDllfu8GNa1e4cuECrz33Ii8+/RyfefFlLo3HlOlYz792jU9+4imWT5+lt3KSpZWVSJCCQkLgNkvVLnCPMG1ZIoSmLUlqG0yb7ozgKhUJrjSvMv1MZhvHLyuZHSGR4ukB5l4gTMd5rz9BQSGzAd50YDj//fg2hHR/iAAqXYcgwYOEuI/gZxdIQ+A9CIT957+gtscSC2K8wNsS/XaHd509z7mzZ2gVBa2iTZG3MSZDG43SClEAkQQEH7B1ha0yqqqiqg1GAqFepaMF7StysfTbBZkSlHdQlTignpSRxCZCHOoSb0tMlmPyHJ21EW0ICNQ1rq7wtkKcRSsFpoUoBUrhg1DVJeVkSD3Yot7Zot71hLpFpVuYfhfVbuP7Bd2VNieXBN0P9IzjRLdmUE8YTsac6m5QVSWXBhcJ235BjB9QNKR0ymslEQSjqUrH1u6Aa+trrG9vUtoa6xzdTodMa7IsT6RDzZ7jzX4RZoxmtvYAzhwf9kGmD/kgiYQk0oD45C6OdMM6y2i8y2Cww8bGJhvr61x65WUuvPQ8rzz9LC8/+xIXNnfZCmEPb9gqKz7xqac5+8hjPPKOd3Lm3Hmg8XL7Kbla4MFCQ4ohNhP2kWPmyeIeUqxg6ilmjhQza+fN3/NEWOaO3Pwd5lb5vecXprMi0xWRR08HdUzbM9NBX9qxqDToU0xT2wpxoEZ0nITQHCXtcO5UF7z4eGJBjI8his4q3aVTBH/QbRfX3TQ9dXMTB08IPo1+HQAiBhGTOgOVepLZdOv89/cdbh/2dWa3gT3nfbOLOITexiihnys6GeTakymLkQojAR0E8WraaTZTZ+IdeIsEiwqeTIRunmM6HSQYDI520SbLstiR+oCrK8rhDuVwFz8Z4e0EQojqCZ2jig661Y1Tedbi6ppqtIuvJgRXoXWGKTwmB3ROCFCOBwy3bjDZWqPaXkOPPRPXoeW75NkSWbaEzpcQrchVRkcZKDKyIqfwBb28RaaEq5ur3OhdY9KaxNpPCzxgkJnHqvEYxzlrRCvKumZ9c4NLl1/j2voNyrqiZTJOrazQzgztLIse47Sr/buese20qvEME4lzJDlqL4n27PEYQ4jcRYFzjqqq2Nre4tULF7j4ygUuvvoql165wGsXXuHypcvcWB+wVTkmB1ytB56/coXVX/gojz/xJKfPnafX7ZEXRbp+tSAaDyqEJH+YDeIE2UN6mzY8Wz/vIZ4N3qbfSft9U1IMUzIugddFTk3b63QgNj/jkQZ8itgAJSRyPDuu0Pzd3BlCED3H98P0WuLs/mwgIIs6EccSC2J8zKBNh9/6+/+/fPlXfjXeWbyfn/tshsJEvRN7eez0URY8zlmcq/E2ei8FIcs6mLyD6Bx0nqaG4hSSxPFwvNF92DMaDvOaw+kDLE5Fhdskx/F8k+4rvX/d0/AwiLFAYTyFWLQEdAAVAuJqCII4mfZvUTcW8NbhrSdYD96i8bSNJm8VgEbhKVotTFaAkii/qCrK0ZDJzha+nhBsjTIGnbdA56i8jcrb+Lom+BpblVTDXVw5wrsSbXIyDz4IKo/TgZPhDrsbVxlvXKPcWkNqz9h0KPQSLV3SMo52pimkQCtDS2kwCm0UOZ4qyxA8J3tL9DrtSIwX7o0HDg2pnZKHxoOlouettjVb21tcu36Ny9euMqlLTq2sMNx9iLrXg45HKdmrE5a9r2r2ZvpBIHnT0j+UII3HWMLUKzblLBJABeqqZDDc4fLVSzz11Cf55Mc+xvOfeZaXL1xla1hS+sA+Z97rUHnPMy+9xEsvvMyT73sfxmTk7Xaa497n9l7gwYBSM+469zpPiKfLHrI871Ge3/aAn3nvGO51f8/vl32bNg6k6fOEhuyG6XFJ6ojmeRm7w2arOZeyAEHF9RKmxDg6l2ZajoZST+UfCxwrvCWIsTY5AN77KUkTEbTJ0KaYG/E1M35hKkmaEsm5/YW9/83I2p7tw9wOm62aCJSjYxjK5LzziSf5nM/7vEi8fLqjZ2c6I5Zz1zU9ex8/976OxNhVBBeJsTGdOK2vIjEO88S4IccpKKxBmLNTRPJsoRJBVtNTC7dil7RdCGmEv/8rB3rJ7wDBIvUIbAvEEILFhwpRsdPzc3NznoDz4Bw4F3DW450n2OQ9BpQyGC2YrEAZg4SA83XUF6ftbVXjbInyHqMzjPPUziF1TT0ZUQ0HTIZDqtGYuhoRbInSFc4G6qpGsjE+BAaba+ysr1Fub1ENxygfsMbhMvBFhlct0CUhc6Ci1VUQdAAtcbiixVMooW00YiAomomDBR4gSOMJa/6FuCiETGvarRa9dpte0cKEQKF0Go76RGD97B6EvR62sJ9tRMw8xs2DXU09cuBRIohqPHYe52rqcsLVa1d44cUX+fSnPs1HPvzzfPqTz3BjY8jY3d49uzses3Ztjd2dXVZPnEyDgUBojrnAA4d5726cbNzrLd7j9Z2+mbqWmc5gyOxpMiPS81+fOSzmtfL73809oeLAbu4x2UglZP8zSaYOYyAOGn2jtxCJz8Dpd/aShuk1znnMw8LZcGxxrImxaM0T7/tcvuy/+2pCEK5du87OzoC6Kun1+rz3ve/jHY8/jjE5BMF5j7MB50IkKs5jbaB2gdqD9YJzMVLVuYBzcXvrPHXz6hzWBqz1OOupa49zLi0Wa2ustTgbo7Hf3Edy21eN1hqtdew4/Ov3H+Y9rsyR1wBBRbKpPGgFQQshRL+RUjlKZ5EoKpUCCRQonzqF+HqzS5r1eYrQyDFoIttnU1m3WoZ8r1QkzDxWh4Dganw5wFcFEgyiDc6l2yFJJRo/uUfwqBk59gGf2lCwDoJHlEGbPMojtAEXI5W981PvufeR4KoAwVhMXVFVJd57yt0B5WCLyWBAOR5jqwneliBCXVYoMySYHOcD2+s32N5Ypx4N8TZ6u72rIYzBjIESTI3PHcpEe3mXBjlYgq0ItkJ7T0sUyoDTLIjxAwal5h628/9CfNDnJmOl3+f06gnseMJkVLDcalNojVKABLywz9uaaHazzxBJc/Owb7zFTUBUQCGiYfpNEBX2bD+pS3Z2NnnllZf42Z/9GX72Zz7M05+5wGBc39F1196ytbnFcHdMZV3SmhJHdXfpMf7v3hFfFaDTOi2CaEEp0CraRUn8W023Jt3Fe5zrUwLkQnQ6eKL06qYD+ORhbC5Jpd14ojWjsC1QE6iBCijT+mZ7iF2hVTE0IB4zrvTJNxPS+QRm3XVI2zkPmxuwvXXndtxzScJURkGKy9gjlZgnwTD3oCA6XRLZnVchzNurWTF927RdtXfDA1tGmumQqVRi+gENC38dOU7fC41EgkiKp+RYIkegceBM7SCoIHhpvMwLHFcca2JMgKLd5sy5MzgXmJQTRAnOOc6dO8+X/Iov4Yu+6ItpdzponeE9U2LsbCTFlfXUdaDyULm9XkGblqr21NZTWkdlPbb21FUkxXXtqOtIgmtrqeqKqq6pbU1d22nKlkNDM+WkVDOQPdgw8xNA04djIPgUXY4iiI55ltI5xrQ5yVOUInKb3mg6Qt93zJmnYM5vnTqT6DmeuqgI+Fmww5shxFNo+q/p8LtZd5fwtsYOB9hhAUUBxoDWMUWPrZDgok5YaTwKLzoR3dg+rAv4JHFA6Rj1jCIEpoTZWYfzMUhDmQyVFejg4nuT4ZxntDtAApTDXarRADspsQ48Bq8geIstKyirmKbNBSaTktprrGR4gYBHhYAN4LUh5Bm+MNhCITogweFqj7WWOlRYWxLqCYUIy0Wbdkez23EL78aDhn16TEVMdSWiCGn214aADR7rPVVtGZUVu+MJ/crSR5GpRGrDTCc5dWSltpNuMBq6KyHgg4qR+KkTUJImlSVSOIKnthVlOeLyaxd5+aXn+cVf+EV+6sd/ik8/e5GyvvN+zwcYj0vKsor9ZyJcN3Fw3xa+5dc35FbIRCHGYJRGZRlKF5gsR8jIMoNSgtZxNgj0dGCiZj1d9GRL6gt8vMe8D3hvmZ9abxDjzwKiAtqEyPXTVpZATY2lpqRmEizj4BkGSx3TJaDSoMRJTCYzlkiObYiLqyNJd3Xs1utm38mfYS3UFj78s/DzP3d3tpy/ptkb2UNRm+670a9P/01/y+Y5EQcMYY5Ez7zN0yPNBnfTY73ZyTFt7NOQmfQaEikPTcaTZsKVOEM6HU+GeKTQSIgkjtU8cfDq/cw7Pg1UlT0HX+CY4VgT4+Adz33q4/yAC9SVZf3GNeqyRJTmHe96kkcffYgn3v0uUEK7BSLRkxIC+OTQ1Aq8Bk3ARHcm4qNLNaiAl4BWHiceLR4tLq4TjxdHEEfQPnpodYje5uBnHtLDdhhPL/7WNpt5cdODMUWU753ekpkShDAfgH7woecuKqQp2b2dY7NdGrAHphKX6Mm+RY9x8nw35GDW4dw9fG2pBgPqbjuOhowBraLEpBoj3kYCqzOCMoQUTOedx9UuziB4BaaIhDfpkK218YnkLc5aggugNDpvkSvQrRyUQekM5x27W1vU5QRXTnB1Fdufygg6AwFfj6lHA1w1IRBnNCwZqnMC0WMohwRXEbwjmAJVdDGdPtJu49sZGKJ22VlKW1K7itpHYtzRilO9PqdXuox3d3D54dh2gcOBim7fffpMRVBxFmNiHVvjETd2dri2s81gZ0C+sUxvfZv28glWg6alc0iBtrMMxpF++CnTnCkppzPE3sUZE4kyG5+2EgJKPN5bxuNdNtbXeOpTn+Jnf/In+bmf+wjPvXqd8jalEwfBpRk537CThLvdc1BRaKLEoI0CZfBGI4n8EqJk1geHlowQVJpJMsTM0AodM5OD+Dj1hgMc3rsUk+BSiseGlkaoFM8cFCgRcBDyOYdC3AsVnso7rHP4EPsTFSBoqEOc3Wm8xfP8EoAUTx0UeAFJzp7p7zrX5R8e9o1Y5vfdzPI1JHO6tUra+fltZW6j2T7j9am5cw7zn9zWWU6/kxwsIc6JEOf10n0wN4gM+77f6P2VEvAqzcLNp4Kb23r/9S1wbHCsiTFAXY559hO/mIjojLC9/HzJ00+/n/d91vsIBPTJ0zEoKjSjU6bTIkoCSgJaGkG9T9NaHiUeJQ6l0msiyF55tPIE5fEhvobg0dpjmgC1EIn1vcaeWLzZy/TGj73UXEdx4MA2zD6b32B/DzDvAZa9qxutcDPdNPNJ3QISiZ4es+lkDokce+ex4wl2PI5ebKOTzrYiTAbga4wpUFkKRNQZzocombEB6wSHQamcgEqzERa8wxLAOnxd42ubHlQGVXTRqj2d7qyHQ3YHA8rRMMpUAK00SmeJlBuCzpHaEazH2fjAVXmb3PSQfIzNMqQuMd5iijam20N3ukirwGcKnzSgVVVSliMqW+KpCb6iJbDa7nCu32d3dZdBcVSjuAXuBGoaqR8JcVMcwQKlc2yPx1zb2ubi+hoXb9xga2uHkROGFioxZJ0lzgWhXWTkRiCJguJt1NDcdDOFOe9eElnG2y4QxBNEUEm3HPA4a9na2uDihZf55C99lJ/8iQ/z/OV1DoETEwgMR0MGg12qqp4jVbfgIXwTiMQMF0prUBq0jqkQRSOipwV2ZppRFRkpGSFowOAbYown+mQhhMg+vQ+pGEScRSSkjD8ABnSSrrkQZ4QMM3mEx+PwWCzWOWyo8SEgKZWiBayJR5wXqewxeSMZUNNDTzfw6f3hP5Jk77v0gE1+ppipQfZu3wR3Ns+Y5ped/s57uLbsIfLNbOQtt4UpCWZvloi005kTOZHiZKCoMU40PHmQG5VMfI1PMx9CykYRjT8NLNxnmwWOD449MYY4Qt+P0WCbT33ik7z7ySdRStHrdmkniUAIcTItpoZN01rJyxt87PyVeHR8BOCJnmKtPFpFz3BohF3NSDzeIlNPsQrgkIMr7hwC3njEP+tI4k3fZHiYbRHmIvOiNIOZjCKN2BuHkpqS43TH77um10157fn4zq4/7Hs/dSJwSF1Nkjy42iJSESxRG1aPcaNtgqujZjgrUHkbydpYFNYHXNAgGWLiZzorIARsVWFTAKi3DlfXSYMcPUl5kZPlOd7VVOMJk7JmNCqpJxVFkZNneQxsEoXOWuhWB+M6aJWh8y62muBsTaEzgsqoywnVuIW4igxHkRXkSyvoToeQG7wEaldTVXHKezLepa4niHKgPIV4lgvD6W6H0XKb3WJ8GJZd4JAwS38lqVqYQURTO8tONebq1hYvX73KMxdf4+KlS2zd2OClC5f59Gde4uKVdXZHNZ/13id47OHTnFzpYVTApJyszdM7NN6uJlYPSURUoRqvnmqC7WZ3XmUrrl25zFOf+CS/9PMf5eUrG4dCihvc2FzjyuXLPLHzbrx3saLaIdz4eZ4hIhhJEgqlUyVBjVYm2lwLRmu0NhiVIaEAX4CfSSqA6C0WCCqg49TMdIYrJMUJxP5K60iKlVaJlIPoZNJG7ByIM0zJA914RhsdMiQ9cSN0Toy6eSs+KuOSeiZqn5vdO8hCJNdm7pCHgim/nT0jAiotKU5j/vEhs2tREq9yfy7jmT55H1k+8Phv0jBC40+JM8F7XOfp85krPXqzac5c+XhdDlwI1N5jbYwpap718QZq9PjxnTTZWxbp2o4l3hLE+EAEx2uvvszTTz/Dysoqq6urdLo9QhAy06Ld6mBMjkiIQWiNh1KF2MNIlFIEFfAqeoG9ngula6b4U3cwC5iOXgYlHieK4O/XiLEZic/Odz73aPN/Mz2rYgQFguATOd6jttjzzbmjTEnx/HXORuKH3S0oUW++0S1g2qc5j7c1WI/H4sshdjTE12X03GYFuuXRheBVhhND0Bp0gc7aaFNE/bqd4KpxCrxMxNh58D4OvLTGKI1kLQiC9SW1S0E7aHTWIm+1oiTCBUQZTNZC8qhx1KagLofYaoLKDEobbCunzgvwFqMCWZaTd/roVoHTBi+Cc57a15RVyXgyxlYjlLIo5REtFCL0jXCqXdAxC4/xg4QopZgnC7EoQlladoYjrm5s8NLlKzz/6kW2L18m3NhgUjt2RDMcTpJcx6K1IzNn6ORCYVSclVA66pKtx9YOW1lwAYVGK4XWGdoYVKaR3KBMJIyCR7xlMNjh4oULfPqjn+DpF16lvMNsMQdOVgEbgy1effUCW1tbWOdQJvYxd1vgIxY9AY2J16oVWhuU0pjMRFKUxSC8TGdkOkdcIsYNpQxNoEMN2hK0wloBE30lokAZQYXottUm1bXQCqNNjA+JhS4JKu0uPVlEEsE1AeWIS0zKQRqzRERVx9SGpLOrfVSF1SE93OcYcLBpG/b21neNqeOkIccNlZcoQUsOl5jlSKbEePrl5vEh8XuqcbnObRNXHdD3T72zB19RoCHisUhTKm2XHEUy48Sh+Xv2jG883j4EXAhUzjGpLVXtqK0l+MgBFEwJsZak4fdNCekFMT6OeMCIcfSKKF1g8hxCrDZTV2O8q9ibDmFuSH4TVJOaq1eu89RTT7M7HKKVpqocZ06f5b3v+yzOnXsIrQ2ZUjRlyeNAb95T06wkeU4CRgQrglMBqxRWC7WNXgGlBWXBasEpd3jpxfbjLnc7G4UrTCK3IeikuGo6r310+BZ70xkhntdZ3Ob5zdH4vXsIhzChGvtg0Tq+CVFTKK7E1VUs/VzXhNpBZVFeoclQRYYUBZJ1MFkLTI5SiuAstpxQj3epq5raujQNrVFKYSQWZAhInGpFobKcVm8JYwx4S5EZjEA1HlFNJjFrRjVKXivQRYYiQ4mLno/g0EZT9DopoEfQxqCzAjGGYDK0yqOn0QekqqOX2XnscBdX7yLBErCM7YRgJ2i/KH/3ICH4OJj1Pk7jah0QpRmNJ2ztDLh6bY2XX77A9kuvEHYGUDW/n2fryiU+9vGPgR8jYUA1epTTy11We12KrMCojO3BkPXNLTY2tthc22QyriiyFkXRptWKS9Fv0+q1KToFeStDSaCejFm/coVPf/TTfPJjT7E+Kt/wOm6Gjmnx+JlHWN9ZY2O4Qz0nhdutxjz3wvP8smtXGY/HKN2ZDhDuBjHDRhNgovHaRB+wzlDkUXucAh0zyRAxYLIkp0je3JACVYMF7RBxGO1wIRAMEDS1i9K7EHSS60kk4NqQZQaTa0THWSqPo8bhcWjjyRKBE+dQDqhjVhlNJNM6Eck9SolZnaZ0nft6X5mt19nMq3wYSMqNuYMznY1oZA9Togyv69dl38K+93eD6RMsRE9wnPCMlokTvjM5hA8x85T1MV+98y7OIBOwzrE7qRiMS8raxj7ehyRLSbmLkpPNSAyqjC1tQYyPIx4AYqzI8iV6/dOcOHmWk6dOs7SyTLvTYTIpGe6OGQwG7A4GOOvQxuCcYzwaMhkPqesR1o6x1QBCyfxt54NifW2LZ/yzvHrxVcpJyXB3xJPveQ+tdoel5XicPCtQAXS6a+N9raYjwSYARVxAqYBTgUwJNr23WtBaqG2UJCgNuhacyOFnpbgjHOyXESVoURgdOzDnFa4ZzkdB1lQvNiv7etC+3wiH3TEcjq9DRBATiasA+JiXODgbq9CFQG1rXHAoVaB1TZ7H2QadtyEr4g8dAt7W1JMxk+GAqiypawfKkBUtsqKYTkWH4KOngYDKcjp5jllZRYtHuRpfT8Bb6smYYCvcJCBZRpZniElRokHj64pga4zJyVotlMkREzWTQWIGDTEGpXM0Gu0FldeIybFBMRqNmOysEeqSEDyVDvhU8HCBBwchxIDOurY459EmINownkzY3R2xtr7J2sXLhPWt192bYbTN5eefxsiYXqvCuAH12dP41VVaeQcthitXrvPSyxe48MpFLrxyiZ2dIZ1On253iW6vT7fXZ+nEEssnl+mv9ukudTFamOzucu3iRT75kU/wwtWNO8ryJwjve+y9fMWv+7Vcv/oazz3zaT7+/LNMXCT3gcDlG1e4fu064/GEVruYzWLdFdIjT0VirADJDEoyNEUcAAeNEY0Kmow0eDakrieJTF0cVNZYwCHKoTNHcJGAKa1wXkcPcrpeUYbM5GTGkBlNUIHaW6y3lMFRYqmCxxED57yekd5G/WIAq5l6PT0gjteXdA97XvY0Dy2HX3di7+xg48bdN+3YbBde/zvuJ8evP72bnPAt+F5k5jYGdCxwFQLOz5FiH4M9K+uo6phhqrYuVjyVmHFkazhhazihrGzMT+/SMzF4gjhEAkYJmRIKrSiMRh9ulOMC9wj3gRgLSufkrR79pZOcOHmO8+cfYWVllVa7hc40tasp65JQxajfldUVzp49T2YyjDGEQNJNlkzKMaPJkO3NdXa217D1GCWCyTOWVk7SLgq2tra5dPkK2xs3mAy22NxY58Tp05jCcOrUaVZXVsizPOrJJJKYJgpYhegJjoaKuiEhJO1wSJ7NmEvW+RSt6hopwS2Hmt02bmW/r5c/7CXITdYIHxqtV0i5L1OXGxo5Rph1LoGk0Z51+IQmnVvcexP/EXtBz6y8iuDCtIZeWpfEjTC1lqQ9NfubBmY02x8Cop48PdRER+8xRdIYZkhWRX2wBynaSJaniPaAC45g6/hESsS4HA8Z7u5QjsfUlcXkRZw6zTOCj/KKibWIjNF5QVa0yIuc3Bg0Hj8ZUltJU7p5ypNJrGDoYuqn4KNEo5oMqScDtMkobJes3cVIF1EqFn0JHucVXscE9UorTNbCtLqYYozoNkEKkBoJDi1QmOQUW+CBQVVV00wnIYAyGVpp8qyg0+5ixODH9cEDVlHoXouio/F+zGh3g20TUKMh5aBksLHLs0+/wCeffo5Xrl5jfTiitA6torY20xmZyWm1C7rdLqsnljlz9gy9Xgc3GbN59SofffoFhnc48M+U4bHHH+fJz30/D73jPKunlhkMhzx98ZXpHe7m9Jw+NAzoEIiGInlBIDSpIlAIBkKGtlkaNCc2LG4msYu0Nc4SYYnChKYfCeg4VsYohSHOwAFEeZ1GgjAej1hfH7G5M+baesn1Xcd2GdipAuM6UDso2tDtw9IKrKxAewmkDSoW2Yy7TN20pJFJQ07jGc4rlY9u4vINsV8ycZPf7m5/0TeVK6SPY22CQGktk8pSpXblQ1OzwFFbR+0iKbZuLzEeTaroLXYu1kLwPhUVa/hAzB6iBIwScrGHnP1jgXuFe0iMBZMtceb8E7z7vZ/N4+98B48++jDLy0t02m2qumRze4Pr69e4ePkGV65dYbgzxo4dZ08/zCOnH2F5aQVjYnELldLu1HXFpJowGG6zO9hCSaDbbaNFGA5HbG1t88rFS9y4fIV69ypgufzKM/z8L5xBZ5rHH38Hjz78MCdWT7K8tIpRGaSgE5EYmKdCQIeoPgr42KmmdWiP8zGrBULKdRvz3Np089wPHOT8O6j72BNl69M1JhIcgwjmCXBK8RT2dnPx0pvqeLOS84RUFGQ6gFCEEKO6Z/mNY3ceZL7W3Oy1ocQypcmHhBAIriZ4h2iN0nnMQBHiFJpyFrEO5QKYDMkKxGh8cARbEnNPRG22tzWTyYjRYIfxaEhVVrTbPVrtLuId3lpC0nI65+n0lyjaBXlmyHON8lCVAbyLxDhv4V2axgvgrSN4h6sdtqwZD3cYD66jjCZvdWjZVVoiGJGY5sqBUx6vXcxqoXN0nmPaPbKyJuvskE1GBK2QMMYYh8/Cghg/YKgmZRyI+YBI7JOMMXQ7XVZdoN/uo/xN3PytFqceOcv58yfotBSu2mV3y1HeuMFrz7/GS89c4KMvXODSaLQnw0HtfEyCSwrEHMQXeVnIs4xWliPBU9cVQ3vn0ptu0eHcYw9x/p2PIPYsp06tsrO+xebmFld3t/BApg1G61m10j3yrjtEk5JQ5kPaIjGOjNMQggGfxeInQaGCQlPhUs6I4F3yFFtiCY6aZsCuBLIMCDpqZXVy7SZdsnU1W1tbXLiww9MveT79KrywDes17HioQtQHtzSsFPDOc/C5nwPveDf0cuhksEcEa/e6CgIpj3E6u9C83i+CdoBneO+6OOA5SGJxwNcPRpg5VW5+GoL1nknp2C0rdkYTxmUdcz8H8L4p0hWlFG5KetPiUyEvP9OCN83Ge6IjwgvBxb66yUolb0baF3ggceTEWETTaq2wvPIQj73jPbzvsz+bd7/3PZw7d5qTp1bwvmY83OXy1U2uXL3Ecy88y/PPP8P6tSsxY0DQeBt47OFHabVysizDZAajDUoprHfUtkun12JlpUueCUv9HhI8N66t4a1jqddns7uMHW8T3ABXjXnxuWdAwcWLr/D4O97BZ733/bz3yffR7fRRKiMEiWnYpkskKsHHimbBNyPGMK2U51O1POtmN9mhE+PmIUGYve63+UHf2fdZQ0yDEIPtgCAy9YTH1E6C8o03uJmbm/fr7lP7+vRfc46+IvgKwaHwBHKCdAkS5+3jflzyxDSa8eTlDPO0eP8U2913NsGDr2uCc2BShSMd85rqWKWD4Dw4j28q+XmPq8tIUFO5ayVRY+ySrpiUtscnPbG1FlyFtyVVVWGtI+AwuUFjEZuhvaMe7VKNh9iqjkS6uUQlcdqX6EGrbcl4tMPO9joigTxvUdeWYFoUYqYdtPMW51XMuqoDSmuyvCBvd8jbPep2H6cC3gaClOCrRSf+gKF2FlJGCm1in6d1RlvnrKqMc6fPcOLUWbZfe5lZbq4IVWQUObhqxNaNEX49cHXsqDfHvPjcZV5Y2+GG87d8JwUCZV1R1tWhXNup5VXOPHSGE6dOkAn0WgWf+wWfz87aJi9fuMCgnHD6/HnOnT9HnpkUdHv37M4m7aej6U8CjoAm4FLazkCI2QtUkpAZPw2utsQgbJ2m4iP2WlGIGQwqH6hrj7UwnpRsb21z4/qYjz1V8ouvBJ4dwHULo9ftAXYtrFm4+BK8uAFfeA0+5wvhnd2Yex8ilW+uQxGbgE6z+xKIsS4+rnepRF7wafsj9tccrHA4aO3M9XGwU1lu62cPc//Pr/QhxMDUScXOqGQwrhhXNmWOEryP3mGf5EtRXhFlEt679MCITqP5Y/j5xafFpSJPLhl8gWOHIyXGojKefP+v4PM//4s5e/ohTp85y+nTp1k9sUKrbdAmcOP6Gq9eeImnPvMZPvbxj3Lx5ReoRgPmG/fViy+w/q4nWVldIdAikGFt7KAdxIIaOFTS+qpEtPJM0+91eOihh8k7S1zq9Vi/cgE73mJ4/TJP7azz4lM9zpx/hHJccebUWYwpyPNYZsk7F0s/O5/KSbtURtqn6ngBW3tsHajTErMS2GmZ6HDoHuPGvZtew97ik/s72Cl1DnPkltQRKYn5LpPgzIcwTWszFTwEUqWnhg77WVc2LfsskTAHR/CRXDvvCHYEdhdFhYhDVBdvMpQqcHOCQWmmJyHJNVLqG2a+4nlSLK+7yjuwog8EWxOcjYn5g5oeLQgxhVNy+OBDJJrO4XyFE4NVJmp5IXaACHneiZ6izKLzHC8SyfBoQD3ZpbIV1lusLyFUuFGXusjRQD0eUY/H1MmrLNqgszx6sosiejSqXUo7YTQaMNgeE1zA6DG1DahWH0yboFo4yfE+6tu18qmUt8JkmryVU3Q61GWPSaixboKzk1ipa0GMHyjY4NGiMXlOkbcwpkArQ8vk6HaXdzzyKJ/1/vdz7fJrjG5cnJHjrIUpNHa4zXq9wW41gkHJeKtmMPKs1YHDobd3hkJpHnn4PKfPnqC/1MEgZATe81nvIVeGz1vbYFRX9FaWeO/73kOn3UYbNb3/7wYuZlXDeNmjjXYSUMqhTZQWoT1ex2wccX6c6LFHY32IqR9s7Oci1Z55zwNQOs/OuGZnYv//7P3XmmRZeqaJvUttYdJVyNQlUAJoNLoxPSTnhMNb4JXw0nhC8oQzDznsJtGNhqoqVKEqtQjlytzkFkvxYK1t7pGZJcMjMhLI/3ksPcPd3MR222t/6/8/wWbjOb/Y8atfLfmHD+G/LeAi/mGksC7CRws4/zsIBu6/A+MMjj1pCZUehEtrtQqgs+bbe7DZ2cK79L5jtnRTtx3//nWdYfgSFv46hJvvMODiryEf70Hz8K0b///VJSu7T+xbPym8w/tI0/cstg2rxtJYj/UQ99S+7J6BTLuGkDgqA4MnDi8/DhkH6Yq19+rfv5d8FRHJ3em7aOhvZ71UYGyqA/78P/wn/uo//jXT0YzxeMJkXFNVhhh7mt2G02ePeP+DX/Ozn/0Dn/zmn/H2qwrnrlnw6PEXTKZTxuMRZVkhZRI4iMGoHcfgZGBdnzp+waOVYjadJn5lYahGNadPPqdZPMY3a7bNii92Wz774Y9ZLBZMJlOkUEih8jjFZ1Cc+G7OhxQr7SLOxX1s9ACSvfP7MXgM/qXYtcRhhYuBQd2xh67xxqVD8NyJub+wxAHPSpAiRQ+T+KxBpNjm1DnJMHQ/xhyWg5CBa2SIWRqCPGJIVAQfekK3AnsBdMkezPQIOQZZ5VciEih+zg1B5pcYuE5HunaiGED7Cx/DEPF9j7cWpc3186RYKuKwORDJrD/kJC7rIhaBEwovVOIjk2J6TVkThSL0fUrRi5G+7+h3G9r1gqbb0ruOZlNhd1e0owmT0QijNN5afG8Tr83HxEMGjNYIAZ5I27dstksuL1ecPo0EC1rCvF8h61MwNao+RBRF7oSQr442EeWFRGmFrgpMXWP7LXQqWbp1Ke76u3qNSkqEVokGU1YoaRBCobXBlAV3jo744ffe5dGPf8SHwbNbnlGMx8xPjqmqiJEtu4srTs8adk0CWa9D/+qgnvDOe+9wfHLIqCoSiSHUPHjzDWajKU3b0nuPKg2HJ0cUpUEJ+aWB+59YMvkAWxsxMTcFBAQZCNoTpCXq3HLVAZRmbzGQN85KaGJ0KK3AK7wf+rZpbfXBs2tazq52PFu0nF85Hj22/LdfwX/fwOpPeNkrC7/+EP6HU9D3YDx5Hm8K8sXcgQkJCPsAyiXQ7P11L0WRwPOt1g0AGa//+dvudqOuvbSjgL0yLyaQLIbVPtMlUghHuk+6+zUd5rkr3XPdotTs8FHsqRPxplfxYDotUpR38C5RIYb2UJRc251eX4Ei6ZiG/LKHgWHM1934pevvd/XtqZdLpZAF66bhsydP0DyjLkqOj6bMZjUxdLTNio8/fp8P3n+fR59++rWgGCAGy2cff4B1gfl8zmQ6oa5r6rqmqGqKskRr0CrQx0CwHcFZttuGtu1xMY2857MJRfEm43HF08c1q4szXLMGJF3Xs1hcJh6zKjCm2tMkgs9dY5/ody4vNM7lbHqbIkwTtyiNXUQc6AMvoQsXyVSOIZAkn4Q3UuYgLyyZmxj39Iv8AEImoaBMTho+hszzDYk2QOYEC4FnEMIForBEXNpph7RoxDAsICIr6Xu82+HaC0L7mFJ0CAMKR9RTYiyJmPw8FmJ+hqj2CyAxr0di4EvHLy1JL1YhBmzXYLsGISUxeGTI5vsqCXL2oozeYnuLcwHrA51zdNbhhULXB+hyipEaUyUOctcnGoZPHwz6dkOzuWC1uGC92mEKxXpSM5nOmB4cUNVjiIoYRJ42BEzpKD2UUVACzrasFhecP33K55/v+PxR+gwaAXd2AcwzUIbxsaEyE4YF3LmUooWSRKWJ0aOUpCg0ttAYLelDoG8Sfv6uXp8yVYVSGlUUSGMQN3LSRIzUpebu0YwfvveQsXEsr+5wfOeYO3eOse2a5cVTPr+4Yrn7qmnBN1n37t7nne+9x9H8gGLAk0oyGtcYpRl7j4/JwLeoiuygEBkyE16ketIR1Fokr1+ZIpaDCiDtdRSdTl7iKLffVA7LjpQCYzTOD3HAgmEw6LzDWsdqvebRkxWfPA48uYDPn8D7mz1l+0+qxxfw6fvpnC+zD/LwmjQ5byRCTpHGZ1Acbeo93Ey+U7e5QxpAYRTX/vDcAIp8dcW+6Wkc8oMMFBAhApIkZA85CjqIpIFJFmoRJUVyFMpkX5kfU+6DQmQOqErWezoI6koyG4NUPW3nsdYSbQfOkhI8PF23o2s3BBSYGagyXWvjAMKHq5C68f/73vHz71vcxpXqu/om6qUC4xAFpxcXBAH9tqXUiof3jzk+HOPsjs1mwa//5X3e/5dfsV1f/o5HimyXj/i4bRnNjpkfHjKfz5nOpozGE+rxiFFlqCuFloHOW1zf0+w6+tYRZfKiHI8qDuYzppMx46ridDLj/PQUgiP4wGKxYD67pDAVdR2BZLfjQ0icIx8ZsK93pM6xh+Dz+CaEPSgWWWJ268A48lx3NoSAlCJFWcYsGMh3lUKilMgb8JjpDok/JfeuGulBkz9u4r8K1GA3f6PDlJMBo8+UibxYBPEctzbGQPAdwW2w/SVu9xShegpUshSLJ0QmBAwhKkQMyBiIQaXxfxapiLygDR7S+0ShW+rAhxixfZuBsSAGg/ISoTUiO/InSkjAdR2265Ni2Tm6rmO32+KFpoiGSo8T5z1fVGNwaUPlFMH19M2a3eqSi6crzk8jUlqqccvsaMXh3Q2T+RGmGCNVRXCe4AMFmqA9sbcIrejbhuXyirPTFV88g89sGqVqYLuA4lFPUZ+iygOK8RFRDjznQPAWlEQWZdruSNBGY0y26osxbfjaWzm039UtlS6KJDTWyXg2eNLi4xxCQqEERwdj3n54wqwKtM0BB4eHzOZzNldXnAXHqfqM8EJw7HZLIjm5+4AHb77JbDpB5017kAJRlZRVledSOUHyhvvPl2ljf0pFsmODyHSEPHCMwoNWCOGSNaIUCBmyt5nPCDruvYKTEFLuAzWcF+k6kGlkXd+z2QSWK1hdwWabntNE/mQay6aHX/4K6hqmB1DV6ftJMpga3IMlhXOZR5xVeCFcA1Q53Pe2KsqBTZepCfxWRHjd1xV7RySR54PEJPaWuVs8DCoH+bYPkd4mWqNWEq1AySSAjvlaFmISqnoCLgwan9S8sW2P7zpC2xK6nth1YFuE75EiIAm4bkNsVsm1R1Ygiz3SvW7NyESpiDeFr/65bvn+uv9dw/hbWS8VGLvdBR/+6pd8XlV06w1aSe7eu8PhwZjV8pzFxVMWZ2fY/g+5Ilt894z1+Zrd5orFYs54OqUsK8qq4uhoyt2TA+pSI/AEa+maHtt7hAloJVDCYJRkXFXY2SESxbieEIKnKmsWlwsKU9F3nun0gLoeURQVA+IRcohOvnZr2G/B89hdingNiHO++q1XfG5vmhejiPc+ibvyiqKVSl1h1JcAdSTElNwjZUDpFHlNCMgQCShiVAQkPuZoz9wNCN5AULl7nBUd+BxvKhHRIWiIfksIK4JbggwoWaJkj6cnxo4oIiEmR4bgI7b39B2EkI6x0gpRgVJF7lanS9ptdYxjjDjbYbstEY+32e3EGIQpkvl/SB1j26eNVrp19M2ObrPFyQJZH+256HhP1zZ0zW7vIBBsx26z4uJszaePI19s0qsvr+B45XkzrLhHwWReMxoXiFKikOhqjBlPMGWBNBLfNGwaz+nS87S7vrh64FmA6gzK8Q4zOseM5shyTtB1nno4BAoZs8hwcBYJERk9SkQKCeI1cDX/rq5r4Fs659KGvA+EPqIKjaoUzu6ojGBUSC67HcvzZ1ydn6FNkdfFEYcndykeXdD614FEkUoqhTEpbU6KFBcsRdoED6SpdM4LriHP7SCMfm9JLFEIZPYyVoj079wJBIi5PT2kzl1XJAV9hP33tcrWaDE57xzNan7wdsfh3LNuYLWC/3QOjy7glxfw6Q6uvuQo8fvKRfj7j9LTT4/g3gMYTUAPPOPc+MRlER7XF/jBtm0AxrcdCZ3CsFITZvD/lwwd4ETNG5wcEkUiInKAjSOmJo5MlDSlUvoiQ2JeDmSJiKQNEQEv0/VYK4lRApFpj855euvZNT2L5ZrNZoPrLa7ruVotOb9csNtusV0HMVCUhrouGNclo7ogSoGLiqCL9LwIpMr85yizwC7kDvLgPBFykEe6pbDb/MH4rmX8rayXeymMDdtn77OFNN8BNpefo02F7ZbE8MfObiPELb5p2bZLdlcThDKooqDZ3cWoSJiOUCIQvafvepwNKTksJi2yFJHCGGaTCZUpOTo4IsRIWRmurpbEIGh3PQcHOw4Oj5hMZpRVjTJlWqjFwM+NeWubbmK4Eff8XIDnNpW3VF/dkccMWgPOJ8C7551l3+fhPjEOSVppxqakQ3mLkn4/qvQofJRpvB8Ho3pDCJoYTKJQSJueQ/UpT14FpBIQLSE2wJbg1zi7IiqRY2Y9QfRE7L6zEEIEB33r2K6TP6QykqJMRviUw5u9Pta3dQyd6+i7LcH3eKXSRbIwyLJCKLPnlnvn8dbhuwbXbrHbLe1mQ1Ajir7He4+NHuc72t2Grtklv9miwNuO7WbN+bnn0RY2+fm3AbYrUKc2AdpxoNYl2tRIXVLWY8rxGGMUSjqCWLPp4WIX2X7pauqApx3MngUm8wvGswPMRCDq4WLiUwxtZM/fSxT1JKFWRAqV6JTf1etTAwXKWkfwFrvrsE2H0ApdKXbrK/A9wvdsry548tknrNZbOht58+13+cGf/ZiDOw8Yjz6nXf8pzNaXVFFc6wfI42+uu4TJT/162nbtfnML4NhcdyBBIxE5oUyiUgYee5lffl1SJBJL2E/QMjAebB7yS0sexunRjg4mzA4ELloQae1sfcG603zwxZZffLTk//63Lb9Y/HHYaRfgl1/AO58UFKOaslYotUH4nmgTdQKfQKmJCUwnAfV1MJ7kdi/8e9eiPejd94quMz6G+5K7xhGSNWaeegoJWiONzJOBdKxvuiT5GDNHOOUMEKEQKb47ZvvNvuvZ7HouLlc8enzK6bMzuu2Gfrvh8uwZzeljYreE0CYOf32EPjxkenLIyfEB1XSOGh0gdE2Q6ShljXqejA56mvSNNPFMZIp07WfPtQ6308P5rr6BevmXwvj84Ci4Db3b/JY7/6HlIe6IcYQZjzk4OOTw8JCqqlAq2WqFvV1a4j6ImEBA8A4hFIVJ8Z99b3HeIZCEEGjbliVLnAu0bcdovGE0mVLXI6Q2ycotz/aFjAgV9nQAEQJCDPbq+faylP43Fpt443tfVvAO9xhA8QCSg3N41+LDFuIWRZ+39hKkQUiNEhohFEqUREqCqAhSEaXZC/eQMjk4SImUCu/ANo7d1ZbNxYL28gw9rzmajFFIfFSIoEAYIhpcCsvom5Zm0+BDpCgLtIIYqrQ8ijRse+69vmCFCLZv6VvwN4ScypWo4JC62HO4o/cI5xB2B/0G+gacJWDxtsN2DS50hH5Dt9vSNTtMYYiuxHUNq2XL6RK2X3rxXYTFFu7sHAdeIFSJqSeU1ZSqHlGNRmgFInbIcoweHaInE8T56iutph3wZAWzpzvGk8eMXKSIEl3UKSpaKaRUecFO494Ufd1nW6FbOrDf1a2Vc5YYwblI31k2iytWl1e0XUPjGi4uz/ji8Wd89vEHfPgv/8IXj07xNlGrNqNz1kfnuK5nVI1YrNf84eZsL68CkV27o2kanPtSYySLrUQkTedi3IurYv7viy6nvUw29DJGfCYjSUAJgRKp8ytFWsu0VMgcHf98zzoFITl3kygn8pQu0QlKo6l1SVQFQVb01rC5DJxf9Xz+aMtvPux4tv7TTrtFB//17y3OGUS8Q5gaiu0zdA867m32U3dz4BjnJxpA8RAKchtlvd9vKGSmSMjM+023/DORgrBCCIhsd9q0LbvdFmKkzImASgikZG+bNnDpfIh0Pvt6S4XSilGhGRUq0TF8oGl7NtuWxdWWzWZN27ZJTJdt+GLw0LfAAuiJ/QK7mbHs36YsS8z4iFExRlYjrNS0zrFc7mibPsV5a4PWiYImpUyx0DKSxx7JzGKYKGf+/Hf17atvcY8oIsuCO/fv88477zCf1tRlGuW7LuTd8eDekEYeMQS8cygl0cpADHS+x3Yu27wZnHfsdju6zrJabSirK6azGZPplNF4TFWPULpIqUg3wXHwCBEQe09en/2Pb//MGNS6z38vj7Nk2tEmjJtlawOFYk+9II3/+w7aNb47R/otSiuU1inMwhRIXaC1QagapCcqgRc6CROyhVFatJIvsaQkOkG3iSwvtlw9W7A7O6UKR4STu8ioEcEk3hYlREPwLaHv6NuW3WaV7IQYEypDyle9KWsYxqsvXiFC37d0rU1jPikRSqJ8jYkRZcrc1YiI4BDBItwW+i3C9YiYRqne9thmi++3uHZJ3zbYrkdQ4W1J3zUslz0X7dePTXc97LqIDRKhS8p6xmh6SFWPKKsSKTzeaVTVUs3vMDleYZ5Z/LZ57nEisAhwfhY5mFwShQJdoqRAFVO00gg5OK1EvHc4l+gh3sbhNPmuXqOy1hFCoO8dzbbh/OKU08ePeXb6lNPzZzx+9ojHj77gyeOnXF7t6H0a/k5FpF8uWTx5jMdQ1yPKoqbpd9/0WwIiu92W7WaD7e11bgfsu8bkTnn6fnh+4vWCz25Nom5JF9EEwCHRKAlKRqSSSC32NDSl9b6jff0OBDkU87nvAiBzwIcUCJMEYC4Y1q3ig8/W/N3PF/xvP2/4p8X19OiPrQh8fBqR/+SYT+eoewVje0EdHaVMvQ3neU4PE0OiTwxEEXmLaszO+iRyCxGpIkqoZHWpIkomDreMIKNAqZC93z2ut1wtV1xcnONtR60FRpD9gpO7U/Aeho53BBtIVp9SI7SmrkrqskRrhRCK3jq2u47NtqVpdyADZW3QtcK6lt5u8W4H3RVp1rYB3+AvRzT33wIMo3KErmoaBNuu5fzsis3pGlXXjGZTJtOKycRglLqebiSJTXKMgjRFvr1D/F294voGgLFKuZZCZ9UD6YrsG+DrXSm+vhLYrUcj7ty5w2RcIrG4bkfouz0YDPHaazB4j8USQ1r40so2qFtV8uUN4GJSFkNH0zS0bcNms2I8GTMejxFap5AKqZAqdVWJkehJ4R45iz18ySXiduvG48a4X6RF5nld/2igTtwAxyIpq1UO2QjWEvuWqARRK6SzCFegjEEYg9A9QvegOlANUhZ5EzCQ7wQiVsioCbYgbAPNomVzsWJ9ccnxuCC6gAgKETUyGIQoIGpi6Aje4fsG163Tx8ELiEVqdQwW9oO445aWmxCgb1o6nflvKqlxdExWdNJZovNE1xNsAu9da2l3jtZGmgixcui+R3YNtlnTb6/wzhGixNuObrtie7VkufTsfsvHIEqBLKaU4yNGsxMmB3eYzo+pqhpTGMDjbEPvJfOTlqN7PePHW9rtI74ME3pg0cL5k4CL57iYwkeUAKXShicJXjzBW0L2cA6JHp1CAL6r16ZSGldK3HLeYZ2l7VouFxd88tknfPrpxzx+csqmSW0ADRwZyYPZCKMUq6fPWPeBy8bSu2/Sufj52jsXDICXa/7rsHbtoWi82XZ78bXU+0AMAoXDBZUJEZ4QTOaUKmImVyD0c766UkiCCMRMBblm7GZUlFfhIFI31Hew3vV8drrj15/0/L/+fsfffOp51MKLGsB44NPznl/8/AK/MdytFUe1Y6qzViCDYpf7NDJe84oHB4vbqo+/OE1NmYGOJiRKJM2GUjptNjJvWOWfB++xfc/i4oKzZ0/ZXTwjrM6J3ZJom+wxn1PkMk1hWP/TxDb9faQxqMIgVYFQhohKgvgg8AGi1JRZoN/2HdEYGM+S0thL0qopQBqsT7Qla3t0sJS6oJSR0HXE1QrX9mysI/gRQowojCB4RwwOH5JQO/hsV5ojpeM3ksX9Xb1ovUJgLBHlEeM773J4/yFRSVwMtH0Cn93pM7j6DGLz+x8qV3A9Asl4NGYyKiF2tNHTyW3OwLhhvi1EchjwFi/BK4GI6WQtihKlVF7sxB6sB+9w1tK0O66uBKNRQV2XuChpPciioJ6OqeqaQmmMVImXGySDScXLGKVcUyKe/3d6v2ksLiAJ3LLALl5fiRAidUJ0XeFDhWtLgjV7/2URLNInuoW0HVI3yGKbNgRaglZIrRLVghJClTh6UeB6SdxG7KqjudqwuVrSncwINiBiOuZEDcKkrwGitwTfEPw6H3+DoEZgSbv6HAuLzBuNFwfHIULbhASMJUgdESYQRQtCIfoO17bYpqXbOJoNtA20HTgJsQKNRXYdomvpdhu6zSrJd0xNsI5mecHqbMlyE/ltk0tTV4yPHjA7epODk7c4Or7DbH5EUZVok+zVbN8QVcWdJrJcO44+O2fx7JTwJbATgVWEZ1fQO4+1z4i2T+IcrTBlnYSkwaXfjQ5BIIRr68FvppJAVGT7pX3S1B4u/dusGPJ5LkAogS4MxajEE1mu15xfXrHNoLgW8HAy4odv3OWtu3d59uyUf/nkEY86yx++or6aMkVBURRIJW+A4wx89v7pw4Qo1y19DKyzKCTSa5SHLgqMEBgZsD5RwogKj87XgyEnL22etdQ46xBGE10khOHnaToXCLjo6PrIZhf4+FHD//a3K/7XX8LPltzq36KLkV988gi7q/n+mxJ/RyEqjzYgcuIdMQnxhk6xzl9v09D6X37xUaK+KJVuMnVvZZ5ASqUy3U5dA+OQOsa7i3O6p0+Jz34J3WekLcOL/LFvzh/S36+Xc9ajezCaJkuPyRx0Cd1dsF16uoNjvIBts2W1ukSoyHh2wEQLShnYhh52PaFr2LoRIU7RhSQ4l7yP99auybUpOE+034UmfVvr1QFjWVDe/x7v/OSnvPX227gQaLqW1XbD1XrNMyL95gLsH7F0REckYrTGGAMhYGVaAlKkYyDEgdJATnX0eAdORCQ6CSxE4gp551L3OKtfQ/BY6+hsh/OOrtXsjGLTWRbbHiskelRR1gXjomBclYxGM0bVFKNLlNDJcuxWa8+2yzyyfBkZ+LA3540hEmUghOxLHK9/WyqFliWyHBPLaYpG9i6tpnIYZ0IYWg7YnNzswQeilwSlkFRIxkh6VJQo65B2Bf2GaHcE2xBdD9HfsFoTScSAyKMni6BD0uZ1rU7OFfT5bwwxDgb/t9MxjjFRhTuTRDMyd1Ui/T47pdv2NCtYX8L6CjYdbHPQVTWDiXaYrsO4fh/mMlwAoutpmhXbZUfvvn6pF1IwOzrh6N6bHN1/i5N7b3F4dMx0NseUBUpJQnD0fQuqYNt6TlYNd++c8Gg8Ybf8qsWhA9YR1BqkDJjiKlEyygrqKZgaHywhuNSNuWmB9U2s4cIwmb/NnXtvMj2YI6Xk6uqK9XJF2+xwfYv3LTF0xGAJPn0mXo+oipdbafpzTX+KQhBl3pgaQ1HVVHU6H++Mah6e3OHO4QmzasLnzSPOevfagWKA9WbNxcUF6/WKtm1ROjnApN18BsbPbf5v77x3+bogvENGjcHSA4WLWJnit6NM4E5JTYhpLQw4JDEJ8bTGOYvWGm8jPiZKBqRrRvCwWjsePWn4+S8a/j8/g3/Y/uk2bb+rVs7yq6cOUdRU4zG1WjNRcW/ddpM+ce2Cza2e6+7TJ6nVr3W6SZUWVa1zQEpOSJUDr0LkaD4HqyUsNtCtuJ0jdHO6kP8u4RQ2K9gdwvgumDI9d7D5OT2sT+m6DWeLM9ZPpkwOjjg4uYMLiub0GawuE49YFcQ4ZkefqDLOZc5KZN8NCz4HHfjvgPG3tF4dMA4e51ocnqgBHxEyIGVEZwu0P/ZsjTGNGa23OK8RIQVxhP3tOpY5EiBbjqUUO5v4rFmUpL1EOYnRCq2HfbVnUB9HPN4FugCXl2s+e7JguVzhui0CizKScjTm3lvv8vY73+fk8ISDySGFKW79UCYQHAkxZGB5TdsIeXQjSMrdENJxzgeM3KABKYnSIIoJanQHoccQXOLOioCUnhhdcq/AEfHJCN05Ii1BdERhUVKhZIlWU4Q8gxAx8nOq4py6auhGgrIElXNLBUk2nfyQQeCQwmK0oyxs8rUULTG2hNCnqGIhcxjI1wkL/8RjGKHv0k2Z7AWa6RrO9rgedivYLODyHC7aJNdYk+JUD5fga0/ddYyDQxUl1fQoiU2ExO2aZBfYQUU60b7MVBjNZrz17ru88c57PHzrbe6/+Saz+SHjySRPMBIX2NoWIQXb3Y7Dozn3H5zwxpsPeCp6dqsNIXeEDOm5ZH6uXQPrpcdUF1DWUM/Qo4jr+0yjSOTD5Gv8vHr8VZQujnj41p/zl3/117z73rscnxwSpeTx46c8fXLGerVjt2uQCooCrO3ZbjZs1kvWVwt2m0va3QXB7fjXCJSVVkSXznfnXJqu2Z5iNObhO+8wOTig33X4tkd2FtU5Ts/XnH52ys+ePmPxB1yUpTIISCPhV7Qz+uzxp/z3//7fuHPvgLv3T5hNpxSmTE4EN6hvac2GWzvpgc5aNCJRuoAeTR8DXYhIO7TnJRKNRyOjQmiHkxGNRxIRUiRqkvfJochHPD3EQHCRvhOcPmn52T8s+ZtfeH71kkDxUJsQ+fgKjvs5x1icaVD2eVCcySEvp1bbBHoHYDzclEoTquTXxiCiu/ZtA9AwOwL3HjQWWPJyzuUWwlNYL0hHYqDpXT9XRNJfSHoUK1XyWI8QsiTamLLE5Qzqg/RYvknXJOczGN533m78+ztQ/G2tV0il8Ph2w8422GAZ2nKDvZkUww7vj6lBROTwLjlPhCwuCn74/5AAZFaPxijw3tF1FucjSiZelM6gOJZmL2RLYRkDQPQ54COwXm1YnF5gr75gOJETI1myvTyjaVqat74H9yPj0fjWj+T+whEGDvPQNQ7JgouskCWdmzKfnwL27kIxCpAKoWtkLRFmkt07fDK5xxJCT/QdxB4RLIQ20x4Czjf4sEaqgDaSwlQIfU6wESkvqMoFo1GPazVFpRDSE6OFaDNFwuaFwyLoUaLHSIsLgehbvG3wrsN7i0DvqXwx3g6Ai4Dt020QTuyjU0MSLm+yOf9lC6fAJQlwChLtoFxH5rsGZztMUWLKKjmTeEdoJSGkxX9Sw2ELlz4zpoWgGI146913ePu9DIrfeIOT+/eZTpMDSmqeeZyz2F4T8cw2Uw4P5zx8eJft6i0mdWBx+oR+tYbWI+x151vkpkzXwXbTYdZLzHZJISXWdim63DmCTx8OnSklr6p0ccKf/9X/zF/9h//IT376E956601O7h4TQuSjj7/g0+PHrK627LYdk2nJ0fEEqaBtO7bbhtVqy+JywWeffswXn7zP4uwDorviXxNAVtlLO0awzrFrWlabLSjF8d17HN+9hxGGZrnl0Qcf8/T8MzaXF2yaDefxdxPGy3LC9OCY4wf3UCKyW1ywvDxnvdngwss9hk3f8E8//xnvvPcGb7/7Bjx8g6Oj40ytSJv88ByN5jZC4FPZEFKYBA5B2kz2CHRMHu4y3HRSUNnXOKC1whHQWeUgh0Q2IgpNQBKioO9gs4o8/qLjZ7/y/N1VWjdedq16yaKr2YUxTnSUIqTBX7wGyM/VbW6C+xaQCQQbndSHxqfusMgrprjxpIP7kZJgimTGLN+GqxI2H0N8xEs7j0UkuVo9D4pTDdNRB74Dv7rxuVPgj5OVkMqLrDSD8OWaNO8zMBbJ5SjHNn7rqgIOgAtenA//baxXCIwDcXPFerlgvVlT6gRAtUxKVPWnoJ0YiD5kCkH6QMZsA+Nj2HdU0789EoGQMqlHiQRv8c6BIINijVIQTIpLToEZFut6nHdILbPHpUz82P2JdP0e/foJj362ZfHklOX3f8LhweHtHL7nniUtwoMFW0qFSrZxRJ+7q0OHNWXwDePIKAY+hcicX0kUOscva2IM2W4mpLAO3SOiQ0WHCB3BjrF2hHOSxnqE7zDRY31LR4e3jhBWmLJhNJGEOKaoKyypW+NFS4g7pDToGAiux/cdvmuw7Taly1mI0VCUc4qiR5clyiRbOLi9JdPZNPUa1rQcXIiziWZhuxSTPDCdB55wJHEFVxtYLbaMZkum80NMVaOI4CRmPGUcAlQ79KGn3gXG28hVB6P5AXfuv8F7P/g+7773Nvcf3OXgaM5kWjMalZSlIW36Yhb/SMpSM5lWHB3PeOedhxgduX9vxvLyLpuLJ2zPn9BebXBtJPZQGCjKlI6lyxRO3rU7vFTYvsc6R9/32Dal3u0pvS+9BEV1l7d/8B/50U9+yt379ynKEh8CXdtjrWO72bFdN2zWDc2uQylFP4koo7BWEWKFKQvGB2Pu2BKh50wOHrK9ekzfXODsBmd3ONcSg/uj35igIL4w1/HFK0UFpfFzCHC1XPHo0SNc8JR1DSiavuXxo2f84y/+maurs99rySZVwcmdN/nJX/wF85MZemQIocdud3TLFdvzKy6fnvPJ00d04eWpMZ9dnvNf/79/w/HRAf5/+GvG0ymVNtdGl3stQY73uCV80fc2d081moDDkLbqDiMsLlhsBsQiP78Jnug9mkCfVt8E8WJMRsE+QFQ4X7BYWL74fMe//GbHzxfwVYnsyymFITQ9tmmxZcDH5GOcYeCecbuvW31RuakVXDoeyf9u+PMxCLSB687x/tdcpmgrmB4mwLpxwDm/H5J9+UMhknZF1Qhd3dDUKKQuGR/c4+TBWyitaNZLVotT1uefEfpVElrKMuOIniT8vvn8HthC2CJHx0zfOKGeH6U136T1mizwJ1uOjuoKpb9dxl8SeBP4My14Y1zxj+uWfw7xtYqUfxX1av9qdsPu8pzl1YLpZEJlCiQRLQT6TwTGhEyVyKOLARiH/L3UgfCEkPnDQiZvSplGddalOOhYaqQq8EESY4onTmPsHuc6nPcoaZAy2V5JWfD1w6lItEu2T37JB23PeH7yokftSw8vUnQy4jp9hxyVHAPCe6IET0jvdViUYha05EaMiMn+Jr2DBI59TKk9Ivv6SjwKhyRdFESwONUixAjXQxsCxA1OdljfIP2O6Bp8bDGFox4bopyiqoouCGRvIe6IYYtQBUp4om0JXYNrdtjdlra3oCMxllT1jqpqkWKEUamHE28JFkcyzSyj3RjTmuwd2DZR3V2XO8h8FYxbYOVgufCM5kuKesxYmRTBrSRSK3Q9pXKBiQsc9J7DnWPTR2ZHd7n7xtu8+c47vPPuW9y9d4eDgymj2lCWGlPIvEFhWG8pCsl4XHJ0PEPpyPxgxGp5h/XyDRbPHnH6+YzFs89plwtc01MUgqos0FkoKaSm71ssEecSnaJvLV1DCgSQt9tE+m1VjR7wl3/9f+J7P/whb7z1NpPZFB8C6/WGEAJdZ1lcLlgurlitGpqdRQhBVRVoY+g6h/XgRYUqRsyPSsrREd///o8Ylw4tLSF0bDdXPH36mLNnT1heXbBeXdE2K7xt+Cqp5WaV1OMD2mZBCN+sk0NkoBApfIDVcsWTx4+RWnN4fIcQJIurDR989CmXV+f8PrSjiyk/+Ol/4D/8D/+ev/jLnxCUY9VcYfsW4QMmCsYUNOdr/p//t/+Ff/jw5/iX1LkLMfKbjz7m7t/8LYdHJ7z13veQZZ141CJNtGLMgQkM6Z4vXn3foxEkeZ0isUwFBoujx+mkCxFCXC8KOcDIikiyd8vUhEjyYQ8BF6CxitNTx2/eX/GLTxwfhd/9SbuNEqKmKmYc1AXadrBrsUWKuh54FGkul7rjL2coNHT3Q+6YynTc8vAyNZIyhSKZHN+4f16EpYJqnHltJWyfgfuU503tBDAGpuk+OgtElIKiRNYjZodHnNy7z2w6JfqAiJGqMkzGNe+99QY//N7b1FXFbtfwxeNn/N0//IyPPvqIsqqZjCfYvmW7XrI4e8b24gtit8ggOY3hRBE5PBrxg++9wf033mJ+cEg9GqFkWqyjT5qmyWTEwXxGUd4+lfJl1h3gf64VP33zLm/cv8f8/c/YPL3kI77pNsGrrVcLjENP/+wRj8cj5vNDZuMJWkpwIXXa/oRZriAlzpE7oZG4p1L4kDjG3qXOb4qbTAhAa4E3edwd82OQuMghOzs43+N8j3U9PnhMSOKXQZz1Oz8qsaO7/A3d4tM/9Wj99oeOQ4cvEf5j3iB426cuoA/YCEIbyqpGG3NNsxA52clotEr+nckcPdILkSkWmiiTbZGP+XsxtVaDLPFaIKqAEgVCNCjVIfwG7ArCBmkaVNiCkbhW0bgRq52iixahVyhToquIUiW+XdDvruh3S9rdEmsDqlJ429I3W9rtBq1HlOV4CG373cf9Dz2GJCeGm1PjOOgmskPD0E0eTPFvpkcBbIGLDZSnLePpFk4i2lToskbI9FkJUWBdYOwiEwd9FMwO73B8/wH3H9zj+OSQ2WxCVZpsGh8Reyg+0IsCSkFdF8CUolBMxhXz+YjNaswo+3cLEVlJaNQlVVlS12OMKVLXQhm8MlgXaHYNzXpFd9Vjr9L1syxfPpVCyAn33vgRb7z1JvWooml3+ODZbDZUZcmorgg+0LY7jIHppGBUGSaTklGlMYWmqgxRKISqQBisHeHdjOlYcTQvmYwMVaWxtuXs7JSzs1Muzs85Pz/j9Nkzzk+fsVycs9suk9tHdJmnL0Ek7+5I8lH9pisiIcr9JthnH3YfIl3bsto0fPTxFzx5/Dm/e44iMKMpb//ZT/npf/wp99+5h54oXPTooBCyQASopGFeTqnNmOnxHeTHBh++aqFZqIKyKGm6BvcCXeXWWj7/7BEfvP8hd998yIO33+bg6AhTFPtGhyRN3JHDyf9i537bRLSISUcxZOCFRKnTaGRQpLhSef18CoRPCXhg0vY8BlQMKCHxPnK57Pj8fMF//9kz/svPev5ul0h2L1rDrO/r3rVAce/uT3jr5JADdc6xfspcBHSbGqcY9uTiwVTu5dSXetLep1GbytdloZL4bng3Udx4U0OzZuCTyyQSDsB6R7JwtVyD4ntQHlxzmIVIXw9mVHeOuHPvLm/ev8fhbJ5sKoVkNCqYjCse3jvh4YN71GVJby3TwyPKyYR3f/ADVBbwd23DZr3m/OKCJ0+fsLy8pNvtiN4zmc85PrnDG2+9xZtvv8vxnTtMJzOqukarHHIlBErCeFwznY0pzLerY3wAHFeKWampJZzUBe9KweMQX0sh78uqV/9X216y+fgD2oMT2sMjpuMJo6pOHUFtiN1+/vL7K1OVxA0yf+LfhqwODns6hLM68cKkyspilYQeIYVyKAVSDuA4OQw473DO4r3Fe0/QmiCHx/46jtKXy+bd5u3VQGca4iaTEM8TnMV2HV2zo+k6dn2PLkqYzxFlhc+ddblXB5eoAgopkQSCiEQHwYvBv2yfGui9SLzDGIlSE03qmFf1IWBRWITbQLMgsASxwfsVXmq6IKEfIbea0lqEvKIoImPvKIoa21zS7S5pt1e0uxU+CCpTg++w3ZZ2t6KuJ+BnN/haL77E74HxsKYP388TQZ8dwwbhSkE6WW7CpQBcBSgXkeNVAz5iTEVZ16jCIFRyPXE+4gLMpUHokun8gIPjE45Pjjk8mDMe1RSFTuEw0UOQ+41MDOl7UkJVFRgjqGtNPykZjwyj2iCjp++6RJFod7i+p6hr6smUsqwwpsILwc55mt0ucXQvV3SXEbuEUgKj1IB5WSVEyb03/px3vvcek8mI3nZsz3cIqdDKUBYFoyoZ9QcfGI8N82mF0YayLCirkqIsUCbdtC6RSiXfaO+ZTyqODsfMZ1MmszFCCVbLK64WC5aLSy4Xl5yfPuP09CnnZ2dcnl9ydbVku92xaxqsTRvgvm+w3fY1AcZ5MpS1PFJKtNE4H+i6ltPzMx59+jG2GSCYeO63AWShOX7zTd770Y947/vf5423HmIqzVVzRQgOl3eAwgtciPQmdVCtknzZM1wIyf17b/LvfvoT3nrjIR9/+Dm/+eAjnpw/+loA/fsqAOeXC95//0NGhzN2zvJ9o5nOZkSfpmBKSrTKNn63sCFumgFop1cQYw/BpyAcoZKd5IAmh6fTiihk5urK1AgNkoBHaHBO8OTZmn98/5z/9y97/ssOrl74laZXMSXh24avBoKMRyf8xU/+E9+7WzPd/Zxx84zKRnQOMR1wqC6+hmN8qzW0hvPNDzzdmGMG82uRQxee54GxEGkhti6BZmWgmsD2CMIOaPOd51AcwmiWXS1yh1pEiAZdzyinB5STA0bTOUZrSmMYjQvGk5piOsHKIvHGlWB6cMiP6gnvfu8HWO+xztO2Hbu2YbPd8v3Nht2uoe96iJGD+ZTjgznTyZhxPaKqKsrCYEyRE/EMRaEpC0NdG6q6QKmXe+RvsyQwAYyI+L6l2awxIvCgVBw3ji++6Rf4Cusb2M4E2K5x0tAozawsmVQHyKMDmu0D1qeRsF1C6PjdAFlSjQ6YTacoJZE38PSeVxx8Nsl3eG9xXoITKZAjx07ub0oglEhf9xqBoRN9w1szewWnLtM3J/SxvaXdrunbbfaltbS7DW2zxTpLIDAajZCjAqVBeIf32cECwa5dslsEjFLUxqB0gRMFURRZYBv2rWkxKG2J6cIhFcqkzruIHoFDyBpBQZAjhNph44SoFFYoRCzpwxx8gQweH9eEaNFKYpsruuYC69bE2GU7H4uQNjtT7HBug7UrtBiBMunvdwv13Pqc1+xh9PhlQfFv+yQ6SHZsgdx1KDBFhSoKUHkU7gISgSlrytGE6XzObD5lMh1RVQZtJEKE/FlNG6kkcHRp4pHHc0oKpFQIUlfLGUFpBGUpqeuSajSmHE1pRg26LFDVCFOPqcoRNkLX9oRdz27XslhYumUSVY88yG3iJb+sini6dsduu2G73SBk2phKpUGnkXknIIYUqpD8xRVlkS42RguMFhRV4lsXhcIYg8AgEIzrkunkBkdbQFmW1KOavqsZtTWj8YjxeETXTvDZY1RKQQyBvtvR7Vb03eYbp1AMFXxev4IjRI8qNNVkxGa7ZdNuWe9WON+k83A8Qc8PkSJkoaxFEji8c8T3f/wj3nznbQ6PZugxuNhhXRJFDUlz3kfW2zWfffaMZ5895Zfv/xL3JbBbjWf84C9+zE/+6qe8/dYbHD+8z8mDB/zTP/2STz57n65fc/NMUdpQlTV1VbFcXmG/JmQkApvNmidPHjM7PuDg5BAXbALGQlJoTVUYYt7chxdU+nddZhhkDJcS2Txa9ijRIYOGoCCI7L4VCEoTlEYJiYgCGcReNN66wGLb8Xe/vuD/8fcNf7u6HVAMcAi8RaLrbiN8QQLHwxEYjY+Yn5xweCg4KgpqIxFrkF1u1hdAHfMAAQAASURBVJI39gNd+6XV0F0QkN2f9lzjQD7IIb+Q7FIRM70CrrnIIf+uyDM6M4PuPqlr7EGMoJxCMUr39eHaEi0odo3jbLkjigVX2w4tk8NUVRdUo5JRWTIqC4rs+KOkSMmuQtH2lrbv6DpLZ3u63tL1Dusk0Zv0VlpJv7Kc79YosUkTBAFKSow2mPxZLUtNYRRFoVFS8D++/Z9e5sG/tZIkP/Sq0BijMVoyrgqOxxWH7YYv/g1xKV4BMB4k/5ljFEj/zur9yhhODg9AHVGPK85PjlicnbG9OMdtV0SXSZA3Hk+oktHsDg/fepe7d08ojNmrXm+GXaToW58pFQ7hkm+ukBopdF5hwvCoKJnGISqro5XKST1KpVFUHB4/ZrXtqwfGgxCl6xrOzk5ZLS4hpo5xs13TNRvKQjGdVJhKUtFRC4mLDhcS0LLOcrW44vLiHKU0R4dHTGaHFKM5qpwQpCWgky2RzOA3JMGJlAIpFEGksdFgoyZ0jZYQdQm6Q4YJ0WicMGipwYwRRgAdNrQ06wXe7hAhWd94tsgiJBP4IiCNRZoeVIMNK9peU0iHUROEvD0ENzhS3Gx4XB/odBtcSn9bDzHZnWmKskQXBiVl2k8FUqqWT2l6WimqskgxpnVBVag0DcTl7l2fPrs+XZCdSxs6nwMvpEgcHu8srm+x3Za+W+HsDkKPUgJdjjGjA6SRKda7qND1mBgFMnZ4sWHXWK6W0PlrQ0LpoHyZhMjoWJz9mg+JGKk5ODpiMp1RFiVFWWIKQ6E1Ssr9ZsA5j5Q+BX9IgQqS4CUxJK/owmgKU1AYg1YJaG+3Wza7Db3tWa1WLK+WXF1ecnl5zsV5olZcLa7YrNesr1YsFguWywu6bpm6h69Red/jnMtULoepCiaHc7a+Y3W1oKVFTDViajj53jvce+MhIjii7TASCiOZzaYc3jmhnmms2LHadZmXKZGqQKkCHyNdsDw5Peef/ubvOf3gA2yzfv7FCMH8/l2O3jhGTwtiBSdvnTA5nDO7c8iD9x/w2acfc3b2hBg89x68wdvvvMPJyTGT8Zh//sef8fOf/yNNdz2QlQhmsznaaJrdlqvLc86ePaZttwQESirqqmYyGhGJuOwV/iK1axIwHh5lOP21sEjdI7whekUMKSXPO4/XGi9MEm/71LkXWuCINN7y6bMt/+svt/znK7it0G0D/NjAe6Nk9bbsQLXwGFgBDo0pRwRhCcKjCsfIp9eXTYn2DZ6XjoufA8aRvQ/G4O8rfKJT6Jw4ctOpQWX6hSAB4sHfOEaoRqndnTd6aJ3VxCbdV5A5zRE8uPMVz9YN5+Y0USazqlpqjTQaJXVyoRIi/bqSSGOQQmOtxfc2ice921u+Dq5PQoiEB3T+fZEacNGl95RS/iRSJ/xgdJp0SCH4v/yfvz3AuJIwrkZMxhMm0yktgum2YbTYptCvb/pFvqJ6icBYIPSI8fE9xvODxBsj0Gw39G2HKSomkyl3Dg+4f/cO1aji+M4RV/fucvbsiPOnh5yfnrI6OyO0a8AiZEE9O+Ho5C7HJ2kUPZ9N05jIZ4rDEM34FRFeyN6tASE9SqZozxBSNw7JPnkrVVo6E57PHekcoRyvtdMv7/D99sOKEMlr9+rqimdPn9C3LV2zY7W8ZLdecjAb8eDeIbgOIyO+H6cAE++x1tK1HZ9/8jHvv/8+Shnefvc9HrzxNod3BBNtUuJfAOc8zluctfi+R0nJdDpnPJ4iTYHUKo17Y+quCy0SXzE6KGqi9gQdiVoiTI0sPUpovHW0bUvbLZFYNI4gI6pQKFNSjdKtqDWmECAszu1QvkDHwan3xStkEOoGbQXJiaLroGvzV5/GmF/ent34cyBF6hYrY1IEqkjuJ1Fl9YuICKmo64rJdMRkUjMelZTZBSWSEhljDwKPwOYo4Ow1jEcIj1ZATNMBbztcv8O1a3y/hWjRWlKPJ/iYALcxEl3VyKJKF0wV8SjaNrCx15rrYRBa3spR/V3Vc3X+Pr/6Rc/J3Td58PBNZgcHTKYTqlENOYHSOU8ISRjbW4fpFcYoyt5Q9yXelclvO1j6LJJq247trmG73bJrtqzXaxaLK66urlhcXrC6umS7WbHdbvAeyrLGW8dyeUXbLnj5Mqk/vkJ0eN/TdTuabksUAV0bZKkIKqBryfTeFFOXvPHuMfceHqUI866lNIpRVSYKSm0QqqdzO1znEcYgTYEuarSM+BhpfMfFesGzLz7BfRkUQ+oqG8fGrll1S2Z2xHxyyMHxAdW0YH5S88a7Rzx9+hRnHQ8ePuTunXuUVQ0Rju/fof5w8hwwNsZwfPeEg+MDJpMapSJtswGZkiKV0smuMXf/g48v3DFudtcXvYEmZQCtAkJZMC3BCYILuMJTGI9TBqd8cqqIqWkSZWDdtnx+fsXff3DB35z5WwPFAMfA2yM4nMHFLiUY577pvkJwdN0VtvfIsMVIm+Kgs0nCvl66qnaYuzmuuwzq+meRRBFzJBN479hn8In4vJ+oiLlzTLJyq+rMbbP5YXUCxUrl6WK+v/ewbmC1xceYmgk+34b7Iq8v2YoEypVJP7Mu3fwNm6IB7EuRMf9gxUa+CHvo+/T6hgvB8B4ytfPbVOmvJjDKUJoi6V4albRF/4a6xXDbwFhoqvE9hNYoo3n4zrv86Cc/5t7du9SjCu8sp2fPWFxeEkPAKM3Dhw959+03qMdjem9Zrdc8HY+ZFAU4S7Nc4pihlWJ+eMTDN97g+OQ48XvKikIrgu0TPzam4A7nfQ76SDtJEZPvZIzJ9xg8Pm+tvQ/Z3UEn4V6MOH99YXYu3U9IjVAqie8Gg8hvSKcpRYqrbtsdl5eXnJ2ecnb6jGePH7FanHJ4MOfddx7y5sMH3L9/j4ODOVonjrUPnqZp+fk//5q//dv/SlGUbDuL0CXleMZkfkCMAds7Li4XnD475XJxyfpqTVmU/OAHP+Sdd95jPJ1Ryix+iIKAwotkJWeFwWmJN2Oi2RELQTQVqoK6KBFeIYVFSU+wLdH3SAyKkrKeMDu4w2hyhCrGSDNCqiJxo+EGwfrFKgLWQ9ckoYqWIAK0O1itYNunC9GOFOyx4+v/2pIsWhvsiUTqRChjkGWNMgVJ7GmYzafMD+eMp1NGkxFFVWK0ghhwtqfvHd6JJP7zaSMnRESb1CQJHqRIHWPv+mRxt1vjuw0ydlSl5OD4gHp2mNfpmKccpCj0EPEu0HfJzmmowVb/JTIp9hVjx9Xlb1guPuGLz445OHrI3fsPODw5Zj6bU1V1diBIm4vBL1YSKQpNXVVMRjXT2YSiMFxdXHL27JTHTx7z+OkjFldnbDcrunZN13UE16WUv+f86DTz4/c4PDrJE4/Xsw8iZMCHnu1uxdXVBZvdit71KCOZzifoWjJ3I2QhqUfg/RJvO7xtkdFgZYkQJYGGKAW9TWsjSiFM6jgW5RjvBG1vcWyg/C1zkRA4//hT/iFu6N0KXUSEiphSoirPaC65I6eMDxXtrqXrOj754gOWyzVXlysef/6Yq9XiuYeczCbcf/sBb73zkHv3jpkcTKjHNVGRfLYBHx0u2EyLiy/cMW7W+aKXh5hSZAgnIShL1IKgA653FBkYF7qglx6jDFJKbO/Y7nZ88uiU//KLK/7zZeT0Fj9CR8A7CmQBpw18tICPQ/KUvX4ax2b1lGb5MbYUeHGOjzt0jAk78tUh2Mur4VUN18QsVZaD6I60ZkcS8Bza2YOa2udXOPhl7l/0AJIHQQ3gsg4jkqkaN+7zFW5c7iork/yVA9fXDi2uBZ0DGBakdXwQAUqZ/q3yh8VasPm9pkXp+r0REqf6uabZK/PAvJVK5xs4G4k+HeemabhYbbkIL6lbPLi/vGZ1q8D48Ogd/v1f/x8wlUJqwbvvvcePf/IT7t27x2Q8wtqex48/59npU7q2xfvA8fExD+/fp6prnPesN2t0CPiuZXlxweVkhEQync64c+cub7zxBvODeeYRxzSytv0+n865lFDn8hgsOJ/CPTK9wjtHiBGRP9UxAlKlx8txoYP62+VbCOnECHkjGLJA7xv50MeY/YpTst+uaXjy9Ckfvf8+y6cfEP2Cq7OKi/MnfP7oAXfv3efw6JBRVVOUJTEmRfs//PyXfPHoQ6TUlKMZ04MjJgdHjOczfBQ0Tc+Tx4/4zfsf8Pnnjzg7PWcynqJUyeHhHUxRUZQVYphYZUseT8TJdAu6SPGbWhBVgTKKqq7QaAgWESO222G7loBF4DHlnPH0DrP5McgSpCEKDaJAycFb9HbK+ZR8J7ZgI2Bh08ClSwl3LalTvOV66b9BZd//Wyj2Zu5CKaTWmLLCjMeYskJrSWE009mY2aymHpWUdeIWIyAGj/UO2wd2jadtHM46oneYQjKaFFSVJupkM2X7Ftc29Lst3WaF223AW0ojMfWEoEZpFOwCfdvR7xqs77A2eRe3bfha+7lXaeQeY0+7e8LT5pyrxSNmB3eYHx4zGc8QKtkiAtllxqdjoTSjqmY6GTOfz1BK8tnHn/Dxh+/z7OwT2n6ZLpq/txy23yGVSQ40r+3FKxCipeu27HYrur4l4qnqAl1NmYuaKD2ogFCRSEuIDfiWKDTBeYJ0yWxRpOTA3tu0se8Vzu3woSEGhbUOIRuKmcL9Flf/2HoWn5/x6M6IBw+PmU5rJtOKoCyqdkyMYXZ8yGa95eMPP+ezx5/x+WdPWDy5IvT575JPX11o7r/zgPvv3OfNd9/g4cO7KCOxwdH2PS5aonMYZ1KzI6Rc9RcV4LWbPL2/4dYwAGN0IKoOryzeeLyL+CLiZcCZSCc8QgqurlY8+uKKv/t4w/9yHnn2Qq/o+TLAAwlVAWc2AeMPQ6JPfLl8vyBuPoWJJKpzgmwStntF1otfrRurpFA5DlqnC8QAPuFGRxb2DhWDA4+ImSZx8x3cIIMM2NPHa4A8gF2Rn3e4UMvcudYmCSgiWaxHArzDS/Yh/VtmR5Ic0IGWuau8R8CpOzw8V7j5OiPXSXqea/Ld67np/rqKQBtg2zp2bU/dalbbhmfbnsXv/e0/oQpNcTjH7hriZvdaLcO3BoylKPjhe3/G//S/+x8pxgXCwNHRCffu3udgPqeuKyBQFYq7d45w1uFDoKpKRvUI7z2b7RZv0+ieGJiMx9y/dx8pFKN6xHQ6TWlQfgjviIisXo4in2MDP9E62q7Ddx26UJR1kYTGGfwGnyKhkRIlAyEUOSgkOVUkjqfH9hbvY9ahFWiTrMRwXx5svapKHrRKwHw+5+DggBAi2+Ul0S8ATwxblhcfsbz8nI9+U6PNCFVUSF0ihSQEz/bqC2Lo8cHy6eefMjo4wknDctegdIF1ng8++oRf/PJfePTZF+wWl4znx7z7vR/xw23LZO4Yx0AyfxuUHslnOODx0RGxCNnnhU4TSXM+ISXGHFJVCkmDIHkfW9chxBytDzHmCKkLhDRIZZCqIApDFLfrxhl99ituoXPpAnRB6qD2fPUv/OVz1wNBCIQuUEWFGdWUozHlaEw1HmOKAiU9WkaM7MFtiDYQ+0CILrl/RIntI03rubzYsLjY0nUd3vdMpxV37x2hZY30GVCuFmyWF6wvTlmfn9O2uyQANzVFIZBVgfWC3uYNomjoestms2F5tWTbvEa0gWhpt09pt884fawQIo1KhZCDbCCLXSNSSJQ0aF1QViOklKxXF/T9mj/2AuRtS7Pb4fotr9WKfKOct8To0UZQjwxTVxHEBB+rdL0vBGWlEDrSu5a+b5MNn1JoIagKgzYaoRWBiA6gQ8CFQAgWJSJSBogKEwLzUeCtdw84M4521eEanxNE2XdqqwPFZKwRocd2a5ptgQ+O1m4IwSOVpgsbdnbJzq7AWPRcUBRjxuMJhSmQAg4OD/jJj3/E/XfuMjkaI6sUQuOjp3ENy/USlwWoSiumbor6Ckfgj69dk/DO4JSgyMOewYzCRIRJYrwYNCIaYoj0IdBsVyyXG371myX/vyeBX/q0ab7NssC/BNANiOY6uPjLpYAHteWhfsJRkBRxg9B+T70daiA4CF4mb/JmbzqTU9QQC53b1yE/+6B21/k+N4FlEvKk70vFXgUdZZ7Qwr7VT75vzF+FSNeZAWSHmBLqYgStrr0oB8P2m1qS4WdptHbzBwlU2NzJtu5Gd9pn4d/QIBtuQ6d4OBYv2QPzd9Qsf31eEpvqyw0eSK/6gsjHqy2xlKyd5bPzDY9svPXPOQLKmWE20yw7R/+aLcG3dq7UZsQP3/0+//4v/x2jeY0oBFoXFKqkKDJfRUsm4xLinb3Lg3OJ97parfHO0TYNXdvgrGVc19y/dw8hJMYUFEUyy+77Pv12TBGeIiZCf5Rib9HW95Zm12LbFlNqKlejdO4Sh4CziW8spQKdFOAxJMJ9EBLvAt46bG+xfbov0RG9wnVNFgW++t1gjIG+bxHAfDbjYH6IEApvtzy/hEaIPd72eLu8oQp5nkMNke3yGf/y61/TWMf55RXVaISQivc//ISPf/Nr+sVjiGu2oeXJ2QXLTcNhZwkxZeohYlZ5pxUnRE+gB9EhRJdcJKKBWBBjWji1hliWEFsiDb3bEtkRxQwp52g9R+kCpQuKokSZAhegsz65Ad3GsSTT03poXVJ8X5GoE3+o+VQEUAJZlOhqRDGaUE7H1OMxVT2iMBJCg4w9+A7XSGxokdESTJX8hYPK0c2W82cXPHp0wW63xbue45MZZSmoikCUARkatoszlmdPWZ4/ZXV+inU9uigwY0EhA6aUSK+IItKoDheh7XtWyxVXl1e09jVbhYD0eXXEHGX8da9wEAr2PexekNDpXMt6eUkMr687p/eWGAOmUIzHFZ4RWgeQKQWwHhVMpjXaCJp2w263Zm00GymRMVBohdYKoRQuBko8DY7eJbqYDBblQAhFEFBMJJP37vDu/VlS53c9Tduz65I/ureOuqo4mY8xwuH7Lc1G4KJjZ3cEAlJpGrul9WscDdVMUUxnnByf8OD+Aw4PDxmPx8ymU44ODpnP5pRFgZN53Q6OXddwtV7QdaltXZUVve0ptX7hLUzbZlcKlZuamUoxGCEMkepaeoTsEEhaIraxPH224tNHW/7Lk8i/+JfHSh8+57+rTgr44WHkzXLNAVAMCPhL+c83lTB7l7pbr6ylQKebNNfAV6vccR14ZmJvB7rvJg8OTwO3ZfjdgSYxfH8ICBmAaRxoFqQ/oMzPM3SRh8ABJdPvp2SvfGDC9UG5eU0cwHXM7yuQ+M3eXQv9BnAcckd4nyZ7s+SNx331NQH+nNTc+RXPi0KnwANSKuNNwBuAM+A3bUt/FlnuOj5eNjzi9qeJZiY4PNBMqpDWpFt+/Bet2+sYA0YItJRorRE6GaJH7/G9pYsBb+Q+In2/s80gtes62qah3TV0bYdzDqU1o9E4p9UppEw84URvSB96GbmOelYSZy3drmW72bJaLXH9Dl1IRtMRBSmGWiCyoC7ZX8mshvXWI6Il5kCQwSM1WEvwHic8OIHdronu6/ZhL79idGxWSzarFV3b4X0gPid2+L2P8DXfatmcfsivNyseP37CaDqnqGqePT2lvzqDmHTQwW54/PSMTx4/YTIfMz8cUwtNxGVQnCgpRIuIFi0CpY6YHFoxJBTGbJQvRERphQ4Fymik1iBGuFDgvEYog0YjhEZJk7jj2VnjhY8j+aIRrqlmsO8T/MElgdHYUE/n1JMDyvGMoq7RpkCISLANfneJb5dE24L3FOWIcjRHFyOELrFRsdk5Lpcdn35+zmdfXLDd7nDecnI5I/iG7WLKSFm0b1hdPOHq7Amb5YLdegVCUIxG1M5DMSbqGi9KYkhODS5EmrZlubhicbG5tY3Ft7m827JZfsLrPOqMwSNEoDCSUV0QfIEMhhgDUgnG2nBQ1pSlodeGVheMXUrUFdEnYGxS8mEU0LkxvbNY57A+IJXKn9P0ifc+aTRstqtq+56mS7feeoKPGGWYTyZMK4OwLburji5Ydr4hKIEuDDY4qrHm5P6cGAVKGw7mc06OJ0wmJVWlqSqIqqdxGxwWFQzeB6y1bPsdu25L31mabkTbN1hvMerFXSnabZ6aZwpsKZM7WzQQBDQtbBuwl5Ft07FpO847uOjhaQsL/9v1Bq+qxgL+7AjensOBhprrVDslE312AMQyZpe08CWYdqtvYESiLpjUJR6+mkxh0JlrPLhR7E2WZVY/k/4gRiUu8Jc7xnL43QyKQ6YtxAGoDq9DXANXyB3rG5SIoU168z43+c6I5JyhVALgUuUuskw3LxN9YkDOIXeYvb/+6ocucn4938AnZYLgR0R+TJp8PoV9OEcJ/AD4CTAH/pnnQfMl8Evgqu243/Wcxsjvz9T840pImNaRsWox3qJewwvSrQHjEDzNZst6ucRJhywlSmgMSbAgdfIIljnBUZEsT3pr2XUd69WK9XrNZrOhaRpsn0bwRVleB3jkE8Xl0I7gAyrbrCmlkFHhraXd7TJwXAKW7a6i6w5QSqO0RAqFVoIoyfZsChEFwTms94kd5APBZbsZH4jOErDY3tNv1xC/mU5TDJ7V1YLF5QVt72h2u8S1li/6p+xxu8csPj9jocYIMyH2HYQEigEILefPnvL+Z59xfGfO/fuHSFkghUsb8aCS3MD3yGApRKDWInVlsqI3+IgXkRAFUSRBoy4KdC/RhQRV4ryitxGpIGhBDCILggfu9+1QWBzXmhBJurgUQEXqGP8hp6uRMJ1PmMyPGc2OqMZzTGWQKhJ9T9+uaS+e0S6e0G/X9G2DKcZU40N0NUGaij5ILtcdp4sdn3x+ySePFmx2HS54jk8ntNsFl8cjpqqljFvWF09YX5zStS3WJn/bejqls4GgR3hZQzHByzLRinwSXF4trthsXyMaxTder9+CfLNitHtgXNeaYDX0Cu+Sh2vhA3WUjFQBUuNkQd15xtYjCBRaYorkliK1JIhBXBzwPiKlSUEpUgABHx3WdvR9y65vabqO1lo67zM1U6W1EwE+0jQ7drstO9+xiz2xUBRVhVSK2eGI0aRCa4M2xd7fVcseFyy7tsH6Hbqv0abGmCrhDOfZ9jta1+K8o7cdne2SeDQG4gteoptNloZlDGZ1mvKHCE7A1sHVDh4v4bMGzkLqqn3zcS+pDhX8aJ6A8Z0ayggiU22FvKboDr7+3iUjiOEN7OeEt/nRL8YJeJoMiAdeY2GgqtKBHsDt0F2NZKVXFq2pbMVWFvl+ueM6UCRkFrMM3VqRob+N6TZoBdxAcSB1nssiAW6trt0r9r78XHeSh26yD9d85MHFwul08y7Zxg30i4E77TLNwg4AOZ2fe+D+CmtOxb+f3uVHYst77YJnfeBjEp1Ck7yxB2B8AJRIfgOcZfZ+INEJd8CzmNLu2lt+jVrC3EAZLNHZ5Hz7mtWtAePGbfmHn/8dxeGUYlaBAS1N6tLmbazUAmMURaGoTUGhNb21bNuWi4sLHj9+wuXFBU3T4EPg8PCYo6NjlEyewz4kQNw0DVeLK5qmYTJKY7m6qjACgvO4rsd11xxgYwomkymj8QilJAKxF7EmT0KZvGdlShzzzuUkS4M0AhMFwals8WZTfPU31DMIMbBYXHL67BlNZ3n69Blnz57iu9ugxyf6Ba4nuqvr7+2ro7v4nPd//RvuHE+5ezLhYFKg6FNHWEqiSI4CwQdc06BcxOh8MYqe6HuCyAEpMqKlRhuT7OFCTO4ZeXcvlUBriZSRGJN1WcihLbfwTvcaDh2vB4ElCRgPYaS/r8aF4OBozmgyRZkSoXQaTYfsGrFd0ayu2C0WbNdL2u0GoSqKaoM0I4Iu2fWRx+drHp9teHTW8PS8oWnTez0/Pefi9CknU8lJbZnrDt+ssVu7j7Q2Zc+46/ERZDmGYoysIBQR2/f01tK0Het1wy0027+rV1Ted4lOEdJGUwaLCn2yZHOeIAK9UWjv9kElJkRqpVEiJru+wV9VqXReqbTRDFEipUHKBIyFjEQczqUEwM52tLbDhoCLCW1JoRFR5jXWsloqViFQBEmFSWKaukIZk51FSBM5pXJjLqV0Op8mcX2MWB9QwaO8TSl/PtCHHllIjDIII4kiRTAPGpAXOqb2ejrvY6JPnQJWpnHxOqQ0y01I5/+LrvK33TNsA3y0grMdTBWMNNQSKgG1glkJkyJFvJsKRiWMCihzE3bod+5ucx0oqwQw98A4P4kx16BUZKVjDsVINNx8EdYxgecyA9kBDA8ivD0FYgDGGRQPADmHcEFMuwCZnaqLInes5fMAeLjvc4bPA80jUy8GgO38DRu3DKYj7MduQl2T1mUEnUG3D88LDl9BTRjz0+kb/NX3HvBu95Q7p2vUouftmMBtS7rGDWybEckBZcpXu8IOXo7gDjACigDtGrY72N428r6Fur2OMYH3v/gFj/6vXyBUWsykUAh0pjkIhBEUJkW8TkZjJvU4jc6aHZeXF6yvLnA25VmacsTb7/0Z0+kMrXRylPCJcrFYXPHJRx+xWl5wcucB77zzDkoptEq7wsGJAhRCag4Oj7l37yGjcX3DpxgyqWLPxZdErLP0XY/N1kbRe6gSj1lKh3MdVXUF629mTEKMXF5e8OjJYzbblo8+/pSzL34D4eq2n+hrvhfAPuLiwzG/Oppw57DmcKwRYQfRpiQzrairMYWpkxl+AFlLTAAdAzF0OVkspY4ZI1GqSN5EIiCFwhiB0lCWkqpOi1wIPTE6YvSEW4rrHRopA53CxGtgXJH4Wb/rLyyBw6nm8OiIsq6zN73HK8C1uN2abr1it92xazq2u5bttiXEHrFu8Rj6qLnY9Hz0xYIvTndsG3A2K6MJbLee7SV8ISPTAk4qmEioVHrx3kNRgfMO5AJdz1DVDOklwUaa1tFZS9t2tK+T6O67+r3V200Oc9niug2+W+PbDa7rU6BPr4j9jl2R/LOVFMntJaQIcZl3fj46ohOIQqEyD1SJ1CAggowy4wOJLhSFMVRFxSR6yI0DISUpcRHwAdtbNvWI9WRCFyy9CAijMGWJUArrk3jZhZDW0RgI0eNw9HkT7GMSAQYLfXBpwh0jQUaqyQgtFHVRoY3KzbvsOf9CBzUDQ1Jn7IoEGCy3Oz+QwMNSUWnFR9v+1h67idA4OL1xKt+Uvg0Tr1rAXMMbU3j7PhzNk9OFyaYNy9sExip3hJVO4+AB1A6it0jm+97gUQ5Cu8GVYk+9yFZvN8HxTW4wuTs80BZ0vBbfDZHTPgvi1I3vDQBV62uAfDMBJcQb3OUIIQsZ2i5Fm3p/bd0WMmAeNgMD9eMmozG8WmA8ZsJfzH7Ef3z3Ln/+xoS7FxumF+nYf49EpfiE9Jk/B57l/18TnnNdehVlgNjCYgVL+3rO7W5VqBrxbHcXf9idRaI0RFIX4csQpPOe7XaT7yqReYcXIzjnaJotfbtm18zpnU2xz84RrM/pNR6QCKEoypqyqimKihBudHsjCUBn1bsPHttZ2qal7xLPOfgU50uEGHt627Brvkkle2C1XnJ6ds5623J6eoprLl7h62mI2095/PEJH98/YDkz6NgSQ4cPFikldTmmNiOM0BhhmM0ncDjFWYPQASFjWje1oqomlOU0O3/0RGRa85wkREOICuccXWdpe0/bB9wtnUku33xeQ1Xeydak3XQDv1MUUAs4PCiZzCZorRI3s9MIJNHukp3abkOza9jtOjabhtVyjXUpCa9zgq0VPF1YPnzU0/62J8tr/VWT7OSOBJzUqUHhHRSWdC0xPWZyhayvkEESysi29SkNbrOh778Dxt+mst0qUWH6hmBbCD0KT8Tjosdby851NGKgk0mMkhRKEJUgeJF47jEkPr+RKKOQMrm9IAoiBVKpRDFTYo9BpJRopdFKobVJVDWuwbR3nrEpmNY1Nnq8iAilUCa5inTO0ztHby29zambwSVbQmXpQ8AhsELghcTLdB4GQGiNVppClVSqpixKtL4RGPEClch1CQw7Xo5FoSCNrN8uBbtXoL+6Ob8cwogWEZ5Y+HQBH7RwZwrzCYzrhEkf36rNwAAqIXVh5T5dcf/zQOruDt73e5FRPkADiN7TKAZwLK5BL6T/D5lLPISDDMJvlQE3RX7eG/QKH268LnUN4PdCutzlJSYADwkAW5t+P4TUHZZcc4tl5iHLyLV4Kr/eEEgdkpd/XRbAQ33Cj998h+89mPPmXDK9lMgmMAJOSJ3hD0ld4Cf531sSp3jz0l/h89U6eLaEtXs9QTG8kkjo31IxZA/Rry8pNaN6RFkWFIUh+phPoch0OuXo5A5IyfzwkKIoiDGmVLeuw/Y9PlrAESPsmoblckXTNNkmLok4gk+0iSSyS4libdPQ7HbpcQYBXiCP8Htc6OndbQzZ/vTaNg1Xm9SFbPueV//x2tCcP+Xzzx/TH48ZlRERO/p+h7MdwSbelfQCLRR37p7w4M371OMqdX1EQCiBMZrxeMZkNCMEibMerTT1uMb7CUI5rB9nUNlhvcBFRbwlbXWfb3rg5RnQDiqXVL0N6cL5dX9pDRyWMJvXGJO8VrebVbLAswrhGuyuoW17ms6ya1tW6xVXiyaFJUVoLKwaeHLFbwfFXyoHbCPM83VgoLKJBliDXm+IoytkEITSs95aFosNi8sF9rV0o/iuflu1zTo55TiLxFMajRmNcEbjeptDYXpsb2lD6k4ZJSm0ShhBpnXLe0uIjpgV9FKZ9GGXBVFmYKwU2kiMSdSmojCUpqQqCqo8kRYqUSOElEgjYTTCGEMkc3+lSDQiwPqQaBg+4kJIIuGQ+O7OO1zw9ERsku5iBbiYushRpoaGkgWlrhkXY+qyRIb03C9SZ/zhK3fBXlaG5+u9hG+Wyr8zJ02e3l85VnxzF/8I7CJ8soNPd6BOoVCpydncpg1A12ZQGLOmbnCPyC9i71UsszAtdyGUulYE7jm7YRAhZXCZQefgfDGI71y4pjOQO8YxpPsNorsBsMsMjAfQLmX6MA9t9pjpGCpeCwONTnyUQOqWZOZF4uBkYB2H95rBuRpoGdzoYL/8nVGJ4k415fBgzKwoKK3Dnzdst5Yd129z8OZ/xPX17Sm3F2P+h1YXkzXq61zfHDD+2hrGdopqNGU8HqFVyiYPQxdDa+pRzcnJMaY0TCcTjFE5ra2j7zt6l5bc5OkbWS4uefzoMUpK+j51gn3weO9wtsdZS/SW4JMi27nhQvLNcYl/V1mfooodIJQGWYN/lV1sR9g94fHHB3TbAw7mJVpa+m5Du1uzW61pN1uiCygUb7z5kN63jKYjet/jY0BIMIVhOpkxm8xRQhEDFGXJuBnR9FNaZymrMatVy2rVEqVBmVF6z7dQlsQj1KQ1TBWgC6i79MMupGN807F6GFceKrh7LJnMJiitcLZnt14RfUuwGhE6bLOjazp2u4b1dsdq1XJ1me0wgV2XhD7rP9QbLtcA6E2+PtgAtgGnQK06QrVAOkEsHVfrjvOzFcvF5nX8KH9Xv6Nc3+y7WUJECq2RUuGNxhlL1yp8CNg8IQs+IGLyFXf5gu99n9e0dPPBMdhlRakJ0iBk4iFrozFFQVmW1FXFqBoxqipc6SiLksIYtNape5w5y4Us91hIDPoCBqoEiTccZZrMDXaYweODp4+BnoAlYEXExpDifAFP4kCXuqYuJtRlhe+TJuRF6g85BSTwEPgzCW9Okijv0sJ/3n19d02QOnB/lrVlG1JX7vmkum+2IuAiuGFMdpvle0BD1Oz9MMQNlyRx46sgA+gMjpHXoHMAxjIkQDl0cfeWbLlTnAXx+y708LsR9h58WrMHxsgs1rsBXBHXIrx9yE9+rMEyrizz64vXgDyELOTLUaRxANzi+r0JrkV3LxkXKwR35Yx7R0ccHkwYjQpM7LFtYJu5xZJ0jRs+ixsSlaIhdYxfl8/o61SvCTAWVOMpJyf3mExnyW1CaWaTCX3XpghcH/ZuFNa2aCOZjCpKo1DRE33AuYB1HTYMOV4R8GzXT/jw/fP8eR1OtJu3b1NJZod3uffGW1hrMdUEF2D16DfE9gtenrvmlyou2D75BbvLOaezOaaSEC3etnTbLb5pIYgUzrHqKJctZQ9N19DbxOgTQjIebRlXS7TSKCEo60RNqMdjTHEJskw7352nGk2Zzw8pivJW3sIAjCVpLdYqCVbKKZQ9yDWMeuhjuq+WqXNWVzCeK2Z3pozGE4gR13d4BDH2ScEcLbZtabY7lqsrri4XrBae7SpNAkWRnH+EBvH7yMxfKg80XRLi9LmZIS20EcQ4EKo1ygli0XN+sebxoxW79avMtfuubqPKIvm8BueJLqSOrRAooTDKoFXiFtfVODezkl2mkql75kMW0/UdvU0BINZ2ifcbIy6KfZiRd5G+F8imYacU2+wbXxUVVVlSmAJjDEan0JAEkCVKKoxWlFpjtEZpA1Im1xkSLyNZY5KpGBGRNSOWgCPiiFgRMyiOhCAICIQ0aFNTlGNqU9I4/yWNyO3VIL6tSaPnN/PXMuOl0iUV/832g8j3v0cKU/ARfkOia3zbriovVGWdOrRF5ghrnTqu4iY94ka3VqlrkBvjtSJwAMbYzM/Nj7/vOt/gC9+kUexBqby+wbU7BFxzigeag83PEbLSJITczc7d5CBTlHQhIbp0Px9SfHXMqX7EDLgje8/krzhSvLxPQoXh+/VDfvzwe7z37jsc3JlTTGuiCPj5PcIXHyDoqUgbtzdJn9+KdLgGOtF39dV6LYDxaDrnz370Y/78z/+co4MjLs4vuVosk/dms0uJSCGlIHmXRnFSQV0XGCEQ3iZ/Yx+wvk/xoc99ICMx9P9KFivJ7Pg+9998G2cd04MjRtM5X5zc44tf/j1h/QF/eDzFi1SEuCS2K9r2nFZmmx0CuJ50yiWO8GLrKZYW08Bmu6Nt2+RFHCJlsaE0BYWWaCUZjWum8wlFVeOCpLeStgfnFMdHJzy4rxiN6lt5BwMwFiRBQCVBjqAeJzV/tXMctJEY0oJajQvq2QhTV0hToKoRZjwm+hS6Ep0jegPOQHT0XcNus2F5tWCxaFkvYbcF8nUjZvqGkPxRflCBZCsVuBYICqDrQawgVB3GLcH0nJ9tWF3479oC38KqqzK5u2DxwSGCQMp0E0Kk0COTNommKCi0Sd7skPy+XeoW930CxW3X0HUNre3prKV3HuESrcF7TwiB3ieamZASlR1jjCkw2qC0xmiDMRqTu8fGGOqiYFSV1GVFWWZXCqEIIvEvhQAVU0amEmLf9dUi4YxAtoiNaUYXAoQsnJZFjS5GFLrEyi55zt9iGRKwvQMcA2OZJkIqv76uzZMjn+4zJNFVJK7/u2M4GcPjS3jf/hsExZCAsRSZp5F98PIGaS+eUzdAMiIBzTD4zA0iuKETHEnKbXnNU74ZJ00GxdwAxtygLUSuecDO51jn3EkeurnOQm/ZO1wMIFkMNAwBUicLhZu+xwMlZOhMy7xzsn16POeuOc2DkO8llELy7vhN/tNf/jU/+sH3mI5rJuOaYjYGpQknDwlqivQXGNKm7vv5d3vS5/i7Vslvr9cCGCc3A01RVpRliRKS4D0+pll2ECloOC2cASFAa5kmHyHgvUsUDCWRhUKZPzTs4ltYAlRZYaoJQllqFMdRgSqwQXD+0Qx7+WuIC17NEh2BHYTd14CvtMO3K8vT0y1CaVyzS/7IPu38d9oidYfSEmUE1aZjtO2RuqDtA70VCFmjzQRTeo6soHC38/cdaBIAVcwd3AJkXWGqEWYSIQcUaF1S1iPK8SRf+CVBCFyUe96kj4LoDPiSGHra3YrtasF6vWWzhs0u2yQFkJv0fP0wyfsj6+u0M7sIegdqC0XsiTqyWXrid22Bb2UpkUGwkkgtiS6m8wYAgZSSoigSpaEo9nHLImZNhC/w3uJchfMW5zqs6+mdpbc2CeGyd3vqLjucTT7hIcR0Tc+dPh8Dtu9pu7QVE4LETZaSUmtqY6iqiqqqk5exLtPmUZt0/qDQSBRyb5EZJcniUQqIEpFBc3LM0AhpEKJACoMStxOva/LXYbx8REoBm5MSwXQ+F/NpCv7asWIC3M+/OwXmJRwdgijhbAGP+TcIiiEpgJFpFDYk2cXMDw4uCdRMTOA0DF3bTEMQWcC2T4oT14Ba3RC0ceOxB07xMDyImSNiM5g2Nn0dRHM6A/aYBYExsLd5C+EGJQIgpNc8GO0N7yXm51Qqd4X79F5MFtgNbAzyr+5zx19OGQyH02Nmx8eMDw8oC0NRFVSjGXVR404eEuqHdJsrdnh60uf3HtduLN/Vb6/XAhi73tI0PbtdS1U0bDZbtpsNQgqkUQhFWjwBocT+sz04VERrU0hEpdGFRhU5g/0Vegi+uhIgNFHo5H5jYDSVyKKkHE25uP8WH/7skO3nfwvh7Bt+rbkN2vW4Z+u02Lk+j68gvRdPkIKgBdYI2p1ltU4LW2hT0iDjI6r5iM4pAgXI6lZenSONkwJJZJMOrUaYEaqeUZR1cjQpa6qqRhdVSgqTkkASe253O7qmwVqH84HoCggR7xqazYL11QXrVc9mkzjFOzK26a7X0ttyTorAzkKxy+ElytK/hh6R39UfViLvNJUSyKhSZ9f5dI2WMYcTyZQeaRRSZ9+IGLNnsSbGlJQX8Xns6/Eh4KPfd4qdT2mS1jlsFvNZ6+idx3qP9YGutzRdR9/3dH3WZeQOrwG0EJRFQT0aUdU1RT2iqEaYsko3ZdBCZwtPmUJFdOIpS62RWmGkRkuFERIlFEpoBPn+USKiTLvJF6j/wA2qK6kzLEnnyzr//5dvjqzBIrlNKKAuoBxDWyQx7Gf+Ol3s31zZHkIWzO1jbd1111ZI8EXqIg9WbQMNYaBVDGBYqNSplUNet7imWQwd5ptCuMFRou9h16SfmSKL4/KdjSKluORudqYdowY6h7+mWRAh2hs0jgyUQ6ZZSJHe07CwhjJbuA0cZ8Fe+arVDU7z7VYpKsp6jDAFTiqi0ghVUBRj6npGe3Qff/wO280TFpyzJv15ZqTP86t2ovi21TcPjIXk+N6bvPn2u9y5e49JVbOZbmg2W6KICC2TKb28npS0XcfV1RVXl5dsl1f0XUtRVkwPD+l6x26z/lcKilPFKAjIxOGTCm1KRspQFDWTyYymafhos8RfDDrUb6o8+95mb0mn5k0ZW/6jBhIA7gQ0iqBNWhi9AFkhp4pxPaauapQ0iFv62EaumegOQIPQBpRB6BJdjSlGE+p6TFXVKG2QUuVJXdjrQoJzeNvjXPZZFWD7DbvNFbv1ht06WWJuQgLGXyX63F7ZkDrTyqWGhv8D52VCgyx+//2+q1dXtmvTMhYC0YXUzbVJ1S+EQClFVAqEx9pACC75su/lEzf0FCIiVcyTbYFAoZQgaEUIScQXvMcVBd45rPPp5gN9BsZV29F2HV3X0dse7zxh8HoPHusdtC19COjeoZoOU5TookTrAq0KpMjuAlJmUCwxRUlRFlSmpDbJB1kpkTjJA1jKa8WLniv382c82oRlhrFyZrY+J1Qa5lKDmegAySNZuNTARQMXHi7/LQfnBJe7rzJTJGTixgy+wJA5ujd+Zy++g313VpksXsv9+jDwirnmFg/8YrgBZnNnWatrXu/NMVwIWXWYfx+RKRs3x3U3qBrDeTOcQ0Na3iDgCy7TJMid4yFVb2h15Mdw8aUB411s+OzsMZNPDnAxMh/VHE2mqFCg54Yz5/hMSz4TkscxdYmH69wlycv4u57Jb69bBcZCGHQ5xdtdCnHYg9PfvpzNDx/yv/+f/o/85V/+Bffv3UHGmJxThMDaHk9IOteBgxYjbdPz5NETLp5+QQx5Trxesri8AAQx/OtmzwQfCT7mqVLqfUghKUyJEpoH9+6ye/f7PNl8Rug+/4ZfbUe6/Az9mKFXc7MnM4zKZM6l14nToMeI0YSD+YQ37h5zPJ9RGsMLOjZ9pTzJLgotQRdJrZ95ZELoRNMREAn4EAkhJjFo3ye7P58cTrx3KdWvC9h2Q7db0W4d7TZ1cje8GrFD03LNEfkDkIQwML5bUIy/Q8avU+02G4gxJb45j3fJ9gwxcI1l6hrndDk1xOhmjmXwKWEyRg8iJrCprjnKQoivuElJASr7CJcFhCjwCHyIKbDD+0S5cA5nLc7Z5ATUtZlSFLAh0nQdoe1AaoTSKF2gTQnyht2iAKUU9ahmNB4zrUaEOiJKhZYFWlyL90AyGHa+SPWGpLXimi4xnCIDKB6+wnWDEq59zxtg3cOz/vXlFIuUCk+/hHib1mxfW0PetE+3kGkVQqY1VcjExY05KCPkLq7Q16BV60TcJtwArQM4zdcJkW3SJOx3f0NnVuXo6YGjPNRAWt93MQauzI3/F1kbs99IDveN178bbvw8hGvBoHe5o8z175K72869tAadpeeTzcdsftnw+Okpx7MZ94+OaFY97X3L5xdn/GOz4NdxxyNSe2o4e3pub0r5r7VuFRgXZs79t77PervganFO6Pv8gXJc77vTLkqrgno048//4t/z7/78p7z39juMRzV92zCZTJhMJnR9R+8cQUSiFDjn6fqOzXrL1cXpNSjO9eV//2utxMMTOY0qorIhuVIKYSJ3jg4Jbz9g8/g+yyeP+OaVV0Nv9st1s2PwJcAcK4gKbSIHI83JtGRapajbeMu5xpG8WGS+WxyiSIeOFblTFSKRwf86pJQ77xNwCTFFVbs0tu77HbbrsG2aur1yBfAfuB7rAurjgvGdO+h69HJf03f1R9VumyZfwfl08wGfL+ZCiT1XVwqZY50lIqv3gw9En7rIKSkyJFCs5D4MRA6JdkIm2pqQGWyLPSAVQqOFRClNoUzCBaR45gEcd31L25d0vaW1FmEdvfUE6wkxEKzFI/EZIEUEMYq0dkmfRH3G0iuH1R6rAz6m8zGJCdPakGKmXwwchwxorLzB6OJ65fnK/blevbak7f2CxNH8Zt3sf3eJkMSDVryC1xgy2cSJ5w/iILgjc3hDTJQH55JAuSiuu8E69+mNzvcN19PESALaKqfj6bwBHIJC9m4XMvsbZ/A97Pq8z0B9eGEDgS13j3PD+tqtav/GbnCMh8fN/OO8+bwG3cPDxfR4ziWKiX95196A47R7zPLJhvnpjMfTI67WLU8uFnzx9At+fvWIRzS/M6Tqu/r6ul0qhYgUZYHYCaLtwLekT4vCmDFVVVFWJZPxiHt37/HwwUPeffdd7k1HiG7Htt2y3W1pNhucTeM85wNCK6RQWNdzdbXiyaPHuO7fLKOL0hhGRZG8S0Ma+IncfVEC5HSEOp5x9eCE3VmJda/rsfpdEwUHsaQSHSPRUdGiQ0uwBvsSNkBdhN6Dz7zJ6y62SDG1PmTAkL6XgDBfuViHEBK1wtkMTnJ66a2/4hcvU8LBnQoznyPKMVF91zF+nWq32RJDDiGyNgVjeEfMF/CECzKYFTeBsST6mNcHTwgui5Zj1jQlYCxkWlelzFxfIbMdWupGJ8BskLK4DvbIfNAEa9OGcC8C1AYTA3WIeB+TRimmW3ocgxBp4ymyCE8KSVkm7+TSJOF1jGBjRMaQaKviukH3wiivSH2amNHu4Kx180I40CqGp+tJG9tn+fZtGEGHDranr+jJ+i4BU+/StK8MiZAt3TXdQWb6QtOmiGVtwPTs0apSyVanMLkDrFKHWMi0gLoebAc2W/oU+pqzLG50afeOFUOJa4eMTCm6pnsMFmy5M4287s/gr2kZ19/MYD1/qAdgfJOOcZO8HvLrf6kV6Lji1K+5uDrj880zxp9OWbdLtvYyaQu+qz+6bplKAdoUSClhMPomIpBUheFgPuPo6JC7d0743rvv8u7bbzGpaoxrac53NNaxbVq2qxVd19FmSyGkQCjF1dWKx4+ecHn+jNd3r/6SS0CpFSOjcVIQvM9rT7I7UyIiCwmV5MFBzfmo4unqdQXGv6scIu7QoUG6LaFf47qSICRC3z6AsxF6mzpySayUaDshxn0DI42gZYq0zgu+GEa9mUsWYwox8Nn2KuRGyeu2PAkF0yPFwZ07hGpCKw3+Jaqov6s/vtqmIwSPsx3O9liXqAshpNRQAUgl9+BYiMTL3W/ccphG8Ndd44RrRQLGAyiWCq00Qso8+Y2pEy0VUhq0KlFK55tC6pSUJ/LnH5noF1IItBBp4pKpD0MeA0IjhCZFEqTfHWggWqp9cIhSSTgdRJ6wiBuw5DaWfEHKoYhpQu8zp8I7UOF5HjFkz3ASZeKMbwcofuUVsqAh3IhOFoPwLIPFIaaz66Gz6WC7yD4aWuUOgvVQVVCqBKalSI9p+wRApUpJRr6A4ibvGJ4jxojc1RVJ5Elh8s9CBsX5A7b3Tobn290DLULcsIMbRHbhBjjOwFhw7au8H36K/XXh5ZfH07ByDav1U/7N4qNbqlsFxkobRnXNbDan3R3RrJe4viHi2e5WONfT7rZ0zZaRlkwUXHlHt1oSEah6jFcFm96x7SyrnBjWtA27puH87JSr8yc4e6tB79+6UiJiREAQEqcVkDGgo0QGj91csnzyCZtnnxDbzbV/+jf8uv/YitHSNCsuLk9BFUyPHaND0NXtj/wD0PYR29s9PcLnKNsQQSiNzF6uMUQiEh0iSvdIlczsYz7SUgik1KmTp/x+8vY6lSmgrAyqqAhS433Evnbw/d92WS+JIWA9OVoZfFRJfJvtzRLIVPvO3ECvjCLsU3STd2tuVggQg68rKUgDofC5ixtiIISIECGbBki8dKgo0BFUjP9/9v470LZlLeuEf29VjTFmXGmnE+4JNxMFCX4iNtB+2tDXT0zYoiJc22ybaRWk1YuA2tqCAbRFJBta7Ia2PwRp0EuDiIKkCzdx0zk7h5VnGKFC/1E15pxr7bXzWnuvfc589hlnrjnmiDVqVL311vM+LzoZuyKRyjWbbZ5ZsQrVGtKqpWtkiIqGcfDRoI6ea41JBrHWJnKlTYboaBgt+uzaPA6Pgjo1hsGkyXQ1M39mMVatYphW0XbaqeEqiskTp6SdVrQFtuDF9Qte1BknmMQtVsljXCTvsJon4NApZ7XoWKmaNGppj9Um0KgaUFMocugU0fBtZVrno7FEtUhG7SwddFj4vjAdIa2hDTNJNwWgOUDXMC4a9U0dU9CGpLzRytD5RGIPx9nwL5J9FpnxR+G09TZPH47XY6xiStHhcIi35xhnhvHeHnU9wXtH3UzY3q2o6ymrecZQBfxoj52rV9A6Y3jhGcxwjQmafevZ3Nljc2uHra1Nxvvb2HrEk+fLPlnEgalDeYcKluAd0nKjghBczXT7JpsXP8TO5Y/i6oYOMUOT8JSlgAyOqp6wtX2LygrrwXCm6HM8Ym2HTgWUNdR1g7Mxxa6zFutcNB6URpmcLM8itQKFcQFtKpQy0TgJyVBRgtY5ymgkGcanCSJQdMDkOVpn1CHy9xv/1NSM1wdUTkARxBHEJSpDmDuoVPTcRs9uS4Nop3ZDCryL3uXg5/SLlmccub5z7/DMOnRxsB0EREcKhSjBKiEkCS3RajZrQiDpHkftY6VCDA5EFgzeDFEZoKLNFKKqhjbRMDbaoNLgU3SOMmaub9sGGCc1jkdBSMF3NoDPYp4Hr6NNZCBJjoHLIBRQB832ds6+y5hryixxAFolreJsTlvwCwoN7YBJqZQARMeRebZg0KqkYpGUlhCJfOS6Jg7o0nGcT3JpdfzsFNDrzpdWTu3ACEci4bqtTy03ecZTZk7HOKx+oUheX81ciUKIaatV5N8FS6w9Eo1416Q62waWPxqEnKFZJ2Q5zmimviI0ZVLEaFPOJGNYSXwebTmbNBBolTLaoEafONM2xCq9cIgljtkwLqf7bG1tUeSaleEqZ1bXyLSOgWLEBjnLNb1Oh2c21ji/OsTu7bLWG2KdRbo9xg42tza5srXNrc0t9vZ3aeoJp28y+skghEBVlkwmExpbY62dBQeIb/D1hK3rr7Jz7RUme7sp/1wU9+4QA0b2n+gd3Ad0hiq66KyHzvuYvE/IM7wSbCsLdQKoLdSVpanLlHa8wLkm8jOVjkaCNqgAWge0MbOO3+iUDpeYNSyYIuoeFzXGRLrdaYHWrdRnasx9iNPtJxgossSDY3XjWXxweFvhXU3wLZ89UhXmwXJR+izSelpjJBnHiRJEUvQJbecvJK4yM1oDkKgXftZHilKp3mu0SoZsUq1op4klCBJkZhhHKTkT6RVpP1HRU+iZ981KCVqpmYEf3zGD6JbXnM24zTLjeB5D7528xAemvRfQ+Jh4p7awaw2briZQ8RS5FB4vDlNxZwkxSDSE5MENbQZOlzy1KhpmtJ5c5nrGBGgmUedSJSpEkDndorGRylBbUNXcm6v1vIK1x7EN1CFKyCk1D5yz6Th1Ez9tw9xYljTjInPVCZdUNayPS1VFz3VTxwZeSTSKXZsg5FEN42iQq2IVWV3H5xprhEA3GcWtvrKbCzwZBVlS6jAqaQ4mo9jaxANvyzANPBoby6fVLbyXQ/p1gGM1jG0z5tq165w/d4ZzZ1Z49plneP6Z5zh75gyrq0N6/S5ZrsmMwniPcg3N3ohyZ5e9/X229ke8cu0qm1tbvPKxD1PWU173T+gwAkwmE3b2d6mbVmg/Rsv6aowd7XDr6ivsbV5j6j2O+G5kxCQWG8QhxuRJ3sMdIZD1yFfX6A2HmKyHznoo1UWZHrrTwUvMwnUSaByUZaAupzSdApMV5K7GhxB57kqD0tGR0E77tovRZC6L/YNShNDB5T1MMcZkDn1KyIlCDAbPssjvhMSL9jaqaixxanD2wkvRkE1eIQkGiLMTcVGJAikz47idkZ6RcpN9EtWnPN57rG1wLip4R+fZ3BMbQkhUoTBP0SzJeE51PTNZGlRFKFEYMcnh1hrHzDjIIjLzQLvkwfYhzO3SNtCvlWfTJhrGOouUpDYo8JikrwLpxIuUVJL2d4DaRDm2bQs7NuCetFNmRod5spdxR4TknV3MEud9Sq2cMsO5RY8l87+b5NHVKnp1W++x8lDvwXg/epZ1P3poGxuN0tYj60LiLCcjMEsGtMjc22ubaETOdI/T4tzcK11XcYlpdWOAn068ZGehaaIRXjXJMA7RoK7L9H6SqBitZTkfwD40+mcgywkrK9TDPrUCJx6UiXJCWhYM4NYr3hrDaUASPLMI8KaO91E1UNlk2Ot4HyZlDnQh3tsp6a+eFI45wYenmpbs7u7iXY13AVd7JpOSs9MNVlaHdHs5Ra7BWUIarQWERhms0nhRcaraL337R8Ozu7/LtVubVHVF01R41+Cbknq0TbVzg91XP8zOZMyUOEuiiBJDJVFy6FSqPIsG00cNV+msrtEZrJDlPXTWBSkQVaDyAh8CjT2Zjqp1ApRVQ7cuyV1DCD7S1FLQkdIaWo9xMo5jBjKDMT6GmiqD4Al5n7zYI+9MKEZQhidfo9sMrKLa6cQ2PkUR5Elf3RKLGK4/Q2TueqKv2CBkcYDGfFAjLBrGEp9t/DGlWY7Puc1yV5ZTqqoE3ELCj5YWEetAazxaHzPkwdwwVkajVOw6RIRcZxRZgRIVAwMTbaN1GooInoDHo4KLXvBI5IjHbVNEk04qCiUGUYYgmiCSHFnJ4/2oaGfGAwQN2PkMdGNht4FrRN3xcBrErk57gEioYkBcG8gWiH9rlXJst+s9syx3LV3B++SN9QsBfERqQ1VFQ86nkRbJw+wFMEmuJNVb5+f0CmCWgU635/LzY7QSb8HPDeOqip9aQ5aM9pYOUlUxaNC23mIXpxVsA6FhxkWYGcWt+1wfLqkHw3PnoCjorq3SHw5xWmhUwBpwBkKmCJkhGIVXcdwQgk8ayklHuWnidTbN3EZ3xHJMjh7EtiPnZBg/2mW/FnDMhnH0Qu1sbrF5bY+LH5UkFN+ll7SJhys9hsMe3U5BJ8/pmJxcZxgVuZlNCPQHA1aHq9zavnk8DeFrCoGt3S1evX49ZqCqS2w9oZnuM92+SrV1hWZ0dUF2LCKmPjmF7auY6BEoekinT94fkvVWMJ0Buuhjsi6iCkTliMqxHvwJGcbex/ajrKCqarquIYhPmUrjlLLSOtI4tY/TwK23TmmUjslAtM4QCXjbpyh6dLoTehqmdp5340lhPtEuKcA6RMmszKDDIzbkSxwruoMNgGQUC6IMkriMIQjeR0m2QEh1ca5LDERvsZJIZQBwDl/XeBS19YRgmcfttcF8MuMhi1L40NC4Bi8hpmlGI6FVwYhcZWN6qLyPUTopYcQltNeN0KajDiFO/UZjP51TqRgfNfM4S7rraMQ4Ypprl+ggj4TEMSYZxcEnkp6ODrfawVZojeIl7g8jIjG7iEvrLbVppDHLgCeRw2VMDNSbUblIrvoQvZZV8nJWQDCJ1zJKBq0BlSTdjERvqUok8cZCqKN3lDAP7HOJTztLEqLmKaPr5EWtm2hMmmR4uxCNYWdhOo3e1taoty4ZxYnGEC1N5mRdH881i359SJxbJev1WD9/gXNnzqI7OVJorIZGe6wCpwUrgSo4Km+p6pq6qrDTEj9NHvCQOrZ24OIWDPzWC17ZJyC2f3pxzHJtmk7RoalGeDvBW0cDTCewswMkb1pvuMLKyoDhcEC/26NbdBn0+vT7A0IIZJ2ClZVVtve2sbOctqfSrHsCCNR1zd6kpKorqioaxeV4i/rmdZhc4061+3SVngHTgU4Xul1M0SEruuRFD5N3EdNBTBEXnaMkB3Schj2hl7cBbo5gt4Kr2xX9azfpD8f0h7cYDC+SFznGZJEyZy1NVTOeTKnKKY1roua2KJQyBGdpminlZMJoBKU7HW1OCDAZQ91Y9re30dmYIAq3HIKeOuhOP5kYrcEaKRStc0e8R5Jh3GoCt5SEkCLioyc5GsZaORwKVTeIqfFex6C+BcNYLSRNEJ2k3zSRo0yc0RNRBFEYpUEbVFags04cNNqo6CJJxioa2yoZwg6VMvFFCkVLtUi0ixCzSkqSOQRoCRetL+6R2QQLM+ltHF/r4AzENmDCaWsrTztuEg3BIi2d+Bl0DFSbFXoyFKVVj0hGWp14ucEx5+EQPZ8upAxydTyHLiBLdId57vOFZBuJRxt8MprVXKVCkhHdpnCWZLDPUkAnw1H5eTKPpoGqBF8nCoNOx09BgTM+f7u0hnLrnn14w7j/wjOsDIa8/IaXePHZ5yi6XXRhKGmYuIqpbyi9pXQN46ZiXJWMJxMm3jOtajwhUSia6PWeVtHInzZQNlCWaV0yik8rVecJ4FgNY6Vz1tdWKXJhe8tSTjY52MQEvLdY29DYmsZWTCuPdyWBGjHEKTolZJ0u2uRzw1gUs7zrr2sIpujSG/TRdYY2Qi2WYAvqVsT8VKKdxtJxxJ91odNFioKsk5PlBXlWJB3s1Pm7gFUeTQCdusjbshMdL+rkRBjVsLlfwfWKSERJd9G2c6eAFvGwCB6aEpqyhtMwVbzEkQgpo6WfGcaxI5/lTFAB0SkArv09RfC32eJiUo9obCjtMVrokBF0D+vqGFyaOKIxSUgysNMUtVaKXJu5eoXWaK1itjqTkZmMIs9ROcTpZBel3lScUo/X6mPOhaDTfK9eeHdkTh1OFuoCmSNtErWRj+V9a6t7cvKJjXFT3sV3v1zOJD8E2kItie18BqwQs3yY9L2Is4MqGceNjZze6QjsLlEtujUo+yBJRykE5sZmMvRwoNIDtMmaazWPQ4gc4yBRnF756JnO9Tywr1VtUIA38ZhlMhZb73Y7fWgTlQM992y3UnC3LerQ56N5jN/+to9jY2WVT37j23jrG16kKArQwuZ0j6ujTbYme+xNx+xOxlS2JJQ1zWhCub9Psz+C0RjG4+gJGU9gMo2GcdlAaeNS+1PKrXyyOGYqRaxUWabpdDqUE8VhNQmlDb1Bn8FwQH/QJTMqcu+NRIFsJagsw+QFmSmoqqhZHD0ly3BJEPJOj8FwhayuyDJFrT3ipkw7HexYE+VjThNMzGVv8hjQYDIoCsgLTJaR5xl5FjtZZTKURB6vT5HDMuvsk0TOExzaHlP8zxJL3BO/+IsfOCCaIBLSwkx1ipaqcAiRTSHzJBwJIcTgu7pusE1FY+uFoMs2410KxCMG7LlEi2h/N8agM0Oe5WQmizrEiTc6S5GeaA8tPSMgeB/1l2dSsa0Rs4C5gka6guQ9VErY291lN049PjSS8zvGI/lIgVXJeVg3KefDI53h9Y5ANJS3iebFgmEcCmjy5LG10QvrR0QqRlvykrbtMjcwW2dIoie4DMoc6iydMixoFMvCFEDqJ7I89j0z5YsUaNEqZrRUgyapTvikneyTF3tRdsPC3DN8uKa069v+6dFUKV5+6SXOrqzxtje+mY97w8vkmcEHz8Xt60x9xaScMnbgy4ZmNKXc3We6u0eztwv7e7A/iobxeBKN42kJ0zqleWU5ArwLjtUwdnbExz7y84QQ05AeZcCsrp/n0z7tMzl3/gyrqwOMBkJMc2otTCYVnm329qforOUrhZn3ZAmh2+uzsrpK3dTUVY7tajqZp5lssz3dwo9PU2bAFCghOk5F6Qx0DioHlRGUIaQsWSCzVPNK1Kw9Ct4TbExmIrOI3yWWeG3jK7/8zx/4LosSCizMXtzhfTj695A0hz3BRyP2XpiTbJJ3uk0/vZBxb2a+tFPSab+50d56huXowWU49GXx+0xEwLG/vXXP670beulUjjjL7xS4Nm6MSAJYJ8ZkLGeWHwWtl7c6evUdEYie53vIIhzm1dztmM2D9Benpd8ElRm8wKScsL2zjUJomppLt67wsasXuXTrGje3t9jc2WZ3b5fJ/h52tA+jEYxH8XMyiQZx2USD+GizbIlDOGaPMTh39/TDq6sbfPzHfTJvePF5zpxZxxgIWMbjMdtbu9y4cYu9/XLOByJ5CE94Cv3pgdDp9hgOBzS2oakNrqvoGE8zOUs92mR/ujmXkHniaCVrdJxKbQ1kNc8WF5Ck3hPQai7jBIlG5oEU1x5mgQ1LLPHaxvbNa0/6El5zMMlO0ylwXycaa5WC8IRoGI+Ik/tLvBbwdNoNla0Zl1NubG2iKkuwnrKc8pHrV3j/lY9x8cZVtrc2me5s48eJNjGdQDmNHuJJGekSS2P4gXHshvG9cPXSK/z4j/0n3vimN/Lc888xXOnR6WbUVcX2zg7Xr9/k1UtXefXVi+zv3GQ+FHzS8fynBQFnHdY6nHM45+P3FA2+6LF58kje4lnQBcwzD8XfQ6JMeAkE5/CiUdqDT1m6EvdRSZyODeFpbeaWWGKJJ40Pby54jMP875JoCJdEyuVylnmJJ433/OIv0TEZF3srrOZdgnVUVcnlrVtcvnWV0fYmYW8XRvvJK1wm2bkmpqpeVuKHxmM3jKtqj5/7mR/lg+9/LxeeeSPnnznP2XMbiA6MxyNu3rzBxz7yYfa3rxA1Apc4jLouKadTrLM0TYktS6bTKeW0pKnLObfqiSIFYUgegy7aeV1p/5d4jCHgXZjrlIojOA8qRNoYoJMkWkBwIdItllhiiSUeFB8Icx0BmA+yX/vzkUIkkkBUs1/itOOVn/rPiCg+YgqMNtFxZBvq8T5htBfpEmUZZeWaNs85r/WK/Fjw2A1jgOArxqOrXLqYxbTmQJZppuWY3b0Rk9He0ii+IwLe+egx9tFrbK3FWotzDu+OCgp4Ekj0iTbvPSwE8C5wvlJMQ/CBIGHB4x3SLm0A0ZyDHB6EMrbEEksskfD61WAxRKUIiH7xpcLTaUe4eYOAUCtDjZoHBZZTmI6jUbxUlTgRyCMLpi+xxBJLLLHEEkssscRrAMsopiWWWGKJJZZYYokllmBpGC+xxBJLLLHEEksssQSwNIyXWGKJJZZYYoklllgCWBrGSyyxxBJLLLHEEkssATyiYSwi7xJ57YpnichfE5EfEpFNEQki8s4jtvm89Nudll/9EOd93Zdr2q4nIl8tIh8UkamIXBSR7xSRlx/inMsyZVam3yAil0WkEpH3iMjveYTzvmbLVUQ+Q0S+WUTeLyITEXlVRP6piLzxiG2ViHyliHxMREoR+XkR+e0Ped5lmcZt/6yI/F8icjXV6Xc94rlf9+UqIm8Tkb8rIr8gIqNUtv9aRD7lIc+7LFORoYj8SxH5kIiMRWRHRP6ziHzJI5z7dV+uR+z3xakduHTS1/ioHuNvAT7rOC7klOJPAF3g/3+XbX6GWAaHl/cC14CfeojzLss14luAPwf8Y+AdwP8EfA7wIyIyeMBzLss04v8A/nvgbwC/CfgPwHc/QiP+Wi7XLwY+Efh7wH8LfAXwacBPi8gLh7b9GuBdwDembX8S+B4RecdDnHdZphF/EDgPfN8xnXtZrvDfAP818B3E9/+PAeeAnxSRT3+I8y7LFHJiOo2/Dnwh8LuB9wHfJSJ/5iHPvSzXBYjIGvB3iDbVySOEsFzusAAqfb6FKKz7zvvc7yWiPPPfetL3cBqX+ylXohq9Bf7aofVfkPb5/Cd9H6dpuc8y/bVH/UY0pq8A+knfx2lagHNHrGvf7b+6sO48MTXnVx/a9keAX3jS93Galvst07S+rdMm1dt3PenrP63LA9TVsySZ1oV1q8A28J1P+j5O0/IgdfUO+/9H4D1P+j5O2/Iw5Qp8M/BvgW8HLp30NR47lSK5ur9WRL5cRF5JrvLvF5HzafmXIrKbpsX/wqF9z4nIP0pT55O0zT8TkeePOPfvSq74Mk0Hf6GIvFtE3n3EMf/Xhanj94vIH7qf+wvhoVPI/V5iKovveJidl+UKxDzSGtg7tH4nfT5Q3V2WKQAtrecHDq3/QeDZhd/vG6/lcg0h3Dxi3SvATWDxej6f6DX67kObfzfwyXKPKcIj7mtZpjxS+3skluUKIYRbIVkaC+t2gQ9yqPzvB8syvSs2ecjEzMtyPXCezwa+BPgf7nXs48JJZb77vcAvEqdpLhBd4N8JDImd8jcDvwP4GyLynhDCv0n7bRDT8nwlsZCeA74c+A8i8nEhhBJARH4D8E+Bfw38WeJU0N8BOsQXnLTdCvDjxCnmdwEfJXZi/1BEihDC3z+Ru4cvBX4mhPCLx3zc1025hhD2ReS7gD8pIv+JSEl5CfhbwM8TvXHHgddNmTJPd3U4AViVPj+JSK04Drwmy1VEPp7oIX7fwupPJJbhhw5t/kvp8xPSeR8Vr6cyfZx4XZeriGwQ3/1ve5Dj3wOvuzIVESE6c1aB357O8/sf5Pj3gddVuYpIlu7pb4UQPiTymNLePqJL/F3xEAfWBWIBmoV1X5/W/08L6wxwA/i2uxxfAy+kfX/rwvqfIFYOWVj36Wm7dy+s+0vEyvDWQ8f9x8CtxWu8x33eN5WCyAsKwJ9cluujlWu6zm9K27TLT3LEVMyyTO9dpkSedgD+20PrvzWt/8plud71Xg3wo+ma1xfWfzNw7S7P4vcuy/TByvSIbR6ZSrEs1ztu+0+BCfCWZZk+fJkCf5x5P1UDf2xZVx+tXIlxRR8COun7t3PaqRR3wf8dQlicQnh/+vy37Yr0+4eID2cGEfmjEiO6R8RpiFfTT29Pv2vgM4D/PaSSSsf7L9zulfkC4D8BHxUR0y7pOs4QPTnHjS8jZi//Zydw7NdbuX4tcQrlfwQ+lzhaPgP8gIj0j+kcr6cy/SHiiPzvichnici6iPx+4Hel349z6vq1WK7fCPwa4EtCCNsPsN9xYVmmJ4PXbbmKyFcSg8X+eAjh8IzHo+D1WKb/G/CZxICybwH+voj84Qc4/v3gdVOuIvIW4KuIdbN8gOM9Mk6KSnG40tR3Wd9pv4jInyBGKn49UY1gm8gl/cmF7c4CGXF0cRjXD30/T/TaNHe4zjN3vIOHgIgUwH8HfH8I4dZxHjvhdVOuIvKJxGjVPxBC+CcL6/8TcdT8B4C/+6jn4XVUpiEEKyJfRBy0/cTCdXwl8A3A1Uc9xwJeU+UqIn8D+EPAl4UQfujQz9vAmojIYodCnL4E2Lqfc9wHXk9l+jjxuixXEfkjwF8jehu/9X6O/QB43ZVpiNzZlj/7gyLSA/4XEfnWEMKdzv+geD2V698D/h1RMWUtrcvjbrIGVCGE6f2c50FxUobxw+KLgR8JIXx5u0JuD1y5RXwY54/Y/wLzURBE8vsN4E/d4XwfePhLPRJfCKzzkEF3J4insVw/OX0ekLsLIfyyiOwAH38M53gUPI1lSgjhvcCnStSC7hMHGb8t/Xxc/OJHwakrVxH5KuAvAH8ihPBdR2zyS0ABvJmDPOPWa/Lee53jhPE0lunTgKe2XEXk9wL/APjbIYSvu9dxHyOe2jI9Aj9NnEG+AJy49u498DSW6ycQ44qO8s5vEx1jf/pe53kYnDbDuMftKgS/b/FLCMGJyE8Dv11E3tV6aCRqML6Rgw/vB4n6rq+GEI4aBR03voxYub7/MZzrQfA0lmurV/irgF9oV4rI24A14PIJnfd+8TSW6eK1fSxdS0bkxv1QCOHDJ33e+8CpKlcR+ZNESs9XhRC+8Q6b/SCxQ/k9wFcvrP8S4BdDCIenIR83nsYyfRrwVJariPxWYqDdt4QQ/scHPc8J46ks0zvgc4ERR3tgHzeexnL9Yha83glfQeQ8/w5OcLBx2gzjHwT+goj8ReA/A78O+KIjtvsrRL7k94rINxOnAN5FNKYWeZLfAPxO4MdE5BuIo5g+8HHAfxVC+M13uxgR+VxiVOYzadVnJH4OIYR/dWjb86SozGOcNjkuPI3l+mNE9Ym/LSLrxNH3i0Qy/i5P3iv/NJZpyyl8hahb/CJRAudF4LPv98ZPGKemXEXki4kR2T8I/Ds5mMVyL3nfCSHcEJGvB75SRPaJSX9+Z7r2L3yguz8ZPHVlmrb9DOBl5tKMn5CoQAD/JoQwueednyyeunIVkc8B/jmxbf32Q9tVIYSfvb9bPzE8jWX6h4lSlz9MNNbOECmVXwR8RQih5snjqSvXEMJPHrHvO4n19N33vONHQXiEyD3uHDn5tYfWvTOtf8uh9e8Gfnzhexf4h0Sezj4x8cAbOSIimRgw8AGiTNIvAb8V+Fngew9tt058iB8l8m5uEI2uP30f9/duDioizJYjtv0z6bdPf5QyXZbrge3OAH8b+GVgClwkBji8fVmmD12mX5uOXxF5Y98BvLCsq0fe27ffqUxZiNBO22rioO2VdD2/AHzRskwfqUzvtu3Ly3J98HJty+AOy8eWZfpQZfprgH9DjNGoiLOZPwz8xod5/5fles99T1yVQtLJnnqIyBuI/L6vCyF8zZO+ntcKluV6/FiW6clgWa7Hj2WZngyW5Xr8WJbpyeD1WK5PpWEsIl1idOUPEzm9bwL+PJEg/okhhOOMrn/dYFmux49lmZ4MluV6/FiW6clgWa7Hj2WZngyW5Rpx2jjG9wtH5FJ+I3G6fUx04f+O18uDOyEsy/X4sSzTk8GyXI8fyzI9GSzL9fixLNOTwbJceUo9xkssscQSSyyxxBJLLHHcOKnMd0ssscQSSyyxxBJLLPFUYWkYL7HEEkssscQSSyyxBKfEMBaR3yIif/YJnftdIhIk5vl+3UJEPi+Vw6+/j22DiLzrMVzWUw8RebeIvPtJX8dphIh8u4h87JiP+XKqn+88zuM+TVi2aY8X91PeC+3r5z3qeR52/9cKln3V7Uhl8i4RORU23Z2QnsfX3sd2B/rN43h/HgSnpeH8LcCvJ0ZDLnH68Vk8+RSXSzz9+BpiWs8llnit42eI7eaTTg/+esPrpa/6PGJyjq/lYCKOpxV/7Eme/LQYxvcFESlCCNWTvo7XO8IRGWmWWOJBEe4jBbXElNU2LKOETw2W7fCDI4SwB9yz3VyW7fFi2Vc9nQgLGTCfBJ64211Evh34MuD55CoPIvKxBdf5bxORfywiN4mZuu44BXvUtLWInBORfyAiF0WkSp/fJSLFXa7pC0RkJCLfeNqnJh4EIvI2EfleEbkhIqWIvCoi33NoCrCX7vtWWr5bRNYOHefA9NTCVOIni8i/F5GJiFwVkb/6Wiq/e0FEvlhE3p/q2S+JyG89Ypu3p2ewIyJTEflJEfmCI7b7XelYpYi8R0S+8GmhZYjIW9I79tF0jx8RkX8oMbX34nYH3uMFGsQfE5G/KSJXiNmX1kTknem3zxGR70vv56aIfJNE7c27Xc9nisi/EpFL6Xo+ICJ/7fB+qXx/XER+vYj8TKrHv3iH5/gpIvKvRWQ7HfM/iMh/9UgFd/x4o4h8fyqrV0TkLy++j/dTFxfe7U8SkX8rMc34v0y/fb6I/ISI7KZzfEBE/vKh/Z+GcjoufPyd2j85Yip4ob79JhH5WRGpSJ4yEfmVIvJj6f2/LCJ/CZAncVNPAvIa76seoI08ss2XaCN9e/r7XURvMUCT7i8sbPusiHxnKqNKRH5BRL7k0PHa9vXXiMi/FJF9EbkuIl+Zfv+CVEfHIvJTIvLph/YXEfkzqQ2oU5l+o4isHH378lUyb4//HxH51Pu57yMO9NtSuzVJ7dj3iMiL99rvXjgNHuOvAc4Bnwl8YVpXAavp778P/ADwe4HOgxw4VbKfADaIUwy/AJwHfjOQp/Mc3udLgW8B/moI4Z5cmKcM3w9sA3+UKN79PPAODg6Q/i4xXeTvBt4O/E2ituGX3cfxvw/4VuCvA58P/CXitM67juPiTzMk8t3+GbGMv5xYp/8ukBHTayIizwE/TkzJ+ceBXeB/AL5fRP5/IYQfSNv9BuCfAv8a+LPpWH+HWP8/+Nhu6uHxHDF9958m1rc3AX+RmDb1s+5j/68Cfgr4Q8RUy+XCb99NNMz+AfCrgL8M9ImpUe+EF4GfI6YT3Qc+Me33JuCLD237ZuJz++vEd+TLge8RkY8LIXwIQEQ+jajt+bPAHwQmwB8BflhEfk0I4b/cxz0+Dnwv8G3EtK2/Cfhq4nP5tvutiwv4P4F/AvzPgBeRNxHr578C/ioxJexbiWUKPFXldFz4Ph68/Xsb8PeI/eBHgC0ROQv8O+Aasd2tgD9HrMevF7zW+6pHbSMX8S3AG4DfD/xaYhkAICJ94EeJ6Zv/YjrnlwDfJSK9EMI3HzrWdwDfCXwz8DuAv5YGG+8Avg4YEcv5+0TkzSGEOu33dcBXAt8E/F/AJxDr9KeIyOeGEBbpHV8KvEpsdwpi+/EjIvLWEMLW/d60iPwRYprrb0vHGBKf34+KyK8IIezf77Fuw0nnnL6fhSPyXxM5M4FD+bkXtv/YEevfzcEc5n+VWEl+5V3O/a50HkPM8NIAf+BJl8kJlPHZdJ9feIff2/L+jkPrv5FomMjCugP51RfK8CsO7fuPiR3v2pO+/8dQvv+ByB9UC+t+NQv534H/BbAs5LUnGn4fAH5mYd1PAL94qMw/nfvIJX8al/Ru/dp0/b9yYf2B9xh4OW3zM4v3nn57Z/rtfz20/qvSO/62Q8d45x2uRdL1fAmxIzyz8Nu70/v/1oV159Px/+LCuh8B3gfkh57j+4DvOwXl3b6Pv+/Q+vcAP/SAdbE91p86dKwvSutX7nIdp7qcTqC879j+MW9fP+9QffPApx7a7+uIA40XFtb1iQZieNL3+xjK83XXV3HnNvLdHNHmAx8Dvv2I+zKHtvvjh+tdWv/DwA1Ap+/vTNv95UPXdIPYJr5xYf0Xpm0/N33fIA7evv3QOb7k8HNM328B/YV1L6dzfM2d7vvw+wMMiIP5bz10zjemd+dPP8rzeBqmub/3Efb9b4CfCiH87H1s+w1Ej8oXhRC+5RHOeVqxSfRI/A0R+YMi8tY7bPf9h76/hziqu3Af5/iXh77/C2IF/qQHudCnDSKiiTMe/yosjIxD5Ld9bGHTzwF+MiTPY9rGAf8c+FQRWUnH+gzgfw/pTU/b/Rfgoyd6I8cEEclF5C9KpIJMiY3ej6Wf334fh/i+xXs/hKPqmCJ6j+90PSsi8j+LyIeJDXgDfBfRSD78HvxyCOGX2y8hhBvEzuHFdKwu8LnA9xA9pyZN7wqxs/mc+7i/x4XD7/IvMvc63rMuHtr3cDv8c8Ry/Bci8kUicn7xx6esnI4LD9P+fSyE8HOH1n0W8dlcbFeEEMZET9zrAa/5vuoY2sj7xecAl0MI7z60/ruJM5GfcGj9bKYohGCBDwEfDCEs9j3vT58vpM9fTZyB/+5Dx/oXxMH35x5a/29SfW7P8zEi//5BPOWfBawA/7RtW1L7cjFd3yO1L0+DYfwoaQjPcP8Rqb+L2HH88COc79QiGRq/Afhp4vTRBxOv6Y8e2vTwVEZLN7kfGsv1O3x//kGu9SnEWSJl4vD9c2jdBkfX52tEg2F94Vg37nGs04y/TvRgfDfwG4lG629Lv91PPbrbO/8wdezbiFP4f4/4DnwmkTZw1PUcNZVXLWy3QfR6/iViZ7a4/HFg/XFyFe+Bo97lxfu4V11cxIFtk0H9+cQ+5LuAa4nr13aCT1M5HRcepm4e9QyePeJYRx3/NYnXSV/1qG3k/eJu73n7+yK2D32v77AODrYlHD5PMqw3jzjHner2g5R9OxD/YW5vXz6ZaPs9NE4Dx/heOMpzVBJHKIdxhvggWrTcpPvB/xf4IeAHROQdIYTRA13lU4AQwkeALxURAT6F2EH9A4kBUNNjOMUF4kh/8TvA5WM49mnGLeILeZSn4gLwSvp7i5iH/jCeIdbzbWJu+ob5i3/4WK8+6sU+Bnwx8J1hgaMvIoMH2P9O3mKIZfBLh77DHeqYiHSIMQXvCiH83YX1n/wA17OIHeL09zcRuXi3IRzk051W3E9dXMRtzySE8O+Bfy8xkPmzidS17xeRl3ntlNOD4G7t35362qPq+lXu3Ja8LvA66Kvut40siZ7RwzhsbN4JWxztgX5m4fdHRXuMZ1hom5MH98wR57hT3X6Qsm/tvHdysD9o8fD8Yk6Px7gC7hpZfgivABdE5Fy7QkTezO0V4IeAXyUin3Ifx/wlIo/lrUTj+EE68qcKIeLniIFdcHzTR//doe9fTCTrv+eYjn8qkaagfwr4IjkY9f//IfKnWvwo8KuT4dBuo4HfCfxsCGEvHeungd+eOoV2u08n8qeeBvSIxv0ift8xHfuoOuaB/3SH7Qui5/Lw9bzzYU6epgB/jNhZ/0wI4acPLw9z3CeAe9bF+z1QCKEKIfw7YlBOn8hHfK2U04PguNq//0h8Nu1UdRtE9Zse7fKePryG+6r7bSNfAd4mIjNHoIh8DjHQbBGtt/ywHfWjwBtE5LMPrf/dxFnJ45BF+0miF/lwIPPvJA4I331o/TtSfQaiGhGRjvEfH+CcP0E0ft9yVNsSQvjAA97DAZwWj/F7gY00VfLTHIxCPwrfQ4x4/G4R+Xri9PNXEj13i/gGYgX4YYnZVt6Ttv3NwB8Jh6IWQwjvkyin8++BfysiX3B4m6cVIvIriFG8/xuRN6SJxoElRkAfftEeBn8wGYY/RZxm/QNET93uMRz7tOOvEAdi3yci/4jI3/pq5lNWEOvjO4H/W0T+CrBHlGd6G3E67fCxvldEvplYZ9+VjvU0eNl+EPgyEXkPsa79NuDXHNOx3yEif4s06CWW1Xcu8oIXEULYFZGfBL5cRK4S24j/nkebMv2zwP9DbCP+CdHDdxb4NGIwy1c8wrEfF+63Lh6JFBH+OcQo+ovM2+ArREoavDbK6UFwx/ZvYYx7P/gG4rP4IYlSXK0qxXF4Sk89Xid91f22kf+CqM7zrRLl2d5IfK8OX2dr4H65iPwA4NLg89uBPwX8HyLyVURq6e8hUlX+cHLEPBJCCFsi8reBrxSRMbFN+HiiEtiPczsXfEqs23+L6Lj4amL78w0PcM49EflzwDclB+kPEMvkeSKn+d0hhH/2KDf1xBeil+GfE6fvAjFg6fPS37/+Dvv8FmIDPAV+nhho924ORXASp6S/mdgo18RG/DuAItwhmpPoNb5EHMHcMer6aVpSOXwHUe5rQpze+FHg88PBqM9ff2i/d6b1Ly+su1Ok7ycRBxVTohH3NSyoNLzWFyJP/QPEjuyXgN96uE4SZzW+j/gSl8TR9hcccazffcSxfpYjVFpO20I0fv5Fep+3idJzn8khtQjurEpxmyrMQj38HKJ02CjV4W8Cukcc452H1v0A0cNwgxi9/hs5WiXgx48498e4PeL649M93kjP6BJRvuwdp6D8b2vT7lDe96yLdznWZ6XncDHd/1Wiw+LtT0s5nUB537H9486qFLfVt/RbK3VXEqeY/xLRgAhP+n4fQ3m+5vsq7rONTNv+YeCX07X+BFGh6ECbRBw8fFN6z/xiPSFy1r+L6BSoiLK1X3KHsnvLofW31VGOaKeJcQl/hthn1ak9+CYO2U9pv68jSsddSvX7x7hdmeXd3EWVYmH9O9Jz3Et15ZeJMnyf8CjPR9LBl1jioSFzgfEsRML9EscMEXkD0bPwdSGEr3nS1/O4ISLvJAbRvTUsKCksscQSS9wvln3VEveD00KlWGKJJRKS1NXXEyNubxHF3/88cUT8WpQSXGKJJZZYYolTgaVhvMQSpw+OGOH7jcSo3jaQ6XeEEB5FvnCJJZZYYokllrgLllSKJZZYYoklllhiiSWW4PTItS2xxBJLLLHEEkssscQTxdIwXmKJJZZYYoklllhiCZaG8RJLLLHEEkssscQSSwDHFHwnIoeIyhou/EbYeDN4AyED0iKACiAhfUL63/xvISrWLfKfRUCp9DnbgCjZ55ln1pS4jaR9RMXrER2/IwvbtKLrAUnHkvT3YTn2ENrLUfEzxOOEtD9+sQgkXpN4kED4yb/wQOru8XYPlylELeyGpyPHw8kihPDAZQp3KteI3/x5v4sv/Z1fivU13jd47/HOUU4rJuMJe9evsHP5VcQ5BoMh/TPnKJ5/AX32PGPvmNQ1dryPHe9QXrvE/isfJlz+KL3NK3TGuygXkADTEEV4x0SZCUd8O3LiE+6IRuVDfHfIR0bb/JQdMQZeys7wSZ/8q/mkz/os3v7pn0FvZQ1UBtYTyopgG5z3BB9wzuFsgwsB7xzOO6x32OBpiEr5Fo/Hg/dIetf+yld86THV1SVaPGxdvSASHFHoMwBnEc6zQocCjUbQOAyBDp4u0MXoHJPlZFmHPO+iJIdgCEGwztM4RxXquLiK2tfYYAkhxNoQYtvXNV16eZ+u6VFkPbKsh8n6ZMUA0xtQDAb0hwN6gy7BTmkmO0z3Nxnv3KSepJxIIihlUFpjtEZrjajkiwngvSeEgEZQqFgHfcBZR9M0eO8QEQQhhMDWdIsf2fw/sSnJ18OU653q6mDlDH/z6/8RX/iFvwXnPN6DKIUoRSBqmzofsN7jvMe5gLWOunLUtaNpHHVt2d3a5NbNG1y/coVLFy+xeXOTyaSknFY0tcc2jrrx1I3DWnBOUdcN+/sjRqMdgr9ElICtjrrMBSjy4gXe8tZP5lM+9Vfwqb/yk3nr297M+WfOsrGxRpYZjNEorVBKxXIUQUSlru9on9iF1du6vocu0yUiTqKvWuLhy/V+cEKqFMK8mzfMjGIMc4O2XRZw5H0GZo7toJKhu4gjDOP2M6SlNY5RM+NbWgM5nWNmEEdx5yPvqN2qtad9kHSEaODHQEZJ24dFa3qJpwCiQImQiRCUgNJYCdS+wZcjqv0dJrtb5EqhhwOKTCPB461FaU1WFIgK6DyALwl2TDANnb7Q2clQdUNoLGpaoStLxwVWiEaqT58lsB0ce9UO29UuVwmM0vV5YzC9LkFrGueorY2dtycN9DQoDeIJAbxKRjFgg6d2Hhs8FklLIKY9EoIPt72OSzxZeOYtqUqfQiDg09A9toXxuydgcV6lRLNCCIKWADi8FxprqW3DyI/ZD2P22GfEHpYST0NIA25B061XGdbrrKl1Vot1Bn2DyQYYk1F0CjrdDnlRoE1GXU8oy4rxeMxotE852YstrCiUMmitUUrNFhE1d34AWhRGFCoQDWPnsU2Dcw5Rc8O4sQ3hJCupzP0vACKxxU/ZEtKA01M3lqpsKKc1k0lNWdY4G2gax7Ur13n1ox/hA+9/L+/9pV9g88YlGmvxzgM5SgpEeijdw5g+Wd5DJAO6aO2xfo04ZN7k7g4QT11d5P3v22Pz5nVu3rjFzZs3+YRPejtvfsvLrK4O6fV7GEzsnpSglCIWukAISOtAWmKJJQ7g5AxjydKi02kM0XObDGJZMGZVMlIXbeW5zZqcvGr+OWOAtEbzQgMye9kX3/hkFC8YxK0nYn7FntYHLHdofMPCr+GA4zoQJCx4uuVIr/OjY9GYX+IkIBLSwEehBJQH5RrcdEy1v8No+ybdooM6fw5jhBA83kXDWGc5kgm+A6JqFBUUgc7AUOz0kbIiVA35tKKYlLhpha9qmqqhsg2j4NgnsElMCXSdEG0c0kRL0aFY6ROMYlqVhKzAZKBQ6EA0ioHgJRrFEnAEHB4boPHQ+IATcCp6qi3gfcD7cLJGxxIPDLPwGZdoFoeZyRyfV2wRooPABwtewFqgwQmoED2cU1szDhM22WaHLaZs4xkTa8LBZz9ilym7TP2EurQEMWT5gJ4S8iyn0+mQ5znaGHwIVHXNZDJlNBpRjkczA1hrg1L6gMdSKwUqtcSiMEqTKY0EQULAW4e1Fuc9Ku0XQqBu6pOvoypNZqrU8Uj0YnufPMS1ZTKtGI+mjPan7I9KppOKEARnPTeubfLqq5d47y+9l1c/+tOEsLdwcElPsg9sIHIWY85gTD/eVYgGcvx9j3t7jT3ebXP9+s8z/o8j9kf7WFujFDzz7AXOnj1Dp9PBGIPJDJkxaC2pK4z97TxT9eGOd4klXr84OR1jBbG3bn1hLb2A+KkS1UBxhBeYAx6FA9QHNfeUzLY7MOOwSJEg/R0bOUn0iraRnjM4JDaGPh4reo4PXk6A2EDL3DAmeYXbTIczT/OiYXysHuPW4/7I6c2XOALOR8qBDgHBo4In+AaaEl+NGe9tcfPGVfq9Luefu8Cab9DEqWCPoBACCiRDd4d0zgV0r0N3bZ1itA/TGlfWhLKORnFZ4cuaZjRlurtH2BuxV41QfkygPvCUtRjWz5xneGYDGwI3bt2iVzb0+kOKrMBgosEB+BBw3mO9x7ZTwInt4wN4mfsYA8RtrcX7pWF8mrCKwhNY9JN6fPo7OgjUgboXvYJaabTRGB2dASF4mqZhN2xzg1tscwXHiLt7JGscN9lhl1HYZDQp8SEj763QI6CzHGUMguCDx9qGuq6pqoqyKlGiUEqjtUUpk9hmPvksYvsbPcjRKDZKxzbXBYL3OO8ggFIapQTvPZV9DIZxCASZU0qCV1jrqWvHaFKxN5qytzdmd3fM/t6Y/f0p1bTBmAytDNNp9B6X0ykhTA4fnOjO3wH2CeEWTXOWplklzq4KMWt5w4NN39SM9j/Mz//shPF4j5s3r/OWt76FN77pZdbX1xkMBgyHQ4bDAd1eF600Soc4merntMTFYdYSS7yecXKGsQTQieYQwgLfN60TD+Kiodt6g+98sLlhPOMGt1btYUN4YXsWDysHd01/zL56idN2M9rzwYappVKQLje0O/o4bR1a6sTs9gPq2GkUGdH7XbOc9z5+RMPYosQjwYO3iK2RpiLUE6ajHbZHt2hsj+l0H29r8uBQgAvQuIBXCpRBd/roPCMfrtDbeIa8agjTCjdtoKoJVUNIRnIzmqA2d3Fb23R3N+ns3SKf3sCECXXyDvaLFc489xzDM2dwApvb21SNi4ZuN5CbHKMzIFZD55JRHNpPcKjEpPezafPI9Qw0yUO3xOnBkB4Wx4SS+gD9TBAMCkNAJ/NYgVIoY9DaYHTkl4YA1jrKMGWLW2xzGcf+A1xFjeU6twhk1YDe/gb9M2dZCT7OlngiXcdZbFNTNTUVFRIU2mm0M2gVt7XBpuFYvAMlBiMaowyZaPCB4F1qSyOtSSuDUgrvPbVr2dYniTDzyocgBB+pE9Npw97ehM2tfba29tne3mN3Z8R4NKEqG7qdHkXRYTq1EBTBW+7uwHBEr/AY6KWljSEp77HvUagpp6/yvl/c4eaNy1y5co3trV2ef/55zp07x/kL5xE0WmdkuaCRhcnX0P63xBJLcKKGcQqUk3BwWWzgZ3TgcNC4jSsP/jk7RrJIZ58w51swo1LI4q6EOVUizNdKyzkWSXyy+SXTHnbxiiQkg1gIijl/WeZe49n0pvjjN4xVD3xG9MBPbr/AJR4JkUfoQYdoGLua0FQYsXSN0C80/UyRKXBNRTkdo8uSrKrxVvDK442AEYLWBFXgdU5dKIIRgrGEvEEai2oc0jhUY5FpTXe4j1vfY3XvFtO9W9i9W+SjLcauwiGsnznHmWefozsc4kShbSDvFGhjQCm8aFxoOafglY484xDwKhrKLnkg8XFkpwAkoIJCgr5tlmSJJ4uKCoejTpSagugjzsjI6aDo4DEEyQgqB52hdIbSBkQRvKesK0b1lGvhBjd5Bcf0Ia4k4LjBFf9fGN8YsVXt8sJ4l/W1DQaDPs10RLBNGky6xFz3uBiRgfIKj8PjcLiZ11uFBhs0xmscesaflvRPBw0u4L1gg6Ph5A3j1ij2iX7S1IH9vYrd3QnbW2O2tkfs7IzY3Z2wtzdhf2/EdFqSmQmZMdT1GO8FrTtEJ4a9xxkd0Us8Yt5rPcgAVTMPQg84t8v1q+9lPB5x6+Yub37zm3j5jW/kzW+u0DrHmIK8AyYL0RGk4kxo210vhaqWWOJEOcZqZjRCy2tKvx2mQtzGnZUjVrWGdJjvLgc/W86wyHy1ACrMlSaicSyzywrttbaX2h5uRpE4eFlq5nae3VD6zaXvbTDgzEY/PvRWwfag9On8rabB8WNtPaPb0QTr2dqpqZt77/O0ow2wQQXEO1xT4eoJOjT0cljpF2wM+wTRWFszHu2jeyPIxzTKUocMXxikyAiZgFEErXFSUJoMCIh2aBswAYwPZB5M4+isloTJmPXJNjLZYlDtcaEZM21KSlvT6ffYeOY5iv4QZaLhY7IOed5B6wwRgxeTpjaieYGo6PUS8MrjJLFTfUBDGl8GTAho2vq7xGnBPg2e6D/0QJ9AhiKnoJA+ynTxkuNVBigCOs3KCc4HGufYr8fcCLe4weWHNIpbBDybbIf/wmjnJlvjTV448xLPPfMcuViwDVoCGo9ErZM0y9Yabu3icEkPRVBYVBqwqdQz+OjZTIxqwUMQGmpqypOnUkiacQmeuglMp5btnX1u3Nhjd3vC7m7kFo/HJZNRxf7ehNH+CO9jO2xMQCtPlvWI3eu9DOMWiwE2D4JV4kziPjBNx6gZ7X2E975nmxvXb7G7M4FgWBmu0+uuUNSBrMhjnK4CbRTaKERJOtYSS7y+cXIe45m12npzFz3Gi9sc4Sk+bFEu8ojbqZ82cngW5Db3JCtRM5/w3Dhueb9yYEwciF612e7tXkKc2lvYcnYZoT1ynIuaBesJzBQybrvXR4VA3oXOAOoSZtPex28c57nwyR9/nnNrPWgaPvixm7z/I2Psa53aLJEvWZU1k8kOk63rTHZuYSd7VONddrauMZ6MqJ2nefVjbO5N6V3ZxgyuMPUZZcgp1lYYnN2gvz6kvzYg7/UJRiOSXjXROBO5voEYoKSyQDBdVNEl7+V06wJxq3RCSdVUlE2FyQydlRXEZOgswxiNNhlKRRlCF5KyRIsgEBQiGlEpoFDCrGrq2Xg14L3HOIP41/oDfrqwydyN0JKoDEJGTlH00cWQoHMaK4wmEyZujzElU6aUoabGMmbEhBGO+piuqqLhI1xpJtgbEwoFG8Me3Qy6eU43L3C2SK4BhcYkqkernqFSex3wKfYkirWp5FaIxnX7l0t/RYLGyc+StWxb6yyjccXuTsnmZqRPjPZKJpOG8WTK3t4+29s73Lq1yc72NmU5pWlqet2C4aBDXVuijkh5glcrxHmEITFobzctrfe4ZG93zNbmPjdvbHPlyi207mCKLM00BURDUWQUnQytNR/3/IsneL1LLPF04OQM47BImbiTUXzE3wd4xO06f5DjoJJBLHMv8IwekdQm1IIXOTHwUicz8yu3TLJkLyzwkiVKHYkSCD7JsEUjWeY2dDxyUOkvEHykWIgnSBvadIwwBvIOdPowaUW+IDa+x+fSffZcwae+7Tzn1vvgGvqdwHhS8soVx3HGZwmJYi7zvxWAijXFRondYz3n3aCUxjtPub/P7s2rXP3IB7l1+aPY6R623OPm1jZXxyVjIIwvwfXriPplgvSYoqnJWB88z4svvZkX3vIyz7/pJbqmi/cOp5uoFpAo920QXFCglBC0wmUZQQok7yNOIcFgbE7HdmJwksmxaUwYiIZwkJBUJfxsrBTZPoJGRSNYBC8kMwN0CKhE7wfw1mOc5qFlITv/dTrxof3bGZ4ZR2m2MjUFiwPKwzsuLhxsOg7Uh/k08izMXhZnc2T+2wJH9/bzyjxwYBZAcASnKrTn8vM2rj1Oswn1ezkunfF94rPuAQMgI3JDM5VTdAdkvVXEdBhNasb7m1wMl9nkFhVl8tmeFByBq2z6jK3ddYb5s6z1h5hBn3o6RAWPd7F4gheCCzhcciCERKcI+JnZG0sshY4mqoXD0prPjoqSCeMTvKeEEK+laiy7uyNuXN9la3PC3u6UqnRY66mqiu2dba5fv861a1e5deMG1XQPZ8dkeZ+VlTUm4z3mAXUn1YC1JZej1Soi61h3g0jL6KLUKv3eGbQqGI8qLl+6zmRcz4JvEQ8a+v0Ow5U+eW74db9qaRifRuQmevIjxcfdsUbN57IFkn410ioOhRl/f9FtGELABz+fIReS/rWO53J3aUtOsHorAa0VSgTrHHe7jOPGCRnGh3nEdym5ttOEmThFWPQQL27HfNtoBLeG8VxiTdL6RbaFpP0O9oMtpzjEkCSJ3jUlCiUagiSDWKVPZqoTiwYxPnmnQ5Qi8snA9gDqmE1jY6DIoVNA1QFXEQ1il5aHrzmt3dIrhLe/tMqLF1ZY7xtCA5M1w4sbils3HPuPaH+LQC+PS7+ATINJL1fwqamXaBSXNUwrGFdzs/8QgeVYYXQM9HG2ZrS7w5WLr/Lqh96PbyaId+wG2CaGy9SA9Q3ONwT2Zh4yu+1Yy1eozpwhjCvUqiOEGpybzTYEYba9eI0KUS3VK4sz4DA4n8UECEbQPkOJJhiDBZT3qOAIXpBgCV5wTghhXuu1KLzEuhtCwPsYYY8PsT8M8zw7zjm0F7x/yJL91C9Lpz00oFXEEygB3f52F6NzhkWjdp4QInrB0+dspWcmN9bGIKhUS2ThXMHP91loc2anChIrXvK0HzSo221aY9ilv9uRSDrv1Q/BB1+BBwpuuzdaEa8c0CiMMhSdLnmvj5gulZ1gCYwZM+WwEsJJwWO5zo3JFdZ2O6wNC4yCLM/odDsEF41i7wKu8amOaYSWaBHL1s9kMuO3lpvc+pctDktDRZU4xid8VyFgG8d0WrOzs8+tW9vs7pTs79dUZUNdN9y8dZNLl17h0sWLbN+6TFNeJ3pqp9Rln83qbBpkFsSnd1ze+sMwQAetegwGqxijGI87VPUErTKKokOedalrz+atXayFq1duxkQ/3oEKKCMMV/tsbKxSFEsaxWmEEuFNL7yIVorpZMrO7g67kzGO0Irg0oriFlroFgVZ3sN0ekimwQS8anBMcb7GO4W3GvEG8QYbLLWf0PiYEAqt6PdX6fSG7O7tsXtzm9Acsi0ETCF0ipxgPb62NDaqHx0HOpliOOyysTak08nYvLXJza0J1f0ykx4RJ0ilWOz4ZMFrLAuWzfy32arUn91evq03pzV+BfGCkoBq9/eH9Ynj6GjWn6b/R6dSSI6lkPpthRGNlgwtOa2ofOt9E5lJvSdPSAwK8S5GLs8c5K2cnPjZlOGxwehoSWodO3+nU7nMec0PChG4cG7AGy6sIL6il8Obnlsnw1OPRtSjXaqtG8h+Q889DHEjozXatYI3nO/y4vmCXuHJlcU2JU3lKadQllDV0DRga6ACcfEIC4PZEzOMizyj2ylwWYbygeneHlvlGIUnT3ehiN1dzmINFywZgT7PZed5ZnWD9aJPF0E3dZR8S/WilQlMg3aCSCxPFyWqLJZGLLU4yhDNBq0NaI1ViiaN7J2LcnLiHSE4nPOEoGelpIkj7RCijrF3Dmsd4jzaK1QyjqNMoSAuicg8DLpF/FQLw5aZDSxpKkDNpwcW38/Fz6PYR8LcEG4T9hxoWpLCTbvzbApCDlaUsPiH3F6JZo5n4aDhzcLs1+Lfi0mF0jGLLsdZO/vEurZO9Bh3MGgUWmuyzJBlGZiMPM/p6x5Du8Ieu8nX+jhQshle5dJuTq/Q9DqGuokeyZjlTuMFlHiMz/DeoRqVVDsFhcYlrZQ4C+KThODco2yxNDRY7In6wFs455hOK/b3xuzs7LG9vcNo1DAZN+zs7LK5ucmrr7zKqx/9IJO9VyBsM2eBA1SEMCL6+dsneNyGsRCHSmfIzAV6vVW63U6ceQoDtM5RSQ6vriybt7bY3xuRX7mB0goXfHxGuSYrMtbWhuzvjiiK/Jivc4ljwWwsLwvNWmTv90xOkRnEW1RwFFkcEBVFERNOZZqgHU5JDH/1UJYWa2ucbQheQIHJQStF7dJck4qzNvMZskNofQ/EGU9tFCo4lIMmHLnHkTjK4awgDrK1kBlFnmmKjqEo5jPJJ40TVKWAltM4dwCFBY/S3GM722Vm+x5OjpE6q9kGiZUmggoBzVx3eNHR3CpFtDJr7aRdpGE4RHw6f1QhEB8gOLy3SJCZypzWBqUVyZFMEI9TIQnBq1iRksEStY4jYSPe+jEbxiYZxTOtndYwfojDGeHFN6zwaz/9Tbzp+TNMJ2NsPWW1n4FtGO1tsX/9BtuXRlRbYDx0eAA9DAHTLxDfEKqKC+uGT3zTKi9e6KGpCXbCeNSwv+eRJlb4qgZbJuM4zLN/Zdye4/C4oY2ik2fQLejlGeI9lpD4nUInDY0scYQugCZD0wcGiKxyfuUcF1ZXWe0UFBJQto6vwYKRGOKOMXeAB4LHWxf1hHXASqDBUuNjfdMKpRWNAiVRo7iByMewqd56Fq4qzWCgaNP8eu8JzqMdZJ7kbU7yhP523e4HglFtC3lgRmeuPa7mxrEc9vknQ3PR3px5f9vfWTCKDxvGwjz5T5gbxIunObzP7NoWzt+W4cwjnR6OdxBs7CBUm1lQQBabznRwbe40qn8oXGCeDqKLkJOjMRilMSnlb9CKLMsYFEM27Dq77LLLNif3liwiYLnGVZ9RbArrRR+NJVNCoTtkSkWDV+lUxAYRwZU+eYoT7S09eD/zG8d/Nv1zJAm3k78dbOOYTEp2d0dsbe1w6+YW43HFeFxz7epVLl58hasXP0Q1/jDMclIeRk1sJaKP/3hhgDXgDEav0e8PKYoclXhRxkTCTYgaotjGU9UTgh/PZj5j+yNknYxOt8A2juACRVEc87UucRwIBOq6RolQliV1E4NyNZBlJj43r5FgyUyWErnoaCqkgOvY/CoEDb6maSrq2uMsmFzRKTTaCBI8wYP3FmcbrLOzenPoovAuYBubHCyt9c4DNT2a2916sfmODs/WkWmMptNRREl0w5G5L44RJ+gxZsFr1HZy8xGPpBKceXcX+qpZH3bbl0hXAIUKEhfxqQ9Mhdh2su2OCwk9goSovRkavK1woSLYGu8apnUgVAE/bgjjiuCS10U0Ki/iUhhUniGZQUyGMQXGdNE6R1Q0WOXgjR9rcy4qEDSgVfQazyZSHvws/UHOr/ykZ/mklzd4w5mCYTal1/fYQqNoKMcjbly8ysUPjHj1KlwsYzDQ/SgoayJVoiig0x9H1YM+nF91rOYTaBrGkxHjSc14FJiMYDxJyxSmLvpgbFpakshJd41NPUVj2Vgd4J55hpeefYHJzVvkQdPXXZxtmNopTbCpqzYYBhg1JGRdpOgxWO3QyRu0lCAVXhXUoqgSJUdcsq1cqq/eRYkrG5OLWJ8mAggo5UEUYsAroRZmElgaPXsndAiYYFGJhBXLKSpSuBBw3sXMdi6gPYiPVKCgFBqFJWAJNA8bxGmSQT7zCDN731Oe7WQctxKOKTXtohe3NYpbY/WAtOMdPMbALIHP/JVLbvD29/S/GX94cT8WPCJh3vgE4sisLqEaQ7kXR2vdPnSHkPdAFcSsnvMSP+AxPwYMk9FoaDsLTU5Mw0ya0fI+emdXhysE9zxSKq5yjU1uPRbqAVSUfIxfbm7RaQYMGbAhq6zn66x2V1DoNEALkGg9rZYxyWssrYrPwr/D31X6d7II1I1lb3fCrZvbXL16g1devcitm9ts3trm+pVX2d/5GMHd4N4xHZ6oU3x89SG2922gnWBdyWjkmUxNyjQY63mbcTDTBpNl5LrAaI0xBqU1yijEaHSmyfKM/qBPvz8gy5dUitOIAGxtb+Odo2xqGh/JRhYYlSW2adDJw6qVIQNsU+OaBmUEkyskA5QjhOgg8c7jHDQWgvJkPnpvvI9eZRcszlucs0f73Tz4GsrGzS7yYfrno+a2ZpaTRJpcEI/ONN1eh65k5Fk3SlKeIE42813bSSWPrah5ENzcOJ47gtv1LPZ/wvxLiFH8kmKYhaQ2EfyC99cveJxbEyEa0t47fLA4N8U1I2wzxtdTfD2FqYVpDbs74DaZRxMbHF2c9ED3YqfY7UGvh+qv0Otv0OkO0aqDkjzZ5K3L6nibRaViKt/ESo9eqzD3Et4vBmtdPvsz3sg7fu3beXYoTDavUk12MdpgtVBOp4z2drny6pj3XoSLllni2APXw+3vTAY814Xnnk22jgtkCjoaBr1A5kZM94XNTcf2NkwShaJsoLIwDfFcbbfzOPxeLapyTHAl6/0unQsXuP7CS9S3RuS+oG+GuLKhGk0oJxPKZooFsqyP7vegmxO6OXqYYVQFjAkywNGhJmOqheBSnXUBrV00ip0F5wjO4r2jSY5K0WCUIiSt0aBC9CCHFDIRVPL4ClmAjneY9DRicJ/gvGCdp3HRMMZHr3/ULVZ4Y8g01BKPbR9Wd9u0ihsyr4oHEvKoBSpF61U+1JK29u8ibYHFv9tVrZHbnof5wHvRMIaDDcr8wuZ/h4VBdEr9Oz+lh3oK4y3Yuw71PqyegSwkOlMnjnAWy0wdr+HWpQM4fMqCqFDkUmBMRiBmLAx4tNIMhyt0sg7dUZ/18Rku1pe5xEWm7B7rNR0Ni2OHMbtM6FCF8+ja0DM9jI589xDaANGWLhFHQW0AUBtQevCfmhnFpPs/SQSgrhp2d/a5fnWTSxev8tGPfJTLFy+xv3UZ11yFB+Jwlzx6D5CmlyiYJwIxxNZ4inXTtlNIyjdZ4hcLKsvpdAqKTkGn00lpvAt0ppE08yhK0ekUdHsdzAkbG0s8PHYnoyPt0yp4GusxQM8oMmPwPuBsha1qjBY6/RxTaLyJ9IjgYk/uHVFpyoFzsf1zHpwD5yxOO5xzd+2Ejz2HGanGq7jEwEEwmaHX71EUPQb91UgjO0GcKMdYUtT9bBaZVi5trivceoph/nlbWzJbn/YTH2kUkLwNHiUpAIdo/Hrv8C6OeLy1hMZhmwpfT/DlPpRb0IwBD1qhewM6qz1KB26vgrDPfDyTQyjAdsH1IAxAhpA7Ml3QzTsokyEq4BzYpBZwvHVG0ErhtSa0HGMRYqCQ4X55bN1+xmd8ygt8xic8wwvnOnSlxpqYABZrsXXD3q0trl/Z4+KlwCUb/R5H4Q4DSZoaqt342LwH56OhWxnY3/S4ADt7sDuFykMdYjNv0108aELU48Lu1iY7t25w7pnzdIxm0Omx2luF2qBCBx90MmIbHCUOi3EOaSzkiT7jwNaWpsoIzZDgujgB5yUaUu3chgdJgV4hRH6w92BDTIPgNUlOScA5vJqXd1ACyqJ8HPCF1Ji19mDUYVW4IPF4wWOdwztH4yAKYmlqCRjx1K5h6izWPyQ3VeuDnuL2hb+bYdxGWrYIJLmNA5wHDm60OFqerzqwsGAUL8waLaxc+GwN4WQYqwXDuBEgFaytoKliT0LyfrezNiHMr1k/+CD1bqioaZ+6JkWUt7xxlzq44KLfVWtUUTAMQxDFeGfKtt9hyj5Hv6kngUBgyg7X6YQOw3JAP+ujsjirFb3Fc6eILBAkDl/hoVDpmZl8YlcewLrAaDzl2tWbfPhDr/DeX3w/H/nAe6gmlyHsHXGV93Xkh7wiQ6RiFMyD+NTCkqF1TpYdXIzJyEyWsh8asjwnL3KMiZ575xvKyRTrHS4Eggi9bo9hPSAzS47xacXdap4n9ptiPXnjKIzFVRVVWUbnla0wHU3IBafBWXdgks4tNMU+gLVA1eCaElvax9YZt/E7/S6srOb0hl2KIsNohcm6iFasr53h3NkLdDrdE72WEzOMRQJ6wVsTnUkhcUfavjN5DRKpfN67zT1FM8+vgKg47akSNSJJ2QM2HSvSJHA1zpY01ZSmKnGjdo5+H5pdcNsQdommmALVY7j6Jl5+6Tm2VgLXXtml3i8Tv9ARR/4VMIUwjWoQtsaQ08/XGfQib9kTqAMEF6I2sgoHnFuPCq0VXhusMbFjnhFDO8zNyrvtD5/wljP8us98iTc9N6SbNbhqGocWQdFUY8q9PTav7HLxw56L+w8eX++ALQdszRl2LsSr8wCSfB0hluiilsYR/sHHiq2dW9y8doU3DAr6QcjEUJgOVRkoa0s1aSinlomvmTDB0eA8FFOPkhoxGShNEENWGny1BrYPoggSvb9KdHRMeuKgJrRvhY/BRtbifMArH0fLyuOVIyxOqyiXBpQBkcTf9D5GFBM9by7E/GPREeBwwdLYJkYQY7CSYcSjRdO4hso1uIfVMc5aj/HC0o54D1Ap2lFwOyW00NzPmBP3emEO/X4Ue6E1isPi9zsgJONWJV3A9vy6pX0sGPbaxCgVk0X6iFZzwxoi1/oYjbd93AF/oSVOhTrnaKxFW4tohxKdVHEEk2d06NAZdSjqgjbW4XEiULLDJiN3FkOG0TmQVH5C6xxRhFZN5FB7385EzYcv85m/k7tmaJxnb3/C5cvX+eAHPsRHP/geqvEHOR4pzLZfuxcyWlY5dFAqR1QePe6uleXU5FmH4XCF1dVV+oMB3V5MSZ2ZDBEdPX7WxVlapbDOUk5LxuMxe6N9RpNRHCwHYbCywsbGGTpF5xjuc4nHgTbmZrHFroHKOrrOYZuGMlU3VzpM45AsJs51hybMWjnU1pFlLbgmUFPiq8enkdYB1nqwup4zXFul6PViv6c1eVHQ7fd47rnnePnFlxkOhid6LScafDfXEoboQZaZVu08MUeYG8Vt398GBSWJtbnKUzSMhcgVDt5hXY13Nc5WcfqgmeDLMb4cEaoRVCOoJuCSUTszx1o4dOZ4+cVzfPav+iSm5YRXL53j5o3r7GxvMdrbo5pOaeoa0QaVFUhWoHLDcE1xbk0zWFFYB3UTEBfSlHRL6Ti+bkkrgzcGmyXDOJXfYinfDavDnLe9YZ2Xz/U4O8jIjaeqIx/Nec9oZ4dbV/e4ecWztQU74eF8JBaYhDSKZe4Ntsy9w0/SAL4TnK2oRyOq/RGZ0uAsSsAFR2kttW9oxM/E8SyOkhLnPXldkDUe5QzaeXRToaopMp2gnKDyQDAetInSaS66SAVFEINXAesDDVWc3rKxlILITL5QSVJUEZ9UVTQ6DY4sHufndSHmH2vl2mIQWfAWH3xUvRYQ7xAXO03r6ln2rgeG0e1pjzCOjzKMmVGjbvMa3xNy+1e5ffWMIhEO/b64Xevtba+lLb8gUGTQ64FfBWWhGcLKmcgxLjpxMKAOGcb6eA23ihShnS67omYaSqhKfFliVCcme9EpiYYkxYo8pzAdirqHwuBOJDtm4onf4aGV7HKTm2iXUbgumTYopdHao53Ghag8oWb9QDSAVVI3Dkm0TacHqGhVgU4G3numZc3u3ogb1ze5dvUG1WST4zGKDXA2Has1bhcHBW0FNUSDuAd0EdUlLzoURYcQAuNRlGU0JqfX63Hm7BmeffZZev0eEKk11bSkLCvKckpZVok+6GmahqosmU4nTKa72Ha2FKEuzxK8p9vtHcO9LnHSUEDHZDjvmaQ2WyFkRqNN7Mudn4si1sTuRNISUragttnyHuo6qlU2FVClCb3w+IziDFgxsLGiWFnt0Rt2UXlOVTfY4MhyQ3/YZ319lXPnzrCy8pQaxkIyjA/NsCqYKTjN+6hwYL9oECu0kuQdjhQJxBGCjRGTrsY2JVU5wU7HuPEujHeh3I5eYd8yVe9thvWHPT7lV7yVd3z+52Ayxa1bN7h67QqXL1/m6tVr3Lxxg52dXUQptDH4EI2O7nDAxtkOeS9jWmpGE3B1oAyOgE+RmsdoGGtDMBlVZqIxIjA3O+/egGstvPH5DZ7d6NKVhpycXHcIBjIzxVvH1vURFz/iuXkL9pvYMT/wNRI9xTAPoIP43GuOJtufFhSiMC5Q708Yh0BdlXgamlAzdRanwXdApgbtchyOCkuFZWAht5rc66g76zx5WaPHU4wD4x1O13id4YPCekVQOcbkIBmNEhodsFbhggJrEZe8mAKiwox3pVVAJRHioCRNpfkDXjZBkuEZjQ3lHDo4PJ5GYCoeHyLv2TtHsHYhm+IDwqj2pLcbxq0xvCij1uJAdO09cJQ3+SiD+LYNwtFGcbuiNY5nbnxJtkoGahV6XWjWwFsoenHRJkVQtheWDmzS/R3TK78QFgjAlIaMEb7u46ZDCl2RSTGjlkXntmBQ5HlOd9JFk+Ee6k2+G4SYirghpiG+3fAONNziGgVdVt0qJovBYCJgrUU7PXMctGF1sabGRNFtuJ1OyiptXryTgneeyXjK7s6Izc0ddrf3CfdFLWp7NZ0+2z5ncYs1BoM341ygaSqsmxDChNgiLoYWBxa75BAEpXO6vT5aCSF4plOh2+3RH/Y4e+4ML770AsYYtre3uXnzBpcvX+LWrWvU1QjrplEqMg3lo7pAO7M4b5ldZdnbLWjsSektL/GoaGuFBzpK08k7MdlHOSWI0O8P6Pa6iG0I5ZQQwmzmZbEP1s1CM7zAViunRGWotko+ZvSB1T5srHboD3tknQwnwjQ4rIszH91Oh16vS6+b0+ucrILKCVIpkmG8SPNrZ4IP0CSYUylaIzq0Dy5EveBQ40OFdSXWTnH1BFuO8OUIN9mH6R7UO2D3OagpeX8wHcXZ8yu8+c0vsLG2Qlm9yM3r13nllVd49dWLXL58hVs3bxGCR5RgnaV2lrw7ZLg2QGUd9kYZGkU9bZgkPp0/rh4yFhLKZDFyODdzPWN7KDL+Duj3c156foMLG0OGnZxu0SHv9AkuDj58UzPZd+xuwW4ZecUP6ytpX8YmLa3X6zR6iRfR1TldZfBlQ9M0hKaC0FDZCXvllMYGghMaSipGTBkzocLhWPcdwmgF6GOkh3Ry2N+PHtOqgk5ByAq8zvGS41ROSIOboBWlV9QhJhjwgZj4IwXmRQ59wGvQykeZ7KS8EBI1pSYqSywGtyoEQtI59n7mIqgJlCI4BBcAF/WQHzaSQjI1p/4uGqGLPOPEiZ9TpsLtpzvSyA0HPh7i6hau54hz+MBBz3FarzPIc/A9CCuJbqE4oERxOJunPl4qRTvj0nZwUwKGCcIU1dQoZ9HOEUz0vSJRkskoQ5EXdFWPzHeo7ygr9rAwwArRKL5zT9owYp9dbGhQImRZFqf1bfRiSuLXa4l1NQSLQ+OwSeHY0ubB0yjmDOXjb0m8c+ztjdjd3Wc8qWJMgKykWUbLfAC0KJOpiRPAHVAdRBtCswXcZG6OCNqscebseQiaqrJMJlMm0zHWVgRqCIvaOwKiEZWjTYHJCrQxZFrR6/UwRqFVy9mOs5KNbRiP96ND5+pHqMpXiG6N++kHHbCHrcdU7czPEqcOuTIQAi54lCh88Cil6Hd7ZJ0Oa2fP0uv3GW9tsl9OITDT3m8jFWZDLweStQ7HCNdyHZ+AUZwBawWsDYVBv6DINaiAdQ3O1lgXZ0rzPCfLTHSYqpP1Zp+YYayVQqsYMRT7neiRaR/OXLlJUGqRZxynmYNzOOuwTU1VjWiqPdxkByabMN2EZid6hUPbMD98Y1nWJZt7W2xt3mLYLRgUPcL6GepJCbUjR7PS7eN9bMCsq6l9jSl69FZWQA3oFAalYDqF/f0G7wJe3Qdl8r4h6E4HKXJUZvBFHqd6mwp8Seyk7mzKrq/0OLM2YGN9nfUzF1gZDjB5gbMe1zTU0ykiAWPii7TLw/lnWppB6z/JmXcj7W+nFcNiwFp3QAfwTYMKDsEynu5yY7TJ2FfUeEpKxtSM8UyI5bROxfPNPue3C85NVzBNRSd4srqi6RTURYErOvi8hy8GSL5CCJ7aN9RKKL2m8ipmUpQM0QrlFWAjpSN4nIuBi8YHjEqJZgRqNfcBCQEJAfEeCQ5va1wTRfa0MXijsaKp0VEKL8RgWP0I709eZLNADh9aXn1gUSqxlZM64DAOh97ahd8WhefuhMQwORoL633rxJXb38eg/JwOERZl5NLGAVLawIPe7pZDfYKG8YT5BHtONHUyGnKaGMMQUiBnW+ZClONSUdu0l/XoVgPGbPIo7ePtyDEU2Dt4i+fwVExxeLQ2FEUnyUH56FhPI6M4M0gMlrZN5McGi58pGAcUMUHBwZC940PwDbdubjIcPoOSjPWNc2hlKKcXaCMkYj8WZxlCCCAKpQzKRFk0rRSjHYOtLLBFmxJIJI80EpUhYhDRaJ3RuAYfHH6m1yrkWU6Wxzm3EBwSPE1tCRp6vS6DQY/9vT22t7dwzjEej0ECe3u73Lx5map8ldgXPAiiotMxi6oscZwQiZQYosFYTRu6WcHK6iob585x/vk30B8MuOwd+7duIsQhW9uHt8Zxm6TKqAxtGsSF250ajxl9gbOrsDI0ZEYIrqZqaqbWU9c1TjQigjaaEDxlNUWrk73YE/QYSwy+k7CggrSQuGNhBjKk/3vvCd7jaourG+y0xE6nNJOtaBCXN8DeIgp6Hd+Iwfqa3fEOm1ubnF9fZ9DpMCh6rPVXqFZjZKbyQt1Msa7Geo31Gp0XdLoGrxRVreh1AkUWMMqhfTTWj89rrFB5gclztNH43MRp3rKCaYf4KI82jPNC8+z5AWdXu6yuDBgOV+n1utEzEaApp9TTCYQ4S2x5OBrFIhazxLVKy+1E3mn1HPfygn7RIfcW6xwmeCRYymqfLX+LHRomxG6npYm0tXATMHgypvTKhv6WUCsITUWV51RFhi16+O4A1ffolYIQsmgUS2ASNBUGQaOUjoF1KokSBlKmOo9xgUx7jA8zdkEThArBhphlTIIH34BraMopTTlGC3S7XZACJxobDDalMA+QMuA9XLkVhZmxEaJnIqR3PU3vi5oNfmcNcKoQM5/rgqd57oddmOu7E5LjVg6uOvDHYTt3ET7ROXw46MGOl6kiHeq2cffCxbaDgABWq2Ot21Pm71AaIs3qnFIGraKRJRJlt1SieimjyTsdup0O3apHfAOPj8QUU42YlIvu7pWmoYxcYq3J8hjc5X1AKZ2ys0li2QS8ddjG4lyDd1HX23mfDGih9v44PQ0H4L3j2rXrDIbP0NSWbrdPnheEcJY2BUHUfo2Ldw4fojErKurHKoHgHHu2SjEEe0BBnvejt1zivJlzAWPcnE+oQOn47LrdLt1OB2tt4gSPmI5HKOUpijWKomA3eKbTfabTfbY2b4AEnC1xbosHk5ObQxQo/YQsoyXuiRA8NvjZ2+YA7R1FnrO2vs75CxcYrK6wt7WFZDmiS7IQ404Mi/kABG1yyPsoKVEuxsjMQgYeMxSw2oGzZzP6gw6iobI1ZVkzqS21A/JOtCe1IYSQ1DZO9mJPzDBWPqCdB+/jJJjo2XRf7Esc3lqcr7G+ihSJakIox4TxGKYjKPfBjcGPiN3EyZhVkjr2yXjC3u4emTa4pmE0LhmPp2xv73Ht+k32dnfZH+/R2AoXGrJOl97qHrpYpWo6jCaGclrhXBkb0XCM7bgIojKU1mgtNFqlTHiG+Bgzorl2sHxEC889t8qbnhtwfkXR0TX1dA9bj2jqhs0bV9jfuUk5mUTns53zhB+1KxViKEnG3PPl4bGkHXgY+CTiaPBoCXTwFMGifSQqtB7vnMiJajPixQnIMCu3LgFjK5rRLo0tmWjN2GiqrKApenTONgy0QULASY1VOV4KvMoISkfeb3CElj8uDUp5jAgZYSaI0BqZNsTAvWhweFSIknK+KanKfSajnRjAJUOM6iIqx+gc8SpqJtsYXyYPaRhnmZl5Lm/39CaP8R3SNIf5Zsx4ycwH0u2/2YYLXtuZeXrAfk4ekLAw8l4YgIfb/I3x2P7AxcyuPMmwh+gwPnB+WdwwHkNrnByfcbE4eT8X7+rRY51+sUq/u0rWGWDyDkYptFaYPCpm6CyPiUCia/bYrml+Zfc2igE8jiZYPAptOmitCUFj8oaY6TzpZoSAdx7bxJlC5yw2fTofPcVl45GqVbM45jsKjlc+9lEa1+XyxVtsb+3hQ5opIH4G7/GuJXTE+qy0QilNlmUYo1E6o9NbxzVdCDVF0eWZC29gfX3IeFwymeyzs7PDeLwdFZTQtNkgRWnyvEfR6Ub5ONtQV2Nss4NgqcoRnU6Puq4RyfBun7q+ydwf+LDuDCGkTK9LnE6UKanHIlySpBUCWgsmyygGQ7rrF5hKwXha4uqYEMSL4E0HX/SxgwGh16Mqx7iwFdW67HHHIdwfMoH1ddi4MMQoQzmpsNMp1aihrgJOQaYyVAAlhhAU1gbqE1bLODlVihBQ3oGLXkwvgZA0TD0B5yqcnVLX+9hyH6bbUUx/ugXNPoRWPeKE31YNZligjKEuK0b7ozht1yTO2d6Im5tbXL58nevXb3Dr1i2qpsLjKfo9hmu7dAbriBniQpfpNHoTQoDgJMlsHQcEUTHVY6YVpU68TaXjTczICwc7jcFqn4974zleutDnzCBg/JjxXoxSnkymbN28xv7uJuXEUZVRqiUHBtyu3/HgVxy9XR3mzLyWe3waKRXWxqlcraN3tsBRYDHUM4ajJ97TORTnzJBnBqsM85wb21e53owZAj2EzNfYkaWZwITAWGCscuqsy0oQev0BojVIhZeCYBowBTYorFI0wdEER0h8SyUBIxqjVEya0GYmFom0HRc5Szp4VLBRa7mpmU7HjPd3yXSgyD0qc4jpkmmP9oqQMh5LA/Kwam15bEZCO8UMc7sxEA2IQ+9B6xWe28OtN3luGPvkyZ0n/ThkXIeDB4wD7uT9TVPThz3JR6Y35aDeweyakqRcEJUM4+T1PvII0Bh1rIZx+1ZntO9RQZ91+maDYX+DXn8dU/TQWY7ogNKCzgxBKUxLuZpd3XGinZS9d/vs8dTB4VEoU5BlHZCCHEeWg8ni8w0u4JzH2hAN4yZKFzZNg01JBnLlkOqk5vsD169cZH9asHtrFzuZMq+lrXHc3qsCyRIXOFIp8qKg6ORkOmdtfYMiz+h1Cwb9HivDFYw2VNWU6XSX0fgy3l8jGrRtkiYIXlHaDuWkx5yANgV2CVjKcpuy7CHSx5gO3j0K6W0RsX47+/hUCJZ4MBzZaoWAtzFdvUjAZIbOcI3+2eeoQpdRGGG1RSkVZSa7Q0J3gO30sEVBM9qGJo8JBtjifnMhHBcUMMhh/ZzhzIUNfO2xVQ2NxY0DrgTVAdOL0qlG5SgyvNc4f7K8n5MzjH2DsyW+mWKdpfFRlN43NaGpCNV+zCZVbSdt4dGCMfwY0RN65zZY39hguDIEEfZHI6qyYn9/xGg8ZTKtmJYNZWWp6kBtBS+aUIFMGpowwRQx1WZmclZWOtROqP1cAeo4EBCMzuh1OkyNwTY1lCNgRJxCu73sOt0+K8MeGQ12f5OtySY7CpSKnopyPIrpI23MgNvUUSzomXS0zUe4XkskvQRi5w7zTv7hJvxOFi5ETrttHEynqLLENDXG1ijiNe8QjRXBs0HDcxfO8/aXXmT3xnluXbtCNdojTMbUbop1nmnaZx9oEJwa4Xb7hNGZmLwj6ZTqrIvOcqzWOIHKO6bO4l2ccYmcfYPWBuUV4iOHX2sdsz8CEnzUtvUNoS7x9RRrS7yr4syFq5BGY8QhTRkpCE6BVdCoh+5fM90asy1H4nZiw6KJeviVUMkgluRVjtzjsOCTlNsPexuid3fmYZ5tPreiZ7EOt13F3Bi/7RRBRXEPL3PD+Ih3OhCojaI+RqLes4BB6KLp0mPAGYZynpX+eQbDDbq9VXTRQWUmcqU1kYcnQlZ0KDpdjMqOvUmNShJwPxUm4KiDI4jB5AM63QGZswRxZHlAZ22K2oC3gZgM0mOtxVqLaRqstYQQKFSD7J1ch9g0NVVdgzZIpx958SrRaRYWURqT5bMUy8YYipRIo1MYik7GcNBlfXVIrxvpI+VkgnMV48kW3l9nrhDfeupaZYt2fTtXsBiZEVuTEHKapgOzCIdHhcPXe5TNcUjTLfE4oUXo5BnDQZ/1jTV2KxhOFFOzQdmdokMM2lRFl4npMjUFlRhsgBCGMDUwCswDaR8PBOgIrKzAmWcGXHjxBaT29IsuHZOh/TbjiSNfGdI/+yzPnnuWsxsXWFvfYGUwIMtOznSFEzOMPd43NM0EW+5RVVN8XcF0EhUkml1wu8QX+0nlOUtYzVl75jxnz59j/cwZMlHs7ewy2h8xGU+ZTqfUdYPzAZEMY7p4yfEqenCtFWgcQVkyZel2evTNkMYJU+tjqsXjQIidhtYZg26f/WwPW9dQ7xEb06OmQoS804tcuWqb0eY21WRMWVr6gy5rGxu4piG4GF9UVdC4mHlmox+TA+74h+9XHdGf0RApFS3ho8v9pCN5/Ih6n1WcERiN8OUUXVdkzqKJTUdrrF0HzvmG1UGXT/iEt6Le8gKT3S1e+eUP8qH3vY9b+xUT4pBll7a7Cxhf4/c2CXs3CeIJOkd0js67mLyg1kJQ0DhLWTc0HrRqPVMZYnJQiqANeZaTk5MFjQkK5T1iLWIrqEpCNcHbCoWLkf+hQbkpTMs4v+AF4xXiDMpnSHi4qPTWMD6gRz7jL4TFjwXMV6ikWqFUpFpFJZrkMZ4dWOaHbQ92wLZtDdJIlls0cmPGtUQpCK2y+L3fS0n7xmuSmYd+RlZOMRTtDVZG7mG8PxjeyBoaQ86AQq1QmA26vQ1W1s4zGK5R9IfovIBM4SXgVUCZWIZ5t0On0ydXnbmi/zFC32cYnCdQE/A6I+sM6fRXo162cujcoYwnWIezPokzSAwI9q3HuI4zOc6Thxo54SQf2uT0V7toZciygizL0xlbEn3kR+dZFg1jpVA6fs/yjE4no9PNWVsbcO7MGkWeMdrb41ZT0TQTquoWHKkS4jlqxu/oq0xis4+EVieo1RDaIgafLvG0QEFUK+l2WF9b4ez5s+z5HitVj3FRUQ4cKuuwsbGB6Q3YdhpvoawcYVpD3U3TuSWwDY8xS2YO9Duwsi5sPHuGZ196kcwJG4NVBnmXzMJoVDE4+wZWn3mJM8+/xNnzz7OyfobBYPi0poQO1JNb1HZK2L8G1R64EnybZOMJG8MtFKhhzsrqKoPhkF6vTxaEqqhoGodzQlZMCSiaxlE3Pi4uidDXDqoKmZTorMJkE/KixGRTKhsYlw3NsU1PeSajMd2yIddx6m7SKaKslD2641Mm55lzK5xf7VLYbeqqZlrWTEtHp98n7/QwuaeYTCl6Db2Bo19C0QXTBbXDgeRkD4OG+LpZ5l7jxaC104TaN0zKkmk5QaYTMu8ZaMNQKwaJe93CAa/6hvd89AM8s97jmY1VVjoZw2GXjTOrIJayqilqSx2i57ztzvRoG/noB1H9FSbG4Do98pU19HBIVhgKo7DO4xqHQYHKCDrDuxzb5FgxOGWojaXIHEY0mRe08yhXI7ZE2wrVyt00JbmAChmZD9A0hLohTGr81BGsiimR/MMZUEUWO9Tb7OIU9Xab+gSwuEaIAUyiUqJgnQILiQbfzP97wLo+yryVRKWIvyQ/+tyTfSD67uD+RwXvRbs3JI+xP0ClmF9Ke2wwWg5woB8VL/XeiChDlvUx2QBdrJIVK3QH63T6Q/JuF5VFTrGTqG6gjEKU0Ov1WVlb5WzvHK/uDqnYOaarEhTFA+kKe9GovCDvDegMVvCJ2K4yh2iLtw7VOIINBAvK+NkgD22QpkE5j6k6J2oYd3urPPPMBUQMRmfkRZcsy8GHGAzYWGzTQGAeTEqkCWmtZ8GPWZ5RFAWdbg+jI1d4d2+Xm7euYu3Nu5Tb4wx+W6TB1Jw+N8USB9BqNi7AiLDS77G+usLq6gqD4ZBsF0JeoXpdCp1T9FdZO38e3e0znVhkUuNHNTQl6BJkDyTKjEZz9fFEABUK1tbg7IWC9bMXGKyfJbOgGmG0tQ+SEXB0e0M2Ns5z9uwznD33LP3hKkWnh9ZPqcc47F5OkfE3OJ7sQScAA7pf0E/pNE2W0zUFIppOd8CkV9JYMOYydWWZTEpG45KyrpNxbKPcDoGAIYhBqS4iHcoqSqB5d0zR4MEx2dpCb5xjo5+RdTvIxjphfwTbu0QvxKKZoBie3eDjXt7g5fMDwk5B02SozoAiFwZnnmHjwvM4ZyltYFxaztZ7qI6nbmAyhak/HgPWE+cG2nf7FAyJjsTEVexPxwwnY4pqSlcLZ/o9znS6rFdjehxMkb0N/NutG1x+94/wqatDPunlF+h3Cy684SznL6xhq4qtW1uYq5vULnCDSKsYec/WziZhZ5NaBGMMZ86ss37hPEWvi+kUZAjdAI0YnO7Q+JyyaSilpvEZNmRYY2l0jRaFRjDBY7AY31CEhiw4rK2pqxF5EJTPKXxAVRP8/ojxjR3GW3vYJuC9ilJUD4FOvmAYH8BBusLdn/tcslEkEaiT3KEPMbDwTvzgmYkbouKBc56Q9D5FWv3kNsveArVDWhZpmEsSp0yDrUlNgGAlyujBnAOdaBsz/nOIgWTHiRdeflOU99JdlO6C7iGmR9YZkHe66CJHGTPTvlQS0EajjGZI9G6/9MzL3Bzf4JL9CDVT/D0l1u4GQbFCj5VkGN/Pm6wQKTBFl2IwoBgOo6IDDqUtiMWrBsQSlCOokBLbeFAqzdAIQVlEGzgxw1hx/pk38Na3vDnqpqLI8y7GZDS1pa4qqsmU6XRCXbVe7Eh1ChBjPzJDCBYlnrKTU5UlNZ7NzU0uX7rE9tYr3NkAbakTjwuntRVe4iiYXGPLg+9toRQba2ucP3eO1dU1ik4PF0ZMygYbCorBKsO1sww3ziFFFx1GuLJttzz4AlwBvks0jPvMAzmB+37HHwxCTCy6ccbw7HPPsrr+LKYYIOJBTyitYmu/Yn+vZj0Y8v4Kw7WzrJ89T97tx3D3Y3RAHIWTM7ubGzx6+NYJo4Bi0GPQ7WJQ1NMaG2r2dvfY29tnb2/Mq69c4n3v/2Xe+773sb2zy2QSOdM+ZbcLwaWK1j4oHYMpQiuudFwVKxC2bzLZfZa13hm63R6d1RWmZ9ZhtAPNhJmou2RkK0Pe9PJZXj5bcKavmNZdpnYVKQJWDMVwHdXpEZoaU3Qoen1WNhrQY/Z2YFoer1c3cKprAgAT27Czv0exs42UY1YyTeYta0XGG7ThJWdnbO72qe4AP2Ut9eY2Upc8f26NjTNDBt0Ovd4QJYGt7T22xjU7zAXX2zmTOgR00yA3b9JM9uj2CopBl26vy7Dbx2c9KueY0iDOYH2GkS5egauhEU+DgHhU8GRYcixBWyTEdOnBWZRoOpmilwmhmWL3tsl2NjGjETXhDgz1+0M3O5hkZv7/ObM4HOkNk5mZ2hqXEBI9IakAtBrChGj8KR15nyIcNLajlWutA9vgnEspiKOur2rTNc+M4wXucXudM8N4gRHt47AXH45QpUjptgMQJFJKjrHBPnvheWKgV0FQBUEKvOqgsi4m76BMjhhD0NFkDRLQifNqTJzef+GFF5iWU/pX+9yor3KLKzTsPtT1GIYMOY/BUFMR7ku3JiarUHkH04mLcw4VXBTpF4dXNSI1XkXj2HuPcgHRGi9pIsMpTJYvDHCOGzF72KDfZX9/grUOrYQiyxEfcE0kjjhnqeuKqqpomhrnHMEnOTqjcS4qiue5pt8r8K7hxo0bXL16Eed27nL+p6GFXOJJIcs1tnIH6n4/yziztsaZ9XW63S5Ka3wQahsIOqPoDugMVil6A7zOCaqk8RBsgMbHTsgaUD1QZ6IkFV3mOtithlureFKn71EfZz7T8GCOz5AO3e/3WF17BqTD1s6Uam/MzpUbfPTSDT5yZURZec5OKpwYsk6P/soqJutQW/fIM9n3wgn6o09JeFX7bANzHbK2Hz+jWVtbZb3o4qcVN65c4/rV67zvAx/k1VcvcePGTS6+eokrl16hrJ70/QTYvYbbfgF57jzdTofVXoVb26CeWNjqRuM4VJj1nLe/pceveLHLuqmQxtHr9yl6XaomUDtPMJrNvRGumjAtpwiBPM/o9XJsXVOOoaeitu1ppD2cBEpnubm9RXXxVXZHO5zPM1aKjBUlfOKZDfqjES/WNR+0lleIKqWB2DS8CnT3p1wZT1m9fJ2z/Yxnzg4pVEZhhPMCe2Huce6lfafEdVdcwO6VDPdKzqldnlsxnHvhAllfmODZc5amFMomQ3UMnV6PWhSVF6rgqHyDDw2Zb2ioCdoSVI3yDVmu6Pc7rKz2GWpF6Wr8eJcVX7MicCs8WuhFJztodC6ax9F8PBh4N/slcS5EJEZY+6QV6yMFpJpOor520nTOigzd6ZFlOZI0fENKsRnP4BFnsW4KdRNFuUOMC8izlDQhKca0VzGTaRNJXmSV1HNiMCMqJdGQaD4rRVIjSEQNn1LteiE3x+vJKHpDAjF1eCCLxrEqUKaDMgVisuQxjuoPiEfrOBgoVAE9QZI+bq/Tg48EdpvNOJC6i3UpZNErkxTI43xYjzM8S5cee+wxZo/7E3RUoDPEZMxymhOQ0EqdBbwyiCi8bvDaolyUplLWRR+DFrT3FGUH4eF48PdGYDre5+qVy1x69TJl2XDhwnNsbJylaRrKsmQ8GjPe36csS+qmwbqG4KLOchuEVzcl1lYILtIo6pIrl6+wu3vpPsrr+LSml3htQeuDczQGONPtcXZ9nZXBAC0xq6L1ICbHqA55t4vJMnwI1E1NXVc05RRGE9ibwKQE52OisN4bQM6CVOBrqMoYdCQadBGN5nocP6UAMRAm4PcgXCGSBe8fpYNAjkiPa9d3uXh1l5uXr3L5Q5e49Mom1/csRuDMtS1eGk+pvEeMAWMIDtwJWyUnS9R43Fh05rQ6R4Z5ts2CaJGkGTl9puDMyirDrKAeT7n0yiV+7md/nnf/2I9z8dKrVNMJ7rioEMeB5hZhezMKxGcFvU6Xejhk+3wgFJ2Yds9VnH/R8GlvyXn7Wc/QlIhz5P0VdKeHLmuYllTlmP3xLr6a4KopPjiUjsZHt9vQ7wdWMsjt6dUdPm7UznFzf5ednesM7ZS6FJ5VmjP9HhfWBjzXP8vbA/zyzU1+djzmA8Tkr2VaLgE3PZjKc6GqeFPZcK6fgfd0lbDqAhukFJjMxZiuEWkZrXGqPZzfs/T3d+lnOZlvcHXG/lTImwxRXaQ7jM8lBBrno+KLb3C+xPsSpWqUlGShiglBco0pCpTErJKUFV3vKZRi7ByWB8+X1aKTJS9e0g4+SKCYN+bh0BJ/jBxk5z3eNtimoq4mVOWYan+PerSPNkLeMeS+T2YchekjIUPICBI1WKOiRQCxNKEm+BqFw2ghl0DHRG+uVyHJKYa0EI1iBaIWDON0efiAD8mLPMviRzKMA+ITD9kLhTneqfCsM8QjeIk0LVQBOkd0gSRvsdIGlGDwIGGmVGKMIcsyOkWHXm+A1pobN65ycbvDlKO1gBWGAWdY4wIiBUE0PsSkzAU5K7qP9TVbfhPLPnczrlsIBmM66MwgRiFGUMm41dqgtcIrhQh4JXgRglExA7fReEUcnIRA0esgSp+QY9Uz2t/m8sWLvPrh92LrKdPRmGpa4n2griomkynj8YiqrqJSho/eYogBedpomqZDY0sIFiVQTcfcuH4ZZ7dO4qKXeJ0gHJK36gNnBkPWV1YYdLuIKGrr8WhM3iHTPbIipilvrKW0jqossdMy2gmTSTR+XRN5DSsb0VmsHfgKdrZhZw/yHgzW4iz4uILaJhkhD24SpXXHBuxluM82AaCysLtv2dyasjO+xdUbm1x55SZbmxU2vd9NgBv/b3t/HmhJdpV3or+1d4znnDvlWJVVJZWkkoQAMYPBA/Bs3NBgY4wHBtMGv7b9PLSf3Xa/9kDT1mOw/V7zjN3Q3Tb0czO63Z7w0GCMMZYAMxiBACGpqlSqysqqyunOZ4ppD/3Hjrj35K2blTcz762c4pOiTt44Ma4Tsffaa3/rW7tzpmVN43ygjGmFF8GdMB//wXWMhXD1nfL9wRL2mn1HuRPO7da3+0VpxFKakirF9voGlzdf4QM/87N8/OMfC1O49x1msPMqu5s7YSToLFqBEh/0U7MMiXPW1iLOrUWsjhpiwCmP10HTtJxP2N5YZzreZj4bIzgGaYRSQmUqGlOjE83ymufCGcv2ZbhoH41YRuVhowlyZ7uAweOdYTgZ8/h0zEgJwzQidpBFEW9xsOssEzwFYY5kHbhCcHZfmztOzytOE3ShFfAWhDUdcXYwYpBGIJZr0ynDoubZ9joiAsXS7swxso6VFHxCbBJylSE6D85RPMCpjNppxCZgQJkKZRowM4wZY0xJYWtUZIm3h0xUitQRSkY4O2VuSzba676zCXZIOm7tLTjKHZ/XEzjDzjtM09DUDcW8YDabMZ2OmU62qeZjIlORuIbRMGWgB4ycZeghw4JL8ETYJswKetUWvPEWH2usSlESoaOYNNYkOji1ThRWHKbxWO/2eM1aq0C3aDnGSHDgvJOg+GC75kTwOIKumEVsjcKhdUKmDSK3Vmo4KlzU1rwTDSpuI68JKoohivcoIqLCtQZxD0ErRaSD5jkiDCVnZXmZ5eEyo+1lJmy0NRwhJiIlIyEnZ5U1Oc9SfhqfRNhI4ZzFGQvGQm2oXEXJlKPOLyhylrMBw0FClgpZAk4rPIISjXgJWuyt8gqRx+0xZ4Q4SVBa4/GkZR745yeE8fY6k3FFU1wEanbWFc56lIpwzlHVJVVZ7GnH7km44bGisI3GugZjKrwzONswn06YzS4f2V49ehyG+bS6Ifd4SSlWh0MGWUKsNU4UvnWK0wE4GYCOaazDFSWzsmI+m+HqEnwDkQPdAEUQRmgICXm712C2DuUOmBmoEcRnQQZgW0lPacMd4kAb0EmINtuoLcZ262e9dvDRF3Z4df1XmRUNZWk5GIP0gBFBohgvYL1BuZC6r+S4WtnD8eA5xp1D3PHFM/brDi86w579GqoN+zSYBSpwFGnyOCYCtjY3ufjRl3nppU/cp04xgIfqOsXGFuOlJTLlEK3a+I+DNEEPMpaXY5ZHisGwRozHi8GLxhrDbHeLzSuX2N6cMJ5Y0gROnU3I8oSiKmlqR56kDFdSHrswpqo843W4/gjkatQIW4TSmymhDYiAJ4DYQ249+bwh18La2ik+KV/GlBXz+YyLszEv4BkTJpVqgrOZAufaY1wQ4UK6zBMrZ3hi7RynlwYsDRRbW68RPfscaWPZJrC5FNBMHaWfUuo5RkUonZElQ0TaCoipR6cRhohZk1I1iriakpQN2kyg3qIqC8qqprYGmwyZJkssNQkDvUrpLIaKDQJX+k5nBtKOv/sGLmGnDOG9x7qQuNqYhqacM59M2dneZXtri8nWJvXOOn6+g840p0YpabLEyDtWRDFESLzB+wTnNGXlcbXHao2PY3QUk2mNjxMUEUpFJJEijjrKhsKgwATunGrLVUeRoKPWMW6pFqrlGzuvwvvVRZnbilOYCm8qlHiyyJNHZl++7RjgVBauR+mgzhDFrQpFjNLxXhlo1RZQ0XsllhVaFIIiUoLECYM8Z2k0Ylkts+tGjClISDjDWVY4zZAVBtEq+dIK8fISNlYYTesYG6rplFm9zZyidaqPdp8xA5YHA0aDmCwNwSkfCb5N9vROwCtEorYmeSgx6Lzf0xGOoggE0iwL250Qitkm3m/T0QCdfY2djSjM0EiEc02Ipt0wjdvawasgNVdo5rXGNBVNVVLOtvFu58Su+dFBSui8F5PDHh34hVkSAUZRzPJoQB7HaKXwKEQ0OslJBzGNH+BUHKLITcV4PmM+n+OrNu8rITjGbgZ2DDMDxQZsPw9+mz0buy2orhIcrpTQI0q7xNDEEGvI2qn4agjNNvjDJAlvRFlDuf7GvY5XGok0jiClCkHO8WAE/bhxcq3MIfIid3SMLsIbs+/4hrq7YUna7zV7/DWc3897swv7dSK07XrrDJU31BjmVcFkNsWY+1RBo4OfY3a3KesLLJ0eMlzKwXkmWmNURJxmkCgmCFMihomQSIX3jqacMt1eZ/vKmPVrju3d8DyL1AyXG8rC09RgsoomsYjyrK7CqW1Ybx7+PGajFEUc4bRCjKNkv6jHJrBCaBoS61na2iZLZkgcUcWCbyk7tQ+P3FWCg7xHsQBe9p7HyzGPV3Me297k6eVTvPNtT5KfPsMTb93EbGyyMTOMm/BYY2BnF67iGEsNqkbnc5Yyz/KpjHi4ilobsuMGDHc9O40hqiv0dEw020Fmu0xmNXUFbjrGqSl2aYCxKU2yhgzBTD3ebKPv4tdNo0UO0+vR8YeNMdRt1K0oZsxnU3Z2dtja3Ga8vk597TrsbEI1BW3J3/4W3vnEWdZWh6TDCJ16nJ9TlBXeRzSNYjy2TCaGBo2LYlSaorOcOMvI4pQszUIVQxUcLY/QSEiQjYwP0VYV6BFaXMtztkGeyzm8tbgmSIqJhGp62lu0tyhv0L4hixQrgwy9HMq1H9fsio8SkLa6pdb4KMLpCNEKqzvJsBBZldZB1hIiyN5LcGqdw1qD95APhpxKz1IUczzCkCGnOMsSa+R6iWQwIspyJErDdGnL7dZKAp/bExKOj9ywC5mkrGQRg9gTU6F9EaZlJcL7BC8RWilsFKG0xzpPyCwN9hfVyvSJoOLoRLPRvTuo7FOB38LbIZ6E/dRZz76CxJ6cSVjnQ8GcelazW5c4s8V9q8r0ICBNUMkyUXQGWzhseY07n9t6gLHASdPAKI4YpAmxCsQkrzU6SlBa4bXgrcYSkpHrsqQsylZq0EIEcSo0dgaT16C8BqoAu9k6xYvwHK6bvdDmNwqa7gK7ZL3jgfOexjbM5mO2t67hnWI2KTG15bd+9mce23kO4uQc45C3cXRvqosEd05slyjXJUAmB77r1nXtk1Ztso0H0+zLNNqFY3dc43Zd0zRMbcXcNZTO0NjX1yO/exx3Q25gtoNpLPnyCmujlFRr8jSmsGAlotbBkV1yQXA+0gW22Kac7jLd3GbnimP9GmyUkE8hjqGuPVUVTFcknjRt0ATH+cwALu4+/FxjI0KVZegkBmNwhG5wG7hMGGut0Rbfdo64LJESVC6cW1JEcUwyM6wVlped5xJwkcDd7ZareH7TNwzqHd6xMeELU+Hdn/I0S5/2SbxHpuy+dpmdVzaYbjXMi9AFvAxseRhaWJsZ1qY7rPlTnBkqVs+tsNvkjMqC68ZipzP89g56sgtFjbLhvHpek27MSXEoleOyIUiMixKiXUPejO/42U+6iHEXSOjg2zLa1mGcwTYl5XSX8e4OO9sb7GxtsHV9nfL6dbh6BbZfA7MT3tvRKmujZ/j0d7+D0Shm7gomTcG4KpjXBms0ZSlsblRsbFRUjeBUhM4GZMtLjJaWWBkNiZZApQkJHh2FCLBWHmlLb4eEO0GJQ3BB0aKusHWNaZpQqbM2iA0qBToSEgWZ9qTKk2jHKE04t6TJVxMSrY6PdqRDxDjwpVrnWGu8aqvxSafUEfQ/glOs2si4CxVHjaGpa5qmIc0GnFo9T9N4IjMkJWVZVsnjIVGeo7MEF2mcC8+6bQVBxIb+9Ebt5qNAyCVmKVXk2hC5OcooQj2/NOhCo7FKYXWEIQTcvTU4Iyhrca3qh0hIJLxZSe7jwcFfzhN4k5YwNenYd3I73t7BYuJtNMdX2Fqxn6Lb43ahTuU8+Z73MozPwDxh9+oO1y/XGNMVBnt0EOVg2vz/FBilMWkUofGI82il0XECGqxYjBeMg9pYyqqmrEqsMeA9sRZGIoybCbZ8GdwrQVr3tsj7i5kjb+wIR63PfCftovOOui6Y7G6xfjVhPqvYvLpNWVTwNX/4Do54NJxsxPiw5OfF4FLXjnQ84az9dxfV7SLFnYPcKUx0x+iiwo4wDaeb0Ho3BI+mbD87/nFXbs2Gv+3AMK4KCmeQJCJfGqJ1FCoz3bdwMN/Ejsd4hDTPMYMEU0bYCmqvqBxslI61SnNupJAoVJCaT3aZjkvGO7BewmuAqmHyKqxstk26QJpBnsEghSyCSIWf5mF3jIki/GgIqyNcWWBdGCdvE2zVCZuM2B+jGcAUHmUso5HweKzIXcRqUbNGoGQ8x35iWyd8UwEfwnLpykUen17j7FrGuTMx573lzJkVRlmDHRvicUVRGQaEV2HoYTirSLe2WFrd4czalLRu2NnexWxex29vwe4YihLTqk3kgMIzLBvyGtQggXSAS2NcFuHtDLUzJ/J35tJlXdJZqMmM86FyWdPxh4uC2XzGZDJmd3eb3e0ttjevM9+8jl1fh43rML4G/irhKUvAJwyHCU8+dprBMGG73EXNwYjDimB1SNBCCZVxTOcGX5UQ18zmDcWswqxU0DTEEeSZRnwDItimoZnPqZu6rWwnJFFEojWRs0TW4H2Dkwa0CZQVFEmkSCJNnijyWMjjkHA3GmScPr1MUu0ca0UmlbQcYy2gdUg8iTqnWPAqFC/ZTwxkL6JqrKGqKqqypCxLprMpxll0mjAcrOFnKVolZNkQiTS1gLcuFAsShfMtZdwB1odqdN4FFY8jO8Y+zHDNd6gmGxS5IPUslECXAbFeJtIeFQWHX+1FvhWqlZ9TTvbis0ovcuXeLHSRsoWpRuBGx7jr5LoOrfsO7jyl9RHHQFh9+2O8+53vIqpyptcKfDxhR45vRuZBQj7QzCqLs5ALDLOUWAlNWTKfTPCjglLVzMua6bxiasDpmLqqKYuSsqhCpVwTFH+KaoabbILb4aR79kPl54+IpqmYTna4dvkVZpNNrr52nRc/fonJ7pRv/xt/6/gu8gBOzjE+2H4mBEWIxTaji+TGhGFQ2w/sSeZ16KpWLtIhunN0peSFvY55b13T7rfoGFfsO9SxZTwvKJwjHQ44ff4saZ63XJbjwglEC+pN3OZ1qnmB9Us43xGpwbmIovZsTx27ucKEMSVVVTKd7DKbWGYlbLAvHTZpIG6C87sELE1hKYMyD85xUwfnaszDze5SUUy0soKcOoOfjmnGFQXQ5ZNH7IuaDAnO8ZyWU9yA2TYMFOTpMqtxzvmm5BwVQxzPwp6OcQcLXHeW67tT2J2SvAzvyoSvePsF3nXhKc6/bcS7xlOevnyFqzu7TMqK0jiiuUdd2gReIWJIVMfoi5sk1zZI5+vE1ZTa+z2neEh4xZaNZWShiWKaQYYVTWMVTbWKmW5hmlvzwg5D2uoYB3UGR2McxtRUsynj3V22trZY31xne3uL2XiHcncbt7MJOxuwuw3VdjuF1zkRocxgkscsLw0YDTN85ECHRIxB3eAkoqg883LG1iRiVs3w4wJmc+ysZjqe0UxnmGJGpB1ZKqRxBM5RFgWT8S5FMQ/cYVEsDwasDAYMk4hcC3EiaNHEOiKJI7I4Jos1WazJE80gi8iSiDRNyLOE4WCAm+2SJsfnGOskbyPGbdRYC15LeIaUYz+6IK3cnAq0Fe+oq4b5rGA2nTKbT9ndHTOZTamsQeUZeTJARQk6Tqmbmsl0SmMbkjwnERUUJJS0ba/DGofxppVJOrpjXPpttjdeYzmDyM2ohktonRFHy4yyBsl8GInHcUtf9KAklAcHvArKIQJ7695cdFPJ3T0vOsbd94tTo2rhu67j6nFbEIjPZTzx2HnOrS7TbFnKpoBiirYzHkWbJmmESS3VHJZixTDPEe+ZjcewsQHpElOTsrVTs7VTMLU5EitMYyhnBfOixDUG6hq3s0m5fR12uloTJ4u7CTOW84Lx1gZ1sYMxFc9+9CUuvrJNbU42eHlyjnFHeeg4vssEr6uLIncD7S5i3EmrdVjkAxv2qRl24TvbftfZ6OCxOzpF138crLlReKqyofFwZmWFM3qFs+fPMR3vHKclTgC7+M1LTLfeTXl2BWcNCosWhZLA5yxrS2MFpWPiNEZFGlq6nuVGlenFKOYU2LFwbtbOruRhgxWCAzh+s2/1TYTXoTwzy2v406eYV9eQytEQ7KUJgwcINkvYH8N17kLiIKkKliRlmZghBoVjGXiewDe+WVNUe/h44fmN13ZYcgPOPHGKtbWzaJ1wbnWX6WzCfD4PUlHesVLX5LvbTCcNxeVX2d3dIqEmwzEEzmpobHDIa2DmZpjJBnWiMYmHRHCRg1HO8NRZsnrpjuwWS+CzBoWJink5ZzafsbW5yfr6da5dvcLm1ctU25tQzqGcwXwK8wnUU8KT5UFa6oBKkOGAfJQxGKQsDXMkgjSJGeWDoMOtI+aNp6nHTOea+bxhurEL0xk4i48U5WzARjlC+xKaKbFWeNMwn0zY2dxkPp1ivUXFEefOnEafO8fKmVXOnllmeZiSxJo0jkjTmCyJyVoHOW0d5EgLkVLBFWpq1HQSuEjHBLnBMQ6LtNxfsG18UlC+dYpRrbxc4OrWxlI2hnlZU9YN1gs6SUnzjDTKQqlxUZSTCfPxmHlZknpP6ixJmhDHEcpZvLUYazBYDG1BoyPCMGMyvcbOdYj9nHo0Io4GpOkqMrAwdNAM8VmKTTQmUqGgh+/SNYOU22I57pPDokO7iM45PgyH8QUXM8B73Al0BJFYxLa0pmKMLcY417YVjxJa1ZwkFgyeYRoxHOYogcnODjPROJUxncPGhmN7xzLHoeIYZy3VvKQuK7AWyjJIsV29AtWYfc4q7Jedun9grWU+GzPembO1tcFLl6ZUb8K46OQc45jQPjTs2z7mRoe14fUOctcGdTkOsO/Yds5w5wh3+y7SXTjw78XI9cHf3AW9akXEY+fO8+TTp3jlxYtcfuUSVXk/EwcszC4yefUVds6ukdmaSCmSNjknVYZMGvIIRoOUlWWFn5+iHp9ltLpOlpqbtvOW4BwnwKCBQQSZhjNZkDEc31/vzbHCepjrCJcG7cZyZcp0a0JqQqS4G49NCUoTS+yP6Rbptc41OBoUgXbxlnabAcGxfo6bxzwq4D/szHht9wUuvXKVt6+ssDoasro05PGzZ1gapIgN3FejFYXWvLZ5lcvza/wmlpowlvmCDD71U86ir064dLlkw8MVP8dNPgHVOmq6SrY0YDjMWMlTzr/jKRJ1h82BNdi6YjzZZXd3h92dbbZ3trh+7SobVy4zv3oZf/01KMYhmUzr8L6nCuIcbAyDDFZXkDzFm5poacjq6ogoVuRZQpbEnBot4bwLJZvjiMJ4UtlErKKa7PDiqxNstRHSnb2HeUY9ybm8fY2tSxniGlxV0uxuY7au46tp+EWjlNkTb0N/0jM8OXo3T55+jAvnT5NlCXGs0TiUQJSkRHGC9hBZh5kXVLu7VLtjyvGYV37tVynHx6dXGyWDLlTaOsbh3x6Hbhs/5YNzjBecD66k8xavLOgGFTXoxJDkMJKUaGBxcR4WBGuhcB6r15n7iqIs0bUwyoYsjQZoPN7UNNbQEJzj24kBCTWF3WK8A5EUNMUSSTIkzwqoGkxVINkQBkN8EuPjCHRQXUE0gkYkLJxwlEjUELzG+51Dvr2dubKHuJF8M+ChfK3glfwTLOsBfgK7kzHzcpf64Sf0vR4enHE4G56rKIkZjIaIVuzsbDGbTKjmhsnylCuzAeNZTqUUkiR457Fl3U5pto7xeAzFJkEB5jSoFVAWzBb42yvUcdJQCrw3TMY7XL82pX6TJgtOzjGO2rJpi9SJzrPoKA1dNcEuitsN2G/GA7/T9uZm+znBGY+gOH3qFO968h1c/vRP5SMf+TCfeP6FE5cEuSv4KfWVl9l64gJnhsJSEmNF46wiFohxJAryNGI0SpAzZ3HVmOvnXmN5ZRN9i9BvFwWNJPgsowg2t3kzZl7uGSxCoSJMOsCNVphXJcp6op0ZiQ3UhJIQNZ8QnOPThGIdnZBN026zx9YBzgB5AqmD3EHhQkLdzbr5EvhN79mYTnjnbMo7BkM+5bGzPHb2FO9429Pk4pCm5uruFi9uXmNnd4PXjOUK+9T8yUhx+lOf5NrgKvONK+xUXdduoN6CzR2m04TZaED++OOceeztrC4t35Hd5uNd5mXBtY1rXNu4xubGOtsb19m9cgV75TXYvALmOoEqMQA1hOEgVFyKsvDenz3Nqbe/nbXlJcbjbRLlOX9mlSSNyNOUVMeBA6w1qqUTjMua7cGUy6kjc1Nkfh0m10In4DzMYtiJsNc8M7GhhratwG4RCDJdKxtRvDjmSizM3nKGPIIzKzlLSzmpFnw1w5sGyTNUnuJKg5lUVOubbLx4ie2XX2V87Sov/OaHqMrj61iiuIugt4sOn048CgcuOMYCtH+2GtEWrx0qsejMEVtIJUalntQLTZxhoxTjwBpDXBYQJzQ4SiZ4V+OKCq0MsVJ4Y6lsRUlNTcVRE5/CpKGjYcy8gXi7xlUFWTrE5iW+rmmqOSofIdUA0hxJUnScEccZOkoQlSAqBokQ6+6OsHgL5INVkCHzyZRHcbr+vkIBWy9t85I8R2JS6rFl0uy0A7NHED4kwnoC1z7KM7yCyWTMelExnRl2BnOucZbCn8bHMZiMoDJgCS+7Df+ua/AVSAKjNcjaCOYYqI4uxXg3uCFH+w22i2OF1kJT18yKN2/IeYI6xipUKehS+7cIobbOOe6c39tRrjhuOA/WgLPEkWa0NODpp5/iMz7r06hNzdVXr2KNIcsHaB0zGe/i75vEPIffvsjOxccYveMJzqycpqksrqlxGJxqpaZMTRQNOPvYBZZHA+bjCZvXf4X8asHkDfq3BFjO4exZyFIoG0jGh+dTPizwXjCS0CRDTL6GXzIE+rZCJlMm3rFNeJR3CY+z4cbc0Y7KXrJfyU4BcQ2PESLIOfCbwIvtsQ77GRxBCeOK9/zibMrjn5jylTubfLU2XDi1wihRvHL9FT74sRf4tWsFV/3+7+KA9crxscuX+NDLYy5Vh8S7vMOXJfOy5NJkzGlfoM6euSO7feijv868Kri2dZ2trXWq3S3szk6YsptvB4vonD3NxUTD0gBW19oqSga1ssyptRWePneW5InzrAxSPvmtb2FlNCJNEmKlSaKYNI5pjGF9c5uXX7vGhz70EX7+g7/JJz78Eczl58AWhFh+At4E3dlmyr46dNb+cotWb6C5zNalZX7jubOsPX6euTW8/cyI05HDbV3DFzPyx54kPfcE050Zm5eu8vKv/gYf/5mf57UXn2U+3eGlcpv6DhMYD4PTafiHtBxiL4GC2zrA+IXJMAHwoRiJckisifIolIaOcqKqpraOxkIlEbVEKO8x1pCPlhgtrzCebDFrNqnZZNvPcbMpCREKRelnjNmlZpejOsah0GjU1rkzGFNTVzrMrkjgyBk7R9UDpMyRZIBKcuJ0hM9GJMkgzCjoDKVTMCdb72q4cpokXaWYXsH7R1AS7D6Dm3peff4aOlJoo6iteWj7njeEQBylNLrB46icZ1rXEAc9eGcrpuMttqaWsXJ4rWCQh0iwjgllneMwW5dZWFmF+kJwiNcGoH0o7lHakO/BneWaHAWpwOooYjSMsN7QGMt45plWh495B8OMpeVlZrMRS1tzip3gtp00TjD5TvYdX9ivm3s/IQj9gTMo5cnSmMcunOPTPv1TSbOESy+/gjWOp9/2DFk25D/89Ad46eMfO9GoxW3BbGJfe4nqwhmSfIh2BW5aYLzBKNtKNVUoBqydOU/y2AWmuxOuXdlk6dee5fruze9jqGFtFc6cDTPfbjcErB5mxxgEKzEmGmJzF0bXjUCt8SZiWmwz955dQsS4IsxuD9pluf27E0+Zt9ukBHfsLPAU8JTAe2O4ZOHjFl4AXuFw/nbHLLoI/PjWmCefe5b3vOU8Z1eGfPTSK/zC1YJfc4FHvLjPZgG//twmH7566yD/vDK89vIl1HT9DmwGH/rYr9PUJfV4Ez/eavnD01By1NUw0JCvhQeorsIFLg1hbS3wZW1NNMjJ04RTSyPeeu40bzlzmgtnzrIyHJLGMcordKSJk5iqbli/vs2zH/sEv/zzH+K5X/p5zOQTBK5yZ4Gl1qJjbozNd1NUB1HiJpd58YVLRGfP0xiDfnIF4hr32ovIZAc3nyNodq5u8/KHn+fXf/o/8J8+9H5eMtO9ktrH+W541SbySfef4BzvrVrctnWMvQTnmCgikhiiDBXnRJlBNwZtHN4Kzgl4i0SWbDBieWWF3Z0hmzsW2KZily2fEJERk1BRUbPNjdkJN4cAK6SMSImJQrvhLbapMQpqZcFXGDNF6gxVZKh4iEqGpHmJaprAQbYeSQS8DhHjE2t9hKWV04yWz3L98ipN+WbIrPVc5FvBlR6HpXlUI8WEpyRNcmodMoFK49gtSyDCK4/SPtQoaBoaicNgUtbC9KQWUFErTKDDqPrU6TCjnypkKcV7C/MZVAqKBswVQlvaFbO5++dTAUOBC6cz3vL2J1k9tULdzJnMxrx8ZYPqckV9YLwtAqPlEUurK5T1KU5Pd6ltxc7k5J3jk1WluN+fZQuuCrqlTVlQlXOcNSSR5tzZM6wsLzMYjHjH258hTQdsbW5x7fKrzKf3SwqagdkVqvE4VOtzDm8arDdUYigqmM4r5mWD8ZosHZIun2J45jzD0Quwe3jkJxd461lYOw0+Cu/KrIbqfv897xph8tdLBJKCHrRyHS5IO9gKV82YEKgQ3aRHl7Q4Y78+kyYkLHbUhpgQKc6AgYfVOmginyFQMj7aLttvcHVXPHzgpWu8cHWD5VTziZ2K33SheEj306SEqLQ38NIrQf/4VvDA9YnBzia33PYwTOfbrXZ4CdoF7rCkkEgoE5olqLUVJFLYnV0oylACLdQzBwVNXfPi5deoTE0UCedXV0CEOIqIVIR4j1YK3WomF0XJzvaE7a0dzLxz2A4mFhwWX1zMvl1EBskp6ijl6rjitY0xm7njVFqjdndQk2309Ss0ccb6a5tcefETvHrpBV4xUzbvyGpHwIHyx0KozNfpQ3fj8z2mWqtIYb3HOI+1nsZ4rKNdBOskTOQ5j/PhU0SRZjl5nqN3NfgGmLXJdhElKeEJKzhqox6hGDBs3eIIIUEkRnSEaI3saTE78A3WKrzEeImxUYVt6lBYRVu8C3PISoLy8clAiOOM4XCJdLBGU17hZMs4d9qkXQvyMOv99LgbiAhpnFNnNZGqMc6xPZkQkTPIM07nKVWlqKqYosooGx+Sp2wDLiZwidvpJdGQ5+Hfw5RkdYQToSlKGJ2H0eOwuwWznVAW2o4JvZsn9C6e0NMVhOe2kwRb5Mx2CE6gBpY1nFmOOHf+NKfPnGN5bRXnKrLpkOu7BV5eH74RIEkyRqMV8DXOl0h0Gfdazfxo4/M7xsk5xl2i3P0MC75sMGVFOZ8xHY+ZT6eUZUmeZTzx+AXOP/44b3/b24njlE+88AK/+Ru/wQvPT+6fqLEdU493ME2NaxrENDjXUGGYV57JvGQyL5lXDVEOLs6Jlk+ztJojrzWvcxESgWfOwDufCRrGZQGTGezOYG4e7tiGoFBehZE1UeBgqRwSH0YLjQErYGYUeF4mNBFbhMp4WwQnd43AO16FvdLSHYVe2Hdeu+KNuYLIB2f6WR+OddirMwd+zniSiUFNDEV7/m7bAaH09Gp7LVf80ZmSM6C84765Cg5xHkE6BJcFilJTgSlRwyFnz50j0pqrkcZu74Qyor4lougIPyuYXbvCC9cuE0VwZnmJM6trKFFEWuOtQ9rSx977Vh+5pK6aQNm64clU7GtAJtyoJXvYExyDugDnn4bVU0yNsDkuGe8K86wirioiUzPZ3mBiHNde2eDaKxe5vnv9hFVaDsiTiSCi2rLU7vU5xx6scxhraRoTdKSbBmMamsZQG0djHcYJ1imss1hnwHnSJCHPc2JJKDzszzN0xKDbQ0JOzpAY3f4vRukElcToOEbHGh1rVKTwosM0L4DzeBuUdZwLpV9bqelQ6e+2r+ToUCoiy3KGS2tMt0fgjy+R8kYkhBkNzf4Mxkk5xl01LN2eozjBc/U4CYgokjgnzy15NscAm+Nd8ghOP3aKwdISsYuJqpjpZsRrOxGIC7JStgZlwt/dVGaagNbI2gqDs6dxUcq4dvjawJNNCFzs7MD2Fmzvwng3jKwlCX6PmYGbgN8hzMgJN1ZbCwNrmCAyZTkynF1JOH1mlZVTp0gHI+J0gNY5Xml0fOVwd0qEKIoZjpbIB4p0IKBqiuYK0QmrAJycY1zb+z9/wYEvDa5usLXBNBYcJDomHaScOX2a82fPcmptlUhHnD93hieeuMCliy9RV/cLL6Sm3rnOdLyDQhilioiITEOkwTQNs9mc8XgCSjGvGpyKOHduxNInxkzK/Y41Bk4nsDoKXP3dGUzGMJ7AZAqFfbgdY5xHWVA+wpG0UWMTnD5lQC+Fz5YoYfBcJfCNNwmR27cCTxOc1YgQF1qsTbOoAts1J0sOzknoshKCpNt19jnMfmF5I+Zj1V5HxT6l/3Zwp+PYKAmV2ZQkIQDsHeIMRTnFFR7JInQSobVG0iQ0zCrIzEgSk+Y59cTjtmrcVsnLr77Gh1dXeGztFO+68CR5FIcofJuD5r0LutzTCcVkF/zBae/FUflRXCkfnPT5DMZjXJRQp45JXrM1MESzBlU6Il2A2WVnN0jnlbZ+08b+LZEiOMZtmQ3XDgh8Gz52rVNsWoe4W6yxNNaEIh3WY73Ceo81hsY0WBuiPnGUkOkhEzfCs8OdO1AxKUM0GaotheOJ8VGMVzFeRXgVhc5Za5RKQBJEUrROUCpCJApqFEqhlEZpjY70iUq2OefxKOI4QamUk0sn6YbJXffbOcjHiU5cckRwwhUhAn6do9JhetwfCG2/JooysnyIcoa6sTTOEScJS0tLqGiANSmXneNa3bpermqlbl1wjFsaVldJU+cjkuVT+HRI1AiNlVCO3VhY3YWVHVgdw+40+HMEHXiqAqoZFLtQTVqnW9qpUR+ULeavQlPiERoPReOYlQ1qMqdW2+SzmiQRqmrOeFpgb/quCVpHDAYj0swxnW+yvLWOO+HKhyfoGD8Ao1IPvnS4xoIFhSaLc1ZGK4gohtmQVMdgLRZPnqWcP3uGldVTrF+7fK+vvoXHTK6xs3GdM2srnF1K2gx+w3LeoHxNMZuyu71B05Ts7u5QNQ0XLqzx2e/Z5fp6QVk5jAn0o0TDvISXXwmFPWYFVFVI9C8faq8YcA5tIFMRpYDRLjQqvgGrgqyYHbA/fTQHHAXwEsExXmefW9xJbi8TmBi0e3Wa0TOC81oSgtLnCIHp8zpoSW/7wGXu9KN3eOOxpiU4xCcV57oZTo9yRGviKEZrjXIh6XMDy7ScYeuGnfkcpTXGubaKm4A4kkHChTOn2M0TNqe7MNlhdu0aH44j3nH+Ap/3zLtZitOgFxwEbXHOUJRzJuMdivF19rnFHQ5qg9wKBvwV2JzDfEzcVChdMYmXuDpQ+LJGrCfXnlRZZtaFukEiB2O6JwIBRLU0CkLiXRc1ds4FCkX776ZuaJoaYwy2MVhrMc7ub9fSLZwTjOuiymE7pSJGyQq7zTkq1rmzqm2CZkTCAGkrNnkEJxGWCINGE6rbRSpGogSlU5SkKJUjKkNHOVqnKBWjdISKInQcoeMYkZOzeFXXVHUTBhoHaCzHi06WqTvHcXbDXfmhsAgZWqftb14Swh89HiR476kbh1IRaTYkMhWCQUQhSpMkCWo4wjDkVGkYzA3jSsCVIYcKT6jVI2FmxmtQCSpJSfIRLl9GG01j25lSA8gQZAnSAlZKaFohBWvBFFDPoZhC1SYyKx+mPSMH5RZc3g45Ot4wNVCNDbvFDsnmHJ1tEicJWSJ413BtfQd/iLvoPVgb2rUkScjyZZaXlhgOE6rmQXWMHxCY7ZIrl67w6pnXeHJ0HoVCOU2apSQ6RjyURYl1FmcNeRqqXK3fR2lovh5T7q4TrSSsDnIGMSRiGWqDLadMdz1aO6bTnM2NTca7O6Sx8PQTI06NLNNpxbzwVHXIl5rPYKeAwgRpsbaC9sNf3NQ6pLboltpgjATqhANcO5reK+HYxX9D9MUTnNjn2WdjTggR3scIsm4JoStcrEzeDZRbYR1yAjf5tEDhg0O8S4jzGG5MsrtdnNQTu5QlrT5xFD4FvA9RPpQGZ5jPZ8GpMU3LKfEgHh1plgYZWguTM2eoFWBqxltbbI0nlFWNsYGnFhxBS2MaimLOeLyLKQ4jnnQJDreTOFKD34C5ob5s2KrGXJysMFnO0MoSx4q1BFbziFqnSJ6RxClJPTmxnGK/RyIOvGK8x3qPsxbbLov/ttbuRYld+53zrRPdkZK9D5FRH8TzrTNYZ/EeojhhNFplpTrPdXOZMLi43ScmYcgqA5aIyPacYNEqlLTWGqc1XkcQp0GeTedEKkckQ6mMKB6goyDXpqMEFUftok+QY+yp65qqbMLM4YlS5TzBtq1KCzHhpTiOYJJmvxpWDWJxvsD7jhJzv8x09jgqvPNMpkWoU4BGVIxSsjdgTJKEdJDjoxFr05qVYcXcWEzRUj5jIBYkjvE6Bh96IuV9eDejmEhiRGm8bZ9Dp0OCkY4Dv1KbVmVMwCWQ+ZB70/gQOBLbRo5t2I+gOtO1H8ZD7QXbOKwpQSqSGPCGojj8uY9jzWg0Isty0jRCKSHNMvI8YVifrN7yI+8Yu3HD87/0Eex6wfiVbR4//zirq2ucOXsab6GpDdubO8zmU7bWN5lOphSz+6z6jquh2CTxawyTNUZxQ+ImxM02ZbnJ5qyhLMZEacbGxgbXr11nd3uL6e6E6cQym0NVh5kWZ2A6h50mOHddhPL+GQacIJzFFTXGQ2MN1GWo1NbxWJVAvDDqJgrZvd7t5elUwMcJJbc/AVwg0CueIDjHa4SJza60dHuUvchj7QM9t5sI7dLI6nbfu1FYPbF8fqVorGVSFMwbg3cW7y1NXQdn2TmYzVupQ9ftBALGNcyqOVmc8NSTFxifWmV7exMtEEcR1gd6gPI+BEyVpyxLZrMZk91d7KFlrDvnYJG4chR4YIemmPPKq6+ycW3IUrbMyvIKZ8+fRpZPM0hW0cuawZmS0dIqo9kGE07GtradX7yBPuA91jmctRhjMMa2lAiDtSZEitv9VLuvEsHjwnRlG5pxPgwePA6lhCiKSNOcleU1TFUy2XmKgi7x5uhIWeY051hRA4ZRSqTCK6IjQcdCFCuiRBOlMTpNiLKMWOdEeoAiQyQjijOiNEcnGSpJ0UmCxHGYaThBKkVVVMwmBcW8wrmTTLyDGzVtOj78nQxEDjtuxw+Xbix0DMftce/g2VjfJIpSYi2sZhFLaUqSDYjjUHQoTjJIMpZyYTmzbOxOMdNJ0CzWDpdGMBihsgGOFJRFygm+mCBxQqxylBHspILxHDZ3YHsn9IG2ClQzadPNVVty2BRh8Q2IAW9D+zLehvJGWcfRMOP8+bPE2YDKgHUeLQ1NNWNebmAOMgxEOP/YeZ5++zs4d/5xYE4xLxAvDLIM1+QnavFH3jEGaKYVz3/keXZe2+KdTz/De97zyWgVsby8TJI2FOWM7e1ttre2KYqCur7Pqlx4C+U2qpkQ+4KUisxtI9U6s8k15rZgOtlGooTNjU021reZjutQmXcGs3nIK9NJcCVmZj8Pde8U9+jW3lQ4oPZYa3BNExqVxoSSeGiIIkiiIH/jdEhG0BlIGXZu+9KGEOHdIKhXvAK8jSDV9iTBWX6MfRm3RdGmxa6xKxTpCM70MqEbPbmErzucplbQGMukmFPPZqAVEoWkuXy0jK1L6tkE55rAb4taXc1YY51ldz5DjxSnlpcYDQfUtsHWNUqFBLGmaUK02Tmct1RlSVmWgef/Ot3gxTKbin1axVGfYA9UeCrmzQ7zZpPt4nHqeMDyBeGsHhANFcnanMHyMoOriuiEGG/W3Ujqv4E+4RzGGJrGtMl1TesUh++UksDpVkLQO+6Gtj50Xt7jcYh4RAUeX5pkyHAFVxu2J09Q2h08L3B09nnECmc5NTjHKMkZxDFtoBilHEoHBzlJNHEWEWcxUZoS6wGRDFCSARlRlKKTDJ2k6DhBohiJdDjQicFTlxWzaUFZlK0yx0liMWrczSUdd/jhPm21F4vWdAMdBRIJKhK6XExXWlxlwyu+6Dc9ElGaG9FUE5qqoFYJuRqhlgckaYaOE6I4IktiVJIwTCzDWNDNDHauwGwS8mLSGJZOhYIekoHO8cMcN1lBa02SOWKjsJNtuL4FV9dhfSMU/PBtNTbdgLah6pdygcNsW8qamDACc8B0FhL08CApWR5z4akLvO3tT5MNRswqS103iK+ZT3cYzwp2N29URMqynHe845289a1v49z5xxjvXmW86/AOsjjD5cNDrHR86B3jFs5arm5eY3N7i5devsRnXfssdBTxlihiOp2wvb1LVTdEcUQc32c8Le9xxS7F9hUmuUdnDaLH+Pl1iu1rmLJGJVt40Wxtl2xteZp2Lr8qoJgFygRl6AJr/0i2PSApoodosWhfY5wOkWFJQ6NQF4Ea0BDmhhwtvUKDVGG6ye634I4wwPgEgQ6xQUiO22iXU4TUmJRAodAE33pOiNY37d+d65cSHOSuTs7xYkSIad8+jAKDx7ZTatnSEudOneLsaMSpLMWahs35mGk5p3aW2lmKxlE2lroxbGxus721SxQneA91MSVLYqq6DtPbVYXxEGkhajRVVRFHEcsrq+jkNLbqZnC6bP9RkNrLNKJX8LO0lR26lSxW1m6z+OQ31KZmd14xLSrqqsIrg1GC0XKiegLOmH1Zto5H7FulBufaaHGzFyXuluA8K5xze9Fm5zzOBp5xp2SnRFASEnHQINpjVUMSDRhGK6T2POXeU3srKDLOsBY9xtLSKZayAYM0C79ZJK1T7NDt3zpWxHEYPEUqIyJHyBDSoF4RpWHwpKNQF1bUHY/bjgrbNBRFRVOVnKxU294ZCYSr7s1/yKBAYkEPYpLRiHx1leHKafKlVZI0JU5itHLgHZHyxBriOHwqDPPdHaY7O8zGE4rxHLwljQUljqaumIwN61d9EKV56OGBBussRZXgvEJUhKhQLl0rTaQE7RpcOaXeuAzbz4YCBDiYaxgv45JToEeQjKhdSZl6BtSkqw0DH1EW67B7Ba5egmuXwE5bx9js0d9A2mS+mr02VQRUDCpr+8AgWBqvjHj729b4lHe/nXe9+50MhkvM6zALGIllNt0GBR/64K/TlPuD0TwfcPbcY5w5c5aV5TXm0y2a2mCMw6MRdbKua+8YH0DjGl7deAX7q44nnnqS0fIS48k2W9tbFGWFUpo4Tu71Zb4OtphQbF9hV5fokSNKC1yxyXirpJyCl5rGws4YdnZbVkDccopdO0Xv92NDx8V4e5AgkqD0AK0ciiQ4xT4Ko+TEhkgxXeNA6yDroBUpLiQeuAJ859bub9apV2wRosnXaIt9EJLuOvm2gv26bG1ztMfUStrtMo47r1wBT4I8cUd7O2nzOSJB0ogLZ07xKW95mrecOsVjS0uAY6cYMy5nTKqScVGwMSlY351xeX2DancbM50Fbicesghzeo2q6RzjEu08RglRozB1TRrHrK2tka9cYHp9K+zHiFByOoNhznBtQD5ImU5OUe7OQrlornE4GSVBcQbHJq+XdzPMi4p5WVLXBcSeBo+RkxXaMiY8Q84HnmEXKcZ5vHdYGzqY4BDvy5sFdQWDs/u0A0+gT3Quv1KCahUfpI0oO+1xUUYcVQzTVZbr8zR+ht0rV+MWlk4nBYSYjFOc5S2sDR9jtHSapcGQYZYRJTqUdY2EKPIo5UO9AfGIBMdd+wTlM8SnKJ+iOodYx3tSbqFwidtLrD8JWGsoixpvuzfvVjiO8MFipsFDgiGkj8ecvnCKJx5/nCefeDtPPfFOLpx/mrOnH2Nl+VRX8oWmqajrGc5UYGvEN3garKmYj3eZ7m4zm+wwn4zBNySJYF3JzmSdV157lZ/7uevsXr/XN/xmwlE3Dd4L0oXWlaCV4AXENthygtu9Am6dG1onW0CxBaQgyzipqQZCHnuyBKxKUbOruJ1LsPkcmOcJA7eaMNOkwgLscdj3EtE12CH7IqURJKusPPkY7/6kp3jve9/NJ3/KexgtrVLUIQk4T4TpZIvd8S4XL17i+qv7P2SWDVhbO83q6mmWRsus64S6spjGI0RodbI+WO8Y3wTXtq7ygZ/7ANe3rmJdQ91UFGXFdDZndtLq0ncAaxpMPcGUgtUe42t8VeGqEOgsKyiqfZUJR2AAzHxw2Gr2SyI8ZM30keEcVE3r8BgflFUa1yYtEargGR0WG9RKQr+m9kfTvivnMWWvQWn/OydQK3YJhTy6QsUVcJ5AlUjZT9Trihd38dCSQKUYtf8+PodMhTPfodeRKkU+GHJ+ZZmlPOeZx57gnY89wenBgFESA5azzZDS1BS2YV43bE9L1sdTPpoP+biD0hiYbwXt42gY8s1ciIjiHTrSJFGYrcnTnDxOyXXU5tgbwlDCAXXQUS5qbB4ypb1poOkoFTd7ug2OMYfLZlmsraltTeObvU4dUXvvzIlwjNvCFtBGfPe0fR2+VZfwbQkoJeCD2C8irfRYq0KxiDCLrdDt9qICRSXyQOKwcUYWpWhisjhnNF5iy5yj8RMUNUKJYo60tlR4MlYYyXlWssdYWTpLPlghzjN0mhInKvCKYyFqo9KIa53i8ASLSxGfIC7B2xgvEV401gviPdhwz421uMPS148J3gvOtslER2oFH7k5tVtjFQZv0bztred46twFLpx6nAtnH+P82gqnRimj2JEzx5qGpirx1RxbzrCmRJwJ1Chl8K5G1TMSJkhSkY08SmvSPKZxnkY0etPhTlLY+j6FR4UgjY5amcyg2BPhiXDE3oDtBrIH0YTFF7Ab0WyPqIcpaM/cCu7iS/Dyi1C+QAgiHPYeHNbidRQ2Q+j/BLRmMMhZW13m9Ooqp1dXGa2sUZlwjDxVjAYxTz31Fp5661vZur6NacvfpdkAraNQ1KhdlIpI4pzRYBljsmOw5M3RO8Y3gfOWZ1/4GJdeu0icRERxBCJY65neN5Xv9uE8OFPhzRTXCE4bfOPwNuh8l1OYFlC1KhMlQX5txn5sxPNoPxDeOcraYXGBW1w2UNYEwWsPtWmdZQHbOcd+33EG9otLRARXeMZiA2UINIpp+23N/rgbgmN8in0HuJNpC8zXcLSl9nOx1tvdISI0mHc24BsozcpoxDNPPcEzFy5wYfUM55dWUd5h6xLrDGuDDK/AaY31MK1qdqYFg3SAsY7n6xI73gwjOHKUFpS32CYI38VxQp5lJGlKWdRkOia2Hmk6K7YJIRThfpoB1W6DmIJ6WrWyQrvc3GKOm7O3Hd4bGt9gxeAklGoRidAiexMIxw3X0nI639bvKUrsO70eH+pOiSBa48XhvGDFhW0XxHgFQCmUUkSiEA1RDBohQhF5hQw9bsmyPFzh9OpZHp88wWSyS11NcWaGN1NopmDnaDwKTxQNibNlktEq6dIqaTYiTiJUqgNvNFEtd1QhyuGxeLEgvq12HYdMdx+DRHg0zqvwWlkfouHG0NgGd6LOqGptfZLzAA8xElh7JuZtT57lHecucHa4xkqcMKor1HiLqq5o4phtJZTljPl8QlFMKYoZ3tZBp1x5dOQRbbGmxDYVIo5IPEkWo2QAqqYxY2bljPst3edNgWhERyjdUSmkLR7liXEkYlG3zHqwYDZw4+vU2ym1KZnOK3jpOZh8At4wpfiN3sEuKyZQtpJYMcxiBnlMniUMsoTYabwIeaLQYnn88cd55p3v4rVXr3H10isoFZEPRuAFax22cQiKOEoY5CPiSHC2l2u7Z3DOMp1NXy+Teh9CKRAs3pSY2lM6h699J2OIjiCO2nwyggtwWErS/V6T5USxF6H0QYOxsUEHsi23TWNCXWzTOspO2qmlTp6mIzx0zkg33fT61rsErrCvZNrVClqlLencrhP2lUm7SauUEF3eIvyOHXHjdsTJbkQFXCKoMN/B3ls71HWDHQ6xWU5thVnZ4GxDWc7x3gYlgjgmStJQ1KGoUNM50XROMp2jixJbN2BqqGv8bMbWa1d4Ln+W8anTLC+NGOYD0jRne3OXixdf5tVLl5hOrnJjVKOz3BhflhRV2nLhjhoFPIgg0aeyjDiJ2uIaEn76qJ3q9yczx2IXtDq9DxH0Lloc1AYc0tIqhFbVbS+ZSSHiUbp14lWIDotSOK1xSuNVKBiilSbTMZFolFOIhXqloi4qyqKgLArqaoqpJ9hyiqvG+LpAe4dCUHECSQ7JAElyVJIQJUGFQkegouCEiwYR1zrFliBLLcExJgEbnGLvBGy4H4dDMGEuoDL4E4wYQzcYaeijwbcPSeHsas6FpWXOZTmrOmbkIG8scVEjVuGiCieepphhiim2nOGqOd4Z1KLSnLOhpLFvQiJpLMSZJsk11kagJaR5PILTm9ZWVLXBedrCN1E7S2TxrgFXI0dq62p8XVLPdqmqErM7hekr3Fl6tyX0YEIXNZYoJU9j8jQmjSMSLaSRRhMhSpHGgibjqSee4NM//TOwRvjwr3+ENM741E99L29561tZWl4JRX50RJxkDPIlfJZyqPDxMaJ3jB94CKI9aRKSRbGGumwLNTZBfk0iGK1AOgK7ATvV8QgDPXRwdk8QPdAkbEgk6P5dN6HqiW3LuLoFftWeY+xgb/qna+kPR0ngHltCLPMKIVq8RuAbL6qbZuxHk6t232m7TBY+x+wXBTl60+EXjnb72Lr4MkUcYy9fYXN5iXOnT3NmbS1oQjQ1WivSLCHNM7I8R+uY8WTOxtYur7zyKpsvXcSsXw06gdbCrMBe3+RS+THcpausraywsrxMng9I0pzJzoSP/NpH+fhzH6Y2G29wT1WbOHI30Ei6xOjUKQaDAQoJtAYFLolxEuFOKHHKVYHr3CXg+Y464bvfNvxb4UFkj7SjcIhSRNJVlgudp4o0KooQrduqcxFKa5IoJotSYhWhUYgVTG1o6oaqqqjKiqaaY+opppwG57ieI86hPHgd4aMYq2KsRDgleOWQllOsFYhqqUbKozrN084x9hokxovCeQEveBPUM8R5xIbod1M0+BPMtAoRastDmQh3U3SCkaEYy91AASMHA+vJjCdTjtRbUrEk2hNrjegYrzVRokmJaeKcJg9lqlWbpKm0R8RSmYLKFCjtiRNFNkzJlwb4piRKB0En/RGkUhhXMi9mGGsQrdFxhBePsTWmLrDVHDnyYN3TVCVmPA2ln/3k1rvc/MoIPVkJrKHTVQZZRJZGJLFCa4i0R7eD9Fh7oizmqbc8Sba0zJNPvZXP+dzPJ1IJj58/z5m1NUajGCVB9CBNM7TygZJ1wr977xg/sOhiiRGiw1SyrfdjHeKD5G6mIBuEiHHdAKp3im8K6eSs/P78NYRosTX7DvKeHu/BTlrYL/7cOcYt3+oQi3uCE1sROMfXCU7xKoEuMSQ4xDH7jnEn8bYCnCE0RZbwuxftcTYJZaWvEcgRJz0LsPnSyyhvWbcNH1fCyvIKaytraK3w3pGkCUtLI5aWl1hZWSWJU66vb3L5ynUuvfYa6xuXcdUsRF5FYDrDbG5zmVe5Loo0W2K0copsMECpiPl0xsZrF2mqm3HgjhMKleYsLS+RZxlaNHiHNY7GeUrkxN4lU+wXjwkupA+cqYVOQUSC8kS7eMC3EmzSRpN0FKPjmCgOn6GanEbrmEhHxHFCliQkOkZ5jfKCNTaUkm6Cg2yakqYqMHXghLq6BGcRDxbBKoXxQm0DW9ticNImU+5xituBo7hAo5D2Vtw+j3BvbEmrtexCRrByYEqzX/TkRNANch/mbrHrNzotnAH7hevvrtpfJLAcxyxHMUtRzEApUg+R82gbiogqQgnwKFakKsb4DONyEIfSIcgjODwNuo5CgqiGKNVkSUIaD7A+KJkg8cnWYblv4ZiVM2ZVQWFq5rbGNVAbR1HvUje7+CMGBLx1+LIKmq0HtIfvDF02zC6+yfGmDNQpb7CuxtgSZxtEafBCFCnW1pZZOXWaJy48xew9FQpFnuVECpp6xnSyifMOYw3GehCNVicrUfMwtwAPIbrGLGGfdVriGks1haKEuKu2CyyNYHgahlmgUFQlFPVxKxo8RJBWq9j5tqBHOzTtHGZZzMpfzM6HfW+l61wt+45xx+E9HB3PeEZwbLuuKm+XjH3O8ZDw66cL33UFQxTht50SItFXCeSIrfa4M04mFmbWXwRvqbHM8GxvKF5eKN2rVEScDBiN1jh79jHyfMClVy6yvXUFaw5JEmlN1abSUc+uM9m6gqgYjw0FbfxBWbWTgiA6IkkTkiQliRJ8U2DnJcW8YHaCU3rNJExpSkt5EAEVwqyBU9hSI8ILrwKfCkAEUb6Nvqkw3ao1URSFRUfh73Z9LIoECQVntEd5IRaF05BowSYaazQ2S7AmxzUjnKn3EgGt8xjvMdZTW4fxDkODwYTpXR8oEN47/N4709rNL6hA+XArHr83U+qcx0vQL/X1yVakE9HoeIgpT7OfBfCgohtsdMnAXRGRDGSISNePhKqVvo3U3w2yDM4sDzm9vMyplWUyH6FM+E0dFuss4sMgKYo0PtJor4ldFAZPun0unMEY0DYiMjFaIHKa2KXENkEZCzbGGX3SM+r3LWbllI3JDld2l4hGkNca4xt2y2uUzQbWHKVOrcObGuoI6qqlhB2XHtUcO3mVrfVlJrMnmZYzpuUEF1mqusaLZ5hlDAZD8sFamA1MEgZ5gqBI4gjvDLO5h3nDvJqwvr1BWZQoiUO28Qmid4wfJEQXIDobpvvNmOD2NOBhXoZnxRBG7lqCpneawnAkmB1PXe2XeO5xCPYib21Ubi+rKkzr4lpu8YJU1Y2OsVr4ruuYug5poerbTdA5yOOFoyUEB3jEfpGPYbussq+F3EWYWzlqThMKiXT0ii1C0t8VQiS54BjdytdFJ9wNXFBrDbYoKYtttreuEUUDqmqDozfADvz0Dai8iyVSjhud8oMKzikKW1vqacF8VlGfYM9cz6fBGW4T5rQIXlTrKAe5NYkUgt7X+m0XpRxKBVk2aaXZtAiRUsQSaBYRiijsjXYOJRYl+362FyFSGu81Lta4JMG5DG8t3pn9Cny2i+Y4EuswztLQhGRFazA2lKi2rSPdOcbOhU/ZG3h6VEu3cM7h8WBDgqFzDlfZE+UWKhWRpEMqfa6lVHRpsrdbXvzNRjcY7+aWhkCOqBylMnSUEkVJoNPoGCUpSBwSHB0Y67DGvk7B5HaRZ4pTowGnlpdYW1oicgpbGpyPwnMaRBRQWrWJmO2z5TQiDqUceIutBbwjchrrIrSF2Goiq9FGI43GNxrbyCObI1nWJa9cu0Iy9BRMGC1pRBk2JuvMqk2sPYphPK6e440FM+WYewWwNdOddeblnFk9Z1JOKV3BZLqD9YbVpRXW/CniLGcpPoVKI1wSRA60FoyxVI3HUTOe7XL1+lVmswqtsyBVd4LoHeMHCdaCnYHfJWgV7OsSNMC2DwyfyIcIYj4PEm1J6tmZwNYkBEM1j3iS3c1gqlDtBA9mgU/ctP82pu2YFx3j7gXtIqSL33cdVsp+UtitdUsX2cqdzsKY/cjxkH1neAU4S3CAV9r9um48ITjIZ4GnCU73DoFm8XHg+VteyXHDY+0Ua9+Iy3w74mcanZxlafUCggT5J29BgzE19XwHbyfc3fRgjZ3vsL07YTweMh0kNJM5460Z87nF+ZjDkiuPA64uAz1CKbxWe86vqCDP5HRw1AWHiEZ53UZbPd7plpPs9pJHfevIekJingSXM0xzWodXFq8VTulQlAwISg1CS2NGKRWeT6UQ53DaIdainEY7R+TAeEtEhHYRxjRoq7GmO7drEwkdztmQ7OYs4kKhh/B+hQRH72idYo+0lf5OkkohEpGlOVW+RjUXcJ3+S8X+u+sO+eze1sNmkmB/4HYY/epIV8Y+xSMCiRGJQRKUSlAqRUUDoihH6wE6zoh0jG5nBrRWiEirZCJYo7BWYYzH+OCMhuIxd+sYC6M8YzTIGAxzYmJc5sO1xgNUkoWKhnEcGDLe4h1oHwrNxFqBFxpvUVZw7bOmnCe2HtWEwZGtG0xlaGr3yEaMnfNcvbrD9s6YFy4mrJ2JGC4L42nN7qQ+Yj6wx5ut9t8n4xHYxuBQVMaxO53hfM3G5hWMqZmtnaYxliw/xak1QYlu5RhdyDNwDmstxbzi2tUNPv7xlzAWVlbOEid9SegeHfyc0EhvcbPOvpt+NsC4gnkRJJl2JrAzC831yZYVfoBhKphPQ6TY2eAc2zo4xnUdMhn3HN6us9PsTz/5hfWw36El7b8T9mvbHdIJLR52ocHvXOqGEL+K2iN1TnLnGK8unLGjXqwSeMtdARFL4DI/3x7rlds00YlCMojWoNnhxiIbh0Ol53nne7+Az/uczyWOE8p5QWMbvBi2dnZ47vkXufziC7jiZe7ceXXQ7DDe2OT6IGGoPXYyZXdjwrwweHL21Ug6QszxwJomDBO0CkoNWoOKUK6NHHtB0UbbxOPF45XGS0jUw7mQBOcc3trgaCjd0hmC84lvK+K1+saujU4rLSgV5i3EK3wrURdoGjo41aJCgpxSKKeJCBJy1js0GuU0xmgaG+GMwdoQ+e2q+FlrsdKx5FuJNO9wtNxkWl+5laYz5iSpFIGukqQZo6VVojilqdewpsa5JmT7+9YZ3nPgD3OMF+6lPe7+TFKnpb3Vfr4RIiBH9DJRshrk8JIBcZIHSk+SEEVxW4k1IY4SdBSjJArRtDZZ0zmHs47GGJqmpqotVRnoL95brA2a7cZ00fw7Nh9ZrhjkKVmWkmcpSZThidBRgkpyVJyi4gRRUahqWVm884gPVeJjpREvaGsQo7FKhefSB46yMhbfNJiqoSkbTGkf2Yhxh6p0XLtcsr4OyUpgw9h9+fwj4HYcYgGVtjk43Xv7xnDW4dBUxjGezimrCVeuXaOuC+rGoHTC6dNFYC4C1gQ5R7xgTENVNsymBVevrvPSi5dRUYzSSwxHfYGPHnvotAVvHQFzwMRBaWBJQrXGOIO8rXFwMmWFH3B4F154z36S3V5nuN9Z70MWls45Vod83yXidaJqN4mK+sNXH9ykc5JLwhNREty+FfbTalYINAthn6/cTbYusy8LdxzQyTmcrfB2Bhxmp6NBRTnR8hr1Vg2+ZJ+GUnBYAy44BsOcs4+dZXlpCWc7HqvlyrVrXLm6weXbikDfbNsxzcaLXJxeYf2lhMR5otpTVo4wNEkJTvGtnJ3bg7e2Tajz4BWCIN4ELWLxIAprPEo0ogTv24IjHc1Cqf0KdzokrKiFRUv7fTcak86ZCurIzkk4Dy2lr9VM7cpMe/GID86xU4Hy4PEo30WFW1sqwSmNtkFqjlaH2VqLs60sYmRxkcFHFtsYxAgYISglC851/MeTgm+j2GFwG5Q8knBvSfIMAAAns0lEQVROF+FcEmYjuoiW6+hVnVPcvbzdjFD3HHX0FtlvXxiyz/rv0qU76lXgAOtomWSwwmC0Qj5YIsuHJElKHCdoHRFp3f6O4VpFtW2MD5H2UA0xDCZMY0IFybqmKhuq0lFVDtOExZo2sfhuegTpUjLCAEMpRZqmxOmANB8SD0bEaY6KQtn3yWTKeAKmrvC2aWcjJDjGSugmSBQds823zn7Qta6bhqpqHtmI8UG4BsrOPTix6WAP7vbauLo0TCclk3FJkqRh0KwykkSjVIazEVVpmewWRNrQNKGPVMozL6ZcvbLOq69cYWN9m/msZnk1Z2l5hdW10ydziy16x/iBwu1FvQoHVRPa5SyHpaWQW+ZLkEdRGP0o8LRc4m7xB3jFXVT4YCfdxWph30FedJg7Sbd2OvQY0uC6LnidEIPuXLQBwSnu1EcW2VgRe9Xtj03p6C1PP4OxhqKYU5dz6npOU8+xdecoHw06ThiOcuodHQqoyCoqP4UrrrT0oRvhzIyimGG9JU5j4iQh0gqFZz6fh4hpc7Nqdq87Oyp9S9AKbrbYrzdIuAd/DVvAuAAQIhIiVtHpKYwDmisct2PsfJhSDHcUFH3DbyagAr2guzzRCvF+zylWWlBaobVuFShahYo24S5so4Kz3CVKttX0nPeIDc+5V4Tz69Z1E7Ugs9Y6uXtXGBxjnEU7T4THR+CtoMThtbuhGErHUSYO2uCuMbjaYKIaqTvqiAVRNMbsJ8OeUNDY2pq6KqjriqauA9Wj1Y7G21YtobV5JxwNIUy3d00LeQbSkrW7Qgxt4qR3A/BnwNcgDYJHqaAekqQDsnxAludkWR6cyyQjimNUK72HhOegK/RiTNC0dRZsWxDFNE2IEtehrHpd19RVTVMZmsZhG9+O9Rej3XdnWKEdDLUVG5M4YTQaMVxZI19eJR2MiHSMsx6tNU3TUHiL7YIRBHuFZ/hG89I2w9aDdZa6qalK0zvGizhaEPdNRV3WbG9N2N2eEScpWarIs1W0FgbZCKVyyrlle3OMVhG2pVuJOMaTbS5dfJWXPnGJzfUdrPXkgwFnzp7l7LlzJ3rdvWP8EKMBtsYwzKGYQlNAWcC86aPFh2IxALzImOg0WP3iRouO8cEOpXOKYb/TEfZLF6ftd8dTSMASqDFj9hP2tgjR5C5G3RUQiQnx12sc36T/uXNnWy6oxVhLWZbM53M2r19lPl3H27Z64C3u1buGaroLbgIY8EXY1x/u2Ho74+Xnf52fyQecPXeOwWBIlqaksWZ9fYPrVy7jzRtVcFqADFl98l0AzLauU+08D/5mFvIYKgzb7fjG88aVou4MdVOHcq9KhQjm3ginVabwgU6xx7vdV21rI3cEJ0raL2EvWhscYBV0iPGtv+nbMWAb++3GhWKDc9IevHPwOgdwb6LDd8dol8Xj+Zb73F6r9/slrlX7rokKzrxyGhU5tG9JHwLae0QvvlfHj7p8ls2rV26MDLN/n4dP6Rxct/C3J3h1LkS9fdd2yOK2rV9tBWcUttaU09aRVrqNBketKonccBa827f1go1xDu9tu4R/O2+Dw3rDPR28j7uAh7r21E2NaRq8ce0MTjtYAuhmMiQM4Gjl+PZnEdgbnFlnscaEaHaX+Kw83kBjGqqqpCrNXV/2Q4X7sFOvq5KPP/ciZVlx+sway8s5WR6HJZ2zmU1ZvzzmYv4qgoRZJB8KAE2nu1x65WUuvvQJrl6/jLEWrRRxFBHH8Yled+8YP7DopHhuXr7UAldboVxrAq1iTKBR9O3JIVAqhNTNQvEO1UZ99pyLxeIdEOzf/QaLU6JdJ7ZIHO7YwUJwUbtSHcc3zO9UKTrHt6ObzQl0CtWuv0Zwno8Do9EACOoHXmj1b2tWV1cY7z5GOS8wdY0xBmMcxjSYZoY1Bd539gNTTTF1Ab7jF0/xVcPNZ0oc852X+NDPXkVFOSrKiKKMOM4wxlBMulj6raHSZc6eP0OWxszWBlx6dpt6equhQw3+evvv43+jiqJEiQR5tSgijiJu4NgKiFeoBS3lThbNudYRcqFIjRcTWEKdg6qjUEVPh7LWi1HITjEiJL15lDiUDxFh5d0NdIrueny7r3UmqFJ4Q+Nt+M2twdrw707izXm/V8VPAdrThQPD9UGozKck6NtG4fNkMcW7Oytyc1s4xCft3OkHtpKbh2LuqMqSumloTENdV8xnc5yKqK2QVQ1ZPkCUtNUUS0zTYI1FKWn5547GtvrZTUNdN3hv0aoNLCjbOsUFVXkfeoL3Evdhp+6d5dInPs7Vy6+SDwcsLY9YPbXC0mgQ+PFaoQltmDUGY+rgGGOpmpKdnS12dnbZ3Z2F8tC+pqwK5rOTLUfcO8YPHLpqap1z9sYt6diBbWd4O17qffj+3B8Q3eoJ2YXQG90ccui49xzdzjG23Oj8LibbyIG/O03RTsZNt9+XHDcxrIthvkhQKllhn2dcs89wPA6I6LZsp0a0Ioo9cZqTZgNWV9dwJjhpznmMhbqxTKdTJpMJZTGmrsYYU2GNwdlFOoLn1vQEj3dzbD3fK3AT3Orb4RaD0jF5njIcZMTac22wRD09yjFO7m36wMv/NpRsbss2K6XRosO/pZWPUx0fNg5V7VSododKEBWjVILoGFFxcGiV2jtGtHesBUe3vR/nXOvzdlPbXfSy1VQGbozetpE+67DeYrEYAofYOYd1Ifrn2oQw2oQ68HTaGqr1Dn2bGBaUEhzOGayxTIptardzYvbucXcoZ46yrGjqhqYxVFWNVwVWFLWF2lgaa9FRRFkW1HVN09SBvuTbnsw7jLU01tAYszeYcpqgc2wE0zRt8t6j1JPdXnt2P8HZmnJaU0532b6uuJJmJGkaqmHi8MbstRO+k2mUdlbJ+jDRIRAl4FzNZLIbKEUniN4xfqCgCNPwCcGRunW0sZtm73EEyGKEWLih8kCnbdzNGe85xHAjtaJzhhelmxYd5IgbRdQ7B7pmn/hwfKiAywT94oMs6OM602xWB7OojhcapocFHyqpaY2KhEggERXUNJaGnDqzRtOUGFNQlXPms4Ld3QnT8TbedJqanjuTW7u9TsTWU2aTMd42VOWcpr73JPwXig/e5h5yyL8PRlkPi7oets4f+t2tYrb+Df66/Y79VrSFHvcTmgrKMvCZjWmomwanKqzSJBJhlcaLQumIqqoxTRjwOGfQCgyCuHaWwdg2gdDuD8K8YB2hkExtMI9S5e6HBd7RlHOa8vaK56gEsmEoh72xcY3p7GSL7/SO8QOJLvW0VyM+diyU1wXYS7y7wSM42EHfLFu+k29aJC9rbnSeO65URBjoHKVi0e1jb6r2BI49mzV7+UdeETRulRBpRazDdJlohWhNFAVd1VzaEsfe4mmoyoLpdMZoecb2zhnGkznVfI5rZmB32SeFnAy8mbG9sU6RD6jKgqY4fs7wyeMg1/Vkz9CjxyIaA3VlMKbBWkdjDc4YXN3gowZfVyAapQ1VXQc+qbNYG9JKO/65ICgRtFLoSIdWU2vQGica5xymNti7rV78QKGj5y224DcbKB7sjxYHuQ/W4FIU6BjyJVheUejIM5mOmc1PNnDRO8YPFBwPdpnSBwBebgz2drKkh+bJHWxkFikVi5HiziVd1DxdlEfqGrKYB5Hs0ljPXkVZT0iW6YpCeEuXUe5ciPgo1VY78x6PxTuLtYKOMoZLMVG6xNJqQ1HWlEXBfDqhno/xrg4HaqP43lu8mYM/BqfZF2y++nFEkkDnMBu3sXOnVW05maLbPXrc/wi0dodvqTdaayKliLRCC6huIOzAeYvzDhFFFAlxFJFnGZESTBTTJDFZnDBIU7x3KCUYcRRiUOLaiPK9vuM3ETIIVD+/2Oe02bG4/W7HdzTLg3M73T6WMI9457KaJw3RkKSQD4XBUBgMI0bLGcPhCO8HNE2O8z2VokePNwemhOl6SL5r6lDwwzRhjtAddL46Z7ZrqA5KWjj29XM6R3jRKbbc6BzDfae1c0Q03Rhgr/hfcHoD+0RapzhQt4NTTJsp3yVjWZRSxFFCmmvyUZCkMiYk4dRlRVWVGNslOQYemjENs8mE+XRMU8yxTRkEPf2i7Y8Kj6+vtMoBtxNXF0SfYmXlNLiSnZ2Lt7Fvjx4PD6yB6xsll5d3yfQmo2lNHKfEaYZOU6IkDTrGophPZxTTAucdSimyNGGUD0jiCEyDa0qqoqIuQvIdOEpXM24Krl7bZnuzeaSoFEk2QOkk8PJ9y7+l1QXHI3R5Al1D3Eo77qnS+KBigsM7ExZvcL5T/ei2OWpEuevLjte5TnM4fS5mdXXA0vKAfJCQ5wn5YEiW5ZRVxHQmNM3JJuL2jnGPHh3qy3DlGvvO7s04kkd9Ke/PEflxIzis7Knx78UzlGBVlzgWNHO1Vojq1BNC1Mi7IOjuESJAt7JOXaOvY02mUjxJW9jCB+UFa0mSiGyQBY3WusE0wdm2TYUpZzhTtIOao0RI7mSaMSJOMgZ5hmn6LPkejy7qBt7/S2P+4y9/mDj+SCjYQUjWpEvwbClqYUDcvmutvKCSUOo8zAb5tqT5/jvpCVUVm8bTGP+ItK4BokLFSdX6rsH99XsiNUE4qbNd2752zmubQ+7Ft1KLOuzgNeLiUCXzoFO8Z9yFvnAx7gO8vvrjwf5xsZ88mhOd5RF5NiLSAxQpuBhnU7xNcTZFSUySKPQJK9T0jnGPHnvouNtH2a5HB2Nap1O6bOI2ots6uLp1jgNnMDjHgQ6xoMNqbVCtMBYdGZQoGhOy2/E+cM1a+S5RKpxOIM+yoGnZ8RMlyHo556irirIsKeYldVHQ1DXWNPhWEsjbGlxX7uROoEANSZIYpcA/sFpbPXocD7yH2nrqR4rncPLY16ruUhHbqHDrrHb60EFNxgSqijN47/Z0zTsN8qD6At6FapmhOuP+ICWcsDtzW+hK3MKs4L68Y7d9KPgDvqV37BdeaQdDXaXIvVLSr6+0GMcRWbKMtzlVofHWU1eeugoFaYwBFSm0TkMfcoLoHeMePXrcFXxXgU1coBb7ILMlXdR3L+jg9xpyEd+WG/bd//ejHBa8OKyxGBPKxWoUvi2HFZzQtpRxFLWlj4PDrbUmihQiYExDU9cURc18Xu9lzHdZ8818ji+3uPOKdUEOTWsJnOl+wNSjR48TQOcYd2Fb2SsWI3t0iUClcN0OQQ/Y+wWiX3Cmne+c2MWcmBv/SUe/cLROs9pXZiKUm5c9hzxE+/eL+7Rl5X04kBKF961Dblygu1ETuM77bWYUpWiVgo9xFpqmE4ryNLEnaiBukzKVPlmOsexVTerRo0ePHj169OjR4xHGycaje/To0aNHjx49evR4QNA7xj169OjRo0ePHj160DvGPXr06NGjR48ePXoAvWPco0ePHj169OjRowfQO8Y9evTo0aNHjx49egB36RiLyPtE5KGUtRCRzxGR7xWRZ0VkLiKXRORHRORth2x7UUT8IctX3eG5e7vub/+EiPwDEbkqIpWIvCQif/MOzvvI21REvukmz2m3PHYH537k7dpue1pE/q6IvCgiRfucfo+InL2D8/Y2Dduead/99damvyQiX3oX536Y7fpWEfmXIvJya6sNEfmAiHz5IdtmIvI/iMiVdttfEJEvvMPz9jYN2/4NEflJEdls29JvustzP/J2vV1f4Viv8W7k2kTkSeBJ7/0vHt8l3R8Qke8EvgD4EeAjwBPAtwDngM/w3r+ysO1F4FngfQcO85z3fvsOzt3bNWz7NPAfgZeA/xG4BjwNPOO9/5bbPO8jb9PWSXvHwd2Bfw286L3/vDs4d29XEQF+DngX8N8DHwM+GfhW4AXgC/xtNLS9TUFEUuCXgTPANwNXgf8S+H3A7/bev/8Ozv0w2/VTgL8IvB94FVgG/gTwFcAf8N7/84Vtf6Rd//8CXgT+LPCfE57TX7vN8/Y2DdtOgF8j2POPAn/Me//9d3HuR96ut+MrHDv2K6r0y+ICnD1k3VsJktffemD9ReCH7/U1PwjLbdr1J4D/BMT3+rrv5+V2bHrIdr+DoLL+Z+/1fdxvy1HtSnCIPfAnD2z7p9r1777X93K/LLdh029obffFC+sE+A3gP93r+3gQFkIBr1eAf72w7tNbu/6xA9s9B/yre33N9/tymE3b9ar9fKa17zfd62t9kJabPKt33K/d7XLsVIp2GuHbReQvtaHyuYj8mIica5d/LCK7IvKKiPzlA/ueFZG/LyLPt/u9IiL/UESeOOTcX9eG2EsR+bCIfKWIvF9E3n/IMf+eiLwmYSr+WRH5k7e6N+/9+iHrXgbWCSOXE0NvVxCRdwBfCny397651XFvhd6mN8U3EsoQ/e+3Os9h6O0KQNJ+jg9svtN+3lY729sUgM8HCr8QGfahZ/xJ4HMPu/Zb4WG262Hw3htglxvr3H8l0AD/x4Ht/hHwpRIi9UdGb9O99XdaV/5Q9Ha9tz7Y3Xr576NtrxbWeeBl4McIofH/O6HD+AnCtPh/B3wJ8Pfbbb98Yd93A38X+APAFwJfS5hOuwhkC9v9bsKo4V8AX07o3F8ELgPvX9humTASvkQI1X8J8D8QCnX/uTu43/e01/zfHFh/kfCjzgl1Dn8R+KrernduV8J0lAf+IPDvWrtuAz8InO5temfP6oFt8va5/Wf9s3pXz6oAHyBM930OMAI+D/go8OO9Te/Ipn8X2D1k2+9ot/3S3q6H3qMiRN8eI9B6auB3LXz/jwgUv4P7/eH2/j6lt+nt2fTAtscSMe7tevS24iSWu9v55j/e80C0sO5vt+v/u4V1EXAd+N/e4PgaeKrd9/cvrP954DdpOdLtus9ut1v88b4FKIF3Hjju9wEbi9d4hHuNCJ3fdWDtwHffTXDkfgfBkXt/ey3f0Nv1zuwK/JX2vGPgfwJ+J/AngU3gg7RTV71Nb+9ZPbDd17XX8ZV38pz2dr3huyHwz9vzd8v/CeS9Te/o/f8z7Xnfc2D7n27Xf11v10Ov4TsXnr8J8NUHvv9J4BcP2e9L2n1+R2/T27PpgW1P2jF+JO26cF+37NeOYzkpubZ/50NovMOz7ee/7Va0379A+HH2ICJ/WkR+XUSmhLD6pfard7ffa0JU5p/51lrt8X6FkKS1iC8Dfgl4SUSibmmv4zQhQeao+B7gtxKc3e3FL7z3f857/4Pe+5/13v9T4HcRnLfbVk+4BR4lu3bP5vu993/We//T3vvvJXSYn02gWRwHHiWbHsQ3EhqZH7+NYx8Vj5pdv48w/f+ngC9qPz8H+Kciclzt7KNk039I6GB/QETeK0Gh4q8Rol0QolrHhYfJrn8H+Fzg9wL/BviHIvJ7jrDfcaO36cngUbbrUfu1u0Z0Qsc9eNH1G6zPuj9E5M8R1Af+NiFjdpvgIP3iwnZngJjQoR/EtQN/nyOM4G7GUT190ztYgIj8LUK08hu99z95q+2991ZE/gnw/xGRx733V45yniPgUbLrZvv57w6s77b7TMLLdLd4lGy6uN3jhCjRdx9oaI8Lj4xdReQrCNH3L/He//t29c+IyIuE5/X3Av/yKOe5BR4Zm3rvd0Tkq4EfICTcAXyCEEn7NuC42lR4iOzqvX+VkOkP8H+2vNDvJMxe0F7jWw/Z9VT7uXWrcxwRj5JN30w8kna9XR/sbnFSjvGd4muBf++9/0vdCnm9Zt0G4cc4d8j+59kfBUFwrq4Df/4m53vuVhckIt8M/GUCb+aHbrX9IfC33uTE8SDa9SO3OMSxJjvcAR5Emy7iGwjTaT9wq+O+yXgQ7fre9vOXD6z/T+3nezgex/hO8SDaFO/9z0pIwn2G8Kw+T+jUC+BXbnWONwH3nV0PwQeBv7Dw90eA3y8iA+/9fGH9JxOcqRfu4BzHiQfRpg8CHli7HoMPdtu43yrfDXj9COSPLf7hvbcEA/4BEZFuvYh8NnDwh/4J4JOAS977Dx6yTN7oYkTk/wl8O/DN3vvvOepNtFMKX9Oe9+pR9ztBPIh2/UWCdulBysSXtZ8HnZA3Gw+iTRfxR4Hf8LepW/om4EG0a/eOH9SB/i3t52tvdI43AQ+iTbvr8t77j3vvn23v408AP+S9n73Rfm8S7iu7HkRL4fnthEh7h39NiAr+oYXtuv7qJ7331e2c4wTwINr0QcADadc79cHuFvdbxPgngL/ccsn+EyHh6g8est1fJ0xR/qiIfC9hCuB9hA5qMZL4XYQX/mdF5LsIo5gh4Qf9Hd7733ezCxGRryVwYH4C+GkR+fyFr8fe+4+2230dQXT+xwk6fOcJgumfRZhevR/wwNnVe29E5K8A3y8if4+Q2PQMISv9/YQknHuJB86mC9t/FvCpwF/i/sODaNd/Tnguf1BEvo3A+/uk9hpfAX70qDd/QngQbYqECpe/QohkPUOIFjfAXz3qjZ8w7ie7vo9Ah/iP7XEfIxRE+Tzg67vtvPcfEpH/A/g7IhITuKN/muD4/JHbu/0TwQNn03bbLwLOttsAfE7L5cWHvKN7jQfOrrfbrx0r/MlkTn77gXXf1K5/5sD69wM/t/B3DvwvBJ26CYFr8rZ23/cd2PfrCT9GRTs9BHwI+NED260RfsSXCFNF14GfBf7CLe7t+7kxw3xxWczO/HyCk3aN0GjvAD/FHcgJ9Xbdt+vC9v8FIUu2IvAKvxsY9Ta9K5v+3fZZPX+nz2hv1xvtSkh0+f+35yjbz+8Dnuhtesc2/QcEDmLdfn43cKp/Vg+9t68k9EPX23O8DPwr4Lcdsm1O4JpebZ/VX2KhkEpv0zuy6ftv9lz3dr0zu3Kb/dpxLndVEvp+goQSii8A3+G9/7Z7fT0PC3q7Hj96m54MerseP3qbngx6ux4/epueDB5Fuz6QjrGIdCPenyJMs70d+G8JNIZP8cenAvFIobfr8aO36cmgt+vxo7fpyaC36/Gjt+nJoLdrwP3GMT4qLIGX8j0EWZAZIYT/hx6VH+6E0Nv1+NHb9GTQ2/X40dv0ZNDb9fjR2/Rk0NuVBzRi3KNHjx49evTo0aPHceN+k2vr0aNHjx49evTo0eOe4L5wjEXkq0TkL96jc79PRHyr5fjIQkTe31ae6XFMEJEvbp+tLznCtr6VsXnk0L//Dz5aO/7Oe30dx4n2/X2fHF9J7xNB+/x++xG2u6GNX2ifvvgEL++OcZR38zjuoTvPne5/v6JvV+8c98sL/1XAPfkBe/S4T/AFwP96ry/iHuGr6N//Bx1/naCN+jDhiwn3db/0k3eLP9MuDxN+ldB2/uq9vpD7EF9F367eER4ob15EUn/vK/P06HHs8N7/4r2+hvsd/fvfo8edw59kQYR7BO/9mFAl9Q3Rtx03R2+b1+Oej4RF5PuBbwSeaEPvXkQuLkyRfLWIfJ+IrBOKaCAi3y8iFw851uvoACJyVkT+ZxF5RUSq9vOHRCR9g2v6MhGZisj33O/TaHcCEflaEXm2tcdHROT3H7LNu0XkR0VkR0QKEflFEfmyQ7b7uvZYpYh8WES+8lGiZYjIu1o7XW9tcElE/smBKaRB+yxttMsPi8jqgePcQKVYmIp6r4j8BxGZi8gVEfnWh+mZ7N//ew8R+fT2Gd5s3/XnROSvtt/9ZyLy4+2zNxeR3xSRvyQiemH/bhr6mxd+w/fdg/t4pv1tX2rv40UR+V9EZO3Adoe2T+1z9/3tv99HiBYDNN19LWz7uIj8YPs+VyLyGyLyDQeO903tfr9VRP6xiExE5NqCbb9MRD4kIjMR+WUJpXcX9xcR+a/b36Nuf4PvEZHlw29fvllEXm3v/WdE5DOOct+HHOir2/Z+3rb//0RE3nKr/U4Q77lZGyiHUCna+/w5Efm9rX0r2ki5iHymiPyshLb6NRH5FkAOO+mDjL5dvTvcDxHjbyOUUvxcQkUUCNVQVtp/fzfwbwgV0LLbOXDbIP48ofzgtwO/AZwjlHBO2vMc3OePEqa0v9V7f0ve1oMGCXzXfwj8GKEk8FlCJbSYUO0GEbkA/ByhQs5/BewSylz/mIj8Hu/9v2m3+93AjxCq1vzF9lh/h/A7Pf+m3dS9xY8B24SyqhvAE8CXc+Og8+8SKg19PfBu4P9LkMX5xiMc/18QKoD9TeBLgW8hlOZ833Fc/H2A/v2/hxCRzyNUyXoB+K8JFebeCXxau8nbgX9P+B1K4HMIz95Z4K+023wB8AuESlV/v1336klf+yG4QCjB/RcI7+Tbgb8G/DjhGm8H/yvwJKFU7W8nvK8AiMgQ+ACh8tdfa8/5DcAPicjAe/+9B471A8APAt8L/CHgb0gYGH85oZT4lNAm/AsReYf3vm73+w5C+ev/CfjXwCcT3pdPF5Ev8t4vluj9o8AlQnudAt8K/HsReaf3fuuoNy0if4pQIe1/a4+xRPi9PyAin+a9nxz1WMeIf8Htt4HvAv5Hgr1eBLZE5Ayh4tpVQttbEcqM30un/6TQt6t3g5Msq3fUhdCgvnpg3RcTSv/96E22v3jI+vdzY7nmbyU0aJ/5Bud+X3ueiCBk3QB//F7b5ARt/R+BjwJqYd3ns1BmEfhOwLBQZhLQBMf5VxfW/TyhXLMsrPts3oSSjffDQqgj74GvvMn33TP8AwfWfw/ByVi02w2lOReey79yYN/vIwxYVu/1/R+jHfv3/97Z/mcIjt3gCNtKa6dvJjiei23I68rV3uulvdbf3l7bZy6sv+E5WVh/Efj+w56NA9v9V+36Lz6w/qcIZW51+/c3tdv99weu6Xr7nL1tYf1Xttt+Ufv3KYKD8f0HzvENB9uc9u8NYLiw7un2HN92s/teeMe+uP17RAiC/IMD53wboeTvX3iTf79btoEH72HhPh3wGQf2+472Pp5aWDdsbefv9fN6Avb7fvp29Y6W+zqc3eJH72Lf/wz4Ze/9h46w7XcB/2/gD3rvH8okqHb683OBf+oXog0+8FsvLmz6hcAveu9fWNjGAv878Bkistwe63OAf+bbN6Hd7lcItdMfBWwSohF/S0T+hIi88ybb/diBvz9MiOqcP8I5/vGBv/8RoQP71Nu50AcY/ft/QhCRAfDbgB/x3s9vss3jIvL3ReRlglPREKJEq4Qo0X0DEUlE5K9JoHYVhGv92fbrdx/jqb4QeM17//4D63+YEKX75APr/033D++9IUTnn/feL7aTz7afT7Wfn0+Ivv3wgWP9I0LQ4osOrP9x7/1s4TwXCdzb24mUfwGwDPyIiETdQhg4PUu473uBO2kDL3rvf+3Aui8g9GuvdCtam/3r47jIBwx9u/oGeBAc47uptnKao0/pfR0h+vlTd3G++x1nCJSJa4d8t7juFIfb/SoharS2cKzrtzjWQ4t2QPC7gQ8Spvmel8Br/NMHNj04ldlNNR1lCuugLbu/n7ida32A0b//J4c1Qh9wqI1aHuC/An4PwRn+nYSB9Xe0m9zWFOybgL9JiFT9MPAVwOcBX91+d5zX+kbtY/f9IrYP/F3fZB3sX2d3jBvO0zrWm4ec42Zt+u20E91A56cIg4rF5b2E9+le4E7awMN+n8cPOdZhx38U0Lerb4AHwTE+TF+wJIymD+Lgi9txPo+C30XgGv0bERkd/fIeKGwQGrnDIpWL67YIZSEP4jHC77G9cKzDokZHiYQ+FPDev+i9/6OESNFnEjhs/7OI/OfHdIqDtuz+fu2Yjn+/o3//Tw7bhCnnm9noHYRZob/svf8+7/3Peu8/yALf9j7D1wI/6L3/du/9T3vvfxnYOWS7mz0/B53Nm+GN2sfu+7tFd4wbztNGcE8fco6btem3005stp/fRBgAHVz+5G0c6zhxJ23gYe3GlUOOddjxHwX07eob4H5xjCsgv43tXwbOi8jZboWIvIPXT5f9JPB5IvLpRzjmRwj8m3fygP2IR0VLh/hl4A8uZoWKyG8hcNI6fAD4fBF5emEbDXwN8CHv/bg91geBPyAisrDdZxM4aY8UfMCvsa8beVxUhz984O+vJSTrfPiYjn8/oH//7wFa+sTPAd8gIofZf9B+Nt0KEYmBP3LItjW39xueBAYsXGuLP3bIdi8D7xKRPSdARL6QkGi2iG5m5+B9fQB4UkR+24H1X0+YQTsOWbRfJNj0aw+s/xoCb/P9B9Z/eZsUCEDbdn8+ISnyqPh5Anf3Ge/9Bw9ZnrvNezguHFcb+AuEfq2jq3SJlL/37i7vvkXfrt4h7hfH+KPAKRH50yLyuSLy3lts/08II54fFpEvFZE/AvxLwkhmEd9F4ID+lIj8eRH5nSLyh0XkR0TkYCOI9/5jhB/xHcC/PWybhwB/HfgkQgb0V4jINxE4XFcXtvkuQqTl34nI14vI7yHwsN5FSLxZPNanAD8qIl/eZp7+k/ZYixnTDyVE5NMkyAj9KRH5EhH5UkJWviFEjo8Df6LlTf5uEflO4I8D3+m93z2m498P6N//e4f/hhAR+gUR+S9E5P8mIv+liHw38DFCZ/kdIvIHReT3Af/uJsf5KPAV7XP6Oa2yzZuNnwC+UUT+jASZub8H/NZDtvtHhHv+B+17+ycI7+3Bd6pzcP+SiPwWEfmc9u/vBz4O/HMR+eMSZKh+iECr+pY2aHBX8EFJ4v8H/HER+Tvt/fx54O8RBjMH8xYK4CclVDv7GoItxoR34KjnHBNUGv6qiPw9Efl9EuS9/oiIfK+IfP3d3tcd4rjawO8CZgQ7fY2IfBXBySuO93LvG/Tt6p3iXmf/tXlbQ0Ji1zbhh7nIfvbkl9xkn68icFcK4NcJhPD3cyDbmDDV/72EaZSakEjwA0DqD2RPLuzzTgKH5heA5XttnxOw99cRFCYqwoju9x+0HWGU+C8InUVJiGB82SHH+vpDjvUhDsl6fdiW9tn6AYI03ZwwvfkB4Evb7w99htnPVn96Yd3NVCk+FfgP7XN+lSDDo0763t5kO/bv/721/2cSBr47rT2fJdAnAD6D4IjNW5t8K8ExOfj8/jbgV9q24oZn+U28jzMEp3e7XX6EQAHwwDcd2Pb/QXBuC0Kk9LN5vSqFJkilXScM9P3Cd48DP0RwGiqCZNU3HDhH954/c2D9+4GfO7Du6XbbP76wTggSes+1z+6V9nqWD+zrCbzvv9b+RiUh6fAzDjnv+xf+7t6xLz6w3ZcT2pxx+7t/nCCX9slv8u/ZvZs3bQMPu4fD7Lvw3We1tikJVIxvISSH+ZO8l3ux0Lerd7xIe8E9ehwLRORJQtb1d3jvv+1eX8+DCtkvMBD7kHDTo0ePHj169Dhh3A8FPno8oGh5iX+bkHG6QRDU/28JUYYHRpqlR48ePXr06NEDese4x93BErKmv4fA2ZsRpqn+kPf+buRgevTo0aNHjx493nT0VIoePXr06NGjR48ePbh/VCl69OjRo0ePHj169Lin6B3jHj169OjRo0ePHj3oHeMePXr06NGjR48ePYDeMe7Ro0ePHj169OjRA+gd4x49evTo0aNHjx49gN4x7tGjR48ePXr06NED6B3jHj169OjRo0ePHj2A3jHu0aNHjx49evTo0QPoHeMePXr06NGjR48ePYDeMe7Ro0ePHj169OjRA+gd4x49evTo0aNHjx49gN4x7tGjR48ePXr06NED6B3jHj169OjRo0ePHj0A+L8AOP3Z7In7Z58AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "from matplotlib import pyplot as plt\n", + "import numpy as np\n", + "\n", + "label_list = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"rog\", \"horse\", \"ship\", \"truck\"]\n", + "print(\"The 32 images with label of the first batch in ds_train are showed below:\")\n", + "ds_iterator = ds_train.create_dict_iterator()\n", + "ds_iterator.get_next()\n", + "batch_1 = ds_iterator.get_next()\n", + "batch_image = batch_1[\"image\"].asnumpy()\n", + "batch_label = batch_1[\"label\"].asnumpy()\n", + "%matplotlib inline\n", + "plt.figure(dpi=144)\n", + "for i,image in enumerate(batch_image):\n", + " plt.subplot(4, 8, i+1)\n", + " plt.subplots_adjust(wspace=0.2, hspace=0.2)\n", + " image = image/np.amax(image)\n", + " image = np.clip(image, 0, 1)\n", + " image = np.transpose(image,(1,2,0))\n", + " plt.imshow(image)\n", + " num = batch_label[i]\n", + " plt.title(f\"image {i+1}\\n{label_list[num]}\", y=-0.65, fontdict={\"fontsize\":8})\n", + " plt.axis('off') \n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 使用Summary算子记录数据\n", + "\n", + "在进行训练之前，需定义神经网络模型，本流程采用AlexNet网络。\n", + "\n", + "MindSpore提供了两种方法进行记录数据，分别为：\n", + "\n", + "- 通过Summary算子记录数据。\n", + "- 通过 `SummaryCollector` 这个callback进行记录。\n", + "\n", + "下面为在AlexNet网络中使用Summary算子记录输入图像和张量数据的配置方法。\n", + "\n", + "- 使用 `ImageSummary` 记录输入图像数据。\n", + "\n", + " 1. 在 `__init__` 方法中初始化 `ImageSummary`。\n", + " \n", + " ```python\n", + " # Init ImageSummary\n", + " self.image_summary = P.ImageSummary()\n", + " ```\n", + " \n", + " 2. 在 `construct` 方法中使用 `ImageSummary` 算子记录输入图像。其中 \"Image\" 为该数据的名称，MindInsight在展示时，会将该名称展示出来以方便识别是哪个数据。\n", + " \n", + " ```python\n", + " # Record image by Summary operator\n", + " self.image_summary(\"Image\", x)\n", + " ```\n", + " \n", + "- 使用 `TensorSummary` 记录张量数据。\n", + "\n", + " 1. 在 `__init__` 方法中初始化 `TensorSummary`。\n", + " \n", + " ```python\n", + " # Init TensorSummary\n", + " self.tensor_summary = P.TensorSummary()\n", + " ```\n", + " \n", + " 2. 在`construct`方法中使用`TensorSummary`算子记录张量数据。其中\"Tensor\"为该数据的名称。\n", + " \n", + " ```python\n", + " # Record tensor by Summary operator\n", + " self.tensor_summary(\"Tensor\", x)\n", + " ```\n", + "\n", + "当前支持的Summary算子:\n", + "\n", + "- [ScalarSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ScalarSummary): 记录标量数据\n", + "- [TensorSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.TensorSummary): 记录张量数据\n", + "- [ImageSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ImageSummary): 记录图片数据\n", + "- [HistogramSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.HistogramSummary): 将张量数据转为直方图数据记录\n", + "\n", + "以下一段代码中定义AlexNet网络结构。" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "import mindspore.nn as nn\n", + "from mindspore.common.initializer import TruncatedNormal\n", + "from mindspore.ops import operations as P\n", + "\n", + "def conv(in_channels, out_channels, kernel_size, stride=1, padding=0, pad_mode=\"valid\"):\n", + " weight = weight_variable()\n", + " return nn.Conv2d(in_channels, out_channels,\n", + " kernel_size=kernel_size, stride=stride, padding=padding,\n", + " weight_init=weight, has_bias=False, pad_mode=pad_mode)\n", + "\n", + "def fc_with_initialize(input_channels, out_channels):\n", + " weight = weight_variable()\n", + " bias = weight_variable()\n", + " return nn.Dense(input_channels, out_channels, weight, bias)\n", + "\n", + "def weight_variable():\n", + " return TruncatedNormal(0.02)\n", + "\n", + "\n", + "class AlexNet(nn.Cell):\n", + " \"\"\"\n", + " Alexnet\n", + " \"\"\"\n", + " def __init__(self, num_classes=10, channel=3):\n", + " super(AlexNet, self).__init__()\n", + " self.conv1 = conv(channel, 96, 11, stride=4)\n", + " self.conv2 = conv(96, 256, 5, pad_mode=\"same\")\n", + " self.conv3 = conv(256, 384, 3, pad_mode=\"same\")\n", + " self.conv4 = conv(384, 384, 3, pad_mode=\"same\")\n", + " self.conv5 = conv(384, 256, 3, pad_mode=\"same\")\n", + " self.relu = nn.ReLU()\n", + " self.max_pool2d = P.MaxPool(ksize=3, strides=2)\n", + " self.flatten = nn.Flatten()\n", + " self.fc1 = fc_with_initialize(6*6*256, 4096)\n", + " self.fc2 = fc_with_initialize(4096, 4096)\n", + " self.fc3 = fc_with_initialize(4096, num_classes)\n", + " # Init TensorSummary\n", + " self.tensor_summary = P.TensorSummary()\n", + " # Init ImageSummary\n", + " self.image_summary = P.ImageSummary()\n", + "\n", + " def construct(self, x):\n", + " # Record image by Summary operator\n", + " self.image_summary(\"Image\", x)\n", + " x = self.conv1(x)\n", + " # Record tensor by Summary operator\n", + " self.tensor_summary(\"Tensor\", x)\n", + " x = self.relu(x)\n", + " x = self.max_pool2d(x)\n", + " x = self.conv2(x)\n", + " x = self.relu(x)\n", + " x = self.max_pool2d(x)\n", + " x = self.conv3(x)\n", + " x = self.relu(x)\n", + " x = self.conv4(x)\n", + " x = self.relu(x)\n", + " x = self.conv5(x)\n", + " x = self.relu(x)\n", + " x = self.max_pool2d(x)\n", + " x = self.flatten(x)\n", + " x = self.fc1(x)\n", + " x = self.relu(x)\n", + " x = self.fc2(x)\n", + " x = self.relu(x)\n", + " x = self.fc3(x)\n", + " return x" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 使用 `SummaryCollector` 记录数据\n", + "\n", + "下面展示使用`SummaryCollector`来记录标量、直方图信息。\n", + "\n", + "在MindSpore中通过`Callback`机制，提供支持快速简易地收集损失值、参数权重、梯度等信息的`Callback`, 叫做`SummaryCollector`（详细的用法可以参考API文档中[mindspore.train.callback.SummaryCollector](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector)）。`SummaryCollector`使用方法如下： \n", + "\n", + "`SummaryCollector` 提供 `collect_specified_data` 参数，允许用户自定义想要收集的数据。\n", + "\n", + "下面的代码展示通过 `SummaryCollector` 收集损失值以及卷积层的参数值，参数值在MindInsight中以直方图展示。\n", + "\n", + "\n", + "\n", + "\n", + "```python\n", + "specified={\"collect_metric\": True, \"histogram_regular\": \"^conv1.*|^conv2.*\"，\"collect_graph\": True, \"collect_dataset_graph\": True}\n", + "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_01\", \n", + " collect_specified_data=specified, \n", + " collect_freq=1, \n", + " keep_default_action=False, \n", + " collect_tensor_freq=200)\n", + "```\n", + "\n", + "- `summary_dir`：指定日志保存的路径。\n", + "- `collect_specified_data`：指定需要记录的信息。\n", + "- `collect_freq`：指定使用`SummaryCollector`记录数据的频率。\n", + "- `keep_default_action`：指定是否除记录除指定信息外的其他数据信息。\n", + "- `collect_tensor_freq`：指定记录张量信息的频率。\n", + "- `\"collect_metric\"`为记录损失值标量信息。\n", + "- `\"histogram_regular\"`为记录`conv1`层和`conv2`层直方图信息。\n", + "- `\"collect_graph\"`为记录计算图信息。\n", + "- `\"collect_dataset_graph\"`为记录数据图信息。\n", + "\n", + "  程序运行过程中将在本地`8080`端口自动启动MindInsight服务并自动遍历读取当前notebook目录下`summary_dir`子目录下所有日志文件、解析进行可视化展示。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 导入模块" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import mindspore.nn as nn\n", + "from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor\n", + "from mindspore.train import Model\n", + "from mindspore.nn.metrics import Accuracy\n", + "from mindspore.train.callback import SummaryCollector\n", + "from mindspore.train.serialization import load_checkpoint, load_param_into_net\n", + "from mindspore import Tensor\n", + "from mindspore import context\n", + "\n", + "device_target = \"GPU\"\n", + "context.set_context(mode=context.GRAPH_MODE, device_target=device_target)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 定义学习率\n", + "\n", + "以下一段代码定义学习率。" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "\n", + "def get_lr(current_step, lr_max, total_epochs, steps_per_epoch):\n", + " \"\"\"\n", + " generate learning rate array\n", + "\n", + " Args:\n", + " current_step(int): current steps of the training\n", + " lr_max(float): max learning rate\n", + " total_epochs(int): total epoch of training\n", + " steps_per_epoch(int): steps of one epoch\n", + "\n", + " Returns:\n", + " np.array, learning rate array\n", + " \"\"\"\n", + " lr_each_step = []\n", + " total_steps = steps_per_epoch * total_epochs\n", + " decay_epoch_index = [0.8 * total_steps]\n", + " for i in range(total_steps):\n", + " if i < decay_epoch_index[0]:\n", + " lr = lr_max\n", + " else:\n", + " lr = lr_max * 0.1\n", + " lr_each_step.append(lr)\n", + " lr_each_step = np.array(lr_each_step).astype(np.float32)\n", + " learning_rate = lr_each_step[current_step:]\n", + "\n", + " return learning_rate\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 执行训练" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "============== Starting Training ==============\n", + "epoch: 1 step: 1, loss is 2.3037791\n", + "epoch: 1 step: 2, loss is 2.3127236\n", + "epoch: 1 step: 3, loss is 2.3156757\n", + "epoch: 1 step: 4, loss is 2.2910595\n", + "epoch: 1 step: 5, loss is 2.3042145\n", + "epoch: 1 step: 6, loss is 2.3150084\n", + "epoch: 1 step: 7, loss is 2.2808924\n", + "epoch: 1 step: 8, loss is 2.3073373\n", + "epoch: 1 step: 9, loss is 2.308782\n", + "epoch: 1 step: 10, loss is 2.2957213\n", + "\n", + "...\n", + "\n", + "epoch: 10 step: 1550, loss is 0.54039395\n", + "epoch: 10 step: 1551, loss is 0.25690028\n", + "epoch: 10 step: 1552, loss is 0.26572403\n", + "epoch: 10 step: 1553, loss is 0.4429163\n", + "epoch: 10 step: 1554, loss is 0.25716054\n", + "epoch: 10 step: 1555, loss is 0.38538748\n", + "epoch: 10 step: 1556, loss is 0.12103356\n", + "epoch: 10 step: 1557, loss is 0.16565521\n", + "epoch: 10 step: 1558, loss is 0.4364005\n", + "epoch: 10 step: 1559, loss is 0.428179\n", + "epoch: 10 step: 1560, loss is 0.42687342\n", + "epoch: 10 step: 1561, loss is 0.6419081\n", + "epoch: 10 step: 1562, loss is 0.5843237\n", + "Epoch time: 115283.798, per step time: 73.805\n", + "============== Starting Testing ==============\n", + "============== {'Accuracy': 0.8302283653846154} ==============\n" + ] + } + ], + "source": [ + "network = AlexNet(num_classes=10)\n", + "net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n", + "lr = Tensor(get_lr(0, 0.002, 10, ds_train.get_dataset_size()))\n", + "net_opt = nn.Momentum(network.trainable_params(), learning_rate=lr, momentum=0.9)\n", + "time_cb = TimeMonitor(data_size=ds_train.get_dataset_size())\n", + "config_ck = CheckpointConfig(save_checkpoint_steps=1562, keep_checkpoint_max=10)\n", + "ckpoint_cb = ModelCheckpoint(directory=\"./models/ckpt/mindinsight_dashboard\", prefix=\"checkpoint_alexnet\", config=config_ck)\n", + "model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", + "\n", + "summary_base_dir = \"./summary_dir\"\n", + "os.system(f\"mindinsight start --summary-base-dir {summary_base_dir} --port=8080\")\n", + "\n", + "# Init a SummaryCollector callback instance, and use it in model.train or model.eval\n", + "specified = {\"collect_metric\": True, \"histogram_regular\": \"^conv1.*|^conv2.*\", \"collect_graph\": True, \"collect_dataset_graph\": True}\n", + "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_01\", collect_specified_data=specified, collect_freq=1, keep_default_action=False, collect_tensor_freq=200)\n", + "\n", + "print(\"============== Starting Training ==============\")\n", + "model.train(epoch=10, train_dataset=ds_train, callbacks=[time_cb, ckpoint_cb, LossMonitor(), summary_collector], dataset_sink_mode=True)\n", + "\n", + "print(\"============== Starting Testing ==============\")\n", + "param_dict = load_checkpoint(\"./models/ckpt/mindinsight_dashboard/checkpoint_alexnet-10_1562.ckpt\")\n", + "load_param_into_net(network, param_dict)\n", + "acc = model.eval(ds_eval, callbacks=summary_collector, dataset_sink_mode=True)\n", + "print(\"============== {} ==============\".format(acc))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## MindInsight看板\n", + "\n", + "在本地浏览器中打开地址：`127.0.0.1:8080`，进入到可视化面板。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/mindinsight_panel.png)\n", + "\n", + "在上图所示面板中可以看到`summary_01`日志文件目录，点击**训练看板**进入到下图所示的训练数据展示面板，该面板展示了标量数据、直方图、图像和张量信息，并随着训练、测试的进行实时刷新数据，实时显示训练过程参数的变化情况。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/mindinsight_panel2.png)\n", + "\n", + "### 标量可视化\n", + "\n", + "标量可视化用于展示训练过程中标量的变化趋势，点击打开训练标量信息展示面板，该面板记录了迭代计算过程中的损失值标量信息，如下图展示了损失值标量趋势图。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/scalar_panel.png)\n", + "\n", + "上图展示了神经网络在训练过程中损失值的变化过程。横坐标是训练步骤，纵坐标是损失值。\n", + "\n", + "图中右上角有几个按钮功能，从左到右功能分别是全屏展示，切换Y轴比例，开启/关闭框选，分步回退和还原图形。\n", + "\n", + "- 全屏展示即全屏展示该标量曲线，再点击一次即可恢复。\n", + "- 切换Y轴比例是指可以将Y轴坐标进行对数转换。\n", + "- 开启/关闭框选是指可以框选图中部分区域，并放大查看该区域，可以在已放大的图形上叠加框选。\n", + "- 分步回退是指对同一个区域连续框选并放大查看时，可以逐步撤销操作。\n", + "- 还原图形是指进行了多次框选后，点击此按钮可以将图还原回原始状态。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/scalar_select.png)\n", + "\n", + "上图展示的标量可视化的功能区，提供了根据选择不同标签，水平轴的不同维度和平滑度来查看标量信息的功能。\n", + "\n", + "- 标签选择：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的标量信息。\n", + "- 水平轴：可以选择“步骤”、“相对时间”、“绝对时间”中的任意一项，来作为标量曲线的水平轴。\n", + "- 平滑度：可以通过调整平滑度，对标量曲线进行平滑处理。\n", + "- 标量合成：可以选中两条标量曲线进行合成并展示在一个图中，以方便对两条曲线进行对比或者查看合成后的图。\n", + " 标量合成的功能区与标量可视化的功能区相似。其中与标量可视化功能区不一样的地方，在于标签选择时，标量合成功能最多只能同时选择两个标签，将其曲线合成并展示。\n", + "\n", + "### 直方图可视化\n", + "\n", + "\n", + "直方图用于将用户所指定的张量以直方图的形式展示。点击打开直方图展示面板，以直方图的形式记录了在迭代过程中所有层参数分布信息。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram_panel.png)\n", + "\n", + "如下图为`conv1`层参数分布信息，点击图中右上角，可以将图放大。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram.png)\n", + "\n", + "下图为直方图功能区。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram_func.png)\n", + "\n", + "上图展示直方图的功能区，包含以下内容：\n", + "\n", + "- 标签选择：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的直方图。\n", + "- 纵轴：可以选择步骤、相对时间、绝对时间中的任意一项，来作为直方图纵轴显示的数据。\n", + "- 视角：可以选择正视和俯视中的一种。正视是指从正面的角度查看直方图，此时不同步骤之间的数据会覆盖在一起。俯视是指偏移以45度角俯视直方图区域，这时可以呈现不同步骤之间数据的差异。\n", + "\n", + "### 图像可视化\n", + "\n", + "图像可视化用于展示用户所指定的图片。点击数据抽样展示面板，展示了每个一步进行处理的图像信息。\n", + "\n", + "下图为展示`summary_01`记录的图像信息。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/image_panel.png)\n", + "\n", + "通过滑动上图中的\"步骤\"滑条，查看不同步骤的图片。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/image_function.png)\n", + "\n", + "上图展示图像可视化的功能区，提供了选择查看不同标签，不同亮度和不同对比度来查看图片信息。\n", + "\n", + "- 标签：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的图片信息。\n", + "- 亮度调整：可以调整所展示的所有图片亮度。\n", + "- 对比度调整：可以调整所展示的所有图片对比度。\n", + "\n", + "### 张量可视化\n", + "\n", + "张量可视化用于将张量以表格以及直方图的形式进行展示。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/tensor_func.png)\n", + "\n", + "上图展示了张量可视化的功能区，包含以下内容：\n", + "\n", + "- 标签选择：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的表格数据或者直方图。\n", + "- 视图：可以选择表格或者直方图来展示tensor数据。在直方图视图下存在纵轴和视角的功能选择。\n", + "- 纵轴：可以选择步骤、相对时间、绝对时间中的任意一项，来作为直方图纵轴显示的数据。\n", + "- 视角：可以选择正视和俯视中的一种。正视是指从正面的角度查看直方图，此时不同步骤之间的数据会覆盖在一起。俯视是指 偏移以45度角俯视直方图区域，这时可以呈现不同步骤之间数据的差异。\n", + "\n", + "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/tensor.png)\n", + "\n", + "上图中将用户所记录的张量以表格的形式展示，包含以下功能：\n", + "\n", + "- 点击表格右边小方框按钮，可以将表格放大。\n", + "- 表格中白色方框显示当前展示的是哪个维度下的张量数据，其中冒号\":\"表示当前维度的所有值，可以在方框输入对应的索引或者:后按Enter键或者点击后边的打勾按钮来查询特定维度的张量数据。 假设某维度是32，则其索引范围是-32到31。注意：可以查询0维到2维的张量数据，不支持查询超过两维的张量数据，即不能设置超过两个冒号\":\"的查询条件。\n", + "- 拖拽表格下方的空心圆圈可以查询特定步骤的张量数据。\n", + "\n", + "### 计算图可视化\n", + "\n", + "点击计算图可视化用于展示计算图的图结构，数据流以及控制流的走向，支持展示summary日志文件与通过`context`的`save_graphs`参数导出的`pb`文件。\n", + "\n", + "![graph.png](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/caculate_graph.png)\n", + "\n", + "上展示了计算图的网络结构。如图中所展示的，在展示区中，选中其中一个算子（图中圈红算子），可以看到该算子有两个输入和一个输出（实线代表算子的数据流走向）。\n", + "\n", + "![graph_sidebar.png](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/graph_sidebar.png)\n", + "\n", + "上图展示了计算图可视化的功能区，包含以下内容：\n", + "\n", + "- 文件选择框：可以选择查看不同文件的计算图。\n", + "- 搜索框：可以对节点进行搜索，输入节点名称点击回车，即可展示该节点。\n", + "- 缩略图：展示整个网络图结构的缩略图，在查看超大图结构时，方便查看当前浏览的区域。\n", + "- 节点信息：展示选中的节点的基本信息，包括节点的名称、属性、输入节点、输出节点等信息。\n", + "- 图例：展示的是计算图中各个图标的含义。\n", + "\n", + "### 数据图可视化\n", + "\n", + "数据图可视化用于展示单次模型训练的数据处理和数据增强信息。\n", + "\n", + "![data_function.png](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/data_function.png)\n", + "\n", + "上图展示的数据图功能区包含以下内容：\n", + "\n", + "- 图例：展示数据溯源图中各个图标的含义。\n", + "- 数据处理流水线：展示训练所使用的数据处理流水线，可以选择图中的单个节点查看详细信息。\n", + "- 节点信息：展示选中的节点的基本信息，包括使用的数据处理和增强算子的名称、参数等。\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 单独记录损失值标量\n", + "\n", + "\n", + "为了降低性能开销和日志文件大小，可以单独记录关心的数据。单独记录标量、参数分布直方图、计算图或数据图信息，可以通过配置`specified`参数为相应的值来单独记录。单独记录图像或张量信息，可以在AlexNet网络的`construct`方法中使用`ImageSummary`算子或`TensorSummary`算子来单独记录。\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 关闭MindInsight服务\n", + "\n", + "在终端命令行中执行以下代码关闭MindInsight服务。\n", + "\n", + "```shell\n", + "mindinsight stop --port 8080\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 注意事项和规格\n", + "1. 为了控制列出summary文件目录的用时，MindInsight最多支持发现999个summary文件目录。\n", + "2. 不能同时使用多个 `SummaryRecord` 实例 （`SummaryCollector` 中使用了 `SummaryRecord`）。\n", + "\n", + " 如果在 `model.train` 或者 `model.eval` 的callback列表中使用两个及以上的 `SummaryCollector` 实例，则视为同时使用 `SummaryRecord`，导致记录数据失败。\n", + "\n", + " 自定义callback中如果使用 `SummaryRecord`，则其不能和 `SummaryCollector` 同时使用。\n", + "\n", + " 正确代码:\n", + " ```\n", + " ...\n", + " summary_collector = SummaryCollector('./summary_dir')\n", + " model.train(2, train_dataset, callbacks=[summary_collector])\n", + "\n", + " ...\n", + " model.eval(dataset， callbacks=[summary_collector])\n", + " ```\n", + "\n", + " 错误代码：\n", + " ```\n", + " ...\n", + " summary_collector1 = SummaryCollector('./summary_dir1')\n", + " summary_collector2 = SummaryCollector('./summary_dir2')\n", + " model.train(2, train_dataset, callbacks=[summary_collector1, summary_collector2])\n", + " ```\n", + "\n", + " 错误代码：\n", + " ```\n", + " ...\n", + " # Note: the 'ConfusionMatrixCallback' is user-defined, and it uses SummaryRecord to record data.\n", + " confusion_callback = ConfusionMatrixCallback('./summary_dir1')\n", + " summary_collector = SummaryCollector('./summary_dir2')\n", + " model.train(2, train_dataset, callbacks=[confusion_callback, summary_collector])\n", + " ```\n", + "3. 每个summary日志文件目录中，应该只放置一次训练的数据。一个summary日志目录中如果存放了多次训练的summary数据，MindInsight在可视化数据时会将这些训练的summary数据进行叠加展示，可能会与预期可视化效果不相符。\n", + "4. 当前 `SummaryCollector` 和 `SummaryRecord` 不支持GPU多卡运行的场景。\n", + "5. 目前MindSpore仅支持在Ascend 910 AI处理器上导出算子融合后的计算图。\n", + "6. 在训练中使用Summary算子收集数据时，`HistogramSummary` 算子会影响性能，所以请尽量少地使用。\n", + "7. 为了控制内存占用，MindInsight对标签（tag）数目和步骤（step）数目进行了限制：\n", + " - 每个训练看板的最大标签数量为300个标签。标量标签、图片标签、计算图标签、参数分布图（直方图）标签、张量标签的数量总和不得超过300个。特别地，每个训练看板最多有10个计算图标签、6个张量标签。当实际标签数量超过这一限制时，将依照MindInsight的处理顺序，保留最近处理的300个标签。\n", + " - 每个训练看板的每个标量标签最多有1000个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", + " - 每个训练看板的每个图片标签最多有10个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", + " - 每个训练看板的每个参数分布图（直方图）标签最多有50个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", + " - 每个训练看板的每个张量标签最多有20个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", + "8. 由于`TensorSummary`会记录完整Tensor数据，数据量通常会比较大，为了控制内存占用和出于性能上的考虑，MindInsight对Tensor的大小以及返回前端展示的数值个数进行以下限制：\n", + " - MindInsight最大支持加载含有1千万个数值的Tensor。\n", + " - Tensor加载后，在张量可视的表格视图下，最大支持查看10万个数值，如果所选择的维度查询得到的数值超过这一限制，则无法显示。\n", + "\n", + "9. 由于张量可视（`TensorSummary`）会记录原始张量数据，需要的存储空间较大。使用`TensorSummary`前和训练过程中请注意检查系统存储空间充足。\n", + "\n", + " 通过以下方法可以降低张量可视功能的存储空间占用：\n", + "\n", + " 1）避免使用`TensorSummary`记录较大的Tensor。\n", + "\n", + " 2）减少网络中`TensorSummary`算子的使用个数。\n", + "\n", + " 功能使用完毕后，请及时清理不再需要的训练日志，以释放磁盘空间。\n", + "\n", + " 备注：估算`TensorSummary`空间使用量的方法如下：\n", + "\n", + " 一个`TensorSummary数据的大小 ＝ Tensor中的数值个数 * 4 bytes`。假设使用`TensorSummary`记录的Tensor大小为`32 * 1 * 256 * 256`，则一个`TensorSummary`数据大约需要`32 * 1 * 256 * 256 * 4 bytes = 8,388,608 bytes = 8MiB`。`TensorSummary`默认会记录20个步骤的数据，则记录这20组数据需要的空间约为`20 * 8 MiB ＝ 160MiB`。需要注意的是，由于数据结构等因素的开销，实际使用的存储空间会略大于160MiB。\n", + "10. 当使用`TensorSummary`时，由于记录完整Tensor数据，训练日志文件较大，MindInsight需要更多时间解析训练日志文件，请耐心等待。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 总结\n", + "\n", + "本次体验流程为完整的MindSpore深度学习及MindInsight可视化展示的过程，包括了下载数据集及预处理过程，构建网络、损失函数和优化器过程，生成模型并进行训练、验证的过程，以及启动MindInsight服务进行训练过程可视化展示。读者可以基于本次体验流程构建自己的网络模型进行训练，并使用`SummaryCollector`以及Summary算子记录关心的数据，然后在MindInsight服务看板中进行可视化展示，根据MindInsight服务中展示的结果调整相应的参数以提高训练精度。\n", + "\n", + "以上便完成了标量、直方图、图像和张量可视化的体验，我们通过本次体验全面了解了MindSpore执行训练的过程和MindInsight在标量、直方图、图像、张量、计算图和数据图可视化的应用，理解了如何使用`SummaryColletor`记录训练过程中的标量、直方图、图像、张量、计算图和数据图数据。" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/tutorials/notebook/mindinsight/mindinsight_image_histogram_scalar_tensor.ipynb b/tutorials/notebook/mindinsight/mindinsight_image_histogram_scalar_tensor.ipynb deleted file mode 100644 index 862c259b04a435aff6077c9ba975bffa5e091705..0000000000000000000000000000000000000000 --- a/tutorials/notebook/mindinsight/mindinsight_image_histogram_scalar_tensor.ipynb +++ /dev/null @@ -1,1285 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# 标量、直方图、图像和张量可视化\n", - "\n", - "可以通过MindSpore提供的接口将训练过程中的标量、图像和张量记录到summary日志文件中，并通过MindInsight提供的可视化界面进行查看。\n", - "\n", - "接下来是本次流程的体验过程。\n", - "\n", - "## 整体流程\n", - "\n", - "1. 下载CIFAR-10二进制格式数据集。\n", - "2. 对数据进行预处理。\n", - "3. 定义AlexNet网络，在网络中使用summary算子记录数据。\n", - "4. 训练网络，使用 `SummaryCollector` 记录损失值标量、权重梯度等参数。同时启动MindInsight服务，实时查看损失值、参数直方图、输入图像和张量的变化。\n", - "5. 完成训练后，查看MindInsight看板中记录到的损失值标量、直方图、图像信息、张量信息。\n", - "6. 分别单独记录损失值标量、直方图、图像信息和张量信息并查看可视化结果，查看损失值标量对比信息。\n", - "7. 相关注意事项，关闭MindInsight服务。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 准备环节\n", - "\n", - "### 下载数据集\n", - "\n", - "本次流程使用CIFAR-10二进制格式数据集，下载地址为：。\n", - "\n", - "CIFAR-10二进制格式数据集包含10个类别的60000个32x32彩色图像。每个类别6000个图像，包含50000张训练图像和10000张测试图像。数据集分为5个训练批次和1个测试批次，每个批次具有10000张图像。测试批次包含每个类别中1000个随机选择的图像，训练批次按随机顺序包含剩余图像（某个训练批次包含的一类图像可能比另一类更多）。其中，每个训练批次精确地包含对应每个类别的5000张图像。\n", - "\n", - "执行下面一段代码下载CIFAR-10二进制格式数据集到当前工作目录，如果已经下载过数据集，则不重复下载。" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "*Checking DataSets Path.*\n", - "Downloading CIFAR-10 DataSets.\n", - "data_batch_1.bin is ok\n", - "data_batch_2.bin is ok\n", - "data_batch_3.bin is ok\n", - "data_batch_4.bin is ok\n", - "data_batch_5.bin is ok\n", - "Downloaded CIFAR-10 DataSets Already.\n" - ] - } - ], - "source": [ - "import os, shutil\n", - "import urllib.request\n", - "from urllib.parse import urlparse\n", - "\n", - "\n", - "def callbackfunc(blocknum, blocksize, totalsize):\n", - " percent = 100.0 * blocknum * blocksize / totalsize\n", - " if percent > 100:\n", - " percent = 100\n", - " print(\"downloaded {:.1f}\".format(percent), end=\"\\r\")\n", - "\n", - "def _download_dataset():\n", - " ds_url = \"https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz\"\n", - " file_base_name = urlparse(ds_url).path.split(\"/\")[-1]\n", - " file_name = os.path.join(\"./DataSets\", file_base_name)\n", - " if not os.path.exists(file_name):\n", - " urllib.request.urlretrieve(ds_url, file_name, callbackfunc)\n", - " print(\"{:*^25}\".format(\"DataSets Downloaded\"))\n", - " shutil.unpack_archive(file_name, extract_dir=\"./DataSets/cifar-10-binary\")\n", - "\n", - "def _copy_dataset(ds_part, dest_path):\n", - " data_source_path = \"./DataSets/cifar-10-binary/cifar-10-batches-bin\"\n", - " ds_part_source_path = os.path.join(data_source_path, ds_part)\n", - " if not os.path.exists(ds_part_source_path):\n", - " _download_dataset()\n", - " shutil.copy(ds_part_source_path, dest_path)\n", - "\n", - "def download_cifar10_dataset():\n", - " ds_base_path = \"./DataSets/cifar-10-batches-bin\"\n", - " train_path = os.path.join(ds_base_path, \"train\")\n", - " test_path = os.path.join(ds_base_path, \"test\")\n", - " print(\"{:*^25}\".format(\"Checking DataSets Path.\"))\n", - " if not os.path.exists(train_path) and not os.path.exists(train_path):\n", - " os.makedirs(train_path)\n", - " os.makedirs(test_path)\n", - " print(\"{:*^25}\".format(\"Downloading CIFAR-10 DataSets.\"))\n", - " for i in range(1, 6):\n", - " train_part = \"data_batch_{}.bin\".format(i)\n", - " if not os.path.exists(os.path.join(train_path, train_part)):\n", - " _copy_dataset(train_part, train_path)\n", - " pops = train_part + \" is ok\"\n", - " print(\"{:*^20}\".format(pops))\n", - " test_part = \"test_batch.bin\"\n", - " if not os.path.exists(os.path.join(test_path, test_part)):\n", - " _copy_dataset(test_part, test_path)\n", - " print(\"{:*^20}\".format(test_part+\" is ok\"))\n", - " print(\"{:*^25}\".format(\"Downloaded CIFAR-10 DataSets Already.\"))\n", - "\n", - "download_cifar10_dataset()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "下载数据集后，CIFAR-10数据集目录（`DataSets`）结构如下所示。\n", - "\n", - "```shell\n", - " $ tree DataSets\n", - " DataSets\n", - " └── cifar-10-batches-bin\n", - " ├── test\n", - " │   └── test_batch.bin\n", - " └── train\n", - " ├── data_batch_1.bin\n", - " ├── data_batch_2.bin\n", - " ├── data_batch_3.bin\n", - " ├── data_batch_4.bin\n", - " └── data_batch_5.bin\n", - "\n", - "```\n", - "\n", - "其中：\n", - "- `test_batch.bin`文件为测试数据集文件。\n", - "- `data_batch_1.bin`文件为第1批次训练数据集文件。\n", - "- `data_batch_2.bin`文件为第2批次训练数据集文件。\n", - "- `data_batch_3.bin`文件为第3批次训练数据集文件。\n", - "- `data_batch_4.bin`文件为第4批次训练数据集文件。\n", - "- `data_batch_5.bin`文件为第5批次训练数据集文件。\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 数据处理\n", - "\n", - "好的数据集可以有效提高训练精度和效率，在加载数据集前，会进行一些处理，增加数据的可用性和随机性。下面一段代码定义函数`create_dataset_cifar10`来进行数据处理操作，并创建训练数据集(`ds_train`)和测试数据集(`ds_eval`)。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "import mindspore.dataset as ds\n", - "import mindspore.dataset.transforms.c_transforms as C\n", - "import mindspore.dataset.vision.c_transforms as CV\n", - "from mindspore.common import dtype as mstype\n", - "\n", - "\n", - "def create_dataset_cifar10(data_path, batch_size=32, repeat_size=1, status=\"train\"):\n", - " \"\"\"\n", - " create dataset for train or test\n", - " \"\"\"\n", - " cifar_ds = ds.Cifar10Dataset(data_path)\n", - " rescale = 1.0 / 255.0\n", - " shift = 0.0\n", - "\n", - " resize_op = CV.Resize(size=(227, 227))\n", - " rescale_op = CV.Rescale(rescale, shift)\n", - " normalize_op = CV.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))\n", - " if status == \"train\":\n", - " random_crop_op = CV.RandomCrop([32, 32], [4, 4, 4, 4])\n", - " random_horizontal_op = CV.RandomHorizontalFlip()\n", - " channel_swap_op = CV.HWC2CHW()\n", - " typecast_op = C.TypeCast(mstype.int32)\n", - " cifar_ds = cifar_ds.map(operations=typecast_op, input_columns=\"label\")\n", - " if status == \"train\":\n", - " cifar_ds = cifar_ds.map(operations=random_crop_op, input_columns=\"image\")\n", - " cifar_ds = cifar_ds.map(operations=random_horizontal_op, input_columns=\"image\")\n", - " cifar_ds = cifar_ds.map(operations=resize_op, input_columns=\"image\")\n", - " cifar_ds = cifar_ds.map(operations=rescale_op, input_columns=\"image\")\n", - " cifar_ds = cifar_ds.map(operations=normalize_op, input_columns=\"image\")\n", - " cifar_ds = cifar_ds.map(operations=channel_swap_op, input_columns=\"image\")\n", - "\n", - " cifar_ds = cifar_ds.shuffle(buffer_size=1000)\n", - " cifar_ds = cifar_ds.batch(batch_size, drop_remainder=True)\n", - " cifar_ds = cifar_ds.repeat(repeat_size)\n", - " return cifar_ds\n", - "\n", - "ds_train = create_dataset_cifar10(data_path=\"./DataSets/cifar-10-batches-bin/train\")\n", - "ds_eval = create_dataset_cifar10(\"./DataSets/cifar-10-batches-bin/test\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 抽取数据集图像\n", - "\n", - "执行以下一段代码，抽取上步创建好的训练数据集`ds_train`中第一个`batch`的32张图像以及对应的类别名称进行展示。" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The 32 images with label of the first batch in ds_train are showed below:\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr4AAAHdCAYAAAD7D3ocAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOy92ZIjSZam9+GYmhkMgMO3WDIjIyu7Ors4XV1dXd2sJqd7hsslecMX4B1fgs/BJ6AIr/kKFIpQhkNSRmSkpWW6qjKrco/Nd8dmMFNTtQNeqBkAR3hEZkZEZkZV2h/iEu6AwTaoqf76n/8c7a1WKzp06NChQ4cOHTp0+FOH/NAn0KFDhw4dOnTo0KHD94GO+Hbo0KFDhw4dOnT4UaAjvh06dOjQoUOHDh1+FOiIb4cOHTp06NChQ4cfBTri26FDhw4dOnTo0OFHgY74dujQoUOHDh06dPhRoCO+HTp06NChQ4cOHX4U6Ihvhw4dOnTo0KFDhx8FOuLboUOHDh06dOjQ4UeBjvh26NChQ4cOHTp0+FGgI74dOnTo0KFDhw4dfhToiG+HDh06dOjQoUOHHwU64tuhQ4cOHTp06NDhRwHzJnbS6/VWb2I/bxsM8J8B94EDwiwhBvr06TNA2ANiFAEiasCxpCRnQMSQPoLwv6w+633bY/+p3tM3hdVq9a3vKXT39evwKve1u6cvx6u21aeX/nu7r7rS21698f9mi/a02ssSQJr3g5Yi21e8UhRhBXj12MpRuQpbFuiqZjjMGA4zTC/suoc0+w6fUVUQgZ7wb/7+Hzj74p/Cbru2+sbxKvf0//nkfAWgKKiiGr4zVUBBw7fIalWh6qm8ZXJ9yeXlFelKOe4lZCaibww9I9gYrIGsn5INh0i1Au+RVRh/Tc+Q9VOMSZHIIJLgRfDSw+mKvPJQQ4qQKkiVI3YB3qHqwfQh3YN+hkrMKjKoAYxSAk4UzypcB6HphRbeQ5prEhWuryecXZzz+PyEL59+wceffcz//b/9r2/knkLXVr8Or3pf4Q0R3z9VeGAODAgPQAk4QCiJKIm4ashwIMJ9+tRUgCMiok/9w518hw4d/qgh8v0F5ORG8E+f/11X6JroBlK6/WlFNvvYOW3V8JIiiBgioxg11GLwCtKLkF4ECL3emjqv6baKoD2o8dT4N3K9Hd4gZP3tBqLYvKwoqivq2uO1wnsHKM5XzOc515NLTOVY+VY+UlTAJrCMhdEoY7i3RznPsbMCcT4ITybhaG/MXrZHmg3o94eIiSFOUBEiAK9IXUHp0OtLdHqFuBK0RvbGyP5d2D9EkwyNM6J+AhqhAmpWNPwWGtKL9GgvThB6KmT9jKPDu1ReuZ5fk/YH3/ut7/Bq6Ijv1+AaSAALTIAF3KCzETCg5IiSI/YbIhwxIAJimsewQ4cOHb4Vet8j8X0xGuIrEK35sLChN0GVla3XdOu012rtemvBYEDBG4+oYCRBMPREEL2pPCuC7ylLZ8kry9wWr3c5v/ifIYIoUY5SZZQ6hrFH3ALsHO+XFIVntlRyn1DWA4j2QY5BhhCPITMMRjGpcUABmjNzQu3bmKDQSoM9qRmmkMVKVCsGh68szlV457AIGqVINkbSPXwNTiMEyARGqbA3EGIK7OyUYnGGY4gnQ9L7xNl96ngPJUZFQAsETyaWLHKoLVBXUKuCSSCKUYmBMOlwNfgopl5PNL49eoEZgkIksBJCI6g89cpRqcdVDl9XKB5XWa7nE07OT0jyAldW9LyjdiVePHlsKPrCcG/I3uEhl89OuDo5RXNLrMpIEn567x3uHB5xuH/I/t4h0WCIxBmSJJjE0FOPLubU+Rw5eYo+ewq2RKiR4zuYdx7C8TvocB8dHkJ/CGlG3TcYhJWRph0LYnrNtE7pNZM4EWGYDunHQ1Th7PKEQRy/Xtvs8L2hI75fgwWQNr8fEGwPSiC/lqAAx8A+++xzF6iJgD367NEn7mzUHTp0eMuw0q+jOWvNCwR6CrAC7a1fXzUETxHWce2W/K5/NBxLNq/TAzGCMQbvaWLJwkrB1pvjKlDLikqVk6sznpydU52dvd6FN9cdA1EkiApSB9KmCuqioF/HgoTAOkHeqAEPagNpr5XYtGqggSgC3w6nrfItpLGQ4ohrEG8RtZjaIzik1URSwWQZko0orFJ7wNVUPowxSRQRSSDVJjaoxvg6EFWrdbAXSIjLCwpSQxTOV8QRRikAT40iqqg4aqeoqyGKiSQmes2hyjREcf3tidL+QwRVcKsVpVbYconN56yWOVVdIVWN9xVWlIKEXCK0TNCy4OzqksdffUlxPUWKigzQdx5QHh9j796junMf+hl1nNEzhr4xpL5iWMzJFnPM2Rnm9BRxNpxZPqcqc5heo9k+OjqE/WMYjvGHe3C4R0/6GDGoiZBeoL2RKEZAfEODBbyGFiKq9FwXjfhjQUd8vwGmBLvDfWJ+wkMgwlIzZ8mEJRHCHnfZ44hWDx7QZ19i4ujNKb7/47/9O/pZH+9946dqlBYJM1ARWf8NoF7DtnV4IBWQSBDT/NEMVOsJuzbhKWpqlAhtoliKFh5XVai1+LJien3N9DrsIQb6CQz3oL+3hxmMMcN9VDK8GIwIcWQQCZ3jtpClqje8YbI1ZK5DZq5GVZtrjHgdEf1Xzf+m+bmpXe0GcDd/7wZ327t3G1ZsBv7tfd2Game7hHA/t/fjbznW7nm2CHqTrD9hgeXWcdr91Vu/f/WCc+vwp4mW9OpLye+G9IqGtrgmv6uG/PZYk17dtv023kgVqLee782DL4HrRoL0DD3phUD3yuNUUV83pAm8gFXP6cUFH3/6CdSz17v49rJUwvMvEeAb0sva1BlJhEQxPQIhDxflAvE14NRgNCI2YKIY1UbtbcOBEhHFkJmaFCVVD86BLYiiIJ2IROGeJDFZlhKPMkDxWuNQnCrWQ2ohMjFCTGpSvA85JQrUClo3UoxALIrBI3giCRekDQlFlYiYOlJEBVxQXxET9v+qzLe30fsNTV8TnCtbloEeagTvFKuOpcspFlO0yPHqQWvq2lP2hKUouSR4V+KLmJPLK758/BXTkzP8FPpA//IM7t+lnj9Al1N8f0ARpYhJGAoMfcVhMYd8SnJ+BZeXiPNEAjq/RhdTVvuX6GgfzQ7h/gPYv4vKPRgaJE2AXtM+w7iaqIb+tRkw69CQQjtaKfjO2vjHgo74fgN44BSIcMApFcqSiinKvNnmiClH7BE3/4S77Ok+b9L1Mz4ak/UzvPchdNV04lEbkiEQ4JBYoNTGhU5SzVpVERHiOAIND2ldr5106wGrnakTBSKq1lIWJb7IWV5fM79Upj74n1uNZ1hBeQmjYs7wDozilHSQkWVDjBHEGEzTz8ct0w5HCeehulaNmj5z7a/yrqZWD8j6Wl8VSfP/LvFt02m2Ibf8tLiNiLZ0c/dn94w3HrjA4Vdb28Vb59h+Pt45XjvAbP5OMOyRMKLPHhEJjjkVExacrz/bEt0WEV0H8GPDrtJ7e1LbRnPVWlk1fcVmUtU8l1Lj65ob08KVsuqBXym+XrFcFpSuDG+xIpKINE4wEmFrh/rQ1zhWzOdzZrMJ+bJoyBpgDLXAR59/xm9+/9Hr34DagUTBUlGD1hYfWdRaam9R59CoeZqFQGSlVXtbK0O4j4qhVsW0E/Ed3hNpDepRu8D5HNwCsTlRHBTvVCI0ShAcceSIUSLqcG6qrGzwvDogqQPxFhxoDNTN5HXTb4YOfIlSoGqpxYMWgazXjQ9BLfhAmn21xC8KVCJM3PaGr3BL65C+Jqprlb9Gwa8QJVhbDKhfYcnxviIR2OsnRHWBKWuq2lN6T4FQmJilGLQSKJWi9BRFhS23DmoLWM7Q5R5qZzhqrPFQh/uYVSXkUySfExU5xlmiqqLVfGoUb0v8ZIamV2hlYTlHUkUOM0za9sIhMhH1wlhotgcmCc1C4rDHlVavdP86fP94a8e9JsH3rUEFPAbmLLEEC8Q2ZsCXzNkD9oF9Iu6SbvniXh+SpEgalFN807mshZlNp6WqWGtvKDrSbCNRhNAQYTR0nk7RGwxNGlU1dGSVeubzOdPzS84v4ZxAerfddml73Us4fDxHa2F0r88w29+6gNBh6FqUDteiQLQmvpuQWZtNi3ik6axfN+Fnm/i1RPU2YgvNmMeGIG5/lbuEdpv4tu/rzt+3KcarnW2239v+2X4WNsQ3QYlJOKDPXdLDe4zu3MX0E2ZfPcNPHwOXKIpvrr1Ve7ug3I8PL7I33Kb8hjanVL7eTErFEAmIhEQ3p0rVRIVEzPpzugpqZaXKdT5ltlgE3VE9/ThlNByS9VNcVeO9kqinVzvOry548uQRV9MJHgUjmDRGxfCHTz/hN5/89g3cBSVSDf2yB8XhvEWtR60D9UASbAihYwobNmnNIYbSUE6tm4c6ah7elvkGYUFQxCu1zaGYIX4ObkGkQiwxaFMRKKqI8ZjWaVsrOAVX4ySo3r6Ro01D26gbFZe2Fwfqmlpz0BwVB7FHnUXrIggdupYUwueWM2yeh48a4VW9Dq3Q0vOKaCDkXhT14WiJGCSSUFFBwdc1cSSMsxipDKb0WK2wXilEsB6ciRq5H0qrlGXNaotX9jxIUYBdoOUcL1AhYaKhNdglveUcWcwx5YK4qkh8hdEVlXrUO/x8jo/61CZGfYnaOcl+iiwOkKyPQehhSABjBHoNLxFFxQQRPQJJgF6Fqn2l+/enihi424ejozthaPeQJAnj4R6jfp/AqjzGBDFsPplydTlhMqlYEDhG/h2d21tLfH8OPCMkl70t8MDlS95fEQjwDEh4SkTOHv03dvzIyNrSEL65hjWuY4svDr8HtRcims8TlOEICZ1kcDE1TK9ueHGwSWjhmV9fc3kZvpOrW3ZfsVGAUdDzKWQD4v6QJOsjZpsigsrGYtH6pVqRB1rbRvt+CKnezBl+NewS0/b3lz0IvZ1tdyEveP3r3ts9D+H589o+h5Zcjxgy4B08Bg8Y9snee4/+e+8wuHOMJAmVV+x8AZqiFGvSu0vIO3yXGHPv/v0f+iSAW5Terb+dr7FFQeU9aRJjojAxVhFm84LJbA6qHO/tMUxTkjQhSqCoLLkrgp3KN3Npabyn3mNXjmcnJ1xMJtTqwHviNGY8HJL1M2rf2JvEIBiePH3Kl4++4mJyhUNBhDhNwQgfffYJj7969Nr3IcIG+2tNsDaIBTyirrkn9ZY9wzVcdjeeYwLx3OqPgjh888mV2iNYIm8RzQNJszNCKM0gpKSpIGSkOEQ8Rh3iNZhHPZuZN54IR4QF4obE1ptzUgUcvpihOsEZj3fNNagLk5f1mQbU1ZzVcgLeUb9Gt6qtQq01vg42PFcrPQwRBiNRUE2lh1OL+pL++JC9wxynii+WlCqIKILBJEPiZIgIwedceqrCr2f/JeCWUMUWnxf4YolKjBKH8ma1RyoL1kJR4IsCV8xZVRXe19QS4ZMCbxJWUYRKgs8SalHk8oDe1WEQc80UYzLUGLxEaAxqBIxBe4YVIYTZy6+J64LhH4Gc0GMT7XxRmmgb+XwV4TEj5EGNxzA+hIOj9xiO3wmRHa+MB2Pu3bnDwfiA5eIKXy1JEugnwmwyY3JxxcXZJfN8wWJSMisDnyoJU86S5+2Br4K3lvjeJxCpt4n4fht8CkyYMmD65nba2BXambls931tU/CtahoGrs1HBaKoUVFlSz1VYoQ62pBikcBA1VdorSwrz+JSOed20pux8aV6wneWV2AuT0myDGQfkwhIGghuU3Jmo/xuyG1rx5B1QglhkDAbVfrl3sSXY5s636bg3oZdJfe213cJ7ovGke1jfx0hbrFtyTAYDvk5Bx9+SFl6lqWF8ZD++w/oP7iL2d+nhyAnl0g2gNzcIL2rnWN0BPi7QsYvf/XXHBwcfu9HXrXf6i1fbvvs6EqRkI6PLQrOzs6Y5zlHhwfBnhQnSJpydjHhk88fsXLKzz54j3tHB4xFyPoxeVFwMrtgmecUeY6IEEUxCBTeUdQVXzx+zMnFBXVdot4RGyFLU/pxCgisBK9BaD09O+HRk2dcTWfUAio9ImMggqfPnsH5xWvfm1ht6AOVJjNJiaUO/ZIJ51GrpXYeVROIY2t1wIA4QnwrqK+RmpBLhgYuvH7wawRPhMWIQ8QFW4XNqZt3DRkmE1LJiMUiOFIskXoip9SuBokwRjDGYdQiVIjaIFNoheKaCB6gFba4Qu0FsQGb1k2/EYipNr1fW3gusjMoJmAX4O3tDeYbQHUrH0OVSj1ea4wIqemRxglJkjI0Kf2+cJClWF1hFeYI5/MCaxwZocJE1N8nToa4ylFWSyqnUG5IpQfmKyhyKBYVLnd4sQh9kgQS9RjvofJUlUOXOW5aICVEDohrVoMimKdNihiFIkfzhOrqDJ4MWc2n9AjVRsJtETSNqfsGiVMwfaIk1BFmPmVvMeeevP3EVwj5SvuE3KUZN8eEiGC1a8eLb+tabu+AevAOnj57gn9yjlaKVp6jgyO0/AB/dIfHX33G9PqSLEsY9A1VWVLMF0HxLQPBNcARMALGAgsN1bVe906/tcT3zemkPxwueblC/G0hhOSuoKw0qoI0g1ndhiNDSSAjBt2hYuFzBiTZlA1qyGdwNMUhAY0aEaXyFVXpsdM55xWc7JxPDzgEjgkNMSckUk2a9/VaUf2M4/vHCO/S5xCTppAka3VofRJt1ynRmti30UNtGb7QKBuvLk+0d6R9cGTr7227wzchqO2Z7NoSbrMs3LbPFZsOpj32amc/7esJQwbcIeWYe3//X3D8t7/g6mqCXlxRAsujEWXfcDDqk2DQfgqS0AZq651jd4T3u8SYP//wZ9w9vkuWfb+1PVdrtfL571hXLQEC0CZpa8XV5Jrff/IJz05O+OD99zm4c4c0y4izIR9/8jn/4T/+J3yl9JqQtjfKfpzx7OKMT558xsXVBVeXV4j0SNM+YnoN8XV88egJJxdnuMri6wJBiXsNmZNAfn2lOLfi4uqa64tLyHOQ3s2M0tkC5q8vIojOmwcg3CARkBhEPRIFiwaqqLdQR6w8IEXYKDJACpoSaYbUGUbSUPuhhgq7fmgTlFQdhoqUAiiwOqe2k01/IBmZj0hdSuyGYFJiazHWI74GC3EaEashVkvs5hidEQMWRcwQqW1zSEVdTj15BJPHTBPBDSBLB2RJEtwXW1UHBKiX55BfQDEj6H+v1ivUWoehRz2iiqscuSsZphGZyTASExtDEocqF8PUUOZz7HxObzBn2h8SJzWxicmiBJMMMSal6OWUpeKqOiQGbuEamDvYywvsYgm9GCSmp0kQCpwDa6mLJXaa45tkHAMYB/EUTKpIUkBcoJmgUsETsJpj4xS1NVoq0sxoNMvwWYr0B4jJMGlGZAx1VTG8PuHBdMLbjja6OSaM+fsEghniHhuRpWr+rvh25Lff7FeC64SLM7ikWo85s+UViVaUsyt+97svOdMmWXHr/CqCstsH3jHw7j3Dw+P7PDh6h9nZBRdPnlCVr0d931ri+5Q/LrW3BwzZyPADYI/XDczvIhSAN9L2223nbTZ+WJoTqF+g+jQ1HDdVIVrlNxBrFIx4YhyVrbFFznI65fyWs3mfQHz3DSybdjjfev8EqKZg7SUigSAbOSRNkqDqRnF4IOpWgd69X1s6qkRIKMZzoyrEt8XLSOn2zy1ngHDTe77rz73tZ3ub3d9bJbZF2/H4rW3CcfskPGTv8KccfPgB+3/3CwZ/8QGTxyml91zOplyfnVJdPOOnvuIwG1Euc7QKitD2dbYdS3Ay8kaTL/800MaW3ddt+BIUnJxcgUakafr1m78ArXLb02/e4FUVXTVJRuvJrQRCrGFBAWiSX2tP5Wuenp3ym49/z2effcmssNyf56RZRn845Ld/+IR/+pffgq54/+E9sr2UVVpjGfDlySM++vz3PDs74/zshJ4I/axPZMIKWpUP1RiuJnNcVVDVlpWvaWo4YCRFCLac2in1PIf5HJxtHsK2D1vBPP/28tNt98fNwjPdVI0gNQhxUHLFEYkjViVGKXwBVkOfGRtCUlkBkkEavMIxzQphKMvG7xFp474XR4wDzcEXrBPNGnIdeU9sY8QIdRShXqFQcA5xoYqGqCAaI84ifkLsZkjURMx8AT5H8FArWs3g8jFcfAKJYTmI0IMj4oMjElIg2E20dnhvqYorKK4ItOfVsXQVpplIRd6xdBVFaYmiPhmKp0Z8xQrBY/GVxeY5+XRKsVyGyb+JIe6DicMiElqjK4+wCt7gHcxpwt7e4WsHtUNL29hUVuhySV2W+NJi87AthP41pun3bON0iWGVXAEONQ5dLUKN42lONS9IEHo9E2r9ZmGCgukjEiMiePUkVc7+9LZ46NuFFZv+f0y4DxVBsFpyc0xqx75v+ti1Qth4DJKEstFwczysgMlkAVoy0aA659wUhVorRh+4dw/+7KcfcG98xN3xAUOUvjqqZcnr4K0lvs94PoHsbcQhwZZxhwH7HGApWVJSU/KmdbU2SCXGhGoNPpAa0zyAa//vVqLYet1FWhXVBEsB7TaCafy06prm2cQefbmkmi6YX1bPfRcZ8A5w2IfBPsgcrpehYW/jCqCE9NkliRiSOEWyIaaxVSiCb0JvWits51+oQhSMGUH9jV6L9Ibr3zxcrcr7IuyS3tvQktN2v9uWgq9Ti7c9t2br9e1tgui1R5LdZ/CvPuTw7/6K/l/8Gb1371NXQY2/Xkx5Nr1isczZP9onM6apghH20h6rvZ6q+dkundYB/vEf/gfSNCONYy4vz/nD73/LNH/8CntyLPNzPvvckpjXbLAEAtz7BlPoFYEjelVqVbyvm6hKcJ+uvMfXfh3lKcqCwhY8Ozvh408+4ze//YhaDNPC0h9mpMOM3332e37/0e8A4a/+8s8ZHqQ4Yzmoh3z25BEff/oJz86f8uzkFAT6WYqJDV6V1UqZTHPq3AYy68sQ/6wDAaskA8wmd6wqgy9T63VEC88bzXJWO4NIm+XlgTolIgMsKg6JPLGGhSbEWbC+KUMTN5EoAT8E6xGjmFhIiUhx9NTSbBkqYKgiYgPhrUOlBa0LcD7YJ4xFrBCZJuHUOnQBUmjg4bUgThATE2ERO0HcHIniJs+jIDJF8+1asBO4egJM1g95iaPOsiBWq8U7i7MF3ha4N0B6AayvqHUVRBPvsZVl6QriOsNRY9SjlYdKWeHwbkE+nwfim4f8g5UIKxNRG0G9UjdtxaDoLdUS8ub7806bfZf4KMK0EzwbKnV4W1K6TZKUIYxdrbKphDmPTgHmqM/R5TXOlizPa9x1yLEzBiQF+lmIpEkEatZROWOUveo1F1f5ntD2/0fAXW7mJi1gzVy+SWy1x0bl3QOODmAwCoFbMQNMskSqmyR6UkJZeqbc7jOOCWsm9IHD4yPe+7OHjJM+46RPUo0wVYUfvV4FjbeW+L79c6eAI1rSO2BIjCWnokRfSHleHSYOSQIhruTxEiOE+rbGhEdZWgPbmvBqU/ycNdF0CGvPl1fQCu8s1XxJVRa4YoGv5kzPz7k8m/LolqhCS6RqA8sSzpeh6sVts8Mr4J+nMM9P+UkFinB4fJdBmqKBAocHrbFwbK8hpG1otn4zNRJvMVc893A/J5JvvfaijuBFpHeXyO5aI9pb26q9urO/cDyDjAbI0R565w5X3uPPz7laLilNggz2yPbvEGcD3n3vAR+8c5+iVMpaKP/pU675lLzZX0no3Aq2LS4d4D4Pf/qXZGlKlgoHB2NiUT76yHGdn77C/hbAguo1InKV3lgI+Ebbu72NhlB96Wucqygqt25zXhVnw4phWdZnMBowtUsury746uwZXz59yuNHz4j2DrgqKuLMEGeGj//wCTx7BAi/+eQjvCl4NjlgfJTx8ecf8eWzR1xeX7HIpyCwqAw9A6tKofaQr4JqqlXwkXoHtQ39TpNjEIivhjqobdLYd1QS1dmwzLxGIbHX+VFzLIdQBUIqEIsnjiwRFbUjLHEmUWN3sNRUOHEhEdgQFqewOe1EUw2hQzMaSC+B9Fq1xM4HEiAeLZRaLVVhsXGOLcAvpcmCDzQ6rMjmwE3BTpo+vEYZQDwK79EQWX1284InV/ijI+oUfFFg7QJb5FTLnJXdIr3SrsD27eF93djtgqe2sEuKckkSZyyzJaoercIY431BXeaU5yfYy3OKIqeuHJV3LCvLUmkmbSCrisw7+u7FhFLKcO+pkjDJqFdh7CtLtCypS4dlQ3xTQj/b1ugwNP11EULzWih+moOH/hz6dWgSOCgLsFGBUqzzU9q69F7COPjHgDYRfcCGtBqCj7a1GbT4uu7rALibwDvvhclBCHoYBqMjstEhmGsO5gvKxZJyFva/ICi9SuuWv2k1PCAIioFEZ4gxTBYzLhdPkUVFr6pCAuhr4K0lvm+/TTzcvEOE+xw1a7SFpSYd+lqB0hceT4Q0CuQ1tBKPIhgxSGyadR2a5hPR+H43doYwzLSSqoLW1FT4wqLlknx6znJ2TT69Zjmfcn0dSpd9dcu5OEAGQAr5PNhSXqYdWOAjD/LVKYPhHnv7h002d5uE1y5doQ1x37SAIGCHhv665cx2bQy3/b8+7s5rtz1qrf2hfX+X9O5+Zvu17bCSueW9VpUFQcYZHIzh3iHXsznXZzPKpaeUGBmNGXiPYch7D9/l/fffJffCTBJmv32fxH663mdJCGlBGAxGt1zTjxF37j7k4Qc/Y5ilZCkcHx+QDQQT1/y7/+9ViO/ro63Tuk1+14/u7sZNoKdCsd6TV45FUayfG+8cRVlii4KD40PSgxEzm3Nyfcajk2d8cfoMJk/58tMR53NLlCgmVq4/+Qx4AsAfPv2IghnHkxH7B32+fPKIr55+ySrPg6ILYBqLRuXB1lDFYXasVVByfVNT1tPEmWlU3Xb4+24XAVjZSbivUVMnxkGtYVUuwgLCSCSk4kjxxCypnQ/nLQJxDK1CTA1xjKQpYmdEdkJdhzKTdSys1/zUYHNwaqm8x0rYQ/hmLN5arFgKmeF8hNrQx2tb7gwJVRlcjvgZ6kz4XuMR2H0wFtUF9eJ2uUhtjroYa2fYxYyimFNv1YnqAcPslauZ4VzVdNkOypKishS2IOnn5FWG85xVLvUAACAASURBVJYqz7HzKVVVUBdz/MUJen1BVVdoD6paKbwj9wrew0pJekJqIHuBkiq0wnkZVlhDEBNKpql1eOsQG/q8OZu+vLU7RFv7UcL8wU83ZSxbZbhtmTmQ102fXYf3UjaE7Y+lmFmr8KaE8aW1PJjm9a8b87axDzx8D375tx9SVsrpyZRlCcODuxzdecDg4A73q4r5xVNm56ecnHiufbiXWXPcdqwzbHzHR33YPxRGoxHGJFzOTjh7/IiBFw7MgOS5qvvfDm8t8f1jgAceo0Q8JqyrEzHFsSQ8VCmvtcjYLWjI4RYJFCA2bYmyZoUIANWwMpCCUDe5YRHeW4qyRJ3F+4piPg2LUlxfcvr4nGnJuobelJc7HT9aglk+X9P3RVgBv6mg/OdPCMvHCf3+kP5w22naehEbGtwQ99f2OHwD3KbObr8uO9vIznbf5hgvIsTb9ouE0DkcZB8wfO8Bvp9wPpszXSxZLJcYSRgdjhnsDdg73MOIMr57l2Q8Jj+Y4/eH6J195MnG4mB3jvlHIlJ850jTmCRLyUYpoywmjpTavYO3Of/yn465zt9kmuo3wyePAuGMRDASbbWz8FuvF34PxV5Ca8xtUHpzWzCb57g61DV1VUVRFJSF5ajIWfqSZ0+f8OTJE56dnVBeTYAFnF6xtECqkKxgskWmvvyKx1IwLfbYP0q5uLpglU/AVkGxZRUamCoUFZR1UH19ayxqFd0GtfLNeo43iCbLaaU0SmpKJFk4v3UdVkEixYgLBQO1aB7QdkJuoXZ4AbUxOANuAsVFqO5ADRojcUocGZCKWsKqbzVBAJ9VUFcQa1DKXJyjEsrC1Q5QaWojR2izKFFlZ2ihWE6xWkBqIDV4G2HtDCbPbrtiimJGkhq8LYLiutOpv7Z9rE16rCp8njMvplzPp4SV4qDfE3yxxM6nlMUcN5sQXZ4j8ylGFekbRpUntxa8p48QS8hEiZzyQD1P2UzYAf4MeADcAcbeo5VSGyVaRWSqpFVT2cHfrFveLhBk2ZDahE3/2woXLZltyzm3IkVFY7GgXdBlPb35VuPADw0hXMuMzbW30b/2frTXs+DFPKAivNlPxlRVyaK8ZjIr0WSJN4uweJUkqCRoL1Rwab/Hgo3LCTYJdveBO3fgzp07KIbryYyzi0uenM7IFBZmQSKvR1074vuaOGl+MrRZeSc0mCNgSJ/4TbooFXDBv7cuIWMMEhmMxNzMDhOkJY1NkVyta4r5lOvLU8r5nGo55fLslJNTzynPV234OryqHeVTQP/pI1Th/rsPeDdLgh1DQ0h2vb779gIcb2jp510Su2tj0J2/t1UCuEmGX5X0bh9r+/Vq65gJIVnyiL/m/q//juwXP+cUePb4SbhHCnfujbh775gkG4SloY1hdG+MJlDuZczHfey9I+TJe8CTdS3E7fPZHkx+zDASlntNUyEbpaTpEJH7xBH8m3/9X/G73/wLn51++jV7+TZaydfj3/3TP9MTIelFxCZBJKgcbUnCXiSNzSkKz48IReWwriIvCyazGbaqsNaGnyJMeg8P93jndJ+ri6dcnD3h0aOv4OKiOeczmLStRLnxlK8u4LML5lcZ83vD4Nl1RcMKkk1lGVuHJC0gDGvfRfzrFdGSWyVUc6DARBXqHLVatA5KMLUSiyeNldxu2YyVcG1mTh1VaEpgPsUVsnyMEpZ6j9KUON0jMRlRBHXsiSnWz9ySJhG4CKpXnYBGNrgWdrKLasAKFIsgnC/wVHIFiwIkp1JgcRkSA29BtcgpUvDW4px/PpzaRAtetcVWGmwqrijIp9ecT684m57jlnMo5wzFgPVUsznF/AI7vSSbTsmm1/RFSHWAuArNC7x6DuOEgySlrixqFxRN9O8pG+X218AvCCHxg6JGpQ61iA0MqxXiQlRhO1E4aPqbKF1LWl9sMWtsQoQW3PafRfN7Ruin2328ehrrm8N2IZQXxU7akmWWENFdEMaeQfPZAZuSZmNCUnpJEMJ2W9g1sJiAlsJyoZydL3l2UXGWnzKaeI6OxhyMRizKioWrWOhNy/5uz2AJffH99+7wzoP3mM1KvvjqKY+enPDFRbBmjPEkr+kJ6IjvG0BbDHpbuwhfaEn8nWhqemMxivWa78HTsCZPstmc0llsueTy9BmPPv4N1+dLrn0ot3bO979K3udA/c8f8RfzOaI1e3t7JFmGMaYhlYIX1jkubwovUly333+ZPWFX9f26/dz2/os+03rOQqKAcMTfsPfzn8ODd8mTmMvpnNPZjH6/T9xPgm+usmASqigkLZ4sPAsDMzX40Yj0g/c5Pvk1s2czFswp2HSIu9fyY0aEw9kF3gnqmgx7VaQWDseHvPvOAz4//XxTI/dWtDGeN6P9/MePfo+IkEQxaZIQSQ/TCzVbBeiJIEYwvSjUuxXBqcd7pXCWZZFjK0flKpx12Kqico6itpRVznx2Sn59xWzR5lbDzZTiF/QKIhD1msazAumBiWAlUFXrWuJvI9oAqZhmskPjOW50waDySViRS0IxhzS+UUZ2jQiLUIDmiM5Io81Gw8iSiZBGrQfUN8fbJABXBLFhCOsEoHYcqQEUlk0xiLbYmBLGluABLpDii/DCy5a4UgCLiYNYYrW55K1OTeXVld/p5VVQwotg21BbYqyjJ3l4XQymVuJlQWIrfK30nSO2LhBHVXAOLYM1Z7+vHCohWqBQjvZI9qe8P63XuQm/AH4CZBFkCuoV7xVRDb5cHyzjld9EXdt7Z9iowNtWhe3bJVvbt/e9JbftbWu9sUGX/+HzJbYXp4CbauruNi2hb8WWlvi2pHd7nGwjg/3mM9s85xBIEphczbiezJktKmYK5WRJ7pXBIIHxaL0EwTchnIMx9EdDMAmLcsLp2SUXV4F4LwiTxtclrm8N8W0Vrpogv/+xYExoEGc7r88Ije7N1iNuJNymvJEYQeJ4s8rZ1iPcNjRRwVPhq4rldM75V1/x+2fLt6JqxldA9dkTsiTm3QfvcleOSZM9agnkd1N0/c2hHcq3k8l2LQ1s/b2rAt9Gel+k/L6M9L6IFPeBjIyj+Be8/9/+N/DgHcq9PaYK54slp2eXHN855sAkVGVJOZuhlZKLpzQDcqtkMZjaYMb7jH72MwbJgPL/Uqan/54l1+tQ3W6H/6OGWqp8gk3BxoBT7GKJtY7UpByPj3h48IBHk6+r8tA4CE1Gkg6QVzVOAv/ht7/FGCGNU7K4T2JCPVRgTX6hRxQJxiSYqIdvVk6r65rKV9SuxvsKp2F5YO88eVkwzyf44gpXzJjnOTdjAS/rGWLYG9Df26PMV0HxxYRSVPhAfJ9r2G/PFCtuB2CBLAaRsGobqkRNAq209pG4RnxNmoa+vF4vFhTKxgYCUQALYsnJ2j4XyBLI4oLMhKcswod6v9ysfFNt/b1dKnEbL5RObDMUfN3aE000Q0zLPDSQl7qxVseN+v2KTfXi7CmigcBK5TGVJ1OlX+REtmgqKQip96Ae6TXRyNIj6onLElPXiC9RYE+VYaIYEWIxxPvH3BfDYmFxtkRKy3t2zqFnnR+pSqgKtFrRb1a9qzzIKnCLdvXNKlzuc7erJa+7Eb3tOuut9awlwK3a25LeH7qFt6LJ7rm0hVHa9rdLfFt1VwiJZXujsANl84HSg8w2idEtHoxgsGc4v7jk5HpGvgzN0Sr0ZiXvVWWzAIuQiNAXJdMXG5z2gdE+mP4ArzBbFJxczrioNrywnfy87r36wfE+oRRGKMwdbvgPTcq+KfZ5sY93VwV+XbRLe0bNCjESmUbxNVu9VtNpR5tH2KtnPp1yfvKUJ5+d8gnfv8L7IpwAX3zyBQYYJgnJMCMyCZGE6hXeb1PF18eLlNv2tXab3d93ye0ucW0Vge33brND7L7Xvt92WkMMR8NfcvjLX7H3q18wNwnT5ZKn8zlXiwWLfMneeC94/ipHvliglbIwUIgynwVP6KgP474hPbhD34wYXC0YLksG8/+XihzHm9Qm324cmge88/5PWCwKHp1/yWaJlQ3GjSqBd9iiQJ2jWFq89WRpxsHBIQ8ePGRRzLi2L5qaN84/EQ6PRxweHmDMq1udLk5Ow3Odpgz6KXGUECfpc23XiGDiYIcg6q37AlUNiwvUoQ+odYXWivMe52rEFVCV1PotEsrSIQz2Ge+PAaWsg2aURAMqcayX2i1c08nEEKeBgUijQXndNHyJG/YmDYn+btFWl2udU2oLinwWFsrR4H+I4+DU9a7COU9N08dHQZiJokCgYyFUbHAxNKXM0qgh1c3/aB7sCqGK3EuNb9+6T1ZYtcbIlyCs7aDEBvCKbWzKN473dSf3EhTXlxgRkl5CXyKkWZ1vYHMGZU7mPdlKGKIkKJF31GWJWotUVVj4ghqoUBEygVQNiUAa9+nvCcfZEL9XUC8LKBaMp9Avc3ytjfXao75CBJIK1Hl6fiObhL2Ha24d57uRvN2+sCWM7TYRQZhot43YXVXzh0f7NW7N0dZoSfFuRHj7XsQGRo3nQQDJwAwC8XUJzC42lshj4MH7A8QknJ1fcXH1fLxoMr3k6qpPOblkOdNgy2nQ+q3L5nwPgXcHMD4agDEsqorpbMHF4mby/JtIIvzBv6v3gb8n3ISS4Cib890S3x6bm94a1F81dUX5/kIcxiRh5TPf1G8QQZpyYEIbq2oeyaYEjqJURcHlyTM+++i3fMbbQ3pbfOYh+eILhsOE/l7G0OwjcXNdjSyr28P9G1yy+DZSu9sBvkjRbd9rP7Or5r7I7nBbp9snTP7u7f3n3P/HfyT5yfuUgwFnkxmfP3nGs4tLKlX6aR/B4D2UtqnJqiF0m1Jxel2yWHpG+wNGBwOGKgx8wuLO+/R/WXH8G7DT/+PGyjx/yvjFT37Fuw8/5MEHH+Jczen5OY8//5QvvvqYikt6xIzJ+POffsjd4yPqulF6lxabW9TWxP2Mg0j4ycOfYGLhq6fPeHT+mOcDiW2RpBGDQcbB0Zis/zrOPwk2AprOXkDk+Yoxda9Jeu0pQrRVlmpjf6IGrTyepryhgUhiTJyRDVOW31SVjTNGgzFH4zsYMcxNgihkkmH7FblJqQaOtWk0Dis1Igq9JsFt1dgh3Ap8b9NvFU2JmKrenMob7qzaZFnnwqlcWZjN5kRRO8cQsmyIU2U2t1xNdk7BBOLbiqfqlaKY4eymW4oaNqQ+KF/ehaIXTjcK5ZtCT7/+Fo3HIQ8ONBTasLDanmM0im/8irPg1BckYhhlhuN+hvT3kGFCNFVMNSexOVlVkTmHUY/4imo6wc2v6VUVRoM3wYuiaYJIwqrfw8cJksRIEhM3Ewdf5qi3KKEtK819rx1SFQiuYbkVq6acX1vNprWQtMvTbCeswYYQtzaI24SLb1L//YfCdoWgXbSKdWvtoPm7tTq096j24Cva3HP6CYzGd+iLsOCCe16ZTMI6Cx8cwP2fvM9stuDq0wnX1fNC37OzkunZp+t7vU1ahWCtOG7+v3MP7r7X5+DoLmXlmU1mXEwXXPNyJ8+r4Aclvn8J/A3wt9zH4jhnToRbVxR4k+S3R5Dx77OZ2LYNICJ8Ia9yPNfsL+O7z0+OTIqYJCxH3Cw+cWPhCqLQYqOgDocyYJ6izDk/fcwXU/+cJeNtQAE8KuH45JTju3fpD4ckWVverK3q28Ze2tHj1bE9g7+N9G4T0t3P3WZV2CW/t4XJdvfTqg+tapABB/Jz7v76V7z7b/8ePxhzXiqX+SmfP37C6fkZewdHjPcOQEKdT+s9dVERe8+gr4iWTB9f8/SyZHD/mJE7YhiPGJuM5PAh/b0Rpi+U/+cTSn63ru37p4o/v/tTfv33v+bdD/6Shz/5V0DEZLHg8Vef8sXvf8rl6WOyLGYQCz/78EOODg6ZTWZMJguKRYGzDnXKeDDCDMdEccz4cIxIymKx5Lq4rcxZWLY2ywYcHY4YjLJXvwDZlOype0olGl5a3eROgoJEIeog7YSYpp00qaIKKh5XQyJAZDBRQio1aZayXMfbXnpCkKQMsj32944wYkjjFPFKjOCsI4v6VM4H9mciTJpi+gbf86iEVbiEkGjrpiW1dcGrHEXkNsXbNJQOa5+i9Sx1FZLnvnzNXnarGykI/Xf7HKYCw0yBghrPZJf0sjmluCHKTgm1eNscvnDZRNKUJXaBaDob/hYTFMk3NeEUfUkBOAODAxgOwcSBgHsfzme7ASnhfF81Ub7vC1ITsycZd9KEvknIzAitctylp1fMGSxz0iIPrMpXcJ2j8xzxFcY71CiJAS8pwhBNBRLBp30SFRIitK4RFO8L0GrddymExUJcEQQgD1RhRrAifC1tUlprdWiX4m1/2hVXtwlxq5puR8a2k8e+bpGi7xsrnldz2fp7vWIdm/GpVYDbCUBFyFk1jTycJGNGB++iScKSmEouqUzJeAYffHiHew8eUupjrpa3r7TrCFzuNgjNQlgJPHgPjh/e4c6D96nUcHU14fLqmqvJ8wl1bwI/GPE9Bv474CEDjohYrJuXI2bOAcrHfPtKAy/CivDFTAjqWlu8uQ1ZvOqM4oyNp+ZFHq03BYkSRFJUHSINpQq+hvXvEoX/FQkhzrKknM6YX56/1d7pKfDoyZTh/hcgYZKC6SPpdoFJvWm0ewVsk9QVvLQa4K6V4UXbbCvHL/rZ/cz2rHsE3OFD7v/dXzP8yXswGDCrKp6cXHA5maAC6WjAaG/E/tE+g/6AftKnJxoK5COUlcM7x3IxwS4KJAVJVmCWqEmaLNhQYN8cvsfo+imW6Z8s8d1D+PMP/4x3Hzzk7t27jMeHeK2xWjMeH3D/nQdkg5TxMGU8yLj7zgNGowPUC9RCllq8C/d2OMqI45h4kWKymLuznAezCdefTnieLKbsH4zJ0hTvwC5f4w63aq0J/4sRekZu8EEQIultjXabgkw9GkIDbWEX0FAyywnEkScViGIDg4xQx+xlULi64vxkn+FoiHOWyiuuKHDzgmJpqfMiqLlpAv2Ywzv7xDIgbkyRK8ISr1p5JHFhVS4JtbyNKLURVrqdZ98cdy3tvRl5oSU5bX/dlhnLNKjqvIhQKmFhORNK+jobXB11Ww1NgnpauLBAm7dBXfbN52ptLRBhd+3y86+aAv0y0htlkGWQDcKxCw/FLf67lWsU8FfEvf0II8JeUmJ0SuzBIFRujto5TM9Znp9RXZ6gvkCdUjfrmQRvMNQCakCrikoMGkEyGAIgEtHDsKoWLIs5bjrFWOi17JXGatBrvrt6oy4u4cYaqu2tb4leyzpae0BLerdJ7XZUUNgQZbjZSn9oFXjbcrltvVA2SwAPuDkmbVcrqgiVMyYljEvYMzA6ViLTJxuNONIKNQLJlNH+AhC+evyU86s5/QHsL3muctAusuY8UsK4NwD6g+a7J0ExqFfKRcliOmP6Hc0qfjDi+98D/xN/Q5+Ix1xxRUzKEaM45WF/Dxf3kat/z8kb5vsrbnpv22LWr0NY24fgHhsi/eQ19vciiAlhIGmUnDaDLYyPApEQiQl9gYD3jiqfk1+cMZ0+72x8h+BRTiSsPPO775AFtdmwL+pfHfAxwG+fYOIIiQ2D/Xtk2d76OlFCgfLXwK5Ku+vPvk3R3U6C2/38iz63S5p3t287pgGwz/u8+w//JR/81/8aPbgLoyHTp5f84fMvuZxN8SijwzH7dw65e3wXIwYjJhAIPF4dK1tRLpcs5pfUkxkLSmq/pDT9sFS08Yip6OM5/PBDxpeHTD//39+mQlNvDO+NR/z8r/6GX/3yV3z44YeMDh8wPj6isB6nit69SyoR6t/l4GjM8eGY4cEhg9EYISU1WajqQFguNhtlRHHE6HrEaDbAOoevHefn55zNdpPdUu7fPWKQDXGeb+ef3YVIWE5LhF4CEvUwsaCrNqzeC+puT5qJXLvchYbs7SYy0A58lhXqV3iUqlLSvqICSRbD4RCWz3ufn8eM1ZcnPInTZqQXqskSnjwlTKy2a4Zk5AgHWYzJDL20F0pO+RUqntpVCA7thWs1gAwFr+m6Vnn75KxW4f83tbxFzf/P3rt9yXVcZ56/3CfOiXNOXiqrCoUiARAgRFG0rFZ7uUdy99irl726H/t/ncd5mafx9HTbPVZLli2LupAAARBA3bPycq5xIuchIiqzCgUQrAIEUO3NlcxCVlbmuUZ88e1vf/vFMb/DAcT4EjnJerTGMb5aQV1CuXDMb+gARuzOz2J+3g2ih9NQJn3YyB0oHWUOME+ewWF5fSa4JyAatH/0c8f4lqVjepvL2I/Gd4q+Isi4s5PQs5A0NborSawQozDVEXZxTHPwNfbLRywPX5R5rcsELWDnHa09pDEl3eam1yImIDFNNWFxekQ9OT82hzk3WeIkPaz8dsNcbzg/BoffhQgODQEUr8c66A2FYGrtfeE6elfAN7DQ4ViECP+2rNheX7eGwQHQ8HPlH1/41z4EbhnYKixK5YxGW0gu5FsDxsUNJkXB1/sn/Pq3X1C1kG/lqEFDUxlOp5eztHeAu1swSiGNoPILQqUBSbGuJzRNM6eclkwOL2eR30S8M+DrdrH13rcVx0z4LTV7bYy0u0TscviWu/jA+e5Z1423LiMQIPZ6XuvlDevtZ7B0YjDGYGYlJ08fc/Dl73j4q3++FIiHAcJY7z//ljc9+B1eZrMSYgEcHR6SbYyRdIO+csV7NohZ7fWGl4t/vV68sMYrXwpszSXvWWcFLvODXP99KC7Icc4NKTdJ+ztsfO82w+/dQ0ZD5qZh9vyAo/0jimLhFKN5Tpxq8iRFBNIkIUlSmqaiqgztyZTj519ztPeM6cOnrkjoYJuyv0OpwvBXuKNrCnTVsNEuXqjQ/WOICNgabzHIc7dANC1tV1O3JXXVUhYlbe0SmUrFxLFG6ZwozhxFFrcQ1XTWlf91wEBrtNbotkDbmtHmiPF0zM7ONvvTp6xfLRv9MaPNAf1cO+rvOvbT1vdLtZZlYzFYWtu5b1uCd/JGeiDS82BXYZVC6NGt9aBtug7bVti2chO1B8xJLOgkcZ5drxvtHs2vp6z4siAaOLfxwILmq9/z4GBEcmvI8HYfMmhoaLsaW9fQmLPatqWXNS3LFsrKNcAIqCLGd027XvT7DuBJCxfJ+AToDyDJlUO3r4i2dXte1yvQC1D76ncDL7Sr9rjMFccp338iXrGt153t/PrBgb/OdVkO6y5rQ2OMy/+2mznW9SqxmJ3Qs9AZi206ys4gjaF88oDFw99hnzwiPjxPHCSAGnrJiH9NBCSJIFNO5iDOlq8RQ4VyxYZKaLQv6mM1JgeLMrh87A1uBq86qwFAvuwwhM8Msonw3Wbt9+8igsRh3a2hY5VFkLXndQeHsD8BA5WsAOsUR+RVXcGsmGCOLUfTfabVnGnVMqsajucNVWsxKOg5GG2NeeESi4DbAndvuW08nsB87sjBObAD3BxUkCwYj43rbt40WG9Fd9l9cV3P5HcGfFMgpiOmQ6g45ZRfAw+oGNAxJOXgvVHPvCcR/Hpj5W9Q8XJXd5ysv4xNu6CanXD07BEPP/9nfjt/EZQHfWvwh/T16G+NAQyWL0FjNMcNIBe/rwCOTgr6J0eMtj9wE7kIIjHWtmfM71XjVWP7ZWyuXPj3uq5s/XER9IbH+k0bZA2b3Gb79o8ZfHSb5MYIfWOT0e1dZDyi3J+wv3/E4cExdVEgKiFLMwajAWmauKrlNKGfD2BqMcuCajLh4Ivf8fzZb4A9t5V2B2aTtT04xZWOLqix76XW+7oRAfe2dtje3CbTfSSKaDtLbNq1Jg5zTO0q8OM4RsWaWGskzhz1IO5xNmlGAjpHBhplB2hqBuMx2/NNdna3uHmwzf70gLC02765yXg8QuvcXSvXGcJC225jQHV0jaERV7CDdTnyALh6ShDp0YkQ1cqxvb2AKoSu6+iaiqWpoad8cZsiTmJirSFZ54deJ77FkqmY0vx+ypEao8YxRnnf3KpxuX8BektYeuhx2oZLdZV3zoH+9UenbOSZzbkDqev357APgxFkWQRiXjrpYp1eNjyvM8dLXgTUF0PEaW5j7YvOIre+uW7Esgb+rLts7Noq/hsZ3Steq8Xpics4mCWmajDlgnZRUj76PfNHj5Cnzqo0WHsGcJndcI4BPYYsiVAof89ZlsrSYjGm8HlwAVOzjMBmft9822Dxbhs1q2I/u3ZSAiiML3z/eoTXLo7lFw/NRb1wkAgsebf1EsEVCFZSjIIV2NVr7wmg9+LzutxmFl4zMJ1PKGzFo/3HnBYF8wqKFgoLlQVUgsSpyz56S7oQH6fwb390E504Wc/J/j5Pnzgf/3A3PwfqOYwmcz5sLNZYbO1yHwkvjjQaZyN73eP1TsK5NxSMSdmgT84pLa6ZwhMKKv75j1Z/eOXwOr9VZx9/+557ciNd2xTMT47YO7Vc1siyw80r6yunUPH5NnqOB2VuuNHWbVXW03sGOC3g5HjKblm65K0oJFa+eheu09UiMADrsc76Xvb+kAZL8Wbza79rWRVGhNcu/nzeEicl37zP+Mc/ZHjvI2SkkTzF5ClzY5k3DYu6orOWLM9RiSZPc7IkRSmFsEQphVKxl7wIzbSkeLbHeYHNgX/8rxM7WjMaDNAqdgu7tqau50SFBqWpy5a2KGnrAtvWYCNs66y9tESIzoh07cWbLZ01dJ2lo8WiiWJBaY3ONf1Rn92b28zv3iE7yrC4iqbd3S02NwcgmrbtrmNAAmaxdpMISxE6FdFbU/D1/P/OujQSsRTBWAFRrlV5r4eDyC4BbJctbRNRLoSpFebThWsx/LaJhq8mmCpatbyqu/NoweBoyue8iDjfUIvB0cgT6f4mXizcUdlIYGsHNsbCoK+IVM3WGA4uqD8igUEGmR84s6DhsueP3jr7CH5ZlDj5QaYhV56lNNAW7vBfF/uqsH6JvbY7dgA7Em+/Foxc3/DE2jbLsysyQtEZaKqKpixpZ5zVU6yPu6Id48tgTBdtxldX5AAAIABJREFUAso10QAiGrR0WIkwokAUSyv0sozexiZRksBgjjUdoiKiaEUEgcUsWyJjkQrsAqQ+z9QGYBsKi5O1n+FFAiSM4SF6/hH+LrhDvesIx/fMoYGVbjnUIYV9D9neAJAvSr/D+VwYeLL/NUaEr48aisatQwK9nCQDDAlNayiKilnjvjsczzu7ip0bm0yOj3h2eMiTr5yc4qLMSAGiEqyBYl4wmVRMuVz6s8l3GPg+Ap5wSkrMztYNdo4rNjim5bvj4fsHDyfmRdT5m1K8a7pLPCoasVhbs1jMOOLlLG5I16yvBoPB+psu0guplLDdwQj8siiA09MZVVVhjXFOFolb9RvDKn93xVgHvxdlDuvvCcdAEZha2OUeCTkSRdhlx9xOWHBCQXVGUF0Evu25z99E3dpB3/+I9HsfYZVL6VVNQzmfM286GgtxmnPz5oCeKNeoRPzgbj3Tr9zQZCyYssFyeK1j8l2PCNA6RkUuv9t1NXVdsJhPHAIgpq5bbD3H1iVtWdCpiDLLyAYZAwStc1pdEmvlCpLKmq7tqOuSOBa6zhJFEVrH6Eyzs7sNYtmebtO2Dh5v7+ww3u7TdhFl2TpT/atGN/crRo+qIqGzPZRKkChGibP8Q6Anxi8IlaPAAIOTM7gsiQVl6HWda6zWwGRhWWA4OZzAyR+geXUJfNW5yuaBrBQS4RAteHXK6Q2syIdjzxRqx7j2R461HY/hxjb0+5m7jmphd9cSiXdu6ByI1NrJFJSXsGQx9L1xh+BPld8fWZ0KNz77U6G1kzgIYApYTC5zlv72oTNQmQO44bsHmbdeE7ed3chrfX0XgJ44d7nrhCFCIaASlEBXGpSqEARl3b0ZHuAJ/hTI+9j+JkZvg1IuYykWJQYl1hVhGoAES4zoJQyGYGpMU2GtdSRAktCTntsGLMZ0YAxSN3SnM3q/OqbGXV6n/vu38EzoEPRNSJOYSOUuy2IbrG1ZGvdP0/rki4XIQLp02yXizmOKK9BrSt4Oa/QaEYB8cKVY34wlqwI/xUrPu946++L1l+OyH4WF/WeGaQVHfsG0MYJRX1DJCJUOmRcdk6M5J/Pm7PbdAu5/IIwHA6rpjAdfHPKr45cfng9vJIzG21hgNp2xP3e0zcWZPgK2UIwG14O+7wz4/h54wIwdNrgzHLN5PGOL47e2cgqMXcd3V9cokRCJ0x26W9xDOAGiCJEOhXXpWWsxpvnGRcQ68A2pHi8Te2MRlIBrRbjErFw1gpdy+P4KmM1hMVuwWJQMk9TbtfmCnTe0XRfZ38u0SYrQr3zIbvwZH/30z8i3RqhcY6xhenjA6d4R0yfPOZp9hWGGoTorrgiDDMCAlHzjNv17dxjc/ZD8w123SKkq2tMZlS3oRFBJykArkrTvU5YGYzsv5vRslQgWS2sspqiwb60M4LsROQ6QROImT2sNpq1pyzkLYoyx1EXN0d4JdVFiTI1SMTrP6JcjV0zl3UM6XMOHtq1prQPQsRYntQFExeg0Zrw9RmvFdj2mqGvKumUw2iTr58TGaXHtdRZpxnvNWONabEUCS2FJjJAhKLcwAjdZ+7YkSy+atQhLL3VwizCfDaoMzbR1AtWqgekcpm/DNOiSWAKHwMx+e5DwBgalkQe+WrvisnDTj0YwHPXRWhOLUNcaY0uyzINZbwCr1mlB/7q1nLUjro3T7EYCiYYoXm26raFuPQD0rg/lBCaz6+9amjgQn/V9vxBxwFcrt69Yt49KOZlA7Y99KMJrrmGUakmwIljRGFFYVWAkda+x1iZ6/TkCY3zGSlwbvKU4FU7SE+8x69Le+CZNPVGIZCAWsU5LGsXKLQSl5+cGC8YhU6ks9Bd0nx9TdSvgm+Ezj1ugbkNy6xaR2kBFfcQ6xavtWqypsKaFtsIad6LFgLUNuqtRkdBPNZlS5AjVvIJfvBvka1/yc4izJLH/fQC9FZcTXRmQbMDkFB76BhLBulUbGKsUSVJUkmOLOYvp9NyadTOFT+/fpypOOD0+5PErQC/AaPMGcZJiTMP0dPJSwm4MjMcjRvnwFZ/2zfHOgO8+8H8BKYf8sP0eW6TcB34I/Ne38H0GxyRv+n9/F8GvJAkSiytSAISIKJIz305BgRiUysjyIZLm2Je66LmLeANIc0FnCeVpRWHeLOgdswLVDW51GRYfLaHIMewP5wDjwdERXz96zC6WtN93qX51vUv2HFN+yXOIDCfu3+FD7n/2F9z7q3/P5v2P2PnkHirPkcQNstV8Tj1dUJwcM3l+QDedY07nNNMFzXROUxaUxtnAJLfvoD/6kMHdj9j46DYyGhBbi0oSlErp94dsjDvKokFESBJNY5ZUVUXVNLRNQ2UaesolrExV0UxP6SYnqD/6VhSvDnf9tygsOrJowS0CrWF+/JyDp0959OgpP/+Hn2OtpZ9l7H7wAUSwvbNDPZ9SnhwwOdpjcvCc+XxC6/1kzaiP1dBZ6xhc2xILDDLNQMcUdc5sPicqS7RWxBFAzGAAnb3G9SpefmBb3zNUoBW6KKaTlp7EzuKMjs4WLG0NIr6BhbuaJRJ60vOFfkJnheXXh/Dlm+AYrxFXwQf6in+3Fls3cIfUH9ZQNqF1htaZkxAB1mT0+zX11lrT9GhNBd2tPofOMcAa36zCF75F8cqurK5hb8/J7udTWDQOiF3nrk38/gy3YTCG/tAz2b798LqmtR04VrsNvUWsZ7Jb5/hQlldPpEm/j0VYdMKsMpSNk25UVijjlZwupM8FaE9g8ZsZ9Gew8YSlEogakA4t0AgsiTA2wSZDrNpAxX1UktFTCfScu9GyEVoBa51jr7HWycI7C42hWVhOOsdozvDtdP2jOnZMrTVP6eITUMNVD6i1So3IdmfkjWsoKnQSEacJ/X6foc7pS0SbdeQ8elOqnG8VoaBNcPPpxXXMwL8e4XDQY5xsb+QfQfYQspMj5RuKVav6IA0MUrAJHJcNUk6BipPjydn3RbheCXd2txjlOY+/+IKH+98suts/Ldx9kvbYP6xeiloqYJSmbG98RxlfgJ8DP6SiXbT0JWLX5tyn4BdcXe6Q4W74l3nWnrWffEWEjibvW4hyur0oWrVGOGtgQZjqFJFKSLIhie6/sphrCAxzRb7ZR2nvn3f6Zvc8lcCGCGLsWfecoPkNsrOzfWRVeDebnXJw8JzB5ghjDPqsAOfqcXE1LLwIhoO0YZd73P/L/8T3/uNf8Mlf/ZTRh7tkN2842UVoC9s0mKaiLuYUs1PMvMIUFc28oJstaIqCqvFm66MRMhyiBiNUPgClKKxBmQSdGNpmQFMZmr5xwDdNaBpDUVYsipJ5MccUjtE0tqNpCpr5BDO/HMS8bV/p9y0iWoQO5T1Wtdc3lvMpB0dTfvf5P/O4fOTeXEPdlty5/zHStbTlnPlUMzl4zvMnj1jMJ1g6VByxOerTZtqxSdZC2yJYstjlK0SgrmtUXTvjkRbHSkl0vV4r1ntMtbXLT59p+jWomGWk6JSAGDBepScrlhfp0UXQE2EpvorKKpi+Y9B71XidHhvfEP2B/8Hf9JFSKImRSBOryEkTOsAKOsmwg9od37UI6pUAesUqYonJ0LSmxbQGazsE11SkrVuktdjaOcZNzKvnt2+6b3vAjz6D/hiyIejcs7xeQyzRKpNmO6cjhvPA1lrvM+yJ/+vYmZFrrFlSloZFXbOoWxatobJQC/QjSLuVh6wCZ8k3A5mBOqrB16+hoNdzz1Z1GFVCVrLMLAwEJSmitOuwIK4YzmVoGtoOjLGYYONnFa1VlKz8ZcMjgLxu4gsV4xKSkk45+ccK6bKacl6YfkonnhZBVEJkz18nf8gIxXX+Lj93DfVYyerB7fsCD3wVqIGiMobK9xYBUAO3+KjWrokkhXjkMgazytCzFrFO9xvithLufnCDzZFLlUyPXbn1qyIGLELVNMxNw/QVh3EBmKrCNtfDKe8U+D4Gfg3snR4Bll1SPqPgR8DfX/L+cOLWjccvRu4fEZd7wL1OE4f3EfSCk/ApAaMEZ2fmX+fM2AgQlEpI+gOyPCfl8n3WQI6QZxsMh0PSYYJWKUpNOT56c2lPZ0IhznvWNOfmrmDvEwrewj0WtMambpidnlAVc9fXPU2Qq7YXuiQuG+cTXFbgHj/i+3/zN9z/z3/Jzfu3SW9uQKIoiwpp5Ix1FU+fREnCcDhkmfUxxmLb1oPihsZYx0SolE4pSDSRSliKkBiDMda9R3U00lArx/hGqSI1iiRVqFgwpmE2ndJWFbWxLE5PmB/sMa+fXZrB+F8J9LorPyKKI/JU08+8BVmsaNuaydExvz344tzfHC+OMHVJnsbYek55YHnw+b/wy5//jMn0CK1jhqMRg0FGfxQcMHHdx9oWU7uqmbouKedzFpM5SltibbABaF7HzqycullmUTukFLRC0cLNZJmGvoZMfA9an0eXHihB9RKU9Jze0+ebLVBc3KZADb7Pps6v6jTzbeKFhchZ41ssHXSRJ/q6M2GnwPnqJb9qj/zniQWNkIlCrDOSLUvDvDCUpWN5J1N49jXsv4bC45vu28++Dx9/5tjdSLl1kHhdr2N79RmhYG2NbR1YD9sbgLvxcossaFivGhqMWE5PZzw/PeH09JTT0xltVdGIKxyMgb5xxhzCyqwDQLXQO8tirk5RsL1UNyFKS9JMoXe20JsfILoPSUJnDY3pKOvCSZhaRzIs7dLdO6mCrRh13NLD3UKhffEC75K3WBWqrR//ixKNMF+EIjmJoEwWzNSCvkBbvbEazCtFAL7BPSjM4mNWi45QhxI62d66fYuKHodff82eWTlV2DmkwwHpaE7/2B2zZACtKBpjMBXkiSVJEnbGChrLza0t7t3+kPEgp5g85+uHj6mMr2N9yTZr4JObY7736ac000OmB199Yzb+Z5NjNibH1zhS7xj4AvwW2OOYG/TZJeYHpPyIiqc4YBziNi4tD+6EPr74QbidGfr3vQz4fpfDMbuC8sVsgANeAYD5Rg8q1mT5kHTYp3/J50S449SXDfrpBhsbA/LNjH42RETzuzcJfJVjqlWkiJUhMfacTZC98AiLagWYynJ6ekI1m2GbCuzQHYPoOmhi9b3heX1gi4FNdrn3n/4Ln/yXv+bTv/kJSSrEifOpLKrK62yVsyUKjLZSpPnI7XCQY/hzY1zWjbKxVMY6PWnQCFqnY2s6S9sYGhVRVQIK1+oV0A0oJSxmM8RaqqaiMQ3F5JDF82cUPH+vMcvVYgQI6MgVrtFBW7uJfG3JcnERrAS0VmS5Jss0WseYumYyOXhBwjOjpK1qMiXMi5K6nvLw83/h7/7H/03FlB4xu+NdPvnkPnfu7CIh1936dlx1TWsN7bymns6ZTxfEyiK69RTcNX18F4tVz8/1E9zhgPC8hq4GNIhjzYIZqWJJqgyxUkjkWSzrmlwU4YKPgFs4M9UWOOnez8pizfWO41qEJpChZtCFRfDezeKkDnTdGSDG3+Ph7euLdAFiidAIWqLQsZpFDYtjOD6GZ3twXDpp83VJlU8/hh/8G/j4U0XbGWo/lIhAFGlE9VGRJsKi6LBtglUlWHPW5AgsdNAae6ZRvlak0FnDSbPg0ckhx6czjk4WdHWNAW5o6CvYtg5k6nrFUK47JKzXecCqKCs/hXinJM4ishubZB/eRQ2HEKfUXUNkGprZDHt6SluV3kbQQNdiC0G2N1DHrvi3gbNCtxJ3aa03pfAkPnDe1zeA9eCPW+LWRqpc1au8axeqxm9HgmNzCxwA3vDP+NeCy8MY+PDOPQ6LiuKrr8/JI04MbJGT93N2c0ODpYmgbiuKxtA1zkEkSRRbwxG3tna4/cEHfHznDpG1/NP/POQ3j6dUfptehijub434/ief8ukPfsCTh4bDr7/4xiHohOtju3cOfD8Hfs6cHwG7RNwj53+no0/LL4G/ZbViCfGyMdDgDvDGK97zXQ5X0xRGYDcgu2YWrBU8yZn9V6KTM/3seoQCshM7w54K9KGTiOFwm50PNrnx5SmHzdG1t3cE7NzcJkKcT2Vnmc3PN+esWd2IgVQ5yygJZ4yqNd4c+w0yvuvfFQbAG/TZ+vA/kP3wM9TODkYlWGtoGkvTGBYLv/0iRCIksSJNhBTHaqMsYiyhxNtiMcaxLNZa/+DMkU2CRR3WL9cTp1dToJLIvc9alCiiJCZOUuZFQVHUFHt7FAcPqL+TivWXhQAjktEIJRFxDLHX7kZdTGc1kbUILbYzFIvWTWACcdQRCWgVoWOFjgWtYsbjEbc//JDkwefngMdN2Wa8ue18bMsppi452duj8jmSJS3PJ0+YTo7o6hokQmJBrKPI6tbQ1gXlvGQxXTA9mdIyx9oY0wntNWU5DCMwfho+4jzCD2aWQw3DFWToeXZXJdCLwYpFxAFeY13Xt7PNCjddslzZXL2vwDeshq+5fZeZbITXFM4NR4IOa+29DbwgWxEgixRaErTVZK3C2piytpTTmr09ePw1fMmby77s7Dr3hhZzzkECgKgD22K9yMLiijzDXEEUsoSu6Cyy9py129Vj1eB36Rn0TkA2NuhnMOxmpAY4ge54tYYLpzSMwUFdAO5SjPD2zT8Q+j/+CfreD1EffkK3fYdlrLGiMLahMx0yHKA3h9C0GOvsGMR22HqOfn6H6Okh3cJlPxXOdWCGu7T6a9uxTsJwYbuCP24QGgYHhfDvt99u69URticc1wCCw36FQjbrf1cBz5484tHRjOcXPksLJGkfwZKamqatmJ1OXKkBkKWwMcq5sTHgg61N7n1wkzxRTI/3mR8f8/irBzw2zrb2ZcdlCORpimkqjg+fc7q/z/QbiuDeVLxz4AvwMyo2iNlFc1eGDPs3+IFOuHP4GDhlgVupuJrlV8cxTlx97Xv5PYxIIu9usOIpQzdfrOt3LwTwK8SJJg1X+FoswXftMlTzI+o9QyvC5o3vsbGxye27NUe//y3Lb1TnvDpub29ya/c2pmloa0PbGNT8vME1nHfaOHN68CxLcHOwXUtnPbP9hiLAheDcsMFttu7/BzZ++r+R/OA+djygMm4QtRjqumFa1GftVBMR8lS7AgwrDuBbgxIAewZ0W+NYX+OfseJZGkF5dlgEkkixVJYksY4pj92CwRpDTxQqSlHJADuZUs+mLJ4fMOfBHxHbGwNbRKMB/b52oDcChSLGEqMROn99ODHt9OSYyaQl1z67EHWoWIh1RBxHqFjY2XYlrX++9yf8/RefA5ASc/+TT9ne2UHiGIulrUuOj1685qcnR65cX1vExh5ctFhTUNc1ZTlnOp0ynUyZFi2L0lLUndNNXmM2zMcZplvSJJUDNQvcYxMYZ/SGGp1BrA2d7TlQ2wuLKSdxANfxzVrrJu71FEewU4ksvR7EfZDMNf5715M44Ab7IE5cN1q9RqwPH2dDKX5/w2KU1cLUrv2N4Lqu+dvbty2O6cc5ca3QxJSts3Uop7D/zHmWvsnYuOFs1erCbZfvrO3HHIPEDtq01ks0una1A9H6yV/VWYR9uypbJH5cFpz9G5FA1CMbbrGR3WTT7JOdPkEW7vvWtajroDec3m7tPel9GP74J4z+4j/CnU/ohjs0+abrRAdY22CMATMk6RqUcS3cwaIw2HJB+vQr1JOHNL+enMn+tnGJlD4OVwQZZYDw6+xt2MYActfXRQHwCu+e8YXVNgTgG629vv7IcBjg4Vdf87tLPicdpKTpgIaGpmqxZUPpSwMkhY1Ryo2NETfHG3x4Y8xHt26wmEx58ughXz14wO+fN99Y0NYHlFYU8yl7jwr2976+VJb5NmpV3gvg+7fALjO+h/BRCrvDlBtbY06PK76yMx5gecBKP1O94rNS3EriVe/5roasrdjPXsOxvpFYOhz4xbhCARFxvpPV5fK9DrcCXpyeYoDt7RJubBJnQ7bzWxwWVwe+9we77O7uMBhuUMzmtOUptnEWNC+7iIMu7Wzvzq1eIr+n1wt74Vnhq1VJGX/0E4Z/9mPk9m0anVJYmC2CA6KlqjoWRYUx1rOwgrE+8d4pFIJSQqeslzEYp+31Gt7Grgi80DI25Cmt3+GeB8OiXOENS+sXNy1KJaRpBnVHdTyh2XtO/V4Mt28iBsCYZGvEKNf0B5pYQMfWMb5dSxRZd30IRFi6tsZWJW3ZrhX1WAeYBcRPUdlAsxNt89kPPuWrJwfEKuPWR3e5e/c+Ou8znZfURU1rWzL94pA40hodO4Y/9I85S4PbGtu5NHLki2qKouVkcdGk9ttHloXrSzlv0gQ3wI0jki1NlilUYkA6jFm6rILfNgs0dtVAY2l8y9r6wiaF2VscoBptaTiG5+Uf2JYpIJ+LRUVhUAi/u2aUpb9+hJXHLrhT6X90gPH8tlmcftb3hzjbPB0psjjBlpZ6WlIvWtrSqVSO39waHXCFwoOR0/UGtvdMu+v34czyp3PX/lmGCTy1bc9+fcYYB9nclbfMS+xEUJG7R4QOHaeM+kOGbYOaPaGpVscOVkXMgYVcb3LU4S737AefYW/dZ7p5g7lKmMwWLE4WVHVD17j2bRZPTixXnyI9tzi2bcXT2YKnVXvGak5xLLzFuzvgssQZ5xsSLde2E84zvutsr8KBuDd8uq8UQcpw8RY3rJwbAigOzSsu3uk3ExhvbJHnA2jmJNIjWfvE3XHK3d1dbgxydgY5uYJ2eszzR4/41T/+mt9Mv1mKoIHROEVJj+nshP1nUw6mL3d/eNPg970Avo+B/wP4kFM22pSNGQybgg074x4Re1hnb8WrC9sS4PvAfVwa4wsuz4wFYbf2n9fyekVv7zrE/3c2UIkQRxHSdf73vopYjPM/xJLIaj9fFh3w9PQU8/N/ZPPmCeUCX0S2yzfXZL4YdzZ/wF/89KdEzBBb0SyOODo44uh09lKtzwDHusbixufW4j0TcYOqVsQqyAKuPkSvp7rDJDYG+tG/Y/DjP0Nuf0SBcLB3gMKyNNZVficRTWOYzxuapsGYBoVgGoOpDCZPHRiOFYmjX5z/bmdpjKEzS1fAZlYzjVIRdWK8Zlv5jlxOEx1FQiIK23NMikhEkqakgwGmqCi+3qPiq/eDmbt27IIekY5GjEc5o4Fm1NfEEWS6Q6whokboiHHHo61qjHSYXNPW4sFumKIcG2zamkgVLjWnYz795BNG41uIZOjBmFhrWmv55989IFOOwfv0T/4E0y4o5nNGowE7N3f48x//KR9sjej84F/aCCoQcQyXzjXbOxrJx7A/pWbBtDyis7417xXDSu2+MTIrs20NbIDuG1RiMJRY252BhtZAt05bhfkqzNqhrB1WXWWApYLGM6zSj+FtAl+Nu+nCRguXF68Jobe9r/YXlteEF4cHDvhmueukdvZVayBY1qlQHIlgvIdvloH1zedEQClNpjSn82P2Hh5TGihbxXz65ueUrQ+dVy+cKamIgkWO9j+Lg3NhhAy2bdbvl4lWWYjOA98YIL46cFMSE4klTjRJEqOspVdXpHXEoI6JFyXNc5jtvygluMxXPzDA+rOI7Cd/xYMk58HTY778pwc8ePiEp0/3qWcLTFX5nfEThTUoJWR5QhJH6EQhWB59/oCjC0YmQSf6S+BDXBKlz3lwuy67CHrjsG5cf6Q44Pw+dG9b4pJCPVbkX9iXdaAeit4v2p79ya0xdz+6TaoGaJUjvgXxllKoAQwGOZ998gnf+/guiW1QNBw+3+fv//6/8w9PLU9fYxtvAh/cGDDa2EAU7D+f8uQVF9/bKNJ+L4AvwEPg74Dd9pjvtSlppYGCnJaIlT7lVREE8kF/87L3h7E05gUp13sdQccblulRKPJaA79EAg3YzoHf88mtV8dB8Qy7n2GtwtrkSqusmA/Z3r5DrDeYHZ5QnR7w/NkBByennPBy/2TFSt7Qrp8Qz1wrEUQp15zjGtTPRf2WAAmbqJs7MBphkoSyaeimU/I0IVEJSaJQaUJVFUxmE9qmxZrGdc7yKUXBFaCpxlAqwVqLMa2TNzTOuSE8zvZJKZKkQyURShLP9MqZNzNWWHbub1uzdBpNpTBFQXN4QvMKj+bvTowg2yYZjxgNMkYDD3wHGq0gizsHeq0ToUrnqu2dO0hHlmlsOwBaJHJnt21b6rZGVI2KQwlLxGg0YjS+Q22EeW05Ppnz4MlDHn71FZ/ev8fdWzuMxmP+7b/7c4SWXGvGowG3Ptgmy2M6Ol/sr2hjIY4F0TEZMWowIOsyov6cZLAgy0YcH02p66sDyNrU2M7bLa2DMdvR2pKlXfqMgvtdt3TM7hllFpIB4aJv115b9xFcS/eXpnVM2tuMUIEcgPn6IHPh3id2WmURoScR5pqj9fGxA7CtgUHfAd7YM6hB+rDu8QvQipcU4J7DvBGJQkeaXA8o4xKJQFmFjjUR5o1rFcfbblsvbQboB/pLaQFZvYU10AthTrnedhVVSetbmWVZTpbOyLVgm5LZaU1zsg97brGwyflTfgbQWQG0YFhSf3CH2XCT3z0/5L8//z3/9OVXHP369zD7FuKub5jEOuCJf6wXXX/beAaX1tO8i5iyalG8fqkErW/KqthNEPokSC6ko5SP73/EztYY2yhohNJabNWiGsuNNOHGcMCtQc5unjI7njKd7PPkyy/45WuC3g1glEPeH5AkytXPvAMA9t4AX4D/E8hp+RtabrTQenFDy+vZhMxx/GSK0++8DPiGJKS84j3vZYgiiiKfunJXiwVoQ3LTDyDWUjeGuq6pytd3KVpiMc0pxgg1cgWN7wb9ZINiUfPw4SP2Hn3O6emXnPKagnX7IjCNVXzWuEKJILJeAvHt4yLj4ObenEYJRdPAvKDqWhrT0M9zslSjaoWqhMVixv7hPsY0YC1pkhBhSRJFJKB8IZy1BmMMbeP0Z8b4RYi1mG6JiiNnOZcomsSQJKnTYiNo5XW/FprWUFWOZZ7OK4qqoWkM7XRKe7z/3bp2Lw3Ht0eDAePBgO3NPuNBTr/vgW/ckccglM5mqm0+3Kn3AAAgAElEQVTBttg2oqZDRIMdODbD1HStq+iu65q6LlGxJm4dD9NZi84y8kHG9GDKw6+e8ct/+ZxffP7fgDkPvrjDX//VX3Jnd8TH9z9hoAVo0Toi62tU7DMqGoyN0W2MbjMyEeKBRg0+IMp22C07vldYJpMpi+mcrr06iCxC3tJb9K6LDyvpVhTTun4ngNuQ9YUVGxBC48BnzKr6JYLlEk4X9u2nvy6yu+vl86x+7vVWdQ0We222F2AyWXUvA+97Kw5Qipc0tDVY48jTyG9L2LTOQBdmTdHEyYD+YAw7QjQX5k3EohD6e19ir+e4dC42tmBry8lRQte3i9FZt82hWG3N6OdcXCyTCE08rjqq7h8e+MtNGG4MWdQFi6Zmcfycg6d7tM9cBwlnE8m5QrH1tVdgI5WG7Id9Tja32Zuc8P/+4z/yt//jFy/Sk68T34K5uW727H2RV4akTqgJhUDwOGw0Etj6KGec7GKTm0iikMSgUsvmICdRCtPgbMuKiup0im0qtgYpQxES09BMJzz68jf85jf7/L7gG0FvhAPboVnifDEnSZx0810sGN4r4FviurntAD/EpcWDBuh1L6pD//xNXNh3URkpStYGMqHrvE4qiLXEsYkBeDXGddH5NgDJmFMWV5pkNhmpHdKsz6KoePToGY9Pv/xWjHEoMDljqRUonaBUjIoVEszC3xDwDcyDJceIomoMtq6pmookTSiLgsU8RSlXLDSZTtjb28PaBkEYZClposjzzM3VktA2FVXV0LQVTdN60GtY2lWxW5ykJGlCqjVpYrGp22ORxGW0xfWtb0tDUTXMipL5vKKsG/fZsxlNe/RHIHMYQ3/EaNRnNOozHo0Yj3MGuaY/UGgFubaIiaATMCVtLdgKBI2SyB33WLnOa7R0HZRtTVkXKK3Raq3xRGbReU7dHvDoyVf84vOfEcRQhX3CF189Ynfn33Drzl12xhmtmWK7mmwQE0kHEiGRoLQQZxrdatooQsuI0QcfMtq+jyXGoqnrmrYqzgohrxQFq1xwySpXeSYw5TxlFsDuRZHf+nuCbnZdqBqE9Za339JyXeQZti0A3Qino1axc0gRpwK1dkljG9c+9poxnzutLjgAHAWKET+udg4Ut7UDw8QeKK7NlKGwSohJdJ9hf4tsK2Oj63M8tUwWLVm2x/JKSO3yuPcxjLdYK25ehUQvgtmzgjx/rENysLtk0AigN7risLp/dIhSCbo/JBv06dcDhk3JwXHFk73OVZzjSKke7rINOtog3Q6XbQz0vx8hf/pjTqKMxweH/Ld/+qergd5/jbMIqqFEYPwBbNy9RzK4Szq464CvmiMU2KbANI3TTDeWajFnPnHo64O0IbHGtSg+Njz43T4/K7751Axxt1ieu3vOWlxTJ2NIlbNa7ps/7Cl+r4AvwHNcsds93M3wbctDUtx99t1nw14MwVmVxbFLpYuXPVhwo/VZAYjF1C1VZSjst1vJzq985E6YmgX2dJtUXK/O1wW9ochhXZOUAGk/YWNjg/4wR6caUbHv/37N3NyFsGhsrrCJ01qI9VIDX3RmsIgVmqZhUcwxTeVZxpwsz+gPBhjTYRpD0xmapqM1BuMbWAQW2OIWKYkxtDZ10ofOnz9JEBWTGgUiGGOpGsNsXnN8MuVoMqdqOurGUBxOMBz+EVzjGelowM72JtvjEePNPqOBJstici3o2KU8pbNgOmzrCv2MWIgi4sgBIQu0rYZW+0Ik8cU7LdY6VZ5YiGyNlhZrS+bzY5xH2Com8ym1NaAFGWiiMiZq3dXWtcZXxcdnIFMiTaw1okfk2YCsn0GUE4mmsy2mvcxF+1uE4FZmEZezu+tCxPCa73r1Aou6/p6L3xEeBjeAvk1LM69VTjwFJT1BpIeIcp27nKIdVz/aYa1laS1d176RQX13d6XxHfRdJ7cgeQCo/Xi5jrGVdxcR7YvizrCnQscZg/4QFQ+JsxtEezNKM7negudCbN6A23fcthqvuThrXrR2fteZ3vW4CHovA8kiL5FQvEY0TYmxDUQRUdrDYuklgvRTR/Ou+VD/GgdwPgM27rt9GmYRWKGxguQb9O9/gv34+zz98gn/z//3C8zhd3+J/y7ioswhBwYfQL6ZIOkAVOIkPE0D1RxrJlTzCfW8oJoaqpnlyZOv+Qo3NEyPLePjQ0ZfHJKm8KvXAL3g6q0UzmJxMMhJ9Ygk0Yipqco5xR8Y9MJ7CHwB/gFXpLaFK6963XX+iG83bgcdTMd3o8uVeG2oiJwNrOfGqigMyhZjnNThzbWieJ1omPMMbIJqstf+q3Aeiguv6azPYHODfDgkyRIkUX5/3xzwtQBxCmmCTcRNgIEKcjJbr/uz1FXDfD6nqRwtZkzLaDigrArfpUjRWWcZZYyhaarV6tm60qiltU7eYAxN4hWLvuWlahIav5AxForGMJ0XnExmnBxPfHEcVMdO3/tduGZfHjH0c0aDjJ3tLcbjAaORZpBrtHZuJFo5v0hap+01dSj0s66hQKy8pVFHXedIVIMEOazFthar6jNZkKJFSUvXlizmL/pUT6dT6q5FdEysY7AaK15ge+ZL1kFnvcw+Js40Ku+jBwN0PiBWGTp9Qx0XwmWuWGl8w4gdQG/H+duhx8ohYZ1dfYEojIDlGjXIikV+m5FCEkPWi1FK01PqrCmNsUvazlJbg7EdnbUsrXGILfSYvWbc2HG3topdodqg757BF9X6esTQ5jcQ4ko5NwUl4cwqJIrROqM/GDGSjI0bGaX5mr3jAntVFHlJ/OAzuHPbE0GtdwuT8wztWRtlf5m+zJvXXngfrCQeV426XiAmdmOYUs47Ohakr51v2AmrVCzwCNi4A5/9+x/Rv3mLm5tbgNAYg1UJsn0Du7HNsy+e8NtfPrrGlv1hY8T7VSS/nvgRIB/A5i7IYAeVD0ASp5lvKkw1p6kmnB7uUZxOKeaWYmp4zOq2e+ofL0zWrxEV7h4abYwYj3fJ0pyjp4+YFvM/iG/vxXgvgS+4Qrd7uIM+YzWWb+BWLoV/PZAACrjjf95/ze94B5rqNxIhZR5CABtdhIMWiN4wN/p6Madh8S3omYs3aIRbHeaDIZvb22TDPirRa807rh7nJQ7+O/PE+19ZzNLQmcZ1qjEGY82ZtKIxlqpZUlXOPgcs89mcyWTicIN16MGinG+qdXY7qw5u1jO/HoqJIrWhs5tBNRVFFaMWiqaxVFVLURbMJhOOj46oKyd1mM2evTd6sqvHmHw0Yrw5YjDKGQwy8jxGZzFxLCgNKnYAGDHQdkQXHPtt1NG2Flu3aG3QegBiiSJPj3Yd2BprW+gK6jlMj2KoDxjnL15I9+/tsD3SxJSuoqkrnaOEdcxvFxCCEQRFrDVa94kHm2RZH601URwhEhFdvyvAi7rXGN+yGLcyfIk2NliC9RJeiKUNEtul/7fQWxPdLrPOifHeBusbQdoHnWhi5YBSoFqNhdZ3E2tNR7c0rsLM2FUe/A0wvuOx3xTP3Eq0YnslAqthMPBqkNizvX1IMohi9zfKClEbucLD2lDOa/p5Rj7MmM8XPHz0mIcP38x0fvcjx1JnmTg/8AvFaSEuI5jXL8F1+zLbrQFlr5C7TggGWYI1JaYRbLWAtkSZjgQuLV76agb7JiOWPlHmerLWnWs33BY1VXPArx+/TrnU+xPvE+iFtUUbbrjIBzAafwT5TcjHYJ1rflVVVNNjZofPePrsOcdzx8C+SdKswsmMRsMKxhYRxfT0lP3jc2ui14oeq+K8q8Z7C3wf+keFOzDO2t5Zle3gLNBO/e9bHOi95//9iNcbt4PO9021gX/r4ZlHYwydz1tF3lFdzmqNg7WZoHVyacviP0R8Exs5AHKEU+zZim/dr77fFzZvbLO7u8toYxOdpHSxQlp7LewbVq8hKwwgg9z1XwSMaSiqAhFh0O/TNOasVbQxLitUNxbaBozhJDlxn2bAGotSCUolXo8tPpXrJqHGGExnULZyKSZlGImTNjTWQlVhJ1A1nWuPPK+YTWdMTo44ePqE2fSU6emEU754J6vkNxcDeuMdPvhgh+2dbcbbI7JMo7QgWlA6cqxvLK7zr/Ladf/XgiAx2NbStU6723bQWpevVZ6qF2ugtXR1TduW1OUBi/lT7LTm4x2BeoefPzigAn589w5//Vd/yt1bA+L2BDudE1l3tbhOiDHYjs4IdDGgyfSAbHMbPRqj+5vEOvaLpJbOyvVbwcZrzxcTKOsmo+JaEotAFBpYSM8D4qUHPD5D1At/bmEJPZficNttBbvRw7QlzN98ejm6CaPRyLmzSITtCWbpCg9bs6QyDaZrwTQOnS1ZeUutu1RcIwLwDQBQxD97uzLVd5kGxqBEiCVGtDi2JQDJGnesLEznc/b299E3Yhju8vjJE/7uv/6an72Bbf3pT+DuXbixCyq27lz7fhTBisy0KxAbmGzgrGANVqA3vHcdAMPqtatGLA30XAFvZwq6RYldLIjLKRsNHFyiG5+ewq8enjBTmxynJaZnKbuKsqpZlDWzWcHv/+dvrr5R/xqrxoy4THh/4xZbN/4Um44w+ZZfTFqaZs5i/xlPHzznN99SGvm6sQQOKlAHE0ajbWKV8/TpnCdX+KwbcG1c894C3xA/x6e9ccB3CycbCoxXBGdFcMGf/dtED3dxfBf0kq4FpaXrujXG1xIRc67VkLf+UklCmroGFsGL8CqxpTZI05SiaJjY63bJdnP4WCXopE9TnJyds3AOAZra6ZSNsVjr8nqRKGx0PTRxcT4SQA0SlspSN3Mao6jmFSKKalBSzud+2yzT0xMWJ8fM9/cwx4dI01BvDZiOR6i8T7Kx5YBvopynZRqK2FICuBBfxKZUhCgn3WiMoaxqjDEopYiU07x1xjI5nrCYz6mKBfOD55w+/Zr6O8/39hmNBq6obZCRZRqdKXQcEceC1jFaK3QMOga/4qPz6W/x6ztLSxzHNFKj45hcA511TVtEVppHWlpTQz2nLiZgYFML7GYoGfHs+ZQ/vb/Dlm6J2xO6yZxCxUSRQsUaqzVWabpIYyVGLOhOUIMBeX+MzkZIrJ339XomRq4nd+h5f8aAt7rA8IavWAbng1Cf5gHv+h/hFg0iEYJrkOLayi4RInokDg1bBVZYIswzeBuU76Afk2XJWZc56wck423ZlsuAyJZusLrsZr1m9PvnQWBoZCGy0sIGZt39s8PSQQfC+bRaZ2vKxYJTjjhAMbSaxw8f8+W3pbAuiZ0h3L0DW9vOecJvCrBKIgUAfFnYzsu0wr/tedC72ofzIPgqkXtnDJTTZpuypTEliVmg1i30LsTnv/+Cp7MT8kdDbARWltR1xenRjOVJ+a8FbdeM9drVBEiSDZJki2ll2J88pzyeMJ9MmOw9Zv/wmAOuD3oz3Box1OGGUxhcHVIFpmmYTSZ8fYXPD3jtukPBew98Q7eRDf8Y4tD+kJXN2YxVofKr/HtfFv0r/M1NnPF1imOk33RrysvCWq9bXJM62DOhn6+N7dwrKo5JlCbRKbqqzrkbfZv4IN3gzp175NmQqmx48mSPp9UTrqo56OGOdz/tk+iUWSH0sCxZETwNkBcwm1UUs5KmbqETRMWI2GvJHcIxCGxvhBANNZ1YqvmMwgpV0aAkoZ7OKaKYZunszY7295ju73H69RcsfaPHyQFkB8KQ+2x+9kNQjjrK+zmj0RjskESlvkpdUCiSJEanOWnictF1VTOdTplNfbLMOlCiVMpkNqOoCtqqoDjcoy7/EFfa24wYspzRoM9wkJNlmjxTaB0jEWgdk2lNniuU+FasncFKDRKjY+uu+dbSIsSqRiIhjmMyP9Nr5XSkUQwSW6jBdi22ntPWC4gUfTUg29RsDnb5wb1dtsaauD2GCZTzDqs1sc5hMKIdjRBH8mK1EJER2wFxNiIbjIjzAYimi4JY0hIRIfH1htdsXW/rZzHrv0I8aHRGLkLPesDr742l3w5EHCksgooVkVrPDMUICmt7WCMsO+iJYLR9K0urfl+jdUzdGky7dBZllhfRmN/0c/u+/nyN0FoDHSY2tO3aR65LRqxTWVgvUQpybYkNUYzLJiCYTrEop0xagdJij2oePnzIdVWpKR70bimG/Qik9Qu/82B3HazGAhfXWeuHdB0sX4zrgF6AYepSDlES0aGwhaWWAmVa5BUTazeBk8kxJ/z/7L15kCRJdt73Cw+P8Iw8qzq7+pqenp7G7O4cAIjFQUo0QIIoUaAkUiYSMpGSEQQpGmniH6AOQkZBBCFIBkk0EwSjboK6KJGEGSWToIMgQdBEAwlIAghAi11wsbO7M9M9PX1VV9edlZkR4eGhP557ZlZ19VVHd1VNfmbZWR2ZcXlGeHz+/HvfO0Lft08x9voQzxJfnSrStEuqFtjauMOHH3yNe5/c5L4tjiTCu+hf5xGZ4loBG37f4/BZF5rNNq4oWdtcOVAgrunfDzuZduKJ7ywaSOTXIA2ww1QDDEJ8YyQqnPvPxjx9hiwkTKc8P/nVwDVEWpEgJHyZ402GBt/pUe0ivrtzVqRnm5SoDFNd7E962/4rOXLuey9EA9y48RavX7tBr9NmPLa0O4u0bjf5+ur7Bz4PDb78sprcnOH4wjE8BJZWVni4ssrC+UuUFoyREzoK+eTkOMhQzZQdO2ZjOGRUgnMKk7QZDraIlWJkh4yLMesr99i8e3NCegNGOEZ8SPFVJeWgmoZ2d4EiH0sdeTSN5kwmttbEvoyxGo8Zj8esPFxhdfURxbigKHJSbWh1F8jHBVsb2wy3Nxht3mWvE8HpgyE1RshtZsiaBpPEpIkiTmJMojGJJtNGigsoh7M+UysOfh4JDl+OyrMSpZTYT6FIjCJJYrQBFVckpo1zORUl2B2UjjFthTIZiWmjkrYwnSqHcY5zOaXJUF2ompkEQ00MTQ3GoFVCghZ9b7ON0l4AGiKtzssjVHIora9J/fUOMA3KekResC4uJLJrNSlRS/Au88mwcaRIU0OSBOIb7CI0rpKKaKFHMcYePfFtgkk0WkPup3UU9SQ3TymIKojwbjCzzRY6uSNIyjBGVI8qjoEcH8wVOJEF2NJL+MNn/lhCnonypX8rV5C7EYNSkW+M2Bg94vbtwz8HLp+HCxeh084wiZ+x8MmNgaQ+SdPrZj6blUDsjeyq+PAShwATl6AiP4NVkaohRu2QKK/W0RxJYuIcLwY380KJK9FwMGTlzj2+fvOrzywr/DxoIQHJBSS5ry2yYQbFpAeaVIprppBiGW9v8OjRwXoYxeNujQfBqSG+s3re8ArR3tq/9xAXiPeQG27br/cxogneDyFn5NzMstCoFUIK93Zk1i8PiY2l3++zSPZRQs342TjcpEQxOGyZU+Q54+GIwfZ4X2vOS42Ua9dv0L/Qp93roVTKJ3fusrK6yub6Jpubm7z79ru8+63fzPVr11ns9clzy8VrK1y6+hqjX7TcWf/ghY87lH7c2dmhyC1jmUjcF++vbJJ98DG9/gWuXh+TmIwjCfsw1T0pupQKNja3+eThGqVTtNrn0L0Gg+EO1pWMx2PyYszG7U+oebLubIOve/fwBpvrfba2LrJ5fpP+YEizvTCRNySNBo1mE20apI2E8XjE8vIDVh/eZ2tzwGBrm2azw8XLV7FFwc72Flt37lLy8ZGc+6uFJss0xsToWPlqfVJ6O1EKo2KMikl8YmbioKxiKicJZcRSmLsEVCmOC2VlsWVF6ZxIHOK2d2bQmGaCyZq0XBdXjiDf8o4MGpQRiUJiiEtJplKVAquJjSFut4mzLqrZhWYXTAbaEMcGpRKU8QmXocqBF+UqVZEkRnTBh7heTTwTwlMKpx1VkG/46o0iX9AoIpQDF4XESSACrSIhvlqTJglJmvpIq8U55XM0p4PlWIlM5FBlrPZDCra2lFZkFkSgiVExklxnAaewuqYklgp0MVNzjPpwMz0BSZKIVKyaal/xm7ZWXBNsLu4OMTPJb2G8UE5dI2NVgSuxtmDz0SM2Phyxtv3MYmHPRKsLzSxcOU5syPy5T6K62kufZxRubqaJJpXoZv4fktpiNUOA4dAGJM6tg4PCblBVDleto+Mc04RWB+IWVMftDz3HY7OZO/ikNsQhaHX5Y4qi4IOvfnlf0vui1+0i8DpyPxSIjeyDsTzfN9ltCa6B5hpoPWZY8FTSHY5/v2MZMyXUh8GpIb4DpoTXMa2bHQpV1AgR7QD/EJIAt4ZEYpv+u/u5PYQ+LbhFzEoCguZ0mcd/qDFTf3mHhPgXkdrfx64X9v69gfzaEAH2vXhuLYUtGI2GbO/zsEiAq1eu8O5738Rb77zN69fforPY54Nbt7l77z63b93ik1u3ePedz/Hu57+Z69eu0V9cwjnH1YfrXL52n51yzODvbLNRvGh1Nz9osGOULZ7ql18BH9y+zevX32Q0ymm2Q8d/ePIb4yvGdLoUCta3Nti6+zGQwuWUdrfHznCL0RCKfExR5GyuP2n4tBdjau6ytb7F1mCb8djSXRig05QkSdGNJmkzI00bqDRlOBywfP8uKw/ukj9chfIRQ85hlULZivHGBsOd5933CYdKMA2DMZpEK09yZXYhiRVGKUwsLxHy+OwtJ3YFiRI9e+IqnHdZcLakcqXo35WBRBEnGUkzw7SaJElFEjsUJaockZc5eV5iS4eKE5TSxCZHVxZnE1yZo0yGyrqo9gJkXVTWxWfaoZKERCUSPVUJQsWlBrCKIUH0vnKsB2cVRk9tGZyCWjlszGS2J1IyGNB4/a5W1FaqOCol/5dy3xFxLBKbJJHqglXks+1n7U0AHfk8T6kYf2TQGmpXYauS2rufxFoR+8dm7WppWwdKpTjtq1H66grWWsTS4HDsVym1y892lhA6J6Q3HwkBTuKpjZmqptGzEKH2Ga2UVc7KyoivfVmeOYbDVfFqtSELaesh0uumfud4Mr6fOmTWsSHol8O5zmp896mDcXBU6zjnKAvLqKxwVop/NI0Q36x5/LOhcwhiJPo6RIjnAOE2QwfFwzUePVzj5j7rXUI4zCOE6zwrV2oRMRr4hqswGsCtDVlv5Qnf3wR6DrQnvU8qMNYCLmuwTrHq3GPOEoEDHhanhvgCfBmRF1xm2gBvAh8gRPcNxBj7G1Fc7TTY3h6yhnRUQ6Qz2ms5MkLcI15HLhiDEKIgq4BpLe9bM+t93W/zMkK2Axk/zhkdScCQniz0WaEcrnXOPyOk3G0+sozH5b4dcAn8xke3WF3fZnlllW/ZKfjse9/IW9c/w9tvfxPrm6usr6/T6bXoL3ZpZRlOKTbXt7lz9zYffOV9lu9+xM4BSC9MC1E9q+ddAL79276Da9evY0wm07jPsd6zECODHE2D1e0N7n7h11kb+Hh9skDn/BKd/kVGg22G22ts3brJYPsmLy4z2IZym0effJ1HE96qmObbamI0DkvNCLmCwuQQbNy7D8MBVMs8uUs5ZXCrbK232Wg16TabJLElSUrKTPSfVUNBbrAmQccKoxJcPsDlIxzl5PfPx0Py0YhhmZNXDlSMSQzaGBIt78o0iRstTJZgTOIrdlWYSvyQpfKhQhGjnDdw9YPH2BhUawHd7hK3l4D2JKO+LEVCgXOopPTkrSLEAkskkp27EuUcU3uGF2wq/y4FHqQkLM7iA6aTz2IV0iZj0GpGtIDPea2pnaWoHHZc4mpL5UqKAqrZEsd4Hf1hRZ9NA8Pdj846kiRObIxDCvHIwMaiakeMI1KSAhrhS9o6R1UU1PUM6zwkyrz0QYJqGuVw0qy29OWKPcnUKiRXesyUgFZ+pkKWVYxGcoduc/hngElFJ5loh4odMdPfNDjaTfx4w6nEoDzhnDg7zMYHqpnAbtC+xX5ZJa5xB0VZjMSSsYRizOTeKkq5vkbPKqM6x5EhDMwaSAAuRZ4qa0xvoR5yjXaY1j5ItX/yWJFA3mf/3MJArDt++x/fEU51H+FDoX7OIjKL/jHTyO9dnh2ycsCKhc0jG5Xtj1NFfMOo8SIyYtgGrjCNuF5BbM3efOd13mgllPdW2L63ySPE4uw+T/bay5EfpeNfIZkuUBTHbuIL8kMGp4mSiSX8MUJNJQ540uu8P2zpRDerZfrSFiW2tE8cueXAzfVVyi9+kX5/ictX3+Dt9z7P1es3yMsxY5dTFGPyfIQtCoqdnJ3RkJWHy3xy+xarD+481sGfQ26aIXLTHGZ2Kwbefvtd3nnvXZYuXiH1dmOHrYgULFBTQNNigwHD9RUmyvFGRrtzjmb3HHmRM7aWwfZ9jk5bG4ZIMiR5nGI4pEvZgu0VJEXgtOt6Z+EYb20x2NpiZ7CFMZAZBc6Ie4eTKGqVK5qJVEar8hGuHFJV1j/0S/IyZ5TnFNZSOu/ioJOJxCBWCUobtG6SmIysnaFMTJxoKsSDVQwPlBRkw6GqEhLvBpEYyNrEpk2FHJsrSyprUaXDGkBVJL5MeOznjJ2noKVzqBIqdXDiO4GXLeCdEML0vERzQUVqEvVFOUmwpJ4ZHQPUVK4UVzBXUFUltlDYynv7+rYwSLLpE59QMTR6BmMyNu9vPP7xQkK312H945mex7P00lXSxpFPUhVrDhR1qLYuIVZrKeoKZR22rqb1bY/gWShWkO6xfsRVkOdCgKtymiydAFUItXqJgAJUAkkVHL4deSl3an4Eh5kYXykuwbNXr2MPjRDYTWjCMHBRouBJpiq4XclwanY1T+5VwkRKcdCutbB+QGjFia6qZJm14AqozkK41/DillGvAIF/aKY5PBXyPB4jt9LrwNtAqwnty02pyrY9ZLQFDTtdbwV5lufILdxFIrJNpsR6C+FiYWwTI6T3LeAz7zThK0NCNtCTco1m4Xg5fsiniviC2JtdQyK8V5AfZ43dUY7h5jbDoaLaGPrcZSGzzf03CYgMIkF+6PN+O+EH7vhlmsc1vF9hUor82PX7Wu8mvoDv8Bwucd4wS2N0SppmpCZ95v16Z3OVL/zar+FQjHPHtVZ33mIAACAASURBVOu30M6hnKPZSuk0DK1Wm8XFRbSDjZVV1jdXWV2/z/2Vu5PBSBfoNyDNNMW6fWG5xzkNS0tL9HqLaKXp9RZ54xtu8NrVq3TaPfHGPYKstmAHKrRzzGgyrm1A5zXOXbqCSVOKjTWGn9xm58P34UDGK4dF7o/yDHr6uFUe3F7FjrYYXLkIXCTPfaniREmym4K2abHQbuPKHFfmlDYnz3NPekvy0gpZQaNME9POMCbz5Ff0wPKQd9jSonAUpZA/Si+diDWxErKsdCYyCRVDYqhiTeW0pxuedFQisaj8PeJciRTJiFFK7rRqhjUKZXlaz/NklM6ikCITKIX1pa8FwmocMRF+xie4OjjnSY6nZZ7Y28rhXEXtLNaV2DKi8jVWlINaKayOqAqe+IRqnlO0OxkmNWzef/zzipKdYs+Qtw5RSiflv30EN0ZPeu7JzLv3us6LgqL0ZYoDmzyKIFAlladdNXUBwu/fKKgk41XkNyokswEaysA7PffMK4sqHIxgY0umiY9CHaIMVApGpdz7u9wYlBBzG3R4bkpokxDSnW0n/x036//r/7YKlLdMLu3Bm3dWW1zja45Yv69wcMdhDvsycQpILwg/D8GdHvKsGyKXQYgEgwQRN4ZQfDicyDbD+NIiQcXZJ0/NboKbMS2OEZ71ERL8ugS8+Tq8/dlvwPIhF+4Oub0lT9FncSTHy6mie+qI79cRycPv653jnc9cYZuK7api+eP73FsTRchwZYONJCYZytRogoxinlWN8y5CNSrkAllgGtY/59/3an0tL48WKa3QWgopBDgN2ie3KSV6P60MmWmQphkNnj2C+vLtD9je3mFze8Tr167RQ9FD8dqlS2SvXaZ57RpL1xdptzpsbm+xPt7k/vJt2reaqPWh3GQNWOz3ULFifXP9heq+Lyh467Nv8e57v4U3rr1BK+vQaGYY06DRyPyU3eyj/HCYun3szNxkDS5dvcbixddopAnj1XUGN2+SPyWZ7XixOzJ8FvFo5SZlOSBrOfI8I2spmkmMURUmBpt1SdwiVBXKWUajIVuDDQbDETt5Re5idNIiVk3iJMM0F8iMEf/cRPsp4gpXVZSluECUlBLttbHIgkwTbRIUhjhpQqJRRkp0VcSeb4l/8GwmlKtiXOULaGBB5bgq2AqqSZDXHuKBb62wGmUlkueco3a1BFDVNAQ5sb8NTw3nO/apkbHon22FrQqcq/3//fH5iJ2ioo6VlNt9wv27sJjRbhl08oTe1EFR7jNYq6GsQTs/iFD+y7PVJZ2Q86oo5fcKiRxBjnEEmcPBDVHi8m5yGIlP9tBOiK4IkaY5fs5JouXIywomucRDhxvB9vYRSqITIaU7udfoziSixZWPpHryGiLliZrIob3ncDhwcDmTIiDBw7i002TGEtneQS9VB9Qz1a+d81Hg8HvNXR1eChKChE/4zgJCcEO175Sp3GULCfatcDBOH4a2YTYcv+8ecKUB1268xmc++xat7gJv3djgi7/yG7iHHNrq76hw6ogvwC8A72xucG7jHElT0WxAQ01/2Kr0frdM77eE6WjnaVhlWghjjIx8OkzJ84tYgETPuc/nxq7pqFARLMy9TUmhUhIdNln63Pv/ZP0+vdsf46yjnVs6Y0t+cRn3cJXR5jbjLGWkFasrD9je3GRne0gxspORZGl9RTMcgxcgvRlw+XKf169d58aNG7xx/S1aWROdGnBT67ZZC7fqCDx4gh4pIDJtmp0eWmt21lZYe/8rbLgvHXo/h8Px6pxOAjY3VvjSF7e4eGGB/rk2rWZCUpUoSrrNNhvtrrePqhiNdhgMBwzyEQ4DSUa7q6QaWJKgtffexScxEVM5KCcjRUfu8kkETJFA6YhLCBZpGs8ekrCGlDBw/hgmoUdnqWzFpMCBKpkOrf30dHy4Ydq4lN4rUkwid7XzyVYR1N6STIP39VVEftrbASghdkKKK1xVYIspvVH1bkWDkECJZO976WVgTExqRJ7w5APfZ5kNx+Em0d06siFATejcgnSrniHxE9J7BKEg56vxOWdFG+u1vTAlggqfzDajLHBIxDfPCXmWqArKDSg24O7jqo9n4olZ9Er25QrRGQexgw7jGK9FxssV8ElsE4mD34xGCLqCibSHUqKxLpxHOJfHSt4/P6p6GvUNThO2mkrHoxTquavDsSJCJAZSEVXe200YD6e3Twchw6GgxSaHD2TPjmeG/lU4SNOUhQXZW7fZZePBKo8e3mP5KfsM9XmehqMymzmVxHcA/J84Nj74gNdRXL7SYXttWwofMK3sETQlsyOS57Hs2PKvO0jy2ueRkcyLktiaw+lc9yJoegV7dGqTwg6+m1aQpopWW/E8TLQGxuMRo50ho5V1Vh9tsn3vASu3brN46yP668sUmeb2vft8cu8On3z9I9bHxeQccwvFikTcX0Sj0+9prl27zpVLl+n2+mRGSK+ORTjp/PSssw6LDeGaI8HsdZA2WpA2GA2HPLr5VTZ2/v7R7ORQOKITPeEodnI+ubnM6sNl+osZ5CNcDpmGtkmorISjcisesBbIutDu9kmSLmoRcWaIk4k3dOzDX8458WWtKhyOyuZyTZUVysWQl7gkx7gKk5WUzpC4DOUSnE4mJqhCIH2015U4SiqXkFRWSiMrDXHIFIhlKkbF0xDLATDYJ3Aa4ROYYoiVk9ke5UhVSqymvZvIBipPLKUqmrPemDYSEuUiX7JY12DlHKPJ6PnxfesEklTuR+ue0rPtF90LxNezWeUcDi37s4gbRVVTO4ebtS5TyFMqjDsOiaqSQdBEk5rLNTVb4jcQ4ipYCFlwKeQOdkY+gOkj8dtrsLXMvpnyT0M7g4uX4ZNb3l3DI/ZP5MJnb+vYl1D2OuNJkMF6kq683tiTXzlJf33484yRKPUkIhtszUAmJ9Q0Ie4gsLMaX58oN+sgkSSnozLqaUWMSD8vAG0F3e40+u+G8nwuEDJ87Ty4MdgxjI/Ye7VG8qi6BWwNpH9oaA2NBhfOLfLmlQfcuecey5UC6CPHGyZ5noQ2ojM+LHE9lcQXRPIwBr4Rx3v3RHkS+o9w/wev36Bhccio53mJ2cDv5yJTve+rxNScPsDTexVmPv2HyqGUQ6uYRpaSDfb38n1s+5Vs35UOOy5w4xGD9VU+WV0m3X7I2MSsr6+zurbO/bu3dhHHMZNS8s+NGLi0dJHXr13n4sUr9DqLpI1MSO9MggYloEFZhWXmoXhkyFA6xeIoNjfYWD0pNeLTZ3/lDGG4A8OdvVfq/ldUYwQX3RadxRyIxe1EpkCkapqvcFZWEq2VKeMS50qq3HriixBfZQCFqyzateT7lUGZEkwiSXcAlWh6XZVTuZy4HOGMQbmcWGmIJeJbERNrg4o12EMw3/AEmHlA1QoKLQlXsRYuaOIKpy2xVlPa62ZJngNXT6aegzZP7OB8OncqeuJUaQrl5/v3QNzVamw1Jh/vEz4ISWBPYDkuhASpcURYPRNOrZkm6h7WVeIpCGSsLCV6a0soC2nL1DA5nsr7+To7LWIxymEw8NO7CbgRrK3BygHMbT5zA3p9Wb+YiRYnib9uPZms9AyJkcPA5vIi8e4PanLpB/XIBBOVWDh//8/s13QiZZEPmkJRWJmJKLyEoiinzg6VkzHhHMeHi4gUcyEV0tvrybUzyqe3YwE0m3DxTXHeKAZgX3S09hyokSTP8bjAWoctLK4oyNKUC4vnOH/v0b7E92JTMx4+OzcoQzTEjUPGhE71JfmJf30JcXO4ijdrRiQKjxAP3l8D/l9EX3KQjMEvMQ04vMarSXUCmT5IU421Eu2Vjio89ENFNLDFEGsLrBd3JTxf5Ll/rs/1G9e5+G3fweXFJTZXl1m+d4+7n9zil3/tV1ixw8cqvCXA2/3LqFQKMayvr/LwOZhpG/jM9T7vvvcen/3c2/T7l+l0uqhYy7RyFZ6IfprWOUl4yfNDOzs8jorR+jobH39Mfu8uLyZoOT5E/TeoV8eclOM5SRgDG+slS1dKn2hUMcxFFFkm4qpLKVHMOFFoEhLEHq1SDktJVYpBmqOkiiGpchKbo8tcfKySBIwmRiLJWAelI88H5PkWqBKTGZKGmcgs5MpU6KRNkhyCTQRM5tlnliVQpT573kGhwCYlacL05nQS4Y0RH19NTJYmIuVwiEOAU0SeVSmVSCGRJKME7ifL2D2dxriAte0dyhEM9hpsgig9gkRhL7QvGOMT6URuYrGoiaRgslogcLOviT7iRRtwN0beriz3fr2hSptGfioXfG+ZSouD/nWUw6iAohKybIewtgWPnpB/2khhoQ8P9iQBNoDegrct2/MEdsixpSXEXi0T/HhdKcc78sduZif8Yrw7ycSpTP72JLSc9SE2kriXGCG9WSa1WQ56pQ62pU2tlWMvComiD8fe1e7TMXH1yjBJvC/kGm40vMa6nMpPasCm4BrQbUPzCjRvHk/q9HkAlfLowRofv/817t3+BJuPKQo3mZGf7c6uAufOXeDO8N4T/X0DwgSQPmRM6FQT34C7yM3eRAjvGjLCuY/4yH0VSYg7qOxgC/gICbGfQwThr2LqJkmkUpSaJLa4qbZXia9k6MALW3jyK73h80g8ms2Mxf4iN955l3ff+ybu3LlN/JXfZKXYZvk3hvvG3pZQvP1NvwWHY/nBA0Z5AcNnGze+djHlxls3eP3aNS5evEyns4g2LZ9yUnhyW81EuR22dFhrfRngI4YbMHq0TFmdFPl9yuLCAqPVNqM58d0XmxZ2tnYY5iUmt+hEiG8TMzUoreQhHwaJKgZxSACcJL45B5WKPQX28l6noUwgVyhtREZROihhNBowGGzgyNFGkRiDaWYY08SzSIwpcUnT+9UeEpNI6R54MmiRvq2elHFmShIjKQISaUWqEkys/SheBs4RoocWK8SYRGcoC2maYPfe8Q7GIxhs8ngEIWSD7TdPaSSiGjuIbST6XU90K+V2Vxx7GkkK+rVDIFRny3PvM+tk6j8UqAja3gnnrkTW4Eohc0UpxL1yQjSG+f7PlWYKn/tGOH8Zzt2ClfsS3a2AcwtCNpNEyO0sqsp7CdvHPwtygjKX45dcDjkn5aZtN7lcfHQ7uEA4H+aNATzpTUQqf6iI73Dk92NhPJb2LTwJzsd+v3vZzhxHhlB4a+L3omfGjmpG4qLklTYbnD/XJeXhkRPfHnCuKwPp4daQO7c+4cObkgCf+OO6iHCzPtBtt1lot0mT1nPlrmrkmt87YHxRnGrie4GpTVkHuaHvA3/P/72CRH1/ncNrbR8g4vFZD7uXDuUfUjPliiWvTd71JMFNkY9yRqMRxcg9V3+TALbYYTTcJC+2KVxO1spYunqZG8N3+LbbX2P5zi0qV4CzmKxJ2mhx4623+fxv/w421zZBwfr6yjOJbxe4cf0G165eZ3Fxicy0UMr4TGTpod0ksc1OiK+cqybVB+yhn4ge4ChHDzlaVfbhYIdb2KMsn3UG8f7Ne5juOV67UrLEJVAxMYoqUSQqJokhjjVJrCYPg6n2302S1xJKykqRuJyyAvIEV1kcCp04YrzvVwmDrR22trbYGqySlyOgwjSbZFmLxBiSxJBlbYxpo2IFfM/BTi4kH+2NdEYzy30SgwUGVvq98LHIGfymnCJOFIlOUCi0U0RBUaQQ5hT5vkWFltkDb/W1L7m1CLHZr2NMIIlA1RJtjvxB1XiSWTscNfWz0luOgPiW5ZQIgI+oinX0RCMbbCtDZBqmFUMnJaP98SRGAiJ78/myNmgzlRGYTAZgQ+erwfn10z2RqzBFbYbTQLdysl9XePmFm37Xznj47ocg7QiGIzHI7wHeJ3hm1QOqcvKhJ+WVEN+8kAHCaAwjX9Ti6OVpcwRoZJbbIoMx5x01XOFjY3gbzwEMN2AhdZhG9kyXqxfFBeBqCuf652ikbZE6lAUF3l3Cyng5kO3wPC8Kx8DtPJWExwhRXuBopDOnkvhmwHcjVdtA7qkNJNJ7B4nOBrnDfsGJg2IdCcs/j0XYcUDN/DGhuJ4Iq1hJ0MWn546LMePxmNH4+Ui6AWy+w3hnnSIfUrqCtNNgSV8B5dhZ/zZWr11CJoctptNBtzq8ceMtvvk7fhv3b91mc7DO3dsfYZafLFA/B7x+rcMb19/i9Ws3WOwvkaYNtNJQqWmE14GzlrycjgMdoHWy/0P5EIjNAq7coHYnqzpaOdzCzYnvU1EDN2/eRimDydokSUYcazISEhOjlEaRoJSveTWjkw+k11H5WK+irBSxBZyjLCWsVgXTVB/xHWwN2Fpf5/7yPVZXl8nLnKyd0czaZO2MrN2mnXXJTIbaG7Z7EQTN7JN66UDMAvkpp8Qs8hwyCRpRpdBKkyYpCTFGpZPlkjZahrQzmUzaZ34o8sHifW/uGh7TQXk0Gv44nByU8hl0VR2kTDWln70i8ntWe17hfA+J4OAQrgEVA/FMOzE143Cllz8gEdPQE1VqGmFNDGQtHpszzroSSY0T0Q5nmbyGO744hX8ZI+S38J107bxUoJDjkmDAVPdb5nJc4KPDM785btfbBM4TY4VEkZUSQh6S4wKJPyhyT8bHXuIwHsqyUSHLzlSkN2JaAck7FwLTAeF+EqBjhkKuzQJfOMQPiIKmf0J8CxhtAAuOZto8oLv4/vitKZy/cI4sy1jotmmkbRmoFRaL8LNHyG2yi4+olLF1FOPhvk+6GHhTpzS0khcWfQS+hqeO+KZIVZB3gM8ytdDYRqK99xACfFwmyB/w6mKCDuksNXpXR6ViJd6lSp4SSqdCJr0Q5lnSuAxYaqcoCnY211lducP9233SRkO0f4nj0mtX6PRaIrNQ4Px8Q6vVZLC9zs5wA1cOaaSOPvI77EUXuHy5ybXX32DpwiV6vUWytIUi9gk4EuG1yHs5M23rJu97E/wOC03/whLba2NGJ6pWRMFo8yZ7r7ajsnM5S9jc2uLmzQ/ptrs0TYZJFFrFaFeRICUSSlcRVxZbllSlxVYlVSVPZBUYAcgMwxhcbKmkhJt3ayhxY9HNjvIReW5ZfbjG1z/8kJWVrUnwtbug6Pa6XLl8hXOLSxhz8LhKSOAIU5aVE2I0wd4IaIj8RUJqUuXtsKLgSaum0gYlVR6VinCIfZi1BaWDkS0oisdvsuSAlldJpEhVRBwrT7Tl4F0lyWyVC3IHIb1ViB6HQ5gNYx+SRIVAeXBDUOHvWRLjicMol2n8AsiVd65xQnoniWEG2l12Ed+IqWygrKbJXkpJ0CRJp/IK04ROF1Yfyfcbrd2SA++YJ82R++3sOSc30z9WQarhhBQHBD0vvrRxGGfgib8qDi51CJj4CE8OTPx9JwO004wY+fEMoezn9GTDtdnglRDfMO51iFuDHcsAr/DTEA2mCa2UMjPcbTc514WPDhnBS5AA5Hvf8DpKaza2dlhbXoGNVYqHmrsPZbZ8mceb5gGwsvFwYq+233j6HNBuLoArcNYyKIpd0qiD4tQR3w4SdX0L+Byhcpv0/3c4/jSgVzoR7ofuOtk9XccM6ZUoj6bT6tBIM2bzXfbDAnBxEfr9RbQrGGw+ZPXeLW6nCZ3eIp1eG6UU/Qt9Fs4vkqQarRWFtYwKcUW/d/cWqw/vUozXMYllqQd2c1p0CfxURRsuX73Ca9eu019aotXponXDn5qXNVipKjX1TlWe7IbXEdO+5DL9Cxexdv2EEV+YfZqGijwa6UBeRnWb04T1jQ1WVlZY6C+StZsYU2GdlA6WeV6wVQn5SCzNrIXYVztT8YT8OFdRugpnY1Aap5zYWqFxuaXMS/Jhji0rNtbW+XB5a9dvMVxzPFjb4PbNDd58c4FW6+BxFQOTiKdz0sfN8tFKicvDrqhTJFPpqfIld5WaEl5muZ2SpbEI/6yz5LYkryp28vG+muJUP2U2fNbJfg+0iom19BvBdtH5+7uiCrUVqKZ2y9On+azc4yik/QkT8xuYsVqeIW2VJ43DEWzveI1vAnklEU3tSWOClzG0du/CaNHvVoFA+0goMWSpEO4wqDeZZOLnuSSJddpev+ijsTixUcMn4blc9hk0jrPPgaqSaHEZ9Mg+tKaUBDcyBfFMwtNeBn1Q4hskM1qB9VZ5wd9dDvJg2z12hIt59rz1nv/7se+E7CbT78QyQYFWcm8U41dTbsjNvrxFX1lI5B2mblSp/3JDp3TbDc4vcqCp6xbwBiL7bAKvXW3y9vWrPFrbYvXuMg/XhuwgvOwWYkDwpBngIDvb7/MO0KKBVglFYSnGBQOkjQ/7/Dt1xHcVadANhAAkiIddk09TJCz2ddgr/7ciVlqS27QiNYa00SBJ9BOjo8Hwut+Afq9Dr5XhSOWCcA5nc+llXYrWDdJOG60TTJqiEs362jqjnXWGwy1GO48Yb67T0DkXFxukZY+e3mRzW6a6LFBq6J/vceXSZa5cuUpvsU+zKY58wdje2VIiQP4Jp2JFrJREhCrnfYwrjjLk2+4u0syajIYn22nylFTMfKV4//1/ICb+VQUXluh222iTTBiDchbyoVzbpS9nlYjfA07hSi+1qUociioxOJWgYoNTjtJW2LxklOcMdrZYfrj8xA54DHzl5gbSUx0MSbjMPSMpmRZ/CBE+C9KLR/Le0JAlCqNijNKkSAlmE2vSNCHVMc468nLIYFRhcZR2RFHtULoShyH3ko69MIkiacRsNsvHypQ1OjDeYd9OONYaHUt0GTUzkKUKeXZCeieOFEzJ0lP0qweCJ36V85eFXzybL1xa2NkR0ruxJaTXaSGgo9wXkki8ZEBB0hSXmlC+vZWJdAIk4hu85E1r6qDgEPJrDLS6QlaTREiwMdMXDlTJpNiE0iKNSLzvrlLTCG8Y49kSRiM5B9RUyxsivs7bjoXmLX3C30HrraQ+EBMh7VSlQvitk8HbMDTyq+zEQtQgSBRmo7Z+wLgvFJNwqVbij6yRvyNkEKS9pKiKXo3jk0JmUwHaelpGWCtoOCGQ4JePvQyy2KLxgv6sEVLX4Abw2qWUZqJpaE2z3aYYbLHxUEjvHabN+tEBzylCMm+ajQY6Tb2Vt6LtrL9nD8cBTh3xBZEbfAm5fi/T4EpynoXyDj1ejfb2ZWM26qkU3rJISK9ONEprH7168sxgmPoIQRhQpGmDRqdHr7dIf7FPq9MSyYQvT5ooKfSpKijGY7bX1xlurzMerqLdmF6m6fR7NBjTJCdrjNkezBhoN5o0mz3anR6ZaZKaVOyEXNBeysUc+/BLmJLVMPlMHphHhRZaaZy1jDY/DVfO2UYFfPi1r5K1myQmIcliMjKcE2V6UuaoQHrdCBHF+hBgpaBSVNbiqpIKBS6W+f2Yiaa+AvI8Z2uww9bO8V4zwUo3RCgnJNsTwzDl7oA4hUYCjTQhSwwNpcmUJlYRCkWiFGmSkOiU3BaMRiO2RwO2hqOJ1RUxZM0ca+N9nytJGtPKMs5fUGxt5BTbTLyzFrptHhUD8Zfdex7aMwsVDt1SUU2T24IWa78p8aOK9IbNOR+JnSV/lZBDG4hvDlsDee2MfCGLRKLto1yie0k8Q0aVRFNDnaDMQDpbQEKJJCITdzzStl/Xk+3ESDIcsXd7MEKkjPHBbi3ku/J+5sZMiS/4Km+z7R5PpRooIWe5kahvHArBDPcZXxyQ+CapSHAU8u58wY8aOU+rxHIv6NBfKlrsjtjODBJR4jaSeFmQisDW3gFjZhMq9qRXeVmvmvLhEPFtpK8usJ0iUd2GkuuoEUv7N5oie5gM8BQMC9gYWh5uLE+KmzxvwPA14AqwtADnOk0hvalmWIz56OaHfHJnzAMkOBk0xweFAhpKkaYarTWNVNNot+n1FuguLJAe0tbhVBLfVcSbtwS+GehUTSo+HXb/Mv0/hQrTtEqhEo3W4jlqS4mnPKkvM0if0DLiD9xIG7R6i7QXz9Nf7LO4uIjJMj89KU8sSZDxW7RS5GJnsMXGw/ukWHqZQlOiAGMSGllBhUOjsHFKp9ej0+mQZW0aDUOiDaGIvHW7O17RIMq5OO/Dk8QOp920oMCh0cBZy3BnE57pIPiyoSDpQ7nJvO7R82Pblty++aGU7E4UrczgqFBYSpuTDHOwQ1wlxFeZBOcMSjkUDmsdZVniVIKEqBToCtC+wItlNBqysb7K6nFHsLwuVCkgiiY+rQA4/xyPhPQaE9MwDbLUkGmxLTM6lvvf+Wp2Wsv9VFXk4xGD7RFr61DW8nCPG0IE6v0IKJDEmswY6KU00hbjnqUcVyRpRNZIaTSkwMMuGDlYR9CiWkpbUrppha/J/mZf/hzFgPSoGlQkBeCrn+VCep0DlYP18oRiBKMdKHLvAuHXLZ1EcLVC2IInTSaGdgtWtiVK1TZCcLVPmNPB1cFIf2sMk/KiaSVtVGVCQozX4mofUU5C6MxPsbtgQ+a7wAoglB8OUd9SzqucGRhNrqP48bGEQ75/UPzKL4OrfdGK8URZdDLwDPlamGofw9M9P/eLCNcznwUrkFcAIYkik1GegBcWBoVUZwvXr0UGb4OxaGtDNbfnefJlTB2tlAJXFBTWYgtYWx9wc22q5T0KxWAFvphNgaKgbbr0212uX7vGxStXaDSyQ23/VBJfkIpqa8CQMbg17nP26UEFE60rsNvWzCf0JFqDc4yHQ1zpyFIwxeOzTG18pZdOymKnQ6fXo3/pCov9iyxduMRibxGl1fT550f0Kla4SgTyJk2xo5I7t+7gijGdFFIVZBKFeKUmmlajh8p69PsX6bQWaTQyUi3FOESm4btiVXoVQ9Bdzli34VBaYVRKdWRSB8VosM3yg3ucvKtnkXPnL7P2CChPltvEScfdlVUUJSZ2tJoxCSWKkri06HIIZY5zI5wq0XmCShKSJEeptpi+OyZz3wqDUjnEUOYlNi8ZDDZYXr5z7DprFe45IIoVjhqHQ/vLP1FSBCEzGVnWIssMiY5Jldi3pcqHt1w9qYzmvA92UVbkIyj8U8p50itmnxrqx9lLZzy9/gAAIABJREFUmsYin7IOXEykFGlsUcrhsDRbMUVRUYRECAWNLlRRQelqXOGo6uk0uA3yhkB2g6437PoYIoQjf2xlKc4JoTyxUpAn8n/rK7pVTAmjS0R6ks8M0AOpzRI414eNbTjXEQLSyiRxTWWQ+X1mmRDiQG5jLz+IcyGLMbK9SelkgFhqqSiQQUQ1k9IRIpPOV6Hz76Pck14tMgSTicwi7DcQ03Buk+M5YJsODq7mOVl42g39rM9eoXPFZJyohNi6ElY24JaXGQaFR8OJVGa7gHtrQvYzno/4WnzRIECtQTkYMi5k2SOkXsJRp8hsA+lwC5tqWotdrp5f4DPXrvLG9TdoNA/nSXFqiS9I5PeXgIRHLDPVWJ1ZVNXk4cUMKZxmaQuRtNaxvbODtSVpqugV7rEa2B2g14HFdoter01vcZHLFy/Tv3SVxX6fXm8Rh8PaQoimv7tkaqQi1ZqGMVjruH9nlXxYkHmRf5aKhU+jk5I0Mpq9Pq3FJfoXrtDuLZKmTRJjULFCEwpywDTy68MhqGn9ebyVmauO9KItywEbKydN5pCStBY4f/4C4+GI4eac+L4oPlnZotu+Sb+boVRJgiNBpA5Sclj8d+MkEe26qaTohPMlipW4pyhy8AUxinxEXuZsbW3w4MHx/yZCfBXRRK7gUE6mjnFCfDtJi4VWm1arRdOYaV+gFEkc41xNXUufkeeOorBUZUVlC0l+8p1CrT3XryKc26cSRSJJs1prnAZqR6ykdLJzBa4uMVnKxSz48zrR9tuSwlUTt4HSl7J9LLIb3iumBPgYRhazEV9bCgEOFa+UH/Q4K5/HeElBiNBWkPiKasqrYIy3Kju3AGUH2n2xN8syIbEEGYMSCURm/LremEd7KUTlZLuhIlWIkIOPGO8Sd08m4YT8eiIbCkcUvlStTsRSLRxPKFLhvNFIMhM9VgdlvXOcDChPfH0y5aNi6qyUIc97hTyftwq4v+bLGJ9DIojPQCiSo/zfW4WstsnxGQpsINUJF8ZbtNPXuXxhgW+4dpG3rl+j3W4fatunmviCNPrf4JXNMrxUFIXFGCvliv2yeFK0AorxiHEFKyvLfPD1j/jo1k0+GbjH0msyplm4Wdak11lkqd+nf+EC/f4SrU6HhslwVSmm2M7O+OtKz9tqtbi4dJHxtR1W3/s86yv3GW6vU9kRaTOllRn6ly/TPb9Eb/ESvf5Feot9ur1FWi1Jg7YzGRbiLRmjnPICDW93VIHy+9Rao+KjE7TEaCqGnLzUsYJyNASnJpZ0c7w4bt5cQZVfwCRgtMxElMNSCI8TUrOwAAv9jH5fCxNwDleJzKEsvejTgqsUg8EWo50t7t/7mPWX0OEkqYg8daRROsKhIQlOFJDFCe1Gi17WIjOGRppQU8stGkUTmZBEVx3FuCD3NXuzRoNzvR1S7XWoRqJB5Qi2tx/3rokVOOuweUFlhSxWNhdbOFdgbQkalI4lyByDq8Wfd1x67afzgeRAbJ+GICc4YgR3OW38T1tClTP1ZPXH1am8FypMsvnzEnYyiaJmBroGFjJ54UvEpm1IWpLwpryO1GigOXV7mI3oBjcRY3xyWYgm79Mewdlj1/cUUoK4LdI1DKQtsVDLrfcJ9tFn7fcR+HOo3HZUwrE5Xg2ayO+bNmBzAGsDsXYNUEBbieZXN+T2WxuCbsPVNmw7iRA/DTUS3Q3mLQNkH8cZ6K6QKLItLCmObiOlqRXajlHF4a7aU0984dNBekF8Nq2VnluiOr4T9cP/8aggHxUsP3jIRx99wM2V7X2T/YIjS6yhkWZ0ez3655dYWlpicWmJ1BiRMVhfGc4hjgrIQxQc7WaTdrMJDna2C+7eXuTe7Y8YbK+Ttlo0Ok2Wrn6Wy1ev0b90if7SZRpZy3f6ctEWhZ3ohpVSmFRjnaJySiIv1Ux0G0ncS3xU+yhgjGaYh0mcEwY3EB3nJPr9qlInTi+GwG/e2aKJDPZKJPl1lk+9tgrX3IjEbGHabZ/uD5VT2CoUuBBN6tbGBjvrqzy4t/xSjj/1SaWRVuhYTRKpEh2TxpqmNnRMRrfRwkSahlIUzlKqkACqsVisq3BVTVWW5KMRKEVmDHEc0enILBJKUVp48KDArj1+rcUx1LaiKkpcCbUFW+WURY6tHaX3l411hfbkqnYwroUkT6K3lt1WZbMIsodjhPbE13jCWeZgzfSaiJVk7sfMaFW9I0DuRCqRqKlet5tBJ5NZq+D24BRgmLgpzEZ5TTL12g1toLQcT2K8K8Ns28yQ5ErNfKb8R7H055lfnhgh6LmP/Co9dYhIjF/PSaQ5JNGBXPZznE5kTWj4SoHFFqy53VFYBXQXmqh0iDIS6R0M4cIlWLoAtgmPfg3qZzwGx4hkIiSrH5b0hgDc0yQSW8gMTKocC82UpgbtCtQhlYlngvh+WqBDcorvNSfetnb6bm1Jo9XjxufexWQ3Wb53h3tjtyumuYXcGL0CrGqQtfv0+hfIOh2xf1IS5a182qdCyRSncyjt/FQogKK/5Hj7vbdZutjn9etXGY92aGQpaZZx6fJl+v0lmp0OrVZHZBjOUlpHnouXqjhQzBbgCEFlqUaXxumE6E5J79HMyw3zT45kO8eDbTY21hiNfYLVnPgeCBWiFXuSr/xdB+sfwmC4zGBkMaaLMS2cMuRWkpnyMqbMS9ZW7rG6ssz6S/opznUWQCniWKNTLYXNYtAqIkGRKoVRCTrySacqmmQ52cpRVTl5UVCUJXk+prBhhkVOIFUalJYkswpGo5zt5f3DrCYFW1mZRreRTK87S4wTwwZPqpyTh2Lld2Nnn5Bu5gW7tbxun7+PAQvnmFZEY5rcFqCUSBg0TPXeWoirQ0hlCLjGSr6XWwcJmAUvkUb2ESqqTSQOXlYQyvtOqseFY3HTpLSgvTU+KqtnNb2zXaCS7RqvN1YjJFFvANXIH2sy4w2Mj0S73duJ40+THejZQqMLzZZGNVLU5vCxQKBCAlcqlb+HJWztQOoJs3USLc6fQXxLJNJ7mKeRwSeAIuPfbZ5OfC8C77x5njevXeXihT5trWE8xBbjQ5HXOfE9RdDeBN7aKemtbIVSFofyo3bF4mKfdz//eZYuX2Jl+QH9j77GF27d39WxbQHbuaLAoLIWzU4XnUqPaF3pI7tMk3xioFKTKINkuEPWaXHZaPoXFynG17BlKYl2OqHZamMamayj1JSou5KiKORmtNZHr73BfTyNaGutp1KOXVHeT0cXvb21ybg4aUl3Zw9D4NZ9cG6V7mJOd6ECEoZ5yah05HnJaJSzfG+D7Zc4/ui2eqLVTRJSY0DVvliAsEPtIAo2fzjpA5BqaKWtyKuCvCgp8pzSWlxVoZXGOostLTrWKDSqgnzsGG0/ee5MJ+CslQi4Z3fKS66UBpUoifw68eS1zgd5Z8tKwZRwuT2vit2Z88eUOdjtq326j9DHKRQxiZ9BK/McVzl0rIh18A+rqJzDlnKuNp/qhjG7u6mwG+OJa+xJqnOgSyZ9bOWmPLaqvNNWLIONZMbhQZUzXFVNSXBIsqPyAfXKu1N4+7MQDYbpe6ymiybLDtakc7xiNDJF2myiGyk6HeInHCbBLg2MiwKt5Voaj0US0chE+rAxEG3w8yBnOmP8olhEai5c0NDswu21Zxe6uwK8+9nPcO3qFc4vnKOhFW48pnCHI69z4nuKoB6rCTmT4DCzTKeGXqdHUYyx1rK6vkJ66/6uqnM53k4IJOqahnkwv72JcwR+2UyimdffOZyQcaUwaYLNxHEh9jZkaZqik1R0hl7PGyqyzbpThDNxTslDfcZJfZrAFx991bYTDmst9ZHMQc4jxs/CEJnGNs2cshQXh6LMyXNLnpfkw5FU0HqJSHSCihSpTkl0Ctr5bH7nCW816QAcahe3BLCuxlqZubEh2uunup3YPMyEG0UH/CRIoYR65p6duUeRKHSYvp+8hy88K4obPqt5uqXUEUDrxIdzw5IKkBmnOJBfpSS8VTkqSmKdoL2vWKw0pRULRpdb4fXeyziw0lBUIjDfSXTYE1gLEy/Y4PO7V/UREs4mcjbk//F+7RhPdb+7iPfedt+7E/WUz+Y4NVBaTxLbw++/Nz3VzRDb4KhS+Op+RbGnFPoxweCrvXWh1QE94JlmSo0GdNpNmo0mjVRLfoOzu6dpDoCorufFT+eYY4455phjjjnmOPuYj/PmmGOOOeaYY4455vhUYE5855hjjjnmmGOOOeb4VGBOfOeYY4455phjjjnm+FRgTnznmGOOOeaYY4455vhUYE5855hjjjnmmGOOOeb4VODIiW8URT8aRdGZtYqIoug/iKLo56IoWo2iqI6i6A+/pP2e2XaNoujboyj6i1EUvR9F0TCKottRFP3VKIrefAn7Psvt+kYURf97FEUfR1E0iqLoURRFPx9F0T91zPs9s226F1EU/ZDvB37xmPdzptvUt+F+r2855v2e6XYFiKLonSiK/md//4+iKPpqFEX/6jHu78y2aTi3J7yOtQToWW5XgCiKrkVR9D/45/8wiqKvRVH0Y1EUtY58X0dtZxZF0VXgal3Xv3SkGz4hiKJoG/h14CPgDwF/pK7rv/QS9ntm2zWKoh8H/mHgrwJfBl4D/ixwAfiWuq6PrcTaGW/X94B/A/h54A7QBf4Y8M8A31vX9f96TPs9s206iyiKbgBfQooPfb2u6+88xn2d6Tb1D/S/BPzkno++VNf18Bj3e9bb9duBv4P0Af8dsAl8BmjXdf0Tx7TPM9um4dz2LG4BPwv8dF3X/8Jx7/uMtmsL+AJSH+NHgdvAdwD/LvB/1HX9+490f3Mf3xdDFEWqrmsXRdFbwNd5ScT3LCOKoqW6rlf2LHsDuAn8WF3XP/JqjuzsIYoijbTrr9d1/Xte9fGcZkRR9LeAW8DnAH2cxPeswxPff7+u6x9+1cdyVhBFkQJ+A/haXde/91Ufz1lFFEXfB/yPwO+u6/pnXvXxnEZEUfRPAn8L+J66rn9uZvmfA34Q6B7lAPilSB38NMCPRVH0p/y0604URT8TRdEF//qfoijajKLokyiK/vSedZeiKPpJH/Ye+u/8VBRFr+2z73/RT5ePoyj6jSiK/lk/tfvze753Poqi/yqKortRFOV+nT/+POdX1y+jxsnjOMvtupf0+mUfAytI9PfYcJbbdT/UdW2RqE95kPWfB5+GNo2i6F8CvhX4oRdqnAPi09CmrwJnvF2/G3gXOJbI7pNwxtt0P3w/sIwQt2PDGW/X1L9v7Vm+gfDUiKNEXddH+kLC1PWeZTXwMfAzyDTrv+xP8GeB/xv4YeCfQKa5auCfnln3c8B/Anwv8I8AfwD4FSTS0pj53u9ECjT+b34f34/IEe4BPz/zvS7wVSSU/sf8fv8jpMDkD7zAeb7lj/UPH3UbfprbdWZ77/hj/sF5ux6uXfFVZYFLiISkAP7xeZserE2RsvPLyGwPyDTyL86v00O1aQ2sIpVWh8j0/HcdZ5ue9XYFfsQf3+8EfgkZ7D4E/lMgm7fp4Z9ViOyhAv7j+bV6qGu1AXwN+LvIYK0N/A7gPvBfHnlbvsQf52vIdGBY9hN++Q/PLNP+xvzvn7L9GHjdr/t7Z5b/P8A/wMs3/LJv9d+b/XH+LDAGPrNnu/818Gj2GJ9xnieF+J6pdp053r/rj3lx3q6Ha1fgx/12a2Ab+H3zNj14mwL/DfALYT+8WuJ7Vtr0LwO/H/gu4A8CX0SI2nfP2/Vg7Qr8Bb+9NeDfQyLAP4gMLH563qZH8qz6Ib/9bz7O6/TT0K5ITs8vMH1W1X5dddRt+TLtzP52LdOsAe/798n0gP/8A6TxJ4ii6E9EUfTFKIoGgEVGFCAjFqIoioFvB/6X2reg397/h+gZZ/G7gF8GbkZRpMPLH0cfGW2cJpzFdv3Pgd8O/MG6rtdfYL2jxFlq1z+PJAr8HuBvAj8VRdHvfo71jhqnvk2jKPouJKn1T8zu5xXi1Lep3+b31XX91+q6/oW6rv8K8J1IROnHntkCx4Oz0K7h+f5X6rr+kbquf76u6x9HEob+uSiKXvaz7iy06V78IeALdV1/6QXWOWqc+naNoqgB/DWE/H4f8I8C/yYyGP4vntkCLwh91Bt8CvYSmOIpyxvhP1EU/QAyNfMTSEOsIzf0L8187zySDfhwn/0u7/n/BSRa+ySNY/+JZ3AycabaNYqi/xD448D31zMi91eAM9OudV3fQVwdAP6612X9OPDXn7XuEeMstOlPAv8tcCeKogW/TAOx//+oruv8KesfNc5Cmz6Guq63oyj6GeCPvsh6R4iz0K6r/v1v71n+c8CfA74F+M2nrH/UOAttOkEURb8VeBv4157n+8eIs9CufxSZkXirrusP/bK/F0XRJvAXoyj6C3Vdf/Ep678QXibxPSj+APB/1XX9p8KC6HF/10dIY1/YZ/2LTEcxIJ3BQ+BJPoZfPfihniqcuHaNoujPAP8W8Cfruv7Lz/r+CcWJa9d98Ku8+s76RXCS2vQd//pX9vlsHfjXkQj7ScdJatMnIUKmO08TTlK7ftm/723DkCj0ShK1D4CT1Kaz+H4kSvpTz/n9k4aT1K7fBKzPkN6Av+/f30HkT0eC00B8mzye6fdHZv9T13UVRdGvAt8bRdGPhpB8FEXfBrzJ7h/nZ4EfAG7Xdb3fKObTghPVrlEU/UlkWvPP1HX9n73o+icIJ6pd9yISi6PvBPZ2MCcZJ6lN/7F9lv15RB/3A8h04mnASWrTxxBFURdJpPnlw27rJeMktevfRJIFfxe7Z3e+x7//6gtu71XhJLUpfrspQhz/Rr2PK9EpwUlq1wfAYhRFb9V1PduH/jb/fvcFt/dUnAbi+7PAn46i6N9G2P/vAP75fb737yBTOD8d/f/svXuQJFte3/fJk6dOZWVlVdeje3oed+be2dm5d/ey7MICyxpbLLIBbSCMwIDMy4QcshRYUhjZVshhh1FgYxyW5TC2BZYDSWbRwxAWGIQMLIt4aQW+sLC73H3dy937mvere6qrq7KzTp085T/Oya7qmu6Znpma6e6Z/E7kdFXWydevTmV+z+98f79fEPwkzkX/wziDzo5sfwynG/l4EAQ/hhuF1HFTFn9qMpn8ubudTBAEHwJWcBHyAF/p9TFMJpOfe5ALPCAcGrsGQfBdOPLwUeA3gyD44MzH/clk8jin4x4Wh8muPwx0cNG913B99i8CHwC+54Gv8PHj0Nh0Mpn89vy6IAh6uMCNOz47xDg0Ng2C4G/gNIW/hdP1PosLwjoOfO+DX+KB4NDYdTKZrHnp2A8FQdDHZcr4Sly2h5+eIxiHGYfGpjP4Zty99acf5IIOCQ6TXT+CK7b0K0EQ/CiOUH8lLmDuj3DPsIXhKBDf/xZo4aYQI1y0/5/BpdPYxmQy+fUgCL4X9yX9As7z8p/jfuQbM+02giD4Gr/+v8Dlie3hvqSf38f5/Dc44XWBv+oXWHSuuUeLw2TXD+Ns92G/zOJ3cNqfo4LDZNdP4iQN3wUs4W5Uf4y7CS30RvKIcZhs+qTgMNn0VeDb/LKE80L9LvAXJ5PJH9xtw0OIw2TX4nw2gb+CG0xcxaWY+pEHvsLHj8NmU3Ayh3Uef5zEInFo7DqZTN7yDq8fxs38LgMXgZ/EFbZZqCznia7cFrgSf1/EGe4o/dAPNUq7PhqUdl08SpsuHqVNHw1Kuy4epU0fDY66XZ8Y4hsEQQ0XnfgvcYLsdwB/EyfA/pLJZHL1AE/vyKK066NBadfFo7Tp4lHa9NGgtOviUdr00eBJtOtRkDrsFzlOE/bjuLQZQ1wy5O88il/MIUJp10eD0q6LR2nTxaO06aNBadfFo7Tpo8ETZ9cnxuNbokSJEiVKlChRosTd8Dgrt5UoUaJEiRIlSpQocWAoiW8JAIIg+G1f0avELgiC4IeDIJj4EowlDhBlX300KO26GJR2vH8EQfCtQRD8Zwd07PLe/pShJL4lSpQoUaJEiYPEt+LyuJYo8chxJIlvEATVgz6HEiUWibJPlyhR4kHwNN07nqZrPQg8LfY99MR3ZhriPUEQ/JqvkvZ/Bw7/aRAErwZBoIMguBoEwY/7Upez268EQfAzQRD0gyC4HQTBTwVB8C1+n193MFd1sAiC4LuCIHglCIJREASfC4Lg23Zp80IQBL8QBEEvCIKtIAheCoJgvrgEQRB8t99XFgTBZ7xtn+SpvrNBEPxyEASDIAjeDoLgbwWuDDCwP7vt1af9Z38mCILfC4Jgwx/j1SAI/tbc9u8LguCXfH/eCoLgd4Mg+FOP5eofM8q++mhQ2nUx2Kcdl4Mg+HtBEFz27V4JguAv79LubBAE/zQIgpu+3afn93e3e8dRRRAEH8EVhDjlr20SBMFbQRB8nX/97wVB8PeDILgJXC+2CYLgrV32dUe/CxwH+N+DILjo7XoxCIJ/HNyF5AVB8GF///3x2fv7k4S9+lLg8ERzq6OkafnnwD8E/jauTN6PAv8l8BPAvwBexFWjeV8QBB+aqfTx/wBf6tt+Efh24O8+3lM/PAiC4OuB/wv4ZVz1lRXgfwUquAorBEFwEvjXuIo/fw1XneWvAr8cBME3TyaTX/XtvgH4p8Av+X0t40oPR8CfPL6reqz4BeCncOUZ/11cJb+LwE/t124z2NGngyB4B86WP4erqqOB87i8iQAEQfB+XCqZTwF/CUiBHwD+ZRAEXzOZTP7oUVz0QaDsq48GpV0Xg33asYmrQlfDVaV6E1cd6+8FQVCdTCZ/17c7Dfw+cANXSesmrvzrzwdB8K2TyeSX5g4//zw8yvgRnO2+CvgWv26Eq+IH7nn9q8B/gOtX+0YQBG3g93Dlhf874GXgGPDnAOWPM7/N9wP/APiRo1ic4QHw9HGryWRyqBfczWIC/ODMug6QAR+Za/t9vu23+Pff6N//+bl2v+TXf91BX98B2PN3gc8DYmbdV3t7/LZ//z8BBnjnTJsQdzP/5My63wM+i0+L59e9f3ZfT8oy0w//w7n1nwE+dp92u6NP+/Xf4dc373IevwF8AVBzx/gC8IsHbacF27zsq6VdD+2yTzv+EO5ZdX5u27+PKwYg/ft/iCO73bl2vw58eub9rveOo74AHwEuza37On+tv7BH+7d2Wf/bs/0O50DIgS+/y7ELm0pcYYYx8B8dtE0eg83v6Es8JdzqKLnwf2Hm9QeBKvBP5tr8LO5m/aGZdvnctuA8ak8dgiAIcaPqn5vM1L6eTCa/D7w10/RrgZcmk8kXZ9rkwM8AXxYEQdPv6yuBn5/4Hu/bfRLn1XhS8ctz7z8LnPGv72m3uW3n++WncTfdnw2C4DuCIDg2+2HgKuh8CPhnOA+xDFwkcoCrqvO1D35ZhwtlX300KO26GNyHHT+M8+S+Wfxe/W/213DFAF6cafcrwMYu7d63j3vHk4yHudZvBD4xmUw+tY+2P4abwfuOyWTyDx7imEcNTx23OkrEd7ZCSGeXdUwmEwOszXx+Arg9mUzGc/u6/kjO8PBjGTcNt9v1z67rMGdbj2s4ktWe2deNe+zrScP63PsR0+m3/dhtFvP994u4aVAB/GPgWhAEvx8EQXGz6eC8bj+EI8izy18D2k+QHq3sq48GpV0Xg/3a8RhuEDH/e/1n/vPuTLvv36Xd35lrV+BIVsx6QDzMtXaBS/ts+93A53BOhKcJTx23Okoa39kScwX5OI7rqAD4EXIX9wWB+/LaQRBU5r6g1Ud5oocYt3A3092ufxV4279ex9l2Hsdx38M6Tls6xt2wd9vXhYc92SOI/dhtFneUTZxMJr8F/JYPvPg3cVN1vxwEwXNAD6fB+gngH+12ArPepyOOsq8+GpR2XQz2a8c13MDgB/fYz6sz7T6O01nuhitz75+mkqu7XWuG0+jOY/b5D+57OrXP4/w7wMeAXw2C4Jsmk8ngvs7y6OKp41ZH1Tv0Es7T9l1z6/99HJn/nZl2ITAfafudj/TsDin8VOUngO8IdmYi+GrguZmmvwN80JOtok2Is++nJpPJpt/XHwLfHgRBMNPuK4Czj/AyDjPuabf97mgymYwmk8lvAv8jUAfOTiaTokb6+3A6yz+cXxZ4LQeKsq8+GpR2XQzuw44fBd4FXNjt9zpzT/go8F7gc3u0uyMI6wnDCBcAuF+8DawGQbBcrAiC4Bzwwly7jwEfCILgffvY5+dwuuLzwEeDIGjcx/k8KXg6uNVBi4zvtTAjPJ9b/9/79f8LTsfzg7gI5I+zM9jg4zhP2V/x7X4S54mYAF970Nd3APb8epzX8JeAPwv8BdxN5CrTgIyTuECL14DvAb4Zpz/LgQ/P7OsbvB1/Efgm3FTdG35fv3nQ1/qY+uFH8EEW92G3vfb1A7go8e/Daam+HUcsLgM13+b9wAAX9PJdM+1+FPgfDtpOZV89/Etp18dqxyVc4Omr/vf9p70t/wbwz2f2dQYnI/kELrXXh3BFHf5r4P+cabfrveOoL7jn9wT4j3Ha6S9lGtz29bu0fydOc/prOHnY9+LiLa6wM7it5fvwTX+Mfxv487hMJI3dbIojz1dwwYuNg7bNI7T5rn2Jp4BbHfgJPMSXE+DSvryKS/t0FTcF3Jxrt4ITZm/6L+kf+RvLBHjfQV/fAdn0u73dRrhR7rdxZzTsC7iH2QZuWuklZh54M+2+Z5d9fYpdInGP8nKXfvgRZqKL92O3u+zr38Cllrno7XkVpwV8Ya7du32fvuHbXcI9fL/poO1U9tWjsZR2fax2bOMCp97EPatu4EjDX5/b1zO4NFqXmT7Tfh34vpk2u947jvqCm9X6GeC2v763uAvx9dt8K47sbgF/jCNfO2zv2x3DkbKr3q4XgZ8GqnvZFOf1vQT8f9wly85RXvbqSzwF3CrwF/BUIQiCn8CNzjuTJ38K6bEiCIJncDn9fnTydORALHFEUfbVR4PSriVKPJ04KtzqKAW3PRCCIPgLuOmmz+HE8B/GTTlKyjO5AAAgAElEQVT9ncP8xRwF+PRa/zMuCvYWrtDC38QFvTxN6WBKHHKUffXRoLRriRJPJ44yt3riiS8wBP46cA6Xn+5N4L9imiamxIMjx0V//jgu4rMIvvrOyWTyNKXbKXH4UfbVR4PSriVKPJ04stzqqZQ6lChRokSJEiVKlHj6cFTTmZUoUaJEiRIlSpQocV8oiW+JEiVKlChRokSJpwIl8S1RokSJEiVKlCjxVGAhwW3qPwkmQoCUIKRAa4vRbu9CQqQESaRQQmCsRVuDsQKDwGqLzgxgmdZaFYAgt8alqM5wMcICFztYLBICAVKAsIDxTF6BETD26wCXZrzYfgYhoPy5SwmZhazYxu9bSYlEoK3FGMvYWJfdrthncZzi/Y7hhGDyv+UB94lA/eUJ4x7waVz+bY8KLnteNHNduV+K6u5D/3cWMdA9DY3noPUirL8BN16HcQphBYY9GO+7sNgOLAPveO40J0+fpts9Rpjn3gQhIYKcu1fRDUNBnrs2/8e/+MV9HXMymdy3TQGCIHhgUXsDaEjX16x1XSlJBEtLTaKkTtJOEFIgJSgpqIUQCUEsq0gh0SJCC0W81KbRahHXG0RRhLWGwSBlazjcPlZVRiT1BlYIjACNZagz0vGI/nqP4cYGVhvQBoFEqAht4XOf/Rxvvv06Ww94jQ9i14ex6dOAg+irTwPKvrp4lDZdPMrf/6PBg9oVFkR8owhPEiVCSqQAq3CsUYIUAoHFGoO1xrEGK9wD27rtwGA9+bXWYjxXygsiWRDWglR6ojkBxn5d4K9ISL+d8Vc4S0h38XEb4bmrBW1gov3+ZGEhixWA9WdY8DjL7q9nz/MepG9PjD+Nywv9xtx6XKro0H88i12I/TYioBHCsQby1CmM2oS850lywZj3geIY2c7V4fY/ECGQ54Dj4zD9CmZhbY4Q4TbptQ9oqseBY3ETicakGTmua40ABhbosTVK0XoTIUFKiwQiLAoQ1iKFJFxaQSytoBp1VLVKHEeoqsKMBUppzGhqIVUNiZRw3dxqrNHo0ZAsSxFk1BTutxZJhKwhqxEWiX3XeVa7bS6+dZGr69cZ7nItJUqUKFGixNOKhRDfOJGAQAiJQIJy750rFsdosgyjzTZxFNYiMEgrUPj2OMKbWdDWe4AFaDklUI68CraZcUGWJEz81WyTZcenp17ZWeYlPFH2sN6LZzXOmyucJ1gB0lq3C2Pd/maX2XOYJ74PJST5xN4fpXustzhCGvllzJSghgBrUB/zruN1Lmd1bg8j32YI42zXXW7jJNANgRiyEF7rbX9UESCEIAz9cXIIw5nvLM/ZHi+I0HcJ/6nNPVOGmS0ODepJh26jRiuJyPprXE8zNDu7lRmAHGj6PY3yMwcSUBZCC2Ptumz7WUMrShBCECcxS+0lhBAYPcIYjdGuAwvcLEkUSbQ1aG2wdguT9UkHt2FswBiUUCgpUVVLFEukimklz6KfPcUL555lY+02t65f5+blG/xx7yal+6BEiRIlSjztWAjxldI/sIVAeLa3TQqsI4zGOhetxBFK4T+UFpTRCDv1+mU48osAKwVGWIzCEw6BsTApDgB3el4LVmLYSUb9PgM55aXKgrT+fPB/Z0kvxdS2ZaBBpM4rrL1XWBRke95buZuL83Giwk4n7k3g5iZs/jqfJYK1a365Apv3IJwN4LkYmi3oW8j1jo/DsEIYKoQIqbgV21IHgNwLHma/CyHCO2wktgnw4UCIpC5BsIXRGjMebHe1YswjcaY2gDRgjes7iBkljAAVQdw9wepz72Tl1Gna3RWipOk2EIJIa6w1WGuxxiKVJKyCtIJISIwVbNgRdjhg7fpVBmtrRBVFpKrEcZOldpc4bvlBIdgspWJT4ip0uxFn+/C2PYxDixIlSpQoUeLxYTEaX1XsRiCEe+oLBMaAMGCsxXhiqwRECCSO9Eptkb6dxBHfFD9jL8EKgZUCK4ST+lpLaqzT784vBWY9sbt4YZ28YuqVU8YtwnrCO7NIHHk3FoTXGgvtSA4CpAIrIbWe/BbO69ljHgQUu3uGLwKV34Ysg2x8p1yiwErLeY0rGXQjqudOI1SFrXATNtd2NK2EFSqhICy+f2sL0YM3fzGqmDuGfx+GwhGyO0YPB4eGalGLIKq6mQljNNqY7S5VeE/95MAOB7+Y63dCgawL6t0TnDh9luUTp2h2j6GiKiZzalwbu1GaGbtjCSVBCYSFSEqMFkhrIN1g7eLbXHkbIglRDO2lq+jBKu3WMrJaQwoBozHCGuLIokQErNK6fZs31zW3H4sFS5QoUaJEicOHhVVuE0L4xXl4rbUYbcm0dUE41ukdYylpCkEkLJGwKAWRAKVsIacl9XIHK8BIp8E1EgbGsm4MVjMlvrAtawg887hjSnfOG2x9O2Eh0pBoaAlIpD8X4YhwodTILKQG7C3o33SEQ9ZAJm4fxk/jD4tzKWQWM+f0WJEBa+wt233jHkFsFSDKoF5xOoa6gDDDMgaxCdFORh36ay6kDiF4mzufp1c/3B2HiPQCWJ0ythZhLEZYp1vPpuqZAoWJi6+86l9HEqIEao2YpNslWV7m5NlzNLvLVOPY9W1jMcZgZmQ91hqMJ9iyapFSUlGSGEE7TdFG8+YXXydDMzQQ9iFNwdrrWGuJo5iKUIyM248eGUfYzYhaVbAioWd2+Y2UKFGiRIkSTwEWQnydl3dKfq0wWOMeuDo1WONIbyQliXXEN8GSCEssIFFeliqc9negnVfXEV6LFgItLOsYjIFMO243hh3uNiGc9KBIdLBr8JmFiSfA0jh+l6RwMoFjEcTCnce2YSwMDPQzGFwELkDUguYxH9SnnGM1A4ZzpLciD7Am9D5j1XZFBefprYwhqkIFBBljxqD67rMZiILseqVCjifDVmxLXrY/9Cg0vrPyhm3d7yGARrtBktlJdneTbhemnu1uUQzdlVW6J46z/NyzdI+t0lo9xVKni6oqrAVjR5ix+63k1klBjLVoPUJgkSZGKElVVZGqRscYhJBEjSVG3CT1x940IK+D5CY6TlBKOUKtDdpopxFGIKWi05KcvWW4Cg+c+aFEiRIlShwAlv80tZUVwmp1yi1kyH5EgjmAyQlllWo1xBi3Ni8+DKEqQ8JQ+nUuO0Dodz7KDZgcKUMI5cwxvaQxHzEa5YzMiHw0YjQagZR7nps7nxk3Up5jDYDZ21Nm/Ae52aPB/rAYXlaE48/8tXhtrzHbnjIhQSmIkXSEpRNBIiUdJYmlWwSQZppUGzQWjSXDknqPsc6MI8QW+uC8ZMpJDqTyxLfwwgrnSAxwpLhgJjIDNQA5BAZgB6COQXPJSS6EATt2ba2FSHkCdAPsNacJbrYgEe4YUsIA5+0zEnLpiJ8sjnvUMMbLJHL3IkvZ2lx3hDhjZxo3cC+KwLaCBOd4cfReBtiXH/jAMOvJ1TOviyVgd6+pxHl7G+0O3dXjLJ86zYnTz9FcPUacLKFqEcJLQJz23frASovJvZfWaKRw8h4ha6h6jJRuVKeimBPPnuPU9Ztcv+wc+xlucHZ7A6wdECnlPccWPYKRBiEtUmRUK4LlDsQpfCE7zN9AiaOAJTo0621a7RZboxFra2vctlcP+rRKlHgi8a1/6Qc4fvw41aqaIb53p3EFcfWJlgjDkKqsYvzdPy/IJlCtVgmlJJ8hpKHffz4auViWMCTkTq9ebgwjT3jzPHfEFwjD3c8vL8jrNpmFHJfC1uS7P5lyY/b87H6wEOJrvZHcdK1w8TqeZFrrSKPRroWMLTGWZiI5Fks6ccyxRpMkiomVQgpBlmmyTKOtJjMpgy3NQGcoBEZnWGvQxvNYBSYCGUlUJBHCukAk47W5YsYJK5xnWRRSgHUwPej1nJRBW9B9SHtgPMETQOuY02nano+bG0BsoCmACGTkYr5i6zJQaH/MI1seZAzcAOrMuNaZEt+C/M5B4D29FOR31hCW2f7qBgSHK5htHmNc+uMigYf/ugHXn+ad6tJ/vtSNaK2s0Fpdpbu6StztEDWXqFR93j9gVvdscQNErQ3GjDFji1QhUkVU6jEyTohUFRXFJO0OL7z3KxBRwiuf+wzm5evbWSYGqUuoItDudzd2GSWsvwBjQSpLowqNmsJe11w1lJrfEveFAMl7n/lSuivLLK+s0Gw2abZajMeG4VafCxcu8K9e+ji6TKZXosRCkecj9GhEnpspoRzpPds7zhrucDOFYUg+8zA2MyQ3z3P3uWOh2+23PycnNyEwgpEnxcYR1jzPMab4m28fM7+Ldzb37QqESEd+tz/fua3JCw/1w5HfhRBfk2We8JrtlGZuMY4A+2wIwkKsJC0RcSyKOdmOOdZqcXL5OEkcE0cxUgh0pjFao3VGlvXppyn9dEASDVAqJR6kMBggjSWVkIppoQmERfoiGduZG7x8IZFwTAp0z9LvQ3oFsp7jcbfeclrf3g1YH0y9fAp43jgPL30fEJc5bXBLgmoK4hgGqWWQeUepcQFvRYaIIwmLC3zbBNZx7nWF68l3ZD4T+Fg2YIbOhkA+/cmFIZD7QLYdAu3DjSKLw7zkYd5nHQFJC0d4T5ygc+IUzdVTJMsnqC8tASCdkBdMjhXaX73TxBf63txaEAKpqqgoQtVqRFGMkE0qQvDuL/tKOieeQ9WW0OOXuHX5MnbgBm5ZMSiZG3RKnC5eWlBViGOIT8ecNoo/uNgryW+JfSGmyb/1VV/DV3/wq2m327RaLWpJjXrSBCwmH3Hp0iVq1Sq//jsfIysFNSUOI6rAEi4Zu/HvpX+f+b+7bXPSb9cFNoAr/m/BpDbmtlmwb8epDYx/fW/yl+chYWj8c0vOrJ/b1ji6WRBfZl6befI5k9NIbpNdgHxbPgFsyyHuRnx3YPuBKpGhI7nz3meAMOehZyoXQny1xhFOYV0CfyGQUqAQRP4BrAzEQtBUMZ2kRafZotNq0Wp16LSOEcURkVIIIVCxC4gzOkPrFJUOiNMBIh4gGgPURp/01i0Y9LllDKZgm1L7vKhmu7Kai4qHREmOS8k7paLf63PBQjpwfT4FrmVg33C1IdZxmyucx+9Yz1elA1o4Em01CCvpxAlJE9Z1j2tmpzP0SMocdkOG+4HvgTAsxpRea1QQ4NwriITL82G9jsj12qNhnNkBENyp9S1+gBWcg7zRiEiaTeJmk2pjCdVYolpfQsZNnw/a+hkSp33Xfk/C20gISei18i640vrU1YKKkEgpaba7IBQnnz3H2bXbqIpkcP0y2W2DtaBnOqH1J2xnLsYMIbMaFRnipM4qpde3xP5w9sQZzp17jnPnzpIkdepJk2q1SrVWwWLJ7RhrR5x/4R2s3/5yfvfl3zvoUy5R4k5sVyBiSnrBaRb3YlUj4AKw6t9LHAmuzrSZJc3HmE4RLgihvJNJ77ZuJ+S25xd2enDBk2AZItnJ02fb7UoU/UqXztYwfbjvj+3nudl57n4XITm5ceSXfPbgroEhZ3+q5r2xGI9v8XAVIK1BKl+RzYJyJc9ceVcpacUxnVaHztIyneYxOs0WzeYyKoqQVR8cZwFjsSPn+VVRnzjuI9M+Kh6g4oi+cUFBdjBgoF1lNSGNS39WzP16qUIsoKUkx6KId8Qx16KUG8YHyuFsew3n1OyxU7uZ4qLg44Ej0QmO+DovmqCTJNiG4Ep/QGwMmZ3x9MoniPzeFSGi6Iq+s7pv3UIotkdoIgRbFLfYnnc5qHPeH7bwSS5w5HE+fVmh9Y2AOII4WaK+tES01EYlTUd4kwRZT5DWIIwBtEvvZ6eZUIRPBSdliKxIpBSERcAm/q+QSFklaXZRcZNTaz36t2+DMdwYa25vXcVmMzmmZ65jW2BhpjKeOLI06oJWIqgPbDkxXeKeOPXsGc74pVqrUatWEZWKkziFLqw4FJbzz59lNNpaMPGdz6lSosQDopi5jJiyIMO9XYk5cB1Hdrs44rs0s/2QqUywC7Vqdbe9PDDCEOSMZvZeBDAMpb88X1fVN5dSet5mdkofCrIrQd4HPZR76Izz3AfDAcZ7qOU8UZ/3CIdTxptj8H5jTC4JZUGwHw6L0fjOBDsZA5qpR6tIY5ZEsNxscmz5GCdPPMPxzjLLnWMkUeSi0kKJFVX3hLeesUoXPSaVRtqM2EYs+8wN/VT7L+YWmV5nYCAduGA47QWZwgeeJVJyTMUckxFNIm5pQdpznt1icDZi99mNDEeI48F0ht9oyAZgM0NMhIoUHRXRiQaOTHu98B0FNJ5QhAXTJYfcTglXKKazHj7QcDuJw5xOIAwP7whhzO7ZHIp1Fnefa7YhaSckzSb1RkLUWEI1Gshagojq2GwIRZEK3y9kRYKNsJEr522xWDt2eaatQVqNzA0VjEuRVlPIikTbiHa7SXdlhXT9Fub2TezgNls2w2cPZGb8tV1oYzaln9GgtzbolaS3xD7RbCZO09tsIqsVqhVFKENEKPyIP2ecN1ldPcFWOiJAMlkYWS1KxTy5CJnOLll2fyaVWBDmjXs/XSsBVkHUoerHY3nmJ9UkUIfGEtQXy3t3EvN7enpn2s16b+X0bxHXtsMLLO+H8s4dSsod0ohZUj1PeKUMHRme9eya7Q1hh9RhJofEAljr4rJt2WkVYXf2GoxB+EpWzUix3Ek4vnqMk8ef4VhrmeXWiqtyZUdYKhS10kQRGRcKhAoRNkOpCIF1pVyVQGtQQqEzQ7+fYbOUQeYD1PyVRcplkWhKybKKWJYRLRQyg0F/fzeVCY74Fh6/4urS1MkdEhGRqJjlSLGsPFH2pMM+2ffoHQiZitQtPsWdLUoXW1fGeJs5+hdF9TbALiBS81GikL4UmCWV4FLbLbWaNJaaxI0GcbJElCRUkyYibiBUhNUa44dP1pfBVkoihAKjwCps7mYylBAIO0YYg7RjpLVElZBYSZSSWCFotdssd7sMl5fJ1rro2+vkwwzt0ywbf87FeW4TdU9+zQjGOnPZUUqUuAcCqjSbderNhGazjqhUqMgKIhQIWczxWBIBqyePQxjy4pkX+dyFlxdw9GLo9uRqhqs4aV2NKekVPMlXfAgwYkq49vu8zoG6ywR1bNWRWzOE0RCGEQzrUJfQieR9eU33Bf+QnZc3hMi7OkKLj+Y9s3KX7Azb24T351ktdL6zxzDG7LmfwhtceJ7vOF9PmgvyK2dzpj4kFuPxnXm9PTWLz+urFLGSNKOETmeZTucYnc4xkihGVZxH0NgQiUSgQChskUBVWGwO1mh02sfoAdZkYDQtKTDNJoNuhjYG2buF7qcue4SXGyTCZV5YVjHHo4QmAjJDlmrS3aqa7YEUJ4HwdN550AaQDS0CQSIjWrWE5WSdfgqRdbmGi8IWTwcK6iu2PaOFN3Tem1sodPKcKQc+5JqHCe4eWcTwzRJJCSRtSLpt4mYbmSSOCUuFsZYs3STLMka31zAba2A0whiqUtBsxCglkY0EVZOoqMpWTVGRgjiqoKR1Cf1MBTNSmEghZIQQikhJoiiiVqshlXKuh4qbERF6em6KaWGNIsWekE4LfOv6XeXbD4zAH7fIbZzPrAuFC7Sbz4pR8X8fJgX1PCooDDDZUWi6xIPgRGuV1dVVWq0m1VrVjWqdBgdRKfIZ5khbpVqr02yNOfXMmQUR3/3MQx9dBLgZ80RC7NNnZto9ewTub1l05hGh0DveJ7T0vNm4mW5joF6HpTNuBrD+SO43uxPVHOPI73zr/XqF53C/pNdtwzQn8Pbxd6bq3W2DqaP3zobbquFdgtweBgsjvoUOUc4sKlJIBC2VcKy5zMnTZzh+/AzLK6eJrIZx5gLcDSAlQsTO+2UzR3otWHKybEDau4YerGOyFKylufwMcaeJEtBJIqKrAs0lbN8w0O5cWgI6MuFk1OJMo0U01OjBOoPbjsjuF6m/xgE79fCDvsv5m6iI5bjFM62MHutc04Z+CjqdpjZ+shGSI4AKhBXn3cVOs5nBDp2AsGJb61sgxw1yDjtyinkJqElQkZtZ6K626J44RdxdobrURtTqGCEYGcPmzVtovcXg+g2GN68jsVQlNJOYpH6KKFpCNuoI0UZvbaGzFCEMShhnN5OxteUCR4W0RPU2UgmqEmo1hYoUKIkRwt0XPeMUxvVTxU7Pr/IzIVd6cPkR2em7v9/ZRWsnqbA5yAokCVRjGA7c7wfhbJgkUG9AugVf+Bx8/vPQPQEnnoVPvgS9BzzRGiFdVlnhBFe5zkXeWOh1Pk04/+ILPHf+LN3jHWRcmRZsEbgKj34oGBJSTSx1G/LM2bMc/+w5rvVfX8AZWJw/dMyT5lKQQKcJTQXtqsQaw+bQpSiUXvl3j3qbJe4XRbrOB3nuGLft2sDHww2BEZzuwJmuj5cbFQ0XBxeAtgspnZn/nye7++GwhYd15y7vkx4WkoTZ53oOSLPDxEUYXWiAMJ+qHFxFi7kTK5xku2d4eFAshPhKAOEfqqLwhgmEdUnNlFQ0kyad5jJJ3ESpGJEZrB17fXARVun9aMJpe106JovONOkgI+31yTZ6YAzLcQslFbHNaNqUY0JzUlpEBJlyI41WHNNKEo4rRWzBZJpb6+us93zasX1C46QOhQerCAjt9aDf66PTJgpBq57QTAfEwqVxswanN37SUeT2mo1u2zO60zHgItANps0Ou9cXnFcyjhzpjSKIaoI4ljQ7XdRSC9loQhRjhEAbg9ncIB30GW5s0Lt6md7layglSOoKVjoYs4qoCGpxhIoidKTYymowHoLpobOMNBticuj3e6goob2SkSx1GOkMKQVSRchqhIxiQqUQUiOVO1cXYDolvsJnJBloeBt46xHZ6ew5V7lwlHria91vspY4oqszGGWOpFdjaDRcysDb63DNu6CNdplXHlQylKDo0mFVrLC6uoq6JdkYb9BnbXEX+hQhxEl0XFIW4e/7IYSCiqx4Lhpi7Yjx1oh+7zbXr1znWv/tBzhaMV80i8T/TXf57OhDCahVJJGKsNJg8wyDiy8p5Q6PAA8T2DAE1sGO/Hfji2HdCKGbgKz6ZSEnem9sSxl2fYZOA9b23sECnr27uXeL3W4zX+lnhouQuyKXxMilMINdXcThnOzhoU/1ofcAPu3S1NMrhEDYqZNPCUWz3qSVLBNHDaRwakkngnbzriKU05LH27pPMNqiM0M6yOivDejfWscaTdw9jlIKmfWI0ht0TJ8z0tL0UZpKRSRJk2bcJKkoImNZH6RcuzHg1vr9DfLGTIsTtHDe4k2gtw7rt3oMek2EgmYUkyhFLFKE92RP9s4t/eQgnB2D5j66278m3Gls/3CEfBroJsK53L6HF3ES0WgoIglxJGg0qiRJRNJeIUrayKgB1RiD0xLY1HL7+mX6N29y8+JFbly8TBIr2t0lIgk21y71XxQRN5YQVYVQGpNaRps9smyTtbWbDIYpohIhZJVTWrOiDXo4QiCIqgoVN5DxEjJS5DW97emVxi+4B6u1rlhLn0dHegFWus4JMYp8lgnr5BWq4tbXa66diiBpQKPpgkGqFUeYBz233XAA/Vv7O2ZIgy5NaggUITUUcSVmdXmZ1W6XWjXCXDa8Mv4TeiX5vW+MRiMGgwFmNAIsYaXi8raL0MmZfJ7uEWO2BkOuX7nOa6+8woN5ve68FyzRYsyYlDGLFcQcPKz/z2VucXNKFoEhQ1tLNd27WmSJA8AGrghWkf5sw73elHAjgWoHqrMZIxaM3aqh3VPWUCSj37GfxTqb5r3E27mGw513AenJr8/w724dZuTW7OJpLrTCd+QffkAszOO7re21biq7IL+FxzeJmjTjFpGMEUhHao0LGZJqXjXpKLO11pVdzQxZZkgzQ5ZprNboYYpO+zDoIYfrJDrlONBSoCKJiiKiWBHVFGZkMFuG/nqPK5dhff3+r1EwzetbeIv7PUd8e2tNZDsiakZECGLhSMZsntdF4kQrJszr3Ny8eXiifnebfWEf1++9xf6Zeeix1D1BEkuU0NSqECeKuBFRqycIGaEt2KHGbmboNCMbbnD74kX6aze4ff0m69dv0ukuITB0ugnGOBIhpEBGygfQVbB6gLWGbDigv7bO+u01ciQGiRQRAoXFBU9EUZ0obhLXW+h6A7IBkXD5q4V2i7RuUJpljvQ+KolDgUrkibaeBtR5OSjg7m1SQhRDLXLec6UcEY4T5/21gBmzrx9RSJt3cpbVapt6S6KqIVJCqCStuMtSvUWtkRA1a9hXLX8w2qDU/N4frl+/zvXr1xkMhtvC7YoM3eweAALC0H3f4zFmNKLfvx9R2d0wPcpUOX54ETB9uO6Houe4NIO2Zr10sOJ+KxXjfsCUpPdQoSheUTzkBjivbwJXulAN3dT9A0ps98Ye+9tblTCbzWG+0d4nt4jMCe6gMy9NcceVnoOHM2eRu/y9Yb5rVbb5bBELPK2H3IlP3WX8lKYVIKxECUmsFLGKiWTdkV5j3TSw1j4ADmAmDVZRycqA1tYLxwUCRRQnCJVhbUY2WEdv3sL0U5SBjvW5hAUIYd2Y2WrW+ym9XsqVKz0uXIL1ucpjq7jUfHdDEdU/GyIzAG7dhBs3rpHQIpYJ0hhiTzqUgvGCydwS0KocI4xiyMZcG/cO/PEt7vLj3jnTMSv6hak63DUMD73GN2J59QQ1BbneRPhQR2slZgQ61WSDNbbMButrPa5fvsytq1fZvHkVnentoZ2UkqV2ghm7UsXGWKxfsCBywFpMukU2GDK4vcHG9R7ap8pr11aoiYQoXkLVm9RkRLu+xKDdhY0W6dYGkUqpRf6ZqcFuuXRyA+Aij9bbC7BxG6IK6JHz3Fo7Q37llPgOU+j3Id6AWh2GW9Dtwld90AX3DDNIMxjc3PtYIW3exTt5/vQ5VlbaVGqCKJZEcUSlWoERWC1oSMmJZ08T1WPE5xWf3niZ7I5SSyX2whcvvcqrr5znhXedx4zHCCHIjVP3W5MjRAhYKlgacY3lbovVlS7XhhcWcHTLbdZxd5IRhzHQrc5O121jq4MAACAASURBVE0h1pjPDb8X1jXIDY2SbhbIWMvWyJBmDzcrf6SxL+/JAWDI7l/KBujr8FoG9ehuhPRBEe7q7d2R7gvukC4UpHeHPniP57ZbvZgTn1U/5tKR3zuPVZwkhMY13vUrl26LEB7aZ7GQqyuyj/mCVE6qIJ3XVwlFJCNilRCpGIFyRFZbtDZIqVHzEWB+3sdY69tZ9BgQkjhOXAoOm5FtDtCDPmbgZAhRkbt3u2qEwYwzzKDH+o0B1y7DpUsuf2+BVeAd3Jv4FhkdFHPE9xbcuKFBrSNjEEYTC4h9kFFlgcS3CnSo0IpWqIQRlfqYqNdjiJNeDBZ3qAeE0/8Vkt/pWA6mjHfu11akGDyMN7c51JorrKycQIgR2SZYPfSi2Qq5Bp0Z1m4PWd9IefP1N7nw9u4BPfXBgGyrizaWsbHY8Rhr7TQNSFHAZStjtJkyXO/Tv7nGZupmPVrxMpFs0F6RRNWEqNDQt1YwjTYy61GzEJvUkb4MMuFIZQpcfQy2ur3uPLm5j3bW2nlARDDNry2LkQBQUc77KwJY6sCXvR/6A7h5G95+++7E952c5fnT53n++XeystpGRBMqcYVmI6GW1Lh9dYO167dp1BucfvZZ2sttqq0Y9WnFv7r6G4/BGk8OXnvtNdbX1sjHBkSFIoVZkcxMCEEFSKpVVlptVleWFzjKKuoLHi6PbwU3E7jEznzfhZMkZX+pM28DUQaNrQxZVWAtmbZk5v5iUp4oJBI2Dtq1cx9YA+pgh48mGHEvH5Mhn+p7Pemd9/Dulg3svgPY7htzoXjbQl6YPvTNtEobYPbgA9Lnf/BVjR8KC7nqZjTNW2uNIFJNItkkwpUsllJu53XFWuzYYLIROktBgbUGcpfAX1j33hiD1SN0lqEzjU4z0AahBEpIdDbAGpe+rLgFFiPsTFuM0ejMkgG9/oB04AJqjJ16bgvd7n5+Vht+mdVZWZw3Kk0hTQ3RYIDJMlc5zk6DiRaCCpzvNuh0ztPpPud0AXFIxiZh79qCDvJgcJOdoa/fNjPiLGpq7+jIxQu7/YOY/jAOt9bBae4EUlShGrmzddMLGCsYbmbcuHiN19+6yLVbe9PLS5mhud7n5CAjyzSZ1mRZhlKZf1rmMDJYbbBZhhlusbk+4MrlNS5ft1x+a42l9md49p3nOHv+BZJGGwPUhKAZRcg4JrIWhXWMs2ax0gVhyAGsGLj0iG31//6807hZCxNPfou6NIXjX8ywhMCTYR/XigW2MthM4cpbex8nYpVn2ic5+cwJTp4+QbPVYJBtIIWgoqpEKiGuGbKGIYpiELDUbvPiu15ERRV6H+vx8vCPHrE1nhycO3+OTrdLGAqsGTMy1s3U2BwhoBJWyLHUqlVWux2++gMfoNfr8wev/f4Cjm6AOiFNKtvJ7w4WVVzIXZ1p6F1Rg+l+6xdpHFm62TNI4cqPD6wjxE+Wovk+kB0h0gvOC3ydaWW4x3H6cprI7K6V3O7I2vDoTmnnIQsto3v2G4yv1uZigJzS1xtKgjSu/Z563gU4yhZy6Z3E/9CtABsTqxZJdAyhNUJrlJRIIfyUmKtcZbKUbOBctTYZO/KL9hIHjR0ZtM4c8R2kZP0UabXT7ypF2huQpnY71ct2Mgign0FqNanJSLXT9A42XLU1gxudN/1fgS94sQcq7LzpzE5ZFcS3twFRAlGckhm2cwlL8eAGrv7Zk4gc6rmlQ8Sp+nOcbpwmVh2iSossG0OvzmaUs3kR8pvXaPvzvfGAx7wblmDPSWEXSlRQXtdp3XfhClfM9t/t395U8+4HB4c/o4NBoBEuoEclCKm2B1tag862ePNP3uRa/y7uSY+LV9d4diNlsJmRphmRylBy6HObCFddIsswm1tkGynrl9d49brlGrg50fXL1F6/zAc+/xrnX3g3J888S7fThXpMYttUrEAhsMaAzTEyZcNkqBaczCDRYKXEiIg00wyNXqgP7Qt/ssCd3QXPh8/x/AvnOP/COzj3wjmq1ZALF0fYiSWSMfVaA9sErCTwTHtlZYXnv+R5nv+S513hhY/Cy5sl+b0Xnn/uRd77vveyurqKFAI7MpjRFvnYYLfGVCoCVa1QrVZpVmtUWy2+6Ru/gfPnz/NPfrbNx1766H0ecbaUU4GYM/UzJEmy2waPFTWgjcsVH0moKx+nIFw2H+3Lg+93OF8USzLeoa1x3Omp1vYemiCW+8ADxBA9KAqH7W7yhgK7BbDNe3rlXBuzqIJSuXT7LvZn8u0SxcZXD9mOuQunNBiT7xnMNlt6+UGxUM7vMhkYtB7QHwhiKYik9AJvg7GZm/YUkGUpaerScxuTgqmCVQgMjFOs3sBkfbLBLQYb6/R7PaRNkUQYZRmsW7IMlPQE0ydX1Rb6qXEljP2SaedFKu5AMXCyCa0YBusutdNeuNtIewu4sQ7LPVfAoOm9z1I4fW+0W0aefSK+0SQkpzIeQ6VCpd4gap4kirvU6y0q4zHjKMRWoNNdYTzsUScktiG9/hpXbl7kCzcX4wk+A5wCbgJf3OXzcDZqKR9DPsaiIVTOCxyG26O0fLqRa374+e42cgsbwy0sVaSsEMmKS6knpSs3rA0nTz/D8LURG/ru9dA2sVy/eZuNjU3SzYxIDalVI6y3pfUu0sFgwMW3L/OFgvTOYAv49MWrCAFRVbIUC2IFYilG2BxhLXpLk6UaIWMa7ZiwUaEjE4xMQNZARWTaoLUbdB4lJLTZyG/z+5/4Q27cusnlq5dZaiaMsgHVKPJlHA3WCipKIYTA+JLRIhAkSYN3vOM8X/M1G7zxa28w2J5KL7Eb3v3iu1ldPc7WcItXX3mV9etr3L55C5uOwOQ06wmtbpNut0NnZYXOygpJUuWZ0yd44YXzfPaVV7jSe+s+jnhnfwwQjEaagxZ2FVXWYqbpDeOae64XZ23sNCh6vwR2jCO/TzXZLXEfCKde1RnsqumdfrjLfnamH5WEmAXoD2fTq20Xowin7129ZDmrepie0ayHeMFYsLPbCX2zbABZhkgSoqTpvX8GY7YwwmIsZDolSwdIITA6xdoITORYo93CmgEm65EN1hn0e/R6PaTNkBhsBGnPSResjwgH3EjbQj+FXuZmSTLLthyimEKNBZw8CcsJXLGQPoSL9JaG9T50hn6E76dxI+mJ7wOivhYjrAU9JIxCKqdaVFqniFod4rhFZMcQQdSIEGGFsBLRUjGNqEmuNWm2wZVrl3n9yuu8ceFt3nzrddi8tydyHieA9ws4puBa5gKj5gfhYVh4fHOXgy7PcLd6CypyAggRei3KVLieEzr+G4Z3VHw5lLDQH2QIWaEhFSJSqFpELaqBHmG3NKeeOU2tWufjn/zX99zdjZtr9Pspw3RIHMWMogykRIaO+Fosg40BF98esFd40Abw+ttXOb7a4vSJJVqRII5jFyhnLP1x6m4cqk6sYpJqG9VaRrVXqKjYFdoYW8bGHCLV5L0RIlhlhTVu83b+Fpe+eJGLb7/NyrE23UZCu912adwsVOsN4qTpPfOa3BoQgnqU8I5zz6HiCr/30ku8vPGJg76sQ4uAKu958UVWj68yHKTcunadC6+9zpuvvI4ZbVEFlrsdnnnmGZ458wzPjUfUalXqtRiZJLzrXef5iktfzvpvrLP/ItnzPbJGhQrajLDm4Cb/C01vA1dlrRa5QM4oEphtqb5FGK9n9/I6zf4IbUl6S9wf9tD07sPTO7+P2be75wS+TxQp/cFx3GL9HqeRz3HcRWZymMVCiK/2YiYnd7BYq50HjNgl2JcC5/FN0cY4eYF2Hl8hBFr3MVrB2BWy0FmPLL1Fv3eDW5evcO3GDa5cTd3Elx7QjGD9uquMFscQN0AlEBlILdy4Bb2ho14aR5C1lzxkOP4VK2glMIhh8BCe2dvAtWuwfAxSDcIT8UQ4zdeDGvjSG5/c8f4Lb7zF0h+9RqPVIm4sEdoxw/5NsGOarQ6NRpdOa4Vus0tFCGDMZjoG0aTVOM6J43AtrDPp98Hufy7myyI4fxzqGrgG5y18dq6NGF4j3xSgWjBW5HkfdMqmFQx1SKjqEDWoRHWiekRUqZDPljHOd/w5pHCpSjJtyIxFIYgrESJqoBoJZphizSZLnS5x0uBdtzd45c3P3HWP1oLWhmyoyaKUtKqwUoKSjLIMY0YMsy3698gF3UlgKVEkMSzVK8QqxGQGMzJk2iJTUDWFkm1k3KLWbhO1l5BRgowSLCHW5keK+B7nFEuiwWV7FdDc4AL98W1OXV7l+fazKClJkyZZI0UqNzIWhbh4AjY3CBE5gqwkJ1ZXeLlM8LAn3nvuvayePIkFXn/9Db7wqT/m85/9Aq/efJ0JjgSeEB3Onz3LCy+cJ7fQareptTo0kzqnnjnBe973IpeuXuFTn9+v3lfM/HWPTE1KnxxxQBrfgvTWcY4NFUFUdVUKoyhCe+YrjPvRzmp95wltwM5AuJLwlnhw3J1pOD3v4yqncSdmszm4IhWaPDcuf4PxdDj3xSzMTGj8gtOYbZ/PInaS+Qez9SNdly3fkUAZCYQE0BjjvE8GXBni4QAhLTrtoSOFiSOwAp32GPRvsH7jCpfeusClt6/x1iW3W51CEsGtSy6orNmEpAlJy8kNUgNXrsJ66q7OCOj1fc7dnpse1j7cVuEC87Im91fDeA4XUjjeh2c0qFjRSiIGwtK32gUYLQQZGxdfYuPivdp1QNQJogphGDqdDGB0BmMnP8CuwHZqoN3RBt7fgfefgdUK2DXIIjiduqm4ItivDS4/nHoD8tg9Acab5FnO5ZtwsQdhDGGzQWvlOKtnTtFdWfFzHtPplfzQsy4XeTUaW7QBi3SleaIEmSxhcjDpENVYoikFL7znRQbacunyF9hrVKWqDU9+NcN0CyUlVipsVaKHQ9KtjMwYsl23dngxhi97/7s5+9wJVroRrcgSCUMqIjIMSglUVMWYGKFayJovIGPHLhoVgwhDRKV62GMLt5GwxKo4TlyJyUdT22ZscoEhJzaW0N0uY60xOscalw9cSJBCQuASqxujUUrRVm3andYBXtHhxvH6Gd7z3vfRancY9Ad88o8+xa/+zq/saLMJbNp1Xn99nZs316jHCWfPnGG1WiNJEla7K7z7xXdz4dIVPvX5T7A/T4MAmsQkpPRxd54RrjzL4++sAS4Yuub/RhHUVJGhRKGqCoubqUE4SY3PJniHZK7Yl8QHZHPYB/4lDjPmHbmFt/dhsjbc77Z7VVQLi9VSuoIWeU5uRqRpn5EZQe6quYVhlaqMCcOqK8axLfH3muAF8t/FeHxn72FS+GhtgYwESkmkFFirMTrFYNEWMp2RpilSWrK0h4kjbBZjBE7e0LvGratXuPDmNS5cggtXnApCa0gU3LjkPKzNviO/zcxpbFMDl65AL8VljJAu5dgts3OKvpAjNBMQx2C1t3dKsxpwDFfidTeMcHKHVIMSkiiKiaUhNiAeux9tHew6k9R1l/31lQiICMioAy+ebvClpxs817Q8W9fUdY/eOCXbdMRX4/S+13CejxBgDQQpIk4Zb7qKu2+k8BkgSiFKNzmtc+JWi1ZnhTD0OXwtXh9sD7nH0f1UtC3sKrBCuewOUQLZiLGsEMcRcT3ilJQQJXTefpaX/+BTzCcRC2mRJE1AunR9W5qhGILSYCQ6c7Mjlr1/pM8BH/jgl/Ml7zrN6dUlltsxDWWQaLAaoyVSCWRNEdkEoZYQKvZPW7O9iErV/biOAPMNqPOu6ntZba2ANtRGEeOZhJpj34ukrwI5vSSLQOJ4r8BojdEVolihIkWrUadFld6RjKZ5tHjP+97Du7/0BZqtJutrN3nz9bf2bJsDn+6/xfnX32bt+k1a7Ra0WrRaCeeqz/KO8+c43nmGa+v7ye1bQdGkVW+zNdxisn2H2OJx99XdSG+kXCVCpVzAdUVVsQgsxvU9f0/ba8JmPvVZiRL3h1kpwp1l2QriWgSu3U/A2oNkfNitnPD0DA35aIQZpaTD26Rpn356i9FoCKZKjqRabVCtNonjmLBaRSIZ5Ybc5F5IGRJKuZBqcwshvipqgnCBZkL656m2qFgR1RVKCrAGrVOUj1hNB336/XWwikFbEClLLA0Cy/q1t7j25hu89SdX+OzLcOkWXPJ3hhs+Z+86bpSc/P/svV1zHEee7vdTVlZWdVV1odGNJkAQpEjJnNHozMzO2d1Yr+2zdoRPnLBv7A9gX/gz+sbhCEf4YuOc9c7GnB2d1axeKJHiG0iAIIBGdVd1VlZWyhdZhRcKpDgiJBEz+itaeGF3o7q7KvPJJ5//81SQVJDs+MQn7eBxddL64DgLeN8BcgVZBvkqjKf+uOM1uP/EyxZ2NF00gQfGf/nXsLYOH30MH31yPkCezbzvqNQG6axfDPSi4re+vCb3a/z7Fs40P//NiOtXcjaGMYEeQfM5jWmZzr10JMZv+yn892kDoxbCBp7seevOf+mefdnd0lmF1o2PN8UHNRzbnx37I7yt5ZMXvEe0xHUpao7Qg2AZQRyjcdhlhVCSq++9y+T6LVav/4zd3X0Onu56MDAMWR8lXF3PUXGGdcJLEoT3/XIaqDXCwvraOn/1m4rRvafMCssM+Pm169x49xq3bq1x6+aU9WnCOIsYpgLZLsHCfA7W+UhwFcdA6vdlpfTOq60Pz3BaexcJRzdZv8015O+u/nv+6sPfII1DlyVa13y2/ATJOyQo1oMp/+ZXH7B+9Sqrq2Py0SqDYUoSJwj5zrH7i7EW2dRIJ4mEIM0yVhkz+0Fcji9PDUXOr//tL/ngg/fJBxkhIYMofeVjDPD0yS7bXz1mlGfkg4goixjnObeub/Hhhz9n5z++CvgKfPvYAAgwtuXrM6P4iIDojV/b61bASZNajPdoTxQMBl7XGyuFGnjw66mOEGUsstvfPQ9ufI13vvophvin+qNKBmeiiXsQaDuXW/gjDQ/O0fIeA+aXuiec7UTr09hoAwK6fp3OokwGkiCA4nCH+d4Ddnbu8ODuH3i+c9ezvbQEMiEKIgISpIxIkilRvkoQRVS197DO81WGq1NWV6fk+SrRG0biXUyAhYoRnZOBDMHWFoNFRgKpZOdraLFaYyQ4J9DLikovkEKyqBTJQlApwFhmT5+w8/gJO4/g2TNvz9X3WxfdQffD4BF0qc+gFn7QfRVnkwOjEWSJl0jkuf95tAG/XsDjHXi87dlbp7xN2S//AvJxTGU0RQG755igVhWYJWjr0M5inejSqt52MPHNChq4lgdsjBJGSUobhyTDFZL8gGECI31iIS/o+WIPhF3jmeD/cs7zek9fr9cT/eqz63mjdW/5Vp8H5UJ0ntSIjt3pZA9dKkNtDUtrGKaKfDwhGebk0+vMCs3T+494+ugRsXAMU0GeKYRKaZ3sQh4somsFD60FJxgNR9y6dYuV8TUQMS6I+fnta1y/tk6WetlPmkA2kMTKLzqttohAeJcG4a9BhNcPu8Ab9Nm2e0nWYo1FUp/JTnz7KuIXw7/hV7/+Sz74+Ycsnx+y2N+n0TeJHwikeodBlrC6OuT6u9dZWV0lyTIG6YA4SQjj8HhnAXxcutYQhiFfxzFKSrLBwK/QfqrjunFjixs3NllfX/cWjS3k0eBbH3e4X7C3+5xif0Y1nRGFU7JcsTYacWNri1jkaHdek1sPeP3n1NJSNTWnR/WAjIRvP4Y3qX5a7cc4ybGCDyUhUl06p5TIUBDJCCklzoF09ti+89vqJ9D7U/0xdQxuX8V6vmhNxsuB3stBLy9H0Gd+38kV2w6FSS9taF8Ax1Wxw97Opzy48zu++OQfmO2cRWlh0B8PJPmUZHUDGQ0piTBSUm+8CxHkee7/5BsGb1wM4ytcZ0rvoHFY7S2UjDS42HVODhXaWhAK4aCqZmi9QAKLI4HCgKkwleH+lw/58mOYPfODjDttccE3V9D9715nk/JGBls3YLwO+ejkFo98N26yBuNN7wxRGb+1XWlYPNHMDr3O5KWrdOejlStjqZxAW3dWBnJJKg19l3LQJbE1bYNuGrTxWtwWD3SnnDR7xAm4AHZm8M9802xoDfjgZzeZjEY+kvdUOaB96xlfLwsIA3HMjDrncA1e3N4xwhbro0ZbR9ZlWqlBSOYgWUlIioxIOmIVolIf7+fDtX1Mt8UhnAXrCJ0gjlPWpyGrE0Gc5cTDFdYnCaM8RjqNoEI6gZQSFSusNjgRIaRECEnQTcBCOP9hCcCFPjq4kwN49xD7FisdBvy3H/wH/urXf80vb3/I9clV9vB2LZNqgrM1gXwHlcTkKxnpMCfJMqLBgDiJieJefwl1o3GNwzUtta69/rdpMdoQIH9i4F6otY010iz1p82iJrSO9dEqK0QcvWLEtdYyOzyknBUsZwVBlBGlllxEXJ2s8/6tm/zr3X8585grIifPxhSF5Rn7QEWLo3IDTk7OCAc032OkQ0+k9KC3v4X4+Uh1EgepJFJKwm4x7P3khbd35IQc+Kl+qgur4LytfvlqINe2x5LHs0Tp+YEW8ryItzMP6Ryc5AnbK1uJ7UKr/HP5x9fFAbau2H7wKXd+//ds3/0ts+fffNqme1wDLA/3KA73iFanyNUNVHqNJIlIkpQg8js9b9rwdiHAV8qODXPezsxojV44TGy8INk5rNVoKiwSYR2LakbVpbIUiwOwFaYsqA409z83fPkHr9dV53mYv0HduAFb12FtDfKRJB8l5JMYGUuEkowq2KgEsyPN84MFs2LBwYFvjCtm3hpNcQ7IdgAC4wTCWLQT3sT8Eo58UQxJ6OOWHS1N09I0mqbrj4POxzKEOIXR0C/45q2XiZznhvoXH97k9vu3ieOhv2xOvy8vhFq8nWXAOu8ZLUQfQuhv1jP7IlA4NMa2SOcjt52AOFYIIRmuDCnmQ1QAiZIMYg88XZ/05BzCeR5ZOB/+nA5i0mGOTFJG0zUm61NiDEoYzEJTzZcIQpTIiKIIrMFJ45lpKREORAOEzoPfLu1FIo+1+MI5fNzg21l/+5u/5X/9X/4nfnn7l0yG6wyIsWWF1UtcOyYQLVK+g4wVSZqQjVaIOqZXJTEqilCx8prp2mFMg7Hav+eNw1QGUxlkN4H82aZkvVABkvX1DbJkQAhUyyVR07KZj/hFfpN/LD5/6WMbaorZjMVhQX1YMkgrxKgmC0O2plPeu3HrDPC9IlNubG4xzsY8fXzAXrHD11j8Hl//iUhgwNecaLkv/jX78b1fhr8Ifo/ZXiWIlUSFEilDBN0i8jiy83s5vJ/qpwK+aV32+vUqiUAPoLvGuJcJfU9LeI7NeZUHv92/SyKCoOawnlMc7LBz92O++Pi3LF8ze7sBmsM9BlHE8Oo1ojQgyROiC9D3wkVJHfpJs9s6FdYh+m1Uq9EaKmeR1iJaCcayKCqqhZ9vZ8pi4wVaVpiFwxgfSJHH3q1h575vprqIGnXyhiQXxEmCHMQgYoSKUbEikwqZxMjYIAcLVDKj0jsw8411i8VZqUVfcQJSCpRUKBUjnYVLGAoAXuqg5zPmzhCIgHI+Z77foLuTVuFfbzzxIDkOoWp4KXBdA0ajMXGaepcJdzrKDU6MtN5m5OvAPGB+tI6KB6hBDI3D2QajDdYabGM7Z5P+4vTaWeMspna0rTcyFFIglERIgRMO6yy1s57pFb35kfMErfQLMq8hFF5H74z3usYghEOKFiFdJylyGNuBbgdN45/ftQ4h+52Z46PrFmzu5Oe3rP77v/0P/Dd/83d8+PMPmE7WCaykqTTG1BhjsM74picZIpUkiCPCKEapGNFnIHd71aLvp3QOaxyttWgHaEtTGexlXKV+jzXJV0kGA1zraJZLbLGAZc3AwWSQMSwE85cA0ACoF0uWRUG5f0g0GBIlGdQNgzBilJ910bi2ucnNzauspyPCqma3OJ1AGZ762gD739tIcZql7b+eBr5CejmflBIZKgIhj0FvT370/+F+Ynx/qoutN4N932SKkS/8tneDeBU0tND21HFvQdY9vcT6lDZb0dqa54/vsHP3Y3YefHQG9J6ea168RjIgSiDfeI/hzQ9ZvfYL8ukGebKKjKILyXi9GGM3pxG9ltX6CUaAB751hcGha40wPrbNakdRVCzmYENQAowCLR1Wd7HCSeeRmMGTA3hYvLn8LqBzgMggyxRqqJBK4YREiBghY2KVo8iQsUMmFSLOePa8QMgKa09S3l6Mi04ykJFvdMjijEJXnn2zl3DoM1DN93F6RtuALjTzHWgKz9TGAoadLlp0b4QuAX3+hTkdBgzznDCOfMaFo0Mhvvr38u1/pyyLw0OGKznOWoRzOGOojQe91rpTC50T2scsa6rKYrrtGSEFUgkfl4vBtQbjDI4GJxwEnp1tpCCSAqEChAQhLKBxtsTYCpwHviL0EzLCN7SZusZ2Fl6eSbZdJLjtGKluYu7kFX1P29sGfH/zwX/H//6//R9cv/ouN67eQn4tqWYL5mXJUi9YmgXaaLTVvOMESK9nVipEdkltwMlHIUG8A+Bojfc5xoAVFr1saNrLFeLxfVc+GqHCiNY66kVFs1jQLpZELeRhxDUmfHYOJdE3gzV1zbJYsDgoiJIZUZbROEckAkbZ2Qa5G1tb3FxfZzPL4WDG4/shNRYIEUSUPgsSv5/0/Zjag5e59JrI0+fCsc5X0HnTK0IhfaOR6DyiT/3/bbuWfqqf6nSdtio7kT98C9PbVds99uQq7LW93W9sS9vW2Lpg78Gn3Pno79m5e+KJNcHboPrWNr+ns8AD3o0Ipu/CcOuvSTZ+ST79GdHa+wSR8vah9hTQfoO6EODrjO22hLrLvnN4QDic05jaoY33GnMV2MpQFJpq4eOLBeASEIkHwaNV3zjgV9awcQU2C7h7AcdqbZ+s430WrTUI61kggey2iGPPygmfg7y2tklRPCTLDUrg9ZoCFs6D8QAPfOlsbKTwYQfWOKz5/gbp76PW8ELzpmxo24ZGg55DOYNG+89HpmayoAAAIABJREFUpL4hcDzptNWFZ4F7ve8acEbG07ZUWntHBwLv6tDvIZ7c6Qd6hW9WX892KY9WyPMVnDUYbaiEn5ZdJ0bvr4OeWawqzXyu0aY5ZpScEDgc2nrWUroa5YyPeRJAJJDQnU8doxQ4wOKcAadxNKdWmQ7nmi4y3GJqzzIb67BWgGi8hEK0CGTHfPrHif6o36LZ+vrwQ/7m13/LL3/+G1KVoQgxVUM91yxnC3S5xCxr6sqga4NUqruuBUJIpFRnOjQc3sXCNB1D3zRYY3GiRSCw2mDx4TqX40z8/isKIwIBtmmo6yXNYomrlwjbMAAmYsCaEzx/YbngpeQtLBvcssZWNW1d45bdPpmDUJ4EUATA1uZVtqZT1qOIejTgfZmBXWIRx7trhobvE/T2dfrzP7070imDThheAnCdXOlFQa/7ie79wWq8Agd/Hukzxy1j9vz+rnMjirs6Briv0PJ+G+j1f5wO657cN6CFtqatamxZMC92KIvnbN/5Pdt3Pj1DWp44OZ0sMgfABvDee7Dx4a9Ipr8kWvsVUb4F6TXvEWHb7uubjwEXAnwL3fviOroGcpQEqQy4CmctRjvcwmELgy40xZ5PUuub15SAcQ5rY8je893qVQGz53Cz9DID89BH5n7XavEev+NVGCWWRB2glCQ2CqUUcRwjMDhRIYlJpEQmI3526wOyZINF9RHPiwUL7YMxrPPHpZRvkKtqR2x1B6gtxhiMvlyj38+BSQpt6ZOHy7n/HOazDuCPYTSByRWYToZordmZNYT4hreNIfy7OdwDPsGf2PcqiD+7i9YwnU4Zj4b+j/VvTXA5+F5fBxxtP2WyMqYdjqjsAr00ngXqmFfRLXyMcSzQ7O3PKBalPydah3TOs7/C4hYl0lZEWJSzWCVwSvhdCOnVhmHg2Uyv/a27ZMQasIgQHyqHpbEGrQ3LSlNVNWVlqKzDOIGUDoXzZvtdA05j8cEOx6D37UC+fzn9H/i7f/d3/OzWB4hCMF8esV/teTnCUqOLJdWsoio0i/mS+awEJFJpkoEfSgPhXRxEt8iyxlBXJcXBAfWy9kC3f+1IjDE4WiLE96YfvWzVNA113dDUS+pliK1LmuUCV9WIumHgYERIQ81p2OFBYk1QW6Tzk0zQgmid7xmo6zMSsA+uvs/Pb9/mxiAhryvCSYZ8f50bj1t2l45dt6Sl5vAHVF+3nCSrdRDXfxUA3oWl6TverEBKcNb5xFJru6+XZ1S7lBWkrN64wsq1Cff/4+9+7KP5Yaq19PDXg99T4PM4tAJehHbyhUAL+aJWNvgWecPZu3q5A4C0BLRE1lJVzyn2djjcecz23U/ZuXuHOx/9wzd26p/jA2/634f4PICNDfjFr1bZ+PCvaPNfYaP3qMkxJNjOHg3aNzV0AC4I+FrTeWN2HT9CdMwgDqzGOoe0EmscRht05XW/xgAWb3HW+G2kJIG1dcnaWkbxfIYUsFn5+z57Do9eUxz9sjooYHYIs7GXPMTK2znF2YLYJL7BpbM9ki4GoRiNxjiRsLG5w8bml8wWPhrZvuB1U1iQukIsKiqtMda91Lz8ba2bYxgnfqIyjWd5226+EcLLT9K0a2yjpW0bj1sb/5g0gI0YYu1tzZ7htzT2dvaIw5AkFAzTkIDAgxLgRCN0SaYJ85TF4TXyfBWjLCKKiVSIiqWf6CUY4yjLkmVpmc2OmM+1P8GFwFrRpb857NKAMSirUTRYFWIHsju3JMgA1QqsA9v6HQqchcZ1s7HodL0+Elwvl1RLQzXXLOYagwfRsmu6kQJCCTIQniw+Q/P++O9/xlVW41USMpqjhqd3tjG6odEGGss7DlpjaMqaRmuWiwVH8wKEQEUxad7gnPO6XwfuHceybnBLQ3FQ8Hx3B2vssQ7znQ7WtI3BYVFIAsxPrC9Q1zVNXWObBtc0/vu6obU1NBZBg6Qh4qzTjQVKSvaXTyj21qmnEw92bUvTWsrFgkVRoIgw1FzbusbW5iYTHIP9GiIBkwFJnTEoamRRs7CCA+CH5PXOc/douyZUZx1OeAZKhN3P0kuHjm+veO6f3EMuoNoS2pY0jX/sI/lBq22/4Vh2AnqBl4HeE8T4TVb4dUHvSXnA6304C6riOYd7D9h7fJedB1+wfecO23fvfMPdCfwscxoM9/s4QQBJukqSTzHJFOSY2qpORwwXudtzMVIH8AC263KPY9+cJqzFCoOzovOz9e4OtqYzzPcTsXIcMwNKgorHxKMrODLgGcYaBLDzDD6582aD32LhI4wPCkjik1ucFsQqRsXeTkrI7JiVkDImTjLG4zEb1xPErIIjH5aB8h9HZUFr0IWjeP6MotOkXSYb3/8KeG8DJqrT7p6qOPa3pPtKC2ZW0Szp8y+g8mNR2Hj2d4IHvhF+hbf/6AmjFPI0JkpzVBAhCGgJcMGLqum3uQz7u0/JshyVriCT3jO3c2dowdiKqjK4BvZnJXppkYMBKkmwjUAb3xxrKofTBpYlwlZkcUyWKnTiMChc0KIUDIzDaIsJHaJxpyXSWOfQeokD5osFi0XJfFGxWJQgFSL2AmAhBIEMUKFECYnBA0Bne23rjw98LZadR7t8wh94urrN0/VHKKmIpCJWIVGoEICuKkxdU5RzDp/vAw4Vh6SLlFpXGKO6a8+xWBxRlgWHz/fZfrCNFJIsyxgob5MlEBhdY7EoQlZwzLF/9u4Oy+WS2ta0zgeeNE1D0xhcU+PqGsfS2/C98DgDPAYWzNi8+xXTUU44GpFvrlM2hr39XXb3npAPUool3Hh3i/X1KXm5gMM9MuG81CcLkQEMohB7EFHXDZ+zRNNNID9wOTwBE0mD6SRICAgagQttJ6NztM552Y375ojWJ8D1zhGWb/ee//OuFP/unA96Dh8+JN18daDKn3oFpxvSXoHovLLhPIj7ejDw2Hq/SxP2C4+KYu8BOw8+Yufup76R7fEDDp9/09L0VVXhsxCqEqo6oE0ibKBoibzn/KmjvAiUcEHNbZ6EMhUIAyIDlXnQZzFdmINEdBZPvURD9sAXzxArgc88T8bE+SZS5MQiw7X3kULzeBc23hT4VjBbwMGhT9/RMZgYskQTq4LMWaSwyG7F7oT0cohEkU9HbGxdwcqHaJz3F078AFcdwYGGxRwS6bCqY7IvEfB9V8D7GyGiaQgbj2XBn+hhZ12W5t7FgQbqjhFG+5/bLpQ+wA9XY+Aq/qTWePBb7jxBj0aEQEv3ZCY48cC+JNXqXY5mUwZOkIgYcAgpaKzz6YXOea3t0rGoDNYKMhGhYknjJKILO1keGcyixJYFTs/J0wHDJCZbcWinQAhiFZIoiZaWZQlRp0UVEvy+isM6jW0si7JkPl94CcCRRqYwUGnn2et1Xd59RHa7MwIj8IlxP+o76qvGsMsu9pFh+ChjOxoyGo1YXVlhmGcM0xSlFPP5EeWiYD47ZL/dwx04ZKRIhxnVsiSofHtS61r2956yt7/L/rM9th8+JR2kTNemMBzxDgLh3sFojcMSoRigkJTs/5kzv4vFgrryjK8Hc43XRjcW19Q0LI8bVF6sqrs95D7XHk/JNqdM6yXLesn+wR57+7uMRjlr6xvc2NpiOp0SBVCHAoRDRTDIJINIkg+gDhzVoWOvDHjSOfn+ENUP3/1Gq7FQG1ChwUqB6DvcO8tA1y0S3Es0vgkn0ceq01EY6+e0+Q/yit72SoGV7hbh3619PIVy/sx/tL//Qx3c21uvuf//XUHv2eqcG+oK6j2qvXs8+Je/Z/ve79i+0zL7Ds/o8MB3XrXktYRWYUk7y7QA2n6kuZgR+WLszKyXO5gD0HtQDaCIYZT4m1MaIQVxl2aG8KyhGvkmsXEM46FnE5WUCGKcy3BSQAzJqmbME65cN2xuwefnJKe9bi2sjxaeFZBEHvQa5XW61s0YGccIiGMJMkZIe9zVkCQJ4/GImT5AVQUYcLEfEK3xrL8GbI8YL5mf4yiHSAVed9J4y7JejZCk/jNLQ59R32oPek3pm99aA6GC2Hnmu8QD4CF+6KL/fgbt/U/Qyec06RCRDn16SJziwuRHed3frSoWh4cgE9TA+M95aY6b3LTWVNWS5dIiiIkGQ+JBxnT9OkZrluUhB7tH3P2Xz2kPtvGDesU7KBIZs5KvMFqfcuvGFYS7SiYlAxyxc1hqJNbriZUgkNAKizWGRVEwOzhkvqjQxhB3qYq9q5e31fAaRNE1tkkhsL0H6Y9cX3PEIaAQGJbU9YJy94jZQUYSR0SxQkg4Kg6Zlwc8ZYddnnLQ7DF7eMhieYSMYThewVpLY2q2nz7i6fY2daVxpkWsgbOrvOMcXztHa713svTtrcjjrnzHEfbPlo2rXMnu3i6LosQOsr5lk7auqe2CJTUl/lp/2bb9V0C6+3vcnQGDq2OK0LFsCsLIcXNrk9FkytbVddIkJSwXtIMBJAMYJISNYYAkd4Jl6Kil48uy4MmP8In4ttKu10SAkT5i3PeWClwXYNHiiZC+ES50p6xO8b/vZPleby89+dM7gopTf8+d+ruWPwdpxBWINhHpJpNskzSKyeMVnm5/wd7BHeAL/Nl2tuYPn33jd3/KFQRBh3N9mMXr9KOd1GngGHDCpH/7kwRBgOwf39YUe9sc7nzC47u/5cGd33K4TefDclJ/zPK0qGFvryIpapJV2YVgtLS2Jej0vT4W7s1h64UBXzRU+zC7f3JBmzG4KYgckkSjUP7+PfCNIQ9hrGCUeQbWbx8pLBkIiUgkiXCoRLLx7D4bW5bp4+/u61vhge/BzDtHmBh01OsywbkCoQSOzstXemspIQRxFjOajMkWz1EHhd/ajrvBKQZT+20ryo699oTdpalJAnEY0hhNC4SxZ3njGMaThDiAoK1Bt5TaA9ymgrL074Hq3gu0XwD0zK/As/o929HugKaFeEYwnCFGj3DplDaZ/kiv/LtVXc2Q5SqDFW+LJRRoXaHNkvm8oDw8Am1hOGVFpsRxztr0OrP9PY4O5+zuFrQHn+ENXXx9jaa0mvJgxpODQ6xumGZD1leGxM6RuMaDXmeQA9lFo/rpURvN0XzB7HDGYm4x2qIS38AlRdBZyFm/Od07UHiCDfkWNbe1HDFDIRGUCGZExI1CNRIx9418M/aZM6foVPQLDtnnkHLvkPgLyXC8gtGaslry6OFXPOQeEsmUddJqgLMN4HDW29DhQPlYD0IkIYoYRUTBo3OMFEM8H1Xzpx148Xxvj3JR0IxGx+t4Z2tqCpb47cxX9TE8xMtXoi8+Z/rBJk0eUdcLwhBubK1z8933ubaxSZZmsChokohgECGSiIFLESKCIGIZOhrZ8v/df/hGr+cKfuqveD17zBcV8AY/T1gJNrR+wWgtUkoa6wNr+vD1UxbSPmWVEy5EiZMUONuFNAnnx8nTikbDiRSid/j+k0zVDiYw2WRj/T02x79gc3KblTRlvLLCZ5+mfPqx5vGzZ5wHfP88V6anE9xeDeNeHlncvjL6WEoPN3utgZSda4StwdYc7j3m8Z3/zMNP/oHte/5jkPg5P8GPjzu8/vlaAc8PS1YPa6KtgKAfYWXP9tYdTn9LgO/iGZgFzB5Cc7/rdlVew5jFHhRa6YW8QoCKOnmD8xd+Fns7sDgDGedAjDHC66a1Q5Ig4iuoXJOtPmaUwN53bHJb4kFv1hnaK9EB4AbG/YAmC0YuJiMnUQKFQiiFihOSLCHJE5IRqKK7v+0kDSH02QxOdB3+FyMm+d4rw9uTqTgGpwmaxoPe4ZBhPmI8nhAELc18HzPbwxWN1zQbTxAbOmZDn6wph5xYnKV40NuzHhVgtL8/+0C+B8lFxZT8UDXH6DmunmPFAGEFWi85Khcsi8rnXhsHKwqZ5gxWJqxO1znY32dn9zlHn37BadD7zSq489XnrK8OWVtJiKcxkyz10d/WkuFtu0LpaKwFC7paclRWaNNinMNhEaFDhr5nSIreJq07Z3s8/FbwvSdVdUvb/t2JUIRemIDwHhZY3JkmIQ3cYxfz8J+IHyZoNEsqnh8PvYYprnNwiYjjAVa0CGNQSqFKRUjgIXcggIiwlTxi+9xjjPHntubVn+JlrodPH/L48WPGoSRbLCnrgkVVsGBJw4njwelynHwmGX5bP8oGiEFEPsm5Eb1L3mzw/tZtrm7cYDpdJ4oGtIOEMBvhshw3G0BVIwKfiha2jtBZojd8PWvrV6F1GF2iFwuO8OD9PDb1nZc8h3O+t9Q2DoM5AceiZ2hP/LEVJzteMR4MHE8Jzu8U9j3SCj8XxaoD2afybHrfm7fpGr3QWtEQl9SU1O0RZd3LGiS7+0c82z/ATxSvWf2q9E+tgj504qxLw8vqpLENz5aeuVj7JVnP/J40v8n+q227HQsLVUlVF9QHzykOH7Nz53fsfvI7DreL44VdcOqW4DEAvD74tbalxt+Cc2UNLx7rd6sLgWUHX4CeQXOv+4UDNBw+8fZkbsUDRIe/qAUnF7RUHvDGGcR5jhyuYUWM1o7FwjArFuR5QpaNEaklGe0wHluPnL5jPQXccx9B3I8mMw2bBgoHhXFcsQVXxBXWEomSMVJlqKQithn56ohRNeaAA6pO4tBhBzpDCGTsb5cF+P6Pt8dcv5kTpwLhGlw7Y2U65crGdVaubJCPN2gbzXzvPrMWmmdPqDTMK9Ctd35o9An7FeA1vgEwBaZTCGK/wKhL2J57sOzo4otn3e1SlaapDqjNCkiDsIJ5WbE8PIJZ1WleFMSrpJNr5OvXGF+9xr/+4Q88+exL4J9e62/80z9/ztpoRJZc4913R5ijJUvriKVkkKYo4TClw9gFel5RFIdY5+ULhAYhLUq2flIVXuMrhEQI2y3YBKJ9W/je86vGnJnHegucMYqIkMeUNHgA84hDzg/OhoiI4XCFlZURKysjmtpitGGQpUSHA5SQDAYKqQKkFKQmY+Nom50XnqfBD54TIgSCGUsO+eMaOi5DGRyf/OEPRLZmXQrqYo/nxR4zuqAhPMYI8ddy/xn1AG8LmBJx9f0bjDfXmby/xbUkJEgHTLN1RoMR48kUGeWQLZHjKcvRmHpvjzZYEjYgaktVVtTFEkH7Ro4Iq2tX/PzjfBNzuVhQlUcsZkUnNvLj0te8+nqwzgPTfrKTVqKU8laWOGzjgW8sIO0at0NOGF1BB3rdKeDb7YSmqaBpAeNw2h9QL3nQ5x3MZa4I3wUtS+Ahh4uacqF5sl0SyZRUrnD3098Bv+e1kOwN+J///S1+sf4eY73CSh3xZLfk2ZOS/+sf/5Gd8xjjS1SBDLx/7ikAK4MA27Znfj77GH9/GVgCeewE3DG5L/pARN3vJNjufJVAXXN4sMPhzj0e3/2E7XsfUdz9LYc7FTV+HDj+e6eeZQMfSnGv/vaxMQKiKEAGfpeo19UHREhqAiLsBYDes6/3DUo/9yEG36gKqgNYJBBbfACE9Be3XnhtqOvov4UDHStGA0UsBYl0VMahjUUZUE4g44zx2hobN3Z45/GbaZ528fZj/UC0tvCM78GhZzErU2FcgRMaNXLkiTymcIVSqCRBqAOs7twsJJCcDGoq9mz2ZQG+IowJVIKIA2IaCFvSfEiUpgRCgWtoG40pNVpXmI7tbTu9eUjXrIHX+U7wfWvxEIYTSCaeDS9nMGshnnsGxHGy5gxfcmxvdblDTLXmz+MoxpjO3uR4S0aBylDJCirJCJMEJ8J+1nytqtnjYLZAWxAqxckQ1zG0fTBDrTXV0ZKy0uilpQ36kIw+SrFrM+8e5cMsThZq7hI1FsKJBY7G4HyUx2uVRSCkRKoBKh5gbe13ogKFRDGIIrIsI4okUgqsNUyPpmj2vrEuq4CUmsRnPaLwoOdPDpyAb9gaCC9BmAzIdmc0cHzrz6x+K38IjICbrLN1c4tbt99nurXJaDolyAbIwYBRNCGLcsJo4NvVQonLMtp8lSpNWe5L6kVJfVDwsKh5UjQ0QMbLo5K/rfaLBUpIhnFMNhgiZcwwz0mGM7KjfaqFpnJnIZY49fUYDDtvqdpjXyc8++Fwne1g9350rkU9o3ta24vzY2j3FP7xzr/X/ffWnUgeNJdU53sDr3k5XZugbkMkoa79jjFHNdQPMYCVLa6MoIy6B78GYA1gNYB0HyJqUiwrRNTWp43dSFN2yssNfL8NTpwLel/2uE5iA96crL/XMXMrgdpgq4L54WPufPxbtu/+oYsgvsMprH2sY+8lORUeyK4BeQTJS4Cv4AQkDxNI8pwg8qIzzgRVSMAgT1TGb1QXAstMwUvZuuK51+4qA7n0QQ+JgsUBzA68qH/HwdoSCgVXFIyFwCnQxqEbiJ1PoJJKsbZxhRu3Ksb/UPwxGx/n1rK7vcPJAFTM4PlzmB2BsQdYCkabjmwscUgPGbp0NycExjm084y2UJ1LRa9hvkQa36OqwRAQhClxCGEakyYpKgCaknKm0UXBfH+H+f6MqoSm8TIH4TpmI4dkCCqFcAjpBMJhCIS0QUA1n9O2oEoPensTmhb/8+V0YzRU5YJYCIRKfFKfhVOblwSDIXGaIwepD6WQ6rW7cPs6mvvIbBkniEiB9m4OTWNwpqY6qlgcLqjKkmVHNIvISx2cazpNr49Z9t03HjiI0/+7ZPuoS/rGqtdfRLRYwEfOKhn7mPSl8QvaQBEnGSsrKySx4h0haK3mymCKXWoc8zOShoLelsowQLHCENG5QRxc6Cv9cSvLc2Q2QOUhCMdoc4qra5gdYPET2pKT7fiQLn4Uwe1f3+b2hx+y/uH7TLeuMZhMEFFIEEYMZE4UDoCQpYNWhLgsp86HLLOcPRHxvHjK3uPH7NXwvHLUQMroO+9OPHq6R6wk66tTksGQeJgQK0k2HKJXR+hywbI6YnE081peL98/PsNOXyrOQWM5QcTOeia4l0J0D1KcMGeq4086U5Vji7Rev2sN1N3uqLV+fdw7Rl5K0Ase+B51tyvAJtz8GXzwiyvUdcmz7ZKHT2Be0wlFH+IC6y/usua191Fa0E/gq/Ir2vSAIznmIF5h/6jkYL+mLP90m+BOA97zAixOqoeoHbw9jgGG1ueqApJA4l1W2oLq8DE7dz/iP/+//yf37nx0RrYg8GxvgP/o+n9r8J9aAIzt+Ucj8MBY4sHveBXy1VVPtp2nPQ5ewMJvUBcCfL9+RXfDfNaxoBbyBMYZkPhGsFmBjy3WUAlwqw4yhxMapO9K18agjEVZi1CS0XTMxg3DOC/YvyBh3df4Y3C2A74ziA+gMgXaPWPj9oJsTVMZg3HOr8AdWOeBr+0ZX3XiTSxlD3ovB/J9tLPHXE9pVUwoBEOVEochQoS0bYsr5+jZPtVsj3Lm3y8C3wAXAsOhT3Qbjj3gTSZj1HBEmKaU8znzskDPfUaFcJCKs3kVcQxRApcRMbRaY2SMjC1f1/YUm+uF31Ion+aGoHH47wfpHynt8PSsCKT37TWaxdz6mONaszg4ZD47QlcaYzpcLcCJFtezva5rajv+no4FPvX1EtV3WflrWkxv3dZFzVrngzyUComThDTLyQaRt6WzCdPplHbX4mqHozwzDc/peX2DZMgKKTEphxxeSqASIMnGqxwdeG/kK+mYLE8Jo5AgihACRlenhCJAPI6Qu0+PVUo9Cz/A2xhev7rFzQ/f572/+Dn55ibZZILMcpCCIAgJkQgnqBtHQ+1PxyjCZjl1vso8SXhsG+7t73PoAubAnBRBRvQdGYXSFpSdC1GiYlbEEBUnyDgnUzFxkpLojEGSsqzm6LJCa4s2LzC+1qd3iu41SwGhgLpnabtbz/oKTprZQgmy6wc5vuw68NungTrn+2L75rbLbKu3sQI7KX7r/D345a/gF7ev8MGH71Es9vki/QJin/a5jPzCIIpLlnUJ5R+HdJY1/PMz+IojrnDEGL9APQKevMbjBSeEzLF7cNrN75cmFfksrDvLAnvxwMn9JO2xKlces8CtrajrksOnd3n+4FPufPwP3Lnz0Td21hwnqtPzdLxzfC/cebWGl0L0+Hs4hijPiVTiE+lOMdLggbmU7Rmm+bvWhQDfIO6S9F4CgA87oJTEkHe2ZYXxMsiq8myr1DCqDEVRgXNoXXlrKFEhlEAqiZSSOEsYrY1Z24QvLrCjxFWgCzh4Dk8O/Af6fAYHi3tsvPceMosxVGgMlTYUi4qFtj6B6xQD0A9kpl/xi8sBKL7UsLM/o2k3II6J44BQBASBoG2bzp5rjq4adNmJ14cQjiCPYTiC0RTiPEaECYQpTVOh9+fM9vaY78NsH+YH0BTeHi0J/LkTelKYQHEpgS+296YWoJtO29trkRx26SUi1bxisahACILxmPbplNfzJxFcvXaFLEtxWOaHh+xub7OQjkMF0jZYbdBHVRe96w/Fb6mGx0b74CdbIfpoVa9FPP13/tSrpKAsS3RVobWhaRqsMQgkoYyJwoh4MCDOEkLpWfXVeh2kJNyPSY922WH3eBKtOVETO/YJSQnJmCJ5doFJQz9U5aNV1jeuMs5HLJdLtqZTBlmGnyAFUTRgsLHJZHXKcDRmPJ1SFgVFMfM7C8AgTdjausG1rRtsvX+TfDohzAa0CGhBdOk4PrrYUoewPPb5amkGCW4yphmP2YsivnKOEkcBFFgKGtwbCqOO7AH/5asF090pV6Zj0iQiGShvpxkPyZQiTnNsVqGrOeV8gTXmWJdr4MwOSe/e0LPD9tQ/y07qIDs9iBCglF+ZisBQS9CyI1/wbjmGE+eJyzGDvLxkL06WcGsT/vpXt7n93ia3bm9ydLQClOQrJZubJYsyIJYTJBMebu7z+e/34Ys/vkvt5Sr/V5fjHD/l76KOiOgTdt+iCsCeBsHyhOWVXpUfBQAlxc5jqr173Pv4t9z5/X/i4affBL19OV7OyVugaL/ZkpUB70dw7V0f/lW2EOUBgcy7YwkIOruTY9WgfcsYX5X4FLNXMb+l9slrV9ZgNIJC+4ayRacVVRUUpSUpFuiqYjYTyNihEodp6e4mAAAgAElEQVSMJSpVJDL3CWpTydqGZOUze2GLMIsHvrMDvzJsge0Z7HwE7/3FQ7KrOTJxiBgWWjNbVCyqk1X9aeB7TKydoQje/nqyP0cTgIiJ4/DYw7dtHY3WmLKkLEBXEISQKv9ZTjdgZTImn14hTIfoUqN1Q7WzTTGbM9uGg23PpjeVf2/zDYjHEKVeGtGCb+j47Ed8A75ztXxt8bSA6RVOFi/eaGh1jakqqnlJMa8QQjKZTHjGBq8DfCMx5erVdYbDBFrH4mjGs6dPUY1GsvRbqYGEBkw3UwoBoZRIIX14BT6iGOc8QLEK19jO1ivooosv+xT77VVQUi4WLKsaozVGG1pjO8ZXEccJSZwwSFJUJPFYzREPYrI8ZVKsou4JNE+P12g1vt/cAhNqMsasM0WywLxF4Ffhdbev2vAd5SM2r276vYpAMJAhaRSB8NZGIhowmmSsDgasTaeUWzOWRUEx8zBDRgFZnnJ1c4v1q1vkG1vkozEuSmhEAM4du4iY2kDdsggcS+EQA0mYRbgkhcmEejLlIBpwH3csS6toaIW9kHG1wfCk2uboQcl0vMradMTqypAkjokjAY0Ba6iKIxI1Yz4/opgVx7pb+Ka2uQe+p3/fSx2OG9kCkFIhpW+Ii5UhXBpKoOqsIEs8mOif5zKXtBx7Xd3ejPivf/VvuXFjzOaNCUdHKZKS69c0pYXaSrARtDFXrkRICf969OTVJ+3bVLdh9W//GilrFnU/Krwl1cKJ6W/QMacBbSAJkMggIoos1FAdbPPgk9/x8T/+P3x+784b/dk5Z9cOGXANeO9ncONd2C28S1eUjCHKgYiAqBNldODXdk4TvRHFG9aFAN/NDzzJtXgOsx1oF5z7eRcGPnsIz2Z+VWtzH5KQdOTYkyeaZ88e++hH0/ngKrhx6zFbN8esrY3JRyP/txb2Qk6pdXzn8TjxOtUXF2n7wD///g4udmRXEvK1hPtPdvjyyxmP971Ew0gQQ5AJvmGhO7DLNmh9/MWcz764SzMKYBQccypOa4rZM+ZFRaW9XV08hHwVxuOE9WvXiOOUIAxpyob53g7z/T32t73URWs/AYwmEG74xr94AmHuAbQIQ7QDrZtLmmFfnexPur7/uuRY2PbVhG3ZMa0Y9p5u0xgNr2nO9Ktf/xu2rl3FWcPdu5/z1Ref8eDOA6+Tlj6IJYslUigv35XeMjCOY+IkJk4GXfBDD4Lpwitcp1lsTwVc/GlXhWV3vsvTp9vEKHRt0Nq3zQ/TIcN8hXS0ynAlQ8YBQgqcEgRpwurVdcTXMFqbMPziE748/Iw9PCAL6BuQLBUFIBggiC4oHPNNKsA3m+b4bdwlL08JW1+fsj6dIgK/WIrw01BIS72wFPXSMy9NSxRF5JtXyTfWWGv8oBcIgYwG5KurhKMRLhqw6JSDwglc09K4Gtqa5d4+dbnERZImCghJIVmFKCQajck2rpJuvUu+dZPyyR5zV/qjd+JCuzFLZpQHM+4fACiGaoXpZMQoVgxj7yct1YjBUOGQaD3DVO64h/X0GN//fNp+rMaT2caC7dCslLZzVpEI4Wisoar8htcel3EMfHnd/xjvbX8DVuJfILnO/j7oUnNwVPPVPU2pa08+tUcU+yVHR0eUpSRFMt2U7B3Zy2FP9gUcHv3Of/+WHG/v+CCDgLb1YCsIIAhaZFtjTU19WPG8OGTnwSfsPP6Cex/9PV/d+fiN/3aDfxu28Frg1QjWNjxZtjrtfmm7Jvn8XZL8BlGycaLxbW3XztajsoCLQL4XMipfud7N+de9rdmzXXj2DL7u42y7pW9poXoGOweQZZDlHmwmzrOIzw9gsfCr3l4v8g5eC6ztAZXWrGmNcBJdvTmovAbcBG6OwMX+GM8D03e+hHjtS67oNdZYY+fZAU+2YWcGX3fJDIOwC3CwJzd7mVAv8AB49PAJcQlJCXHgB+xGa+aFB7C68XOOSmE4GjG6ssFoep0gAK01Zl6iZzNm234RNN/z4DaIvTQineA9kPMhIvbxFk5AoBsQJYrqbRkv/ojS4MxJ4xgtJye+AfeI+pFiexCBEujF3K/shHqNk1hx69a7TKYT9MEDnu4+4un2I3YfedBrFLQpiNwSD/xf9myvIB4oBgNFHMVIGXkbs6DTHAp3DIDpksrEJXN2+K61zyGH+/sM1ZDWOqz5GiEhSTKS1N/iLEVFCpTACIdMEoaDiDROSAYRQSho/qBxR/fxalg/dviBfokkIjneAP9xagUPdgfdLcMvtV4GfFdlzurqiNFk5HcIREhoW0RjoW6oqxpXtzgEDYLpOGc0WWUQSkLRO3QLCEOiwYAwGeBEiMFv84eu7wFraJqG4mCf8miOyBRkCQwEYZsgo4gwy4nGE7L1dQbrm7T7S1j2vNGS788DxjA3e8yfzpnEE65ORqzEijxNUSLCOb8zYu3MX/acBb+nAe/pm+ZkupYa0trilENJb9HUN9Ed8acFeoFjtnZlJSKKbiBZpzw64uBZyf5+zZOHFmstUlrKsuTJ9hMePoQ0g5WVFa5OUtKfSe5//JZFE69wvu73LWWnuxwK5LG1ryUIWmx1SHW4w+HOAz7+T/83X3z8W5bVxTqTrwHvvQtb78LaFuRTKEqoavygZFYJojXk8CqqY31pz4O5b1NksTxxS1IZjFrP1lp7Fgj2fTVCQDby2+QxoCy4Ay99qMRZr8KvgYMKnh+BFRXPnlWYJdy7/2YJNhF+EM4kbKxBvAYyh/vP4asXBClHFTx8CJV7zsIteHagqbr7hH2XLp03sT3R+V42Bq0Gdvc1VwLQncVC03v0Np0UARChjycO4xghBM61NLqinO0zn+2hdQMB5BNIh5x4lXUa+qqE+WyOa+e03UWoGy95uXygt68S57Sf4W3AWUT7BVQVyy8129SYckF7eAju/GCE0xXFV3E49g92mT3dZvbkHqZcMIz9uZsNfFONrqCqLFXXdAN0YNezvFIIZCCOfxYIj0S6s/QYt/wZVMGc+VFBmc0RXYNHnESEiSTNhqT5kHQ4RCbKM75SYJxhmKXkaU4rHEZYjpb76E9L5HLvuAEpBSLewauD3Y+246PwPtpJ933vqyvxIPi8OXtmC5qmQYaSbJAxGAygtrCsaZY1gXPQQhRFRDIiDBIQEYQRIvLe0IjQr3RlRCskgQi73HOJaOlMu4HGsdRLlmWBIEJgCDMJdY6IBkghGUQZgygnG4wIw6gb8AWx2CSKBt/zO6jZ19vsb28TosiJu6hhi0AfSxv697QHwDHfnFR7v/L+vkLw/7P3tjGSJPl53y8jIzOyMqururq3Z3pndmdvd+/2bkmetUeexCMpmQQpQjJM0LDgF1mwTfoDbcuAYBsyYJimIMIvtKEPgmQBpiTLJkHZkm2RAmTYEiSS4JkWBZ54Oq5u9265u9yXme2Z6Z6e7qquqszKyIyM8oeIrKqZndmdl57Z7d16gOrqysrKzIrKzHjiH8//+WOsddaHdkaRl4ynLtJ7du9/H46TE7hxTfP2Gyc0TY4xU65dO+K1b1+jaXLC0KBzzbUbcOMqZH3Y3j6hPHHe7x87fKyS3ZaevI0xixIUyDa5TUIY+jLATt2rx0P0eI/h3tvsv/0K+5dfY/jWy+jidM7CCHgOeCqE3Wdg9ymIezDM4eoY9vbd/7IHYW8Xme6i1ACpUgySxhiQrZtv+/djFPFdJNhKV7ShJ6Hbd4sWtiy565yNHx73dmDzCef5KksnDzBTJ/Kf35bNNyrh5himhdvG9BgOHqCARQZcSFwSQTue6SawuwWbzzrd6avvAneQtOztQWFhbEqmpYtKS7HM1BVwi7jrjOS0vQ+HIxhljpxaQPjM47Lxp5uAKIQoCp0OOA5pmpp8MmZyvM/kaEJdu3Oi72UNDRENoUuQG0M+cZHgcrykh5Xfx9lFga1nBNIyN3e6rK7CtGL2tnHCdjt0yz4QKecvnsdYw42DAw6vXObm5bdIKku345MKYzegPDlxmzUxiAzwmlUplSO9wpFeKV1RXqfpve04PyXEt8Qwtifk0ylKJsQyIZIbpFmX7sYG2Uaf7mYPqSKIBVYJImvobbrCFyIRBIngJD9kNj2ByxWlzqkxKAQJwhUx+Iho7wCn5fVxk4VXZltoouPfv73fngNWa5SM6PW6bA62sIVGj2fUYsqsBmxDRymUUgjZgVCBjBEqIpKOBCMiaAuohKFzbyBc3BOttVhTo2czptMpERppFaqX0WhNaCGKFCrK6HR6dNIeUjhZUEiXz5y/SLfXe/QN6VFTcbSSua1wbZzi+gCBD+jgCjQlbqLAXU5+NnM1FwTh7M6ErjDGMikMI+4uP/mkoDpquHYt5513jjBG05iSK2+f8PLLN2hKX2zNQOXF85NjmNxgmS24xodghfwuaKKTNUgZOtuk0Je00wV6fMhw7032Xvsab7/ym1x94/C+B153ywzZxLnYvfgMPPOMk0WmPRhruHITLl+HV96Am417/9JgB9QuYfYEoUyX8gdwtTTaruo0LB04JeJrfIi2jXTGiSOEi/cNlDHEneX0f3fLT3kbN+MbV5BWkAonl6xbmUQJYtNN6VbGkd/pA5DeDeAz5+Czl5wcY/9dF6Htdt0Pl6aOiD9xDrbfeH9xxKaEwxuOWFhv/C9i34CVt6/RgHdzWJQrPmPTx28VcOEI8k0QibuJR7jzTQA0EIcQRhFhFELTUOYT6rLAVqXXDkGUuohwHKfkeUGZlxQTp/nNR1AcOekEAKGfIjzTxLfC2hIpLSaWzKs73RIOYTzh3u/kfaytuXlwnWpoOLp6mdH1Mc/0odd1pLcbuUFKVbrrQuJkKEuiKwiFWJyHbdmLZTELaEfQZ3Ww9iCYMKOYzhCpJJaurVSiUGlK2u3SyTJk6hiMjQWxNXT6m2SbfXddS8v5w4sMb1yjmk0pjofM8jHSiQSQnvw+7ibt4/S8rbwhojXVW5YLVzhSfCeMxyO01jS2ccQfd144iYJ1fl1CYG3o5G01WIVbFgoa4WOhQgChT66MCHGvsY2z8apBz2ry8RRpI1St6eQ90NrvT9BRKYPeJuefPM/2zhbXRtCTPbYHXbKs+whb8YPR+pUmCBf4kILKGqy3EYxj6ChBLGNiYqrMYhuL8VXdYiWJE+ktDsHUI0Txyb/44mwLU4acHGkkxnsuZnTDPkaeoKQrdHWUgW0jvB/HSO+HwU37PGZj+rtEQo1BG0OBxjB2wzc9xuib3Hz7ZYZvfZ29b3+Nq28cMsQN5lI+2Dk5wlVjbUteHN62/jYu0vvcc/DMZ2B31x1Z0bgA5t51uHzVkV4LoHbobb1A2nuKMGyH63f7ji0ejiycCvFt5SBC+MQa6chv28dKHImSi9I1TtIVJ/6mKlyG/1YMYsdlpi/Mw60reEGCK99ofActuK/6oE9swqVL8B0vghnDbhfKY0il0xCf80lBT5yHz+7C0e01SnH7m06dnCNuyxFbR/xL483MWz7RdeTvrBSwaHEFZ+dWXIKNDJKNkDhWRITYxlVvo4EoiRxhbUrKckJT54gQkjQk6m0QJZsUeU6Vl4xulBzvuxF8nvtBDW56PfQzo5WFsIF7LsH1MYQ1JbEUJHHM5Phut4/7qeulOTq8gS1ukDQzqtExMwPPdGGw4QaJiXTXi/F+7xu0NmYSqXznKnCVt4R15AWNpfHct+1wP7pp+Y8CMwpmdUliOk4KImNiGZPECZ2kQ5p2kZ0IEkfoDIa0m5FsdCG2WGEY7Oywfe485cmY2IKsLLYxWGsJQ0nkLaseF/q4PredUm8VRpIlAf4w4nt0cMBofEx30KWqNbEQqG7kpBtaUaPBk9ZaSeq6Ia6dLZ7BODlUaB3ZFaCEIoqUo71WOEszLLUNmRWG6VSjrMHWmuqkwM401NYZ6KuYrXM7fObZZ7l+fY8re+/S62UoFTkLxI8QTsGfIqQkTgVUBcaUi59bCEkSpyRJRi90Pt7WD4VkHNNJYoQQGGORUnF4fJ0D7l3fu41LhDsLeuDdFy/RmJAs6ZMlGY0BmSgylXDh/CVoJJicRBowmrI8Itcn3DjI2bvCx8wS7Da0JZeV4ziXLl3g6UsX2N7OUOq+6xQ9GBbtE0IoPT10IdPGaHRRMM7HDMdjhsWQpjhE5/sML7/M8PLLjPf0oqdKufXe4KLFvnqgdkGv3V3Y2XE8qiog3Ydv4U79XVzu1Isvwgvf4b35VcrxsOBg6CK9b1+F/bG3RO3B7lMvsfvUd9EbXESqHo0bFrHwHF7EiNzrcCWy/aA4lZ+lyt1xOH9CIF3eZMH9I9oosPd2idsVrEssi6WLvgrDohyuxVuCeUJZTd37sQC56Qinvgncgw57cwueeNKRXwrYTGB8ANUIxmPnJVdZV1nuuc/C3g24egcmUFe+HpdPhitLH6EuWHIa/znjBwJnDVfxsoMoIu5tkiYZUeP8fOtyBLYgjF3PU9c1dTmBpnJJLUlMnG4R9TaZjHMmowmjQzh8zwU72yZNNyDJHOmN2iS6hjM932dtSSwTOp2YMulQl/cxMrsjRsymx8ymy9F16wfaTRJiDBHGXRuV0xIK3DUo49jZmbVzse4IsbTG058uons7phQUzMhMD+vJvxDS+R5HCXGiCJMYEkkiXMupLCXudhAxEEJ/Z4ftnfOUoxGyMoRFhS41pS6RSULcVYuo3qPEgFulDG2X0N5iW/Ib+oWqnZnj/dbrNw8PGB0fsTnYRNczZKSIpER1FLarEFh0ranrGj0T6GiGCK3bqK2xUQTWeG9uSRgKFiy1cbM6NRZTwzjXjMY5HS3o6JDe9swRX12DskgRstnr8dSFJ3nyyfM8+dQOCmdxJT/iszdhi7S7QZJIOh0XAReiXEymWARCRsQqJku6yLjjhpbCEquYJE38uWERUnI8GnJ0UN5TNdIB8IXPf57Ll99jr3yA6c/HjK985cucnJQYLcmyDCVDMtUl62dkWUY/2yaUhgQDUoMpKU3O7732JkfHrzH7WGlpV7AFnUuK8+e7bPXP0d/e5nPPPsvnnvsc585nZJkLQJwqWhL4Pu4nl8uBRmuMLijyMZPhTQ6Hh9w83mf/5j7oQ3R+lfH+G0z3bxU3ZMDAV2DNthzxVcolxk3GzkFo9yLs7sD4EIqhKzl9ebgsUbx7EXaegieeUkxyJ2+4OnR5WVdvOtI7AzYH8MRTX+SJi9/FYPd5VPoEkGJM6CUNrea1/aKN18o+/DT6qfwqC2cZ65LVixGYwpHZpJUE+Jq0RjpCOHUWicTCFbQwPhoszFJGIPDE10+1ydRFW20bDTYwPQezMS6T8ubdj7ESMJ7BjbEvnXzOjTZGN8AewM0CynfBxILP/oHPMjY3uPrbt5XWiqGz5QTawksvWuu1QMI84Rb38rk5W3ZmLQ6AOtvAZhu+NFuMiAQ0sestmwRERFVBaBuoG0IRI+IUDZzs75O/+Tb7b8HJvneDKN2gYmMHNrbdc5IlNE3jybNbJ3jvbEQx7gRjwEiLtZY4jqnLO1GL+8Hy7GlwWfl9II1AWkNVGqYTGA/dmjFOS7W906U3GJB0O8iOdLKb0HsGWus65sYukzCF8MltZ2x64iEwZsYRI9AR9mZIMuiT9Tcoq5raGCrTII1x9yMEUgpiKQmlJJQdZByyff4i06eH2NKQkBCRkE8nyMmYzkaPja0+Mo4f+Xdx5SWW3UHNMuEKPPkVTnqL8BFgC5l+/9k5Afb2rtDNOmz2eogsI1Kps2ZTAt1Icq0ptEZPZxxOD4mGIZ2xQmUZUb+H6m0SCUmkOtCBRjU0NWgBWs8Yz6aMjg65sr/H9et7bPYUm0axOd6knk5hPKaxlqgR9GqNFZbPpSmzzS3G4xzGQ+wj0ZCtuu626mgJdFHpBv005Yn+JucHG5zb7pHFMDm5wXR8SA3Y0ZDKeocGU6JnlvE4J0kKYuk6NCss6UaXDZ3R6/c4f/48/f426cYGF65f58033uH1sfnAxO2XvvsP8cM//Ef59muv8crLr/Lq1TcfQVucDjqXFJ978ZKv3uW04UmSEEqFDCXSl8eVOB2qlA3GOOPfxmxz5YrhrZOP2/e7xL/wx77M5168QJIYlJKoJKOrMra2t+nLLShB06AfgTi5vZWHsKiTZACMoSoKtB4z3L/K3lvf5ubhVYaH++jxmFA2qBB6WchubwPz1As0O0O0PqYYFxgDGz2XbCazEFLFeFxQjB3BHQ4d8dU4XqoLL+9UK9KHEIYaXnkL3hpqDg7heOwkDkeelEcKzu0qXnjpR3nm+e9hsPM8qvcEhBmmgYZlQlsDLsHNGBaZ8KfQpqczHPH9pfU6V1v7Sm3ehzduJV/SJ7tZR3KKwkVOZbpyk27vPWK56YXHaLy0DKuMM8YWqfOErXpgNkHf4H0R4FC6Ahvj0nkI2x5spu7YKMDkcFzB6CZs7vY4d+EpnhqlnP/Gyxys9gw9R7xlsgiaLTTLLVlfnakP2lTfM4YTICejDBOqMEYhIIwRYeO8s6KIBktZN8RNjWgaV/awCanrmuPDgtFVOHwXjiYu4pQkkG7Bzmdg59IW2eYuUdKjzMfk+YRyMoF8QkxzZjObrXWkt7YGGQuitEtdjDiN4U+IkzGcwxUOwVqqHMYnMJku68SlXUm3P6CzkRJ3YmQsWMgu242tkF5r7cJP9NOGE3IkKaJO2CkL6spgKoNpnL0SJlgOxIVYuGMgY6SK2dgasLl9Hj2eQWmxlUXIGDsP6G9usjnYJkoePfFd9TdolVarfuQCZ74QCicvsl7yre5yod3cP2C0tcVsewdd1zQdg40kSoCwhtpq6lqjdY6eFWgFWkd0tzfJIovqRAil3GyEr7JmQ7A0zNDks5zj6Yibx4dcPzzANj2EyJhNp9TjCXo8ItKaBkE0zcmM5gkh+Ey3y0GuGRcV9akmBMS44UMGpHTilI1enyRzJYw3+5v0+wM2u13OD7bpdVMG/QRMyXvvvs5771iiyQRE7Jw8rMWWYCrnEZ3MDJGMFyOTyvjBsUqQMibpJsg4YXOwjbGS6auv8d4HONw8+/zn+QMv/SEaGzM90Vy+epXJ+2pjfTzQ3+pz7mIPSFBkjuzKVsPpBpJShii/3AVIHfk9d6nkwoUD3nrtg4mv6H8Re/LwnrP3hi/yR378B/mBH/oyFy70MU3uXBSkQpKQKIWSEoyhnC6KHp8awtXnZmlNBoaGhlyPmRzfZP/6Zd5841WG1/eY3bwKaDZ7PQaDAT21wcXBkxA+SSihMJqiyCl0QZg2XmYEcWoYN/uMh/sMtSO0SkN43SUhoqEpllLXBlelbTwEPQR9GW7c4TJVg5Dd51/iue/4Xi4+8xJS9QhVRtMS28Z4Ztp+eDXMvfr/g+N0Ir4+vGBroGo7iiVptRUILwMorZMUVF4iIIX7H1hWPGstwVbzJFpZhPERX59QJqWrHGf9cZQJDPdw5Nd37k0MNwqwN1yzdVPoKh9InznJRCI8QTeG2JSI1HLuEhz8vv+SKQQ9t4/K9yyuJLEn0Lj/A+EivUHsju2sBtFeeWOfhA1SsU28mRKFLlzUUGPrmqYqqW1NSeOmTCcjinxEPoaJ9+9tSxv3dmBrF7YvwtbFLbKti1RlST66wuTogMmooczdYOiskl6AeVVjpMDU7rISIiSIE+bVg3dKGW5qcwvY6cG5AfRSl2hUFK4c+NRnjHcEqKxLnLqp1UXBikh6FwfciqHwmejWR3vFgth9mjBliiIhJmE205SVpqoqqrKmKiusACnmSBkSSuGnsS0Sd2EnnS4bg22qaeFG4tZV5BJCknY6dJLMl6Z9tIhY+sfi/199DUvS245vlkZ278fB9UOe7G0y7g1QozFCKkwYIuuasLZgGjJtiWpNVM9odAO1c/eNoohYRIjaQm1oOoXPJpcgI9Az6mLMbHzMwf4ee3vvINilo84zHo04vrZPWNe0Pr3NTGO1pjsec0koOlmPkbLUzYMOJnfwQ0SUGJB2U7KNlG53g26WkqVd0qxDt9ulkyQknQ69bpdutkGWdun1+2ykMWknpsiHTPIxBwfXSZI+cbzht11gjXEJzyUYUyLlspT4rK4oypxy5jrLXq9H0slQScr2znmeeabCvneVt8v35wN8Zutpzp1/mnRjkyzrEycbxCID+/EkvsjEmbg30hWfbb1kpVoWEaNBN86JwBgf/kXS3z7Hi1/8Etdu5Lz1ym/heu8LiH4fe3INMOx+7ku89KWXePONC7z18u/yqE10d7/4NOcubiGT1mkgI8SAkRhC506lDYSGO5r7PCTuTPcaJJqm0VT6JsPJHkVzjOqF7Kodws8MSFXG7hO77G5dpNfrMRjs+MGxQZsGXWkqNIT+gSaUBTLbJ0wPkenbTPRlpgW8ftM97vMgAeheDHnhO76XSy98L73BM6B6GBljGjdV7orJuQ04A4dlyeJmtZDFQ457T434Wgt25ohpe0NdEF+J8/G3rn9oJQLWOgmC8VFV6+s9LiKp3hnBCi8SbwtDmCXxTRNX/U0kQArVhiOixdQRqXnp9j2ZwqR0nsBJ7Ah3Il1UOo2d/CGNQBpDYgpEYjl3KeVZU3BsoRQguu67GC/pwC4T+lYj1tZn9j7m3JZTxbePYefqhN3NhM0sRSRuNGYraKqaqiyxZUkUgo0iJkM4vOaKVuQjV7Q3AZRwsobdF2DrwnNsbF8kjjbYf/fbHO9fY3TdfaYtjnG2UVFXLlHKuXoIEplAmjh1bVXSVOU9BQEULklgS8L2Jmz1Ybsfs9XrYoopNq8ocji2Ti+lcAPAJN0gSTJnZbYapVw9EQW+ZIV/KVp/3zNmQfKQKJkxYUqMoiwKKl1RlxVVVdFUlc+8jZFC+XLPbS6gQAo3vbmxtY3VFTRzhHU3A4tAyZA0TQkfQyS9/SXtyvPqozVUQDpNfRvtv9uvXQHHe3tMBpso2UEQYaxFzgoUIR3VIZWKyGriWlOFNXpWIUKLUopICCJTg64x6Z5edz4AACAASURBVAwjY2TaJcq6oAvsbEwxPuLwYI89crLDQ7Z6GZPhiJG67pIurHVk2lqiuqGjDb1QkXUFx/bBc2DP9b4TKSWbg212zj9BbzCg1+uzsdGl2+2SZhmdJKG7kZLGiiyJSTopaZKRJCmdTopMJDKGo+MbHBzuoy73iMdd4qQLVviIe0FVuE7bVCAqu/g9hCgR45KTkymlqTh3/hwXL15kZ+c8W1s7WBIMCYdvfuuWlIdLm8/w+S98J08++TTdjQFJ2iXppHSSLhQfXvr8o4BMFKFUNA00voKCMYaEkNB36k7a4B0uaJC4jKr+9jZf+FKf/vY5rnzlB1Eq4fz5LbJ+l5OjI0qt2d7aYmt7my9/6SWu/OEf4Jd+4W9C/uikEc++eIHtS9tI5cRFEknrkm0MaGPQeNLrCfxp4s7XrEsKbBiS6+uMx5fJ9Zg4hd7uLruDATs7u1zcfYHdnedQcos0fsJJMRoXKW6fTVjQkKP1EN0MCdNDVHoT1DNcPfw1ZsXegx/8AJ75/Jd57os/wHMvfJlQ7YLcAFbkDbJZnCe35LU1q3NYd22Ie8bpJLeV3tbGk1spHbFs9bzWOkeGBUG2LCxyRDsv1z77/1u3B+E1aa3VUpvc49IM3YK2gAZuFpLupntOKp/0U0A9dTpckzivYOkJqvX6GGucLZQ5Lpnad6kqgVEl3Qs+Im3Axssfog2O3cJrV4IQYuU7nkW8Azy3D1+4mLPlK42EYUhdOw+SxlejaOqauiwYjyAfuygvONLb95XaNrYhSlJsbZkc7kOzz+G77zI6dLZm+eTWAhlnF25UZkzlPTzdCRB3FHEco+JtpIwpiyn5yQmYig0pELbCFgZrXbslws1KDDZcdHcji9lMEza7G2z3uxwWBeMCJsUyF3Ajht5OjyTrEqnYSxeEJ14u89RVncIROFhEeqWURCI6uyfrQ2DCBInk5GTIyfEm08k2uigwdUaYSGcLh59waixNaTB+KmpunX5LRAlJ1sc8YTBznyhXG8JA3DLAeFRo9xBxlxmT2+9VwhWl+aDL7a18xuY//WfUvU3CTg8VKTJCukKRKegoMFajraYWhk5T05lqsnFON1KI2iJ0jZ3OsKGCboUoNGI8wx4eog8OOBhdpwDe02PSd94lHOfUB4c82evRjRSZUnSlIosUYe0s+maRYtY8+Kn6B7/vK0gp6aZd+oNtur0NNja6pGlCmmUkSYKKYxIVk0hBLCJiGXtbNkljLXVRYArD8dERJ+MT8qJ0xSgQCBm7xDtrkXKKMU465icHEUDHuqALSIqi4ubhCGtCRqOCycQwnRqslVzcfNZ9TsakG30uXnyaJy8+zdFRwe987WV+9xvf4pVXX2eveO8BW+PRI8u26fbPuwbQZiHJCqVEhu2060okQEqQilAm9KQky0K6/fM8/VwJoaTfz5BSkucXKMup+72UJMkUaivju7/yZb7x64+O+F567jm2zm0TZsrrT/0boTv2tjxEWzPi1LEg1Eu0RmYhIZnqMdjZJe0NwDxDL8sYDAYMBgNUtotJM6ePCB3ZNXJVPhC63wRQSoJO6e0MUPIpULtURvJW72VuvPb1+z7saBde/OL38twLL7G7+wxpNqAJXaEKaIO8ZvkdbwsOGeSC654GTTiVn6b0WZfLZBkgBhu5iG6b9NbKINpE81bOewvxXSGMkuW60m9f4j4oDAjNItV9dd9JBlK5DGJjnAal8OvJlIXHrhVOtmD98ckKxhXIoxEyBtmB9DwUJ96yDBbktiXbwh+zuQOhP+u+qO/VcDQu2C1rwrrxlZe8dMUKwjihyWvyEUwmzqi9aVwiTRzC9i5s7jq/5lAklJMx5fiEfNRwfNV5+lblkvTWZ7y92pFbXdWEFiwGISAVGZ0kpbe9zWCwA1ZT6xKFpSdB2hIzHWLyKWnkOkUlIY4tsRDEUtCNYgbdlM0s48BeZngCY+Mi6zGQ9GCws0PS3UDG8bI6G66iVkvAFk3cWEQkvddv5GURH0mjfaSYYzhhyJAh/aMNJuPzVDONNY1LagslQduWlcUUrXeGxHitJkISpRldC9YKRCCpTqaYslzemB4Dbovrv+9N6418rV3GGO6GOfC6tcjRMb3RMVtyi163x6YSdIxEaYm2NcoaatVghaWja3p5gYoEQmuYzTCiQ40k7BbQHcB0SnN4iDk44MDv6yYQz8bovTHjPaizHZ7a3iLb3EYNtuhlynd60SJf5EEDPl/5/u9HCEESJySdjDhJSNKEjoqJE7UguXEkkA0ILKEP1daVcUV4qpzpbMrBzavcODxiPJlSzirXPwmJjFIArxMvGeMi1AFuYBsBMk4RccysatDHJ5yMpoDA1DGVlcSdLhcvPc/m1g7bOzv0+lvESUIcd7h29TqvfPPbfOuVV3jj4Jsf8kt+tOhn59jafhpMg9GGxhgMxkVKb2EfBmMaJytSXRKZed1sxjkkK7oIwJDrHFNuucnxxiATher3+a4/+CW+8et/+5F8F/HsS1x69hL97a32iN1vLCVqodBQ7jm8/fudDppb0liXCAlRYcagt0MYZoQyRCmFVCmxLzhjQsUE45wHXIqac0BbbMXQLFwoUlTaQ4VgMugNnmGwc5EnnnqBV7MB73z9V+/peKMBPPPCUzzz/Hfx3AtfZLDzDCrdRSpF48prYEzj2ssXp1j9dqEvnBWaZmFz5vTND3fOn85P44ezUjqZQdzxtmbiVp2ZWHnIlgC3pHU1WiqWpDeG98/dtaTSyyvaAeMi3yHy225nITw5tQDeT1jEXhhee+lEgStEUbjjSbrQUy4pSxh/vF6m0R5fe6wLb2Gz1ClLseDkZxaHOE12g6BqICaCMIMk9JqHGhtVWOH0ZW1RlaTnIpXJhpOglDkUk2Oq3Dl+lJMl6Q39NKzwlmZn0gZjgaUQvancNEeDxaSZ032mPTZ3zrmIUGOQtiKjJDY5ZAJRJXTTiG4Sg7AYq8FYpLF0ACkUtobxScmN42W0NxHQu3CeJ55+mmwQE8eSWLXnoKuixSoJFgLhizZEMlpUdPu0osFQMKaYTKhmBaaqsK0Waw4Yi6HCWDdlHes5tmOp6pqmBKxExq5k5aaMSbOMaTpEn0z8NO6jRavXXZU63AJPeH0hNYx1lYM/TC5Q+G1JQJkRaqxQKvLXrEtcgxrRsQgpUNqg6hkdHYK1iNowowCrENrAzGCOhuh3rjB+461bHCWu+ecOYPMjBAKFQoUdlEgxQqIjmOiam3XNg9Z7eP755yH0BFVIhD//W327NYbSVEwrjZ2VmLJCFyXVbEZRlUxz54U6mh5xeHidq1feYf+9dzDjETMO6SBJkEgf9ZAIEmciuOwzSJCyT6xSIuFKzFazkrKYUVSaCYJBGtPfTsk6W2wOLtDvbTu3kdLw+uvv8M1Xv8nQXOb+fMEfP4yR6FJicBrYBokxZiFdDJGgnH2VxkWXFCHasCCTMpEoJEhHNBuv+zTS8R8TAqHzLb/0uWd56it/gr3f/run/l1efOFLnLv4LFmWUJoSrbVzbTBm4TubACZUTg7TkuDTRisFkCFulhFcq0liFTMIIVQSpVKnpY6lW9c4SUHrANGKS9oYqgwNspWjoPxihYpDCAekWYgKG0I9Jkbz+td/80MPdXc3ZDDYQCnQxZji+BCdG9S4ADmgCdUihBvSOkMsw+Wm0aCd9tcsDvXhW/VUfpeWZMrMF3eIHblsLcfARbEELKO6LTFtybF/bmdjF+QYr6n1+t6lzs7DLN9rHRaQXurgCXDcyi789oWbTUG2oWTjJRHlUl9sjZNUWOs+a9XyuFrtsmh7BeEq0IFP3PPE/KwT3xpARERRQujZqYg2gJSwKaEuEE1JlBSEidcPxk5rqjZ9tZaJK02cT6CeuOIVTbWM7oah8/ENBTQRfEyTk+8Rqye0F6wDZaVdIhlth7sUjObFBG2mdKhIpCVJJBuDFGstk0qANgg0trJUpWZqNcOhqyzYnn4bW9A/9wQb588Tx8ad2xKUtAik0576kanAT9+LiFA4m6624xfhWT5bHxbL36T9eVpdVmOhaRrQDVBhSoOtLMZU2NIsbsZCgOxu0EkzpIiZygRTPYyd3f0d/V1fC5yVWQRC4WZt+PBE0m1c6dEuoLDO0WGmsVg0mpoaI9x5LToR1tYuwU3UtC4ixvqa59p1vPXRMfm1A/LjK+/bX+4OkR6CrgWlDbKoQdVo2TAl4mA25cp0xqx+MJWvK+oCAu924gu5VGWNMQY9KymmU6bDE8qTKfl4wnQyZTo+YZKPGY2POJkccXxyk/H4kGF1QL1iIzTEEAGphT6ChJgBhgxLjXCxzngDIbtEMqWXZggshZggbEFRlRTkUFR0jwripEAmU6oypJxpyrLi8rtXPen9+N8sT45Ljo5zNKsU3SC1ox4qBOn1sY32pFg3gKsCKp1XgOvT/QMMpXG6Vgj9MteRZ/0+P/hDP8LfPThh9s6vn+I3eZat7afJ1LaLnpKjTY7zHHaDFxpDrhISCVl7sOEpU18/kHZRULeoCaXrn1nKimUDjVZIFG544QiMlBJljNdSa0LjI78YwtZGDgDt+HUTAsrLbws2JDy1O4AXX2Dy7te5dvPu52AnhWLcsH/5DcbDIYe9qwwGu/R6F+lt7RDGO648sU9AUN7uTqqUMPRlOAqN1oUb4FQNMlSE6uGdck7lV0l8FTOZ+IptSynhMnvYR3jx8gCBi6AaLzdYDQe7BAC/8dbFoY2mwsLxQUiWyW5mcU4sorDG73f1c8ZvT3iSJoWL/poV+QNmqf2tcMfYGmO2xy7wcgf/RVt5wy3xnTMdvXR1x2prsViapiGMfJm1KIM6JgxjaCpsfEyYNIQZEEKy6bS9demLkUyclrcqbpsB8BGoMFqWUz3buHP0pR4fszc+Zu/yVQZPvocQTltjqwKbHyFNSUe4KoKbfclgkDrD8LKEyiAqS6jd9SINvFzeWlJ74yZc/8ZrPHmzRIgKISyJtMQSlBSe6EpkJBeRrjbxzXUcbSQY/vyf/K8fS0t93GBomPtpIesJsLWu0t3ytcAiXJl1K9z79RyM8PkIMXEUE8XubLYITP3oia+97f87aeAW92HcoHM2+3Da1JY9bgtjGGqqFYdgTU1tLWiBMhZVC4xVNNZVX2vCBpoUbIhtNESCejajHo3uuG+BI9qbsksmFJEFqw16qpkqwQjLwXjMlcMRY/1g/i95Pl0MaKx10+vGWBdxLUtmk5zpeMzo6Ih8OGY6mjCdjMmnE6b5mPH0kIkdMWXI/C4tWON0va6dKhK6pEJhhKQSgjRJkVGKlBmx6rprkRQpS0qbU0wLBAlFbjg8HGOtZJhOnetIWXE0OuIskF6A45Oc46OSUoJGuiCl18JKKdGEqNIFMZuy7cPbXrT0dLbBNC4iqRLHJ7VxYp3QSybkQvwq+dznnuWLX3yJf3qqxFdiSonRITJJaGPYWrqELOOjvtKTENVGs0/bb7rxJPU2qYiP2a78ZWG22zQhmKpV07IQTDQayH0lNI1EY3DmvEZrdNPQ6BBjJJXW5JNDitxVfqM4ZONulYU9jIbxTRgPG6TaJwz36fW+ze7OUwwGF0l7O6Rq4I9GkcqUNE1RqodSigZDXlRUOkfnTpMcqowsTVnxcXsgnArx7Q1YCnHx1cxWiKrwhFcKSKx7tA4OBjedbr1vuPASCcGS7FZmqaFdJGrIlYgsPolOLt/zkjyM8cdTutcFPhptvbesci4PxCDSlWNuSyyzJPCsJFXYNmqtPOH1x7iQKxufe3fG2dxk3FAWORGCJoQmjAgjQZikCJHQNCVVGEHUkGwAG5BuQ3Zug/xoQjN2xDZNvO3SSi/dyhyW4vxPOKoRw8ujD15nbOC9eyhFuIIJ8Oqx5dXjtx782Dz+/N966E2cSRiqpQdrq931F77xNyJr3ajcIJDClZ+1rbtL7KLmUaxI0oTGWCpjkebxRHxb+7J65XXtH8a6ZDYhnJY+n8LBPfLG9t5XA1NqNAJBhCAip2ZKTlQLOnQQVhDpGWgQwlmS2TCiQRGu2OlZXd8xJrAJPIlkZ/s8Xev2oTVoNKNaMLKWvf0Rbx3vMfrAEg93x95b71BbQzWbUEzHFOWMWV5SFgVlPqOcFuT5lGI8oZxMqYoZlS3RFBgqKnIM3iroQ3AClFiepKAnOqTdLvFGn9Br7gUSUwtEFJNubNDrx6TbcH5mOSlKRsMTbh4dcTwcLvpQhKVaKPsfz7n1MDg5yTk6yTFSggIpk6W+17hIn5dCo73trQuUul5BGzCmpDTaRXwbUGHoZmAA0AueHErptmkkKvkAaqNehPPn3P9X3sQJbbahfwHOXWCwfY7hb78MrHoDv8N7V65x42jMli/j6wh84qq4lvg/LkatmyUJPlW0nWTLe/1r462+XLN5UUlrf1U4y7JCG2gaX4Y49NHeAkVDKF20t8j30XpMURQUuqAoDHnecPPmkNe++TLjySFKud3e+JBuql5xH6v9OG16s+Hw7cvsDC4zGISkassnyivSrEevNyBNvf0hhqLQ5FpTFRW51mTpgLQ3IFQS+I8fuBlPh/j2WOSmlf4knk5YTLFJHySIvWa3a6Dy/rlYR3xNAiLz/NlHYo3BGYF7gmu9Tq3Vq4kYbL2UMBi4pVCG9RkcrYVaWcHcQt1GhnEWZrHXAreRaaB1KHE36FV5RXsDaqPOMQutcdXKJPCyi+rsE9/RBPJZThKFiDrEJhlN6Cq5JVGETXIsEcQl0aaTLWTnEtKdHapyQplDFAOJS3hrbgtHNb69WvK7xhofBWo8kbUrI3Z/zVtjsZXB+ukoIQymJR3GTfcIK5BBSCwTkqSDMZbEzr0/5aNFm5ddszxsu7qs8fcj42ZYxjmLxLIPQs1SAagBi0bQ1jQTjJkxQiMtDGqI6hBVC5g1COGSK0TUIRRghUVYgalr6pm+o764B+xk22xubqPGDdZYtGmojWZkBEfGcv34mLc5fmDK984bb2BMxWR6zPHRAScnJ0yHY4oiZ1YUVJRUlNTkrMqVbkdwj/tzU/yGTSz9TkJvsIVpoKrMIkASK0XWfYJuf8C2SLDEXL1+g+Ojb3I0PWLCCB+yIUJiyDkrxNeeaI5PclAZIdJpYGVI4rULzr7VYHRD6WVD0riglHvLoLWm9HZBiQmdK1OrATWaxl9jzlwsJCEjURmuzuXt9Y63+fwP/Qg/8CM/wIVLl7h27RrvXbnCkxcusH1+m0z1yZI+/+jv/UN+4xf+W5a+wIa9azc4OjpBZYosU4SJouW8MnERa/wxu8ejaNGlJtf5Ikta5W6ziPs2Ltza5Ji8oNFjdFGgC02Ta5SSpCoFCsImh7BBSTe/rY8vMykOGY/HjPWQ4VAzPNa8/XaxmGWcPqTZfg1cG8Jw2JDibPgaIE1hMIA0c9GwpmnIc6g05IUreNYbwMbOYOE+8aA4FeJbVUud7iJd2DgCG8csTNN93s/Sh9csSWtb9U3g+x25nCYXyiektc4MOMkBZqlzNP7bWAl0vAzB+PVafZCBwrr3Eu/fG+OWtyTd4jqKRTEN/72kZOHasCL/cxpm69rArDykYKFXPss4nMBxUZNkACFNI5xVh4gIo5DGLkffYQhRBCBompAwCkkSV64UH3G3zdJKyTY+Atw4KVEYfVTfco1PO06YMGWG9oUHBIJYJsRSUdkKY6zTQicxMlHIpM3edTcvKaWLahpLNa2xlUU2AcKeyi32A+FkCMtobzs2d1TJTbsyg0aDnTqrwnux+b+GG5B2/PY3MSgsNYaQGZoZtd9fIzR1DbNpg60joIOwiqjTQamUelxjKdh7a4/X9QGv3WF/x8C7+QHZ9S7dRiJxhTM0hoOi5rox7HH4UHTvn3z172MbS1VNqMopuqowGCy1T0GzSAxiZeTTSttYeQZHeO5FabwPzMwho+tHbJ2MyDY2Sbt90qRLsrFNJ+4iIuXzYZy0xlQGaw0VU+A6Lck9a1bnIsvY6vchy5BJhlSJszLz2lzjO+4QiTKh76ul98k1oBzRbTzPUYlCKukdH8CYhsa4SKszKZAkoeLcxXOQPQv5y7cd0RRjShSSrW6G6W/DBdje7pOpDCUVSkKWKJzKfeVKyZ0xXZJIuv0uYRLSGE2mS/JSUYba9WXG+Ogzp57cun/5a+hhz0d422370QJtad+KJh+jh0OaPMdUE3Se02hNoytSFZJmihCDoiFTkMYhqYJxsU9RFIxzZ5d5eAj7w1uldaeFds6mvXeZwiU0jicNk/GtOQhu4O0qwo2L4cMqHU6P+MJKwpev4CYTRzCFT3Sj9GTSOzFU1ksURDu6w0ki/IEVufMyT3tuqjyWy320+5E+KmPau5NcyhSE9bpI6TS80hNhIdy2Euky4mUbAZZertC4SDL4wK4/oDbpxdQs/ImN9VFnT+Zt5b9jK7044zrffWCUN2ztuE7Q2b+FhDYkthFNI6gb6zS6oSO/QsRunTAlyibEJejItU2Fty6rXfTXWj9+bbz0YY01PgJoDDkllXFaXyEkUsYomTI1OU0NMpbEiUImCYFylnGBcSet02kC1lIVJdYYpJVOMvGI4WI+tOY6jojibsMRLkJrtXs9Y+mgcC94D1c1sO0oMiwRGum3Zfw+LBZjZkzzGj2L/A47dHtd6Ghmumaqa65eucKbwBt32NcRjpR3xtfYZJOO6FJHEUYIrszGXGHM3kPWdvy9g398x+UBy36nfURejuC6lXY4YRbRdLh3InoCjLEUxXvsliXdpEO306Hf30bGqUuKrljswxjXqWjGnIXI7t3Q72ds97chywiTzDlqtA4yjtdiZIkqw9bcGBm2cgeJlAadgHFhXlSiSMLEEWBcNbimja42TjCXqYwL586z89xzHL5yO/HVoBsSKemrDLYdyZUqQcnQSTHCkH6W4IjvCsocMKgwpNfNSDJFWWoamSFDTaanlKahCV3k2jyCKczDy/+MPA2pdIFptIvyhhpniWwwuqCpCvLjY4Z7Dc3EDXhXKDKpf6jQEc1BCoOei7g2rnkWxHe/ePhaeJvA3QR+q4KlKZCOnXTvbkKmGqhPobzrqRDf3/07p7GVu2MCt1Sw+TAMH9WB3Adafd1Zx8UUko1NGhtRVoJKuGIIVRLRRAlh1iPZ3oX8BpSFc3IoRti6ocon2BJGJUxy5/Nb5i7iW3OrtCG0EH68nXnW+IRjSs5Ml5jaUlWGsqiQsmCWFxSFJoxclDeMJGm3gxACaxpsZV2J2qpCa7euc36RjyVhM2RJcH3MgRoXJZnhqrdHK6/vF/8ceB34PI4KCL+9Nq2jC+gDGCmQfirOmhm2HpF1Z6RqxHQ8ZaynfJOc37nLfgqcBGObmlpCJxVYKahDwfEMjqkfWUnzObdKRcCReZeiuBxUwFLzfL+xvDmORDT2kOrykPHRIecu5qSD83S6A5JOD1NVVLqkGN7gevU68PGsyHavGB4dcXBwzUV7s7736ZVOnmBcxNY4Ia8LUgFgnK0rrnSuW+QiWEpmyFChksRZda1G/rzna45mmucopXBF3/OVlRT5NOfGlWtcy/quiIMxJEpDotAmx4ArMKTCW8OO/QwpDabR5NOcxpSY0h2/MQaj29h1iDaaR6F1+J1fvYN/bgrdnpOc6txppacfUFJ4hhtkdhpQDYw1DIeQ+rZsGhZCn/vhXXfD/WSsDHk8vOnRz8OtcWYhgacvbbLZ2wSgqUNsGHpz5RDCGJGkpJs72LCisgV25sht2UxoSidnmPjRY567SK/PBwJcB7qoyHLGo+NrnG3MKKiqCmMaqsow0wV2CsV0RlmWJN0O4GQNcRIThTFNXWGoKI2hqQ3T6YzJyRgpBSpKFv6wjxqKJSFbsaenZNnxzB9i+yWOAG+wjCpv4nS527idR7OlzKId+PfGIxQjxv443uSDtfxuvRDVjaCrsDJEIyimAj179OmvbZLgqgvGqtShhU/heCA4TwbDaPoOs7cs5y+WPPm0pNvvMTWGysyYTY7QZ5z0AnB0g+vXrhAmGYnqo7IMmUhMaTCNoTENZVm6pDXjYritjlWbEmNOQEofAU6QKGSYoWRCIhWhkiTJUu9pAG00xwft5HyfW4lvQp5PuXbtgMzp9wBXEQ6lKI1B64aToxPXAWpFy343zvcB0KUmz0/QeYjRDfgq9K3EoTUXM6Z5LBp/CpgWUAzdRMu9YgaLoYXGkeAW7f3jNAaa99OtP65g4Zr4rnFXbAJpkiCiGGxzS4dlF674IWEUYUW86PUaP89at3reGqrGvW6jUS28scMtHc0aa3wUWLg6eBG/NZamMRhjmTfWr4HzPA580QMhFklvrSNEVdVgpY9+Pr5b7Gre7eq1+jCE93a0EaCAZaS5JdptB7eabNde27l/fFgEqf1cI4QrUBAKJxuLoH4wI4f7xpxbO+vbSfBqAuGDYoY7M0pzQlWVWGu8vzBgnUf0o0TA6Z4Xd0WjKXXpk9HK1ogXrbWPkhp06b1jTZug5chvqZ1XrpQSIyUyNDRymc9jpCExGYtittKRzbIx6MZ4rcH7rz9jGsrSRW1D6VLijAqdnlxryrKtyOY36ulf4quxuYptJTQJpnSR69Wk7YV2+XGQ3hXcD+ldfAYWtq2raA0ZPqmxqGA+fyyn/xprrLHGGmusscYaa3ykWAfZ1lhjjTXWWGONNdb4VGBNfNdYY4011lhjjTXW+FRgTXzXWGONNdZYY4011vhUYE1811hjjTXWWGONNdb4VGBNfNdYY4011lhjjTXW+FTgkRDfIAh+NgiCT6xdRBAEzwZB8MtBEIyCIMiDIPiNIAi+fIrb/6S3388FQfCPgiA4CoJgHgTBT37Auj8VBMHvBUGggyB4PQiC//AB97luU7feTwRB8CtBEFz26/3iQ+73U9+uQRA8GQTBfxcEwdeDIDgJguAwCIJfD4LgX3zAfX7q29Sv9wtBELwWBME4CIJpEAT/PAiCPxMEd+jjZgAAIABJREFUwQMZ+q7b9Y6f+f4gCKxf/76999Ztuljvq/792x//yQPud92uy3UHQRD8pSAIrngesPew/dajivj+DeD7HtG2P1IEQbAN/GPgu4D/APiT/q3fCILgxVPazSe2/Tz+DNAB/u8PWikIgp8C/hrwK8AfB/4O8D8GQfCnH2Cf6zZ1+LeB54Ff5f6K6twN63aF7wH+TeDvAf8a8JO4mg9fDYLgxx5gn+s2degAfwX414E/Afwa8JeBv/iA+1236wqCIIhw99eDh9jnuk2X+CauLVYf//sD7nfdrjjSi+NbfxT4GeBHgf+Mhywqt/bxvU8EQfAzwM8CX5jP57/vl2XA28D/O5/P/42P8PDOBIIgEPP53AZB8FlcMad/bz6f/+Jt60jgGvAP5vP5T6ws/1+AHweenM/nn4Sq0KeCe2nT1fX8/3vAr83n8598rAd7hnCP5+omMJ3P52ZlmQS+BRzM5/MHivx+UnGv5+pdPvu3gR+bz+cbj/IYzyLut12DIPhp4E/hBmw/DUSr5/Aa93Vf/Sog5/P5H37Mh3gmcR/t+leBfwn44nw+P41ADfAYpQ4+nP3fBEHwZ/00ax4Ewf8TBME5//g//TThe0EQ/Oe3fXYnCIK/FgTBG0EQFH6dvxUEwcU77PvfCtzUeBkEwStBEPy4n4b46m3rPREEwc8HQXDVh89/LwiCf/8evt5XgDdb0gswn89z4P8DfuxBpovu8B0+ye1HS7w+BN8H7AD/623L/yauSup93WDWbXp/690r1u0K8/l8dDth8K9fBt533B+GdZt+II7wxbvuF+t2vWU/zwP/JfAf8RCVYtdt+miwbtdFQPHfBf7GaZJeePwli/8d4FXcxXYe+EvAL+FKwP8D4K/jprX++yAIXpnP53/ff24LN3X4XwCHwAXgzwK/FQTBF+bzeQkQBMGPAv8b8H/595/w+0iAN9qDCIKgB/wWLtT+s8A7wB8Dfj4IAjWfz//KB3yHhjuXadd+e88Dr99zi9wfPgntd6/4Tv/86m3Lv+WfvwP4jVPYz6epTR8nPtXtGgRBjBu8ffMUN/upa9MgCNrqyF3gR4CfAP7CaW3f41PXrsDPA788n89/MwiCHz7F7bb4NLbpl4IgOAFS4DXgL8/n8//5FLcPn652/R6//YMgCH4Z+Jdx/OvXgP90Pp+/88Bbns/np/7ANcT8tmVzXMPJlWV/0S//mZVlErgB/MIHbD8Envaf/VdXlv8T3EkRrCz7br/eV1eW/TncSfC527b7PwE3V4/xDvv+C7jy9NsrywQuXD8Hvm/dfndvv9vW/6zf9k/e4b2f9u8lty2XfvmfW7fp/bXpHdbdA35xfa6ebrv69X8OV+r+j6zb9MHbFPgxv87ct+fPrc/Vh2tXnM7/GDi32i73uv11m97x/f8K+CngB4F/BZeXcsv3Xbfr/bUrLn9qjstF+T9w+t4/BVz2j40Hadv5fP7Y7cx+dX7rlODv+ed/2C7w7/8+7kdZIAiCPx24rN4pbqrrin/r8/79EPgy8Ctz32p+e9/AjUZW8ceBrwHvBEEg24c/jm1cNPFu+Ks4ovtLQRA8HwTBk8D/ADzr33+UUyOfhPa7VwTtIZzCtv5/9t48ypLsru/83BsRL96aL/faq7uqW62WWqglAZawhFpHYMCAWAxmEzODfbwdj32AwYOxOUAb0JmxGZuZMYYjMFjWmMUslgyIXdZKS6Klbkm9VZdqX3LPl2+P/cb8cW/ke5mVVZVVlS+XrvieE/nyxbsRceMXN+793t/9LbfC/STT3cR9K1chxPcBPwb8TJqmn9jBU9+PMv0E8JVo55b/E/hnQoj37OD54T6SqxBiEvi3wL9M03TpXs93C9w3MjXX/sk0TX8lTdOPpWn639M0/Q7gg8CPCyGqO3ENg/tJrhk/vQh8T5qmf56m6W8A3wWcRE/g7gq7beqwtul7eIv9xeyLEOKfosnlvwP+d1NeAp8eKjcNOOiZzmZs9lqdRc80bmbbNHWzG0jT9IIQ4t3Af0A3LoBngJ9HexvO3+zYHcCBl98doGE+J9ko08lNv98r7ieZ7ibuS7kKId4FvA/41TRNf2onz819KNM0TVvAZ83XDwshQuAnhBC/mKbp9R26zP0k15811/1toZ0yYVDXuhDCT7XPyr3ifpLpzfCbwLcBXwZ8aofOeT/JddV8/sUmIv4ZIUQbeOPdnni3ie/d4nuAD6dp+iPZDiHEqU1lVtAPYXaL4w8xmN2AFugS8IM3ud4tbXTTNP09IcQHgUeAME3T80KIXwKupml65VbH7hH2lfy2icyW9zE2Et9sJvniDlzjXnAQZXoQcGDlKoT4GnTIvQ+gQx3uFxxYmW6Bz6IH7FPAThHfu8VBlOtr0URsdYvfVtARHr5tB65ztziIMr0ZdmvVcjs4iHLNOMDN5HfXq+sHhfiWuTHm6N8Z/pKmaSKE+CzwHUKIJ7MZghDiy9Gd5PBD+xN0HLkrd7vck6ZpgjZgRwhxFB3H8+fu5ly7gH0nv23gU+gX8d1oY/YMmX3aX47outvFQZTpQcCBlKsQ4qvQpOHDwPene+QNfhMcSJneBE+gB8ILu3zdrXAQ5fpDwPimfT+Adhr8Wu4tpu9O4CDK9Gb4PsADntvl626FAyfXNE2vmfp8nRBCDNXnq4Ax4Om7PfdBIb5/AvxzoeMO/hXwTnSg+M34KeDPgA8IIX4Zrbp/Elhg4+zg59FE9RNCiJ9Hz04qwKNoZ5RvvVlFhA76/W+Aj6Eb0mNoT8kX0LZT+xH7Rn4AQogn0KHKDptdX2HsjkjT9HfNZySE+Al0worraPL7TuDvAv80TdOtImvsJg6cTE251zLQmpeAB4QQWb0/lqbp8u1vfaQ4cHIVQjwKfAg9Ufs54MuFEOvnSNP009u891HhIMr0m9AD8x+gB9waOp7nPwDem6bp3B3c/6hw4OSapunntzjuHebfj6V7H8f3wMlUCPHVaJv+/wZcAuroicS3AD+2Q6Yj94oDJ1eDH0PbDf+uEOI/mmPeg7Zt/o1t3flWSO/SK+5WGzf3SPzZTft+wOx/eNP+jwKfHPpeQodfWUZn7PhD9AwkBZ7cdOz3oR9CgCaj3w48C3xgU7kJ9MO7iLaHWUI7UvzQbe7NNtdfNNc4j7abKufyu738huqXbrVtUfYfoj1ZA3TkjH+cy/TuZcrAg3ur7R25XO9crkP3tq02nct0WzJ9FE0krpq6LKIzOL0bkHkfcG/96lZyYWejOtw3MkXbuv4x2vQmALro6AjfezftNJfrDWX/Jlq766NNLN4PHLpb2aZp+srP3CaEOI52QntPmqY/s9f1OWjI5bfzyGU6GuRy3XnkMh0NcrnuPHKZjgavRLm+ooivEKKE9lr8C/Sy42ngR9GG2Y+laTrKiAsHHrn8dh65TEeDXK47j1ymo0Eu151HLtPR4H6R60Gx8d0uErTNyC+gw2lkqYT/9ivlgY0Yufx2HrlMR4NcrjuPXKajQS7XnUcu09HgvpDrK0rjmyNHjhw5cuTIkSPHzbDbmdty5MiRI0eOHDly5NgTHEjiK4T4qBDio3tdjxwaQognhRDv3Ot67EfkbfXWMG0nFTrdZY49RN5W9xfyfnUj8vZ579hOfyuEeIcp8457uM49n2OUOJDEN8e+w0+h4wLmyJEjR46dQd6v5tgLPAN8lfl8RSInvjly5HjFQgjh7nUdcuTIcfBwv/YdaZq20zT9dJqmmzO9bcBBls++J75CiO8RQpwRQgRCiBeEEN++RZlXCyE+IIRoCiE8IcSnhRDfsEW57zXn8oUQzwkhvuV+Xz4RQjxuZLdqZPeyEOJfmN++TgjxR0KIeSFEXwjxvBDiR4QQ1tDxmXfkj5uljVQI8eRe3MteI2+r94RTQogPCSG6QojLQoifFEKs90/bkdvQMt7rhBB/ajIB/bb57euFEE8JIVrmGi8LIX5y0/GPCyF+XwixZq7xl0JnZXrFIW+ro0Xer94bttk+p4UQvySEuG7KnRFC/IMtyp0SQvy6EGLZlPv85vPdqu94heI1QoiPmPY3L4T46ay/FVuYKZj3+ZNCiHcJIZ4VQgTAPza/zQghfkMI0TZ9xfu5MS32vsK+tqsTQnwtOi3dh4AfQaer+38AB51ZBCHEUXRGnw7wT4AW8L8CHxJCfHOapn9syv0N4NeB3zfnmgb+b6CIzgx230EI8dfQGVTOAT8MXANeBbzeFDkNfBj49+isKV+Bzigzg04lCHpJ5FPA+4D3mn3XRl33/Ya8rd4zPgD8J3QmoHcB/wqdtes/bVduQ/jvwK8C/xpQQojTaFn+LvDT6CxDr0K3bwCEEG9Ch+15Fvj7QB/4R8BfCCH+epqmnxvFTe8F8rY6WuT96r1hm+1zDPhLdEayJ9HZw74e+CUhhJum6b835U4An0FnFfthdOay7wZ+TwjxbWma/v6my2/oO0Z3l3uODwK/BvwfaLn9BPp+n7zFMY8A/y/wM8AFoGH2/zfgceBforOrfje6be9f3Evat1Fv6Ib9IkMpKoE3o1PbfdR8/7+AmKGUfYCFfkGeGdr3FPA8JoSb2fem4XPdbxvwcTS5uG26ZUCgJ0o/DqxteiY3pFK837a8rd613J409/V3Nu1/DvizO5Rbdq4f3HSu7zT7x25Rjw8DLwGFTdd4CfjgXstph2Wet9XRyjfvV+9Nfttpnz+BnjS8atOxv4JOvGCb77+KJrtTm8r9OfD5oe9b9h2vtG3oPn9sC7l10Jrad7AphT16IqeAN2w67m+Yst+zaf8fbz7Hftr2ramDWfb5SuB30zRdn3mlafoZ4NJQ0bcDn07T9NxQmQT4TeANQogxc66vAH4vNU/FlHsGPVO87yCEKANvBX49TdP+TcocEUK8VwhxGa0li4CfRb8cs7tW2X2OvK3uCD606fvzwEnz/23ltunYD2z6/nl02/0tIcR3CiE2tF2hsxU9AfwOWkNsC+31LNAZjN5+97e1v5C31dEi71fvDXfQPr8Brcm9mL2v5p39U3TihdcOlfsjoLVFuce30Xe8UrHZjOO3gCrwulsccylN089v2vdV6KQXv7fF+fYt9i3xRS+ZOcDiFr8N75sEtsoosoAeuCaGzrV0m3PdT5hAP/8tl8+Mvc/vA9+M7pTfie6Q3mOKFHehjgcFeVu9dzQ2fQ8YtLHtyG0YG8oa8vb16Pb+/wELQojPCCGeGDq/hdYiRZu2fwJMiCF74wOOvK2OFnm/em/YbvucRU/ONr+vv2N+nxoq9z9vUe7nNpXL8IrJTnYbbJZv9v3YLY7ZSjZHgLU0TaPbnH9fYT/b+K6gG+ihLX47BFw2/zfQKfY24zBa1d5A2+tFbD2bPgRcudfKHkCsoZcubtbQH0Jrc/6nNE3/S7ZTCPGuXajbQUPeVkeL7chtGDeko0zT9CPAR4T2RH4r2tb3Q0KIB4Em+l34D8D7t6rAsPbpgCNvq6NF3q/eG7bbPlfRE64fvMl5Xh4q9wm0ze5WmNv0/X5JZXsIbac7/B3gOjfnhVvJZh6tGHA2kd+tnt++wb7VYphltaeB7xQbvbvfDDw4VPRjwFvMAJaVsdAG1s+madox5/os8B1CCDFU7suBUyO8jX0Lswz3SeD7zVLvZpTN53pjFkI4wLu3KBuinQzuS+RtdeS4rdy2e6I0TYM0Tf8H8G+ACnAqTdMsH/3jaPvVz27edvBe9hR5Wx0t8n713nAH7fNPgEeBK1u9r0N9wp+gnQpfuEm5YHfubN/huzZ9/x6gizYxuxN8Cr1a9h1bnG/fYj9rfEEH8P4z4INCiPeivTv/FXq5LcPPAz8A/LkQ4qeANjrMxiPAN21xrg8IIX4ZvaTypDnXK0Wbc6f4Z+gB7lNCiH+LXp47DbwB7U17GXiPECJBd9Q/fJPzvAh8kxDiT9Aaj7k0TTfPpF/pyNvq6LBduW0JIcQ/Qi+L/hHa6Wga+BdobU/W0f9vaKekPxVC/CpakzGNdtSy0jT9sc3nPcDI2+pokfer94btts/vBj4hhPh5tIa3gibDX52m6beacj8J/BXwcSHEL6DthCfQtqyn0zT9u6O/nX2Jv28mFk+jzcD+HvBkmqbNoTnsbZGm6Z8LIT4JvFcIMc0gqsOtbIX3HnvtXXe7DfhedKMOgBeAb0d7GH50qMyr0eE5WmhPz08D37DFub5vi3M9C3xgr+9zD+X7RuAP0Mu9HnAG+OfmtzegtRd9dOf90+gXJAUeHDrHW4HPGdmn6Bdoz+9tD2SZt9U7l9mTps3Ym/a/D+1MsW253eJcX4UOU3TVyHMebQv46k3lXoN2ylgy5a6h7TG/ca/llLfVg7Xl/equtM8JNAG+iNaOL6FXbn5o07mOA/8RvYwfmvf/z4HvHyqzZd/xStuG7vN1wEdM21xAhyiTpsw72Dqqwydvcs4ZtNNrx7T39wPfuvkc+2kTpuL3JYQQx9GxFt+TpunP7HV9cuS4GfK2muOgIG+rOXLk2M+4b4ivsbf6d+jwRCvopacfRRthP5am6f3izZljnyNvqzkOCvK2miNHjoOG/W7ju5NI0B7Jv4AOYZI5tPztvHPOsc+Qt9UcBwV5W82RI8eBwn2j8c2RI0eOHDly5Mhxf2PfhjPLkSNHjhw5cuTIkWMnkRPfHDly5MiRI0eOHPcFcuKbI0eOHDly5MiR475ATnxz5MiRI0eOHDly3BfYkagOQojcQ+4WSNN0+6lQDBZb954zPFUmcZJSKEClCq/Xo9fp8sXPf5Hnv/ACn336WZ7+9GfpBufu9XLbwCHe8pZH+fLXP0DJtbAs6HYjer2QTs+j2/MIwhSFQEkbW9pYtkW9VmN8rMavve+XBvd2FzKFrdvqv/6p91CvjyMlKAVRpLOJSreMVXCRjoPlOEhp6U9L4jgOlrSQgGWBTMAiodVq0em2WFxc5Pq163Q6HU6eeohDR45y6OgRZo8cx7IkSaKfjWXdOPdMEoVlSaS0NuxXKiFJFMVyGafg8tRffoannnqK337/r6Bj4d877kautbd9b6oAD0VKArYFjqTggu1KHBskDhKJlDZYFirR9yOlhXRspHSwpEQ6EiklWBZxN6LXjGm9dAXOXgVeQsdI32GcfiMTX/YGTpx+mCgOiKOASMUoFaGITSFFFCXEsUIlCqUiiENkErD2F/8Dejev19221X/4wz+ahmFMo+Wz1olwLIdCoUChUMRxXP2/XaBUdKmVJLVqgSOzdWYP1Qi8kMXFFi9//nO88KmPc/3CWWygYEseet0befxt7+TQqUeYffARKvVpZLFKoegyVZeUHIijmCiO6IfgRRArG2VVQEqUUqQowsgjDn2Cfo9ur0Wv16bXbuF7HQg9iLucPlHjyEyZK/OXuHTtEh9/6jN84g+f07EgtouJGqzdKN+7kasQ/0sKEqQLdRdKDpQtqLpQdsDvw+ocXP4LdGK/UUPyund9L1/zdV/D2776CWzHwXFc/S5IC62bGuojLJBYWFLvk5ty46lEGdEaAZu+/93f+i2svPDx29bm7mS6sU+dRYf5mAEmgSI6awRAAXDNviKwjM5SsWy2Djr/rWWOPWq2E+hcz5ajw4Z8IdLp7zpmy97SncYEOj7foaG6FwAHLWGFrk8DnbnBR/fE2W8Aczs4VuUY4G77Vdj34cwkOlV5b68rciAhDJNLAZXExHHCytIKl85f4HOffYa/+tTTvPT8y3SDy7tUoyYXLq8wWa/xwLExxsdcBApUAkM8f2NfLhFCIuRdt/HbIkkUSeKTRIrA92m2OgRRhHTKOMUytYlxapN1XOkiLQfHsbCkhWNJpGVhSUjw8ft9Go0VFucXWWus4fsRjuPgOOBYIIfHL0N4h8ltdt/ZcLf+ZQjW0CBoWdIMgHu7cNO98hyQgIr1SCwBRxI6ilAmpnrm7jIhxOhZRiJBOeiuyNxsFIEKoKOgGaATMo0QcoG1xWdoBxdI4lhfO0mySuqqqwS6IXgBxBFEChwJrntL0nsvULEiVZjrA5YW2foOpVDrP0qkYJ0JKSBWoJQkVhKFjZIShURaRYqFMhY2MlakcQJxrJ+D0u1p/fhYoWJFrAAVkhriG6sY3+8TBT363S6tTpNep0WvvYbvdZBxgIx9oikbFdqU7AKz9TonDx1h/IFzNC942xdEcyflOyCE6+RbSt3m+gEsXIKFp9kd0gugeP4PfhPXdTh65DCzM4eZmT2MlI5+1tmrbZmqJyAtbp8MWm34QCnrpkV3Gi00aXXQ5NdBj+ARA8KbkdsOOtPJytDxCTBujovM1kOT5yjSxDIjl5jzjIr49k09sjorUw85dA+O2TIJW6bOw+Q3x/7CrhLfV33Nd+lORoLvRwRRgNfz6DbXcC2bSqnE1PQ0j7z6UR5++BFKpRKlWpUL5y9x7sJ52q0usVK6Q7CNBsnWGiIh1k+tBwMzXpFq7QRKE0FpSVCQpAqlFJCilELJGEhQ+hBQgmxuZ0kXSYE4ViRKYUmJbUukrQcepSKSOCCOfVSakqh4cP19AJUq4jAh8DyuXr7CU598ik98/Cm+8Nyn2N1JRcDS/BIXKiUmxxzGx1xSLUFSpTBPCqREIkEK/Uz1nxHWKyLod/D7XVqNFosrDVp9H8etUazUmPVnwUqgVscpOpA4JKbblSgiEoJ+l05jhfn5a1w5dx0/8ikWi1TrVYpFB6cosRw5xGpNN2nJ9TFN36FhONYQyR26dUUClgRpIaWF4zjodrqHuPrC3l7/XlAA1Dxcnye5ih6xshc3G90yDtzkxpd6hD2oUui+yVwzVUP7wfQ9SmulpURKEEOvikIT3RibhAKgtYi2XaFSrlG0HWSsEEkMSiFJ149NDelNEkUcx6SxQtk2IIhVQqxCIq9Hv9uh02vTWWvQaTfpdJqEvS62CrBViN8pEI8XKEmLmVqdh44d5fFHHuZT7ecIV9gedlTvlT1gm+EJBO016K/B6tNoXeJuQvG53/2vHJ6d5U1vfBNj1SpWzSIxD0NPkq1BWzQdwjAn3oxED4R6PFS7OzUO0IRRAmUJFQusSGtDa2gSaTg8fTaSXtCvZNlsjtmnzHl7aLKcEWnf7L8dSkDF/O+bbTtkOTDXdsz/oamfMveR9b7W0JbJOiO/OfYfdpX4Xj97jiCOSKIAVhuQNshGkqwxto48xsOnHuDEsRmyZR7XBkeAW5CULRdse30ZKCO5UoK0pD4i6/iVGRjS1JBciW0JSNEkS2EIlzLT6BTQmkWFRMU2KIlKNVm2baNxkfqVUXGEimMU0frAAbEmcmjly16R3+GxQilFEPh0Oz2uXb7OM08/w6Xzlxgd6c1e/a3uPsXr9YiiiDhJzHO5Rc+8rigcLfFtNjrMX7vCysoK3Y6PryzKtQilFEsk9PstJqcPcfyBU0jZJ4giokiREKGShF5rhdbKIt1Gl363g+sWmajVqU9OUKvWKJYdZMG6gfRuuG/N+m9U+W4a2aShxJajTS60jcYIxfNKRgi0GVKNoUesjPRK9PqmY/4P0CNmYvaXGJlCWqmhDkRJ3V7QE/1BP6eQgC3BtoX+bWhDFrBtF2lrba5SEmkXqJRquE6Rgg22VNi2wrZAsMUk09Qh6yGzQUNfQ+heOI4hSYzCX5HpjePQx+t2USpAxSFFKTk8NcFrHzrJpeIczVZs1NNsj8HcMzY9aK8PgYLVF4DngTvQRO8oPD70X96P125RHxvj6IMPMjE1g7Qkar2bkEjHMqsRmZ5fN0Ur22HIcdZlJEjkZnuIXcAyWmdeUzBluvcaUEcT0CYDbeosA23wsOa0wkZTB9AmBZlZRMT2n1YR/bpmc4c7GfnawCID7bM09aygyXnWZWRQ5IR3v2NXiW//6ll0c7/5ixjPv0QSPcGxo4cJAp9ex8O1JQWZ4joWpZKL45aQZtkuiULCOEFKiW0LLEtim447Voo0SYyWWF9T2xkOaYWVIlGAtDXnkAIpbb0SFkKcQBxKEgEYfqGU1uqq0CNJAhQxUkosAZmeJdMc78Ui9DrpNfeslCIIA3q9DtevXuWZp5/FTxdGcGUL3cVkEOgmFqEZhtayeF6PKAwHWt71pXAxqL00d3Kj0nMEiGi1lrh0/gyLcysEEchilUnTe/X7TVaWIfADJicnSRJo9fv0+z18PyCIfDprS6wtXkMmDlbkUHVr1Kt1Dk3PUK6Vcd0ijiMHYiL7dzBEZTdpWRuZrtxkB2xZWgsksZCWhXAc0l0hDa9QNNmo3R3W+Dpocpttw8iOGaElxrplAxJh+q7NVUBqTa1lGzJqYyZVNtK2kHYBIYukuqfCtl0q5QrlYgFHSgpmtcwcYlZbTPcxUC+ba2WqATmYn6mUNI5RsUKqAemVKKIgwu+ZhWoVUbJtjk5NgIqZnqnR6vdYWFtjcbVFuMgukV+MSUgE/TZEXeCLu3jxm6C9wCc/9hEefewxSvUqE1NTSCdrZPoZOZY2Q8nI73D3QabltYz1mJlDJ8MmE7sED018K7pKzKCJ77jRAHciTT4dtA1tEW3eALCAJrY1NOk9DZyegcSHxY6+lS+xkWzeDsMmFv4d3kvLfA4TWsvUr2buITN/gJz0HgTsso3vrUkvgDV+jKnxGq5UnLt4jk//5V9x5epVLl+/TqIU5VqNUqmGbTsgJGEUEkURjmVTcI2DjMwGCDHQFEqJJSSFgm1s1DJNsVxfJsw6e6UUSayItSKXolvBsV2iOCJOtB2jRCFFgiNjEArblggpCWJ93jhbX9pDTZxCO+J4nke70WRhfp6VlRX8tMNoXs+MNVjopjWJnuPHZkupVmFy3MEtFoxJgxhoM/SHqftAfFIKpBhhU7Ucoiih0+nTajXoB1CuwvHjxzlyZIZWq8laa42L58+xuLhCq9thfnGFtWaLKPLxgwBHJpSJOD5zkiOHjlO0tLObpfSGSrCwjNPa0CKlZSZjgLLWpbAl1gmwXt7AMRpfy7JGZuN2X2Cz8OSZJmBNAAAgAElEQVTQ/hDdCF2yBaiBza1t9o8IqcrseG+snBz6KoVA2hLbtrHt4V8z+mkTKUnoh8RhRLfvEYURqJiCVNgyU7mGgLNlvyXM2VCaeGWm3NL8rzvOmDiJieMIpULiOCCMbYJQgAog9rXW17aplcr0fI9W3KO12iG8svPy2xrGLUmtQXsVuICmWvsD4dWz/Odf/hXe/rVfyzd+27cyc+wo1Wodxy2hoggv8bAdG8dyNh640V5q8AiNZdTuWfgOsIQmmdlocwiYrMGJww71ZsRMB1Z9WFS6XISeg0YMXFgngXEHxosQJnp/RmC3O4LZGMc4U58G27eeGTPHJAzIc9GcL9NQZ2YYTVPv7B5y7F/sMvG9PQt89cMnOTw5hkXMxZdf5g9/57cgGJhEdHHAqkPB1aqvKIIg0E4mrgsFfUtCSJyCg20XcBzLfDqUSg5xkhKFIUopCgW9X9o2Ukr8MCSKEiI/JkxiUJLZ2cNUKlV8PyCKAmwbHEdiFwSWpU0gbBuEISYq0rbDwJ6ofDO9qVKKOI4IfI+1xgrz16+zstrg7jUbDnqOW0K/4u0tysQMzP0Pw6EjCKlwpKLmBEwVPaYrMSXX0YOpWVkVZhRdN5JQoKTQ2iUpsezRObdZ0iFS0A98WgF0AavfoVouc/zILJHfZ6nXZ37lCtfmlwm3kF+I1mBYwRWOTRzCkRJHh4jQKhfTSzvS0u2WAZG1pLYO2+7gJK0smoRjnOecnPjuJDYTP4+br6lWbrJ/B6DNtIwG1di76+oNOpXMBt6ybO13sO4EOkx+BbFS9IKAHn06vT5h2Eepku7LbIiNc2K2MqYrgNH0mtUyMxuVSmlviCHyi9LmDioOScKQJA5AhQShQxjakPio0CMNQxxbUim7yDVJFMW0F3dTO+CiFTBr6Cgha7t47e2hd/E5/vi/LnPqVafAAueog+s4BH2fIAiojtWxnMxSNjOH0ezWMv1MIs3zsszPe8F80SPEi2iS+GXA1Di8+qETHA07tIIOzTBiOUqYW4DLc5qU9tBa1gSYAuplqDngWwOyeScYZ2BX3Gd7tr2loeOW0WQ201rXzW8Z+c5sjpvo+ud98f7HvorqUB6b5k2vfy3TkzWCXpv5K+ch2Gz6HkGycuNAFLCBz6VoMhKyGSWGdIngGMIcxRCn2r5h3aRdly9IcK0jRF4fr9fDLkhUSWs1ncJgjTRVFkmQEPmhXqUXe2HoYCi3UqgkJgp8Ou02c9fn+dKZs8xfnePuiW+FgbHjzRCxrgorl6nPzuLaUJCKsuUxYXeouQGWNawq26TF2gBj1zjKGYTl4DhFiuXyuodwX0WgIsoFB8cCSGg1GluS3mF0oj5Rv08S9PH7HfodB6ds4ZRdvTxpSW1OzjDpvdGcAUCZkGc3mDqsr0trs4jR2j/fh7iTkWuEvqGxiolVFp0B1FDEhWGqmE0epfFs0wpYtR59IYwTgjDExyfBo9dt0VlbpVK2UfUiFCQQgw1SlZFmNUylamASBqz/M2T6oM1HhY4gI7PJdmjsxHziqKi1wH5E4vt0A49Wv0+j1+PawjIXLs7tsoXBWQaxBPbKnncbaC/whx/8AJcvXeb0w69mamKKbrdPv9vlkde8hhMPPkC5WqZaLiGlpoJKqYH7AENBK/bI5G4YmdWQY4HjQM0p4lQsyiSUgSRp0GwmuH3tLJatHYZA0IKOBWFTa18zx7Jxbv8ESwxCj3WA1W3U1UaT3MxoL0KPqcNa3Mw5L2AQUm2UYdVy7Cx2lfhWRYk4jfFvshDwte98K29585fjui7tlQWW566OoBabXpWod5t1iYjVuascmqwTeS163Sa2I1GxA6oAsUNakESBtnjzvIheL9YxaB1nj0iJHrCiMMHrezRWGpz/0nk+9/SzXLh4t2uK4+huJEDrRG814nuAA4UCY/UxCkJhkVCRFm7BwnEDkDbamXwr0puNrtZgOXWUcpSScr3O9KHjrCw2WG11aQP9TgO/1wUS3GKmM7g9gn6HVmsFSyoUEbLs4JTLJJFap/jDZnfrmt8htUxCsoHwrq9kDu2TloV0JI6zR+qcHCNFmKTmHQHF8IpHZg+vEGazpXZwy/wWVKpIVEgcBkRhlzDwiEzf53XWWF28RqWY0q9JpAwRqoK0E+y4hnRdBGnGtjeRX9O7ZK+o1D4Vtl3QvhEYMy8VQxyhTISbKIzpeR6rrRbzjTWuLa3yzOev7AH3fOoOy7vsle3vlac+wpWnPoI4+ghT07O011qErS5f97f+Fm9/4m2cOH2a0oMncMtFkiTZ2IlmFmcZ9pD5nkDb6paByIfG8gI4CuVAUixSLNapVSLGa02m+gPb2QRtJ6wAv6G/Z+YENQaxfZfZWpObkZvMpne7Jg7jDJzWfAbKs8y5rcPAoa3NgPTeqe3w3eDtT3wHlVKJoakoYQJ9Dzo9xfwcrC2ClpaPHqtbaCldN0eUgNcwGMczj91MgrbZNoWZXFcIbm5MHXOuW8GsACPQTzEAjgHTzE5P89jj0xRd3YENerrB0lua3eywPeQ9YFeJ78lTRwg8j2arzWp/I3F61ekHectfeyMPP3iMpaUVlhevI9X+8Ngp2IpaSeD1ItKwQ6wgxMaSLpYqoCKIIkUYK/qewvMSypUqFVnB3gulutLawl6vw8ryMlcuX+alF1/mmWe/QHhD8JjtIkCPUtt9JhG4FvVqhTDoEPV6dJUHQQh+hE2HoJeCFAMt1aYzSJUyUC2NrudOsCjX6hw7eYrG8goXWmcBWJm7ztKVc3Q7XVTkEwW3v/cAWGnMIx1Ft9Og3p9EVosU63Xqibbz1Spf7YstLbmu9R2GhUVCsk6Gs4glKtGEF7Tm15EOTuFOF/9yHAREcUwcQ6wkKLke+k+tGx8AKGxSJCZco/FTCOMQv9+h11ujtdagPbSk31pe4OLZF1DRGoRr1GcmKNTquGN1Yj+kUJ2g1w/peRF+rAhDRSptnGIF2y4AAqUgjuP1ZXZpa22zjmuTQqyQsUIZ4t4PQtbafa7ML3Pm0jXOnV/bx4aQJeCNaOKwyu6HN9uIdO4sK3Nn17//2ft+jbjb5a3vfIKjszO41TEiEqLhebncoPPdM+I7i3ZOO4EmlGEHlq/3iSzwE3DGfcaPFak5ZU5MOUStZVZ9bXXdB55Fa18PsdFt+igwzYCADtvXZokw+tzaSulmKJvPLBnFMFnO6GQ09Htvi3KjwPTMa/ilX/xFxserQEqsIIig2VKsLCkWl2IuXEi5eF6xeNHj6pkWjeV5mlwn5DJwBbgKvIkHHv8aossrBM0lFF0kARYhJSQuNgVcHBxcXIpG8h4eIYrEsogrNrFtk1oWYbdB4F2gwVVznWW01EPgtcCDwMOUeYgqFpJlHDwOTz7Isdce44lvPMU7vuk09XEbx1aDoX6YY99ADu5NlrvKyh44dYye5zHWqeMuLbC42KRcdjh89Aivf/3rcCSsLM7RXmvS6zR51UMP0Go2KJRKuK6rvZMtizQFzwvw/IBWu0W71dWew4khJpZxv05u1eQt6tOzuK6LWypy9dyZLUsJd5rHv+y1HDk8hYr7eL0CSoXYUiFVBKkiCVJ6vT7dXkAYSqJEUi65uK5N0dl9UpItcfa6XeavXefy+Qsszs8R0uDuR5s77T4shA1SKrrNBkvXLpP0mlRkRLkQc3hSMVmXlMfr1OpjJstcSuYnk0XQEFsx4h1GosBxq9SnZ5mcnKbAWUJgaanB+ZdepBMldKNoWzP6GFjstohUn3KrRt3vMnH8JDNRRKJAOtJEZNBvrrNF+1CmUtYWxNgyM16JhWNZJoNcTnxfiYgNaUzTQZqKja/CQAcrM40vgJTEYUKn02JtbY3VtY0LvGvBGufOnsHvrdFrLFKfmqA8Pk6pXqcytYxdquMniiiGUOlcHcIuUq7VcYsVik4Ru+Dq5BbKJC5RgxpKUwcpbTNhtQlCRasTML+4ts9J7wMw9XqYmIBzL7F7iSzuBGt85MMf5uixw7z5K78C5/AsSWblYLqKoSAPwK2dZkcFmwFBzexl8aHTgOWODhE2fhlOqQUKxUmOjo8jj8LchWUuo39/jkE4s8MYBzkGWtkMi2iyPIfWb2Y6zLtB1s/32ZgnMjPXyKKmZ8krsvi9ozZzkBKEsMx7pUh86PUU8/Nw+XzElYs+C9cjFq9GLM21WF5eZJWLaPOeq2itrAckVMYd2tcTYjwUHWw8BBERFgpJQoEQB0UJaYK4hXjaz7dQolov4o6XKI6X8JsJzfM1VL/E0npeu2zMykLhpLgoSlgUKOm4yJ2IxlybtaWAZkNRcMCqS+2gO2i4g8+tlM13iV0lvoeOzNLp96iOVTh8bEYr3cslxsbqjNfG6PXaXL3Yo9vr0ev0mZ0e513f/PUUS2XcSgmnUERa2nwgiGLCMMHrefQ8j3azQ7OpI8yPj4+jFHQ7HVrtNo1Gg8ZqgygKiOKYmdlZTj3wIIcOH9bE13X5xMen+OIXXtCZm1KoTU9y+vQpHnjgBDOz0xRdhzDs4PUbhLGOaFlwLJyCTRTHeKQQB9jSQcoCtYrL5FiVUnGEbt+3goJep8fy4gLz16+ztLTM7i7Z2cY8QdFpNYmXLwFLdEnoogha4E06HEJRqpSIlRiMnVthhORXAeViEcYnqU9Or6fYXASc89f0Epfcyl58a6wBQT/C7TfoRD1Odh6jH/ioRGFZOvWxUd0CmswmaqChkdlOtA1wRn4TNQimYyFNFjlnS/Kc4y5QYuv5XRa9YXiRSjKw/BnRiJclr1CZ89L68qYyOl5jAy6V2bLjwPN91lZXaSwv0w42Et8GcGHuGr3OKqurdSYm6lTqdar1cZzx69hunSzxRSr1p110qU/MUKrUqFQnqJRq655tYeih4hDiULfrdMikJ4YoVPT6EatrHeYW9zPpfTs89jp48KgOc3bui+zXyqZr5zj38jmCvm6wWVaxjU3FpMAZ+NPuGmy0pvcw2kEtS1yhIuhHum+9jH6lDi1A5WjE5GSFIkUuXFhmBh1vI0Uv0LfQhLRsznkYTYAzD5uK+cwSZ2y3r94KmYlDZgSQoWC2YQ6Wkd7dCWFmQhUao/4wUTRbcP1qwNkzHudf7NBaCmiueDQWV1jlMnAGeAEzmqE7MoVd1Hb9CX1iukh8LBJi4wybEujMjkTYOggiER4WDpVSibFph/EjFcaOjuOtwWI8TvRSnQYVYkoMTCT0/wJhIkLauLjYCHpRSHOpQ3O5T2MlolKxKdV0ptYNBFdu+twB7Crxbfab9Po9er0e7W6PTq+DCiPSROE6BSoFF8eyUCqFVFGq1qjXx3FLJdxSBcd1sQsull0girOsRVmmNUWp5OBYBcZqNWzLYrxe50gcE0aBjuJg6lEfrzMzM8P4eJ2C4+AWirgFl0de/QhS2Fi2Q6VWY3p6nLGxGm5ZZ66o1grMTo+h0gRpgVtwcNwCUoLnhXh+hJQlpO0yNj5JvTaG4+xNVi0pJbalQ7hp/rnbZvcVlFJ0Oj067RZ6uG0znOtGSBewSZVAKam1R1titGrfBO1xUSyXcctlqhLaSneeF9AvyZ3EgJcMlsKiVsBKq0Gj0+JYpMP5RREEUUASRZmZJI7lYDkDIistiUrUuk1vRnpVonAsnbXNkpZ2btvCMS7HHWIcDr0JFheBeXRzzTBs/pbB1cdYLiQXRlMlFcfaBl5lldD0RgcB3Ahh4veGYUQYt5mbm+O5Lz7Pi2fO3nBe0Le40vEodTxqLFCruVRqY8hiBWwXpA3CRhZc7GKRgluiVB6jVKlRG5ugWqvjFAoUXAeFIo4Del6HsNsi8fskKkbEirVOlzgOOXfpKi+fucD83Ha8AV2mDx+hPj6BEjb9IGbxwrP3IsrbYAJ4DTz8aqiPaWZ26SqaMOxfLCwtsNZu4QfBUNqwREfc2APfEoGJ1Wu2GTQ5LbMxGiBmXx1NiscdqFkWFRIiK2HcgZlIH5uFyO6i3RGzqA5ZGuQs8dUiWts7x0Yt7Z3CZkBiM3VCptmtm+tnxDdznt8tp7b1bO8xBLFiZSniwjmfF55pc+a5Fc69uEKv2aZLC59F4CJa07uIltIYWmde4uqVJvFqB4WPTYwNFLTBFBIbC2mi7ocEps8BKOEwc6zGidfOMnZojPGjE3gtl4lSiCSm91KPS6yv3QJVoEaJIi42LjZFSggcFD38rkdjocvVL7VwHahUXUpF2/R5Q0u/pmY7hd0lvr0GXt+n0+8yv7DA2kJ0S8MYCzgy7eCUKpr8uiWKbgmr4Ga6Doqui1MqUavUqNbq1CpVatUKpXKJktHmFgoujlvAsSzsYpGS61Kp6PM5BYuCVeDYseN0Oh5uuUKlXKJQKGAZbxHP7+B5Haanq5w4NqODvRdsSiUXt+RSsAvGSUvguBUKTgWn4GqHjz2I7CClxMbGlha2yfJ0c1I5KpRI45ROvw+dNgMmEaGJr42U+lOtN+6MXd6onhiltYNChzSTrkO5VqNWq0JrYKx/q05NYDpGqb2VE5MRNVMEhsBKY4VGa41+4JMkoFSE3+8T+D5+4GNJi7Ih3dKSuI6rY/5u0PJq0puoRJNkEx3CkmxpI3wQUUAPLg12Nwh8eRq+7K2TnH5smhfOXeaLMtgYTHQr2FCchNnxGleude5NxXQzJOseZIY0pDdOwMx3bWIrCcMQz4+ZvzrPF557Ad+7eXaNzFmnDdAJoJPlxBrAAkrCpVAs4xRciqUq9fEJ6vUJSpUSlWqFQqlAqVwkikN8v08cR7peKTQaPTqyw4VLc1y8dvvQYdPTszz0qtMcPXyMsfEJYuHS8SJ+fyTE9zBwGo4+DIdnYHZW726vweXL7Fdtb4aFhQXarTZBEGC5Euk6JCqzAmdIQybvbOZ+h/gGc6nhUGMKTW5raG3selpfU6eK0sR2Eqg7ULOgTEIoFeNlmGkNElpkyMivY67VQ/cVw9ncGujXNpsHwO3XOaum/lk0CcxngRtj9w6v31rcEExq9JAQxYqep1heCbn0JY8Xnm1w5rl5Frx59FRhCT21vYBef2wzSDN5FHBYu9yGtEcRD4fYGCdYJn6SXJ+kpER4eEgkDg6lUpUjx6o8/OgMY4fqjB0eI/LKNCZtiBLa822CZkiDAJ8A3QKqlCjhYlHAwrEcJDFp0sWnx+pch2vnmkxM2Bw5bqHUIJ34Vq7vO4Hd9bwSAqdYoCKrTM9OYxWa+D0PrwVEgzxftgCnAJWKZGysBlKbN6RKESYRMkiIwphYKbpSpy9esW1s4VAqlRifmMR1i+vpcG3bwbFtbMfGKjgUrIImvE4B1y3hFl2jcRRMTE4xOTlJuaJjXEobYuWboOwRpaKtl55tC9uyIU4J4oAwUERxjJR9pGyY8+mlyre96fGRinVzpras0QgpsQoujluhWBnbPKaNGDYgiGKBTs2WLQjp5ZYEh0RZKCUgNqGTFKhB1uh1rGeQGhEkIC0Hy4JiuUx1cnID8R1GAZ1/vlx2KZYryIKL4xRBJliOot9apt8KiPqDYy6ef4kE6LZaXLl4Bd+PaHVa9Lt9+n4fy5LU63Xq9UmOHz/OsZPHmJ6cYfrQrDZjGA5WnygSJ3Nu1eYOrxSNbw3dPWeRVncLp09P8siDp3n45DEKjFNzrnJ5Zp7ry0NZ1RNMbCVgGqaOSR4+eYLZyRm8lWdYfmnn22fBkiaetSI2S1sKHVtcD0yDAUqhnc0aq2ssr65x8dIl/Na1e65DAnTTALyAgudSjxOkion97nq0h3KlyPjMBFJKEuNf4CeKKITQ7xBHARcv3r4u47UJJsfHKFs2qBi/38WL+6y2RhH64SjwGpg6DNNTUHR1zO21VW3bq+40+sPuw19YYnVtlXa3S9Up4eKsE8vhqXCSZScdETIN7Dj6Hc6cvrJeK3MI89Hjes0FJ4BJH8brMD6uFQeR3yfoJBBpolnb4lorwDXzW58B4W0yMIfIiHFmaXqryPWbCZAz9Dmc7Q30NCgbQi1zzRFGM7wBkVJEYUqzFTN/PeLMGY/z57pcvd5kxWugpXPVbE203ZaN1vRW0MQ3BtrQPw+0kXg4pNhYFChg6fyQQ/2KieqCwsGh6EgKRRvHLdDtKpoXAyDFTktYbpVyfYxKs46Hj4+HoIJDBQcXG7MCbdv67IlFiqTTjJi72uPIiQrdVoVwRmEX1CDV9iBn945hV4mvWy5Qdhxsp8CRE0eRCBzbpiQdpEqxlIIkJgoToiRGxfp/Lwjxo2jdvCFKUsIowPN8vL6H1/Po9Ac6oow33YmXZbVUoVqd5MTJE5x48BgTU+NUyi6VikulVqJUysiyNoVQysbreURBQK/n0Vhr0253CIKIMAzp9zw8zyNWir/3/e/eUTnegPXsTplBoEKmCmnbFNwKbm2S0tQRuHSCjeHBRw2bhCLYJYhq6Ll4BXDxlY2vbJJYL+mnSpusZAlP5abGvjl/1U7CSnQ8XMdxKNfqTB45gn3xyg2a3ing5CMnqU/WqZZrerXBcZCWg9J+1SzNu8xfu0LfjwhNlZdbiuVnXuBzz7wA/PYt6/LwQw/xlW95C294w5t421c/QX1yEmkyvm0w+JBgORZW0cFyD76N77A2Zbdp/KmTx3j05KO85uRpHph8mDc91KbdbdEJGrS9Jo2VJp22hxdHeJFienKa2UNHOHnoGNPjk6hmwode2nmNZKVo6xi+fkgUG3sL5RiPN9YDnmjWqwj8kOvX5njpzFm+8PmdX6YPCUiVwu+1CDtrLHmtdZ3okSslipUyVsEllZJWL6DVDYmC1i3PWbZcKuUy1VKJUrlI0XHxPY+VhXn8OGWt42u74B3DNwIOVOswNQZVF0omjnt/AV5+CbwP7+D1Rgi1zNzcddbWVnBKs1THyiRmiTp7i+LImAQmo+vvs2xqk0BNDvL1RAzsbpsYDbAL47NQq0GtAo4s4lhF+hF0mh1WGz5+X9d+HB3w6vqm6100dzfFgPj6DFbmjqGJcWaPW0OPetmKUpZNLov80AUyHWMZPUJlSSpqQ2VX0aYUlrnf4WvuBsIIeoHi0qWIZ5/2eOn5NhfONrh8fYmYJVO7C2i7XswdjFGghqREhENCB9ad3B0cHAqUcbCxcCnhGJMHHTsmXh95U0qAa0ssKUliOH/R49LVDpVagQdPlQiSGoVSnSrj9PBoUqDEOC7jlCjhrEd/sVBYCAoIEtorIVfOtjl8pMLpR8eYOeJQsZVxcjPT+h0mv7tKfFtrLYqlIk7RxXVcbNfFLTiUXO3lZ8c6dmRsyG+v16PXa9HptGkaUhmjtR9RGBMnCX4/wA82kqK7CSvS9Xr0PJ8o8Wl1ViiWHBQJUirK1RKlskvNLO2hJCoFzw8JvIBOp0ej0aTZbOmrK4ijkChONmZCGgHSjAoNe4alEKqUrh+y0AqYa4QstWBgrZQtFN0bsgxxW0PHNQ5RILLr2uiuIiBOY7rdmE6nR7VWAMtCpUNWYOuNfBfCOgxfzvjubjVMOA44joIkIIrASnwSy9LaCpUQEUEYUHYd6uWIfvfOO8Zz58+ztDjPtSuX6fX7nHr4IY4fP8XE9CEdF9qS68Ze0tIpiw9COLNbtxX9hHto6e96EHh7nDipEkQlvUxsK0pjFiUqTKppJksevY5HM/BoeR6VUoWSWwUcgkjpNOUjwPFjh4jCmNVWF9Xs6bB/Q/3JeqrrWNFqdWh1Wlw8f5kXnj9Dc2nnY6DPVKY4cmgaqRJU6NMaIr5rPY9q6FGqVoiRrKxt09IySUhCnz6KMIrodPvEKKIood33CaMd1qm501q9WHKzF1pnVVhagLnL4H1mZ683Yly9OsfCwjLVsTJj9Spev00/6BMFisiPcIouZbeqs6CMCB9FT1qPAkfVILzXJNrON4vje3gGZiahPq6d7ZIEIt/H7/n0etDpQ6Opj1foUWoKTdM26/znGERd6HFj36JHn8GIVxn6xByrGPQ1MZoAdxn4sWYEObufLA5wVna30WkkXLoQcu5MyEtfbHLubJv562vEtNFTi6aptQRcBHVc406mEAgiBrF4FQXjvibXtbzraidj4ytx0VFZYiI8AlqtFu3VBo2VMVaXFMsL0OsUqLgB3ppHGAukcCGNjZlEk4AukjEk4yRRHTcaA6tgHHQt/ChFLHu0lgIaSxH9rqKo9WM7Hs0hw64S38bSMsVSCbfkUinXKFXLiERhK1BSkiozQCrtoRwEAY1Gg8WlZRaXengD0zFsS5shqFi/QDfcmIBCwcGxbO0FncYIYeMORVlI4gTP8/CjbE6TsNpYZrVxc5uA4pjJQrsvkv4oQyjiQbtI9YsZEbPWD7iy3OHstTYLS9liT9Yd3BtsdCdyc32OblrRulNd1tSyl8+m6TlU2g5jNRvbLUHBZWCuqlu7UWyNFDpqglxPkpGgbuhIM2da3+9A4uN3oI/CMlq3KEmIVARS4lqSeg2iABZvbca+JdrdPp/4xCdZWl7hzW/567z5rV/No45DrVanWq+ve2hbcn9HdZhBe3ZX0G1yHvjSTcr6DJYjRx0PczPCuEIztFnpgReA5yVIW1IqVai4NQoSxqoK2WyiaIFM8DxYpkc3UCw1myOp1+kHj+L5IdbSKn6cEIQSP07XXwgpwbZt4iSmsdqh3W1x5sx5rl54aUT1Oc6RQ4cQsU8UeLTWFul6mvr6gIzA8j2iePujVEBMEMTIwFuf4o70+W+YUCfa9tUCVheg+3n2htLcPS5fuMzclevMzkwxNR2wsrRCc22VVrtLe63L7NQsR0+cYKS2Duh396LZMrjAqzF2vDV4+FGYKpcpOxX6jSatZkSnCc1l6EUDs/o+A6O4ujl+s9b3VvF5M9t1i4Ft8bDtbuZ4fLMpVcDAUrZkyu+PjGwxLz4fcO7FgDPPrXDlahOfHlpqLbQtb4rucauMMQXYRESERCQEQIhFEZcyLi7OOv3FEN8EgaSAg0sBiYtFBY8OK7RIkoSlxTrOfInGiphWBoQAACAASURBVKDVlPieTcl2STodgkBTamV685R5EtZYZRbFYUJmtM13MgaAxCYhpt/3aK94NBcjOmsp1UlBScpBXL4dxq4S39WlFdyCg1ty6ZU6lCoVSq5Lu1jEEYKClFm4flIFjUaDlZUVGo0e3U1+BnGWM3ALWEC9XmV8fIKx+pjRkqRIqe16BUIbiUcRzVaLVrNJt+fhB7d3ZvDb9ySCnYdJLwpKa0wVdLyA5bUG5y8v8dL5Bc5cWoJ29prrGdy9oHDbM5gAM9LSETdkFg/KYTDXjkiJ8XounlemaDtYOENOGbu34G1t+txKSZ+l0FSRTz9oEWVhoyOwIpCO9vAv16Bar2IVy0TVPv014zx0F3j5zBk63b62P5YOJ089RLlaw3FMNAfHMsR3byKH3Ax14AE06T01qZVrvQDqDT1Y3czac5jw3E5DvJNoKkmjr6h0Y8I4Iohis/TpgFXCKdnYtsSJQEY6goFPiIoCYgntYDSz4KNHZvD6PmGSsNbsoYjxY9MSlQ53FoYh/XaPxuoi80tzfOnsaEJMlIGjh6Y4dmQaFQf4vQ71WhW8gRmCB9iewr+DqWqy6XP0GI4tMOQZ2O+yvYS2+wtf+qvP8Zmjh1FxQHt1gctXL7CyuozXDwj6EQ88eArbcui2d9NqXiNAx9aNgMokjB86gRNBEiV0eorlZU16s4X3jJRmo7DDIIvbzSINboUlBuYXDoMIDJmJQo/t98n7Qr+1joCXn+9y6UyHK1fn8dc1vE0GuescYAzb2NZGmFXy9eRTAQKJQ3ldKz5wJBPr/wuj99Vmo1Lb+tPRdtULY7jXSnrC0pAElkMhKEAQEnciklRHBNZPYQ49bemyhkdMSIQ00X21y5x2cAevq2j+/+S92ZcbSZbm94O5mxscDscWiI1BJllM5lbZmTVV03t3TfWMljMP0tHDSOdojl70v+hBT/oT9KwH6ehBsx316Z5Rd1VnV01WVe6VmUwmyWAwGDsCgANwuJu7QQ/mHkBwSW4RJLP75okMRgQC7uFubvbZvd/9vn3DsGfoXnCs4uc5XckXq+pwAKCpVDSuO0IUTToVURjkCTsHlcBDa2v0lRkLtp6kcdqtQLPZoNNpsbyyRrfbITOGWZ4X72twhIMrXTJjCIKAehBw1O8TRRFJlqC1xnFA1SSiYhgcPP20HCoIQx/nHO1kZwYWh0bZTLez2+PvPvyKv/vVF3zwqy/JNjeZP+q2DPJ0U8npKGVcHh0hqBa4hdeOV05hZTGpDE2azebN6wiEqJz0cr64cE6O+CgVoBk2aSKchMkADgens90dbbllMoSq40BRQX3eXOy9u3f4N//m33EcTfjzv/gZ3dVVhJTImkK+Yjq+FeBNrGBOF7i8Dlcvge/bTKq4Cb19u9A9bhmW2AXwRYDfnWRMmI0RyZDMJGRGI7QhikGZHE8oyAQDYRgrS7MVxvLVpMLe6HOIZujjOg7N0KfZqKGzCeNJTtnrOYk1k2zK4d5dbty4zvadG5x+ps/ObndJuSyFPp2wik7BNSl+VZ7aoMx49k3eCwtHzIteovj6lAbA9yyye/zN//W/86sPfsGljWXubd1iPDqm2V6jvbTC4Ni2mUU7uy/l9EpAiwPTqaR30Gd60KN/AIPo0ZnXsmHOw7ZmdbAQ6knng2dxbHv1I+Krzw7Zu3PMlG3mbXslOaQBKKr4uLgkJGgSUsbYq2Fz1xkajS5oZTOck1pLmUAR5OSkJORoUhKG9BjSs4bD/RB/SzHoGSbDGRUczKHCw6UyE0zRxBxhm+y2sSzsHEiISEhI8IlpcQmfDgKJoIrJJIMe9A8N44Gg3gLXBVmi1DMsWrxQ4HsCH2eQLW7tHhMO0KiDb2AYw+who9+TEAZ1fD+gFgS0Wy3W1lZZXV8j0xl5lpPqFJ1qHNehqqogIAoCgnod4bk4yiXPUrJZTqfT4vLlDRqtkN9+9BtufLPD7Ak3zW9eWeXa6z+g2+kgX1DjkUGQGYPONNv3evz8F1/y81/9jvT2V9gi8yLrqZT8hjnn5ywjAL8FoiDpOB5lU9v9ofNCzaEY/qeAPC8m72vXPqewAn70AmhyuzlLJnaPvRg9YEODzMHecg8lJUrq51ZFOtq/y1/+v/+B7uoqP/mDP6AW1pGOh5TilQC+ZUPIVeBtD4I61H24dAkuXwkI6oo0MUCf/X2bkbnJo7N8FeaNKbZn+HxjX49pJEOc1BY2dUEdkolBZQLfNbiOYOxoYiUQxkHkGUJayVtzTgn3MAxwPZfmOKAR+kSTDCFiMIY8y0jSlHF0zM2bd9i+89kDv18NLjEdD5j3oisurWywtf/wrPDDsuxV7L1o132WGlXaDZ94AuSaWs3D58UqcJxpnKS8Ts87378wTO99yTf35hSXwegGg62baA1S1iB7OcB3SkEqcSRTLdndnXCwObf6Lef4RSOI8msPu2KEWK6vZq7r+48zxnzz1T6TE/WGRb0KnxpNJBpFiiZhTFyA3hL4Qpn/TkkKasOssEKHxWcgO8kTjxkzos8hI3qAoEeI2lKMcs2UlBkuycxH4eMTokmI6WFB717xjoeUbOmUlBRDyDpeJcAREld4zDKP4ZHheD8jGrq0xiBKcvYZLwIvVs7sGcIDrlxp8IPXX8cP6mTGkJsZWW7IMoPruohKBeX5SOnjCsvpdV0X3/cRAvzAR0mXPDMkSVJQHwRmluErhTGGTt7BVTYFLSqwtNzh6rUf0O20cAQsd27x8UdfMHkCOt/123vE8ZhOu413jqCkIoqsrxBkOmN7t8f23iG/+ug6v/zoK6a3dyi7N22UTPFFoOtwWj9MLHx+1mxRF/wAmw4TdlfiK4gfBJUJCUkyRWU+0jw81/uiwK8jLJCs1+sPfU1eiGZIMW/TK6MGNNtQCyVS1izHTOYgB2ciBxoPDvnV333AxsXX+NOf/pR2Z9kqURRqFC8zfoStH1wGrl0Bv25v//JqQGupi+tKEBpVi/GDhGD8aLejpeK9Auzoi5lPnecVGk2SJcSJzXAYk9l5oBgPIzHGQZDohCRLAdts4BuBj6CXns8IdV2Bl7kE1SrNwOfYi3GB8XjMZBpxfDzgcG+b3r2H261Px2XTSxkJBwc7jzxe219iEh+RY++PiwW+fgWCIAAEyVQTT6aMRyOciqDN6fL0Kx+LnMFcFF3jAup1iJd49egOJUXsWbjHhu0vPic567zGU4TEXvKDXc3NfJPepi3QJ8ytge+PKXY8lcC4fJ8AazXS58X3AbwakTPhFraucshcrsynhqKOPBEfs5SCEvAm2LtQoVTnnZGQFd1AkGJIMVQLQOySk5IVENZCaNtC6OJaQkQ+o4aLRJ60wVlI3SdhwIxhcWwfWx8uZ3Wv+H6PMYfIWZMwb+H7PkYbhr2IvS3orGk85dNdk3hV7wHDnueNVx74rq87/Pif/og//pM/ZX3jUmFdrIgnCXmeWceiioBCESCZpvT7Q6LIWhiPx2OajYBWuwMY4nFMMo2J44RpAiLwkZ6L8hUt3cFTEll1WV9f4803r7FxcZkfXL7E9vYWYdDk53/7AZMnqOdt7Y3Y2jv/RomKgBlWP/PbzT0++PBz/upvPiG6/mvmffJV5oC3FCUt99qltaDADsrKfT9/0oJRyaDqAOsQNuZvqzyoBxA/mBqbEZMkCVpPUcbnoXlewaM5CGcRTnEQB6rVGp3OMksCju5bMAxAbhvBm4mlOpTgNwS6ywpZayFl0zqeV8ERi21bzxef//YDcgSrq+u89/4/QQhhaQ8vGfj+c0AJWL8Gl94Brx7ghx1Uo4UfNCx1JYrxwpSgs0NjbEfd/XSZi9gOcB+73MfYEaywmZ5HLZTPG0ZkxDplFI/JyDE6IxcCxxEgMpvqB3Ru8yAgyB1Q2kFlgtHkfMZm1XXBM4S+tT8/UAMEhuFgwN29AXu37kD+cGc2Gz3ub8uZzmIaCN66epWPb944AaxVJ2Dl0gWyUcgsmZClCSadoqoeNVUlDNsYI5hOU4ZRxGAwwHNhbcUl28/OfXNypnEfRQzj2F3r8WXInhD4uj+AK1foXLrAhbUVrv/yF6S3PjzjExWsXniTixvrHO3vcHvzwaz+42PI4ZcvT5O4ioVW9yIw0fREIaFcYRY53mbh37YwPm9yO5FLwzbNllrBfV6VxrMXFZ9hr2gPcHDpsEaAQuEgidFMCshqwWe5jYA53DPAFM0MjSFDkqFIi1bAGTPGDJie+n1BhRAfWbxLRhOfQPrW/Cn16M8G7HNIwi7zDG8Lh5AKLoJKMefbdMaQbURhbeH7K2RpxvFOzNZXCdIPQRikG9Lp2ib3swS/rzzw3dvP+eqrzzCznPWNdVqdDkEQonVKVjgeGAOzLCPLM5KpZjCM0EmZuRH4MoPQRSmfoFXFUC3K60WTyAxmjgThUimMKcJmyHK3ReAHtNsJJst45+13SJOMX3/4CVH/8WDm/ffepdVqnvEVOY3GdGaI4ymHvSFf3djk7//zF3z71U3mRoqFWDQwz/imnPZhLRGqZC4olTMnyz8un1O+f4Yd7HuwJUG59iOPwSTMJcUXc32SiihLjRVOn+vCn3zOsnAW/DpUa5KV1S4bF9c5unM6O2b5apJmR9CsafKJYTSyE/BrTehcfLPQSpRoPSHREyb6bM/7y9/+gs+++Bf85I//gEmUWLke5+UCXwNoA3EfhofQcBJUKBEiQPod4gyixNCLcvrjh+eu6liO9DrzrZiLveYN7Mi6dx4nf0nx43c22Fhu0WkFmNxgyDHMraAt7jVokxcVJ+uNJYTEcV026/I71E2ePT7/+DOrm5sbBv2Y8SBiMh7ROzhg79Y9yB8nWfZwSDDFUFWSNeWzlcRUgKV6larIEDUPPEE6dUndwrRUZ+weHZH85iOUdBAYMJosT2nUA2qDwQu2r3rOEI79WKQ21WrQXoGDco68PyRwEZbXYX2N4MIyaysrLC21WWo3UA7cqkl6X3zE2bFLDa5JUGhq5+i8dp6xD/yOec2xrA6UjWyLbYb5ws/u/7qkPyz+uwTE3y8djueN0oWtjoNPizY+NawCkp25DHmxQc+wufHyKpfra0p+sr5rchw0LhXkCfFhtgB4QeFRxadFQICLLLLFhkqprlkFE8/QpKTklJloh5AGLYT1bCPBMGTMjJycPXrMrMzagUbgkJGTyzr4LTw/pbMi2Eh960Drnh34fWnAd1HZ1TrQQ16BbDbf8QFMc/j6d316hx+wdqHL+vo6rW7HLlDGkGc5mc7Qqc0cJnHMOLKal0FQp9Fo0qq7kAWo0KFVb+NVFVJZS+GKcHArAhl0cP3QToi4CBccYQdPxRi8ikf6Zk6gAtqNLp//7gZxktFud9m+u8vR3p1Tf9/apSv89L/4l2xcvHAm12t2ypWtDEtxiMYxB4d9rl/f4tO//wJGW1jgWw74kjFZDpvF6WURaJZ3RGB3eos7xIcVpkuzx1LwpQS+uxBPIS76apULyrFc37yN3Q3aqOAiKvcndB+SQTvHeV8sHFzJKqvdLhcvXuTT+4CvvWKKemeV1WYX5UhGvQGTwYj2+grN9YtEkymjaYKODNNcME3Ovl/9i0+/YHtzuwC84qVnfMttUdy3rrcqyAi7Akf6CNUlRzPWI3qjjEH8YDNLFbiCzfau3PezmPmtP3Pg68Nrf/Amv//OOit1n5ay4/00GUjYxSTHGupoC351IcGIkHzZ8M4F+H7xySdWicavkRqXKIqIo4iDgz3Ib/KsD0UO+J7H2mob78BQcV1qoUJhqPoSgWLqCmIXptOEJJ5yPBxwY3cfF2g70G4FNMMqzTDEr36PgK+zQLc6ee5zUMo6uB2s8dCR9v5/x+rVazQuX6DRbNJu1KnX69SVwq9JlrtLXL76A25dvcaNjz8k2jobSTknT5Ak+M/Ve1fnZcLDb7GZ2kUXtEXgWzauL4LicmSXd6jEC/l9/y4zyt+X4ff8EQGSGiEtmtRpUMUnLRrZrLdjxuyBq1j64grslSvscoscbH7Ccy/X8vLrOlVCAhrUaRW6wOLEP0BQQCYBJjakhXhaBcGMgDoNGrRw8ZH4jInJMYwYAbvM6HNESl5QMg2CZKdBJmL8EK68FZIXqkniDJt+XgrwdZhbkzrYP8oIS7mS2ORebmBSrKjTGezswSQ+pD84ph6GGGOYGYPJZxiTodOETGNtMhPbDVgPe0SDOplOiEcDgnpI4NdRtRpK+XhK4QgHz/UQQQdHhYSNDo12xzJckzEmi6n7PspTdLsruELi1xqsXrhE/3hMOoM0yx4AvjozHPdHqOpZug4VA7kEwQImccLd7SN+d32LG7d2YFRyfxYbyVLmCpmLfjPlg1BuP+47zsnPAubT0eLrZlgYswjucmwzXR/rHBMALZygS8X3yIYC4jp2qsro1Kr4NYUjvVMA1B69Usg2PfkVetYQxblLKQmbTTYuXqTFh6cYkimQU0VWm9Saq9SrIVJNqYUTVLOGdkK0gKnRTPOcRNup6Kzjq+vX+fLLL1leXqXb6b70vpyyXqCBJIEsExij0LkiMYI4hfE4JYpiRvFp7h7AGlYFIgR8D9LUjo5yWj63Jsf1DkFouavkOSIv6Q32aEJYtQ8jpB2HUoLKyXNjq03SRbiS1dDn63M4vQ/+5u8QwsULAnCr3N0bsL03YHS8zcNB7/3M84eHAeJ4TNVzubi+ZP9e10M4IF274GWFQ1OmM0aTmMHMnFBTmkAYhgSBR1Ct4rpPdtxXIvISbsn5JTQGlA9iCdQGJA8C37DRZml5hbVLV6g36/hKUVMKpQRKSmpS0m40ka6k0V7ixsfLbH/69zwv+1nHQ/SojkmeQy/DVZC93LzoJpaiULY4L2rtGk4/6yX4vR8kLwLjxRXn+9yW+CzhoGgRUqeGd8LrzciYYtDMB/Zi4mrGvJE9Yk4SeVRiRgA+FVwkCgdlQSkZo+IYDh6+CfE8n2rYxNcZbqwQBfc3xz/5zyNAEiKooslJyUnpA0NmKI6L44HCkCHuwcqGz3AQk06tX4Hrluf1/PHCgW+FecK9/BN08QSYAn+5shDWMFA6EWfA0RB6wxxB/9TtOtFfXXhPB4gGcKRG7O1c52v/ul3EDDiug6eqSKnwXInjeeSVACN8rr75Fm++83v4ShIN+hiT8O7bb7K+sorv+ciupNHscOXqG2zvHrKzt8+dzQdLjke7B/zdL/6WIAj43/7X/+UMrpw5oWYA1iAuNxwPIm7cusuvP77BnbuH2OlDMieVLxLcE+ZXvsKc2rC4EyxLILP7XltKgZfZ430e1DZYjLh4HwOqy9JKh0B5mKSDiceINMadpfhK40uN4ylrZ3gSlYX7OWNmzq+dwSlHUA6OlNTDkIsXX+O9t1/n5199e+q12kg0VTQ1tFPHqdWQsol2DJOpZjLRTKaGSWKYTvW5wIGDW9/yyccf8/Y771Kr1V66EpNbYEWhyoUrQOcBcSphnDEexsTDIXE/PjUawWZslrFbJL/4Xszch2hI6fV3LmfOYGzYPhxiAkXmK1zpIl2JK4VtYnQcq4AlrHyZwMUrHkKpXFyl6DQeVCs5i/jrj6ztcFl8f3zt4MkKgQI4ONhjuRnSbtdxXAedGVIDQpgT2cdZZkim01OgF2BldYXXr12FLAE9tRvU70voIp+v89MIStZAVWFp7aEJ3+joiMloBKL4pTwHoyGTGMfgSkW17XLhjXdZWr/E2sZlNq9e48ZHH3L4TNxcG6PRMfGxIJk8h3ZGVb50PsAxi9oi8/G8+LEIehdvzSLoTR/x9T+eCArCQYCHhyE/kSzTxOSkwIxKQVEoqQ3zVuEYC3wfd9UsPhAoXHzAK/QYNCPGzNAIAgK9Sl01aa9fIjGCYOsARQ+HjBkGhYeHi6KKIkAgC4G0lJRecS5lBbgBhES4uDNBr99kHMUkicF1xQNpt+eJFw58S/hyf6sVeQGIhe2w91wLUtOJ/fmiVuT9C8D9+cbyc5pjCZgPzBk5nMh8nI7N7V2iUYLvKw52d8h0TLNWo9Ns4bkSFfi0lUJWfVRQx1Ch0Wo85A+N2bxxVg5K99EcZqV0Wc5Rb8SNmzt8+sW3HG/vMheyLoFsWQhazMz6zFUSSzZlye0t78giZCuHiVj4eBIem52enEDR7bZoNZuILEWkCULHVNKYWd7HZAN74yuL77/w1xsws/NL+564xRmDFAJZrbG6epG3332fyWjEb+7OW3eS1JCklrMqquA4CjyHyXjAYdQnGQ2YRCMGvUNGo/MogANM+OzTzyxAf+3iOR3jyUP6RZK0SKKZXJEaCUmOmSTE47H9GM2n3/J5VtgMYoB97rPMjtRySiwFe84ln9gbc+/mHmDYafl0fIVSHsqTuNJFSQflunjKcnndIiPqOg5C5DQCn6Dh8AS+N88UT2/u8GSbQwMc9yOWwiphoHBcQRTFpNOUNNXkM5hOpqRpQprpU8fvVgSXLl/i6rWrREcHREc9pOPxvYEf1WKXVlIeTA5TbcGhUKBK+tZ9880333CwssTK0BpCuGpeqheAUBJkjeZyE3XhEktLK6ysrbFyYYP/5/8YwvHmM51uBBzvH/FcrQJKvXTgC3Y/UQpbLrZcwaOb3MqOlPtpEN/hX/UPOuoo6lRRSAROASHjAvwmJxlfuxlVzLO6CXY0PWnlwE7oDgqBxFApGudK+48pERrjuHjNBssbG6Qmo353C38WnDCMFRKJiyr+E0gSmxrh+IS1PcDe1bLO59JHMDgeEEUJSZwjlVXQOat4KVSHkg06YZ6jcLCbaS+zQCRDonObMatJqAWSqvJwXAdReNZbuoM1tc3ymCyz8l6m+MhzyFLQsyd/SPa3d/gi+JLRYMDRnuV4rnU7LHc7+H4NX/kYUSFHkKQ57Xaby5cvn/EVWoyFGc8UJVgH4olm53DMJ9d3+e3nm3zz6U0Y3gCun/6dE7WF2cLXEvtQlN42JQAus8LlpuAsFrMOy90u6+srNMMGJo4xSYxIPYz20LEmjcfFhFbc14LMIwwYYSzd/jz5DnluuTUn20lBvdPlzfd/QrXVJvmP/4nPb1kzzq39PaYaomRKZ9KlVgtQqsadu5t8++1X9AeHjzzMWcbXv/0l3dVVXn/3Xab65YIOUVCMMwNxCnESkcR9cpGTEhOP+uRJH8w891BGSbLRQN/YeWEPW0/YY54lOpfoj+G3Y+79dvPJ+MPCtSLNrrK89avrNK9uMPj1zfM8y6eIJ9sezIBkBqmxE2Y8Ttjf22PnYECUnL4/ga94I2zS7LRZv7jBhfU1Xn9tnUurS2zfvsm9TLO01MbbHz+RwdBLj3evWrCba0gyGA1hPymMLRzY3uehNJLsM8Z/8xUfApevXePC2gXa7SWUkigpqUqBVlWrre0U1Jn2EsvvvMvP/vX/zL17mxztH9Hb34cbv+Vp5tbbzzn1NVfWGBw9G/A+yygd1Up1lkWwe38m737VhzLK1UtwutL7CuD6h4b/+Jc8VdRQRQ43YVbkTTUJ2QI3Vyx4ss2TfD0ebdL8sLDNaeCdsIDjE50Nuz4rFJ1LDS6+3eXtP9og6MKwv09yPWHMETF9qoQF4HWsug8CnwZ1BA5rxf2dYZ+HYywIjpmR0h+2GA5jBlGG9K2xlXNqT/rsQPilqzqUcKycsrXGer5XNGnxw1hDXfj4fojvB6jqvLRojCYzGXmWkOkxOktOjCq0hopj3y/VpzPH3xWbtzZBzzOam5tbjMcTTFFuz41hmhpmQuIon5rvW7/a/MVQ7I0xjJOU7YMhN24d8O3WAQx3sNza+2fJHAs3SoKJx5yuUO7LyigL0Wfov1QPCcOAMAipBzWygsuNnNmyo3HIUnHC4108ewMIM8PMzhn4wgNThazVWL/4GtVayPVbd06AL8DB8R4GmEwnhGGTWljjzt3bLwz02kjYuXuP0WBA8rKBr23+P9lsJmlCPO6hSHDxMbEtsZUU7ofx80qdkXKK7nHOoPdZwmQQZ0BcVOgOGeyMYet8ZPXXnZAkn9I7h2xqRsGdr0CWpRwPInaTBwFEKFyWV1e5cvUK7/zwh1xYX2GpWSOoSkaDI3p7irARsqoE9xLzymfhum9cQGvNcBQzOzyEUQLHR5DoQr7ju4weNLPffMxtbUVqkolBVSU16SIFKOkgpcB1HRwp7IawWuPCW++wdOkqo+GAeDDk57IJX/7lC/qLoebXzqX58lkixiolLxpWLI7uRShTAt9Fsc0yTcPC77+qHF8fS+E6y/BwEZgC7Fr9hvQE9NokoI1yfU+ZC0M+TUgcaggkM8TJu88JpQIPSdCu0lwJWL7UIE7GtFaWCG8dgM6YoZEo3ELr156Vg0MVH6hRJ6LJnIKxyEFuoZOEJElJYmOZRXBmsg4vHfjeHyWVIZ+d/t5+b8h+b4gD+H6AIx1cIZDSRXoSIQw4AjNzSFKBzi0dYAYgLXUizQrHuEdEd/0Sh7v7p0AvWKDpOg5VpfCVQhuDERmDccJkOLIc33MDvfMtjhAw1RlJmrO3P+Trb+/y5fVNDvfLvfSjOLelKFQLezVLiHF/Y9pZRxcn6IBQxEmKdKVNv59EKbddfhT5P2O/K4TNAlOod5xfOMX/F/MOAkdKVFUh1WlrVoCj4z2OjvfwpELKKuPJi19aDg/3GfR6TKLohR97MdQCaU84YEYw3ttBtFx8OvjGwVcufh286HSmpuzOLqe8Ed89kl+52Dq/bO+VN98lSWLUzj124rPdBhhAuC6yWoNYM52ah2bNMgzScwlqVcK6T1DzEAJr6ywEyq8RBCHNZpNo//iVv28/ef8ymdYcHfe5vZUziA9g6xDMLhb0PmbTP/oQPphwZ2ufO2srIASedBDkOMbYDLByEdIBBxxX4EuF4zjoaYKJNYw0L1JpYTR69bz1ysxtin3W9X0/KyPHmgOVDoKLGd+yOe5VNZsuz/NsI8OQkhVqMxk5sxOKol1RNTmaGFtTj3l6BXQJtPBps3hmpQAAIABJREFUUCkMVGyG3S/UHwBS25hoUoxJ0IXVe1bJwAVXS3wCXBT2zs2hpgt4BXs4OpE6KO+iwLY6NwmcGtJ4oMWchVksws/b8fPSVB2eFcbkwCgeL1BMBVW/hu+7qMDKcSW5RmcOxjggwBUuFVfgpYbcJMzyB7OHr7/9Lu++9yP+41//NaPeaTn2PJshvcKCVimEMRhc9HTE7u4+m5t3Hng/sA9s7UyeSmHpr4DOZozGKTuHEde/2eHLr2/B/j6Pz9KW3loj7J475vxZUh1qQYtKRZLEmlgkuBgcBOIE9FogPh/IBnPfvr+ktZxXOA5Ws1Vwot0KIKWkWqtSrdoSZvoQol2qE1L9csR0hnt79Ho9Js/T+HIGoRabhw0whvEYfJPhqn2EalFXiiCMUDvz5hY4zSwvt2NDntwGd3kD/tV//zMGw4TtrR63bx9y50bv2f6Qh1A7X2ZcefOHTJMJ0g/gZoU4HhMxPpOn1gJfiVtV4DrEj0gIGFO4YFYVzaBG3VcIcmZZBq7Aq/nUwjrtVpPo+JiBfrVdtf7p+5eYas3Bfh1XTPnNFmCOsRSxJ40vbJZ/y4rv2VKwpYmNcUFUrb2jENbtZmWNSr1pM5dGvATg++oRAUputOHBbpH7CTvltr5afJbMFeGnvLrAtwTrZxt5AW0pMr5Wt1cUzeAWDKfMTlEWn1ZXukWVJooQkGRYAwmJTUTNTigVwuacTUpmNDkZM2EwjsDBQ+HjnGR7bd6+AjhUkAh8XGp4TChKhicm1R1qNKh5Ph4SUZYDzdnNLS8F+J4t3DJM4xFT64L38KhoKxWhF6jzjo/rFTelIqg3OiAEo+jBSSLVU0ZRhCclfs2Hiov0BNEk4uatG9y+9e0DvwOgKuCewcifYTBZRjYzbO8OuLnZ47OvttneOWSaJEVnUWlq/aiUdh/r+vIiczIRURTh932qngdZg0YY4CuPLIVZpsFktq+t6CSfC9w8nP7wosJS/qwdcK1ao1kPOTh+VQqGRZgBh3s7DA5eJMXiwbj8hi2bJyNIIognkMa2cBIPgaCPoIXvuzTITpX/AjgR4CnVHJ50mq748KOfrHPlWoeD/TE6TRj2xzx8G/qIaMFPfnYV37FTbqJz0iRDZ4Ysz9GZIUkyMq3RRcUoNoZEG0zNQwSK/Nfnc/391gXEZEzYSmi3B7i4iDgjIud59UI0tpHQZFCVHsttxe5x8sDc7AoXT1Qg00THR7gmxfWs5ko8mpCmGpNnJ89xmcV7ZUNPQGcIEmoSwoZPtNyFg2WejlxjmG/Pyn8PsR2eCpLCajipQW3EDIdMSCqOsCUStQbJI/jEZxyz4zOkrj1nvIZNv5RkOw971b9rNM+w4NdjrttL8bl0b3vVoovVJq9VH/PCp4yUBBdBWoDeDFNo6lrjiZSY/KSFuGwhfBrE5VPBR+AutNRbZSeb9ZVkhR2JwIW0Qj6GuJcS9cb0e0Oi0QBBjjjZ3liYOSvW9ZJ0GeCjqeMAE2bFsQNClmjQphMs4Tu+zfiecY7ulaM6nEvMKDTT5uF4dZqNNr4f4PuWP5wkKegHuTA6TRgMhiilqAcBrhK40mMUjblx/QaDg9MZYgdoFKYM2TOuT6VhRaXQYEtzQ5Jptnb6/PqTTb64fpft3R55noPjYqkMG9hpxMH2ypf2w2U354vOTA6h36fnuXbPJwT1IEBWfbKxwegUYTKrKyFsfxmzWcHjWcwJvgzwa/l6JpfU6opmGHJ4PDjZcZZUo5ed3drf2aF3/HKB79V37OMVHUJvx1Il0wiyiQW+DmB8ja8kYZDRHZ9WmlwEvRFPDpx+9i9avPPeBt1lhc4SBn1Fo+XiuJA/yXPnw5/812/z3/y3P8F3BL6wDXp2qnBwsFWA3FBUHXLbF4XV8dUCjAP/9v/+Gz79dzuPO9pThxesghNTD2PCxrFtVEgjTJ4RIcmfwyEsB/LUzk+qqlhbXWI0useN+/bNnuvZZW6aMtjbw4yHVH0PxxNMxhFxkpJmKcakJ0vcqwx8J6MhOtcYHSOlYaldJ7q0BvHA8n2fqL+hTpn/mjNRFyUjEyhYjBgJk0KQT+XMpIR6DXwJt0bAjfP4M09HfJZa8g/GP3mj+EchxGtSO1RNQeDNAcdAS8GFFnQbWEyWgrNpCSaPY8mXivE15tnelLk/6asUFeBaCG+8AbXgbN9bk5JTKbzXTFEps4YVhozZyTgsPz9tf4BCUgOck9ENFvba/2bYp3yGwGGmXdIxTA410f6Y48Nj+vSsDXGxvalQKdjHZS1X4FgxNABcDC4ZDiEeLVqsENKi21jCdwNMJmxj/xmCgH8cwPchkcdDBk5AxfVRgWQ4ipkkD19IlK/QOsUYU5gs2JuQpgmj8YP8yrbvIJiRpAbznH0pMwNpbjg6HrHfi/j6m20++90dbty6R3/UZzaZckJkJmRBmJJ54fisLDSfNsagd8n2Mw5Nhu8rVrpLiIoDeYZJE8xsdlI0YfaQsV1840U3MNjdrQW/9VqNbqfJ/t49hok9oZq0QL78+mXFrVvfMj58uW1gKxsdsiRBMra2xY4Fj8MIxim4MdAZY/AJfOiMT3dsO8yLcn2ebHP/zu/DH/7Z22xsLNPq+iTaEA1TOt06jQYcPwHb4bXfX+Hd91/j2lvrKAoQbsAYgRACWdiXm4UTspQbc7LsGGO49XGXTzl74Ot7IcK4+H6Tmh8Quw4Y28ySP+dyXwGEMQiTUvMEa0shZhpy4/Z8PnOAUEmqDuTJmHv3IqR0aDbr1Go+o/GYyWhCPE1I04Qse3UbjcrQWqO11fF1haCuqjSbDQZrK7A9KqbK7wK/bSh4j/NRfL/+gObEQEgIS3uw3W+41RquAikEUTaCrRcAfM85fvgH9rNTAt208AmBk8viYLOfSwGEVcjHYKZAFXpf258/7gkqWaulT+WUV1PNYQ24cBVW34DgzIGvpgLFDGCzvrPCeth+TDmt2fu021BRgGmDYXrqeRaUmv42XZExI+rFHO302Q532dvqMepN0GSFxrC4bz23X80oVYJ9AiSQUSEvTC4CGoSEfpPAC6gYiU6ePYH4qPhHC3whIRvd5nDkc7i7jB0kD0eply9fRimFX/Pxg4DMwCSxhPFGs0W11cJFIIVAGCvWFo8nxPmzZ+hPOK0CRuOET363zS8/+oZPv7jNJ5/fZLp/ZG2ypgnMyvMuheLOfhF+9rgDuk+6E9PzfeK1NVtCThN0klDBlkgxwj4NMwOzysnDQaUQaHFfDJPL5MbyfItWYolDMwxZX19h5+4mw8ROtVqDVC+fEDo+vPuyT4HW6lWSeEgc3cB1rb3vOIF+ZIGsswPNdZCtBCWhG0A2nj9tEgs1nrTV8uI1+Ff/00/5s5++i3BcXFHI8zkeSQIr63c47n13w98bfxbyxz99i42NkDiJiZIEYzJbdShuq3IFQrjWpIB5xsEUC40ojC3i+Hw2lnXh4bkZmSsZuB5G5xzP4ifmP9c4rZm8GAHgE1MzMR2vSmepRtNZ5ovbEdvYnOaaH7DSqRPWBNNpny9v3CbNZqyvdlhZbp/IRkaDiCgaMM3PNClzPpHXcNDggCTGcRSOkLjtOplYA921VIURcDTAjswS3JYykCXIWNQ9L+en0vZdWdm7UOG2G/jtNrV6HVWvgbR4WDsx0y3FuVfiVNvykM4pOlfnTWcSTqrrZeOZcKxMcq0K1Sp4DuRTC36bG7C0AVtfweY9+JpFQ/sHI8Juria8mqAXoFOD1iUIL0EQnu17T0kQZOiTpjY4DXjLf5fbhPufyO+iQwKkZKcE53TByK3ikGGFYCrM8BgSsdnbZtiL2L/TYzQeM06mSAIcvCKvK5gtqE1kC5Vcl5qlb5HhkVMpXl8FVMWaZ2gtiMYZHV2Za/mL599evxTntqAC49nLLxPbiCF/NCvQVYKKEOhcW4m0NGUwGrNzeMidzS32d/eY9iOq9ZBq4JPEGTpOmD6X4vg8Up0xGETcvr3Frz/8gq9vbjK9tQ26fOzL3d6UuVXwqxb2QRNCIITlCpk8J81SPGGKgTy/XuWDMis4QiXAeNEh7EkT1mqsLi/TbLbgsAC+wEvqaXvlwlUBRoPr+ghnjMEq1fWA0ly3swNhbCGjkoUEWmav4xD72icBdJ4HP/2LDa5eXSFs2OPqzCq6CFfhKg8V+HhBRPoIBZ/mBrz3zhXevLJBpxWgTjIYReWhGGqFJwcl/dzkc8pDjgFhkMK1GcRziGw0INMTTBJDlpJl6VNBpJJeeP91rWDzlqFn8EVKzXGRLrh1ybIDB7kFxh4pbjaFNCbRU+7t7DOJQU8nTEZjhOviCpeDowP2BxaIvMo0Bxv2rgo0jrTOe7JWQyU1sgKdBfUlNA3S/RiOpjDqQzJgbgZkmIOM8t5LLN2sZgFvVdrGNiXxa5J6rUatXsNv1i2vSxqaSYupfwni8836ukstsnsPuoueVYStAvg61h/EKfYC0il4vNJeDlWAXgHoKeQhVAMIaxCGEH4D1d/Br3g0qE14dTO9ZVRDqDbsZ/kQb6vniZyUGV4BektzilKybJFqM+U06C1dV0tpgUfNWeUYh5LKM8OQn4BXUXB1ISLCkJIwxhzkCBxUQXJwEYWqRUnIgHJizQpOsot1wiw3TTZSDCkmN2TZjCQxTGJDMp1xlh5WLxT4VoG3rl7i8uvX0KnlreosZ6ynjCcThnFENIoYRSNmZ1VCrizD7IhnzUVkiaHX69E/6uMrH6UUm1vbfHn9Oj//259zfGBrqtNRn+no7ECnEIIsz4jjmMP9A7795jqfffSfyfZKq+CE+SCecj4cXv++j6Jln5h5UfNhi4DCLp1lvbmFs3yBpZU1avUQt1ChNrnGYHBEURKZGRaLpXNra/FigK/JF2zc5oqFYVhjZXWF1eUufGszrCXn7EniVeEDn1cYLckzm+WS7hgh5yNlFztaekCrX2QaXdtUBXYkbwJPmrf+vffh9Tc3EMbQ2+vZBkgj6Pcyer2EQZyhpSLoKFw/IUkgL9yzqy34weWAt966wg+vXmO9sULL9wkdH1MDCtArACEsH80RhjwTGA1JpkkTQ6IT4izFZHkhS3E+Y/PozldkJiWeDMjiCJ1Ov4Pg4FIhwGWCgz6Z2O/P7TjACrDmw1IoCDyN76Z4QpCLGX4VvLGdVY5jTfPoHnFgmGaGUWwl5s3+mOFgjHCtJNrRIHtkX/GrFo5ju+AdCUJKVK1Go91gMtm3Dm4k5PUWjWaNWLUZrwBbW7BZ5hiPOA0syu1Rzbq+NRq47SYCSJMJGJCug3IFSkpqStmMrwTTbnP8xjXST59ElefZ48LGOnfuPbtt8uMiqNqrUJVQdU7nvqUA6YEnLBDGmZtP5MK+PpQ2E9xZglYI41/BL7/jeK96vsHx7P11PLv3OcuYnfgFzhWR5mB3UQ93MWx9Z74SfVeKoUykwZzDbncy+UnzuY2UmLQ4/hpNGvUO0mvhCR8zTTDxlHE+JGG8cM4wIyvYvjECUyS6LLnCkJMSkSQjq2IzrjOIEqaJIDPzi/m8cr4vFPi2fMHrr1/l/R//xModVCyBWpPT6/c46O2yu7/H3u4u+/t7RL34ubr5Lr7zM/7oT/6QnXt77Ozsc+uTv+RZAHC/f0w0iqgNApRS3Lx1m1//5rcnoPc8oiKKxXaScHR4xL2tTbK9z7GTb5l1KPNRZ73/FUDHfrRa4Ic4fgPyhPx433YxVX3rVetV7edsatv59ZQT8JqAhTwdltbW6XSXCGo+jqjYsnKmMRVOqTfMTJkBnp9K5ZwzvicOQmLxsbbHcxyohSGr3S7dbvup3reBLRknzBUV/yFGllBw2dXJviETtiy5hx0BVqTGgq7lBS5ojycHvQDX3gq58loXjGFwOMTqPgv6R5rDXko/SsBVNFe7tAKXhu8gHIErYLnV4vLGBiurHVa7HZqNkLpfw3d9XGk3V44AhEE4BlGUs00OJssZxRmx0IyxXW2J0RYr63MCvtvXbXbZTNGTY9Jp/MjNU9Vf4c2rVzHJADMZMD7a4SjRDwDfNtCuwFLTJawKqiLFMwIXa4DquTZBkWLH7PgYpu0jtBEnjME9oJdAJQGxUBj9PkS+kO0SAqp1RV3XEfvKUseMwWhDzVc0l1bIhc/tZAT796yTEkec3k4UrVZCwdISXneJ5aU2RmuGx0foJEZJFykdlLTgVyiBIx1E03Dt2mV+NziCzQ/P5w9WbdbWlp9O6eQpoybtPFktAGwJbKVjgV8haWzn1mKyzbHX36nazGinU8W8VqXV7HN0AAc34eFaSd+DqkJ5AYRD7qjHvfopI8OcaqzUzAHvo9xWa9j+n/J3SuWkh0WZSCsFJu/nsC9aPdnOjDECnBg/dOg0OyinRTIckTAgGU2YFAbTlUINvwS44DIriGP2CJqMBGtOPGYcx4zHMaOhJo4lWWYwppijnzNeKPDdjQ2/+MUHbG7tofwayvcJW22WL3SZCagHPpcvrbO+1iWZvonWCXEcn3xEUcRwMGQ4jG0W55FRZ+3NH/HP/tmf86d/9sf0jyN6/SF/FYZ8+ov/86nP23U8/CBgZgxRNKLf63Gwez5uTafDoLXGZJqlpk+ttcZkrIuxXcGWLmKsC/pZNjgZLNMqgv4Asi4Vr4Irq8hOC9EMkL5CKsUMQWoEoxHMxiPbzlv1wfchWQHq0FnB84PCajrDaMiTmEzHSOEgXPHg4Re/VRE4lfMDvvNDOuTFc20w5MZO10opmp1l2t1lyobksOj2vz/H7zHvPC73p//QfeV7hwfoJCEaJ4y15feO0jmFIceCqAm2e7vNfNp92lE7GDocHkDYcvGVjzAO4BL1D9nePqB3ELO6FBKsd9lYbbC+HOAWWdx64NNptQjrAUHgE/gKKQVKlGR8y2fIMeQiw3HtpK+zQtGhWDikCz4uSrqAoO6fcQdLEePjmxhm6CxlMB4TzR4tpzeNI46Pj0ijQ5LoiAnmoQDhEJjMoLqX4bvH+O6Uie/h4hLFKel0nk0p75En7EIRMs9LvlyvwGeP3d27UHCRtYFca6SQFpQZYDQh3d1l16mxdqlOfalBtVFn2m0XDn3lXy6wT3rR6CatCH86TegPRgij0dqafMiqREkXIYRV4dF2fnEQNJZWWL30A/Y2v+Q8Cvidy5ep1+tn/r6LUQ2s/GPNg2ptDo0cQHhWHcXBgTwnF9ry6HMwVcCAFhJRlci0Sm2jxtoPJ7x1AAfRw/Pgr3rlbDyBgyNw7uVU+2etsW6YnYDRlDm392Gg18GmG5awY7WkRfgPee1ilBr/5QpWrr12rp2nh6wxhYtimqeMx0OEOcQXCek4JY81CWOmjIsG9gqlgr99zxRzAkEtc1kXmWuPKXqaEI8TokHMZCxJ4pwsdBHe9yzjC7Afa/a/+gqwJ//a+gbviffodFuETZ9ao0sYBARhaEtESjEpgG/vqMf+3h57+/vsbG9zeNjj8KhPet98cfGdH/NnP/1zfvYXP+VP/+SPSdKMaZLTanTY3NxmsPXBU52z40p8P8DMYDIe0+v1GI3Ov+BijCHTKSbXdBs+b72+SpIKMhR97bEfSbjXh1nG+VAdEmDPCq63Q6SqopptqkqhfIXve6SZYRIn6DxmSgazFJyAiu8z6/owhWp3FeUHIFxmZkaexmTTBDOdYhxvbv1ljG1yK6siJ1xLyw0+rzAnMoGWp2T/RbErdZDVkKaUdDortNs1tNZ0Wk1Aw90BfWwxaV1CVVkOqzYwmcBEz/flr3JcxU6NzxL9gx46y4hHmjixwPdRCg1lxvBZo9+HvUODKyXK8SF3EHgM+/vs3ukxHCdcu/waV19b4e2r61x7rVvQFoxtPpXKNksK6/woEDgO5LnG5IbEaMtBE9YVCWGVG3ReNpzmuI6D65SLgDXQOY+IBt9iDKRpxkDPhfwf8WoGR0dEycFjgcEE2JuBvwc1NyauxmAgnkKanDYXEBT6BEIQ8HzmQ69C7G5vW41uqXBkFUyO6wiE4xTKjwlM9klR6JWL1JWk3ayxs9yCHbWgnaWwNR1pOb2qBghINGNj4ZqLQUqJdCVSVcFxbHWr0JOXUtJu10muXGFv8314ynXpcRFcepPLV6/SaJ4z8K1a4FsNoBbYLb9T5ngXNxW5JtdjTKrBm4+jVDp4RaqgSoMLb0+YTGHvr+A353rm5xOTKRwegJaW5nG2UdIbylWlzPg+bIVZATo4tIAqOSl2TX9cFnrRS7OMUhqV4vsuZbOnwCchYTQeYGKHhBhbDBMkxOgC+NotkJ0zbWhsBriCQZCRMmWKIUaRoPWUSZQQDaaMxz5xYsgzg3EFznOmfV+qqsMM2NzZJvlVQmupTtgIqDcDmo0GjWZI4AfUmw1mZkaWZSRFKarhB6hLl1ldXqM/jGxGd6jpDzJaS2v80R/9Ie+99y6XLl2i0WqSZ4Ysh7feeZN/+V/9l/z7/2CIdr6LRTSPZqfBj370Huvr6/QOD+n1DukPXoyRgeu6+DWflZUlLv/gEuPxmO3DhJ1+Rj7AjrtqAnGAhV6PAr6CCoqG9GmFPiudECkquEKQTGOiYUTvuM/u7FHwrEe29QmR0yEKGnhhE08plFKkuWESx+THQ5hEwAyEgwoCpq4DSqD8AMeViApk6ZQks5wkIXLE4ioLnC7HFF+XrzmnyAEjimW+BNsLmtlSCoSsUQtDmp0uk2hgf2eqT86yBjTbdqKrSpiMLfBNsIDlVc5S+FhKxrM2IG9tb5Obgg8/tJmaQ+A8npJ+P6Z3GLHRbaGEQxznxJOIOB6BTmj4cPVii2uvLbO+3KAVlj5xdoDlC7y1rGitcHIXYzIyA5nJSAruuTUoMhgDuRGQC5uhsF2PGGHbN4w8n2m07VuaxRiI9OMB5zB58irUNrZyP70NTd/mbkzRbFiaC0ig6UDdt8CpTozPq91Y9LiISxczEVueoWNzk0bnFrzWJnbXqg2jwZDB8RGSnE63TW+la4trwKlysQErAVNkyHIraZADFJstKZUlfpbiIMWvSqmoN1p0L13mcOcQsqdxkPvuaC2t4NfriLMmmt4X93Zta4SnQAUlGLN5X5vXxmZ49ZR8ahVxcmxLBQ6I6hSPKZIBOso5mEDfsTSI7xWPpohobBVt8h7IMzawmBM9Mub0hoeVv+tU6OIQIvCKeS/myX0xS2DtUqo7zHd95Ryq8FBIBDExeW44zhMEfQQeAklKRE6Mg4OHV8Bc20xsThrmPESx8JaNc6I4TKYzJiNNHGWMxxmplqgzuKavhJzZ7sEhu49wn1rqSJRvy5K1IKTZCGm1O1z5wWU6nY51U8ohzhyS1CNsLvHO773PxsYGyyvLgOXLusC1q6/xP/7r/4GNSxv8+3+7zlcff0KpX7R29QIbGx1WV1qsrNjP3U6bbrtFpxGAyfhoPOZg/5DDw/vOtSI405bDIlzHpbvcwfM8HFWl0V3h5x9t88Vv7nI0HtoJWgqIy+zDgMXlcUktc+Xq61x5/SqXX7/MlcuX8X1Jw1fkWW6b5yYxk1HE4eEBv/z//hNff/45d/OHAeAY8m0YbpMO7eP34AKogC64PvWwg5CCJAfl+3hKwsyQJDEYK+KoPGuoJ2Qx7LWZLwpiVhRFLCo+T6KDKZh/lMeWYCx1FMcBpMRxHGrNJqsXLrJzFwYHe0Sj5AQoVFXRxevYpplc6+8F6IU5yHlWNtqXX87HfhTbIsQ25yOq19tL2L+zA9fW8JVL76DPne0ecdQnVIZOJ+DH765weaOLlC45GZnRaJNYRYbifWwDm8DHB6FIMmMzvplmrG3RzcRl1lPaiVlIHMe1AEZCLgQZkIrzARb/P3vvFiNJluZ5/exc7Ji5+S08IyMyMiozq6prenq2p6eZ2d7Vsssuu0JCYh944QUESGgEj7wgIYHEwyAkxAMIgXiClUAIXkC7i5YRWjFCq2F6lplhenq6e7qnp+lLVWdV5S0ybn4xNzsX5+Ecc4/MyszKyqyojq72f8nLMzzczSzMzc75n+/7f//vC/tgHTyewmzJJwq19oAdCXP/fJ+X4/To13HR0wk2SuI1nQN7Yxjv7LAiMDInlM3PN/FtFnXKxiyoFw2iiKTUNhYGQ5AaGgvKMH38mPds4NrumFu3buDtjLMP3yGW+DWsz0RIOl9TAm4TKpcStEEXBtUz6foRKZtkCUHgpaHs73DnzjtIL3jwhz/kVQuxn4SmN9qhKErkJRcG//F36IK7SL3Jlnk8vptLEo+y81gG0tHjGLxJRMd6WEJ4DP4M5gPYW8Yx9CK160R+sW3vZwM1ig15XiZ1dxSg9zhFfD/1oaEjoJ284Vl34xC4QcUBIGhxeGZEodkDXj4z3MkpOoV26lACgEIimFAiWDFjypxTluuLv0dORUlNSYNBAz62IE6ZVIcjoMjJ6WwAN1tXSCWwS5iftUzPLbMzT90EqkGIPbtegxVcCeL7Ijw+Xt8i5PIRVaUZDgYcfXjEaGeIC9ERQOgB5H0GwxNcWPHg/odc39tjMplQlRVlVSGEZHd3wld+9S/RWssXv/g2XdeEg4MJu9eHDIeG0aikZ3KMkQg8H9z7gHt37/L13/un/N7v/dFHb7ZLIL3QEfYcrTWNhePTJcdHZxzffwQPj1iv+noBxQ6VqqhEw3BQcmN/l4M3Djm88yaHt25xcHiT/ZsHaCkochGF4s7R2Iambjg9iRXyk73r/OD7f86P7t59hQlOE6dPRdM6dF6hZUlp4uDrnKVp5zh7ilh5ci3QSiSbs1RRH0KiuxddIy4fIRCLmrrqLC9icQICnUb1Xm/EwRtvsJhOOXr0gBmbW08UIHo9Urk4mCleN6/dwOSzxKue6e99h/jdmTh3PSCSrcsg/MJAHTw/fnDMqQ08fHjKwwdTfNOCNijdp6nhbLrpWmR9SwjJdUSA0gIpNLlWoEDqrkuRiEqblIEIKU7S9ZAX0sQDECSTnu7/l4NCrQ/5Ew0xRXrlwaH+AAAgAElEQVQEVCodeTFm6TEmJkdHbJIsRaZQSuNYIXNQV72k/mMwHF3D+lgZ36TuYgvbYH1I9y2JwGrwnuXsjLpnsMM+2vTg1h24V4I7JYYju2YWIg4i3seNaonUGmUKtCnQ2hA6SwNihFQogxSRHPdHQ67tXeMB1/g06jWya+9w7dp1TNl/wqnmMvD9d+Oz1MRbhadKopKnL5Z1xLeb1b0GX6T3eJANiDnoZJnTBfe6ci6RdqHTth/x2QQWiiFxavMwewmt1mIJAx8vqU8X9YXnZ/kKlcAukiGQE9vddI2dO0eST4LuW1ylh0r7qBgzZEiJp6ElYNdSikiYW2LmTAEKH4MQ623GpsakUjdNjsIgKZAYJAYlDULk4HJ8k1HPA7Zd8bQ92qvgyhPfi2g9tOeWk/Nj3vvgGY4KWUWxc43r+zfY29vn4OYbHB4esn/jgJsHh1zb3eXgxiG337rDeDyk/utfWydkqqrAGJ0myJq6nrKYnvPo0UO+/n9/nT/9xh/zvR+82AtRlRWuflmTq5dDJiKZePRwyve/d493f/Ae7r0fgk+FFjLj5s0Bt/avcXhjj5s3dtnd2+fwzi2Gk11G4wn90YiqjM03UpZ2baXgXHRX2F3MKXsld95+m8M7b3Hze3/GN/6fP+Dh6pNM6wOQQ0CxmNeMJyN6gwEmj2fZt3Pm9TmiOWYoWoyWCJUhVQZe0Ml4RZfnQKQo3eWSXx+6m7tTMMYbKv6rI7+CXm/A7dtvcvzoEfzwBzi6DjREYe/gAI+MpElD0D8fLGFjWPNq+BOid2eexuQjXt7q7RNBQTkx1EbzzR/fI7z7kHpuaeaBca6Y9GPc8v6DmvnsFEjl5D7aVAkdO/GZUpEbQZnnUEq0VASRJAxBRneHIAi2u/Z6BFkiVB4dTHAQGkKI07u8rNVNKqp2DpYvObv3UAykIQTH3DefyCTrFLhNLIXp6rnFKnVqUkB+qYqjzwR7N25hrUUWJ1jxmNlswWJR06zlCoaiF/W6y0UNTcNstuDxyTkBgbp5Byd6cP8hNOeAive+lDFS7NMCS0uEVhSFQRmDNAaSVlxKEEphdAFJY94rewxHIzAjaF6P+GbX/hm++KtfZO/GIcZ0FPHy8BfvxmcpE/mFjdRj8yPCR/cX78HJC7+4eIjJkl5Z6Plodabtpk3xE40y0ttf1JB5eGvzgcZCk2rAMg+rKS+92i+K6ENc9CJxbz7G0KlNag/5qWt85zxpOfY0JkgmSAYpLlyz4oxXH5U7OUK3gNb02WXMNYbkDCaKZb1iXgsyApH4tnTyCIehRSNxF6pnIh9wiX7GZaNGYcgp08Og8wKEQWFwTtEsoG3C2gXqdfBzRXw/Fqs5y+M5d49/yt0/B+SEnZuHvHnnTe68+RZvvvU2X/21rzIeT+gPKnb3JmgR/cSzLDZKaJoZTeNoanBNy3Q65cMP7/HjH72MAfhlELQYSVjWDedn57SLc/Ap+ZPBnTfGfOmdA37pnVu8dectbr/1Jrs3Dji4dQdTDci1QarN3be6QCRFIpYQcNbS6w+4fnBAf2fMeHeXZdPw8Bt//AmOdRCjJSHg53OYQFWV5Ek661YeX8+hnjKsPNpEjZ0QkGUAq7XhyTqCIjst3WclGJBPPAEQJD6A7vXYf+Mmo59Okm6uWQ8LVg4IehdImlBtEXqKprlyveSfRuc68apX77fY2KNrnu0k+WlgNAEzHBByw73TGdN5C0HFRZMqGag+rah4cBo4r9sk2RZ06UFtHMYoSqsoK3BCQ1BITFoQCkLymPRBE2QseEPnKFEiTYnSJcE3UQezzk5cDuZTWLioGXxxYVvEMCsxSpHninbxMrHej6JLZnYWfGczy9m8wWlB4z6r/MvloRwOUc5SBjALy6zxNKFhhQZdoLSi1+/HucAGVo2ntZbZrEFpzXC0x0nQrKyAc5OkcjZmiJYhrgBFGl+FBKXj+BAC1kaXHqM1QmwYogCEFBiloVe+3s1j7vArv/plbt45ZLizgw+eZnm5RoqrpKVxEtzFCOeFi2VtdX0xWHeR/K49JQEb5YtWguyBPNuQ3Y4ApyAxPZ5PfPvX4dbb8UMyh2UbHRfw8Wuyczi+C81L3FxCQzWG8SiS9w/tiz/XdJr8T31weNF3uQsMkQwAiaVmxYxIel/UD+9F6EivTs+KPkP2eYPhbsZgXzGdn1O/t2S+skzp9CCdTEJg0eSEVFwn6IqCbSrSc4mRZORoKgwVUhcIoRHCIEVJaBXNHHyTavlfcyD6fBHfp+GPObl7zMnd7/DNrwPZdf7y3/jnuHXrFvv7+4x3dlBKIDMoChk7+aiAlDFJKMQKYwrefOtN2qbmO9/+M6aL56+e3SW1LpVKsDPW3Drs4e2EQXGIFAfcubXHL3/xDoeHBxze3Gc03mG8M6bqDTDlAJXa/Prgnhxw0j82KyeBUJqiGjIWGYfWoU3J0cNjHnz4gO/dexnSfx1IE0FTgws4H9srZkKhpCC0Ahsc2HieosQhHcEqIDpu2wl9RWx4eNmuDv65X6lI0jRPsCC1ZjTZZbKzz2hynQ9PohjEEQc6G6JYTQgPuo/uDTB9C7NwpT18u/nnVeNCHXG+bH/Nqq8w1QRlxpR9RTANAoPAYMoSV1bMdcmpKKlFdKaVEro2cUJEv9o+JRV9LCWBPo4KlSQFzjaEADZYXLK9KkWJMCWqLClNRQg51iqCg+AFuSgv5e999y4sfWwC8uHHvDcHhAh418bmGqvQZZ2jDvkl9/lTOnfOSLZPG1j88C6qzHlw9nIE/Cpj1kSf3sYTixNFAboh1xpNH6M1ZWkgQN9KpsKQaYMNGm8FWmuMGbHcA3o9OHkMJyeATd0aUpGbNoDAOsv5+SyOD9YTQqA0PXp9C6FEy9gCuWlm1M0szuwvA3UzGmivfd0jJrduUfZ7CK1IQkg+FePTF2HImsgqAB+ngVXLJjC5CfRtqicvDjwXHbMkGB/JrSxALEDYj0Z8u03FYMlH4UXsSXIxj5d3BiwptjK6nqLGH3NhS58a8Q1hlFoy/+Rbz39/45OU43JVJhcwBHbJ1qS3YcUpcfR4XelMlw+MS+mSAePxdfYPR+y9NWQ6P0blhtUPcqZrB/DOGaLEUtJGoQ95Cm512uMVFk1NToNAY+ijifKcIARa5GhhEFbRzgVtDcGGF8zZL4fPN/F9GqtHfOPr/5Bv0Ofa4RfYv3GAkgKpJNUgj4Vf/ZLRoGQyGbG/O6GqSt586y36g4rDW7d4dP8hD4+OOTo95dHRMatLald6EUoKxqOc27crqnyXg0lgPBzxlV/7Ml/+ta8y3t1jON598kMhthqMLVbDemQQQSQ/3e6NYh0ZKwcjTFmRm5Ld3RucHh3z6P6HzKdnvDd7UdL0FjCEzMSleohefL6NN4AQCqmSvs1bQhsvctWx3nBh2OqEXBd+EJdc3AYe7y/oey/sHcD75DwoI/EdTXYZjXYoxU+o05sWVrD0UcKPAJlbdDGiNwj0ei1mNuNTt3T8FLHxPbiayIBycB3Tm2DKCaVQiODQokTpODhaaThXBmQZm2ZKUEIShCekTmwCQR1KGkosFYEKyxCNRwRiwZGNzy7dG7k2SFNiyiraGgaHamTkHU5g9KdtUh/x01jrw31evCgZA0IqhIDW21QyAgZNZRQqVwgRCMsG1zhqni6D3aDzW+7wI2DxuKFHw2MuScLyGaJuLDZ4bICAJiTdti40faMpjcaITmIVpQ9NE1PXGRq8RvcMvV6fptdnvmggPIyVTCFVxmpDpjVBSqz1nJzN0AuLtQEfPP2+BfooBME0BO9pFjWLumuL/PGYvP0mZaHQKu7X2gUhwLWbtzD9MtqnIdcKysuE6loWixSJTXIF66OmN4QL19qF0K3Mo3zxopmQAnQivYVPBDdFfbsi3E7D3jnNSp5d5BbyuHDMZZRLiCRXCMRgh9AxYiwLWH7vxbFUQSS+RS+2YK4qWE7h3o+f/5lNddJngQkF1wnkBAQrGmIx26fh77/OawJQ0meye4ObhwfcfvsGs+YxQgvqM8f9B8dMOWbTSc6mIH48shgzzskIWBrsmvjWlGhyKvKsAqlipU8mKUQBLsNOBXYBrg3rzp+vil8s4rvGjMcffIvHH7xgyQbsXB9z+84tJqMBZdWjUDn7Nw84uH0LpMQUPUzZ48GHD/jdf/I7nBxdTttJUxhu375Nv19Rzxc0TY3Jc3Z3dxnv7mKMwYcY09mUhaXyFJEyb08FTIXoCro2r4XEhnVRIpXizhfe4Wt/469jbcN7v/OPX3CEiTCu6pT6i8L7YD1aaYwxlKbC2hm1yHDegYiTgxfRuyGEAMFFfh4y1mLkVUgHenk6tScWAcQU5YYEX9D7So3WgsnuLrffeYejkxN++JOfAFCfHfHtP/hDrt/c5+BgD60Lev0RhdYslrO43l18NjZ4r4I0V11Z/NXfqPjK197mxpuHjA92aWyNW3f5Eyip0UKhlMGYCpHEhoGkShACHwIBu+7QpuQcERxhMee0jcSkbRoaZ6mbNhZ+uoDkLgKBMSUqL6OvrwtIPErBH/34dZyJn48BL5cpDcDSOxrvnoh8TbDkSlFVOSaXrMqc0CxZnNeUPqaIF8So2SExUdq1xPkmmyjaZbhz/KywtJ7GBs5nlpPzhvNFw6oJ0NP0hzv0e2WUIhDojSz9xYKTkxnnZzOC87jGs5zNYDaDx4/g5F3gfXBjWB7AsM/g2jW0jrK5elFTL8KFKqfoFF5oiZaSQkq8XzI7OeL86DE0Lzejm14ZPYJFDAxoLcAHbNMwPzun1++DACMMWnZs83JoWK93oVMbcTchyeuXnRMWrGufRY911DcnZssyWBfBaUCHtC0PsgKZMujhwuYsMQPxrDOmrsNkDOMqEXIZpRPLLvKcFHSyiiR6QnQ4eR75DTIS+S7OpTWMrsHpI6ifFS1O719+Jsy3BBSWgF87jnxapBfiWa/T85xTTjl+eMLe3j5hPiTMW+qH0Bz7VODWSR06mUTLks0ipRNhBeIiveaMQEmGpmQEeJr6jLCCQpfgBmANoi3BSoJXW+J7mTh5dMrJo1N2divefPNNDt845PruPpPdXSaTCYe37nBw84B7H9wjhJp/9Pf/j0s5Dq1zDm7c4MbeXnxhXaEGrKO6lkBGEGod2SI9P8/N5unXQyrwyHVBZgquHx7yxeVXOHr4kN7v/OMXuP+plG/q6NMCmBOcQ0lFrg25KVlpkyrUfdy5UrEQLJD0kqytzNZsNB1k2LDTS0DctpQkY+z0kBcjwDJKL3ROfzzi4OCQ3f331sQ3buaMR+97JqMRxaiPrgZQFfhTsMEzHlnOzhZXzt6sU8Fe1cKlEfCXfuWAr375Bndu77G/P4lDpvfrJJyUEiUUQmikrvDoqBMMkfR2Oss2NFhrcS49h4bQQj1rOK8ddROYN5Z53TCvW+q6YV7XWGcRqkRqg7Wx4CwvNWWV8733L6d1eVmCq1/ue2n4aLrXEbPcZaHpVwWhaQm5wISG/CysO+jtS/jCL8HBzT2W6oCHc8Xd3/8Wj668Ov2To7FRu7tYWs4XS9zCQmNjmrXXpxoNKYxGAMZayrJPsFAvGhrX4KyFs3O4fx/qToRyAojYqVIbhqMdpPA0zQLXNBvjWh39vhqtWS6XlEqwVILgFixm58zOz+N7PxYl2vSQUiCEROOjUjgEFsEyqxeMrQOiNCPOBVFScRkokvVCITfuCwC00F5oVEERfW1FFRejXbxEXfiQILk7EF8TnuimMAfOPiIFfm4GYjyGwQAGvXhMuYwzk/VJ2pZkF1KDrmAwitv/4Dnb6+Rc1m9aMVcDmEzgg+fIJGyIuuLLR3Qh8ayIc+9jPt1Orh2i/37NlNn5jHoKq3qAW8yxp4LWpqY/69JY2MgkMlZJOhjSa56WwCo2vOCMinH8yCrQsCQswdkx2BrRSrACWpFiYa83W22J70vg5GjOyePv8uMfv8tkMmY4HrMzGnPz8JAbN29S1zUPPriHkjHTfzkQdF17V/HH+L8LUVuR/mPz0iciM52NU4dcF5S9ikF/RF9qFs/09wU4g1XOpqLTAi3Lec10Nsf5jLoOuOmU5dKuCbZIUbhVAB9Wa0lGCJ3EwRNcwHmLs5c3CT+tF7rId6XsalG7kVpQmB67+/scHLxBLv6UNlzU5c249/77WLtLf9SjKDS66NFHYnSP0aDhvfffv7S/5VXQDU1X1Xnt1iHsH+4ymPTRBiwNzrvUZbjTjuUYIVJSN14rXscBsjMcCzJ+j8FI8NH5IISovZwrTV1C4wJ1A3WTnms4m1pmTYpheIcTnpoAvkFZTenmlxIt/yv//F/mbLZAff9dfv/o+YnY53lF71UVO6MB/aIg+MDj8zMen8yZpy5wO+lBiJG3UuVUVQ5Kcmt3wKOjF9XL/5wiBaKcC5GUhnhtCCHQQscW6qToaSpGQ0jKQkfbfaGZ6o6gdkZwAI/jtt+FB0ZT9jS+acgQrEy0PMu0RmtN2evRKwqk1gTX0CxmzBYLFotZSsV93B/hsNZjipJ+2UMSsPWMEJYp0CGjd2+I3tTWXYyTfvoIadVsLxJKEQluJZPDG8RubV0/9+STDsnlgtT9jUjRi5AkD574Yr6J9nbD9YJntzQuEyEdD2BQxOixBEQLy2Uq5JXr8MaTeuPnQCYF33y6sVhj+YKPdUUPl6sySWiJniw1MW59OZnniBPOecQR73H3JxU6D0zPHvGDH36LD/gJjjNS5x82buDxLEUJRhcJjj/F+G+N5ZQ5I5aM02sNrDz1/JSpGTCcC9p5iWskoRWs/OsFwrbE92WxgrPjOWfHc7p1YVEKru8fUJaGxeycspRMZ5dpHRMvnHXtV3oN4sB94acLg6dIKodwgQU/b3B98j1S6ahrrIZUgxGcPq8ytHO07dIhCfOG2bymWQaUagjzOaGxFy66DBAE72KhUEhJkJVLr4MKAt9Gy7XLw8XVadJ/iRT9TedVIiMJFmB6AybXDzg4uM3+/nXu3nuSyJ6f3QMREPoA3SvQRQ+tC/RghJaCh48eUTdXx+qsI72fWR3GJ0BGrMzeO7zO8FqFyMHZmiZlOhTd9yMog0QLH0WGMprQobrOa3FiVUogpEqm/hIfYuFi0wfXCBrrcC5FBhtB3UqOTi2nM9LrnroO1KGJV7urGdjzF1oqvSq+9nf+FmenUxiO+PB3/4ifzJ892D+L9BbAwfXrlJVBKMG0nvLobM6DC7fRMWleXoFCUCpBUUh0nnP7+oh3j844vtLK70+OgCQEFwltY8E7ELHJbiyHVOBlHIsaS9NYBNAzBi80FJ7pYwHOEqnXxfv4MTiNu2uYXhuhtI9dH40B3YvSBK3p9Up6pkRJsL6maWrq2YzlbEH8Rvq8uE2IpQkwVAW90Q7CW7CWxXKJ1BqhNSIVtFnn8cvLVZu2yYNXdDbFAZAxyioSSwzEYjObJAeIjRyCPEVlBRi50fAWHvQSQgFtaoyxTI+ufcOzcD3JHAaJ+EobNb4E0M3GP5gLxDu5UD4TWXqrXcLidEPYvY3uELF09hmfsXHfl4+GjS/PZc8rMzwPeUgfOQvYb59xYh/xI77FjCPikrpbAZn0/DTNDBfCSIE2OUHUjKg5TxnIJlLi+hR1VlLPDHbex9sC5wNubRPyatgS39fAsg7cffd5yZHLwqa1w/qVlMoVbKS83WSYbT729Gaef6cnsqdyTVUN6I9GDMeTFxDfp6NRyXBGCJq2hZQ6FAKEytBKIjIIYRUjLy4QVsmfL6wuRLBJGl+Pc5eYdg0Qgsd7jRAhNtPgwjEgU1gituDUhWE0GjHZj4VuTxNfgPOTB1BogoBeYdAyNUlItkZXCZ1h3FWM+B4OoJxUtDQcn59yLhwSv66W7ghLqwzWlBiVo4xBa41SKhKBIBA2Xn9d4leIOOiGZLKuCVgVk4ZBglUCZ2LjGI2hyiEETQiCpgk0jad20YrnYFzy00vwO9i/eZ3eoOTN+SFffnSbsz95l+OXXFdfA7QCESxt45ifn3H8jDnxAZFmje4GpPyQ/k5LS8Hs7CQlJj9faBYLmsbSNA0suxMSYtgyBLx1eAJ103B2NkstjlPtgxZoNEpJnEif+Qjuw3kJTY0bGrJRn15Px3oQY9DG0O/Fdu+CQGNnWGux1sWQqdTghnxcf7yTx4/p93p4fw0QWAfWBmQh0KmFtveeYC113TUVuBysT0OahCRddiUuNiVx4Zk6QdPaSB7Xjgdy83O4sAL3PkZrfRsD8xsRXXx+3oxQFLFt/Nr+zINsoWihauNQvhTx+AoR5RRh+fwahxs9GBcpiu3jZyoZtcNomAt4+NTpHWiYFBsZyOXisy2jg4aGmlOOEHbFKQ+ZcZf4zXQnoiMYq/VrIY24IR2vp8Wvu9AFZgyYMU5zbkh7OkPMDdNpn+npmPmJoZnGRePrYEt8f16QUlcfsW4RTzytkQk2S9N1PueparanyO8qFcN1ZFnrnLLfZzSesLN7neLdH6aWhC+CJpYK7EBVEVihc0U5KNG6YtWUZCEnA7xzeB/wLj5Wq9U66Nzpj6XsNL6XqIz1UQwahI03p4w3qKSTOsSmBsjoual1QW8Ao9F1RpPd5272/N77LJZL9iYTBkUPv1wSrMVdwXZuFx2FrhIO3oRgBD89esBxc453DTI4VC7QUqNUJCMDkzM0JjZqKSvyMnYMLI1BolEhEt7gSNH4GCGJfCfgg+iy3nEiTtErpUWMmuYqFreZPiCSRCLQOst3/3SfP/y9h5/+Hy9atFlx82DAr3/lNiIPfOv7P2V6DNPV86e6fWBUgV+eMV8GFu2SxyfNc9//I2A5h0ffdZTyPl7Bu82rO39eZZydn+OsxS4WSa6QUq/W4qylaZZYD7PZjNPHJ5zNTuiZgtKYdXGrkCKNnc8bC9+HpoFHO3FMHd6gLPsUvZKy18MogdIysq212kLEkKnW4M3H9uJdfXCXk9KwMxxipGSxjIS+N5AIER0dnA00jWW2uFw7mYtSsW4R7W2KpKY5pm1Txzaf8mv5k80uOgnD2vs2eZDJAHYB7SIS0ykvXhJkpDbBaSUfSIVxyxj1HYXo5LDUKcuVdF72OV7Ze8CtFEFGR3JeaehJsBUMlsAAjs82X1kG3Cjg1gCK6hkb/bmHRJHj8cyYMuOU6BPTECmlJMovFBvpY4ZFoVJ+UbKgTd3c4u8bVlRMGSJRFClE0TJHrE6YnY45PTrn/Khkepy/9jpuS3x/npDY4Es72n6EyTyD2ryA7SilKcuK/mjIcGeXvhmzfGHLGkmsRd+F8gaYEuvb2HGrMmhZ4mcFotUx1RUcLqyiA1qafzrSm2Uxkp2LRFhWlxnxDXjvY2QwWISXT0izhBBJ5hBPltaxaGQwGTHamfCi8Lk7OeKhDdhBHztd0DRXz9OsuwSuYnxv71ChjGE6mzKtp2BjK+LSGIzRsaBNa+paMTeaqjaU1YjKlVSuTz+UUf/rY9ahJcQC+8YjJATfWePE3KdQnbxFgAroIKiEJA/QHw4YVEOEVkih8dbimpZbB+NL+duDq1HCcW1U8Etf2EfnMNkpeHjvlAcfnHH2uGFuY2lVF63aBd64Bj2j4+TSNthkx/UifECM15jEPj6PpBfg9OgEay3tbBb9xgOAp1ksWCaCuLSe8/NzTk5OaM9OCaMhWio0AnRqr76uXH8WLNGEDjjvJY2tQAmDMSVGx4K00IldZSLTMrVMBpi9KCUHhA+ZPh4x293DGpNs2gJ9sYn4WhtYNg2LxeWmv5+I0naaXp80v6RobwtuyYbh5sRWxcCqC7akuIxPC8+OUPt5XCO8qJitwyAdSqcrthbEMkomtI+k10pYplopa+PxteGja40d4G3g1jUYjTdR7EJAlcd9zJfAEB6lwriMmG25XsDNAVTDVzqlnyEq4ujxSSSaAonE0TKlZcYZm+VIIOp6OwFdmbaf4xE4AhlLAg2b0uQmvWfOgnOiGWWFRGCZsQKmJ6eUZsjpUZ/zI8MqvF5LvC3x3WKNpwm1Ss4LpTEMhwNGwz5Hj55FfEuigXYJ7EB/H3YPgJrZ41O0qRgMJ7Ffi8yQKg7qARGL15wjhEAWQIqMXCqkik0rVKai598ldh8K1kcCZANBhCh7CDGVJUTU9mq9ocJCCLSUjAYjdvf3uHFwh/v3fvLc7bvZMY9ml1P5/7roWi53VjNXCTmwd22fyXjMeDjAlAalBVpJTFkidA5J6+t9jXM1jsB0PqeuHXNTc6YMRhryRJC1MhiTo4xGS4WSseOsMiVaGRBRHxn5UKxONgE8EmMURhsCAR8aCJFEa3U5sfL68b24IgwNg7Dk9rWcnd4uzc0xiy95mtmSWV1zNp1SLxc0bcug32NQxYYarrU0Nme2bLk5b3h8AifHcLSKpTAX0d37DU/WZH/eMP/udyNjqeskdYgMaWoD76MpB32sDcwXM9zJOcwWtAjOAKs1Ta1Zns9S8wh4vk1YAGbQzGg/eMgHFiZ7lgD0Ck1pYvbIlH363rJsGpx1LGYzVtYSGfDHjHmP73L/3g7D0ZDQWISIzhT94QihNU2I1m32kjNMPmXaZTLXDUlmv86gBKCNbYhT3XPUvnaOBym7oiVUIkZTq6T1HYhIdjttb1dC/SzsAJNBlEeEabQu8wF6IhLiQQHXkl54IZNsYg7TxUfHvh3gN4Bf+WW4dgN6KeKLTHrmROaXA8gHMd65IO0HeKOAW1V0lbiKeIu/xiFvc4u3aWj4gPf4ET/kiD95qc+b5MYQKx0ujiYdke0WbV2xWzwzSxwWi6S5oPTtE93Ih2gyPDUNDkeNYIqiRHrB6oPAhz/J6d9QDM7KtM1Xw5b4bvF8ZAKlMvI8x5Ql5rmCpXF6FKDHMJjAaBemD2A+o2lTU0KRJBgdT/COFYFVCLBapWhv9FjNRYaUKhXtBay9PF8Y7wPeeoQUMerrQ3QCSEUa6+5yXaGb1OhC0uLEMWsAACAASURBVOv12BmP2N3dfSHxvcroLC2vIqoBDKuKcVmxWw7pVyV5z2DKHGVKUBrnLM476uaUWW2pXY2zgibUuEZRixata5SI+srcQElsD1v6WF2jBaAMwpjYSlaqqPsTjhA8Skty4nUZm79YgnUx1CVEypd++nDz6PsscBQ4VE+y0x8grgkICtc66nrOfFGzdI7GuVhUI8DbluWypmkNi8ZhW4+74Wgbx8PHc95/BI+nayMuSiKp6BKTNZ9T8nv3PTbRqI4QSnjcY94f0jQhSpEWi+jVu1iAFiy1Bm3xUsLiYujSEKfRZ5HU1Pj5cXR8OBYKY3qIEFBojBJoqbC6pCh7lMM+1lvapSHSvo9b7M+Yn5+gtKHQAmMMpuxhemWMpYU4joVL/iJdG6O0wl+QGaRaK5EyCCJEwohPw7/fjDs+pMI2YvyxIFKaQkKv2xabbm3PwpDYO3TQi58XnbOmj4S8KuCahpvjeKxTYNpG06+nHeQMsSXTnQG8eQN6g2jDtpZuEP82KyOJtgVcF3A9ySgGxH1NdCTbVw0DDvmb/AvcOXyH27e+SNMsee/uu1w/uslv8xd8fFzdI2jxzGk5I5Lbi7g4Hs6I4syc+O12l0YAFBkqvV5hKDGpuUVI95enwVMzpwQMx0d7HD/cpXGvR123xHeL50MAqxj98qvAs2uyNGvSC3EU7C78PEaCdVmhVEkWHIhkbxI83qVy+6iES921MrpObfGhWHlL01xicQY+iu59wPqo95PWYoVFa4/VAimjrnPTZlkgpGTQG7B3/fk636uOjgJ0dbhXChLqxlHbhrppkFrgZaD1Fl/P03dmYxtsLCE4jNCUZQlBRCmN91GPS8A1Ya0hFEKgQiAITyMDpbO0jUGgENIkGyhLCAGtoqRCG4lyiuAbrGsI1hGAenY52QilFDGiLSAIRNKUKiK5zXNBrkqqUuFCwIdAtpafOqx3OBdtAp0LrJzDtYFpXfN4OmM+q5nVS3AeBSgCSwfWwVkD02W0f7IWlguY201S1LFphWx5trPE1cQHxPFpmZ77QA+mFhYOJ5q4oFnUsGiiVtcWEHz8HiRJl9VJHRqer7YOrK2lTqLn1z0psdeGSEYEIyAsaFILd10YhqMhtjCcuSa1Qn4xMiHo9/vsjIYMRzvs7OzQM70YFQ0BoQ26uGQRk401yJYUDSUS3ZxEVmVcG3Y2kSKqO9B5GnNEyjolH+BCRjlBQYz+6pTVXhApUk3MUHSLdsFGXjCqIsnVRfyFDLDfg0kVo8GDQYGXoH0MS58uYHn/ycK2XRKJHsRjXU7jfXBxASFSFLu7FEwO4+UF8i6jJVt+9co5+DJf4Te+9FcZ9XcZl7s0qkXsx7/px8d/k+/xT3mxJVpNzYwFR3x8X0mIRLpko/stEUkukVNQcJ2SXaKYKI15WAItPj1bzqkpmdVzzusWX79em/gt8d1ijdWFYo2saxyRCUImcAHcR8ZPTZT/j9LPdWxi0XXU0iVIiSoGZLpE+Cbp2hQrn8hm2DSrEAREJhA4MhQiCJRIUojLNNMP0TnCp0HMSoFcWryOPp4GiRUB3Y3cIQWDtWbQH7C7v8v1/uTKyhlehBWbQf+qEV+hoHENtcuZuxoaMASEapg3UdoQB0aHMTllbjCipDIVQcCsrfGNxfoQrfFkwAQAgZECh6PF4mmwTY1RBk0kvyHEyn4fQsx2GINxitxogrepmUGU6NT15RBfKRVhlWqhQ/RzESEkW7Z4r1SqQIhiXb+adRLlEIv26H4GSO3LrQu01mMTEXYhxN85aKylbR3zRcu0tiwWC+q6YblcMJ3OaZbQNtHaaVbDfAHzJkaOp8TYz9Umwfd50sMkmdA6C3Xyn/IN1Iv4WDbQs6n4NcR6YR8ubOMis9FsbKVI/56x9iM4AYTkFE/fRIGRtTXWRpNoowxmaGKjYR+Ynp1DuPvCv0YCo1GfGzducG13D1H2kTrqz23w6FAQLqej9gZp5eySrtcQiV9FIq5+4+6wljXo6Lwgk6ShS3oLEmFOmZiqsz4jXl+GSKEqnrTInQDXE8Ed9OL2Q1oEXh/A9dTFbTQoCAiWFsJiyodzy+KDzRioiCR6Qor0CphOoWmjfMOmJE8u4zEionIm17HQzRBJu0nHZ65Y2uQN3ubLfIVf/9LXEEFBkLTWMtQV/aLiwfE9FJpv8/tEw8NnoaZmSqwEeNk/cE48qzlQkmEwCAyGCTfY5U1AEHu/1TTMUzfRmq7JusMwrWvOao+rX28xtyW+W3wE2XqmTBASZSpU2Sfe0hdVVo64Bo+lQyCgnsHZMQwGZNUOmTRYG6jbFhqLah1OOqTonM3jRCKADIeKXewRIpCRse7qdonwPsTlvfBgLV5bGmuRImrnZNDJ1SyO3ALQQmCqHqP+iMlk5+eS+MJGancVsD+GB0kyZgyUVYkxGq1AikCgwQVoXE2bCt083cJJInDYsjNQT2Ql/it2MkPQBs/poqG2cZEFlqBLnNIoUUYdbxDY0BBCQLoY5Vc67iUER7AO2zQ4F2jqyyke8olgBRdw68LPi1ohYlRPCDIhUOlX67cEniS+iTi7PNm2uZA4XGclGAu7XOuoly1107JYVtT1kuWiZtrvMV8sqOcNi4VDK+jkzYuGdVvSq3Q9fRRPf1fnxBM1gqOdTUVVgKiD0WAMqtdDCbDNMvr/PrPlS8mTkeCOIHvimTmJVr9Y7ovAaNTD+yVSBpSIBbO9VLSJVDTB0/75ES+SPCgtMNqgez102QdJ9KYOfq1Rf17nzk8N6QvP8k3EtpIwkNHWq9BxWJXd9QjkeTy1HcGVJDkE8ee8I8hES7CFBmMjSa6IcoJuJuquN++j9Rkh7SvpcX2IRWgyAPY0uke0cHoOi2ksbKuIV8KEGMYpevGbWzTQzOPnbcp+aAE2HX+e2PegB/s+WakVMBhH+cZVwohdvsJXuXPwFsNqgF8GljagMgFCMzYDfuXOL2M+lBir+CZ/hHtmLzuXxs1PMu51zTX6gMKhaPAIBCscEDAYFBUWzRzHnCVtMlWUlBQMEaMqau1fk7puie8Wa3yE8KbZUyiFLgfowQ5R1vAg/b6rsz3jiUtpehZJ5GBANZiglKRpLWHewLJFNZZ+7jEp0sQqpKl8FU3kBaisc2FL2t9LDPiuq4c7XW/q2iSXS5ZCIKSm0AXopEHOJVJIhNYUxjAYjNgZjV64j4/Y0G3xEVzP4fZhjCDO6tiytxqWlKUhNwJ0SIVlHhcsNlg6WhtLyC2g0M4ilIheqzo6N/iQNGMiUFuo7RmqBi09SkCwJa3UFKYCWSFjP6xEW1RMCkSDs5j2dhbXNjSNi56wl4CY6YjElxAQKnY7FJBIbVh3ilQKgojENqwXkwlrwpGUdSKy4bDuEra573Ub8CZQFJqqLWjalmVTMl8U5Lmi0JKpEEjmeBto2kh+r9gc/wkxA+5DvZOYl45/lFJgBPR7VP2SYBtm5y7KH9Jk/SQ6w/5oz7RB977zuK/HlqUILBd9lAZjFP1eD20KTK9Hf9jH9PoY0+N79x/DyZ8+98i1lGgTG2MIrWmsjRp0HzXownvkZYcNlpDJjbShIhLekYCx3HjqJoey9XqiI7habqK9miSDWAuAI5HUBRSLuO1AJL5dxLeT2yyb1JmtTQ0q0r3RJGbcLFIZVnp9OoXzefymRmkbN9K2RRE9hxdzmM7jdttl3IbWG4IbUuOO3hiKIZikJx6Pk5TjCqXR3uQdfm3869y5cZtSa+rWIoLDr2IXtVIbvnT4RW5M9hi/t8Ph8SH/lH/CQ7771JYc4pUqAI6I31YFmBQmC7TUeGoUFQM5xHkJzHEI6iTCUgzojfdQu0NCYbDq9VribYnvFcW//Zu/+bM+hOSdG3hw/yHf+IM/4KO14PBk286EVXrrqWbGjfTbkF78OOH8Z4+AxwaPCJ3WVYBcxjTWInVtA7QtsNZGdwAhWCwWsQMUKa38DGTECSBozXlz9QRfTzc8+VniUQuPLoyxP/oh/Bf/+XfWP7/s4kGmGTh1wN5kny+ys8SXMxEnYJUiT7mOer0kQ1+7e0SCEh/BRu7TNLGb24eXFOhfLKIOdRU8YeXQTpErkZpvbCK5ToHyEuVE0sdHqdFa73vh706cd/3ZpIbofhtJrBJkKJQUKCVQyqFkXAR3bm9SBFZivk5Dz5dgV/FU/3zmPe7Hx7NMYu9/tHzn2Xj0gt9d1Ew+jhf6o41WuhsV332p/Wwwvfun/NHdjhiXaWuf8ThzFu+zJj1+Fo2uP4CYcngvPV4RR8C3IV7En8KFnAHhf3j97bwuciq+wNu8c/Alro/3UMGjEoEVwRJCiwiecVnRLwu0VBzsHmB+oPhd4D5/zmYBl6NeWXrYZVji55dYzjleF7YNwwidCeRKoSgoyQlMqMwtyoNDhKlo5ha/nBKXKa+GLfG9ovjtf/Df/6wP4VOABV6sUbsS8AG8X2vCPBbpZSK1guligbUeLZeIhVy3Hl0sFiyms/T87M5dMYCkY5jgChLfESmlx9Wv4n9Zcu5bXjrfvgrxrVcxPV8nn9kQIj0KSuG9IMs2BW4QI7heCZyKHrOdJEJpib9w1oQAT0YmQnKneMZO02IhhO5/qf2qEOQqx5kW14u+VSsCmagJWYzSh1MIq3guX9x3bIvLweVZPm7xargKAQWAQw64xh6DYkgpcoRvEcGiQoujJQSLXzl8SlQMi4qqKFlMfx13L/AX7PFdvsmKKTkG+Vo1Nw2b5XYkvhpBRclwNUShiGanBkMfpQeUe4cU1R5YgX08xb9mjmlLfLf4hcff/4e/zWA0iCbyOpYcSxFTh1JKtNBoHV0rhBRoEclv0yx58OE97r3/Po8e/uiZ23bAo8ZeSdILz47hb3E10CxiyU0IyQBIeZTKEKITdcSoblCCEASii/iKEDW/IU/kOGmgL+iDReZh1dkKXUQqak0vOxdYdRp8BSpXFE4R0LjQIwiBDXNGXS+IU3Crz7Ed2hZb/Jwhp2TCLhUVEgUu0C5rnLUQWlbOEnxD27q19CkvDKbsc3vvNs4FykeGOTXH3KcklqG9XBbkWYijVydw8an98YAxY84pGRKb0Rvy3pgw2CEzFcFDezYl1E0aoP7KKx/Blvhu8QuP9376/Z/1IWyxxUewbGKhVAiRzAYnCCppfEUsiBJCsAoSt5YvRHmSEoI89mdeR4azJwrjwlPPT2LVEd/kDuFcSJHnQKYE+SrHFoHWB0rjGA1iqEgQI79uBQ+5OhGvLbb4RcWAPiVVzOzVc+ZaU88rvHe0zuJsi21rrHXRsk+oWPEHlLrk1t4h4Dh+9IhH9Ak0WOY85AT3WrmdkliuGGiZcso5Q85Qsk8x2EGZkhrDPGjctKaZ3kfVC0Q9JXOvV/SzJb5bbLHFFlcQzbLzxA7R1F0JwqrT8YLPBFIpggsoFYlvCAGCi9KH0DWAAZGBEDJateEIYXWhMUvaYeeeEi7Q4lU6gmT7EJs5ZqxQ6FyRt4GiVzDygQxLRmpScxor+Y/4zBWnW2yxxQX0GVDSAwJ1PaWWkrkpgcBq5XHtEts0NM4SggTyJLESlKqgmlToXHM8P2KwKKk5YcoxE055+NrEtyIKoxqmzDlnzsgIqhsTTNWnOVvhz8Cdz7H1lIwjBOeI1xSnbYnvFltsscUVhGu7qEZqfcUqmfuQ3BlIhBjatnNG6Sy0YhylszoT68hvjAgHYiWnSL60XXHmukgzfSa6aHCBDEdtsHMB52IBrECgdUGvJ1iFJhbPaShr6J3FwrATttKHLbb4rJFToiiT63RDvTyn1oq6qSB4BAHrG9pmTtu2rMhBJONnIWJmaZWjhWJYjWj8NeZN7Lf2Du8wYMKP+CGfTDRniG1C3qDPEEmDouUah+z1folrX7jF4OYEh+J0fspyfkJdH9HwGMUUQ/vaLjJb4rvFFltscQXh1q2ikp0Z4IJYuzI4QiTELiRtQkiRXYFLFmhrOYQQ+NQXEQKrpP31xOK5QICwWhNfpSQrpTYyBwRK5UTpxYrgYye44AMZgiIvgBxBjsxbepVlxwWqCs7mMJnB1EcCvPzIX7rFFltcBkqGSAwBaGmp6ymLXFPXfYSIzg6ubbDLefQlFwZU8kEXEiEylFOoTLDTG0bfabFC1dBnwptAScWf8buf4Kgq4DojDthjl14G5QBGkwOGe7eYvHFItTtgvmjJPjylXpyz4AFT3sPgCWjUtrhtiy222OLzB2c7r9iue5vCq03MVwgBagUqW3dAVCr5HQcQYoUToFSK3iYZhBCCTCl86uTWkV7ovM2S44PbRHkVAiUEjtj0QxBlFoUGJyGsFFIFciXRbY4PjrByCDGnyGFYwaKFnTmc1FsCvMUWlw3FBMMOkONZxYivnVLXmpmpYqQ1tDhbY5c11lucKAkudTNUGuEUIlNAxrA3wuFpg6V2gb2yYlj2KR6UNAT+P77Piy39OggUCsWUmgArjZ/m5AMbG+tkIEJAW0vulhQ0WAKGHENAoaPl6Gudmy222GKLLa4cvGuTz3Kkn46NnCH46K3rAhDEBYkCZKvomh98jMGKaH4GoSU4h1AKgcB3n+uEvSJEiisyOoNe0blACBUrwlNRnU32ZuQCGWAVBCKLZFwphwuSEHKWbUMIjjyHMiTj/zn0zuGRf9Lddosttvj0kDNG0INkPmaxLNuGpa5p7BwZaqyr8bambRuCc1jlQAWsMghXIaWlFRqEoMxLKt9D6QpUTa8asj+eUBYl+sOCvh3wTb7Oy9zVAwItR7QcU1MiVyUcDzGThmFtcXkNdY22lhJHIEcwRBAwrx3v3RLfLbbYYosrixWsu7R1XTjCKqROcoGgwrpDRSSxglVqzLHBhW4V4aM+C+veHk83+Vh/unOSyMiS2bXIOslFR8Vj84uwgiAUSsRoslISJRyB2MLV5dC2UBZQzqPd3+L1T9MWW2zxFNYL3qTVD6kZzqprae1stDJLDSzCykPIo3tLlw0Km2FBCokUKo0GAiVyCmVYlbA3vsb40ZhYsPbxxFcScFgsgRWgEbRLi3Uu7t9JcAERApKAJIsyKgICsW689KrIVqut4cwWW2yxxRZbbLHFFp9//Hy3WN9iiy222GKLLbbYYouXxJb4brHFFltsscUWW2zxC4Et8d1iiy222GKLLbbY4hcCW+K7xRZbbLHFFltsscUvBF6L+GZZ9ltZln1uq+OyLPtPsyz7P7Mse5xl2SrLsn/rGe/52+l3z3v8tVfY7y/8eU3v62VZ9h9nWfaDLMvqLMvuZln2P2ZZ9uYr7HN7Tlmf0/8yy7IPsixbZln2nSzL/vXX2O/n9rxmWfa1LMv+2yzLvp9l2SLLsp9mWfY/Z1n21jPeK7Is+w+zLHs3nddvZVn2r7zifrfnNL7338uy7H/PsuxeuqZ/6zX3/Qt/XrMs+2KWZf9VlmXfzrJsls7tP8qy7KuvuN/tOc2yQZZl/0uWZT/MsmyeZdlplmV/mGXZv/Ea+/6FP6/P+Ny/lsaB91/3GF434vv3gH/2dQ/iCuPfJfpz/PYL3vMnxHPw9ON7wH3g/32F/W7Pa8TfA/594L8D/i7wHwF/C/i/sizrf8J9bs9pxD8AfhP4z4B/Gfh94H/KsuzffMX9fp7P678KfBn4r4F/CfgPgN8A/jjLsltPvfc/AX4L+G/Se/8A+F+zLPu7r7Df7TmN+HeAPeB/+5T2vT2v8C8Cf+f/Z+/NoyPJ7jrfz72x5CqppMoqVVXvu40XjDHm2MwAh8OMeQMMY/CAeWOGZoZheAzD8sx5YPzAjTEwMwz4sQ3PbPbYhgEGnoH3bBrMYjBgG+MFcJvudndXVy9VpWpVpjIzMjIylhvvj98NZSpLUkklqSSV4ntOKJURN27c+OWNG9/7u78F+O/AVwLfBpwAPqKU+txruG4pU/CRyHw/hoyp/yvwMPAupdR3X+O1S7lOQCl1DHgrwql2jjzPy22DDdD2824kpOb9WzzvNiT83Y/v9z0cxG0rckVIXAr86NT+L7PnvGq/7+MgbVuU6T9a7xhCls8Dzn7fx0HagBPr7Cue7TdP7DsJjIAfmir7x8Df7fd9HKRtqzK1+4s+7dp++8B+t/+gbtvoqy1sGNOJfXNIMr137vd9HKRtO311g/M/BPz9ft/HQduuRa7ALwB/ALwDeGanbdh1Uwerin6LUur1SqlzVvX/XqXUSbv9plKqa5etv3fq3BNKqbfZpe3Qlvk1pdRN61z7662qvFiu/edKqQ8opT4wVa6llPp5u7Q7sud8y1buL89zcw1iAfgGQCGz6m2jlCsgLzuHK6Nhr9jPbfXdUqYAFGY3vz+1/0Hg9MTxLeNGlmue51fk38zz/BySl3OyPa9CtD7vnir+buBF6ipLeOvcVylTdjT+rotSrpDn+XJumcTEvi7wKFPy3wpKmW6Ky0CyhXJXoJTrmut8AfA64D9cre6tYq8yt30D8ClkGWUR+L+AdwIzyEv3F4B/CfwnpdTf53n+PnveApLC/Q2IEM4Arwf+Uin1vDzPIwCl1D8BfhX4PXu8Za9RRR5gbLlZZCm3hixDnkVeUj+vlKrkef4ze3T//xr4eJ7nn9rleo+MXPM87yul3gV8h1LqI4jJyG3AjwN/i2jTdgNHRqZAZj/jqf0j+/lC24bdwA0pV6XU8xEN7z9M7H4BIsPHpoo/ZD8/y153pzhKMr2eONJyVUotIM/+27dT/1Vw5GSqlFKIsmYO+Bp7nX+7nfq3gCMlV6WUZ+/px/M8f0xEvAvYocr6Aalizb4cEZA7se8n7f7/c2KfC1wC3r5J/Q5wiz331RP7/wr58dXEvpfach+Y2PcDyI99z1S9vwgsT7bxKve5ZVMHxC4nB76jlOvO5Grb+XO2TLF9mHWWSkqZXl2miJ10DvwvU/t/xe5/QynXTe/VBf7Mtnl+Yv8vABc3+S2+oZTp9mS6TpkdmzqUct2w7K8imaPvLmV67TIFvp3xeyoGvq3sqzuTK+LX8xhQtd/fwX6bOmyC9+d5nk58f9h+/kGxwx5/DBH+KpRS/5sSj+gAsfF8yh66zx53gJcBv51bSdj6Ps6VWpUvAz4CnFVKucVm23Ec0cTsNr4RWd74tT2o+6jJ9S3IEsf3AF+EzHaPA7+vlGrs0jWOkkz/EJlR/7RS6hVKqXml1L8Fvt4e382l5RtRrj8LvBJ4XZ7nnckmIy+GaeySemIVR0mm1xNHVq5KqTcgzljfnuf59IrFTnAUZfobwOchDlu/BPyMUurfb6P+reDIyFUpdTfwRqRvRtuo76rYK1OH6U4Rb7K/WnxRSv1HxNPvJxFv/g5iy/nhiXItwENmB9NYmvp+EtG6bGRnc3zDO7gGKKUqwNcC783zfHk367Y4MnJVSr0A8fb85jzPf3li/0eQWe83Az+10+twhGSa53mqlHoNMin7q4l2vAHxmL2w02tM4IaSq1Lqx4BvAb4xz/M/nDrcBuaVUmryhQHMTxzfDRwlmV5PHEm5KqW+FfhRRFv4K1upexs4cjLNxXa1sF99UClVB/6rUupX8jy/JlvfdXCU5PrTwJ8AH1YS1QHEl0LZ76M8z4dbuc409or4XiteC/xxnuevL3aoKx1DlhFhn1zn/EXGsxgQ4/JLwHducL1Hrr2p6+KfIy+7a3Jq20McRrm+yH6uCQeX5/lnlFIrwPN34Ro7wWGUKXmefxp4iZJYyA1kEvHV9vBu2ffuBAdOrkqpNyKTsO/I8/xd6xR5CKgAd7HWzrfQenz6atfYYxxGmR4GHFq5Kglf+N+An8jz/EeuVu91xKGV6Tr4G2QFeBHYcezZHeIwyvWzEL+e9bTrHUTx9V1Xu856OGjEt86VXvzfNPklz/NMKfU3wNcopR4oNCxKYhDewdof50EkvulTeZ6vN4vZbXwj0iHeex2utR0cRrkW8fpeDvxdsVMpdS9wDHh2j667VRxGmU627UnbFg+xTfvDPM8f3+vrbgEHSq5Kqe9ATG7emG/ssPEgomH5V8APTex/HfCpPM+nlwmvNw6jTA8DDqVclVKvRhzZfinP8+/Z7nX2GIdSphvgi4CA9TWo1xuHUa6vZUJrbfF9wOciDnzXPJk4aMT3QeB7lVLfD/w18CXAa9Yp9ybEXvE9SqlfQFT0DyBkadJO8a3A1wEfVEq9FZmFNIDnAf84z/Ov2qwxSqkvQoJ7n7K7XmbtY8jz/Lemyp7EejXu4rLGbuEwyvWDSPSGn1BKzSOz51sRY/cu+69VP4wyLWz6ziFxe29FQsTcCnzBVm98j3Fg5KqUei3i0fwg8CdqbRbGntWek+f5JVv3G5RSfSSpzdfZtm/6u10nHDqZ2rIvA25nHLrws6ypDsD78jwPr37re4pDJ1el1BcC/wNRJrxjqtwoz/NPbPHe9wqHUab/HgkF+UcIGTuOmDy+Bvi+PM+no+jsBw6dXPM8//A6596P9NMPXP2WN0G+A884NvY8fMvUvvvt/run9n8A+IuJ7zXg5xE7mT4SWP8O1vHoRQzyH0HCCD0EvBr4BPCeqXLzyI90FtHKXEJI1Xdt4f4+wNqIAqvbOmW/2x773J3ItJTrmnLHgZ8APgMMgacRB4L7Sples0zfYusfIXZb/x24peyr697bOzaSKRMezrasg0zKztn2/B3wmlKmO5LpZmVvL+W6fbkWMthge7KU6TXJ9JXA+xAfiRGyGvlHwJdfy/NfyvWq5+44qoOylR16KKVuRuzrfiTP8x/e7/bcKCjluvsoZbo3KOW6+yhlujco5br7KGW6N7gR5Xooia9SqoZ4J/4RYpB9J/B/IAbYL8jzfDe9048MSrnuPkqZ7g1Kue4+SpnuDUq57j5Kme4NjopcD5qN71aRIbaMP4sshw8QFfu/vFF+mH1CKdfdRynTvUEp191HKdO9QSnX3Ucp073BkZDrodT4lihRokSJEiVKlCixXexV5rYSJUqUKFGiRIkSJQ4USuJbhiRkOgAAIABJREFUokSJEiVKlChR4khg34ivUuodSqknd7nO25VSuY31VmIKSqkHrHw2tO1WSn2xLfPFO7jOjusosfp7fcl+t2MvsZU+WeLg4yj01euNUqZroZT6gFLqA/vdjsOAclzdHPup8f1hJD5ciYOFjwOvsJ8l9hdvQgKNlyhx0FH21d1HKdMSJfYA+zYbyLeQHtWmU03z0gPvuiHP8x5wRcaUaSilKnmej65Dk0qUuGaU/bREiRLXgnLs2BiHXTa7rvFVSt2tlHqXUuqsUmqolHpCKfXzNu3sZLk1pg4TZgrfppT6L0qp80jmkGNKqfvtsS9USv2OUipQSl1WSv2cjTu3WXs+Tyn1W0qpZ2x7HlFK/ej0eXYZ5S+UUl+qlPq4UipUSn1KKfUv1qnzs5VSv6eU6tg6/1Ip9Y93JLjri+crpf7U3uMFpdSblVIa1jdTmJDNVyqlPqGUGgHfZo+dUEr9mlKqp5RaUUq9Ezi2L3d1gGD7yHtsPy363RvssX+qlHqflX3Rz16vlHImzi8me2+0v0eulHpgP+7lOuEOpdR77bN9Tin1g0WfBFBK3WfluWLl+WGl1JdNVjCxvPdCpdQfKEnZ/Jv22KuUUn+llOraazyilPrBqfMP+3N9TSj76u6jlOnOoJR6rVLqYaXUSCn1kFLqitVhpVRLCbd41pZ7WCn1LeuUu0Mp9atKqedsuU9O17fZ2HHIUY6r62AvNL5nkHzV3wV0kADI34+k9HvFFs5/I/BR4FuQNKDRxLF3IwL/b8DLgR9E8kPfv0l9twKfRFLd9YEX2PPuBF47VfYu4KeAH0OCN78e+C2l1PPyPH8MQCn1UiSu3SeAfweEwLcCf6SUemWe5x/bwj3uN34H+BXkPl8F/ACSh/uBTc65F/hpxETlCaBt9/8/wGcjv/FnkPzdP7MXjT4sUEq9HEkZ+RiSyvoZ4B7gxbbIncAfI3KKgJchsj8BfJ8t8wrgQ0i/fZvd98xet30f8R7g7UgKzK8EfghJUf12pdQZ4C+Q5/fbgS7wH4D3KqW+Is/z35+q63eBXwb+M2CUUncCvwf8FvBmJL3mPcjvANwwz/W2UfbV3Ucp051BKfWlwK8B70XewSeQ97KHpNJFKTUL/CWSivcBJG3uq4CfV6KN/Blb7hbgI0g63e9GUvZ+HfDbSql/kef5701dfs3YsXd3ed1QjqvrYac5j7eQc9oF/hGSh/lzpnIuPznx/XZb5uPY+MLr5KP+v6f2vxEJuHzvVB33b9AWZdvzOqRTH5/KbZ0A90zsO2nr//6JfX8M/APgT+xz7L7f2Wt57vC3eMDK5/um9v8i0vmPAV9sy3zxlGwM8JKp8/6JLfvaqf2/P13HUdqAP0cGl/oWyhZ98o3IRFFPHLsiN/uNtk30yW+a2v/3wB/a//8rkDKRj94+c48AH1+nru+cqus1dv/sJu04tM/1DuVf9tVSpgdqQwjtp6dk8flWHh+w338AmTTcM3XuLyJKK9d+/2WE7B6fKvd+4JMT39cdOw7rVo6rm297YergK6W+3y47DBEy+UF7+L4tVPE7uZXOOpheevh1xFzj5Zu0Z1Yp9Z+VUo8jphMJ8C5kwLlnqvhn8jz/TPElz/NLyEzxVltXDfgi4H8iMx5XidekQlL8feEW7u8gYD05NoEXbnLOk3mef3Jq3yuQicFvr1PfkYRSqg58AfCreZ6HG5Q5rZR6m1LqHDJLToC3IBOPk9etsQcL7536/insc4c8Vx/O7aoLQJ7nGfA/gJdY7c8k3jP1/ZOIjH9dKfUapdQaGd9Az/W2UPbV3Ucp053Bmnt8HvBbeZ6valzzPP8I8ORE0S9DNLlni+fVPrN/gGQc+6yJcu8DuuuU++wtjB2HHeW4ug72IqrDjyEzhHcDX46Q0q+2x6pbOH+ztHhLG3y/aZNz3o6o1n8a0VB+HqLOX689ba7EaKLcAjJb+QHkB5/cvh2Yn7SfOcC4Fjmu97ucBjp5nidXqf8oYR55rtZdlrT94/eAr0Bedl+C9MkfsUW28ozciJh+9qafu/X630VkEJ2f2r+mrB3YX4X8Lu8CLiqlPqKU+qKJ+m+E53q7KPvq7qOU6c7QQkwa1nuHTO47iRCn6ef1f9rjxyfK/et1yv34VLkCN0xaXotyXF0He2Hj+1rgnXmev6XYoZRqbuP8zSI4LAIPTX0HeHa9wkqpKvBVwAN5nv/UxP4XbaM9k1hBlvx/DnjnegUmZ6kHGIuIne7kdxA5btQn1vtdLiCd15siv4vrlD0q6CB9ZKNJxF2ITd835Hn+7mKnUuorr0PbDivaSP74aZxC+uX04H5FX83z/E+BP1VKVRCN3JsRW7bbuXGe6+2i7Ku7j1KmO8MyQozWe4csAufs/5eR1djv3KCeRybKfRCxS10P56e+H6UIUkd2XN0L4ltHOu4kvmmX6v5a4E8mvr8WEexfb1C+gsw4pttz/7VcPM/zgVLqg4gz18cP8cvwa4H/NPH9tUCALIO8ZBv1fAiR79ew1rxh2mnwyCDP81Ap9RfA65RSb87zfDhVpG4/V/ukkrB9/2qd6mLEeeOo48+A71JK3Z7n+ZOwuiT6dcAn8jzvb7WiXELw/ImdjP8ucEee5x+9QZ7rbaHsq7uPUqY7Q57nmVLqo8BrlFIPFM+iUurzER+egvg+CPxH4ClrkrgRHkRM8h5a57c46jiy4+peEN8HgW9USv094tX61cArd6nuf6aU+nHgDxETijch2uVH1yuc53lXKfVh4PVKqQvIbPLfsPmS/tXwvyPOC3+glPplROvZAl4KOHmef99mJx8Q/Du7xPBRZKnimxGt+IpSasuV5Hn+fjvIv00p1WIc1WEzW+GjgO9BBpUPKaV+Aln2vBOZVLweGbx/RCmVIS/A796gnk8DX66UehDRJJ3P83xaQ3EU8FZksvp+pdSbgB4STu9exJxqUyilvhVZFn0f4nTUAt6AaHs+ZYvdCM/1taDsq7uPUqY7w5uQd/zvKKXehkR1+CFkCb7AW5F3zQeVUm9FNLwN4HnAP87z/KtsuR9EFGN/rpT6WcROeB55R92Z5/m/2fvbObA4uuPqbnvL2Zv6deRB7QC/itgwrYm2wMZRHb55nTrvt8e+EJlNBIga/ueA2jp13D+17/eRqAWXgJ9FftT1Ihf8xTrXfhJ4x9S+59t7vITYzDyD2G39s+vhkbiD3+YBe98vBP4UGCKDyQ9jPWjZOKrDFbKxx04gxvB9ZGnjnYh5yZGN6mDl8jnA/2tlMgQeBr7XHnsJEkYmtH3nzcjkIwdun6jjC4CPId7LOTI52fd726M+6U7tnx4f7kPC8HWtPD4MfNkW63qFHTeets/rBcQW8L6pcofyuS776sHbSpnuWH5fj5DZEWLe+Gr7HvrARJl5hLydRbTjlxCzhu+aqutm4JcQU77YPv/vB143UWbdseOwbuW4uvmmbMMONJRS9yNOavfkEx6IJUqUKFGiRIkSJUpsFTeip3KJEiVKlChRokSJElegJL4lSpQoUaJEiRIljgQOhalDiRIlSpQoUaJEiRI7RanxLVGiRIkSJUqUKHEkUBLfEiVKlChRokSJEkcCJfEtUaJEiRIlSpQocSSwKwkslFJbNxT2gDkkCu9+5fGoINFn67YNGRLBbsR4KhDabRJ1YMZuRRLmxJYrzsfWV0XuVUN+Lt96VgiLbcn0CCLPty9T2Llc3/RT7ycMe/zm23+Rc48+uJOqNsEpbrvzFk7VLvORh564evFdxLXIteyrm+Na++pjK1EOhjRNidPUjpcG1/XxfRetJ4fvlDiOePqpx3n2qcfo9Tv0eivEcUSSRoDBc11838VgMMYw6PXpBT0WT53mRS96EbfdejvGpJg4ZhSFxGGI1uCiiaKIC0vPsdxe4bnlDu2VHmjQWpOmMYM4YnllhWc7l1nuBQxTQ5QatEnBpPbTkAQDaBsY7I9c5x2VZ0aCuY6uWvro4Vpk+qIXOnmSGaIQltswCPaiZYcX+/WuutFxrXKFvcnctjkKorifyeuqCIktkisnCHHN7LYeKqwSWTSSqBd7rmO3gjTrTeopcajx0Kf+Bg/oDXt7eJXLPNP1uHLmVeIowZh09X/XHQ8uWgMYjIkxxhBGIf3eCisrHZ67dJ7ly8+Rmhi/XsWv+xjqpGlMFEV0w4AwHBD0ejSbTWYXFjjWWsCt+qQYcpMSm5ReENBb6cq1NcRRzEqvRxAOiNMYg0EbGcbjNCWKIqLRkDgMScOAOErJ44wsCCCCbISwzX1G1YckhuE23z91ZEgvyfKVeOhRQ56xv+/0EiW2getPfOG6vM9dIN3oYNVuDYTwwpjIFg9whhDdbKK8t8kFtT1enGsm6i5xw+A9v/FuavU6wcWP7uFVErLLT3Pu8tVLvrQG3hAeRdIklrhxkKaWKWpjOYVlFkaTojHGEMcxnfYy588/y4WlC/R6bbq9NtWqy9yxY1QrPq5bJU4i+mHA8kqHTnuZi0sXuOee+zh9+y0cO76A9n1GJhaCHEd0ggHLnTbaGDSaNI7ptDsEYUgQhkRxRG5YJd7tbofOygq9dptBr08WGRn/+vsju41Q84EM0m2StAYw13CoNhtcWOqxhUdzT6AAH3ldHZRpcZ7sdwtKXC9syqsOEfaH+O4xXERBu+EPVJg2aORd4iBT+kLJ1mdMYCc1udjvxbHi+yQhHjHWIpcDwg2HrPcQwV4qe7cBB3jJy+F4De78KHz0MhzMtIZ3IOo+w5VqIY08scVDZqaOORP/Tx6fXAVUjJdi7Nfifz1xCgby1J6b2i2zbUvsp4+MHhXWn+0W95BPtXX6vlxQ17wSB0BqYjCG1KTEaUQSx6Rxiu/7eL4PwCiOCYIuwyggikOGcUgYRwRRRnulS7XuUa9XMaQstzt0ej1WgoB+GDJKY2IMsTEEUYjRYNKUNE5JDWjXx6RiFhEZQxin9IKQbm+FlX6XIAgZBAFBGNDr9RkEEUmAjIEH9O3oOOMetR30gGiQ4Q96ZIi1W8DaXng9UJBeB+nm5Vp4ieuFm6qwMAePLh3+lY8bjvj6wGJ9hn64gaqheH8W2ljPbjNybgzr259NuwEmjN+JeuL/gviKWV2JEnuGWeAlr4R7b4GLt8Dsg/DY0/vdqrW47+t/kpd/6VfQH40IRjGpgcTkxDEkGRgcUnxylGjhDGhytDbMVjwWZnwavqbmKTxdHBOCJvauGu24KFehXY3WGlRhDqDRq/TAoEyKmydgUkyaYtIReTKE0YhoMCAJhyR+g9htkDlNUneO1G0wTA0jg6gJE3BNhiKl6hu8isH3wHMNrpujtEIDfsXF81z0DvyH0zS2Wt2QYTQgDEKCIKBer9NsNkFr4jiiF/YILfGN4ogoHjEIAnorK1SbVeaOzaI1dFfatFdWGPT79MKQKI3JMMQmZRANSYzYEZvUyDjo+GLza1LiFMI4Fq1xt8Pl5YtcunSZ5UuhjJc7MGOouFCrAC4YDb09XLrwHJ/hNTR2xPhlPwPcfPM9+M0ay+ef5Hyvd90IqG8/Hcbvq5L8ltguTjlwcZvmmLffBGfmG2TDAY/3YLEBZ26BwQCefhpW9qape4Ibivi6wB2tFkmUbMw5C80uCDENWTVniD1kRDnGWrtfGGtvNeNBvrDvhbU2vaW9U4nrgA7wxJ/BqRdAvQf3zu53i67EgAoD3WCkfVInxThCRbULjgGNg7Ya3wpWx+uAr6FZdTjWcKk6UHE1zupDNf50XRfXc1FabFG1Bq1ArRJfMBPPos6twxUjTNRnuNxj1LlIcukCSfcykT/HsHKMrHYSZhLShiZzK+C6UrErdVc1NCqaWgUqVXC0wVUZGoPODcrTaHdnw+sg6GGMwZiENI6I05A4DjGkxKlog8MoEvvefpdhGBLHMakx4Gr8ehW0ZhQXJhOaerOJdl28WhW/3iSKY5a7behqMJDFKXEUEwYhYTAkCkNGlnCvdC7T77VZ6bXp9rpEEaLZbUDjOJY0g4khju0SeCF7l/G4WyjZLUapbNcDjtPA9xFD322imEI1qi3mT93C/GKL1uIit/WW+dAnPrHnBLQOzNv/i4XGiP3RPJc43EiuwQfp4cfhvDfgrOVCTw3gqYd3t13XCzcU8X3erYvcfOxmzj5xdmPz2gaygukgA3AfIb8zjEnxAkJqB4wjNkSMSa6xx4rRZ9IcoiS9Ja4jHv0ruLMLd5yCO+r73ZorcWmQstRPiDNDllpmiqJQoNrQAGg0rlZ4Ciq+puIKqfQ9vdb83gY30Bob1cAF4+DqHGNEZawBnUOx3q6NJrOmEBkeBnBJMUnCsH2R3rmHGZx9mNHFpxlWWgwbLdL5W6B1L7RSTGMOU20I49WKmvZxtYPyXbyqi+eAq1M0sbWESMgTQ5KaHQWM7PXEuUxrgyEmTmOSVKI0DKOAOE4Jh0N6vR7DMCSMQqI0JUfMFOpNTWpSoli049r1aVarNGdTjDlGpV4njGLCaIU4HhFFMcMgJApCer2AQX9Ab2WFXrvHMOgRhQOx3R0ixLUC+DB7DO6+/Q4q2sWkMWkcMex1CYKIXgyjGLQP2tVEsSHZR1Mhx5nF8x3cwfK2rDGKRysDmnMt5hdv5sxtt+P7GR4xjqf54F9/bA9aLJgBWsD8ot3hNPCChEEQE5vDv/Rc4vriWgJvXAYu3yDmm9ed+PrATMXj8mgPJJgkjKKQbtDeeDGrMEsoHNEKzW4RbWKO8Sg3aRYxadebMbbhnbYDLm17S1xHvA/wHoJXPA1z9fF87qAgTqCbKCG9RqFcLRpeLcvaQnwVLhqlNK4rEQxcT0JlAWQYLKclM2ByIbPosf+pKDkNngalzZhv6vE8NDWKkdGkRpMZGAwMF7spS8tD+peGjC4NGVb6jOoVTNJHmx7KrKBrCbo6wPU8PN/nWKOGbtSo4VI14DpWq5xrMBmkkJNj9M5mwCsry3ILLmBSa087wOQG0KuRGkZhSBSFpNGINI6tdjglTWPiOCaOI4wxuL6L77pUm1VmmnV8rTFxQhiGdNttVlZW6HfFpCKMQqIwYtAbMOwFQnQLs+iJwVX5UK/WaVSruGhSDMYeNwayFKLUmlZj1oZ93AcMhwmDYbZl0quAmt3E4rzK/Pwsc7NzzM418L0M3/E5c2aRu+fhsT0w05gBTgOt26B1+z1EcY04qRJf7NAZLhOb0q21xPZw1CdK1534OkBr4TT0l7kc7K5f6tkLbYLlFS5uVsja5vrHoOpBrwhT1ke0uHOgqvIOW0XAWrOGgjBP1lnsPyiutiWODH4XeLoH9/UOFukVuIzwJfyV1rg4GG21vA4UzFQDrtY41lRBI5EM4glfstxGMjBm7LjmZgYvNlRcqLgGp5jYao3WBq0gdkTbOwK6MfQSaI/gUtfhiV6NsH8M0tOAC3kT0hkYzICnwQyBgdhRNCrQaDBcmENrqFVcXAWpsR50xsb3suElzU7UvUB3+TlbT4rJU6IwJIwimRBobR3fxHRDpwnEESbsEQcBYRiw0h2QpDFpaqytrsh08cxJGrfdYltoSHsrtJ99mgvnz7Oy0mUY2vUyI/L2NFAH33UxxjC0ERu8OtSqGtc49Ns90jgmCPoEQcQggOGQA+fkdqmzvC0b3yairPGAmgezCwu0Zps0fCCOGA67BHEPkh7HGz6PdXY/ZlsLOHMnnLn3xdx8z+dy/nzM5U5CfDGhkyyVZg4lSmwT1534DgESWKi2mPENT7af2bW6B8AguYqWpdDeelCfATSEVUiLSA+A50HiQV4Q3Aqi5SiczCe1vZMEePp7iRLXCR+HfQuxtDk8YiqMaaDG1S7K0ROBF0SDuRq3ITfERhzYMmODNKya9Rp50VvD3VTLpnONjyZzIUOTIYQ0AfqpIUgNl0fwbD/hQn9EJxjC5RFc9KF7HEbWVz/1QXlABdIhtM9DewmiPszUYKZG51SLzqmTtFsLzNVq1CtVXNfD9T0qno9X8fB9H0c71snu2nD28YeRBBYJaSoRHeI4pVr18atVtC5CmkWEQUAUhSRhgEkGRO0VLp+/RBwlpDbsWGxScmNouhr/pptw4xhtUrLeCoNLF1l5ZonOCuQjROiuQblYLTw49jf0DCQGdAy5MQRRn6jdZxiJo8tBNvVa2QbprQCzdk7jAI0atOarzNbAyQLiICMaLDMIlsi6SyzUMm4Cnt3F9s4A1VnwF1o0Wrcx27qTQRyRMMTxl0mvKUZFiRJroebhltvhqU/sd0uuD3af+LaA5c2LnF1+Cg9NVfubF9wLjIABxH24CMx6sHgMng0R0lqFuDBfKNRnHuM4MjDW+E46vJXjT4l9xoEkvoUdrrYRAwqaawyFohRjMBhSNMSGzBiUTtE6x9XGur8ptE1kpDGY3NryGjGJEKsJjXE0Q9dlBHTSlHZiuDBIeXaQc7kfi5FaL4JOCL0BdFPo1yCesWRaQ6Ihy2DYkYHiwt8CT8OSJ/fCApxscX7xFOdnZmFuHneuxcyxFgutFgsnT3Ks6tGoVXH1tQ8Mn3n4IdHqZjFxnFhtt2G2MWMjNWiiOGYYBJI8Io6ouuD74KYxhAPSMCZOxUREueC5PhWtqfuuOPnFEToeUdHQ9CGpQmQgTcU5LR+Nh7ohZo0GdzS8sZdMa4BvY4dlq2ZuEfGwS9BZwhn6RINlhsNlsqhHRsZuv9EyYDmA7PwyNJagdoEsruLVajSbVer45SJjiW3j7hfb4a8LrTPwghfC3ILsW3l0bdlbW9BagCiD88/BygEJ57kT7D7xPYG8zNobF4mBGENoNsnwUGFvRtXClrePDGhzMN+Edh2GDcaex0XY0QJF1rbi2KRmd73wpCVKXGcEHDwbX8k0pjHGFZaKFn7pSOYvhVl1STfGCK/KU1AZWhsynVJxXVxLbgsjAs9SaF9rPA2OjeSQ4DJMUy6ncGGY8vQw42I3hpUEOhFcCmElhHYI/RDikagvI+shpI2MimYIaRe6F4GPcUWwnksNuHQaWIDZU6SnbqNz6hY6d9zBULtQreB6Psa/duJ77vHPgDHEcUqaRqtjTLiwgElaaK3pBQFBr0cSR5g0Zf5YnfqxOtrEmCgmDWE0klP9Cui6QaeypXFEGgfEgxDiGFdDxYZVDlPIdmvV3k5wbMQyHB+0A0kTqILng+9qTGiIV6B3AJJeVBBTOMcmvHAQEuwTkyUDkmCZZOgQDJeJomUYDiCWcrsR5N9FdC0Z0DOQPAf+7DJObYnZ2UUazRrVqkOj4RDuQvrnEocAVWQ2VmSYvdrvPgszC7A4L323vQydNtx1F3zOZ7fAH6CrQ+YWoLXYwPEdOu0ejzZgsAzd5+D5d8GLn1+jdQJiZ8hyV6I7PPo0ODVwCud+II5gFEjbVA1qVZlwg0ySZbmfVSWisjG10+J+YlvGsfe5tGuSuwK7T3zriINYBnQ3L7qZbVKtCsO9UicU5NeDgQdB1fJYb+I4jE0XprOw3chqjhKHGi+znx/Z11ZMwNFUHUtXDRQGvUVsB4216c1tpAbMaj6Kmq+peT4zVYdmzcWd0BBrY9AaPFfjOTBKEoZRyPIw5h/6I84NUiJrnks/gSCF/hA6fRgM7daHfhuCZWSBehmJjlwEjRrYfetFqBwg6UI86C1A7xycOwXnbuL86dOcv/02ji+eolatwKv/6TWJLo0iDEZiDsex8F5jGAQ9li0LLqI5SFzjnJmmC9RFrhNmFqmNhhGblKefOQ++xsQSISLo9VlpB/R74oyYpWx5Iu9p8F0ksoVrozf4SLANt9DGa/wU/NRq+gGjDaEPqavt766JUkNyQMbWmiW92pdsb1UH5howPz+L9h0MCXEWQRaRZeJEGHXklbcbZs3F5HU1bq8D8XBAt7skUSSyLvHgHMnw/C5c7TrDozQJ3ALueLX0wdkmzDV8PN+j5teIyciyjGgY0e3ERIGk4vZtX/VrPlXPo1ar0fCb1GqzZENYOr/M8rMd5ucaVGfnmD+1wMKiptqU/jUIEqrNHvML0DoBzgt97rprnjvvmWd+wSFxOrSGQ1ovHfC8bixs2rEhsmLIsowsTiADv+bh+D5xHJNlxn5Kr3Yc2QpkmWwmgyyxfd+DP/8NYI+6994Q3+KpvQrx3QgngXs9OMuV9lIKsXuaNLPNKTRBW4xnWBBfDVTlvZhhv0/a6U5+TkZ2KLW7JQ4gHOAlwC0cIOLramquxiCe9ONHRzS2q05txsbYBRvhTFPxFHMNWJjxWWh4eNqGQEvz1Xxvrg+ep7jUTuj1B1xod3nkXBsuxxC4MPAhjGGQitph0INkIJreYQdG54BzCIltI7ZatyEqhxFXNyBJENXEEow+Bedasp29j8u33w61BnBtxDeOIzEDMSkmzUitqQO9jDgMrX3viNjG6XVdSOOanKz1ajK7YtjKbFSGZ5/t0+n9A2kKaSwvm03ZmnUYdFwhua4Wwjtb92nWazSqVZr1Kn7Vx6/6aN8l1YYUQxob0tSgI4OOhMCnaUpkYnydEpJjTAqpwaQS/3e/oQDP9/H9GN+H2YasCs425pmfa5JkDkEcQzzEmCGQEEWw3JMMb7uJIkub54DJAga9JWoMcKgSDZ8U58/Dhp35fG4MD1Qd8sJs8ZDj9W98NTgONd+n2qjiOA6+45ORCVmMY7IowsQZ2rJJ39P4jo/jO/iOQ5b5xLFD0BnyzOPLnH98CSeTvnvm9nnuuvcEOBndbkCcdWg22sy1OrTOzNM6dYJb7prnlrvmacz7ZAyFJzkOGQ7GKeyAgGEGmYN2HBycVXYbxQFxEoPsXYOCCGdka5YpY0uElzsP8ul37I1s9865bTrT5xbhIsT3dCJcc4W1Gv0TiEJ5sm9PRhcbchUUURwy5L0WQTeyjmxwpXZ3svLDOMiUODJ4GfCSWVhswU1P7K6TzbWi6hrqniHR4KTynBWPkdaSzMLXRTILVjWVWkPDh9mqS9PXNPxc/K1iQ6oNejU5RUoyMvS9zkEXAAAgAElEQVTaF3n23Dkee6YNzw4kIcNoBqKGLEMPh7J+t3IWGIEykPeR6fVZxpEtC61vYb+0HcORjFUSvNSDS09DtXLNsovDgQ3jJtnTCrnFaUqqY7GLjo2kF9YS43gQ9Lm0DIPekNBO7hszmgaa1Ij9dJEhOk03IL2+ZFt2XahWoVL38X2fqu/hakk34mLwXJ+qq/G1xsVAGpNGKSYyRCYlNjGDICaKEghjdJgSh2C/0tWQ2R/dcSC76uB9fdDU0PRlJcEHar5Ds+pQ8x18MpJsiBlGDAcBw2GPQZQRdHecwG4NcuQdFyJE3B9Akg2Ik6dx4hpZVGPQ6+3a9a4rJkOJWkj4QlkVSq+VtGqbNOUGIL0AWSMDJyN2MjIyHGCQOaJZzSDLYtCZVboKKU4ciEhkbpFBFMjwt/x0m4c/9QSPfvxxSDKcAJbOz7N8fhHHgc5zA5Y7Xc4/1yPIoEGPYdVhiZg4CPAL2uowtlUCYhwcA85QDgrptU5PvoQPjLN4lfQ6OOMRNZP7kqaOya+xZZb3sHPvPvEdIZ3aYct2ujVWY6EzD9wMtMIx31wGnkNMXFqMie8kP+0iZrtXxQn72Wc1pmQaIiNWEbFh0qmt1PIeSXw+cCdwHHgPB4NEboYXA/d78IrPgdlZ+Ion4G373ShgxjHM+TmjFGItNl+p0asKSUeDq8a2kS4GxxKpYxVNq+HS8AwVJR5aI5NBAkY7pAaydEicDDh39jH+7m8+xuVzFyVGYTwD+mZ5m/bbMLgEo0eADwMJ5DXkob+wTqt7yGhjbS6uCU9D/vQWZuIbI1yJVum3MTa6gkbMHyYGJA3WcRDa7YiVdkQUiePKzBycPHmCxuwxEpOSGkOcRkRxSM8d0FtJ1/gozJyoMn9sjuZsnUazie/7+L6L7/rUKx46T0njISaOSKMYE0WkcUS4MiCOI0ZxShRDEAu5bW9mhzhx3YNil66ARs1htubgZ/IKaDgODb9K1XEgDsiCAcNOj14npheIr2TA3mVQy7EpkyMYRhFZNyJodugsH7hocVuC34DYQd6/HhxvyAoCDiQRtLuWwG4XCTeUGeJDS2vTognhzUjizGpRxdpAez6+X8N3JJoNmSHLMuIsI+rEdNtDzj3U4ZH/D5hIaf/xv+pwstFB+9Driaa1eRc0zkByIqOTLMPFZbI2wmMnHD0Ls9zM2r/XMtH8ZhPENnOEpBu7RyzV1mp3s9U/9sP6Fzs+XHp892Q5jd0nvn2kA047gG2CIohCHXndgJgf9BFCuzJRXZexMrkwFSou4zBOK7kGFcQ+YgbJylakIrZCXpN1rTBrYGJ/6TZ7ZNAEPg/Rnp5C+mQX+BCyGH4Q4QB3AGECD31CuvDZfW5TAUWKtkGXXCWqScfG13W1EN+Klv+tIhJljHzXEtIstjFyjcmIE0OWju1Xh8MR0WhAv98m6D0L/fMQ1iBbAHxrQDyAGSNeFNmAq2eVL3LxTnhu7ANiG0JRaxupwZoaaDtr0NZY2tUa7RZOhGJQ4kcGtGF2YY75VovZYwurPDMIV+j02lxu968g5v1uhHI9YmMIo1Q0vK7Gd11qVRdMyqi3QhR2SUNxnksjsRwpxugi8uNhgcvYZM7Hmi5arZljl5TjWOPgSJg6R36Ppg/RPnSPOJJ2HRAF+bo41RKloAGiCIIQdAVO3waLp+t0g5D+QFZ6KkgKcM8X4gtw+SqRoVwPmk3xS02KkKNVG/d7Haeou++QJ7kbwsU9dJraTSz1nlv9f5X0ZoYsjhkOLVt0wPcdqrUqvl8T84EsI8kS4mHMMMiIOrAUsG6HuTRgzZL6yuN2lX1WFCj4dpKCdZx2JkynGHM3f9X8wn7acwpyvBmyDPJJ5aKWxDiHy7lthW2bBsTIq2YOedWEiJb3AmJ9VxDZHNH8aoTDFmVDxhy2br/nICR3AWEzDeThaNg2GuQHL9IXY/cVjSnaXsbmveExj9jF3oyQ3VOIpreOPCAvsN+XgEfs50UOhpbqNuA+ZHH+aeCTPWnfn+9rqyaQZ5CmqxH/tALQYi+qwXOh5mhc69imjEZLGjQck5GMMtIYhlo0vllqMCbHsSNXNBwRDoYMh11G0SVILkJWl/XS1LoPzwFzFahU4NmtULJi/X8ylMv1R5aI57NfF5MD33dFA+t6OK6La8mu67q4rmvJr5wbRinNlYja7AILCydozh3DcV2063K56zFKI0lEMY0Yekt9erq/5tYdBxpVsRMebDHqgrp6kX2FS/HS9omJVxf9nEJvZd/sWQzJIJLl3lqGV3OYzTyyLCaIwRuMndDWVbzsIhzsayo5uMS3VoeXvvw0nufhAFEyIuj30RWHm287TetMi/ZKl063T9gNCNsrVLwqM/U5kiQhSpbpdq0ZDDau9BRuWoSFlhDZfgTVGagsyPManBwRBTCyq7r33AF333YH2tF0+iP6/WcYHAJl1nIwzshXOIDFMSRDGCZjQun7GbXGgGptsFouiWEYiflQPLR9pclVQ80C8CSsNCCYE81rPJGxNh/bLBQPzxhFVIbJF+N6L0ln6njM2lV1bRfk9nCZdfeJb2F8u02yOJo4rTBjaHPlIBLb4zb3BExcqvgNPCBeQOwiFmzhOuN4vIV9b5HlqaikSKdZpDDerolfiUOHz0c0vMcRk5uim0w+GMftdh+iCb5st4sI2Vzi+ppCVIFFhKDfgrQ7BR4F/ga4dB3bclWYFEkpAS5q1YjX04aqjQbQcIX0am1QxkhOYiOObmmR6habGM3IiFC1wR+y1BBHGVk0IEu6kLZtKDIDSQ1MBXQDZhrMGJc7PDibbMUsKmftctD1R6U21vb6Vd+SWwfturi+j2tZbqHJ1Whc36da9UHHDELZK05xUPF96vU6/XBF3Ao3Yk7rhGfMgN42mZbPwV55bmqf4vf1jQQC9K29sbafDuJtPozBqyXgg5850KzRMBm1IML3xG55r5W/hd6m0LQd1IxtJ09Dq7XIzEyFiueTJBHdfhvPc7j5tkXmT8/Tmq/TD+ZYfq7Ns1lCxamwMDdHFBnmZto062LMk2gYTtgDKyQ1e2seji3M4NX7VEYwM+8xt7iI42i67Q5Rf0ASGkjg5jOLnFxsYTJAh8wtHA7i25kIDrBKfI0Q2Wg4dgJzfGjGULUEMkO04MNAJm1ZEW+7scULZ0APsg6S5bbYZ+QjhXHugslOP7nkM0GW13Co6XMmiW82UabGniocd5/4TurAt4GU1Qhjohli4/ueNMEttkms2kQXD0w0cUIxoNdZjS+5SnhDRAtcsPASNzQ+nzFxtJmsV/sg63zCmBjPIsTzXsQitIcQ4YIUdxm7S+0mbrLtLcx8LjJ26rzMASO9gLgqGLGW1Rpls074WmLx+gZcA4qcyUgPNjNFEfCMIgaa1gZXGzynYut28HOJCCHGv0NRhyUa4mOQDySuFhUaGI6hmedqxHc3TBw0cEzSQF4jbr553tr3GmIy0tQQRSNcP8OP09WoDlEUY6yD26lTxzlx6iS9IOCZZ55Fux1mVwLmFvq0WseZP95ipTtgZWVlYwPRhi8qox06l5yuwZMHVC0p8YTl7a1xcMSPiGoNqjUH3xeHnFXH9Uymb46ursYurWU1vGaEMwC6rMYs3av2zgPzVTEdOMhObaMRXFpuY7I55o85Y66TRURRn6SvqTsOzfkZSCI6nsbDkXHWgYbnMVMfUSxIRHpM8hfmYH4Oqr6Hl2lm6jPUZ2IWFk/QOt0iMxlJvwMO1Gc0TubhaE3QDwnDhAtLbZYPialDezJESGb1cHYSFg8lIgsGsCS4Zu1tcUQzHA5Za3J6ArgdePIqF25AvQrzmTh6Vq19r+9I8IZOIo5n/Wmymklb1hDZQnlYDKd6qvyk3cSkNnmPx43dJ76bmTkUoc6mPDoL2JwSq7LZJL3FqvK2IL7rjjnFj1443BUO1g42FyTjsBHFVpLeGxp3I93wZsakd1LpP+nkPkQekMkJlotk+zvlip2ZZ7fi3F4oAQS6yVrNcEGOC2Jdm7hGz5aLkC45a/dN9/+bELOLYlFlOFH3ZQ7m4oRQ13TVHhWthLjqHF8J+VVmgvAiMla6GCrsEe3Y0GcKR3m4vo8y4OMyShUqNWBiyIfCQGJtozYMROtrGtSAmVQze9VW20C0O9H4VlpwchEq1auX3QCnb76VNE3phSFpb4UoHRKGMW6c4ropcRzT70l/W4W+zMzCMfpBwPJzACGX2yHzvZDcgOv79Hs9VlY2pv7OsXmyKBS10uDa3KdmgdOn4MmDYmw+BR/wnCIdpyNaXhz8WobvOfh+huNkOPahkleag1PY+EroUvzBAD+IcWJgsHda2Dlk3Gm1oLPM5i/HfUaYwNLSMp6GZqMKXqFjSkiiPqMwY35ugYWFOZKwwVOOs/re99DUK1Wa9RE6Ec2lZ2VdqwjpXTyhqdoJZbNaoTo3x8nFeRYXF4iSiEvnIPEMda9OxaljMugGIe2VPs9cWCY+JO/3XqE5ydZ8SDKIQrNq7XNSLBF1wHUsKR6wVgk5L8evRnxvPSH267MZzDuw4ENDrHwIMlnlrMXw6FCyugNjIjvZrkkC7LC+tncjbfAev8x2n/hOBtmdtJUtzOUmHcmm2OoJxr9NcUohxwI1xHrB5kyiwtpgDKvjwWDiWlXGzKVgywUJLtow/aNcDXuVWa7EnuIFjInnpJYX1mp4p1cSJsfKMBHynCCmpENbMLGfrmvt1RP5PM6YqBb1TJPt4ljdtu0i8LeMA+K7CJmYLL+XmuVdg8lkIwMUrnZtKDKzmoUNk62mL9ZaybK9EU1nZmJW0xxrSXzgu5rUJkNI04QszsiNBmqSMoiENREZ+glc7HO20yXLO1swcyumQ9caikzDQkO2HYQzc6tVTByTBAG9YEQYRQxDcF2D54uW1xRjqh3H6s0ZZo8dw+DSOj0QObk1qtUqqTH0goCVIKDX23itV7saqj5G5+SDid7lw1Qw5g3hA77vUyE+sMNkkmVjBZTj4DtQ9R28mkO16eMzxF+djvpUq3NkmS9lfZ+a49CYz5glYMgAfzCxFLyLqAANDbU5yZaVOQfXvhegtwSfXAlZWnqK8xeWOdas4nkZzRnAJJhRSNQNCTttnnziAo/+XZtGHU4vJtTrTTwcjs14mH5CZ6Kbes54AeW550IhxVVwKhB22yRhABqioE8SQpcQnWV0uyO6bVhu72FSrD1AWNjLTnOlaVJZwDo6pQUhGk6U7SEOUlcZ/O4GXjwrCSwW52dpOhkNIppkNIBhDZabcH4WZpfgwwFCumzGttXrTWt6Ya0OYZIIb2QWsYfYG+JbMP8qY4/B4ia9if8ngvGeRDRaRbSxwvw2ZK2tbwvR1t29AK061J8Zl4uYyJlRaHkjxmS3EPwMcAwZUaZnHJsJvvCeq9qtTRnx4ZDhC2+BJISLl8fEsSC5BcFcD9PJhnoJuAVzLt50iQzM9brsXrQVe4WquNAOW00GqZDj4jwPqM2CNwdnPwV/3JPZ9WVkDJvUEnc5OA52m8GkMSaLAI2LRpscR6tVfSrk5KRgyS2riYk1qcmIkxSDITcpSoPr5mCUlDaa0SgmHsWkqSu2vHoO6EM+tt8kGEIggRGfuupyTg0xhqsxjjEziakZb6UBDQ8GK7K7oaE5C6ea+K05Kjsgvn6jRqohSg2Xe6FoemIYuRITWVvSqyriQ+h40FxYYP74CeqNeVJTZRjGRLHY+KYmpbPSpd1ub5xcqOKifVeiR1Q1/SBYZVkLp+v4rubiueCq7K7qQr3eYI74QJnfKCaUSnEi72aHMZn1HJpNn7l5TdVpUnMyyKQ/aDziRDS+1Rpov8q8UyNpzDPMlqh2OiCZi3eN7Cug5cH8CWjMgt+EbBkGBzxNcT6C88/A+WdC/ErIrTfDbWc8Rt2EvifpoCsePP4YfOxROKFBP7/N4mnwHE1rYYEoew7aMiooDV5FnDwd7XDhguE5++5VQPdSSNQ/S70BQVvkkyQQhiPOPbV/ctgRCo3tJNFlnU9Ym+7XmTivyF7ZRjS9HdagjihUagi3euW98LxTPre3znBb6wy1JMAfdqhmAXNZROZrus0aS55DM16i9xx8untlvavYyNEtse0sjNYTruvLbPeJb5W1DGHVnZu1pDdj9b1UxO89gQi/0GppxpOGgpQUVRRkpeCghYXCFStA09qJYjZURBuf1EjX2TzS0QxjZr6NcG0lDhAse/RCqA/H5gspa3/O6ezVRf8ruO6qNnhShWszYyUT+zyA+pjYJsjyUBgKAWco5HeVHF+U84u0CoUNb8iYqwzt94NOegGyJCGJhAZkGlLXxXUdjGPDcGlINbg5aEeTU+iGIcMAKdokYsZgciAlM1pCZhkXfIM7U2P2RIuTw7u4hIHeE/bqhdqhmPlu1ezAtdsW4viOBuNJtYckxsh6sHKBeDQgdvzNz9+0GS5upcZcq8VNRpPGKWka47qSUKIIYQaASXE1LJ46he9XcV24+bRLODJEUUoYJQyikCAMGYSb0LIkZRQGpC5UXS0is8RXEosoGnMw6LCp5jdNIYrSA7ciX/QvgV5Nneo4Dp4vGa9AE8cG7WQ4jsNia5FW606yOKPTG5AxIMoCsmEGTpXZZoP5Fix2YbDUoTfYHeJbAeYt6fUbDr0g41JHnJ4Oso3vNOIRPPY4PHs24dYzcLolDmrMwMiS166BZ87BKGzjVWTSFXYNUSDjYQ6EAbSXIfSSVdILcmxpCZpzMDMD7Ta0V+C5jcjYYcGkv9R6P/i0oq4ISzJ5Psjwt4HdbPEe6SBK4dbjUM1i5nkSf8HBz4boOCAbRrSjmMyTBTy/6dNK4B7g09faGQub4H3gUbtPfAsFR8FYG4zJb/GCKDSxtvMeQ/jkSQ0LdQiN2Ks4oyvJyCqHzsYEpM44fPCaAacg18XnpJnFpCdd8U7czA9FI+vWLcaq5ZL4HjoUS2XeHNSsxrUIgVdMuAqCO41JElx8nybAif2TwOpA49nr1qzGNwyhd3lMkD3Gmt8eshr1DBI6bch4RWPIgTbtWxdxHBOPhuQYlNa4mcakDkZrMtfFcTSu65G7NlsYrOGamgxMjDYytTVxitHiaJRRAQf0TJWZkyc5o+5kJUuJH1liPMoXv5aLzHpqbL5QXGTfGeukr4qi01TsKakRQ8zO8tbrWA+uj9Yus/M+fn1e0hVjxFzEdfEdF9f1cV1t9xtqVRfXd3G1T7PZIookHm93JSBeukAYtknCTZapDBAOyKo+qfZwqi7ZTLrqlIh2aBxr4vkxKxc2fuNJ/NbkQA+RjjNmCY7v4HtVPN8BYuIMnCzD9x3m589w3/NfzCCIyR5/gt4AhvFQeEatRrXRYD52iM44dLIMBpJRbYtR366AAhaqMNuExmyD2WaNjIxep8PTdqm66KEOh8fibmjg0WdAJ8C8jb5ku2IMXAiE3M7MQXPOEHXHpBckhFenzboz/gGwfB7CGej24bkDbf+1RWRT2yScqc9pB7HJcsX+DR7G0cTnExnMPQ53zQPZRXRsII5JBhlBgE1kMcABWrEQ31OZKGiuCfs0QOxNyuJJ1WtBLGFMQifitRVE2AMadaj7EEaQjKSKyRWdyd/f88TzsO5BxQpvxNQrbZLQVv9/9t6tuY1kz/b7MbMqUTcABBoSJW2p1Ze92z3d44gT4fCcsSN8ifCDH/wR/OJv4PC38sN584ufHOE4EXMcMWdm9uwzvWef7lbfRJGCBAHEpVCFzMqkH7KKKFLUrSVKVItLARHEhSgAVVkr/7n+a7El4O2d6eysKeTZI0lj87Ph149qV3in+PZbr8FNEhjd8cs8vQD2v/Wks30cNsT2FMFtoU2Q2yu/Z4/lsO7ODRaQxLBePD3RbUh34xF8H09+Lds65ftGegGOncFW+uSQP64ELgiwQVDrTyGwGjYWIbZLMA5AF1RFjtisoFziXIEVJU6ACAZI2UUEHYIgxpETJgFx2kGrtnClmd3u1Nd32bbDnnd2bEyjUs5fzz+nzNnoXtsc92QM+fWt/oEIvJyhpdoQAm9nJjzBDQNV66bdieOFCBxBEKFUiggsCE1RlhzjKMv1i5lSqb1tVxASRIoKX00WQQiBIhCACEE92/mhwvP/y7jPHlMXx6xFSq/ZlTJCxRlK+aV0bO5DA9A+GtZ6zfLg2ggZW+RCU1hvbixjRa5ilMzxVv6vh34Eg6Gk1+sRxTEqitDGgPQlTFVb3IHX+3LJZQ9tHAPfjf0CTpJ6i8IGTX95Ooc94wsEzWpvg8VziNJ85S/v0cfxfDzPIeusm8J5jzv7e/iMv9XCf8aPfJ/9BOO93MsgSqCAvJZS2BykzVEr6OfQL1+D+L4jvHni2wRD1DF3NAN3h+35ptF0KP+7XoMNQah6YlL6gupZGVpbXZBEfrlkmfgDBc5w0bYOok51OYVmGxqS3jS+pWxPDA1RblwqlvX7an5e4b3D3xvoG/gK+OK/hht9+KgHVPDwh6fJb1NxbSsazhJh0/pZnbm/kVFArQk+M3A3Nn5Ns9r9+tKmZJeRPLwKHMe4Y68zdcfHCAe28ro9YSqE2OCqJbY6ojrOfSCvc5h8iZlPccv6YpcYkeN2HKjrBJ1dku51ut3rGGGw5CCNj3xD+U4gkYBphrkOfslGsRWMnCWmKfSGEKSwWfnlp+NWlVi6808eF1C5CEVA5XwTn6s/MCcg8HFtIAKOa220q3cyEQU4Aa4JtKjqepnzMglblS/uvtrAjqtIA3CBRIva+zFIcEFA5RwVlf84n0N8LZfXa7YCCueIJYQqJgxjQpWiIomUOdJqcCtMuSJfTJiMfybrDdjbG5CmEMaWvIixEqyUSBYUhaYoLQter/ksjCFKU9IsRoURSG+3FkU9humCKJPEUcg8L1/ZW/kyoAJ+mcHN8vzDJgem69NGTC/TMPibIbwNzlZ82zLRZ+l928W8syeOIXDw4pf9C/DJBD75Bm5FMJLbEVMqYAVCg54Aj09nWLwvuDg7szZ5bITLZxlEszRYm4VHHb/ka91pTWMbzZJ0FAp2E8WqW7KePy0tPtHcNcar5xFf6u3U9f1N1Sar72sqLY2sweA1wEtep5BzhXeIb/EuC7eLWnfbg2QExNt9rt2vdtbOrC1z4MzvzaX93OYxZ8lwc70hvE2V932XpZ2FsxXOVljnD5nj2onAHdfODccFTk/YbB6S5/cx5RSHJ1ZmcUTxZEI1nWCPJlT6CCMcxwLoxATxiMHoDsNrdxFpB9IARw6h8heVwk5Sf1lNIEXTTVFxbrZYGMOg6xvl5htvi2ZSoPAPb5wNqvpP7nBhYmuFr+Qa506C5BzUEmRRl3/rqpkDTozjAgICnBA4YUBUgMZVJWX5EvTA4Z3yq+LER9gSYLTD6YoKjalM3V3nB8KzApJjfMX3MqPCx6tKKZEq9hcpCaVFskIaQBtWswmHB/fY4w63B38AmWLog4RSl+S2wNaRWra1M7SjkF8FcazqRjs/SlgLOEkcZ/T6kjCVKAVzU7J5T5f0C2A8hdUzdsc5/pScsK09vYcc//XQvOF2Y1tb2nDe8dWsOjUptG1C3Kufv8JXeJ4xbq2B74A//FT3dWbQi0GrlpKibpyT5ct3TlwmvHniW+I/naT+va07ca3rCSdJIqmBfgijDvTXsD+F0p1PfKfU5CRJGeyNCKN9RiPD/Hv4ceVJBOAJbdvGDE5Xf2vT8ROtcXsAEa3nJvXlbBjGFd5bFHgCzP9VxxLHsF/AD2wrvD22tmdt4tscMOGZn2fR9to97zFNw9oT4PvXfD+XGZtyw6b0MYlC7LAjgtqP1yEwLPMHPHr4RyYPv2F58EdYHm4lUEv8AH3uybGg4j6P5X0e9/+O4M4u6ce/Y8kAurucRIxUQ7w4bclWDNVczhn5zTHMAx9v7PqgKlC7+AO/Icz2dI76BUFpgXCwsd4Rw1ddfRCIlyA7dhAohNfvOIFzsHYCV0GgDa7SBK4kqNZU6wlm9nJmW8dzeLgsWyfXJjPzNDofdfl8eIOvb33K/g/3+Mf797aGOu6yt2BZrGsxA2FP3B0iMqTUyBDG0wmHkwXXH08orEVFCovFWcNkfMDB+ICyMKRSs9eTxDJlsTKstMbUvZVaexvEF6EDDERIaGG1yLFW4ixYJEqFjEYpk/mYJ7MF88ZK6j3Fvx4+//4Ovum9OfUe8uat4i41xpyu+GZsrX2eVQ1uGtmaxrF2dVjVz1f4w/k5c+B7eKldnIMqQdcEWMXUOl+/oJYOYLDivet3upiKb2P71cgaYCt7aMsfajaRSE98hx3IZEQYlZj1+Tv5cf0ShB2StEsY7pF1pwzHa5Kzs98l26CKNntpE/DzBo6m+a6ZbjYkfspW9tDyzrzC+4OGcN6n5d9bbEMgYNtM1njqtmOM24S3LXtoV4Ab6cKa00lw7W1ogisevLm3dilhjabYlAghkEGAcscgJBWWQGiK/CGPH/6Z1c//AL882k5Az/rHPfMFgClU7ohNt2Cn/ynHo99DdwBVCiYCW8Ck3Viww3ON6zYhiPoMI8F3tS7B5FBu3t76fVWXcytfaSWgLpfX0oYA74ncMvT1sgfhZcfCP1C4ih23gTJ/tTHrZfx6VcC169f56r/4Kzo47t+/xz61NtNd/gFSisbOrLkBkAolI6TMQDryyZTJxJPeOOvTG/SIswhtNflqwWwyRuJDGHqZIs76RD1NuFiRFwaZhmjnWB+8eCIwEJAqhbRQFgXa+uI7QqGQqBhWecmTF5tRv9c4ZnvqbfaiOX7M/GDwjEa+EzRuDw3xbWsOmtsLth9g49LYeGM+h/iu8bx7gA+ykAUIWReCovqYiSAeQl+x5UbvCS4msrjt1wun49ja55/a63fUh1EE/bBP4hKi6JBwfVof2d7gELBOsrYhjgQTQmnWT9cjHH7aAv4IGtbXDZ4UP0+y4PDt9SXsDKEb1cL68jnPec/RLPxeVl3em0DDq36zzqIAACAASURBVJoktTaBPXvcNiT5vAQ3eJoMtzW+a07LHjhz34LfnqzhXBQbjvINspalhgQEUiBEjhQFk8c/sjr4Czx5tF3aa0rtjZzgZSDgZrwhurHk4eeOGZUneoWDiYFHITzqw/0KXw1e40soD/CV4A7eTfwudPoQdoEBHJcwfwibKX7veXsH/6qqMLbiaL1mul7i1g6CiiSKSNIYoSICoahEhCAAobzLA7JeqBI4DbrUiLwkKN98BXZ5OGOe/YD7/C6JKBh1YLHxn6y4hOW53cgvM/rTlCTKYlQkkAqkbJXPRIwKQUQKudJYkTOZzvjmL39iMBqxt7eHikMEklFvSL5asVjMSHsjRsMevcInvyGcbwhyz2MwdZpkCqNByijrkRcl44XBWQsqRFMwmc0otc9j+VDQbgt6DX+U9xMNy2+KhI0vb4N2Vbep4qrW9aYpqrE5az7Al9x/DoA9oJRgYx+V7CQo6yu+TQW4NkJ5r2yHLqa5rd1YdtZNwZy5AHt9GEWCodwjNCGdzhiFo8PTxLepvhkLayNwoosNE9Zm//lZEk3CRVRvx5yXc2ZYeWuZmxIelfDY/TaJYVMDaxfpf8tovAthO09rSGx7Pwrx4895koezcoe2xve8saUh0Zc+be1NojDM84IA54sSQiAFCI4IxIzp5Ad48IP/UJodr3Ee2/DSXk1KwI0Y9vZy/vAHy1Hs+HmZs7/aYJ+EcBTBkwDGMcwLWK5hOoTJAD/DvQk7I0hvQKcHqg8ig+oYNj/gTwNvl8nlxqGrium64MniCFyJcxrbyxBVhooSXNjBqQqIIHCoSqBUx69WONAVuLJCrDXBunphRs+vweLhIbZ4SCTWDHqCG4/d9uR7yXD3888Bi9Yaa/0nsa34+t8tgAoRcUhMhlRzDDBbaPbHY27uTbBWMxqNkAgGvQGmKFgtfKTwrb0ehQZtS6zV5NpQls//1O/24Osv7qKEIpQKUxjKua4t00Bby+P3hFS8KTSONs1Y+irBqr8J1PZhpxrWzhLfki3Bbchxm2udTXl7hRP8AXAX0JHX9xZz7yISAVEtqQglRI1uuHmN92BidjF2Zm00H3LbzaFpdrPAEsaH8KjreJyU7BpJFCmuU3Ibb6/RYBe4hZ+FWCyH0znfHoz5/mDOH1cvjKD22/ArBJXRBJjUVQy2S9iXsKDxq3EiIfkA0ZZJnW1Ma38mzyK0Dc6uzjeEuCHSjfzhgyG9APnPLI6+RTiNBDpSoQJJ5RZYt6Q8euI/mLMHUyMpehlIuHMXbl6Hu9cEwxszqtTxeCOYrCV6L2WTpyxzweNFxXxZkS+PWU6BX4CfgSCvvXiVj5QiArsLZQrZXZ/+RoFXgr8drFzFuix58PCAg3v3Tn0euwqSTpPeJqhcgBOKJOkSJT0GWcKtYUaHiqAqWU4mLJ5c0KRWQ7V+DNWaJHJ0Y7CB7wu8bPBVXd9ARt3cZq2tya/F1reVWhArhY0lYX/A6MYe9uGYyQwODyzGfMeTwYTBYMRgMGAwustgdIvBtQGD0YDZbOZT33SOXQCrrV/4ubAgQ4moNcYijpBKUpaWeW5fUGx5nw0Pn42mNtUMBR9cSGqb6TcEtn17U2VtB1c0Fd/mpKbZLlQ1yW4vGb51hC8JTPB/a2YgqscgGccUWEqgRG81w+9J1exiie9Zq42G9DYdh7Vq/Zccbnfh0+EapCSMQm4NS+ZTrzNp3NBGeOJ7sw8Ox/50yv/z7Zz/8Cs2rdFutnvtGqOJiG1GRaPRNPVjuvVtS/wO8Vsivx862gT4bBNb27LsWZXdphLcSMLPOj68RytBbw75jxzPh1i9weLQKiYIO1SmAF3A7Mh/KI2oryFL7VTF85BBMBCksSTtKO58usPNG4I7w4AvR1Pi/oqi2qGwAVrvUukBhZUc6WOKjSPPK46mFQ/vaR7cK/lpWfFkc+TtyhReH5zfBNmHj+7625ZzMI94W1OXdeWYlwUHBw+e+iyOtL94NAOqhtkK3wYEt4Xgejfho14HvVo+N5TydSAs6PUjnKtIlA9eWLtXE4XsBnD0FgZTa3Wd0iaRIsYYjbXbYBuwGGspnCRFEhIT94Z8JByroqDcX7AwPiyhdzDjD59J0myPW3du8dmXnwEFghKtC39uKcHOwZqtxPLJOdtVavyDYgWxQEWKSKUU5eL5pFdEIDO/Fu3Ady/8NnCMP9KaOtkH5+pwnma3vYrS/N4+qbRlpk2hseFfv8LvbQJMrG/OPAAy6zcrSwesrKawmpXVz+wVvqy4WOJ7YrNTo/1FNBXgJTxYwd8L6Fyb8PvuCmstWQbDEq6tt4Fvu8AggW4imC6XjOfmV5Fe2HbbSzzZ7eNJbVL/3phCLNnuSx1OecmfkOPfovThQ0YzyLab0trV3IbEnucXHnF6svS8x38QyB/UOvsKVITs7hLTZZlvoCg9mWyvCgn/UD/Qp/ijvmlV3ngT/7hAXou5kSniKKDT2aEXHxMHjgzHNXfEwAkq4agQuE6B7WzYHAeelmjYaMi7JeOdgkex5adJwf7MMhYxh0GXVV6AM7DZgTiFcBeudyGIYL0iyYEjODf9t54BvUTg8XMxXy1YLBaQ/7rp0r5z7M9X3JivEFyc9birYLF4jBABBBAoH1CgX1LqEAKJeDtjaV4sCKVCqh5ZHNZeWYWf9FqD1n7qK6Sk0BEqhijukWYRpSnYGy+YTbaT2Ly0zAq4nY64/fnXWLsAPacsCga9n8nCAxbWv8wAiOuqylnyGyoIsxQVp6g0Q66AOIPFs9u5dsIIFQ+RqgcyxVrJZrLit9Q90LThfJBjpz1z/SzxPa9y23ZxeAP4CchK3xVxQN0bN4HFZE5RlOSF5bsZ752J8psnvo2et5lxtPfas7832hDgJwc/jeGrccnXIXx5V9CJJMn69PcYJhKk5ptfDH98zU1tGpyaTemyNaWArTz5bJqxxRPjLnUhiPdrNto0sTXGFFfE/Wm0x4/2ilOzHzS9V3Damrr52W5s++CqvGfxg/euCEbwu3DJR+kN9nXBo3kOq5Zp4VN6m7rhTA4h2gW5gOAeBId8pNf0FiuCwnNRpmAGHYpUsIk1Vdd64zEBjhlOPGTHKTqVpCOCWh7gGF3b8Ptuye8fFhxNLffdkp8p+L8nBeU8B73wth9dA58X/C+f5Xwm4LoFHsN8XGvfaj2tDWqDBV6/GWd//0c2z4sXfkk8L1Vph1c8/pUgloKOgE1enRSc7j9yKKVxAnQA+QomL+ih2AFuUjuh6LczDt0/GJPFESqLiLI97MoXWozRtQ5XIqX0/UJKoVTMaK/H3qjHYDRgNLrF5GDC+GBMvtAQ9TiY5NzVEjm4RU/t0YtAqozZLGe1yFlN7kMOez0Y7UXIn0rm5vRq4ehWyuDOXaSKIYyReg5RiRQrcFvyuwNkgx5ROkCqPkqlqLhHmg6RCP7yF8N6Yrns/gcK6Cb1CqvxDXsTnt4HzouYadCjdhsNYWq4sBWNd4ZmYelsLHGDt9Bnewz8pzO3/XkCnX+fs3mPZyMXY2fWEF847eMreSEL+DOQGfi4dGysZEs/gY6E0L/EPuUbjck7wkspYMvdk9alnfbXbhBpSPGS90f32/juX5Hel4Pl9OL2s5osyjP3tVeZrgCjJXyUTtkNoMxz7GbDk+eOB0fADQgq7zslHLiCqCyI57UEqusNGJISwmqD0CFVbr2LTz1suMDAjkUcS4RWCLGDCARIiDsOF1u6xlIK6BgLlWG0LNgXBRwXIDV0KhhV/JefVnwZecLmHsIihXwKpoLKbPNw3sRxZcs16IudNg043Vf4Iux2EhLVQSnBavMYV/kxb7oCBBgHKw3T/Hwf9jYaOWJjsPM2sCkBDIW2GAtaW/Ky8PHEdYRxFsXEceZjmaVEqR5pf0TaH9Lv7TEZTRiMxkwmC/ICVlZiVUphQ9IoRfVjBnsFtz//mnyhmdyfY35aEMcQKUUoS2Kzfc83BAz2PiEe3MKi0EhsLJDDkrSE9eMDcAs6UcpgGCOjHqgYKVOkiomimDQLUSjSKGR9GdvAOrC7K5DaIYyPK+4mvtLfl37+2x97kU67eNjntFSswS7waR+iCMIOJGuI1jB5hg3qe4mzVdS2FOwd430mvXARxLdtEdZug28fi81Y/owv8Sfg9hRmxlCSABFSdDAhrKXDWcujcxj0Dk9ne78sjtieABpTikbekHA6naSRQzRSiPbbeJ/I7xV+HV5krfiejwlvHNfw1ZndDXQfQricct1B38HUelWiww9GzdjuAxMdlfwOmINZQDUFc8iIrdVhbwNDCR8dwyCEOHJsAiiaPwKIHdgJJEEVEIgdX+Ws/LQvEM67IcQhwjkmxQ6dQiPEBlhDVcCOX6qKpKObWLqxX+1xMVhFHSPsCa8AQlGT7p3XO866SuAICGuLsIvCS/cQCupJg0IFiiD0+R4VsNSwyJ9fXT6LDb6H422vlm0slIVlPl8xWyyYzWZIoVAqpNePGVy7zvXRyPvmqpgwypBhnyiOGIwUvUHOYG/KbLJgsiiZrQriXo/JIkfGCpVFyMEdPvlaEUcjxvdn5PmfsNIym2ty7ceIAPikB7e/+JLRna9R2S3mhWVRWLQK6e3F2GhA3BtQFAsfqCG9d5U3pNBYDaGUGO2rSs7k7+ATfQE6kA4TuklIaNaEzpAkvjmznwpGScimv6GfQDKG79b+HaTAp3uwzr2kSJqtROTTAXx6G4zxk62mRhYmMJlDcbb5IuwAsn7Ce9rG3fZ0+2D1H28Gb574tgfojC2DbCKBG4sN2J7lzoy8j4D9deOFKpGiA2GKloKNMBgMR2fqCQpvvbHm14cCNHG1Dfltmtmat9GuBEdsC9uS016Da7iMTj5XuMJbhwK+ZquPrzbgNr5iE6feWjvmNPEN8MdRAcwDw0/uAbZagl0wwB+TJX4oCYBu4a27QgFOBixQFM7HOQRAKAJCdlDUOgTnQ5FxwA6IANIwIE4lqQsIdIUTpd9Qp33OcgWBOCaUjk7o348Lajmvg6rWP4kd77TgAny08mtgNwlwAWRZhNqUXFRmwauQ86qyde+G9w0GTQUcbfy4/ap4JxTNWBZ5iZotmM0WPHm8IOmn9GRIHPUYDPYY7e35GGILUqUgU+JsxGg0Qo8soyJntljQG88YT+aorM+jaYFKe8Q9ySDb42Zvj97gFn/6j99xcLAgn4+Zzcbk9fmuD3z29d9w+4svGYzuQHQLo1csbImRFjUaMkgHZFmPYjWjzCcUqxkai7O+CU9Ih5JgtSCUCmsvQQutEDAc+kZRLCqUdJMOWQdCB6EtSSNHEkG/H7E3TLBDy6C/IcuWuHvw7Ro+DuHjUcI8K5mvHWINYu4P4U8/C/n45pD5dMp0brACZAc6CSQJLI1gXXv9R/0hne4AY2GzWVNM5zB5TxNAGscGuCK+r4GLbW5rR/w2bg45nhw3gsiEc5ukT3bLJMOSYklwIVgVInAMGfMQf9Lss21Oe519oSG8DdlN2coc2qFzzW1h/bhZfXuj6pCc37l7hSt8aPgc+BQ/DBzhK3xz4AZwrfJDQTMktK3kTrSnzveYNZrFGX6VpTnWCiBfwM+PIDjsEQa3UZuUTmeH+NgQUxGEEAaOEH9MB+6YEE0kLB1Z0gk29AlIhOSHPOVBHjHNFZigTk8DpGN1qPnXb8F2apL3CI4ewOwRHM1hsfQ6380xbJy/OOB//5Wf3c1hD3RFJQS95SEb/eYkATucXsV6GVgHi3zNIn9ad7x4H5a5WjgajzkazzgpUegBd/Y+5+6duwwGI6TMKO2KQmsyDbkBqS2qMHVKVkzUk4xkRNQbUVrLKjeMJwu0lawGPfaGQ6QckN36A707U8ZzzXeL8UnewEd7txjd+Zp09AlapehSsigUubGsCstiNafMZ9jFhNV0zGKyz2Zx4LdXQNLv0euNiDJBJEGqi3BpfkUIYJjS6ydYHA5LKL1qBGe8c4UzlGuHKUEIw7AryJI+t4d7DK9ZiH5BfL/mWgpSJIQyJIwMw0QyvBmSdDt88vsRo65kWZasxnPWZjsZFmFIEoaEMiIRCcnuiGQ4wlnHepkzto58OYfNJa/8/ltOh1QMIRkp1gsNj/GD4Zz3rrHsMuDimttgW+FtYNlGWsG2EnyOgn1a35VEKc52wHawUmJCSygc/bqF6CZwqwN2s03F+rU4W+mN8JWdRqnRFHDaScZdts1MTTd/WG//lZzgCh86PgE+24FHx34y+4htmGJW+ZNVE+rR2Pk2k8cOvpJ79jiqWj9/aZ77EPToNsjPYDOCToByG/quRCpDEGriwJKGjpSKBEP3WNNnRY8lu3FItxPwfZ7yYNlhXYQ+ssgIf+DnwEPDX3a8mfsEcHOYT2A6gfEjeDStiThbP/fXgSe+DhdELFcliwezN0Z8m8nIZfFG3eVdNCe11uWs5Patu9y98wdU7KUE2mpyo1kZSC2IwiFjTRQpYhWSZSlxf8TASg7GY8Y/HVACs5Wl1BIVj+jFI3q3vmRwd8XiX344eY/XencY3P2Kj+58SX90i1lhmeeauZastGRRWMazBWYxQ68mrCeH0JBeAAfFbMEgligylGqI77tFMOyQ9BOy3cSvglqHtIYQA2aNdAasozSed4YS1tcMo27C7ZsfI0gQYRfkPqEFYSQyDJECkt0uo5t77O2N+OSaQLHml4NDlus5y9JPnoWCJBJEnQQbdSHK6A5H9Ed7OGNZJnPYbPhhMrn0xPd//d/+D7TWzGYT5vMpg2Gf4aDHTz/9wHf3/pUnP439A6+I7yvjYiq+bdsNceb2Bk3nz5kIYMU28M0Tzi2TdlicNTgBw6xLspqy14FRF0oHa3PafeFVsdt63YazN2S4Ib6NrKG5rUlCbtwgmoC4klfTu13hCr9FHAI/H3vC+wNb0nsPr7XMedp8qe3cs3wJ9jgHmArYH4G7DsshKIHWOY+NBJbgjiHuQL8PQV1PdpqdakHPLelHkIaCh0XIrOjAky4sE3Ch9xsuDfyi+U8TmIdwPYBq7knv4fRiDKSqsoSq4lhrqqp6o70DCZenF+E68PWnn/PPPz5g+o6W6dflhB+++wFrLCpUEMf+dCUFpbaUK4OUK+QcikKQK4lsdXHPJhNmszEyj5GRIi9zFmVBqmL2JwsWRkI2ACJ2GKJ613Ey43CSc7i4z3iaM5mtyLUh15qizClWU471jJ1yBqunzeiyEEa9mNEgJe1lIKNaXvDu0O93yYYZ3X4X68BYi3SGjt2QGkvoQOg1y7XB1PaFgogsGXJ97y7d7ggXXiPsf8qjwzGH3/+CiRJCETK6eZMv/vqv+Pjj63SZY5ZjoqQP4RhcfX4OIepEyDTBhh3oRIShRGLZmA3L6ZTp9BDmb6ud8tcjHfaIrEUqSZSlZFFMFMcMrpXc1harJEfZgSdNK06bHr/0i+CXzZsO7ss9F3hjuJjI4g1bacN5OrfG3aEVWwz++7vNlnxaEWG0p77OOTAWU26wrmRvNyNMQzKxJJVrwvrvJU+92Muhh+/U7te/l2yXVBsiDqftzRrJw7C+fott2kzj/XuBPSlXuMKlxx/xpLBpHm1j/+mHvwZG8MMeHN2A3RRE5den8xzyjU8RCEdw92OIujQB3cflEfPNnDkbb0hL05k2BHbBpVBMochhvGGFd5758xvd9vOxfPgQ5xzVSrNYLN+oD2+jo36RDLmLHxMdfnXV4GUSCt+0eD2FTq9L1OvxYDLjuyfr5650NX62EX5S5ICvP7/Gf/8//09U/+Gf+Pf/9Pev9b5eHoodBMcnRFvzz9/8v/zzNxEdMka37nDr9l2u37lLsTLM1QprNVbnWKspdYG1xmuALUxmcyaTBUYpUBEWibYSrERahS40crDHjT/8dygpUEoxKy3jP33DbL5iefAIyrPfsG+IORYFuKfLejevwd07e4xu7aHSQd3Q9Q67SzK4vtenOxzS7Q+9bb+xhM4QmZLEdhBmDWZNuJzDck2SdkjCAbvd29y89SV7tz+n/6nj46Xl7/7u7/j77w/pJn2ybp/bv/+af/O3f8uXX3zKfP9PjH+GZHdE2P2WKPScP+yEJN0+YdTFiBCnBFhDuV5x9Pgxv9z7M0f7l5/0AoyLca0x91X9AlgVBTLtceeLz+j/rsfBnR7373zHZm79QHsfb7j7MuR3AHwC3Rt3GAwG5Dbnyb+7987VMm8DF6fxbUqmzQS0LX9o7jszu2i8dBu5QCkFxlkcAmwI1kuEUJJOpEgiULZE2O2f/rVoNMKNVrfJ2nCtTRf4Ab/Z7JJtRfisE8Sard/vFa7wIePnt/IqtYt6XtSjWgXLHIqCE1dlE8CyC3YAMvVPM12odr3+8FhvxyfRBZECgZc7GPClkbcHrTVUjqra+M71N/i3S3yB50VkurFpbL92hI+Nv9OF331yh95wRDK8xmj6hOinn/jPPz955rjXZ9ssHFHnzVUF8/WUXL/NMG/L8bkksWRDyfgAVBgSxgqJqeOHJboEawvKIkdrjcQiLRRlWe9rMZaI2UozmeXMZgWruaHKNcxnUNb+umEI1tYC9mcJNVPA+AbLM0iAOFJIJbBYijJnUViK4h06OmxgNp0gw5AkSZBhRJgkhM4gSwebNThfKY+iiF1nyJIuSdInSfpESZck22WQdQhHEf17v0B/j3B3QNTvk4xu0h/cpD+8TjntI6OETtal3x+CyCmN8SVfai4gBRaJWZcs1yWHhw/eG9IL8M1PfwIk0kqsBl1oSmMJpUVGoPWC2eIhm5XdEhXFae/V5yH2j1chqNiX6p4M4MK6aC8RLkbj2zDQtrdTeOa2c0rqon5YKPy4UCIxtjYGdqa2cAGHIAwFqhZCuNqhpOHSvtf41dBnWy3esM3agK2M4axqoylsNw1xSf2YJvkN3nl/7RWu8IHAAI+h+gh0DM5BucQ3xTVHoQRi2OmB6vqOmyDzqWzOevcGV0HlwCkgBbtTV4ADtrTt7RA0VzlwjgqHC3ijs+gFL39+O3sODYDfBfDp7QGffvIxu9du0rvxOwaTx0glCJTju/0ZyzMcrD7PooJ6PK2t0MZHK/713jccPHqzawDPx/OZQcWEyThGSpBWEyuLLiWqsGizqolvQQyEtcWmxILVYDW6XDE+OGR9bwxm/PQLvNSScskO9qkK+g7UIRkhGke+ytFWM8tLNqt3GFxh4OGPc3DQiSJ2+5IozRBIjFljrfNaXyxhKOhnCf1ul35/lyRJEGEHEXZIoiFh1KU7uk1y7Xd0+kM6/T6d4R5ht0/YSZFhgggTkrRPfzjAIXDzNdaCMQ4nrQ+vkbBcr5kv1xz8/Db3r9fH99/+x1rzZSG3WzlDY4DdlpS25d0Z21X1Z+3mjS1VATZfoTOFlto/94r4vibOVn3bOa7nwOCTlkJZf4/O94X6Mm+9FyNqra1BNJrfEsrNtm/uVSX+N9hWew1b4tvO4DibvEzrNsdWChHh98kBfp+8qvj+dpHxAUcRX0o8AjeEquc74o5ztvXKeup6jP8pOiBj2Ikg6HqijPMpFA0jq4QnwropA1e8zW87EMLbvAUBMtGk+ZvrYzmHir0Sen3oZQm9NCbLIuJE0csiRrtdlqNdKpfzyyNNUXini6ZRWNRKkvZ4erSEH3+5x5PZxS/TB/hv8GUaj4tyynwW0ssUxUJhtff/xWq0LZFYhIJYUl/XGFmfNYoZxWJyPul9SXSwPiKW0+eRPj7NGCm9/tiuKArLZFa+lTSv52ID0+mcfn+OChM6ifN1K/BG0DJEhpYoCukAw8GA/u6ITtJHhhFOhhCGnthGGTLbQyZdZNqHKMHKsPbtDSFMCZM+/f6ITQnlxrEuvXbYGC/6tcByXTKdz9/WfPXN4b7eam9z/MLAnC3xbaDYzirbRPhZld+mMoy/X+scvZKUsjj9d3/DePPEt+n+arCmVcp9/lM1MKuN7aUAh8Ficc5TUmNBIgkFWFOy1nPMao058vvDEVtd7stC4bW9EX5wKdkOMs0mNx69ba/eBs1rrVvPaZrgUj6Y/eiDxNulQe8vbnUgjGFawPJCZ4KP4HhYsxoJOxs4bk9bK6iOwMZQdWAn8I91zXqOA3EMsqqJMN6g93iDH2Hu8TadZ5MsARzOVSglEWJJNoOpfn0HhNfZbyv8ODlezXGHPxMtZ6jDfRZlwZOjKeV6QShglMFKQFn6grqrfBNy04gE/gRUOJhMNOYtdNu9LOkNkGSRQikATVnMsUiktmRZzN7gOr0sJo0VWShZ5HOmszEL67C6JJKWfvR631PGae7SLJaGIUjlz5ereUHpNIt5wWR2OWKK9RomkymWDtqFdJMIiSRLdumGlkRYEiXpCMlwOKJ78zbR8BomSlhbKJcl6+WU+dJgXEjpBKK0zJaGR9MV/SRiVYKTHcJOn2T3GsnasVwbpN2gRYhDUG4Mm41jtdywXptzMwMuNab4L7yxiWlUW+0shKaPqumzfBmPX3H6fqs1i8WMArttavqNL1VfbMW3wRpfUm3wjOAK8A0UAkgcnvRivU4HQ+IkQoaEIbjlBrOaszmCtds2z7Td0l4Gt+tLYzm8ZsvVU7ZcvU162/tYUwtq6konlQ289OGKGP128YE0wL4W/urOkP/hf/xbZEfxYDLh//unf+HhzxdlXFXgvVQ6sBODMrBp2FS9nuPmUKX+EkTetaE5ykWd+yjwBNhZqDRbRezr1klfDWmvFl85R5okZLsZN25pSq0pS4OuKioHG11RatDap1xtuNgI4Ao/3j1arFjo+wgR4MQOjmOqWpoR4QgygRKOdQDrNRS1b3NV+cov+BNQie9BfCsQ0n+vLXSQKCFRMkRI6bmElMg4RoVgbYnWOdZKkJZeP2N0bY+be3v00ph+phiPDwCDm88ptUZhGGSSo9cgESnb80yb08QxhBFY64M48sIynqzYLC4H8WUN08dzrI0wLsQMdhl2O4T9PkkiGESSpPba3R2OtpnD1wAAIABJREFU6O99TNgfYcOEpYPVesN8vWG+3GBsxNpIXAnT9YbJfEk/CcGAFQkq7ZNkIzpdQ7hcI5zE0aG0krVxrIxhuS45zkuI6thzSa2lNJe7CjzjdMW3jbM2seedjM6S4bOob9cajq3dPk5xRXxfGed9AY2epLFCEDzTRPIUcRUQhhF2A1tLM4cxhs167ncG5//kFC9NedngiB084R3Vf3nCNrK40e3CdpZ91uz9vEnWhq26Q7QuV/ht4mpS82J8/Pmn3Ll7h9lyxeqHny+Q9DY4hPzwnNs7wA04KnzVt5qDueGb2HYyEJHX/DpvsE9VsbWoOSt2ejsIAu88gRMgBCIQhIEiiSqqzKdj+Im3wzl/KXWO1iXrNcyP6rhXXn0l7HkwwJMV9AMIAoiUb1ZyAK7y5Nc5nLMIoYmiiiRx6HJbSAd/wtUVaAOmejt2pP3hCPCkUQJpnNKL4vrDsVijMdqgAaUkUvogicV8Rq/fY+/WHlk6YDTaYzDaO5FaRlGPj0Y3sTJEO9C9uiRn4cdv7j+1HTuRJFMhygK6RBtfxGtWoSNOWtuestCUCkBRriSFK5jNNZv8kpDezFe0jh0crddsHs9wFiJ1zRNSF7Ii9FMjGYEYEtLHmIT1EuxyzWy+4fE059tf9tk/PCBJIpIkpBM6fu6GCGuQdondGKY6xEQjTGIwqcGYI8qN8/03EgQh/UGI69feJKGlXG/Il2uYvkmflAvAwYsf8lw864BvqsYlIOFYsjVS/40T3gYXV/FtGF9DeNvBFhI/izmH+B7TMoKQEhFFSGcxxtd+rRXgLKb0jtWNi4LlxR3KbdzFE982D2+ef1at0RDc8sz97VCLthSicXqIuCK+Hyoi/MHVTKAuwuf1fcCNm3uMrl+nNJY8f4cd52zw/hIS8prUHjuQH/nM4eDYE8xjXTe41c1uouJdOd6KoJ46O4dAIJyol97r+wEhBIEQiNqfTLuSqtKUpSZflZSloSwr5keag8e/Llb4PBxtIKpqTh4FdBJfGjiuvDSjqok4QYhyjk5lqZIK62rC7hxlWaE1vlq98UT4oldRer3ByXWlFIM4pZdmWK3RRmOKktVqBa5EhgIkFIucxdwTy1t3bhPFKb3egH5vhDUlVmuUyuj3RuSFIY1ydOYpqyTEFpbFouD2nT1u3dqj14voxTFKSig0xXxGPjlgMTvA6tI7RxRQzn1aXlNcUYBKfQ8MpGhjWaw0m7dWLn8x1PAaer2G9RJKTWGWzMMOIw3IBIOktBEiTIAEQZ8lfYzpsF6DMxsOxkc8Gs94cDBh/HhCtxuRlR26ScijcUIkBB3pp3xrF7EJu7hojeussRFsTE5p/RlcSugkCVEYIkIJCmbTI/Ly8P2SPVwE2qlwH9gJ6mLtzDpsfWuaqXEz23jGbGQH6AiIIrBhRBl1MGaDNa4WyEukcCSRQBh3Isid8/JJaU0UMWylDSu2s+r2BU5z+EbG0N78huy61nNeJ0jjCu8P2g2QbXwgE+cXorIF5WbBdPaI6ePLEOnyk9fsrnJYPWIrTkq9POK4aWJrpuAbfOnl7ZiytVFV1XbVSAiEOD2Nboivr7Se/Od79wJBFCmECFDKoVSHTrRhtNAczb1o49dUgLvAoAPDHnR7AUkSo4ItG98JBNIFiLrqG4mAAEfkOCHDzjlcVREGxhPOylFpiEqYF7C8wHmGkgqkVzwoGYIEazVIiFG+0V0XkFu01WhrKYsCazRFvmI6mTCbzJhNFqRx5i3JrGY2mzKZ7DOZjnk0GbPKc0qt0YVm0I98499A0YssPeVII+0DMHSJlQULfKVee1MIjHdPw7Vs7GzdrLVagVktWOX20oSQNND7Y4hC6CQQdUmTPkk3ozSW/YOJn/hg6WZDukmfwTVYyy79vqHb9RM8rCEU0O+GfHxrSLeb0M0Srl8bkfV9k9u6XLFeGybTJZPJlMODCYeHY5bLJc46wqhDvz+k3+0iQoGSktKUzPMV06M1jM+6in+g+AAcHM7DxWp8G2lDOw7NsbXZOKvzDSHqQCL9xYYdNp2ITd7OPhYIGZJECulKH33oXl6qE+MT2pqmgSZprdHodjgdU9z28A1bt4Nfmjrb+AZPV4yv8NtBs4DRHDjNBMdw5fBwHmy1ocjnHE0fMZm8/VDap+GAB/iugIytJ1AMx00YeS0xOCG+M96FGNA5d9K7EuBJ7lny61qPrUUPgCMIgChAKoGrBGkmGA59WbgsNeVKo3FUVFSuQlcWW1XoSlNph3Pe1U14lQUqgEgFhEFAogKiKCRSAYGKTrapIegIgXPu5Dhxribnwm9nVVW4qiIKNJUKqdwxzlUkGjJd8ZcHF0fnhJLeokyGvuIKaKOJQ0UUR0gJ+UqxsJZCF5S6RJcWC5SrgicPJ8wGYxaTCXkWI4VESMtivmD//n1mc5/gVhQl2lmclQzSHnEck0aCVGliCbH0UguNBptjzZxCgyl95dtav1Sv2yob6ffURQ6byzrSOHyE6rBPrz9g0L9GKDtsjGF+OGG1XLI2hn5/Sb9/jb0NmChjz1icg6QTgXN0Qhh0E+7eHJJ1+3S7fQbDPt3ukDDssFxumK0th5MVv+xP2D8Yc/TLGPIVpB3iQYebt/rcvH37ZMNm8yMOp3P0fO09ha/wweJiiG8Tb6bYssR26oNrXdoIoVT18pmEUEgEvqENwrojQiIFpFGH0JaUzld7H7/kpjUVW8PWnL05BLYxyU/HFDfXzytcnyW9DUkuW2//Cr8NtK2pr/BiuMqw0RuKIuddeus/jZzTqtJLopFsoaoqTyqF9y4/RXkbAuy2Vd7tzf6XQAEuABUgCAgChRAKV9WubzVRrnDYSqOdo9IVzjWX0wN0IAQBgiAQhAEgAm+5VhP0c+Fa233i6ODr2M2/AAcigAB2hOAipSVSCqSU/hJKhLXYrbYOqIlx/U8hvOTBSkIpsaUhny+YjCekUcxg2KPX82EoWmt0WVIUJWW5wlrflqbSlFQKlPQ2nFobrF6AtehCk+c5pTW++qw82XWArU88su47MtanZ7/YGOV5HU1vCxFhmBBFGVjH/GjK48NDr6t1G55ka+Z7DsKEm7e9cDAMQ5KoQwdB0gmR0vv1J9kuSdqnm2Uk3V0IBZP5imXpOHy85Nsf9tGPDmE8ARxECZ0oopMkJEmXzabEGMNyueHhwQQOP9Ay5xVO8OaJbzuWB55mCo7nHpPH1ES0GS3rUoEKO16jUw/4nU5CZOds8KT3ZWUO4AnrvHW9ZBtA0ciRYVvhbVeAm7f0IuL7fi91+1lLGCREUYQQChCU6zWb6hHv+7u7wttDZSqcqai09YnAV3hpHONqUiZwwm0bw0Trh2iRXQEQIITzhv4tkskJcXZeDwyo+nkV4ERA6BwuEDgXcNxIEurGucaCTAiBDGpdcVPpFYKqqivO9UZuny9ONL3tS1U5tN7UBBtwjsr593yROCG9dVFF1iTRAk4brLW+bqNCpASrvPyBujospESXmsl4TKQUaRyR3bpFFKfEWUZeLFBS+kQ6XZ8ldAlaIUKfwpUv5sxXC8qi8O4EDUHOLKaozzNntHRSQ744ty2mRsROGKMUKAlWF1hdYt07Uqhbn5oGYIxhMpvBpNV0ujqkImS9t0cn6pAlGcPuLoN+grQWnGO4mzEY9ok6XaK0T5h0CaMEY2F/PGFjLIeTCfrbH2EzwX86Hehm9Lt90jACB2W5YT5fsn84hns/8rxP8QofBt488W0EtF22OoKzmt4XkN91/RDnjLeesQASh/AJbQZkFCHCkDWG83q4nwfD0zGcjSqjQVP1barAbULcCC+a4nWDpqmtuf9dz7lfHglffvEV169/RK+XEUUJSRSRJBlRkqGCCIdgtViw/8tP7P/yEz/+cI+5fnGrzL/56m9QUYcgiHh4MOaH/X/hSgjy4cBVFa6yp9v53xU6QJrCRkN++c3oqsohhO97cM7naQgcwm3Lu41aTAhOZAYNxd1Wf+vb2jrgQNTSCd/b50RA5RxNkIfnup6gNmS1Ib6nJBeibr6riXlDeKmrwK5+fuXcyfUT8qsrqkrX7hTurewiDek9IcASpJUnsgf/mJA4ilHWr+81j7XWYrWlXJWMD8ZIJHt73hdIKUWUpkQrhVIhsvShFtZabFlgpQAVg1KsipLJZMJsBkb7ZrUslURxhJWlTxyrlxp9tVfWfUjPOKOIiCTrE8eRb3wTYEuJK6TXDRv79oOUDGxKQ7nesFouOT7PQWE1Zj6f47RBSgg7Id0kIZSCEMi6Gf3+BhllhJ0uyAgrOqw2PqRivi6ZPJ7D5pDtkrIk7SQknQhrDeujOZPZlPHjKUc/H3JFeq8AFyV1iPDEt/EwbNtnNElubd1vOyZNw5GCpYGlWTMrl6w3XpagjYO1xS4NwzshCV1+ZsqrtMzM2botSDzZbXh647vb2Jk1tzfE1+K3owm5aJ86281w7Wa3y440GPLf/Lf/FV999SUqCAiUIo4ioihBqehEw+cqwVGmcHpFVS6YT8fMH22JrwSSQNXWMwmdJOFv/ubf8ldf/TVKRYhA8ejhhG/+/Ff8wz/+Ew8efvvu3vQV3hqcMRxbjQoE3T6U71Lmu4H+EDadDuWOg9XlnppqVyGclxgcI9gR4pTFWUNsPd9tfjovFTvzt7ZiBOcb3gIBOzVFrq1tRV2JbaKS67oxiC3ppfUT6olNrds9PlPVxbElzpXbNrdVtqX1rUm2461UJ6PQN+IpJVEqIlWKLIqx2mKdJiwiIqUwaYw1GqsNuia8AFJJVvmC8cF9xuN9kBYpAGm5NRqhbI60KyJpyCXoIkfaAl1CHEtkL0bJCKUGGDvjx3rVfQfLMLWEygcKepWDoighX1lW5dPRxUEYEWcxURwjRYSSqtYvey2wlZJQKyJd8Dh/y6t0kzHFZMzPJ2Wg87Ch+P5b/uGPH2M2G6wxWHOLfjej300whLiwLkEZWC9zHq8eczie8I//8Ee++ZdvqGZTTtZqswR2uySRYj6b8MuP35PP5zBfgcm5ylG9QoOL0/g2QtemJFHW15tSa9v2oGGJdbeYLmEufNTgkVuxQVAgKTFsXI5ZGW4dJSTS8qrp28f1pkStS5etnKFBU+1t7M4k20rxWZlD27O3OczbTW+XGV//9Rf89Vdf8IcvPkOXpU+JCiKUUl531+gAncNVJbgSnGZH+He2A9zaHbF34zq3bu0xuj4kyzK6WcYXX33Fxx9/QhBEBEHEo8mU3eEuSRLxf/67K+L7IcBWPgXt/2/vzMPkuKpD/ztV3TXdM62RZjTSWLIsL8gsNsYirGb1SyAQ8tghmMQBk4WQhQAheSE4JAqG5MsLL4QYHg9IiDExjywsIQk4BhKBHTDYIY6NbFkYa99Hs6lnuqe6q27+OLe6a0o9mrU1PZr7m6++nr59azu1nTr3LLmc0N3D4kuOLZJcHJIrFJnqDjDljnI6PoOm76yae71cDlOvWz9Ya/n1knuPTXmmzgVAoqBO8xlrfhdrIG60xamQPuuMa31TckCcSyzJXiOIrW6V3jiOWyq9asWta3a4ep04Ninrb90qzRDZQnn1uP33S99adn3UylssFOkp9RCGIbWKj09IFPgEFY/ID6hRgTCkQoTnq1tENaxwpDqOXx1nYF8f6wf66B8osXGwl0IxoKdYJOopQFjWZ1pYJa5ERD0BYVjSteeLRKkcUgY4NQG5Cqz1Ie/7+BSohRGVKpiGZShPVyGgVArIBz5B3sf380CAj49nE2wEQaDln3xffR/OteI7VyVz6jiPPPQQed+jK69n4+DAQGOgtxZDYo4aHi1z4Ohx9u0/xO6HdnFy7yMwcRrya2BtN0H/Wg2OiyJGhoepHPghrsyQoxVLr/gmCXVPo+kT+mmaT6tMT3Cb9sFPFGLr57CHpDjoIdRxuBvDJBWGNCb7qHpVLOQ5apiepSF5NCSW3kThTdyVEytwtuZGOnAt+147S9a2jkDwKBXyGgEehwwPHeP0+CjValUjv6tVqpMhk9WQ0dEJRkdHOXniGCP1phTW5QpcdcXj2HrJVjZt2czGgQG6u7spdBfoH9hId6FbFWgvR3epxObNmxkvT9KVG2Cq7oIMznviGAHy+Rw9xTzTrpJliME5dbyG31PTnImZzek06ta9oFF7zga7edalwQCSCmZr+ArEzSbSyq1VRpMiE17DNYJGZ89aleM4xounLcDObz2F034J9v+4lS9v4ieM0WC6hu9vKr7Zs/Ni64a0ET+MNCUmEYQhUT4kquTVJWHaFFOrRWrttdFvcaQpziphhSS78/GRIY4e2c9EOaBSLlCZGKJaGaFWHSeqTqilOFQ/3pGRIcphhRo+xL6mVmvUnlXqMVTKEAYRRPrALPX0kA+KFIs9GiyGpl+LwwphFBHW9Enj4+OHqMk3iohiz/onF9sr1MVy4Hs8dPwERw8e4DHbLuXizZvYtGVzw60xOQGPnhxizyN72XvgAObIIc0VvHYNbNlIsVszPdWqE5w8dBjGzm2VRcfKoj0W36R2sA8MArZoSkPxPY2mR0gbI2Lbbu+nTVtM1U4DpEtUZGOy50vWfxeaFdoSpTdxV067KieZGrpSy0pcG7JVBLM+wJ1GNwW6uwPyOY84DhkZPsahQwcYOnGKoRMnGBkeZXiWpPK9pW6eeMXjufzxj2XLli1s3LiRfBAQBIEGzcRqS4rwKHWX8DbniPG46klXcu/3vnGudtWxXMRGXzJzOXp6iqztrTE2DoV1sPGCIocPVYjOcaawaAL8fExUZF6KrwAbUFvWBO0flk+nM9NINS/lY5sj8XpQvDPSQ3qedRa1n6r0Jr679Wn+urkzfHfBi4PmclC3hYbyTBKHETc+IT5D8VXl19j/69aqa//s9jaU4PgcKL4x0+JGoiAk9APiKCKKVeklijQ9r3VxiKKImCSvb41q2LSenho+zpEjBcbHYGLEx/dreISElTFqYUgUWv010rRp8ViIVwwISn3kvTxZxRdgsgoSgolDuvMeff0l+voG6Ovro69/gPGxMSrVCcbHfMLxMd1G+wRSX2XNEOHjgW9TJHU6U4cYfeAQ/7FnDz/YdimXXHwpUyFUpwAi4ijm0KFDmAMPML0gdzcXbR0k70NUqzF0etgpvY5ZWXrFt5/pBcYTEidZHZWZ7oTmZb63JCkIvHiSLGuJsjpJUwnuIrEvNxXixNKb+PUmSjOcmaUNmjp+xyu+pRK96/rpH9jIxgsuoDxeJpcLWNc7QH//BsrlCcrlSUbHxzl5bIjRuN7wM1uDx8Vbt7J9+9VctX07W7ZsZf3GjfT29qqvcC7XGLrUOlke9TqsjWOCoMD27ds5fOg4R0/sXkYJONpPbPPA+hSD3LTKY3F9imiZLK5RHb0RzAODvnqfqyh5q+s2A9uskqk37TphXRXV2Cqsvte8laZz62osXNqPIFFK9f/0MuJG1yRoLXtzjhuW3XQ0mjpZnHm383I5AqCei6nXPTxPtdsYGwgHhFbhrdehPp/0PAugt7cXiJrPpzBiIioTJSWLI3VQi6PQWndr1KIKYaQjYCPlMqO15jDFWBVGyuNEkWaI8PwQ368wMV5lZAjKw2qADQC/BPmCTRBhFe2Z8IEgX6B3bT+9vX16Xw2S8ce4mUXI9/H9GLsDqfn1mEVYZb+NSDeYZIhzsUwdZXzXUe7f8wDUskpBi6ITUyMMDx+HWo2pyTL1o4ut8+tYDSy54htshjDRGNNm0WxyXGgOyaUrRczI0vgo9aY2I/HFTay4ifLbjVp704pvmWbdjXQVZo+mL28VfX8PaCq+nUx3by/r+gfYeMFmtmzZiucFDGzczGS5zOTkJPV6nbBeZ2qyytDwEOPj45TLk4TVKbZuvYiLtl7CZZddxhO3P4m+df10d5cIggAvp/k9G8nsSaw6euBLpRJXb9/O8PAon/+iU3zPdzyBfN4jyOdtuVWo16AyEbc93uTibZdSCHy6unIcPryfUyfsWFIEvucRSdzMhZj3IShAV143cGLiDFeMc5saKp6m/Da2Ie2mkEygyisevueR80hC0xopf1OLtddj3WZr8Kh7Obuc6a4NStKWCpFL+/PaLlm9N3GbIJdTN416DGGdugeEdWLPusBqM/Wo/YaCvr5eiKAW1QhrIWFYoVpO7tQ2gA2I4pAwrFILK1TiKtUwZLxc5lR1+jhjHRifqIDvU4x9rUBByPgYHDmipU+S0cO+EvQUA6CgBSrCmTVF3w/oXbtWrbx9ffSu7QVbRS4MQ+JQLcW+52v6Mt9Tlwy/qfQ2MlG02Z9ocMCjOhUzupSG1trp2fsATA4zcdSD06dhygWvOebGkiu+YXKNJRkbTtNMnwDTI74SpTcdDDcjC3udXAdcDGxNLX4Mtdyk/XXXos4Ua2gqvWm/3sTVAZoeG55tm0Jv4El7nJqvky2++48c4P7de8gVCgyNlpksT1KthsT1GFP3AHVLqBPjBd0EJSjlCoT1OrnuXurkGBot8/3djxIER8jlcnherhkEk7MPbaDxluN5TE5W2X/gAOPjnVc0wLG0hEZVxa6uLvrXrmHj+iEmKhHlCpyabx7CedI3OMCWrZdRLHZRLAZU6nVOnXhEf6xClM9cnbVIo78X40O1hDT9aL3GdTRNv4wTxVh/8YjJJdFNOa+hNCevnI2leckV6VlXJK/pTkHTEqx91UrreU2/hHpa4UUtt2dgu9dRy7D6xKp7Q+xBnNPYudjuV+Oz3T7fNkDNw8evQS2MKU+UUz+nUpslptqqR0REJWy9cUNjVfxCgShfIsgH+H6IXxmHgro5hEDeg6AHBgc3MV6uMTJSoXyWrCJaz0ID1nzfV//isEYYhlSrFa0MF4ZqNU4KbuR98r4e/9hagCM7XzvpLvh4ccyYZyt+n2uGzvNYkTxQDGxJP/RC6eTgoRXA0rs6jKGaYOLkmgqymKZNpotcwPQUCktw8TwOeApwVTdsWAMDa6A6AUPH4UAMjwAnaJ4/g3ZKK7QTqKU3fduIaSrC2YIWjWANmsFtnaz4Qshdd/07B4+c5MLNm+yYqEeOAM/LAXXiOCSMq1SrE9TrYSNi+9CxIbq7D1AIuikUusnlrLLrqcKsliM7HOrFdiBUH9BhGLLvwCH2Prp/Wffe0X6ieo0YQyEIWLduLYOVAabC4+z+YfvXvWZNL8XiGrqKXXQVA3JBT/NHg17gbR5aXwxxnFI4afrfJtXcEifZpt6pV1kdD+pgkpdOL/nF9vKaeX/TrgwxmoUjKUaRXq+6UlgFvLHC5t0traQ3X3VjdV8gpJ4EusWpEhXJqnP233NgTq9WqvixjcuIIlUkK1VVHq1SHCa5foseBb9ISARRgB8EUD1zHG+qBmEtICj2UQigWAD8gHJliOg4EGoK3961fQwMbiZihKGh8lkTLdQJqYYVqmGBiQmtAhfVImpR2FB6a7WQMNIyEX4+IO/7FItFfN9rKMnNAhntI6rViCN1qXA1apaObVdeqa4s1j9saGSE4bGT1KNIc/+Flc4fVu5Qll7xTZtHY5paY6LoJr97qCtEOrI7QJXfRQa7PH0Qrh2A7evgqjUehXxMt6f+Vnt98A81fXWTl6dBoC+12YlPb+Kn66d2K220TqzISXs2gVDnv5hV2f/o/ex/9P5Um4cejCR2eSEs0RuMY0UThnXqUR086OnuYmDdWmphhWPHxhlpZ1BbHorFHrq6io3KYmE2cqrDL85vPXB6WqrE5L4iZ5knpUuS6LTTPXS1PeenZiAVXBY1A8zSqqznaVasactKXhri5vxJdoZkmiLx8e+Md4zKRKVRuCLtJ5vohjViohCCIAIP/IKPHwb4YURQLFpFI52pXf+vRj7lUJWUYpCn0OszuDmgWBymUq7h+z49vQPkgxIwTqUatQhrm04UQVgNqQRVVdKjWjPYzm6w7/sEnt8IKA6CPPh+w31dLcXtvQ/v39vWxa9aHtm1y/7XCSWozy+WXvENaVp10yR++ZNMy97Q8O9NtMYeFuUjkOuCZ2+BZ2yAS7sLDBZKxNUyUa1KOAW1SdW3t6DuDYky2203MblZ11JT2gsjXbEt8e+F5i0wn5lWJomzxmKX4Vjt1GohtWpIjAFi1vQUiQfWc9mlZfY8GnN6FreCErCuBP3r1U/YeMLpSsQPDp59Pj/w6Crk6Cp6VCoVxk5PMD7eonpUB7PoAer0ENRSLOs8MOdVK1WtEhbktdpaUYcdw5pafpN0ZjGQD3z8vBatyBcDeggg6CUi0GpqYcRkeRxqY5QrPocOjlPdoI5yPT0DbL5okM0XQXmkTBTWKBbzTJRDxscrTJSrZ1FlAnKePj3CKGZioky14hPGEVFY08pwnk8+X6Qn8AmCPPkgwPd9m0kixPd9LZjh+3i+z7GRGVfm6Hic0rvULL3im1h5E3eH5Ma7xn5OMF3xTZdFA9VAN6PK7wKeU1etg2dthqcMDtLvdbHG9xmlRrlapVZVd4cCsAnYSNOym2xqUqAiCU6rof0TA0m6EJ3PdL+7hgWB6a7LDsdqpR5G1Go1dXfxoFTsoljoJ4piCoUTDJ+qcGxILYU5D7oCTc1Z6umiv7SGCzaso6eni+5igSiOmZiaYnjsNB4HeeTgzI8E3/fJ5TWLxOjoBMNDJxg5tbIUX8fSU6mqxRffI/ChWCwSBAET5Qn1mQ3DhmuAFwT4fh4/CPAjn54gT4EiftCDny8R1iJOHT/OqZN56mGVw0fGwM9TKGj2iPWbB+kv9TI2PM7E+ASV8jAj4yOMj1WpVKafuUIBnyKB5+MXbSllIIpCyhVNsRZbpZxSiWLBp1gM6Cn1EOStpRcYC9UynA8CSj1FwjBq/OZwOJT25PGF6SbPANUUT6M+wOkhzi6aaYVs5TY8VAGe53PqkhI8cxAG85CvRVSjSaq1GuXhMUaHYegknLaBn+l0ZT7NmhuTqG6eJE9LDzKkjSdJZofsLSVRgJNMELNDVNZyAAAY+0lEQVQNZzkc5zORaQZCedioc/HpX9dLPt/F4MAUW7ZUAUPOE/J+jkKhi2KXDhn3FgvkfF9zTROTz+fo6srjPSnHpgtG2P2DUY61qGITnq4xNjrEiRO9HD50mMOHj1Afd1Hfq51STy+er5XNAi8AIiJ8Qr+qQW1+QKEIQcEjKGoJ4LBaJapUIFAf2kLvWkqlAcCnVOiht7eX8vgQI2MnIAqoVmFoaIJK5ThHgxHGx8uE5SqViRFq5THGxys6byNzfIDvBbruUl79dPM21qSmPr01m18YIAjyFEpFCqUCxZ4iQVAgbwPyoiimFBUZGOhjYMOAGmWiiHvu/+6yyNvh6ETaU7kt7RuQaIchqk0mWUrKNK3DSdqzODVvkltsHuN9z7kAnjUImwpryEcwNTnJ5OlJRk/CyHEYHlK9O4mjS2LrfOAkTS+MdOqyZLPS7+dpi25a8U2HfCTBce5R61jNNNO92mR2HuR8j/61ffSvb+Y29EUDp3K2mILkBC82GpRVr0OsuV97PA/jeWy+4AImKnU2btzPnh/s5b49GduvgdGRkxR7ujl8+CDloQ5J1eBYVnrWrgUg8Dxb18HH8yIC39Ncu4FHsRBQKAX4gZpdx8ar1MIqeQIKfQX6e/vo2zBIPt9Db68WlTh+ZD8T5QmiKKBSiYiiMseHxiCKqZZDKtWQqDJOrTyG74Pn5+kNevD9ks004RHh09tTpLS2p1EeeaI8QaVSJQpDrTjn+/iFgKDUQ7GnRMFafG0WYWsp9rn4os1cfNHFFK3y/pFPLZ/MHY5OQ4xZfMiBiHRC3ELHYow5WzxKS5xMz85CZApOrrOx1Ofq9ouL/I/nPNm6OsSIB57vEQQB+a6uRu5YzwPfE804IFYdNkaLHSQWYw/I5RDx8II8UT3myIkhjp4YYs8PH+XBPXFjRKeTcOdqe3D31aXHyXTpcdd/e1ioXKGdrg4Oh2PV0yiD64F4YIiJjZavjSOByGa88zxiq/D6SbqtGIhiPNFlSKIkI1oAAcPangDvgj6KhW1s2jzG0RNDPLg74nRllg1zOBwOx6rEKb4Oh6NtNF/Jm+ntTGxsxgFDjGherDhqmHql4TOk/Tw/KdDgNQJG6xgwMfmcT7ErT29vD+Q9onrE3uKQU3wdDofD0ZIlcXVwOBwOh8PhcDg6HZdty+FwOBwOh8OxKnCKr8PhcDgcDodjVeAUX4fD4XA4HA7HqsApvg6Hw+FwOByOVYFTfB0Oh8PhcDgcq4JFKb4isuN8TbIsIk8VkY+LyG4RmRSRAyJym4hc2qLvPhExLaZXLHDdTq7N/heKyCdF5JiITInIXhH5owWsd9XLVERumOE8TaYLFrDuVS9X23e9iHxIRB4VkYo9Tz8sIhsWsF4nU+07YK/9k1am3xGRFy1i3eezXC8WkX8Qkf1WVkMislNEfqJF34KI/ImIHLV9vy0iz1vgep1Mte8fisgdInLK3ktvWOS6V71c56srzGsbFpPOTES2AFuMMXcvdkM6DRH5AHANcBuwC7gQeA+wEdhujDmY6rsP2A3syCzmYWPMyALW7eSqfS8B/h3YC/w5cBy4BNhmjHnPPNe76mVqlbDHZGcH/hF41Bjz9AWs28lVRIC7gMcCvwc8BFwB3ATsAZ5l5nGjdTIFEekC7gEGgBuBY8DPAy8HXmiM2bmAdZ/Pcr0S+A1gJ3AI6AV+EfhJ4NXGmM+n+t5m238LeBT4VeAngGuMMffNc71Optr3NHAfKs83AG8yxtyyiHWvernOR1eYN8YYN7WYgA0t2i5GU++/N9O+D/jr5d7mlTDNU663A98F8su93Z08zUemLfo9FzDAry73fnTaNFe5ogqvAd6c6fsW2/645d6XTpnmIdPrreyuTbUJcD/w3eXej5UwoQWqDgL/mGq72sr1TZl+DwNfWu5t7vSplUxtu2c/t1n53rDc27qSphnO1QU/12abltzVwZr53yci77Sm7AkR+WcR2WinvxWRMRE5KCK/nZl3g4h8TET2WNP2QRH5jIhc2GLdr7cm8KqIPCAiL7Pm8p2ZfgMi8lEROSw6VL5bRN48274ZY062aNsPnETfPNqGkyuIyGOAFwE3G2Nqsy13NpxMZ+SNQAh8drb1tMLJFYDAfo5nuo/az3ndZ51MAXgmUAG+kepngDuAp7Xa9tk4n+XaCmNMHRgD0vfPl9nvf5Pp91ngRaKW9jnjZNpoj1vPsTCcXNusgy1SS9+BvR+l2gywH/hn1HT9c+gD4XZ02Pp3gRcAH7N9X5Ka93HAh4BXA88DrkOHu/YBhVS/F6Ja/xftOt6IDjEcAXam+vWib7IHUFP6C4A/ASLgrQvY3yfYbf7NTPs+9KBNAlPA3cArnFwXLld0uMgArwG+auU6AtwKrHcyXdi5mulTtOft59y5uqhzVVAFbRfwVKAEPB14EPiyk+mCZPohYKxF3/fbvi9ycm25jx5qPbsAHRYOgR9L/f5Z1AUvO99P2f270sl0fjLN9F0Si6+T69zvFQuSb5sOzh4gl2r7U9v+u6m2HHAC+KuzLN8HLrLzvjLV/i3g+1gfZdv2I7Zf+uC8B6gCl2eW+wlgKL2Nc9jXHPpwOwH0ZX67GVXUnosqajvttlzv5LowuQLvsusdBz4C/CjwZuAUcC92aMnJdH7naqbf6+12vGwh56mT67TfeoDP2/Un0z8BRSfTBV3/v2LX+4RM/3+17a93cm25DR9InX+ngVdlfr8DuLvFfC+w8zzXyXR+Ms30bbfiuyrlmtqvWZ9rc5nalc7sq0ZN1wm77ee/JA3290dQ4TcQkV8Wkf8SkTJQR98oQN9YEBEftap8zlhp2OV9Dw2CSvNi4DvAXhHJJZPdjvVoAMpc+TDwLFSZnRawZox5qzHmVmPMncaYvwd+DFXO5p19YBZWk1yTc3OnMeZXjTH/aoz5OPpAfArqBrEUrCaZZnkjOmz05Xkse66sNrl+Ah2efwvwfPv5VODvRWSp7rOrSaafQc/NT4nIVXZY9d2otQrUKrVUnE9y/TPgacBLga8AnxGR/5neZFTRyCJzWPZ8WE0yPZesZrnO9bk2K7nFzHwWshsVnqW9kHwRkbei0ft/ikacjqAK0N2pfgNAHtX6sxzPfN+IvoHN5CO6fsY9SCGaPuvNwBuNMXfM1t8YE4nI3wF/LCKbjDFH57KeObCa5HrKfn410570ezJ6sSyW1STTdL9NqJXn5syNdKlYNXIVkZ9ErecvMMZ83TZ/U0QeRc/XlwL/MJf1zMKqkakxZlREXg18Cg1oA/ghagm7CViqeyqcR3I1xhxCI+UB/sn6ZX4AHX0AGAa2tpi1L/X7UrCaZHouWZVyna8ONhvtUnwXynXA140x70wa5MycbUOosDe2mH+Q5lsMqPJ0AnjbDOt7eLYNEpEb0WH3XzfGfHq2/ulZ7Wert+tzzUqU6y77OZP8ljSYYAGsRJmmuR4d7vrUbMs9x6xEuV5lP+/JtH/Xfj6BpVF8F8pKlCnGmDtFg1y3oefqHvShXQG+N9s6zgEdJ9cW3Au8PfV9F/BKEek2xkym2q9AlaVHFrCOpWQlynQlsGLluggdbEY6rXJbN2e+Qbwp/cUYE6ECerWINIZnROQpQPZA3g48HjhgjLm3xXT6bBsjIr8OvA+40Rhz81x3wpr8X2vXe2yu87WRlSjXu9HcnS/OtCffs0rGuWYlyjTNG4D7zTzzdp4DVqJck2s8mwf5Gfbz8NnWcQ5YiTJNtssYY35gjNlt9+MXgU8bY8pnm+8c0VFyzWJdbJ6DWsoTvoRa9V6b6pcDXgfcYYyZms862sBKlOlKYEXKdaE62Gx0msX3duC3rS/Xd9GApte06Pf76BDiF0Tk46iJfgf6AEpbAj+IXtB3isgH0beQHvSAPdcY8/KZNkRErkN9UG4H/lVEnpn6edwY86Dt93o0qfqX0Tx0g2hC8Kegw5+dwIqTqzGmLiLvAm4Rkf+HBg5tQ6O6d6JBLsvJipNpqv+PAE8E3knnsRLl+nn0vLxVRG5C/e4eb7fxIPCFOe99e1iJMk2GN/8DtURtQ629NeB35rzn7aWT5LoD6Eej+4+hkfI/j76M/XTSzxhzn4j8DfBnIpJHfTd/GVVsfmZ+u98WVpxMbd/nAxtsH4CnWl9ajMb9LDcrTq7zfa7NC9OeyMP3ZdpusO3bMu07gbtS34vAR9GghtOor8eldt4dmXl/GhX2FHb4BvhP4AuZfn3oQdqLDuWcAO4E3j7Lvt3C9Ajt9JSObnwmqoQdR2/KY8DXWEC6HSfXplxT/X8WjTKdQv36bgZKTqaLkumH7Lk6uJjr38l1Wt+LgL+066jaz08AFzqZLlimn0R9AEP7eTPQ787Vlvv2MvQ5dMKuYz9q3X12i75F1NfzmD1Xv0OqUIiT6YJkunOm89rJdWFyZZ7PtflMiypZ3EmIlvh7BHi/Meam5d6e8wUn16XHybQ9OLkuPU6m7cHJdelxMm0P56NcV6TiKyLJG+vX0GGwy4D/hboZXGmWLovCqsLJdelxMm0PTq5Lj5Npe3ByXXqcTNvDapFrp/n4zpUI9Qv5MJo2YwI1sb/2fDkwy4ST69LjZNoenFyXHifT9uDkuvQ4mbaHVSHXFWnxdTgcDofD4XA45kunpTNzOBwOh8PhcDjawopRfEVkp63s4VggIrJDRIzN2zhTn2ttn2sXsZ5FL6MTsPuxQ5au5GxbsLJ+3xz6TbuGOuU4icgrROQ3lmnds14TKxUn15WNleGPLvd2nG84uTZZrXpVRz/QHcvC94Br6IzKSMvNtWhew/PlOvkVO3UarwCWRUE7z3FyXdn8Pppv1bG0OLmuctzb+AyISJdZ/io25xxjzDhaNe2srFb5rGTMYhJ+dwDunGsPTq4Oh+Nc0Cn3mo60ZInIdSKyW0SmRGSXiLyyRZ8BEfmoiBy2/XaLyJtb9LtURG4TkZO2333Z5aWG5Z4oIv9iK678bRt3cbl5goj8m4hMishREXlvMpzfavjbDofcJSIvFZH/FJEprOVQRDaIyGdEZFxERkXkVmDdcuyUiGwTkU+LyF4RqYjIo/Yc6cv0azm8IyL7ROQW+/8O1DIAULMyMam+m0TkVhEZsufV/SJyfWZ5N9j5niUifysip0XkuIj8jv39xVaeEyJyj2hpyPT8IiLvEJGHRSS0x+rDItLbevflRhE5ZPf9myKyfS773WJBrxKRu+35MSoifyciW2ebbyFYeb8RuDCRsT0OyXn4KhH5hIicRIvEICK3iMi+Fss6Y//s+fl/ReSgPU4H7TnSdZZterGIlK2sO/IeORtOrsuLiFwtIl8QkVP2enw4dd3/uIh82V7PkyLyfRF5p4j4qfmTe82NqeO3Yzn2pZNwcl044vSqBh1n8RWRFwCfAf4ZLam6Aa00lUeriWAf/P+OViPZgVYOeRHwUdE3ipttv4vQqjQngHegVUteB3xORF5hjPlSZvX/gFZf+mOml+c73/giWhXpj1C5vQfd3x1nmeexwJ8DNwGPAsO2/fPA1cC7gR+g8l2ymtrzZDNa4entwAiag/DdaDnpa+a5rL8AtqClFJ+DpnkBQER6gG+glWvejZalvR74tIh0G2M+nlnWp4BbgY8DrwX+UETWAS9BS92Wgf8NfFFEHmOMCe1870fLs34E+EfgClT+V4vI840x6XP0DcAB4NeALuC9wNdF5HJjzDBzRETeglb4+Su7jDXoefENEXmSmWcN9jlwE3qNPw2t6ANazWet/f9m4CtoBb/CfBYs+sLzLbQ85vuA+4GNaInxwK4nO88b0GN/0wpP1u7kukyIyNPRylmPoM+dQ8DlwJNsl8uAr6PHoAo8Fb3GNgDvsn2uAb6NVq/6mG071O5t72ScXBeO06syLKbsWzsmVPAPAl6q7RmkytShiloVuDwz7yfQpMs5+/0v0YOyPtPvq8B9qe877PLfttz732bZJvv5rhZyO41aaq+1fa5N/b4TPWG3Z+Z7oe17Xab9K9llLNP+5lCl1QBPzuzPzhb99wG3tJBXLtPv11rtH5r0+wTg2+832H6/l9mmE2jJ4EtT7S+zfZ9vv/fbc/yWzDqut/1elmoz9rzvSbVdYtdx00z7nT3WQAktuf3JzDovQUtSnrUU5SKO0y3AoUxbsm1fmKH/vhbt2f17L/rC8uSzrLtxjNFE7TXgF5bzvHVyXdkT8E30Zbh7Dn3FyuhG9GU9/dw7o0Ttap6cXBclO6dXpaaOGm6yQxJPA/7epKxZxpjvoEpJwovRN469IpJLJuBf0KTLV6T6fRkYa9HvajlzyPgL7divDiQ73PBZVOl54lnm2WeMuS/Tdg36APxci+Wdc0QkEJF32+GZCvqwvdP+/LglXNXzgMPGmJ2Z9r9G36SvyLR/JfnHGFNHLRZ7jDF7U31228+L7OczUcvtX2eW9VmgDjw/0/5lY8xEaj37UF/t+Vi6rwF6gdsy18shu33Pm8eylorFXJM/DtxjjPnPOfT9IPAHwGuMMX+xiHWuFJxc24CIdAPPBm4zxkzO0GeTiHxMRPajL5Q11HK+DrWcOzI4uS4cp1edSae5OgygpvfjLX5Lt20EtqEndivWp/q9wU4z9RtPfT9vKpPMQla+yfcLW/yW0Eo2m4ARY0z2OMy0jHbzR8BbUYvUt1Ar9hbUHWNew7mz0E9reRxL/Z5mJPM9nKENmtuZLGPaeowxdRE51WIdM10zV7Zon4nk4fC1GX7PbvO5YDHX5Hrgv+bY9/XALmbe9/MNJ9f20IfGzrQcPre+zV9C3bJ2oC+UFTQDx40s7X3qfMLJdeE4vSpDpym+Q6jQB1v8Ngjst/+fQoeL3zbDch5O9bsT9S1pxZHM99VSxm4Q9dNNfwc4zMznRCvZHAX6RCSfUX5bHb9zwXXArcaYRk5bESm16FdFLZtZssrkTAzT2oJ8gf08NcflzLaOZJm7kkb7Zr2+xTpmumYOz2OdyTJvSK8zxVL7986FVuddFfUlzZKVyxD6MjcXfgy4A/iKiLzEGFOe11auPJxc28MI6hY2k3weg/qe/qwxpjGaIyIvPQfbtpJxcl04Tq/K0FGuDsaYCLgHeE066ldEnoH6GSbcDjweOGCMubfFdDrV70nArhn6LXtajWXipzLfr0MDrL4/z+V8G/CBV7dY3nLQzZlvq29q0W8/8FgRaTzkReR5aCBXmuT8KGbavwFsEZFnZ9p/Gr1xPDSfjZ6Bu+36s7J8Hfpy8o1M+0ts0B0AInIJ6i7x7XmsM7GSb5vhenl4tgUskCnOlPHZ2A8MishA0iAij+HMl5E7gKeLyNVzWOYu1P/1cuB2EcmeCysRJ9dzjB2Gvwu4XkRayb7bfjbuUyKSB36mRd+Q+R2/8xYn14Xj9Koz6TSLL2gKqTvQCPePoT6Tf0BzGBnUb+x1wJ0i8kH0TaQHPWjPNca83Pb7PeC7wDdF5MOoP0sf6st6mTHm59q/Ox3JL9oL4B40avMXgB3GmFERmfNCjDFfFZG7gI/Zh2WS1eFsvsLt5HbgjSLyAOpH+yrgWS36fRZ4M/BJ0bRPl6KJ/scy/ZK8t+8Uka8AkTHmXjQI6G3A50XkRnT47WfQYL9fsjeaRWGMGRaRPwV+R0QmUJ+qJ6A+a3eh0blpKsAdIvInqG/wH6DDTR+cxzrHReS3gI+IyAbUN3kMtbI8Hw2C+Mzi9qwlDwL9IvLLwL2o5fFs/B2ateA2K6MBNPvFUKbfB9GXka+JVrZ7wPZ9OfAWk8lQYYx5SDSN37+hStqLs31WGE6uy8Nvoi+m3xaR/4PeHy4DtqMR9fuB94tIhCpq75hhOQ8CPykit6MWzyPGmKw1bTXh5LpwnF6VZrmj61pNqF/Yw6jFYhfwSs6MLO5DD9Re9A3uBGp+f3tmWVvQNDqHbb+jaPTh9ak+O2gRvX++Tan9fCL6EKqgJ/5N2GhPZs7qcNcMy9wA/H/UUjiKpu16eXYZ52j/BlCldsROt6FO/Qa4IdP3l1BFvYJaOp/CmVkdfDSV2Al0mM2kftsEfBpVCqbQlE7XZ9Zxg133tkz7GfJE37wNqch3NDL5HfZaSM7djwC9mXkNmvrs3ejDoGqvhWwWjuw1dMaxtu0vsefHuJXPI2j6uyvadNx67Dk0YrdnX2rbXjDDPK9ARygqqL/pj2f3z/bbiKaRO2pleBBNL9eVuSZyqXkut3L8dlbWK2lycl1W2T8ZTUE4amW5G/ht+9t29OV10srjvajxwQCXpJbxbOA/7PVsUOPEsu+bk+vKnHB6VWMSu4EOh8PhcDgcDsd5TUf5+DocDofD4XA4HO3CKb4Oh8PhcDgcjlWBU3wdDofD4XA4HKsCp/g6HA6Hw+FwOFYFTvF1OBwOh8PhcKwKnOLrcDgcDofD4VgVOMXX4XA4HA6Hw7EqcIqvw+FwOBwOh2NV4BRfh8PhcDgcDseqwCm+DofD4XA4HI5VgVN8HQ6Hw+FwOByrAqf4OhwOh8PhcDhWBU7xdTgcDofD4XCsCpzi63A4HA6Hw+FYFTjF1+FwOBwOh8OxKnCKr8PhcDgcDodjVfDfl6uf7+KljrQAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "from matplotlib import pyplot as plt\n", - "import numpy as np\n", - "\n", - "label_list = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\", \"dog\", \"rog\", \"horse\", \"ship\", \"truck\"]\n", - "print(\"The 32 images with label of the first batch in ds_train are showed below:\")\n", - "ds_iterator = ds_train.create_dict_iterator()\n", - "ds_iterator.get_next()\n", - "batch_1 = ds_iterator.get_next()\n", - "batch_image = batch_1[\"image\"].asnumpy()\n", - "batch_label = batch_1[\"label\"].asnumpy()\n", - "%matplotlib inline\n", - "plt.figure(dpi=144)\n", - "for i,image in enumerate(batch_image):\n", - " plt.subplot(4, 8, i+1)\n", - " plt.subplots_adjust(wspace=0.2, hspace=0.2)\n", - " image = image/np.amax(image)\n", - " image = np.clip(image, 0, 1)\n", - " image = np.transpose(image,(1,2,0))\n", - " plt.imshow(image)\n", - " num = batch_label[i]\n", - " plt.title(f\"image {i+1}\\n{label_list[num]}\", y=-0.65, fontdict={\"fontsize\":8})\n", - " plt.axis('off') \n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 使用Summary算子记录数据\n", - "\n", - "在进行训练之前，需定义神经网络模型，本流程采用AlexNet网络，以下一段代码中定义AlexNet网络结构。\n", - "\n", - "MindSpore提供了两种方法进行记录数据，分别为：\n", - "- 通过Summary算子记录数据\n", - "- 通过 `SummaryCollector` 这个callback进行记录\n", - "\n", - "下面展示在AlexNet网络中使用Summary算子记录输入图像和张量数据。\n", - "\n", - "- 使用 `ImageSummary` 记录输入图像数据。\n", - "\n", - " 1. 在 `__init__` 方法中初始化 `ImageSummary`。\n", - " \n", - " ```python\n", - " # Init ImageSummary\n", - " self.image_summary = P.ImageSummary()\n", - " ```\n", - " \n", - " 2. 在 `construct` 方法中使用 `ImageSummary` 算子记录输入图像。其中 \"Image\" 为该数据的名称，MindInsight在展示时，会将该名称展示出来以方便识别是哪个数据。\n", - " \n", - " ```python\n", - " # Record image by Summary operator\n", - " self.image_summary(\"Image\", x)\n", - " ```\n", - " \n", - "- 使用 `TensorSummary` 记录张量数据。\n", - "\n", - " 1. 在 `__init__` 方法中初始化 `TensorSummary`。\n", - " \n", - " ```python\n", - " # Init TensorSummary\n", - " self.tensor_summary = P.TensorSummary()\n", - " ```\n", - " \n", - " 2. 在`construct`方法中使用`TensorSummary`算子记录张量数据。其中\"Tensor\"为该数据的名称。\n", - " \n", - " ```python\n", - " # Record tensor by Summary operator\n", - " self.tensor_summary(\"Tensor\", x)\n", - " ```\n", - "\n", - "当前支持的Summary算子:\n", - "\n", - "- [ScalarSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ScalarSummary): 记录标量数据\n", - "- [TensorSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.TensorSummary): 记录张量数据\n", - "- [ImageSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ImageSummary): 记录图片数据\n", - "- [HistogramSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.HistogramSummary): 将张量数据转为直方图数据记录" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "import mindspore.nn as nn\n", - "from mindspore.common.initializer import TruncatedNormal\n", - "from mindspore.ops import operations as P\n", - "\n", - "def conv(in_channels, out_channels, kernel_size, stride=1, padding=0, pad_mode=\"valid\"):\n", - " weight = weight_variable()\n", - " return nn.Conv2d(in_channels, out_channels,\n", - " kernel_size=kernel_size, stride=stride, padding=padding,\n", - " weight_init=weight, has_bias=False, pad_mode=pad_mode)\n", - "\n", - "def fc_with_initialize(input_channels, out_channels):\n", - " weight = weight_variable()\n", - " bias = weight_variable()\n", - " return nn.Dense(input_channels, out_channels, weight, bias)\n", - "\n", - "def weight_variable():\n", - " return TruncatedNormal(0.02)\n", - "\n", - "\n", - "class AlexNet(nn.Cell):\n", - " \"\"\"\n", - " Alexnet\n", - " \"\"\"\n", - " def __init__(self, num_classes=10, channel=3):\n", - " super(AlexNet, self).__init__()\n", - " self.conv1 = conv(channel, 96, 11, stride=4)\n", - " self.conv2 = conv(96, 256, 5, pad_mode=\"same\")\n", - " self.conv3 = conv(256, 384, 3, pad_mode=\"same\")\n", - " self.conv4 = conv(384, 384, 3, pad_mode=\"same\")\n", - " self.conv5 = conv(384, 256, 3, pad_mode=\"same\")\n", - " self.relu = nn.ReLU()\n", - " self.max_pool2d = P.MaxPool(ksize=3, strides=2)\n", - " self.flatten = nn.Flatten()\n", - " self.fc1 = fc_with_initialize(6*6*256, 4096)\n", - " self.fc2 = fc_with_initialize(4096, 4096)\n", - " self.fc3 = fc_with_initialize(4096, num_classes)\n", - " # Init TensorSummary\n", - " self.tensor_summary = P.TensorSummary()\n", - " # Init ImageSummary\n", - " self.image_summary = P.ImageSummary()\n", - "\n", - " def construct(self, x):\n", - " # Record image by Summary operator\n", - " self.image_summary(\"Image\", x)\n", - " x = self.conv1(x)\n", - " # Record tensor by Summary operator\n", - " self.tensor_summary(\"Tensor\", x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv2(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv3(x)\n", - " x = self.relu(x)\n", - " x = self.conv4(x)\n", - " x = self.relu(x)\n", - " x = self.conv5(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.flatten(x)\n", - " x = self.fc1(x)\n", - " x = self.relu(x)\n", - " x = self.fc2(x)\n", - " x = self.relu(x)\n", - " x = self.fc3(x)\n", - " return x" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 使用 `SummaryCollector` 记录数据\n", - "\n", - "下面展示使用`SummaryCollector`来记录标量、直方图信息。\n", - "\n", - "在MindSpore中通过`Callback`机制，提供支持快速简易地收集损失值、参数权重、梯度等信息的`Callback`, 叫做`SummaryCollector`（详细的用法可以参考API文档中[mindspore.train.callback.SummaryCollector](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.train.html#mindspore.train.callback.SummaryCollector)）。`SummaryCollector`使用方法如下： \n", - "\n", - "`SummaryCollector` 提供 `collect_specified_data` 参数，允许自定义想要收集的数据。\n", - "\n", - "下面的代码展示通过 `SummaryCollector` 收集损失值以及卷积层的参数值，参数值在MindInsight中以直方图展示。\n", - "\n", - "\n", - "\n", - "\n", - "```python\n", - "specified={\"collect_metric\": True, \"histogram_regular\": \"^conv1.*|^conv2.*\"}\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_01\", \n", - " collect_specified_data=specified, \n", - " collect_freq=1, \n", - " keep_default_action=False, \n", - " collect_tensor_freq=200)\n", - "```\n", - "\n", - "- `summary_dir`：指定日志保存的路径。\n", - "- `collect_specified_data`：指定需要记录的信息。\n", - "- `collect_freq`：指定使用`SummaryCollector`记录数据的频率。\n", - "- `keep_default_action`：指定是否除记录除指定信息外的其他数据信息。\n", - "- `collect_tensor_freq`：指定记录张量信息的频率。\n", - "- `\"collect_metric\"`为记录损失值标量信息。\n", - "- `\"histogram_regular\"`为记录`conv1`层和`conv2`层直方图信息。\n", - "\n", - "  程序运行过程中将在本地`8080`端口自动启动MindInsight服务并自动遍历读取当前notebook目录下`summary_dir`子目录下所有日志文件、解析进行可视化展示。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 导入模块" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import mindspore.nn as nn\n", - "from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor\n", - "from mindspore.train import Model\n", - "from mindspore.nn.metrics import Accuracy\n", - "from mindspore.train.callback import SummaryCollector\n", - "from mindspore.train.serialization import load_checkpoint, load_param_into_net\n", - "from mindspore import Tensor\n", - "from mindspore import context\n", - "\n", - "device_target = \"GPU\"\n", - "context.set_context(mode=context.GRAPH_MODE, device_target=device_target)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 定义学习率\n", - "\n", - "以下一段代码定义学习率。" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "\n", - "\n", - "def get_lr(current_step, lr_max, total_epochs, steps_per_epoch):\n", - " \"\"\"\n", - " generate learning rate array\n", - "\n", - " Args:\n", - " current_step(int): current steps of the training\n", - " lr_max(float): max learning rate\n", - " total_epochs(int): total epoch of training\n", - " steps_per_epoch(int): steps of one epoch\n", - "\n", - " Returns:\n", - " np.array, learning rate array\n", - " \"\"\"\n", - " lr_each_step = []\n", - " total_steps = steps_per_epoch * total_epochs\n", - " decay_epoch_index = [0.8 * total_steps]\n", - " for i in range(total_steps):\n", - " if i < decay_epoch_index[0]:\n", - " lr = lr_max\n", - " else:\n", - " lr = lr_max * 0.1\n", - " lr_each_step.append(lr)\n", - " lr_each_step = np.array(lr_each_step).astype(np.float32)\n", - " learning_rate = lr_each_step[current_step:]\n", - "\n", - " return learning_rate\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 执行训练" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "============== Starting Training ==============\n", - "epoch: 1 step: 1, loss is 2.3056953\n", - "epoch: 1 step: 2, loss is 2.3169184\n", - "epoch: 1 step: 3, loss is 2.2732773\n", - "epoch: 1 step: 4, loss is 2.3223817\n", - "epoch: 1 step: 5, loss is 2.300379\n", - "epoch: 1 step: 6, loss is 2.2816362\n", - "epoch: 1 step: 7, loss is 2.3317387\n", - "epoch: 1 step: 8, loss is 2.2595024\n", - "epoch: 1 step: 9, loss is 2.3138928\n", - "epoch: 1 step: 10, loss is 2.294712\n", - "\n", - "...\n", - "\n", - "epoch: 10 step: 1553, loss is 0.23232733\n", - "epoch: 10 step: 1554, loss is 0.35622978\n", - "epoch: 10 step: 1555, loss is 0.24221122\n", - "epoch: 10 step: 1556, loss is 0.2082262\n", - "epoch: 10 step: 1557, loss is 0.29972154\n", - "epoch: 10 step: 1558, loss is 0.32628897\n", - "epoch: 10 step: 1559, loss is 0.44762093\n", - "epoch: 10 step: 1560, loss is 0.4621265\n", - "epoch: 10 step: 1561, loss is 0.13807176\n", - "epoch: 10 step: 1562, loss is 0.40322578\n", - "Epoch time: 242827.643, per step time: 155.459\n", - "============== Starting Testing ==============\n", - "============== {'Accuracy': 0.8299278846153846} ==============\n" - ] - } - ], - "source": [ - "\n", - "network = AlexNet(num_classes=10)\n", - "net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n", - "lr = Tensor(get_lr(0, 0.002, 10, ds_train.get_dataset_size()))\n", - "net_opt = nn.Momentum(network.trainable_params(), learning_rate=lr, momentum=0.9)\n", - "time_cb = TimeMonitor(data_size=ds_train.get_dataset_size())\n", - "config_ck = CheckpointConfig(save_checkpoint_steps=1562, keep_checkpoint_max=10)\n", - "ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_alexnet\", config=config_ck)\n", - "model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", - "\n", - "summary_base_dir = \"./summary_dir\"\n", - "os.system(f\"mindinsight start --summary-base-dir {summary_base_dir} --port=8080\")\n", - "\n", - "# Init a SummaryCollector callback instance, and use it in model.train or model.eval\n", - "specified = {\"collect_metric\": True, \"histogram_regular\": \"^conv1.*|^conv2.*\"}\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_01\", collect_specified_data=specified, collect_freq=1, keep_default_action=False, collect_tensor_freq=200)\n", - "\n", - "print(\"============== Starting Training ==============\")\n", - "model.train(epoch=10, train_dataset=ds_train, callbacks=[time_cb, ckpoint_cb, LossMonitor(), summary_collector], dataset_sink_mode=True)\n", - "\n", - "print(\"============== Starting Testing ==============\")\n", - "param_dict = load_checkpoint(\"checkpoint_alexnet-10_1562.ckpt\")\n", - "load_param_into_net(network, param_dict)\n", - "acc = model.eval(ds_eval, callbacks=summary_collector, dataset_sink_mode=True)\n", - "print(\"============== {} ==============\".format(acc))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## MindInsight看板\n", - "\n", - "在本地浏览器中打开地址：`127.0.0.1:8080`，进入到可视化面板。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/mindinsight_panel.png)\n", - "\n", - "在上图所示面板中可以看到`summary_01`日志文件目录，点击**训练看板**进入到下图所示的训练数据展示面板，该面板展示了标量数据、直方图、图像和张量信息，并随着训练、测试的进行实时刷新数据，实时显示训练过程参数的变化情况。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/mindinsight_panel2.png)\n", - "\n", - "### 标量可视化\n", - "\n", - "标量可视化用于展示训练过程中标量的变化趋势，点击打开标量信息展示面板，该面板记录了迭代计算过程中的损失值标量信息，如下图展示了损失值标量趋势图。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/scalar_panel.png)\n", - "\n", - "上图展示了神经网络在训练过程中损失值的变化过程。横坐标是训练步骤，纵坐标是损失值。\n", - "\n", - "图中右上角有几个按钮功能，从左到右功能分别是全屏展示，切换Y轴比例，开启/关闭框选，分步回退和还原图形。\n", - "\n", - "- 全屏展示即全屏展示该标量曲线，再点击一次即可恢复。\n", - "- 切换Y轴比例是指可以将Y轴坐标进行对数转换。\n", - "- 开启/关闭框选是指可以框选图中部分区域，并放大查看该区域，可以在已放大的图形上叠加框选。\n", - "- 分步回退是指对同一个区域连续框选并放大查看时，可以逐步撤销操作。\n", - "- 还原图形是指进行了多次框选后，点击此按钮可以将图还原回原始状态。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/scalar_select.png)\n", - "\n", - "上图展示的标量可视化的功能区，提供了根据选择不同标签，水平轴的不同维度和平滑度来查看标量信息的功能。\n", - "\n", - "- 标签选择：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的标量信息。\n", - "- 水平轴：可以选择“步骤”、“相对时间”、“绝对时间”中的任意一项，来作为标量曲线的水平轴。\n", - "- 平滑度：可以通过调整平滑度，对标量曲线进行平滑处理。\n", - "- 标量合成：可以选中两条标量曲线进行合成并展示在一个图中，以方便对两条曲线进行对比或者查看合成后的图。\n", - " 标量合成的功能区与标量可视化的功能区相似。其中与标量可视化功能区不一样的地方，在于标签选择时，标量合成功能最多只能同时选择两个标签，将其曲线合成并展示。\n", - "\n", - "### 直方图可视化\n", - "\n", - "\n", - "直方图用于将用户所指定的张量以直方图的形式展示。点击打开直方图展示面板，以直方图的形式记录了在迭代过程中所有层参数分布信息。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram_panel.png)\n", - "\n", - "如下图为`conv1`层参数分布信息，点击图中右上角，可以将图放大。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram.png)\n", - "\n", - "下图为直方图功能区。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram_func.png)\n", - "\n", - "上图展示直方图的功能区，包含以下内容：\n", - "\n", - "- 标签选择：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的直方图。\n", - "- 纵轴：可以选择步骤、相对时间、绝对时间中的任意一项，来作为直方图纵轴显示的数据。\n", - "- 视角：可以选择正视和俯视中的一种。正视是指从正面的角度查看直方图，此时不同步骤之间的数据会覆盖在一起。俯视是指偏移以45度角俯视直方图区域，这时可以呈现不同步骤之间数据的差异。\n", - "\n", - "### 图像可视化\n", - "\n", - "图像可视化用于展示用户所指定的图片。点击图像展示面板，展示了每个一步进行处理的图像信息。\n", - "\n", - "下图为展示`summary_01`记录的图像信息。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/image_panel.png)\n", - "\n", - "通过滑动上图中的\"步骤\"滑条，查看不同步骤的图片。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/image_function.png)\n", - "\n", - "上图展示图像可视化的功能区，提供了选择查看不同标签，不同亮度和不同对比度来查看图片信息。\n", - "\n", - "- 标签：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的图片信息。\n", - "- 亮度调整：可以调整所展示的所有图片亮度。\n", - "- 对比度调整：可以调整所展示的所有图片对比度。\n", - "\n", - "### 张量可视化\n", - "\n", - "张量可视化用于将张量以表格以及直方图的形式进行展示。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/tensor_func.png)\n", - "\n", - "上图展示了张量可视化的功能区，包含以下内容：\n", - "\n", - "- 标签选择：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的表格数据或者直方图。\n", - "- 视图：可以选择表格或者直方图来展示tensor数据。在直方图视图下存在纵轴和视角的功能选择。\n", - "- 纵轴：可以选择步骤、相对时间、绝对时间中的任意一项，来作为直方图纵轴显示的数据。\n", - "- 视角：可以选择正视和俯视中的一种。正视是指从正面的角度查看直方图，此时不同步骤之间的数据会覆盖在一起。俯视是指 偏移以45度角俯视直方图区域，这时可以呈现不同步骤之间数据的差异。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/tensor.png)\n", - "\n", - "上图中将用户所记录的张量以表格的形式展示，包含以下功能：\n", - "\n", - "- 点击表格右边小方框按钮，可以将表格放大。\n", - "- 表格中白色方框显示当前展示的是哪个维度下的张量数据，其中冒号\":\"表示当前维度的所有值，可以在方框输入对应的索引或者:后按Enter键或者点击后边的打勾按钮来查询特定维度的张量数据。 假设某维度是32，则其索引范围是-32到31。注意：可以查询0维到2维的张量数据，不支持查询超过两维的张量数据，即不能设置超过两个冒号\":\"的查询条件。\n", - "- 拖拽表格下方的空心圆圈可以查询特定步骤的张量数据。\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 单独记录数据\n", - "\n", - "以上流程为整体展示Summary算子能记录到的所有类型数据，也可以单独记录关心的数据，以降低性能开销和日志文件大小。\n", - "\n", - "### 单独记录损失值标量\n", - "\n", - "1. 配置`specified`参数为：\n", - "\n", - "```python\n", - "specified={\"collect_metric\": True}\n", - "```\n", - "\n", - "2. 配置`summary_collector`为：\n", - "\n", - "```python\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_loss_only\", \n", - " collect_specified_data=specified, \n", - " collect_freq=1, \n", - " keep_default_action=False)\n", - "```\n", - "\n", - "  运行以下一段代码，单独记录损失值标量信息。" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "============== Starting Training ==============\n", - "epoch: 1 step: 1, loss is 2.316685\n", - "epoch: 1 step: 2, loss is 2.3051994\n", - "epoch: 1 step: 3, loss is 2.2948198\n", - "epoch: 1 step: 4, loss is 2.3207984\n", - "epoch: 1 step: 5, loss is 2.3364246\n", - "epoch: 1 step: 6, loss is 2.2956452\n", - "epoch: 1 step: 7, loss is 2.2634928\n", - "epoch: 1 step: 8, loss is 2.3085115\n", - "epoch: 1 step: 9, loss is 2.254295\n", - "epoch: 1 step: 10, loss is 2.3339896\n", - "\n", - "...\n", - "\n", - "epoch: 10 step: 1556, loss is 0.40271574\n", - "epoch: 10 step: 1557, loss is 0.5172653\n", - "epoch: 10 step: 1558, loss is 0.3401278\n", - "epoch: 10 step: 1559, loss is 0.4081525\n", - "epoch: 10 step: 1560, loss is 0.31565452\n", - "epoch: 10 step: 1561, loss is 0.41298962\n", - "epoch: 10 step: 1562, loss is 0.24210417\n", - "Epoch time: 54385.175, per step time: 34.818\n", - "============== Starting Testing ==============\n", - "============== {'Accuracy': 0.8254206730769231} ==============\n" - ] - } - ], - "source": [ - "class AlexNet(nn.Cell):\n", - " \"\"\"\n", - " Alexnet\n", - " \"\"\"\n", - " def __init__(self, num_classes=10, channel=3):\n", - " super(AlexNet, self).__init__()\n", - " self.conv1 = conv(channel, 96, 11, stride=4)\n", - " self.conv2 = conv(96, 256, 5, pad_mode=\"same\")\n", - " self.conv3 = conv(256, 384, 3, pad_mode=\"same\")\n", - " self.conv4 = conv(384, 384, 3, pad_mode=\"same\")\n", - " self.conv5 = conv(384, 256, 3, pad_mode=\"same\")\n", - " self.relu = nn.ReLU()\n", - " self.max_pool2d = P.MaxPool(ksize=3, strides=2)\n", - " self.flatten = nn.Flatten()\n", - " self.fc1 = fc_with_initialize(6*6*256, 4096)\n", - " self.fc2 = fc_with_initialize(4096, 4096)\n", - " self.fc3 = fc_with_initialize(4096, num_classes)\n", - "\n", - " def construct(self, x):\n", - " x = self.conv1(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv2(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv3(x)\n", - " x = self.relu(x)\n", - " x = self.conv4(x)\n", - " x = self.relu(x)\n", - " x = self.conv5(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.flatten(x)\n", - " x = self.fc1(x)\n", - " x = self.relu(x)\n", - " x = self.fc2(x)\n", - " x = self.relu(x)\n", - " x = self.fc3(x)\n", - " return x\n", - "\n", - "lr = Tensor(get_lr(0, 0.002, 10, ds_train.get_dataset_size()))\n", - "network = AlexNet(num_classes=10)\n", - "net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n", - "net_opt = nn.Momentum(network.trainable_params(), learning_rate=lr, momentum=0.9)\n", - "time_cb = TimeMonitor(data_size=ds_train.get_dataset_size())\n", - "config_ck = CheckpointConfig(save_checkpoint_steps=1562, keep_checkpoint_max=10)\n", - "ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_alexnet\", config=config_ck)\n", - "model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", - "\n", - "# Init a SummaryCollector callback instance, and use it in model.train or model.eval\n", - "specified = {\"collect_metric\": True}\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_loss_only\", collect_specified_data=specified, collect_freq=1, keep_default_action=False)\n", - "\n", - "print(\"============== Starting Training ==============\")\n", - "model.train(epoch=10, train_dataset=ds_train, callbacks=[time_cb, ckpoint_cb, LossMonitor(), summary_collector], dataset_sink_mode=True)\n", - "\n", - "print(\"============== Starting Testing ==============\")\n", - "param_dict = load_checkpoint(\"checkpoint_alexnet_1-10_1562.ckpt\")\n", - "load_param_into_net(network, param_dict)\n", - "acc = model.eval(ds_eval, callbacks=summary_collector, dataset_sink_mode=True)\n", - "print(\"============== {} ==============\".format(acc))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "此时点击打开MindInsight**训练列表**看板中的`./summary_loss_only`目录，如下图所示，可以看到只记录有损失值标量信息。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/loss_scalar_only.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 单独记录参数分布直方图\n", - "\n", - "1. 配置`specified`参数为只记录`conv1`层直方图信息：\n", - "\n", - "```python\n", - "specified = {\"histogram_regular\": \"^conv1.*\"}\n", - "```\n", - "2. 配置`summary_collector`为：\n", - "\n", - "```python\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_histogram_only\", \n", - " collect_specified_data=specified, \n", - " collect_freq=1,\n", - " keep_default_action=False)\n", - "```\n", - "\n", - "  运行以下一段代码记录`conv1`层参数直方图信息（为了减少内存占用和减少训练时间，在后续的训练中设置`epoch`为1）。" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "============== Starting Training ==============\n", - "epoch: 1 step: 1, loss is 2.2941067\n", - "epoch: 1 step: 2, loss is 2.2770133\n", - "epoch: 1 step: 3, loss is 2.2869372\n", - "epoch: 1 step: 4, loss is 2.261474\n", - "epoch: 1 step: 5, loss is 2.3582025\n", - "epoch: 1 step: 6, loss is 2.290377\n", - "epoch: 1 step: 7, loss is 2.2761602\n", - "epoch: 1 step: 8, loss is 2.3452077\n", - "epoch: 1 step: 9, loss is 2.2613692\n", - "epoch: 1 step: 10, loss is 2.3617961\n", - "\n", - "...\n", - "\n", - "epoch: 1 step: 1552, loss is 0.9691028\n", - "epoch: 1 step: 1553, loss is 1.1841048\n", - "epoch: 1 step: 1554, loss is 1.3479778\n", - "epoch: 1 step: 1555, loss is 1.2386065\n", - "epoch: 1 step: 1556, loss is 1.0223479\n", - "epoch: 1 step: 1557, loss is 1.1582826\n", - "epoch: 1 step: 1558, loss is 0.87887794\n", - "epoch: 1 step: 1559, loss is 0.956085\n", - "epoch: 1 step: 1560, loss is 1.3973256\n", - "epoch: 1 step: 1561, loss is 1.234511\n", - "epoch: 1 step: 1562, loss is 1.0787828\n", - "Epoch time: 62971.025, per step time: 40.314\n", - "============== Starting Testing ==============\n", - "============== {'Accuracy': 0.5687099358974359} ==============\n" - ] - } - ], - "source": [ - "lr = Tensor(get_lr(0, 0.002, 1, ds_train.get_dataset_size()))\n", - "network = AlexNet(num_classes=10)\n", - "net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n", - "net_opt = nn.Momentum(network.trainable_params(), learning_rate=lr, momentum=0.9)\n", - "time_cb = TimeMonitor(data_size=ds_train.get_dataset_size())\n", - "config_ck = CheckpointConfig(save_checkpoint_steps=1562, keep_checkpoint_max=10)\n", - "ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_alexnet\", config=config_ck)\n", - "model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", - "\n", - "# Init a SummaryCollector callback instance, and use it in model.train or model.eval\n", - "specified = {\"histogram_regular\": \"^conv1.*\"}\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_histogram_only\", collect_specified_data=specified, collect_freq=1, keep_default_action=False)\n", - "\n", - "print(\"============== Starting Training ==============\")\n", - "model.train(epoch=1, train_dataset=ds_train, callbacks=[time_cb, ckpoint_cb, LossMonitor(), summary_collector], dataset_sink_mode=True)\n", - "\n", - "print(\"============== Starting Testing ==============\")\n", - "param_dict = load_checkpoint(\"checkpoint_alexnet_2-1_1562.ckpt\")\n", - "load_param_into_net(network, param_dict)\n", - "acc = model.eval(ds_eval, callbacks=summary_collector, dataset_sink_mode=True)\n", - "print(\"============== {} ==============\".format(acc))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "此时点击打开MindInsight**训练列表**看板中的`./summary_histogram_only`目录。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram_only.png)\n", - "\n", - "在MindInsight面板中，如上图所示，只展示了直方图信息。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/histogram_only_all.png)\n", - "\n", - "点击进入直方图面板，如上图所示，只展示了`conv1`层的直方图信息。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 单独记录张量数据\n", - "\n", - "1. 在AlexNet网络的`__init__`方法中初始化`TensorSummary`。\n", - "2. 在AlexNet网络的`construct`方法中使用`TensorSummary`算子记录张量数据。\n", - "3. 配置`summary_collector`为：\n", - "\n", - "```python\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_tensor_only\", \n", - " collect_specified_data=None, \n", - " collect_freq=1,\n", - " keep_default_action=False, \n", - " collect_tensor_freq=50)\n", - "```\n", - "\n", - "  运行以下一段代码只记录张量数据。\n" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "============== Starting Training ==============\n", - "epoch: 1 step: 1, loss is 2.3178232\n", - "epoch: 1 step: 2, loss is 2.3172731\n", - "epoch: 1 step: 3, loss is 2.3256557\n", - "epoch: 1 step: 4, loss is 2.3034613\n", - "epoch: 1 step: 5, loss is 2.318819\n", - "epoch: 1 step: 6, loss is 2.2775433\n", - "epoch: 1 step: 7, loss is 2.322216\n", - "epoch: 1 step: 8, loss is 2.2980762\n", - "epoch: 1 step: 9, loss is 2.3208668\n", - "epoch: 1 step: 10, loss is 2.3162236\n", - "\n", - "...\n", - "\n", - "epoch: 1 step: 1552, loss is 1.5239879\n", - "epoch: 1 step: 1553, loss is 1.3195564\n", - "epoch: 1 step: 1554, loss is 1.2827079\n", - "epoch: 1 step: 1555, loss is 1.0843871\n", - "epoch: 1 step: 1556, loss is 1.2715582\n", - "epoch: 1 step: 1557, loss is 1.4982214\n", - "epoch: 1 step: 1558, loss is 1.0394028\n", - "epoch: 1 step: 1559, loss is 1.0470619\n", - "epoch: 1 step: 1560, loss is 1.1495018\n", - "epoch: 1 step: 1561, loss is 1.0332686\n", - "epoch: 1 step: 1562, loss is 1.1649165\n", - "Epoch time: 113988.939, per step time: 72.976\n", - "============== Starting Testing ==============\n", - "============== {'Accuracy': 0.5647035256410257} ==============\n" - ] - } - ], - "source": [ - "class AlexNet(nn.Cell):\n", - " \"\"\"\n", - " Alexnet\n", - " \"\"\"\n", - " def __init__(self, num_classes=10, channel=3):\n", - " super(AlexNet, self).__init__()\n", - " self.conv1 = conv(channel, 96, 11, stride=4)\n", - " self.conv2 = conv(96, 256, 5, pad_mode=\"same\")\n", - " self.conv3 = conv(256, 384, 3, pad_mode=\"same\")\n", - " self.conv4 = conv(384, 384, 3, pad_mode=\"same\")\n", - " self.conv5 = conv(384, 256, 3, pad_mode=\"same\")\n", - " self.relu = nn.ReLU()\n", - " self.max_pool2d = P.MaxPool(ksize=3, strides=2)\n", - " self.flatten = nn.Flatten()\n", - " self.fc1 = fc_with_initialize(6*6*256, 4096)\n", - " self.fc2 = fc_with_initialize(4096, 4096)\n", - " self.fc3 = fc_with_initialize(4096, num_classes)\n", - " # Init TensorSummary\n", - " self.tensor_summary = P.TensorSummary()\n", - "\n", - " def construct(self, x):\n", - " x = self.conv1(x)\n", - " # Record tensor by Summary operator\n", - " self.tensor_summary(\"Tensor\", x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv2(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv3(x)\n", - " x = self.relu(x)\n", - " x = self.conv4(x)\n", - " x = self.relu(x)\n", - " x = self.conv5(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.flatten(x)\n", - " x = self.fc1(x)\n", - " x = self.relu(x)\n", - " x = self.fc2(x)\n", - " x = self.relu(x)\n", - " x = self.fc3(x)\n", - " return x\n", - "\n", - "lr = Tensor(get_lr(0, 0.002, 1, ds_train.get_dataset_size()))\n", - "network = AlexNet(num_classes=10)\n", - "net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n", - "net_opt = nn.Momentum(network.trainable_params(), learning_rate=lr, momentum=0.9)\n", - "time_cb = TimeMonitor(data_size=ds_train.get_dataset_size())\n", - "config_ck = CheckpointConfig(save_checkpoint_steps=1562, keep_checkpoint_max=10)\n", - "ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_alexnet\", config=config_ck)\n", - "model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", - "\n", - "# Init a SummaryCollector callback instance, and use it in model.train or model.eval\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_tensor_only\", collect_specified_data=None, collect_freq=1, keep_default_action=False, collect_tensor_freq=50)\n", - "\n", - "print(\"============== Starting Training ==============\")\n", - "model.train(epoch=1, train_dataset=ds_train, callbacks=[time_cb, ckpoint_cb, LossMonitor(), summary_collector], dataset_sink_mode=True)\n", - "\n", - "print(\"============== Starting Testing ==============\")\n", - "param_dict = load_checkpoint(\"checkpoint_alexnet_3-1_1562.ckpt\")\n", - "load_param_into_net(network, param_dict)\n", - "acc = model.eval(ds_eval, callbacks=summary_collector, dataset_sink_mode=True)\n", - "print(\"============== {} ==============\".format(acc))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "此时点击打开MindInsight**训练列表**看板中的`./summary_tensor_only`目录，如下图所示，可以看到只记录有张量信息。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/tensor_only.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 单独记录图像\n", - "\n", - "1. 在AlexNet网络的`__init__`方法中初始化`ImageSummary`。\n", - "2. 在AlexNet网络的`construct`方法中使用`ImageSummary`算子记录输入的图像。\n", - "3. 配置`summary_collector`为：\n", - "\n", - " ```python\n", - " summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_image_only\", \n", - " collect_specified_data=None, \n", - " collect_freq=1, \n", - " keep_default_action=False)\n", - " ```\n", - "\n", - "  运行以下一段代码只记录张量数据。" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "============== Starting Training ==============\n", - "epoch: 1 step: 1, loss is 2.3218322\n", - "epoch: 1 step: 2, loss is 2.3194063\n", - "epoch: 1 step: 3, loss is 2.3358874\n", - "epoch: 1 step: 4, loss is 2.3217764\n", - "epoch: 1 step: 5, loss is 2.3040292\n", - "epoch: 1 step: 6, loss is 2.2832437\n", - "epoch: 1 step: 7, loss is 2.31645\n", - "epoch: 1 step: 8, loss is 2.3330472\n", - "epoch: 1 step: 9, loss is 2.3014827\n", - "epoch: 1 step: 10, loss is 2.310499\n", - "\n", - "...\n", - "\n", - "epoch: 1 step: 1552, loss is 1.293778\n", - "epoch: 1 step: 1553, loss is 1.385756\n", - "epoch: 1 step: 1554, loss is 1.2190051\n", - "epoch: 1 step: 1555, loss is 1.2079114\n", - "epoch: 1 step: 1556, loss is 1.0067248\n", - "epoch: 1 step: 1557, loss is 1.2732614\n", - "epoch: 1 step: 1558, loss is 1.0397598\n", - "epoch: 1 step: 1559, loss is 1.1968248\n", - "epoch: 1 step: 1560, loss is 1.2161392\n", - "epoch: 1 step: 1561, loss is 1.135769\n", - "epoch: 1 step: 1562, loss is 1.4535972\n", - "Epoch time: 152835.351, per step time: 97.846\n", - "============== Starting Testing ==============\n", - "============== {'Accuracy': 0.5597956730769231} ==============\n" - ] - } - ], - "source": [ - "class AlexNet(nn.Cell):\n", - " \"\"\"\n", - " Alexnet\n", - " \"\"\"\n", - " def __init__(self, num_classes=10, channel=3):\n", - " super(AlexNet, self).__init__()\n", - " self.conv1 = conv(channel, 96, 11, stride=4)\n", - " self.conv2 = conv(96, 256, 5, pad_mode=\"same\")\n", - " self.conv3 = conv(256, 384, 3, pad_mode=\"same\")\n", - " self.conv4 = conv(384, 384, 3, pad_mode=\"same\")\n", - " self.conv5 = conv(384, 256, 3, pad_mode=\"same\")\n", - " self.relu = nn.ReLU()\n", - " self.max_pool2d = P.MaxPool(ksize=3, strides=2)\n", - " self.flatten = nn.Flatten()\n", - " self.fc1 = fc_with_initialize(6*6*256, 4096)\n", - " self.fc2 = fc_with_initialize(4096, 4096)\n", - " self.fc3 = fc_with_initialize(4096, num_classes)\n", - " # Init ImageSummary\n", - " self.image_summary = P.ImageSummary()\n", - "\n", - " def construct(self, x):\n", - " # Record image by Summary operator\n", - " self.image_summary(\"Image\", x)\n", - " x = self.conv1(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv2(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.conv3(x)\n", - " x = self.relu(x)\n", - " x = self.conv4(x)\n", - " x = self.relu(x)\n", - " x = self.conv5(x)\n", - " x = self.relu(x)\n", - " x = self.max_pool2d(x)\n", - " x = self.flatten(x)\n", - " x = self.fc1(x)\n", - " x = self.relu(x)\n", - " x = self.fc2(x)\n", - " x = self.relu(x)\n", - " x = self.fc3(x)\n", - " return x\n", - "\n", - "lr = Tensor(get_lr(0, 0.002, 1, ds_train.get_dataset_size()))\n", - "network = AlexNet(num_classes=10)\n", - "net_loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True, reduction=\"mean\")\n", - "net_opt = nn.Momentum(network.trainable_params(), learning_rate=lr, momentum=0.9)\n", - "time_cb = TimeMonitor(data_size=ds_train.get_dataset_size())\n", - "config_ck = CheckpointConfig(save_checkpoint_steps=1562, keep_checkpoint_max=10)\n", - "ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_alexnet\", config=config_ck)\n", - "model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n", - "\n", - "# Init a SummaryCollector callback instance, and use it in model.train or model.eval\n", - "summary_collector = SummaryCollector(summary_dir=\"./summary_dir/summary_image_only\", collect_specified_data=None, collect_freq=1, keep_default_action=False)\n", - "\n", - "print(\"============== Starting Training ==============\")\n", - "model.train(epoch=1, train_dataset=ds_train, callbacks=[time_cb, ckpoint_cb, LossMonitor(), summary_collector], dataset_sink_mode=True)\n", - "\n", - "print(\"============== Starting Testing ==============\")\n", - "param_dict = load_checkpoint(\"checkpoint_alexnet_4-1_1562.ckpt\")\n", - "load_param_into_net(network, param_dict)\n", - "acc = model.eval(ds_eval, callbacks=summary_collector, dataset_sink_mode=True)\n", - "print(\"============== {} ==============\".format(acc))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/image_only.png)\n", - "\n", - "在MindInsight面板中，如上图所示，只展示了输入图像信息。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 对比看板\n", - "\n", - "对比看板用于多次训练之间的数据对比。\n", - "\n", - "点击MindInsight看板中的**对比看板**，打开对比看板，可以得到多次（不同）训练搜集到的标量数据对比信息。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/multi_scalars.png)\n", - "\n", - "上图展示了`summary_01`（上图中红色曲线）和`summary_loss_only`（上图中蓝色曲线）的标量曲线对比效果，横坐标是训练步骤，纵坐标是标量值。\n", - "\n", - "![](https://gitee.com/mindspore/docs/raw/master/tutorials/notebook/mindinsight/images/multi_scalars_select.png)\n", - "\n", - "上图展示的对比看板可视的功能区，提供了根据选择不同训练或标签，水平轴的不同维度和平滑度来进行标量对比的功能。\n", - "\n", - "- 训练：提供了对所有训练进行多项选择的功能，用户可以通过勾选或关键字筛选所需的训练。\n", - "- 标签：提供了对所有标签进行多项选择的功能，用户可以通过勾选所需的标签，查看对应的标量信息。\n", - "- 水平轴：可以选择“步骤”、“相对时间”、“绝对时间”中的任意一项，来作为标量曲线的水平轴。\n", - "- 平滑度：可以通过调整平滑度，对标量曲线进行平滑处理。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 关闭MindInsight服务\n", - "\n", - "在终端命令行中执行以下代码关闭MindInsight服务。\n", - "\n", - "```shell\n", - "mindinsight stop --port 8080\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 注意事项和规格\n", - "- 在训练中使用Summary算子收集数据时，`HistogramSummary`算子会影响性能，所以请尽量少地使用。\n", - "- 不能同时使用多个 `SummaryRecord` 实例 （`SummaryCollector` 中使用了 `SummaryRecord`）。\n", - "- 为了控制列出summary文件目录的用时，MindInsight最多支持发现999个summary文件目录。\n", - "- 出于性能上的考虑，MindInsight对比看板使用缓存机制加载训练的标量曲线数据，并进行以下限制：\n", - " - 对比看板只支持在缓存中的训练进行比较标量曲线对比。\n", - " - 缓存最多保留最新（按修改时间排列）的15个训练。\n", - " - 用户最多同时对比5个训练的标量曲线。\n", - "- 为了控制内存占用，MindInsight对标签（tag）数目和步骤（step）数目进行了限制：\n", - " - 每个训练看板的最大标签数量为300个标签。标量标签、图片标签、计算图标签、参数分布图（直方图）标签、张量标签的数量总和不得超过300个。特别地，每个训练看板最多有10个计算图标签、6个张量标签。当实际标签数量超过这一限制时，将依照MindInsight的处理顺序，保留最近处理的300个标签。\n", - " - 每个训练看板的每个标量标签最多有1000个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", - " - 每个训练看板的每个图片标签最多有10个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", - " - 每个训练看板的每个参数分布图（直方图）标签最多有50个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", - " - 每个训练看板的每个张量标签最多有20个步骤的数据。当实际步骤的数目超过这一限制时，将对数据进行随机采样，以满足这一限制。\n", - "- 由于`TensorSummary`会记录完整Tensor数据，数据量通常会比较大，为了控制内存占用和出于性能上的考虑，MindInsight对Tensor的大小以及返回前端展示的数值个数进行以下限制：\n", - " - MindInsight最大支持加载含有1千万个数值的Tensor。\n", - " - Tensor加载后，在张量可视的表格视图下，最大支持查看10万个数值，如果所选择的维度查询得到的数值超过这一限制，则无法显示。\n", - "- 由于张量可视（`TensorSummary`）会记录原始张量数据，需要的存储空间较大。使用`TensorSummary`前和训练过程中请注意检查系统存储空间充足。 通过以下方法可以降低张量可视功能的存储空间占用：\\\n", - "  1）避免使用`TensorSummary`记录较大的Tensor。\\\n", - "  2）减少网络中`TensorSummary`算子的使用个数。\n", - "- 功能使用完毕后，请及时清理不再需要的训练日志，以释放磁盘空间。\n", - "- 备注：估算`TensorSummary`空间使用量的方法如下：\n", - " - 一个`TensorSummary`数据的大小 ＝ Tensor中的数值个数 * 4 bytes。假设使用`TensorSummary`记录的Tensor大小为32 * 1 * 256 * 256，则一个`TensorSummary`数据大约需要32 * 1 * 256 * 256 * 4 bytes = 8,388,608 bytes = 8MiB。`TensorSummary`默认会记录20个步骤的数据，则记录这20组数据需要的空间约为20 * 8 MiB ＝ 160MiB。需要注意的是，由于数据结构等因素的开销，实际使用的存储空间会略大于160MiB。\n", - " - 当使用`TensorSummary`时，由于记录完整Tensor数据，训练日志文件较大，MindInsight需要更多时间解析训练日志文件，请耐心等待。" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 总结\n", - "\n", - "本次体验流程为完整的MindSpore深度学习及MindInsight可视化展示的过程，包括了下载数据集及预处理过程，构建网络、损失函数和优化器过程，生成模型并进行训练、验证的过程，以及启动MindInsight服务进行训练过程可视化展示。读者可以基于本次体验流程构建自己的网络模型进行训练，并使用`SummaryCollector`以及Summary算子记录关心的数据，然后在MindInsight服务看板中进行可视化展示，根据MindInsight服务中展示的结果调整相应的参数以提高训练精度。\n", - "\n", - "以上便完成了标量、直方图、图像和张量可视化的体验，我们通过本次体验全面了解了MindSpore执行训练的过程和MindInsight在标量、直方图、图像和张量可视化的应用，理解了如何使用`SummaryColletor`记录训练过程中的标量、直方图、图像和张量数据。" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.5" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/tutorials/notebook/programming_guide/dtype.ipynb b/tutorials/notebook/programming_guide/dtype.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..3a4d47dcde38c12b1135570924e2bd0c2f623c31 --- /dev/null +++ b/tutorials/notebook/programming_guide/dtype.ipynb @@ -0,0 +1,111 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# dtype" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 概述\n", + "\n", + "MindSpore张量支持不同的数据类型，包含`int8`、`int16`、`int32`、`int64`、`uint8`、`uint16`、`uint32`、`uint64`、`float16`、`float32`、`float64`、`bool_`，与NumPy的数据类型一一对应。\n", + "\n", + "在MindSpore的运算处理流程中，Python中的`int`数会被转换为定义的`int64`类型，`float`数会被转换为定义的`float32`类型。\n", + "\n", + "详细的类型支持情况请参考https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.html#mindspore.dtype。\n", + "\n", + "以下代码，打印MindSpore的数据类型int32。" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Int32\n" + ] + } + ], + "source": [ + "from mindspore import dtype as mstype\n", + "\n", + "data_type = mstype.int32\n", + "print(data_type)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 数据类型转换接口\n", + "\n", + "MindSpore提供了以下几个接口，实现与NumPy数据类型和Python内置的数据类型间的转换。\n", + "\n", + " * `dtype_to_nptype`：将MindSpore的数据类型转换为NumPy对应的数据类型。\n", + "\n", + " * `dtype_to_pytype`：将MindSpore的数据类型转换为Python对应的内置数据类型。\n", + "\n", + " * `pytype_to_dtype`：将Python内置的数据类型转换为MindSpore对应的数据类型。\n", + "\n", + "以下代码实现了不同数据类型间的转换，并打印转换后的类型。" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Int64\n", + "\n" + ] + } + ], + "source": [ + "from mindspore import dtype as mstype\n", + "\n", + "np_type = mstype.dtype_to_nptype(mstype.int32)\n", + "ms_type = mstype.pytype_to_dtype(int)\n", + "py_type = mstype.dtype_to_pytype(mstype.float64)\n", + "\n", + "print(np_type)\n", + "print(ms_type)\n", + "print(py_type)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/tutorials/notebook/programming_guide/tensor.ipynb b/tutorials/notebook/programming_guide/tensor.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..fc4e13fb3dfae27a4f0996aefcf31a15fd7be21f --- /dev/null +++ b/tutorials/notebook/programming_guide/tensor.ipynb @@ -0,0 +1,149 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Tensor" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 概述\n", + "\n", + "张量（Tensor）是MindSpore网络运算中的基本数据结构。张量中的数据类型可参考[dtype](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/dtype.html)。\n", + "\n", + "不同维度的张量分别表示不同的数据，0维张量表示标量，1维张量表示向量，2维张量表示矩阵，3维张量可以表示彩色图像的RGB三通道等等。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "> 本文中的所有示例，支持在PyNative模式下运行，暂不支持CPU。" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 张量构造\n", + "\n", + "构造张量时，支持传入`Tensor`、`float`、`int`、`bool`、`tuple`、`list`和`NumPy.array`类型。\n", + "\n", + "`Tensor`作为初始值时，可指定dtype，如果没有指定dtype，`int`、`float`、`bool`分别对应`int32`、`float32`、`bool_`，`tuple`和`list`生成的1维`Tensor`数据类型与`tuple`和`list`里存放数据的类型相对应。\n", + "\n", + "代码样例如下：" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[1 2]\n", + " [3 4]] \n", + "\n", + " 1 \n", + "\n", + " 2 \n", + "\n", + " True \n", + "\n", + " [1 2 3] \n", + "\n", + " [4. 5. 6.]\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "from mindspore import Tensor\n", + "from mindspore.common import dtype as mstype\n", + "\n", + "x = Tensor(np.array([[1, 2], [3, 4]]), mstype.int32)\n", + "y = Tensor(1.0, mstype.int32)\n", + "z = Tensor(2, mstype.int32)\n", + "m = Tensor(True, mstype.bool_)\n", + "n = Tensor((1, 2, 3), mstype.int16)\n", + "p = Tensor([4.0, 5.0, 6.0], mstype.float64)\n", + "\n", + "print(x, \"\\n\\n\", y, \"\\n\\n\", z, \"\\n\\n\", m, \"\\n\\n\", n, \"\\n\\n\", p)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 张量的属性" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 属性\n", + "\n", + "张量的属性包括形状（shape）和数据类型（dtype）。\n", + "\n", + " * 形状：`Tensor`的shape，是一个tuple。\n", + "\n", + " * 数据类型：`Tensor`的dtype，是MindSpore的一个数据类型。\n", + "\n", + "代码样例如下：" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(2, 2) Int32\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "from mindspore import Tensor\n", + "from mindspore.common import dtype as mstype\n", + "\n", + "x = Tensor(np.array([[1, 2], [3, 4]]), mstype.int32)\n", + "x_shape = x.shape\n", + "x_dtype = x.dtype\n", + "\n", + "print(x_shape, x_dtype)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/tutorials/notebook/quick_start.ipynb b/tutorials/notebook/quick_start.ipynb index 6110b8aeda958ec86cfa7d0ae890dd0eff96da72..4180ec9e87ab99a8afaa73920ee704548dc9b1a8 100644 --- a/tutorials/notebook/quick_start.ipynb +++ b/tutorials/notebook/quick_start.ipynb @@ -525,7 +525,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "使用MindSpore定义神经网络需要继承`mindspore.nn.cell.Cell`，`Cell`是所有神经网络（`Conv2d`等）的基类。\n", + "使用MindSpore定义神经网络需要继承`mindspore.nn.Cell`，`Cell`是所有神经网络（`Conv2d`等）的基类。\n", "\n", "神经网络的各层需要预先在`__init__`方法中定义，然后通过定义`construct`方法来完成神经网络的前向构造，按照LeNet5的网络结构，定义网络各层如下：" ] diff --git a/tutorials/training/source_en/advanced_use/apply_gradient_accumulation.md b/tutorials/training/source_en/advanced_use/apply_gradient_accumulation.md index 76012d6196c00d1419108b0b58017d913c96fd6d..0ee7673c662d38680a970f955bee0e119104c609 100644 --- a/tutorials/training/source_en/advanced_use/apply_gradient_accumulation.md +++ b/tutorials/training/source_en/advanced_use/apply_gradient_accumulation.md @@ -36,6 +36,7 @@ The ultimate objective is to achieve the same effect as training with N x mini-b The MNIST dataset is used as an example to describe how to customize a simple model to implement gradient accumulation. ### Importing Library Files + The following are the required public modules and MindSpore modules and library files. ```python @@ -65,7 +66,9 @@ Use the `MnistDataset` API provided by the dataset of MindSpore to load the MNIS The following uses the LeNet network as an example. You can also use other networks, such as ResNet-50 and BERT. The code is imported from [lenet.py]() in the lenet directory of model_zoo. ### Defining the Training Model + The training process is divided into three parts: forward and backward training, parameter update, and accumulated gradient clearance. + - `TrainForwardBackward` calculates the loss and gradient, and uses grad_sum to implement gradient accumulation. - `TrainOptim` updates parameters. - `TrainClear` clears the gradient accumulation variable grad_sum. @@ -134,8 +137,8 @@ class TrainClear(Cell): ``` ### Defining the Training Process -Each mini-batch calculates the loss and gradient through forward and backward training, and uses mini_steps to control the accumulated times before each parameter update. After the number of accumulation times is reached, the parameter is updated and the accumulated gradient variable is cleared. +Each mini-batch calculates the loss and gradient through forward and backward training, and uses mini_steps to control the accumulated times before each parameter update. After the number of accumulation times is reached, the parameter is updated and the accumulated gradient variable is cleared. ```python class GradientAccumulation: @@ -202,6 +205,7 @@ class GradientAccumulation: ``` ### Training and Saving the Model + Call the network, optimizer, and loss function, and then customize the `train_process` API of `GradientAccumulation` to train the model. ```python @@ -226,18 +230,20 @@ if __name__ == "__main__": ``` ## Experiment Result + After 10 epochs, the accuracy on the test set is about 96.31%. -**Training Execution** +**Training Execution:** + 1. Run the training code and view the running result. - ```shell - $ python train.py --data_path=./MNIST_Data + ```bash + python train.py --data_path=./MNIST_Data ``` The output is as follows. The loss value decreases during training. - ```shell + ```text epoch: 1 step: 27 loss is 0.3660637 epoch: 1 step: 28 loss is 0.25238192 ... @@ -252,17 +258,17 @@ After 10 epochs, the accuracy on the test set is about 96.31%. The model file `gradient_accumulation.ckpt` is saved during training. -**Model Validation** +**Model Validation:** Use the saved checkpoint file to load the validation dataset through [eval.py]() in the lenet directory of model_zoo. -```shell -$ python eval.py --data_path=./MNIST_Data --ckpt_path=./gradient_accumulation.ckpt --device_target=GPU +```bash +python eval.py --data_path=./MNIST_Data --ckpt_path=./gradient_accumulation.ckpt --device_target=GPU ``` The output is as follows. The accuracy of the validation dataset is about 96.31%, which is the same as the result when the value of batch_size is 32. -```shell +```text ============== Starting Testing ============== ============== {'Accuracy': 0.9631730769230769} ============== -``` \ No newline at end of file +``` diff --git a/tutorials/training/source_en/advanced_use/apply_host_device_training.md b/tutorials/training/source_en/advanced_use/apply_host_device_training.md index 1dfbfa88e44a3345d497c0ddcacc7934b447ffd4..e1f3d69be7c5858bbf527f60f5dfbcb2a7e3d7f7 100644 --- a/tutorials/training/source_en/advanced_use/apply_host_device_training.md +++ b/tutorials/training/source_en/advanced_use/apply_host_device_training.md @@ -18,13 +18,14 @@ In deep learning, one usually has to deal with the huge model problem, in which the total size of parameters in the model is beyond the device memory capacity. To efficiently train a huge model, one solution is to employ homogenous accelerators (*e.g.*, Ascend 910 AI Accelerator and GPU) for distributed training. When the size of a model is hundreds of GBs or several TBs, the number of required accelerators is too overwhelming for people to access, resulting in this solution inapplicable. One alternative is Host+Device hybrid training. This solution simultaneously leveraging the huge memory in hosts and fast computation in accelerators, is a promisingly -efficient method for addressing huge model problem. +efficient method for addressing huge model problem. In MindSpore, users can easily implement hybrid training by configuring trainable parameters and necessary operators to run on hosts, and other operators to run on accelerators. This tutorial introduces how to train [Wide&Deep](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/recommend/wide_and_deep) in the Host+Ascend 910 AI Accelerator mode. + ## Preliminaries -1. Prepare the model. The Wide&Deep code can be found at: , in which `train_and_eval_auto_parallel.py` is the main function for training, +1. Prepare the model. The Wide&Deep code can be found at: , in which `train_and_eval_auto_parallel.py` is the main function for training, `src/` directory contains the model definition, data processing and configuration files, `script/` directory contains the launch scripts in different modes. 2. Prepare the dataset. The dataset can be found at: . Use the script `src/preprocess_data.py` to transform dataset into MindRecord format. @@ -50,19 +51,23 @@ This tutorial introduces how to train [Wide&Deep](https://gitee.com/mindspore/mi ## Configuring for Hybrid Training 1. Configure the flag of hybrid training. In the function `argparse_init` of file `src/config.py`, change the default value of `host_device_mix` to be `1`; change `self.host_device_mix` in function `__init__` of `class WideDeepConfig` to be `1`: + ```python self.host_device_mix = 1 ``` 2. Check placement of necessary operators and optimizers. In class `WideDeepModel` of file `src/wide_and_deep.py`, check the placement of `EmbeddingLookup` is at host: + ```python self.deep_embeddinglookup = nn.EmbeddingLookup() self.wide_embeddinglookup = nn.EmbeddingLookup() ``` + In `class TrainStepWrap(nn.Cell)` of file `src/wide_and_deep.py`, check two optimizer are also at host: + ```python - self.optimizer_w.sparse_opt.add_prim_attr("primitive_target", "CPU") - self.optimizer_d.sparse_opt.add_prim_attr("primitive_target", "CPU") + self.optimizer_w.target = "CPU" + self.optimizer_d.target = "CPU" ``` ## Training the Model @@ -73,7 +78,7 @@ and `RANK_TABLE_FILE` is the path of the above `rank_table_1p_0.json` file. The running log is in the directory of `device_0`, where `loss.log` contains every loss value of every step in the epoch. Here is an example: -``` +```text epoch: 1 step: 1, wide_loss is 0.6873926, deep_loss is 0.8878349 epoch: 1 step: 2, wide_loss is 0.6442529, deep_loss is 0.8342661 epoch: 1 step: 3, wide_loss is 0.6227323, deep_loss is 0.80273706 @@ -90,7 +95,7 @@ epoch: 1 step: 10, wide_loss is 0.566089, deep_loss is 0.6884129 `test_deep0.log` contains the runtime log (This needs to adjust the log level to INFO, and add the `-p on` option when compiling MindSpore). Search `EmbeddingLookup` in `test_deep0.log`, the following can be found: -``` +```text [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:34.928.275 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/network-VirtualDatasetCellTriple/_backbone-NetWithLossClass/network-WideDeepModel/EmbeddingLookup-op297 costs 3066 us. [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:34.943.896 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/network-VirtualDatasetCellTriple/_backbone-NetWithLossClass/network-WideDeepModel/EmbeddingLookup-op298 costs 15521 us. ``` @@ -99,7 +104,7 @@ showing the running time of `EmbeddingLookup` on the host. Search `FusedSparseFtrl` and `FusedSparseLazyAdam` in `test_deep0.log`, the following can be found: -``` +```text [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:35.422.963 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/optimizer_w-FTRL/FusedSparseFtrl-op299 costs 54492 us. [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:35.565.953 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/optimizer_d-LazyAdam/FusedSparseLazyAdam-op300 costs 142865 us. ``` diff --git a/tutorials/training/source_en/advanced_use/apply_parameter_server_training.md b/tutorials/training/source_en/advanced_use/apply_parameter_server_training.md index 50acc0f46e5625a44b1dc590e3bbb66a78cc97db..1161cf1fd005946a0f4b796e39d393c7ed9bace3 100644 --- a/tutorials/training/source_en/advanced_use/apply_parameter_server_training.md +++ b/tutorials/training/source_en/advanced_use/apply_parameter_server_training.md @@ -1,4 +1,4 @@ -# Training with Parameter Server +# Training with Parameter Server `Linux` `Ascend` `GPU` `Model Training` `Intermediate` `Expert` @@ -17,20 +17,21 @@ ## Overview + A parameter server is a widely used architecture in distributed training. Compared with the synchronous AllReduce training method, a parameter server has better flexibility, scalability, and node failover capabilities. Specifically, the parameter server supports both synchronous and asynchronous SGD training algorithms. In terms of scalability, model computing and update are separately deployed in the worker and server processes, so that resources of the worker and server can be independently scaled out and in horizontally. In addition, in an environment of a large-scale data center, various failures often occur in a computing device, a network, and a storage device, and consequently some nodes are abnormal. However, in an architecture of a parameter server, such a failure can be relatively easily handled without affecting a training job. In the parameter server implementation of MindSpore, the open-source [ps-lite](https://github.com/dmlc/ps-lite) is used as the basic architecture. Based on the remote communication capability provided by the [ps-lite](https://github.com/dmlc/ps-lite) and abstract Push/Pull primitives, the distributed training algorithm of the synchronous SGD is implemented. In addition, with the high-performance collective communication library in Ascend and GPU(HCCL and NCCL), MindSpore also provides the hybrid training mode of parameter server and AllReduce. Some weights can be stored and updated through the parameter server, and other weights are still trained through the AllReduce algorithm. The ps-lite architecture consists of three independent components: server, worker, and scheduler. Their functions are as follows: -- Server: saves model weights and backward computation gradients, and updates the model using gradients pushed by workers. +- Server: saves model weights and backward computation gradients, and updates the model using gradients pushed by workers. - Worker: performs forward and backward computation on the network. The gradient value for backward computation is uploaded to a server through the `Push` API, and the model updated by the server is downloaded to the worker through the `Pull` API. - Scheduler: establishes the communication relationship between the server and worker. - ## Preparations + The following describes how to use parameter server to train LeNet on Ascend 910: ### Training Script Preparation @@ -41,17 +42,18 @@ Learn how to train a LeNet using the [MNIST dataset](http://yann.lecun.com/exdb/ 1. First of all, Use `mindspore.context.set_ps_context(enable_ps=True)` to enable Parameter Server training mode. -- This method should be called before `mindspore.communication.management.init()`. -- If you don't call this method, the [Environment Variable Setting](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/apply_parameter_server_training.html#environment-variable-setting) below will not take effect. -- Use `mindspore.context.reset_ps_context()` to disable Parameter Server training mode. + - This method should be called before `mindspore.communication.management.init()`. + - If you don't call this method, the [Environment Variable Setting](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/apply_parameter_server_training.html#environment-variable-setting) below will not take effect. + - Use `mindspore.context.reset_ps_context()` to disable Parameter Server training mode. 2. In this training mode, you can use either of the following methods to control whether the training parameters are updated by the Parameter Server: -- Use `mindspore.nn.Cell.set_param_ps()` to set all weight recursions of `nn.Cell`. -- Use `mindspore.common.Parameter.set_param_ps()` to set the weight. -- The size of the weight which is updated by Parameter Server should not exceed INT_MAX(2^31 - 1) bytes. + - Use `mindspore.nn.Cell.set_param_ps()` to set all weight recursions of `nn.Cell`. + - Use `mindspore.common.Parameter.set_param_ps()` to set the weight. + - The size of the weight which is updated by Parameter Server should not exceed INT_MAX(2^31 - 1) bytes. 3. On the basis of the [original training script](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/lenet/train.py), set all LeNet model weights to be trained on the parameter server: + ```python context.set_ps_context(enable_ps=True) network = LeNet5(cfg.num_classes) @@ -62,7 +64,7 @@ network.set_param_ps() MindSpore reads environment variables to control parameter server training. The environment variables include the following options (all scripts of `MS_SCHED_HOST` and `MS_SCHED_PORT` must be consistent): -``` +```bash export PS_VERBOSE=1 # Print ps-lite log export MS_SERVER_NUM=1 # Server number export MS_WORKER_NUM=1 # Worker number @@ -78,6 +80,7 @@ export MS_ROLE=MS_SCHED # The role of this process: MS_SCHED repre Provide the shell scripts corresponding to the worker, server, and scheduler roles to start training: `Scheduler.sh`: + ```bash #!/bin/bash export PS_VERBOSE=1 @@ -90,6 +93,7 @@ export MS_ROLE=MS_SCHED # The role of this process: MS_SCHED repre ``` `Server.sh`: + ```bash #!/bin/bash export PS_VERBOSE=1 @@ -102,6 +106,7 @@ export MS_ROLE=MS_SCHED # The role of this process: MS_SCHED repre ``` `Worker.sh`: + ```bash #!/bin/bash export PS_VERBOSE=1 @@ -114,26 +119,31 @@ export MS_ROLE=MS_SCHED # The role of this process: MS_SCHED repre ``` Run the following commands separately: + ```bash sh Scheduler.sh > scheduler.log 2>&1 & sh Server.sh > server.log 2>&1 & sh Worker.sh > worker.log 2>&1 & ``` + Start training. 2. Viewing result Run the following command to view the communication logs between the server and worker in the `scheduler.log` file: - ``` + + ```text Bind to role=scheduler, id=1, ip=XXX.XXX.XXX.XXX, port=XXXX Assign rank=8 to node role=server, ip=XXX.XXX.XXX.XXX, port=XXXX Assign rank=9 to node role=worker, ip=XXX.XXX.XXX.XXX, port=XXXX the scheduler is connected to 1 workers and 1 servers ``` + The preceding information indicates that the communication between the server, worker, and scheduler is established successfully. Check the training result in the `worker.log` file: - ``` + + ```text epoch: 1 step: 1, loss is 2.302287 epoch: 1 step: 2, loss is 2.304071 epoch: 1 step: 3, loss is 2.308778 diff --git a/tutorials/training/source_en/advanced_use/apply_quantization_aware_training.md b/tutorials/training/source_en/advanced_use/apply_quantization_aware_training.md index 6868003cbb8b6e815ac79df7d45bbd2b39294ceb..1e18ce380d6296ca8f10dadfebd6bc89180158fd 100644 --- a/tutorials/training/source_en/advanced_use/apply_quantization_aware_training.md +++ b/tutorials/training/source_en/advanced_use/apply_quantization_aware_training.md @@ -39,6 +39,7 @@ Currently, there are two types of quantization solutions in the industry: quanti ### Fake Quantization Node A fake quantization node is a node inserted during quantization aware training, and is used to search for network data distribution and feed back a lost accuracy. The specific functions are as follows: + - Find the distribution of network data, that is, find the maximum and minimum values of the parameters to be quantized. - Simulate the accuracy loss of low-bit quantization, apply the loss to the network model, and transfer the loss to the loss function, so that the optimizer optimizes the loss value during training. @@ -99,7 +100,7 @@ class LeNet5(nn.Cell): Tensor, output tensor Examples: >>> LeNet(num_class=10, num_channel=1) - + """ def __init__(self, num_class=10, num_channel=1): super(LeNet5, self).__init__() @@ -129,10 +130,10 @@ class LeNet5(nn.Cell): def __init__(self, num_class=10): super(LeNet5, self).__init__() self.num_class = num_class - + self.conv1 = nn.Conv2dBnAct(1, 6, kernel_size=5, activation='relu') self.conv2 = nn.Conv2dBnAct(6, 16, kernel_size=5, activation='relu') - + self.fc1 = nn.DenseBnAct(16 * 5 * 5, 120, activation='relu') self.fc2 = nn.DenseBnAct(120, 84, activation='relu') self.fc3 = nn.DenseBnAct(84, self.num_class) @@ -164,17 +165,17 @@ net = quant.convert_quant_network(network, quant_delay=900, bn_fold=False, per_c The preceding describes the quantization aware training from scratch. A more common case is that an existing model file needs to be converted to a quantization model. The model file and training script obtained through common network model training are available for quantization aware training. To use a checkpoint file for retraining, perform the following steps: - 1. Process data and load datasets. - 2. Define an original unquantative network. - 3. Train the original network to generate a unquantative model. - 4. Define a fusion network. - 5. Define an optimizer and loss function. - 6. Generate a quantative network based on the fusion network. - 7. Load a model file and retrain the model. Load the unquantative model file generated in step 3 and retrain the quantative model based on the quantative network to generate a quantative model. For details, see . + 1. Process data and load datasets. + 2. Define an original unquantative network. + 3. Train the original network to generate a unquantative model. + 4. Define a fusion network. + 5. Define an optimizer and loss function. + 6. Generate a quantative network based on the fusion network. + 7. Load a model file and retrain the model. Load the unquantative model file generated in step 3 and retrain the quantative model based on the quantative network to generate a quantative model. For details, see . ### Inference -The inference using a quantization model is the same the common model inference. The inference can be performed by directly using the checkpoint file or converting the checkpoint file into a common model format (such as AIR or MINDIR). +The inference using a quantization model is the same the common model inference. The inference can be performed by directly using the checkpoint file or converting the checkpoint file into a common model format (such as AIR or MINDIR). For details, see . @@ -183,7 +184,7 @@ For details, see - [Converting Dataset to MindRecord](#converting-dataset-to-mindrecord) - - [Overview](#overview) - - [Basic Concepts](#basic-concepts) - - [Converting Dataset to MindRecord](#converting-dataset-to-mindrecord-1) - - [Loading MindRecord Dataset](#loading-mindrecord-dataset) + - [Overview](#overview) + - [Basic Concepts](#basic-concepts) + - [Converting Dataset to MindRecord](#converting-dataset-to-mindrecord-1) + - [Loading MindRecord Dataset](#loading-mindrecord-dataset) @@ -19,6 +19,7 @@ Users can convert non-standard datasets and common datasets into the MindSpore data format, MindRecord, so that they can be easily loaded to MindSpore for training. In addition, the performance of MindSpore in some scenarios is optimized, which delivers better user experience when you use datasets in the MindSpore data format. The MindSpore data format has the following features: + 1. Unified storage and access of user data are implemented, simplifying training data loading. 2. Data is aggregated for storage, which can be efficiently read, managed and moved. 3. Data encoding and decoding are efficient and transparent to users. @@ -96,7 +97,7 @@ The following tutorial demonstrates how to convert image data and its annotation 5. Create a `FileWriter` object, transfer the file name and number of slices, add the schema and index, call the `write_raw_data` API to write data, and call the `commit` API to generate a local data file. - ```python + ```python writer = FileWriter(file_name="test.mindrecord", shard_num=4) writer.add_schema(cv_schema_json, "test_schema") writer.add_index(indexes) @@ -141,7 +142,7 @@ The following tutorial briefly demonstrates how to load the MindRecord dataset u The output is as follows: - ``` + ```text sample: {'data': array([175, 175, 85, 60, 184, 124, 54, 189, 125, 193, 153, 91, 234, 106, 43, 143, 132, 211, 204, 160, 44, 105, 187, 185, 45, 205, 122, 236, 112, 123, 84, 177, 219], dtype=uint8), 'file_name': array(b'3.jpg', dtype='|S5'), 'label': array(99, dtype=int32)} diff --git a/tutorials/training/source_en/advanced_use/custom_debugging_info.md b/tutorials/training/source_en/advanced_use/custom_debugging_info.md index 201968f5804065cb24db3f6f2fdb7232505a0848..96331030f21b7dc2215ece5a3fd5e03bcc936731 100644 --- a/tutorials/training/source_en/advanced_use/custom_debugging_info.md +++ b/tutorials/training/source_en/advanced_use/custom_debugging_info.md @@ -24,7 +24,7 @@ This section describes how to use the customized capabilities provided by MindSpore, such as `callback`, `metrics`, `Print` operators and log printing, to help you quickly debug the training network. -## Introduction to Callback +## Introduction to Callback Here, callback is not a function but a class. You can use callback to observe the internal status and related information of the network during training or perform specific actions in a specific period. For example, you can monitor the loss, save model parameters, dynamically adjust parameters, and terminate training tasks in advance. @@ -39,7 +39,7 @@ MindSpore provides the callback capabilities to allow users to insert customized Usage: Transfer the callback object in the `model.train` method. The callback object can be a list, for example: ```python -ckpt_cb = ModelCheckpoint() +ckpt_cb = ModelCheckpoint() loss_cb = LossMonitor() summary_cb = SummaryCollector(summary_dir='./summary_dir') model.train(epoch, dataset, callbacks=[ckpt_cb, loss_cb, summary_cb]) @@ -58,7 +58,7 @@ The callback base class is defined as follows: ```python class Callback(): - """Callback base class""" + """Callback base class""" def begin(self, run_context): """Called once before the network executing.""" pass @@ -68,11 +68,11 @@ class Callback(): pass def epoch_end(self, run_context): - """Called after each epoch finished.""" + """Called after each epoch finished.""" pass def step_begin(self, run_context): - """Called before each epoch beginning.""" + """Called before each epoch beginning.""" pass def step_end(self, run_context): @@ -129,7 +129,7 @@ Here are two examples to further explain the usage of custom Callback. The output is as follows: - ``` + ```text epoch: 20 step: 32 loss: 2.298344373703003 ``` @@ -221,12 +221,16 @@ print('Accuracy is ', accuracy) ``` The output is as follows: -``` + +```text Accuracy is 0.6667 ``` + ## MindSpore Print Operator -MindSpore-developed `Print` operator is used to print the tensors or character strings input by users. Multiple strings, multiple tensors, and a combination of tensors and strings are supported, which are separated by comma (,). -The method of using the MindSpore `Print` operator is the same as using other operators. You need to assert MindSpore `Print` operator in `__init__` and invoke it using `construct`. The following is an example. + +MindSpore-developed `Print` operator is used to print the tensors or character strings input by users. Multiple strings, multiple tensors, and a combination of tensors and strings are supported, which are separated by comma (,). +The method of using the MindSpore `Print` operator is the same as using other operators. You need to assert MindSpore `Print` operator in `__init__` and invoke it using `construct`. The following is an example. + ```python import numpy as np from mindspore import Tensor @@ -250,8 +254,10 @@ y = Tensor(np.ones([2, 2]).astype(np.int32)) net = PrintDemo() output = net(x, y) ``` + The output is as follows: -``` + +```text print Tensor x and Tensor y: Tensor shape:[[const vector][2, 1]]Int32 val:[[1] @@ -313,7 +319,7 @@ The input and output of the operator can be saved for debugging through the data You can set `context.set_context(reserve_class_name_in_scope=False)` in your training script to avoid dump failure because of file name is too long. 4. Parse the Dump file. - + Call `numpy.fromfile` to parse dump data file. ### Asynchronous Dump @@ -321,6 +327,7 @@ The input and output of the operator can be saved for debugging through the data 1. Create dump json file:`data_dump.json`. The name and location of the JSON file can be customized. + ```json { "common_dump_settings": { @@ -369,30 +376,31 @@ The input and output of the operator can be saved for debugging through the data ``` ## Log-related Environment Variables and Configurations + MindSpore uses glog to output logs. The following environment variables are commonly used: - `GLOG_v` - - The environment variable specifies the log level. + + The environment variable specifies the log level. The default value is 2, indicating the WARNING level. The values are as follows: 0: DEBUG; 1: INFO; 2: WARNING; 3: ERROR. -- `GLOG_logtostderr` +- `GLOG_logtostderr` The environment variable specifies the log output mode. When `GLOG_logtostderr` is set to 1, logs are output to the screen. If the value is set to 0, logs are output to a file. The default value is 1. - `GLOG_log_dir` - - The environment variable specifies the log output path. - If `GLOG_logtostderr` is set to 0, value of this variable must be specified. - If `GLOG_log_dir` is specified and the value of `GLOG_logtostderr` is 1, logs are output to the screen but not to a file. + + The environment variable specifies the log output path. + If `GLOG_logtostderr` is set to 0, value of this variable must be specified. + If `GLOG_log_dir` is specified and the value of `GLOG_logtostderr` is 1, logs are output to the screen but not to a file. Logs of C++ and Python will be output to different files. The file name of C++ log complies with the naming rule of `GLOG` log file. Here, the name is `mindspore.MachineName.UserName.log.LogLevel.Timestamp`. The file name of Python log is `mindspore.log`. -- `MS_SUBMODULE_LOG_v` +- `MS_SUBMODULE_LOG_v` The environment variable specifies log levels of C++ sub modules of MindSpore. - The environment variable is assigned as: `MS_SUBMODULE_LOG_v="{SubModule1:LogLevel1,SubModule2:LogLevel2,...}"`. - The specified sub module log level will overwrite the global log level. The meaning of sub module log level is the same as `GLOG_v`, the sub modules of MindSpore are categorized by source directory is shown in the below table. + The environment variable is assigned as: `MS_SUBMODULE_LOG_v="{SubModule1:LogLevel1,SubModule2:LogLevel2,...}"`. + The specified sub module log level will overwrite the global log level. The meaning of sub module log level is the same as `GLOG_v`, the sub modules of MindSpore are categorized by source directory is shown in the below table. E.g. when set `GLOG_v=1 MS_SUBMODULE_LOG_v="{PARSER:2,ANALYZER:2}"` then log levels of `PARSER` and `ANALYZER` are WARNING, other modules' log levels are INFO. Sub modules of MindSpore grouped by source directory: @@ -424,4 +432,4 @@ Sub modules of MindSpore grouped by source directory: | mindspore/core/gvar | COMMON | | mindspore/core/ | CORE | -> The glog does not support log rotate. To control the disk space occupied by log files, use the log file management tool provided by the operating system, such as: logrotate of Linux. \ No newline at end of file +> The glog does not support log rotate. To control the disk space occupied by log files, use the log file management tool provided by the operating system, such as: logrotate of Linux. diff --git a/tutorials/training/source_en/advanced_use/custom_operator_ascend.md b/tutorials/training/source_en/advanced_use/custom_operator_ascend.md index c205cff59a5ed61e6cf2b5d6d2be5bea9b178034..ec197f4a58d0310adbe4321378fb7ed5d1c0d7a1 100644 --- a/tutorials/training/source_en/advanced_use/custom_operator_ascend.md +++ b/tutorials/training/source_en/advanced_use/custom_operator_ascend.md @@ -25,6 +25,7 @@ When built-in operators cannot meet requirements during network development, you To add a custom operator, you need to register the operator primitive, implement the operator, and register the operator information. The related concepts are as follows: + - Operator primitive: defines the frontend API prototype of an operator on the network. It is the basic unit for forming a network model and includes the operator name, attribute (optional), input and output names, output shape inference method, and output dtype inference method. - Operator implementation: describes the implementation of the internal computation logic for an operator through the DSL API provided by the Tensor Boost Engine (TBE). The TBE supports the development of custom operators based on the Ascend AI chip. You can apply for Open Beta Tests (OBTs) by visiting . - Operator information: describes basic information about a TBE operator, such as the operator name and supported input and output types. It is the basis for the backend to select and map operators. @@ -38,6 +39,7 @@ This section takes a Square operator as an example to describe how to customize The primitive of an operator is a subclass inherited from `PrimitiveWithInfer`. The type name of the subclass is the operator name. The definition of the custom operator primitive is the same as that of the built-in operator primitive. + - The attribute is defined by the input parameter of the constructor function `__init__`. The operator in this test case has no attribute. Therefore, `__init__` has only one input parameter. For details about test cases in which operators have attributes, see [custom add3](https://gitee.com/mindspore/mindspore/blob/master/tests/st/ops/custom_ops_tbe/cus_add3.py) in the MindSpore source code. - The input and output names are defined by the `init_prim_io_names` function. - The shape inference method of the output tensor is defined in the `infer_shape` function, and the dtype inference method of the output tensor is defined in the `infer_dtype` function. @@ -75,10 +77,12 @@ To compile an operator implementation, you need to compile a computable function The computable function of an operator is mainly used to encapsulate the computation logic of the operator for the main function to call. The computation logic is implemented by calling the combined API of the TBE. The entry function of an operator describes the internal process of compiling the operator. The process is as follows: + 1. Prepare placeholders to be input. A placeholder will return a tensor object that represents a group of input data. 2. Call the computable function. The computable function uses the API provided by the TBE to describe the computation logic of the operator. 3. Call the scheduling module. The model tiles the operator data based on the scheduling description and specifies the data transfer process to ensure optimal hardware execution. By default, the automatic scheduling module (`auto_schedule`) can be used. 4. Call `cce_build_code` to compile and generate an operator binary file. + > The input parameters of the entry function require the input information of each operator, output information of each operator, operator attributes (optional), and `kernel_name` (name of the generated operator binary file). The input and output information is encapsulated in dictionaries, including the input and output shape and dtype when the operator is called on the network. For details about TBE operator development, visit the [TBE website](https://support.huaweicloud.com/odevg-A800_3000_3010/atlaste_10_0063.html). For details about how to debug and optimize the TBE operator, visit the [Mind Studio website](https://support.huaweicloud.com/usermanual-mindstudioc73/atlasmindstudio_02_0043.html). @@ -88,12 +92,12 @@ For details about TBE operator development, visit the [TBE website](https://supp The operator information is key for the backend to select the operator implementation and guides the backend to insert appropriate type and format conversion operators. It uses the `TBERegOp` API for definition and uses the `op_info_register` decorator to bind the operator information to the entry function of the operator implementation. When the .py operator implementation file is imported, the `op_info_register` decorator registers the operator information to the operator information library at the backend. For details about how to use the operator information, see comments for the member method of `TBERegOp`. > The numbers and sequences of the input and output information defined in the operator information must be the same as those in the parameters of the entry function of the operator implementation and those listed in the operator primitive. - +> > If an operator has attributes, use `attr` to describe the attribute information in the operator information. The attribute names must be the same as those in the operator primitive definition. ### Example -The following takes the TBE implementation `square_impl.py` of the `Square` operator as an example. `square_compute` is a computable function of the operator implementation. It describes the computation logic of `x * x` by calling the API provided by `te.lang.cce`. `cus_square_op_info ` is the operator information, which is defined by `TBERegOp`. For the specific field meaning of the operator information, visit the [TBE website](https://support.huaweicloud.com/odevg-A800_3000_3010/atlaste_10_0096.html). +The following takes the TBE implementation `square_impl.py` of the `Square` operator as an example. `square_compute` is a computable function of the operator implementation. It describes the computation logic of `x * x` by calling the API provided by `te.lang.cce`. `cus_square_op_info` is the operator information, which is defined by `TBERegOp`. For the specific field meaning of the operator information, visit the [TBE website](https://support.huaweicloud.com/odevg-A800_3000_3010/atlaste_10_0096.html). Note the following parameters when setting `TBERegOp`: @@ -128,7 +132,7 @@ cus_square_op_info = TBERegOp("CusSquare") \ .output(0, "y", False, "required", "all") \ .dtype_format(DataType.F32_Default, DataType.F32_Default) \ .dtype_format(DataType.F16_Default, DataType.F16_Default) \ - .get_op_info() + .get_op_info() # Binding kernel info with the kernel implementation. @op_info_register(cus_square_op_info) @@ -185,17 +189,20 @@ def test_net(): ``` Execute the test case. -``` + +```bash pytest -s tests/st/ops/custom_ops_tbe/test_square.py::test_net ``` The execution result is as follows: -``` + +```text x: [1. 4. 9.] output: [1. 16. 81.] ``` ## Defining the bprop Function for an Operator + If an operator needs to support automatic differentiation, the bprop function needs to be defined in the primitive of the operator. In the bprop function, you need to describe the backward computation logic that uses the forward input, forward output, and output gradients to obtain the input gradients. The backward computation logic can be composed of built-in operators or custom backward operators. Note the following points when defining the bprop function: @@ -204,6 +211,7 @@ Note the following points when defining the bprop function: - The return value of the bprop function is tuples consisting of input gradients. The sequence of elements in a tuple is the same as that of the forward input parameters. Even if there is only one input gradient, the return value must be a tuple. For example, the `CusSquare` primitive after the bprop function is added is as follows: + ```python class CusSquare(PrimitiveWithInfer): @prim_attr_register @@ -228,6 +236,7 @@ class CusSquare(PrimitiveWithInfer): ``` Define backward cases in the `test_square.py` file. + ```python from mindspore.ops import composite as C def test_grad_net(): @@ -241,12 +250,14 @@ def test_grad_net(): ``` Execute the test case. -``` + +```bash pytest -s tests/st/ops/custom_ops_tbe/test_square.py::test_grad_net ``` The execution result is as follows: -``` + +```text x: [1. 4. 9.] dx: [2. 8. 18.] ``` diff --git a/tutorials/training/source_en/advanced_use/cv_resnet50.md b/tutorials/training/source_en/advanced_use/cv_resnet50.md index 6c59dcbbcf0ee5f11745416c9221d0c8ee879a03..a4b80602073d479d1cd86b6f228b10a9468a4ee0 100644 --- a/tutorials/training/source_en/advanced_use/cv_resnet50.md +++ b/tutorials/training/source_en/advanced_use/cv_resnet50.md @@ -26,8 +26,8 @@ Computer vision is one of the most widely researched and mature technology field This chapter describes how to apply MindSpore to computer vision scenarios based on image classification tasks. - ## Image Classification + Image classification is one of the most basic computer vision applications and belongs to the supervised learning category. For example, determine the class of a digital image, such as cat, dog, airplane, or car. The function is as follows: ```python @@ -42,7 +42,6 @@ MindSpore presets a typical CNN, developer can visit [model_zoo](https://gitee.c MindSpore supports the following image classification networks: LeNet, AlexNet, and ResNet. - ## Task Description and Preparation ![cifar10](images/cifar10.jpg) @@ -54,6 +53,7 @@ The CIFAR-10 dataset contains 10 classes of 60,000 images. Each class contains 6 Generally, a training indicator of image classification is accuracy, that is, a ratio of the quantity of accurately predicted examples to the total quantity of predicted examples. Next, let's use MindSpore to solve the image classification task. The overall process is as follows: + 1. Download the CIFAR-10 dataset. 2. Load and preprocess data. 3. Define a convolutional neural network. In this example, the ResNet-50 network is used. @@ -66,6 +66,7 @@ Next, let's use MindSpore to solve the image classification task. The overall pr The key parts of the task process code are explained below. ### Downloading the CIFAR-10 Dataset + CIFAR-10 dataset download address: [the website of Cifar-10 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html). In this example, the data is in binary format. In the Linux environment, run the following command to download the dataset: ```shell @@ -78,7 +79,6 @@ Run the following command to decompress the dataset: tar -zvxf cifar-10-binary.tar.gz ``` - ### Data Preloading and Preprocessing 1. Load the dataset. @@ -86,10 +86,8 @@ tar -zvxf cifar-10-binary.tar.gz Data can be loaded through the built-in dataset format `Cifar10Dataset` API. > `Cifar10Dataset`: The read type is random read. The built-in CIFAR-10 dataset contains images and labels. The default image format is uint8, and the default label data format is uint32. For details, see the description of the `Cifar10Dataset` API. - The data loading code is as follows, where `data_home` indicates the data storage location: - ```python cifar_ds = ds.Cifar10Dataset(data_home) ``` @@ -138,7 +136,6 @@ tar -zvxf cifar-10-binary.tar.gz cifar_ds = cifar_ds.repeat(repeat_num) ``` - ### Defining the CNN CNN is a standard algorithm for image classification tasks. CNN uses a layered structure to perform feature extraction on an image, and is formed by stacking a series of network layers, such as a convolutional layer, a pooling layer, and an activation layer. @@ -153,10 +150,8 @@ network = resnet50(class_num=10) For more information about ResNet, see [ResNet Paper](https://arxiv.org/abs/1512.03385). - ### Defining the Loss Function and Optimizer - A loss function and an optimizer need to be defined. The loss function is a training objective of the deep learning, and is also referred to as an objective function. The loss function indicates the distance between a logit of a neural network and a label, and is scalar data. Common loss functions include mean square error, L2 loss, Hinge loss, and cross entropy. Cross entropy is usually used for image classification. @@ -173,7 +168,6 @@ ls = SoftmaxCrossEntropyWithLogits(sparse=True, reduction="mean") opt = Momentum(filter(lambda x: x.requires_grad, net.get_parameters()), 0.01, 0.9) ``` - ### Calling the High-level `Model` API To Train and Save the Model File After data preprocessing, network definition, and loss function and optimizer definition are complete, model training can be performed. Model training involves two iterations: multi-round iteration (`epoch`) of datasets and single-step iteration based on the batch size of datasets. The single-step iteration refers to extracting data from a dataset by `batch`, inputting the data to a network to calculate a loss function, and then calculating and updating a gradient of training parameters by using an optimizer. @@ -210,8 +204,6 @@ res = model.eval(eval_dataset) print("result: ", res) ``` - - ## References -[1] https://www.cs.toronto.edu/~kriz/cifar.html +[1] diff --git a/tutorials/training/source_en/advanced_use/cv_resnet50_second_order_optimizer.md b/tutorials/training/source_en/advanced_use/cv_resnet50_second_order_optimizer.md index b91878138c5d3f4ffd388becda17a9a9ab4de34c..8f2919d643678cb02f2ba20c2d6f23669446945c 100644 --- a/tutorials/training/source_en/advanced_use/cv_resnet50_second_order_optimizer.md +++ b/tutorials/training/source_en/advanced_use/cv_resnet50_second_order_optimizer.md @@ -37,7 +37,6 @@ Common optimization algorithms are classified into the first-order and the secon Based on the existing natural gradient algorithm, MindSpore development team uses optimized acceleration methods such as approximation and sharding for the FIM, greatly reducing the computation complexity of the inverse matrix and developing the available second-order optimizer THOR. With eight Ascend 910 AI processors, THOR can complete the training of ResNet-50 v1.5 network and ImageNet dataset within 72 minutes, which is nearly twice the speed of SGD+Momentum. - This tutorial describes how to use the second-order optimizer THOR provided by MindSpore to train the ResNet-50 v1.5 network and ImageNet dataset on Ascend 910 and GPU. > Download address of the complete code example: @@ -47,12 +46,12 @@ Directory Structure of Code Examples ```shell ├── resnet_thor ├── README.md - ├── scripts + ├── scripts ├── run_distribute_train.sh # launch distributed training for Ascend 910 └── run_eval.sh # launch inference for Ascend 910 ├── run_distribute_train_gpu.sh # launch distributed training for GPU └── run_eval_gpu.sh # launch inference for GPU - ├── src + ├── src ├── crossentropy.py # CrossEntropy loss function ├── config.py # parameter configuration ├── dataset_helper.py # dataset helper for minddata dataset @@ -61,20 +60,19 @@ Directory Structure of Code Examples ├── resnet_thor.py # resnet50_thor backone ├── thor.py # thor optimizer ├── thor_layer.py # thor layer - └── dataset.py # data preprocessing + └── dataset.py # data preprocessing ├── eval.py # infer script └── train.py # train script - ``` The overall execution process is as follows: + 1. Prepare the ImageNet dataset and process the required dataset. 2. Define the ResNet-50 network. 3. Define the loss function and the optimizer THOR. 4. Load the dataset and perform training. After the training is complete, check the result and save the model file. 5. Load the saved model for inference. - ## Preparation Ensure that MindSpore has been correctly installed. If not, install it by referring to [Install](https://www.mindspore.cn/install/en). @@ -85,7 +83,7 @@ Download the complete ImageNet2012 dataset, decompress the dataset, and save it The directory structure is as follows: -``` +```text └─ImageNet2012 ├─ilsvrc │ n03676483 @@ -97,19 +95,22 @@ The directory structure is as follows: │ n02504013 │ n07871810 │ ...... - ``` + ### Configuring Distributed Environment Variables + #### Ascend 910 + For details about how to configure the distributed environment variables of Ascend 910 AI processors, see [Parallel Distributed Training (Ascend)](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/distributed_training_ascend.html#configuring-distributed-environment-variables). #### GPU -For details about how to configure the distributed environment of GPUs, see [Parallel Distributed Training (GPU)](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/distributed_training_gpu.html#configuring-distributed-environment-variables). +For details about how to configure the distributed environment of GPUs, see [Parallel Distributed Training (GPU)](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/distributed_training_gpu.html#configuring-distributed-environment-variables). ## Loading the Dataset During distributed training, load the dataset in parallel mode and process it through the data argumentation API provided by MindSpore. The `src/dataset.py` script in the source code is for loading and processing the dataset. + ```python import os import mindspore.common.dtype as mstype @@ -165,17 +166,18 @@ def create_dataset(dataset_path, do_train, repeat_num=1, batch_size=32, target=" > MindSpore supports multiple data processing and augmentation operations, which are usually combined. For details, see [Data Processing](https://www.mindspore.cn/tutorial/training/en/master/use/data_preparation.html). - ## Defining the Network + Use the ResNet-50 v1.5 network model as an example. Define the [ResNet-50 network](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/resnet/src/resnet.py), and replace the `Conv2d` and `Dense` operators with the operators customized by the second-order optimizer. The defined network model stores in the `src/resnet_thor.py` script in the source code, and the customized operators `Conv2d_thor` and `Dense_thor` store in the `src/thor_layer.py` script. -- Use `Conv2d_thor` to replace `Conv2d` in the original network model. -- Use `Dense_thor` to replace `Dense` in the original network model. +- Use `Conv2d_thor` to replace `Conv2d` in the original network model. +- Use `Dense_thor` to replace `Dense` in the original network model. > The `Conv2d_thor` and `Dense_thor` operators customized by THOR are used to save the second-order matrix information in model training. The backbone of the newly defined network is the same as that of the original network model. After the network is built, call the defined ResNet-50 in the `__main__` function. + ```python ... from src.resnet_thor import resnet50 @@ -188,15 +190,14 @@ if __name__ == "__main__": ... ``` - ## Defining the Loss Function and Optimizer THOR - ### Defining the Loss Function Loss functions supported by MindSpore include `SoftmaxCrossEntropyWithLogits`, `L1Loss`, and `MSELoss`. The `SoftmaxCrossEntropyWithLogits` loss function is required by THOR. The implementation procedure of the loss function is in the `src/crossentropy.py` script. A common trick in deep network model training, label smoothing, is used to improve the model tolerance to error label classification by smoothing real labels, thereby improving the model generalization capability. + ```python class CrossEntropy(_Loss): """CrossEntropy""" @@ -214,6 +215,7 @@ class CrossEntropy(_Loss): loss = self.mean(loss, 0) return loss ``` + Call the defined loss function in the `__main__` function. ```python @@ -236,6 +238,7 @@ The parameter update formula of THOR is as follows: $$ \theta^{t+1} = \theta^t + \alpha F^{-1}\nabla E$$ The meanings of parameters in the formula are as follows: + - $\theta$: trainable parameters on the network - $t$: number of training steps - $\alpha$: learning rate, which is the parameter update value per step @@ -296,7 +299,6 @@ if __name__ == "__main__": Use the `model.train` API provided by MindSpore to easily train the network. THOR reduces the computation workload and improves the computation speed by reducing the frequency of updating the second-order matrix. Therefore, the Model_Thor class is redefined to inherit the Model class provided by MindSpore. The parameter for controlling the frequency of updating the second-order matrix is added to the Model_Thor class. You can adjust this parameter to optimize the overall performance. - ```python ... from mindspore.train.loss_scale_manager import FixedLossScaleManager @@ -316,15 +318,21 @@ if __name__ == "__main__": ``` ### Running the Script + After the training script is defined, call the shell script in the `scripts` directory to start the distributed training process. + #### Ascend 910 + Currently, MindSpore distributed execution on Ascend uses the single-device single-process running mode. That is, one process runs on a device, and the number of total processes is the same as the number of devices that are being used. For device 0, the corresponding process is executed in the foreground. For other devices, the corresponding processes are executed in the background. Create a directory named `train_parallel`+`device_id` for each process to store log information, operator compilation information, and training checkpoint files. The following takes the distributed training script for eight devices as an example to describe how to run the script: Run the script. -``` + +```bash sh run_distribute_train.sh [RANK_TABLE_FILE] [DATASET_PATH] [DEVICE_NUM] ``` + Variables `RANK_TABLE_FILE`, `DATASET_PATH`, and `DEVICE_NUM` need to be transferred to the script. The meanings of variables are as follows: + - `RANK_TABLE_FILE`: path for storing the networking information file - `DATASET_PATH`: training dataset path - `DEVICE_NUM`: the actual number of running devices. @@ -361,17 +369,22 @@ In the preceding information, `*.ckpt` indicates the saved model parameter file. The name of a checkpoint file is in the following format: *Network name*-*Number of epochs*_*Number of steps*.ckpt. #### GPU + On the GPU hardware platform, MindSpore uses `mpirun` of OpenMPI to perform distributed training. The process creates a directory named `train_parallel` to store log information and training checkpoint files. The following takes the distributed training script for eight devices as an example to describe how to run the script: -``` + +```bash sh run_distribute_train_gpu.sh [DATASET_PATH] [DEVICE_NUM] ``` + Variables `DATASET_PATH` and `DEVICE_NUM` need to be transferred to the script. The meanings of variables are as follows: + - `DATASET_PATH`: training dataset path - `DEVICE_NUM`: the actual number of running devices During GPU-based training, the `DEVICE_ID` environment variable is not required. Therefore, you do not need to call `int(os.getenv('DEVICE_ID'))` in the main training script to obtain the device ID or transfer `device_id` to `context`. You need to set `device_target` to `GPU` and call `init()` to enable the NCCL. The following is an example of loss values output during training: + ```bash ... epoch: 1 step: 5004, loss is 4.2546034 @@ -391,7 +404,7 @@ The following is an example of model files saved after training: ├─ckpt_0 ├─resnet-1_5004.ckpt ├─resnet-2_5004.ckpt - │ ...... + │ ...... ├─resnet-36_5004.ckpt │ ...... ...... @@ -435,40 +448,54 @@ if __name__ == "__main__": # define model model = Model(net, loss_fn=loss, metrics={'top_1_accuracy', 'top_5_accuracy'}) - + # eval model res = model.eval(dataset) print("result:", res, "ckpt=", args_opt.checkpoint_path) ``` ### Inference + After the inference network is defined, the shell script in the `scripts` directory is called for inference. + #### Ascend 910 + On the Ascend 910 hardware platform, run the following inference command: -``` + +```bash sh run_eval.sh [DATASET_PATH] [CHECKPOINT_PATH] ``` + Variables `DATASET_PATH` and `CHECKPOINT_PATH` need to be transferred to the script. The meanings of variables are as follows: + - `DATASET_PATH`: inference dataset path - `CHECKPOINT_PATH`: path for storing the checkpoint file Currently, a single device (device 0 by default) is used for inference. The inference result is as follows: -``` + +```text result: {'top_5_accuracy': 0.9295574583866837, 'top_1_accuracy': 0.761443661971831} ckpt=train_parallel0/resnet-42_5004.ckpt ``` + - `top_5_accuracy`: For an input image, if the labels whose prediction probability ranks top 5 contain actual labels, the classification is correct. -- `top_1_accuracy`: For an input image, if the label with the highest prediction probability is the same as the actual label, the classification is correct. +- `top_1_accuracy`: For an input image, if the label with the highest prediction probability is the same as the actual label, the +classification is correct. + #### GPU On the GPU hardware platform, run the following inference command: -``` + +```bash sh run_eval_gpu.sh [DATASET_PATH] [CHECKPOINT_PATH] ``` + Variables `DATASET_PATH` and `CHECKPOINT_PATH` need to be transferred to the script. The meanings of variables are as follows: + - `DATASET_PATH`: inference dataset path - `CHECKPOINT_PATH`: path for storing the checkpoint file The inference result is as follows: -``` + +```text result: {'top_5_accuracy': 0.9287972151088348, 'top_1_accuracy': 0.7597031049935979} ckpt=train_parallel/resnet-36_5004.ckpt -``` \ No newline at end of file +``` diff --git a/tutorials/training/source_en/advanced_use/dashboard.md b/tutorials/training/source_en/advanced_use/dashboard.md index 739e117a1637cd12eb7d0ab8ba144f8fb5e1b8e6..c2709a946a74f5da0b17ba53e5d9d8c2720ca100 100644 --- a/tutorials/training/source_en/advanced_use/dashboard.md +++ b/tutorials/training/source_en/advanced_use/dashboard.md @@ -170,7 +170,6 @@ Figure 13 shows tensors recorded by a user in a form of a histogram. Click the u ## Notices - 1. Currently MindSpore supports recording computational graph after operator fusion for Ascend 910 AI processor only. 2. When using the Summary operator to collect data in training, 'HistogramSummary' operator will affect performance, so please use as few as possible. @@ -196,6 +195,6 @@ Figure 13 shows tensors recorded by a user in a form of a histogram. Click the u Remarks: The method of estimating the space usage of `TensorSummary` is as follows: - The size of a `TensorSummary` data = the number of values in the tensor * 4 bytes. Assuming that the size of the tensor recorded by `TensorSummary` is 32 * 1 * 256 * 256, then a `TensorSummary` data needs about 32 * 1 * 256 * 256 * 4 bytes = 8,388,608 bytes = 8MiB. `TensorSummary` will record data of 20 steps by default. Then the required space when recording these 20 sets of data is about 20 * 8 MiB = 160MiB. It should be noted that due to the overhead of data structure and other factors, the actual storage space used will be slightly larger than 160MiB. + The size of a `TensorSummary` data = the number of values in the tensor \* 4 bytes. Assuming that the size of the tensor recorded by `TensorSummary` is 32 \* 1 \* 256 \* 256, then a `TensorSummary` data needs about 32 \* 1 \* 256 \* 256 \* 4 bytes = 8,388,608 bytes = 8MiB. `TensorSummary` will record data of 20 steps by default. Then the required space when recording these 20 sets of data is about 20 * 8 MiB = 160MiB. It should be noted that due to the overhead of data structure and other factors, the actual storage space used will be slightly larger than 160MiB. -6. The training log file is large when using `TensorSummary` because the complete tensor data is recorded. MindInsight needs more time to parse the training log file, please be patient. \ No newline at end of file +6. The training log file is large when using `TensorSummary` because the complete tensor data is recorded. MindInsight needs more time to parse the training log file, please be patient. diff --git a/tutorials/training/source_en/advanced_use/debug_in_pynative_mode.md b/tutorials/training/source_en/advanced_use/debug_in_pynative_mode.md index c705b21fc86e377ced59c9f47769ecbf40ec25fc..063c565e79ac0d3664edfc4821f043ff112e664c 100644 --- a/tutorials/training/source_en/advanced_use/debug_in_pynative_mode.md +++ b/tutorials/training/source_en/advanced_use/debug_in_pynative_mode.md @@ -26,7 +26,7 @@ By default, MindSpore is in PyNative mode. You can switch it to the graph mode b In PyNative mode, single operators, common functions, network inference, and separated gradient calculation can be executed. The following describes the usage and precautions. -> In PyNative mode, operators are executed asynchronously on the device to improve performance. Therefore, when an error occurs during operator excution, the error information may be displayed after the program is executed. +> In PyNative mode, operators are executed asynchronously on the device to improve performance. Therefore, when an error occurs during operator excution, the error information may be displayed after the program is executed. ## Executing a Single Operator @@ -73,12 +73,12 @@ Output: [ 0.05016355 0.03958241 0.03958241 0.03958241 0.03443141]]]] ``` - ## Executing a Common Function Combine multiple operators into a function, call the function to execute the operators, and output the result, as shown in the following example: -**Example Code** +**Example Code:** + ```python import numpy as np from mindspore import context, Tensor @@ -97,9 +97,9 @@ output = tensor_add_func(x, y) print(output.asnumpy()) ``` -**Output** +**Output:** -```python +```text [[3. 3. 3.] [3. 3. 3.] [3. 3. 3.]] @@ -107,7 +107,6 @@ print(output.asnumpy()) > Parallel execution and summary are not supported in PyNative mode, so parallel and summary related operators cannot be used. - ### Improving PyNative Performance MindSpore provides the Staging function to improve the execution speed of inference tasks in PyNative mode. This function compiles Python functions or Python class methods into computational graphs in PyNative mode and improves the execution speed by using graph optimization technologies, as shown in the following example: @@ -140,9 +139,10 @@ tensor_add = P.TensorAdd() res = tensor_add(x, z) # PyNative mode print(res.asnumpy()) ``` -**Output** -```python +**Output:** + +```text [[3. 3. 3. 3.] [3. 3. 3. 3.] [3. 3. 3. 3.] @@ -153,7 +153,7 @@ In the preceding code, the `ms_function` decorator is added before `construct` o It should be noted that, in a function to which the `ms_function` decorator is added, if an operator (such as `pooling` or `tensor_add`) that does not need parameter training is included, the operator can be directly called in the decorated function, as shown in the following example: -**Example Code** +**Example Code:** ```python import numpy as np @@ -176,9 +176,10 @@ y = Tensor(np.ones([4, 4]).astype(np.float32)) z = tensor_add_fn(x, y) print(z.asnumpy()) ``` -**Output** -```shell +**Output:** + +```text [[2. 2. 2. 2.] [2. 2. 2. 2.] [2. 2. 2. 2.] @@ -187,7 +188,7 @@ print(z.asnumpy()) If the decorated function contains operators (such as `Convolution` and `BatchNorm`) that require parameter training, these operators must be instantiated before the decorated function is called, as shown in the following example: -**Example Code** +**Example Code:** ```python import numpy as np @@ -209,9 +210,9 @@ z = conv_fn(Tensor(input_data)) print(z.asnumpy()) ``` -**Output** +**Output:** -```shell +```text [[[[ 0.10377571 -0.0182163 -0.05221086] [ 0.1428334 -0.01216263 0.03171652] [-0.00673915 -0.01216291 0.02872104]] @@ -245,12 +246,11 @@ print(z.asnumpy()) [ 0.0377498 -0.06117418 0.00546303]]]] ``` - ## Debugging Network Train Model In PyNative mode, the gradient can be calculated separately. As shown in the following example, `GradOperation` is used to calculate all input gradients of the function or the network. Note that the inputs have to be Tensor. -**Example Code** +**Example Code:** ```python from mindspore.ops import composite as C @@ -267,15 +267,15 @@ def mainf(x, y): print(mainf(Tensor(1, mstype.int32), Tensor(2, mstype.int32))) ``` -**Output** +**Output:** -```python +```text (2, 1) ``` During network training, obtain the gradient, call the optimizer to optimize parameters (the breakpoint cannot be set during the reverse gradient calculation), and calculate the loss values. Then, network training is implemented in PyNative mode. -**Complete LeNet Sample Code** +**Complete LeNet Sample Code:** ```python import numpy as np @@ -312,7 +312,7 @@ class LeNet5(nn.Cell): Lenet network Args: num_class (int): Num classes. Default: 10. - + Returns: Tensor, output tensor @@ -346,8 +346,8 @@ class LeNet5(nn.Cell): x = self.relu(x) x = self.fc3(x) return x - - + + class GradWrap(nn.Cell): """ GradWrap definition """ def __init__(self, network): @@ -376,9 +376,9 @@ loss = loss_output.asnumpy() print(loss) ``` -**Output** +**Output:** -```python +```text 2.3050091 ``` diff --git a/tutorials/training/source_en/advanced_use/debugger.md b/tutorials/training/source_en/advanced_use/debugger.md index 13528eb1f156e387bd8f4ba04b5be4269c3010b6..2000412df467cb5e61a5acbee08a2e32ffbb2a65 100644 --- a/tutorials/training/source_en/advanced_use/debugger.md +++ b/tutorials/training/source_en/advanced_use/debugger.md @@ -29,19 +29,19 @@ In `Graph Mode` training, the computation results of intermediate nodes in the c - Visualize the computational graph on the UI and analyze the output of the graph node; - Set a conditional breakpoint to monitor training exceptions (such as INF), if the condition is met, users can track the cause of the bug when an exception occurs; -- Visualize and analyze the change of parameters, such as weights. +- Visualize and analyze the change of parameters, such as weights. ## Operation Process - Launch MindInsight in debugger mode, and set Debugger environment variables for the training; - At the beginning of the training, set conditional breakpoints; -- Analyze the training progress on MindInsight Debugger UI. +- Analyze the training progress on MindInsight Debugger UI. ## Debugger Environment Preparation At first, install MindInsight and launch it in debugger mode. MindSpore will send training information to MindInsight Debugger Server in debugger mode, users can analyze the information on MindInsight UI. -The command to launch MindInsight in debugger mode is as follows: +The command to launch MindInsight in debugger mode is as follows: ```shell mindinsight start --port {PORT} --enable-debugger True --debugger-port {DEBUGGER_PORT} @@ -67,7 +67,7 @@ Besides, do not use dataset sink mode (Set the parameter `dataset_sink_mode` in ## Debugger UI Introduction -After the Debugger environment preparation, users can run the training script. +After the Debugger environment preparation, users can run the training script. Before the execution of the computational graph, the MindInsight Debugger UI will show the information of the optimized computational graph. The following are the Debugger UI components. @@ -103,22 +103,22 @@ Figure 2: The Graph Node Details When choosing one node on the graph, the details of this node will be displayed at the bottom. The `Tensor Value Overview` area will show the input nodes and the outputs of this node. The `Type`, `Shape` and `Value` of the `Tensor` can also be viewed. -For GPU environment, after selecting an executable node on the graph, right-click to select `Continue to` on this node, -which means running the training script to the selected node within one step. +For GPU environment, after selecting an executable node on the graph, right-click to select `Continue to` on this node, +which means running the training script to the selected node within one step. After left-click `Continue to`, the training script will be executed and paused after running to this node. ![debugger_tensor_value](./images/debugger_tensor_value.png) Figure 3: `Tensor` Value Visualization -Some outputs of the node contain too many dimensions. +Some outputs of the node contain too many dimensions. For these `Tensors`, users can click the `View` link and visualize the `Tensor` in the new panel, which is shown in Figure 3. ![debugger_tensor_compare](./images/debugger_tensor_compare.png) Figure 4: Previous Step Value Compare For Parameter Nodes -In addition, the output of the parameter nodes can be compared with their output in the previous step. +In addition, the output of the parameter nodes can be compared with their output in the previous step. Click the `Compare with Previous Step` button to enter the comparison interface, as shown in Figure 4. ### Conditional Breakpoint @@ -127,13 +127,14 @@ Click the `Compare with Previous Step` button to enter the comparison interface, Figure 5: Set Conditional Breakpoint (Watch Point) -In order to monitor the training and find out the bugs, users can set conditional breakpoints (called `Watch Point List` on UI) to analyze the outputs of the +In order to monitor the training and find out the bugs, users can set conditional breakpoints (called `Watch Point List` on UI) to analyze the outputs of the specified nodes automatically. Figure 5 displays how to set a `Watch Point`: + - At first, click the `+` button on the upper right corner, and then choose a watch condition; - Select the nodes to be watched in the `Node List`, tick the boxes in the front of the chosen nodes; - Click the `OK` button to add this `Watch Point`. -The outputs of the watched nodes will be checked by the corresponding conditions. Once the condition is satisfied, the training will pause, and users can analyze +The outputs of the watched nodes will be checked by the corresponding conditions. Once the condition is satisfied, the training will pause, and users can analyze the triggered `Watch Point List` on the Debugger UI. ![debugger_watch_point_hit](./images/debugger_watch_point_hit.png) @@ -146,7 +147,7 @@ Users can further trace the reason of the bug by analyzing the node details. ### Training Control -At the bottom of the watchpoint setting panel is the training control panel, which shows the training control functions of the debugger, +At the bottom of the watchpoint setting panel is the training control panel, which shows the training control functions of the debugger, with four buttons: `CONTINUE`, `PAUSE`, `TERMINATE` and `OK`: - `OK` stands for executing the training for several steps, the number of the `step` can be specified in the above bar. @@ -160,20 +161,20 @@ The training will be paused until the `Watch Point List` is triggered, or the nu 1. Prepare the debugger environment, and open the MindInsight Debugger UI. ![debugger_waiting](./images/debugger_waiting.png) - + Figure 7: Debugger Start and Waiting for the Training - + The Debugger server is launched and waiting for the training to connect. 2. Run the training script, after a while, the computational graph will be displayed on Debugger UI, as shown in Figure 1. 3. Set conditional breakpoints for the training, as shown in Figure 5. - + In Figure 5, the conditions are selected, and some nodes are watched, which means whether there is any output meeting the conditions in the training process of these nodes. After setting the conditional breakpoint, users can set steps in the control panel and click `OK` or `CONTINUE` to continue training. 4. The conditional breakpoints are triggered, as shown in Figure 6. - + When the conditional breakpoints are triggered, users can analyze the corresponding node details to find out the reason of the bug. ## Notices diff --git a/tutorials/training/source_en/advanced_use/distributed_training_ascend.md b/tutorials/training/source_en/advanced_use/distributed_training_ascend.md index 6e3bc78be0a939fe5cdfc471c43fb461a4bfedcf..8bf810bc6ddf3fedaa8344091edc28d58bbd3706 100644 --- a/tutorials/training/source_en/advanced_use/distributed_training_ascend.md +++ b/tutorials/training/source_en/advanced_use/distributed_training_ascend.md @@ -12,6 +12,8 @@ - [Calling the Collective Communication Library](#calling-the-collective-communication-library) - [Loading the Dataset in Data Parallel Mode](#loading-the-dataset-in-data-parallel-mode) - [Defining the Network](#defining-the-network) + - [Hybrid Parallel Mode](#hybrid-parallel-mode) + - [Semi Auto Parallel Mode](#semi-auto-parallel-mode) - [Defining the Loss Function and Optimizer](#defining-the-loss-function-and-optimizer) - [Defining the Loss Function](#defining-the-loss-function) - [Defining the Optimizer](#defining-the-optimizer) @@ -32,6 +34,8 @@ This tutorial describes how to train the ResNet-50 network in data parallel and automatic parallel modes on MindSpore based on the Ascend 910 AI processor. > Download address of the complete sample code: +Besides, we describe the usages of hybrid parallel and semi-auto parallel modes in the sections [Defining the Network](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/distributed_training_ascend.html#defining-the-network) and [Distributed Training Model Parameters Saving and Loading](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/distributed_training_ascend.html#distributed-training-model-parameters-saving-and-loading). + ## Preparations ### Downloading the Dataset @@ -70,6 +74,7 @@ The following uses the Ascend 910 AI processor as an example. The JSON configura "status": "completed" } ``` + The following parameters need to be modified based on the actual training environment: - `server_count`: number of hosts. @@ -78,13 +83,14 @@ The following parameters need to be modified based on the actual training enviro - `device_ip`: IP address of the integrated NIC. You can run the `cat /etc/hccn.conf` command on the current host. The key value of `address_x` is the IP address of the NIC. - `rank_id`: logical sequence number of a device, which starts from 0. - ### Calling the Collective Communication Library The Huawei Collective Communication Library (HCCL) is used for the communication of MindSpore parallel distributed training and can be found in the Ascend 310 AI processor software package. In addition, `mindspore.communication.management` encapsulates the collective communication API provided by the HCCL to help users configure distributed information. > HCCL implements multi-device multi-node communication based on the Ascend AI processor. The common restrictions on using the distributed service are as follows. For details, see the HCCL documentation. +> > - In a single-node system, a cluster of 1, 2, 4, or 8 devices is supported. In a multi-node system, a cluster of 8 x N devices is supported. > - Each host has four devices numbered 0 to 3 and four devices numbered 4 to 7 deployed on two different networks. During training of 2 or 4 devices, the devices must be connected and clusters cannot be created across networks. +> - When we create a multi-node system, all nodes should use one same exchanger. > - The server hardware architecture and operating system require the symmetrical multi-processing (SMP) mode. The sample code for calling the HCCL as follows: @@ -97,10 +103,11 @@ from mindspore.communication.management import init if __name__ == "__main__": context.set_context(mode=context.GRAPH_MODE, device_target="Ascend", device_id=int(os.environ["DEVICE_ID"])) init() - ... + ... ``` In the preceding code: + - `mode=context.GRAPH_MODE`: sets the running mode to graph mode for distributed training. (The PyNative mode does not support parallel running.) - `device_id`: physical sequence number of a device, that is, the actual sequence number of the device on the corresponding host. - `init`: enables HCCL communication and completes the distributed training initialization. @@ -109,7 +116,6 @@ In the preceding code: During distributed training, data is imported in data parallel mode. The following takes the CIFAR-10 dataset as an example to describe how to import the CIFAR-10 dataset in data parallel mode. `data_path` indicates the dataset path, which is also the path of the `cifar-10-batches-bin` folder. - ```python import mindspore.common.dtype as mstype import mindspore.dataset as ds @@ -122,12 +128,12 @@ def create_dataset(data_path, repeat_num=1, batch_size=32, rank_id=0, rank_size= resize_width = 224 rescale = 1.0 / 255.0 shift = 0.0 - + # get rank_id and rank_size rank_id = get_rank() rank_size = get_group_size() data_set = ds.Cifar10Dataset(data_path, num_shards=rank_size, shard_id=rank_id) - + # define map operations random_crop_op = vision.RandomCrop((32, 32), (4, 4, 4, 4)) random_horizontal_op = vision.RandomHorizontalFlip() @@ -155,13 +161,76 @@ def create_dataset(data_path, repeat_num=1, batch_size=32, rank_id=0, rank_size= return data_set ``` + Different from the single-node system, the multi-node system needs to transfer the `num_shards` and `shard_id` parameters to the dataset API. The two parameters correspond to the number of devices and logical sequence numbers of devices, respectively. You are advised to obtain the parameters through the HCCL API. + - `get_rank`: obtains the ID of the current device in the cluster. - `get_group_size`: obtains the number of devices. ## Defining the Network -In data parallel and automatic parallel modes, the network definition method is the same as that in a single-node system. The reference code is as follows: +In data parallel and automatic parallel modes, the network definition method is the same as that in a single-node system. The reference code of ResNet is as follows: + +In this section we focus on how to define a network in hybrid parallel or semi-auto parallel mode. + +### Hybrid Parallel Mode + +Hybrid parallel mode adds the setting `layerwise_parallel` for `parameter` based on the data parallel mode. The `parameter` with the settig would be saved and computed in slice tensor and would not apply gradients aggregation. In this mode, MindSpore would not infer computation and communication for parallel operators automatically. To ensure the consistency of calculation logic, users are required to manually infer extra operations and insert them to networks. Therefore, this parallel mode is suitable for the users with deep understanding of parallel theory. + +In the following example, specify the `self.weight` as the `layerwise_parallel`, that is, the `self.weight` and the output of `MatMul` are sliced on the second dimension. At this time, perform ReduceSum on the second dimension would only get one sliced result. `AllReduce.Sum` is required here to accumulate the results among all devices. More information about the parallel theory please refer to the [design doc](https://www.mindspore.cn/doc/note/en/master/design/mindspore/distributed_training_design.html). + +```python +from mindspore import Tensor +import mindspore.ops as ops +import mindspore.common.dtype as mstype +import mindspore.nn as nn + +class HybridParallelNet(nn.Cell): + def __init__(self): + super(HybridParallelNet, self).__init__() + # initialize the weight which is sliced at the second dimension + weight_init = np.random.rand(512, 128/2).astype(np.float32) + self.weight = Parameter(Tensor(weight_init), name="weight", layerwise_parallel=True) + self.fc = ops.MatMul() + self.reduce = ops.ReduceSum() + self.allreduce = ops.AllReduce(op='sum') + + def construct(self, x): + x = self.fc(x, self.weight) + x = self.reduce(x, -1) + x = self.allreduce(x) + return x +``` + +### Semi Auto Parallel Mode + +Compared with the auto parallel mode, semi auto parallel mode supports manual configuration on shard strategies for network tuning. The definition of shard strategies could be referred by this [design doc](https://www.mindspore.cn/doc/note/en/master/design/mindspore/distributed_training_design.html). + +It should be noticed that the operators without shard strategies would be regraded as data parallel. If a parameter is used by multiple operators, each operator's shard strategy for this parameter needs to be consistent, otherwise an error will be reported. + +In the above example `HybridParallelNet`, the script in semi auto parallel mode is as follows. The shard stratege of `MatMul` is `{(1, 1), (1, 2)}`, which means `self.weight` is sliced at the second dimension. + +```python +from mindspore import Tensor +import mindspore.ops as ops +import mindspore.common.dtype as mstype +import mindspore.nn as nn + +class SemiAutoParallelNet(nn.Cell): + def __init__(self): + super(SemiAutoParallelNet, self).__init__() + # initialize full tensor weight + weight_init = np.random.rand(512, 128).astype(np.float32) + self.weight = Parameter(Tensor(weight_init), name="weight") + # set shard strategy + self.fc = ops.MatMul().shard({(1, 1),(1, 2)}) + self.reduce = ops.ReduceSum() + + def construct(self, x): + x = self.fc(x, self.weight) + x = self.reduce(x, -1) + return x +``` ## Defining the Loss Function and Optimizer @@ -195,7 +264,7 @@ class SoftmaxCrossEntropyExpand(nn.Cell): self.sparse = sparse self.max = P.ReduceMax(keep_dims=True) self.sub = P.Sub() - + def construct(self, logit, label): logit_max = self.max(logit, -1) exp = self.exp(self.sub(logit, logit_max)) @@ -252,11 +321,14 @@ def test_train_cifar(epoch_size=10): model = Model(net, loss_fn=loss, optimizer=opt) model.train(epoch_size, dataset, callbacks=[loss_cb], dataset_sink_mode=True) ``` + In the preceding code: + - `dataset_sink_mode=True`: uses the dataset sink mode. That is, the training computing is sunk to the hardware platform for execution. - `LossMonitor`: returns the loss value through the callback function to monitor the loss function. ## Running the Script + After the script required for training is edited, run the corresponding command to call the script. Currently, MindSpore distributed execution uses the single-device single-process running mode. That is, one process runs on each device, and the number of total processes is the same as the number of devices that are being used. For device 0, the corresponding process is executed in the foreground. For other devices, the corresponding processes are executed in the background. You need to create a directory for each process to store log information and operator compilation information. The following takes the distributed training script for eight devices as an example to describe how to run the script: @@ -318,6 +390,7 @@ cd ../ The variables `DATA_PATH` and `RANK_SIZE` need to be transferred to the script, which indicate the path of the dataset and the number of devices, respectively. The necessary environment variables are as follows: + - `RANK_TABLE_FILE`: path for storing the networking information file. - `DEVICE_ID`: actual sequence number of the current device on the corresponding host. - `RANK_ID`: logical sequence number of the current device. @@ -327,7 +400,7 @@ The running time is about 5 minutes, which is mainly occupied by operator compil Log files are saved in the `device` directory. The `env.log` file records environment variable information. The `train.log` file records the loss function information. The following is an example: -``` +```text epoch: 1 step: 156, loss is 2.0084016 epoch: 2 step: 156, loss is 1.6407638 epoch: 3 step: 156, loss is 1.6164391 @@ -417,7 +490,7 @@ strategy = ((1, 1), (1, 8)) net = DataParallelNet(strategy=strategy) # reset parallel mode context.reset_auto_parallel_context() -# set parallel mode, data parallel mode is selected for training and model saving. If you want to choose auto parallel +# set parallel mode, data parallel mode is selected for training and model saving. If you want to choose auto parallel # mode, you can simply change the value of parallel_mode parameter to ParallelMode.AUTO_PARALLEL. context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL, device_num=8) ``` @@ -478,10 +551,10 @@ strategy = ((1, 1), (1, 8)) net = SemiAutoParallelNet(strategy=strategy, strategy2=strategy) # reset parallel mode context.reset_auto_parallel_context() -# set parallel mode, data parallel mode is selected for training and model saving. If you want to choose auto parallel +# set parallel mode, data parallel mode is selected for training and model saving. If you want to choose auto parallel # mode, you can simply change the value of parallel_mode parameter to ParallelMode.AUTO_PARALLEL. context.set_auto_parallel_context(parallel_mode=ParallelMode.SEMI_AUTO_PARALLEL, - strategy_ckpt_save_file='./rank_{}_ckpt/strategy.txt'.format(get_rank)) + strategy_ckpt_save_file='./rank_{}_ckpt/strategy.txt'.format(get_rank)) ``` Then set the checkpoint saving policy, optimizer and loss function as required. The code is as follows: @@ -510,12 +583,14 @@ For the three parallel training modes described above, the checkpoint file is sa Only by changing the code that sets the checkpoint saving policy, the checkpoint file of each card can be saved on itself. The specific changes are as follows: Change the checkpoint configuration policy from: + ```python # config checkpoint ckpt_config = CheckpointConfig(keep_checkpoint_max=1) ``` to: + ```python # config checkpoint ckpt_config = CheckpointConfig(keep_checkpoint_max=1, integrated_save=False) @@ -525,4 +600,4 @@ It should be noted that if users chooses this checkpoint saving policy, users ne ### Hybrid Parallel Mode -For model parameter saving and loading in Hybrid Parallel Mode, please refer to [Saving and Loading Model Parameters in the Hybrid Parallel Scenario](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/save_load_model_hybrid_parallel.html). \ No newline at end of file +For model parameter saving and loading in Hybrid Parallel Mode, please refer to [Saving and Loading Model Parameters in the Hybrid Parallel Scenario](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/save_load_model_hybrid_parallel.html). diff --git a/tutorials/training/source_en/advanced_use/distributed_training_gpu.md b/tutorials/training/source_en/advanced_use/distributed_training_gpu.md index 49bd34a74af32c9d84cf1e34271d147b8b64cc02..69d9e0fcc20d247b11686a06f0b5ca6cc08f1860 100644 --- a/tutorials/training/source_en/advanced_use/distributed_training_gpu.md +++ b/tutorials/training/source_en/advanced_use/distributed_training_gpu.md @@ -1,6 +1,6 @@ # Distributed Parallel Training (GPU) -`Linux` `GPU` `Model Training` `Intermediate` `Expert` +`Linux` `GPU` `Model Training` `Intermediate` `Expert` @@ -70,7 +70,7 @@ from mindspore.communication.management import init if __name__ == "__main__": context.set_context(mode=context.GRAPH_MODE, device_target="GPU") init("nccl") - ... + ... ``` In the preceding information, @@ -110,7 +110,7 @@ mpirun -n 8 pytest -s -v ./resnet50_distributed_training.py > train.log 2>&1 & The script requires the variable `DATA_PATH`, which indicates the path of the dataset. In addition, you need to modify the `resnet50_distributed_training.py` file. Since the `DEVICE_ID` environment variable does not need to be set on the GPU, you do not need to call `int(os.getenv('DEVICE_ID'))` in the script to obtain the physical sequence number of the device, and `context` does not require `device_id`. You need to set `device_target` to `GPU` and call `init("nccl")` to enable the NCCL. The log file is saved in the device directory, and the loss result is saved in train.log. The output loss values of the grep command are as follows: -``` +```text epoch: 1 step: 1, loss is 2.3025854 epoch: 1 step: 1, loss is 2.3025854 epoch: 1 step: 1, loss is 2.3025854 @@ -124,6 +124,7 @@ epoch: 1 step: 1, loss is 2.3025854 ## Running the Multi-Host Script If multiple hosts are involved in the training, you need to set the multi-host configuration in the `mpirun` command. You can use the `-H` option in the `mpirun` command. For example, `mpirun -n 16 -H DEVICE1_IP:8,DEVICE2_IP:8 python hello.py` indicates that eight processes are started on the host whose IP addresses are DEVICE1_IP and DEVICE2_IP, respectively. Alternatively, you can create a hostfile similar to the following and transfer its path to the `--hostfile` option of `mpirun`. Each line in the hostfile is in the format of `[hostname] slots=[slotnum]`, where hostname can be an IP address or a host name. + ```bash DEVICE1 slots=8 DEVICE2 slots=8 diff --git a/tutorials/training/source_en/advanced_use/enable_graph_kernel_fusion.md b/tutorials/training/source_en/advanced_use/enable_graph_kernel_fusion.md index 6ef3b5c3751ac89d1197764c4091b03902fd4408..c18de80d685d415230262d0183e2f18dae2c3320 100644 --- a/tutorials/training/source_en/advanced_use/enable_graph_kernel_fusion.md +++ b/tutorials/training/source_en/advanced_use/enable_graph_kernel_fusion.md @@ -100,7 +100,6 @@ context.set_context(enable_graph_kernel=True) 2. `BERT-large` training network Take the training model of the `BERT-large` network as an example. For details about the dataset and training script, see . You only need to modify the `context` parameter. - ## Effect Evaluation diff --git a/tutorials/training/source_en/advanced_use/enable_mixed_precision.md b/tutorials/training/source_en/advanced_use/enable_mixed_precision.md index 8a031c231e3b9b324ff7bf2e209411daf9d27c13..1ca25936cbe2ae6b44983a1c75cf52d7878955d5 100644 --- a/tutorials/training/source_en/advanced_use/enable_mixed_precision.md +++ b/tutorials/training/source_en/advanced_use/enable_mixed_precision.md @@ -16,7 +16,7 @@ ## Overview -The mixed precision training method accelerates the deep learning neural network training process by using both the single-precision and half-precision data formats, and maintains the network precision achieved by the single-precision training at the same time. +The mixed precision training method accelerates the deep learning neural network training process by using both the single-precision and half-precision data formats, and maintains the network precision achieved by the single-precision training at the same time. Mixed precision training can accelerate the computation process, reduce memory usage, and enable a larger model or batch size to be trained on specific hardware. For FP16 operators, if the input data type is FP32, the backend of MindSpore will automatically handle it with reduced precision. Users could check the reduced-precision operators by enabling INFO log and then searching 'reduce precision'. @@ -42,6 +42,7 @@ This document describes the computation process by using examples of automatic a To use the automatic mixed precision, you need to invoke the corresponding API, which takes the network to be trained and the optimizer as the input. This API converts the operators of the entire network into FP16 operators (except the `BatchNorm` and Loss operators). You can use automatic mixed precision through API `amp` or API `Model`. The procedure of using automatic mixed precision by API `amp` is as follows: + 1. Introduce the MindSpore mixed precision API `amp`. 2. Define the network. This step is the same as the common network definition. (You do not need to manually configure the precision of any specific operator.) @@ -93,6 +94,7 @@ output = train_network(predict, label) ``` The procedure of using automatic mixed precision by API `Model` is as follows: + 1. Introduce the MindSpore model API `Model`. 2. Define the network. This step is the same as the common network definition. (You do not need to manually configure the precision of any specific operator.) @@ -169,7 +171,8 @@ model.train(epoch=10, train_dataset=ds_train) MindSpore also supports manual mixed precision. It is assumed that only one dense layer in the network needs to be calculated by using FP32, and other layers are calculated by using FP16. The mixed precision is configured in the granularity of cell. The default format of a cell is FP32. The following is the procedure for implementing manual mixed precision: -1. Define the network. This step is similar to step 2 in the automatic mixed precision. + +1. Define the network. This step is similar to step 2 in the automatic mixed precision. 2. Configure the mixed precision. Use `net.to_float(mstype.float16)` to set all operators of the cell and its sub-cells to FP16. Then, configure the dense to FP32. diff --git a/tutorials/training/source_en/advanced_use/evaluate_the_model_during_training.md b/tutorials/training/source_en/advanced_use/evaluate_the_model_during_training.md index d1bae7be1a799c11a10b2db113ca0c53d78fcb56..2b36b20bb6df7864bfda5c4dba086b4263f23d78 100644 --- a/tutorials/training/source_en/advanced_use/evaluate_the_model_during_training.md +++ b/tutorials/training/source_en/advanced_use/evaluate_the_model_during_training.md @@ -20,6 +20,7 @@ For a complex network, epoch training usually needs to be performed for dozens or even hundreds of times. Before training, it is difficult to know when a model can achieve required accuracy in epoch training. Therefore, the accuracy of the model is usually validated at a fixed epoch interval in training and the corresponding model is saved. After the training is completed, you can quickly select the optimal model by viewing the change of the corresponding model accuracy. This section uses this method and takes the LeNet network as an example. The procedure is as follows: + 1. Define the callback function EvalCallBack to implement synchronous training and validation. 2. Define a training network and execute it. 3. Draw a line chart based on the model accuracy under different epochs and select the optimal model. @@ -52,7 +53,7 @@ class EvalCallBack(Callback): self.eval_dataset = eval_dataset self.eval_per_epoch = eval_per_epoch self.epoch_per_eval = epoch_per_eval - + def epoch_end(self, run_context): cb_param = run_context.original_args() cur_epoch = cb_param.cur_epoch_num @@ -90,51 +91,52 @@ if __name__ == "__main__": eval_per_epoch = 2 ... ... - + # need to calculate how many steps are in each epoch, in this example, 1875 steps per epoch. config_ck = CheckpointConfig(save_checkpoint_steps=eval_per_epoch*1875, keep_checkpoint_max=15) ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet",directory=ckpt_save_dir, config=config_ck) model = Model(network, net_loss, net_opt, metrics={"Accuracy": Accuracy()}) - + epoch_per_eval = {"epoch": [], "acc": []} eval_cb = EvalCallBack(model, eval_data, eval_per_epoch, epoch_per_eval) - + model.train(epoch_size, train_data, callbacks=[ckpoint_cb, LossMonitor(375), eval_cb], dataset_sink_mode=True) ``` The output is as follows: - epoch: 1 step: 375, loss is 2.298612 - epoch: 1 step: 750, loss is 2.075152 - epoch: 1 step: 1125, loss is 0.39205977 - epoch: 1 step: 1500, loss is 0.12368304 - epoch: 1 step: 1875, loss is 0.20988345 - epoch: 2 step: 375, loss is 0.20582482 - epoch: 2 step: 750, loss is 0.029070046 - epoch: 2 step: 1125, loss is 0.041760832 - epoch: 2 step: 1500, loss is 0.067035824 - epoch: 2 step: 1875, loss is 0.0050643035 - {'Accuracy': 0.9763621794871795} - - ... ... - - epoch: 9 step: 375, loss is 0.021227183 - epoch: 9 step: 750, loss is 0.005586236 - epoch: 9 step: 1125, loss is 0.029125651 - epoch: 9 step: 1500, loss is 0.00045874066 - epoch: 9 step: 1875, loss is 0.023556218 - epoch: 10 step: 375, loss is 0.0005807788 - epoch: 10 step: 750, loss is 0.02574059 - epoch: 10 step: 1125, loss is 0.108463734 - epoch: 10 step: 1500, loss is 0.01950589 - epoch: 10 step: 1875, loss is 0.10563098 - {'Accuracy': 0.979667467948718} - +```text +epoch: 1 step: 375, loss is 2.298612 +epoch: 1 step: 750, loss is 2.075152 +epoch: 1 step: 1125, loss is 0.39205977 +epoch: 1 step: 1500, loss is 0.12368304 +epoch: 1 step: 1875, loss is 0.20988345 +epoch: 2 step: 375, loss is 0.20582482 +epoch: 2 step: 750, loss is 0.029070046 +epoch: 2 step: 1125, loss is 0.041760832 +epoch: 2 step: 1500, loss is 0.067035824 +epoch: 2 step: 1875, loss is 0.0050643035 +{'Accuracy': 0.9763621794871795} + +... ... + +epoch: 9 step: 375, loss is 0.021227183 +epoch: 9 step: 750, loss is 0.005586236 +epoch: 9 step: 1125, loss is 0.029125651 +epoch: 9 step: 1500, loss is 0.00045874066 +epoch: 9 step: 1875, loss is 0.023556218 +epoch: 10 step: 375, loss is 0.0005807788 +epoch: 10 step: 750, loss is 0.02574059 +epoch: 10 step: 1125, loss is 0.108463734 +epoch: 10 step: 1500, loss is 0.01950589 +epoch: 10 step: 1875, loss is 0.10563098 +{'Accuracy': 0.979667467948718} +``` Find the `lenet_ckpt` folder in the same directory. The folder contains five models and data related to a calculation graph. The structure is as follows: -``` +```text lenet_ckpt ├── checkpoint_lenet-10_1875.ckpt ├── checkpoint_lenet-2_1875.ckpt @@ -148,7 +150,6 @@ lenet_ckpt Define the drawing function `eval_show`, load `epoch_per_eval` to `eval_show`, and draw the model accuracy variation chart based on different `epoch`. - ```python import matplotlib.pyplot as plt @@ -166,7 +167,6 @@ The output is as follows: ![png](./images/evaluate_the_model_during_training.png) - You can easily select the optimal model based on the preceding figure. ## Summary diff --git a/tutorials/training/source_en/advanced_use/improve_model_security_nad.md b/tutorials/training/source_en/advanced_use/improve_model_security_nad.md index 7af1a43349250261213fdd0b6419b76da6910a1a..05e021edaaec801236970eeed05a4101c313c89d 100644 --- a/tutorials/training/source_en/advanced_use/improve_model_security_nad.md +++ b/tutorials/training/source_en/advanced_use/improve_model_security_nad.md @@ -33,6 +33,7 @@ This section describes how to use MindArmour in adversarial attack and defense b > The current sample is for CPU, GPU and Ascend 910 AI processor. You can find the complete executable sample code at > +> > - `mnist_attack_fgsm.py`: contains attack code. > - `mnist_defense_nad.py`: contains defense code. @@ -133,18 +134,18 @@ The LeNet model is used as an example. You can also create and train your own mo return nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding, weight_init=weight, has_bias=False, pad_mode="valid") - - + + def fc_with_initialize(input_channels, out_channels): weight = weight_variable() bias = weight_variable() return nn.Dense(input_channels, out_channels, weight, bias) - - + + def weight_variable(): return TruncatedNormal(0.02) - - + + class LeNet5(nn.Cell): """ Lenet network @@ -159,7 +160,7 @@ The LeNet model is used as an example. You can also create and train your own mo self.relu = nn.ReLU() self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2) self.flatten = nn.Flatten() - + def construct(self, x): x = self.conv1(x) x = self.relu(x) @@ -191,7 +192,7 @@ The LeNet model is used as an example. You can also create and train your own mo model = Model(net, loss, opt, metrics=None) model.train(10, ds_train, callbacks=[LossMonitor()], dataset_sink_mode=False) - + # get test data ds_test = generate_mnist_dataset(os.path.join(mnist_path, "test"), batch_size=batch_size, repeat_size=1, @@ -218,16 +219,16 @@ The LeNet model is used as an example. You can also create and train your own mo logits = net(Tensor(batch_inputs)).asnumpy() test_logits.append(logits) test_logits = np.concatenate(test_logits) - + tmp = np.argmax(test_logits, axis=1) == np.argmax(test_labels, axis=1) accuracy = np.mean(tmp) LOGGER.info(TAG, 'prediction accuracy before attacking is : %s', accuracy) - + ``` The classification accuracy reaches 98%. - - ```python + + ```python prediction accuracy before attacking is : 0.9895833333333334 ``` @@ -274,7 +275,7 @@ LOGGER.info(TAG, 'The average structural similarity between original ' The attack results are as follows: -``` +```text prediction accuracy after attacking is : 0.052083 mis-classification rate of adversaries is : 0.947917 The average confidence of adversarial class is : 0.803375 @@ -351,7 +352,7 @@ LOGGER.info(TAG, 'The average confidence of true class is : %s', ### Defense Effect -``` +```text accuracy of TEST data on defensed model is : 0.974259 accuracy of adv data on defensed model is : 0.856370 defense mis-classification rate of adversaries is : 0.143629 @@ -359,5 +360,4 @@ The average confidence of adversarial class is : 0.616670 The average confidence of true class is : 0.177374 ``` -After NAD is used to defend against adversarial examples, the model's misclassification ratio of adversarial examples decreases from 95% to 14%, effectively defending against adversarial examples. In addition, the classification accuracy of the model for the original test dataset reaches 97%. - +After NAD is used to defend against adversarial examples, the model's misclassification ratio of adversarial examples decreases from 95% to 14%, effectively defending against adversarial examples. In addition, the classification accuracy of the model for the original test dataset reaches 97%. diff --git a/tutorials/training/source_en/advanced_use/lineage_and_scalars_comparision.md b/tutorials/training/source_en/advanced_use/lineage_and_scalars_comparision.md index 68cfe7d1a65fd5bc186e5a2f4cbb721ec3135886..73337706653defa72ead334ac953b702dcbb4d48 100644 --- a/tutorials/training/source_en/advanced_use/lineage_and_scalars_comparision.md +++ b/tutorials/training/source_en/advanced_use/lineage_and_scalars_comparision.md @@ -105,6 +105,7 @@ Figure 9 shows the scalars comparision function area, which allows you to view s ## Notices To ensure performance, MindInsight implements scalars comparision with the cache mechanism and the following restrictions: -- The scalars comparision supports only for trainings in cache. + +- The scalars comparision supports only for trainings in cache. - The maximum of 15 latest trainings (sorted by modification time) can be retained in the cache. - The maximum of 5 trainings can be selected for scalars comparision at the same time. diff --git a/tutorials/training/source_en/advanced_use/migrate_3rd_scripts.md b/tutorials/training/source_en/advanced_use/migrate_3rd_scripts.md index f84d41e41347b26b210a10513bd7336704afec58..140c2c5683b11fa06481d36c3f98a709e8b6e222 100644 --- a/tutorials/training/source_en/advanced_use/migrate_3rd_scripts.md +++ b/tutorials/training/source_en/advanced_use/migrate_3rd_scripts.md @@ -275,4 +275,3 @@ Models trained on the Ascend 910 AI processor can be used for inference on diffe ## Examples - [Model Zoo](https://gitee.com/mindspore/mindspore/tree/master/model_zoo) - diff --git a/tutorials/training/source_en/advanced_use/migrate_3rd_scripts_mindconverter.md b/tutorials/training/source_en/advanced_use/migrate_3rd_scripts_mindconverter.md index ce21f4c2521c29c14f0778cdcd25cb35f5a20f26..1bf17792d50b319f44f60dd7fcc1e46a43f3b6a5 100644 --- a/tutorials/training/source_en/advanced_use/migrate_3rd_scripts_mindconverter.md +++ b/tutorials/training/source_en/advanced_use/migrate_3rd_scripts_mindconverter.md @@ -22,12 +22,10 @@ MindConverter is a migration tool to transform the model scripts from PyTorch to Mindspore. Users can migrate their PyTorch models to Mindspore rapidly with minor changes according to the conversion report. - ## Installation Mindconverter is a submodule in MindInsight. Please follow the [Guide](https://www.mindspore.cn/install/en) here to install MindInsight. - ## Usage MindConverter currently only provides command-line interface. Here is the manual page. @@ -71,7 +69,7 @@ optional arguments: > The AST mode will be enabled, if both `--in_file` and `--model_file` are specified. -For the Graph mode, `--shape` is mandatory. +For the Graph mode, `--shape` is mandatory. For the AST mode, `--shape` is ignored. @@ -80,11 +78,8 @@ For the AST mode, `--shape` is ignored. Please note that your original PyTorch project is included in the module search path (PYTHONPATH). Use the python interpreter and test your module can be successfully loaded by `import` command. Use `--project_path` instead if your project is not in the PYTHONPATH to ensure MindConverter can load it. > Assume the project is located at `/home/user/project/model_training`, users can use this command to add the project to `PYTHONPATH` : `export PYTHONPATH=/home/user/project/model_training:$PYTHONPATH` - > MindConverter needs the original PyTorch scripts because of the reverse serialization. - - ## Scenario MindConverter provides two modes for different migration demands. @@ -96,13 +91,12 @@ The AST mode is recommended for the first demand. It parses and analyzes PyTorch For the second demand, the Graph mode is recommended. As the computational graph is a standard descriptive language, it is not affected by user's coding style. This mode may have more operators converted as long as these operators are supported by MindConverter. -Some typical image classification networks such as ResNet and VGG have been tested for the Graph mode. Note that: +Some typical image classification networks such as ResNet and VGG have been tested for the Graph mode. Note that: > 1. Currently, the Graph mode does not support models with multiple inputs. Only models with a single input and single output are supported. > 2. The Dropout operator will be lost after conversion because the inference mode is used to load the PyTorch model. Manually re-implement is necessary. > 3. The Graph-based mode will be continuously developed and optimized with further updates. - ## Example ### AST-Based Conversion @@ -123,8 +117,8 @@ line : [UnConvert] 'operator' didn't convert. ... For non-transformed operators, the original code keeps. Please manually migrate them. [Click here](https://www.mindspore.cn/doc/note/en/master/index.html#operator_api) for more information about operator mapping. - Here is an example of the conversion report: + ```text [Start Convert] [Insert] 'import mindspore.ops.operations as P' is inserted to the converted file. @@ -137,7 +131,6 @@ Here is an example of the conversion report: For non-transformed operators, suggestions are provided in the report. For instance, MindConverter suggests that replace `torch.nn.AdaptiveAvgPool2d` with `mindspore.ops.operations.ReduceMean`. - ### Graph-Based Conversion Assume the PyTorch model (.pth file) is located at `/home/user/model.pth`, with input shape (3, 224, 224) and the original PyTorch script is at `/home/user/project/model_training`. Output the transformed MindSpore script to `/home/user/output`, with the conversion report to `/home/user/output/report`. Use the following command: @@ -194,12 +187,9 @@ class Classifier(nn.Cell): ``` -> `--output` and `--report` are optional. MindConverter creates an `output` folder under the current working directory, and outputs generated scripts and conversion reports to it. - +> `--output` and `--report` are optional. MindConverter creates an `output` folder under the current working directory, and outputs generated scripts and conversion reports to it. ## Caution 1. PyTorch is not an explicitly stated dependency library in MindInsight. The Graph conversion requires the consistent PyTorch version as the model is trained. (MindConverter recommends PyTorch 1.4.0 or 1.6.0) -2. This script conversion tool relies on operators which supported by MindConverter and MindSpore. Unsupported operators may not successfully mapped to MindSpore operators. You can manually edit, or implement the mapping based on MindConverter, and contribute to our MindInsight repository. We appreciate your support for the MindSpore community. - - +2. This script conversion tool relies on operators which supported by MindConverter and MindSpore. Unsupported operators may not successfully mapped to MindSpore operators. You can manually edit, or implement the mapping based on MindConverter, and contribute to our MindInsight repository. We appreciate your support for the MindSpore community. diff --git a/tutorials/training/source_en/advanced_use/nlp_sentimentnet.md b/tutorials/training/source_en/advanced_use/nlp_sentimentnet.md index a697aa0c93ef8afc4350b9ed2db97b02df142f4f..e3dba0158c68eca3b0c1db799acfeef63825ae8d 100644 --- a/tutorials/training/source_en/advanced_use/nlp_sentimentnet.md +++ b/tutorials/training/source_en/advanced_use/nlp_sentimentnet.md @@ -47,6 +47,7 @@ Vertical polarity word = General polarity word + Domain-specific polarity word According to the text processing granularity, sentiment analysis can be divided into word, phrase, sentence, paragraph, and chapter levels. A sentiment analysis at paragraph level is used as an example. The input is a paragraph, and the output is information about whether the movie review is positive or negative. ## Preparation and Design + ### Downloading the Dataset The IMDb movie review dataset is used as experimental data. @@ -54,15 +55,17 @@ The IMDb movie review dataset is used as experimental data. The following are cases of negative and positive reviews. -| Review | Label | +| Review | Label | |---|---| | "Quitting" may be as much about exiting a pre-ordained identity as about drug withdrawal. As a rural guy coming to Beijing, class and success must have struck this young artist face on as an appeal to separate from his roots and far surpass his peasant parents' acting success. Troubles arise, however, when the new man is too new, when it demands too big a departure from family, history, nature, and personal identity. The ensuing splits, and confusion between the imaginary and the real and the dissonance between the ordinary and the heroic are the stuff of a gut check on the one hand or a complete escape from self on the other. | Negative | | This movie is amazing because the fact that the real people portray themselves and their real life experience and do such a good job it's like they're almost living the past over again. Jia Hongsheng plays himself an actor who quit everything except music and drugs struggling with depression and searching for the meaning of life while being angry at everyone especially the people who care for him most. | Positive | Download the GloVe file and add the following line at the beginning of the file, which means that a total of 400,000 words are read, and each word is represented by a word vector of 300 latitudes. -``` + +```text 400000 300 ``` + GloVe file download address: ### Determining Evaluation Criteria @@ -79,16 +82,17 @@ F1 score = (2 x Precision x Recall)/(Precision + Recall) In the IMDb dataset, the number of positive and negative samples does not vary greatly. Accuracy can be used as the evaluation criterion of the classification system. - ### Determining the Network and Process Currently, MindSpore GPU and CPU supports SentimentNet network based on the long short-term memory (LSTM) network for NLP. + 1. Load the dataset in use and process data if necessary. 2. Use the SentimentNet network based on LSTM to train data and generate a model. Long short-term memory (LSTM) is an artificial recurrent neural network (RNN) architecture used for processing and predicting an important event with a long interval and delay in a time sequence. For details, refer to online documentation. 3. After the model is obtained, use the validation dataset to check the accuracy of model. > The current sample is for the Ascend 910 AI processor. You can find the complete executable sample code at +> > - `src/config.py`: some configurations of the network, including the batch size and number of training epochs. > - `src/dataset.py`: dataset related definition, including converted MindRecord file and preprocessed data. > - `src/imdb.py`: the utility class for parsing IMDb dataset. @@ -97,8 +101,11 @@ Currently, MindSpore GPU and CPU supports SentimentNet network based on the long > - `eval.py`: the evaluation script. ## Implementation + ### Importing Library Files + The following are the required public modules and MindSpore modules and library files. + ```python import argparse import os @@ -118,6 +125,7 @@ from mindspore.train.serialization import load_param_into_net, load_checkpoint ### Configuring Environment Information 1. The `parser` module is used to transfer necessary information for running, such as storage paths of the dataset and the GloVe file. In this way, the frequently changed configurations can be entered during runtime, which is more flexible. + ```python parser = argparse.ArgumentParser(description='MindSpore LSTM Example') parser.add_argument('--preprocess', type=str, default='false', choices=['true', 'false'], @@ -138,13 +146,14 @@ from mindspore.train.serialization import load_param_into_net, load_checkpoint ``` 2. Before implementing code, configure necessary information, including the environment information, execution mode, backend information, and hardware information. - + ```python context.set_context( mode=context.GRAPH_MODE, save_graphs=False, device_target=args.device_target) ``` + For details about the API configuration, see the `context.set_context`. ### Preprocessing the Dataset @@ -156,15 +165,14 @@ if args.preprocess == "true": print("============== Starting Data Pre-processing ==============") convert_to_mindrecord(cfg.embed_size, args.aclimdb_path, args.preprocess_path, args.glove_path) ``` -> After successful conversion, `mindrecord` files are generated under the directory `preprocess_path`. Usually, this operation does not need to be performed every time if the dataset is unchanged. +> After successful conversion, `mindrecord` files are generated under the directory `preprocess_path`. Usually, this operation does not need to be performed every time if the dataset is unchanged. > For `convert_to_mindrecord`, you can find the complete definition at: - > It consists of two steps: +> >1. Process the text dataset, including encoding, word segmentation, alignment, and processing the original GloVe data to adapt to the network structure. >2. Convert the dataset format to the MindRecord format. - ### Defining the Network ```python @@ -178,11 +186,13 @@ network = SentimentNet(vocab_size=embedding_table.shape[0], weight=Tensor(embedding_table), batch_size=cfg.batch_size) ``` + > For `SentimentNet`, you can find the complete definition at: ### Pre-Training The parameter `pre_trained` specifies the preloading CheckPoint file for pre-training, which is empty by default + ```python if args.pre_trained: load_param_into_net(network, load_checkpoint(args.pre_trained)) @@ -217,6 +227,7 @@ else: model.train(cfg.num_epochs, ds_train, callbacks=[time_cb, ckpoint_cb, loss_cb]) print("============== Training Success ==============") ``` + > For `lstm_create_dataset`, you can find the complete definition at: ### Validating the Model @@ -238,12 +249,15 @@ print("============== {} ==============".format(acc)) ``` ## Experimental Result + After 20 epochs, the accuracy on the test set is about 84.19%. -**Training Execution** +**Training Execution:** + 1. Run the training code and view the running result. + ```shell - $ python train.py --preprocess=true --ckpt_path=./ --device_target=GPU + python train.py --preprocess=true --ckpt_path=./ --device_target=GPU ``` As shown in the following output, the loss value decreases gradually with the training process and reaches about 0.2855. @@ -263,11 +277,11 @@ After 20 epochs, the accuracy on the test set is about 84.19%. ``` 2. Check the saved CheckPoint files. - + CheckPoint files (model files) are saved during the training. You can view all saved files in the file path. ```shell - $ ls ./*.ckpt + ls ./*.ckpt ``` The output is as follows: @@ -276,12 +290,12 @@ After 20 epochs, the accuracy on the test set is about 84.19%. lstm-11_390.ckpt lstm-12_390.ckpt lstm-13_390.ckpt lstm-14_390.ckpt lstm-15_390.ckpt lstm-16_390.ckpt lstm-17_390.ckpt lstm-18_390.ckpt lstm-19_390.ckpt lstm-20_390.ckpt ``` -**Model Validation** +**Model Validation:** Use the last saved CheckPoint file to load and validate the dataset. ```shell -$ python eval.py --ckpt_path=./lstm-20_390.ckpt --device_target=GPU +python eval.py --ckpt_path=./lstm-20_390.ckpt --device_target=GPU ``` As shown in the following output, the sentiment analysis accuracy of the text is about 84.19%, which is basically satisfactory. @@ -290,4 +304,3 @@ As shown in the following output, the sentiment analysis accuracy of the text is ============== Starting Testing ============== ============== {'acc': 0.8419471153846154} ============== ``` - diff --git a/tutorials/training/source_en/advanced_use/optimize_data_processing.md b/tutorials/training/source_en/advanced_use/optimize_data_processing.md index dd973909e001655eb9970b725301c802249a9de1..3feeee91b9c5a466464151eef53c12c5bdaa06ce 100644 --- a/tutorials/training/source_en/advanced_use/optimize_data_processing.md +++ b/tutorials/training/source_en/advanced_use/optimize_data_processing.md @@ -53,7 +53,7 @@ import numpy as np The directory structure is as follows: -``` +```text dataset/Cifar10Data ├── cifar-10-batches-bin │   ├── batches.meta.txt @@ -76,6 +76,7 @@ dataset/Cifar10Data ``` In the preceding information: + - The `cifar-10-batches-bin` directory is the directory for storing the CIFAR-10 dataset in binary format. - The `cifar-10-batches-py` directory is the directory for storing the CIFAR-10 dataset in Python file format. @@ -93,6 +94,7 @@ MindSpore provides multiple data loading methods, including common dataset loadi ![title](./images/data_loading_performance_scheme.png) Suggestions on data loading performance optimization are as follows: + - Built-in loading operators are preferred for supported dataset formats. For details, see [Built-in Loading Operators](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.dataset.html). If the performance cannot meet the requirements, use the multi-thread concurrency solution. For details, see [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#multi-thread-optimization-solution). - For a dataset format that is not supported, convert the format to the MindSpore data format and then use the `MindDataset` class to load the dataset. If the performance cannot meet the requirements, use the multi-thread concurrency solution, for details, see [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#multi-thread-optimization-solution). - For dataset formats that are not supported, the user-defined `GeneratorDataset` class is preferred for implementing fast algorithm verification. If the performance cannot meet the requirements, the multi-process concurrency solution can be used. For details, see [Multi-process Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#multi-process-optimization-solution). @@ -113,7 +115,8 @@ Based on the preceding suggestions of data loading performance optimization, the ``` The output is as follows: - ``` + + ```text {'image': Tensor(shape=[32, 32, 3], dtype=UInt8, value= [[[235, 235, 235], [230, 230, 230], @@ -148,7 +151,7 @@ Based on the preceding suggestions of data loading performance optimization, the The output is as follows: - ``` + ```text {'data': Tensor(shape=[1431], dtype=UInt8, value= [255, 216, 255, ..., 63, 255, 217]), 'id': Tensor(shape=[], dtype=Int64, value= 30474), 'label': Tensor(shape=[], dtype=Int64, value= 2)} @@ -169,7 +172,7 @@ Based on the preceding suggestions of data loading performance optimization, the The output is as follows: - ``` + ```text {'data': Tensor(shape=[1], dtype=Int64, value= [0])} ``` @@ -182,6 +185,7 @@ The shuffle operation is used to shuffle ordered datasets or repeated datasets. ![title](./images/shuffle_performance_scheme.png) Suggestions on shuffle performance optimization are as follows: + - Use the `shuffle` parameter of built-in loading operators to shuffle data. - If the `shuffle` function is used and the performance still cannot meet the requirements, adjust the value of the `buffer_size` parameter to improve the performance. @@ -202,7 +206,7 @@ Based on the preceding shuffle performance optimization suggestions, the `shuffl The output is as follows: - ``` + ```text {'image': Tensor(shape=[32, 32, 3], dtype=UInt8, value= [[[235, 235, 235], [230, 230, 230], @@ -237,7 +241,7 @@ Based on the preceding shuffle performance optimization suggestions, the `shuffl The output is as follows: - ``` + ```text before shuffle: [0 1 2 3 4] [1 2 3 4 5] @@ -255,6 +259,7 @@ Based on the preceding shuffle performance optimization suggestions, the `shuffl ## Optimizing the Data Augmentation Performance During image classification training, especially when the dataset is small, users can use data augmentation to preprocess images to enrich the dataset. MindSpore provides multiple data augmentation methods, including: + - Use the built-in C operator (`c_transforms` module) to perform data augmentation. - Use the built-in Python operator (`py_transforms` module) to perform data augmentation. - Users can define Python functions as needed to perform data augmentation. @@ -271,6 +276,7 @@ The performance varies according to the underlying implementation methods. ![title](./images/data_enhancement_performance_scheme.png) Suggestions on data augmentation performance optimization are as follows: + - The `c_transforms` module is preferentially used to perform data augmentation for its highest performance. If the performance cannot meet the requirements, refer to [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#multi-thread-optimization-solution), [Compose Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#compose-optimization-solution), or [Operator Fusion Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#operator-fusion-optimization-solution). - If the `py_transforms` module is used to perform data augmentation and the performance still cannot meet the requirements, refer to [Multi-thread Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#multi-thread-optimization-solution), [Multi-process Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#multi-process-optimization-solution), [Compose Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#compose-optimization-solution), or [Operator Fusion Optimization Solution](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/optimize_data_processing.html#operator-fusion-optimization-solution). - The `c_transforms` module maintains buffer management in C++, and the `py_transforms` module maintains buffer management in Python. Because of the performance cost of switching between Python and C++, it is advised not to use different operator types together. @@ -324,7 +330,7 @@ Based on the preceding suggestions of data augmentation performance optimization The output is as follows: - ``` + ```text before map: [0 1 2 3 4] [1 2 3 4 5] @@ -392,6 +398,7 @@ Data processing is performed on the host. Therefore, configurations of the host ### Multi-thread Optimization Solution During the data pipeline process, the number of threads for related operators can be set to improve the concurrency and performance. For example: + - During data loading, the `num_parallel_workers` parameter in the built-in data loading class is used to set the number of threads. - During data augmentation, the `num_parallel_workers` parameter in the `map` function is used to set the number of threads. - During batch processing, the `num_parallel_workers` parameter in the `batch` function is used to set the number of threads. @@ -401,6 +408,7 @@ For details, see [Built-in Loading Operators](https://www.mindspore.cn/doc/api_p ### Multi-process Optimization Solution During data processing, operators implemented by Python support the multi-process mode. For example: + - By default, the `GeneratorDataset` class is in multi-process mode. The `num_parallel_workers` parameter indicates the number of enabled processes. The default value is 1. For details, see [GeneratorDataset](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.dataset.html#mindspore.dataset.GeneratorDataset). - If the user-defined Python function or the `py_transforms` module is used to perform data augmentation and the `python_multiprocessing` parameter of the `map` function is set to True, the `num_parallel_workers` parameter indicates the number of processes and the default value of the `python_multiprocessing` parameter is False. In this case, the `num_parallel_workers` parameter indicates the number of threads. For details, see [Built-in Loading Operators](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.dataset.html). diff --git a/tutorials/training/source_en/advanced_use/performance_profiling.md b/tutorials/training/source_en/advanced_use/performance_profiling.md index f18d59a6e706c5c76e3a0ecad9af5efaddba9f58..e69d6daf16666a88d42fa67838648261cbf45b5b 100644 --- a/tutorials/training/source_en/advanced_use/performance_profiling.md +++ b/tutorials/training/source_en/advanced_use/performance_profiling.md @@ -22,6 +22,7 @@ ## Overview + Performance data like operator's execution time is recorded in files and can be viewed on the web page, this can help users optimize the performance of neural networks. ## Operation Process @@ -52,10 +53,10 @@ from mindspore import Model, nn, context def test_profiler(): # Init context env context.set_context(mode=context.GRAPH_MODE, device_target='Ascend', device_id=int(os.environ["DEVICE_ID"])) - + # Init Profiler profiler = Profiler() - + # Init hyperparameter epoch = 2 # Init network and Model @@ -67,17 +68,15 @@ def test_profiler(): train_ds = create_mindrecord_dataset_for_training() # Model Train model.train(epoch, train_ds) - + # Profiler end profiler.analyse() ``` - ## Launch MindInsight The MindInsight launch command can refer to [MindInsight Commands](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/mindinsight_commands.html). - ### Performance Analysis Users can access the Performance Profiler by selecting a specific training from the training list, and click the performance profiling link. @@ -87,6 +86,7 @@ Users can access the Performance Profiler by selecting a specific training from Figure 1: Overall Performance Figure 1 displays the overall performance of the training, including the overall data of Step Trace, Operator Performance, MindData Performance and Timeline. The data shown in these components include: + - Step Trace: It will divide the training steps into several stages and collect execution time for each stage. The overall performance page will show the step trace graph. - Operator Performance: It will collect the execution time of operators and operator types. The overall performance page will show the pie graph for different operator types. - MindData Performance: It will analyse the performance of the data input stages. The overall performance page will show the number of steps that may be the bottleneck for these stages. @@ -103,14 +103,14 @@ Step Gap (The time between the end of one step and the computation of next step) Figure 2: Step Trace Analysis -Figure 2 displays the Step Trace page. The Step Trace detail will show the start/finish time for each stage. By default, it shows the average time for all the steps. Users can also choose a specific step to see its step trace statistics. The graphs at the bottom of the page show the execution time of Step Gap, Forward/Backward Propagation and Step Tail (The time between the end of Backward Propagation and the end of Parameter Update) changes according to different steps, it will help to decide whether we can optimize the performance of some stages. +Figure 2 displays the Step Trace page. The Step Trace detail will show the start/finish time for each stage. By default, it shows the average time for all the steps. Users can also choose a specific step to see its step trace statistics. The graphs at the bottom of the page show the execution time of Step Gap, Forward/Backward Propagation and Step Tail (The time between the end of Backward Propagation and the end of Parameter Update) changes according to different steps, it will help to decide whether we can optimize the performance of some stages. In order to divide the stages, the Step Trace Component need to figure out the forward propagation start operator and the backward propagation end operator. MindSpore will automatically figure out the two operators to reduce the profiler configuration work. The first operator after `get_next` will be selected as the forward start operator and the operator before the last all reduce will be selected as the backward end operator. **However, Profiler do not guarantee that the automatically selected operators will meet the user's expectation in all cases.** Users can set the two operators manually as follows: + - Set environment variable `FP_POINT` to configure the forward start operator, for example, `export FP_POINT=fp32_vars/conv2d/BatchNorm`. - Set environment variable `BP_POINT` to configure the backward end operator, for example, `export BP_POINT=loss_scale/gradients/AddN_70`. - #### Operator Performance Analysis The operator performance analysis component is used to display the execution time of the operators during MindSpore run. @@ -120,7 +120,8 @@ The operator performance analysis component is used to display the execution tim Figure 3: Statistics for Operator Types Figure 3 displays the statistics for the operator types, including: -- Choose pie or bar graph to show the proportion time occupied by each operator type. The time of one operator type is calculated by accumulating the execution time of operators belonging to this type. + +- Choose pie or bar graph to show the proportion time occupied by each operator type. The time of one operator type is calculated by accumulating the execution time of operators belonging to this type. - Display top 20 operator types with the longest execution time, show the proportion and execution time (ms) of each operator type. ![op_statistics.png](./images/op_statistics.PNG) @@ -128,6 +129,7 @@ Figure 3 displays the statistics for the operator types, including: Figure 4: Statistics for Operators Figure 4 displays the statistics table for the operators, including: + - Choose All: Display statistics for the operators, including operator name, type, execution time, full scope time, information, etc. The table will be sorted by execution time by default. - Choose Type: Display statistics for the operator types, including operator type name, execution time, execution frequency and proportion of total time. Users can click on each line, querying for all the operators belonging to this type. - Search: There is a search box on the right, which can support fuzzy search for operators/operator types. @@ -135,7 +137,7 @@ Figure 4 displays the statistics table for the operators, including: #### MindData Performance Analysis The MindData performance analysis component is used to analyse the execution of data input pipeline for the training. The data input pipeline can be divided into three stages: -the data process pipeline, data transfer from host to device and data fetch on device. The component will analyse the performance of each stage in detail and display the results. +the data process pipeline, data transfer from host to device and data fetch on device. The component will analyse the performance of each stage in detail and display the results. ![minddata_profile.png](./images/minddata_profile.png) @@ -144,38 +146,44 @@ Figure 5: MindData Performance Analysis Figure 5 displays the page of MindData performance analysis component. It consists of two tabs: The step gap and the data process. The step gap page is used to analyse whether there is performance bottleneck in the three stages. We can get our conclusion from the data queue graphs: + - The data queue size stands for the queue length when the training fetches data from the queue on the device. If the data queue size is 0, the training will wait until there is data in the queue; If the data queue size is above 0, the training can get data very quickly, and it means MindData is not the bottleneck for this training step. - The host queue size can be used to infer the speed of data process and data transfer. If the host queue size is 0, it means we need to speed up the data process stage. -- If the size of the host queue is always large and the size of the data queue is continuously small, there may be a performance bottleneck in data transfer. +- If the size of the host queue is always large and the size of the data queue is continuously small, there may be a performance bottleneck in data transfer. ![data_op_profile.png](./images/data_op_profile.png) Figure 6: Data Process Pipeline Analysis Figure 6 displays the page of data process pipeline analysis. The data queues are used to exchange data between the MindData operators. The data size of the queues reflect the data consume speed of the operators, and can be used to infer the bottleneck operator. The queue usage percentage stands for the average value of data size in queue divide data queue maximum size, the higher the usage percentage, the more data that is accumulated in the queue. The graph at the bottom of the page shows the MindData pipeline operators with the data queues, the user can click one queue to see how the data size changes according to the time, and the operators connected to the queue. The data process pipeline can be analysed as follows: + - When the input queue usage percentage of one operator is high, and the output queue usage percentage is low, the operator may be the bottleneck. - For the leftmost operator, if the usage percentage of all the queues on the right are low, the operator may be the bottleneck. -- For the rightmost operator, if the usage percentage of all the queues on the left are high, the operator may be the bottleneck. +- For the rightmost operator, if the usage percentage of all the queues on the left are high, the operator may be the bottleneck. To optimize the perforamnce of MindData operators, there are some suggestions: + - If the Dataset Operator is the bottleneck, try to increase the `num_parallel_workers`. - If a GeneratorOp type operator is the bottleneck, try to increase the `num_parallel_workers` and replace the operator to `MindRecordDataset`. - If a MapOp type operator is the bottleneck, try to increase the `num_parallel_workers`. If it is a python operator, try to optimize the training script. -- If a BatchOp type operator is the bottleneck, try to adjust the size of `prefetch_size`. +- If a BatchOp type operator is the bottleneck, try to adjust the size of `prefetch_size`. #### Timeline Analysis -The Timeline component can display: +The Timeline component can display: + - The operators (AICore/AICPU operators) are executed on which device. - The MindSpore stream split strategy for this neural network. - The execution sequence and execution time of the operator on the device. Users can get the most detailed information from the Timeline: + - From the High level, users can analyse whether the stream split strategy can be optimized and whether the step tail is too long. - From the Low level, users can analyse the execution time for all the operators, etc. Users can click the download button on the overall performance page to view Timeline details. The Timeline data file (json format) will be stored on local machine, and can be displayed by tools. We suggest to use `chrome://tracing` or [Perfetto](https://ui.perfetto.dev/#!viewer) to visualize the Timeline. + - Chrome tracing: Click "load" on the upper left to load the file. - Perfetto: Click "Open trace file" on the left to load the file. @@ -184,8 +192,9 @@ Users can click the download button on the overall performance page to view Time Figure 7: Timeline Analysis The Timeline consists of the following parts: + - Device and Stream List: It will show the stream list on each device. Each stream consists of a series of tasks. One rectangle stands for one task, and the area stands for the execution time of the task. -- The Operator Information: When we click one task, the corresponding operator of this task will be shown at the bottom. +- The Operator Information: When we click one task, the corresponding operator of this task will be shown at the bottom. W/A/S/D can be applied to zoom in and out of the Timeline graph. diff --git a/tutorials/training/source_en/advanced_use/performance_profiling_gpu.md b/tutorials/training/source_en/advanced_use/performance_profiling_gpu.md index e989af720650ac32483bcdf43237c693dd4d3be2..be91784497ba1cea230ecc92ea2a165cabab3cfc 100644 --- a/tutorials/training/source_en/advanced_use/performance_profiling_gpu.md +++ b/tutorials/training/source_en/advanced_use/performance_profiling_gpu.md @@ -18,6 +18,7 @@ ## Overview + Performance data like operators' execution time is recorded in files and can be viewed on the web page, this can help the user optimize the performance of neural networks. ## Operation Process @@ -25,9 +26,8 @@ Performance data like operators' execution time is recorded in files and can be > The GPU operation process is the same as that in the Ascend chip. > > - > By default, common users do not have the permission to access the NVIDIA GPU performance counters on the target device. -> If common users need to use the profiler performance statistics capability in the training script, configure the permission by referring to the following description: +> If common users need to use the profiler performance statistics capability in the training script, configure the permission by referring to the following description: > > @@ -48,7 +48,7 @@ class StopAtStep(Callback): self.start_step = start_step self.stop_step = stop_step self.already_analysed = False - + def step_begin(self, run_context): cb_params = run_context.original_args() step_num = cb_params.cur_step_num @@ -61,7 +61,7 @@ class StopAtStep(Callback): if step_num == self.stop_step and not self.already_analysed: self.profiler.analyse() self.already_analysed = True - + def end(self, run_context): if not self.already_analysed: self.profiler.analyse() @@ -73,7 +73,6 @@ The code above is just an example. Users should implement callback by themselves The MindInsight launch command can refer to [MindInsight Commands](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/mindinsight_commands.html). - ### Performance Analysis Users can access the Performance Profiler by selecting a specific training from the training list, and click the performance profiling link. And the Performance Profiler only supports operation analysis and Timeline Analysis now, the others modules will be published soon. @@ -83,6 +82,7 @@ Users can access the Performance Profiler by selecting a specific training from Figure 1: Overall Performance Figure 1 displays the overall performance of the training, including the overall data of Step Trace, Operator Performance, MindData Performance and Timeline: + - Operator Performance: It will collect the average execution time of operators and operator types. The overall performance page will show the pie graph for different operator types. - Timeline: It will collect execution time for operations and CUDA activity. The tasks will be shown on the time axis. The overall performance page will show the statistics for tasks. @@ -98,7 +98,7 @@ Figure 2: Statistics for Operator Types Figure 2 displays the statistics for the operator types, including: -- Choose a pie or a bar graph to show the proportion time occupied by each operator type. The time of one operator type is calculated by accumulating the execution time of operators belong to this type. +- Choose a pie or a bar graph to show the proportion time occupied by each operator type. The time of one operator type is calculated by accumulating the execution time of operators belong to this type. - Display top 20 operator types with the longest average execution time, show the proportion of total time and average execution time (ms) of each operator type. The bottom half of Figure 2 displays the statistics table for the operators' details, including: @@ -123,4 +123,4 @@ The usage is almost the same as that in Ascend. The difference is GPU Timeline d > The usage is described as follows: > -> \ No newline at end of file +> diff --git a/tutorials/training/source_en/advanced_use/protect_user_privacy_with_differential_privacy.md b/tutorials/training/source_en/advanced_use/protect_user_privacy_with_differential_privacy.md index 299f44d107e0c1c8f60353e725418e1aab0ecd9d..5653313d5046bb17377adb961635d4cd89991ad7 100644 --- a/tutorials/training/source_en/advanced_use/protect_user_privacy_with_differential_privacy.md +++ b/tutorials/training/source_en/advanced_use/protect_user_privacy_with_differential_privacy.md @@ -22,7 +22,7 @@ Differential privacy is a mechanism for protecting user data privacy. What is privacy? Privacy refers to the attributes of individual users. Common attributes shared by a group of users may not be considered as privacy. For example, if we say "smoking people have a higher probability of getting lung cancer", it does not disclose privacy. However, if we say "Zhang San smokes and gets lung cancer", it discloses the privacy of Zhang San. Assume that there are 100 patients in a hospital and 10 of them have lung cancer. If the information of any 99 patients are known, we can infer whether the remaining one has lung cancer. This behavior of stealing privacy is called differential attack. Differential privacy is a method for preventing differential attacks. By adding noise, the query results of two datasets with only one different record are nearly indistinguishable. In the above example, after differential privacy is used, the statistic information of the 100 patients achieved by the attacker is almost the same as that of the 99 patients. Therefore, the attacker can hardly infer the information of the remaining one patient. -**Differential privacy in machine learning** +**Differential privacy in machine learning:** Machine learning algorithms usually update model parameters and learn data features based on a large amount of data. Ideally, these models can learn the common features of a class of entities and achieve good generalization, such as "smoking patients are more likely to get lung cancer" rather than models with individual features, such as "Zhang San is a smoker who gets lung cancer." However, machine learning algorithms do not distinguish between general and individual features. The published machine learning models, especially the deep neural networks, may unintentionally memorize and expose the features of individual entities in training data. This can be exploited by malicious attackers to reveal Zhang San's privacy information from the published model. Therefore, it is necessary to use differential privacy to protect machine learning models from privacy leakage. @@ -32,14 +32,14 @@ $Pr[\mathcal{K}(D)\in S] \le e^{\epsilon} Pr[\mathcal{K}(D') \in S]+\delta$ For datasets $D$ and $D'$ that differ on only one record, the probability of obtaining the same result from $\mathcal{K}(D)$ and $\mathcal{K}(D')$ by using a randomized algorithm $\mathcal{K}$ must meet the preceding formula. $\epsilon$ indicates the differential privacy budget and $\delta$ indicates the perturbation. The smaller the values of $\epsilon$ and $\delta$, the closer the data distribution output by $\mathcal{K}$ on $D$ and $D'$. -**Differential privacy measurement** +**Differential privacy measurement:** Differential privacy can be measured using $\epsilon$ and $\delta$. - $\epsilon$: specifies the upper limit of the output probability that can be changed when a record is added to or deleted from the dataset. We usually hope that $\epsilon$ is a small constant. A smaller value indicates stricter differential privacy conditions. - $\delta$: limits the probability of arbitrary model behavior change. Generally, this parameter is set to a small constant. You are advised to set this parameter to a value less than the reciprocal of the size of a training dataset. -**Differential privacy implemented by MindArmour** +**Differential privacy implemented by MindArmour:** MindArmour differential privacy module Differential-Privacy implements the differential privacy optimizer. Currently, SGD, Momentum, and Adam are supported. They are differential privacy optimizers based on the Gaussian mechanism. Gaussian noise mechanism supports both non-adaptive policy and adaptive policy The non-adaptive policy use a fixed noise parameter for each step while the adaptive policy changes the noise parameter along time or iteration step. An advantage of using the non-adaptive Gaussian noise is that a differential privacy budget $\epsilon$ can be strictly controlled. However, a disadvantage is that in a model training process, the noise amount added in each step is fixed. In the later training stage, large noise makes the model convergence difficult, and even causes the performance to decrease greatly and the model usability to be poor. Adaptive noise can solve this problem. In the initial model training stage, the amount of added noise is large. As the model converges, the amount of noise decreases gradually, and the impact of noise on model availability decreases. The disadvantage is that the differential privacy budget cannot be strictly controlled. Under the same initial value, the $\epsilon$ of the adaptive differential privacy is greater than that of the non-adaptive differential privacy. Rényi differential privacy (RDP) [2] is also provided to monitor differential privacy budgets. @@ -84,7 +84,7 @@ TAG = 'Lenet5_train' ### Configuring Parameters 1. Set the running environment, dataset path, model training parameters, checkpoint storage parameters, and differential privacy parameters. Replace 'data_path' with you data path. For more configurations, see . - + ```python cfg = edict({ 'num_classes': 10, # the number of classes of model's output @@ -322,28 +322,29 @@ ds_train = generate_mnist_dataset(os.path.join(cfg.data_path, "train"), acc = model.eval(ds_eval, dataset_sink_mode=False) LOGGER.info(TAG, "============== Accuracy: %s ==============", acc) ``` - + 4. Run the following command. - + Execute the script: - + ```bash python lenet5_dp.py ``` - + In the preceding command, replace `lenet5_dp.py` with the name of your script. - + 5. Display the result. The accuracy of the LeNet model without differential privacy is 99%, and the accuracy of the LeNet model with Gaussian noise and adaptive clip differential privacy is mostly more than 95%. - ``` + + ```text ============== Starting Training ============== ... ============== Starting Testing ============== ... ============== Accuracy: 0.9698 ============== ``` - + ### References [1] C. Dwork and J. Lei. Differential privacy and robust statistics. In STOC, pages 371–380. ACM, 2009. @@ -351,6 +352,3 @@ ds_train = generate_mnist_dataset(os.path.join(cfg.data_path, "train"), [2] Ilya Mironov. Rényi differential privacy. In IEEE Computer Security Foundations Symposium, 2017. [3] Abadi, M. e. a., 2016. *Deep learning with differential privacy.* s.l.:Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. - - - diff --git a/tutorials/training/source_en/advanced_use/save_load_model_hybrid_parallel.md b/tutorials/training/source_en/advanced_use/save_load_model_hybrid_parallel.md index fe7b6d8c7ba28449f73dc473d6e2bd1fc6d49f5f..aa75b23de65b09a999c685bdfdc94563ab8f2762 100644 --- a/tutorials/training/source_en/advanced_use/save_load_model_hybrid_parallel.md +++ b/tutorials/training/source_en/advanced_use/save_load_model_hybrid_parallel.md @@ -90,11 +90,11 @@ Finally, save the updated parameter list to a file through the API provided by M Define the network, call the `load_checkpoint` and `load_param_into_net` APIs to import the checkpoint files to the network in rank id order, and then call `parameters_and_names` API to obtain all parameters in this network. -``` -net = Net() +```python +net = Net() opt = Momentum(learning_rate=0.01, momentum=0.9, params=net.get_parameters()) net = TrainOneStepCell(net, opt) -param_dicts = [] +param_dicts = [] for i in range(rank_size): file_name = os.path.join("./node"+str(i), "CKP_1-4_32.ckpt") # checkpoint file name of current node param_dict = load_checkpoint(file_name) @@ -115,7 +115,7 @@ In the preceding information: Call the `build_searched_strategy` API to obtain the slice strategy of model. -``` +```python strategy = build_searched_strategy("./strategy_train.ckpt") ``` @@ -131,7 +131,7 @@ The parameter name is model\_parallel\_weight and the dividing strategy is to pe 1. Obtain the data value on all nodes for model parallel parameters. - ``` + ```python sliced_parameters = [] for i in range(4): parameter = param_dicts[i].get("model_parallel_weight") @@ -142,32 +142,32 @@ The parameter name is model\_parallel\_weight and the dividing strategy is to pe 2. Call the `merge_sliced_parameter` API to merge the sliced parameters. + ```python + merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) ``` - merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) - ``` - + > If there are multiple model parallel parameters, repeat steps 1 to 2 to process them one by one. ### Saving the Data and Generating a New Checkpoint File 1. Convert `param_dict` to `param_list`. - ``` + ```python param_list = [] for (key, value) in param_dict.items(): each_param = {} each_param["name"] = key if isinstance(value.data, Tensor): - param_data = value.data + param_data = value.data else: - param_data = Tensor(value.data) + param_data = Tensor(value.data) each_param["data"] = param_data param_list.append(each_param） ``` 2. Call the `save_checkpoint` API to write the parameter data to a file and generate a new checkpoint file. - ``` + ```python save_checkpoint(param_list, “./CKP-Integrated_1-4_32.ckpt”) ``` @@ -186,7 +186,7 @@ If you need to load the integrated and saved checkpoint file to multi-device tra Call the `load_checkpoint` API to load model parameter data from the checkpoint file. -``` +```python param_dict = load_checkpoint("./CKP-Integrated_1-4_32.ckpt") ``` @@ -205,7 +205,7 @@ The following uses a specific model parameter as an example. The parameter name In the following code example, data is divided into two slices in dimension 0. - ``` + ```python new_param = parameter_dict[“model_parallel_weight”] slice_list = np.split(new_param.data.asnumpy(), 2, axis=0) new_param_moments = parameter_dict[“moments.model_parallel_weight”] @@ -214,8 +214,10 @@ The following uses a specific model parameter as an example. The parameter name Data after dividing: - slice_list[0] --- [1, 2, 3, 4] Corresponding to device0 - slice_list[1] --- [5, 6, 7, 8] Corresponding to device1 + ```text + slice_list[0] --- [1, 2, 3, 4] Corresponding to device0 + slice_list[1] --- [5, 6, 7, 8] Corresponding to device1 + ``` Similar to slice\_list, slice\_moments\_list is divided into two tensors with the shape of \[1, 4]. @@ -223,7 +225,7 @@ The following uses a specific model parameter as an example. The parameter name Obtain rank\_id of the current node and load data based on rank\_id. - ``` + ```python rank = get_rank() tensor_slice = Tensor(slice_list[rank]) tensor_slice_moments = Tensor(slice_moments_list[rank]) @@ -233,7 +235,7 @@ The following uses a specific model parameter as an example. The parameter name 3. Modify values of model parameters. - ``` + ```python new_param.set_data(tensor_slice, True) new_param_moments.set_data(tensor_slice_moments, True) ``` @@ -244,8 +246,8 @@ The following uses a specific model parameter as an example. The parameter name Call the `load_param_into_net` API to load the model parameter data to the network. -``` -net = Net() +```python +net = Net() opt = Momentum(learning_rate=0.01, momentum=0.9, params=parallel_net.get_parameters()) load_param_into_net(net, param_dict) load_param_into_net(opt, param_dict) @@ -273,40 +275,38 @@ User process: 1. Run the following script to integrate the checkpoint files: - - - ``` + ```python python ./integrate_checkpoint.py "Name of the checkpoint file to be integrated" "Path and name of the checkpoint file generated after integration" "Path and name of the strategy file" "Number of nodes" ``` integrate\_checkpoint.py: - ``` + ```python import numpy as np import os import mindspore.nn as nn from mindspore import Tensor, Parameter from mindspore.ops import operations as P from mindspore.train.serialization import save_checkpoint, load_checkpoint, build_searched_strategy, merge_sliced_parameter - + class Net(nn.Cell): def __init__(self,weight_init): super(Net, self).__init__() self.weight = Parameter(Tensor(weight_init), "model_parallel_weight", layerwise_parallel=True) self.fc = P.MatMul(transpose_b=True) - + def construct(self, x): x = self.fc(x, self.weight1) return x - + def integrate_ckpt_file(old_ckpt_file, new_ckpt_file, strategy_file, rank_size): weight = np.ones([2, 8]).astype(np.float32) net = Net(weight) opt = Momentum(learning_rate=0.01, momentum=0.9, params=net.get_parameters()) net = TrainOneStepCell(net, opt) - + # load CheckPoint into net in rank id order - param_dicts = [] + param_dicts = [] for i in range(rank_size): file_name = os.path.join("./node"+str(i), old_ckpt_file) param_dict = load_checkpoint(file_name) @@ -315,21 +315,21 @@ User process: for _, param in net.parameters_and_names(): param_dict[param.name] = param param_dicts.append(param_dict) - + strategy = build_searched_strategy(strategy_file) param_dict = {} - + for paramname in ["model_parallel_weight", "moments.model_parallel_weight"]: # get layer wise model parallel parameter sliced_parameters = [] for i in range(rank_size): parameter = param_dicts[i].get(paramname) sliced_parameters.append(parameter) - + # merge the parallel parameters of the model - merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) + merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) param_dict[paramname] = merged_parameter - + # convert param_dict to list type data param_list = [] for (key, value) in param_dict.items(): @@ -339,14 +339,14 @@ User process: param_data = value.data else: param_data = Tensor(value.data) - each_param["data"] = param_data - param_list.append(each_param) - + each_param["data"] = param_data + param_list.append(each_param) + # call the API to generate a new CheckPoint file save_checkpoint(param_list, new_ckpt_file) - + return - + if __name__ == "__main__": try: old_ckpt_file = sys.argv[1] @@ -363,10 +363,10 @@ User process: Before the script is executed, the parameter values in the checkpoint files are as follows: - ``` + ```text device0: name is model_parallel_weight - value is + value is [[0.87537426 1.0448935 0.86736983 0.8836905 0.77354026 0.69588304 0.9183654 0.7792076] [0.87224025 0.8726848 0.771446 0.81967723 0.88974726 0.7988162 0.72919345 0.7677011]] name is learning_rate @@ -380,7 +380,7 @@ User process: device1: name is model_parallel_weight - value is + value is [[0.9210751 0.9050457 0.9827775 0.920396 0.9240526 0.9750359 1.0275179 1.0819869] [0.73605865 0.84631145 0.9746683 0.9386582 0.82902765 0.83565056 0.9702136 1.0514659]] name is learning_rate @@ -390,11 +390,11 @@ User process: name is moments.model_weight value is [[0.2417504 0.28193963 0.06713893 0.21510397 0.23380603 0.11424308 0.0218009 -0.11969765] - [0.45955992 0.22664294 0.01990281 0.0731914 0.27125207 0.27298513 -0.01716102 -0.15327111]] + [0.45955992 0.22664294 0.01990281 0.0731914 0.27125207 0.27298513 -0.01716102 -0.15327111]] device2: name is model_parallel_weight - value is + value is [[1.0108461 0.8689414 0.91719437 0.8805056 0.7994629 0.8999671 0.7585804 1.0287056 ] [0.90653455 0.60146594 0.7206475 0.8306303 0.8364681 0.89625114 0.7354735 0.8447268]] name is learning_rate @@ -402,7 +402,7 @@ User process: name is momentum value is [0.9] name is moments.model_weight - value is + value is [[0.03440702 0.41419312 0.24817684 0.30765256 0.48516113 0.24904746 0.57791173 0.00955463] [0.13458519 0.6690533 0.49259356 0.28319967 0.25951773 0.16777472 0.45696738 0.24933104]] @@ -416,16 +416,16 @@ User process: name is momentum value is [0.9] name is moments.model_parallel_weight - value is + value is [[0.14152306 0.5040985 0.24455397 0.10907605 0.11319532 0.19538902 0.01208619 0.40430856] [-0.7773164 -0.47611716 -0.6041424 -0.6144473 -0.2651842 -0.31909415 -0.4510405 -0.12860501]] ``` After the script is executed, the parameter values in the checkpoint files are as follows: - ``` + ```text name is model_parallel_weight - value is + value is [[1.1138763 1.0962057 1.3516843 1.0812817 1.1579804 1.1078343 1.0906502 1.3207073] [0.916671 1.0781671 1.0368758 0.9680898 1.1735439 1.0628364 0.9960786 1.0135143] [0.8828271 0.7963984 0.90675324 0.9830291 0.89010954 0.897052 0.7890109 0.89784735] @@ -439,7 +439,7 @@ User process: name is momentum value is [0.9] name is moments.model_parallel_weight - value is + value is [[0.2567724 -0.07485991 0.282002 0.2456022 0.454939 0.619168 0.18964815 0.45714882] [0.25946522 0.24344791 0.45677605 0.3611395 0.23378398 0.41439137 0.5312468 0.4696194 ] [0.2417504 0.28193963 0.06713893 0.21510397 0.23380603 0.11424308 0.0218009 -0.11969765] @@ -453,7 +453,7 @@ User process: 2. Execute stage 2 training and load the checkpoint file before training. The training code needs to be supplemented based on the site requirements. - ``` + ```python import numpy as np import os import mindspore.nn as nn @@ -497,7 +497,7 @@ User process: load_param_into_net(net, param_dict) opt = Momentum(learning_rate=0.01, momentum=0.9, params=parallel_net.get_parameters()) load_param_into_net(opt, param_dict) - # train code + # train code ... if __name__ == "__main__": @@ -506,7 +506,7 @@ User process: label = np.random.random((4, 4)).astype(np.float32) train_mindspore_impl_fc(input, label, weight1) ``` - + In the preceding information: - `mode=context.GRAPH_MODE`: sets the running mode to graph mode for distributed training. (The PyNative mode does not support parallel running.) @@ -515,10 +515,10 @@ User process: Parameter values after loading: - ``` + ```text device0: name is model_parallel_weight - value is + value is [[0.87537426 1.0448935 0.86736983 0.8836905 0.77354026 0.69588304 0.9183654 0.7792076] [0.87224025 0.8726848 0.771446 0.81967723 0.88974726 0.7988162 0.72919345 0.7677011] [0.8828271 0.7963984 0.90675324 0.9830291 0.89010954 0.897052 0.7890109 0.89784735] @@ -536,7 +536,7 @@ User process: device1: name is model_parallel_weight - value is + value is [[1.0053468 0.98402303 0.99762845 0.97587246 1.0259694 1.0055295 0.99420834 0.9496847] [1.0851002 1.0295962 1.0999886 1.0958165 0.9765328 1.146529 1.0970603 1.1388365] [0.7147005 0.9168278 0.80178416 0.6258351 0.8413766 0.5909515 0.696347 0.71359116] @@ -550,5 +550,5 @@ User process: [[0.03440702 0.41419312 0.24817684 0.30765256 0.48516113 0.24904746 0.57791173 0.00955463] [0.13458519 0.6690533 0.49259356 0.28319967 0.25951773 0.16777472 0.45696738 0.24933104] [0.14152306 0.5040985 0.24455397 0.10907605 0.11319532 0.19538902 0.01208619 0.40430856] - [-0.7773164 -0.47611716 -0.6041424 -0.6144473 -0.2651842 -0.31909415 -0.4510405 -0.12860501]] + [-0.7773164 -0.47611716 -0.6041424 -0.6144473 -0.2651842 -0.31909415 -0.4510405 -0.12860501]] ``` diff --git a/tutorials/training/source_en/advanced_use/summary_record.md b/tutorials/training/source_en/advanced_use/summary_record.md index 133317016f590f38bf63463de75dad948ba0a2c6..eb3760d6759e2d292cd0185be2d3062b9719b0c3 100644 --- a/tutorials/training/source_en/advanced_use/summary_record.md +++ b/tutorials/training/source_en/advanced_use/summary_record.md @@ -41,6 +41,7 @@ The `Callback` mechanism in MindSpore provides a quick and easy way to collect c When you write a training script, you just instantiate the `SummaryCollector` and apply it to either `model.train` or `model.eval`. You can automatically collect some common summary data. The detailed usage of `SummaryCollector` can refer to the `API` document `mindspore.train.callback.SummaryCollector`. The sample code is as follows: + ```python import mindspore import mindspore.nn as nn @@ -126,9 +127,10 @@ model.eval(ds_eval, callbacks=[summary_collector]) ### Method two: Custom collection of network data with summary operators and SummaryCollector -In addition to providing the `SummaryCollector` that automatically collects some summary data, MindSpore provides summary operators that enable customized collection of other data on the network, such as the input of each convolutional layer, or the loss value in the loss function, etc. +In addition to providing the `SummaryCollector` that automatically collects some summary data, MindSpore provides summary operators that enable customized collection of other data on the network, such as the input of each convolutional layer, or the loss value in the loss function, etc. The following summary operators are currently supported: + - [ScalarSummary](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.ops.html#mindspore.ops.ScalarSummary): Record a scalar data. - [TensorSummary](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.ops.html#mindspore.ops.TensorSummary): Record a tensor data. - [ImageSummary](https://www.mindspore.cn/doc/api_python/en/master/mindspore/mindspore.ops.html#mindspore.ops.ImageSummary): Record a image data. @@ -193,7 +195,7 @@ class MyOptimizer(Optimizer): self.histogram_summary(self.weight_names[0], self.paramters[0]) # Record gradient self.histogram_summary(self.weight_names[0] + ".gradient", grads[0]) - + ...... @@ -252,18 +254,18 @@ The detailed usage of `SummaryRecord` can refer to the `API` document `mindspore The sample code is as follows: -``` +```python from mindspore.train.callback import Callback from mindspore.train.summary import SummaryRecord class ConfusionMatrixCallback(Callback): def __init__(self, summary_dir): self._summary_dir = summary_dir - + def __enter__(self): # init you summary record in here, when the train script run, it will be inited before training self.summary_record = SummaryRecord(summary_dir) - + def __exit__(self, *exc_args): # Note: you must close the summary record, it will release the process pool resource # else your training script will not exit from training. @@ -274,7 +276,7 @@ class ConfusionMatrixCallback(Callback): cb_params = run_context.run_context.original_args() # create a confusion matric image, and record it to summary file - confusion_martrix = create_confusion_matrix(cb_params) + confusion_martrix = create_confusion_matrix(cb_params) self.summary_record.add_value('image', 'confusion_matrix', confusion_matric) self.summary_record.record(cb_params.cur_step) @@ -291,24 +293,28 @@ the `save_graphs` option of `context.set_context` in the training script is set In the saved files, `ms_output_after_hwopt.pb` is the computational graph after operator fusion, which can be viewed on the web page. ## Run MindInsight + After completing the data collection in the tutorial above, you can start MindInsight to visualize the collected data. When start MindInsight, you need to specify the summary log file directory with the `--summary-base-dir` parameter. The specified summary log file directory can be the output directory of a training or the parent directory of the output directory of multiple training. The output directory structure for a training is as follows -``` + +```text └─summary_dir events.out.events.summary.1596869898.hostname_MS events.out.events.summary.1596869898.hostname_lineage ``` Execute command: + ```Bash mindinsight start --summary-base-dir ./summary_dir ``` The output directory structure of multiple training is as follows: -``` + +```text └─summary ├─summary_dir1 │ events.out.events.summary.1596869898.hostname_MS @@ -320,6 +326,7 @@ The output directory structure of multiple training is as follows: ``` Execute command: + ```Bash mindinsight start --summary-base-dir ./summary ``` @@ -327,6 +334,7 @@ mindinsight start --summary-base-dir ./summary After successful startup, the visual page can be viewed by visiting the `http://127.0.0.1:8080` address through the browser. Stop MindInsight command: + ```Bash mindinsight stop ``` @@ -339,12 +347,13 @@ For more parameter Settings, see the [MindInsight related commands](https://www. 2. Multiple `SummaryRecord` instances can not be used at the same time. (`SummaryRecord` is used in `SummaryCollector`) - If you use two or more instances of `SummaryCollector` in the callback list of 'model.train' or 'model.eval', it is seen as using multiple `SummaryRecord` instances at the same time, and it will cause recoding data failure. + If you use two or more instances of `SummaryCollector` in the callback list of 'model.train' or 'model.eval', it is seen as using multiple `SummaryRecord` instances at the same time, and it will cause recoding data failure. If the customized callback uses `SummaryRecord`, it can not be used with `SummaryCollector` at the same time. Correct code: - ``` + + ```python ... summary_collector = SummaryCollector('./summary_dir') model.train(2, train_dataset, callbacks=[summary_collector]) @@ -353,7 +362,8 @@ For more parameter Settings, see the [MindInsight related commands](https://www. ``` Wrong code: - ``` + + ```python ... summary_collector1 = SummaryCollector('./summary_dir1') summary_collector2 = SummaryCollector('./summary_dir2') @@ -361,7 +371,8 @@ For more parameter Settings, see the [MindInsight related commands](https://www. ``` Wrong code: - ``` + + ```python ... # Note: the 'ConfusionMatrixCallback' is user-defined, and it uses SummaryRecord to record data. confusion_callback = ConfusionMatrixCallback('./summary_dir1') @@ -371,4 +382,4 @@ For more parameter Settings, see the [MindInsight related commands](https://www. 3. In each Summary log file directory, only one training data should be placed. If a summary log directory contains summary data from multiple training, MindInsight will overlay the summary data from these training when visualizing the data, which may not be consistent with the expected visualizations. -4. Currently, `SummaryCollector` and `SummaryRecord` do not support scenarios with GPU multi-card running. \ No newline at end of file +4. Currently, `SummaryCollector` and `SummaryRecord` do not support scenarios with GPU multi-card running. diff --git a/tutorials/training/source_en/advanced_use/test_model_security_fuzzing.md b/tutorials/training/source_en/advanced_use/test_model_security_fuzzing.md index ae637cdf6d71fff01473574b34e21f70cc708836..8d61b0abdbaceb7871c764641783ca65e5a93f3a 100644 --- a/tutorials/training/source_en/advanced_use/test_model_security_fuzzing.md +++ b/tutorials/training/source_en/advanced_use/test_model_security_fuzzing.md @@ -75,7 +75,7 @@ For details about the API configuration, see the `context.set_context`. images = data[0].asnumpy().astype(np.float32) train_images.append(images) train_images = np.concatenate(train_images, axis=0) - + # get test data data_list = "../common/dataset/MNIST/test" batch_size = 32 @@ -101,7 +101,7 @@ For details about the API configuration, see the `context.set_context`. The data mutation method must include the method based on the image pixel value changes. - The first two image transform methods support user-defined configuration parameters and randomly generated parameters by algorithms. For user-defined configuration parameters see the class methods corresponding to https://gitee.com/mindspore/mindarmour/blob/master/mindarmour/fuzz_testing/image_transform.py. For randomly generated parameters by algorithms you can set method's params to `'auto_param': [True]`. The mutation parameters are randomly generated within the recommended range. + The first two image transform methods support user-defined configuration parameters and randomly generated parameters by algorithms. For user-defined configuration parameters see the class methods corresponding to . For randomly generated parameters by algorithms you can set method's params to `'auto_param': [True]`. The mutation parameters are randomly generated within the recommended range. For details about how to set parameters based on the attack defense method, see the corresponding attack method class. @@ -144,7 +144,7 @@ For details about the API configuration, see the `context.set_context`. # make initial seeds initial_seeds = [] for img, label in zip(test_images, test_labels): - initial_seeds.append([img, label]) + initial_seeds.append([img, label]) initial_seeds = initial_seeds[:100] ``` @@ -174,7 +174,7 @@ For details about the API configuration, see the `context.set_context`. 6. Experiment results. - The results of fuzz testing contains five aspect data: + The results of fuzz testing contains five aspect data: - fuzz_samples: mutated samples in fuzz testing. - true_labels: the ground truth labels of fuzz_samples. @@ -188,8 +188,8 @@ For details about the API configuration, see the `context.set_context`. ```python if metrics: - for key in metrics: - LOGGER.info(TAG, key + ': %s', metrics[key]) + for key in metrics: + LOGGER.info(TAG, key + ': %s', metrics[key]) ``` The fuzz testing result is as follows: diff --git a/tutorials/training/source_en/quick_start/linear_regression.md b/tutorials/training/source_en/quick_start/linear_regression.md index d437a48ee9811ff373afb270101f620a9a325d5e..94dec03fb35e0e9f493722ef72efe8597a2d77bb 100644 --- a/tutorials/training/source_en/quick_start/linear_regression.md +++ b/tutorials/training/source_en/quick_start/linear_regression.md @@ -29,7 +29,6 @@ Author: [Yi Yang](https://github.com/helloyesterday)    Edit    - ## Overview Regression algorithms usually use a series of properties to predict a value, and the predicted values are consecutive. For example, the price of a house is predicted based on some given feature data of the house, such as area and the number of bedrooms; or future temperature conditions are predicted by using the temperature change data and satellite cloud images in the last week. If the actual price of the house is CNY5 million, and the value predicted through regression analysis is CNY4.99 million, the regression analysis is considered accurate. For machine learning problems, common regression analysis includes linear regression, polynomial regression, and logistic regression. This example describes the linear regression algorithms and how to use MindSpore to perform linear regression AI training. @@ -50,7 +49,6 @@ Complete MindSpore running configuration. Third-party support package: `matplotlib`. If this package is not installed, run the `pip install matplotlib` command to install it first. - ```python from mindspore import context @@ -67,7 +65,6 @@ context.set_context(mode=context.GRAPH_MODE, device_target="CPU") `get_data` is used to generate training and test datasets. Since linear data is fitted, the required training datasets should be randomly distributed around the objective function. Assume that the objective function to be fitted is $f(x)=2x+3$. $f(x)=2x+3+noise$ is used to generate training datasets, and `noise` is a random value that complies with standard normal distribution rules. - ```python import numpy as np @@ -81,7 +78,6 @@ def get_data(num, w=2.0, b=3.0): Use `get_data` to generate 50 groups of test data and visualize them. - ```python import matplotlib.pyplot as plt @@ -98,10 +94,8 @@ plt.show() The output is as follows: - ![png](./images/linear_regression_eval_datasets.png) - In the preceding figure, the green line indicates the objective function, and the red points indicate the verification data `eval_data`. ### Defining the Data Argumentation Function @@ -112,7 +106,6 @@ Use the MindSpore data conversion function `GeneratorDataset` to convert the dat - `batch`: combines `batch_size` pieces of data into a batch. - `repeat`: multiplies the number of datasets. - ```python from mindspore import dataset as ds @@ -125,13 +118,12 @@ def create_dataset(num_data, batch_size=16, repeat_size=1): Use the dataset argumentation function to generate training data and view the training data format. - ```python num_data = 1600 batch_size = 16 repeat_size = 1 -ds_train = create_dataset(num_data, batch_size=batch_size, repeat_size=repeat_size) +ds_train = create_dataset(num_data, batch_size=batch_size, repeat_size=repeat_size) print("The dataset size of ds_train:", ds_train.get_dataset_size()) dict_datasets = ds_train.create_dict_iterator().get_next() @@ -142,11 +134,12 @@ print("The y label value shape:", dict_datasets["label"].shape) The output is as follows: - The dataset size of ds_train: 100 - dict_keys(['data', 'label']) - The x label value shape: (16, 1) - The y label value shape: (16, 1) - +```text +The dataset size of ds_train: 100 +dict_keys(['data', 'label']) +The x label value shape: (16, 1) +The y label value shape: (16, 1) +``` Use the defined `create_dataset` to perform argumentation on the generated 1600 data records and set them into 100 datasets with the shape of 16 x 1. @@ -158,7 +151,6 @@ $$f(x)=wx+b\tag{1}$$ Use the Normal operator to randomly initialize the weights $w$ and $b$. - ```python from mindspore.common.initializer import Normal from mindspore import nn @@ -175,7 +167,6 @@ class LinearNet(nn.Cell): Call the network to view the initialized model parameters. - ```python net = LinearNet() model_params = net.trainable_params() @@ -184,18 +175,18 @@ print(model_params) The output is as follows: - [Parameter (name=fc.weight, value=Tensor(shape=[1, 1], dtype=Float32, - [[-7.35660456e-003]])), Parameter (name=fc.bias, value=Tensor(shape=[1], dtype=Float32, [-7.35660456e-003]))] - +```text +[Parameter (name=fc.weight, value=Tensor(shape=[1, 1], dtype=Float32, +[[-7.35660456e-003]])), Parameter (name=fc.bias, value=Tensor(shape=[1], dtype=Float32, [-7.35660456e-003]))] +``` After initializing the network model, visualize the initialized network function and training dataset to understand the model function before fitting. - ```python from mindspore import Tensor x_model_label = np.array([-10, 10, 0.1]) -y_model_label = (x_model_label * Tensor(model_params[0]).asnumpy()[0][0] + +y_model_label = (x_model_label * Tensor(model_params[0]).asnumpy()[0][0] + Tensor(model_params[1]).asnumpy()[0]) plt.scatter(x_eval_label, y_eval_label, color="red", s=5) @@ -206,10 +197,8 @@ plt.show() The output is as follows: - ![png](./images/model_net_and_eval_datasets.png) - As shown in the preceding figure, the initialized model function in blue differs greatly from the objective function in green. ## Defining and Associating the Forward and Backward Propagation Networks @@ -237,7 +226,6 @@ A forward propagation network consists of two parts: The following method is used in MindSpore: - ```python net = LinearNet() net_loss = nn.loss.MSELoss() @@ -258,7 +246,6 @@ Parameters in formula 3 are described as follows: After all weight values in the function are updated, transfer the values to the model function. This process is the backward propagation. To implement this process, the optimizer function in MindSpore is required. - ```python opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) ``` @@ -267,7 +254,6 @@ opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) After forward propagation and backward propagation are defined, call the `Model` function in MindSpore to associate the previously defined networks, loss functions, and optimizer function to form a complete computing network. - ```python from mindspore.train import Model @@ -280,7 +266,6 @@ model = Model(net, net_loss, opt) To make the entire training process easier to understand, the test data, objective function, and model network of the training process need to be visualized. The following defines a visualization function which is called after each training step to display a fitting process of the model network. - ```python import matplotlib.pyplot as plt import time @@ -293,7 +278,7 @@ def plot_model_and_datasets(net, eval_data): x1, y1 = zip(*eval_data) x_target = x y_target = x_target * 2 + 3 - + plt.axis([-11, 11, -20, 25]) plt.scatter(x1, y1, color="red", s=5) plt.plot(x, y, color="blue") @@ -306,7 +291,6 @@ def plot_model_and_datasets(net, eval_data): MindSpore provides tools to customize the model training process. The following calls the visualization function in `step_end` to display the fitting process. For more information, see [Customized Debugging Information](https://www.mindspore.cn/tutorial/training/en/master/advanced_use/custom_debugging_info.html#callback). - ```python from IPython import display from mindspore.train.callback import Callback @@ -315,7 +299,7 @@ class ImageShowCallback(Callback): def __init__(self, net, eval_data): self.net = net self.eval_data = eval_data - + def step_end(self, run_context): plot_model_and_datasets(self.net, self.eval_data) display.clear_output(wait=True) @@ -330,7 +314,6 @@ After the preceding process is complete, use the training parameter `ds_train` t - `callbacks`: Required callback function during training. - `dataset_sink_model`: Dataset offload mode, which supports the Ascend and GPU computing platforms. In this example, this parameter is set to False for the CPU computing platform. - ```python from mindspore.train.callback import LossMonitor @@ -345,13 +328,12 @@ print(net.trainable_params()[0], "\n%s" % net.trainable_params()[1]) The output is as follows: - ![gif](./images/linear_regression.gif) - - Parameter (name=fc.weight, value=[[2.0065749]]) - Parameter (name=fc.bias, value=[3.0089042]) - +```text +Parameter (name=fc.weight, value=[[2.0065749]]) +Parameter (name=fc.bias, value=[3.0089042]) +``` After the training is complete, the weight parameters of the final model are printed. The value of weight is close to 2.0 and the value of bias is close to 3.0. As a result, the model training meets the expectation. diff --git a/tutorials/training/source_en/quick_start/quick_start.md b/tutorials/training/source_en/quick_start/quick_start.md index b7bad45c0f0ef93ab96835c001f95dce81dcb6c3..f9f2dc93c54cfb2295951458a49ed12ab786fa11 100644 --- a/tutorials/training/source_en/quick_start/quick_start.md +++ b/tutorials/training/source_en/quick_start/quick_start.md @@ -32,6 +32,7 @@ This document uses a practice example to demonstrate the basic functions of MindSpore. For common users, it takes 20 to 30 minutes to complete the practice. During the practice, a simple image classification function is implemented. The overall process is as follows: + 1. Process the required dataset. The MNIST dataset is used in this example. 2. Define a network. The LeNet network is used in this example. 3. Define the loss function and optimizer. @@ -39,7 +40,7 @@ During the practice, a simple image classification function is implemented. The 5. Load the saved model for inference. 6. Validate the model, load the test dataset and trained model, and validate the result accuracy. -> You can find the complete executable sample code at . +> You can find the complete executable sample code at . This is a simple and basic workflow. For applying to other advanced and complex applications, extend this basic process as appropriate. @@ -61,7 +62,7 @@ Download the files, decompress them, and store them in the workspace directories The directory structure is as follows: -``` +```text └─MNIST_Data ├─test │ t10k-images.idx3-ubyte @@ -71,6 +72,7 @@ The directory structure is as follows: train-images.idx3-ubyte train-labels.idx1-ubyte ``` + > For ease of use, we added the function of automatically downloading datasets in the sample script. ### Importing Python Libraries and Modules @@ -78,8 +80,7 @@ The directory structure is as follows: Before start, you need to import Python libraries. Currently, the `os` libraries are required. For ease of understanding, other required libraries will not be described here. - - + ```python import os ``` @@ -156,7 +157,7 @@ def create_dataset(data_path, batch_size=32, repeat_size=1, rescale_op = CV.Rescale(rescale, shift) # rescale images hwc2chw_op = CV.HWC2CHW() # change shape from (height, width, channel) to (channel, height, width) to fit network. type_cast_op = C.TypeCast(mstype.int32) # change data type of label to int32 to fit network - + # apply map operations on images mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns="label", num_parallel_workers=num_parallel_workers) mnist_ds = mnist_ds.map(operations=resize_op, input_columns="image", num_parallel_workers=num_parallel_workers) @@ -182,7 +183,6 @@ Perform the shuffle and batch operations, and then perform the repeat operation > MindSpore supports multiple data processing and augmentation operations, which are usually used in combined. For details, see section [Data Processing](https://www.mindspore.cn/tutorial/training/en/master/use/data_preparation.html) in the MindSpore Tutorials. - ## Defining the Network The LeNet network is relatively simple. In addition to the input layer, the LeNet network has seven layers, including two convolutional layers, two down-sample layers (pooling layers), and three full connection layers. Each layer contains different numbers of training parameters, as shown in the following figure: @@ -195,7 +195,7 @@ You can initialize the full connection layers and convolutional layers by `Norma MindSpore supports multiple parameter initialization methods, such as `TruncatedNormal`, `Normal`, and `Uniform`, default value is `Normal`. For details, see the description of the `mindspore.common.initializer` module in the MindSpore API. -To use MindSpore for neural network definition, inherit `mindspore.nn.cell.Cell`. `Cell` is the base class of all neural networks (such as `Conv2d`). +To use MindSpore for neural network definition, inherit `mindspore.nn.Cell`. `Cell` is the base class of all neural networks (such as `Conv2d`). Define each layer of a neural network in the `__init__` method in advance, and then define the `construct` method to complete the forward construction of the neural network. According to the structure of the LeNet network, define the network layers as follows: @@ -237,7 +237,7 @@ class LeNet5(nn.Cell): Before definition, this section briefly describes concepts of loss function and optimizer. - Loss function: It is also called objective function and is used to measure the difference between a predicted value and an actual value. Deep learning reduces the value of the loss function by continuous iteration. Defining a good loss function can effectively improve the model performance. -- Optimizer: It is used to minimize the loss function, improving the model during training. +- Optimizer: It is used to minimize the loss function, improving the model during training. After the loss function is defined, the weight-related gradient of the loss function can be obtained. The gradient is used to indicate the weight optimization direction for the optimizer, improving model performance. @@ -291,9 +291,9 @@ from mindspore.train.callback import ModelCheckpoint, CheckpointConfig if __name__ == "__main__": ... # set parameters of check point - config_ck = CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10) + config_ck = CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10) # apply parameters of check point - ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet", config=config_ck) + ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet", config=config_ck) ... ``` @@ -318,24 +318,27 @@ def train_net(args, model, epoch_size, mnist_path, repeat_size, ckpoint_cb, sink if __name__ == "__main__": ... - - epoch_size = 1 + + epoch_size = 1 mnist_path = "./MNIST_Data" repeat_size = 1 model = Model(network, net_loss, net_opt, metrics={"Accuracy": Accuracy()}) train_net(args, model, epoch_size, mnist_path, repeat_size, ckpoint_cb, dataset_sink_mode) ... ``` -In the preceding information: + +In the preceding information: In the `train_net` method, we loaded the training dataset, `MNIST path` is MNIST dataset path. ## Running and Viewing the Result Run the script using the following command: -``` + +```bash python lenet.py --device_target=CPU ``` -In the preceding information: + +In the preceding information: `Lenet. Py`: the script file you wrote. `--device_target CPU`: Specify the hardware platform.The parameters are 'CPU', 'GPU' or 'Ascend'. @@ -375,7 +378,6 @@ In the preceding information: After obtaining the model file, we verify the generalization ability of the model. - ```python from mindspore.train.serialization import load_checkpoint, load_param_into_net @@ -396,23 +398,24 @@ if __name__ == "__main__": test_net(network, model, mnist_path) ``` -In the preceding information: +In the preceding information: `load_checkpoint`: This API is used to load the CheckPoint model parameter file and return a parameter dictionary. `checkpoint_lenet-3_1404.ckpt`: name of the saved CheckPoint model file. `load_param_into_net`: This API is used to load parameters to the network. - Run the script using the following command: -``` + +```bash python lenet.py --device_target=CPU ``` -In the preceding information: + +In the preceding information: `Lenet. Py`: the script file you wrote. `--device_target CPU`: Specify the hardware platform.The parameters are 'CPU', 'GPU' or 'Ascend'. After executing the command, the result is displayed as follows: -``` +```text ============== Starting Testing ============== ============== Accuracy:{'Accuracy': 0.9663477564102564} ============== ``` diff --git a/tutorials/training/source_en/quick_start/quick_video.md b/tutorials/training/source_en/quick_start/quick_video.md index 05fdbb0749055b1e273d0761fe35aba4a89956b1..4c8b795bfccbd080b9d91d3647b177a89ab20347 100644 --- a/tutorials/training/source_en/quick_start/quick_video.md +++ b/tutorials/training/source_en/quick_start/quick_video.md @@ -108,7 +108,6 @@ Provides video tutorials from installation to try-on, helping you quickly use Mi - ## MindSpore Experience @@ -209,6 +208,14 @@ Provides video tutorials from installation to try-on, helping you quickly use Mi + + + + +## Training Process Visualization-MindInsight + + + @@ -371,4 +402,4 @@ Provides video tutorials from installation to try-on, helping you quickly use Mi - \ No newline at end of file + diff --git a/tutorials/training/source_en/quick_start/quick_video/loading_the_dataset_and_converting_data_format.md b/tutorials/training/source_en/quick_start/quick_video/loading_the_dataset_and_converting_data_format.md index 6e6d6c115eacc4cde4713f3ec63b4d9d9b3f7b92..cdd21d37f314ae52f230eeef6c16721190184ffa 100644 --- a/tutorials/training/source_en/quick_start/quick_video/loading_the_dataset_and_converting_data_format.md +++ b/tutorials/training/source_en/quick_start/quick_video/loading_the_dataset_and_converting_data_format.md @@ -5,4 +5,3 @@ - \ No newline at end of file diff --git a/tutorials/training/source_en/quick_start/quick_video/mindInsight_dashboard.md b/tutorials/training/source_en/quick_start/quick_video/mindInsight_dashboard.md index 52b39f1841978bb0b2129b1fcd87a10c76bdc60b..8a159643e6b0de231dcc83f8835930d6a9586cf9 100644 --- a/tutorials/training/source_en/quick_start/quick_video/mindInsight_dashboard.md +++ b/tutorials/training/source_en/quick_start/quick_video/mindInsight_dashboard.md @@ -8,4 +8,4 @@ **Install now**: -**See more**: \ No newline at end of file +**See more**: diff --git a/tutorials/training/source_en/quick_start/quick_video/mindInsight_installation_and_common_commands.md b/tutorials/training/source_en/quick_start/quick_video/mindInsight_installation_and_common_commands.md index cca2d82e2a0859c8b15a87a4678c1ef3ff08fb6c..a0dae43ea1fb50de88e82e82ad6fac19028aca14 100644 --- a/tutorials/training/source_en/quick_start/quick_video/mindInsight_installation_and_common_commands.md +++ b/tutorials/training/source_en/quick_start/quick_video/mindInsight_installation_and_common_commands.md @@ -8,4 +8,4 @@ **Install now**: -**More commands**: \ No newline at end of file +**More commands**: diff --git a/tutorials/training/source_en/quick_start/quick_video/mindInsight_lineage_and_scalars_comparision.md b/tutorials/training/source_en/quick_start/quick_video/mindInsight_lineage_and_scalars_comparision.md index 4960609f2998245eb4af5c1abb547c45c33c9e8f..5b762e715851344fe9d288882196614060bad2d9 100644 --- a/tutorials/training/source_en/quick_start/quick_video/mindInsight_lineage_and_scalars_comparision.md +++ b/tutorials/training/source_en/quick_start/quick_video/mindInsight_lineage_and_scalars_comparision.md @@ -6,4 +6,4 @@ -**See more**: \ No newline at end of file +**See more**: diff --git a/tutorials/training/source_en/quick_start/quick_video/mindInsight_performance_profiling.md b/tutorials/training/source_en/quick_start/quick_video/mindInsight_performance_profiling.md new file mode 100644 index 0000000000000000000000000000000000000000..953ca8844386844edb24faefb57ce3d69d89e43a --- /dev/null +++ b/tutorials/training/source_en/quick_start/quick_video/mindInsight_performance_profiling.md @@ -0,0 +1,13 @@ +# MindInsight Performance Profiling + +[comment]: <> (This document contains Hands-on Tutorial Series. Gitee does not support display. Please check tutorials on the official website) + + + +**See more**: + + + + \ No newline at end of file diff --git a/tutorials/training/source_en/quick_start/quick_video/quick_start_video.md b/tutorials/training/source_en/quick_start/quick_video/quick_start_video.md index 8af0e535a933d34b8ed7d176ecbf57980d029013..914e35c6a520ebb06589aaf09c87011537a2628c 100644 --- a/tutorials/training/source_en/quick_start/quick_video/quick_start_video.md +++ b/tutorials/training/source_en/quick_start/quick_video/quick_start_video.md @@ -8,4 +8,4 @@ **View code**: -**View the full tutorial**: \ No newline at end of file +**View the full tutorial**: diff --git a/tutorials/training/source_en/quick_start/quick_video/saving_and_loading_model_parameters.md b/tutorials/training/source_en/quick_start/quick_video/saving_and_loading_model_parameters.md index 2fc01870d5cc721dd56d5da841c3564f56ca11c1..78a12a04c0f41419e4afd4d8641be8c5004cb135 100644 --- a/tutorials/training/source_en/quick_start/quick_video/saving_and_loading_model_parameters.md +++ b/tutorials/training/source_en/quick_start/quick_video/saving_and_loading_model_parameters.md @@ -6,4 +6,4 @@ -**View the full tutorial**: \ No newline at end of file +**View the full tutorial**: diff --git a/tutorials/training/source_en/use/load_dataset_image.md b/tutorials/training/source_en/use/load_dataset_image.md index ee57af47a4b2344d9cfc338158ee391ef7a7ec1a..67ea6b6352ccc4650c0069519fae3fcde270b5e7 100644 --- a/tutorials/training/source_en/use/load_dataset_image.md +++ b/tutorials/training/source_en/use/load_dataset_image.md @@ -28,7 +28,7 @@ This tutorial uses the MNIST dataset [1] as an example to demonstrate how to loa 1. Download and decompress the training [Image](http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz) and [Label](http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz) of the MNIST dataset to `./MNIST` directory. The directory structure is as follows. - ``` + ```text └─MNIST ├─train-images.idx3-ubyte └─train-labels.idx1-ubyte diff --git a/tutorials/training/source_en/use/load_dataset_text.md b/tutorials/training/source_en/use/load_dataset_text.md index e63808dbeda5034934dfcce8ea0d4ada6fd6b7d8..e4e9c34cdc20a1e91092794357d21f7ca658ca25 100644 --- a/tutorials/training/source_en/use/load_dataset_text.md +++ b/tutorials/training/source_en/use/load_dataset_text.md @@ -27,7 +27,7 @@ This tutorial briefly demonstrates how to load and process text data using MindS 1. Prepare the following text data. - ``` + ```text Welcome to Beijing! 北京欢迎您！ 我喜欢English! @@ -35,7 +35,7 @@ This tutorial briefly demonstrates how to load and process text data using MindS 2. Create the `tokenizer.txt` file, copy the text data to the file, and save the file under `./test` directory. The directory structure is as follow. - ``` + ```text └─test └─tokenizer.txt ``` @@ -69,7 +69,7 @@ The following tutorial demonstrates loading datasets using the `TextFileDataset` The output without tokenization: - ``` + ```text Welcome to Beijing! 北京欢迎您！ 我喜欢English! @@ -99,7 +99,7 @@ The following tutorial demonstrates how to perform data processing such as `Slid The output is as follows: - ``` + ```text ['大', '家', '早', '上', '好'] ``` @@ -118,7 +118,7 @@ The following tutorial demonstrates how to perform data processing such as `Slid The output is as follows: - ``` + ```text [['大', '家'], ['家', '早'], ['早', '上'], @@ -145,7 +145,7 @@ The following tutorial demonstrates how to perform data processing such as `Slid The output is as follows: - ``` + ```text c a d @@ -178,7 +178,7 @@ The following tutorial demonstrates how to use the `WhitespaceTokenizer` to toke The output after tokenization is as follows: - ``` + ```text ['Welcome', 'to', 'Beijing!'] ['北京欢迎您！'] ['我喜欢English!'] diff --git a/tutorials/training/source_en/use/load_model_for_inference_and_transfer.md b/tutorials/training/source_en/use/load_model_for_inference_and_transfer.md index d402b51700d83d2c95d8d7fcc0cbe57062becb22..8e6e891492d0a263b87a3b19835aea474c9d95db 100644 --- a/tutorials/training/source_en/use/load_model_for_inference_and_transfer.md +++ b/tutorials/training/source_en/use/load_model_for_inference_and_transfer.md @@ -1,4 +1,4 @@ -# Loading a Model for Inference and Transfer Learning +# Loading a Model for Inference and Transfer Learning `Linux` `Ascend` `GPU` `CPU` `Model Loading` `Beginner` `Intermediate` `Expert` @@ -50,6 +50,7 @@ The `eval` method validates the accuracy of the trained model. In the retraining and fine-tuning scenarios for task interruption, you can load network parameters and optimizer parameters to the model. The sample code is as follows: + ```python # return a parameter dict for model param_dict = load_checkpoint("resnet50-2_32.ckpt") @@ -105,11 +106,11 @@ The `load_checkpoint` method returns a parameter dictionary and then the `load_p ### For Transfer Training -When loading a model with `mindspore_hub.load` API, we can add an extra argument to load the feature extraction part of the model only. So we can easily add new layers to perform transfer learning. This feature can be found in the related model page when an extra argument (e.g., include_top) has been integrated into the model construction by the model developer. The value of `include_top` is True or False, indicating whether to keep the top layer in the fully-connected network. +When loading a model with `mindspore_hub.load` API, we can add an extra argument to load the feature extraction part of the model only. So we can easily add new layers to perform transfer learning. This feature can be found in the related model page when an extra argument (e.g., include_top) has been integrated into the model construction by the model developer. The value of `include_top` is True or False, indicating whether to keep the top layer in the fully-connected network. -We use GoogleNet as example to illustrate how to load a model trained on ImageNet dataset and then perform transfer learning (re-training) on specific sub-task dataset. The main steps are listed below: +We use GoogleNet as example to illustrate how to load a model trained on ImageNet dataset and then perform transfer learning (re-training) on specific sub-task dataset. The main steps are listed below: -1. Search the model of interest on [MindSpore Hub Website](https://www.mindspore.cn/resources/hub/) and get the related `url`. +1. Search the model of interest on [MindSpore Hub Website](https://www.mindspore.cn/resources/hub/) and get the related `url`. 2. Load the model from MindSpore Hub using the `url`. Note that the parameter `include_top` is provided by the model developer. @@ -142,7 +143,7 @@ We use GoogleNet as example to illustrate how to load a model trained on ImageNe super(ReduceMeanFlatten, self).__init__() self.mean = P.ReduceMean(keep_dims=True) self.flatten = nn.Flatten() - + def construct(self, x): x = self.mean(x, (2, 3)) x = self.flatten(x) @@ -197,7 +198,7 @@ We use GoogleNet as example to illustrate how to load a model trained on ImageNe data, label = items data = mindspore.Tensor(data) label = mindspore.Tensor(label) - + loss = train_net(data, label) print(f"epoch: {epoch}/{epoch_size}, loss: {loss}") # Save the ckpt file for each epoch. @@ -218,7 +219,7 @@ We use GoogleNet as example to illustrate how to load a model trained on ImageNe classification_layer = nn.Dense(last_channel, num_classes) classification_layer.set_train(False) softmax = nn.Softmax() - network = nn.SequentialCell([network, reducemean_flatten, + network = nn.SequentialCell([network, reducemean_flatten, classification_layer, softmax]) # Load a pre-trained ckpt file. @@ -237,4 +238,4 @@ We use GoogleNet as example to illustrate how to load a model trained on ImageNe res = model.eval(eval_dataset) print("result:", res, "ckpt=", ckpt_path) - ``` \ No newline at end of file + ``` diff --git a/tutorials/training/source_en/use/publish_model.md b/tutorials/training/source_en/use/publish_model.md index 2d208ae971327c20bda76fd21d1e6545ec501525..fc0e39991d8292a0861d16430ae1cbc77b265d51 100644 --- a/tutorials/training/source_en/use/publish_model.md +++ b/tutorials/training/source_en/use/publish_model.md @@ -14,15 +14,15 @@ ## Overview -[MindSpore Hub](https://www.mindspore.cn/resources/hub/) is a platform for storing pre-trained models provided by MindSpore or third-party developers. It provides application developers with simple model loading and fine-tuning APIs, which enables the users to perform inference or fine-tuning based on the pre-trained models and thus deploy to their own applications. Users can also submit their pre-trained models into MindSpore Hub following the specific steps. Thus other users can download and use the published models. +[MindSpore Hub](https://www.mindspore.cn/resources/hub/) is a platform for storing pre-trained models provided by MindSpore or third-party developers. It provides application developers with simple model loading and fine-tuning APIs, which enables the users to perform inference or fine-tuning based on the pre-trained models and thus deploy to their own applications. Users can also submit their pre-trained models into MindSpore Hub following the specific steps. Thus other users can download and use the published models. This tutorial uses GoogleNet as an example to describe how to submit models for model developers who are interested in publishing models into MindSpore Hub. ## How to publish models -You can publish models to MindSpore Hub via PR in [hub](https://gitee.com/mindspore/hub) repo. Here we use GoogleNet as an example to list the steps of model submission to MindSpore Hub. +You can publish models to MindSpore Hub via PR in [hub](https://gitee.com/mindspore/hub) repo. Here we use GoogleNet as an example to list the steps of model submission to MindSpore Hub. -1. Host your pre-trained model in a storage location where we are able to access. +1. Host your pre-trained model in a storage location where we are able to access. 2. Add a model generation python file called `mindspore_hub_conf.py` in your own repo using this [template](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/googlenet/mindspore_hub_conf.py). The location of the `mindspore_hub_conf.py` file is shown below: @@ -47,11 +47,11 @@ You can publish models to MindSpore Hub via PR in [hub](https://gitee.com/mindsp | ├── gpu | ├── 0.7 | ├── ascend - | ├── 0.7 + | ├── 0.7 | ├── googlenet_v1_cifar10.md │   ├── tools | ├── md_validator.py - | └── md_validator.py + | └── md_validator.py ``` Note that it is required to fill in the `{model_name}_{model_version}_{dataset}.md` template by providing `file-format`、`asset-link` and `asset-sha256` below, which refers to the model file format, model storage location from step 1 and model hash value, respectively. @@ -60,7 +60,7 @@ You can publish models to MindSpore Hub via PR in [hub](https://gitee.com/mindsp file-format: ckpt asset-link: https://download.mindspore.cn/model_zoo/official/cv/googlenet/goolenet_ascend_0.2.0_cifar10_official_classification_20200713/googlenet.ckpt asset-sha256: 114e5acc31dad444fa8ed2aafa02ca34734419f602b9299f3b53013dfc71b0f7 - ``` + ``` The MindSpore Hub supports multiple model file formats including: - [MindSpore CKPT](https://www.mindspore.cn/tutorial/training/en/master/use/save_model.html#checkpoint-configuration-policies) @@ -81,6 +81,6 @@ You can publish models to MindSpore Hub via PR in [hub](https://gitee.com/mindsp python md_validator.py ../assets/mindspore/ascend/0.7/googlenet_v1_cifar10.md ``` -5. Create a PR in `mindspore/hub` repo. See our [Contributor Wiki](https://gitee.com/mindspore/mindspore/blob/master/CONTRIBUTING.md#) for more information about creating a PR. +5. Create a PR in `mindspore/hub` repo. See our [Contributor Wiki](https://gitee.com/mindspore/mindspore/blob/master/CONTRIBUTING.md#) for more information about creating a PR. -Once your PR is merged into master branch here, your model will show up in [MindSpore Hub Website](https://www.mindspore.cn/resources/hub) within 24 hours. Please refer to [README](https://gitee.com/mindspore/hub/blob/master/mshub_res/README.md#) for more information about model submission. +Once your PR is merged into master branch here, your model will show up in [MindSpore Hub Website](https://www.mindspore.cn/resources/hub) within 24 hours. Please refer to [README](https://gitee.com/mindspore/hub/blob/master/mshub_res/README.md#) for more information about model submission. diff --git a/tutorials/training/source_en/use/save_model.md b/tutorials/training/source_en/use/save_model.md index d21e33e2e0ce98b402c65e0494d44990da5189ba..98b4938598d87bdd39d394827194e953efd13b25 100644 --- a/tutorials/training/source_en/use/save_model.md +++ b/tutorials/training/source_en/use/save_model.md @@ -35,6 +35,7 @@ During model training, use the callback mechanism to transfer the object of the You can use the `CheckpointConfig` object to set the CheckPoint saving policies. The saved parameters are classified into network parameters and optimizer parameters. `ModelCheckpoint` provides default configuration policies for users to quickly get started. The following describes the usage: + ```python from mindspore.train.callback import ModelCheckpoint ckpoint_cb = ModelCheckpoint() @@ -61,7 +62,7 @@ Create a `ModelCheckpoint` object and transfer it to the model.train method. The Generated CheckPoint files are as follows: -``` +```text resnet50-graph.meta # Generate compiled computation graph. resnet50-1_32.ckpt # The file name extension is .ckpt. resnet50-2_32.ckpt # The file name format contains the epoch and step correspond to the saved parameters. @@ -100,6 +101,7 @@ When you have a CheckPoint file, if you want to do inference on device, you need If you want to do inference on the device, then you need to generate corresponding MINDIR models based on the network and CheckPoint. Currently we support the export of MINDIR models for inference based on graph mode, which don't contain control flow. Taking the export of MINDIR model as an example to illustrate the implementation of model export, the code is as follows: + ```python from mindspore.train.serialization import export import numpy as np @@ -133,7 +135,7 @@ input = np.random.uniform(0.0, 1.0, size=[32, 3, 224, 224]).astype(np.float32) export(resnet, Tensor(input), file_name='resnet50-2_32.air', file_format='AIR') ``` -Before using the `export` interface, you need to import` mindspore.train.serialization`. +Before using the `export` interface, you need to import`mindspore.train.serialization`. The `input` parameter is used to specify the input shape and the data type of the exported model. diff --git a/tutorials/training/source_zh_cn/advanced_use/apply_deep_probability_programming.md b/tutorials/training/source_zh_cn/advanced_use/apply_deep_probability_programming.md index da0cbc368bf90e5b97ac2f204a6c6a7ee6dee537..9c03363c1ad8e6894b2e83f3745fe5259f9dbbbf 100644 --- a/tutorials/training/source_zh_cn/advanced_use/apply_deep_probability_programming.md +++ b/tutorials/training/source_zh_cn/advanced_use/apply_deep_probability_programming.md @@ -1,4 +1,5 @@ # 深度概率编程 + `Ascend` `GPU` `全流程` `初级` `中级` `高级` @@ -29,16 +30,20 @@ ## 概述 + 深度学习模型具有强大的拟合能力，而贝叶斯理论具有很好的可解释能力。MindSpore深度概率编程（MindSpore Deep Probabilistic Programming, MDP）将深度学习和贝叶斯学习结合，通过设置网络权重为分布、引入隐空间分布等，可以对分布进行采样前向传播，由此引入了不确定性，从而增强了模型的鲁棒性和可解释性。MDP不仅包含通用、专业的概率学习编程语言，适用于“专业”用户，而且支持使用开发深度学习模型的逻辑进行概率编程，让初学者轻松上手；此外，还提供深度概率学习的工具箱，拓展贝叶斯应用功能。 本章将详细介绍深度概率编程在MindSpore上的应用。在动手进行实践之前，确保，你已经正确安装了MindSpore 0.7.0-beta及其以上版本。本章的具体内容如下： + 1. 介绍如何使用[bnn_layers模块](https://gitee.com/mindspore/mindspore/tree/master/mindspore/nn/probability/bnn_layers)实现贝叶斯神经网（Bayesian Neural Network, BNN）； 2. 介绍如何使用[variational模块](https://gitee.com/mindspore/mindspore/tree/master/mindspore/nn/probability/infer/variational)和[dpn模块](https://gitee.com/mindspore/mindspore/tree/master/mindspore/nn/probability/dpn)实现变分自编码器（Variational AutoEncoder, VAE）； 3. 介绍如何使用[transforms模块](https://gitee.com/mindspore/mindspore/tree/master/mindspore/nn/probability/transforms)实现DNN（Deep Neural Network, DNN）一键转BNN； 4. 介绍如何使用[toolbox模块](https://gitee.com/mindspore/mindspore/blob/master/mindspore/nn/probability/toolbox/uncertainty_evaluation.py)实现不确定性估计。 ## 使用贝叶斯神经网络 + 贝叶斯神经网络是由概率模型和神经网络组成的基本模型，它的权重不再是一个确定的值，而是一个分布。本例介绍了如何使用MDP中的bnn_layers模块实现贝叶斯神经网络，并利用贝叶斯神经网络实现一个简单的图片分类功能，整体流程如下： + 1. 处理MNIST数据集； 2. 定义贝叶斯LeNet网络； 3. 定义损失函数和优化器； @@ -47,12 +52,14 @@ > 本例面向GPU或Ascend 910 AI处理器平台，你可以在这里下载完整的样例代码： ### 处理数据集 + 本例子使用的是MNIST数据集，数据处理过程与教程中的[实现一个图片分类应用](https://www.mindspore.cn/tutorial/training/zh-CN/master/quick_start/quick_start.html)一致。 ### 定义贝叶斯神经网络 + 本例使用的是Bayesian LeNet。利用bnn_layers模块构建贝叶斯神经网络的方法与构建普通的神经网络相同。值得注意的是，`bnn_layers`和普通的神经网络层可以互相组合。 -``` +```python import mindspore.nn as nn from mindspore.nn.probability import bnn_layers import mindspore.ops.operations as P @@ -98,7 +105,9 @@ class BNNLeNet5(nn.Cell): x = self.fc3(x) return x ``` + ### 定义损失函数和优化器 + 接下来需要定义损失函数（Loss）和优化器（Optimizer）。损失函数是深度学习的训练目标，也叫目标函数，可以理解为神经网络的输出（Logits）和标签(Labels)之间的距离，是一个标量数据。 常见的损失函数包括均方误差、L2损失、Hinge损失、交叉熵等等。图像分类应用通常采用交叉熵损失（CrossEntropy）。 @@ -107,7 +116,7 @@ class BNNLeNet5(nn.Cell): MindSpore中定义损失函数和优化器的代码样例如下： -``` +```python # loss function definition criterion = SoftmaxCrossEntropyWithLogits(sparse=True, reduction="mean") @@ -116,9 +125,10 @@ optimizer = AdamWeightDecay(params=network.trainable_params(), learning_rate=0.0 ``` ### 训练网络 + 贝叶斯神经网络的训练过程与DNN基本相同，唯一不同的是将`WithLossCell`替换为适用于BNN的`WithBNNLossCell`。除了`backbone`和`loss_fn`两个参数之外，`WithBNNLossCell`增加了`dnn_factor`和`bnn_factor`两个参数。`dnn_factor`是由损失函数计算得到的网络整体损失的系数，`bnn_factor`是每个贝叶斯层的KL散度的系数，这两个参数是用来平衡网络整体损失和贝叶斯层的KL散度的，防止KL散度的值过大掩盖了网络整体损失。 -``` +```python net_with_loss = bnn_layers.WithBNNLossCell(network, criterion, dnn_factor=60000, bnn_factor=0.000001) train_bnn_network = TrainOneStepCell(net_with_loss, optimizer) train_bnn_network.set_train() @@ -136,9 +146,10 @@ for i in range(epoch): print('Epoch: {} \tTraining Loss: {:.4f} \tTraining Accuracy: {:.4f} \tvalidation Accuracy: {:.4f}'. format(i, train_loss, train_acc, valid_acc)) ``` + 其中，`train_model`和`validate_model`在MindSpore中的代码样例如下： -``` +```python def train_model(train_net, net, dataset): accs = [] loss_sum = 0 @@ -172,17 +183,22 @@ def validate_model(net, dataset): ``` ## 使用变分自编码器 + 接下来介绍如何使用MDP中的variational模块和dpn模块实现变分自编码器。变分自编码器是经典的应用了变分推断的深度概率模型，用来学习潜在变量的表示，通过该模型，不仅可以压缩输入数据，还可以生成该类型的新图像。本例的整体流程如下： + 1. 定义变分自编码器； 2. 定义损失函数和优化器； 3. 处理数据； 4. 训练网络； 5. 生成新样本或重构输入样本。 + > 本例面向GPU或Ascend 910 AI处理器平台，你可以在这里下载完整的样例代码： + ### 定义变分自编码器 + 使用dpn模块来构造变分自编码器尤为简单，你只需要自定义编码器和解码器（DNN模型），调用`VAE`接口即可。 -``` +```python class Encoder(nn.Cell): def __init__(self): super(Encoder, self).__init__() @@ -218,11 +234,13 @@ encoder = Encoder() decoder = Decoder() vae = VAE(encoder, decoder, hidden_size=400, latent_size=20) ``` + ### 定义损失函数和优化器 + 接下来需要定义损失函数（Loss）和优化器（Optimizer）。本例使用的损失函数是`ELBO`，`ELBO`是变分推断专用的损失函数；本例使用的优化器是`Adam`。 MindSpore中定义损失函数和优化器的代码样例如下： -``` +```python # loss function definition net_loss = ELBO(latent_prior='Normal', output_prior='Normal') @@ -231,24 +249,30 @@ optimizer = nn.Adam(params=vae.trainable_params(), learning_rate=0.001) net_with_loss = nn.WithLossCell(vae, net_loss) ``` + ### 处理数据 + 本例使用的是MNIST数据集，数据处理过程与教程中的[实现一个图片分类应用](https://www.mindspore.cn/tutorial/training/zh-CN/master/quick_start/quick_start.html)一致。 ### 训练网络 + 使用variational模块中的`SVI`接口对VAE网络进行训练。 -``` +```python from mindspore.nn.probability.infer import SVI vi = SVI(net_with_loss=net_with_loss, optimizer=optimizer) vae = vi.run(train_dataset=ds_train, epochs=10) trained_loss = vi.get_train_loss() ``` + 通过`vi.run`可以得到训练好的网络，使用`vi.get_train_loss`可以得到训练之后的损失。 + ### 生成新样本或重构输入样本 + 利用训练好的VAE网络，我们可以生成新的样本或重构输入样本。 -``` +```python IMAGE_SHAPE = (-1, 1, 32, 32) generated_sample = vae.generate_sample(64, IMAGE_SHAPE) for sample in ds_train.create_dict_iterator(): @@ -257,16 +281,21 @@ for sample in ds_train.create_dict_iterator(): ``` ## DNN一键转换成BNN + 对于不熟悉贝叶斯模型的DNN研究人员，MDP提供了高级API`TransformToBNN`，支持DNN模型一键转换成BNN模型。目前在LeNet，ResNet，MobileNet，VGG等模型上验证了API的通用性。本例将会介绍如何使用transforms模块中的`TransformToBNN`API实现DNN一键转换成BNN，整体流程如下： + 1. 定义DNN模型； 2. 定义损失函数和优化器； 3. 实现功能一：转换整个模型； 4. 实现功能二：转换指定类型的层。 + > 本例面向GPU或Ascend 910 AI处理器平台，你可以在这里下载完整的样例代码： + ### 定义DNN模型 + 本例使用的DNN模型是LeNet。 -``` +```python from mindspore.common.initializer import TruncatedNormal import mindspore.nn as nn import mindspore.ops.operations as P @@ -332,9 +361,10 @@ class LeNet5(nn.Cell): x = self.fc3(x) return x ``` + LeNet的网络结构如下： -``` +```text LeNet5 (conv1) Conv2dinput_channels=1, output_channels=6, kernel_size=(5, 5),stride=(1, 1), pad_mode=valid, padding=0, dilation=(1, 1), group=1, has_bias=False (conv2) Conv2dinput_channels=6, output_channels=16, kernel_size=(5, 5),stride=(1, 1), pad_mode=valid, padding=0, dilation=(1, 1), group=1, has_bias=False @@ -347,9 +377,10 @@ LeNet5 ``` ### 定义损失函数和优化器 + 接下来需要定义损失函数（Loss）和优化器（Optimizer）。本例使用交叉熵损失作为损失函数，`Adam`作为优化器。 -``` +```python network = LeNet5() # loss function definition @@ -361,10 +392,12 @@ optimizer = AdamWeightDecay(params=network.trainable_params(), learning_rate=0.0 net_with_loss = WithLossCell(network, criterion) train_network = TrainOneStepCell(net_with_loss, optimizer) ``` + ### 实例化TransformToBNN + `TransformToBNN`的`__init__`函数定义如下： -``` +```python class TransformToBNN: def __init__(self, trainable_dnn, dnn_factor=1, bnn_factor=1): net_with_loss = trainable_dnn.network @@ -375,18 +408,21 @@ class TransformToBNN: self.bnn_factor = bnn_factor self.bnn_loss_file = None ``` + 参数`trainable_bnn`是经过`TrainOneStepCell`包装的可训练DNN模型，`dnn_factor`和`bnn_factor`分别为由损失函数计算得到的网络整体损失的系数和每个贝叶斯层的KL散度的系数。 MindSpore中实例化`TransformToBNN`的代码如下： -``` +```python from mindspore.nn.probability import transforms bnn_transformer = transforms.TransformToBNN(train_network, 60000, 0.000001) ``` + ### 实现功能一：转换整个模型 + `transform_to_bnn_model`方法可以将整个DNN模型转换为BNN模型。其定义如下: -``` +```python def transform_to_bnn_model(self, get_dense_args=lambda dp: {"in_channels": dp.in_channels, "has_bias": dp.has_bias, "out_channels": dp.out_channels, "activation": dp.activation}, @@ -413,90 +449,94 @@ bnn_transformer = transforms.TransformToBNN(train_network, 60000, 0.000001) Cell, a trainable BNN model wrapped by TrainOneStepCell. """ ``` + 参数`get_dense_args`指定从DNN模型的全连接层中获取哪些参数，`get_conv_args`指定从DNN模型的卷积层中获取哪些参数，参数`add_dense_args`和`add_conv_args`分别指定了要为BNN层指定哪些新的参数值。需要注意的是，`add_dense_args`中的参数不能与`get_dense_args`重复，`add_conv_args`和`get_conv_args`也是如此。 在MindSpore中将整个DNN模型转换成BNN模型的代码如下： -``` +```python train_bnn_network = bnn_transformer.transform_to_bnn_model() ``` + 整个模型转换后的结构如下： -``` +```text LeNet5 (conv1) ConvReparam in_channels=1, out_channels=6, kernel_size=(5, 5), stride=(1, 1), pad_mode=valid, padding=0, dilation=(1, 1), group=1, weight_mean=Parameter (name=conv1.weight_posterior.mean), weight_std=Parameter (name=conv1.weight_posterior.untransformed_std), has_bias=False (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (conv2) ConvReparam in_channels=6, out_channels=16, kernel_size=(5, 5), stride=(1, 1), pad_mode=valid, padding=0, dilation=(1, 1), group=1, weight_mean=Parameter (name=conv2.weight_posterior.mean), weight_std=Parameter (name=conv2.weight_posterior.untransformed_std), has_bias=False (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (fc1) DenseReparam in_channels=400, out_channels=120, weight_mean=Parameter (name=fc1.weight_posterior.mean), weight_std=Parameter (name=fc1.weight_posterior.untransformed_std), has_bias=True, bias_mean=Parameter (name=fc1.bias_posterior.mean), bias_std=Parameter (name=fc1.bias_posterior.untransformed_std) (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - + (bias_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (bias_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (fc2) DenseReparam in_channels=120, out_channels=84, weight_mean=Parameter (name=fc2.weight_posterior.mean), weight_std=Parameter (name=fc2.weight_posterior.untransformed_std), has_bias=True, bias_mean=Parameter (name=fc2.bias_posterior.mean), bias_std=Parameter (name=fc2.bias_posterior.untransformed_std) (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - + (bias_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (bias_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (fc3) DenseReparam in_channels=84, out_channels=10, weight_mean=Parameter (name=fc3.weight_posterior.mean), weight_std=Parameter (name=fc3.weight_posterior.untransformed_std), has_bias=True, bias_mean=Parameter (name=fc3.bias_posterior.mean), bias_std=Parameter (name=fc3.bias_posterior.untransformed_std) (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - + (bias_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (bias_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (relu) ReLU (max_pool2d) MaxPool2dkernel_size=2, stride=2, pad_mode=VALID (flatten) Flatten ``` + 可以看到，整个LeNet网络中的卷积层和全连接层都转变成了相应的贝叶斯层。 ### 实现功能二：转换指定类型的层 + `transform_to_bnn_layer`方法可以将DNN模型中指定类型的层（nn.Dense或者nn.Conv2d）转换为对应的贝叶斯层。其定义如下: -``` +```python def transform_to_bnn_layer(self, dnn_layer, bnn_layer, get_args=None, add_args=None): r""" Transform a specific type of layers in DNN model to corresponding BNN layer. @@ -513,16 +553,18 @@ LeNet5 Cell, a trainable model wrapped by TrainOneStepCell, whose sprcific type of layer is transformed to the corresponding bayesian layer. """ ``` + 参数`dnn_layer`指定将哪个类型的DNN层转换成BNN层，`bnn_layer`指定DNN层将转换成哪个类型的BNN层，`get_args`和`add_args`分别指定从DNN层中获取哪些参数和要为BNN层的哪些参数重新赋值。 在MindSpore中将DNN模型中的Dense层转换成相应贝叶斯层`DenseReparam`的代码如下： -``` +```python train_bnn_network = bnn_transformer.transform_to_bnn_layer(nn.Dense, bnn_layers.DenseReparam) ``` + 转换后网络的结构如下： -``` +```text LeNet5 (conv1) Conv2dinput_channels=1, output_channels=6, kernel_size=(5, 5),stride=(1, 1), pad_mode=valid, padding=0, dilation=(1, 1), group=1, has_bias=False (conv2) Conv2dinput_channels=6, output_channels=16, kernel_size=(5, 5),stride=(1, 1), pad_mode=valid, padding=0, dilation=(1, 1), group=1, has_bias=False @@ -530,55 +572,58 @@ LeNet5 in_channels=400, out_channels=120, weight_mean=Parameter (name=fc1.weight_posterior.mean), weight_std=Parameter (name=fc1.weight_posterior.untransformed_std), has_bias=True, bias_mean=Parameter (name=fc1.bias_posterior.mean), bias_std=Parameter (name=fc1.bias_posterior.untransformed_std) (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - + (bias_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (bias_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (fc2) DenseReparam in_channels=120, out_channels=84, weight_mean=Parameter (name=fc2.weight_posterior.mean), weight_std=Parameter (name=fc2.weight_posterior.untransformed_std), has_bias=True, bias_mean=Parameter (name=fc2.bias_posterior.mean), bias_std=Parameter (name=fc2.bias_posterior.untransformed_std) (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - + (bias_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (bias_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (fc3) DenseReparam in_channels=84, out_channels=10, weight_mean=Parameter (name=fc3.weight_posterior.mean), weight_std=Parameter (name=fc3.weight_posterior.untransformed_std), has_bias=True, bias_mean=Parameter (name=fc3.bias_posterior.mean), bias_std=Parameter (name=fc3.bias_posterior.untransformed_std) (weight_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (weight_posterior) NormalPosterior (normal) Normalbatch_shape = None - + (bias_prior) NormalPrior (normal) Normalmean = 0.0, standard deviation = 0.1 - + (bias_posterior) NormalPosterior (normal) Normalbatch_shape = None - - + + (relu) ReLU (max_pool2d) MaxPool2dkernel_size=2, stride=2, pad_mode=VALID (flatten) Flatten ``` + 可以看到，LeNet网络中的卷积层保持不变，全连接层变成了对应的贝叶斯层`DenseReparam`。 ## 使用不确定性估计工具箱 + 贝叶斯神经网络的优势之一就是可以获取不确定性，MDP在上层提供了不确定性估计的工具箱，用户可以很方便地使用该工具箱计算不确定性。不确定性意味着深度学习模型对预测结果的不确定程度。目前，大多数深度学习算法只能给出预测结果，而不能判断预测结果的可靠性。不确定性主要有两种类型：偶然不确定性和认知不确定性。 + - 偶然不确定性（Aleatoric Uncertainty）：描述数据中的内在噪声，即无法避免的误差，这个现象不能通过增加采样数据来削弱。 - 认知不确定性（Epistemic Uncertainty）：模型自身对输入数据的估计可能因为训练不佳、训练数据不够等原因而不准确，可以通过增加训练数据等方式来缓解。 @@ -587,7 +632,7 @@ LeNet5 以分类任务为例，本例中使用的模型是LeNet，数据集为MNIST，数据处理过程与教程中的[实现一个图片分类应用](https://www.mindspore.cn/tutorial/training/zh-CN/master/quick_start/quick_start.html)一致。为了评估测试示例的不确定性，使用工具箱的方法如下: -``` +```python from mindspore.nn.probability.toolbox.uncertainty_evaluation import UncertaintyEvaluation from mindspore.train.serialization import load_checkpoint, load_param_into_net @@ -610,4 +655,3 @@ for eval_data in ds_eval.create_dict_iterator(): epistemic_uncertainty = evaluation.eval_epistemic_uncertainty(eval_data) aleatoric_uncertainty = evaluation.eval_aleatoric_uncertainty(eval_data) ``` - diff --git a/tutorials/training/source_zh_cn/advanced_use/apply_gradient_accumulation.md b/tutorials/training/source_zh_cn/advanced_use/apply_gradient_accumulation.md index 96eccdf5381618c39603cbdb1e265adff5831238..38897c1dd87c1a4b9ab515fbc673a15108fd0324 100644 --- a/tutorials/training/source_zh_cn/advanced_use/apply_gradient_accumulation.md +++ b/tutorials/training/source_zh_cn/advanced_use/apply_gradient_accumulation.md @@ -36,6 +36,7 @@ 以MNIST作为示范数据集，自定义简单模型实现梯度累积。 ### 导入需要的库文件 + 下列是我们所需要的公共模块及MindSpore的模块及库文件。 ```python @@ -65,7 +66,9 @@ from model_zoo.official.cv.lenet.src.lenet import LeNet5 这里以LeNet网络为例进行介绍，当然也可以使用其它的网络，如ResNet-50、BERT等, 此部分代码由`model_zoo`中`lenet`目录下的[lenet.py]()导入。 ### 定义训练模型 + 将训练流程拆分为正向反向训练、参数更新和累积梯度清理三个部分： + - `TrainForwardBackward`计算loss和梯度，利用grad_sum实现梯度累加。 - `TrainOptim`实现参数更新。 - `TrainClear`实现对梯度累加变量grad_sum清零。 @@ -134,6 +137,7 @@ class TrainClear(Cell): ``` ### 定义训练过程 + 每个Mini-batch通过正反向训练计算loss和梯度，通过mini_steps控制每次更新参数前的累加次数。达到累加次数后进行参数更新和 累加梯度变量清零。 @@ -202,6 +206,7 @@ class GradientAccumulation: ``` ### 训练并保存模型 + 调用网络、优化器及损失函数，然后自定义`GradientAccumulation`的`train_process`接口，进行模型训练。 ```python @@ -226,13 +231,15 @@ if __name__ == "__main__": ``` ## 实验结果 + 在经历了10轮epoch之后，在测试集上的精度约为96.31%。 -**执行训练** +**执行训练：** + 1. 运行训练代码，查看运行结果。 ```shell - $ python train.py --data_path=./MNIST_Data + python train.py --data_path=./MNIST_Data ``` 输出如下，可以看到loss值随着训练逐步降低： @@ -247,17 +254,17 @@ if __name__ == "__main__": epoch: 10 step: 448 loss is 0.06443884 epoch: 10 step: 449 loss is 0.0067842817 ``` - + 2. 查看保存的CheckPoint文件。 训练过程中保存了CheckPoint文件`gradient_accumulation.ckpt`，即模型文件。 -**验证模型** +**验证模型：** 通过`model_zoo`中`lenet`目录下的[eval.py]()，使用保存的CheckPoint文件，加载验证数据集，进行验证。 ```shell -$ python eval.py --data_path=./MNIST_Data --ckpt_path=./gradient_accumulation.ckpt --device_target=GPU +python eval.py --data_path=./MNIST_Data --ckpt_path=./gradient_accumulation.ckpt --device_target=GPU ``` 输出如下，可以看到使用验证的数据集，正确率在96.31%左右，与batch_size为32的验证结果一致。 diff --git a/tutorials/training/source_zh_cn/advanced_use/apply_host_device_training.md b/tutorials/training/source_zh_cn/advanced_use/apply_host_device_training.md index 6e8b3ebc390223daee20e7496b853b21a27cf2d2..c171c55efb9ab4d016ebbeecb634bc52c4bfafc2 100644 --- a/tutorials/training/source_zh_cn/advanced_use/apply_host_device_training.md +++ b/tutorials/training/source_zh_cn/advanced_use/apply_host_device_training.md @@ -47,19 +47,23 @@ ## 配置混合执行 1. 配置混合训练标识。在`src/config.py`文件中，设置`argparse_init`函数中的`host_device_mix`默认值为`1`，设置`WideDeepConfig`类的`__init__`函数中`self.host_device_mix`为`1`： + ```python self.host_device_mix = 1 ``` 2. 检查必要算子和优化器的执行位置。在`src/wide_and_deep.py`的`WideDeepModel`类中，检查`EmbeddingLookup`为主机端执行： + ```python self.deep_embeddinglookup = nn.EmbeddingLookup() self.wide_embeddinglookup = nn.EmbeddingLookup() ``` + 在`src/wide_and_deep.py`文件的`class TrainStepWrap(nn.Cell)`中，检查两个优化器主机端执行的属性。 + ```python - self.optimizer_w.sparse_opt.add_prim_attr("primitive_target", "CPU") - self.optimizer_d.sparse_opt.add_prim_attr("primitive_target", "CPU") + self.optimizer_w.target = "CPU" + self.optimizer_d.target = "CPU" ``` ## 训练模型 @@ -69,7 +73,7 @@ 运行日志保存在`device_0`目录下，其中`loss.log`保存一个epoch内中多个loss值，其值类似如下： -``` +```text epoch: 1 step: 1, wide_loss is 0.6873926, deep_loss is 0.8878349 epoch: 1 step: 2, wide_loss is 0.6442529, deep_loss is 0.8342661 epoch: 1 step: 3, wide_loss is 0.6227323, deep_loss is 0.80273706 @@ -84,7 +88,7 @@ epoch: 1 step: 10, wide_loss is 0.566089, deep_loss is 0.6884129 `test_deep0.log`保存pytest进程输出的详细的运行时日志（需要将日志级别设置为INFO，且在MindSpore编译时加上-p on选项），搜索关键字`EmbeddingLookup`，可找到如下信息： -``` +```text [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:34.928.275 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/network-VirtualDatasetCellTriple/_backbone-NetWithLossClass/network-WideDeepModel/EmbeddingLookup-op297 costs 3066 us. [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:34.943.896 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/network-VirtualDatasetCellTriple/_backbone-NetWithLossClass/network-WideDeepModel/EmbeddingLookup-op298 costs 15521 us. ``` @@ -92,7 +96,7 @@ epoch: 1 step: 10, wide_loss is 0.566089, deep_loss is 0.6884129 表示`EmbeddingLookup`在主机端的执行时间。 继续在`test_deep0.log`搜索关键字`FusedSparseFtrl`和`FusedSparseLazyAdam`，可找到如下信息： -``` +```text [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:35.422.963 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/optimizer_w-FTRL/FusedSparseFtrl-op299 costs 54492 us. [INFO] DEVICE(109904,python3.7):2020-06-27-12:42:35.565.953 [mindspore/ccsrc/device/cpu/cpu_kernel_runtime.cc:324] Run] cpu kernel: Default/optimizer_d-LazyAdam/FusedSparseLazyAdam-op300 costs 142865 us. ``` diff --git a/tutorials/training/source_zh_cn/advanced_use/apply_parameter_server_training.md b/tutorials/training/source_zh_cn/advanced_use/apply_parameter_server_training.md index 6fab9ad461d8518db4f4f7beda17402b60408996..e586363322df7a9e32aadc504fff8234587bb636 100644 --- a/tutorials/training/source_zh_cn/advanced_use/apply_parameter_server_training.md +++ b/tutorials/training/source_zh_cn/advanced_use/apply_parameter_server_training.md @@ -17,6 +17,7 @@ ## 概述 + Parameter Server(参数服务器)是分布式训练中一种广泛使用的架构，相较于同步的AllReduce训练方法，Parameter Server具有更好的灵活性、可扩展性以及节点容灾的能力。具体来讲，参数服务器既支持同步SGD，也支持异步SGD的训练算法；在扩展性上，将模型的计算与模型的更新分别部署在Worker和Server两类进程中，使得Worker和Server的资源可以独立地横向扩缩；另外，在大规模数据中心的环境下，计算设备、网络以及存储经常会出现各种故障而导致部分节点异常，而在参数服务器的架构下，能够较为容易地处理此类的故障而不会对训练中的任务产生影响。 在MindSpore的参数服务器实现中，采用了开源的[ps-lite](https://github.com/dmlc/ps-lite)作为基础架构，基于其提供的远程通信能力以及抽象的Push/Pull原语，实现了同步SGD的分布式训练算法，另外结合Ascend和GPU中的高性能集合通信库(HCCL和NCCL)，MindSpore还提供了Parameter Server和AllReduce的混合训练模式，支持将部分权重通过参数服务器进行存储和更新，其余权重仍然通过AllReduce算法进行训练。 @@ -30,6 +31,7 @@ Parameter Server(参数服务器)是分布式训练中一种广泛使用的架 - Scheduler：用于建立Server和Worker的通信关系。 ## 准备工作 + 以LeNet在Ascend 910上使用Parameter Server训练为例： ### 训练脚本准备 @@ -51,6 +53,7 @@ Parameter Server(参数服务器)是分布式训练中一种广泛使用的架 - 被设置为通过Parameter Server更新的单个权重大小不得超过INT_MAX(2^31 - 1)字节。 3. 在[原训练脚本](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/lenet/train.py)基础上，设置LeNet模型所有权重通过Parameter Server训练： + ```python context.set_ps_context(enable_ps=True) network = LeNet5(cfg.num_classes) @@ -61,7 +64,7 @@ Parameter Server(参数服务器)是分布式训练中一种广泛使用的架 MindSpore通过读取环境变量，控制Parameter Server训练，环境变量包括以下选项(其中`MS_SCHED_HOST`及`MS_SCHED_PORT`所有脚本需保持一致)： -``` +```text export PS_VERBOSE=1 # Print ps-lite log export MS_SERVER_NUM=1 # Server number export MS_WORKER_NUM=1 # Worker number @@ -90,6 +93,7 @@ export MS_ROLE=MS_SCHED # The role of this process: MS_SCHED repre ``` `Server.sh`: + ```bash #!/bin/bash export PS_VERBOSE=1 @@ -102,6 +106,7 @@ export MS_ROLE=MS_SCHED # The role of this process: MS_SCHED repre ``` `Worker.sh`: + ```bash #!/bin/bash export PS_VERBOSE=1 @@ -114,26 +119,31 @@ export MS_ROLE=MS_SCHED # The role of this process: MS_SCHED repre ``` 最后分别执行： + ```bash sh Scheduler.sh > scheduler.log 2>&1 & sh Server.sh > server.log 2>&1 & sh Worker.sh > worker.log 2>&1 & ``` + 启动训练 2. 查看结果 查看`scheduler.log`中Server与Worker通信日志： - ``` + + ```text Bind to role=scheduler, id=1, ip=XXX.XXX.XXX.XXX, port=XXXX Assign rank=8 to node role=server, ip=XXX.XXX.XXX.XXX, port=XXXX Assign rank=9 to node role=worker, ip=XXX.XXX.XXX.XXX, port=XXXX the scheduler is connected to 1 workers and 1 servers ``` + 说明Server、Worker与Scheduler通信建立成功。 查看`worker.log`中训练结果： - ``` + + ```text epoch: 1 step: 1, loss is 2.302287 epoch: 1 step: 2, loss is 2.304071 epoch: 1 step: 3, loss is 2.308778 diff --git a/tutorials/training/source_zh_cn/advanced_use/apply_quantization_aware_training.md b/tutorials/training/source_zh_cn/advanced_use/apply_quantization_aware_training.md index d9a6c653a7a4c2b292b2be8213658408aae6c4fe..14183e2221312cf8e652117fa5fe2e93f06afc2e 100644 --- a/tutorials/training/source_zh_cn/advanced_use/apply_quantization_aware_training.md +++ b/tutorials/training/source_zh_cn/advanced_use/apply_quantization_aware_training.md @@ -39,6 +39,7 @@ ### 伪量化节点 伪量化节点，是指感知量化训练中插入的节点，用以寻找网络数据分布，并反馈损失精度，具体作用如下： + - 找到网络数据的分布，即找到待量化参数的最大值和最小值； - 模拟量化为低比特时的精度损失，把该损失作用到网络模型中，传递给损失函数，让优化器在训练过程中对该损失值进行优化。 @@ -59,12 +60,12 @@ MindSpore的感知量化训练是在训练基础上，使用低精度数据替 感知量化训练模型与一般训练步骤一致，在定义网络和最后生成模型阶段后，需要进行额外的操作，完整流程如下： -1. 数据处理加载数据集。 -2. 定义原始非量化网络。 -3. 定义融合网络。在完成定义原始非量化网络后，替换指定的算子，完成融合网络的定义。 -4. 定义优化器和损失函数。 -5. 转化量化网络。基于融合网络，使用转化接口在融合网络中插入伪量化节点，生成量化网络。 -6. 进行量化训练。基于量化网络训练，生成量化模型。 +1. 数据处理加载数据集。 +2. 定义原始非量化网络。 +3. 定义融合网络。在完成定义原始非量化网络后，替换指定的算子，完成融合网络的定义。 +4. 定义优化器和损失函数。 +5. 转化量化网络。基于融合网络，使用转化接口在融合网络中插入伪量化节点，生成量化网络。 +6. 进行量化训练。基于量化网络训练，生成量化模型。 在上面流程中，第3、5、6步是感知量化训练区别普通训练需要额外进行的步骤。 @@ -99,7 +100,7 @@ class LeNet5(nn.Cell): Tensor, output tensor Examples: >>> LeNet(num_class=10, num_channel=1) - + """ def __init__(self, num_class=10, num_channel=1): super(LeNet5, self).__init__() @@ -129,10 +130,10 @@ class LeNet5(nn.Cell): def __init__(self, num_class=10): super(LeNet5, self).__init__() self.num_class = num_class - + self.conv1 = nn.Conv2dBnAct(1, 6, kernel_size=5, activation='relu') self.conv2 = nn.Conv2dBnAct(6, 16, kernel_size=5, activation='relu') - + self.fc1 = nn.DenseBnAct(16 * 5 * 5, 120, activation='relu') self.fc2 = nn.DenseBnAct(120, 84, activation='relu') self.fc3 = nn.DenseBnAct(84, self.num_class) @@ -164,13 +165,13 @@ net = quant.convert_quant_network(network, quant_delay=900, bn_fold=False, per_c 上面介绍了从零开始进行感知量化训练。更常见情况是已有一个模型文件，希望生成量化模型，这时已有正常网络模型训练得到的模型文件及训练脚本，进行感知量化训练。这里使用checkpoint文件重新训练的功能，详细步骤为： - 1. 数据处理加载数据集。 - 2. 定义原始非量化网络。 - 3. 训练原始网络生成非量化模型。 - 4. 定义融合网络。 - 5. 定义优化器和损失函数。 - 6. 基于融合网络转化生成量化网络。 - 7. 加载模型文件重训。加载已有非量化模型文件，基于量化网络重新训练生成量化模型。详细模型重载训练，请参见。 + 1. 数据处理加载数据集。 + 2. 定义原始非量化网络。 + 3. 训练原始网络生成非量化模型。 + 4. 定义融合网络。 + 5. 定义优化器和损失函数。 + 6. 基于融合网络转化生成量化网络。 + 7. 加载模型文件重训。加载已有非量化模型文件，基于量化网络重新训练生成量化模型。详细模型重载训练，请参见。 ### 进行推理 @@ -180,11 +181,11 @@ net = quant.convert_quant_network(network, quant_delay=900, bn_fold=False, per_c - 使用感知量化训练后得到的checkpoint文件进行推理： - 1. 加载量化模型。 - 2. 推理。 + 1. 加载量化模型。 + 2. 推理。 - 转化为ONNX等通用格式进行推理（暂不支持，开发完善后补充）。 - + ## 参考文献 [1] Jacob B, Kligys S, Chen B, et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 2704-2713. diff --git a/tutorials/training/source_zh_cn/advanced_use/convert_dataset.md b/tutorials/training/source_zh_cn/advanced_use/convert_dataset.md index b025f48e7066029a21b553d09b1a91f663ab9efc..1e60f89a8cf434850c5b5b95327bcc05324cb2f2 100644 --- a/tutorials/training/source_zh_cn/advanced_use/convert_dataset.md +++ b/tutorials/training/source_zh_cn/advanced_use/convert_dataset.md @@ -16,9 +16,10 @@ ## 概述 -用户可以将非标准的数据集和常用的数据集转换为MindSpore数据格式，即MindRecord，从而方便地加载到MindSpore中进行训练。同时，MindSpore在部分场景做了性能优化，使用MindSpore数据格式可以获得更好的性能。 +用户可以将非标准的数据集和常用的数据集转换为MindSpore数据格式，即MindRecord，从而方便地加载到MindSpore中进行训练。同时，MindSpore在部分场景做了性能优化，使用MindSpore数据格式可以获得更好的性能。 MindSpore数据格式具备的特征如下： + 1. 实现多变的用户数据统一存储、访问，训练数据读取更简便； 2. 数据聚合存储，高效读取，且方便管理、移动； 3. 高效数据编解码操作，对用户透明、无感知； @@ -96,7 +97,7 @@ MindSpore数据格式的目标是归一化用户的数据集，并进一步通 5. 创建`FileWriter`对象，传入文件名及分片数量，然后添加Schema文件及索引，调用`write_raw_data`接口写入数据，最后调用`commit`接口生成本地数据文件。 - ```python + ```python writer = FileWriter(file_name="test.mindrecord", shard_num=4) writer.add_schema(cv_schema_json, "test_schema") writer.add_index(indexes) @@ -141,7 +142,7 @@ MindSpore数据格式的目标是归一化用户的数据集，并进一步通 输出结果如下： - ``` + ```text sample: {'data': array([175, 175, 85, 60, 184, 124, 54, 189, 125, 193, 153, 91, 234, 106, 43, 143, 132, 211, 204, 160, 44, 105, 187, 185, 45, 205, 122, 236, 112, 123, 84, 177, 219], dtype=uint8), 'file_name': array(b'3.jpg', dtype='|S5'), 'label': array(99, dtype=int32)} diff --git a/tutorials/training/source_zh_cn/advanced_use/custom_debugging_info.md b/tutorials/training/source_zh_cn/advanced_use/custom_debugging_info.md index 0d933a30b9f5d3699e9e7f0ce515c232a0901a98..9a9584f1767e620302058e3743eb3cb1febdc27c 100644 --- a/tutorials/training/source_zh_cn/advanced_use/custom_debugging_info.md +++ b/tutorials/training/source_zh_cn/advanced_use/custom_debugging_info.md @@ -41,7 +41,7 @@ MindSpore提供`Callback`能力，支持用户在训练/推理的特定阶段， 使用方法：在`model.train`方法中传入`Callback`对象，它可以是一个`Callback`列表，例： ```python -ckpt_cb = ModelCheckpoint() +ckpt_cb = ModelCheckpoint() loss_cb = LossMonitor() summary_cb = SummaryCollector(summary_dir='./summary_dir') model.train(epoch, dataset, callbacks=[ckpt_cb, loss_cb, summary_cb]) @@ -60,7 +60,7 @@ model.train(epoch, dataset, callbacks=[ckpt_cb, loss_cb, summary_cb]) ```python class Callback(): - """Callback base class""" + """Callback base class""" def begin(self, run_context): """Called once before the network executing.""" pass @@ -70,11 +70,11 @@ class Callback(): pass def epoch_end(self, run_context): - """Called after each epoch finished.""" + """Called after each epoch finished.""" pass def step_begin(self, run_context): - """Called before each epoch beginning.""" + """Called before each epoch beginning.""" pass def step_end(self, run_context): @@ -131,7 +131,7 @@ class Callback(): 输出： - ``` + ```text epoch: 20 step: 32 loss: 2.298344373703003 ``` @@ -172,7 +172,6 @@ class Callback(): 具体实现逻辑为：定义一个`Callback`对象，初始化对象接收`model`对象和`ds_eval`(验证数据集)。在`step_end`阶段验证模型的精度，当精度为当前最高时，手动触发保存checkpoint方法，保存当前的参数。 - ## MindSpore metrics功能介绍 当训练结束后，可以使用metrics评估训练结果的好坏。 @@ -224,13 +223,17 @@ print('Accuracy is ', accuracy) ``` 输出： -``` + +```text Accuracy is 0.6667 ``` + ## Print算子功能介绍 + MindSpore的自研`Print`算子可以将用户输入的Tensor或字符串信息打印出来，支持多字符串输入，多Tensor输入和字符串与Tensor的混合输入，输入参数以逗号隔开。 `Print`算子使用方法与其他算子相同，在网络中的`__init__`声明算子并在`construct`进行调用，具体使用实例及输出结果如下： + ```python import numpy as np from mindspore import Tensor @@ -254,8 +257,10 @@ y = Tensor(np.ones([2, 2]).astype(np.int32)) net = PrintDemo() output = net(x, y) ``` + 输出： -``` + +```text print Tensor x and Tensor y: Tensor shape:[[const vector][2, 1]]Int32 val:[[1] @@ -315,7 +320,7 @@ val:[[1 1] 3. 执行用例Dump数据。 可以在训练脚本中设置`context.set_context(reserve_class_name_in_scope=False)`，避免Dump文件名称过长导致Dump数据文件生成失败。 4. 解析Dump数据。 - + 通过`numpy.fromfile`读取Dump数据文件即可解析。 ### 异步Dump功能介绍 @@ -372,6 +377,7 @@ val:[[1 1] ``` ## 日志相关的环境变量和配置 + MindSpore采用glog来输出日志，常用的几个环境变量如下： - `GLOG_v` @@ -379,13 +385,13 @@ MindSpore采用glog来输出日志，常用的几个环境变量如下： 该环境变量控制日志的级别。 该环境变量默认值为2，即WARNING级别，对应关系如下：0-DEBUG、1-INFO、2-WARNING、3-ERROR。 -- `GLOG_logtostderr` +- `GLOG_logtostderr` 该环境变量控制日志的输出方式。 该环境变量的值设置为1时，日志输出到屏幕；值设置为0时，日志输出到文件。默认值为1。 -- `GLOG_log_dir` - +- `GLOG_log_dir` + 该环境变量指定日志输出的路径。 若`GLOG_logtostderr`的值为0，则必须设置此变量。 若指定了`GLOG_log_dir`且`GLOG_logtostderr`的值为1时，则日志输出到屏幕，不输出到文件。 @@ -428,6 +434,3 @@ MindSpore子模块按照目录划分如下： | mindspore/core/ | CORE | > glog不支持日志文件的绕接，如果需要控制日志文件对磁盘空间的占用，可选用操作系统提供的日志文件管理工具，例如：Linux的logrotate。 - - - diff --git a/tutorials/training/source_zh_cn/advanced_use/custom_operator_ascend.md b/tutorials/training/source_zh_cn/advanced_use/custom_operator_ascend.md index ed1a76fb97058cb30837566de9128166f1405011..6bd704a066c9206985ab3888079438e0bbc7ea67 100644 --- a/tutorials/training/source_zh_cn/advanced_use/custom_operator_ascend.md +++ b/tutorials/training/source_zh_cn/advanced_use/custom_operator_ascend.md @@ -25,6 +25,7 @@ 添加一个自定义算子，需要完成算子原语注册、算子实现、算子信息注册三部分工作。 其中： + - 算子原语：定义了算子在网络中的前端接口原型，也是组成网络模型的基础单元，主要包括算子的名称、属性（可选）、输入输出名称、输出shape推理方法、输出dtype推理方法等信息。 - 算子实现：通过TBE（Tensor Boost Engine）提供的特性语言接口，描述算子内部计算逻辑的实现。TBE提供了开发昇腾AI芯片自定义算子的能力。你可以在页面申请公测。 - 算子信息：描述TBE算子的基本信息，如算子名称、支持的输入输出类型等。它是后端做算子选择和映射时的依据。 @@ -38,6 +39,7 @@ 每个算子的原语是一个继承于`PrimitiveWithInfer`的子类，其类型名称即是算子名称。 自定义算子原语与内置算子原语的接口定义完全一致： + - 属性由构造函数`__init__`的入参定义。本用例的算子没有属性，因此`__init__`没有额外的入参。带属性的用例可参考MindSpore源码中的[custom add3](https://gitee.com/mindspore/mindspore/blob/master/tests/st/ops/custom_ops_tbe/cus_add3.py)用例。 - 输入输出的名称通过`init_prim_io_names`函数定义。 - 输出Tensor的shape推理方法在`infer_shape`函数中定义，输出Tensor的dtype推理方法在`infer_dtype`函数中定义。 @@ -75,10 +77,12 @@ class CusSquare(PrimitiveWithInfer): 算子的计算函数主要用来封装算子的计算逻辑供主函数调用，其内部通过调用TBE的API接口组合实现算子的计算逻辑。 算子的入口函数描述了编译算子的内部过程，一般分为如下几步： + 1. 准备输入的placeholder，placeholder是一个占位符，返回一个Tensor对象，表示一组输入数据。 2. 调用计算函数，计算函数使用TBE提供的API接口描述了算子内部的计算逻辑。 3. 调用Schedule调度模块，调度模块对算子中的数据按照调度模块的调度描述进行切分，同时指定好数据的搬运流程，确保在硬件上的执行达到最优。默认可以采用自动调度模块（`auto_schedule`）。 4. 调用`cce_build_code`编译生成算子二进制。 + > 入口函数的输入参数有特殊要求，需要依次为：算子每个输入的信息、算子每个输出的信息、算子属性（可选）和`kernel_name`（生成算子二进制的名称）。输入和输出的信息用字典封装传入，其中包含该算子在网络中被调用时传入的实际输入和输出的shape和dtype。 更多关于使用TBE开发算子的内容请参考[TBE文档](https://support.huaweicloud.com/odevg-A800_3000_3010/atlaste_10_0063.html)，关于TBE算子的调试和性能优化请参考[MindStudio文档](https://support.huaweicloud.com/usermanual-mindstudioc73/atlasmindstudio_02_0043.html)。 @@ -92,7 +96,7 @@ class CusSquare(PrimitiveWithInfer): ### 示例 -下面以`Square`算子的TBE实现`square_impl.py`为例进行介绍。`square_compute`是算子实现的计算函数，通过调用`te.lang.cce`提供的API描述了`x * x`的计算逻辑。`cus_square_op_info `是算子信息，通过`TBERegOp`来定义。 +下面以`Square`算子的TBE实现`square_impl.py`为例进行介绍。`square_compute`是算子实现的计算函数，通过调用`te.lang.cce`提供的API描述了`x * x`的计算逻辑。`cus_square_op_info`是算子信息，通过`TBERegOp`来定义。 `TBERegOp`的设置需要注意以下几点： @@ -128,7 +132,7 @@ cus_square_op_info = TBERegOp("CusSquare") \ .output(0, "y", False, "required", "all") \ .dtype_format(DataType.F32_Default, DataType.F32_Default) \ .dtype_format(DataType.F16_Default, DataType.F16_Default) \ - .get_op_info() + .get_op_info() # Binding kernel info with the kernel implementation. @op_info_register(cus_square_op_info) @@ -185,17 +189,20 @@ def test_net(): ``` 执行用例: -``` + +```bash pytest -s tests/st/ops/custom_ops_tbe/test_square.py::test_net ``` 执行结果: -``` + +```text x: [1. 4. 9.] output: [1. 16. 81.] ``` ## 定义算子反向传播函数 + 如果算子要支持自动微分，需要在其原语中定义其反向传播函数（bprop）。你需要在bprop中描述利用正向输入、正向输出和输出梯度得到输入梯度的反向计算逻辑。反向计算逻辑可以使用内置算子或自定义反向算子构成。 定义算子反向传播函数时需注意以下几点： @@ -204,6 +211,7 @@ output: [1. 16. 81.] - bprop函数的返回值形式约定为输入梯度组成的元组，元组中元素的顺序与正向输入参数顺序一致。即使只有一个输入梯度，返回值也要求是元组的形式。 例如，增加bprop后的`CusSquare`原语为： + ```python class CusSquare(PrimitiveWithInfer): @prim_attr_register @@ -228,6 +236,7 @@ class CusSquare(PrimitiveWithInfer): ``` 在`test_square.py`文件中定义反向用例。 + ```python from mindspore.ops import composite as C def test_grad_net(): @@ -241,12 +250,14 @@ def test_grad_net(): ``` 执行用例: -``` + +```bash pytest -s tests/st/ops/custom_ops_tbe/test_square.py::test_grad_net ``` 执行结果: -``` + +```text x: [1. 4. 9.] dx: [2. 8. 18.] ``` diff --git a/tutorials/training/source_zh_cn/advanced_use/cv_mobilenetv2_fine_tune.md b/tutorials/training/source_zh_cn/advanced_use/cv_mobilenetv2_fine_tune.md index e206515f68a906e939db574b88f0748ce9c3cf05..1743089f80e081542ecf6a5061668d84fbeb0e52 100644 --- a/tutorials/training/source_zh_cn/advanced_use/cv_mobilenetv2_fine_tune.md +++ b/tutorials/training/source_zh_cn/advanced_use/cv_mobilenetv2_fine_tune.md @@ -293,26 +293,26 @@ Windows系统输出信息到交互式命令行，Linux系统环境下运行`run_ - 开始增量训练 - - 使用样例1：通过Python文件调用1个GPU处理器。 + - 使用样例1：通过Python文件调用1个GPU处理器。 - ```bash - # Windows or Linux with Python - python train.py --platform GPU --dataset_path [TRAIN_DATASET_PATH] --pretrain_ckpt ./pretrain_checkpoint/mobilenetv2_cpu_gpu.ckpt --freeze_layer backbone - ``` + ```bash + # Windows or Linux with Python + python train.py --platform GPU --dataset_path [TRAIN_DATASET_PATH] --pretrain_ckpt ./pretrain_checkpoint/mobilenetv2_cpu_gpu.ckpt --freeze_layer backbone + ``` - - 使用样例2：通过Shell脚本调用1个GPU处理器，设备ID为`“0”`。 + - 使用样例2：通过Shell脚本调用1个GPU处理器，设备ID为`“0”`。 - ```bash - # Linux with Shell - sh run_train.sh GPU 1 0 [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_cpu_gpu.ckpt backbone - ``` + ```bash + # Linux with Shell + sh run_train.sh GPU 1 0 [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_cpu_gpu.ckpt backbone + ``` - - 使用样例3：通过Shell脚本调用8个GPU处理器，设备ID为`“0,1,2,3,4,5,6,7”`。 + - 使用样例3：通过Shell脚本调用8个GPU处理器，设备ID为`“0,1,2,3,4,5,6,7”`。 - ```bash - # Linux with Shell - sh run_train.sh GPU 8 0,1,2,3,4,5,6,7 [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_cpu_gpu.ckpt backbone - ``` + ```bash + # Linux with Shell + sh run_train.sh GPU 8 0,1,2,3,4,5,6,7 [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_cpu_gpu.ckpt backbone + ``` ### Ascend加载训练 @@ -322,68 +322,68 @@ Windows系统输出信息到交互式命令行，Linux系统环境下运行`run_ - 开始增量训练 - - 使用样例1：通过Python文件调用1个Ascend处理器。 + - 使用样例1：通过Python文件调用1个Ascend处理器。 - ```bash - # Windows or Linux with Python - python train.py --platform Ascend --dataset_path [TRAIN_DATASET_PATH] --pretrain_ckpt ./pretrain_checkpoint mobilenetv2_ascend.ckpt --freeze_layer backbone - ``` + ```bash + # Windows or Linux with Python + python train.py --platform Ascend --dataset_path [TRAIN_DATASET_PATH] --pretrain_ckpt ./pretrain_checkpoint mobilenetv2_ascend.ckpt --freeze_layer backbone + ``` - - 使用样例2：通过Shell脚本调用1个Ascend AI处理器，设备ID为“0”。 + - 使用样例2：通过Shell脚本调用1个Ascend AI处理器，设备ID为“0”。 - ```bash - # Linux with Shell - sh run_train.sh Ascend 1 0 ~/rank_table.json [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_ascend.ckpt backbone - ``` + ```bash + # Linux with Shell + sh run_train.sh Ascend 1 0 ~/rank_table.json [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_ascend.ckpt backbone + ``` - - 使用样例3：通过Shell脚本调用8个Ascend AI处理器，设备ID为”0,1,2,3,4,5,6,7“。 + - 使用样例3：通过Shell脚本调用8个Ascend AI处理器，设备ID为”0,1,2,3,4,5,6,7“。 - ```bash - # Linux with Shell - sh run_train.sh Ascend 8 0,1,2,3,4,5,6,7 ~/rank_table.json [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_ascend.ckpt backbone - ``` + ```bash + # Linux with Shell + sh run_train.sh Ascend 8 0,1,2,3,4,5,6,7 ~/rank_table.json [TRAIN_DATASET_PATH] ../pretrain_checkpoint/mobilenetv2_ascend.ckpt backbone + ``` ### 微调训练结果 - 查看运行结果。 - - 运行Python文件时在交互式命令行中查看打印信息，`Linux`上运行Shell脚本运行后使用`cat ./train/rank0/log0.log`中查看打印信息，输出结果如下： + - 运行Python文件时在交互式命令行中查看打印信息，`Linux`上运行Shell脚本运行后使用`cat ./train/rank0/log0.log`中查看打印信息，输出结果如下： - ```bash - train args: Namespace(dataset_path='./dataset/train', platform='CPU', \ - pretrain_ckpt='./pretrain_checkpoint/mobilenetv2_cpu_gpu.ckpt', freeze_layer='backbone') - cfg: {'num_classes': 26, 'image_height': 224, 'image_width': 224, 'batch_size': 150, \ - 'epoch_size': 200, 'warmup_epochs': 0, 'lr_max': 0.03, 'lr_end': 0.03, 'momentum': 0.9, \ - 'weight_decay': 4e-05, 'label_smooth': 0.1, 'loss_scale': 1024, 'save_checkpoint': True, \ - 'save_checkpoint_epochs': 1, 'keep_checkpoint_max': 20, 'save_checkpoint_path': './', \ - 'platform': 'CPU'} - Processing batch: 16: 100%|███████████████████████████████████████████ █████████████████████| 16/16 [00:00 本例面向Ascend 910 AI处理器硬件平台，你可以在这里下载完整的样例代码： 下面对任务流程中各个环节及代码关键片段进行解释说明。 - ## 下载CIFAR-10数据集 + 先从[CIFAR-10数据集官网](https://www.cs.toronto.edu/~kriz/cifar.html)上下载CIFAR-10数据集。本例中采用binary格式的数据，Linux环境可以通过下面的命令下载： ```shell @@ -81,7 +80,6 @@ wget https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz tar -zvxf cifar-10-binary.tar.gz ``` - ## 数据预加载和预处理 1. 加载数据集 @@ -89,10 +87,8 @@ tar -zvxf cifar-10-binary.tar.gz 数据加载可以通过内置数据集格式`Cifar10Dataset`接口完成。 > `Cifar10Dataset`，读取类型为随机读取，内置CIFAR-10数据集，包含图像和标签，图像格式默认为uint8，标签数据格式默认为uint32。更多说明请查看API中`Cifar10Dataset`接口说明。 - 数据加载代码如下，其中`data_home`为数据存储位置： - ```python cifar_ds = ds.Cifar10Dataset(data_home) ``` @@ -141,7 +137,6 @@ tar -zvxf cifar-10-binary.tar.gz cifar_ds = cifar_ds.repeat(repeat_num) ``` - ## 定义卷积神经网络 卷积神经网络已经是图像分类任务的标准算法了。卷积神经网络采用分层的结构对图片进行特征提取，由一系列的网络层堆叠而成，比如卷积层、池化层、激活层等等。 @@ -156,7 +151,6 @@ network = resnet50(class_num=10) 更多ResNet的介绍请参考：[ResNet论文](https://arxiv.org/abs/1512.03385) - ## 定义损失函数和优化器 接下来需要定义损失函数（Loss）和优化器（Optimizer）。损失函数是深度学习的训练目标，也叫目标函数，可以理解为神经网络的输出（Logits）和标签(Labels)之间的距离，是一个标量数据。 @@ -175,7 +169,6 @@ ls = SoftmaxCrossEntropyWithLogits(sparse=True, reduction="mean") opt = Momentum(filter(lambda x: x.requires_grad, net.get_parameters()), 0.01, 0.9) ``` - ## 调用`Model`高阶API进行训练和保存模型文件 完成数据预处理、网络定义、损失函数和优化器定义之后，就可以进行模型训练了。模型训练包含两层迭代，数据集的多轮迭代（`epoch`）和一轮数据集内按分组（`batch`）大小进行的单步迭代。其中，单步迭代指的是按分组从数据集中抽取数据，输入到网络中计算得到损失函数，然后通过优化器计算和更新训练参数的梯度。 @@ -214,4 +207,4 @@ print("result: ", res) ## 参考文献 -[1] https://www.cs.toronto.edu/~kriz/cifar.html +[1] diff --git a/tutorials/training/source_zh_cn/advanced_use/cv_resnet50_second_order_optimizer.md b/tutorials/training/source_zh_cn/advanced_use/cv_resnet50_second_order_optimizer.md index 855cb086c5438b1700d3d4ac40d917310bbe8ebc..3c0c523802b016c1035d638c1b06cfdf1b441b4f 100644 --- a/tutorials/training/source_zh_cn/advanced_use/cv_resnet50_second_order_optimizer.md +++ b/tutorials/training/source_zh_cn/advanced_use/cv_resnet50_second_order_optimizer.md @@ -37,7 +37,6 @@ MindSpore开发团队在现有的自然梯度算法的基础上，对FIM矩阵采用近似、切分等优化加速手段，极大的降低了逆矩阵的计算复杂度，开发出了可用的二阶优化器THOR。使用8块Ascend 910 AI处理器，THOR可以在72min内完成ResNet50-v1.5网络和ImageNet数据集的训练，相比于SGD+Momentum速度提升了近一倍。 - 本篇教程将主要介绍如何在Ascend 910 以及GPU上，使用MindSpore提供的二阶优化器THOR训练ResNet50-v1.5网络和ImageNet数据集。 > 你可以在这里下载完整的示例代码： 。 @@ -47,12 +46,12 @@ MindSpore开发团队在现有的自然梯度算法的基础上，对FIM矩阵 ```shell ├── resnet_thor ├── README.md - ├── scripts + ├── scripts ├── run_distribute_train.sh # launch distributed training for Ascend 910 └── run_eval.sh # launch inference for Ascend 910 ├── run_distribute_train_gpu.sh # launch distributed training for GPU └── run_eval_gpu.sh # launch inference for GPU - ├── src + ├── src ├── crossentropy.py # CrossEntropy loss function ├── config.py # parameter configuration ├── dataset_helper.py # dataset helper for minddata dataset @@ -61,20 +60,20 @@ MindSpore开发团队在现有的自然梯度算法的基础上，对FIM矩阵 ├── resnet_thor.py # resnet50_thor backone ├── thor.py # thor optimizer ├── thor_layer.py # thor layer - └── dataset.py # data preprocessing + └── dataset.py # data preprocessing ├── eval.py # infer script └── train.py # train script - + ``` 整体执行流程如下： + 1. 准备ImageNet数据集，处理需要的数据集； 2. 定义ResNet50网络； 3. 定义损失函数和THOR优化器； 4. 加载数据集并进行训练，训练完成后，查看结果及保存模型文件； 5. 加载保存的模型，进行推理。 - ## 准备环节 实践前，确保已经正确安装MindSpore。如果没有，可以通过[MindSpore安装页面](https://www.mindspore.cn/install)安装MindSpore。 @@ -85,7 +84,7 @@ MindSpore开发团队在现有的自然梯度算法的基础上，对FIM矩阵 目录结构如下： -``` +```text └─ImageNet2012 ├─ilsvrc │ n03676483 @@ -99,17 +98,21 @@ MindSpore开发团队在现有的自然梯度算法的基础上，对FIM矩阵 │ ...... ``` + ### 配置分布式环境变量 + #### Ascend 910 + Ascend 910 AI处理器的分布式环境变量配置参考[分布式并行训练 (Ascend)](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_ascend.html#id4)。 #### GPU -GPU的分布式环境配置参考[分布式并行训练 (GPU)](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_gpu.html#id4)。 +GPU的分布式环境配置参考[分布式并行训练 (GPU)](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_gpu.html#id4)。 ## 加载处理数据集 分布式训练时，通过并行的方式加载数据集，同时通过MindSpore提供的数据增强接口对数据集进行处理。加载处理数据集的脚本在源码的`src/dataset.py`脚本中。 + ```python import os import mindspore.common.dtype as mstype @@ -165,17 +168,18 @@ def create_dataset(dataset_path, do_train, repeat_num=1, batch_size=32, target=" > MindSpore支持进行多种数据处理和增强的操作，各种操作往往组合使用，具体可以参考[数据处理](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/pipeline.html)和[数据增强](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/augmentation.html)章节。 - ## 定义网络 + 本示例中使用的网络模型为ResNet50-v1.5，先定义[ResNet50网络](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/resnet/src/resnet.py)，然后使用二阶优化器自定义的算子替换`Conv2d`和 和`Dense`算子。定义好的网络模型在在源码`src/resnet_thor.py`脚本中，自定义的算子`Conv2d_thor`和`Dense_thor`在`src/thor_layer.py`脚本中。 -- 使用`Conv2d_thor`替换原网络模型中的`Conv2d` -- 使用`Dense_thor`替换原网络模型中的`Dense` +- 使用`Conv2d_thor`替换原网络模型中的`Conv2d` +- 使用`Dense_thor`替换原网络模型中的`Dense` > 使用THOR自定义的算子`Conv2d_thor`和`Dense_thor`是为了保存模型训练中的二阶矩阵信息，新定义的网络与原网络模型的backbone一致。 网络构建完成以后，在`__main__`函数中调用定义好的ResNet50： + ```python ... from src.resnet_thor import resnet50 @@ -188,15 +192,14 @@ if __name__ == "__main__": ... ``` - ## 定义损失函数及THOR优化器 - ### 定义损失函数 MindSpore支持的损失函数有`SoftmaxCrossEntropyWithLogits`、`L1Loss`、`MSELoss`等。THOR优化器需要使用`SoftmaxCrossEntropyWithLogits`损失函数。 损失函数的实现步骤在`src/crossentropy.py`脚本中。这里使用了深度网络模型训练中的一个常用trick：label smoothing，通过对真实标签做平滑处理，提高模型对分类错误标签的容忍度，从而可以增加模型的泛化能力。 + ```python class CrossEntropy(_Loss): """CrossEntropy""" @@ -214,6 +217,7 @@ class CrossEntropy(_Loss): loss = self.mean(loss, 0) return loss ``` + 在`__main__`函数中调用定义好的损失函数： ```python @@ -236,6 +240,7 @@ THOR优化器的参数更新公式如下： $$ \theta^{t+1} = \theta^t + \alpha F^{-1}\nabla E$$ 参数更新公式中各参数的含义如下： + - $\theta$：网络中的可训参数； - $t$：迭代次数； - $\alpha$：学习率值，参数的更新步长； @@ -296,7 +301,6 @@ if __name__ == "__main__": 通过MindSpore提供的`model.train`接口可以方便地进行网络的训练。THOR优化器通过降低二阶矩阵更新频率，来减少计算量，提升计算速度，故重新定义一个Model_Thor类，继承MindSpore提供的Model类。在Model_Thor类中增加二阶矩阵更新频率控制参数，用户可以通过调整该参数，优化整体的性能。 - ```python ... from mindspore.train.loss_scale_manager import FixedLossScaleManager @@ -316,15 +320,21 @@ if __name__ == "__main__": ``` ### 运行脚本 + 训练脚本定义完成之后，调`scripts`目录下的shell脚本，启动分布式训练进程。 + #### Ascend 910 + 目前MindSpore分布式在Ascend上执行采用单卡单进程运行方式，即每张卡上运行1个进程，进程数量与使用的卡的数量一致。其中，0卡在前台执行，其他卡放在后台执行。每个进程创建1个目录，目录名称为`train_parallel`+ `device_id`，用来保存日志信息，算子编译信息以及训练的checkpoint文件。下面以使用8张卡的分布式训练脚本为例，演示如何运行脚本： 使用以下命令运行脚本： -``` + +```bash sh run_distribute_train.sh [RANK_TABLE_FILE] [DATASET_PATH] [DEVICE_NUM] ``` + 脚本需要传入变量`RANK_TABLE_FILE`、`DATASET_PATH`和`DEVICE_NUM`，其中： + - `RANK_TABLE_FILE`：组网信息文件的路径。 - `DATASET_PATH`：训练数据集路径。 - `DEVICE_NUM`：实际的运行卡数。 @@ -361,17 +371,22 @@ epoch: 42 step: 5004, loss is 1.6453942 `*.ckpt`：指保存的模型参数文件。checkpoint文件名称具体含义：*网络名称*-*epoch数*_*step数*.ckpt。 #### GPU + 在GPU硬件平台上，MindSpore采用OpenMPI的`mpirun`进行分布式训练，进程创建1个目录，目录名称为`train_parallel`，用来保存日志信息和训练的checkpoint文件。下面以使用8张卡的分布式训练脚本为例，演示如何运行脚本： -``` + +```bash sh run_distribute_train_gpu.sh [DATASET_PATH] [DEVICE_NUM] ``` + 脚本需要传入变量`DATASET_PATH`和`DEVICE_NUM`，其中： + - `DATASET_PATH`：训练数据集路径。 - `DEVICE_NUM`：实际的运行卡数。 在GPU训练时，无需设置`DEVICE_ID`环境变量，因此在主训练脚本中不需要调用`int(os.getenv('DEVICE_ID'))`来获取卡的物理序号，同时`context`中也无需传入`device_id`。我们需要将device_target设置为GPU，并需要调用`init()`来使能NCCL。 训练过程中loss打印示例如下： + ```bash ... epoch: 1 step: 5004, loss is 4.2546034 @@ -391,7 +406,7 @@ epoch: 36 step: 5004, loss is 1.645802 ├─ckpt_0 ├─resnet-1_5004.ckpt ├─resnet-2_5004.ckpt - │ ...... + │ ...... ├─resnet-36_5004.ckpt │ ...... ...... @@ -436,40 +451,53 @@ if __name__ == "__main__": # define model model = Model(net, loss_fn=loss, metrics={'top_1_accuracy', 'top_5_accuracy'}) - + # eval model res = model.eval(dataset) print("result:", res, "ckpt=", args_opt.checkpoint_path) ``` ### 执行推理 + 推理网络定义完成之后，调用`scripts`目录下的shell脚本，进行推理。 + #### Ascend 910 + 在Ascend 910硬件平台上，推理的执行命令如下： -``` + +```bash sh run_eval.sh [DATASET_PATH] [CHECKPOINT_PATH] ``` + 脚本需要传入变量`DATASET_PATH`和`CHECKPOINT_PATH`，其中： + - `DATASET_PATH`：推理数据集路径。 - `CHECKPOINT_PATH`：保存的checkpoint路径。 目前推理使用的是单卡（默认device 0）进行推理，推理的结果如下： -``` + +```text result: {'top_5_accuracy': 0.9295574583866837, 'top_1_accuracy': 0.761443661971831} ckpt=train_parallel0/resnet-42_5004.ckpt ``` + - `top_5_accuracy`：对于一个输入图片，如果预测概率排名前五的标签中包含真实标签，即认为分类正确； - `top_1_accuracy`：对于一个输入图片，如果预测概率最大的标签与真实标签相同，即认为分类正确。 + #### GPU 在GPU硬件平台上，推理的执行命令如下： -``` + +```bash sh run_eval_gpu.sh [DATASET_PATH] [CHECKPOINT_PATH] ``` + 脚本需要传入变量`DATASET_PATH`和`CHECKPOINT_PATH`，其中： + - `DATASET_PATH`：推理数据集路径。 - `CHECKPOINT_PATH`：保存的checkpoint路径。 推理的结果如下： -``` + +```text result: {'top_5_accuracy': 0.9287972151088348, 'top_1_accuracy': 0.7597031049935979} ckpt=train_parallel/resnet-36_5004.ckpt ``` diff --git a/tutorials/training/source_zh_cn/advanced_use/dashboard.md b/tutorials/training/source_zh_cn/advanced_use/dashboard.md index bd5f41918a33ddf1c34d048b91e949d428cbc2d0..1779b5a0161fd0cdfa9de16155acff8d83357587 100644 --- a/tutorials/training/source_zh_cn/advanced_use/dashboard.md +++ b/tutorials/training/source_zh_cn/advanced_use/dashboard.md @@ -17,7 +17,7 @@    - + ## 概述 @@ -196,4 +196,4 @@ 备注：估算`TensorSummary`空间使用量的方法如下： 一个`TensorSummary数据的大小 ＝ Tensor中的数值个数 * 4 bytes`。假设使用`TensorSummary`记录的Tensor大小为`32 * 1 * 256 * 256`，则一个`TensorSummary`数据大约需要`32 * 1 * 256 * 256 * 4 bytes = 8,388,608 bytes = 8MiB`。`TensorSummary`默认会记录20个步骤的数据，则记录这20组数据需要的空间约为`20 * 8 MiB ＝ 160MiB`。需要注意的是，由于数据结构等因素的开销，实际使用的存储空间会略大于160MiB。 -6. 当使用`TensorSummary`时，由于记录完整Tensor数据，训练日志文件较大，MindInsight需要更多时间解析训练日志文件，请耐心等待。 \ No newline at end of file +6. 当使用`TensorSummary`时，由于记录完整Tensor数据，训练日志文件较大，MindInsight需要更多时间解析训练日志文件，请耐心等待。 diff --git a/tutorials/training/source_zh_cn/advanced_use/debug_in_pynative_mode.md b/tutorials/training/source_zh_cn/advanced_use/debug_in_pynative_mode.md index 3879fbfa69d4265dcd5551119b61b78a0b0dc6f6..80e4b8a6e815fd0fc344fd580fc8bbab76525570 100644 --- a/tutorials/training/source_zh_cn/advanced_use/debug_in_pynative_mode.md +++ b/tutorials/training/source_zh_cn/advanced_use/debug_in_pynative_mode.md @@ -75,12 +75,12 @@ print(output.asnumpy()) [ 0.05016355 0.03958241 0.03958241 0.03958241 0.03443141]]]] ``` - ## 执行普通函数 将若干算子组合成一个函数，然后直接通过函数调用的方式执行这些算子，并打印相关结果，如下例所示。 -**示例代码** +**示例代码：** + ```python import numpy as np from mindspore import context, Tensor @@ -99,7 +99,7 @@ output = tensor_add_func(x, y) print(output.asnumpy()) ``` -**输出** +**输出：** ```python [[3. 3. 3.] @@ -109,7 +109,6 @@ print(output.asnumpy()) > PyNative不支持并行执行和summary功能，图模式的并行和summary相关算子不能使用。 - ### 提升PyNative性能 为了提高PyNative模式下的前向计算任务执行速度，MindSpore提供了Staging功能，该功能可以在PyNative模式下将Python函数或者Python类的方法编译成计算图，通过图优化等技术提高运行速度，如下例所示。 @@ -142,7 +141,8 @@ tensor_add = P.TensorAdd() res = tensor_add(x, z) # PyNative mode print(res.asnumpy()) ``` -**输出** + +**输出：** ```python [[3. 3. 3. 3.] @@ -155,7 +155,7 @@ print(res.asnumpy()) 需要说明的是，加装了`ms_function`装饰器的函数中，如果包含不需要进行参数训练的算子（如`pooling`、`tensor_add`等算子），则这些算子可以在被装饰的函数中直接调用，如下例所示。 -**示例代码** +**示例代码：** ```python import numpy as np @@ -178,7 +178,8 @@ y = Tensor(np.ones([4, 4]).astype(np.float32)) z = tensor_add_fn(x, y) print(z.asnumpy()) ``` -**输出** + +**输出：** ```shell [[2. 2. 2. 2.] @@ -189,7 +190,7 @@ print(z.asnumpy()) 如果被装饰的函数中包含了需要进行参数训练的算子（如`Convolution`、`BatchNorm`等算子），则这些算子必须在被装饰等函数之外完成实例化操作，如下例所示。 -**示例代码** +**示例代码：** ```python import numpy as np @@ -211,7 +212,7 @@ z = conv_fn(Tensor(input_data)) print(z.asnumpy()) ``` -**输出** +**输出：** ```shell [[[[ 0.10377571 -0.0182163 -0.05221086] @@ -247,12 +248,11 @@ print(z.asnumpy()) [ 0.0377498 -0.06117418 0.00546303]]]] ``` - ## 调试网络训练模型 PyNative模式下，还可以支持单独求梯度的操作。如下例所示，可通过`GradOperation`求该函数或者网络所有的输入梯度。需要注意，输入类型仅支持Tensor。 -**示例代码** +**示例代码：** ```python from mindspore.ops import composite as C @@ -269,7 +269,7 @@ def mainf(x, y): print(mainf(Tensor(1, mstype.int32), Tensor(2, mstype.int32))) ``` -**输出** +**输出：** ```python (2, 1) @@ -277,7 +277,7 @@ print(mainf(Tensor(1, mstype.int32), Tensor(2, mstype.int32))) 在进行网络训练时，求得梯度然后调用优化器对参数进行优化（暂不支持在反向计算梯度的过程中设置断点），然后再利用前向计算loss，从而实现在PyNative模式下进行网络训练。 -**完整LeNet示例代码** +**完整LeNet示例代码：** ```python import numpy as np @@ -314,7 +314,7 @@ class LeNet5(nn.Cell): Lenet network Args: num_class (int): Num classes. Default: 10. - + Returns: Tensor, output tensor @@ -348,8 +348,8 @@ class LeNet5(nn.Cell): x = self.relu(x) x = self.fc3(x) return x - - + + class GradWrap(nn.Cell): """ GradWrap definition """ def __init__(self, network): @@ -378,7 +378,7 @@ loss = loss_output.asnumpy() print(loss) ``` -**输出** +**输出：** ```python 2.3050091 diff --git a/tutorials/training/source_zh_cn/advanced_use/debugger.md b/tutorials/training/source_zh_cn/advanced_use/debugger.md index 35214c624c4f8598917cce8bbfdb274d52f5cdcd..6652fe3ca6af1a5033fa28962f9e301a7c85725a 100644 --- a/tutorials/training/source_zh_cn/advanced_use/debugger.md +++ b/tutorials/training/source_zh_cn/advanced_use/debugger.md @@ -22,6 +22,7 @@ ## 概述 + MindSpore调试器是为图模式训练提供的调试工具，可以用来查看并分析计算图节点的中间结果。 在MindSpore图模式的训练过程中，用户无法从Python层获取到计算图中间节点的结果，使得训练调试变得很困难。使用MindSpore调试器，用户可以： @@ -37,6 +38,7 @@ MindSpore调试器是为图模式训练提供的调试工具，可以用来查 - 在MindInsight调试器界面分析训练执行情况。 ## 调试器环境准备 + 开始训练前，请先安装MindInsight，并以调试模式启动。调试模式下，MindSpore会将训练信息发送给MindInsight调试服务，用户可在MindInsight调试器界面进行查看和分析。 MindInsight调试服务启动命令： @@ -72,6 +74,7 @@ mindinsight start --port {PORT} --enable-debugger True --debugger-port {DEBUGGER 图1： 调试器初始页面 ### 计算图 + 调试器将优化后的最终执行图展示在UI的中上位置，用户可以双击打开图上的方框 (代表一个`scope`) 将计算图进一步展开，查看`scope`中的节点信息。 面板的最上方展示了`训练端地址`（训练脚本所在进程的地址和端口），训练使用的`卡号`, 训练的`当前轮次`等元信息。 @@ -119,11 +122,12 @@ mindinsight start --port {PORT} --enable-debugger True --debugger-port {DEBUGGER 图6： 查看触发的条件断点 -图6展示了条件断点触发后的展示页面，该页面和`节点列表`所在位置相同。触发的节点以及监控条件会按照节点的执行序排列，用户点击某一行，会在计算图中跳转到对应节点，可以进一步查看节点信息分析INF等异常结果出现的原因。 +图6展示了条件断点触发后的展示页面，该页面和`节点列表`所在位置相同。触发的节点以及监控条件会按照节点的执行序排列，用户点击某一行，会在计算图中跳转到对应节点，可以进一步查看节点信息分析INF等异常结果出现的原因。 ### 训练控制 监测点设置面板的下方是训练控制面板，该面板展示了调试器的训练控制功能，有`继续`、`暂停`、`结束`、`确定`四个按钮。 + - `确定`代表训练向前执行若干个`轮次`，需要用户在上方的输入框内指定执行的`轮次`数目，直到条件断点触发、或`轮次`执行完毕后暂停； - `继续`代表训练一直执行，直到条件断点触发后暂停、或运行至训练结束； - `暂停`代表训练暂停； @@ -134,20 +138,20 @@ mindinsight start --port {PORT} --enable-debugger True --debugger-port {DEBUGGER 1. 在调试器环境准备完成后，打开调试器界面，如下图所示： ![debugger_waiting](./images/debugger_waiting.png) - + 图7： 调试器等待训练连接 - + 此时，调试器处于等待训练启动和连接的状态。 2. 运行训练脚本，稍后可以看到计算图显示在调试器界面，见图1。 3. 设置条件断点，见图5。 - + 图5中，选中检测条件，并勾选了部分节点，代表监控这些节点在计算过程是否存在满足监控条件的输出。 设置完条件断点后，可以在控制面板选择设置轮次点击`确定`或者`继续`继续训练。 4. 条件断点触发，见图6。 - + 条件断点触发后，用户查看对应的节点信息，找出异常原因后修改脚本，解掉bug。 ## 注意事项 diff --git a/tutorials/training/source_zh_cn/advanced_use/distributed_training_ascend.md b/tutorials/training/source_zh_cn/advanced_use/distributed_training_ascend.md index f0752c2b060cd84a1ab23a0f7200ffe672686f1e..d9368cc9c7f7bd9e832d7dd46549e173bfde0e85 100644 --- a/tutorials/training/source_zh_cn/advanced_use/distributed_training_ascend.md +++ b/tutorials/training/source_zh_cn/advanced_use/distributed_training_ascend.md @@ -12,6 +12,8 @@ - [调用集合通信库](#调用集合通信库) - [数据并行模式加载数据集](#数据并行模式加载数据集) - [定义网络](#定义网络) + - [手动混合并行模式](#手动混合并行模式) + - [半自动并行模式](#半自动并行模式) - [定义损失函数及优化器](#定义损失函数及优化器) - [定义损失函数](#定义损失函数) - [定义优化器](#定义优化器) @@ -34,6 +36,8 @@ > > +此外在[定义网络](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_ascend.html#id7)和[分布式训练模型参数保存和加载](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_ascend.html#id13)小节中我们针对手动混合并行模式和半自动并行模式的使用做了特殊说明。 + ## 准备环节 ### 下载数据集 @@ -81,13 +85,14 @@ - `device_ip`表示集成网卡的IP地址，可以在当前机器执行指令`cat /etc/hccn.conf`，`address_x`的键值就是网卡IP地址。 - `rank_id`表示卡逻辑序号，固定从0开始编号。 - ### 调用集合通信库 MindSpore分布式并行训练的通信使用了华为集合通信库`Huawei Collective Communication Library`（以下简称HCCL），可以在Ascend AI处理器配套的软件包中找到。同时`mindspore.communication.management`中封装了HCCL提供的集合通信接口，方便用户配置分布式信息。 > HCCL实现了基于Ascend AI处理器的多机多卡通信，有一些使用限制，我们列出使用分布式服务常见的，详细的可以查看HCCL对应的使用文档。 +> > - 单机场景下支持1、2、4、8卡设备集群，多机场景下支持8*n卡设备集群。 > - 每台机器的0-3卡和4-7卡各为1个组网，2卡和4卡训练时卡必须相连且不支持跨组网创建集群。 +> - 组建多机集群时需要保证各台机器使用同一交换机。 > - 服务器硬件架构及操作系统需要是SMP（Symmetrical Multi-Processing，对称多处理器）处理模式。 下面是调用集合通信库样例代码： @@ -100,10 +105,11 @@ from mindspore.communication.management import init if __name__ == "__main__": context.set_context(mode=context.GRAPH_MODE, device_target="Ascend", device_id=int(os.environ["DEVICE_ID"])) init() - ... + ... ``` 其中， + - `mode=context.GRAPH_MODE`：使用分布式训练需要指定运行模式为图模式（PyNative模式不支持并行）。 - `device_id`：卡的物理序号，即卡所在机器中的实际序号。 - `init`：使能HCCL通信，并完成分布式训练初始化操作。 @@ -112,7 +118,6 @@ if __name__ == "__main__": 分布式训练时，数据是以数据并行的方式导入的。下面我们以CIFAR-10数据集为例，介绍以数据并行方式导入CIFAR-10数据集的方法，`data_path`是指数据集的路径，即`cifar-10-batches-bin`文件夹的路径。 - ```python import mindspore.common.dtype as mstype import mindspore.dataset as ds @@ -158,13 +163,76 @@ def create_dataset(data_path, repeat_num=1, batch_size=32, rank_id=0, rank_size= return data_set ``` + 其中，与单机不同的是，在数据集接口需要传入`num_shards`和`shard_id`参数，分别对应卡的数量和逻辑序号，建议通过HCCL接口获取： + - `get_rank`：获取当前设备在集群中的ID。 - `get_group_size`：获取集群数量。 ## 定义网络 -数据并行及自动并行模式下，网络定义方式与单机一致。代码请参考： +数据并行及自动并行模式下，网络定义方式与单机写法一致，可以参考[ResNet网络样例脚本](https://gitee.com/mindspore/docs/blob/master/tutorials/tutorial_code/resnet/resnet.py)。 + +本章节重点介绍手动混合并行和半自动并行模式的网络定义方法。 + +### 手动混合并行模式 + +手动混合并行模式在数据并行模式的基础上，对`parameter`增加了模型并行`layerwise_parallel`配置，包含此配置的`parameter`将以切片的形式保存并参与计算，在优化器计算时不会进行梯度累加。在该模式下，框架不会自动插入并行算子前后需要的计算和通信操作，为了保证计算逻辑的正确性，用户需要手动推导并写在网络结构中，适合对并行原理深入了解的用户使用。 + +以下面的代码为例，将`self.weight`指定为模型并行配置，即`self.weight`和`MatMul`的输出在第二维`channel`上存在切分。这时再在第二维上进行`ReduceSum`得到的仅是单卡累加结果，还需要引入`AllReduce.Sum`通信操作对每卡的结果做加和。关于并行算子的推导原理可以参考这篇[设计文档](https://www.mindspore.cn/doc/note/zh-CN/master/design/mindspore/distributed_training_design.html#id10)。 + +```python +from mindspore import Tensor +import mindspore.ops as ops +import mindspore.common.dtype as mstype +import mindspore.nn as nn + +class HybridParallelNet(nn.Cell): + def __init__(self): + super(HybridParallelNet, self).__init__() + # initialize the weight which is sliced at the second dimension + weight_init = np.random.rand(512, 128/2).astype(np.float32) + self.weight = Parameter(Tensor(weight_init), name="weight", layerwise_parallel=True) + self.fc = ops.MatMul() + self.reduce = ops.ReduceSum() + self.allreduce = ops.AllReduce(op='sum') + + def construct(self, x): + x = self.fc(x, self.weight) + x = self.reduce(x, -1) + x = self.allreduce(x) + return x +``` + +### 半自动并行模式 + +半自动并行模式相较于自动并行模式支持用户手动配置并行策略进行调优。关于算子并行策略的定义可以参考这篇[设计文档](https://www.mindspore.cn/doc/note/zh-CN/master/design/mindspore/distributed_training_design.html#id10)。 + +用户在使用半自动并行模式时，需要注意，未配置策略的算子默认以数据并行方式执行，如果某个`parameter`被多个算子使用，则每个算子对这个`parameter`的切分策略需要保持一致，否则将报错。 + +以前述的`HybridParallelNet`为例，在半自动并行模式下的脚本代码如下，`MatMul`的切分策略为`{(1, 1),(1, 2)}`，指定`self.weight`在第二维度上被切分两份。 + +```python +from mindspore import Tensor +import mindspore.ops as ops +import mindspore.common.dtype as mstype +import mindspore.nn as nn + +class SemiAutoParallelNet(nn.Cell): + def __init__(self): + super(SemiAutoParallelNet, self).__init__() + # initialize full tensor weight + weight_init = np.random.rand(512, 128).astype(np.float32) + self.weight = Parameter(Tensor(weight_init), name="weight") + # set shard strategy + self.fc = ops.MatMul().shard({(1, 1),(1, 2)}) + self.reduce = ops.ReduceSum() + + def construct(self, x): + x = self.fc(x, self.weight) + x = self.reduce(x, -1) + return x +``` ## 定义损失函数及优化器 @@ -255,7 +323,9 @@ def test_train_cifar(epoch_size=10): model = Model(net, loss_fn=loss, optimizer=opt) model.train(epoch_size, dataset, callbacks=[loss_cb], dataset_sink_mode=True) ``` + 其中， + - `dataset_sink_mode=True`：表示采用数据集的下沉模式，即训练的计算下沉到硬件平台中执行。 - `LossMonitor`：能够通过回调函数返回Loss值，用于监控损失函数。 @@ -322,6 +392,7 @@ cd ../ 脚本需要传入变量`DATA_PATH`和`RANK_SIZE`，分别表示数据集的路径和卡的数量。 其中必要的环境变量有， + - `RANK_TABLE_FILE`：组网信息文件的路径。 - `DEVICE_ID`：当前卡在机器上的实际序号。 - `RANK_ID`：当前卡的逻辑序号。 @@ -331,7 +402,7 @@ cd ../ 日志文件保存`device`目录下，`env.log`中记录了环境变量的相关信息，关于Loss部分结果保存在`train.log`中，示例如下： -``` +```text epoch: 1 step: 156, loss is 2.0084016 epoch: 2 step: 156, loss is 1.6407638 epoch: 3 step: 156, loss is 1.6164391 @@ -485,7 +556,7 @@ context.reset_auto_parallel_context() # set parallel mode, data parallel mode is selected for training and model saving. If you want to choose auto parallel # mode, you can simply change the value of parallel_mode parameter to ParallelMode.AUTO_PARALLEL. context.set_auto_parallel_context(parallel_mode=ParallelMode.SEMI_AUTO_PARALLEL, - strategy_ckpt_save_file='./rank_{}_ckpt/strategy.txt'.format(get_rank)) + strategy_ckpt_save_file='./rank_{}_ckpt/strategy.txt'.format(get_rank)) ``` 然后根据需要设置checkpoint保存策略，以及设置优化器和损失函数等，代码如下： @@ -514,12 +585,14 @@ context.reset_auto_parallel_context() 只需要改动设置checkpoint保存策略的代码，将`CheckpointConfig`中的`integrated_save`参数设置为Fasle，便可实现每张卡上只保存本卡的checkpoint文件，具体改动如下： 将checkpoint配置策略由 + ```python # config checkpoint ckpt_config = CheckpointConfig(keep_checkpoint_max=1) ``` 改为 + ```python # config checkpoint ckpt_config = CheckpointConfig(keep_checkpoint_max=1, integrated_save=False) diff --git a/tutorials/training/source_zh_cn/advanced_use/distributed_training_gpu.md b/tutorials/training/source_zh_cn/advanced_use/distributed_training_gpu.md index c1d5cb03a3080e84b9e9ed82142831349a38f266..0b2f4a1eeb38e37f31340aa0239c343f2691a878 100644 --- a/tutorials/training/source_zh_cn/advanced_use/distributed_training_gpu.md +++ b/tutorials/training/source_zh_cn/advanced_use/distributed_training_gpu.md @@ -70,7 +70,7 @@ from mindspore.communication.management import init if __name__ == "__main__": context.set_context(mode=context.GRAPH_MODE, device_target="GPU") init("nccl") - ... + ... ``` 其中， @@ -110,7 +110,7 @@ mpirun -n 8 pytest -s -v ./resnet50_distributed_training.py > train.log 2>&1 & 脚本需要传入变量`DATA_PATH`，表示数据集的路径。此外，我们需要修改下`resnet50_distributed_training.py`文件，由于在GPU上，我们无需设置`DEVICE_ID`环境变量，因此，在脚本中不需要调用`int(os.getenv('DEVICE_ID'))`来获取卡的物理序号，同时`context`中也无需传入`device_id`。我们需要将`device_target`设置为`GPU`，并调用`init("nccl")`来使能NCCL。日志文件保存到device目录下，关于Loss部分结果保存在train.log中。将loss值grep出来后，示例如下： -``` +```text epoch: 1 step: 1, loss is 2.3025854 epoch: 1 step: 1, loss is 2.3025854 epoch: 1 step: 1, loss is 2.3025854 @@ -124,6 +124,7 @@ epoch: 1 step: 1, loss is 2.3025854 ## 运行多机脚本 若训练涉及多机，则需要额外在`mpirun`命令中设置多机配置。你可以直接在`mpirun`命令中用`-H`选项进行设置，比如`mpirun -n 16 -H DEVICE1_IP:8,DEVICE2_IP:8 python hello.py`，表示在ip为DEVICE1_IP和DEVICE2_IP的机器上分别起8个进程运行程序；或者也可以构造一个如下这样的hostfile文件，并将其路径传给`mpirun`的`--hostfile`的选项。hostfile文件每一行格式为`[hostname] slots=[slotnum]`，hostname可以是ip或者主机名。 + ```bash DEVICE1 slots=8 DEVICE2 slots=8 @@ -145,4 +146,4 @@ echo "start training" mpirun -n 16 --hostfile $HOSTFILE -x DATA_PATH=$DATA_PATH -x PATH -mca pml ob1 pytest -s -v ./resnet50_distributed_training.py > train.log 2>&1 & ``` -在GPU上进行分布式训练时，模型参数的保存和加载可参考[分布式训练模型参数保存和加载](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_ascend.html#id12) \ No newline at end of file +在GPU上进行分布式训练时，模型参数的保存和加载可参考[分布式训练模型参数保存和加载](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/distributed_training_ascend.html#id12) diff --git a/tutorials/training/source_zh_cn/advanced_use/enable_mixed_precision.md b/tutorials/training/source_zh_cn/advanced_use/enable_mixed_precision.md index 7181afe3cca19a2a293bd0a09e3db02bd5f8b3bb..9f52c1ac9681f82f7e9f641a7ed955f8d9941bd9 100644 --- a/tutorials/training/source_zh_cn/advanced_use/enable_mixed_precision.md +++ b/tutorials/training/source_zh_cn/advanced_use/enable_mixed_precision.md @@ -42,6 +42,7 @@ MindSpore混合精度典型的计算流程如下图所示： 使用自动混合精度，需要调用相应的接口，将待训练网络和优化器作为输入传进去；该接口会将整张网络的算子转换成FP16算子(除`BatchNorm`算子和Loss涉及到的算子外)。可以使用`amp`接口和`Model`接口两种方式实现混合精度。 使用`amp`接口具体的实现步骤为： + 1. 引入MindSpore的混合精度的接口`amp`； 2. 定义网络：该步骤和普通的网络定义没有区别(无需手动配置某个算子的精度)； @@ -93,6 +94,7 @@ output = train_network(predict, label) ``` 使用`Model`接口具体的实现步骤为： + 1. 引入MindSpore的模型训练接口`Model`； 2. 定义网络：该步骤和普通的网络定义没有区别(无需手动配置某个算子的精度)； @@ -169,6 +171,7 @@ model.train(epoch=10, train_dataset=ds_train) MindSpore还支持手动混合精度。假定在网络中只有一个Dense Layer要用FP32计算，其他Layer都用FP16计算。混合精度配置以Cell为粒度，Cell默认是FP32类型。 以下是一个手动混合精度的实现步骤： + 1. 定义网络：该步骤与自动混合精度中的步骤2类似； 2. 配置混合精度：通过`net.to_float(mstype.float16)`，把该Cell及其子Cell中所有的算子都配置成FP16；然后，将模型中的dense算子手动配置成FP32； @@ -220,4 +223,4 @@ train_network.set_train() # Run training output = train_network(predict, label) -``` \ No newline at end of file +``` diff --git a/tutorials/training/source_zh_cn/advanced_use/evaluate_the_model_during_training.md b/tutorials/training/source_zh_cn/advanced_use/evaluate_the_model_during_training.md index ce8172c6236837449633b315a8a125d63bd06c4f..ca8e2c515002d1092f59bd004215b516406a5193 100644 --- a/tutorials/training/source_zh_cn/advanced_use/evaluate_the_model_during_training.md +++ b/tutorials/training/source_zh_cn/advanced_use/evaluate_the_model_during_training.md @@ -22,6 +22,7 @@ 在面对复杂网络时，往往需要进行几十甚至几百次的epoch训练。在训练之前，很难掌握在训练到第几个epoch时，模型的精度能达到满足要求的程度，所以经常会采用一边训练的同时，在相隔固定epoch的位置对模型进行精度验证，并保存相应的模型，等训练完毕后，通过查看对应模型精度的变化就能迅速地挑选出相对最优的模型，本文将采用这种方法，以LeNet网络为样本，进行示例。 流程如下： + 1. 定义回调函数EvalCallBack，实现同步进行训练和验证。 2. 定义训练网络并执行。 3. 将不同epoch下的模型精度绘制出折线图并挑选最优模型。 @@ -54,7 +55,7 @@ class EvalCallBack(Callback): self.eval_dataset = eval_dataset self.eval_per_epoch = eval_per_epoch self.epoch_per_eval = epoch_per_eval - + def epoch_end(self, run_context): cb_param = run_context.original_args() cur_epoch = cb_param.cur_epoch_num @@ -92,21 +93,21 @@ if __name__ == "__main__": eval_per_epoch = 2 ... ... - + # need to calculate how many steps are in each epoch，in this example, 1875 steps per epoch config_ck = CheckpointConfig(save_checkpoint_steps=eval_per_epoch*1875, keep_checkpoint_max=15) ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet",directory=ckpt_save_dir, config=config_ck) model = Model(network, net_loss, net_opt, metrics={"Accuracy": Accuracy()}) - + epoch_per_eval = {"epoch": [], "acc": []} eval_cb = EvalCallBack(model, eval_data, eval_per_epoch, epoch_per_eval) - + model.train(epoch_size, train_data, callbacks=[ckpoint_cb, LossMonitor(375), eval_cb], dataset_sink_mode=True) ``` 输出结果： - + ```text epoch: 1 step: 375, loss is 2.298612 epoch: 1 step: 750, loss is 2.075152 epoch: 1 step: 1125, loss is 0.39205977 @@ -118,9 +119,7 @@ if __name__ == "__main__": epoch: 2 step: 1500, loss is 0.067035824 epoch: 2 step: 1875, loss is 0.0050643035 {'Accuracy': 0.9763621794871795} - ... ... - epoch: 9 step: 375, loss is 0.021227183 epoch: 9 step: 750, loss is 0.005586236 epoch: 9 step: 1125, loss is 0.029125651 @@ -133,10 +132,9 @@ if __name__ == "__main__": epoch: 10 step: 1875, loss is 0.10563098 {'Accuracy': 0.979667467948718} - 在同一目录找到`lenet_ckpt`文件夹，文件夹中保存了5个模型，和一个计算图相关数据，其结构如下： -``` +```text lenet_ckpt ├── checkpoint_lenet-10_1875.ckpt ├── checkpoint_lenet-2_1875.ckpt @@ -150,7 +148,6 @@ lenet_ckpt 定义绘图函数`eval_show`，将`epoch_per_eval`载入到`eval_show`中，绘制出不同`epoch`下模型的验证精度折线图。 - ```python import matplotlib.pyplot as plt @@ -168,7 +165,6 @@ eval_show(epoch_per_eval) ![png](./images/evaluate_the_model_during_training.png) - 从上图可以一目了然地挑选出需要的最优模型。 ## 总结 diff --git a/tutorials/training/source_zh_cn/advanced_use/improve_model_security_nad.md b/tutorials/training/source_zh_cn/advanced_use/improve_model_security_nad.md index 68020090ef4af4cd0e311deacc06c1ae3873479f..ed4b2d5b3f6f738334bb5d8e4e94a11aafae9e35 100644 --- a/tutorials/training/source_zh_cn/advanced_use/improve_model_security_nad.md +++ b/tutorials/training/source_zh_cn/advanced_use/improve_model_security_nad.md @@ -25,6 +25,7 @@ 本教程介绍MindArmour提供的模型安全防护手段，引导您快速使用MindArmour，为您的AI模型提供一定的安全防护能力。 AI算法设计之初普遍未考虑相关的安全威胁，使得AI算法的判断结果容易被恶意攻击者影响，导致AI系统判断失准。攻击者在原始样本处加入人类不易察觉的微小扰动，导致深度学习模型误判，称为对抗样本攻击。MindArmour模型安全提供对抗样本生成、对抗样本检测、模型防御、攻防效果评估等功能，为AI模型安全研究和AI应用安全提供重要支撑。 + - 对抗样本生成模块支持安全工程师快速高效地生成对抗样本，用于攻击AI模型。 - 对抗样本检测、防御模块支持用户检测过滤对抗样本、增强AI模型对于对抗样本的鲁棒性。 - 评估模块提供多种指标全面评估对抗样本攻防性能。 @@ -32,6 +33,7 @@ AI算法设计之初普遍未考虑相关的安全威胁，使得AI算法的判 这里通过图像分类任务上的对抗性攻防，以攻击算法FGSM和防御算法NAD为例，介绍MindArmour在对抗攻防上的使用方法。 > 本例面向CPU、GPU、Ascend 910 AI处理器，你可以在这里下载完整的样例代码： +> > - `mnist_attack_fgsm.py`：包含攻击代码。 > - `mnist_defense_nad.py`：包含防御代码。 @@ -132,18 +134,18 @@ def generate_mnist_dataset(data_path, batch_size=32, repeat_size=1, return nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding, weight_init=weight, has_bias=False, pad_mode="valid") - - + + def fc_with_initialize(input_channels, out_channels): weight = weight_variable() bias = weight_variable() return nn.Dense(input_channels, out_channels, weight, bias) - - + + def weight_variable(): return TruncatedNormal(0.02) - - + + class LeNet5(nn.Cell): """ Lenet network @@ -158,7 +160,7 @@ def generate_mnist_dataset(data_path, batch_size=32, repeat_size=1, self.relu = nn.ReLU() self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2) self.flatten = nn.Flatten() - + def construct(self, x): x = self.conv1(x) x = self.relu(x) @@ -190,7 +192,7 @@ def generate_mnist_dataset(data_path, batch_size=32, repeat_size=1, model = Model(net, loss, opt, metrics=None) model.train(10, ds_train, callbacks=[LossMonitor()], dataset_sink_mode=False) - + # 2. get test data ds_test = generate_mnist_dataset(os.path.join(mnist_path, "test"), batch_size=batch_size, repeat_size=1, @@ -203,7 +205,7 @@ def generate_mnist_dataset(data_path, batch_size=32, repeat_size=1, test_inputs = np.concatenate(inputs) test_labels = np.concatenate(labels) ``` - + 3. 测试模型。 ```python @@ -217,15 +219,15 @@ def generate_mnist_dataset(data_path, batch_size=32, repeat_size=1, logits = net(Tensor(batch_inputs)).asnumpy() test_logits.append(logits) test_logits = np.concatenate(test_logits) - + tmp = np.argmax(test_logits, axis=1) == np.argmax(test_labels, axis=1) accuracy = np.mean(tmp) LOGGER.info(TAG, 'prediction accuracy before attacking is : %s', accuracy) ``` - + 测试结果中分类精度达到了98%。 - - ```python + + ```python prediction accuracy before attacking is : 0.9895833333333334 ``` @@ -272,7 +274,7 @@ LOGGER.info(TAG, 'The average structural similarity between original ' 攻击结果如下： -``` +```text prediction accuracy after attacking is : 0.052083 mis-classification rate of adversaries is : 0.947917 The average confidence of adversarial class is : 0.803375 @@ -349,7 +351,7 @@ LOGGER.info(TAG, 'The average confidence of true class is : %s', ### 防御效果 -``` +```text accuracy of TEST data on defensed model is : 0.974259 accuracy of adv data on defensed model is : 0.856370 defense mis-classification rate of adversaries is : 0.143629 @@ -358,4 +360,3 @@ The average confidence of true class is : 0.177374 ``` 使用NAD进行对抗样本防御后，模型对于对抗样本的误分类率从95%降至14%，模型有效地防御了对抗样本。同时，模型对于原来测试数据集的分类精度达97%。 - diff --git a/tutorials/training/source_zh_cn/advanced_use/lineage_and_scalars_comparision.md b/tutorials/training/source_zh_cn/advanced_use/lineage_and_scalars_comparision.md index f7a55d1c793bd401e67156859c4dce0536293100..555ef0fee8d2c087de64f6852395e3c3ba70bdda 100644 --- a/tutorials/training/source_zh_cn/advanced_use/lineage_and_scalars_comparision.md +++ b/tutorials/training/source_zh_cn/advanced_use/lineage_and_scalars_comparision.md @@ -106,6 +106,7 @@ MindInsight中的模型溯源、数据溯源和对比看板同训练看板一样 ## 注意事项 出于性能上的考虑，MindInsight对比看板使用缓存机制加载训练的标量曲线数据，并进行以下限制： + - 对比看板只支持在缓存中的训练进行比较标量曲线对比。 - 缓存最多保留最新（按修改时间排列）的15个训练。 - 用户最多同时对比5个训练的标量曲线。 diff --git a/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts.md b/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts.md index 7b94469d212649f5dc183f295a6618dfa192328a..d382670fc2dfbb67b8bd3a51e107b0a20df5a9da 100644 --- a/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts.md +++ b/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts.md @@ -36,10 +36,12 @@ 以ResNet-50为例，[Conv](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.Conv2d)和[BatchNorm](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.nn.html#mindspore.nn.BatchNorm2d)是其中最主要的两个算子，它们已在MindSpore支持的算子列表中。 如果发现没有对应算子，建议： + - 使用其他算子替换：分析算子实现公式，审视是否可以采用MindSpore现有算子叠加达到预期目标。 - 临时替代方案：比如不支持某个Loss，是否可以替换为同类已支持的Loss算子；又比如当前的网络结构，是否可以替换为其他同类主流网络等。 如果发现支持的算子存在功能不全，建议： + - 非必要功能：可删除。 - 必要功能：寻找替代方案。 @@ -68,7 +70,7 @@ MindSpore与TensorFlow、PyTorch在网络结构组织方式上，存在一定差 2. 加载数据集和预处理。 使用MindSpore构造你需要使用的数据集。目前MindSpore已支持常见数据集，你可以通过原始格式、`MindRecord`、`TFRecord`等多种接口调用，同时还支持数据处理以及数据增强等相关功能，具体用法可参考[准备数据教程](https://www.mindspore.cn/tutorial/training/zh-CN/master/use/data_preparation.html)。 - + 本例中加载了Cifar-10数据集，可同时支持单卡和多卡的场景。 ```python @@ -78,7 +80,7 @@ MindSpore与TensorFlow、PyTorch在网络结构组织方式上，存在一定差 ds = de.Cifar10Dataset(dataset_path, num_parallel_workers=4, shuffle=True, num_shards=device_num, shard_id=rank_id) ``` - + 然后对数据进行了数据增强、数据清洗和批处理等操作。代码详见。 3. 构建网络。 @@ -234,13 +236,13 @@ MindSpore与TensorFlow、PyTorch在网络结构组织方式上，存在一定差 ``` 如果希望使用`Model`内置的评估方法，则可以使用[metrics](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/custom_debugging_info.html#mindspore-metrics)属性设置希望使用的评估方法。 - + ```python model = Model(net, loss_fn=loss, optimizer=opt, loss_scale_manager=loss_scale, metrics={'acc'}) ``` 类似于TensorFlow的`estimator.train`，可以通过调用`model.train`接口来进行训练。CheckPoint和中间结果打印等功能，可通过`Callback`的方式定义到`model.train`接口上。 - + ```python time_cb = TimeMonitor(data_size=step_size) loss_cb = LossMonitor() @@ -256,6 +258,7 @@ MindSpore与TensorFlow、PyTorch在网络结构组织方式上，存在一定差 #### 精度调试 精度调优过程建议如下两点： + 1. 单卡精度验证时，建议先采用小数据集进行训练。验证达标后，多卡精度验证时，再采用全量数据集。这样可以帮助提升调试效率。 2. 首先删减脚本中的不必要技巧（如优化器中的增强配置、动态Loss Scale等），验证达标后，在此基础上逐个叠加新增功能，待当前新增功能确认正常后，再叠加下一个功能。这样可以帮助快速定位问题。 diff --git a/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts_mindconverter.md b/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts_mindconverter.md index 4128c54a2c594fac6eb46e8964326cdc8538bfe4..6cf61899c7b2a90c36839a3f4a5c132432f41f12 100644 --- a/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts_mindconverter.md +++ b/tutorials/training/source_zh_cn/advanced_use/migrate_3rd_scripts_mindconverter.md @@ -22,14 +22,10 @@ MindConverter是一款将PyTorch模型脚本转换至MindSpore的脚本迁移工具。结合转换报告的提示信息，用户对转换后脚本进行微小改动，即可快速将PyTorch模型脚本迁移至MindSpore。 - - ## 安装 此工具为MindInsight的子模块，安装MindInsight后，即可使用MindConverter，MindInsight安装请参考该[安装文档](https://www.mindspore.cn/install/)。 - - ## 用法 MindConverter提供命令行（Command-line interface, CLI）的使用方式，命令如下。 @@ -79,15 +75,13 @@ optional arguments: 另外，当使用基于图结构的脚本生成方案时，请确保原PyTorch项目已在Python包搜索路径中，可通过CLI进入Python交互式命令行，通过import的方式判断是否已满足；若未加入，可通过`--project_path`命令手动将项目路径传入，以确保MindConverter可引用到原PyTorch脚本。 - > 假设用户项目目录为`/home/user/project/model_training`，用户可通过如下命令手动项目添加至包搜索路径中：`export PYTHONPATH=/home/user/project/model_training:$PYTHONPATH` - > 此处MindConverter需要引用原PyTorch脚本，是因为PyTorch模型反向序列化过程中会引用原脚本。 - ## 使用场景 MindConverter提供两种技术方案，以应对不同脚本迁移场景： + 1. 用户希望迁移后脚本保持原有PyTorch脚本结构（包括变量、函数、类命名等与原脚本保持一致）； 2. 用户希望迁移后脚本保持较高的转换率，尽量少的修改、甚至不需要修改，即可实现迁移后模型脚本的执行。 @@ -101,7 +95,6 @@ MindConverter提供两种技术方案，以应对不同脚本迁移场景： > 2. 基于图结构的脚本生成方案，由于要基于推理模式加载PyTorch模型，会导致转换后网络中Dropout算子丢失，需要用户手动补齐； > 3. 基于图结构的脚本生成方案持续优化中。 - ## 使用示例 ### 基于AST的脚本转换示例 @@ -121,6 +114,7 @@ line x:y: [UnConvert] 'operator' didn't convert. ... ``` 转换报告示例如下所示： + ```text [Start Convert] [Insert] 'import mindspore.ops.operations as P' is inserted to the converted file. @@ -133,7 +127,6 @@ line x:y: [UnConvert] 'operator' didn't convert. ... 对于部分未成功转换的算子，报告中会提供修改建议，如`line 157:23`，MindConverter建议将`torch.nn.AdaptiveAvgPool2d`替换为`mindspore.ops.operations.ReduceMean`。 - ### 基于图结构的脚本生成示例 若用户已将PyTorch模型保存为.pth格式，假设模型绝对路径为`/home/user/model.pth`，该模型期望的输入样本shape为(3, 224, 224)，原PyTorch脚本位于`/home/user/project/model_training`，希望将脚本输出至`/home/user/output`，转换报告输出至`/home/user/output/report`，则脚本生成命令为： @@ -147,10 +140,8 @@ mindconverter --model_file /home/user/model.pth --shape 3,224,224 \ 执行该命令，MindSpore代码文件、转换报告生成至相应目录。 - 基于图结构的脚本生成方案产生的转换报告格式与AST方案相同。然而，由于基于图结构方案属于生成式方法，转换过程中未参考原PyTorch脚本，因此生成的转换报告中涉及的代码行、列号均指生成后脚本。 - 另外对于未成功转换的算子，在代码中会相应的标识该节点输入、输出Tensor的shape（以`input_shape`, `output_shape`标识），便于用户手动修改。以Reshape算子为例（暂不支持Reshape），将生成如下代码： ```python @@ -194,7 +185,6 @@ class Classifier(nn.Cell): ``` - > 其中`--output`与`--report`参数可省略，若省略，该命令将在当前工作目录（Working directory）下自动创建`output`目录，将生成的脚本、转换报告输出至该目录。 ## 注意事项 diff --git a/tutorials/training/source_zh_cn/advanced_use/nlp_bert_poetry.md b/tutorials/training/source_zh_cn/advanced_use/nlp_bert_poetry.md index d595dbef535f308296582fc1edec0628d35d278b..27cbdb6297c7c7091272562aaa64d79554952cac 100644 --- a/tutorials/training/source_zh_cn/advanced_use/nlp_bert_poetry.md +++ b/tutorials/training/source_zh_cn/advanced_use/nlp_bert_poetry.md @@ -87,7 +87,7 @@ BERT采用了Encoder结构，`attention_mask`为全1的向量，即每个token 样例代码可[点击下载](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com:443/DemoCode/bert_poetry_c.rar)，可直接运行体验实现写诗效果，代码结构如下: -``` +```text └─bert_poetry ├── src ├── bert_for_pre_training.py # 封装BERT-Base正反向网络类 @@ -107,7 +107,7 @@ BERT采用了Encoder结构，`attention_mask`为全1的向量，即每个token ├── poetry_client.py # 客户端代码 ├── ms_service_pb2_grpc.py # 定义了grpc相关函数供bert_flask.py使用 └── ms_service_pb2.py # 定义了protocol buffer相关函数供bert_flask.py使用 - + ``` ## 实现步骤 @@ -118,7 +118,6 @@ BERT采用了Encoder结构，`attention_mask`为全1的向量，即每个token ### 数据准备 - 数据集为43030首诗词：可[下载](https://github.com/AaronJny/DeepLearningExamples/tree/master/keras-bert-poetry-generator)其中的`poetry.txt`。 BERT-Base模型的预训练ckpt：可在[MindSpore官网](http://download.mindspore.cn/model_zoo/official/nlp/bert/bert_base_ascend_0.5.0_cn-wiki_official_nlp_20200720.tar.gz)下载。 @@ -127,7 +126,7 @@ BERT-Base模型的预训练ckpt：可在[MindSpore官网](http://download.mindsp 在`src/finetune_config.py`中修改`pre_training_ckpt`路径，加载预训练的ckpt，修改`batch_size`为bs，修改`dataset_path`为存放诗词的路径，默认的`BertConfig`为Base模型。 -``` +```python 'dataset_path': '/your/path/to/poetry.txt', 'batch_size': bs, 'pre_training_ckpt': '/your/path/to/pre_training_ckpt', @@ -135,7 +134,7 @@ BERT-Base模型的预训练ckpt：可在[MindSpore官网](http://download.mindsp 执行训练指令 -``` +```bash python poetry.py ``` @@ -145,19 +144,20 @@ python poetry.py `generate_random_poetry`函数实现随机生成和续写诗句的功能，如果入参`s`为空则代表随机生成，`s`不为空则为续写诗句。 -``` +```python output = generate_random_poetry(poetrymodel, s='') #随机生成 output = generate_random_poetry(poetrymodel, s='天下为公') #续写诗句 ``` `generate_hidden`函数实现生成藏头诗的功能，入参`head`为隐藏的头部语句。 -``` + +```python output = generate_hidden(poetrymodel, head="人工智能") #藏头诗 ``` 执行推理指令 -``` +```bash python poetry.py --train=False --ckpt_path=/your/ckpt/path ``` @@ -165,7 +165,7 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 随机生成： -``` +```text 大堤柳暗， 春深树根。 东望一望， @@ -178,7 +178,7 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 续写 【天下为公】: -``` +```text 天下为公少， 唯君北向西。 远山无路见， @@ -191,7 +191,7 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 藏头诗 【人工智能】: -``` +```text 人君离别难堪望， 工部张机自少年。 智士不知身没处， @@ -206,7 +206,7 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 在使用Serving部署服务前，需要导出模型文件，在`poetry.py`中提供了`export_net`函数负责导出MINDIR模型，执行命令: - ``` + ```bash python poetry.py --export=True --ckpt_path=/your/ckpt/path ``` @@ -216,7 +216,7 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 在服务器侧启动Serving服务，并加载导出的MINDIR文件`poetry.pb`。 - ``` + ```bash cd serving ./ms_serving --model_path=/path/to/your/MINDIR_file --model_name=your_mindir.pb ``` @@ -225,7 +225,7 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 预处理及后处理通过Flask框架来快速实现，在服务器侧运行`bert_flask.py`文件，启动Flask服务。 - ``` + ```bash python bert_flask.py ``` @@ -235,36 +235,38 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 可用电脑作为客户端，修改`poetry_client.py`中的url请求地址为推理服务启动的服务器IP，并确保端口与服务端`bert_flask.py`中的端口一致，例如： - ``` + ```python url = 'http://10.155.170.71:8080/' ``` 运行`poetry_client.py`文件 - ``` + ```bash python poetry_client.py ``` 此时在客户端输入指令，即可在远端服务器进行推理，返回生成的诗句。 - ``` + ```text 选择模式：0-随机生成，1：续写，2：藏头诗 0 ``` - ``` + + ```text 一朵黄花叶， 千竿绿树枝。 含香待夏晚， 澹浩长风时。 ``` - ``` + ```text 选择模式：0-随机生成，1：续写，2：藏头诗 1 输入首句诗 明月 ``` - ``` + + ```text 明月照三峡， 长空一片云。 秋风与雨过， @@ -275,13 +277,14 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 何道逐风君。 ``` - ``` + ```text 选择模式：0-随机生成，1：续写，2：藏头诗 2 输入藏头诗 人工智能 ``` - ``` + + ```text 人生事太远， 工部与神期。 智者岂无识， @@ -290,10 +293,8 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path 细读鉴赏一下，平仄、押韵、意味均有体现，AI诗人已然成形。 - > 友情提醒，修改其他类型数据集，也可以完成其他简单的生成类任务，如对春联，简单聊天机器人等，用户可尝试体验实现。 - ## 参考文献 [1] [BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) @@ -301,4 +302,3 @@ python poetry.py --train=False --ckpt_path=/your/ckpt/path [2] [https://github.com/AaronJny/DeepLearningExamples/](https://github.com/AaronJny/DeepLearningExamples/) [3] [https://github.com/bojone/bert4keras](https://github.com/bojone/bert4keras) - diff --git a/tutorials/training/source_zh_cn/advanced_use/nlp_sentimentnet.md b/tutorials/training/source_zh_cn/advanced_use/nlp_sentimentnet.md index 9dcc7157aebbdbc6ca221656b57322f84525e722..e73bbcaf3b629a66a603977dd014998a5292b9f5 100644 --- a/tutorials/training/source_zh_cn/advanced_use/nlp_sentimentnet.md +++ b/tutorials/training/source_zh_cn/advanced_use/nlp_sentimentnet.md @@ -48,6 +48,7 @@ $垂直极性词 = 通用极性词 + 领域特有极性词$ 按照处理文本的粒度不同，情感分析可分为词语级、短语级、句子级、段落级以及篇章级等几个研究层次。这里以“段落级”为例，输入为一个段落，输出为影评是正面还是负面的信息。 ## 准备及设计 + ### 下载数据集 采用IMDb影评数据集作为实验数据。 @@ -55,15 +56,17 @@ $垂直极性词 = 通用极性词 + 领域特有极性词$ 以下是负面影评（Negative）和正面影评（Positive）的案例。 -| Review | Label | +| Review | Label | |---|---| | "Quitting" may be as much about exiting a pre-ordained identity as about drug withdrawal. As a rural guy coming to Beijing, class and success must have struck this young artist face on as an appeal to separate from his roots and far surpass his peasant parents' acting success. Troubles arise, however, when the new man is too new, when it demands too big a departure from family, history, nature, and personal identity. The ensuing splits, and confusion between the imaginary and the real and the dissonance between the ordinary and the heroic are the stuff of a gut check on the one hand or a complete escape from self on the other. | Negative | | This movie is amazing because the fact that the real people portray themselves and their real life experience and do such a good job it's like they're almost living the past over again. Jia Hongsheng plays himself an actor who quit everything except music and drugs struggling with depression and searching for the meaning of life while being angry at everyone especially the people who care for him most. | Positive | 同时，我们要下载GloVe文件，并在文件开头处添加新的一行，意思是总共读取400000个单词，每个单词用300纬度的词向量表示。 -``` + +```text 400000 300 ``` + GloVe文件下载地址：。 ### 确定评价标准 @@ -74,22 +77,23 @@ $精度（Accuracy）= 分类正确的样本数目 / 总样本数目$ $精准度（Precision）= 真阳性样本数目 / 所有预测类别为阳性的样本数目$ -$召回率（Recall）= 真阳性样本数目 / 所有真实类别为阳性的样本数目$ +$召回率（Recall）= 真阳性样本数目 / 所有真实类别为阳性的样本数目$ -$F1分数 = (2 * Precision * Recall) / (Precision + Recall)$ +$F1分数 = (2 \* Precision \* Recall) / (Precision + Recall)$ 在IMDb这个数据集中，正负样本数差别不大，可以简单地用精度（accuracy）作为分类器的衡量标准。 - ### 确定网络及流程 我们使用基于LSTM构建的SentimentNet网络进行自然语言处理。 + 1. 加载使用的数据集，并进行必要的数据处理。 2. 使用基于LSTM构建的SentimentNet网络训练数据，生成模型。 > LSTM（Long short-term memory，长短期记忆）网络是一种时间循环神经网络，适合于处理和预测时间序列中间隔和延迟非常长的重要事件。具体介绍可参考网上资料，在此不再赘述。 3. 得到模型之后，使用验证数据集，查看模型精度情况。 > 本例面向GPU或CPU硬件平台，你可以在这里下载完整的样例代码： +> > - `src/config.py`：网络中的一些配置，包括`batch size`、进行几次epoch训练等。 > - `src/dataset.py`：数据集相关，包括转换成MindRecord文件，数据预处理等。 > - `src/imdb.py`： 解析IMDb数据集的工具。 @@ -98,8 +102,11 @@ $F1分数 = (2 * Precision * Recall) / (Precision + Recall)$ > - `eval.py`：模型的推理脚本。 ## 实现阶段 + ### 导入需要的库文件 + 下列是我们所需要的公共模块及MindSpore的模块及库文件。 + ```python import argparse import os @@ -119,6 +126,7 @@ from mindspore.train.serialization import load_param_into_net, load_checkpoint ### 配置环境信息 1. 使用`parser`模块，传入运行必要的信息，如数据集存放路径，GloVe存放路径，这样的好处是，对于经常变化的配置，可以在运行代码时输入，使用更加灵活。 + ```python parser = argparse.ArgumentParser(description='MindSpore LSTM Example') parser.add_argument('--preprocess', type=str, default='false', choices=['true', 'false'], @@ -139,12 +147,14 @@ from mindspore.train.serialization import load_param_into_net, load_checkpoint ``` 2. 实现代码前，需要配置必要的信息，包括环境信息、执行的模式、后端信息及硬件信息。 + ```python context.set_context( mode=context.GRAPH_MODE, save_graphs=False, device_target=args.device_target) ``` + 详细的接口配置信息，请参见`context.set_context`接口说明。 ### 预处理数据集 @@ -156,15 +166,14 @@ if args.preprocess == "true": print("============== Starting Data Pre-processing ==============") convert_to_mindrecord(cfg.embed_size, args.aclimdb_path, args.preprocess_path, args.glove_path) ``` -> 转换成功后会在`preprocess_path`路径下生成`mindrecord`文件； 通常该操作在数据集不变的情况下，无需每次训练都执行。 +> 转换成功后会在`preprocess_path`路径下生成`mindrecord`文件； 通常该操作在数据集不变的情况下，无需每次训练都执行。 > `convert_to_mindrecord`函数的具体实现请参考 - > 其中包含两大步骤： +> > 1. 解析文本数据集，包括编码、分词、对齐、处理GloVe原始数据，使之能够适应网络结构。 > 2. 转换并保存为MindRecord格式数据集。 - ### 定义网络 ```python @@ -178,11 +187,13 @@ network = SentimentNet(vocab_size=embedding_table.shape[0], weight=Tensor(embedding_table), batch_size=cfg.batch_size) ``` + > `SentimentNet`网络结构的具体实现请参考 ### 预训练模型 通过参数`pre_trained`指定预加载CheckPoint文件来进行预训练，默认该参数为空。 + ```python if args.pre_trained: load_param_into_net(network, load_checkpoint(args.pre_trained)) @@ -217,6 +228,7 @@ else: model.train(cfg.num_epochs, ds_train, callbacks=[time_cb, ckpoint_cb, loss_cb]) print("============== Training Success ==============") ``` + > `lstm_create_dataset`函数的具体实现请参考 ### 模型验证 @@ -238,12 +250,15 @@ print("============== {} ==============".format(acc)) ``` ## 实验结果 + 在经历了20轮epoch之后，在测试集上的精度约为84.19%。 -**执行训练** +### 执行训练 + 1. 运行训练代码，查看运行结果。 + ```shell - $ python train.py --preprocess=true --ckpt_path=./ --device_target=GPU + python train.py --preprocess=true --ckpt_path=./ --device_target=GPU ``` 输出如下，可以看到loss值随着训练逐步降低，最后达到0.2855左右： @@ -261,13 +276,13 @@ print("============== {} ==============".format(acc)) epoch: 20 step: 389, loss is 0.1354 epoch: 20 step: 390, loss is 0.2855 ``` - + 2. 查看保存的CheckPoint文件。 - + 训练过程中保存了CheckPoint文件，即模型文件，我们可以查看文件保存的路径下的所有保存文件。 ```shell - $ ls ./*.ckpt + ls ./*.ckpt ``` 输出如下： @@ -276,12 +291,12 @@ print("============== {} ==============".format(acc)) lstm-11_390.ckpt lstm-12_390.ckpt lstm-13_390.ckpt lstm-14_390.ckpt lstm-15_390.ckpt lstm-16_390.ckpt lstm-17_390.ckpt lstm-18_390.ckpt lstm-19_390.ckpt lstm-20_390.ckpt ``` -**验证模型** +### 验证模型 使用最后保存的CheckPoint文件，加载验证数据集，进行验证。 ```shell -$ python eval.py --ckpt_path=./lstm-20_390.ckpt --device_target=GPU +python eval.py --ckpt_path=./lstm-20_390.ckpt --device_target=GPU ``` 输出如下，可以看到使用验证的数据集，对文本的情感分析正确率在84.19%左右，达到一个基本满意的结果。 @@ -290,6 +305,3 @@ $ python eval.py --ckpt_path=./lstm-20_390.ckpt --device_target=GPU ============== Starting Testing ============== ============== {'acc': 0.8419471153846154} ============== ``` - - - diff --git a/tutorials/training/source_zh_cn/advanced_use/optimize_data_processing.md b/tutorials/training/source_zh_cn/advanced_use/optimize_data_processing.md index 30c79d1cbc7d233fe3cc48513a7d55da416b1bae..256a17af7d87826fc63aa65c5a718db3c0065277 100644 --- a/tutorials/training/source_zh_cn/advanced_use/optimize_data_processing.md +++ b/tutorials/training/source_zh_cn/advanced_use/optimize_data_processing.md @@ -54,7 +54,7 @@ import numpy as np 目录结构如下所示： -``` +```text dataset/Cifar10Data ├── cifar-10-batches-bin │   ├── batches.meta.txt @@ -77,6 +77,7 @@ dataset/Cifar10Data ``` 其中： + - `cifar-10-batches-bin`目录为CIFAR-10二进制格式数据集目录。 - `cifar-10-batches-py`目录为CIFAR-10 Python文件格式数据集目录。 @@ -94,6 +95,7 @@ MindSpore为用户提供了多种数据加载方式，其中包括常用数据 ![title](./images/data_loading_performance_scheme.png) 数据加载性能优化建议如下： + - 已经支持的数据集格式优选内置加载算子，具体内容请参考[内置加载算子](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.dataset.html)，如果性能仍无法满足需求，则可采取多线程并发方案，请参考本文[多线程优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id16)。 - 不支持的数据集格式，优选转换为MindSpore数据格式后再使用`MindDataset`类进行加载，具体内容请参考[MindSpore数据格式转换](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/dataset_conversion.html)，如果性能仍无法满足需求，则可采取多线程并发方案，请参考本文[多线程优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id16)。 - 不支持的数据集格式，算法快速验证场景，优选用户自定义`GeneratorDataset`类实现，如果性能仍无法满足需求，则可采取多进程并发方案，请参考本文[多进程优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id17)。 @@ -115,7 +117,7 @@ MindSpore为用户提供了多种数据加载方式，其中包括常用数据 输出： - ``` + ```text {'image': Tensor(shape=[32, 32, 3], dtype=UInt8, value= [[[235, 235, 235], [230, 230, 230], @@ -150,7 +152,7 @@ MindSpore为用户提供了多种数据加载方式，其中包括常用数据 输出： - ``` + ```text {'data': Tensor(shape=[1431], dtype=UInt8, value= [255, 216, 255, ..., 63, 255, 217]), 'id': Tensor(shape=[], dtype=Int64, value= 30474), 'label': Tensor(shape=[], dtype=Int64, value= 2)} @@ -171,7 +173,7 @@ MindSpore为用户提供了多种数据加载方式，其中包括常用数据 输出： - ``` + ```text {'data': Tensor(shape=[1], dtype=Int64, value= [0])} ``` @@ -184,6 +186,7 @@ shuffle操作主要是对有序的数据集或者进行过repeat的数据集进 ![title](./images/shuffle_performance_scheme.png) shuffle性能优化建议如下： + - 直接使用内置加载算子的`shuffle`参数进行数据的混洗。 - 如果使用的是`shuffle`函数，当性能仍无法满足需求，可通过调整`buffer_size`参数的值来优化提升性能。 @@ -204,7 +207,7 @@ shuffle性能优化建议如下： 输出： - ``` + ```text {'image': Tensor(shape=[32, 32, 3], dtype=UInt8, value= [[[254, 254, 254], [255, 255, 254], @@ -239,7 +242,7 @@ shuffle性能优化建议如下： 输出： - ``` + ```text before shuffle: [0 1 2 3 4] [1 2 3 4 5] @@ -257,6 +260,7 @@ shuffle性能优化建议如下： ## 数据增强性能优化 在图片分类的训练中，尤其是当数据集比较小的时候，用户可以使用数据增强的方式来预处理图片，从而丰富数据集。MindSpore为用户提供了多种数据增强的方式，其中包括： + - 使用内置C算子（`c_transforms`模块）进行数据增强。 - 使用内置Python算子（`py_transforms`模块）进行数据增强。 - 用户可根据自己的需求，自定义Python函数进行数据增强。 @@ -273,6 +277,7 @@ shuffle性能优化建议如下： ![title](./images/data_enhancement_performance_scheme.png) 数据增强性能优化建议如下： + - 优先使用`c_transforms`模块进行数据增强，因为性能最高，如果性能仍无法满足需求，可采取[多线程优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id16)、[Compose优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#compose)或者[算子融合优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id18)。 - 如果使用了`py_transforms`模块进行数据增强，当性能仍无法满足需求，可采取[多线程优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id16)、[多进程优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id17)、[Compose优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#compose)或者[算子融合优化方案](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/optimize_data_processing.html#id18)。 - `c_transforms`模块是在C++内维护buffer管理，`py_transforms`模块是在Python内维护buffer管理。因为Python和C++切换的性能成本，建议不要混用算子。 @@ -326,7 +331,7 @@ shuffle性能优化建议如下： 输出： - ``` + ```text before map: [0 1 2 3 4] [1 2 3 4 5] @@ -394,6 +399,7 @@ shuffle性能优化建议如下： ### 多线程优化方案 在数据pipeline过程中，相关算子一般都有线程数设置参数，来提升处理并发度，提升性能，例如： + - 在数据加载的过程中，内置数据加载类有`num_parallel_workers`参数用来设置线程数。 - 在数据增强的过程中，`map`函数有`num_parallel_workers`参数用来设置线程数。 - 在Batch的过程中，`batch`函数有`num_parallel_workers`参数用来设置线程数。 @@ -403,6 +409,7 @@ shuffle性能优化建议如下： ### 多进程优化方案 数据处理中Python实现的算子均支持多进程的模式，例如： + - `GeneratorDataset`这个类默认是多进程模式，它的`num_parallel_workers`参数表示的是开启的进程数，默认为1，具体内容请参考[GeneratorDataset](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.dataset.html#mindspore.dataset.GeneratorDataset)。 - 如果使用Python自定义函数或者`py_transforms`模块进行数据增强的时候，当`map`函数的参数`python_multiprocessing`设置为True时，此时参数`num_parallel_workers`表示的是进程数，参数`python_multiprocessing`默认为False，此时参数`num_parallel_workers`表示的是线程数，具体的内容请参考[内置加载算子](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.dataset.html)。 diff --git a/tutorials/training/source_zh_cn/advanced_use/performance_profiling.md b/tutorials/training/source_zh_cn/advanced_use/performance_profiling.md index cea56ee54da1bb4d6785807ca53909d321ec57cc..c603d4164123779e9ac35e60180b58e372808da6 100644 --- a/tutorials/training/source_zh_cn/advanced_use/performance_profiling.md +++ b/tutorials/training/source_zh_cn/advanced_use/performance_profiling.md @@ -22,6 +22,7 @@ ## 概述 + 将训练过程中的算子耗时等信息记录到文件中，通过可视化界面供用户查看分析，帮助用户更高效地调试神经网络性能。 ## 操作流程 @@ -31,11 +32,13 @@ - 在训练列表找到对应训练，点击性能分析，即可在页面中查看训练性能数据。 ## 环境准备 + 在使用性能分析工具之前，要确保后台工具进程（ada）正确启动，要求用户使用HwHiAiUser用户组的用户或root启动ada进程，并使用同用户跑训练脚本，启动命令为：`/usr/local/Ascend/driver/tools/ada`。 ## 准备训练脚本 为了收集神经网络的性能数据，需要在训练脚本中添加MindSpore Profiler相关接口。 + - `set_context`之后，初始化网络和HCCL之前，需要初始化MindSpore `Profiler`对象。 > Profiler支持的参数可以参考： @@ -54,10 +57,10 @@ from mindspore import Model, nn, context def test_profiler(): # Init context env context.set_context(mode=context.GRAPH_MODE, device_target='Ascend', device_id=int(os.environ["DEVICE_ID"])) - + # Init Profiler profiler = Profiler() - + # Init hyperparameter epoch = 2 # Init network and Model @@ -69,7 +72,7 @@ def test_profiler(): train_ds = create_mindrecord_dataset_for_training() # Model Train model.train(epoch, train_ds) - + # Profiler end profiler.analyse() ``` @@ -78,7 +81,6 @@ def test_profiler(): 启动命令请参考[MindInsight相关命令](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/mindinsight_commands.html)。 - ### 性能分析 用户从训练列表中选择指定的训练，点击性能调试，可以查看该次训练的性能数据。 @@ -88,6 +90,7 @@ def test_profiler(): 图1：性能数据总览 图1展示了性能数据总览页面，包含了迭代轨迹（Step Trace）、算子性能、MindData性能和Timeline等组件的数据总体呈现。各组件展示的数据如下： + - 迭代轨迹：将训练step划分为几个阶段，统计每个阶段的耗时，按时间线进行展示；总览页展示了迭代轨迹图。 - 算子性能：统计单算子以及各算子类型的执行时间，进行排序展示；总览页中展示了各算子类型时间占比的饼状图。 - MindData性能：统计训练数据准备各阶段的性能情况；总览页中展示了各阶段性能可能存在瓶颈的step数目。 @@ -108,6 +111,7 @@ def test_profiler(): 迭代轨迹在做阶段划分时，需要识别前向计算开始的算子和反向计算结束的算子。为了降低用户使用Profiler的门槛，MindSpore会对这两个算子做自动识别，方法为： 前向计算开始的算子指定为`get_next`算子之后连接的第一个算子，反向计算结束的算子指定为最后一次all reduce之前连接的算子。**Profiler不保证在所有情况下自动识别的结果和用户的预期一致，用户可以根据网络的特点自行调整**，调整方法如下： + - 设置`FP_POINT`环境变量指定前向计算开始的算子，如`export FP_POINT=fp32_vars/conv2d/BatchNorm`。 - 设置`BP_POINT`环境变量指定反向计算结束的算子，如`export BP_POINT=loss_scale/gradients/AddN_70`。 @@ -120,6 +124,7 @@ def test_profiler(): 图3：算子类别统计分析 图3展示了按算子类别进行统计分析的结果，包含以下内容： + - 可以选择饼图/柱状图展示各算子类别的时间占比，每个算子类别的执行时间会统计属于该类别的算子执行时间总和。 - 统计前20个占比时间最长的算子类别，展示其时间所占的百分比以及具体的执行时间（毫秒）。 @@ -128,6 +133,7 @@ def test_profiler(): 图4：算子统计分析 图4展示了算子性能统计表，包含以下内容： + - 选择全部：按单个算子的统计结果进行排序展示，展示维度包括算子名称、算子类型、算子执行时间、算子全scope名称、算子信息等；默认按算子执行时间排序。 - 选择分类：按算子类别的统计结果进行排序展示，展示维度包括算子分类名称、算子类别执行时间、执行频次、占总时间的比例等。点击每个算子类别，可以进一步查看该类别下所有单个算子的统计信息。 - 搜索：在右侧搜索框中输入字符串，支持对算子名称/类别进行模糊搜索。 @@ -143,6 +149,7 @@ def test_profiler(): 图5展示了MindData性能分析页面，包含迭代间隙和数据处理两个TAB页面。 迭代间隙TAB页主要用来分析数据准备三个阶段是否存在性能瓶颈，数据队列图是分析判断的重要依据： + - 数据队列Size代表Device侧从队列取数据时队列的长度，如果数据队列Size为0，则训练会一直等待，直到队列中有数据才会开始某个step的训练；如果数据队列Size大于0，则训练可以快速取到数据，MindData不是该step的瓶颈所在。 - 主机队列Size可以推断出数据处理和发送速度，如果主机队列Size为0，表示数据处理速度慢而数据发送速度快，需要加快数据处理。 - 如果主机队列Size一直较大，而数据队列的Size持续很小，则数据发送有可能存在性能瓶颈。 @@ -154,20 +161,22 @@ def test_profiler(): 图6展示了数据处理TAB页面，可以对数据处理pipeline做进一步分析。不同的数据算子之间使用队列进行数据交换，队列的长度可以反映出算子处理数据的快慢，进而推断出pipeline中的瓶颈算子所在。 算子队列的平均使用率代表队列中已有数据Size除以队列最大数据Size的平均值，使用率越高说明队列中数据积累越多。算子队列关系展示了数据处理pipeline中的算子以及它们之间的连接情况，点击某个队列可以在下方查看该队列中数据Size随着时间的变化曲线，以及与数据队列连接的算子信息等。对数据处理pipeline的分析有如下建议： + - 当算子左边连接的Queue使用率都比较高，右边连接的Queue使用率比较低，该算子可能是性能瓶颈。 - 对于最左侧的算子，如果其右边所有Queue的使用率都比较低，该算子可能是性能瓶颈。 - 对于最右侧的算子，如果其左边所有Queue的使用率都比较高，该算子可能是性能瓶颈。 对于不同的类型的MindData算子，有如下优化建议： + - 如果Dataset算子是性能瓶颈，建议增加`num_parallel_workers`。 - 如果GeneratorOp类型的算子是性能瓶颈，建议增加`num_parallel_workers`，并尝试将其替换为`MindRecordDataset`。 - 如果MapOp类型的算子是性能瓶颈，建议增加`num_parallel_workers`，如果该算子为Python算子，可以尝试优化脚本。 - 如果BatchOp类型的算子是性能瓶颈，建议调整`prefetch_size`的大小。 - #### Timeline分析 Timeline组件可以展示： + - 算子分配到哪个设备（AICPU、AICore等）执行。 - MindSpore对该网络的流切分策略。 - 算子在Device上的执行序列和执行时长。 @@ -175,6 +184,7 @@ Timeline组件可以展示： 通过分析Timeline，用户可以对训练过程进行细粒度分析：从High Level层面，可以分析流切分方法是否合理、迭代间隙和拖尾时间是否过长等；从Low Level层面，可以分析算子执行时间等。 用户可以点击总览页面Timeline部分的下载按钮，将Timeline数据文件 (json格式) 保存至本地，再通过工具查看Timeline的详细信息。推荐使用 `chrome://tracing` 或者 [Perfetto](https://ui.perfetto.dev/#!viewer) 做Timeline展示。 + - Chrome tracing：点击左上角"load"加载文件。 - Perfetto：点击左侧"Open trace file"加载文件。 @@ -183,12 +193,12 @@ Timeline组件可以展示： 图7：Timeline分析 Timeline主要包含如下几个部分： + - Device及其stream list：包含Device上的stream列表，每个stream由task执行序列组成，一个task是其中的一个小方块，大小代表执行时间长短。 - 算子信息：选中某个task后，可以显示该task对应算子的信息，包括名称、type等。 可以使用W/A/S/D来放大、缩小地查看Timeline图信息。 - ## 规格 - 为了控制性能测试时生成数据的大小，大型网络建议性能调试的step数目限制在10以内。 diff --git a/tutorials/training/source_zh_cn/advanced_use/performance_profiling_gpu.md b/tutorials/training/source_zh_cn/advanced_use/performance_profiling_gpu.md index 6b98ea73a57f15127956b710ac956487978eff51..5ebda9c4bb1285988444a4ae066d03004a80aa2e 100644 --- a/tutorials/training/source_zh_cn/advanced_use/performance_profiling_gpu.md +++ b/tutorials/training/source_zh_cn/advanced_use/performance_profiling_gpu.md @@ -18,6 +18,7 @@ ## 概述 + 将训练过程中的算子耗时等信息记录到文件中，通过可视化界面供用户查看分析，帮助用户更高效地调试神经网络性能。 ## 操作流程 @@ -25,7 +26,6 @@ > 操作流程可以参考Ascend 910上profiler的操作： > > - > 普通用户默认情况下无权访问目标设备上的NVIDIA GPU性能计数器。如果普通用户需要在训练脚本中使用profiler性能统计能力，则需参考以下网址的说明进行权限配置。 > > @@ -33,6 +33,7 @@ ## 准备训练脚本 为了收集神经网络的性能数据，需要在训练脚本中添加MindSpore Profiler相关接口。 + - `set_context`之后，需要初始化MindSpore `Profiler`对象，GPU场景下初始化Profiler对象时只有output_path参数有效。 - 在训练结束后，调用`Profiler.analyse`停止性能数据收集并生成性能分析结果。 @@ -49,7 +50,7 @@ class StopAtStep(Callback): self.start_step = start_step self.stop_step = stop_step self.already_analysed = False - + def step_begin(self, run_context): cb_params = run_context.original_args() step_num = cb_params.cur_step_num @@ -62,7 +63,7 @@ class StopAtStep(Callback): if step_num == self.stop_step and not self.already_analysed: self.profiler.analyse() self.already_analysed = True - + def end(self, run_context): if not self.already_analysed: self.profiler.analyse() @@ -74,7 +75,6 @@ class StopAtStep(Callback): 启动命令请参考[MindInsight相关命令](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/mindinsight_commands.html)。 - ### 性能分析 用户从训练列表中选择指定的训练，点击性能调试，可以查看该次训练的性能数据（目前GPU场景只支持算子耗时排名统计和Timeline功能，其他功能敬请期待）。 @@ -84,6 +84,7 @@ class StopAtStep(Callback): 图1：性能数据总览 图1展示了性能数据总览页面，包含了迭代轨迹（Step Trace）、算子性能、MindData性能和Timeline等组件的数据总体呈现： + - 算子性能：统计单算子以及各算子类型的执行时间，进行排序展示；总览页中展示了各算子类型平均执行时间占比的饼状图。 - Timeline：统计了算子以及CUDA activity，在时间轴排列展示；总览页展示了Timeline中执行情况汇总。 @@ -98,10 +99,12 @@ class StopAtStep(Callback): 图2：算子类别统计分析 图2展示了按算子类别进行统计分析的结果，包含以下内容： + - 可以选择饼图/柱状图展示各算子类别的时间占比，每个算子类别的执行时间会统计属于该类别的算子执行时间总和以及平均执行时间。 - 统计前20个平均执行时间最长的算子类别。 图2下半部分展示了算子性能统计表，包含以下内容： + - 选择全部：按单个算子的统计结果进行排序展示，展示维度包括算子位置（Device/Host）、算子类型、算子执行时间、算子全名等；默认按算子平均执行时间排序。 - 选择分类：按算子类别的统计结果进行排序展示，展示维度包括算子分类名称、算子类别执行时间、执行频次、执行总时间的比例、平均执行时间。点击每个算子类别，可以进一步查看该类别下所有单个算子的统计信息。 - 搜索：在右侧搜索框中输入字符串，支持对算子名称/类别进行模糊搜索。 @@ -120,7 +123,6 @@ class StopAtStep(Callback): GPU场景下，Timeline分析的使用方法和Ascend场景相同，不同之处是，GPU Timeline展示的是算子信息和CUDA activity的信息。使用方法参考： -> 样例代码与Ascend使用方式一致可以参考： +> 样例代码与Ascend使用方式一致可以参考： > > - diff --git a/tutorials/training/source_zh_cn/advanced_use/protect_user_privacy_with_differential_privacy.md b/tutorials/training/source_zh_cn/advanced_use/protect_user_privacy_with_differential_privacy.md index 827919e43ed88e6925f422ba908b3bde2ebb9e9e..d936ee4d962520ee2413ee20fcc9ab40c2d2092f 100644 --- a/tutorials/training/source_zh_cn/advanced_use/protect_user_privacy_with_differential_privacy.md +++ b/tutorials/training/source_zh_cn/advanced_use/protect_user_privacy_with_differential_privacy.md @@ -22,7 +22,7 @@ 差分隐私是一种保护用户数据隐私的机制。什么是隐私，隐私指的是单个用户的某些属性，一群用户的某一些属性可以不看做隐私。例如：“抽烟的人有更高的几率会得肺癌”，这个不泄露隐私，但是“张三抽烟，得了肺癌”，这个就泄露了张三的隐私。如果我们知道A医院，今天就诊的100个病人，其中有10个肺癌，并且我们知道了其中99个人的患病信息，就可以推测剩下一个人是否患有肺癌。这种窃取隐私的行为叫做差分攻击。差分隐私是防止差分攻击的方法，通过添加噪声，使得差别只有一条记录的两个数据集，通过模型推理获得相同结果的概率非常接近。也就是说，用了差分隐私后，攻击者知道的100个人的患病信息和99个人的患病信息几乎是一样的，从而无法推测出剩下1个人的患病情况。 -**机器学习中的差分隐私** +### 机器学习中的差分隐私 机器学习算法一般是用大量数据并更新模型参数，学习数据特征。在理想情况下，这些算法学习到一些泛化性较好的模型，例如“吸烟患者更容易得肺癌”，而不是特定的个体特征，例如“张三是个吸烟者，患有肺癌”。然而，机器学习算法并不会区分通用特征还是个体特征。当我们用机器学习来完成某个重要的任务，例如肺癌诊断，发布的机器学习模型，可能在无意中透露训练集中的个体特征，恶意攻击者可能从发布的模型获得关于张三的隐私信息，因此使用差分隐私技术来保护机器学习模型是十分必要的。 @@ -32,14 +32,14 @@ $Pr[\mathcal{K}(D)\in S] \le e^{\epsilon} Pr[\mathcal{K}(D') \in S]+\delta$ 对于两个差别只有一条记录的数据集$D, D'$，通过随机算法$\mathcal{K}$，输出为结果集合$S$子集的概率满足上面公式，$\epsilon$为差分隐私预算，$\delta$ 为扰动，$\epsilon, \delta$越小，$\mathcal{K}$在$D, D'$上输出的数据分布越接近。 -**差分隐私的度量** +### 差分隐私的度量 差分隐私可以用$\epsilon, \delta$ 度量。 - $\epsilon$：数据集中增加或者减少一条记录，引起的输出概率可以改变的上限。我们通常希望$\epsilon$是一个较小的常数，值越小表示差分隐私条件越严格。 - $\delta$：用于限制模型行为任意改变的概率，通常设置为一个小的常数，推荐设置小于训练数据集大小的倒数。 -**MindArmour实现的差分隐私** +### MindArmour实现的差分隐私 MindArmour的差分隐私模块Differential-Privacy，实现了差分隐私优化器。目前支持基于高斯机制的差分隐私SGD、Momentum、Adam优化器。其中，高斯噪声机制支持固定标准差的非自适应高斯噪声和随着时间或者迭代步数变化而变化的自适应高斯噪声，使用非自适应高斯噪声的优势在于可以严格控制差分隐私预算$\epsilon$，缺点是在模型训练过程中，每个Step添加的噪声量固定，在训练后期，较大的噪声使得模型收敛困难，甚至导致性能大幅下跌，模型可用性差。自适应噪声很好的解决了这个问题，在模型训练初期，添加的噪声量较大，随着模型逐渐收敛，噪声量逐渐减小，噪声对于模型可用性的影响减小。自适应噪声的缺点是不能严格控制差分隐私预算，在同样的初始值下，自适应差分隐私的$\epsilon$比非自适应的大。同时还提供RDP（R’enyi differential privacy）[2]用于监测差分隐私预算。 @@ -336,7 +336,8 @@ ds_train = generate_mnist_dataset(os.path.join(cfg.data_path, "train"), 5. 结果展示。 不加差分隐私的LeNet模型精度稳定在99%，加了Gaussian噪声，自适应Clip的差分隐私LeNet模型收敛，精度稳定在95%左右。 - ``` + + ```text ============== Starting Training ============== ... ============== Starting Testing ============== diff --git a/tutorials/training/source_zh_cn/advanced_use/save_load_model_hybrid_parallel.md b/tutorials/training/source_zh_cn/advanced_use/save_load_model_hybrid_parallel.md index b72b90fcf2949f4401ca5a5f9cd7126f1a081d41..40bca9703031ef6f25d470fff5602f5cb26655d3 100644 --- a/tutorials/training/source_zh_cn/advanced_use/save_load_model_hybrid_parallel.md +++ b/tutorials/training/source_zh_cn/advanced_use/save_load_model_hybrid_parallel.md @@ -72,9 +72,6 @@ MindSpore模型并行场景下，每个实例进程只保存有本节点对应 4. 执行阶段二训练。 - - - ## 对保存的CheckPoint文件做合并处理 ### 整体流程 @@ -85,18 +82,19 @@ MindSpore模型并行场景下，每个实例进程只保存有本节点对应 最后，将更新之后的参数列表，通过MindSpore提供的API保存到文件，生成新的CheckPoint文件。对应下图中的Step4。 -![img](./images/checkpoint_integration_process.jpg) +![img](./images/checkpoint_integration_process.jpg) ### 准备工作 #### 按逻辑顺序导入CheckPoint文件 定义网络，调用`load_checkpoint`、`load_param_into_net`接口，按逻辑顺序将CheckPoint文件导入网络，之后调用`parameters_and_names`接口获取网络里所有的参数数据。 -``` -net = Net() + +```python +net = Net() opt = Momentum(learning_rate=0.01, momentum=0.9, params=net.get_parameters()) net = TrainOneStepCell(net, opt) -param_dicts = [] +param_dicts = [] for i in range(rank_size): file_name = os.path.join("./node"+str(i), "CKP_1-4_32.ckpt") # checkpoint file name of current node param_dict = load_checkpoint(file_name) @@ -116,7 +114,8 @@ for i in range(rank_size): #### 获取模型参数切分策略 调用`build_searched_strategy`接口，得到模型各个参数的切分策略。 -``` + +```python strategy = build_searched_strategy("./strategy_train.cpkt") ``` @@ -130,45 +129,48 @@ strategy = build_searched_strategy("./strategy_train.cpkt") 参数名称为"model_parallel_weight"，切分逻辑为4卡场景。 -1. 针对涉及模型并行的参数，获取所有节点上的参数数据。 +1. 针对涉及模型并行的参数，获取所有节点上的参数数据。 - ``` + ```python sliced_parameters = [] for i in range(4): parameter = param_dicts[i].get("model_parallel_weight") sliced_parameters.append(parameter) ``` + > 如果要保证参数更新速度不变，需要对优化器中保存的参数，如“moments.model_parallel_weight”，同样做合并处理。 2. 调用`merge_sliced_parameter`接口进行参数合并。 - ``` - merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) + ```python + merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) ``` > 如果存在多个模型并行的参数，则需要重复步骤1到步骤2循环逐个处理。 ### 保存数据生成新的CheckPoint文件 -1. 将`param_dict`转换为list类型数据。 +1. 将`param_dict`转换为list类型数据。 - ``` + ```python param_list = [] for (key, value) in param_dict.items(): - each_param = {} - each_param["name"] = key - if isinstance(value.data, Tensor): - param_data = value.data - else: - param_data = Tensor(value.data) - each_param["data"] = param_data + each_param = {} + each_param["name"] = key + if isinstance(value.data, Tensor): + param_data = value.data + else: + param_data = Tensor(value.data) + each_param["data"] = param_data param_list.append(each_param） ``` 2. 调用`save_checkpoint`接口，将参数数据写入文件，生成新的CheckPoint文件。 - ``` + + ```python save_checkpoint(param_list, “./CKP-Integrated_1-4_32.ckpt”) ``` + 其中， - `save_checkpoint`： 通过该接口将网络模型参数信息存入文件。 - `CKP-Integrated_1-4_32.ckpt`： 新生成的CheckPoint模型参数文件名称。 @@ -185,7 +187,7 @@ strategy = build_searched_strategy("./strategy_train.cpkt") 调用`load_checkpoint`接口，从CheckPoint文件中加载模型参数数据。 -``` +```python param_dict = load_checkpoint("./CKP-Integrated_1-4_32.ckpt") ``` @@ -204,7 +206,8 @@ param_dict = load_checkpoint("./CKP-Integrated_1-4_32.ckpt") 1. 对模型参数数据做切分。 如下代码示例，在维度0上，将数据切分为两个切片。 - ``` + + ```python new_param = parameter_dict[“model_parallel_weight”] slice_list = np.split(new_param.data.asnumpy(), 2, axis=0) new_param_moments = parameter_dict[“moments.model_parallel_weight”] @@ -213,24 +216,28 @@ param_dict = load_checkpoint("./CKP-Integrated_1-4_32.ckpt") 切分后的数据情况： - slice_list[0] --- [1, 2, 3, 4] 对应device0 - slice_list[1] --- [5, 6, 7, 8] 对应device1 + ```text + slice_list[0] --- [1, 2, 3, 4] 对应device0 + slice_list[1] --- [5, 6, 7, 8] 对应device1 + ``` 与`slice_list`类似，`slice_moments_list` 也被切分为两个shape为[1, 4]的Tensor。 -2. 在每个节点分别加载对应的数据切片。 +2. 在每个节点分别加载对应的数据切片。 获取本节点的rank_id，根据rank_id加载数据。 - ``` + + ```python rank = get_rank() tensor_slice = Tensor(slice_list[rank]) tensor_slice_moments = Tensor(slice_moments_list[rank]) ``` - - `get_rank`：获取当前设备在集群中的ID。 -3. 修改模型参数数据值。 + - `get_rank`：获取当前设备在集群中的ID。 - ``` +3. 修改模型参数数据值。 + + ```python new_param.set_data(tensor_slice, True) new_param_moments.set_data(tensor_slice_moments, True) ``` @@ -240,8 +247,9 @@ param_dict = load_checkpoint("./CKP-Integrated_1-4_32.ckpt") ### 步骤3：将修改后的参数数据加载到网络中 调用`load_param_into_net`接口，将模型参数数据加载到网络中。 -``` -net = Net() + +```python +net = Net() opt = Momentum(learning_rate=0.01, momentum=0.9, params=parallel_net.get_parameters()) load_param_into_net(net, param_dict) load_param_into_net(opt, param_dict) @@ -266,43 +274,44 @@ load_param_into_net(opt, param_dict) > > 本文档附上对CheckPoint文件做合并处理以及分布式训练前加载CheckPoint文件的示例代码，仅作为参考，实际请参考具体情况实现。 -### 示例代码 +### 示例代码 1. 执行脚本对CheckPoint文件做合并处理。 - 脚本执行命令： - ``` + 脚本执行命令： + + ```bash python ./integrate_checkpoint.py "待合并的CheckPoint文件名称" "合并生成的CheckPoint文件路径&名称" "策略文件路径&名称" "节点数" ``` integrate_checkpoint.py： - ``` + ```python import numpy as np import os import mindspore.nn as nn from mindspore import Tensor, Parameter from mindspore.ops import operations as P from mindspore.train.serialization import save_checkpoint, load_checkpoint, build_searched_strategy, merge_sliced_parameter - + class Net(nn.Cell): def __init__(self,weight_init): super(Net, self).__init__() self.weight = Parameter(Tensor(weight_init), "model_parallel_weight", layerwise_parallel=True) self.fc = P.MatMul(transpose_b=True) - + def construct(self, x): x = self.fc(x, self.weight1) return x - + def integrate_ckpt_file(old_ckpt_file, new_ckpt_file, strategy_file, rank_size): weight = np.ones([2, 8]).astype(np.float32) net = Net(weight) opt = Momentum(learning_rate=0.01, momentum=0.9, params=net.get_parameters()) net = TrainOneStepCell(net, opt) - + # load CheckPoint into net in rank id order - param_dicts = [] + param_dicts = [] for i in range(rank_size): file_name = os.path.join("./node"+str(i), old_ckpt_file) param_dict = load_checkpoint(file_name) @@ -311,21 +320,21 @@ load_param_into_net(opt, param_dict) for _, param in net.parameters_and_names(): param_dict[param.name] = param param_dicts.append(param_dict) - + strategy = build_searched_strategy(strategy_file) param_dict = {} - + for paramname in ["model_parallel_weight", "moments.model_parallel_weight"]: # get layer wise model parallel parameter sliced_parameters = [] for i in range(rank_size): parameter = param_dicts[i].get(paramname) sliced_parameters.append(parameter) - + # merge the parallel parameters of the model - merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) + merged_parameter = merge_sliced_parameter(sliced_parameters, strategy) param_dict[paramname] = merged_parameter - + # convert param_dict to list type data param_list = [] for (key, value) in param_dict.items(): @@ -335,14 +344,14 @@ load_param_into_net(opt, param_dict) param_data = value.data else: param_data = Tensor(value.data) - each_param["data"] = param_data - param_list.append(each_param) - + each_param["data"] = param_data + param_list.append(each_param) + # call the API to generate a new CheckPoint file save_checkpoint(param_list, new_ckpt_file) - + return - + if __name__ == "__main__": try: old_ckpt_file = sys.argv[1] @@ -354,14 +363,15 @@ load_param_into_net(opt, param_dict) print("Fail to integrate checkpoint file) sys.exit(-1) ``` - + 执行结果： 脚本执行前，CheckPoint文件中参数值： - ``` + + ```text device0： name is model_parallel_weight - value is + value is [[0.87537426 1.0448935 0.86736983 0.8836905 0.77354026 0.69588304 0.9183654 0.7792076] [0.87224025 0.8726848 0.771446 0.81967723 0.88974726 0.7988162 0.72919345 0.7677011]] name is learning_rate @@ -372,10 +382,10 @@ load_param_into_net(opt, param_dict) value is [[0.2567724 -0.07485991 0.282002 0.2456022 0.454939 0.619168 0.18964815 0.45714882] [0.25946522 0.24344791 0.45677605 0.3611395 0.23378398 0.41439137 0.5312468 0.4696194]] - + device1： name is model_parallel_weight - value is + value is [[0.9210751 0.9050457 0.9827775 0.920396 0.9240526 0.9750359 1.0275179 1.0819869] [0.73605865 0.84631145 0.9746683 0.9386582 0.82902765 0.83565056 0.9702136 1.0514659]] name is learning_rate @@ -385,11 +395,11 @@ load_param_into_net(opt, param_dict) name is moments.model_weight value is [[0.2417504 0.28193963 0.06713893 0.21510397 0.23380603 0.11424308 0.0218009 -0.11969765] - [0.45955992 0.22664294 0.01990281 0.0731914 0.27125207 0.27298513 -0.01716102 -0.15327111]] - + [0.45955992 0.22664294 0.01990281 0.0731914 0.27125207 0.27298513 -0.01716102 -0.15327111]] + device2： name is model_parallel_weight - value is + value is [[1.0108461 0.8689414 0.91719437 0.8805056 0.7994629 0.8999671 0.7585804 1.0287056 ] [0.90653455 0.60146594 0.7206475 0.8306303 0.8364681 0.89625114 0.7354735 0.8447268]] name is learning_rate @@ -397,10 +407,10 @@ load_param_into_net(opt, param_dict) name is momentum value is [0.9] name is moments.model_weight - value is + value is [[0.03440702 0.41419312 0.24817684 0.30765256 0.48516113 0.24904746 0.57791173 0.00955463] [0.13458519 0.6690533 0.49259356 0.28319967 0.25951773 0.16777472 0.45696738 0.24933104]] - + device3： name is model_parallel_weight value is @@ -411,16 +421,16 @@ load_param_into_net(opt, param_dict) name is momentum value is [0.9] name is moments.model_parallel_weight - value is + value is [[0.14152306 0.5040985 0.24455397 0.10907605 0.11319532 0.19538902 0.01208619 0.40430856] [-0.7773164 -0.47611716 -0.6041424 -0.6144473 -0.2651842 -0.31909415 -0.4510405 -0.12860501]] ``` 脚本执行后，CheckPoint文件中参数值： - ``` + ```text name is model_parallel_weight - value is + value is [[1.1138763 1.0962057 1.3516843 1.0812817 1.1579804 1.1078343 1.0906502 1.3207073] [0.916671 1.0781671 1.0368758 0.9680898 1.1735439 1.0628364 0.9960786 1.0135143] [0.8828271 0.7963984 0.90675324 0.9830291 0.89010954 0.897052 0.7890109 0.89784735] @@ -434,7 +444,7 @@ load_param_into_net(opt, param_dict) name is momentum value is [0.9] name is moments.model_parallel_weight - value is + value is [[0.2567724 -0.07485991 0.282002 0.2456022 0.454939 0.619168 0.18964815 0.45714882] [0.25946522 0.24344791 0.45677605 0.3611395 0.23378398 0.41439137 0.5312468 0.4696194 ] [0.2417504 0.28193963 0.06713893 0.21510397 0.23380603 0.11424308 0.0218009 -0.11969765] @@ -446,10 +456,9 @@ load_param_into_net(opt, param_dict) -0.12860501]] ``` - 2. 执行阶段2训练，训练前加载CheckPoint文件。其中训练代码部分，需要根据实际情况补充。 - ``` + ```python import numpy as np import os import mindspore.nn as nn @@ -458,24 +467,24 @@ load_param_into_net(opt, param_dict) from mindspore import Tensor, Parameter from mindspore.ops import operations as P from mindspore.train.serialization import load_checkpoint, load_param_into_net - + from mindspore.communication.management import init devid = int(os.getenv('DEVICE_ID')) context.set_context(mode=context.GRAPH_MODE,device_target='Ascend',save_graphs=True, device_id=devid) init() - + class Net(nn.Cell): def __init__(self,weight_init): super(Net, self).__init__() self.weight = Parameter(Tensor(weight_init), "model_parallel_weight", layerwise_parallel=True) self.fc = P.MatMul(transpose_b=True) - + def construct(self, x): x = self.fc(x, self.weight1) return x def train_mindspore_impl_fc(input, label, ckpt_file): param_dict = load_checkpoint(ckpt_file) - + for paramname in ["model_parallel_weight", "moments.model_parallel_weight"]: # get layer wise model parallel parameter new_param = parameter_dict[paramname] @@ -486,23 +495,23 @@ load_param_into_net(opt, param_dict) tensor_slice = Tensor(slice_list[rank]) # modify model parameter data values new_param.set_data(tensor_slice, True) - + # load the modified parameter data into the network weight = np.ones([4, 8]).astype(np.float32) net = Net(weight) load_param_into_net(net, param_dict) opt = Momentum(learning_rate=0.01, momentum=0.9, params=parallel_net.get_parameters()) load_param_into_net(opt, param_dict) - # train code + # train code ... - + if __name__ == "__main__": input = np.random.random((4, 8)).astype(np.float32) print("mean = ", np.mean(input,axis=1, keepdims=True)) label = np.random.random((4, 4)).astype(np.float32) train_mindspore_impl_fc(input, label, weight1) ``` - + 其中， - `mode=context.GRAPH_MODE`：使用分布式训练需要指定运行模式为图模式（PyNative模式不支持并行）。 @@ -511,10 +520,10 @@ load_param_into_net(opt, param_dict) 加载后的参数值： - ``` + ```text device0： name is model_parallel_weight - value is + value is [[0.87537426 1.0448935 0.86736983 0.8836905 0.77354026 0.69588304 0.9183654 0.7792076] [0.87224025 0.8726848 0.771446 0.81967723 0.88974726 0.7988162 0.72919345 0.7677011] [0.8828271 0.7963984 0.90675324 0.9830291 0.89010954 0.897052 0.7890109 0.89784735] @@ -532,7 +541,7 @@ load_param_into_net(opt, param_dict) device1： name is model_parallel_weight - value is + value is [[1.0053468 0.98402303 0.99762845 0.97587246 1.0259694 1.0055295 0.99420834 0.9496847] [1.0851002 1.0295962 1.0999886 1.0958165 0.9765328 1.146529 1.0970603 1.1388365] [0.7147005 0.9168278 0.80178416 0.6258351 0.8413766 0.5909515 0.696347 0.71359116] @@ -546,5 +555,5 @@ load_param_into_net(opt, param_dict) [[0.03440702 0.41419312 0.24817684 0.30765256 0.48516113 0.24904746 0.57791173 0.00955463] [0.13458519 0.6690533 0.49259356 0.28319967 0.25951773 0.16777472 0.45696738 0.24933104] [0.14152306 0.5040985 0.24455397 0.10907605 0.11319532 0.19538902 0.01208619 0.40430856] - [-0.7773164 -0.47611716 -0.6041424 -0.6144473 -0.2651842 -0.31909415 -0.4510405 -0.12860501]] + [-0.7773164 -0.47611716 -0.6041424 -0.6144473 -0.2651842 -0.31909415 -0.4510405 -0.12860501]] ``` diff --git a/tutorials/training/source_zh_cn/advanced_use/summary_record.md b/tutorials/training/source_zh_cn/advanced_use/summary_record.md index 40d9a9fde98731b0c709c73e744684137d12afec..dd3da7678f7e6f154d2127ae436d02a5a6937991 100644 --- a/tutorials/training/source_zh_cn/advanced_use/summary_record.md +++ b/tutorials/training/source_zh_cn/advanced_use/summary_record.md @@ -17,7 +17,7 @@    - + ## 概述 @@ -43,6 +43,7 @@ MindSpore目前支持三种方式将数据记录到summary日志文件中。 即可自动收集一些常见信息。`SummaryCollector` 详细的用法可以参考 `API` 文档中 `mindspore.train.callback.SummaryCollector`。 样例代码如下： + ```python import mindspore import mindspore.nn as nn @@ -131,6 +132,7 @@ model.eval(ds_eval, callbacks=[summary_collector]) MindSpore除了提供 `SummaryCollector` 能够自动收集一些常见数据，还提供了Summary算子，支持在网络中自定义收集其他的数据，比如每一个卷积层的输入，或在损失函数中的损失值等。 当前支持的Summary算子: + - [ScalarSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ScalarSummary)：记录标量数据 - [TensorSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.TensorSummary)：记录张量数据 - [ImageSummary](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.ops.html#mindspore.ops.ImageSummary)：记录图片数据 @@ -254,18 +256,18 @@ MindSpore支持自定义Callback, 并允许在自定义Callback中将数据记 样例代码如下： -``` +```python from mindspore.train.callback import Callback from mindspore.train.summary import SummaryRecord class ConfusionMatrixCallback(Callback): def __init__(self, summary_dir): self._summary_dir = summary_dir - + def __enter__(self): # init you summary record in here, when the train script run, it will be inited before training self.summary_record = SummaryRecord(summary_dir) - + def __exit__(self, *exc_args): # Note: you must close the summary record, it will release the process pool resource # else your training script will not exit from training. @@ -276,7 +278,7 @@ class ConfusionMatrixCallback(Callback): cb_params = run_context.run_context.original_args() # create a confusion matric image, and record it to summary file - confusion_martrix = create_confusion_matrix(cb_params) + confusion_martrix = create_confusion_matrix(cb_params) self.summary_record.add_value('image', 'confusion_matrix', confusion_matric) self.summary_record.record(cb_params.cur_step) @@ -288,31 +290,34 @@ model.train(cnn_network, train_dataset=train_ds, callbacks=[confusion_martrix]) ``` 上面的三种方式，支持记录计算图, 损失值等多种数据。除此以外，MindSpore还支持保存训练中其他阶段的计算图，通过 -将训练脚本中 `context.set_context` 的 `save_graphs` 选项设置为 `True`, 可以记录其他阶段的计算图，其中包括算子融合后的计算图。 +将训练脚本中 `context.set_context` 的 `save_graphs` 选项设置为 `True`, 可以记录其他阶段的计算图，其中包括算子融合后的计算图。 在保存的文件中，`ms_output_after_hwopt.pb` 即为算子融合后的计算图，可以使用可视化页面对其进行查看。 ## 运行MindInsight + 按照上面教程完成数据收集后，启动MindInsight，即可可视化收集到的数据。启动MindInsight时， 需要通过 `--summary-base-dir` 参数指定summary日志文件目录。 其中指定的summary日志文件目录可以是一次训练的输出目录，也可以是多次训练输出目录的父目录。 - 一次训练的输出目录结构如下： -``` + +```text └─summary_dir events.out.events.summary.1596869898.hostname_MS events.out.events.summary.1596869898.hostname_lineage ``` 启动命令： + ```Bash mindinsight start --summary-base-dir ./summary_dir ``` 多次训练的输出目录结构如下： -``` + +```text └─summary ├─summary_dir1 │ events.out.events.summary.1596869898.hostname_MS @@ -324,6 +329,7 @@ mindinsight start --summary-base-dir ./summary_dir ``` 启动命令: + ```Bash mindinsight start --summary-base-dir ./summary ``` @@ -331,13 +337,13 @@ mindinsight start --summary-base-dir ./summary 启动成功后，通过浏览器访问 `http://127.0.0.1:8080` 地址，即可查看可视化页面。 停止MindInsight命令： + ```Bash mindinsight stop ``` 更多参数设置，请点击查看[MindInsight相关命令](https://www.mindspore.cn/tutorial/training/zh-CN/master/advanced_use/mindinsight_commands.html)页面。 - ## 注意事项 1. 为了控制列出summary文件目录的用时，MindInsight最多支持发现999个summary文件目录。 @@ -349,7 +355,8 @@ mindinsight stop 自定义callback中如果使用 `SummaryRecord`，则其不能和 `SummaryCollector` 同时使用。 正确代码: - ``` + + ```python ... summary_collector = SummaryCollector('./summary_dir') model.train(2, train_dataset, callbacks=[summary_collector]) @@ -359,7 +366,8 @@ mindinsight stop ``` 错误代码： - ``` + + ```python ... summary_collector1 = SummaryCollector('./summary_dir1') summary_collector2 = SummaryCollector('./summary_dir2') @@ -367,7 +375,8 @@ mindinsight stop ``` 错误代码： - ``` + + ```python ... # Note: the 'ConfusionMatrixCallback' is user-defined, and it uses SummaryRecord to record data. confusion_callback = ConfusionMatrixCallback('./summary_dir1') @@ -377,4 +386,4 @@ mindinsight stop 3. 每个summary日志文件目录中，应该只放置一次训练的数据。一个summary日志目录中如果存放了多次训练的summary数据，MindInsight在可视化数据时会将这些训练的summary数据进行叠加展示，可能会与预期可视化效果不相符。 -4. 当前 `SummaryCollector` 和 `SummaryRecord` 不支持GPU多卡运行的场景。 \ No newline at end of file +4. 当前 `SummaryCollector` 和 `SummaryRecord` 不支持GPU多卡运行的场景。 diff --git a/tutorials/training/source_zh_cn/advanced_use/test_model_security_fuzzing.md b/tutorials/training/source_zh_cn/advanced_use/test_model_security_fuzzing.md index dde3f397c448a934218d4e57f322fb574b96898d..148399f8fda9f614f9499852ceb35b0ae870d884 100644 --- a/tutorials/training/source_zh_cn/advanced_use/test_model_security_fuzzing.md +++ b/tutorials/training/source_zh_cn/advanced_use/test_model_security_fuzzing.md @@ -10,7 +10,7 @@ - [导入需要的库文件](#导入需要的库文件) - [参数配置](#参数配置) - [运用Fuzz Testing](#运用fuzz-testing) - +    @@ -75,7 +75,7 @@ context.set_context(mode=context.GRAPH_MODE, device_target="Ascend") images = data[0].asnumpy().astype(np.float32) train_images.append(images) train_images = np.concatenate(train_images, axis=0) - + # get test data data_list = "../common/dataset/MNIST/test" batch_size = 32 @@ -105,7 +105,7 @@ context.set_context(mode=context.GRAPH_MODE, device_target="Ascend") 中对应的类方法。算法随机选择参数，则`params`设置为`'auto_param': [True]`，参数将在推荐范围内随机生成。 基于对抗攻击方法的参数配置请参考对应的攻击方法类。 - + 下面时变异方法及其参数配置的一个例子： ```python @@ -174,12 +174,12 @@ context.set_context(mode=context.GRAPH_MODE, device_target="Ascend") ``` 6. 实验结果。 - + fuzzing的返回结果中包含了5个数据：fuzz生成的样本fuzz_samples、生成样本的真实标签true_labels、被测模型对于生成样本的预测值fuzz_preds、 生成样本使用的变异方法fuzz_strategies、fuzz testing的评估报告metrics_report。用户可使用这些返回结果进一步的分析模型的鲁棒性。这里只展开metrics_report，查看fuzz testing后的各个评估指标。 ```python if metrics: - for key in metrics: + for key in metrics: LOGGER.info(TAG, key + ': %s', metrics[key]) ``` @@ -199,4 +199,4 @@ context.set_context(mode=context.GRAPH_MODE, device_target="Ascend") ​ Fuzz生成的变异图片： - ![fuzz_res](./images/fuzz_res.png) \ No newline at end of file + ![fuzz_res](./images/fuzz_res.png) diff --git a/tutorials/training/source_zh_cn/advanced_use/test_model_security_membership_inference.md b/tutorials/training/source_zh_cn/advanced_use/test_model_security_membership_inference.md index a5057865f58984a0b5665591d4195d6846e36bf5..559e1faf214329efc226a82f76906650d7a70243 100644 --- a/tutorials/training/source_zh_cn/advanced_use/test_model_security_membership_inference.md +++ b/tutorials/training/source_zh_cn/advanced_use/test_model_security_membership_inference.md @@ -10,7 +10,6 @@ - [建立模型](#建立模型) - [运用MembershipInference进行隐私安全评估](#运用membershipinference进行隐私安全评估) - [参考文献](#参考文献) -    @@ -30,7 +29,9 @@ ## 实现阶段 ### 导入需要的库文件 + #### 引入相关包 + 下面是我们需要的公共模块、MindSpore相关模块和MembershipInference特性模块，以及配置日志标签和日志等级。 ```python @@ -57,9 +58,11 @@ LOGGER = LogUtil.get_instance() TAG = "MembershipInference_test" LOGGER.set_level("INFO") ``` + ### 加载数据集 这里采用的是CIFAR-100数据集，您也可以采用自己的数据集，但要保证传入的数据仅有两项属性"image"和"label"。 + ```python # Generate CIFAR-100 data. def vgg_create_dataset100(data_home, image_size, batch_size, rank_id=0, rank_size=1, repeat_num=1, @@ -111,9 +114,11 @@ def vgg_create_dataset100(data_home, image_size, batch_size, rank_id=0, rank_siz return data_set ``` + ### 建立模型 这里以VGG16模型为例，您也可以替换为自己的模型。 + ```python def _make_layer(base, args, batch_norm): """Make stage network of VGG.""" @@ -178,10 +183,11 @@ def vgg16(num_classes=1000, args=None, phase="train"): ``` ### 运用MembershipInference进行隐私安全评估 + 1. 构建VGG16模型并加载参数文件。 - + 这里直接加载预训练完成的VGG16参数配置，您也可以使用如上的网络自行训练。 - + ```python ... # load parameter @@ -195,8 +201,8 @@ def vgg16(num_classes=1000, args=None, phase="train"): args.padding = 0 args.pad_mode = "same" args.weight_decay = 5e-4 - args.loss_scale = 1.0 - + args.loss_scale = 1.0 + # Load the pretrained model. net = vgg16(num_classes=100, args=args) loss = nn.SoftmaxCrossEntropyWithLogits(sparse=True) @@ -205,7 +211,7 @@ def vgg16(num_classes=1000, args=None, phase="train"): load_param_into_net(net, load_checkpoint(args.pre_trained)) model = Model(network=net, loss_fn=loss, optimizer=opt) ``` - + 2. 加载CIFAR-100数据集，按8:2分割为成员推理模型的训练集和测试集。 ```python @@ -221,9 +227,9 @@ def vgg16(num_classes=1000, args=None, phase="train"): ``` 3. 配置推理参数和评估参数 - + 设置用于成员推理的方法和参数。目前支持的推理方法有：KNN、LR、MLPClassifier和RandomForestClassifier。推理参数数据类型使用list，各个方法使用key为"method"和"params"的字典表示。 - + ```python config = [ { @@ -232,7 +238,7 @@ def vgg16(num_classes=1000, args=None, phase="train"): "C": np.logspace(-4, 2, 10) } }, - { + { "method": "knn", "params": { "n_neighbors": [3, 5, 7] @@ -258,13 +264,13 @@ def vgg16(num_classes=1000, args=None, phase="train"): } ] ``` - + 我们约定标签为数据集的是正类，标签为测试集的是负类。设置评价指标，目前支持3种评价指标。包括： - * 准确率：accuracy，正确推理的数量占全体样本中的比例。 - * 精确率：precision，正确推理的正类样本占所有推理为正类中的比例。 - * 召回率：recall，正确推理的正类样本占全体正类样本的比例。 + - 准确率：accuracy，正确推理的数量占全体样本中的比例。 + - 精确率：precision，正确推理的正类样本占所有推理为正类中的比例。 + - 召回率：recall，正确推理的正类样本占全体正类样本的比例。 在样本数量足够大时，如果上述指标均大于0.6，我们认为目标模型就存在隐私泄露的风险。 - + ```python metrics = ["precision", "accuracy", "recall"] ``` @@ -273,11 +279,11 @@ def vgg16(num_classes=1000, args=None, phase="train"): ```python inference = MembershipInference(model) # Get inference model. - + inference.train(train_train, train_test, config) # Train inference model. msg = "Membership inference model training completed." LOGGER.info(TAG, msg) - + result = inference.eval(eval_train, eval_test, metrics) # Eval metrics. count = len(config) for i in range(count): @@ -286,16 +292,16 @@ def vgg16(num_classes=1000, args=None, phase="train"): 5. 实验结果。 执行如下指令,开始成员推理训练和评估： - - ``` + + ```bash python example_vgg_cifar.py --data_path ./cifar-100-binary/ --pre_trained ./VGG16-100_781.ckpt ``` 成员推理的指标如下所示，各数值均保留至小数点后四位。 以第一行结果为例：在使用lr（逻辑回归分类）进行成员推理时，推理的准确率（accuracy）为0.7132，推理精确率（precision）为0.6596，正类样本召回率为0.8810，说明lr有71.32%的概率能正确分辨一个数据样本是否属于目标模型的训练数据集。在二分类任务下，指标表明成员推理是有效的，即该模型存在隐私泄露的风险。 - - ``` + + ```text Method: lr, {'recall': 0.8810,'precision': 0.6596,'accuracy': 0.7132} Method: knn, {'recall': 0.7082,'precision': 0.5613,'accuracy': 0.5774} Method: mlp, {'recall': 0.6729,'precision': 0.6462,'accuracy': 0.6522} @@ -303,4 +309,5 @@ def vgg16(num_classes=1000, args=None, phase="train"): ``` ## 参考文献 + [1] [Shokri R , Stronati M , Song C , et al. Membership Inference Attacks against Machine Learning Models[J].](https://arxiv.org/abs/1610.05820v2) diff --git a/tutorials/training/source_zh_cn/advanced_use/use_on_the_cloud.md b/tutorials/training/source_zh_cn/advanced_use/use_on_the_cloud.md index 47680705b2a44f4f25d103ab25612414410df8e7..bf3dfd79ce8be8343042b7b90500edf752e212df 100644 --- a/tutorials/training/source_zh_cn/advanced_use/use_on_the_cloud.md +++ b/tutorials/training/source_zh_cn/advanced_use/use_on_the_cloud.md @@ -53,7 +53,7 @@ ModelArts使用对象存储服务（Object Storage Service，简称OBS）进行 2. 新建一个自己的OBS桶（例如：ms-dataset），在桶中创建数据目录（例如：cifar-10），将CIFAR-10数据按照如下结构上传至数据目录。 - ``` + ```text └─对象存储/ms-dataset/cifar-10 ├─train │ data_batch_1.bin @@ -73,7 +73,7 @@ ModelArts使用对象存储服务（Object Storage Service，简称OBS）进行 为了方便后续创建训练作业，先创建训练输出目录和日志输出目录，本示例创建的目录结构如下： -``` +```text └─对象存储/resnet50-train ├─resnet50_cifar10_train │ dataset.py @@ -87,7 +87,7 @@ ModelArts使用对象存储服务（Object Storage Service，简称OBS）进行 “执行脚本准备”章节提供的脚本可以直接运行在ModelArts，想要快速体验ResNet-50训练CIFAR-10可以跳过本章节。如果需要将自定义MindSpore脚本或更多MindSpore示例代码在ModelArts运行起来，需要参考本章节对MindSpore代码进行简单适配。 -### 适配脚本参数 +### 适配脚本参数 1. 在ModelArts运行的脚本必须配置`data_url`和`train_url`，分别对应数据存储路径(OBS路径)和训练输出路径(OBS路径)。 @@ -125,13 +125,14 @@ MindSpore暂时没有提供直接访问OBS数据的接口，需要通过MoXing ``` ### 适配8卡训练任务 + 如果需要将脚本运行在`8*Ascend`规格的环境上，需要对创建数据集的代码和本地数据路径进行适配，并配置分布式策略。通过获取`DEVICE_ID`和`RANK_SIZE`两个环境变量，用户可以构建适用于`1*Ascend`和`8*Ascend`两种不同规格的训练脚本。 1. 本地路径适配。 ```python import os - + device_num = int(os.getenv('RANK_SIZE')) device_id = int(os.getenv('DEVICE_ID')) # define local data path @@ -311,7 +312,6 @@ ModelArts教程 - ## 概述 回归问题算法通常是利用一系列属性来预测一个值，预测的值是连续的。例如给出一套房子的一些特征数据，如面积、卧室数等等来预测房价，利用最近一周的气温变化和卫星云图来预测未来的气温情况等。如果一套房子实际价格为500万元，通过回归分析的预测值为499万元，则认为这是一个比较好的回归分析。在机器学习问题中，常见的回归分析有线性回归、多项式回归、逻辑回归等。本例子介绍线性回归算法，并通过MindSpore进行线性回归AI训练体验。 @@ -47,7 +46,6 @@ 设置MindSpore运行配置 - ```python from mindspore import context @@ -66,7 +64,6 @@ context.set_context(mode=context.GRAPH_MODE, device_target="CPU") `get_data`用于生成训练数据集和测试数据集。由于拟合的是线性数据，假定要拟合的目标函数为：$f(x)=2x+3$，那么我们需要的训练数据集应随机分布于函数周边，这里采用了$f(x)=2x+3+noise$的方式生成，其中`noise`为遵循标准正态分布规律的随机数值。 - ```python import numpy as np @@ -80,7 +77,6 @@ def get_data(num, w=2.0, b=3.0): 使用`get_data`生成50组测试数据，可视化展示。 - ```python import matplotlib.pyplot as plt @@ -97,10 +93,8 @@ plt.show() 输出结果： - ![png](./images/linear_regression_eval_datasets.png) - 上图中绿色线条部分为目标函数，红点部分为验证数据`eval_data`。 ### 定义数据增强函数 @@ -111,7 +105,6 @@ plt.show() - `batch`：将`batch_size`个数据组合成一个batch。 - `repeat`：将数据集数量倍增。 - ```python from mindspore import dataset as ds @@ -124,13 +117,12 @@ def create_dataset(num_data, batch_size=16, repeat_size=1): 使用数据集增强函数生成训练数据，并查看训练数据的格式。 - ```python num_data = 1600 batch_size = 16 repeat_size = 1 -ds_train = create_dataset(num_data, batch_size=batch_size, repeat_size=repeat_size) +ds_train = create_dataset(num_data, batch_size=batch_size, repeat_size=repeat_size) print("The dataset size of ds_train:", ds_train.get_dataset_size()) dict_datasets = ds_train.create_dict_iterator().get_next() @@ -141,11 +133,12 @@ print("The y label value shape:", dict_datasets["label"].shape) 输出结果： - The dataset size of ds_train: 100 - dict_keys(['data', 'label']) - The x label value shape: (16, 1) - The y label value shape: (16, 1) - +```text +The dataset size of ds_train: 100 +dict_keys(['data', 'label']) +The x label value shape: (16, 1) +The y label value shape: (16, 1) +``` 通过定义的`create_dataset`将生成的1600个数据增强为了100组shape为16x1的数据集。 @@ -157,7 +150,6 @@ $$f(x)=wx+b\tag{1}$$ 并使用Normal算子随机初始化权重$w$和$b$。 - ```python from mindspore.common.initializer import Normal from mindspore import nn @@ -174,7 +166,6 @@ class LinearNet(nn.Cell): 调用网络查看初始化的模型参数。 - ```python net = LinearNet() model_params = net.trainable_params() @@ -183,18 +174,18 @@ print(model_params) 输出结果： - [Parameter (name=fc.weight, value=Tensor(shape=[1, 1], dtype=Float32, - [[-7.35660456e-003]])), Parameter (name=fc.bias, value=Tensor(shape=[1], dtype=Float32, [-7.35660456e-003]))] - +```text +[Parameter (name=fc.weight, value=Tensor(shape=[1, 1], dtype=Float32, +[[-7.35660456e-003]])), Parameter (name=fc.bias, value=Tensor(shape=[1], dtype=Float32, [-7.35660456e-003]))] +``` 初始化网络模型后，接下来将初始化的网络函数和训练数据集进行可视化，了解拟合前的模型函数情况。 - ```python from mindspore import Tensor x_model_label = np.array([-10, 10, 0.1]) -y_model_label = (x_model_label * Tensor(model_params[0]).asnumpy()[0][0] + +y_model_label = (x_model_label * Tensor(model_params[0]).asnumpy()[0][0] + Tensor(model_params[1]).asnumpy()[0]) plt.scatter(x_eval_label, y_eval_label, color="red", s=5) @@ -205,10 +196,8 @@ plt.show() 输出结果： - ![png](./images/model_net_and_eval_datasets.png) - 从上图中可以看出，蓝色线条的初始化模型函数与绿色线条的目标函数还是有较大的差别的。 ## 定义前向传播网络与反向传播网络并关联 @@ -236,7 +225,6 @@ $$J(w)=\frac{1}{2m}\sum_{i=1}^m(h(x_i)-y^{(i)})^2\tag{2}$$ 在MindSpore中使用如下方式实现。 - ```python net = LinearNet() net_loss = nn.loss.MSELoss() @@ -257,7 +245,6 @@ $$w_{t}=w_{t-1}-\alpha\frac{\partial{J(w_{t-1})}}{\partial{w}}\tag{3}$$ 函数中所有的权重值更新完成后，将值传入到模型函数中，这个过程就是反向传播过程，实现此过程需要使用MindSpore中的优化器函数，如下： - ```python opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) ``` @@ -266,7 +253,6 @@ opt = nn.Momentum(net.trainable_params(), learning_rate=0.005, momentum=0.9) 定义完成前向传播和反向传播后，在MindSpore中需要调用`Model`函数，将前面定义的网络，损失函数，优化器函数关联起来，使之变成完整的计算网络。 - ```python from mindspore.train import Model @@ -279,7 +265,6 @@ model = Model(net, net_loss, opt) 为了使得整个训练过程更容易理解，需要将训练过程的测试数据、目标函数和模型网络进行可视化，这里定义了可视化函数，将在每个step训练结束后调用，展示模型网络的拟合过程。 - ```python import matplotlib.pyplot as plt import time @@ -292,7 +277,7 @@ def plot_model_and_datasets(net, eval_data): x1, y1 = zip(*eval_data) x_target = x y_target = x_target * 2 + 3 - + plt.axis([-11, 11, -20, 25]) plt.scatter(x1, y1, color="red", s=5) plt.plot(x, y, color="blue") @@ -305,7 +290,6 @@ def plot_model_and_datasets(net, eval_data): MindSpore提供的工具，可对模型训练过程进行自定义控制，这里在`step_end`中调用可视化函数，展示拟合过程。更多的使用可参考[官网说明]()。 - ```python from IPython import display from mindspore.train.callback import Callback @@ -314,7 +298,7 @@ class ImageShowCallback(Callback): def __init__(self, net, eval_data): self.net = net self.eval_data = eval_data - + def step_end(self, run_context): plot_model_and_datasets(self.net, self.eval_data) display.clear_output(wait=True) @@ -329,7 +313,6 @@ class ImageShowCallback(Callback): - `callbacks`：训练过程中需要调用的回调函数。 - `dataset_sink_model`：数据集下沉模式，支持Ascend、GPU计算平台，本例为CPU计算平台设置为False。 - ```python from mindspore.train.callback import LossMonitor @@ -344,13 +327,12 @@ print(net.trainable_params()[0], "\n%s" % net.trainable_params()[1]) 输出结果： - ![gif](./images/linear_regression.gif) - - Parameter (name=fc.weight, value=[[2.0065749]]) - Parameter (name=fc.bias, value=[3.0089042]) - +```text +Parameter (name=fc.weight, value=[[2.0065749]]) +Parameter (name=fc.bias, value=[3.0089042]) +``` 训练完成后打印出最终模型的权重参数，其中weight接近于2.0，bias接近于3.0，模型训练完成，符合预期。 diff --git a/tutorials/training/source_zh_cn/quick_start/quick_start.md b/tutorials/training/source_zh_cn/quick_start/quick_start.md index b9d211561fbd82befa9f3ac7bee738773bd76008..10cbb10ec939b5054e5295f72b35e902ad860e5d 100644 --- a/tutorials/training/source_zh_cn/quick_start/quick_start.md +++ b/tutorials/training/source_zh_cn/quick_start/quick_start.md @@ -36,6 +36,7 @@ 下面我们通过一个实际样例，带领大家体验MindSpore基础的功能，对于一般的用户而言，完成整个样例实践会持续20~30分钟。 本例子会实现一个简单的图片分类的功能，整体流程如下： + 1. 处理需要的数据集，这里使用了MNIST数据集。 2. 定义一个网络，这里我们使用LeNet网络。 3. 定义损失函数和优化器。 @@ -45,7 +46,6 @@ > 你可以在这里找到完整可运行的样例代码： 。 - 这是简单、基础的应用流程，其他高级、复杂的应用可以基于这个基本流程进行扩展。 ## 准备环节 @@ -66,7 +66,7 @@ 目录结构如下： -``` +```text └─MNIST_Data ├─test │ t10k-images.idx3-ubyte @@ -76,6 +76,7 @@ train-images.idx3-ubyte train-labels.idx1-ubyte ``` + > 为了方便样例使用，我们在样例脚本中添加了自动下载数据集的功能。 ### 导入Python库&模块 @@ -83,8 +84,7 @@ 在使用前，需要导入需要的Python库。 目前使用到`os`库，为方便理解，其他需要的库，我们在具体使用到时再说明。 - - + ```python import os ``` @@ -161,7 +161,7 @@ def create_dataset(data_path, batch_size=32, repeat_size=1, rescale_op = CV.Rescale(rescale, shift) # rescale images hwc2chw_op = CV.HWC2CHW() # change shape from (height, width, channel) to (channel, height, width) to fit network. type_cast_op = C.TypeCast(mstype.int32) # change data type of label to int32 to fit network - + # apply map operations on images mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns="label", num_parallel_workers=num_parallel_workers) mnist_ds = mnist_ds.map(operations=resize_op, input_columns="image", num_parallel_workers=num_parallel_workers) @@ -187,7 +187,6 @@ def create_dataset(data_path, batch_size=32, repeat_size=1, > MindSpore支持进行多种数据处理和增强的操作，各种操作往往组合使用，具体可以参考[数据处理](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/pipeline.html)和与[数据增强](https://www.mindspore.cn/doc/programming_guide/zh-CN/master/augmentation.html)章节。 - ## 定义网络 我们选择相对简单的LeNet网络。LeNet网络不包括输入层的情况下，共有7层：2个卷积层、2个下采样层（池化层）、3个全连接层。每层都包含不同数量的训练参数，如下图所示： @@ -196,11 +195,11 @@ def create_dataset(data_path, batch_size=32, repeat_size=1, > 更多的LeNet网络的介绍不在此赘述，希望详细了解LeNet网络，可以查询。 -我们对全连接层以及卷积层采用`Normal`进行参数初始化。 +我们对全连接层以及卷积层采用`Normal`进行参数初始化。 MindSpore支持`TruncatedNormal`、`Normal`、`Uniform`等多种参数初始化方法，默认采用`Normal`。具体可以参考MindSpore API的`mindspore.common.initializer`模块说明。 -使用MindSpore定义神经网络需要继承`mindspore.nn.cell.Cell`。`Cell`是所有神经网络（`Conv2d`等）的基类。 +使用MindSpore定义神经网络需要继承`mindspore.nn.Cell`。`Cell`是所有神经网络（`Conv2d`等）的基类。 神经网络的各层需要预先在`__init__`方法中定义，然后通过定义`construct`方法来完成神经网络的前向构造。按照LeNet的网络结构，定义网络各层如下： @@ -242,7 +241,7 @@ class LeNet5(nn.Cell): 在进行定义之前，先简单介绍损失函数及优化器的概念。 - 损失函数：又叫目标函数，用于衡量预测值与实际值差异的程度。深度学习通过不停地迭代来缩小损失函数的值。定义一个好的损失函数，可以有效提高模型的性能。 -- 优化器：用于最小化损失函数，从而在训练过程中改进模型。 +- 优化器：用于最小化损失函数，从而在训练过程中改进模型。 定义了损失函数后，可以得到损失函数关于权重的梯度。梯度用于指示优化器优化权重的方向，以提高模型性能。 @@ -296,9 +295,9 @@ from mindspore.train.callback import ModelCheckpoint, CheckpointConfig if __name__ == "__main__": ... # set parameters of check point - config_ck = CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10) + config_ck = CheckpointConfig(save_checkpoint_steps=1875, keep_checkpoint_max=10) # apply parameters of check point - ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet", config=config_ck) + ckpoint_cb = ModelCheckpoint(prefix="checkpoint_lenet", config=config_ck) ... ``` @@ -307,7 +306,6 @@ if __name__ == "__main__": 通过MindSpore提供的`model.train`接口可以方便地进行网络的训练。`LossMonitor`可以监控训练过程中`loss`值的变化。 这里把`epoch_size`设置为1，对数据集进行1个迭代的训练。 - ```python from mindspore.nn.metrics import Accuracy from mindspore.train.callback import LossMonitor @@ -324,23 +322,26 @@ def train_net(args, model, epoch_size, mnist_path, repeat_size, ckpoint_cb, sink if __name__ == "__main__": ... - - epoch_size = 1 + + epoch_size = 1 mnist_path = "./MNIST_Data" repeat_size = 1 model = Model(network, net_loss, net_opt, metrics={"Accuracy": Accuracy()}) train_net(args, model, epoch_size, mnist_path, repeat_size, ckpoint_cb, dataset_sink_mode) ... ``` + 其中， 在`train_net`方法中，我们加载了之前下载的训练数据集，`mnist_path`是MNIST数据集路径。 ## 运行并查看结果 使用以下命令运行脚本： -``` + +```bash python lenet.py --device_target=CPU ``` + 其中， `lenet.py`：为你根据教程编写的脚本文件。 `--device_target CPU`：指定运行硬件平台，参数为`CPU`、`GPU`或者`Ascend`，根据你的实际运行硬件平台来指定。 @@ -402,23 +403,24 @@ if __name__ == "__main__": test_net(network, model, mnist_path) ``` -其中， +其中， `load_checkpoint`：通过该接口加载CheckPoint模型参数文件，返回一个参数字典。 `checkpoint_lenet-1_1875.ckpt`：之前保存的CheckPoint模型文件名称。 `load_param_into_net`：通过该接口把参数加载到网络中。 - 使用运行命令，运行你的代码脚本。 + ```bash python lenet.py --device_target=CPU ``` + 其中， `lenet.py`：为你根据教程编写的脚本文件。 `--device_target CPU`：指定运行硬件平台，参数为`CPU`、`GPU`或者`Ascend`，根据你的实际运行硬件平台来指定。 运行结果示例如下： -``` +```text ... ============== Starting Testing ============== ============== Accuracy:{'Accuracy': 0.9663477564102564} ============== diff --git a/tutorials/training/source_zh_cn/quick_start/quick_video.md b/tutorials/training/source_zh_cn/quick_start/quick_video.md index 75c15f0c82a95af47410f087654a6f07d399f1fc..8da94f547ad1a63eae72fb82bfc64acf82cf357d 100644 --- a/tutorials/training/source_zh_cn/quick_start/quick_video.md +++ b/tutorials/training/source_zh_cn/quick_start/quick_video.md @@ -108,7 +108,6 @@ - ## 体验MindSpore @@ -209,11 +208,34 @@ +
+ + + + play + + + + + + + +
- ## 使用可视化组件MindInsight @@ -426,4 +448,4 @@ - \ No newline at end of file + diff --git a/tutorials/training/source_zh_cn/quick_start/quick_video/inference.md b/tutorials/training/source_zh_cn/quick_start/quick_video/inference.md new file mode 100644 index 0000000000000000000000000000000000000000..9bccf474586b17c133300dcdeaf7d57f51e7da83 --- /dev/null +++ b/tutorials/training/source_zh_cn/quick_start/quick_video/inference.md @@ -0,0 +1,9 @@ +# 多平台推理 + +[comment]: <> (本文档中包含手把手系列视频，码云Gitee不支持展示，请于官方网站对应教程中查看) + + + +**更多内容**： \ No newline at end of file diff --git a/tutorials/training/source_zh_cn/use/load_model_for_inference_and_transfer.md b/tutorials/training/source_zh_cn/use/load_model_for_inference_and_transfer.md index a5e79747edb17264a5d862dbbd9bd80dbd28d4c1..c38c10191788bdaf2249932da565fc3c48832789 100644 --- a/tutorials/training/source_zh_cn/use/load_model_for_inference_and_transfer.md +++ b/tutorials/training/source_zh_cn/use/load_model_for_inference_and_transfer.md @@ -48,6 +48,7 @@ acc = model.eval(dataset_eval) 针对任务中断再训练及微调（Fine Tune）场景，可以加载网络参数和优化器参数到模型中。 示例代码如下： + ```python # return a parameter dict for model param_dict = load_checkpoint("resnet50-2_32.ckpt") @@ -103,7 +104,7 @@ model.train(epoch, dataset) ### 用于迁移学习 -通过`mindspore_hub.load`完成模型加载后，可以增加一个额外的参数项只加载神经网络的特征提取部分，这样我们就能很容易地在之后增加一些新的层进行迁移学习。*当模型开发者将额外的参数（例如 `include_top`）添加到模型构造中时，可以在模型的详情页中找到这个功能。`include_top`取值为True或者False，表示是否保留顶层的全连接网络。* +通过`mindspore_hub.load`完成模型加载后，可以增加一个额外的参数项只加载神经网络的特征提取部分，这样我们就能很容易地在之后增加一些新的层进行迁移学习。*当模型开发者将额外的参数（例如 `include_top`）添加到模型构造中时，可以在模型的详情页中找到这个功能。`include_top`取值为True或者False，表示是否保留顶层的全连接网络。* 下面我们以GoogleNet为例，说明如何加载一个基于ImageNet的预训练模型，并在特定的子任务数据集上进行迁移学习（重训练）。主要的步骤如下： @@ -140,7 +141,7 @@ model.train(epoch, dataset) super(ReduceMeanFlatten, self).__init__() self.mean = P.ReduceMean(keep_dims=True) self.flatten = nn.Flatten() - + def construct(self, x): x = self.mean(x, (2, 3)) x = self.flatten(x) @@ -180,10 +181,10 @@ model.train(epoch, dataset) optim = Momentum(filter(lambda x: x.requires_grad, loss_net.get_parameters()), Tensor(lr), 0.9, 4e-5) train_net = nn.TrainOneStepCell(loss_net, optim) ``` - + 5. 构建数据集，开始重训练。 - 如下所示，进行微调任务的数据集为垃圾分类数据集，存储位置为`/ssd/data/garbage/train`。 + 如下所示，进行微调任务的数据集为垃圾分类数据集，存储位置为`/ssd/data/garbage/train`。 ```python dataset = create_dataset("/ssd/data/garbage/train", @@ -197,7 +198,7 @@ model.train(epoch, dataset) data, label = items data = mindspore.Tensor(data) label = mindspore.Tensor(label) - + loss = train_net(data, label) print(f"epoch: {epoch}/{epoch_size}, loss: {loss}") # Save the ckpt file for each epoch. @@ -218,7 +219,7 @@ model.train(epoch, dataset) classification_layer = nn.Dense(last_channel, num_classes) classification_layer.set_train(False) softmax = nn.Softmax() - network = nn.SequentialCell([network, reducemean_flatten, + network = nn.SequentialCell([network, reducemean_flatten, classification_layer, softmax]) # Load a pre-trained ckpt file. @@ -237,4 +238,4 @@ model.train(epoch, dataset) res = model.eval(eval_dataset) print("result:", res, "ckpt=", ckpt_path) - ``` \ No newline at end of file + ``` diff --git a/tutorials/training/source_zh_cn/use/publish_model.md b/tutorials/training/source_zh_cn/use/publish_model.md index 9d3aadc66768f33a1b3b376d2a8fc22f93887019..51778851255c74789c6ae2cd24063268fa1c8757 100644 --- a/tutorials/training/source_zh_cn/use/publish_model.md +++ b/tutorials/training/source_zh_cn/use/publish_model.md @@ -24,9 +24,9 @@ 1. 将你的预训练模型托管在可以访问的存储位置。 -2. 参照[模板](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/googlenet/mindspore_hub_conf.py)，在你自己的代码仓中添加模型生成文件`mindspore_hub_conf.py`，文件放置的位置如下： +2. 参照[模板](https://gitee.com/mindspore/mindspore/blob/master/model_zoo/official/cv/googlenet/mindspore_hub_conf.py)，在你自己的代码仓中添加模型生成文件`mindspore_hub_conf.py`，文件放置的位置如下： - ```shell + ```bash googlenet ├── src │   ├── googlenet.py @@ -39,7 +39,7 @@ 3. 参照[模板](https://gitee.com/mindspore/hub/blob/master/mshub_res/assets/mindspore/ascend/0.7/googlenet_v1_cifar10.md#)，在`hub/mshub_res/assets/mindspore/ascend/0.7`文件夹下创建`{model_name}_{model_version}_{dataset}.md`文件，其中`ascend`为模型运行的硬件平台，`0.7`为MindSpore的版本号，`hub/mshub_res`的目录结构为： - ```shell + ```bash hub ├── mshub_res │   ├── assets @@ -47,19 +47,20 @@ | ├── gpu | ├── 0.7 | ├── ascend - | ├── 0.7 + | ├── 0.7 | ├── googlenet_v1_cifar10.md │   ├── tools | ├── md_validator.py - | └── md_validator.py + | └── md_validator.py ``` + 注意，`{model_name}_{model_version}_{dataset}.md`文件中需要补充如下所示的`file-format`、`asset-link` 和 `asset-sha256`信息，它们分别表示模型文件格式、模型存储位置（步骤1所得）和模型哈希值。 - ```shell + ```bash file-format: ckpt asset-link: https://download.mindspore.cn/model_zoo/official/cv/googlenet/goolenet_ascend_0.2.0_cifar10_official_classification_20200713/googlenet.ckpt asset-sha256: 114e5acc31dad444fa8ed2aafa02ca34734419f602b9299f3b53013dfc71b0f7 - ``` + ``` 其中，MindSpore Hub支持的模型文件格式有： - [MindSpore CKPT](https://www.mindspore.cn/tutorial/training/zh-CN/master/use/save_model.html#checkpoint) diff --git a/tutorials/training/source_zh_cn/use/save_model.md b/tutorials/training/source_zh_cn/use/save_model.md index 0a2312e85106badd0e5be989acee0612aab899c1..0e5e9a3c1105bab12594792cfa03950b6f514d39 100644 --- a/tutorials/training/source_zh_cn/use/save_model.md +++ b/tutorials/training/source_zh_cn/use/save_model.md @@ -34,6 +34,7 @@ 通过`CheckpointConfig`对象可以设置CheckPoint的保存策略。保存的参数分为网络参数和优化器参数。 `ModelCheckpoint`提供默认配置策略，方便用户快速上手。具体用法如下： + ```python from mindspore.train.callback import ModelCheckpoint ckpoint_cb = ModelCheckpoint() @@ -60,7 +61,7 @@ model.train(epoch_num, dataset, callbacks=ckpoint_cb) 生成的CheckPoint文件如下： -``` +```text resnet50-graph.meta # 编译后的计算图 resnet50-1_32.ckpt # CheckPoint文件后缀名为'.ckpt' resnet50-2_32.ckpt # 文件的命名方式表示保存参数所在的epoch和step数 diff --git a/tutorials/tutorial_code/evaluate_the_model_during_training/README.md b/tutorials/tutorial_code/evaluate_the_model_during_training/README.md index 19b9474a559963f0ba47134d2da9cf118dc24321..acd60348ea8a95aed80e0e36f20186b294fdc158 100644 --- a/tutorials/tutorial_code/evaluate_the_model_during_training/README.md +++ b/tutorials/tutorial_code/evaluate_the_model_during_training/README.md @@ -1,7 +1,9 @@ -使用数据集: [MNIST](http://yann.lecun.com/exdb/mnist/) +# README + +使用数据集: [MNIST](http://yann.lecun.com/exdb/mnist/) 下载后按照下述结构放置： -``` +```text ├─evaluate_the_model_during_training.py │ └─MNIST_Data @@ -14,4 +16,4 @@ train-labels.idx1-ubyte ``` -使用命令`python evaluate_the_model_during_training.py >train.log 2>&1 &`运行（过程较长，大约需要3分钟），运行结果会记录在`log.txt`文件中。 \ No newline at end of file +使用命令`python evaluate_the_model_during_training.py >train.log 2>&1 &`运行（过程较长，大约需要3分钟），运行结果会记录在`log.txt`文件中。