From 6a9ade456c5d7406109665aa0fec064342f70491 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=AE=A6=E6=99=93=E7=8E=B2?= <3174348550@qq.com> Date: Thu, 18 Sep 2025 11:18:56 +0800 Subject: [PATCH] convert rst files to md files --- .../{auto_parallel.rst => auto_parallel.md} | 60 ++++++--------- .../parallel/images/tensor_redistribution.png | Bin 0 -> 49235 bytes .../{auto_parallel.rst => auto_parallel.md} | 70 +++++++----------- 3 files changed, 51 insertions(+), 79 deletions(-) rename docs/mindspore/source_en/features/parallel/{auto_parallel.rst => auto_parallel.md} (79%) create mode 100644 docs/mindspore/source_en/features/parallel/images/tensor_redistribution.png rename docs/mindspore/source_zh_cn/features/parallel/{auto_parallel.rst => auto_parallel.md} (70%) diff --git a/docs/mindspore/source_en/features/parallel/auto_parallel.rst b/docs/mindspore/source_en/features/parallel/auto_parallel.md similarity index 79% rename from docs/mindspore/source_en/features/parallel/auto_parallel.rst rename to docs/mindspore/source_en/features/parallel/auto_parallel.md index a4b82b6d21..941ac02b2f 100644 --- a/docs/mindspore/source_en/features/parallel/auto_parallel.rst +++ b/docs/mindspore/source_en/features/parallel/auto_parallel.md @@ -1,21 +1,17 @@ -Automatic Parallel Strategy Search -==================================== +# Automatic Parallel Strategy Search -.. image:: https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.6.0/resource/_static/logo_source_en.svg - :target: https://gitee.com/mindspore/docs/blob/r2.6.0/docs/mindspore/source_en/features/parallel/auto_parallel.rst - :alt: View Source On Gitee +[![View Source On Gitee](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.6.0/resource/_static/logo_source_en.svg)](https://gitee.com/mindspore/docs/blob/r2.6.0/docs/mindspore/source_en/features/parallel/auto_parallel.md) The auto-parallel mode allows the user to automatically build the cost model and find a parallel strategy with shorter training time without paying attention to the strategy configuration. Currently MindSpore supports the following two different auto-parallel schemes: - Sharding Strategy Propagation Algorithm: propagation of parallel strategy from operators configured with parallel strategy to operators not configured. When propagating, the algorithm tries to pick the strategy that triggers the least amount of tensor rearranging communication. - Double Recursive Strategy Search Algorithm: Its cost model based on symbolic operations can be freely adapted to different accelerator clusters, and can generate optimal strategy fast for huge networks and large-scale multi-card slicing. -Automatic Parallel Strategy Search is the strategy search algorithm based on the operator-level model parallel, and to understand the principles, it is first necessary to understand the basic concepts in MindSpore operator-level parallel: distributed operators, tensor arranging, and tensor rearranging. Operator-level parallel is an implementation of Single Program Multiple Data (SPMD). The same program is executed on different data slices. +Automatic Parallel Strategy Search is the strategy search algorithm based on the operator-level model parallel, and to understand the principles, it is first necessary to understand the basic concepts in MindSpore operator-level parallel: distributed operators, tensorarranging, and tensor rearranging. Operator-level parallel is an implementation of Single Program Multiple Data (SPMD). The same programis executed on different data slices. MindSpore converts a stand-alone version of a program into a parallel version. The conversion is fine-grained, replacing each operator in the stand-alone version of the program with a distributed operator, while ensuring that the replacement is mathematically equivalent. -Distributed Operators ------------------------- +## Distributed Operators Distributed operators running on multiple devices guarantees computational semantic equivalence with the stand-alone version of the operator. That is: given the same input, the distributed operator always gets the same output as that of the stand-alone version. @@ -26,46 +22,42 @@ Taking the matrix multiplication operator (MatMul) as an example, inputs are two In addition to Single Program (SP), Multiple Data (MD) also needs to be specified, i.e., the device is specified to get one slice of the data. To do this, we first define the sharding strategy. -Sharding Strategy ------------------- +## Sharding Strategy The sharding strategy of an operator is a two-dimensional array representing the number of slices in each dimension of each input tensor for that operator. Here the slices are all uniformly sliced. From the sharding strategy, one can derive **Tensor Arrangement** to characterize how tensors are distributed across devices. -Tensor Arrangement -------------------- +## Tensor Arrangement Given a sharding strategy for an operator, a **tensor arrangement** that can derive the input and output tensors of that operator. Tensor arrangement is composed of a **logical device matrix** and a **tensor mapping**: - The logical device matrix, which is shared by the input and output tensor of this operator, is a one-dimensional array representing how the devices are organized. - The tensor mapping is a two-dimensional array that represents a dimension of the tensor sliced into a dimension of the logical device matrix. -Taking the matrix multiplication operator (MatMul) as an example, its inputs are two matrices X and W, and the output is Y = MatMul(X, W). Configure the operator with a sharding strategy of [[2, 1], [1, 4]], and the obtained tensor arrangement and computations performed on each device are shown below. X is uniformly sliced into 2 parts along the rows, and W is uniformly sliced into 4 parts along the columns (Figure (b) below). Based on the sharding strategy, the logical device matrix and tensor mapping are derived as shown in Figure (c) below. The coordinates of the individual devices are thus determined, describing their positions in the logical device matrix. The distribution of the tensor in each device is determined by the coordinates of the device. From column '2' of the table in figure (c) below: device 0-device 3 get :math:`X_0` slice, device 4-device 7 get :math:`X_1` slice. From column '4' of the table in figure (c) below: device 0 and device 4 get :math:`W_0` slice, device 1 and device 5 get :math:`W_1` slice, device 2 and device 6 get :math:`W_2` slice, device 3 and device 7 get :math:`W_3` Slicing. Therefore, the calculations on each device are also determined as shown in figure (d) below. +Taking the matrix multiplication operator (MatMul) as an example, its inputs are two matrices X and W, and the output is Y = MatMul(X, W). Configure the operator with a sharding strategy of [[2, 1], [1, 4]], and the obtained tensor arrangement and computations performed on each device are shown below. X is uniformly sliced into 2 parts along the rows, and W is uniformly sliced into 4 parts along the columns (Figure (b) below). Based on the sharding strategy, the logical device matrix and tensor mapping are derived as shown in Figure (c) below. The coordinates of the individual devices are thus determined, describing their positions in the logical device matrix. The distribution of the tensor in each device is determined by the coordinates of the device. From column '2' of the table in figure (c) below: device 0-device 3 get $X_0$ slice, device 4-device 7 get $X_1$ slice. From column '4' of the table in figure (c) below: device 0 and device 4 get $W_0$ slice, device 1 and device 5 get $W_1$ slice, device 2 and device 6 get $W_2$ slice, device 3 and device 7 get $W_3$ Slicing. Therefore, the calculations on each device are also determined as shown in figure (d) below. -.. image:: ./images/tensor_layout.png +![image](./images/tensor_layout.png) For two operators with data dependency (i.e., the output tensor of one operator is used by the second operator), the tensor arrangement defined by the two operators for that data-dependent tensor may be different (due to different logical device matrices or different tensor mappings), and thus **tensor rearrangement** is proposed to convert the inconsistent arrangement. The definition of tensor rearrangement is given here and the specific algorithm is omitted. -Tensor Rearrangement ---------------------- +## Tensor Rearrangement Given two inconsistent tensor arrangements of the same tensor, tensor rearrangement is able to convert the source arrangement to the destination arrangement while ensuring that the communication cost incurred by the conversion is minimized. The communication cost here refers to the amount of data communicated by each device. -Taking two matrix multiplication operators as an example: Z = MatMul(X, W), O = MatMul(Z, Y). In order to make the tensor rearrangement work, the two matrix multiplication operators are configured with different sharding strategies that make the arrangement of tensor Z inconsistent. In the figure (a) below, the output tensor Z of the first matrix multiplication operator is sliced by rows, however, the second matrix multiplication operator requires the tensor Z to be complete, so the tensor rearrangement infers that the AllGather operator needs to be inserted here to complete the conversion :sup:`[1]`. In figure (b) below, the output tensor Z of the first matrix multiplication operator is sliced by rows, however, the second matrix multiplication operator requires that the tensor Z is sliced by columns, so the tensor rearrangement deduces that the AllToAll operator needs to be inserted here to complete the conversion. +Taking two matrix multiplication operators as an example: Z = MatMul(X, W), O = MatMul(Z, Y). In order to make the tensor rearrangement work,the two matrix multiplication operators are configured with different sharding strategies that make the arrangement of tensor Z inconsistent. In the figure (a) below, the output tensor Z of the first matrix multiplication operator is sliced by rows, however, the second matrix multiplication operator requires the tensor Z to be complete, so the tensor rearrangement infers that the AllGather operator needs to be inserted here to complete the conversion [1]. In figure (b) below, the output tensor Z of the first matrix multiplication operator is sliced by rows, however, the second matrix multiplication operator requires that the tensor Z is sliced by columns, so the tensor rearrangement deduces that the AllToAll operator needs to be inserted here to complete the conversion. -[1]: Note: the AllGather operator and the Concat operator actually need to be inserted. +![image](./images/tensor_redistribution.png) -Strategy Propagation Algorithm ------------------------------------------------- +[1] Note: The AllGather operator and the Concat operator actually need to be inserted. -Sharding Strategy Propagation Algorithm -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +## Strategy Propagation Algorithm + +### Sharding Strategy Propagation Algorithm The sharding strategy propagation algorithm means that the user only needs to manually define the strategies for a few key operators, and the strategies for the rest of the operators in the computation graph are automatically generated by the algorithm. Because the strategies of the key operators have been defined, the cost model of the algorithm mainly describes the redistribution cost between operators, and the optimization objective is to minimize the cost of the whole graph redistribution. Because the main operator strategy has been defined, which is equivalent to compress the search space, the search time of this scheme is shorter, and its strategy performance relies on the definition of the key operator strategy, so it still requires the user to have some ability to analyze the definition strategy. -.. note:: - Hardware platforms supported by the sharding strategy propagation algorithm include Ascend, in addition to both PyNative mode and Graph mode. +> Hardware platforms supported by the sharding strategy propagation algorithm include Ascend, in addition to both PyNative mode and Graph mode. Related interfaces: @@ -75,8 +67,7 @@ Related interfaces: In summary, the sharding strategy propagation algorithm requires the user to manually configure the sharding strategy of the key operator. -Basic Principles -^^^^^^^^^^^^^^^^^^^^^^^^ +#### Basic Principles Given a computation graph, **Sharding Propagation** is a functionality that propagates the Sharding Strategies from configured operator to the whole graph, with the goal of minimizing the communication cost in Tensor Redistribution. @@ -90,19 +81,15 @@ The following figure illustrates an example process of applying Sharding Propaga 1. Given an computation graph with some configured strategies, it first enumerates possible strategies for non-configured operators, as shown in figure (b). 2. Next, it enumerates possible strategies and the Tensor Redistribution costs for each edge. Demonstrated in figure (c), the strategy for an edge is defined as a pair [*s_strategy*, *t_strategy*], where *s_strategy* and *t_strategy* denote Sharding Strategy for source operator and target operator, respectively. -3. Finally, starting from the configured operator, it determines the next operator’s Sharding Strategy, such that the communication cost in Tensor Redistribution is minimized. The propagation ends when the Sharding Strategies for all operators are settled, as shown in figure (d). +3. Finally, starting from the configured operator, it determines the next operator's Sharding Strategy, such that the communication cost in Tensor Redistribution is minimized. The propagation ends when the Sharding Strategies for all operators are settled, as shown in figure (d). -.. image:: ./images/sharding_propagation.png - :alt: An example process of applying Sharding Propagation - :target: ./images/sharding_propagation.png +[![An example process of applying Sharding Propagation](./images/sharding_propagation.png)](./images/sharding_propagation.png) -Double Recursive Strategy Search Algorithm -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +### Double Recursive Strategy Search Algorithm The double recursive strategy search algorithm is based on Symbolic Automatic Parallel Planner (SAPP). The SAPP algorithm is able to instantly generate optimal strategy for huge networks and large-scale slicing. SAPP is modeled based on the principle of parallel, and describes the topology of hardware clusters by building an abstraction machine, and optimizes the cost model through symbolic simplicity. The cost model compares the relative costs of different parallel strategy rather than the predicted absolute delay, thus greatly compressing the search space and guaranteeing minute-level search times for 100-card clusters. -.. note:: - Hardware platforms supported by the double recursive strategy search algorithm include Ascend, and need to run in Graph mode. +> Hardware platforms supported by the double recursive strategy search algorithm include Ascend, and need to run in Graph mode. Related interfaces: @@ -110,8 +97,7 @@ Related interfaces: No additional configuration is required for the double recursive strategy search algorithm, except for the ``AutoParallel`` above. -Basic Principles -^^^^^^^^^^^^^^^^^^^^^^^^ +#### Basic Principles The double recursive strategy search algorithm is a fully automatic operator-level strategy search scheme, where the user does not need to configure the model in any way, and the algorithm automatically searches for parallel policies that minimize the communication cost. @@ -124,4 +110,4 @@ For the first problem, the double recursive strategy search algorithm summarizes For the second problem, the double recursive strategy search algorithm builds a symbolic cost model, whereas the cost model of the traditional approach focuses on how to accurately predict the absolute delay of different strategies. The cost model of the double recursive strategy search algorithm compares the relative cost of different strategies, and thus saves significantly the cost of profiling. -Therefore, the double recursive strategy search algorithm is able to quickly generate optimal strategies for huge networks and large-scale cluster slicing. In summary, the double recursive strategy search algorithm is modeled based on the parallel principle, describes the hardware cluster topology by building an abstract machine, and simplifies the cost model by symbolization. Its cost model compares not the predicted absolute latency, but the relative cost of different parallel strategies, which can greatly compress the search space and guarantee minute-level search times for 100-card clusters. \ No newline at end of file +Therefore, the double recursive strategy search algorithm is able to quickly generate optimal strategies for huge networks and large-scale cluster slicing. In summary, the double recursive strategy search algorithm is modeled based on the parallel principle, describes the hardware cluster topology by building an abstract machine, and simplifies the cost model by symbolization. Its cost model compares not the predicted absolute latency, but the relative cost of different parallel strategies, which can greatly compress the search space and guarantee minute-level search times for 100-card clusters. diff --git a/docs/mindspore/source_en/features/parallel/images/tensor_redistribution.png b/docs/mindspore/source_en/features/parallel/images/tensor_redistribution.png new file mode 100644 index 0000000000000000000000000000000000000000..aaa18548288e323656514376c3f3ad9887ff2a8a GIT binary patch literal 49235 zcmdSARajeH*aZj_37$YHP~3_ZE$#t=6)2SAPO;+dZlR=Dp}4n5ixepC5Zp_V;O_3u zgzx`n=9#&iySacoCkOV~XYbecde@42rz($+^AZOI1qEMGK~@6=1)Tu}1=SybiTp<5 zG;a&}f$FLu4@M~)rrbtepj%0+NTZ-s#NytWVj!=vofLFkQBVlrPd}(h>;x1jD6LzH zveKGfM*EpqP0uIJdq(oLDW)4kFn)Kvkj_`MlhQ1ykSj5XoW}LoQ)OW*YU1kHt9T(G4QaXYF_6kPN>{gUZAEc}oL1Ow<~S9#6g0NeS7>Q*hv@j|uQ_o~8O#>C)6 z3L=IuC=d=kY`Gn7aNpf1DX*rwapuq1_lfW=n0s4|>jG$8h4B+k?hi zTQJlmKlRZV@bm{yzvy2uLjLayt^Z$K^nVjQsO#Jq&fGN$JRK6eXnRJeV`!MEk}WY` zR=0wKe9yxUJ6O>Q+EhE1Gdg-#Ebr}6)|nc*5!Z9hBtm4FAp9>|C@nIWaDAW?E06j$B{9XLPG9d5y!0UYxzbRo{q9*-o{} zsxXfchUBCEH!}6tS1OfX)g%xvICPdQ|9RbP7v6h@0h2=D0J{E+7%EW_&`RBE%xUd^ zR`EuhEW-wAd%n#G%XGx~l<^@P02cCC@V2To1gHxmXZ61eMkibHG(|KW({Pd!i-B+d zW4y@q*Azh7X%Px%{Th*e;Qjgkm?m=UBD>1ZAf!EWA?W{im%z{hc`crmdH-1dJv2vOzc+_bMlweJ^`S1GN7yvsEEn|9{Qn|1qimPZv;}r77dUb7SoP z97X;5=}nQ}#miGy-I zj;&60hxG4@dCx`NT8H_&*Xu*2wF{q~_O10*f2!zV(|C-;jWzP~bZ^de@6*HeNTA;1!+q(~ntuB?m?1o_DtYe~r~*)TK+cIC%m{&M1d|;nciuEk zB^&s~?;2D)%+HkKBF9mQN=tird%c-$kFZESM{Xz^DBK1aQGSd%cvme_E_LufT-3H+ zQ-uGwiA7>h&X2rVQjf5_w_EUdX5Skwv!9^MS>b)UaqajtGo~e-Eq(UW$R<}{-1GGhu{|fTTb29<8F$10iIM_5`IrJUe zqUZnK?<1Y9ihxR_ZlA*bPdo+FT_??hiT!Mi`z)HO=>D7A+r`HNcnG27-Dnh9{cr0r z`o_&H-waf>yo|oD%u~v$Lbj6}jr%pTc5`*k7o#$S>mTvNF6UeYoR*|~k-;&Og$O`q z9C%-Szs?otOtKS4C$krV%jq=hYB8K4%sKEN`EX3=X|VhjK4)1&Im<%{%hFeK zA2+iDTPzVzc2b?;KB?=0j7c}P$_U5Sd%Lr(v3SC10G*PWT%?UjFBVu9gSw=kF=sf1OiWJBL@{SgFHsifAdI75yuN#FIg~Ew_*>;U)+NcR#14jreXJRX1ePsh z_TbSo6Nk&?Os(bxOX7RH41C<2v(MMLdi~@r&oFl;jeP9`UrTQeXWCEyHYK7DD|+|& z0%qdYs+KKrv0uB`aGvha&_`bT3=_|3cN^Nlx02yH>`aY|>IL~7D8_&$)EsbsgLsPl z>?OZL-hViT=~$2ie&oRo(=2IyxEb!<^kjaUGHNmd5d%>p19_7e!_n4~5Sql}ovUYw zZE5v4Ts~9dYad9{!_6+|pHN%hUb@Kd&kZjYd>u)tL#$*`PTw#D_44KU<;F9q=gu9a zbFzvvpk+kDS0_(?Q0#Z$a=E#i*M~rl3MKFRjM)j`09hTnFet8=l4IR6W?<=w5Hd;` z8dqu5BVEc+$85AmdQQLd93(G>({J0IAE^k$e~Tt3=RdAp$*8x0A9PLwMuM|Fobgt6)q72+Cj^$Llo#k z1RUBQ_O22G_r;ZMmvqlsaq^<-YY2MNc@RRGVK47=q`GQELbQ-lz?lJ=dDZ*kLig@3 z7d@9tle&|Q18i_fIsdp%8BQ%~c*lu`pmrl;!I*b`%h_M$nt6}e>q9N9$A`<-OhHdg z0}q6Sg}4G;nm~M*jR7*xaF(MnxU4buhTDu6AfBRi_1~ijZDZkJSx5~qkB2t?0KazL zTu|)DZ*1WG*TS^Yrg&{zD3_ugFWC3IUpt3Nj z)Vm?afK^B_@`HG|uBp8KH3@8o{-gd=MZ(M`A{p?UIo^3--7k zj(Qn42K#5p^=}w}3hc>6uT}h76=MyXXsYN|X(!tr}dryVvtm}yQ{O)d9<1kZNPnOr&nEXC6 zG&<8w;LVI~l7bTj{hI4YSY#7_<}lz;NvKZ@fqS?B!Sb-ztYicI)6j5m`KASAT4$hBwZp zdrGEw9=@e`-%3}JI#xz=3XNFM^+cCes1HNua`_$xxLSoOvN2?M49dSMzfvYX07&$J zDdgrW?oI}tGSns})=$>sbOplqYL7zq^0zKKf Z_`B`%T&_utNcZ*((WB6-`sj! zs0q|sW(|G%U|yB>`nT}V#%xDgSIuwT%-7Z%_m@Vef$A!mLzfTf+ID)cKUog#d@7k( zv7F2+ORZjeTzPrfu)m@;8Q`pJ+uG)9KUCtnyKaAf!QN=GKFO{#D_~z@f9p}R!S)OB zIa8Y>`i7M>k&j*v_zUo>Acmob==`}wl1~)*V#;~6PjtAL!k`Ih(<@Vn>pwbIoK=d) z*%(}QJ8Hv%MXN04gRoj;OmCQTUKdHvaDhMzWXH_lfe)uN$!)y7H$FrBQ`5+E$LfO> z8c&EW=hZ$kx6C3dQv34>?aRfXw-VSKo!^yAytjWO&!q5}+pWN(9K=Zs=;2Ug`iSLJ zCoq$(JN>2v%L(nN-bPdv>Jxrdg#jG8kQr=kco|vZmeQWM1c~IbFy$+ zBi7EjCs-hH%G;Tq3l)lAU)AgW?Dbjvp-jpQ-k5<{qUD|ijRdsS$TTu1Aps?T`A-66wm$vua!JL1sp(=Fn{cAy)7`BQ67eW^%csXdQB z#;nPtOHIF??et;oo`vD0{^I$M{%Q^-7z_1Hq1VvpWdST9uF`g@48LF__GK2f-SdL+ z9@h|w7w$A@a>7Z1QJ_3l^*?xDf+@2CBKD3EVNeuJ75KA6wSqYqLrrRb0kWieNE0UI zzQ`tkd9d|1U(5_x4yKqQtNuoPd25B95y{6rO@trYhdJ_V!*ANg{u2>|GzUi69y2x% z%!40tQe#L6Nm^X{SqtmVmoB&5KKJJ%PY?KEQnCPS5IvUwqBGns!}Y+M(=K#aM>zQQ zHI+K#_Y$%;aw;#DM+OtOOG-^X=rgx|5P5dMjm;*0mjbP+o>vv^i_4t}b|N}7>*Tru zFFyK>D%vy#za`*DhmW!Kg{oW+`r@-(pY z6!XXyzg#)JSl2dAGHhl#i9~Rbd411ffJI6?O5E<5_fWdRC`>r2fvD3c|9+gUgDSTf z7~Z0vbD_ym) z@$S5S)E!AIa@`uuEz28V#zPO!T(rn_74g4uyS#A>yiNDb{+wWLI<_|R zsEOqZ>O0i`z-6GF!l1T;8T<%UHD`LsO38OF79pAC{tp2g-fub`9_Az9lI&nvAuF~W zFZfd0u=A6Xobq-Nne&jvrdRUO!srMFGtYks{gP&#j*Z_up{bT?9eY}ps{^$a)Ot50gFPXFdVPoVEtOLaB@ z5IxP^-CMEctcogc*_%6JdaFq=$KJuzyB>svEY=2PSyNYH@@hT}6MG(I`2A?`{$? z+{Lv-^6n_q)x2PSz17+e8lQc)d@FE09=kubT|R^LI-qi7_Wo1VK-+5W^FM~8bCYSa z?G1xQ3l~N<%CjF!zm)Vq<_)MQ1aZcD2_nRz>b8l;TMHgFSWQj-`9eX{FB7@;qTU+y z{g5(!i3}hw%w$naG3z3U*wg!#z!pq_GC*o@;qzYUc54E&HilTSm71h9{wafTa;>_i zn3w!)g&&19T>xlqe?TT7*9pqlW{TXTd<)cT!)+`%#Ds!#PN7sjTe%1BBE4p?tM{yT zyT*?{an#U6!~yzm_oG~l@#C6a@Zh0V5Se^m+sco_V!9IK8I9`^dOE99wR&TS_H%iY)@q!x|@2Uvc+f*BA) zfprDuONF6panKh~U@#E1E~tV!8HlR6xA&D?+GQW*D-4hDSqo_-5S7fo{TvKM!3#0b z7oY>bp=NyB)vHkk0Aa7iO>b}~4-%FCb_r_8jv=-1TkDN~m>lHph!E--c0kyx%`%4e^FME_FT%5!)LoB?|T>s*Qd<`X_;wJhnjm zV!D(3cmi#o7846BK+U?`B+%6cb!FFy#kH*}O=hjT7*?x0F*Utjs=f;BPV=kp?xVM$UZ-;X>kpW!a^Ii#IoNN_lt(2Ha zhRS$GA1x$If)h0!LnNZ_mJpTt<9$$E5EeQ;8iZDvdx{6QJk%Z5ZZd3+r}}_Z4r1pP zxE`@WJErGmQ#ZfPbOD>m9@7c|`BXzRM3jSz+CsE}yNOhj^^b|UIbz2}f1`4mjwD&$ zOJC=TsYmEcLu8^*W%Cg!ig_By$|gSLxD-*RTzZQWZP=kt1np9(B$MWR*)MQSgAyFZ zXa#?HN{wy=M4p_o^a&OCtC>eIvt0l1ymft>;SyKjk32eGNPf_>W)I0=VO;x8smbuq zVh}xC{e@iKM=^sCmv^kR0!g3x4Pe$Z92F8Xshe|EfYjr&(KZF#zo>;JU=uo$!YTTwX7d!-%v^jzD$CDvc~oZMil6} z{a*HYn{C8xAAp;$o|{M|m9u&p?kfAr4-&=NlgK1nCWT`2mJ#v>?bIq=Tmb$p`?~m- z_O_4!X0Z$8xe^->J)Q;Zd-ET2*jzjpGmuGXU09Z)<6I6P3$?}>6^J_x-PIekrZa}Ovt+C18wM%+6hjPFb7NhdQ9!!bARxD<6Gxi$St zzpnK2Wz66j3Maj%flI+yoQh&|&%LhKlQXy8a@J^^4KY1c#H)I>Hf=}UuSq|olT&Za zCB1}{C~`uW6F$b)wSEPD{hmhl_QwnQAXUou2p%+rzXFigJTuUC267-DE8O|D>@oLG zWk0(7{HB7hszwC7k9;9ykyE%9-$0iA+}K*j8`4{50;Fcs#}^=#Fyj_}-z~h|mO(LJ zBz{bx0)?ilW{T8rvmPg!qE4OhayZ*dmC)$AfN3C*}q!1y-|SgMfqoAhUTVOtQ|2 zKgdvFY&<*RLT!GhKcEaUdUQ&$zWfuTqr0e0P8`x-@w% z`q!#ti6QvWexeNEHqo|RZ%|krImMJYc?;6#Wv^&jAf#NZ9AdkDdVY+@@j_LU*R_$bcrm zbcC`qTU&l=`CS4OetQ0|e1B&1u)v(Y_76gZ!aNdb8dPTvtGR1_gmU$K*&6$VzI{>`xwAXSB{*g?Ued z6CU25X9PkyB}s+`pM_Py`c4%zk^xQPNFa`LIoI02I+2`&(IW*CWn%@3?H5+142_DT z01I-{rUXY6yC~D$L_t1wzp08GsdS*)6gn*5tsE#+(O;y3hi@Uyf_( z{LvcKMOU)%6@}dMCS)zf`6gPnU4-eWuCbR5vVl`PXs;Z@%$Y-+UuAi}PEY2#- z6rAIwJX4~W5=LZQ@cM3S%mYjcyCq01`LZAK$Zd$}4Oy3KqWv`HjR$+#^ojc-MZb}V zb~ru;JvEnn;%?_xc}C93>4@aOZo$hG(N1sK5{=7HzCY=C^0Fdc5O1SmNf0^E3k(0{ z+Q~Ckf}dvz_I*Hgza)HbbEF>V)rjY+s<-e4wh5mC)~N$uRUb+IJvlV zdMZ}cDq(f;5F2$ax+v)TTi{s4q{Xw`(RXU^uQt)ZV{_897QpYajPNe0+tg^FOW@P8 z?L~VJ&9y%7*I#scU)Ed>yf!xYdc$PD3x`f3gE(Y+R<);&Xys1*?=!_YnKF0|KSn^7R z6&1*e)=_;m`5JGJr@&_3^_cwPiq38UB z_^OvkoH<%WvIiE|8OCQ1ji5yeh~ema>*Je-sLJ+rhr~pTm>sq>oqR!L z)0}eVUCw83y!sYfkUpQ)>7Kq%Zq6)`dG&>N^**L`T6{z@#c9P(7ywOIK)hSYJ3$Gki7O-U-_`ZXg);XOELdHNL8x)q3d{&JlOuZCc`2CVs# zgbsr8p2ZN^IC1#&lC8{Oxm~Wy;$QOB)u(Wf@X2#v0GNm$o9IVyyoLxiRMN1~qXK1zl$R5ENFmh>S{ozt?{1yu59tR(+*Ip!$BQ1FPrI{-(ROGA@^ z>7VmB83Jd;_o-%Fx*u=;HoiD~o%V@OUv}C-?P|t%T7d6%^jGWV`c>oN@LMllyoJAe z{fnvh_Jweli5SbqO>UAk14qfz0av(ZmZ$xqt?`?y8O_e-Og zZZ7>}Z+!?#`x4tZo#~dTv@jI_-=H%GX5h6PK70xSzB;BKs`^k<>QD&@=Pes0gwejUQw3kwr**V8LNVNFAbOM(zC{7zA z_WO~-?_R2cd!OC&#~J{v?Q1JYES3^z=a|YFlzH zb*Nx2KCxxt_Fg({6I>hPBkDGmu3dUOcf4iVzVXKdZUVY6i-jg{2MVfD+LqLPvgV-N zpkv<|n?m7{FDN3v-G-f036{vN_J5ONWGbp0?>2R2DxX%7e3Yk9U&3){!gFI0a61a2JTJHWW=v>@xC1?eRiGyM) zhS)4^Im|CKVmd~OE{8mHI_-M+heX0UbY}k?1r+ADt~fu+Ti#QvaxjbS>lQ8CGaSO> z)zbVo$q%%f(!36>9ucX#=W^6(m1m;mUYk^2lzu#&duN-(vxGi&Hdg>sx*o6A%Z)cx zs0)v_M#^?vA%db0*45coo)cJEu322*L?Fd+Vvd@wl3LV|;Nl#%C6vbN7|8-?c@RC8 zLBPjRxX?S;>t1Ato?}!Ilc4p5rUo-m4!4zfi)DM+VmLjJ&Hp3i@XFEfT(%D%?a$Fi z_J8ItayS_cQk`_+t0)1R$R3@myEKYBz9MsHXd09bkFz2c+%6-q_|B$`*_|3jk>xSdJ=wxu@@?(GaOU~e(Km63k4ts_?DVQcRKA|BfV=pXYIv})xZ*iM63iYdgAQeN-$z;tBgeChcSxhiX zrL?vDC1qH(U_mPWyhO#HPYwcR3AN=fEUmoGhh0jC2o_k5jcBc#*6$D4j2ioDR7BF} zCA;i`rv$Hew5v{yc=ZFX;l7b>e4?IBh{Vw)qRo)V9#o1O3LGn+-Rq~WXknB5GDh+v zH`$Vx(mSo=M5vyaazp^U}BS{>0p2&tUk_$n0 zKBz)-u2D2gk^eAdIj5cZNhM(u};3{zM#N=2Ir{z`bUA9>UbvKG24&3nh9 z<~GRcs8S)u^GTW+k1$Fj1}6%Q&Th5%mtE#0pP>Vlx2!DIaZ}i({S&OkFxUZQ!kyiI zN00tok87zpsAmG%uRWLqz0%UR=f4~b&s8mC=s%U;q>QClso~KftCxGT*6efz$Rc6R zbW30-4q7^{ZM`32+JZexP+cABt@ye};LJDy(;@S4D;n=ABE`uSfHiEaJ{SW2RINSs z{t1&yL8&=5OuiE8H>DH9Z%MVOx!+kh^tqk?*cF!CE-DPK;yRM(@kFD3T-0!kXdk(u z5LcZQiH{7i2wfMCb~oxUwaq6XT7l~gq?{k&O}==DqLZ>~Rx+C{)=#?^Vm_Yl%NpA< zd^S7G_glxfDj(ruKUfe3DT}z*1TkC`rcrisFFp8x255YV-}a? zvs3JlD7ME6+FYFOl*jE0-y+Cau5 z;8N|@cRru&IrXjyvd*md@+X>@yWV?ut*`Ct6ok@X#jTW6st$&g6(oJa!})Mf-~@^2 z^G^$IMwxV+cMYY8yu#r&AbSn#9P&ptWIXom3NU!s@i3>K5NPHj-kqim5nIG(S8)b0 zoQ2DsFkz$Ctt`U9VXyYRLq}h&GOMlD8+&?bi&v9|lY3v^CA1p-Vs??IY>sMeU!6Z3 z8YXx2XorVQfjO7x#yYsh2dY)pelNZ2<&h3~8^DfSS>Lu#RClCZRUJ)QHB&A38VmVH z2^{DqgGNs67zY7#QVEAYo{vt^>T(#7{Ro-Egyj{AIDUJAk{&;z$;zP<4QC!nhJ`wv z^!m{)zUaPqB5i5E`$>uAb-64qB}wfV><35husKTUrii*BSQa7x;sC?n8;)z@W~ui% z?SnLsRIrLfd*Mq?zy0Ii;5g~;pu7auGEx#gvAsdmShKW@>KC;lb0b%HMx^yI5jkLu zKHix`wrvJB?qveEA{_E+y6chY3iI*qgyniEqiBNykeOx_Ubh?66*W91B@j+7QjX|d z=bMn_+*I}kAN9MxWPho9;MoA%2gtbf^AO;mc7`mfSwTlp#z)sz4&K%eTb+fg0xY;b zL_BH1Tq z-R8^yLesG4GNB@XcB^@JSB01qhJRCaww~L6yC#x2>ik!YROPc}zlFJm(sE0zipUg`60w zB5jFHN!GjpEzv#TwkqS4(!UFP!Qw+@x6Qf(?*tR1KYc?WW&u4XrNp;an`H|Dw@lx1 zn+G2>byP#_>;%l2^c}y<&A1$p=7As#^kQb{>u)BY9-49TVF;hCuV!(A-Gf}H08TG; zx;WZ|zn8LJT}PR`7M~xB@0Jmpdg4z z654+}eM?Dg!tX9eHw%a5OgLY3=pKCG3wN=)#b^qcX(1fzI!di5?A|>Ze;K08fnCT7<4X#2Lz~EM~*@>=?=ObW^lQag`ut z*gG3n^UTabSy6OW#$|(XyP-rSng5F>f2oAp{pz-FeLAXDIcd92&-8m7gzGDdw0A$V zF7X#yBT1N9!HEK$vc$D2+SIlkaDYlRM_xEjjIN(46d*6mbo`!GWQI!A!wple&aW_3 z$o?fno!;`nT3`cBzx+D3>={EW^e-5I>O2|7KBh;n;8EwumU`eO< zQWj6p)bU?ySc@D!gptY}LS8S(;bkz2gomv@QhPLO*OS(^thc6wcyxfaxt|@G%)n_ zj?gQPIE3+Qp@gHyWVGdC#0hhpU|!NxX;Y^y!_`8$u53f8_W2I)A)i&OaYK2u@r8)x z7==bDsX!HVwANs8pn|TP-%@JMh%TEN^y4)1wMmEz#qd56DIlAy>Ca*?mJS)X7pVA| zyb0`N;q(O&r(vj{MYAK`PwX99SJ36NhYeyT`JmTl4KsVI zWoJ;AK#a}%>1_fBgysapg4f%~ZK-|4%0~=hkZ@{hheUzOWKuKEpu-GQg=UrCHqnxv zkMmfBM1*r}zKxZ|P_%mUd91>X+b5j;CB*$VqiGGF1AXZhV-;dKoY?NH-f;R_MmEVGZzSXH(r2C%8x*_-%*QoHzr zPuXjT=DTk{MczbxOnR*EXz)lmgun5YO(f_i!9>ek34FX=Jl|2zGT-rlQ9$x>!s9ym zM#RS5Id9K*HysUe7!tflxy`=)Y;M~Px=ADu%m31MWGHe5I4Fk9afgA_2qmseE^otN zqp4h&aFFbPQxW4-w7Mlx)!1<2&$5m0 zDIY$vEHxTY%y{01ZGy=xJ84lFsR^MNt%m$`Y*NYVgOkV27QLj-zpeaYc2@a6QV}UK z>k6D7tXhaNN{qKgBB&Q6w9YS%)02ig^qJ$6^Xf0>1zrz5?3^d}Qng=RH9IT_mu9a) zR388mi^I2f{=Q5uVD#IU^}npX)Tnsr9Nm`YAIokDRDGp7X%wHU9>z446T1Xh)%!`X zrHln{3hSI48^CZ9rdF_}B}I~?qj}WKHZ;d=3!W_#I{rXG$G_phgR5_%n05sh4Oh$m zULxy_a=luPY`lEdixHaBb_&1r&gvW|VQ}GQPzSz%7|P6lmPgN)2Qt>bb{CvIrd+)e z5=7Sw8}#c^nu+iu^5yMSl#Dr5+Qg2>xyhHNg3MF3U1QDGOV(fTg|lF>X^3>Twa0ZG z5jJ||u4zbPB!@Hrm}ttAL)y^>MCyofog{>`g=;1FFGpj+pwZAYZ9f=yEJd+m@x1fdN;) zR^@cN6^kb-{4E1Vcopya8tXM&clJ%aQIq$omBGPbR=`d9R}KjWNuQ-dF@<@eY-Oy_ zR)4ak-u}`SikK1MUz|mvuZ*!!BALjTCfqPHMj5YXnbLUkeq8`DTBtq2l5SyH*8p9R zKujEt$ef!V&w;MBe9?&F66PP#e;7@Xw|uSAWKv#b@aj9-K>qoDMC5{6 zf!a`)@AO>&sqODzMjA8*EwTi--8yj|2=f_;bU3F29!Jk7Of}|&6|c@pq=tdRxJ*)1 zr^(^=a1|T>#a4eBYQYHY7bLs4f@k(~i9Pakzsjt0>Uoj-to*YXx)bSIqm6y}> z2Zebu!RsEC4!RD(ytBlmnH)WHv6b-l=C>2eT$r#$%}sI+fH~ zQ_v=wir_6zEA_wF{P0#43oZd!Ll%T4*e*Hm3Rnf_aK+w{RkS@TP9eMf$PGUVGdT;h zjiBwUR`9SY$9XYWfv;sa8D)6_Wf2!k4hxqe2I!Jd62qPD|-8@ z57w^lLm%*kH&2dpUV#4NMp&ZroO*{fARbIlzRqWM!bEHdCNFc)yUKTOthAMYXsqsQdi1z^0{OJt|@CRri$ ze*11Vl$XqaB7mY6EOJv9hsluRb;0ct6yILNOix~3Ujx8`Jx3YAV#rQpmk71>XfYCm zi}NMKDN`qWtYe0)tzdqxAoeKbc%G$}Wlp4p!OaGqh`1ug7z~fmw=nPMCD`m`@fOAb z!OOKowkxL@h`!DSZ@vL0@=SX}cpgAsKwaJvr|yTP0@hR98bJ5c@ZSn^wd_$+A7eh3 z7X;XEio4|B+a?h<vPuJlF?66b zRE%H*!+{GFvo8vk7j*Rz?@Xia;BSSBds#bCm@K+eEVs~;M#kzRE!lIuEEXEFa;O2| z15_FMu8-SCjiGxJlu|dG7;E?Lf+f;R78oI2&lsyj>6A%^ui&l{8otScXMw$nekosu zT~VRley$XNiTHN6rzHYYUDeVnqXq%I$WD_QzMo3uJBT4=02wG>pz>5P$>^p;Vz^TG za;clw_Wi(@kVaT?G7F4f%Wz)x_TE&$`bMCp<8DsIT-N_v=3+<8t*NW1{j7}Lz5B@s z!z> zd7@eX^3v$oZG7RnDnGH|l=UIqge#28nt#M$L1UHaON*WHwImc=Vs`+kHt@;p%}@l<+ow$vnB&@>s~moY_dXOVdUcDv{EhGsa?A z`6=;v34nq@`9y-q1suAQ!n`b6Jh{1kMOwu@(Fi+}AQLY->bU4!vZII`z9^lz{&31m zvAN2W&sgU7X|`p@i;8V5f)44(@>RPZ#&LpPjUBjnerlkM` zrufAM*t*fByM^SP+xXuZ8AoQ$0|qdFcMNLV}STp*AM)Xp^^-H16qK1>D7tT+I?{<;!gB# zN-t9GP&7|RgTd%0oxY^GoV2p&R=l^O@Y_OZ^g;OBO>Nui-(vK~WPho_O`DC8EH~Az zUF>P}=t2Bs@8M&zdLQA$UqEX$S#l($R}Gfnm(|`QyfH>916Z;?_mj8hCWr>Bns^o9 zm`tL`;;ejgE*`30r5+{^j%$dTj*PFDyj|F2OTB^*gAtn_#2AA_IPzQ|v}Gm$^At-o?ZCtX zHzOdF3yuXrok~@QI~~hhwViXZ_09*=Wf3GORkk|O*KtD?!(1|)yU8EE?V$@Zgsi0` za&<5t`2Z5y!d)S8yJ3vEl(_4pplDypk1fo2QSrL6yahCJZS4$De-to5>yRhwY0>%9 zq`@H3xZ_ za)CLsG^B;^{{G6BmvAYZuifhrut5Ac&(P_g;S-(7ZrwHO+4KHAbx~?Hq-@Q|W+f}L zWjh+nB+H(m(3mC_WD>#Ml(C(i>=Gm;%pm(dkDgd(GR`hO*=D7sQ<(8}H zKRkO)Ix!tlm0V3lT5juGULAVxk#1k=A0>9wp#W3B_dj`}G?tYRW%Vt|^9QvJ-diq3 z{7w{dkwq%HRH8nga&?3Tj{?}jXuZigI+5CpZ6gK+ZZ>uz;7t$!^rJ`meTa}hSWA{( zm`eAAT}}#^_b%nj`j2EImn@s`wSo5N5FN9Q`xJ~H(*}@r00e8{gO|M`vQSCv%=Ea* zG^~EK5inad-~X;AP@xiF&|b0<@ii~^_H2FsYUaRW+(#b4kCkN;IJbRS0RYj9doiqB z4}{2bQ#XeXHhfTB>KZgb-<)jpbY%MVJP%znkt~>K8qWpqo>D;zZtA{^Sxtj*Wz==-CBi$!z-Mz={V+)qr#`QsQXXc z@XfH%5~KoT(VKA8@Y2 zHZu)1V_XCJyWtZQtqY&j`DT}tA$bi(6*Y{u_is}PFO_gaB_KLaOlqL^*0 zQb2jrz;YD$Ad;<)7`l)>pgU;T;b+sC6rE;YY zcvG7l_)ve!DJ5kk^3GC(Z3-j^0YoKr=ZAVcUj({o$G!{7s`ba_Zy%5IGP>vQbkxvGJ!WX``meX7o>!Sv7lrrxj}Icr0V-&jce)V0UP5si57irmp+bUDtXRi^VZ+z@q*%&8_tNb-xwQEw^BqSQ44!-5WpA-Sw^ykWd~8QB+MZ|Q^WE6z z0=a|$ly)ecJMQL-9w;4VJ5mj~WZ5lMS2%CdYJh7wtjKf`-OpHu6zmF4Yz$_ZS8WW{ zI9jUJs#&SJ4*cwO8L+K|yok+P-`_%BEl8GkE$@-zn8r548`(}b)Ge7sWep}4`sTHuF zo%zXLw%4!sBkaBxDb3j`B-T>7@2sFB`%SyB5-*p6ZV@_)ebTAUe4G|OHMkHg&wb9E z$GN!9KiiktEwEE<(%zVK0uk3Q>j)Kl%3K^>00Qh-fJlv1hI7Od?c+f-T-xuz|8joQ z;BAy^DZ~+V&IW}0a1<$7r}~jKjx0XNR^|HRcOXggOh0iZDmcT-;pS}nJ(0Pr8FG26 z$=abL`PLsTS`Qj__mB-A_mwxxFP}u3v(!w>#o`pic$vti{y5VRlNcVzf4_Rl<5643 z6hnJ4Y&S!*ymQFsDD9C0Ch_;UX8^qd^bA>Zb^cUN<~*p%+YS+=Tz~c~{^X=nZQOo1 z&s)H}yS&Ew7+ijzZtreHQDHp{E^kWZcbx6G9q*W_v9&5~s>ws5Gm~ZR`n!JyU51Z3 zv+5{Lqw&2=hMYuglN*JM}uINpwGdy6-dF_3}@rs%8moN z_MA=rUXgS?6l7XtMTW?KHXJCj^v42-lvCEW+kR_X!p$;-!%akI$*}Pt6xV84Et(_s zzzHZEp#BLd%Px@BjRLv-RCwoofv}n-(Kav*yx%px_sn}nl5}MbCP%hpDU$W`h$ehq z5?7{6K2Z=AttcUxaA<@HH%Px3DFK~D>X$acs+8E(#5`ebM8oh3;t``yp2%@0Cav3( zqB-{Knp%+K3m`Q|Op0S{sF2=Qsu^1=KX^8vj%Ck7%U(+R<@_Jion=^6ZMf%Ax>MM6 zN`rJaxRH`l0SO5Kk?!u=bc0AYD1wBPG$`FlcSv{lJlpp@=geF)ALg1*XFl$Wy%%e( zr~m)^{@rb@^ue7uFj=>S7Y`%9@`Rp&;lBCjml>5cG@I?Qjn)gvU$%c!Xv zRpg|BJM=AT#k-x!3eG9ksa*TOLb0TXS`Zzb$V6bzBTiR^vIAl;Dwyo!!K|T@K5!-ev271w8_4e@QHn#ER+AcKTy*^x2cbj)#zqFJ|B#GjLe!a$g(m1>p zCIu7PD$diZnXPN%qEmm4m!gD&VEl*Nl9d+=I=cy5=<@3L*vKG`I-Qfw(!UR*egApo94q{R0M=NVYk8ee~4ObAL@Wu0&2ltaI)-7jy^ssaKlr!{dD|b@f36MTHy5Gn zMD?kiEeq|0EibCeTD7AUu_FOv%}>f7Rhh;^or)PTk)xNW{n10j1@vD0?Zl@(3Bg05 zeS~?`K~!%U94xRjg2DJx4tZzYhV0mBoU_Y^XeR0Gn9k+~J@c;5f z?H_1*eM3SkSQ8c0!VYiq9XG;9kF)M?y4I?Si4KS+0kGhPU;TNk8liM$ywG0V%8eG? zBLKL7Kb4$>5O+WcM^A*>`-AX0K!VUK3kLhX`qi)#fjKA5?UB6D7Dg~QUg$aherZ#a zrHGt!3Q{A_C;s_$iM??rNA0YJ0I!Z*59qCi0CJKC|H0oE2DmNR|M%P$7-L_f`|-{X zk~h^wSw7SAAPLy_^OVw)rqvpFgydcF=rN#8)O$bw$k*eLv1oR-Gj#^y0vh(cL7%=G zALVi0dQmHx405OIt-;>yGTxfke9&p;-+Q4WB%qf|SOclsRo-V)U1cqI9!ye5Tw{HK zxDxNmTkaZK9=Ao3^6mftgUi?$aWC*acj7-)?sLJjz4Sdc3taZ<*Bf-q9hLwKqw@A_ zW{Apt75^H1{pJh0TgL`Hz-Ng>z$wn{Ur_x5tWNC_&tT%}EO$9@=!Zg$Z!t)0{Vk+ZM>gam5b&=71~h zGNJE!sf!V~ASETd8Y_GAFLeYBpYrh|ddTe@*c#mOSApEQ8TfkQJFo-fRVSNGnMB?` z&hMqiAwnSOvlQ?o?#6F~HuDp9!r98czI>(gV?l!qn4Tc^-MX@e@hl+}~Z>+#YsO|J^}05$EawRznkJ zD09f~SGofq`PrK2M?rS>P77-~xozW!$N8Q$2*)l5FdH!ZL6^=g$REf5sa0;!q{k!W zcGSbP`^5J$0_YtkHW{f14KBM|2AH&DS)la>s5~}kMUVS=zrd4bU;Xp*AXsud?(g`K zEFt$s+GjTMWm z0_Fz0H^Hz><5&czZ2O+WWjlQmnYZf^PnzxeHb(NS$5aHJ-I+e9n;fn7)PRt^1`CCM zU@3TX7{=-kqeZru;eZYs(1@YB=F`&Spw*;mIfLGL3Cz2hfa{ykFYYl)gx z9oT}^03`IwWyV_YKH;B{jiydf0mJVvwg(a2oaj9?EBD-86eeJ%snI3LO-~iH%~Y!C zJpx#qh7t+)TYz}D$m06GksGl!K$OsrO6|4S9GKH$w&F4Gqk{QZuH<8`%Nm`Lg&y>_yW8L8z|Wr0$p&W1AG zzqb{q%x>H%vR%4T$d6^1e|psS1WZ-Tw%r9^ha|4+g6;Ca_2-(X_}<0dqgSa)Tr!y4 zC8*~?>-W<^k}}AaXD1hP`>N{;Qpj&r`Bp)Wfop)2L>L1|H{N{Qg#o&Ch9^9(vF>!g z<$nGzjp)4l5fXl|=0D6G+FUu}5Imh$T|+>Nv1$A4FIU~C+)JMbYvK2WMnaYx z_>jcFMy+Q2wdIroQ^s35S_+s;qktmuNDjahrs+@Gt#17*mz3w+pwzYqkR? z(;0N8^hH}1te^l+{f4^yvQ?W&-}~DPyI2;nfGN&s(LEtJ?#c>G;CJIErOj7cXM>{q zb;lfu^^WTex3}OnzI~~g?zLW-uCQ*oJsV5R{&zV^FD^MY{HS8qSS;|23+$~#um0J) z!T-#Rx=Q-*Farz>?KsH)^g)N9umZ%sU#EQvP3F*LRhIpCJO?rJrbCumi2v<8;3C2p zWB;yQU~Pocq5k7f|4ly75q|5iUTj@@wOw9VI39va@ot63x7hBzrS)v9!T9H&GodXd zSH<6^?Ge9dm6L7ri%no|>ZdHtyMDTAfQ1lgyz=_kS;as{J6fqBmpbsUOIs7VR_~HN zVdk-}quDXqhmu@s#Eu$2lkGK=k2d{va=ZGMY5i<(4qmPUd1ARE8Lj7k$R= zFPQl4_(7#4uj7yFv`67o$S`-#;7^Z?2562;9X91=3n#5UiptjpmX7`mhq4o@5x0q_ zd}QuPNgivpaaoVmrR8*AVSHL}$UjHhuib8R>3w6X{8M~K+)Dhrjuv=Z6R{@brUD`% z0{QCCYyyw06T)pemdnRRlhzzqL;uCNVMELjuOq7;$R1g1d(NPhYjIVo)~ILO2w1(q znJ3N*xhyBggzBSo2`tmeRTtl2;8TA)JA8Y;MsR`!xK-Ve0B#mvOeygTdPZSZ#KeaG zZAGTy>-;EN@W$}96ZyDfKK1|+$*gBAcJIiHu^b|Q=k{#qbd zk2vx`N2fB%c^?nwZ*65H!lPGp3n%9(n9A?gsoeYx6Ca{!%aimKC7Plv@@5hnkX8vhg7B_6MiUNf?@;SgZGnLyOJHQ z8d->I6AOQm7q4@6_cE^zua)M{dzKoH*cPZ#%Zf0?PqJ#m0>6B|j&yy)D;8*Rp0!I_ z5_+z4E~=9$+U;}`EWs-rfBwy;b=0JI;W<-6PCBv29p0Ty@3AUtsJqz2l4XKp6M!Lc zZUy1d7XHHCws00OEN!9Us0%)F!avZL_QK5vKXtnKTe zh6f#6`z`BV3l=``l$z@gJ?>hkF2_EI78AeO^LyOF{kS4)SorZ7`CXLqM4KdT?LtV` z>;2b{sU^F%eO)(&OUS^^H z?x;^pN@VcmR7oB1!f88I;B=jL*_Zgfe{r5WZ|)#e?$zh|d?I0G*)3tZ;>D2R)h$y} z7#Ih%_2RCvn-{#hyw~nbS#OR!YAiFht{6JZD{IG6do6O-TTZG-`y84aKAd=xW^Q+S zD{7^fHqZe85gcM@bc0f5B>Lf9e~Q3fiKu~Ei~m6INz0?*uN}+2<~{xnUZBY7%ud~eWkJAOZKGsJI?b-Eu_%Tb-PWP8z7zseh&+0xQl$Rrw7kwnd7AZICZ?tPIC@nN)1oMW0_v0-VO6CTr@ae-e{waTj! zmBcJ7CTPDbCIon$y}YjA@6P&@u<_xKQu73_IJ!S8rZ9Dj>w;KB*?MQ2%QxSu_g#@L z6MR!>3JU3KLf~;&8qS~MUxLSBKW83vpkgfNN&Aw1S~Dhm@OW9}OO$)d8yn$J9u29> z=o1gL4ba{pC1-fk>TDjje+%mBA1G1wsZ!3qGidXA} z*UDsvY?zf%&4umTcwFxVOvAx*h#J^-ZSU|Y^L$I(D3d)Ki4i;hZHnk9!4g-WPvjuZ zJ2DoJ8onJjba+l>B-?n+@BU89w|ZdvC^{paZpnhjK6yo`Px05rJjv75A!AmaxxCpy z=eN~@^;GpqQ#)&=N(+dMqbt5Pv%k-Lj+mV>Dzmhj*F?7}2~Urcq8IDgPA>9<_L&Fy z7hFVAmk0CSp4zrKr{d*)gs+Q`h1;*vE8DXip3@Jo3n$+_`;xZ69&w1ze`g=FGu9Up z8mipwm*rxHTd8+F_4&J#Lfug`<>Kw&Tcw%v?iG>z0>QE_1K;;2A5ivEPrE(eG<|tP zMsmQiuUehi@%&sOjvn5*82&6swvM`MF?HWwScaZg=FCB-g3#NZ%^;N{rgXg1AUc1d zwam_@p6f~d0AC`aq}wH?GQ%>Z*+ps@wRGYjUx(x;c6B1*qgBN+Q-KMGKh)dZIUUv) z9cHB8V0wd}V~;3Jr@}5S73G`O%Z68;oHf!5PduF7e2XC-Lom^TJOM`6r+;ReLA@>f zrZ52y&&^f?|73IoZfow(JWL_#}1vxkLQU@MDCu9 zE_CZX#}vNT+g@OwXp8ewu7^)rjM5o+xm27p&Kgj=XKxt<_nRKrU{uJ}yellj@a2i4 z6-hT05%&?W@ZoLzrSv%ChO|U^ecbC_RVu@v8RybwH35s3?-4(ZDOfr^ zswrYA3eJ>#%hq=~L@(CFrF3P|tC#Ya5=dq5OO|5J7cps*xdu^qBKc}K?FYhQ+hSBusdAtwBt>*TY=bMzb-nHBIJw*|amaeVH^L!$A(|I}@ zwxO=i7=!=_s3U&!0>(QCy=2)LYFYilH@Kdt)h04P!ALWE@kV@!7 zr(5miwn7ei7Z11%MqXZ85;^S$!?J`jMe)Vq6Z!+77(SCtnWrzc>BaNB-?HjSe714& z)tS5iT#KX}>*P~~1U;cn8}w}ReG^?j7Ik4?HoAbR6|NxVbUc!ClgCsmi@x{U6~Me$;|Ja{vDh?F=;i|34Vi|2=N>{g&9xgba+X4v2BP zKz%)!=*{9E*Y1cGT1Eqi;D3(o%WxeW-yc%3;B$Rxxi(~7JI4r2qQ8F}VP^iX6Fgo8 z5XArEIRD?_gNEihTu~ru0|KipkbbgSUfsC6RSU@rd36~&Cr*L$Mm;vUs-Ecb#2 zsd9$M?+`4~<#Ke|Po%&Fk%RA@fZ&*j34_z7Yoi3FyTR8YA>8e7=lbv0;85g$)z1ID z_o;`lzpn1ibpJyN7ie_v*HYf>^e|;PmH#Ur<&FY*M2=KLQ_}=+KVpExcki$bw;AC6 z-QKrZyhA4okN$b_uwLNW;hFXVmG&KQ zGE1;2MScm)+R0ju1$3&VKK>erGTFX=^`w^nWc&KIzl^5&~g^V!{SRQVm~ zC%b1|l6ND(H0A_q=HFla^NQl)K}PJzz!AS$KV-e)a$WL%3Omv3{`UL~;KHu52xMS4 ze_uxxobC*S&Q&pnzi)OK(82^<0P1KOgekozWO`l=m;J5brste5(29saQzdfuf2sg+ z$KUV&zKba7^SwFiIp>LQOC6!QeaRfRUx2eoQEB&YQ<2j8{ss(&UC@MbhrfOyZ`vi$ zIPP^(`|jvfMwhHWhZ68D?s25t2JQ`bq8@NTfK9)aE^FAH6+CICIie^BWm zG=^%oU4T(F(+X(fF?~!U&EB5p2Hfyovn!9Le*2b){*{htQvBP)mZcDKU_5;K*K6tx zF8%=nGIc#(6pfS9c&Sc}+oAC}nTXTIB7rV(1n7$&asE>ldz$TLy!` z>4Kxg2Y@<9NB0l>KX%C3y3<@&F)>lfxZI^#@)6Zv=;;Ps4TUzkN+{2pBZh{U5jmJWk%#HUjn1 z#`Ej*!BSFQvpQ=Tv^dzo~sL z+7{wPi`(GE7T~7Wsc}t~#}d6m?a zh?C;)06Y>%$d|2|_t>5S#)=g{H5aF8*aPf$&1PZRG?+0qd`03;tN|LX66oQB(ZYWq z-+@wl>sra{x~E6ZZ5*WO1wf+Cf?B*XXokPw!n_Mm%cqS(Db3*Fdo!6)$Ah9nxAXZ5 z307aejaPzV0`};@B;0b0NKgH> zZ1~mp4wUcLE;I*RsEyASW~@7ES{kf%t-mGR!wrF4r{Pd&qG2{Cg6L9Sj09^~6HQPz zPGB!UvE030Q%{7nZLsDYqCGqSFM0;VFK;#y@u-(V{OjPx2M0bS0NdV{gR}!1-B!?k zw%(OgFo9$rxP1)Fpyo^n{#O8R61BzZL~ET1+$RjK3}zSpgaB<;=#FobPF1p53~t5g%up@B+V_jktf5(V=01(M`L=AdiXFoVp7*dL z8#B-c%kRPbw>)T+!|j^l8i?DexjbIKwoF3J9Llu_486;`J7MzuQfY|S#GJ(n_2*9X zQ$GW#{)LNpff`cXh|s(G^~`PC9?BOBpfP6AkzX_x1ocv=#(=66+I`8HadAve!}vmY z25|q!_cPdUZw~jA?*BMzLGBa9=M`T4tS6;cFh=%UChsw50efweDE`^(*9eH_sFY5z zEw56=z(d)zRuH7nxQ5~-DTCZsBLcW_wM`R`>2KB0$5ESpZ?h6e(g~30+l?qk$ay~S z3_bzT-D7ijYDbl1CegjGL63WFi^jdg$zx6-!$pKHBh&}btcd`8Bo!|j3r0L=d^KmEa z2nl}~FjZXwg7DO(MCGpK#R+~rUCqc9PZzNMXxXK$C%*byzLBwAQ2JB#=dHy{}H}dw7o(E*t=cOXx|2#^bM8iyy53f9|ZCK4w)s4T=!9b5Ta1{fbd6 z8Xf`0hA8zz`A2UR+RgpG@4Y3czu>T)+vpB@0sZ!egvqm6wh$%)ZO#N{+|v*Ib2~kK zR=Xef^}F?i`zxg{s4^YZr5Vrv7S{Pp^$aO$_?f#=nNJYXou<}GNMAee8%@m>-#o5^ zXZgErhO3gNn7cR`A~FQuUHA70G)TTvt7me#b>0@P++9|e?{rc7R9_x9S`Yl0j5+&! ztG%8S(SR(_@n>fCqr1Qm@g#+a9&NL=*Ojx+I%+>f zbtZBqta4O02lLtVmocqBnF(UevuzT^FK}bPbhq!aQ_*#0nS{6otNh^yd?K{-oLM!W z?yN4E=i=k;F)p$9rDJB5YMd4m?f09GhgmwfQ@Cpcp7`Xk$Z^fFH(qPX1x#N7{yO_u z&Y>HQ2OF{lu?jXAGG@-*onV0Mh|x1^*CqbZ)LEm0#_EHuZ+UqU?9iQ6#esKNnsMT3 z8Q5Xix9>1VS`{7%$lUZfLm?vD8a41d#THB_Clk?=wlr;tkAgS{kcCFnT?Elu?bgba(Iv}XGK^XjfT zyd=b1sw)y+v%9V7*lAnI+OVux)`QsH7flk{@lUx~Ax z!BSFy+O7HTLF-H7;v^n9*Q|B`z6G(I!WP2S2E^}V%3=&JshH$K)!YABn{Ow&feG4? zDC9%qPtSmbMahLWq5LONIXdvjF6_iFcJ(rDi;|dY2-iG~iHbC6a^Iw2ruT4zKJYh; zFj1-ZEluG|I+fRvtx<{OwOgM)H#qAl0tEfFCXjGPP#;`+`6y z6v93wR{dMIpWzBFtjj;b@VD>VtRil2IG&D{K5b{qbq|4ho~fFt>bh#E`BY1jAk~a%PAj;={N)Xex zCd@f&3hBHB++QKHv9e{6gb(!UL6m}dUuO_16OE3Qo2#I@8%9fxhVMt4@j0b~Sl4QaD%Y~CzA*oizCGxr zHq{4ssLBtjf`=~wq*2TKj~e4E5?X|j!1Gb*A9sMWD8etKg@-eVi!K!wW!!8wj?d>e z$j05=DO0cv_A`}|H%qvCqLH6?DOt+#b=?-_)}(hNAGtyZpbD^JscBC0af`Wk$!kr~ z9*ndPr`@qP28(7=Fhv>~h1Zg00~{hhU@@35kc)m4D&MT0g~up0y??mmA5EGQ1}dxg zrlTUE%Wp=SAS5Q_#NY=EtCB|@3>6axNQQB3R)-xK{w(J$Jp(Nxs~S^5Xah0sT5v7M z(7sj`Eg4u5cw4`2y@Qc8Rn&L$exWwQ#PKy}_0s_|EtuaA1MaLDkEp^?XC-ZaV|d4ijT`i?YVfzsdQiq0(w|XG>GZdB7S% zB{**7QU2gDKpva5vSQFayrA@nSy($41c6g26Ct<8JL2FFW`h?a;4h+a31!9XeS}zY z!pp1~1mjVg`(~+S-e*WNmTb3=9+U0O5_%TxKpJFABe4Uj{kQ zjLq#9HmdG42$N~$4TyV#4Ezwqw4(HM^AE5MH}4OXiq z&4Kts#cGwPh@ZG|b=1_EJfU3clJs$4TjA4NH}fJB9}SP$p=Xh?`;;1NvD3=4P=aom z0xP>J7`2x9_=-K->_NR2heZR8LPTkdkr?yJ_{7&sp3wg&Q2|L^A&oY9QuCWw#W}~a z<#NFGMsS%v$s8%eXiG^X8|`bNeUR0b`WC&&XK=P{yx+Wfk!z0hQ*6Dr8*kp-sXKRT zqYHWW5q+v{^JSrB%Da_Btz7%oY%iCaV}Z@}`>6yDiEmG4ZjE`ys z-$)@i$|!sC24j^pPVz*(qW&G>!Na1tp9XEs%qa(H#yN8)Tp!j$D-~wus?1f*b(6?d zM3nQ?9&$f~?A`-j)&^@P2x+9ZL=J6Lnhc4*}fX>&5G^D-4Rlf#1+oGtkJ z3My)$y~r%C>#w~@X}S5~T!#O0PO3ybpEkTA}ZjgxxB%eEnt0tM$?81C(gEm}?Hg zi-bkW(^-G>_F>KgTts5k{dfHD<^GJfift6fM$#$XqXR6dhk` zJ?c3>r*wUt>anhy+;A!fk(r2*Z=@CS@Fd+LLReU38#hwwyFTxnj>uYt0@)oaHE1en z;cGZKS#!SDLEmoTvv&8Bk$pu^Td3uFcpo!V*bkL7N<_w~jEg`u8d=VqOuthnC=4(@ zNAKk>{c2w?{tmL!z?MN;9BFelm-doZBPc)&w7Tg`+3EGYpTqGw7WrQKxd57+*dM5-LJ3hM{gbGfso!`#=ln{P3He%*?umbkiGw1{-x5egc(l7*QhOj|2 zIU-)U;SnzEXRsc15Jw|Wp$5^+&1B4=Rv0o!!}n5tt$_*0g?DZ99b>#xZ2GIJd7eVK zz{z-rnX4x6lw;7L(FSgSNLS00rIW^rkAyB_0iw*Dry}uQB4TMbzPXN@J3Z3&EW9N0 zG@sfGOUG^BuKwtpQBitpKmXcg@|EZ0A97m)_o>e>hS0>`aYf_FDiFNcoX_$5d;^d2 zji>sPkitm~1}arA5n>URnZ^)m&K}*8cV{++HyMFA{V3lnSa>@h22B#g|L_?MF%!Ga zpP*fBkn8CA1|p{^a|n^5jG&vH1=_(u>u3L;enA{>F$+5)Ngm$2XY;zN!HZ8Sq8dR! zbbj6_dcloa8MA64hd}~c01nBXQDPR&3_mt+j$#te7O*JLmkYge4oCN%PF zO5W8Mr+$(Ao<}L_NN)fNv)sD(i!fs(I%eATp!rdz;=(E00ERGdo=-o1_ER8? zAPm3c5|r%AU}AF?ooIaRAT5XO?IMR>$oYe?lG{N>A(Gh-R6tOhT6Bq6dhW#w)iRNJ zK)>+t6G;ed0OE|>0JkxkF@-Th?lcMt9cD28Q<9bV{lky@qv&0&tF9%6ml9cB?{WwS zPt6j)&c+9V%i%G8y<>dXxmlF8w?&ePo!uW_Czz^? z&Ti3*SJ}f9)P6NxSvj3L^JZq%Ww0&t>M3V9>qYUBA#v<GcjKSVAcqZA3~ zi_g0^pJrvS>?mcZ=5gn=1Ts-pQ7=9qc`2!5Ea&}I{Xin$3O`=W#=F}gSdZ*P6pd-X zR3?Nu4x)~*2}(>%;e@$>-7C8&SnF@U9JZ~|a6is?EuyvRZ@7w+uPdLM5$bi&%RHHz}0+m#zhNkcc;hmDr!H41Z?m7LxNR(8rv zJsym=>e?ZZE!7iY0sS%c(y^aPyem2FvV=F&-kammCV@7wVp)|1%^;xNj30CwR#VzX zt9QU9#QHQS`Yy`S3pD)S@ziwG#lC1Pc?NmAtciu}P_dg%2>IP_h-b-IZH*c&sr!hR zeU$alwu8(2YyI8DCS2wQl`jn34@o?>1h+CBHlHLDTnPt-F5UXeYY;t))W!(ZTMeNx zM-?#Scc1Kh;EgM$x{11&LrS7(NXpHeDW=Czghk*7IP^s&!(mGFq|+dlf6FYzFxm%$ zP=N^%7bX=6dx)hjVhdr1ge#LOhKA$^Tgg9Qbf-!Uw!c@&BM;*VcrIMF4CHnv&V;%; z{CGzYV%(sP8N?AuCq}K7mdC1egVWNrzicGkFSZxyj?KNC>|5jLObI!LGCi(>-?Wy4F3ib3gsF(~dKlq7ni+3_mY2+b72f`rO;-*Dcrgro86-Q9Rv- zhz9u8rz&6?zZ!t_M#`uH>ihz0u3t^;6nc-KOv_6dxx3+W!vN&Q5BM9VwQEqFWWK^4 zxh2}dy2QggCpzwvo^d7Qt_^M!dR)42Lz7@dwGXMIac`|&G%!HX2}q5l{)!mzDDTwL zur70=`U03ZS4v;Z46h}lYTVwcMWA*)O5%nHZDZZ_g#QGp_D}SDYv2b^nljcNjT#*`IVnsU#yrA;P1Bw9<)H zu}HUsOnP1E5lH4ehC&`8!@ASYkk$OHZtFV=J-G{Ooc{5SCN!Z-RZog>KY4g?5 zX_=E|o*{S1l6K9k!+Fekoo-5a@%boCVFzcjpzxN-I<9cnp!oT0C0kwrN(^t~@CR1c z-b9wcz%Z{zVwm8>m=Gb$7#SG!p_;JuS2sfsg^*qL97w0O(S?{DYXdf*197>9Yd1?9 z_wWPdKOlvs2s5S;2-Pq>9j zDJ!SKCe#=8JIZJ7-qV{k8?M|PCz9KJg5o{H*F+a&E0VBGH$33>dlpTzC9W=%$W%rj z_)?m)h&d-pF2ArTSN}&d=JW(J?K%;n^;#@a>K#H20{RANHc={zE+v(g<8suYUiKJ8 z@Mp{x>GgyIR1uT9?f4XbocE8VA}J_U2zp}){e;Ld8s1UBYa5vt(};1sq&-JoDkh-e zgQ+p}5*j^&%jB3Z30`7R;Xj!*lnpg7CegF=Z>g}5U60U7KlpF~bQ67MRCmS?jEQ<| zz%%&@PnX%NLPHHi!Tz*P+a?fs=XE@Wx7~l7{7Dvio%ne;GVKf$^;<3H2$qXC`56fa z-7+6ul{j|AvL~v^$ECW@i#MAdt(^6c%JOCSNS%X zpY@4VDH#Ze4u58fPg%+2xu(S(S^QlRw(otC*V1PG@ti!b(HptF%OvZAN|ZL*dD+@_ z-@PGyvs-WTknFS13}~vYE0ttLI&c4k5Iz@bd((&IO%jjHL&Ja(s^^T>^W*u`0UDA( zeb~9t*Y9{Vc;fHO(c`d?0^t@%*m0Z8Gsm1o;pi?VMC>($fGJ?`l7zX+eGB=8lK4%N zCI>W4YMVObK+wx+>{(9xf= zjzt}LY`qxzdX&TK3{rwP4r8-%7B7d+Jv3rga6JF!;SkV-+fl%Rjlc#(L~t7o^YRTs z3}Ykn#zh5XOawyX1|C1WERwB4E7s#B79sSh+yuy;mi|J6MhQNUCD! zJ!%Z^k@?%@Fp{ha^EBV3$o2SxHLR-XEJ6c3G@SacfT-Ox!9U~9QI$OJZ(aM|`1X9> zBTa0&ZoTpN{kr$v<&yzoL7Dz)*9}5ZFYB1+H!&?a5@P0S+Bg8nq;>^K&5j!6XO9$# zh9bgM_sDZjB8y z+QMIChF`Wyp=YK6i^tuHc6?Uej2@n~GZ&Dd#(JH--mkrX<6)=sGXuLkfQBNEtkqAdZ7qF6touu8cq?E~Z10Zdkjw;= z7<_OPl768FoW(c}d;jYvydni3Y5kuQP$C|647)Sgo6^RHKE$W>doGm0J>BrNuFZ-w zC?1LOq1e?T__1A8#IUg8H9VoZ2GJUkl18ld;l_s}y#!`qgdr?we0wf(>aQs#&Tqj| zmx{j+*KUMxp-vq+j|==PBf3n!auJ|XAYOXm#VMjMv~(8ZDw%@GN*x3=O;(ngb_= zwL;>dB=~Vo+5i$A+z?EGV{||vTlYBU4U6>7v-QXQ+<3hM- z?Bl%-HI^S}{PYVjXk-I-Wk>ZdhU+}d?i!gU35Vc?t^*o%R^Hm-HXKX|>IomKdwq-7 z7KE>sGwiy5nu)*S^BngS=aAorDSKT-OqC3!oq7sYP#gwLu_@$jp8LM78Rs$kUYdZ` zbs==j?AvqjsXUG%}AIWE_(6U$p^5smTUmEc(P|m21PRP7Wqn_E$+! zSKvhOY9k6vdbRpU7#BhBJ4l8KG?fbrPE5I<81HgF?mZ1KH*dlaf%FL}BSjLlJt;kj{spyExCjwE3{Wo{?lrPFLN z;o|BerQc7@j``??wVBK%_C1nsC6iK&UhE#8@OU3K+ji}Lr!?6S>4SayUS^=(t_;S( zZl2{`Lym=%$+WznkU!|TwEbg46e`q0+S*)KW`w8Dz1MDY0i*b_nz}UGO&IINp@R@e zSUd*Og5B5cRpwOY$h{abo^cK>(x4yuue1xvi!o9o?wvxM)Y~Nvsc@73R`zx?m*&El zk%v^IOS+RW%R*LQL=2RiD0{-^R55&ml%7H9zV!sNY|)SQ(d3Z#aNCL9KQgm5s($bD z#yMgD*}E^eHRMI5qMZ$DFt)bhm=O|K&H!C5{W_gb#?(b` z4mxf*3J_)4y~;}TcIr7Hm9xjHmiOkouamgEmKLZP=WoQQDJgbneTeJ05Ey;e{8j3Nd zd-PN*Z2^WV!e(tYiu`yHOA)B3dw0L;A+10Bw2mzb%?qZ z9=yucmm8)Pdyo@wh7seMY1GT$9{UijOSZEcIf$&6TM6M*h$=)Z6pei+L4qhiI9HXM z2D+#oL^J5U>{T!CwvGQWEL(v*CMqH}P>Y^cT}-uqFSwX0*GVg>EQ@Y zyF9Jm@$^hK8wb3s_>rHGX{d%RaysPXImh0;=QIo&!#C<=|7|4(!ENl?z%rr_T0u6% z%KzkKU`K)#b|kGv_06 zLO;eb52GO5ElV#dg?Sggmvc06(m#}!!WX`1$zEGe;|#*lmG6 zuE14)x$yRexMqF+Ha#a?x>trkzE_XuR{{hTvb^aaZPaRP_lM1HiI|#`dXpGsJYK%U z9SV8bB?`e0l2FL;euqWesf-1|!fd379|~S+mCEg}#N(8ii+!GrF1rD#x4>hCh~+O3 zJw=lesH2*d%oIhEbQnXzoc}dD!Ie%jZw3#ai_5|@^ns-owr1`@b*)mcY z66;5%VhH3h4%F>|ywi_d5$R?UVozqx5FAU|DaY&r7 zp5Cdst5pcDRU*~Jsrx6>F@Bk*9U3?c-L>sHpK0DOUAHknz0xf&(@i&$a~`EPdosIW z7JwFi7PcXUm7N!z#xty-K-_U0tWGV`Q6fg-H;AzJ8%7_Aq+W|UTrcPKM_exI?S>s; znKUJF9ZqN)v+URQs1FM?kqB)qkav!wq>e_A2YK{A(TMUA)C1Il9;ofSa8_?kbdQwc z)#&dgaXP~qJT;#nsi#{Bq$X6L$DQroIiONX;cvX1dnu=*?<-ujnK^P~{0`?{VS%Y# zy5>}*cdTjfs?)arOuOaIM^KkLZrI^DzRWZKQi<}WFS8a;ndISUek2Z4@{ENA3=zu%(4B0&beMgSlToAx($YyF*vg#1&~o7R0O}Q z+cvnYSr^Q6GNk>mC5A5l%-orVTNJ6}N?>EL$6FPW6>ug4JLDIsE=-JFOf-_c>WQ4k z6`IqQ;!`e#TJjEI>u*FEjcZ(dnQm-r`Dd?ZkP8tgxi_s}R#io^tbXe(uwxDx|Ngyo#kM`aw zs*dh`&5KCm{)h;O-XO6Ao~2cMBE)LU0N0?i!rn9xPbb`SSf{{inNU z&8(Te=!?l6YlWg}zq_{8`#z6LkH5Jdg_&5g7llBQ)sYaD6L`HWOb~W#Dtit>a{kHu zjlIXNk_L;1UAKH055bUYK~elgy;Ze^B85v>Q~0x_SIz46Kd>5C9jKebuZb>Uc@CY6 zKpC(QIwb7QDofLqMT@oFj=%{! z#n542wiL`0f`g1g_q@l9ih)hRQIN=@yodf1SSUFv>#3;&>6`q}6l^Ntsa5A96$_1E z4S{G}2>l+w%7>C6M;I1^)?a$%xuu&ml1YaZbablo5P->UmZ%}^F8js!>K|NH86#g) z2wG0G_t^;}fL8M9vpJ0@6jJL26Yi4`mncNg%HJ@7)OVU!_n1sKvlW|F&LFc0afj|EM(4(79(cI>9%DQPwsXY$IkeeQBxad${D9 zUw`_-X!axsn}AL(O`wrlV(Xhi|hVB$`)jHUj5Lbj7~& zwT#|3w%HRM^C|k=?(M{aj<)WXZHq{4_Qflr9a?*ST~xIrp_0i`qmmUCS7$bi0UzUq z$PrB`G5Z}zjcFKC&SHY)Uuy9f8^>?Ph^vd!gXQ{H)8E_q<gCAEO&aTe(*_<^zxbs<@WMi3yHxL4r#jt5)`{oTD0n^zz=0@!i-Xg zS5ZFAQ){(gdvJ6$(KDH6Qb?lhg2a#vRcs^;D9zHOJ53x-feAMWVh85B6Yr80Y8+|M z2GyjHMG$g6N7^4Sh^)g~!HYUr;-tI~N1+V^)q=&XxE?PuX7zZ+s7@(}_@u-1U%a#~6SQqceEAZ?9@(o+TN^3$ zbajm9I)a`~1Ra;MP_7|}EMP*Jxgj#H1K*RQ$Ykx;d7H6(L8S#^5m-nRe;WA{kz?r{ zQ?FbW-BW`nvgp`k<}%OfXip2j$UP@+aKUyQ}R~hLHz|l7g=rd~ly|2BU-dFPAD# zZpIL-7(x!~xr+CY!%UC~Q$>ld;U6Jy=|6E~TMt-18ATux-GZHuSj1RC48XK`aw+UE znzf14vOAv0fpaA!d9y*j_=6dRZ4MO@zRL&~1O`);BR^uy=vS_W4rbE+7pWVX7ov}| znH(FK#`)&7rH3LoCjy98&w{Sg?OhYu(1a&ofn& z>y&>}zDu&1*jKxdzudm@S<4Jt=uXdwF8N815S3+j&|#M}!M!RT2iB0u_Y_gR@G+t@ zMLjw!;}TeY&AhaEdt%{5!1B1s;azC2#5$O z)Y*D^LhD6{+?L8Jgy7_$Q>YBFCjoX9;p8{$)hlH`J*?8|hDKxBHfJoB`g(5Kha^NS ze<}*%V)u2h8#WgdA9BjQtmR3nj`P-+&1$Lf48DlN+-vxoCJ@oRiZlr2{03(Mbe zVbwP*-zy?%r?g^9GN9XkuEH5yve$ zOOp1I72jz1D9+>i1mc^xmaRB!)ZFha-$gpOlw{06#6v8@lWZq7%CVKDJ#P9lnw>O9 zVWHNwZ&C}R+ID^-X5qJH?c~+x*XrF^7|)+9*DU?n9uai@`sVB8(;`iy1u*xC)&MCP z8^)h9tUJB)QkOCl=a@4+dtOUz>IFG_<>pldOEwi4dfP@&7mJr%o}e}lfk8MpZs02i zfge?21h~jJ#|-kC2i@g4X6QR4`rSL`DAr@XDi}EAbfA=97K6SI0hVNPvTjKjG(L(? z)zItJDZp|N5I%@^q|zN^2Qo!ZR%>KjDqX9Q9^&HJ4rAeNq=bnoq8v32xebNH57a1A zNpO9jF=w7`s6sZ8K!AdvbV=;u`1NRE3Lm3PgG=*^b73TXd-C3%i?8uBl zvPhE{b1;!fun1#H#!{S;hWIQ@KRn>l2- zOgdiRHIwljMmZUPw%_7*(I~LNsEm{3e7j6YyF!j6r0(*onT6Kx$tdR^JM|Vd1(f?I zsEkT_X!17_S?y8NE6rnL=Sg?Mbwwgj7T4FvG?qwb< zooseYzV44?&)fQ}!bnGv5@KjG2odu!ympWa2)Fpe1vH;1PlF^_H0w!SD@vPLCJo)h z^>i$Ti~LNm5VnJXk*`5V?&jA*abt12?484>C3^JNP=0h~?HDSz2L>w4HU$c8W)Sgb zVsSd7#`)nuu3m?wElJVo+_5G1dBK$7#koK@Bl+f^@&z%R0+ z4Jl5NxmHiNpIV;{M0b*Am6v75R;3$nip;sY`Krx=7)VY1Wtt2eF+p3_>y#apP_>`x zfS-b85u+(?Ey5&7h~ja`97XGg6T^dK8vzp9^>GR@RrCO>fDx8P-KLC-zGIp zyXsV&0fJeU9scuj=o1(sY+P@9AcQ9?%Qza3ur|J$kP8*-ST;7wj~q5aU>aH)ocY4_ zm-zsGgF@eyj5i%Cp%GHU(A5z1*&6vj3UNyr@NjTh3YJk91;Kz}BZH54_y1K0a9SvJ!)IuIB?C zhW*r~dW{Z`enbqVqYdnJVc2?&bO|r-`|J1feD|YI!YEkcdx|La4sv6Q|H8{`cz8$ zjp>!^26w^t;C$GQEQ;C)ZZnzILR7z5`#r=2vf6P@-*-eVeWq0W%Ct3!x!J|}VuYv* zQ`}W4i#SJ2@&RCERbKFHUTl#(C zJo#0Me+>HB&($^ET~)BWQ&j-kx|}Ge$41ku__Kf^v=PMwwa%#OSFgzQO99!Yrfo<6 zgGP?nDx}suTlQh)T<6zQCHEb_YOjR$=N}#XV3gM-{uz4P)Oc}%v2k0)$T|Jb7``Tb zBjLbu91HAzpiyhKMe|g7a_`g3O!{$W`g3aQ^G#U=@ouo5yBCNe%{LuKYfj7D9J~>Q z+M>mSNWY&KisRaU86r+-GDl5^!dXVhzqG;Pg|PX-I4l(WsK>cppI~M$2^1P}{*WLQ zthW6=*@%dheyk(Q&}e;9e-mFw_sW;U_J^@B@q&bW_37?U! zZfq%w9DaT(dgC|#o^z=vOM>piPz*p0vianHCznp695tiyu?iXCp<};!V~yL_uFGMB zUtJOQ9WPS$zWa-$wv?UM@bd+Hl%&jIp`SyJ!Uq)Ojn?{$nUMTJ`(mbGgY zgL44F*naln_VdM(XA{#lW6upQw@m#y@Krc{B+G6D1W_F660GXozz1i zVW$KY9Ui`O#j-UrV=(QUR!rJDTb5kGf>sQ)CwSiFq+=eAgXEZ}4McXpaKK+@r}JQJ zYDqu~l^?Cv{@L`7tG|>15hp((xmG^CKGD@}E&y#ldl9QPF>roDF!x)tKw{l*oxLH! zv~O9I0uBUCEw}xiL%Fw$WUrc&Karxo0HN?eK~kRu-eO-^kIm4xSzp2?fBEae(KtRCmZb#NJomxwe&+ELGiYR%jv8%K~E#_0Gu>r!^=9&Qk|ik zqyWNM=sn-&#hh-z*v5i@tuKhjM9whxalCT_!thH_k%OzYs@sfCgw;xZXw08VFWMg&9vEwLK zKGXS7KS`}B$Q#hlm|D-hT3QbfAhWJ4%JOkxF7jfOLl3&UT8|SIRwvLuC4~7cSUugz zdXlza8VgYI)YxO5CTtej;Yq%?`uIt)`sQm8&X6&ei0Vm*qrD5PQmZRYB@Bax?Nf0< zhjh9bp~wyvPXhj?W7~C?Kl{Paa#Fi*C`6eMb&Vs5*$)K08o5LvIg?4eMA$DM0Ld`t} zaq^DLmMlrVxV3dKa50+XK@=Uav+*x3x9NTSP2Ti*5y-%kMEW}F=)YNzi-@|Fb{H;> z@8$N~XN0Vorj#8+sONJ%g)2KZg|>A5cL&`WERz&;zI5EVD(2hUmf3B2N|xm|>zKHtrgb z>A$Q_9+UQ(b&VXodefx+b801lY~IeXz8zJmO2~GMxJB3lYC{|PN?z8Qgd%6P3XX!> z2f_F%B~B>txw&nNotERtjmim>rt6Ko+&QMi-K+(sEV9c)mntrhvhA@0HIo&vu#Bv1 zVyD-Ft*_%4d_sn8;V281(SRfDX3Mi%YlOj+g@~V!LAcIE1(P~hFclzsjy?@0%>Nwp z$ejm`h(g5w^L)adgkQdX4WRlLqgYaXG6!LlA}P5J5`NnBj(95S!7ya;G4>K>9yjFK z$=_BVG!esdW?)+w6(e<2409eIM-V!tujjMfc!wohu2>3(doSl(K8VPHYN=z8?2 z_O8Uu;p?lzQ=7VU4&sFl%VdFRvq1%c+RHCEwfC`~7jUG?&3h{vGP4`Dat}2P4y+gL zYoa|Oy|qx}(y~~zWynJ+NtmZ-r?|kUu$_ec_o78vx82}sbP%|DNV=!gje-ujUkXE_ zRNg4?-h%`J7Q#X$a8!C5^F#2ESXR&hjFgXsGZ~)Jdpm)~v>VXy z6!e=q!RqicAvyxbeimM-&d^a1x(uCU7?fM@Wtr(Zh%QYdj#WB6s)1Ez8$UT=gn>^~ zUO{(=6i-?)+qEaDWEvEj&My9zf5K}%4^`3QXn<7vUQcxdUG#Yv8h$xZa%CpQ9>=9Z zFQPVSen<>eVYd6gE;?=8>ht#VzQ+^047VHuh>W+oZ<{FQAsr7%o z`^#NB-8qo-{-M+}(*FXmT->bZ9&FvzT)Ld^3VxoZyDjSW%Um?gpBAWkqqT7%XuxG_ zNl8*$6@&SDo-9gqq1MB00hQy$*}$6ZOrdJQitH;#H8ue&d8Qprv=4N%Y1=uCMc=R> zd(3;nd$a?bp->7;Baq8fjY$$8ov5`Hs2(qc#cnf2dbLM2^LDU>Biv89x<;HJNoGM_ zHsd6ujUZP>DdC6PQ1Zlf#qTUU3l7K_Y4btVpkZg*5dKZq=W}W9XeJCUZ%_uJxcpFG zJBwuDYi&Vp^;fVwIZgVQOxGf>aCrnaq1gn6oEvf9c>wc$uNMVwZ|P;R!IGl`fnM(L z^PI_tv%XYR${VrD|6wNPA$dVap9fPL-!*Hm?>sTi=j*$**WXdNkVzf!sInTadTaGy zOC4|{<-3(w=dd>_HP`Vg>Y(IB)WIZi>)e&Y_8jil7T2j}{x@yEqJ~9>9n7+S)zo^B zaBj3XT74zX@Gz=Pbf}r>sl90=7mNQ|SXb>ZS^Guwewqb>to0VzLwWV}=#C)jONtyV zuVC7G-m&)k-W@q~ir9w+9IN}~f^b6spEmBRquoynEc7JNemWHXk>{hpb zMi!d6XL*md0-GD}4!v!>h?Gh!^a)^1(3~>u029J zSkRLp;PmSc+3-tl!VKmdi9au^$+ab;Ttb^Xlirw>D>80!7d{)a!V#}3edH!&rAEgL zW+sMEJ+krzuk}e|SCE#j2`qwdAWLm0g{@sWFRk4w@hX#`FzO15`~f-BzJl-oCWYN-Zr0frP@fKdvtU0mS2@p){a zrB%rD(pvyeL$xiT*efhjqVt;Lm7@*CBl|leKG8-kTEt^Q|^TH^+gCBv94{)4`5TzBsiZI2K8Ewc@?IDi?1x3HJY3dsuE;io=wlZS z0!#FF=S+!To*w7d!^hDiSs2Z;&|MF_a%F4fpX&ZlV`qTa@h3MD=O4VvJq^SdO^=QN zHQMzT2`yPDV43iOY5=b8RPNXl7_b$X4P-Xz+Yz;yU`Q2zZ$HXFCYuIbRKjeFvt&YY zHAHsn*bp5xJc&-;2l|}js*Cii^<@;@-8C2KbX%Bj??&_$YOkCy>Q+RUj)DC*1$!d_ zz|&G1ph632i&v!DC6+x3xq#>UdPgWk|HUjpCnl^-k(;_XYyp)AU>5-k5y+c?A52c9 zX;is&DR^pVmkHcmU0P$!ej%G)Vo=`z*r27~S6%t;*8;Qh7;pMG5)KqdiwL*B-QkSAn(K2bxo zKgVkw1Artfeu2wC#I{f-$#{w==nz0;!L;GR(j7|T%xSzI9NK?F+NTuI*}Yy@8mFP^ zTAk*0g?qg>=0;t7d95zX4M4uXLP)3+fEpEwCAR_m(?fKlwW_di!~)$)4M4h%LPuB; z>KfLDm&9)Lor6X7fPQY|T}T#7^cF%q5TNPUziomZ9}V~?u!??qE)GW%^ARZn%zLt* z^nko1!MOdpbBF#1*6h0xG`To=TYKhrPa>EazA&VB?U=_{V9CN%k+Et4qRYa_lkT~$ z>&-xsw`dV~05WCunne?e7KX7VB6CB_5dMfxoCJKrwfF>({>z_f2ZZn{+{n1Yc^zNe zwnvUUv~mU-o+$&d`^)%|h1z$qL00ix0FYKaL*NjJ<{6JRDNv%X4HOpr`48{A3NI7f zj(!aAT!8#G&02tw(P$1UaIIer!OXH!WR#AV1b8%zkR(UO>Tvg?7|B1@sb8w5kmqLrY*rVXg9QIy@zEJ@wt6}6`aIZroJ^>4&Etq2 zXyG-m09=g)0KWOprK+^u79HxQpJ!s8U7v1bTm3`yf>L*(7c-Tc!>i~>a{js%R6_r~ zF8uepRQMiS;dk=)cAfzbr=9@N;ujC*zyGWPaQSu7yV9W!&D3aX{%4~40Dr!6)7OQ^ zX<59kRmAPrwZTwJ%&qW0zZ?ci20}T`;JuR{zLXRV2a!dS3ynCC3)m)k9L$a50MLIp zU~7>W_+%qB2Pa%{{kSe@Wefyh@V4-QD%)*DdkNb<$flzAaOfl=emY0t( zRxJXTd&AoVw?8?Ik_2qS@hbO#>ZE#*t95C`wstGUK-9z>s0(&q#%bvI<_Nq^tSDHn z#c<(;MqHmFY;`F;qS$IQ#}lCF%mdJuU#*F&aKqGXXJXiSFc8=2^2R;3D^coEoPT35 z31^#;UIPa;2LO`43Yf6w0cZnzpxuL7mhiZd|HX51QTNi;BL3BEIHjN2Z*wpVgJ9_j z^yYJWv+uCh6sa&t zL%=}6`2~@_HEFfGolgju80@A$CASK^1ImvYLQqKlHb+=rctQjo*ab+L%~wz7?Y3Cj z_9)TFJ^(N@9;bg8aP5Je-=hufr$m6jqcg);@fXEpg7`d>_| zUIHKd!nDn zcjjy6d5AGB8q(y%Sq3R@P&2Mtc)FbtD*mgH41vM7s>7pFkM^1p7fC;Xzw?lCM~jRc zZw`!O@Akcbb8(lo_aqB&S4%BpC|WmgmmA`=*M83B=-mHy4?IS=@aH|NB{h(yjcS5f zs2J-B@rKL8OiFTmK4sQ^mkY-M1T$@O%q0WBamkSrj~LuF11!v{fks$bNyq^Bf&-Lx zoPZXn=P+tl3KL~CV)H&;_&nVz=k%`t2_R|;!4*Cj-fHMy+irslpND~3|h(vbQzZgb%*X8=Z zc3_k>0G7DDyV>X6?P$~ot#EkT-&saP`!BKb-=F{YcVPSjY?BE;s(Hz>D$!QMK{GXFEdQL|vVW z{AJaLZxny^5p>p_rWKPqsavk>fIi|3KIyPJW7|JoeO=iN{Swv6A=EgRH`z2l_8{Wf z++;7&MP#we z4+{&M8;3h9Wxc24Daj#wbt+wUi;b733uD?IFZ6}FKVWl{i@2-Fez}s;dCpKLb5pCV zP-k*FYolKL46v|90*jP{d;)^@j1(a9s5MEgG2|y(oAgf&+bdHzT9}>On2^|5R7B1 zXtL9hVUkxq@$7{&2Jgx#pg*?&$Jw3E5xhcKxhYi4a7iFq(iPCzy7qXKaL*7!f0{k!q3rVIqWt;Usax~IyVM-r zVt+=ZcYa7cocBMwcrL!E-q|1ZrX63XQK+uJ^Th$bsM`cJue^HwBH{Ab-#U`rs{Pnf z>;XAq#n4AX_3Dsg`N!lj{iMLKpMwt$SW9R~s=VviCCgC&`~5NdDQT|pBk7=uEm=ye zu|ecH=6%YM|3ppy?Xq0$d4D0+N1Agu#a$Ax(?3&<8GTeMr5Y*v@{!@j&1S&(bK?`cr3~tewnhL-1%{3@Whb$NVl=^*(4DjaQXr7+9%|F zNry&bZN&lKWlHGs0+2E>%fwtv$UI-(9X}Hhj465ehD=%Y+rej@Vz2es0n@>{ex>VO zil-|XmG7n5g;b)S@xz72k@VL0ilo-e0)nJ_FlIweyA|2;ZBlRg@7uV=Q=w+y=Z;LB zWBQC+k^`sR=KxT=x&IjhC-RnS1>OX%;00Jj)|4Hr>EtK`Sr+ zG`*+Hd~Deq{rF{tH#;=^RzoNKtb9$Ulm70g@b0oI(Z}HIROa$t=uX4O)%U21 zI&jbb*JI0v+pmwAH@SA};Jp=cCj)D_&Xto`w|=Xq5`lrf_u1dav*qfu7z3qlc^E5z zx=y&GA_AbqqNkk&%y`Wp-0O5Zxz{)veNK<$B>#Q#pizBQ!Z2fIveYT)8o_RFy1Un# zto9(ByJN;;=k>ShC#^Xq&pN62j| zA7BAMA8&~bLeK5iGjCCRHB#dVi@u!kwbzm|;!I%hYe<C18pf{1;5Bg_fcv5e7OCOhreC`x--Z5;yRuaor3*N?4yf@GSkn*MOz@}O| z8LRS}QFtMzO+Hcr8Ltc9{cFukVb{uw-93*K*9a?*Y|;wd=|_`&AzP@_z9Q8o%@YpV zT>t3+d+f848wSzhn*`w9(Ui>_kR9ng{wXwCurK0L`Q~ut%K}f~PXTp7Ssk5&`@wJk z&;Rj1nv1HXLg7L0wB`Z(TGyFHjqSEns8EgIuls<8Es6mpge|d&ebS$R-~x5@O@8m& z-`*s?l%0qo!!p#myzaB< zcGsdf2RaT=r&^e%b~tZrHY@q4dHDd$e-N=d;6pVwyU#RLGBfOy(tQmjdlUzV`WBs?a!lrJ znquG4UX|Y;OuuPlF!b)lISH_;Dr%t^Ijkub2#y0&xu<3JW){MDUuIp^rUS%#?`}?-VehW$l7}ANHV|Eh4(wY^_Gd(->*-3kX z;7>O?Id|bfcb_}^iM17nlg0jAS}_kgzRLuMe6w6$`?iUV0}+}*7}c^9-y!k*e5k+I z3P!P_+cS~lqWJ-W^!d~P!FT~Np7;7|y?YMtZl9{7{E82FtBZfdaUIYCTUwTs>&44C z8_ss?FK5o(PHVOk5Y#i;^XKw@#Qxr9CHa7?)>6B9m6(!)eI$49Pw4;X8bC}pm-;9M z*Hi%$T;r9?vae#;Unh>5hhiJyvC*`c*OX7=&z;U12c^%Iw{5pTHR6?;BL(eCJ0m(rxBYi(05{$~_uKXM3BNl1Pt5IU9gk zj=$Mx`I3LDdh9&h-lE~bJFJAnQaMm9eZT5sD8Q>H61Y31@}vzj%fdFFv$lV8`Ymz; zwGprMn+JUYq*7k@NuBPxQMh_KNG74-_Y@Abwb`E->&n=Lg|(T-J#71s5J$7Wo{Rmz zo(pJk6foPADmsz#cm_oSynESFJz#0+4_OHMhEFGtK`J!XSlnN_(k)m?E5cAuKooh@ z0xBQ_25b35_;#TGH>Oj=WB2U%jY|Rr+^QnOOs+W8%i~8Aqln?(Q?viof)L2i1GQ>_wG=h>yy+9*pR$66e7US~ G$bSPW^D1%x literal 0 HcmV?d00001 diff --git a/docs/mindspore/source_zh_cn/features/parallel/auto_parallel.rst b/docs/mindspore/source_zh_cn/features/parallel/auto_parallel.md similarity index 70% rename from docs/mindspore/source_zh_cn/features/parallel/auto_parallel.rst rename to docs/mindspore/source_zh_cn/features/parallel/auto_parallel.md index 86a2b88674..8ced6dc818 100644 --- a/docs/mindspore/source_zh_cn/features/parallel/auto_parallel.rst +++ b/docs/mindspore/source_zh_cn/features/parallel/auto_parallel.md @@ -1,21 +1,17 @@ -自动并行策略搜索 -======================== +# 自动并行策略搜索 -.. image:: https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.6.0/resource/_static/logo_source.svg - :target: https://gitee.com/mindspore/docs/blob/r2.6.0/docs/mindspore/source_zh_cn/features/parallel/auto_parallel.rst - :alt: 查看源文件 +[![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.6.0/resource/_static/logo_source.svg)](https://gitee.com/mindspore/docs/blob/r2.6.0/docs/mindspore/source_zh_cn/features/parallel/auto_parallel.md) 自动并行策略搜索模式能够让用户无需关心策略配置，自动地建立代价模型，找到训练时间较短的并行策略。当前MindSpore支持如下两种不同的自动并行策略搜索方案： - 切分策略传播算法：由配置并行策略的算子向未配置的算子传播并行策略。在传播时，算法会尽量选取引发张量重排布通信最少的策略。 - 双递归策略搜索算法：其基于符号运算的代价模型可以自由适配不同的加速器集群，对于巨大网络以及大规模多卡切分能够保证瞬间生成最优策略。 -自动并行策略搜索算法是基于算子级模型并行的策略搜索算法，要了解其原理，首先需要了解MindSpore算子级并行中的基本概念：分布式算子、张量排布和张量重排布。算子级并行属于“单程序多数据”（Single Program Multiple Data，SPMD）的一种实现，即同一段程序执行在不同数据分片上。 +自动并行策略搜索算法是基于算子级模型并行的策略搜索算法，要了解其原理，首先需要了解MindSpore算子级并行中的基本概念：分布式算子、张量排布和张量重排布。算子级并行属于"单程序多数据"（Single Program Multiple Data，SPMD）的一种实现，即同一段程序执行在不同数据分片上。 MindSpore将单机版本的程序转换成并行版本的程序。该转换是细粒度的，会将单机版本程序中每个算子替换成分布式算子，同时保证替换是数学等价的。 -分布式算子 ----------------------- +## 分布式算子 运行在多设备上的分布式算子保证了与单机版本算子计算语义的等价性。即在给定相同输入的情况下，分布式算子与单机版本算子总是得到相同的输出。 @@ -24,50 +20,44 @@ MindSpore将单机版本的程序转换成并行版本的程序。该转换是 - 情况一：如果矩阵X在4台设备上都有副本，而W按列切分4份，每台设备有一份，那么单机版本MatMul算子对应的分布式算子同样是MatMul；即每台设备上都将执行MatMul算子。 - 情况二：如果将X按照列切分4份，W按行切4份，每台机器各得到X和W的一个分片，那么单机版本MatMul算子对应的分布式算子是MatMul->AllReduce；即每台设备上都将顺序执行MatMul和AllReduce两个算子，才能保证数学等价性。 -除了“单程序”（Single Program，SP）外，“多数据”（Multiple Data，MD）也需要指定，也就是哪台设备得到数据的哪个切片。为此，我们先定义切分策略（Sharding Strategy）。 +除了"单程序"（Single Program，SP）外，"多数据"（Multiple Data，MD）也需要指定，也就是哪台设备得到数据的哪个切片。为此，我们先定义切分策略（Sharding Strategy）。 -切分策略 ----------------------- +## 切分策略 算子的切分策略是一个二维数组，表示该算子的每个输入张量中的每个维度的切片数量。这里的切分都是均匀切分。 -由切分策略，可以推导出 **张量排布** ，用以描述张量是如何分布在各个设备上的。 +由切分策略，可以推导出 **张量排布**，用以描述张量是如何分布在各个设备上的。 -张量排布 ----------------------- +## 张量排布 -给定一个算子的切分策略，能够推导出该算子的输入和输出张量的 **张量排布** 。张量排布是由 **逻辑设备矩阵** 和 **张量映射** 构成的，其中： +给定一个算子的切分策略，能够推导出该算子的输入和输出张量的 **张量排布**。张量排布是由 **逻辑设备矩阵** 和 **张量映射** 构成的，其中： - 逻辑设备矩阵是该算子的输入和输出张量共用的，为一维数组，表示设备的组织形式； - 张量映射是二维数组，表示张量的某一维切分到逻辑设备矩阵的某一维。 -同样以矩阵乘算子（MatMul）为例，其输入是两个矩阵X和W，输出为Y = MatMul(X, W)。给算子配置切分策略为[[2, 1], [1, 4]]，从而得到的张量排布和每台设备上执行的计算如下图所示。其中，X沿行均匀切分为2份，W沿列均匀切分为4份（如下图(b)）；根据切分策略，推导出逻辑设备矩阵和张量映射，如下图(c)所示；各个设备的坐标因此也确定下来了，描述了其在逻辑设备矩阵中的位置。张量在各个设备中的分布由设备的坐标决定。由下图(c)中表的‘2’列得出：设备0—设备3得到 :math:`X_0` 分片，设备4—设备7得到 :math:`X_1` 分片。由下图(c)中表的‘4’列得出：设备0和设备4得到 :math:`W_0` 分片，设备1和设备5得到 :math:`W_1` 分片，设备2和设备6得到 :math:`W_2` 分片，设备3和设备7得到 :math:`W_3` 分片。因此，各台设备上的计算也确定下来了，如下图(d)所示。 +同样以矩阵乘算子（MatMul）为例，其输入是两个矩阵X和W，输出为Y = MatMul(X, W)。给算子配置切分策略为[[2, 1], [1,4]]，从而得到的张量排布和每台设备上执行的计算如下图所示。其中，X沿行均匀切分为2份，W沿列均匀切分为4份（如下图(b)）；根据切分策略，推导出逻辑设备矩阵和张量映射，如下图(c)所示；各个设备的坐标因此也确定下来了，描述了其在逻辑设备矩阵中的位置。张量在各个设备中的分布由设备的坐标决定。由下图(c)中表的'2'列得出：设备0---设备3得到$X_0$ 分片，设备4---设备7得到 $X_1$分片。由下图(c)中表的'4'列得出：设备0和设备4得到 $W_0$分片，设备1和设备5得到 $W_1$ 分片，设备2和设备6得到 $W_2$分片，设备3和设备7得到 $W_3$分片。因此，各台设备上的计算也确定下来了，如下图(d)所示。 -.. image:: ./images/tensor_layout_zh.png +![image](./images/tensor_layout_zh.png) -对于有数据依赖的两个算子（即一个算子的输出张量被第二个算子使用），两个算子对于该数据依赖张量定义的张量排布可能不同（由于逻辑设备矩阵不同或张量映射不同），因此提出了 **张量重排布** ，用以转换不一致的排布。这里给出张量重排布的定义，省略了具体算法。 +对于有数据依赖的两个算子（即一个算子的输出张量被第二个算子使用），两个算子对于该数据依赖张量定义的张量排布可能不同（由于逻辑设备矩阵不同或张量映射不同），因此提出了**张量重排布**，用以转换不一致的排布。这里给出张量重排布的定义，省略了具体算法。 -张量重排布 ----------------------- +## 张量重排布 给定同一张量的两个不一致的张量排布，张量重排布能够将源排布转换到目的排布，同时保证转换产生的通信代价最小。这里的通信代价指的是每台设备通信的数据量。 -以两个矩阵乘算子为例：Z = MatMul(X, W), O = MatMul(Z, Y)。为了使得张量重排布起作用，两个矩阵乘算子配置了不同的切分策略，使得张量Z的排布不一致。在下图(a)中，第一个矩阵乘算子的输出张量Z是按行切分的，然而第二个矩阵乘算子要求张量Z是完整的，因此张量重排布推导出这里需要插入AllGather算子完成转换 :sup:`[1]` 。在下图(b)中，第一个矩阵乘算子的输出张量Z是按行切分的，然而第二个矩阵乘算子要求张量Z是按列切分的，故张量重排布推导出这里需要插入AllToAll算子完成转换。 +以两个矩阵乘算子为例：Z = MatMul(X, W), O = MatMul(Z, Y)。为了使得张量重排布起作用，两个矩阵乘算子配置了不同的切分策略，使得张量Z的排布不一致。在下图(a)中，第一个矩阵乘算子的输出张量Z是按行切分的，然而第二个矩阵乘算子要求张量Z是完整的，因此张量重排布推导出这里需要插入AllGather算子完成转换[1]。在下图(b)中，第一个矩阵乘算子的输出张量Z是按行切分的，然而第二个矩阵乘算子要求张量Z是按列切分的，故张量重排布推导出这里需要插入AllToAll算子完成转换。 -.. image:: ./images/tensor_redistribution_zh.png +![image](./images/tensor_redistribution_zh.png) -[1]：注：实际上需要插入AllGather算子和Concat算子。 +[1] 注：实际上需要插入AllGather算子和Concat算子。 -策略传播算法 ----------------- +## 策略传播算法 -切分策略传播算法 -~~~~~~~~~~~~~~~~ +### 切分策略传播算法 切分策略传播算法是指：用户仅需手动定义计算图中几个关键算子的策略，其余算子的策略由算法自动生成。因为关键算子的策略已被定义，所以该算法的代价模型主要描述算子之间的重排布代价（Redistribution Cost），优化目标为全图重排代价最小。主要算子策略已被定义，意味着压缩了搜索空间，这种方案的搜索时间较短，其策略性能依赖于关键算子策略的定义。因此使用切分策略传播算法需要用户具备一定的分析、定义策略的能力。 -.. note:: - 切分策略传播算法支持的硬件平台为Ascend，此外还同时支持 PyNative 模式和 Graph 模式。 +> 切分策略传播算法支持的硬件平台为Ascend，此外还同时支持 PyNative 模式和 Graph 模式。 相关接口： @@ -77,8 +67,7 @@ MindSpore将单机版本的程序转换成并行版本的程序。该转换是 总而言之，切分策略传播算法需要用户手动配置关键算子的切分策略。 -基本原理 -^^^^^^^^ +#### 基本原理 给定计算图， **切分策略传播** （Sharding Propagation）使得策略由配置的算子传播到整张计算图的所有算子。在传播过程中，策略选择的目标是最小化张量重排布产生的通信代价。 @@ -91,27 +80,24 @@ MindSpore将单机版本的程序转换成并行版本的程序。该转换是 下图所示的是切分策略传播的一个流程实例： 1. 在给定带有切分策略的计算图后，首先为没有配置切分策略的算子枚举其可行的策略，如下图(b)所示。 -2. 为每条边枚举重排布策略和相应的代价。如下图(c)所示，这里的重排布策略定义为二元组[ *s_strategy* , *t_strategy* ]，其中 *s_strategy* 表示的是源算子（下图(c)中的ReLU）的切分策略， *t_strategy* 表示的是目的算子（下图(c)中的MatMul）的切分策略。 -3. 当沿着一条边传播到下一个算子时（如图中ReLU切分策略已确定，为[2, 4]，下一步要决定MatMul算子的切分策略），在表中选择引起通信代价最小的目的算子策略（即为MatMul选择[[2, 4], [4, 1]]）。最后，所有算子的切分策略都被确定，如下图(d)所示。 +2. 为每条边枚举重排布策略和相应的代价。如下图(c)所示，这里的重排布策略定义为二元组[*s_strategy* , *t_strategy* ]，其中 *s_strategy*表示的是源算子（下图(c)中的ReLU）的切分策略， *t_strategy*表示的是目的算子（下图(c)中的MatMul）的切分策略。 +3. 当沿着一条边传播到下一个算子时（如图中ReLU切分策略已确定，为[2, 4]，下一步要决定MatMul算子的切分策略），在表中选择引起通信代价最小的目的算子策略（即为MatMul选择[[2,4], [4, 1]]）。最后，所有算子的切分策略都被确定，如下图(d)所示。 -.. image:: ./images/sharding_propagation_zh.png - :alt: 切分策略传播的流程实例 - :target: ./images/sharding_propagation_zh.png +[![切分策略传播的流程实例](./images/sharding_propagation_zh.png)](./images/sharding_propagation_zh.png) -双递归策略搜索算法 -~~~~~~~~~~~~~~~~~~~~ +### 双递归策略搜索算法 双递归策略搜索算法（Symbolic Automatic Parallel Planner，简称SAPP）基于符号化自动策略生成。SAPP算法能够对于巨大网络以及大规模切分瞬间生成最优策略。SAPP基于并行原理建模，通过建立抽象机来描述硬件集群拓扑，并通过符号化简优化代价模型。其代价模型比较的不是预估的绝对时延，而是不同并行策略的相对代价，因此能够大大压缩搜索空间，在百卡集群上能够保证分钟级的搜索时间。 -.. note:: - 双递归策略搜索算法支持的硬件平台为Ascend，需要在 Graph 模式下运行。 +> 双递归策略搜索算法支持的硬件平台为Ascend，需要在 Graph 模式下运行。 相关接口： + ``mindspore.parallel.auto_parallel.AutoParallel(net, parallel_mode="recursive_programming")`` + 除了以上 ``AutoParallel`` 接口，双递归策略搜索算法无需额外配置。 -基本原理 -^^^^^^^^^ +#### 基本原理 双递归策略搜索算法是一种全自动的算子级策略搜索方案，用户无需对模型进行任何配置，算法可以自动搜索出通信代价最小的并行策略。 -- Gitee