From 3e47c235dfe2b23e7e657cd4092fcaf9ee442a69 Mon Sep 17 00:00:00 2001 From: whytao <1396718917@qq.com> Date: Fri, 5 Jan 2024 10:39:25 +0800 Subject: [PATCH 1/4] =?UTF-8?q?=E5=BC=95=E7=94=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- app/en/blogs/RDMA/RDMA_Network_Guide.md | 469 ++++++++++++++++++ .../figures/en-us_image_0000001745579744.png | Bin 0 -> 21367 bytes .../figures/en-us_image_0000001745579748.png | Bin 0 -> 27063 bytes .../figures/en-us_image_0000001745579752.png | Bin 0 -> 23246 bytes .../figures/en-us_image_0000001745738904.png | Bin 0 -> 57358 bytes .../figures/en-us_image_0000001745738908.png | Bin 0 -> 32459 bytes .../figures/en-us_image_0000001745738916.png | Bin 0 -> 9099 bytes .../figures/en-us_image_0000001745738920.png | Bin 0 -> 82292 bytes .../figures/en-us_image_0000001792578593.png | Bin 0 -> 22315 bytes .../figures/en-us_image_0000001792578597.png | Bin 0 -> 21411 bytes .../figures/en-us_image_0000001792578601.png | Bin 0 -> 11952 bytes .../figures/en-us_image_0000001792658853.png | Bin 0 -> 23074 bytes .../figures/en-us_image_0000001792658857.png | Bin 0 -> 6575 bytes .../figures/en-us_image_0000001792658861.png | Bin 0 -> 11826 bytes .../public_sys-resources/icon-caution.gif | Bin 0 -> 2251 bytes .../RDMA/public_sys-resources/icon-danger.gif | Bin 0 -> 2251 bytes .../RDMA/public_sys-resources/icon-note.gif | Bin 0 -> 2052 bytes .../RDMA/public_sys-resources/icon-notice.gif | Bin 0 -> 2178 bytes .../RDMA/public_sys-resources/icon-tip.gif | Bin 0 -> 1305 bytes .../public_sys-resources/icon-warning.gif | Bin 0 -> 2251 bytes ...eak-modules_Script_for_OS_Compatibility.md | 173 +++++++ app/en/blogs/weak-modules/figures/weak.png | Bin 0 -> 133551 bytes .../public_sys-resources/icon-caution.gif | Bin 0 -> 2251 bytes .../public_sys-resources/icon-danger.gif | Bin 0 -> 2251 bytes .../public_sys-resources/icon-note.gif | Bin 0 -> 2052 bytes .../public_sys-resources/icon-notice.gif | Bin 0 -> 2178 bytes .../public_sys-resources/icon-tip.gif | Bin 0 -> 1305 bytes .../public_sys-resources/icon-warning.gif | Bin 0 -> 2251 bytes ...56\346\224\271\346\226\271\346\263\225.md" | 173 +++++++ app/zh/blogs/weak-modules/figures/weak.png | Bin 0 -> 133551 bytes .../public_sys-resources/icon-caution.gif | Bin 0 -> 2251 bytes .../public_sys-resources/icon-danger.gif | Bin 0 -> 2251 bytes .../public_sys-resources/icon-note.gif | Bin 0 -> 2052 bytes .../public_sys-resources/icon-notice.gif | Bin 0 -> 2178 bytes .../public_sys-resources/icon-tip.gif | Bin 0 -> 1305 bytes .../public_sys-resources/icon-warning.gif | Bin 0 -> 2251 bytes 36 files changed, 815 insertions(+) create mode 100644 app/en/blogs/RDMA/RDMA_Network_Guide.md create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001745579744.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001745579748.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001745579752.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001745738904.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001745738908.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001745738916.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001745738920.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001792578593.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001792578597.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001792578601.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001792658853.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001792658857.png create mode 100644 app/en/blogs/RDMA/figures/en-us_image_0000001792658861.png create mode 100644 app/en/blogs/RDMA/public_sys-resources/icon-caution.gif create mode 100644 app/en/blogs/RDMA/public_sys-resources/icon-danger.gif create mode 100644 app/en/blogs/RDMA/public_sys-resources/icon-note.gif create mode 100644 app/en/blogs/RDMA/public_sys-resources/icon-notice.gif create mode 100644 app/en/blogs/RDMA/public_sys-resources/icon-tip.gif create mode 100644 app/en/blogs/RDMA/public_sys-resources/icon-warning.gif create mode 100644 app/en/blogs/weak-modules/Modification_of_the_weak-modules_Script_for_OS_Compatibility.md create mode 100644 app/en/blogs/weak-modules/figures/weak.png create mode 100644 app/en/blogs/weak-modules/public_sys-resources/icon-caution.gif create mode 100644 app/en/blogs/weak-modules/public_sys-resources/icon-danger.gif create mode 100644 app/en/blogs/weak-modules/public_sys-resources/icon-note.gif create mode 100644 app/en/blogs/weak-modules/public_sys-resources/icon-notice.gif create mode 100644 app/en/blogs/weak-modules/public_sys-resources/icon-tip.gif create mode 100644 app/en/blogs/weak-modules/public_sys-resources/icon-warning.gif create mode 100644 "app/zh/blogs/weak-modules/OS\345\205\274\345\256\271\346\200\247weak-modules\350\204\232\346\234\254\344\277\256\346\224\271\346\226\271\346\263\225.md" create mode 100644 app/zh/blogs/weak-modules/figures/weak.png create mode 100644 app/zh/blogs/weak-modules/public_sys-resources/icon-caution.gif create mode 100644 app/zh/blogs/weak-modules/public_sys-resources/icon-danger.gif create mode 100644 app/zh/blogs/weak-modules/public_sys-resources/icon-note.gif create mode 100644 app/zh/blogs/weak-modules/public_sys-resources/icon-notice.gif create mode 100644 app/zh/blogs/weak-modules/public_sys-resources/icon-tip.gif create mode 100644 app/zh/blogs/weak-modules/public_sys-resources/icon-warning.gif diff --git a/app/en/blogs/RDMA/RDMA_Network_Guide.md b/app/en/blogs/RDMA/RDMA_Network_Guide.md new file mode 100644 index 00000000..e4be4866 --- /dev/null +++ b/app/en/blogs/RDMA/RDMA_Network_Guide.md @@ -0,0 +1,469 @@ +--- +title: "RDMA_Network_Guide" +date: '2023-12-27' +category: 'blog' +tags: ['openGauss'] +archives: '2023-11' +author: 'why涛' +summary: "RDMA_Network_Guide" +--- + +# Identifying CX4/CX5 NICs + +Run the following command: + +``` +lspci |grep Mellanox +``` + +Command output: + +``` +81:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] +81:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] +``` + +# Installing the MLNX Driver + +1. Download the driver package that matches the OS from [https://network.nvidia.com/products/infiniband-drivers/linux/mlnx\_ofed/](https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed/). + +  + +2. Create a directory and mount the OS image file to this directory. Change the OS image name to the actual one. + + ``` + mkdir -p /mnt/iso + mount openEuler-22.03-LTS-x86_64-dvd.iso /mnt/iso + ``` + +3. Configure the OS image source, for example, the local image, to obtain dependencies required during the installation. + 1. Open the image source file. + + ``` + vim /etc/yum.repos.d/openEuler.repo + ``` + + 2. Press **i** to enter the insert mode and retain only the following content: + + ``` + [OS] + name=OS + baseurl=file:///mnt/iso + enabled=1 + gpgcheck=0 + ``` + + 3. Press **Esc**, type **:wq!**, and press **Enter** to save the file and exit. + 4. Cache the software package. + + ``` + yum makecache + ``` + +4. Upload the driver package to the server and decompress it. Change the driver package name to the actual one. + + ``` + tar -zxvf MLNX_OFED_LINUX-5.4-3.7.5.0-openeuler22.03-x86_64.tgz + ``` + +5. Go to the driver package directory extracted after the decompression and run the following command to install the driver: + + ``` + ./mlnxofedinstall --without-depcheck --without-fw-update --force + ``` + + If the system displays a message indicating that the kernel does not support the driver version, run the following command: + + ``` + ./mlnxofedinstall --add-kernel-support + ``` + +6. Configure the system to automatically start the driver upon system restart. + + ``` + chkconfig --add openibd + /etc/init.d/openibd start + chkconfig openibd on + ``` + +7. Reboot the server after the installation is complete. + +# Verifying the Installation + +1. Check the RoCE LAG function of the driver. + 1. Check whether the RoCE LAG function is enabled. + + ``` + find /sys/ -name roce_lag_enable | xargs cat + ``` + + - If the command output is **1**, the function is enabled. + - If the command output is **0** or no command output is displayed, the function is disabled. + - The function is expected to be disabled. If the function is enabled, go to [1.b](#li519083722516). + + 2. Disable the RoCE LAG function. + + ``` + sed '/load_module mlx5_core/a\ files=`find /sys -name roce_lag_enable`;for file in $files;do echo 0 > $file;done' -i /etc/init.d/openibd + ``` + + 3. Reboot the node to apply the modification. Then, perform [1.a](#li389014811257) again to check whether the modification takes effect. + + ``` + reboot + ``` + +2. Query the driver version. + + ``` + ofed_info -s + ``` + + If the queried driver version is the same as the version installed in [Installing the MLNX Driver](installing-the-mlnx-driver.md), the driver version is correct. + +3. Load the MST tool. + + ``` + mst start + ``` + + If the following information is displayed, the loading is successful. + + ``` + Starting MST (Mellanox Software Tools) driver set + Loading MST PCI module - Success + Loading MST PCI configuration module - Success + Create devices + Unloading MST PCI module (unused) - Success + ``` + +4. Query the device path and network port. + 1. Query the device paths of RoCE and IB cards. + + ``` + mst status + ``` + + Command output: + + ``` + MST modules: + ------------ + MST PCI module is not loaded + MST PCI configuration module loaded + + MST devices: + ------------ + /dev/mst/mt4119_pciconf0 - PCI configuration cycles access. + domain:bus:dev.fn=0000:81:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 + Chip revision is: 00 + ``` + + A device path **/dev/mst/**_mst\_typeN_ \(_N_ can be 0, 1, 2, ...\) enumerated in the **MST devices** field indicates a CX card. For details about the mapping between **mst\_type** and CX NIC models, see [Table 1](#table93856218287). + + **Table 1** Mapping between mst\_type and CX NIC models + + +
+ | ++ | +
---|---|
+ | ++ | +
+ | ++ | +
+ | ++ | +
+ | ++ | +
4Dx-1Z+CHH_^gDtvg1(o
zJ9}9aW>GAR#wp+uMtF`PX>~}@Q??bFdA(C><(^!#j`?)o*5t#AO}F;4vtyZjKSQ{O
zXHi0^ctbYHh^vBv6wTn>J2%uY_(Nz9s$!^OazFot6>=2Py}6JxMNvg9D4b~o-vHVV
zJSl_av#+N(TVzk3Nx_OrTJf<4)qYv$bpxtGD?YCA4cv+ag^F>29){RQ2|f=U6FHZZ
znXI4n87l);*P%_Ms1!5XIIGNgOKdVDhLcsq{so{;**MNj`PDqT&M-)o-r_NhD|wC&
zp}^H!Ue~3W!y?`;-LEtSfyZB-*QKJv-NbOE&uApWd7-s4;rv>vXq@(a4^F@4=w0dN
znBMTFaE;cpx8M-^UfFSx21ioqp7m?Y__z-LxT|;EQH&E65T;Kix^tmTn?FvKU##Yd
zit;bu;d06Pg$6Yd47oIJfBC*M+4$wQDO2ATJKSZcN^a%=
zjO;nYdoYW-0K*AZaqCk~CSUL6CQ3)8FRRgohDo1d@v_wbSZ}I2Ia^hacobba?cO-l
z6g~7~w%U9+A^{0md6k7pNkT~}F_WBF*HN*w>epsh}A69x8QEZsT
z6>rugD=21+`n2>p;Fv}*BB%<1>}U3}olkCUEDnw%AwRw5rIOIDxq;mVAVO+jc!#2Q
z@-(B+rxcD)MQk4>YJ5&((p3sd(BKEpBAz*1PbRAScMm@^)`B7LS)jet5+2---QpYy
ze)!0fsY}W0pz_n#yXNn$no(%Em6FJ_ayE?R;wpDg7AR_3)NsWhr9Wr$d<#0>&i6HJ
zCyP~6%rPLO{KssBV~o3gIqJua
zlqjCh((Kq^+ixZ2R*h~8p=R=maI5`IX5%sWfC_~hLxkh?-+9RQF%I@RY5o?lT%H@o
z;4oQmNWGit`_Ye1AMC5$zk!12ydBI$v=~bExuAY)HD8kAFDf|G4G