From 0a861ca194892a32aaf8e5d13dbbedcc36b4821d Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Wed, 11 Jan 2023 11:52:24 +0000 Subject: [PATCH 01/99] update README.md. Signed-off-by: MacChen1 --- README.md | 93 ++++++++++++++++--------------------------------------- 1 file changed, 27 insertions(+), 66 deletions(-) diff --git a/README.md b/README.md index 9500aae..c591be7 100644 --- a/README.md +++ b/README.md @@ -3,11 +3,11 @@ ## 概述 数据已经成为土地、劳动力、资本、技术之后的第五生产要素。在这种情况下，openEuler社区需要从技术角度考虑如何更充分的发挥数据在生产、生活中的作用，要构建大数据处理相关能力，由本SIG负责。 -## sig-BigData愿景 +## 愿景 构建和完善openEuler社区下的大数据生态，打造活跃的大数据交流平台，丰富和提升大数据组件功能和性能， 发掘用户大数据需求，孵化有潜力的大数据组件。 -## 该SIG的业务范围 +## 业务范围 - openEuler上的大数据基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 - openEuler上大数据平台，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 - openEuler上的大数据组件和平台的性能优化，让大数据在openEuler上更好用。 @@ -31,70 +31,31 @@ - sinever -## repository地址： - -- https://gitee.com/src-openeuler/jupyter -- https://gitee.com/src-openeuler/hadoop -- https://gitee.com/src-openeuler/libhdfs -- https://gitee.com/src-openeuler/gl2ps -- https://gitee.com/src-openeuler/lzip -- https://gitee.com/src-openeuler/zookeeper -- https://gitee.com/src-openeuler/kafka -- https://gitee.com/src-openeuler/zeppelin -- https://gitee.com/src-openeuler/ibis -- https://gitee.com/src-openeuler/presto -- https://gitee.com/src-openeuler/rain -- https://gitee.com/src-openeuler/ignite -- https://gitee.com/src-openeuler/bigtop -- https://gitee.com/src-openeuler/flume -- https://gitee.com/src-openeuler/notebook -- https://gitee.com/src-openeuler/datafu -- https://gitee.com/src-openeuler/epstool -- https://gitee.com/src-openeuler/flink -- https://gitee.com/src-openeuler/flume -- https://gitee.com/src-openeuler/ftgl -- https://gitee.com/src-openeuler/giraph -- https://gitee.com/src-openeuler/gl2ps -- https://gitee.com/src-openeuler/hadoop-3.1 -- https://gitee.com/src-openeuler/hadoop -- https://gitee.com/src-openeuler/hbase -- https://gitee.com/src-openeuler/hive -- https://gitee.com/src-openeuler/hue -- https://gitee.com/src-openeuler/ibis -- https://gitee.com/src-openeuler/ignite -- https://gitee.com/src-openeuler/impala -- https://gitee.com/src-openeuler/jupyter -- https://gitee.com/src-openeuler/kite-sdk -- https://gitee.com/src-openeuler/knox -- https://gitee.com/src-openeuler/kudu -- https://gitee.com/src-openeuler/kafka -- https://gitee.com/src-openeuler/libhdfs -- https://gitee.com/src-openeuler/libiodbc -- https://gitee.com/src-openeuler/llama -- https://gitee.com/src-openeuler/lzip -- https://gitee.com/src-openeuler/mahout -- https://gitee.com/src-openeuler/notebook -- https://gitee.com/src-openeuler/oozie -- https://gitee.com/src-openeuler/parquet-format -- https://gitee.com/src-openeuler/phantomjs -- https://gitee.com/src-openeuler/presto -- https://gitee.com/src-openeuler/protobuf2 -- https://gitee.com/src-openeuler/qfs -- https://gitee.com/src-openeuler/qrupdate -- https://gitee.com/src-openeuler/qscintilla -- https://gitee.com/src-openeuler/rain -- https://gitee.com/src-openeuler/redis5 -- https://gitee.com/src-openeuler/redis6 -- https://gitee.com/src-openeuler/spark -- https://gitee.com/src-openeuler/storm -- https://gitee.com/src-openeuler/telegraf -- https://gitee.com/src-openeuler/ycsb -- https://gitee.com/src-openeuler/zeppelin -- https://gitee.com/src-openeuler/zookeeper -- https://gitee.com/src-openeuler/delta -- https://gitee.com/src-openeuler/iceberg -- https://gitee.com/src-openeuler/hudi -- https://gitee.com/src-openeuler/apache-atlas +# 项目清单： +| 项目 | 版本 | 状态 | 类别 | +| ---- | ---- | ---- | ---- | +| [hadoop](https://gitee.com/src-openeuler/hadoop) | 2. | 已落版本 | 说明 | +| [hadoop-3.1](https://gitee.com/src-openeuler/hadoop-3.1) | 3.1 | 已落版本 | 说明 | +| [hbase](https://gitee.com/src-openeuler/hbase) | 3.1 | 已落版本 | 说明 | +| [hive](https://gitee.com/src-openeuler/hive) | 3.1 | 已落版本 | 说明 | +| [spark](https://gitee.com/src-openeuler/spark) | 3.1 | 已落版本 | 说明 | +| [flink](https://gitee.com/src-openeuler/flink) | 3.1 | 已落版本 | 说明 | +| [libhdfs](https://gitee.com/src-openeuler/libhdfs) | 3.1 | 已落版本 | 说明 | +| [zookeeper](https://gitee.com/src-openeuler/zookeeper) | 3.1 | 已落版本 | 说明 | +| [kafka](https://gitee.com/src-openeuler/kafka) | 3.1 | 已落版本 | 说明 | +| [zeppelin](https://gitee.com/src-openeuler/zeppelin) | 3.1 | 已落版本 | 说明 | +| [presto](https://gitee.com/src-openeuler/presto) | 3.1 | 已落版本 | 说明 | +| [rain](https://gitee.com/src-openeuler/rain) | 3.1 | 已落版本 | 说明 | +| [ignite](https://gitee.com/src-openeuler/ignite) | 3.1 | 已落版本 | 说明 | +| [bigtop](https://gitee.com/src-openeuler/bigtop) | 3.1 | 引入中 | 说明 | +| [flume](https://gitee.com/src-openeuler/flume) | 3.1 | 已落版本 | 说明 | +| [storm](https://gitee.com/src-openeuler/storm) | 3.1 | 已落版本 | 说明 | +| [delta](https://gitee.com/src-openeuler/delta) | 3.1 | 已落版本 | 说明 | +| [iceberg](https://gitee.com/src-openeuler/iceberg) | 3.1 | 已落版本 | 说明 | +| [hudi](https://gitee.com/src-openeuler/hudi) | 3.1 | 已落版本 | 说明 | +| [hue](https://gitee.com/src-openeuler/hue) | 3.1 | 已落版本 | 说明 | +| [oozie](https://gitee.com/src-openeuler/oozie) | 3.1 | 已落版本 | 说明 | +| [ycsb](https://gitee.com/src-openeuler/ycsb) | 3.1 | 已落版本 | 说明 | ## 路线图 -- Gitee From dce272998a41e8d09fd1ae48ab9946b13fc2a476 Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Wed, 11 Jan 2023 11:55:15 +0000 Subject: [PATCH 02/99] update README.md. Signed-off-by: MacChen1 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index c591be7..1dc8d0b 100644 --- a/README.md +++ b/README.md @@ -61,7 +61,7 @@ ## 路线图 ## 联系方式 -![输入图片说明](image.png) -## [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) +### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) bigdata@openeuler.org +### [微信公众号]：openEuler-bigdata-sig -- Gitee From f66a029bf20c99e921609d1882a46a7fb193ac57 Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Wed, 11 Jan 2023 11:57:15 +0000 Subject: [PATCH 03/99] update README.md. Signed-off-by: MacChen1 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1dc8d0b..47bdcf8 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,7 @@ - 两周一次 + 按需出发，请订阅邮件列表获取具体时间和会议链接 - 线上 +- 订阅邮箱：bigdata@openeuler.org ## 成员 @@ -62,6 +63,5 @@ ## 联系方式 ### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) -bigdata@openeuler.org -### [微信公众号]：openEuler-bigdata-sig +### 微信公众号：openEuler-bigdata-sig -- Gitee From 7ffd5b050ffb8b70846b93d8bfdfb7348cdd933e Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Wed, 11 Jan 2023 12:00:06 +0000 Subject: [PATCH 04/99] update README.md. Signed-off-by: MacChen1 --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 47bdcf8..dd89abe 100644 --- a/README.md +++ b/README.md @@ -58,10 +58,12 @@ | [oozie](https://gitee.com/src-openeuler/oozie) | 3.1 | 已落版本 | 说明 | | [ycsb](https://gitee.com/src-openeuler/ycsb) | 3.1 | 已落版本 | 说明 | - -## 路线图 +# 创新项目 +## VectorBlas(https://gitee.com/openeuler/vectorBlas) ## 联系方式 ### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) ### 微信公众号：openEuler-bigdata-sig +# 路标： + -- Gitee From aa4db1c68339bcd7afde085f7dfa614e573906de Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Wed, 11 Jan 2023 12:02:19 +0000 Subject: [PATCH 05/99] update README.md. Signed-off-by: MacChen1 --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index dd89abe..3dcd444 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,6 @@ | [presto](https://gitee.com/src-openeuler/presto) | 3.1 | 已落版本 | 说明 | | [rain](https://gitee.com/src-openeuler/rain) | 3.1 | 已落版本 | 说明 | | [ignite](https://gitee.com/src-openeuler/ignite) | 3.1 | 已落版本 | 说明 | -| [bigtop](https://gitee.com/src-openeuler/bigtop) | 3.1 | 引入中 | 说明 | | [flume](https://gitee.com/src-openeuler/flume) | 3.1 | 已落版本 | 说明 | | [storm](https://gitee.com/src-openeuler/storm) | 3.1 | 已落版本 | 说明 | | [delta](https://gitee.com/src-openeuler/delta) | 3.1 | 已落版本 | 说明 | @@ -57,9 +56,12 @@ | [hue](https://gitee.com/src-openeuler/hue) | 3.1 | 已落版本 | 说明 | | [oozie](https://gitee.com/src-openeuler/oozie) | 3.1 | 已落版本 | 说明 | | [ycsb](https://gitee.com/src-openeuler/ycsb) | 3.1 | 已落版本 | 说明 | +| [bigtop](https://gitee.com/src-openeuler/bigtop) | 3.1 | 计划引入中 | 说明 | # 创新项目 -## VectorBlas(https://gitee.com/openeuler/vectorBlas) +| 项目 | 版本 | 状态 | 类别 | +| ---- | ---- | ---- | ---- | +| [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | ## 联系方式 ### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) -- Gitee From b504847082b32586d9a09cebf474ea8a158c9b26 Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Wed, 11 Jan 2023 12:03:41 +0000 Subject: [PATCH 06/99] update README.md. Signed-off-by: MacChen1 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3dcd444..e004958 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ - sinever -# 项目清单： +# 项目清单 | 项目 | 版本 | 状态 | 类别 | | ---- | ---- | ---- | ---- | | [hadoop](https://gitee.com/src-openeuler/hadoop) | 2. | 已落版本 | 说明 | @@ -67,5 +67,5 @@ ### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) ### 微信公众号：openEuler-bigdata-sig -# 路标： +# 路标 -- Gitee From ba6e2dcc79d4c560c87325f8535ece9d6fda7da7 Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Wed, 11 Jan 2023 12:08:44 +0000 Subject: [PATCH 07/99] update README.md. Signed-off-by: MacChen1 --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index e004958..212bc7c 100644 --- a/README.md +++ b/README.md @@ -22,11 +22,11 @@ ## 成员 ### Maintainer列表 - - sinever + - [sinever](https://gitee.com/sinever) - njlzk - - yangzhao_kl - - wuzeyi1 - - unioah + - [yangzhao_kl](https://gitee.com/yangzhao_kl) + - [wuzeyi1](https://gitee.com/wuzeyi1) + - [macchen1](https://gitee.com/macchen1) ### Committer列表 @@ -35,7 +35,7 @@ # 项目清单 | 项目 | 版本 | 状态 | 类别 | | ---- | ---- | ---- | ---- | -| [hadoop](https://gitee.com/src-openeuler/hadoop) | 2. | 已落版本 | 说明 | +| [hadoop](https://gitee.com/src-openeuler/hadoop) | 2.3 | 已落版本 | 说明 | | [hadoop-3.1](https://gitee.com/src-openeuler/hadoop-3.1) | 3.1 | 已落版本 | 说明 | | [hbase](https://gitee.com/src-openeuler/hbase) | 3.1 | 已落版本 | 说明 | | [hive](https://gitee.com/src-openeuler/hive) | 3.1 | 已落版本 | 说明 | -- Gitee From c890a464a5d2578af7b55acd711d61986ec9fe1c Mon Sep 17 00:00:00 2001 From: xiexing Date: Thu, 9 Mar 2023 08:23:21 +0000 Subject: [PATCH 08/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20Docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Docs/.keep | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 Docs/.keep diff --git a/Docs/.keep b/Docs/.keep new file mode 100644 index 0000000..e69de29 -- Gitee From 190bf5baceccb6eed3d251c5395df787c3ce125e Mon Sep 17 00:00:00 2001 From: xiexing Date: Thu, 9 Mar 2023 08:23:53 +0000 Subject: [PATCH 09/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20=E7=A7=BB=E6=A4=8D?= =?UTF-8?q?=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" new file mode 100644 index 0000000..e69de29 -- Gitee From 9d3098f0023f9739d4bf89a225dd6871561c7c9b Mon Sep 17 00:00:00 2001 From: xiexing Date: Thu, 9 Mar 2023 08:24:18 +0000 Subject: [PATCH 10/99] =?UTF-8?q?add=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" new file mode 100644 index 0000000..e69de29 -- Gitee From f13c14ab2c6449b5bf4ba3ff6e7acd8cf35dc0d0 Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 01:21:57 +0000 Subject: [PATCH 11/99] update README.md. Signed-off-by: xiexing --- README.md | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 212bc7c..3c2183c 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@ # bigdata -## 概述 -数据已经成为土地、劳动力、资本、技术之后的第五生产要素。在这种情况下，openEuler社区需要从技术角度考虑如何更充分的发挥数据在生产、生活中的作用，要构建大数据处理相关能力，由本SIG负责。 ## 愿景 构建和完善openEuler社区下的大数据生态，打造活跃的大数据交流平台，丰富和提升大数据组件功能和性能， @@ -15,18 +13,20 @@ ## 会议 -- 两周一次 + 按需出发，请订阅邮件列表获取具体时间和会议链接 -- 线上 +- 会议主题： OpenEuler bigdata meetup +- 会议时间： 每周四 16:00-17:00(GMT+08:00) 中国标准时间-北京 每周一次 + 按需出发， +- 点击链接入会，或添加至会议列表：xxxxxxxx +- 会议 ID：xxxxxx +- 手机号一键拨号入会 - 订阅邮箱：bigdata@openeuler.org +- 复制该信息，打开手机腾讯会议v2.13以上版本即可参与 ## 成员 ### Maintainer列表 - - [sinever](https://gitee.com/sinever) - - njlzk - - [yangzhao_kl](https://gitee.com/yangzhao_kl) - - [wuzeyi1](https://gitee.com/wuzeyi1) - - [macchen1](https://gitee.com/macchen1) + - 杨昭 [yangzhao_kl](https://gitee.com/yangzhao_kl) email: yangzhao1@kylinos.cn + - 吴泽艺 [wuzeyi1](https://gitee.com/wuzeyi1) email: zaki.wu@huawei.com + - 陈强 [macchen1](https://gitee.com/macchen1) email: mac.chenqiang@huawei.com ### Committer列表 @@ -62,6 +62,15 @@ | 项目 | 版本 | 状态 | 类别 | | ---- | ---- | ---- | ---- | | [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | +| [openEuler开源实习](https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata)||| + +## 链接 +openEuler SDS sig组简介: + +邮件列表归档地址： + +openEuler bigdata 月刊归档地址： + ## 联系方式 ### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) -- Gitee From 09269530be5f12805888ce67b8c411215af8ce54 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Fri, 10 Mar 2023 01:42:27 +0000 Subject: [PATCH 12/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../1.jpg" | Bin 0 -> 27873 bytes .../hadoop.md" | 194 ++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/1.jpg" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/1.jpg" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/1.jpg" new file mode 100644 index 0000000000000000000000000000000000000000..ec877fbfab90775e85180cc2194e88fce4f0fcff GIT binary patch literal 27873 zcmd42cT|(zw=NnGM5RgZpwg78bg2r`L_m585S1DNgkGdXQ9!zYfPjK@=^_ce_ugwl z4NZCi0s%re`S#xD+_U$&pL)$G+W;P!J3Z4Cek z2?^j8@dda-0n`B3NJ;;B6E8C2oBRejIT;!GEeeY3H>hq=QBmHaq@<>yzfDa;M?*<@ zoB1{!10xd?6BR8BD>EZ2JtGt2KMx_fM*I&M`Au^2n~cy8%!l3%^Ek z4?s#sa*dAUsuREi0FaOqNBhr!|KlPdC64hr#f_V{D2W{qw*jOi*RGM0UHfO$#NNTg z-vMNFhNhPGb0cFDQ!{f5OItg82S+EcvzNDzub+Rw+YcYZ!XrLKMkOYFNlr=q znwFlEo0nfu__L_^cNH8_T~k|E-`>&L)!ozE*FQcnIW;{qJBM6Budc0cZ2sBW#vUI1 zJw7?bot^(97YX3nzl-(1B>OLN(Gle$B_q2=M)8kaB&2@Cc8!jV{O+Uc^iK^aUVAXy zlX!oFQ8gjEvh60Xq#=gsjpx`cW5#A$(eQ zXKjavMA30&iF_7UVgjF#8sd5R+ghV{5mJ47-WpZ?5QCCJgWn#~-Q--OGOg(>3w(;D zXoG#km_tsIb}YGKH?n>@Evom=tB!-_=lL=QRhd;21s6V71NSjny|^+geMk2ZPQY4* z@>7NyW+5&?t|9i}QCox8eX2RNT3b26Z}0U?H&VYL4-HNc`09NwGf0T6WeChYQwVo+ zwSo+1j*BoKGeqgew2Lu`mlf1iNs9dm8>-iu03L8DPQF-sJJG0%q38*r#q`b~g4uuf42JME4FhWM3HHyqkjZ?^awV_MFKJQo|iBH#zs9WvYIlAU$1Kx9zHfHF}I{ zKAM9w+<5E0s7dsXc+{JJx$lI&^)Zz7{R>(HC*s{;QR|}JhgD7+* zJ}gMEOa}2|=NfP#w<&SF2RN~j`G9-FoLSB`YJG7F=36)v`8;&6ket0f7ILCSoU)~8 zvDZ__TxK9lSd|q-X6Hw5y^t>RgqL0V=hV&GNF8Q<(Wi9HGje1{>6=~9TRS#1I8eX_ zgqa|LQOfhBEVv96oH(%6Idd~r^E!FY=UV;d)|rSz?gm@$$x;-E9i9QYg^M;D6ZzY6 z`%gWh8UZa>XhWkQzy>*=NX5DAx^Z*qjv=xhXnJjI^3y&J3upag?ViTHD?qil=JXXn zHh4q676gZ00ctK6flKPH%{I7b+XBr3e|>`qct$W6LR6_Rb8%WbQwbel#50d*WG3h7`vA_x>m-(x7U?gpTJC;(q zE}x^l1Hr43Qtt;dI(VtqTxVOCJteJaBb9lbSU%%2UQWE3j6TI;rZi#Y_OlglV5~`eitV+@M_Y7_DDl^{JBHO2? zw{`5VX}Iy~)aLv9Wqr}*gl;_JC>&&|D09L;!BT`bn*Frism?+cm4bSIkJu_2MEjdx z0ZLK#J%?}pxg{n!iLQ@@p(DZyFcqKRL-+b221O_}>ZadL)4uI9b_sQ6wHlIoH`If3oYPL8@Az_MqVuk2no zg)hX=P~3wbLnE14kei*te^%;IXaM<-1eOQUU)Q-Ehgy*zJu*Fx%Zf|6H2DP~by8Ds ztD<~}&s)pw!8%1QbKS|`p2m9+VQIbRmOwY}a(Hvxh2%9eoGV(CQkyW|^vk>~b>N7} z?Owp^qw7XPk>-towcW>Ty+4l+BF>k>T|4Vzn6cD596emS^nH{Mdo2t&;m?Sc%98K1 z$d8L$xxgvv!wP$fVJ`j&Z-6C%A7fu3IwScqt=hj3_1+VwSB@zaz=-LH!922C~oMs>mne+g)>cv@ey_V;vv@S=d%}vZ+l@1JA_UEL`;SO`}$;5;2YfBqv@4u{! za9ga=1e9w`Z|&}9*T&pUWU*k>X78<|;1y~O=kB~6IKwR_WfBlbeRl%5g$o|B*-VRz zz0prg1$U1Fdvnj zDW}b|ozU)!vQ~*!1ZRU)NC=Eh=^9N1JDK;!rDJu>8>@gf?W0EZszZ#8LvxM>v(`vYu#ZE?ZspL3#D2v6>HE zmM?xuPtF)IJwrVZYk3nfXremA+tRD0VYwFMib1UgsZdTe>IN1q;3PXnf%DZ#K4>jB zjn9d@T7I)YH+{F5XMrR{)4s2tV!xF7WBNh=>dSU?lWT zZK5z`a`MnN`Z37em@dclb>buXmPW!5VSnJjRcw04GuHuKe~iz#K%D(mw#629G(p>y znJ}@7xp__})9J05I-(%Gsg*|AAFQW*i`Jb9onue~c&Ur*_cPwOT=dfJL^@527ovP9cYg|k=80BjcfJCU;{{h@ zX>G99!-$nSs6+IHT1WNt^o*_#pGf~WKR%ZADaWgi!JTm|We!2A*n^MwU`$18N`*)1 zEUu*;sXZ2!xP|F^?Zbl-_@;A`1;5GbT9r`K_v(54SI4^pD-$8m_L1Y7O+;V#Y05M# zcxDSW5Xh}>y%eKNfg5R+XOypQ%5Qs37Reaz;==(J7o9fviVQUVJOp?>+8_Px{k$11 zRl&_f^;;IK59diZxtEVCtZ+-K6(m6#SmvVDSXNvdEi=Rq5*(1X%{;x)KJ1`SaJWxa zdX(+PWO@JOL!Zm5=EeFmalNKs(G5!2&iNHUlA#k+br2u)U}bXpSc&@1?pbl*+HdNzk` zIm#iC-&5Jl=hmFzO)AMh&C;2M+NR_+xgVA^chJYGe)tL(D=cbB(K)h)Dl%&7st5 zGEEBG{a!M0%d*EAZv9t)h{EH|@|@kznKC$6Y%@>T_M$-3C(Em) z?aQaI8*~!r@LFO|BNj;Z?Ujt;^X5rh8njRs)37Sr3H&bDc}~6=FDkY5#@ZRlIALF# za!46@?^m6L)nYks!YR4FC+GexSs`n&vLgv@lKZgohflbdJ48&cxRF07%SUcfgX&)BJ;2jO_xl4#VOZH zh5fgid;DnB5U>??V+E0Tq%|szSGdf+0^C@WZ@OH}aFt(NK!t}2H$=w38X~V>vn_v= zew4+5SDJ3*S`ojwxNWJNM4nmxkM@u@MC~n+6Seous5b!)oOGGvpk8~(=TW%|;=tK< zFWtg3*oz>W;-iDa-<&^3=6^9Xw{D&BvJxqPJ+_PO$?P}kclZ5J=k>m6^?(^KF{Mn6 zQ|pkIX+WdOY&*Pc^jVuR$7Q7~4jWn)ll?V^a3Gm$_~DWVZ&;!Ih`@;_%*$;ltV!mt zUwzb}&PHO(D$IIVwugsI@ik^+N^oYk)Kw)YW#TDKg`<~JUU`19*qbQRK!-cB3_?7~ z5icPHxS9*8?JK}dJkJX3_B^DSXL5P!Z&ynurLLD=iaA#~|3^N9F3$tctOhFke_~>q zThcCgSKQnR>{ByIgucdPqdvNO@?QNF;A1>+wAA|o1+fboE3>Pu@e4!`_I6uT-?r@q zeO>*I3tL^fiMv<8GbiS64e)w*)2dn!)!8Yo9zXku$J&%yyhJRf($4nvU;`NbBE=00f=e70(>u9`)i>7GBoP`57PhIzH`P?Mg19!%ai#1 z?Y5>WwMx8q*DBySN_X(1Wi=m%42Ys*vX$#MkO^TXAblVa=OeX;#~uzkA5V@JO?I*<4g zwflxla^Bs#|L;v+)D|rpt@w1MNu>i>7$se%^JJp(L{#YLWjYXOMM{y%@W{B8FX)F) zxo_0{9rvwQIZ^ik|63y&>&5hz$MkNi2|s8f^S(0`Yi8Os<0D+AY1mjhDNR{t85rGA zWNG1$5NEM4n>eV=m+0wb;`HxRfxzppQp37uups|v;3VQ=HXG=)@1s*EW~Y^^U`w_3 zIaiQ!)ZzF4&BXoJiTwZYGp!x2;v`sh8Ppc7qbsX}kPpuYh6VW1WVofKN{Z$B-AWTv z4H(XmHo0L{(J$A_l#H%;4_&Uf{s*~VxeBF3*+As2ESs}_be|@HMyRs9Z>J0~y3R!M zy>xfi`iT!-ji>Q1S>fq$y+_;=ROtPkr?=bp7a{k%z9kF1uuir?h^n`@zd z?j*gqNbFsFfNU6lT<2DzlRQ^uOlj3wxY6}pol}kZ{fov`F5Sn#bQ>=ij<%_I3txTF zL{oxj;i$j~>a6O|wb)?vBM=#9$4wN5lW+#Wj5n7u+Oc_D5zQ2|JSO5AVXtv$mjro8 z9R*?w8LHn5{mW%@zG;KG+|T}jsNXr~kL?@@%nj%=d9mEi`-&@>qc1?F3g8X_^{BwZ zv(UkUT}R-M+rhM5&60ruicd>m&TM;YPU{x4NL0Y{S4RS|d~e2|YuvE9ri(q@V=3@& zP{SsH2{f)8UCJ^}9G#ZA&Y8jR20n|d-RVWcuZn#M)-~D>IKx_{~oH#b715WaHbJaiHn52^| z**9ywQy%yocA_^bl|_l`f=`qTmEcT=_dU=YP0xES6pR}xy;(4*t##$m(E@Ar?i1Di z;{N>B3c;xmX>fYa2gRGNDiKhIE&|nik-rOXxD3cM+EYEPex{G2QXHCP>(;>fKxbj`Cy@tXv!XVPy8mK>MW;JeP_~vvPGr-F1smkw=DC;W$cuVy= zJ!#$0<3ItnmxY1#_YL}m1~dMXDicz6DuT^E;$I|)!(rsxJhs298=5A#wTHsKn3nKH zry0{H2$j44@ZK{)jD)?7`%thrubf@5u&{-enUq3V@Ij&2lvRLFTo7mm&-Xl_wRYF}N$KT! zHQz|~GVv{5%N$X!n-)RD@|w8Hsf(_EFKJ8!mi#htSYnM+pEu@;S$0f^#o)ntxOy<+ z?hmDc7VjPYg&C9%sv*9ykEYeQawqlm48Y6)4UOCE@ukTrkO*tJyrB&7M_*-;e73@MFjb>)f^t+n9?Ryf3aW?-~ zG`Wcx3w;YVI}5tv-{8TrGew@AuZ2`YrWX2)4t(xUBqpIk0{!$(|GL#-3Jpg8?#!WH zPPiRNVd;og0=6y}>HPR_Rx()ZEBWd(jZjUDGPnM68vEj@*@VO7Js+KD))(y%&yG9a zvBkBgF|}o|+d+m^(;~-()uXzZmF1sOjT#^wd=lh7zK1!pRDlbO-x~a`vHff`!Ky03 z+J52QE_R=Zj<1NRIa_AfLh!aU5LSZsVTo%7+P#rqa(8f z%u9($A01FS)(ciziVo}_urINyf!&(!TUvm{D|7xKf(s)N$HL+a7EAL>bHUkrQ(}^m z0mB*D3#9MfiQm(p{yf{;9fT+C1V}(irW;?suJBjUm=J>xgiU#WWuIO=7w)?KHKt5} zn}LqM*_Vx;X5Pc9t88*w_K_8EfWN6zj4#gVYf0|4&#iG!DZWLmcJ;@U^cx$#479um zIwx?&OL7S@eBtKetK;>qz6X$-rLcP%=BQ#c3Dy75+I7DHNzG1y?znrrR<99w2)(du9{Z!3vH# zi}O?7REXzGKkUMNKqq6SHX}46ecrEJ^fga-G$N!LwF-n(XJwti zEN72qy3+}#gv4>Y>!omJ;VN_vwXzwiOlAdXx)q_esgIyh2uzh|(~=;Gl{2m@v&@>n>F(cfXy5YyDuh}?MZ>g7Ij9;nM_P^w2o zD(F7`ZqX{P*yQ~ycbcW{5tMLu3oD#?3}`$9Ul7yPfXwYV%sAA?IrXDTZ~y-Mqf~?R z^5F=t49VK*72vj_$4d0bM`E_osOtyGF~?Q(q}2NdNVPC_{6rkjAYa#hA8(_xuhd`A z8WxlA`(4cGkYje5KYj)9IROO|w1GtiH4Q8+ps7pb<*6c1PYdmVqWH2nsdH?Ff1t(a zajvhcm{f-Lo6MVZISf6Xhd_0{FrA;Ay}8uCmXD@_TyS(N(C|zOj7Cjfkm|Tk$pS>( zS_4QHcM&|1QMV#tdGI@y`A%g_piq;<`6w=p;Vb(!mM=!4nS0zcR<7-SA};7AoApz zK$CdS()EK7y1ihXP6BH^mOth*;xxQ$BQdan0{WG2fbZVTTi+UytzfYYsX5Pxs!j_jw%x~8xZf^h|2&@h8F*h5+b$Nq& zf_{@E!Kv1$;Yo3)gfiUHy4#P^g{ z{3k_*WPvV5LL2e*T2wcdrHfj!i-#KjAZO9ZFM)`|J(P~iDno8RPuX*Jv#X^Pt^Amh zUGY}1!@9b$gt6pV6nGu* zNm9(WF|pWjjWVoTIAgAWAS%ATMO2gnWdBHd0+hKljt|1is>M1?!~go2Y1GDAnA_bp zx1$3+A&h5LZ96U9ZWe~H(;!yjcP}Ze5-JoG^3VE|lOWP#ao6wPF@H`vR%CA7@AlG| z^ZikGmd^>t1>}1JKFw6OuG{@-H|*B23SH1!!qRl}qopgr>i{0iuQiDP7-Zoxi}0Om zQF1+A_R?l#c_+(x+|-$QYlsNLq7JIA0Pmjp`RxpxFW@VVqzO_%=Eg4D*P zziWcVd7A?1xNwjn?n5WVfj1`x#j4DY5+qC0H1FItQ>=(l7IJEq!#S=f(Hda(4v97; zL!t9~TJ+c$s9AmSqUyb9QH{v3;^u#ZdDII==FlWl z5#-FCr~U=Xv_`}#U|eiHO;go)l|rjauTWf51)mDS@x&)k*LMGt&nJN^mahP`5}TbB zb1*9$1d0wO2w+A&LGj|=Dagf)ocR&{u6KTPd1V`F?D8!4^STF(>mzP?UKj5gd7s*H zeP5t*s!`|15~MnbkL6r6+FEof<97;2;~!pavrx(>hxg_B+Q`l$_U>W_+P(Z&(I?-R z52(8Zu^Mpfw*2AFy~F+!Bu9m`E?k@a?2TF6W46KHj^7)euEyaqE}p5cW`+6MT>m|O z{1c9ozJVZ1>Quh#3fi@?(r2xyq^5FMB;{PE)Y9ubYqklneegUn)mlY|(6EiK-|PQ! zW`n!}w4M<-xIN1JWiL3|2wXEni?OIY{N63c5A5(fXy4cnp%qc~6BKkO!vd?7 zNhVxFD4jzjryc7<2?s{;I+)8+epRdAv=WCFBxGM=*_k-B1_mFWX_e{%Eifn9D=O4m zDaxGIf>u?5@q0=>73*J}@6NMUk#M{|Ocg$K^z{rV&sv4Xz_aKf>4Fh>SziGn7P`wj zE`_QN%VAM@l!)H@{H}FCvLDCmt{bIcS{*1=1H~*}0ScN9M#Gw$7>K6W0!ps`va3i^ zfWos(Hrki!`_W|k74_ufLZ<1n#Vjb&0z}@0KG%r7MV)8A!HLf^(?(~_MF25b;f?x5 zR=C^_S^3nky}gYYDN0C2c*_J5d@uHm`ztzy@GsX7*L)01DX+crAjtS{>P{mTXw@-@ z3H=rQo3onvTV^^8s%~E&B?kV^yMEnbQIq4{t9+7m$b9o1|J4qbteXTu|MdNtXO&n` zyr?(k5FGL#!T_>U0q^io$Pc|Y@FuH1;hUA+fxWTZ&;ysX>XtM9({1E}HJ{u8BLNoD z;yF=;7@@!(bX<&3;wW3wb0Le|)80x>3mH1G0^{a|q0gK?V&@xE#Nm@?Eyq*P=Gt3|J+=!44%$c%BSM^Jhnp3m#AD;(wbCRq#CdK4Kh;q1H0w3@ zCB+t*+j(Y8J~aBQcosqqd8--S=u@yTzf~psZhu_VSh#`j3b0j)%o2QXM7>Wt%(6Zb zBD7z1@l?48Egp@8Tk_21VgC}|79xqkiemzdYk6~k8+zXKouNFqBp++G_jsiV90Ut4 z3{t6^P*htDugNlPO0tn`aPsVaJwu$L>`{VPJ4l@QhzRZj+jJUkgS1wu&oMk{ph;VxNp}fkcD*%}rfd$nlUl^tM*b6Oe%f8Lc@w+bV zpp>oKx7U(CY)FXM(caEyg2!ICeAY#@-Kz@NRkn(~;3f}&L)jy-{tyt4f9rV)8ZBM725XavW z`(j8($OAgDF+Y@$gNlrRLYf){htBQRfW7#4-Y8HgGBxHnZU0A0!{q8XUIlXVWh&K& z!N!y)Px*N@!Zhaped#Flk3VVT7Krb<&2y{l>r6hDGL6GPO!NwX;HwUK5+S-CoK-D* zK;ZP+gmDIe^Zkje2RIMad~hMLC^RvlJt3x85WQ!|z~42dLGnJ`F`@Mp>jQPgGP}Cc ziUysv{rMKYX4k^(w~)2$A}S1EBO91qbiN#{rW_}=ml zribU=X*!(PJvEx8cz@62)&Q|$P5b^1ud-Qb5Y-Ia+6=9iCB69V2zN_g-00E3U5j|+ z_{=HW(}qp=5Z4wleH)>ONM-)!rsXBT9}n~3MEzT8Ui1qAD8i_HeL#l`QpR+5FPMYA?Xfc%+wwGV=Ifs~`iq8Mv_bx-9u3 zLmL_=5(kRz3K8w+Q@$dFml=*-kkW? z9LO4%aG@z4)66>SFRVA=(-5d&t-@lVnYlY}u>VRa$BC5P(6FC9cD(n77J5>C!u+G_;gF0=^+K^8wJ3x<$Mr-<>7p~ zMHzms`BKmPc>mH0if!S;fqQhb5Kt6~2r#z*=Q?oEEv z)E;tpFKDmM^07?y^uQH>Q6ZLRId-Z2!Y{%Ys~I5{r0!yCt=8ddCaHA%L2h`C z^4YJ32d(x|7HryQ?>1TR;ETsQ%{ND-OSX=iC|^$ERZd`gd)M0o(oeOHJ{cd?JgZdZ zES62;%V2xw;*f^5MbAVi3xKR~QMA`ba&W6g-^CD0hDxsn9FC4t%tagXozgYW;?C@@ zeb0Uy*ML6Vvo8*^!z^YOZZqJ1Q!ZGq1mY?bU;B9FbZ_?$PM?RgtzWkvo=G%$fq(B_ ztF+0ZfUkEA$v7x)E%OO3y9~ONk`v`=lMhM4gq-$RmQHjP>6WM!;(RCUpwt81iUW1_ znx;#f&Z7nL%)}C|mJ-h%os?zu2TUrE-6pBfS-vFpHYuBVkc;B@?G>i(c zjPC?7N{uiE$zlqk1F{#@(NL<^Y59;ENRbc_QB;?>EK=P~Z>bK>Bbhd4-M+xmMBENjHx1R(*u{Q8BX<5`*bP|cdr5@`c8 z281Xt!G35zPmcYYDXOz)krTKRSix#F|EKp)|E??i@ZB@ER0SJt6UjsJ_SJ7VVh!B} zcLi9{f3L`OnZ%u;5}^T=HZK$v_=)`yHTK;%Hv9%f%0oR*eXa)q%{%+s1Z(Egw||re z`Ytb7&LW99@&O1Jn73VDnofkjM1dXpF6SHXoey<=^3u%7j;6Jq(Z5?a=!D)keN}$l zdGlXTPqp%+km>r0Zd%H1Q`vEUggC+>@}O+L@wJy%E^n@IW%Wcq$?bnuD*k_4sp44v zRRsm+<&SG`A%_%#kW0VB{Y<8So8D7|%^H~L+ki~oKvuqGK9Ib0+WPg9|B!P`vu z=XS_1=C5#ki~x?q)Anu9LpGoX`~q6^va0JOv%3RLYC_-M!_YiQ%b-vgRilzuu@f?- z!Li&qy{oGrwglx&NhXn>*vXf&u%_kt={++yUERPX?jF@fk5G_pHJN5&I<01>*h)MY`~cJz{1QE|5$FvX-+LW3Pv)e8fxJ z=i}VBZ0=4ht{iiDK-a!(D}0kuspEA~HVkcfa8{M1S|if{I{#JAbp?omIdDPSL)fv( z`CjOFs`+gjmYSLjt5MN%H})%lJ~%H*I{mUof=9;$9&%{j$o^}gPrMu zlKN+?=tto1=HnzgE$yd6PbFrZSuzDew<5gtBsgi_O7b!?5N43fgAgi1H@qM4J*Y<4V#wk2ly{80Pf7DjF+&IEsPsw9uG zd$E0yjJYA4b-O|b8_}i9(!)cA7tG#P=4#ew@ONn4^0n-0E{=9OSpuiH^CM)|(|W$` z)HxW8Cd;)QJ%>44m+x%_s{lY$fpef~B;l?)<4cnNA2kX@kY`4_3N_vy%|nXwYQxSt zE3>s+P;rY|KR>w82#gN!?URb!U?r_I6{_rx$r2x)RVWV`x4Aowhpvo-)@RlfE$twu zHa~vV3H)iL)YUmCt+Q@DyjH~txUT<%+yOVz%`J{AS|%3EWS6LR9d$?BN`h3Y#@z;! zv`TA_@{g!XJ?rM`#-;p5`1T*K`M{!5mG9vEa&Tsbqkl8{q@FoRE%=1BFQI_&A;~AV z2Jd+<@qL6weudt7$l4xtYV7PhnH2I?Y7cp;pMw`WsF3mnS9B=_?m^T3aEo20dk>M0 zbjc9QTqqN){)f%#h)zog*hUx5?b3HG0}}r@hF~==^w0?{m{>=eM}%r*owz9@(CX!p zt0nPtkWA7#tz#hA+W_n;qU$Mo`cxLll{U5buy^byS@pDgom_~}WkJwWbeW)GO_C~x zkX5`R0n{mnRLdNR?*Dpa@6E8Eocq$6ZdQ#mFzxB%(u^xWEH@w4v;j;1y3l+mn0Pz6 z2z27-x2~XH6RkhsW4I^u4mC_8f221w=R;)so{5&NH=h$oO)Zt}Pk z0?zj5R`o%?3@(;5+Ak(lcoZWFz$KJjsuCpE=E&s#EQ^y8pB+8fSq?yphays)$m zSrzle|DhOaAZGdCMbYvZKfEZk>!tuA1cU*t+g4*!cTqq5K~J!_8tgnWJ_=7sUN;TT z+69Ttj_`LrRCKpGTkEZCEG->kr+?{cc+(zZd7jb3!`b+_7=IUASv#pHS>iv?ZTdF2 zX%H!s1amFY5iwGis#=1dRhwQ8FI0bwWIO%ET9M-H)1XhF80gJ+Grj_pfLk&*OaA4l z%$lp|Xr=^xs!rF87Ylk?ilf<~{n9qjgD&0d!aJM-#XZL9wC(#~ zp0ZjJAABJX&ZHZ~+ttHjfEHN&P)lO?kb(pyxH6wHvs_4zq<6!_9F1MaI`#QI$-Od# zu!Oo297q!vP=S-^YJ9pfiqwtER%A(lFh`%HenEciY`^y`QLENpC$`g$)3z8lgw;fg zW_~Yr;+DXWw}dJ_MQiQtOMf@q)NZJpJpPG>Cahazy)xo^{k4Ph^@4=w(^7lWEa@-F zYbeVyll`)_lq-O+la&$N2*{V!XK{G|?Ez*oSXVdstOOF5ESk-ZDsAZMeak~#XQy-@ zZx1<{(){Kps#F^)J?UttvY;`J%)JY3{|F$qB|`bu_S?lmSOvr^lm0c4jOKpI!tiiP zJlRhnanK#WrUYQB!P=gP%qSC+_fGk#`TA2NUd9!MpvhjTD~0zpZkmtGLM>IrsT((q28G06;9R_@tEAK(WcA;pM51F*ncn!Bkwd+MIX7EG;q=jbWb)kaRb_V?7^0%3$dv|)UqHyw22tz z#l6^&dNUfjXysID5lQ|_V{)Cpg+)C7geSBk^$3{*1Sl>O~@Hk~{;aaLJX{_11z1KKh z%!TQA z{kyB19F#6(KmR<~QQ<@^cp2l8gFV27BF)M$Nn`WX&mw|*V4sVaI@Hk*o=4Cx2{Y*} ztKxAfgoil40)NFIc}zjPV}2S)zj0!Hih8Kz`JV}iG^m}1`1((Uv3G1pma_Xto`Rx( zTaIVl=BI%m%mm1KIIxQcLj+yh8!-AOviMl@CIYdPVLda<<8ZMgc-~0dL#_bty{vrX z%EGk^TJ=gow=BhlVgJNHbMujvjF^4j5vi@SyG*F26v^O8Q zZ)?>N-QSaLB4V#54lLCX4T;R=g^xb--B^1am14DwO~^!;{udzv|GaZ$f|!g~c#A&Y z(VHyYm%3CPHw{IPZwo}3+ZlyxY5+<|Z`_HAG4Wpw$TL=V^u23hY?t^J@IXB9)mt~2 zC`9p=Sv_;UPocA^AA3kPSoo<2*oQ$~MB+gQZ!W;(J=tvy4WTlQcmHdc!0P5}Sn1O+ zF}wkEWx;Z_LniTtoAl38dx)IXXVS(tUt#+Ghk60rh%n910bz^=PssoG#w1!+Iy3VU z1ytkj>Kx{cWBpRq!0@h^f-lkIJy2N!LlLFSwQT?<2xAs2R-(^{yaY(;>;PD}Tl?jT z-OK0e=8mkr%3nx6{99bE1A#~%Ps&V>_IY?nep(IZ7uoMz7JA>u_wAMGNK7LMIzHm< z7fVmzeg*cew5#@-3F zhTh=iWh60;s*{QR3EYxF=2go++Det^BV(zcykVa}nnnj=b$Izli-Py<=FehpfWd!D z-+Dv}Ye}lSShi!qoU<%5%R}&K**)Y!AEk)V*Is%u&xc()V($LEBbI=||CFjQK`z8j zdl6Z?e%!h!xbI!fzHv)PP8auZ4Bg8SR;avHoXeLqN}?Zpn>we=x!zSQ^5@A>em z{-h|slbEPjj#GZPx}2T&t6RVBgx=`wGRbxSPec6=*6@G&?!PtsqS1fP7@0gb)h9*% zqC_>&?Hkm}$9X_t-Cg@AFE-LAaT=^e_QSMnY&%aF|Kvmg?w6#z70WKNSh7Z%&o5Y- zYy4gqp%THOa^=*)tWJ_q!F4W1}Mz+kF-r8*0E$6>`gFYgj@|UZEGDKPaGVO0L#foFJ^w%$Q*w4*%Yey9ue&pr7<86To{`6Hu0orGxMZ4VeAJkXkI1_yw=^~ zC|MiO?DKM{?w!gbxxFaQX+ub9#sv!YQ#2>*S?YY*rSH)r?+c%XNzosksyag1!ZT#= zy8^>&Fg#Btg231Tt*9eI2$XdGhWB)Ut@uO2Wcb8sj|1zO=c6-Kd@J`ivkx;raUV|7 zaq;L_9>WP~QO`w4u%R69q+PsrRTTOP(C5D>Q2YHZ{fq4}0j{}>P|5IDK{56s_Rqg8 zvTJMZStCw<{2g0bFtnDPhd5-VuE0kADjY<$kcDtLc?P{6R}>bE46nePtVjlz*q{cW zFt#Qgv-mzOx?FFGa-P9g7p!}_t(51RyAyd%A=hW5KkXs7D2ax^9y~SFT!dBk86vDcm`+kUsy<%^*pK7 zU(BiLt0|E8u;%%s0?zu>e1F`>TH)oiEBlhEd3fH*xzZN~(APC>-hlNurbIK-SS`PQ zjCznZo|6WmCzZ=yG}qNvBa9r~s7~>7eyhYZRW@N!@9C)xF1Gou?D&f?$qm4mMH21= z1x!gjt(e1QwVyXQ%Z_hLwx5_4NpE-DQ>X8&xT!5@T{?2d$1ud|y$uLkN0mhza{W%L z=*oj9^G|pVnJ49w#w!VTLdLRghujAv!%{yT*Sf7tx*6_Y?)@Fm(XOz2aTXHvt217r zAjHI1sMh+2E9>UIvY>zXQLAqzbzLpIVbo;}EXrFuQI=nem?zkl%8Ih3^EnPMYTnZ3 zi_?RO4^)12mV4`ymr=-nk})xNs36O9l96coe0WBEIOoec|CCBs;SmaZID;ZchYW?# z;A~f5Ol=gOUV-Rw_Iv0f2dqu9uXGw!G*hPsc$5cW33oxg`_gQ}Q5u=EF#ZE-s`P)-(<>0$Hyx_#-@f54=M!Pi+Rr}#=|AdQR-d+ zYvr>c6YdF-|SGfx)8&3&pYZrt*WpZOd=2jMIJM zbrzQwOND8z0P}a(b%Vu;MrwKYT*Rlbdmr$D-#wf)52 z6#!QF8(zHB1inr}0lwGlL{>!R4w+~)xOh%H>zEQ&3T7{;bmUKgClvg2{TnC@uE5u5 zYGyiRTkyM&6)o-pe-5)%9s8LgH|1L-f~k383FH`-jx0LoOp39HPn%eWNZ)GN^y+Lc zHxY3GqiU$w{Id$u>+2TzN$#9s!%wj-t<6MK6D?}gdj1g0Ne~$mo9;d`ofLEJ3R~;< z9a=|yN?v|4Um``!dOz3oZs1>IJGw)ZN3+L$_6#X;yjr<|Nrgbo0QXAj%FL>Zth@4(gz~k_NHH~q4h89Rx7hx)TwuHF*O^kO z$$3H_^zqZiq863Jt)o`|q}7VpAnWW{ag!<0{d@Ld0u8z{)51I1%!=7c$Y+9CZc{y9 z_hNI$)6fU8i;u9)(agz3+cHJ3YIK6I;ClPA_qBB!1Jk&$;;4Y+)O{_kx14 zLpi$uh1+Q`zsg*t!eQRuVa&($6Q)-6IgQwa$LuCPZ#)%8)?%h1)Twru#C;$7cV}-^HF}Lm8{Kz=b3^ zPsA6$I{Mjai`RBkPt~->=DSwSrC{!>Wd#<@Qd+6)TATG#+k$k@$s?3f;xG*ahxm^= zAMnU-&_gC5Z+g?p%ogNv8k3#KP74G5)j^6^!@jS-7;#t6mUXE=NR%a;aE0zvp8Gg0 zJB*j%9$WSJbLZY$@3_rGlhg2s$hQaWK0W^}I6z|6#4t~$(ge%)Xb3j&)SW@RHdpg_ z=EH#@*Kh6)PrM==MDE|bdyIGWCRd2-@EU4hixkJZkaeJfj#%2%NE3Y~=e!wnzR#ro zkxHtuo^P*EFGrPV)LB?c^n=&7!pPrW1_ANe5Tx_3I-+>;!hUa_t{Ip{HYpJ(-l&TZhy+ei%e=cMqfmLKb; zq10gE$(|7_*v=H`kXz5o;)jFV&nQ=$m7hOB%Frs*y!lLdUz;UB4-aNZh&4<+B@R_f zHH3AeP5-`>t6$+$;s<3Xvzo7ecztH%$05#OJY}y882oLwVv-n_niC$sHh4#(1;l@X z|A|C`_iwCxV4sauOY#(Kw2=64pVQ$W9@hpF#@kJHv#TQP_xpD>mc@VAsO~BDwJ&QN zFpenWJb5nbicr|LiAL8bs~vo2BvTMuU&SilnX6t6%H)lV(a-w)lZ~z`R=-D{T7=Of zFwGz;Whr~PoM~L;xrf`N$Si4&LwYt5M2N3*&_1)BZ-nAf1L6S^YrFTbqUz+y)_u)7 zXt7Y)IWC3RWVfY!n@`i6Yb*C-eW4fUHb{>tC$N+gDD##efp4p6i!I|~0b%_mz(HU1 zTM={9wEHEYF1z_Q(#lx3nBhO!ION+*i!shW*&JOjUgwI<%q6Ka9377zqlMmo|LM32 z)Ma$3J1!-1&Obldnu**Vnf7+V)z4)EL18ZD_k|U(uEPZ8S&j5hHq+{C5=6V+B#OQ4 z-EFC`WQ4@IL?Hq-E3mT`bGXXae;Myb*Pv3uy)ltHR+%!9A|xWuS)57KVtw(GKi_u^ zc|f~99eZJP{K)G~x!0XsB{mLl)GW#Tr8z^1)c3gXVV!;|=ZWvGX_m&0P!Y{r2AtFT7fpg_Fsr6Pu|4hK33ZJb`xkTlDsgZ z-B3M04~u&Q#^pTS=imJ0^R|W!X7d|F5ObPv&6d)#ecR5p{?ZckM}|lfgs(41fThWB z3W1l{8kMWDghR`6-s^dRJDozGHh)6$laR`>gWLunzNmU}f5k5yO#p{ zJo29(o_ZQrJYx6)<9a_wB-cciOtU=kT#-qyKhj?{^MiOg=&P6#BRS+Py(S;54Vv4`G|ay!1X*E12~Qzmdw)Sh znWx%8ryyGP^Dz1=C@QYSe~Y}TSXM?2&?aSDK>c!J4;=fuW<3IGmcUtV;l&Bx(6`Lp z$-B`4{!uLxBFxEtVIv8JUqX6li+Qzv$}7l@sLo?1->JqEC3b#53Gd+NQ?#87Q9|sD zW;;g&4UYe+WuBl_3+U8Hzo=$edMl`Ut2OSlt5#BaA-txqsW6x0y^+{!N6xJ_0fUGY zX~P={TGdA3uTyI!G@XuI+{{@T8Q0!k-oCdtJ+L>cGDo5QWIM9MBG@&!q@Jx}=!RmGh{R;eEP*P5(drGA3l2Km4$-t(KBJ<8y^3#G3FLk{xy!E9? zpT;(|C6ni`+X^CU(0E7+>ig}jX;T^=3j*?fL7O{{dAYbsXU)(3|{%q_)kq+LJh*B2DteXMsm^4+w%SzqeoceeJ7 z)o32$zO0xhi)943d3{-0L10JXC_+?qzah}y7ZS{j**(uv`RWZ4s)^8u61d$9{1dV4 zK3w{Ph9>h*uH;rqRaUWEOW(F^va@ePw23gc=)Z z^>MsG{jifPyLSs|Rt05E3N^)NlowQ5^6oWx22QJh&B2W3%1-fW*9g%ly1qxdwrjw4 z=L!P4GPF9Hc{kZqmuvqg+tn3G5H@i;Q%RV0fDTC^=zjHv15t|z{2@jdV%P>(Y`kwFmDVq2B~1T?Ub`>{WxafnMardmTIjBkYfW3AyJ zJUB#f&5rSWPL4#jPAdAz=HI`v_|EF-{6oY_ZdMI&lzrrtSRx5_s9J%i6$HBQp#GaZ zh^@=vKT2$0?KPS;gR*mUx< zp4NF3Cv+ymsRC@VYl7BeWQ7#F>=1AWPv;UnPTO>Y+~F1-X2k0pc*JDrqttJ0DZe}a z?kt&B!o>j=Y%j?tfCvOl9lz`4+Xl^_>a2-^*rNY(76NA^jHaa&>nr;Y7q8b<5p&!r zp4Rw`8p1G8~d9DZsT^f5%zy@ zeXcT*p191B5xY8xTpF67goz0-?yK*MKS!HTQxViP>E9ho)wx#&9WD^T z*!h_HKWX@of7~TKjeuN6wY1p^_NjI2k8k$rg=we1=r0u~;0-ILOSa%A(gN$P#o&aC zIQ6w*Zep1k6t;mfZH>gzi<6dWy~FG*eTZazzJ>L zBV&MNJg08fIu~oguQ>xRK__;Fr}ed#w#4 z{?HjR#%Caj>@P-TRBgiP&MiEgs3H-}iAJ!|uT!dGO6!(h-F58>)_xIX_G~Y|H^*u8 zkNlgzo4{t~+g;r`$YFX`Jq->Y)3*cN`q-C454dl7Lc?MVx|i>iv2+0crWJv65^X~X zmg_JK+Cf6h{+8vOMd3v37(U|kQvr#a*x1eGhwVvbRNpf2kQ+iQF_oqpu`kPb z8p2YdB4E}@TUY3x&C*QM-yL{drRjI&!69`U9=sVFQ1+6`$`)4^*L*AI%D(UVR_*R! z?99EGBfDx$4QhBO%bJV$@-M)Vae|`IpYYbX;~2M&M<+-BM@=e|ne^7j9AI(87>8ll zwPIQid@06!%S!A-5-{bmm&RHg+$}K zD0Ex8a92}dk+CIwY(pb8rvP2PfCg(@A&xLIZ5#K410E>U8?mV7ePT& zV|2w>@iww^%WZvR-C_$l)4yN8v#!KUZ-kt}gzEBuZ@(RZs|%zJmipyBzO^76tzl6G z&CBuou!MWrCOAs$XuF93DbMQQNXTGaZf{-YePLLrB;(=$LCo$feA#w$92&rkp^a!A zv;4y)lQSxO?F^(%`}SGxgv~Cd7RCv57(nt|gi4?7>o8REu0?4mvBz`M`{Xg4?s0{t zPY&_z&tc%lAU@XJqH$*1%Ho|hcd-4)25uL8ec8DO8a*L>YU_RJ55#UI7Vx4Ll@J_0cGn`;mGAj;yj>9kGezjY7?Cb1P~I$o#6_cozk=qGHpT`% ze)#4DuT_|N^x~Ri=pn=1(;d@cgBO&s_+6StFj_vBrNs24bsrhBsZOwmGu1fk3~ohE z`6YE9Uk!~PSBM;U5G^?q3vr0fe!qX&6&O)|PJB3aiBy*yVIJ|2nE=nfCxvZGg(IIl z`mmfHNE$ViBqVGdkNh$5t}2lB1SZSO*?h*pMB_zyLb1x!{&&!oT`+$TUtg)%XqEis zNE<|wVdrT(k|2MZh-uFIl%7#Z0T7ix+6 zg-&9M7XWn9u?qT6bW(St@^5sq`Zqdp*V@(lz{Kq8lDpr}*9?itFi3r4Hr<}bT;06u zogG!$%w?Uky{4X24u>0**)P(1USgFF&hK-7lddFwy`WN4te)($mBdKBS$f7g=46(t zqRq1P2mnkB^I3P8FMhJ|vh1XblH_w;Uo~-srRGWI->s+${llYK~pKoSco`u_a-=CMfDxU}!C!LV|?kr#St zB8z1mb9WDJM__O*j4hnndq@21&;S+E0&HFo;MoO-}_yp^|#HI9(cK`!J_PC2dhUG zMe65-Io54fIeqyVMqYf0Z#u>Gg}$GLrAFxZP0i>)X}ao{*9+g)>MIPSG!2nGd8uH+q@R z^6(%!)!=sS&C8SfFn9S}#%i0#cBT%lM+{Z%kB6@-%A`@7mCkIl{kNSQRo&k zUCCEfJ(OWI$z_S;B>iP_;nMu83X~PxCd#`csni*5RNVe!- zEu&@e8&1Z0g*EiRrsPbHj3w8nW@TH2 zGk<>GSt6S)-i0Fqb{vPo@j6s44w`RB0hgI6)U5wO@XEr1O_usgn^pj79M z%N?tgMr82rxkPKg$M@TKtxcozM@-)emwo^9u|_RBOM&|dN5t&hpc&mJl- zH){vN!%QCfc>5h@@mrKV?6hJ5;B!2T|H*cz0hh$VwB1&CboDqi=f<4leaIG1MeMhZ zgFKO-LlDvVCuqA~yp&RefDIG#R9Iy1;m)`ToJrn5*nxAO3rp+24#Ft6EID9R*cc<( zQR52u0WkD^cQF#rxc?5}WL|X*@{LAyKhStn=9fLf0oWt&H$ai&=LgEcj!%7x{BU9y z5j!J$S?4yZW=q>V1lV`KS#7JYx0{`Jpf{P@?;}m1E0`dXYZsZkFrFFYT#I(uOGS&) zDJ=w9ESAAniOOzC7#WFh2SanmxSaG~>3O`Kr{l;7&R-0R`1zNjt_99D2Q!e~-YQ%k zqT0$(8;V%x$g3g@C@DVRx9&S9uNgUdL+miumQZf8$C>v%XJ#WInm`0Q3xojK{DNFl zhB>t#g&hIN1?LA>czOe!90$Htt)uuh879XiJ1w1aQxn0j&XJT$}nfX? zVAwWq`T3Hk+%bc60$8!}&eHZW0QI2oqhVu=@F5e5! zgB~ge>$xzCCn>Rb1l57M>|kqBs(zil_EAMm*vxu!{X-26cHwT%58l68&-oizFL*I> zYHVn%y!_SVyU0LZm{%15Sa>c4he?1x2`6$copwLVZSLi7-Mjy4 zYn;mKZDT~m(Tp|!p;o06s1RU(2)*VlW;5n!%M7);(_C6@_i|=S_gYaHqU*qRM7x`n zhbc7Oj&IM0NGUK+4yeN>Ki%@)B^GB}EmEo=*a_X?p7xzH0F?5%hd0X0PY--Ti^}A- za!#F*Mj@n?8GoE)D`RAqk-o=2m&+30OWTMG&#uC>Du}OAgWm6-?Gx|!;UJzKbBT}d ztIjJEuTIUOjURhvjT(Vh_)7kD9VYD>ba7xE$qo1C%A_RZ!SxSekkb-=<0i>bGiNKQ z5xR0ht7&$iwzMxHf5C+nLKmT~b@fo}lDzOKxzNtm>nA_@R>(|C=&ao(c2jnKz}|gJ z%)=AQi5Max&UYX-VG=W3(3)0JLl4|9&l`1JvmIty>+=hwan)<<$&5(5ybH+n>OHHy zV)&3lYl|TY-#@9uybf%QP1pmBHg+p_-J1Q8sl9NK_Tl|*-0YXGRSHFrhQpgOCff*# z^oE}B@(m8z>f(-0=fQLG-O`5{w)0;HD~| z3fj%RcFUMcT>zE@Vu{@DE@I$~$*fezJ+2v@-G~ZtOu>%tb-| zh0Sq+SHPUb`#!NqcDC&1hWgDI#visvTS35X**>%qwLz(6M>>~mpRMYD^CabcnCUw| zCEti#S2ihp`=geO^Hf86-3rSD2t3$dq?6~;+VT(q@agMFsead_;y|#4LuV|U^wRvi z;y^P#`pCfK^T__Ox?UahCaYw5af-wDCtLaa#`>C$bsLcl&6BSbw=OCSETN|%*+y;k zieRuFj(ZD_x0K0U4?-VWhqzS+r{}&=IcUbt2SAxdOxUJY0m+V|`3Li@k1iXf<#h5+ z-ZK6Jta<1YT>O~jn6S+C^Hg9>MznAK6|vhiB{D}lL7u+gQkPYmFpR{-yYe=Z<tqb?ale_A3IBQdkzkzosn9@=DAp2}7c83UyPNoXTd)?A=vi zYEy~fXbU);CjW?`8!m&x@s6T!Ms>nnF=r#G68%(UHCgaio&%F8;@d((1?4cHBwOrk zblD2lTz#^QiwMP3wL%8b8?|2!Qg!_!DbnL9^CbI!Vc}(Xzqo_-hxH)pgUn{qo@gUV z8^Ax=KxSEXpmv_aEC|zq#%0}<_7Ih}2<&n08>T!?PQJ(?KYt=IL(g5D=U~`wbe~Br z_Z{chs?FG6q?Dz}yekt97AbZi6AG!`_*lk9fa{4k+?d4wAf;-6bMXUNV;`L-QcI-j{k%H~}NI#@yXS=tn+6MZJ{yqWpm zqY6n<{8xy|f=%K{W#H1qu1r?RP-nnMXO~4~;!n1EBuO`GaC=)6)M|c!>C{e*Yr&ji zdU08`&9*89A_9>`3n%ZlN!6fkqupNVRoHa2!`<6K%1tr^MtLLX(0tV7-ckLJjN^bi zrosREaq`ew^PqoPLzi!r;5nPbiIEF`<R_Ub~b%wLT`(SYDp1R ztD2mtb^uc9RH;hI3mbkEU7@!ajtN;$$mMs$OU=Z!wO2BJ4$KG=TeprmsJo%ui6 zhBkuI7yO<7r3J^6;c#Bka-l0#B+-mrv{6%&Rb2dc+z`)592*4UhNvdoB61TIPVULV z2{qpT-N5|jyT+B_s+Kd9^1|Gbm87Bme%2NwaxbS8BH~KqhVU6tdyXuS)ys=ec1jZX zKzs5HiRCRoK=Xc9uYL34tO+2Xo2r+#Yga`8v|CC9Fk$Gv#McLd7alPrx02FA2{{

IlN1yzX&Vt8zC|@9?`Mc}G;Is%vfWZvtQ8 zdG}0iSw=c8As~?Kx8l_A5pA^~hwd{5%2iT3k#%M7O_0#Mux_-#wQQ1W(Y!)=3sOC= z&=xP|l&n_AyHNA_o+C8zjLMR7LNZtA)V^v^7lE6Jq*6}Xl9`diU~}rH<>gkKm_na> zSe1mKc&*_&6S}M(_M3W4>yLe4*jp3@eWjf6N!hSjRr|?ySxx_&Ej=)D4Be0J#ZnC0 zzyJnS#vwa#GHPH8MtfV3Dt*^w2dKVJk>Yxyq>g?ChlYBais`7R+?VE>cExdOBO(TQ zj;H%#HhN**6sT|i)UFHMr@R=KpJYIQ1DPx97IPqd9mM@L8J66t#SJW#FI*5#Y*la# z5PxhHYyglmvBSJa$U@VjxuT!-c|LjkUzWpqOu*(VAk>Rwr!bM0gA=v=Y8)tq@|S(f zyfYi;lv9Fwy(-xA0drdHw>hnRwEO%d!`^mZ;3u0nq&peY8@LNHL=8>wJYOd<_Cs>+ zGegm)6CzQDSd+zVDqP3Zen$IV^oid|##^_&b#c7WH(=-e}SQfPvh zVBd7D&yNN#sL7UM&}jUr>5rFh5!JbsNuJ|!%MX`Uq{CU zt=*(&s1oi47F`VLHjzDDd;p V_A2wg&9wa2`Ii6k8T51b{{U&ecFh0) literal 0 HcmV?d00001 diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" index e69de29..d9f5878 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" @@ -0,0 +1,194 @@ +### **Hadoop-3.3.4 移植指南** + +编写日期：2023年03月09日 + +#### **1. 软硬件环境** + +##### **1.1 硬件** +CPU 鲲鹏920 +网络 Ethernet-10GE +存储 SATA 1T +内存 Hynix 512G 2400MHz +##### **1.2 OS** +EulerOS 22.03 +Kernel 5.10.0-60.18.0.50.oe2203.aarch64 +OpenJDK 1.8.0_342 +Maven 3.8.6 +Hadoop 3.3.4 +#### **2.编译环境准备** +##### **2.1 编译工具安装** +`yum –y install wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 cmake patch protobuf` +##### **2.2 安装openjdk** +安装指南参考链接：https://gitee.com/openeuler/bishengjdk-8/wikis/%E6%AF%95%E6%98%87JDK%208%20%E5%AE%89%E8%A3%85%E6%8C%87%E5%8D%97?sort_id=2891179 +##### **2.3 安装maven** +###### 2.3.1 下载并安装目录 + +``` +下载： wget https://archive.apache.org/dist/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz --no-check-certificate +安装： tar -zxf apache-maven-3.8.6-bin.tar.gz -C /home/installed/ +修改maven环境变量，在/etc/profile文件末尾增加下面代码： +export JAVA_HOME=/home/installed/jdk1.8.0_342 // 注意jdk等安装路径 +export MAVEN_HOME=/home/installed/apache-maven-3.8.6 +export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH +source /etc/profile +``` +###### 2.3.2 配置本地仓、远程仓等 +``` + +/path/to/local/repo + + + + huaweimaven + huawei maven + http://cmc-cd-mirror.rnd.huawei.com/maven/ + central + +``` +#### 3 移植分析 +| 原始jar | so文件 | +|---------------------------------|-------------------------------------------| +| lz4-1.7.1.jar | liblz4-java.so | +| netty-all-4.1.61.Final.jar | libnetty_transport_native_epoll_x86_64.so | +| leveldbjni-all-1.8.jar | libleveldbjni.so | +| snappy-java-1.0.5.jar | libsnappy.so | +| wildfly-openssl-1.0.7.Final.jar | libwfssl.so | + +#### 4.编译依赖包 +##### 4.1 编译wildfly-openssl-1.0.7.Final.jar +``` +下载源码：wget https://github.com/wildfly/wildfly-openssl/archive/1.0.7.Final.tar.gz --no-check-certificate +解压安装：tar -zxf wildfly-openssl-1.0.7.Final.tar.gz && cd wildfly-openssl-1.0.7.Final +编译命令：mvn clean package -DskipTests=true -Dmaven.javadoc.skip=true -Denforcer.skip=true +``` +Q1:修改libwfssl/src/ssl.c，868行在J2S(hostname)前增加(void *) + +Q2:手动下依赖 +wget https://repository.jboss.org/nexus/content/repositories/thirdparty-releases/org/apache/maven/plugins/maven-compiler-plugin/3.8.0-jboss-2/maven-compiler-plugin-3.8.0-jboss-2.jar https://repository.jboss.org/nexus/content/repositories/thirdparty-releases/org/apache/maven/plugins/maven-compiler-plugin/3.8.0-jboss-2/maven-compiler-plugin-3.8.0-jboss-2.pom --no-check-certificate 到本地仓库对应目录 + +##### 4.2 编译leveldbjni-all-1.8.jar +###### 4.2.1 github下载源码 +1）master分支支持aarch64，编译前需要修改pom.xml文件里面的版本号，也可在windows下载master分支的代码压缩包，再上传到服务器进行编译。 + +``` +vim /etc/profile +在最后一行加入 +export GIT_SSL_NO_VERIFY=1 +保存并退出 + +git config --global url."https://".insteadOf git:// + +git clone git://github.com/fusesource/leveldbjni.git +``` +2）进入当前目录 +``` +cd leveldbjni +export LEVELDBJNI_HOME=`pwd` +``` +3）修改pom.xml文件 +注释无关操作系统类型（freebsd64、sunos64-amd64、sunos64-sparcv9、linux64-ppc64le），避免引起编译不通过。 +如下： +``` +245 +246 full +247 +248 leveldbjni-osx +249 leveldbjni-linux32 +250 leveldbjni-linux64 +251 +252 leveldbjni-win32 +253 leveldbjni-win64 +254 +255 +256 +257 leveldbjni-linux64-aarch64 +258 leveldbjni-all +259 +260 +``` +并注释 +``` +288 +289 +295 +... +315 +316 +``` +第60、61行，将编译的目标module加入leveldbjni-all和linux64模块。 +``` + 57 + 58 + 59 leveldbjni + 60 leveldbjni-linux64 + 61 leveldbjni-all + 62 + 63 +``` +4）修改leveldbjni-linux64-aarch64/pom.xml文件，第77行，修改后文件对应内容如下： +``` + 76 + 77 aarch64 + 78 leveldbjni + 79 false +``` +5）修改leveldbjni-all/pom.xml文件 +注释无关操作系统类型（freebsd64、sunos64-amd64、sunos64-sparcv9、linux64-ppc64le），避免引起编译不通过。将dependency相关的依赖注释掉，注释内容如下： +``` + 75 + ... + 93 +``` +将build的native code部分注释掉，并修改153行，将x86-64修改为aarch64，表明生成的leveldbjni-all-1.8.jar里面对应linux64目录下的so也为aarch64，后续在ambari的nodemanager启动过程中，默认使用的是该目录的so： + + + + + + -- Gitee From eabd72f8cda0c3aca7df416c5f91362e6b1b3284 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Fri, 10 Mar 2023 02:01:38 +0000 Subject: [PATCH 13/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hadoop.md" | 47 +++++++++++++++++-- 1 file changed, 44 insertions(+), 3 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" index d9f5878..4c753ad 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" @@ -67,9 +67,7 @@ Q2:手动下依赖 wget https://repository.jboss.org/nexus/content/repositories/thirdparty-releases/org/apache/maven/plugins/maven-compiler-plugin/3.8.0-jboss-2/maven-compiler-plugin-3.8.0-jboss-2.jar https://repository.jboss.org/nexus/content/repositories/thirdparty-releases/org/apache/maven/plugins/maven-compiler-plugin/3.8.0-jboss-2/maven-compiler-plugin-3.8.0-jboss-2.pom --no-check-certificate 到本地仓库对应目录 ##### 4.2 编译leveldbjni-all-1.8.jar -###### 4.2.1 github下载源码 1）master分支支持aarch64，编译前需要修改pom.xml文件里面的版本号，也可在windows下载master分支的代码压缩包，再上传到服务器进行编译。 - ``` vim /etc/profile 在最后一行加入 @@ -185,7 +183,50 @@ export LEVELDBJNI_HOME=`pwd` 109 provided 110 ``` -将build的native code部分注释掉，并修改153行，将x86-64修改为aarch64，表明生成的leveldbjni-all-1.8.jar里面对应linux64目录下的so也为aarch64，后续在ambari的nodemanager启动过程中，默认使用的是该目录的so： +将build的NativeCode部分注释掉，并修改部分内容，表明生成的leveldbjni-all-1.8.jar里面对应linux64目录下的so也为aarch64，后续在ambari的nodemanager启动过程中，默认使用的是该目录的so：修改后内容如下： +``` +147 +148 META-INF/native/windows32/leveldbjni.dll;osname=Win32;processor=x86, +149 META-INF/native/windows64/leveldbjni.dll;osname=Win32;processor=x86-64, +150 META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86, +151 META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86-64, +152 META-INF/native/linux32/libleveldbjni.so;osname=Linux;processor=x86, +153 META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=aarch64, +154 META-INF/native/aarch64/libleveldbjni.so;osname=Linux;processor=aarch64 +155 +156 +``` +6）修改所有pom.xml文件中的版本号，将所有pom.xml文件中的版本“99-master-SNAPSHOT”改为 “1.8”： + +find . -name pom.xml | xargs sed -i 's/99-master-SNAPSHOT/1.8/g' + +7）编译 +mvn clean package -P download -P linux64-aarch64 -DskipTests + +#### 5 编译hadoop +##### 5.1 编译 +``` +编译前安装snappy snappy-devel： +yum install -y snappy snappy-devel + +编译命令：mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/lib64 -Dbundle.snappy -Dmaven.javadoc.skip=true +``` +###### 5.2 问题 +Q1：找不到sasl。 + +报错信息：Cound not find a SASL library (GSASL (gsasl) or Cyrus SASL (libsasl2)) + +缺失相关依赖，解决： +`yum install -y cyrus-sasl.aarch64 cyrus-sasl-lib.aarch64 cyrus-sasl-devel.aarch64` + + + + -- Gitee From 797b5b2476c6249e34a7ca7ee6cb0a2904c170c4 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Fri, 10 Mar 2023 02:02:06 +0000 Subject: [PATCH 14/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" index 4c753ad..c93ee9e 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" @@ -216,7 +216,7 @@ yum install -y snappy snappy-devel 编译命令：mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/lib64 -Dbundle.snappy -Dmaven.javadoc.skip=true ``` -###### 5.2 问题 +##### 5.2 问题 Q1：找不到sasl。 报错信息：Cound not find a SASL library (GSASL (gsasl) or Cyrus SASL (libsasl2)) -- Gitee From b414bab738dbe230ce993e190f322609f2fd0f65 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Fri, 10 Mar 2023 02:08:26 +0000 Subject: [PATCH 15/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hadoop.md" | 32 +++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" index c93ee9e..cd54202 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" @@ -217,13 +217,41 @@ yum install -y snappy snappy-devel 编译命令：mvn package -DskipTests -Pdist,native -Dtar -Dsnappy.lib=/usr/lib64 -Dbundle.snappy -Dmaven.javadoc.skip=true ``` ##### 5.2 问题 -Q1：找不到sasl。 +###### Q1：找不到sasl。 -报错信息：Cound not find a SASL library (GSASL (gsasl) or Cyrus SASL (libsasl2)) +描述信息：Cound not find a SASL library (GSASL (gsasl) or Cyrus SASL (libsasl2)) 缺失相关依赖，解决： `yum install -y cyrus-sasl.aarch64 cyrus-sasl-lib.aarch64 cyrus-sasl-devel.aarch64` +###### Q2：yarn下载失败，代理或网络的问题 + +描述：[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.11.2:yarn (yarn install) on project hadoop-yarn-applications-catalog-webapp: Failed to run task: 'yarn ' failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1] + +解决： +1.安装Node.js +下载地址: http://nodejs.cn/download/ + +2.配置npm + +内网限制原因，需要修改npm源或者配置代理，才可以通过npm正常下载。 + +修改npm源 +``` +npm config rm proxy +npm config rm http-proxy +npm config rm https-proxy +npm config set no-proxy .huawei.com +npm config set registry http://cmc-cd-mirror.rnd.huawei.com/npm +``` +3.安装yarn +``` +npm install –g yarn +yarn config list 查找strict-ssl选项，将true设置为false， +yarn config set strict-ssl false +yarn config list 确认修改完成。 +``` + -- Gitee From 2ff1a231d8550acb4ad8cfdf4d9070767397d40c Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Fri, 10 Mar 2023 02:12:59 +0000 Subject: [PATCH 16/99] =?UTF-8?q?=E5=88=A0=E9=99=A4=E6=96=87=E4=BB=B6=20Do?= =?UTF-8?q?cs/=E7=A7=BB=E6=A4=8D=E6=8C=87=E5=8D=97/1.jpg?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../1.jpg" | Bin 27873 -> 0 bytes 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/1.jpg" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/1.jpg" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/1.jpg" deleted file mode 100644 index ec877fbfab90775e85180cc2194e88fce4f0fcff..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 27873 zcmd42cT|(zw=NnGM5RgZpwg78bg2r`L_m585S1DNgkGdXQ9!zYfPjK@=^_ce_ugwl z4NZCi0s%re`S#xD+_U$&pL)$G+W;P!J3Z4Cek z2?^j8@dda-0n`B3NJ;;B6E8C2oBRejIT;!GEeeY3H>hq=QBmHaq@<>yzfDa;M?*<@ zoB1{!10xd?6BR8BD>EZ2JtGt2KMx_fM*I&M`Au^2n~cy8%!l3%^Ek z4?s#sa*dAUsuREi0FaOqNBhr!|KlPdC64hr#f_V{D2W{qw*jOi*RGM0UHfO$#NNTg z-vMNFhNhPGb0cFDQ!{f5OItg82S+EcvzNDzub+Rw+YcYZ!XrLKMkOYFNlr=q znwFlEo0nfu__L_^cNH8_T~k|E-`>&L)!ozE*FQcnIW;{qJBM6Budc0cZ2sBW#vUI1 zJw7?bot^(97YX3nzl-(1B>OLN(Gle$B_q2=M)8kaB&2@Cc8!jV{O+Uc^iK^aUVAXy zlX!oFQ8gjEvh60Xq#=gsjpx`cW5#A$(eQ zXKjavMA30&iF_7UVgjF#8sd5R+ghV{5mJ47-WpZ?5QCCJgWn#~-Q--OGOg(>3w(;D zXoG#km_tsIb}YGKH?n>@Evom=tB!-_=lL=QRhd;21s6V71NSjny|^+geMk2ZPQY4* z@>7NyW+5&?t|9i}QCox8eX2RNT3b26Z}0U?H&VYL4-HNc`09NwGf0T6WeChYQwVo+ zwSo+1j*BoKGeqgew2Lu`mlf1iNs9dm8>-iu03L8DPQF-sJJG0%q38*r#q`b~g4uuf42JME4FhWM3HHyqkjZ?^awV_MFKJQo|iBH#zs9WvYIlAU$1Kx9zHfHF}I{ zKAM9w+<5E0s7dsXc+{JJx$lI&^)Zz7{R>(HC*s{;QR|}JhgD7+* zJ}gMEOa}2|=NfP#w<&SF2RN~j`G9-FoLSB`YJG7F=36)v`8;&6ket0f7ILCSoU)~8 zvDZ__TxK9lSd|q-X6Hw5y^t>RgqL0V=hV&GNF8Q<(Wi9HGje1{>6=~9TRS#1I8eX_ zgqa|LQOfhBEVv96oH(%6Idd~r^E!FY=UV;d)|rSz?gm@$$x;-E9i9QYg^M;D6ZzY6 z`%gWh8UZa>XhWkQzy>*=NX5DAx^Z*qjv=xhXnJjI^3y&J3upag?ViTHD?qil=JXXn zHh4q676gZ00ctK6flKPH%{I7b+XBr3e|>`qct$W6LR6_Rb8%WbQwbel#50d*WG3h7`vA_x>m-(x7U?gpTJC;(q zE}x^l1Hr43Qtt;dI(VtqTxVOCJteJaBb9lbSU%%2UQWE3j6TI;rZi#Y_OlglV5~`eitV+@M_Y7_DDl^{JBHO2? zw{`5VX}Iy~)aLv9Wqr}*gl;_JC>&&|D09L;!BT`bn*Frism?+cm4bSIkJu_2MEjdx z0ZLK#J%?}pxg{n!iLQ@@p(DZyFcqKRL-+b221O_}>ZadL)4uI9b_sQ6wHlIoH`If3oYPL8@Az_MqVuk2no zg)hX=P~3wbLnE14kei*te^%;IXaM<-1eOQUU)Q-Ehgy*zJu*Fx%Zf|6H2DP~by8Ds ztD<~}&s)pw!8%1QbKS|`p2m9+VQIbRmOwY}a(Hvxh2%9eoGV(CQkyW|^vk>~b>N7} z?Owp^qw7XPk>-towcW>Ty+4l+BF>k>T|4Vzn6cD596emS^nH{Mdo2t&;m?Sc%98K1 z$d8L$xxgvv!wP$fVJ`j&Z-6C%A7fu3IwScqt=hj3_1+VwSB@zaz=-LH!922C~oMs>mne+g)>cv@ey_V;vv@S=d%}vZ+l@1JA_UEL`;SO`}$;5;2YfBqv@4u{! za9ga=1e9w`Z|&}9*T&pUWU*k>X78<|;1y~O=kB~6IKwR_WfBlbeRl%5g$o|B*-VRz zz0prg1$U1Fdvnj zDW}b|ozU)!vQ~*!1ZRU)NC=Eh=^9N1JDK;!rDJu>8>@gf?W0EZszZ#8LvxM>v(`vYu#ZE?ZspL3#D2v6>HE zmM?xuPtF)IJwrVZYk3nfXremA+tRD0VYwFMib1UgsZdTe>IN1q;3PXnf%DZ#K4>jB zjn9d@T7I)YH+{F5XMrR{)4s2tV!xF7WBNh=>dSU?lWT zZK5z`a`MnN`Z37em@dclb>buXmPW!5VSnJjRcw04GuHuKe~iz#K%D(mw#629G(p>y znJ}@7xp__})9J05I-(%Gsg*|AAFQW*i`Jb9onue~c&Ur*_cPwOT=dfJL^@527ovP9cYg|k=80BjcfJCU;{{h@ zX>G99!-$nSs6+IHT1WNt^o*_#pGf~WKR%ZADaWgi!JTm|We!2A*n^MwU`$18N`*)1 zEUu*;sXZ2!xP|F^?Zbl-_@;A`1;5GbT9r`K_v(54SI4^pD-$8m_L1Y7O+;V#Y05M# zcxDSW5Xh}>y%eKNfg5R+XOypQ%5Qs37Reaz;==(J7o9fviVQUVJOp?>+8_Px{k$11 zRl&_f^;;IK59diZxtEVCtZ+-K6(m6#SmvVDSXNvdEi=Rq5*(1X%{;x)KJ1`SaJWxa zdX(+PWO@JOL!Zm5=EeFmalNKs(G5!2&iNHUlA#k+br2u)U}bXpSc&@1?pbl*+HdNzk` zIm#iC-&5Jl=hmFzO)AMh&C;2M+NR_+xgVA^chJYGe)tL(D=cbB(K)h)Dl%&7st5 zGEEBG{a!M0%d*EAZv9t)h{EH|@|@kznKC$6Y%@>T_M$-3C(Em) z?aQaI8*~!r@LFO|BNj;Z?Ujt;^X5rh8njRs)37Sr3H&bDc}~6=FDkY5#@ZRlIALF# za!46@?^m6L)nYks!YR4FC+GexSs`n&vLgv@lKZgohflbdJ48&cxRF07%SUcfgX&)BJ;2jO_xl4#VOZH zh5fgid;DnB5U>??V+E0Tq%|szSGdf+0^C@WZ@OH}aFt(NK!t}2H$=w38X~V>vn_v= zew4+5SDJ3*S`ojwxNWJNM4nmxkM@u@MC~n+6Seous5b!)oOGGvpk8~(=TW%|;=tK< zFWtg3*oz>W;-iDa-<&^3=6^9Xw{D&BvJxqPJ+_PO$?P}kclZ5J=k>m6^?(^KF{Mn6 zQ|pkIX+WdOY&*Pc^jVuR$7Q7~4jWn)ll?V^a3Gm$_~DWVZ&;!Ih`@;_%*$;ltV!mt zUwzb}&PHO(D$IIVwugsI@ik^+N^oYk)Kw)YW#TDKg`<~JUU`19*qbQRK!-cB3_?7~ z5icPHxS9*8?JK}dJkJX3_B^DSXL5P!Z&ynurLLD=iaA#~|3^N9F3$tctOhFke_~>q zThcCgSKQnR>{ByIgucdPqdvNO@?QNF;A1>+wAA|o1+fboE3>Pu@e4!`_I6uT-?r@q zeO>*I3tL^fiMv<8GbiS64e)w*)2dn!)!8Yo9zXku$J&%yyhJRf($4nvU;`NbBE=00f=e70(>u9`)i>7GBoP`57PhIzH`P?Mg19!%ai#1 z?Y5>WwMx8q*DBySN_X(1Wi=m%42Ys*vX$#MkO^TXAblVa=OeX;#~uzkA5V@JO?I*<4g zwflxla^Bs#|L;v+)D|rpt@w1MNu>i>7$se%^JJp(L{#YLWjYXOMM{y%@W{B8FX)F) zxo_0{9rvwQIZ^ik|63y&>&5hz$MkNi2|s8f^S(0`Yi8Os<0D+AY1mjhDNR{t85rGA zWNG1$5NEM4n>eV=m+0wb;`HxRfxzppQp37uups|v;3VQ=HXG=)@1s*EW~Y^^U`w_3 zIaiQ!)ZzF4&BXoJiTwZYGp!x2;v`sh8Ppc7qbsX}kPpuYh6VW1WVofKN{Z$B-AWTv z4H(XmHo0L{(J$A_l#H%;4_&Uf{s*~VxeBF3*+As2ESs}_be|@HMyRs9Z>J0~y3R!M zy>xfi`iT!-ji>Q1S>fq$y+_;=ROtPkr?=bp7a{k%z9kF1uuir?h^n`@zd z?j*gqNbFsFfNU6lT<2DzlRQ^uOlj3wxY6}pol}kZ{fov`F5Sn#bQ>=ij<%_I3txTF zL{oxj;i$j~>a6O|wb)?vBM=#9$4wN5lW+#Wj5n7u+Oc_D5zQ2|JSO5AVXtv$mjro8 z9R*?w8LHn5{mW%@zG;KG+|T}jsNXr~kL?@@%nj%=d9mEi`-&@>qc1?F3g8X_^{BwZ zv(UkUT}R-M+rhM5&60ruicd>m&TM;YPU{x4NL0Y{S4RS|d~e2|YuvE9ri(q@V=3@& zP{SsH2{f)8UCJ^}9G#ZA&Y8jR20n|d-RVWcuZn#M)-~D>IKx_{~oH#b715WaHbJaiHn52^| z**9ywQy%yocA_^bl|_l`f=`qTmEcT=_dU=YP0xES6pR}xy;(4*t##$m(E@Ar?i1Di z;{N>B3c;xmX>fYa2gRGNDiKhIE&|nik-rOXxD3cM+EYEPex{G2QXHCP>(;>fKxbj`Cy@tXv!XVPy8mK>MW;JeP_~vvPGr-F1smkw=DC;W$cuVy= zJ!#$0<3ItnmxY1#_YL}m1~dMXDicz6DuT^E;$I|)!(rsxJhs298=5A#wTHsKn3nKH zry0{H2$j44@ZK{)jD)?7`%thrubf@5u&{-enUq3V@Ij&2lvRLFTo7mm&-Xl_wRYF}N$KT! zHQz|~GVv{5%N$X!n-)RD@|w8Hsf(_EFKJ8!mi#htSYnM+pEu@;S$0f^#o)ntxOy<+ z?hmDc7VjPYg&C9%sv*9ykEYeQawqlm48Y6)4UOCE@ukTrkO*tJyrB&7M_*-;e73@MFjb>)f^t+n9?Ryf3aW?-~ zG`Wcx3w;YVI}5tv-{8TrGew@AuZ2`YrWX2)4t(xUBqpIk0{!$(|GL#-3Jpg8?#!WH zPPiRNVd;og0=6y}>HPR_Rx()ZEBWd(jZjUDGPnM68vEj@*@VO7Js+KD))(y%&yG9a zvBkBgF|}o|+d+m^(;~-()uXzZmF1sOjT#^wd=lh7zK1!pRDlbO-x~a`vHff`!Ky03 z+J52QE_R=Zj<1NRIa_AfLh!aU5LSZsVTo%7+P#rqa(8f z%u9($A01FS)(ciziVo}_urINyf!&(!TUvm{D|7xKf(s)N$HL+a7EAL>bHUkrQ(}^m z0mB*D3#9MfiQm(p{yf{;9fT+C1V}(irW;?suJBjUm=J>xgiU#WWuIO=7w)?KHKt5} zn}LqM*_Vx;X5Pc9t88*w_K_8EfWN6zj4#gVYf0|4&#iG!DZWLmcJ;@U^cx$#479um zIwx?&OL7S@eBtKetK;>qz6X$-rLcP%=BQ#c3Dy75+I7DHNzG1y?znrrR<99w2)(du9{Z!3vH# zi}O?7REXzGKkUMNKqq6SHX}46ecrEJ^fga-G$N!LwF-n(XJwti zEN72qy3+}#gv4>Y>!omJ;VN_vwXzwiOlAdXx)q_esgIyh2uzh|(~=;Gl{2m@v&@>n>F(cfXy5YyDuh}?MZ>g7Ij9;nM_P^w2o zD(F7`ZqX{P*yQ~ycbcW{5tMLu3oD#?3}`$9Ul7yPfXwYV%sAA?IrXDTZ~y-Mqf~?R z^5F=t49VK*72vj_$4d0bM`E_osOtyGF~?Q(q}2NdNVPC_{6rkjAYa#hA8(_xuhd`A z8WxlA`(4cGkYje5KYj)9IROO|w1GtiH4Q8+ps7pb<*6c1PYdmVqWH2nsdH?Ff1t(a zajvhcm{f-Lo6MVZISf6Xhd_0{FrA;Ay}8uCmXD@_TyS(N(C|zOj7Cjfkm|Tk$pS>( zS_4QHcM&|1QMV#tdGI@y`A%g_piq;<`6w=p;Vb(!mM=!4nS0zcR<7-SA};7AoApz zK$CdS()EK7y1ihXP6BH^mOth*;xxQ$BQdan0{WG2fbZVTTi+UytzfYYsX5Pxs!j_jw%x~8xZf^h|2&@h8F*h5+b$Nq& zf_{@E!Kv1$;Yo3)gfiUHy4#P^g{ z{3k_*WPvV5LL2e*T2wcdrHfj!i-#KjAZO9ZFM)`|J(P~iDno8RPuX*Jv#X^Pt^Amh zUGY}1!@9b$gt6pV6nGu* zNm9(WF|pWjjWVoTIAgAWAS%ATMO2gnWdBHd0+hKljt|1is>M1?!~go2Y1GDAnA_bp zx1$3+A&h5LZ96U9ZWe~H(;!yjcP}Ze5-JoG^3VE|lOWP#ao6wPF@H`vR%CA7@AlG| z^ZikGmd^>t1>}1JKFw6OuG{@-H|*B23SH1!!qRl}qopgr>i{0iuQiDP7-Zoxi}0Om zQF1+A_R?l#c_+(x+|-$QYlsNLq7JIA0Pmjp`RxpxFW@VVqzO_%=Eg4D*P zziWcVd7A?1xNwjn?n5WVfj1`x#j4DY5+qC0H1FItQ>=(l7IJEq!#S=f(Hda(4v97; zL!t9~TJ+c$s9AmSqUyb9QH{v3;^u#ZdDII==FlWl z5#-FCr~U=Xv_`}#U|eiHO;go)l|rjauTWf51)mDS@x&)k*LMGt&nJN^mahP`5}TbB zb1*9$1d0wO2w+A&LGj|=Dagf)ocR&{u6KTPd1V`F?D8!4^STF(>mzP?UKj5gd7s*H zeP5t*s!`|15~MnbkL6r6+FEof<97;2;~!pavrx(>hxg_B+Q`l$_U>W_+P(Z&(I?-R z52(8Zu^Mpfw*2AFy~F+!Bu9m`E?k@a?2TF6W46KHj^7)euEyaqE}p5cW`+6MT>m|O z{1c9ozJVZ1>Quh#3fi@?(r2xyq^5FMB;{PE)Y9ubYqklneegUn)mlY|(6EiK-|PQ! zW`n!}w4M<-xIN1JWiL3|2wXEni?OIY{N63c5A5(fXy4cnp%qc~6BKkO!vd?7 zNhVxFD4jzjryc7<2?s{;I+)8+epRdAv=WCFBxGM=*_k-B1_mFWX_e{%Eifn9D=O4m zDaxGIf>u?5@q0=>73*J}@6NMUk#M{|Ocg$K^z{rV&sv4Xz_aKf>4Fh>SziGn7P`wj zE`_QN%VAM@l!)H@{H}FCvLDCmt{bIcS{*1=1H~*}0ScN9M#Gw$7>K6W0!ps`va3i^ zfWos(Hrki!`_W|k74_ufLZ<1n#Vjb&0z}@0KG%r7MV)8A!HLf^(?(~_MF25b;f?x5 zR=C^_S^3nky}gYYDN0C2c*_J5d@uHm`ztzy@GsX7*L)01DX+crAjtS{>P{mTXw@-@ z3H=rQo3onvTV^^8s%~E&B?kV^yMEnbQIq4{t9+7m$b9o1|J4qbteXTu|MdNtXO&n` zyr?(k5FGL#!T_>U0q^io$Pc|Y@FuH1;hUA+fxWTZ&;ysX>XtM9({1E}HJ{u8BLNoD z;yF=;7@@!(bX<&3;wW3wb0Le|)80x>3mH1G0^{a|q0gK?V&@xE#Nm@?Eyq*P=Gt3|J+=!44%$c%BSM^Jhnp3m#AD;(wbCRq#CdK4Kh;q1H0w3@ zCB+t*+j(Y8J~aBQcosqqd8--S=u@yTzf~psZhu_VSh#`j3b0j)%o2QXM7>Wt%(6Zb zBD7z1@l?48Egp@8Tk_21VgC}|79xqkiemzdYk6~k8+zXKouNFqBp++G_jsiV90Ut4 z3{t6^P*htDugNlPO0tn`aPsVaJwu$L>`{VPJ4l@QhzRZj+jJUkgS1wu&oMk{ph;VxNp}fkcD*%}rfd$nlUl^tM*b6Oe%f8Lc@w+bV zpp>oKx7U(CY)FXM(caEyg2!ICeAY#@-Kz@NRkn(~;3f}&L)jy-{tyt4f9rV)8ZBM725XavW z`(j8($OAgDF+Y@$gNlrRLYf){htBQRfW7#4-Y8HgGBxHnZU0A0!{q8XUIlXVWh&K& z!N!y)Px*N@!Zhaped#Flk3VVT7Krb<&2y{l>r6hDGL6GPO!NwX;HwUK5+S-CoK-D* zK;ZP+gmDIe^Zkje2RIMad~hMLC^RvlJt3x85WQ!|z~42dLGnJ`F`@Mp>jQPgGP}Cc ziUysv{rMKYX4k^(w~)2$A}S1EBO91qbiN#{rW_}=ml zribU=X*!(PJvEx8cz@62)&Q|$P5b^1ud-Qb5Y-Ia+6=9iCB69V2zN_g-00E3U5j|+ z_{=HW(}qp=5Z4wleH)>ONM-)!rsXBT9}n~3MEzT8Ui1qAD8i_HeL#l`QpR+5FPMYA?Xfc%+wwGV=Ifs~`iq8Mv_bx-9u3 zLmL_=5(kRz3K8w+Q@$dFml=*-kkW? z9LO4%aG@z4)66>SFRVA=(-5d&t-@lVnYlY}u>VRa$BC5P(6FC9cD(n77J5>C!u+G_;gF0=^+K^8wJ3x<$Mr-<>7p~ zMHzms`BKmPc>mH0if!S;fqQhb5Kt6~2r#z*=Q?oEEv z)E;tpFKDmM^07?y^uQH>Q6ZLRId-Z2!Y{%Ys~I5{r0!yCt=8ddCaHA%L2h`C z^4YJ32d(x|7HryQ?>1TR;ETsQ%{ND-OSX=iC|^$ERZd`gd)M0o(oeOHJ{cd?JgZdZ zES62;%V2xw;*f^5MbAVi3xKR~QMA`ba&W6g-^CD0hDxsn9FC4t%tagXozgYW;?C@@ zeb0Uy*ML6Vvo8*^!z^YOZZqJ1Q!ZGq1mY?bU;B9FbZ_?$PM?RgtzWkvo=G%$fq(B_ ztF+0ZfUkEA$v7x)E%OO3y9~ONk`v`=lMhM4gq-$RmQHjP>6WM!;(RCUpwt81iUW1_ znx;#f&Z7nL%)}C|mJ-h%os?zu2TUrE-6pBfS-vFpHYuBVkc;B@?G>i(c zjPC?7N{uiE$zlqk1F{#@(NL<^Y59;ENRbc_QB;?>EK=P~Z>bK>Bbhd4-M+xmMBENjHx1R(*u{Q8BX<5`*bP|cdr5@`c8 z281Xt!G35zPmcYYDXOz)krTKRSix#F|EKp)|E??i@ZB@ER0SJt6UjsJ_SJ7VVh!B} zcLi9{f3L`OnZ%u;5}^T=HZK$v_=)`yHTK;%Hv9%f%0oR*eXa)q%{%+s1Z(Egw||re z`Ytb7&LW99@&O1Jn73VDnofkjM1dXpF6SHXoey<=^3u%7j;6Jq(Z5?a=!D)keN}$l zdGlXTPqp%+km>r0Zd%H1Q`vEUggC+>@}O+L@wJy%E^n@IW%Wcq$?bnuD*k_4sp44v zRRsm+<&SG`A%_%#kW0VB{Y<8So8D7|%^H~L+ki~oKvuqGK9Ib0+WPg9|B!P`vu z=XS_1=C5#ki~x?q)Anu9LpGoX`~q6^va0JOv%3RLYC_-M!_YiQ%b-vgRilzuu@f?- z!Li&qy{oGrwglx&NhXn>*vXf&u%_kt={++yUERPX?jF@fk5G_pHJN5&I<01>*h)MY`~cJz{1QE|5$FvX-+LW3Pv)e8fxJ z=i}VBZ0=4ht{iiDK-a!(D}0kuspEA~HVkcfa8{M1S|if{I{#JAbp?omIdDPSL)fv( z`CjOFs`+gjmYSLjt5MN%H})%lJ~%H*I{mUof=9;$9&%{j$o^}gPrMu zlKN+?=tto1=HnzgE$yd6PbFrZSuzDew<5gtBsgi_O7b!?5N43fgAgi1H@qM4J*Y<4V#wk2ly{80Pf7DjF+&IEsPsw9uG zd$E0yjJYA4b-O|b8_}i9(!)cA7tG#P=4#ew@ONn4^0n-0E{=9OSpuiH^CM)|(|W$` z)HxW8Cd;)QJ%>44m+x%_s{lY$fpef~B;l?)<4cnNA2kX@kY`4_3N_vy%|nXwYQxSt zE3>s+P;rY|KR>w82#gN!?URb!U?r_I6{_rx$r2x)RVWV`x4Aowhpvo-)@RlfE$twu zHa~vV3H)iL)YUmCt+Q@DyjH~txUT<%+yOVz%`J{AS|%3EWS6LR9d$?BN`h3Y#@z;! zv`TA_@{g!XJ?rM`#-;p5`1T*K`M{!5mG9vEa&Tsbqkl8{q@FoRE%=1BFQI_&A;~AV z2Jd+<@qL6weudt7$l4xtYV7PhnH2I?Y7cp;pMw`WsF3mnS9B=_?m^T3aEo20dk>M0 zbjc9QTqqN){)f%#h)zog*hUx5?b3HG0}}r@hF~==^w0?{m{>=eM}%r*owz9@(CX!p zt0nPtkWA7#tz#hA+W_n;qU$Mo`cxLll{U5buy^byS@pDgom_~}WkJwWbeW)GO_C~x zkX5`R0n{mnRLdNR?*Dpa@6E8Eocq$6ZdQ#mFzxB%(u^xWEH@w4v;j;1y3l+mn0Pz6 z2z27-x2~XH6RkhsW4I^u4mC_8f221w=R;)so{5&NH=h$oO)Zt}Pk z0?zj5R`o%?3@(;5+Ak(lcoZWFz$KJjsuCpE=E&s#EQ^y8pB+8fSq?yphays)$m zSrzle|DhOaAZGdCMbYvZKfEZk>!tuA1cU*t+g4*!cTqq5K~J!_8tgnWJ_=7sUN;TT z+69Ttj_`LrRCKpGTkEZCEG->kr+?{cc+(zZd7jb3!`b+_7=IUASv#pHS>iv?ZTdF2 zX%H!s1amFY5iwGis#=1dRhwQ8FI0bwWIO%ET9M-H)1XhF80gJ+Grj_pfLk&*OaA4l z%$lp|Xr=^xs!rF87Ylk?ilf<~{n9qjgD&0d!aJM-#XZL9wC(#~ zp0ZjJAABJX&ZHZ~+ttHjfEHN&P)lO?kb(pyxH6wHvs_4zq<6!_9F1MaI`#QI$-Od# zu!Oo297q!vP=S-^YJ9pfiqwtER%A(lFh`%HenEciY`^y`QLENpC$`g$)3z8lgw;fg zW_~Yr;+DXWw}dJ_MQiQtOMf@q)NZJpJpPG>Cahazy)xo^{k4Ph^@4=w(^7lWEa@-F zYbeVyll`)_lq-O+la&$N2*{V!XK{G|?Ez*oSXVdstOOF5ESk-ZDsAZMeak~#XQy-@ zZx1<{(){Kps#F^)J?UttvY;`J%)JY3{|F$qB|`bu_S?lmSOvr^lm0c4jOKpI!tiiP zJlRhnanK#WrUYQB!P=gP%qSC+_fGk#`TA2NUd9!MpvhjTD~0zpZkmtGLM>IrsT((q28G06;9R_@tEAK(WcA;pM51F*ncn!Bkwd+MIX7EG;q=jbWb)kaRb_V?7^0%3$dv|)UqHyw22tz z#l6^&dNUfjXysID5lQ|_V{)Cpg+)C7geSBk^$3{*1Sl>O~@Hk~{;aaLJX{_11z1KKh z%!TQA z{kyB19F#6(KmR<~QQ<@^cp2l8gFV27BF)M$Nn`WX&mw|*V4sVaI@Hk*o=4Cx2{Y*} ztKxAfgoil40)NFIc}zjPV}2S)zj0!Hih8Kz`JV}iG^m}1`1((Uv3G1pma_Xto`Rx( zTaIVl=BI%m%mm1KIIxQcLj+yh8!-AOviMl@CIYdPVLda<<8ZMgc-~0dL#_bty{vrX z%EGk^TJ=gow=BhlVgJNHbMujvjF^4j5vi@SyG*F26v^O8Q zZ)?>N-QSaLB4V#54lLCX4T;R=g^xb--B^1am14DwO~^!;{udzv|GaZ$f|!g~c#A&Y z(VHyYm%3CPHw{IPZwo}3+ZlyxY5+<|Z`_HAG4Wpw$TL=V^u23hY?t^J@IXB9)mt~2 zC`9p=Sv_;UPocA^AA3kPSoo<2*oQ$~MB+gQZ!W;(J=tvy4WTlQcmHdc!0P5}Sn1O+ zF}wkEWx;Z_LniTtoAl38dx)IXXVS(tUt#+Ghk60rh%n910bz^=PssoG#w1!+Iy3VU z1ytkj>Kx{cWBpRq!0@h^f-lkIJy2N!LlLFSwQT?<2xAs2R-(^{yaY(;>;PD}Tl?jT z-OK0e=8mkr%3nx6{99bE1A#~%Ps&V>_IY?nep(IZ7uoMz7JA>u_wAMGNK7LMIzHm< z7fVmzeg*cew5#@-3F zhTh=iWh60;s*{QR3EYxF=2go++Det^BV(zcykVa}nnnj=b$Izli-Py<=FehpfWd!D z-+Dv}Ye}lSShi!qoU<%5%R}&K**)Y!AEk)V*Is%u&xc()V($LEBbI=||CFjQK`z8j zdl6Z?e%!h!xbI!fzHv)PP8auZ4Bg8SR;avHoXeLqN}?Zpn>we=x!zSQ^5@A>em z{-h|slbEPjj#GZPx}2T&t6RVBgx=`wGRbxSPec6=*6@G&?!PtsqS1fP7@0gb)h9*% zqC_>&?Hkm}$9X_t-Cg@AFE-LAaT=^e_QSMnY&%aF|Kvmg?w6#z70WKNSh7Z%&o5Y- zYy4gqp%THOa^=*)tWJ_q!F4W1}Mz+kF-r8*0E$6>`gFYgj@|UZEGDKPaGVO0L#foFJ^w%$Q*w4*%Yey9ue&pr7<86To{`6Hu0orGxMZ4VeAJkXkI1_yw=^~ zC|MiO?DKM{?w!gbxxFaQX+ub9#sv!YQ#2>*S?YY*rSH)r?+c%XNzosksyag1!ZT#= zy8^>&Fg#Btg231Tt*9eI2$XdGhWB)Ut@uO2Wcb8sj|1zO=c6-Kd@J`ivkx;raUV|7 zaq;L_9>WP~QO`w4u%R69q+PsrRTTOP(C5D>Q2YHZ{fq4}0j{}>P|5IDK{56s_Rqg8 zvTJMZStCw<{2g0bFtnDPhd5-VuE0kADjY<$kcDtLc?P{6R}>bE46nePtVjlz*q{cW zFt#Qgv-mzOx?FFGa-P9g7p!}_t(51RyAyd%A=hW5KkXs7D2ax^9y~SFT!dBk86vDcm`+kUsy<%^*pK7 zU(BiLt0|E8u;%%s0?zu>e1F`>TH)oiEBlhEd3fH*xzZN~(APC>-hlNurbIK-SS`PQ zjCznZo|6WmCzZ=yG}qNvBa9r~s7~>7eyhYZRW@N!@9C)xF1Gou?D&f?$qm4mMH21= z1x!gjt(e1QwVyXQ%Z_hLwx5_4NpE-DQ>X8&xT!5@T{?2d$1ud|y$uLkN0mhza{W%L z=*oj9^G|pVnJ49w#w!VTLdLRghujAv!%{yT*Sf7tx*6_Y?)@Fm(XOz2aTXHvt217r zAjHI1sMh+2E9>UIvY>zXQLAqzbzLpIVbo;}EXrFuQI=nem?zkl%8Ih3^EnPMYTnZ3 zi_?RO4^)12mV4`ymr=-nk})xNs36O9l96coe0WBEIOoec|CCBs;SmaZID;ZchYW?# z;A~f5Ol=gOUV-Rw_Iv0f2dqu9uXGw!G*hPsc$5cW33oxg`_gQ}Q5u=EF#ZE-s`P)-(<>0$Hyx_#-@f54=M!Pi+Rr}#=|AdQR-d+ zYvr>c6YdF-|SGfx)8&3&pYZrt*WpZOd=2jMIJM zbrzQwOND8z0P}a(b%Vu;MrwKYT*Rlbdmr$D-#wf)52 z6#!QF8(zHB1inr}0lwGlL{>!R4w+~)xOh%H>zEQ&3T7{;bmUKgClvg2{TnC@uE5u5 zYGyiRTkyM&6)o-pe-5)%9s8LgH|1L-f~k383FH`-jx0LoOp39HPn%eWNZ)GN^y+Lc zHxY3GqiU$w{Id$u>+2TzN$#9s!%wj-t<6MK6D?}gdj1g0Ne~$mo9;d`ofLEJ3R~;< z9a=|yN?v|4Um``!dOz3oZs1>IJGw)ZN3+L$_6#X;yjr<|Nrgbo0QXAj%FL>Zth@4(gz~k_NHH~q4h89Rx7hx)TwuHF*O^kO z$$3H_^zqZiq863Jt)o`|q}7VpAnWW{ag!<0{d@Ld0u8z{)51I1%!=7c$Y+9CZc{y9 z_hNI$)6fU8i;u9)(agz3+cHJ3YIK6I;ClPA_qBB!1Jk&$;;4Y+)O{_kx14 zLpi$uh1+Q`zsg*t!eQRuVa&($6Q)-6IgQwa$LuCPZ#)%8)?%h1)Twru#C;$7cV}-^HF}Lm8{Kz=b3^ zPsA6$I{Mjai`RBkPt~->=DSwSrC{!>Wd#<@Qd+6)TATG#+k$k@$s?3f;xG*ahxm^= zAMnU-&_gC5Z+g?p%ogNv8k3#KP74G5)j^6^!@jS-7;#t6mUXE=NR%a;aE0zvp8Gg0 zJB*j%9$WSJbLZY$@3_rGlhg2s$hQaWK0W^}I6z|6#4t~$(ge%)Xb3j&)SW@RHdpg_ z=EH#@*Kh6)PrM==MDE|bdyIGWCRd2-@EU4hixkJZkaeJfj#%2%NE3Y~=e!wnzR#ro zkxHtuo^P*EFGrPV)LB?c^n=&7!pPrW1_ANe5Tx_3I-+>;!hUa_t{Ip{HYpJ(-l&TZhy+ei%e=cMqfmLKb; zq10gE$(|7_*v=H`kXz5o;)jFV&nQ=$m7hOB%Frs*y!lLdUz;UB4-aNZh&4<+B@R_f zHH3AeP5-`>t6$+$;s<3Xvzo7ecztH%$05#OJY}y882oLwVv-n_niC$sHh4#(1;l@X z|A|C`_iwCxV4sauOY#(Kw2=64pVQ$W9@hpF#@kJHv#TQP_xpD>mc@VAsO~BDwJ&QN zFpenWJb5nbicr|LiAL8bs~vo2BvTMuU&SilnX6t6%H)lV(a-w)lZ~z`R=-D{T7=Of zFwGz;Whr~PoM~L;xrf`N$Si4&LwYt5M2N3*&_1)BZ-nAf1L6S^YrFTbqUz+y)_u)7 zXt7Y)IWC3RWVfY!n@`i6Yb*C-eW4fUHb{>tC$N+gDD##efp4p6i!I|~0b%_mz(HU1 zTM={9wEHEYF1z_Q(#lx3nBhO!ION+*i!shW*&JOjUgwI<%q6Ka9377zqlMmo|LM32 z)Ma$3J1!-1&Obldnu**Vnf7+V)z4)EL18ZD_k|U(uEPZ8S&j5hHq+{C5=6V+B#OQ4 z-EFC`WQ4@IL?Hq-E3mT`bGXXae;Myb*Pv3uy)ltHR+%!9A|xWuS)57KVtw(GKi_u^ zc|f~99eZJP{K)G~x!0XsB{mLl)GW#Tr8z^1)c3gXVV!;|=ZWvGX_m&0P!Y{r2AtFT7fpg_Fsr6Pu|4hK33ZJb`xkTlDsgZ z-B3M04~u&Q#^pTS=imJ0^R|W!X7d|F5ObPv&6d)#ecR5p{?ZckM}|lfgs(41fThWB z3W1l{8kMWDghR`6-s^dRJDozGHh)6$laR`>gWLunzNmU}f5k5yO#p{ zJo29(o_ZQrJYx6)<9a_wB-cciOtU=kT#-qyKhj?{^MiOg=&P6#BRS+Py(S;54Vv4`G|ay!1X*E12~Qzmdw)Sh znWx%8ryyGP^Dz1=C@QYSe~Y}TSXM?2&?aSDK>c!J4;=fuW<3IGmcUtV;l&Bx(6`Lp z$-B`4{!uLxBFxEtVIv8JUqX6li+Qzv$}7l@sLo?1->JqEC3b#53Gd+NQ?#87Q9|sD zW;;g&4UYe+WuBl_3+U8Hzo=$edMl`Ut2OSlt5#BaA-txqsW6x0y^+{!N6xJ_0fUGY zX~P={TGdA3uTyI!G@XuI+{{@T8Q0!k-oCdtJ+L>cGDo5QWIM9MBG@&!q@Jx}=!RmGh{R;eEP*P5(drGA3l2Km4$-t(KBJ<8y^3#G3FLk{xy!E9? zpT;(|C6ni`+X^CU(0E7+>ig}jX;T^=3j*?fL7O{{dAYbsXU)(3|{%q_)kq+LJh*B2DteXMsm^4+w%SzqeoceeJ7 z)o32$zO0xhi)943d3{-0L10JXC_+?qzah}y7ZS{j**(uv`RWZ4s)^8u61d$9{1dV4 zK3w{Ph9>h*uH;rqRaUWEOW(F^va@ePw23gc=)Z z^>MsG{jifPyLSs|Rt05E3N^)NlowQ5^6oWx22QJh&B2W3%1-fW*9g%ly1qxdwrjw4 z=L!P4GPF9Hc{kZqmuvqg+tn3G5H@i;Q%RV0fDTC^=zjHv15t|z{2@jdV%P>(Y`kwFmDVq2B~1T?Ub`>{WxafnMardmTIjBkYfW3AyJ zJUB#f&5rSWPL4#jPAdAz=HI`v_|EF-{6oY_ZdMI&lzrrtSRx5_s9J%i6$HBQp#GaZ zh^@=vKT2$0?KPS;gR*mUx< zp4NF3Cv+ymsRC@VYl7BeWQ7#F>=1AWPv;UnPTO>Y+~F1-X2k0pc*JDrqttJ0DZe}a z?kt&B!o>j=Y%j?tfCvOl9lz`4+Xl^_>a2-^*rNY(76NA^jHaa&>nr;Y7q8b<5p&!r zp4Rw`8p1G8~d9DZsT^f5%zy@ zeXcT*p191B5xY8xTpF67goz0-?yK*MKS!HTQxViP>E9ho)wx#&9WD^T z*!h_HKWX@of7~TKjeuN6wY1p^_NjI2k8k$rg=we1=r0u~;0-ILOSa%A(gN$P#o&aC zIQ6w*Zep1k6t;mfZH>gzi<6dWy~FG*eTZazzJ>L zBV&MNJg08fIu~oguQ>xRK__;Fr}ed#w#4 z{?HjR#%Caj>@P-TRBgiP&MiEgs3H-}iAJ!|uT!dGO6!(h-F58>)_xIX_G~Y|H^*u8 zkNlgzo4{t~+g;r`$YFX`Jq->Y)3*cN`q-C454dl7Lc?MVx|i>iv2+0crWJv65^X~X zmg_JK+Cf6h{+8vOMd3v37(U|kQvr#a*x1eGhwVvbRNpf2kQ+iQF_oqpu`kPb z8p2YdB4E}@TUY3x&C*QM-yL{drRjI&!69`U9=sVFQ1+6`$`)4^*L*AI%D(UVR_*R! z?99EGBfDx$4QhBO%bJV$@-M)Vae|`IpYYbX;~2M&M<+-BM@=e|ne^7j9AI(87>8ll zwPIQid@06!%S!A-5-{bmm&RHg+$}K zD0Ex8a92}dk+CIwY(pb8rvP2PfCg(@A&xLIZ5#K410E>U8?mV7ePT& zV|2w>@iww^%WZvR-C_$l)4yN8v#!KUZ-kt}gzEBuZ@(RZs|%zJmipyBzO^76tzl6G z&CBuou!MWrCOAs$XuF93DbMQQNXTGaZf{-YePLLrB;(=$LCo$feA#w$92&rkp^a!A zv;4y)lQSxO?F^(%`}SGxgv~Cd7RCv57(nt|gi4?7>o8REu0?4mvBz`M`{Xg4?s0{t zPY&_z&tc%lAU@XJqH$*1%Ho|hcd-4)25uL8ec8DO8a*L>YU_RJ55#UI7Vx4Ll@J_0cGn`;mGAj;yj>9kGezjY7?Cb1P~I$o#6_cozk=qGHpT`% ze)#4DuT_|N^x~Ri=pn=1(;d@cgBO&s_+6StFj_vBrNs24bsrhBsZOwmGu1fk3~ohE z`6YE9Uk!~PSBM;U5G^?q3vr0fe!qX&6&O)|PJB3aiBy*yVIJ|2nE=nfCxvZGg(IIl z`mmfHNE$ViBqVGdkNh$5t}2lB1SZSO*?h*pMB_zyLb1x!{&&!oT`+$TUtg)%XqEis zNE<|wVdrT(k|2MZh-uFIl%7#Z0T7ix+6 zg-&9M7XWn9u?qT6bW(St@^5sq`Zqdp*V@(lz{Kq8lDpr}*9?itFi3r4Hr<}bT;06u zogG!$%w?Uky{4X24u>0**)P(1USgFF&hK-7lddFwy`WN4te)($mBdKBS$f7g=46(t zqRq1P2mnkB^I3P8FMhJ|vh1XblH_w;Uo~-srRGWI->s+${llYK~pKoSco`u_a-=CMfDxU}!C!LV|?kr#St zB8z1mb9WDJM__O*j4hnndq@21&;S+E0&HFo;MoO-}_yp^|#HI9(cK`!J_PC2dhUG zMe65-Io54fIeqyVMqYf0Z#u>Gg}$GLrAFxZP0i>)X}ao{*9+g)>MIPSG!2nGd8uH+q@R z^6(%!)!=sS&C8SfFn9S}#%i0#cBT%lM+{Z%kB6@-%A`@7mCkIl{kNSQRo&k zUCCEfJ(OWI$z_S;B>iP_;nMu83X~PxCd#`csni*5RNVe!- zEu&@e8&1Z0g*EiRrsPbHj3w8nW@TH2 zGk<>GSt6S)-i0Fqb{vPo@j6s44w`RB0hgI6)U5wO@XEr1O_usgn^pj79M z%N?tgMr82rxkPKg$M@TKtxcozM@-)emwo^9u|_RBOM&|dN5t&hpc&mJl- zH){vN!%QCfc>5h@@mrKV?6hJ5;B!2T|H*cz0hh$VwB1&CboDqi=f<4leaIG1MeMhZ zgFKO-LlDvVCuqA~yp&RefDIG#R9Iy1;m)`ToJrn5*nxAO3rp+24#Ft6EID9R*cc<( zQR52u0WkD^cQF#rxc?5}WL|X*@{LAyKhStn=9fLf0oWt&H$ai&=LgEcj!%7x{BU9y z5j!J$S?4yZW=q>V1lV`KS#7JYx0{`Jpf{P@?;}m1E0`dXYZsZkFrFFYT#I(uOGS&) zDJ=w9ESAAniOOzC7#WFh2SanmxSaG~>3O`Kr{l;7&R-0R`1zNjt_99D2Q!e~-YQ%k zqT0$(8;V%x$g3g@C@DVRx9&S9uNgUdL+miumQZf8$C>v%XJ#WInm`0Q3xojK{DNFl zhB>t#g&hIN1?LA>czOe!90$Htt)uuh879XiJ1w1aQxn0j&XJT$}nfX? zVAwWq`T3Hk+%bc60$8!}&eHZW0QI2oqhVu=@F5e5! zgB~ge>$xzCCn>Rb1l57M>|kqBs(zil_EAMm*vxu!{X-26cHwT%58l68&-oizFL*I> zYHVn%y!_SVyU0LZm{%15Sa>c4he?1x2`6$copwLVZSLi7-Mjy4 zYn;mKZDT~m(Tp|!p;o06s1RU(2)*VlW;5n!%M7);(_C6@_i|=S_gYaHqU*qRM7x`n zhbc7Oj&IM0NGUK+4yeN>Ki%@)B^GB}EmEo=*a_X?p7xzH0F?5%hd0X0PY--Ti^}A- za!#F*Mj@n?8GoE)D`RAqk-o=2m&+30OWTMG&#uC>Du}OAgWm6-?Gx|!;UJzKbBT}d ztIjJEuTIUOjURhvjT(Vh_)7kD9VYD>ba7xE$qo1C%A_RZ!SxSekkb-=<0i>bGiNKQ z5xR0ht7&$iwzMxHf5C+nLKmT~b@fo}lDzOKxzNtm>nA_@R>(|C=&ao(c2jnKz}|gJ z%)=AQi5Max&UYX-VG=W3(3)0JLl4|9&l`1JvmIty>+=hwan)<<$&5(5ybH+n>OHHy zV)&3lYl|TY-#@9uybf%QP1pmBHg+p_-J1Q8sl9NK_Tl|*-0YXGRSHFrhQpgOCff*# z^oE}B@(m8z>f(-0=fQLG-O`5{w)0;HD~| z3fj%RcFUMcT>zE@Vu{@DE@I$~$*fezJ+2v@-G~ZtOu>%tb-| zh0Sq+SHPUb`#!NqcDC&1hWgDI#visvTS35X**>%qwLz(6M>>~mpRMYD^CabcnCUw| zCEti#S2ihp`=geO^Hf86-3rSD2t3$dq?6~;+VT(q@agMFsead_;y|#4LuV|U^wRvi z;y^P#`pCfK^T__Ox?UahCaYw5af-wDCtLaa#`>C$bsLcl&6BSbw=OCSETN|%*+y;k zieRuFj(ZD_x0K0U4?-VWhqzS+r{}&=IcUbt2SAxdOxUJY0m+V|`3Li@k1iXf<#h5+ z-ZK6Jta<1YT>O~jn6S+C^Hg9>MznAK6|vhiB{D}lL7u+gQkPYmFpR{-yYe=Z<tqb?ale_A3IBQdkzkzosn9@=DAp2}7c83UyPNoXTd)?A=vi zYEy~fXbU);CjW?`8!m&x@s6T!Ms>nnF=r#G68%(UHCgaio&%F8;@d((1?4cHBwOrk zblD2lTz#^QiwMP3wL%8b8?|2!Qg!_!DbnL9^CbI!Vc}(Xzqo_-hxH)pgUn{qo@gUV z8^Ax=KxSEXpmv_aEC|zq#%0}<_7Ih}2<&n08>T!?PQJ(?KYt=IL(g5D=U~`wbe~Br z_Z{chs?FG6q?Dz}yekt97AbZi6AG!`_*lk9fa{4k+?d4wAf;-6bMXUNV;`L-QcI-j{k%H~}NI#@yXS=tn+6MZJ{yqWpm zqY6n<{8xy|f=%K{W#H1qu1r?RP-nnMXO~4~;!n1EBuO`GaC=)6)M|c!>C{e*Yr&ji zdU08`&9*89A_9>`3n%ZlN!6fkqupNVRoHa2!`<6K%1tr^MtLLX(0tV7-ckLJjN^bi zrosREaq`ew^PqoPLzi!r;5nPbiIEF`<R_Ub~b%wLT`(SYDp1R ztD2mtb^uc9RH;hI3mbkEU7@!ajtN;$$mMs$OU=Z!wO2BJ4$KG=TeprmsJo%ui6 zhBkuI7yO<7r3J^6;c#Bka-l0#B+-mrv{6%&Rb2dc+z`)592*4UhNvdoB61TIPVULV z2{qpT-N5|jyT+B_s+Kd9^1|Gbm87Bme%2NwaxbS8BH~KqhVU6tdyXuS)ys=ec1jZX zKzs5HiRCRoK=Xc9uYL34tO+2Xo2r+#Yga`8v|CC9Fk$Gv#McLd7alPrx02FA2{{

IlN1yzX&Vt8zC|@9?`Mc}G;Is%vfWZvtQ8 zdG}0iSw=c8As~?Kx8l_A5pA^~hwd{5%2iT3k#%M7O_0#Mux_-#wQQ1W(Y!)=3sOC= z&=xP|l&n_AyHNA_o+C8zjLMR7LNZtA)V^v^7lE6Jq*6}Xl9`diU~}rH<>gkKm_na> zSe1mKc&*_&6S}M(_M3W4>yLe4*jp3@eWjf6N!hSjRr|?ySxx_&Ej=)D4Be0J#ZnC0 zzyJnS#vwa#GHPH8MtfV3Dt*^w2dKVJk>Yxyq>g?ChlYBais`7R+?VE>cExdOBO(TQ zj;H%#HhN**6sT|i)UFHMr@R=KpJYIQ1DPx97IPqd9mM@L8J66t#SJW#FI*5#Y*la# z5PxhHYyglmvBSJa$U@VjxuT!-c|LjkUzWpqOu*(VAk>Rwr!bM0gA=v=Y8)tq@|S(f zyfYi;lv9Fwy(-xA0drdHw>hnRwEO%d!`^mZ;3u0nq&peY8@LNHL=8>wJYOd<_Cs>+ zGegm)6CzQDSd+zVDqP3Zen$IV^oid|##^_&b#c7WH(=-e}SQfPvh zVBd7D&yNN#sL7UM&}jUr>5rFh5!JbsNuJ|!%MX`Uq{CU zt=*(&s1oi47F`VLHjzDDd;p V_A2wg&9wa2`Ii6k8T51b{{U&ecFh0) -- Gitee From 046e0c18f2b1efcb1210c4aa195de83cf742e208 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Fri, 10 Mar 2023 02:13:22 +0000 Subject: [PATCH 17/99] =?UTF-8?q?add=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/spark.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 6b2d3c9f954577a30f5000f36d15ccde837424a5 Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 02:18:13 +0000 Subject: [PATCH 18/99] update README.md. Signed-off-by: xiexing --- README.md | 108 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 72 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index 3c2183c..cb8164b 100644 --- a/README.md +++ b/README.md @@ -14,12 +14,10 @@ ## 会议 - 会议主题： OpenEuler bigdata meetup -- 会议时间： 每周四 16:00-17:00(GMT+08:00) 中国标准时间-北京 每周一次 + 按需出发， +- 会议时间： 每周四 16:00-17:00(GMT+08:00) 中国标准时间-北京 每两周一次 + 按需出发，详情可见订阅邮箱。 - 点击链接入会，或添加至会议列表：xxxxxxxx - 会议 ID：xxxxxx -- 手机号一键拨号入会 - 订阅邮箱：bigdata@openeuler.org -- 复制该信息，打开手机腾讯会议v2.13以上版本即可参与 ## 成员 @@ -28,48 +26,86 @@ - 吴泽艺 [wuzeyi1](https://gitee.com/wuzeyi1) email: zaki.wu@huawei.com - 陈强 [macchen1](https://gitee.com/macchen1) email: mac.chenqiang@huawei.com -### Committer列表 - -- sinever - -# 项目清单 -| 项目 | 版本 | 状态 | 类别 | -| ---- | ---- | ---- | ---- | -| [hadoop](https://gitee.com/src-openeuler/hadoop) | 2.3 | 已落版本 | 说明 | -| [hadoop-3.1](https://gitee.com/src-openeuler/hadoop-3.1) | 3.1 | 已落版本 | 说明 | -| [hbase](https://gitee.com/src-openeuler/hbase) | 3.1 | 已落版本 | 说明 | -| [hive](https://gitee.com/src-openeuler/hive) | 3.1 | 已落版本 | 说明 | -| [spark](https://gitee.com/src-openeuler/spark) | 3.1 | 已落版本 | 说明 | -| [flink](https://gitee.com/src-openeuler/flink) | 3.1 | 已落版本 | 说明 | -| [libhdfs](https://gitee.com/src-openeuler/libhdfs) | 3.1 | 已落版本 | 说明 | -| [zookeeper](https://gitee.com/src-openeuler/zookeeper) | 3.1 | 已落版本 | 说明 | -| [kafka](https://gitee.com/src-openeuler/kafka) | 3.1 | 已落版本 | 说明 | -| [zeppelin](https://gitee.com/src-openeuler/zeppelin) | 3.1 | 已落版本 | 说明 | -| [presto](https://gitee.com/src-openeuler/presto) | 3.1 | 已落版本 | 说明 | -| [rain](https://gitee.com/src-openeuler/rain) | 3.1 | 已落版本 | 说明 | -| [ignite](https://gitee.com/src-openeuler/ignite) | 3.1 | 已落版本 | 说明 | -| [flume](https://gitee.com/src-openeuler/flume) | 3.1 | 已落版本 | 说明 | -| [storm](https://gitee.com/src-openeuler/storm) | 3.1 | 已落版本 | 说明 | -| [delta](https://gitee.com/src-openeuler/delta) | 3.1 | 已落版本 | 说明 | -| [iceberg](https://gitee.com/src-openeuler/iceberg) | 3.1 | 已落版本 | 说明 | -| [hudi](https://gitee.com/src-openeuler/hudi) | 3.1 | 已落版本 | 说明 | -| [hue](https://gitee.com/src-openeuler/hue) | 3.1 | 已落版本 | 说明 | -| [oozie](https://gitee.com/src-openeuler/oozie) | 3.1 | 已落版本 | 说明 | -| [ycsb](https://gitee.com/src-openeuler/ycsb) | 3.1 | 已落版本 | 说明 | -| [bigtop](https://gitee.com/src-openeuler/bigtop) | 3.1 | 计划引入中 | 说明 | - # 创新项目 | 项目 | 版本 | 状态 | 类别 | | ---- | ---- | ---- | ---- | | [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | -| [openEuler开源实习](https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata)||| + +# 开源实习 +openEuler开源实习是openEuler社区和社区合作单位共同发起的线上实习项目，旨在鼓励在校学生积极参与开源社区，在实际的开源环境中提升实践能力，在社区中成长为优秀的开源人才。由社区提供实习任务，并提供导师辅导，学生通过实习申请后，可在社区领取任务，每完成一个任务可获得相应积分，积分累计达规定量后，可获得实习证明和实习工资。openeuler社区发布大量的实习任务，欢迎大家积极参与。 + +openEuler开源实习地址：https://www.openeuler.org/zh/internship/ + +openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata + +## 项目清单： + +| 项目 | 版本 | 状态 | +| ---------------- |---------------- |---------------- | +| [accumulo](https://gitee.com/src-openeuler/accumulo) | 2.0.1 | 已落版本 | +| [alluxio](https://gitee.com/src-openeuler/alluxio) | 2.7.0 | 已落版本 | +| [ambari](https://gitee.com/src-openeuler/ambari) | 2.7.6 | 已落版本 | +| [apache-atlas](https://gitee.com/src-openeuler/apache-atlas) | 2.1.0 | 已落版本 | +| [apache-ranger](https://gitee.com/src-openeuler/apache-ranger) | 2.1.0 | 已落版本 | +| [avor](https://gitee.com/src-openeuler/avor) | 无 | 已建仓 | +| [bigtop](https://gitee.com/src-openeuler/bigtop) | 无 | 已建仓 | +| [blaze](https://gitee.com/src-openeuler/blaze) | 0.10.1 | 已落版本 | +| [calcite](https://gitee.com/src-openeuler/calcite) | 1.30.0 | 已落版本 | +| [datafu](https://gitee.com/src-openeuler/datafu) | 1.6.1 | 已落版本 | +| [delta](https://gitee.com/src-openeuler/delta) | 1.2.1 | 已落版本 | +| [druid](https://gitee.com/src-openeuler/druid) | 0.22.1 | 已落版本 | +| [flink](https://gitee.com/src-openeuler/flink) | 1.13.0 | 已落版本 | +| [flume](https://gitee.com/src-openeuler/flume) | 1.10.0 | 已落版本 | +| [giraph](https://gitee.com/src-openeuler/giraph) | 1.3.0 | 已落版本 | +| [hadoop](https://gitee.com/src-openeuler/hadoop) | 3.3.4 | 已落版本 | +| [hadoop-3.1](https://gitee.com/src-openeuler/hadoop-3.1) | 3.1.4 | 已落版本 | +| [hbase](https://gitee.com/src-openeuler/hbase) | 2.2.5 | 已落版本 | +| [hive](https://gitee.com/src-openeuler/hive) | 3.1.3 | 已落版本 | +| [hudi](https://gitee.com/src-openeuler/hudi) | 0.11.0 | 已落版本 | +| [hue](https://gitee.com/src-openeuler/hue) | 无 | 已建仓 | +| [ibis](https://gitee.com/src-openeuler/ibis) | 1.3.0 | 已落版本 | +| [iceberg](https://gitee.com/src-openeuler/iceberg) | 0.13.0 | 已落版本 | +| [ignite](https://gitee.com/src-openeuler/ignite) | 2.9.0 | 已落版本 | +| [impala](https://gitee.com/src-openeuler/impala) | 无 | 已建仓 | +| [kafka](https://gitee.com/src-openeuler/kafka) | 2.8.2 | 已落版本 | +| [kite-sdk](https://gitee.com/src-openeuler/kite-sdk) | 无 | 已建仓 | +| [knox](https://gitee.com/src-openeuler/knox) | 1.6.1 | 已落版本 | +| [kudu](https://gitee.com/src-openeuler/kudu) | 无 | 已建仓 | +| [libhdfs](https://gitee.com/src-openeuler/libhdfs) | 无 | 已建仓 | +| [llama](https://gitee.com/src-openeuler/llama) | 无 | 已建仓 | +| [mahout](https://gitee.com/src-openeuler/mahout) | 14.1 | 已落版本 | +| [oozie](https://gitee.com/src-openeuler/oozie) | 5.2.1 | 已落版本 | +| [parquet-format](https://gitee.com/src-openeuler/parquet-format) | 2.9.0 | 已落版本 | +| [presto](https://gitee.com/src-openeuler/presto) | 0.26 | 已落版本 | +| [qfs](https://gitee.com/src-openeuler/qfs) | 2.2.4 | 已落版本 | +| [redis5](https://gitee.com/src-openeuler/redis5) | 5.0.4 | 已落版本 | +| [redis6](https://gitee.com/src-openeuler/redis6) | 6.2.7 | 已落版本 | +| [spark](https://gitee.com/src-openeuler/spark) | 3.2.2 | 已落版本 | +| [storm](https://gitee.com/src-openeuler/storm) | 1.2.4 | 已落版本 | +| [ycsb](https://gitee.com/src-openeuler/ycsb) | 0.17.0 | 已落版本 | +| [zeppelin](https://gitee.com/src-openeuler/zeppelin) | 无 | 已建仓 | +| [zookeeper](https://gitee.com/src-openeuler/zookeeper) | 3.8.0 | 已落版本 | + +### 说明 + +| 组件状态 | 说明 | +| -------- | ---------------------------------------------------------- | +| 已落版本 | 该组件已落openEuler发行版本 | +| 引入中 | 该组件已在openEuler建仓且代码正在从上游社区引入，待进入openEuler发行版本 | +| 孵化中 | 该组件由BIGDATA SIG所孵化，待进入openEuler发行版 | +| 已建仓 | 该组件在openEuler中已建仓，尚需从上游社区引入源码. Help wanted~ | +| 待引入 | 该组件有需求，但尚未在openEuler中建仓 | + ## 链接 -openEuler SDS sig组简介: +openEuler bigdata sig组 简介: https://gitee.com/openeuler/community/tree/master/sig/bigdata + +openEuler bigdata 月刊归档地址： 邮件列表归档地址： -openEuler bigdata 月刊归档地址： +会议纪要地址： + ## 联系方式 -- Gitee From f3720cf51edbdff25ef27400c715e5df783a9d8a Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 02:21:43 +0000 Subject: [PATCH 19/99] add readme-English.md. Signed-off-by: xiexing --- readme-English.md | 116 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 readme-English.md diff --git a/readme-English.md b/readme-English.md new file mode 100644 index 0000000..b67f0dc --- /dev/null +++ b/readme-English.md @@ -0,0 +1,116 @@ +# bigdata + + +## 愿景 +构建和完善openEuler社区下的大数据生态，打造活跃的大数据交流平台，丰富和提升大数据组件功能和性能， +发掘用户大数据需求，孵化有潜力的大数据组件。 + +## 业务范围 + - openEuler上的大数据基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 + - openEuler上大数据平台，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 + - openEuler上的大数据组件和平台的性能优化，让大数据在openEuler上更好用。 + - openEuler上的大数据相关能力集成，新芯片和软件进入openEuler时的支持。 + +## 会议 + +- 会议主题： OpenEuler bigdata meetup +- 会议时间： 每周四 16:00-17:00(GMT+08:00) 中国标准时间-北京 每两周一次 + 按需出发，详情可见订阅邮箱。 +- 点击链接入会，或添加至会议列表：xxxxxxxx +- 会议 ID：xxxxxx +- 订阅邮箱：bigdata@openeuler.org + +## 成员 + +### Maintainer列表 + - 杨昭 [yangzhao_kl](https://gitee.com/yangzhao_kl) email: yangzhao1@kylinos.cn + - 吴泽艺 [wuzeyi1](https://gitee.com/wuzeyi1) email: zaki.wu@huawei.com + - 陈强 [macchen1](https://gitee.com/macchen1) email: mac.chenqiang@huawei.com + +# 创新项目 +| 项目 | 版本 | 状态 | 类别 | +| ---- | ---- | ---- | ---- | +| [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | + +# 开源实习 +openEuler开源实习是openEuler社区和社区合作单位共同发起的线上实习项目，旨在鼓励在校学生积极参与开源社区，在实际的开源环境中提升实践能力，在社区中成长为优秀的开源人才。由社区提供实习任务，并提供导师辅导，学生通过实习申请后，可在社区领取任务，每完成一个任务可获得相应积分，积分累计达规定量后，可获得实习证明和实习工资。openeuler社区发布大量的实习任务，欢迎大家积极参与。 + +openEuler开源实习地址：https://www.openeuler.org/zh/internship/ + +openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata + +## 项目清单： + +| 项目 | 版本 | 状态 | +| ---------------- |---------------- |---------------- | +| [accumulo](https://gitee.com/src-openeuler/accumulo) | 2.0.1 | 已落版本 | +| [alluxio](https://gitee.com/src-openeuler/alluxio) | 2.7.0 | 已落版本 | +| [ambari](https://gitee.com/src-openeuler/ambari) | 2.7.6 | 已落版本 | +| [apache-atlas](https://gitee.com/src-openeuler/apache-atlas) | 2.1.0 | 已落版本 | +| [apache-ranger](https://gitee.com/src-openeuler/apache-ranger) | 2.1.0 | 已落版本 | +| [avor](https://gitee.com/src-openeuler/avor) | 无 | 已建仓 | +| [bigtop](https://gitee.com/src-openeuler/bigtop) | 无 | 已建仓 | +| [blaze](https://gitee.com/src-openeuler/blaze) | 0.10.1 | 已落版本 | +| [calcite](https://gitee.com/src-openeuler/calcite) | 1.30.0 | 已落版本 | +| [datafu](https://gitee.com/src-openeuler/datafu) | 1.6.1 | 已落版本 | +| [delta](https://gitee.com/src-openeuler/delta) | 1.2.1 | 已落版本 | +| [druid](https://gitee.com/src-openeuler/druid) | 0.22.1 | 已落版本 | +| [flink](https://gitee.com/src-openeuler/flink) | 1.13.0 | 已落版本 | +| [flume](https://gitee.com/src-openeuler/flume) | 1.10.0 | 已落版本 | +| [giraph](https://gitee.com/src-openeuler/giraph) | 1.3.0 | 已落版本 | +| [hadoop](https://gitee.com/src-openeuler/hadoop) | 3.3.4 | 已落版本 | +| [hadoop-3.1](https://gitee.com/src-openeuler/hadoop-3.1) | 3.1.4 | 已落版本 | +| [hbase](https://gitee.com/src-openeuler/hbase) | 2.2.5 | 已落版本 | +| [hive](https://gitee.com/src-openeuler/hive) | 3.1.3 | 已落版本 | +| [hudi](https://gitee.com/src-openeuler/hudi) | 0.11.0 | 已落版本 | +| [hue](https://gitee.com/src-openeuler/hue) | 无 | 已建仓 | +| [ibis](https://gitee.com/src-openeuler/ibis) | 1.3.0 | 已落版本 | +| [iceberg](https://gitee.com/src-openeuler/iceberg) | 0.13.0 | 已落版本 | +| [ignite](https://gitee.com/src-openeuler/ignite) | 2.9.0 | 已落版本 | +| [impala](https://gitee.com/src-openeuler/impala) | 无 | 已建仓 | +| [kafka](https://gitee.com/src-openeuler/kafka) | 2.8.2 | 已落版本 | +| [kite-sdk](https://gitee.com/src-openeuler/kite-sdk) | 无 | 已建仓 | +| [knox](https://gitee.com/src-openeuler/knox) | 1.6.1 | 已落版本 | +| [kudu](https://gitee.com/src-openeuler/kudu) | 无 | 已建仓 | +| [libhdfs](https://gitee.com/src-openeuler/libhdfs) | 无 | 已建仓 | +| [llama](https://gitee.com/src-openeuler/llama) | 无 | 已建仓 | +| [mahout](https://gitee.com/src-openeuler/mahout) | 14.1 | 已落版本 | +| [oozie](https://gitee.com/src-openeuler/oozie) | 5.2.1 | 已落版本 | +| [parquet-format](https://gitee.com/src-openeuler/parquet-format) | 2.9.0 | 已落版本 | +| [presto](https://gitee.com/src-openeuler/presto) | 0.26 | 已落版本 | +| [qfs](https://gitee.com/src-openeuler/qfs) | 2.2.4 | 已落版本 | +| [redis5](https://gitee.com/src-openeuler/redis5) | 5.0.4 | 已落版本 | +| [redis6](https://gitee.com/src-openeuler/redis6) | 6.2.7 | 已落版本 | +| [spark](https://gitee.com/src-openeuler/spark) | 3.2.2 | 已落版本 | +| [storm](https://gitee.com/src-openeuler/storm) | 1.2.4 | 已落版本 | +| [ycsb](https://gitee.com/src-openeuler/ycsb) | 0.17.0 | 已落版本 | +| [zeppelin](https://gitee.com/src-openeuler/zeppelin) | 无 | 已建仓 | +| [zookeeper](https://gitee.com/src-openeuler/zookeeper) | 3.8.0 | 已落版本 | + +### 说明 + +| 组件状态 | 说明 | +| -------- | ---------------------------------------------------------- | +| 已落版本 | 该组件已落openEuler发行版本 | +| 引入中 | 该组件已在openEuler建仓且代码正在从上游社区引入，待进入openEuler发行版本 | +| 孵化中 | 该组件由BIGDATA SIG所孵化，待进入openEuler发行版 | +| 已建仓 | 该组件在openEuler中已建仓，尚需从上游社区引入源码. Help wanted~ | +| 待引入 | 该组件有需求，但尚未在openEuler中建仓 | + + +## 链接 +openEuler bigdata sig组 简介: https://gitee.com/openeuler/community/tree/master/sig/bigdata + +openEuler bigdata 月刊归档地址： + +邮件列表归档地址： + +会议纪要地址： + + + +## 联系方式 +### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) +### 微信公众号：openEuler-bigdata-sig + +# 路标 + -- Gitee From c215a92fc25d501cc1c5ada3d95e7566ba6c0797 Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 02:35:10 +0000 Subject: [PATCH 20/99] update readme-English.md. Signed-off-by: xiexing --- readme-English.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/readme-English.md b/readme-English.md index b67f0dc..e7714e1 100644 --- a/readme-English.md +++ b/readme-English.md @@ -1,23 +1,23 @@ # bigdata -## 愿景 -构建和完善openEuler社区下的大数据生态，打造活跃的大数据交流平台，丰富和提升大数据组件功能和性能， -发掘用户大数据需求，孵化有潜力的大数据组件。 +## expectation +Build and improve the big data ecosystem in the openEuler community, build an active big data communication platform, and enrich and improve the functions and performance of big data components, +Explore users' big data requirements and incubate potential big data components. -## 业务范围 +## Business Scope - openEuler上的大数据基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 - openEuler上大数据平台，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 - openEuler上的大数据组件和平台的性能优化，让大数据在openEuler上更好用。 - openEuler上的大数据相关能力集成，新芯片和软件进入openEuler时的支持。 -## 会议 +## meeting -- 会议主题： OpenEuler bigdata meetup -- 会议时间： 每周四 16:00-17:00(GMT+08:00) 中国标准时间-北京 每两周一次 + 按需出发，详情可见订阅邮箱。 -- 点击链接入会，或添加至会议列表：xxxxxxxx +- meeting subject： OpenEuler bigdata meetup +- meeting time： Thursday 16:00-17:00(GMT+08:00) China Standard Time - Beijing Once every two weeks + Departure on demand，For details, see the subscription email address.。 - 会议 ID：xxxxxx - 订阅邮箱：bigdata@openeuler.org +- 点击链接入会，或添加至会议列表：xxxxxxxx ## 成员 -- Gitee From 5a1efe29b81e2f0bc37875e4513b5478f9133cd1 Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 02:36:51 +0000 Subject: [PATCH 21/99] =?UTF-8?q?add=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper=E7=A7=BB=E6=A4=8D=E6=8C=87=E5=8D=97.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper\347\247\273\346\244\215\346\214\207\345\215\227.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper\347\247\273\346\244\215\346\214\207\345\215\227.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper\347\247\273\346\244\215\346\214\207\345\215\227.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper\347\247\273\346\244\215\346\214\207\345\215\227.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 0111896e6328c9af6abdbe4a33c6c0e794e2718a Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Fri, 10 Mar 2023 03:21:00 +0000 Subject: [PATCH 22/99] update readme-English.md. Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- readme-English.md | 47 ++++++++++++++++++++++++----------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/readme-English.md b/readme-English.md index e7714e1..1955779 100644 --- a/readme-English.md +++ b/readme-English.md @@ -6,41 +6,42 @@ Build and improve the big data ecosystem in the openEuler community, build an ac Explore users' big data requirements and incubate potential big data components. ## Business Scope - - openEuler上的大数据基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 - - openEuler上大数据平台，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 - - openEuler上的大数据组件和平台的性能优化，让大数据在openEuler上更好用。 - - openEuler上的大数据相关能力集成，新芯片和软件进入openEuler时的支持。 + - OpenEuler provides basic big data running capabilities, including data collection, data transmission, data storage, data analysis, and data visualization. + - OpenEuler integrates various common tools and software to provide a unified user interface, making big data easier to use on OpenEuler. + - OpenEuler optimizes the performance of big data components and makes big data better used on OpenEuler. + - OpenEuler integrates big data capabilities to support new chips and software when entering OpenEuler. -## meeting +## Meeting -- meeting subject： OpenEuler bigdata meetup -- meeting time： Thursday 16:00-17:00(GMT+08:00) China Standard Time - Beijing Once every two weeks + Departure on demand，For details, see the subscription email address.。 -- 会议 ID：xxxxxx -- 订阅邮箱：bigdata@openeuler.org -- 点击链接入会，或添加至会议列表：xxxxxxxx +- Meeting subject： OpenEuler bigdata meetup +- Meeting time： Thursday 16:00-17:00(GMT+08:00) China Standard Time - Beijing Once every two weeks + Departure on demand，For details, see the subscription email address.。 +- Meeting ID：xxxxxx +- Subscription Email Address：bigdata@openeuler.org +- Click the link to join the meeting or add it to the meeting list.：xxxxxxxx -## 成员 +## Member -### Maintainer列表 - - 杨昭 [yangzhao_kl](https://gitee.com/yangzhao_kl) email: yangzhao1@kylinos.cn - - 吴泽艺 [wuzeyi1](https://gitee.com/wuzeyi1) email: zaki.wu@huawei.com - - 陈强 [macchen1](https://gitee.com/macchen1) email: mac.chenqiang@huawei.com +### Maintainer List + + - Zhao Yang [yangzhao_kl](https://gitee.com/yangzhao_kl) email: yangzhao1@kylinos.cn + - ZeYi Wu [wuzeyi1](https://gitee.com/wuzeyi1) email: zaki.wu@huawei.com + - Qiang Chen [macchen1](https://gitee.com/macchen1) email: mac.chenqiang@huawei.com # 创新项目 -| 项目 | 版本 | 状态 | 类别 | +| innovative project | version | status | category | | ---- | ---- | ---- | ---- | -| [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | +| [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | Introducting| math library | -# 开源实习 -openEuler开源实习是openEuler社区和社区合作单位共同发起的线上实习项目，旨在鼓励在校学生积极参与开源社区，在实际的开源环境中提升实践能力，在社区中成长为优秀的开源人才。由社区提供实习任务，并提供导师辅导，学生通过实习申请后，可在社区领取任务，每完成一个任务可获得相应积分，积分累计达规定量后，可获得实习证明和实习工资。openeuler社区发布大量的实习任务，欢迎大家积极参与。 +# Open-source Internship +OpenEuler open-source internship is an online internship program jointly sponsored by the openEuler community and community partners. It aims to encourage students to actively participate in the open source community, improve their practical capabilities in the actual open source environment, and grow into excellent open source talents in the community. The community provides internship tasks and provides tutoring. After passing the internship application, students can receive tasks in the community. They can obtain corresponding points for each task completed. After the accumulated points reach the specified amount, they can obtain internship certificates and internship salaries. The openeuler community has released a large number of internships, and you are welcome to participate actively. -openEuler开源实习地址：https://www.openeuler.org/zh/internship/ +openEuler open-source internship link address：https://www.openeuler.org/zh/internship/ -openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata +openeuler bigda internship tasks link address ：https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata -## 项目清单： +## event list： -| 项目 | 版本 | 状态 | +| event | version | status | | ---------------- |---------------- |---------------- | | [accumulo](https://gitee.com/src-openeuler/accumulo) | 2.0.1 | 已落版本 | | [alluxio](https://gitee.com/src-openeuler/alluxio) | 2.7.0 | 已落版本 | -- Gitee From dab157fea2f69a212c67a2cec41e7e6611359597 Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 03:48:44 +0000 Subject: [PATCH 23/99] =?UTF-8?q?rename=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper=E7=A7=BB=E6=A4=8D=E6=8C=87=E5=8D=97.md=20to?= =?UTF-8?q?=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87=E5=8D=97/zookeeper.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper.md" | 86 +++++++++++++++++++ ...73\346\244\215\346\214\207\345\215\227.md" | 0 2 files changed, 86 insertions(+) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" delete mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper\347\247\273\346\244\215\346\214\207\345\215\227.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" new file mode 100644 index 0000000..89ec539 --- /dev/null +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" @@ -0,0 +1,86 @@ +### **Hadoop-3.3.4 移植指南** +编写日期：2023年03月10日 + +### 1. 介绍 +Zookeeper是一个分布式的，开放源码的分布式应用程序协调服务，是Google的Chubby一个开源的实现，是Hadoop和HBase的重要组件。它是一个为分布式应用提供一致性服务的软件，提供的功能包括：配置维护、域名服务、分布式同步、组服务等。 + +关于Zookeeper的更多信息请访问Zookeeper官网。 + +### 2. 环境要求 +#### 2.1 硬件要求 +| 项目 | 说明 | +| ---------------- |---------------- | +|服务器|对服务器无要求| +|CPU|aarch64架构| +|磁盘分区|对磁盘分区无要求| +|网络|可访问外网| +#### 2.2 软件要求 +| 项目 | 说明 | +| ---------------- |---------------- | +|版本|openeuler 22.03| +|OS|5.10.0-60.65.0.90.oe2203.aarch64| +|GCC|10.3.1| +|OpenJDK|1.8.0_342| +|Maven|3.6.3| +|cmake|3.22.0| +|zookeeper|3.8.1| +### 3. 安装基础库 +#### 3.1 安装gcc maven jdk-1.8.0 + +yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 maven java-1.8.0-openjdk.aarch64 java-1.8.0-openjdk-devel.aarch64 + +#### 3.2 安装依赖 +yum install -y wget vim openssl-devel zlib-devel autoconf automake libtool ant svn make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel + +#### 3.3 配置maven +修改Maven配置文件中的：本地仓路径、远程仓等。 + +配置文件路径：“/etc/maven/settings.xml” + +说明： + +``` + +/path/to/local/repo + + + + huaweimaven + huawei maven + https://repo.huaweicloud.com/repository/maven/ + central + +``` +### 4 移植分析 +| 原始jar包 |so文件 | +| ---------------- |---------------- | +|jline-2.14.6.jar| libjansi.so| + +### 5 编译依赖包 + +### 6 编译Zookeeper + +1. 从官网上下载Zookeeper-release-3.8.1源码并解压。 + + wget https://codeload.github.com/apache/zookeeper/tar.gz/refs/tags/release-3.8.1 + + tar -zxf release-3.8.1.tar.gz + +2. 进入解压后目录并安装依赖。 + + cd zookeeper-release-3.8.1 + + +3. 执行编译。 + + mvn clean package -Dmaven.test.skip=true + +4. 编译成功后将在源码目录下“zookeeper-assembly/target”生成zookeeper-3.8.1.tar.gz包。 + +5. 使用鲲鹏分析扫描工具扫描编译生成的tar包，或者使用find.so脚本查找，确保没有包含有x86的so和jar包。 + + + + + + diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper\347\247\273\346\244\215\346\214\207\345\215\227.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper\347\247\273\346\244\215\346\214\207\345\215\227.md" deleted file mode 100644 index e69de29..0000000 -- Gitee From 5b59e984737632d52e0cdf106a9dc7649fba0caf Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 07:41:50 +0000 Subject: [PATCH 24/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper.md.=20zookeeper=20aarch64=E7=A7=BB=E6=A4=8D?= =?UTF-8?q?=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper.md" | 42 +++++++++---------- 1 file changed, 19 insertions(+), 23 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" index 89ec539..c3bcc8a 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" @@ -55,29 +55,25 @@ yum install -y wget vim openssl-devel zlib-devel autoconf automake libtool ant s | 原始jar包 |so文件 | | ---------------- |---------------- | |jline-2.14.6.jar| libjansi.so| - -### 5 编译依赖包 - -### 6 编译Zookeeper - -1. 从官网上下载Zookeeper-release-3.8.1源码并解压。 - - wget https://codeload.github.com/apache/zookeeper/tar.gz/refs/tags/release-3.8.1 - - tar -zxf release-3.8.1.tar.gz - -2. 进入解压后目录并安装依赖。 - - cd zookeeper-release-3.8.1 - - -3. 执行编译。 - - mvn clean package -Dmaven.test.skip=true - -4. 编译成功后将在源码目录下“zookeeper-assembly/target”生成zookeeper-3.8.1.tar.gz包。 - -5. 使用鲲鹏分析扫描工具扫描编译生成的tar包，或者使用find.so脚本查找，确保没有包含有x86的so和jar包。 +``` +jline-2.14.6.jar aarch64下载地址：https://repo.huaweicloud.com/kunpeng/maven/jline/jline/2.14.6/jline-2.14.6.jar +将下载的jline-2.14.6.jar包放入“maven本地仓库下jline/jline/2.14.6/”下。 +``` +### 5 编译Zookeeper +#### 5.1 从官网上下载Zookeeper-release-3.8.1源码并解压。 +``` +wget https://codeload.github.com/apache/zookeeper/tar.gz/refs/tags/release-3.8.1 +tar -zxf release-3.8.1.tar.gz +``` +#### 5.2 进入解压后目录并安装编译。 +``` +cd zookeeper-release-3.8.1 +mvn clean package -Dmaven.test.skip=true +``` +#### 5.3 zookeeper-3.8.1.tar.gz包路径 +编译成功后将在源码目录下“zookeeper-assembly/target”生成zookeeper-3.8.1.tar.gz包。 +#### 5.4 使用鲲鹏分析扫描工具扫描编译生成的tar包，或者使用find.so脚本查找，确保没有包含有x86的so和jar包。 +如果有，则参考4，将jar包替换本地仓库，并重新编译zookeeper。 -- Gitee From 473f02ddb20904f6f85d853fa17d324050d1eb5c Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 07:43:43 +0000 Subject: [PATCH 25/99] =?UTF-8?q?add=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/flink.md.=20=E5=88=9B=E5=BB=BAflink=20aarch64?= =?UTF-8?q?=E7=A7=BB=E6=A4=8D=E6=8C=87=E5=8D=97flink.md=E6=96=87=E4=BB=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 80779a032eb26e0876460046087352745fa7a03a Mon Sep 17 00:00:00 2001 From: xiexing Date: Fri, 10 Mar 2023 09:27:22 +0000 Subject: [PATCH 26/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/flink.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../flink.md" | 140 ++++++++++++++++++ 1 file changed, 140 insertions(+) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" index e69de29..eab23b4 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" @@ -0,0 +1,140 @@ +### flink-1.13.0移植指南 +编写日期：2023年03月10日 +### 1 简介 + Flink是一个分布式、高性能、高可用的开源大数据处理框架，具有基于内存计算，流式处理等特点，用于对无边界和有边界数据流进行有状态的计算。 +### 2 环境要求 +#### 2.1 硬件要求 +| 项目 | 说明 | +| ---------------- |---------------- | +|服务器|对服务器无要求| +|CPU|aarch64架构| +|磁盘分区|对磁盘分区无要求| +|网络|可访问外网| +#### 2.2 软件要求 +| 项目 | 说明 | +| ---------------- |---------------- | +|版本|openeuler 22.03| +|OS|5.10.0-60.65.0.90.oe2203.aarch64| +|GCC|10.3.1| +|OpenJDK|1.8.0_342| +|Maven|3.6.3| +|Flink|1.13.0| +### 3 安装基础库 +#### 3.1 安装gcc maven jdk-1.8.0 +yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 maven java-1.8.0-openjdk.aarch64 java-1.8.0-openjdk-devel.aarch64 +#### 3.2 安装依赖 +yum -y install gcc gcc-c++ gcc-gfortran.aarch64 libgcc.aarch64 make cmake libtool autoconf automake ant wget git vim epel-release clang openssl openssl-devel golang +#### 3.3 配置maven +修改Maven配置文件中的：本地仓路径、远程仓等。 +配置文件路径：“/etc/maven/settings.xml” +说明： +``` + +/path/to/local/repo + + + + huaweimaven + huawei maven + https://repo.huaweicloud.com/repository/maven/ + central + +``` +### 4 移植指南分析 +#### 4.1 软件移植分析结果 +|原始jar包|so文件| +| ---------------- |---------------- | +|scala-compiler-2.11.12.jar | libjansi.so +|flink-shaded-netty-4.1.49.Final-13.0.jar | liborg_apache_flink_shaded_netty4_netty_transport_native_epoll_x86_64.so| +|frocksdbjni-5.17.2-artisans-2.0.jar | librocksdbjni32.so, librocksdbjni64.so, librocksdbjnile.so| +|flink-shaded-netty-tcnative-dynamic-2.0.30.Final-13.0.jar | liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_64.so, liborg_apache_flink_shaded_netty4_netty_tcnative_linux_x86_64_fedora.so| +|conscrypt-openjdk-uber-1.3.0.jar | libconscrypt_openjdk_jni-linux-x86_64.so +|beam-vendor-grpc-1_26_0-0.3.jar | liborg_apache_beam_vendor_grpc_v1p26p0_netty_tcnative_linux_x86_64.so| +### 5 依赖库编译 +#### 5.1 scala-compiler-2.11.12.jar +scala-compiler-2.11.12.jar aarch64下载地址：https://mirrors.huaweicloud.com/kunpeng/maven/org/scala-lang/scala-compiler/2.11.12/scala-compiler-2.11.12.jar +#### 5.2 frocksdbjni-5.17.2-artisans-2.0.jar +frocksdbjni-5.17.2-artisans-2.0.jar aarch64下载地址：https://mirrors.huaweicloud.com/kunpeng/maven/com/data-artisans/frocksdbjni/5.17.2-artisans-2.0/frocksdbjni-5.17.2-artisans-2.0.jar +#### 5.3 beam-vendor-grpc-1_26_0-0.3.jar +##### 5.3.1 安装gradle-5.4.1 +``` +wget https://services.gradle.org/distributions/gradle-5.4.1-bin.zip --no-check-certificate +unzip gradle-5.4.1-bin.zip +export PATH=`pwd`/gradle-5.4.1/bin:$PATH +``` +##### 5.3.2 移植beam-vendor-grpc +``` +wget https://github.com/apache/beam/archive/v2.19.0.tar.gz -O beam-v2.19.0.tar.gz +tar -zxf beam-v2.19.0.tar.gz +``` +##### 5.3.3 进入beam目录 +``` +cd beam-2.19.0/vendor/grpc-1_26_0/ +vim build.gradle +version = "0.3" + +repositories { + maven { url "https://mirrors.huaweicloud.com/kunpeng/maven/" } + mavenLocal() + maven { url "https://mirrors.huaweicloud.com/repository/maven/"} +} +``` +##### 5.4.3执行编译 +``` +gradle build +编译好的beam-vendor-grpc-1_26_0-0.3.jar在build/libs目录下 +请将编译完成的jar包替换到/.../org/apache/beam/beam-vendor-grpc-1_26_0/0.3/自己搭建的Maven仓库中。 +``` +#### 5.4 conscrypt-openjdk-uber-1.3.0.jar +#### 5.5 flink-shaded-netty + +### 6 Flink-1.3.0编译 +#### 6.1 从github上下载flink-release-1.13.1源码并解压： +``` +wget https://github.com/apache/flink/archive/release-1.3.0.tar.gz +tar -zxf flink-release-1.13.0.tar.gz +cd flink-release-1.13.0 +``` +#### 6.2 修改vim flink-runtime-web/pom.xml +说明：国内访问外网地址，由于网络限制或者网速问题，会出现下载报错或者长时间卡住，更换为国内地址。 +``` +@@ -271,6 +271,7 @@ under the License. + + + ci --cache-max=0 --no-save ++ https://mirrors.huaweicloud.com/repository/npm + + true + +@@ -283,6 +284,7 @@ under the License. + + + run build ++ https://mirrors.huaweicloud.com/repository/npm + + + +@@ -262,6 +262,7 @@ under the License. + + + v10.9.0 ++ https://mirrors.huaweicloud.com/nodejs/ + + + +``` +#### 6.3 执行编译： +``` +mvn clean install -DskipTests +多线程编译可以如下命令： +mvn clean install -DskipTests -Dfast -T 32 -Dmaven.compile.fork=true +如flink-runtime-web_2.11编译报错，可使用如下命令继续编译，节约时间。 +mvn clean install -DskipTests -rf :flink-runtime-web_2.11 +``` +#### 6.4 编译结果 +编译完成后在flink-release-1.13.0/flink-dist/target/flink-1.13.0-bin/生成目标目录flink-1.13.0。 +#### 6.5 使用鲲鹏分析扫描工具扫描编译生成的tar包，确保没有包含有x86的so和jar包。 +如果有，则参考4，将jar包替换本地仓库，并重新编译flink。 + + + -- Gitee From ae56e5d3c3ba17288bc49097d4449d63f7209f56 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 02:14:31 +0000 Subject: [PATCH 27/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/spark.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../spark.md" | 297 ++++++++++++++++++ 1 file changed, 297 insertions(+) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" index e69de29..a622687 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/spark.md" @@ -0,0 +1,297 @@ +### **spark-3.3.0 移植指南** + +编写日期：2023年03月10日 + +#### **1. 软硬件环境** + +##### **1.1 硬件** +CPU 鲲鹏920 +网络 Ethernet-10GE +存储 SATA 1T +内存 Hynix 512G 2400MHz +##### **1.2 OS** +EulerOS 22.03 +Kernel 5.10.0-60.18.0.50.oe2203.aarch64 +OpenJDK 1.8.0_342 +Maven 3.8.6 +Hadoop 3.3.4 +#### **2.编译环境准备** +##### **2.1 编译工具安装** +`yum –y install wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 cmake patch protobuf` +##### **2.2 安装openjdk** +安装指南参考链接：https://gitee.com/openeuler/bishengjdk-8/wikis/%E6%AF%95%E6%98%87JDK%208%20%E5%AE%89%E8%A3%85%E6%8C%87%E5%8D%97?sort_id=2891179 +##### **2.3 安装maven** +###### 2.3.1 下载并安装目录 + +``` +下载： wget https://archive.apache.org/dist/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz --no-check-certificate +安装： tar -zxf apache-maven-3.8.6-bin.tar.gz -C /home/installed/ +修改maven环境变量，在/etc/profile文件末尾增加下面代码： +export JAVA_HOME=/home/installed/jdk1.8.0_342 // 注意jdk等安装路径 +export MAVEN_HOME=/home/installed/apache-maven-3.8.6 +export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH +source /etc/profile +``` +###### 2.3.2 配置本地仓、远程仓等 +``` + +/path/to/local/repo + + + + huaweimaven + huawei maven + http://cmc-cd-mirror.rnd.huawei.com/maven/ + central + +``` +#### 3 移植分析 +| 原始jar | so文件 | +|---------------------------------|-------------------------------------------| +| commons-crypto-1.1.0.jar | libcommons-crypto.so | +| jline-2.12.1.jar | libjansi.so | +| leveldbjni-all-1.8.jar | libleveldbjni.so | +| netty-all-4.1.74.Final.jar | libnetty_transport_native_epoll_x86_64.so | +| scala-compiler-2.12.15.jar | libjansi.so | + +#### 4.编译依赖包 +##### 4.1 编译commons-crypto-1.1.0.jar + +``` +下载源码：wget https://github.com/apache/commons-crypto/archive/refs/tags/rel/commons-crypto-1.1.0.tar.gz --no-check-certificate +解压：tar -zxf commons-crypto-1.1.0.tar.gz +进入目录：cd commons-crypto-rel-commons-crypto-1.1.0 +执行编译命令：mvn clean install -DskipTests +``` +##### 4.2 编译leveldbjni-all-1.8.jar +1）master分支支持aarch64，编译前需要修改pom.xml文件里面的版本号，也可在windows下载master分支的代码压缩包，再上传到服务器进行编译。 +``` +vim /etc/profile +在最后一行加入 +export GIT_SSL_NO_VERIFY=1 +保存并退出 + +git config --global url."https://".insteadOf git:// + +git clone git://github.com/fusesource/leveldbjni.git +``` +2）进入当前目录 +``` +cd leveldbjni +export LEVELDBJNI_HOME=`pwd` +``` +3）修改pom.xml文件 +注释无关操作系统类型（freebsd64、sunos64-amd64、sunos64-sparcv9、linux64-ppc64le），避免引起编译不通过。 +如下： +``` +245 +246 full +247 +248 leveldbjni-osx +249 leveldbjni-linux32 +250 leveldbjni-linux64 +251 +252 leveldbjni-win32 +253 leveldbjni-win64 +254 +255 +256 +257 leveldbjni-linux64-aarch64 +258 leveldbjni-all +259 +260 +``` +并注释 +``` +288 +289 +295 +... +315 +316 +``` +第60、61行，将编译的目标module加入leveldbjni-all和linux64模块。 +``` + 57 + 58 + 59 leveldbjni + 60 leveldbjni-linux64 + 61 leveldbjni-all + 62 + 63 +``` +4）修改leveldbjni-linux64-aarch64/pom.xml文件，第77行，修改后文件对应内容如下： +``` + 76 + 77 aarch64 + 78 leveldbjni + 79 false +``` +5）修改leveldbjni-all/pom.xml文件 +注释无关操作系统类型（freebsd64、sunos64-amd64、sunos64-sparcv9、linux64-ppc64le），避免引起编译不通过。将dependency相关的依赖注释掉，注释内容如下： +``` + 75 + ... + 93 +``` +将build的NativeCode部分注释掉，并修改部分内容，表明生成的leveldbjni-all-1.8.jar里面对应linux64目录下的so也为aarch64，后续在ambari的nodemanager启动过程中，默认使用的是该目录的so：修改后内容如下： +``` +147 +148 META-INF/native/windows32/leveldbjni.dll;osname=Win32;processor=x86, +149 META-INF/native/windows64/leveldbjni.dll;osname=Win32;processor=x86-64, +150 META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86, +151 META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86-64, +152 META-INF/native/linux32/libleveldbjni.so;osname=Linux;processor=x86, +153 META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=aarch64, +154 META-INF/native/aarch64/libleveldbjni.so;osname=Linux;processor=aarch64 +155 +156 +``` +6）修改所有pom.xml文件中的版本号，将所有pom.xml文件中的版本“99-master-SNAPSHOT”改为 “1.8”： + +find . -name pom.xml | xargs sed -i 's/99-master-SNAPSHOT/1.8/g' + +7）编译 +mvn clean package -P download -P linux64-aarch64 -DskipTests + +#### 4.3编译netty-all-4.1.74.Final.jar + +``` +下载源码：wget https://github.com/netty/netty/archive/netty-4.1.74.Final.tar.gz --no-check-certificate +解压：tar zxf netty-4.1.74.Final.tar.gz +进入目录修改pom.xml文件，第545行：将tcnative.verision 改为2.0.36.Final + +从kunpeng镜像仓下载已经移植的netty-tcnative jar包放至本地仓库路径下对应目录io/netty/netty-tcnative/2.0.36.Final/ +wget https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-tcnative/2.0.36.Final/netty-tcnative-2.0.36.Final.jar https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-tcnative/2.0.36.Final/netty-tcnative-2.0.36.Final.pom https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-tcnative/2.0.36.Final/netty-tcnative-2.0.36.Final.jar.sha1 https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-tcnative/2.0.36.Final/netty-tcnative-2.0.36.Final.pom.sha1 --no-check-certificate + +修改pom.xml，注释以下代码 +1190 + +编译：mvn package install -DskipTests +``` +#### 4.4编译scala-compiler-2.12.15.jar +##### 4.4.1sbt工具下载配置： +``` +下载sbt：wget https://github.com/sbt/sbt/releases/download/v1.3.10/sbt-1.3.10.zip --no-check-certificate +解压：unzip sbt-1.3.10.zip +mv sbt-1.3.10 sbt +进入目录sbt/conf并配置sbtconfig.txt： +vim sbtconfig.txt +``` +sbtconfig.txt内容如下： +``` +-XX:ReservedCodeCacheSize=128m +-XX:+CMSClassUnloadingEnabled1 + +# Set the extra sbt options + +-Dsbt.log.format=true +-Dfile.encoding=UTF8 + +-Dsbt.repository.config="/root/.sbt/repositories" +-Dsbt.override.build.repos=true + +#以下网络代理根据是否需要再进行配置 +-Dhttp.proxyHost= #例：-Dhttp.proxyHost=http://*.*.*.* 网络代理ip地址 +-Dhttp.proxyPort= #网络代理ip端口 +-Dhttps.proxyHost= #例：-Dhttp.proxyHost=http://*.*.*.* 网络代理ip地址 +-Dhttps.proxyPort= #网络代理ip端口 +``` +配置仓库代理： +``` +[repositories] + local + huaweicloud-maven: https://repo.huaweicloud.com/repository/maven/ + maven-central: https://repo1.maven.org/maven2/ + sbt-plugin-repo: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext] + typesafe: https://dl.bintray.com/typesafe/ivy-releases/ , [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext], bootOnly + +``` +##### 4.4.2编译scala-compiler-2.12.15.jar +``` +下载源码：wget https://codeload.github.com/scala/scala/tar.gz/v2.12.15 +解压：tar -zxf scala-2.12.7.tar.gz +cd scala-2.12.15/ + +编译命令：sbt package + +编译完成之后的 scala-compiler.jar 在如下路径下 +build/pack/lib/scala-compiler.jar + +进行mvn构建本地仓jar包 +mvn install:install-file -Dpackaging=jar -Dfile=build-sbt/pack/lib/scala-compiler.jar -DgroupId=org.scala-lang -DartifactId=scala-compiler -Dversion=2.12.15 +``` +#### 5编译spark-3.3.0 +命令：mvn -Pyarn -Phadoop-3.3 -Dhadoop.version=3.3.4 -Phive -Phive-thriftserver -DskipTests clean package + +打包（记得修改./dev/make-distribution.sh里的mvn路径为自己maven的路径） + +第39行：MVN="$SPARK_HOME/build/mvn" + +打包命令：./dev/make-distribution.sh --tgz -DskipTests=true -Phive -Phive-thriftserver -Pyarn + + + + + + + -- Gitee From d78c5d37665e4e01bf07dc2ea369685cc092c73f Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 02:23:43 +0000 Subject: [PATCH 28/99] =?UTF-8?q?add=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hbase.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" new file mode 100644 index 0000000..e69de29 -- Gitee From f4117c664d955e015a0f236e992b4cefb538f7e3 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 02:54:42 +0000 Subject: [PATCH 29/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/flink.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../flink.md" | 320 +++++++++++++++++- 1 file changed, 319 insertions(+), 1 deletion(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" index eab23b4..c7b7b29 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" @@ -23,7 +23,7 @@ #### 3.1 安装gcc maven jdk-1.8.0 yum -y install gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 maven java-1.8.0-openjdk.aarch64 java-1.8.0-openjdk-devel.aarch64 #### 3.2 安装依赖 -yum -y install gcc gcc-c++ gcc-gfortran.aarch64 libgcc.aarch64 make cmake libtool autoconf automake ant wget git vim epel-release clang openssl openssl-devel golang +yum -y install gcc gcc-c++ gcc-gfortran.aarch64 libgcc.aarch64 make cmake libtool autoconf automake ant wget git vim clang openssl openssl-devel golang #### 3.3 配置maven 修改Maven配置文件中的：本地仓路径、远程仓等。 配置文件路径：“/etc/maven/settings.xml” @@ -86,8 +86,326 @@ gradle build 请将编译完成的jar包替换到/.../org/apache/beam/beam-vendor-grpc-1_26_0/0.3/自己搭建的Maven仓库中。 ``` #### 5.4 conscrypt-openjdk-uber-1.3.0.jar +``` +##### 5.4.1准备conscrypt编译环境 +步骤1 安装cmake，需要（3.0及以上），可使用yum安装，cmake –version查看版本。 +步骤2 安装ninja。 +1.下载ninja，并解压。 +wget https://github.com/ninja-build/ninja/archive/v1.10.0.zip +unzip v1.10.0.zip +2.定位到目录ninja-1.10.0。 +cd ninja-1.10.0 +3.编译ninja。 +./configure.py --bootstrap +4.将里面的ninja 文件复制到/usr/bin目录下。 +cp ninja /usr/bin +步骤3 配置go环境 +1.下载，并解压。 +cd /usr/local/ +wget https://golang.org/dl/go1.16.6.linux-arm64.tar.gz +tar –xf go1.16.6.linux-arm64.tar.gz +2.配置环境变量，并生效。 +export GOROOT=/usr/local/go +export PATH=$PATH:$GOROOT/bin +source /etc/profile +步骤4 构建boringssl环境 +1.下载boringssl源码，并解压。 +wget https://github.com/google/boringssl/archive/master.zip +unzip master.zip +mv boringssl-master boringssl +2.配置环境变量。 +vim /etc/profile +export BORINGSSL_HOME=/home/boringssl +source /etc/profile +3.进入boringssl 目录。 +cd boringssl +构建64位版本和32位版本。因为系统架构为aarch64,可以不构建32位版本。 +64位版本： +mkdir build64 +cd build64 +cmake -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_ASM_FLAGS=-Wa,--noexecstack \ + -GNinja .. +ninja +注: + 1. 找不到libwind，yum安装libunwind-dev。 + 2. yum golang下载失败，可手动安装go，并配置环境变量。 + 3. ninja时，go：golang get 报x509：certificate错误，可以配置。 +export GOPROXY=http://mirrors.tools.huawei.com/goproxy/ +export GO111MODULE=on +32 位版本： +mkdir build32 +cd build32 +cmake -DCMAKE_TOOLCHAIN_FILE=../util/32-bit-toolchain.cmake \ + -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_ASM_FLAGS="-Wa,--noexecstack -m32 -msse2" \ + -GNinja .. +ninja +##### 5.4.2编译gradle源码 +1．定位到/home 目录。 +cd /home +2．查看conscrypt编译对应的gradle版本。 +进入conscrypt目录。 +vim /home/conscript/gradle/wrapper/gradle-wrapper.properties +3．下载gradle-v4.8.1源码 。 +wget https://github.com/gradle/gradle/archive/v4.8.1.zip +解压v4.8.1.zip。 +unzip v4.8.1.zip +重命名为gradle，并定位到gradle目录。 +mv gradle-4.8.1 gradle && cd gradle +4．编辑NativePlatforms.java +vim subprojects/platform-native/src/main/java/org/gradle/nativeplatform/platform/internal/NativePlatforms.java +在44行后面增加一行代码。 +ArchitectureInternal aarch64 = Architectures.forInput("aarch64"); +在73行后面增加一行代码。 + platforms.add(createPlatform(linux, aarch64)); +5．gradle构建。 +./gradlew +运行完成后/home/gradle/subprojects/platform-native/build/libs 下会生成gradle-platform-native-4.8.1.jar。 +注：1. 这里UnknownHostException问题，请看问题1 gradle代理配置解决。 + 2. 出现Reason: invalid distance too far back错误，删除/root/.gradle/wrapper/下目录，重新运行./gradlew即可 +6．定位到conscrypt目录。 +cd /home/conscript +运行构建命令。 +./gradlew +会自动下载gradle-4.8.1-all，gradle的默认的缓存目录：/root/.gradle/wrapper/dists。 +7．定位到下载的gradle 中删除gradle-platform-native-4.8.1.jar +cd /root/.gradle/wrapper/dists/gradle-4.8.1-all/6fmj4nezasjg1b7kkmy10xgo2/gradle- +4.8.1/lib/plugins ; +rm -rf gradle-platform-native-4.8.1.jar; +将步骤3生成的jar包替换 +cp /home/gradle-4.8.1/subprojects/platform-native/build/libs/gradle-platform-native-4.8.1.jar ./ +8．重启电脑 +Reboot +##### 5.4.4编译conscrypt +1.下载conscrypt源码 +定位到/home目录。 +cd /home +下载源码，并解压 +wget https://github.com/google/conscrypt/archive/1.3.0.zip --no-check-certificate +unzip 1.3.0.zip +mv conscrypt-1.3.0 conscrypt && cd /conscript +2.编译conscrypt。 +步骤 1 定位到openjdk目录 +cd openjdk +步骤 2 修改build.gradle文件 +vim build.gradle +1.10行增加arm64 = 'aarch64' +2.修改19行nativeClassifier64Bit = classifierFor(osName, arm64) +3.在第280行增加platform配置。 + linux_aarch64 { + architecture "aarch64" + operatingSystem "linux" + } +4.将构建jni-library 时64位架构改为“linux_aarch64”。 +注释第293行，修改第294行arch64Name为"linux_aarch64" +修改315行修改为aarch64。 +5.Clang 或gcc编译时args增加指定头文件地址。(10.3.0为gcc版本，根据自己gcc版本配置) +"-I$jniSourceDir/main/include", +"-I$jniSourceDir/unbundled/include", +"-I/usr/include/c++/10.3.0", +"-I/usr/include/c++/10.3.0/aarch64-redhat-linux ", +修改生成库信息代码。 +def archName = binary.targetPlatform.architecture.name +def archNameR = archName +if(archName == 'aarch64'){ + archNameR = 'aarch_64' +} +archName = archName.replaceAll('-','_') +def classifier = classifierFor(osName, archName) +def classifierR = classifierFor(osName,archNameR) +def sourceSetName = sourceSetName("$classifier") +def source = binary.sharedLibraryFile +修改第419行和第421行内容为classifierR +完成后单击“Ecs”，输入:wq保存退出。 +6.返回上一级目录。修改“conscrypt”根目录下的build.gradle文件 +vi build.gradle +修改改第39行，并增加40行-43行，构建aarch通道。 + gcc(Gcc){ + target("linux_aarch64") { + cppCompiler.executable = "/usr/bin/gcc" + } + } +在第172-174行增加代码。 +dependencies { +errorprone("com.google.errorprone:error_prone_core:2.3.3") +} +完成后单击“Ecs”，输入:wq保存退出。 +7.定位到conscrypt目录。 +cd /home/conscrypt +8.构建。 +./gradlew clean build --stacktrace -x :conscrypt-openjdk:linux_aarch64Test -Dorg.conscrypt.openjdk.buildUberJar=true +注：1.如找不到linux_aarch64Test，则使用./gradlew clean build，跳过测试。 + 2.构建找不到conscrypt_openjdk_jni-linux-aarch_64，可在根目录，find一下，将找到的so文件按照要求放入对应目录下，比如/usr/lib。 + 3.Tset构建失败，注释build.gradle下Test类。 +9.定位到openjdk-uber/buil/libs目录。 +cd /openjdk-uber/build/libs +cp /home/conscrypt-1.3.0/openjdk/build/linux_aarch_64/native-resources/META-INF/native/libconscrypt_openjdk_jni-linux-aarch_64.so +10.解压conscrypt-openjdk-uber-1.3.0.jar。 +jar xvf conscrypt-openjdk-uber-1.3.0.jar +11.将/home/conscrypt/openjdk/build/linux_aarch_64/native-resources/META-INF/native\ 目录下的libconscrypt_openjdk_jni-linux-aarch_64.so 放入conscrypt-open-uber-1.3.0.jar 中的META-INF/native 下。 +cp /home/conscrypt/openjdk/build/linux_aarch_64/native-resources/META-INF/native\ /libconscrypt_openjdk_jni-linux-aarch_64.so META-INF/native +12.jar打包，移植结束。 +jar cvf conscrypt-openjdk-uber-1.3.0.jar META-INF/ org +将编译好的jar包放入/root/.gradle/caches/modules-2/files-2.1/org.conscrypt/conscrypt-openjdk-uber +``` #### 5.5 flink-shaded-netty +##### 5.5.1 编译安装apr-1.6.5 +``` +1.下载apr-1.6.5源码。 +wget https://archive.apache.org/dist/apr/apr-1.6.5.tar.gz +2.解压后编译安装。 +tar -zxvf apr-1.6.5.tar.gz +cd apr-1.6.5 +./configure +make +make install +注：如./configure 报错：apr-1.7.0 rm: cannot remove 'libtoolT': No such file or directory +则vim configure +30392 ，修改RM='$RM'修改为RM='$RM -f' +30390 PACKAGE='$PACKAGE' +30391 VERSION='$VERSION' +30392 RM='$RM -rf' +30393 ofile='$ofile' +``` +##### 5.5.2 编译安装netty-tcnative-parent-2.0.30.Final +``` +1.下载netty-tcnative-parent-2.0.30.Final源码。 +wget https://codeload.github.com/netty/netty-tcnative/tar.gz/netty-tcnative-parent-2.0.30.Final +mv netty-tcnative-parent-2.0.30.Final netty-tcnative-parent-2.0.30.Final.tar.gz +2.解压源码包。 +tar -zxvf netty-tcnative-parent-2.0.30.Final.tar.gz +3.进入解压后目录。 +cd netty-tcnative-netty-tcnative-parent-2.0.30.Final +4.修改pom.xml，屏蔽boringssl-static的编译。 +vim pom.xml +603 +601 openssl-dynamic +602 openssl-static +603 +604 +5.编译打包到maven本地仓库。 +mvn install +``` +##### 5.5.3 编译安装netty-all-4.1.49.Final +``` +1.下载netty-4.1.49.Final源码。 +wget https://github.com/netty/netty/archive/netty-4.1.49.Final.tar.gz +2.解压源码包。 +tar -zxvf netty-4.1.49.Final.tar.gz +3.进入netty解压目录。 +cd netty-netty-4.1.49.Final +4.编译打成jar包，netty-all-4.1.49Final.jar放置于“all/target”目录。 +mvn install -DskipTests +``` +##### 5.5.4 flink-shaded-netty-4.1.49.Final-13.0编译 +``` +1.下载flink-shaded-release-13.0安装包。 +wget https://codeload.github.com/apache/flink-shaded/tar.gz/release-13.0 +2.解压安装包。 +mv release-13.0 flink-shaded-release-13.0.tar.gz +tar -zxvf flink-shaded-release-13.0.tar.gz +3.进入解压后的目录。 +cd flink-shaded-release-13.0 +4.修改pom.xml +因为只需单独编译打包flink-shaded-netty-4，注释掉其余不需要的module。 +vim pom.xml +58 + 58 + 59 + 60 + 61 + 62 flink-shaded-netty-4 + 63 + 64 + 65 + 66 +5.编译打成jar包。 +mvn install package –DskipTests +flink-shaded-netty-4.1.49.Final-13.0.jar放置于“flink-shaded-netty-4/target/”目录。 +将编译好的jar包替换本地仓库中 +``` +##### 5.5.5 flink-shaded-netty-tcnative-dynamic-2.0.30.Final-13.0.jar +``` +1.进入flink shaded-release-13.0 +cd flink-shaded-release-13.0 +2.修改“flink-shaded-release-13.0/pom.xml”。因为只需单独编译打包flink-shaded-netty-tcnative-dynamic，注释掉其余不需要的module。 +vim pom.xml + + + + + + flink-shaded-netty-tcnative-dynamic + + + +3.修改“flink-shaded-release-13.0/flink-shaded-netty-tcnative-dynamic/pom.xml”。 +vim flink-shaded-netty-tcnative-dynamic/pom.xml 在第94行下方插入如下代码。 + + io.netty + netty-tcnative + ${netty.tcnative.version} + linux-aarch_64 + jar + false +${project.build.directory}/native_libs + + +添加完成之后，在第156行下方插入如下代码。 + + + + + + + + + + + + + + + + + + + +注释掉第175-第198行。 + +4.编译打包到maven本地仓库。 +mvn install +编译好的jar包在flink-shaded-netty-tcnative-dynamic/target目录下，并将编译完成的jar包替换到本地仓库中。 +``` ### 6 Flink-1.3.0编译 #### 6.1 从github上下载flink-release-1.13.1源码并解压： ``` -- Gitee From 4fca8ef3b1b93aa72216273a4680cc72a0ca2b46 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 02:57:27 +0000 Subject: [PATCH 30/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/flink.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../flink.md" | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" index c7b7b29..59746ca 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/flink.md" @@ -79,15 +79,15 @@ repositories { maven { url "https://mirrors.huaweicloud.com/repository/maven/"} } ``` -##### 5.4.3执行编译 +##### 5.3.3执行编译 ``` gradle build 编译好的beam-vendor-grpc-1_26_0-0.3.jar在build/libs目录下 请将编译完成的jar包替换到/.../org/apache/beam/beam-vendor-grpc-1_26_0/0.3/自己搭建的Maven仓库中。 ``` #### 5.4 conscrypt-openjdk-uber-1.3.0.jar +##### 5.4.1 准备conscrypt编译环境 ``` -##### 5.4.1准备conscrypt编译环境 步骤1 安装cmake，需要（3.0及以上），可使用yum安装，cmake –version查看版本。 步骤2 安装ninja。 1.下载ninja，并解压。 @@ -143,7 +143,9 @@ cmake -DCMAKE_TOOLCHAIN_FILE=../util/32-bit-toolchain.cmake \ -DCMAKE_ASM_FLAGS="-Wa,--noexecstack -m32 -msse2" \ -GNinja .. ninja +``` ##### 5.4.2编译gradle源码 +``` 1．定位到/home 目录。 cd /home 2．查看conscrypt编译对应的gradle版本。 @@ -179,7 +181,9 @@ rm -rf gradle-platform-native-4.8.1.jar; cp /home/gradle-4.8.1/subprojects/platform-native/build/libs/gradle-platform-native-4.8.1.jar ./ 8．重启电脑 Reboot +``` ##### 5.4.4编译conscrypt +``` 1.下载conscrypt源码 定位到/home目录。 cd /home -- Gitee From f741fde931b296cfb2434dddbc60f2df740b6539 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 03:03:02 +0000 Subject: [PATCH 31/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20=E9=83=A8=E7=BD=B2?= =?UTF-8?q?=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" new file mode 100644 index 0000000..e69de29 -- Gitee From 30f5990c222b9746ae19f4d3cb1cdfd288c797a1 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 03:03:20 +0000 Subject: [PATCH 32/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20=E9=83=A8=E7=BD=B2?= =?UTF-8?q?=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" new file mode 100644 index 0000000..e69de29 -- Gitee From 4191ec55cd55e34f2cea82ff6fedcdd9484a5c5e Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 03:23:32 +0000 Subject: [PATCH 33/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hbase.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hbase.md" | 257 ++++++++++++++++++ 1 file changed, 257 insertions(+) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" index e69de29..3204c22 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" @@ -0,0 +1,257 @@ +### **Hbase-2.5.0 移植指南** + +编写日期：2023年03月13日 + +#### **1. 软硬件环境** + +##### **1.1 硬件** +CPU 鲲鹏920 +网络 Ethernet-10GE +存储 SATA 1T +内存 Hynix 512G 2400MHz +##### **1.2 OS** +EulerOS 22.03 +Kernel 5.10.0-60.18.0.50.oe2203.aarch64 +OpenJDK 1.8.0_342 +Maven 3.8.6 +Hadoop 3.3.4 +#### **2.编译环境准备** +##### **2.1 编译工具安装** +`yum –y install wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 cmake patch protobuf` +##### **2.2 安装openjdk** +安装指南参考链接：https://gitee.com/openeuler/bishengjdk-8/wikis/%E6%AF%95%E6%98%87JDK%208%20%E5%AE%89%E8%A3%85%E6%8C%87%E5%8D%97?sort_id=2891179 +##### **2.3 安装maven** +###### 2.3.1 下载并安装目录 + +``` +下载： wget https://archive.apache.org/dist/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz --no-check-certificate +安装： tar -zxf apache-maven-3.8.6-bin.tar.gz -C /home/installed/ +修改maven环境变量，在/etc/profile文件末尾增加下面代码： +export JAVA_HOME=/home/installed/jdk1.8.0_342 // 注意jdk等安装路径 +export MAVEN_HOME=/home/installed/apache-maven-3.8.6 +export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH +source /etc/profile +``` +###### 2.3.2 配置本地仓、远程仓等 +``` + +/path/to/local/repo + + + + huaweimaven + huawei maven + http://cmc-cd-mirror.rnd.huawei.com/maven/ + central + +``` +#### 3 移植分析 +| 原始jar | so文件 | +|---------------------------------|-------------------------------------------| +| commons-crypto-1.1.0.jar | libcommons-crypto.so | +| jine-2.14.6.jar | libjansi.so | +| leveldbjni-all-1.8.jar | libleveldbjni.so | +#### 4.编译依赖包 +##### 4.1编译commons-crypto-1.1.0.jar + +``` +下载源码：wget https://github.com/apache/commons-crypto/archive/refs/tags/rel/commons-crypto-1.1.0.tar.gz --no-check-certificate +解压：tar -zxf commons-crypto-1.1.0.tar.gz +进入目录：cd commons-crypto-rel-commons-crypto-1.1.0 +执行编译命令：mvn clean install -DskipTests +``` +##### 4.2 编译leveldbjni-all-1.8.jar +1）master分支支持aarch64，编译前需要修改pom.xml文件里面的版本号，也可在windows下载master分支的代码压缩包，再上传到服务器进行编译。 +``` +vim /etc/profile +在最后一行加入 +export GIT_SSL_NO_VERIFY=1 +保存并退出 + +git config --global url."https://".insteadOf git:// + +git clone git://github.com/fusesource/leveldbjni.git +``` +2）进入当前目录 +``` +cd leveldbjni +export LEVELDBJNI_HOME=`pwd` +``` +3）修改pom.xml文件 +注释无关操作系统类型（freebsd64、sunos64-amd64、sunos64-sparcv9、linux64-ppc64le），避免引起编译不通过。 +如下： +``` +245 +246 full +247 +248 leveldbjni-osx +249 leveldbjni-linux32 +250 leveldbjni-linux64 +251 +252 leveldbjni-win32 +253 leveldbjni-win64 +254 +255 +256 +257 leveldbjni-linux64-aarch64 +258 leveldbjni-all +259 +260 +``` +并注释 +``` +288 +289 +295 +... +315 +316 +``` +第60、61行，将编译的目标module加入leveldbjni-all和linux64模块。 +``` + 57 + 58 + 59 leveldbjni + 60 leveldbjni-linux64 + 61 leveldbjni-all + 62 + 63 +``` +4）修改leveldbjni-linux64-aarch64/pom.xml文件，第77行，修改后文件对应内容如下： +``` + 76 + 77 aarch64 + 78 leveldbjni + 79 false +``` +5）修改leveldbjni-all/pom.xml文件 +注释无关操作系统类型（freebsd64、sunos64-amd64、sunos64-sparcv9、linux64-ppc64le），避免引起编译不通过。将dependency相关的依赖注释掉，注释内容如下： +``` + 75 + ... + 93 +``` +将build的NativeCode部分注释掉，并修改部分内容，表明生成的leveldbjni-all-1.8.jar里面对应linux64目录下的so也为aarch64，后续在ambari的nodemanager启动过程中，默认使用的是该目录的so：修改后内容如下： +``` +147 +148 META-INF/native/windows32/leveldbjni.dll;osname=Win32;processor=x86, +149 META-INF/native/windows64/leveldbjni.dll;osname=Win32;processor=x86-64, +150 META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86, +151 META-INF/native/osx/libleveldbjni.jnilib;osname=macosx;processor=x86-64, +152 META-INF/native/linux32/libleveldbjni.so;osname=Linux;processor=x86, +153 META-INF/native/linux64/libleveldbjni.so;osname=Linux;processor=aarch64, +154 META-INF/native/aarch64/libleveldbjni.so;osname=Linux;processor=aarch64 +155 +156 +``` +6）修改所有pom.xml文件中的版本号，将所有pom.xml文件中的版本“99-master-SNAPSHOT”改为 “1.8”： + +find . -name pom.xml | xargs sed -i 's/99-master-SNAPSHOT/1.8/g' + +7）编译 +mvn clean package -P download -P linux64-aarch64 -DskipTests + +#### 4.3编译jine-2.14.6.jar +依赖关系：jline-2.14.6--> jansi-1.12 --> jansi-native-1.7 +##### 4.3.1 编译jansi-native-1.7 + +``` +下载源码:wget https://github.com/fusesource/jansi-native/archive/jansi-native-1.7.tar.gz --no-check-certificate +解压并进入目录：tar zxf jansi-native-1.7.tar.gz +cd jansi-native-jansi-native-1.7 +``` +修改pom.xml的hawtjni-version：改为1.16与openEuler yum源下载版本对应：vim pom.xml +39 +``` +39 1.16 +``` +修改280行 maven-hawtjni-plugin 为 hawtjni-maven-plugin +``` +280 hawtjni-maven-plugin +``` +执行编译： +``` +mvn install -Dplatform=linux64 +``` +结果在“./target”目录下 +##### 4.3.2 编译jansi-1.12 +下载Jansi 1.12版本源码，并进入源码目录 +``` +wget https://github.com/fusesource/jansi/archive/jansi-project-1.12.tar.gz +tar -zxvf jansi-project-1.12.tar.gz +cd jansi-jansi-project-1.12/ +``` +修改pom.xml里jansi-native的版本为1.7: vim pom.xml +40 +``` +40 1.7 +``` +执行编译： +``` +mvn install -Dmaven.javadoc.skip=true +``` +##### 4.3.3 编译jline-2.14.6 +下载jline-2.14.6源码，并解压进入目录 +``` +wget https://github.com/jline/jline2/archive/refs/tags/jline-2.14.6.tar.gz --no-check-certificate +tar -zxvf jline-2.14.6.tar.gz +cd jline2-jline-2.14.6 +``` +执行编译： +``` +mvn install -Dmaven.javadoc.skip=true -DskipTests +``` +#### 5编译hbase-2.5.0 +编译命令： +``` +mvn clean package -DskipTests assembly:single +``` -- Gitee From 497c25e286835f70a55f82b1635804dbcfc4e99e Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 03:24:21 +0000 Subject: [PATCH 34/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hbase.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hbase.md" | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" index 3204c22..68f134b 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hbase.md" @@ -202,9 +202,9 @@ find . -name pom.xml | xargs sed -i 's/99-master-SNAPSHOT/1.8/g' 7）编译 mvn clean package -P download -P linux64-aarch64 -DskipTests -#### 4.3编译jine-2.14.6.jar +##### 4.3编译jine-2.14.6.jar 依赖关系：jline-2.14.6--> jansi-1.12 --> jansi-native-1.7 -##### 4.3.1 编译jansi-native-1.7 +###### 4.3.1 编译jansi-native-1.7 ``` 下载源码:wget https://github.com/fusesource/jansi-native/archive/jansi-native-1.7.tar.gz --no-check-certificate @@ -224,7 +224,7 @@ cd jansi-native-jansi-native-1.7 mvn install -Dplatform=linux64 ``` 结果在“./target”目录下 -##### 4.3.2 编译jansi-1.12 +###### 4.3.2 编译jansi-1.12 下载Jansi 1.12版本源码，并进入源码目录 ``` wget https://github.com/fusesource/jansi/archive/jansi-project-1.12.tar.gz @@ -239,7 +239,7 @@ cd jansi-jansi-project-1.12/ ``` mvn install -Dmaven.javadoc.skip=true ``` -##### 4.3.3 编译jline-2.14.6 +###### 4.3.3 编译jline-2.14.6 下载jline-2.14.6源码，并解压进入目录 ``` wget https://github.com/jline/jline2/archive/refs/tags/jline-2.14.6.tar.gz --no-check-certificate -- Gitee From 866fbe45351773ccd1c650246db3012273eaeca4 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 03:25:01 +0000 Subject: [PATCH 35/99] =?UTF-8?q?add=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" new file mode 100644 index 0000000..e69de29 -- Gitee From c8ddca5c0f321709e349656f97a3b4d9cb07a43b Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 03:48:13 +0000 Subject: [PATCH 36/99] =?UTF-8?q?=E5=88=A0=E9=99=A4=E6=96=87=E4=BB=B6=20Do?= =?UTF-8?q?cs/=E9=83=A8=E7=BD=B2=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" deleted file mode 100644 index e69de29..0000000 -- Gitee From 6760ed1b5d564d24f8c1ff98a881c743435583f4 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 03:50:16 +0000 Subject: [PATCH 37/99] add Docs/README.md. Signed-off-by: xiexing --- Docs/README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 Docs/README.md diff --git a/Docs/README.md b/Docs/README.md new file mode 100644 index 0000000..3782320 --- /dev/null +++ b/Docs/README.md @@ -0,0 +1,14 @@ +CloudNative SIG WiKi +此处存放CloudNative云原生相关资料供开发者、用户参考 + +目录说明 +1-开发指南：存放开发者关注的文档，如k8s集群本地测试调试方法等 +k8s集群本地测试调试方法 +2-用户指南：存放用户指南，如k8s集群二进制部署方法、SIG组件版本关系配套说明等，更多的是基于场景的解决方案类 +k8s镜像发布流程暂行办法 +k8s二进制部署指南 +openEuler容器镜像使用方法 +3-特性指南：SIG组内特性维度的文档，包括但不限于设计、架构等 +4-其他：不属于上述三类的文档 +说明 +开发者在提交文档之后，请更新此文件，刷新相关目录章节 \ No newline at end of file -- Gitee From 2c2aba16715d16a2bc63d476917a981178003d8f Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 03:50:29 +0000 Subject: [PATCH 38/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hive.md" | 96 +++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" index e69de29..ec95371 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" @@ -0,0 +1,96 @@ +### **hive-3.1.3 移植指南** + +编写日期：2023年03月13日 + +#### **1. 软硬件环境** + +##### **1.1 硬件** +CPU 鲲鹏920 +网络 Ethernet-10GE +存储 SATA 1T +内存 Hynix 512G 2400MHz +##### **1.2 OS** +EulerOS 22.03 +Kernel 5.10.0-60.18.0.50.oe2203.aarch64 +OpenJDK 1.8.0_342 +Maven 3.8.6 +Hadoop 3.3.4 +#### **2.编译环境准备** +##### **2.1 编译工具安装** +`yum –y install wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 cmake patch protobuf` +##### **2.2 安装openjdk** +安装指南参考链接：https://gitee.com/openeuler/bishengjdk-8/wikis/%E6%AF%95%E6%98%87JDK%208%20%E5%AE%89%E8%A3%85%E6%8C%87%E5%8D%97?sort_id=2891179 +##### **2.3 安装maven** +###### 2.3.1 下载并安装目录 + +``` +下载： wget https://archive.apache.org/dist/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz --no-check-certificate +安装： tar -zxf apache-maven-3.8.6-bin.tar.gz -C /home/installed/ +修改maven环境变量，在/etc/profile文件末尾增加下面代码： +export JAVA_HOME=/home/installed/jdk1.8.0_342 // 注意jdk等安装路径 +export MAVEN_HOME=/home/installed/apache-maven-3.8.6 +export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH +source /etc/profile +``` +###### 2.3.2 配置本地仓、远程仓等 +``` + +/path/to/local/repo + + + + huaweimaven + huawei maven + http://cmc-cd-mirror.rnd.huawei.com/maven/ + central + +``` +#### 3 移植分析 +| 原始jar | so文件 | +|---------------------------------|-------------------------------------------| +| commons-crypto-1.1.0.jar | libcommons-crypto.so | +| jline-2.12.1.jar | libjansi.so | +| leveldbjni-all-1.8.jar | libleveldbjni.so | +| netty-all-4.1.17.Final.jar | libnetty_transport_native_epoll_x86_64.so | +| scala-compiler-2.12.15.jar | libjansi.so | + +#### 4 编译依赖包 +##### 4.1 编译 + + +#### 5 编译Hive-3.1.3 +##### 5.1 提升hive的guave版本 +原因：集群中所安装的Hadoop-3.3.4中和Hive-3.1.3中包含guava的依赖，Hadoop-3.3.4中的版本为guava-27.0-jre，而Hive-3.1.3中的版本为guava-19.0。由于Hive运行时会加载Hadoop依赖，故会出现依赖冲突的问题。 + +修改pom.xml文件 + +将pom.xml文件中147行的 19.0修改为27.0-jre +##### 5.2 无法推断类型变量V问题解决 +Error:Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project hive-llap-common: Compilation failure: Compilation failure + +原因：无法推断类型变量V (实际参数列表和形式参数列表长度不同) + +解决：修改下列源码 +vim druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java +``` +28 import com.google.common.collect.Iterators; +29 +30 import java.io.IOException; +31 + import java.util.Collections; +32 import java.util.Iterator; +33 import java.util.List; +``` + +``` +44 +45 private ScanResultValue current; +46 +47 - private Iterator> compactedValues = Iterators.emptyIterator(); +48 + private Iterator> compactedValues = Collections.emptyIterator(); +49 +50 @Override +51 protected JavaType getResultTypeDef() { +``` + + + -- Gitee From 7dbdd73e999790afdcc86058ef0b28f725919a1f Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:10:41 +0000 Subject: [PATCH 39/99] =?UTF-8?q?update=20Docs/README.md.=20docs=20?= =?UTF-8?q?=E6=96=87=E4=BB=B6=E5=A4=B9=E6=8F=8F=E8=BF=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- Docs/README.md | 40 ++++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/Docs/README.md b/Docs/README.md index 3782320..a51f9a0 100644 --- a/Docs/README.md +++ b/Docs/README.md @@ -1,14 +1,26 @@ -CloudNative SIG WiKi -此处存放CloudNative云原生相关资料供开发者、用户参考 - -目录说明 -1-开发指南：存放开发者关注的文档，如k8s集群本地测试调试方法等 -k8s集群本地测试调试方法 -2-用户指南：存放用户指南，如k8s集群二进制部署方法、SIG组件版本关系配套说明等，更多的是基于场景的解决方案类 -k8s镜像发布流程暂行办法 -k8s二进制部署指南 -openEuler容器镜像使用方法 -3-特性指南：SIG组内特性维度的文档，包括但不限于设计、架构等 -4-其他：不属于上述三类的文档 -说明 -开发者在提交文档之后，请更新此文件，刷新相关目录章节 \ No newline at end of file +# Bigdata SIG WiKi + +此处存放Bigdata相关资料供开发者、用户参考 + +## 目录说明 +- **1-移植指南**：存放开发者关注的文档，如hadoop aarch64移植步骤及问题等 + - [hadoop移植指南](移植指南/hadoop.md) + - [hive移植指南](移植指南/hive.md) + - [hbase移植指南](移植指南/hbase.md) + - [spark移植指南](移植指南/spark.md) + - [flink移植指南](移植指南/flink.md) + - [zookeeper移植指南](移植指南/zookeeper.md) +- **2-部署指南**：存放bigdata组件部署指南，如hadoop部署指南等类 + - [hadoop部署指南](部署指南/hadoop.md) + - [hive部署指南](部署指南/hive.md) + - [hbase部署指南](部署指南/hbase.md) + - [spark部署指南](部署指南/spark.md) + - [flink部署指南](部署指南/flink.md) + - [zookeeper移植指南](移植指南/zookeeper.md) +- **3-checkSo工具**：存放checkSo工具，及使用方法 + - + +- **4-其他**：不属于上述三类的文档 + +## 说明 +开发者在提交文档之后，请更新此文件，刷新相关目录章节 -- Gitee From 478f96fe43ad0196189ad2a699eb148854a4b2dc Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:11:51 +0000 Subject: [PATCH 40/99] update Docs/README.md. Signed-off-by: xiexing --- Docs/README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/Docs/README.md b/Docs/README.md index a51f9a0..89daab5 100644 --- a/Docs/README.md +++ b/Docs/README.md @@ -18,8 +18,7 @@ - [flink部署指南](部署指南/flink.md) - [zookeeper移植指南](移植指南/zookeeper.md) - **3-checkSo工具**：存放checkSo工具，及使用方法 - - - + - **4-其他**：不属于上述三类的文档 ## 说明 -- Gitee From 7e639f7121d7736e4040be6fb6bb96ff4aa76ec3 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:12:19 +0000 Subject: [PATCH 41/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20checkSo=E7=AD=89?= =?UTF-8?q?=E5=B7=A5=E5=85=B7=E5=8C=85?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/.keep" diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/.keep" "b/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/.keep" new file mode 100644 index 0000000..e69de29 -- Gitee From e83a64b97f3ec732d8733e59adb0dce0b73f6239 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:17:25 +0000 Subject: [PATCH 42/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20checkSo.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../checkSo.md/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo.md/.keep" diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo.md/.keep" "b/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo.md/.keep" new file mode 100644 index 0000000..e69de29 -- Gitee From 26692635c57a3a97446d1e3fd27871c86923cc5e Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 06:17:33 +0000 Subject: [PATCH 43/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hive.md" | 110 +++++++++++++++++- 1 file changed, 109 insertions(+), 1 deletion(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" index ec95371..ce7c6b1 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" @@ -71,7 +71,7 @@ Error:Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6. 原因：无法推断类型变量V (实际参数列表和形式参数列表长度不同) 解决：修改下列源码 -vim druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java +1) vim druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java ``` 28 import com.google.common.collect.Iterators; 29 @@ -91,6 +91,114 @@ vim druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryReco 50 @Override 51 protected JavaType getResultTypeDef() { ``` +2) vim llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/AMReporter.java + + +``` +174 Thread.getDefaultUncaughtExceptionHandler().uncaughtException(Thread.currentThread(), t); +175 } +176 } +177 - }); +177 + },MoreExecutors.directExecutor()); +178 // TODO: why is this needed? we could just save the host and port? +179 nodeId = LlapNodeId.getInstance(localAddress.get().getHostName(), localAddress.get().getPort()); +180 LOG.info("AMReporter running with DaemonId: {}, NodeId: {}", daemonId, nodeId); + +... + +274 LOG.warn("Failed to send taskKilled for {}. The attempt will likely time out.", +275 taskAttemptId); +276 } +277 - }); +277 + },MoreExecutors.directExecutor()); +278 } +279 +280 public void queryComplete(QueryIdentifier queryIdentifier) { + +... + +342 amNodeInfo.amNodeId, currentQueryIdentifier, t); +343 queryFailedHandler.queryFailed(currentQueryIdentifier); +344 } +345 - }); +345 + },MoreExecutors.directExecutor()); +346 } +347 } +348 } catch (InterruptedException e) { + +``` +3) vim llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapTaskReporter.java +``` +128 sendCounterInterval, maxEventsToGet, requestCounter, containerIdStr, initialEvent, +129 fragmentRequestId, wmCounters); +130 ListenableFuture future = heartbeatExecutor.submit(currentCallable); +131 - Futures.addCallback(future, new HeartbeatCallback(errorReporter)); +131 + Futures.addCallback(future, new HeartbeatCallback(errorReporter),MoreExecutors.directExecutor()); +132 } +``` +4) vim llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorService.java +``` +175 executionCompletionExecutorService = MoreExecutors.listeningDecorator( +176 executionCompletionExecutorServiceRaw); +177 ListenableFuture future = waitQueueExecutorService.submit(new WaitQueueWorker()); +178 - Futures.addCallback(future, new WaitQueueWorkerCallback()); +178 + Futures.addCallback(future, new WaitQueueWorkerCallback(),MoreExecutors.directExecutor()); +179 } +180 +181 private LlapQueueComparatorBase createComparator( +``` +5) vim ql/src/test/org/apache/hadoop/hive/ql/exec/tez/SampleTezSessionState.java +``` +19 package org.apache.hadoop.hive.ql.exec.tez; +20 +21 - import com.google.common.util.concurrent.Futures; + - import com.google.common.util.concurrent.FutureCallback; + - import com.google.common.util.concurrent.ListenableFuture; + - import com.google.common.util.concurrent.SettableFuture; +22 + import com.google.common.util.concurrent.*; +23 +24 import java.io.IOException; +25 import java.util.concurrent.ScheduledExecutorService; +26 import javax.security.auth.login.LoginException; + +... + +126 public void onFailure(Throwable t) { +127 future.setException(t); +128 } + - }); +129 + }, MoreExecutors.directExecutor()); +130 return future; +131 } +132 +``` +6) vim ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java +``` +18 package org.apache.hadoop.hive.ql.exec.tez; +19 +20 + import com.google.common.util.concurrent.*; +21 import org.apache.hadoop.hive.metastore.api.WMPoolSchedulingPolicy; +22 import org.apache.hadoop.hive.metastore.utils.MetaStoreUtils; +23 +24 import com.google.common.annotations.VisibleForTesting; +25 import com.google.common.collect.Lists; +26 import com.google.common.collect.Sets; +27 import com.google.common.math.DoubleMath; + - import com.google.common.util.concurrent.FutureCallback; + - import com.google.common.util.concurrent.Futures; + - import com.google.common.util.concurrent.ListenableFuture; + - import com.google.common.util.concurrent.SettableFuture; + - import com.google.common.util.concurrent.ThreadFactoryBuilder; +28 +29 import java.util.ArrayList; +30 import java.util.Collection; + +... + +``` + + + -- Gitee From 7a86f6971c4fead31232ce0808041441ba7300a3 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:18:31 +0000 Subject: [PATCH 44/99] =?UTF-8?q?=E9=87=8D=E5=91=BD=E5=90=8D=20Docs/checkS?= =?UTF-8?q?o=E7=AD=89=E5=B7=A5=E5=85=B7=E5=8C=85/checkSo.md=20=E4=B8=BA=20?= =?UTF-8?q?Docs/checkSo=E7=AD=89=E5=B7=A5=E5=85=B7=E5=8C=85/checkSo?= =?UTF-8?q?=E5=B7=A5=E5=85=B7=E4=BD=BF=E7=94=A8=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo.md/.keep" => "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" (100%) diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo.md/.keep" "b/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" similarity index 100% rename from "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo.md/.keep" rename to "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" -- Gitee From 4e74cdade382db4f2fd34f7454d5fbfb78a70b06 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:18:52 +0000 Subject: [PATCH 45/99] =?UTF-8?q?add=20Docs/checkSo=E7=AD=89=E5=B7=A5?= =?UTF-8?q?=E5=85=B7=E5=8C=85/checkSo=E5=B7=A5=E5=85=B7=E4=BD=BF=E7=94=A8?= =?UTF-8?q?=E6=8C=87=E5=8D=97/checkSo.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../checkSo.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" "b/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 940e0bf2014c537581e1e3ff7384df3ec439c271 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:21:00 +0000 Subject: [PATCH 46/99] =?UTF-8?q?update=20Docs/checkSo=E7=AD=89=E5=B7=A5?= =?UTF-8?q?=E5=85=B7=E5=8C=85/checkSo=E5=B7=A5=E5=85=B7=E4=BD=BF=E7=94=A8?= =?UTF-8?q?=E6=8C=87=E5=8D=97/checkSo.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../checkSo.md" | 6 ++++++ 1 file changed, 6 insertions(+) diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" "b/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" index e69de29..4a85a21 100644 --- "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" +++ "b/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" @@ -0,0 +1,6 @@ +### checkSo工具使用指南 +#### checkSo工具下载地址 +https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/bigdata/Tools/checkSo.zip +#### checkSo工具指南 +参考以下链接 +https://bbs.huaweicloud.com/forum/thread-22679-1-1.html -- Gitee From a1077b7c07b58ded78c1271da0842cfd87517951 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:25:35 +0000 Subject: [PATCH 47/99] update Docs/README.md. Signed-off-by: xiexing --- Docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Docs/README.md b/Docs/README.md index 89daab5..57cc220 100644 --- a/Docs/README.md +++ b/Docs/README.md @@ -18,7 +18,7 @@ - [flink部署指南](部署指南/flink.md) - [zookeeper移植指南](移植指南/zookeeper.md) - **3-checkSo工具**：存放checkSo工具，及使用方法 - + - [checkSo工具使用指南](checkSo等工具包/checkSo工具使用指南/checkSo.md) - **4-其他**：不属于上述三类的文档 ## 说明 -- Gitee From a8d5ec5446585c49f40e2bbcf149f617ba7c6f9e Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:25:58 +0000 Subject: [PATCH 48/99] =?UTF-8?q?=E5=88=A0=E9=99=A4=E6=96=87=E4=BB=B6=20Do?= =?UTF-8?q?cs/.keep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Docs/.keep | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 Docs/.keep diff --git a/Docs/.keep b/Docs/.keep deleted file mode 100644 index e69de29..0000000 -- Gitee From 1f284e91313360f70651f3fee6de82a3006e8e0d Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:26:09 +0000 Subject: [PATCH 49/99] =?UTF-8?q?=E5=88=A0=E9=99=A4=E6=96=87=E4=BB=B6=20Do?= =?UTF-8?q?cs/=E7=A7=BB=E6=A4=8D=E6=8C=87=E5=8D=97/.keep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/.keep" deleted file mode 100644 index e69de29..0000000 -- Gitee From 633edd9ade61dd7056e947fe3e04b1d21c684dfc Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:26:27 +0000 Subject: [PATCH 50/99] =?UTF-8?q?=E5=88=A0=E9=99=A4=E6=96=87=E4=BB=B6=20Do?= =?UTF-8?q?cs/=E9=83=A8=E7=BD=B2=E6=8C=87=E5=8D=97=20/.keep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227 /.keep" deleted file mode 100644 index e69de29..0000000 -- Gitee From e7aaf5f9b298c800c3f4007fcfb1d403e18b2c9f Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:26:53 +0000 Subject: [PATCH 51/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20=E9=83=A8=E7=BD=B2?= =?UTF-8?q?=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" new file mode 100644 index 0000000..e69de29 -- Gitee From 94a4c31a80e268736f5ac6bd37d32142f85bc404 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:27:14 +0000 Subject: [PATCH 52/99] =?UTF-8?q?add=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop" new file mode 100644 index 0000000..e69de29 -- Gitee From df12bf618d7140f084e1497fbd0533b37e94621e Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:27:43 +0000 Subject: [PATCH 53/99] =?UTF-8?q?=E9=87=8D=E5=91=BD=E5=90=8D=20Docs/?= =?UTF-8?q?=E9=83=A8=E7=BD=B2=E6=8C=87=E5=8D=97/hadoop=20=E4=B8=BA=20Docs/?= =?UTF-8?q?=E9=83=A8=E7=BD=B2=E6=8C=87=E5=8D=97/hadoop.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop" => "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" (100%) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" similarity index 100% rename from "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop" rename to "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" -- Gitee From eba6c94f543fe573ac9b4b81db2be0500f4aa90e Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:34:58 +0000 Subject: [PATCH 54/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" index c3bcc8a..a745998 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" @@ -1,4 +1,4 @@ -### **Hadoop-3.3.4 移植指南** +### **zookeeper-3.8.1移植指南** 编写日期：2023年03月10日 ### 1. 介绍 -- Gitee From 7f6cbb3561d61de1f0acb8b2f3cf35cbb3f506bd Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 06:35:47 +0000 Subject: [PATCH 55/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper.md" | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" index a745998..54600bd 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/zookeeper.md" @@ -8,14 +8,14 @@ Zookeeper是一个分布式的，开放源码的分布式应用程序协调服 ### 2. 环境要求 #### 2.1 硬件要求 -| 项目 | 说明 | +| 硬件 | 说明 | | ---------------- |---------------- | |服务器|对服务器无要求| |CPU|aarch64架构| |磁盘分区|对磁盘分区无要求| |网络|可访问外网| #### 2.2 软件要求 -| 项目 | 说明 | +| 软件 | 说明 | | ---------------- |---------------- | |版本|openeuler 22.03| |OS|5.10.0-60.65.0.90.oe2203.aarch64| -- Gitee From ed82e52bbfcab886f6c832376cc3eacefc021699 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 06:51:24 +0000 Subject: [PATCH 56/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hive.md" | 168 ++++++++++++++++++ 1 file changed, 168 insertions(+) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" index ce7c6b1..1c2f44b 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" @@ -195,7 +195,175 @@ Error:Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6. ... +1088 } +1089 +1090 private void failOnFutureFailure(ListenableFuture future) { + - Futures.addCallback(future, FATAL_ERROR_CALLBACK); +1091 + Futures.addCallback(future, FATAL_ERROR_CALLBACK, MoreExecutors.directExecutor()); +1092 } +1093 +1094 private void queueGetRequestOnMasterThread( + +... + +1921 +1922 public void start() throws Exception { +1923 ListenableFuture getFuture = tezAmPool.getSessionAsync(); + - Futures.addCallback(getFuture, this); +1924 + Futures.addCallback(getFuture, this,MoreExecutors.directExecutor()); +1925 } +1926 +1927 @Override + +... + +1975 case GETTING: { +1976 ListenableFuture waitFuture = session.waitForAmRegistryAsync( +1977 amRegistryTimeoutMs, timeoutPool); + - Futures.addCallback(waitFuture, this); +1978 + Futures.addCallback(waitFuture, this,MoreExecutors.directExecutor()); +1979 break; +1980 } +1981 case WAITING_FOR_REGISTRY: { +``` +7) vim llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java +``` +744 }, 10000L, TimeUnit.MILLISECONDS); +745 +746 nodeEnablerFuture = nodeEnabledExecutor.submit(nodeEnablerCallable); + - Futures.addCallback(nodeEnablerFuture, new LoggingFutureCallback("NodeEnablerThread", LOG)); +747 + Futures.addCallback(nodeEnablerFuture, new LoggingFutureCallback("NodeEnablerThread", LOG),MoreExecutors.directExecutor()); +748 +749 delayedTaskSchedulerFuture = +750 delayedTaskSchedulerExecutor.submit(delayedTaskSchedulerCallable); +751 Futures.addCallback(delayedTaskSchedulerFuture, + - new LoggingFutureCallback("DelayedTaskSchedulerThread", LOG)); +752 + new LoggingFutureCallback("DelayedTaskSchedulerThread", LOG),MoreExecutors.directExecutor()); +753 +754 schedulerFuture = schedulerExecutor.submit(schedulerCallable); + - Futures.addCallback(schedulerFuture, new LoggingFutureCallback("SchedulerThread", LOG)); +755 + Futures.addCallback(schedulerFuture, new LoggingFutureCallback("SchedulerThread", LOG),MoreExecutors.directExecutor()); +756 +757 registry.start(); +758 registry.registerStateChangeListener(new NodeStateChangeListener()); + +``` +8) vim llap-common/src/java/org/apache/hadoop/hive/llap/AsyncPbRpcProxy.java +``` +171 CallableRequest request, LlapNodeId nodeId) { +172 ListenableFuture future = executor.submit(request); +173 Futures.addCallback(future, new ResponseCallback( + - request.getCallback(), nodeId, this)); +174 + request.getCallback(), nodeId, this),MoreExecutors.directExecutor()); +175 } +176 +177 @VisibleForTesting + +... + +283 LOG.warn("RequestManager shutdown with error", t); +284 } +285 } + - }); +286 + },MoreExecutors.directExecutor()); +287 } +288 +289 @Override +``` +##### 5.3 整合spark版本，以下以整合spark3.3.0为例 +1）修改hive源码目录下pom.xml文件第201开始： +``` +3.3.0 +2.12 +2.12.15 ``` +2）vim ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUtils.java +``` +31 import org.apache.hadoop.hive.ql.plan.ColStatistics.Range; +32 import org.apache.hadoop.hive.serde.serdeConstants; +33 import org.junit.Test; + - import org.spark_project.guava.collect.Sets; +34 + import org.sparkproject.guava.collect.Sets; +35 +36 public class TestStatsUtils { +``` +3) vim spark-client/src/main/java/org/apache/hive/spark/client/metrics/ShuffleWriteMetrics.java +``` +47 } +48 +49 public ShuffleWriteMetrics(TaskMetrics metrics) { + - this(metrics.shuffleWriteMetrics().shuffleBytesWritten(), + - metrics.shuffleWriteMetrics().shuffleWriteTime()); +50 + this(metrics.shuffleWriteMetrics().bytesWritten(), +51 + metrics.shuffleWriteMetrics().bytesWritten()); +52 } +53 +54 } + +``` +4) vim spark-client/src/main/java/org/apache/hive/spark/counter/SparkCounter.java +``` +19 +20 import java.io.Serializable; +21 + - import org.apache.spark.Accumulator; + - import org.apache.spark.AccumulatorParam; +22 import org.apache.spark.api.java.JavaSparkContext; +23 + import org.apache.spark.util.LongAccumulator; +24 +25 public class SparkCounter implements Serializable { +26 +27 private String name; +28 private String displayName; + - private Accumulator accumulator; +29 + private LongAccumulator accumulator; +30 +31 // Values of accumulators can only be read on the SparkContext side. This field is used when +32 // creating a snapshot to be sent to the RSC client. + +... + +54 +55 this.name = name; +56 this.displayName = displayName; + - LongAccumulatorParam longParam = new LongAccumulatorParam(); +57 + String accumulatorName = groupName + "_" + name; + - this.accumulator = sparkContext.accumulator(initValue, accumulatorName, longParam); +58 + this.accumulator = sparkContext.sc().longAccumulator(accumulatorName); +59 } +60 +61 public long getValue() { + +... +86 +87 return new SparkCounter(name, displayName, accumulator.value()); +88 } + - + - class LongAccumulatorParam implements AccumulatorParam { + - + - @Override + - public Long addAccumulator(Long t1, Long t2) { + - return t1 + t2; + - } + - + - @Override + - public Long addInPlace(Long r1, Long r2) { + - return r1 + r2; + - } + - + - @Override + - public Long zero(Long initialValue) { + - return 0L; + - } + - } +89 +90 } +``` +##### 5.4 修复插入数据的bug +源码按照链接修复16个类：https://github.com/gitlbo/hive/commit/c073e71ef43699b7aa68cad7c69a2e8f487089fd + + + -- Gitee From b552d95e3edfff7466dae62c9453d45b6c2159c8 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 07:22:17 +0000 Subject: [PATCH 57/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hive.md" | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" index 1c2f44b..ab2d3bc 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" @@ -362,6 +362,47 @@ Error:Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6. ##### 5.4 修复插入数据的bug 源码按照链接修复16个类：https://github.com/gitlbo/hive/commit/c073e71ef43699b7aa68cad7c69a2e8f487089fd +#### 6 编译Hive-3.1.3 +``` +编译命令：mvn clean package -Pdist -DskipTests -Dmaven.javadoc.skip=true +``` +##### 6.1 问题：protoc-2.5.0linux-aarch_64.exe本地仓缺失 +1) 下载并解压源码 +``` +wget https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz + +tar -zxf protobuf-2.5.0.tar.gz + +mv protobuf-2.5.0 /opt/tools/installed/ + +cd /opt/tools/installed/protobuf-2.5.0 +``` +2) 安装依赖库 +``` +yum -y install patch automake libtool +``` +3) 上传protoc.patch到服务器，打补丁，其中protoc.patch的路径视实际情况而定。 +``` +wget https://obs-mirror-ftp4.obs.cn-north-4.myhuaweicloud.com/tools/protoc.patch + +cp protoc.patch ./src/google/protobuf/stubs/ + +cd ./src/google/protobuf/stubs/ + +patch -p1 < protoc.patch + +cd - +``` +4) 编译并安装到系统默认目录 +``` +./autogen.sh && ./configure CFLAGS='-fsigned-char' && make –j128 && make install +``` +5) Protoc部署在本地Maven仓库中 +``` +mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=/usr/local/bin/protoc +``` + + -- Gitee From 6da7969ca29921d7a559de164bfbe54d60b89924 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 07:36:11 +0000 Subject: [PATCH 58/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../hadoop.md" | 372 ++++++++++++++++++ 1 file changed, 372 insertions(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" index e69de29..43f2a84 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" @@ -0,0 +1,372 @@ +### hadoop-3.3.4部署指南 +#### 1- 环境要求 +##### 1.1 建议版本 +| 软件 | 说明 | 获取方法 | +| ---------------- |---------------- |---------------- | +|OpenJDK|1.8.0_342|yum安装或者官网获取| +|hadoop|3.3.4|官网获取，aarch64版本需移植，参考移植指南| +|zookeeper|3.8.1|官网获取，aarch64版本需移植，参考移植指南| +##### 1.2硬件要求 +``` +最低配置：任意CPU、一根内存（大小不限）、一块硬盘（大小不限）。 +具体配置视实际应用场景而定。 +操作系统要求： 适用于CentOS 7.4~7.6、openeuler-20.03、openEuler-22.03操作系统。 +说明 +本文以openeuler 22.03为例，介绍Hadoop（1+3）集群部署。 +``` +##### 1.3 集群环境规划 +本章节规划以四台机器分别作为集群的节点1、节点2、节点3、节点4。各个节点数据 +| 机器名称 | IP地址 | 硬盘数 | OS & JDK| +| ---------------- |---------------- |---------------- |---------------- | +|server1|IPaddress1|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent1|IPaddress2|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent2|IPaddress3|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent3|IPaddress4|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +##### 1.4 软件规划 +| 机器名称 | 服务名称 | +| ---------------- |---------------- | +|server1|Namenode、ResourceManager| +|agent1|QuorumPeerMain、DataNode、NodeManager、JournalNode| +|agent2|QuorumPeerMain、DataNode、NodeManager、JournalNode| +|agent3|QuorumPeerMain、DataNode、NodeManager、JournalNode| +#### 2- 配置部署环境 +``` +步骤1 依次登录节点1-4，将节点的主机名分别修改为server1、agent1、agent2、agent3。 +hostnamectl set-hostname 主机名 --static +步骤2 登录所有节点，修改“/etc/hosts”文件。 +在hosts文件中添加集群所有节点的“地址-主机名”映射关系。 +IPaddress1 server1 +IPaddress2 agent1 +IPaddress3 agent2 +IPaddress4 agent3 +步骤3 登录所有节点，关闭防火墙。 +systemctl stop firewalld.service +systemctl disable firewalld.service +systemctl status firewalld.service +步骤4 登录所有节点，配置SSH免密登录。 +1. 生成密钥，遇到提示时，按回车。 +ssh-keygen -t rsa +2. 在每台机器上配置SSH免密登录（包括配置自身节点的免密）。 +ssh-copy-id -i ~/.ssh/id_rsa.pub root@节点IP +步骤5 登录所有节点，安装OpenJDK，可使用指定版本jdk。 +yum install -y java-1.8.0 +java -version +``` +#### 3-部署 ZooKeeper +##### 3.1 编译并解压 ZooKeeper +``` +步骤1 参考ZooKeeper移植指南编译出“zookeeper-3.8.1.tar.gz”部署包。 +步骤2 将zookeeper-3.8.1.tar.gz放置于agent1节点的“/usr/local”目录下，并解压。 +mv zookeeper-3.8.1.tar.gz /usr/local +cd /usr/local +tar -zxvf zookeeper-3.8.1.tar.gz +步骤3 建立软链接，便于后期版本更换。 +ln -s zookeeper-3.8.1 zookeeper +``` +##### 3.2 添加 ZooKeeper 到环境变量 +``` +步骤1 打开配置文件。 +vim /etc/profile +步骤2 添加ZooKeeper到环境变量。 +export ZOOKEEPER_HOME=/usr/local/zookeeper +export PATH=$ZOOKEEPER_HOME/bin:$PATH +步骤3 使环境变量生效。 +source /etc/profile +``` +##### 3.3 修改 ZooKeeper 配置文件 +``` +步骤1 进入ZooKeeper所在目录。 +cd /usr/local/zookeeper/conf +步骤2 拷贝配置文件。 +cp zoo_sample.cfg zoo.cfg +步骤3 修改配置文件。 +vim zoo.cfg +1. 修改数据目录。 +dataDir=/usr/local/zookeeper/tmp +2. 在最后添加如下代码，其中server.1-3是部署ZooKeeper的节点。 +server.1=agent1:2888:3888 +server.2=agent2:2888:3888 +server.3=agent3:2888:3888 +4 创建tmp目录作数据目录。 +mkdir /usr/local/zookeeper/tmp +步骤5 在tmp目录中创建一个空文件，并向该文件写入ID。 +touch /usr/local/zookeeper/tmp/myid +echo 1 > /usr/local/zookeeper/tmp/myid +``` +##### 3.4 同步配置到其它节点 +``` +步骤1 将配置好的ZooKeeper拷贝到其它节点。 +scp -r /usr/local/zookeeper-3.4.6 root@agent2:/usr/local +scp -r /usr/local/zookeeper-3.4.6 root@agent3:/usr/local +步骤2 登录agent2、agent3，创建软链接并修改myid内容。 +● agent2： +cd /usr/local +ln -s zookeeper-3.8.1 zookeeper +echo 2 > /usr/local/zookeeper/tmp/myid +● agent3： +cd /usr/local +ln -s zookeeper-3.8.1 zookeeper +echo 3 > /usr/local/zookeeper/tmp/myid +``` +##### 3.5 运行验证 +``` +步骤1 分别在agent1，agent2，agent3上启动ZooKeeper。 +cd /usr/local/zookeeper/bin +./zkServer.sh start +说明 +您可以分别在agent1，agent2，agent3上停止ZooKeeper。 +cd /usr/local/zookeeper/bin +./zkServer.sh stop +步骤2 查看ZooKeeper状态。 +./zkServer.sh status +``` +#### 4-部署Hadoop +##### 4.1 编译并解压 Hadoop +``` +步骤1 参考Hadoop 3.3.4 移植指南 编译出Hadoop软件部署包 “hadoop-3.3.4.tar.gz”。 +步骤2 将“hadoop-3.3.4.tar.gz”放置于server1节点的“/usr/local”目录，并解压。 +mv hadoop-3.3.4.tar.gz /usr/local +cd /usr/local +tar -zxvf hadoop-3.3.4.tar.gz +步骤3 建立软链接，便于后期版本更换。 +ln -s hadoop-3.3.4 hadoop +``` +##### 4.2 添加 Hadoop 到环境变量 +``` +步骤1 编辑/etc/profile文件。 +vim /etc/profile +步骤2 在文件底部添加环境变量，如下所示。 +export HADOOP_HOME=/usr/local/hadoop +export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH +步骤3 使环境变量生效。 +``` +##### 4.3 修改 Hadoop 配置文件 +``` +说明 +Hadoop所有的配置文件都在“$HADOOP_HOME/etc/hadoop”目录下，修改以下配置文件 +前，需要切换到“HADOOP_HOME/etc/hadoop”目录。 +cd $HADOOP_HOME/etc/hadoop +``` +修改 hadoop-env.sh +``` +修改环境变量JAVA_HOME为绝对路径，并将用户指定为root。（为jdk安装的路径） +echo "export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-3.oe2203sp1.aarch64/jre/" >> hadoop-env.sh +echo "export HDFS_NAMENODE_USER=root" >> hadoop-env.sh +echo "export HDFS_SECONDARYNAMENODE_USER=root" >> hadoop-env.sh +echo "export HDFS_DATANODE_USER=root" >> hadoop-env.sh +``` +修改 yarn-env.sh +``` +修改用户为root。 +echo "export YARN_REGISTRYDNS_SECURE_USER=root" >> yarn-env.sh +echo "export YARN_RESOURCEMANAGER_USER=root" >> yarn-env.sh +echo "export YARN_NODEMANAGER_USER=root" >> yarn-env.sh +``` +修改 core-site.xml +``` +步骤1 编辑core-site.xml文件。 +vim core-site.xml +步骤2 添加或修改ÑnĒªñà†ì²Ñn标签范围内的参数。 + + fs.defaultFS + hdfs://server1:9000 + + + hadoop.tmp.dir + /home/hadoop_tmp_dir + + + ipc.client.connect.max.retries + 100 + + + ipc.client.connect.retry.interval + 10000 + + + hadoop.proxyuser.root.hosts + * + + + hadoop.proxyuser.root.groups + * + +``` +在节点server1上创建目录。 +mkdir /home/hadoop_tmp_dir +修改 hdfs-site.xml +``` +步骤1 修改hdfs-site.xml文件。 +vim hdfs-site.xml +步骤2 添加或修改ÑnĒªñà†ì²Ñn标签范围内的参数。 + + dfs.replication + 1 + + + dfs.namenode.name.dir + /data/data1/hadoop/nn + + + dfs.datanode.data.dir + /data/data1/hadoop/dn,/data/data2/hadoop/dn,/data/data3/hadoop/dn,/data/data4/hadoop/dn,/ +data/data5/hadoop/dn,/data/data6/hadoop/dn,/data/data7/hadoop/dn,/data/data8/hadoop/dn,/data/data9/ +hadoop/dn,/data/data10/hadoop/dn,/data/data11/hadoop/dn,/data/data12/hadoop/dn + + + dfs.http.address + server1:50070 + + + dfs.namenode.http-bind-host + 0.0.0.0 + + + dfs.datanode.handler.count + 600 + + + dfs.namenode.handler.count + 600 + + + dfs.namenode.service.handler.count + 600 + + + ipc.server.handler.queue.size + 300 + + + dfs.webhdfs.enabled + true + +``` +``` +节点agent1、agent2、agent3分别创建dfs.datanode.data.dir对应目录。 +举例：mkdir -p /data/data{1,2,3,4,5,6,7,8,9,10,11,12}/hadoop +``` +修改 mapred-site.xml +``` +步骤1 编辑mapred-site.xml文件。 +vim mapred-site.xml +步骤2 添加或修改ÑnĒªñà†ì²Ñn标签范围内的参数。 + + mapreduce.framework.name + yarn + true + The runtime framework for executing MapReduce jobs + + + mapreduce.job.reduce.slowstart.completedmaps + 0.88 + + + mapreduce.application.classpath + + /usr/local/hadoop/etc/hadoop, + /usr/local/hadoop/share/hadoop/common/*, + /usr/local/hadoop/share/hadoop/common/lib/*, + /usr/local/hadoop/share/hadoop/hdfs/*, + /usr/local/hadoop/share/hadoop/hdfs/lib/*, + /usr/local/hadoop/share/hadoop/mapreduce/*, + /usr/local/hadoop/share/hadoop/mapreduce/lib/*, + /usr/local/hadoop/share/hadoop/yarn/*, + /usr/local/hadoop/share/hadoop/yarn/lib/* + + + + mapreduce.map.memory.mb + 6144 + + + mapreduce.reduce.memory.mb + 6144 + + + mapreduce.map.java.opts + -Xmx5530m + + + mapreduce.reduce.java.opts + -Xmx2765m + + + mapred.child.java.opts + -Xmx2048m -Xms2048m + + + mapred.reduce.parallel.copies + 20 + + + yarn.app.mapreduce.am.env + HADOOP_MAPRED_HOME=/usr/local/hadoop + + + mapreduce.map.env + HADOOP_MAPRED_HOME=/usr/local/hadoop + + + mapreduce.reduce.env + HADOOP_MAPRED_HOME=/usr/local/hadoop + +``` +修改 yarn-site.xml +``` +步骤1 编辑yarn-site.xml文件。 +vim yarn-site.xml +步骤2 添加或修改ÑnĒªñà†ì²Ñn标签范围内的参数。 + + yarn.nodemanager.aux-services + ɍü†ÃñžɌm†ÝàžšñžȎä¯ñďžɍȍü†ÃñžɌ + ɍĒn†ÃɌìàñžɍȍĒn†ÃɌ + + + yarn.nodemanager.aux-services + ɍü†ÃñžɌm†ÝàžšñžȎä¯ñďžɍȍü†ÃñžɌ + + + yarn.resourcemanager.hostname + server1 + + + yarn.resourcemanager.bind-host + 0.0.0.0 + + + yarn.scheduler.maximum-allocation-mb + 65536 + + + yarn.nodemanager.resource.memory-mb + 102400 + + + yarn.nodemanager.resource.cpu-vcores + 48 + + + yarn.log-aggregation-enable + true + + + yarn.client.nodemanager-connect.max-wait-ms + 300000 + + + yarn.nodemanager.vmem-pmem-ratio + 7.1 + + + yarn.nodemanager.vmem-check-enabled + false + + + yarn.nodemanager.pmem-check-enabled + false + + +``` + + -- Gitee From 2c2ee074bebbd2351e9cf5a9f41ab2b1c995c9ac Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 08:01:07 +0000 Subject: [PATCH 59/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hive.md" | 74 +++++++++++++++++-- 1 file changed, 69 insertions(+), 5 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" index ab2d3bc..3478ec1 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hive.md" @@ -49,13 +49,77 @@ source /etc/profile | 原始jar | so文件 | |---------------------------------|-------------------------------------------| | commons-crypto-1.1.0.jar | libcommons-crypto.so | -| jline-2.12.1.jar | libjansi.so | -| leveldbjni-all-1.8.jar | libleveldbjni.so | +| jline-2.12.jar | libjansi.so | | netty-all-4.1.17.Final.jar | libnetty_transport_native_epoll_x86_64.so | -| scala-compiler-2.12.15.jar | libjansi.so | #### 4 编译依赖包 -##### 4.1 编译 +##### 4.1 编译commons-crypto-1.1.0.jar +``` +下载源码：wget https://github.com/apache/commons-crypto/archive/refs/tags/rel/commons-crypto-1.1.0.tar.gz --no-check-certificate +解压：tar -zxf commons-crypto-1.1.0.tar.gz +进入目录：cd commons-crypto-rel-commons-crypto-1.1.0 +执行编译命令：mvn clean install -DskipTests +``` +##### 4.2 编译jline-2.12.jar +依赖关系：jline-2.12 --> jansi-1.12 --> jansi-native-1.7 +###### 4.2.1 编译jansi-native-1.7 + +``` +下载源码:wget https://github.com/fusesource/jansi-native/archive/jansi-native-1.7.tar.gz --no-check-certificate +解压并进入目录：tar zxf jansi-native-1.7.tar.gz +cd jansi-native-jansi-native-1.7 +``` +修改pom.xml的hawtjni-version：改为1.16与openEuler yum源下载版本对应：vim pom.xml +39 +``` +39 1.16 +``` +修改280行 maven-hawtjni-plugin 为 hawtjni-maven-plugin +``` +280 hawtjni-maven-plugin +``` +执行编译： +``` +mvn install -Dplatform=linux64 +``` +结果在“./target”目录下 + +###### 4.2.2 编译jansi-1.12 +下载Jansi 1.12版本源码，并进入源码目录 +``` +wget https://github.com/fusesource/jansi/archive/jansi-project-1.12.tar.gz +tar -zxvf jansi-project-1.12.tar.gz +cd jansi-jansi-project-1.12/ +``` +修改pom.xml里jansi-native的版本为1.7: vim pom.xml +40 +``` +40 1.7 +``` +执行编译： +``` +mvn install -Dmaven.javadoc.skip=true +``` +###### 4.3.3 编译jline-2.12 +下载jline-2.12源码，并解压进入目录 +``` +wget https://github.com/jline/jline2/archive/refs/tags/jline-2.12.tar.gz --no-check-certificate +tar -zxvf jline-2.12.tar.gz +cd jline2-jline-2.12 +``` +修改pom.xml文件第120行jansi.version的版本1.11为1.12 +``` +120 1.12 +``` + +执行编译： +``` +mvn install -Dmaven.javadoc.skip=true -DskipTests +``` + +##### 4.3 netty-all-4.1.17.Final.jar 移植到本地仓对应目录io/netty/netty-all/4.1.17.Final/下 +可直接从鲲鹏镜像仓拉取： +``` +wget https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-all/4.0.17.Final/netty-all-4.0.17.Final.jar https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-all/4.0.17.Final/netty-all-4.0.17.Final.jar.sha1 https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-all/4.0.17.Final/netty-all-4.0.17.Final.pom https://repo.huaweicloud.com/kunpeng/maven/io/netty/netty-all/4.0.17.Final/netty-all-4.0.17.Final.pom.sha1 --no-check-certificate +``` #### 5 编译Hive-3.1.3 @@ -275,7 +339,7 @@ Error:Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6. ``` 3.3.0 2.12 -2.12.15 +2.12.13 ``` 2）vim ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUtils.java ``` -- Gitee From f0246d06b5a82e9ab326da6d34af819895ae8a36 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 08:26:41 +0000 Subject: [PATCH 60/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.=20hadoop=20=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../hadoop.md" | 113 ++++++++++++++++-- 1 file changed, 106 insertions(+), 7 deletions(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" index 43f2a84..36a1f6a 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" @@ -254,7 +254,7 @@ vim mapred-site.xml mapreduce.framework.name yarn - true + true The runtime framework for executing MapReduce jobs @@ -319,12 +319,12 @@ vim yarn-site.xml 步骤2 添加或修改ÑnĒªñà†ì²Ñn标签范围内的参数。 yarn.nodemanager.aux-services - ɍü†ÃñžɌm†ÝàžšñžȎä¯ñďžɍȍü†ÃñžɌ - ɍĒn†ÃɌìàñžɍȍĒn†ÃɌ + mapreduce_shuffle + true yarn.nodemanager.aux-services - ɍü†ÃñžɌm†ÝàžšñžȎä¯ñďžɍȍü†ÃñžɌ + mapreduce_shuffle yarn.resourcemanager.hostname @@ -366,7 +366,106 @@ vim yarn-site.xml yarn.nodemanager.pmem-check-enabled false - + + yarn.scheduler.minimum-allocation-mb + 3072 + + + yarn.app.mapreduce.am.resource.mb + 3072 + + + yarn.scheduler.maximum-allocation-vcores + 48 + + + yarn.application.classpath + + /usr/local/hadoop/etc/hadoop, + /usr/local/hadoop/share/hadoop/common/*, + /usr/local/hadoop/share/hadoop/common/lib/*, + /usr/local/hadoop/share/hadoop/hdfs/*, + /usr/local/hadoop/share/hadoop/hdfs/lib/*, + /usr/local/hadoop/share/hadoop/mapreduce/*, + /usr/local/hadoop/share/hadoop/mapreduce/lib/*, + /usr/local/hadoop/share/hadoop/yarn/*, + /usr/local/hadoop/share/hadoop/yarn/lib/* + + + + yarn.nodemanager.local-dirs /data/data1/hadoop/yarn/local,/data/data2/ +hadoop/yarn/local,/data/data3/hadoop/yarn/local,/data/data4/hadoop/yarn/local,/data/data5/hadoop/yarn/ +local,/data/data6/hadoop/yarn/local,/data/data7/hadoop/yarn/local,/data/data8/hadoop/yarn/local,/data/ +data9/hadoop/yarn/local,/data/data10/hadoop/yarn/local,/data/data11/hadoop/yarn/local,/data/data12/ +hadoop/yarn/local + + + yarn.nodemanager.log-dirs /data/data1/hadoop/yarn/log,/data/data2/ +hadoop/yarn/log,/data/data3/hadoop/yarn/log,/data/data4/hadoop/yarn/log,/data/data5/hadoop/yarn/log,/ +data/data6/hadoop/yarn/log,/data/data7/hadoop/yarn/log,/data/data8/hadoop/yarn/log,/data/data9/ +hadoop/yarn/log,/data/data10/hadoop/yarn/log,/data/data11/hadoop/yarn/log,/data/data12/hadoop/yarn/ +log + +``` +``` +节点agent1、agent2、agent3分别创建yarn.nodemanager.local-dirs对应目录。 +举例：mkdir -p /data/data{1,2,3,4,5,6,7,8,9,10,11,12}/hadoop/yarn +``` +修改 slaves 或 workers ``` - - +步骤1 确认Hadoop版本，3.x以下的版本编辑slaves文件，3.x及以上的编辑workers文件。 +步骤2 编辑文件（本文版本3.1.1）。 +vim workers +步骤3 修改workers文件，只保存所有agent节点的IP地址（可用主机名代替），其余内容均 +删除。 +agent1 +agent2 +agent3 +``` +##### 4.4 同步配置到其它节点 +``` +步骤1 所有节点依次创建journaldata目录。 +mkdir -p /usr/local/hadoop-3.3.4/journaldata +步骤2 拷贝hadoop-3.1.1到agent1、agent2、agent3节点的“/usr/local”目录。 +scp -r /usr/local/hadoop-3.3.4 root@agent1:/usr/local +scp -r /usr/local/hadoop-3.3.4 root@agent2:/usr/local +scp -r /usr/local/hadoop-3.3.4 root@agent3:/usr/local +步骤3 分别登录到agent1、agent2、agent3节点，为hadoop-3.3.4建立软链接。 +cd /usr/local +ln -s hadoop-3.3.4 hadoop +``` +##### 4.5启动 Hadoop 集群 +``` +步骤1 启动ZooKeeper集群。 +分别在agent1，agent2，agent3节点上启动ZooKeeper。 +cd /usr/local/zookeeper/bin +./zkServer.sh start +步骤2 启动JournalNode。 +分别在agent1，agent2，agent3节点上启动JournalNode。 +说明 +只在第一次进行格式化操作时，需要执行步骤2-步骤4，完成格式化后，下次启动集群，只需要 +执行步骤1、步骤5、步骤6。 +cd /usr/local/hadoop/sbin +./hadoop-daemon.sh start journalnode +步骤3 格式化HDFS。 +1. 在server1节点上格式化HDFS。 +hdfs namenode -format +2. 格式化后集群会根据core-site.xml配置的hadoop.tmp.dir参数生成目录。 +本文档配置目录为“/home/hadoop_tmp”。 +步骤4 格式化ZKFC。 +在server1节点上格式化ZKFC。 +hdfs zkfc -formatZK +步骤5 启动HDFS。 +在server1节点上启动HDFS。 +cd /usr/local/hadoop/sbin +./start-dfs.sh +步骤6 启动Yarn。 +在server1节点上启动Yarn。 +cd /usr/local/hadoop/sbin +./start-yarn.sh +步骤7 观察进程是否都正常启动。 +使用jps命令查看进程 +``` +##### 4.6 验证hadoop +在浏览器中输入URL地址，访问Hadoop Web页面，URL格式为“http:// +server1:50070”。 -- Gitee From ff3aa8f4f683dbbe6622b0fbfb968aaabf22bfe4 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 08:29:08 +0000 Subject: [PATCH 61/99] =?UTF-8?q?add=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 3b58aa8e19708fbfd62af69d70f53c2d761f1259 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 08:31:30 +0000 Subject: [PATCH 62/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper.md" | 3 +++ 1 file changed, 3 insertions(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" index e69de29..affa3d4 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" @@ -0,0 +1,3 @@ +### zookeeper-3.8.1部署指南 + +详情参考hadoop部署指南 [zookeeper部署](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md) \ No newline at end of file -- Gitee From fd1eaad7f7c12517e006cd1564014123b7198a83 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 08:32:03 +0000 Subject: [PATCH 63/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/zookeeper.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../zookeeper.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" index affa3d4..d9bc751 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/zookeeper.md" @@ -1,3 +1,3 @@ ### zookeeper-3.8.1部署指南 -详情参考hadoop部署指南 [zookeeper部署](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md) \ No newline at end of file +详情参考hadoop部署指南 [hadoop部署指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md) \ No newline at end of file -- Gitee From de9001fe9aa7d0b619d5b00c569f2da9e3be93fd Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 08:32:37 +0000 Subject: [PATCH 64/99] =?UTF-8?q?add=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hbase.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 223fe594e88b6bbf694514949187e8aead309a0b Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 08:45:34 +0000 Subject: [PATCH 65/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../hadoop.md" | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" index 36a1f6a..15fb3d7 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" @@ -4,8 +4,8 @@ | 软件 | 说明 | 获取方法 | | ---------------- |---------------- |---------------- | |OpenJDK|1.8.0_342|yum安装或者官网获取| -|hadoop|3.3.4|官网获取，aarch64版本需移植，参考移植指南| -|zookeeper|3.8.1|官网获取，aarch64版本需移植，参考移植指南| +|hadoop|3.3.4|官网获取，aarch64版本需移植，参考[hadoop移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hadoop.md)| +|zookeeper|3.8.1|官网获取，aarch64版本需移植，参考[zookeeper移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/zookeeper.md)| ##### 1.2硬件要求 ``` 最低配置：任意CPU、一根内存（大小不限）、一块硬盘（大小不限）。 @@ -54,8 +54,8 @@ java -version ``` #### 3-部署 ZooKeeper ##### 3.1 编译并解压 ZooKeeper +步骤1 参考ZooKeeper移植指南“[zookeeper移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/zookeeper.md)”编译出“zookeeper-3.8.1.tar.gz”部署包。 ``` -步骤1 参考ZooKeeper移植指南编译出“zookeeper-3.8.1.tar.gz”部署包。 步骤2 将zookeeper-3.8.1.tar.gz放置于agent1节点的“/usr/local”目录下，并解压。 mv zookeeper-3.8.1.tar.gz /usr/local cd /usr/local -- Gitee From cb9dbee537231a92136d8c2edc805291bb15f163 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 08:46:31 +0000 Subject: [PATCH 66/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" | 1 + 1 file changed, 1 insertion(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" index 15fb3d7..64a16cc 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hadoop.md" @@ -1,4 +1,5 @@ ### hadoop-3.3.4部署指南 +编写日期 2023.3.13 #### 1- 环境要求 ##### 1.1 建议版本 | 软件 | 说明 | 获取方法 | -- Gitee From 46ef5720a0ecf40044676b97c4a232c70359e8e8 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 09:05:19 +0000 Subject: [PATCH 67/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hbase.md.=20=E5=A2=9E=E5=8A=A0hbase=E9=83=A8=E7=BD=B2?= =?UTF-8?q?=E6=8C=87=E5=8D=97?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../hbase.md" | 182 ++++++++++++++++++ 1 file changed, 182 insertions(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" index e69de29..19a75fa 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" @@ -0,0 +1,182 @@ +### Hbase-2.5.0 移植指南 +编写日期 2023.3.13 +#### 1- 环境要求 +##### 1.1 建议版本 +| 软件 | 说明 | 获取方法 | +| ---------------- |---------------- |---------------- | +|OpenJDK|1.8.0_342|yum安装或者官网获取| +|hadoop|3.3.4|官网获取，aarch64版本需移植，参考[hadoop移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hadoop.md)| +|zookeeper|3.8.1|官网获取，aarch64版本需移植，参考[zookeeper移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/zookeeper.md)| +|hbase|2.5.0|官网获取，aarch64版本需移植，参考[hbase移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hbase.md)| +##### 1.2硬件要求 +``` +最低配置：任意CPU、一根内存（大小不限）、一块硬盘（大小不限）。 +具体配置视实际应用场景而定。 +操作系统要求： 适用于CentOS 7.4~7.6、openeuler-20.03、openEuler-22.03操作系统。 +说明 +本文以openeuler 22.03为例，介绍Hadoop（1+3）集群部署。 +``` +##### 1.3 集群环境规划 +本章节规划以四台机器分别作为集群的节点1、节点2、节点3、节点4。各个节点数据 +| 机器名称 | IP地址 | 硬盘数 | OS & JDK| +| ---------------- |---------------- |---------------- |---------------- | +|server1|IPaddress1|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent1|IPaddress2|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent2|IPaddress3|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent3|IPaddress4|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +##### 1.4 软件规划 +| 机器名称 | 服务名称 | +| ---------------- |---------------- | +|server1|Namenode、ResourceManager、Hmaster| +|agent1|QuorumPeerMain、DataNode、NodeManager、JournalNode、HRegionServer| +|agent2|QuorumPeerMain、DataNode、NodeManager、JournalNode、HRegionServer| +|agent3|QuorumPeerMain、DataNode、NodeManager、JournalNode、HRegionServer| +#### 2- 配置部署环境 +``` +步骤1 依次登录节点1-4，将节点的主机名分别修改为server1、agent1、agent2、agent3。 +hostnamectl set-hostname 主机名 --static +步骤2 登录所有节点，修改“/etc/hosts”文件。 +在hosts文件中添加集群所有节点的“地址-主机名”映射关系。 +IPaddress1 server1 +IPaddress2 agent1 +IPaddress3 agent2 +IPaddress4 agent3 +步骤3 登录所有节点，关闭防火墙。 +systemctl stop firewalld.service +systemctl disable firewalld.service +systemctl status firewalld.service +步骤4 登录所有节点，配置SSH免密登录。 +1. 生成密钥，遇到提示时，按回车。 +ssh-keygen -t rsa +2. 在每台机器上配置SSH免密登录（包括配置自身节点的免密）。 +ssh-copy-id -i ~/.ssh/id_rsa.pub root@节点IP +步骤5 登录所有节点，安装OpenJDK，可使用指定版本jdk。 +yum install -y java-1.8.0 +java -version +``` +#### 3-部署zookeeper +参考 [hadoop部署指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md) +#### 4-部署hadoop +参考 [hadoop部署指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md) +#### 5-部署hbase +##### 5.1 下载并解压 HBase +步骤1 下载HBase。 +官网获取 或者 aarch64版本需移植，参考[hbase移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hbase.md)| +``` +步骤2 将hbase-2.5.0-bin.tar.gz放置于server1节点的“/usr/local”目录，并解压。 +mv hbase-2.5.0-bin.tar.gz /usr/local +tar -zxvf hbase-2.5.0-bin.tar.gz +步骤3 建立软链接，便于后期版本更换。 +ln -s hbase-2.5.0 hbase +``` +##### 5.2 添加 HBase 到环境变量 +``` +步骤1 编辑“/etc/profile”文件。 +vim ȍžìȍÝàÑĒÞ +步骤2 在文件底部添加环境变量，如下所示。 +export HBASE_HOME=/usr/local/hbase +export PATH=$HBASE_HOME/bin:$HBASE_HOME/sbin:$PATH +步骤3 使环境变量生效。 +source /etc/profile +``` +##### 5.3 修改 HBase 配置文件 +``` +说明 +HBase所有的配置文件都在“HBASE_HOME/conf”目录下，修改以下配置文件前，切换到 +“HBASE_HOME/conf”目录。 +cd $HBASE_HOME/conf +``` +修改 hbase-env.sh +``` +步骤1 编辑hbase-env.sh文件。 +vim hbase-env.sh +步骤2 修改环境变量JAVA_HOME为绝对路径，HBASE_MANAGES_ZK设为false。 +export JAVA_HOME=/usr/local/jdk8u252-b09 +export HBASE_MANAGES_ZK=false +export HBASE_LIBRARY_PATH=/usr/local/hadoop/lib/native +``` +修改 hbase-site.xml +``` +步骤1 修改hbase-site.xml文件。 +vim hbase-site.xml +步骤2 添加或修改ÑnĒªñà†ì²Ñn标签范围内的部分参数。 + + + hbase.rootdir + hdfs://server1:9000/HBase + + + hbase.tmp.dir + /usr/local/hbase/tmp + + + hbase.cluster.distributed + true + + + hbase.unsafe.stream.capability.enforce + false + + + hbase.zookeeper.quorum + agent1:2181,agent2:2181,agent3:2181 + + + hbase.unsafe.stream.capability.enforce + false + + +``` +修改 regionservers +``` +步骤1 编辑regionservers文件。 +vim regionservers +步骤2 将regionservers文件内容替换为agent节点IP（可用主机名代替）。 +agent1 +agent2 +agent3 +``` +拷贝 hdfs-site.xml +``` +拷贝hadoop目录下的的的hdfs-site.xml文件到“hbase/conf/”目录，可选择软链接或 +拷贝。 +cp /usr/local/hadoop/etc/hadoop/hdfs-site.xml /usr/local/hbase/conf/hdfs-site.xml +``` +##### 5.4 同步配置到其它节点 +``` +步骤1 拷贝hbase-2.5.0到agent1、agent2、agent3节点的“/usr/local”目录。 +scp -r /usr/local/hbase-2.5.0 root@agent1:/usr/local +scp -r /usr/local/hbase-2.5.0 root@agent2:/usr/local +scp -r /usr/local/hbase-2.5.0 root@agent3:/usr/local +步骤2 分别登录到agent1、agent2、agent3节点，为hbase-2.5.0建立软链接。 +cd /usr/local +ln -s hbase-2.5.0 hbase +``` +##### 5.5 启动 HBase 集群 +``` +步骤1 依次启动ZooKeeper和Hadoop。 +步骤2 在server1节点上启动HBase集群。 +/usr/local/hbase/bin/start-hbase.sh +步骤3 观察进程是否都正常启动。 +jps +server1： +ResourceManager +NameNode +HMaster +agent1： +NodeManager +DataNode +HRegionServer +JournalNode +QuorumPeerMain +``` +##### 5.6 停止 HBase 集群(可选) +在server1节点上停止HBase集群。 +/usr/local/hbase/bin/stop-hbase.sh +##### 5.7 验证 HBase +打开浏览器，可通过URL地址，访问HBase Web页面，URL格式为“http:// +server1:16010”。其中，“server1”填写HMaster进程所在节点的IP地址， +“16010”是HBase 1.0以后版本的默认端口，可通过修改“hbase-site.xml”文件的 +“hbase.master.info.port”参数进行设置。 +通过观察Region Servers数量是否与agent数目相等（本文是3个agent）判断集群是否 +正常启动。 -- Gitee From 0aca7b6b526e76ef653109712dfebdfe12399e8d Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 09:09:22 +0000 Subject: [PATCH 68/99] =?UTF-8?q?add=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" new file mode 100644 index 0000000..e69de29 -- Gitee From fcc2a5cdd8914d243099794846a916c1a1b8c6ba Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 09:19:17 +0000 Subject: [PATCH 69/99] =?UTF-8?q?=E6=96=B0=E5=BB=BA=20=E5=85=B6=E4=BB=96?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\345\205\266\344\273\226/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\345\205\266\344\273\226/.keep" diff --git "a/Docs/\345\205\266\344\273\226/.keep" "b/Docs/\345\205\266\344\273\226/.keep" new file mode 100644 index 0000000..e69de29 -- Gitee From 3513f88c6d68ebcb0aa0042defd87abda95afe08 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 09:22:28 +0000 Subject: [PATCH 70/99] =?UTF-8?q?=E9=87=8D=E5=91=BD=E5=90=8D=20Docs/checkS?= =?UTF-8?q?o=E7=AD=89=E5=B7=A5=E5=85=B7=E5=8C=85=20=E4=B8=BA=20Docs/?= =?UTF-8?q?=E5=B7=A5=E5=85=B7=E5=8C=85?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../.keep" => "Docs/\345\267\245\345\205\267\345\214\205/.keep" | 0 .../.keep" | 0 .../checkSo.md" | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/.keep" => "Docs/\345\267\245\345\205\267\345\214\205/.keep" (100%) rename "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" => "Docs/\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" (100%) rename "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" => "Docs/\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" (100%) diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/.keep" "b/Docs/\345\267\245\345\205\267\345\214\205/.keep" similarity index 100% rename from "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/.keep" rename to "Docs/\345\267\245\345\205\267\345\214\205/.keep" diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" "b/Docs/\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" similarity index 100% rename from "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" rename to "Docs/\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/.keep" diff --git "a/Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" "b/Docs/\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" similarity index 100% rename from "Docs/checkSo\347\255\211\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" rename to "Docs/\345\267\245\345\205\267\345\214\205/checkSo\345\267\245\345\205\267\344\275\277\347\224\250\346\214\207\345\215\227/checkSo.md" -- Gitee From dbf023588e3fb25d65baab05d2fe38581a78fda1 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 09:23:01 +0000 Subject: [PATCH 71/99] update Docs/README.md. Signed-off-by: xiexing --- Docs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Docs/README.md b/Docs/README.md index 57cc220..3914b00 100644 --- a/Docs/README.md +++ b/Docs/README.md @@ -17,7 +17,7 @@ - [spark部署指南](部署指南/spark.md) - [flink部署指南](部署指南/flink.md) - [zookeeper移植指南](移植指南/zookeeper.md) -- **3-checkSo工具**：存放checkSo工具，及使用方法 +- **3-工具包**：存放checkSo工具，及使用方法 - [checkSo工具使用指南](checkSo等工具包/checkSo工具使用指南/checkSo.md) - **4-其他**：不属于上述三类的文档 -- Gitee From b04098000f8d70491b036044ae158c901c44b4f4 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Mon, 13 Mar 2023 10:44:48 +0000 Subject: [PATCH 72/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hive.md" | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" index e69de29..8ad03d3 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" @@ -0,0 +1,55 @@ +### hive-3.1.3部署指南 +编写日期 2023.3.13 +#### 1- 环境要求 +##### 1.1 建议版本 +| 软件 | 说明 | 获取方法 | +| ---------------- |---------------- |---------------- | +|OpenJDK|1.8.0_342|yum安装或者官网获取| +|hadoop|3.3.4|官网获取，aarch64版本需移植，参考[hadoop移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hadoop.md)| +|hive|3.1.3|官网获取，aarch64版本需移植，参考[hive移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hive.md)| +##### 1.2硬件要求 +``` +最低配置：任意CPU、一根内存（大小不限）、一块硬盘（大小不限）。 +具体配置视实际应用场景而定。 +操作系统要求： 适用于CentOS 7.4~7.6、openeuler-20.03、openEuler-22.03操作系统。 +说明 +本文以openeuler 22.03为例，介绍Hadoop（1+3）集群部署。 +``` +##### 1.3 集群环境规划 +本章节规划以四台机器分别作为集群的节点1、节点2、节点3、节点4。各个节点数据 +| 机器名称 | IP地址 | 硬盘数 | OS & JDK| +| ---------------- |---------------- |---------------- |---------------- | +|server1|IPaddress1|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent1|IPaddress2|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent2|IPaddress3|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent3|IPaddress4|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +##### 1.4 软件规划 +| 机器名称 | 服务名称 | +| ---------------- |---------------- | +|server1|Namenode、ResourceManager、RunJar、RunJar| +|agent1|DataNode、NodeManager、JournalNode| +|agent2|DataNode、NodeManager、JournalNode| +|agent3|DataNode、NodeManager、JournalNode| +#### 2- 配置部署环境 +``` +步骤1 依次登录节点1-4，将节点的主机名分别修改为server1、agent1、agent2、agent3。 +hostnamectl set-hostname 主机名 --static +步骤2 登录所有节点，修改“/etc/hosts”文件。 +在hosts文件中添加集群所有节点的“地址-主机名”映射关系。 +IPaddress1 server1 +IPaddress2 agent1 +IPaddress3 agent2 +IPaddress4 agent3 +步骤3 登录所有节点，关闭防火墙。 +systemctl stop firewalld.service +systemctl disable firewalld.service +systemctl status firewalld.service +步骤4 登录所有节点，配置SSH免密登录。 +1. 生成密钥，遇到提示时，按回车。 +ssh-keygen -t rsa +2. 在每台机器上配置SSH免密登录（包括配置自身节点的免密）。 +ssh-copy-id -i ~/.ssh/id_rsa.pub root@节点IP +步骤5 登录所有节点，安装OpenJDK，可使用指定版本jdk。 +yum install -y java-1.8.0 +java -version +``` \ No newline at end of file -- Gitee From 8fe432fddd400c2026458d329f0bc9fdcfd28e8d Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 11:16:11 +0000 Subject: [PATCH 73/99] =?UTF-8?q?add=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/flink.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 1c74e69e7a02536efca51166d680ef4cc9fb7be4 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 11:18:40 +0000 Subject: [PATCH 74/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hbase.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" index 19a75fa..ffac455 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" @@ -14,7 +14,7 @@ 具体配置视实际应用场景而定。 操作系统要求： 适用于CentOS 7.4~7.6、openeuler-20.03、openEuler-22.03操作系统。 说明 -本文以openeuler 22.03为例，介绍Hadoop（1+3）集群部署。 +本文以openeuler 22.03为例，介绍flink（1+3）集群部署。 ``` ##### 1.3 集群环境规划 本章节规划以四台机器分别作为集群的节点1、节点2、节点3、节点4。各个节点数据 -- Gitee From be6deb6f40bb73ef26e63b4d2401713afd07e0f5 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 13 Mar 2023 11:28:27 +0000 Subject: [PATCH 75/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hbase.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" index ffac455..618f834 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hbase.md" @@ -72,7 +72,7 @@ ln -s hbase-2.5.0 hbase ##### 5.2 添加 HBase 到环境变量 ``` 步骤1 编辑“/etc/profile”文件。 -vim ȍžìȍÝàÑĒÞ +vim /etc/profile 步骤2 在文件底部添加环境变量，如下所示。 export HBASE_HOME=/usr/local/hbase export PATH=$HBASE_HOME/bin:$HBASE_HOME/sbin:$PATH -- Gitee From bc2ba8a83c53eb1e8382501676f989c137a6a773 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Tue, 14 Mar 2023 01:28:44 +0000 Subject: [PATCH 76/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../hive.md" | 150 +++++++++++++++++- 1 file changed, 149 insertions(+), 1 deletion(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" index 8ad03d3..4315ec5 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" @@ -52,4 +52,152 @@ ssh-copy-id -i ~/.ssh/id_rsa.pub root@节点IP 步骤5 登录所有节点，安装OpenJDK，可使用指定版本jdk。 yum install -y java-1.8.0 java -version -``` \ No newline at end of file +``` +#### 3- 部署Hive +在部署Hive之前请先安装部署好Hadoop,参考https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md +##### 3.1 配置Hive环境变量（下面hive的路径已经提前做了软连接的操作） +``` +vim /etc/profile +#增加以下内容： +# HIVE_HOME +export HIVE_HOME=/usr/local/hive +export PATH=$PATH:$HIVE_HOME/bin + +#保存退出 source 使其生效 +source /etc/profile +``` +##### 3.2 配置Hive文件 +##### 3.2.1 配置hive-env.sh +``` +cd $HIVE_HOME/conf +cp hive-env.sh.template hive-env.sh + +vim hive-env.sh + +export JAVA_HOME= #修改为环境中jdk路径 +export HADOOP_HOME=/usr/local/hadoop #修改为环境中hadoop路径 +export HIVE_CONF_DIR=/usr/local/hive/conf #修改为环境中hive配置文件路径 +``` +##### 3.2.2 配置hive-site.xml +``` + + + + + javax.jdo.option.ConnectionURL + jdbc:mysql://server1:3306/hive?createDatabaseIfNotExist=true + + + + javax.jdo.option.ConnectionDriverName + org.mariadb.jdbc.Driver + + + + javax.jdo.option.ConnectionUserName + root + + + + javax.jdo.option.ConnectionPassword + root + + + + hive.exec.local.scratchdir + /tmp/hive + + + + hive.exec.scratchdir + /tmp/hive + + + + hive.downloaded.resources.dir + /tmp/${hive.session.id}_resources + + + + hive.querylog.location + /tmp/hive + + + + hive.server2.logging.operation.log.location + /tmp/hive/operation_logs + + + + hive.metastore.client.connect.retry.delay + 1s + + + + hive.metastore.client.socket.timeout + 600s + + + + hive.metastore.uris + + + + + hive.server2.enable.doAs + true + + + + hive.server2.thrift.http.port + 10001 + + + + hive.server2.thrift.port + 10000 + + + + hive.server2.transport.mode + binary + + + + hive.metastore.warehouse.dir + /user/hive/warehouse + + +``` +##### 3.2.3 安装mariadb并配置hive元数据库 +``` +openEuler系统直接通过yum命令安装 +yum install -y mariadb* + +启动mysql服务 +systemctl start mariadb + +进入mysql ：mysql -uroot -p + +create database hive + +set password for 'root'@'localhost'=password('root'); +grant all privileges on *.* to root@'%' identified by 'root' with grant option; +让设置立即生效 +flush privileges; + +``` +##### 3.2.4 初始化数据库 +schematool -initSchema -dbType mysql +##### 3.3 启动Hive +``` +启动hadoop集群：statr-all.sh +启动mysql服务：service mysqld start +在安装了hive的机器上输入hive + +简单查看数据库：Show databases; +hive创建数据库：create database bigdata; +show database; +``` + + -- Gitee From 03b0a71015935d614ff0e180c54c1b5e08ab73ac Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Tue, 14 Mar 2023 01:29:44 +0000 Subject: [PATCH 77/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" index 4315ec5..8177f64 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" @@ -54,7 +54,7 @@ yum install -y java-1.8.0 java -version ``` #### 3- 部署Hive -在部署Hive之前请先安装部署好Hadoop,参考https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md +在部署Hive之前请先安装部署好Hadoop,参考[hadoop部署指南]（https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md） ##### 3.1 配置Hive环境变量（下面hive的路径已经提前做了软连接的操作） ``` vim /etc/profile -- Gitee From 4d434b8fb11313c34384c29fbef8026b18580ca7 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Tue, 14 Mar 2023 01:30:12 +0000 Subject: [PATCH 78/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hive.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" index 8177f64..db7d34f 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/hive.md" @@ -54,7 +54,7 @@ yum install -y java-1.8.0 java -version ``` #### 3- 部署Hive -在部署Hive之前请先安装部署好Hadoop,参考[hadoop部署指南]（https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md） +在部署Hive之前请先安装部署好Hadoop,参考[hadoop部署指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md) ##### 3.1 配置Hive环境变量（下面hive的路径已经提前做了软连接的操作） ``` vim /etc/profile -- Gitee From 6642926ca9cb3c052434f52d52315ec7f27c3511 Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Tue, 14 Mar 2023 01:35:07 +0000 Subject: [PATCH 79/99] =?UTF-8?q?add=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/spark.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" new file mode 100644 index 0000000..e69de29 -- Gitee From 42e6fe48da59a085d93a34b3d8b726b2307a9184 Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 02:23:41 +0000 Subject: [PATCH 80/99] =?UTF-8?q?add=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/README.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" new file mode 100644 index 0000000..e69de29 -- Gitee From c368c8f1d71c372931c4366ea8cd1686e91261fa Mon Sep 17 00:00:00 2001 From: Hann <12582270+Hann0321@user.noreply.gitee.com> Date: Tue, 14 Mar 2023 02:25:23 +0000 Subject: [PATCH 81/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/spark.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Hann <12582270+Hann0321@user.noreply.gitee.com> --- .../spark.md" | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" index e69de29..7200396 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/spark.md" @@ -0,0 +1,132 @@ +### spark-3.3.0部署指南 +编写日期 2023.3.14 +#### 1- 环境要求 +##### 1.1 建议版本 +| 软件 | 说明 | 获取方法 | +| ---------------- |---------------- |---------------- | +|OpenJDK|1.8.0_342|yum安装或者官网获取| +|hadoop|3.3.4|官网获取，aarch64版本需移植，参考[hadoop移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hadoop.md)| +|spark|3.3.0|官网获取，aarch64版本需移植，参考[spark移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/spark.md)| +##### 1.2硬件要求 +``` +最低配置：任意CPU、一根内存（大小不限）、一块硬盘（大小不限）。 +具体配置视实际应用场景而定。 +操作系统要求： 适用于CentOS 7.4~7.6、openeuler-20.03、openEuler-22.03操作系统。 +说明 +本文以openeuler 22.03为例，介绍Hadoop（1+3）集群部署。 +``` +##### 1.3 集群环境规划 +本章节规划以四台机器分别作为集群的节点1、节点2、节点3、节点4。各个节点数据 +| 机器名称 | IP地址 | 硬盘数 | OS & JDK| +| ---------------- |---------------- |---------------- |---------------- | +|server1|IPaddress1|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent1|IPaddress2|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent2|IPaddress3|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent3|IPaddress4|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +##### 1.4 软件规划 +| 机器名称 | 服务名称 | +| ---------------- |---------------- | +|server1|Namenode、ResourceManager、RunJar、RunJar| +|agent1|DataNode、NodeManager、JournalNode| +|agent2|DataNode、NodeManager、JournalNode| +|agent3|DataNode、NodeManager、JournalNode| +#### 2- 配置部署环境 +``` +步骤1 依次登录节点1-4，将节点的主机名分别修改为server1、agent1、agent2、agent3。 +hostnamectl set-hostname 主机名 --static +步骤2 登录所有节点，修改“/etc/hosts”文件。 +在hosts文件中添加集群所有节点的“地址-主机名”映射关系。 +IPaddress1 server1 +IPaddress2 agent1 +IPaddress3 agent2 +IPaddress4 agent3 +步骤3 登录所有节点，关闭防火墙。 +systemctl stop firewalld.service +systemctl disable firewalld.service +systemctl status firewalld.service +步骤4 登录所有节点，配置SSH免密登录。 +1. 生成密钥，遇到提示时，按回车。 +ssh-keygen -t rsa +2. 在每台机器上配置SSH免密登录（包括配置自身节点的免密）。 +ssh-copy-id -i ~/.ssh/id_rsa.pub root@节点IP +步骤5 登录所有节点，安装OpenJDK，可使用指定版本jdk。 +yum install -y java-1.8.0 +java -version +``` +#### 3- 部署Spark +在部署Hive之前请先安装部署好Hadoop,参考[hadoop部署指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E9%83%A8%E7%BD%B2%E6%8C%87%E5%8D%97/hadoop.md) +##### 3.1 安装Scala +``` +wget https://downloads.lightbend.com/scala/2.12.15/scala-2.12.15.tgz +tar zxf scala-2.12.15.tgz -C /usr/local/ +ln -s /usr/local/scala /usr/local/scala-2.12.15 +``` +##### 3.2 配置Spark环境变量（下面Spark的路径已经提前做了软连接的操作） +``` +vim /etc/profile +#增加以下内容： +# SPARK_HOME +export SPARK_HOME=/usr/local/spark +export PATH=$PATH:$SPARK_HOME/bin +# SCALA_HOME +export SCALA_HOME=/usr/local/scala +export PATH=${SCALA_HOME}/bin:${PATH} + +#保存退出 source 使其生效 +source /etc/profile +``` +##### 3.3 配置Spark文件 +cd $SPARK_HOME/conf +###### 3.3.1 配置 spark-env.sh +``` +cp spark-env.sh.template spark-env.sh + +vim spark-env.sh + +export JAVA_HOME=/usr/local/jdk1.8.0_342 +export SCALA_HOME=/usr/local/scala +export HADOOP_HOME=/usr/local/hadoop +export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop +``` +###### 3.3.2 配置 spark-defaults.conf +``` +cp spark-defaults.conf.template spark-defaults.conf + +vim spark-defaults.conf + +park.eventLog.dir hdfs://server1:9000/spark_history +spark.eventLog.compress true +spark.eventLog.enabled true +spark.serializer org.apache.spark.serializer.KryoSerializer +spark.history.fs.logDirectory hdfs:///server1:9000/spark_history +spark.yarn.historyServer.address server1:18080 +spark.history.ui.port 18080 +spark.jobhistory.address http://server1:19888/history/logs +``` +###### 3.3.3 配置 log4j2.properties +``` +cp log4j2.properties.template log4j2.properties +``` +###### 3.3.4 配置 slaves +``` +vim slaves +server1 +agent1 +agent2 +agent3 +``` +###### 3.3.5 将hadoop配置文件core-site.xml，hdfs-site.xml拷贝倒spark/conf目录下 +``` +cp $HADOOP_HOME/etc/hadoop/core-site.xml $HADOOP_HOME/etc/hadoop/hdfs-site.xml $SPARK_HOME/conf +``` +###### 3.3.6 将hive配置文件hive-site.xml拷贝倒spark/conf目录下（配置hive on spark模式才进行） +``` +cp $HIVE_HOME/conf/hive-site.xml $SPARK_HOME/conf +``` +##### 3.4 启动Spark +``` +sh $SPARK_HOME/sbin/start-all.sh +sh $SPARK_HOME/sbin/start-history-server.sh +# sh $SPARK_HOME/sbin/start-thriftserver.sh --driver-class-path $SPARK_HOME/jars/${jdbc驱动名称} + +``` -- Gitee From f30324f71bd9ad53ca55953f320a54140573bad7 Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 02:26:50 +0000 Subject: [PATCH 82/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/README.md.=20=E5=A2=9E=E5=8A=A0readerme=E4=BF=A1?= =?UTF-8?q?=E6=81=AF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../README.md" | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" index e69de29..343c49f 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/README.md" @@ -0,0 +1,15 @@ +# Bigdata DOCS 部署指南说明 + +此处存放Bigdata部署相关资料供开发者、用户参考 + +## 目录说明 +- **1-部署指南**：存放bigdata组件部署指南，如hadoop部署指南等类 + - [hadoop部署指南](hadoop.md) + - [hive部署指南](hive.md) + - [hbase部署指南](hbase.md) + - [spark部署指南](spark.md) + - [flink部署指南](flink.md) + - [zookeeper移植指南](zookeeper.md) + +## 说明 +开发者在提交文档之后，请更新此文件，刷新相关目录章节 -- Gitee From 47c0977e8b2cb96c6fb8441b818b62da079aec49 Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 02:28:27 +0000 Subject: [PATCH 83/99] =?UTF-8?q?add=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/README.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../README.md" | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 "Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" new file mode 100644 index 0000000..d3c20b9 --- /dev/null +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" @@ -0,0 +1,12 @@ +# Bigdata 移植指南 目录说明 + +此处存放Bigdata移植资料供开发者、用户参考 + +## 目录说明 +- **1-移植指南**：存放开发者关注的文档，如hadoop aarch64移植步骤及问题等 + - [hadoop移植指南](hadoop.md) + - [hive移植指南](hive.md) + - [hbase移植指南](hbase.md) + - [spark移植指南](spark.md) + - [flink移植指南](flink.md) + - [zookeeper移植指南](zookeeper.md) \ No newline at end of file -- Gitee From c0291644520773447ee13d293844c64a337a08e1 Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 03:01:20 +0000 Subject: [PATCH 84/99] =?UTF-8?q?update=20Docs/=E9=83=A8=E7=BD=B2=E6=8C=87?= =?UTF-8?q?=E5=8D=97/flink.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../flink.md" | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" index e69de29..ff8f50a 100644 --- "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" +++ "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/flink.md" @@ -0,0 +1,166 @@ +### Flink-1.13.0部署指南 +编写日期 2023.3.13 +#### 1- 环境要求 +##### 1.1 建议版本 +| 软件 | 说明 | 获取方法 | +| ---------------- |---------------- |---------------- | +|OpenJDK|1.8.0_342|yum安装或者官网获取| +|flink|1.13.0|官网获取，aarch64版本需移植，参考[flink移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/flink.md) +|hadoop|3.3.4|官网获取，aarch64版本需移植，参考[hadoop移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/hadoop.md)| +|zookeeper|3.8.1|官网获取，aarch64版本需移植，参考[zookeeper移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/zookeeper.md)| +##### 1.2硬件要求 +``` +最低配置：任意CPU、一根内存（大小不限）、一块硬盘（大小不限）。 +具体配置视实际应用场景而定。 +操作系统要求： 适用于CentOS 7.4~7.6、openeuler-20.03、openEuler-22.03操作系统。 +说明 +本文以openeuler 22.03为例，介绍flink部署。 +``` +##### 1.3 集群环境规划 +本章节规划以四台机器分别作为集群的节点1、节点2、节点3、节点4。各个节点数据 +| 机器名称 | IP地址 | 硬盘数 | OS & JDK| +| ---------------- |---------------- |---------------- |---------------- | +|server1|IPaddress1|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent1|IPaddress2|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent2|IPaddress3|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +|agent3|IPaddress4|系统盘：1 * 4TB 数据盘：12 * 4TB HDD|openeuler-22.03 & OpenJDK-1.8.0_342| +##### 1.4 软件规划 +| 机器名称 | 服务名称 | +| ---------------- |---------------- | +|server1|StandaloneSessionClusterEntrypoint、(Namenode、ResourceManager)| +|agent1|TaskManagerRunner、(DataNode、NodeManager、JournalNode)| +|agent2|TaskManagerRunner、(DataNode、NodeManager、JournalNode)| +|agent3|TaskManagerRunner、(DataNode、NodeManager、JournalNode)| +#### 2- 配置部署环境 +``` +步骤1 依次登录节点1-4，将节点的主机名分别修改为server1、agent1、agent2、agent3。 +hostnamectl set-hostname 主机名 --static +步骤2 登录所有节点，修改“/etc/hosts”文件。 +在hosts文件中添加集群所有节点的“地址-主机名”映射关系。 +IPaddress1 server1 +IPaddress2 agent1 +IPaddress3 agent2 +IPaddress4 agent3 +步骤3 登录所有节点，关闭防火墙。 +systemctl stop firewalld.service +systemctl disable firewalld.service +systemctl status firewalld.service +步骤4 登录所有节点，配置SSH免密登录。 +1. 生成密钥，遇到提示时，按回车。 +ssh-keygen -t rsa +2. 在每台机器上配置SSH免密登录（包括配置自身节点的免密）。 +ssh-copy-id -i ~/.ssh/id_rsa.pub root@节点IP +步骤5 登录所有节点，安装OpenJDK，可使用指定版本jdk。 +yum install -y java-1.8.0 +java -version +``` +#### 3-flink部署 +Flink的部署有3种模式，分别是local模式、Standalone模式、yarn模式。其中local就是单机模式，一般来说用于本地开发测试；Standalone跟yarn模式都可以支撑集群部署、实现HA，但是两者在任务分配机制、内存管理等内容上有比较大的差异。一般在处理计算数据量级非常大的生产环境，使用flink on yarn的模式更多一些。 + +下载flink。 官网获取 或者 aarch64版本需移植，参考[flink移植指南](https://gitee.com/macchen1/bigdata/blob/change-bigdat/Docs/%E7%A7%BB%E6%A4%8D%E6%8C%87%E5%8D%97/flink.md)。 +##### 3.1 本地模式 +解压并直接执行 ./bin/start-cluster.sh 即可启动flink服务。 +##### 3.2 Standalone模式 +``` +步骤1 将flink-1.13.0-bin.tar.gz放置于server1节点的“/usr/local”目录，并解压。 +mv flink-1.13.0-bin.tar.gz /usr/local +tar -zxvf flink-1.13.0-bin.tar.gz +步骤2 建立软链接，便于后期版本更换。 +ln -s flink-1.13.0 flink +``` +###### 3.2.1 添加 flink 到环境变量 +``` +步骤1 编辑“/etc/profile”文件。 +vim /etc/profile +步骤2 在文件底部添加环境变量，如下所示。 +export FLINK=HOME=/usr/local/flink +export PATH=$FLINK_HOME/bin:$PATH +步骤3 使环境变量生效。 +source /etc/profile +说明 +flink所有的配置文件都在“FLINK_HOME/conf”目录下，修改以下配置文件前，切换到 +“FLINK_HOME/conf”目录。 +cd $FLINK_HOME/conf +``` +###### 3.2.2配置flink conf文件 +配置flink-conf.yaml +``` +vim flink-conf.yaml +jobmanager.rpc.address: server1 +注： +如需配置flink的历史任务地址则需要配置hadoop，配以下参数： +# 指定由JobManager归档的作业信息所存放的目录，这里使用的是HDFS +jobmanager.archive.fs.dir: hdfs://server1:9000/completed-jobs/ +# 指定History Server扫描哪些归档目录，多个目录使用逗号分隔 +historyserver.archive.fs.dir: hdfs://server1:9000/completed-jobs/ +# 指定History Server间隔多少毫秒扫描一次归档目录 +historyserver.archive.fs.refresh-interval: 10000 +# History Server所绑定的ip，0.0.0.0代表允许所有ip访问 +historyserver.web.address: 0.0.0.0 +# 指定History Server所监听的端口号 +historyserver.web.port: 8082 +内存配置用户自定义，略 +``` +配置 master +``` +vim masters +修改为server1的地址 +``` +配置 work +``` +vim workers +agent1 +agent2 +agent3 +``` +##### 3.2.3 同步配置到其它节点 +``` +步骤1 拷贝hbase-2.5.0到agent1、agent2、agent3节点的“/usr/local”目录。 +scp -r /usr/local/flink-1.13.0 root@agent1:/usr/local +scp -r /usr/local/flink-1.13.0 root@agent2:/usr/local +scp -r /usr/local/flink-1.13.0 root@agent3:/usr/local +步骤2 分别登录到agent1、agent2、agent3节点，为flink-1.13.0 建立软链接。 +cd /usr/local +ln -s flink-1.13.0 flink +``` +###### 3.2.4 启动flink +./start-cluster.sh +###### 3.2.5 查看flink +可通过master所在机器地址查看运行状态：server1:8081 +##### 3.3 flink on yarn +配置hadoop。再参考flink Standalone + +##### 3.4 flink 绑核 +``` +vim bin/taskmanager.sh +注释如下信息： +60 #if [[ $STARTSTOP == "start-foreground" ]]; then + 61 # exec "${FLINK_BIN_DIR}"/flink-console.sh $ENTRYPOINT "${ARGS[@]}" + 62 #else + 63 # if [[ $FLINK_TM_COMPUTE_NUMA == "false" ]]; then + 64 # # Start a single TaskManager + 65 # "${FLINK_BIN_DIR}"/flink-daemon.sh $STARTSTOP $ENTRYPOINT "${ARGS[@]}" + 66 # else + 67 # Example output from `numactl --show` on an AWS c4.8xlarge: + 68 # policy: default + 69 # preferred node: current + 70 # physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 2 9 30 31 32 33 34 35 + 71 # cpubind: 0 1 + 72 # nodebind: 0 1 + 73 # membind: 0 1 + 74 read -ra NODE_LIST <<< $(numactl --show | grep "^nodebind: ") + 75 for NODE_ID in "${NODE_LIST[@]:1}"; do + 76 # Start a TaskManager for each NUMA node + 77 numactl --membind=$NODE_ID --cpunodebind=$NODE_ID -- "${FLINK_BIN_DIR}"/flink-daemon.sh $ STARTSTOP $ENTRYPOINT "${ARGS[@]}" + 78 done + 79 # fi + 80 #fi + +查看是否生效 +jps +taskset –pc 进程号 +可查看到TaskManagerRunner 在哪几个cpu上运行。 +``` + + + -- Gitee From b3cbca177ca25027f57d983a75c90ba8d50165ec Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 03:03:31 +0000 Subject: [PATCH 85/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/README.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../README.md" | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" index d3c20b9..a3c0113 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/README.md" @@ -9,4 +9,6 @@ - [hbase移植指南](hbase.md) - [spark移植指南](spark.md) - [flink移植指南](flink.md) - - [zookeeper移植指南](zookeeper.md) \ No newline at end of file + - [zookeeper移植指南](zookeeper.md) +## 说明 +开发者在提交文档之后，请更新此文件，刷新相关目录章节 \ No newline at end of file -- Gitee From ead481c1fd93c84f47e9c8a9227a246aef17b0fb Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 03:06:38 +0000 Subject: [PATCH 86/99] update README.md. Signed-off-by: xiexing --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cb8164b..64e8757 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# bigdata +# bigdata [English](readme-English.md) ## 愿景 -- Gitee From 92fbbdd1b6368329d6cf7d5a20d8ab4c2f7b3c8a Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 03:21:17 +0000 Subject: [PATCH 87/99] update readme-English.md. Signed-off-by: xiexing --- readme-English.md | 110 +++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 55 deletions(-) diff --git a/readme-English.md b/readme-English.md index 1955779..fd23700 100644 --- a/readme-English.md +++ b/readme-English.md @@ -27,7 +27,7 @@ Explore users' big data requirements and incubate potential big data components. - ZeYi Wu [wuzeyi1](https://gitee.com/wuzeyi1) email: zaki.wu@huawei.com - Qiang Chen [macchen1](https://gitee.com/macchen1) email: mac.chenqiang@huawei.com -# 创新项目 +# innovative project | innovative project | version | status | category | | ---- | ---- | ---- | ---- | | [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | Introducting| math library | @@ -43,62 +43,62 @@ openeuler bigda internship tasks link address ：https://www.openeuler.org/zh/si | event | version | status | | ---------------- |---------------- |---------------- | -| [accumulo](https://gitee.com/src-openeuler/accumulo) | 2.0.1 | 已落版本 | -| [alluxio](https://gitee.com/src-openeuler/alluxio) | 2.7.0 | 已落版本 | -| [ambari](https://gitee.com/src-openeuler/ambari) | 2.7.6 | 已落版本 | -| [apache-atlas](https://gitee.com/src-openeuler/apache-atlas) | 2.1.0 | 已落版本 | -| [apache-ranger](https://gitee.com/src-openeuler/apache-ranger) | 2.1.0 | 已落版本 | -| [avor](https://gitee.com/src-openeuler/avor) | 无 | 已建仓 | -| [bigtop](https://gitee.com/src-openeuler/bigtop) | 无 | 已建仓 | -| [blaze](https://gitee.com/src-openeuler/blaze) | 0.10.1 | 已落版本 | -| [calcite](https://gitee.com/src-openeuler/calcite) | 1.30.0 | 已落版本 | -| [datafu](https://gitee.com/src-openeuler/datafu) | 1.6.1 | 已落版本 | -| [delta](https://gitee.com/src-openeuler/delta) | 1.2.1 | 已落版本 | -| [druid](https://gitee.com/src-openeuler/druid) | 0.22.1 | 已落版本 | -| [flink](https://gitee.com/src-openeuler/flink) | 1.13.0 | 已落版本 | -| [flume](https://gitee.com/src-openeuler/flume) | 1.10.0 | 已落版本 | -| [giraph](https://gitee.com/src-openeuler/giraph) | 1.3.0 | 已落版本 | -| [hadoop](https://gitee.com/src-openeuler/hadoop) | 3.3.4 | 已落版本 | -| [hadoop-3.1](https://gitee.com/src-openeuler/hadoop-3.1) | 3.1.4 | 已落版本 | -| [hbase](https://gitee.com/src-openeuler/hbase) | 2.2.5 | 已落版本 | -| [hive](https://gitee.com/src-openeuler/hive) | 3.1.3 | 已落版本 | -| [hudi](https://gitee.com/src-openeuler/hudi) | 0.11.0 | 已落版本 | -| [hue](https://gitee.com/src-openeuler/hue) | 无 | 已建仓 | -| [ibis](https://gitee.com/src-openeuler/ibis) | 1.3.0 | 已落版本 | -| [iceberg](https://gitee.com/src-openeuler/iceberg) | 0.13.0 | 已落版本 | -| [ignite](https://gitee.com/src-openeuler/ignite) | 2.9.0 | 已落版本 | -| [impala](https://gitee.com/src-openeuler/impala) | 无 | 已建仓 | -| [kafka](https://gitee.com/src-openeuler/kafka) | 2.8.2 | 已落版本 | -| [kite-sdk](https://gitee.com/src-openeuler/kite-sdk) | 无 | 已建仓 | -| [knox](https://gitee.com/src-openeuler/knox) | 1.6.1 | 已落版本 | -| [kudu](https://gitee.com/src-openeuler/kudu) | 无 | 已建仓 | -| [libhdfs](https://gitee.com/src-openeuler/libhdfs) | 无 | 已建仓 | -| [llama](https://gitee.com/src-openeuler/llama) | 无 | 已建仓 | -| [mahout](https://gitee.com/src-openeuler/mahout) | 14.1 | 已落版本 | -| [oozie](https://gitee.com/src-openeuler/oozie) | 5.2.1 | 已落版本 | -| [parquet-format](https://gitee.com/src-openeuler/parquet-format) | 2.9.0 | 已落版本 | -| [presto](https://gitee.com/src-openeuler/presto) | 0.26 | 已落版本 | -| [qfs](https://gitee.com/src-openeuler/qfs) | 2.2.4 | 已落版本 | -| [redis5](https://gitee.com/src-openeuler/redis5) | 5.0.4 | 已落版本 | -| [redis6](https://gitee.com/src-openeuler/redis6) | 6.2.7 | 已落版本 | -| [spark](https://gitee.com/src-openeuler/spark) | 3.2.2 | 已落版本 | -| [storm](https://gitee.com/src-openeuler/storm) | 1.2.4 | 已落版本 | -| [ycsb](https://gitee.com/src-openeuler/ycsb) | 0.17.0 | 已落版本 | -| [zeppelin](https://gitee.com/src-openeuler/zeppelin) | 无 | 已建仓 | -| [zookeeper](https://gitee.com/src-openeuler/zookeeper) | 3.8.0 | 已落版本 | - -### 说明 - -| 组件状态 | 说明 | +|[accumulo](https://gitee.com/src-openeuler/accumulo)|2.0.1|Existing Version| +|[alluxio](https://gitee.com/src-openeuler/alluxio)|2.7.0|Existing Version| +|[ambari](https://gitee.com/src-openeuler/ambari)|2.7.6|Existing Version| +|[apache-atlas](https://gitee.com/src-openeuler/apache-atlas)|2.1.0|Existing Version| +|[apache-ranger](https://gitee.com/src-openeuler/apache-ranger) |2.1.0|Existing Version| +|[avor](https://gitee.com/src-openeuler/avor)|None|Established warehouse| +|[bigtop](https://gitee.com/src-openeuler/bigtop)|None|Established warehouse| +|[blaze](https://gitee.com/src-openeuler/blaze) |0.10.1|Existing Version| +|[calcite](https://gitee.com/src-openeuler/calcite)|1.30.0|Existing Version| +|[datafu](https://gitee.com/src-openeuler/datafu)|1.6.1 |Existing Version| +|[delta](https://gitee.com/src-openeuler/delta) |1.2.1 |Existing Version| +|[druid](https://gitee.com/src-openeuler/druid) |0.22.1 |Existing Version| +|[flink](https://gitee.com/src-openeuler/flink) |1.13.0 |Existing Version| +|[flume](https://gitee.com/src-openeuler/flume) |1.10.0 |Existing Version| +|[giraph](https://gitee.com/src-openeuler/giraph)|1.3.0 |Existing Version| +|[hadoop](https://gitee.com/src-openeuler/hadoop)|3.3.4 |Existing Version| +|[hadoop-3.1](https://gitee.com/src-openeuler/hadoop-3.1)|3.1.4|Existing Version| +|[hbase](https://gitee.com/src-openeuler/hbase) |2.2.5|Existing Version | +|[hive](https://gitee.com/src-openeuler/hive)| 3.1.3 |Existing Version| +|[hudi](https://gitee.com/src-openeuler/hudi)| 0.11.0 |Existing Version| +|[hue](https://gitee.com/src-openeuler/hue)|无|Existing Version| +|[ibis](https://gitee.com/src-openeuler/ibis)|1.3.0|Existing Version| +|[iceberg](https://gitee.com/src-openeuler/iceberg)|0.13.0|Existing Version| +|[ignite](https://gitee.com/src-openeuler/ignite)|2.9.0 |Existing Version| +|[impala](https://gitee.com/src-openeuler/impala)|None|Established warehouse| +|[kafka](https://gitee.com/src-openeuler/kafka) |2.8.2| Existing Version| +|[kite-sdk](https://gitee.com/src-openeuler/kite-sdk)|None|Established warehouse| +|[knox](https://gitee.com/src-openeuler/knox)|1.6.1|Existing Version| +|[kudu](https://gitee.com/src-openeuler/kudu)|None|Established warehouse| +|[libhdfs](https://gitee.com/src-openeuler/libhdfs)|None|Established warehouse| +|[llama](https://gitee.com/src-openeuler/llama) |None|Established warehouse| +|[mahout](https://gitee.com/src-openeuler/mahout)|14.1|Existing Version| +|[oozie](https://gitee.com/src-openeuler/oozie) |5.2.1|Existing Version| +|[parquet-format](https://gitee.com/src-openeuler/parquet-format)|2.9.0|Existing Version| +|[presto](https://gitee.com/src-openeuler/presto)|0.26|Existing Version| +|[qfs](https://gitee.com/src-openeuler/qfs)|2.2.4|Existing Version| +|[redis5](https://gitee.com/src-openeuler/redis5)|5.0.4|Existing Version| +|[redis6](https://gitee.com/src-openeuler/redis6)|6.2.7|Existing Version| +|[spark](https://gitee.com/src-openeuler/spark) |3.2.2|Existing Version| +|[storm](https://gitee.com/src-openeuler/storm) |1.2.4|Existing Version| +|[ycsb](https://gitee.com/src-openeuler/ycsb)|0.17.0|Existing Version| +|[zeppelin](https://gitee.com/src-openeuler/zeppelin)|None|Established warehouse| +|[zookeeper](https://gitee.com/src-openeuler/zookeeper) |3.8.0|Existing Version| + +### explain + +| Component Status |explain | | -------- | ---------------------------------------------------------- | -| 已落版本 | 该组件已落openEuler发行版本 | -| 引入中 | 该组件已在openEuler建仓且代码正在从上游社区引入，待进入openEuler发行版本 | -| 孵化中 | 该组件由BIGDATA SIG所孵化，待进入openEuler发行版 | -| 已建仓 | 该组件在openEuler中已建仓，尚需从上游社区引入源码. Help wanted~ | -| 待引入 | 该组件有需求，但尚未在openEuler中建仓 | +| Existing Version | This component has been released in the openEuler version| +| Introducting |This component has been created in OpenEuler and its code is being imported from the upstream community. It will be released in OpenEuler.| +| In hatching |This component is incubated by the BIGDATA SIG and will be released in the openEuler release.| +| Established warehouse | This component has a warehouse in OpenEuler, and the source code needs to be imported from the upstream community. Help wanted~ | +| To be introduced| This component is required, but the warehouse has not been created in openEuler. | -## 链接 +## inkage openEuler bigdata sig组 简介: https://gitee.com/openeuler/community/tree/master/sig/bigdata openEuler bigdata 月刊归档地址： @@ -109,7 +109,7 @@ openEuler bigdata 月刊归档地址： -## 联系方式 +## contact details ### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) ### 微信公众号：openEuler-bigdata-sig -- Gitee From b70b7caa21e9731870b14dd0586be174ac38e0b3 Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 03:22:54 +0000 Subject: [PATCH 88/99] =?UTF-8?q?add=20Docs/=E5=B7=A5=E5=85=B7=E5=8C=85/RE?= =?UTF-8?q?ADME.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- "Docs/\345\267\245\345\205\267\345\214\205/README.md" | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 "Docs/\345\267\245\345\205\267\345\214\205/README.md" diff --git "a/Docs/\345\267\245\345\205\267\345\214\205/README.md" "b/Docs/\345\267\245\345\205\267\345\214\205/README.md" new file mode 100644 index 0000000..915dab0 --- /dev/null +++ "b/Docs/\345\267\245\345\205\267\345\214\205/README.md" @@ -0,0 +1,10 @@ +# Bigdata SIG WiKi + +此处存放Bigdata相关资料供开发者、用户参考 + +## 目录说明 +- **1-工具包**：存放checkSo工具，及使用方法 + - [checkSo工具使用指南](checkSo工具使用指南/checkSo.md) + +## 说明 +开发者在提交文档之后，请更新此文件，刷新相关目录章节 -- Gitee From 06cc1e18bd05b79c3ea41d6067d6165e061dcc7c Mon Sep 17 00:00:00 2001 From: xiexing Date: Tue, 14 Mar 2023 03:32:46 +0000 Subject: [PATCH 89/99] update README.md. Signed-off-by: xiexing --- README.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 64e8757..67f4ae4 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ -# bigdata [English](readme-English.md) +# bigdata +[English](readme-English.md) ## 愿景 @@ -38,9 +39,9 @@ openEuler开源实习地址：https://www.openeuler.org/zh/internship/ openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata -## 项目清单： +## 组件清单 -| 项目 | 版本 | 状态 | +| 组件 | 版本 | 状态 | | ---------------- |---------------- |---------------- | | [accumulo](https://gitee.com/src-openeuler/accumulo) | 2.0.1 | 已落版本 | | [alluxio](https://gitee.com/src-openeuler/alluxio) | 2.7.0 | 已落版本 | @@ -108,9 +109,10 @@ openEuler bigdata 月刊归档地址： -## 联系方式 -### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) -### 微信公众号：openEuler-bigdata-sig +## 联系方式： +邮箱地址：https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/ + +微信公众号：openEuler-bigdata-sig + -# 路标 -- Gitee From f3f84b35fac43e4ad54dfd8ddf96ee0b9ddfe374 Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Tue, 14 Mar 2023 09:14:52 +0000 Subject: [PATCH 90/99] update README.md. Signed-off-by: MacChen1 --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 67f4ae4..e2052a3 100644 --- a/README.md +++ b/README.md @@ -7,10 +7,11 @@ 发掘用户大数据需求，孵化有潜力的大数据组件。 ## 业务范围 - - openEuler上的大数据基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 - - openEuler上大数据平台，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 - - openEuler上的大数据组件和平台的性能优化，让大数据在openEuler上更好用。 - - openEuler上的大数据相关能力集成，新芯片和软件进入openEuler时的支持。 +在openEuler平台上持续构建大数据生态： +1.基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 +2.统一界面，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 +3.性能优化，让大数据在openEuler平台上发挥极致性能。 +4.集成能力，新芯片和软件进入openEuler时的支持。 ## 会议 -- Gitee From e56a93ca5c5f8c590e3a32b21668fe0e28cccb98 Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Tue, 14 Mar 2023 09:16:42 +0000 Subject: [PATCH 91/99] update README.md. Signed-off-by: MacChen1 --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index e2052a3..7b5abda 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,9 @@ [English](readme-English.md) -## 愿景 +## 愿景和职责 构建和完善openEuler社区下的大数据生态，打造活跃的大数据交流平台，丰富和提升大数据组件功能和性能， 发掘用户大数据需求，孵化有潜力的大数据组件。 - -## 业务范围 在openEuler平台上持续构建大数据生态： 1.基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 2.统一界面，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 -- Gitee From f70ab2870367316323dbc9c5fadadff405ee4613 Mon Sep 17 00:00:00 2001 From: xiexing Date: Wed, 15 Mar 2023 02:08:46 +0000 Subject: [PATCH 92/99] update README.md. Signed-off-by: xiexing --- README.md | 43 ++++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 7b5abda..662b14d 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,20 @@ # bigdata -[English](readme-English.md) +[English version introduction](readme-English.md) ## 愿景和职责 构建和完善openEuler社区下的大数据生态，打造活跃的大数据交流平台，丰富和提升大数据组件功能和性能， 发掘用户大数据需求，孵化有潜力的大数据组件。 在openEuler平台上持续构建大数据生态： -1.基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 -2.统一界面，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 -3.性能优化，让大数据在openEuler平台上发挥极致性能。 -4.集成能力，新芯片和软件进入openEuler时的支持。 +- 基础运行能力，包括数据采集、数据传输、数据存储、数据分析、数据可视等。 +- 统一界面，将各种常用的工具软件集成提供统一的用户界面解决让大数据在openEuler上更易用。 +- 性能优化，让大数据在openEuler平台上发挥极致性能。 +- 集成能力，新芯片和软件进入openEuler时的支持。 ## 会议 -- 会议主题： OpenEuler bigdata meetup -- 会议时间： 每周四 16:00-17:00(GMT+08:00) 中国标准时间-北京 每两周一次 + 按需出发，详情可见订阅邮箱。 -- 点击链接入会，或添加至会议列表：xxxxxxxx -- 会议 ID：xxxxxx -- 订阅邮箱：bigdata@openeuler.org +- 会议时间： 周四 16:00-17:00(GMT+08:00) 中国标准时间-北京 每两周一次 + 按需出发，详情可见订阅邮箱。 +- 订阅邮箱地址：bigdata@openeuler.org ## 成员 @@ -30,6 +27,8 @@ | 项目 | 版本 | 状态 | 类别 | | ---- | ---- | ---- | ---- | | [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | +|OmniRuntime算子下推 Hive特性||引入中|bigdata| +|Spark SQL Native||引入中|bigdata| # 开源实习 openEuler开源实习是openEuler社区和社区合作单位共同发起的线上实习项目，旨在鼓励在校学生积极参与开源社区，在实际的开源环境中提升实践能力，在社区中成长为优秀的开源人才。由社区提供实习任务，并提供导师辅导，学生通过实习申请后，可在社区领取任务，每完成一个任务可获得相应积分，积分累计达规定量后，可获得实习证明和实习工资。openeuler社区发布大量的实习任务，欢迎大家积极参与。 @@ -47,8 +46,8 @@ openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail | [ambari](https://gitee.com/src-openeuler/ambari) | 2.7.6 | 已落版本 | | [apache-atlas](https://gitee.com/src-openeuler/apache-atlas) | 2.1.0 | 已落版本 | | [apache-ranger](https://gitee.com/src-openeuler/apache-ranger) | 2.1.0 | 已落版本 | -| [avor](https://gitee.com/src-openeuler/avor) | 无 | 已建仓 | -| [bigtop](https://gitee.com/src-openeuler/bigtop) | 无 | 已建仓 | +| [avor](https://gitee.com/src-openeuler/avor) | 无 | 引入中 | +| [bigtop](https://gitee.com/src-openeuler/bigtop) | 无 | 引入中 | | [blaze](https://gitee.com/src-openeuler/blaze) | 0.10.1 | 已落版本 | | [calcite](https://gitee.com/src-openeuler/calcite) | 1.30.0 | 已落版本 | | [datafu](https://gitee.com/src-openeuler/datafu) | 1.6.1 | 已落版本 | @@ -62,17 +61,17 @@ openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail | [hbase](https://gitee.com/src-openeuler/hbase) | 2.2.5 | 已落版本 | | [hive](https://gitee.com/src-openeuler/hive) | 3.1.3 | 已落版本 | | [hudi](https://gitee.com/src-openeuler/hudi) | 0.11.0 | 已落版本 | -| [hue](https://gitee.com/src-openeuler/hue) | 无 | 已建仓 | +| [hue](https://gitee.com/src-openeuler/hue) | 无 | 引入中 | | [ibis](https://gitee.com/src-openeuler/ibis) | 1.3.0 | 已落版本 | | [iceberg](https://gitee.com/src-openeuler/iceberg) | 0.13.0 | 已落版本 | | [ignite](https://gitee.com/src-openeuler/ignite) | 2.9.0 | 已落版本 | -| [impala](https://gitee.com/src-openeuler/impala) | 无 | 已建仓 | +| [impala](https://gitee.com/src-openeuler/impala) | 无 | 引入中 | | [kafka](https://gitee.com/src-openeuler/kafka) | 2.8.2 | 已落版本 | -| [kite-sdk](https://gitee.com/src-openeuler/kite-sdk) | 无 | 已建仓 | +| [kite-sdk](https://gitee.com/src-openeuler/kite-sdk) | 无 | 引入中 | | [knox](https://gitee.com/src-openeuler/knox) | 1.6.1 | 已落版本 | -| [kudu](https://gitee.com/src-openeuler/kudu) | 无 | 已建仓 | -| [libhdfs](https://gitee.com/src-openeuler/libhdfs) | 无 | 已建仓 | -| [llama](https://gitee.com/src-openeuler/llama) | 无 | 已建仓 | +| [kudu](https://gitee.com/src-openeuler/kudu) | 无 | 引入中 | +| [libhdfs](https://gitee.com/src-openeuler/libhdfs) | 无 | 引入中 | +| [llama](https://gitee.com/src-openeuler/llama) | 无 | 引入中 | | [mahout](https://gitee.com/src-openeuler/mahout) | 14.1 | 已落版本 | | [oozie](https://gitee.com/src-openeuler/oozie) | 5.2.1 | 已落版本 | | [parquet-format](https://gitee.com/src-openeuler/parquet-format) | 2.9.0 | 已落版本 | @@ -83,18 +82,16 @@ openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail | [spark](https://gitee.com/src-openeuler/spark) | 3.2.2 | 已落版本 | | [storm](https://gitee.com/src-openeuler/storm) | 1.2.4 | 已落版本 | | [ycsb](https://gitee.com/src-openeuler/ycsb) | 0.17.0 | 已落版本 | -| [zeppelin](https://gitee.com/src-openeuler/zeppelin) | 无 | 已建仓 | +| [zeppelin](https://gitee.com/src-openeuler/zeppelin) | 无 | 引入中 | | [zookeeper](https://gitee.com/src-openeuler/zookeeper) | 3.8.0 | 已落版本 | ### 说明 | 组件状态 | 说明 | | -------- | ---------------------------------------------------------- | -| 已落版本 | 该组件已落openEuler发行版本 | -| 引入中 | 该组件已在openEuler建仓且代码正在从上游社区引入，待进入openEuler发行版本 | -| 孵化中 | 该组件由BIGDATA SIG所孵化，待进入openEuler发行版 | -| 已建仓 | 该组件在openEuler中已建仓，尚需从上游社区引入源码. Help wanted~ | | 待引入 | 该组件有需求，但尚未在openEuler中建仓 | +| 引入中 | 该组件已在openEuler建仓且代码正在从上游社区引入，待进入openEuler发行版本 | +| 已落版本 | 该组件已落openEuler发行版本 | ## 链接 -- Gitee From a5487361bb39c7a1bf06307934ba0cb4c7648ed8 Mon Sep 17 00:00:00 2001 From: xiexing Date: Wed, 15 Mar 2023 02:12:01 +0000 Subject: [PATCH 93/99] =?UTF-8?q?update=20Docs/=E7=A7=BB=E6=A4=8D=E6=8C=87?= =?UTF-8?q?=E5=8D=97/hadoop.md.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: xiexing --- .../hadoop.md" | 26 +++++++++++-------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" index cd54202..66c3839 100644 --- "a/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" +++ "b/Docs/\347\247\273\346\244\215\346\214\207\345\215\227/hadoop.md" @@ -4,17 +4,21 @@ #### **1. 软硬件环境** -##### **1.1 硬件** -CPU 鲲鹏920 -网络 Ethernet-10GE -存储 SATA 1T -内存 Hynix 512G 2400MHz -##### **1.2 OS** -EulerOS 22.03 -Kernel 5.10.0-60.18.0.50.oe2203.aarch64 -OpenJDK 1.8.0_342 -Maven 3.8.6 -Hadoop 3.3.4 +##### **1.1 硬件** +| 硬件 | 说明 | +| ---------------- |---------------- | +|CPU|鲲鹏920| +|网络|Ethernet-10GE| +|存储|SATA 1T| +|内存|Hynix 512G 2400MHz| +##### **1.2 OS** +| 软件 | 说明 | +| ---------------- |---------------- | +|EulerOS|22.03| +|Kernel|5.10.0-60.18.0.50.oe2203.aarch64| +|OpenJDK|1.8.0_342| +|Maven|3.8.6| +|Hadoop|3.3.4| #### **2.编译环境准备** ##### **2.1 编译工具安装** `yum –y install wget openssl-devel zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel gcc.aarch64 gcc-c++.aarch64 gcc-gfortran.aarch64 libgcc.aarch64 cmake patch protobuf` -- Gitee From f14c9d89f4fcaf92fa38178ed552bc0e6cb87e92 Mon Sep 17 00:00:00 2001 From: xiexing Date: Wed, 15 Mar 2023 06:18:29 +0000 Subject: [PATCH 94/99] update README.md. Signed-off-by: xiexing --- README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 662b14d..d6d0003 100644 --- a/README.md +++ b/README.md @@ -19,23 +19,23 @@ ## 成员 ### Maintainer列表 - - 杨昭 [yangzhao_kl](https://gitee.com/yangzhao_kl) email: yangzhao1@kylinos.cn - - 吴泽艺 [wuzeyi1](https://gitee.com/wuzeyi1) email: zaki.wu@huawei.com - - 陈强 [macchen1](https://gitee.com/macchen1) email: mac.chenqiang@huawei.com + - 杨昭 [yangzhao_kl](https://gitee.com/yangzhao_kl) email地址: yangzhao1@kylinos.cn + - 吴泽艺 [wuzeyi1](https://gitee.com/wuzeyi1) email地址: zaki.wu@huawei.com + - 陈强 [macchen1](https://gitee.com/macchen1) email地址: mac.chenqiang@huawei.com # 创新项目 | 项目 | 版本 | 状态 | 类别 | | ---- | ---- | ---- | ---- | -| [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | -|OmniRuntime算子下推 Hive特性||引入中|bigdata| -|Spark SQL Native||引入中|bigdata| +|[VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | 引入中| 数学库 | +|OmniRuntime算子下推 Hive特性|1.0|引入中|bigdata| +|Spark SQL Native|1.0|引入中|bigdata| # 开源实习 -openEuler开源实习是openEuler社区和社区合作单位共同发起的线上实习项目，旨在鼓励在校学生积极参与开源社区，在实际的开源环境中提升实践能力，在社区中成长为优秀的开源人才。由社区提供实习任务，并提供导师辅导，学生通过实习申请后，可在社区领取任务，每完成一个任务可获得相应积分，积分累计达规定量后，可获得实习证明和实习工资。openeuler社区发布大量的实习任务，欢迎大家积极参与。 +openEuler开源实习是openEuler社区和社区合作单位共同发起的线上实习项目，旨在鼓励在校学生积极参与开源社区，在实际的开源环境中提升实践能力，在社区中成长为优秀的开源人才。 openEuler开源实习地址：https://www.openeuler.org/zh/internship/ -openeuler bigda 实习任务地址：https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata +bigdata 实习任务发布地址：https://www.openeuler.org/zh/sig/sig-detail/?name=bigdata ## 组件清单 @@ -99,9 +99,9 @@ openEuler bigdata sig组 简介: https://gitee.com/openeuler/community/tree/mast openEuler bigdata 月刊归档地址： -邮件列表归档地址： +邮件列表归档地址： -会议纪要地址： +会议纪要地址：[会议纪要](https://gitee.com/openeuler/bigdata/wikis/sig-bigdata%E4%BB%8B%E7%BB%8D) -- Gitee From a630d97fba2279b2f285ffde04ae72017f45a328 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 20 Mar 2023 02:48:03 +0000 Subject: [PATCH 95/99] update README.md. Signed-off-by: xiexing --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index d6d0003..5da3b4c 100644 --- a/README.md +++ b/README.md @@ -86,12 +86,12 @@ bigdata 实习任务发布地址：https://www.openeuler.org/zh/sig/sig-detail/? | [zookeeper](https://gitee.com/src-openeuler/zookeeper) | 3.8.0 | 已落版本 | ### 说明 - -| 组件状态 | 说明 | -| -------- | ---------------------------------------------------------- | -| 待引入 | 该组件有需求，但尚未在openEuler中建仓 | -| 引入中 | 该组件已在openEuler建仓且代码正在从上游社区引入，待进入openEuler发行版本 | -| 已落版本 | 该组件已落openEuler发行版本 | +| 组件状态 | 说明 | +|------|----| +| 待引入 | 该组件有需求，但尚未在openEuler中建仓 | +| 引入中 | 该组件已在openEuler建仓且代码正在从上游社区引入，待进入openEuler发行版本 | +| 已落版本 | 该组件已落openEuler发行版本 | + ## 链接 -- Gitee From a4991eca4fdfbde979103d218c5c86339d901277 Mon Sep 17 00:00:00 2001 From: xiexing Date: Mon, 20 Mar 2023 03:20:15 +0000 Subject: [PATCH 96/99] update readme-English.md. Signed-off-by: xiexing --- readme-English.md | 54 +++++++++++++++++++++-------------------------- 1 file changed, 24 insertions(+), 30 deletions(-) diff --git a/readme-English.md b/readme-English.md index fd23700..af6a1d7 100644 --- a/readme-English.md +++ b/readme-English.md @@ -13,11 +13,8 @@ Explore users' big data requirements and incubate potential big data components. ## Meeting -- Meeting subject： OpenEuler bigdata meetup -- Meeting time： Thursday 16:00-17:00(GMT+08:00) China Standard Time - Beijing Once every two weeks + Departure on demand，For details, see the subscription email address.。 -- Meeting ID：xxxxxx -- Subscription Email Address：bigdata@openeuler.org -- Click the link to join the meeting or add it to the meeting list.：xxxxxxxx +- Meeting time： Thursday 16:00-17:00(GMT+08:00) China Standard Time - Beijing Once every two weeks + Departure on demand，For details, see the subscription email address. +- subscription Email Address：bigdata@openeuler.org ## Member @@ -31,9 +28,11 @@ Explore users' big data requirements and incubate potential big data components. | innovative project | version | status | category | | ---- | ---- | ---- | ---- | | [VectorBlas](https://gitee.com/openeuler/vectorBlas) | 1.0 | Introducting| math library | +|OmniRuntime operator pushdown Hive characteristic|1.0|Introducting|bigdata| +|Spark SQL Native|1.0|Introducting|bigdata| # Open-source Internship -OpenEuler open-source internship is an online internship program jointly sponsored by the openEuler community and community partners. It aims to encourage students to actively participate in the open source community, improve their practical capabilities in the actual open source environment, and grow into excellent open source talents in the community. The community provides internship tasks and provides tutoring. After passing the internship application, students can receive tasks in the community. They can obtain corresponding points for each task completed. After the accumulated points reach the specified amount, they can obtain internship certificates and internship salaries. The openeuler community has released a large number of internships, and you are welcome to participate actively. +OpenEuler open-source internship is an online internship program jointly sponsored by the openEuler community and community partners. It aims to encourage students to actively participate in the open source community, improve their practical capabilities in the actual open source environment, and grow into excellent open source talents in the community. openEuler open-source internship link address：https://www.openeuler.org/zh/internship/ @@ -48,8 +47,8 @@ openeuler bigda internship tasks link address ：https://www.openeuler.org/zh/si |[ambari](https://gitee.com/src-openeuler/ambari)|2.7.6|Existing Version| |[apache-atlas](https://gitee.com/src-openeuler/apache-atlas)|2.1.0|Existing Version| |[apache-ranger](https://gitee.com/src-openeuler/apache-ranger) |2.1.0|Existing Version| -|[avor](https://gitee.com/src-openeuler/avor)|None|Established warehouse| -|[bigtop](https://gitee.com/src-openeuler/bigtop)|None|Established warehouse| +|[avor](https://gitee.com/src-openeuler/avor)|None| Introducting| +|[bigtop](https://gitee.com/src-openeuler/bigtop)|None| Introducting| |[blaze](https://gitee.com/src-openeuler/blaze) |0.10.1|Existing Version| |[calcite](https://gitee.com/src-openeuler/calcite)|1.30.0|Existing Version| |[datafu](https://gitee.com/src-openeuler/datafu)|1.6.1 |Existing Version| @@ -67,13 +66,13 @@ openeuler bigda internship tasks link address ：https://www.openeuler.org/zh/si |[ibis](https://gitee.com/src-openeuler/ibis)|1.3.0|Existing Version| |[iceberg](https://gitee.com/src-openeuler/iceberg)|0.13.0|Existing Version| |[ignite](https://gitee.com/src-openeuler/ignite)|2.9.0 |Existing Version| -|[impala](https://gitee.com/src-openeuler/impala)|None|Established warehouse| +|[impala](https://gitee.com/src-openeuler/impala)|None| Introducting| |[kafka](https://gitee.com/src-openeuler/kafka) |2.8.2| Existing Version| -|[kite-sdk](https://gitee.com/src-openeuler/kite-sdk)|None|Established warehouse| +|[kite-sdk](https://gitee.com/src-openeuler/kite-sdk)|None| Introducting| |[knox](https://gitee.com/src-openeuler/knox)|1.6.1|Existing Version| -|[kudu](https://gitee.com/src-openeuler/kudu)|None|Established warehouse| -|[libhdfs](https://gitee.com/src-openeuler/libhdfs)|None|Established warehouse| -|[llama](https://gitee.com/src-openeuler/llama) |None|Established warehouse| +|[kudu](https://gitee.com/src-openeuler/kudu)|None| Introducting| +|[libhdfs](https://gitee.com/src-openeuler/libhdfs)|None| Introducting| +|[llama](https://gitee.com/src-openeuler/llama) |None| Introducting| |[mahout](https://gitee.com/src-openeuler/mahout)|14.1|Existing Version| |[oozie](https://gitee.com/src-openeuler/oozie) |5.2.1|Existing Version| |[parquet-format](https://gitee.com/src-openeuler/parquet-format)|2.9.0|Existing Version| @@ -84,34 +83,29 @@ openeuler bigda internship tasks link address ：https://www.openeuler.org/zh/si |[spark](https://gitee.com/src-openeuler/spark) |3.2.2|Existing Version| |[storm](https://gitee.com/src-openeuler/storm) |1.2.4|Existing Version| |[ycsb](https://gitee.com/src-openeuler/ycsb)|0.17.0|Existing Version| -|[zeppelin](https://gitee.com/src-openeuler/zeppelin)|None|Established warehouse| +|[zeppelin](https://gitee.com/src-openeuler/zeppelin)|None| Introducting| |[zookeeper](https://gitee.com/src-openeuler/zookeeper) |3.8.0|Existing Version| ### explain -| Component Status |explain | -| -------- | ---------------------------------------------------------- | -| Existing Version | This component has been released in the openEuler version| -| Introducting |This component has been created in OpenEuler and its code is being imported from the upstream community. It will be released in OpenEuler.| -| In hatching |This component is incubated by the BIGDATA SIG and will be released in the openEuler release.| -| Established warehouse | This component has a warehouse in OpenEuler, and the source code needs to be imported from the upstream community. Help wanted~ | -| To be introduced| This component is required, but the warehouse has not been created in openEuler. | - +| Component Status | explain | +|------|----| +|To be introduced | This component is required, but the warehouse has not been created in openEuler | +| Introducting | This component has been created in OpenEuler and its code is being imported from the upstream community. It will be released in OpenEuler.| +| Existing Version | This component has been released in the openEuler version | ## inkage -openEuler bigdata sig组 简介: https://gitee.com/openeuler/community/tree/master/sig/bigdata - -openEuler bigdata 月刊归档地址： +openEuler bigdata sigintroduction: https://gitee.com/openeuler/community/tree/master/sig/bigdata -邮件列表归档地址： +openEuler bigdata Monthly magazine archive address： -会议纪要地址： +Mailing List Archive Address： +Address of meeting minutes：[Address of meeting minutes](https://gitee.com/openeuler/bigdata/wikis/sig-bigdata%E4%BB%8B%E7%BB%8D) ## contact details -### [邮件列表](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) -### 微信公众号：openEuler-bigdata-sig +### [message list](https://mailweb.openeuler.org/hyperkitty/list/bigdata@openeuler.org/) +### WeChat official account：openEuler-bigdata-sig -# 路标 -- Gitee From c716ec541d1dd5c881817409aa7dbfa7f2a581e6 Mon Sep 17 00:00:00 2001 From: MacChen1 Date: Mon, 20 Mar 2023 11:07:25 +0000 Subject: [PATCH 97/99] update README.md. Signed-off-by: MacChen1 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 5da3b4c..bd62b7e 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ # bigdata -[English version introduction](readme-English.md) +[ _English version introduction_ ](readme-English.md) ## 愿景和职责 -- Gitee From 3873437603075cf8247757854900f25d96ca913f Mon Sep 17 00:00:00 2001 From: Hann <307510108@qq.com> Date: Tue, 21 Mar 2023 01:56:09 +0000 Subject: [PATCH 98/99] =?UTF-8?q?=E5=88=A0=E9=99=A4=E6=96=87=E4=BB=B6=20Do?= =?UTF-8?q?cs/=E5=B7=A5=E5=85=B7=E5=8C=85/.keep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\345\267\245\345\205\267\345\214\205/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 "Docs/\345\267\245\345\205\267\345\214\205/.keep" diff --git "a/Docs/\345\267\245\345\205\267\345\214\205/.keep" "b/Docs/\345\267\245\345\205\267\345\214\205/.keep" deleted file mode 100644 index e69de29..0000000 -- Gitee From a46a4102da1da81adc96bec0ff389a74f536d7bb Mon Sep 17 00:00:00 2001 From: Hann <307510108@qq.com> Date: Tue, 21 Mar 2023 01:56:25 +0000 Subject: [PATCH 99/99] =?UTF-8?q?=E5=88=A0=E9=99=A4=E6=96=87=E4=BB=B6=20Do?= =?UTF-8?q?cs/=E9=83=A8=E7=BD=B2=E6=8C=87=E5=8D=97/.keep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 "Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" diff --git "a/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" "b/Docs/\351\203\250\347\275\262\346\214\207\345\215\227/.keep" deleted file mode 100644 index e69de29..0000000 -- Gitee