diff --git "a/content/zh/post/DarkAthena/MPT\350\207\252\345\212\250\345\214\226\346\265\213\350\257\225\345\267\245\345\205\267\345\256\236\350\267\265.md" "b/content/zh/post/DarkAthena/MPT\350\207\252\345\212\250\345\214\226\346\265\213\350\257\225\345\267\245\345\205\267\345\256\236\350\267\265.md" new file mode 100644 index 0000000000000000000000000000000000000000..59fe0fca8616773b5e8e9692f8a41fd11d5dfcf1 --- /dev/null +++ "b/content/zh/post/DarkAthena/MPT\350\207\252\345\212\250\345\214\226\346\265\213\350\257\225\345\267\245\345\205\267\345\256\236\350\267\265.md" @@ -0,0 +1,144 @@ ++++ + +title = "MPT自动化测试工具实践" + +date = "2023-02-07" + +tags = ["MogDB"] + +archives = "2023-02" + +author = "DarkAthena" + +summary = "MPT自动化测试工具实践" + +img = "/zh/post/DarkAthena/title/img31.png" + +times = "10:20" + ++++ + +# 创建测试环境目录及用户 + +``` +mkdir /opt/mpt_test -r +useradd mpt_test +chown mpt_test /opt/mpt_test +``` + +# 下载mpt并解压 + +最新版本地址下载地址见wiki +http://wiki.enmotech.com:8090/pages/viewpage.action?pageId=29361193 + +``` +su - mpt_test +cd /opt/mpt_test +wget https://cdn-mogdb.enmotech.com/mpt/v1.3.0/mpt_x86_64_v1.3.0.zip +unzip mpt_x86_64_v1.3.0.zip +``` + +# 下载配置文件 + +http://wiki.enmotech.com:8090/pages/viewpage.action?pageId=29361193 + +``` +cd mpt_x86_64_v1.3.0 +# 下载测试配置文件至此目录 ，比如 MPT_TMPL_NORMAL_v1.1_20230109.xlsx +``` + +# 申请license ,会收到邮件，手动上传到服务器上 + +``` +./mpt_x86_64 --apply-license +cat > license.json +``` + +# 创建PTK安装配置文件 + +``` +exit ## 切换回root +cd /opt/mpt_test +cat > /opt/mpt_test/config.yaml + +global: + # # cluster name (required) + cluster_name: "mpt_test" + # # system user for running db + user: "mpt_test" + # # system user group, same as username if not given + # group: "omm" + # # base directory for install MogDB server, + # # if any of app_dir, data_dir, log_dir and tool_dir not config, + # # PTK will create corresponding directory under base_dir + base_dir: "/opt/mpt_test/mogdb" + # # default password ：Enmo@123 + db_password: "pTk6NDk0MjZiOGQ8PTxFPT8/QkZiR0dlOFo2bWd3a2pxb3BrQXdKTHpHNXJLUFVUckNHNDRoemg5SE05RDQ=" + +db_servers: + - host: "127.0.0.1" + # # database port + db_port: 26100 + db_conf: + checkpoint_segments: 64 + wal_keep_segments: 64 + +ctrl+d ## 保存文件 +``` + +# 安装PTK + +``` +curl --proto '=https' --tlsv1.2 -sSf https://cdn-mogdb.enmotech.com/ptk/install.sh | sh +ptk checkos -f /opt/mpt_test/config.yaml +``` + +# 安装数据库 (如果重新测试，则从这一步开始,主要是为了释放空间，以及避免测试用例编写遗漏了清理环境) + +``` +ptk cluster stop -n mpt_test ## 这里为了删库重测 +ptk uninstall -n mpt_test ## y y n --删除数据库但保留用户 +ptk install -f /opt/mpt_test/config.yaml --skip-create-user -y +ptk cluster install-plugin -n mpt_test +``` + +# 安装数据库兼容性提高组件 + +``` +su - mpt_test +gsql -r +create extension whale; +create extension orafce; +\q + +cd /opt/mpt_test + +wget https://gitee.com/enmotech/compat-tools/repository/archive/master.zip +unzip master.zip +cd compat-tools-master +gsql -f runMe.sql +``` + +# 执行测试（对于测试时间长的，建议nohup执行） + +``` +cd /opt/mpt_test/mpt_x86_64_v1.3.0 +nohup ./mpt_x86_64 -H localhost -P $PGPORT -U $PGUSER -c MPT_TMPL_NORMAL_v1.1_20230109.xlsx -r MPT_Report-$(date +%Y%m%d-%H%M%S).docx --cmd-spliter=xxxxxxx -T W & + +ctrl+c +``` + +# 观察日志 + +``` +tail -f nohup.out +``` + +# 注意 + +## 测试需要预留足够的磁盘空间， 其中pg_log可能会占很大的空间，如果空间紧张,需要手动清理 + +``` +# 查看目录大小 +du -h --max-depth=1 /opt/mpt_test/mogdb +``` diff --git "a/content/zh/post/DarkAthena/MogDB\357\274\2103.0.3.6\347\211\210\346\234\254\357\274\211\346\265\213\350\257\225oracle\350\257\255\346\263\225\347\232\204DBLINK\345\212\237\350\203\275\346\223\215\344\275\234\346\255\245\351\252\244.md" "b/content/zh/post/DarkAthena/MogDB\357\274\2103.0.3.6\347\211\210\346\234\254\357\274\211\346\265\213\350\257\225oracle\350\257\255\346\263\225\347\232\204DBLINK\345\212\237\350\203\275\346\223\215\344\275\234\346\255\245\351\252\244.md" new file mode 100644 index 0000000000000000000000000000000000000000..280f47cc7bf658ab6a97862cf91961ce91d47910 --- /dev/null +++ "b/content/zh/post/DarkAthena/MogDB\357\274\2103.0.3.6\347\211\210\346\234\254\357\274\211\346\265\213\350\257\225oracle\350\257\255\346\263\225\347\232\204DBLINK\345\212\237\350\203\275\346\223\215\344\275\234\346\255\245\351\252\244.md" @@ -0,0 +1,114 @@ ++++ + +title = "MogDB（3.0.3.6版本）测试oracle语法的DBLINK功能操作步骤" + +date = "2023-02-07" + +tags = ["MogDB"] + +archives = "2023-02" + +author = "DarkAthena" + +summary = "MogDB（3.0.3.6版本）测试oracle语法的DBLINK功能操作步骤" + +img = "/zh/post/DarkAthena/title/img31.png" + +times = "10:20" + ++++ + +# 安装POC版本MogDB + +``` +ptk install -f config.yaml -p https://cdn-mogdb.enmotech.com/mogdb-media/3.0.3.6/MogDB-3.0.3.6-CentOS-x86_64.tar.gz +``` + +# 添加插件 + +``` +ptk cluster install-plugin -n xxx -p https://cdn-mogdb.enmotech.com/mogdb-media/3.0.3.6/Plugins-3.0.3-CentOS-x86_64.tar.gz --skip-check-version +``` + +# 切换到数据库用户 + +``` +su - omm +``` + +# 手动下载oracle_fdw + +oracle_fdw没有打到插件包里去，需要单独下载 + +``` +wget https://cdn-mogdb.enmotech.com/mogdb-media/3.0.1.4/oracle_fdw_CentOS_x86.tar.gz +``` + +# 解压 并复制到对应目录 + +``` +tar -xvf oracle_fdw_CentOS_x86.tar.gz +cd oracle_fdw_CentOS_x86 +cp *.sql $GAUSSHOME/app/share/postgresql/extension/ +cp *.control $GAUSSHOME/app/share/postgresql/extension/ +cp *.so $GAUSSHOME/lib/postgresql +``` + +# 下载oracle客户端并配置环境变量 + +``` +wget https://download.oracle.com/otn_software/linux/instantclient/218000/instantclient-basic-linux.x64-21.8.0.0.0dbru.zip +unzip instantclient-basic-linux.x64-21.8.0.0.0dbru.zip +vi ~/.bash_profile +export LD_LIBRARY_PATH=/home/omm/instantclient_21_7:$LD_LIBRARY_PATH +export PATH=/home/omm/instantclient_21_7:$PATH +source ~/.bash_profile +``` + +# 重启数据库 + +``` +gs_ctl restart +``` + +# 连接数据库并创建插件 + +``` +gsql -r +create extension oracle_fdw; +``` + +# 创建服务器 + +``` +create server ora_sv foreign data wrapper oracle_fdw options(dbserver 'xxx.xxx.xxx.xxx:1521/pdb1'); +``` + +# 给服务器设置用户名和密码 + +``` +create user mapping for mogdb的用户名 server ora_sv options(user 'oracle的用户名',password 'oracle的密码'); +``` + +可能会报错 + +> ERROR: No key file usermapping.key.cipher +> Please create usermapping.key.cipher file with gs_guc and gs_ssh, such as :gs_ssh -c “gs_guc generate -S XXX -D $GAUSSHOME/bin -o usermapping” + +先去操作系统输入以下命令，并输入密码 + +``` +gs_guc generate -D $GAUSSHOME/bin -o usermapping +``` + +然后回来创建 + +``` +create user mapping for mogdb的用户名 server ora_sv options(user 'oracle的用户名',password 'oracle的密码'); +``` + +# 测试 + +``` +select * from scott.emp@ora_sv +``` diff --git a/content/zh/post/DarkAthena/images/01fa1633e6829a9adafc9eaf4f3a502c.png b/content/zh/post/DarkAthena/images/01fa1633e6829a9adafc9eaf4f3a502c.png new file mode 100644 index 0000000000000000000000000000000000000000..5b4ba30b87df1419fc61efdf9249464a1e6f765d Binary files /dev/null and b/content/zh/post/DarkAthena/images/01fa1633e6829a9adafc9eaf4f3a502c.png differ diff --git a/content/zh/post/DarkAthena/images/035778d7d2f842f0e43b07f06be50dcc.png b/content/zh/post/DarkAthena/images/035778d7d2f842f0e43b07f06be50dcc.png new file mode 100644 index 0000000000000000000000000000000000000000..700d92fbe1ce554e78b769a226e3a06008e4c33f Binary files /dev/null and b/content/zh/post/DarkAthena/images/035778d7d2f842f0e43b07f06be50dcc.png differ diff --git a/content/zh/post/DarkAthena/images/042e5465f559ceac88bdaf9fec73c530.png b/content/zh/post/DarkAthena/images/042e5465f559ceac88bdaf9fec73c530.png new file mode 100644 index 0000000000000000000000000000000000000000..8a578799bed7d5a2ace3501a70b3542195cc9a32 Binary files /dev/null and b/content/zh/post/DarkAthena/images/042e5465f559ceac88bdaf9fec73c530.png differ diff --git a/content/zh/post/DarkAthena/images/20230201-49d75e33-2a1e-4eef-b312-aae978af089a.png b/content/zh/post/DarkAthena/images/20230201-49d75e33-2a1e-4eef-b312-aae978af089a.png new file mode 100644 index 0000000000000000000000000000000000000000..3b4e82c5e249f42869f641bb74fea5f5828e1ff9 Binary files /dev/null and b/content/zh/post/DarkAthena/images/20230201-49d75e33-2a1e-4eef-b312-aae978af089a.png differ diff --git a/content/zh/post/DarkAthena/images/20230201-858fc5aa-cd0a-4e31-9141-1c87fb24b9c8.png b/content/zh/post/DarkAthena/images/20230201-858fc5aa-cd0a-4e31-9141-1c87fb24b9c8.png new file mode 100644 index 0000000000000000000000000000000000000000..d62aff052e096962f19d59f6b667fa0c33a6d594 Binary files /dev/null and b/content/zh/post/DarkAthena/images/20230201-858fc5aa-cd0a-4e31-9141-1c87fb24b9c8.png differ diff --git a/content/zh/post/DarkAthena/images/20744cd1b2784c25b9f1d08e2a08b26b.png b/content/zh/post/DarkAthena/images/20744cd1b2784c25b9f1d08e2a08b26b.png new file mode 100644 index 0000000000000000000000000000000000000000..93eff530895579c6c04746e045c5fd8dc70b5151 Binary files /dev/null and b/content/zh/post/DarkAthena/images/20744cd1b2784c25b9f1d08e2a08b26b.png differ diff --git a/content/zh/post/DarkAthena/images/34ed0aa9c0765d744f1ae4b2e23bbcc9.png b/content/zh/post/DarkAthena/images/34ed0aa9c0765d744f1ae4b2e23bbcc9.png new file mode 100644 index 0000000000000000000000000000000000000000..eee1ecbeff4cdbb77d202e642ea6a544c86ef81a Binary files /dev/null and b/content/zh/post/DarkAthena/images/34ed0aa9c0765d744f1ae4b2e23bbcc9.png differ diff --git a/content/zh/post/DarkAthena/images/3c5edcfd66c542453e6284b38773b911.png b/content/zh/post/DarkAthena/images/3c5edcfd66c542453e6284b38773b911.png new file mode 100644 index 0000000000000000000000000000000000000000..9839062800e067f307c326290f049c8e6277bba5 Binary files /dev/null and b/content/zh/post/DarkAthena/images/3c5edcfd66c542453e6284b38773b911.png differ diff --git a/content/zh/post/DarkAthena/images/43f831521c831e975a7ef7258e3f9625.png b/content/zh/post/DarkAthena/images/43f831521c831e975a7ef7258e3f9625.png new file mode 100644 index 0000000000000000000000000000000000000000..b240534e3d5839c046c2ba1c54be95dffa3c3b1d Binary files /dev/null and b/content/zh/post/DarkAthena/images/43f831521c831e975a7ef7258e3f9625.png differ diff --git a/content/zh/post/DarkAthena/images/5e59536d4830366906e336284f44007c.png b/content/zh/post/DarkAthena/images/5e59536d4830366906e336284f44007c.png new file mode 100644 index 0000000000000000000000000000000000000000..b2433f1bcf35fd6e33b8c0761036c75ddc290e4d Binary files /dev/null and b/content/zh/post/DarkAthena/images/5e59536d4830366906e336284f44007c.png differ diff --git a/content/zh/post/DarkAthena/images/74a4eafe28730bc309d5685e8b09b93c.png b/content/zh/post/DarkAthena/images/74a4eafe28730bc309d5685e8b09b93c.png new file mode 100644 index 0000000000000000000000000000000000000000..0464f15c50c09ab3501a6ee96b6b39f9a3dfd735 Binary files /dev/null and b/content/zh/post/DarkAthena/images/74a4eafe28730bc309d5685e8b09b93c.png differ diff --git a/content/zh/post/DarkAthena/images/7a908b5bc6c6a9fc2aa805f4ef6c8544.png b/content/zh/post/DarkAthena/images/7a908b5bc6c6a9fc2aa805f4ef6c8544.png new file mode 100644 index 0000000000000000000000000000000000000000..760cf6dde434f6422720cf3bb5ca191912bdbfba Binary files /dev/null and b/content/zh/post/DarkAthena/images/7a908b5bc6c6a9fc2aa805f4ef6c8544.png differ diff --git a/content/zh/post/DarkAthena/images/88a31fd96492dcec9b34f627b99d8052.png b/content/zh/post/DarkAthena/images/88a31fd96492dcec9b34f627b99d8052.png new file mode 100644 index 0000000000000000000000000000000000000000..628674007c7fc39c1a247d3c24d4ad157e2c3f0d Binary files /dev/null and b/content/zh/post/DarkAthena/images/88a31fd96492dcec9b34f627b99d8052.png differ diff --git a/content/zh/post/DarkAthena/images/961af150a73bdb6abedefb408a16d581.png b/content/zh/post/DarkAthena/images/961af150a73bdb6abedefb408a16d581.png new file mode 100644 index 0000000000000000000000000000000000000000..a6668dc636700fa8be2c269da905a55788e9dda0 Binary files /dev/null and b/content/zh/post/DarkAthena/images/961af150a73bdb6abedefb408a16d581.png differ diff --git a/content/zh/post/DarkAthena/images/9bff3ede038f944c36cf0d1b3c0ff1d9.png b/content/zh/post/DarkAthena/images/9bff3ede038f944c36cf0d1b3c0ff1d9.png new file mode 100644 index 0000000000000000000000000000000000000000..ae18407df3ce65b67c94422c9f2a576a658e0851 Binary files /dev/null and b/content/zh/post/DarkAthena/images/9bff3ede038f944c36cf0d1b3c0ff1d9.png differ diff --git a/content/zh/post/DarkAthena/images/a26b52b8cd3094bdc59d405302f7b8d8.png b/content/zh/post/DarkAthena/images/a26b52b8cd3094bdc59d405302f7b8d8.png new file mode 100644 index 0000000000000000000000000000000000000000..1c47ba7ce75b738024524cc6d33d474871c9a74f Binary files /dev/null and b/content/zh/post/DarkAthena/images/a26b52b8cd3094bdc59d405302f7b8d8.png differ diff --git a/content/zh/post/DarkAthena/images/abcef55aaba11d0adcdcc83fc632da1a.png b/content/zh/post/DarkAthena/images/abcef55aaba11d0adcdcc83fc632da1a.png new file mode 100644 index 0000000000000000000000000000000000000000..12d5c2a7bd4041bc5dfd7b7040d6a6ce430eaf95 Binary files /dev/null and b/content/zh/post/DarkAthena/images/abcef55aaba11d0adcdcc83fc632da1a.png differ diff --git a/content/zh/post/DarkAthena/images/b82d6a1a629cf548e5cbfb024b439757.png b/content/zh/post/DarkAthena/images/b82d6a1a629cf548e5cbfb024b439757.png new file mode 100644 index 0000000000000000000000000000000000000000..28fb38649d53cfc980b9d8895fe5c07dd41304d5 Binary files /dev/null and b/content/zh/post/DarkAthena/images/b82d6a1a629cf548e5cbfb024b439757.png differ diff --git a/content/zh/post/DarkAthena/images/ca2a4405d05cce042f494a37b9039c97.png b/content/zh/post/DarkAthena/images/ca2a4405d05cce042f494a37b9039c97.png new file mode 100644 index 0000000000000000000000000000000000000000..b3e5fd7bb22f772f8e9cd60e1b3fd303680006e0 Binary files /dev/null and b/content/zh/post/DarkAthena/images/ca2a4405d05cce042f494a37b9039c97.png differ diff --git a/content/zh/post/DarkAthena/images/cc3f6b4b0fccbb1452cfcf894bef1115.png b/content/zh/post/DarkAthena/images/cc3f6b4b0fccbb1452cfcf894bef1115.png new file mode 100644 index 0000000000000000000000000000000000000000..2066a7ae7608809092260b6d215b6684c934474c Binary files /dev/null and b/content/zh/post/DarkAthena/images/cc3f6b4b0fccbb1452cfcf894bef1115.png differ diff --git a/content/zh/post/DarkAthena/images/f06e8c49015bfe49ae1fd3311235c5b6.png b/content/zh/post/DarkAthena/images/f06e8c49015bfe49ae1fd3311235c5b6.png new file mode 100644 index 0000000000000000000000000000000000000000..099deb7d835cc2a6b41431fd1d1c84d72a1dbab8 Binary files /dev/null and b/content/zh/post/DarkAthena/images/f06e8c49015bfe49ae1fd3311235c5b6.png differ diff --git a/content/zh/post/DarkAthena/images/f7c379f7ed4611bfee7273041cea2113.png b/content/zh/post/DarkAthena/images/f7c379f7ed4611bfee7273041cea2113.png new file mode 100644 index 0000000000000000000000000000000000000000..879fccd317a4d8e0d52d8a85286db5575ee5aac3 Binary files /dev/null and b/content/zh/post/DarkAthena/images/f7c379f7ed4611bfee7273041cea2113.png differ diff --git "a/content/zh/post/DarkAthena/\343\200\220openGauss\343\200\221\345\234\250WPS\350\241\250\346\240\274\351\207\214\345\210\266\344\275\234\350\277\236\346\216\245\345\210\260openGauss\347\232\204\345\256\236\346\227\266\345\210\267\346\226\260\346\212\245\350\241\250.md" "b/content/zh/post/DarkAthena/\343\200\220openGauss\343\200\221\345\234\250WPS\350\241\250\346\240\274\351\207\214\345\210\266\344\275\234\350\277\236\346\216\245\345\210\260openGauss\347\232\204\345\256\236\346\227\266\345\210\267\346\226\260\346\212\245\350\241\250.md" new file mode 100644 index 0000000000000000000000000000000000000000..8d601b2c08a068abb72cc87f4c5c7f9a1ea97095 --- /dev/null +++ "b/content/zh/post/DarkAthena/\343\200\220openGauss\343\200\221\345\234\250WPS\350\241\250\346\240\274\351\207\214\345\210\266\344\275\234\350\277\236\346\216\245\345\210\260openGauss\347\232\204\345\256\236\346\227\266\345\210\267\346\226\260\346\212\245\350\241\250.md" @@ -0,0 +1,107 @@ ++++ + +title = "【openGauss】在WPS表格里制作连接到openGauss的实时刷新报表" + +date = "2022-12-06" + +tags = ["openGauss"] + +archives = "2022-12" + +author = "DarkAthena" + +summary = "【openGauss】在WPS表格里制作连接到openGauss的实时刷新报表" + +img = "/zh/post/DarkAthena/title/img.png" + +times = "10:20" + ++++ + +## 前言 + +其实我的数据库启蒙，是在一家甲方公司。 +当时一进这家公司，就见到了通过连接数据库自动刷新的excel表。当时学会了这招就一发不可收拾，制作出各种自动刷新的报表。 +想象一下，有些高管不喜欢打开各种复杂的业务系统或者报表系统，一上班就直接打开桌面上的可刷新表格文件，就能看到昨日的报表数据以及今日的实时数据。 +当年智能手机还未普及，没有移动端报表,每晚的值班经理需要查询当日数据编辑短信发送给高管，也是在电脑上用EXCEL直接刷出数据，而且提前在EXCEL里写好了公式，拼接成了短信文本内容，复制粘贴到飞信就把业绩短信发出去了，多少年来一直都是这么用的，只是后来改成了粘贴到微信发送。 + +在当时，这也算是极低成本的半自动化了，好不惬意！ + +当时连接的主要是Oracle数据库，现在突然想起，是不是我们的国产数据库也可以这样连接呢？ + +## 原理 + +其实原理很简单，就是在windows的odbc数据源中配置好对应的数据库连接，然后在excel或者wps表格中选择导入数据/odbc数据源，选择需要的表及字段，或者直接写个sql查询也行，就可以把数据返回到表格中。当数据库中的数据发生变化时，只需要在表格中点击刷新或者或者设置刷新频率即可 + +## 步骤 + +### 安装ODBC驱动 + +[windows端ODBC驱动下载（支持sha256） https://obs.myhuaweicloud.com/dws/download/dws_8.1.x_odbc_driver_for_windows.zip](https://obs.myhuaweicloud.com/dws/download/dws_8.1.x_odbc_driver_for_windows.zip) + +下载好后，解压，根据使用的wps或msoffice是64位还是32位版本，执行对应的安装程序（目测大多数人安装的都是32位版本），一路“next”就行了。 +不知道自己安装的是32还是64位的，可以把程序打开，到任务管理器里找到对应的程序，会有显示 +![image-1668874050903](./images/3c5edcfd66c542453e6284b38773b911.png) + +### 添加ODBC数据源 + +1. 依次打开 + 控制面板-管理工具-ODBC Data Sources (32-bit) + ![image-1668874143580](./images/9bff3ede038f944c36cf0d1b3c0ff1d9.png) +2. 点击右边的添加按钮，找到 PostgreSQL Unicode,双击 + ![image-1668874242003](./images/01fa1633e6829a9adafc9eaf4f3a502c.png) +3. 然后填写连接信息，并点击Test + ![image-1668874405597](./images/f06e8c49015bfe49ae1fd3311235c5b6.png) +4. 提示连接成功，确定，点击SAVE保存，会弹出安全提示，意思就是连接信息会保存在系统注册表里，不安全，问是否继续，点"是"（本篇不讨论安全问题,请自行斟酌） + ![image-1668874472622](./images/a26b52b8cd3094bdc59d405302f7b8d8.png) +5. 然后就可以看到数据源里多了一个刚刚新建的数据源 + ![image-1668874765760](./images/43f831521c831e975a7ef7258e3f9625.png) + +### 在WPS中引用数据源 + +1. 新建一个空白表格 +2. 点击 数据 - 导入数据，选择ODBC DSN + ![image-1668874917682](./images/cc3f6b4b0fccbb1452cfcf894bef1115.png) +3. 选择刚刚新建的数据源，确定，下一步 + ![image-1668874959657](./images/035778d7d2f842f0e43b07f06be50dcc.png) +4. 选择需要的表，并将需要的字段移到右边，点击下一步 + ![image-1668875300934](./images/34ed0aa9c0765d744f1ae4b2e23bbcc9.png) +5. 选择手工输入连接语句，点击下一步 + ![image-1668875424131](./images/f7c379f7ed4611bfee7273041cea2113.png) +6. 预览没有问题，点击完成 + ![image-1668875476891](./images/7a908b5bc6c6a9fc2aa805f4ef6c8544.png) +7. 选择数据需要存放的开始单元格，点击确定 + ![image-1668875523076](./images/88a31fd96492dcec9b34f627b99d8052.png) +8. 数据就放回到表格中了 + ![image-1668875560923](./images/5e59536d4830366906e336284f44007c.png) + +### 验证数据刷新 + +1. 在数据库中执行sql修改数据，比如让所有人的工资翻倍 + +``` +update scott.emp set sal=sal*2 where 1=1; +``` + +![image-1668875774103](./images/b82d6a1a629cf548e5cbfb024b439757.png) +[windows端gsql下载 （支持sha256）https://obs.myhuaweicloud.com/dws/download/dws_8.1.x_gsql_for_windows.zip](https://obs.myhuaweicloud.com/dws/download/dws_8.1.x_gsql_for_windows.zip) +\2. 在表格数据区域点击鼠标右键，刷新数据 +![image-1668875838278](./images/74a4eafe28730bc309d5685e8b09b93c.png) +\3. 可以看到数据都变更了 +![image-1668875866935](./images/abcef55aaba11d0adcdcc83fc632da1a.png) + +### 配置数据自动刷新 + +1. 在表格数据区域点击鼠标右键，点击数据区域属性 + ![image-1668875958517](./images/ca2a4405d05cce042f494a37b9039c97.png) +2. 设置刷新频率，并勾选打开文件时刷新数据 + ![image-1668876065494](./images/20744cd1b2784c25b9f1d08e2a08b26b.png) + +## 总结 + +本篇只举了个最简单的例子，实际上，新建多个查询，结合表格的公式，并调整表格的样式，是可以制作出相当精美的报表的，而且只用做一次，以后一直都能用，数据都是可以实时从数据库中获取的。 +另外，以上操作在MSOFFICE中的EXCEL也是类似的，只是EXCEL还有个MSQUERY的程序，可以可视化编辑多表的关联关系，有兴趣的可以自己尝试一下。 + +> - **本文作者：** [DarkAthena](https://www.darkathena.top/) +> - **本文链接：** [https://www.darkathena.top/archives/opengauss-wps-excel-realtime-report-sha256](https://www.darkathena.top/archives/opengauss-wps-excel-realtime-report-sha256) +> - **版权声明：** 本博客所有文章除特别声明外，均采用[CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) 许可协议。转载请注明出处！ diff --git "a/content/zh/post/DarkAthena/\343\200\220openGauss\343\200\221\346\212\212\345\272\224\347\224\250\345\274\200\345\217\221\344\270\255\347\232\204\350\256\276\347\275\256\345\256\242\346\210\267\347\253\257\345\255\227\347\254\246\347\274\226\347\240\201\345\276\200\347\273\206\344\272\206\350\257\264.md" "b/content/zh/post/DarkAthena/\343\200\220openGauss\343\200\221\346\212\212\345\272\224\347\224\250\345\274\200\345\217\221\344\270\255\347\232\204\350\256\276\347\275\256\345\256\242\346\210\267\347\253\257\345\255\227\347\254\246\347\274\226\347\240\201\345\276\200\347\273\206\344\272\206\350\257\264.md" new file mode 100644 index 0000000000000000000000000000000000000000..a064cfb6a8364f55805d25a264c64e2db457e3bf --- /dev/null +++ "b/content/zh/post/DarkAthena/\343\200\220openGauss\343\200\221\346\212\212\345\272\224\347\224\250\345\274\200\345\217\221\344\270\255\347\232\204\350\256\276\347\275\256\345\256\242\346\210\267\347\253\257\345\255\227\347\254\246\347\274\226\347\240\201\345\276\200\347\273\206\344\272\206\350\257\264.md" @@ -0,0 +1,141 @@ ++++ + +title = "【openGauss】把应用开发中的设置客户端字符编码往细了说" + +date = "2023-02-07" + +tags = ["openGauss"] + +archives = "2023-02" + +author = "DarkAthena" + +summary = "【openGauss】把应用开发中的设置客户端字符编码往细了说" + +img = "/zh/post/DarkAthena/title/img31.png" + +times = "10:20" + ++++ + +## 前言 + +早前写过两篇有关Oracle字符集的文章 +[【ORACLE】谈一谈Oracle数据库使用的字符集,不仅仅是乱码](https://www.darkathena.top/archives/about-oracle-charset) +[【ORACLE】谈一谈NVARCHAR2、NCHAR、NCLOB等数据类型和国家字符集](https://www.darkathena.top/archives/about-nvarchar2-and-national-charset) +基本说明了 ”**数据字符编码**“、”**客户端字符编码**“、”**数据库字符编码**“三者的关系，这些关系，对于openGauss/MogDB/postgresql其实是一样的，即”**数据字符编码**“和”**客户端字符编码**“应保持一致，且对应的字符集为”**数据库字符集**“的”**子集**“。但是实际应用开发中，”**客户端字符编码**“在没有进行主动设定时，往往会受各种因素干扰，比如各种环境变量和数据库参数。本文就针对各种客户端，通过文档加实验来对客户端字符编码应该如何设置来做个分析说明（注：本文的测试环境操作系统为centos7.9） + +## 设置客户端字符编码的各种方式 + +在openGauss中，可以通过很多方式来设置客户端字符集，但实际上最终影响的都是在数据库连接中的 **client_encoding**，该值可以通过执行 **show client_encoding;** 来进行查看。 + +### 操作系统环境变量 + +1. LANG + +``` +export LANG=zh_CN.UTF-8 +``` + +1. LC_CTYPE + +``` +export LC_CTYPE=zh_CN.UTF-8 +``` + +1. PGCLIENTENCODING + +``` +export PGCLIENTENCODING=UTF-8 +``` + +### 数据库级参数 + +1. alter databse dbname set client_encoding=‘UTF-8’; + (注意：alter system set client_encoding=‘UTF-8’; 无法执行) + +### 数据库会话中的参数(会话级) + +1. set client_encoding=‘UTF-8’; +2. set client_encoding to ‘UTF-8’; +3. set names ‘UTF-8’; +4. alter session set client_encoding = ‘UTF8’; + +### gsql元命令(会话级) + +1. \encoding UTF-8 + +### 驱动连接参数 + +1. libpq + +``` +conninfo = "dbname=postgres port=26100 host='192.168.56.117' client_encoding='UTF-8' application_name=test connect_timeout=5 sslmode=disable user='xxx' password='xxx' "; +``` + +1. jdbc + +``` +urlString = "jdbc:opengauss://192.168.56.117:26100/postgres?batchMode=off&allowEncodingChanges=true&characterEncoding=GBK"; +``` + +1. psycopg2 + +``` +conn = psycopg2.connect(database="postgres", user="xxx", password="xxx", host="192.168.56.117", port="26100",client_encoding="GBK") +``` + +### postgresql.conf + +``` +client_encoding= 'UTF8' +``` + +(注：由于设置postgresql.conf的方式在我测试的各种情况下均未产生影响，为节省篇幅，下文不再列出这种方式) + +## 各客户端中取值的优先级和默认值 + +由于可以设置的方式太多,那么必然会出现各个设置互相冲突时，以哪个为准的问题。 +首先明确一点，当前会话执行的客户端字符编码，一定是可以通过 +**show client_encoding;** 这个命令查看的，也就是说，无论怎么设置，最终都是为了影响会话里的 **client_encoding** 参数。接着，我们可以进行各种组合尝试，来确认这个参数的默认值及获取来源 +![image](./images/961af150a73bdb6abedefb408a16d581.png) + +根据以上测试结果，可以得到如下几个结论 +对于gsql + +1. 不论数据库字符编码为何值，其client_encoding会从PGCLIENTENCODING获取 +2. 当PGCLIENTENCODING没有设置时，会从环境变量LC_CTYPE获取 +3. 当LC_CTYPE没有设置时，会从环境变量LC_ALL获取 +4. 当LC_ALL没有设置时，会从环境变量LANG获取 +5. 当LANG没有设置时（unset LANG），默认为sql_ascii + +![image.png](./images/20230201-49d75e33-2a1e-4eef-b312-aae978af089a.png) + +对于libpq + +1. 不论LANG为何值，其client_encoding会从PGCLIENTENCODING获取 +2. 当PGCLIENTENCODING没有设置时，会从database的client_encoding获取(select * from pg_catalog.pg_db_role_setting ) +3. 当database的client_encoding没有设置时，默认为数据库建库时的字符编码（select getdatabaseencoding();） + +![image.png](./images/20230201-858fc5aa-cd0a-4e31-9141-1c87fb24b9c8.png) + +简单来说，就是**libpq**不认LANG、LC_CTYPE，**gsql**不认alter database和数据库字符编码。在不修改应用程序代码时，想指定客户端字符编码，最佳方式为设置PGCLIENTENCODING，因为这样才能保证两者的表现一致。 + +以上同样的测试，我用**psycopg2**和**jdbc**也测了一遍，和**libpq**的表现是完全一致的，也就是说，设定PGCLIENTENCODING对各种程序开发是最通用的设定客户端字符集编码的方案。（单一程序处理多种字符编码的情况本文暂不考虑） + +## 其他 + +附上我用libpq测试[之前文章中的用例](https://www.darkathena.top/archives/about-oracle-charset)结果，可以发现能插入的结果和ORACLE表现是完全一致的，有区别的是，ORACLE中字符编码错了可能也能插入，而MogDB/openGauss/postgresql中则会报错，这个点是好是坏，也只能见仁见智，区分场景了。但我个人偏向于应尽量避免让错误的数据进入数据库 +![image-1675252053480](./images/042e5465f559ceac88bdaf9fec73c530.png) + +本文结论为参考了各种官方文档，加上自己的测试得到的结果，如有不对，请联系我指出。 + +## 参考链接 + +https://docs.mogdb.io/zh/mogdb/v3.0/character-set-support +https://docs.mogdb.io/zh/mogdb/v3.0/1-gsql +http://postgres.cn/docs/13/libpq-connect.html + +- **本文作者：** [DarkAthena](https://www.darkathena.top/) +- **本文链接：** https://www.darkathena.top/archives/about-opengauss-client-encoding +- **版权声明：** 本博客所有文章除特别声明外，均采用[CC BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/) 许可协议。转载请注明出处！ diff --git a/content/zh/post/Frank/images/HIS6hdLiNFe6czt6hwP2pi_NnvwiFnaRrKxs1DhPu5s.png b/content/zh/post/Frank/images/HIS6hdLiNFe6czt6hwP2pi_NnvwiFnaRrKxs1DhPu5s.png new file mode 100644 index 0000000000000000000000000000000000000000..e09d3e87ce50c71bedfdd60d265cde33d9cf33c1 Binary files /dev/null and b/content/zh/post/Frank/images/HIS6hdLiNFe6czt6hwP2pi_NnvwiFnaRrKxs1DhPu5s.png differ diff --git a/content/zh/post/Frank/images/JDztFoVEkBL515qAiQljPxPByxTa4pjIe15g50Ciyl4.png b/content/zh/post/Frank/images/JDztFoVEkBL515qAiQljPxPByxTa4pjIe15g50Ciyl4.png new file mode 100644 index 0000000000000000000000000000000000000000..6809266dc95ea47b98580bfe93b40f6695a24747 Binary files /dev/null and b/content/zh/post/Frank/images/JDztFoVEkBL515qAiQljPxPByxTa4pjIe15g50Ciyl4.png differ diff --git a/content/zh/post/Frank/images/K6wZLF-stZaD6c4LzgP5UMEjo0qxjuOw-Irj42Q33fw.png b/content/zh/post/Frank/images/K6wZLF-stZaD6c4LzgP5UMEjo0qxjuOw-Irj42Q33fw.png new file mode 100644 index 0000000000000000000000000000000000000000..21d0edd387699dd71a4a894a7cf00bc4b2b41577 Binary files /dev/null and b/content/zh/post/Frank/images/K6wZLF-stZaD6c4LzgP5UMEjo0qxjuOw-Irj42Q33fw.png differ diff --git a/content/zh/post/Frank/images/OAlETvVs-VRdXpoBOenwSil6WkhILHKdzb1OEkU1lYY.png b/content/zh/post/Frank/images/OAlETvVs-VRdXpoBOenwSil6WkhILHKdzb1OEkU1lYY.png new file mode 100644 index 0000000000000000000000000000000000000000..11573df1b8cc5d81b526d984d043053c3469a2cd Binary files /dev/null and b/content/zh/post/Frank/images/OAlETvVs-VRdXpoBOenwSil6WkhILHKdzb1OEkU1lYY.png differ diff --git a/content/zh/post/Frank/images/PNcP_TtELNoc1JnjdxdJFZJAd5FlK94zcMgv63wDs-w.webp b/content/zh/post/Frank/images/PNcP_TtELNoc1JnjdxdJFZJAd5FlK94zcMgv63wDs-w.webp new file mode 100644 index 0000000000000000000000000000000000000000..6641239a402a2365254aaaf4632c7fb548fa6587 Binary files /dev/null and b/content/zh/post/Frank/images/PNcP_TtELNoc1JnjdxdJFZJAd5FlK94zcMgv63wDs-w.webp differ diff --git a/content/zh/post/Frank/images/Rr8WMD2FqdxOic2kFbxXJic35fdB_YAqnmerRQwqQG4.png b/content/zh/post/Frank/images/Rr8WMD2FqdxOic2kFbxXJic35fdB_YAqnmerRQwqQG4.png new file mode 100644 index 0000000000000000000000000000000000000000..d48c78c5a9408a80d4c50d8148e65c01b67d271e Binary files /dev/null and b/content/zh/post/Frank/images/Rr8WMD2FqdxOic2kFbxXJic35fdB_YAqnmerRQwqQG4.png differ diff --git a/content/zh/post/Frank/images/T311n-kMrG-vp6DcsMlDiHUBfuC6BfNcHDMMYjzs1No.png b/content/zh/post/Frank/images/T311n-kMrG-vp6DcsMlDiHUBfuC6BfNcHDMMYjzs1No.png new file mode 100644 index 0000000000000000000000000000000000000000..b25b528cf83df21498005703e1794991d72aa1c0 Binary files /dev/null and b/content/zh/post/Frank/images/T311n-kMrG-vp6DcsMlDiHUBfuC6BfNcHDMMYjzs1No.png differ diff --git a/content/zh/post/Frank/images/WH2_Mzd71Rk3VVjf1TECjxhpzKx-tAku9kfp3MX1aS0.png b/content/zh/post/Frank/images/WH2_Mzd71Rk3VVjf1TECjxhpzKx-tAku9kfp3MX1aS0.png new file mode 100644 index 0000000000000000000000000000000000000000..0ebe9a1e6b1ebc0ff6ffa5b00a381349d366e1bd Binary files /dev/null and b/content/zh/post/Frank/images/WH2_Mzd71Rk3VVjf1TECjxhpzKx-tAku9kfp3MX1aS0.png differ diff --git a/content/zh/post/Frank/images/ZvFmq2I9sxqe8M4iAg23tiFMEXKRCEMGAtOy8DUsLaQ.png b/content/zh/post/Frank/images/ZvFmq2I9sxqe8M4iAg23tiFMEXKRCEMGAtOy8DUsLaQ.png new file mode 100644 index 0000000000000000000000000000000000000000..93c57effdc39f78ef9870eb7afe6b0a4d3daa633 Binary files /dev/null and b/content/zh/post/Frank/images/ZvFmq2I9sxqe8M4iAg23tiFMEXKRCEMGAtOy8DUsLaQ.png differ diff --git a/content/zh/post/Frank/images/aNZdKt9BgaGiInzMuH3jNar-18Ca0fCHGN0vtksiH3w.png b/content/zh/post/Frank/images/aNZdKt9BgaGiInzMuH3jNar-18Ca0fCHGN0vtksiH3w.png new file mode 100644 index 0000000000000000000000000000000000000000..1bf61b7fb8142e7397f1b01a405ea6a1dd33668d Binary files /dev/null and b/content/zh/post/Frank/images/aNZdKt9BgaGiInzMuH3jNar-18Ca0fCHGN0vtksiH3w.png differ diff --git a/content/zh/post/Frank/images/aebKG7cndlMYcoRcq7qs5m-GYbhGj74zOR2NeJKh4GI.png b/content/zh/post/Frank/images/aebKG7cndlMYcoRcq7qs5m-GYbhGj74zOR2NeJKh4GI.png new file mode 100644 index 0000000000000000000000000000000000000000..d238d7109e309e9b70efa0802faf747899560c57 Binary files /dev/null and b/content/zh/post/Frank/images/aebKG7cndlMYcoRcq7qs5m-GYbhGj74zOR2NeJKh4GI.png differ diff --git a/content/zh/post/Frank/images/lWgH2foPpaW6bs4tZ71JzhLuKib1-kFxJhaWd3RwyQk.png b/content/zh/post/Frank/images/lWgH2foPpaW6bs4tZ71JzhLuKib1-kFxJhaWd3RwyQk.png new file mode 100644 index 0000000000000000000000000000000000000000..915ec38ba9bae0f1d12ccbedc84ee80642fa4253 Binary files /dev/null and b/content/zh/post/Frank/images/lWgH2foPpaW6bs4tZ71JzhLuKib1-kFxJhaWd3RwyQk.png differ diff --git a/content/zh/post/Frank/images/ofpefUbUWTyuI5WDrTWmcWbLLsqGXYJM6MPgODEKgOI.png b/content/zh/post/Frank/images/ofpefUbUWTyuI5WDrTWmcWbLLsqGXYJM6MPgODEKgOI.png new file mode 100644 index 0000000000000000000000000000000000000000..378bdc4ff8de8155419a9d199cfddfa1a702a7f2 Binary files /dev/null and b/content/zh/post/Frank/images/ofpefUbUWTyuI5WDrTWmcWbLLsqGXYJM6MPgODEKgOI.png differ diff --git "a/content/zh/post/Frank/\346\265\213\350\257\204\346\212\245\345\221\212\357\274\232\346\226\207\344\273\266\345\257\274\345\205\245\345\223\252\345\256\266\345\274\272\357\274\237.md" "b/content/zh/post/Frank/\346\265\213\350\257\204\346\212\245\345\221\212\357\274\232\346\226\207\344\273\266\345\257\274\345\205\245\345\223\252\345\256\266\345\274\272\357\274\237.md" new file mode 100644 index 0000000000000000000000000000000000000000..06725338837f6c65230b0c8f7f19e6c908179a34 --- /dev/null +++ "b/content/zh/post/Frank/\346\265\213\350\257\204\346\212\245\345\221\212\357\274\232\346\226\207\344\273\266\345\257\274\345\205\245\345\223\252\345\256\266\345\274\272\357\274\237.md" @@ -0,0 +1,604 @@ +# 测评报告：文件导入哪家强？ +# 文件导入哪家强？ +![image](images/PNcP_TtELNoc1JnjdxdJFZJAd5FlK94zcMgv63wDs-w.webp) + +# 引子 +> 最近业务上遇到一个场景，需要将一个/多个文本文件导入到与其结构对应的表中。功能需求比较简单，大部分的关系数据库基本都支持这个功能。基于上面的场景把手头上的几款开源数据库和国产数据库的文件导入功能进行了性能对比。 + +# 摘要 +本文将针对文件导入功能，在MySQL、PostgreSQL、达梦8、LvorySQL、openGauss、OceanBase六款数据库进行了对比测试。测试过程中使用相同的硬件环境和操作系统，且各数据库使用安装时的默认参数。 + +> ***注意：*** + +> 1. ***由于Oracle没有提供ARM相关的安装介质，所以本次测试并未对比Oracle的sqlldr；*** +> 2. ***所有被测数据库都未经过优化，全部使用默认参数；*** + +# 测试环境 +## 服务器 +|CPU|Kunpeng-920| +| ----- | ----- | +|Architecture|aarch64| +|On-line CPU(s) list|0-7| +|CPU主频|2600MHz| + +## 操作系统 +```bash +NAME="openEuler" +VERSION="20.03 (LTS-SP2)" +ID="openEuler" +VERSION_ID="20.03" +PRETTY_NAME="openEuler 20.03 (LTS-SP2)" +ANSI_COLOR="0;31" +``` +## 系统性能 +* 以下是unixbench跑分，供对比参考 + +```bash +------------------------------------------------------------------------ +Benchmark Run: Wed Dec 07 2022 15:48:16 - 15:57:14 +8 CPUs in system; running 1 parallel copy of tests + +Dhrystone 2 using register variables 40999324.8 lps (10.0 s, 2 samples) +Double-Precision Whetstone 4314.8 MWIPS (9.3 s, 2 samples) +Execl Throughput 3762.5 lps (29.8 s, 1 samples) +File Copy 1024 bufsize 2000 maxblocks 685100.0 KBps (30.0 s, 1 samples) +File Copy 256 bufsize 500 maxblocks 189282.0 KBps (30.0 s, 1 samples) +File Copy 4096 bufsize 8000 maxblocks 1992147.0 KBps (30.0 s, 1 samples) +Pipe Throughput 1231178.1 lps (10.0 s, 2 samples) +Pipe-based Context Switching 105636.0 lps (10.0 s, 2 samples) +Process Creation 8963.6 lps (30.0 s, 1 samples) +Shell Scripts (1 concurrent) 7087.6 lpm (60.0 s, 1 samples) +Shell Scripts (8 concurrent) 3055.4 lpm (60.0 s, 1 samples) +System Call Overhead 907956.1 lps (10.0 s, 2 samples) + +System Benchmarks Index Values BASELINE RESULT INDEX +Dhrystone 2 using register variables 116700.0 40999324.8 3513.2 +Double-Precision Whetstone 55.0 4314.8 784.5 +Execl Throughput 43.0 3762.5 875.0 +File Copy 1024 bufsize 2000 maxblocks 3960.0 685100.0 1730.1 +File Copy 256 bufsize 500 maxblocks 1655.0 189282.0 1143.7 +File Copy 4096 bufsize 8000 maxblocks 5800.0 1992147.0 3434.7 +Pipe Throughput 12440.0 1231178.1 989.7 +Pipe-based Context Switching 4000.0 105636.0 264.1 +Process Creation 126.0 8963.6 711.4 +Shell Scripts (1 concurrent) 42.4 7087.6 1671.6 +Shell Scripts (8 concurrent) 6.0 3055.4 5092.4 +System Call Overhead 15000.0 907956.1 605.3 + ======== +System Benchmarks Index Score 1258.0 + +------------------------------------------------------------------------ +Benchmark Run: Wed Dec 07 2022 15:57:14 - 16:06:18 +8 CPUs in system; running 16 parallel copies of tests + +Dhrystone 2 using register variables 326630138.1 lps (10.0 s, 2 samples) +Double-Precision Whetstone 35757.3 MWIPS (8.7 s, 2 samples) +Execl Throughput 23047.5 lps (29.7 s, 1 samples) +File Copy 1024 bufsize 2000 maxblocks 525702.0 KBps (30.0 s, 1 samples) +File Copy 256 bufsize 500 maxblocks 139688.0 KBps (30.0 s, 1 samples) +File Copy 4096 bufsize 8000 maxblocks 1834232.0 KBps (30.0 s, 1 samples) +Pipe Throughput 9754452.8 lps (10.0 s, 2 samples) +Pipe-based Context Switching 1181849.8 lps (10.0 s, 2 samples) +Process Creation 42295.7 lps (30.0 s, 1 samples) +Shell Scripts (1 concurrent) 32399.6 lpm (60.0 s, 1 samples) +Shell Scripts (8 concurrent) 4529.9 lpm (60.1 s, 1 samples) +System Call Overhead 3386402.8 lps (10.0 s, 2 samples) + +System Benchmarks Index Values BASELINE RESULT INDEX +Dhrystone 2 using register variables 116700.0 326630138.1 27988.9 +Double-Precision Whetstone 55.0 35757.3 6501.3 +Execl Throughput 43.0 23047.5 5359.9 +File Copy 1024 bufsize 2000 maxblocks 3960.0 525702.0 1327.5 +File Copy 256 bufsize 500 maxblocks 1655.0 139688.0 844.0 +File Copy 4096 bufsize 8000 maxblocks 5800.0 1834232.0 3162.5 +Pipe Throughput 12440.0 9754452.8 7841.2 +Pipe-based Context Switching 4000.0 1181849.8 2954.6 +Process Creation 126.0 42295.7 3356.8 +Shell Scripts (1 concurrent) 42.4 32399.6 7641.4 +Shell Scripts (8 concurrent) 6.0 4529.9 7549.8 +System Call Overhead 15000.0 3386402.8 2257.6 + ======== +System Benchmarks Index Score 4252.8 + +``` +# 数据准备 +使用benchmarksql-5.0中的历史表作为被测表，建表脚本如下： + +```sql +create table bmsql_history ( + hist_id integer, + h_c_id integer, + h_c_d_id integer, + h_c_w_id integer, + h_d_id integer, + h_w_id integer, + h_date timestamp, + h_amount decimal(6,2), + h_data varchar(24) +); +``` + + +## 测试数据生成脚本 +根据字段类型写了一个生成测试数据的脚本，代码如下： + +```python +# -*- coding=utf-8 -*- + +import csv +import random +import time +import string + +# 创建列表，保存header内容 +header_list = ["hist_id", "h_c_id", "h_c_d_id", "h_c_w_id", "h_d_id", "h_w_id", "h_date", "h_amount", "h_data"] + +g_count = 0 + +def random_list(n): + data_list = [] + global g_count + for i in range(n): + g_count = g_count + 1 + l = [g_count, + random.randint(0,1000), + random.randint(0,1000), + random.randint(0,1000), + random.randint(0,1000), + random.randint(0,1000), + time.strftime("%Y-%m-%d %H:%M:%S", time.localtime()), + #random.randint(0,1000), + round(random.uniform(0, 9999.0), 2), + ''.join(random.sample(string.ascii_letters + string.digits, 20)) + ] + data_list.append(l) + + return data_list + +# 以写方式打开文件。注意添加 newline=""，否则会在两行数据之间都插入一行空白。 +with open("bmsql_history.csv", mode="w", encoding="utf-8", newline="") as f: + + # 基于打开的文件，创建 csv.writer 实例 + writer = csv.writer(f) + + # 写入 header。 + # writerow() 一次只能写入一行。 + writer.writerow(header_list) + + # 写入数据。 + # writerows() 一次写入多行。 + for i in range(10000): + writer.writerows(random_list(1000)) + +``` + + +## 测试数据 +执行脚本后会生成10000000行测试数据，具体如下图： + +![image](images/HIS6hdLiNFe6czt6hwP2pi_NnvwiFnaRrKxs1DhPu5s.png) + +# 性能测试 +## PostgreSQL COPY +### 简介 +`COPY` moves data between PostgreSQL tables and standard file-system files. `COPY TO` copies the contents of a table *to* a file, while `COPY FROM` copies data *from* a file to a table (appending the data to whatever is in the table already). `COPY TO` can also copy the results of a `SELECT` query. + +If a column list is specified, `COPY TO` copies only the data in the specified columns to the file. For `COPY FROM`, each field in the file is inserted, in order, into the specified column. Table columns not specified in the `COPY FROM` column list will receive their default values. + +`COPY` with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server. When `PROGRAM` is specified, the server executes the given command and reads from the standard output of the program, or writes to the standard input of the program. The command must be specified from the viewpoint of the server, and be executable by the PostgreSQL user. When `STDIN` or `STDOUT` is specified, data is transmitted via the connection between the client and the server. + +### 语法 +```sql + Copy ::= COPY table_name [ ( column_name [, ...] ) ] + FROM { 'filename' | STDIN } + [ [ USING ] DELIMITERS 'delimiters' ] + [ WITHOUT ESCAPING ] + [ LOG ERRORS ] + [ REJECT LIMIT 'limit' ] + [ WITH ( option [, ...] ) ] + | copy_option + | TRANSFORM ( { column_name [ data_type ] [ AS transform_expr ] } [, ...] ) + | FIXED FORMATTER ( { column_name( offset, length ) } [, ...] ) [ ( option [, ...] ) | copy_option [ ...] ] ; +``` + + +### 测试 +* PostgreSQL版本（编译安装） + +![image](images/WH2_Mzd71Rk3VVjf1TECjxhpzKx-tAku9kfp3MX1aS0.png) + +```sql +postgres=# copy bmsql_history from '/home/postgres/bmsql_history.csv' delimiter ',' +postgres-# ; +COPY 10000000 +Time: 19829.354 ms (00:19.829) +``` +耗时：约20s + +## IvorySQL COPY +### 简介 +参考PostgreSQL简介。 + +### 语法 +参考PostgreSQL简介。 + +### 测试 +* IvorySQL版本（通过编译安装） + +![image](images/aebKG7cndlMYcoRcq7qs5m-GYbhGj74zOR2NeJKh4GI.png) + +![image](images/Rr8WMD2FqdxOic2kFbxXJic35fdB_YAqnmerRQwqQG4.png) + +```sql +postgres=# copy bmsql_history from '/home/ivory/bmsql_history.csv' delimiter ','; +COPY 10000000 +Time: 21108.218 ms (00:21.108) +``` +![image](images/OAlETvVs-VRdXpoBOenwSil6WkhILHKdzb1OEkU1lYY.png) + +耗时：约20s + +## openGauss COPY +### 简介 +参考PostgreSQL简介。 + +### 语法 +参考PostgreSQL简介。 + +### 测试 +> 分别用编译安装和安装包安装了openGauss，测试结果基本一致。 + +#### 编译安装openGauss +```sql +openGauss=# vacuum; +VACUUM +openGauss=# analyze; +ANALYZE +openGauss=# \timing +Timing is on. +openGauss=# copy bmsql_history from '/home/omm/bmsql_history.csv' delimiter ','; +COPY 10000000 +Time: 112710.938 ms +openGauss=# select version(); + version +------------------------------------------------------------------------------------------------------------------------------------------------------------ + (openGauss 3.1.0 build 2586b083) compiled at 2022-11-28 15:46:36 commit 0 last mr debug on aarch64-unknown-linux-gnu, compiled by g++ (GCC) 7.3.0, 64-bit +(1 row) + +``` +#### 安装包安装极简版 +```sql +openGauss=# truncate bmsql_history; +TRUNCATE TABLE +openGauss=# \timing +Timing is on. +openGauss=# copy bmsql_history from '/home/omm/bmsql_history.csv' delimiter ','; +COPY 10000000 +Time: 122703.615 ms +openGauss=# select version(); + version +------------------------------------------------------------------------------------------------------------------------------------------------------- + (openGauss 3.1.0 build 4e931f9a) compiled at 2022-09-29 14:19:54 commit 0 last mr on aarch64-unknown-linux-gnu, compiled by g++ (GCC) 7.3.0, 64-bit +(1 row) + +Time: 0.568 ms + +``` +耗时：约120s + +## MySQL LOAD +### 简介 +1. MySQL load data 语句能快速将一个文本文件的内容导入到对应的数据库表中（一般文本的一行对应表的一条记录）； +2. 数据库应用程序开发中，涉及大批量数据需要插入时，使用 load data 语句的效率比一般的 insert 语句的高很多； +3. 可以看成select … into outfile语句的反操作，select … into outfile将数据库表中的数据导出保存到一个文件中。参考MySQL 5.7 官方手册 ； + +### 语法 +```sql +LOAD DATA + [LOW_PRIORITY | CONCURRENT] [LOCAL] + INFILE 'file_name' + [REPLACE | IGNORE] + INTO TABLE tbl_name + [PARTITION (partition_name [, partition_name] ...)] + [CHARACTER SET charset_name] + [{FIELDS | COLUMNS} + [TERMINATED BY 'string'] + [[OPTIONALLY] ENCLOSED BY 'char'] + [ESCAPED BY 'char'] + ] + [LINES + [STARTING BY 'string'] + [TERMINATED BY 'string'] + ] + [IGNORE number {LINES | ROWS}] + [(col_name_or_user_var + [, col_name_or_user_var] ...)] + [SET col_name={expr | DEFAULT}, + [, col_name={expr | DEFAULT}] ...] +``` + + +### 测试 +```sql +-- LOAD DATA LOCAL INFILE '文件路径' INTO TABLE tableName FIELDS TERMINATED BY ','; + +LOAD DATA LOCAL INFILE '/root/bmsql_history.csv' INTO TABLE bmsql_history FIELDS TERMINATED BY ','; +``` +![image](images/T311n-kMrG-vp6DcsMlDiHUBfuC6BfNcHDMMYjzs1No.png) + +耗时：69.58s + +![image](images/lWgH2foPpaW6bs4tZ71JzhLuKib1-kFxJhaWd3RwyQk.png) + + + +## DM8 dmfldr +### 简介 +dmfldr（DM Fast Loader）是 DM 提供的快速数据装载命令行工具。用户通过使用dmfldr 工具能够把按照一定格式排序的文本数据以简单、快速、高效的方式载入到 DM 数据库中，或把 DM 数据库中的数据按照一定格式写入文本文件。 + +![image](images/ofpefUbUWTyuI5WDrTWmcWbLLsqGXYJM6MPgODEKgOI.png) + +### 语法/参数说明 +```bash +[dmdba@host-10-208-227-136 ~]$ dmfldr help +version: 03134283938-20221019-172201-20018 +格式: ./dmfldr KEYWORD=value + +例程: ./dmfldr SYSDBA/SYSDBA CONTROL='/opt/data/fldr.ctl' + +USERID 必须是命令行中的第一个参数 +字符串类型参数必须以引号封闭 + +关键字 说明（默认值） +-------------------------------------------------------------------------------- +USERID 用户名/口令， 格式:{[/] | /}[@][