From 370b3b7096525fea2faec359ed81663f3775161d Mon Sep 17 00:00:00 2001 From: hongyedba Date: Wed, 10 Jul 2024 19:24:21 +0800 Subject: [PATCH] =?UTF-8?q?feat(mdb):=20=E5=8F=91=E5=B8=83v2.3=E7=89=88?= =?UTF-8?q?=E6=9C=AC=EF=BC=8C=E8=B0=83=E6=95=B4=E5=AE=89=E8=A3=85=E6=AD=A5?= =?UTF-8?q?=E9=AA=A4=EF=BC=8C=E6=96=B0=E5=A2=9E=E4=BD=BF=E7=94=A8=E9=99=90?= =?UTF-8?q?=E5=88=B6?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- product/en/docs-mdb/v2.0/exceptions.md | 1 - product/en/docs-mdb/v2.0/install.md | 50 +++++++++++++------ .../en/docs-mdb/v2.0/releases/release-2.0.md | 22 ++++++++ product/en/docs-mdb/v2.0/restriction.md | 31 ++++++++++++ product/en/docs-mdb/v2.0/toc.md | 1 + product/zh/docs-mdb/v2.0/exceptions.md | 1 - product/zh/docs-mdb/v2.0/install.md | 50 +++++++++++++------ .../zh/docs-mdb/v2.0/releases/release-2.0.md | 22 ++++++++ product/zh/docs-mdb/v2.0/restriction.md | 31 ++++++++++++ product/zh/docs-mdb/v2.0/toc.md | 1 + 10 files changed, 180 insertions(+), 30 deletions(-) create mode 100644 product/en/docs-mdb/v2.0/restriction.md create mode 100644 product/zh/docs-mdb/v2.0/restriction.md diff --git a/product/en/docs-mdb/v2.0/exceptions.md b/product/en/docs-mdb/v2.0/exceptions.md index 2fc6b3fc..e8978d70 100644 --- a/product/en/docs-mdb/v2.0/exceptions.md +++ b/product/en/docs-mdb/v2.0/exceptions.md @@ -34,7 +34,6 @@ This article describes common exception scenarios and exception information in M | Exception Information | Cause | Solution | |--------|------------------------------------------------------|------------------------------------------------------------------------------------------------------| |Online REDO LOG files or archive log files do not contain the offset scn 283058996. Please perform a new snapshot.| scn is not in the archive log file | After data synchronization is reset, data is synchronized again | -|ORA-01555: snapshot too old| There is too much data in the table | The synchronous correspondence table is ignored | |FATAL: no pg_hba.conf entry for replication connection from host "27.209.10.108", user "mtk_user", SSL off| pg_hba.conf is not configured in the source database | Refer to the source database for instructions on configuring pg_hba.conf |FATAL: terminating Walsender process due to replication timeout.Detail: now time(2022-12-06 01:40:14.299346+08) timeout time(2022-12-06 01:40:14.298937+08) last recv time(2022-12-06 01:40:08.298937+08) heartbeat time(2000-01-01 08:00:00+08).Hint: try increasing wal_sender_timeout or check system time.| wal_sender_timeout parameter configuration problem | Refer to the source instructions for configuring the wal_sender_timeout parameter in postgresql.conf | |ERROR: replication should connect HA port in thread_pool| The database opens the connection pool | The connection pool needs to be closed | diff --git a/product/en/docs-mdb/v2.0/install.md b/product/en/docs-mdb/v2.0/install.md index 3a1bcb1f..f345df3c 100644 --- a/product/en/docs-mdb/v2.0/install.md +++ b/product/en/docs-mdb/v2.0/install.md @@ -22,7 +22,7 @@ The system in which MDB is to be deployed must meet the following requirement. | Java | 11+ | | MogDB | Database: 2.0+ Plug-ins: wal2json | | Oracle | 11g, 12c, 19c | -| MySQL | 5.7, 8.0.x (not support MariaDB) | +| MySQL | 5.7, 8.0.x (partial compatible for MariaDB > 10.4) | |PostgreSQL|Database: 9.6, 10, 11, 12, 13, 14 Plug-ins: decoderbufs, wal2json| |SQL Server|22016 Service Pack 1 (SP1) and later| @@ -30,7 +30,7 @@ The system in which MDB is to be deployed must meet the following requirement. ### Download Software -Download the latest MDB installation package： [MDB latest](https://www.mogdb.io/downloads/mdb/all) +Download the latest MDB installation package： [MDB Latest](https://www.mogdb.io/downloads/mdb/all) ### Upload and Decompress the Software Package @@ -44,7 +44,7 @@ tar -zxvf mdb_v1.0.0.RELEASE_linux_x86_64.tar.gz The MDB repository must use a MogDB (openGauss) database, and the version cannot be earlier than 2.0. -For details about how to install MogDB, see [MogDB Single Node Installation](https://docs.mogdb.io/en/mogdb/v2.1/installation-on-a-single-node). +For details about how to install MogDB, see [MogDB Single Node Installation](https://docs.mogdb.io/zh/mogdb/v5.0/installation-on-a-single-node). After MogDB is installed, use gsql to log in to the database as a super administrator. @@ -70,14 +70,14 @@ MDB depends on Java 11. Therefore, make sure that the Java version is not earli java -version ``` -If the Java version is earlier, you need to visit the following Oracle official website to download the JDK/JRE software package corresponding to Java 11. +If the Java version is earlier, you can download the software package newer than openJDK 11 from the official website of Java. The download link is as follows - + -Download the jdk-11.0.15.1_linux-x64_bin.tar.gz software package, and upload it to the server and decompress it. +Download the openjdk-11.0.0.2_linux-x64.tar.gz software package, and upload it to the server and decompress it. ```shell -tar -zxvf jdk-11.0.15.1_linux-x64_bin.tar.gz +tar -zxvf openjdk-11.0.0.2_linux-x64.tar.gz # Obtain the current path: /usr/local pwd @@ -86,7 +86,7 @@ pwd vi ~/.bash_profile # Replace the JAVA_HOME path and then add the following content to the end of the file. -export JAVA_HOME=/usr/local/jdk-11.0.15 +export JAVA_HOME=/usr/local/jdk-11.0.0.2 export CLASSPATH=$:CLASSPATH:$JAVA_HOME/lib/ export PATH=$PATH:$JAVA_HOME/bin @@ -101,14 +101,14 @@ java -version # The following shows the command output: # -------------- -# java version "11.0.15" 2022-04-19 LTS -# Java(TM) SE Runtime Environment 18.9 (build 11.0.15+8-LTS-149) -# Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.15+8-LTS-149, mixed mode) +# openjdk 11.0.0.2 2024-07-02 +# OpenJDK Runtime Environment 18.9 (build 11.0.0.2+2-2) +# OpenJDK 64-Bit Server VM 18.9 (build 11.0.0.2+2-2, mixed mode) ``` ### Check the Web Port -MDB Server requires a web listening port (default: 55437). During installation, make sure that the port is available. +MDB Server requires a web listening port (updatable, default: 55437). During installation, make sure that the port is available. ```shell netstat -anp | grep LISTEN | grep 55437 # Make sure that no process is in the listening status. @@ -119,7 +119,7 @@ netstat -anp | grep LISTEN | grep 55437 # Make sure that no process is in the For ZooKeeper and kafka installation, make sure that the `/zookeeper` and `/kafka-logs` directories do not exist. If they are residual files, run the following commands to delete them. ```shell -rm -rf /zookeeper # : MDB configuration item, default: /tmp +rm -rf /zookeeper # : MDB configuration item, default: ../mdb_data rm -rf /kafka-logs ``` @@ -185,7 +185,29 @@ The following describes the configuration items. | messageMaxSize | 50 | topic message max size（unit: M）| | mvdServerPort | 9876 | MVD server port | -### Configure MTK and SCA +### Configure ulimit + +During the data synchronization process, some temporary data files need to be stored in the local file system of MDB. For tables containing LOB fields, LOB data files will also be stored additionally. It is necessary to adjust the maximum limit of open handles for the current MDB user to avoid the number of open file handles exceeding the default limit during the synchronization process. + +```shell +# Note: The following command lifts the restrictions for all users. You can modify the '*' to the MDB installation user to only lift the MDB restrictions, and re-login to take effect. +cat >> /etc/security/limits.conf <> /etc/sysctl.conf < 1. +- Fix the issue that when there are both indexes and constraints on the same field, the index is created as unique. +- Fix the issue of incorrect check constraint verification in the structure comparison from MogDB to MySQL. The MySQL check constraint column is parsed from the expression. +- Fix the issue that the limit on the number of new channels does not remove draft channels. + ## v2.2.0(2024.05.17) ### Features diff --git a/product/en/docs-mdb/v2.0/restriction.md b/product/en/docs-mdb/v2.0/restriction.md new file mode 100644 index 00000000..44ef4e9e --- /dev/null +++ b/product/en/docs-mdb/v2.0/restriction.md @@ -0,0 +1,31 @@ +--- +title: Restrictions +summary: MDB Restrictions +author: hongrihua +date: 2022-06-30 +--- + +# Restrictions + +This article introduces some current usage restrictions of the MDB system, and it is necessary to pay attention to avoiding related scenarios during the actual synchronization process. + +## debezium + +### Mixed case scenarios + +1. For all source databases, if there are mixed-case name scenarios and the values are the same after ignoring the case, an error will be reported during source streaming data parsing. + +## Oracle + +### Column type restrictions + +1. Oracle Logmnr cannot obtain the before image of the large object columns. In the filtering conditions of the update/delete operations of tables without primary keys, the filtering conditions of the LOB columns will not be carried. If there are multiple pieces of data in the table where all columns except the LOB columns are exactly the same, then the update/delete of these data cannot be correctly synchronized to the target, which may lead to the deletion of multiple rows or the update of multiple rows. +2. Data synchronization of LONG/LONG RAW types is not supported. If these two types of columns exist in the table, an error will be reported during the data synchronization process. + +## MySQL + +### Field length restrictions + +1. The maximum total length of a single row cannot exceed 65,535 bytes (excluding text/blob related types). +2. The length of a single character field cannot exceed 65,535 bytes. For varchar of the utf8mb4 character set, its length cannot exceed 16,384. +3. The total length of the primary key and unique key cannot exceed 767 bytes. diff --git a/product/en/docs-mdb/v2.0/toc.md b/product/en/docs-mdb/v2.0/toc.md index 7208a0bc..88f9ff16 100644 --- a/product/en/docs-mdb/v2.0/toc.md +++ b/product/en/docs-mdb/v2.0/toc.md @@ -24,6 +24,7 @@ + [PostgreSQL](./source_postgresql.md) + [SQL Server](./source_sqlserver.md) + [Target Database Requirements](./target_db.md) ++ [Usage Restriction](./restriction.md) + [Frequently Exceptions](./exceptions.md) + Release Note + [2.0](releases/release-2.0.md) diff --git a/product/zh/docs-mdb/v2.0/exceptions.md b/product/zh/docs-mdb/v2.0/exceptions.md index 0ecf5275..1512eab5 100644 --- a/product/zh/docs-mdb/v2.0/exceptions.md +++ b/product/zh/docs-mdb/v2.0/exceptions.md @@ -34,7 +34,6 @@ date: 2023-02-01 | 异常信息 | 异常原因 | 解决方法 | |--------|-----------|----------| |Online REDO LOG files or archive log files do not contain the offset scn 283058996. Please perform a new snapshot.|scn不在归档日志文件中|数据同步复位后重新数据同步| -|ORA-01555: snapshot too old|表数据量过大| 忽略同步对应表| |FATAL: no pg_hba.conf entry for replication connection from host "27.209.10.108", user "mtk_user", SSL off|源库未配置pg_hba.conf|参考源库说明配置pg_hba.conf| |FATAL: terminating Walsender process due to replication timeout.Detail: now time(2022-12-06 01:40:14.299346+08) timeout time(2022-12-06 01:40:14.298937+08) last recv time(2022-12-06 01:40:08.298937+08) heartbeat time(2000-01-01 08:00:00+08).Hint: try increasing wal_sender_timeout or check system time.|wal_sender_timeout参数配置问题|参考源库说明配置postgresql.conf中的参数wal_sender_timeout| |ERROR: replication should connect HA port in thread_pool|数据库开启连接池|需关闭连接池| diff --git a/product/zh/docs-mdb/v2.0/install.md b/product/zh/docs-mdb/v2.0/install.md index 27f0bb2d..4db39790 100644 --- a/product/zh/docs-mdb/v2.0/install.md +++ b/product/zh/docs-mdb/v2.0/install.md @@ -22,7 +22,7 @@ MDB 部署的系统需求如下： | Java | 11+ | | MogDB | Database: 2.0+ Plug-ins: wal2json | | Oracle | 11g, 12c, 19c | -| MySQL | 5.7, 8.0.x (不支持MariaDB) | +| MySQL | 5.7, 8.0.x (部分兼容MariaDB > 10.4) | |PostgreSQL|Database: 9.6, 10, 11, 12, 13, 14 Plug-ins: decoderbufs, wal2json| |SQL Server|2016(SP1)及以后版本| @@ -30,7 +30,7 @@ MDB 部署的系统需求如下： ### 下载软件 -下载最新版 MDB 安装包： [MDB latest](https://www.mogdb.io/downloads/mdb/all) +下载最新版 MDB 安装包： [MDB Latest](https://www.mogdb.io/downloads/mdb/all) ### 上传软件并解压 @@ -44,7 +44,7 @@ tar -zxvf mdb_v1.0.0.RELEASE_linux_x86_64.tar.gz MDB 当前资料库必须使用 MogDB(openGauss) 数据库，版本不低于 2.0。 -MogDB 安装请参考： [MogDB单实例安装](https://docs.mogdb.io/zh/mogdb/v2.1/installation-on-a-single-node) +MogDB 安装请参考： [MogDB单实例安装](https://docs.mogdb.io/zh/mogdb/v5.0/installation-on-a-single-node) MogDB 软件安装完成后，在 MogDB 数据库服务器中，使用 gsql 客户端工具以本地超级管理员用户登录数据库 @@ -70,14 +70,14 @@ MDB 依赖 Java 11，需要确保当前安装服务器中，Java 版本不低于 java -version ``` -若 Java 版本过低，则需要从 Oracle 官网下载 Java 11 对应的 JDK/JRE 软件包，下载链接如下： +若 Java 版本过低，可从 Java 官网下载不低于 openJDK 11 的软件包，下载链接如下： - + -下载 jdk-11.0.15.1_linux-x64_bin.tar.gz 软件包，并上传到服务器中解压使用： +下载 openjdk-11.0.0.2_linux-x64.tar.gz 软件包，并上传到服务器中解压使用： ```shell -tar -zxvf jdk-11.0.15.1_linux-x64_bin.tar.gz +tar -zxvf openjdk-11.0.0.2_linux-x64.tar.gz # 获取当前路径, 当前路径: /usr/local pwd @@ -86,7 +86,7 @@ pwd vi ~/.bash_profile # 替换JAVA_HOME路径后, 在文件末尾输入以下内容: -export JAVA_HOME=/usr/local/jdk-11.0.15 +export JAVA_HOME=/usr/local/jdk-11.0.0.2 export CLASSPATH=$:CLASSPATH:$JAVA_HOME/lib/ export PATH=$PATH:$JAVA_HOME/bin @@ -101,14 +101,14 @@ java -version # 以下为示例输出： # -------------- -# java version "11.0.15" 2022-04-19 LTS -# Java(TM) SE Runtime Environment 18.9 (build 11.0.15+8-LTS-149) -# Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.15+8-LTS-149, mixed mode) +# openjdk 11.0.0.2 2024-07-02 +# OpenJDK Runtime Environment 18.9 (build 11.0.0.2+2-2) +# OpenJDK 64-Bit Server VM 18.9 (build 11.0.0.2+2-2, mixed mode) ``` ### 检查 Web 端口占用 -MDB Server 需要一个 Web 监听端口（默认 55437），在安装的时候，需要确保该端口未被占用。 +MDB Server 需要一个 Web 监听端口（可修改，默认 55437），在安装的时候，需要确保该端口未被占用。 ```shell netstat -anp | grep LISTEN | grep 55437 # 确保没有处于 LISTEN 状态的进程 @@ -119,7 +119,7 @@ netstat -anp | grep LISTEN | grep 55437 # 确保没有处于 LISTEN 状态的 zookeeper 和 kafka 安装，需要确保`/zookeeper` 和 `/kafka-logs`目录不存在。如果确定是残留文件，请执行以下命令。 ```shell -rm -rf /zookeeper # : MDB临时文件配置项, 默认值: /tmp +rm -rf /zookeeper # : MDB临时文件配置项, 默认值: ../mdb_data rm -rf /kafka-logs ``` @@ -185,7 +185,29 @@ ls -ltr | messageMaxSize | 50 | topic最大消息（单位: M）| | mvdServerPort | 9876 | MVD 服务端口 | -### 配置MTK 和 SCA +### 配置 ulimit + +在数据同步过程中，需要在 MDB 本地文件系统中存储部分临时数据文件，对于含有 LOB 字段的表，还会额外存储 LOB 数据文件，需要调整当前 MDB 用户的可打开句柄的最大限制，避免同步过程中打开文件句柄数据超过默认限制。 + +```shell +# 注意：以下命令放开了所有用户的限制，可将 * 修改为 MDB 安装用户只放开 MDB 限制，重新登录后生效 +cat >> /etc/security/limits.conf <> /etc/sysctl.conf < 1 +- 修复同一个字段上同时存在索引和约束时，索引被创建为 unique 的问题 +- 修复 MogDB 到 MySQL 结构比较 check 约束校验不正确的问题，MySQL check 约束列从表达式解析 +- 修复新增通道数量限制未去除草稿通道的问题 + ## v2.2.0(2024.05.17) ### Features diff --git a/product/zh/docs-mdb/v2.0/restriction.md b/product/zh/docs-mdb/v2.0/restriction.md new file mode 100644 index 00000000..53abbbcc --- /dev/null +++ b/product/zh/docs-mdb/v2.0/restriction.md @@ -0,0 +1,31 @@ +--- +title: 使用限制 +summary: MDB 使用限制 +author: hongrihua +date: 2022-06-30 +--- + +# 使用限制 + +本文介绍了 MDB 系统目前的一些使用限制，在实际同步过程中需要注意避免相关场景。 + +## debezium + +### 大小写混合场景 + +1. 所有源端数据库，如果存在大小写混合场景，且忽略大小写之后值是相同的，那么在源端解析的时候会报错 + +## Oracle + +### 字段类型限制 + +1. Oracle Logmnr无法获取大对象 LOB 字段的 before image，无主键表 update/delete 操作过滤条件中不会携带 LOB 字段的过滤条件。若表中存在除 LOB 字段之外，其他字段都完全一致的多条数据，那么对这些数据的 update/delete 将无法正确同步到目标端，可能导致删除多行，或者更新多行。 +2. 暂不支持 LONG/LONG RAW 类型的数据同步，若表中存在这两种字段，则数据同步过程中会报错。 + +## MySQL + +### 字段长度限制 + +1. 单行最大总长度不能超过 65535 字节（不包括 text/blob 相关类型） +2. 单个字符字段的长度不能超过 65535 字节，对于 utf8mb4 字符集的 varchar，其长度不能超过 16384 +3. 主键唯一键的总长度不能超过 767 字节 diff --git a/product/zh/docs-mdb/v2.0/toc.md b/product/zh/docs-mdb/v2.0/toc.md index 56a55c68..6df4b659 100644 --- a/product/zh/docs-mdb/v2.0/toc.md +++ b/product/zh/docs-mdb/v2.0/toc.md @@ -24,6 +24,7 @@ + [PostgreSQL](./source_postgresql.md) + [SQL Server](./source_sqlserver.md) + [目标库说明](./target_db.md) ++ [使用限制](./restriction.md) + [常见异常](./exceptions.md) + 发布记录 + [2.0](releases/release-2.0.md) -- Gitee