diff --git a/README.en.md b/README.en.md index 02be9d5395e0ce10bdffac3ced84c764cea0a429..81eb8f06378a026190b1385ffd05f22c65e446df 100644 --- a/README.en.md +++ b/README.en.md @@ -1,9 +1,9 @@ # openGauss-tools-datachecker-performance -#### Description +## Description Opengauss data migration verification tool, including full data verification and incremental data verification. -#### Software Architecture +## Software Architecture Full data verification: JDBC is used to extract the source and target data, and the extraction results are temporarily stored in Kafka. The verification service obtains the extraction results of the specified table from Kafka through the extraction service for verification. Finally, output the verification results to the file in the specified path. @@ -21,13 +21,13 @@ Install kafka (start zookeeper and kafka service) -#### Installation +## Installation 1. Download and start Kafka 2. Obtain the data verification service jar package and the configuration file template (datachecker-check.jar/datachecker-extract.jar, application.yml, application sink.yml, application source.yml) 3. Copy the jar package and configuration file to the specified server directory, configure the relevant configuration file, and start the corresponding jar service. -#### Instructions +### Instructions **Start zookeeper** @@ -103,6 +103,9 @@ Source side service configuration modification application-source.yml file bootstrap-servers is the working address of kafka, which can not be modified by default Data Source Configuration + The tool uses the druid data source by default. Users can customize the connection pool parameters + InitialSize: 100 Default initial connection size + MaxActive: 500 Default number of active database connections ``` **Target side service startup configuration** @@ -116,6 +119,9 @@ Target side service configuration modification application-sink.yml file bootstrap-servers is the working address of kafka, which can not be modified by default Data Source Configuration + The tool uses the druid data source by default. Users can customize the connection pool parameters + InitialSize: 100 Default initial connection size + MaxActive: 500 Default number of active database connections ``` **Start datachecker performance service** @@ -153,9 +159,66 @@ nohup java -Dspring.config.additional-location=config/application.yml -jar datac 3. After the validation service is started, it will detect whether the table data information on the extraction end has been loaded. If the loading is not completed within a certain period of time, the validation service will automatically exit. At this time, you need to query the table information loading progress of the source and destination, and view the loading progress through the log information. Or restart the verification service directly. 4. The incremental verification service is started, and the source side configuration file config application source needs to be modified Debezium enable: true in yml and configure other debezium related configurations. Start the service to start the incremental verification service + +5. The service startup parameters and deployment methods will have a certain impact on the verification performance. + Start parameters: check the service and extraction process, add the java virtual machine parameters - Xmx, - Xms parameter settings (1G-10G), and keep the process parameters consistent + Deployment method: deploy the extraction service to the database nodes of the source and target respectively, and the verification service and kafka can be deployed on separate machines. ``` +### Filter Rule Property Description + +In the verification service, you can configure filter rules to implement user-defined filter rules. Currently, the tool supports adding table level, row level, and column level filtering rules. + +- Table level verification filtering: filter the current database table by configuring a black and white list. The black and white list configuration is mutually exclusive, that is, black and white lists cannot be configured at the same time. + + ``` + Black and white list configuration rules: + 1. Configure one or more table names separated by commas, for example: t_table_1,t_table_2,t_table_three + 2. Regular expression, add examples: ^\w$, ^[a-zA-Z][a-zA-Z0-9_]$ + table: + white: t_table_1,t_table_2,t_table_three + #black: t_table_one + ``` + +- Row level filtering: filter the records in the current table by adding rules to the table. Sort the table data in ascending order according to the primary key, and obtain the records to be verified in the specified table according to the table name, query quantity, and offset information configured by the user. + + ``` + The configuration method is as follows: table_name: 10,100 + If table_ Name The primary key of the table is named id, and the current rule is equivalent to SQL select * from table_name order by id asc limit 10 , 100 + row: + t_table_1: 10,100 + ``` + +- Column level filtering: filter the fields in the current table by adding rules to the table. Column level filtering includes two rules, including rules and exclusive rules. Including rules means that only the configured field list is verified, and exclusive rules means that the configured field list is not verified. Inclusion rules and exclusive rules are mutually exclusive rules. Inclusion rules and exclusive rules cannot be configured in the same table at the same time. When we check the table data, the table must contain a primary key, so when the inclusion rule does not configure a primary key field, we will automatically add a primary key column to the inclusion rule; When the exclusion rule is configured with a primary key field, we will automatically delete the primary key column from the exclusion rule + ``` + Column level filter rule configuration: + 1. The default includes rule configuration methods, such as table_name: field1,field2,... field + 2. Include rule configuration methods, such as table_name: include(field1,field2,... field n) + 3. Exclusive rule configuration method, such as table_name: exclude(field1,field2,... field n) + The default inclusion configuration method cannot be mixed with inclusion rules and exclusive rules; The effective configuration methods are limited to the above three methods. + column: + t_table_1: include(id,portal_id,func_id,name,width,last_upd_time) + t_table_2: exclude(name,height,last_upd_time) + t_table_3: id,width,height,name,last_upd_time + ``` + + **Rule configuration template** + +``` +rules: + # Filter rule switch: enable=true enables the filter rule + enable: true + table: + white: t_table_1,t_table_2,t_table_3 + #black: t_table_1 + row: + t_table_1: 10,100 + column: + t_table_1: include(id,portal_id,func_id,name,width,last_upd_time) + t_table_2: exclude(name,height,last_upd_time) + t_table_3: id,width,height,name,last_upd_time +``` **Developer local startup service** @@ -184,19 +247,10 @@ The current version does not support the verification of geographic location geo -#### Contribution +## Contribution 1. Fork the repository 2. Create Feat_xxx branch 3. Commit your code 4. Create Pull Request - -#### Gitee Feature - -1. You can use Readme\_XXX.md to support different languages, such as Readme\_en.md, Readme\_zh.md -2. Gitee blog [blog.gitee.com](https://blog.gitee.com) -3. Explore open source project [https://gitee.com/explore](https://gitee.com/explore) -4. The most valuable open source project [GVP](https://gitee.com/gvp) -5. The manual of Gitee [https://gitee.com/help](https://gitee.com/help) -6. The most popular members [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/) diff --git a/README.md b/README.md index 950d96982bee54a85b43552ffb435f5f59606b8d..9ae549e49c2af3dab8ac868aa88eae671fdee974 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,9 @@ # openGauss-tools-datachecker-performance -#### 介绍 +## 介绍 openGauss数据迁移校验工具 ，包含全量数据校验以及增量数据校验。 -#### 软件架构 +## 软件架构 全量数据校验，采用JDBC方式抽取源端和目标端数据，并将抽取结果暂存到kafka中，校验服务通过抽取服务从kafka中获取指定表的抽取结果，进行校验。最后将校验结果输出到指定路径的文件文件中。 增量数据校验，通过debezium监控源端数据库的数据变更记录，抽取服务按照一定的频率定期处理debezium的变更记录，对变更记录进行统计。将统计结果发送给数据校验服务。由数据校验服务发起增量数据校验，并将校验结果输出到指定路径文件。 @@ -13,13 +13,13 @@ openGauss数据迁移校验工具 ，包含全量数据校验以及增量数据 JDK11+ kafka安装（启动zookeeper，kafka服务） -#### 安装教程 +## 使用步骤 1. 下载并启动kafka 2. 获取数据校验服务jar包，及配置文件模版（datachecker-check.jar/datachecker-extract.jar,application.yml,application-sink.yml,application-source.yml） 3. 将jar包以及配置文件copy到指定服务器目录，并配置相关配置文件，启动相应的jar服务即可。 -#### 详细使用说明 +### 详细使用说明 **启动Zookeeper** @@ -93,6 +93,9 @@ bin/connect-standalone -daemon etc/kafka/connect-standalone.properties etc/kafka bootstrap-servers 为kafka工作地址，默认安装可不修改 数据源配置 + 工具默认采用druid数据源，用户可以自定义配置连接池参数 + initialSize: 100 默认初始连接大小 + maxActive: 500 默认激活数据库连接数量 ``` @@ -108,6 +111,9 @@ bin/connect-standalone -daemon etc/kafka/connect-standalone.properties etc/kafka bootstrap-servers 为kafka工作地址，默认安装可不修改 数据源配置 + 工具默认采用druid数据源，用户可以自定义配置连接池参数 + initialSize: 100 默认初始连接大小 + maxActive: 500 默认激活数据库连接数量 ``` @@ -139,6 +145,64 @@ nohup java -Dspring.config.additional-location=config/application.yml -jar datac 2、抽取服务在启动后，会自动加载数据库的表相关信息，如果数据量较大，则数据加载会比较耗时。 3、校验服务启动后，会检测抽取端的表数据信息是否加载完成，如果在一定时间内，未完成加载，则校验服务会自行退出。这时需要查询源端和宿端的表信息加载进度，通过日志信息查看加载进度。或者直接重新启动校验服务。 4、增量校验服务启动，需要修改源端配置文件\config\application-source.yml 中 debezium-enable:true并配置其他 debezium相关配置，服务启动即可开启增量校验服务 +5、服务启动参数与部署方式，会对校验性能产生一定的影响。 + 启动参数：校验服务和抽取进程 增加java虚拟机参数 -Xmx、 -Xms 参数设置（1G -10G）各进程参数保持一致即可 + 部署方式：将抽取服务分别部署源端和目标端的数据库节点，校验服务和kafka可以部署在单独机器上。 +``` + +### 过滤规则特性说明 + +在校验服务，通过配置过滤规则，来实现自定义过滤规则实现。工具当前支持表级，行级，列级过滤规则添加。 + +- 表级校验过滤：通过配置黑白名单对当前数据库表进行过滤。黑白名单配置是互斥的，即配置黑白名单不能同时配置。 + + ``` + 黑白名单配置规则： + 1、配置一个或者多个表名称,中间用逗号分隔，例如：t_table_1,t_table_2,t_table_3 + 2、正则表达式，示例：^\w$ 、 ^[a-zA-Z][a-zA-Z0-9_]$ + table: + white: t_table_1,t_table_2,t_table_3 + #black: t_table_1 + ``` + +- 行级过滤：即通过给表添加规则，实现对当前表记录进行过滤。根据主键对表数据按升序进行排序，根据用户配置的表名、查询数量、偏移量信息，获取指定表的待校验记录。 + + ``` + 配置方式 如下： table_name: 10,100 + 如果table_name表主键名为id,当前规则等效SQL select * from table_name order by id asc limit 10 , 100# + row: + t_table_1: 10,100 + ``` + +- 列级过滤：即通过给表添加规则，实现对当前表字段进行过滤。列级过滤包含两种规则，包含规则，和排他规则。包含规则即只校验配置的字段列表，排他规则即不校验配置的字段列表。包含规则和排他规则是互斥规则，同一张表中不能同时配置包含规则和排他规则。由于我们在校验表数据时，表必须包含主键，所以当包含规则未配置主键字段，我们会自动添加主键列到包含规则中；当排他规则配置了主键字段，我们会自动从排除规则中删除主键列 + + ``` + 列级过滤规则配置： + 1、默认包含规则配置方式，例如 table_name: field1,field2,... field + 2、包含规则配置方式，例如 table_name: include(field1,field2,... field n) + 3、排他规则配置方式，例如 table_name: exclude(field1,field2,... field n) + 默认包含配置方式不能与包含规则，排他规则混合配置；且有效配置方式仅限上述三种方式。 + column: + t_table_1: include(id,portal_id,func_id,name,width,last_upd_time) + t_table_2: exclude(name,height,last_upd_time) + t_table_3: id,width,height,name,last_upd_time + ``` + +**规则配置模版** + +``` +rules: + # 过滤规则开关，enable=true 则开启过滤规则，enable=false关闭过滤规则 + enable: true + table: + white: t_table_1,t_table_2,t_table_3 + #black: t_table_1 + row: + t_table_1: 10,100 + column: + t_table_1: include(id,portal_id,func_id,name,width,last_upd_time) + t_table_2: exclude(name,height,last_upd_time) + t_table_3: id,width,height,name,last_upd_time ``` **开发人员本地 启动服务** @@ -168,19 +232,10 @@ MYSQL需要5.7+版本 -#### 参与贡献 +## 参与贡献 1. Fork 本仓库 2. 新建 Feat_xxx 分支 3. 提交代码 4. 新建 Pull Request - -#### 特技 - -1. 使用 Readme\_XXX.md 来支持不同的语言，例如 Readme\_en.md, Readme\_zh.md -2. Gitee 官方博客 [blog.gitee.com](https://blog.gitee.com) -3. 你可以 [https://gitee.com/explore](https://gitee.com/explore) 这个地址来了解 Gitee 上的优秀开源项目 -4. [GVP](https://gitee.com/gvp) 全称是 Gitee 最有价值开源项目，是综合评定出的优秀开源项目 -5. Gitee 官方提供的使用手册 [https://gitee.com/help](https://gitee.com/help) -6. Gitee 封面人物是一档用来展示 Gitee 会员风采的栏目 [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/) diff --git a/config/application.yml b/config/application.yml index 5c52b27ea88ec72db38a78d3eb5b9ac996396cc3..0994390391b90680057004e3d3a7dd5a469bca57 100644 --- a/config/application.yml +++ b/config/application.yml @@ -12,9 +12,47 @@ data: sink-uri: http://127.0.0.1:9002 # server.port=9002 # Configure the black and white list startup mode of the checklist # [black starts the black list verification,white starts the white list verification] - black-white-mode: BLACK - black-list: # like table1,table2 +rules: + # filter rule switch: enable=true enables the filter rule, and enable=false closes the filter rule + enable: false + # Table level verification and filtering: filter the current database table by configuring a black and white list. + # The black and white list configuration is mutually exclusive, that is, the black and white list cannot be configured at the same time. + # If the black and white list is configured at the same time, only the white list will take effect. + # black and white list configuration rules: + # 1. Configure one or more table names separated by commas, for example: t_ table_ 1,t_ table_ 2,t_ table_3 + # 2. Use suffix wildcards, for example: ^\w+$ , ^[a-zA-Z][a-zA-Z0-9_]+$ + table: + # white: t_test_1,t_test_2,t_test_3,t_test_4 + # white: t_test_1 + # white: ^[a-zA-Z][a-zA-Z0-9_]+$ + # black: t_test_9 + # Row level filtering is to filter the records that need to be verified in the current table by adding rules to the table. + # Sort the table data in ascending order according to the primary key, and obtain the records to be verified in the specified table according to the table name, + # query quantity, and offset information configured by the user. + # The configuration mode is as follows: + # table_name: 10,100 + # If the primary key name of table_name is id, the current rule is equivalent to select * from table_name order by id asc limit 10 , 100 + row: + # t_test_1: 10,100 + # Column level filtering is to filter the fields that need to be verified in the current table by adding rules to the table. + # Column level filtering includes two rules: inclusion rules and exclusive rules. + # Including rules means that only the configured field list is verified, and exclusive rules means that the configured field list is not verified. + # Inclusion rules and exclusive rules are mutually exclusive rules. + # Inclusion rules and exclusive rules cannot be configured in the same table at the same time, otherwise the rule will not take effect. + # When we verify the table data, we require that the table must contain a primary key. + # Therefore, if the primary key field is not configured in the inclusion rule, we will automatically add the primary key column to the inclusion rule. + # In addition, if the primary key field is configured in the exclusion rule, we will automatically delete the primary key column from the exclusion rule + # Column level filter rule configuration: + # 1. Include rule configuration, such as table_name: include(field1,field2,... field n) + # 2. Include rule configuration, such as table_name: field1,field2,... field + # 3. exclusion rule configuration, such as table_name: exclude(field1,field2,... field n) + # The default inclusion configuration cannot be mixed with inclusion rules and exclusive rules + # The effective configuration are limited to the above three methods. + column: + # t_test_1: include(id,portal_id,func_id,name,width,last_upd_time) + # t_test_2: exclude(name,height,last_upd_time) + # t_test_3: id,width,height,name,last_upd_time diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/ColumnRule.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/ColumnRule.java new file mode 100644 index 0000000000000000000000000000000000000000..90b19686977722db41657717a649f4c394fd78e9 --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/ColumnRule.java @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.annotation; + +import org.opengauss.datachecker.check.config.validator.RuleConfigColumnValidator; + +import javax.validation.Constraint; +import javax.validation.Payload; +import java.lang.annotation.Documented; +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + +/** + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +@Documented +@Target({ElementType.FIELD}) +@Retention(RetentionPolicy.RUNTIME) +@Constraint(validatedBy = RuleConfigColumnValidator.class) +public @interface ColumnRule { + + String message() default "table column rule is not valid"; + + Class[] groups() default {}; + + Class[] payload() default {}; +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/RowRule.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/RowRule.java new file mode 100644 index 0000000000000000000000000000000000000000..0570430f2772a1fd0468bbc1e5d496c5287a2c79 --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/RowRule.java @@ -0,0 +1,46 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.annotation; + +import org.opengauss.datachecker.check.config.validator.RuleConfigRowValidator; + +import javax.validation.Constraint; +import javax.validation.Payload; +import java.lang.annotation.Documented; +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + +/** + * RowRule + * + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +@Documented +@Target({ElementType.FIELD}) +@Retention(RetentionPolicy.RUNTIME) +@Constraint(validatedBy = RuleConfigRowValidator.class) +public @interface RowRule { + + String message() default "table row rule is not valid"; + + Class[] groups() default {}; + + Class[] payload() default {}; +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/TableRule.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/TableRule.java new file mode 100644 index 0000000000000000000000000000000000000000..28c3d423fd687bd23b21ab6b19fca32bb75f0ee6 --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/annotation/TableRule.java @@ -0,0 +1,46 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.annotation; + +import org.opengauss.datachecker.check.config.validator.RuleConfigTableValidator; + +import javax.validation.Constraint; +import javax.validation.Payload; +import java.lang.annotation.Documented; +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.RetentionPolicy; +import java.lang.annotation.Target; + +/** + * TableRule + * + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +@Documented +@Target({ElementType.FIELD}) +@Retention(RetentionPolicy.RUNTIME) +@Constraint(validatedBy = RuleConfigTableValidator.class) +public @interface TableRule { + + String message() default "table rule is not valid"; + + Class[] groups() default {}; + + Class[] payload() default {}; +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFallbackFactory.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFallbackFactory.java index a53cc093c9304e53add6f057c8a04885d1e76e69..4ff9412c13bfe1c435de3f14add69288cb9e4431 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFallbackFactory.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFallbackFactory.java @@ -15,7 +15,9 @@ package org.opengauss.datachecker.check.client; -import org.opengauss.datachecker.common.entry.enums.CheckBlackWhiteMode; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.CheckMode; +import org.opengauss.datachecker.common.entry.enums.RuleType; import org.opengauss.datachecker.common.entry.extract.ExtractConfig; import org.opengauss.datachecker.common.entry.extract.ExtractTask; import org.opengauss.datachecker.common.entry.extract.RowDataHash; @@ -150,13 +152,17 @@ public class ExtractFallbackFactory implements FallbackFactory getEndpointConfig() { - return Result - .error("Remote call, Get the current endpoint configuration information, abnormal“"); + return Result.error("Remote call, Get the current endpoint configuration information, abnormal“"); } + /** + * Distribution Data Extraction Filter Rules + * + * @param rules rules + */ @Override - public void refreshBlackWhiteList(CheckBlackWhiteMode mode, List tableList) { - + public Result distributeRules(CheckMode checkMode,Map> rules) { + return Result.error("Remote call, Distribution Data Extraction Filter Rules exception"); } } } diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFeignClient.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFeignClient.java index b9f210560f4240c6dfb80c8dd48a1cc2233a2dbe..7e3bc05b2a72944278735559d5b524c347db88a4 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFeignClient.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/ExtractFeignClient.java @@ -15,7 +15,9 @@ package org.opengauss.datachecker.check.client; -import org.opengauss.datachecker.common.entry.enums.CheckBlackWhiteMode; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.CheckMode; +import org.opengauss.datachecker.common.entry.enums.RuleType; import org.opengauss.datachecker.common.entry.extract.ExtractConfig; import org.opengauss.datachecker.common.entry.extract.ExtractTask; import org.opengauss.datachecker.common.entry.extract.RowDataHash; @@ -70,7 +72,7 @@ public interface ExtractFeignClient { * @param taskList Source side task list * @return Request results */ - @PostMapping("/extract/config/sink/task/all") + @PostMapping("/extract/sink/task/all") Result buildExtractTaskAllTables(@RequestParam(name = "processNo") String processNo, @RequestBody List taskList); @@ -183,11 +185,11 @@ public interface ExtractFeignClient { Result getEndpointConfig(); /** - * Update black and white list + * Distribution Data Extraction Filter Rules * - * @param mode Black and white list mode enumeration{@linkplain CheckBlackWhiteMode} - * @param tableList Black and white list - table name set + * @param rules rules + * @return void */ - @PostMapping("/extract/refresh/black/white/list") - void refreshBlackWhiteList(@RequestParam CheckBlackWhiteMode mode, @RequestBody List tableList); + @PostMapping("/extract/rules/distribute") + Result distributeRules(@RequestParam(name = "checkMode") CheckMode checkMode,@RequestBody Map> rules); } diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/FeignClientService.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/FeignClientService.java index c7a876ed3eca982705c013cbfba32c45987549f8..693d649140c090cfedb214489aa495355708c6f1 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/FeignClientService.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/client/FeignClientService.java @@ -15,7 +15,10 @@ package org.opengauss.datachecker.check.client; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.CheckMode; import org.opengauss.datachecker.common.entry.enums.Endpoint; +import org.opengauss.datachecker.common.entry.enums.RuleType; import org.opengauss.datachecker.common.entry.extract.ExtractConfig; import org.opengauss.datachecker.common.entry.extract.ExtractTask; import org.opengauss.datachecker.common.entry.extract.TableMetadata; @@ -234,4 +237,8 @@ public class FeignClientService { return null; } } + + public void distributeRules(Endpoint endpoint, CheckMode checkMode, Map> rules) { + getClient(endpoint).distributeRules(checkMode, rules); + } } diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/DataCheckProperties.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/DataCheckProperties.java index 47cfca98ad329ae9704ae5bbcfe89e5a3036c220..1b0a4a34e574e506ad09b793ca3b95c840661ebe 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/DataCheckProperties.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/DataCheckProperties.java @@ -77,10 +77,6 @@ public class DataCheckProperties { @NotEmpty(message = "The root directory of data verification results cannot be empty") private String dataPath; - /** - * Add black and white list configuration - */ - private CheckBlackWhiteMode blackWhiteMode; /** * statistical-enable : Configure whether to perform verification time statistics. * If true, the execution time of the verification process will be statistically analyzed automatically. diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/RuleConfig.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/RuleConfig.java new file mode 100644 index 0000000000000000000000000000000000000000..5f2d716b4ec0f8c3bff915321a9a84180dedff58 --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/RuleConfig.java @@ -0,0 +1,47 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.config; + +import lombok.Data; +import org.opengauss.datachecker.check.annotation.ColumnRule; +import org.opengauss.datachecker.check.annotation.RowRule; +import org.opengauss.datachecker.check.annotation.TableRule; +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.context.annotation.Configuration; +import org.springframework.validation.annotation.Validated; + +import java.util.Map; + +/** + * RuleConfig + * + * @author ：wangchao + * @date ：Created in 2022/11/29 + * @since ：11 + */ +@Validated +@Data +@Configuration +@ConfigurationProperties(prefix = "rules", ignoreInvalidFields = true) +public class RuleConfig { + private boolean enable; + @TableRule + private Map table; + @RowRule + private Map row; + @ColumnRule + private Map column; +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigColumnValidator.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigColumnValidator.java new file mode 100644 index 0000000000000000000000000000000000000000..49c4ec60145a71d9618d50f3f72c54274d0b5bef --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigColumnValidator.java @@ -0,0 +1,38 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.config.validator; + +import org.opengauss.datachecker.check.annotation.ColumnRule; + +import javax.validation.ConstraintValidatorContext; +import java.util.Map; +import java.util.regex.Pattern; + +/** + * RuleConfigColumnValidator + * + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +public class RuleConfigColumnValidator implements RuleConfigValidator { + private Pattern table_name_pattern = Pattern.compile("^[A-Za-z][A-Za-z0-9_]+$"); + private Pattern column_pattern = Pattern.compile("^[A-Za-z][A-Za-z0-9_(),]+$"); + @Override + public boolean isValid(Map value, ConstraintValidatorContext context) { + return RuleConfigValidatorUtil.isValid(value, table_name_pattern, column_pattern); + } +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigRowValidator.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigRowValidator.java new file mode 100644 index 0000000000000000000000000000000000000000..5458390bef5bfa724a4e743d9dbb8d1a376313dc --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigRowValidator.java @@ -0,0 +1,39 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.config.validator; + +import org.opengauss.datachecker.check.annotation.RowRule; + +import javax.validation.ConstraintValidatorContext; +import java.util.Map; +import java.util.regex.Pattern; + +/** + * RuleConfigRowValidator + * + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +public class RuleConfigRowValidator implements RuleConfigValidator { + private Pattern table_name_pattern = Pattern.compile("^[A-Za-z][A-Za-z0-9_]+$"); + private Pattern row_pattern = Pattern.compile("^\\d+(\\,\\d+)"); + + @Override + public boolean isValid(Map value, ConstraintValidatorContext context) { + return RuleConfigValidatorUtil.isValid(value, table_name_pattern, row_pattern); + } +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigTableValidator.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigTableValidator.java new file mode 100644 index 0000000000000000000000000000000000000000..baa30b4f04cdda46b44b098419dc96df39bf6fdb --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigTableValidator.java @@ -0,0 +1,37 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.config.validator; + +import lombok.extern.slf4j.Slf4j; +import org.opengauss.datachecker.check.annotation.TableRule; + +import javax.validation.ConstraintValidatorContext; +import java.util.Map; + +/** + * TableRuleConfigValidator + * + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +@Slf4j +public class RuleConfigTableValidator implements RuleConfigValidator { + @Override + public boolean isValid(Map value, ConstraintValidatorContext context) { + return RuleConfigValidatorUtil.isValid(value); + } +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigValidator.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigValidator.java new file mode 100644 index 0000000000000000000000000000000000000000..317a7b47395edb331a8a93381c037f718ab7e14e --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigValidator.java @@ -0,0 +1,30 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.config.validator; + +import javax.validation.ConstraintValidator; +import java.lang.annotation.Annotation; +import java.util.Map; + +/** + * RuleConfigValidator + * + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +public interface RuleConfigValidator extends ConstraintValidator> { +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigValidatorUtil.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigValidatorUtil.java new file mode 100644 index 0000000000000000000000000000000000000000..22de91c970a0419bbe79ecadafc0a22b8f5bf68c --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/config/validator/RuleConfigValidatorUtil.java @@ -0,0 +1,88 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.config.validator; + +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.collections4.MapUtils; +import org.apache.commons.lang3.StringUtils; + +import java.util.Map; +import java.util.Map.Entry; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.regex.Pattern; +import java.util.regex.PatternSyntaxException; + +import static org.opengauss.datachecker.common.constant.RuleConstants.RULE_BLACK; +import static org.opengauss.datachecker.common.constant.RuleConstants.RULE_WHITE; + +/** + * RuleConfigValidatorUtil + * + * @author ：wangchao + * @date ：Created in 2022/12/7 + * @since ：11 + */ +@Slf4j +public class RuleConfigValidatorUtil { + + public static boolean isValid(Map value, Pattern tableNamePattern, Pattern textPattern) { + if (MapUtils.isEmpty(value)) { + return true; + } + AtomicBoolean valid = new AtomicBoolean(true); + for (Entry entry : value.entrySet()) { + String tableName = entry.getKey(); + String ruleText = entry.getValue(); + valid.set(tableNamePattern.matcher(tableName).matches() && textPattern.matcher(ruleText).matches()); + if (!valid.get()) { + log.error("RuleConfig tableName={} rule={} error! ", tableName, ruleText); + break; + } + } + return valid.get(); + } + + public static boolean isValid(Map value) { + if (MapUtils.isEmpty(value)) { + return true; + } + final String tableRule = getTableRule(value); + if (value.size() == 1 && isValidTableRule(tableRule)) { + return true; + } else { + log.error("RuleConfig of table rule name , black or white rule ={} is invalid rule", tableRule); + return false; + } + } + + private static String getTableRule(Map tableRules) { + return tableRules.containsValue(RULE_BLACK) ? tableRules.get(RULE_BLACK) : + tableRules.getOrDefault(RULE_WHITE, ""); + } + + private static boolean isValidTableRule(String tableRule) { + if (StringUtils.isBlank(tableRule)) { + return false; + } + try { + Pattern.compile(tableRule); + return true; + } catch (PatternSyntaxException ex) { + log.error("RuleConfig of table rule {} error,", ex.getPattern()); + return false; + } + } +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/controller/CheckBlackWhiteController.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/controller/CheckBlackWhiteController.java deleted file mode 100644 index c0bcfeb5a907b0dd6a477f59f80b539a9112fe02..0000000000000000000000000000000000000000 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/controller/CheckBlackWhiteController.java +++ /dev/null @@ -1,144 +0,0 @@ -/* - * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. - * - * openGauss is licensed under Mulan PSL v2. - * You can use this software according to the terms and conditions of the Mulan PSL v2. - * You may obtain a copy of Mulan PSL v2 at: - * - * http://license.coscl.org.cn/MulanPSL2 - * - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, - * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, - * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. - * See the Mulan PSL v2 for more details. - */ - -package org.opengauss.datachecker.check.controller; - -import io.swagger.v3.oas.annotations.Operation; -import io.swagger.v3.oas.annotations.Parameter; -import io.swagger.v3.oas.annotations.tags.Tag; -import org.opengauss.datachecker.check.service.CheckBlackWhiteService; -import org.opengauss.datachecker.common.web.Result; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.validation.annotation.Validated; -import org.springframework.web.bind.annotation.PostMapping; -import org.springframework.web.bind.annotation.RequestBody; -import org.springframework.web.bind.annotation.RequestMapping; -import org.springframework.web.bind.annotation.RestController; - -import java.util.List; - -/** - * @author ：wangchao - * @date ：Created in 2022/5/25 - * @since ：11 - */ -@Tag(name = "CheckBlackWhiteController", description = "Verification service - black and white list management") -@Validated -@RestController -@RequestMapping -public class CheckBlackWhiteController { - - @Autowired - private CheckBlackWhiteService checkBlackWhiteService; - - /** - * Add white list this function clears the historical white list and resets the white list to the current list - * - * @param whiteList whiteList - * @return request result - */ - @PostMapping("/add/white/list") - public Result addWhiteList( - @Parameter(name = "whiteList", description = "whiteList") @RequestBody List whiteList) { - checkBlackWhiteService.addWhiteList(whiteList); - return Result.success(); - } - - /** - * Update white list this function adds the current list to the white list on the basis of the current white list - * - * @param whiteList whiteList - * @return request result - */ - @PostMapping("/update/white/list") - public Result updateWhiteList( - @Parameter(name = "whiteList", description = "whiteList") @RequestBody List whiteList) { - checkBlackWhiteService.updateWhiteList(whiteList); - return Result.success(); - } - - /** - * Remove white list this function removes the current list from the current white list - * - * @param whiteList whiteList - * @return request result - */ - @PostMapping("/delete/white/list") - public Result deleteWhiteList( - @Parameter(name = "whiteList", description = "whiteList") @RequestBody List whiteList) { - checkBlackWhiteService.deleteWhiteList(whiteList); - return Result.success(); - } - - /** - * Query white list - * - * @return white list - */ - @PostMapping("/query/white/list") - public Result> queryWhiteList() { - return Result.success(checkBlackWhiteService.queryWhiteList()); - } - - /** - * Add blacklist list this function clears the historical blacklist and resets the blacklist to the current list - * - * @param blackList blackList - * @return request result - */ - @PostMapping("/add/black/list") - public Result addBlackList( - @Parameter(name = "blackList", description = "Blacklist list") @RequestBody List blackList) { - checkBlackWhiteService.addBlackList(blackList); - return Result.success(); - } - - /** - * Update blacklist list this function adds the current list to the blacklist on the basis of the current blacklist - * - * @param blackList blackList - * @return request result - */ - @PostMapping("/update/black/list") - public Result updateBlackList( - @Parameter(name = "blackList", description = "Blacklist list") @RequestBody List blackList) { - checkBlackWhiteService.updateBlackList(blackList); - return Result.success(); - } - - /** - * Remove blacklist list this function removes the current list from the blacklist based on the current blacklist - * - * @param blackList blackList - * @return request result - */ - @PostMapping("/delete/black/list") - public Result deleteBlackList( - @Parameter(name = "blackList", description = "Blacklist list") @RequestBody List blackList) { - checkBlackWhiteService.deleteBlackList(blackList); - return Result.success(); - } - - /** - * Query blacklist list - * - * @return blackList - */ - @Operation(summary = "Query blacklist list ") - @PostMapping("/query/black/list") - public Result> queryBlackList() { - return Result.success(checkBlackWhiteService.queryBlackList()); - } -} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/AbstractCheckLoader.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/AbstractCheckLoader.java index 25f211af5e225975ab2a4edf0ac97242d5a772fd..a316fbcc86338cd03a89b8fcfe1741bfbdfb1718 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/AbstractCheckLoader.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/AbstractCheckLoader.java @@ -31,7 +31,7 @@ import javax.annotation.Resource; */ @Slf4j public abstract class AbstractCheckLoader implements CheckLoader { - protected static final int RETRY_TIMES = 3; + protected static final int RETRY_TIMES = 30; private static ConfigurableApplicationContext applicationContext; @Resource diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckEnvironment.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckEnvironment.java index 9c869926ef4dc6133e5b234817cfc6f0d9ef6699..1f0313d299b10eda9e9054e515afaccc6eba3b3f 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckEnvironment.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckEnvironment.java @@ -15,12 +15,15 @@ package org.opengauss.datachecker.check.load; +import org.opengauss.datachecker.common.entry.common.Rule; import org.opengauss.datachecker.common.entry.enums.CheckMode; import org.opengauss.datachecker.common.entry.enums.Endpoint; +import org.opengauss.datachecker.common.entry.enums.RuleType; import org.opengauss.datachecker.common.entry.extract.Database; import org.springframework.stereotype.Service; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.concurrent.ExecutorService; import java.util.concurrent.atomic.AtomicReference; @@ -36,6 +39,7 @@ import java.util.concurrent.atomic.AtomicReference; public class CheckEnvironment { private static final Map EXTRACT_DATABASE = new HashMap<>(); private static final AtomicReference CHECK_MODE_REF = new AtomicReference<>(); + private static final Map> RULES = new HashMap<>(); private static String exportCheckPath = ""; private static boolean metaLoading = false; @@ -131,4 +135,8 @@ public class CheckEnvironment { public ExecutorService getCheckExecutorService() { return threadPoolExecutor; } + + public void addRules(Map> rules) { + RULES.putAll(rules); + } } diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/BlackListLoader.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckRuleDistributeLoader.java similarity index 41% rename from datachecker-check/src/main/java/org/opengauss/datachecker/check/load/BlackListLoader.java rename to datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckRuleDistributeLoader.java index 6240d6b64976363ef4cc9a9974a71c41e2673db5..fb15bf27ce3f21804de0170b68ef4edf9dfb2132 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/BlackListLoader.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckRuleDistributeLoader.java @@ -16,47 +16,52 @@ package org.opengauss.datachecker.check.load; import lombok.extern.slf4j.Slf4j; -import org.opengauss.datachecker.check.service.CheckBlackWhiteService; -import org.opengauss.datachecker.check.service.EndpointMetaDataManager; -import org.opengauss.datachecker.common.entry.enums.CheckBlackWhiteMode; -import org.opengauss.datachecker.common.exception.CheckingException; -import org.springframework.beans.factory.annotation.Value; +import org.opengauss.datachecker.check.client.FeignClientService; +import org.opengauss.datachecker.check.config.RuleConfig; +import org.opengauss.datachecker.check.modules.rule.RuleParser; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.CheckMode; +import org.opengauss.datachecker.common.entry.enums.Endpoint; +import org.opengauss.datachecker.common.entry.enums.RuleType; import org.springframework.core.annotation.Order; import org.springframework.stereotype.Service; import javax.annotation.Resource; -import java.util.Arrays; +import java.util.List; +import java.util.Map; import java.util.Objects; /** - * MetaDataLoader + * CheckRuleLoader * * @author ：wangchao - * @date ：Created in 2022/11/9 + * @date ：Created in 2022/10/31 * @since ：11 */ @Slf4j -@Order(101) +@Order(98) @Service -public class BlackListLoader extends AbstractCheckLoader { - @Value("${data.check.black-list}") - private String[] backList; - @Value("${data.check.black-white-mode}") - private CheckBlackWhiteMode blackWhiteMode; +public class CheckRuleDistributeLoader extends AbstractCheckLoader { @Resource - private CheckBlackWhiteService checkBlackWhiteService; + private FeignClientService feignClient; + @Resource + private RuleConfig config; + /** + * Initialize the verification result environment + */ @Override public void load(CheckEnvironment checkEnvironment) { - try { - if (Objects.equals(blackWhiteMode, CheckBlackWhiteMode.BLACK)) { - if (backList.length != 0) { - checkBlackWhiteService.addBlackList(Arrays.asList(backList)); - } - log.info("check service load back list success.{}", backList); - } - } catch (CheckingException ex) { - log.error("check service load back list exception :", ex); + RuleParser ruleParser = new RuleParser(); + CheckMode checkMode = checkEnvironment.getCheckMode(); + if (Objects.equals(CheckMode.INCREMENT, checkMode)) { + config.getTable().clear(); + config.getRow().clear(); } + final Map> rules = ruleParser.parser(config); + feignClient.distributeRules(Endpoint.SOURCE, checkMode, rules); + feignClient.distributeRules(Endpoint.SINK, checkMode, rules); + checkEnvironment.addRules(rules); + log.info("check service distribute rules success."); } } diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckStartLoader.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckStartLoader.java index 36df41a2c9447bbb6685e5846d3508e4403620d3..d78c92b406180439ff3adcfae9866afc1b53980c 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckStartLoader.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/load/CheckStartLoader.java @@ -24,6 +24,7 @@ import org.springframework.stereotype.Service; import javax.annotation.Resource; import java.time.Duration; import java.time.LocalDateTime; +import java.util.Objects; /** * CheckStartLoader @@ -41,6 +42,9 @@ public class CheckStartLoader extends AbstractCheckLoader { @Override public void load(CheckEnvironment checkEnvironment) { + if (Objects.equals(CheckMode.INCREMENT, checkEnvironment.getCheckMode())) { + return; + } final LocalDateTime startTime = LocalDateTime.now(); checkService.start(CheckMode.FULL); final LocalDateTime endTime = LocalDateTime.now(); diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/check/DataCheckService.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/check/DataCheckService.java index 33656deac0d45b77a8f4d3576083977ecb72ff96..2193ab0812ec10502ca3b6d91d8279ae570892f9 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/check/DataCheckService.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/check/DataCheckService.java @@ -48,6 +48,7 @@ public class DataCheckService { private CheckEnvironment checkEnvironment; @Resource private ThreadPoolTaskExecutor asyncCheckExecutor; + /** * submit check table data runnable * @@ -79,8 +80,8 @@ public class DataCheckService { */ public void incrementCheckTableData(String tableName, String process, SourceDataLog dataLog) { IncrementDataCheckParam checkParam = - new IncrementDataCheckParam().setTableName(tableName).setBucketCapacity(dataCheckConfig.getBucketCapacity()) - .setDataLog(dataLog).setProcess(process); + new IncrementDataCheckParam().setSchema(getSinkSchema()).setTableName(tableName).setProcess(process) + .setBucketCapacity(dataCheckConfig.getBucketCapacity()).setDataLog(dataLog); final IncrementCheckThread incrementCheck = new IncrementCheckThread(checkParam, dataCheckRunnableSupport); asyncCheckExecutor.submit(incrementCheck); } diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/ColumnRuleParser.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/ColumnRuleParser.java new file mode 100644 index 0000000000000000000000000000000000000000..6ca3cb7f53d2da063fa3f9541e2ab321d1e57ff8 --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/ColumnRuleParser.java @@ -0,0 +1,82 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.modules.rule; + +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.StringUtils; +import org.opengauss.datachecker.common.constant.RuleConstants; +import org.opengauss.datachecker.common.entry.common.Rule; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * ColumnRuleParser + * + * @author ：wangchao + * @date ：Created in 2022/12/3 + * @since ：11 + */ +@Slf4j +public class ColumnRuleParser { + private static final String EXCLUDE = RuleConstants.RULE_EXCLUDE; + private static final String EXCLUDE_BRACE = RuleConstants.RULE_EXCLUDE + RuleConstants.RULE_LEFT_BRACE; + private static final String INCLUDE = RuleConstants.RULE_INCLUDE; + private static final String INCLUDE_BRACE = RuleConstants.RULE_INCLUDE + RuleConstants.RULE_LEFT_BRACE; + private static final String RIGHT_BRACE = RuleConstants.RULE_RIGHT_BRACE; + private static final String SPACE = ""; + + /** + * Parse the corresponding table column filtering rules according to columnRuleConfig + * + * @param columnRuleConfig columnRuleConfig + * @return rules + */ + public List parse(Map columnRuleConfig) { + List rulesList = new ArrayList<>(); + columnRuleConfig.entrySet().stream().filter(entry -> StringUtils.isNotBlank(entry.getValue())) + .forEach(entry -> { + final String text = entry.getValue(); + final String name = entry.getKey(); + if (isDefaultColumnRule(text) || isColumnIncludeRule(text)) { + rulesList.add(new Rule(name, formatColumnRule(text, INCLUDE_BRACE), INCLUDE)); + } else if (isColumnExcludeRule(text)) { + rulesList.add(new Rule(name, formatColumnRule(text, EXCLUDE_BRACE), EXCLUDE)); + } else { + log.error("rule column invalid, key={} , value={}", name, text); + } + }); + return rulesList; + } + + private String formatColumnRule(String text, String keyword) { + return text.replace(keyword, SPACE).replace(RIGHT_BRACE, SPACE); + } + + private boolean isDefaultColumnRule(String text) { + return !text.contains(INCLUDE_BRACE) && !text.contains(EXCLUDE_BRACE); + } + + private boolean isColumnIncludeRule(String text) { + return text.startsWith(INCLUDE_BRACE) && text.endsWith(RIGHT_BRACE) && !text.contains(EXCLUDE_BRACE); + } + + private boolean isColumnExcludeRule(String text) { + return text.startsWith(EXCLUDE_BRACE) && text.endsWith(RIGHT_BRACE) && !text.contains(INCLUDE_BRACE); + } + +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/RowRuleParser.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/RowRuleParser.java new file mode 100644 index 0000000000000000000000000000000000000000..59e4a637aec78220f6d7095f7c9d7e60ae3ebbdd --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/RowRuleParser.java @@ -0,0 +1,49 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.modules.rule; + +import org.apache.commons.lang3.StringUtils; +import org.opengauss.datachecker.common.entry.common.Rule; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * RowRuleParser + * + * @author ：wangchao + * @date ：Created in 2022/12/3 + * @since ：11 + */ +public class RowRuleParser { + + /** + * Resolve the corresponding row record filtering rules according to rowRuleConfig + * + * @param rowRuleConfig rowRuleConfig + * @return rules + */ + public List parse(Map rowRuleConfig) { + List rulesList = new ArrayList<>(); + rowRuleConfig.forEach((name, text) -> { + if (StringUtils.isNotBlank(text)) { + rulesList.add(new Rule(name, text)); + } + }); + return rulesList; + } +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/RuleParser.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/RuleParser.java new file mode 100644 index 0000000000000000000000000000000000000000..cbefd3c5fc43380a35f7f4e39ffb602ba8ba45a3 --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/RuleParser.java @@ -0,0 +1,95 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.modules.rule; + +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.collections4.CollectionUtils; +import org.apache.commons.collections4.MapUtils; +import org.opengauss.datachecker.check.config.RuleConfig; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.RuleType; +import org.springframework.util.Assert; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * RuleParser + * + * @author ：wangchao + * @date ：Created in 2022/11/30 + * @since ：11 + */ +@Slf4j +public class RuleParser { + + /** + * Filter Rule Configuration Resolution + * + * @param config Filter Rule Configuration + * @return Filter Rule + */ + public Map> parser(RuleConfig config) { + Assert.notNull(config, "the rule config cannot be empty"); + if (config.isEnable()) { + Map> rules = new HashMap<>(); + parseTableRules(rules, config.getTable()); + parseRowRules(rules, config.getRow()); + parseColumnRules(rules, config.getColumn()); + return rules; + } + return new HashMap<>(); + } + + private void parseTableRules(Map> rules, Map tableRuleConfig) { + if (MapUtils.isEmpty(tableRuleConfig)) { + return; + } + TableRuleParser tableRuleParser = new TableRuleParser(); + // Rule filtering, filtering illegal or invalid rules + final Map tableRules = tableRuleParser.filterRules(tableRuleConfig); + Rule rule = tableRuleParser.parser(tableRules); + if (Objects.nonNull(rule)) { + putRules(rules, RuleType.TABLE, List.of(rule)); + } + } + + private void parseRowRules(Map> rules, Map rowRuleConfig) { + if (MapUtils.isEmpty(rowRuleConfig)) { + return; + } + RowRuleParser rowRuleParser = new RowRuleParser(); + List rulesList = rowRuleParser.parse(rowRuleConfig); + putRules(rules, RuleType.ROW, rulesList); + } + + private void parseColumnRules(Map> rules, Map columnRuleConfig) { + if (MapUtils.isEmpty(columnRuleConfig)) { + return; + } + ColumnRuleParser columnRuleParser = new ColumnRuleParser(); + List rulesList = columnRuleParser.parse(columnRuleConfig); + putRules(rules, RuleType.COLUMN, rulesList); + } + + private void putRules(Map> rules, RuleType ruleType, List ruleList) { + if (CollectionUtils.isNotEmpty(ruleList)) { + rules.put(ruleType, ruleList); + } + } +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/TableRuleParser.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/TableRuleParser.java new file mode 100644 index 0000000000000000000000000000000000000000..5331fe6970dd86281d489956105640a51fcfb0ae --- /dev/null +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/modules/rule/TableRuleParser.java @@ -0,0 +1,73 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.modules.rule; + +import org.apache.commons.collections4.MapUtils; +import org.apache.commons.lang3.StringUtils; +import org.opengauss.datachecker.common.constant.RuleConstants; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.springframework.util.Assert; + +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Map.Entry; +import java.util.stream.Collectors; + +/** + * TableRuleParser + * + * @author ：wangchao + * @date ：Created in 2022/12/3 + * @since ：11 + */ +public class TableRuleParser { + private static final String RULE_WHITE = RuleConstants.RULE_WHITE; + private static final String RULE_BLACK = RuleConstants.RULE_BLACK; + private static final List RULE_WHITE_BLACK = List.of(RULE_WHITE, RULE_BLACK); + + /** + * Parse the corresponding black-and-white list filtering rule according to tableRuleConfig + * + * @param tableRules tableRules + * @return rule + */ + public Rule parser(Map tableRules) { + Assert.isTrue(MapUtils.isNotEmpty(tableRules), "table rule is not empty!"); + String ruleType = getTableRuleType(tableRules); + return new Rule(ruleType, tableRules.get(ruleType)); + } + + /** + * Filter invalid black-and-white list rules according to tableRuleConfig + * + * @param tableRuleConfig tableRuleConfig + * @return rules + */ + public Map filterRules(Map tableRuleConfig) { + return tableRuleConfig.entrySet().stream().filter(entry -> isWhiteBlackRule(entry.getKey())) + .filter(entry -> StringUtils.isNotBlank(entry.getValue())).collect( + Collectors.toMap(entry -> entry.getKey().toLowerCase(Locale.ENGLISH), Entry::getValue)); + } + + private boolean isWhiteBlackRule(String key) { + return RULE_WHITE_BLACK.contains(key.toLowerCase(Locale.ENGLISH)); + } + + private String getTableRuleType(Map tableRules) { + return tableRules.containsKey(RULE_WHITE) ? RULE_WHITE : RULE_BLACK; + } +} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/service/CheckBlackWhiteService.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/service/CheckBlackWhiteService.java deleted file mode 100644 index 129a9cc49ccef5ed5422ed38b60e9590bbd4244b..0000000000000000000000000000000000000000 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/service/CheckBlackWhiteService.java +++ /dev/null @@ -1,131 +0,0 @@ -/* - * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. - * - * openGauss is licensed under Mulan PSL v2. - * You can use this software according to the terms and conditions of the Mulan PSL v2. - * You may obtain a copy of Mulan PSL v2 at: - * - * http://license.coscl.org.cn/MulanPSL2 - * - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, - * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, - * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. - * See the Mulan PSL v2 for more details. - */ - -package org.opengauss.datachecker.check.service; - -import lombok.extern.slf4j.Slf4j; -import org.opengauss.datachecker.check.client.FeignClientService; -import org.opengauss.datachecker.check.config.DataCheckProperties; -import org.opengauss.datachecker.common.entry.enums.CheckBlackWhiteMode; -import org.opengauss.datachecker.common.entry.enums.Endpoint; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.stereotype.Service; - -import java.util.ArrayList; -import java.util.List; -import java.util.Set; -import java.util.concurrent.ConcurrentSkipListSet; - -/** - * @author ：wangchao - * @date ：Created in 2022/6/22 - * @since ：11 - */ -@Slf4j -@Service -public class CheckBlackWhiteService { - private static final Set WHITE = new ConcurrentSkipListSet<>(); - private static final Set BLACK = new ConcurrentSkipListSet<>(); - - @Autowired - private FeignClientService feignClientService; - - @Autowired - private DataCheckProperties dataCheckProperties; - - @Autowired - private EndpointMetaDataManager endpointMetaDataManager; - - /** - * Add white list this function clears the historical white list and resets the white list to the current list - * - * @param whiteList whiteList - */ - public void addWhiteList(List whiteList) { - WHITE.clear(); - WHITE.addAll(whiteList); - refreshBlackWhiteList(); - log.info("add whitelist list [{}]", whiteList); - } - - /** - * Update white list this function adds the current list to the white list on the basis of the current white list - * - * @param whiteList whiteList - */ - public void updateWhiteList(List whiteList) { - WHITE.addAll(whiteList); - refreshBlackWhiteList(); - log.info("update whitelist list [{}]", whiteList); - } - - /** - * Remove white list this function removes the current list from the current white list - * - * @param whiteList whiteList - */ - public void deleteWhiteList(List whiteList) { - WHITE.removeAll(whiteList); - refreshBlackWhiteList(); - log.info("delete whitelist list [{}]", whiteList); - } - - /** - * Query white list - * - * @return whiteList - */ - public List queryWhiteList() { - return new ArrayList<>(WHITE); - } - - public void addBlackList(List blackList) { - BLACK.clear(); - BLACK.addAll(blackList); - refreshBlackWhiteList(); - log.info("add blackList list [{}]", blackList); - } - - public void updateBlackList(List blackList) { - BLACK.addAll(blackList); - refreshBlackWhiteList(); - log.info("update blackList list [{}]", blackList); - } - - public void deleteBlackList(List blackList) { - BLACK.removeAll(blackList); - refreshBlackWhiteList(); - log.info("delete blackList list [{}]", blackList); - } - - public List queryBlackList() { - return new ArrayList<>(BLACK); - } - - private void refreshBlackWhiteList() { - final CheckBlackWhiteMode blackWhiteMode = dataCheckProperties.getBlackWhiteMode(); - if (blackWhiteMode == CheckBlackWhiteMode.WHITE) { - // White list mode - feignClientService.getClient(Endpoint.SOURCE).refreshBlackWhiteList(blackWhiteMode, new ArrayList<>(WHITE)); - feignClientService.getClient(Endpoint.SINK).refreshBlackWhiteList(blackWhiteMode, new ArrayList<>(WHITE)); - } else if (blackWhiteMode == CheckBlackWhiteMode.BLACK) { - // Blacklist mode - feignClientService.getClient(Endpoint.SOURCE).refreshBlackWhiteList(blackWhiteMode, new ArrayList<>(BLACK)); - feignClientService.getClient(Endpoint.SINK).refreshBlackWhiteList(blackWhiteMode, new ArrayList<>(BLACK)); - } - endpointMetaDataManager.load(); - } - -} diff --git a/datachecker-check/src/main/java/org/opengauss/datachecker/check/service/impl/CheckServiceImpl.java b/datachecker-check/src/main/java/org/opengauss/datachecker/check/service/impl/CheckServiceImpl.java index c29c91c1ea7995c2bcc5bb0438dbd5cb3d19d2d4..f46d2d18f4e5e0df08390b89b2051ae7975b9b2a 100644 --- a/datachecker-check/src/main/java/org/opengauss/datachecker/check/service/impl/CheckServiceImpl.java +++ b/datachecker-check/src/main/java/org/opengauss/datachecker/check/service/impl/CheckServiceImpl.java @@ -100,8 +100,6 @@ public class CheckServiceImpl implements CheckService { private CheckEnvironment checkEnvironment; @Value("${data.check.auto-clean-environment}") private boolean isAutoCleanEnvironment = true; - @Value("${data.check.check-with-sync-extracting}") - private boolean isCheckWithSyncExtracting = true; /** * Enable verification service diff --git a/datachecker-check/src/main/resources/application.yml b/datachecker-check/src/main/resources/application.yml index 16b38aa150f0a54d5725d2a7cbf8ca5bf726c9b6..a28979ae0a10288b89ef76adbd6115b7a043089c 100644 --- a/datachecker-check/src/main/resources/application.yml +++ b/datachecker-check/src/main/resources/application.yml @@ -22,10 +22,6 @@ spring: request-timeout-ms: 300000 fetch-max-bytes: 536870912 # 512M -feign: - okhttp: - enabled: true - logging: config: classpath:log4j2.xml @@ -34,8 +30,6 @@ data: data-path: local_path/xxx bucket-expect-capacity: 10 health-check-api: /extract/health - black-white-mode: BLACK - black-list: # statistical-enable : Configure whether to perform verification time statistics. # If true, the execution time of the verification process will be statistically analyzed automatically. statistical-enable: false @@ -45,8 +39,45 @@ data: check-with-sync-extracting: true retry-fetch-record-times: 5 error-rate: 30 +rules: + # filter rule switch: enable=true enables the filter rule, and enable=false closes the filter rule + enable: false + # Table level verification and filtering: filter the current database table by configuring a black and white list. + # The black and white list configuration is mutually exclusive, that is, the black and white list cannot be configured at the same time. + # If the black and white list is configured at the same time, only the white list will take effect. + # black and white list configuration rules: + # 1. Configure one or more table names separated by commas, for example: t_ table_ 1,t_ table_ 2,t_ table_3 + # 2. Use suffix wildcards, for example: ^\w+$ , ^[a-zA-Z][a-zA-Z0-9_]+$ + # table: + # white: t_test_1 + # white: t_test_1 + # white: ^[a-zA-Z][a-zA-Z0-9_]+$ + # black: t_test_9 + # Row level filtering is to filter the records that need to be verified in the current table by adding rules to the table. + # Sort the table data in ascending order according to the primary key, and obtain the records to be verified in the specified table according to the table name, + # query quantity, and offset information configured by the user. + # The configuration mode is as follows: + # table_name: 10,100 + # If the primary key name of table_name is id, the current rule is equivalent to select * from table_name order by id asc limit 10 , 100 + # row: + # t_test_1: 10,100 - - - + # Column level filtering is to filter the fields that need to be verified in the current table by adding rules to the table. + # Column level filtering includes two rules: inclusion rules and exclusive rules. + # Including rules means that only the configured field list is verified, and exclusive rules means that the configured field list is not verified. + # Inclusion rules and exclusive rules are mutually exclusive rules. + # Inclusion rules and exclusive rules cannot be configured in the same table at the same time, otherwise the rule will not take effect. + # When we verify the table data, we require that the table must contain a primary key. + # Therefore, if the primary key field is not configured in the inclusion rule, we will automatically add the primary key column to the inclusion rule. + # In addition, if the primary key field is configured in the exclusion rule, we will automatically delete the primary key column from the exclusion rule + # Column level filter rule configuration: + # 1. Include rule configuration, such as table_name: include(field1,field2,... field n) + # 2. Include rule configuration, such as table_name: field1,field2,... field + # 3. exclusion rule configuration, such as table_name: exclude(field1,field2,... field n) + # The default inclusion configuration cannot be mixed with inclusion rules and exclusive rules + # The effective configuration are limited to the above three methods. + # column: + # t_test_1: include(id,portal_id,func_id,name,width,last_upd_time) + # t_test_2: exclude(name,height,last_upd_time) + # t_test_3: id,width,height,name,last_upd_time \ No newline at end of file diff --git a/datachecker-check/src/main/resources/log4j2.xml b/datachecker-check/src/main/resources/log4j2.xml index 94250dda5ef9ad0805daa9d3c8bc10c749848ee8..f86c97519ddf7c25f2d0ca275c904824d3220188 100644 --- a/datachecker-check/src/main/resources/log4j2.xml +++ b/datachecker-check/src/main/resources/log4j2.xml @@ -41,14 +41,14 @@ - + - - + + diff --git a/datachecker-check/src/test/java/org/opengauss/datachecker/check/modules/rule/ColumnRuleParserTest.java b/datachecker-check/src/test/java/org/opengauss/datachecker/check/modules/rule/ColumnRuleParserTest.java new file mode 100644 index 0000000000000000000000000000000000000000..deed8c0ac49c6342b20be6c478fa14581af5b088 --- /dev/null +++ b/datachecker-check/src/test/java/org/opengauss/datachecker/check/modules/rule/ColumnRuleParserTest.java @@ -0,0 +1,121 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.modules.rule; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.opengauss.datachecker.common.entry.common.Rule; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +class ColumnRuleParserTest { + + private ColumnRuleParser columnRuleParser; + + @BeforeEach + void setUp() { + columnRuleParser = new ColumnRuleParser(); + } + + @Test + void testParse_only_include() { + // Setup + final Map columnRuleConfig = new HashMap<>(); + columnRuleConfig.put("t_test_1", "include(id,name,width,last_upd_time)"); + final List expectedResult = List.of(new Rule("t_test_1", "id,name,width,last_upd_time", "include")); + + // Run the test + final List result = columnRuleParser.parse(columnRuleConfig); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + void testParse_include_default_include() { + // Setup + final Map columnRuleConfig = new HashMap<>(); + columnRuleConfig.put("t_test_2", "id,width,name,last_upd_time"); + final List expectedResult = List.of(new Rule("t_test_2", "id,width,name,last_upd_time", "include")); + + // Run the test + final List result = columnRuleParser.parse(columnRuleConfig); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + void testParse_only_exclude() { + // Setup + final Map columnRuleConfig = new HashMap<>(); + columnRuleConfig.put("t_test_3", "exclude(name,height,last_upd_time)"); + final List expectedResult = List.of(new Rule("t_test_3", "name,height,last_upd_time", "exclude")); + + // Run the test + final List result = columnRuleParser.parse(columnRuleConfig); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + void testParse_has_include_exclude() { + // Setup + final Map columnRuleConfig = new HashMap<>(); + columnRuleConfig.put("t_test_4", "include(id,portal_id,func_id,name,width),exclude(name,height,last_upd_time)"); + final List expectedResult = new ArrayList<>(); + + // Run the test + final List result = columnRuleParser.parse(columnRuleConfig); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + void testParse_has_default_include_exclude() { + // Setup + final Map columnRuleConfig = new HashMap<>(); + columnRuleConfig.put("t_test_5", "id,portal_id,func_id,name,width,exclude(name,height,last_upd_time)"); + final List expectedResult = new ArrayList<>(); + + // Run the test + final List result = columnRuleParser.parse(columnRuleConfig); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + void testParse_has_default_include_include() { + // Setup + final Map columnRuleConfig = new HashMap<>(); + columnRuleConfig.put("t_test_5", "id,portal_id,func_id,name,width,include(name,height,last_upd_time)"); + final List expectedResult = new ArrayList<>(); + + // Run the test + final List result = columnRuleParser.parse(columnRuleConfig); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } +} diff --git a/datachecker-check/src/test/java/org/opengauss/datachecker/check/modules/rule/RuleParserTest.java b/datachecker-check/src/test/java/org/opengauss/datachecker/check/modules/rule/RuleParserTest.java new file mode 100644 index 0000000000000000000000000000000000000000..65f95a714752d6f0e44214f52adb3be184a05075 --- /dev/null +++ b/datachecker-check/src/test/java/org/opengauss/datachecker/check/modules/rule/RuleParserTest.java @@ -0,0 +1,204 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.check.modules.rule; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.opengauss.datachecker.check.config.RuleConfig; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.RuleType; + +import java.util.List; +import java.util.Map; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * RuleParserTest + * + * @author ：wangchao + * @date ：Created in 2022/10/31 + * @since ：11 + */ +public class RuleParserTest { + + private RuleParser ruleParserUnderTest; + + @BeforeEach + void setUp() { + ruleParserUnderTest = new RuleParser(); + } + + @Test + @DisplayName("test rule parse enable false") + void testParser_enable_false() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(false); + config.setTable(Map.ofEntries(Map.entry("black", "value"))); + config.setRow(Map.ofEntries(Map.entry("value", "value"))); + config.setColumn(Map.ofEntries(Map.entry("value", "value"))); + + final Map> expectedResult = Map.ofEntries(); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse enable true") + void testParser_enable_true() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("black", "value"))); + + final Rule rule = new Rule("black", "value"); + final Map> expectedResult = Map.ofEntries(Map.entry(RuleType.TABLE, List.of(rule))); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse black white all has value") + void testParser_black_white_all() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("black", "blackValue"), Map.entry("white", "whiteValue"))); + + final Rule white = new Rule("white", "whiteValue"); + final Map> expectedResult = Map.ofEntries(Map.entry(RuleType.TABLE, List.of(white))); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse black white but white has no value") + void testParser_black_white_all_empty_white() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("black", "blackValue"), Map.entry("white", ""))); + + final Rule black = new Rule("black", "blackValue"); + final Map> expectedResult = Map.ofEntries(Map.entry(RuleType.TABLE, List.of(black))); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse black white but black has no value") + void testParser_black_white_all_empty_black() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("black", ""), Map.entry("white", "whiteValue"))); + + final Rule white = new Rule("white", "whiteValue"); + final Map> expectedResult = Map.ofEntries(Map.entry(RuleType.TABLE, List.of(white))); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse black white, and only has black") + void testParser_black_only_black() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("black", "blackValue"))); + + final Rule black = new Rule("black", "blackValue"); + final Map> expectedResult = Map.ofEntries(Map.entry(RuleType.TABLE, List.of(black))); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse black white, and only has black ,but black value is empty") + void testParser_black_only_black_and_empty_value() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("black", ""))); + final Map> expectedResult = Map.ofEntries(); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse black white, but only has white") + void testParser_black_only_white() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("white", "whiteValue"))); + + final Rule white = new Rule("white", "whiteValue"); + final Map> expectedResult = Map.ofEntries(Map.entry(RuleType.TABLE, List.of(white))); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } + + @Test + @DisplayName("test rule parse black white, and only has white ,but white value is empty") + void testParser_black_only_white_and_empty_value() { + // Setup + final RuleConfig config = new RuleConfig(); + config.setEnable(true); + config.setTable(Map.ofEntries(Map.entry("white", ""))); + final Map> expectedResult = Map.ofEntries(); + + // Run the test + final Map> result = ruleParserUnderTest.parser(config); + + // Verify the results + assertThat(result).isEqualTo(expectedResult); + } +} diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/constant/RuleConstants.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/constant/RuleConstants.java new file mode 100644 index 0000000000000000000000000000000000000000..9a404bac8c87b36227bb42da808a2279b1d7cdd3 --- /dev/null +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/constant/RuleConstants.java @@ -0,0 +1,34 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.common.constant; + +/** + * RuleConstants + * + * @author ：wangchao + * @date ：Created in 2022/5/24 + * @since ：11 + */ +public interface RuleConstants { + String RULE_WHITE = "white"; + String RULE_BLACK = "black"; + String RULE_INCLUDE = "include"; + String RULE_EXCLUDE = "exclude"; + String RULE_LEFT_BRACE = "("; + String RULE_RIGHT_BRACE = ")"; + String RULE_SPLIT = ","; + String RULE_REGEX = "regex"; +} diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/common/Rule.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/common/Rule.java new file mode 100644 index 0000000000000000000000000000000000000000..8e74cf6fb7a1f903a152c8e55cf33bcd41d771f4 --- /dev/null +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/common/Rule.java @@ -0,0 +1,46 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.common.entry.common; + +import lombok.Data; + +/** + * Rule + * + * @author ：wangchao + * @date ：Created in 2022/11/30 + * @since ：11 + */ +@Data +public class Rule { + private String name; + private String text; + private String attribute; + + public Rule() { + } + + public Rule(String name, String text) { + this.name = name; + this.text = text; + } + + public Rule(String name, String text, String attribute) { + this.name = name; + this.text = text; + this.attribute = attribute; + } +} diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/enums/DataBaseMeta.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/enums/DataBaseMeta.java index 8a0acdf2e33db2e030a26508ec5ce7574a3290cb..676d48ad9d3f51589146fa2b2dc7e96472cc01db 100644 --- a/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/enums/DataBaseMeta.java +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/enums/DataBaseMeta.java @@ -30,6 +30,10 @@ public enum DataBaseMeta implements IEnum { * DataBaseHealth */ HEALTH("HealthMeta", "DataBaseHealth"), + /** + * COUNT + */ + COUNT("COUNT", "COUNT"), /** * TableMetaData */ diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/enums/RuleType.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/enums/RuleType.java new file mode 100644 index 0000000000000000000000000000000000000000..6eb57de7b4d377d9d675b5da1753378c0949e35c --- /dev/null +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/enums/RuleType.java @@ -0,0 +1,39 @@ +package org.opengauss.datachecker.common.entry.enums; + +import lombok.Getter; + +/** + * RuleType + * + * @author ：wangchao + * @date ：Created in 2022/11/30 + * @since ：11 + */ +@Getter +public enum RuleType implements IEnum { + /** + * table + */ + TABLE("table"), + /** + * Sink + */ + ROW("row"), + /** + * Sink + */ + COLUMN("column"); + + private final String code; + private String description; + + RuleType(String code) { + this.code = code; + } + + /** + * DataSourceType api description + */ + public static final String API_DESCRIPTION = "RuleType [TABLE ,ROW, COLUMN]"; + +} diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/extract/ConditionLimit.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/extract/ConditionLimit.java new file mode 100644 index 0000000000000000000000000000000000000000..5fceaf39814273e52c37e76b26f5040a9a938bfc --- /dev/null +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/extract/ConditionLimit.java @@ -0,0 +1,36 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.common.entry.extract; + +import lombok.Data; + +/** + * ConditionLimit + * + * @author ：wangchao + * @date ：Created in 2022/12/5 + * @since ：11 + */ +@Data +public class ConditionLimit { + private int start; + private long offset; + + public ConditionLimit(int start, long offset) { + this.start = start; + this.offset = offset; + } +} diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/extract/TableMetadata.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/extract/TableMetadata.java index aa27e3919cd8323ab1a09d2447c271dd46662543..22284cb5b18df001f585702f2662f138db8214b3 100644 --- a/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/extract/TableMetadata.java +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/entry/extract/TableMetadata.java @@ -53,4 +53,6 @@ public class TableMetadata { * Table column properties */ private List columnsMetas; + + private ConditionLimit conditionLimit; } diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/thread/ThreadPoolFactory.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/thread/ThreadPoolFactory.java index 5174f828d53827d28375f2b02953f628e4260564..d64ea3bcb9271d98ebc2ef904d519cc2480f7fd3 100644 --- a/datachecker-common/src/main/java/org/opengauss/datachecker/common/thread/ThreadPoolFactory.java +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/thread/ThreadPoolFactory.java @@ -40,7 +40,7 @@ public class ThreadPoolFactory { private static final double IO_WAIT_TIME = 4.0; private static final double CPU_TIME = 1.0; private static final double POOL_QUEUE_EXPANSION_RATIO = 1.2; - private static final double CHECK_POOL_QUEUE_EXPANSION_RATIO = 0.5; + private static final double CHECK_POOL_QUEUE_EXPANSION_RATIO = 1; private static final double CORE_POOL_SIZE_RATIO = 2.0; /** @@ -79,11 +79,8 @@ public class ThreadPoolFactory { } private static ExecutorService createThreadPool(String threadName, int corePoolSize, int threadNum, int queueSize) { - - log.info("Thread name is {}, corePoolSize is : {}, size is {}, queueSize is {}", threadName, corePoolSize, - threadNum, queueSize); + log.debug("ThreadPool corePoolSize is : {}, size is {}, queueSize is {}", corePoolSize, threadNum, queueSize); BlockingQueue blockingQueue = new LinkedBlockingQueue<>(queueSize); - ThreadPoolExecutor threadPoolExecutor = new ThreadPoolExecutor(corePoolSize, threadNum, 60L, TimeUnit.SECONDS, blockingQueue, new CheckThreadFactory("check", threadName, false), new DiscardOldestPolicy(log, threadName)); diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TaskUtil.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TaskUtil.java index 591395116ea98ee156a1607a91790c22bcf6b166..cc229b42c08f6032d9f0e64588004666efd1d379 100644 --- a/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TaskUtil.java +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TaskUtil.java @@ -23,7 +23,7 @@ package org.opengauss.datachecker.common.util; * @since ：11 */ public class TaskUtil { - public static final int EXTRACT_MAX_ROW_COUNT = 10000; + public static final int EXTRACT_MAX_ROW_COUNT = 100000; /** * Calculate the number of segmented tasks according to the total number recorded in the table diff --git a/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TopicUtil.java b/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TopicUtil.java index 9108b55ce3662b1c9730863f82e964732c7ea3cd..2994445d916150dc8f672b41c5c4c43057d7abef 100644 --- a/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TopicUtil.java +++ b/datachecker-common/src/main/java/org/opengauss/datachecker/common/util/TopicUtil.java @@ -25,7 +25,7 @@ import org.opengauss.datachecker.common.entry.enums.Endpoint; * @since ：11 */ public class TopicUtil { - public static final int TOPIC_MAX_PARTITIONS = 16; + public static final int TOPIC_MAX_PARTITIONS = 4; public static final int TOPIC_MIN_PARTITIONS = 1; private static final String TOPIC_TEMPLATE = "CHECK_%s_%s_%s"; diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/AdapterContext.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/AdapterContext.java new file mode 100644 index 0000000000000000000000000000000000000000..cb00932606232b44129f072975c6cc6847042a90 --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/AdapterContext.java @@ -0,0 +1,29 @@ +package org.opengauss.datachecker.extract.adapter; + +import org.opengauss.datachecker.common.entry.enums.DataBaseType; +import org.opengauss.datachecker.extract.adapter.service.CheckRowRule; +import org.opengauss.datachecker.extract.util.SpringUtil; + +import java.util.HashMap; +import java.util.Map; + +/** + * AdapterService + * + * @author ：wangchao + * @date ：Created in 2022/12/2 + * @since ：11 + */ +public class AdapterContext { + private static final Map, String>> CONTEXT = new HashMap<>(); + + static { + CONTEXT.put(DataBaseType.MS, Map.ofEntries(Map.entry(CheckRowRule.class, "mysqlCheckRowRule"))); + CONTEXT.put(DataBaseType.OG, Map.ofEntries(Map.entry(CheckRowRule.class, "openGaussCheckRowRule"))); + } + + public static T getBean(DataBaseType databaseType, Class classz) { + final String beanName = CONTEXT.get(databaseType).get(classz); + return SpringUtil.getBean(beanName, classz); + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/CheckRowRule.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/CheckRowRule.java new file mode 100644 index 0000000000000000000000000000000000000000..6ae67659e1e0a767e985acbb1f75c64037504619 --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/CheckRowRule.java @@ -0,0 +1,38 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.adapter.service; + +import org.opengauss.datachecker.common.entry.common.Rule; + +/** + * CheckRowRule + * + * @author ：wangchao + * @date ：Created in 2022/12/2 + * @since ：11 + */ +public interface CheckRowRule { + String CHECK_RULE_STATEMENT = "select count(1) from %s.%s where %s limit 1; "; + + /** + * Check whether the filter rule of table rows is valid + * + * @param schema schema + * @param rule rule + * @return true|false + */ + boolean checkRule(String schema, Rule rule); +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/CheckRowRuleService.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/CheckRowRuleService.java new file mode 100644 index 0000000000000000000000000000000000000000..60f937dee1c62bc85f44a825c7d9c04e39980394 --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/CheckRowRuleService.java @@ -0,0 +1,68 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.adapter.service; + +import lombok.extern.slf4j.Slf4j; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.springframework.dao.DataAccessException; +import org.springframework.jdbc.core.JdbcTemplate; + +import javax.annotation.Resource; + +/** + * CheckRowRuleService + * + * @author ：wangchao + * @date ：Created in 2022/12/2 + * @since ：11 + */ +@Slf4j +public abstract class CheckRowRuleService implements CheckRowRule { + @Resource + private JdbcTemplate jdbcTemplate; + + @Override + public boolean checkRule(String schema, Rule rule) { + boolean result = false; + try { + String checkStatement = String.format(CHECK_RULE_STATEMENT, convert(schema), convert(rule.getName()), + convertCondition(rule.getText())); + jdbcTemplate.execute(checkStatement); + result = true; + } catch (DataAccessException ex) { + log.error("rules of row is invalid!,schema={},rule={}", schema, rule); + } + return result; + } + + /** + * convert text + * + * @param text text + * @return text + */ + protected abstract String convert(String text); + + /** + * Row level rule condition semantics support ( > < >= <= = !=) Six types of conditional filtering + * and compound conditional statements; + * Composite conditional statements must be spliced with and, for example, a>1 and b>2; + * + * @param text text + * @return condition + */ + protected abstract String convertCondition(String text); +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/mysql/MysqlCheckRowRule.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/mysql/MysqlCheckRowRule.java new file mode 100644 index 0000000000000000000000000000000000000000..f9a2fe52023b0fa835b8c0e6c34d5b1c068c1c2d --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/mysql/MysqlCheckRowRule.java @@ -0,0 +1,27 @@ +package org.opengauss.datachecker.extract.adapter.service.mysql; + +import lombok.extern.slf4j.Slf4j; +import org.opengauss.datachecker.extract.adapter.service.CheckRowRuleService; +import org.springframework.stereotype.Service; + +/** + * MysqlCheckRowRule + * + * @author ：wangchao + * @date ：Created in 2022/12/2 + * @since ：11 + */ +@Slf4j +@Service +public class MysqlCheckRowRule extends CheckRowRuleService { + + @Override + protected String convert(String text) { + return "`" + text + "`"; + } + + @Override + protected String convertCondition(String text) { + return text; + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/opengauss/OpenGaussCheckRowRule.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/opengauss/OpenGaussCheckRowRule.java new file mode 100644 index 0000000000000000000000000000000000000000..fa3803a17a646cbb56e6800489c6dae85762bcc9 --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/adapter/service/opengauss/OpenGaussCheckRowRule.java @@ -0,0 +1,27 @@ +package org.opengauss.datachecker.extract.adapter.service.opengauss; + +import lombok.extern.slf4j.Slf4j; +import org.opengauss.datachecker.extract.adapter.service.CheckRowRuleService; +import org.springframework.stereotype.Service; + +/** + * OpenGaussCheckRowRule + * + * @author ：wangchao + * @date ：Created in 2022/12/2 + * @since ：11 + */ +@Slf4j +@Service +public class OpenGaussCheckRowRule extends CheckRowRuleService { + + @Override + protected String convert(String text) { + return "\"" + text + "\""; + } + + @Override + protected String convertCondition(String text) { + return text; + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/cache/MetaDataCache.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/cache/MetaDataCache.java index 84d22d8b96cf1c97dee546d677abf345b26c6e54..f40da535d92d8dfd11ab55233cc95bde3d730f07 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/cache/MetaDataCache.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/cache/MetaDataCache.java @@ -46,27 +46,24 @@ public class MetaDataCache { /** * Initializing the Metadata Cache Method */ - public static void initCache() { - if (CACHE == null) { - CACHE = CacheBuilder.newBuilder() - //Set the concurrent read/write level based on the number of CPU cores; - .concurrencyLevel(1) - // Size of the buffer pool - .maximumSize(Integer.MAX_VALUE) - // Removing a Listener - .removalListener((RemovalListener) remove -> log - .debug("cache: [{}], removed", remove.getKey())).recordStats().build( - // Method of handing a Key that does not exist - new CacheLoader<>() { - @Override - public TableMetadata load(String tableName) { - log.info("cache: [{}], does not exist", tableName); - MetaDataService metaDataService = SpringUtil.getBean(MetaDataService.class); - return metaDataService.queryMetaDataOfSchema(tableName); - } - }); - } - log.info("initialize table meta data cache"); + static { + CACHE = CacheBuilder.newBuilder() + //Set the concurrent read/write level based on the number of CPU cores; + .concurrencyLevel(1) + // Size of the buffer pool + .maximumSize(Integer.MAX_VALUE) + // Removing a Listener + .removalListener((RemovalListener) remove -> log + .debug("cache: [{}], removed", remove.getKey())).recordStats().build( + // Method of handing a Key that does not exist + new CacheLoader<>() { + @Override + public TableMetadata load(String tableName) { + log.info("cache: [{}], does not exist", tableName); + MetaDataService metaDataService = SpringUtil.getBean(MetaDataService.class); + return metaDataService.queryMetaDataOfSchema(tableName); + } + }); } /** @@ -85,9 +82,6 @@ public class MetaDataCache { public static Map getAll() { try { - if (CACHE == null) { - initCache(); - } return CACHE.asMap(); } catch (Exception exception) { log.error("put in cache exception ", exception); diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractCleanController.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractCleanController.java index b3aa492d51fa1ed96f32c838e1983aeb6edb8081..69fc5684fb4105b0e0bdaf286606c35878edd57c 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractCleanController.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractCleanController.java @@ -54,7 +54,6 @@ public class ExtractCleanController { @Operation(summary = "clear the endpoint information and reinitialize the environment") @PostMapping("/extract/clean/environment") Result cleanEnvironment(@RequestParam(name = "processNo") String processNo) { - metaDataService.init(); dataExtractService.cleanBuildTask(); kafkaManagerService.cleanKafka(processNo); return Result.success(); diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractController.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractController.java index f22412fba0c1aa294bab940622f079b4cd5d4916..b3ca64d986550df8fe2f55f02b885c252b3271ec 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractController.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractController.java @@ -70,18 +70,6 @@ public class ExtractController { return Result.success(metaDataMap); } - /** - * refreshing the block list and trust list - * - * @param mode {@value CheckBlackWhiteMode#API_DESCRIPTION} - * @param tableList tableList - */ - @Operation(summary = "refreshing the block list and trust list") - @PostMapping("/extract/refresh/black/white/list") - void refreshBlackWhiteList(@RequestParam CheckBlackWhiteMode mode, @RequestBody List tableList) { - metaDataService.refreshBlackWhiteList(mode, tableList); - } - /** * source endpoint extraction task construction * @@ -106,7 +94,7 @@ public class ExtractController { * new verification process cannot be enabled. */ @Operation(summary = "sink endpoint task configuration") - @PostMapping("/extract/config/sink/task/all") + @PostMapping("/extract/sink/task/all") Result buildExtractTaskAllTables( @Parameter(name = "processNo", description = "execution process no") @RequestParam(name = "processNo") String processNo, @RequestBody List taskList) { diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractRuleController.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractRuleController.java new file mode 100644 index 0000000000000000000000000000000000000000..a533b9f0b2a7621ed7afee3f80c6d44283487389 --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/controller/ExtractRuleController.java @@ -0,0 +1,57 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.controller; + +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.CheckMode; +import org.opengauss.datachecker.common.entry.enums.RuleType; +import org.opengauss.datachecker.extract.load.EnvironmentLoader; +import org.opengauss.datachecker.extract.service.RuleAdapterService; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.RequestBody; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; + +import javax.annotation.Resource; +import java.util.List; +import java.util.Map; + +/** + * Filter Rules service + * + * @author ：wangchao + * @date ：Created in 2022/6/23 + * @since ：11 + */ +@RestController +public class ExtractRuleController { + @Resource + private RuleAdapterService ruleAdapterService; + @Resource + private EnvironmentLoader environmentLoader; + + /** + * Distribution Data Extraction Filter Rules + * + * @param rules rules + */ + @PostMapping("/extract/rules/distribute") + public void distributeRules(@RequestParam(name = "checkMode") CheckMode checkMode, + @RequestBody Map> rules) { + ruleAdapterService.init(rules); + environmentLoader.load(checkMode); + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/DataBaseMetaDataDAOImpl.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/DataBaseMetaDataDAOImpl.java index 9bdc6fea4187c9e0d4eb91f80b9af8caeb1d289f..f3a5ab788d442b343302510f16a6db6ec2e9b5ba 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/DataBaseMetaDataDAOImpl.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/DataBaseMetaDataDAOImpl.java @@ -15,25 +15,29 @@ package org.opengauss.datachecker.extract.dao; -import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.opengauss.datachecker.common.constant.Constants; -import org.opengauss.datachecker.common.entry.enums.CheckBlackWhiteMode; +import org.opengauss.datachecker.common.entry.enums.CheckMode; import org.opengauss.datachecker.common.entry.enums.ColumnKey; import org.opengauss.datachecker.common.entry.enums.DataBaseMeta; import org.opengauss.datachecker.common.entry.enums.DataBaseType; import org.opengauss.datachecker.common.entry.extract.ColumnsMetaData; +import org.opengauss.datachecker.common.entry.extract.ConditionLimit; import org.opengauss.datachecker.common.entry.extract.MetadataLoadProcess; import org.opengauss.datachecker.common.entry.extract.TableMetadata; import org.opengauss.datachecker.common.util.EnumUtil; import org.opengauss.datachecker.extract.config.ExtractProperties; +import org.opengauss.datachecker.extract.load.ExtractEnvironment; +import org.opengauss.datachecker.extract.service.RuleAdapterService; import org.springframework.jdbc.core.JdbcTemplate; +import org.springframework.jdbc.core.PreparedStatementSetter; import org.springframework.jdbc.core.RowCountCallbackHandler; import org.springframework.jdbc.core.RowMapper; import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate; import org.springframework.stereotype.Component; -import org.springframework.util.CollectionUtils; +import javax.annotation.PostConstruct; +import javax.annotation.Resource; import java.sql.ResultSet; import java.sql.SQLException; import java.util.ArrayList; @@ -41,8 +45,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Objects; -import java.util.concurrent.atomic.AtomicReference; -import java.util.stream.Collectors; +import java.util.Optional; import static org.opengauss.datachecker.extract.constants.ExtConstants.COLUMN_INDEX_FIRST_ZERO; @@ -55,134 +58,100 @@ import static org.opengauss.datachecker.extract.constants.ExtConstants.COLUMN_IN */ @Slf4j @Component -@RequiredArgsConstructor public class DataBaseMetaDataDAOImpl implements MetaDataDAO { + @Resource + private RuleAdapterService ruleAdapterService; + @Resource + protected JdbcTemplate JdbcTemplateOne; + @Resource + private ExtractProperties extractProperties; + @Resource + private ExtractEnvironment extractEnvironment; - private static final AtomicReference MODE_REF = - new AtomicReference<>(CheckBlackWhiteMode.NONE); - private static final AtomicReference> WHITE_REF = new AtomicReference<>(); - private static final AtomicReference> BLACK_REF = new AtomicReference<>(); - - protected final JdbcTemplate JdbcTemplateOne; - - private final ExtractProperties extractProperties; private volatile MetadataLoadProcess metadataLoadProcess = new MetadataLoadProcess(); + private volatile DataBaseType dataBaseType; + private volatile String schema; + + @PostConstruct + public void initDataBaseType() { + this.dataBaseType = extractProperties.getDatabaseType(); + this.schema = extractProperties.getSchema(); + } @Override public boolean health() { String sql = MetaSqlMapper.getMetaSql(extractProperties.getDatabaseType(), DataBaseMeta.HEALTH); List result = new ArrayList<>(); - JdbcTemplateOne.query(sql, ps -> ps.setString(1, getSchema()), new RowCountCallbackHandler() { - @Override - protected void processRow(ResultSet rs, int rowNum) throws SQLException { - result.add(rs.getString(1)); - } - }); + JdbcTemplateOne + .query(sql, (PreparedStatementSetter) ps -> ps.setString(1, schema), new RowCountCallbackHandler() { + @Override + protected void processRow(ResultSet rs, int rowNum) throws SQLException { + result.add(rs.getString(1)); + } + }); return result.size() > 0; } - @Override - public void resetBlackWhite(CheckBlackWhiteMode mode, List tableList) { - MODE_REF.set(mode); - if (Objects.equals(mode, CheckBlackWhiteMode.WHITE)) { - WHITE_REF.set(tableList); - } else if (Objects.equals(mode, CheckBlackWhiteMode.BLACK)) { - BLACK_REF.set(tableList); - } else { - WHITE_REF.getAcquire().clear(); - BLACK_REF.getAcquire().clear(); - } - } - @Override public List queryTableMetadata() { final List tableNameList = new ArrayList<>(); - String sql = MetaSqlMapper.getMetaSql(extractProperties.getDatabaseType(), DataBaseMeta.TABLE); - JdbcTemplateOne.query(sql, ps -> ps.setString(1, getSchema()), new RowCountCallbackHandler() { - @Override - protected void processRow(ResultSet rs, int rowNum) throws SQLException { - tableNameList.add(rs.getString(1)); - } - }); + String sql = MetaSqlMapper.getMetaSql(dataBaseType, DataBaseMeta.TABLE); + JdbcTemplateOne + .query(sql, (PreparedStatementSetter) ps -> ps.setString(1, schema), new RowCountCallbackHandler() { + @Override + protected void processRow(ResultSet rs, int rowNum) throws SQLException { + tableNameList.add(rs.getString(1)); + } + }); return getAllTableCount(filterTableListByBlackWhite(tableNameList)); } private List filterTableListByBlackWhite(List tableNameList) { - if (Objects.equals(MODE_REF.get(), CheckBlackWhiteMode.WHITE)) { - final List whiteList = WHITE_REF.get(); - if (CollectionUtils.isEmpty(whiteList)) { - return tableNameList; - } - return tableNameList.stream().filter(whiteList::contains).collect(Collectors.toList()); - } else if (Objects.equals(MODE_REF.get(), CheckBlackWhiteMode.BLACK)) { - final List blackList = BLACK_REF.get(); - if (CollectionUtils.isEmpty(blackList)) { - return tableNameList; - } - return tableNameList.stream().filter(table -> !blackList.contains(table)).collect(Collectors.toList()); - } else { - return tableNameList; - } + return ruleAdapterService.executeTableRule(tableNameList); } - private List filterBlackWhiteList(List tableMetaList) { - if (Objects.equals(MODE_REF.get(), CheckBlackWhiteMode.WHITE)) { - final List whiteList = WHITE_REF.get(); - if (CollectionUtils.isEmpty(whiteList)) { - return tableMetaList; - } - return tableMetaList.stream().filter(table -> whiteList.contains(table.getTableName())) - .collect(Collectors.toList()); - } else if (Objects.equals(MODE_REF.get(), CheckBlackWhiteMode.BLACK)) { - final List blackList = BLACK_REF.get(); - if (CollectionUtils.isEmpty(blackList)) { - return tableMetaList; - } - return tableMetaList.stream().filter(table -> !blackList.contains(table.getTableName())) - .collect(Collectors.toList()); + private List getAllTableCount(List tableNameList) { + final List tableMetadataList = new ArrayList<>(); + metadataLoadProcess.setTotal(tableNameList.size()); + final Map conditionMap = ruleAdapterService.executeRowRule(); + if (isNoTableRowCount()) { + tableNameList.forEach(tableName -> { + TableMetadata tableMetadata = constructTableMetadataObject(tableName); + tableMetadataList.add(tableMetadata); + metadataLoadProcess.setLoadCount(tableMetadataList.size()); + }); } else { - return tableMetaList; + tableNameList.forEach(tableName -> { + TableMetadata tableMetadata = constructTableMetadataObject(conditionMap, tableName); + tableMetadataList.add(tableMetadata); + metadataLoadProcess.setLoadCount(tableMetadataList.size()); + }); } + return tableMetadataList; } - @Override - public List queryTableMetadataFast() { - List tableMetadata = new ArrayList<>(); - String sql = MetaSqlMapper.getMetaSql(extractProperties.getDatabaseType(), DataBaseMeta.TABLE); - JdbcTemplateOne.query(sql, ps -> ps.setString(1, getSchema()), new RowCountCallbackHandler() { - @Override - protected void processRow(ResultSet rs, int rowNum) throws SQLException { - final TableMetadata metadata = - new TableMetadata().setTableName(rs.getString(1)).setTableRows(rs.getLong(2)); - log.debug("queryTableMetadataFast {}", metadata.toString()); - tableMetadata.add(metadata); - } - }); - return filterBlackWhiteList(tableMetadata); + private boolean isNoTableRowCount() { + final CheckMode checkMode = extractEnvironment.getCheckMode(); + return Objects.equals(checkMode, CheckMode.INCREMENT); } - private List getAllTableCount(List tableNameList) { - final List tableMetadata = new ArrayList<>(); - String sqlQueryTableRowCount = MetaSqlMapper.getTableCount(); - final String schema = getSchema(); - final Boolean isConvertTableName = isOpenGauss(); - metadataLoadProcess.setTotal(tableNameList.size()); - tableNameList.forEach(tableName -> { - final Long rowCount = JdbcTemplateOne.queryForObject( - String.format(sqlQueryTableRowCount, schema, isConvertTableName ? convert(tableName) : tableName), - Long.class); - tableMetadata.add(new TableMetadata().setTableName(tableName).setTableRows(rowCount)); - metadataLoadProcess.setLoadCount(tableMetadata.size()); - }); - return tableMetadata; + private TableMetadata constructTableMetadataObject(String tableName) { + return new TableMetadata().setTableName(tableName).setTableRows(0); } - private Boolean isOpenGauss() { - return Objects.equals(extractProperties.getDatabaseType(), DataBaseType.OG); - } - - private String convert(String tableName) { - return "\"" + tableName + "\""; + private TableMetadata constructTableMetadataObject(Map conditionMap, String tableName) { + TableMetadata tableMetadata = new TableMetadata(); + if (conditionMap.containsKey(tableName)) { + final ConditionLimit conditionLimit = conditionMap.get(tableName); + tableMetadata.setTableName(tableName).setTableRows(conditionLimit.getOffset()) + .setConditionLimit(conditionLimit); + } else { + String sqlQueryTableRowCount = MetaSqlMapper.getTableCount(dataBaseType); + final Long rowCount = + JdbcTemplateOne.queryForObject(String.format(sqlQueryTableRowCount, schema, tableName), Long.class); + tableMetadata.setTableName(tableName).setTableRows(Optional.ofNullable(rowCount).orElse(0L)); + } + return tableMetadata; } @Override @@ -199,33 +168,23 @@ public class DataBaseMetaDataDAOImpl implements MetaDataDAO { public List queryColumnMetadata(List tableNames) { Map map = new HashMap<>(Constants.InitialCapacity.EMPTY); map.put("tableNames", tableNames); - map.put("databaseSchema", getSchema()); + map.put("databaseSchema", schema); NamedParameterJdbcTemplate jdbc = new NamedParameterJdbcTemplate(JdbcTemplateOne); - String sql = MetaSqlMapper.getMetaSql(extractProperties.getDatabaseType(), DataBaseMeta.COLUMN); - return jdbc.query(sql, map, new RowMapper<>() { + String sql = MetaSqlMapper.getMetaSql(dataBaseType, DataBaseMeta.COLUMN); + List columns = jdbc.query(sql, map, new RowMapper<>() { int columnIndex = COLUMN_INDEX_FIRST_ZERO; @Override public ColumnsMetaData mapRow(ResultSet rs, int rowNum) throws SQLException { - ColumnsMetaData columnsMetaData = new ColumnsMetaData().setTableName(rs.getString(++columnIndex)) - .setColumnName(rs.getString(++columnIndex)) - .setOrdinalPosition(rs.getInt(++columnIndex)) - .setDataType(rs.getString(++columnIndex)) - .setColumnType(rs.getString(++columnIndex)) - .setColumnKey(EnumUtil.valueOf(ColumnKey.class, - rs.getString(++columnIndex))); + ColumnsMetaData columnsMetaData = new ColumnsMetaData(); + columnsMetaData.setTableName(rs.getString(++columnIndex)).setColumnName(rs.getString(++columnIndex)) + .setOrdinalPosition(rs.getInt(++columnIndex)).setDataType(rs.getString(++columnIndex)) + .setColumnType(rs.getString(++columnIndex)) + .setColumnKey(EnumUtil.valueOf(ColumnKey.class, rs.getString(++columnIndex))); columnIndex = COLUMN_INDEX_FIRST_ZERO; return columnsMetaData; } }); - } - - /** - * Dynamically obtain the schema information of the current data source - * - * @return database schema - */ - private String getSchema() { - return extractProperties.getSchema(); + return ruleAdapterService.executeColumnRule(columns); } } diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaDataDAO.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaDataDAO.java index 7d45f493e37b604330327ac7249823db4836d960..50839600aec9c79af15207f8e141b21d035ae6e2 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaDataDAO.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaDataDAO.java @@ -15,7 +15,6 @@ package org.opengauss.datachecker.extract.dao; -import org.opengauss.datachecker.common.entry.enums.CheckBlackWhiteMode; import org.opengauss.datachecker.common.entry.extract.ColumnsMetaData; import org.opengauss.datachecker.common.entry.extract.MetadataLoadProcess; import org.opengauss.datachecker.common.entry.extract.TableMetadata; @@ -44,28 +43,13 @@ public interface MetaDataDAO { */ MetadataLoadProcess getMetadataLoadProcess(); - /** - * Reset black and white list - * - * @param mode Black and white list mode{@link CheckBlackWhiteMode} - * @param tableList tableList - */ - void resetBlackWhite(CheckBlackWhiteMode mode, List tableList); - - /** + /** * Query table metadata * * @return table metadata information */ List queryTableMetadata(); - /** - * Quick query table metadata - directly from information_ Schema acquisition - * - * @return table metadata information - */ - List queryTableMetadataFast(); - /** * Query the metadata information of the corresponding column of the table * diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaSqlMapper.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaSqlMapper.java index 52c1355e4f5142405580e74cea08f67435df6164..f145df03f5d487342c68020802df777bdbc5aad2 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaSqlMapper.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/dao/MetaSqlMapper.java @@ -37,16 +37,19 @@ public class MetaSqlMapper { dataBaseMySql.put(DataBaseMeta.TABLE, DataBaseMySql.TABLE_METADATA_SQL); dataBaseMySql.put(DataBaseMeta.COLUMN, DataBaseMySql.TABLES_COLUMN_META_DATA_SQL); dataBaseMySql.put(DataBaseMeta.HEALTH, DataBaseMySql.HEALTH_SQL); + dataBaseMySql.put(DataBaseMeta.COUNT, DataBaseMySql.TABLE_COUNT_SQL); DATABASE_META_MAPPER.put(DataBaseType.MS, dataBaseMySql); Map dataBaseOpenGauss = new HashMap<>(); dataBaseOpenGauss.put(DataBaseMeta.TABLE, DataBaseOpenGauss.TABLE_METADATA_SQL); dataBaseOpenGauss.put(DataBaseMeta.COLUMN, DataBaseOpenGauss.TABLES_COLUMN_META_DATA_SQL); - dataBaseOpenGauss.put(DataBaseMeta.HEALTH, DataBaseMySql.HEALTH_SQL); + dataBaseOpenGauss.put(DataBaseMeta.HEALTH, DataBaseOpenGauss.HEALTH_SQL); + dataBaseOpenGauss.put(DataBaseMeta.COUNT, DataBaseOpenGauss.TABLE_COUNT_SQL); DATABASE_META_MAPPER.put(DataBaseType.OG, dataBaseOpenGauss); Map databaseO = new HashMap<>(); databaseO.put(DataBaseMeta.TABLE, DataBaseO.TABLE_METADATA_SQL); databaseO.put(DataBaseMeta.COLUMN, DataBaseO.TABLES_COLUMN_META_DATA_SQL); - databaseO.put(DataBaseMeta.HEALTH, DataBaseMySql.HEALTH_SQL); + databaseO.put(DataBaseMeta.HEALTH, DataBaseO.HEALTH_SQL); + databaseO.put(DataBaseMeta.COUNT, DataBaseO.TABLE_COUNT_SQL); DATABASE_META_MAPPER.put(DataBaseType.O, databaseO); } @@ -55,8 +58,8 @@ public class MetaSqlMapper { * * @return table row count sql */ - public static String getTableCount() { - return "select count(1) rowCount from %s.%s"; + public static String getTableCount(DataBaseType databaseType) { + return DATABASE_META_MAPPER.get(databaseType).get(DataBaseMeta.COUNT); } /** @@ -77,6 +80,7 @@ public class MetaSqlMapper { * Health check SQL */ String HEALTH_SQL = "select table_name from information_schema.tables WHERE table_schema=? limit 1"; + String TABLE_COUNT_SQL = " select count(1) rowCount from `%s`.`%s`"; /** * Table metadata query SQL @@ -100,7 +104,7 @@ public class MetaSqlMapper { * Health check SQL */ String HEALTH_SQL = "select table_name from information_schema.tables WHERE table_schema=? limit 1"; - + String TABLE_COUNT_SQL = " select count(1) rowCount from \"%s\".\"%s\""; /** * Table metadata query SQL */ @@ -128,6 +132,7 @@ public class MetaSqlMapper { * Health check SQL */ String HEALTH_SQL = ""; + String TABLE_COUNT_SQL = ""; /** * Table metadata query SQL diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/debe/DataConsolidationServiceImpl.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/debe/DataConsolidationServiceImpl.java index 16bd24cb8510180e535194039fb5e13d4608c63e..1d07b4307be1c1ec03d6e7989e6b282a37098490 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/debe/DataConsolidationServiceImpl.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/debe/DataConsolidationServiceImpl.java @@ -77,7 +77,6 @@ public class DataConsolidationServiceImpl implements DataConsolidationService { @PostConstruct public void initIncrementConfig() { if (extractProperties.isDebeziumEnable()) { - metaDataService.init(); ThreadUtil.newSingleThreadExecutor().submit(new DebeziumWorker(debeziumListener, kafkaConfig)); } } diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/AbstractExtractLoader.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/AbstractExtractLoader.java index 40314b17b51b2e8d6f4e01ef2fc76cce1ba85af2..f20b21e5e6136ce534aa99a9bb63c055a2a81c0c 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/AbstractExtractLoader.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/AbstractExtractLoader.java @@ -16,11 +16,8 @@ package org.opengauss.datachecker.extract.load; import lombok.extern.slf4j.Slf4j; +import org.opengauss.datachecker.extract.util.SpringUtil; import org.springframework.boot.SpringApplication; -import org.springframework.boot.context.event.ApplicationReadyEvent; -import org.springframework.context.ConfigurableApplicationContext; - -import javax.annotation.Resource; /** * AbstractCheckLoader @@ -31,9 +28,6 @@ import javax.annotation.Resource; */ @Slf4j public abstract class AbstractExtractLoader implements ExtractLoader { - private static ConfigurableApplicationContext applicationContext; - @Resource - private ExtractEnvironment extractEnvironment; /** * Verification environment global information loader @@ -41,17 +35,6 @@ public abstract class AbstractExtractLoader implements ExtractLoader { @Override public abstract void load(ExtractEnvironment extractEnvironment); - /** - * Handle an application event. - * - * @param event the event to respond to - */ - @Override - public void onApplicationEvent(ApplicationReadyEvent event) { - applicationContext = event.getApplicationContext(); - load(extractEnvironment); - } - /** * shutdown app * @@ -60,6 +43,6 @@ public abstract class AbstractExtractLoader implements ExtractLoader { public void shutdown(String message) { log.error("The check server will be shutdown , {}", message); log.error("check server exited ."); - System.exit(SpringApplication.exit(applicationContext)); + System.exit(SpringApplication.exit(SpringUtil.getApplicationContext())); } } diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/EnvironmentLoader.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/EnvironmentLoader.java new file mode 100644 index 0000000000000000000000000000000000000000..0767d52bb729ab5e84651ef27a71723caa02f78f --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/EnvironmentLoader.java @@ -0,0 +1,44 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.load; + +import org.opengauss.datachecker.common.entry.enums.CheckMode; +import org.opengauss.datachecker.extract.util.SpringUtil; +import org.springframework.stereotype.Service; + +import javax.annotation.Resource; +import java.util.Map; + +/** + * EnvironmentLoader + * + * @author ：wangchao + * @date ：Created in 2022/12/2 + * @since ：11 + */ +@Service +public class EnvironmentLoader { + @Resource + private ExtractEnvironment extractEnvironment; + + public void load(CheckMode checkMode) { + extractEnvironment.setCheckMode(checkMode); + final Map beans = SpringUtil.getBeans(ExtractLoader.class); + beans.values().forEach(loader -> { + loader.load(extractEnvironment); + }); + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractEnvironment.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractEnvironment.java index 94985cb05d88e3516e43407b711bcc314a9e92b3..fc6f5cdf8f8af76c844abb20d98e29adee10c5ee 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractEnvironment.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractEnvironment.java @@ -15,7 +15,9 @@ package org.opengauss.datachecker.extract.load; +import org.opengauss.datachecker.common.entry.enums.CheckMode; import org.springframework.stereotype.Service; + import java.util.concurrent.ExecutorService; /** @@ -28,12 +30,21 @@ import java.util.concurrent.ExecutorService; @Service public class ExtractEnvironment { private boolean loadSuccess = false; + private CheckMode checkMode; private ExecutorService threadPoolExecutor = null; public void setLoadSuccess(boolean loadSuccess) { this.loadSuccess = loadSuccess; } + public void setCheckMode(CheckMode checkMode) { + this.checkMode = checkMode; + } + + public CheckMode getCheckMode() { + return checkMode; + } + public void setExtractThreadPool(ExecutorService threadPoolExecutor) { this.threadPoolExecutor = threadPoolExecutor; } diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractLoader.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractLoader.java index d2fafc703e14b8e9d52f42e93c8e681892fc61c6..898eba23b330a93f5c530d12c90d6d3256be1d39 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractLoader.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/load/ExtractLoader.java @@ -15,9 +15,6 @@ package org.opengauss.datachecker.extract.load; -import org.springframework.boot.context.event.ApplicationReadyEvent; -import org.springframework.context.ApplicationListener; - /** * ExtractLoader After the Check service is started, load the verification environment information * @@ -25,7 +22,7 @@ import org.springframework.context.ApplicationListener; * @date ：Created in 2022/10/31 * @since ：11 */ -public interface ExtractLoader extends ApplicationListener { +public interface ExtractLoader { /** * Verification environment global information loader * diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/ColumnRuleAdapterService.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/ColumnRuleAdapterService.java new file mode 100644 index 0000000000000000000000000000000000000000..8b205b4ba752ba9b5b925e386e8bc245cc56f4cf --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/ColumnRuleAdapterService.java @@ -0,0 +1,90 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.service; + +import org.apache.commons.collections4.CollectionUtils; +import org.apache.commons.lang3.StringUtils; +import org.opengauss.datachecker.common.constant.RuleConstants; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.ColumnKey; +import org.opengauss.datachecker.common.entry.extract.ColumnsMetaData; +import org.springframework.stereotype.Service; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import java.util.function.Function; +import java.util.stream.Collectors; + +/** + * ColumnRuleAdapterService + * + * @author ：wangchao + * @date ：Created in 2022/12/1 + * @since ：11 + */ +@Service +public class ColumnRuleAdapterService { + private static final String SPLIT = RuleConstants.RULE_SPLIT; + private static final String EXCLUDE = RuleConstants.RULE_EXCLUDE; + + public List executeColumnRule(List rules, List columns) { + if (CollectionUtils.isEmpty(rules)) { + return columns; + } + return filterColumnByRule(columns, rules); + } + + private List filterColumnByRule(List columns, List rules) { + if (CollectionUtils.isEmpty(rules)) { + return columns; + } + Map ruleMap = translateColumnRules(rules); + Map> ruleTextsMap = translateColumnRule(rules); + return columns.parallelStream().filter(metaData -> { + if (!ruleMap.containsKey(metaData.getTableName())) { + return true; + } + if (Objects.equals(metaData.getColumnKey(), ColumnKey.PRI)) { + return true; + } else { + final Rule rulesOfTable = ruleMap.get(metaData.getTableName()); + final List ruleTextsOfTable = ruleTextsMap.get(metaData.getTableName()); + if (StringUtils.equals(rulesOfTable.getAttribute(), EXCLUDE)) { + return !ruleTextsOfTable.contains(metaData.getColumnName().toLowerCase(Locale.ENGLISH)); + } else { + return ruleTextsOfTable.contains(metaData.getColumnName().toLowerCase(Locale.ENGLISH)); + } + } + + }).collect(Collectors.toList()); + } + + private Map translateColumnRules(List rules) { + return rules.stream().collect(Collectors.toMap(Rule::getName, Function.identity())); + } + + private Map> translateColumnRule(List rules) { + Map> ruleMap = new HashMap<>(); + rules.forEach(rule -> { + ruleMap.put(rule.getName(), Arrays.asList(rule.getText().toLowerCase(Locale.ENGLISH).split(SPLIT))); + }); + return ruleMap; + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/DataExtractServiceImpl.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/DataExtractServiceImpl.java index 02a866297b686abe1fda395d1fe7636427970d33..1fb135e5110f8c9d50627f7c709eb23378d013f9 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/DataExtractServiceImpl.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/DataExtractServiceImpl.java @@ -15,6 +15,7 @@ package org.opengauss.datachecker.extract.service; +import com.google.common.collect.Lists; import lombok.extern.slf4j.Slf4j; import org.apache.commons.collections4.CollectionUtils; import org.opengauss.datachecker.common.constant.Constants; @@ -59,7 +60,6 @@ import java.util.Map; import java.util.Objects; import java.util.Set; import java.util.concurrent.ExecutorService; -import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; import java.util.stream.Collectors; @@ -375,19 +375,13 @@ public class DataExtractServiceImpl implements DataExtractService { if (Objects.isNull(metadata)) { throw new TableNotExistException(tableName); } + int partitionSize = compositeKeys.size() / MAX_QUERY_PAGE_SIZE; if (compositeKeys.size() > MAX_QUERY_PAGE_SIZE) { List result = new ArrayList<>(); - AtomicInteger cnt = new AtomicInteger(0); - List tempCompositeKeys = new ArrayList<>(); - compositeKeys.forEach(key -> { - tempCompositeKeys.add(key); - if (cnt.incrementAndGet() == MAX_QUERY_PAGE_SIZE) { - result - .addAll(dataManipulationService.queryColumnHashValues(tableName, tempCompositeKeys, metadata)); - tempCompositeKeys.clear(); - } + final List> partitionList = Lists.partition(compositeKeys, partitionSize); + partitionList.forEach(subList -> { + result.addAll(dataManipulationService.queryColumnHashValues(tableName, subList, metadata)); }); - result.addAll(dataManipulationService.queryColumnHashValues(tableName, tempCompositeKeys, metadata)); return result; } else { return dataManipulationService.queryColumnHashValues(tableName, compositeKeys, metadata); diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/MetaDataService.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/MetaDataService.java index 5b3d2ea92fc99b7d625e0fcca0d3e979b5eb118a..9d9249a5115460cd33db7518bcb28fa0b7005a27 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/MetaDataService.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/MetaDataService.java @@ -29,7 +29,6 @@ import org.springframework.scheduling.annotation.Async; import org.springframework.stereotype.Service; import org.springframework.util.CollectionUtils; -import javax.annotation.PostConstruct; import java.util.List; import java.util.Map; import java.util.function.Function; @@ -47,15 +46,6 @@ import java.util.stream.Collectors; @RequiredArgsConstructor public class MetaDataService { private final DataBaseMetaDataDAOImpl dataBaseMetadataDAOImpl; - private boolean isQueryTableRowCount = true; - - /** - * Metadata cache load - */ - @PostConstruct - public void init() { - MetaDataCache.initCache(); - } /** * Return database metadata information through cache @@ -107,18 +97,6 @@ public class MetaDataService { return tableMetadata.stream().collect(Collectors.toMap(TableMetadata::getTableName, Function.identity())); } - /** - * refresh black or white list - * - * @param mode mode{@value CheckBlackWhiteMode#API_DESCRIPTION } - * @param tableList tableList - */ - public void refreshBlackWhiteList(CheckBlackWhiteMode mode, List tableList) { - dataBaseMetadataDAOImpl.resetBlackWhite(mode, tableList); - init(); - log.info("refresh black or white list ,mode=[{}],list=[{}]", mode.getDescription(), tableList); - } - /** * query table Metadata info * @@ -140,10 +118,6 @@ public class MetaDataService { return MetaDataCache.get(tableName); } - public void updateMetaDataOfSchemaByCache(String tableName) { - MetaDataCache.put(tableName, queryMetaDataOfSchema(tableName)); - } - /** * query column Metadata info * @@ -161,11 +135,7 @@ public class MetaDataService { } private List queryTableMetadata() { - if (isQueryTableRowCount) { - return dataBaseMetadataDAOImpl.queryTableMetadata(); - } else { - return dataBaseMetadataDAOImpl.queryTableMetadataFast(); - } + return dataBaseMetadataDAOImpl.queryTableMetadata(); } private List getTablePrimaryColumn(List columnsMetaData) { diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/RowRuleAdapterService.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/RowRuleAdapterService.java new file mode 100644 index 0000000000000000000000000000000000000000..4ff257ecfd47ad737a656827ce159df3a6c1e321 --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/RowRuleAdapterService.java @@ -0,0 +1,67 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.service; + +import lombok.extern.slf4j.Slf4j; +import org.apache.commons.lang3.math.NumberUtils; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.extract.ConditionLimit; +import org.opengauss.datachecker.extract.adapter.AdapterContext; +import org.opengauss.datachecker.extract.adapter.service.CheckRowRule; +import org.opengauss.datachecker.extract.config.ExtractProperties; +import org.springframework.stereotype.Service; + +import javax.annotation.Resource; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * RowRuleAdapterService + * + * @author ：wangchao + * @date ：Created in 2022/12/1 + * @since ：11 + */ +@Slf4j +@Service +public class RowRuleAdapterService { + + @Resource + private ExtractProperties extractProperties; + + public Map executeRowRule(List rules) { + Map conditionMap = new HashMap<>(); + rules.forEach(rule -> { + conditionMap.put(rule.getName(), getConditionLimit(rule.getText())); + }); + return conditionMap; + } + + private ConditionLimit getConditionLimit(String ruleText) { + ConditionLimit conditionLimit = null; + final String[] condition = ruleText.split(","); + if (NumberUtils.isDigits(condition[0]) && NumberUtils.isDigits(condition[1])) { + conditionLimit = new ConditionLimit(Integer.parseInt(condition[0]), Long.parseLong(condition[1])); + } + return conditionLimit; + } + + private boolean checkRule(Rule rule) { + CheckRowRule checkRowRule = AdapterContext.getBean(extractProperties.getDatabaseType(), CheckRowRule.class); + return checkRowRule.checkRule(extractProperties.getSchema(), rule); + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/RuleAdapterService.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/RuleAdapterService.java new file mode 100644 index 0000000000000000000000000000000000000000..2f21a2bd3482d13554f94d9e1ca23ec4d71c43d7 --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/RuleAdapterService.java @@ -0,0 +1,76 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.service; + +import org.apache.commons.collections4.CollectionUtils; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.opengauss.datachecker.common.entry.enums.RuleType; +import org.opengauss.datachecker.common.entry.extract.ColumnsMetaData; +import org.opengauss.datachecker.common.entry.extract.ConditionLimit; +import org.springframework.stereotype.Service; + +import javax.annotation.Resource; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * RuleAdapterService + * + * @author ：wangchao + * @date ：Created in 2022/12/1 + * @since ：11 + */ +@Service +public class RuleAdapterService { + private static final Map> RULES = new HashMap<>(); + + @Resource + private TableRuleAdapterService tableRuleAdapterService; + @Resource + private ColumnRuleAdapterService columnRuleAdapterService; + @Resource + private RowRuleAdapterService rowRuleAdapterService; + + public void init(Map> rules) { + RULES.clear(); + RULES.putAll(rules); + } + + public List executeTableRule(List tableList) { + final List rules = RULES.get(RuleType.TABLE); + if (CollectionUtils.isEmpty(rules)) { + return tableList; + } + return tableRuleAdapterService.executeTableRule(rules, tableList); + } + + public List executeColumnRule(List columns) { + final List rules = RULES.get(RuleType.COLUMN); + if (CollectionUtils.isEmpty(rules)) { + return columns; + } + return columnRuleAdapterService.executeColumnRule(rules, columns); + } + + public Map executeRowRule() { + final List rules = RULES.get(RuleType.ROW); + if (CollectionUtils.isEmpty(rules)) { + return new HashMap<>(); + } + return rowRuleAdapterService.executeRowRule(rules); + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/TableRuleAdapterService.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/TableRuleAdapterService.java new file mode 100644 index 0000000000000000000000000000000000000000..363dd1f1056c44d040c7320249e635d5a6c212fe --- /dev/null +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/service/TableRuleAdapterService.java @@ -0,0 +1,75 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.service; + +import org.apache.commons.collections4.CollectionUtils; +import org.opengauss.datachecker.common.constant.RuleConstants; +import org.opengauss.datachecker.common.entry.common.Rule; +import org.springframework.stereotype.Service; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +/** + * TableRuleAdapterService + * + * @author ：wangchao + * @date ：Created in 2022/12/1 + * @since ：11 + */ +@Service +public class TableRuleAdapterService { + private static final String WHITE = RuleConstants.RULE_WHITE; + private static final String BLACK = RuleConstants.RULE_BLACK; + private static final String SPLIT = RuleConstants.RULE_SPLIT; + private static final String REGEX_PREFIX = "^"; + private static final String REGEX_SUFFIX = "$"; + private static final String SUB_REGEX_PREFIX = "("; + private static final String SUB_REGEX_SUFFIX = ")"; + private static final String REGEX_DELIMITER = "|"; + private static final Map EXECUTORS = new HashMap<>(); + + static { + EXECUTORS.put(WHITE, (pattern, table) -> pattern.matcher(table).matches()); + EXECUTORS.put(BLACK, (pattern, table) -> !pattern.matcher(table).matches()); + } + + public List executeTableRule(List rules, List tableList) { + if (CollectionUtils.isEmpty(rules)) { + return tableList; + } + final Rule rule = rules.get(0); + final TableRuleExecutor tableRuleExecutor = EXECUTORS.get(rule.getName()); + String ruleRegex = rule.getText(); + if (!rule.getText().startsWith(REGEX_PREFIX) && !rule.getText().endsWith(REGEX_SUFFIX)) { + final String[] ruleArr = rule.getText().split(SPLIT); + ruleRegex = Arrays.stream(ruleArr).map(subRule -> SUB_REGEX_PREFIX + subRule + SUB_REGEX_SUFFIX) + .collect(Collectors.joining(REGEX_DELIMITER)); + } + Pattern pattern = Pattern.compile(ruleRegex); + return tableList.parallelStream().filter(table -> tableRuleExecutor.apply(pattern, table)) + .collect(Collectors.toList()); + } + + @FunctionalInterface + interface TableRuleExecutor { + boolean apply(Pattern tablePattern, String table); + } +} diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilder.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilder.java index 175587ddcf8183509e7227c57ff212134d8b5ed2..53435f92ac6c0c10f30f9e5d053b7eed7a5c37e2 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilder.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilder.java @@ -16,6 +16,7 @@ package org.opengauss.datachecker.extract.task; import org.opengauss.datachecker.common.constant.Constants.InitialCapacity; +import org.opengauss.datachecker.common.entry.extract.ConditionLimit; import org.opengauss.datachecker.common.entry.extract.ExtractIncrementTask; import org.opengauss.datachecker.common.entry.extract.ExtractTask; import org.opengauss.datachecker.common.entry.extract.SourceDataLog; @@ -83,8 +84,11 @@ public class ExtractTaskBuilder { if (Objects.nonNull(metadata)) { // tableRows is the current table data amount counted in the table metadata information long tableRows = metadata.getTableRows(); - if (tableRows > EXTRACT_MAX_ROW_COUNT) { - + final ConditionLimit conditionLimit = metadata.getConditionLimit(); + if (Objects.nonNull(conditionLimit)) { + taskList.add(buildTask(metadata)); + taskCountMap.put(tableName, 1); + } else if (tableRows > EXTRACT_MAX_ROW_COUNT) { // Construct extraction tasks based on table metadata information List taskEntryList = buildTaskList(metadata); taskCountMap.put(tableName, taskEntryList.size()); diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/QuerySqlTemplate.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/QuerySqlTemplate.java index 9ac4c0aaa890d1a94b13434da5d381a4e15ccc01..cc93cb26c66c9d30d0cebe31c08a6574737d0aee 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/QuerySqlTemplate.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/QuerySqlTemplate.java @@ -58,6 +58,8 @@ public interface QuerySqlTemplate { */ String JOIN_ON = ":joinOn"; + String ORDER_BY = ":orderBy"; + /** * Query SQL statement fragment: Query SQL statements in the scenario without offset */ @@ -66,7 +68,7 @@ public interface QuerySqlTemplate { /** * Query SQL statement fragment: SQL statement for fragment query using offset in single primary key scenario */ - String QUERY_OFF_SET = "SELECT :columnsList FROM :schema.:tableName LIMIT :start,:offset"; + String QUERY_OFF_SET = "SELECT :columnsList FROM :schema.:tableName :orderBy LIMIT :start,:offset"; /** * Query SQL statement fragment: SQL statement for fragment query using offset in single primary key scenario diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/SelectSqlBuilder.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/SelectSqlBuilder.java index 68174bfe9e6442c5f7ad5476ca80ade424ed20c1..df3e848c0e7dcf920bceb1cbdbace979a5b01b35 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/SelectSqlBuilder.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/task/sql/SelectSqlBuilder.java @@ -15,10 +15,10 @@ package org.opengauss.datachecker.extract.task.sql; -import lombok.AllArgsConstructor; import lombok.Getter; import org.opengauss.datachecker.common.entry.enums.DataBaseType; import org.opengauss.datachecker.common.entry.extract.ColumnsMetaData; +import org.opengauss.datachecker.common.entry.extract.ConditionLimit; import org.opengauss.datachecker.common.entry.extract.TableMetadata; import org.springframework.lang.NonNull; import org.springframework.util.Assert; @@ -32,10 +32,9 @@ import java.util.stream.Collectors; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.COLUMN; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.DELIMITER; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.MYSQL_ESCAPE; -import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.MYSQL_DELIMITER; -import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.OPENGAUSS_DELIMITER; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.OPENGAUSS_ESCAPE; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.OFFSET; +import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.ORDER_BY; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.SCHEMA; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.START; import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.TABLE_NAME; @@ -49,11 +48,13 @@ import static org.opengauss.datachecker.extract.task.sql.QuerySqlTemplate.TABLE_ **/ public class SelectSqlBuilder { private static final Map SQL_GENERATE = new HashMap<>(); + private static final Map ESCAPE = new HashMap<>(); private static final long OFF_SET_ZERO = 0L; private static final SqlGenerateTemplate GENERATE_TEMPLATE = (template, sqlGenerateMeta) -> template.replace(COLUMN, sqlGenerateMeta.getColumns()) .replace(SCHEMA, sqlGenerateMeta.getSchema()) .replace(TABLE_NAME, sqlGenerateMeta.getTableName()) + .replace(ORDER_BY, sqlGenerateMeta.getOrder()) .replace(START, String.valueOf(sqlGenerateMeta.getStart())) .replace(OFFSET, String.valueOf(sqlGenerateMeta.getOffset())); private static final SqlGenerateTemplate NO_OFFSET_SQL_GENERATE_TEMPLATE = @@ -69,6 +70,9 @@ public class SelectSqlBuilder { SQL_GENERATE.put(DataBaseType.MS, OFFSET_GENERATE); SQL_GENERATE.put(DataBaseType.OG, OFFSET_GENERATE); SQL_GENERATE.put(DataBaseType.O, OFFSET_GENERATE); + ESCAPE.put(DataBaseType.MS, (key) -> MYSQL_ESCAPE + key + MYSQL_ESCAPE); + ESCAPE.put(DataBaseType.OG, (key) -> OPENGAUSS_ESCAPE + key + OPENGAUSS_ESCAPE); + ESCAPE.put(DataBaseType.O, (key) -> key); } private String schema; @@ -133,47 +137,55 @@ public class SelectSqlBuilder { Assert.isTrue(Objects.nonNull(tableMetadata), Message.TABLE_METADATA_NULL_NOT_TO_BUILD_SQL); List columnsMetas = tableMetadata.getColumnsMetas(); Assert.notEmpty(columnsMetas, Message.COLUMN_METADATA_EMPTY_NOT_TO_BUILD_SQL); - if (offset == OFF_SET_ZERO || !isDivisions) { + final ConditionLimit conditionLimit = tableMetadata.getConditionLimit(); + if (Objects.nonNull(conditionLimit)) { + return buildSelectSqlConditionLimit(tableMetadata, conditionLimit); + } else if (offset == OFF_SET_ZERO || !isDivisions) { return buildSelectSqlOffsetZero(columnsMetas, tableMetadata.getTableName()); } else { return buildSelectSqlOffset(tableMetadata, start, offset); } } - public String buildSelectSqlOffset(TableMetadata tableMetadata, long start, long offset) { + private String buildSelectSqlConditionLimit(TableMetadata tableMetadata, ConditionLimit conditionLimit) { List columnsMetas = tableMetadata.getColumnsMetas(); - String tableName = tableMetadata.getTableName(); String columnNames = getColumnNameList(columnsMetas, dataBaseType); + final String schemaEscape = escape(schema, dataBaseType); + final String tableName = escape(tableMetadata.getTableName(), dataBaseType); + final String orderBy = getOrderBy(tableMetadata.getPrimaryMetas(), dataBaseType); SqlGenerateMeta sqlGenerateMeta = - new SqlGenerateMeta(schema, convertTableName(tableName, dataBaseType), columnNames, start, offset); + new SqlGenerateMeta(schemaEscape, tableName, columnNames, orderBy, conditionLimit.getStart(), + conditionLimit.getOffset()); return getSqlGenerate(dataBaseType).replace(sqlGenerateMeta); } - private static String convertTableName(String tableName, DataBaseType dataBaseType) { - if (DataBaseType.MS.equals(dataBaseType)) { - return MYSQL_ESCAPE + tableName + MYSQL_ESCAPE; - } else if (DataBaseType.OG.equals(dataBaseType)) { - return OPENGAUSS_ESCAPE + tableName + OPENGAUSS_ESCAPE; - } - return tableName; + private String getOrderBy(List primaryMetas, DataBaseType dataBaseType) { + return "order by " + primaryMetas.stream().map(ColumnsMetaData::getColumnName) + .map(key -> escape(key, dataBaseType) + " asc") + .collect(Collectors.joining(DELIMITER)); + } + + public String buildSelectSqlOffset(TableMetadata tableMetadata, long start, long offset) { + List columnsMetas = tableMetadata.getColumnsMetas(); + String tableName = escape(tableMetadata.getTableName(), dataBaseType); + String columnNames = getColumnNameList(columnsMetas, dataBaseType); + SqlGenerateMeta sqlGenerateMeta = new SqlGenerateMeta(schema, tableName, columnNames, start, offset); + return getSqlGenerate(dataBaseType).replace(sqlGenerateMeta); + } + + private String escape(String content, DataBaseType dataBaseType) { + return ESCAPE.get(dataBaseType).escape(content); } private String buildSelectSqlOffsetZero(List columnsMetas, String tableName) { String columnNames = getColumnNameList(columnsMetas, dataBaseType); - SqlGenerateMeta sqlGenerateMeta = - new SqlGenerateMeta(schema, convertTableName(tableName, dataBaseType), columnNames, 0, 0); + SqlGenerateMeta sqlGenerateMeta = new SqlGenerateMeta(schema, escape(tableName, dataBaseType), columnNames); return NO_OFFSET_GENERATE.replace(sqlGenerateMeta); } - private static String getColumnNameList(@NonNull List columnsMetas, DataBaseType dataBaseType) { - if (DataBaseType.MS.equals(dataBaseType)) { - return MYSQL_ESCAPE + columnsMetas.stream().map(ColumnsMetaData::getColumnName) - .collect(Collectors.joining(MYSQL_DELIMITER)) + MYSQL_ESCAPE; - } else if (DataBaseType.OG.equals(dataBaseType)) { - return OPENGAUSS_ESCAPE + columnsMetas.stream().map(ColumnsMetaData::getColumnName) - .collect(Collectors.joining(OPENGAUSS_DELIMITER)) + OPENGAUSS_ESCAPE; - } - return columnsMetas.stream().map(ColumnsMetaData::getColumnName).collect(Collectors.joining(DELIMITER)); + private String getColumnNameList(@NonNull List columnsMetas, DataBaseType dataBaseType) { + return columnsMetas.stream().map(ColumnsMetaData::getColumnName).map(column -> escape(column, dataBaseType)) + .collect(Collectors.joining(DELIMITER)); } private SqlGenerate getSqlGenerate(DataBaseType dataBaseType) { @@ -181,13 +193,37 @@ public class SelectSqlBuilder { } @Getter - @AllArgsConstructor static class SqlGenerateMeta { - private final String schema; - private final String tableName; - private final String columns; - private final long start; - private final long offset; + private String schema; + private String tableName; + private String columns; + private String order; + private long start; + private long offset; + + public SqlGenerateMeta(String schema, String tableName, String columns) { + this.schema = schema; + this.tableName = tableName; + this.columns = columns; + } + + public SqlGenerateMeta(String schema, String tableName, String columns, long start, long offset) { + this.schema = schema; + this.tableName = tableName; + this.columns = columns; + this.order = ""; + this.start = start; + this.offset = offset; + } + + public SqlGenerateMeta(String schema, String tableName, String columns, String order, long start, long offset) { + this.schema = schema; + this.tableName = tableName; + this.columns = columns; + this.order = order; + this.start = start; + this.offset = offset; + } } @FunctionalInterface @@ -201,6 +237,15 @@ public class SelectSqlBuilder { String replace(SqlGenerateMeta sqlGenerateMeta); } + @FunctionalInterface + interface SqlEscape { + /** + * @param key key + * @return Return + */ + String escape(String key); + } + @FunctionalInterface interface SqlGenerateTemplate { /** diff --git a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/util/SpringUtil.java b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/util/SpringUtil.java index 9c66c46e7501b7a5a8ad22ca915add45df9c81e4..fe86704c3ea12587a85b5f8ea1055397a92e20cc 100644 --- a/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/util/SpringUtil.java +++ b/datachecker-extract/src/main/java/org/opengauss/datachecker/extract/util/SpringUtil.java @@ -21,6 +21,8 @@ import org.springframework.context.ApplicationContext; import org.springframework.context.ApplicationContextAware; import org.springframework.stereotype.Component; +import java.util.Map; + /** * SpringUtil * @@ -74,6 +76,9 @@ public class SpringUtil implements ApplicationContextAware { return getApplicationContext().getBean(clazz); } + public static Map getBeans(Class clazz) { + return getApplicationContext().getBeansOfType(clazz); + } /** * Get the corresponding bean instance according to the clazz type * diff --git a/datachecker-extract/src/test/java/org/opengauss/datachecker/extract/service/TableRuleAdapterServiceTest.java b/datachecker-extract/src/test/java/org/opengauss/datachecker/extract/service/TableRuleAdapterServiceTest.java new file mode 100644 index 0000000000000000000000000000000000000000..98b998320fb3dbe1c3a85ed859e65f44fa7c4792 --- /dev/null +++ b/datachecker-extract/src/test/java/org/opengauss/datachecker/extract/service/TableRuleAdapterServiceTest.java @@ -0,0 +1,144 @@ +/* + * Copyright (c) 2022-2022 Huawei Technologies Co.,Ltd. + * + * openGauss is licensed under Mulan PSL v2. + * You can use this software according to the terms and conditions of the Mulan PSL v2. + * You may obtain a copy of Mulan PSL v2 at: + * + * http://license.coscl.org.cn/MulanPSL2 + * + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, + * EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, + * MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE. + * See the Mulan PSL v2 for more details. + */ + +package org.opengauss.datachecker.extract.service; + +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.opengauss.datachecker.common.entry.common.Rule; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.List; +import java.util.stream.Collectors; + +import static org.assertj.core.api.Assertions.assertThat; +/** + * TableRuleAdapterServiceTest + * + * @author ：wangchao + * @date ：Created in 2022/10/31 + * @since ：11 + */ +public class TableRuleAdapterServiceTest { + + private TableRuleAdapterService tableRuleAdapterService; + private List tableList = new ArrayList<>(); + + private String table_prefix = "t_test_"; + + @BeforeEach + void setUp() { + tableRuleAdapterService = new TableRuleAdapterService(); + mockTableList(); + } + + private void mockTableList() { + for (int idx = 0; idx < 300; idx++) { + tableList.add(table_prefix + idx); + } + } + + private void sort(List expected) { + expected.sort(Comparator.naturalOrder()); + } + + @DisplayName("black & white with actual table name") + @Test + void test_black_white_with_actual_table_name() { + // Setup + final List rules = + List.of(new Rule("white", "t_test_196,t_test_158,t_test_156"), new Rule("black", "t_test_152")); + + // Run the test + final List result = tableRuleAdapterService.executeTableRule(rules, tableList); + sort(result); + + List expected = new ArrayList<>(Arrays.asList("t_test_196", "t_test_158", "t_test_156")); + sort(expected); + // Verify the results + assertThat(result).isEqualTo(expected); + } + + @DisplayName("white with actual table name") + @Test + void test_white_with_actual_table_name() { + // Setup + final List rules = List.of(new Rule("white", "t_test_196,t_test_158,t_test_156")); + + // Run the test + final List result = tableRuleAdapterService.executeTableRule(rules, tableList); + sort(result); + + List expected = new ArrayList<>(Arrays.asList("t_test_196", "t_test_158", "t_test_156")); + sort(expected); + // Verify the results + assertThat(result).isEqualTo(expected); + } + + @DisplayName("black with actual table name") + @Test + void test_black_with_actual_table_name() { + // Setup + // final List rules = List.of(new Rule("black", "t_test_196")); + final List rules = List.of(new Rule("black", "t_test_196,t_test_158,t_test_156")); + + // Run the test + final List result = tableRuleAdapterService.executeTableRule(rules, tableList); + sort(result); + + List expected = tableList.stream().filter( + table -> !(table.equals("t_test_196") || table.equals("t_test_158") || table.equals("t_test_156"))) + .collect(Collectors.toList()); + sort(expected); + // Verify the results + assertThat(result).isEqualTo(expected); + } + + @DisplayName("white with regex table name ") + @Test + void test_white_with_one_regex_table_name() { + // Setup + final List rules = List.of(new Rule("white", "^[\\w]+$")); + + // Run the test + final List result = tableRuleAdapterService.executeTableRule(rules, tableList); + sort(result); + + sort(tableList); + // Verify the results + assertThat(result).isEqualTo(tableList); + } + + // @DisplayName("white with regex table name contains 0 ,9 ") + // @Test + // void test_white_with_one_regex_table_name_contains_0_9() { + // // Setup + // final List rules = List.of(new Rule("white", "^[\\w]+[0,9]$")); + // + // // Run the test + // final List result = tableRuleAdapterService.executeTableRule(rules, tableList); + // sort(result); + // List expected1 = tableList.stream().filter(table -> table.contains("9")).collect(Collectors.toList()); + // List expected2 = tableList.stream().filter(table -> table.contains("0")).collect(Collectors.toList()); + // expected1.addAll(expected2); + // final List collect = + // expected1.stream().distinct().sorted(Comparator.naturalOrder()).collect(Collectors.toList()); + // // Verify the results + // assertThat(result).isEqualTo(collect); + // } +} diff --git a/datachecker-extract/src/test/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilderTest.java b/datachecker-extract/src/test/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilderTest.java index 05fb94b11ab8d4b924b7cdd065fd62e1c0f018b0..78483b736e28f97b49a5857d3b8847f5f355cb46 100644 --- a/datachecker-extract/src/test/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilderTest.java +++ b/datachecker-extract/src/test/java/org/opengauss/datachecker/extract/task/ExtractTaskBuilderTest.java @@ -23,7 +23,6 @@ import org.opengauss.datachecker.extract.service.MetaDataService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.test.context.SpringBootTest; -import javax.annotation.PostConstruct; import java.util.List; import java.util.Set; @@ -46,7 +45,6 @@ public class ExtractTaskBuilderTest { * init */ public void init() { - MetaDataCache.initCache(); MetaDataCache.putMap(metadataService.queryMetaDataOfSchema()); }