# csx-ods-canal

**Repository Path**: xwcoding/csx-ods-canal

## Basic Information

- **Project Name**: csx-ods-canal
- **Description**: 数据同步工具
- **Primary Language**: Java
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2021-09-05
- **Last Updated**: 2021-09-05

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# csx-ods-canal

数据仓库ods同步程序是，基于canal canal做的全量同步及增量同步程序，部署程序包括csx-ods-fullload和csx-ods-increment

工程结构如下

```shell

-csx-ods-canal			#父级工程
--csx-ods-adapter		#适配器
----csx-ods-clickhouse	#clickhouse适配器
----csx-ods-tidb			#tidb适配器
--csx-ods-core			#核心代码及工具包
--csx-ods-fullload		#全量同步程序
--csx-ods-increment		#增量同步程序
```

## 全量同步（csx-ods-fullload）
全量同步程序原理，先查询源数据库并binlog位置，全量扫描源库分页查询并插入到目标库的临时表（新创建），同步完后修改表名为正式表名；启动canal订阅源库binlog日志并处理全表处理期间的增量数据；单端数据延迟低于2s（可配置）结束canal日志订阅

1. 配置说明

```shell

# web server configruation
# server
spring.application.name=csx-ods-fullload
server.port=8081
server.servlet.context-path=/
spring.resources.static-locations = classpath:/META-INF/resources/,classpath:/resources/,classpath:/static/
#======================================================================
#clickhouse configuration
bsf.clickhouse.enabled=true
spring.datasource.clickhouse.type=com.alibaba.druid.pool.DruidDataSource
spring.datasource.clickhouse.username=default
spring.datasource.clickhouse.password=
spring.datasource.clickhouse.url=jdbc:clickhouse://10.252.193.21:8123/default
spring.datasource.clickhouse.driver-class-name=ru.yandex.clickhouse.ClickHouseDriver
#tidb configuration
bsf.tidb.enabled=true
spring.datasource.tidb.type=com.alibaba.druid.pool.DruidDataSource
spring.datasource.tidb.username=tidb_test
spring.datasource.tidb.password=1234565
spring.datasource.tidb.url=jdbc:mysql://10.252.216.12:4000?useUnicode=true&characterEncoding=UTF8&useSSL=false&allowMultiQueries=true
spring.datasource.tidb.driver-class-name=com.mysql.cj.jdbc.Driver

#fulload thread size
fullload.parallelCount=20
#max scan table page size
fullload.maxcount=1000
fullload.freshTable.enabled=true


#db strategy
canal.dbs=basic_data_new

canal.batch.size=1
canal.ods.delay.limit=5000
#datasource strategy 
spring.datasource.basic_data_new.username=
spring.datasource.basic_data_new.password=
spring.datasource.basic_data_new.address=10.252.193.29:3306

#db table -> clickhouse table strategy
strategy.basic_data_new.base_address_info=basic_data_new.base_address_info
strategy.basic_data_new.base_inventory_provice=basic_data_new.base_inventory_provice
strategy.basic_data_new.base_inventory_relation=basic_data_new.base_inventory_relation



# redis configuration
bsf.redis.enabled=true
bsf.redis.nodes = 10.252.193.36:7000,10.252.193.37:7005,10.252.193.37:7003,10.252.193.36:7002,10.252.193.36:7001,10.252.193.37:7004

#bsf health monitor configuration,detail configruation ,please refer to (BSF)[https://gitee.com/yhcsx/csx-bsf-all]
bsf.health.enabled = true
bsf.health.io.autoClear = true
bsf.health.warn.enabled = true
bsf.health.check.enabled = true
bsf.health.export.enabled = true
bsf.health.export.elk.enabled = true
bsf.health.export.cat.enabled = true
bsf.health.log.statistic.enabled = true
bsf.health.dump.enabled = true
bsf.health.ping.enabled = true
bsf.health.doubtapi.enabled = true
bsf.health.doubtapi.threshold = 3145728
bsf.health.ping.enabled=true
bsf.health.file.enabled = true
bsf.health.file.timeSpan = 20
bsf.health.strategy.cpu.process = [>0.9]
bsf.health.strategy.cpu.system = [>0.8]
bsf.health.strategy.io.current.dir.usable.size = [<500]
bsf.health.strategy.memery.system.free = [<2048]
bsf.health.strategy.memery.jvm.max = [<1024]
bsf.health.strategy.tomcat.threadPool.queue.size = [>20]
bsf.health.strategy.tomcat.threadPool.active.count = [>100]
bsf.health.strategy.log.error.count = [>80]
bsf.health.strategy.log.incre.count = [>4000]
bsf.health.warn.dingding.system.access_token = 09cf9d76adfc600310fc33c918c9a6acd10f50ed707e0683d11c0dc9c17ceb32
bsf.health.warn.dingding.project.access_token = 


```

2. 启动

```shell
nohup java -jar csx-ods-fullload-1.0-SNAPSHOT.jar >/dev/null 2>&1 &
```

3. 同步操作

打开同步任务提交页面:http://{server_id}:{port}/sync.html
文本框提交同步任务，格式例如 ，"db.table1,db.table2"。



## 增量同步程序（csx-ods-increment）

启动canal订阅源库binlog日志并处理全表处理期间的增量数据

1. 配置同全量配置
2. 启动

```shell
nohup java -jar csx-ods-increment-1.0-SNAPSHOT.jar >/dev/null 2>&1 &
```