# budo-warehouse

**Repository Path**: quickthinking/budo-warehouse

## Basic Information

- **Project Name**: budo-warehouse
- **Description**: 基于Binlog等技术的数据同步工具，类似otter，and，more
- **Primary Language**: Java
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 34
- **Created**: 2019-07-02
- **Last Updated**: 2020-12-19

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# budo-warehouse

基于Binlog等技术的数据同步工具，类似otter，and，more


## 功能特性

1. 数据入口支持 Binlog MQ/Kafka 接口调用(Dubbo/HTTP)

2. 数据出口支持 ElasticSearch/Solr/Mongo/CVS文件/Hive/HBase MQ/Kafka 接口调用(Dubbo/HTTP) 邮件(发送变更告警)

3. 支持所有入口出口的动态搭配，级联串联；例如 Binlog > Kafka > ElasticSearch

4. 支持根据表和类型筛选部分变更事件；只同步新增，不同步删除

5. 支持异结构数据同步（增加一个列或处理列的值）


## 核心配置
```
// 数据端点(入口或出口)
class DataNode {
    // jdbc:mysql://127.0.0.1 
    // async:activemq://127.0.0.1:61616
    private String url;

    private String username;

    private String password;
}

// 传输通道
class Pipeline {
    // 动态筛选事件(SPEL表达式) 
    // #{eventType == 'UPDATE' and tableName.startsWith('backup_')}
    private String eventFilter;

    // 数据入口
    private Integer sourceDataNodeId;

    private String sourceSchema;

    private String sourceTable;

    // 数据出口
    private Integer targetDataNodeId;

    private String targetSchema;

    private String targetTable;
}
```


# 代码导读 
[CODE-GUIDE.md](CODE-GUIDE.md)


## 构建

需要使用Maven私服

```
<repository>
    <id>budo-snapshot-repository</id>
    <url>https://repo.rdc.aliyun.com/repository/1726-snapshot-ZLxSbx/</url>
</repository>
```

## 启动

[budo-warehouse-web-launcher](https://gitee.com/budogroup/budo-warehouse/tree/master/budo-warehouse-web-launcher)


## 参考资料

[Mysql Binlog](https://baike.baidu.com/item/Mysql%20Binlog)

[DataX](https://github.com/alibaba/DataX)

[Otter双向回环控制](https://github.com/alibaba/otter/wiki/Otter双向回环控制)

[Oracle 物化视图](https://github.com/alibaba/yugong/wiki/DevDesign)

[通过Interface收发MQ消息 budo-dubbo-protocol-async](https://gitee.com/budogroup/budo-common/tree/master/budo-dubbo-protocol-async)

[budo-csv-jdbc-driver](https://gitee.com/budogroup/budo-common/tree/master/budo-csv-jdbc-driver)

[budo-elasticsearch-jdbc-driver](https://gitee.com/budogroup/budo-common/tree/master/budo-elasticsearch-jdbc-driver)

[budo-hbase-jdbc-driver](https://gitee.com/budogroup/budo-common/tree/master/budo-hbase-jdbc-driver)

[budo-solr-jdbc-driver](https://gitee.com/budogroup/budo-common/tree/master/budo-solr-jdbc-driver)

[budo-mongo-jdbc-driver](https://gitee.com/budogroup/budo-common/tree/master/budo-mongo-jdbc-driver)