# spring-data-flow-poc-demo

**Repository Path**: cckevincyh/spring-data-flow-poc-demo

## Basic Information

- **Project Name**: spring-data-flow-poc-demo
- **Description**: Spring Cloud Data Flow Demo
- **Primary Language**: Java
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2020-07-23
- **Last Updated**: 2023-05-18

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

##  环境准备

### Solace

https://docs.solace.com/Solace-SW-Broker-Set-Up/Docker-Containers/Set-Up-Single-Win-Container.htm

1. Install Docker
2. Clone the GitHub repository

> git clone https://github.com/SolaceLabs/solace-single-docker-compose.git

> cd solace-single-docker-compose/template

1. Run the following command

> docker-compose -f PubSubStandard_singleNode.yml up -d

配置文件如下，可以自己改默认的8080端口

```dockerfile
# docker-compose -f PubSubStandard_singleNode.yml up
version: '3.3'

services:
  primary:
    container_name: pubSubStandardSingleNode
    image: solace/solace-pubsub-standard:latest
    shm_size: 1g
    ulimits:
      core: 1
      nofile:
        soft: 2448
        hard: 38048
    deploy:
      restart_policy:
        condition: on-failure
        max_attempts: 1
    ports:
    #Port Mappings:  Ports are mapped straight through from host to
    #container.  This may result in port collisions on commonly used
    #ports that will cause failure of the container to start.
      #Web transport
      #- '80:80'
      #Web transport over TLS
      #- '443:443'
      #SEMP over TLS
      #- '943:943'
      #MQTT Default VPN
      #- '1883:1883'
      #AMQP Default VPN over TLS
      #- '5671:5671'
      #AMQP Default VPN
      #- '5672:5672'
      #MQTT Default VPN over WebSockets
      #- '8000:8000'
      #MQTT Default VPN over WebSockets / TLS
      #- '8443:8443'
      #MQTT Default VPN over TLS
      #- '8883:8883'
      #SEMP / PubSub+ Manager
      - '9090:8080'
      #REST Default VPN
      #- '9000:9000'
      #REST Default VPN over TLS
      #- '9443:9443'
      #SMF
      - '55555:55555'
      #SMF Compressed
      #- '55003:55003'
      #SMF over TLS
      #- '55443:55443'
    environment:
      - username_admin_globalaccesslevel=admin
      - username_admin_password=admin
      - system_scaling_maxconnectioncount=100
```

![](./img/solace.png)

### Mysql

获取mysql docker images

```shell
docker pull mysql:5.7.25
```



启动mysql

```shell
docker run -p 3306:3306 --name mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=task -d mysql:5.7.25
```

create database
```sql script
create database data_flow_demo;
```


### Spring Cloud Data Flow

本地安装

- download Spring Cloud Data Flow Server & shell

  ```bash
  wget https://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-server/2.5.3.RELEASE/spring-cloud-dataflow-server-2.5.3.RELEASE.jar
  wget https://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-shell/2.5.3.RELEASE/spring-cloud-dataflow-shell-2.5.3.RELEASE.jar
  ```

- Download Spring Cloud Skipper

  ```shell
  wget https://repo.spring.io/release/org/springframework/cloud/spring-cloud-skipper-server/2.4.3.RELEASE/spring-cloud-skipper-server-2.4.3.RELEASE.jar
  ```

启动命令

- 启动Data Flow Server，并指定数据存储的db

  ```bash
  java -jar spring-cloud-dataflow-server-2.5.3.RELEASE.jar  --spring.datasource.url=jdbc:mariadb://192.168.99.100:3306/data_flow_demo?useMysqlMetadata=true --spring.datasource.username=root --spring.datasource.password=password --spring.datasource.driver-class-name=org.mariadb.jdbc.Driver
  ```

  

- 启动Skipper

  ```bash
  java -jar spring-cloud-skipper-server-2.4.3.RELEASE.jar
  ```

- 启动shell

  ```
  java -jar spring-cloud-dataflow-shell-2.5.3.RELEASE.jar
  ```



## 项目构建

- maven 命令

  ```bash
  mvn clean install -pl common,jpa-common,task-common,stream-common,task-message-sender,message-adaptor,process-service,web-service,consumer-service
  ```

  

## 应用的注册和启动

- 注册

在启动的pring-cloud-dataflow-shell命令行中执行如下命令，将application注册到Spring Cloud Data Flow Server

```shell
app register --name task-message-sender --type task --uri maven://com.poc:task-message-sender:jar:1.0

app register --name message-adaptor --type source --uri maven://com.poc:message-adaptor:jar:1.0

app register --name process-service --type sink --uri maven://com.poc:process-service:jar:1.0

app register --name web-service --type source --uri maven://com.poc:web-service:jar:1.0

app register --name consumer-service --type sink --uri maven://com.poc:consumer-service:jar:1.0
```



- Create Stream

  ```shell
  stream create --name adaptor-processor --definition "message-adaptor  | process-service"
  
  stream create --name web-consumer --definition "web-service  --server.port=8762 | consumer-service"
  ```

  ![](./img/create-stream.png)

 - Create Task

   ```shell
   task create --name task-message-sender --definition "task-message-sender"
   ```

   ![](./img/create-task.png)

- 启动

  - deploy stream

    ```
    stream deploy --name adaptor-processor
    stream deploy --name web-consumer
    ```

    ![](./img/deploy-stream.png)

  - launch task

    ```shell
    task launch task-message-sender
    ```



## 项目架构

![](./img/poc-data-flow.png)