# πFlow

**Repository Path**: bluejoe/piflow

## Basic Information

- **Project Name**: πFlow
- **Description**: 可配置图形化大数据流水线系统，包含丰富的处理器组件，提供Shell、DSL、Web配置界面、任务调度、任务监控等功能
- **Primary Language**: Scala
- **License**: BSD-2-Clause
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 61
- **Forks**: 13
- **Created**: 2017-10-17
- **Last Updated**: 2025-07-13

## Categories & Tags

**Categories**: big-data

**Tags**: None

## README

# πFlow
a full big data flow system involving online data collection, real time computation, data output

```
        ______ _
       |  ____| |
 ______| |__  | | _____      __
/_ _  _|  __| | |/ _ \ \ /\ / /
 || ||_| |    | | (_) \ V  V /
 |/  \/|_|    |_|\___/ \_/\_/
	
```
# Starting piflow-server with maven

```
cd piflow-core
mvn test -DskipTests -PstartServer
```

# Running piflow-shell with maven

Since piflow-shell depends on piflow-core, so install piflow-core project first:
```
cd piflow-core
mvn clean install -DskipTests
```

Then, run mvn with following arguments:
```
cd ./piflow-shell
mvn clean test -PrunShell
```

Now, piflow-shell will be started in which you can type commands:
```
πFlow> SeqAsSource(1 to 1000) > DoMap[Int, Int](_ + 1) > ConsoleSink()
πFlow> res0!
```

type ```:quit``` to quit the shell.

# Tests and Examples

- BatchFlowTest: tests batch flow building and execution
https://gitee.com/bluejoe/piflow/blob/master/piflow-core/src/test/scala/cn/piflow/flow/BatchFlowTest.scala

- JobManagerTest: tests job management functions, including list, kill, pause operations
https://gitee.com/bluejoe/piflow/blob/master/piflow-core/src/test/scala/cn/piflow/flow/JobManagerTest.scala

- DslTest: tests PiFlow DSL
https://gitee.com/bluejoe/piflow/blob/master/piflow-core/src/test/scala/cn/piflow/flow/DslTest.scala

- ShellRunner: starts a PiFlow shell
https://gitee.com/bluejoe/piflow/blob/master/piflow-shell/src/main/scala/cn/piflow/shell/ShellRunner.scala