# Arctic

**Repository Path**: quickthinking/Arctic

## Basic Information

- **Project Name**: Arctic
- **Description**: Arctic 是由网易开源的流式湖仓系统，Arctic 在 Iceberg 和 Hive 之上添加了更多实时场景的能力，并且面向 DataOps 提供流批统一，开箱即用的元数据服务
- **Primary Language**: Java
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: https://www.oschina.net/p/arctic
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 4
- **Created**: 2022-10-27
- **Last Updated**: 2022-10-27

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

![logo](site/docs/ch/images/arctic_logo_for_git.png)

Welcome to arctic, arctic is a streaming lake warehouse system open sourced by NetEase.
 Arctic adds more real-time capabilities on top of iceberg and hive, and provides stream-batch unified, out-of-the-box metadata services for dataops, 
allowing Data lakes much more usable and practical.
## What is arctic
Arctic is a streaming lakehouse service built on top of apache iceberg table format. 
Through arctic, users could benefit optimized CDC、streaming update、fresh olap etc. on engines like flink, spark, and trino. 
Combined with efficient offline processing capabilities of data lakes, arctic can serve more scenarios where streaming and batch are fused. 
At the same time, the function of self-optimization、concurrent conflict resolution and standard management tools could effectively reduce the burden on users in data lake management and optimization.
![Introduce](site/docs/ch/images/arctic_introduce.png)

Arctic services are presented by deploying AMS, which can be considered as a replacement for HMS (Hive Metastore), or HMS for iceberg. 
Arctic uses iceberg as the base table format, but instead of hacking the iceberg implementation, it uses iceberg as a lib. 
Arctic's open overlay architecture can help large-scale offline data lakes quickly upgraded to real-time data lakes, without worrying about compatibility issues with the original data lakes, 
allowing data lakes to meet more real-time analysis, real-time risk control, Real-time training, feature engineering and other scenarios.
## Arctic features

* Efficient streaming updates based on primary keys
* Data auto bucketing and self-optimized for performance and efficiency
* Encapsulating data lake and message queue into a unified table to achieve lower-latency computing
* Provide standardized metrics, dashboard and related management tools for streaming lakehouse
* Support spark and flink to read and write data, support trino to query data
* 100% compatible with iceberg / hive table format and syntax
* Provide transactional guarantee for streaming and batch concurrent writing

## Modules

Arctic contains modules as below:

- `arctic-core` contains core abstractions and common implementions for other modules
- `arctic-flink` is the module for integrating with Apache Flink (use arctic-flink-runtime for a shaded version)
- `arctic-spark` is the module for integrating with Apache Spark (use arctic-spark-runtime for a shaded version)
- `arctic-trino` now provides query integrating with apache trino, built on JDK11
- `arctic-optimizing` exposes optimizing container/group api and provides default implemetion
- `arctic-ams` is arctic meta service module
  - `ams-api` contains ams thrift api
  - `ams-dashboard` is the dashboard frontend for ams
  - `ams-server` is the backend server for ams

## Building

Arctic is built using Maven with Java 1.8 and Java 11(only for `trino` module).

* To build Trino module need config `toolchains.xml` in `${user.home}/.m2/` dir, the content is
```
<?xml version="1.0" encoding="UTF-8"?>
<toolchains>
    <toolchain>
        <type>jdk</type>
        <provides>
            <version>11</version>
            <vendor>sun</vendor>
        </provides>
        <configuration>
            <jdkHome>${yourJdk11Home}</jdkHome>
        </configuration>
    </toolchain>
</toolchains>
```
* To invoke a build and run tests: `mvn package -P toolchain`
* To skip tests: `mvn -DskipTests package -P toolchain`

## Engines supported

Arctic support multiple processing engines as below:

| Processing Engine | Version                   |
| ----------------- |---------------------------|
| Flink             | 1.12.x, 1.14.x and 1.15.x |
| Spark             | 2.3, 3.1                  |
| Trino             | 380                       |

## Quickstart

Visit [https://arctic.netease.com/ch/docker-quickstart/](https://arctic.netease.com/ch/docker-quickstart/) to quickly explore what arctic can do.

## Join Community 
If you are interested in Lakehouse, Data Lake Format, welcome to join our community, we welcome any organizations, teams and individuals to grow together, and sincerely hope to help users better use Data Lake Format through open source. 

Join the Arctic WeChat Group: Add " `kllnn999` " as a friend on WeChat and specify "Arctic lover".