# HuaweiCloud-Heron-plugins

**Repository Path**: HuaweiCloudDeveloper/huaweicloud-heron-plugins

## Basic Information

- **Project Name**: HuaweiCloud-Heron-plugins
- **Description**: Huawei-Heron-plugins
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2023-03-14
- **Last Updated**: 2025-06-16

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

<!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
-->
# Heron Uploader 新增 OBS 的支持

## 项目背景

Heron 用于处理实时数据流，允许在数据到达时立即进行处理和分析。它通常用于构建大规模的实时分析系统，例如实时推荐、实时计算等。Heron 的设计涉及多个组件和层次，以下是一些包含在 Heron 设计图中的关键组件：
- Scheduler
- Uploader
- Topology（拓扑）
- Stream Manager
- Heron监控

![图片1](images/1.png)

其中 Heron Uploader 会分发 topology 的 jar 到各个服务器上来执行，Heron Uploader 本身支持：HDFS，Local File System，Amazon S3 等，如今新增 OBS 。

### Documentation

[Heron in Apache Incubation](https://github.com/apache/incubator-heron)

## 使用文档

### 使用前准备

#### 1 华为云相关服务准备

(1) 在华为云 OBS 控制台，创建一个桶，用于存放上传的拓扑</br>
https://console.huaweicloud.com/console/#/obs/manager/buckets

(2) 在华为云 —— 我的凭证，生成访问密钥</br>
https://console.huaweicloud.com/iam/#/mine/accessKey

(3) 操作系统建议用：Ubuntu 20.04

(4) 在服务器创建 /opt/software 目录，在此目录下搭建 Heron 项目
```
mkdir -p /opt/software/heron
```

#### 2 Java 环境准备

(1) 在oracle官网下载JDK11：https://www.oracle.com/

(2) 上传到服务器 /opt/software 目录下，将其解压
```
cd /opt/software
# 解压 Java 压缩包
tar -zxvf jdk-11.0.17_linux-x64_bin.tar.gz
```

(3) 配置环境变量
```
# 编辑全局变量文件
vim /etc/profile
```

(4) 在最后一行配置以下信息并保存
```shell
# JAVA11 HOME
export JAVA_HOME=/opt/software/jdk-11.0.17
export CLASSPATH=$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin
```

(5) 使配置文件生效
```
source /etc/profile
```

(6) 检查配置是否生效，出现java版本信息，代表配置生效
```
java -version
```

![图片2](images/2.png)

### Heron源码编译安装包

#### 1 操作系统建议用：Ubuntu 20.04

#### 2 编译部署方式可参考：[Compiling Linux](https://github.com/apache/incubator-heron/tree/master/website2/docs/compiling-linux.md)

(1) 更新 Ubuntu
```
sudo apt-get update -y
sudo apt-get upgrade -y
```

(2) 安装所需的库
```
sudo apt-get install git build-essential automake cmake libtool-bin zip ant \
libunwind-setjmp0-dev zlib1g-dev unzip pkg-config python3-setuptools -y
```

(3) 设置以下环境变量
```
export CC=/usr/bin/gcc
export CCX=/usr/bin/g++
```

(4) 安装 Bazel
```
cd /opt/software
wget -O bazel.sh https://mirrors.huaweicloud.com/bazel/6.0.0/bazel-6.0.0-installer-linux-x86_64.sh
chmod +x bazel.sh
# 运行安装脚本
/opt/software/bazel.sh --user
```

(5) 安装 python 开发工具
```
sudo apt-get install  python3-dev python3-pip
sudo apt install python3.8-venv
```

(6) 确保 Bazel 可执行文件位于 PATH 中
```
export PATH="$PATH:$HOME/bin"
# 检测 Bazel 是否安装成功
bazel version
```

![图片6](images/6.png)

(7) 获取最新版本的 Heron 源代码
```
cd /opt/software
git clone https://gitee.com/HuaweiCloudDeveloper/huaweicloud-heron-plugins.git && cd huaweicloud-heron-plugins
```

(8) 检查使用 Bazel 构建 Heron 时所需环境配置是否满足要求
```
./bazel_configure.py
```

![图片7](images/7.png)

(9) 构建项目
```
chmod -R +x /opt/software/huaweicloud-heron-plugins
bazel build heron/...
```

(10) 构建包
```
bazel build scripts/packages:binpkgs
bazel build scripts/packages:tarpkgs
# 构建成功后，将在 bazel-bin/scripts/packages/ 目录中发现 Heron 安装包：heron.tar.gz 和 heron-install.sh
```

### 安装Heron并启动

#### 1 解压并配置环境变量

(1) 上传到服务器 /opt/software/heron 目录下，将其解压
```
cd /opt/software/heron
# 解压 heron 压缩包
tar -zxvf heron.tar.gz
cd /opt/software/heron/dist
# 解压 heron-core 压缩包
tar -zxvf heron-core.tar.gz
```

(2) 配置环境变量
```
# 编辑全局变量文件
vim /etc/profile
```

(3) 在最后一行配置以下信息并保存
```shell
# HERON HOME
export PATH=/opt/software/heron/bin:$PATH
```

(4) 使配置文件生效
```
source /etc/profile
```

(5) 检查配置是否生效，出现heron版本信息，代表配置生效
```
heron version
```

![图片3](images/3.png)

#### 2 配置uploader.yaml

```yaml
# uploader 类型设置为 OBS
heron.class.uploader: org.apache.heron.uploader.obs.ObsUploader
# upload 期望上传的 OBS 桶名称
heron.uploader.obs.bucket: xxx
# upload 期望上传的 OBS AK
heron.uploader.obs.access_key: xxx
# upload 期望上传的 OBS SK
heron.uploader.obs.secret_key: xxx
# upload 期望上传的 OBS endpoint
heron.uploader.obs.endpoint: obs.xxx.myhuaweicloud.com
# upload 期望上传的 OBS 桶路径前缀
heron.uploader.obs.path_prefix: xxx/xxx/
```

#### 3 上传拓扑

(1) 提交拓扑 WindowedWordCountTopology
```
heron submit local /opt/software/heron/examples/heron-streamlet-examples.jar \
org.apache.heron.examples.streamlet.WindowedWordCountTopology \
WindowedWordCountTopology --deploy-deactivated
```

![图片4](images/4.png)

(2) OBS Uploader 设置的路径为 heron/，对应路径中有如下对象

![图片5](images/5.png)

### 更多文档

更多Heron的使用可参考：[Heron Documents](https://github.com/apache/incubator-heron/tree/master/website2/docs)