# lagou-kafka

**Repository Path**: dingjunjun_codeSpace/lagou-kafka

## Basic Information

- **Project Name**: lagou-kafka
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 1
- **Created**: 2022-10-24
- **Last Updated**: 2022-10-24

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

##  服务搭建

| 名称  | ip           | 软件安装                            |
| ----- | ------------ | ----------------------------------- |
| node1 | 10.10.10.211 | jdk、kafka、zoopeeker               |
| node2 | 10.10.10.212 | jdk、kafka、zoopeeker               |
| node3 | 10.10.10.213 | jdk、kafka、zoopeeker               |
| nginx | 10.10.10.214 | nginx、ngx_kafka_module、librdkafka |

#### 服务器配置

node1、node2、node3服务器不同的是

```shell
# zookeeper集群编号文件
$ echo 节点id > /var/lagou/zookeeper/data/myid
# kafka配置文件
$ vim /opt/kafka_2.12-1.0.2/config/server.properties
broker.id=节点id-1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://node节点id:9092
log.dirs=/var/lagou/kafka/kafka-logs
zookeeper.connect=node1:2181,node2:2181,node3:2181/myKafka
```



```shell
# 下载安装包
$ wget https://repo.huaweicloud.com/java/jdk/8u181-b13/jdk-8u181-linux-x64.rpm
$ wget https://repo.huaweicloud.com/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
$ wget https://repo.huaweicloud.com/apache/kafka/1.0.2/kafka_2.12-1.0.2.tgz

# 关闭防火墙
$ systemctl status firewalld
$ systemctl stop firewalld
$ systemctl disable firewalld

# 设置hosts
$ yum -y install vim
$ vim /etc/hosts
10.10.10.211 node1
10.10.10.212 node2
10.10.10.213 node3

# 安装Java环境
$ rpm -ivh jdk-8u181-linux-x64.rpm
$ vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64
export PATH=$PATH:$JAVA_HOME/bin

$ source /etc/profile
$ java -version

# 安装zookeeper环境
$ tar -zxf zookeeper-3.4.14.tar.gz -C /opt
$ cd /opt/zookeeper-3.4.14/conf
$ cp zoo_sample.cfg zoo.cfg
$ vim zoo.cfg
server.1=node1:2881:3881
server.2=node2:2881:3881
server.3=node3:2881:3881
dataDir=/var/lagou/zookeeper/data

$ mkdir -p /var/lagou/zookeeper/data
$ echo 1 > /var/lagou/zookeeper/data/myid

$ vim /etc/profile
export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/var/lagou/zookeeper/log
$ source /etc/profile

# 安装kafka环境
$ tar -zxf kafka_2.12-1.0.2.tgz -C /opt
$ vim /etc/profile
export KAFKA_HOME=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA_HOME/bin
$ source /etc/profile

# kafka配置
$ vim /opt/kafka_2.12-1.0.2/config/server.properties
broker.id=0
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://node1:9092
log.dirs=/var/lagou/kafka/kafka-logs
zookeeper.connect=node1:2181,node2:2181,node3:2181/myKafka

```

####  启动集群

```shell
# 启动Zookeeper
$ zkServer.sh start

# 启动kafka
$ kafka-server-start.sh -daemon /opt/kafka_2.12-1.0.2/config/server.properties
```

####  nginx服务配置

```shell
# 关闭防火墙
$ systemctl status firewalld
$ systemctl stop firewalld
$ systemctl disable firewalld

# 设置hosts
$ yum -y install vim
$ vim /etc/hosts
10.10.10.211 node1
10.10.10.212 node2
10.10.10.213 node3

# 安装依赖
$ yum install gcc-c++ -y

$ wget https://github.com/brg-liuwei/ngx_kafka_module/archive/refs/tags/v0.9.1.tar.gz
$ mv v0.9.1.tar.gz librdkafka-1.5.2.tar.gz 
$ wget https://github.com/edenhill/librdkafka/archive/refs/tags/v1.5.2.tar.gz
$ mv v1.5.2.tar.gz librdkafka-1.5.2.tar.gz 

# 安装配置kafka客户端库
$ tar -zxf librdkafka-1.5.2.tar.gz 
$ cd librdkafka-1.5.2
$ ./configure
$ make
$ sudo make install

# ngx_kafka_module
$ tar -zxf ngx_kafka_module-0.9.1.tar.gz

# 安装NGINX
$ wget https://repo.huaweicloud.com/nginx/nginx-1.17.8.tar.gz
$ tar -zxf nginx-1.17.8.tar.gz 
$ cd nginx-1.17.8
$ yum install gcc zlib zlib-devel openssl openssl-devel pcre pcre-devel -y
$ ./configure --add-module=/root/ngx_kafka_module-0.9.1
$ make
$ sudo make install
# 修改nginx.conf文件
$ cd /usr/local/nginx/conf
$ vim nginx.conf
## 在http{}中添加
kafka;
kafka_broker_list node1:9092 node2:9092 node3:9092;
## 在http{server{} }中添加
location /log { 
  add_header 'Access-Control-Allow-Origin' $http_origin;
  add_header 'Access-Control-Allow-Credentials' 'true';
  add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
  add_header 'Access-Control-Allow-Headers' 'DNT,web-token,app-token,Authorization,Accept,Origin,Keep-Alive,User-Agent,X-Mx-ReqToken,X-Data-Type,X-Auth-Token,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
  add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
  if ($request_method = 'OPTIONS') {
    add_header 'Access-Control-Max-Age' 1728000;
    add_header 'Content-Type' 'text/plain; charset=utf-8';
    add_header 'Content-Length' 0;
    return 204;
  }
  kafka_topic tp_log_01;
}

# 让操作系统加载模板
$ echo "/usr/local/lib" >> /etc/ld.so.conf
$ ldconfig
```

####  启动nginx

```shell
# 启动nginx
$ /usr/local/nginx/sbin/nginx

# 测试
$ curl localhost/log -d "hello ngx_kafka_module" -v

# 新开一个窗口，记录kafka的消费情况
$ kafka-console-consumer.sh --bootstrap-server node1:9092 --topic tp_log_01 --from-beginning
```