diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628215828665.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628215828665.png new file mode 100644 index 0000000000000000000000000000000000000000..2260fb08875635bd16372c1ad8c8b665064020b9 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628215828665.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628220021575.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628220021575.png new file mode 100644 index 0000000000000000000000000000000000000000..f403b795c6584469cc8145c5ace4b96dc9312c49 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628220021575.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628220055254.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628220055254.png new file mode 100644 index 0000000000000000000000000000000000000000..871f4e536e5e89630f47a40fcea6f14f2055f456 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628220055254.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628221218147.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628221218147.png new file mode 100644 index 0000000000000000000000000000000000000000..c44ef5e2a6fbd9c37b633a837e165058742faab3 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628221218147.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628222317161.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628222317161.png new file mode 100644 index 0000000000000000000000000000000000000000..228a55bb7d6446501667c5290bcdcd34712b259c Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628222317161.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628223551680.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628223551680.png new file mode 100644 index 0000000000000000000000000000000000000000..baa1813d8009652421c8efd6f62bf685ce714f2d Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628223551680.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628230657507.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628230657507.png new file mode 100644 index 0000000000000000000000000000000000000000..ecf5d499df0837efc617019437bcfbd1ef4c24c0 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628230657507.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628232228635.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628232228635.png new file mode 100644 index 0000000000000000000000000000000000000000..e2bd29bf8b34e044eb6211a06afef83ed191493e Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628232228635.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628232627644.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628232627644.png new file mode 100644 index 0000000000000000000000000000000000000000..97d2474d7c386a72071c75b67c2e3cdc11c442f0 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240628232627644.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629002535910.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629002535910.png new file mode 100644 index 0000000000000000000000000000000000000000..52b25d3fdd08e8793c7eab20ae1133abf18493fc Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629002535910.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629004728952-1719846865054.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629004728952-1719846865054.png new file mode 100644 index 0000000000000000000000000000000000000000..967d51fc76230672150e16fa9c9ccf857b87ffd8 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629004728952-1719846865054.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629004728952.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629004728952.png new file mode 100644 index 0000000000000000000000000000000000000000..967d51fc76230672150e16fa9c9ccf857b87ffd8 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629004728952.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629005025167-1719846865055.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629005025167-1719846865055.png new file mode 100644 index 0000000000000000000000000000000000000000..29bc64c97a7637151b93f3d920b797b4754dd188 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629005025167-1719846865055.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629005025167.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629005025167.png new file mode 100644 index 0000000000000000000000000000000000000000..29bc64c97a7637151b93f3d920b797b4754dd188 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629005025167.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629100837464-1719846786300.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629100837464-1719846786300.png new file mode 100644 index 0000000000000000000000000000000000000000..203d04bfe61169b78cbc37f686275cd92115abe9 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629100837464-1719846786300.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629100837464.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629100837464.png new file mode 100644 index 0000000000000000000000000000000000000000..203d04bfe61169b78cbc37f686275cd92115abe9 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629100837464.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629103345529-1719846786300.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629103345529-1719846786300.png new file mode 100644 index 0000000000000000000000000000000000000000..af1a52c78928f04e94b3bf4164ecc35a4ebd3bf4 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629103345529-1719846786300.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629103345529.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629103345529.png new file mode 100644 index 0000000000000000000000000000000000000000..af1a52c78928f04e94b3bf4164ecc35a4ebd3bf4 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629103345529.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110237598.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110237598.png new file mode 100644 index 0000000000000000000000000000000000000000..324c109ccd80f2f38cd3ad274f9c61e2ff58ba20 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110237598.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110257968.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110257968.png new file mode 100644 index 0000000000000000000000000000000000000000..774b0945d8d32c4b006ec7ebde6baf6bce0b37b7 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110257968.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110422825.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110422825.png new file mode 100644 index 0000000000000000000000000000000000000000..419055a8df6877d887347a328af9d78d52334cac Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629110422825.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629111347623.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629111347623.png new file mode 100644 index 0000000000000000000000000000000000000000..4b2903dfb57bd3c5b8c47cfb300f6c9e374b5e30 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629111347623.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629125404861-1719846786300.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629125404861-1719846786300.png new file mode 100644 index 0000000000000000000000000000000000000000..68229e715a1c76224bcb98a92966c2338960a694 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629125404861-1719846786300.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629125404861.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629125404861.png new file mode 100644 index 0000000000000000000000000000000000000000..68229e715a1c76224bcb98a92966c2338960a694 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240629125404861.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225825908.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225825908.png new file mode 100644 index 0000000000000000000000000000000000000000..c83602172c1a52632703b7adc7eafee4a222875c Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225825908.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225835987.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225835987.png new file mode 100644 index 0000000000000000000000000000000000000000..2ab2057b0d22f8812c550bd1849b2602f627ad62 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225835987.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225859103.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225859103.png new file mode 100644 index 0000000000000000000000000000000000000000..d4bdc3e084866cea67317d927bacab8d07418ccc Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701225859103.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230129538.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230129538.png new file mode 100644 index 0000000000000000000000000000000000000000..23b9f7f7e1090eedeae0d58c0fbd2aeedc9a82c9 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230129538.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230418479.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230418479.png new file mode 100644 index 0000000000000000000000000000000000000000..e84feb2f67aabef71f1e690ea722faf103276c3e Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230418479.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230426540.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230426540.png new file mode 100644 index 0000000000000000000000000000000000000000..6f3faa191ae508ef7734235429c0428f5cf7bdf4 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230426540.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230628054.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230628054.png new file mode 100644 index 0000000000000000000000000000000000000000..960fea50294642725f069a77269be21f16682ca0 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230628054.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230907670.png b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230907670.png new file mode 100644 index 0000000000000000000000000000000000000000..a756e5ba894431bd9db9d51592f67ef307326ea5 Binary files /dev/null and b/src/content/posts/2024-01/arm_kafka_test.assets/image-20240701230907670.png differ diff --git a/src/content/posts/2024-01/arm_kafka_test.md b/src/content/posts/2024-01/arm_kafka_test.md new file mode 100644 index 0000000000000000000000000000000000000000..164621dd3a56f41d156134134a21b8da72f12c8c --- /dev/null +++ b/src/content/posts/2024-01/arm_kafka_test.md @@ -0,0 +1,281 @@ +--- +title: Arm平台上的Kafka测试 +author: 刘玥骁 +pubDate: 2024-07-01 +categories: ['2024 年第一期'] +description: 'Arm上的Kafka性能测试与部分功能测试' +--- + +## 1. 我的任务 + +我的任务是进行大数据组件Kafka在ARM架构Kunpeng 920服务器和RISC-V架构SG2042服务器上的功能支持与性能测试。我完成了Kafka在Kunpeng 920服务器上的性能测试,记录了表格中要求的所有性能数据,并进行了Topic相关的功能测试;然而在RISC-V架构遇见了JDK环境问题导致Kafka无法运行,没有进行性能测试和功能测试。 + +## 2. 前置知识 + +### 2.1 消息队列 + +kafka是一个分布式的基于发布/订阅模式的消息队列,我通过阅读博客和实践,重点了解了消息队列概念、Kafka简介和在性能测试中使用到的逻辑概念topic,内容如下: + +消息队列就是将**需要传输的数据存放在队列中**,可用于解耦复杂系统,属于生产者-消费者模型,有以下两种模式: + +1. 点对点模式:每个消息只有一个接收者(Consumer)(即一旦被消费,消息就不再在消息队列中);发送者和接收者间没有依赖性,发送者发送消息之后,不管有没有接收者在运行,都不会影响到发送者下次发送消息;接收者在成功接收消息之后需向队列应答成功,以便消息队列删除当前接收的消息;![image-20240701225825908](arm_kafka_test.assets/image-20240701225825908.png) + +2. 发布/订阅模式:每个消息可以有多个订阅者;发布者和订阅者之间有时间上的依赖性。针对某个主题(Topic)的订阅者,它必须创建一个订阅者之后,才能消费发布者的消息。为了消费消息,订阅者需要**提前**订阅该角色主题,并保持在线运行;![image-20240701225835987](arm_kafka_test.assets/image-20240701225835987.png) + +### 2.2 Kafka简介 + +我们通常将Apache Kafka用在两类程序:1. 建立实时数据管道,以可靠地在系统或应用程序之间获取数据;2. 构建实时流应用程序,以转换或响应数据流。模型如下:![image-20240701225859103](arm_kafka_test.assets/image-20240701225859103.png) + +1. Producers:可以有很多的应用程序,将消息数据**放入**到Kafka集群中。 + +2. Consumers:可以有很多的应用程序,将消息数据从Kafka集群中**拉取**出来。 +3. Connectors:Kafka的连接器可以将数据库中的数据**导入**到Kafka,也可以将Kafka的数据**导出**到数据库中。 +4. Stream Processors:流处理器可以Kafka**拉取**数据,也可以将数据**写入**到Kafka中。 + +Kafka 的 topic 是一个逻辑概念,用于对消息进行分类。在 Kafka 中,消息以流的形式组织,而 topic 就是这些消息流的名称。每个 topic 可以被视为一个消息类别或者消息队列。用途如下: + +1. **消息分类**:topic 允许你将消息按照某种特定的类别进行分组。例如,一个公司的 Kafka 集群可能会有多个 topic,分别用于存储用户行为日志、系统监控数据、交易记录等。 + +2. **数据持久化**:Kafka 是一个持久化的消息系统,存储在 topic 中的消息会保留在磁盘上,直到它们被消费或者超过了设置的保留策略。 + +3. **分区**:为了能够横向扩展和并行处理,每个 Kafka topic 可以被分割成多个分区(partitions)。分区允许多个生产者和消费者并发地读写数据,提高了系统的吞吐量。 + +4. **复制**:为了提高数据的可靠性和可用性,每个分区可以被复制到多个 broker 上。Kafka 通过复制因子(replication factor)来定义每个分区的副本数量。 + +5. **消费者组**:Kafka 支持消费者组(consumer groups)的概念,允许多个消费者实例协调工作,共同消费同一个 topic 中的消息。 + +6. **消息顺序**:在同一个分区内,Kafka 保证消息的顺序。如果应用需要严格的顺序保证,可以通过单分区或者有序的分区键来实现。 + +7. **保留策略**:Kafka 允许你设置消息的保留时间或大小,超过这个时间或大小的消息将被删除。这有助于控制存储使用。 + +8. **创建和删除**:你可以使用 Kafka 提供的命令行工具或 API 来创建和删除 topic。 + +9. **配置**:topic 可以有自己的配置,如保留时间、清理策略、最小和最大消息大小等。 + +## 3. ARM架构 + +### 3.1 安装**Kafka和Zookeeper** + +1. 安装与解压 + +```shell +wget https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz +tar -zxvf kafka_2.12-2.2.0.tgz +#openjdk使用的毕昇自带的8u412 +wget http://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz +tar -zxvf zookeeper-3.4.6.tar.gz +``` + +2. 在~/.bash_profile中添加环境变量 + +```shell +export KAFKA_HOME=~/kafka_2.12-2.2.0 +export PATH=$KAFKA_HOME/bin:$PATH +export ZOOKEEPER_HOME=~/zookeeper-3.4.6 +export PATH=$ZOOKEEPER_HOME/bin:$PATH +``` + +3. 配置config/server.properties,修改的内容如下: + +![image-20240628232627644](arm_kafka_test.assets/image-20240628232627644.png) + +### 3.2 运行Kafka + +1. 先启动Zookeeper:`bin/zookeeper-server-start.sh config/zookeeper.properties` + +2. 然后启动Kafka服务:`bin/kafka-server-start.sh config/server.properties` + +![image-20240628215828665](arm_kafka_test.assets/image-20240628215828665.png) + +3. 测试是否成功启动:在服务器上打开一个生产者,然后把输入的每行数据发送到Kafka中,再打开一个消费者,当有数据往Kafka的test主题发送消息,这边就会进行消费。 + + ```shell + bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test + #后面光标提示数据数据,然后回车就会发送到kafka中了 + bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning + ``` + + ![image-20240628220021575](arm_kafka_test.assets/image-20240628220021575.png) + + ![image-20240628220055254](arm_kafka_test.assets/image-20240628220055254.png) + +### 3.3 **性能测试** + +参考[教程](https://blog.taliove.com/kafka-perf-test)。 + +1. 创建topic:创建名为`test-rep-one`,分区为6,复制因子为1的主题。数据的保留时间为1小时。 + + ```bash + $ bin/kafka-topics.sh \ + --create \ + --topic test-rep-one \ + --partitions 6 \ + --replication-factor 1 \ + --config retention.ms=86400000 \ + --bootstrap-server localhost:9092 + ``` + +2. 使用不同的生产者性能测试脚本。以下示例将使用上面创建的主题来存储300万条数据,每条消息的大小为1KB。`-throughput`设为-1时表示消息会尽快生成。 + + ```bash + $ bin/kafka-producer-perf-test.sh \ + --topic test-rep-one \ + --throughput -1 \ + --num-records 3000000 \ + --record-size 1024 \ + --producer-props acks=all bootstrap.servers=localhost:9092 + ``` + + ![image-20240628221218147](arm_kafka_test.assets/image-20240628221218147.png) + +3. 测试消费者性能。这样根据第2、3步,得到了非公共的性能指标(黄色部分)。 + + ```bash + $ bin/kafka-consumer-perf-test.sh \ + --topic test-rep-one \ + --broker-list localhost:9092 \ + --messages 3000000 + ``` + +![image-20240628222317161](arm_kafka_test.assets/image-20240628222317161.png)![image-20240701230418479](arm_kafka_test.assets/image-20240701230418479.png)![image-20240701230426540](arm_kafka_test.assets/image-20240701230426540.png) + +4. 加入公共指标:对于kafka-producer-perf-test.sh和kafka-consumer-perf-test.sh,将第2、3步的命令添加在脚本后即可;而对于kafka-topics.sh,我选择了列举topics的命令进行测试:`performance_counter_920.sh "bin/kafka-topics.sh --list --bootstrap-server localhost:9092" ./`。因为每次测试的结果都不同,我未记录第2、3步中的结果,公共和非公共性能指标均记录自本次结果,运行截图如下: + + 1. kafka-topics.sh + + ![image-20240628232228635](arm_kafka_test.assets/image-20240628232228635.png) + + 2. kafka-producer-perf-test.sh:![image-20240628230657507](arm_kafka_test.assets/image-20240628230657507.png) + + 3. kafka-consumer-perf-test.sh:![image-20240628223551680](arm_kafka_test.assets/image-20240628223551680.png) + +> 因为组长在bash_profile配了这个,所以系统 会在该路径找sh脚本,就无需进入performance目录下执行了![image-20240629002535910](arm_kafka_test.assets/image-20240629002535910.png) + +#### 3.3.1 测试结果 + +| **比较类目** | **CPU** | **ARM(鲲鹏920-7260)** | +| --------------------------- | --------------------- | ------------------------- | +| kafka-topics.sh | duration_time | 1780170866 | +| kafka-topics.sh | task-clock | 3267.100 | +| kafka-topics.sh | cycles | 7202976617 | +| kafka-topics.sh | instructions | 7892467747 | +| kafka-topics.sh | cache-references | 2260246244 | +| kafka-topics.sh | cache-misses | 35076202 | +| kafka-topics.sh | branches |