Kafka自带的性能测试脚本

Dec 23, 2016 | | Say something

压测写入消息

./kafka-producer-perf-test.sh --topic test_perf --num-records 1000000 --record-size 1000 --throughput 20000 --producer-props bootstrap.servers=localhost:9092
88126 records sent, 17625.2 records/sec (16.81 MB/sec), 968.5 ms avg latency, 1409.0 max latency.
113096 records sent, 22587.6 records/sec (21.54 MB/sec), 249.2 ms avg latency, 681.0 max latency.
63205 records sent, 12615.8 records/sec (12.03 MB/sec), 1200.1 ms avg latency, 2560.0 max latency.
137255 records sent, 27451.0 records/sec (26.18 MB/sec), 381.6 ms avg latency, 902.0 max latency.
100020 records sent, 20000.0 records/sec (19.07 MB/sec), 4.2 ms avg latency, 142.0 max latency.
100020 records sent, 20004.0 records/sec (19.08 MB/sec), 1.1 ms avg latency, 18.0 max latency.
29195 records sent, 5839.0 records/sec (5.57 MB/sec), 1365.7 ms avg latency, 3544.0 max latency.
22680 records sent, 4528.8 records/sec (4.32 MB/sec), 5279.9 ms avg latency, 6772.0 max latency.
81390 records sent, 16100.9 records/sec (15.36 MB/sec), 3051.3 ms avg latency, 7415.0 max latency.
139410 records sent, 27693.7 records/sec (26.41 MB/sec), 1115.1 ms avg latency, 1637.0 max latency.
118290 records sent, 23610.8 records/sec (22.52 MB/sec), 1226.4 ms avg latency, 1804.0 max latency.
1000000 records sent, 17737.415304 records/sec (16.92 MB/sec), 962.23 ms avg latency, 7415.00 ms max latency, 831 ms 50th, 3932 ms 95th, 6894 ms 99th, 7285 ms 99.9th.

kafka-producer-perf-test.sh 脚本命令的参数为:
--topic topic名称,本例为test_perf
--num-records 总共需要发送的消息数,本例为1000000
--record-size 每个记录的字节数,本例为1000
--throughput 每秒钟发送的记录数,本例为20000
--producer-props bootstrap.servers=localhost:9092 发送端的配置信息,本例只指定了kafka的链接信息
可以看到本例中,每秒平均向kafka写入了16.92 MB的数据,大概是17737条消息,每次写入的平均延迟为962.23毫秒,最大的延迟为7415.00毫秒,831 ms内占50%

压测消费消息

[root@kafka bin]# ./kafka-consumer-perf-test.sh --zookeeper localhost:2181 --topic test_perf --fetch-size 1048576 --messages 1000000 --threads 1
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2016-12-24 09:18:26:409, 2016-12-24 09:18:36:231, 553.8979, 56.3936, 580804, 59132.9668

kafka-consumer-perf-test.sh 脚本命令的参数为:
--zookeeper 指定zookeeper的链接信息,本例为localhost:2181 ,如果使用新的纯java客户端则使用另外的配置
--topic 指定topic的名称,本例为test_perf
--fetch-size 指定每次fetch的数据的大小,本例为1048576,也就是1M
--messages 总共要消费的消息个数,本例为1000000,100w
可以看到本例中,总共消费了553.8979M的数据,每秒为56.3936, 总共消费了580804条消息,每秒为59132.9668

大家在压测的时候肯定不会让压测脚本和kafka服务器跑在同一台电脑上,比较压测脚本也要占用很多服务器资源,影响压测准确性。如果没有多余的服务器,也可以这样简单的执行以下,了解kafka在你给定的硬件上的一个大概的吞吐量,本文由javacoder.cn整理,转载请注明出处。

Posted in: Kafka

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">