Apache Kafka专题二、Kafka安装和基本使用.md
一、前置条件
Kafka依赖于Java和zookeeper，需要一起部署。这里我们使用了3台服务器。
10.140.100.14
10.140.100.26
10.140.100.32
主机之间不需要设置hosts，但要注意本机hostname必须对应本机IP地址。
二、安装部署ZooKeeper
ZooKeeper我用的版本是zookeeper-3.4.10。直接下载解压，然后配置文件如下
##@@file: zoo.cfg
#datadir
dataDir=/data/apps/zookeeper-3.4.10/data
#clusterIP
server.1=10.140.100.14:2888:3888
server.2=10.140.100.26:2888:3888
server.3=10.140.100.32:2888:3888然后每个机器将自己的ID写入myid文件
#@以第三台为例
echo "3" >/data/apps/zookeeper-3.4.10/data/myid启动zookeeper
/data/apps/zookeeper-3.4.10/bin/zkServer.sh start三、部署kafka集群
kafka有源码包和二进制包，这里我们直接下载二进制包
cd /data/apps/
wget http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz
tar zxvf kafka_2.11-1.0.0.tgz 解压后，编辑config/server.properties
#@broker.id必须唯一
broker.id=2
#@listeners使用自己的IP
listeners=PLAINTEXT://10.140.100.32:9092
log.dirs=/data/kafka-logs
zookeeper.connect=10.140.100.14:2181,10.140.100.26:2181,10.140.100.32:2181启动kafka
/data/apps/kafka_2.11-1.0.0/bin/kafka-server-start.sh -daemon server.properties四、创建测试topic
[root@kuiswenden-1 bin]# @@创建topic
[root@kuiswenden-1 bin]# ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic test
Created topic "test".
[root@kuiswenden-1 bin]# @@列出topic
[root@kuiswenden-1 bin]# ./kafka-topics.sh --zookeeper localhost:2181 --list
test
test02
[root@kuiswenden-1 bin]# @@查看topic详情
[root@kuiswenden-1 bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
Topic:test PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2
Topic: test Partition: 1 Leader: 2 Replicas: 0,1,2 Isr: 2
Topic: test Partition: 2 Leader: 1 Replicas: 1,2,0 Isr: 1,0,2
[root@kuiswenden-1 bin]# 五、模拟生产者和消费者
生产一部分数据
[root@kuiswenden-3 config]# /data/apps/kafka_2.11-1.0.0/bin/kafka-console-producer.sh --broker-list 10.140.100.32:9092 --topic test
>abcdefghijklmn
>消费端查看数据
[root@kuiswenden-1 config]# /data/apps/kafka_2.11-1.0.0/bin/kafka-console-consumer.sh --zookeeper 0.140.100.14:2181 --from-beginning --topic test
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
abcdefghijklmn六、简单压力测试
[root@kuiswenden-1 ~]# /data/apps/kafka_2.11-1.0.0/bin/kafka-producer-perf-test.sh --topic test --num-records 15000000 --record-size 100 --throughput 75000000 --producer-props acks=1 bootstrap.servers=10.140.100.32:9092,10.140.100.26:9092,10.140.100.14:9092 buffer.memory=67108864 compression.type=none batch.size=8196
2765043 records sent, 553008.6 records/sec (52.74 MB/sec), 70.1 ms avg latency, 388.0 max latency.
3991707 records sent, 798341.4 records/sec (76.14 MB/sec), 2.7 ms avg latency, 22.0 max latency.
4069509 records sent, 813901.8 records/sec (77.62 MB/sec), 2.4 ms avg latency, 52.0 max latency.
3999520 records sent, 799904.0 records/sec (76.28 MB/sec), 3.6 ms avg latency, 30.0 max latency.
15000000 records sent, 742096.670460 records/sec (70.77 MB/sec), 15.27 ms avg latency, 388.00 ms max latency, 2 ms 50th, 143 ms 95th, 221 ms 99th, 285 ms 99.9th.
[root@kuiswenden-1 ~]# 可以看出kafka可以轻松吞吐50-60M数据。
参考资料
http://www.infoq.com/cn/articles/kafka-analysis-part-1?utm_source=infoq
http://blog.csdn.net/shirdrn/article/details/7183503
http://www.jasongj.com/kafka/high_throughput/