Kafka之安装配置 （二）

目录
[TOC]

Kafka之安装配置

本篇基于Linux CentOS来介绍安装的，需要先安装好JDK

1 Zookeeper安装配置

Zookeeper是安装Kafka集群的必要组件，Kafka通过Zookeeper来实施对于元数据信息的管理，包括集群、borker、主题、分区等内容。

Zookeeper是一个开源的分布式协调服务。分布式应用程序可以基于Zookeeper实现注入数据发布/订阅、负载均衡、命名服务、分布式协调/通知、集群管理、Master选举、配置维护等功能。

在Zookeeper中共有3个角色：

• leader
• follower
• observer

observer不参与投票，默认情况下Zookeeper中只有leader与follower两个角色

也可以通过我的这篇文章来了解一下：Zookeeper

1.1 下载ZK

官网：https://zookeeper.apache.org/releases.html

1.2 安装配置

下载好安装包后将其复制到/opt目录下，具体命令如下：

#切换到/opt目录下
[root@hopestation opt]# cd /opt
#显示详细信息
[root@hopestation opt]# ll apache-zookeeper-3.6.2-bin.tar.gz 
-rwxr-xr-x 1 root root 12515974 Apr 14 22:55 apache-zookeeper-3.6.2-bin.tar.gz
#解压文件
[root@hopestation opt]# tar zxvf apache-zookeeper-3.6.2-bin.tar.gz 
#进入到解压后的文件夹
[root@hopestation opt]# cd apache-zookeeper-3.6.2-bin
#查看当前目录
[root@hopestation apache-zookeeper-3.6.2-bin]# pwd
/opt/apache-zookeeper-3.6.2-bin

配置环境变量

#编辑/etc/profile
[root@hopestation apache-zookeeper-3.6.2-bin]# vim /etc/profile

• 增加这句话：export ZOOKEEPER_HOME=/opt/apache-zookeeper-3.6.2-bin
• 然后把它放到环境变量里就ok了，注意与已经存在的编辑变量用“冒号”分割

export ZOOKEEPER_HOME=/opt/apache-zookeeper-3.6.2-bin
export PATH=$PATH:/usr/local/lib/erlang/bin:/usr/local/rabbitmq_server-3.8.3/sbin:$ZOOKEEPER_HOME/bin
#然后 wq退出

#使环境变量生效
[root@hopestation apache-zookeeper-3.6.2-bin]# source /etc/profile
#查看环境变量
[root@hopestation apache-zookeeper-3.6.2-bin]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/lib/erlang/bin:/usr/local/rabbitmq_server-3.8.3/sbin:/root/bin:/usr/local/lib/erlang/bin:/usr/local/rabbitmq_server-3.8.3/sbin:/opt/apache-zookeeper-3.6.2-bin/bin

修改配置文件

#进入zk的pwdconf目录
[root@hopestation apache-zookeeper-3.6.2-bin]# cd conf
#当前目录
[root@hopestation conf]# pwd
/opt/apache-zookeeper-3.6.2-bin/conf
#复制一份新的配置文件 因为zk启动会需要名为zoo.cfg的文件
[root@hopestation conf]# cp zoo_sample.cfg zoo.cfg
[root@hopestation conf]# ls
configuration.xsl  log4j.properties  zoo.cfg  zoo_sample.cfg
#编辑zoo.cfg
[root@hopestation conf]# vim zoo.cfg
#然后按i，进入编辑模式。编辑：wq退出

zoo.cfg 内容

# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/tmp/zookeeper/data
dataLogDir=/tmp/zookeeper/log
#其他内容不变

而默认情况下linux是没有这两个目录的，所以需要创建出来

#创建
[root@hopestation conf]# mkdir -p /tmp/zookeeper/data
[root@hopestation conf]# mkdir -p /tmp/zookeeper/log
#查看
[root@hopestation conf]# cd /tmp/zookeeper
[root@hopestation zookeeper]# ls
data  log
#然后需要在 data目录下创建一个myid文件，并写入一个数值作为服务器的编号
[root@hopestation zookeeper]# cd data
[root@hopestation data]# touch myid
[root@hopestation data]# vim myid
#i编写，比如写入0 然后:wq退出
[root@hopestation data]# cat myid
0

启动zookeeper

[root@hopestation conf]# zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.2-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@hopestation conf]# zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/apache-zookeeper-3.6.2-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Error contacting service. It is probably not running.

2 Kafka安装配置

官网下载地址：http://kafka.apache.org/downloads.html

#解压
[root@hopestation opt]# tar zxvf kafka_2.13-2.8.0.tgz 

将kafka目录添加到环境变量里：/opt/kafka_2.13-2.8.0

#编辑
[root@hopestation opt]# vim /etc/profile
export KAFKA_HOME=/opt/kafka_2.13-2.8.0
export ZOOKEEPER_HOME=/opt/apache-zookeeper-3.6.2-bin
export PATH=$PATH:/usr/local/lib/erlang/bin:/usr/local/rabbitmq_server-3.8.3/sbin:$ZOOKEEPER_HOME/bin:$KAFKA_HOME
#生效
[root@hopestation opt]# source /etc/profile

参考下图

修改broker配置文件$KAFKA_HOME/conf/server.properties

[root@hopestation config]# vim server.properties 
#broker的编号，若集群中有多个broker，则每个broker的编号需要设置的不同
broker.id=0
#broker对外提供服务的入口地址
listeners = PLAINTEXT://localhost:9092
#存放消息日志文件的地址
log.dirs=/tmp/kafka-logs
#Kafka所需的Zookeeper集群的地址，我们假设Kafka和Zookeeper都安装在本机
zookeeper.connect=localhost:2181

配置好了之后就可以启动Kafka了

#启动
[root@hopestation kafka_2.13-2.8.0]# bin/kafka-server-start.sh config/server.properties 
#查看是否有Kafak进程
[root@hopestation kafka_2.13-2.8.0]# jps -l
29936 kafka.Kafka #这个就是了
29697 org.apache.zookeeper.server.quorum.QuorumPeerMain

错误及解决

连接Zookeeper失败

[2021-04-24 23:19:12,074] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING

解决

#关闭zookeeper
[root@hopestation ~]# zkServer.sh stop
#再次启动 查看log
[root@hopestation ~]#  zkServer.sh start-foreground
#绑定8080端口 失败，端口占用
org.apache.zookeeper.server.admin.AdminServer$AdminServerException: Problem starting AdminServer on address 0.0.0.0, port 8080 and command URL /commands   org.apache.zookeeper.server.admin.JettyAdminServer.start(JettyAdminServer.java:171)
Caused by: java.net.BindException: Address already in use

• 可以直接关闭掉占用8080端口的程序
• 也可以修改zookeeper的zoo.cfg配置文件

[root@hopestation opt]# cd apache-zookeeper-3.6.2-bin/conf
[root@hopestation conf]# vim zoo.cfg
#增加这句话，使用空闲的端口就可以了
admin.serverPort=2182

参考文献

• 深入理解Kafka-核心设计与实践原理
• Kafka官网：http://kafka.apache.org