淄博网站开发网泰快,seo推广教程seo高级教程,网站架构软件,下列关于网站开发中网友上传单机模式链接#xff1a;https://blog.csdn.net/wsdhla/article/details/133032238 kraft集群模式链接#xff1a;部署Kafka_kafka 部署-CSDN博客 zookeeper选举机制举例#xff1a; 目前有5台服务器#xff0c;每台服务器均没有数据#xff0c;它们的编号分别是1,2,3,4,5…单机模式链接https://blog.csdn.net/wsdhla/article/details/133032238 kraft集群模式链接部署Kafka_kafka 部署-CSDN博客 zookeeper选举机制举例 目前有5台服务器每台服务器均没有数据它们的编号分别是1,2,3,4,5,按编号依次启动它们的选择举过程如下 服务器1启动给自己投票然后发投票信息由于其它机器还没有启动所以它收不到反馈信息服务器1的状态一直属于Looking(选举状态)。服务器2启动给自己投票同时与之前启动的服务器1交换结果由于服务器2的编号大所以服务器2胜出但此时投票数没有大于半数所以两个服务器的状态依然是LOOKING。服务器3启动给自己投票同时与之前启动的服务器1,2交换信息由于服务器3的编号最大所以服务器3胜出此时投票数正好大于半数所以服务器3成为领导者服务器1,2成为小弟。服务器4启动给自己投票同时与之前启动的服务器1,2,3交换信息尽管服务器4的编号大但之前服务器3已经胜出所以服务器4只能成为小弟。服务器5启动后面的逻辑同服务器4成为小弟。 Docker 自定义静态网络 Docker默认提供了3种网络模式生成容器时不指定网络模式下默认使用bridge桥接模式。每次容器启动时ip会发生变更无法达到我们集群的效果。 创建自定义网络模式指定网段。 docker network create --subnet172.18.0.0/16 net-kafka 若报错Error response from daemon: Pool overlaps with other one on this address space 说明指定的网络地址段已存在 1、Docker部署zookeeper集群
zk使用官方镜像 1.1、主机创建目录
[zk01]
D:/mysoft/dev/docker-zk-cluster/zk01/conf
D:/mysoft/dev/docker-zk-cluster/zk01/data
D:/mysoft/dev/docker-zk-cluster/zk01/datalog[zk02]
D:/mysoft/dev/docker-zk-cluster/zk02/conf
D:/mysoft/dev/docker-zk-cluster/zk02/data
D:/mysoft/dev/docker-zk-cluster/zk02/datalog[zk03]
D:/mysoft/dev/docker-zk-cluster/zk03/conf
D:/mysoft/dev/docker-zk-cluster/zk03/data
D:/mysoft/dev/docker-zk-cluster/zk03/datalog
1.2、修改配置文件
分别拷贝到zk01、zk02、zk03
[zoo.cfg]dataDir/data
dataLogDir/datalog
tickTime2000
initLimit5
syncLimit2
autopurge.snapRetainCount3
autopurge.purgeInterval0
maxClientCnxns60
standaloneEnabledtrue
admin.enableServertrue
server.1172.18.0.11:2888:3888;2181
server.2172.18.0.12:2888:3888;2181
server.3172.18.0.13:2888:3888;2181
1.3、指定ip创建启动容器
docker run -d --name zk01 -p 2181:2181 --network net-kafka --ip 172.18.0.11 -v D:/mysoft/dev/docker-zk-cluster/zk01/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk01/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk01/datalog:/datalog zookeeperdocker run -d --name zk02 -p 2182:2181 --network net-kafka --ip 172.18.0.12 -v D:/mysoft/dev/docker-zk-cluster/zk02/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk02/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk02/datalog:/datalog zookeeperdocker run -d --name zk03 -p 2183:2181 --network net-kafka --ip 172.18.0.13 -v D:/mysoft/dev/docker-zk-cluster/zk03/conf:/conf -v D:/mysoft/dev/docker-zk-cluster/zk03/data:/data -v D:/mysoft/dev/docker-zk-cluster/zk03/datalog:/datalog zookeeper
zk01可以正常启动zk02和zk03启动失败 警告这个正常是因为节点启动时选举机制的原因 zk02 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 2 at election address /172.18.0.12:3888 zk03 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888 异常 zk02 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Exception while listening to address /172.18.0.11:3888 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Leaving listener thread for address 172.18.0.11:3888 after 3 errors. Use zookeeper.electionPortBindRetry property to increase retry count. ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - As Im leaving the listener thread, I wont be able to participate in leader election any longer: 172.18.0.11:3888 zk03 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Exception while listening to address /172.18.0.11:3888 ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - Leaving listener thread for address 172.18.0.11:3888 after 3 errors. Use zookeeper.electionPortBindRetry property to increase retry count. ERROR org.apache.zookeeper.server.quorum.QuorumCnxManager - As Im leaving the listener thread, I wont be able to participate in leader election any longer: 172.18.0.11:3888 原因是没有配置quorumListenOnAllIPs默认为false需要设置为true
[zoo.cfg]dataDir/data
dataLogDir/datalog
tickTime2000
initLimit5
syncLimit2
autopurge.snapRetainCount3
autopurge.purgeInterval0
maxClientCnxns60
standaloneEnabledtrue
admin.enableServertrue
quorumListenOnAllIPstrue
server.1172.18.0.11:2888:3888;2181
server.2172.18.0.12:2888:3888;2181
server.3172.18.0.13:2888:3888;2181 quorumListenOnAllIPs 当设置为true时ZooKeeper服务器将在所有可用IP地址上侦听来自其对等方的连接而不仅是在配置文件的服务器列表中配置的地址。它会影响处理ZAB协议和快速领导者选举协议的连接。默认值为false。 有一些WARN:
### 1、只启动zk01
# zk01
01:21:17.746 [QuorumConnectionThread-[myid1]-3] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 2 at election address /172.18.0.12:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)01:21:20.904 [QuorumConnectionThread-[myid1]-1] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)### 2、 只启动zk01、zk02
# zk01
01:21:46.504 [QuorumConnectionThread-[myid1]-3] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)# zk02
01:21:43.350 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.01:21:48.824 [QuorumConnectionThread-[myid1]-2] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /172.18.0.13:3888
java.net.NoRouteToHostException: No route to host (Host unreachable)### 2、 启动zk01、zk02、zk03
# zk01正常
01:31:34.579 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:31:34.579 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:31:34.580 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
01:31:34.584 [QuorumConnectionThread-[myid1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
01:31:34.584 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
01:31:34.585 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
01:31:34.585 [QuorumConnectionThread-[myid1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
01:31:34.585 [QuorumConnectionThread-[myid1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
01:31:34.585 [QuorumConnectionThread-[myid1]-21] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 -- sid:2)
01:31:34.586 [QuorumConnectionThread-[myid1]-21] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 -- sid:2)
01:31:34.585 [QuorumConnectionThread-[myid1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
01:31:34.586 [QuorumConnectionThread-[myid1]-22] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 -- sid:3)
01:31:34.586 [QuorumConnectionThread-[myid1]-22] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 -- sid:3)
01:31:52.173 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - node0 scavenging sessions
01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.SessionHandler873610597dftMaxIdleSec-1 scavenging sessions
01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.SessionHandler873610597dftMaxIdleSec-1 scavenging session ids []
01:31:52.174 [Session-HouseKeeper-383bfa16-1] DEBUG org.eclipse.jetty.server.session - org.eclipse.jetty.server.session.DefaultSessionCache57d7f8ca[evict-1,removeUnloadablefalse,saveOnCreatefalse,saveOnInactiveEvictfalse] checking expiration on []# zk02
01:35:15.331 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:35:15.332 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:35:15.333 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
01:35:15.334 [QuorumConnectionThread-[myid1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
01:35:15.334 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
01:35:15.335 [QuorumConnectionThread-[myid1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
01:35:15.335 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.12:50246
01:35:15.335 [QuorumConnectionThread-[myid1]-27] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 -- sid:2)
01:35:15.335 [QuorumConnectionThread-[myid1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.12:50246
01:35:15.335 [QuorumConnectionThread-[myid1]-27] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 -- sid:2)
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid1, electionAddr[/172.18.0.11:3888]}
01:35:15.335 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:35:15.335 [QuorumConnectionThread-[myid1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
01:35:15.335 [QuorumConnectionThread-[myid1]-28] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 -- sid:3)
01:35:15.336 [QuorumConnectionThread-[myid1]-28] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 -- sid:3)
01:35:34.600 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.11:49750
01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.11:49750
01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid1, electionAddr[/172.18.0.11:3888]}
01:35:34.601 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:35:59.199 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.13:56956
01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.13:56956
01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid1, electionAddr[/172.18.0.11:3888]}
01:35:59.200 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.# zk03
01:36:59.201 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:36:59.201 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Queue size: 1
01:36:59.202 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 2 already, it is in the lastCommittedView
01:36:59.202 [QuorumConnectionThread-[myid1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 2
01:36:59.203 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Server 1 knows 3 already, it is in the lastCommittedView
01:36:59.203 [QuorumConnectionThread-[myid1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 2 using election address: /172.18.0.12:3888
01:36:59.203 [QuorumConnectionThread-[myid1]-15] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 -- sid:2)
01:36:59.203 [QuorumPeer[myid1](plain0.0.0.0:2181)(securedisabled)] INFO org.apache.zookeeper.server.quorum.FastLeaderElection - Notification time out: 60000 ms
01:36:59.203 [QuorumConnectionThread-[myid1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Opening channel to server 3
01:36:59.203 [QuorumConnectionThread-[myid1]-15] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 -- sid:2)
01:36:59.204 [QuorumConnectionThread-[myid1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Connected to server 3 using election address: /172.18.0.13:3888
01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.13:35192
01:36:59.204 [QuorumConnectionThread-[myid1]-16] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - startConnection (myId:1 -- sid:3)
01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.13:35192
01:36:59.204 [QuorumConnectionThread-[myid1]-16] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Have smaller server identifier, so dropping the connection: (myId:1 -- sid:3)
01:36:59.204 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid1, electionAddr[/172.18.0.11:3888]}
01:36:59.205 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.12:35956
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.12:35956
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid1, electionAddr[/172.18.0.11:3888]}
01:37:15.342 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] INFO org.apache.zookeeper.server.quorum.QuorumCnxManager - Received connection request from /172.18.0.11:46420
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Sync handling of connection request received from: /172.18.0.11:46420
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] DEBUG org.apache.zookeeper.server.quorum.QuorumCnxManager - Initial message parsed by 1: InitialMessage{sid1, electionAddr[/172.18.0.11:3888]}
01:37:34.609 [ListenerHandler-0.0.0.0/0.0.0.0:3888] WARN org.apache.zookeeper.server.quorum.QuorumCnxManager - We got a connection request from a server with our own ID. This should be either a configuration error, or a bug.找到原因了因为data/myid没有修改目前3个节点的配置全部为1分别改为1、2、3就正常了。 启动zkui
docker run -d --name zkui -p 9090:9090 --network net-kafka -e ZKLIST172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 maauso/zkui
访问localhost:9090登录之后报错KeeperErrorCode ConnectionLoss for /
删除data\version-2\snapshot.0依然不行 stat is not executed because it is not in the whitelist. envi is not executed because it is not in the whitelist.
4lw.commands.whitelist* 16:49:58.466 [NIOWorkerThread-10] DEBUG org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /172.18.0.2:42978 (no session established for client)
16:49:58.499 [NIOServerCxnFactory.AcceptThread:/0.0.0.0:2181] DEBUG org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket connection from /172.18.0.2:42980
16:49:58.500 [NIOWorkerThread-11] WARN org.apache.zookeeper.server.NIOServerCnxn - Close of session 0x0
java.io.IOException: ZooKeeperServer not runningat org.apache.zookeeper.server.NIOServerCnxn.readLength(NIOServerCnxn.java:569)at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:349)at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:153)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)at java.base/java.lang.Thread.run(Unknown Source)
好像改为zk的data/myid配置上面的问题就没有了。 2、Docker部署kafka集群
kafka没有官方镜像使用wurstmeister/kafka镜像 2.1、主机创建目录
[kafka01]
D:/mysoft/dev/docker-kafka-cluster/kafka01/config
D:/mysoft/dev/docker-kafka-cluster/kafka01/data
D:/mysoft/dev/docker-kafka-cluster/kafka01/logs
D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts[kafka02]
D:/mysoft/dev/docker-kafka-cluster/kafka02/config
D:/mysoft/dev/docker-kafka-cluster/kafka02/data
D:/mysoft/dev/docker-kafka-cluster/kafka02/logs
D:/mysoft/dev/docker-kafka-cluster/kafka02/hosts[kafka03]
D:/mysoft/dev/docker-kafka-cluster/kafka03/config
D:/mysoft/dev/docker-kafka-cluster/kafka03/data
D:/mysoft/dev/docker-kafka-cluster/kafka03/logs
D:/mysoft/dev/docker-kafka-cluster/kafka03/hosts
2.2、修改配置文件
分别拷贝到kafka01、kafka02、kafka03
[server.properties]broker.id1
advertised.listenersPLAINTEXT://172.18.0.21:9092
# port9092
# log.dirs/kafka/kafka-logs-1ddafafe54d0
listenersPLAINTEXT://0.0.0.0:9092
zookeeper.connect172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181
2.3、指定ip创建启动容器
# docker run -d --name kafka01 --restartalways --nethost -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc/hosts -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_HOST_NAME192.168.180.42 -e JMX_PORT9999 -e HOST_IP127.0.0.1 -e KAFKA_ADVERTISED_PORT9092 -e KAFKA_ZOOKEEPER_CONNECT127.0.0.1:2181 -e KAFKA_BROKER_ID1 -e KAFKA_HEAP_OPTS-Xmx6G -Xms6G wurstmeister/kafka
# docker run -d --name kafka01 --restartalways --network net-kafka --ip 172.18.0.21 -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_HOST_NAME172.18.0.21 -e KAFKA_ADVERTISED_PORT9092 -e KAFKA_ADVERTISED_LISTENERSPLAINTEXT://172.18.0.21:9092 -e KAFKA_LISTENERSPLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID1 -e KAFKA_LOG_DIRS/opt/kafka/logs wurstmeister/kafkadocker run -d --name kafka01 -p 9192:9092 --restartalways --network net-kafka --ip 172.18.0.21 -v D:/mysoft/dev/docker-kafka-cluster/kafka01/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka01/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka01/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka01/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERSPLAINTEXT://172.18.0.21:9092 -e KAFKA_LISTENERSPLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID1 wurstmeister/kafkadocker run -d --name kafka02 -p 9292:9092 --restartalways --network net-kafka --ip 172.18.0.22 -v D:/mysoft/dev/docker-kafka-cluster/kafka02/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka02/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka02/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka02/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERSPLAINTEXT://172.18.0.22:9092 -e KAFKA_LISTENERSPLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID2 wurstmeister/kafkadocker run -d --name kafka03 -p 9392:9092 --restartalways --network net-kafka --ip 172.18.0.23 -v D:/mysoft/dev/docker-kafka-cluster/kafka03/hosts:/etc -v D:/mysoft/dev/docker-kafka-cluster/kafka03/data:/opt/kafka/data -v D:/mysoft/dev/docker-kafka-cluster/kafka03/config:/opt/kafka/config -v D:/mysoft/dev/docker-kafka-cluster/kafka03/logs:/opt/kafka/logs -e KAFKA_ADVERTISED_LISTENERSPLAINTEXT://172.18.0.23:9092 -e KAFKA_LISTENERSPLAINTEXT://0.0.0.0:9092 -e KAFKA_ZOOKEEPER_CONNECT172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 -e KAFKA_BROKER_ID3 wurstmeister/kafka kafka容器异常一直自动反复重启UP状态维持1分钟左右 [Configuring] advertised.listeners in /opt/kafka/config/server.properties
[Configuring] port in /opt/kafka/config/server.properties
Excluding KAFKA_HOME from broker config
[Configuring] log.dirs in /opt/kafka/config/server.properties
[Configuring] listeners in /opt/kafka/config/server.properties
Excluding KAFKA_VERSION from broker config
[Configuring] zookeeper.connect in /opt/kafka/config/server.properties
[Configuring] broker.id in /opt/kafka/config/server.properties
log4j:ERROR Could not read configuration file from URL [file:/opt/kafka/bin/../config/log4j.properties].
java.io.FileNotFoundException: /opt/kafka/bin/../config/log4j.properties (No such file or directory)at java.base/java.io.FileInputStream.open0(Native Method)at java.base/java.io.FileInputStream.open(Unknown Source)at java.base/java.io.FileInputStream.init(Unknown Source)at java.base/java.io.FileInputStream.init(Unknown Source)at java.base/sun.net.www.protocol.file.FileURLConnection.connect(Unknown Source)at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(Unknown Source)at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)at org.apache.log4j.LogManager.clinit(LogManager.java:127)at org.slf4j.impl.Log4jLoggerFactory.init(Log4jLoggerFactory.java:66)at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:72)at org.slf4j.impl.StaticLoggerBinder.clinit(StaticLoggerBinder.java:45)at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)at kafka.utils.Log4jControllerRegistration$.clinit(Logging.scala:25)at kafka.Kafka$.clinit(Kafka.scala:30)at kafka.Kafka.main(Kafka.scala)
log4j:ERROR Ignoring configuration file [file:/opt/kafka/bin/../config/log4j.properties].
log4j:WARN No appenders could be found for logger (kafka.utils.Log4jControllerRegistration$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
指定-e KAFKA_LOG_DIRS也依然不可以手动拷贝一份log4j.properties到映射的目录D:/mysoft/dev/docker-kafka-cluster/kafka01/logs下问题就解决了。 但是出现另外一个异常
[2023-12-23 15:20:50,136] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,204] INFO Socket connection established, initiating session, client: /172.18.0.21:39814, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,273] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,749] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,811] INFO Socket connection established, initiating session, client: /172.18.0.21:49980, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:50,914] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,268] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,356] INFO Socket connection established, initiating session, client: /172.18.0.21:40344, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,446] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,617] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2023-12-23 15:20:52,817] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:52,855] INFO Socket connection established, initiating session, client: /172.18.0.21:39828, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:53,036] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2023-12-23 15:20:53,037] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2023-12-23 15:20:53,076] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2023-12-23 15:20:53,222] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTINGat kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:271)at kafka.zookeeper.ZooKeeperClient.init(ZooKeeperClient.scala:125)at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)at kafka.server.KafkaServer.startup(KafkaServer.scala:191)at kafka.Kafka$.main(Kafka.scala:109)at kafka.Kafka.main(Kafka.scala)
[2023-12-23 15:20:53,290] INFO shutting down (kafka.server.KafkaServer)
[2023-12-23 15:20:53,375] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2023-12-23 15:20:53,440] INFO shut down completed (kafka.server.KafkaServer)
[2023-12-23 15:20:53,515] ERROR Exiting Kafka. (kafka.Kafka$)
[2023-12-23 15:20:53,590] INFO shutting down (kafka.server.KafkaServer)
回头看zk的控制台发现报错java.io.IOException: ZooKeeperServer not running跟用zkui客户端连zk集群一样的错通过修改zk的data/myid配置之后3个kafka服务都能正常启动了而且也不会反复自动重启了。
# kafka01
[2023-12-25 02:08:01,176] INFO [KafkaServer id1] started (kafka.server.KafkaServer)# kafka02
[2023-12-25 02:09:56,391] INFO [KafkaServer id2] started (kafka.server.KafkaServer)# kafka03
[2023-12-25 02:10:14,479] INFO [KafkaServer id3] started (kafka.server.KafkaServer) 虽然kafka服务在启动的初始阶段仍然有一下异常但是目前没有发现有什么影响 [2023-12-25 00:52:15,882] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:15,932] INFO Socket connection established, initiating session, client: /172.18.0.21:49568, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:15,984] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:17,040] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:17,094] INFO Socket connection established, initiating session, client: /172.18.0.21:59734, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:17,148] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:17,979] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:18,033] INFO Socket connection established, initiating session, client: /172.18.0.21:50098, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:18,088] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:19,308] INFO Opening socket connection to server zk01.net-kafka/172.18.0.11:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:19,357] INFO Socket connection established, initiating session, client: /172.18.0.21:49580, server: zk01.net-kafka/172.18.0.11:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:19,403] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:19,974] INFO Opening socket connection to server zk03.net-kafka/172.18.0.13:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:20,027] INFO Socket connection established, initiating session, client: /172.18.0.21:59746, server: zk03.net-kafka/172.18.0.13:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:20,075] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:20,549] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2023-12-25 00:52:20,812] INFO Opening socket connection to server zk02.net-kafka/172.18.0.12:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:20,860] INFO Socket connection established, initiating session, client: /172.18.0.21:50110, server: zk02.net-kafka/172.18.0.12:2181 (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:21,016] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2023-12-25 00:52:21,017] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2023-12-25 00:52:21,062] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2023-12-25 00:52:21,164] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTINGat kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:271)at kafka.zookeeper.ZooKeeperClient.init(ZooKeeperClient.scala:125)at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:431)at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:456)at kafka.server.KafkaServer.startup(KafkaServer.scala:191)at kafka.Kafka$.main(Kafka.scala:109)at kafka.Kafka.main(Kafka.scala)
[2023-12-25 00:52:21,213] INFO shutting down (kafka.server.KafkaServer)
[2023-12-25 00:52:21,274] INFO App info kafka.server for 1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2023-12-25 00:52:21,319] INFO shut down completed (kafka.server.KafkaServer)
[2023-12-25 00:52:21,371] ERROR Exiting Kafka. (kafka.Kafka$)
[2023-12-25 00:52:21,425] INFO shutting down (kafka.server.KafkaServer) 启动sheepkiller/kafka-manager
# docker run -d --name kafka-manager -p 9000:9000 --network net-kafka --ip 172.18.0.31 -e ZK_HOSTS172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 sheepkiller/kafka-managerdocker run -d --name kafka-manager -p 9000:9000 --network net-kafka -e ZK_HOSTS172.18.0.11:2181,172.18.0.12:2181,172.18.0.13:2181 sheepkiller/kafka-manager 或者使用provectuslabs/kafka-ui:latest
# docker run -d --name kafka-ui -p 8080:8080 --network net-kafka --ip 172.18.0.32 -e DYNAMIC_CONFIG_ENABLEDtrue provectuslabs/kafka-uidocker run -d --name kafka-ui -p 8080:8080 --network net-kafka -e DYNAMIC_CONFIG_ENABLEDtrue provectuslabs/kafka-ui
