湖南营销型网站建设多少钱,在线旅游网站平台有哪些,常州酒店网站建设,seo搜索引擎实战详解引言
刚接触大数据以及部分接触大数据多年的伙伴可能从来没有自己搭建过一套属于自己的大数据集群#xff0c;今天就花点时间聊聊怎么快速搭建一套属于自己、且可用于操作、调试的大数据集群
正文
本次搭建的组件都有以下服务以及对应的版本
hadoop#xff08;3.2.4…引言
刚接触大数据以及部分接触大数据多年的伙伴可能从来没有自己搭建过一套属于自己的大数据集群今天就花点时间聊聊怎么快速搭建一套属于自己、且可用于操作、调试的大数据集群
正文
本次搭建的组件都有以下服务以及对应的版本
hadoop3.2.4zookeeper3.9.1kafka2.13-3.6.1
组件下载地址
上述组件都是apache旗下的通过此地址找到对应的版本下载使用即可 https://archive.apache.org/dist/hadoop/common/但如果下载速度慢的话可以考虑通过这个地址进行加速下载 https://mirrors.tuna.tsinghua.edu.cn/apache/后面这个地址仅用于学习请勿用于商用
hadoop
hadoop是大数据最基本的底座在将安装包解压后修改下 ./etc/hadoop 目录下重要的四个配置内容
core-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --
configuration!-- 指定 namenode 的通信地址 --propertynamefs.default.name/namevaluehdfs://localhost:9000/value/property!-- 指定hadoop运行时产生文件的存储路径 --propertynamehadoop.tmp.dir/namevalue/Users/lin/dev/bigdata/hadoop-3.2.4/temp/value/property
/configurationhdfs-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --
configurationpropertynamedfs.permissions.enabled/namevaluefalse/value/propertypropertynamedfs.replication/namevalue1/value/property propertynamedfs.namenode.name.dir/namevalue/Users/lin/dev/bigdata/hadoop-3.2.4/data/namenode/value/propertypropertynamedfs.datanode.data.dir/namevalue/Users/lin/dev/bigdata/hadoop-3.2.4/data/datanode/value/propertypropertynamedfs.namenode.secondary.http-address/namevaluelocalhost:9001/value/propertypropertynamedfs.webhdfs.enabled/namevaluetrue/value/propertypropertynamedfs.http.address/namevalue0.0.0.0:50070/value/property
/configurationyarn-site.xml
?xml version1.0?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--
configurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/property
/configurationmapred-site.xml
?xml version1.0?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --
configurationpropertynamemapreduce.framework.name/namevalueyarn/valuefinaltrue/finaldescriptionThe runtime framework for executing MapReduce jobs/description/propertypropertynameyarn.app.mapreduce.am.env/namevalueHADOOP_MAPRED_HOME/Users/lin/dev/bigdata/hadoop-3.2.4/value/propertypropertynamemapreduce.map.env/namevalueHADOOP_MAPRED_HOME/Users/lin/dev/bigdata/hadoop-3.2.4/value/propertypropertynamemapreduce.reduce.env/namevalueHADOOP_MAPRED_HOME/Users/lin/dev/bigdata/hadoop-3.2.4/value/property
/configuration改完上面四个配置后通过./sbin/start-all.sh指令启动集群通过访问地址 http://localhost:50070 可看到hdfs服务已经正常启动
接下来简单验证下服务是否正常工作
展示文件目录
创建一个自定义目录data
上传一个本地文件到hadoop集群
通过上述演示已完整的部署本地的Hadoop服务
zookeeper
解压后通过指令bin/zkServer.sh start启动服务即可通过指令查询可看到已经启动服务
接下来简单进行验证下首先通过指令bin/zkCli.sh进入客户端
kafka
解压kafka安装包后通过指令nohup bin/kafka-server-start.sh config/server.properties 21 进行服务的后台启动。通过linux指令可以看到kafka服务已经正常启动
接下来进行简单验证下
创建Topic
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic TestKafkaTopic1消费Topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TestKafkaTopic1 --from-beginning写Topic
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TestKafkaTopic1查看消费情况
通过上述几步操作能看到我们的kafka服务也正常工作了
小结
以上就是搭建一个简单的本地调试环境的流程最好是都能手动操作一次对这几个基础服务都有一定的了解
