下载类网站做多久才有流量,速拓科技是做网站,页面设计蓝色与什么颜色搭配最和谐,网站空间使用方法基于Docker和YARN的大数据环境部署实践
目的
本操作手册旨在指导用户通过Docker容器技术#xff0c;快速搭建一个完整的大数据环境。该环境包含以下核心组件#xff1a;
Hadoop HDFS/YARN#xff08;分布式存储与资源调度#xff09;Spark on YARN#xff08;分布式计算…基于Docker和YARN的大数据环境部署实践
目的
本操作手册旨在指导用户通过Docker容器技术快速搭建一个完整的大数据环境。该环境包含以下核心组件
Hadoop HDFS/YARN分布式存储与资源调度Spark on YARN分布式计算Kafka消息队列Hive数据仓库JupyterLab交互式开发环境
通过清晰的步骤说明和验证方法读者将掌握
容器网络的搭建WeaveDocker Compose编排文件编写技巧多组件协同工作的配置要点集群扩展与验证方法 整体架构 #mermaid-svg-EqwYiwbZncaxM6M0 {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-EqwYiwbZncaxM6M0 .error-icon{fill:#552222;}#mermaid-svg-EqwYiwbZncaxM6M0 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-EqwYiwbZncaxM6M0 .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-EqwYiwbZncaxM6M0 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-EqwYiwbZncaxM6M0 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-EqwYiwbZncaxM6M0 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-EqwYiwbZncaxM6M0 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-EqwYiwbZncaxM6M0 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-EqwYiwbZncaxM6M0 .marker.cross{stroke:#333333;}#mermaid-svg-EqwYiwbZncaxM6M0 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-EqwYiwbZncaxM6M0 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-EqwYiwbZncaxM6M0 .cluster-label text{fill:#333;}#mermaid-svg-EqwYiwbZncaxM6M0 .cluster-label span{color:#333;}#mermaid-svg-EqwYiwbZncaxM6M0 .label text,#mermaid-svg-EqwYiwbZncaxM6M0 span{fill:#333;color:#333;}#mermaid-svg-EqwYiwbZncaxM6M0 .node rect,#mermaid-svg-EqwYiwbZncaxM6M0 .node circle,#mermaid-svg-EqwYiwbZncaxM6M0 .node ellipse,#mermaid-svg-EqwYiwbZncaxM6M0 .node polygon,#mermaid-svg-EqwYiwbZncaxM6M0 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-EqwYiwbZncaxM6M0 .node .label{text-align:center;}#mermaid-svg-EqwYiwbZncaxM6M0 .node.clickable{cursor:pointer;}#mermaid-svg-EqwYiwbZncaxM6M0 .arrowheadPath{fill:#333333;}#mermaid-svg-EqwYiwbZncaxM6M0 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-EqwYiwbZncaxM6M0 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-EqwYiwbZncaxM6M0 .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-EqwYiwbZncaxM6M0 .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-EqwYiwbZncaxM6M0 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-EqwYiwbZncaxM6M0 .cluster text{fill:#333;}#mermaid-svg-EqwYiwbZncaxM6M0 .cluster span{color:#333;}#mermaid-svg-EqwYiwbZncaxM6M0 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-EqwYiwbZncaxM6M0 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} spark-submit 资源分配 执行任务 生产/消费 读写数据 元数据查询 Jupyter RM NM1 DN1 Kafka Spark HDFS Hive MySQL 组件功能表
组件名称功能描述依赖服务端口配置数据存储Hadoop NameNodeHDFS元数据管理无9870 (Web UI), 8020Docker卷: hadoop_namenodeHadoop DataNodeHDFS数据存储节点NameNode9864 (数据传输)本地卷或Docker卷YARN ResourceManager资源调度与管理NameNode8088 (Web UI), 8032无YARN NodeManager单个节点资源管理ResourceManager8042 (Web UI)无Spark (YARN模式)分布式计算框架YARN ResourceManager无集成在YARN中JupyterLab交互式开发环境Spark, YARN8888 (Web UI)本地目录挂载Kafka分布式消息队列ZooKeeper9092 (Broker)Docker卷:kafka_data、kafka_logsHive数据仓库服务HDFS, MySQL10000 (HiveServer2)MySQL存储元数据MySQL存储Hive元数据无3306Docker卷: mysql_dataZooKeeper分布式协调服务Kafka依赖无2181Docker卷:zookeeper_data 关键交互流程 数据存储 HDFS通过NameNode管理元数据DataNode存储实际数据。JupyterLab通过挂载本地目录访问数据同时可读写HDFS。 资源调度 Spark作业通过YARN ResourceManager申请资源由NodeManager执行任务。 数据处理 Kafka接收实时数据流Spark消费后进行实时计算。Hive通过HDFS存储表数据元数据存储在MySQL。
环境搭建步骤
1. 容器网络准备Weave
# 安装Weave网络插件
sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod x /usr/local/bin/weave
# 启动Weave网络
weave launch
# 验证网络状态
weave status
#在其他节点上运行
weave launch 主节点IP2. Docker Compose编排文件
创建 docker-compose.yml核心配置如下
version: 3.8services:# ZooKeeperzookeeper-1:image: bitnami/zookeeper:3.8.0privileged: true #使用二进制文件安装的docker需要开启特权模式每个容器都需要开启该模式container_name: zookeeper-1hostname: zookeeper-1ports:- 2181:2181environment:- ALLOW_ANONYMOUS_LOGINyes- TZAsia/Shanghaivolumes:- zookeeper_data:/bitnami/zookeepernetworks:- bigdata-netdns:- 172.17.0.1restart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7# Kafkakafka-1:image: bitnami/kafka:3.3.1container_name: kafka-1hostname: kafka-1environment:- KAFKA_BROKER_ID1- KAFKA_CFG_ZOOKEEPER_CONNECTzookeeper-1:2181- ALLOW_PLAINTEXT_LISTENERyes- TZAsia/Shanghaiports:- 9092:9092volumes:- kafka_data:/bitnami/kafka # Kafka数据持久化- kafka_logs:/kafka-logs # 独立日志目录depends_on:- zookeeper-1networks:- bigdata-netdns:- 172.17.0.1restart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7# Hadoop HDFShadoop-namenode:image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8container_name: hadoop-namenodehostname: hadoop-namenodeenvironment:- CLUSTER_NAMEbigdata- CORE_CONF_fs_defaultFShdfs://hadoop-namenode:8020- HDFS_CONF_dfs_replication2- TZAsia/Shanghaiports:- 9870:9870- 8020:8020networks:- bigdata-netdns:- 172.17.0.1volumes:- hadoop_namenode:/hadoop/dfs/namerestart: alwayshadoop-datanode:image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8container_name: hadoop-datanodehostname: hadoop-datanodeenvironment:- CORE_CONF_fs_defaultFShdfs://hadoop-namenode:8020- HDFS_CONF_dfs_replication2- TZAsia/Shanghaidepends_on:- hadoop-namenodenetworks:- bigdata-netdns:- 172.17.0.1restart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7# YARNhadoop-resourcemanager:image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8container_name: hadoop-resourcemanagerhostname: hadoop-resourcemanagerports:- 8088:8088 # YARN Web UIenvironment:- CORE_CONF_fs_defaultFShdfs://hadoop-namenode:8020- YARN_CONF_yarn_resourcemanager_hostnamehadoop-resourcemanager- TZAsia/Shanghaidepends_on:- hadoop-namenodenetworks:- bigdata-netdns:- 172.17.0.1restart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7hadoop-nodemanager:image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8container_name: hadoop-nodemanagerhostname: hadoop-nodemanagerenvironment:- CORE_CONF_fs_defaultFShdfs://hadoop-namenode:8020- YARN_CONF_yarn_resourcemanager_hostnamehadoop-resourcemanager- TZAsia/Shanghaidepends_on:- hadoop-resourcemanagernetworks:- bigdata-netdns:- 172.17.0.1volumes:- ./hadoop-conf/yarn-site.xml:/etc/hadoop/yarn-site.xml # 挂载主节点的Hadoop配置文件,用于上报内存与cpu核心数restart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7# Hivehive:image: bde2020/hive:2.3.2container_name: hivehostname: hiveenvironment:- HIVE_METASTORE_URIthrift://hive:9083- SERVICE_PRECONDITIONhadoop-namenode:8020,mysql:3306- TZAsia/Shanghaiports:- 10000:10000- 9083:9083depends_on:- hadoop-namenode- mysqlnetworks:- bigdata-netdns:- 172.17.0.1restart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7# MySQLmysql:image: mysql:8.0container_name: mysqlenvironment:- MYSQL_ROOT_PASSWORDroot- MYSQL_DATABASEmetastore- TZAsia/Shanghaiports:- 3306:3306networks:- bigdata-netdns:- 172.17.0.1volumes:- mysql_data:/var/lib/mysqlrestart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7# JupyterLab集成Spark on YARNjupyter:image: jupyter/all-spark-notebook:latestcontainer_name: jupyter-labenvironment:- JUPYTER_ENABLE_LAByes- TZAsia/Shanghai- SPARK_OPTS--master yarn --deploy-mode client # 默认使用YARN模式- HADOOP_CONF_DIR/etc/hadoop/conf # 必须定义- YARN_CONF_DIR/etc/hadoop/conf # 必须定义ports:- 8888:8888volumes:- ./notebooks:/home/jovyan/work- /path/to/local/data:/data- ./hadoop-conf:/etc/hadoop/conf # 挂载Hadoop配置文件./hadoop-conf代表在docker-compose.yml同目录下的hadoop-confnetworks:- bigdata-netdns:- 172.17.0.1depends_on:- hadoop-resourcemanager- hadoop-namenoderestart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 7volumes:hadoop_namenode:mysql_data:zookeeper_data:kafka_data:kafka_logs:hadoop-nodemanager:networks:bigdata-net:external: truename: weaveHadoop配置文件
yarn-site.xml:
configurationpropertynameyarn.resourcemanager.hostname/namevaluehadoop-resourcemanager/value/propertypropertynameyarn.resourcemanager.address/namevaluehadoop-resourcemanager:8032/value/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/property
/configurationcore-site.xml:
configurationpropertynamefs.defaultFS/namevaluehdfs://hadoop-namenode:8020/value/property
/configuration将这两个文件放置到hadoop-conf目录下。
3. 启动服务
# 启动容器
docker-compose up -d# 查看容器状态
docker-compose ps4. 验证服务是否可用
验证HDFS
(1) 访问HDFS Web UI 操作浏览器打开 http://localhost:9870。 预期结果 Overview 页面显示HDFS总容量。 Datanodes 显示至少1个活跃节点对应 hadoop-datanode 容器。
(2) 命令行操作HDFS
docker exec -it hadoop-namenode bash
# 创建测试目录
hdfs dfs -mkdir /test
# 上传本地文件
echo hello hdfs test.txt
hdfs dfs -put test.txt /test/
# 查看文件
hdfs dfs -ls /test
#解除安全模式
hdfs dfsadmin -safemode leave预期结果成功创建目录、上传文件并列出文件。
如图所示 ** 验证YARN**
(1) 访问YARN ResourceManager Web UI
操作浏览器打开 http://localhost:8088。预期结果 Cluster Metrics 显示总资源如内存、CPU。Nodes 显示至少1个NodeManager对应 hadoop-nodemanager 容器。
(2) 提交测试作业到YARN
# 进入Jupyter容器提交Spark作业
docker exec -it jupyter-lab bash
# 提交Spark Pi示例作业
spark-submit --master yarn --deploy-mode client --class org.apache.spark.examples.SparkPi /usr/local/spark/examples/jars/spark-examples_*.jar 10预期结果 作业输出中包含 Pi is roughly 3.14。在YARN Web UI (http://localhost:8088) 中看到作业状态为 SUCCEEDED。
如图所示
若是出现报错Permission denied: userjovyan, accessWRITE, inode“/user”:root:supergroup:drwxr-xr-x
报错原因
当前运行 Spark 的用户是jovyanJupyter 默认用户
Spark 提交任务后会自动尝试在 HDFS 上创建目录 /user/jovyan
但是这个目录不存在或者 /user 目录不允许 jovyan 写入
所以 HDFS 拒绝创建临时目录导致整个作业提交失败
解决方法
创建目录并赋权
进入NameNode 容器
docker exec -it hadoop-namenode bash然后执行 HDFS 命令
hdfs dfs -mkdir -p /user/jovyan
hdfs dfs -chown jovyan:supergroup /user/jovyan这一步允许 jovyan 用户有权写入自己的临时目录。 提示可以先执行 hdfs dfs -ls /user 看是否有 jovyan 子目录。 最后再次执行 spark-submit 后可以看到 控制台打印 Submitting application application_xxx to ResourceManagerYARN 8088 页面 出现作业记录状态为 RUNNING 或 FINISHED
验证Spark on YARN通过JupyterLab
(1) 访问JupyterLab
操作浏览器打开 http://localhost:8888使用Token登录通过 docker logs jupyter-lab 获取Token。预期结果成功进入JupyterLab界面。
(2) 运行PySpark代码
在Jupyter中新建Notebook执行以下代码
from pyspark.sql import SparkSession
spark SparkSession.builder \.appName(jupyter-yarn-test) \.getOrCreate()# 测试Spark Context
print(Spark Version:, spark.version)
print(YARN Cluster Mode:, spark.sparkContext.master)# 读取HDFS文件
df spark.read.text(hdfs://hadoop-namenode:8020/test/test.txt)
df.show()# 读取数据
local_df spark.read.csv(/data/example.csv, headerTrue) # 替换为实际文件路径
local_df.show()预期结果 输出Spark版本和YARN模式如 yarn。成功读取HDFS文件并显示内容 hello hdfs。成功读取CSV文件需提前放置测试文件。
如图所示
验证Hive
(1) 创建Hive表并查询
#使用docker cp命令将jdbc驱动放入容器内部示例
docker cp mysql-connector-java-8.0.12.jar 容器ID或容器名称:/opt/hive/lib
docker exec -it hive bash
#重新初始化 Hive Metastore
schematool -dbType mysql -initSchema --verbose
#查询MetaStore运行状态
ps -ef | grep MetaStore
# 启动Hive Beeline客户端
beeline -u jdbc:hive2://localhost:10000 -n root//驱动下载链接https://downloads.mysql.com/archives/c-j/
若是执行上述命令报错可以按照以下步骤来进行更改
1、配置hive-site.xml文件
?xml version1.0 encodingUTF-8 standaloneno?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?!--Licensed to the Apache Software Foundation (ASF) under one or morecontributor license agreements. See the NOTICE file distributed withthis work for additional information regarding copyright ownership.The ASF licenses this file to You under the Apache License, Version 2.0(the License); you may not use this file except in compliance withthe License. You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.
--configuration
propertynamejavax.jdo.option.ConnectionURL/namevaluejdbc:mysql://192.168.0.78:3306/metastore_db?createDatabaseIfNotExisttrue/valuedescriptionJDBC connect string for a JDBC metastore/description/propertypropertynamejavax.jdo.option.ConnectionDriverName/namevaluecom.mysql.cj.jdbc.Driver/value/property propertynamejavax.jdo.option.ConnectionUserName/namevalueroot/value/propertypropertynamejavax.jdo.option.ConnectionPassword/namevalueroot/value/property!-- Metastor--rashproperty namehive.metastore.uris/namevaluethrift://localhost:9083/value/propertypropertynamehive.metastore.warehouse.dir/namevalue/user/hive/warehouse/value/propertypropertynamehive.server2.thrift.bind.host/namevalue0.0.0.0/value/propertypropertynamehive.server2.thrift.port/namevalue10000/value/property/configuration
//将此配置使用docker cp命令拷贝至hive容器内的/opt/hive/conf目录下。
2、配置core-site.xml文件
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configuration
propertynamefs.defaultFS/namevaluehdfs://hadoop-namenode:8020/value/property
/configuration//hadoop-namenode容器IP可以在宿主机执行weave ps 命令获取配置文件修改完毕后通过docker cp命令将文件拷贝至hive容器内的/opt/hadoop-2.7.4/etc/hadoop目录与/opt/hive/conf目录。
启动metastore
#在hive容器内部执行
hive --service metastore 启动hiveserver2
#在hive容器内部执行执行此命令需要先关闭hadoo-namenode的安全模式
hive --service hiveserver2 --hiveconf hive.root.loggerDEBUG,console执行HQL
CREATE TABLE test_hive (id INT, name STRING);
INSERT INTO test_hive VALUES (1, hive-test);
SELECT * FROM test_hive;预期结果输出 1, hive-test。
(2) 验证MySQL元数据
docker exec -it mysql mysql -uroot -proot
use metastore_db;
SELECT TBL_NAME FROM TBLS;预期结果显示 test_hive 表名。
如图所示
验证Kafka
(1) 生产与消费消息
docker exec -it kafka-1 bash
# 创建主题
kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092
# 生产消息
echo hello kafka | kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092
# 消费消息需另开终端
kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092预期结果消费者终端输出 hello kafka。
如图所示
** 验证本地数据挂载**
在JupyterLab中
左侧文件浏览器中检查 /home/jovyan/work对应本地 ./notebooks 目录。检查 /data 目录是否包含本地挂载的文件例如 /path/to/local/data 中的内容。
子节点设置
version: 3.8services:# HDFS DataNode 服务hadoop-datanode:image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8privileged: true #使用二进制文件安装的docker需要开启特权模式每个容器都需要开启该模式container_name: hadoop-datanode-2 # 子节点容器名称唯一例如按编号命名hostname: hadoop-datanode-2environment:- CORE_CONF_fs_defaultFShdfs://hadoop-namenode:8020 # 指向主节点NameNode- HDFS_CONF_dfs_replication2- TZAsia/Shanghainetworks:- bigdata-netdns:- 172.17.0.1volumes:- ./hadoop-conf:/etc/hadoop/conf # 挂载主节点的Hadoop配置文件restart: always
# extra_hosts:
# - hadoop-namenode:10.32.0.32logging:driver: json-fileoptions:max-size: 100mmax-file: 5# YARN NodeManager 服务hadoop-nodemanager:image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8privileged: true #使用二进制文件安装的docker需要开启特权模式每个容器都需要开启该模式container_name: hadoop-nodemanager-2 # 子节点容器名称唯一hostname: hadoop-nodemanager-2environment:- YARN_CONF_yarn_resourcemanager_hostnamehadoop-resourcemanager # 指向主节点ResourceManager- CORE_CONF_fs_defaultFShdfs://hadoop-namenode:8020- TZAsia/Shanghainetworks:- bigdata-netdns:- 172.17.0.1volumes:- ./hadoop-conf/yarn-site.xml:/etc/hadoop/yarn-site.xml # 挂载主节点的Hadoop配置文件,用于上报内存与cpu核心数depends_on:- hadoop-datanode # 确保DataNode先启动可选restart: alwayslogging:driver: json-fileoptions:max-size: 100mmax-file: 5# 共享网络配置必须与主节点一致
networks:bigdata-net:external: truename: weave # 使用主节点创建的Weave网络yarn配置文件
configurationpropertynameyarn.resourcemanager.hostname/namevaluehadoop-resourcemanager/value/propertypropertynameyarn.resourcemanager.address/namevaluehadoop-resourcemanager:8032/value/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.resource.memory-mb/namevalue4096/value/propertypropertynameyarn.nodemanager.resource.cpu-vcores/namevalue8/value/property
/configurationPs:内存大小和cpu核心数需要按照实际情况填写。 ####################################################################################### 今日推荐 小说《异种的营养是牛肉的六倍》 简介【异种天灾】【美食】【日常】 变异生物的蛋白质是牛肉的几倍 刚刚来到这个世界刘笔就被自己的想法震惊到了。 在被孢子污染后的土地上觉醒了神厨系统是不是搞错了什么各种变异生物都能做成美食吗 那就快端上来罢 安全区边缘的特色美食饭店有点非常规的温馨美食日常。
