云南住房和城乡建设厅网站首页,郑州发布会最新消息,做木工网站,偷别人的WordPress主题文章目录 概述基础知识Hadoop组件构成Hadoop配置文件 环境准备配置Hadoop配置下载配置环境变量 Hadoop运行模式Standalone Operation#xff08;本地#xff09;官方DemoWordCount单词统计Demo Pseudo-Distributed Operation#xff08;伪分布式模式#xff09;配置修改启动… 文章目录 概述基础知识Hadoop组件构成Hadoop配置文件 环境准备配置Hadoop配置下载配置环境变量 Hadoop运行模式Standalone Operation本地官方DemoWordCount单词统计Demo Pseudo-Distributed Operation伪分布式模式配置修改启动DFS【9870】Hadoop-DFS数据清空格式化启动DFS组件访问DFS前端页面不同版本的Hadoop的NameNode端口有变dfs命令使用主要用来操作文件复制物理机文件中hadoop中文件展示以及读取文件内容创建目录、文件使用mapreduce进行计算hadoop里面某个文件夹的内容 启动Yarn组件【8088】配置修改启动访问yarn前端页面运行计算dfs某个目录所有文件的单词数yarn页面有运行记录 启动MapReduce组件配置修改启动日志采集系统查看任务日志启动日志聚集任务执行的具体详情上传到HDFS组件中未启动前启动 概述 Hadoop 分布式系统基础架构 解决问题 海量数据存储、海量数据的分析计算 官网https://hadoop.apache.org/ HDFSHadoop Distributed File System 分布式文件系统用于存储数据 Hadoop的默认配置【core-site.xml】 https://hadoop.apache.org/docs/r3.3.6/hadoop-project-dist/hadoop-common/core-default.xml 配置Hadoop集群中各个组件间共享属性和通用参数以实现更好的性能和可靠性 hadoop目录\share\hadoop\common\hadoop-common-3.3.6.jar Hadoop的默认配置【hdfs-site.xml】 https://hadoop.apache.org/docs/r3.3.6/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml 配置HDFS组件中各种参数以实现更好的性能和可靠性如数据块大小、心跳间隔等 hadoop目录\share\hadoop\hdfs\hadoop-hdfs-3.3.6.jar Hadoop的默认配置【mapred-site.xml】 https://hadoop.apache.org/docs/r3.3.6/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml 配置MapReduce任务执行过程进行参数调整、优化等操作 hadoop目录\share\hadoop\mapreduce\hadoop-mapreduce-client-core-3.3.6.jar Hadoop的默认配置【yarn-site.xml】 https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/yarn-default.xml 配置YARN资源管理器ResourceManager和节点管理器NodeManager的行为 hadoop目录\share\hadoop\yarn\hadoop-yarn-common-3.3.6.jar 基础知识 #mermaid-svg-68Uz1VThxlaZxDYU {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-68Uz1VThxlaZxDYU .error-icon{fill:#552222;}#mermaid-svg-68Uz1VThxlaZxDYU .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-68Uz1VThxlaZxDYU .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-68Uz1VThxlaZxDYU .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-68Uz1VThxlaZxDYU .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-68Uz1VThxlaZxDYU .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-68Uz1VThxlaZxDYU .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-68Uz1VThxlaZxDYU .marker{fill:#333333;stroke:#333333;}#mermaid-svg-68Uz1VThxlaZxDYU .marker.cross{stroke:#333333;}#mermaid-svg-68Uz1VThxlaZxDYU svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-68Uz1VThxlaZxDYU .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-68Uz1VThxlaZxDYU .cluster-label text{fill:#333;}#mermaid-svg-68Uz1VThxlaZxDYU .cluster-label span{color:#333;}#mermaid-svg-68Uz1VThxlaZxDYU .label text,#mermaid-svg-68Uz1VThxlaZxDYU span{fill:#333;color:#333;}#mermaid-svg-68Uz1VThxlaZxDYU .node rect,#mermaid-svg-68Uz1VThxlaZxDYU .node circle,#mermaid-svg-68Uz1VThxlaZxDYU .node ellipse,#mermaid-svg-68Uz1VThxlaZxDYU .node polygon,#mermaid-svg-68Uz1VThxlaZxDYU .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-68Uz1VThxlaZxDYU .node .label{text-align:center;}#mermaid-svg-68Uz1VThxlaZxDYU .node.clickable{cursor:pointer;}#mermaid-svg-68Uz1VThxlaZxDYU .arrowheadPath{fill:#333333;}#mermaid-svg-68Uz1VThxlaZxDYU .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-68Uz1VThxlaZxDYU .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-68Uz1VThxlaZxDYU .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-68Uz1VThxlaZxDYU .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-68Uz1VThxlaZxDYU .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-68Uz1VThxlaZxDYU .cluster text{fill:#333;}#mermaid-svg-68Uz1VThxlaZxDYU .cluster span{color:#333;}#mermaid-svg-68Uz1VThxlaZxDYU div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-68Uz1VThxlaZxDYU :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Hadoop三大发行版本 Apache最基础版本入门学习友好 Cloudera大型互联网企业用的多 收费 Hotronwords文档好 #mermaid-svg-I5uW2DALcNqJvQFw {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-I5uW2DALcNqJvQFw .error-icon{fill:#552222;}#mermaid-svg-I5uW2DALcNqJvQFw .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-I5uW2DALcNqJvQFw .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-I5uW2DALcNqJvQFw .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-I5uW2DALcNqJvQFw .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-I5uW2DALcNqJvQFw .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-I5uW2DALcNqJvQFw .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-I5uW2DALcNqJvQFw .marker{fill:#333333;stroke:#333333;}#mermaid-svg-I5uW2DALcNqJvQFw .marker.cross{stroke:#333333;}#mermaid-svg-I5uW2DALcNqJvQFw svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-I5uW2DALcNqJvQFw .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-I5uW2DALcNqJvQFw .cluster-label text{fill:#333;}#mermaid-svg-I5uW2DALcNqJvQFw .cluster-label span{color:#333;}#mermaid-svg-I5uW2DALcNqJvQFw .label text,#mermaid-svg-I5uW2DALcNqJvQFw span{fill:#333;color:#333;}#mermaid-svg-I5uW2DALcNqJvQFw .node rect,#mermaid-svg-I5uW2DALcNqJvQFw .node circle,#mermaid-svg-I5uW2DALcNqJvQFw .node ellipse,#mermaid-svg-I5uW2DALcNqJvQFw .node polygon,#mermaid-svg-I5uW2DALcNqJvQFw .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-I5uW2DALcNqJvQFw .node .label{text-align:center;}#mermaid-svg-I5uW2DALcNqJvQFw .node.clickable{cursor:pointer;}#mermaid-svg-I5uW2DALcNqJvQFw .arrowheadPath{fill:#333333;}#mermaid-svg-I5uW2DALcNqJvQFw .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-I5uW2DALcNqJvQFw .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-I5uW2DALcNqJvQFw .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-I5uW2DALcNqJvQFw .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-I5uW2DALcNqJvQFw .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-I5uW2DALcNqJvQFw .cluster text{fill:#333;}#mermaid-svg-I5uW2DALcNqJvQFw .cluster span{color:#333;}#mermaid-svg-I5uW2DALcNqJvQFw div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-I5uW2DALcNqJvQFw :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Hadoop优点 高可靠性Hadoop底层维护多个数据副本以即使adoop某个计算元素或存储出现贴故障也不会导致据的丢失 高扩展性在集间分配任务数据可方便的扩展数刻以干十的节点 高效性在MapReducel的思想下Hadoop是并行工作的以加快任务处理速度 高容错性能够自动将失败的任务重新分配 #mermaid-svg-3iXt1U8OqHJzQkh9 {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .error-icon{fill:#552222;}#mermaid-svg-3iXt1U8OqHJzQkh9 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-3iXt1U8OqHJzQkh9 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .marker.cross{stroke:#333333;}#mermaid-svg-3iXt1U8OqHJzQkh9 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-3iXt1U8OqHJzQkh9 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .cluster-label text{fill:#333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .cluster-label span{color:#333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .label text,#mermaid-svg-3iXt1U8OqHJzQkh9 span{fill:#333;color:#333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .node rect,#mermaid-svg-3iXt1U8OqHJzQkh9 .node circle,#mermaid-svg-3iXt1U8OqHJzQkh9 .node ellipse,#mermaid-svg-3iXt1U8OqHJzQkh9 .node polygon,#mermaid-svg-3iXt1U8OqHJzQkh9 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-3iXt1U8OqHJzQkh9 .node .label{text-align:center;}#mermaid-svg-3iXt1U8OqHJzQkh9 .node.clickable{cursor:pointer;}#mermaid-svg-3iXt1U8OqHJzQkh9 .arrowheadPath{fill:#333333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-3iXt1U8OqHJzQkh9 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-3iXt1U8OqHJzQkh9 .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-3iXt1U8OqHJzQkh9 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-3iXt1U8OqHJzQkh9 .cluster text{fill:#333;}#mermaid-svg-3iXt1U8OqHJzQkh9 .cluster span{color:#333;}#mermaid-svg-3iXt1U8OqHJzQkh9 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-3iXt1U8OqHJzQkh9 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Hadoop 1.x组成 HDFS数据存储 Common辅助工具 MapReduce计算以及资源调度 Hadoop组件构成 #mermaid-svg-GHZCcW8G0Vl4fvuA {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .error-icon{fill:#552222;}#mermaid-svg-GHZCcW8G0Vl4fvuA .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-GHZCcW8G0Vl4fvuA .marker{fill:#333333;stroke:#333333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .marker.cross{stroke:#333333;}#mermaid-svg-GHZCcW8G0Vl4fvuA svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-GHZCcW8G0Vl4fvuA .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .cluster-label text{fill:#333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .cluster-label span{color:#333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .label text,#mermaid-svg-GHZCcW8G0Vl4fvuA span{fill:#333;color:#333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .node rect,#mermaid-svg-GHZCcW8G0Vl4fvuA .node circle,#mermaid-svg-GHZCcW8G0Vl4fvuA .node ellipse,#mermaid-svg-GHZCcW8G0Vl4fvuA .node polygon,#mermaid-svg-GHZCcW8G0Vl4fvuA .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-GHZCcW8G0Vl4fvuA .node .label{text-align:center;}#mermaid-svg-GHZCcW8G0Vl4fvuA .node.clickable{cursor:pointer;}#mermaid-svg-GHZCcW8G0Vl4fvuA .arrowheadPath{fill:#333333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-GHZCcW8G0Vl4fvuA .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-GHZCcW8G0Vl4fvuA .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-GHZCcW8G0Vl4fvuA .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-GHZCcW8G0Vl4fvuA .cluster text{fill:#333;}#mermaid-svg-GHZCcW8G0Vl4fvuA .cluster span{color:#333;}#mermaid-svg-GHZCcW8G0Vl4fvuA div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-GHZCcW8G0Vl4fvuA :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Hadoop 2.x组成 HDFS数据存储 涉及进程【NameNode、DataNode、SecondaryNameNode、Journal Node、NFSGateway】 Common辅助工具 MapReduce计算 涉及进程【JobTracker、TaskTracker、MapTask、ReduceTask】 Yarn资源调度 涉及进程【ResourceManager、NodeManager、ApplicationMaster、TimelineServer】 #mermaid-svg-SUDHZItnXr19F782 {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-SUDHZItnXr19F782 .error-icon{fill:#552222;}#mermaid-svg-SUDHZItnXr19F782 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-SUDHZItnXr19F782 .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-SUDHZItnXr19F782 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-SUDHZItnXr19F782 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-SUDHZItnXr19F782 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-SUDHZItnXr19F782 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-SUDHZItnXr19F782 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-SUDHZItnXr19F782 .marker.cross{stroke:#333333;}#mermaid-svg-SUDHZItnXr19F782 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-SUDHZItnXr19F782 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-SUDHZItnXr19F782 .cluster-label text{fill:#333;}#mermaid-svg-SUDHZItnXr19F782 .cluster-label span{color:#333;}#mermaid-svg-SUDHZItnXr19F782 .label text,#mermaid-svg-SUDHZItnXr19F782 span{fill:#333;color:#333;}#mermaid-svg-SUDHZItnXr19F782 .node rect,#mermaid-svg-SUDHZItnXr19F782 .node circle,#mermaid-svg-SUDHZItnXr19F782 .node ellipse,#mermaid-svg-SUDHZItnXr19F782 .node polygon,#mermaid-svg-SUDHZItnXr19F782 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-SUDHZItnXr19F782 .node .label{text-align:center;}#mermaid-svg-SUDHZItnXr19F782 .node.clickable{cursor:pointer;}#mermaid-svg-SUDHZItnXr19F782 .arrowheadPath{fill:#333333;}#mermaid-svg-SUDHZItnXr19F782 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-SUDHZItnXr19F782 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-SUDHZItnXr19F782 .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-SUDHZItnXr19F782 .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-SUDHZItnXr19F782 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-SUDHZItnXr19F782 .cluster text{fill:#333;}#mermaid-svg-SUDHZItnXr19F782 .cluster span{color:#333;}#mermaid-svg-SUDHZItnXr19F782 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-SUDHZItnXr19F782 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} HDFS分布式文件系统组成 NameNode存储文件元数据属性信息比如文件名、文件权限等以及每个文件的快列表以及其所在的DataNode DataNode文件块数据、以及快数据的校验和 Secondary NameNode监控HDFS状态的程序每隔一段时间获取HDFS元数据快照 JournalNode协调多个NameNodes之间共享编辑日志 NFSGateway提供给用户访问 HDFS 文件系统的一种途径, 它将 HDFS 文件系统映射为标准的网络文件系统 (NFS) 挂载点 #mermaid-svg-zaglz3alXL7blLLk {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-zaglz3alXL7blLLk .error-icon{fill:#552222;}#mermaid-svg-zaglz3alXL7blLLk .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-zaglz3alXL7blLLk .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-zaglz3alXL7blLLk .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-zaglz3alXL7blLLk .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-zaglz3alXL7blLLk .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-zaglz3alXL7blLLk .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-zaglz3alXL7blLLk .marker{fill:#333333;stroke:#333333;}#mermaid-svg-zaglz3alXL7blLLk .marker.cross{stroke:#333333;}#mermaid-svg-zaglz3alXL7blLLk svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-zaglz3alXL7blLLk .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-zaglz3alXL7blLLk .cluster-label text{fill:#333;}#mermaid-svg-zaglz3alXL7blLLk .cluster-label span{color:#333;}#mermaid-svg-zaglz3alXL7blLLk .label text,#mermaid-svg-zaglz3alXL7blLLk span{fill:#333;color:#333;}#mermaid-svg-zaglz3alXL7blLLk .node rect,#mermaid-svg-zaglz3alXL7blLLk .node circle,#mermaid-svg-zaglz3alXL7blLLk .node ellipse,#mermaid-svg-zaglz3alXL7blLLk .node polygon,#mermaid-svg-zaglz3alXL7blLLk .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-zaglz3alXL7blLLk .node .label{text-align:center;}#mermaid-svg-zaglz3alXL7blLLk .node.clickable{cursor:pointer;}#mermaid-svg-zaglz3alXL7blLLk .arrowheadPath{fill:#333333;}#mermaid-svg-zaglz3alXL7blLLk .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-zaglz3alXL7blLLk .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-zaglz3alXL7blLLk .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-zaglz3alXL7blLLk .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-zaglz3alXL7blLLk .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-zaglz3alXL7blLLk .cluster text{fill:#333;}#mermaid-svg-zaglz3alXL7blLLk .cluster span{color:#333;}#mermaid-svg-zaglz3alXL7blLLk div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-zaglz3alXL7blLLk :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} YARN资源调度组成 ResourceManager主节点负责协调集群资源并分配给各个应用程序 NodeManager从节点负责管理单个节点上的资源并执行任务 ApplicationMaster新应用程序提交到集群时会创建一个ApplicationMaster实例 TimelineServer用户方便地查询 Hadoop 集群中运行过的历史作业和应用程序信息 #mermaid-svg-PXGZFX6oLsoRKPRv {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-PXGZFX6oLsoRKPRv .error-icon{fill:#552222;}#mermaid-svg-PXGZFX6oLsoRKPRv .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-PXGZFX6oLsoRKPRv .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-PXGZFX6oLsoRKPRv .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-PXGZFX6oLsoRKPRv .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-PXGZFX6oLsoRKPRv .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-PXGZFX6oLsoRKPRv .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-PXGZFX6oLsoRKPRv .marker{fill:#333333;stroke:#333333;}#mermaid-svg-PXGZFX6oLsoRKPRv .marker.cross{stroke:#333333;}#mermaid-svg-PXGZFX6oLsoRKPRv svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-PXGZFX6oLsoRKPRv .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-PXGZFX6oLsoRKPRv .cluster-label text{fill:#333;}#mermaid-svg-PXGZFX6oLsoRKPRv .cluster-label span{color:#333;}#mermaid-svg-PXGZFX6oLsoRKPRv .label text,#mermaid-svg-PXGZFX6oLsoRKPRv span{fill:#333;color:#333;}#mermaid-svg-PXGZFX6oLsoRKPRv .node rect,#mermaid-svg-PXGZFX6oLsoRKPRv .node circle,#mermaid-svg-PXGZFX6oLsoRKPRv .node ellipse,#mermaid-svg-PXGZFX6oLsoRKPRv .node polygon,#mermaid-svg-PXGZFX6oLsoRKPRv .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-PXGZFX6oLsoRKPRv .node .label{text-align:center;}#mermaid-svg-PXGZFX6oLsoRKPRv .node.clickable{cursor:pointer;}#mermaid-svg-PXGZFX6oLsoRKPRv .arrowheadPath{fill:#333333;}#mermaid-svg-PXGZFX6oLsoRKPRv .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-PXGZFX6oLsoRKPRv .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-PXGZFX6oLsoRKPRv .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-PXGZFX6oLsoRKPRv .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-PXGZFX6oLsoRKPRv .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-PXGZFX6oLsoRKPRv .cluster text{fill:#333;}#mermaid-svg-PXGZFX6oLsoRKPRv .cluster span{color:#333;}#mermaid-svg-PXGZFX6oLsoRKPRv div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-PXGZFX6oLsoRKPRv :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} MapReduce计算组成 JobTracker主节点负责协调集群资源并分配给各个任务 TaskTracker从节点负责管理单个节点上的资源并执行任务 JobHistoryServer管理和存储MapReduce作业历史记录信息 MapTask和ReduceTask执行任务的Java程序。新作业被提交到集群时会创建多个Mapper和Reducer实例处理数据 #mermaid-svg-kCuqdQn5ul3GO2LW {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-kCuqdQn5ul3GO2LW .error-icon{fill:#552222;}#mermaid-svg-kCuqdQn5ul3GO2LW .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-kCuqdQn5ul3GO2LW .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-kCuqdQn5ul3GO2LW .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-kCuqdQn5ul3GO2LW .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-kCuqdQn5ul3GO2LW .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-kCuqdQn5ul3GO2LW .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-kCuqdQn5ul3GO2LW .marker{fill:#333333;stroke:#333333;}#mermaid-svg-kCuqdQn5ul3GO2LW .marker.cross{stroke:#333333;}#mermaid-svg-kCuqdQn5ul3GO2LW svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-kCuqdQn5ul3GO2LW .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-kCuqdQn5ul3GO2LW .cluster-label text{fill:#333;}#mermaid-svg-kCuqdQn5ul3GO2LW .cluster-label span{color:#333;}#mermaid-svg-kCuqdQn5ul3GO2LW .label text,#mermaid-svg-kCuqdQn5ul3GO2LW span{fill:#333;color:#333;}#mermaid-svg-kCuqdQn5ul3GO2LW .node rect,#mermaid-svg-kCuqdQn5ul3GO2LW .node circle,#mermaid-svg-kCuqdQn5ul3GO2LW .node ellipse,#mermaid-svg-kCuqdQn5ul3GO2LW .node polygon,#mermaid-svg-kCuqdQn5ul3GO2LW .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-kCuqdQn5ul3GO2LW .node .label{text-align:center;}#mermaid-svg-kCuqdQn5ul3GO2LW .node.clickable{cursor:pointer;}#mermaid-svg-kCuqdQn5ul3GO2LW .arrowheadPath{fill:#333333;}#mermaid-svg-kCuqdQn5ul3GO2LW .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-kCuqdQn5ul3GO2LW .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-kCuqdQn5ul3GO2LW .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-kCuqdQn5ul3GO2LW .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-kCuqdQn5ul3GO2LW .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-kCuqdQn5ul3GO2LW .cluster text{fill:#333;}#mermaid-svg-kCuqdQn5ul3GO2LW .cluster span{color:#333;}#mermaid-svg-kCuqdQn5ul3GO2LW div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-kCuqdQn5ul3GO2LW :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} MapReduce计算过程 Map并行处理输入数据 Reduce对Map结果进行汇总 Hadoop配置文件 配置文件路径 hadoop目录/etc/hadoop #mermaid-svg-UitrCcpuSsafPP6q {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-UitrCcpuSsafPP6q .error-icon{fill:#552222;}#mermaid-svg-UitrCcpuSsafPP6q .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-UitrCcpuSsafPP6q .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-UitrCcpuSsafPP6q .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-UitrCcpuSsafPP6q .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-UitrCcpuSsafPP6q .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-UitrCcpuSsafPP6q .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-UitrCcpuSsafPP6q .marker{fill:#333333;stroke:#333333;}#mermaid-svg-UitrCcpuSsafPP6q .marker.cross{stroke:#333333;}#mermaid-svg-UitrCcpuSsafPP6q svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-UitrCcpuSsafPP6q .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-UitrCcpuSsafPP6q .cluster-label text{fill:#333;}#mermaid-svg-UitrCcpuSsafPP6q .cluster-label span{color:#333;}#mermaid-svg-UitrCcpuSsafPP6q .label text,#mermaid-svg-UitrCcpuSsafPP6q span{fill:#333;color:#333;}#mermaid-svg-UitrCcpuSsafPP6q .node rect,#mermaid-svg-UitrCcpuSsafPP6q .node circle,#mermaid-svg-UitrCcpuSsafPP6q .node ellipse,#mermaid-svg-UitrCcpuSsafPP6q .node polygon,#mermaid-svg-UitrCcpuSsafPP6q .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-UitrCcpuSsafPP6q .node .label{text-align:center;}#mermaid-svg-UitrCcpuSsafPP6q .node.clickable{cursor:pointer;}#mermaid-svg-UitrCcpuSsafPP6q .arrowheadPath{fill:#333333;}#mermaid-svg-UitrCcpuSsafPP6q .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-UitrCcpuSsafPP6q .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-UitrCcpuSsafPP6q .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-UitrCcpuSsafPP6q .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-UitrCcpuSsafPP6q .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-UitrCcpuSsafPP6q .cluster text{fill:#333;}#mermaid-svg-UitrCcpuSsafPP6q .cluster span{color:#333;}#mermaid-svg-UitrCcpuSsafPP6q div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-UitrCcpuSsafPP6q :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Hadoop配置文件 core-site.xml所有组件共享的通用属性和参数、修改文件系统、集群名称、日志路径等 hdfs-site.xmlHDFS特定的属性和参数、数据快大小、副本数、NameNode、DataNode的存储路径 mapred-site.xml配置MapReduce任务执行过程进行参数调整、优化等操作 yarn-site.xml配置YARN资源管理器ResourceManager和节点管理器NodeManager的行为 环境准备
配置
//修改主机名
//more /etc/sysconfig/network 内容如下 //不同机器取不同的HOSTNAME不要取成一样的
NETWORKINGyes
HOSTNAMEhadoop107 ////固定IP地址 自行百度
ifconfig
more /etc/sysconfig/network-scripts/ifcfg-ens33 //
// 查看自定义主机名、ip的映射关系 more /etc/hosts
ping 主机名Hadoop配置
下载 官网 https://hadoop.apache.org/releases.html 配置环境变量
//将压缩包解压到指定目录
mkdir -p /opt/module/ tar -zxvf hadoop-3.3.6.tar.gz -C /opt/module///进入解压后的软件目录
cd /opt/module/hadoop-3.3.6//设置环境变量
vim /etc/profile//此文件末尾添加下面四行内容
## Hadoop
export HADOOP_HOME/opt/module/hadoop-3.3.6
export PATH$PATH:$HADOOP_HOME/bin
export PATH$PATH:$HADOOP_HOME/sbin//使环境变量生效
source /etc/profileHadoop运行模式 #mermaid-svg-WhJEewenAp9zAjMS {font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}#mermaid-svg-WhJEewenAp9zAjMS .error-icon{fill:#552222;}#mermaid-svg-WhJEewenAp9zAjMS .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-WhJEewenAp9zAjMS .edge-thickness-normal{stroke-width:2px;}#mermaid-svg-WhJEewenAp9zAjMS .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-WhJEewenAp9zAjMS .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-WhJEewenAp9zAjMS .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-WhJEewenAp9zAjMS .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-WhJEewenAp9zAjMS .marker{fill:#333333;stroke:#333333;}#mermaid-svg-WhJEewenAp9zAjMS .marker.cross{stroke:#333333;}#mermaid-svg-WhJEewenAp9zAjMS svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-WhJEewenAp9zAjMS .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-WhJEewenAp9zAjMS .cluster-label text{fill:#333;}#mermaid-svg-WhJEewenAp9zAjMS .cluster-label span{color:#333;}#mermaid-svg-WhJEewenAp9zAjMS .label text,#mermaid-svg-WhJEewenAp9zAjMS span{fill:#333;color:#333;}#mermaid-svg-WhJEewenAp9zAjMS .node rect,#mermaid-svg-WhJEewenAp9zAjMS .node circle,#mermaid-svg-WhJEewenAp9zAjMS .node ellipse,#mermaid-svg-WhJEewenAp9zAjMS .node polygon,#mermaid-svg-WhJEewenAp9zAjMS .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-WhJEewenAp9zAjMS .node .label{text-align:center;}#mermaid-svg-WhJEewenAp9zAjMS .node.clickable{cursor:pointer;}#mermaid-svg-WhJEewenAp9zAjMS .arrowheadPath{fill:#333333;}#mermaid-svg-WhJEewenAp9zAjMS .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-WhJEewenAp9zAjMS .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-WhJEewenAp9zAjMS .edgeLabel{background-color:#e8e8e8;text-align:center;}#mermaid-svg-WhJEewenAp9zAjMS .edgeLabel rect{opacity:0.5;background-color:#e8e8e8;fill:#e8e8e8;}#mermaid-svg-WhJEewenAp9zAjMS .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-WhJEewenAp9zAjMS .cluster text{fill:#333;}#mermaid-svg-WhJEewenAp9zAjMS .cluster span{color:#333;}#mermaid-svg-WhJEewenAp9zAjMS div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-WhJEewenAp9zAjMS :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 默认模式 生产真正使用 分布模式 本地模式Local Standalone Operation 伪分布模式Pseudo-Distributed Pseudo-Distributed Operation 完全分布模式Fully-Distributed Fully-Distributed Operation Standalone Operation本地 参考 https://hadoop.apache.org/docs/r3.3.6/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation 官方Demo 官方Demo统计文件中某个正则规则的单词出现次数 # hadoop目录
cd /opt/module/hadoop-3.3.6# 创建数据源文件 用于下面进行demo统计单词
mkdir input# 复制一些普通的文件
cp etc/hadoop/*.xml input# 统计input里面的源文件规则是dfs[a-z.]的单词个数并将结果输出到当前目录下的output目录下 输出目录不得提前创建运行时提示会报错
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input output dfs[a-z.]# 查看统计结果
cat output/*
cat output/part-r-00000# 显示出来的结果跟grep查出来的一样WordCount单词统计Demo
//创建数据目录
mkdir -p /opt/module/hadoop-3.3.6/input/wordCountData cd /opt/module/hadoop-3.3.6/input///文件数据创建 用于demo测试
echo cat apple banana wordCountData/data1.txt
echo dog wordCountData/data1.txt
echo elephant wordCountData/data1.txtecho cat apple banana wordCountData/data2.txt
echo dog wordCountData/data2.txt
echo elephant queen wordCountData/data2.txt//查看数据内容
more wordCountData/data1.txt
more wordCountData/data2.txt//开始统计wordCountData文件目录下的单词数
hadoop jar /opt/module/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /opt/module/hadoop-3.3.6/input/wordCountData wordCountDataoutput//查看统计结果
cd /opt/module/hadoop-3.3.6/input/wordCountDataoutput
cat ./*Pseudo-Distributed Operation伪分布式模式 参考 https://hadoop.apache.org/docs/r3.3.6/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation 概述 单节点的分布式系统用于测试使用 配置修改 核心配置文件修改 vim /opt/module/hadoop-3.3.6/etc/hadoop/core-site.xml
configuration!-- 默认是本地文件协议 file: --propertynamefs.defaultFS/namevaluehdfs://192.168.19.107:9000/value/property!-- 临时目录 默认/tmp/hadoop-${user.name} --propertynamehadoop.tmp.dir/namevalue/opt/module/hadoop-3.3.6/tmp/value/property/configuration核心配置文件修改 vim /opt/module/hadoop-3.3.6/etc/hadoop/hdfs-site.xml
configuration!-- 集群设置为1 默认3 --propertynamedfs.replication/namevalue1/value/property/configuration启动DFS【9870】
Hadoop-DFS数据清空格式化
hdfs namenode -format启动DFS组件 注意 启动过程中可能遇到非root用户、JAVA_HOME找不到的现象导致启动失败自行参考下面的问题解决 # 未启动hadoop时所系统所运行java程序
jps# 启动hadoop相关的应用程序
sh /opt/module/hadoop-3.3.6/sbin/start-dfs.sh# 查看启动hadoop的应用变化
jps访问DFS前端页面不同版本的Hadoop的NameNode端口有变 浏览器NameNode前端页面 http://192.168.19.107:9870/ dfs命令使用主要用来操作文件 帮助文档 hdfs dfs --help 复制物理机文件中hadoop中
hdfs dfs -mkdir /testhdfs dfs -put /opt/module/hadoop-3.3.6/input /test文件展示以及读取文件内容
hdfs dfs -ls -R /hdfs dfs -cat /test/input/core-site.xml创建目录、文件
hdfs dfs -mkdir -p /test/linrchdfs dfs -touch /test/linrc/1.txt使用mapreduce进行计算hadoop里面某个文件夹的内容
hdfs dfs -ls /test/input# 对hadoop里面某个文件夹内容进行单词统计
hadoop jar /opt/module/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /test/input/wordCountData /test/input/wordCountDataoutput2hdfs dfs -ls /test/input# 查看统计结果
hdfs dfs -cat /test/input/wordCountDataoutput2/*启动Yarn组件【8088】
配置修改
强制指定Yarn的环境变量 /opt/module/hadoop-3.3.6/etc/hadoop/yarn-env.sh
export JAVA_HOME/www/server/jdk8/jdk1.8.0_202yarn-site.xml添加如下两个配置 /opt/module/hadoop-3.3.6/etc/hadoop/yarn-site.sh
configuration!-- Site specific YARN configuration properties https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/yarn-default.xml --propertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.resourcemanager.hostname/namevalue192.168.19.107/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HO
ME/value/property!-- 查看任务日志时防止其重定向localhost导致页面打开失败 --propertynameyarn.timeline-service.hostname/namevalue192.168.19.107/value/property/configuration启动
//非常重要必须回到hadoop的目录里面进行启动我也不知道为什么
cd /opt/module/hadoop-3.3.6//不要使用 sh命令启动否则报错我也不知道为什么
./sbin/start-yarn.sh 访问yarn前端页面 浏览器 http://ip:8088 yarn页面端口配置 https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/yarn-default.xml的【yarn.resourcemanager.webapp.address】 运行计算dfs某个目录所有文件的单词数yarn页面有运行记录 //单词计算开始
hadoop jar /opt/module/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /test/input/wordCountData /test/input/wordCountDataoutput3启动MapReduce组件
配置修改
强制指定mapred的环境变量 /opt/module/hadoop-3.3.6/etc/hadoop/mapred-env.sh
export JAVA_HOME/www/server/jdk8/jdk1.8.0_202mapred-site.xml添加如下配置 /opt/module/hadoop-3.3.6/etc/hadoop/mapred-site.xml
configuration!-- The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn --propertynamemapreduce.framework.name/namevalueyarn/value/propertypropertynamemapreduce.application.classpath/namevalue$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*/value/property!-- mr运行日志采集系统配置 --propertynamemapreduce.jobhistory.address/namevalue192.168.19.107:10020/value/propertypropertynamemapreduce.jobhistory.webapp.address/namevalue192.168.19.107:19888/value/property/configuration启动日志采集系统 mapred --daemon start historyserver查看任务日志 启动日志聚集任务执行的具体详情上传到HDFS组件中
未启动前 启动 注意 如果yarn组件已经启动修改yarn的配置需要重新启动使得配置生效 #停止日志系统
mapred --daemon stop historyserver#停止yarn组件
cd /opt/module/hadoop-3.3.6
./sbin/stop-yarn.sh yarn-site.xml添加如下配置 /opt/module/hadoop-3.3.6/etc/hadoop/yarn-site.sh
configuration!-- Site specific YARN configuration properties https://hadoop.apache.org/docs/r3.3.6/hadoop-yarn/hadoop-yarn-common/yarn-default.xml --propertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.resourcemanager.hostname/namevalue192.168.19.107/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HO
ME/value/propertypropertynameyarn.timeline-service.hostname/namevalue192.168.19.107/value/property!-- 日志聚集启动 --propertynameyarn.log-aggregation-enable/namevaluetrue/value/property!-- 日志聚集的日志保留的时间单位秒 --propertynameyarn.log-aggregation.retain-seconds/namevalue2592000/value/property/configuration#启动yarn组件
cd /opt/module/hadoop-3.3.6
./sbin/start-yarn.sh#启动日志系统
mapred --daemon start historyserver # 重新运行一个任务
hadoop jar /opt/module/hadoop-3.3.6/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /test/input/wordCountData /test/input/wordCountDataoutput5
