怎么做网站表格,合肥建筑材料市场信息价官网,电商详情页图片,德州网站建设价格文章目录 #xff08;132#xff09;YARN常用命令查看任务查看日志查看容器查看节点状态rmadmin更新配置查看队列 #xff08;133#xff09;生产环境核心配置参数#xff08;135#xff09;生产环境核心参数配置案例#xff08;140/141#xff09;Tool接口案例参考文献… 文章目录 132YARN常用命令查看任务查看日志查看容器查看节点状态rmadmin更新配置查看队列 133生产环境核心配置参数135生产环境核心参数配置案例140/141Tool接口案例参考文献 本章我是仅做了解所以很多地方并没有深入去探究用处估计不大可酌情参考。
132YARN常用命令
查看任务
列出所有Applicationyarn application -list
根据Application状态过滤出指定Application如过滤出已完成的Applicationyarn application -list -appStates FINISHED
Application的状态有ALL、NEW、NEW_SAVING、SUBMITTED、ACCEPTED、RUNNING、FINISHED、FAILED、KILLED。
杀掉某个Applicationyarn application -kill application-id
其中application_id是一串形如application_1612577921195_0001的字符串。
列出所有Application尝试的列表yarn applicationattempt -list ApplicationId
打印ApplicationAttempt的状态yarn applicationattempt -status applicationAttemptId
查看日志
非常重要。
查询某个Application的日志yarn logs -applicationId application-id
查询container日志yarn logs -applicationId ApplicationId -containerId ContainerId
查看容器
列出所有容器yarn container -list ApplicationAttemptId
打印容器状态yarn container -status ContainerId
只有在任务运行的时候才能看到container的状态
查看节点状态
列出所有节点yarn node -list -all
就是打印出集群下所有服务器节点的运行状态和地址信息啥的。
rmadmin更新配置
加载队列配置yarn rmadmin -refreshQueues
可以实现对队列配置信息的动态的修改无需停机。
查看队列
打印队列信息yarn queue -status QueueName
比如说yarn queue -status default就是打印默认的队列
会打印出队列的状态、当前容量等等。
133生产环境核心配置参数
同样仅做了解所以直接截教程的图了 RM默认并发是50线程
这里有个虚拟核数的概念需要简单介绍一下。
首先需要知道集群里每个NM都有自己的一套配置参数并不严格要求每个NodeManager的配置参数都必须是一样的。
这样做主要是考虑到节点间性能差异较大的情况。比如说节点1的单核CPU性能是节点2单核CPU性能的两倍那么将二者一视同仁来分配任务的话就有问题了。这时候就可以开启节点1的虚拟核功能把一个物理核视为两个虚拟核这时候节点1和节点2的单核虚拟核CPU性能就接近了也方便RM来分配任务。
即不同NM的话一个物理核数作为几个虚拟核数来使用是不一样的。这样做是为了防止因节点CPU性能不同不好统一管理各个CPU。
所以如果有CPU混搭的情况如有节点是i5有节点是i7这种是有需要开启虚拟核的。
“物理内存检查机制”是为了防止节点内存超出导致崩溃默认打开
135生产环境核心参数配置案例
需求从1G数据中统计每个单词出现次数。服务器3台每台配置4G内存4核CPU4线程。
块大小使用默认的128M1G/128M8所以整个任务需要启用8个MapTask1个ReduceTask以及1个MrAppMaster。
平均每个节点运行811/3台 约等于 3个任务假设采用433分布。
基于以上需求和硬件条件可以做出如下思考
1G数据量不大可以使用容量调度器
RM处理调度器的线程数量默认50太大了没必要可以削成8
不同节点CPU性能一致不需要开启虚拟核
其他配置暂且不表。
直接把教程里的yarn-site.xml配置参数贴出来吧方便之后查看。
!-- 选择调度器默认容量 --
propertydescriptionThe class to use as the resource scheduler./descriptionnameyarn.resourcemanager.scheduler.class/namevalueorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler/value
/property!-- ResourceManager处理调度器请求的线程数量,默认50如果提交的任务数大于50可以增加该值但是不能超过3台 * 4线程 12线程去除其他应用程序实际不能超过8 --
propertydescriptionNumber of threads to handle scheduler interface./descriptionnameyarn.resourcemanager.scheduler.client.thread-count/namevalue8/value
/property!-- 是否让yarn自动检测硬件进行配置默认是false如果该节点有很多其他应用程序建议手动配置。如果该节点没有其他应用程序可以采用自动 --
propertydescriptionEnable auto-detection of node capabilities such asmemory and CPU./descriptionnameyarn.nodemanager.resource.detect-hardware-capabilities/namevaluefalse/value
/property!-- 是否将虚拟核数当作CPU核数默认是false采用物理CPU核数 --
propertydescriptionFlag to determine if logical processors(such ashyperthreads) should be counted as cores. Only applicable on Linuxwhen yarn.nodemanager.resource.cpu-vcores is set to -1 andyarn.nodemanager.resource.detect-hardware-capabilities is true./descriptionnameyarn.nodemanager.resource.count-logical-processors-as-cores/namevaluefalse/value
/property!-- 虚拟核数和物理核数乘数默认是1.0 --
propertydescriptionMultiplier to determine how to convert phyiscal cores tovcores. This value is used if yarn.nodemanager.resource.cpu-vcoresis set to -1(which implies auto-calculate vcores) andyarn.nodemanager.resource.detect-hardware-capabilities is set to true. The number of vcores will be calculated as number of CPUs * multiplier./descriptionnameyarn.nodemanager.resource.pcores-vcores-multiplier/namevalue1.0/value
/property!-- NodeManager使用内存数默认8G修改为4G内存 --
propertydescriptionAmount of physical memory, in MB, that can be allocated for containers. If set to -1 andyarn.nodemanager.resource.detect-hardware-capabilities is true, it isautomatically calculated(in case of Windows and Linux).In other cases, the default is 8192MB./descriptionnameyarn.nodemanager.resource.memory-mb/namevalue4096/value
/property!-- nodemanager的CPU核数不按照硬件环境自动设定时默认是8个修改为4个 --
propertydescriptionNumber of vcores that can be allocatedfor containers. This is used by the RM scheduler when allocatingresources for containers. This is not used to limit the number ofCPUs used by YARN containers. If it is set to -1 andyarn.nodemanager.resource.detect-hardware-capabilities is true, it isautomatically determined from the hardware in case of Windows and Linux.In other cases, number of vcores is 8 by default./descriptionnameyarn.nodemanager.resource.cpu-vcores/namevalue4/value
/property!-- 容器最小内存默认1G --
propertydescriptionThe minimum allocation for every container request at the RM in MBs. Memory requests lower than this will be set to the value of this property. Additionally, a node manager that is configured to have less memory than this value will be shut down by the resource manager./descriptionnameyarn.scheduler.minimum-allocation-mb/namevalue1024/value
/property!-- 容器最大内存默认8G修改为2G --
propertydescriptionThe maximum allocation for every container request at the RM in MBs. Memory requests higher than this will throw an InvalidResourceRequestException./descriptionnameyarn.scheduler.maximum-allocation-mb/namevalue2048/value
/property!-- 容器最小CPU核数默认1个 --
propertydescriptionThe minimum allocation for every container request at the RM in terms of virtual CPU cores. Requests lower than this will be set to the value of this property. Additionally, a node manager that is configured to have fewer virtual cores than this value will be shut down by the resource manager./descriptionnameyarn.scheduler.minimum-allocation-vcores/namevalue1/value
/property!-- 容器最大CPU核数默认4个修改为2个 --
propertydescriptionThe maximum allocation for every container request at the RM in terms of virtual CPU cores. Requests higher than this will throw anInvalidResourceRequestException./descriptionnameyarn.scheduler.maximum-allocation-vcores/namevalue2/value
/property!-- 虚拟内存检查默认打开修改为关闭 --
propertydescriptionWhether virtual memory limits will be enforced forcontainers./descriptionnameyarn.nodemanager.vmem-check-enabled/namevaluefalse/value
/property!-- 虚拟内存和物理内存设置比例,默认2.1 --
propertydescriptionRatio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio./descriptionnameyarn.nodemanager.vmem-pmem-ratio/namevalue2.1/value
/property140/141Tool接口案例
生产环境下比较有用的一个功能。仅做了解吧本节我其实并没有深入只做了简单的复制。
通过tools接口可以实现我们自己程序的参数的动态修改
接下来以自定义实现WordCount为例。
在编写代码的时候pom.xml里要引入
dependenciesdependencygroupIdorg.apache.hadoop/groupIdartifactIdhadoop-client/artifactIdversion3.1.3/version/dependency/dependencies创建类WordCount并实现Tool接口
package com.atguigu.yarn;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;import java.io.IOException;public class WordCount implements Tool {private Configuration conf;//核心驱动Overridepublic int run(String[] args) throws Exception {Job job Job.getInstance(conf);job.setJarByClass(WordCountDriver.class);job.setMapperClass(WordCountMapper.class);job.setReducerClass(WordCountReducer.class);job.setMapOutputKeyClass(Text.class);job.setMapOutputValueClass(IntWritable.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.setInputPaths(job, new Path(args[0]));FileOutputFormat.setOutputPath(job, new Path(args[1]));return job.waitForCompletion(true) ? 0 : 1;}Overridepublic void setConf(Configuration conf) {this.conf conf;}Overridepublic Configuration getConf() {return conf;}public static class WordCountMapper extends MapperLongWritable, Text, Text, IntWritable {private Text outK new Text();private IntWritable outV new IntWritable(1);Overrideprotected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {String line value.toString();String[] words line.split( );for (String word : words) {outK.set(word);context.write(outK, outV);}}}public static class WordCountReducer extends ReducerText, IntWritable, Text, IntWritable {private IntWritable outV new IntWritable();Overrideprotected void reduce(Text key, IterableIntWritable values, Context context) throws IOException, InterruptedException {int sum 0;for (IntWritable value : values) {sum value.get();}outV.set(sum);context.write(key, outV);}}
}新建WordCountDriver
package com.atguigu.yarn;import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import java.util.Arrays;public class WordCountDriver {private static Tool tool;public static void main(String[] args) throws Exception {// 1. 创建配置文件Configuration conf new Configuration();// 2. 判断是否有tool接口switch (args[0]){case wordcount:tool new WordCount();break;default:throw new RuntimeException( No such tool: args[0] );}// 3. 用Tool执行程序// Arrays.copyOfRange 将老数组的元素放到新数组里面// 相当于是拷贝从索引为1的参数到最后的参数int run ToolRunner.run(conf, tool, Arrays.copyOfRange(args, 1, args.length));System.exit(run);}
}然后执行
[atguiguhadoop102 hadoop-3.1.3]$ yarn jar YarnDemo.jar com.atguigu.yarn.WordCountDriver wordcount /input /output注意此时提交的3个参数第一个用于生成特定的Tool第二个和第三个为输入输出目录。此时如果我们希望加入设置参数可以在wordcount后面添加参数例如
[atguiguhadoop102 hadoop-3.1.3]$ yarn jar YarnDemo.jar com.atguigu.yarn.WordCountDriver wordcount -Dmapreduce.job.queuenameroot.test /input /output1参考文献
【尚硅谷大数据Hadoop教程hadoop3.x搭建到集群调优百万播放】
