大数据—— YARN

枕上从妨一夜睡，灯前读尽十年诗。这篇文章主要讲述大数据—— YARN相关的知识，希望能为你提供帮助。
源码见：https://github.com/hiszm/hadoop-train
YARN产生背景

文章图片

YARN架构详解

文章图片

Client

向RM提交任务
杀死任务

ResourceManager

ResourceManager 通常在独立的机器上以后台进程的形式运行，它是整个集群资源的主要协调者和管理者。
负责给用户提交的所有应用程序分配资源，它根据应用程序优先级、队列容量、ACLs、数据位置等信息，做出决策，然后以共享的、安全的、多租户的方式制定分配策略，调度集群资源。

NodeManager

NodeManager 是 YARN 集群中的每个具体节点的管理者。
主要负责该节点内所有容器的生命周期的管理，监视资源和跟踪节点健康。具体如下：
- 启动时向 ResourceManager 注册并定时发送心跳消息，等待 ResourceManager 的指令；
- 维护 Container 的生命周期，监控 Container 的资源使用情况；
- 管理任务运行时的相关依赖，根据 ApplicationMaster 的需要，在启动 Container 之前将需要的程序及其依赖拷贝到本地。

ApplicationMaster

在用户提交一个应用程序时，YARN 会启动一个轻量级的进程 ApplicationMaster。
ApplicationMaster 负责协调来自 ResourceManager 的资源，并通过 NodeManager 监视容器内资源的使用情况，同时还负责任务的监控与容错。具体如下：
- 根据应用的运行状态来决定动态计算资源需求；
- 向 ResourceManager 申请资源，监控申请的资源的使用情况；
- 跟踪任务状态和进度，报告资源的使用情况和应用的进度信息；
- 负责任务的容错。

Container

Container 是 YARN 中的资源抽象，它封装了某个节点上的多维度资源，如内存、CPU、磁盘、网络等。
当 AM 向 RM 申请资源时，RM 为 AM 返回的资源是用 Container 表示的。
YARN 会为每个任务分配一个 Container，该任务只能使用该 Container 中描述的资源。ApplicationMaster 可在 Container 内运行任何类型的任务。例如，MapReduce ApplicationMaster 请求一个容器来启动 map或reduce 任务

YARN执行流程

文章图片

客户端client向yarn集群提交作业 , 首先①向ResourceManager申请分配资源
Resource Manager会为作业分配一个Container(Application manager),Container里面运行这(Application Manager)
Resource Manager会找一个对应的NodeManager通信②，要求NodeManager在这个container上启动应用程序Application Master ③
Application Master向Resource Manager申请资源④(采用轮询的方式通过RPC协议)，Resource scheduler将资源封装发给Application master④,
Application Master将获取到的资源分配给各个Node Manager，并监控运行情况⑤
Node Manage得到任务和资源开始执行作业⑥
再细分作业的话可以分为先执行Map Task,结束后在执行Reduce Task 最后再将结果返回給Application Master等依次往上层递交⑦

YARN环境部署
http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.15.1/hadoop-project-dist/hadoop-common/SingleCluster.html

YARN on Single Node

Configure parameters as follows:
etc/hadoop/mapred-site.xml:

< configuration> < property> < name> mapreduce.framework.name< /name> < value> yarn< /value> < /property> < /configuration>

etc/hadoop/yarn-site.xml:

< configuration> < property> < name> yarn.nodemanager.aux-services< /name> < value> mapreduce_shuffle< /value> < /property> < /configuration>

【大数据—— YARN】Start ResourceManager daemon and NodeManager daemon:
$ sbin/start-yarn.sh
Browse the web interface for the ResourceManager; by default it is available at:
ResourceManager - http://localhost:8088/
Run a MapReduce job.
When you\'re done, stop the daemons with:
$ sbin/stop-yarn.sh

[hadoop@hadoop000 hadoop]$ pwd /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/etc/hadoop [hadoop@hadoop000 hadoop]$ vi mapred-site.xml [hadoop@hadoop000 hadoop]$ vi yarn-site.xml[hadoop@hadoop000 sbin]$ jps 7234 NodeManager 8131 Jps 7588 NameNode 7962 SecondaryNameNode 7116 ResourceManager 7791 DataNode

http://192.168.43.200:8088/cluster

文章图片

[hadoop@hadoop000 hadoop]$ pwd /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/share/hadoop [hadoop@hadoop000 hadoop]$ ls commonhttpfsmapreducemapreduce2yarn hdfskmsmapreduce1tools [hadoop@hadoop000 hadoop]$ pwd /home/hadoop/app/hadoop-2.6.0-cdh5.15.1/share/hadoop [hadoop@hadoop000 hadoop]$ cd mapreduce [hadoop@hadoop000 mapreduce]$ ls hadoop-mapreduce-client-app-2.6.0-cdh5.15.1.jar hadoop-mapreduce-client-common-2.6.0-cdh5.15.1.jar hadoop-mapreduce-client-core-2.6.0-cdh5.15.1.jar hadoop-mapreduce-client-hs-2.6.0-cdh5.15.1.jar hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.15.1.jar hadoop-mapreduce-client-jobclient-2.6.0-cdh5.15.1.jar hadoop-mapreduce-client-jobclient-2.6.0-cdh5.15.1-tests.jar hadoop-mapreduce-client-nativetask-2.6.0-cdh5.15.1.jar hadoop-mapreduce-client-shuffle-2.6.0-cdh5.15.1.jar hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar lib lib-examples sources

提交example案例到YARN上运行

hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar pi 2 3

文章图片

[hadoop@hadoop000 ~]$ hadoop dfs -cat /wc/input/1.txt DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.helloworldhello hello helloworld [hadoop@hadoop000 ~]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.15.1.jar wordcount /wc/input /wc/output[hadoop@hadoop000 ~]$ hadoop dfs -cat /wc/output/part-r-00000 DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.hello4 world2

文章图片

提交流量统计案例到YARN上运行

mvn clean package -DskipTests 注意在当前的项目环境
(base) locahost:untitled5 jacksun$ mvn clean package -DskipTests [INFO] Scanning for projects... [INFO] [INFO] -----------------------< org.example:untitled5 > ------------------------ [INFO] Building untitled5 1.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ untitled5 --- [INFO] Deleting /Users/jacksun/IdeaProjects/untitled5/target [INFO] [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ untitled5 --- [INFO] Using \'UTF-8\' encoding to copy filtered resources. [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ untitled5 --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 15 source files to /Users/jacksun/IdeaProjects/untitled5/target/classes [INFO] [INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ untitled5 --- [INFO] Using \'UTF-8\' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/jacksun/IdeaProjects/untitled5/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ untitled5 --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 2 source files to /Users/jacksun/IdeaProjects/untitled5/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.22.1:test (default-test) @ untitled5 --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ untitled5 --- [INFO] Building jar: /Users/jacksun/IdeaProjects/untitled5/target/untitled5-1.0-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time:43.078 s [INFO] Finished at: 2020-09-02T10:04:51+08:00 [INFO] ------------------------------------------------------------------------ (base) locahost:untitled5 jacksun$ ls D:accessoutputsrc Hadoop.imlinputpom.xmltarget (base) locahost:untitled5 jacksun$ cd target/ (base) locahost:target jacksun$ ls classesmaven-status generated-sourcestest-classes generated-test-sourcesuntitled5-1.0-SNAPSHOT.jar maven-archiver (base) locahost:target jacksun$

(base) locahost:target jacksun$ scp untitled5-1.0-SNAPSHOT.jar hadoop@192.168.43.200:~/lib/
hadoop@192.168.43.200\'s password:
untitled5-1.0-SNAPSHOT.jar100%18KB 750.6KB/s00:00
(base) locahost:target jacksun$

- 到编译后的`/target/`目录jar包和相关的数据上传到服务器`scp xxx hadoop@localhost:dir`- 再上传到`hdfs `用`Hadoop fs -put/dir`

hadoop jar untitled5-1.0-SNAPSHOT.jar com.bigdata.hadoop.mr.access.AccessYARNApp /access/input/access.log /access/ouput/

- 执行作业` hadoop jar xx.jar `完整的类名和包名` args参数`

20/09/02 10:13:22 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/09/02 10:13:22 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
20/09/02 10:13:23 INFO input.FileInputFormat: Total input paths to process : 1
20/09/02 10:13:24 INFO mapreduce.JobSubmitter: number of splits:1
20/09/02 10:13:24 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1598998523059_0003
20/09/02 10:13:25 INFO impl.YarnClientImpl: Submitted application application_1598998523059_0003
20/09/02 10:13:25 INFO mapreduce.Job: The url to track the job: http://hadoop000:8088/proxy/application_1598998523059_0003/
20/09/02 10:13:25 INFO mapreduce.Job: Running job: job_1598998523059_0003
20/09/02 10:13:35 INFO mapreduce.Job: Job job_1598998523059_0003 running in uber mode : false
20/09/02 10:13:35 INFO mapreduce.Job:map 0% reduce 0%
20/09/02 10:13:42 INFO mapreduce.Job:map 100% reduce 0%
20/09/02 10:13:51 INFO mapreduce.Job:map 100% reduce 33%
20/09/02 10:13:53 INFO mapreduce.Job:map 100% reduce 67%
20/09/02 10:14:01 INFO mapreduce.Job:map 100% reduce 100%
20/09/02 10:14:03 INFO mapreduce.Job: Job job_1598998523059_0003 completed successfully
20/09/02 10:14:03 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=1185
FILE: Number of bytes written=575593
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2444
HDFS: Number of bytes written=552
HDFS: Number of read operations=12
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Job Counters
Killed reduce tasks=1
Launched map tasks=1
Launched reduce tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=13914
Total time spent by all reduces in occupied slots (ms)=71064
Total time spent by all map tasks (ms)=4638
Total time spent by all reduce tasks (ms)=23688
Total vcore-milliseconds taken by all map tasks=4638
Total vcore-milliseconds taken by all reduce tasks=23688
Total megabyte-milliseconds taken by all map tasks=14247936
Total megabyte-milliseconds taken by all reduce tasks=72769536
Map-Reduce Framework
Map input records=23
Map output records=23
Map output bytes=1121
Map output materialized bytes=1185
Input split bytes=110
Combine input records=0
Combine output records=0
Reduce input groups=21
Reduce shuffle bytes=1185
Reduce input records=23
Reduce output records=21
Spilled Records=46
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=696
CPU time spent (ms)=8510
Physical memory (bytes) snapshot=783241216
Virtual memory (bytes) snapshot=16559239168
Total committed heap usage (bytes)=674758656
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=2334
File Output Format Counters
Bytes Written=552
[hadoop@hadoop000 lib]$ hadoop fs -ls /access/ouput/
Found 4 items
-rw-r--r--1 hadoop supergroup0 2020-09-02 10:14 /access/ouput/_SUCCESS
-rw-r--r--1 hadoop supergroup393 2020-09-02 10:13 /access/ouput/part-r-00000
-rw-r--r--1 hadoop supergroup80 2020-09-02 10:13 /access/ouput/part-r-00001
-rw-r--r--1 hadoop supergroup79 2020-09-02 10:13 /access/ouput/part-r-00002
[hadoop@hadoop000 lib]$ hadoop fs -cat /access/ouput/part-r-00000
13480253104,180,180,360
13502468823,7335,110349,117684
13560436666,1116,954,2070
13560439658,2034,5892,7926
13602846565,1938,2910,4848
13660577991,6960,690,7650
13719199419,240,0,240
13726230503,2481,24681,27162
13726238888,12481,44681,57162
13760778710,120,120,240
13826544101,264,0,264
13922314466,3008,3720,6728
13925057413,11058,48243,59301
13926251106,240,0,240
13926435656,132,1512,1644
[hadoop@hadoop000 lib]$

![](https://s4.51cto.com/images/blog/202108/10/7c575fed28554c03b5464a8c5ae029a1.png?x-oss-process=image/watermark,size_14,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=)- 到`http://192.168.43.200:8088/cluster/`观察结果![image.png](https://s2.51cto.com/images/20210810/1628576259203148.png?x-oss-process=image/watermark,size_14,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_20,type_ZmFuZ3poZW5naGVpdGk=)