锐客网

  1. 首页 > 睿知 > it技术 > >

对Hadoop2.7.2文档的学习-Yarn部分Yarn Application

IT知识applicationApplicatio

非淡泊无以明志，非宁静无以致远。这篇文章主要讲述对Hadoop2.7.2文档的学习-Yarn部分Yarn Application相关的知识，希望能为你提供帮助。
Writing YARN Applications文档中的启动过程：
Application submission client向Yarn ResourceManager提交一个Application，RM、NM、AM处理流程。
首先，创建一个YarnClient对象并start它，然后Client可以设置ApplicationContext。为app准备第一个container来contain ApplicationMaster,然后提交Application。
RM在已经指定的Container中启动ApplicationMaster。AM与YARN集群通信，处理Application的执行。在app启动的过程中（app的启动过程中，AM与RM的通信是异步的），AM的主要工作包括：
（1）与RM通信，协商为之后的Containers分配资源（通过AMRMClientAsync对象，AMRMClientAsync.CallbackHandler指定事件的处理方法）；
（2）Container分配之后，与NodeManagers通信，启动它们所在节点的app的Containers（启动一个Runnable对象，当为Containers分配资源之后，启动containers。作为启动Container的一部分，AM需要指定带有启动信息的ContainerLaunchContext）。
在Application执行的过程中，ApplicationMaster通过NMClientAsync对象与NodeManagers通信。所有Containers的事件由NMClientAsync.CallbackHandler处理。
一个callback handler处理Client的start，stop，status update以及error。
 
 
（1）  创建一个YarnClient对象并start它，然后Client可以设置ApplicationContext，然后向ResourceManager提交Application。
（2）RM向NM发出指令，为该App启动第一个Container，并在其中启动ApplicationMaster
（3）AM向RM注册
（4）AM采用轮询的方式向RM的YARN Scheduler申请资源
（5）当AM申请到资源后（即获取到了空闲节点的信息），与NodeManagers通信（多个NodeManager），请求启动计算任务
（6）NodeManagers根据资源量的大小、所需的运行环境，在Container中启动任务。
【对Hadoop2.7.2文档的学习-Yarn部分Yarn Application】（7）各个任务向AM汇报自己的状态和进度，以便AM掌握各个任务的执行情况
（8）APP运行完成后，AM向RM注销并关闭自己。
 
 

对Hadoop2.7.2文档的学习-Yarn部分Yarn Application

文章图片

 
对Hadoop2.7.2文档的学习-Yarn部分Yarn Application

文章图片

 
 
原文：
The general concept is that an  application submission client  submits an  application  to the YARN  ResourceManager  (RM). This can be done through setting up a  YarnClient  object. After  YarnClient  is started, the client can then set up application context, prepare the very first container of the application that contains the  ApplicationMaster  (AM), and then submit the application. You need to provide information such as the details about the local files/jars that need to be available for your application to run, the actual command that needs to be executed (with the necessary command line arguments), any OS environment settings (optional), etc. Effectively, you need to describe the Unix process(es) that needs to be launched for your ApplicationMaster.
The YARN ResourceManager will then launch the ApplicationMaster (as specified) on an allocated container. The ApplicationMaster communicates with YARN cluster, and handles application execution. It performs operations in an asynchronous fashion. During application launch time, the main tasks of the ApplicationMaster are: a) communicating with the ResourceManager to negotiate and allocate resources for future containers, and b) after container allocation, communicating YARN *NodeManager*s (NMs) to launch application containers on them. Task a) can be performed asynchronously through an  AMRMClientAsync  object, with event handling methods specified in a  AMRMClientAsync.CallbackHandler  type of event handler. The event handler needs to be set to the client explicitly. Task b) can be performed by launching a runnable object that then launches containers when there are containers allocated. As part of launching this container, the AM has to specify the  ContainerLaunchContext  that has the launch information such as command line specification, environment, etc.
参考：
（1）《Hadoop The Definitive Guide 4th》
（2）http://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html

    推荐阅读


    上一篇：Android Studio打开出现（Default activity not found）

    下一篇：android中资源的使用