Resource Manager component

The Resource Manager is a primary core component of YARN. Each client interaction will always involve the Resource Manager for some reason. YARN provides a few major components for the Resource Manager, as shown here: 

• Client component: The Resource Manager has exposed methods to clients for initiating RPC communication with itself. The YARN Resource Manager provides the ClientRMService class, exposes the API for application requests, such as submitting a new job, or a new application request, and killing the application. It also contains the API to get the cluster metrics. It also provides the AdminService class, which is used by the cluster administrator for managing the Resource Manager services. Admin can check cluster health, access information, and refresh cluster nodes. Admin uses the rmadmin command to execute any operation that internally uses the service provided by the AdminService class.
• Core component: The scheduler and the application manager are the major part of the Resource Manager core interface. The following are a few interfaces provided by the Resource Manager:
  •  YarnScheduler provides APIs for resource allocation and cleanup operation. YarnScheduler works on the pluggable policy and based on the configuration, it distributes cluster resources to the applications.
  • RMStateStore is another core interface that provides implementation for the Resource Manager state store so that in case of failure, the Resource Manager can recover its state using the implementation provided. A few implementation of this class are FileSystemRMStateStore, MemoryRMStateStore, ZKRMStateStore, NullRMStateStore. The state of the Resource Manager plays an important role in the Resource Manager high availability implementation. The ZKRMStateStore is a very reliable implementation for this purpose as it uses the ZooKeeper to maintain state.
  • SchedulingMonitor provides an interface to define the scheduling policy and a way to edit schedules at regular intervals. It also interfaces to monitor containers and the tuning of resources.
  • RMAppManager is responsible for managing the list of applications running on the YARN cluster. It collects and stores runtime information of the application and provides information to YARN on request.
• Node Manager component: The Resource Manager is a master node and Node Manager is slave node. Node Manager sends heartbeats, resource information, status information, and so on to the Resource Manager at regular intervals. The Resource Manager keeps an updated status of each Node Manager, which helps it in allocating resource to the application master. ResourceTrackerService is responsible for responding to the Node Managers' RPC requests. It contains APIs for the registration of new nodes to the cluster and receiving heartbeats from the Node Managers. It also contains the NMLivelinessMonitor object, which helps in monitoring all the live and dead nodes. In general, if the node does not send a heartbeat for 10 minutes, it is considered as dead and will not be used to launch any new containers. You can increase or decrease the interval by setting YARN.am.liveness-monitor.expiry-interval-ms in the YARN configuration (time in milliseconds). The constructor of ResourceTrackerService looks like the following code: 

        public ResourceTrackerService(RMContext rmContext,    
        NodesListManager nodesListManager, NMLivelinessMonitor 
        nmLivelinessMonitor, RMContainerTokenSecretManager 
        containerTokenSecretManager, NMTokenSecretManagerInRM 
        nmTokenSecretManager

• NMLivelinessMonitor is responsible for tracking all the dead and live nodes. As discussed earlier, any Node Manager that does not send a heartbeat to the Resource Manager for 10 minutes is considered dead and will not be used for running any container on it. 

• Application master component: YARN launches a new application master for every new application request it receives from a client. The application master acts as a mediator between the Resource Manager and the Node Manager for resource requirements of its application. The Resource Manager provides an API to manage the application master. The following are some classes and internals of how it works:
  • AMLivelinessMonitor works very similarly to NMLivelinessMonitor and sends heartbeats to the Resource Manager. The Resource Manager keeps track of all the live and dead application masters and if it does not receive heartbeats from any application master for 10 minutes, then it marks the application master as dead. The containers used by these dead application masters are then destroyed and the Resource Manager launches a new application master for the application on a new container. If still the application master fails, the Resource Manager repeats the process for four attempts and then sends a failure message to the client. The number of attempts for relaunching the application master is configurable. 
  • ApplicationMasterService is responsible for responding to the application master's RPC request and provides an API for the new application master registration, container requests from all application masters, and so on. It then forwards the request to the YARN scheduler for further processing.