Configuring Resource Manager high availability

Let's look into how we configure Resource Manager high availability and what are the configuration changes required in the YARN cluster. The YARN configuration changes are available in YARN-site.xml and the following changes are required in order to enable HA:

• Enabling high availability: The first step is to enable the YARN cluster for resource manager high availability and change the YARN.resourcemanager.ha.enabled property to true as shown in the code:

        <property>
          <name>YARN.resourcemanager.ha.enabled</name>
           <value>true</value>
        </property>

• Assign IDs to Resource Manager: Each Resource Manager in the YARN cluster must be configured with unique IDs for its identification. That means if we have three Resource Managers, we must have three IDs, for example:

        <property>
          <name>YARN.resourcemanager.ha.rm-ids</name>
          <value>rm1,rm2,rm3</value>
        </property>

• Attaching Resource Manager hostname to IDs: The IDs configured will be linked with the Resource Manager hostname, which will be uniquely assigned. The hostname mentioned in the following configuration must be replaced by your resource manager hostname or IP address, for example:

        <property> <name>YARN.resourcemanager.hostname.rm1</name>         
        <value>resourcemanager1</value> </property> <property>            
        <name>YARN.resourcemanager.hostname.rm2</name>         
        <value>resourcemanager2</value> </property><property>         
        <name>YARN.resourcemanager.hostname.rm3</name>         
        <value>resourcemanager3</value> </property>

• Configuring Resource Manager web application: The following properties set the port for the RM web application:

        <property>
          <name>YARN.resourcemanager.webapp.address.rm1</name>
          <value>resourcemanager2:8088</value>
        </property>
        <property>
          <name>YARN.resourcemanager.webapp.address.rm2</name>
          <value>resourcemanager2:8088</value>
        </property>
        <property>
          <name>YARN.resourcemanager.webapp.address.rm3</name>
          <value>resourcemanager3:8088</value>
        </property>

• ZooKeeper address: The ZooKeeper plays a very important role in the state store and leader election. It also provides the fencing mechanism for failover. The following property must be set with all zooKeeper node address:

        <property>
          <name>YARN.resourcemanager.zk-address</name>
          <value>Zkhost1:2181,Zkhost2:2181,Zkhost3:2181</value>
        </property>

You can then check the Resource Manager state using the ID assigned to each resource manager and YARN admin commands. For example, the following command will give the state of resourcemanager1:

YARN rmadmin -getServiceState rm1