Fault tolerance enhancement on Apache Hadoop 3.0.0-alpha2 by supporting more than 2 NameNodes.

NameNode is the most critical resource in Hadoop core cluster. Once very large files loaded into the Hadoop Distributed File System (HDFS), the files get broken into block-sized chunks as per the parameter configured (64 MB by default). The chunks are then stored as independent units across the data nodes in the cluster. The primary responsibility of the data nodes is to hold the actual data in the form of chunk and NameNode holds the information where all the chunks located/stored in the data nodes. Basically, NameNode manages the filesystem namespace. By maintaining filesystem tree and the metadata for all the files and directories in the tree, the NameNode is getting recognized as Master Node in the entire cluster. Besides, the NameNode stores data nodes location, replicas and other details. If the NameNode in the cluster gets crushed or removed/isolated, we won’t be able to perform any kind of operations in the data nodes and eventually, Hadoop core cluster becomes incompetent. Prior to Hadoop 2.0.0, the NameNode was a single point of failure (SPOF) in an HDFS cluster. Analyzing the importance of the NameNode, standby or secondary NameNode was introduced in Hadoop 2.0.0. Secondary NameNode is not a backup of NameNode (Active), but instead works as helper of the primary/active NameNode by storing a copy of FsImage file and edits log. It also, periodically applies edits log records to FsImage file and refreshes the edits log. The NameNode leverages the updated FsImage file in order to avoid re-applying the EditLog records during its startup process. If NameNode fails/crushes, File System metadata can be recovered from the last saved FsImage on the Secondary NameNode but Secondary NameNode can’t be placed as the primary NameNode.

A group of separate daemons called JournalNodes has been utilized by the secondary NameNode to keep its state synchronized with active NameNode. The JournalNodes internally relies on Quorum-based Storage, acts as the middleman between the active and secondary NameNode. For any namespace modification performed by the active NameNode, it securely logs record of the modification to the JournalNodes. The standby/secondary NameNode has the capability of reading the edits from the JournalNodes and is constantly monitoring them for changes to the edit log. The combination of a single active NameNode with a single Standby NameNode can be utilized as an initial implementation of HDFS NameNode high-availability. The single point of failure in the Hadoop cluster prior to Hadoop 2.0.0 release has been eliminated using the former procedure.
Due to the adoption of Hadoop framework by many organizations to process vast/very large amount of enterprise data, some of the deployments require high degree of fault tolerance. Apache Hadoop 3.0.0-alpha2 is going to address this by supporting more than 2 NameNode. With this new feature, the user can run multiple standby NameNodes. With the configuration of three NameNodes and five JournalNodes, the Hadoop cluster would have the capacity to tolerate the failure of two nodes rather than just one.

Written by
Gautam Goswami