Hadoop Cluster Disaster Recovery Solution 1/2

Hi Folks, whenever we think about the cluster setup and design. We always think about DR how can we save our Data from cluster crash. Today we discuss about the disaster recovery plan for hadoop cluster, what would be steps we can take and how far we can save our data.

Type of Cluster design across data center

1. Synchronous Data Replication between cluster
2. ASynchronous Data Replication between cluster

lets talk about the Synchronous Data writing between cluster. Here is the pictorial view of data center design.

1.  

1. When a client is writing a HDFS file, after the file is created, it starts to request a new block. And the Active NameNode of primary cluster will allocate a new block and select a list of DataNodes for the client to write to. By using the new mirror block placement policy, the Active NameNode can guarantee one or more remote DataNodes from the mirror cluster are selected at the end of the pipeline.
2. The primary cluster Active NameNode knows the available DataNodes of the mirror cluster via heartbeats from mirror cluster’s Active NameNode with the MIRROR_DATANODE_AVAILABLE command. So, latest reported DataNodes will be considered for the mirror cluster pipeline which will be appended to primary cluster pipeline.
3. As usual, upon a successful block allocation, the client will write the block data to the first DataNode in the pipeline and also giving the remaining DataNodes.
4. As usual, the first DataNode will continue to write to the following DataNode in the pipeline.
5. The last local DataNode in the pipeline will continue to write the remote DataNode that following.
6. If there are more than one remote DataNodes are selected, the remote DataNode will continue to write to the following DataNode which is local to the remote DataNode. We provide flexibility to users that they can even configure the mirror cluster replication. Based on the configured replication, mirror nodes will be selected. 

Synchronous Namespace Journaling 

1. As usual, the primary cluster Active NameNode writes the edit logs to Shared Journal of the primary cluster.
   
2. The primary cluster Active NameNode also writes the edit logs to the mirror cluster Active NameNode by using a new JournalManager.
   
3. As usual, the primary cluster Standby NameNode tails the edit logs from Shared Journal of the primary cluster.
   
4. The mirror cluster Active NameNode writes the edit logs to Shared Journal of the mirror cluster after applying the edit logs received from the primary cluster.
   
5. As usual, the mirror cluster Standby NameNode tails the edit logs from Shared Journal of the mirror cluster. 
   

Points to Remember

1. Synchronous Data writing is good when the data is very critical and we cant afford to lose consistency at any point of time.
2. It Actually increase the latency of hadoop data writing, which impact performance of the hadoop cluster.
3. Required more network bandwidth and  stability to cope with synchronous replication.