Hadoop Cluster Disaster Recovery Solution 2/2

Hi Folks, In our last blog we have discussed about the synchronous data replication across the cluster which is pretty much expensive in term of network and performance. Today we will talk about  the asynchronous data replication which less expensive then the previous one.

So lets start, how we can go with asynchronous data replication, what kind or design we required to setup that and how we will make this work.

In above picture we can see the designing of cross data replication, let focus how it works between two clusters.

1. When a client is writing a HDFS file, after the file is created, it starts to request a new block. And the primary cluster Active NameNode will allocate a new block and select a list of DataNodes for the client to write to. For the file which needs only asynchronous data replication, no remote DataNode from mirror cluster is selected for the pipeline at Active NameNode.
2. As usual, upon a successful block allocation, the client will write the block data to the first DataNode in the pipeline and also giving the remaining DataNodes.
3. As usual, the first DataNode will continue to write to the following DataNode in the pipeline until the last. But this time the pipeline doesn’t span to the mirror cluster.
4. Asynchronously, the mirror cluster Active NameNode will actively schedule to replicate data blocks which are not on any of the local DataNodes. As part of heartbeats it will send MIRROR_REPLICATION_REQUEST which will contain batch of blocks to replicate with target DataNodes selected from mirror cluster. The mirror cluster doesn’t need to aware of real block location in primary cluster.
5. As a result of handling the MIRROR_REPLICATION_REQUEST, the primary cluster Active NameNode takes care of selecting block location and schedules the replication command to corresponding source DataNode at primary cluster.
6. A DataNode will be selected to replicate the data block from one of the DataNodes in primary cluster that hold the block.
7. As a result of the replication pipeline, the local DataNode can replicate the block to other DataNodes of the mirror cluster.

Asynchronous Namespace Journaling 

Synchronous journaling to remote clusters means more latency and performance impact. When the performance is critical, the admin can configure an asynchronous edit log journaling. 

1. As usual, the primary cluster Active NameNode writes the edit logs to Shared Journal of the primary cluster.
   
2. As usual, the primary cluster Standby NameNode tails the edit logs from Shared Journal of the primary cluster.

3. The mirror cluster Active NameNode tails the edit logs from Shared Journal of the primary cluster. And applies the edit logs to its namespace in memory.
   
4. After applying the edit logs to its namespace, the mirror cluster Active NameNode also writes the edit logs to its local Shared Journal.
   
5. As usual, the mirror cluster Standby NameNode tails the edit logs from Shared Journal of the mirror cluster. 
   

Points to remember

1. Better performance and low letency then the synchronous data replication.
2. Chance of data loss while asynchronous data replication and primary went down.
3. Required when performance is critical then the data.