Hadoop Installation (type RPM )

Hi Folks,

Today we are going for RPM installation of hadoop. It is also pretty easy as my last hadoop installtion was ,  So lets try it out.

Requirement

• Java JDK (download from here)
• hadoop-0.20.204.0-1.i386.rpm  (Download from here)

Installation

1. Installation of Java and set Java Home on /etc/profile by export JAVA_HOME=/usr

sudo ./jdk-6u26-linux-x64-rpm.bin.sh

2. Hadoop RPM installation

sudo rpm -i hadoop-0.20.204.0-1.i386.rpm

3. Setting up Single Node cluster

sudo /usr/sbin/hadoop-setup-single-node.sh 

You will get many question while setting we up hadoop , like creation of directories and some configuration related, you need to give answer in y.

 For MultiNode Setup You Need to run below commands

3- Setting up Multinode Cluster

sudo /usr/sbin/hadoop-setup-conf.sh \
  --namenode-host=hdfs://${namenode}:9000/ \
  --jobtracker-host=${jobtracker}:9001 \
  --conf-dir=/etc/hadoop \
  --hdfs-dir=/var/lib/hadoop/hdfs \
  --namenode-dir=/var/lib/hadoop/hdfs/namenode \
  --mapred-dir=/var/lib/hadoop/mapred \
 --mapreduce-user=mapred \
  --datanode-dir=/var/lib/hadoop/hdfs/data \
  --log-dir=/var/log/hadoop \
  --auto

 Where $namenode and $jobtracker are the Hostname of respective Nodes where you want to run the services, you have to fire this command on everyNode.

4. Now after installation you have to format the namenode

sudo /usr/sbin/hadoop-setup-hdfs.sh

5.  For Starting services you can do as below

• For single Node

for service in /etc/init.d/hadoop-* ;do sudo  $service  start ; done

•  For Multinode
• on Master Node

sudo  /etc/init.d/hadoop-namenode start
sudo  /etc/init.d/hadoop-jobtracker start 
sudo  /etc/init.d/hadoop-secondarynamenode start 

• on Slave Node

sudo  /etc/init.d/hadoop-datanode start
sudo  /etc/init.d/hadoop-tasktracker start 

6. You can Create a User Account for you self on HDFS by below command

sudo /usr/sbin/hadoop-create-user.sh -u $USER

Now You can run the word count program as given in previous post. Please try it out and let me know if faced any issue in this.

Thanks

Hadoop Installation (CDH4 - Yum installation)

Hi Folks,

Today we are going for yum installation of CDH4. its pretty easy one.

Requirement

•  Oracle JDK 1.6
•  CentOS 6.4

Installation

1. Downloading the CDH4 Repo file

sudo wget -O /etc/yum.repos.d/cloudera-cdh4.repo http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/cloudera-cdh4.repo

2.  Download cloudera cdh4

sudo yum install hadoop-0.20-conf-pseudo

3. Formatting the namenode

sudo -u hdfs hdfs namenode -format

4.Starting HDFS Services on respective nodes

• Namenode Services on Master Node
  

  sudo service hadoop-hdfs-namenode start
  sudo service hadoop-hdfs-secondarynamenode start

• Datanode Services on Master Node(becoz its pseudo mode)

sudo service hadoop-hdfs-datanode start

5. Creating Hdfs Directories on Master

sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
sudo -u hdfs hadoop fs -mkdir /user

6. Creating Map-reduce Directories on  Master node

sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred
sudo -u hdfs mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs chown hdfs:hadoop /var/lib/hadoop-hdfs/cache/mapred
sudo -u hdfs chown -R mapred /var/lib/hadoop-hdfs/cache/mapred 

 7. Starting Mapreduce Services on master and on Slaves

• JobTracker Services on Master Node

   sudo service hadoop-0.20-mapreduce-jobtracker start

•  TaskTracker Service on master Node

  sudo service hadoop-0.20-mapreduce-tasktracker start

8. Creating Home Directory for Users like hdfs and mapred, replace $user with hdfs and mapred

sudo -u hdfs hadoop fs -mkdir /user/$USER
sudo -u hdfs hadoop fs -chown $USER /user/$USER

 9. Update export in .profile

export HADOOP_HOME=/usr/lib/hadoop

 10. You can check hdfs directory by

sudo -u hdfs hadoop fs -ls  /

Try running any sample job by cmd below.

sudo -u hdfs hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5  10

 NOTE: Please comment you have any problem in it.