Storm Installation on CentOs

Hi Folks, Storm is one of best real-time processing System now a days, companies have started using it on large scale. It is one of the best distributed real-time computation System similarly like spark.

You might have heard the integration of storm with many data-pipe line for data processing, Its having one unique feature which is different from other and it is once its start it will run like forever until you kills it.

Basic difference from hadoop is, it is for realtime processing unlike hadoop which is for batch processing.

So lets see how we can get it on our system up and running.

Step 1 :- Download the storm from official apache site and unzip it you will find the couple of folder and storm jar

 $  wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip
 $ unzip storm-0.8.1.zip

[storm@kafka ~]$ ll storm-0.8.1
total 4780
drwxr-xr-x. 2 storm storm    4096 Sep  6  2012 bin
-rw-r--r--. 1 storm storm   19981 Sep  6  2012 CHANGELOG.md
drwxr-xr-x. 2 storm storm    4096 Jun 25 08:19 conf
drwxrwxr-x. 4 storm storm    4096 Jun 25 05:24 data
drwxr-xr-x. 2 storm storm    4096 Sep  6  2012 lib
-rw-r--r--. 1 storm storm   12710 Sep  6  2012 LICENSE.html
drwxr-xr-x. 2 storm storm    4096 Sep  6  2012 log4j
drwxr-xr-x. 2 storm storm    4096 Jun 25 08:21 logs
-rw-------. 1 storm storm   25640 Jun 25 07:32 nohup.out
drwxr-xr-x. 4 storm storm    4096 Sep  6  2012 public
-rw-r--r--. 1 storm storm    3730 Sep  6  2012 README.markdown
-rw-r--r--. 1 storm storm       6 Sep  6  2012 RELEASE
-rw-r--r--. 1 storm storm 4789764 Sep  6  2012 storm-0.8.1.jar

Step 2:- Download zeromq from its official site 

$ wget http://download.zeromq.org/zeromq-2.1.7.zip

$ unzip  zeromq-2.1.7.zip

you will see bunch of file and make file , now you need to build it through mvn 

$ cd  zeromq-2.1.7

$  ./configure && make 

if its failing during build you need to run below commands to install required libraries

sudo yum install libuuid*
sudo yum install uuid-*
sudo yum install gcc-*  
sudo yum install git
sudo yum install libtool*

Step 3: After configuration of zeromq we need jzmq from git.

 $  git clone https://github.com/nathanmarz/jzmq.git
 $ ./Makefile.am
 $  sed -i 's/classdist_noinst.stamp/classnoinst.stamp/g' src/Makefile.am
 $  ./autogen.sh
 $ ./configure && make install 

Step 4: Download the zookeeper from its official site

$ wget http://www.webhostingreviewjam.com/mirror/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz 
$ unzip zookeeper-3.4.6.tar.gz
$ mkdir zookeeper-3.4.6/data

 Now update the zoo.conf with data folder and port no.


dataDir=~/data
# the port at which the clients will connect
clientPort=2181 
 
Step 5: Update the storm configuration file and make some entries

$ vi storm.yaml

########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "192.168.99.141"     // your ip address
storm.zookeeper.port: 2181
nimbus.host: "192.168.99.141"    // your ip address
nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
supervisor.childopts: "-Djava.net.preferIPv4Stack=true"
worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
nimbus.thrift.port: 8627
ui.port: 8772
storm.local.dir: "/home/storm/storm-0.8.1/data"  // your data dir path
java.library.path: "/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/" 
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703 

Step 6:- Start the services nimbus, supervisor , ui 

before that add below to .profile file

export STORM_HOME="/home/storm/storm-0.8.1"
export JAVA_HOME="/usr"
export PATH=$STORM_HOME/bin:$JAVA_HOME/bin:$PATH
export ZOOKEEPER_HOME="/home/zookeeper/zookeeper-3.4.6"
export PATH=$ZOOKEEPER_HOME/bin:$PATH
 
Now start the services in background.

$  zkServer.sh start
$ nohup storm nimbus &
$ nohup storm supervisor &
$ nohup strom ui & 

Now you can see the services running like

[storm@kafka ~]$ jps
3354 core
3247 nimbus
3440 Jps
3332 supervisor
3083 QuorumPeerMain

 You can view the web ui at http://localhost:8772