Hi Folks, Storm is one of best real-time processing System now a days, companies have started using it on large scale. It is one of the best distributed real-time computation System similarly like spark.
You might have heard the integration of storm with many data-pipe line for data processing, Its having one unique feature which is different from other and it is once its start it will run like forever until you kills it.
Basic difference from hadoop is, it is for realtime processing unlike hadoop which is for batch processing.
So lets see how we can get it on our system up and running.
Step 1 :- Download the storm from official apache site and unzip it you will find the couple of folder and storm jar
$ wget https://github.com/downloads/nathanmarz/storm/storm-0.8.1.zip
$ unzip storm-0.8.1.zip
[storm@kafka ~]$ ll storm-0.8.1
total 4780
drwxr-xr-x. 2 storm storm 4096 Sep 6 2012 bin
-rw-r--r--. 1 storm storm 19981 Sep 6 2012 CHANGELOG.md
drwxr-xr-x. 2 storm storm 4096 Jun 25 08:19 conf
drwxrwxr-x. 4 storm storm 4096 Jun 25 05:24 data
drwxr-xr-x. 2 storm storm 4096 Sep 6 2012 lib
-rw-r--r--. 1 storm storm 12710 Sep 6 2012 LICENSE.html
drwxr-xr-x. 2 storm storm 4096 Sep 6 2012 log4j
drwxr-xr-x. 2 storm storm 4096 Jun 25 08:21 logs
-rw-------. 1 storm storm 25640 Jun 25 07:32 nohup.out
drwxr-xr-x. 4 storm storm 4096 Sep 6 2012 public
-rw-r--r--. 1 storm storm 3730 Sep 6 2012 README.markdown
-rw-r--r--. 1 storm storm 6 Sep 6 2012 RELEASE
-rw-r--r--. 1 storm storm 4789764 Sep 6 2012 storm-0.8.1.jar
Step 2:- Download zeromq from its official site
$ wget http://download.zeromq.org/zeromq-2.1.7.zip
$ unzip zeromq-2.1.7.zip
you will see bunch of file and make file , now you need to build it through mvn
$ cd zeromq-2.1.7
$ ./configure && make
if its failing during build you need to run below commands to install required libraries
sudo yum install libuuid*
sudo yum install uuid-*
sudo yum install gcc-*
sudo yum install git
sudo yum install libtool*
Step 3: After configuration of zeromq we need jzmq from git.
$ git clone https://github.com/nathanmarz/jzmq.git
$ ./Makefile.am
$ sed -i 's/classdist_noinst.stamp/classnoinst.stamp/g' src/Makefile.am
$ ./autogen.sh
$ ./configure && make install
Step 4: Download the zookeeper from its official site
$ wget http://www.webhostingreviewjam.com/mirror/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz
$ unzip zookeeper-3.4.6.tar.gz
$ mkdir zookeeper-3.4.6/data
Now update the zoo.conf with data folder and port no.
dataDir=~/data
# the port at which the clients will connect
clientPort=2181
Step 5: Update the storm configuration file and make some entries
$ vi storm.yaml
Step 6:- Start the services nimbus, supervisor , ui########### These MUST be filled in for a storm configuration
storm.zookeeper.servers:
- "192.168.99.141" // your ip address
storm.zookeeper.port: 2181
nimbus.host: "192.168.99.141" // your ip address
nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
supervisor.childopts: "-Djava.net.preferIPv4Stack=true"
worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
nimbus.thrift.port: 8627
ui.port: 8772
storm.local.dir: "/home/storm/storm-0.8.1/data" // your data dir path
java.library.path: "/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
before that add below to .profile file
export STORM_HOME="/home/storm/storm-0.8.1"
export JAVA_HOME="/usr"
export PATH=$STORM_HOME/bin:$JAVA_HOME/bin:$PATH
export ZOOKEEPER_HOME="/home/zookeeper/zookeeper-3.4.6"
export PATH=$ZOOKEEPER_HOME/bin:$PATH
Now start the services in background.
$ zkServer.sh start
$ nohup storm nimbus &
$ nohup storm supervisor &
$ nohup strom ui &
Now you can see the services running like
[storm@kafka ~]$ jps
3354 core
3247 nimbus
3440 Jps
3332 supervisor
3083 QuorumPeerMain
You can view the web ui at http://localhost:8772
