Tuesday, May 01, 2012

Axis2 clustering on Tomcat


In this post I will discuss how to setup a Axis2 cluster using two Tomcat servers.  It's not my intention to describe Axis2 cluster architecture or cluster configuration language, if you need such details refer provided references at the end of this post. Axis2 cluster implementation is based on pure API which you can implement using any Java multicast communication framework, by default Axis2 provide a cluster implemantation based on Tomcat Cluster Communication Module also known as Apache Tribes. Following diagram illustrate the design we are going to discuss in this post but I skip load balancer setup for simplicity but in real world scenario you could use load balancer such as Apache2 server or WSO2 Load Balancer that support for more advanced options.


Pre-requirements   

1. Apache Tomcat server. ( Version 7.x.x preferable. )
2. Apache Axis2 WAR distribution (Version 1.6.2 preferable.)


Here I use same machine to setup two Tomcat server instances hence it's required to change server configuration of one instance. Let's say Node-1 having default configuration and Node-2 having custom configuration. Open the server.xml file and change server port, port nubers of HTTP and AJP Connectors as follows.    

Node-1 (default configuration)
 <Server port="8005" shutdown="SHUTDOWN">  
 <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />  
 <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />  

Node-2 
 <Server port="9005" shutdown="SHUTDOWN">  
 <Connector port="9090" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="9443" />  
 <Connector port="9009" protocol="AJP/1.3" redirectPort="9443" />  

Now install Axis2 WAR distribution on both server instances and stop both servers to edit axis2 configuration file. Open axis2.xml file and edit cluster settings as follows. 

 Node -1

     <parameter name="AvoidInitiation">false</parameter>   
     <parameter name="domain">sample.cluster.domain</parameter>  
     <parameter name="mcastBindAddress">127.0.0.1</parameter>  
     <parameter name="localMemberHost">127.0.0.1</parameter>  
     <parameter name="localMemberPort">4000</parameter>  


  Node -2 

     <parameter name="AvoidInitiation">false</parameter>   
     <parameter name="domain">sample.cluster.domain</parameter>  
     <parameter name="mcastBindAddress">127.0.0.1</parameter>  
     <parameter name="localMemberHost">127.0.0.1</parameter>  
     <parameter name="localMemberPort">4001</parameter>   

Note that in a real network setup it's required to edit mcastBindAddress, localMemberHost settings in addition to  localMemberPort value but in my local machine only localMemberPort has changed.  After this has done start the Node -1, if there is no issue in your setup you could able to see log messages as follows on node-1 server console. 

 [INFO] Initializing cluster...  
 [INFO] Cluster domain: sample.cluster.domain  
 [INFO] Using multicast based membership management scheme  
 Apr 30, 2012 6:37:13 PM org.apache.catalina.tribes.transport.ReceiverBase bind  
 INFO: Receiver Server Socket bound to:/127.0.0.1:4000  
 Apr 30, 2012 6:37:13 PM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket  
 INFO: Setting cluster mcast soTimeout to 500  
 Apr 30, 2012 6:37:13 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4  
 Apr 30, 2012 6:37:14 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Done sleeping, membership established, start level:4  
 Apr 30, 2012 6:37:14 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:8  
 Apr 30, 2012 6:37:15 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Done sleeping, membership established, start level:8  
 [WARN] Local member advertising its IP address as 127.0.0.1. Remote members will not be able to connect to this member.  
 [INFO] Local Member 127.0.0.1:4000(sample.cluster.domain)  
 [INFO] No members in current cluster  
 [INFO] Cluster initialization completed.  
 Apr 30, 2012 6:37:15 PM org.apache.catalina.startup.HostConfig deployDirectory  
 INFO: Deploying web application directory /home/sagara/dev/servers/axis2-clustering/tomcat/node1/webapps/ROOT  
 Apr 30, 2012 6:37:15 PM org.apache.coyote.AbstractProtocol start  
 INFO: Starting ProtocolHandler ["http-bio-8080"]  
 Apr 30, 2012 6:37:15 PM org.apache.coyote.AbstractProtocol start  
 INFO: Starting ProtocolHandler ["ajp-bio-8009"]  
 Apr 30, 2012 6:37:15 PM org.apache.catalina.startup.Catalina start  
 INFO: Server startup in 3068 ms  


Now start the Node -2 and monitor log messages on Node-2 console 
 [INFO] Initializing cluster...  
 [INFO] Cluster domain: sample.cluster.domain  
 [INFO] Using multicast based membership management scheme  
 Apr 30, 2012 6:37:50 PM org.apache.catalina.tribes.transport.ReceiverBase bind  
 INFO: Receiver Server Socket bound to:/127.0.0.1:4001  
 Apr 30, 2012 6:37:50 PM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket  
 INFO: Setting cluster mcast soTimeout to 500  
 Apr 30, 2012 6:37:50 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4  
 [INFO] New member 127.0.0.1:4000(sample.cluster.domain ) joined cluster.  
 Apr 30, 2012 6:37:51 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Done sleeping, membership established, start level:4  
 Apr 30, 2012 6:37:51 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:8  
 Apr 30, 2012 6:37:51 PM org.apache.catalina.tribes.io.BufferPool getBufferPool  
 INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl  
 Apr 30, 2012 6:37:52 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers  
 INFO: Done sleeping, membership established, start level:8  
 [WARN] Local member advertising its IP address as 127.0.0.1. Remote members will not be able to connect to this member.  
 [INFO] Local Member 127.0.0.1:4001(sample.cluster.domain )  
 [INFO] Members of current cluster  
 [INFO] Member1 127.0.0.1:4000(sample.cluster.domain )  
 [INFO] Trying to send initialization request to 127.0.0.1:4000(sample.cluster.domain )  
 [INFO] Received configuration initialization message  
 [INFO] Trying to send initialization request to 127.0.0.1:4000(sample.cluster.domain )  
 [INFO] Received state initialization message  
 [INFO] Cluster initialization completed.  
 Apr 30, 2012 6:37:52 PM org.apache.catalina.startup.HostConfig deployDirectory  
 INFO: Deploying web application directory /home/sagara/dev/servers/axis2-clustering/tomcat/node2/webapps/ROOT  
 Apr 30, 2012 6:37:52 PM org.apache.coyote.AbstractProtocol start  
 INFO: Starting ProtocolHandler ["http-bio-9090"]  
 Apr 30, 2012 6:37:52 PM org.apache.coyote.AbstractProtocol start  
 INFO: Starting ProtocolHandler ["ajp-bio-9009"]  
 Apr 30, 2012 6:37:52 PM org.apache.catalina.startup.Catalina start  
 INFO: Server startup in 13334 ms   


Additionally now you should able to see following log messages on Node-1 server console. 
 Apr 30, 2012 6:37:50 PM org.apache.catalina.tribes.io.BufferPool getBufferPool  
 INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl  
 [INFO] New member 127.0.0.1:4001(sample.cluster.domain) joined cluster.  
 [INFO] Received GetConfigurationCommand initialization request message from 127.0.0.1:4001(sample.cluster.domain)  
 [INFO] Received GetStateCommand initialization request message from 127.0.0.1:4001(sample.cluster.domain) 


If you have followed me up to this point you have successfully setup a Axis2 cluster with two nodes. Now you can deploy any cluster aware web service on this cluster. For testing purposes let's write following POJO service and will deploy on both servers. I have given service code and service.xml  below. 

 package sample;  
 import org.apache.axis2.context.MessageContext;  
 public class Count {  
   public int count() {  
     int count;  
     MessageContext mc = MessageContext.getCurrentMessageContext();  
     Object ob = mc.getConfigurationContext().getProperty("count");  
     if (ob == null) {  
       count = 1;  
     } else {  
       count = (Integer) ob;  
       count++;  
     }  
     mc.getConfigurationContext().setProperty("count", count);  
     return count;  
   }  
 }  

Note that we use  ConfigurationContext to store our count values, ConfigurationContext is replicate among cluster members.


 <service name="count">   
   <parameter name="ServiceClass">sample.Count</parameter>  
   <operation name="count">  
   <messageReceiver class="org.apache.axis2.rpc.receivers.RPCMessageReceiver" />  
   </operation>  
 </service>  

In order to test our cluster aware service we need a Web service client, for the simplicity I will use a browser and Axis2 REST support.  Since we skip the load balancer setup following two different URLs can be use to invoke services . Once you invokes services on both servers you can notice that the count value is shared among two Axis2 instances clearly.

 http://localhost:8080/axis2/services/count/count  
 http://localhost:9090/axis2/services/count/count   

References


No comments: