Juju Java Cluster – Part 3

In my previous post, we added Memcached to our cluster.  In this post, I’ll write a bit more about the Tomcat configuration options that are available including JMX monitoring.  I’ll also show how easy it is to enable session clustering.

Java cluster with JMX and session clusteringConfiguration and Monitoring

All charms come with many options available for configuration.  Each is selected to allow the same tuning you would typically perform on a manually deployed machine.  Configuration options are shown per charm when browsing the Charm Store (jujucharms.com/charms/precise).  The Tomcat charm provides numerous options.  For example, to tweak the JVM options of a running service:

juju set tomcat "java_opts=-Xms768M -Xmx1024M"

This sets the Java heap to a miminum and maximum of 768Mb and 1024Mb respectively.  If you are debugging an application, you may also set:

juju set tomcat "java_opts=-Xms768M -Xmx1024M -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=webapps"

To create a ‘.hprof’ Java heap dump you can inspect with VisualVM or jhat each time an OutOfMemoryError occurs.

To open a remote debugger:

juju set tomcat debug_enabled=True

This will open a JDWP debugger on port 8000 you can use to step-through code from Eclipse, Netbeans etc.  (Note: The debugger is never exposed to the Internet, so you need to access it through a ssh tunnel – ‘ssh -L 8000:localhost:8000 ubuntu@xxx.computer.amazonaws.com’, then connect your IDE to localhost port 8000).

A useful part of the JVM is JMX monitoring.  To enable JMX:

juju set tomcat jmx_enabled=True
juju set tomcat "jmx_control_password=<password>"
juju set tomcat "jmx_monitor_password=<password>"

This will start a remote JMX listener on ports 10001, 10002 and set passwords for the ‘monitorRole’ and ‘controlRole’ users (not setting a password disables that account).  You can now open VisualVM or JConsole to connect to the remote JMX instance (screenshot below).  (Note: JMX is never exposed to the Internet, so you need to access it through a ssh tunnel – ‘ssh -L 10001:localhost:10001 -L 10002:localhost:10002 ubuntu@xxx.computer.amazonaws.com’, then connect your JMX client to port 10001).  You can easily expose your own application specific MBeans for monitoring by adding them to the platform MBeanServer.

Juju JMX monitoringOptions are applied to services and to all units under a service.  It isn’t possible to apply options to a specific unit.  So if you enable debugging, you enable it for all Tomcat units.  Same with Java options.

Options may also be applied at deployment time.  For example, to use Tomcat 6 (rather than the default Tomcat 7), create a ‘config.yaml’ file containing the following:

tomcat:
  tomcat_version: tomcat6

Then deploy:

juju deploy --config config.yaml cs:~robert-ayres/precise/tomcat

All units added via ‘add-unit’ will also be Tomcat 6.

Session Clustering

Previously, we setup a Juju cluster consisting of two Tomcat units behind HAProxy.  In this configuration, HTTP sessions exist only on individual Tomcat units.  For many production setups, the use of load balancer sticky sessions and a non-replicated session is the most performant where HTTP sessions are either not required or expendable in the event of unit failure.  For setups concerned about availability of sessions, you can enable Tomcat session clustering on your Juju service which will replicate session data between all units in the service.  Should a unit fail, any of the remaining units can pickup the subsequent requests with the previous session state.  To enable session clustering:

juju set tomcat cluster_enabled=True

We have two choices of how the cluster manages membership.  The preferred choice is using multicast traffic, but as EC2 doesn’t allow this, we must use static configuration.  This is the default, but you can switch between either method by changing the value of the ‘multicast’ option.  Like everything else Juju deployed, any new units added or removed via ‘add-unit’ or ‘remove-unit’ are automatically included/excluded from the cluster membership.  This easily allows you to toggle clustering so that you can benchmark precisely what latency/throughput cost you have by using replicated sessions.

In summary, I’ve shown how you can tweak Tomcat configuration including enabling JMX monitoring.  We’ve also seen how to enable session clustering.  In my final post of the series, I shall show how you can add Solr indexing to your application.