java.lang.OutOfMemoryError

               Out of memory errors can be pretty tough to crack - especially on high volume applications. The main problem is the lack of tools that work well under heavy loads. Most commercially available tools that works nicely in development fails miserably under high load in the production environments. Unfortunately toughest of the OOMs show up only under production loads; that too usually over a period of time, thus making it next to impossible to reproduce them with the usual load/stress tests. One of the reasons why the commercial tools don't work very well under heavy loads is because they are extremely intrusive. This affects the JVM performance so much that the server CPU usually hits the wall the moment you turn on production traffic. The usual debugging mechanisms, such as rigorous code reviews looking for possible memory leaks and concurrency/performance issues, analysing access logs with error/application logs and verbosegc output, secluding suspect applications into separate JVMs etc helps, and should be sufficient to uncover the culprit code in most cases, but not always!

               So what do you do when everything else fails?! Well.. welcome to the world of the amazing JVMPI/JVMTI APIs. Using JVMPI it is possible to put hooks into JVM to do things that are normally not possible in Java - things such as profiling Java heap, generating object allocation stacktraces etc. This extremely helpful information from the deep insides of the JVM is usually more than sufficient to crack the toughest of the memory leaks. The most important thing here is, unlike the commercial tools which are usually catch all (ie., track everything that happens inside the JVM) you get to pick and choose the specific events that you are interested in. For eg, you can tell JVM to notify only when an object of a particular type or size is created. Even more importantly, you get the ability to generate this information only when you want it. This really is the key to make this work in production environments - the fact that you can programmatically activate/deactivate the tool - to take periodic snap shots of the JVM heap or get allocation stacktraces for a single request etc. The trick is to implement an easily accessible switch that allows to enable/disable the JVMPI module as and when needed - could be as simple as a simple JSP page running in the suspect VM that hooks into the JVMPI module via JNI.

               Sounds interesting and ready for a deep dive? Pl refer to the sample JVMPI utility at http://virtualmachine.de/ and the official JVM PI site.