ActiveMQ / Spring Blues

Well, this post is just a whine about my problems with two technologies which individually I have used extensively and have both worked well, but if used together have been disastrous for me.

JMS (in particular ActiveMQ): I created a logging framework for our web applications including a utility app that glues the pieces together. Basically this setup uses Log4j JMSAppender, ActiveMQ and a few pieces that I coded to allow dynamic control of logging levels and aggregation and daisy-chaining of the multiple clusters of servers at our data center back to our main office. This setup works great and was a piece of cake to setup.

Spring (using Appfuse 2.0.2 as a start): I used this to create a backend processor for our web application to offload various functions from the main web applications and give us an asynchronous way for making our entire setup more scalable. This approach works great also.

Individually these two technologies have been fantastic and I would highly recommend to anyone considering there use - with one caveat. Using them together. What?!? All my statements together make no sense. And I would agree.

My problems with them has come down to two different scenarios which have rather soured me on both and I don't know where to put the blame. I know much of the blame is on me but I don't know what I am doing wrong and reading the forums for both of these products I know I am not the only one. The two areas I attempted, unsuccessfully, to use these together are the following:

1) For passing data from the legacy web applications I attempted to use JMS to send the messages asynchronously to the Spring based application (running on Tomcat). Additionally for queuing up processing of various reports and tasks I sent JMS messages within the Spring based application to itself. I have done these thing previously several times in several projects and it has worked phenomenally well. Only I was not using Spring, and in one case was using JMS capabilities in JBoss. My problems with this attempt had to do with horrendous memory and connection leaks. I never had these problems using virtually identical code in other projects. There is even a bug that may be related to this (https://issues.apache.org/activemq/browse/AMQ-1790) that indicates it is resolved, but I am doubtful of that (didn't work for me) and at least one other poster suggests the same.

2) The JMS / JMSAppender logging setup that works so well everywhere else fails miserably withing the Spring environment. I get the "peer did not send his wire format" exception and there is even an open JIRA bug that has been languishing forever on JIRA for this as well (https://issues.apache.org/activemq/browse/AMQ-1473).

These two issues in all my years of programming have been the only two I have not been able to solve so far. I am sure there must be a solution to both, but it frustrates me that I have been unable to find it and I see others with the same basic issues with no resolution either. I don't even know where to point the blame. Is it more ActiveMQ or Spring, or is it just the combination of the two. For me, the only time I have had problems with either is if I am using them together.

I would welcome any constructive comments or suggestions. I am currently researching alternatives to ActiveMQ. Not sure what my best alternatives to Spring are, other than simply going back to traditional J2EE appserver platforms like JBoss.

Update November 17, 2012

Well this post was forever ago, but I thought I would update anyway. The JMS problems was due to a connection leak with Spring at the time. We have since switched to AMQP with RabbitMQ and it is a great improvement. It seems strange to say the problem was Spring, and not ActiveMQ, yet ActiveMQ is what we dumped, not Spring. We ending incorporating new tools outside of Java such as Erlang and JMS did not fit well for us as compared with AMQP. I think AMQP has a lot to credit JMS for what people learned in messaging (yeah, I know messaging concepts predated JMS but most of those earlier technologies were too expensive to license for your average small shop).

I continue to use Spring, but the Appfuse stuff has been dumped as it did not seem to be kept current. The main portion of Spring I am using is Spring Integration and it has been fantastic. A significant portion of our server setup uses Erlang and we communicate between different components using RabbitMQ and Spring Integration.

In place of Appfuse for a quick project startup I have been using Spring Roo, but I usually yank Roo out fairly quickly as it seems to get in the way much past the initial bootstrap. I am sure there are people who effectively use Roo for long term development but all the black magic seems to get in the way for me. It also makes me nervous if I don't completely understand all the mechanism, and some of the slick easy setup with Roo translates to abstraction that obscures what is really going on. The nice thing about Roo is it does get you up and running quickly, and they do make it very easy to remove at any point.

Log4j aggregator

I have created a prototype log4j aggregator for our web applications using Apache ActiveMQ and a little java glue code. It acts as both a server and a client and has command line options to optionally startup ActiveMQ as an embedded JMS instance.

Basically, all I am trying to do is be able to pipe log events from servers (production, qa, development) and get them to the people who need them in a real-time fashion. The obvious issues of security, scalability, etc. have been addressed (in a simple fashion so far), but I also need allow for better control, configuration of the selection, filtering, and monitoring of the events. I am also trying to investigate the assistance of IONA's FUSE HQ.

The current version is a standalone application which can be run as a daemon, but I am looking to follow this up with a web server deployable version.

I am considering using Camel or maybe even ServiceMix for an enhanced version, as well as maybe using IONA's FUSE equivalents for production.

This prototype has run well in development for some months, but is in need of enhancement for ease-of-use reasons. I am investigating getting the project hosted as an open source project, although there are probably existing projects that do this already, and are much more mature, better documented, tested, etc., although I have not found such a solution yet. 

Some random notes (in case someone has any thoughts/solutions/alternatives) on some of the issues I have run into:

Server source identification: I am sure I can probably use the NDC/MDC capabilities of log4j to somehow embed the originating server information into each log message, but my current solution of simply passing the server information in a JMS wrapper is working well enough. I will have to come back to this and learn more on log4j before I do a phase II of the project.

ActiveMQ: This has worked well except for known glitches with the 5.1.0 release in areas of idle connections, and a few others. Trying to identify how to best incorporate snapshot versions into my build process has not been successful. This is due to my still learning maven2 more than ActiveMQ faults.

Maven2: I have liked using maven2, but I still struggle with the learning curve of "The Maven Way". This book is phenomenal and is helping me a lot in this area and I will be buying the print version as soon as I can. Kudos to everyone behind this. My most recent hurdle is simply getting the use of snapshots on remote repositories working as I test out potential fixes in development versions with my code. When I reference this repository I get a file (e.g. commons-cli-2.0-20080529.153957-1.jar), but it references commons-cli-2.0-SNAPSHOT.jar in the packaged META-INF/MANIFEST.MF for the Class-Path. I hate being a n00b when I know it is such a simple problem.

Spring Framework: I am not actually using this in the project yet, but I have been trying to get up to speed on this for Phase II. I have found a great help from AppFuse in this area.

I would welcome any suggestions, experience, insights on any of this.

Update November 17, 2012

I am using RabbitMQ and Spring Integration, not Camel or JMS. We have done several variations on the messaging approach to logging, but have moved away from a pure log4j approach. Some of our stuff is using Logback, some Log4j, slf4j, Log4erl, etc., but is still somewhat a hodgepodge.

The messaging hub paradigm for logging has been extremely useful for us during development but I am unsatisfied with our overall solution for the production environment. Conceptually, slf4j should make the underlying implementation transparent, but due to slf4j binding version compatibility problems this didn't work seamlessly.

Changes I have made of some of the server components to allow dynamic changing of logging levels has helped a bit but the framework does not extend across all components. In short, we still have a ways to go.

Just started - nothing to say yet