Saturday, June 7, 2008

Log4j aggregator

I have created a prototype log4j aggregator for our web applications using Apache ActiveMQ and a little java glue code. It acts as both a server and a client and has command line options to optionally startup ActiveMQ as an embedded JMS instance.

Basically, all I am trying to do is be able to pipe log events from servers (production, qa, development) and get them to the people who need them in a real-time fashion. The obvious issues of security, scalability, etc. have been addressed (in a simple fashion so far), but I also need allow for better control, configuration of the selection, filtering, and monitoring of the events. I am also trying to investigate the assistance of IONA's FUSE HQ.

The current version is a standalone application which can be run as a daemon, but I am looking to follow this up with a web server deployable version.

I am considering using Camel or maybe even ServiceMix for an enhanced version, as well as maybe using IONA's FUSE equivalents for production.

This prototype has run well in development for some months, but is in need of enhancement for ease-of-use reasons. I am investigating getting the project hosted as an open source project, although there are probably existing projects that do this already, and are much more mature, better documented, tested, etc., although I have not found such a solution yet. 

Some random notes (in case someone has any thoughts/solutions/alternatives) on some of the issues I have run into:

Server source identification: I am sure I can probably use the NDC/MDC capabilities of log4j to somehow embed the originating server information into each log message, but my current solution of simply passing the server information in a JMS wrapper is working well enough. I will have to come back to this and learn more on log4j before I do a phase II of the project.

ActiveMQ: This has worked well except for known glitches with the 5.1.0 release in areas of idle connections, and a few others. Trying to identify how to best incorporate snapshot versions into my build process has not been successful. This is due to my still learning maven2 more than ActiveMQ faults.

Maven2: I have liked using maven2, but I still struggle with the learning curve of "The Maven Way". This book is phenomenal and is helping me a lot in this area and I will be buying the print version as soon as I can. Kudos to everyone behind this. My most recent hurdle is simply getting the use of snapshots on remote repositories working as I test out potential fixes in development versions with my code. When I reference this repository I get a file (e.g. commons-cli-2.0-20080529.153957-1.jar), but it references commons-cli-2.0-SNAPSHOT.jar in the packaged META-INF/MANIFEST.MF for the Class-Path. I hate being a n00b when I know it is such a simple problem.

Spring Framework: I am not actually using this in the project yet, but I have been trying to get up to speed on this for Phase II. I have found a great help from AppFuse in this area.

I would welcome any suggestions, experience, insights on any of this.

Update November 17, 2012

I am using RabbitMQ and Spring Integration, not Camel or JMS. We have done several variations on the messaging approach to logging, but have moved away from a pure log4j approach. Some of our stuff is using Logback, some Log4j, slf4j, Log4erl, etc., but is still somewhat a hodgepodge.

The messaging hub paradigm for logging has been extremely useful for us during development but I am unsatisfied with our overall solution for the production environment. Conceptually, slf4j should make the underlying implementation transparent, but due to slf4j binding version compatibility problems this didn't work seamlessly.

Changes I have made of some of the server components to allow dynamic changing of logging levels has helped a bit but the framework does not extend across all components. In short, we still have a ways to go.

No comments: