Spring Boot, Apache Camel, Actuator and Hawt.io - A Winning combination

I was recently asked how I would do the following:
  • Watch for a zip file and manifest file being copied into a directory
  • Check contents of the zip file against the manifest
  • Copy the contents of the zip to a remote system using an interface (tbd)
  • Must be able to scale to high(ish) volumes 
Sounds simple....right?

Having encountered numerous problems like this before, and having the war wounds to prove it, I recalled all of the nightmare scenarios in a series of flash backs:
  • Raw reading of files, possibly by multiple readers, and the file locking problems that ensue
  • Programmatically extracting zip files usually involves quite a bit of laborious and low-level code - whilst checking against the manifest
  • Implementing manually will either end up with brittle bespoke code is difficult to change; or lots of abstractions with lots more code than should be required for such a simple set of operations
Time to get out my 'go-to' tool for this kind of task - Apache Camel.  After downloading my template project from GitHub, I was ready to go.  Implementing was easy; four 'routes' and four small classes later I was done (GitHub).  It really was only a few lines of code.

The routes are here and hopefully readable even if you don't know Camel.
  1. MainRoute - High-level process
    1.  Watch for meta-file and process
    2.  Process zip file
    3.  Check if there are any files in the manifest not found in the zip
    4.  Done
  2. ProcessMetaFile
    1.  Read each line and place the filename in an array for checking later
  3. ProcessZipFile
    1.  Extract each file from the zip
    2.  Check if it was in the manifest
    3.  If not -Use the 'UnknownFile' route
    4.  If it was - process the file
  4. UnknownFile - File in zip not expected
    1.  Log the details
    2.  Copy file to the error directory

I realised while writing the application that my base Spring Boot/Camel application is quite a bit out of date, it was written when Spring Boot had just been released.

Time to update my base project :)

I have used Spring Boot on a number of occasions recently so wanted to include the following:
  • Latest version of Spring Boot
  • Spring Actuator - for metrics
I had also played with a fledgling version of Hawt.io a couple of years ago and wanted to see how far it had come - I was not disappointed.

After hunting through a selection of samples, Q&A and blogs entries I was able to integrate all the above in a single stand-alone application, which is available in GitHub.  So what do we need?
  1.  Spring Boot
    1. Dependency injection
    2. Inversion of Control
    3. Actuator - RESTful metrics endpoints
    4. Auto configuration of Apache Camel using starter dependency
      1. No boiler plate required - just define the routes
  2. Apache Camel
    1. Great implementation of the Enterprise Integration patterns
  3. Jolokia
    1. Converts JMX to RESTful json - required for Hawt.io
  4. Hawt.io
    1. Dynamic monitoring of your application; and more specifically the Camel Routes.
Final result

Using this base application you can quickly build integration applications that use the powerful components and adapters included with Apache Camel.  Added to that, Actuator provides in-depth metrics via REST endpoints.

To top it all, dynamic monitoring of your Camel routes  as they process data, among other things, via the Hawt.io HTML 5 user interface - directly from your application.


Real world applications

For a production application you will probably want to remove Hawt.io from the same application.  It can just as easily connect from a remote host to provide the same monitoring without placing the resource burden on the processing application.

Hawt.io JVM monitoring
Hawt.io monitoring Apache Camel routes



Popular Posts