Skip to main content

SOA and Batch - Let's Get Technical - Part 1

Two weeks ago, Zapthink published a fluff piece Batch Processing with SOA (by Ronald Schmelzer).

The article says just two simple things:

1 - You can make batch process controls into callable services.

2 - On-demand batches ("workload automation") are the future of batch processing.

The article ignores what are the key topics of a batch SOA conversation, namely:

A - Can web services be utlized in the context of a batch process (or workflow automation or whatever you want to call it)?

B - Can an ESB provide an enviroment for a batch process, or a significant portion thereof?

In other words, is there a convergence between batch processes and SOA integration tools (web services, ESB, or even governance)? Can these respective spaces leverage one another, or are they fundementally incompatible?

This issue has come up in the context of two clients within the past 2 weeks. The first question usually asked is WHY? Why would you consider an ESB in the context of batch processing?

There's a few answers to this question:

1. The first is one of the nominal goals of SOA, reuse. Making connections is a big part of service development. Depending on the environment and technologies involved, making a connection to a given system can be a major effort in and of itself. Security considerations, gateways, protocol translations and adapters required, just making contact with the right people and receiving the right approvals. The actual labor (programming or configuring) required isn't usually that time consuming, but finding the right people to get in touch with, going through the security approval processes, getting adapters installed or gateways configured, and coordinating it all, that is time consuming.

So, you've got a service means you have a valid working connection to a given system, with a valid working transaction - business function - service. Some batch process comes along that has a new requirement for your service, why not use it and save all that setup time? (And the related value of having one code source - function basis, real reuse.)

2. The second is ESB tools are very good at data transformations - in particular at the programming of data transformations. Visual drag-and-link development and integrated converter functions make the type of data format transformations often involved in batch processes a good fit. Further reinforcing this thought is that the ESB usually has a trained group of developers just waiting to create those transformations.

(As opposed to batch developers, which are usually doing this effort as part of their primary project - which is not exlusively batch work - and therefore are less likely to have such specialized tools or, if having them - be less skilled in their use. If they have tools, it's ETL tools [Extract Transform Load] which are very good at what they do BUT tend to be significantly more expensive than ESB tools, and because of their limited use they're harder to justify.)

These reasons, and perhaps just the general Integration Compentency Centers tend to be thought of when connecting things - regardless of the processing model - are dragging many an ESB and/or integration team into implementing batch processes in the context of the Enterprise Service Bus (ESB). This, however, is usually a mistake.

Why will be covered in my next article.


Popular posts from this blog

Integration Spaghetti™

I’ve been using the term Integration Spaghetti™ for the past 9 years or so to describe what happens as systems connectivity increases and increases to the point of … unmanageability, indeterminate impact, or just generally a big mess.  A standard line of mine is “moving from spaghetti code to spaghetti connections is not an improvement”.(A standard “point to point connection mess” slide, by enterprise architect Jerry Foster from 2001.)In the past few days I’ve been meeting with a series of IT managers at a large customer and have come up with a revised definition for Integration Spaghetti™ :Integration Spaghetti™ is when the connectivity to/from an application is so complex that everyone is afraid of touching it.  An application with such spaghetti becomes nearly impossible to replace.  Estimates of change impact to the application are frequently wrong by orders of magnitude.  Interruption in the integration functioning are always a major disaster – both in terms of the time and peopl…

Solving Integration Chaos - Past Approaches

A U.S. Fortune 50's systems interconnect map for 1 division, "core systems only".

Integration patterns began changing 15 years ago. Several early attempts were made to solve the increasing problem of the widening need for integration…

Enterprise Java Beans (J2EE / EJB's) attempted to make independent callable codelets. Coupling was too tight, the technology too platform specific.

Remote Method Invocation (Java / RMI) attempted to make anything independently callable, but again was too platform specific and a very tightly coupled protocol.

Similarly on the Microsoft side, DCOM & COM+ attempted to make anything independently and remotely callable. However, as with RMI the approach was extremely platform and vendor specific, and very tightly coupled.

MQ created a reliable independent messaging paradigm, but the cost and complexity of operation made it prohibitive for most projects and all but the largest of Enterprise IT shops which could devote a focused technology tea…

From Spaghetti Code to Spaghetti Connections

Twenty five years ago my boss handed me the primary billing program and described a series of new features needed. The program was about 4 years old and had been worked on by 5 different programmers. It had an original design model, but between all the modifications, bug fixes, patches and quick new features thrown in, the original design pattern was impossible to discern. Any pattern was impossible to discern. It had become, to quote what’s titled the most common architecture pattern of today, ‘a big ball of mud’.

After studying the program for several days, I informed my boss the program was untouchable. The effort to make anything more than a minor adjustment carried such a risk, as the impact could only be guessed at, that it was easier and less risky to rewrite it from scratch.
If they had considered the future impact, they never would have let a key program degenerate that way. They would have invested the extra effort to maintain it’s design, document it property, and consider t…