SOA as Interface Simplification

Integration is tough. Traditional IT applications are spending as much as 40 percent of their budget on integration. As the environment complexity increases as well as the number of connections per system, that number may increase to 60 percent.

Why?

At the basic level every system has it’s internal data model and logical model. Every interface has to bridge and convert those models (for the interfaced elements) across two systems. Then there’s the practical aspects – matching connectivity technologies (or bridging them), matching security patterns, simply determining appropriate error handling, human contacts, etc.

SOA has standardized the interface technologies and provided a wealth of tools to bridge the issues where standards don’t match.

Most organizations are using these tools today, whether intentionally or because the programmers are using recent development tools that use SOA interface technologies by default (the more likely situation).

Interfacing significantly easier naturally leads to more interfacing per application or per software development problem. The result: simplified interfacing but complicated inter-application dependencies and the growth of the accidental integration architecture a.k.a. interface spaghetti.

Solving Integration Chaos - Past Approaches

A U.S. Fortune 50's systems interconnect map for 1 division, "core systems only".

Integration patterns began changing 15 years ago. Several early attempts were made to solve the increasing problem of the widening need for integration…

Enterprise Java Beans (J2EE / EJB's) attempted to make independent callable codelets. Coupling was too tight, the technology too platform specific.

Remote Method Invocation (Java / RMI) attempted to make anything independently callable, but again was too platform specific and a very tightly coupled protocol.

Similarly on the Microsoft side, DCOM & COM+ attempted to make anything independently and remotely callable. However, as with RMI the approach was extremely platform and vendor specific, and very tightly coupled.

MQ created a reliable independent messaging paradigm, but the cost and complexity of operation made it prohibitive for most projects and all but the largest of Enterprise IT shops which could devote a focused technology team (until recently).

Prior to SOAP (Simple Object Access Protocol) and SOA (Service Oriented Architecture) one could carefully manage the integration chaos but one couldn’t solve it. And frankly, few IT shops carefully managed it.

The picture above is a real picture. Each line represents a major transaction connection between "core" business systems at a very large IT enterprise. While each individual connection was created in response to a business need as interpreted by the IT systems creators, the overall result speaks for itself.

From Spaghetti Code to Spaghetti Connections

Twenty five years ago my boss handed me the primary billing program and described a series of new features needed. The program was about 4 years old and had been worked on by 5 different programmers. It had an original design model, but between all the modifications, bug fixes, patches and quick new features thrown in, the original design pattern was impossible to discern. Any pattern was impossible to discern. It had become, to quote what’s titled the most common architecture pattern of today, ‘a big ball of mud’.

After studying the program for several days, I informed my boss the program was untouchable. The effort to make anything more than a minor adjustment carried such a risk, as the impact could only be guessed at, that it was easier and less risky to rewrite it from scratch.
If they had considered the future impact, they never would have let a key program degenerate that way. They would have invested the extra effort to maintain it’s design, document it property, and consider the impact of quick changes.

Nowadays a similar pattern is happening in many IT shops with integration. It starts with “just” connecting a system or two. Technology, methodology, and architecture is not considered – after all it’s just a connection or two. After a few years, that IT shop is faced with tens to hundreds of connections via all kinds of different connection technologies (and gateways), a wide variety of data formats, limited or no reuse of connections, and a systems integration infrastructure that looks like a plate of spaghetti!

Minor changes to one system begin impacting multiple other systems. Slowdowns of one database cause impact throughout the environment, even impacting systems seemingly with no relation to the database involved.

The accidental integration architecture begins to rear its ugly head.

Figure 1 represents a key business system at a US Fortune 50 IT organization that grew, in an unplanned fashion, into a highly integrated system. Systems stability degraded tremendously, and system agility became an impossibility.

Accidental Enterprise Integration

Sometime in the last 15 years the way systems were developed began to change. Or rather, the purpose of the systems began to change. The big box system, that massive application that “did everything”, completely covering the functionality of the company (such as ERP systems or CRM systems) or the functionality of the large business department (such as Billing), began to decompose.

Business people started demanding a level of detailed functionality in what were previously niche areas of the big system. The niches became complete systems of their own. Whether a logical break off of the big box system or independent smaller systems of their own, suddenly supporting business functionality was outside the box. And real integration began.

Conceptually integration is nothing new. The billing system has been passing data to the accounting system (for example) for the past 40 years. What has changed is the type of integration and the significance of the integration.

Previously (as an example) the billing system would package up the financial data after a billing run and send it over to the accounting system for appropriate corporate financial updates. The billing system wasn’t dependent on sending the data, and the accounting system would continue to operate just fine (if somewhat out of date) without receiving the update. They remained independent systems, neither directly affected or reliant upon the other.

However (as an example) when the customer management system was developed and the business responsibility for maintaining and managing the customer data moved out of the billing system, the billing system could no longer operate without a direct customer feed from the customer management system. Systems became directly dependent upon each other (aka tightly coupled) and the importance of integration moved from a minor component of the IT environment to a key operational factor.

This has been a gradual shift with many an IT shop not realizing the true impact. They have developed an accidental enteprise integration and an associated accidental integration architecture.

We'll explore what this means in the next article.

A Big Ball of Mud

The #1 software architecture pattern: A Big Ball of Mud.

If applications architecture degenerates over time into a Big Ball of Mud, does unmanaged integration between those applications degenerate into a Untanglable Ball of Yarn?