I’ve been using the term Integration Spaghetti™ for the past 9 years or so to describe what happens as systems connectivity increases and increases to the point of … unmanageability, indeterminate impact, or just generally a big mess. A standard line of mine is “moving from spaghetti code to spaghetti connections is not an improvement”.
(A standard “point to point connection mess” slide, by enterprise architect Jerry Foster from 2001.)
In the past few days I’ve been meeting with a series of IT managers at a large customer and have come up with a revised definition for Integration Spaghetti™ :
Integration Spaghetti™ is when the connectivity to/from an application is so complex that everyone is afraid of touching it. An application with such spaghetti becomes nearly impossible to replace. Estimates of change impact to the application are frequently wrong by orders of magnitude. Interruption in the integration functioning are always a major disaster – both in terms of the time and people required to resolve it and in the business impact of it.
Even as the spaghetti bound application is nearly impossible to replace, as the current state continues it continues to grow worse as additional connections are made to these key applications and derivative copies of the data are taken from it, or clones created to avoid it (and thereby creating another synchronization and connection point).
Such spaghetti takes multiple forms but often involves ALL forms with multiple generations of technology connections, including excessive point to point connections, tightly coupled connection technologies, database triggers, business logic embedded in EAI process steps, many batches in and out from and to many destinations, ETL loads and extracts to/from other databases, multiple services providing nearly (but not exactly) identical data sets, and the involvement of many message queues.
Anything is done to avoid dealing with the giant plate of spaghetti.
Systems will integrate with systems that integrate with it, piggybacking existing connectivity and putting a burden on the subsidiary system, to avoid directly connecting into the spaghetti. They’ll go to a secondary or tertiary data source to avoid going direct. Everyone knows avoid the spaghetti if at all possible and will spend double to triple the integration effort to do so.
If the primary system is replaced, it’s not unusual that the new system won’t be integrated into all the old connections – this would require actually understanding each existing connecting, extracting it and redirecting/reconnecting it to the new system – rather the OLD SYSTEM will stay around to act as the connection point for all the existing spaghetti connections and the new system will become an integration, taking data feeds or a regular ETL load, off the old system! Meaning the old system lives forever!
Does this problem every get resolved? Yes. When the other side of the connections gets replaced, the new systems on that side will be integrated with what replaced the core spaghetti bound system. If the IT shop is lucky after a generation or so the spaghetti bound system can be shut down.
Unfortunately in major Enterprise IT shops finding some spaghetti integrations is not unusual. IT management is loathe to acknowledge such a problem to the business and will continue work-arounds until it directly impacts business goals. Otherwise it remains just another hidden IT enterprise IT expense.