Making the Cloud, Microservices, and Enterprise IT work!

Loriane Lawson over at IT Business Edge has a penchant for touching upon particularly interesting integration problems.  This week she asks … It's interesting: Writing custom code for data integration is nearly universally frowned upon by experts – and yet, I'm seeing a lot of discussion about creating your own metadata solutions for support with integration. My question is: If you're getting away from hand-coding, why would you want to delve into customized metadata solutions? The article focuses upon some of the technical issues and technical approach to such a discussion.  I’d like to focus on the business issues, and how those should be directing the technical approach (but aren’t). Every application involved in integration brings along it’s database (or general data) model and it’s internal data object model.  These models were developed to meet the functional goals of the particular application.  For example, the “customer” representation in t...

This is a classic IT question.  Should one go with picking and choosing Best of Breed applications in the various niches that one’s IT shop needs, or go with an Application Suite?  SOA and Integration has some significant input to this question, and impact from this question.  And the same question applies not only to business toolsets, but also to SOA toolsets (best of breed ESB, design time governance, run time governance, SOA security tools, BPM, etc, or a suite?) With best of breed, we might end up with one company’s ERP system and another company’s CRM system, a third company’s manufacturing system and a fourth company’s financials. With historical integration patterns, the issue of data interchange between the systems was mostly handling by large scale data exports and imports, usually performed as batch processes at end of day (or end of week or end of month).  These processes could be described as “dump all the (insert primary data type here, such as c...

  I’ve been using the term Integration Spaghetti™ for the past 9 years or so to describe what happens as systems connectivity increases and increases to the point of … unmanageability, indeterminate impact, or just generally a big mess.  A standard line of mine is “moving from spaghetti code to spaghetti connections is not an improvement”. (A standard “point to point connection mess” slide, by enterprise architect Jerry Foster from 2001.) In the past few days I’ve been meeting with a series of IT managers at a large customer and have come up with a revised definition for Integration Spaghetti™ : Integration Spaghetti™ is when the connectivity to/from an application is so complex that everyone is afraid of touching it.  An application with such spaghetti becomes nearly impossible to replace.  Estimates of change impact to the application are frequently wrong by orders of magnitude.  Interruption in the integration functioning are always a major disast...

  A number of industry analysts have been speaking of SOA Design Time Governance failure for some time.  As I’ve written previously, primarily this was because the majority of enterprise IT shops hadn’t reached either the SOA maturity level to deal with it or had a large enough service catalog to have a need to address it with tools. I’ve seen a lot of change in this in the past year, as many organizations are suddenly asking for help in defining requirements for SOA governance tools. But what of the cutting edge IT shops , the early SOA adopters who ran into SOA governance needs years ago and started working with SOA Design Time Governance tools of earlier generations?  (I admit to being one of these, having led the purchase of a design time governance tool for my U.S. Fortune 50 employer at the time, about 7 years ago.) Most of these projects FAILED!  (Including the one I ran.)  The tools were complicated and somewhat rigid, the processes to make it s...

Addressing a service presents a few problems.  By putting the URL (or queue name if using messaging), you unintentionally couple between the service consumer and the physical instance of the service.  (Meaning what server it’s on, IP address, etc.) Applications often unintentionally become tightly coupled simply by addressing connections directly, by IP address, server name, or queue name. This is unacceptable, as any change in the physical layer results in software changes. (Hardcoding such information is clearly a major mistake, but even placing it in a configuration file or database entry still results in application manipulation due to physical layer changes.) Replace a server, redeploy all consumers of the services exposed on that server?  Ouch.  Even moving from development to test to production becomes a challenge (as you have to recompile or reconfigure as the consumer needs to repoint to the new provider instance in each environment). UDDI was origi...

A month ago I wrote how suddenly I’m seeing a surge of interest in SOA Governance.  In an interesting follow up to that article, I found myself sitting in a vendor sales presentation for a full SOA suite of tools to a potential customer.  And this customer asked a very good question… “We haven’t seen a lot of SOA suite sales in our region (and therefore haven’t seen many sales of your product). Why not?” While many of the Fortune 500 and leading edge companies approached SOA with significant investment and (somewhat) serious commitment, others did not. The innovators and early adopters have been “doing SOA” for 6-10 years.  Two things forced them into major vendor ESB products and SOA suites from the start.  First, these were the only tools (operating at an enterprise level) early on the market and they provided EAI and protocol + technology bridging that settled them into a solid enterprise IT niche.  Second, the extending of all the various develo...

Loraine Lawson over at the IT Business Edge Integration blog discusses Microsoft’s entry into the MDM space in her most recent post .  She notes that: “I was also intrigued to learn that the whole thing is API-based. Why does that matter? As Hayler explains, it allows ISVs to build apps on top of MDS. That's a good thing, since the description of MDS sounds pretty bare-bones at this point. The idea seems to be that ISVs will be able to build on better user interfaces and create support for things like version comparison, which, oddly, it does not provide out of the box.” I was surprised at this bit of thinking.  It’s always nice when a vendor product provides ways of extending it’s functionality.  It’s occasionally useful for enterprise IT shops, and often useful for niche vendor’s or ISV’s looking for opportunities to extend a major vendor’s product(s).  But that’s some old school thinking. Since the early days of COM (Microsoft’s Common Object Model),...

“What’s a Service?” my client asked.  Not what’s the technology of a service.  Rather this client is starting a real Service Oriented Architecture project.  Real meaning creating a new ‘application’ by creating a series of services that model granular business functions, and orchestrating those business services into business process workflows (both by BPM and by composite services) – layering on a User Interface and presto, it’s a composed application. But they asked “what’s a service?”  More specifically, what level of business functionality should become a service?  A surprisingly tough question to answer , and when starting with an existing environment the answer usually is “we work with whatever level the existing applications are exposing”.  Not as bad as it sounds as the existing “transactions” usually have developed over time to model that business sweet-spot – preventing the need for the architects to actually address this question. But when...

What should be a service?  Not what API should be exposed or what function of the application should be exposed.  At the higher levels of SOA maturity when we actually begin to architect pieces of business functionality as services, what should a service be?  From the analysis perspective, from the business perspective, and that ultimately becomes the actual IT modules and blocks of code. I had to give a presentation on that exact question.  It was surprisingly challenging, as almost every situation I’ve been in finds either organizations exposing application level functional API’s as services – and sometimes combining/composing those into wider services or enterprise services – or wrapping various blocks of IT functionality to offer more business flexibility (in theory from the IT perspective), as services. Suddenly someone was asking, “ok, assuming we’re going to build a real Service Oriented system from scratch, at what level of granularity do we plan the s...

Suddenly this year clients are calling me up about SOA Governance, service libraries, service monitoring and control.  First SOA was dead, then SOA governance was dead (even saw an article that SOA governance is more than just dead, it’s a murderer, killing the future by stopping cloud computing).  What’s going on with all my customer calls? Here’s a simple fact about services.  Until a significant percentage of frequently used business processes have been exposed, and exposed at a relatively granular (detailed) level (without being too granular that they require re-composition and orchestration every use), the major SOA ROI (return on investment) doesn’t happen.  Kind of like a real world physical library, it won’t have much traffic until either it has a large collection or a collection of popular material. Most SOA efforts begin Bottom Up, with programmers and projects beginning to use SOA technologies simply because they are available and enable getting...

  I was recently working with a client on SOA governance / service management.  They’ve build several hundred enterprise services and major integration points, and trying to manually manage them through a web site (or Excel spreadsheet or the brains of the integration department people) is just getting out of hand. This isn’t so unusual, it’s become the classic SOA design time governance problem encountered 2-3 years into the SOA maturity cycle.  While there are those claiming SOA Governance is dead, focused on Design Time Governance (I guess this would be a subset argument of the SOA is Dead punditry), the problem keeps cropping up in Enterprise IT. To be fair to the (Design Time) Governance is Dead argument, few organizations have successfully deployed SOA design time governance.  It’s extremely tricky to insert a tool that touches so many points of the Software Development Lifecycle (SDLC) into the process and not have it rejected or avoided.  It requi...

  The Datapower has an internal structure of components that can inherit or be reused, depending on their place in the inheritance chain.  Here’s the secret internal architecture of the DataPower:   And here’s the Datapower’s capabilities in a nutshell:   Multi-Protocol Gateway: (superset of XML Firewall) Transformations – Any-to-any transformation engine: MPGW can parse and transform arbitrary binary, flat text, and XML messages, including EDI, COBOL Copybook, ISO 8583, CSV, ASN.1, and ebXML. Transport Bridging – protocols such as HTTP, HTTPS, MQ, SSL, IMS Connect, FTP, and more Message-level Security - Messages can be filtered, validated, encrypted, and signed, helping to provide more secure enablement of high-value applications. Supported technologies include WS-Security, WS-Trust, SAML, and LDAP. Logging - logging and audit trail, including non-repudiation support Web Service Proxy: Schema Validation Policy Application SLA Monitori...

  Ok, this is a little more practical and technical than we usually get, but important nonetheless. The IBM Datapower, as well as similar devices from Layer 7 and Cisco (and others) provide SOA security, attack prevention, and a number of ESB-like abilities (somewhat of an ESB lite).  However, these boxes tend to be EXPENSIVE, as well as having a series of add-on software modules that raise the price significantly. The latter is a shock to many people.  One thinks of the Datapower as a physical device, like a network router.  While it is a physical device and much of it’s performance is based on optimized software placed in ASIC hardware chips, it’s also a very sophisticated software platform.  As such they’ve done the standard vendor thing of separating many of the sophisticated abilities as separate software add-on modules – adding on ability and adding on price.  For example (if I remember correctly), the ability of the Datapower to connect to a da...

  Validation.  Security headers, SAML, WS-Security, etc are useless unless they’re being validated against SOMETHING.  Usually this something is an LDAP, and in a Microsoft environment the Active Directory.  If you’ve got a security device such as a Datapower, it can be configured to perform this validation.  So can many SOA runtime governance and SOA security tools.  Use it! Exclusivity.  Sources of requests must be validated.  If there’s a Datapower (or similar) device in the picture, this means both providers and ESB’s only accept requests from the security device.  If not, security agents have to be configured to validate sources.  If no agents exist, there’s a hole to be exploited. Protocol Independence.  Security rules must remain true regardless of protocol…http, https, ftp, s/ftp, MQ, Tibco, etc. To act in this fashion, every message must contain the XML security header whether it’s sent by HTTP or another protocol suc...

Encryption Every web service call should be SSL encrypted (HTTP/S). Similarly, any web service operation or file transfer sent via FTP should utilize Secure-FTP (S/FTP). Basic level security requires that the receiving source have a valid signed certificate, but it does not require dual-side certificates or any validation of the certificate beyond it being a valid source on a valid root chain. The objective of this requirement is encryption, not authentication. Now encryption will frequently be argued against.  “It’s high overhead, slows things down too much.” and “Setting up security on every server is time consuming.” First, we must layer our security.  By exposing services we’re transmitting internal application data outside the security perimeter of the application!  This data must be protected from view, and dropping a network sniffer onto a developer’s workstation or a development or test server is a trivial exercise.  So even if there’s som...

The first and foremost feature of an IBM DataPower is as a security device.  However, most organizations turn their Datapower over to their security team and ignore it afterwards.  The security team(s) generally use it as a perimeter security device – as a firewall and filter for exposing SOA services out to the Internet (or via VPN connections, as who can trust a vendor’s network anymore).  It works in this capacity very well but is far more capable than just this narrow role.   With SOA breaking down the outer perimeter of our internal applications, security must now be layered and extended to EVERY exposed service or interface.  There’s two general approaches to providing this: The agent based model, where an agent is installed upon every server / application / application container and controls access to each service.  The other is an agentless model, where each web service is routed through a control point – in this case the Datapower, and the...

I had a very interesting meeting with a client a few weeks back. (For those who don't read regularly, I do high level IT consulting.) This client has been doing SOA for a few years. Actually though, they're doing SOI - Service Oriented Integration - not SOA, Service Oriented Architecture. Meaning they're creating and/or exposing lots of services from existing big box applications, and using web service technology to connect apps together. After a couple of years of SOI they have several hundred exposed services and application interconnections. Their architects have spotted that some services have much higher reuse patterns than others and are trying to apply architecture standards such as standard entities and interface patterns to every new service - to up the reuse, decrease the maintenance, and move towards SOA architecture goals and ROI. This is a very natural step along the SOA maturity path. Along comes a change in senior business management. They bring a new...

Cloud Computing is clearly well into the hype cycle. Never being one to miss some good hype, I've been paying attention to advise my clients on whether and when to pay attention to this rising option. One of the big factors in consideration is my current location... I'm working outside the U.S. My local country is heavily wired, offers high speed broadband (2-10mbit) to 100% of the country, 50mbit home connectivity in the major cities (100mbit next year), and even offers cell based mobile internet connectivity at 2mbit or 3mbit (competing companies). Company and office connectivity is typically equal or better. The in-country data centers and backbones between them are very high speed. Ping times from in-country web sites typically run 30ms across the various data centers, backbones and ISP's / hosting sites. All in all, compared to the US that's seriously high speed. Yet, all of that is in-country. The one area where the local internet infrastructure is weak is t...

When Anne Thomas Manes (of the Burton Group) famously declared in January of 2009 that "SOA is dead", everyone rushed around to understand what she meant. Being that a year later she's still giving presentations on SOA Governance and other SOA topics, clearly she didn't mean that SOA was a failed technology. (There are plenty of IT technologies that come along with much hype but never quite translate into practical usage patterns or benefits for Enterprise IT, and therefore fade away as quickly as they arrived.) Today when I'm talking with IT organizations the majority are doing some level of SOA. So clearly SOA has moved along the adoption curve. The innovators struggled with it but got and touted their early advantage. The early adopters picked it up and integrated it into their enterprise IT model. We're clearly past even the early majority and a good way into the late majority. The late majority are organizations that 2 years ago weren't consider...

Every IT system has an internal data and object model. This model is matched with a relational model that turns into the supporting database. When systems interface, bridging the data & object model from one system to another is a significant effort. Signifcant effort means up to 50% of the integration effort! The historical method is for the exposing system to slightly simplify and expose it’s model, and the receiving system to create significant manual code to transform from the exposed model to it’s internal model. Over the past ten years or so, a large number of industry IT consortiums have been working to create industry data+transaction standards. These standards are designed to be highly interoperable, covering each business process and data objects for the particular industry. They appear rather complex (they are rather complex) as they’re designed to cover all aspects of an industry object with necessary flexibility. However, literally years of thought have gone into co...

Twitter, in and of itself, is pretty stupid. Or to be a bit more analytical and precise, as a web version of a cell phone Short Message Service (aka SMS), it's incredibly limited functionality provides little room for practical value. (It actually started as a way to reflect a message from one cell phone out to a group of friends on their cell phones, via a web based facility.) By itself, Twitter is a very limited tool that would be permanently consigned to a narrow audience as a small utility function. You can easily think of 5-10 such services that you've tried and (probably) discarded, a few of which you keep using for their narrow purpose. So what differentiates Twitter from hundreds of other narrow utilities that came (and mostly went)? In a word, a Service Oriented Interface. Twitter started from day 1 exposing a simple straightforward Web Service interface. Even further, they never offered a feature without simultaneously exposing it. In other words, there is a Twi...