Making the Cloud, Microservices, and Enterprise IT work!

  BI, Business Intelligence, has taken hold at almost every mid-size or larger IT organization.  It commonly means extracting key data elements from all the main systems and databases in the organization and compiling it all together in the Business Intelligence Data Warehouse.  And the primary method for doing this is ETL – extract, transform, and load.  Basically meaning batch-style data loads performed daily, weekly, or monthly (from the source systems to the data warehouse). Setting it up is expensive and time consuming as it requires building a large capacity database and ETL processes for every important data source in the company.  The ETL processes by themselves are often not enough as data duplication and data quality problems quickly float to the surface and have to be resolved to a sufficient level to continue (resolved in the data warehouse, not in the data sources). However, it’s relatively easy to demonstrate the business value of the res...

Jason Bloomberg over at Zapthink has a devastating indictment of vendors selling “Private Clouds” and “Platform as a Service”. If you’re headed towards cloud computing (and we all are, it’s just a question of how long over the next decade till you get there), read it.

Web service security is a tricky business.  EVERY service exposed by any service provider, be it .Net, Java, the Mainframe, or any other provider needs to be secured.  Certainly if it’s exposing sensitive data (say customer data), allowing activation of a business process, and most especially if it’s involving a financial transaction. But how do you do it?  While every vendor and (almost) every technology announces compatibility with every web service security buzzword (WS-Security, SAML, X.509, etc.), they don’t describe how to actually make use of all this security data attached to the web service request. I’ve had recent discussions with IBM, Oracle, and Software AG (as leading SOA middleware tool providers) on this exact topic and the results are disappointing. The architecture model for this says that to provide SOA security I should use the tools as a SOA security layer, allowing my services to go about their business and the security tools to grab and pr...

SOA security is getting more attention.  Recent document leaks show the need of inside-the-network security, and recent attacks on service providers by Wikileaks supporting hacker-nets show the need for AT LEAST “decent” SOA security.  As ZapThink wrote recently … If you’ve been even peripherally paying attention to the news lately, then you know about Wikileaks and the battle going on between supporters and opponents of the group. While you might think that these incidents have nothing to do with you, enterprise architect, IT manager, developer, vendor, or consultant at a private company or government institution, they most certainly do. You are about to become a part of the frontline in the Cyberwar battle… Just as firms plan for server outages and deployment problems, now every firm must have a contingency plan to deal with the potential outage of their most important online suppliers… Your company uses Google for mail, a specialty Software-as-a-Service (Saa...

Is BPM (Business Process Modeling and Execution) the SOA killer application? I had a very interesting interaction with a global VP of BPM of BPM for a major vendor.  The encounter went like this… VP on behalf of vendor demonstrates BPM tools to potential client.  Everyone oohs and ahhs at the impressive presentation, graphics, and overall possibilities of BPM.  The vendor product looks impressive, it works, and actually has reasonable (runtime) performance. Then I ask the VP “how many processes can this client create in a year?”.  After all, creating a BPM workflow / process is relatively quick and easy.  In theory a couple of days to create it, another week to test it and deploy it.  Maybe a mid-sized organization should be able to generate 26 processes per year per person assigned to BPM. The VP responded, “7”.  “Seven”, I remarked, “why so few?”  He explained first of all the BPM processes won’t have all their steps exposed (at all o...

I sat with my client on a major service orientation project as IBM presented their updates to CICS.  My client is a few versions behind (could actually find no compelling reason to bother to upgrade) on their mainframe and wanted to review if IBM’s added capabilities would offer any abilities the project could utilize. IBM’s CICS support for web services is well known by now.  Their CICS updates keep the web service support up to the latest WSDL – SOAP – and WS-* standards.  If you need or want to expose COBOL programs as a web service (and this can be valuable to allow older applications to be partially utilized as transaction engines in their area of expertise) IBM’s keeping the capabilities in line. IBM has also continued to expand a large variety of internal CICS capabilities.  This was a major time warp for me as CICS command line functions, com areas and linkage sections, and various transaction batch and script coding abilities are so dated.  As an...

Loriane Lawson over at IT Business Edge has a penchant for touching upon particularly interesting integration problems.  This week she asks … It's interesting: Writing custom code for data integration is nearly universally frowned upon by experts – and yet, I'm seeing a lot of discussion about creating your own metadata solutions for support with integration. My question is: If you're getting away from hand-coding, why would you want to delve into customized metadata solutions? The article focuses upon some of the technical issues and technical approach to such a discussion.  I’d like to focus on the business issues, and how those should be directing the technical approach (but aren’t). Every application involved in integration brings along it’s database (or general data) model and it’s internal data object model.  These models were developed to meet the functional goals of the particular application.  For example, the “customer” representation in t...

This is a classic IT question.  Should one go with picking and choosing Best of Breed applications in the various niches that one’s IT shop needs, or go with an Application Suite?  SOA and Integration has some significant input to this question, and impact from this question.  And the same question applies not only to business toolsets, but also to SOA toolsets (best of breed ESB, design time governance, run time governance, SOA security tools, BPM, etc, or a suite?) With best of breed, we might end up with one company’s ERP system and another company’s CRM system, a third company’s manufacturing system and a fourth company’s financials. With historical integration patterns, the issue of data interchange between the systems was mostly handling by large scale data exports and imports, usually performed as batch processes at end of day (or end of week or end of month).  These processes could be described as “dump all the (insert primary data type here, such as c...

  I’ve been using the term Integration Spaghetti™ for the past 9 years or so to describe what happens as systems connectivity increases and increases to the point of … unmanageability, indeterminate impact, or just generally a big mess.  A standard line of mine is “moving from spaghetti code to spaghetti connections is not an improvement”. (A standard “point to point connection mess” slide, by enterprise architect Jerry Foster from 2001.) In the past few days I’ve been meeting with a series of IT managers at a large customer and have come up with a revised definition for Integration Spaghetti™ : Integration Spaghetti™ is when the connectivity to/from an application is so complex that everyone is afraid of touching it.  An application with such spaghetti becomes nearly impossible to replace.  Estimates of change impact to the application are frequently wrong by orders of magnitude.  Interruption in the integration functioning are always a major disast...

  A number of industry analysts have been speaking of SOA Design Time Governance failure for some time.  As I’ve written previously, primarily this was because the majority of enterprise IT shops hadn’t reached either the SOA maturity level to deal with it or had a large enough service catalog to have a need to address it with tools. I’ve seen a lot of change in this in the past year, as many organizations are suddenly asking for help in defining requirements for SOA governance tools. But what of the cutting edge IT shops , the early SOA adopters who ran into SOA governance needs years ago and started working with SOA Design Time Governance tools of earlier generations?  (I admit to being one of these, having led the purchase of a design time governance tool for my U.S. Fortune 50 employer at the time, about 7 years ago.) Most of these projects FAILED!  (Including the one I ran.)  The tools were complicated and somewhat rigid, the processes to make it s...

Addressing a service presents a few problems.  By putting the URL (or queue name if using messaging), you unintentionally couple between the service consumer and the physical instance of the service.  (Meaning what server it’s on, IP address, etc.) Applications often unintentionally become tightly coupled simply by addressing connections directly, by IP address, server name, or queue name. This is unacceptable, as any change in the physical layer results in software changes. (Hardcoding such information is clearly a major mistake, but even placing it in a configuration file or database entry still results in application manipulation due to physical layer changes.) Replace a server, redeploy all consumers of the services exposed on that server?  Ouch.  Even moving from development to test to production becomes a challenge (as you have to recompile or reconfigure as the consumer needs to repoint to the new provider instance in each environment). UDDI was origi...

A month ago I wrote how suddenly I’m seeing a surge of interest in SOA Governance.  In an interesting follow up to that article, I found myself sitting in a vendor sales presentation for a full SOA suite of tools to a potential customer.  And this customer asked a very good question… “We haven’t seen a lot of SOA suite sales in our region (and therefore haven’t seen many sales of your product). Why not?” While many of the Fortune 500 and leading edge companies approached SOA with significant investment and (somewhat) serious commitment, others did not. The innovators and early adopters have been “doing SOA” for 6-10 years.  Two things forced them into major vendor ESB products and SOA suites from the start.  First, these were the only tools (operating at an enterprise level) early on the market and they provided EAI and protocol + technology bridging that settled them into a solid enterprise IT niche.  Second, the extending of all the various develo...

Loraine Lawson over at the IT Business Edge Integration blog discusses Microsoft’s entry into the MDM space in her most recent post .  She notes that: “I was also intrigued to learn that the whole thing is API-based. Why does that matter? As Hayler explains, it allows ISVs to build apps on top of MDS. That's a good thing, since the description of MDS sounds pretty bare-bones at this point. The idea seems to be that ISVs will be able to build on better user interfaces and create support for things like version comparison, which, oddly, it does not provide out of the box.” I was surprised at this bit of thinking.  It’s always nice when a vendor product provides ways of extending it’s functionality.  It’s occasionally useful for enterprise IT shops, and often useful for niche vendor’s or ISV’s looking for opportunities to extend a major vendor’s product(s).  But that’s some old school thinking. Since the early days of COM (Microsoft’s Common Object Model),...

“What’s a Service?” my client asked.  Not what’s the technology of a service.  Rather this client is starting a real Service Oriented Architecture project.  Real meaning creating a new ‘application’ by creating a series of services that model granular business functions, and orchestrating those business services into business process workflows (both by BPM and by composite services) – layering on a User Interface and presto, it’s a composed application. But they asked “what’s a service?”  More specifically, what level of business functionality should become a service?  A surprisingly tough question to answer , and when starting with an existing environment the answer usually is “we work with whatever level the existing applications are exposing”.  Not as bad as it sounds as the existing “transactions” usually have developed over time to model that business sweet-spot – preventing the need for the architects to actually address this question. But when...

What should be a service?  Not what API should be exposed or what function of the application should be exposed.  At the higher levels of SOA maturity when we actually begin to architect pieces of business functionality as services, what should a service be?  From the analysis perspective, from the business perspective, and that ultimately becomes the actual IT modules and blocks of code. I had to give a presentation on that exact question.  It was surprisingly challenging, as almost every situation I’ve been in finds either organizations exposing application level functional API’s as services – and sometimes combining/composing those into wider services or enterprise services – or wrapping various blocks of IT functionality to offer more business flexibility (in theory from the IT perspective), as services. Suddenly someone was asking, “ok, assuming we’re going to build a real Service Oriented system from scratch, at what level of granularity do we plan the s...

Suddenly this year clients are calling me up about SOA Governance, service libraries, service monitoring and control.  First SOA was dead, then SOA governance was dead (even saw an article that SOA governance is more than just dead, it’s a murderer, killing the future by stopping cloud computing).  What’s going on with all my customer calls? Here’s a simple fact about services.  Until a significant percentage of frequently used business processes have been exposed, and exposed at a relatively granular (detailed) level (without being too granular that they require re-composition and orchestration every use), the major SOA ROI (return on investment) doesn’t happen.  Kind of like a real world physical library, it won’t have much traffic until either it has a large collection or a collection of popular material. Most SOA efforts begin Bottom Up, with programmers and projects beginning to use SOA technologies simply because they are available and enable getting...

  I was recently working with a client on SOA governance / service management.  They’ve build several hundred enterprise services and major integration points, and trying to manually manage them through a web site (or Excel spreadsheet or the brains of the integration department people) is just getting out of hand. This isn’t so unusual, it’s become the classic SOA design time governance problem encountered 2-3 years into the SOA maturity cycle.  While there are those claiming SOA Governance is dead, focused on Design Time Governance (I guess this would be a subset argument of the SOA is Dead punditry), the problem keeps cropping up in Enterprise IT. To be fair to the (Design Time) Governance is Dead argument, few organizations have successfully deployed SOA design time governance.  It’s extremely tricky to insert a tool that touches so many points of the Software Development Lifecycle (SDLC) into the process and not have it rejected or avoided.  It requi...

  The Datapower has an internal structure of components that can inherit or be reused, depending on their place in the inheritance chain.  Here’s the secret internal architecture of the DataPower:   And here’s the Datapower’s capabilities in a nutshell:   Multi-Protocol Gateway: (superset of XML Firewall) Transformations – Any-to-any transformation engine: MPGW can parse and transform arbitrary binary, flat text, and XML messages, including EDI, COBOL Copybook, ISO 8583, CSV, ASN.1, and ebXML. Transport Bridging – protocols such as HTTP, HTTPS, MQ, SSL, IMS Connect, FTP, and more Message-level Security - Messages can be filtered, validated, encrypted, and signed, helping to provide more secure enablement of high-value applications. Supported technologies include WS-Security, WS-Trust, SAML, and LDAP. Logging - logging and audit trail, including non-repudiation support Web Service Proxy: Schema Validation Policy Application SLA Monitori...

  Ok, this is a little more practical and technical than we usually get, but important nonetheless. The IBM Datapower, as well as similar devices from Layer 7 and Cisco (and others) provide SOA security, attack prevention, and a number of ESB-like abilities (somewhat of an ESB lite).  However, these boxes tend to be EXPENSIVE, as well as having a series of add-on software modules that raise the price significantly. The latter is a shock to many people.  One thinks of the Datapower as a physical device, like a network router.  While it is a physical device and much of it’s performance is based on optimized software placed in ASIC hardware chips, it’s also a very sophisticated software platform.  As such they’ve done the standard vendor thing of separating many of the sophisticated abilities as separate software add-on modules – adding on ability and adding on price.  For example (if I remember correctly), the ability of the Datapower to connect to a da...

  Validation.  Security headers, SAML, WS-Security, etc are useless unless they’re being validated against SOMETHING.  Usually this something is an LDAP, and in a Microsoft environment the Active Directory.  If you’ve got a security device such as a Datapower, it can be configured to perform this validation.  So can many SOA runtime governance and SOA security tools.  Use it! Exclusivity.  Sources of requests must be validated.  If there’s a Datapower (or similar) device in the picture, this means both providers and ESB’s only accept requests from the security device.  If not, security agents have to be configured to validate sources.  If no agents exist, there’s a hole to be exploited. Protocol Independence.  Security rules must remain true regardless of protocol…http, https, ftp, s/ftp, MQ, Tibco, etc. To act in this fashion, every message must contain the XML security header whether it’s sent by HTTP or another protocol suc...

Encryption Every web service call should be SSL encrypted (HTTP/S). Similarly, any web service operation or file transfer sent via FTP should utilize Secure-FTP (S/FTP). Basic level security requires that the receiving source have a valid signed certificate, but it does not require dual-side certificates or any validation of the certificate beyond it being a valid source on a valid root chain. The objective of this requirement is encryption, not authentication. Now encryption will frequently be argued against.  “It’s high overhead, slows things down too much.” and “Setting up security on every server is time consuming.” First, we must layer our security.  By exposing services we’re transmitting internal application data outside the security perimeter of the application!  This data must be protected from view, and dropping a network sniffer onto a developer’s workstation or a development or test server is a trivial exercise.  So even if there’s som...

The first and foremost feature of an IBM DataPower is as a security device.  However, most organizations turn their Datapower over to their security team and ignore it afterwards.  The security team(s) generally use it as a perimeter security device – as a firewall and filter for exposing SOA services out to the Internet (or via VPN connections, as who can trust a vendor’s network anymore).  It works in this capacity very well but is far more capable than just this narrow role.   With SOA breaking down the outer perimeter of our internal applications, security must now be layered and extended to EVERY exposed service or interface.  There’s two general approaches to providing this: The agent based model, where an agent is installed upon every server / application / application container and controls access to each service.  The other is an agentless model, where each web service is routed through a control point – in this case the Datapower, and the...