Making the Cloud, Microservices, and Enterprise IT work!

Web service security is a tricky business.  EVERY service exposed by any service provider, be it .Net, Java, the Mainframe, or any other provider needs to be secured.  Certainly if it’s exposing sensitive data (say customer data), allowing activation of a business process, and most especially if it’s involving a financial transaction. But how do you do it?  While every vendor and (almost) every technology announces compatibility with every web service security buzzword (WS-Security, SAML, X.509, etc.), they don’t describe how to actually make use of all this security data attached to the web service request. I’ve had recent discussions with IBM, Oracle, and Software AG (as leading SOA middleware tool providers) on this exact topic and the results are disappointing. The architecture model for this says that to provide SOA security I should use the tools as a SOA security layer, allowing my services to go about their business and the security tools to grab and pr...

Is BPM (Business Process Modeling and Execution) the SOA killer application? I had a very interesting interaction with a global VP of BPM of BPM for a major vendor.  The encounter went like this… VP on behalf of vendor demonstrates BPM tools to potential client.  Everyone oohs and ahhs at the impressive presentation, graphics, and overall possibilities of BPM.  The vendor product looks impressive, it works, and actually has reasonable (runtime) performance. Then I ask the VP “how many processes can this client create in a year?”.  After all, creating a BPM workflow / process is relatively quick and easy.  In theory a couple of days to create it, another week to test it and deploy it.  Maybe a mid-sized organization should be able to generate 26 processes per year per person assigned to BPM. The VP responded, “7”.  “Seven”, I remarked, “why so few?”  He explained first of all the BPM processes won’t have all their steps exposed (at all o...

I sat with my client on a major service orientation project as IBM presented their updates to CICS.  My client is a few versions behind (could actually find no compelling reason to bother to upgrade) on their mainframe and wanted to review if IBM’s added capabilities would offer any abilities the project could utilize. IBM’s CICS support for web services is well known by now.  Their CICS updates keep the web service support up to the latest WSDL – SOAP – and WS-* standards.  If you need or want to expose COBOL programs as a web service (and this can be valuable to allow older applications to be partially utilized as transaction engines in their area of expertise) IBM’s keeping the capabilities in line. IBM has also continued to expand a large variety of internal CICS capabilities.  This was a major time warp for me as CICS command line functions, com areas and linkage sections, and various transaction batch and script coding abilities are so dated.  As an...

Loraine Lawson over at the IT Business Edge Integration blog discusses Microsoft’s entry into the MDM space in her most recent post .  She notes that: “I was also intrigued to learn that the whole thing is API-based. Why does that matter? As Hayler explains, it allows ISVs to build apps on top of MDS. That's a good thing, since the description of MDS sounds pretty bare-bones at this point. The idea seems to be that ISVs will be able to build on better user interfaces and create support for things like version comparison, which, oddly, it does not provide out of the box.” I was surprised at this bit of thinking.  It’s always nice when a vendor product provides ways of extending it’s functionality.  It’s occasionally useful for enterprise IT shops, and often useful for niche vendor’s or ISV’s looking for opportunities to extend a major vendor’s product(s).  But that’s some old school thinking. Since the early days of COM (Microsoft’s Common Object Model),...

Suddenly this year clients are calling me up about SOA Governance, service libraries, service monitoring and control.  First SOA was dead, then SOA governance was dead (even saw an article that SOA governance is more than just dead, it’s a murderer, killing the future by stopping cloud computing).  What’s going on with all my customer calls? Here’s a simple fact about services.  Until a significant percentage of frequently used business processes have been exposed, and exposed at a relatively granular (detailed) level (without being too granular that they require re-composition and orchestration every use), the major SOA ROI (return on investment) doesn’t happen.  Kind of like a real world physical library, it won’t have much traffic until either it has a large collection or a collection of popular material. Most SOA efforts begin Bottom Up, with programmers and projects beginning to use SOA technologies simply because they are available and enable getting...

Encryption Every web service call should be SSL encrypted (HTTP/S). Similarly, any web service operation or file transfer sent via FTP should utilize Secure-FTP (S/FTP). Basic level security requires that the receiving source have a valid signed certificate, but it does not require dual-side certificates or any validation of the certificate beyond it being a valid source on a valid root chain. The objective of this requirement is encryption, not authentication. Now encryption will frequently be argued against.  “It’s high overhead, slows things down too much.” and “Setting up security on every server is time consuming.” First, we must layer our security.  By exposing services we’re transmitting internal application data outside the security perimeter of the application!  This data must be protected from view, and dropping a network sniffer onto a developer’s workstation or a development or test server is a trivial exercise.  So even if there’s som...

The first and foremost feature of an IBM DataPower is as a security device.  However, most organizations turn their Datapower over to their security team and ignore it afterwards.  The security team(s) generally use it as a perimeter security device – as a firewall and filter for exposing SOA services out to the Internet (or via VPN connections, as who can trust a vendor’s network anymore).  It works in this capacity very well but is far more capable than just this narrow role.   With SOA breaking down the outer perimeter of our internal applications, security must now be layered and extended to EVERY exposed service or interface.  There’s two general approaches to providing this: The agent based model, where an agent is installed upon every server / application / application container and controls access to each service.  The other is an agentless model, where each web service is routed through a control point – in this case the Datapower, and the...

I had a very interesting meeting with a client a few weeks back. (For those who don't read regularly, I do high level IT consulting.) This client has been doing SOA for a few years. Actually though, they're doing SOI - Service Oriented Integration - not SOA, Service Oriented Architecture. Meaning they're creating and/or exposing lots of services from existing big box applications, and using web service technology to connect apps together. After a couple of years of SOI they have several hundred exposed services and application interconnections. Their architects have spotted that some services have much higher reuse patterns than others and are trying to apply architecture standards such as standard entities and interface patterns to every new service - to up the reuse, decrease the maintenance, and move towards SOA architecture goals and ROI. This is a very natural step along the SOA maturity path. Along comes a change in senior business management. They bring a new...

When Anne Thomas Manes (of the Burton Group) famously declared in January of 2009 that "SOA is dead", everyone rushed around to understand what she meant. Being that a year later she's still giving presentations on SOA Governance and other SOA topics, clearly she didn't mean that SOA was a failed technology. (There are plenty of IT technologies that come along with much hype but never quite translate into practical usage patterns or benefits for Enterprise IT, and therefore fade away as quickly as they arrived.) Today when I'm talking with IT organizations the majority are doing some level of SOA. So clearly SOA has moved along the adoption curve. The innovators struggled with it but got and touted their early advantage. The early adopters picked it up and integrated it into their enterprise IT model. We're clearly past even the early majority and a good way into the late majority. The late majority are organizations that 2 years ago weren't consider...

Twitter, in and of itself, is pretty stupid. Or to be a bit more analytical and precise, as a web version of a cell phone Short Message Service (aka SMS), it's incredibly limited functionality provides little room for practical value. (It actually started as a way to reflect a message from one cell phone out to a group of friends on their cell phones, via a web based facility.) By itself, Twitter is a very limited tool that would be permanently consigned to a narrow audience as a small utility function. You can easily think of 5-10 such services that you've tried and (probably) discarded, a few of which you keep using for their narrow purpose. So what differentiates Twitter from hundreds of other narrow utilities that came (and mostly went)? In a word, a Service Oriented Interface. Twitter started from day 1 exposing a simple straightforward Web Service interface. Even further, they never offered a feature without simultaneously exposing it. In other words, there is a Twi...

Reader Francesco Annunziata asked the following excellent question... ERP Vendors are embracing the field of BPM e and SOA. Sooner or later their solutions will catch up with those of current leaders. On the one hand, if they can’t show much deeper integration with their ERP suite than any other BPMS vendor can offer, they will be just another behind-the-curve BPMS struggling for market share (a quote from the Column 2 blog ). On the other hand, If they can take advantage of their ERP by showing a superior capability of integrating their applications I think the BPM deals will by default go to them. Those considerations boil down to my question: Is SOA, among other things, leveling the vendors' playing field in terms of integration and composite applications? In other words, will a potential Client be able to choose a BPM system (and other applications) only considering its capability to meet the company needs, "taking for granted" that the well implemented Service Orient...

For some years I've been dealing with IBM SOA oriented products. About 3 years ago I had a chance to be in an IBM center and discuss their EBS product strategy with some top IBM SOA experts. As IBM had (and now has even more) products in the space, I was trying to make sense of where to position which product that was being pitched to our large enterprise IT. At the end of the conversation I was not successful. Recently I was speaking to a top MDM expert about Oracle's product strategy in the MDM space. I was commenting on Oracle's "product", for which I had recently received a vendor pitch. He responded that Oracle has 5 products competing in the MDM space (and primary MDM tools). Today I'm trying to produce an architecture model for a medium sized IT shop that purchased IBM DataPower to include within their existing SOA model (and fit with existing tools). In my search I came across this slide from IBM... I see... one product provides fast connectivity...

What is Mainframe SOA? For the purposes of this article, I'm referring to exposing web services from COBOL applications in a mainframe environment. (The specific mainframe environment I'll discuss in this article is CICS.) Today there's a variety of tools and native functions that allow expose of CICS transactions or COBOL subroutines (or full applications) as Web Services from the mainframe. In brief, here's what these tools and functions come to solve: - Formerly COBOL did not handle XML natively. - While CICS began to expose web server type functionality some years ago, it did not have any way to link a web request to a COBOL routine. About 3 years ago both of these problems were solved natively, in somewhat of a limited fashion: - Enterprise COBOL integrated XML processing functions into the base language capabilities. However, COBOL is still limited to creating internal data structures using the standard COBOL data blocks. This means it's poor at handling ...

New generation SOA tools such as ESB’s (Enterprise Service Buses), Governance Tools, and Monitoring & Security tools change the integration approach and model. Instead of point to point connections degenerating into interface spaghetti over time, these tools provide centralized facilities for managing and controlling interfaces and inter-system connectivity. They allow you take control of the integration space, manage & control connections, and provide the potential for connection reuse. They also bridge the technology gaps in many a legacy environment between the interface technologies and methodologies of today and those of the past. Using them allows you to move to Centralized Integration Control and Management. This is extremely valuable to any large IT shop in preventing connectivity chaos and integration spaghetti, as well as significantly simplifying and (critical over time) standardizing interface methodology. This is Service Oriented Integration. It is not, by itsel...

Integration is tough. Traditional IT applications are spending as much as 40 percent of their budget on integration. As the environment complexity increases as well as the number of connections per system, that number may increase to 60 percent. Why? At the basic level every system has it’s internal data model and logical model. Every interface has to bridge and convert those models (for the interfaced elements) across two systems. Then there’s the practical aspects – matching connectivity technologies (or bridging them), matching security patterns, simply determining appropriate error handling, human contacts, etc. SOA has standardized the interface technologies and provided a wealth of tools to bridge the issues where standards don’t match. Most organizations are using these tools today, whether intentionally or because the programmers are using recent development tools that use SOA interface technologies by default (the more likely situation). Interfacing significantly easier natura...

(Continuing the discussion of Batch Processing and SOA.) This article focuses on USING services (consuming a web service) as part of a batch oriented process. As mentioned in Part 1 , there's a strong tendency to want to consume (web) services as part of a newly developed batch process - usually due to reuse (other reasons discussed in the Part 1). It's pretty natural to say "well, services are reusable, and here we have a process that needs to use them. So, use them." However, there's a natural architecture incompatibility that has to be address. Web services are, by nature, remote. Batch process are oriented around local processing of large quantities of data/transactions, by marshalling large amounts of local resources to maximize the throughout. Here's an example to explain better... In the ancient days of programming, say before 1990 or so, the majority of systems were built using local files. Relational databases didn't exist yet, or were just...

Two weeks ago, Zapthink published a fluff piece Batch Processing with SOA (by Ronald Schmelzer). The article says just two simple things: 1 - You can make batch process controls into callable services. 2 - On-demand batches ("workload automation") are the future of batch processing. The article ignores what are the key topics of a batch SOA conversation, namely: A - Can web services be utlized in the context of a batch process (or workflow automation or whatever you want to call it)? B - Can an ESB provide an enviroment for a batch process, or a significant portion thereof? In other words, is there a convergence between batch processes and SOA integration tools (web services, ESB, or even governance)? Can these respective spaces leverage one another, or are they fundementally incompatible? This issue has come up in the context of two clients within the past 2 weeks. The first question usually asked is WHY? Why would you consider an ESB in the context of batch processing? ...

The article of the year is from Ann Thomas Manes at The Burton Group: SOA is Dead, Long Live Services. Nothing like a provacative title to get your attention. SOA from a vendor perspective is whatever tool they are selling in the space: ESB, Design Time Governance, Runtime Governance, Policy Management, Registry & Repository, etc. What they always sell is the tool, never the process. Since the A in SOA is Architecture, and the true value of SOA is SOA for the Enterprise (as opposed to SOA for Integration ), it takes implementation of a SOA architecture pattern to get the hyped SOA benefits. But as noted above, this is not what vendors are selling. As most organizations focus on SOA for Integration and layer on a few tools rather than applying architecture and an organizational change process (which, face it, is hard), only incremental benefits will be gained. This is not a surprise, IT managers are rarely rewarded for shaking things up and taking risks. Much of SOA is about c...

SOA and IT News of Note will be an occasional article here at Making SOA Work. Modeling SOA can be a challenge. Good news, the folks at OMG have been working on it... Highlights: "It's an extension of the UML, the Unified Modeling Language, and it contains modeling constructs for things like contracts and service and providers and consumers, all things that you would expect if you were going to do the architectural delivery for a SOA-based project," The specification will be published as a UML profile, meaning it can be used in any UML modeling tool. It can be extended and built upon, Harrison said. Tools vendors are expected to incorporate it in their products, A little more at Infoworld.

Why is a Cisco router $10,000 and a Linksys router $100? Or perhaps the better question is why are your network guys willing to pay $10,000 for a Cisco router when a Linksys router can be purchased for $100? The answer is that Cisco routers have multiple layers of reliablity, managability, and security that Linksys routers don't. But 100x (100 times) difference? Is it really worth 100 times difference? The network is the backbone of your IT infrastructure. If your network goes down, all the IT systems stop working and so does much (if not all) of your company's operations. And keeping that up, being alerted early to problems, being able to reroute and resolve problems, that is indeed worth 100 times difference. When creating traditional big-box applications, the traditional components are: The Application You Wrote The Container or Runtime The Application Server The Database The Database Server (When we move to an application with a web GUI, you can add a web server in ther...