SOA Governance is Alive, Alive!

Suddenly this year clients are calling me up about SOA Governance, service libraries, service monitoring and control.  First SOA was dead, then SOA governance was dead (even saw an article that SOA governance is more than just dead, it’s a murderer, killing the future by stopping cloud computing).  What’s going on with all my customer calls?

Here’s a simple fact about services.  Until a significant percentage of frequently used business processes have been exposed, and exposed at a relatively granular (detailed) level (without being too granular that they require re-composition and orchestration every use), the major SOA ROI (return on investment) doesn’t happen.  Kind of like a real world physical library, it won’t have much traffic until either it has a large collection or a collection of popular material.

Most SOA efforts begin Bottom Up, with programmers and projects beginning to use SOA technologies simply because they are available and enable getting the job done (rather than planned, architected, and adjusting development processes to handle the new approach).  Therefore the services exposed aren’t necessarily ones with high (re)use potential, they’re whatever the particular project that was using some SOA tech happened to need for their deliverables.  Even if the enterprise moves to Middle Out, still the SOA development is oriented around the projects driven by the business need of the moment.

This makes traditional SDLC (software development life cycle) sense, IT should be driven by the business needs and goals!  But like a well planned data center doesn’t run new central network wires every time a new rack is installed, nor install a server per application (rather they manage overall “capacity”), and a well planned building doesn’t run new wires to each new office (but has trunk lines and break-out boxes strategically placed throughout the areas), some top down planning is required to QUICKLY get SOA ROI by focusing on the highly (re)useable services and enterprise business processes.

Since few organizations have done that, they don’t start to see real SOA ROI or value until they’ve built 200+ services.  Neither are they seeing significant multiple use (I don’t think “reuse” is the right term for services as reuse implies multiple installation, which services avoid) until they hit that milestone.  And without multiple use they don’t start to encounter SOA complexity, spaghetti integration, library and catalog issues, or governance issues.  Neither does service security become a big issue until key business processes are being exposed.  (Monitoring does become an early issue that most struggle through manually.)

Therefore it takes years for the average IT enterprise 3+ years of SOA and service exposing projects to either start experiencing significant service (re)use and the associated difficulties of managing a large quantity of active services.

Hence the sudden calls about SOA Governance.  SOA was being declared dead by the pundits as it reached the tipping point of moving from the early adopters to the early majority.  For the IT pundits SOA may be dead…as an area of interest and punditry.  They moved on to TNBG (The Next Big Thing), in this case Cloud Computing – which as far as I can tell is still stuck with the Innovators (as I’m not hearing even early mainstream type solutions from the vendors).

But for the majority, their SOA efforts have been building their libraries and are entering later stages of implementation maturity, thereby encountering the need for services management a.k.a. (also known as) SOA Governance.  Both design time needs in the form of catalogs and repositories as well as some service lifecycle control, and runtime needs in the form of detailed monitoring,  transaction monitoring, root cause analysis, and SLA definition/control/alerts.

Surprise IT pundits, SOA Governance is going mainstream.  Given the responses I saw watching governance trying to be sold two years ago (and my own efforts to implement a SOA Governance tool set 6 years ago), I’m pretty surprised myself.

Fortunately, though it was slow going for sales for the vendors they haven’t been standing still.  Today’s vendor offerings in the SOA Governance space are sophisticated, capable, and offer some surprisingly advanced features.  The vendors have been some very complete governance suites that cover the requirements of the space extremely well.

SOA Governance is alive, alive!

We Don’t Centralize That

 

I was recently working with a client on SOA governance / service management.  They’ve build several hundred enterprise services and major integration points, and trying to manually manage them through a web site (or Excel spreadsheet or the brains of the integration department people) is just getting out of hand.

This isn’t so unusual, it’s become the classic SOA design time governance problem encountered 2-3 years into the SOA maturity cycle.  While there are those claiming SOA Governance is dead, focused on Design Time Governance (I guess this would be a subset argument of the SOA is Dead punditry), the problem keeps cropping up in Enterprise IT.

To be fair to the (Design Time) Governance is Dead argument, few organizations have successfully deployed SOA design time governance.  It’s extremely tricky to insert a tool that touches so many points of the Software Development Lifecycle (SDLC) into the process and not have it rejected or avoided.  It requires adjustments to the enterprise SDLC, a well defined process that provides and demonstrates direct (and early) benefits to each of the roles of the users, and balancing of the overhead it adds against the benefits to make sure adoption occurs.  In other words, some carrots, some sticks, some enforcement, and some internal sales and ongoing demonstration of value is necessary to make it part of the organization culture.  Without those efforts, the tool may survive for some cataloging purposes but the Design Time Governance implementation will be a failure.

So back to my client.  As we gathered the requirements and reviewed their SDLC process, I encountered the database people.  I invite them to Design Time Governance requirements gathering for two reasons.  First, most design time management tools include general reuseable “asset” management capabilities – and therefore the tool can serve more than just web services but other reusable enterprise components such as stored procedures. 

 

Second, the database people are often a valuable resource relative to Design Time Governance, as web service reusable assets follow a similar model and SDLC as stored procedures.  Those enterprises using sophisticated stored procedures have to develop IT processes for cataloging them, maintaining a list of who’s using them, who supports them, and what they do.  While various IT enterprises are at different levels of sophistication on stored procedures, almost every database group instinctively understands the issues relating to stored procedure reuse and sharing.  Therefore the database group’s example can be held up either as a model to pattern for services, something to be understood and taken to the next level for services, or at least have some people at the table that can provide some concrete examples to which the rest of IT can relate.

So at this customer the database people arrived for the requirements discussion and I began asking my standard questions about how they manage shared stored procedures.  They responded, “we don’t”.  Ok, not so unusual to find no management (unfortunately).  I asked a few more questions to understand if they thought an asset repository would help them.

They expounded, “We don’t DO stored procedures. Since we don’t have any way (tools or methodology) for managing a shared object, we require our applications to keep their logic and complex SQL in their code.  This prevents any management issues associated with reusable objects in the database.”

That was a first.  They don’t do stored procedures because they can’t manage reuse.  Wow.

I guess there’s a certain wisdom there.  No deployment and dependency confusion.  No unexpected change impact.  No confusion of ownership or support.  They keep their environment clean by eliminating the ability that could confuse it.

Still, that was a first.

(It’s possible the first graphic was borrowed from Weblayers – a pretty cool SOA vendor, on this page.  Then again, it came up on a Google image search without the page, so I can’t say for certain where Google handed it over from :-)

IBM DataPower Architecture and Features

 

The Datapower has an internal structure of components that can inherit or be reused, depending on their place in the inheritance chain.  Here’s the secret internal architecture of the DataPower:

 

And here’s the Datapower’s capabilities in a nutshell:

 

Multi-Protocol Gateway: (superset of XML Firewall)

Transformations – Any-to-any transformation engine: MPGW can parse and transform arbitrary binary, flat text, and XML messages, including EDI, COBOL Copybook, ISO 8583, CSV, ASN.1, and ebXML.

Transport Bridging – protocols such as HTTP, HTTPS, MQ, SSL, IMS Connect, FTP, and more

Message-level Security - Messages can be filtered, validated, encrypted, and signed, helping to provide more secure enablement of high-value applications. Supported technologies include WS-Security, WS-Trust, SAML, and LDAP.

Logging - logging and audit trail, including non-repudiation support

Web Service Proxy:

Schema Validation

Policy Application

SLA Monitoring

Load-Balancing w/Peers based on SLA

UDDI Directing

Governance Tie-In

XML Firewall:

Protect Externally Exposed Services

XML Threat Protection

Heavy Authentication, Authorization, Validation

Dos, Virus, SQL Injection Prevention, XML Node Attack Prevention

Attachment transformation/conversion/interface with virus scanner

SLA Monitors

Web Application Firewall:

To protect an internet exposed web application in the DMZ.

XSL Accelerator:

Accelerate XML Parsing
Accelerate Schema Validation
Accelerate XSLT Processing
XML Compression/Decompression ***no one uses it anymore

A quick way to select the best service can be found by answering some questions:

_ Does the service use a WSDL? If so, then use WS-Proxy.

_ Does the service uses multiple transports? If so, then use MPGW.

_ Does the service use only XML over HTTP? If so, then use XML firewall.

_ Is there only non-XML and HTTP traffic? If so, then use Web application firewall.

…and that’s a wrap on our Datapower information at this time.

Datapower – Balancing and Failover

 

Ok, this is a little more practical and technical than we usually get, but important nonetheless.

The IBM Datapower, as well as similar devices from Layer 7 and Cisco (and others) provide SOA security, attack prevention, and a number of ESB-like abilities (somewhat of an ESB lite).  However, these boxes tend to be EXPENSIVE, as well as having a series of add-on software modules that raise the price significantly.

The latter is a shock to many people.  One thinks of the Datapower as a physical device, like a network router.  While it is a physical device and much of it’s performance is based on optimized software placed in ASIC hardware chips, it’s also a very sophisticated software platform.  As such they’ve done the standard vendor thing of separating many of the sophisticated abilities as separate software add-on modules – adding on ability and adding on price.  For example (if I remember correctly), the ability of the Datapower to connect to a database stored procedure and expose it as a (secured) service is an add-on feature.

Because it’s an expensive device, and it’s advertised as a high availability device, and it’s considered to be a hardware platform like a router, and they want to isolate development/test from production, many organizations are taking one for development / test / QA, and one for production.  (And often none for disaster recovery!)

This is a mistake in two ways.  First, even high availability devices fail.  And if the Datapower (or other hardware based SOA security device) is layered into the security control of all SOA services, then if the device fails ALL SOAP WEB SERVICES (or at least all that go through the device for security, runtime management, and/or ESB lite abilities) are offline until the device is physically replaced.  Being it’s expensive, you can assume IBM doesn’t have them sitting around at the local office.

Second, the device offers a multi-tenancy ability.  It has several ways to automatically separate logical instances from each other, and reasonable internal protection of one tenant impacting another.

For these reasons, it’s highly advised to cluster your Datapowers and use the multi-tenancy features to logically separate your development/test from production environments. 

And don’t forget disaster recovery!  If you are heavily reliant on this device and don’t have one in disaster recovery, then if you lose your primary data center you can be out of service for an extended time until a physical replacement can arrive and be reconfigured.  The basic rule is whatever it touches and whatever those services touch will be offline until it’s replace.  That could be a VERY big deal as integration has spread far and wide throughout the enterprise.

A Basic SOA Security Model – Part 2

 

Validation.  Security headers, SAML, WS-Security, etc are useless unless they’re being validated against SOMETHING.  Usually this something is an LDAP, and in a Microsoft environment the Active Directory.  If you’ve got a security device such as a Datapower, it can be configured to perform this validation.  So can many SOA runtime governance and SOA security tools.  Use it!

Exclusivity.  Sources of requests must be validated.  If there’s a Datapower (or similar) device in the picture, this means both providers and ESB’s only accept requests from the security device.  If not, security agents have to be configured to validate sources.  If no agents exist, there’s a hole to be exploited.

Protocol Independence.  Security rules must remain true regardless of protocol…http, https, ftp, s/ftp, MQ, Tibco, etc. To act in this fashion, every message must contain the XML security header whether it’s sent by HTTP or another protocol such as FTP or MQ.  Security is independent of protocol.

Audit Database. Sufficient transaction content, particularly the header, must be saved for auditing, problem research, and potential forensic research.  This has to be done well, meaning an independent asynchronous process that causes no overhead to the services.  A message queue with an independent logging process works very well to solve this.  However it can generate a large quantity of data, so a good archiving and/or deletion policy has to be in place as well.

Attack Protection. Attack protection is an important feature that may not be necessary for internal services but is critical for internet facing services.  It prevents various forms of XML and Injection attacks. However, it requires a security device such as the Datapower or Layer 7 XML boxes.

A Basic SOA Security Model – Part 1

Encryption

Every web service call should be SSL encrypted (HTTP/S). Similarly, any web service operation or file transfer sent via FTP should utilize Secure-FTP (S/FTP). Basic level security requires that the receiving source have a valid signed certificate, but it does not require dual-side certificates or any validation of the certificate beyond it being a valid source on a valid root chain. The objective of this requirement is encryption, not authentication.

Now encryption will frequently be argued against.  “It’s high overhead, slows things down too much.” and “Setting up security on every server is time consuming.”

First, we must layer our security.  By exposing services we’re transmitting internal application data outside the security perimeter of the application!  This data must be protected from view, and dropping a network sniffer onto a developer’s workstation or a development or test server is a trivial exercise.  So even if there’s some overhead, we must provide some protection.  Second, today’s server CPU’s will process SSL with little overhead (3 to 8%), and even less if there’s a Datapower in the loop (as it’s optimized for it).  As far as the setup time, the setup overhead is brief and a one time cost (per server).

Basic Service Security.

Ever service should be required to include:

a. The Requesting Application
b. The Requesting Server (IP)
c. The Requesting Environment (version, platform)
d. The User Context (what user in the requesting App activated a function needing this service)
e. Authorization ID+Password
-- i. For this service or just from that application?
-- ii. Different per environment (Dev, Test, Production)

Now depending on whether you’re handling this manually or not, some of this information will be included automatically.  Note we’re not discussing HOW this information is included here.  (For example SAML for the authorization ID/password, and/or WS-Security for some of the other information.)  Rather, what should be included.

Security control will differ by organization and maturity of the SOA implementation.  But here’s what needs to be known…

- The Requesting Application.  The simplest level of access control is to specify that the exposed service can be used by specific other applications.  For example, the Customer Service application can call the Get Bill service of the Billing System.

- The Requesting Server.  We want to make sure that not only is the service being called from an application that is allowed to do so, but from an instance of the application that’s allowed to do so.  A simple way to do this is by checking the source IP of the request.  Yes, this is WEAK and easily spoofed (meaning it’s not difficult to fake a source IP on a TCP/IP request).  But, it’s simple and better than nothing.  (Note it’s included in the network packet already, so it doesn’t need to be added, just checked.)

- The Requesting Environment.  Cross-connecting between development or test and production is a not infrequent SOA mistake.  The security should protect against this, but we must give it enough information to do so.

- The User Context.  Some environments have a federated identity model in place.  This may be as simple as Microsoft’s Active Directory or as complex as Tivoli’s Identity Management for managing identity across multiple technology platforms (such as Windows to Unix to Mainframe).  If so, this feature may not need to be part of the SOA security or rather just pass a token from the identity environment.  In either case the idea is that the user context in the application requesting use of a service should be passed to the application(s) exposing the service(s) for auditing purposes (logging who requested this action) and for security purposes (are they authorized to request this action).  This can be challenging in an environment without identity management as when we connect the applications a complete new user based from the requesting application may require authorization entries into the providing application’s security controls.  Therefore many organizations ignore this area.  This is a mistake…even if not used for actual authorization the data should be logged for auditing purposes.