Instant Realtime BI with SOA BI

 

BI, Business Intelligence, has taken hold at almost every mid-size or larger IT organization. 

It commonly means extracting key data elements from all the main systems and databases in the organization and compiling it all together in the Business Intelligence Data Warehouse.  And the primary method for doing this is ETL – extract, transform, and load.  Basically meaning batch-style data loads performed daily, weekly, or monthly (from the source systems to the data warehouse).

Setting it up is expensive and time consuming as it requires building a large capacity database and ETL processes for every important data source in the company.  The ETL processes by themselves are often not enough as data duplication and data quality problems quickly float to the surface and have to be resolved to a sufficient level to continue (resolved in the data warehouse, not in the data sources).

However, it’s relatively easy to demonstrate the business value of the resulting Business Intelligence Center, as that cross-system data provides business process statistics, results, and reporting as none of the systems can standing alone.  Further, intensive data mining can be performed that can’t be applied against the source systems (either because they’re operating live and can’t handle that depth of data access or because the value points come out by the combinations across systems and complete business processes.)

Most businesses that implement BI consider it a win and see their ROI.

Now as I’ve been implementing a new enterprise SOA strategy at a major customer, the BI team arrived in an architecture strategy presentation meeting and asked “ok, how’s this (an enterprise SOA strategy including common entities and processes) different from BI?”

In essence they’re pointing out that they already have integrated to everything (or at least every important data source), they’ve already translated between all those disparate data models to a single enterprise model (the structure of their data warehouse), and they’ve already handled the cross-system conflicts of object model conflicts and different representations of the same data. 

The primary differences between BI ETL integrations and SOA integrations is small.  One, BI is a timed infrequent (daily or less) data extract, SOA is a real time integration.  Two, BI tends to work in a true ETL model…extract the data in the source system’s format, transform it to the data warehouse’s preferred format, and load it into the data warehouse.  SOA, when it moves beyond a point to point connectivity technology does several transformations, first from the source format to XML, second from the source data model to a company or industry standard, and third for legacy situations from the company standard to the destination system’s requirements (though this need fades with time if a company standard is selected). 

But conceptually BI and SOA integrations are doing much of the same thing!

Now BI is being faced with a new requirement…realtime.  Companies are seeing such a value in BI that they’re asking why they can’t get the reports and analyses RIGHT NOW (rather than tomorrow or next week).

The SOA vendors offer a solution to part of this desire with their BAM – Business Activity Monitoring tools.  These tools (examples include IBM Websphere Business Monitor and Software AG Webmethods Optimize) monitor data elements passing through web service requests and use it to build a real time image of what’s happening – an image that can be identical to the BI image generated 24 hours later through summarized data reporting.

However, the integration teams and Integration Competency Centers have generally been unsuccessful at selling this ability to their business users.  This isn’t because it doesn’t solve the problem but rather because the business users of BI and integration are different.  BI is actually being used by key business users.  Integration is generally service the IT executives as their “customer”, and therefore BAM doesn’t fit in with their normal “offerings”.

Realtime BI has become one of the “hot” IT topics for 2011.  There’s two relatively easy solutions to help BI become realtime, though they’re somewhat problematic politically as they violate current organization structures at many IT shops.

Solution #1 – Get BAM SOA tools but give it to the BI team to use as part of the offerings to their data needing business customers.

Solution #2 – IT shops with a good library of existing connections and services can begin to echo certain update services directly to the data warehouse and BI team for realtime handling.  (Realtime in this case means sent to a queue, the realtime update processes can’t stop and wait for the data warehouse to process the updates.)

In both cases it requires the BI teams to begin leveraging the integration teams infrastructure.  However, the connection can bring BI the realtime options it needs with minimal effort.

Other models, such as Change Data Capture utilities, are great for increasing vendor software sales (and they do work and are impressive tools) but aren’t necessary… IF we can get two disciplines with somewhat different historical goals to begin working together.

Those that do will get a big and relatively inexpensive win.

Private Clouds? Maybe Not.

Jason Bloomberg over at Zapthink has a devastating indictment of vendors selling “Private Clouds” and “Platform as a Service”.

If you’re headed towards cloud computing (and we all are, it’s just a question of how long over the next decade till you get there), read it.

What’s With Web Service Security???

Web service security is a tricky business.  EVERY service exposed by any service provider, be it .Net, Java, the Mainframe, or any other provider needs to be secured.  Certainly if it’s exposing sensitive data (say customer data), allowing activation of a business process, and most especially if it’s involving a financial transaction.

But how do you do it?  While every vendor and (almost) every technology announces compatibility with every web service security buzzword (WS-Security, SAML, X.509, etc.), they don’t describe how to actually make use of all this security data attached to the web service request.

I’ve had recent discussions with IBM, Oracle, and Software AG (as leading SOA middleware tool providers) on this exact topic and the results are disappointing.

The architecture model for this says that to provide SOA security I should use the tools as a SOA security layer, allowing my services to go about their business and the security tools to grab and process the security data added on to the requests.

This means, for example, that my service requesting systems could activate a SOA security module (provided by a vendor tool) and get an appropriate security context added on to the service request.  This might be composed of a WS-Security block with certificates or keys and a SAML block composed of requesting application context (instance information – production/test/etc) and user context (user name or user id or with federated identity management session id).

My middle step, the ESB usually, would automatically processes the security context, authenticate the source and authorize the requested action.  It would then make any calls to providing system with an updated security context.

The providing systems would filter the requests through a SOA security agent, which would perform the same security actions (process the security context, authenticate, authorize, log for auditing – triple-A security).

That’s a reasonable expectation for SOA security tools.  Now where are the vendors at?

Oracle used to be the closed to this, with Amberpoint offering agents for providers and a central-agent/server for environments that couldn’t handle agents or installations where agents weren’t desired.  However, Oracle has apparently removed security functionality from Oracle and built a new Oracle Web Services Manager tool that does not have agents.  Rather, it allows policies to be created and validates security as the services pass through the central security server or through security enabled Oracle SOA tools (such as their ESB.)  [They did say they intend to add agents for select environments, such as SAP and .Net, over the next year.]

IBM never fully detailed the model.  Rather, they allow Websphere Registry & Repository (WSRR) to define policies which can then be pushed to a Datapower (a physical XML firewall device), which can then act as a central security service doing the authentication, authorization and log for auditing.  The providing systems have to be manually programmed to only accept the requests from the Datatpower and the requesting systems have to manually generate the security header, completing the security picture.  This model is fine for external web service security (exposing internal web services across the firewall for internet requests) but isn’t so great for internal.

Software AG’s options are wider but more confusing.  Their design time repository, CentraSite, can define and generate security policies.  These policies can be pushed to the Mediator add-on for their ESB, which can then authenticate, authorize and log for auditing as service requests arrive at the ESB, or can be pushed to Layer-7 physical XML firewall devices {Layer-7 does have a “software appliance” as well} for enforcement like the Datapower in the IBM option.  Interestingly IF you have Software AG’s SOA runtime monitoring tool, Insight, the Insight agents can be tasked by the Mediator ESB add-on to perform authentication and authorization (with the monitoring tool already doing logging).

In all cases, the requesting system is responsible for generating the security information required.  No one (as far as I know) is providing an agent or client side plug-in that takes over the security layer tasks from the service requestor.

Now, it’s worth noting  that all the functionality mentioned above can be created manually without too much difficulty using the native features of ESB’s (whether from IBM, Oracle or Software AG, or others), and Java as well as .Net provide a host of included classes and API’s for handling web service security. (Even IBM CICS does nowadays.)

The question is can I just load in my web service security layer or do I have to take a tool from a vendor and STILL spend time, either on the requesting side, providing side, or both, to build the steps necessary to complete the web service security picture.

At the moment, it seems this is still the case.  No matter what tool combination is used, I’m still going to be defining the security standards I want to use and writing some code to inject those security protocols (or interpret them) into my web service requests.

10 years into web services this is disappointing.  And dealing with customers looking to increase the SOA maturity level of their environments as well as some late comers to SOA, they just don’t understand why this would be necessary.