Architecting the Software Development Team

I was recently deep in an architecture process when I was asked by a team member for some help in understanding the team operating model.  Or more specifically, a team leader and architect were asking me whether they should be coding a particularly difficult area of the system.

Being in architect mode, I immediately ran to the white board and architected the optimum software development team process.  Here it is…

 

What I was trying to describe was the role at each level, which is achieved through experience, and the flow between them.  So while an architect or team leader may be able to develop code at 3-5 times the rate of a Junior Programmer, everyone above can fill the roles below – but the reverse is not true.  So if that architect is programming, no one is architecting!

And while the architect might be able to code without an architecture or design (because he or she has an image of it in his or her mind), if he (or she) does so there is no architecture or design for other programmers that come along in the future and have to deal with that code (maintain it, extend it, interface with it, etc). 

When the senior people do this for base system components, they make a major team mistake that leaves everyone else struggling with their now undiagrammed undocumented features in the future.

It may get it done faster now, but it’s usually a mistake in the long run.

CloudCon – Funny Vendor Quotes

I’m sitting at CloudCon III – SaaSCon 2011.  It’s “a Cloud Conference with a focus on Software as a Service.”  Sadly the presentations are of limited value, with the same confusion I noted in my “Impressions” article (i.e. every IT business marketing department is trying to take advantage of it and rebrand their abilities “Cloud”).

While not of particular technical knowledge value, they to tend to result in humorous statements by the presenters…

IBM 

“80% of Fortune 100 companies are using IBM cloud capabilities.”  Wow, you’ve got 80 customers?  Really?  (Those Fortune 100 companies, that spent from $200 million to $1 billion per year on IT costs, are generally using some of every capability of every major IT vendor.)

Google

“4 million businesses have gone Google.”  As of 2007, the US Census Bureau reported there are 29,413,039 businesses in the U.S.  Assuming Google’s talking just about the U.S. (and I don’t think they were), that’s a 13% market penetration!  Wow!  (Not!)  If were were to take 2011 numbers and go worldwide, it might be 3% penetration.  Double wow!  (Double not!)

“We expected cloud email to be a growth industry and a challenge to Microsoft in the Enterprise.”  Chuckle.  This is your Cloud goal?  Email?

"Chromebooks – nothing but a browser, configured via the cloud, automatic upgrades, subscription model. Strong processor, wifi, 3G, battery lasts a full day. No hard drive.  Easy to replace a traditional laptop.  Happy IT managers and end users.”  Oh, and costs $499 in the US for a 12.1 inch netbook, $200 more than a Windows netbook with 1/2 the ability.  #Fail

“What the cloud offers: Enhanced Security”.  You’ve didn’t actually say this?  You couldn’t have actually said this!  Savings, definitely.  Ease of access to abilities, yes.  Flexibility, definitely.  Enhanced security, no way in h#ll.  If you’re going cloud you BETTER be spending A LOT more time layering on the security!

Symantec

“Software as a Service, Cloud, Managed Services, Hosted Services, Outsourcing – we just change the name now and then to keep it fresh.”  Well that was refreshingly honest.

“Strength in depth.  A cloud based solution, a gateway based solution, a desktop based solution, all from different vendors.  It’s expensive, but when places are serious about security this is what they do.”  Is someone really talking straight?  So unusual not sure if I can handle it.

Impressions from a SaasCon – CloudCon

I’m sitting at CloudCon III – SaaSCon 2011.  It’s marketed as a Cloud Conference with a focus on Software as a Service.  Here’s what I’m seeing…

a.  Computer hardware vendors selling small footprint office workstations.  It’s not a surprise that computer vendors for the office have finally decided to abandon the standard desk-drawer PC box for a cigar sized box.  (Anyone who opens a standard PC box will find the components would fit in a cigar box anyway, the rest is open space or fans.)  The surprise is it took so long and that they’re selling them as “cloud workstations” and spending their money trying to market them at a Cloud Conference.

b.  Network hardware vendors selling the next speed in network infrastructure, 10 gigabit.  Apparently since everything’s “in the cloud” you need yet more network bandwidth to get to it.  This is some nice marketing fluff since Cloud doesn’t increase your bandwidth needs, it just shifts it from inside your internal network to some external vendors as well – to which your network connections are inevitably slower simply due to WAN costs.  As far as the internal network goes, network storage devices, SOA and web services and heavy application infrastructure has already resulted in the massive network speed and capacity increases.  Not that I’d complain about deploying apps or integrations on a 10gbit network, it certainly makes non-local devices respond even more as if local.  But again, not new, not “cloudy”, just another marketing ploy.

Anyone notice I haven’t mentioned anything Software as a Service oriented yet?

c.  Consulting vendors.  “We’ve got Cloud experience and expertise.”  Sure you do.  Reminds me when early in my career I was seeing advertisements for “5 years of Visual C++ programming experience” when Visual C++ had only been general release for 1 year.  On the serious side, I did have a conversation with a major consulting vendor division VP who told me they have started recommending the use of some “private cloud” resources – which they translate to mean some storage or computing resources hosted in the consulting vendor’s data center.  So for some consulting vendors Cloud means outsourcing a customer’s storage or computing requirements to the vendor data center and/or hosting the customer’s applications for them in the consulting vendor data center.

d.  Utility service vendors.  Symantec “virus protection as a cloud service”, somebody offering Fax as a Cloud service (there’s something incredibly weird about offering a 1980’s technology as a 2010’s cloud ability), central Email management as a Cloud service, and Telephony as a Cloud service.  The last one is kind of interesting though again not what I think of when I think Software as a Service.  Since the PBX moved to VOIP (voice over IP) and the office phone handsets moved to TCP/IP network connected digital devices (the less technical may not have noticed over the last 5 years their office phone moved from being connected to a phone wire to being connected to the office network), it makes sense you could move the PBX to a Cloud service.

e. The Big Vendors.  Did you know IBM offers cloud services?  IBM Smart Cloud!  It’s IBM, it’s Cloud, it’s Smart.  Marketing at it’s best.  Not much to actually say or show beyond “we’ve got lots of data centers and cloud offerings around the world”.  Ok, we know it’s IBM and they’ll (probably) make just about anything you want work…if you’ve got money and time.

f.  Data center hosting vendors.  They can host your servers, they can virtualize for you, they can host your storage, your backups, your network services…oh and by the way they’ll host your Private Cloud (which for them is just your collection of servers in their data center).  A minor twist on what they already do.

f.  Far off in a little corner by themselves were a few real Software as a Service vendors.  A CRM vendor, a Project Management vendor, real life Software as a Service vendors offering their applications and their abilities and various pricing models.

Net net, it tells me that Cloud and Software as a Service remains a confusing poorly defined poorly understood tech space.  Every IT business marketing department is trying to take advantage of it and rebrand their abilities “Cloud”. 

But like the hype cycle for SOA, many try but not that many are offering actual value in the space.  The Cloud and Software as a Service market has a lot of growing to do and maturity to gain before it stabilizes.  There’s definitely value to be gained right now, but also the possibility to be taken by ridiculous claims and expensive products and services offering marginal value.

Batch Out to Web Services?

Calling web services from the mainframe has become a frequent question.  But as applications (and data) may be migrated off the mainframe to apps now hosted on Linux or Windows servers, the old trustworthy batch jobs may suddenly need to access remote systems and web services to do their job.

Here’s how one person phrased their problem…

We are currently looking at doing a partial migration away from a MainFrame.  Some of the functionality written in Mainframe Cobol and is called from Mainframe Batch programs.  We would like to move these cobol programs off the mainframe.  Question - If we moved the functionality in the cobol programme to a Java or .Net web service, is the a way to call this web service from a Mainframe batch programme?

Technically this is an easy answer.  Yes, web services can be invoked from the mainframe.  They can even be directly invoked from CICS and from IBM Enterprise COBOL (as of CICS TS 3.1).  There are some technical limitations to this, Enterprise COBOL web services cannot deal with complicated XML structures and all XML data types – which can make it a challenge to call pre-defined web services with modern standards.  But if the web services are being created directly to service the Enterprise COBOL call, no problem (technically speaking).

Architecturally, this type of batch web service invocation does have a major flaw.  Anyone doing batch programming knows that database commits can cause significant performance problems for batch, and therefore careful management of the database commits (and other database activities) are part of every batch implementation (commonly commits are only done every 100 transactions or more).

Similarly, every web service invocation has an overhead cost.  Multiply this by tens of thousands or hundreds of thousands of transactions and your batch process will spend most of it’s time waiting to make the web service connection.  And that time may run to hours or more.

The solution is similar to the database commit approach.  The web service must be designed to pass multiple transaction requests through a single invocation.  The communication connection is made and an array of transaction requests (in mainframe COBOL speak) or a list of SOAP documents (in web service speak) are transmitted during the connection. 

Naturally the receiving web service must be designed to handle multiple transaction requests in a single invocation, and practically this is not a problem in any modern environment (such as Java or .Net).  It is an unusual pattern that most don’t consider, but even in most normal circumstances there is no reason that a web service shouldn’t handle multiple transactions included in a single request body or multiple request bodies in a single communication instance.

This is the only HTTP SOAP based approach to this problem.  Other alternatives include queuing, loading all the requests into a messaging system (such as IBM Websphere MQ Series), with the processing system having a reasonably large thread pool for pulling and processing the messages as they arrive.

These ‘mixed environment’ batches are already very common, and many organizations have no intention to move away from the ‘large processing job’ approach.  As resources spread even farther and into the cloud, this problem will grow ever more ‘interesting’.

What Cloud, Which Cloud, Where Cloud?

Cloud Computing has strongly entered it’s hype cycle.  Just as everyone ran to relabel everything Service Oriented and ESB-this or that, now everything is being relabled Cloud this or that. 

As soon as that happens we enter the technology confusion cycle.  (Sometimes one thinks this is intentional on the part of vendors, so you can’t tell exactly where their product fits or where it lacks.)

Let’s see if we can do a little bit of Cloud Clarification™…

-> Cloud Computing is about pushing applications, components, modules, abilities to an on-demand model with remote capacity.

-> Software as a Service is about renting software abilities via a vendor exposing abilities, modules, business processes for remote use.

-> Cloud Infrastructure, which is also being called Cloud Computing, is about renting remote computing/hardware capacity on demand and in fractional increments.

About the best picture I’ve seen describing this is here… (though there’s some details in it I’d quibble about)

 

Today most Cloud ‘things’ being sold or used are Cloud Infrastructure, meaning remote storage or remote computing capacity.  What makes them different from just renting a server from a hosting vendor somewhere is it’s usually available on demand (renting a server, whether physical or virtual, usually involves some wait and setup time as the server is prepared for you, which includes installing or allocating the right amount of memory and disk) and is charged in fractional increments.  For example, Amazon’s S3 storage service charges per gigabyte per hour. 

The other big Cloud activity is Software as a Service (SaaS).  SaaS is making serious inroads in major IT shops as Salesforce.com and others are making an excellent case for simply using their software remotely, with a lower cost than a major CRM, ERP or accounting software purchase plus associated installation, administration, and server costs required for a normal software install.  And not to forget ongoing maintenance costs, support, and high availability redundancy.  Most major software vendors are preparing Software as a Service editions of their major application products.  CA, SAP, Peoplesoft, BMC… Salesforce.com has proven the SaaS model and the majors are trying to follow (not always an easy task with many major application products having old software bases.)

Real “Cloud Computing”, the ability to dynamically deploy some code or modules or components or services to on-demand container environments has not yet had major penetration.  It’s understandable as the complexity is much higher and has to be coordinated with development environments and tools.  Many start-ups are trying to create the right combination, and I’m sure we’ll see increasing traction here soon.

So the question remains,

• Will I be able to mix and match business processes and capabilities from multiple vendors business process (formerly application) portfolios?

• Can I “deploy” my integrated orchestrated capabilities in an on-demand environment?

• Will it let me gain a strategic business advantage by creating unique processes exactly matching my business goals?

• Will I be using and only paying for the exact capacity I need?

• Will I be able to change my processes quickly and easily as market conditions change?

Not yet, but to some of them there is a partial yes, and all of them are in sight.

CICS Web Service Compatibility

IBM has done significant work to allow mainframe based applications to expose and consume web services.  They’ve particularly targeted CICS and languages COBOL, PL/I and C++. 

While many vendors (including IBM) offer a variety of tools to provide easy web service bridging, IBM’s CICS efforts offer a direct path without loading and managing additional utilities.  There was concern in the past of the CPU load this added to CICS, but IBM handled those problems over the years with the current edition having good performance with reasonable overhead.

While IBM recommends using modern development tools such as their Rational Application Developer for Z/Os (RD for Z) to automatically generate and build the binding and WSDL’s necessary for a service, their CICS command line based utility is probably used by most that do so (this being DFHLS2WS).  With a short series of configuration sessions (and limited options), it will take program information and data areas and generate appropriate WSDL and binding files.

Which is where it gets interesting.

IBM’s mainframe folks have worked hard to generate a very exacting and 100% compliant WSDL while dealing with the difficult aspects of fixed length fields, fixed array quantities, EBCIDIC to UTF-8 translations, and unique language storage models (such as COBOL’s COMP-3 packed decimal field).  For the most part they’ve done a good job of mapping 2nd generation languages’ internal storage models to XML and the latest schema/XSD capabilities.  But in a few areas they’ve seriously fallen down.

1. Arrays.  Empty arrays are sent through in XML as…repetitive empty copies of the field/tag.  So if in COBOL you have a 05 PHONE-NUMBERS PIC X(9) OCCURS 50 (that’s an array of fields called phone-numbers of 9 characters), in XML you get <PHONE-NUMBERS> repeated 50 times, with no values if empty but still there.

2. There’s sophisticated namespace usage by IBM in the resulting WSDL, with the WSDL having one namespace, the request tags having another and the response tags having a third.  At the start of each section the DFHLS2WS utility names an XSD complex type “ProgramInterface”.  Before doing that they change the namespace, but to NOT tag the names in the section with the namespace.  The strict rules of XSD’s say this is acceptable.  BUT when .Net programmers import the CICS generated WSDL into Visual Studio or Java programmers import it into Eclipse (including IBM’s Rational Application Developer for Websphere), these tools reject the WSDL due to a duplicate name (the repeating of ProgramInterface without the namespace pre-pended but after the namespace change).

Technically, that’s a bug in Visual Studio, Eclipse and Rational Application Developer, as those tools aren’t handling the namespace change and implicitly placing the names in that section in the namespace.

Practically it means IBM’s CICS team outsmarted themselves with the XSD sophistication of the output of DFHLS2WS, using a feature that’s not well supported by the developers of the development tools that will be importing the resulting WSDL.

Interoperability standards are critical.  But vendor interoperability isn’t perfect.  While we shouldn’t have to go to the lowest common denominator, going to the highest isn’t wise.

[ There is no solution for this particular DFHLS2WS problem beyond writing a utility to automatically modify the output WSDL, or manually modifying it.  Maddeningly the IBM documentation even says it is likely to need to be modified rather than offering options or flexibility! ]

Cloud Computing, As a Service, and Taxes

In the past months a number of articles were published about Cloud Computing and taxes.  As a techie, choosing a software vendor on the basis of taxes may not be something considered.  But, depending on the jurisdiction, software purchases can be charged sales taxes, service taxes, or value added taxes.  While one might think ‘rented software’ isn’t a purchase and can’t be charged a sales tax, in some places it may be considered a capital acquisition (and taxes as a sale) while in others it could be charged a valued added tax (which taxes services as well as sales in European countries and Israel) or might be subject to various business taxes (such as a franchise tax in Texas where every company’s data on the Cloud is considered a local franchise and taxes as such.)

First an important definition of terms, because almost EVERY “Cloud Computing will be Taxed” article I read got it absolutely WRONG…

Cloud Computing – renting computing resources from a remote vendor on the basis of computing units rather than physical equipment.  For example, I need web serving ability that can handle a site taking 100,000 visits an hour and storage for 5,000 high resolution photos.  I do not rent servers, computers, or hard disks, I rent “capacity” and often pay for capacity x price-per-capacity-unit per day of use.

When using Cloud Computing resources, I do not know where the resources I’m using are or what physical equipment is involved.  I just need some “capacity” to “run my stuff”.

Software / Platform / Integration / Infrastructure AS A Service – renting use of particular software capabilities with the software operating at the remote vendor.  For example, I need CRM (Customer Resource Management) functions for my business, so I decide to use Salesforce.com which provides me a series of software-package abilities that I access remotely running somewhere on their servers (or in their Cloud).  I do not rent servers and install their software, I rent “software capacity” measured in software usage units (per day / month / or year), such as number of users from my office + number of customer records I store in their software x price-per-software-unit x amount of time used.

Cloud Computing is something IT people might talk about and use, particularly infrastructure people.  …As a Service is something the business people might talk about, such as “should we buy SAP CRM or use Salesforce.com Software as a Service”? 

As states and countries are becoming desperate for taxes, all kinds of weird tax schemes are being extended to cover data centers and software services.

The whole point of Cloud Computing and Software (etc.) as a Service is that the user (the company using the services) do not need to worry about where the supporting equipment is, what it is, is it secure / backed up / managed / etc.  All the physical operational details are managed by the Cloud or As a Service vendor.

Even further, the vendors themselves are expected to managed and balance their physical capacity, so from where your service is being provided, upon what your service is running and where your data is located may change as the vendor rebalances their customer load.

However, if we factor taxes into the picture it’s possible using software in a location or having your data in a location could subject you to local taxes.  Which means these questions suddenly apply (which are the opposite of the point and goals of Cloud Computing and As a Service)…

In what state / country / jurisdiction is the service user (company) located?

In what state / country / jurisdiction is the service provider located?

When companies are multi-state and multi-national, those questions can be even more complicated.

In what state / country / jurisdiction is the computing power located (where’s it running, or where’s the data center it’s running within)?

In what state / country / jurisdiction is the data storage located?

You can’t just ask “where’s the server”, as the server may be virtualized or relocated depending on capacity requirements at any given time.  With current Cloud Computing technology, it’s probably staying within 1 physical data center though.

Does the service provider have a physical location in the state / country / jurisdiction where the service user is located?

(This one’s extra complicated.)  Is the service user using the provided service for their own business or reselling the provided service in some way?  (As an example, photo hosting site SmugMug uses Amazon S3 cloud storage service to store people’s photos.)

So now when considering Cloud or As a Service, one must also consider the tax implications in the ‘buyer’s’ jurisdiction, the ‘seller’s’ jurisdiction, the jurisdiction of the data center providing the computing power and the jurisdiction of the data center providing storage capacity (if different from the computing power).

As noted above, Texas (somewhat accidentally) tries to charge a business franchise tax if Texas based data centers are providing your Cloud or As a Service services (the assumption being your computing or data in their state is a physical presence in their state – which is why Cloud vendors all left Texas).

Similarly California recently tried to tax Amazon.com by stating local people in your affiliate program qualifies as a local office (to which Amazon responded by cancelling their affiliate program for anyone with a California address).

To date the Cloud vendors are solving the problem by avoiding jurisdictions where such taxes might have impact.  But in major deals it’s appropriate to involve your lawyers and accountants to verify contract and local law details that may apply.

Much internet growth was fueled by the opportunity to avoid brick-and-mortar taxes.  Let’s hope politicians don’t overly tax the Cloud market before it has a chance to develop it’s advantages and provide a business base worth taxing.

Basic Enterprise Web Service Security Concepts

In the (near) past, security was handled by the user interface.  The user interface acted as the sole entry point to the application, and therefore all application security was oriented around user permissions.

Added web services is like having great locks on your front door but opening all the windows in your house.  Lots of entry points, each of which needs security.

There’s a few basic enterprise web service security concepts that need to be understood to understand web service security.

Web service security may operate from a user context, an application context, or both.

User Context: Application 1 includes in the (web) service request to application 2 information about the user who performed an action causing the request. Application 2 then decides if the service is permitted based on the user requesting it in application 1.

This requires applications 1 and 2 to have a common user security framework (application 2 has to recognize application 1’s user and be able to check if that user is authorized to request the service operation being requested.)

User Validation – How can application 2 know that the user sent by application 1 has been validated by application 1? One answer would be to send through the user’s password, but application 1 rarely has access to the password (as it may be under the control of an external security system such as Microsoft Active Directory), and sending a password in a message has it’s own security risks.

A solution frequently select is Single Sign On software. This is integrated into both applications 1 and 2, and when the user logs in gives the application a “user token” instead of user information. This user token can then be passed in the message, and application 2 can simply ask the Single Sign On utility if the user token is valid and active (is the user still logged in).

If applications 1 and 2 have no common user context, no shared user base or shared security source, then user context security can’t be used.  Rather, the best that can be done is application 1 can pass along the name or ID of the user who performed a function resulting in the web service request, and application 2 can store it (for logging or auditing purposes), but can’t check any sort of permissions (as the user is unknown to application 2).

Application Context: Is application 1 allowed to activate a particular service in application 2? Is application 1’s test environment allowed to activate that service in application 2’s production environment? (probably not.)

Application context is about whether the source of the request (the source from a program / code / environment perspective) is allowed to request the action being asked from the destination (program and environment).

Enforcement: Some ESB’s (Enterprise Service Buses) have internal features to enforce some of this type of security. (Some require add on modules.) However, even if the ESB is enforcing this type of security, the end points of the requests (the service providing systems) must also be protected and have service security enforcement. Otherwise, what is to stop a developer (or hacker that gets into the internal network) from directly accessing a product business web service from a workstation or laptop? (Nothing.) Further, services are intentionally designed to be easy to use and understand (therefore security through obscurity may no longer help.)

Complete enforcement is best done using SOA security tools. These will either include an agent on each end point or route all services through security enforcement gateways (with the end points only accepting requests via the gateways).  It is possible to create your own security enforcement function in front of services (such as with IBM Websphere where a “handler” can be inserted in the Web Service engine), but is generally not recommended (as you would have to recreate it for each technology exposing services – which the vendor’s provide.)

Agent Based Security Model

 

Gateway Based Service Security Model

A Code Weapon

 

Stuxnet: Anatomy of a Computer Virus

Early Signs of SOA Success

I’ve been working with a client for an extended period of time.  This large IT department has had a variety of SOA tools and technologies available and has been doing major systems integration for 10 years.  Yet while their SOA tools have allowed them to integrate quicker than manual development, their integration methodology (essentially none) has given them 0% reuse.

Reuse is a fine objective, but it may not actually be valuable depending on the business and IT organization goals.  In this client’s case we did an extensive evaluation of IT current state, IT future state plans and goals, and business goals.  That may sound like a lot of overhead to determine future state integration and SOA approaches, but in the current economic climate architecture for architecture’s sake is simply not acceptable (if it ever was).

Or to put it another way, when IT is aligned with and demonstrating direct business value then IT is valued by the business.  And this attitude has to filter down to enterprise architecture, integration and SOA.

This is not to justify SOA (service oriented architecture).  Rather, SOA must justify it’s overhead by demonstrating how it’s going to provide value in meeting the IT and business goals.

At this client we identified 3 primary business and IT drivers for integration:

1. Business and IT systems agility.  This client is in a dynamic business environment and is frequently reinventing parts of their business, leading to an unusually high volume of major application replacement and feature revamping.

2. Reliability.  As the complexity of the interconnections between systems and applications had been increasing, reliability was suffering, sometimes with real dollar measurable business impact of downtime or data loss.  (Correspondingly more and better people were needed for support as more time was spend on more complicated problems.)

3. Integration Cost Reduction.  Integration (and integration support) were taking higher and higher percentages of project budgets and the trend continues to grow.

As I noted at the beginning of the article, I’ve been working with this client for some time.  Meeting this client’s goals is mostly about IT process changing and IT thinking change (though some solutions can be met with select tools for certain parts of the problem), for which we’ve been planning and preparing and planting the seeds as we’re reviewing projects in progress before the new processes are complete.

This week I saw in the organization the first signs of real SOA success.  I was sitting with an integration architect who was describing how he just saved 75% of the integration of 3 projects because we designed the services used by the first project in a reusable pattern. 

And that’s how it starts.

The goal is not reuse, the goal is aligning IT to meet the business goals.  Reuse is a method.  And seeing my client beginning to have success with the method and start meeting their goals…that’s exciting!

(Now we have to quickly put in place the KPI’s [key performance indicators] to measure the success and report it to all levels of IT management.  That’s the way to reinforce the positive people pattern and get the integration people positive recognition.)

Unionized IT and SOA

Labor unions are rarely found in IT organizations.  It’s not unheard of, but generally high pay rates and frequent job mobility have made labor or trade unions appear to be of limited benefit to the employee – and therefore rejected.

In general labor unions impose rules on the management that require employees with the greatest seniority (most time at the company) be promoted to more senior positions as they open up.  And they require new employees to be brought in at the most junior level.

IT, with frequently changing technologies, requires bringing in subject matter experts and promoting those demonstrating top technical skills to technical leadership positions.

Recently I’ve been doing some consulting on a large scale IT project, which involves quite a bit of Service Oriented Architecture and involves a unionized IT department.  They’re struggling both with the technical aspects of a major technology and architectural approach change and with the union job impacts of such.

In particular, like the classical unionized worker, much of IT is certain that it’s doing it’s job function in “the right way”.  It knows this because it’s the documented union approved procedure, and therefore the “right” one.

Similarly because of this the existing environment has been extremely slow to adopt new technology or new methods, as each such change requires negotiations with the union.  This has led to much of the IT operations literally remaining as green screen mainframe applications written in a standard 2nd generation programming language.

As an interesting aside, I heard that one of the union contract terms for pay includes a multiplier based on the CPU (mips) capacity of the mainframe.  (I guess the assumption would be if the “computer” is doing more then either the workers are doing more or their work is more valuable.)  This has literally led to increased computing power being delayed due to the impact on labor costs (and therefore maintaining slower application response time across the whole work force).

It’s hard to see how one can operate within union work rules and have an agile and integrated IT environment.  Perhaps there are agile unions that could make such a thing possible, but traditional union patterns do not seem compatible with agile IT.

Instant Realtime BI with SOA BI

 

BI, Business Intelligence, has taken hold at almost every mid-size or larger IT organization. 

It commonly means extracting key data elements from all the main systems and databases in the organization and compiling it all together in the Business Intelligence Data Warehouse.  And the primary method for doing this is ETL – extract, transform, and load.  Basically meaning batch-style data loads performed daily, weekly, or monthly (from the source systems to the data warehouse).

Setting it up is expensive and time consuming as it requires building a large capacity database and ETL processes for every important data source in the company.  The ETL processes by themselves are often not enough as data duplication and data quality problems quickly float to the surface and have to be resolved to a sufficient level to continue (resolved in the data warehouse, not in the data sources).

However, it’s relatively easy to demonstrate the business value of the resulting Business Intelligence Center, as that cross-system data provides business process statistics, results, and reporting as none of the systems can standing alone.  Further, intensive data mining can be performed that can’t be applied against the source systems (either because they’re operating live and can’t handle that depth of data access or because the value points come out by the combinations across systems and complete business processes.)

Most businesses that implement BI consider it a win and see their ROI.

Now as I’ve been implementing a new enterprise SOA strategy at a major customer, the BI team arrived in an architecture strategy presentation meeting and asked “ok, how’s this (an enterprise SOA strategy including common entities and processes) different from BI?”

In essence they’re pointing out that they already have integrated to everything (or at least every important data source), they’ve already translated between all those disparate data models to a single enterprise model (the structure of their data warehouse), and they’ve already handled the cross-system conflicts of object model conflicts and different representations of the same data. 

The primary differences between BI ETL integrations and SOA integrations is small.  One, BI is a timed infrequent (daily or less) data extract, SOA is a real time integration.  Two, BI tends to work in a true ETL model…extract the data in the source system’s format, transform it to the data warehouse’s preferred format, and load it into the data warehouse.  SOA, when it moves beyond a point to point connectivity technology does several transformations, first from the source format to XML, second from the source data model to a company or industry standard, and third for legacy situations from the company standard to the destination system’s requirements (though this need fades with time if a company standard is selected). 

But conceptually BI and SOA integrations are doing much of the same thing!

Now BI is being faced with a new requirement…realtime.  Companies are seeing such a value in BI that they’re asking why they can’t get the reports and analyses RIGHT NOW (rather than tomorrow or next week).

The SOA vendors offer a solution to part of this desire with their BAM – Business Activity Monitoring tools.  These tools (examples include IBM Websphere Business Monitor and Software AG Webmethods Optimize) monitor data elements passing through web service requests and use it to build a real time image of what’s happening – an image that can be identical to the BI image generated 24 hours later through summarized data reporting.

However, the integration teams and Integration Competency Centers have generally been unsuccessful at selling this ability to their business users.  This isn’t because it doesn’t solve the problem but rather because the business users of BI and integration are different.  BI is actually being used by key business users.  Integration is generally service the IT executives as their “customer”, and therefore BAM doesn’t fit in with their normal “offerings”.

Realtime BI has become one of the “hot” IT topics for 2011.  There’s two relatively easy solutions to help BI become realtime, though they’re somewhat problematic politically as they violate current organization structures at many IT shops.

Solution #1 – Get BAM SOA tools but give it to the BI team to use as part of the offerings to their data needing business customers.

Solution #2 – IT shops with a good library of existing connections and services can begin to echo certain update services directly to the data warehouse and BI team for realtime handling.  (Realtime in this case means sent to a queue, the realtime update processes can’t stop and wait for the data warehouse to process the updates.)

In both cases it requires the BI teams to begin leveraging the integration teams infrastructure.  However, the connection can bring BI the realtime options it needs with minimal effort.

Other models, such as Change Data Capture utilities, are great for increasing vendor software sales (and they do work and are impressive tools) but aren’t necessary… IF we can get two disciplines with somewhat different historical goals to begin working together.

Those that do will get a big and relatively inexpensive win.

Private Clouds? Maybe Not.

Jason Bloomberg over at Zapthink has a devastating indictment of vendors selling “Private Clouds” and “Platform as a Service”.

If you’re headed towards cloud computing (and we all are, it’s just a question of how long over the next decade till you get there), read it.

What’s With Web Service Security???

Web service security is a tricky business.  EVERY service exposed by any service provider, be it .Net, Java, the Mainframe, or any other provider needs to be secured.  Certainly if it’s exposing sensitive data (say customer data), allowing activation of a business process, and most especially if it’s involving a financial transaction.

But how do you do it?  While every vendor and (almost) every technology announces compatibility with every web service security buzzword (WS-Security, SAML, X.509, etc.), they don’t describe how to actually make use of all this security data attached to the web service request.

I’ve had recent discussions with IBM, Oracle, and Software AG (as leading SOA middleware tool providers) on this exact topic and the results are disappointing.

The architecture model for this says that to provide SOA security I should use the tools as a SOA security layer, allowing my services to go about their business and the security tools to grab and process the security data added on to the requests.

This means, for example, that my service requesting systems could activate a SOA security module (provided by a vendor tool) and get an appropriate security context added on to the service request.  This might be composed of a WS-Security block with certificates or keys and a SAML block composed of requesting application context (instance information – production/test/etc) and user context (user name or user id or with federated identity management session id).

My middle step, the ESB usually, would automatically processes the security context, authenticate the source and authorize the requested action.  It would then make any calls to providing system with an updated security context.

The providing systems would filter the requests through a SOA security agent, which would perform the same security actions (process the security context, authenticate, authorize, log for auditing – triple-A security).

That’s a reasonable expectation for SOA security tools.  Now where are the vendors at?

Oracle used to be the closed to this, with Amberpoint offering agents for providers and a central-agent/server for environments that couldn’t handle agents or installations where agents weren’t desired.  However, Oracle has apparently removed security functionality from Oracle and built a new Oracle Web Services Manager tool that does not have agents.  Rather, it allows policies to be created and validates security as the services pass through the central security server or through security enabled Oracle SOA tools (such as their ESB.)  [They did say they intend to add agents for select environments, such as SAP and .Net, over the next year.]

IBM never fully detailed the model.  Rather, they allow Websphere Registry & Repository (WSRR) to define policies which can then be pushed to a Datapower (a physical XML firewall device), which can then act as a central security service doing the authentication, authorization and log for auditing.  The providing systems have to be manually programmed to only accept the requests from the Datatpower and the requesting systems have to manually generate the security header, completing the security picture.  This model is fine for external web service security (exposing internal web services across the firewall for internet requests) but isn’t so great for internal.

Software AG’s options are wider but more confusing.  Their design time repository, CentraSite, can define and generate security policies.  These policies can be pushed to the Mediator add-on for their ESB, which can then authenticate, authorize and log for auditing as service requests arrive at the ESB, or can be pushed to Layer-7 physical XML firewall devices {Layer-7 does have a “software appliance” as well} for enforcement like the Datapower in the IBM option.  Interestingly IF you have Software AG’s SOA runtime monitoring tool, Insight, the Insight agents can be tasked by the Mediator ESB add-on to perform authentication and authorization (with the monitoring tool already doing logging).

In all cases, the requesting system is responsible for generating the security information required.  No one (as far as I know) is providing an agent or client side plug-in that takes over the security layer tasks from the service requestor.

Now, it’s worth noting  that all the functionality mentioned above can be created manually without too much difficulty using the native features of ESB’s (whether from IBM, Oracle or Software AG, or others), and Java as well as .Net provide a host of included classes and API’s for handling web service security. (Even IBM CICS does nowadays.)

The question is can I just load in my web service security layer or do I have to take a tool from a vendor and STILL spend time, either on the requesting side, providing side, or both, to build the steps necessary to complete the web service security picture.

At the moment, it seems this is still the case.  No matter what tool combination is used, I’m still going to be defining the security standards I want to use and writing some code to inject those security protocols (or interpret them) into my web service requests.

10 years into web services this is disappointing.  And dealing with customers looking to increase the SOA maturity level of their environments as well as some late comers to SOA, they just don’t understand why this would be necessary.

Intermediate SOA Security Questions

SOA security is getting more attention.  Recent document leaks show the need of inside-the-network security, and recent attacks on service providers by Wikileaks supporting hacker-nets show the need for AT LEAST “decent” SOA security.  As ZapThink wrote recently…

If you’ve been even peripherally paying attention to the news lately, then you know about Wikileaks and the battle going on between supporters and opponents of the group. While you might think that these incidents have nothing to do with you, enterprise architect, IT manager, developer, vendor, or consultant at a private company or government institution, they most certainly do. You are about to become a part of the frontline in the Cyberwar battle…

Just as firms plan for server outages and deployment problems, now every firm must have a contingency plan to deal with the potential outage of their most important online suppliers…

Your company uses Google for mail, a specialty Software-as-a-Service (SaaS) supplier for its B2B network that just happens to use Amazon’s cloud computing infrastructure, and processes its online payments with an Internet-based credit card gateway. The first thing that the company notices is that it can’t process payments because the credit card processor is under siege. Then, its B2B network goes down because Amazon is under attack. Finally, you can’t even send requests for support to either the card processor or the B2B network because Gmail is down due to a distributed denial of service attack (DDOS). At this point, the company is effectively knocked off the net from a business standpoint, even if its own website is up and operational.

For just how many hours can you afford to be disconnected as collateral damage in a Cyberwar? You say you have no contingency plans to deal with direct or consequential damage from malicious attacks? Then you are due for a Cyberwar Crisis Point…

The more interconnected you are and dependent on third-parties for your IT capabilities, the more threatened you will be by these attacks. Disconnecting yourself from your suppliers and the Internet is not an option these days.

If you’re Service Oriented, your applications are exposing more and more services inside your network and connecting more and more to services outside the network.  If you are actually working with Service Oriented Architecture, the whole model of your business process interaction is cooperating services.

All this means is that a Service Security Layer is critical as the IT threats have increased significantly.  Along these lines I’m sending some vendors some questions, to understand how their products can offer at least BASIC and hopefully some INTERMEDIATE level SOA security capabilities that can be directly plugged into the average IT site…

Basic High Level SOA Security Questions for Vendors:

[ Basic means this is the average security the average IT shop should have.  If you are dealing with financial transactions or sensitive data, more than this is appropriate.  High level means these are approach and toolset questions, not a long list of details of particular protocols supported.  You should always ask your vendors what protocols and methodologies are supported and verify they match your CURRENT requirements and show good possibilities of covering FUTURE requirements. ]

Vendor, security is an integral part of today's SOA and ESB environments.  These questions are to understand how or what part of your SOA products or suite provides features for an average SOA security model, and how.

Here's what I mean by an average SOA security model:

1. Triple-A.

Authenticate - All incoming service requests, regardless of protocol (HTTP, FTP, MQ, SMTP, etc) have the source of the request authenticated.

Authorize - Validate that the requesting source is permitted to operate the service being requested against a source list of services and authorized requestors.

Audit – Appropriate information on the servicer request is captured and stored for later forensic or debugging research.  Management of the audit contents should be automatic and tightly restricted (meaning scheduled archiving and purging).

Question : Which tools do you provide that offer these required abilities?

Question : Which features (protocols, tools) do your tools offer for these abilities?

2. User context and integration with identity management tools.

It's often not feasible to synchronize security between the front end user presentation tools and the back end servicing systems.  Sometimes this is because federated identity may not be in place, other times because attempting to map actual users against each service exposing business process steps or even IT utility functions is just not reasonable.  Regardless, the ability to accept an actual User ID and pass it along as part of the service request remains a key features, to allow back end systems to trace activity back to the user initiating the action.

In environments using identity management tools, the user context may be represented as only a session token, and to store the actual user information for auditing requires querying an identity management service with the session token.

Question : Which tools and to what level of ability do they do this (versus requiring development of a unique service to do so)?

Question : With what enterprise security environment and IDM tools do your SOA security tools integrate?

 

3. Validations and Prevention.

The general assumption is that the ESB and services are not going to come under attack from inside the network.  However, recent document leaks have demonstrated to many customers that internal security is important as well.  Further, many service problems often involve programming errors on service request content and occasional placement of completely inappropriate data types.

Therefore we often require products offer schema validation ability at each step of the service (meaning the ESB should validate and the providing system should also validate – for it can't guarantee a request will arrive only from the ESB).  Similarly the ability to offer some level of attack prevention even inside the internal network is appropriate.  While it may seem unlikely an internal service is going to come under a DNS attack, we've seen this happen in production not due to an intentional DNS attack but due to a programming error looping a service call tens-of-thousands of times against a providing system.

Question : Which of your tools can do this?

Question : What protections and validations do they offer?

Question : How's the performance (or rather what's the performance impact)? [ It's understood if this is done on the servers as opposed to within an XML appliance the CPU impact can be significant. ]

 

4. All the above apply to each step of a service path.  Every exposed service needs to be secured, validate the requestor, log the request, and validate the safety of the request. 

Question : How do your security tools provide these abilities to every service being exposed?

Question :  Any technology limitations on where your tools can provide such abilities?

Question :  Which of your tools are required?

 

5. External Service Exposure.

Customers are more and more needing the ability to expose services outside the internal network.  Further, I'm writing this article in Israel, which is a major target for hacking attacks for ideological reasons.  So I'd like to understand the various models for securely and safely exposing web services to external customers using SOA tools.

Question : Are XML Appliance/Firewall products part of your product offerings or integration with other vendor's appliances?

Question : Please explain the interface and management of the relationship between your SOA (ESB or Security or Governance) tools and the Appliance/Firewalls.  (Meaning can I publish and operate all from 1 console or do I need to put it in a Design Time Governance or Policy Tool, push it into an ESB, then export it to an Appliance and go into the appliance management console to complete the process?

Question : Some customers are looking for a less expensive method than installing multiple XML appliances (primary, failover, disaster recovery, etc.)  Is there a secure software model using your tools for doing this?  (Perhaps, for example, installing an instance of your ESB in the DMZ and creating a secure channel to an internal instance of the ESB?)  What, how, how secure, and what additional modules or products are required to do this?

BPA, BPM, BPMS, Business Modeling???

Is BPM (Business Process Modeling and Execution) the SOA killer application?

I had a very interesting interaction with a global VP of BPM of BPM for a major vendor.  The encounter went like this…

VP on behalf of vendor demonstrates BPM tools to potential client.  Everyone oohs and ahhs at the impressive presentation, graphics, and overall possibilities of BPM.  The vendor product looks impressive, it works, and actually has reasonable (runtime) performance.

Then I ask the VP “how many processes can this client create in a year?”.  After all, creating a BPM workflow / process is relatively quick and easy.  In theory a couple of days to create it, another week to test it and deploy it.  Maybe a mid-sized organization should be able to generate 26 processes per year per person assigned to BPM.

The VP responded, “7”.  “Seven”, I remarked, “why so few?”  He explained first of all the BPM processes won’t have all their steps exposed (at all or at the right level of granularity).  This turns almost every BPM creation into an integration project (and explains why many BPM tools come with a large number of adapters and other integration components – because they have to struggle to get to the process steps or data needed for them). 

Then he added  that few businesses actually have their processes mapped out.  So the managers must get together with a business process modeler, model the process, run it through a few senior managers who revise it, send it back down where they adjust it, and finally get the process approved.  Then when they deploy it they find out that how the managers think things are working is not exactly how they ARE working…and it goes through a revision to reality.  Naturally the whole thing can take a few months.

So what has happened?  I see BPM as having divided into two separate directions, with most tools focused on one side or the other.  The first are very pure business driven Business Process Modeling and Execution tools.  These tools and projects are driven by the business with limited IT involvement, and while they directly solve business problems they are limited in their ability to do so due to IT not offering the access to the IT systems/services performing the business steps.

The second are Business Process Automation tools and features, some of which have become part of a number of Enterprise Service Bus products (others which are tightly integrated with ESB’s and architecturally a close layer above).  In many cases the ESB logic of specific integrations has been morphing into partial (and sometimes complete) business process automation routines.  And the ESB’s themselves often offer a fully visual drag-and-drop drag-and-connect modeling-like environment for creating them.

As I note to the confusion of my colleagues, a BPM process may be exposed as a service and be composed of services.  It’s activating event may be a service request, and it’s terminating action may be to activate a service.

There may be no differentiating factor between the approaches from the tool standpoint (and both IBM and Software AG have been busy buying both the higher business approach tools and the lower BPM automation tools – with roadmaps for merging them together), but there is a common differentiator from the IT standpoint…

When processes are particularly long running (days or longer) and/or include stops for human input (or interaction or approval), these are typically the business oriented BPM processes.  When the processes are short running (seconds to hours) and/or don’t normally require human interaction (except in case of an exception), these are typically business process automation routines.

Business Process Automation offers significant possibilities to IT architecture and inter-system and component interaction designs and should directly be incorporated into the Enterprise Architecture and Enterprise Integration Layer (or tightly coupled to it right above).

Business Process Modeling and Execution (execution of the model) has shown less success as an enterprise APPROACH and should be targeted at specific business problems where it will directly bring it’s ROI.

Back to the Future or the Past?

I sat with my client on a major service orientation project as IBM presented their updates to CICS.  My client is a few versions behind (could actually find no compelling reason to bother to upgrade) on their mainframe and wanted to review if IBM’s added capabilities would offer any abilities the project could utilize.

IBM’s CICS support for web services is well known by now.  Their CICS updates keep the web service support up to the latest WSDL – SOAP – and WS-* standards.  If you need or want to expose COBOL programs as a web service (and this can be valuable to allow older applications to be partially utilized as transaction engines in their area of expertise) IBM’s keeping the capabilities in line.

IBM has also continued to expand a large variety of internal CICS capabilities.  This was a major time warp for me as CICS command line functions, com areas and linkage sections, and various transaction batch and script coding abilities are so dated.  As an example when my children see me occasionally drop to a DOS box or telnet into a Unix command line prompt on some web servers they freak out.  “Wow, you’re like in the guts of the machine or something.  Whoa.”  Manipulating your files, whether on your local machine or on a server, through command line prompts is just a dated concept to which they can’t relate.  Drag and drop is the mode of the day.

As a better example…a Fortune 500 company I worked for in the US had a tremendous problem with turnover (employees who would leave quickly) in the call center.  While call center work is rather tedious and employees staying more than 1.5 years is unusual, their problem was a bit different.  The majority of the customer care applications were mainframe green screen (being displayed in an emulation window on a windows workstation).  The employees, mostly fresh out of high school or college, couldn’t figure out how to work without a mouse!  The result was it taking an extra 2 months to train them…and then they’d leave at 5 months due to the discomfort!

The most amazing thing offer was in CICS 4.1.  Here IBM has added….wait for it…Event Processing!  CICS now has an event processing module that will allow you to code a series of event triggers against program comareas or linkage sections.  The Event Processing Module will monitor activity as it passes between programs and activate the coded event upon hitting the triggering event activity.

I was astounded as the idea of manually coding events off the interaction of modules calling each on the mainframe presented itself!  Astounded as I tried to reconcile IBM’s attempt to keep the mainframe relevant by adding the latest technology possibilities with the reality of the narrow limited implementation.  When I think of event processing handling, I simply don’t think of CICS command line coding and monitoring inter-program communication areas.  I’m astounded someone at IBM tried to marry these two concepts.

Mainframes still aren’t going away.  And continuing to enhance their abilities to expose transactions and integrate into a service oriented enterprise makes a lot of sense.  Leveraging in the latest architecture approaches into a 70’s command line interface…not so much.

The Integration Data Model

Loriane Lawson over at IT Business Edge has a penchant for touching upon particularly interesting integration problems.  This week she asks…

It's interesting: Writing custom code for data integration is nearly universally frowned upon by experts – and yet, I'm seeing a lot of discussion about creating your own metadata solutions for support with integration.

My question is: If you're getting away from hand-coding, why would you want to delve into customized metadata solutions?

The article focuses upon some of the technical issues and technical approach to such a discussion.  I’d like to focus on the business issues, and how those should be directing the technical approach (but aren’t).

Every application involved in integration brings along it’s database (or general data) model and it’s internal data object model.  These models were developed to meet the functional goals of the particular application.  For example, the “customer” representation in the CRM (customer resource management) application will be different from the “customer” representation in the billing application.

The different requirements of that representation lead to a different logical implementation of the representation.  In the CRM application it may be a series of relational tables representing different aspects of the full customer – and a series of classes that provide the different portion of customer interaction.  In the billing application customer may be a single table and a single class, as only basic customer information is needed for the billing process (which is focused on charges and activity).

Business-wise, the billing department doesn’t think about the full customer relationship.  Customer service or future sales are not part of their business process.  Their view of the customer is a narrow slice and it’s reflected in their application.

When we want to integrate these systems we’re faced with translating between these different models of customer.  And in many integrations the data mapping effort is THE major effort of the project.

Not the implementation of the mapping and data transformation.  ETL tools and ESB tools do this easily with visual drag-and-drop mapping and/or simple transformations scripting (with the ability to drop back to a code level for very complex element transformations).  Even third party tools are available that offer map-and-drop-into-your-environment deployments (see Altova’s MapForce for example). 

Rather, the effort is focused on the actual analysis and modeling necessary to produce a transformation map that can be implemented.  Gathering accurate information on the exact meaning of each data element is often very challenging.  Not at the database layer (where often there’s a decent data model with some minimal documentation, or at the very least a basic one can be generated by the database for you) but at the internal application level.  Because it’s the internal application data object model that’s being exposed.

And good documentation on that layer is rare, often requiring ‘digging into the code’ to get an accurate answer.  The problem can be worse when dealing with a vendor application.  There one is reliant purely upon detailed vendor documentation being accurate, sufficiently detailed and up to date.

So the analysis and design of the mapping between the applications is often the major effort of an integration project.  As such one solution approach is to put major effort into Metadata Management.  Building or buying the tools necessary to capture and track this information as it’s analyzed and implemented becomes a major way of gradually improving the future.

Personally I recommend a different approach.  Because I’ve seen huge metadata management teams in large corporate IT shops be basically completely ineffective, I recommend an approach that reduces the mapping problem altogether.

By creating a layer of Standard Enterprise Entities for integration, preferably based on an industry standard such as ACORD for Insurance, HR-XML for HR transactions, or one of the many XML industry standards, we force all integrations to map into and out of the standard.  This gives us a fixed set of metadata to work with, and over time as the individual applications develop new functionality and expose it, rather than expose in their proprietary format and spend time metadata mapping they begin exposing directly in the enterprise entity standard.  Further along the maturity cycle the enterprise standard begins to extend deeper into the applications as they begin to model their internal representations around the standard (avoiding even internal mapping).

Following this approach significantly reduces the metadata and mapping problem over the long term (5 years) and provides the short term benefits of speedier integrations (as the enterprise teams become comfortable working with the entities and can quickly utilize services exposed in that pattern without mapping).

Best of Breed vs. Suites

This is a classic IT question.  Should one go with picking and choosing Best of Breed applications in the various niches that one’s IT shop needs, or go with an Application Suite?  SOA and Integration has some significant input to this question, and impact from this question.  And the same question applies not only to business toolsets, but also to SOA toolsets (best of breed ESB, design time governance, run time governance, SOA security tools, BPM, etc, or a suite?)

With best of breed, we might end up with one company’s ERP system and another company’s CRM system, a third company’s manufacturing system and a fourth company’s financials.

With historical integration patterns, the issue of data interchange between the systems was mostly handling by large scale data exports and imports, usually performed as batch processes at end of day (or end of week or end of month).  These processes could be described as “dump all the (insert primary data type here, such as customers), then dump all the (day’s/week’s/month’s) transactions” followed by a specially written import program that read the foreign systems’ format and wrote the transactions (either directly into the database or processed through the exposed transaction API).

As these systems chained along the full business process, the latter systems in the chain might not be updated (assuming a daily batch) for 3 or 4 days.  Further, other systems that came along needing the same data wouldn’t necessarily go the the primary source (let’s say the first system in the chain), they would go to the easiest access point for the data (whichever system had the easiest API to use or easiest database to access) to pull it.

The result over time in these cases was an initial chain of connections that degraded into a web of batch extract, transfer, and loads hubbing off the original chain.

But today in our best of breed scenario there is an expectation of real time feeds between the systems.  The integrations are significantly harder as the systems have to be intricately connected, dealing with differences in data formats, connection protocols and paradigms, and transaction processing models. 

The integration effort in the Best of Breed scenario has gone up significantly, and the success or failure of the primary systems is completely dependent on a successful reliable intricate integration!

In the case of the Suite approach, each individual module (or major application) may not be the best in class solution.  Some parts of the suite may be excellent, others average and some just barely acceptable.  Yet, if the vendor has done their job well, the various modules are already well integrated.  Not having to build, manage and maintain an integration layer among the suite components is the key advantage of the suite.

Does that mean I recommend the suite approach over the best of breed approach?  Not necessarily.  There are good counter arguments such as avoiding vendor lock-in and being able to take advantage of particular applications that offer exceptional abilities in their area.

So how does one deal with a best of breed approach, whether total or partial (say you use SAP for many things but still use a few other solutions)?

The answer is integration processes and standards.  Building a well defined integration architecture layer that provides logical decoupling, as well as forcing your internal IT shop (and the vendors if possible) into XML industry standards is critical.

And what of SOA suites?  Mixing and matching the integration tools is just as challenging as the applications.  One can select, say, Software AG CentraSite for Design Time Governance / Services Catalog, IBM Datapower for security enforcement with SOA Software’s ServiceManager for runtime control and monitoring, and Oracle’s Fusion ESB.  Technically that should all be possible and should work.  On a practical basis, I don’t know of anyone who has succeeding in doing so.  (More often one finds most tools from one vendor and perhaps one component from another, and the IT shop dealing with the extra work of the particular bridge between that one integration point.)

I find it interesting that the proliferation of open standards and easy integration is driving us back to suites.  Not because we can’t connect everything but because it’s simply not worth the effort to do so.

Of course, those trying to do so is what’s keeping my work schedule completely full.  Hmm, maybe I shouldn’t have written this article.

Integration Spaghetti™

 

I’ve been using the term Integration Spaghetti™ for the past 9 years or so to describe what happens as systems connectivity increases and increases to the point of … unmanageability, indeterminate impact, or just generally a big mess.  A standard line of mine is “moving from spaghetti code to spaghetti connections is not an improvement”.

(A standard “point to point connection mess” slide, by enterprise architect Jerry Foster from 2001.)

In the past few days I’ve been meeting with a series of IT managers at a large customer and have come up with a revised definition for Integration Spaghetti™ :

Integration Spaghetti™ is when the connectivity to/from an application is so complex that everyone is afraid of touching it.  An application with such spaghetti becomes nearly impossible to replace.  Estimates of change impact to the application are frequently wrong by orders of magnitude.  Interruption in the integration functioning are always a major disaster – both in terms of the time and people required to resolve it and in the business impact of it.

Even as the spaghetti bound application is nearly impossible to replace, as the current state continues it continues to grow worse as additional connections are made to these key applications and derivative copies of the data are taken from it, or clones created to avoid it (and thereby creating another synchronization and connection point).

Such spaghetti takes multiple forms but often involves ALL forms with multiple generations of technology connections, including excessive point to point connections, tightly coupled connection technologies, database triggers, business logic embedded in EAI process steps, many batches in and out from and to many destinations, ETL loads and extracts to/from other databases, multiple services providing nearly (but not exactly) identical data sets, and the involvement of many message queues.

Anything is done to avoid dealing with the giant plate of spaghetti.

Systems will integrate with systems that integrate with it, piggybacking existing connectivity and putting a burden on the subsidiary system, to avoid directly connecting into the spaghetti.  They’ll go to a secondary or tertiary data source to avoid going direct.  Everyone knows avoid the spaghetti if at all possible and will spend double to triple the integration effort to do so.

If the primary system is replaced, it’s not unusual that the new system won’t be integrated into all the old connections – this would require actually understanding each existing connecting, extracting it and redirecting/reconnecting it to the new system – rather the OLD SYSTEM will stay around to act as the connection point for all the existing spaghetti connections and the new system will become an integration, taking data feeds or a regular ETL load, off the old system!  Meaning the old system lives forever!

Does this problem every get resolved?  Yes.  When the other side of the connections gets replaced, the new systems on that side will be integrated with what replaced the core spaghetti bound system.  If the IT shop is lucky after a generation or so the spaghetti bound system can be shut down.

Unfortunately in major Enterprise IT shops finding some spaghetti integrations is not unusual.  IT management is loathe to acknowledge such a problem to the business and will continue work-arounds until it directly impacts business goals.  Otherwise it remains just another hidden IT enterprise IT expense.

Signs of Industry Governance Failure and Recovery

 

A number of industry analysts have been speaking of SOA Design Time Governance failure for some time.  As I’ve written previously, primarily this was because the majority of enterprise IT shops hadn’t reached either the SOA maturity level to deal with it or had a large enough service catalog to have a need to address it with tools.

I’ve seen a lot of change in this in the past year, as many organizations are suddenly asking for help in defining requirements for SOA governance tools.

But what of the cutting edge IT shops, the early SOA adopters who ran into SOA governance needs years ago and started working with SOA Design Time Governance tools of earlier generations?  (I admit to being one of these, having led the purchase of a design time governance tool for my U.S. Fortune 50 employer at the time, about 7 years ago.)

Most of these projects FAILED!  (Including the one I ran.)  The tools were complicated and somewhat rigid, the processes to make it successful (the IT people-process changes necessary to successfully incorporate the tool) weren’t well understood.  As a result most design time governance projects morphed into simple service catalogs.  The complicated and expensive design time governance tools turned into a simple library website.  And sometimes not for services (as asset management tools with flexible templates, they easily serve the needs of other reusable IT assets – such as architecture designs). 

So today I find this interesting announcement in my inbox:

--------
SOA Software Announces Repository Federation Solution

Repository Manager supports downstream, horizontal, vertical, and custom federation scenarios

July 26, 2010 – SOA Software, (blah blah blah), today announced the availability of comprehensive SOA federation capabilities in its…product. These capabilities help customers share software development asset information effectively within and outside the enterprise, govern how this information is shared, and integrate this information (blah blah blah).

Downstream Federation:  (products) federation capabilities for leading service registries, including SOA Software’s Policy Manager, IBM WSRR, HP SOA Systinet, TIBCO ActiveMatrix and SAP ESR, automatically synchronize service definitions, supporting information and governance states both to and from the run-time environment.  This is unique in the market and provides customers with granular control of their various registries from a single platform…

--------

What strikes me about this?  Now we have a vendor who’s going to cross-load information from previous (failed) installations of design time governance tools.  I mean, why else would an enterprise have multiple design time governance tools (or even instances of the same tool)?  These tools, by nature, handle multiple libraries or catalogs of information. 

So multiples means previous failures have devolved into niche catalogs of select assets, become departmental tools or limited in use by a few major project teams or architect led projects.  (Given the price structure of these tools enterprises do not choose them as departmental solutions.  These are enterprise-wide priced tools.)

All this says to me that governance is making a comeback.  However, the key to any governance project is not the particular capabilities of the tools, but the processes and methodologies necessary to get a successful implementation and tool use acceptance.  Incorporating design time governance into the software development lifecycle is the key success factor.  How a vendor is going to help the customer to make that happen is the first question any customer needs to ask.

(Photo credit – SOA World Magazine)

Where does UDDI fit in the average Integration?

Addressing a service presents a few problems.  By putting the URL (or queue name if using messaging), you unintentionally couple between the service consumer and the physical instance of the service.  (Meaning what server it’s on, IP address, etc.)

Applications often unintentionally become tightly coupled simply by addressing connections directly, by IP address, server name, or queue name. This is unacceptable, as any change in the physical layer results in software changes. (Hardcoding such information is clearly a major mistake, but even placing it in a configuration file or database entry still results in application manipulation due to physical layer changes.)

Replace a server, redeploy all consumers of the services exposed on that server?  Ouch.  Even moving from development to test to production becomes a challenge (as you have to recompile or reconfigure as the consumer needs to repoint to the new provider instance in each environment).

UDDI was originally created as a runtime service lookup to decouple the logical use from the physical implementation.  Since service addressing must use outside facilities to allow adjusting addresses without touching the service, whether from the consumer/request or within the integration bus. UDDI fits the bill.

(Some IT shops try to get off easy by using just DNS to solve this problem.  This can be helpful in a single environment, such as the replacement of a production server.  But not for redirection between environments.)

However, some vendors have expanded basic UDDI systems into something much larger – basically expanding into Design Time Governance “registry and repository” capabilities…

Every organization using services should (but doesn’t have to) implement a UDDI.  The extended capabilities that have been loaded into many UDDI products often confuse the issue.

Example – IBM’s Websphere Registry and Repository started as a good production quality UDDI and was initially extended to handle MQ as well (handling queue addresses in addition to URL’s).  But then it was extended with a partial set of design time governance features, confusing the issue if I need a UDDI and have other design time governance tools.

In at least one conversation I’ve had with a vendor they had a hard time understanding what I meant by “runtime UDDI”, being focused on design time repository abilities.  UDDI’s become rather divorced from it’s original and primary functionality.

Surge of SOA Suites?

A month ago I wrote how suddenly I’m seeing a surge of interest in SOA Governance.  In an interesting follow up to that article, I found myself sitting in a vendor sales presentation for a full SOA suite of tools to a potential customer.  And this customer asked a very good question…

“We haven’t seen a lot of SOA suite sales in our region (and therefore haven’t seen many sales of your product). Why not?”

While many of the Fortune 500 and leading edge companies approached SOA with significant investment and (somewhat) serious commitment, others did not.

The innovators and early adopters have been “doing SOA” for 6-10 years.  Two things forced them into major vendor ESB products and SOA suites from the start.  First, these were the only tools (operating at an enterprise level) early on the market and they provided EAI and protocol + technology bridging that settled them into a solid enterprise IT niche.  Second, the extending of all the various development technologies into the web service space hadn’t happened yet – therefore the major tools were the only way to “do SOA”.

The majority of adopters (from the charge above) have had other options.  Various containers and servers extended to provide parts of SOA functionality.  EAI tools and gateway tools that provide parts of ESB functionality.  A variety of small vendor SOA tools and even open source projects to fill in function needs.

As such, before investing hundreds of thousands of dollars (or more!) in a major SOA suite and major implementation effort, most IT shops (particularly where I am located, a small country) have started with small tools.  These departmental sized tools have allowed them to get their feet wet with SOA without the large investment risk of a major suite and associated implementation project.

Suddenly though, as with my article on Governance, many of these IT shops are finding they’re hitting the capacity and capability limits of these departmental tools.  Having been “doing SOA” for 2-4 years and hitting the 100-300 major service range, the more limited SOA tools hit reliability and performance problems, lack capabilities for managing the library of services (at design time or, more importantly in this case, at runtime), don’t do much about security, are harder to use than the visual design and scripting oriented SOA suite tools, and don’t well integrate out into governance tools or BPM tools.

Net net, the value proposition of a SOA suite becomes apparent, and the fears of a high investment in an unknown area no longer exist as they’re already familiar with some level of SOA and looking to move up in SOA maturity (to solve developing SOA management problems and to begin to realize higher level SOA benefits).

This is good news for Software AG, IBM, Oracle, and Tibco.  But the window of opportunity is closing for Microsoft, HP, Fujitsu, Alcatel-Lucent, Progress, and others.  Niche product vendors will be dependent on how well their specialty solutions play with the major suites (examples: Layer 7, Intel SOA Expressway, Weblayers, iWay).  The jury is still out on whether the growing open source SOA suite Mule make an impact.

Microsoft Demonstrates Component Thinking

Loraine Lawson over at the IT Business Edge Integration blog discusses Microsoft’s entry into the MDM space in her most recent post.  She notes that:

“I was also intrigued to learn that the whole thing is API-based. Why does that matter? As Hayler explains, it allows ISVs to build apps on top of MDS. That's a good thing, since the description of MDS sounds pretty bare-bones at this point. The idea seems to be that ISVs will be able to build on better user interfaces and create support for things like version comparison, which, oddly, it does not provide out of the box.”

I was surprised at this bit of thinking.  It’s always nice when a vendor product provides ways of extending it’s functionality.  It’s occasionally useful for enterprise IT shops, and often useful for niche vendor’s or ISV’s looking for opportunities to extend a major vendor’s product(s).  But that’s some old school thinking.

Since the early days of COM (Microsoft’s Common Object Model), followed by COM+ and ActiveX, Microsoft has been exposing component level capabilities of their products.  Thus, not only could their products be used for their primary purpose, but they could be leveraged to provide modular capabilities to other applications.

Take a look here…  This is a presentation slide showing the object palette of the BPM tool from Acentn, Agilepoint:

It’s a little hard to see (click for larger image), but notice the Microsoft exposed functionality – Exchange server can be used to create meetings, appointments, and read them later (calendar engine), Excel can be used to read data, write data, and perform calculations (calculation engine), Active Directory can be directly accessed to manage a user base (user management engine), and Sharepoint + Infopath can be access to generate a variety of automated portal functions.  Even Word can be used as a print engine.

So when Microsoft says their MDM tool is exposing it’s functionality via an API layer (SOA based I hope), my thought is how I’ll be able to orchestrate that functionality into business solutions I’m architecting, and how well they’ll fit various BPM processes being created.

Extending the application is nice, decomposing the application into function modules (with an easy to access open standards SOAP API) is Service Oriented Architecture.  In this area Microsoft’s thinking is way ahead of the competition.