Business Drivers

I had a chat with a former co-worker at a US Fortune 50 about IT direction, what is the status for the people there now? Leading in to the conversation is the fact they formerly had a US $1 BILLION dollar IT budget, 5,000 US IT workers and another 2,000 positions outsourced (to India). First they cut that budget to $800 million, then cut a straight 25% of the US IT workforce...

Friend: I miss you so much in the office... these people will chase their tales for months, its insane.

Akiva: So basicaly Company has pulled back from IT being strategic to just another basic function they have to put up with?

Friend: Its funny because everyone is frustrated... they all want it done the "right" way... but won't invest the money to do it that way. So we spin round and round trying to find a cheap, but "right" way to do it and it never happens. Today I'm chasing another department to get 'free' services that they don't give away free (without internal chargeback).

Akiva: I'm working with some organizations now who have been doing that for years and suddenly have complete enterprise spaghetti that they can't change. They're desperate to be where Company was and has now turned from. Because the organizations that can change are going to start KILLING them.

Friend: That's going to be us again... its like we're on a cycle... and every few years it will circle back to what they wanted to go with a couple years ago... Everyone wants to do the right thing, but then money drives the decision and we never can. The architects are very frustrated.

Akiva: Well, there's 2 parts to that. The first is, basically the right thing is just get it done, damn the future. If that's the business driver, embrace it and be successful. And tell the architects that... their goal is to design the cheapest fastest and still stable system they can. Embrace the goal and be successful. The problem for the architects ss they're still thinking structure for long term maintenance and change, and flexibilty, reuse, and ease of interfacing.

Friend: But the architects are being told by the Architect Manager to do things the right way... so they get conflicting messages and gets them even more frustrated... Breaking the architects of that is very hard but its only because they knows how it will turn on us in a couple years and the mess we'll be in then.

Akiva: If there's one lesson I learned from corporate IT, it's forget the boss driver and grab the business driver.

Friend: Very true... business always wins out

Akiva: And about the future mess - so? That's the decision the business is taking. They decided to outsource and cut their IT knowledge base. Then they decided to cut to barebones IT capabilities. If they feel the money savings short term is worth the long term price....well, that's business. And you know what, they may be right! If those changes now keep the company profitable and other companies die, then they'll have the market share and reduced competition to re-improve the enterprise in the future.

Some tactical changes in IT approach make sense to match those goals. When there's a conflict and those business goals haven't filtered down to IT, and you have Architecture Management fighting with Project Management with completely misaligned goals - major problems result.

SOA and Batch - Part 3, Conflict

(Final Part of the discussion of Batch Processing and SOA.)

A web service access pattern is usually designed for "online" or real-time(ish) access. Most web services are designed for single request / single transaction access. A account record is requested, a customer record is updated, a transaction of some sort is executed. Rarely are services designed to handle multiple requests, 10, 100, 1000, 10,000 in a single service connection.

Further reinforcing the single operation model of most web services is the communication pattern. Most services are firmly operating in a request/reply pattern – regardless of whether the actual transaction operation is a query or update, whether the reply is a simple acknowledgement or a complete data response.

Therefore, if a batch of requests comes in, the ESB is intermediating between communication patterns. On one side is a single request, a “file” with many requests (tens to millions). On the other side is a service model that accepts single requests. As each request has a communications and processing overhead, as an example 50ms (a normal TCP/IP request/reply overhead), a moderate batch of 100,000 requests would have a time impact of 80 minutes only for the communications. If the actual transaction processing was another 50ms (that would be a very fast transaction), the total time would increase to 160 minutes or 2.6 hours.

This intermediation between communication models causes a significant time overhead, as well as utilizing a significant amount of resources in the ESB environment for minimal benefit. Further, the time overhead can create competition between batch jobs.

These are risks that have to be mitigated when using this pattern.

In summary, SOA and Batch are not well suited for each other. However, all too often we find this pattern developing or being pushed from the 'reuse' mindset. If you find yourself here, be careful to manage and mitigate the model conflicts.