Expertise and Offshoring

While I was working as a senior manager in a Fortune 500 IT department, we went through a recession, and a few years later through the U.S. outsourcing offshoring trend.

During the 2001 recession, management reacted with reasonable downturn planning. Business management requested a certain level of cutbacks (say 7%) and IT listed projects that could reasonably be cancelled. Things difficult to cut back (without very serious planning) such as operations - keeping the lights on, normal systems maintenance & support operations, planned regulatory system work, and required upgrades (due to vendors end-of-support-life), these were all off limits. New development and system enhancements were on the line.

IT management also reacted in a normal downturn mode. Middle IT management had gotten a bit fat, so they trimmed middle management. For the remainder, they turned to their managers and instructed them to lifeboat. Those employees who weren't considered key - who weren't a subject matter expert, just a plain old team member - and who were lower rated in performance, were thrown overboard.

It was moderately painful as a manager to be judging people such a way, but the cutbacks didn't really bite as the method was focusing on trimming the least efficient staff (of course such judgements weren't always perfect). When the recession ended and business started picking up - with a resulting increase in demand to the IT department, the IT management made an interesting decision NOT to significantly increase staff but rather to try offshoring.

The offshoring calculus was interesting. The Indian vendors (such as Satyam, Cognizant, Infosys, Patni, and Tata among others) arrived offering offshore "resources" (people, but we never referred to them that way, just "resources") at a rate/cost of 3.2 : 1, meaning we could rent 3.2 consultants for the amount we paid to 1 full time employee (including all associated costs, taxes, equipment, office space, etc). Further, we could bring them onshore, directly replacing employees in our office, at a rate of 1.6 : 1. (While legally H1B and L1 US visa's don't actually allow for this beyond temporary role filling, in practice we consistently were instructed to use them for exactly that purpose.)

In theory, you're getting a massively larger number of people working on your software projects for the same amount of money, basically 3 to 1. Our IT department literally doubled in size, by pure number of staff working on our projects. But productivity did not increase. The output of the department did not double, rather it increased very marginally (maybe 25%).

What happened was systems expertise started dwindling. Every software system has some design patterns, both technical and business. Business goals, processes and rules are incorporated into the code and class/object structures. Algorithms and routines are selected and created to accomplish the software goals. And, like the layout of a city and the understanding of what it takes to get from one point to another within the city, the system gains it's own unique structure. Exactly like a large city versus a small one, bigger systems with many modules and functions take longer to learn to traverse than smaller ones.

People who spent significant time within a system become Subject Matter Experts - they understand the business goals which it is meeting (and which it's doing well or not so well), how it's doing it, which parts of the system are strong and which are weak, and when asked to modify the system have some idea of what it will take. Depending on the complexity of the system, it takes 6 months to 1 1/2 years to become a true Subject Matter Expert (SME).

Offshore teams do not arrive with any subject matter expertise. Further, there tends to be a lot of employee motion among such teams - staying on the same project for a year and a half is an unusually long engagement by their standards. (Not that an offshoring company won't be engaged on a project that long, but hot developers are moved to the latest hot project, lower level coders are brought in, and moving between jobs and companies has been a very frequent activity among Indian IT workers.) So the teams are constantly climbing the learning curve on the same systems, and losing the SME efficiency that long term IT employees have.

At the Fortune 500 IT shop, we did not solve this problem. Further, it seemed to be growing worse. Even though studies were showing offshoring value returns from 6% - 20% gains in value/reductions in cost (meaning though the number of people working on a project would be 2x or 3x versus having a local IT staff, the value returned was only a 6% savings over having that local IT staff), the offshore trend kept accelerating.

During a management offsite, myself and co-worker manager were asked to create a presentation on approaches to solving the problem. For having a small group of managers and 1 hour to solve the problem, I think we actually did present a number of good solutions. Here is that presentation...

Bench Strength

View more presentations from akivam.

Speaking Engagement

I'll be a featured speaker at:

The CTO & IT Architects Forum
February 23, 2009
8:30 AM

Crowne Plaza Hotel - HaYarkon
Tel Aviv, Israel

My topic: Is SOA Dead? No SOA ROI? Getting SOA Value instead of SOA Buzz

Here's a link to the event flyer (in Hebrew). The presentation will be in English.

Hope to see you there.

SOA and Batch - Part 2, The Access Pattern

(Continuing the discussion of Batch Processing and SOA.)

This article focuses on USING services (consuming a web service) as part of a batch oriented process.

As mentioned in Part 1, there's a strong tendency to want to consume (web) services as part of a newly developed batch process - usually due to reuse (other reasons discussed in the Part 1). It's pretty natural to say "well, services are reusable, and here we have a process that needs to use them. So, use them."

However, there's a natural architecture incompatibility that has to be address. Web services are, by nature, remote. Batch process are oriented around local processing of large quantities of data/transactions, by marshalling large amounts of local resources to maximize the throughout. Here's an example to explain better...

In the ancient days of programming, say before 1990 or so, the majority of systems were built using local files. Relational databases didn't exist yet, or were just gaining commercial viability (the first commercial RDBMS was released for the mainframe in 1978, and Oracle as a company formed in 1979). Data was accessed locally using a variety of file indexing schemes (ISAM). Programmers were confident of being able to access 1 record in 1 file very quickly, and the data handling patterns were designed around that mode of access. (Code tables were buffered, step-through processes the norm.)

In such an access model, moving the data - the tables - away from local access is a nightmare. The model is based on very speedy access to each 1 record, the indexing routines being a library loaded into the local program requiring very speedy reading through index files or index header records. If the file access speed drops from 3ms to 6ms, the program might literally run twice as slow. Bad, very bad.

This model was fundementally incompatible with relational databases. In relational databases, the whole data structure moves off the local machine to a "database server" and is optimized around a multi-file single read operation that returns many records in a single request. The reason this is critical is because of the overhead...

Marshall, send, and receive a network request: 30ms
Create a request string, interpret, process: 20ms
Interpret multi-record answer: 10ms

That's 60ms, 20 times SLOWER than traditional local indexed file access. If we attach the average program of 1980 to a relational database on the network, performance falls through the floor as each record request becomes 20 times slower.

But when we adjust the data handling pattern to request the data needed through it's relational structure, returning multiple records across multiple tables in a single data access request (a SQL call), we might get the equivalent of 100 individual file index and data table reads returned in a single call, getting 300ms of direct table access in 60ms, 5 times faster with the added advantage of dividing our processing across 2 servers.

The key was changing the data access model to take advantage of the new technology advantages.

A web service access pattern is usually designed for "online" or real-time(ish) access. Details of this in my next article.

SOA and Batch - Let's Get Technical - Part 1

Two weeks ago, Zapthink published a fluff piece Batch Processing with SOA (by Ronald Schmelzer).

The article says just two simple things:

1 - You can make batch process controls into callable services.

2 - On-demand batches ("workload automation") are the future of batch processing.

The article ignores what are the key topics of a batch SOA conversation, namely:

A - Can web services be utlized in the context of a batch process (or workflow automation or whatever you want to call it)?

B - Can an ESB provide an enviroment for a batch process, or a significant portion thereof?

In other words, is there a convergence between batch processes and SOA integration tools (web services, ESB, or even governance)? Can these respective spaces leverage one another, or are they fundementally incompatible?

This issue has come up in the context of two clients within the past 2 weeks. The first question usually asked is WHY? Why would you consider an ESB in the context of batch processing?

There's a few answers to this question:

1. The first is one of the nominal goals of SOA, reuse. Making connections is a big part of service development. Depending on the environment and technologies involved, making a connection to a given system can be a major effort in and of itself. Security considerations, gateways, protocol translations and adapters required, just making contact with the right people and receiving the right approvals. The actual labor (programming or configuring) required isn't usually that time consuming, but finding the right people to get in touch with, going through the security approval processes, getting adapters installed or gateways configured, and coordinating it all, that is time consuming.

So, you've got a service means you have a valid working connection to a given system, with a valid working transaction - business function - service. Some batch process comes along that has a new requirement for your service, why not use it and save all that setup time? (And the related value of having one code source - function basis, real reuse.)

2. The second is ESB tools are very good at data transformations - in particular at the programming of data transformations. Visual drag-and-link development and integrated converter functions make the type of data format transformations often involved in batch processes a good fit. Further reinforcing this thought is that the ESB usually has a trained group of developers just waiting to create those transformations.

(As opposed to batch developers, which are usually doing this effort as part of their primary project - which is not exlusively batch work - and therefore are less likely to have such specialized tools or, if having them - be less skilled in their use. If they have tools, it's ETL tools [Extract Transform Load] which are very good at what they do BUT tend to be significantly more expensive than ESB tools, and because of their limited use they're harder to justify.)

These reasons, and perhaps just the general Integration Compentency Centers tend to be thought of when connecting things - regardless of the processing model - are dragging many an ESB and/or integration team into implementing batch processes in the context of the Enterprise Service Bus (ESB). This, however, is usually a mistake.

Why will be covered in my next article.