International Clouds

Cloud Computing is clearly well into the hype cycle. Never being one to miss some good hype, I've been paying attention to advise my clients on whether and when to pay attention to this rising option.

One of the big factors in consideration is my current location... I'm working outside the U.S. My local country is heavily wired, offers high speed broadband (2-10mbit) to 100% of the country, 50mbit home connectivity in the major cities (100mbit next year), and even offers cell based mobile internet connectivity at 2mbit or 3mbit (competing companies). Company and office connectivity is typically equal or better.

The in-country data centers and backbones between them are very high speed. Ping times from in-country web sites typically run 30ms across the various data centers, backbones and ISP's / hosting sites. All in all, compared to the US that's seriously high speed.

Yet, all of that is in-country. The one area where the local internet infrastructure is weak is to-the-US bandwidth. Typical ping times to the US run 250ms at non-peak times, and 400-600ms during peak times. This is not only a delay problem but also a bandwidth problem. (It's 2-3 times better to Western Europe, but that's still not so good.)

Some of this is because the local internet users greatly appreciate American media, music, TV shows, movies, video clips, video chatting and Skype'ing people in the US. High bandwidth activities that push capacity utilization on the international links way up during those evening hours when everyone gets home from work. And since the long-haul international pipes are very expensive, the local backbone providers keep the utilization high, just below the serious pain point, to justify the cost.

What's all this mean relative to cloud computing? In local IT shops, considering cloud resources within the local (national) internet loop may be a very viable option with a very high speed local loop. Further, in theory even web 2.0 applications could utilize cloud resources to present functionality directly onto our user's desktops.

But utilizing US based resources is a much more iffy proposition. Let me not waffle there, it's not an option. Though (for example) Amazon S3 may offer a MUCH better cost:utilization ratio than renting a server for disk capacity at a local hosting center - or dropping some EMC storage into my enterprise data center (with the requisite backups and disaster-recovery site duplication), the international performance considerations make it not a viable option.

So the question for cloud computing in my current country base is...will cloud service vendors offer international clouds? Since the local size / market is about the same as a single mid-size US state, will cloud service vendors consider this market worthwhile? (Perhaps as either a test market to get facilities up to speed for larget markets, or offering franchise opportunities to local businesses that wish to offer the services. Or perhaps it leaves an opening for small (by US standards) start-ups to offer and develop services which they can then try to push into the bigger US or European markets?)

In any case, two recent topics - one on whether a "private" cloud / inside company cloud really qualifies as a cloud and noting that Google is getting into the data link business both bear on this question.

Along the SOA Tipping Point

When Anne Thomas Manes (of the Burton Group) famously declared in January of 2009 that "SOA is dead", everyone rushed around to understand what she meant. Being that a year later she's still giving presentations on SOA Governance and other SOA topics, clearly she didn't mean that SOA was a failed technology. (There are plenty of IT technologies that come along with much hype but never quite translate into practical usage patterns or benefits for Enterprise IT, and therefore fade away as quickly as they arrived.)

Today when I'm talking with IT organizations the majority are doing some level of SOA. So clearly SOA has moved along the adoption curve. The innovators struggled with it but got and touted their early advantage. The early adopters picked it up and integrated it into their enterprise IT model.

We're clearly past even the early majority and a good way into the late majority. The late majority are organizations that 2 years ago weren't considering SOA, organizations that have little drive to change or business requirements that stability be a strong priority, i.e. utility companies, government IT, or back office military IT. Today all these organizations are either beginning to use SOA technologies, run actual SOA projects, or find themselves in a bottom-up situation where SOA tech has been implemented at the lower level in some projects and need to begin to rationalize and manage the results.

As an example of this, I visited a customer who's mainframe department was very enthusiastically talking about how they'd exposed over 100 web services directly from CICS [as IBM's CICS 3.1 and Enterprise COBOL allow relatively quick and easy exposure of transactions as services]. A year earlier these same people were shaking their heads over newfangled XML, and mainframe connectivity was a carefully managed process using MQ or other various gateway tools. When the mainframe guys are excited about web services and exposing modular functionality, it's a clear sign SOA has passed into the Late Majority.

However, while most IT shops are now doing some level of SOA, few of them have embraced or implemented the methodology, IT management, and IT-business interaction changes that are necessary to gain most of the benefits of SOA. SOA has succeeded as a technology set but the accompanying people changes have not penetrated. And most IT organizations are losing their SOA benefits because of this.

SOA technology without methodology is a net loser. (Not the methodology of low level integration pattern methods, rather the methodologies that bring changes to high level architecture, process modeling, IT-business interaction, and IT management methods.) Everyone (mostly) is doing it because the need to live-connect systems and the practical spread of the business processes across systems must be handled - and SOA technology offers a relatively easy way (from a pure connectivity and composition standpoint) of doing so. But (mostly) everyone is also complaining about the new problems it brings. (The problems by themselves are a good topic for a separate article.)

For SOA to succeed within the organization, the organization must adjust it's software development lifecycle (SDLC) and IT management patterns to accomodate and MAXIMIZE the new technology pattern. To date, few enterprise IT shops have done so.

IBM expresses this in their Rational Rules for Software Development, in typical engineering speak...

Rational System Engineering
The Six Principles of System Development
Rule #6

Development Organization should Reflect Product Architecture.

“Technology dictates a change in architecture, and organizations that do not adapt experience a loss of productivity and effectiveness…”

How should the organization adapt? Not having seen a simple guide to do so, I think that will be my next series of articles.

"How to Adapt your IT Enterprise to Get Positive SOA Value". This is an IT management problem, and IT architecture problem, a project management problem, and a SDLC problem.

Don't Want No Stinking Data Standards!

Every IT system has an internal data and object model. This model is matched with a relational model that turns into the supporting database. When systems interface, bridging the data & object model from one system to another is a significant effort. Signifcant effort means up to 50% of the integration effort!

The historical method is for the exposing system to slightly simplify and expose it’s model, and the receiving system to create significant manual code to transform from the exposed model to it’s internal model.

Over the past ten years or so, a large number of industry IT consortiums have been working to create industry data+transaction standards. These standards are designed to be highly interoperable, covering each business process and data objects for the particular industry. They appear rather complex (they are rather complex) as they’re designed to cover all aspects of an industry object with necessary flexibility. However, literally years of thought have gone into covering their respective industries with enough flexibility to cover individual company variation yet complete enough to have the full range of business operations.

Many an IT shop struggles with agreeing upon an integration data standard and then spends year after year adjusting, expanding, and fixing the standard to meet their changing company, regulatory and industry needs. Other shops avoid the issue complete, with every exposed web service being a unique operation. This leads to "get customer" returning similar but formatted wildly differently results from different systems in the same shop. Further, throw in an ERP or CRM and some vertical industry specific tools, and you've got data soup...each data/interface format providing a unique perspective to it's given application.

While the output may be XML and therefore somewhat human readable, without it's appilcation and business context it's not fully understandable. This means to understand that web service exposed 4 years ago you need either:

a. The guy or team who wrote it. (If they're still around.) - or -
b. The well written documentation. (If it exists. Ha!) - or -
c. To dig into the application and code context to figure out what it is.

Let's do some algebra on that equation. C = (Time + Expense * lots of it) / each project that encounters the problem. Since this equation is divided by each project encountering the problem, the actual expense and impact often doesn't bubble up as a major problem.

To avoid this significant embedded expense in the integration process, I strongly recommend spending the time to select an appropriate industry standard that matches the company’s needs. Standardizing the "integration" or "web service" data format, preferably with an industry standard, is a highly recommended best practice.

It does have initial overhead, often significant initial overhead to come up to speed and gain understanding of the particular standard's object model and flexibility of the format. Yet there is no question that the payback as interfaces start to standardize across the company becomes significant (and the ROI loss for not doing so real).

Further, vendor products (and outsourcing vendor providers) are arriving with industry standards in place!

Some example standards:

• ACORD: the Insurance Industry Standard
• OAGIS: eCommerce, Logistics, CRM
• B2MML: Manufacturing & Automation
• Rossettanet: Engineering & Product Info

Some organizations try to standardize on a proprietary vendor format. What's wrong with a nice SAP or Oracle or Baan format? In a word, lock-in. Lock-in to a particular vendor’s data model and system mindset. Portability and reusability is not the vendor's goals.

Standard Enterprise Entities will enhance your integration reuse and agility significantly. Using Industry Standard Enterprise Entities is a great fit for most companies.

The SOA of Twitter vs Buzz

Twitter, in and of itself, is pretty stupid. Or to be a bit more analytical and precise, as a web version of a cell phone Short Message Service (aka SMS), it's incredibly limited functionality provides little room for practical value. (It actually started as a way to reflect a message from one cell phone out to a group of friends on their cell phones, via a web based facility.)

By itself, Twitter is a very limited tool that would be permanently consigned to a narrow audience as a small utility function. You can easily think of 5-10 such services that you've tried and (probably) discarded, a few of which you keep using for their narrow purpose.

So what differentiates Twitter from hundreds of other narrow utilities that came (and mostly went)? In a word, a Service Oriented Interface.

Twitter started from day 1 exposing a simple straightforward Web Service interface. Even further, they never offered a feature without simultaneously exposing it. In other words, there is a Twitter "Engine" and there is a Twitter Web Site. The Twitter web site is just another client of the engine, and the engine is fully open for use by anyone.

Further, Twitter never pushed their web site as the primary place to access Twitter, nor even the best place. Their own web site advertises the API and the many products others have created to access Twitter.

So while Twitter had a very narrow set of capabilities, thousands of developers thought of ways to use and leverage Twitter "if only it had ....", and then created the missing functionality. The result was URL shorteners (aka bit.ly), fancy clients (aka Tweetdeck), clients for other devices (iPhone, Blackberry, etc), interfaces into blogging platforms for "notifications", even games (such as Foursquare).

All of this is a perfect example of well defined SOA utility services at the right level of granularity. Simple and easy to use interfaces offering discrete bits of function. Not too discrete that I have to figure out how to put multiple together to work, each completes a "business function" by itself. But not too high level that I'm locked into the vendor's business model.

The result has been an explosive growth of functionality layered upon Twitter.

Now lets take a look at Google Buzz. Google comes along focused on the friend/follow model of Twitter and extends it by automatically creating those relationships using a new algorithm that scans your Gmail interactions to determine relationships. They then offer a richer and more capable client, which can accept longer messages, pictures and/or video. They tie it in to all their providing platforms...if I blog on Blogger it shows up on Buzz, if I read it in Reader and "like" it, it shows up on Buzz. My Tweets can reflect to Buzz, any pictures posted on Flickr or Picasa are buzz'd.

So the client presentation and the native interfacing are worlds ahead of Twitter. But, what Twitter offers for client interaction may be only 5% of what one can do with Twitter, due to all the leveraged API developed tools and features by others. By offering a complete and easy to use SOA interface set from day one, Twitter reached hundreds to thousands of times it's own team's development capacity by allowing full public use of the engine for development of other tools.

Buzz, on the other hand, is currently offering a very limited API - with the API focused on reading the (Buzz stream) data. Further, the API is oriented around public yet somewhat complex standards. "Update feeds are returned in the standard Atom Syndication Format (RFC 4287). Update feeds may contain references to rich media elements, such as audio, images, or video. When available, these media elements will be exposed via the MediaRSS extension..." (Not to worry, it is possible to link an external source into Buzz if "sites are publicly associated with a Google profile, to associate a new feed with a Google Buzz account, the owner of the external site must point back to the Google profile (directly or by association). To make this claim use XFN markup. Site feeds may be published in either the standard Atom Syndication Format, or in many of the variants of the RSS format.")

To summarize Google's position, "Over the next several months Google Buzz will introduce an API for developers, including full/read write support for posts with the Atom Publishing Protocol, rich activity notification with Activity Streams, delegated authorization with OAuth, federated comments and activities with Salmon, distributed profile and contact information with WebFinger, and much, much more."

Well, as a highly technical developer I appreciate a rich feature set and the ability to interface deeply. However, I've worked with some lightweight web site developers and watched them integrate Twitter into their site (not a Twitter widget but integrate to publish site updates as Twitter tweets) in 3-5 lines of PHP code. And frankly I greatly appreciate being able to drop in functionality like Twitter in a few lines of code.

The fact that the Twitter API is so simple to understand and therefore easy to use is the reason that there are tens and hundreds of already developed features which move Twitter from an API based model to a widget or 3rd party utility model.

And in my evalation it is the reason that Buzz will be unsuccessful. If features in the basic vendor web presentation were the success or failure in this space, Twitter would have died long ago. Even providing a better micro-blogging/short message service engine doesn't make a difference.

Twitter's success is it's easy to use SOA web services. And Google, being overly technically oriented, completely missed that point.

There's an important lesson here for Enterprise IT and enterprise architecture. Quick and easy to use tops a very well architected full featured but hard to understand and use API. And when quick and easy is also well architected and powerful, the value is easy to see and the reuse multiplies.