Oct 21, 2014

CIO Interview–Integration (SOA / SOAP / Web Services) Impact

Some years ago I interviewed a CIO of a Fortune 500 IT vendor as part of an Integration Improvement project. His responses helped shape the goals and roadmap of the project, as business drivers and goals should always be taken into account in how one models the architecture and integration space.   The interview gives a great view of IT management business drivers in the integration space.  Company identifying information has been removed.

Question: How is integration?

-- Almost everyone at our company is an expert on Information Systems.

-- Our culture - everyone (department/division) believes their own numbers and requirements and doesn't believe anyone else's. Words like "mutual understanding" or "common terminology" didn't exist in the past.

  • The reason there is no real enterprise integration because of how the company is running. Each unit built their own systems.
  • The other units came to IT and said "buy this for us or build this for us, exactly per our requirements".
  • Result, we have a large number of systems, majority home grown because building to our own specs was (considered) easier (in the past.)

Today we want to do the opposite. We want to create best practices and show to our customers "this is the best way to manage your business". To implement the best practice by picking the best packages (vendor software) that does those processes. (Best of breed approach - but the business idea behind it is best-of-breed to implement specific target business processes, some of which may be offered as solutions to customers.)

Now we are not "developing" but coming with a solution. But there is no one company that can bring the best solutions across all the lines of business. Therefore we understand that we'll need multiple vendors to provide all the needs.

We have a variety of lines of business. And there are different requirements in the different lines. So we have to come up with the best practices for each business unit...and they don't want to bother with integration (they don't see integration are part of their business goals - interoperating with other business units is not a focus for them). But they want us (IT) to build them one integrated view (across business units) of what's going on.

The result - different packages (that we didn’t build), each from a different vendor. Some according to standards, some not [because sometimes the best solutions aren't standards compliant) and sometimes even different solutions from the same vendor are operating per different standards (a result of many vendor portfolios being built via acquisition).

The expectation is to build a common integration layer that can integrate the various tools from the different vendors….AND provide a common process view and data view that we can analyze.

...and the cost of maintaining this should go down (in comparison to today) and should continue to decrease.

So adding new systems shouldn't create a linear growth in the cost. The company wants to measure IT...what is the cost according to what it should have been by adding more and more systems? And how does it compare to the current cost of today. They want us to reduce the real maintenance cost of today even though we're adding systems and capabilities.

They want IT costs NOT TO GROW but provide additional capabilities (do more, spend the same).

How to do it? Some kind of miracle (we have to find).

Question: What's the time frame to meet these goals.

Answer: No one expects us to come up with all of this, this year. But what they do expect is new systems work should be meeting the new goals.

ROI - people have to commit to improved project impact by showing over 5 years how costs in particular areas (for example integration support) will be reduced.  In order to make decisions we need to quantify everything as best as possible...even knowing some are assumptions that you can't really quantify at this stage. So we need to build a strict ROI but we can, for example, quantify values of agility (an example I provided).

Because we're in the process of replacing many older systems, we can easily quantify major ROI returns much faster. (Meaning switching old systems to a new method might have a very long ROI period, but incorporating the new method with the implementation of a new system eliminates a "change over" cost as that's already part of the old system replacement cost.)

After we make a decision on best internal practices, the cost of new projects will automatically include the overhead of doing the interface according to the right method/pattern/etc. And it's our job to convince the managers that it has the long term value in reducing the maintenance costs.


Question: Cultural drivers?

Answer: Our main driver, being able to respond to customers needs and changing business. As a company operating in technology fields, the technology is constantly changing as is the business environment.

We must be able to integrate new things very fast, and do so without destabilizing existing systems or significantly increasing support costs.

Today the capabilities we (IT) give to the company are very low. We're not meeting the business requirements of new functionality the business demands. There is a long back log of features the business has requested that we can't provide within our available budget and resources. We need to be able to delivery fast and agile.

Aug 20, 2014

How an Open Data Feed changed Israel’s Civil Defense


Israel is under frequent, as frequent as every 10 minutes, rocket attack from Gaza.  While the Iron Dome rocket interception system has become famous as a technological marvel in the defense of the country, there are other technological marvels of note as well.

The first step of any civil defense system is getting the civilians out of the way or under cover.  Israel has a network of neighborhood bomb shelters, building bomb shelters, and (in new construction) a “hardened room” in every private residence and on every floor of every office building.  When air raids were measured in hours or tens of minutes, all of these were adequate together with a nationwide network of air raid sirens.

But modern circumstances have brought two new problems:

- Rocket attack warnings are measured in SECONDS.  Fifteen seconds in towns near border regions, and 90-120 seconds in the center of the country.

- As the country has suffered suburban sprawl, built malls and cinema mega-plexes, modern skyscrapers, joined the problems of traffic jams, and built it all with modern climate control (meaning sealed or closed windows and A/C), HEARING alarm sirens has become a problem.

Like any government agency, the Israeli Civil Defense department – a division of the Israeli army – has implemented big project approaches to these problems.  A radio based pager like messaging device… too expensive except for large businesses or office buildings (which can then manually alert tenants).  A cell based pager messaging device with digital output… with reception problems and a complicated interface requiring special software – again making it of limited use.  The newest addition, SMS messages to all cell phones from cell towers in an alert area, a massive project that required integration with all the cell phone providers but only results in a regular SMS “ding” – making it useless.

A lot of effort and a lot of money with the problem continuing to grow and the current solutions offering only limited impact.

But then something amazing happened.  The Civil Defense department public web site integrated a real time alert box onto the site.  It was unnoticed by almost everybody except for a young man in southern Israel in a community frequently targeted.  Since a web page is, by nature, open source, he looked into the page to determine where they were getting their data – their real time data of civil defense alerts for Israel.

He took the data feed, a nicely formed JSON data URL, set up a server polling it, and built an Android client.  This became the first “Code Red Israel” alert app.  Someone else contacted him and asked to use his server, and built an iPhone edition.  This was 2 years ago.

With the current conflict and the terrorists expanding their targeting to civilian cities and towns across Israel, the apps gained notoriety.  But so did the people interested in creating additional abilities, options, and clients.  And an explosion in apps and abilities has been created over the past two months.

Examples include: real time alert monitoring web pages (in Hebrew and English), extensions for Chrome, iPhone and iPad apps that offer various sounds, filtering by city, maps of alert locations, commenting to share thoughts of being targeted, Android apps to do all of the same – in Hebrew, English, or Russian (major languages used by segments of the population in Israel).  And like any app category, there’s become a competition between apps on offering the most useful features – even though most of the apps / pages / extensions do not charge or even offer ads (meaning they’re covering their development and server costs out of pocket).

Today while waiting in line at a grocery store or sitting in an office, almost everyone’s phone will go off if there’s an alert – some for only the local area, some for the whole country (as each person prefers).

There’s a key additional point.  The data feed seems to be providing the alert data seconds before the actual sirens go off.  So it is possible to get the alert up to ten seconds before the sirens actually go off!  (Depending on the speed of the monitoring service.)

So while the Civil Defense department spent years and millions on building up a technological infrastructure, their biggest success was by accidentally offering an open JSON data feed.

My personal alert project is a web 2.0 site at http://IsraelSirens.com.

Jul 18, 2014

MDM & SOA - Layer, Repurpose or Replace?

An Architect Friend sent me this extended architecture question...

I recently joined a company that provides business consulting (via many MBAs) services related to sales and marketingMost clients are large pharma companies.

In addition to consultants, there are business process outsourcing teams (offshore) that do operations (like incentive plan management, report distribution, etc.)  There is also BI/reporting group that creates BI/DW solutions (custom ones using template approach) for large clientsPlus there is an Software Development group (SD).

Over the years the Software Development group of the company created various (10+) browser-based (.NET/SQL) point-solutions/tools to help consultants (and eventually some head-quarters users) perform specific tasks. For example:

- Designing sales territories and managing the alignment of reps to territories
- Custom ETL-based tools to perform incentive calculation
- Some Salesforce-like platform for creating custom form-based apps
The applications are architected as single-tenant – with some deployment tricks to be able to deploy an “instance per client” on the web servers. The databases are isolated per client/instance.  The tools are sold as if they are part of an integrated suite, but they aren’t natively integrated and require custom integration.

There is a custom grown ETL-like tool for interconnecting the tools to each other (but not standard connections since the data models are all “flexible” and not well defined) plus Informatica and Boomi to get data from clients.  Some clients use one tool, some use 2, some use 3, etc.  Some tools are used directly by the client, but most are used by the consulting teams on behalf of the client.
Lately, there is desire to make it all “integrated” across the company (SD + BI + all else)Two main themes are emerging (even prior to me joining): “common data model” and “SOA”.  There is also the question of letting existing applications function as-is and developing new ones on a more proper architecture versus trying to evolve the existing apps.
However, the understanding of how this applies to an Enterprise looking inward on its own systems and trying to align them, versus Independent Software Vendor (ISV) looking to build software for other Enterprises did not yet sink in… and concepts are being confused…
The tension between a standardized productized software versus customized (consulting company) software solution is not yet resolved.

I wanted to ask if you had experience in environments were an ISV was trying to define the enterprise architecture of their solutions for customers versus their own internal architecture.

Are there any case-studies or resources you could point me to get some reference architecture examples?

I usually do not like “next gen” approaches, but I am not seeing much potential in evolution of existing assets into an integrated state (they have a lot of “baggage” and features that were there but don’t play nicely with “integrated” world-view).

Here's my answer:

I wanted to ask if you had experience in environments were an ISV was trying to define the enterprise architecture of their solutions for customers versus their own internal architecture.

-        No, though I have built integration competency centers and projects that were providing service environments across very large scale enterprises of disparate divisions.

Are there any case-studies or resources you could point me to get some reference architecture examples?

-        Not that I know of.  I'm not much of a fan of such studies, mostly because the requirements and details are always highly complex, and those details directly affect the approaches taken.  Studies and reference architectures provide a nice high level structure – but the more you try to keep to them in the details the less effective they are (as they are mismatched to the exact situation).  I use bits of Togaf-9 from opengroup.org, bits of CBDI from Everware http://www.everware-cbdi.com/, and various tidbits picked up from Zapthink (though every few years they discuss the benefits of yet another framework).

I usually do not like “next gen” approaches, but I am not seeing much potential in evolution of existing assets into an integrated state (they have a lot of “baggage” and features that were there but don’t play nicely with “integrated” world-view).

-        It's a pretty standard problem: how to balance between what is, how it can be extended / expanded / reused, and what should be replaced / redeveloped / moved up to a new generation of technology, pattern and features.

- The problem you describe sounds like it crosses between SOA / integration and MDM (master data management).  Sometimes a SOA façade can provide an MDM operational model, with composite services doing multi-system queries, combining or rationalizing the result, and presenting single meaningful "views".  In other cases it's the SOA abilities enabling MDM to do it's job, which often involves signification bi-directional synchronization.

- The MDM tools tend to be heavy, and the business and systems analysis work (which system wins when data is in conflict, for example) is a major portion of the success or failure.

- That said, IF you are only trying to get views of the data, I am hearing reports of good success with some of the easier BigData tools (such as MongoDB).  Success meaning they are able to develop and deploy meaningful business results in months, whereas MDM and big integration SOA projects almost always take over a year.

Aug 26, 2013

NSA Tapping and Cloud Computing

th (4)Recent revelations from the U.S. have informed the public that conspiracy theorist fantasy's are all too real, the U.S. National Security Agency (NSA) has been installing taps at key Internet points to absorb vast quantities of email and Internet traffic.

As IT professionals, publically available information (no inside information or secret information is utilized in the preparation of this article) and an understanding of the Internet as a series of routers and servers, we understand that “taping key Internet points” means copying streams of Internet traffic via router configurations and having monitoring software installed on email servers and the like (directing copies to the government monitoring servers).

The NSA isn’t secretly tapping into some sort of vast Internet cable bundles.  Rather they’re walking into AT&T, Google, Yahoo, Microsoft, Internet backbone and primary service providers, and installing software on their routers and servers as well as installing NSA receiving servers on their networks and premises. 

A few months ago ZapThink published an article questioning why the move into Cloud Computing via Public Cloud Vendors has been relatively slow…

Cloud Computing: Rethinking Control of IT : Jason Bloomberg, April 24, 2013

In my role as a globetrotting Cloud consultant, I continue to be amazed at how many executives, both in IT and in the lines of business, still favor Private Clouds over Public. These managers are perfectly happy to pour money into newfangled data centers (sorry, “Private Clouds”), even though Amazon Web Services (AWS) and its brethren are reinventing the entire world of IT.

Their reason? Sometimes they believe Private Clouds will save them money over the Public Cloud option. No such luck: Private Clouds are dreadfully expensive to build, staff, and manage, while Public Cloud services continue to fall in price. Others point to security as the problem. No again. OK, maybe Private Clouds will give us sufficient elasticity? Probably not. Go through all the arguments, however, and they’re still dead set on building that Private Cloud. What gives?

The true reason for this stubbornness, of course, is the battle over control…

Why do (IT) executives crave control so badly? Two reasons: risk mitigation and differentiation. If that piece of technology is outside your control, then perhaps bad things will happen: security breaches, regulatory compliance violations, or performance issues, to name the scariest.

(The article continues why this isn’t really true and mitigated through SLA agreements and gives you the advantage of separating the responsibility and control.)

ZapThink misses the major point in my mind. 

There’s been a variety of Cloud Computing articles superficially discussing potential legal complications of a corporation having some of their business data in a different legal or national jurisdictions from their business.  And we’ve seen some practical challenges of major US service providers being hit with EU privacy standards violations (for example).  What, for example, would happen if a customer sued the Cloud Vendor for “deletion rights” (an EU data right) for a EU customer who had signed up with a US Internet service provider that happened to use an EU based Cloud Resource Vendor for their storage?  IT executives naturally shudder at the business / legal complexity of data crossing state / national / international borders.

With the NSA monitoring revelations, we see much worse concerns.

In the case above, I could end up in a lawsuit outside my jurisdiction.  If I’m a small company, this could be catastrophic (if I’m a large corporation, only ridiculously expensive).  But at least all I have is a legal risk.  As long as the Cloud Resource or Platform Vendor is living up to their contractual responsibilities and technical features, my data and computing results remain private and controlled – though now I have the additional party of the Cloud Vendor in the mix.

With NSA monitoring, the Cloud Vendor may be forced (or coerced or tempted with financial payments) to provide monitoring access without being permitted to notify me (fully legal).  My company would thereby have no legal recourse to attempt to protect our data because we wouldn’t even know such monitoring is occurring.  (In instances of the monitoring revelations, it’s also become clear that the NSA is doing so with “secret court” orders which prevent the service providers / vendors from letting anyone know such monitoring is being requested or occurring.)

Suddenly the controlling or paranoid IT executive is looking smart. 

We now have established the fact that if your data leaves your premises, it may be secretly tapped / copied / monitored, and you’ll likely never know (thereby offering no legal recourse to challenge it).  And while we certainly want our national security resources to be able to do their jobs and provide national safety, we also know that such authority is subject to misuse – and misuse of key company data could cost millions or even billions, or put a company out of business.

When police or investigative authorities arrive with a court order, we may legally challenge it as well as trying to keep the access as narrow as possible.  Even further, it may be our IT people providing the data (so we know exactly what’s leaving and what the potential business impact is).  If the NSA is monitoring or accessing Cloud Vendors, our data is leaving without any control or even knowledge on our part.  Our business risk is potentially unlimited.

The advantages of the Cloud now carry a real risk.

For personal use, cloud services now carry the same risk.  If you’re tying your Android phone to Google account sync or your iPhone to the iCloud, your contacts are now (probably) being monitored.  How about if you’re using a Cloud backup service or online file storage (Googe Drive, Microsoft SkyDrive, etc)?  We don’t know, but with recent revelations, we’d be foolish to assume the data is being kept private from national security authorities – which included revelations that NSA employees used the service to spy on personal love interests.

If it leaves your premises and it’s not encrypted and kept encrypted at the destination, it’s only appropriate nowadays to assume it’s being monitored.  And if it is encrypted, you may warrant special attention.

Welcome to the digital age.  Your government is now online.

Aug 20, 2013

The Reality - Email Privacy or the Lack Therefore

Today a top law blogger freaked out after realizing that their emails can be read and monitored...

"The owner of (a encrypted protected email service that just shut down) tells us that he's stopped using email and if we knew what he knew, we'd stop too.  There is no way to (blog) without email. Therein lies the conundrum.  What to do?

...the simple truth is, no matter how good the motives might be for collecting and screening everything we say to one another, and no matter how "clean" we all are ourselves from the standpoint of the screeners, I don't know how to function in such an atmosphere.

I feel (unclean), knowing that persons I don't know can paw through all my thoughts and hopes and plans in my emails...  They tell us that if you send or receive an email from outside the US, it will be read.  (And many emails inside the US are accidentally picked up by those capture engines.)  If it's encrypted, they keep it for five years, presumably in the hopes of tech advancing to be able to decrypt it against your will and without your knowledge.

I hope that makes it clear why I can't continue. There is now no shield from forced exposure. …no one can feel protected enough from forced exposure any more to say anything the least bit (controversial or security related) to anyone in an email, particularly from the US out or to the US in, but really anywhere. You don't expect a stranger to read your private communications to a friend. And once you know they can, what is there to say?"

Much of the Internet and the abilities we take for granted today were never conceived to be the large world-wide network they have become today.  Email, as used today, and it’s base protocol (SMTP) are not encrypted and follow the normal network routes to get where they are going.  The term “email” is a mistake, because it’s NOT a sealed letter – it’s a open postcard (an open sheet of paper with an address on it).

This means:

- The sending post office and receiving post office can and do read the sending and received address AND the full content.

- Every network it passes through along the way (remember, the “Internet” is all cross connected networks”) can also capture and read the sending and receiving addresses AND the full content.  (Today the average number of networks things pass through are 8-16.)

- All the little people along the way that operate all the stuff that makes this happen, such as system administrators, database administrators, network administrators, have all the tools in front of them every day as part of their jobs to read any of this stuff they want to.

That’s been the case from day 1 with Internet email.  NO public free email service NOR any service offered by any Internet ISP encrypts email traffic or storage.  It’s all traveling and sitting on open unprotected readable pieces of paper (so to speak).  The ONLY protection has been “privacy policies” of the companies and them being shamed (and losing customers) if they didn’t provide a reasonable semblance of isolation of your emails.

Even so, YOU have voluntarily given them the right to “paw through your data” since day one, and much worse!

- Your emails are being automatically scanned by Google / Yahoo / Microsoft Hotmail-Outlook.com, "to serve you ads and understand their customers."  Note the ads displayed on the email pages are context sensitive to the email being read, this doesn’t happen by magic but by your email being auto-scanned for keywords.

- Your phone location (meaning your body’s location) is being sent to Google and Apple, "to provide better mapping services and location based app responses (letting an app know where you are to tell you about something near by)."  You can’t turn on GPS / Location Services without giving them permission to track you moment by moment. 

- If that’s not enough, the cellphone service provider logs every call, who you called or who called you (number on both ends – which you see on your phone bill) AND the approximate location of your phone when the call began (tracked by which cell tower you connected to and the strength of your signal, meaning how far from the tower you are – combined with where the other towers are near that one this can be used to narrow your location to within 1/10 of a mile).

- Of course, every SMS is logged (source and destination) as well.

- Your office computer is tracking what web sites you browse, in some cases what programs you are running, in some cases even how fast and what you are typing, and reporting it to your IT department and/or your boss.  If you are visiting whatever your office considers improper sites, the IT department and boss are being alerted.

- Your office emails are being scanned for improper words and phrases (sexual, harassing, racist, violent, threatening), alerting the IT department and your boss if there's a hit.  They may also be scanned to see if you are sending out company proprietary information.

- Anytime you access any Google or Yahoo or Microsoft service, or for that matter any web site that chooses to collect the data, they know from where you did it (by IP address – which if correlated with the ISP will give a physical location down to where the connecting router is or actual cell location) and from what type of equipment you did it.  Are you at home or in the office, on a desktop workstation, laptop, iPad or Galaxy phone, running Windows or a Mac or iOS or Android, which browser, and what size screen.  This information can be obscured (with special fully legal utilities) except for the IP address, which can be via a VPN or anonymous service (also fully legal).  But few people do so (and because few people do so, there’s a suspicion about people who do.)

Nothing is new with any of this.  These are all a basic part of the operating model of all of these services from day one.  If you wish to avoid it, you have to avoid using these services.  (There are a few alternatives that theoretically offer protection, privacy or lack of logging – naturally such services are costly.)

However, there has been a FAÇADE of privacy and protection.  And there has been a general thought that given the huge amount of such data, there may be anonymity and protection through obscurity.  The façade has always been false – the data has always been easily accessible.  And if it exists, police or courts or other authorities will go after it if it’s in their interest to do so.  (There are companies who intentionally limit their logs of such things to 2 weeks or 3 months, to prevent them being called into any court case.)

The big deal now is that the U.S. government has joined the game and is, via various means, grabbing or monitoring some of the data above.  And with new levels of computer processing power and “BigData” huge data set analysis techniques, Big Brother isn’t occasionally poking in to take a look – he’s constantly monitoring.

Given that the companies involved have basically being doing the same since day one, the primary change is that the façade has been blow away.

There are ways to reduce exposure without completely eliminating use of such services.  But if you’re using Google or Microsoft or Facebook (or other such web sites or phone services), assume EVERYTHING you do is completely 100% public.  Email, chat, SMS, sites you browse, pictures you share – it’s all being tracked … correlated, profiled.

Being we talk of BigData capabilities and BAM – Business Activity Monitoring, we in the IT industry shouldn’t be surprised.

Blog Widget by LinkWithin