Aug 20, 2014

How an Open Data Feed changed Israel’s Civil Defense


Israel is under frequent, as frequent as every 10 minutes, rocket attack from Gaza.  While the Iron Dome rocket interception system has become famous as a technological marvel in the defense of the country, there are other technological marvels of note as well.

The first step of any civil defense system is getting the civilians out of the way or under cover.  Israel has a network of neighborhood bomb shelters, building bomb shelters, and (in new construction) a “hardened room” in every private residence and on every floor of every office building.  When air raids were measured in hours or tens of minutes, all of these were adequate together with a nationwide network of air raid sirens.

But modern circumstances have brought two new problems:

- Rocket attack warnings are measured in SECONDS.  Fifteen seconds in towns near border regions, and 90-120 seconds in the center of the country.

- As the country has suffered suburban sprawl, built malls and cinema mega-plexes, modern skyscrapers, joined the problems of traffic jams, and built it all with modern climate control (meaning sealed or closed windows and A/C), HEARING alarm sirens has become a problem.

Like any government agency, the Israeli Civil Defense department – a division of the Israeli army – has implemented big project approaches to these problems.  A radio based pager like messaging device… too expensive except for large businesses or office buildings (which can then manually alert tenants).  A cell based pager messaging device with digital output… with reception problems and a complicated interface requiring special software – again making it of limited use.  The newest addition, SMS messages to all cell phones from cell towers in an alert area, a massive project that required integration with all the cell phone providers but only results in a regular SMS “ding” – making it useless.

A lot of effort and a lot of money with the problem continuing to grow and the current solutions offering only limited impact.

But then something amazing happened.  The Civil Defense department public web site integrated a real time alert box onto the site.  It was unnoticed by almost everybody except for a young man in southern Israel in a community frequently targeted.  Since a web page is, by nature, open source, he looked into the page to determine where they were getting their data – their real time data of civil defense alerts for Israel.

He took the data feed, a nicely formed JSON data URL, set up a server polling it, and built an Android client.  This became the first “Code Red Israel” alert app.  Someone else contacted him and asked to use his server, and built an iPhone edition.  This was 2 years ago.

With the current conflict and the terrorists expanding their targeting to civilian cities and towns across Israel, the apps gained notoriety.  But so did the people interested in creating additional abilities, options, and clients.  And an explosion in apps and abilities has been created over the past two months.

Examples include: real time alert monitoring web pages (in Hebrew and English), extensions for Chrome, iPhone and iPad apps that offer various sounds, filtering by city, maps of alert locations, commenting to share thoughts of being targeted, Android apps to do all of the same – in Hebrew, English, or Russian (major languages used by segments of the population in Israel).  And like any app category, there’s become a competition between apps on offering the most useful features – even though most of the apps / pages / extensions do not charge or even offer ads (meaning they’re covering their development and server costs out of pocket).

Today while waiting in line at a grocery store or sitting in an office, almost everyone’s phone will go off if there’s an alert – some for only the local area, some for the whole country (as each person prefers).

There’s a key additional point.  The data feed seems to be providing the alert data seconds before the actual sirens go off.  So it is possible to get the alert up to ten seconds before the sirens actually go off!  (Depending on the speed of the monitoring service.)

So while the Civil Defense department spent years and millions on building up a technological infrastructure, their biggest success was by accidentally offering an open JSON data feed.

My personal alert project is a web 2.0 site at

Jul 18, 2014

MDM & SOA - Layer, Repurpose or Replace?

An Architect Friend sent me this extended architecture question...

I recently joined a company that provides business consulting (via many MBAs) services related to sales and marketingMost clients are large pharma companies.

In addition to consultants, there are business process outsourcing teams (offshore) that do operations (like incentive plan management, report distribution, etc.)  There is also BI/reporting group that creates BI/DW solutions (custom ones using template approach) for large clientsPlus there is an Software Development group (SD).

Over the years the Software Development group of the company created various (10+) browser-based (.NET/SQL) point-solutions/tools to help consultants (and eventually some head-quarters users) perform specific tasks. For example:

- Designing sales territories and managing the alignment of reps to territories
- Custom ETL-based tools to perform incentive calculation
- Some Salesforce-like platform for creating custom form-based apps
The applications are architected as single-tenant – with some deployment tricks to be able to deploy an “instance per client” on the web servers. The databases are isolated per client/instance.  The tools are sold as if they are part of an integrated suite, but they aren’t natively integrated and require custom integration.

There is a custom grown ETL-like tool for interconnecting the tools to each other (but not standard connections since the data models are all “flexible” and not well defined) plus Informatica and Boomi to get data from clients.  Some clients use one tool, some use 2, some use 3, etc.  Some tools are used directly by the client, but most are used by the consulting teams on behalf of the client.
Lately, there is desire to make it all “integrated” across the company (SD + BI + all else)Two main themes are emerging (even prior to me joining): “common data model” and “SOA”.  There is also the question of letting existing applications function as-is and developing new ones on a more proper architecture versus trying to evolve the existing apps.
However, the understanding of how this applies to an Enterprise looking inward on its own systems and trying to align them, versus Independent Software Vendor (ISV) looking to build software for other Enterprises did not yet sink in… and concepts are being confused…
The tension between a standardized productized software versus customized (consulting company) software solution is not yet resolved.

I wanted to ask if you had experience in environments were an ISV was trying to define the enterprise architecture of their solutions for customers versus their own internal architecture.

Are there any case-studies or resources you could point me to get some reference architecture examples?

I usually do not like “next gen” approaches, but I am not seeing much potential in evolution of existing assets into an integrated state (they have a lot of “baggage” and features that were there but don’t play nicely with “integrated” world-view).

Here's my answer:

I wanted to ask if you had experience in environments were an ISV was trying to define the enterprise architecture of their solutions for customers versus their own internal architecture.

-        No, though I have built integration competency centers and projects that were providing service environments across very large scale enterprises of disparate divisions.

Are there any case-studies or resources you could point me to get some reference architecture examples?

-        Not that I know of.  I'm not much of a fan of such studies, mostly because the requirements and details are always highly complex, and those details directly affect the approaches taken.  Studies and reference architectures provide a nice high level structure – but the more you try to keep to them in the details the less effective they are (as they are mismatched to the exact situation).  I use bits of Togaf-9 from, bits of CBDI from Everware, and various tidbits picked up from Zapthink (though every few years they discuss the benefits of yet another framework).

I usually do not like “next gen” approaches, but I am not seeing much potential in evolution of existing assets into an integrated state (they have a lot of “baggage” and features that were there but don’t play nicely with “integrated” world-view).

-        It's a pretty standard problem: how to balance between what is, how it can be extended / expanded / reused, and what should be replaced / redeveloped / moved up to a new generation of technology, pattern and features.

- The problem you describe sounds like it crosses between SOA / integration and MDM (master data management).  Sometimes a SOA façade can provide an MDM operational model, with composite services doing multi-system queries, combining or rationalizing the result, and presenting single meaningful "views".  In other cases it's the SOA abilities enabling MDM to do it's job, which often involves signification bi-directional synchronization.

- The MDM tools tend to be heavy, and the business and systems analysis work (which system wins when data is in conflict, for example) is a major portion of the success or failure.

- That said, IF you are only trying to get views of the data, I am hearing reports of good success with some of the easier BigData tools (such as MongoDB).  Success meaning they are able to develop and deploy meaningful business results in months, whereas MDM and big integration SOA projects almost always take over a year.

Aug 26, 2013

NSA Tapping and Cloud Computing

th (4)Recent revelations from the U.S. have informed the public that conspiracy theorist fantasy's are all too real, the U.S. National Security Agency (NSA) has been installing taps at key Internet points to absorb vast quantities of email and Internet traffic.

As IT professionals, publically available information (no inside information or secret information is utilized in the preparation of this article) and an understanding of the Internet as a series of routers and servers, we understand that “taping key Internet points” means copying streams of Internet traffic via router configurations and having monitoring software installed on email servers and the like (directing copies to the government monitoring servers).

The NSA isn’t secretly tapping into some sort of vast Internet cable bundles.  Rather they’re walking into AT&T, Google, Yahoo, Microsoft, Internet backbone and primary service providers, and installing software on their routers and servers as well as installing NSA receiving servers on their networks and premises. 

A few months ago ZapThink published an article questioning why the move into Cloud Computing via Public Cloud Vendors has been relatively slow…

Cloud Computing: Rethinking Control of IT : Jason Bloomberg, April 24, 2013

In my role as a globetrotting Cloud consultant, I continue to be amazed at how many executives, both in IT and in the lines of business, still favor Private Clouds over Public. These managers are perfectly happy to pour money into newfangled data centers (sorry, “Private Clouds”), even though Amazon Web Services (AWS) and its brethren are reinventing the entire world of IT.

Their reason? Sometimes they believe Private Clouds will save them money over the Public Cloud option. No such luck: Private Clouds are dreadfully expensive to build, staff, and manage, while Public Cloud services continue to fall in price. Others point to security as the problem. No again. OK, maybe Private Clouds will give us sufficient elasticity? Probably not. Go through all the arguments, however, and they’re still dead set on building that Private Cloud. What gives?

The true reason for this stubbornness, of course, is the battle over control…

Why do (IT) executives crave control so badly? Two reasons: risk mitigation and differentiation. If that piece of technology is outside your control, then perhaps bad things will happen: security breaches, regulatory compliance violations, or performance issues, to name the scariest.

(The article continues why this isn’t really true and mitigated through SLA agreements and gives you the advantage of separating the responsibility and control.)

ZapThink misses the major point in my mind. 

There’s been a variety of Cloud Computing articles superficially discussing potential legal complications of a corporation having some of their business data in a different legal or national jurisdictions from their business.  And we’ve seen some practical challenges of major US service providers being hit with EU privacy standards violations (for example).  What, for example, would happen if a customer sued the Cloud Vendor for “deletion rights” (an EU data right) for a EU customer who had signed up with a US Internet service provider that happened to use an EU based Cloud Resource Vendor for their storage?  IT executives naturally shudder at the business / legal complexity of data crossing state / national / international borders.

With the NSA monitoring revelations, we see much worse concerns.

In the case above, I could end up in a lawsuit outside my jurisdiction.  If I’m a small company, this could be catastrophic (if I’m a large corporation, only ridiculously expensive).  But at least all I have is a legal risk.  As long as the Cloud Resource or Platform Vendor is living up to their contractual responsibilities and technical features, my data and computing results remain private and controlled – though now I have the additional party of the Cloud Vendor in the mix.

With NSA monitoring, the Cloud Vendor may be forced (or coerced or tempted with financial payments) to provide monitoring access without being permitted to notify me (fully legal).  My company would thereby have no legal recourse to attempt to protect our data because we wouldn’t even know such monitoring is occurring.  (In instances of the monitoring revelations, it’s also become clear that the NSA is doing so with “secret court” orders which prevent the service providers / vendors from letting anyone know such monitoring is being requested or occurring.)

Suddenly the controlling or paranoid IT executive is looking smart. 

We now have established the fact that if your data leaves your premises, it may be secretly tapped / copied / monitored, and you’ll likely never know (thereby offering no legal recourse to challenge it).  And while we certainly want our national security resources to be able to do their jobs and provide national safety, we also know that such authority is subject to misuse – and misuse of key company data could cost millions or even billions, or put a company out of business.

When police or investigative authorities arrive with a court order, we may legally challenge it as well as trying to keep the access as narrow as possible.  Even further, it may be our IT people providing the data (so we know exactly what’s leaving and what the potential business impact is).  If the NSA is monitoring or accessing Cloud Vendors, our data is leaving without any control or even knowledge on our part.  Our business risk is potentially unlimited.

The advantages of the Cloud now carry a real risk.

For personal use, cloud services now carry the same risk.  If you’re tying your Android phone to Google account sync or your iPhone to the iCloud, your contacts are now (probably) being monitored.  How about if you’re using a Cloud backup service or online file storage (Googe Drive, Microsoft SkyDrive, etc)?  We don’t know, but with recent revelations, we’d be foolish to assume the data is being kept private from national security authorities – which included revelations that NSA employees used the service to spy on personal love interests.

If it leaves your premises and it’s not encrypted and kept encrypted at the destination, it’s only appropriate nowadays to assume it’s being monitored.  And if it is encrypted, you may warrant special attention.

Welcome to the digital age.  Your government is now online.

Aug 20, 2013

The Reality - Email Privacy or the Lack Therefore

Today a top law blogger freaked out after realizing that their emails can be read and monitored...

"The owner of (a encrypted protected email service that just shut down) tells us that he's stopped using email and if we knew what he knew, we'd stop too.  There is no way to (blog) without email. Therein lies the conundrum.  What to do?

...the simple truth is, no matter how good the motives might be for collecting and screening everything we say to one another, and no matter how "clean" we all are ourselves from the standpoint of the screeners, I don't know how to function in such an atmosphere.

I feel (unclean), knowing that persons I don't know can paw through all my thoughts and hopes and plans in my emails...  They tell us that if you send or receive an email from outside the US, it will be read.  (And many emails inside the US are accidentally picked up by those capture engines.)  If it's encrypted, they keep it for five years, presumably in the hopes of tech advancing to be able to decrypt it against your will and without your knowledge.

I hope that makes it clear why I can't continue. There is now no shield from forced exposure. …no one can feel protected enough from forced exposure any more to say anything the least bit (controversial or security related) to anyone in an email, particularly from the US out or to the US in, but really anywhere. You don't expect a stranger to read your private communications to a friend. And once you know they can, what is there to say?"

Much of the Internet and the abilities we take for granted today were never conceived to be the large world-wide network they have become today.  Email, as used today, and it’s base protocol (SMTP) are not encrypted and follow the normal network routes to get where they are going.  The term “email” is a mistake, because it’s NOT a sealed letter – it’s a open postcard (an open sheet of paper with an address on it).

This means:

- The sending post office and receiving post office can and do read the sending and received address AND the full content.

- Every network it passes through along the way (remember, the “Internet” is all cross connected networks”) can also capture and read the sending and receiving addresses AND the full content.  (Today the average number of networks things pass through are 8-16.)

- All the little people along the way that operate all the stuff that makes this happen, such as system administrators, database administrators, network administrators, have all the tools in front of them every day as part of their jobs to read any of this stuff they want to.

That’s been the case from day 1 with Internet email.  NO public free email service NOR any service offered by any Internet ISP encrypts email traffic or storage.  It’s all traveling and sitting on open unprotected readable pieces of paper (so to speak).  The ONLY protection has been “privacy policies” of the companies and them being shamed (and losing customers) if they didn’t provide a reasonable semblance of isolation of your emails.

Even so, YOU have voluntarily given them the right to “paw through your data” since day one, and much worse!

- Your emails are being automatically scanned by Google / Yahoo / Microsoft, "to serve you ads and understand their customers."  Note the ads displayed on the email pages are context sensitive to the email being read, this doesn’t happen by magic but by your email being auto-scanned for keywords.

- Your phone location (meaning your body’s location) is being sent to Google and Apple, "to provide better mapping services and location based app responses (letting an app know where you are to tell you about something near by)."  You can’t turn on GPS / Location Services without giving them permission to track you moment by moment. 

- If that’s not enough, the cellphone service provider logs every call, who you called or who called you (number on both ends – which you see on your phone bill) AND the approximate location of your phone when the call began (tracked by which cell tower you connected to and the strength of your signal, meaning how far from the tower you are – combined with where the other towers are near that one this can be used to narrow your location to within 1/10 of a mile).

- Of course, every SMS is logged (source and destination) as well.

- Your office computer is tracking what web sites you browse, in some cases what programs you are running, in some cases even how fast and what you are typing, and reporting it to your IT department and/or your boss.  If you are visiting whatever your office considers improper sites, the IT department and boss are being alerted.

- Your office emails are being scanned for improper words and phrases (sexual, harassing, racist, violent, threatening), alerting the IT department and your boss if there's a hit.  They may also be scanned to see if you are sending out company proprietary information.

- Anytime you access any Google or Yahoo or Microsoft service, or for that matter any web site that chooses to collect the data, they know from where you did it (by IP address – which if correlated with the ISP will give a physical location down to where the connecting router is or actual cell location) and from what type of equipment you did it.  Are you at home or in the office, on a desktop workstation, laptop, iPad or Galaxy phone, running Windows or a Mac or iOS or Android, which browser, and what size screen.  This information can be obscured (with special fully legal utilities) except for the IP address, which can be via a VPN or anonymous service (also fully legal).  But few people do so (and because few people do so, there’s a suspicion about people who do.)

Nothing is new with any of this.  These are all a basic part of the operating model of all of these services from day one.  If you wish to avoid it, you have to avoid using these services.  (There are a few alternatives that theoretically offer protection, privacy or lack of logging – naturally such services are costly.)

However, there has been a FAÇADE of privacy and protection.  And there has been a general thought that given the huge amount of such data, there may be anonymity and protection through obscurity.  The façade has always been false – the data has always been easily accessible.  And if it exists, police or courts or other authorities will go after it if it’s in their interest to do so.  (There are companies who intentionally limit their logs of such things to 2 weeks or 3 months, to prevent them being called into any court case.)

The big deal now is that the U.S. government has joined the game and is, via various means, grabbing or monitoring some of the data above.  And with new levels of computer processing power and “BigData” huge data set analysis techniques, Big Brother isn’t occasionally poking in to take a look – he’s constantly monitoring.

Given that the companies involved have basically being doing the same since day one, the primary change is that the façade has been blow away.

There are ways to reduce exposure without completely eliminating use of such services.  But if you’re using Google or Microsoft or Facebook (or other such web sites or phone services), assume EVERYTHING you do is completely 100% public.  Email, chat, SMS, sites you browse, pictures you share – it’s all being tracked … correlated, profiled.

Being we talk of BigData capabilities and BAM – Business Activity Monitoring, we in the IT industry shouldn’t be surprised.

Blog Widget by LinkWithin