Making the Cloud, Microservices, and Enterprise IT work!

I attended a major vendor conference this week.  This particular vendor has a a modern line of current modern products, and also a set of 2nd generation mainframe products.  They’ve wisely bought a series of smaller product companies over the past decade to ensure their future. But the old product set continues to live on as well.  Yes they’re offering modern interfaces, web service enabling abilities and so forth.  And naturally they’re still investing in the original product set, as a cash cow should be milked as long as possible. One of the speakers I wanted to hear was speaking in the older product set track, so I made my way to that conference area.  As I entered it a generational change occurred.  EVERYONE, and I mean everyone, was 55 years old or older – with the majority seemingly very very close to retirement. There’s a clear hint from this sight for those using this product set.  We normally think of end of life for a technology as w...

I met with a major vendor technology evangelist recently (actually he called himself a technology space regional CTO) on the topic of noSQL and big data.  Now I’ll admit, like any technologist and architect, I have my areas of specialty and areas of less specific knowledge.  And as the IT industry continues to develop new technologies and approaches yearly, it’s challenging to stay up to date. So while I know relational databases well, can write good SQL and understand architecturally when to use database power functions like triggers, store procedures and the like, as well as database implementation level issues such as clustering, failover, performance, etc, I had not been paying attention to recent industry changes such as noSQL, graph databases and big data. This tech evangelist presented the concept of a particular noSQL tool, emphatically stating that the day of the relational database was over.  Relational databases are dead as a future technology.  H...

I was recently deep in an architecture process when I was asked by a team member for some help in understanding the team operating model.  Or more specifically, a team leader and architect were asking me whether they should be coding a particularly difficult area of the system. Being in architect mode, I immediately ran to the white board and architected the optimum software development team process.  Here it is…   What I was trying to describe was the role at each level, which is achieved through experience, and the flow between them.  So while an architect or team leader may be able to develop code at 3-5 times the rate of a Junior Programmer, everyone above can fill the roles below – but the reverse is not true.  So if that architect is programming, no one is architecting! And while the architect might be able to code without an architecture or design (because he or she has an image of it in his or her mind), if he (or she) does so there is no archi...

I’m sitting at CloudCon III – SaaSCon 2011.  It’s “a Cloud Conference with a focus on Software as a Service.”  Sadly the presentations are of limited value, with the same confusion I noted in my “Impressions” article (i.e. every IT business marketing department is trying to take advantage of it and rebrand their abilities “Cloud”). While not of particular technical knowledge value, they to tend to result in humorous statements by the presenters… IBM   “80% of Fortune 100 companies are using IBM cloud capabilities.”  Wow, you’ve got 80 customers?  Really?  (Those Fortune 100 companies, that spent from $200 million to $1 billion per year on IT costs, are generally using some of every capability of every major IT vendor.) Google “4 million businesses have gone Google.”  As of 2007, the US Census Bureau reported there are 29,413,039 businesses in the U.S.  Assuming Google’s talking just about the U.S. (and I don’t think they were), that’s...

I’m sitting at CloudCon III – SaaSCon 2011.  It’s marketed as a Cloud Conference with a focus on Software as a Service.  Here’s what I’m seeing… a.  Computer hardware vendors selling small footprint office workstations.  It’s not a surprise that computer vendors for the office have finally decided to abandon the standard desk-drawer PC box for a cigar sized box.  (Anyone who opens a standard PC box will find the components would fit in a cigar box anyway, the rest is open space or fans.)  The surprise is it took so long and that they’re selling them as “cloud workstations” and spending their money trying to market them at a Cloud Conference. b.  Network hardware vendors selling the next speed in network infrastructure, 10 gigabit.  Apparently since everything’s “in the cloud” you need yet more network bandwidth to get to it.  This is some nice marketing fluff since Cloud doesn’t increase your bandwidth needs, it just shifts it fro...

Calling web services from the mainframe has become a frequent question.  But as applications (and data) may be migrated off the mainframe to apps now hosted on Linux or Windows servers, the old trustworthy batch jobs may suddenly need to access remote systems and web services to do their job. Here’s how one person phrased their problem… We are currently looking at doing a partial migration away from a MainFrame.  S ome of the functionality written in Mainframe Cobol and is called from Mainframe Batch programs.  We would like to move these cobol programs off the mainframe.  Question - If we moved the functionality in the cobol programme to a Java or .Net web service, is the a way to call this web service from a Mainframe batch programme? Technically this is an easy answer.  Yes, web services can be invoked from the mainframe.  They can even be directly invoked from CICS and from IBM Enterprise COBOL (as of CICS TS 3.1).  There are some tech...

Cloud Computing has strongly entered it’s hype cycle.  Just as everyone ran to relabel everything Service Oriented and ESB-this or that, now everything is being relabled Cloud this or that.  As soon as that happens we enter the technology confusion cycle.  (Sometimes one thinks this is intentional on the part of vendors, so you can’t tell exactly where their product fits or where it lacks.) Let’s see if we can do a little bit of Cloud Clarification™… -> Cloud Computing is about pushing applications, components, modules, abilities to an on-demand model with remote capacity. -> Software as a Service is about renting software abilities via a vendor exposing abilities, modules, business processes for remote use. -> Cloud Infrastructure , which is also being called Cloud Computing , is about renting remote computing/hardware capacity on demand and in fractional increments. About the best picture I’ve seen describing this is here… (though there’s some ...

IBM has done significant work to allow mainframe based applications to expose and consume web services.  They’ve particularly targeted CICS and languages COBOL, PL/I and C++.  While many vendors (including IBM) offer a variety of tools to provide easy web service bridging, IBM’s CICS efforts offer a direct path without loading and managing additional utilities.  There was concern in the past of the CPU load this added to CICS, but IBM handled those problems over the years with the current edition having good performance with reasonable overhead. While IBM recommends using modern development tools such as their Rational Application Developer for Z/Os (RD for Z) to automatically generate and build the binding and WSDL’s necessary for a service, their CICS command line based utility is probably used by most that do so (this being DFHLS2WS).  With a short series of configuration sessions (and limited options), it will take program information and data areas and ge...

In the past months a number of articles were published about Cloud Computing and taxes.  As a techie, choosing a software vendor on the basis of taxes may not be something considered.  But, depending on the jurisdiction, software purchases can be charged sales taxes, service taxes, or value added taxes.  While one might think ‘rented software’ isn’t a purchase and can’t be charged a sales tax, in some places it may be considered a capital acquisition (and taxes as a sale) while in others it could be charged a valued added tax (which taxes services as well as sales in European countries and Israel) or might be subject to various business taxes (such as a franchise tax in Texas where every company’s data on the Cloud is considered a local franchise and taxes as such.) First an important definition of terms, because almost EVERY “Cloud Computing will be Taxed” article I read got it absolutely WRONG… Cloud Computing – renting computing resources from a remote vendor ...

In the (near) past, security was handled by the user interface.  The user interface acted as the sole entry point to the application, and therefore all application security was oriented around user permissions. Added web services is like having great locks on your front door but opening all the windows in your house.  Lots of entry points, each of which needs security. There’s a few basic enterprise web service security concepts that need to be understood to understand web service security. Web service security may operate from a user context, an application context, or both. User Context: Application 1 includes in the (web) service request to application 2 information about the user who performed an action causing the request. Application 2 then decides if the service is permitted based on the user requesting it in application 1. This requires applications 1 and 2 to have a common user security framework (application 2 has to recognize application 1’s user and b...

  Stuxnet: Anatomy of a Computer Virus

I’ve been working with a client for an extended period of time.  This large IT department has had a variety of SOA tools and technologies available and has been doing major systems integration for 10 years.  Yet while their SOA tools have allowed them to integrate quicker than manual development, their integration methodology (essentially none) has given them 0% reuse. Reuse is a fine objective, but it may not actually be valuable depending on the business and IT organization goals.  In this client’s case we did an extensive evaluation of IT current state, IT future state plans and goals, and business goals.  That may sound like a lot of overhead to determine future state integration and SOA approaches, but in the current economic climate architecture for architecture’s sake is simply not acceptable (if it ever was). Or to put it another way, when IT is aligned with and demonstrating direct business value then IT is valued by the business.  And this attit...

Labor unions are rarely found in IT organizations.  It’s not unheard of, but generally high pay rates and frequent job mobility have made labor or trade unions appear to be of limited benefit to the employee – and therefore rejected. In general labor unions impose rules on the management that require employees with the greatest seniority (most time at the company) be promoted to more senior positions as they open up.  And they require new employees to be brought in at the most junior level. IT, with frequently changing technologies, requires bringing in subject matter experts and promoting those demonstrating top technical skills to technical leadership positions. Recently I’ve been doing some consulting on a large scale IT project, which involves quite a bit of Service Oriented Architecture and involves a unionized IT department.  They’re struggling both with the technical aspects of a major technology and architectural approach change and with the union job imp...

  BI, Business Intelligence, has taken hold at almost every mid-size or larger IT organization.  It commonly means extracting key data elements from all the main systems and databases in the organization and compiling it all together in the Business Intelligence Data Warehouse.  And the primary method for doing this is ETL – extract, transform, and load.  Basically meaning batch-style data loads performed daily, weekly, or monthly (from the source systems to the data warehouse). Setting it up is expensive and time consuming as it requires building a large capacity database and ETL processes for every important data source in the company.  The ETL processes by themselves are often not enough as data duplication and data quality problems quickly float to the surface and have to be resolved to a sufficient level to continue (resolved in the data warehouse, not in the data sources). However, it’s relatively easy to demonstrate the business value of the res...

Jason Bloomberg over at Zapthink has a devastating indictment of vendors selling “Private Clouds” and “Platform as a Service”. If you’re headed towards cloud computing (and we all are, it’s just a question of how long over the next decade till you get there), read it.

Web service security is a tricky business.  EVERY service exposed by any service provider, be it .Net, Java, the Mainframe, or any other provider needs to be secured.  Certainly if it’s exposing sensitive data (say customer data), allowing activation of a business process, and most especially if it’s involving a financial transaction. But how do you do it?  While every vendor and (almost) every technology announces compatibility with every web service security buzzword (WS-Security, SAML, X.509, etc.), they don’t describe how to actually make use of all this security data attached to the web service request. I’ve had recent discussions with IBM, Oracle, and Software AG (as leading SOA middleware tool providers) on this exact topic and the results are disappointing. The architecture model for this says that to provide SOA security I should use the tools as a SOA security layer, allowing my services to go about their business and the security tools to grab and pr...

SOA security is getting more attention.  Recent document leaks show the need of inside-the-network security, and recent attacks on service providers by Wikileaks supporting hacker-nets show the need for AT LEAST “decent” SOA security.  As ZapThink wrote recently … If you’ve been even peripherally paying attention to the news lately, then you know about Wikileaks and the battle going on between supporters and opponents of the group. While you might think that these incidents have nothing to do with you, enterprise architect, IT manager, developer, vendor, or consultant at a private company or government institution, they most certainly do. You are about to become a part of the frontline in the Cyberwar battle… Just as firms plan for server outages and deployment problems, now every firm must have a contingency plan to deal with the potential outage of their most important online suppliers… Your company uses Google for mail, a specialty Software-as-a-Service (Saa...

Is BPM (Business Process Modeling and Execution) the SOA killer application? I had a very interesting interaction with a global VP of BPM of BPM for a major vendor.  The encounter went like this… VP on behalf of vendor demonstrates BPM tools to potential client.  Everyone oohs and ahhs at the impressive presentation, graphics, and overall possibilities of BPM.  The vendor product looks impressive, it works, and actually has reasonable (runtime) performance. Then I ask the VP “how many processes can this client create in a year?”.  After all, creating a BPM workflow / process is relatively quick and easy.  In theory a couple of days to create it, another week to test it and deploy it.  Maybe a mid-sized organization should be able to generate 26 processes per year per person assigned to BPM. The VP responded, “7”.  “Seven”, I remarked, “why so few?”  He explained first of all the BPM processes won’t have all their steps exposed (at all o...

I sat with my client on a major service orientation project as IBM presented their updates to CICS.  My client is a few versions behind (could actually find no compelling reason to bother to upgrade) on their mainframe and wanted to review if IBM’s added capabilities would offer any abilities the project could utilize. IBM’s CICS support for web services is well known by now.  Their CICS updates keep the web service support up to the latest WSDL – SOAP – and WS-* standards.  If you need or want to expose COBOL programs as a web service (and this can be valuable to allow older applications to be partially utilized as transaction engines in their area of expertise) IBM’s keeping the capabilities in line. IBM has also continued to expand a large variety of internal CICS capabilities.  This was a major time warp for me as CICS command line functions, com areas and linkage sections, and various transaction batch and script coding abilities are so dated.  As an...

Loriane Lawson over at IT Business Edge has a penchant for touching upon particularly interesting integration problems.  This week she asks … It's interesting: Writing custom code for data integration is nearly universally frowned upon by experts – and yet, I'm seeing a lot of discussion about creating your own metadata solutions for support with integration. My question is: If you're getting away from hand-coding, why would you want to delve into customized metadata solutions? The article focuses upon some of the technical issues and technical approach to such a discussion.  I’d like to focus on the business issues, and how those should be directing the technical approach (but aren’t). Every application involved in integration brings along it’s database (or general data) model and it’s internal data object model.  These models were developed to meet the functional goals of the particular application.  For example, the “customer” representation in t...

This is a classic IT question.  Should one go with picking and choosing Best of Breed applications in the various niches that one’s IT shop needs, or go with an Application Suite?  SOA and Integration has some significant input to this question, and impact from this question.  And the same question applies not only to business toolsets, but also to SOA toolsets (best of breed ESB, design time governance, run time governance, SOA security tools, BPM, etc, or a suite?) With best of breed, we might end up with one company’s ERP system and another company’s CRM system, a third company’s manufacturing system and a fourth company’s financials. With historical integration patterns, the issue of data interchange between the systems was mostly handling by large scale data exports and imports, usually performed as batch processes at end of day (or end of week or end of month).  These processes could be described as “dump all the (insert primary data type here, such as c...

  I’ve been using the term Integration Spaghetti™ for the past 9 years or so to describe what happens as systems connectivity increases and increases to the point of … unmanageability, indeterminate impact, or just generally a big mess.  A standard line of mine is “moving from spaghetti code to spaghetti connections is not an improvement”. (A standard “point to point connection mess” slide, by enterprise architect Jerry Foster from 2001.) In the past few days I’ve been meeting with a series of IT managers at a large customer and have come up with a revised definition for Integration Spaghetti™ : Integration Spaghetti™ is when the connectivity to/from an application is so complex that everyone is afraid of touching it.  An application with such spaghetti becomes nearly impossible to replace.  Estimates of change impact to the application are frequently wrong by orders of magnitude.  Interruption in the integration functioning are always a major disast...

  A number of industry analysts have been speaking of SOA Design Time Governance failure for some time.  As I’ve written previously, primarily this was because the majority of enterprise IT shops hadn’t reached either the SOA maturity level to deal with it or had a large enough service catalog to have a need to address it with tools. I’ve seen a lot of change in this in the past year, as many organizations are suddenly asking for help in defining requirements for SOA governance tools. But what of the cutting edge IT shops , the early SOA adopters who ran into SOA governance needs years ago and started working with SOA Design Time Governance tools of earlier generations?  (I admit to being one of these, having led the purchase of a design time governance tool for my U.S. Fortune 50 employer at the time, about 7 years ago.) Most of these projects FAILED!  (Including the one I ran.)  The tools were complicated and somewhat rigid, the processes to make it s...

Addressing a service presents a few problems.  By putting the URL (or queue name if using messaging), you unintentionally couple between the service consumer and the physical instance of the service.  (Meaning what server it’s on, IP address, etc.) Applications often unintentionally become tightly coupled simply by addressing connections directly, by IP address, server name, or queue name. This is unacceptable, as any change in the physical layer results in software changes. (Hardcoding such information is clearly a major mistake, but even placing it in a configuration file or database entry still results in application manipulation due to physical layer changes.) Replace a server, redeploy all consumers of the services exposed on that server?  Ouch.  Even moving from development to test to production becomes a challenge (as you have to recompile or reconfigure as the consumer needs to repoint to the new provider instance in each environment). UDDI was origi...