Making the Cloud, Microservices, and Enterprise IT work!

What is a Service or Component?  It sounds like a simple question.  But I was approached by an organization that had done no component or service oriented or even object oriented development.  So the question was much more complicated… What should a service / component / object be?  What amount of business functionality should it include?  Or, in technical terms, what granularity to your business services or components? This question came up again for me, as I’m now dealing with the same question relative to Business Processes (BPM or BPA workflows).  That will be a future post. But for how granular should a service or component be?  Here’s my presentation on the topic… What is a SOA Service - from the Business Perspective? How detailed, what level of functionality does it expose? from Akiva Marks

My customer presented me with a major enterprise architecture task.  “We’re building new systems in a new environment, it’s time to modernize our batch processing.  Please recommend a toolset and processing model to move our legacy batches to our new environment.”  I didn’t realize it at the time, but this was a trick question. I spent an extensive amount of time investigating modern batch tools as well as Java development and batch.  What I found was a very limited set of such tools, but more important as I consulted various architects I know around the world, the question of “why are you trying to modernize batch?” kept arising. The question is less straightforward than it sounds.  SOA, a component orientation, real time business and event driven business make batch exceptionally challenging, but more importantly much less relevant. Attached is the result of my research, a whitepaper on a modern approach to batch in light of Service Oriented Archite...

(Large Government Agency) has begun one of the largest government IT projects in (Country). (Large Government Agency) has a strong desire to move into the future, provide more efficient and user friendly systems with web and mobile interfaces, and move from 2 nd generation code based systems (requiring over 150 maintenance programmers) to a 5 th generation rules and workflow based system. Unfortunately failure to upgrade their software environment and approach in the past left them with not only 2 nd generation code and user interface (NATURAL by Software AG and 3270 green screens running in an IBM Mainframe environment) but with a 2 nd generation architecture . The mixed screen & processing logic, pure mainframe batch processing model and narrowly specialized data model was judged unable to be extended for use as a transaction engine or even be service enabled. Certainly no part of a 3270 green screen 80x25 character based user interface can carry forward into a web ...

Whether we’re integrating systems or building new SOA architected applications (with heavy cross-component communication that may be flowing between physical machines or across virtual machine clusters – which again drop down to separate physical machines), the network backbone makes a significant difference. A good enterprise data center is running at least a 1gbit backbone, with portions or major interconnects at 10gbit.  Further, those good network engineers are properly segmenting the network, which makes sure the heavy traffic patterns have maximum capability (and aren’t going to be slowed down by users streaming internet radio). Between applications in the core of the data center, we rarely run into network capacity or network speed as our major performance problem.  (Credit those network engineers.)  This doesn’t mean the issue should be ignored, if some applications are integrated in a high speed query or transactional pattern, there can definitely be a m...

- When linking two systems, with one system providing data to another system, the providing system's physical environment must be sized for the capacity of the requesting system and the reliability of the requesting system. Another way of saying this is that the providing system must meet or exceed the quality of service or SLA of the requesting system. - In the example I was reviewing, the HR system was the base for the desired information and was sized for the user capacity of the HR department and the reliability impact of an outage of the HR department. If it is to be used as a real-time providing system for Department B, it's capacity must be increased from the HR department (10 users) to the capacity of the Department B user base, the main business area (3,000 users). It must also have it's redundancy increased to provide no outages. - Alternatively, when there is a mismatch between capacity and reliability of the providing system and requesting system, the ...

SOA design time governance products come with a variety of methods to categorize the services or assets.  Interestingly I’m currently working on a project involving a document management system, and the select document management tools comes with almost an identical selection of categorization methods.  These include trees, taxonomies, and domains among others. In some of the earlier SOA design time governance implementations I performed, we spent significant time working on the categorizations – trying to make the catalog and information easy to traverse for the various user categories that would encounter in.  (In the case of design time governance, this might be analysts, architects, developers, QA, and IT management.) We invariably found designing the categories and approaches took a tremendous amount of time, with every constituency having different ideas and requesting various adjustments to the approach.  To some extent it became the never ending quest ...

Software AG’s Natural language and development environment for the mainframe (IBM z/OS and CICS) offered many nice improvements over COBOL for mainframe software development.  Software AG developed a nice market niche in the mainframe development tools market, and millions of lines of code were developed and continue to run till today. Like IBM’s version of COBOL, the vendor has struggled to extend Natural’s lifespan and the life of the code written in it.  Initially Software AG created a communication bridging tool (called EntireX), allowing bi-direction communication between the Natural environment and Web Services or MQ or Java or .Net.  But like IBM came out with “native” web services as part of CICS 3 and Enterprise COBOL, Software AG has web service enabled Natural.  And web service enablement includes the handling of (simplified) XML.  From the code sample I was able to find, it appears that Natural IS NOT handling the building of the HTTP header...

I attended a major vendor conference this week.  This particular vendor has a a modern line of current modern products, and also a set of 2nd generation mainframe products.  They’ve wisely bought a series of smaller product companies over the past decade to ensure their future. But the old product set continues to live on as well.  Yes they’re offering modern interfaces, web service enabling abilities and so forth.  And naturally they’re still investing in the original product set, as a cash cow should be milked as long as possible. One of the speakers I wanted to hear was speaking in the older product set track, so I made my way to that conference area.  As I entered it a generational change occurred.  EVERYONE, and I mean everyone, was 55 years old or older – with the majority seemingly very very close to retirement. There’s a clear hint from this sight for those using this product set.  We normally think of end of life for a technology as w...

I met with a major vendor technology evangelist recently (actually he called himself a technology space regional CTO) on the topic of noSQL and big data.  Now I’ll admit, like any technologist and architect, I have my areas of specialty and areas of less specific knowledge.  And as the IT industry continues to develop new technologies and approaches yearly, it’s challenging to stay up to date. So while I know relational databases well, can write good SQL and understand architecturally when to use database power functions like triggers, store procedures and the like, as well as database implementation level issues such as clustering, failover, performance, etc, I had not been paying attention to recent industry changes such as noSQL, graph databases and big data. This tech evangelist presented the concept of a particular noSQL tool, emphatically stating that the day of the relational database was over.  Relational databases are dead as a future technology.  H...

I was recently deep in an architecture process when I was asked by a team member for some help in understanding the team operating model.  Or more specifically, a team leader and architect were asking me whether they should be coding a particularly difficult area of the system. Being in architect mode, I immediately ran to the white board and architected the optimum software development team process.  Here it is…   What I was trying to describe was the role at each level, which is achieved through experience, and the flow between them.  So while an architect or team leader may be able to develop code at 3-5 times the rate of a Junior Programmer, everyone above can fill the roles below – but the reverse is not true.  So if that architect is programming, no one is architecting! And while the architect might be able to code without an architecture or design (because he or she has an image of it in his or her mind), if he (or she) does so there is no archi...

I’m sitting at CloudCon III – SaaSCon 2011.  It’s “a Cloud Conference with a focus on Software as a Service.”  Sadly the presentations are of limited value, with the same confusion I noted in my “Impressions” article (i.e. every IT business marketing department is trying to take advantage of it and rebrand their abilities “Cloud”). While not of particular technical knowledge value, they to tend to result in humorous statements by the presenters… IBM   “80% of Fortune 100 companies are using IBM cloud capabilities.”  Wow, you’ve got 80 customers?  Really?  (Those Fortune 100 companies, that spent from $200 million to $1 billion per year on IT costs, are generally using some of every capability of every major IT vendor.) Google “4 million businesses have gone Google.”  As of 2007, the US Census Bureau reported there are 29,413,039 businesses in the U.S.  Assuming Google’s talking just about the U.S. (and I don’t think they were), that’s...

I’m sitting at CloudCon III – SaaSCon 2011.  It’s marketed as a Cloud Conference with a focus on Software as a Service.  Here’s what I’m seeing… a.  Computer hardware vendors selling small footprint office workstations.  It’s not a surprise that computer vendors for the office have finally decided to abandon the standard desk-drawer PC box for a cigar sized box.  (Anyone who opens a standard PC box will find the components would fit in a cigar box anyway, the rest is open space or fans.)  The surprise is it took so long and that they’re selling them as “cloud workstations” and spending their money trying to market them at a Cloud Conference. b.  Network hardware vendors selling the next speed in network infrastructure, 10 gigabit.  Apparently since everything’s “in the cloud” you need yet more network bandwidth to get to it.  This is some nice marketing fluff since Cloud doesn’t increase your bandwidth needs, it just shifts it fro...

Calling web services from the mainframe has become a frequent question.  But as applications (and data) may be migrated off the mainframe to apps now hosted on Linux or Windows servers, the old trustworthy batch jobs may suddenly need to access remote systems and web services to do their job. Here’s how one person phrased their problem… We are currently looking at doing a partial migration away from a MainFrame.  S ome of the functionality written in Mainframe Cobol and is called from Mainframe Batch programs.  We would like to move these cobol programs off the mainframe.  Question - If we moved the functionality in the cobol programme to a Java or .Net web service, is the a way to call this web service from a Mainframe batch programme? Technically this is an easy answer.  Yes, web services can be invoked from the mainframe.  They can even be directly invoked from CICS and from IBM Enterprise COBOL (as of CICS TS 3.1).  There are some tech...

Cloud Computing has strongly entered it’s hype cycle.  Just as everyone ran to relabel everything Service Oriented and ESB-this or that, now everything is being relabled Cloud this or that.  As soon as that happens we enter the technology confusion cycle.  (Sometimes one thinks this is intentional on the part of vendors, so you can’t tell exactly where their product fits or where it lacks.) Let’s see if we can do a little bit of Cloud Clarification™… -> Cloud Computing is about pushing applications, components, modules, abilities to an on-demand model with remote capacity. -> Software as a Service is about renting software abilities via a vendor exposing abilities, modules, business processes for remote use. -> Cloud Infrastructure , which is also being called Cloud Computing , is about renting remote computing/hardware capacity on demand and in fractional increments. About the best picture I’ve seen describing this is here… (though there’s some ...

IBM has done significant work to allow mainframe based applications to expose and consume web services.  They’ve particularly targeted CICS and languages COBOL, PL/I and C++.  While many vendors (including IBM) offer a variety of tools to provide easy web service bridging, IBM’s CICS efforts offer a direct path without loading and managing additional utilities.  There was concern in the past of the CPU load this added to CICS, but IBM handled those problems over the years with the current edition having good performance with reasonable overhead. While IBM recommends using modern development tools such as their Rational Application Developer for Z/Os (RD for Z) to automatically generate and build the binding and WSDL’s necessary for a service, their CICS command line based utility is probably used by most that do so (this being DFHLS2WS).  With a short series of configuration sessions (and limited options), it will take program information and data areas and ge...

In the past months a number of articles were published about Cloud Computing and taxes.  As a techie, choosing a software vendor on the basis of taxes may not be something considered.  But, depending on the jurisdiction, software purchases can be charged sales taxes, service taxes, or value added taxes.  While one might think ‘rented software’ isn’t a purchase and can’t be charged a sales tax, in some places it may be considered a capital acquisition (and taxes as a sale) while in others it could be charged a valued added tax (which taxes services as well as sales in European countries and Israel) or might be subject to various business taxes (such as a franchise tax in Texas where every company’s data on the Cloud is considered a local franchise and taxes as such.) First an important definition of terms, because almost EVERY “Cloud Computing will be Taxed” article I read got it absolutely WRONG… Cloud Computing – renting computing resources from a remote vendor ...

In the (near) past, security was handled by the user interface.  The user interface acted as the sole entry point to the application, and therefore all application security was oriented around user permissions. Added web services is like having great locks on your front door but opening all the windows in your house.  Lots of entry points, each of which needs security. There’s a few basic enterprise web service security concepts that need to be understood to understand web service security. Web service security may operate from a user context, an application context, or both. User Context: Application 1 includes in the (web) service request to application 2 information about the user who performed an action causing the request. Application 2 then decides if the service is permitted based on the user requesting it in application 1. This requires applications 1 and 2 to have a common user security framework (application 2 has to recognize application 1’s user and b...

  Stuxnet: Anatomy of a Computer Virus

I’ve been working with a client for an extended period of time.  This large IT department has had a variety of SOA tools and technologies available and has been doing major systems integration for 10 years.  Yet while their SOA tools have allowed them to integrate quicker than manual development, their integration methodology (essentially none) has given them 0% reuse. Reuse is a fine objective, but it may not actually be valuable depending on the business and IT organization goals.  In this client’s case we did an extensive evaluation of IT current state, IT future state plans and goals, and business goals.  That may sound like a lot of overhead to determine future state integration and SOA approaches, but in the current economic climate architecture for architecture’s sake is simply not acceptable (if it ever was). Or to put it another way, when IT is aligned with and demonstrating direct business value then IT is valued by the business.  And this attit...

Labor unions are rarely found in IT organizations.  It’s not unheard of, but generally high pay rates and frequent job mobility have made labor or trade unions appear to be of limited benefit to the employee – and therefore rejected. In general labor unions impose rules on the management that require employees with the greatest seniority (most time at the company) be promoted to more senior positions as they open up.  And they require new employees to be brought in at the most junior level. IT, with frequently changing technologies, requires bringing in subject matter experts and promoting those demonstrating top technical skills to technical leadership positions. Recently I’ve been doing some consulting on a large scale IT project, which involves quite a bit of Service Oriented Architecture and involves a unionized IT department.  They’re struggling both with the technical aspects of a major technology and architectural approach change and with the union job imp...

  BI, Business Intelligence, has taken hold at almost every mid-size or larger IT organization.  It commonly means extracting key data elements from all the main systems and databases in the organization and compiling it all together in the Business Intelligence Data Warehouse.  And the primary method for doing this is ETL – extract, transform, and load.  Basically meaning batch-style data loads performed daily, weekly, or monthly (from the source systems to the data warehouse). Setting it up is expensive and time consuming as it requires building a large capacity database and ETL processes for every important data source in the company.  The ETL processes by themselves are often not enough as data duplication and data quality problems quickly float to the surface and have to be resolved to a sufficient level to continue (resolved in the data warehouse, not in the data sources). However, it’s relatively easy to demonstrate the business value of the res...

Jason Bloomberg over at Zapthink has a devastating indictment of vendors selling “Private Clouds” and “Platform as a Service”. If you’re headed towards cloud computing (and we all are, it’s just a question of how long over the next decade till you get there), read it.

Web service security is a tricky business.  EVERY service exposed by any service provider, be it .Net, Java, the Mainframe, or any other provider needs to be secured.  Certainly if it’s exposing sensitive data (say customer data), allowing activation of a business process, and most especially if it’s involving a financial transaction. But how do you do it?  While every vendor and (almost) every technology announces compatibility with every web service security buzzword (WS-Security, SAML, X.509, etc.), they don’t describe how to actually make use of all this security data attached to the web service request. I’ve had recent discussions with IBM, Oracle, and Software AG (as leading SOA middleware tool providers) on this exact topic and the results are disappointing. The architecture model for this says that to provide SOA security I should use the tools as a SOA security layer, allowing my services to go about their business and the security tools to grab and pr...

SOA security is getting more attention.  Recent document leaks show the need of inside-the-network security, and recent attacks on service providers by Wikileaks supporting hacker-nets show the need for AT LEAST “decent” SOA security.  As ZapThink wrote recently … If you’ve been even peripherally paying attention to the news lately, then you know about Wikileaks and the battle going on between supporters and opponents of the group. While you might think that these incidents have nothing to do with you, enterprise architect, IT manager, developer, vendor, or consultant at a private company or government institution, they most certainly do. You are about to become a part of the frontline in the Cyberwar battle… Just as firms plan for server outages and deployment problems, now every firm must have a contingency plan to deal with the potential outage of their most important online suppliers… Your company uses Google for mail, a specialty Software-as-a-Service (Saa...