Data Integration Blog -

In a word – Yes. But it might not appear that way yet. So let me play the angel’s advocate and see where the logic takes us.

The first point to make is that cloud computing is still young, in fact it’s very young – just out of kindergarten. So there are many CIO and data center managers that don’t have the confidence to dive in and do some serious computing in the cloud. And they have good reasons to hold back. Pioneers get scalped, as they say.

There are legitimate concerns about the cloud in respect of security, manageability and actual service levels (as opposed to promised service levels) that suggest a cautious approach. But even the most conservative CIO believes that such problems will be solved and forgotten at some point in the future. So let’s not worry about concerns that will inevitably fade away.

Is The Cloud Less Expensive?
Yes. Of course it is. It’s economic because the running costs are much closer to a utility than those of a normal data center. And it’s particularly true for very large cloud data centers. The simple fact is that if you build your data center in the right place and make it large enough then: the hardware is cheaper (because of volume discounts and you can build it yourself if you like), the power is cheaper (both for A/C and for the computers), the floor space is cheaper and the land is cheaper, the operational staff are probably less expensive and there’s much less for them to do (because of similar workloads). There’s no way that the normal data center can run anything like as economically. So even if a given cloud service isn’t particularly cheap right now, it will be in time, as the economies of scale increase.
 
The Analysis Deficit
I was having a conversation with the CMO for a BI software vendor the other day, and he suggested that there is an “analysis deficit.” The idea is easiest to understand graphically, so here’s a graph:

So we’ve been storing analyzable data at an exponential rate, the volume roughly increasing by about 60 percent per year. (Note that the vertical axis here is exponential.) Naturally we have been analyzing more and more of the data we store as well, and we’ve been doing that because there is value in doing it. But there is a gap between what we could analyze and what we do analyze and this gap appears to be increasing.

How do we know that?

This is slightly speculative, so the answer is that we don’t know that for sure, but it is very likely, because the BI Market just keeps on growing (20 to 30%) each year and we can only presume that the software and hardware sold is being used rather than not. Also we note the recent (and increasingly popular) use of Hadoop to analyze petabyte heaps of data. The data heaps we can analyze keep growing in size.

Can Data Centers Cope With The Growth in Computing Requirement?
The problem with managing a data center is that costs increase most of the time in a gradual way, then the cost suddenly escalates in a big steep jump (measured in millions or tens of millions of dollars). That happens when you run out of data center space. The growth pattern is similar to that of an insect, which sheds its exoskeleton for a burst of growth and then for a long time just grows slowly into the new exoskeleton.

This cost situation was exacerbated when computers started running hot (from about 2001 onwards) putting further cost pressure on data centers – and yet the demand for compute power never slowed – and not just in the area of BI. We kept on adding applications and we kept on storing more data and we used more and more BI tools too.

The only way data centers have to meet this problem – which is not going to pack its bags and go away – is to use the cloud. Without the cloud, may data centers would now be bursting at their seams. And conveniently, the cloud seems very economic.

But What Can You Put In The Cloud?
It is easier to approach this question from the opposite direction and ask “what applications are going to be difficult to migrate to the cloud?

Certainly, the mission critical applications, especially those which demand careful management and have many dependencies, are not going to the cloud any time soon. They will be the last to move, if they move at all.

But it’s possible to migrate software development and software testing to the cloud – and it makes good sense to migrate many BI applications to the cloud – even if the initial data load takes a long time. That’s because even though BI often runs against very high volumes of data, the results of BI tools very rarely consist of high volumes of data – and that means that the communication traffic need not be high.

So there may be a problem getting the BI data into the cloud – because of bandwidth limits – but as long as your cloud service is happy to use physical media for data loads, that’s a one-time problem.

All of this makes sense, but it makes even more sense if you need to deal with “big data”, and by big data I mean heaps of data that are measured in the tens or hundreds of terabytes (or worse). The point here is that you are not going to do that in your own data center – unless you are Google or Yahoo! or Microsoft et al. If you are going to do that, you do it in the cloud or not at all.

In Summary
Really, the point here is that some BI applications are natural for the cloud and the larger an application gets, in terms of sheer data volumes, the more it makes sense to fire it into the cloud. The cloud is really convenient for BI.