“Big Data is at the foundation of all the megatrends that are happening today, from social to mobile to cloud to gaming.” Chris Lynch – Vertica Systems
A good candidate for the fastest growing technology market out there is the Cloud space – Gartner estimated that the Infrastructure as a Service (IaaS) market grew by 32.8% over 2014 and reached $ 16.5 Billion this year. The growth is expected to continue at a CAGR of over 29% through till 2019. These are big numbers but there is stiff competition in the growth stakes from Big Data. IDC recently estimated that the market for Big Data technology and services would grow at a comparable CAGR of 26.4% and reach over $ 41 Billion by 2018. Chances are if you are a serious player in either of these games your shareholders are happy today.
From our vantage point as a technology partner to businesses looking to leverage technology for business impact, we have a fair idea of why these segments seem to be marching ahead in lockstep with each other. Frankly, we view the relationship between the sectors as symbiotic. Our experience has been that organizations looking to deploy Big Data and Cloud solutions together derive much more benefit that those looking to deploy one without the other. Let’s explore why?
The Big Data questions
First consider just why organizations go in for Big Data solutions – clearly it’s because there is such a profusion of data out there and they believe they can leverage it to make more informed decisions. The Evans Corp. “Big data and Advanced Analytics Survey 2015” found that 40.8% of the organizations going for Big Data (as opposed to just database) solutions did so because of the total size of the data being processed. This is the “Volume” argument Big Data proponents make.
Then there is the “Velocity” of the data. This typically refers to the sheer amount of data being generated in very short spans of time. We like to extend this to also include a discussion about the variable rate at which data is created – often a tsunami like burst followed by a period of relative calmness. The other side of the “Velocity” story is the rate at which the data has to be analyzed and insights presented to the business so that real-time actions can be taken. The goal posts have shifted from “batch processing” of data to “real time analysis” now.
Into the mix let us throw in just how complex this whole operation is. The data is generated from several different sources and the post-analysis insights have to be presented to the teams most affected by it immediately. The people who could do the most with this information may be located at several different locations around the world and accessing the data in so many different ways.
The Cloud Case
Going back to the Evans Corp. survey mentioned earlier, the top 2 barriers the surveyed organizations cited in leveraging Big Data solutions were the availability of the relevant tools (10.9%) & the cost of storing the data (10.2%). If you consider the complex demands placed on the IT & Storage Infrastructure it becomes clear why a scalable, on-demand, pay-as-you-use Cloud model would work well with Big Data. The emphasis these days is shifting from the traditional data warehouses to a more flexible or elastic infrastructure. The attraction is obvious, using someone like AWS to house your data allows you a significant amount of flexibility – to add more storage as your data grows and even to scale it back when needed. AWS Data Science chief, Mark Wood, has quoted the example of the Human Genome Project that rapidly moved from gigabytes of data to terabytes and then petabytes of data – a move tough to make with traditional, on-premise approaches. In the case of geographically dispersed teams access can be provided with a degree of simplification of the internal infrastructure – something IT Teams appreciate. With so many clear benefits it’s no surprise that, based on what we see, pretty much all Big Data initiatives today include a significant nod to the Cloud.
No relationship is perfect
That being said though there are still some gaps, vendors would call them market opportunities. Among the chief concerns is moving such vast amounts of data around, to the Cloud IaaS to be stored and back to the Analytics tools for processing, is not yet a solved problem. Speeds are low, data corruption, while lower than it used to be, is still a factor and bandwidth is a cost factor. Then there is the security of the data – enterprises generally believe in-premise to be safer than the Cloud. Organizations vary of the Cloud for these reasons are, understandably, more circumspect in how they make the move.
Journalist Adrian Bridgwater has written, “Positive business transformation requires a journey into what Gartner calls the ‘Nexus of forces’: Mobile, Cloud, Social Media and Big Data. Bringing these four independent trends together to deliver new business opportunities should be every firm’s goal.” Our view is that while some concerns remain, over the long term all Big Data strategies will be built around the Cloud – the combination is just too powerful to ignore.