Five Reasons Not To Neglect The Coming Data Avalanche

By Johna Till Johnson
On Oct 25, 2012
Wednesday, October 24, 2012

Tempted to dismiss “big data” as industry hype? Bad idea.

Nemertes has conducted in-depth primary research with 220 individuals at 200 organizations in 2012 to understand, among other things, the impact of big data. We asked about big data drivers, technologies, and challenges, and the types of organizational structures that work well (and not so well) in big data initiatives.

The upshot? There are plenty of reasons to take big data seriously. Here are five:

1.     Big data really is different. Unlike traditional data, big data comes from a broad spectrum of sources: existing databases, mountains of unstructured and semi-structured documents, audio and video streams, Web and social media sources, and the emerging “network of things” (including sensor networks and machine-to-machine communications).  Because the sources are so varied—and often unplanned-for-- technology professionals are likely to find themselves with a mismatch between the architecture on their roadmaps and what’s actually needed: “We’re planning parallelized processing for large volumes of structured data; then mining unstructured data,” says a vice president at a very large financial services company. 

2.    There’s a lot of it, and it’s growing fast. The cutoff point for what gets described as “big data” appears to be roughly 1 Petabyte, but the average amount of data in organizations that describe themselves as having big data initiatives is 20 Petabytes—and it’s growing dramatically. “We’re expecting petabytes (of growth) per year,” says an architect in a very large energy company. And nearly half of companies will have big data initiatives underway by 2014, or are assessing such initiatives.

3.    It’s often real-time.  Unlike with traditional data stores, big data is often real-time, whether streaming audio or video, or (more often) from real-time activities: social media, sensor networks, machine-to-machine communications. Mining those data streams for good information requires tools and technologies (such as streaming databases) that aren’t common in most organizations.

4.    The information it provides can be critical. With all the technical challenges posed by big data, you can’t afford to ignore it. The driving motivation behind “big data” is “big information”: Information that can ensure an organization makes significant decisions quickly. Sometimes it’s the technologists that realize this; other times it’s business executives. But regardless of who first notices that there’s gold in that data, the rush to mine it is on.

5.     You probably aren’t organized properly to deal with it. Fewer than 40% of big data initiatives are led by a dedicated executive—yet the success of such initiatives correlates with that kind kind of focus. Developing the organizational structure that ensures success is both critical and challenging, as it requires close integration among lines of business and (multiple) technology groups.

The bottom line: Big data is real, it’s here, and it’s driving substantial changes in technology and organization.

If you haven’t already, register for passes at the E2 Innovate Big Data conference with Priority code: WEEJ12 to attend my track and discuss the Big Data Avalanche with attendees on November 12-15 at E2 Innovate, Silicon Valley. This code offers $500 off the current price of Conference passes or a free expo pass.