Embrace “Big Data” Opportunities
[view:https://www.youtube.com/watch?v=J_60hhYRvlo&feature=channel_video_title]
By Joe Mullich
The era of "big data" has begun, and companies are pondering how to handle the massive mounds of information being produced at lightning speed. At the moment, many people feel big data is a big problem. In a 2010 survey by Bloomberg Businessweek Research Services, a large percentage of senior executives expressed frustration with their business intelligence and analytics systems' ability to handle large and complex data sets.
Respondents were asked to rate their satisfaction levels on a scale of 1 to 10, with 1 indicating no satisfaction and 10 indicating a high level of satisfaction.
Source: Bloomberg Businessweek Research Services' 2010 survey of 128 senior executives at U.S. and European companies with at least $500 million in revenues.
Kenneth Cukier, a global authority on technology and a correspondent for The Economist, published a landmark report on big data in the magazine in 2010. He says many companies look at big data from the wrong perspective.
"When we want to extract value from data, it's not simply a question of using smart algorithms and good statistical models, as we did in the past," Cukier says. "What we want is more data, because more data will give us more information."
At first glance, that might seem counterintuitive. We have moved from a time of information scarcity to one of information surplus, and the challenge people now encounter is making sense of all of the data they currently have. Why would they need even more?
Cukier responds with an example of a company that starts with a database containing 300 million records of names. The company then combines that database with another database holding 50 million names. Adding the 300 million records with 50 million records results in... 200 million names.
Reason: When the data sources were combined, there were more identifiers to indicate which records were redundant and could be eliminated. The result was a smaller database, but one that held richer and more accurate information. Cukier says this is a "conceptual shift" that people need to make about big data. The point is not simply to add more data but to add the right data that provides greater specificity. Big data is not additive; effectively applied, it is a filtering mechanism.
Smart managers and companies, Cukier continues, will be the ones who look for ways to leverage big data to do just that. He points to Farecast, a part of the Microsoft® BingTM search engine, which can suggest whether travelers should buy an airline ticket now or wait for the price to come down by examining 225 billion flight and price records.
"That is a business model that simply could not have existed in the past," Cukier says. "There was no way of getting tens of millions of records at any moment in time with which to make these predictions and to score the degree of confidence you have in the prediction."
He goes on to add, "This is a perfect example of a big data business that was created by just applying a lot of computing power and a really important conceptual shift about the value of data and new business opportunities that could be created."
The same approach could be used to help consumers find the optimal price for used cars, houses, and a host of other products. "The mind boggles at the unlimited nature of how you could apply the same sort of metrics to a variety of products," he says. "We are going to see this everywhere."
A key is to think creatively. Instead of looking at the mounting stores of data as a problem to be managed, see them as an opportunity to be mined for new ideas. Such opportunities are rising in tandem with all of the new sources of data-from social networks and genomics research to Internet search indexing and call detail records. In just one example, a 2010 Bloomberg Businessweek article noted that a Boeing jet generates 10 terabytes of information per engine every 30 minutes of flight, meaning a cross-country flight by a twin-engine Boeing 737 would produce 240 terabytes of data.
"The stakes and potential for monetization are huge in a world where roads have sensors and can communicate with the vehicles passing over them to determine traffic patterns, find more sustainable ways to route cars, and perhaps even generate data to be sold to insurance companies or other businesses seeking to tap transportation information," the Bloomberg Businessweek article continued.
These vast stores of digital information, intelligently managed and applied, can uncover business trends, reduce dangerous drug interactions, combat crime, and be put to a host of other uses. And in a world of big data, valuable correlations can reveal themselves without computers being explicitly programmed to find them.
Cukier points out that analytics is already becoming ubiquitous. At eBay, for example, analytics is used by one-third of the 18,000-person workforce, many of whom are call-center workers who view analytics on their desktops when calls come in. "It infuses the entire organization," he says. "As goes an Internet company like eBay, so goes the rest of us-eventually."
Executives should be embracing the big data opportunity now. "It's also a mandatory requirement," Cukier exclaims. "If you don't do it, you're out of luck and out of a job, because the whole basis for your company will be an analytical, statistical, data-driven model. If you don't know this information, you won't be valuable to your company anymore."
Join the BI TV LinkedIn Group to stay up-to-date with all the new releases
Follow BI TV on Twitter