Big Data Needs To Think Bigger

Spend enough time in Silicon Valley, and of all the buzz words you’ll hear neatly tucked in with “graph,” “serendipity,” and “personalization” is one often uttered though, on the whole, not yet fully understood: “Big Data.” On the surface, everyone realizes the opportunity. Data is being generated at lightning speed, the cost of storing is tiny, and new technologies are available to help manage, organize, and secure the data. Earlier this month, LinkedIn co-founder and Greylock partner Reid Hoffman delivered a presentation on this topic at SxSW, and starting next week, GigaOM’s annual big data conference “ Structure’ ” kicks off in NYC. At the consumer level, while we are wowed by pretty visualizations, the real advancements in big data technologies cover (1) how data is structured and stored, (2) how it is organized and retrieved, and, most interesting to me, (3) how underlying mathematics can be written into algorithms to leverage the data and help discover entirely new things. I’ll paraphrase from one data scientist, LinkedIn’s Peter Skomorach , who notes on Quora that cheap data storage allows users to leverage asymmetric information, larger data sets increase the likelihood that n