Fast Times at "Big Data" High

"Big Data" is showing up more and more in the Information Technology News so you might be asking yourself "What is Big Data"?  Big Data is the representation of extremely large data sets and the management of these data sets using new and improved systems that are specifically made to handle these data sets quickly.  These data sets are usually filled with auto-generated data that was collected using sensors or cameras. [1]

One of the tools generated to work with Big Data are NoSQL databases.  With the large amount of data manipulation going on, a new way of data storage and access needed to be created.  NoSQL databases focus on retrieving items from strorage efficiently and in real-time. [2] A few of the pros of using NoSQL databases includes elastic, transparent scalability, and more relaxed data model restrictions making data model changes easier to implement. [3]  NoSQL databases though still have a ways to go as the next generation of technicians are struggling to keep up and it's far from a hands-off solution right now.


With the introduction of new "databases" in the market also welcomes new management packages like Hadoop, Pig, and Hive.  Hadoop "is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model." [4]  Hadoop is comprised of two parts: the Hadoop Distributed File System and high-performance parallel data processing. [5] Pig and Hive both are related to Hadoop, and it's functionality.  Pig is "a high-level data-flow language and execution framework for parallel computation," and Hive is "a data warehouse infrastructure that provides data summarization and ad hoc querying." [6]

So why is "Big Data" important? I believe this quote from a recent article sums it up: "The amount of data in our world has been exploding, and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey's Business Technology Office." [7]  Big Data is where the future is as more time passes and more equipment are being developed to collect more and more data from the world.  The faster you can manipulate that data and get it back out into the world, the more successful you will be going forward.  The age of starting a program and walking away from a few days until it's done running are over and no longer acceptable to standards in today's age.


More Information about Big Data in the Media Recently:
Adobe Unleashes Predictive Analytics on Big Data Complexity
Will Google Big Query Transform Big Data Analysis?
Customers, Big Data, and the Internet of Things
Mobile, social and big data drive cloud computing boom: studies
Microsoft destined to follow in big data