Fast Times at "Big Data" High

"Big Data" is showing up more and more in the Information Technology News so you might be asking yourself "What is Big Data"?  Big Data is the representation of extremely large data sets and the management of these data sets using new and improved systems that are specifically made to handle these data sets quickly.  These data sets are usually filled with auto-generated data that was collected using sensors or cameras. [1]

One of the tools generated to work with Big Data are NoSQL databases.  With the large amount of data manipulation going on, a new way of data storage and access needed to be created.  NoSQL databases focus on retrieving items from strorage efficiently and in real-time. [2] A few of the pros of using NoSQL databases includes elastic, transparent scalability, and more relaxed data model restrictions making data model changes easier to implement. [3]  NoSQL databases though still have a ways to go as the next generation of technicians are struggling to keep up and it's far from a hands-off solution right now.


With the introduction of new "databases" in the market also welcomes new management packages like Hadoop, Pig, and Hive.  Hadoop "is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model." [4]  Hadoop is comprised of two parts: the Hadoop Distributed File System and high-performance parallel data processing. [5] Pig and Hive both are related to Hadoop, and it's functionality.  Pig is "a high-level data-flow language and execution framework for parallel computation," and Hive is "a data warehouse infrastructure that provides data summarization and ad hoc querying." [6]

So why is "Big Data" important? I believe this quote from a recent article sums it up: "The amount of data in our world has been exploding, and analyzing large data sets—so-called big data—will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey's Business Technology Office." [7]  Big Data is where the future is as more time passes and more equipment are being developed to collect more and more data from the world.  The faster you can manipulate that data and get it back out into the world, the more successful you will be going forward.  The age of starting a program and walking away from a few days until it's done running are over and no longer acceptable to standards in today's age.


More Information about Big Data in the Media Recently:
Adobe Unleashes Predictive Analytics on Big Data Complexity
Will Google Big Query Transform Big Data Analysis?
Customers, Big Data, and the Internet of Things
Mobile, social and big data drive cloud computing boom: studies
Microsoft destined to follow in big data

For Your 'Leaner' Wednesday

ALT Tag: "Our company is agile and lean with a focus on the long tail. Ok, our company is actually a polecat I found in my backyard."


Brought to you by the awesomeness that is XKCD: http://xkcd.com/1032/

The "Leaner" Side of Life

Lean, or Lean Thinking, or the Toyota Way, is a concept that defines a general process of 'continuous improvement' and 'respect for people'.  This process can be applied to many situations, but this article mostly describes it in terms of production of a service.  Part of the 'continuous improvement' aspect, also known as kaizan,a is to allows challenge everything and to always be open to change.  Toyota defined this as "always being dissatisfied with the status quo."

[Figure 1. From "Lean Primer" by Craig Larman and Bas Vodd.]

The 'Lean-Thinking House' visually shows these two concepts working in tandem to build a solid foundation for product development and management.  The top tier, aka the 'roof', represents the goal trying to be accomplished.  On the bottom, you have the 'foundation', which represents solid management that applies and teaches lean thinking to their teams and base their decisions off of lean long-term thinking.  On the two sides, you have the two pillars that define lean thinking.  And in the middle, you have the heart that is product development and the 14 principles.  The 14 principles outline the various principles that have been developed by Toyota over all their decades of product development and include items such as 'move toward flow' and 'grow leaders from within'.  The product development area represents focusing "on creating more useful knowledge and learning better than the competition."  Then using that knowledge and making sure that it does not go to waste.  Lean thinking is about creating high-value, low-cost information to utilize in their development processes as shown below.

[Figure 2. From "Lean Primer" by Craig Larman and Bas Vodd.]

As for my thoughts on the lean methodology, I think it's an interesting concept and would be open to working in such an environment.  It definitely has it's benefits as a developer especially since your employer is actually trying to enhance your knowledge and to maintain your loyalty as an employee.  I'd be curious to see how this might be implemented in my work environment in education.  Many times we are 'firefighters' in addition to our development duties.  Just how this would work out in an environment where each individual wears many hats from customer support, to system administration, to development, to management and administrative duties, I don't know.  Would this only work in a profit driven company?  How easily could this be accomplished with other organizations?  There actually is a group on campus currently training up on Lean Six Sigma; it will be interesting to see what they do with this knowledge and if they are successful in implementing it and getting buy-in from their management.  I'll update you hear if I learn more about their implementation.  That's one thing I wish this primer would have delved deeper into, is how this might work and does currently work in companies another than Toyota.

"Hammer"ing Out the BPM System

According to Michael Hammer, "Business Process Management (BPM) is a comprehensive system for managing and transforming organizational operations, based on what is arguably the first set of new ideas about organizational performance since the Industrial Revolution."  In his article, "What is Business Process Management?", he discusses topics such as it's origin and its development.  Hammer credits the work of Shewhart and Deming as the first to approach BPM with statistical analysis of problems occurred during production; this methodology was too generally defined for Hammer's taste.  His work then is the second definitive work on the subject matter.


Hammer then goes on to discuss the process management cycle, shown below, which was eventually developed by merging the two definitive methodologies of himself and Shewhart and Deming.  This cycle is derived from Deming's Plan-Do-Check-Act cycle.  Hammer then goes on to discuss how "through process management, an enterprise can create high-performance processes, which operate with much lower costs, faster speeds, greater accuracy, reduced assets, and enhanced flexibility."  And he goes on into the operational benefits that might be realized through such as process.  But he also notes that this road is not always easy; there are certain processes that must be changed to allow BPM to flourish: process design, process metrics, process performers, process infrastructure, and the process owner.  Hammer then continues on to discuss the items that need to be able to change to be a success such as leadership, governance, culture, and the expertise.

[Figure 1. From "What is Business Process Management?" by Michael Hammer]

As a summary, Hammer reviews the main principles of process management: All work is a process.  Any process is better than no process.  A good process is better than a bad process.  One process version is better than many.  Even a good process must be performed effectively.  Even a good process can be made better.  Every good process eventually becomes a bad process.  After that Hammer begins to discuss how Enterprise Process Model (EPM) can be used a graphical tool to enhance BPMN.  An example is shown in the graphic below.  Lastly, Hammer gives his ideas on where BPM might be going in the future.

[Figure 2. From "What is Business Process Management?" by Michael Hammer]

In my opinion, Hammer definitely is one of the frontiersmen of BPM.  While his writing can be a little hard to follow, his methodology is not.  It's quite simple and has a bit of lean thinking's 'challenge everything' mantra.  He describes most things in three phases: what it was doing, what it is doing, what it will be doing.  Always a continuous improvement process.  It will definitely be interesting to see where the BPM methodology will go in the future and what it's adoption rates will be.