“Big Data” reached its Peak in 2012, according to the Gartner and based on their Hype Cycle. This is what skeptics are saying:   

• “Correlation is as good as causation, if you have enough data”. If you’re disappointed with Big Data, you’re not paying attention is a must read blog.
• The biases of digital vs. traditional qualitative non-digital research. In particular for those that are incorrectly using “Big Data” as a synonym of “Web data”
• Big Data can be bad to companies if it’s misused or confused as “strategy”. Here
• The Big Data fad exists because of the compulsory "need to feel in control”
• One of the risks of Big Data is that it might focus a company even more internally than in the past.
• Multiple conversations gravitate on the amount of “right” data as well as it’s “quality”
• On the extraction of value: Big Data reminds me of the gold rush, we’re early at a point to really understand how deep the insights can be. Most are sure at the end of the road most companies will use Big Data for more than just storage automation.
• Many will just opt-out Big Data. “Big Data is not truth”  and many more blogs to come…

Still in doubt? You should not be, consume the right resources. There are countless Big data reports, covering marketing and every imaginable application. I’ll highlight key nuggets:

1. Updated. The IDC digital universe 2020 was refreshed last December, documents the data explosion.
2. “Big Data: the next frontier for innovation, competition and productivity” by McKinsey turned out to be the main reference for the scarcity of Analytics Human Resources.
3. The emerging returns of Big Data by Tata is a great summary of the state of Big Data across geographies, industries and solutions.
4. From the IT Industry, The Software & Information Industry Association published their “Data-Driven innovation” report
5. “The evolution of data products” is one of my favorite reports, it helped me understand why Big Data can be more important than the PC itself.
6. Beyond Information Technology, Mary Ann Liebert’s “Big Data Journal” covers the short and long term potential of Big Data.
7. If the industry won’t solves Privacy and Security concerns, the government will. The “Shakespeare review of Public Sector information” provides great suggestions on establishing National Data Initiatives.
8. For the Investor there is no direct link for “Big Data II” but you might try to contact Merryll Lynch and ask for a copy. A nice analysis of the companies and products behind this revolution. On a related note, Gartner is actually talking now of the creation of a whole new Data-Driven Economy.
9. Missed on my original list. The “Unreasonable effectiveness of data”. Presumes “Invariably, simple machine learning models and lots of data trump more elaborate models based on less data”

Stay tuned. Follow @luisdans or subscribe to my “Big Data and Emerging Tech” flipboard magazine. (On Windows you need to get BlueStacks first)

• “The evolution of data products” by Mike Loukides describes our final destination as <Disappearing data>. A world where the use of computers is irrelevant, the information will be delivered on a <human time>.
• The inaugural issue of the “Big data” Journal by Mary Ann Liebert Publishers. The perspective and interview articles highlight the phenomena across many developed and emerging markets. A must read.

By combining PCs embedded in our bodies (either temporal tattoos or robotic extensions), ubiquitous computing, big data and a new internet we’ll suddenly will start living on a new augmented universe.  The documents bring a new level of clarity around the “big data” buzzword.

In the future… we’ll be able to infinitely analyze any decision, for example:

• At dinner time we’ll be presented multiple recommended options. We’ll be able to evaluate if consuming a particular one can extend our life for a few minutes, if it will result on an increase our medical insurance cost and how a particular option can modify economic and physical long term objectives.
• Before leaving our home for an external event, an automated system will remind us of taking a specific suit to the dry cleaner - based on future activities in our calendar and the time it takes to have to do the dry cleaning. It will evaluate the lowest cost and most impactful moment to get this done. The self-driving car will optimize the route every few milliseconds. The next few days and years will be fine tuned for a  just-in-time life.

Our Facebook experience today is unique every single time we visit the site. Imagine a same concept applied to everything around us. If desired, evaluate any decision by their infinite consequences. Play and visualize a “what if” scenario from any previous day in our lives. Every new solution creates new problems, we’ll need to tackle negative aspects as the new reality takes over our present.

With certain limitations, the world will adapt to what we want and to what certain virtual agents favor. We’re not living on the Information era, just on its beginning.

What are the most influential papers in the world of Big Data?  Let me suggest:

And to capture the information deluge word of mouth from the last few days, visit my “Digital Genesis”

I believe Database technologies are shaping the future of IT. Consider how this important workload can be delivered:

1. As traditional software-only and hardware servers
2. Over the public cloud, enabling new scenarios. Either as a “fabric” cloud offering or a plain RDMBS software appliance.
3. Hardware Database appliances will be the delivery mechanism for private clouds. After ~16B have been spent by Oracle, SAP, EMC, IBM and Microsoft on acquisitions (Netezza, Sybase, Greenplum, 3PAR…) during the last few months, there is little question that full stack databases technologies are the future. Most of these vendors sell hardware/software singularities as the new market standard.
4. There is a thin line on when a Reference Architecture (RA) ends; the SQL Server Fast Track Data Warehouse reduces hardware testing and provides better performance out-of-the-box than costly fine-tuning that has take place over many years. Before a computing appliance exists, there must be a RA that is used by OEMs and system builders.

The fundamental change is automation going beyond a few servers to thousands of servers. While we’re still early in cycle of data crossing boundaries (e.g. MDM), the new models are arriving – become a leader and embrace computing appliances and hardware+software reference architectures for database and other IT workloads.

• Managing hundreds of TBs continues a challenge to traditional software-only solutions
• Reference architectures deliver specialized fine tuning to specific workloads, e.g. Microsoft SQL Server Fast Track Data Warehouse 2.0 can alleviate the problem on the majority of the organizations
• IBM, Britton-Lee, Netezza and most recently Oracle have participated as database machine vendors. There are niche markets for building special-purpose computer hardware for specialized workloads. None of them has delivered a long-term solution.
• During the last 12 months there has been growing activity on database appliances: XML database appliances, database security sniffer appliances and those announced by vendors to create private clouds. Some of them for the very large database, some of them for analytics on smaller markets.

The new delivery model for information management will include embedded, in-premise, cloud, software appliances, hardware appliances and database machines. A big task is ahead, evaluating the benefits and limitations of each model on deployment, isolation, extensibility, etc.  Higher level automation is knocking on the door… but a lot more DBAs will be required to manage the Exaflood.

“The information age is an idea that the current era will be characterized by the ability of individuals to transfer information freely, and to have instant access to knowledge that would have been very difficult or impossible to find previously” Wikipedia.

Computer miniaturization advances plus the internet reaching a critical mass is shaping the modern society. The social generation expects that second and third tier web to be 7x24: MySpace has supported over 4.4M concurrent users, gambling site bWin running 30,000 transactions per second. Data volumes are exploding with event data streaming from sources such as RFID, sensors and web logs – the ability to monitor, analyze and act before information gets stored provides opportunity to more a new “near real-time”.

The consumerization of technology is here but according to IDC, more than 20% of enterprise IT spending goes to acquire and service mission critical workloads.   This week, I started a new role at Microsoft Corp. as Director of Technical Product Management for SQL Server.  I’m very exited to lead the future of the database: where it runs today and and on highly balanced Appliances that integrate hardware (CPU, IO, storage), software reference architectures for specific workloads. e.g. the Fast Track Data Warehouse. Stay tuned for the PASS 2010 summit.

The application platform for the “Unlimited information era” will be the result of the many changes I will continue discussing on this blog…  PLEASE VOTE and collaborate of my proposed “Future of non-cloud databases and application platform” session at SXSW.

• An internet scale document living on the cloud has significant differences to the design for 140 characters. In addition to the vast amounts of information being produced, it’s getting harder to get “all the data” stored on memory. This means there is never a consistent global view of the whole structure. Implementing an “eventually consistent” version of data is a hard problem that impact the design and responsiveness of future applications. For example, Microsoft Outlook implements an <offline> mode – simulating expected results on the client requires additional logic. It works very well. Most twitter clients wait for user actions to take place, instead of doing the work on the background – however the operation might actually never take place on the cloud. More on this later.
• Social computing is allowing many users to participate on collaborative creation, this means that data structures need to use more unique identifiers so versioning and consolidation can take place. There is a significant performance and UI impact for building these multi-screen applications. Envision how memory requirements would affect excel for adding multiple versions of each cell.
• The Social generation will continue releasing additional personal information, combined with location and activity. See data life of the future. A key way to combat the information overflow is to display information that is relevant on very specific moments. We should expect this functionality on future e-mail and communication clients. Acquiring, tagging and classification will continue relevant.
• It’s expected that applications will leverage information from many sources as well as information on public cloud databases (Dallas, AWS). Pay attention to the emerging OData proposal. Toby Segaran published a great presentation on Social Data analysis.

The problem goes beyond data to visualization, we can argue that after 25 year of the existence of Microsoft Word, most users cannot produce a good looking document. Three months ago I blogged on storage limits. Emerging topics also include proposals allowing cloud developers to control their data travels across devices.  Yes, the Exabyte age is just arriving…

Let me share with you the perspective that I have as owner of the Microsoft Cloud initiative for Latin America:

• On March 4, 2010 Steve Ballmer announced that moving forward, the Cloud was the #1 priority for Microsoft. There were no service announcements, but only the aspiration to bring the false perceptions to reality and to lead the "enterprise cloud”. Previously it was not our “top” priority.
• All industry players and companies agree to three benefits the cloud brings: Agility as #1, a new economic model with no upfront costs as second and finally getting rid of deployment and management functions that do not add value. Secondary benefits include mobile access, lowering carbon consumption and others.

Microsoft is the only Cloud provider that:

1. Provides flexibility to run in-premise or on the cloud.

o Traditional Enterprise IT vendors want to enable the datacenter so that non-IT departments can consume services per hour/terabyte.

o Internet companies want to move “everything to the cloud”. They cannot offer a “dedicated cloud” for specific customizations. Microsoft recommends to companies over 15,000 seats to evaluate dedicated cloud inside Microsoft datacenters. Mode details to follow.

o Only Microsoft – the leader on software business – offers the capability to execute processes in-premise, on private or remote private cloud or on a public cloud.

2. Microsoft is that only company that has solid offers across IAAS, PAAS and SAAS. Sure, you can choose SaaS offerings from multiple vendors, but how are they going to be integrated long term? Sure, you can choose an IaaS provider but how well integrated is it to their SaaS offerings?

o IaaS – Infrastructure as a Service. Microsoft enables enterprises to provide software services and computing/storage per consumption with two main offerings: virtualization and physical+virtual systems management. The competitive offer costs at least 6 times more but the channels makes a lot more money selling it…

o PaaS – Platform as a Service. A new level of abstraction and automation beyond IaaS enables to build custom and commercial solutions for the cloud. Targeting to developers using .NET, Java and PHP among others. If you are building new solutions, you must consider it.

o SaaS - Software as a Service. Microsoft software as a service portfolio is already extensive, but let’s focus on Microsoft BPOS (Business Productivity Online Services): It includes e-mail (Exchange online), collaboration (Sharepoint Online), web conferencing (Live meeting) and real-time communications (Office communications Online). Starting at $10 per user per month and down if provides great performance, service level agreements, reliability, control, end user high productivity and great mobile devices support.

3. Microsoft leads the “enterprise cloud”. We believe that the consumer cloud differs significantly from the enterprise cloud. On the first one, Microsoft has over 15 years experience: 600M users, 369M hotmail users… Most IaaS competitors cannot demonstrate they operate at this levels. Let’s focus on the businesses

o Microsoft already has more than 1.4M paying customers. 7 of the top 10 energy companies, 13 of the top 20 telecos, 15 of the 20 top financial institutions and 16 of the top 20 pharmaceutical companies already have chosen Microsoft Online. More than 500 government entities, 50% of Fortune 50 companies… Over 40 million paid seats. Other SaaS providers have many “free” customers but limited paying customers.

On Latin America we launched the Microsoft cloud services were made available April 9, 2010. Customers on Argentina, Colombia, Mexico, Brazil were running production systems on top of our Windows Azure Platform that same day. We exceeded 10,000 SaaS paid seats on the first three weeks. I’m currently responsible for the Cloud readiness of over 1000 employees in the region – delivering the message and benefits to ALL our customers and partners in the very short term is the next step but… Microsoft already leads the enterprise cloud.

Some frequent questions:

How do I get ready to move to the Cloud? As you do today, needs and requirements need to be understood. The existence of a management and identity infrastructures will be important. Move low risk applications immediately. Activate your developers to Windows and SQL Azure.

What about Cloud security? Nothing connected to the internet can be considered 100% safe. Microsoft relies on third party certifications (SAS, ISO, FISMA…). Read more on privacy. Some leading SaaS providers DO NOT allow third parties to inspect their datacenters.. Security suggestions for software developers.

Do I own my data on the Cloud? Each customer owns and controls their data. Microsoft doesn’t access, modifies or deletes information unless required to do so. Open Data Protocol (OData) and vertical/government specific (e.g.  OGDI) allow for publishing data on a consistent format. Furthermore – our commercial model is not based on advertising or mining of user data.

What if the Cloud is not available? Service level agreements are important, we offer 99.9% and above financially backed plans. Cloud only is NOT “good enough” for many customers.

Does moving to the clouds means immediate savings and higher productivity? Not for all. Many large customers have the infrastructure, people and processes to run their IT at a very low fixed cost. But most do not. In general there is immediate productivity because we offer the same software on-premise now on the cloud.

Will my current application run on the Cloud? We are working to deliver a consistent software development model. Today Windows Azure offers a new layer of abstraction and automated management. Microsoft offers great interoperability HTTP, XML, SOAP, REST for connecting cloud services to in-premise solutions.

What is the licensing model for the Cloud? Another unique advantage is that Microsoft offers licensing that can cover in-premise, on the cloud or hybrid users in a single model. Our customers decide when is the right moment to move to the cloud.

• Cloud computing. No need to provide more details on this one, benefits outweigh problems.
• PCs took over 10 years to arrive to many desktops at global organizations. Many visionaries think that the mobile transformation will take less than 5 year. All agree that portable devices will be the most common way to get to what’s left of any Intranet.  “The New world of wireless” from Scott A. Snyder goes further detailing the org chart effect organizations might experience.
• Social Media, customer communities and social networks are main communication channels. However Twitter, facebook and other do not really have an enterprise product. Expect more than analytics.
• Enterprise Search and Retrieval (ESR), Master Data Management (MDM) and Enterprise Risk Management will increase relevance.
• Innovation from virtually collocated teams will continue growing.
• Great coverage of interoperability.
• Green IT will be adopted for three main reasons: cost savings, government policy and end user demand.

All the above topics under a single umbrella. Simon is only missing more comments on NUI, Augmented Reality and geo-location from my perspective to have one of the best ever compilations.

B Side. The next challenge is deeply understanding the impact the industry will live given the re-tooling of the enterprise. I hope to be wrong but most likely it will be  painful:

• Several visionaries predict that hosting companies on developed countries won’t be able to compete with industry “gorillas”
• Software as a services eliminates a lot of “non-value” adding channels
• The IT profession will evolve to higher forms, e.g. Business Intelligence instead of installing/managing a RDBMS.

Let’s move now to the real Enterprise 2.0 era and the ecosystem for the next decade.

• Less than 4 months after commercial availability of elastic computing there were over 22 definitions for Cloud computing. Today, there is still no common understanding for the concept but agreement that it has gone mainstream.
• “The state of the internet operating system” from Tim O’Reilly clearly describes the current reality: We all use the cloud as consumers and soon most organizations will be a hybrid of internal and external IT services as predicted by industry analysts.

This only means that the wait is over, the cloud is Web 3.0. Probably we should call it the “complex cloud” or “complex web” because it is very unstructured.

Sorry Tim: Semantic is fantastic but it has not crossed the chasm. Often we cannot tell what we really want to tell…

The web 3.0 is defined by interoperability; thousands of screens (different sizes) and applications on many clouds. PAAS is the real enabler of SAAS… and not the other way around:

ReadWriteWeb declared “Web 2.0 is dead” on December 2005… they missed BusinessWeek’s 1991 article Old software never dies…  The web 2.0 is great; it has helped consolidate the Social Media.

The web 3.0 is already here: It’s called cloud computing. We still need a breakthrough on software development: “Until the software equivalent of a bridge collapses sw engineers aren't compelled to a higher form of excellence”, until then, the apps will slowly continue flourishing.