SPARK

  • Article

The weekend before last (the weekend before Mix06) I attended the Microsoft ‘SPARK’ conference at the Venetian in Las Vegas. SPARK was billed as chance to explore and discuss the issues in the evolving architectural landscapes of Web 2.0, SOA and Software as a Service (SaaS). The tag line behind SPARK was that it was ‘as much about understanding the questions as the answers’. And so it was too – I’m not sure we got too many definitive answers but we sure did raise a fair number of important questions, and developed models for looking at these issues that might prove useful…

SPARK

SPARK was attended by a diverse collection of very impressive people, including CTO’s, CEO’s, senior architects, entrepreneurs, journalists, and analysts – about 35 in total. The format of the event was very interesting too. The group was guided through various stages of the discussion by moderators. For each stage we split into small groups to brainstorm ideas and opinions, followed by a free-form discussion between the whole group. Very interactive and not a PowerPoint deck in site!

We were first invited to think about the various environmental drivers that were driving changes in the architectural world. We then prioritized and grouped them so we could see the commonalities and trends between them. Then we proposed a number of models that could be used to describe the issues in a nice graphical way. Finally, the team rationalized all of these models into a single uber-model. Mike Platt of Microsoft then presented this model, and the thinking behind it, at the Mix06 conference itself.

On the whole, I thought the format worked pretty well. Some attendees wanted to dig deeper into technology, some wanted to stay focused on business or social aspects but I think most everybody got something valuable from the experience. I certainly learned a lot of stuff I didn’t know about – the Web 2.0 trends in the industry are more powerful and far reaching than I had first thought. The next 5 years are going to be pretty interesting…

Check out the SPARK blog at <sparklasvegas.wordpress.com/> and photographs from the event on Flickr <www.flickr.com/photos/dionh/tags/spark06/> (I’m the handsome one at the back J).

Web 2.0 != SOA

So what did we spend two days discussing? The majority of the conversation was centered on Web 2.0, and how it meshes with the world of SOA. Are these two sides of the same coin, incompatible oil and water technologies, or alternate paths leading to the same destination?

In some ways, Web 2.0 and SOA represent a clash of world views. Both are underpinned by standards of course, but they solve different problems and in different ways. Web 2.0 is very much about the dynamic, interactive, nature of the web, supporting and enhancing social computing and communities, and providing the end-user with the flexibility to create the interactive experiences they want. SOA on the other hand is about maintaining some level control over a distributed solution, be it in terms of transactions, security, or the careful coordination of interactions between disparate systems that can now interoperate with each other. So when would you to use one technology over the other and what happens when they need to be integrated?

Software as a Service, on the other hand, is about new ways of delivering applications or functionality by leveraging Web 2.0 and/or SOA. SaaS has its own set of challenges of course (such as the integration of SaaS applications with an organization’s other systems, management of service level, reliability and security, etc) but it essentially builds on the capabilities of Web 2.0 and SOA. It is my belief, though, that SaaS will not be able to take off properly until the issues surrounding Web 2.0 and SOA are addressed…

Phase I – The Environmental Drivers

It’s widely recognized that there are big changes underway – changes in the way consumers buy, sell, explore, collaborate and communicate, and changes in the way businesses relate to these consumers, advertise, monetize and operate. We started looking at these issues by trying to identify and understand the environmental drivers – i.e. what factors are driving these changes and what do they mean for businesses and consumers?

Many, many factors, issues and ideas were put forward. I’ve been pondering these for the last week or so and in my mind I categorize them into four buckets – User Experience, Web Critical Mass, Business Agility, and Business Practice. I have listed all of the factors that were discussed in their respective groups below. There are overlaps between some of these and I haven’t tidied them up too much, but I hope you can get an idea of the kind of issues we discussed.

User Experience

  • Users want it now, want it to be rich and want to have control over it.

  • User experience is a key differentiator for an organization but it is becoming increasingly difficult to maintain control over it.

  • User experience drives a closer customer connection.

  • There is a lot of creativity and buzz in the public domain.

  • The ability to search, find, use, and re-use is critical.

  • The ability to collaborate and share content and functionality is important.

  • The advent of rich media.

  • There is a lowering of the barrier to entry – End-users can create and manage their own experiences (this changes who is doing the architecting).

  • Control of identity – Protect me and my data.

  • Generation Internet – Many users were born into a connected world.

  • The Disruptive Innovation of Technology.

  • The Attention Economy.

  • The limitations of web technologies.

Web Critical Mass

  • The web is reaching critical mass – Both in terms of number of connected people and in technology and capability.
  • The ubiquity and maturity of the web ecosystem.
  • Pervasive Web API’s – But publicly exposed API’s can lead to unexpected exploits/usage that was not intended.
  • Increasing bandwidth.
  • Increasing hardware capabilities.
  • Higher level frameworks make building apps easier (interesting side question – “does architecture changes drive frameworks, or vice versa?”).
  • Managing standards & standards compliance.
  • Facing the trustworthiness challenge.
  • Creating and managing internet scale apps – including rapid growth.
  • Increase in using the Web as a platform for business.
  • Dynamic mediation.
  • The increasingly social use of computing.
  • Managing the diversity of connected devices.
  • Eco-system computing – Viral, community driven, social networks.
  • Managing and leveraging user generated content and meta-data to drive business intelligence.

Business Agility

  • The Long Tail!! – See <www.wired.com/wired/archive/12.10/tail.html> and <www.thelongtail.com/>.
  • The drive to reduce costs and to seize new opportunities.
  • Shortened development cycles.
  • Accelerated change, accelerated pace of business activity.
  • Business Impatience – Another type of BI!
  • Flexibility and Agility.
  • Agile development culture.
  • Build to change, rather than build to last.
  • Need to scale up quickly (possibly very very quickly - MySpace are adding 250,000 users a day!).
  • Re-Architecture – Short shelf life of IT solutions, need to design for change from the ground up.
  • Interoperability – Re-using and integrating existing systems.
  • Growth in the sheer quantity information leads to challenges in search, management, spotting patterns, managing relationships, etc.
  • Asymmetric demand – The tipping point.
  • Good enough tools/technology versus innovation.

Business Practice

  • Compliance and legal/management oversight.

  • Business Model Innovation – The need for new business models and ways to remain profitable (the long tail, again).

  • Monetization.

  • The shift of control – Movement of management of identity, trust, relationships to the edge.

  • Licensing, Open Source.

  • Control vs complexity – How to maintain a level of control without increasing the complexity.

  • Serving micro markets – The long tail, again?

  • Security threats – Protecting the organization and customers from security threats.

  • Protecting the organization’s assets and reputation.

  • The inflated expectations of standards.

  • Traceability, manageability, reversibility, accountability.

  • Value Chain Optimization.

  • Balancing risk and opportunity.

Phase II – The Models

So given the factors described above, we started to put together a number of conceptual models to describe the space in a way that architects could easily digest – a whiteboard friendly nice simple picture.

When I look at the drivers above, it seems to me that there are a number of opposing forces at work. On the one hand, there are the exciting opportunities afforded by Web 2.0, leading to a tendency to push more stuff to the ‘edge’. On the other hand, there is a need to balance that to maintain organizational control, leading to a tendency to pull stuff into the center. The edge represents the Web 2.0 world where interaction occurs through lightweight protocols that do not offer much in the way of control or reliability. The center represents the SOA world where interactions occur in a much more controlled and reliable manner. Typically (though this is an overly simplistic model), you can think of the edge as outside the firewall, and the center as behind the firewall.

So what forces are pushing an organization to open up its data and services on the edge?

Connectivity is becoming ubiquitous and more and more people are comfortable to work, shop and find entertainment online. There is a huge (and growing) connected market out there but the ‘old’ ways of connecting to that market are becoming less and less relevant – for example, delivering static content on pre-defined and inflexible schedules is just sooooo old fashioned.

 

Connected consumers are kind of getting used to having things on their own terms, when where and how they want them. Connected consumers need effective ways of finding goods and services which can themselves (increasingly) be delivered to them over the network. And the ability to easily find and receive goods and services allows an organization to more effectively leverage the long tail.

There are other business benefits to leveraging the edge too. Managing large online communities effectively can create a synergistic feedback loop – think of Amazon, e-Bay, Google, MSN. All of these take the data and/or content from interacting with many, many customers, aggregate it and use that data to drive more traffic, provide better results, more targeted advertising or more accurate recommendations. Managing such large communities need not be a huge burden on the organization either, for example X-Box Live and e-Bay both effectively demonstrate self regulating communities.

All of these opportunities can enable an organization to gain significant business advantage. So what are some of the forces slowing this headlong rush into this brave new world?

All organizations face a number of universal business constraints. For instance, an organization is typically required to be profitable and has to protect itself and its customers from legal and ethical issues.

A flexible organization can usually find a way to monetize its operations by employing a flexible business model that takes advantage of the Web 2.0 world – the long tail again! But this requires a certain level of agility within the organization that may not be easy to attain. An organization therefore might have to slow down pushing stuff to the edge in order to maintain its internal consistency.

A more important issue, though, is the requirement that an organization protect itself and its customers. It has to protect its reputation, its IP, its brand, its data, its processes, and all of these things on behalf of its customer and partners. It has to maintain compliance, security, and trust. This really comes down to an issue of control. If an organization exposes too many services and data to the wide world without effective controls in place, then it can be opening itself up to a lot of risk in these areas. Hence there are forces at play that pull stuff into the center where more control can be exerted.

Similarly, all organizations have to differentiate themselves somehow and increasingly this means by providing the user experience that consumers want, but in such a way that clearly is associated with the organization and/or its brand. Being able to control the user experience becomes very difficult when applications are loosely composed from data and services provided by many different organizations. In fact, in the world of mash-ups, the concept of an application with a specific user experience becomes vague. You can effectively create a new experience by recombining the same or similar services from different places, throw in a bit of data munging so you can strip out unwanted stuff and you are away.

Of course, it’s not to say that an organization can’t expose data and services in such a way that unwanted usage can be prevented, but this effectively requires a SOA like architectural approach so that control can be maintained. An organization can even create an application that consumes data and services provided by another organization but if it is to maintain a certain level of reliability and security, then it must have service level agreements in place with the other service provider, which again requires a level of control. These are examples of situations in which the “SOA inside the firewall, Web 2.0 outside the firewall“ does not hold true.

Given all of the factors above, there is clearly some value in opening up data and services to the edge but it has to be done in a way than maintains enough control. The interface between these two worlds is interesting. One question is - what architectural capabilities are required to manage this interface? Our small group came up with four basic capabilities that are required to help balance these two forces:

  • Discovery and Mediation – The ability for consumer or external organization to discover and use the services that you offer and to connect to them in an appropriate way that balances control with openness.

  • Community – The ability to maintain and foster a vibrant, dynamic and rapidly expanding community and to keep that community focused within the organization’s business goal.

  • Trust and Relationships – The ability to manage trust and relationships between organization, systems, and elements of the community to ensure adequate control and protection of the organization.

  • Rich Content – The ability to manage and deliver rich relevant content in an appropriate way that balances control with wide reach.

The Spinning Disk Model

So how to compress all of those concepts, issues and factors into a nice simple picture? I proposed a spinning disk model, shown below. The center of the circle represents the core business capabilities that are typically based on a SOA architecture. Around this core are the capabilities described above. Beyond those capabilities are the Web 2.0 capabilities, and communities of consumers and external organizations.

You can imagine that the disk is spinning so that the opposing forces can be depicted as centrifugal and centripetal forces. Other disks represent similar islands of capabilities in the outer ether that are available to form applications or bridges between these islands. OK the analogy isn’t perfect since islands don’t really spin, unless you’ve had a lot of rum, but I hope you like it anyway.

You can see a photograph of the original whiteboard version here and here [<www.flickr.com/photos/dionh/114941232> and <www.veryard.com/so/2006/03/spark-workshop-3.htm>]

Phase III – The Rationalized Uber-Model

Other models were proposed by the other groups (but they weren’t circular L). The group set to and rationalized all of the models into the single model shown below

Mike Platt collated all of the pertinent details of the discussions over the two day event and presented at the Mix06 conference. You can find the PowerPoint deck he presented here: <www.looselycoupled.com/ev/Spark.ppt>. I knew we couldn’t escape PowerPoint for ever J.

Web 2.0 + SOA + SaaS = Web 3.0?

My view going into SPARK is that Web 2.0 is really about AJAX, which in turn is all about bringing something like a usable user experience to browser based applications, and that while AJAX is an important technology in this regard, it doesn’t change the fundamental model for the web and we still have the same problems when building applications on a platform that was not fundamentally designed as an application platform (hence the need for SOA and SaaS). So I find myself wondering what the web application platform will look like in 5 years time when these issues are addressed – I guess you could call it Web 3.0. Since Web 2.0 will eventually evolve into Web 3.0 anyway, I attended SPARK hoping to understand more about the Web 2.0 world, and to understand the emerging trends and the drivers for them and to ultimately to understand what Web 3.0 might look like.

It turns out that my simplistic view of the Web 2.0 world was, err, too simplistic anyway. A large part of the Web 2.0 phenomenon is indeed about improving the user experience, and AJAX is an important part of that, but it seems to be more about the convergence of ubiquitous connectivity and lightweight protocols to create dynamic communities and the flexible recombination of diverse content and services into useful applications. It seems there are challenges though, such as how to build a wide reach web application that leverage the dynamism of Web 2.0 but in such a way that allows an organization to build a compelling business model around it. These issues may be partly addressed by SOA and SaaS but they in some ways fly in the face of the Web 2.0 philosophy, making their incorporation into the Web 2.0 world challenging.

I believe that from the solution to these integration challenges, Web 3.0 will emerge…