What’s to be done?

Bank’s today are sitting on a ton of data, much of it about you and me. This data is incredibly valuable. But years of acquisitions have led to a patchwork of systems and technology solutions that don’t sit well together. Plus many database systems are designed around standard predictable questions we always needed answers to not the new questions that are driving our business more and more today.

The result is data is hard to access let alone analyze. Even though it is valuable, it’s really hard to get.

Plus even more data is coming down the pike – and unstructured data at that which bank systems are ill equipped to handle after years of investing in structured, relationship based systems. Plus what is data today? Is it words, numbers, images, voices, gestures, presence?

Classifying the data is a great place to start. There is high-value data we need to run the business now and lower value data we may need in the future (or may not – we just don’t know yet). Data we need now should be close to the line of business. Not far away in some remote  central repository. We have more flexibility over lower value data. As long as we can get it when we need it.

This suggests a federated approach to data management. A centralized, single database model assumes all data is equal in value which, of course, it isn’t.

Once we have our data story in place, the technology and security story we need to support it falls more easily into place. Tools like Hadoop, Hive, SQL, analytics and the Cloud each have a clearer role to play and data will flow more easily to where it’s most needed.

And what’s the prize for getting this right?

Many banks have had to cut back and grow smaller since the financial crisis of 2008. One exception is Great Western Bank that has grown 300% since then mainly by harnessing data Big Data tools to understand their customers better and service them individually.(1)

But have some features become so generic, such as pinch-to-zoom, as to make it impossible to produce a smartphone without them? If true will competition suffer?

The answer depends on how consumers react. Will they substitute cool for safe now the market has become less predictable? Or will consumers see an opportunity to express their individualism in the devices that define them and embrace separate choices?

On the one hand picking a safer phone – one not subject to controversy – has become more important.

But individualism and personalization are on the rise. More and more we are defined but the devices we have and the lifestyles they support. Almost every industry has flourished by offering consumers choices and avoiding standardization.

What about the supply side of the equation? Some technologists may now be more fearful in bringing new ideas to market, now the bar on what is different is higher. But the more a product looks different the stronger their defense will be.

In truth, the smartphone industry is still in its early stages. Just like cars might have looked very similar in the thirties and forties so smart phones often resemble each other in our own time.  Sometimes it’s hard to tell them apart without close inspection.

Windows Phone is certainly a separate point of view a distinction even more reinforced when Windows 8 comes to market in October. Also smartphones will not long survive as standalone devices but companion tools in a world of connected devices and applications where the platform becomes more important.

The smartphone industry is a young one but was already beginning to coalesce on devices that looked more and more standardized. The Apple verdict celebrates the importance of a separate point of view reinforcing the importance of creativity in a market that desperately needs it.

Yet it is the very transparency of the internet and the information society it is spawning that is transferring power from banks to their customers. In today’s always interconnected world of unlimited newsbites customers often know more about their banks than banks know about them.

In our down market there has been a shift in banking towards customer intimacy and away from complex products. What’s the point of a synthetic derivative if no one wants it? This represents a 180 degree transformation in the industry’s technology focus at a time when budgets are really tight.

Ironically it is the internet and its related technologies, the tools that have put banks so much on the defensive, that offer a solution. The debate about whether social is better for branding or advertising misses the point. It’s one of the tools that help banks build deeper relationships with customers.

Thanks to the internet of things, an ecosystem of connected devices and applications, we are generating more data about ourselves than at any time in history. Every time we put our photos in the cloud, check in on a social networking site or trawl websites we leave behind digital clues as to who we really are, what relationships we have and lifestyle choices we make.

We are creating a treasure chest of information which, when combined with what banks already know about us, offers huge insight into us as individuals.

Add to this mix the power of analytics and banks can predict with increasing certainty how we will respond to individual campaigns. Instead of receiving blanket offers we may have little use for we can start to receive relevant ones that may actually improve our lives. Even in as basic an area as communications, content can be personalized to our individual needs.

Does the customer want their statement electronically or in paper form? How frequently do they want those statements? Are they interested in savings products or borrowings? Are they about to change jobs or start a new business? Imagine being able to make relevant offers timed to lifestyle events.

In the old world of spray and prey marketing banks spend billions of dollars a year in postage alone and achieve little in terms of customer satisfaction, retention or even growth in customer relationships.

It is time for banks to overcome their fear of social, embrace the new world of market transparency and recognize the new opportunities they have been offered on a plate. Once banks do this perhaps then they will appreciate the real value of Facebook.

Have we forgotten that the real purpose of capital markets is to unlock long term value; not to satisfy the whims of short term investors? And yet here we are bashing the IPO and the company after just a few days. It's like judging a new born baby based on whether or not it looks sufficiently like its parents or siblings.

Sadly whether it is dotcom stocks, IPOs or houses we have come to expect to make money by flipping assets rather than investing in them. A boom and bust society has been the result with little growth in equities. It's hard to build long term wealth with a short term perspective.

This has other implications.

The long term transformational impact of technology generally exceeds the short term hype. In the case of social networking we have seen much hype but are nowhere near realizing the full potential.

Why is this potentially dangerous?

Because we often fail to see the full benefits of technology and so delay its implementation.

We recently passed an important milestone; the ability to store and compute almost anything. This means that many challenges we once thought impossible may now become attainable. And through ventures that may or may not have the word social in them, but may still bring huge social benefits in healthcare, lifestyle even the management of our economy, we may transform the future of our planet.

These initiatives need long term venture capital not short term day trading.

Facebook is an exciting opportunity in a huge long term shift in our global society to wards greaser collaboration, freedom and transparency empowered by technology.

Let's try and put a short term value on that.

By implication, boomers are technophobes or at best neophytes. Future markets are more likely to be shaped by younger people than older ones. Many firms have set up Gen Y research, marketing and HR teams to help plan the future of their products and businesses. The size of this group already eclipses Baby Boomers and will soon represent half the workforce.

But is it really that black and white? The terms ‘Digital Native’ and ‘Digital Immigrants’ are already ten years old and technology has changed a lot since then.  Technology is easier to use, more accessible and more embedded in almost everything we do. It’s harder to be a digital immigrant today. Boomers may prefer email and millennials texting, but they both can function in a real time information society.

The productivity gap has become slight not systemic.

Digital literacy adds a further dimension. Our participation in digital society varies significantly even within generations. Playing games and sharing photos as opposed to writing blogs and wikis suggests very different levels of digital literacy. The U.S. President is no millennial, but he has redefined politics through his sophisticated use of social networking and is reaching out to all of us.

Because of their broader use of technology, millennials may be wired differently than boomers which may make them better at multi-tasking. But that may make them less effective at deeper analysis and decision making. As we move to a more information rich, analytical society employees with deep analytical skills will be more in demand. But there may be fewer of them.

While millennials may be early adopters, Boomers are making a comeback. In May 2009 Microsoft and AARP sponsored dinners for 60 boomers in four cities – San Francisco, Phoenix, Chicago and New York – to explore what modern technology meant to them.[i]

Far from portraying a generation challenged by technology the boomers came across as ready to embrace it, but with their own unique perspective. They feared that their children were allowing technology to shape their lives rather than using it to help create the lives they want.[ii] Millennials may be early adopters of technology, but so too were boomers when they were young – after all this was the generation that produced Bill Gates and Steve Jobs. And that generation invented much of twenty first century technology – a revolutionary combination of flower and computing power.

Boomers have a unique perspective. Standing between their children – the millennials – and their parents – the Silent Generation – they can see both the dangers of over dependence on technology and the consequences of ignoring it. So they are more likely to have a more balanced view and use technology as a tool rather than an addiction.

Another Microsoft study that year reinforced this, focusing on financial services. While millennials were early adopters of technology, boomers were more pragmatic in their approach willing to use a portfolio of tools to achieve their objectives.[iii]

As technology has grown, its adoption spans the generations. Cell phones, including smartphones, are by far the most popular device among American adults of all ages. Again there were some differences. Younger users are more likely to use more functions than older ones. But that’s a long way from being a deep technological divide.

When it comes to social networking, Millennials may be the most connected generation in history. But, everyone uses social networks these days. Users 55 and older represent the fastest growing segment.

So what are the implications for banks?

Thanks to technology, there has been a big shift in power and influence from corporations and institutions to consumers and employees. We are less susceptible to traditional marketing campaigns and company messaging and more influenced by personal, shared experiences.

Thanks to the internet, we often know more about the corporations that service and employ us than they know about us. As a result, it is easier for consumers to compare one product with another and switch providers, and easier for employees to assess and discover new jobs and opportunities.

We live in a more and transparent, real-time and fluid society.

As a result, the competition for the best customers and employees is likely to accelerate. Banks in particular will have to make themselves more attractive destinations for both customers and staff.

In this context the diminishing technology divide between boomers and millennials is becoming less of an issue. Technology is embracing everyone. It is easier to use – more connected, inclusive, pervasive and embedded in everything we do. As a result customers both young and old will have more choices in how to engage with their banks. Similarly banks will have more opportunities to engage with customers.

For example, banking has traditionally been a face to face business. But ‘face to face’ means something different now. It may be a page on Facebook, a video call on Skype or an avatar on Xbox. Customers are more likely to engage with their banks through their smartphones, tablets, perhaps even from within their cars as automation embraces almost everything.

Thanks to the tremendous growth in TV content and the opportunity for interactive TV customers in the future may be just as likely to deal with their banks from their living room than from within the branch. Channels are likely to grow rather than shrink which increases the need for them to be integrated to encourage access to financial services from multiple sources and enable conversations in one channel to be continued in others.

Banks will have more information on customer needs. The growth of the internet and the plethora of connected devices increase the amount of digital information – Big Data – and opportunities to create targeted offers and campaigns. A new era of personalized eBanking will emerge.

In the past banks have had separate physical and digital strategies. This no longer makes sense. Banks will need to combine both physical and digital channels into a single integrated approach.

The bank of the future will be more analytical and less transactional, and more relationship focused. There is already a shift underway from building technology around products to building it around customer relationships and that will only accelerate.

To achieve this, banks and their branches will need new skills. A more sophisticated retail banking workforce will emerge with deeper relationship and advisory skills. This in turn will need new tools to manage it and new incentives to motivate and reward.

 Millennials and Boomers – poles apart or peas in a pod?  A fresh perspective is long overdue.

But sometimes when you think you understand a problem you realize you are looking at it through the wrong lenses. The shortages we face today as a planet in energy, clean water, medicine and nutrition are simply aspects of a single, bigger problem – a shortage of information.

With the rapid changes in technology everything we call information whether it’s a book, a film, a conversation even an awareness of someone’s presence is data that can be digitalized. With the proliferation of mobile computing and communication devices what has come to be called ‘Big Data’ is growing exponentially, 90 per cent of it created in the last couple of years.

But what has information, digital or otherwise, got to do with the fact that the planet is running out of resources or that nearly 11 million children die every year through under nourishment; lack of clean water perhaps being the biggest problem?

Since the dawn of civilization we have been collecting information on papyrus scrolls to silicon chips. But until recently our lives have been defined by a lack rather than an abundance of it. Think of the recent financial crisis – how much of us really understood the risks we were taking?

Now with our ability to store and compute almost anything that is all about to change. We are entering a golden era of computing and with that our ability to manage our resources and our lives better than ever eliminating waste.

In their book ‘Abundance’, Peter Diamandis and Steven Kotler point out that in America today 70 percent of our water is used for agriculture, yet 50 percent of the food we produce gets thrown away. Five percent of our energy goes to pump water but 20 percent of that water is lost through leaky pipes.

A water shortage is really an information shortage. By creating a smart water system that identifies how and where we lose it we can manage it more effectively.

An example is computer-assisted irrigation with GPS tracking and remote sensing technologies that allow farmers to know everything that is going on in their fields. This method of precision agriculture allows farmers to lower their water usage, target it more effectively and increase their productivity.[i]

When most of us think about Big Data we tend to see the commercial opportunities first.

And solving the world’s water supply is a huge commercial opportunity. But it is also a great human opportunity, a chance to lift the world out of poverty, to raise living standards by solving our most basic problems first – our  ability to help each other survive.

According to the Basel Committee, roughly two thirds of counterparty credit risk losses were due to credit value adjustment (CVA) losses and only one third to actual defaults.[i] Managing CVA volatility is one of the biggest challenges facing financial institutions still to this day.

Never mind the algorithmic and compute challenges involved, the acronyms alone are a lot to handle.

CVA – credit value adjustment – is the difference between the market value of a counter party exposure and a risk free position. If derivatives are cleared by CCPs (central clearing counterparties) then such contracts carry a CVA of zero. But if they remain within the bilateral structure of OTC markets – that’s a whole other ball game. CCR’s EPEs, EDEs, DVAs all add to the alphabet soup that basically sum up to the fact that this is not a topic for the technically challenged.[ii] In fact for much of Wall Street, despite the huge risks involved, getting on top of these exposures is still a work in progress.

CVA calculations are typically carried out by the trading desk and sometimes by risk and finance as well. But while finance, risk and trading may share similar models, trading rarely wants to share what part of the curve it’s on to the rest of the business. Plus, to the extent that different lines of business may use different risk methodologies creating the single version of the truth may become even more elusive. That makes the efficient allocation of capital across the enterprise more art than science.

By themselves Monte Carlo simulations are renowned for their complexity; exotics even more so. Calculating the multitude of internal and external scenarios can grow exponentially the more complex the derivative. It requires a huge amount of compute power which for many firms using standard technology can take too long.

CVA adds a further complexity – scenarios on top of scenarios.

New simulation technology is one response – simplifying the regression by reducing the number of scenarios.[iii]

But compute power is another part of the challenge. And not just to calculate the endless scenarios. As more and more data becomes electronic and accessible in digital form so the number of inputs into a scenario can also grow substantially. In analyzing the risks in any position we can not only reply on the huge amount of market information, we can tap social data as well.

Back in the days of open outcry traders in one pit could sense the buzz of markets by listening to the roar in other pits. With electronic trading that intelligence disappeared. To some extent it has replaced by 24X7 news feeds but tweets, blogs and internet activity can provide a ton of extra intelligence anticipating potentials movements in CDS spreads and CVA positions.

The pressure to deliver CVA calculations to tougher regulatory standards brings together two monolithic challenges that plague financial services today – the challenge of Big Data – and the need to access massive compute power at a moment’s notice.

The cloud is an obvious solution. It is in the one place where you can access compute power at the touch of a button. But while some firms are rushing to embrace it, others remain anxious about it. They worry about its security, in particular putting customer data beyond their firewalls. Bursting to the cloud avoids this risk keeping data on premises while accessing almost unlimited computing power at a fraction of the cost of in-house systems.

Most firms will prefer a portfolio of technology solutions – massive parallel process (MPP) allowing many servers to come together on a single problem; Hadoop to tackle the challenges of Big Data; and a range of cloud solutions – private (on premises), hybrid and public cloud provide a range of options for almost every market situation. The back office of the future has to be able to spin on a dime.

On April 10^th JP Morgan Chase, Numerix and Microsoft came together at the Harvard Club to share their perspectives these challenging topics with a number of industry practitioners. [iv]The good news is that most of the tools we need are readily at our disposal.

The challenges, as always, comes in implementing them and that means a true commitment to the analytical enterprise, not just within one line of business, but across the enterprise as well.

To support this mining of Big Data, a slew of new technologies has arisen. The most talked about is Hadoop. But what exactly is Hadoop? How should enterprises leverage it? And how should they not..?

A Schema-less World

Relational databases are at the heart of every Line of Business application today, and most web ones too. For good reason: they have tightly defined schemas, provide data consistency and support a well-defined query framework.

That said the technology can struggle with the variety and size of Big Data. The resources and effort required to store this type of information in a relational model can be significant. At the same time, databases are organized for anticipated queries, whereas businesses really need the potential to ask any question they want.

How Does Hadoop Help?

Historically business-critical data was well organized and the needs predictable. Today’s world is all about uncertainty. A fresh approach to data management is required, and Hadoop helps fulfil this need.

Hadoop at its simplest is a combination of a distributed file-system and a framework for Map/Reduce algorithms. Map/Reduce provides a means to perform analysis without moving the data (too cumbersome), and without requiring a predefined schema (not possible). The file-system stores large amounts of unstructured data in a cost effective, fault tolerant and scalable away.

Simply put, you can:

• Ask any question you want
• Perform the analysis where the data lives
• Look at a complete detailed picture (not a sample nor an aggregate)
• Efficiently store large amounts of varied data

What does this mean for Financial Services?

Examples of industry Big Data include raw tick data from financial exchanges, server logs from e-banking websites, and the output of operational and trading applications. In these cases, data is generated at high velocity, often in a raw yet-to-be-structured format.

Big Data tools can therefore be applied to solve some critical industry problems:

• Building a complete, single view of the customer for a more personal approach to banking.
• Constructing and back-testing complex trading strategies in order to beat the market.
• Calculating exposure accurately and in real-time to take risk management from an art to a science.
• Delivering holistic operational data management to create a more robust and reliable back-office.

These scenarios require the merging of varied data sources, and the ability to ask complex and detailed questions in an efficient way. An integrated data platform is therefore essential, and along with traditional relational databases, Massively Parallel Processing (MPP) appliances, analysis engines, and BI and visualisation tools, Hadoop has an important role to play.

Where does Microsoft fit in?

As part of the complete platform, Microsoft has optimised a distribution of the open source Hadoop project to shine on Windows Azure and Windows Server. In addition Hadoop has been made enterprise ready by integrating it with Microsoft’s systems management, database and BI platforms, as well as providing simplified tools to write and execute Map/Reduce jobs.

How Should Enterprises use Hadoop?

Some firms use Hadoop to test new forms of analysis on data before deciding whether to store it in their data warehouse. Others with heavy and frequent BI needs may leverage Hadoop for ETL (Extract + Transform + Load) workloads to process complex raw data before ingestion into a relational database or directly into a BI cube. Those whose business-needs require lots of ad-hoc analysis may use Excel as their front-end to directly query data stored in Hadoop.

Watch out!

Beware: Hadoop is not the hammer to make everything a nail. Queries are batch driven, not real time. It is not designed to be optimized for specific scenarios. There is no transactional model.

By combining Hadoop with other data technologies, firms can build a complete Analytical Enterprise. Leveraging internal and external Big Data to its fullest, they can gain a clear and accurate view of their world, make better decisions and leave the competition behind as a result.

What is Big Data?

The growth in data, digital data, has become exponential. Every day, we create 2.5 quintillion bytes of data. 90% of the data has been created in the last two years alone. This data comes from everywhere: from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and from cell phone GPS signals and electronic markets to name just a few. 

IDC estimates point to it doubling every other year.

This highly varied data, flowing faster and growing larger than ever before, is “Big Data”.

The tools to manage Big Data can be thought of in three parts:

• Consolidation and aggregation – the assembly of data
• Storage and analysis – the ability to access and analyze data
• Absorption – presenting data so it can be easily consumed and meaningful insight obtained.

Conceived in this way, Big Data technologies involve a platform approach from the creation of data to its final consumption. Self-service BI is an important part of the Big Data story. Data is useless if you can’t do anything with it.

What defines an Analytical Enterprise?

An enterprise that uses data to understand their customers, risk and operations more completely, redefining their business opportunity as a result, we call an analytical enterprise.

The value of Big Data lies in the fact that it offers businesses the opportunity to see everything and miss nothing. The technology of Big Data allows us to store and compute practically anything for the first time in history. The impact can drive significant increases in value across many industries including financial services. Customer segmentation and targeting, risk management, investment management, compliance and financial reporting; even managing the enterprise becomes easier when we have access to more, perhaps unlimited information.

One great example is from a large insurance company. By using sensor technology in cars to monitor driving behavior, and a Big Data platform to analyze the output, they can more finely price insurance risk. By taking a more granular approach they can outperform their competitors on price and portfolio quality.

Big Data is the challenge - Creating the analytical enterprise is the solution.

The true analytical enterprise is one that takes a holistic view of its data platforms and an end-to-end approach to data management. It has the ability to see the wood and the trees, viewing everything in context. With both a laser focus on detail and a landscape view of the organization, an Analytical Enterprise can see its risks and its rewards, giving it the upper hand amongst its peers.

For more information...

To support this mining of Big Data, a slew of new technologies has arisen. The most talked about is Hadoop. But what exactly is Hadoop? How should enterprises leverage it? And how should they not..?

The difference between ‘almost’ and ‘just’ might seem more semantic than systemic, but the reality is quite the opposite. The massive growth in technology in recent years is causing an explosive growth in information about ourselves, our lives and the world in which we live.

Thanks to the internet, not since the Great Library in Alexandria has so much information been stored in one place at one time and with far less chance of burning down. Of course it is not just in one ‘place’ but distributed across a mass of servers, nodes and networks accessible to all of us who hold an internet ‘library card’ (address) thanks to an open sourced software framework called Hadoop.

The implications of this massive explosion of data are huge. We call it ‘Big.’ And big it is, growing at 50% per annum according to IDC estimates and by even more according to others.[ii] But not just big in terms of growth, but also scale, because it covers our tweets, our wikis, our blogs, our photos, even our conversations and gestures thanks to dramatic progress in the way we engage with computers.

Big Data is different from traditional data in that it is more varied in format and structure, more volatile and often travels much more rapidly than traditional forms of information. It is also more valuable, because it provides granular information to marketers as to who we really are and offers new ways for corporations and financial institutions to engage with us with pinpoint accuracy.

In contrast’ traditional’ data is structured, machine readable (by traditional machines) and resides on data bases organized around planned queries – questions we thought we wanted answers to in some cases years ago. But in the last few years, we know the world has changed dramatically.

We have new questions and they are growing all the time. As a result, much of the way corporations are approaching data management and data warehousing in the past has to be revised.

Just a few years ago we believed free markets were self-correcting without knowing as much about them as we thought we did. Now we can manage financial markets in real time with the precision of a Swiss watch. We have the technology to achieve this, but not always the will to implement it.

For those of us who believe many of the world’s problems – war, diseases, poverty, hunger – are largely caused by ignorance, Big Data promises us a new Utopia.

The time has come to grasp it as quickly as we can.