In the first case, I was tasked with guiding development of an executive-level paper and presentation for CIOs faced with the challenges surrounding Mergers, Acquisitions, and Divestitures – common business transactions that can easily derail an organization for months even as it seeks to gain synergies directly related to executing the M/A/D in the first place..... my student team was bright, and as is the norm I built an advisory team of Microsoft-internal business architects and consultants to review our work as it progressed. Read more about this project in my CIO's blog: http://blogs.msdn.com/b/mscio/archive/2012/03/08/a-mergers-amp-acquisitions-playbook-for-cios-part-1.aspx

More recently, just this week my newest student team gave me and my stakeholders a mid-project update on some work we're doing around Social CRM. We're looking to create another CIO-level paper and summarized PPTX deck with actionable advice and a fresh perspective.

If my student teams were simply to stick to the plan and deliver the papers I need, I'd be quite happy. Mission accomplished, right? And they'd be happy with their As and the chance to include a nice little write-up and couple of extra bullet points on their resume.

But here's the thing, and why I'm writing at 3:00 in the morning. Both of my student teams saw opportunities to take our projects to the next level, to a different place, beyond the picture I painted for them. And they did so effectively – in the most amazing way. Now, I'm not a guy who needs to take credit for things I have no business taking credit for, but I have to admit that I set the stage pretty well. I pushed the teams to think beyond the obvious, beyond the research, beyond what everyone else was saying. I didn't want reworked advice and tired axioms published again and again over the last 10 years. I especially didn't want to echo what the analysts were already telling my customers. I demanded – in the nicest way of course – something new. They just over-delivered.

In the first case, my team and I were unhappy with the quality and depth of data we were finding in the contemporary M&A space. We needed something new, and we found that in part by developing and delivering to current and former CIOs a custom survey aimed at identifying real-world lessons learned and if-only-I-could-do-it-again advice. Simply having such primary research at our disposal made our paper much more credible, and yes, full of some very nice nuggets of CIO wisdom and perspectives. But my students took the project further and developed a first-of-its-kind comprehensive M&A framework. Not only is it absolutely unique, but given its chronological orientation, the framework serves as an actionable roadmap as well. I only hoped for a decent paper, but in return I received a differentiating piece of Intellectual Property vetted by CIOs for CIOs. Amazing. And they delivered a great paper and supporting decks (executive-level as well as a very detailed one). Not bad for a bunch of kids..... :). 

In my most recent experience this week, my team has again surprised me. Different team and different students, mind you, but the same promise of shock and awe. We're only half-way through the Social CRM project and the team has already produced a model that captures and describes the key social platforms in terms of potential revenue impact, time-to-value, and more. The step-wise nature of the model seems destined to provide CIOs today with the kind of carefully aligned insight and actionable guidance necessary to not only identify and navigate the smartest path forward but also to develop the underpinnings of the business case necessary for pursuing that path. I wanted a paper....but I'm convinced I'm going to get a whole lot more. Again.

Sure, we can look at these project scenarios as an example of effective collaboration bounded by decent project management and guided by the maxim 'show me something new.' But it's more than that. My students are looking at the same problem space I am, but they're synthesizing it differently. As a leader, I might like to think that my team is being enabled, empowered, and positioned for success based on what I've provided them. And yes, I'm pretty helpful. :) But at the end of the day, I'm really standing on the shoulders of my students. They're the giants, and my customers will benefit mightily from their fresh perspectives and our combined work. As for me, I'm just grateful for each of them and for the chance to be part of developing something truly innovative, and truly inspiring. Imagine what they'll accomplish as they move into our workplace and transform it in front of our eyes. Very cool times ahead of us, for sure.....  :)

Take care all,

 George

An organization can take several precautions to help secure a database, such as designing a secure system, encrypting confidential assets, and building a firewall around the database servers. However, in a scenario where the physical media (such as disk drives or backup tapes) are stolen, a malicious party can simply restore or attach the database and browse the contained data. One solution is to encrypt the sensitive data in the database and protect the keys that are used to encrypt the data with a certificate. This prevents anyone without the keys from using or accessing the data, but this kind of protection must be planned in advance.

As of SQL Server 2008 a new feature called Transparent Data Encryption (TDE) was introduced into the RDBMS product. The new feature differs significantly from Data Encryption introduced with SQL Server 2005, which could be used to encrypt data of certain columns in tables. Microsoft received the following feedback from customers and partners, especially from larger ISVs:

• The encryption technology should be transparent to the underlying applications. The emphasis of such a technology would be less on protecting the data from user access, and more on protection from media loss (backup tapes or disks) or media theft.
• Data encryption should cover the entire database.
• Data encryption should be extended to database backups. 

Transparent data encryption (TDE) performs real-time I/O encryption and decryption of the data and log files. The encryption uses a database encryption key (DEK), which is stored in the database boot record for availability during recovery. The DEK is a symmetric key secured by using a certificate stored in the master database of the server or an asymmetric key protected by an EKM module. TDE protects data "at rest", meaning the data and log files. It provides the ability to comply with many laws, regulations, and guidelines established in various industries. This enables software developers and more specifically system administrators to encrypt data by using AES and 3DES encryption algorithms without changing existing applications.

Two very important technical and operational aspects to consider before implementing SQL Server TDE, is the impact on database performance (CPU and I/O) and the management of certificates and private keys. Activating TDE will directly impact the database performance to varying degrees, depending on which encryption algorithm is chosen. This impact is discussed in more detail in Step #3 below.

Additionally, the management and backup of database certificates and keys are vital to ensure that the encrypted database can be restored in the advent of a disaster. If you cannot retrieve your database certificate and private keys, the data is lost! Step #7 discusses the procedures for creating file level backups of these keys and certificates. SAP fully supports TDE on SQL Server databases and OSS note 1380493 can be referenced for more information.

Step by Step Procedure 

Step #1 – Verify SAP & SQL Server

Verify SAP & SQL Server before starting the TDE procedure. Login to SAP and execute DBACOCKPIT to display some basic database information thus verifying the database is open and working satisfactorily.

Next, using SQL Server Management Studio to verify the database options showing that encryption (TDE) is not enabled.

Step #2 – Full Backup

Execute a full database backup including transaction logs before starting the TDE procedure. Using SQL Server Management Studio or any 3^rd party backup software to start and execute the full backup. Be sure to provide a descriptive label including the text that this backup is not encrypted.

 After the backup completes, verify with DBACOCKPIT and/or SQL Server Management Studio or any 3^rd party backup software.

Step #3 – Encryption Type

Before proceeding, a decision must be made regarding the encryption algorithm that will be used. Microsoft supports multiple encryption algorithms which have different levels of performance and encryption. The table below depicts a simple load comparison conducted using various encryption algorithms and base lined with no encryption.

Supported encryption algorithms are AES with 128-bit, 192-bit, or 256-bit keys or 3 Key Triple DES (see Database Encryption in SQL Server 2008 Enterprise Edition at the end of this post for more information).

The Microsoft LoB CoE recommends using AES_256 or lower for SAP systems in general. Special consideration must be given to systems which already are I/O bottlenecked because of the increased encryption overhead caused by the data getting encrypted/decrypted in the I/O path. Therefore, servers with low I/O and a low CPU load will have the least performance impact. Applications with high CPU usage will suffer the most performance loss.

Selecting a higher encryption level will exacerbate this degradation and large scans which mainly happen on disk will be slowed down. In particular, TRIPLE_DES has significant higher impact than one of the AES algorithms.

Step #4 – Stop SAP

Before starting the encryption procedure, we recommend that all SAP applications utilizing the database be shut down. Stopping the SAP application prevents any DDL statement from being executed from the SAP data dictionary or via SAP transports.

The following operations are not allowed during initial database encryption, key change, or database decryption:

• Dropping a file from a filegroup in the database
• Dropping the database
• Taking the database offline
• Detaching a database
• Transitioning a database or filegroup into a READ ONLY state

The following operations are not allowed during the CREATE DATABASE ENCRYPTION KEY, ALTER DATABASE ENCRYPTION KEY, DROP DATABASE ENCRYPTION KEY, or ALTER DATABASE...SET ENCRYPTION statements.

• Dropping a file from a filegroup in the database.
• Dropping the database.
• Taking the database offline.
• Detaching a database.
• Transitioning a database or filegroup into a READ ONLY state.
• Using an ALTER DATABASE command.
• Starting a database or database file backup.
• Starting a database or database file restore.
• Creating a snapshot.

The following operations or conditions will prevent the CREATE DATABASE ENCRYPTION KEY, ALTER DATABASE ENCRYPTION KEY, DROP DATABASE ENCRYPTION KEY, or ALTER DATABASE...SET ENCRYPTION statements.

• The database is read-only or has any read-only file groups.
• An ALTER DATABASE command is executing.
• Any data backup is running.
• The database is in an offline or restore condition.
• A snapshot is in progress.
• Database maintenance tasks.

Step #5 – Create the Master Key

Create a master key that applies within the entire SQL Server instance. This creates the database master key (DMK), which again is used to encrypt certificates to be created for database encryption. The master key itself is protected by the Service Master Key, which is automatically generated when SQL Server is installed. Be aware that the password needs to accommodate the default Windows password policies if those are enabled on the server you are creating the certificate on.

Step #6 – Create the Certificate

Create a certificate that is also valid for the entire SQL Server. Because no specific encryption password is used, the DMK is used as encryption password.

Step #7 – Backup the Key and Certificate

If the local security certificate is used, back up the certificate. If possible, the backup location should be on a different server. If you cannot retrieve this certificate, the data is lost. It doesn't matter whether there is a valid backup of the encrypted database. In order to restore the backup, a valid certificate is needed. If the only location of the certificate is the production SQL Server instance and the server goes down or the SQL Server instance is lost, the backup is not worth a cent. Therefore it doesn't make sense to keep the backups of the certificates on the same server; they should be stored on a different computer.

The following procedure creates a file level backup of the database certificate (.CER) and corresponding private key (.PVK). The private key file is also encrypted with an additional password as shown in the screenshot.

 Strategies for Managing Keys

The illustration below shows the architecture of TDE encryption and how each key and certificate is protected. In general the generated certificate and keys are stored in the master database within the SQL Server instance. To ensure that the keys are available after a disaster should the SQL Server instance be lost, the keys should be stored in additional locations and media independent of the database server. The following scenarios can be considered:

1. Using a secondary SQL Server database to store the certificates and keys. A smaller SQL Server 2008 instance or even SQL Server 2008 Express instance can be used to accomplish this scenario. The secondary instance must be 100% independent from the encrypted database and should not share any infrastructure components.
2. Storing the backup of the database certificate (.CER) and corresponding private key (.PVK) on a third party file repository with Access Control Lists (ACL). One example would be creating a SharePoint repository, setting the appropriate ACL and then storing these files with that repository. Additionally, to simplify the management of the file pair, both files can be consolidated into a single password protected ZIP file before storage.
3. Storing the backup of the database certificate (.CER) and corresponding private key (.PVK) on removable media such as a USB flash drive and then physically securing the USB flash drive. Additionally, to simplify the management of the file pair, both files can be consolidated into a single password protected ZIP file before storage.
4. Using an extensible key management (EKM) provider to generate and store the keys. The procedures for this step will vary depending on the requirements of the EKM provider, but might include installing a certificate from the EKM provider in the local certificate store, and copying EKM DLL files onto the SQL Server computer. Using an EKM provider is not in scope for this article and thus not discussed further in this document. Using this key/certificate management mechanism would be preferable when multiple systems/databases require encryption. The LoB CoE together with our engineering colleagues are considering a more in depth and focused blog entry for the future.   

Step #8 – Link the master key, certificate and database

Link the master key, the certificate, and the database to be encrypted. This is done by creating an encryption key for the specific database to be encrypted.

Step #9 – Encrypt the database

Execute the database encryption command per the screenshot below:

Step #10 – Monitoring the progress

While the background processes continue to work, this query can be used to get an overview of the progress of work. The progress of the encryption can be monitored using the following transact SQL statement:

SELECT DB_NAME(database_id), encryption_state, key_algorithm, key_length, percent_complete FROM sys.dm_database_encryption_keys

The first column will show the database name. The second column will have a value from 1 to 5 where the values stand for:

1. Unencrypted
2. Encryption in progress
3. Encrypted
4. DEK change in progress
5. Decryption in progress

You will also notice additional CPU utilization for the SQLSERVR.EXE process as shown below via task manager:

Executing the query after encryption is completed displays the encryption_state = 3 and percent_complete = 0.

Step #11 – Start SAP

After the encryption process has completed, start the SAP system. No additional changes or profile parameters are required for SAP to function correctly.

Step #12 – Verification

After the encryption process has completed and the SAP system restarted, small verification checks can be executed to ensure that the system is stable and working correctly. Using SQL Server Management Studio, open the database parameters options and looks at the encryption setting:

DBACOCKPIT can also be executed again to verify that SAP and the database are communicating correctly:

Step #13 – Full Backup

Finally, execute a full database backup including transaction after the TDE procedure. Using SQL Server Management Studio or any 3^rd party backup software to start and execute the full backup. Note that this backup is now encrypted and can only be restored if the database certificate and private key are present on the target SQL Server instance. Any restore will fail if the keys are not installed and available to SQL Server.

One important aspect to note about SQL Server backups after enabling TDE is that compression rates fall significantly.  TDE destroys recurring patterns in the data stream that the compression algorithms use. Our engineering experience has shown that backup compression rates drop to only 1% on average when TDE is enabled and thus the resulting backup set is much larger.

Conclusion

The process of encrypting your Microsoft SQL Server 2008 database is fairly straight forward and painless. The LoB CoE highly recommends performing this process multiple times in a non-production system to ensure that the results are understood and validated before moving into your "live" SAP environments. Additionally, please ensure that the Microsoft KB article 2300689 is referenced and applied before enabling or disabling TDE.

A special thanks to Juergen Thomas and Raymond Smith for helping with this content. Part #2 - Impacts of TDE and Disabling/Removing TDE will be published mid-December 2011 and I expect a total of 4 blogs on this topic:

• Part #1 - Enabling TDE
• Part #2 - Impacts of TDE and Disabling/Removing TDE
• Part #3 – System Copy with TDE
• Part #4 – Log Shipping and Database Mirroring with TDE

Additional Information

Related SAP OSS Notes
1380493    - SQL Server Transparent Data Encryption (TDE)

Restore error after disabling TDE
KB - 2300689 - http://support.microsoft.com/kb/2300689

Database Encryption in SQL Server 2008 Enterprise Edition
http://msdn.microsoft.com/en-us/library/cc278098

Understanding Transparent Data Encryption (TDE)
http://msdn.microsoft.com/en-us/library/bb934049.aspx

How to: Enable TDE Using EKM
http://msdn.microsoft.com/en-us/library/cc645957.aspx

SAP with Microsoft SQL Server 2008 and SQL Server 2005: Best Practices for High Availability, Maximum Performance, and Scalability
http://download.microsoft.com/download/d/9/4/d948f981-926e-40fa-a026-5bfcf076d9b9/SAP_SQL2008_Best Practices_Part_I.docx

We raised some issues with the SAP migration tools during a replatforming project including Unicode conversion from SUN/Oracle to Windows SQL Server. 

Our migration went well and all the log files from the technical migration reported no error, but during the application testing the customer claimed some missing data in a couple of tables.

What happened:

These tables were splitted in at least four parts from the R3TA tool and exported and imported without any errors. After a detailed investigation, we found that some of the packages a empty:

0 rows exported and 0 rows imported.

A cross check inside the source SAP system with the select patterns out of the WHR files showed the missing entries. The select statement with the select criteria out of the HWR files executed directly  on Oracle using sqlplus resulted with 0 rows again.

After tracing the selection of SE16 with ST05, we found the issue:

The WHERE condition inside SE16 was like

• SELECT COUNT(*) from "/BIC/ADC_CNT200" WHERE "CRM_OHGUID" <= '4909283A9B121CF6E10000000A31EA29' AND "CRM_ITMGUI" = ' ';
  Result: 238656 rows

The WHERE clause with the conditions of the WHR files was:

• SELECT COUNT(*) from "/BIC/ADC_CNT200" WHERE "CRM_OHGUID" <= '4909283A9B121CF6E10000000A31EA29' AND "CRM_ITMGUI" = '';
  Result: 0 rows

The difference is the blank in the second condition! So the error is caused by the '' in the where condition created by the R3TA tool.

After changing the WHR file and re-export/re-import the table, we got the same row count on both sides.

Now the customer raised the concern about the consistency of the migrated data. Therefore we checked the row count for all tables on the source and the target system and compared these values with the information recorded in the *.toc (export) and *.log files (import) to be sure we didn't lost any data on the way from Oracle to SQL.

Thereby we found two additional strange phenomena:

1. The cluster table REGUC had app. 18.000 additional rows on the target side, but from the application point of view it was consistent.

2. The row count information from the *.toc and *.log files was not consistent with the row count in the source and target DB

To 1.: The phenomena with REGUC was raised by the data-growth during the UNICODE conversion. The table is a table cluster resp. A physical cluster table with a VARDATA field containing compressed raw data. The content of the VARDATA field is extracted and converted during the migration/Unicode conversion. The length of the field (3.800 bytes) doesn't change - however after the Unicode conversion some data doesn't fit in one record only so you have more records on the target. So a continuation record is created and the table has more records after the migration. This could be verified with the following select statement, which needs to show the same values on the source and target:

• Select count(*) from REGUC where PAGENO=0;
  (table records without continuation records)

To 2.: For some transparent tables (e.g. PA0000 in the HCM system) we had the same row count information in the *.toc (export) and *.log (import) files. Both files showed 145.189 rows, but a select count(*) on the source and target DB showed 142.615.

We reported all the issues to SAP. The issue with the R3TA is already corrected in version 7.20 and higher and will be back-ported to earlier versions (7.0x and 7.1x), too. The issue with the inconsistent row count is still under investigation from SAP. I will update the blog, if we get an response from SAP 

Lessons learned:

Don't trust the log files of the SAP migration tools, you have to verify the consistency of the databases by additional checks.

We've seen Enterprise IT take a remarkably diverse journey over the last decade. Back in 2013, still called "The Year of SaaS," we saw a great amount of consolidation in the increasingly successful SaaS and IT services markets. Through these mergers, the then-Big 4 software providers managed to maintain their competitive positions from years earlier, though they were faced with two "new" enterprise competitors. Initially known for search and later a mixed portfolio of OSs, tools, and a less-than-successful initial application strategy, Google's acquisition spree (particularly of Salesforce.com and several open source enterprise applications providers) created a new contender in enterprise SaaS-based business applications. More surprising, Apple's unforeseen entry into the Enterprise Data Center marked the beginning of a new wave of highly consumable and easily customized software, services, and underlying infrastructure. Leap-frogging one another during the year, the six companies battled on several fronts ranging from technology innovations, standards rationalization, and commoditization plays to tier 1 references, industry alignment, and market cap valuations.

In 12 months, IT's landscape was forever changed. To be sure, businesses around the globe benefitted. During this time though, access strategies to the cloud's growing collection of business-critical applications became increasingly prone to disaster recoverability and business continuity concerns. In the same way, security concerns began to crop up and consume a disproportionate share of IT budgets as data privacy issues and secure accessibility by a global community ill-prepared for the cloud threatened businesses and consumers alike. Unfortunately, at the time it all seemed business-as-usual, and band-aids seemed to be an adequate cure. Looking back, it's obvious what a mistake that kind of thinking turned out to be.

By mid-2014, in the midst of diminished margins and generally tremendous upheaval in their cash cow businesses and consumer applications marketplaces, the Big 6 found it necessary to pursue (in some cases, last-minute) aggressive partnering and vertical M&A strategies. Failed buyout attempts and successful acquisitions of one another underscored the scramble. Vertical mega-acquisitions up and down the delivery, technology, and application stacks added billion-dollar facilities at one end of the spectrum and mobile networking/telecom assets at the other. An especially bright spot emerged as Microsoft completed SaaS-enabling and integrating the last of its business and consumer applications and integration solutions. In doing so, they created the first true business/consumer experience meshing a consumer's business and personal worlds and workloads as loosely or as tightly as those consumers found useful.

The real news towards the end of 2014 revolved around the growth and automated accessibility of “virtual payment,” though, as PayphoneX catapulted into the forefront of daily news due to its significant online-energy payment system breakthroughs. PayphoneX made it easy to own and operate the new breed of hybrid and electric cars, particularly in the world's most congested cities. Payphone’s ability to accept virtual payment for car & mobile systems based exclusively on Microsoft’s Windows Phone XA enabled millions of consumers to not only gain real-time insight into road conditions, traffic, and even air travel, but to also allow automated payment processing at recharging stations around Europe, North America, and eventually the rest of the world. More remarkably, Phone XA and several competing products eventually made it possible to dynamically improve travel conditions using smart reroute analytics for Windows Phone and Windows-engineered vehicles. With a 42% annual growth rate across the world's most populous automobile-centric cities, PayphoneX and its infrastructure partners changed travel like Facebook had changed social networking 5 years earlier.

By the middle of 2015, a new cloud phenomena known today as NewCloud owned the headlines. Pure-play Public Cloud Services (PCS) had finally matured enough to enable consumers, businesses, and their partners, vendors, customers, and friends to connect to one another effortlessly and securely; short-lived Hybrid Cloud models fell away for all but the ultra-conservative or federally regulated. This ubiquitous connectivity ushered in another wave of paperless integration, speeding along the demise of more printed and faxed forms as promised two decades earlier. And this newfound interconnectedness set the stage for yet another round of automated and orchestrated business-to-business and business-to-consumer processes. Conversely, Private Cloud deployments peaked that year as their stop-gap “purpose for being” lost traction in the wake of less expensive, low risk, and highly available NewCloud offerings with their standardized and ubiquitously integrated development and hosting environments.

On the business/technology front, IT departments continued to dramatically shrink. In the space of only 2 years, many of these final bastions of techno-geek empires gave way instead to Business-enabled IT Project Management Offices responsible for the coordination rather than the implementation and management of information technology. The real work of IT was being done by the IT15, a who’s who of Mergers and Acquisitions yielding marriages between various telecommunications and network providers, software providers, systems integrators, traditional hardware vendors, and several hosting and other services providers. Nearly all desktop, network, database, and server administration and support professionals went the way of green-screen terminal service providers as both client and back-end devices became more and more disposable and as business/productivity applications were delivered as a service. Private Data Centers still hosted a few core or differentiating applications but everything else had since been tossed into the cloud (consistent with Geoffrey Moore’s core-vs-context work socialized a decade earlier), where literally a handful of subject matter experts addressed the infrastructure and platform needs of a billion end users.

And then everything changed in 2017 and the world was forced to reckon with the magnitude of all these changes. Despite great progress on many other fronts, security threats continued to mount as data was logically centralized by relatively few providers; so much power consolidated so tightly and wielded by so few was destined to be exploited. To make matters worse, another world-wide recession and the continued re-leveling of global assets set the stage for a new round of terrorist activities. This culminated in the events of October 2017 when $2 trillion worth of trading was compromised and eventually held hostage. Though officially resolved after 2 weeks, the hijacking combined with a related string of dirty bomb detonations orchestrated over 3 continents changed the face of not only global business but communications all up. The resulting exodus of liquidity from public markets devastated the global economy; repercussions continue to emanate today across social, economic, political, and business divides. And in the wake of the event’s success in destabilizing the globe, the subsequent growing incidence of copycat events has weighed heavily on global recovery. Back in those days, the prevalence of hijacking both business data and business processes pushed data security squarely into the limelight as the number one risk management challenge today. The uptick in “backsourcing” IT from the cloud back to a company’s own data center only made matters worse; the economics simply couldn’t justify the risk/reward model over any period of longevity. Companies made huge exit investments to the detriment of their already skittish shareholders and stakeholders, only to ultimately go back tail-between-their-legs to the very clouds they had left the year before.

And so we find ourselves here, today, in 2020. With this new decade of hindsight upon us, can we indeed finally see more clearly? Do we have better answers? We still haven’t solved third world hunger (though we have the resources), we are still too dependent on old world fuels (though we have effective alternatives), we still solve problems by deploying troops first and diplomacy second (though we know better), and we continue to face an increasingly dramatic cultural divide between those who have information and those who need it. Our world of enterprise and consumer technology is certainly much less expensive and much more connected and accessible than only five or ten years ago, too. But has it made a positive difference in our lives?

As we move through this Year of Hindsight, large in-house business application-focused Data Centers exist only for those firms where government regulations and compliance requirements (ITAR, FDA, HIPAA, etc) stipulate data-to-physical asset mapping and control. With much of our business, economic, and personal data continuing to grow even more tightly interconnected, Business Intelligence has again topped the business application charts as the killer app of the year—for the 10^thyear in a row. Interestingly though, this continued focus on BI has done little to change individual work efficiencies. Productivity from an individual perspective remains at the same level we saw five to ten years ago, likely a direct result of the massive increases in information, subsequent analysis, and limitations of the human brain to effectively multi-task. Of course, the breadth of automation and orchestration implemented over the last decade has circumvented many of these challenges, driving record profits for companies around the world while nearly doubling the GNP of several countries. New low-cost “people-pluggable” productivity systems from MBI (MicroBook Inc), GoogBM, AppleZon, and OraATT hold the promise to significantly overcome individual productivity and efficiency limitations. But with fewer and fewer people holding fewer and fewer knowledge worker jobs, the net effects might take years to bear fruit.

There is some good news on the consumer front though. Excess compute power is not only traded on the open market but donated tax-advantageously to the benefit of government programs, research projects, and society in general. Year 2020 has arrived to see many people using large-scale well-performing and secured pay-as-you-go public clouds in the same way that utilities like electricity and water are consumed. The “utility computing” concept envisioned at the turn of the century has finally been realized: a ubiquitous, seamless, and everyday part of life as a consumer on Earth. More than ¾ of the world’s 13 billion inhabitants across 190 countries seamlessly access the cloud and thus each other; business, ongoing education, and indeed a big part of life are being transacted nearly exclusively via the connectedness enabled by NewCloud, all fed by a host of vNext mobile and other tetherless devices. Indeed, the Green Consortium’s work towards solving the battery life problem has opened new doors to online education and 24x7 collaboration despite differing global infrastructure availability and technology standards (a sticky issue handled in large part by standardized data sharing protocols, cloud-access message bus technology, and IPv7).

Major roads, bridges, and ports, and several billion homes and businesses around the globe have been instrumented, resulting in a new wave of innovative cloud services developed and sold for fractions of pennies per transaction. Ninety percent of IT assets are controlled, and 95% of all transactions are at least passively handled, by the Global 10 (G10). People and goods alike are tracked today from birth to death, from creation to consumption, awakening a new level of insight and frankly a new incarnation of Big Brother like the world has only to this point imagined in fiction.

Massive exabyte and zettabyte-sized globally distributed data stores and the inherent physical vulnerabilities of the networks carrying all this information have finally necessitated  government protection and regulation, which in turn along with the growing crisis around data theft and information hostage has pushed the development of more sophisticated security and identification controls. Despite global redundancy and safeguards, with its ability to broadly affect entire countries and markets as well as individual companies and high-visibility industry and political leaders, hacking components of the G10 has become the dominant and favored method of terrorism. To be sure, the recently approved trillion Euro funding initiative will help quell these attacks. But this unforeseen wrinkle in the power shift has recently given rise to the notion that many countries and geographies will soon become second-tier constructs underneath the umbrella of the G10 and its carte blanche access to funding.

In all this, privacy has become a luxury few can afford. Most every business, vehicle, and commerce-related entity is instrumented and tracked for reasons described as “SoS”—for the “safety of society.” In the name of global security and in the wake of near-cataclysmic terrorist close-calls post-2017, there is even serious talk by a small circle of G10 and others global leaders of eventually instrumenting individuals as a prerequisite to buying and selling goods. In hindsight, perhaps this outcome is fairly predictable, if not an obvious outcome of the cloud phenomenon initiated a decade ago. After all, it too was predicted many, many, years ago….

   --- --- --- --- --- --- --- --- ---

Hope you enjoyed our little journey! Your thoughts and feedback are welcomed, as usual.

Best regards,

George Anderson & Juergen Imhoff

A decade later, nothing had really changed….In the early 2000s, BI was still expensive and oftentimes still ‘batch’ oriented. The highest performing BI systems – those capable of delivering complex real-time analytics – could only be afforded by very large organizations with equally large budgets. For the rest of us, Microsoft came out with Analysis Services (OLAP Services) in 2000, and a Reporting Services add-in to SQL 2000 in 2004, and those were nice steps in the right direction. SAP came out with its Google-like BI appliance a bit later and saw some decent adoption in the SAP ecosystem, but even then SAP’s customers didn’t really know what to do with all that power and ability to do real-time analytics. And it was still “SAP” expensive….not exactly BI for the masses. In the end, we had a mix of expensive real-time and inexpensive near-real-time BI solutions, and end-users (actually, their IT organizations making the purchasing decisions) pretty much had to accept the trade-off.

Five years later, and things are finally changing. Cost-effective real-time analytics are here. As the cost of world-class BI systems has fallen, and BI’s capabilities to provide real-time business visibility have grown so much, we’re at a new crossroads. I have come to the conclusion that this new cost model and capabilities combination has moved BI into the realm of mission-critical gotta-stay-up applications. Why? Because a contemporary BI system’s failure may now cause an organization's overall mission at hand to fail – real mission critical impact. Because waiting a day for a report means losing a day’s advantage to your competitors who now have not only the same kind of inexpensive real-time data, but also the ability others have to do mash-ups in a way that uncovers new trends, markets, expense reduction possibilities, and net new OPPORTUNITIES. Cheap real-time BI gives businesses the agility they need today, this minute, right now, to make smarter decisions and employ course corrections....to keep the business not only afloat but ahead of its competitors. Again, mission critical impact.

What next? It’s time that the platform providers shared the kind of prescriptive guidance and practices necessary to host these real-time business capabilities in a mission-critical, unbreakable, resilient, and risk-mitigated manner. Yes, beyond unbreakable... which arguably is the most important attribute but at the end of the day really only represents 'table stakes.' Microsoft is one of very few players that can offer mission critical platforms across a breadth of innovative and cost-effective hardware platforms, from on-premises to cloud, from HP and Dell to Fujitsu and Unisys hardware partners, from CSC and IBM to smaller hosting providers. The combination of these four attributes – unbreakable, innovative, cost-effective, and superior choice – sets the stage for a new generation of mission critical platforms, a new normal in enterprise computing, and a new sense of the phrase 'table stakes.' All of this sounds like something we need to explore in another post…

Thanks for your (gentle!) comments and feedback,

george

Most of the public available data about SAP database compression ratios only shows values for compression of an existing SQL Server database, like: 

·         Using SQL Server Database Compression with SAP NetWeaver:

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/50c5de8f-282a-2e10-cb8a-a5859a0994a4?QuickLink=index&overridelayout=true

·         Quanta case study:

http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000003962

In our project we performed a heterogeneous system copy of a whole SAP landscape based on Netweaver 7.0  from SUN/Oracle to Windows SQL Server including a Unicode conversion of the SAP systems. So we combined the change of the operating system and database software (SAP OS/DB migration) together with the Unicode conversion of the systems. The technical procedure is based on the SAP migration tool R3load, which converts the system during the export of the data to Unicode. 

The target database software is SQL Server 2008 R2 and we are using Page compression for all the database tables. Source database software is Oracle 10.2.0.4.

Until SQL Server 2008 R2, we had the disadvantage of storing the data in a UCS2 format which resulted in a growth of the database after the Unicode Conversion. SAP officially provided the following sizing information for databases in regard of Unicode Conversions:

With SQL Server 2008 R2, Microsoft introduced UCS-2 compression on top of our existing compression functionality, which removes our disadvantage in case of a Unicode conversion of a SAP system: (http://blogs.msdn.com/b/saponsqlserver/archive/2009/09/16/sql-server-2008-r2-ucs2-compression-what-is-it-impact-on-sap-systems.aspx)

Now back to the customer example. In the following table you find the database sizes and compression ratios of five productive (PRD) and development (DEV) SAP systems: 

As you see in the saving column, even with a Unicode conversion, which normally ends up with a growth of the data, we came up with an average saving of 51% for the used database size. If we include the QAS systems, which are 1:1 copies of the PRD systems, the customer came down from app. 10TB total DB space on the Oracle side to app. 5TB on the SQL Server 2008 R2 side.

We only found one strange phenomena in regards to the compression. For legal and compliance reason the customer needs to store his data online for 10 years and needs to protocol all changes to the database tables. Inside a SAP system this changes are stored in the standard table DBTABLOG. In the large ECC 6.0 system this table has > 15millions rows and uses app. 280GB of disk space on the Oracle side. After the migration the size of the table grows up to 520GB on SQL Server 2008 R2, which means near the half of the database size is occupied by one table. This difference is due to Oracle’s BLOB/CLOB compression; a functionality which will compress content in BLOB/CLOB columns. Since DBTABLOG keeps the main content in such columns, there is a reduction in size of such a table compared to SQL Server. SQL Server so far doesn’t have compression implemented for BLOB/CLOB data types. Except for the fact that SQL Server would apply its Page compression algorithms on BLOB/CLOB content which is stored in-row.

To answer one question with another, have they considered moving their non-production (back-end supporting) mission-critical SAP, Microsoft, Oracle (Siebel, JD Edwards, Peoplesoft), and other enterprise business systems to an Infrastructure-as-a-Service (IaaS) cloud? Pundits warn that the cloud is not ready for your business applications, much less your mission-critical ones. After all, traditionally hosted on-premises business applications reflect too many complex integration points (connecting ERP to your analytics, reporting, faxing, warehousing, tax calculation, and other bolt-on systems, for example) subject to potential performance, security, and availability issues. Combined with data complexities, the need to meet stringent SLAs, and all the risks associated with change, “clouding” enterprise applications might seem like the last place a CIO should look to make changes.

Ah, but here’s the thing: while these issues or challenges certainly hold true for production systems (and are starting to be well-addressed, by the way), the arguments against using the cloud for enterprise applications really start to fall apart when we consider moving behind-the-scenes non-production systems to the cloud. Non-production systems used for training technical teams and end user communities, piloting or demonstrating new systems, installing and quickly evaluating functional enhancements, or quickly smoke testing new functionality used automated test tools, can be ideal cloud candidates.

Let’s back up a moment. What is the cloud but simply a term reflecting a number of alternative deployment options or methods of delivering IT, platforms, or finished software solutions? It's just another platform, another option, and a quickly provisioned relatively cheap one at that. The new question on the lips of IT leaders everywhere should be: “why can’t we toss this into the cloud?” 

The cloud can dramatically increase a business’s time-to-value. Most IaaS providers can turn around an organization’s request for compute resources, i.e. the servers and storage resources along with network and other infrastructure services, in minutes to hours. With all the hassles surrounding new system procurement, OS and database licensing, rack-and-stack, and the requisite installation services, the best that most IT departments can promise is several days to weeks or longer. Such time-frames do little to appease business leaders or executive teams that a CIO is doing everything possible to decrease time-to-deploy, reduce the risk of delaying new projects, and show quickly delivered quantifiable business value.

And the cloud can offer financial alternatives as well. From pay as you go models, to converting CAPEX to OPEX, cloud providers give IT organizations a level of financial flexibility they probably don’t have available today. When you consider the acquisition and operational costs of servers, OS licenses, related disk space, and so on (for each of your non-production SAP, Oracle, and Microsoft business applications), the numbers grow quickly. Factor in the benefits related to rapid deployment and the ability to spin up new and wind down no-longer-needed computing resources, and it shouldn’t be difficult to not reduce costs by 1/3 to 1/10th vs traditionally hosted, internally managed systems but to deliver real business value faster than you've done before. True, you lose the tax deductions for depreciation. And there’s cases where regulatory bodies simply eliminate the cloud as a hosting or platform alternative. But in many cases an organization can gain both business flexibility and increased IT employee bandwidth at precisely the time they need it - as our global economy begins to recover and an organization’s business teams begin pushing IT to dust off all those projects shelved over the last 3 years.

A typical Fortune 2000 company running a mix of ERP, Supply Chain, Customer Relationship Management, Portal, and similar mission-critical business applications is likely running a combination of development, test, technical sandbox, business sandbox/prototyping, demonstration, upgrade, and end-user training systems for every production system. To put the cloud opportunity into perspective, there’s conservatively a 3:1 ratio of “cloudable” non-production systems to production systems (because systems in the promote-to-production process – like development and test – might be more effectively housed alongside the production system).  Once we start doing the math, the full impact of these lower costs @ rapid time-to-value @ little to no risk should become apparent. Factor in the ability to move supporting bolt-on non-production (and perhaps production?) systems to the cloud, and the numbers grow even more impressive.

Let me know if you’d like to walk through a real-world risk-informed cost savings model. I could share some work we're developing, and together we could build something very realistic. In the meantime, think about and share with me here the non-production systems in your own workplace that you could transition to the cloud, and all the benefits (besides increasing time-to-value and reducing costs) you might reap in the process…

Best to you!

George

You can probably guess that I am a big fan of compression, and many JD Edwards customers who are now using it are big fans of it as well. However there was one slight *** in SQL Server's Compression armour. In SQL 2008 compression only compressed non-Unicode data. From EnterpriseOne 8.9 and onwards Unicode has been enabled by default. If you upgraded from an earlier release you could get away without using Unicode. The likelihood however is if you upgraded to or installed EnterpriseOne 8.9 or higher the database is now using Unicode. You can tell by looking at the table definitions, if you see NCHAR or NVARCHAR columns then you are using Unicode. If you have a Unicode database that didn't mean however that there was no benefit in using compression, it is still possible to achieve 50%+ compression of a JD Edwards Unicode database when using SQL Server 2008's non-Unicode compression.

However with the release of SQL Server 2008 R2 Unicode compression was now included and the *** in compression’s armour removed. How much of a difference though does Unicode compression now make on a compressed JD Edwards database? To answer that question I ran some tests on a database that contained real customer data. The database was from an ERP 8 installation that was upgraded to ERP 9.0, some data was truncated from the database in order to speed up the upgrade process. The cut down database was still a healthy 371GB in size, uncompressed. The database was upgraded to ERP 9.0 using the standard JD Edwards upgrade process and after all the table conversions had completed the database was now using Unicode.

The database was first compressed (row compression) in SQL Server 2008 and this saw that size of the DB drop from 371GB to 161GB, that is a significant 56% reduction in size. I got a backup of this database and restored to a SQL 2008 R2 server, I then recompressed the database to take advantage of the Unicode compression now available in SQL 2008 R2. The addition of Unicode compression resulted in the database dropping to 118GB that is 68% smaller than the uncompressed database. Even though those numbers were great I decided to go in search of just a little bit more, so I uncompressed the whole database returning it to its original size of 371GB. I then compressed the whole DB again, using row compression, and the result was a database that was 111GB in size or 70% smaller than the uncompressed database.

These numbers demonstrate the kind of benefits you can achieve when using SQL Server 2008 R2 compression on your JD Edwards databases, and remember these numbers were achieved using real customer data. The test highlighted two things. The first is that Unicode compression in SQL 2008 R2 works and practically justifies an upgrade on its own. The second is that if you are using compression in SQL 2008 today, and you are planning an upgrade to SQL 2008 R2, then all you have to do is restore/attach the database to SQL 2008 R2 and recompress and you will get practically the same results as uncompressing the DB and recompressing it completely from scratch. The only question that remains to be answered is when are you planning your upgrade?

There are a couple of best practices that are important to keep your SAP instance running well on SQL Server. The first is to ensure that proportional fill is working correctly. A key best practice of proportional fill is to ensure that your freespace is the same across all data files. For this reason, we have always recommended another best practice was to manually grow your datafiles. This was because when SQL Server autogrows data files, it does so one data file at a time in a round robin fashion. When this happens, SQL Server autogrows the first file, writes to it until it is filled and then autogrows the next data file. Now your proportional fill has been corrupted.

In SQL 2008, trace flag 1117 has been created to address this problem. If trace flag 1117 is set, then when SQL Server determines that it needs to autogrow a data file, it autogrows all of the files at the same time. If you want more detail about this trace flag, please refer to SAP Service Marketplace Note 1238993.

We still recommend that you monitor your data files and grow them manually, growing all of you files at one time and by the same amount, but by setting trace flag 1117, you will put a safety net in place that will protect you increase SQL Server beats you to this task.   

That being said and specific to an SAP, I find that most SAP customers are familiar with SAP MaxAttention, SAP's tailored and comprehensive support solution.  Although not every SAP customer is a MaxAttention support customer, the option for an increased level of support and attention is available for the demands of your business regarding continuity and high availability.

From a Microsoft perspective and specifically for those of you running SAP on the Microsoft Application Platform, you should also be aware of a relatively new offering from Microsoft Services: Microsoft's Premier Mission Critical (PMC) support .  As an add-on to Microsoft Services Premier Support (which, in my strong opinion, every Microsoft platform customer running SAP or an LOB/enterprise application must have), PMC offers additional services for mission critical applications like SAP that are running on the Microsoft Application Platform.  This enhanced service offering is tailored specific to your needs and has specific services ranging from solution validation and architectural reviews to dedicated proactive and problem resolution services.  Working with SAP Support, Microsoft Services will ensure that your entire solution stack is covered.

Finally, a story to wrap up as this topic reminds me of a customer visit to Microsoft several years ago by the U.S. Army.  A colleague of mine was presenting to several generals about running SAP on the Microsoft platform and a high availability discussion started.  However, the conversation was momentarily preempted by one of the older generals who turned to my co-worker and said, "Son, let me tell you about 'mission critical'.  Mission Critical for us is about lives at stake!"  Needless to say, the conversation went on from there with a perhaps more serious tone than with other customers!  But it also highlighted the fact that the definition of "mission critical" is indeed often different from one company or organization to the next and this fact must also certainly be accounted for (hence not forgetting the business architecture piece to even this equation!).

As 2011 is upon us and New Year's resolutions are made, I encourage you to take a step back and think hard about and review your services and support strategy for your SAP and LOB applications.  Microsoft Services and our LOB CoE is here to help.

Until next time,
Bryan...