https://www.microsoft.com/en-us/sqlazurelabs/labs/trust-services.aspx

The update does not need registration code anymore (ignore "Get Invited" link - it is leftover from previous release and will be removed). Just download SDK, visit the portal to create server, and check the documentation and samples.

Note that if you are using previous release - the current and previous services are now both alive, as completely independent services. Each is only compatible with appropriate SDK. You cannot get to the previous service using new SDK or vise versa. Please update to the new SDK/service - it offers many new features (and the old service will be disabled some time in future).

But today the project I've been working recently has shipped our first Lab:

Microsoft Codename "Trust Services"

Trust and Security have been hot topics for the public cloud since its inception.  Corporate IT departments and CIOs have repeatedly expressed concerns over the loss of control associated with moving various levels of sensitive data to a public cloud.  At the same time, the overall benefits of a public cloud are tremendous and continue to gain momentum.  This means that many organizations have a pressing need to migrate to public cloud infrastructure in spite of ongoing concerns about security.

Encryption is one of the fundamental required tools for protecting data in the cloud. However, when you start encrypting data, you are facing with question how do you manage encryption keys and encryption policies? There are several options (you will probably recognize the products using them):

• Have keys owned by the cloud as well, but stored separately from data (Server Side Encryption) - this provides a bit of improvement, but not much of a real security
• Use a single key that unlocks the data (Simple Encrypting Client) - secure, but not manageable, you have to give the same key to all the parties, have no way to revoke keys when one of the parties leaves, etc
• Have an on-premise security middleware - secure, but not very useful, since now all the users (even mobile users) have to go through this middleware - you lose most of the benefits of the cloud, plus have to purchase and maintain on-premises hardware and software
• Store keys, authorizations and encryption policies in the cloud, but signed and encrypted with individual keys given to each party - so that each party can locally verify authenticity of the policies and only authorized parties have access to and are able to decrypt the sensitive data

You can guess the Trust Services follows the latest approach, and provides a unique combination of end-to-end application level encryption and power of the cloud to roam encryption keys, while leaving the owner in full control of his data. It enables data driven applications to work with sensitive data, securely stored in different cloud-based storages while continuing to maintain control over access to this data.

Today we shipped the service (sign up link), the client SDK and four samples of client applications utilizing the service+SDK.

Check out the video at the Trust Services Lab Link, then read more at the Trust Services TechNet article.

I added zip code data from CSV file by loading to PowerPivot, and discarded the CSV since I did not plan to update it. Bad idea. PowerPivot wants to update all data when you do Refresh All, and missing CSV file prevents it from refreshing all other tables too.

What I should have done: load data to my Excel file, then import it to PowerPivot from Excel. Or load data to temporary Excel and copy/paste it to PowerPivot. Later solution is better, since it avoids data duplication.

The lesson - learn your data before importing it to PowerPivot, afterward it might be too late.

The workaround is of course easy - use SQL query rather than table import, but tedious since PowerPivot makes you re-run the wizard for every single table. Here is the query just in case:

select *, convert(VARCHAR(40), guid, 1) AS GuidStr from MyTable 

Then I've imported data, added relationships, and started to create calculated columns. One of the DAX expressions got wrong results, and I realized I missed one relationship which would make the expression return correct result. I've tried to add relationship, but PowerPivot did not let me do it, failing with some very obscure error message (An unexpected error occurred, file 'xmvsquery.cpp', line 1739, ...). Only once I've deleted that calculated column, I could create relationship, then add the column back, and finally get the correct results.

Second lesson for me - tell PowerPivot about all the relationships before doing anything with the data.

Overall, I find PowerPivot useful, but such problems make it way less useable as exploratory tool - you've to know all your columns and all your relationships in advance, and tell PowerPivot about everything. If you miss something you might need to go back and start from scratch, e.g. delete a table with all the calculated columns, and re-import it again to get binary column, or delete the calculated columns to be able to define relationships.

If PowerPivot silently fails to load, and in Event Viewer you've got an event like below, run Office Setup / Add or Remove Features, then check .NET Programmability Support under Microsoft Excel.

Here is the event you might find in event viewer:

Customization URI: file:///C:/Program Files (x86)/Microsoft Analysis Services/AS Excel Client/10/Microsoft.AnalysisServices.Modeler.FieldList.vsto

Exception: Customization could not be loaded because the application domain could not be created.　

************** Exception Text **************

Microsoft.VisualStudio.Tools.Applications.Runtime.CannotCreateCustomizationDomainException: Customization could not be loaded because the application domain could not be created. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Office.Interop.Excel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c' or one of its dependencies. The system cannot find the file specified.

File name: 'Microsoft.Office.Interop.Excel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c'

at Microsoft.AnalysisServices.Modeler.FieldList.ThisAddIn..ctor()

https://blogs.msdn.com/b/somasegar/archive/2010/10/28/making-asynchronous-programming-easy.aspx

I guess soon the idea hack of using C# iterators as poor-man's language support (that caused me to start this blog :)) will finally be deprecated.

But performance was not good – it took about 10 minutes to push ~32mln rows.

I started with simplest debugging technique – remove destination to check if it is the cause, then remove conditional split, etc. We quickly found the problem is caused by the custom component. But it is very simple:

public override void ProcessInput(int inputID, PipelineBuffer buffer)
{
    while( buffer.NextRow() )
    {
        byte[] guid = buffer.GetBytes(m_inputColumn);
        int partition = Partition(guid);
        buffer.SetInt32(m_outputColumn, partition);

        buffer.DirectRow(ComponentMetaData.OutputCollection[0].ID);
    }
}

Can you guess why is it so slow?

There are many potential inefficiencies here, e.g. due to the buffer API we’ve to allocate and copy byte array for every row. But the code that took most of the overall execution time was

    buffer.DirectRow(ComponentMetaData.OutputCollection[0].ID);

Why so slow? You see, the PipelineBuffer object is optimized for performance, but ComponentMetaData object and various collections of inputs/outputs/columns/etc are mostly for design time and were not optimized for performance at all. And there is a good reason for this: these collections do not change during runtime, so one should not call them at runtime. So we moved their usage to PreExecute and then used the cached output ID:

public override void PreExecute ()
{
    m_outputID = ComponentMetaData.OutputCollection[0].ID;
}

and somewhere in ProcessInput(..) -

    buffer.DirectRow(m_outputID);

Actually, this particular component did not need to call DirectRow at all, and using default exclusion group was enough, so we finally removed it. But if you do need to call it – cache the output ID in advance. Our performance has improved about 20 times by this simple change.