Before

<BusinessPartner>

    <Key>1234</Key>

    <Name>Test Company</Name>  

</BusinessPartner>

After

<BusinessPartner>

    <Key>1234</Key>

    <CompanyDetails>  

        <Name>Test Company</Name>

    </CompanyDetails> 

</BusinessPartner>

One way to nest an XML element in T-SQL is to use a subquery. The query for the flat XML output looks something like this:

SELECT [key], name FROM BusinessPartnerTable FOR XML PATH('BusinessPartner')

To create the nested CompanyDetails element, you can use a subquery:

SELECT [key],

    (SELECT name as 'name'

     FROM BusinessPartnerTable c

     WHERE c.[key] = bp.[key]

     ORDER BY [key]

     FOR XML PATH(''), type

    ) as 'CompanyDetails'

FROM BusinessPartnerTable bp

GROUP BY [key]

FOR XML path('BusinessPartner'), root('root')

Notice the use of a subquery to nest the CompanyDetails element.

The attached Nested XML Example.sql file shows an example of how to nest XML in a SQL SELECT statement.

Alternatively, you can fine tune a SELECT column to control the output. For example, 'name' as 'CompanyDetails/Name' will nest the 'name' column in a CompanyDetails element.

SELECT
   [key],
   'name' as 'CompanyDetails/Name'
FROM @BusinessPartner
FOR XML PATH('BusinessPartner')

Nested XML Example.sql

For more information about the Microsoft “Data Explorer” Preview for Excel, see https://www.microsoft.com/en-us/bi/Products/Office.aspx.

How to create the task report with an advanced query

Microsoft “Data Explorer” Preview for Excel enables you to create advanced queries with the Data Explorer formula language. The Data Explorer Formula Language provides a wide variety of formulas that are used to build expressions. To create an advanced query, you start with a blank query and add query steps that contain Data Explorer formulas. To create a query to shape a task table from the Access 2013 Project management web app, you can use the Project Task Query listed below. You need to replace the Server name and Database name in the Sql.Database formula parameters to connect to the SQL Database instance.

Project Task Query

let

Source = Sql.Database("{Server name}.database.windows.net", "{Database name}"),

Access.Projects = Source{[Name="Access.Projects"]}[Data],

HiddenOtherColumns = Table.SelectColumns(Access.Projects,{"Project Name",
"Access.Tasks"}),

#"Expand Access.Tasks" = Table.ExpandTableColumn(HiddenOtherColumns, "Access.Tasks",
{"Task Title", "Description", "Start Date", "Due Date", "Percent Complete"},
{"Access.Tasks.Task
Title", "Access.Tasks.Description", "Access.Tasks.Start Date", "Access.Tasks.Due Date",
"Access.Tasks.Percent Complete"}),

RenamedColumns = Table.RenameColumns(#"Expand Access.Tasks",{{"Access.Tasks.Task
Title", "Task Title"}, {"Access.Tasks.Description", "Description"}, {"Access.Tasks.Start Date",
"Tasks Start Date"}, {"Access.Tasks.Due Date", "Tasks Due Date"}, {"Access.Tasks.Percent
Complete", "Task Percent Complete"}})

in

RenamedColumns

Parameters to replace

In the Project Task Query, you must replace the Sql.Database {Server name} and {Database name} parameters. You get the SQL Database instance connection information from the Access 2013 SQL Server Connection Information dialog box.

To get the SQL Database instance connection information for your Access app:

1. Open your Access 2013 app.

2. Click the FILE ribbon tab.

3. Click Manage > Enable Read-Only Connection.

4. Click Manage > View Read-Only Connection Information. The SQL Server Connection Information dialog box displays the SQL Database instance Server, Database, UserName, and Password.

5. In the Access app Project Task Query, change {Server name}.database.windows.net to the Server name, and change {Database name} to the Database name.

To create the advanced query

With the SQL Server connection information, you can now create the Data Explorer query to import a task report into Excel.

To enable advanced query editing

1. In the DATA EXPLORER ribbon tab, in the Options group, click Settings.

2. In the Settings dialog box, click Enable Advanced Query Editing.

Once you enable advanced query editing, you can create an empty query by using the Write Query option in the From Other Sources ribbon menu.

To create an advanced query

1. In the DATA EXPLORER ribbon tab, click From Other Sources > Write Query.

2. In the Edit Query dialog box, add query steps that contain Data Explorer formulas by using the formula bar or click the query script icon ( ) to display the Edit Query formulas dialog box.

3. In the Edit Query dialog box, replace the default query with the Project Task Query.

4. Click Edit credentials. In the SQL Server Connection Information dialog box, enter the SQL Database instance UserName and Password.

5. In the Access a Microsoft SQL Database dialog box, enter the SQL Database instance UserName and Password.

6. Click Save. Data Explorer builds a query from the custom Project Task Query.

7. Click Done.

After you click Done, Data Explorer executes the query and inserts a task report. To update the task worksheet, you simple click Refresh in the QUERY ribbon tab.

As of today, here is a partial list of how customers plan to use Data Explorer.

1. Build a simple table showing your sales broken down by product in an Excel spreadsheet, you could build this quite easily with Data Explorer. Thanks Chris Webb for this observation.
2. Create a query from your Facebook likes that render an Excel chart. Check out pic.twitter.com/QjuuYZAbcx.
3. Import data from multiple log files. Check out this blog post.
4. Traverse the Facebook Graph. Check out this blog post.
5. Pull data into PowerPivot from new data sources, such as XML, Facebook, and File Folders as refreshable connections. Check out this blog post.
6. Import data from a wide variety of traditional and non-traditional data sources including CSV, Windows Azure SQL Database, Oracle, MySQL, SharePoint list, OData Feed, Facebook.
7. Merge and shape data sources to match your data analysis requirements or prepare it for further analysis and modeling by tools such as PowerPivot and PowerView.
8. Connect to Big Data in Excel 2013.
9. Self-service ETL tool for the Excel power user. Enhances the self-service Business Intelligence experience in Excel by simplifying data discovery and access. Check out this blog post.
10. Create dynamic lookups. Check out this blog post.

One of the goals of efficient data management design is to remove duplicate data that is contained in a data source. To achieve that goal, a data management application is divided into many subject tables so that each fact is represented only once. For example, an invoice design might have an Order table and related Order_Details table, in which one order has many order details. To bring the divided information back together, the columns in the tables must be coordinated so that they show information about the same order. This coordination is accomplished by using table relationships. A table relationship works by matching data in key columns — often a column with the same name in both tables. In most cases, these matching columns are the primary key from one table, which provides a unique identifier for each record, and a foreign key in the other table. An order can be associated with order line item details by creating a table relationship between an OrderID column in the Order table and Order_Details table. In this example, the Order table is the primary table and the Order_Details table is the related table.

In Microsoft “Data Explorer” Preview for Excel, you create a query to bring data source tables back together. In the Order example, you can use the Expand query operation to bring together Order_Details records that are related to the Order table, to combine order line items with each order. The Expand operation widens a subject table to include columns from a related table and expands a subject table with values from the related table. To illustrate:

A subject table has columns A and B.

A related table has column C.

The Expand operation widens a subject table to include column C and expands a subject table with related values from the data source that contains column C.

In the Order example, the Expand operation widens an Order table to include the Order_Details.ProductID, Order_Details.UnitPrice, and Order_Details.Quantity columns to bring together Order records and related Order_Details records. For more information about the Expand operation, see Expand a column containing an associated table and the Combine data from multiple data sources tutorial.

For more information about the Microsoft “Data Explorer” Preview for Excel, see https://www.microsoft.com/en-us/bi/Products/Office.aspx.

The content has several lessons about exposing an Azure SQL Database on Windows Azure as a WCF Data Service. You can download a sample Console app to learn how to use an ACS token to connect to a Windows Azure cloud service, and use the token to LINQ query a Windows Azure WCF Data Service. In addition, the sample application shows how to use EncryptString to save sensitive data (user name and password) on the client.

You can download the word document and sample app at Migrate a WCF Data Service to Windows Azure Lessons on our Windows Azure SQL CodePlex site.

Lesson 1 - Create a local WCF Data Service that connects to Azure SQL Database

How to deploy the sample AdventureWorksLT2012_Azure database to Azure SQL Database

In lesson 1, you learn how to deploy a SQL Server database to an Azure SQL Database account, and create a WCF Data Service on localhost that exposes a Product OData resource including a product image resource stream.

Restricting the SQL Server User

When creating an Azure WCF Data Service, you should create web user accounts on your Azure SQL Database. For more information about adding users to your Azure SQL Database, see Adding Users to Your SQL Azure Database.

How to create a Visual Studio 2012 Windows Azure Cloud Service on localhost

You use Visual Studio 2012 with the Windows Azure SDK installed to create a Windows Azure Cloud Service on localhost.

How to create a Named Resource class to stream an image

This section shows you how to you create a named resource class to expose ThumbnailPhoto as an Edm.Stream entity type.

How to encrypt a connection string for testing on localhost

This section shows you how to protect sensitive connection string data by encrypting the web.config connection string for testing on localhost.

Lesson 2 - Publish a local WCF Data Service to a Windows Azure Cloud Service

In this lesson, you publish the local WCF Data Service created in lesson 1 to a Windows Azure Cloud Service. Sign up for aWindows Azure 90 day free trial, and follow the Windows Azure Account setup instructions.

How to publish a Windows Azure Cloud Service

A step by step guide to publish a local WCF Data Service to a Windows Azure Cloud Service using Visual Studio 2012.

How to encrypt a connection string for Windows Azure

In this section, you encrypt a connection string by opening a Remote Desktop session into your Windows Azure Compute VM.

Lesson 3 - How to implement claims-based authentication using Windows Azure ACS 2.0

In lesson3, you learn how to implement Windows Azure claims-based authentication using the Azure Access Control Service 2.0 (ACS). ACS is a cloud-based service that provides an easy way of authenticating and authorizing users to gain access to your web applications and services while allowing the features of authentication and authorization to be factored out of your code.

How to configure a user name and password ACS Service Identity

In ACS, a service identity is a credential that is registered with an ACS namespace and is intended for use by autonomous applications or clients. In this section, you learn how to configure an ACS Service Identity. You also learn how to configure a Relying Party Application using the Windows Azure portal.

How to obtain service credentials for an HttpModule

In this section, you learn how to create a security module that implements IHttpModule to validate a Windows Azure ACS token to authenticate a WCF Data Service user.

How to configure web.config for a Windows Azure WCF Data Service

In this section, you configure the AdventureWorksLTWindowsAzureWebRole web.config file.

How to create an Azure custom error page

In this section, you create a custom error page that is displayed when access is denied due to invalid credentials.

Lesson 4 - How to configure a Windows Azure WCF Data Service client App for Claims-based authentication

The AzureODataConsoleApp sample console application displays a LoginForm where you enter your Windows Azure ACS user name and password. The sample then connects to your claims-based Windows Azure WCF Data Service using an SWT token retuned using your credentials. A token allows a user to enter their username and password in order to obtain a time limited key token which allows them to fetch a specific resource - without using their username and password. The AzureWebClient.GetTokenFromACS() method in the tutorial source code returns a Windows Azure ACS SWT token representing a service resource scope for a username and password.

The AzureODataConsoleApp demonstrates how to:

• Get an ACS SWT token using AzureWebClient.GetTokenFromACS().
• Get an OData stream using AzureWebClient.OpenRead() passing the SWT token.
• Create a LINQ query to project the OData stream to a List<List<EntityProperty>>.
• Save a bitmap from a web response when querying a named resource stream.
• Use a Login Form to get user name and password
• Use EncryptString to save sensitive data on the client.

In this tutorial you will learn the fundamentals of how to upload an Excel worksheet to Azure SQL Database using SQL Server Integration Services (SSIS). This tutorial assumes prior experience with SSIS and Windows Azure SQL Database. Upon completing this tutorial, you will have a sample database on Windows Azure published from an Excel 2007 or above worksheet. This tutorial will not illustrate data retry scenarios since the example worksheet contains a small data set. However, when the amount of data becomes significant, you will need a retry strategy to compensate for network bandwidth and latency, and the amount of (shared) resources available in a physical hardware node hosting a database.

SSIS is a platform for building enterprise-level data integration and data transformations solutions. You use SSIS packages containing tasks to solve complex business data migration problems. Publishing an Excel worksheet to an Azure SQL Database is similar to an on-premise solution except an Azure SQL table must have a clustered index. For this tutorial, you will configure a Data Flow Task containing an Excel Source task and ADO NET Destination task to publish a worksheet to an Azure SQL Database.

Table of Contents

• Prerequisites
• Overview
• Step 1: Download the sample OlympicsData workbook
• Step 2: Setup the sample Olympics Azure SQL Database
• Step 3: Create an SSIS Package that Uploads Data from Excel Worksheet to Azure SQL Database
• How to configure an Excel Source task on a 64-bit machine
• Retry strategies
• Alternative data movement strategies
• References

Prerequisites

• Azure SQL Database Account. Sign up for a Windows Azure 90 day free trial which includes 1 SQL database, and follow the Windows Azure Account setup instructions.

• Excel 2007 or above.
• Example OlympicsData workbook located at https://msftdbprodsamples.codeplex.com/downloads/get/528843.
• SQL Server 2012 Developer, Standard, or Enterprise with SQL Server Data Tools installed.

Note To test this tutorial without an Azure SQL Database account, you can use an on-premise SQL Server 2012 database. Any SQL Server 2012 instance will work including SQL Server 2012 Express LocalDB.

Overview

After following this tutorial, you will publish a sample Excel worksheet to an Azure SQL Database using SSIS.

Step 1: Download the sample OlympicsData workbook

For this tutorial, you will start with an Excel 2007 or above workbook containing four worksheets: Disciplines, Events, Medalists, and Sports. Download the worksheet from our CodePlex site: https://msftdbprodsamples.codeplex.com/downloads/get/528843.

Olympics data © The Guardian, and reproduced with The Guardian permission 

Step 2: Setup the sample Olympics Azure SQL Database

There are two primary methods to create an Azure SQL Database: Management Portal for SQL Database or SQL Server Management Studio on your desktop. For details on Getting Started with Windows Azure SQL Database, see Getting Started with Windows Azure SQL Database.

From SQL Server Management Studio (SSMS):

1. Connect to your Azure SQL Database account, and the master database.
2. Create the Olympics database:

-- NOTE: For Azure SQL Database, you must be connected to the master database to 

-- create a new database.

USE master

GO

IF EXISTS (SELECT [name] FROM [sys].[databases] WHERE [name] = N'Olympics')

    DROP DATABASE [Olympics]

GO

CREATE DATABASE [Olympics]

GO

Step 3: Create an SSIS Package that Uploads Data from Excel Worksheet to Azure SQL Database

Unlike SQL Server, every table in Azure SQL Database must have a clustered index. A clustered index is usually created on the primary key column of the table.

Note The ADO NET Destination task returns the following error if your Azure SQL Database table does not have a clustered index:

[ADO NET Destination [2]] Error: An exception has occurred during data insertion, the message returned from the provider is: Tables without a clustered index are not supported in this version of SQL Server. Please create a clustered index and try again.

To create the Excel worksheet to Azure SQL Database SSIS package

To publish an Excel worksheet to Azure SQL Database, your package will contain a Data Flow Task, Excel Source task, and ADO NET Destination.

1) Create an SSIS project.

2) Drop a Data Flow Task onto the Control Flow design surface, and double click the Data Flow Task.

3) Drop an Excel Source onto the Data Flow design surface.

Note When using the Excel Source task on a 64-bit machine, set Run64BitRuntime to False. For more information, see How to configure an Excel Source task on a 64-bit machine in this tutorial.

4) Double click the Excel Source task.

5) Click New next to the Excel connection manager drop down.

6) Browse to OlympicsData.xlsx.

7) Click OK.

8) Select “Disciplines$” from the Name of the Excel sheet drop down, and click Preview.

9) Click OK.

10) Drop an ADO NET Destination task under the Excel Source task.

11) Connect a Data Flow Path from the Excel Source to the ADO NET Destination.

12) Double click the ADO NET Destination, and click Yes on the “This component has no available input columns” dialog form.

13) Click New next to Connection manager.

14) Click New on the Configure ADO.NET Connection Manager dialog form.

15) Enter your Azure SQL Database configuration settings (or SQL Server connection settings for on-premise):

a. Enter your Azure server name as {servername}.database.windows.net.

b. Select SQL Server Authentication, and enter your SQL database user name and password.

c. Select the Olympics database from the Connect to a database drop down.

d. Click All.

e. Set Connect Timeout to 500 to increase the length of time (in seconds) to wait for a connection to Azure.

f. Set Encrpt to True to encrypt the SQL Server connection string.

g. Since the sample worksheet data is small, you can use the default packet size. For larger data sets, you should increase network packet size.

16) Click Test Connection.

17) Click OK.

18) Click OK (ADO NET Connection Manager).

19) Click New next to Use a table or view.

20) Replace the default Create Table statement with an Azure SQL Database Create Table statement:

Note The default [DisciplineID] and [SportID] column types are changed to int to optimize the database size, and [DisciplineID] is set as the clustered index.

CREATE TABLE [Discipline] (

  [DisciplineID] int NOT NULL IDENTITY, 

  [Discipline] nvarchar(255),

  [SportID]int,

  CONSTRAINT [PK_Discipline_ID] PRIMARY KEY CLUSTERED 

  (

    [DisciplineID] ASC

  )

)

21) Click OK, and OK on the “Select a table or a view from the list” dialog form.

22) From the Use a table or view drop down, select "dbo"."Discipline", and click Preview.

23) Click Mappings and accept the default Available Input Columns from the Excel worksheet and Available Destination Columns from the ADO.NET Destination Columns.

24) Click OK.

Final Data Flow sequence of tasks

To publish the Excel worksheet to Azure SQL Database SSIS package

1) Right click the design surface, click Execute Task to publish the sample worksheet to Azure SQL Database.

Note You can repeat this process to publish multiple Excel worksheets to Azure SQL Database.

Successful Data Flow execution

To create each Olympics table that corresponds to each OlympicsData worksheet, run the following SQL script:

USE [Olympics]

GO

-- Disciplines Worksheet. This table was created in step 3

IF EXISTS (SELECT [name] FROM [sys].[tables] WHERE [name] = N'Discipline')

DROP TABLE [Discipline]

GO

CREATE TABLE [Discipline] (

[DisciplineID] int NOT NULL IDENTITY,

[Discipline] nvarchar(255),

[SportID]int,

CONSTRAINT [PK_Discipline_ID] PRIMARY KEY CLUSTERED

(

[DisciplineID] ASC

)

)

GO

-- Events Worksheet

IF EXISTS (SELECT [name] FROM [sys].[tables] WHERE [name] = N'Event')

DROP TABLE [Event]

GO

CREATE TABLE [Event] (

[EventID] int NOT NULL IDENTITY,

[DisciplineID] int,

[Event] nvarchar(255),

CONSTRAINT [PK_Event_ID] PRIMARY KEY CLUSTERED

(

[EventID] ASC

)

)

GO

-- Medalists Worksheet

IF EXISTS (SELECT [name] FROM [sys].[tables] WHERE [name] = N'Medalist')

DROP TABLE Medalist

GO

CREATE TABLE Medalist (

[MedalistID] int NOT NULL IDENTITY,

[Event] nvarchar(255),

[Edition] datetime,

[Athlete] nvarchar(255),

[NOC] nvarchar(255),

[Gender] nvarchar(255),

[Medal] nvarchar(255),

[Season] nvarchar(255),

[MedalValue] float,

[DisciplineID]int,

[SportID] int,

CONSTRAINT [PK_Medalist_ID] PRIMARY KEY CLUSTERED

(

[MedalistID] ASC

)

)

GO

-- Sports Worksheet

IF EXISTS (SELECT [name] FROM [sys].[tables] WHERE [name] = N'Sport')

DROP TABLE Sport

GO

CREATE TABLE [Sport] (

[SportID] int NOT NULL IDENTITY,

[Sport] nvarchar(255),

CONSTRAINT [PK_Sport_ID] PRIMARY KEY CLUSTERED

(

[SportID] ASC

)

)

GO

How to configure an Excel Source task on a 64-bit machine

The Excel Source task does not have a 64-bit OLE DB driver. To configure an Excel Source task on a 64-bit machine, set Run64BitRuntime to False on the Debugging node on the SSIS solution properties dialog form.

Note When Run64BitRuntime is True, SSIS will generate the follow error:

SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC0209303. There may be error messages posted before this with more information on why the AcquireConnection method call failed.

Retry strategies

This tutorial does not provide details about retry strategies when moving data to Azure SQL Database. When the amount of data becomes significant, this operation may present some critical challenges that will require a slightly different approach compared to on-premises solutions, mainly due to public network bandwidth and latency, and the amount of (shared) resources available in physical hardware nodes hosting a database in the cloud environment to execute the data loading phase.

• For cases of a single package, incorporate retry by
• Determining the maximum number of times the package should retry before failing.
• Setting up a variable to store your success status.
• Using a FOR loop to retry up to the maximum number of retries if the data flow is not successful.
• Move the worksheet into an on-premise SQL Server database. Then, use SQL Agent and specify what action SQL Server should take if a failure occurs during job execution.
• Develop custom source or destination components for SSIS and incorporate transient fault handling.
• Execute packages programmatically, and place the retry of failed packages into the programmatic execution of the package. You accomplish this by checking the execution status after package execution, and retrying if the package execution was unsuccessful.

Alternative data movement strategies

This tutorial illustrated how to use SSIS to publish an Excel worksheet to Azure SQL Database. However, there are other data movement solutions you can consider:

• Use the SqlBulkCopy class in ADO.NET.
• Run a Bulk Copy Program (BCP.EXE) import / export script. See https://msftdbprodsamples.codeplex.com/releases/view/37304 for an AdventureWorks2012 on Azure SQL Database sample.

Note When running the AdventureWorks2012 on Azure SQL Database sample, change the batch size to 10000 from the default 1000 and the packet size to 16000 from the default 4000.

• Azure Storage Blobs and Queues can be used to implement complex solutions that require intermediate storage between on premise and cloud processes, and the ability to orchestrate phases and operations between the two environments.

References

• Connect to an Excel Workbook
• Excel Source task and ADO NET Destination task
• Getting Started with Windows Azure SQL Database
• Loop through Excel Files and Tables by Using a Foreach Loop Container
• CREATE INDEX (Windows Azure SQL Database)

The AdventureWorks QueryFeed OData sample illustrates an end to end OData workflow with Office scenario. Starting with SQL Server views, the sample shows how to expose selective AdventureWorks views as an OData service. The AdventureWorks product catalog contains two product images: LargePhoto and ThumbNailPhoto. By implementing WCF Data Services 5.0 Named Resource Streams, the sample shows how to stream LargePhoto and ThumbNailPhoto into a Word document. You can select named resources in the QueryFeed workflow activity and select the default named resource that the client host will render in the TablePartPublisher workflow activity.

By providing feedback on this blog, or sending me an email, you can help define any part of the sample. My email address is derrickv@microsoft.com.

The source code is available at CodePlex.

• AdventureWorks OData Feed (https://msftdbprodsamples.codeplex.com/releases/view/94064).
• OData QueryFeed workflow activity (https://msftdbprodsamples.codeplex.com/releases/view/94486).

The sample shows how to integrate the following technologies to address the business scenario of consuming any OData service in an Office application. The sample uses the AdventureWorks database as an example line-of-business database with selective views exposed as OData resources.

• SQL Server 2008R2 or above including views with joins, ROW_NUMBER() OVER, functions, xpath queries, and configuring a NT AUTHORITY\NETWORK SERVICE login and role for IIS.
• Windows Workflow 4.0 including custom activities, activity designers, OData schema aware expression lists, practical use of ModelItem and ModelItemTree, child activities, XML LINQ projections into entity properties, a custom workflow designer, and variables.
• Office (Excel and Word) including hosting an OData related workflow, consuming workflow activity states using tracking participants, extension methods, embedding OData properties into Content Controls, and Open XML.

The attached document explains more about the latest iteration. Also, I added a section on creating a new QueryFeed OData workflow application using the sample workflow designer.

The sample uses several LINQ queries to combine elements to form IEnumerable<EntityProperty>. I would like to use fewer LINQ queries, but ran out of time. The QueryFeed.Execute() method shows how various metadata elements are combined to form an IEnumerable<EntityProperty>.

Named Resource Streams and product photos

A data service can expose binary data. Starting with version 3 of OData, an entity can have multiple related resource streams, which are accessed by name. The sample illustrates how to implement IDataServiceStreamProvider2 to expose AdventureWorks ThumbNailPhoto and LargePhoto product images.

Configuring a data service to support the streaming of binary data requires five steps.

1. Attribute the target entity that has resource streams.
2. Implement IDataServiceStreamProvider2 – required only for named resource streams.
3. Implement IDataServiceStreamProvider – required to support both kinds of binary resource stream.
4. Define a data service that implements the IServiceProvider interface. The data service uses the GetService implementation to access the streaming data provider implementation. This method returns the appropriate streaming provider implementation.
5. Enable large message streams in the Web application configuration and access to binary resources on the server or in a data source.

Step 1 - Attribute the target entity that has resource streams.

The AdevntureWorks sample attributes vProductCatalog. See AdventureWorksModel.Extensions.cs within the code sample.

using System.Data.Services.Common;
 
 namespace Microsoft.Samples.SqlServer.AdventureWorksService
 {
 [NamedStream("LargePhoto")]
 [NamedStream("ThumbNailPhoto")]
 public partial class vProductCatalog { }
 }

Step 2 - Implement IDataServiceStreamProvider2 – required only for named resource streams.

public class ProductCatalogResourceProvider : IDataServiceStreamProvider2
 {
 …
 
 public Stream GetReadStream(object entity, ResourceProperty resourceProperty, string etag, bool? checkETagForEquality, DataServiceOperationContext operationContext)
 {
 vProductCatalog image = entity as vProductCatalog;
 
 if (image == null)
 {
 throw new DataServiceException(500, "Internal Server Error.");
 }
 
 // Return a stream that contains the requested ThumbnailPhoto or 
 LargePhoto
 
 return ProductPhoto(image.ProductID, resourceProperty.Name);
 }

//The sample uses .NET Framework 4.5 SqlDataReader.GetStream();
 private Stream ProductPhoto(int productID, string columnName)
 {
 Stream productPhoto = null;
 
 using (SqlConnection connection = new SqlConnection
 (Properties.Settings.Default.Setting.ToString()))
 {
 using (SqlCommand command = connection.CreateCommand())
 {
 // Setup the command
 command.CommandText =
 string.Format("SELECT {0} FROM Production.vProductCatalogImages WHERE ProductID=@ProductID", columnName);
 
 command.CommandType = CommandType.Text;
 
 // Declare the parameter
 SqlParameter paramID = new SqlParameter("@ProductID", SqlDbType.Int);
 
 paramID.Value = productID;
 command.Parameters.Add(paramID);
 
 connection.Open();
 
 try
 {
 using (SqlDataReader reader = command.ExecuteReader(CommandBehavior.CloseConnection))
 {
 reader.Read();
 if (reader.HasRows)
 productPhoto = reader.GetStream(0);
 }
 }
 catch (SqlException ex)
 {
 //Log the SqlException, such as Invalid column name, in a production application
 }
 
 return productPhoto;
 }
 }
 }
 

Step 3 - Implement IDataServiceStreamProvider – required to support both kinds of binary resource stream. The sample only implements IDataServiceStreamProvider2.GetReadStream().

Step 4 - Define a data service that implements the IServiceProvider interface.

public object GetService(Type serviceType)
 {
 if(serviceType == typeof(IDataServiceStreamProvider2))
 {
 //Return the stream provider to the data service.
 return new ProductCatalogResourceProvider();
 }
 
 return null;
 }
 

Step 4 - Enable large message streams in the Web application configuration.

When you create a data service in an ASP.NET Web application, Windows Communication Foundation (WCF) is used to provide the HTTP protocol implementation. By default, WCF limits the size of HTTP messages to only 65K bytes. To stream large binary data to and from the data service, you configure the Web application to enable large binary files and to use streams for transfer. To do this, add <services> element and <bindings> element to the application's Web.config file:

<system.serviceModel>

    <services>

      <!--The name of the service-->

      <service name="AdventureWorks_ODataService.AdventureWorks">

        <!--you can leave the address blank or specify your end point URI-->

        <endpoint binding="webHttpBinding" bindingConfiguration="higherMessageSize" 

contract="System.Data.Services.IRequestHandler" />

      </service>

    </services>

    <bindings>

      <webHttpBinding>

        <!-- configure the maxReceivedMessageSize value to suit the max size of 

   the request (in bytes) you want the service to recieve-->

        <binding name="higherMessageSize" maxReceivedMessageSize="500000" />

      </webHttpBinding>

    </bindings>

  </system.serviceModel>

See AdventureWorks.OData.Service.sln for a complete sample service. For more information, see Streaming Provider (WCF Data Services).

QueryFeed activity

The sample QueryFeed activity now allows you to select Named Resources.

In addition, you can select a default resource to render in the TablePartPublisher activity.

Finally, as mentioned in my previous post, the attached document details how to edit a QueryFeed workflow using a hosted Windows Workflow designer. After creating a product catalog workflow application, you can render a Word table. The images in the sample table were rendered using Named Resource Streams.

AdventureWorks OData-Named Resource Streams.docx

My original entity properties TablePartPublisherDesigner had style names hard-coded in the activity designer. This was okay to demonstrate the concept of an activity publisher. However, I wanted a way for the client hosting the workflow designer to supply style names for any TablePartPublisherDesigner.  After some research, I decided to use the workflow designer ModelService to get an enumeration of ModelItems from the ModelItemTree. A LINQ statement returns a collection of ModelItems of type TablePartPublisher. From this collection, the ViewService.GetView(modelItem) is used to get each TablePartPublisherDesigner. Once I had each TablePartPublisherDesigner, I was able to add style names to each activity designer from within the workflow designer client.

With this approach, a client application can supply expression data to an activity designer.

The source code at https://msftdbprodsamples.codeplex.com/releases/view/94486 shows how this is done.

The result is an activity with expression items set by the client host:

Excel Host	Word Host

Here is some of the code:

modelService = workflowDesigner.Context.Services.GetService<ModelService>();

//Set StyleNames

IEnumerable<ModelItem> activityCollection = modelService.Find(modelService.Root, typeof(Activity)) ;

var tablePartPublisherDesigners = from mi in activityCollection
                                  where mi.ItemType == typeof(Microsoft.Samples.SqlServer.Activities.ActivityPublishers.TablePartPublisher)
                                  select mi;

foreach (ModelItem mi in tablePartPublisherDesigners)
{
    this.SetStyleNames(workflowDesigner.Context.Services.GetService<ViewService>().GetView(mi) as TablePartPublisherDesigner);
}

private void SetStyleNames(TablePartPublisherDesigner activityPublisher)
    {
        switch (this.ClientName)
        {
            case "Word":
                {
                    activityPublisher.StyleNames.Add("Light Shading - Accent 5");
                    activityPublisher.StyleNames.Add("Light Shading - Accent 2");
                    activityPublisher.StyleNames.Add("Medium List 2 - Accent 5");
                    activityPublisher.StyleNames.Add("Medium Grid 1 - Accent 4");
                    activityPublisher.StyleNames.Add("Colorful List");
                    break;
                }
            default:
                {
                    activityPublisher.StyleNames.Add("TableStyleMedium9");
                    activityPublisher.StyleNames.Add("TableStyleMedium10");
                    activityPublisher.StyleNames.Add("TableStyleMedium20");
                    activityPublisher.StyleNames.Add("TableStyleMedium24");
                    activityPublisher.StyleNames.Add("TableStyleMedium28");
                    break;
                }
        }
    }

• Using LINQ to project OData entity XML to entity classes.
• Creating custom workflow activities.
• Creating custom workflow designers containing OData schema aware expression editors and WorkflowItemsPresenters. A WorkflowItemsPresenter is used to drop OData Filter activities that format a fully qualified OData filter parameter.
• WPF ComboBox items hosting a button and checkbox.
• Hosting the Windows Workflow designer within Microsoft Excel 2010.
• Running a workflow from within a Microsoft Excel 2010 and Microsoft Word 2010 Addin.
• Using a workflow Tracking Participant to subscribe to activity states.
• Creating ModelItem and Office extension methods.
• Open XML 2.0 that renders OData entity properties as a Word 2010 table.

By providing feedback on this blog, or sending me an email, you can help define any part of the sample. My email address is derrickv@microsoft.com.

Source Code

https://msftdbprodsamples.codeplex.com/releases/view/94486.

User Story

The sample addresses the following user story:

As a developer, I want to create an OData feed activity so that an IT Analyst can consume any AdventureWorks OData feed within a business workflow.

The OData QueryFeed sample activity shows how to create a workflow activity that consumes an OData resource, and renders entity properties in a Microsoft Excel 2010 worksheet or Microsoft Word 2010 document. Using the sample QueryFeed activity, you can consume any OData resource. The sample activity uses LINQ to project OData metadata into activity designer expression items. By setting activity expressions, a fully qualified OData query string is constructed consisting of Resource, Filter, OrderBy, and Select parameters. Executing the activity returns OData entity properties.

Resource

Query string part

/CompanySales

EntitySets LINQ for Resource Items

 public IEnumerable<EntitySet> EntitySets
 {
 get
 {
 //Arrange: A url to a WCF 5.0 service is given as InArgument_Url
 XNamespace xmlns = "https://schemas.microsoft.com/ado/2008/09/edm";
 
 //Act: A LINQ to XML query is constructed that projects EntitySet XML to an EntitySet class
 IEnumerable<EntitySet> entitySets = from x in metadata
 .Descendants(xmlns.GetName("EntitySet"))
 select new EntitySet
 {
 Name = x.Attributes("Name").Single().Value,
 Namespace = x.Attributes("EntityType").Single().Value.Split(new char[] { '.' })[0],
 EntityType = x.Attributes("EntityType").Single().Value.Split(new char[] { '.' })[1]
 };
 
 //Return:
 return entitySets;
 }
 }

Filter

Query string part

$filter=OrderYear ne 2006 and Sales lt 10000

EntitySetSchema LINQ for Filter Items

 public IEnumerable<EntityPropertySchema> EntitySetSchema(string resource)
 {
 IEnumerable<EntityPropertySchema> entitySchema = null;
 if (resource != string.Empty)
 {
 XNamespace xmlns = "https://schemas.microsoft.com/ado/2008/09/edm";
 entitySchema = from p in metadata
 .Descendants(xmlns.GetName("EntityType")).Descendants(xmlns.GetName("Property"))
 where p.Parent.Attribute("Name").Value == (from e in this.EntitySets where e.Name == resource select e.EntityType).Single<string>()
 select new EntityPropertySchema
 {
 Parent = p.Parent,
 Name = p.Attribute("Name").Value,
 Type = p.Attribute("Type").Value,
 MaxLength = p.Attribute("MaxLength") == null ? "NaN" : p.Attribute("MaxLength").Value
 };
 }
 
 return entitySchema;
 }

OrderBy

Query string part

$orderby=ProductSubCategory asc,Sales desc

EntitySetSchema LINQ for Filter Items

See Filter

Select

Query string part

$select=ProductSubCategory,Sales

EntitySetSchema LINQ for Filter Items

See Filter

Rendering QueryFeed TableParts

The example QueryFeed workflow renders entity properties in any client that supports .NET Framework 4.0. The sample source code at https://msftdbprodsamples.codeplex.com/releases/edit/94486 shows rendering an EntityProperties TablePart in Microsoft Excel 2010 and Microsoft Word 2010. The sample source code shows how to use an extension method to extend a Microsoft Excel or Microsoft Word range.

Excel InsertEntityTable Extension Method

public static void InsertEntityTable(this Range activeCell, IEnumerable<IEnumerable<EntityProperty>> entityProperties, string styleName)
 {
 int currentColumn = 0;
 int currentRow = 1;
 Range range;
 
 List<string> propertyNames = (from item in entityProperties select item).First<IEnumerable<EntityProperty>>().Select(n => n.Name).ToList<string>();
 int columnCount = propertyNames.Count();
 int rowCount = entityProperties.Count();
 
 Globals.ThisAddIn.Application.ScreenUpdating = false;
 
 //Data Columns
 foreach (string name in propertyNames)
 {
 range = activeCell.get_Offset(1, currentColumn);
 range.FormulaR1C1 = name;
 currentColumn++;
 }
 currentColumn = 0;
 
 //Data Values
 foreach (IEnumerable<EntityProperty> items in entityProperties)
 {
 //row = new TableRow();
 currentRow++;
 foreach (EntityProperty item in items)
 {
 range = activeCell.get_Offset(currentRow, currentColumn);
 range.FormulaR1C1 = item.Value;
 currentColumn++;
 }
 currentColumn = 0;
 }
 
 Worksheet activeSheet = Globals.ThisAddIn.Application.ActiveSheet;
 Range styleRange = activeCell.Range[activeSheet.Cells[2, 1], activeSheet.Cells[rowCount + 2, columnCount]];
 string listObjectName = String.Format("Table{0}", activeSheet.ListObjects.Count);
 
 try
 {
 activeSheet.ListObjects.AddEx(XlListObjectSourceType.xlSrcRange, styleRange, Type.Missing, XlYesNoGuess.xlYes).Name = listObjectName;
 activeSheet.ListObjects[listObjectName].TableStyle = styleName;
 }
 catch
 {
 //Handle exception in a production application
 }
 
 styleRange.Columns.AutoFit();
 
 Globals.ThisAddIn.Application.ScreenUpdating = true;
 }

My next post will show how to edit a QueryFeed workflow using a hosted Windows Workflow designer. A custom designer is included in the source code at https://msftdbprodsamples.codeplex.com/releases/view/94486.

You can read more about the QueryFeed workflow in the attached Introducing the OData QueryFeed Activity document.

Introduction to QueryFeed Activity.docx

As a continuation of my previous blog post, we created a live AdventureWorks OData feed at https://services.odata.org/AdventureWorksV3/AdventureWorks.svc. The AdventureWorks OData service exposes resources based on specific SQL views. The SQL views are a limited subset of the AdventureWorks database that results in several consuming scenarios:

• CompanySales
• Documents
• ManufacturingInstructions
• ProductCatalog
• TerritorySalesDrilldown
• WorkOrderRouting

We will be iterating on this sample, so expect it to improve over time. For instance, the Documents feed currently exposes the document entity type property as an Edm.Binary type. An upcoming iteration will show how to implement Named Resource Streams for the Documents.Document entity type property.

User Story

This sample iteration, on our CodePlex site, addresses the following user story:

As a backend developer, I want to only allow views of the AdventureWorks2012 database to be exposed as an OData public resource so that the underlying schema can be modified without affecting the service.

Source Download

• AdventureWorks OData Service
• AdventureWorks2012 database

How to install the sample

You can consume the AdventureWorks OData feed from https://services.odata.org/AdventureWorksV3/AdventureWorks.svc. You can also consume the AdventureWorks OData feed by running the sample service from a local ASP.NET Development Server.

To run the service on a local ASP.NET Development Server

1. Download the sample from CodePlex.
2. Attach the AdventureWorks2012 database. The AdventureWorks2012 database can be downloaded from https://msftdbprodsamples.codeplex.com/releases/view/93587
3. From Microsoft SQL Server Management Studio, run \AdventureWorks.OData.Service\SQL Scripts\Views.sql to create the OData feed SQL views.
4. Open \AdventureWorks.OData.Service\AdventureWorks.OData.Service.sln in Visual Studio 2012.
5. In Solution Explorer, select AdventureWorks.svc.
6. Press F5 to run the service on https://localhost:1234/AdventureWorks.svc/.

AdventureWorks OData Service Resources

Iteration 2 will address the following user story:

As a frontend developer, I want to consume the AdventureWorks OData feed so that I can use entity data within a business workflow.

The sample will show how to create a QueryFeed workflow activity that can consume any OData feed url and return an enumeration of entity properties. The sample further shows how to create an EntityProperties activity which allows a user designing a workflow to drop a TablePartPublisher into the activity. A TablePartPublisher can be consumed by any client to render entity properties. For example, a Word Add-in can consume EntityProperties and a TablePartPublisher to render an Open XML table. Iteration 2 will show how this is done.

I have most of the code for iteration 2 complete, but not quite ready to publish. I plan to publish iteration 2 in about two weeks. For now, I’ll show you a sample of each AdventureWorks OData resource. Each example table was created by running the custom activities from within Word.

My next blog post will dive into how to create OData workflow activates. Custom activity designers that consume OData metadata and use LINQ to project entity xml into entity classes will be discussed. The metadata entity classes are used to provide expression editor items.

CompanySales Example

QueryFeed Url

https://localhost:1234/AdventureWorks.svc/CompanySales?$top=2&$orderby=OrderYear asc

Result

ManufacturingInstructions Example

QueryFeed Url

https://localhost:1234/AdventureWorks.svc/ManufacturingInstructions?$top=2&$select=ProductName,Instructions,SetupHours

Result

ProductCatalog Example

QueryFeed Url

https://localhost:1234/AdventureWorks.svc/ProductCatalog?$filter=CultureID eq 'en' and ProductSubcategory eq 'Pedals'&$top=2&$select=ProductName,ProductSubcategory,Description,ListPrice

Results

TerritorySalesDrilldown Example

QueryFeed Url

https://localhost:1234/AdventureWorks.svc/TerritorySalesDrilldown?$filter=SalesPersonID eq 275&$top=2

Result

WorkOrderRouting Example

QueryFeed Url

https://localhost:1234/AdventureWorks.svc/WorkOrderRouting?$top=2&$select=WorkOrderID,ProductNumber,ProductName,ScheduledStartDate,ActualStartDate,ScheduledEndDate,ActualEndDate

Result