On-premises BizTalk integration with Azure File Storage

아티클
1/17/2024

Other Resources

Michael Stephenson - Web cast using file share to Azure File Storage

Problem

Azure File Storage (AFS) supports on-premises integration on servers supporting SMB 3.0. For BizTalk integration this allows for AFS to be mapped just like a normal file locations using the File Receive Locations and File Send Ports. In situations where SMB 3.0 is not supported, this project is provided as an alternative strategy by using the Azure Storage client library to integrate with AFS.

Context

The requirement to move files into AFS came about when legacy on-premises solutions needed to exchange messages with virtual machines running in Azure. File storage was chosen over Blob storage because the solutions already supported the exchange of file based messages and because of AFS's support for SMB 3.0. As expected, this made integration with the solutions running in Azure VMs simple without requiring additional development effort. Not expected was the client's environment did not allow for AFS to be mapped within the environment (Error 53). An on-prem BizTalk server was available so this was the natural choice for a reliable messaging service.

The requirement was to reliably push messages from on-premises folders to an AFS share. As AFS supports REST endpoints, this was explored first using the BizTalk WCF adapter with the AzureStorageBehaviorExtension endpoint behavior. The version of the AzureStorageBehavior at the time of this article does not support the File storage type. Weighing the cost of extending the behavior to support File storage versus the simplicity of using the Azure Storage client library, it was decided that the simplicity of the client library outweighed the loss of tracking by not using the WCF REST endpoints.

Solution

The solution was to use the Azure Storage client library to save the files into AFS. This greatly simplified coding as most of the functionality could be accomplished in a single expression. But, as anything in BizTalk, the issue was getting the information required into the expression. In particular this was information like the connection string and directory that would change with each environment (development, test, production). As there was not an existing pattern for this, for instance using the BRE or a config database, it was decided to create a custom receive pipeline component where this information could be supplied.

For the benefit of the community the basic project has been provided on MSDN: On-premises BizTalk integration with Azure File Storage.

Sample Project

The solution, MSDNFileMover, consists of two projects: MSDNFileMover and MSDNFileMoverPromoter. The MSDNFileMoverPromoter contains a property promoter used to allow some values to be specified on receive pipeline while the MSDNFileMover contains the pipeline, schema and orchestration.

MSDNFileMoverPromoter

A single class, PropertyPromoter, is used to allow for the AFS storage connection string and a directory to be specified. There are many posts on the web that illustrate how a BizTalk pipeline component can be created so this will not be repeated here. The purpose is to allow a convenient mechanism to add context to a receive pipeline as shown below:

The following snippet shows the Execute method that promotes the filename, directory and storage connection string so they can be referenced in the orchestration.

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
    string systemPropertiesNamespace = @"http://schemas.microsoft.com/BizTalk/2003/system-properties";
    string filePropertiesNamespace = @"http://schemas.microsoft.com/BizTalk/2003/file-properties";
 
    var id = pInMsg.Context.Read("InterchangeID", systemPropertiesNamespace);
    pInMsg.Context.Promote("InterchangeID", systemPropertiesNamespace, id);
 
    var path = pInMsg.Context.Read("ReceivedFileName", filePropertiesNamespace);
    var filename = System.IO.Path.GetFileName(path.ToString());
 
    pInMsg.Context.Promote("StorageConnectionString", "https://MSDNFileMover.MSDNFileMoverProperties/v1", StorageConnectionString);
    pInMsg.Context.Promote("Directory", "https://MSDNFileMover.MSDNFileMoverProperties/v1", ImportDirectory);
    pInMsg.Context.Promote("Filename", "https://MSDNFileMover.MSDNFileMoverProperties/v1", filename);
 
    return pInMsg;
}

MSDNFileMover

The orchestration is the most interesting part of the MSDNFileMover project as the receive pipeline and properties schema are straight forward. The orchestration receives an xml document and then uses the Azure Storage client library to save the xml document to Azure File Storage. In the case of error, the orchestration will suspend to allow for the orchestration to be resumed and the send attempted again.

The orchestration is shown below:

A couple of things to note about the orchestration. The orchestration is long running but contains an atomic scope. This allows for the retry when a failure happens even though the variables used are not serializable. The orchestration required the Azure Storage nuget package to be installed:

The expression, SendToAFS, uses the message properties to create a new file in AFS.

storageConnection = Message_Inbound(MSDNFileMover.StorageConnectionString);
 
directory = Message_Inbound(MSDNFileMover.Directory);
filename = Message_Inbound(MSDNFileMover.Filename);
 
storageAccount = Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse(storageConnection);
 
client = storageAccount.CreateCloudFileClient();
share = client.GetShareReference("MyShare");
 
rootDirRef = share.GetRootDirectoryReference();
dirRef = rootDirRef.GetDirectoryReference(directory);
 
fileRef = dirRef.GetFileReference(filename);
 
fileRef.UploadText(Message_Inbound.OuterXml, null, null, null, null);

Summary

This project provides an illustration of how Azure File Storage could be integration with an on-premises BizTalk server. The Azure Storage client libraries are used in a single transaction. If multiple operations are performed in this manner, it is recommended to do each one as a separate transaction to prevent the environment being in an unknown state in case of a failure.

It is also recommended that a solution around SMB 3.0 should be explored first. SMB 3.0 might not be supported due to infrastructure restrictions or because of the infamous Error 53 (see links). Extending the AzureStorageBehaviorExtnension and using WCF to communicate with the Azure Storage REST endpoints wsa also explored. This proved cumbersome and challenging for two reasons. The first is the level of control over the BizTalk WCF engine and secondly because of the two step process the REST endpoints require. One call to create the file and the second to fill the content of the file. A simpler and more reliable approach was chosen instead at the sacrifice of message level tracking.

More Information

The following are some links that may be helpful:

다음을 통해 공유