Raveena Dayani — Wed, 29 Jun 2016 15:03:51 GMT

Make sure the projects are of .NET framework 4.5.2 To write a test client which completely mimics the ADF, write a console application with the code as shared.

Following nugets need to be installed first:

Azure.Management.DataFactories – provides ADF management capabilities
Moq – a mocking framework for .NET

Make linked services for azure table storage, batch and azure documentDB:

Azure Storage Linked Service connection string (Input):
- Account Name - schoolrecordsstorage
- Account Key – can be found on the portal
- Default Endpoint Protocol – https

Azure Batch Linked Service (Batch):
- Account Name – schoolrecordsbatch
- Batch URI - https://northeurope.batch.azure.com
- Access Key - can be found on the portal
- Pool Name – streamspool
- Linked Service Name – Azure Storage Linked Service’s name

Azure documentDB linked service connection string (Output):
- Account Endpoint - https://documents.azure.com:443/
- Account Key – can be found on the portal

Create list of all the linked services being used by ADF

Make datasets for azure table storage and documentDB

Azure Storage Dataset definition (Input):
Structure - List() where new DataElement are EnglishScore, MathScore, ChemistryScore, PhysicsScore, ComputerScore, AccountsScore, BusinessStdScore, EconomicsScore
Linked Service Name – associated storage linked service
Type Properties – SubjectsScoreStorage is the table name
External – true (when data in storage is not coming from some other pipeline but coming from external source)
Availability – 1 Hourly
Policy – Specify the retry interval and timeout and maximum no. of retries

Azure Blob Dataset definition (Input):
- Linked Service Name – associated storage linked service
- Type Properties - FolderPath = "binariescontainer"
- External – true
- Availability – 1 Hourly

Azure DocumentDB Dataset definition (Output):
- Linked Service Name – associated documentDB linked service
- Type Properties – TestAssetsR StudentCollection is the collection name
- Availability – 1 Hourly

Create list of all the datasets being used by ADF

Create an activity within the pipeline:

Assign Storage and Azure Batch as inputs to it
Assign DocumentDB as output to it
Type Properties – Give the following details in here:
- Assembly Name: dll StudentGradingAdfActivity.dll
- Entry Point:TransformActivity (i.e., namespace.className of entry class)
- Package Linked Service: Azure Storage Linked Service Name
- Package File: binariescontainer/ StudentGradingAdfActivity.zip (i.e., a zip containing only .dll and .pdb files which has the custom activity dll as well)
- Extended Properties: Containing all the configurable properties, namely, sliceStart, sliceEnd, dataStorageAccountName, docDbEndpointUrl, sinkDbName, dataStorageContainer, azureStorageDatasetName, docDbDatasetName, docDbLinkedServiceName, azureStorageLinkedServiceName, blobDllStorageDatasetName, queueName, storedProcedureName, queueWrapperWriteSize, docsWriteBatchSize, (dataStorageAccountKey, docDbAuthKey, appInsightsInstrumentationKey)
- Policy: Define the policy to follow for pipeline scheduling, namely, order of execution, retry count and timeout.
- Scheduler: 1 Hourly

Create a mock for the IActivityLogger. This will be needed in the next step

Call Execute(IEnumerable, IEnumerable, Activity, IActivityLogger). Pass mock logger to it as logging is not needed while running the adf test client

Transformation of Data from Azure Storage to DocumentDB Using Azure Data Factory Custom Activity