Adding Data FLow activity via .Net SDK not working

Question

I am creating a pipeline in .Net SDK which is using ExecuteDataFLow Activity, here is the script

//Create a Mapping Data Flow resource

var props = new MappingDataFlow() { Description = "Data Flow to purge Cosmos Data",

Sources = new List<DataFlowSource>
    {
        new DataFlowSource
        {
            Name = "cosmosSource"+ (new Random(5).Next()),
            Description="Delete from Cosmos", 

            Dataset = new DatasetReference()
            {
                ReferenceName = srcCosmosDataset.Name,
                Parameters = new Dictionary<string, object>
                {
                    { "databaseName", new Expression("@pipeline().parameters.dbConfig.name") },
                    { "collectionName", new Expression("@item.name") }//source doesn't need partition key
                }
            }
        },
    },

Sinks = new List<DataFlowSink>
    {
        new DataFlowSink
        {
            Name = "cosmosSink"+ (new Random(5).Next()),
            Description="Sink to Cosmos",
             //SchemaLinkedService = new LinkedServiceReference { ReferenceName = "AzureDataLakeStoreLinkedService" },

            Dataset = new DatasetReference
            {
                ReferenceName = sinkCosmosDatasetResource.Name,
                Parameters = new Dictionary<string, object>
                {
                    { "databaseName", new Expression("@pipeline().parameters.dbConfig.name") },
                    { "collectionName", new Expression("@item.name") }
                }
            },

        }
    },

Transformations = new List<Transformation>
    {
         new Transformation
        {
            Name = "alterRow1",
            Description = "Delete if if true",
        }
    },

ScriptLines = new List<string> {
                    //SCRIP LINES IS HUGE, NOT ADDED HERE
}; 
var mappingDataFlow = new DataFlowResource(props, Guid.NewGuid().ToString(), "dfPurgeCosmos" );
var execDataFlowActivity = new ExecuteDataFlowActivity
 { Name = "ExecuteDataFlowActivity", //LinkedServiceName = new LinkedServiceReference { ReferenceName = "AzureDataLakeStoreLinkedService" }, Policy = new ActivityPolicy() { Timeout = "0.12:00:00", Retry = 3, RetryIntervalInSeconds = 30, SecureInput = false, SecureOutput = false }, DataFlow = new DataFlowReference() { ReferenceName = mappingDataFlow.Name, DatasetParameters = new Dictionary<string, object> { { "partitionKeyParam", new Expression("item().partitionKey") } } //AdditionalProperties = new Dictionary<string, object> //{ // { "ClientPasId", new Expression("
};

var forEachDBServiceActivity = new ForEachActivity {
 Name = "ForeachDBServices", IsSequential = false, Items = new Expression("
};

Expected behaviour A clear and concise description of what you expected to happen. As you can see in the script, I have a foreach activity inside which I want to execute Data FLow Activity for each cosmos container. My linked services are proper, but my ExecuteDataFlowActivity is failing due to invalid DataFlowReference. Not sure what is wrong there. Any suggestion would be highly appreciated.

Thanks Asif

Answer 1

@Bhat, Asif

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: I am creating a pipeline in .Net SDK which is using ExecuteDataFLow Activity, here is the script

//Create a Mapping Data Flow resource

var props = new MappingDataFlow() { Description = "Data Flow to purge Cosmos Data",

PowerShellAI ConvertCopy

Sources = new List<DataFlowSource>
    {
        new DataFlowSource
        {
            Name = "cosmosSource"+ (new Random(5).Next()),
            Description="Delete from Cosmos", 

            Dataset = new DatasetReference()
            {
                ReferenceName = srcCosmosDataset.Name,
                Parameters = new Dictionary<string, object>
                {
                    { "databaseName", new Expression("@pipeline().parameters.dbConfig.name") },
                    { "collectionName", new Expression("@item.name") }//source doesn't need partition key
                }
            }
        },
    },

Sinks = new List<DataFlowSink>
    {
        new DataFlowSink
        {
            Name = "cosmosSink"+ (new Random(5).Next()),
            Description="Sink to Cosmos",
             //SchemaLinkedService = new LinkedServiceReference { ReferenceName = "AzureDataLakeStoreLinkedService" },

            Dataset = new DatasetReference
            {
                ReferenceName = sinkCosmosDatasetResource.Name,
                Parameters = new Dictionary<string, object>
                {
                    { "databaseName", new Expression("@pipeline().parameters.dbConfig.name") },
                    { "collectionName", new Expression("@item.name") }
                }
            },

        }
    },

Transformations = new List<Transformation>
    {
         new Transformation
        {
            Name = "alterRow1",
            Description = "Delete if if true",
        }
    },

ScriptLines = new List<string> {
                    //SCRIP LINES IS HUGE, NOT ADDED HERE
}; 
var mappingDataFlow = new DataFlowResource(props, Guid.NewGuid().ToString(), "dfPurgeCosmos" );
var execDataFlowActivity = new ExecuteDataFlowActivity
 { Name = "ExecuteDataFlowActivity", //LinkedServiceName = new LinkedServiceReference { ReferenceName = "AzureDataLakeStoreLinkedService" }, Policy = new ActivityPolicy() { Timeout = "0.12:00:00", Retry = 3, RetryIntervalInSeconds = 30, SecureInput = false, SecureOutput = false }, DataFlow = new DataFlowReference() { ReferenceName = mappingDataFlow.Name, DatasetParameters = new Dictionary<string, object> { { "partitionKeyParam", new Expression("item().partitionKey") } } //AdditionalProperties = new Dictionary<string, object> //{ // { "ClientPasId", new Expression("
};

var forEachDBServiceActivity = new ForEachActivity {
 Name = "ForeachDBServices", IsSequential = false, Items = new Expression("
};

Expected behaviour A clear and concise description of what you expected to happen. As you can see in the script, I have a foreach activity inside which I want to execute Data FLow Activity for each cosmos container. My linked services are proper, but my ExecuteDataFlowActivity is failing due to invalid DataFlowReference. Not sure what is wrong there. Any suggestion would be highly appreciated.

Solution : Looks like I had to first create the Data FLow separately and then use it.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Looks like I had to first create the Data FLow separately and then use it.