Loading Files to Azure

Question

Hello,

Im loading files from SD card to Azure. Here we are manually triggering the pipeline because each file is taking certain time to load. For eg., one file might take less than 5 minutes and the other might take more than an hour. We do not want to schedule a trigger for every one hour as most of the files are taking less than 5 minutes. We want to automate this whole process which picks each file sequentially after the completion of loading. Is there a way where we can build a pipeline so that the pipeline will pick each file in order without manual intervention? If yes, please guide me clearly with every step.

Answer 1

If you put all the files on an Azure Storage Account you can use the For each activity enables you to iterate on an array to perform actions like running a stored procedure on each item/file, but before that, you need to get the array of all the file names in the blob container, Get Metadata activity provides this capability. The below pipeline definition is an example:

{  
    "name": "pipelineTest",  
    "properties": {  
        "activities": [  
            {  
                "name": "Get Metadata1",  
                "type": "GetMetadata",  
                "typeProperties": {  
                    "dataset": {  
                        "referenceName": "YourAzureBlobDataset",  
                        "type": "DatasetReference"  
                    },  
                    "fieldList": [  
                        "childItems"  
                    ]  
                }  
            },  
            {  
                "name": "ForEach1",  
                "type": "ForEach",  
                "dependsOn": [  
                    {  
                        "activity": "Get Metadata1",  
                        "dependencyConditions": [  
                            "Succeeded"  
                        ]  
                    }  
                ],  
                "typeProperties": {  
                    "items": {  
                        "value": "@activity('Get Metadata1').output.childItems",  
                        "type": "Expression"  
                    },  
                    "activities": [  
                        {  
                            "name": "Stored Procedure1",  
                            "type": "SqlServerStoredProcedure",  
                            "typeProperties": {  
                                "storedProcedureName": "[dbo].[StoredProcedureTestWithParameters]",  
                                "storedProcedureParameters": {  
                                    "fileName": {  
                                        "value": {  
                                            "value": "@item().name",  
                                            "type": "Expression"  
                                        }  
                                    }  
                                }  
                            },  
                            "linkedServiceName": {  
                                "referenceName": "AzureSql1",  
                                "type": "LinkedServiceReference"  
                            }  
                        }  
                    ]  
                }  
            }  
        ]  
    }  
}