Data Factory triggered from bob storage creating multiple runs

Louise Kirkham 0

I have an ADF that currently triggers from a blob storage event, blob created, ignoring empty blobs.
Screenshot 2024-06-20 at 01.52.41

Previously this was correctly triggering a single ADF run when a new file was uploaded.

I have recently introduced a Logic App which copies a new file from a Sharepoint to blob storage. This works successfully, writing a single file to blob storage, however this is now triggering ten runs of the ADF pipeline.

Any advice on why this is occurring?
And any suggested solution for preventing multiple repeat runs triggered by the same file?

Currently defining my trigger with:

trigger={
    "properties": {
        "pipelines": [
            {
                "pipelineReference": {
                    "referenceName": df_pipeline_name,
                    "type": "PipelineReference"
                },
                "parameters": {
                    "filename": "@triggerBody().fileName"
                }
            }
        ],
        "type": "BlobEventsTrigger",
        "typeProperties": {
            "blobPathBeginsWith": "/data/blobs/raw/spa_initial_drafts/included",
            "blobPathEndsWith": ".docx",
            "ignoreEmptyBlobs": True,
            "scope": f"/subscriptions/{df_config['subscription_id']}/resourceGroups/{df_config['resource_group_name'].lower()}/providers/Microsoft.Storage/storageAccounts/{storage_account_name.lower()}",
            "events": [
                "Microsoft.Storage.BlobCreated"
            ]
        },
    }
}

phemanth 7,825 Reputation points Microsoft Vendor

2024-06-20T05:38:06.8466667+00:00

@Louise Kirkham

Thanks for the question and using MS Q&A platform.

The ‘BlobCreated’ event is triggered when a blob is created or replaced. Specifically, this event is triggered when clients use the PutBlob, PutBlockList, or CopyBlob operations that are available in the Blob REST API and when the Block Blob is completely committed.

The ‘ignore empty blobs’ option, if set to true, means that blobs with zero bytes will be ignored. However, this might not solve your issue if the Logic App is writing the file in chunks, as each chunk might not be empty.

One possible solution could be to modify your Logic App to write the entire file to blob storage in one operation, rather than in chunks. This would ensure that the ‘BlobCreated’ event is triggered only once, when the entire file is uploaded.

Another approach could be to add a delay or a check mechanism in your ADF pipeline to ensure that the file is completely written before the pipeline is triggered. This could be done by checking the file properties or by implementing a delay.

Hope this helps. Do let us know if you any further queries.

Louise Kirkham 0

Thanks for the response @phemanth , that's helpful to know.

The Logic App was created via the designer, but looking at he scheme code the 'Copy Blob' action is defined as:

"Create_blob_(V2)": {
                                        "inputs": {
                                            "body": "@body('Get_file_content_using_path')",
                                            "headers": {
                                                "ReadFileMetadataFromServer": true
                                            },
                                            "host": {
                                                "connection": {
                                                    "referenceName": "azureblob-1"
                                                }
                                            },
                                            "method": "post",
                                            "path": "/v2/datasets/@{encodeURIComponent(encodeURIComponent('etaidev'))}/files",
                                            "queries": {
                                                "folderPath": "/data/raw/spa_initial_drafts/included/",
                                                "name": "@item()?['{FilenameWithExtension}']",
                                                "queryParametersSingleEncoded": true
                                            }
                                        },
                                        "runAfter": {
                                            "Get_file_content_using_path": [
                                                "SUCCEEDED"
                                            ]
                                        },
                                        "type": "ApiConnection"
                                    },

Are you able to advise how to change this to write in one operation rather than chunks?

phemanth 7,825 Reputation points Microsoft Vendor

2024-06-24T05:48:37.45+00:00
@Louise Kirkham Absolutely! Based on the provided Logic App schema code snippet, here's how you can modify the "Copy Blob" action to write the entire file in one operation:

Current Behavior:

The current Logic App action seems to be using the "Copy Blob" operation with a POST request.

This might be uploading the file in chunks, triggering the ADF pipeline for each chunk.

Solution:

We need to switch to the "Put Blob" operation, which uploads the entire file in a single request.

Here's how to modify the code:

Change Action Type:

In the Logic App designer, locate the "Copy Blob" action.

Change the action type from "ApiConnection" to "Azure Blob Storage - Put blob content."

Update Inputs:

The "Put Blob content" action requires different inputs compared to "Copy Blob."

Update the inputs as follows:

Blob content: This should be set to the dynamic content from the "Get_file_content_using_path" action that retrieves the entire file content from SharePoint.

Blob container name: Set this to the container name where you want to store the file (likely "etaidev").

Blob name: Set this to the desired filename with path, similar to the "name" property in the original action. You can use dynamic content from the "Get_file_content_using_path" action to construct the filename and path.Absolutely! Based on the provided Logic App schema code snippet, here's how you can modify the "Copy Blob" action to write the entire file in one operation:
phemanth 7,825 Reputation points Microsoft Vendor

2024-06-25T09:19:17.4833333+00:00

@Louise Kirkham We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
phemanth 7,825 Reputation points Microsoft Vendor

2024-07-01T04:44:02+00:00

@Louise Kirkham just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

2 answers

Q&A Assist

2024-06-20T01:00:20.35+00:00

AI generated content. This question contains an answer created with AI from Q&A Assist. Learn more
The reason why the Logic App is triggering ten runs of the ADF pipeline is because the Logic App is writing the file to blob storage in chunks, which triggers the ADF pipeline each time a chunk is written. To prevent multiple repeat runs triggered by the same file, you can use the "BlobCreated" event instead of the "BlobCreated (ignore empty blobs)" event in the ADF trigger. This will trigger the ADF pipeline only once when the entire file is uploaded to blob storage.

References:

Data pipelines storage event triggers in Data Factory (Preview)

Use Azure Data Factory to migrate data from Amazon S3 to Azure Storage
Please sign in to rate this answer.
Louise Kirkham 0 Reputation points

2024-06-20T01:03:24.8533333+00:00

This answer suggesting the each chunk being written to the blob makes sense, but I'm not sure how I prevent this from triggering a run until the whole file is complete.

I don't seem to find any info on 'BlobCreated' and 'BlobCreated (ignore empty blobs)' in the documentation.

Suggested advice has been to include 'ignore empty blobs', which hasn't solved the issue.
Sign in to comment
Nandan Hegde 30,716 Reputation points MVP

2024-07-01T06:02:54.42+00:00

As you have blob trigger enabled, you can set up the parameters to capture the file name and start as stated below :

https://www.c-sharpcorner.com/article/get-file-details-triggering-the-azure-data-factory-pipeline-synapse-via-storag/

to debug.

This would help in debugging the time and file name and based on that we can identify what might be the cause for the same? And take actions accordingly
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Data Factory triggered from bob storage creating multiple runs

2 answers