Azure Data Factory on trigger event not working properly

Question

Azure Data Factory on trigger event not working properly

thomasthomas 1

Hi,

I am using Azure Data Factory V2 and want to set up a trigger based on a blob creation in a Gen2 ADLS.

When I upload some file from my laptop (through Azure Storage Explorer) to the folder that the trigger activity keeps checking - everything works just fine. Trigger is firing and it everything looks good.

However when this trigger file is created by one of our applications that uses the Azure Storage Blobs client library for .NET then the event grid captures this event, but it does not deliver it to ADF. In other words, the pipeline is not getting triggered. So I got stuck at this point. All the services are in the East US.

For what it's worth, the Azure Blob storage is of version StorageV2 - which is supported based on the latest documentations.

On top of that what makes this really weird is that if I check the trigger activity in ADF then it does show all the trigger files - including those that were created by the lib above.

For experimental purposes, I downloaded one of these trigger files created by .NET. When I uploaded it from my laptop, it seemed to work. Neither did I change its content nor its name.

Please advise,
Tamas

svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-09-13T09:40:36.03+00:00

Hello @Thomas Thomas ,
Welcome to the Microsoft Q&A platform.

You mentioned that you are uploading file through Application through a custom code. In your code, can you please let me know the method you are using to upload the fille ? If you are using Flush method to upload the file ?

Can you pass the parameter close=true in the flush method? The default is set to false. If the value of close parameter is true and the flush operation completes successfully, the service raises a file change notification with a property indicating that this is the final update.
thomasthomas 1 Reputation point

2021-09-13T10:01:17.723+00:00

Hello @svijay-MSFT ,

Thats the method we use:

azure.storage.blobs.blobclient.uploadasync

So every time when someone pushes a button in some UI, a (non-empty) trigger file is created in the Blob storage under the hood - and we want ADF to trigger some data pipeline.

The name of the trigger file is unique. Neither does the app append to an existing file nor does it overwrite.
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-09-13T18:55:00.767+00:00

Thanks @thomasthomas - what is the SDK version you are using ? As well I did a quick check at my end. Spun up the sample Scripts referenced from here. I tested the behavior in both V11 and V12 SDK. It worked as expected. I'd recommend you test the simple script at your end and see whether that works. The below is the config I had for the trigger :

svijay-MSFT 5,256 Microsoft Employee Moderator

namespace BlobQuickstartV12
{
    class Program
    {
        static async Task Main()
        {
            Console.WriteLine("Azure Blob Storage v12 - .NET quickstart sample\n");
            string connectionString = "<>";
            BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
            //Create a unique name for the container
            string containerName = "quickstartblobs" + Guid.NewGuid().ToString();
            // Create the container and return a container client object
            BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName);
            // Create a local file in the ./data/ directory for uploading and downloading
            string localPath = "./data/";
            string fileName = "quickstart" + Guid.NewGuid().ToString() + ".txt";
            string localFilePath = Path.Combine(localPath, fileName);

            // Write text to the file
            await File.WriteAllTextAsync(localFilePath, "Hello, World!");

            // Get a reference to a blob
            BlobClient blobClient = containerClient.GetBlobClient(fileName);

            Console.WriteLine("Uploading to Blob storage as blob:\n\t {0}\n", blobClient.Uri);

            // Upload data from the local file
            await blobClient.UploadAsync(localFilePath,true);
        }
    }

}

thomasthomas 1 Reputation point

2021-09-16T07:47:53.373+00:00

Hi @svijay-MSFT

We tested your template script and it failed to work in our end.

Again, it works when the webserver is run locally. But it stops working once it is deployed to an Azure App Service. We created a free app service for testing purposes. Indeed, the application can create the trigger file in the Blob, we can see that. However, nothing happens after then. The ADF event trigger does not capture this if the file is created from the app hosted in this app service instance. :(
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-09-17T05:09:59.247+00:00

@thomasthomas ,

I tried to deploy my app in the app service - Unfortunately, I was not able to observe the issue. Just wanted to check with you whether the below numbers (Published Events - Delivery Events Sum) in the Events Section of Storage account. Is there any failure in the delivery?

Also, can you please elaborate the below line from question :

FerreiraN 0

I'm having the same issue, I can successfully upload a Blob using TypeScript with the script below, but the event trigger is not working. When manually uploading a blob my trigger works, through the UI.

import { BlobServiceClient } from "@azure/storage-blob"

async function handleWriteToBlob(blobData, insurance) {
  const jsonData = sendJson(blobData);
  const containerName = "raw-layer";
  const fileName = blobData['timestamp'];
  const blobName = `/folder_name/${insurance}/${insurance}-insurance_form-${fileName.toISOString()}.json`;

  const blobServiceClient = new BlobServiceClient(`https://${accountName}.blob.core.windows.net?${sasToken}`);
  const containerClient = blobServiceClient.getContainerClient(containerName)
  const blockBlobClient = containerClient.getBlockBlobClient(blobName);

  const options = {
    blobHTTPHeaders: {
      blobContentType: 'application/json'
    },
    bufferThreshold: 4 * 1024 * 1024, // 4MB
    maxBuffers: 20 // 80MB
  };

  const result = await blockBlobClient.upload(
    jsonData,
    jsonData.length,
    options
  );

  if (result.errorCode) {
    throw Error(result.errorCode);
  }

  console.log(`Uploaded BlockBlob ${blobName} successfully as: ${jsonData}`);
}

function sendJson(blobData) {
  console.log(`${JSON.stringify(blobData)}`);
  return JSON.stringify(blobData);
}

Your answer

svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-09-13T09:40:36.03+00:00

Hello @Thomas Thomas ,
Welcome to the Microsoft Q&A platform.

You mentioned that you are uploading file through Application through a custom code. In your code, can you please let me know the method you are using to upload the fille ? If you are using Flush method to upload the file ?

Can you pass the parameter close=true in the flush method? The default is set to false. If the value of close parameter is true and the flush operation completes successfully, the service raises a file change notification with a property indicating that this is the final update.
thomasthomas 1 Reputation point

2021-09-13T10:01:17.723+00:00

Hello @svijay-MSFT ,

Thats the method we use:

azure.storage.blobs.blobclient.uploadasync

So every time when someone pushes a button in some UI, a (non-empty) trigger file is created in the Blob storage under the hood - and we want ADF to trigger some data pipeline.

The name of the trigger file is unique. Neither does the app append to an existing file nor does it overwrite.
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-09-13T18:55:00.767+00:00

Thanks @thomasthomas - what is the SDK version you are using ? As well I did a quick check at my end. Spun up the sample Scripts referenced from here. I tested the behavior in both V11 and V12 SDK. It worked as expected. I'd recommend you test the simple script at your end and see whether that works. The below is the config I had for the trigger :
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-09-13T18:55:26.63+00:00

namespace BlobQuickstartV12 { class Program { static async Task Main() { Console.WriteLine("Azure Blob Storage v12 - .NET quickstart sample\n"); string connectionString = "<>"; BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString); //Create a unique name for the container string containerName = "quickstartblobs" + Guid.NewGuid().ToString(); // Create the container and return a container client object BlobContainerClient containerClient = await blobServiceClient.CreateBlobContainerAsync(containerName); // Create a local file in the ./data/ directory for uploading and downloading string localPath = "./data/"; string fileName = "quickstart" + Guid.NewGuid().ToString() + ".txt"; string localFilePath = Path.Combine(localPath, fileName); // Write text to the file await File.WriteAllTextAsync(localFilePath, "Hello, World!"); // Get a reference to a blob BlobClient blobClient = containerClient.GetBlobClient(fileName); Console.WriteLine("Uploading to Blob storage as blob:\n\t {0}\n", blobClient.Uri); // Upload data from the local file await blobClient.UploadAsync(localFilePath,true); } }

}
thomasthomas 1 Reputation point

2021-09-16T07:47:53.373+00:00

Hi @svijay-MSFT

We tested your template script and it failed to work in our end.

Again, it works when the webserver is run locally. But it stops working once it is deployed to an Azure App Service. We created a free app service for testing purposes. Indeed, the application can create the trigger file in the Blob, we can see that. However, nothing happens after then. The ADF event trigger does not capture this if the file is created from the app hosted in this app service instance. :(
svijay-MSFT 5,256 Reputation points Microsoft Employee Moderator

2021-09-17T05:09:59.247+00:00

@thomasthomas ,

I tried to deploy my app in the app service - Unfortunately, I was not able to observe the issue. Just wanted to check with you whether the below numbers (Published Events - Delivery Events Sum) in the Events Section of Storage account. Is there any failure in the delivery?

Also, can you please elaborate the below line from question :
FerreiraN 0 Reputation points

2023-07-18T18:29:42.6166667+00:00

I'm having the same issue, I can successfully upload a Blob using TypeScript with the script below, but the event trigger is not working. When manually uploading a blob my trigger works, through the UI.

import { BlobServiceClient } from "@azure/storage-blob" async function handleWriteToBlob(blobData, insurance) { const jsonData = sendJson(blobData); const containerName = "raw-layer"; const fileName = blobData['timestamp']; const blobName = `/folder_name/${insurance}/${insurance}-insurance_form-${fileName.toISOString()}.json`; const blobServiceClient = new BlobServiceClient(`https://${accountName}.blob.core.windows.net?${sasToken}`); const containerClient = blobServiceClient.getContainerClient(containerName) const blockBlobClient = containerClient.getBlockBlobClient(blobName); const options = { blobHTTPHeaders: { blobContentType: 'application/json' }, bufferThreshold: 4 * 1024 * 1024, // 4MB maxBuffers: 20 // 80MB }; const result = await blockBlobClient.upload( jsonData, jsonData.length, options ); if (result.errorCode) { throw Error(result.errorCode); } console.log(`Uploaded BlockBlob ${blobName} successfully as: ${jsonData}`); } function sendJson(blobData) { console.log(`${JSON.stringify(blobData)}`); return JSON.stringify(blobData); }

Share via

Azure Data Factory on trigger event not working properly

Your answer