ADF Pipeline runs successfully but data doesn't show up

Question

ADF Pipeline runs successfully but data doesn't show up

Rishabh Srivastava 60 Microsoft Employee

Hi team

Using Azure Data Factory pipeline using two web activity Source web is reading data from log analytics workspace w1 table t1 and I am passing the result of source web activity as request body of destination web activity then my destination web is writing aggregated data to log analytics workspace w1 table t2.

User's image

When I run debug its working fine and I can see data reading from source and writing to destination by running query in destination log analytics workspace however when I schedule the trigger on this pipeline to run every 15 mins to write aggregated data from source to destination pipeline run is successful as shown below
User's image

but response is empty from source web activity as shown below

User's image

Please help in resolving this.

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-17T06:56:47.1966667+00:00
@Rishabh Srivastava (CIS)

Thanks for using MS Q&A platform and posting your query.

the issue might lie in how the managed identity is being accessed or utilized by the pipeline during scheduled executions. Here are some additional troubleshooting steps to pinpoint the cause:

Managed Identity Permissions:

Double-check that the managed identity assigned to your Data Factory has the Log Analytics reader role assigned specifically for the Log Analytics workspace (w1) and the table (t1) you're reading from. Inconsistent permissions might explain the occasional failures.

Managed Identity Access during Scheduling:

During scheduled runs, the pipeline relies on the managed identity assigned to the Data Factory itself, not your own identity used in debug mode. Verify if any Azure Resource Manager (ARM) templates or configurations used for deployment might be interfering with managed identity access.

ADF Logging and Monitoring:

Enable diagnostic logging for the Web Activity in your Data Factory. This will capture detailed information about the request and response during scheduled runs. Look for specific errors related to authentication failures or permission issues. You can also leverage Azure Monitor to track pipeline execution history and identify any anomalies.

ADF Managed Identity Configuration:

Ensure the Web Activity configuration explicitly references the managed identity assigned to your Data Factory. You might need to set the identity property within the Web Activity definition to use the managed identity for authentication.

Here are some resources that might be helpful:

Troubleshoot Azure Data Factory and Synapse Analytics security and access control issues: https://learn.microsoft.com/en-us/azure/data-factory/data-movement-security-considerations

Azure Data Factory documentation on Web Activity: https://learn.microsoft.com/en-us/azure/data-factory/the issue might lie in how the managed identity is being accessed or utilized by the pipeline during scheduled executions.

Hope this helps. Do let us know if the issue persists
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-20T04:34:51.2666667+00:00

@Rishabh Srivastava We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-05-20T04:34:51.2666667+00:00

@Rishabh Srivastava We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hello Rishabh Srivastava,

Thanks for posting your question in the Microsoft Q&A forum.

it seems that the issue you're facing is related to the authentication or authorization of the Azure Data Factory pipeline when running on a scheduled trigger. when you run the pipeline in debug mode, the authentication and authorization context is likely different from when the pipeline runs on a scheduled trigger.

To ensure a successful connection to the Log Analytics workspace from the Source Web Activity, verify the authentication method being used. If you're using a managed identity, make sure it has the necessary permissions to read data from the Log Analytics workspace. For service principals or other authentication methods, confirm that the credentials or access tokens are valid and have the required permissions.
Grant the necessary permissions to the managed identity or service principal used by the Data Factory pipeline to read data from the Log Analytics workspace.
Ensure that the linked service is configured correctly with the appropriate authentication method and credentials.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful

Rishabh Srivastava 60 Reputation points Microsoft Employee

2024-05-16T18:49:32.9933333+00:00

Hi @hossein jalilian

I am using Managed Identity and its working fine sometimes but its not consistent.
like I am using same Managed Identity in debug mode and it actually gets response from source web activity. I am using web activity not linked service

Share via

ADF Pipeline runs successfully but data doesn't show up

1 answer

Your answer