Where to find/log Azure Data Factory detailed YARN logs

Question

Where to find/log Azure Data Factory detailed YARN logs

Prince Rastogi Admin Account 21

We are migrating to ADF with almost 1000 of pipelines and data flow. Where it throws some error then it is very generic error which is like not specific error to fix the problem quickly.

When Data Flow triggers then it starts a cluster and somewhere generates detailed YARN logs. We have already enabled Diagnostics on ADF level for all events.

Where can I find those logs for troubleshooting and fix?
If this is not there by default then what setting we need to change to get those detailed YARN logs?

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-12-07T16:23:56.0233333+00:00

@Prince Rastogi Admin Account Just checking in to see if the below information was helpful. If it answers your query, please do click Accept Answer and Yes for "was this answer helpful", as it might be beneficial to other community members reading this thread. If you have any further query, do let us know.

Thank you

1 answer

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-12-07T16:23:56.0233333+00:00

@Prince Rastogi Admin Account Just checking in to see if the below information was helpful. If it answers your query, please do click Accept Answer and Yes for "was this answer helpful", as it might be beneficial to other community members reading this thread. If you have any further query, do let us know.

Thank you

Answer 1

KranthiPakala-MSFT 46,642 Microsoft Employee Moderator

@Prince Rastogi Admin Account Welcome to Microsoft Q&A forum and thanks for reaching out here.

Unfortunately, there is no way a user can access the backend cluster logs of your data flow. Those are managed by Azure and I believe only the ADF product team has access to those logs.

Setting up diagnostic logs only will help track below tables data but not the detailed YARN logs of your ADF mapping data flow.
Screenshot that shows where diagnostic logs tab is located in data factory.

For most of the common issues related to ADF mapping data flow, the product team has consolidated a portion of the list of issues that commonly reported and also provided possible root cause and resolution details in this document. Kindlly refer to this document and see if that helps to identify your specific issue and possible solution

MS Doc:

In case if your issue is not listed in the above troubleshooting guides and are blocked, kindly share the error message details and few details about your pipeline flow. Or if you have a support plan, please file a support ticket and do share the SR# with us here.

Hope this info helps.

Prince Rastogi Admin Account 21 Reputation points

2023-12-01T15:15:21.9733333+00:00

@KranthiPakala-MSFT

We already configured Diagnostic to store on log analytics workspace side and getting same level of error details (not detailed one).
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-12-05T16:13:21.44+00:00

@Prince Rastogi Admin Account Diagnostic logs will only contain the data for the tables mentioned above. For deeper logs, users won't be able to access them. If you need further troubleshooting, I would recommend filing a support ticket as the product team has access to the backend logs.

If you have any feedback for the product team, I would highly encourage you to please log it in Azure data factory IDEAS forum here: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c and please do share the link here so that we can share it with respective product team to further review it.

Share via

Where to find/log Azure Data Factory detailed YARN logs

1 answer

Your answer