Azure CDM conversion issue

Question

Azure CDM conversion issue

Shibhin Mohan 1

Hi, I'm getting filenotfound error in CDM root folder clear activity in ADF dataflow. It was working fine till last Friday. The process is looking for the CDM entity model json file inside the root folder path which is not expected. It should be points to the corpus folder. Please let me know if you need additional information on this issue.

Error : {"message":"at Sink 'sink1': Could not read '/data.cdm.json' from the 'local' namespace. Reason 'java.io.FileNotFoundException: HEAD https://XXXXXXXX.dfs.core.windows.net/cdmcontainer1/cdm/root/data.cdm.json?timeout=90\nStatusCode=404\nStatusDescription=The specified path does not exist.\nErrorCode=\nErrorMessage='. Details:at Sink 'sink1': Could not read '/data.cdm.json' from the 'local' namespace. Reason 'java.io.FileNotFoundException: HEAD https://XXXXXXXX.dfs.core.windows.net/cdmcontainer1/cdm/root/data.cdm.json?timeout=90\nStatusCode=404\nStatusDescription=The specified path does not exist.\nErrorCode=\nErrorMessage='","failureType":"UserError","target":"dataflow1","errorCode":"DFExecutorUserError"}

KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-08-25T17:12:46.263+00:00

Hi @Shibhin Mohan ,

Welcome to Microsoft Q&A platform and thanks for using this forum.

Sorry for the delay in response. My understanding is that your ADF dataflow pipeline was working fine until last Friday and suddenly started to error out with the above error message without any changes to your pipeline code - please correct me if misunderstood your problem. If that is the case, could you please provide the failed pipeline runID as well as failed activity runID and the location of your data factory so that I can with work internal team to check the logs.

Looking forward to your response and thanks for patience.

Thank you
Shibhin Mohan 1 Reputation point

2020-08-27T10:00:30.69+00:00

Hi @KranthiPakala-MSFT The error I have mentioned above only happening in the debug mode. It is working fine when I trigger the published pipeline.
Shibhin Mohan 1 Reputation point

2020-08-27T10:36:39.967+00:00

Requested Informations
ADF Location : Central India
Failed Pipeline Run ID = 2ddc69b6-74aa-42c6-8709-ba794d240495
Activity ID = c4075ddc-8374-4940-8644-de7ab06dd704
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-08-27T23:35:37.637+00:00

Hi @Shibhin Mohan ,

Thank you for sharing additional details. Before I escalate to internal team, could you please confirm if your published pipeline code and debug pipeline code are same? or any changes implemented recently?

Please confirm.

Thanks
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-08-28T22:22:38.25+00:00

Hi @Shibhin Mohan ,

Just wanted to check if you have got a chance to validate if your published pipeline code is same as the pipeline code you are trying to debug? Also by looking at the error message, it seems like the corpus isn't set up properly. Could you please validate that too.

Thank you
Shibhin Mohan 1 Reputation point

2020-08-29T17:24:16.027+00:00

Hi @KranthiPakala-MSFT Not this pipeline run but I have executed one of my CDM Conversion pipeline in debug mode and trigger mode. In debug mode it is throwing error. The clear folder option in CDM is causing the issue.
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-09-02T01:29:39.933+00:00

Hi @Shibhin Mohan ,

Thanks for your response and clarification. I have reached out to internal team to get additional information regarding this issue.
In the meantime could you please share both pipeline run ID's (executed on debug mode and trigger mode), which might help us to understand the difference in behavior. And also please confirm whether the ADF data flow pipeline code is same for both the runs (debug & trigger)?

I strongly doubt that there could be a difference in ADF pipeline code for both (debug run and trigger run). If possible please do share ADF support files to AzCommunity[at]microsoft[dot]com for further analysis with below details.

Email subject: <AttnKranthi : Microsoft Q&A Thread title>
Thread URL: <Microsoft Q&A Thread>
Subscription ID: <your subscription id>
Attached ADF support files
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-09-02T22:50:32.667+00:00

Hi @Shibhin Mohan ,

Following up to see if you have got a chance to see my previous response. If so could you please share the details requested for further investigation.
Thank you
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-09-05T00:33:46.007+00:00

Hi ShibhinMohan-1373 ,

Just checking to see if you have got a chance to see my previous response. If so could you please share the details requested for further investigation.
Thank you
Shibhin Mohan 1 Reputation point

2020-09-08T06:29:05.35+00:00

Hi @KranthiPakala-MSFT Sorry for the delayed reply. I don't have the last successful run id for the pipeline. Please find below the latest run details.

Run ID : b2a0e982-5393-4031-a6da-eaf2c26e37ca
ADF Location : East US
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-09-14T23:52:38.46+00:00

Hi @Shibhin Mohan ,

Sorry for the delay. After having initial investigation, it seems that it requires a deeper analysis to figure out the root cause. For deeper investigation and immediate assistance, If you have a support plan you may file a support ticket, else could you please send an email to AzCommunity[at]Microsoft[dot]com with the below details, so that we can enable a one-time-free support for you to work closely on this matter.

Subject of the email: <Attn - Kranthi : Microsoft Q&A Thread title>
Thread URL: <Microsoft Q&A Thread>
Subscription ID: <your subscription id>

Please let me know here once the email is sent.

Looking forward to your response.
Thank you
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-09-15T20:00:21.687+00:00

Hi @Shibhin Mohan ,

Just checking to see if you have got a chance to see my previous comment? In case if you have a support plan and have already opened a SR, then please do share SR# with us for tracking this issue internally. Otherwise please email the details requested in my previous response.

Let me know here once the email is sent.

Thank you
KranthiPakala-MSFT 46,827 Reputation points Moderator

2020-09-16T21:58:44.343+00:00

Hi,

We still have not heard back from you. Just wanted to check if you are you still facing the issue? In case If you already found a solution, would you please share it here with the community? Otherwise, please do share the details requested above to file a support request.

Thank you
Shibhin Mohan 1 Reputation point

2020-10-05T10:56:35.18+00:00

Hi, I came to know that this is an know issue in the CDM clear folder option. Thanks for all your help