Share via

Azure Data factory V2 pipeline schedule trigger fails to create records in Dynamics CRM after 1 hour.

Kuldeep Patwal 6 Reputation points
2021-09-17T02:30:18.027+00:00

Actually, I have a pipeline for which I have created a schedule trigger to migrate data from one D365 CRM entity to another but after 1 hour of execution it starts to fail the record creation but pipeline still shows In progress as another activity/task it performs successfully. Please follow the below/attached snap shots for your reference.
132868-image.png
132913-image.png
132924-image.png

Error details (which fails only after 1 hour of pipeline trigger execution):
Error code
2108
Troubleshooting guide
Failure type
User configuration issue
Details
Invoking Web Activity failed with HttpStatusCode - 'Unauthorized', message - ''

Also, please let me know, why its taking too much time to execute only 1000 records. How I can optimize it please suggest.

Just FYI: I have tried to set concurrency as 50 and also change the TTL setting as in below SS. Also, tried to change it to every possible option (i.e. Compute Type, Core Count, Time to live & enable Quick re-use) but still did not get any help.
132878-image.png

Error details (which fails only after 1 hour of pipeline trigger execution):
Error code
2108
Troubleshooting guide
Failure type
User configuration issue
Details
Invoking Web Activity failed with HttpStatusCode - 'Unauthorized', message - ''

Please anyone provide solution asap. Much appreciate for any quick solution.
Thanks.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
0 comments

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. MartinJaffer-MSFT 26,061 Reputation points
    2021-09-17T18:03:23.607+00:00

    Hello @Kuldeep Patwal and welcome to Microsoft Q&A.

    Please tell me if I understand this correctly:

    1. your pipeline starts
    2. you get authorization token
    3. you enter a loop. Inside is a web activity. The web activity uses the authorization token.
    4. After an hour of running in a loop, the web activity starts failing with that message.

    If that is the case, then I believe the cause would be the expiration of the authorization token. To fix this, either the token must be extended, or refreshed. I am thinking of ways this could be done without interrupting the loop.

    1 person found this answer helpful.
    0 comments
  2. Kuldeep Patwal 6 Reputation points
    2021-09-22T08:44:28.227+00:00

    Hi @MartinJaffer-MSFT

    Thanks for your reply and support.

    I tried the above approach and customize it with removing variables but it only processed around 500 records (which was 350 earlier) in an hour. So I have changed whole pipeline to copy data activity which takes records from one entity and migrate data into another CRM entity.

    Thanks

    1 person found this answer helpful.
  3. Kuldeep Patwal 6 Reputation points
    2021-09-19T11:52:47.817+00:00

    Hi @MartinJaffer-MSFT ,
    Thanks for your prompt response :)

    Yes, you are absolutely right - and yes I also believe that its causing by expiration of authorization key But still my concern is why its taking too much time to process only around 1100 records.

    Is there please any other ways to optimize it so that records creation in Dynamics CRM would be fast (as I can see - currently its only processing around 300-350 records) and it will not reach to limit of 1 hour (where token is expiring).

    Please suggest.

    Thanks,
    Kuldeep Patwal