azure data factory - service now connector

Question

azure data factory - service now connector

brajesh jaishwal 26

I am trying to use service now connector in azure data factory, for 1 day data pull, copy activity works fine with Actual.{table_name} but not working for Display.{table_name}. I am using self hosted run time, that has 8vCPU and 32 GB memory. have given few days, but not able to solve the issue.

AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2022-06-08T06:16:05.547+00:00
Hi @brajesh jaishwal ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query.
In order to provide you better help, could you please share the below details:

When you say copy activity is working for 'Actual' schema , but not for 'Display' schema , what is the error you are getting for 'Display' schema .

Please provide the screenshot of source and sink settings of your Copy activity to check where went wrong in the configuration.

Thanks, looking forward to your response.

Answer accepted by question author

0 additional answers

Your answer

AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2022-06-08T06:16:05.547+00:00

Hi @brajesh jaishwal ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query.
In order to provide you better help, could you please share the below details:

When you say copy activity is working for 'Actual' schema , but not for 'Display' schema , what is the error you are getting for 'Display' schema .

Please provide the screenshot of source and sink settings of your Copy activity to check where went wrong in the configuration.

Thanks, looking forward to your response.

Answer 1

Hi @brajesh jaishwal ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query.

As we connected and discussed over the issue being internal to the organization , it seems the issue is the query you are running to fetch the incremental data is taking a lots of time to filter out records based on the where clause .

select @{variables('display_column_str')} from Display.@{pipeline().parameters.source_table where @{variables('ChangefeedColumn')}_display_value >= '@{variables('StartTime')}' and @{variables('ChangefeedColumn')}_display_value <= '@{variables('EndTime')}'

Pipeline is working fine if we remove the where clause. So , it doesn't seem to be an ADF issue. However, a deeper investigation is needed on how with the same number of records being retrieved for 'Actual' table is working within a minute of pipeline execution. I would suggest you to keep following up on the Support ticket that you have raised.

Hope this will help.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.
Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

brajesh jaishwal 26 Reputation points

2022-06-08T12:13:51.92+00:00

Hi @AnnuKumari,

Thanks for trying to help solve the issue. I will point to the count query that we tried,

select count(*) from Display.sc_task

There was no result for this query.

For the same, rest request takes few seconds to complete

https://linkedin.service-now.com/api/now/stats/sc_task?sysparm_count=true { "result": { "stats": { "count": "406377" } } }

So I would say something is happening at ADF service now connector side, and agree it needs a deeper investigation. I will follow up this issue via MS Support.
AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator

2022-06-08T13:35:41.92+00:00

I agree @brajesh jaishwal .

It would be much appreciated if you would share the resolution once you hear from the support team. Thanks in advance !

Kindly consider clicking Accept Answer as accepted answer helps community as well.

Share via

azure data factory - service now connector

0 additional answers

Your answer