Share via

Operation gets timed out while fetching data from dataverse using ADF pipeline

Sarthak Behera 0 Reputation points
2024-01-29T15:51:47.44+00:00

I need to copy data from dataverse to Azure Blob storage (as a CSV file) through ADF. The source entity (Entity1) in dataverse has lot many Lookup columns. Consequently, the GUID values are getting copied instead of the required values (eg: name in primary entity). As a workaround, I used the Query option and am trying to build a Fetch XML query having necessary linked entities. Firstly, using an inner join-type isn't allowed. Secondly, the operation times out frequently when I use outer join (to join Entity 2) and use filter condition as stated below.

<link-entity name="Entity2" from="Entity2_Attribute1" to="Entity1_Attribute1" visible="false" link-type="outer" alias="Entity2">
	<attribute name="Entity2_Attribute2" />
</link-entity>
<filter type="and">
	<condition entityname="Entity2" attribute="Entity2_Attribute2" operator="not-null"/>
</filter>

Similar to these, there are lot many linked entities and this is causing the copy activity to time out. Please suggest some better practices through which we can display the required values from other entities without impacting performance.

Azure Data Factory
Azure Data Factory

An Azure service for ingesting, preparing, and transforming data at scale.


Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.