An Azure service for ingesting, preparing, and transforming data at scale.
Operation gets timed out while fetching data from dataverse using ADF pipeline
I need to copy data from dataverse to Azure Blob storage (as a CSV file) through ADF. The source entity (Entity1) in dataverse has lot many Lookup columns. Consequently, the GUID values are getting copied instead of the required values (eg: name in primary entity). As a workaround, I used the Query option and am trying to build a Fetch XML query having necessary linked entities. Firstly, using an inner join-type isn't allowed. Secondly, the operation times out frequently when I use outer join (to join Entity 2) and use filter condition as stated below.
<link-entity name="Entity2" from="Entity2_Attribute1" to="Entity1_Attribute1" visible="false" link-type="outer" alias="Entity2">
<attribute name="Entity2_Attribute2" />
</link-entity>
<filter type="and">
<condition entityname="Entity2" attribute="Entity2_Attribute2" operator="not-null"/>
</filter>
Similar to these, there are lot many linked entities and this is causing the copy activity to time out. Please suggest some better practices through which we can display the required values from other entities without impacting performance.