Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,256 questions
Facing latency issue while accessing data from synapse link
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Niharika Paladugu
15
Reputation points
We have created an azure synapse link to load data from dynamics to data lake. Initially we were querying actual tables but that was randomly throwing error as tables not found due to the continuous refresh by synapse link. Microsoft has advised to use _partitioned tables. But we observed that though actual table refresh is happening in less than 15 mins, partitioned tables refresh is taking around 1 hour 15mins causing delays on data availability in target system. Further processing of synapse Pipeline takes 30 mins. Is there any other approach of using actual tables or or any other approach to avoid these delays.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache 20,351 Reputation points2024-02-08T01:14:10.1+00:00 Hello @Lakshmi Niharika Paladugu Looks like you have already approached the Microsoft Tech support, right?
But we observed that though actual table refresh is happening in less than 15 mins, partitioned tables refresh is taking around 1 hour 15mins
Did you try increasing the frequency of your partitioning job to reduce the delta between the analytical store and partitioned store.
Some of the suggestions that we can explore depends on the data volume/refresh!- You can try partitioning your tables based on the columns that are commonly used in your analytical queries. This can help reduce the amount of data processed and improve query performance.
- You can try increasing the frequency of your partitioning job to reduce the delta between the analytical store and partitioned store. Another option is to use incremental loading instead of full loading. This can help reduce the amount of data processed and improve the overall performance of your pipeline
-
Lakshmi Niharika Paladugu 15 Reputation points
2024-02-08T06:17:33.7833333+00:00 How can I increase the frequency of partitioning job to reduce delta between analytical store and partitioned store. Where do I have to nake this change.
-
QuantumCache 20,351 Reputation points
2024-02-08T20:52:16.8166667+00:00 @Lakshmi Niharika Paladugu frequency of partitioning job means adding a trigger to the Pipeline which will pull the data more frequently.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 5%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
sai ravitej vuppalapati 0 Reputation points2024-12-11T16:36:18.35+00:00 @QuantumCache, @Lakshmi Niharika Paladugu
Where can we find the pipeline which copy data from analytical store to partitioned store? We also having similar situation where partitioned tables are refreshed hourly. We want to reduce the refresh to 30 min.
Appreciate your help.
Sign in to comment
Sign in to answer