Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
2 answers
Incremental refresh for azure synapse with sql serverless pool
Hello I have cosmos collection with updatedDate time filed in epochtime stamp format. I have enabled azure synapse for this with serverless sql pool. I want to do the incremental import on this. I followed the steps as mentioned in…
asked 2022-10-18T10:20:17.373+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 62%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Bhuvnesh Kumar
26
Reputation points
answered 2025-02-18T22:12:26.8833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 76%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Rock
41
Reputation points
1 answer
About partition strategies on Synapse Spark Pools
Hi I recently posted this question: https://learn.microsoft.com/en-us/answers/questions/2153368/about-partitioned-parquet-files-on-adls2 I wonder if there is a way to customize the partition size on Synapse Spark. I have a 40GB table that generates a…
1 answer
check for pipe characters in a field and use if condition on this field
In the dataflow, I am checking for data in a field that could have Pipe (|) characters or alphanumeric characters with Pipe(|) between them. If there is only pipe (|)characters, then use iif condition and mark it as Y . If there are alphanumeric…
asked 2025-02-18T17:14:18.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 23%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Peruka, Sikander reddy
90
Reputation points
commented 2025-02-18T19:48:32.3366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,085
Reputation points Microsoft Vendor
1 answer
Connect Local Machine to Synapse Lake Database using Python
Hi, I am trying to develop tooling for my use case, where I am using PySpark/Spark SQL on Synapse Analytics to create tables in Lake database. Sample code below: spark.sql("CREATE TABLE abc AS SELECT * FROM view") The table abc is created in…
asked 2025-02-17T15:02:43.2566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 88%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHN%3C/text%3E%3C/svg%3E)
Haria, Neel
0
Reputation points
commented 2025-02-18T18:51:12.15+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
13,800
Reputation points Microsoft Vendor
0 answers
Error 1502
Im new to Azure....We have spark job thats has been running just fine for a long time we've had some issues with the data format lately but the pipeline was breaking in a different place than now. Now Im getting this error "errorCode":…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 52%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Synapse notebook script runs ok but stops on time out and stuck in queued in pipelines
I have a python script in a notebook that performs a parquet file schema correction. It is working fine, runs in less than 10 seconds, depending on the number of files to process. Now I need to run it in a pipeline that correct schemas and then send…
asked 2025-02-12T12:27:57.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 39%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Ricker Silva
0
Reputation points
commented 2025-02-18T16:07:37.91+00:00
Ricker Silva
0
Reputation points
0 answers
Why are incorrect v-cores allocated?
I created and ran a notebook with one executor. In the monitor section, i see that 48 cores are allocated and i am unable to utilize more cores in other notebooks. I have to wait almost 30 mins after i stop the spark session to create the session…
asked 2025-02-18T05:46:13.2+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 97%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
M Saad
36
Reputation points
commented 2025-02-18T14:15:21.0366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
8,470
Reputation points Microsoft Vendor
2 answers
Azure synapse spark start-up fail: CLUSTER_CREATION_TIMED_OUT: Livy session has failed.
Since a few days none of my spark pools in Azure synapse analytics are able to start-up. After trying for 15 minutes to start, I either get this message: LivySessionCreationTimeoutException: Livy session creation timed out. workspace: we-psch-a-data-syn,…
asked 2025-02-13T08:09:40.0633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 24%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPT%3C/text%3E%3C/svg%3E)
Pieter Tjerkstra
5
Reputation points
commented 2025-02-18T12:41:31.8733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,295
Reputation points Microsoft Vendor
1 answer
All accountid's in the contact-table are NULL when using Azure Synapse Link or a Synapse pipeline to parquet
I've tried two ways to copy data from our Dynamics 365 to a Storage Account on Azure: Azure Synapse Analytics: creating a pipeline that connects to Dynamics 365 and copies the data the a parquet file on a storage account. I tried fetching the data in…
asked 2025-02-17T20:05:44.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 97%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETD%3C/text%3E%3C/svg%3E)
Tim Derdelinckx
0
Reputation points
edited an answer 2025-02-18T08:17:47.0566667+00:00
Chandra Boorla
8,470
Reputation points Microsoft Vendor
1 answer
Azure Synapse managed Vnet default routing through an NVA firewall
We have deployed a synapse instance with a managed vnet, We also have deployed a palo alto azure fw appliance that we want to inspect all traffic with. i understand I can use Private endpoints when communicating with synapse from other vnets and we have…
asked 2025-02-13T19:44:08.5733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 82%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Shawn Schiebrel
5
Reputation points
commented 2025-02-17T19:17:26.3766667+00:00
Shawn Schiebrel
5
Reputation points
2 answers
One of the answers was accepted by the question author.
Azure Synapse pipeline
Hi, I have a question regarding Synapse. I want to create a pipeline that can delta update the target table “quotes_target” from source table “quotes_source” with filtered quotes data. I have a list of quotes id of the updated quotes after I extracted it…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 14.000000000000002%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Disabling for Index of Azure Synapse Dedicated sql pool
Hi, I am having an Azure synapse dedicated sql pool instance but wanted to disable the default index creation as part creating the tables. Please let me know all steps around doing that. Also let me know how much better performance and storage do we get…
asked 2025-02-13T12:50:35.6466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 33%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Nitish Sharma
20
Reputation points
accepted 2025-02-17T07:15:54.61+00:00
Nitish Sharma
20
Reputation points
2 answers
How to save complete web activity output to ADLS Gen2 in JSON format? Azure Synapse Studio
I want to save the output of my Web activity (which hits an Api and then fetches data from that Api in Json format) into blob storage account container. Also, my web activity is within a foreach - so multiple Api links are being hit and multiple files…
asked 2024-10-24T05:42:30.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 20%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mitashi Malkani
0
Reputation points
commented 2025-02-17T01:51:49.8666667+00:00
Chandra Boorla
8,470
Reputation points Microsoft Vendor
1 answer
SharePoint List not showing in Azure Synapse Analytics
I have been setting up SharePoint List and Pulling the data into Azure Synapse Analytics for Power BI report for long time through Linked Service. This time set up a Linked Service. I can see the SharePoint List from Azure Synapse Analytics Studio but…
asked 2025-02-12T16:32:49.83+00:00
Gabe
1
Reputation point
commented 2025-02-17T01:39:30.18+00:00
Chandra Boorla
8,470
Reputation points Microsoft Vendor
1 answer
How to query Azure sql DB using Synapse Notebooks
I want to query my azure sql db from azure synapse notebook. I have linked connection working and can use copy activities to read and write data from said synapse workspace to that db. What i would like to do is query that db straight from notebooks so i…
asked 2025-02-14T08:35:51.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 7.000000000000001%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJH%3C/text%3E%3C/svg%3E)
Jani Hämäläinen
80
Reputation points
answered 2025-02-14T20:13:31.7233333+00:00
phemanth
13,800
Reputation points Microsoft Vendor
1 answer
Synapse with DL and network restrction using Resource instances
Hello, I am at a loss so would greatly appreciate any assistance. I have created a synapse workspace with managed network and a Datalake gen2 as a linked service. The problem is that I am unable to manage the linked DL from Synapse studio as I just get…
asked 2022-04-05T07:56:57.997+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 30%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
theodorbrander
56
Reputation points
answered 2025-02-14T08:14:56.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 88%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JonasK
0
Reputation points
3 answers
One of the answers was accepted by the question author.
ADF Pipelines to Synapse or Fabric
Hi there, I would like to know if the actively running ADF pipelines could be copied and restored into Synapse or Fabric? (lift and shift). If possible Across accounts (to use as templates across accounts and customize/tune ADF pipelines little bit in…
asked 2025-02-11T17:48:15.4033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 69%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
RJ
326
Reputation points
commented 2025-02-14T07:53:24.98+00:00
Chiugo Okpala
235
Reputation points MVP
1 answer
One of the answers was accepted by the question author.
Is Azure Synapse Analytics expected to support Spark 3.5 in the near future?
Is Azure Synapse Analytics expected to support Spark 3.5 in the near future? Spark 3.5 allows to use Delta Lake 3.0 version which improved overall performance significantly. https://delta.io/blog/delta-lake-performance/ Apache Spark 3.5 compatible…
asked 2025-02-10T15:39:23.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 89%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Akjol Syeryekkhaan
20
Reputation points Microsoft Employee
accepted 2025-02-13T20:59:12.2266667+00:00
Akjol Syeryekkhaan
20
Reputation points Microsoft Employee
1 answer
Github interaction with synapse
We have github repo which is connected to our synapse environment, anytime someone configure git it makes a commit to master branch, actually we want to pull the updates from github rather than updating from synapse to git
asked 2025-02-11T06:16:22.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 61%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETA%3C/text%3E%3C/svg%3E)
Tailor, Anurag
0
Reputation points
commented 2025-02-13T12:53:26.5533333+00:00
Chandra Boorla
8,470
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
How to use linked service in Notebook with pyspark
I have pyspark script in Notebook to read and write data in ADLS Gen2. Below is an sample of the pyspark script. But in the Synapse I only have a linked service created with Service Principle could connect to the ADLS Gen2, so I need to specify in…
asked 2023-03-31T04:24:40.7333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 38%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Shao Peng Sun
91
Reputation points
commented 2025-02-13T11:38:00.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 73%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rajan Sethi
5
Reputation points