5,191 questions with Azure Synapse Analytics tags

Sort by: Updated
1 answer

About partition strategies on Synapse Spark Pools

Hi I recently posted this question: https://learn.microsoft.com/en-us/answers/questions/2153368/about-partitioned-parquet-files-on-adls2 I wonder if there is a way to customize the partition size on Synapse Spark. I have a 40GB table that generates a…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-17T15:14:08.6966667+00:00
Jona 575 Reputation points
commented 2025-02-18T19:59:22.5866667+00:00
phemanth 13,800 Reputation points Microsoft Vendor
1 answer

check for pipe characters in a field and use if condition on this field

In the dataflow, I am checking for data in a field that could have Pipe (|) characters or alphanumeric characters with Pipe(|) between them. If there is only pipe (|)characters, then use iif condition and mark it as Y . If there are alphanumeric…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-02-18T17:14:18.44+00:00
Peruka, Sikander reddy 90 Reputation points
commented 2025-02-18T19:48:32.3366667+00:00
Ganesh Gurram 4,085 Reputation points Microsoft Vendor
1 answer

Connect Local Machine to Synapse Lake Database using Python

Hi, I am trying to develop tooling for my use case, where I am using PySpark/Spark SQL on Synapse Analytics to create tables in Lake database. Sample code below: spark.sql("CREATE TABLE abc AS SELECT * FROM view") The table abc is created in…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-17T15:02:43.2566667+00:00
Haria, Neel 0 Reputation points
commented 2025-02-18T18:51:12.15+00:00
phemanth 13,800 Reputation points Microsoft Vendor
0 answers

Error 1502

Im new to Azure....We have spark job thats has been running just fine for a long time we've had some issues with the data format lately but the pipeline was breaking in a different place than now. Now Im getting this error "errorCode":…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-14T12:27:35.3933333+00:00
Siphe 0 Reputation points
commented 2025-02-18T17:49:33.25+00:00
phemanth 13,800 Reputation points Microsoft Vendor
1 answer

Synapse notebook script runs ok but stops on time out and stuck in queued in pipelines

I have a python script in a notebook that performs a parquet file schema correction. It is working fine, runs in less than 10 seconds, depending on the number of files to process. Now I need to run it in a pipeline that correct schemas and then send…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-12T12:27:57.79+00:00
Ricker Silva 0 Reputation points
commented 2025-02-18T16:07:37.91+00:00
Ricker Silva 0 Reputation points
0 answers

Why are incorrect v-cores allocated?

I created and ran a notebook with one executor. In the monitor section, i see that 48 cores are allocated and i am unable to utilize more cores in other notebooks. I have to wait almost 30 mins after i stop the spark session to create the session…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-18T05:46:13.2+00:00
M Saad 36 Reputation points
commented 2025-02-18T14:15:21.0366667+00:00
Chandra Boorla 8,470 Reputation points Microsoft Vendor
2 answers

Azure synapse spark start-up fail: CLUSTER_CREATION_TIMED_OUT: Livy session has failed.

Since a few days none of my spark pools in Azure synapse analytics are able to start-up. After trying for 15 minutes to start, I either get this message: LivySessionCreationTimeoutException: Livy session creation timed out. workspace: we-psch-a-data-syn,…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-13T08:09:40.0633333+00:00
Pieter Tjerkstra 5 Reputation points
commented 2025-02-18T12:41:31.8733333+00:00
Smaran Thoomu 20,295 Reputation points Microsoft Vendor
1 answer

All accountid's in the contact-table are NULL when using Azure Synapse Link or a Synapse pipeline to parquet

I've tried two ways to copy data from our Dynamics 365 to a Storage Account on Azure: Azure Synapse Analytics: creating a pipeline that connects to Dynamics 365 and copies the data the a parquet file on a storage account. I tried fetching the data in…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-02-17T20:05:44.3333333+00:00
Tim Derdelinckx 0 Reputation points
edited an answer 2025-02-18T08:17:47.0566667+00:00
Chandra Boorla 8,470 Reputation points Microsoft Vendor
1 answer

Azure Synapse managed Vnet default routing through an NVA firewall

We have deployed a synapse instance with a managed vnet, We also have deployed a palo alto azure fw appliance that we want to inspect all traffic with. i understand I can use Private endpoints when communicating with synapse from other vnets and we have…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-13T19:44:08.5733333+00:00
Shawn Schiebrel 5 Reputation points
commented 2025-02-17T19:17:26.3766667+00:00
Shawn Schiebrel 5 Reputation points
2 answers One of the answers was accepted by the question author.

Azure Synapse pipeline

Hi, I have a question regarding Synapse. I want to create a pipeline that can delta update the target table “quotes_target” from source table “quotes_source” with filtered quotes data. I have a list of quotes id of the updated quotes after I extracted it…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-01-17T09:16:28.69+00:00
Lotus88 66 Reputation points
accepted 2025-02-17T08:11:17.11+00:00
Lotus88 66 Reputation points
1 answer One of the answers was accepted by the question author.

Disabling for Index of Azure Synapse Dedicated sql pool

Hi, I am having an Azure synapse dedicated sql pool instance but wanted to disable the default index creation as part creating the tables. Please let me know all steps around doing that. Also let me know how much better performance and storage do we get…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-13T12:50:35.6466667+00:00
Nitish Sharma 20 Reputation points
accepted 2025-02-17T07:15:54.61+00:00
Nitish Sharma 20 Reputation points
2 answers

How to save complete web activity output to ADLS Gen2 in JSON format? Azure Synapse Studio

I want to save the output of my Web activity (which hits an Api and then fetches data from that Api in Json format) into blob storage account container. Also, my web activity is within a foreach - so multiple Api links are being hit and multiple files…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2024-10-24T05:42:30.79+00:00
Mitashi Malkani 0 Reputation points
commented 2025-02-17T01:51:49.8666667+00:00
Chandra Boorla 8,470 Reputation points Microsoft Vendor
1 answer

SharePoint List not showing in Azure Synapse Analytics

I have been setting up SharePoint List and Pulling the data into Azure Synapse Analytics for Power BI report for long time through Linked Service. This time set up a Linked Service. I can see the SharePoint List from Azure Synapse Analytics Studio but…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-02-12T16:32:49.83+00:00
Gabe 1 Reputation point
commented 2025-02-17T01:39:30.18+00:00
Chandra Boorla 8,470 Reputation points Microsoft Vendor
1 answer

How to query Azure sql DB using Synapse Notebooks

I want to query my azure sql db from azure synapse notebook. I have linked connection working and can use copy activities to read and write data from said synapse workspace to that db. What i would like to do is query that db straight from notebooks so i…

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-14T08:35:51.3333333+00:00
Jani Hämäläinen 80 Reputation points
answered 2025-02-14T20:13:31.7233333+00:00
phemanth 13,800 Reputation points Microsoft Vendor
1 answer

Synapse with DL and network restrction using Resource instances

Hello, I am at a loss so would greatly appreciate any assistance. I have created a synapse workspace with managed network and a Datalake gen2 as a linked service. The problem is that I am unable to manage the linked DL from Synapse studio as I just get…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2022-04-05T07:56:57.997+00:00
theodorbrander 56 Reputation points
answered 2025-02-14T08:14:56.9733333+00:00
JonasK 0 Reputation points
3 answers One of the answers was accepted by the question author.

ADF Pipelines to Synapse or Fabric

Hi there, I would like to know if the actively running ADF pipelines could be copied and restored into Synapse or Fabric? (lift and shift). If possible Across accounts (to use as templates across accounts and customize/tune ADF pipelines little bit in…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
asked 2025-02-11T17:48:15.4033333+00:00
RJ 326 Reputation points
commented 2025-02-14T07:53:24.98+00:00
Chiugo Okpala 235 Reputation points MVP
1 answer One of the answers was accepted by the question author.

Is Azure Synapse Analytics expected to support Spark 3.5 in the near future?

Is Azure Synapse Analytics expected to support Spark 3.5 in the near future? Spark 3.5 allows to use Delta Lake 3.0 version which improved overall performance significantly. https://delta.io/blog/delta-lake-performance/ Apache Spark 3.5 compatible…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-10T15:39:23.9866667+00:00
Akjol Syeryekkhaan 20 Reputation points Microsoft Employee
accepted 2025-02-13T20:59:12.2266667+00:00
Akjol Syeryekkhaan 20 Reputation points Microsoft Employee
1 answer

Github interaction with synapse

We have github repo which is connected to our synapse environment, anytime someone configure git it makes a commit to master branch, actually we want to pull the updates from github rather than updating from synapse to git

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
GitHub Training
GitHub Training
GitHub: A web-based hosting service for software development and version control using Git. Acquired by Microsoft in 2018.Training: Instruction to develop new skills.
48 questions
asked 2025-02-11T06:16:22.45+00:00
Tailor, Anurag 0 Reputation points
commented 2025-02-13T12:53:26.5533333+00:00
Chandra Boorla 8,470 Reputation points Microsoft Vendor
2 answers One of the answers was accepted by the question author.

How to use linked service in Notebook with pyspark

I have pyspark script in Notebook to read and write data in ADLS Gen2. Below is an sample of the pyspark script. But in the Synapse I only have a linked service created with Service Principle could connect to the ADLS Gen2, so I need to specify in…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,541 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2023-03-31T04:24:40.7333333+00:00
Shao Peng Sun 91 Reputation points
commented 2025-02-13T11:38:00.3+00:00
Rajan Sethi 5 Reputation points
1 answer One of the answers was accepted by the question author.

Compatibility of STRING_SPLIT function inside store procedure for Azure synapse serverless sql pool

I am trying to create a stored procedure in Azure synapse serverless sql pool. The STRING_SPLIT function works fine when used in a select statement but not inside store procedure. I receive an error that it is not recognized. The database compatibility…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,191 questions
asked 2025-02-12T16:03:14.8066667+00:00
Suraj Shashidhar 65 Reputation points
accepted 2025-02-13T10:18:07.35+00:00
Suraj Shashidhar 65 Reputation points