4,365 questions with Azure Synapse Analytics tags

Sort by: Updated
1 answer

The new ADF/Synapse connector to Salesforce no longer supports QueryAll to query deleted records in Salesforce. How can this be done?

Hi, The previous version of the ADF Salesforce connector (https://learn.microsoft.com/en-us/azure/data-factory/connector-salesforce-legacy?tabs=data-factory) supported querying deleted records in Salesforce: But the new version…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions
asked 2024-02-28T21:15:09.6433333+00:00
Kevin Chadney 10 Reputation points
commented 2024-04-24T18:58:43.8266667+00:00
BhargavaGunnam-MSFT 26,136 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

how to get exception from synapse notebook in pipeline activity

Hi Team, I ran a notebook from the pipeline and it has failed with exception in the notebook but pipeline activity has showing succeeded. How can i get the error details of the notebook from pipeline and will send the error notification through…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-02-07T02:10:49.2866667+00:00
Veera Satya Teja Mummidi 40 Reputation points
commented 2024-04-24T18:25:53.87+00:00
Trishul Sangu [C] 0 Reputation points
1 answer

Interpreter died error after updating spark pool from 3.1 to 3.3 in Azure synapse

I have updated the spark version from 3.1 to 3.3 in spark pool . But now the Spark pool is not working. It give interpreter died error when I run any cell in notebook.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-01-10T06:38:50.11+00:00
Abhijeet Singh 25 Reputation points
commented 2024-04-24T14:49:09.4633333+00:00
Pavlina 0 Reputation points
1 answer

Cosmos Synapse link SQL query error

Hi, we have a Synapse SQL View onto a Cosmos container (using Synapse link). When we run a SELECT * on the view we sometimes get this error: Resolving CosmosDB path has failed with error 'CosmosDB request failed with status code '449';…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,441 questions
asked 2024-04-21T12:33:17.8733333+00:00
David Baker 21 Reputation points
commented 2024-04-24T10:34:11.61+00:00
Smaran Thoomu 9,525 Reputation points Microsoft Vendor
1 answer

Get Metadata synapse activity returns wrong data type from parquet file

When the synapse pipeline creates external table we need data types, so currently we're using Get Metadata synapse activity that returns the columns types from the parquet file of the corresponding table.    For the couple of tables the data type…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-22T11:35:21.6166667+00:00
Nataliia Peresada 0 Reputation points
answered 2024-04-24T10:28:09.5266667+00:00
phemanth 5,735 Reputation points Microsoft Vendor
2 answers

Best resource class for an ADF managed identity database user in a Synapse SQL pool

Hi, for a my Azure data platform solution I'm using Data Factory to implement the pipelines and Synapse Analytics to store the data read from the sources. I've a dedicated SQL pool DW100c. Inside the SQL pool I've specified the ADF system managed…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions
asked 2024-03-06T10:01:21.73+00:00
pmscorca 792 Reputation points
answered 2024-04-24T08:46:57+00:00
Pinaki Ghatak 2,320 Reputation points Microsoft Employee
1 answer One of the answers was accepted by the question author.

Synapse Serverless SQL pool, first query takes longer

While attempting to run a synapse query on an OPENROWSET with format "Delta", the first query takes 15-40 seconds extra compared to every other query run after this irrespective of the filepath the query is run on. Same behaviour is observed…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-15T07:04:40.6433333+00:00
Kemla Mukul 20 Reputation points
accepted 2024-04-24T07:54:11.8+00:00
Kemla Mukul 20 Reputation points
1 answer

How to fix "LeaseIdMissing" or "LeaseNotPresent" Error in Synapse pipeline

Hello, In past couple of weeks we are facing issues with data lake loads. As a part of the process we delete historical data from the data lake and reload the data into the data lake through synapse pipelines. When trying to delete the historical data…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,338 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions
asked 2023-10-30T06:46:56.86+00:00
Kiran 26 Reputation points
answered 2024-04-24T04:10:25.4633333+00:00
Vijayan, Uvaneswaran 20 Reputation points
1 answer

How to query Kusto data in Synapse with cert auth in linked service

Hi, I'm trying to query Kusto data with linked service in Synapse. Synapse can generate the samples automatically as below. It can work when I use secret of application in setting up linked service. But it failed with the error when I tried to use cert…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-10T06:26:26.6+00:00
Meng Zhang 31 Reputation points Microsoft Employee
commented 2024-04-23T21:45:52.0233333+00:00
Venkat P 0 Reputation points
2 answers One of the answers was accepted by the question author.

Copying complex JSON data dynamically using Mapping Data Flows in Azure

In my current data engineering project, I am tasked with building a pipeline that can efficiently handle various JSON structures. One of the key requirements is the ability to dynamically flatten nested arrays within the JSON data using parameters, and…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions
asked 2024-04-02T09:01:14.78+00:00
Curiel, H. (Hosé) 20 Reputation points
accepted 2024-04-23T20:06:24.5266667+00:00
Curiel, H. (Hosé) 20 Reputation points
1 answer

reduce waiting time within a synapse pipeline/ improve performance synapse pipeline

A certain synapse pipeline takes a very long time. When I look at the various activities within this pipeline, the activities themselves don't take much time at all. It's mostly the waiting time between them that is consuming a lot of time. We are using…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-15T12:38:49.0633333+00:00
MG 5 Reputation points
commented 2024-04-23T13:07:42.0033333+00:00
Vinodh247-1375 11,046 Reputation points
1 answer

Single Pyspark notebook to call multiple notebooks through parameters

I have one pyspark notebook1 which loads one ERP system (ex:A ) , another notebook loads one more ERP system ( ex: B) . I want to create another notebook to call above 2 notebooks through parameters .. the notebook should ask parameter to enter ERP …

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-18T09:43:02.48+00:00
Bathini, Shirish 0 Reputation points
answered 2024-04-23T11:29:59.9666667+00:00
Smaran Thoomu 9,525 Reputation points Microsoft Vendor
2 answers

Stored Procedure Activity getting failed

Hi, I have got to use couple of external tables(parquet) in dedicated sql pool. when I run stored procedures in dedicated sql pool using those external tables, its running successfully. But when I started running stored procedure using stored procedure…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-22T11:45:55.77+00:00
Anirudh Govardhanam 0 Reputation points
answered 2024-04-23T11:00:27.5666667+00:00
Smaran Thoomu 9,525 Reputation points Microsoft Vendor
2 answers

LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: d2425fce-9179-49eb-829c-a7b2a0d963ed.

Hi Everyone, I am getting the Below Error Code while running my cells in Synapse Notebook: > LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID:…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,917 questions
asked 2022-11-22T06:13:46.587+00:00
Devender 61 Reputation points
commented 2024-04-23T08:30:09.7233333+00:00
Dom Sadie 0 Reputation points
1 answer One of the answers was accepted by the question author.

How to use IS NOT NULL in synapse dataflow

I would like to use IS NOT NULL in synapse's dataflow, but I can't get it to work. Can you please tell me how to do it? I have also tried notNull, but it doesn't work... I used the dataflow filter.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2022-12-20T06:36:09.603+00:00
NishimuraChinatsu-9854 756 Reputation points
edited a comment 2024-04-23T05:22:45.2966667+00:00
Javid Ali Khan Lavani 0 Reputation points
0 answers

Synapse spark V3.4 bug??

I am unable to create a database using Spark 3.4. Is there any existing issue with spark 3.4?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-22T18:59:04.6833333+00:00
Michael M 40 Reputation points
edited a comment 2024-04-22T19:36:08.8033333+00:00
hossein jalilian 2,375 Reputation points
1 answer One of the answers was accepted by the question author.

Utilizing Azure Key Vault secrets for secure password management in Apache spark connector for SQL Server

In the provided code example, our team demonstrates using a hard-coded username and password for connecting to SQL Server using the Apache Spark connector. However, for enhanced security, I want to utilize a password stored in Azure Key Vault instead of…

Azure SQL Database
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2024-04-18T16:57:26.27+00:00
Thomasville99 260 Reputation points
commented 2024-04-22T17:35:16.8133333+00:00
Thomasville99 260 Reputation points
3 answers One of the answers was accepted by the question author.

Configuring Alert Rules to Not Send Secondary Resolved Notification

We have set up an alert rule to notify us when a pipeline has failed in our synapse works (when pipeline failure > 0). The evaluation is based on a period of 5 minutes with a frequency of evaluation every 1 minute. The set up works well in notifying…

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,802 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2020-12-17T17:38:39.95+00:00
Moore, Payton E 101 Reputation points
answered 2024-04-22T13:25:01.6466667+00:00
Segun Akinyemi 0 Reputation points Microsoft Employee
2 answers

How to use Localhost (On-premise database) as sink in Azure Data Factory? It's possible to have source as On-premise database but why not as "Sink"?

I want to use my local database on-premise as "SINK" in Azure Data Factory. It's possible to have it as a "Source" using self-hosted-integration-runtime but why not as a "SINK"???

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
12,707 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions
asked 2024-04-21T14:04:29.1+00:00
Abhinandan Shrestha 20 Reputation points
answered 2024-04-22T11:06:25.9266667+00:00
Harishga 3,260 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Azure Synapse Serverless SQL backup/recovery

We use Azure Synapse Workspaces, with Serverless SQL pools. How would one implement a disaster recovery solution to support this (across Azure regions, with geo-replicated StorageV2 (general purpose v2) datalake storage accounts.)? The automated…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,365 questions
asked 2022-07-29T12:06:48.387+00:00
Darren-1990 21 Reputation points
edited a comment 2024-04-22T10:57:55.8233333+00:00
Darren-1990 21 Reputation points