The new ADF/Synapse connector to Salesforce no longer supports QueryAll to query deleted records in Salesforce. How can this be done?
Hi, The previous version of the ADF Salesforce connector (https://learn.microsoft.com/en-us/azure/data-factory/connector-salesforce-legacy?tabs=data-factory) supported querying deleted records in Salesforce: But the new version…
how to get exception from synapse notebook in pipeline activity
Hi Team, I ran a notebook from the pipeline and it has failed with exception in the notebook but pipeline activity has showing succeeded. How can i get the error details of the notebook from pipeline and will send the error notification through…
Interpreter died error after updating spark pool from 3.1 to 3.3 in Azure synapse
I have updated the spark version from 3.1 to 3.3 in spark pool . But now the Spark pool is not working. It give interpreter died error when I run any cell in notebook.
Cosmos Synapse link SQL query error
Hi, we have a Synapse SQL View onto a Cosmos container (using Synapse link). When we run a SELECT * on the view we sometimes get this error: Resolving CosmosDB path has failed with error 'CosmosDB request failed with status code '449';…
Get Metadata synapse activity returns wrong data type from parquet file
When the synapse pipeline creates external table we need data types, so currently we're using Get Metadata synapse activity that returns the columns types from the parquet file of the corresponding table. For the couple of tables the data type…
Best resource class for an ADF managed identity database user in a Synapse SQL pool
Hi, for a my Azure data platform solution I'm using Data Factory to implement the pipelines and Synapse Analytics to store the data read from the sources. I've a dedicated SQL pool DW100c. Inside the SQL pool I've specified the ADF system managed…
Synapse Serverless SQL pool, first query takes longer
While attempting to run a synapse query on an OPENROWSET with format "Delta", the first query takes 15-40 seconds extra compared to every other query run after this irrespective of the filepath the query is run on. Same behaviour is observed…
How to fix "LeaseIdMissing" or "LeaseNotPresent" Error in Synapse pipeline
Hello, In past couple of weeks we are facing issues with data lake loads. As a part of the process we delete historical data from the data lake and reload the data into the data lake through synapse pipelines. When trying to delete the historical data…
How to query Kusto data in Synapse with cert auth in linked service
Hi, I'm trying to query Kusto data with linked service in Synapse. Synapse can generate the samples automatically as below. It can work when I use secret of application in setting up linked service. But it failed with the error when I tried to use cert…
Copying complex JSON data dynamically using Mapping Data Flows in Azure
In my current data engineering project, I am tasked with building a pipeline that can efficiently handle various JSON structures. One of the key requirements is the ability to dynamically flatten nested arrays within the JSON data using parameters, and…
reduce waiting time within a synapse pipeline/ improve performance synapse pipeline
A certain synapse pipeline takes a very long time. When I look at the various activities within this pipeline, the activities themselves don't take much time at all. It's mostly the waiting time between them that is consuming a lot of time. We are using…
Single Pyspark notebook to call multiple notebooks through parameters
I have one pyspark notebook1 which loads one ERP system (ex:A ) , another notebook loads one more ERP system ( ex: B) . I want to create another notebook to call above 2 notebooks through parameters .. the notebook should ask parameter to enter ERP …
Stored Procedure Activity getting failed
Hi, I have got to use couple of external tables(parquet) in dedicated sql pool. when I run stored procedures in dedicated sql pool using those external tables, its running successfully. But when I started running stored procedure using stored procedure…
LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID: d2425fce-9179-49eb-829c-a7b2a0d963ed.
Hi Everyone, I am getting the Below Error Code while running my cells in Synapse Notebook: > LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500. Trace ID:…
How to use IS NOT NULL in synapse dataflow
I would like to use IS NOT NULL in synapse's dataflow, but I can't get it to work. Can you please tell me how to do it? I have also tried notNull, but it doesn't work... I used the dataflow filter.
Synapse spark V3.4 bug??
I am unable to create a database using Spark 3.4. Is there any existing issue with spark 3.4?
Utilizing Azure Key Vault secrets for secure password management in Apache spark connector for SQL Server
In the provided code example, our team demonstrates using a hard-coded username and password for connecting to SQL Server using the Apache Spark connector. However, for enhanced security, I want to utilize a password stored in Azure Key Vault instead of…
Configuring Alert Rules to Not Send Secondary Resolved Notification
We have set up an alert rule to notify us when a pipeline has failed in our synapse works (when pipeline failure > 0). The evaluation is based on a period of 5 minutes with a frequency of evaluation every 1 minute. The set up works well in notifying…
How to use Localhost (On-premise database) as sink in Azure Data Factory? It's possible to have source as On-premise database but why not as "Sink"?
I want to use my local database on-premise as "SINK" in Azure Data Factory. It's possible to have it as a "Source" using self-hosted-integration-runtime but why not as a "SINK"???
Azure Synapse Serverless SQL backup/recovery
We use Azure Synapse Workspaces, with Serverless SQL pools. How would one implement a disaster recovery solution to support this (across Azure regions, with geo-replicated StorageV2 (general purpose v2) datalake storage accounts.)? The automated…