Seeking Expertise in Spark SQL CTE Recursive Queries in Azure Databricks
I'm currently diving deep into Spark SQL and its capabilities, and I'm facing an interesting challenge. I'm eager to learn how to write CTE recursive queries in Spark SQL, but after thorough research, it seems that Spark doesn't natively support…
Data flow failing at Sink Step when pipeline is scheduled
I have Pipeline with Data flow debug where data is sinked into Data set. When run manually it runs fine and the parquet file is created. However when scheduled in Pipeline it gives following error. Pipeline runs fine only gives error on schedule…
Using SMI in Synapse Spark Job
Hello We have Synapse Pipeline with Spark Job Definition We currently use SPN to read data from ADLS2 and also to write to kusto with Spark Kusto Connector (Getting a token from SPN) We have saved SPN credentials into AKV. We have urgent requirement to…
How can i integrate hubspot rest api linked service in azure synapse.
When i directly integrated husbpot it's not giving me all the data. But when i try to link that with rest api my test connection is failied. I checked the hubspot api endpoint and from postman i can see all the data tables i want but it is not working in…
Synapse Integration Runtime high availability
I have a question with the Self-Hosted Integration Runtime configuration in a high availability scenario. As per this documentation…
Databricks integration to Azure Synapse Serveless SQL Pool
Hi, What is the way to connect to Databricks to Azure Synapse serverless SQL pool when I am trying to connect using serverless SQL endpoint getting an error as - Py4JJavaError: An error occurred while calling o388.load. :…
Possible bug or issue in Synapse dedicated SQL pool when exporting parquet files
I'm not sure if this is really a bug, but its definitly a frustration for me at least ;-). When trying to write data from Synapse dedicated SQL pools to data lake storage as parquet files (using a CETAS statement) it produces files with non-standard…
Missing columns in CDM entities with Dataverse D365 Synapse Link to ADLS
We have a synapse link setup to load data from Dataverse D365 into Data lake gen2. It has a table 'Contact' and it has around 780+ columns and the synapse link loads this table as Common Data Model and the dataflow reads the CDM. There were columns…
Issue with Copy Dataverse data into Azure SQL using Synapse Link Template
I am experiencing issues using ADF I am trying to use the template "Copy Dataverse data into Azure SQL using Synapse Link" to move f&o data from datalake to Azure…
How to pass the correct audience when calling mssparkutils.credentials.getToken on Azure China (Mooncake) cloud?
I'm using Microsoft Spark Utilities (MSSparkUtils) with linked service to authenticate into Azure SQL using System Assigned Managed Identity (Synapse Workspace) on Azure China (Mooncake) cloud. However, when I call gettoken with the audience type…
How to run stored procedure in Azure Database for PostgreSQL using Azure Databricks Notebook
We have Stored Procedure available in Azure Database for PostgreSQL and we want to call or run or execute the postgreSQL stored procedures in Azure Databricks through Notebook We are attempting to run PostgreSQL stored procedures, through Azure…
How to reset password for Azure synapse dedicated SQL Pool "when Reset SQL Admin password" is disabled?
Hello, When I'm creating a linked service to fetch the data from Dedicated SQL Pool within the Azure Synapse Analytics workspace, it ask for the SQL authentication. Since I hadn't created the username and password at that time, I want to create or reset…
I want to ask wheather you can connect Azure with Power BI in a live connection
If you know a way, an you please explain me how i can do that. I want to create some charts which are live connected power Bi to the working items in Azure. So if a working item changes state, or area etc, it also shows up on the chart as well
Using UAMI in Azure Synapse Notebooks
I have a bunch of Synapse notebooks currently using AppId and secret through Linked Services and successfully running so far. As part of some initiative we need to migrate these notebooks to consuming User Assigned Managed Identity (UAMI). Notebooks…
Changing Synapse Notebook variable values during deployment in Azure DevOps
How can I change the variable values in a Synapse Notebook, such as storage account name, container name, and file location, during deployment from dev to prod in Azure DevOps? The notebook is already being used by other notebooks in my dev environment.
We want to create Lake database entity tables using Synapse Industry Database Templates and map from D365 F&O Dataverse tables created using synapse link for Dataverse.
Using Azure synapse link , Table's where created from D365 data. How to create data entity views for the same
databricks to synapse
Hi Team, I am getting error while connecting databricks to synapse ... here is the code com.databricks.spark.sqldw.SqlDWConnectorException: Exception encountered in Azure Synapse Analytics connector code. def…
create store procedure in Synapse
Hi , How to create store procedure in synapse to create tables Regards
SAP CDC Source in Synapse Dataflow - Full and Incremental Load Discrepancy with Partition Condition
I am writing to report an incident related to the SAP CDC source within a Synapse Dataflow. The issue revolves around the use of full and incremental loads in delta tables, specifically when applying a filter via a partition condition. Currently, the…