Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,667 questions
0 answers
My python code uses AzureCliCredential() function, but it is giving error in running in Synapse/ADF/Databricks notebook .
My python code uses AzureCliCredential() function, but it is giving error in running in Synapse Workspace that your Azure Cli command not found on path . I have tried using other function like DefaultAzureCredentials(), ClientSecretCredentials () also…
asked 2024-07-18T18:55:12.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 80%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUM%3C/text%3E%3C/svg%3E)
Utsav Mori
20
Reputation points
asked 2024-07-18T18:55:12.8266667+00:00
Utsav Mori
20
Reputation points
0 answers
Workflow that logs the completion of certain pipelines into a table
I'm having a lot of difficulty implementing the solutions I had in mind. Previously, I asked for help to create an architecture that would be efficient, easy to maintain, and cost-effective. I received various suggestions, but I can't decide which one to…
asked 2024-07-18T18:08:14.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 31%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYD%3C/text%3E%3C/svg%3E)
Yohanna de Oliveira Cavalcanti
160
Reputation points
asked 2024-07-18T18:08:14.79+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
2 answers
One of the answers was accepted by the question author.
Data Factory Logs --> Catolog Databricks
Good morning, I need assistance in creating a project that captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks. The key requirements for this project are as follows: No Duplicate Logs: Ensuring that the logs are not…
asked 2024-07-17T14:53:09.4466667+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
edited a comment 2024-07-18T15:56:37.4933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
8,485
Reputation points Microsoft Vendor
2 answers
SAS token generation by Databricks to access CSV files from ADLS container folder
Hi Team, There are some csv files zips inside the ADLS container folder. These zip files need to be downloaded for data correction. Downloading the file requires SAS token embedded with zip file path. Databricks has been used to generate the token and…
asked 2024-07-17T05:08:24.3033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 91%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Subhadip Roy
21
Reputation points
commented 2024-07-18T15:47:55.41+00:00
Subhadip Roy
21
Reputation points
2 answers
Getting the size of parquet files from azure blob storage
I have a blob container abcd The folder structure is like below: abcd/Folder1/Folder a, Folder b…..Folder z Inside a particular Folder a/v1/full/20230505/part12344.parquet Similarly Folder b/v1/full/20230505/part9385795.parquet Scenario is I need to get…
asked 2024-07-09T11:57:50.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 89%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
KEERTHANA JAYADEVAN
66
Reputation points
commented 2024-07-18T12:13:50.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 81%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Amrinder Singh
4,270
Reputation points Microsoft Employee
1 answer
How to add parent group for one specific group in databricks?
The question for this is in "databricks" -> "settings" -> "Identity and access" -> "Groups" Here has "admin" group with system managed. But we wonder if a new group can be assigned as a parent…
asked 2024-07-18T09:02:33.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 66%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFT%3C/text%3E%3C/svg%3E)
FIXED-TERM GUAN Lanita (IOB/PAU-AP)
0
Reputation points
answered 2024-07-18T09:12:19.5066667+00:00
Deepanshukatara-6769
7,830
Reputation points
1 answer
One of the answers was accepted by the question author.
Pipeline Executing Databricks Notebook Successfully Despite Stopped Cluster
In Azure Data Factory (ADF), I have a pipeline that executes a notebook in Azure Databricks. I noticed that even when the Databricks cluster is stopped, the ADF pipeline still completes successfully, and the notebook runs without any issues. Is this…
asked 2024-06-20T11:17:32.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 15%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV0%3C/text%3E%3C/svg%3E)
vikranth-0706
205
Reputation points
commented 2024-07-18T07:19:40.9566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,610
Reputation points Microsoft Vendor
1 answer
I am unable to create compute cluster in databricks from azure free trial subcription
I am unable to create compute cluster in databricks from azure free trial subcription. Failed to add 1 container to the compute. Will attempt retry: false. Reason: Azure Quota Exceeded Exception
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 2%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks
Good morning, I need assistance in creating a project that captures logs from Azure Data Factory and inserts them into a Delta Table in Databricks. The key requirements for this project are as follows: No Duplicate Logs: Ensuring that the logs are not…
asked 2024-07-17T15:17:27.44+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
commented 2024-07-18T05:42:00.2866667+00:00
Smaran Thoomu
12,610
Reputation points Microsoft Vendor
0 answers
Datafactory logs for dasboard on databricks
hello! need help! I need to solve a problem that consists of taking completed pipeline logs with specific names and inserting them into a deltatable within Databricks to create dashboards in it. this solution needs to be entirely using Azure tools. I'm…
asked 2024-07-18T01:53:11.08+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
asked 2024-07-18T01:53:11.08+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
3 answers
Azure Data Bricks - User Doesn't have permission to perform this action while connecting to Azure Synapse Dedicate Pool
We are connecting Azure Synapse Analytics - Dedicated Pool using the PySpark Code that runs from Azure Data Bricks using SQL Authentication. While running, we are getting the below error when we use a user with db_datawriter and db_datareader…
asked 2024-06-07T11:26:00.4933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 97%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Praveen Sreeram
1
Reputation point
answered 2024-07-18T00:39:31.7333333+00:00
Vinodh247
13,226
Reputation points
1 answer
Efficient Log Handling and Data Retention in Azure Data Factory and Databricks
I need to create a solution to send logs from Azure Data Factory to the Databricks Unity Catalog. I'm considering the following structure: Whenever an activity run results in either failure or success, the corresponding log will be sent to Azure Logic…
asked 2024-07-17T19:44:42.0666667+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
commented 2024-07-17T22:12:15.19+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
2 answers
How to create Synapse Serverless SQL Pool External Table using Databricks Notebook?
Hello, Can we create Synapse Serverless SQL Pool External Table using a Databricks Notebook? E.g., the script to create an Synapse Serverless SQL Pool External Table from within Synapse is as follows: CREATE EXTERNAL TABLE [SchemaName].[TableName]…
asked 2023-07-24T20:04:56.9766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 81%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAE%3C/text%3E%3C/svg%3E)
Azure Enthusiast
10
Reputation points
edited a comment 2024-07-17T20:21:32.17+00:00
Kafran
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Data Factory Logs to Databricks
I need to create a way to send logs from Data Factory to the Databricks Catalog. What is the most cost-effective and efficient method to achieve this?
asked 2024-07-16T19:26:32.8633333+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
commented 2024-07-17T18:39:54.08+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
1 answer
One of the answers was accepted by the question author.
How to Correctly Pass and Use Boolean Values from Datafactory to Databricks Notebook
How can I correctly pass a Boolean value from Datafactory to a Databricks notebook and use it in conditional logic? I configured a pipeline in Datafactory that calls a Databricks notebook. I attempted to pass a Boolean parameter from Datafactory as a…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 32%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
5 answers
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
asked 2024-07-04T10:41:44.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
30
Reputation points
commented 2024-07-17T14:10:25.3766667+00:00
Smaran Thoomu
12,610
Reputation points Microsoft Vendor
3 answers
One of the answers was accepted by the question author.
Validate Task - Data Factory npm Build Fails bundle.manager.js:53 exit code 255
I have two devops accounts. In 1st account I configured a build pipeline for Azure Data Factory with the attached build.yaml132228-buildyaml.txt I then copied the repo and the build yaml to the 2nd devops account. In the 1st devops the build…
asked 2021-09-15T07:22:51.877+00:00
Rob Bowman
221
Reputation points
answered 2024-07-17T11:56:03.2833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 77%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ6%3C/text%3E%3C/svg%3E)
JyotiranjanMangaraj-6157
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Best Practices for Automating Pipeline Execution Data Collection in Azure Data Factory
Hello everyone, I am looking for the best practices to create an automated workflow for collecting execution data from Azure Data Factory (ADF) pipelines, storing this data in Azure Data Lake Storage Gen2, and consolidating it into a single table for…
asked 2024-07-16T14:23:15.0633333+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
accepted 2024-07-17T11:38:09.2833333+00:00
Yohanna de Oliveira Cavalcanti
160
Reputation points
1 answer
How does ADF Linked Service for Azure Databricks aligns with Job Compute Policy defined inside Azure Databricks
This is a two-part Question. First Part Context: I have ADF which contains several data pipelines. Some pipeline also includes a databricks notebook as an activity. I have created Linked Service Azure Data Factory to facilitate the pipeline which…
asked 2024-07-10T10:25:16.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 26%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Ravineesh
20
Reputation points
commented 2024-07-16T23:14:23.99+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
28,951
Reputation points Microsoft Employee
1 answer
Append in Liquid Cluster enabled table is not completing on DBR 15.3 version
I am trying do analysis with a Partition Table and Liquid Clustered table. As per Azure Databricks recommendation, I am using DBR 15.2 to execute the code. I have created a clustered table as and using an append operation which is specified below. Few…
asked 2024-07-13T07:29:40.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 35%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Sudipta Goswami
20
Reputation points
commented 2024-07-16T17:09:56.4566667+00:00
Sudipta Goswami
20
Reputation points