Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,376 questions
1 answer
Changing Synapse Notebook variable values during deployment in Azure DevOps
How can I change the variable values in a Synapse Notebook, such as storage account name, container name, and file location, during deployment from dev to prod in Azure DevOps? The notebook is already being used by other notebooks in my dev environment.
asked 2024-05-29T04:11:27.8566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 41%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
ADITYA SINGH
0
Reputation points
edited an answer 2024-05-29T09:10:54.5633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
4,325
Reputation points Microsoft Vendor
1 answer
Using UAMI in Azure Synapse Notebooks
I have a bunch of Synapse notebooks currently using AppId and secret through Linked Services and successfully running so far. As part of some initiative we need to migrate these notebooks to consuming User Assigned Managed Identity (UAMI). Notebooks…
asked 2024-05-21T01:42:12.56+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 74%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Siva Gangadhar Galla
0
Reputation points Microsoft Employee
commented 2024-05-29T08:46:21.42+00:00
Harishga
4,325
Reputation points Microsoft Vendor
1 answer
I want to ask wheather you can connect Azure with Power BI in a live connection
If you know a way, an you please explain me how i can do that. I want to create some charts which are live connected power Bi to the working items in Azure. So if a working item changes state, or area etc, it also shows up on the chart as well
asked 2024-05-24T14:43:43.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 37%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Marvin Bylyku
0
Reputation points
commented 2024-05-29T08:10:31.24+00:00
PRADEEPCHEEKATLA-MSFT
80,096
Reputation points Microsoft Employee
0 answers
Is there a way to restrict multiple instances of an ADF pipeline on Same path of event based trigger?
I'm running Spark notebooks in Synapse using an event-based ADF pipeline. If any notebook gets triggered twice, I encounter a conflict error. I understand that I can avoid concurrent execution of the pipeline by setting the concurrency to 1. However, I…
asked 2024-05-28T11:16:32.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 74%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Divya Sharma
1
Reputation point
commented 2024-05-29T08:09:59.56+00:00
Harishga
4,325
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
We want to create Lake database entity tables using Synapse Industry Database Templates and map from D365 F&O Dataverse tables created using synapse link for Dataverse.
Using Azure synapse link , Table's where created from D365 data. How to create data entity views for the same
asked 2024-05-24T06:41:34.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 25%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Suraj Surendran
60
Reputation points
accepted 2024-05-29T08:08:04.9366667+00:00
Suraj Surendran
60
Reputation points
0 answers
databricks to synapse
Hi Team, I am getting error while connecting databricks to synapse ... here is the code com.databricks.spark.sqldw.SqlDWConnectorException: Exception encountered in Azure Synapse Analytics connector code. def…
asked 2024-05-21T09:20:54.4866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
205
Reputation points
commented 2024-05-29T08:05:17.6+00:00
PRADEEPCHEEKATLA-MSFT
80,096
Reputation points Microsoft Employee
1 answer
create store procedure in Synapse
Hi , How to create store procedure in synapse to create tables Regards
asked 2024-05-20T05:47:38.0366667+00:00
Vineet S
205
Reputation points
commented 2024-05-29T08:03:07.3766667+00:00
PRADEEPCHEEKATLA-MSFT
80,096
Reputation points Microsoft Employee
1 answer
SAP CDC Source in Synapse Dataflow - Full and Incremental Load Discrepancy with Partition Condition
I am writing to report an incident related to the SAP CDC source within a Synapse Dataflow. The issue revolves around the use of full and incremental loads in delta tables, specifically when applying a filter via a partition condition. Currently, the…
asked 2024-01-24T15:00:13.5966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 21%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Dev-nimbeo
6
Reputation points
commented 2024-05-29T02:44:28.8366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 2%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPG%3C/text%3E%3C/svg%3E)
Phipps, Garry (IST)
0
Reputation points
2 answers
One of the answers was accepted by the question author.
calling another notebook with mssparkutils.notebook.run - Kernel doesn't exist
i am not able to execute a notebook with mssparkutils.notebook.run. However the magic command %run 'notebook' works. i am getting an error like this: com.microsoft.spark.notebook.msutils.NotebookExecutionException: Fetch notebook content for…
asked 2023-03-15T07:46:42.0666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 69%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGF%3C/text%3E%3C/svg%3E)
Garon Fabienne
20
Reputation points
answered 2024-05-28T16:29:06.04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 27%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
NPizzuti
51
Reputation points
1 answer
Synapse Integration Runtime high availability
I have a question with the Self-Hosted Integration Runtime configuration in a high availability scenario. As per this documentation…
asked 2024-05-28T14:55:50.6366667+00:00
cthivierge
4,056
Reputation points
commented 2024-05-28T15:51:15.25+00:00
cthivierge
4,056
Reputation points
7 answers
Error trying to validate storage account name while creating a new synapse workspace
I am unable to create a new data lake storage (gen2) account name in the basic tab for create Azure Synapse Workspace. The error I get is - " There was an error trying to validate storage account name. please try again ". The error message is…
asked 2022-08-03T07:01:54.587+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 2%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jozure
476
Reputation points
answered 2024-05-28T14:23:07.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 95%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Brenton Burkhalter
0
Reputation points
1 answer
CI/CD Powershell Azure Synapse Workspace with Azure Devops
I am working on CI/CD between Azure Synapse Workspace with Azure Devops. I have source code from someone else who work on this project before and it works well and I try to understand it and create my own project on it. right now, I have issues of the…
asked 2024-05-28T10:04:13.9833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 75%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZM%3C/text%3E%3C/svg%3E)
Zhuoqiong Mo
20
Reputation points
commented 2024-05-28T12:35:42.77+00:00
Zhuoqiong Mo
20
Reputation points
0 answers
How to pass the correct audience when calling mssparkutils.credentials.getToken on Azure China (Mooncake) cloud?
I'm using Microsoft Spark Utilities (MSSparkUtils) with linked service to authenticate into Azure SQL using System Assigned Managed Identity (Synapse Workspace) on Azure China (Mooncake) cloud. However, when I call gettoken with the audience type…
asked 2024-05-21T04:20:59.6566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 75%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Abhishek Bhatt
0
Reputation points Microsoft Employee
commented 2024-05-28T11:37:46.48+00:00
PRADEEPCHEEKATLA-MSFT
80,096
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
pyspark dataframe methods: count, show ... duration?
Hi, I am testing some pyspark methods over a dataframe that I have created from a table, from the dedicated pool and it about 32 million rows length When running for example: "df1.show()" or "df1.count()" both commands last for about…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 18%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJZ%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
0 answers
Assert error output not writing to blob
Hello. I have built a pipeline which gathers data from a .csv file and then goes through a couple of assert activities to check the validity of the data. Valid rows are supposed to be input into the table, while assert-failure rows are setup to be…
asked 2024-05-28T10:17:13.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 92%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDL%3C/text%3E%3C/svg%3E)
Dejvis Leka
0
Reputation points
asked 2024-05-28T10:17:13.85+00:00
Dejvis Leka
0
Reputation points
1 answer
One of the answers was accepted by the question author.
CI/CD Azure Synapse Workspace with Azure Devops
Dear all, I am working on CI/CD between Azure Synapse Workspace with Azure Devops. I have source code from someone else who work on this project before and it works well and I try to understand it and create my own project on it. right now, I have…
asked 2024-05-28T07:59:15.3033333+00:00
Zhuoqiong Mo
20
Reputation points
commented 2024-05-28T09:47:39.4233333+00:00
Zhuoqiong Mo
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Should all active services in my account be visible from Home-->List All?
I've used Azure portal a few time for AI-201 training. I've just started DP-203 training and was attempting to run a beginners MS Learn script that requested Azure Synapse Analytics and related services for exploration purposes. After about 30 minutes,…
asked 2024-05-24T13:37:56.0266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 51%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMI%3C/text%3E%3C/svg%3E)
McKay, Ian
20
Reputation points
accepted 2024-05-28T09:45:46.1033333+00:00
McKay, Ian
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Aritechture best practices to store data from websraping and use it in analytics
I am scrapping data from websites and I want to export what I scrapped as csv files then store them in Azure Data Lake, then apply an ETL and the final data output will be used for Power BI reports and for machine learning do I need to use Azure synapse?…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 64%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
2 answers
The synapse link is not able to write the data in Azure data lake gen 2 storage account
Hi Team, The synapse link is not able to write the data in Azure data lake gen 2 storage account. I am getting the following error message. {"code":"AuthorizationFailed","message":"The client…
asked 2024-05-24T08:27:24.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 60%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Abhishek Sajjan
0
Reputation points
edited an answer 2024-05-28T08:35:39.2433333+00:00
AnnuKumari-MSFT
31,726
Reputation points Microsoft Employee
3 answers
One of the answers was accepted by the question author.
OverrideArmParameters with Synapse workspace deployment@2
I'm trying to deploy a synapse workspace to a Prod environment, overriding parameters (such as linkedServiceUrls etc.). I have got it working using a template-parameters-definition.json file to make the parameters I want available for overriding, and the…
asked 2023-10-09T11:04:09.7666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 90%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMO%3C/text%3E%3C/svg%3E)
Mathias Opland
135
Reputation points
answered 2024-05-28T08:23:08.8033333+00:00
Mathias Opland
135
Reputation points