Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,502 questions
0 answers
Failed to convert the value in 'xxxx' property to 'System.String' type in Data Lake Connector
Environment: Azure Synapse I am developing a pipeline that has a lookup to SQL that is passing the values to GetMetaData. When using @dataset.name of parameter in the dataset, it fails with "Failed to convert the value in name of parameter…
asked 2024-05-31T19:42:29.6733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 18%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDC%3C/text%3E%3C/svg%3E)
Doreen Cecil
0
Reputation points
asked 2024-05-31T19:42:29.6733333+00:00
Doreen Cecil
0
Reputation points
1 answer
How to ignore the records by applying an auditing filed column condition using ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
asked 2024-05-30T09:28:46.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
45
Reputation points
commented 2024-05-31T19:27:52.55+00:00
Amira Bedhiafi
16,951
Reputation points
1 answer
From Synapse notebook, trouble connecting to Azure SQL using system managed identity
I can create a Linked Service to my Database using my Synapse system-managed identity. This works fine, and the "test connection" shows success. But for the life of me, I can't figure out how to do it in a Python notebook. Here is the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 80%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
2 answers
Assert error output not writing to blob
Hello. I have built a pipeline which gathers data from a .csv file and then goes through a couple of assert activities to check the validity of the data. Valid rows are supposed to be input into the table, while assert-failure rows are setup to be…
asked 2024-05-28T10:17:13.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 92%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDL%3C/text%3E%3C/svg%3E)
Dejvis Leka
0
Reputation points
commented 2024-05-31T16:18:50.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
1 answer
CI/CD Powershell Azure Synapse Workspace with Azure Devops
I am working on CI/CD between Azure Synapse Workspace with Azure Devops. I have source code from someone else who work on this project before and it works well and I try to understand it and create my own project on it. right now, I have issues of the…
asked 2024-05-28T10:04:13.9833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 75%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZM%3C/text%3E%3C/svg%3E)
Zhuoqiong Mo
20
Reputation points
commented 2024-05-31T15:52:34.93+00:00
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
0 answers
Error publishing notebook.
I am trying to publish my notebook. I get the following error : CreateOrUpdateNotebook failed: BadRequest. The operation failed because the entity “notebookA” is being renamed to “notebookB”. A retry of rename from exact source to target may work”. It…
asked 2024-05-30T17:31:34.6033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 14.000000000000002%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECT%3C/text%3E%3C/svg%3E)
Candy T
0
Reputation points
edited a comment 2024-05-31T15:47:45.9433333+00:00
BhargavaGunnam-MSFT
27,976
Reputation points Microsoft Employee
0 answers
source files problem from internal server to Gen2 container
Hello, I am trying to connect my interal files system to the gen2 container sink, following this instructions https://arulmouzhi.wordpress.com/2021/04/12/bringing-folder-structure-via-azure-data-factory/ The first one in Allfiles set, I can load…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 2%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENW%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How can I find help validating a certificate?
Indeed, I am a student in cloud computing in Morocco and I do not have enough means to obtain the AZ 900 certification and my family also does not have enough financial means. Could someone help me validate this certificate please?
asked 2024-05-30T15:54:26.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 95%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHY%3C/text%3E%3C/svg%3E)
HAMIDOU YOUSSOUF
20
Reputation points
accepted 2024-05-31T14:55:42.77+00:00
HAMIDOU YOUSSOUF
20
Reputation points
1 answer
azure synapse RBAC not working
I have just created a synapse analytics workspace, on opening it I get an error message similar to the below. From my research, I have gr anted storage blob contributor role to the synapse application as well as the user assigned managed identity. Going…
asked 2024-05-30T22:05:59.0533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 39%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
MrFlinstone
501
Reputation points
commented 2024-05-31T13:17:54.9233333+00:00
MrFlinstone
501
Reputation points
1 answer
Should Serverless Spark pool be able to process less than 5 million rows quickly?
We are downloading about 10K records daily and the trying to merge them into a Deltalake file and noticing that it takes about 35-45 minutes to merge. Is that expected?
asked 2024-05-30T13:03:07.7533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 55.00000000000001%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJF%3C/text%3E%3C/svg%3E)
John Fisher
0
Reputation points
commented 2024-05-31T12:50:51.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,885
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Resolving FileNotFoundError When Reading Parquet Files in Synapse Notebook
In my Synapse Notebook, I aimed to read Parquet files. However, I encountered a 'FileNotFoundError' when attempting to use a wildcard. The folder structure I intend to access is as follows: 'test/year={yyyy}/month={MM}/day={dd}/*.parquet'. Here's the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 64%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECJ%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Proper Usage of Synapse Notebook References
I'm attempting to utilize the Azure Synapse notebook reference outlined in the documentation provided here: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-development-using-notebooks?tabs=preview#notebook-reference In my…
0 answers
We are not getting data in azure data lake Service v2 storage's container.
Hello, We are getting data in the Azure event hub sink through the POSTGRES SQL database but we are not getting files in the container which we have configured in an ADLS GEN 2 and while configuring we have an encounter that we are using a free…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 90%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
1 answer
Running specific pipelines from an Orchestration pipeline
How can I run individual pipelines from an Orchestration pipeline that has multiple pipelines running in parallel? I want to be able to choose which pipeline to run separately from the rest.
asked 2024-05-30T21:38:39.0933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 41%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
ADITYA SINGH
0
Reputation points
commented 2024-05-31T09:41:01.8633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sina Salam
4,546
Reputation points
4 answers
Azure Synapse Analytics Failed to setup debug session
Right now I'm not able to run a data flow task on our Azure Synapse environment. When I try to start a debug session I get the message Failed to setup debug session. When I trigger the pipeline instead the task fails with "Operation on target…
asked 2022-09-20T12:40:49.163+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
answered 2024-05-31T09:04:57.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 74%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
srivally chintakindi
0
Reputation points
0 answers
ADF Data Flows Flatten nested json array values are being populated as null
Hi All I am building a data transamination with ADF data flow using a nested json array of objects , but after parse and flatten the json node itOffer.item.LeadOfer.zdeal.item[].dealNumber I am seeing that the column values are populated as null . I…
asked 2024-05-30T11:06:56.13+00:00
venkat rao
45
Reputation points
edited a comment 2024-05-31T08:49:38.1266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
4,485
Reputation points Microsoft Vendor
1 answer
Access cosmos db through azure synapse analytics notebook using system assigned managed identity linked service
i have made a linked service for cosmos db no sql using system assigned managed identity as auth type and linked service is published as well. Now when i access this linked service from synapse analytics notebook using below code it gives me this…
asked 2023-09-13T09:50:20.6066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 77%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Syed Ali Naqi
0
Reputation points
commented 2024-05-31T08:01:07.61+00:00
Amira Bedhiafi
16,951
Reputation points
2 answers
Seeking Expertise in Spark SQL CTE Recursive Queries in Azure Databricks
I'm currently diving deep into Spark SQL and its capabilities, and I'm facing an interesting challenge. I'm eager to learn how to write CTE recursive queries in Spark SQL, but after thorough research, it seems that Spark doesn't natively support…
asked 2024-05-30T04:27:35.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 57.99999999999999%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anuj, Singh (Cognizant)
50
Reputation points
answered 2024-05-31T07:52:36.89+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
10,885
Reputation points Microsoft Vendor
1 answer
Data flow failing at Sink Step when pipeline is scheduled
I have Pipeline with Data flow debug where data is sinked into Data set. When run manually it runs fine and the parquet file is created. However when scheduled in Pipeline it gives following error. Pipeline runs fine only gives error on schedule…
asked 2024-05-30T07:21:04.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 32%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENZ%3C/text%3E%3C/svg%3E)
Nazish Zahra
0
Reputation points
commented 2024-05-31T07:47:28.8633333+00:00
Amira Bedhiafi
16,951
Reputation points
0 answers
Using SMI in Synapse Spark Job
Hello We have Synapse Pipeline with Spark Job Definition We currently use SPN to read data from ADLS2 and also to write to kusto with Spark Kusto Connector (Getting a token from SPN) We have saved SPN credentials into AKV. We have urgent requirement to…
asked 2024-05-29T14:30:25.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 2%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Jean-Christian Kouamé
0
Reputation points Microsoft Employee
commented 2024-05-31T03:53:18.5466667+00:00
phemanth
6,885
Reputation points Microsoft Vendor