Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,483 questions
4 answers
How to send a mail notification for a failed pipeline in Azure Synapse Analytics?
How can I send a notification email to a specific email address without using a logic app when one of my Synapse Analytics pipelines fails? I would like to include the error message in the email notification.
asked 2024-03-20T15:48:58.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aditya Singh
160
Reputation points
answered 2024-11-07T21:37:08.3566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 15%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
Manuel Dones
0
Reputation points
2 answers
Request Timeout on ADF Web Activity to Synapse
Hello everyone! I have a set of pipelines in Azure Data Factory (ADF) that trigger Synapse via a Web activity POST request. This setup worked until today around 11 AM CET, but now it’s throwing the following error: Error calling the endpoint. Response…
asked 2024-11-05T15:51:24.8766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 34%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEM%3C/text%3E%3C/svg%3E)
Erik Martins Tonon
5
Reputation points
answered 2024-11-07T20:31:14.5633333+00:00
AnnuKumari-MSFT
33,401
Reputation points Microsoft Employee
1 answer
Transforming JSON files using data flow
Hello! I currently have about 60 json files inside a blob container which most of them have different fields and values. I have created a pipeline with a get metadata activity that points to the container, with the field list set to Child items. I have…
asked 2024-11-07T15:23:15.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 17%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
A K
20
Reputation points
edited an answer 2024-11-07T20:06:22.09+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 6%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKD%3C/text%3E%3C/svg%3E)
Keshavulu Dasari
1,080
Reputation points Microsoft Vendor
0 answers
Parametrization for connection string is not working in linked service
Hi there, I have solution - data pipeline in Synapse taking data from mysql database. I need to parametrize it to be able run process across many environemnts (CI/CD). I set many linked services with code referencing on variable in yml file in style…
asked 2024-11-07T19:55:32.4766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 49%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Peter Michalik
0
Reputation points
asked 2024-11-07T19:55:32.4766667+00:00
Peter Michalik
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory
Hello I need to create a new resource in Azure: Azure Data Factory to perform SQL integration. I am trying to create it in V1, since it is for a Development environment and V2 is very expensive. The problem is that it is not letting me create it in V1, I…
asked 2024-11-07T15:30:10.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 16%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFG%3C/text%3E%3C/svg%3E)
Fernando Gutierrez
240
Reputation points
accepted 2024-11-07T19:54:54.6533333+00:00
Fernando Gutierrez
240
Reputation points
0 answers
How to connect to on-prem Analysis server using synapse notebook and uses System assigned managed identity as authentication.
HI Team, I have an on-prem analysis server details like server name and data base name managed by another team, they have provided access to me on the server but now is it possible to use a synapse notebook to connect to on prem analysis svr. Using…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 9%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFS%3C/text%3E%3C/svg%3E)
0 answers
How to connect to on-prem Analysis server using synapse notebook and uses System assigned managed identity as authentication.
HI Team, I have an on-prem analysis server details like server name and data base name managed by another team, they have provided access to me on the server but now is it possible to use a synapse notebook to connect to on prem analysis svr. Using…
1 answer
Copy Files from sharepoint online site to azure datalake storage
Hello We are trying to setup the flow which will copy files from sharepoint online site to azure datalake storage. As per my understanding there are 2 options : Using ADF to pull the files as mentioned in the link below…
asked 2024-11-06T22:12:34.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sourav
100
Reputation points
edited a comment 2024-11-07T19:06:11.19+00:00
Keshavulu Dasari
1,080
Reputation points Microsoft Vendor
1 answer
How to delete ADF Pipeline run history or configure to be deleted after X days ?
Hello, Is there a way to configure ADF Pipeline run history to delete after X days ( say 10 days) ? I'm not able to delete this from UI portal / ADF Studio. How can we delete pipeline run history in ADF ? Thanks, Balaji
asked 2024-11-04T23:23:57.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 43%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBD%3C/text%3E%3C/svg%3E)
Balaji Dabbara
0
Reputation points Microsoft Employee
commented 2024-11-07T18:44:00.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
16,735
Reputation points Microsoft Vendor
2 answers
DataFactory DataFlow Activity ManageVnet dataflow cluster is not started
Hi All, we are unable to start the azure data factory data flow cluster . we are using Mange Vent IR in itally north region when we start the cluster in Debug Mode we get message-The startup of the cluster is taking longer than expected , we recommend…
asked 2024-10-24T12:33:08.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 20%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
manish verma
461
Reputation points
commented 2024-11-07T18:41:44.7633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
11,200
Reputation points Microsoft Vendor
1 answer
Azure Data Factory Private Endpoint Options
I am trying to create private endpoints for ADF in dev and test environment. Since we are following hub and spoke topology, we are creating Private DNS zones in Hub which would be common for dev and test environments as well. There are two private dns…
asked 2024-11-05T05:02:12.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 25%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
M, Akhil
20
Reputation points
commented 2024-11-07T18:40:51.9933333+00:00
Smaran Thoomu
16,735
Reputation points Microsoft Vendor
1 answer
Connection time out issue in Manged vnet IR from ADF to SQL. How to resolve?
We are using Managed Vnet integration runtime to connect between Azure data factory and SQL DB. This was suggested by microsoft since our Azure SQL is private only, and Autoresolve IR wont work. We tried with SHIR earlier, but since we are using…
asked 2024-11-06T07:54:24.8666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 84%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJK%3C/text%3E%3C/svg%3E)
Jerald K Philip
0
Reputation points
commented 2024-11-07T18:40:26.5+00:00
Smaran Thoomu
16,735
Reputation points Microsoft Vendor
1 answer
Dynamic Array Mapping in ADF Copy Activity
I'm attempting to set up dynamic mapping between blob storage in an ADF (Azure Data Factory) Copy activity. In the mapping section, I'm trying to add an additional column using dynamic content from a set variable that is an array. However, I encounter…
asked 2024-11-07T18:39:41.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 59%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Akhil
0
Reputation points
answered 2024-11-07T18:39:41.95+00:00
Q&A Assist
1 answer
How to check text in row 1 of Excel file for about 10 column headers?
I'm using Azure Data Factory (ADF) on the portal. I use the latest Chrome browser to get to ADF. I'm not an admin and have no admin rights. I have an Excel file I want to read. I'm still fairly new to ADF. I'm guessing this could be done in a data flow.…
asked 2024-11-05T12:36:13.51+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 77%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Chuck Roberts
110
Reputation points
edited a comment 2024-11-07T18:35:35.3866667+00:00
phemanth
11,200
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Moving data from Snowflake to Azure SQL DB using Copy Activity
Hi All, I am using COPY activity to move data from Snowflake to Azure SQL DB. First of all, this data movement is not allowed without intermediate staging of the data in ADLS. I would love to avoid this intermediate step. If this is how it will work then…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 27%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
2 answers
Datetime2 error when querying parquet file produced from Synapse copy activity from OData Business Central Source (data seems to be changed from 0001-01-01 to 0000-12-30)
I have a pipeline running a copy activity to copy data from Business Central OData source to parquet file everyday. Everything works fine before 16/07/2024 but problem arises starting from 17/07/2024. The parquet file no longer be able to query from…
asked 2024-07-24T15:13:17.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 10%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVC%3C/text%3E%3C/svg%3E)
Victor Chu
15
Reputation points
commented 2024-11-07T17:54:25.05+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 10%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENN%3C/text%3E%3C/svg%3E)
Nneoma Nwamaghinna
5
Reputation points
1 answer
ADF Copy Activity Sink with Metadata field not handling accented characters
Hi All I have a ADF Copy Activity where I have defined several metadata fields in the Sink tab for my output file. The output filename created by the Copy activity has accented characters in its name for example: Jonsins Blobs statión Full…
asked 2024-01-23T13:21:58.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 14.000000000000002%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDW%3C/text%3E%3C/svg%3E)
D Williams
76
Reputation points
commented 2024-11-07T15:23:40.0866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 68%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMY%3C/text%3E%3C/svg%3E)
Manas Yadav
0
Reputation points
1 answer
How to resolve invalid property name error in ADF copy activity?
ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'BadRequest'. Account: 'rmsdatalakestracc1mqa'. FileSystem:…
asked 2024-05-19T06:23:54.9633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 35%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMY%3C/text%3E%3C/svg%3E)
Mansi Yadav
40
Reputation points
commented 2024-11-07T15:18:22.2366667+00:00
Manas Yadav
0
Reputation points
1 answer
Occassional Issue with passing parameters between activities
We are having a spotty issue where once in every few pipeline runs an activity will not fill in the parameters even though we run the same code 50 times and 49 of those times works. There doesn't seem to be any difference that we can find. Has there been…
asked 2024-10-31T19:01:41.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 20%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMF%3C/text%3E%3C/svg%3E)
Mike Fatal
0
Reputation points
commented 2024-11-07T14:36:04.1866667+00:00
Mike Fatal
0
Reputation points
0 answers
ADF - Timeout when all parts have access to resources.
Hi Community! I’m running into a challenging issue with my Azure Data Factory setup. Here’s what my current infrastructure looks like: Azure Data Factory (ADF) CosmosDB Key Vault Storage Account Private Networks Everything was set up months ago…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 99%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
