Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,534 questions
3 answers
One of the answers was accepted by the question author.
Copy data from Teradata to snowflake
I am running copy activity for copying data from teradata to snowflake. I am getting error : Operation on target Copy data1 failed: ErrorCode=UserErrorOdbcOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ERROR…
asked 2021-08-18T14:50:08.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 48%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Aditya Raj
286
Reputation points
answered 2022-01-15T03:01:48.003+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 81%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMN%3C/text%3E%3C/svg%3E)
Macolino, Nick (Health)
1
Reputation point
0 answers
How to create Custom Event in Synapse Pipeline for SFTP file watch?
I have created a Synapse pipeline that loads data in a synapse dedicated SQL pool from CSV files. The CSV files are placed into SFTP location by the source system and I need to create a file watcher event to trigger the pipeline only when the files are…
asked 2022-01-14T13:07:39.37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 38%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Abhilash Sadhale
21
Reputation points
commented 2022-01-14T21:47:10.783+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,676
Reputation points Microsoft Employee
1 answer
How to prevent data re pull from the beginning if a link broken or timeout occurred
Hi team, I am importing a large dataset (500G +) from a external resource and loading into Azure sql db using ADF copy data function. Sometimes, a link broken or time out can occur. My question is: if a link broken/timeout occurs, how to keep the…
asked 2022-01-07T03:18:13.583+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 54%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZG%3C/text%3E%3C/svg%3E)
Zhang, Gongliang
1
Reputation point
commented 2022-01-14T20:34:59.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,026
Reputation points
0 answers
Delete data between date range in Data Flow
0 I have a data Flow that reads from Parquet files, does some filtering and then loads into a Delta Lake. The data flow would run multiple times and I don't want duplicate data in my Delta Lake. To safeguard this, I thought to implement a delete-insert…
asked 2022-01-04T02:33:48.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 61%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYT%3C/text%3E%3C/svg%3E)
Yash Tamakuwala
1
Reputation point
commented 2022-01-14T09:24:23.777+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 63%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
svijay-MSFT
5,201
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Executing pipeline in AML from ADF suddenly stopped working
I have a pipeline defined in Azure Machine Learning. It was launched every day with Azure Data Factory with Machine Learning Execute Pipeline activity. This solution worked without any issues for a few weeks, but since 12/09/2021 all pipeline runs have…
asked 2021-12-13T12:34:29.747+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 64%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Dorota Szepietowska
26
Reputation points
answered 2022-01-14T07:24:35.053+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 61%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJL%3C/text%3E%3C/svg%3E)
Jessie Liu
1
Reputation point Microsoft Employee
1 answer
One of the answers was accepted by the question author.
auto trigger azure data share using rest api
I would like to create auto trigger using rest api which will trigger the azure data synchronization. I am trying to invoke the rest api call using ADF web activiy but unable to do it. Please help URL :…
asked 2022-01-11T09:59:03.89+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 17%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Arundhati Sen
151
Reputation points
accepted 2022-01-14T05:39:23.14+00:00
Arundhati Sen
151
Reputation points
1 answer
One of the answers was accepted by the question author.
How to delete " from JSON file (at the beginning after [ and at the end before ] )
I am trying to get rid of two double quotation marks from this JSON file. Bottom is image: This is text. I am sharing my JSON file here. [" { "GLDETAIL": { "RECORDNO": "249125-1688259-784668--accrual",…
asked 2022-01-13T01:00:47.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2022-01-13T18:41:48.67+00:00
Anonymous
1 answer
One of the answers was accepted by the question author.
How to put a variable in the "Request body" field of a POST method?
I m trying to use a variable in the "Request body" field, but I cant get it to work. How do I get rid of the additional slashes? This is my expression: @replace(variables('requestBody'),'\','') The variable looks like this: { …
asked 2021-12-23T12:10:41.917+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 44%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Dirk Sachse
106
Reputation points
accepted 2022-01-13T17:19:47.63+00:00
Dirk Sachse
106
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory REST API Pagination Rules
Hi there, I've built a simple pipeline with 2 activities: One "Web" and one "Copy Data" The web activity generates a token to access a REST API and Copy Data uses the token to copy the result to a SQL database. I can preview the…
asked 2022-01-08T00:26:36.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 2%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rafael Silveira
21
Reputation points
accepted 2022-01-13T16:52:09.993+00:00
Rafael Silveira
21
Reputation points
2 answers
derived column pattern
Hello, I have been using a derived column pattern for a while now but today I get an error in the pattern expression as follows: 'Config#lookup' at parameter '1' expects 'string' type of argument This is the expression I am using: …
asked 2022-01-04T15:24:03.487+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
arkiboys
9,621
Reputation points
commented 2022-01-13T16:38:04.207+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
Unable to get PartialArmTemplate in ADF?
PartialArmTemplate is not generated while publishing the ADF pipelines. Only global parameters and linked service folders are coming. What could be the possible reasons for this?
asked 2022-01-03T05:58:29.093+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 93%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPP%3C/text%3E%3C/svg%3E)
Prakhar Parimal
26
Reputation points
commented 2022-01-13T16:28:50.877+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
3 answers
HTTP Response Header details not available
Hello there, I have a requirement to fetch the billions of records of data from the salesforce objects on daily basis, I am using the BULK API 2.0 for this task to get the data quickly as the salesforce direct connector is taking more than 15 hours to…
asked 2022-01-03T07:47:57.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 94%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Muralikrishna Pirla
216
Reputation points
answered 2022-01-13T10:29:49.82+00:00
Muralikrishna Pirla
216
Reputation points
1 answer
Unable to delete resources in my resource group
I have create several resources in a resource group in my MSDN subscription to testing purposes, which I now want to delete. However I get an error when trying to delete Private Endpoints for a Data Factory resource, and I am now stuck with a Data…
asked 2022-01-07T10:13:24.08+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 69%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAH%3C/text%3E%3C/svg%3E)
Allan Hjorth Vallentin
1
Reputation point
commented 2022-01-13T05:36:17.287+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure: Create resource with PowerShell using parameterized JSON definition file
I want to do something like what's shown below, but it's not working. How does one parameterize values inside a JSON configuration file and then pass in the parameter values on the PowerShell command line? Is it possible?? New-AzDataFactoryV2Pipeline…
asked 2021-09-03T14:19:40.723+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 47%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
chrisw
81
Reputation points
commented 2022-01-13T03:32:02.623+00:00
PRADEEPCHEEKATLA-MSFT
77,081
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
How to get rid of backslash "\" and "\r\n" from a JSON file?
I am trying to move data from a JSON file to SQL table, and apparently, there are unnecessary characters ( \ \r\n) that I have to delete in order to map to SQL table using Azure Data Factory. This is an error message: I am not sure…
asked 2022-01-12T01:12:32.35+00:00
Anonymous
accepted 2022-01-13T00:43:29.023+00:00
Anonymous
1 answer
One of the answers was accepted by the question author.
Error Generating ARM Script
Getting below error when trying to publish the changes in ADF Generating ARM templates {"error":{"code":"GatewayTimeout","message":"The gateway did not receive a response from 'Microsoft.DataFactory'…
asked 2022-01-05T02:09:04.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 33%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVB%3C/text%3E%3C/svg%3E)
Venkata Barlapudi
21
Reputation points
commented 2022-01-12T20:27:34.047+00:00
Venkata Barlapudi
21
Reputation points
1 answer
Is it possible to delete Files in blob based on a size
I have around 10k files stored in a container. End goal is to merge them into a single file. But there are a lot of file with no data and just headers resulting in size being 50bytes. Is it possible in ADF (using delete activity) to delete all files…
asked 2022-01-05T05:07:39.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 42%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
sam nick
286
Reputation points
commented 2022-01-12T16:40:16.6+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
How to pass parameters from DevOps to Azure Data Factory Pipeline
Hello Everyone, We have the ADF pipeline where people key in the following parameters values in the Global Parameter section and trigger the pipeline and It's working fine without any issues. But we would like to use DevOps to pass those parameters…
asked 2022-01-01T13:55:11.807+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 98%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
LoganIN
61
Reputation points
commented 2022-01-12T11:58:12.387+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Track data quality issues in Azure Data Factory
Hello! I'm ingesting two "Account" tables from 2 different countries from 2 different Salesforce instances. I'm joining them together on an ID key. A number of columns are supposed to match their records because Salesforce is supposed to…
asked 2021-12-20T19:55:48.683+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 77%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Degenocrat
27
Reputation points
commented 2022-01-12T11:21:06.957+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
ADF - Adding element to JSON object
Hi Guys, Is there a simple way to append to a JSON object within Azure Data Factory? We retrieve the JSON object from Cosmos. The type is shown as 'complex' We then create calculated columns. We then want to add those columns within the JSON…
asked 2021-12-14T10:29:11.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 66%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shezad Chishti
1
Reputation point
commented 2022-01-12T11:05:24.25+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee