Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
1 answer
One of the answers was accepted by the question author.
Data factory pipeline shows success when copying csv to sql azure but data does not appear in the table
When running a data factory pipeline to copy a csv file to sql azure, the job shows as a success and confirms that the proper amount of rows were copied but when querying the table in the database, the data does not show.
asked 2020-11-12T20:57:46.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 74%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Robert Anderson
116
Reputation points
accepted 2020-11-17T14:58:58.18+00:00
Robert Anderson
116
Reputation points
2 answers
Azure Data Factory - Paypal - Lookup error
I am trying to execute a lookup to read records from Paypal: This is the query I am running Select Id,Transactions_Index,Sale_Create_Time,Sale_Parent_Payment,Sale_Amount_Details_Subtotal, Sale_Id from Payment_Transactions_Sale Where Sale_Create_Time…
asked 2020-11-16T04:59:23.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 52%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Ana Del Campo
121
Reputation points Microsoft Employee
answered 2020-11-17T01:22:19.28+00:00
Ana Del Campo
121
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure data factory - query pipeline by data annotations
Is there a way to programmatically query (using .net SDK) list of data factory pipelines by data annotations? I can set data annotation when I run pipeline as explained here, but not sure how to query pipelines or filter pipelines using the same?…
asked 2020-10-23T21:37:28.577+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 84%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJV%3C/text%3E%3C/svg%3E)
JAY VORA
21
Reputation points
commented 2020-11-16T23:38:22.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,061
Reputation points
1 answer
How we can get uptime for all the resources in azure.
Hi guys, I have to check the uptime for the resources in Azure. As for some resources, we can check directly from the Azure portal. I'm able to see uptime for the storage account and SQL database but how can I get the uptime for other resources such…
asked 2020-11-07T07:03:54.573+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 21%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashima Gupta
66
Reputation points
commented 2020-11-16T22:45:46.163+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
0 answers
Expiry Datetime incorrect for Tumbling Window Trigger Catchup Run?
I have a pipeline that primarily has 2 copy activities . The sink for the copy activities is ADLS Gen 1, so it also sets an expiry datetime for each of the files copied over. It uses a tumbling window trigger with a self-dependency. Recently, the…
asked 2020-11-06T19:26:33.123+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 36%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Rasika Athavale
1
Reputation point Microsoft Employee
commented 2020-11-16T22:40:37.95+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
Logs are missing in the Log Analytics Workspace - Azure Data Factory starting 10/23/2020
Hello, Our ADF and the log analytic workspace are hosted under West US2, same as our ADF, and since 10/23, we’ve noticed several pipeline runs do not come with the Succeeded or Failed log, but just InProgress or Queued. It threw off our query since…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 98%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHH%3C/text%3E%3C/svg%3E)
2 answers
Data Factory Loading multiple csv files. Is there any way of dealing with column header differences?
I have csvs in a datalake. Im coping them into SQL using the copy activity the File Path type is a wildcard file path Very occasionally there are slight differences between the column headers. Is there any way that this can be dealt with rather…
asked 2020-11-05T12:56:42.187+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDE%3C/text%3E%3C/svg%3E)
Debbie Edwards
521
Reputation points
commented 2020-11-16T22:16:54.58+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Getting TlsVersionNotPermitted error when SSIS package executing from ADF
Hi, I have created one SSIS package which is reading data from Storage account and putting that into Azure DB. This is working perfectly well when executing from Visual Studio. But when I am triggering it after deployment in ADF, getting below error. …
asked 2020-11-12T18:01:00.723+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 30%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vikas
21
Reputation points
commented 2020-11-16T18:10:09.6+00:00
MartinJaffer-MSFT
26,061
Reputation points
1 answer
One of the answers was accepted by the question author.
Query Azure Function HTTP Get from Azure Data Factory
I have a function like: public static async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest req, ILogger log) { string dateFromString =…
asked 2020-11-13T12:22:03.513+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 3%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECH%3C/text%3E%3C/svg%3E)
Christian Hansen
21
Reputation points
accepted 2020-11-16T13:33:21.307+00:00
Christian Hansen
21
Reputation points
2 answers
Azure data factory web activity body
How do I use parameters in the body of Push method in data factory? It seems to allow static values or a parameter but not both. Say I want my message body as {"count" :@Anonymous } But this isn't working, any ideas
asked 2020-11-10T04:14:56.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 7.000000000000001%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Accelerator
1
Reputation point
commented 2020-11-16T06:31:20.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Is it possible to rename file name with MD5 hash encoding in Azure data factory?
I have this scenario where I have a blob with multiple binary files and I would like to copy these files into another blob storage using Azure data factory. The issue is my destination file name should be MD5 encoded . Now I would like to know if it…
asked 2020-10-05T12:46:22.937+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 85%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
ankit kumar
101
Reputation points
commented 2020-11-14T00:03:32.147+00:00
MartinJaffer-MSFT
26,061
Reputation points
1 answer
Azure data factory Cosmos Db delete record
hello After run an ETl process through ADF I need to run a bluk delete on cosmos DB . I know that ADf doesn't suppor this feature but it could be possible to do it through an azure function. I don't know how to deal with a bulk delete(based on a…
asked 2020-11-12T13:03:33.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 36%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDM%3C/text%3E%3C/svg%3E)
Diana Milena Coronado Granados
1
Reputation point
answered 2020-11-13T02:18:55.527+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 93%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Navtej Singh Saini
4,221
Reputation points Microsoft Employee
1 answer
Linked Service with ADF Connection Failing
I am attempting to connect by Azure Data Lake Analytics account with Azure Data Factory via a Linked Service. I have added the required information in the Edit linked service blade, created an app and a secret for the Service Principal ID and key, set…
asked 2020-11-02T17:23:19.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 73%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Supply Chain Glass
1
Reputation point
commented 2020-11-12T20:29:42.27+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,791
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
More than two time trigger duplicate records inserted in increment load.
Hi Team, I am follow below link create Pipeline i have haapy answer but I have problem when i am insert more two time for day its dupplicate records inserted. please anybody help how to avoid duplicate record in sink table my client didnt…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 79%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
AZURE APP, AZURE POSTGRESQL, EMAIL
I have .net app deployed on azure web app, connected with azure postgres db. I push SAP BW data into azure postgres using azure data factory. Data processed in db and final processed/ not processed output is stored in db table. I want to trigger…
asked 2020-11-11T10:52:48.123+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 41%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Charmi
1
Reputation point
answered 2020-11-12T11:38:33.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 10%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KalyanChanumolu-MSFT
8,316
Reputation points
1 answer
Copy activity output entities and Query activity of ADW
HI All, We have a pipeline that contains copy activity which runs query on Azure SQL DB and generates a blob file. PFB screenshot of copy activity output 1) Wanted to know how Copy duration mentioned in the output is generated. Is it combination of…
asked 2020-11-06T14:15:41.207+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 27%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Priya Jha
871
Reputation points
commented 2020-11-12T06:51:07.097+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Azure Data Factory Issue SFTP Sink tasks fails after copying 19 files
Hi there, I have an Azure Data Factory Pipeline that has a task that is copying CSV files to a SFTP server. { "dataRead": 3570267, "dataWritten": 1491182, "filesRead": 31, …
asked 2020-11-06T12:45:09.263+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 5%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3END%3C/text%3E%3C/svg%3E)
Nathan Dear
1
Reputation point
commented 2020-11-12T04:07:58.06+00:00
Nathan Dear
1
Reputation point
1 answer
Unable to query data factory pipeline runs
I am trying to query against azure pipelines runs for a given data factory pipeline resource. The account I am using has build in role Contributor role to the data factory resource. But I get a permission error when I try to query running pipelines…
asked 2020-11-09T17:40:52.157+00:00
JAY VORA
21
Reputation points
commented 2020-11-12T01:56:44.467+00:00
Saurabh Sharma
23,791
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
How to pass a parameter with more than one key field?
All I have a working pipeline, whereby the parameter LogicalKey, type string in pipeline as well as data flow, is filled with one value. This is the logical key of my Sink. Everything works fine. See ‘Old settings’ Now I changed the parameter. It…
asked 2020-10-21T14:02:19.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 92%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPV%3C/text%3E%3C/svg%3E)
Poel van der, RE (Ron)
431
Reputation points
commented 2020-11-11T08:04:08.083+00:00
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
Add transformation logic dynamically to Data Flow.
We have a Web App which the user uses to upload the file that need to be imported to Azure SQL Database via Azure Data Factory pipeline. Prior to uploading, the Web App provide a functionality where in the user can do the mapping of file columns to…
asked 2020-11-09T21:04:24.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
jigsm
236
Reputation points
commented 2020-11-11T00:22:23+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mark Kromer MSFT
1,146
Reputation points