Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,928 questions
1 answer
Unable to create a New SFTP Linked service connection using Public Key Authentication
Hi, I am currently having an issue setting up a SFTP linked service in ADF using public key authentication and getting the error as 'Invalid Sftp credential provided for 'SshPublickey' Authentication type. DER Length is 114 and cannot be more…
asked 2020-08-31T22:09:18.193+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 7.000000000000001%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Puneet, Mithun
1
Reputation point
commented 2020-09-04T00:35:33.953+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Wait activity actual wait time does not match the waitTimeInSeconds input
We saw this is happening: the actual execution time of the wait activity does not match the input of waitTimeInSeconds. It might be either much longer or much shorter than the actual seconds passed in. Does anyone know what might be the cause? Thank you!…
asked 2020-08-31T18:59:23.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 23%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
Candice Guo
26
Reputation points
accepted 2020-09-03T16:09:45.66+00:00
Candice Guo
26
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory - HTTP Connectors
Hi I am working in Azure Data Factory and doing "Copy Data Activity from HTTP Connector". It works for me but only the first page is getting saved into the database table. My question is how to do pagination in HTTP connector.
asked 2020-08-17T13:33:03.733+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 33%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWD%3C/text%3E%3C/svg%3E)
Wadhwa, Dinesh
21
Reputation points
accepted 2020-09-03T06:59:46.85+00:00
Wadhwa, Dinesh
21
Reputation points
0 answers
How to copy data from Azure SQL Database to Azure Synapse?
Hello Experts, I am pretty new to Azure Data Factory and struggling with a simple problem. I've done this in SSIS but now trying to implement the same in Azure Data Factory. I have a list of tables that I would like to copy the data on an incremental…
asked 2020-08-30T16:39:04.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 19%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Julaayi
1
Reputation point
commented 2020-09-03T04:43:02.267+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
One of the answers was accepted by the question author.
ADF - Copy activity - getting list of files copied
Is there a way in which I can get the list of files copied by the copy activity. I do not want to use the GetMetaData activity, as in the time different between executing the copy activity and the getmetadata activity, more files could have been added to…
asked 2020-09-01T10:11:49.237+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 6%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-09-02T08:02:59.18+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
0 answers
Powershell to update the SQL server Linked service in ADF v2
Hello team, Is is possible to update the SQL server linked service in ADF v2 via windows auth via powershell by passing all the details during run time/dynamically via Azure Devops library rather than having a static file for :…
asked 2020-08-21T13:04:56.897+00:00
Nandan Hegde
30,166
Reputation points MVP
commented 2020-09-01T23:38:05.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,046
Reputation points
2 answers
Azure data factory reporting
Hello I'm using ADF in order to develop an ETL process, at the end of the process I need to have a txt file with these kind of numbesrs: number of files processed Number of records processed (per file) Number of records not processed (per file) …
asked 2020-07-17T12:04:25.027+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 71%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
CORONADO GRANADOS Diana Milena
126
Reputation points
answered 2020-09-01T20:14:17.857+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Triggering a pipeline using web activity - parameters not reflecting
In continuation to the question raised in https://learn.microsoft.com/en-us/answers/questions/81708/triggering-a-pipeline-using-web-activity-not-able.html, I tried to keep the context parameter as default and added a new parameter called interchangeid…
asked 2020-08-31T09:22:56.057+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-09-01T10:26:13.687+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Triggering a pipeline using web activity - not able to pass parameters
From one pipeline, I have a web activity in which I trigger another pipeline. When I try to pass parameters to the pipeline, I am not able to do this. Following are the details. Pipeline 1 - Web Activity In the pipeline which is being…
asked 2020-08-31T09:00:06.413+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-09-01T10:07:19.15+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Using MSA account with Integration Runtime for On-Premise File System
We have a scenario to copy data from on-premise to Cloud Storage and we are using an Integration Runtime for this. In the Linked Service to the on-premise file system, I see an option for providing username/password, or to retrieve the same from a key…
asked 2020-08-13T11:45:22.223+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-09-01T05:31:17.843+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
2 answers
Import incremental data from Azure Data Lake
Good afternoon, I have the following scenario !! I have a table in SQL that always receives changes daily, and I need to send this data to Azure data lake So, today I take this data and send it in the datalake file: Historicos.csv …
asked 2020-08-20T17:58:50.56+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 25%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELS%3C/text%3E%3C/svg%3E)
Lucas Silva
1
Reputation point
commented 2020-08-31T18:20:50.803+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
0 answers
csv incremental data load to database concern
Hi Team, Currently, We have requirement to datalake csv file sync with the on-prime database, according to i have created connection and currently working on loading data from datalake csv(one csv file as a table) file to on-prime database table. In…
asked 2020-08-24T01:43:40.57+00:00
Mahesh Madhusanka
106
Reputation points
commented 2020-08-31T07:52:06.883+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Want to Read Microsoft Team Post in Azure Data Factory
I Have requirement to read Microsoft Team post in Azure Data Factory. currently i am using logic apps , but it generate output in html file and it is difficult to read what is the best way to get data from Microsoft team, so we can read data in…
asked 2020-08-27T08:52:24.843+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 57.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVM%3C/text%3E%3C/svg%3E)
Verma, Manish Kumar
131
Reputation points
commented 2020-08-31T06:13:56.377+00:00
MayankBargali-MSFT
69,926
Reputation points
2 answers
Why does the ADF integration runtime have a dependency on Java for parquet compression purposes?
I've installed the IR (self-hosted) several times over the course of the past six months and have been confused by the dependency on the JDK. (we use this one https://adoptopenjdk.net/ ) You can read about this dependency in the docs here:…
asked 2020-08-07T15:46:37.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 78%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
David Beavon
976
Reputation points
answered 2020-08-28T23:17:59.16+00:00
David Beavon
976
Reputation points
2 answers
How to flat complex json data into sql db using copy activity?
Hi Team, I have data with multiple arrays and inside multiple objects. I am using copy activity to load into database. I Have used collection reference to convert the data. But it's not helping much, out of 1000 users data ,either I get all users with…
asked 2020-08-13T07:31:14.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETG%3C/text%3E%3C/svg%3E)
THIMMAIAH GARI,PRASHANTH,,
201
Reputation points
commented 2020-08-28T22:29:01.183+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
Error in Lookup while getting distinct values
I am trying to do a look up on a table, preview data in the look up gets me the correct data when table option is chosen but when the query option is chosen and if I try to get the distinct values from the table it throws me an error
asked 2020-08-20T06:11:59.817+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 42%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Manjari Manokar
1
Reputation point
commented 2020-08-28T22:26:38.847+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
0 answers
hw to ingest only files from new folder in AWS to ADLS
I have a AWS s3 bucket with 4 folders in it. Each folder has sub folders in it and each subfolder has files in it. The folder structure is like below. For each day there will be another folder with date(for ex, for next day a folder with 2020-01-03 will…
asked 2020-08-25T04:23:49.003+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 37%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
reddy
41
Reputation points
commented 2020-08-28T22:25:33.19+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
How to write Dynamic query in Azure Bulk data pool?
I want to write Dynamic query in Azure SQL Bulk data load.
asked 2020-08-25T10:48:08.257+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 16%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Vinay Raghuwanshi
1
Reputation point
commented 2020-08-28T22:18:43.273+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
ADF - Http as Source and Sink
Can a copy activity, copy from a http source to a http destination? In the "Supported data stores and formats' in ADF documentation, I see that http is supported only as source. Is it that http cannot be a destination ? Any detail on this will be…
asked 2020-08-21T07:02:42.597+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
commented 2020-08-28T19:36:13.48+00:00
MartinJaffer-MSFT
26,046
Reputation points
1 answer
One of the answers was accepted by the question author.
Error on tab delimited file Dataset
Hi, I'm having problems with a tab delimited file on Azure Data Factory. I have a Self-hosted Integration Runtime pointing to a drive on a On-premises server, and a Dataset that is pointing to this tab delimited file. Here we can see the setup of…
asked 2020-08-27T17:50:39.343+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 60%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Alexandre Von Mühlen
71
Reputation points
commented 2020-08-28T18:29:40.31+00:00
Alexandre Von Mühlen
71
Reputation points