Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,549 questions
1 answer
Network and Security Requirement for Azure Data Factory - Azure Integration Runtime
Data Source is SaaS with public endpoint. I am planning to do the movement of data from SaaS to SQL Server in Azure. I am using Azure Integration Runtime on ADF. I want to know below questions: Saas --> SQL server Network requirement from Saas…
asked 2020-09-02T19:45:03.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-09-10T18:38:12.883+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,026
Reputation points
0 answers
parent child relationships using ADF
Could you also provide me some reference for migrating multiple entities with parent child relationships using ADF. How do we create the lookup codes of power apps entities in the adf pipelines
asked 2020-09-07T05:36:07.877+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 88%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKT%3C/text%3E%3C/svg%3E)
Kesavi Thakkellapati
21
Reputation points
commented 2020-09-10T18:22:27.923+00:00
MartinJaffer-MSFT
26,026
Reputation points
2 answers
Problem with cosmos db mongo db connection
Hi, I would like to ask about data migration. I want to migrate data from my cosmos DB MongoDB 3.2 to cosmos DB MongoDB 3.6 and decided to use ADF. Two weeks ago everything worked well, but now I have an error 2200. I double-checked connection strings.…
asked 2020-08-10T14:07:59.113+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 48%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
chodkows
11
Reputation points
commented 2020-09-10T15:46:34.673+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
What consitutes pipeline failure?
I have this pipeline: (It's a simplification of a real pipeline). Execution goes like this: The Fail activity fails (I'm casting a non-numeric string to an int, just btw) The OnFailure activity executes and succeeds The pipeline…
asked 2020-08-16T16:48:40.047+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 77%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Richard Swinbank
372
Reputation points MVP
commented 2020-09-10T11:53:51.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,126
Reputation points
2 answers
BULK INSERT WITH Self hosted IR failing
Hi With Azure ADF copy method BULKINSERT copy data to OnPrem sql server to Azure Synapse its failing with below error. I verified the connection to azure synapse with : telnet <azure server name> 1433 its working. Firewall is opened to…
asked 2020-09-04T15:31:35.147+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 8%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHS%3C/text%3E%3C/svg%3E)
Harish Sunderaraj
176
Reputation points
commented 2020-09-10T05:30:28.307+00:00
HarithaMaddi-MSFT
10,126
Reputation points
1 answer
Copying an entire Data Factory, moving from development to production environment
Hello everybody and thank you for your attention. I currently have many pipelines, linked services, datasets and triggers inside my development Azure Data Factory. The purpose would be having such items copied (each one with its own settings,…
asked 2020-09-07T12:36:32.747+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELC%3C/text%3E%3C/svg%3E)
Leonardo clemente
1
Reputation point
commented 2020-09-10T05:27:01.323+00:00
HarithaMaddi-MSFT
10,126
Reputation points
0 answers
Irregular behavior with Azure Data Factory Data Flows. Debug results different than when running the pipline
Hello, I am building some pipelines for production in Data Factory. For one of my transformation I am using Data Flow. I am basically reading data from Parquet, do some processing, and sink the data into Azure SQL. In my debug session, when…
asked 2020-08-29T10:14:19.323+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 68%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Raef Mousheimish
6
Reputation points
commented 2020-09-09T20:26:52.48+00:00
MartinJaffer-MSFT
26,026
Reputation points
0 answers
Unable to Create Linked Server under Azure Data Factory
I am too experiencing same issue while creating linked server from my data factory to data lake - Were you creating a new linked service or editing an existing one?Ans- Creating a new one. Is your Data Factory in a Vnet or have other security/network…
asked 2020-09-09T05:20:20.55+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 16%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAJ%3C/text%3E%3C/svg%3E)
Amit Jain
1
Reputation point
commented 2020-09-09T20:10:08.203+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Unable to connect to azure server using linked services, load fail after selecting certificate to authenicate
Currently having this issue when creating a link service and now all my previous workflow are unable to connect as well. I have cleared all cache and log out of all accounts and restarted and the issue persist Failed to get response from server, please…
asked 2020-09-08T07:35:45.263+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 30%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWH%3C/text%3E%3C/svg%3E)
wang ho li
1
Reputation point
commented 2020-09-09T20:03:08.967+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Security Recommendations for Azure Data and Analytics Services
I am working on Securing Data and Analytics Services on Azure. I want to know what security controls i can apply after creating of services and what i can apply only during the service creation. Below are the recommendation i have found as of now. Could…
asked 2020-09-02T20:20:43.98+00:00
Akash Verma
21
Reputation points
commented 2020-09-09T11:11:32.767+00:00
PRADEEPCHEEKATLA-MSFT
77,086
Reputation points Microsoft Employee
1 answer
Error while copying Big Query to Data Lake in Azure Data Factory
Hi All, We are facing an issue while copying data from Google Analytics to Data Lake using Big Query connector in ADF. The linked services works fine, the preview of data also works fine. When we run the pipeline we get an error. Also I am running just…
asked 2020-08-31T11:25:15.013+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 41%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBR%3C/text%3E%3C/svg%3E)
Bharath R S
21
Reputation points
commented 2020-09-09T05:46:06.4+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Need East US region Datafactory Ips
We have Azure Sql Server where we enable Ip restrictions. Datafactory Connect with SQL for migration then it shows We need to whitelist ip 20.42.3.103. That is why I need full Ip ranges so that in future it will not get failed because of IP. Below is…
asked 2020-09-03T12:37:54.717+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 3%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Shubham Sharma
1
Reputation point
commented 2020-09-08T17:26:35.06+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF - Reset a pipeline parameter in one of the activities of the pipeline
I want to reset the value of the pipeline parameter in the pipeline. How can I go about doing this ? I have a scenario where one pipeline triggers a run of the child pipeline. Later it gets the status of the child pipeline by calling the rest API. I…
asked 2020-09-03T09:24:56.183+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 6%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan
941
Reputation points Microsoft Employee
commented 2020-09-08T16:29:43.067+00:00
MartinJaffer-MSFT
26,026
Reputation points
2 answers
AzCopy Support for On premise HDFS
Hi, Can we use AzCopy to transfer on-prem hdfs files to ads gen2 ? Thanks, Soumitra.
asked 2020-08-29T04:12:27.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 6%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Soumitra
1
Reputation point
commented 2020-09-08T09:00:12.167+00:00
Sumarigo-MSFT
43,721
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
publish collabarator branch via script
Hi Team, I know that there is publish button on the azure portal page of data factory to publish the collabarator branch to publish branch. However I want to know if it is possible to do it via script and if yes then how.
asked 2020-09-01T10:05:36.703+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 92%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJP%3C/text%3E%3C/svg%3E)
Jigar Patel
41
Reputation points
accepted 2020-09-08T06:40:10.167+00:00
Jigar Patel
41
Reputation points
2 answers
An internal DMS error occurred that caused this operation to fail
Hi Got the below error for unknown reason. Initially thought it could be due to data issue, so enabled the skip incompatible rows option to debug the issue, but after enabling it didn't skip any rows and the copy activity succeeded by copying all…
asked 2020-08-06T15:28:12.38+00:00
Harish Sunderaraj
176
Reputation points
commented 2020-09-05T00:08:00.857+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Synapse- The timeout period elapsed prior to completion of the operation or the server is not responding
Hi I am seeing this timeout error when copying to synapse using staged copy from OnPrem(SQL Server) to Synapse(using polybase). Looking at the error it says Synapse is timing out, but Synapse metrics shows its not touched its peak, we get this error when…
asked 2020-08-18T08:32:35.873+00:00
Harish Sunderaraj
176
Reputation points
accepted 2020-09-04T15:21:54.593+00:00
Harish Sunderaraj
176
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF - Copy Activity - Logging not happening
Following is the setting that I have enabled in my Copy activity. If I test the connection, the blob is reachable. However, there is no logging happening in the container, though files are copied successfully. Please provide some pointers as to…
asked 2020-09-01T10:25:28.847+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-09-04T11:45:37.387+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
1 answer
Unable to start Azure Data Factory SSIS Integration Runtime for my Managed Instance
I'm trying to deploy an Azure Data Factory SSIS Integration runtime for use with my SQL Managed Instance (on the private interface). I keep running into the same issue with this error: Failed to start the integration runtime: SSISintegrationRuntime.…
asked 2020-08-31T11:05:08.297+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 4%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
EJR
1
Reputation point
commented 2020-09-04T08:50:22.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 37%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Monalv-MSFT
5,891
Reputation points
1 answer
Excel to CSV conversion in ADF v2
Hello , Based on the latest feature of Excel support in ADF v2 in source, whne we try to leverage that functionality for converting an excel file to CSV we are stuck in a below situation: In excel a column value is displayed as: -1.36282E-05 But…
asked 2020-07-20T12:01:44.5+00:00
Nandan Hegde
29,886
Reputation points MVP
commented 2020-09-04T08:01:02.397+00:00
HarithaMaddi-MSFT
10,126
Reputation points