Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,160 questions
1 answer
ADF data Flow SQL Server as Sink
Hi I have one issue with the sql server database with the table as parameter on the data set: On the DataFlow the parameter not appear as option but of course gives a message that the parameter is not provided:
asked 2020-09-10T12:07:25.417+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 67%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
jmarciogsousa
6
Reputation points
commented 2020-09-14T07:49:54.267+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF web activity issue
seems web activity can't read HTTP header content type correctly. I am using WEB activity to refresh power BI dataset. it's working normal in ADF till 2020/08/30. I test the web API in local PostMam, it's working. so I think it's not Power…
asked 2020-09-11T07:23:11.59+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 64%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZZ%3C/text%3E%3C/svg%3E)
zhou zhou
21
Reputation points
commented 2020-09-14T05:26:08.943+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Error Testing Connection from SSIS to ADLS Gen2
I'm trying to connect to ADLS Gen2 from SSIS using an Azure Storage Connection Manager using a service account & Service Principal authentication. I'm using the Key Vault client ID for the application ID and secret key for the Application key. I'm…
asked 2020-08-31T17:17:19.263+00:00
atrapp
6
Reputation points
commented 2020-09-14T02:15:53.777+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 37%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Monalv-MSFT
5,896
Reputation points
2 answers
Azure SQL Database Port 1433 Workaround - Data Factory Enable Staging option not working
Hello, I have developed data pipelines and database to store data while I was working from home. However, I am not able to access the database in the corporate network. After some research I found that port 1433 Outbound should be open for Azure SQL…
asked 2020-08-13T16:51:59.177+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 5%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Chandrasekaran, Raji
1
Reputation point
commented 2020-09-11T20:37:28.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,061
Reputation points
2 answers
Installing self hosted integration runtime on Azure virtual machine
We need to install self-hosted integration runtime on Azure Virtual machine (VM) and mainly while configuring which instance of Azure Virtual machine (VM) is preferred.
asked 2020-09-07T10:41:34.13+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 82%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vamshi Kishore KALIKOTA
51
Reputation points
commented 2020-09-11T16:56:05.13+00:00
prmanhas-MSFT
17,901
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
problem with If Condition proper setup
I'm building a flow in my Azure Data factory pipeline where want to setup some conditional logic. The purpose is to validate the blob files were copied properly into Azure SQL Database and if so move the blob files to a backup folder or if not move…
asked 2020-09-10T15:03:35.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 26%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
braxx
436
Reputation points
commented 2020-09-11T11:15:59.3+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Parsing json to azure sql db - mapping
Hello I have a blob storage with json files comming from API and loaded there ones per day. The case is to parse the jsons and import to azure sql db on a daily basis. The jsons contains about 70 attributes with values. The structure is not always the…
asked 2020-09-04T08:07:24.977+00:00
braxx
436
Reputation points
commented 2020-09-11T09:44:16.747+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Azure Data Factory ForEach limit at 5000 items?
Hi, I am using ForEach activity that loops over more than 7000 items in a Azure Data Factory pipeline. However, looking at the run history of my pipeline, ForEach only actually covers 5000 of the items. According to doc:…
asked 2020-09-07T17:14:32.047+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 28.999999999999996%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMZ%3C/text%3E%3C/svg%3E)
Minxiao Zhou
1
Reputation point
commented 2020-09-11T09:42:23.017+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Webapp to extract Excel data
Hi there - I am trying to create a database from existing Excel model files, and then access the data through webportal or through PowerBI. Unfortunately the data is not currently structured as a table and these files are fairly complex. Could I…
asked 2020-09-07T22:48:03.497+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 24%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENA%3C/text%3E%3C/svg%3E)
Nicholas Ahn
1
Reputation point
commented 2020-09-11T09:41:23.79+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Replace some characters within an Excel file via ADF v2
Hello, we have a below architecture wherein: Excel file in blob >>> ADF v2 to convert it into CSV file >>> Azure synapse(polybase) Now the challenge is ADF v2 supports only single delimiter and in case if that delimiter is already…
asked 2020-09-09T05:52:22.66+00:00
Nandan Hegde
31,436
Reputation points MVP
commented 2020-09-11T09:39:31.637+00:00
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
Can't access txt file with json format from Data Lake Gen2 inside a Data Flow
I have stored a txt file with json format (coming from a MongoDB) in my Data Lake Storage Gen2 and inside my Data Flow I would like to access that file, convert it into a column structure, transform the data and shift it to my Azure SQL DB. Currently I…
asked 2020-08-19T11:37:03.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 73%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Joko123
1
Reputation point
commented 2020-09-10T20:03:06.417+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
2 answers
Public IP address ADF
Hello Friends, I need to copy files from third party vendor's SFTP using ADF Copy activity (US West2). The third party needs to whitelist the public IP to provide access to SFTP location. How can I provide Azure Data Factory's public IP or IP…
asked 2020-09-08T16:34:37.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 92%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Abhi Tripathi
6
Reputation points
commented 2020-09-10T18:45:36.933+00:00
MartinJaffer-MSFT
26,061
Reputation points
1 answer
Network and Security Requirement for Azure Data Factory - Azure Integration Runtime
Data Source is SaaS with public endpoint. I am planning to do the movement of data from SaaS to SQL Server in Azure. I am using Azure Integration Runtime on ADF. I want to know below questions: Saas --> SQL server Network requirement from Saas…
asked 2020-09-02T19:45:03.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-09-10T18:38:12.883+00:00
MartinJaffer-MSFT
26,061
Reputation points
0 answers
parent child relationships using ADF
Could you also provide me some reference for migrating multiple entities with parent child relationships using ADF. How do we create the lookup codes of power apps entities in the adf pipelines
asked 2020-09-07T05:36:07.877+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 88%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKT%3C/text%3E%3C/svg%3E)
Kesavi Thakkellapati
21
Reputation points
commented 2020-09-10T18:22:27.923+00:00
MartinJaffer-MSFT
26,061
Reputation points
2 answers
Problem with cosmos db mongo db connection
Hi, I would like to ask about data migration. I want to migrate data from my cosmos DB MongoDB 3.2 to cosmos DB MongoDB 3.6 and decided to use ADF. Two weeks ago everything worked well, but now I have an error 2200. I double-checked connection strings.…
asked 2020-08-10T14:07:59.113+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 48%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
chodkows
11
Reputation points
commented 2020-09-10T15:46:34.673+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
What consitutes pipeline failure?
I have this pipeline: (It's a simplification of a real pipeline). Execution goes like this: The Fail activity fails (I'm casting a non-numeric string to an int, just btw) The OnFailure activity executes and succeeds The pipeline…
asked 2020-08-16T16:48:40.047+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 77%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Richard Swinbank
372
Reputation points MVP
commented 2020-09-10T11:53:51.58+00:00
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
BULK INSERT WITH Self hosted IR failing
Hi With Azure ADF copy method BULKINSERT copy data to OnPrem sql server to Azure Synapse its failing with below error. I verified the connection to azure synapse with : telnet <azure server name> 1433 its working. Firewall is opened to…
asked 2020-09-04T15:31:35.147+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 8%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHS%3C/text%3E%3C/svg%3E)
Harish Sunderaraj
176
Reputation points
commented 2020-09-10T05:30:28.307+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Copying an entire Data Factory, moving from development to production environment
Hello everybody and thank you for your attention. I currently have many pipelines, linked services, datasets and triggers inside my development Azure Data Factory. The purpose would be having such items copied (each one with its own settings,…
asked 2020-09-07T12:36:32.747+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELC%3C/text%3E%3C/svg%3E)
Leonardo clemente
1
Reputation point
commented 2020-09-10T05:27:01.323+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Irregular behavior with Azure Data Factory Data Flows. Debug results different than when running the pipline
Hello, I am building some pipelines for production in Data Factory. For one of my transformation I am using Data Flow. I am basically reading data from Parquet, do some processing, and sink the data into Azure SQL. In my debug session, when…
asked 2020-08-29T10:14:19.323+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 68%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Raef Mousheimish
6
Reputation points
commented 2020-09-09T20:26:52.48+00:00
MartinJaffer-MSFT
26,061
Reputation points
0 answers
Unable to Create Linked Server under Azure Data Factory
I am too experiencing same issue while creating linked server from my data factory to data lake - Were you creating a new linked service or editing an existing one?Ans- Creating a new one. Is your Data Factory in a Vnet or have other security/network…
asked 2020-09-09T05:20:20.55+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 16%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAJ%3C/text%3E%3C/svg%3E)
Amit Jain
1
Reputation point
commented 2020-09-09T20:10:08.203+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee