Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,925 questions
1 answer
Azure Data Factory Copy Data, Azure table storage to table storage only update if Timestamp of destination row is smaller
Would like some help with the following scenario, I have setup a simple copy from one Azure table to another Azure table. What I want to do is only update the destination row if the timestamp of the source row is greater then the destination row. On the…
asked 2020-06-03T05:57:12.057+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 42%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGA%3C/text%3E%3C/svg%3E)
Gavin Armitage
1
Reputation point
commented 2020-06-15T19:47:42.677+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
ADF pipeline errors, Another update is in progress on resource id
Since Monday, June 8th, 2020, our pipelines have been failing half of the time due to a new error: "message":"at Sink '####': com.microsoft.azure.documentdb.DocumentClientException: Another update is in progress on resource id…
asked 2020-06-10T18:34:20.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 41%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAF%3C/text%3E%3C/svg%3E)
A Fisher
1
Reputation point
commented 2020-06-15T17:50:31.133+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
dataflow detailed log
Hi, one of the Azure data factory pipeline failed with this error. where can we find the more detailed information like in which rows of the source file we have issues. {"message":"at Source 'source1': Column 'int_col has incompatible…
asked 2020-06-09T06:30:25.157+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESE%3C/text%3E%3C/svg%3E)
saikumar ellendula
6
Reputation points
commented 2020-06-11T21:04:02.297+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
custom failing a dataflow & custom error messages
[Enter feedback here] Hi, In azure data factory, we have a scenario of failing the dataflow if there are zero records in a file. using the aggregate transformation, we are able to derive the number of records in the file,but we done see a way to…
asked 2020-06-09T06:28:39.657+00:00
saikumar ellendula
6
Reputation points
commented 2020-06-11T20:36:22.433+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure DevOps CI/CD not deploying my pipeline triggers in an Active state
My dev ADF has pipeline triggers that are Active and set to run every hour. Using devops repos & CI/CD, I deployed these triggers from our dev to test ADF environments. The pipeline triggers were successfully created in my test environment, but they…
asked 2020-06-11T15:10:01.457+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 89%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Michael Mitchell
21
Reputation points
commented 2020-06-11T15:23:20.033+00:00
Dave Patrick
426.3K
Reputation points MVP
0 answers
ADF v2 Integration Runtime for Network share paths
HI I am looking to access the Network shared fodler with the help of Intgeration Runtime setup which is consumed by my copy activity, i am not able to access the path of the network share even after providing the required credentials which has the…
asked 2020-06-03T19:52:06.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 88%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
ManojMathe
1
Reputation point
commented 2020-06-11T06:05:02.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 76%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
ChiragMishra-MSFT
956
Reputation points
1 answer
How to add a Joing Query for pipeline Source
Hi Team, Currently i have configure integration Runtime and link services, After then I have to setup a join query for a pipeline source area as per below snap, but our concern is when i create a source connection i'm only allow to select one table…
asked 2020-06-04T02:45:02.047+00:00
Mahesh Madhusanka
106
Reputation points
commented 2020-06-11T06:02:09.483+00:00
ChiragMishra-MSFT
956
Reputation points
4 answers
ADF Data Flow Execution Failed
Hi Team, We are facing issue with Data Flow execution for past 3 hours and not able to run any pipelines. Can you please check the issue? { "errorCode": "4501", "message": "Failed to fetch execution state: '',…
asked 2020-06-04T11:27:17.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 88%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Sathish Kumar Sermadurai
16
Reputation points
commented 2020-06-10T16:50:32.903+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 8%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDF%3C/text%3E%3C/svg%3E)
Dan Faustine
1
Reputation point
3 answers
One of the answers was accepted by the question author.
Azure DataFactory : Set up Self Hosted IR
Hi Folks, I cannot set up a self hosted IR in my local machine. The purpose of this set up is to copy data from a SFTP server via a specific server to azure datalake. I have a admin credentials on my machine, however i get an error…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 9%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
2 answers
Connectivity between Azure data factory and Azure data catalog
Hi Team, We have a use case to use the azure data catalog Metadata information in Azure data factory. We have gone through the documentation however could not find any information on this. Could you help me if the feature is already available in…
asked 2020-06-09T07:29:32.307+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 38%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJR%3C/text%3E%3C/svg%3E)
Jyothi Ragi
1
Reputation point
answered 2020-06-09T07:39:22.73+00:00
PRADEEPCHEEKATLA-MSFT
83,066
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Change node size in Azure Data Factory (Spark)
Hello! How do I change the node Size in Azure Data Factory for HD Insight? I Want to create an Standard_D13_v2. But I didn't find a tutorial for do it:
asked 2020-06-06T00:51:51.307+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 42%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Joaquin Chemile
41
Reputation points
commented 2020-06-09T05:34:06.25+00:00
PRADEEPCHEEKATLA-MSFT
83,066
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Data Flow : Location
Azure Data Flow : is it supported in Location Australia South East? Looks like it is supported in Australia East. My Data Factory is built in Australia South East. To use data flows should I clone to different location? Thanks
asked 2020-06-08T21:30:50.48+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 94%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAA%3C/text%3E%3C/svg%3E)
Alan Anscombe
51
Reputation points
accepted 2020-06-08T22:58:48.367+00:00
Alan Anscombe
51
Reputation points
2 answers
One of the answers was accepted by the question author.
Delay between Activities in Data Factory
We have pipeline that has been running for 6 months in Data Factory V2. Last night there was a 48 minute delay from between two activities , no queued time. Does any know if there was maintenance on Data Factory V2 in West Central US on 5/27/2020 9:40…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 32%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
3 answers
Setting Pane in Mapping Data FLow in Azure Data Factory
Hi Experts , Can anyone advise me why am getting 'Settings pane'/'Data Preview Pane 'for configuring activity in Mapping Data FLows in bottom right hand corner of the screen ?. It was ok yesterday but seems from today am getting this View . Can…
asked 2020-06-02T06:54:42.487+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 42%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
n k
106
Reputation points
commented 2020-06-08T20:33:34.937+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
AZURE SQL MI Error: Stale/aborted version cleanup was aborted for database id '10' due to database shutdown.
Hi, I am trying to restore SQL server .bak file to Azure SQL Managed Instance database through SSMS but getting an error Msg 22003, Level 16, State 1, Line 18 Stale/aborted version cleanup was aborted for database id '10' due to…
asked 2020-06-05T18:47:16.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 57.00000000000001%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH0%3C/text%3E%3C/svg%3E)
HemantSudehely-0557
266
Reputation points
commented 2020-06-08T18:03:41.503+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 93%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Navtej Singh Saini
4,221
Reputation points Microsoft Employee
0 answers
toString does not accept array types
Hi All, Dataflows which have been working in the production environment have now started failing due to this error (DF-FN-006 - toString does not accept array types) This seems to be a recent change and I have quite a lot Mapping Dataflows which…
asked 2020-06-02T21:06:13.117+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 86%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rajneel Sharma
1
Reputation point
commented 2020-06-04T21:38:20.647+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Use single self hosted IR to source on-Prem sql database and target Azure VM sql server data server
I have required to perform Azure Data Factory Copy activity. My source is the On-Prem SQL server database and target sink activity to the Azure VM SQL server data server. I did try by creating 2 self-hosted IR(one for source and one for target) but…
asked 2020-06-03T16:40:42.713+00:00
HemantSudehely-0557
266
Reputation points
accepted 2020-06-04T13:38:37.177+00:00
HemantSudehely-0557
266
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory extracting Activities from Dynamics 365
I have a ADF extracting 'activity' (activitypointer) from Dynamics 365 Online. The ADF runs for 60+ minutes and fails with the below error for the copy activity. The sink is a gen2 data lake. I have 'AutoResolveIntegrationRuntime' for this data factory.…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 87%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
1 answer
ADF extract data from an on-premises oracle database into azure SQL database in real time
We have lots of web application on-premises, and its database include: Oracle, SQL Server, MySQL and SAP HANA. For Oracle DB, is it possible that ADF can extract data in read time? When there is any update in Oracle database, ADF can SYNC these…
asked 2020-06-02T02:02:42.407+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 53%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWF%3C/text%3E%3C/svg%3E)
Wenjian Feng
41
Reputation points
commented 2020-06-03T08:43:10.39+00:00
Wenjian Feng
41
Reputation points
1 answer
How to push data from Azure Data lake to remote file server (network file folder)
Hello Experts, I need your guidance on how to push files from Azure Data lake to remote file server(network file folder). Our requirement is to send files from Azure Data Lake Gen2 to Remote File Server( Network File Folder), once the data pushed…
asked 2020-06-01T08:56:36.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 78%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EF%3C/text%3E%3C/svg%3E)
fbmsdn511
1
Reputation point
commented 2020-06-03T01:24:27.297+00:00
KranthiPakala-MSFT
46,432
Reputation points Microsoft Employee