Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,012 questions
1 answer
How to show the task tab in Azure Data Studio
My connection tab looks like this, it's hard to switch to another connect or SQL file How can I restore it like the preset view ,like this Thanks a lot
asked 2020-09-16T06:04:14.007+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 91%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJT%3C/text%3E%3C/svg%3E)
Jiawen Tang
1
Reputation point
commented 2020-09-24T05:07:31.773+00:00
PRADEEPCHEEKATLA-MSFT
84,371
Reputation points Microsoft Employee
1 answer
Store procedure with output param in lookup
Hi, I am using Lookup activity to execute store procedure with an output parameter. But I have noticed if the store procedure doing Insert, then "message": "Failure happened on 'Source' side.…
asked 2020-09-23T12:52:55.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 98%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
rajesh kumar
1
Reputation point
answered 2020-09-23T18:07:53.133+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Data Factory copy data to another container
Hi Everyone, Could you help me please In Data Factory I created Copy Data pipelines from one Storage account (Containers) to another Storage account (Containers), copy working fine , could tell me please how I can configured to keep every time new…
asked 2020-09-17T23:23:48.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 85%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Alex Corovin
46
Reputation points
accepted 2020-09-23T00:16:21.977+00:00
Alex Corovin
46
Reputation points
1 answer
Copy data from multiple Azure DB's to one Azure DB with Data Factory
I'm not sure that this is possible for this service or not. I have many databases of the same schema that I want to copy data on a monthly data into one single database. I need to be able to use a Table that contains a list of these databases and…
asked 2020-09-17T16:16:24.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 25%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELT%3C/text%3E%3C/svg%3E)
Lee Taylor
31
Reputation points
commented 2020-09-22T20:18:20.293+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,051
Reputation points
1 answer
One of the answers was accepted by the question author.
Does data factory Copy Activity check if file at source has finished writing ?
Hello, I am using a Copy Activity to copy recent files (past 15 minutes by checking LastModifiedDate in the copy activity) from SFTP to ADLS Gen 2. At the moment, to make sure that the files the activity copies are complete in the source and not…
asked 2020-09-22T08:48:33.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 79%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEB%3C/text%3E%3C/svg%3E)
Emilien BOUCAUD
21
Reputation points
accepted 2020-09-22T12:21:30.48+00:00
Emilien BOUCAUD
21
Reputation points
2 answers
Data Factory V2 Storage Account not Refreshing - Because it thinks its blob Storage
I have created a Storage Account set to be Azure Data Lake V2 by setting the hierarchical namepace. And I used Azure Storage Explorer to add some files into the blob Container (Which is an Azure datalake gen2) I'm now creating a Data Lake V2 Linked…
asked 2020-07-31T10:10:17.587+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDE%3C/text%3E%3C/svg%3E)
Debbie Edwards
521
Reputation points
answered 2020-09-22T10:55:12.323+00:00
Debbie Edwards
521
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data factory - Reading nvarchar column from excel source xlsx in azure data factory but numeric data is not being read properly!!
Column col1 has both numeric and character data so reading it as string in excel source(xlsx). Excel data ex. col1 (General Format) DFSDF 23 12345678 Expecting to read the data as it is in azure data factory ,but its altering the numeric data and…
asked 2020-09-10T08:22:01.303+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 83%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gandhikumar, Suganthi
31
Reputation points
accepted 2020-09-22T05:35:06.88+00:00
Gandhikumar, Suganthi
31
Reputation points
1 answer
One of the answers was accepted by the question author.
How to trigger the pipeline when the upstream data appears.
I want my pipeline wait to run until it meet some conditions like the upstream data appears. This feature is supported in Maestro but I could'nt find similar in ADF.
asked 2020-09-17T06:16:25.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 78%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFJ%3C/text%3E%3C/svg%3E)
Feiyue Jiang
41
Reputation points
accepted 2020-09-22T02:23:46.88+00:00
Feiyue Jiang
41
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF Trigger Activation fails when it contains a JSON object parameter containing a property value that begins with "@"
Hi, I believe I have found a bug when trying to activate a trigger in Azure Data Factory. I have been using JSON objects as parameters for a while now and just encountered a publishing error when I try and set the trigger state to Activated=Yes. …
asked 2020-09-04T10:45:10.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 40%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Nicholas
96
Reputation points
commented 2020-09-21T23:53:15.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
0 answers
Why do I have to have GIT
Why do I have to have GIT to use Data Factory? I currently use TFS. Why can't that be used, or my Azure Storage container or.... I hate GIT
asked 2020-09-17T14:04:04.863+00:00
Lee Taylor
31
Reputation points
commented 2020-09-21T23:46:43.76+00:00
MartinJaffer-MSFT
26,051
Reputation points
0 answers
Cannot get the right file path
Hi All, I am new to ADF . I created a sftp linked service successfully. When creating data set (which is an excel file). No matter how many times i give the right path i get the following error message. Very frustrated. please help me. Please see…
asked 2020-09-03T20:07:20.747+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 7.000000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
bk
461
Reputation points
commented 2020-09-21T21:39:20.773+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
2 answers
How to Maintaing log table to Datalake File data Count
Hi Team, Currently we have a setup on data factory pipeline copy some specific data from database and data append on the datalake csv file wise. Currently we have a requirement to maintain a table as a log which is data Synchronization from third party…
asked 2020-09-08T12:36:11.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 19%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mahesh Madhusanka
216
Reputation points
commented 2020-09-21T21:37:07.327+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
2 answers
Extract files from .tar.gz files store in Blob Container
Is it possible to extract files from .tar.gz using Azure Data Factory or Functions App to be ingested by ETL process in ADF? I tried to use 7zip in Functions Apps, which worked fine to extract a test .tar.gz file uploaded to the Functions App but…
asked 2020-09-11T17:23:46.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 83%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
elastiSol
1
Reputation point
commented 2020-09-21T17:07:02.647+00:00
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
What does publishing do in Azure Data Factory do?
What does publishing a pipeline/dataset in data factory do? Does it publish the data somewhere where other users can access? Is it related to publishing a Power BI notebook?
asked 2020-09-19T15:02:01.35+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 94%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Siddhanth Shetty
66
Reputation points
accepted 2020-09-21T11:35:21.723+00:00
Siddhanth Shetty
66
Reputation points
1 answer
Mapping Data Flow Sink to SQL on IaaS VM
Does Mapping Data Flow allow us to sink the data into a SQL DB hosted on a IaaS VM? Currently, there is no option other than Azure Blob Storage or Azure SQL. Any inputs here are highly appreciated. Thanks.
asked 2020-09-15T20:05:01.807+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 73%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Priyaranjan Midde
1
Reputation point
commented 2020-09-19T00:05:59.85+00:00
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
0 answers
ADF SQL to CRM Copy Data Pipeline Error
I am currently working on a Azure Data Factory Mapping. SQL Server to CRM There is a Entity Reference look up field that I am trying to update. Do not know how to. The field is: new_districtlink --> Territory/Territories. My Source does a…
asked 2020-09-14T03:42:15.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 55.00000000000001%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
Shamsuddin, Natasha
1
Reputation point
commented 2020-09-18T23:31:29.247+00:00
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF pipeline stuck in "Queued" status while writing data in SFSC
I am writing to upsert / insert records in SFSC from sql server object . The connection to SFSC is working fine and successful. But when I run the pipeline the status is stuck in "Queued" . It never changes.
asked 2020-09-10T16:52:55.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 61%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Alinkil, Mithun
21
Reputation points
accepted 2020-09-18T20:23:20.773+00:00
Alinkil, Mithun
21
Reputation points
1 answer
Copy Activity Sink to SFTP "useTempFileRename": false option not always working
Has anyone else experienced this issue, is it a bug? When using Copy Activity and ftp to a site which does not allow file rename I set "useTempFileRename": false. But even though ADF will say copy was successful the file sometimes is not…
asked 2020-09-15T14:03:02.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 31%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jay
561
Reputation points
commented 2020-09-18T17:54:12.513+00:00
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
protect JSON-file /ARM template Azure Data Factory
Hi, I have developed an ETL procedure for a specific ERP system in Azure Data Factory. I want to sell the entire procedure to customers so that they can use the ARM template in their own Azure tenant. However, I do not want them to make a copy of…
asked 2020-09-18T11:34:13.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 52%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
markharing@bisqq.nl
61
Reputation points
accepted 2020-09-18T12:28:13.007+00:00
markharing@bisqq.nl
61
Reputation points
1 answer
One of the answers was accepted by the question author.
How to manually cancle a ADF pipline when an activity fails or completes?
Hi, I have a pipeline which has two concurrent list of activities running at the same time lets say the first activity has a flag which makes the until loop in the second list of activity to complete if flag is true. Now the problem is the flag is…
asked 2020-09-14T04:39:40.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 21%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Stramzik
21
Reputation points
commented 2020-09-17T20:25:56.393+00:00
MartinJaffer-MSFT
26,051
Reputation points