Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,160 questions
1 answer
One of the answers was accepted by the question author.
Issue with ADF Data Flow, xml source and optional nodes or attributes
Hi, I need some assistance with ADF Data Flow using an xml data source. We have a data lake full of xml which I need to process. We do have a schema for the xml, but the xml isn't validated against it. We have some optional attributes and nodes in…
asked 2020-09-10T08:35:37.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 57.00000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Michael Payne
121
Reputation points
commented 2020-09-30T14:21:59.96+00:00
Michael Payne
121
Reputation points
2 answers
Azure data factory convert excel file copied from Sharepoint online to parquet
Hi All, I have a requirement to copy excel files from share point online to Azure data lake store(ADLS) Gen1 in parquet format, with the below link I was able to copy file over to ADLS. But I can't convert the file to parquet format as it doesn't…
asked 2020-07-01T04:22:19.69+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 33%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Cooldbguy
1
Reputation point
commented 2020-09-30T10:40:25.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 46%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
Purushotham Chanda
6
Reputation points
0 answers
Azure Batch linked service not showing in Azure Data Factory pipeline
I'm following this guide: https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory In step 6 under Set up an Azure Data Factory pipeline it shows that one can find the Batch Account under the Azure Batch linked…
asked 2020-09-25T10:42:04.337+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 4%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERR%3C/text%3E%3C/svg%3E)
Riis, Rasmus
6
Reputation points
commented 2020-09-30T09:08:21.883+00:00
Riis, Rasmus
6
Reputation points
2 answers
Azure Data Factory Activity wait times extending
We have a set of Azure pipelines that have been running for the past two years without incident. Processing in ADF is orchestration of stored procedures in Azure SQL PaaS and data copies between SQL PaaS databases, all for Data Warehousing. In the past…
asked 2020-09-22T21:28:21.19+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 67%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECT%3C/text%3E%3C/svg%3E)
Clive Trott
1
Reputation point
commented 2020-09-30T09:03:49.593+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Copy Activity going for loop when data get empty
I am trying to extract data from REST API to ADL--->ADL to SQLSTG. In one situation when source data get empty it's going in loop situation. How do i terminate from the loop instead of giving timeout in UNtil activity. How to avoid making multiple…
asked 2020-09-25T09:02:54.16+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETG%3C/text%3E%3C/svg%3E)
THIMMAIAH GARI,PRASHANTH,,
201
Reputation points
accepted 2020-09-30T08:50:42.017+00:00
THIMMAIAH GARI,PRASHANTH,,
201
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF Batch : Bacth Activty starts but never ends
My pipeline is working until it reaches a certain batch activity where process continues but never ends and throws no error message or warning so it has been not possible for me to figure out the problem. The python file to run is very simple (reads a…
asked 2020-09-29T20:23:54.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 97%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
javier
936
Reputation points
commented 2020-09-30T04:57:00.777+00:00
PRADEEPCHEEKATLA-MSFT
85,511
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Case labels in Data Factory Switch activity not editable?
Having an issue with Data Factory's Switch activity -- the UX won't let me edit case labels. When I add a new case, it's created with a default label ("Case1", "Case2" etc) as normal, but the label appears in a non-editable text…
asked 2020-09-28T11:54:44.527+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 75%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Richard Swinbank
46
Reputation points
accepted 2020-09-29T20:43:46.327+00:00
Richard Swinbank
46
Reputation points
7 answers
Azure Data Factory: CDM Inline Dataset - Error on parsing datetime field
Hi, I am trying to use the CDM inline dataset feature in the data flow. I could see the data parsed successfully except for the "datetime" fields. All datetime fields are parsed as NULLS though my source CDM data has values in it. I…
asked 2020-08-17T12:14:20.357+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 98%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGP%3C/text%3E%3C/svg%3E)
Guru Prasath Poongodi Kandasamy
16
Reputation points
answered 2020-09-29T11:03:26.867+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 15%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EET%3C/text%3E%3C/svg%3E)
Ellens, Tim
16
Reputation points
1 answer
Zip on prem files before copy to Azure blob
the main thing I want to do is in the title, even better, would be the ability to query a db (oracle, SQL Server) using ADF and self-hosted integration runtime services, zip the output, copy to blob storage. Thanks
asked 2020-09-24T01:21:02.927+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 81%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dave
1
Reputation point
commented 2020-09-29T09:05:17.707+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Copy Different type of file from Gen1 Azure lake to Azure Gen2 lake with attribute( like last updated)
I need to migrate all my data from Azur data lake Gen1 to Lake Gen2. In my lake we have different types of file mixed (.txt, .zip,.json and many other). We want to move them as-it-is to GEN2 lake. Along with that we also want to maintain last updated…
asked 2020-09-20T16:47:33.427+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 62%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Bhuvnesh Kumar
26
Reputation points
commented 2020-09-29T07:10:41.99+00:00
Bhuvnesh Kumar
26
Reputation points
0 answers
ADF - Restart Pipeline - Start from Failure
On failure of a pipeline, I am trying to rerun the pipeline, by providing the run id and startFromFailure as true. Now in my pipeline, I had a foreach activity which processed 3 files - out of which 2 files were successful and 1 file failed. When I…
asked 2020-09-24T08:15:02.753+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 6%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan
941
Reputation points Microsoft Employee
commented 2020-09-28T22:09:59.01+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
copy from json as source to azure database
Copy from json source to azure sql dw fails to load. I am getting the below error message while loading the data to database. I guess I have to make some changes to mappings.. not sure what exactly. Also in mapping tab I cannot use CollectionReference…
asked 2020-09-28T19:21:54.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 9%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Naveen Kuppili
41
Reputation points
commented 2020-09-28T20:53:05.437+00:00
Naveen Kuppili
41
Reputation points
2 answers
One of the answers was accepted by the question author.
weird behaviour of copy data activity in azure data factory
Hi Experts, i have 2 activity which copies the data from table in azure sql db to other azure sql db which is in VNET. config of source sql db is S7800 DTU and that of destination db is S6400 DTU. the amount the data which is transferred in…
asked 2020-09-22T09:03:04.213+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 42%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
n k
106
Reputation points
commented 2020-09-28T19:30:20.96+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
1 answer
ADF Tumbling Window Trigger:Dependency trigger not triggering properly
Hello, I have created a tumbling window trigger for my Azure Data Factory Pipeline Test_Daily with recurrence as 24 hours. For this pipeline, i have added a dependency trigger lets say Test_Hourly (which runs every hour) with offset as 1.00:00:00(1…
asked 2020-09-22T15:54:25.197+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 62%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Prajval Ramesh (Cognizant Worldwide Limited)
1
Reputation point
commented 2020-09-28T12:25:49.943+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Create On Demand HDInsight from Data Factory in an existing VNet
The Azure Data Factory can create an on-demand HDInsight (linked service), which is automatically deleted after the job running on it finishes. Is it possible to have this on-demand HDInsight created in an existing VNet?
asked 2020-09-23T07:07:55.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 4%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Beniamin Murza
26
Reputation points
commented 2020-09-28T12:11:44.993+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Data factory API Trigger runs by order
The Sample Request and Body in the examples does not show how to sort the order. How do you get the latest succeeded run ? https://learn.microsoft.com/en-us/rest/api/datafactory/triggerruns/querybyfactory Managed to filter the successfull runs…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 70%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIH%3C/text%3E%3C/svg%3E)
1 answer
Give all (inclusing new) datafactories access to my keyvault?
I have several datafactories and there will be more. There is one Azure Keyvault. Is there a way to give all datafactories access to this keyvault without specifying each one individually? Through a securitygroup or such?
asked 2020-09-04T13:57:20.503+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 30%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBH%3C/text%3E%3C/svg%3E)
BIP | Henro Veijer
1
Reputation point
commented 2020-09-25T23:35:58.383+00:00
JamesTran-MSFT
36,541
Reputation points Microsoft Employee
1 answer
Azure Log Analytics recursion - Get parent (or parent's parent) pipeline info for Activity in Data Factory
I have a number of data factories hooked up to a log analytics workspace. Currently I am creating a outage dashboard that looks at failed activities. I would like to show the initiating pipeline in this report for activities that failed. However, that…
asked 2020-09-18T16:09:04.547+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 45%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
John Aherne
516
Reputation points
answered 2020-09-25T23:25:35.737+00:00
olufemia-MSFT
2,861
Reputation points
1 answer
Missing Database
I can see my database in SSMS but I am unable to see it in Data Factory. Unfortunately it is the database that I need to get data into. It is in the same subscription, resource group, and on the same server with the same permissions.
asked 2020-09-17T13:41:32.093+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 25%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELT%3C/text%3E%3C/svg%3E)
Lee Taylor
31
Reputation points
commented 2020-09-25T19:40:39.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
2 answers
ADF Copy activity using PolyBase fails to load "Date" type to Synapse
When loading Date type from a parquet file to Synapse with "Date" type via ADF Copy with PolyBase, the ADF throws the following error: { "errorCode": "2200", "message":…
asked 2020-09-24T14:46:46.05+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 15%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Robert Kostecki
6
Reputation points
answered 2020-09-25T17:15:07.9+00:00
Robert Kostecki
6
Reputation points