Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,161 questions
1 answer
One of the answers was accepted by the question author.
merge rows of same file Azure data factory
Hello I need to merge multiple rows into one row in azure data factory, per example I have the following file PolicyId Driver_name 0001 Adam 0001 Lucy 0002 peter At the end I need to have PolicyId Driver_name1 Driver name2 …
asked 2020-07-03T13:02:35.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 71%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
CORONADO GRANADOS Diana Milena
126
Reputation points
commented 2020-07-05T15:25:16.64+00:00
CORONADO GRANADOS Diana Milena
126
Reputation points
1 answer
One of the answers was accepted by the question author.
How to pull the data from Dynamic 365 to Data lake
<p>Hi <a rel="user" nodeId="74270" href="/answers/users/74270/chiragmishramsft-1092.html">@ChiragMishraMSFT-1092</a> </p> <p>Can you pls help me with the below error:</p> <p>{…
asked 2020-06-30T06:16:02.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2020-07-01T20:59:37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
1 answer
ADF - export to flat file with multiple record types
How to export to flat file with multiple record types in Azure Data Factory from JSON Example output file: Header, 1, H1 Detail1, 1, D11, Test1, 10 Detail1, 2, D12, Test2, 20 Detail2, 1, D21, 100 Detail2, 2, D22, 200 Trailer, 1, T1, 2,…
asked 2020-06-09T19:46:40.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 6%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sam
1
Reputation point
commented 2020-07-01T10:55:23.493+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 76%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
ChiragMishra-MSFT
956
Reputation points
1 answer
ADFV2 Copy Activity Support for more encodingNames
HI I am using the Copyactivity which generate the delimiter text files reading from parquet and applying the respective encodings for the files, we see a list of encoding which are supported by Copy activity (Delimited text format ), and still some…
asked 2020-06-30T22:36:06.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 88%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
ManojMathe
1
Reputation point
answered 2020-07-01T09:58:34.26+00:00
ChiragMishra-MSFT
956
Reputation points
2 answers
Azure Data Factory add Tags in json files
Hello, In the Portal it is possible to add Tags to a Azure Data Factory (I'm not talking about the Annotation in pipelines). But I can't find a way to include them in the ARM Template. I just created a blank ADF, added a Tag and exported the ARM…
asked 2020-06-24T09:24:10.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 51%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHL%3C/text%3E%3C/svg%3E)
Harry Leboeuf
6
Reputation points
answered 2020-07-01T06:39:43.22+00:00
Vaibhav Chaudhari
38,681
Reputation points
2 answers
Data flow error: 4501 - related to WranglingDataFlow
Hi guys, Please, for some reason, I'm getting this error message when I try to debug a pipeline with a wrangling data flow activity. { "errorCode": "4501", "message": "Failed to fetch…
asked 2020-06-04T13:26:01.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 3%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Michael S
1
Reputation point
commented 2020-07-01T01:55:45.807+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 52%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Ashutosh Shukla
1
Reputation point
1 answer
One of the answers was accepted by the question author.
JSON file Node didnt loade in SQL server tables using Azure data factory
HI Team, Below is MY JSON file for SOurce(Blob storage JSON files) I have loading Sql server table using Azure data factory(Copy data activity) in the Mapping tab they Node checkbox I have selected resolutions Checkbox data loaded only…
asked 2020-06-19T12:34:25.987+00:00
Anonymous
accepted 2020-06-30T07:03:03.28+00:00
Anonymous
2 answers
Log shipping
Dear experts I have two node SQL server standard failover cluster. I need to setup single node DR using log Shipping method. Want to ask after switch over to DR how we can setup, plan so that application can connect automatically to DR database…
asked 2020-06-29T05:40:19.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 62%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
mukki mehra
1
Reputation point
answered 2020-06-29T12:55:40.187+00:00
Ronen Ariely
15,106
Reputation points
2 answers
One of the answers was accepted by the question author.
Azure Data Factory Dynamic Content
Hi there, I want to use a variable inside my Data factory Copy activity. I trying to use a variable which I got from an Azure Function. When I run the debug for the variable I can see that I'm getting the value but I can't access it as a…
asked 2020-06-27T10:33:26.767+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 93%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETE%3C/text%3E%3C/svg%3E)
Tahmid Eshayat
286
Reputation points
commented 2020-06-27T14:00:15.667+00:00
Tahmid Eshayat
286
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory Data Copy from Cosmos
We have one scenario, where we have to transfer/copy newly added/updated data from cosmos dB to a SQL Server every 2 hours. Currently, data copy copies all the files, can I configure to only copy the newly added/updated ? Thanks
asked 2020-06-26T03:57:27.08+00:00
Tahmid Eshayat
286
Reputation points
commented 2020-06-27T13:57:22.137+00:00
Tahmid Eshayat
286
Reputation points
1 answer
Data feild comma data impact to csv file creation issue
Hi Team, Currently i have setup a copy activity for a SQL query output .csv file create on the Data Lake destination, But in the one table address level details there have a comma (,) on the inside of the data fields, Due to the that reason, created…
asked 2020-06-19T17:49:07.423+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 19%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mahesh Madhusanka
216
Reputation points
commented 2020-06-26T17:54:55.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,061
Reputation points
1 answer
Doubt with new Azure IR
Hi there. I know this is maybe a trivial question for pretty much all of you, but I am confused with this. To reduce the Pipeline runtime, the administrator has created a new Azure IR. I understand that I need to change the setting on its…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 71%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
0 answers
data bricks scala : data frame column endoing from UTF 8 to windows 1252
HI I am working with data bricks where i have the data in parque and i am generating smaller files out of it , i have a column in this which is string and it has different characters and i have to encode this string value to windows 1252 or windows…
asked 2020-06-22T20:29:26.43+00:00
ManojMathe
1
Reputation point
commented 2020-06-25T22:21:25.563+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
how to replicate ongoing changes of on prem database to data lake?
Like AWS DMS handles ongoing replication and update bucket with respective files, what is available in azure to manage ongoing changes occurring at on-prem sql database? Initially it requires to do a full load in my data lake storage which can be…
asked 2020-06-19T17:52:50.293+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 42%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Seuli Das Majumder
101
Reputation points
accepted 2020-06-25T13:31:24.097+00:00
Seuli Das Majumder
101
Reputation points
2 answers
Extract the date hierarchy from previous data in Data Lake Partition
Hi Team, Would be of great help to get some clear assistance on the below query. Data source is Dynamic365 & SAP Hana. My requirement is to create a data lake partition(ADLS Gen2) with date…
asked 2020-06-07T07:02:45.193+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 96%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
RahulK
101
Reputation points
commented 2020-06-25T07:39:37.837+00:00
ChiragMishra-MSFT
956
Reputation points
1 answer
Datatime format in ADF v2
Hello , I have a requirement wherein I need to convert the below datetime fomat: 2020-06-24T10:30:54.6451783Z into 2020-06-24T10:30:54.64Z meaning yyyy-MM-ddTHH:mm:ss with trailing z. Is there any direct format other than me concatinating the…
asked 2020-06-24T10:56:53.633+00:00
Nandan Hegde
31,436
Reputation points MVP
answered 2020-06-24T23:11:31.483+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
0 answers
Mapping data flow does not load any rows to 1 out of 2 branches
I have this mapping data flow, with two branches from 'RemoveHdlColumns'. When I run this the rows from MergingInput is only loaded through the bottom branch. All the rows are loaded for the 'Select2' transformation, but NO rows hits the…
asked 2020-06-03T10:09:05.363+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 99%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Cedersved
16
Reputation points
commented 2020-06-23T20:54:28.01+00:00
MartinJaffer-MSFT
26,061
Reputation points
2 answers
Designing Table Structure for Accomodating Huge Amount of Data
I am working on a use-case wherein I am expected to receive 80 million records every week and I have to load it in Azure SQL Database, some of the data is dimensional type and some is of fact type, I would be using Azure Data Factory to load the data…
asked 2020-06-09T04:27:29.12+00:00
Pratik Somaiya
4,206
Reputation points
answered 2020-06-23T19:50:50.353+00:00
Mike Ubezzi
2,776
Reputation points
1 answer
Transactions in azure-data-factory
Hello, Am working on one of the adf activity where I am generating the 10 files dynamically with SP outputs using Foreach and Export data. I am trying to implement transactions now where the generation of files should be rolled back if any one of the…
asked 2020-06-02T13:01:44.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 13%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
Purushotham Chanda
1
Reputation point
commented 2020-06-23T19:41:46.333+00:00
MartinJaffer-MSFT
26,061
Reputation points
1 answer
One of the answers was accepted by the question author.
Why do my dataflow pipelines spend 5 minutes in acquiring compute state every time?
I have a bunch of pipelines using dataflows for data transformations each with different sources and sinks. All pipeline exactly takes 5 minutes to spin up a cluster for these dataflows even though they are all triggered almost at the same time. Is there…
asked 2020-05-08T22:50:52.553+00:00
KranthiPakala-MSFT
46,442
Reputation points Microsoft Employee
commented 2020-06-23T13:41:42.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 57.00000000000001%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELS%3C/text%3E%3C/svg%3E)
Lavanya Sreedhar
1
Reputation point Microsoft Employee