Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,528 questions
1 answer
Azure Data Factory - How do we get source and destination of Azure Data Factory pipeline via .net code?
Azure Data Factory - How do we get source and destination of Azure Data Factory pipeline via .net code?
asked 2022-03-10T15:59:17.363+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 51%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
Sekar Nagaraj
1
Reputation point
commented 2022-03-16T16:44:46.37+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to get the activity name by dynamically in the adf
I’m trying to capture the logs with error and store it in ADLS? I don’t have sql and store procedure. In the log files the pipeline name is coming as logging pipeline which is not correct, I should display activity name like profile data load success ,…
asked 2022-03-16T13:38:10.523+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 7.000000000000001%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHL%3C/text%3E%3C/svg%3E)
HemanthKumar loka
226
Reputation points
accepted 2022-03-16T14:03:13.103+00:00
HemanthKumar loka
226
Reputation points
2 answers
One of the answers was accepted by the question author.
Change Region Azure Data Factory
Hi, I created an ADF in the region Germany West and now notice that Data Flows aren't available in this region. Would it be possible to change the region of my ADF or do I have to create the same Data Factory in the correct region?
asked 2021-05-10T10:10:31.613+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 87%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDV%3C/text%3E%3C/svg%3E)
David van der Eng
26
Reputation points
commented 2022-03-16T13:19:54.103+00:00
Nandan Hegde
29,886
Reputation points MVP
1 answer
One of the answers was accepted by the question author.
dataflow - branch - collect all fields to go to sink
Hello, in the screenshot below, I am getting data from .json file... flatten24 has a-lot of the fields. Now I would like to get other fields from flatten23,22,21 so that I have them all to put into sink. How do I do this please? Thank…
1 answer
One of the answers was accepted by the question author.
dataflow - joining flattened fields
Hi, I am flattening a json file data... I am using flatten activity for each array inside a parent.... Basically, when I am happen with the flatten activies and I can see the columns coming through in each flatten activity, how do I put these…
1 answer
One of the answers was accepted by the question author.
expression
Hello, What is wrong with this expression please? I think it requires single quote around surename? Note that I do not get an expression error but when I run this in pipeline copy sink I get an error. When I narrow it down it looks like…
1 answer
Transpose File using ADF Mapping DataFlow
I have a csv file data like below : ![182916-image.png][1] I want to achieve data like below : ![182867-image.png][2] Here first 4 columns have same row values and last 4 columns have separate row values. Please help me how to achieve this using ADF…
asked 2022-03-14T14:31:10.893+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 17%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Arundhati Sen
151
Reputation points
commented 2022-03-16T10:46:27.253+00:00
Arundhati Sen
151
Reputation points
2 answers
How to raise a bug in azure data factory
I would want to report a finding on Azure data factory. May I know what is the procedure for it
asked 2022-03-02T15:08:26.48+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 16%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Madhana Kumar
6
Reputation points
commented 2022-03-16T10:21:43.377+00:00
PRADEEPCHEEKATLA-MSFT
76,921
Reputation points Microsoft Employee
2 answers
Incremental Load in Azure Data Factory
Hi, Requirement: Fetch data from on-premise using ADF and load it into Synapse incrementally. Problem: We don't have a unique value column to perform upsert taking that column as the key. There is a 'change_date' column indicating the last modified…
asked 2022-03-02T08:06:53.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 48%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E3%3C/text%3E%3C/svg%3E)
39861377
141
Reputation points
commented 2022-03-16T06:26:51.28+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee
0 answers
Unable to copy Delta Lake data to Synapse Dedicated SQL Pool with Azure Data Factory
When trying to use data factory to copy data to a Synapse Dedicated SQL Pool Table I received the following error: ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details:…
asked 2022-02-14T17:11:11.963+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 71%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGK%3C/text%3E%3C/svg%3E)
Gary Knowles
1
Reputation point
commented 2022-03-16T05:36:39.913+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Unpivot and Summarize Data Flow in ADF
When making sense of a new data set I often want to perform some basic transformations to get an understanding of which columns have limited selections or are largely empty so I often perform the following basic transformation. These steps are…
asked 2022-03-15T14:33:38.62+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 85%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWP%3C/text%3E%3C/svg%3E)
Walter Pelowski
66
Reputation points
commented 2022-03-15T22:49:20.65+00:00
MarkKromer-MSFT
5,186
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
distinct value of column
Can you please see if what I am doing here correct to get the distinct value of column? I get the correct answer but not sure if this is the way to do it. basically what does this do? name != "venture" Thank you
2 answers
One of the answers was accepted by the question author.
Strange errors ADF
Hello, We encountered some strange errors in our Azure Data Factory pipeline runs. One is from a Notebook activity as seen in the following screenshot: Another error is from the Notebook activity as well: A third error occurred…
asked 2022-03-09T16:47:50.697+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 52%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECD%3C/text%3E%3C/svg%3E)
Chi, Dave
61
Reputation points
commented 2022-03-15T17:02:49.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,021
Reputation points
1 answer
flatten json data
sample json data: ... "parcels": [ { "name": "kljkl", "state": "fgfgf", "quantities": [ { …
asked 2022-03-13T22:49:42.733+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
arkiboys
9,616
Reputation points
commented 2022-03-15T17:00:25.97+00:00
MartinJaffer-MSFT
26,021
Reputation points
1 answer
One of the answers was accepted by the question author.
Incremental Changes from Integration Runtime to Data Factory
Hi- I am attempting to load data from an on-prem sql server to azure data factory (data lake). I will be scheduling the upload once per day. How do I keep the data together, but still have a way to separate the days? Basically I do not want to…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 17%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
2 answers
At export step for Azure data factory pipeline getting error "at Sink ""Exportcustomerss": Java.Lang.NullPointerException
At export step for Azure data factory pipeline getting error "at Sink ""Exportcustomerss": Java.Lang.NullPointerException I can see one record is available for export to azure data lake file. In Sink setting Sink Type is…
asked 2022-02-24T15:23:17.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 78%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Shahid Bashir
1
Reputation point Microsoft Employee
commented 2022-03-15T13:55:12.763+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Is data encrypted when using Self-Hosted Integration Runtime?
Hi- I am simply looking to know if data transfer are encrypted when using Self-Hosted Integration Runtime? I can't find it in any documentation, I only see references to encrypting credentials. Thanks!!
1 answer
Synapse data flow: DFExecutorUserError (User configuration issue)
I have created data flow which transforms one parquet files into other parquet files (I would like to change schema). Both source and sink is configured as ADLS2. As input I have multiple files and some of them are processed correctly but couple of…
asked 2021-10-13T15:49:17.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 35%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
lbedn
1
Reputation point
answered 2022-03-14T23:55:28.103+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 90%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYV%3C/text%3E%3C/svg%3E)
Yash Vijay Mehta
1
Reputation point
1 answer
The current user has exceeded the maximum number of concurrent queries allowed by the server.,Source=Microsoft Salesforce ODBC Driver
Hi team, we are running SQL query on Salesforce as a source but the activity is getting failed with this error - Why it is failing? what is the root cause and solution for it?
asked 2022-03-08T15:42:23.797+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 89%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sarvesh Pandey
71
Reputation points
commented 2022-03-14T22:46:54.937+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
0 answers
How to do full load of BSEG table from SAP ABAP oracle environment using SAP table connector
I'm new to SAP. Can anyone let me know how to implement full load for the BSEG table from SAP ABAP oracle environment using SAP table connector into Azure. As this table has huge volume of data , which field can be used to pull the historical data .
asked 2022-02-22T17:33:56.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 99%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DR
1
Reputation point
commented 2022-03-14T18:08:01.237+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee