Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,109 questions
1 answer
Look up in adf
Hi, What is the difference between data flow look up transformation and normal look up transformation.. Any screen appreciated
2 answers
One of the answers was accepted by the question author.
Function in adf
Hi what are the functions in adf Pls help me with screenshots
asked 2024-07-17T03:51:27.6433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
385
Reputation points
answered 2024-07-17T05:15:15.9666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 46%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
NIKHILANETHIKUNTA
75
Reputation points Microsoft Vendor
1 answer
ADF-Global parameters inside Linked Service
Hi all! My company is starting to make use of DevOps pipeline to share the same code in all the productive and not productive envirorments. Before this, everything was wild. For example, the linked service for the Azure KeyVault has the url-link written…
asked 2023-10-01T08:53:07.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 39%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDA%3C/text%3E%3C/svg%3E)
Dario Antolini
21
Reputation points
commented 2024-07-17T05:05:15.74+00:00
Vinodh247
13,146
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory Copy Activity convert sql string in to jobject
I have a query from Azure SQL database which returns jsonstring like below and want to insert in to Cosmos db as JSON object (not string) "{\"type\":…
asked 2024-07-12T11:22:21.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Vijay Rudrasamudram
40
Reputation points
accepted 2024-07-17T04:51:19.14+00:00
Vijay Rudrasamudram
40
Reputation points
1 answer
Config table in adf
How we can load table from config table In da pipeline using copy activity.. Help me with screenshot
0 answers
Copy activity failed: Failure happened on 'Source' side
Trying to get data from PGSQL to blob using copy activity give below error after query runs for ~30 min Operation on target Copy Customer failed: Failure happened on 'Source' side.…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 65%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
2 answers
Integration runtime setup, Manual setup: Download and install integration runtime Page not found
I am writing to seek assistance regarding two issues I am encountering with Azure Data Factory Integration Runtime setup. Express setup Internal Server Error: Whenever I attempt to select "Option 2: Manual setup" and proceed to "Step…
asked 2024-07-16T16:50:34.3666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 57.99999999999999%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGA%3C/text%3E%3C/svg%3E)
Gulam Azeemuddin
0
Reputation points
edited the question 2024-07-17T03:31:20.3266667+00:00
PRADEEPCHEEKATLA-MSFT
85,031
Reputation points Microsoft Employee
1 answer
Upsert operation in azure data factory
We want to copy data from production and send it to dev env daily. As a solution, we re thinking to create a pipeline in Azure data factory to fetch all data from source database, scramble PII information for a few columns and send it to a destination…
asked 2024-07-12T12:10:45.25+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 27%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERJ%3C/text%3E%3C/svg%3E)
Riya Jain
0
Reputation points
commented 2024-07-16T23:17:16.4966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
28,946
Reputation points Microsoft Employee
1 answer
How does ADF Linked Service for Azure Databricks aligns with Job Compute Policy defined inside Azure Databricks
This is a two-part Question. First Part Context: I have ADF which contains several data pipelines. Some pipeline also includes a databricks notebook as an activity. I have created Linked Service Azure Data Factory to facilitate the pipeline which…
asked 2024-07-10T10:25:16.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 26%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Ravineesh
20
Reputation points
commented 2024-07-16T23:14:23.99+00:00
Bhargava-MSFT
28,946
Reputation points Microsoft Employee
3 answers
How to generate ETL pipeline for Python code which gives 2 JSON file in response
Requirement : I have python code in my local pc folder which returns 2 JSON file as a output (that code is using API and try to store the data of website in JSON file), I want to create automation for running that python code at particular time using…
asked 2024-07-13T15:19:01.0733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 80%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUM%3C/text%3E%3C/svg%3E)
Utsav Mori
20
Reputation points
answered 2024-07-16T16:50:56+00:00
NIKHILANETHIKUNTA
75
Reputation points Microsoft Vendor
1 answer
RFC table options
Hi, i´m using azure data factory and copy step to copy data from sap. To shrink data volume i´ve tried to use the rfc table option in combination with a pipeline variable. This fails with sap error message, that it cannot parsed, when i hardcode the…
asked 2022-12-14T13:05:50.477+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 1%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Müller, André
71
Reputation points
commented 2024-07-16T14:57:28.1966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 49%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Anudeep Burli
0
Reputation points
2 answers
One of the answers was accepted by the question author.
For each count dynamic batch count
I have a scenario in which i need to have provide dynamic batch count in for each loop f. Is there any way to do it. RIghtnow , I don't see any option to give it. We need to give it as hardcoded in pipeline.
asked 2022-01-22T05:48:21.423+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 48%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Aditya Raj
286
Reputation points
commented 2024-07-16T14:40:37.2733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 25%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEN%3C/text%3E%3C/svg%3E)
Ext Nestor Penate (Admin)
0
Reputation points
2 answers
One of the answers was accepted by the question author.
how to read excel file data (stored in adls gen2 ) in Azure Data Factory pipeline
I have source excel file which is coming as showing in the format . Please help how to read excel file with that specified format . Please help . Thank You.
asked 2024-07-15T06:39:57.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 88%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Bathini, Shirish
20
Reputation points
accepted 2024-07-16T13:58:19.0633333+00:00
Bathini, Shirish
20
Reputation points
1 answer
ADF Debug pipeline : Use activity runtime
What is the difference between 'Use data flow debug session' and 'Use activity runtime' in debug in Azure Data Factory? I have a pipeline with a Lookup activity followed by a ForEach activity. A lookup file is used to store parameters, and this file is…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 26%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
2 answers
adf data flow acitivity stuck in queued status when the pipelines run
when I trigger the pipelines, it got stuck in the data flow activity as queued status
asked 2024-05-07T15:40:39.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 93%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
Burcu
5
Reputation points
commented 2024-07-16T11:10:20.7166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Data Factory's Data Flow cost structure
Hi all, I have a question regarding the cost structure for Data Factory's Data Flow. Is the charge solely per cluster/integration runtime, or are the costs doubled if two pipelines run simultaneously, I mean, both using the same cluster/integration…
asked 2024-07-10T17:56:32.1066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 11%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERN%3C/text%3E%3C/svg%3E)
Renan Nascimento
20
Reputation points
accepted 2024-07-16T09:48:50.3933333+00:00
Renan Nascimento
20
Reputation points
1 answer
How do i refrence my dataset parameter in my base url in azure copy rest activity
Good day, Is there a way i can reference my dataset parameter in my rest copy activity on the base url, am trying to create a dynamic url:
asked 2024-07-13T10:37:57.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 28.999999999999996%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOA%3C/text%3E%3C/svg%3E)
Osamwonyi Alekwe
40
Reputation points
commented 2024-07-16T09:10:42.8833333+00:00
Smaran Thoomu
12,445
Reputation points Microsoft Vendor
1 answer
Issues with mi IP address
Hi guys. I am doing a course about Azure and I was checking the section to connect a Blob Storage to SQL Database. When I was setting up my Firewall rules, the first issue I found was that Azure didn't recognize my IPv4 but IPv6 instead so I have to…
asked 2024-07-15T07:53:30.32+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 16%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Reinaldo Munoz
0
Reputation points
commented 2024-07-16T09:00:12.8166667+00:00
AnnuKumari-MSFT
32,011
Reputation points Microsoft Employee
2 answers
Azure ADF Copy activity faild
ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'xxxxx'. FileSystem: 'xxxxxx'. Message: 'The…
asked 2024-07-15T09:35:01.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 81%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMH%3C/text%3E%3C/svg%3E)
Mohammad Hijazi
0
Reputation points
commented 2024-07-16T08:53:55.5533333+00:00
AnnuKumari-MSFT
32,011
Reputation points Microsoft Employee
1 answer
How to deploy user assigned managed identity?
I have a linked service (which is connected to Dataverse) in Azure Data Factory which is being authenticated by user-assigned managed identity. I have created three UAMI for three different environments - Dev, UAT and PROD. The name of the UAMI are -…
asked 2024-07-15T10:49:17.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 4%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED0%3C/text%3E%3C/svg%3E)
DevanshAgarwal-0149
0
Reputation points
edited a comment 2024-07-16T08:26:15.0733333+00:00
DevanshAgarwal-0149
0
Reputation points