Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,257 questions
1 answer
Why does ADF copy activity to Oracle RDBMS configured with concurrent connections and degree of parallelism always fail?
Configuring ADF copy activity to read from Azure Blob Storage and concurrently write to Oracle database (to speed up the copy activity) fails always with ORA-00054 "Resource busy and acquire with no wait error.....". DBA says ADF had X active…
asked 2025-02-18T16:49:47.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 76%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Ajay Devulapalli
0
Reputation points
edited an answer 2025-02-18T21:05:35.59+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
4,085
Reputation points Microsoft Vendor
0 answers
Azure Data factory: Linked service parameterization default values and CI/CD pipelines
I am struggling to understand how ADF parameterization functions when deploying from one environment to another when using DevOps pipelines. Our linked services may be set up with parameters such as "name":…
asked 2025-02-18T15:07:37.1933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 84%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELA%3C/text%3E%3C/svg%3E)
Lockyer, Aaron (SGRE COG B MS SYS BSPM)
0
Reputation points
edited a comment 2025-02-18T20:27:31.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
13,800
Reputation points Microsoft Vendor
1 answer
NAT Gateway for Managed VNet IR
How can I make Azure Managed VNet IR to make requests through a static public I.P? I want to pull data from 'xyz.com/books' api. This API provider doesn't allow requests from everyone. They only allow requests from IPs which are whitelisted. I have…
asked 2025-02-10T22:46:31.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 49%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Amrale, Siddhesh
0
Reputation points
commented 2025-02-18T20:03:47.9533333+00:00
Amrale, Siddhesh
0
Reputation points
1 answer
check for pipe characters in a field and use if condition on this field
In the dataflow, I am checking for data in a field that could have Pipe (|) characters or alphanumeric characters with Pipe(|) between them. If there is only pipe (|)characters, then use iif condition and mark it as Y . If there are alphanumeric…
asked 2025-02-18T17:14:18.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 23%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Peruka, Sikander reddy
90
Reputation points
commented 2025-02-18T19:48:32.3366667+00:00
Ganesh Gurram
4,085
Reputation points Microsoft Vendor
2 answers
ADF Power Query Sink Settings UI Missing Options
Why would my data factory not have the same UI as I see in training videos and QA answers on the Power Query data wrangling part of ADF? I can only see the delete button next to the dataset in the Power Query activity Sink…
asked 2025-01-23T15:50:49.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 24%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EES%3C/text%3E%3C/svg%3E)
Elisa Schrader
20
Reputation points
commented 2025-02-18T19:36:51.0266667+00:00
Elisa Schrader
20
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF Snowflake Connector - Key pair authentication - Failed to parse PEM block containing the private key.
I have created RSA key pair for my snowflake service account. I tested authenticating snowflake with this RSA keys in python. Data from snowflake loaded successfully. Now I'm trying to use this RSA private key for authentication in ADF. I have followed…
asked 2024-04-22T20:11:10.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 36%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Md, Muzaffar
25
Reputation points
commented 2025-02-18T18:59:47.15+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 27%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
PK
20
Reputation points
1 answer
Additional cost for calling Data Factory API
Hi all, Currently we are going to add an Data Factory API call to the Data Factory pipeline. We have a pipeline that run every 10mins and we would like to add a Web activity that call the Data Factory queryPipelineRuns API.…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 87%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
1 answer
Within Azure Data Factory Studio, I get a Failed Connection to my SQL Database
Below is the error I keep getting, I've read all documentation and tried all steps to ensure identification is all there. I'm stuck and have a client deliverable. I would happily pay extra to get over this hurdle as I need to use the data warehouse…
asked 2025-02-14T00:58:25.4166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 28.000000000000004%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDL%3C/text%3E%3C/svg%3E)
Derek Lilley
20
Reputation points
commented 2025-02-18T15:54:51.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
8,470
Reputation points Microsoft Vendor
2 answers
Azure data factory calling restapi hosted on app service
I am calling restapi POST method hosted on app service from azure data factory(web activity) using VNet integration runtime, I am using authentication type as NONE. but it is giving error as Error "403 Forbidden 'the web app you have attempted to…
asked 2025-02-08T05:15:33.36+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 35%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Pankaj Joshi
351
Reputation points
commented 2025-02-18T15:42:10.05+00:00
Chandra Boorla
8,470
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Need help to copy Data from Snowflake to Azure Blob Storage via ADF Pipeline
I want to create an Azure Data Factory pipeline to perform a bulk data copy from Snowflake tables to Azure Blob Storage. Since I have large tables to handle, I aim to organize the data into folders named after each table. Inside each table’s folder, the…
asked 2025-02-14T17:26:39.2166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 67%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENB%3C/text%3E%3C/svg%3E)
Nalini Bhavaraju
0
Reputation points
accepted 2025-02-18T14:58:52.71+00:00
Nalini Bhavaraju
0
Reputation points
1 answer
Running MongoDB Aggregation Operation in Azure Data Factory.
Hi, I want to query my MongoDB Atlas (Not Cosmos DB) to fetch data and process it in Azure Data Factory. I have successfully created a connection to MongoDB and can run filter operations. I want to perform aggregation operations like $lookup with the…
asked 2025-02-18T05:05:42.0433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 68%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anish Sapkota
0
Reputation points
edited a comment 2025-02-18T14:50:58.48+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 99%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
J N S S Kasyap
0
Reputation points Microsoft Vendor
1 answer
Sudden slowness on Azure Datafactory pipelines while fetching data from oracle database
Hello, We have been running ADF pipelines to ingest data from Oracle database. Suddenly it started taking 7-8 hours to finish whereas earlier it was 2-3 hours. No changes on Network end. Self hosted integration runtime was upgraded on 18th Jan-2025 where…
asked 2025-02-18T09:52:26.3766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 49%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHD%3C/text%3E%3C/svg%3E)
Hiranya Deka
0
Reputation points
answered 2025-02-18T14:10:14.7733333+00:00
Vinodh247
28,136
Reputation points MVP
1 answer
One of the answers was accepted by the question author.
What is the cost of failed ADF(Azure Data Factory) Pipeline
I'm getting cost optimization recommendations from Azure Advisor stating that there are failed ADF pipelines. It suggests that deleting them can help save costs. Could you please specify how much I can save by deleting the pipeline?
asked 2025-02-11T10:15:39.06+00:00
Navin Prasad Kumar
81
Reputation points
commented 2025-02-18T12:34:32.7966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
20,295
Reputation points Microsoft Vendor
2 answers
Azure Data Factory & ServiceNow V2 connector - Schema import failed
Hi, I have started to explore using the ServiceNow V2 Connector in ADF to replace an existing on-prem process using the ServiceNow ODBC driver. I have created Datasets for some of the tables, however when creating the Dataset for the 'Incident' table,…
asked 2025-02-13T15:01:53.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 87%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Smith, Darren (IT-LON)
0
Reputation points
answered 2025-02-18T12:29:02.2233333+00:00
Smith, Darren (IT-LON)
0
Reputation points
0 answers
ADF Copy Activity for JSON filesfails to Locate Blob Despite Successful Metadata Retrieval
I’m currently working on an Azure Data Factory (ADF) pipeline that consolidates files from multiple source containers into a single target container. The goal is to gather one file type (lod2) from various resource-based subfolders and versioned folders…
asked 2025-02-17T23:18:32.66+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 10%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDG%3C/text%3E%3C/svg%3E)
Denktas, Gökalp Yigit
0
Reputation points
commented 2025-02-18T12:08:44.96+00:00
Denktas, Gökalp Yigit
0
Reputation points
2 answers
ADF and Azure ML Integration
Hello, In Azure Data Factory (ADF), we have Development (DEV), Staging (Stage), and Production (PROD) workspaces. Similarly, Azure Machine Learning (Azure ML) also has corresponding DEV, Stage, and PROD workspaces. In ADF DEV, I have an Azure Machine…
asked 2025-02-14T09:02:58.9733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 63%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sama Praveen Kumar Reddy
20
Reputation points
accepted 2025-02-18T11:49:59.2333333+00:00
Sama Praveen Kumar Reddy
20
Reputation points
2 answers
Issue with New Snowflake Connector in Power BI Desktop (January 2025 Release)
Issue with New Snowflake Connector in Power BI Desktop (January 2025 Release) 37m ago Hi everyone, Has anyone experienced issues when testing the new Snowflake Connector in the January 2025 release of Power BI Desktop? Version: 2.139.1678.0 (64-bit) –…
asked 2025-02-17T12:30:30.5966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 28.000000000000004%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Antonio Rizziello
0
Reputation points
answered 2025-02-18T11:31:31.23+00:00
Ganesh Gurram
4,085
Reputation points Microsoft Vendor
1 answer
How to Get Data from an API Using Azure Data Factory (ADF) Without Dataflows and Handle Cursor-Based Pagination
Hi, I'm working with Azure Data Factory (ADF) and need to fetch data from an API that uses cursor-based pagination. I want to: Make an initial API call to get the first set of data. Apply pagination using the cursor field to fetch subsequent pages. …
asked 2025-02-17T13:10:14.1266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 28.999999999999996%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOM%3C/text%3E%3C/svg%3E)
Onkar More
0
Reputation points
commented 2025-02-18T11:25:34.72+00:00
Ganesh Gurram
4,085
Reputation points Microsoft Vendor
1 answer
ADF Pipeline Debug and Resume after Breakpoints
I have a ADF Pipeline (Pipeline1) which has around 7 activities. Inside the ADF Pipeline we have an Execute Pipeline, which executes Pipeline2 I am putting a Debug Point inside the Pipeline1 before calling Pipeline2 and also inside Pipeline2 we have…
asked 2025-02-13T18:39:57.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 59%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIN%3C/text%3E%3C/svg%3E)
Immaneni, Nanda
0
Reputation points
answered 2025-02-18T11:21:35.5866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 57.00000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rakesh Govindula
0
Reputation points Microsoft Vendor
1 answer
ADF: copy activity, ordinal mapping and Additional columns on source side.
Hi, I have a pipeline with a copy data activity. I would like to do an ordinal mapping, the source has all the columns needed on the sink side but I also want to add 2 Additional columns on source side which have dynamic content. something comes up…
asked 2025-02-14T14:20:13.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 79%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
eg0087
0
Reputation points
commented 2025-02-18T11:21:31.7733333+00:00
Ganesh Gurram
4,085
Reputation points Microsoft Vendor