Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
1 answer
One of the answers was accepted by the question author.
reusable extract pipeline and modular ADF framework
We are delivering an 80TB data migration project for a healthcare client, migrating from on-prem IBM DB2 (via FlashCopy and CDC) to Azure SQL Hyperscale using Azure-native services. The architecture supports one-time historical, catch-up, and real-time…
Azure Data Factory
asked 2025-07-06T09:15:50.3833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 56.00000000000001%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Janice Chi
160
Reputation points
edited a comment 2025-07-10T08:44:19.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
24,750
Reputation points Microsoft External Staff
Moderator
1 answer
One of the answers was accepted by the question author.
Multiple pipeline runs triggered when a file is created
We are creating a file using mssparkutils put method of size 0kb. Just a simple success file. But when this file is written or kept at gen2 location, the trigger which is storage based trigger, is triggered twice due to which same pipleine is triggered…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-07T10:41:09.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 90%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHK%3C/text%3E%3C/svg%3E)
Harshit Kadhao
20
Reputation points
commented 2025-07-10T06:13:13.0633333+00:00
Harshit Kadhao
20
Reputation points
1 answer
GA date for Data Factory Change Data Capcture
When is GA date for ADF Change Data Capture, as now it is in public preview.
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2024-09-13T01:16:17.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 51%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Shiva_G_Rajan_Adm
0
Reputation points
commented 2025-07-10T03:23:06.6366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 40%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Shiva Ganesh Rajan
0
Reputation points
1 answer
Key-pair Authenication snowflake authenication using linked Service
Please I need assistance with resolve the error message i got while connection to snowflake using Key-Pair Authentication. I followed the procedure outlined in both snowflake and Azure Data factory. I'm still getting the same error message see…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-03T12:26:22.5633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 69%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOO%3C/text%3E%3C/svg%3E)
Owhondah Ogundu Raymond
0
Reputation points
commented 2025-07-09T23:32:17.1933333+00:00
Owhondah Ogundu Raymond
0
Reputation points
1 answer
Recommended approach to update reconciliation log tables during DB2-to-Azure SQL Hyperscale migration using ADF + Databricks
We are migrating data from IBM DB2 to Azure SQL Hyperscale using a combination of Azure Data Factory (ADF) and Azure Databricks. As part of our reconciliation framework, we calculate row counts and hash values within Databricks after each major stage…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-04T08:12:46.2533333+00:00
Janice Chi
160
Reputation points
commented 2025-07-09T19:58:55.5866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 50%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkat Reddy Navari
3,625
Reputation points Microsoft External Staff
Moderator
0 answers
ADF Private Endpoint Resource Stuck in 'Provisioning' State
I'm attempting to provisioning a managed private endpoint within ADF Studio. The challenge is that once I created it, it is now taking well over an hour and I think it's stuck in 'Provisioning' state. Any suggestions about how to undo this and get it…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-09T17:12:35.36+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 0%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJV%3C/text%3E%3C/svg%3E)
John V
0
Reputation points
commented 2025-07-09T19:44:08.6666667+00:00
Venkat Reddy Navari
3,625
Reputation points Microsoft External Staff
Moderator
1 answer
ADF pipeline fails on scheduled trigger but same pipeline gets successful when ran on debug mode..
I have pipeline on ADF, which read the file from Azure storage container and loads the data to Databricks tables .. File is avalilable on path , we use triggers for scheduled run of pipleine. When run on scheduled run , pipeline fails with the below…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-09T13:09:36.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 85%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Namreddy, Sirisha
0
Reputation points
answered 2025-07-09T19:05:09.27+00:00
Venkat Reddy Navari
3,625
Reputation points Microsoft External Staff
Moderator
1 answer
Assert failure rows: Output to sink Option Reverts After Publish
I'm encountering a UI issue in Azure Data Factory while configuring a Data Flow activity inside a pipeline. When I uncheck the "Assert failure rows: Output to sink" option and publish the pipeline, the setting appears to be saved. However,…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-08T13:22:00.2866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 57.99999999999999%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYH%3C/text%3E%3C/svg%3E)
Yahya Hussaini
0
Reputation points
commented 2025-07-09T18:34:12.0866667+00:00
Venkat Reddy Navari
3,625
Reputation points Microsoft External Staff
Moderator
0 answers
TNS connection error with Oracle connector 2.0
After upgrading the ADF oracle connector from 1.0 to 2.0, the connection to our oracle database in ADF pipelines has become unstable. In the pipelines, the look up activity used to query specific tables have consistently succeeded. But when the look up…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-09T17:33:15.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 32%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZY%3C/text%3E%3C/svg%3E)
Zhu, Yueli YZ [NC]
280
Reputation points
commented 2025-07-09T18:13:38.99+00:00
Zhu, Yueli YZ [NC]
280
Reputation points
1 answer
How to have cert authentication and OAuth2 in ADF
How can have cert authentication for mSSL handshake, and OAuth2 client ID/Secret in ADF linked service.
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-07T18:22:56.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 67%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bayan, Masoud
0
Reputation points
commented 2025-07-09T15:16:17.2966667+00:00
Bayan, Masoud
0
Reputation points
1 answer
How to delete Salesforce records from ADF
I know that there is no direct way to delete Salesforce records from ADF. But I do know that there is a work around for this where we can get record IDs using Lookup then pass those inside For Loop and then use web activity with DELETE method to delete…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 43%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E8%3C/text%3E%3C/svg%3E)
2 answers
PstgreSQL Linked Service works, but not with debug.
Hi! I am getting this error when trying to use data preview in a dataflow with my dataset which I get from a postgresql db. Spark job failed: { "text/plain":…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-07T10:51:11.6966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 98%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPA%3C/text%3E%3C/svg%3E)
Pedersen, Anders
0
Reputation points
commented 2025-07-09T13:06:49.2666667+00:00
Pedersen, Anders
0
Reputation points
0 answers
Can't get Data Factory to recognize string as Time Stamp
I keep getting the following error when trying to upload data from a csv file to a database: "Operation on target Copy_SR_to_THER47 failed: ErrorCode=TypeConversionFailure,Exception occurred when converting value '2023-01-25 04:51:07' for column…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-03T15:01:12.6633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 46%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
AE90
20
Reputation points
commented 2025-07-09T10:14:51.46+00:00
Smaran Thoomu
24,750
Reputation points Microsoft External Staff
Moderator
1 answer
One of the answers was accepted by the question author.
My tumbling window trigger in Azure data Factory is not running
My tumbling window trigger is not running. It does not have any dependencies. Please see below screen print, even though the scheduled start time has passed trigger status is "Started" it was supposed to start running at 15:00 EST 07/07.…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-08T05:41:56.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 99%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDG%3C/text%3E%3C/svg%3E)
Dattatreya Gandham
20
Reputation points
commented 2025-07-09T09:26:25.0233333+00:00
Amira Bedhiafi
34,336
Reputation points
Volunteer Moderator
0 answers
How to Implement Fuzzy Matching in Azure Data Factory (ADF) with Similarity and Confidence Scoring Like SSIS Fuzzy Lookup?
Hi everyone, In SQL Server Integration Services (SSIS), the Fuzzy Lookup transformation allows us to match input records against a reference table using fuzzy matching logic. It also automatically generates Similarity and Confidence columns for each…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-08T13:32:24.3366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 21%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Priyanka Rani
0
Reputation points
commented 2025-07-09T08:03:58.11+00:00
Priyanka Rani
0
Reputation points
1 answer
How to resolve Budget Export Error from FinOPS HUB Deployment ADF 'error':{'code':'NotFound','message':'The requested export with export name: finops-hub-monthly-costdetails could not be found.'}}
Hi Team, We have integrated FinOps Hub Toolkit tools into our subscription to leverage cost analytics and insights from our budget spending. However, we are seeing a problem with the FinOPS hub deployed ADF exporting budget data into Storage accounts. We…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-07T07:17:57.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 72%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Manisankar Vr
0
Reputation points
commented 2025-07-09T08:00:46.0666667+00:00
Smaran Thoomu
24,750
Reputation points Microsoft External Staff
Moderator
1 answer
Azure Free trial
I started with an Azure free trial, but I didn't understand how to work with ADF at first, so I decided to save my credits for later. However, Microsoft has now declared that my one-month trial period is over, and they are asking me to switch to a paid…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-08T10:13:20.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 19%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGR%3C/text%3E%3C/svg%3E)
Girija Raut
0
Reputation points
commented 2025-07-09T07:45:19.35+00:00
Smaran Thoomu
24,750
Reputation points Microsoft External Staff
Moderator
2 answers
Continuous vs. Phase-Specific Pipelines for Retry and Reconciliation (ADF + Databricks)
In our healthcare data migration project, we are using a modular, control-table-driven architecture for One-Time, Catch-Up, and Real-Time phases. The pipeline processes are divided across Extract, Transform, and Load stages — primarily orchestrated via…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-07T15:29:47.1333333+00:00
Janice Chi
160
Reputation points
commented 2025-07-09T07:44:22.4033333+00:00
Smaran Thoomu
24,750
Reputation points Microsoft External Staff
Moderator
2 answers
One of the answers was accepted by the question author.
Design Error Handling In ADF
Hello, Please help me on how to design Error handling in ADF. I am moving files from Bronze ADLS Gen2 to Silver Azure SQL. So If data in the file contain error more that 20% of total records than reject entire file and move file to reject bucket, if less…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 79%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
1 answer
Unable to undo or commit changes in Fabric Git, also when trying to access Notebooks getting 404 error.
Everything was working fine until yesterday. We normally pause the fabric capacity at night and resume in the morning. When we resumed today morning, there were multiple file changes detected in Fabric Source Control. We tried to access some…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,655 questions
asked 2025-07-07T18:40:58.1833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 47%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Azure Dev
0
Reputation points
answered 2025-07-08T05:11:35.8433333+00:00
Smaran Thoomu
24,750
Reputation points Microsoft External Staff
Moderator