11,672 questions with Azure Data Factory tags

Sort by: Updated
1 answer

Azure Data Factory Copy Activity Fails – Azure IR Pre-login Handshake Timeout Across Regions

Due to ill-supported support from within Azure, I try getting answers along this way: We are encountering persistent failures in Azure Data Factory copy activities when writing to Azure SQL Database using the Azure Integration Runtime (both AutoResolve…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-20T10:14:57.5433333+00:00
Gerrit Van Even 0 Reputation points
commented 2025-06-25T12:59:09.8066667+00:00
J N S S Kasyap 4,085 Reputation points Microsoft External Staff Moderator
1 answer

Azure Data Factory trigger event not working while uploading using .NET client library

I want to trigger an ADF pipeline on blob creation, file based trigger. The manual upload to blob storage triggers the ADF pipeline correctly. But when i am uploading the same via AzureDataLakeStore .Net libraries from a web app, specifically package - …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-02-25T22:26:52.95+00:00
NATASHA TAHARIYA 0 Reputation points
commented 2025-06-25T09:21:05.95+00:00
Akshaya Sivanantham 5 Reputation points
3 answers

ADF COPY ACTIVITY FAILS DUE TO TIME DATATYPE COLUMN

I am trying to copy data from SQL Server to Snowflake using ADF Pipeline, I am using a copy activity, my source and destination column data types all match , my source column "LastUpdateTime" in SQL Server is a TIME data type . My destination…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2022-06-20T21:14:00.613+00:00
Shaista Lubna 1 Reputation point
answered 2025-06-25T02:25:57.8266667+00:00
Michael Hanna 0 Reputation points
1 answer

upgrade the Amazon RDS for Oracle connector v2 is not working

I am trying to upgrade existing linked service for Amazon RDS for Oracle to v2. When I try to publish the linked service, below properties are empty. Can someone help me on this? I am using below json template for the linked service. { "name":…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-19T05:43:54.0533333+00:00
Ashok Sharma 0 Reputation points
commented 2025-06-24T15:58:35.37+00:00
Smaran Thoomu 25,115 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

How Does Azure Data Factory Dynamically Decide DIU and Parallel Copy Based on File Count vs. File Size?

We are designing a high-throughput ingestion pipeline using Azure Data Factory (ADF) and would appreciate clarification regarding how Data Integration Units (DIU) and Degree of Parallelism (parallelCopies) are determined automatically. As per your…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-23T15:07:23.68+00:00
Janice Chi 320 Reputation points
accepted 2025-06-24T12:26:15.12+00:00
Janice Chi 320 Reputation points
2 answers

Connecting to On-Prem SQL Server through Change Data Capture (ADF) Issue

So I have been trying to get CDC working for the past two days but can't seem to get it to connect to my on-premise SQL Server as it seems to have issues using self hosted integration runtime inside the CDC wizard. I am trying to do incremental loads…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-04T21:17:53.26+00:00
Jason Sherbondy 0 Reputation points
commented 2025-06-23T18:58:37.2633333+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Azure Data Factory (ADF) Managed Batch Account Endpoint issues.

I am attempting to integrate Azure Data Factory with Azure Batch using a private endpoint, specifically through a Managed IR. I created a Managed Vnet, then used az cli to attempt to connect a batch account using a managed private endpoint: az…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-20T14:02:30.27+00:00
Ahmed Said Youssef 20 Reputation points
accepted 2025-06-23T18:01:52.5466667+00:00
Ahmed Said Youssef 20 Reputation points
1 answer

what time is JOB_DURATION for

is JOB_DURATION representing until run start time or until run end time?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-19T16:50:02.2366667+00:00
Zhou, Min (NonEmp) 0 Reputation points
commented 2025-06-23T16:48:02.63+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer

Help with Auto Update Issue on Self-Hosted Integration Runtime (Error Code: 10003)

I'm running into an issue with the auto-update of the Self-Hosted Integration Runtime (SHIR) agent in Azure Data Factory. When I try to manually update the agent from the Data Factory Studio, I get the following error: Download failed: Download…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-19T14:04:53.0733333+00:00
Hardavinder Bhinder 20 Reputation points
commented 2025-06-23T08:13:22.4+00:00
J N S S Kasyap 4,085 Reputation points Microsoft External Staff Moderator
1 answer

How to fetch Schema of a file stored in Azure Blob using ADF

Hi, I'm sending data from Azure Blob to AWS S3 (As AWS S3 isn't supported as sink dataset in Copy Data Activity I'm using Mapping Dataflows). I need to write the schema to the sink, so my pipeline has to write data in one directory and schema file in…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-04-03T07:47:02.5266667+00:00
Madan M R 0 Reputation points
commented 2025-06-23T05:25:53.7566667+00:00
Madan M R 0 Reputation points
1 answer

Guidance on Best Practices for Loading CDC-Merged Data from ADLS to Azure SQL Hyperscale

As part of our healthcare data modernization initiative, we are currently implementing the Catch-Up phase of a large-scale CDC pipeline. The architecture involves: All Insert/Update/Delete logic (CDC merge) being handled in the Databricks branch layer…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-21T09:23:23.7866667+00:00
Janice Chi 320 Reputation points
commented 2025-06-23T01:59:55.5733333+00:00
Smaran Thoomu 25,115 Reputation points Microsoft External Staff Moderator
2 answers

Azure data factory @concat usage..please help

HI, i am trying to call multiple tables from one database such that it should fetch only current week data to sink and i am trying to use where condition in source copy activity where i am defining below query... NOTE: item().source is used to get…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2021-12-02T16:04:13.377+00:00
Vanipriya K 6 Reputation points
commented 2025-06-20T16:31:07.75+00:00
IAZ 0 Reputation points
1 answer

adf split of columns

I have a column like id,job description,etc in a single column how to make it separate column in adf

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-18T16:31:07.7833333+00:00
Bitterlein Konnoth Biju [student] 0 Reputation points
commented 2025-06-20T16:16:21.46+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Deadlock Issue

I am facing deadlock issue in the ADF pipeline. Will enable the retry option in the settings 3 times and the interval 5 mins will solve the problem? I am getting the deadlock issue occasionally only not on regular basis.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-13T15:57:02.2733333+00:00
Avinashkumar Manivannan 20 Reputation points
commented 2025-06-20T15:35:35.22+00:00
Younes 0 Reputation points
2 answers One of the answers was accepted by the question author.

Handling Files and Data with Azure Data Factory

I have created a data pipeline using ADF which processed data from ADLS Gen2 pushed by SAP system and after transformation persists to Azure SQL. Please help me in designing on:- How ADF ensure the files is already processed or not processed. I want…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-20T13:25:53.2166667+00:00
Relay 200 Reputation points
accepted 2025-06-20T15:10:13.7166667+00:00
Relay 200 Reputation points
1 answer One of the answers was accepted by the question author.

Cannot add workarounds=’ 536870912’ connection property to Oracle linked service version 2.0

According to Microsoft we must upgrade azure oracle linked services from version 1.0 to 2.0. My oracle linked service version 1.0 has an additional connection property named "workarounds" see the picture attached. How to incorporate the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-16T16:34:33.25+00:00
John Balta 20 Reputation points
accepted 2025-06-20T13:09:14.5633333+00:00
John Balta 20 Reputation points
2 answers

How to export full Azure Data Factory ARM template (pipelines, datasets, dataflows, linked services) via API/CLI?

I’m trying to automate the export of Azure Data Factory (ADF) ARM templates as part of a CI/CD pipeline. The goal is to get a complete export containing pipelines, datasets, dataflows, linked services, and other artifacts - without manually using the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-18T00:15:14.16+00:00
Napsty 45 Reputation points
commented 2025-06-20T07:58:20.7533333+00:00
Krupal Bandari 770 Reputation points Microsoft External Staff Moderator
1 answer

Best Practice for Handling Multiple Excel Files with Different Schemas and Schedules in Azure Data Factory

I’m working on an Azure Data Factory (ADF) solution where I receive four different Excel files daily as input data for order processing. Each of these files has a unique and fixed schema that will not change over time. In terms of scheduling: Two files…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-19T02:31:14.0166667+00:00
YERNAIDU SIRAPARAPU 45 Reputation points
commented 2025-06-20T06:40:08.2733333+00:00
Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator
1 answer

Sharepoint List extract error from ADF

Dear Support, I am reaching out regarding an issue we are encountering while extracting data from several SharePoint lists using Azure Data Factory (ADF). We have been able to successfully extract 10 out of 14 lists without any problems. However, when…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-17T11:29:13.34+00:00
ABEL ANTON RODRIGUEZ 0 Reputation points
commented 2025-06-20T05:39:03.6366667+00:00
Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator
1 answer

Azure Data Factory - IP is not from a specified range

Hi. I am trying to whitelist my Azure Data Factory Azure Integration Runtime, but the IPs that it is utilizing do not come from DataFactory.WestEurope service tag, but from AzureCloud.WestEurope. The service I'm trying to connect is using IP whitelisting…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-17T14:10:57.3266667+00:00
Ignacy Jakowiecki 6 Reputation points
commented 2025-06-19T18:29:51.7033333+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator