11,582 questions with Azure Data Factory tags

Sort by: Updated
1 answer One of the answers was accepted by the question author.

ADF error 'The object has not yet been initialized'

Hello, We have a pipeline that loads multiple data objects from SAP CDC source which runs okay for the past few weeks. This error only pops up in the last few days. Error: "The object has not yet been initialized." We have asked SAP team, they…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-03T04:03:11.4266667+00:00
Fatehah L 25 Reputation points
edited a comment 2025-06-12T02:59:19.8633333+00:00
Fatehah L 25 Reputation points
2 answers One of the answers was accepted by the question author.

How to fix ClientIPNotAuthorized on ADF

How do I fix the ClientIDNotAuthorized error message? I am on VPN as well.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-05-13T20:02:20.89+00:00
Sherry 20 Reputation points
commented 2025-06-12T00:38:58.1266667+00:00
Pritam Kabiraj 235 Reputation points Microsoft External Staff Moderator
1 answer

Copy activity convert NUMBER to DOUBLE format

Hi we use the ADF copy activity to copy data from an oracle source to azure gen2 container as parquet file. The source data contain data type NUMBER, but the copy activity converts it to DOUBLE format. Could you please provide a solution for this…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-09T19:59:31.38+00:00
Zhu, Yueli YZ [NC] 280 Reputation points
commented 2025-06-11T17:17:40.98+00:00
Chandra Boorla 13,790 Reputation points Microsoft External Staff Moderator
1 answer

How to Capture Row-Level Schema Failures and Redirect Bad Records in ADF During DB2 to ADLS Ingestion

We are ingesting data from an IBM DB2 source into ADLS Gen2 using Azure Data Factory (ADF) Copy activity. The Copy activity is set up with schema validation enabled. During ingestion, if a row violates schema (e.g., data type mismatch), the entire…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-09T14:59:03.35+00:00
Janice Chi 100 Reputation points
commented 2025-06-11T11:53:46.5433333+00:00
J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator
1 answer

Architecture design

Hi friends, thank you for your help as always. My follow up question to below: https://learn.microsoft.com/en-us/answers/questions/2282164/architecture Sorry for silly question. How to get to good hands on the diagraming tools such as Draw IO etc. , I…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-10T13:24:57.7333333+00:00
azure_learner 615 Reputation points
commented 2025-06-11T09:47:19.6466667+00:00
Smaran Thoomu 24,015 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Migration of large tables

we have 800 tables in DB2 and only 25 tables have 90% of data and other 775 tables has10% of data then what will be the best approach while migrating the data first to adlsgen2 and then to hyperscale via DB2-- ADF(SHIR) - ADLSGEN2- DBR(minimal trans) -…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-05-27T07:13:27.2233333+00:00
Janice Chi 100 Reputation points
commented 2025-06-11T03:49:53.1566667+00:00
Smaran Thoomu 24,015 Reputation points Microsoft External Staff Moderator
2 answers One of the answers was accepted by the question author.

ADF Pipeline error: Request content size is too large

I have a pretty basic ADF pipeline that checks a subdirectory for a group of flat files and based on the name of the file, it will load data into a SQL staging table. The ForEach activity has 7 If Condition activities which will run based on the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2021-05-25T15:08:39.773+00:00
Matt Michalowski 31 Reputation points
answered 2025-06-10T22:11:58.58+00:00
Kleber Rebello 0 Reputation points
2 answers One of the answers was accepted by the question author.

Data Factory copy activity is stuck in a "queued" status and not completing the data load

The Data Factory copy activity is stuck in a "queued" status and not completing the data load the same copy activity was running without any issues...now it is goind on queued

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-05-28T18:59:52.84+00:00
mahesh 20 Reputation points
accepted 2025-06-10T18:10:09.7166667+00:00
mahesh 20 Reputation points
1 answer One of the answers was accepted by the question author.

Where to find updates on the Couchbase (Preview) and its release for public use?

I see that there is a preview for a Couchbase connection to copy data from Couchbase to Azure Data Factory. Is there an expected release date for a production connection to Couchbase? Specifically for the use case Copying data to a Couchbase…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-10T15:03:15.76+00:00
Knara, Marcin 20 Reputation points
accepted 2025-06-10T16:07:39.5866667+00:00
Knara, Marcin 20 Reputation points
1 answer

designing a partition-based ingestion strategy

We are currently designing a partition-based ingestion strategy from IBM DB2 (on-prem) to ADLS Gen2 using Azure Data Factory (ADF). Since DB2 is a database and does not have a file system, we cannot use ADF’s Get Metadata activity to estimate partition…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-10T14:07:07.7833333+00:00
Janice Chi 100 Reputation points
answered 2025-06-10T15:17:16.2166667+00:00
J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator
3 answers

JSON to CSV transformation

I would like to below given json data to given csv format using data factory if possible json: { "employee":"123", "bankdetail":[ { "banknumber" : "bank1" }, { …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2023-12-19T17:53:46+00:00
Anonymous
answered 2025-06-10T10:45:35.75+00:00
Anil Peter 0 Reputation points
2 answers One of the answers was accepted by the question author.

Unexpected $500 Billing for Azure Synapse

I activated the Azure Synapse service on August 1st using the default settings. However, I noticed that I am being billed $250 per day, which has accumulated to a total of $500 over the last two days. As a student, I am using this service solely for…

Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
3,525 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,357 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2024-08-03T18:13:18+00:00
Paramjeet Singh 20 Reputation points
commented 2025-06-10T08:37:04.43+00:00
BeresfordBear 0 Reputation points
2 answers One of the answers was accepted by the question author.

Truncated digits for numerical values in Azure Data Factory pipelines

Hello, when I am receiving numerical values within json response from API endpoint, Azure Data Factory is applying scientific notations into numerical values. But in the same time, it's also truncating last digits, see the following example: *Value…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-05-27T08:00:14.6866667+00:00
Stanislav Slovik - Administrator 20 Reputation points
commented 2025-06-10T07:20:39.24+00:00
Stanislav Slovik - Administrator 20 Reputation points
0 answers

Connecting to On-Prem SQL Server through Change Data Capture (ADF) Issue

So I have been trying to get CDC working for the past two days but can't seem to get it to connect to my on-premise SQL Server as it seems to have issues using self hosted integration runtime inside the CDC wizard. I am trying to do incremental loads…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-04T21:17:53.26+00:00
Jason Sherbondy 0 Reputation points
commented 2025-06-09T20:09:33.45+00:00
Chandra Boorla 13,790 Reputation points Microsoft External Staff Moderator
1 answer

Multiple Control tables or single control table

We’re currently working on a large-scale healthcare data migration project where data from on-prem DB2 is being ingested into Azure through a combination of ADF, SHIR, and Databricks. The project supports multiple ingestion modes, including one-time…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-05-23T13:21:43.01+00:00
Janice Chi 100 Reputation points
commented 2025-06-09T18:15:22.5+00:00
Smaran Thoomu 24,015 Reputation points Microsoft External Staff Moderator
1 answer

How should DIU and Degree of Parallelism be configured in ADF when partition sizes vary significantly?

In our project, we are performing large-scale ingestion using Azure Data Factory copy activities. The source data is partitioned with highly variable sizes—some as small as 500 MB and others as large as 3 TB, all coming from the same source system. We…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-09T15:18:20.58+00:00
Janice Chi 100 Reputation points
answered 2025-06-09T16:18:57.63+00:00
J N S S Kasyap 3,300 Reputation points Microsoft External Staff Moderator
1 answer

ADF Linked Service SHIR - Oracle connector 2.0 & TNSNAMES.ORA

Trying to find example on how to configure Linked service using Oracle connector 2.0 to point to TNSNames.Ora file which is available on SHIR server. Documentation is not clear. Can you please send me a link/information on how to make connection using…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-05-30T22:02:37.9533333+00:00
Thakur, Satyajit 0 Reputation points
commented 2025-06-09T14:51:50.6266667+00:00
chaitanya dasari 0 Reputation points
1 answer One of the answers was accepted by the question author.

ADF managed Airflow error : 'ReferenceError: Cannot access 'Z' before initialization'

Hi, I would like to obtain the confirmation that the following error, occurring when creating a new managed Airflow integration runtime in ADF, is caused by the ADF of being new/recent (created in early 2025) and not supporting anymore Airflow as a WOM,…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-06T17:27:49.5133333+00:00
Louis Lalonde 40 Reputation points
edited a comment 2025-06-09T12:13:22.63+00:00
Louis Lalonde 40 Reputation points
2 answers One of the answers was accepted by the question author.

Can we compare the cost of DBR vs DLT via pricing calculator ?

Can we compare the cost of DBR vs DLT via pricing calculator ?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-06T14:19:45.5933333+00:00
Janice Chi 100 Reputation points
accepted 2025-06-09T11:21:44.8733333+00:00
Janice Chi 100 Reputation points
2 answers One of the answers was accepted by the question author.

Airflow Azure Data Factory VNet configuration

Hi, I'm looking to know if it's possible to inject Azure Data Factory (ADF) managed Apache Airflow in a VNet directly from the ADF studio configuration page ? If it's not possible, what is the recommended strategy to obtain this configuration…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,582 questions
asked 2025-06-02T17:55:20.2966667+00:00
Louis Lalonde 40 Reputation points
commented 2025-06-09T09:08:14.36+00:00
Amira Bedhiafi 32,756 Reputation points Volunteer Moderator