11,672 questions with Azure Data Factory tags

Sort by: Updated
1 answer

I have added parameter in Dataset but it is not reflecting in copy activity

I have added parameter in Dataset but it is not reflecting in copy activity while coping all sql tables form azure sql to blob storage , created 2 parameter (schema,table) for pass lookup output values

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-30T07:39:03.91+00:00
Rupesh Boiwar 0 Reputation points
commented 2025-07-01T16:26:38.2366667+00:00
Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator
2 answers

How to make the Chekpoint Key / SAP to stage subscriber process a static value?

Context: I am currently building a pipeline that extracts data from a SAP CDC source, where I want to utilize the delta mechanism this connector enables. My setup is: SAP CDC connector to source system (SAP CRM)- static subscriber name and ODP…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2023-12-09T11:16:12.3066667+00:00
Georgi Ivanov 0 Reputation points
answered 2025-07-01T13:51:58.1933333+00:00
Bröthaler Thomas 0 Reputation points
1 answer One of the answers was accepted by the question author.

Compatibility and Support for Amazon Corretto in Azure Data Factory

Hello Azure Support Team, We are exploring the possibility of replacing Oracle Java with Amazon Corretto (OpenJDK distribution) in our Azure Data Factory (ADF) environment. Before proceeding, we would like to clarify the following: Compatibility: Is…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-23T13:27:38.9433333+00:00
Naveen Prasanth 20 Reputation points
accepted 2025-07-01T10:38:33.73+00:00
Naveen Prasanth 20 Reputation points
2 answers One of the answers was accepted by the question author.

Issue with "Too Many Requests" Error in App Provisioning

I ran the code in the test file within Synapse and observed a "Too Many Requests" error while processing records for App provisioning. We also confirmed this issue via Postman, where the same error was encountered.   Can anyone help me to…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,385 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
8,995 questions
asked 2024-12-04T14:42:12.2733333+00:00
Vaibhav Patil 255 Reputation points
edited the question 2025-07-01T05:22:13.5133333+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer

What are the recommended steps for setting up a Service Principal in Azure for enterprise use, and how should the IT team manage its lifecycle securely?

We’re working on setting up Service Principals (App Registrations) in Azure to support automated data workflows (e.g., Azure Data Factory accessing Azure SQL, ADLS, etc.). The goal is to enable secure, non-interactive authentication while following…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-27T09:33:08.0033333+00:00
YERNAIDU SIRAPARAPU 45 Reputation points
commented 2025-06-30T06:35:17.3033333+00:00
Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator
1 answer

'json' parameter is not valid after updating secret key in ADF web activity

Hello, We have a pipeline with two web activities: The first retrieves a secret key from the key vault. The second uses that secret key to obtain a bearer token. Recently, we updated the expired secret key retrieved in the first web activity. The…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-03-20T15:33:29.6633333+00:00
Anneka Osmun 0 Reputation points
commented 2025-06-30T01:59:11.85+00:00
J N S S Kasyap 4,085 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Breaking Undocumented Change: Oracle Connector v2 for queries ending in a semi-colon | Works with v1 linked service

Breaking Undocumented Change: Oracle Connector v2 on ADF: When using "Query" option in Copy Activity, and if that query ends in a semicolon(;), it breaks during run time on v2 linked service, while this works fine on v1 linked service. …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-26T15:53:49.17+00:00
VIjaya Teja Rebbapragada 20 Reputation points
accepted 2025-06-29T12:37:09.1833333+00:00
VIjaya Teja Rebbapragada 20 Reputation points
1 answer

Help with Metadata-Driven ADF Pipeline & Column Mapping Case Sensitivity

Hi everyone, I'm building a metadata-driven Azure Data Factory (ADF) pipeline to dynamically ingest Excel sales files from multiple source systems (like SAP, Senior, etc.) into raw SQL tables. Pipeline Overview I have created a single pipeline…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-25T17:41:34.31+00:00
YERNAIDU SIRAPARAPU 45 Reputation points
commented 2025-06-27T19:18:15.3566667+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer

OData Queries aren't working when using the Lookup action in Azure Data Factory

I'm trying to list all the files in a specific folder on a SharePoint site that have value in a column called "FakeModified" that is less than 2025-01-01. I am using the Lookup activity in Azure Data Factory with a SharePointOnlineListResource…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-25T16:41:04.47+00:00
Calvin Worst 0 Reputation points
commented 2025-06-27T16:59:56.2266667+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer

Why the size of Data Read in ADF copy activity is much bigger than the table size from BigQuery?

I am currently using Azure Data Factory (ADF) to copy data from a BigQuery table to Azure Blob Storage. However, I noticed a discrepancy where the "Data Read" metric in the copy activity is significantly larger than the actual size of the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-25T02:30:45.58+00:00
SheldonWong-7041 0 Reputation points
commented 2025-06-27T09:14:25.8233333+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
2 answers

How does data factory handle guid

How would you design a data pipeline in Azure Data Factory to move data from an on-premises SQL Server to an Azure Data Lake, ensuring data security, fault tolerance, and performance optimization?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-26T05:31:59.26+00:00
Himanshu Sinha 11 Reputation points
commented 2025-06-27T08:58:17.4+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer

Best way to ingest data from SAP BSEG table

I need to extract the following columns from the BSEG table: BUKRS, BELNR, SHKZG, HKONT, DMBTR, SGTXT, PRCTR, GJAHR I’ve tried: Filtering by HKONT values Partitioning by GJAHR (fiscal year) Combining both filters However, all attempts are resulting…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-25T10:00:20.27+00:00
Srinivas, Shreya 0 Reputation points
commented 2025-06-27T08:56:50.91+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
4 answers

Issue with Oracle Connector Ver. 2.0 in ADF writing to Parquet fles

As recommended I have updated our Oracle connectors in our ADF linked services from ver 1.0 to 2.0 since ver 1.0 will be deprecated by 31st July 2025. Since using ver 2.0 we are getting errors when saving files to parquet format. Below is the error we…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-06T01:44:17.1966667+00:00
Kelman David 25 Reputation points
commented 2025-06-27T08:52:46.6933333+00:00
Mohammed Ben M'hand 0 Reputation points
1 answer

Data Flow Sink Fails with DF-Executor-InternalServerError and Spark Executor Timeout in ADF

I have a Data Flow activity in Azure Data Factory named DF_StageAccountInfo that was working previously without any issues. However, it suddenly started consistently failing with the following error: Operation on target DF_StageAccountInfo failed:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-25T06:02:51.6933333+00:00
Grace Ann Salvame 0 Reputation points
edited an answer 2025-06-27T07:09:20.8533333+00:00
Krupal Bandari 770 Reputation points Microsoft External Staff Moderator
1 answer

Delta vs Parquet Format – Design Clarification for DB2 to Azure SQL Hyperscale Migration

In our current project, we are migrating approximately 80 TB of data across ~800 tables from IBM DB2 (on-prem) to Azure SQL Hyperscale. The architecture involves: Source: IBM DB2 Landing Zone: ADLS Gen2 (CSV format via ADF SHIR) Transformation:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-20T06:58:30.36+00:00
Janice Chi 320 Reputation points
commented 2025-06-26T16:51:03.18+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
1 answer

Azure ADF Copy Activity Fails After running for 11 hours with Transport level error

Hi, I am trying to copy multiple tables using the Copy Activity in the ADF pipeline from the Azure Data Warehouse to the Fabric Warehouse. However, the majority of the copy activities fail after running for a couple of hours with the below error. I am…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-25T06:04:03.4833333+00:00
Suraj Pardeshi 0 Reputation points
commented 2025-06-26T10:41:12.1+00:00
Venkat Reddy Navari 3,630 Reputation points Microsoft External Staff Moderator
1 answer

How do we read a secret value in ADF using Lookup Activity or any other activity

How do we read a secret value in ADF using Lookup Activity or any other activity

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-19T06:04:42.9233333+00:00
Immaneni, Nanda 25 Reputation points
commented 2025-06-26T06:32:04.62+00:00
Smaran Thoomu 25,195 Reputation points Microsoft External Staff Moderator
1 answer One of the answers was accepted by the question author.

Copy data not recognizing existing headers

I'm very new to DF, so I need help figuring out the most efficient way to set up a pipeline.  I have to copy .zip files from an SFTP, store them in Blob storage, and extract .csv files with certain names from the .zip file to process and move into a…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-17T13:11:56.3966667+00:00
AE90 20 Reputation points
commented 2025-06-25T19:11:44.9133333+00:00
AE90 20 Reputation points
1 answer

for first-time load of Account data into Dynamics 365 using Azure Data Factory (ADF).

for first-time load of Account data into Dynamics 365 using Azure Data Factory (ADF). I have a CSV file as source which has Account information. I need to import the Accounts from the CSV file into Dynamics 365 Account table. One of the column is parent…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-19T14:11:57.5166667+00:00
Nagalakshmi Pradeep 0 Reputation points
commented 2025-06-25T18:58:24.5466667+00:00
Chandra Boorla 14,685 Reputation points Microsoft External Staff Moderator
3 answers One of the answers was accepted by the question author.

How to Implement a Metadata-Driven ADF Pipeline to Dynamically Process Latest Files from ADLS?

Hello Community, I need help designing a single, reusable, and metadata-driven Azure Data Factory (ADF) pipeline for a sales order ingestion use case involving multiple source systems like SAP and Senior. Business Scenario We receive Excel files from two…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,672 questions
asked 2025-06-20T09:57:32.63+00:00
YERNAIDU SIRAPARAPU 45 Reputation points
answered 2025-06-25T17:49:19.57+00:00
YERNAIDU SIRAPARAPU 45 Reputation points