Azure Data Factory Copy Activity Fails – Azure IR Pre-login Handshake Timeout Across Regions
Due to ill-supported support from within Azure, I try getting answers along this way: We are encountering persistent failures in Azure Data Factory copy activities when writing to Azure SQL Database using the Azure Integration Runtime (both AutoResolve…
Azure Data Factory
Azure Data Factory trigger event not working while uploading using .NET client library
I want to trigger an ADF pipeline on blob creation, file based trigger. The manual upload to blob storage triggers the ADF pipeline correctly. But when i am uploading the same via AzureDataLakeStore .Net libraries from a web app, specifically package - …
Azure Data Factory
ADF COPY ACTIVITY FAILS DUE TO TIME DATATYPE COLUMN
I am trying to copy data from SQL Server to Snowflake using ADF Pipeline, I am using a copy activity, my source and destination column data types all match , my source column "LastUpdateTime" in SQL Server is a TIME data type . My destination…
Azure Data Factory
upgrade the Amazon RDS for Oracle connector v2 is not working
I am trying to upgrade existing linked service for Amazon RDS for Oracle to v2. When I try to publish the linked service, below properties are empty. Can someone help me on this? I am using below json template for the linked service. { "name":…
Azure Data Factory
How Does Azure Data Factory Dynamically Decide DIU and Parallel Copy Based on File Count vs. File Size?
We are designing a high-throughput ingestion pipeline using Azure Data Factory (ADF) and would appreciate clarification regarding how Data Integration Units (DIU) and Degree of Parallelism (parallelCopies) are determined automatically. As per your…
Azure Data Factory
Connecting to On-Prem SQL Server through Change Data Capture (ADF) Issue
So I have been trying to get CDC working for the past two days but can't seem to get it to connect to my on-premise SQL Server as it seems to have issues using self hosted integration runtime inside the CDC wizard. I am trying to do incremental loads…
Azure Data Factory
Azure Data Factory (ADF) Managed Batch Account Endpoint issues.
I am attempting to integrate Azure Data Factory with Azure Batch using a private endpoint, specifically through a Managed IR. I created a Managed Vnet, then used az cli to attempt to connect a batch account using a managed private endpoint: az…
Azure Data Factory
what time is JOB_DURATION for
is JOB_DURATION representing until run start time or until run end time?
Azure Data Factory
Help with Auto Update Issue on Self-Hosted Integration Runtime (Error Code: 10003)
I'm running into an issue with the auto-update of the Self-Hosted Integration Runtime (SHIR) agent in Azure Data Factory. When I try to manually update the agent from the Data Factory Studio, I get the following error: Download failed: Download…
Azure Data Factory
How to fetch Schema of a file stored in Azure Blob using ADF
Hi, I'm sending data from Azure Blob to AWS S3 (As AWS S3 isn't supported as sink dataset in Copy Data Activity I'm using Mapping Dataflows). I need to write the schema to the sink, so my pipeline has to write data in one directory and schema file in…
Azure Data Factory
Guidance on Best Practices for Loading CDC-Merged Data from ADLS to Azure SQL Hyperscale
As part of our healthcare data modernization initiative, we are currently implementing the Catch-Up phase of a large-scale CDC pipeline. The architecture involves: All Insert/Update/Delete logic (CDC merge) being handled in the Databricks branch layer…
Azure Data Factory
Azure data factory @concat usage..please help
HI, i am trying to call multiple tables from one database such that it should fetch only current week data to sink and i am trying to use where condition in source copy activity where i am defining below query... NOTE: item().source is used to get…
Azure Data Factory
adf split of columns
I have a column like id,job description,etc in a single column how to make it separate column in adf
Azure Data Factory
Deadlock Issue
I am facing deadlock issue in the ADF pipeline. Will enable the retry option in the settings 3 times and the interval 5 mins will solve the problem? I am getting the deadlock issue occasionally only not on regular basis.
Azure Data Factory
Handling Files and Data with Azure Data Factory
I have created a data pipeline using ADF which processed data from ADLS Gen2 pushed by SAP system and after transformation persists to Azure SQL. Please help me in designing on:- How ADF ensure the files is already processed or not processed. I want…
Azure Data Factory
Cannot add workarounds=’ 536870912’ connection property to Oracle linked service version 2.0
According to Microsoft we must upgrade azure oracle linked services from version 1.0 to 2.0. My oracle linked service version 1.0 has an additional connection property named "workarounds" see the picture attached. How to incorporate the…
Azure Data Factory
How to export full Azure Data Factory ARM template (pipelines, datasets, dataflows, linked services) via API/CLI?
I’m trying to automate the export of Azure Data Factory (ADF) ARM templates as part of a CI/CD pipeline. The goal is to get a complete export containing pipelines, datasets, dataflows, linked services, and other artifacts - without manually using the…
Azure Data Factory
Best Practice for Handling Multiple Excel Files with Different Schemas and Schedules in Azure Data Factory
I’m working on an Azure Data Factory (ADF) solution where I receive four different Excel files daily as input data for order processing. Each of these files has a unique and fixed schema that will not change over time. In terms of scheduling: Two files…
Azure Data Factory
Sharepoint List extract error from ADF
Dear Support, I am reaching out regarding an issue we are encountering while extracting data from several SharePoint lists using Azure Data Factory (ADF). We have been able to successfully extract 10 out of 14 lists without any problems. However, when…
Azure Data Factory
Azure Data Factory - IP is not from a specified range
Hi. I am trying to whitelist my Azure Data Factory Azure Integration Runtime, but the IPs that it is utilizing do not come from DataFactory.WestEurope service tag, but from AzureCloud.WestEurope. The service I'm trying to connect is using IP whitelisting…