An Azure service for ingesting, preparing, and transforming data at scale.
Why an activity was not triggered on the Failure of previous Web activity in ADF
My web activity, Get ChangeOrderRequest, is in a sequential ForEach Prime Contract activity and can fail with the message, "Item not found". I have a Set exit message variable activity to be executed upon failure of the Get ChangeOrderRequest…
Azure Data Factory
how connect to a source system - SAP database?
how connect to a source SAP database?This project involves the migration of the existing data warehouse environment from Qlik to Microsoft Azure. The initiative aims to modernize the analytics platform by leveraging Azure’s scalable storage, compute, and…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Azure Data Factory: Managed Identity Authentication Support for Cosmos DB for MongoDB
We need to copy data from Azure Cosmos DB (for MongoDB) to Azure Data Explorer via Azure Data Factory using managed identity authentication for security compliance. Currently, it appears only connection string authentication is supported for Cosmos DB…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Xero 2.0 ADF connector not supporting the same tables as Xero 1.0 connector
The Xero 2.0 connector in ADF no longer supports the ProjectTasks and Time tables while the Xero 1.0 connector does. This is also mentioned in the documentation: …
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Deleted DataFactory Trigger still executing daily for months after being disabled and deleted
We used to execute an SSIS package 2x daily on a Data Factory trigger that started an Integration Runtime, ran the package, and stopped the runtime. This was all supposedly disabled over a year ago. Today I accidentally found from the logs that the…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Join: Microsoft Azure Q&A Champions Program
The Microsoft Azure Q&A Champions Program recognizes and empowers a global community of internal and external Azure experts who help customers succeed by providing high-quality, trusted answers on Microsoft Q&A. The program scales expert led…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Azure Datafactory to Fabric Lakehouse Migration
We used to see a popup in Azure Data Factory about migration to Fabric Lakehouse, but that notification is no longer appearing. Could you let us know if the migration is still required? If so, is there a timeline for it? Attaching the screenshot of how…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
SFTP Linked service using Public & Private Keys
Hello, Trying to create a SFTP linked service using " SSH Public key Authentication "and getting errors. One of the errors I am receiving is 'Invalid Sftp credential provided for 'SshPublicKey' authentication type.Invalid private key…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
SftpFailedToConnectToSftpServer
I am having an issue with a client supplied SFTP account from within Data Factory, I am able to connect with the credentials via FileZilla locally, and with some C# code when deployed, but Data factory cannot create a valid connection to this AWS server.…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
The Validation Activity has a timeout of 13 hours, but i configured it to time out after 7 days in Blusehift Env.
The Validation Activity has a timeout of 13 hours, but i configured it to time out after 7 days in Blusehift Env.. this's my validation activity settings: { "dataset": { "referenceName": "DependencyPath_EU", …
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
unable to see resources and pipelines
Hi Team, I want to see my old resources and all the azure data factories pipeline i created, but this time i am unable to see anything, suggest how to check those pipelines.
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
I could not connect to ADF. it is asking me to pay-as-you-go. previously with *******@gmail.com
Hi, I could not connect to ADF. it is asking me to pay-as-you-go. previously with ****@gmail.com , connected with other user . How could I use free . It is showing my one subscription and all. Even I closed that subscription also. I can not open other…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
ADF pipeline succeeds but data is wrong - common patterns and how to catch them
Wanted to start a discussion around a category of ADF issues that I see repeatedly in production but rarely documented well. These are cases where the pipeline runs successfully with no errors, but the data loaded is incorrect. Examples include column…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
ADF ARM Template validation issue
I am getting error while running : run: | # node ./adf/node_modules/@microsoft/azure-data-factory-utilities/lib/index validate ./adf ${{env.RESOURCEID_DEV}} "message":"undefined not found in linked-service-registery"
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
ADF delete activity failure because of storesettings
ADF delete activity fails with this error. It works fine in Debug mode, but fails in Trigger mode. We checked that the collaboration branch has the same changes as in adf_publish. So not sure why this fails. One observation is whenver we open the delete…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Azure-SSIS IR (ssis-ir03) in ADF (adf-ssis-08) is stuck and cannot be stopped or deleted. Cannot delete ADF resource.
Azure-SSIS IR (ssis-ir03) in ADF (adf-ssis-08) is stuck and cannot be stopped or deleted. Cannot delete ADF resource. Initial error was: “Operation failed. Cannot find key vault” The dependent Key Vault had been soft-deleted, and I successfully…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Data Factory - Copy Activity - Azure Blob Storage to Snowflake V2 - Extremely long queue times
Noticing this morning unusual queue times on copy activities in the second part of the staged copy from Azure to Snowflake (Azure Blob Storage to Snowflake V2). The first part of the copy activity (reading from API for instance) is fine but after it…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Parameterize MongoDB linked service Database Name in ADF
I have many databases in MongoDB (nearly 20). I need to create linked services for all of them in Azure Data Factory. After creating one, I found that linked services for MongoDB are not parameterized for database name. Is there any way to create a…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Table lock whiile runing multiple UPDATE statements on the same Warehouse table.
I am using Microsoft Fabric Data Pipelines with a ForEach activity to run multiple UPDATE statements on the same Warehouse table. When multiple updates run at the same time, I get concurrency or locking errors. I tried reducing the batch count in the…
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
Azure data factory pagination rule cannot work for dataflow mapping but copy activity
Hi! I planned to move some data using Rest API and then do some transformation like join and cleaning. So instead of use move activity I prefer directly use mapping data flow to do pagination. my rest API setting is like this: Base URL: …
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.