9,776 questions with Azure Data Factory tags

Sort by: Updated
1 answer

All ADF dataflow stages disappear after publish

Hello, I made a source for my Dataflow and added several stages to the existing one. I published and Dataflow showed an error with a new source. I tried to delete it, but could not delete it. I published again and all my stages disappeared. It is not…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-29T11:22:44.82+00:00
Darjuš Vasiukevič 20 Reputation points
commented 2024-05-29T14:53:27.96+00:00
Darjuš Vasiukevič 20 Reputation points
1 answer

ADF does not allow MI based auth for Azure table storage. Are there any alternatives for this?

I am trying to move away from key-based access for an azure table storage in ADF, but the only auth options I get are account key and SAS URI. Is there no way to use MI/SP for this? Blob storage has MI based auth but not table, why is this so?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-29T12:15:25.3933333+00:00
Akshita Arora 0 Reputation points Microsoft Employee
answered 2024-05-29T14:50:40.8066667+00:00
Amira Bedhiafi 16,696 Reputation points
1 answer
asked 2024-05-29T14:14:58.2933333+00:00
NK 0 Reputation points
answered 2024-05-29T14:32:50.0833333+00:00
NK 0 Reputation points
5 answers One of the answers was accepted by the question author.

Copy Data Activity failes on datatype uniqueidentifier when Staging is enabled and no column mapping configured.

Using Azure Data Factory Copy Data Activity from Azure SQL Database to Azure Synapse Analytics. The Copy Data Activity failed with InvalidCastException error 'The given value of type String from the data source cannot be converted to type…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2020-06-30T14:49:54.783+00:00
XchangeVisions 301 Reputation points
commented 2024-05-29T12:50:24.26+00:00
Umar Butt 0 Reputation points
1 answer One of the answers was accepted by the question author.

ADF deployment issue

Hi, I am using Azure data factory and setup different branches for developers. I have setup adf publish branch as well so that when developers are done with their changes, we can publish the changes to adf publish branch and then from that branch I can…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-23T10:48:57.88+00:00
RSingh 40 Reputation points
accepted 2024-05-29T12:23:09.3733333+00:00
RSingh 40 Reputation points
1 answer One of the answers was accepted by the question author.

Azure Data Factory: Fetch xml to return all the attributes not working properyl

Hi, Thanks in advance for your response. I am using copy activity to copy the data from one environment to another environment in dynamics 365. If I dint use any filter in the fetch xml query, it works fine. Otherwise i am getting missing key…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-28T14:31:05.47+00:00
Maniraj K V 20 Reputation points
commented 2024-05-29T11:59:50.3333333+00:00
Maniraj K V 20 Reputation points
1 answer

Obvious behavior of dynamic mapping in ADF

I have a single copy activity that moves data from a source parquet file to an azure sql database. When I used auto mapping and tried to change the name of source or sink field it failed, however when I copy the json mapping generated and change one of…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-29T06:44:25.6133333+00:00
Bens Bernard 0 Reputation points
edited the question 2024-05-29T09:46:04.9366667+00:00
ManoharLakkoju 615 Reputation points Microsoft Vendor
1 answer

"No result available for stream" error when performing join and previous step uses cached lookup

I have a data flow where I look up and cache rows that have been updated as part of an ETL pipeline. I then want to join that updated data with the existing data in the target database. The problem seems to be that when doing the join operation the part…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2022-06-23T20:01:38.977+00:00
John Neubecker 6 Reputation points
commented 2024-05-29T09:06:02.9666667+00:00
Moon Do 0 Reputation points
0 answers

Is there a way to restrict multiple instances of an ADF pipeline on Same path of event based trigger?

I'm running Spark notebooks in Synapse using an event-based ADF pipeline. If any notebook gets triggered twice, I encounter a conflict error. I understand that I can avoid concurrent execution of the pipeline by setting the concurrency to 1. However, I…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,486 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,978 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-28T11:16:32.32+00:00
Divya Sharma 1 Reputation point
commented 2024-05-29T08:09:59.56+00:00
Harishga 4,325 Reputation points Microsoft Vendor
0 answers

Unable to resolve Azure database for mysql name from Data Factory

I am creating a linked service from my Azure Data Factory environment, to access my Azure Database for Mysql database. I select "Azure database for Mysql", and I can see all my mysql servers. I select "From Azure subscription" as the…

Azure Database for MySQL
Azure Database for MySQL
An Azure managed MySQL database service for app development and deployment.
729 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-07T07:39:15.9766667+00:00
Ignacio Sales 5 Reputation points
commented 2024-05-29T07:59:20.8933333+00:00
PRADEEPCHEEKATLA-MSFT 80,251 Reputation points Microsoft Employee
0 answers

How to forecast 12-month demand for medicines using Azure AutoML time-series?

I need to predict the monthly demand for medicines for the next 12 months. I have historical data from 2018 to 2023 for each medicine, including a unique drug identifier (Scode) and the date of consumption. Using this data, I aim to forecast the demand…

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,618 questions
Azure Time Series Insights
Azure Time Series Insights
An Azure internet of things (IoT) analytics platform to monitor, analyze, and visualize industrial IoT analytics data at scale.
77 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-24T06:35:22.5733333+00:00
Anil Kumar Nagar 0 Reputation points
commented 2024-05-29T05:41:54.8066667+00:00
santoshkc 5,180 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Azure Data Factory - Copy data activity failed: Failure happened on 'Source' side

Operation on target Copy data1 failed: Failure happened on 'Source' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-04-23T05:58:01.1533333+00:00
Aman Pahariya 0 Reputation points
accepted 2024-05-29T04:34:36.4466667+00:00
Aman Pahariya 0 Reputation points
1 answer

Not able to connect to on premise sql server using self integration run time in azure data factory

I am able connect to on premise sql server using SSMS tool but while connecting through azure data factory using self hosted integration runtime and linked service getting below error. No mistakes in user name and password .Please help me on this. { …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-23T16:30:35.6866667+00:00
Rajesh 0 Reputation points
answered 2024-05-29T03:00:26.21+00:00
Nandan Hegde 29,911 Reputation points MVP
0 answers

Salesforce Bulk API Connector

I am using SOQL instead of ObjectApiName. The SOQL is generated using parameter correctly. The older legacy connector works fine for the same pipeline but the new one is giving this error msg: …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-28T04:11:41.51+00:00
Dev Poudel (Health) 0 Reputation points
commented 2024-05-29T01:29:55.8833333+00:00
Dev Poudel (Health) 0 Reputation points
1 answer One of the answers was accepted by the question author.

i got stuck on deployment is in progress when creating data factory

this is the message that appear to me : status : InternalServerError { "status": "Failed", "error": { "code": "GatewayAuthenticationFailed", "message": "Gateway…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2023-11-09T18:31:24.8633333+00:00
17042440 20 Reputation points
commented 2024-05-28T22:14:36.2733333+00:00
SDanpant 1 Reputation point Microsoft Employee
0 answers

Copy activity throwing an error, when a auto incremented column is there on the Sink side

Hello Team, I am trying to copy data from Amazon S3 to Azure SQL database. On Azure SQL side I have set an autoincrement column(Row_Index) in a table, however the copy activity throws an error that "Column 'Row_Index' does not allow…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-15T22:53:21.2033333+00:00
Shweta Chavan 40 Reputation points
commented 2024-05-28T20:13:19.3933333+00:00
Oury Ba-MSFT 16,901 Reputation points Microsoft Employee
0 answers

Except Unique Assert Transformation in Dataflow

I am trying to use Asset transformation in dataflow to check the dups in source. I am not able to get the correct result when using parameters that will have single or multiple columns as primary keys (example shared below) when using below expression. …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-26T08:02:22.9033333+00:00
Vaibhav 105 Reputation points
commented 2024-05-28T20:11:05.1833333+00:00
Vaibhav 105 Reputation points
1 answer

How to fix "Failure happened on 'Source' side A task was canceled" error

Data load from Salesforce object with 1million records fails with the below error in Azure Data Factory: Failure happened on 'Source' side. 'Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,' The data load…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-28T18:42:40.6+00:00
thara kollu 0 Reputation points
answered 2024-05-28T20:01:12.95+00:00
hossein jalilian 4,285 Reputation points
1 answer

Default TImezone in the Dataflow

Hi Team, Is there anyway to change the default timezone in the datafactory? Currently, when I am creating a column(loadtime) with currentTimestamp() in derived column transformation in the dataflow, then I get the time in UTC. But, I want the default…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-27T20:45:04.3433333+00:00
Vaibhav 105 Reputation points
commented 2024-05-28T20:00:32.4366667+00:00
Vaibhav 105 Reputation points
1 answer

Salesforce V2 Link Service doens't work. error message: a task was canceled Source=mscorlib

I am migrating the ADF salesforce connector to the new V2 connector and now testing it. When testing the connection for the link service, it is good. However, it failed in the copy activity. with message: Failure happened on 'Source' side.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,776 questions
asked 2024-05-16T13:35:58.54+00:00
ChoiLiamhehimhis-5365 25 Reputation points
commented 2024-05-28T19:30:28.1+00:00
thara kollu 0 Reputation points