9,983 questions with Azure Data Factory tags

Sort by: Updated
Updated Created Answers
1 answer

How to unzip csv files, read first line, load into different tables based on the first line content using ADF?

I have Zip folder in Azure container in which we have multiple CSV files. I want to: Unzip CSV files read first line of each file one by one to find text 'XYZ'. based on findings i want to route the file data into two different tables (one with…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-27T09:19:17.7366667+00:00
KT 0 Reputation points
commented 2024-07-01T10:02:52.7733333+00:00
Harishga 5,420 Reputation points Microsoft Vendor
0 answers

How to run a purge command in ADX using ADF?

Approach 1 Created an ADX activity in ADF, and in the command field, I wrote the purge command (which I copied from Azure Docs): .purge table MyTable in database DBName allrecords I tried running the purge command but it keeps returning syntax error:…

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
501 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-12T18:37:13.4033333+00:00
Sailesh Choyal 0 Reputation points
edited a comment 2024-07-01T09:25:17.24+00:00
Sailesh Choyal 0 Reputation points
2 answers

Unable to setup a connection in ADF with SA despite whitelisting the IP of SHIR

Hello. I am having some issues with connecting to Storage Account from ADF. My current setup: Azure Data Factory Virtual Network Static Public IP Address Azure Windows Server 2019 VM with Self-Hosted Integration Runtime installed and Public IP from…

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,857 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-24T15:49:36.65+00:00
11-4688 111 Reputation points
answered 2024-07-01T08:53:22.5966667+00:00
Pinaki Ghatak 2,720 Reputation points Microsoft Employee
2 answers

Azure Data Factory - Azure SQL connectivity error : Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access

Hi I am trying to import data from onprem SQL server database to Azure SQL databases through Azure Data Factor(ADF) Copy activity , this worked for more than a year with no problems ,till few weeks back when I got the error below. I got a workaround it…

Azure SQL Database
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,573 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,252 questions
asked 2024-06-24T13:30:39.4533333+00:00
Mfundo Potolwana 0 Reputation points
answered 2024-07-01T08:49:18.93+00:00
Pinaki Ghatak 2,720 Reputation points Microsoft Employee
1 answer

Copy activity failing with SystemErrorOutofMemory

Hi, I have a pipeline which is executing copy activity, copying data from SQL server and storing to a blob container. It is failing with error -…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-07-01T07:09:02.23+00:00
Mayank Gupta 0 Reputation points
commented 2024-07-01T07:24:19.3666667+00:00
Mayank Gupta 0 Reputation points
2 answers

Data Factory triggered from bob storage creating multiple runs

I have an ADF that currently triggers from a blob storage event, blob created, ignoring empty blobs. Previously this was correctly triggering a single ADF run when a new file was uploaded. I have recently introduced a Logic App which copies a new…

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,573 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
Azure Event Grid
Azure Event Grid
An Azure event routing service designed for high availability, consistent performance, and dynamic scale.
340 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
10,109 questions
asked 2024-06-20T01:00:20.3033333+00:00
Louise Kirkham 0 Reputation points
answered 2024-07-01T06:02:54.42+00:00
Nandan Hegde 30,716 Reputation points MVP
0 answers

Azure Data Factory - Sink dataflow error

Hello , i was trying to connect SFTP connection to the Sink data flow and in directory i gave the path and it was throwing error as Job failed due to reason: at Sink 'sink1': Can't make directory for path ".spark-staging-" under "". i…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-27T16:36:32.6+00:00
Prasad M 0 Reputation points
commented 2024-07-01T05:23:43.69+00:00
Harishga 5,420 Reputation points Microsoft Vendor
0 answers

Pipeline outputs files with empty content

I have a pipeline and it worked well until 6/14. Since 6/15, the pipeline ran successfully and output files with empty content, but I did not modify it. Can anyone help me understand it? Here is the pipeline:…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-27T11:13:12.63+00:00
Yanan Wang 0 Reputation points Microsoft Employee
commented 2024-07-01T05:11:39.49+00:00
Yanan Wang 0 Reputation points Microsoft Employee
2 answers

Issue with Copy Dataverse data into Azure SQL using Synapse Link Template

I am experiencing issues using ADF I am trying to use the template "Copy Dataverse data into Azure SQL using Synapse Link" to move f&o data from datalake to Azure…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,601 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-05-30T07:08:10.6766667+00:00
Dipo Onibile 0 Reputation points
commented 2024-07-01T04:55:32.2433333+00:00
phemanth 7,825 Reputation points Microsoft Vendor
0 answers

I'm encountering inconsistencies with FetchXML in Dynamics 365; it fails to load all the all attributes each time I run the query. Despite setting up FetchXML to filter and retrieve all attributes, it only returns a subset. Why is this ?

Here is the query that I am using ADF copy activity: <fetch version="1.0" output-format="xml-platform" mapping="logical" distinct="false"> <entity name="@{item().tablename}"> …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
Dynamics 365 Training
Dynamics 365 Training
Dynamics 365: A Microsoft cloud-based business platform that provides customer relationship management and enterprise resource planning solutions.Training: Instruction to develop new skills.
68 questions
asked 2024-06-26T05:05:57.9766667+00:00
Namrata Zalewski (She/Her) 0 Reputation points
commented 2024-07-01T04:51:54.8133333+00:00
phemanth 7,825 Reputation points Microsoft Vendor
1 answer

Servicenow linked service used connectivity vendor

I am interested in using Servicenow Linked Service to extract data from Servicenow. There are two linked services available; the legacy version and version 2 that uses table API. The documentation mentions a built-in driver for enabling connectivity. Can…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,601 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-27T10:50:37.5233333+00:00
Amir Bagheripour 0 Reputation points
commented 2024-07-01T03:59:02.4966667+00:00
Harishga 5,420 Reputation points Microsoft Vendor
0 answers

Azure Data Factory executes stored procedure twice (and concurrently)

I have an ADF pipeline executing a stored procedure on an Azure SQL database. Retries is set to 0, so this procedure should be executed exactly once. Now it turns out that it sometimes executes twice, i.e. a second time when the first execution is…

Azure SQL Database
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-26T14:40:32.42+00:00
Matthias Delafontaine 16 Reputation points
commented 2024-07-01T03:54:13.1433333+00:00
Harishga 5,420 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Insert Excel Data in Sharepoint/Onedrive to SQl DB using ADF - cannot use Storage Account

I want to insert the data in Excel placed in a Sharepoint/Onedrive, into a DB using ADF. I found that this can be done using a storage account. My requirement is not to use a Storage account. Can anyone guide me?

Excel
Excel
A family of Microsoft spreadsheet software with tools for analyzing, charting, and communicating data.
1,641 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
SharePoint
SharePoint
A group of Microsoft Products and technologies used for sharing and managing content, knowledge, and applications.
10,109 questions
asked 2024-06-26T07:32:25.61+00:00
Dixan Lal Thomas 20 Reputation points
accepted 2024-06-30T18:25:53.73+00:00
Dixan Lal Thomas 20 Reputation points
1 answer

What could cause this error: Cluster creation failed due to an unexpected system error from a runtime related dependency

During one of the scheduled executions of a pipeline, an error occurred with the following details: Error code: [5000](https://go.microsoft.com/fwlink/?linkid=2117168#error-code-5000) Failure type: User configuration issue Details: Cluster creation…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-25T10:18:32.1533333+00:00
Veena 0 Reputation points
commented 2024-06-30T15:13:42.7466667+00:00
Smaran Thoomu 12,085 Reputation points Microsoft Vendor
1 answer One of the answers was accepted by the question author.

Adding a custom column in a ADF copy activity

Hi, in ADF I'm implementing a copy activity. I'd like to add a custom date column in order to add an actual load date to sink data. Obviously, the load date doesn't come from source data. Is it possible to add a custom date column to value with the…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-29T04:13:38.37+00:00
pmscorca 877 Reputation points
accepted 2024-06-30T06:02:36.1233333+00:00
pmscorca 877 Reputation points
1 answer

Data Factory Time Discrepancy: Seeking Explanation

In Data Factory, I have a total transfer time of 23 minutes, but the sum of reading and writing time is much lower. My reading time is 4:48 and writing time is 7:12. Can someone help me understand this discrepancy ?

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,405 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-26T10:02:49.0066667+00:00
EmaLom 0 Reputation points
answered 2024-06-29T06:13:46.0866667+00:00
EmaLom 0 Reputation points
2 answers

Azure Data Factory: Copy Data Activity: "Schema import failed: Please select a table"

Hello! I am pretty new to Azure Data Factory and trying to build my first data pipeline. So far, I have managed to install an integration run time in my PC and established a connection to the on prem SQL server. I have tested the connection and it says…

SQL Server
SQL Server
A family of Microsoft relational database management and analysis systems for e-commerce, line-of-business, and data warehousing solutions.
13,171 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-22T10:16:15.37+00:00
Bhargav 5 Reputation points
answered 2024-06-28T22:24:33.49+00:00
Arabaj Shaikhh 0 Reputation points
1 answer

Data flow is not sinking all data because I have 84 json files in different sub-folders (of which 78 files are blank, and 6 files have data). But when I delete the 78 blank files (and remain with 6 files with data), the data flow will sink all results.

Hello. Please support, my data flow pipeline is not sinking all data. This happens whenever I have 84 json files each in different sub-folders (of which 78 files are blank, and 6 files have data). But when I delete the 78 blank json files (and only…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-14T17:35:51.8166667+00:00
Babu Musanga 0 Reputation points
commented 2024-06-28T17:19:40.74+00:00
Babu Musanga 0 Reputation points
2 answers

Release Pipeline - ADF deployment

hi, I would like to deploy ADF solution from DEV to UAT to PROD. I prefer using classic ADF release pipeline in Azure devops, I would like to have an option to pick the items for deployment like pipelines, dataflows and datasets. I do not want deploy…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-26T11:37:23.42+00:00
Santhi Dhanuskodi 205 Reputation points
commented 2024-06-28T14:19:11.4733333+00:00
BhargavaGunnam-MSFT 28,446 Reputation points Microsoft Employee
0 answers

Azure Data Factory - Jira Connector Schema Change / missing data

Hi All, We have been using ADF to ingest Jira into our data lake for several months now. The connector has worked with no issues and we have been able to pull all issues and all the custom fields, etc from Jira. Our Monday refresh failed due to a schema…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,983 questions
asked 2024-06-19T05:59:12.65+00:00
Jose 0 Reputation points
commented 2024-06-28T13:42:36.26+00:00
Jose 0 Reputation points