9,444 questions with Azure Data Factory tags

Sort by: Updated
1 answer

Express route

Hi, We have newly setup express route with on-prem. I have some queries. How do we test that we have express route connectivity between Azure to On-prem destination servers ? We want to connect on-prem oracle db over express route and TLS using ADF…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
Azure ExpressRoute
Azure ExpressRoute
An Azure service that provides private connections between Azure datacenters and infrastructure, either on premises or in a colocation environment.
320 questions
asked 2024-04-13T05:05:24.7733333+00:00
Sourav 40 Reputation points
commented 2024-04-13T14:35:09.68+00:00
Sourav 40 Reputation points
1 answer

AZ DATA FACTORY - COPY Activity : 'Incorrect syntax near '>'

Hi, while i am trying to copy data using below dynamic value. SELECT * FROM @{pipeline().parameters.P_DS_Object_api} WHERE lastmodifieddate >= '@{formatDateTime(activity('watermarktable').output.firstRow.Watermarkvalue, 'yyyy-MM-dd HH:mm:ss')}' i am…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,316 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2023-09-07T11:12:25.0733333+00:00
Ram Babu 45 Reputation points
commented 2024-04-13T07:54:41.0966667+00:00
Ivan Casana-Gallen 21 Reputation points
4 answers One of the answers was accepted by the question author.

ADF throwing 403 error while copying

Hi, I have Azure synapse analytics setup with private endpoint and I am running a ADF pipeline but it throws the error: {     "errorCode": "2200",     "message":…

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,316 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2023-09-29T17:10:11.98+00:00
Rajaniesh Kaushikk 476 Reputation points MVP
accepted 2024-04-13T00:49:16.7433333+00:00
Rajaniesh Kaushikk 476 Reputation points MVP
4 answers

Managing private endpoints with "portal" subresource and multiple Data Factory deployments.

Hi everyone, We have several data factories across many subscriptions (data sensitivity isolation, different environments, etc ...). Basically, if a managed service is compatible with the Private Link Service, we use it as our Azure environment is…

Azure DNS
Azure DNS
An Azure service that enables hosting Domain Name System (DNS) domains in Azure.
592 questions
Azure Virtual Network
Azure Virtual Network
An Azure networking service that is used to provision private networks and optionally to connect to on-premises datacenters.
2,119 questions
Azure Private Link
Azure Private Link
An Azure service that provides private connectivity from a virtual network to Azure platform as a service, customer-owned, or Microsoft partner services.
457 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2022-06-02T08:17:33.327+00:00
LukeCloudWalker-6128 36 Reputation points
answered 2024-04-12T19:48:01.18+00:00
Eric Ehret 0 Reputation points
2 answers

How to remove Line Breaks from Excel file in ADF

Hi All, I have a scenario wherein my Excel file has data with line breaks. If i convert this excel file to CSV, even my CSV file contains line break in data and i am getting error while reading this CSV through PolyBase table in Azure Synapse. I…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2020-09-10T05:12:04.197+00:00
Priya Jha 866 Reputation points
commented 2024-04-12T18:51:17.38+00:00
Row 0 Reputation points
0 answers

Azure Datafactory Out of memory error when read from Salesforce

I use Datafactory Copy activity to copy data from Salesforce to ADLS, I am facing with Out of memory error. The file size is 129k of rows 800MB, then I set block size to 100 MB and Max rows per file to 100 000, but the error still exist. What can you…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,330 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-04T13:32:58.54+00:00
Olga Valiuk 0 Reputation points
edited a comment 2024-04-12T16:18:36.4866667+00:00
Olga Valiuk 0 Reputation points
0 answers

How to scan a single data ADLS source for multiple collectioons

We have recently implemented Purview as apart of a data strategy project that involves ADLS Gen 2. We have multiple data pipelines into a single ADSL container. Now in implementing Purview, and looking to create separate Collections for each business…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
Microsoft Purview
Microsoft Purview
A Microsoft data governance service that helps manage and govern on-premises, multicloud, and software-as-a-service data. Previously known as Azure Purview.
915 questions
asked 2024-04-12T16:14:33.4766667+00:00
Oluyemisi 0 Reputation points
1 answer

issue parsing the expression body('DataFlow1ComposeRuntimeVariables')?.DataFlow1f247f5567eb940fb88196f49f1a907ab?.Source.FolderName

I have source dataset added two parameters named FolderName & ABLB_ContainerName and used in data flows of source (source settings). under source options, wildcardpaths: concat($yyyy,'/',$mm,'/',$dd,'/*.json'). data flows are inside switch activity.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-12T08:51:38.6866667+00:00
Anuganti Suresh 160 Reputation points
commented 2024-04-12T12:57:10.3966667+00:00
Anuganti Suresh 160 Reputation points
1 answer

Connect on-premise API from Azure Data Factory pipeline

Hi Guys, I have a requirement to connect on-premise API link from ADF pipeline. I used WEB Activity with self hosted IR with below configuration, getting error code - 2108 .Failure type - User configuration issue  Details - Error calling the endpoint.…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-12T10:46:22.6466667+00:00
SarmisthaS 0 Reputation points
commented 2024-04-12T12:56:45.8+00:00
SarmisthaS 0 Reputation points
1 answer

Azure Datafactory SSIS IR configure Vnet integration with Custom DNS

Configure Azure Datafactory SSIS IR Express Vnet integration with Custom DNS. Followed below document, but it used Blob SAS URI which is not…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-11T13:04:30.5166667+00:00
Prannoy Seal (US) 0 Reputation points
edited a comment 2024-04-12T12:48:37.2766667+00:00
Prannoy Seal (US) 0 Reputation points
0 answers

Why is the Copy-Activity slower than a Stored Procedure in Azure Data Factory?

I am building an Azure Data Factory pipeline to mimic the Stored-Procedure-based workflow, where the first step is to read several JSON-files from a connected Azure Blob Storage and copy the data to our Azure SQL Database. However, the Dataflow takes…

Azure SQL Database
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,405 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-12T12:19:33.7+00:00
RenPanning-1598 20 Reputation points
2 answers One of the answers was accepted by the question author.

Consistent data in data lake gen2

Hi friends, I have to understand how data consistency works in ADLS, I have found this old…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,330 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-04T12:24:04.8466667+00:00
Anshal 1,826 Reputation points
commented 2024-04-12T10:53:58.17+00:00
1 answer

ADF OAuth2 Client Credentials Flow - Connection Test Fails

I'm trying to connect to a REST service requiring an access token (OAuth2) using Azure Data Factory (ADF). In Postman, everything works perfectly with the client credentials flow (client_id/client_secret): I call the token endpoint to get an access…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-11T08:10:39.3+00:00
Pravalika-randstad 220 Reputation points
commented 2024-04-12T09:33:41.4366667+00:00
Smaran Thoomu 8,885 Reputation points Microsoft Vendor
1 answer

ADF pipeline to read the data from UC table to adls gen2 account

Hello Team, We have a requirement to create Azure Datafactory pipeline to read the data from UC table, access on the table is provided ( to Azure Datafactory Managed Identity) and copy the data into adls gen2. Is there a way or article to implement this?…

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,330 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,901 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-11T19:05:05.9733333+00:00
Ashwini Gaikwad 45 Reputation points
commented 2024-04-12T07:44:27.2+00:00
Amira Bedhiafi 14,481 Reputation points
0 answers

Azure Data Flow connection error to Dynamics 365 but pipeline can

I have a pipeline that has a connection to a table in Dynamics 365 which works fine. I need to create a dataflow as I need to lookup a value in the source that then replaces the value with a guid to load into Dynamics 365 but when I do a data preview in…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-03-04T05:21:30.9166667+00:00
Wayne Bartkowski 656 Reputation points
commented 2024-04-12T06:37:26.5266667+00:00
ShaikMaheer-MSFT 37,656 Reputation points Microsoft Employee
1 answer

Azure Data Factory : DF-Executor-UserError (Writing job aborted.)

I have created a pipeline in ADF based on Dataflow. The dataflow shows step by step update I want to make also reflected in the sink. When I earlier used to run pipeline it worked fine. Now it is throwing error. I am sharing the details of the same. …

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-04T04:54:22.1133333+00:00
Nowduru, Sreedhar 0 Reputation points
commented 2024-04-12T06:32:53.1066667+00:00
PRADEEPCHEEKATLA-MSFT 76,361 Reputation points Microsoft Employee
2 answers

How to load distinct values from parquet file to azure sql database using copy data activity in ADF,I don't want to use Dataflows

I wrote the sql query in pre-copy script in SINK of copy data activity,I am getting error as below ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-03-22T23:39:27.3666667+00:00
commented 2024-04-12T06:23:17.3733333+00:00
ShaikMaheer-MSFT 37,656 Reputation points Microsoft Employee
2 answers

Azure DataFactory Dataflow jobs stuck at queued status

We have ETLs in Azure Datafactory (North Europe Region) that use dataflows. A single run of ETL reads small file from blob storage and writes data into Azure SQL database. The ETLs have been running with success for more few months. Today the ETLs have…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-03T23:15:22.85+00:00
Singh, Gurinderbeer (ext) 15 Reputation points
answered 2024-04-12T06:14:33.5766667+00:00
phemanth 5,095 Reputation points Microsoft Vendor
2 answers

ADF Deployment - Code changes in UAT, Prod

I have a dev instance of ADF(Azure Devops repository is configured) and I am going to deploy this to UAT and prod instances using CI/CD pipeline. I have a question in terms of code maintenance in repository. As all code will be in main branch, and that…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-01T07:19:27.2966667+00:00
Santhi Dhanuskodi 125 Reputation points
commented 2024-04-12T05:31:29.5033333+00:00
ShaikMaheer-MSFT 37,656 Reputation points Microsoft Employee
2 answers

How to refresh table with deadlock?

We Azure postgresql database build. on that we have a ADF process that is doing truncate and load from source schema to prod schema every morning. Teams are querying to prod schema continuously to pull data which causes deadlock on table. then adf…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
asked 2024-04-03T18:34:12.2933333+00:00
Grover, Jyoti 40 Reputation points
answered 2024-04-12T05:28:36.2833333+00:00
ShaikMaheer-MSFT 37,656 Reputation points Microsoft Employee