9,336 questions with Azure Data Factory tags

Sort by: Updated
2 answers

Filtering data for last 24 months in Mapping Data Flows

How can I filter data using Mapping Data Flows for the last 24 months, starting from the max date and going back? The date column is in the mm/dd/yyyy format.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,314 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,250 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-19T01:29:58.9966667+00:00
ADITYA SINGH 0 Reputation points
commented 2024-03-28T05:18:06.6166667+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
2 answers

How to load distinct values from parquet file to azure sql database using copy data activity in ADF,I don't want to use Dataflows

I wrote the sql query in pre-copy script in SINK of copy data activity,I am getting error as below ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-22T23:39:27.3666667+00:00
answered 2024-03-28T05:11:56.2666667+00:00
ShaikMaheer-MSFT 37,566 Reputation points Microsoft Employee
1 answer

Azure Data Factory Fuzzy logic implementation for multiple tables in the SQL DB

I am trying to build a solution using Azure Data Factory to implement the Fuzzy Logic for two data source which is stored in Synapse SQL DB. for i.e. Table 1 and Table 2 which has identical structures. I am trying to use ADF to build a pipelines using…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-19T03:46:26.49+00:00
MG 40 Reputation points
commented 2024-03-28T05:11:37.6933333+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
2 answers

ADF Deployment

Hi, I have a ADF dev instance which has all pipelines, dataflows , triggers which are parameterized. If I deploy this to UAT and prod, how ADF will behave? Where can I specify the parameters for these instances? how it will affect the main branch…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-19T08:53:02.5933333+00:00
Santhi Dhanuskodi 125 Reputation points
commented 2024-03-28T05:10:34.9733333+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
1 answer

How to call a REST API end point in ADF using service principal authentication?

I have an ADF workspace in MSFT tenant and I need to invoke a REST API hosted in AME tenant. REST API needs AME based token to accept any request. I have an AME Serice principal which used in the linked service for REST API. I created a REST API source…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-20T04:27:23.51+00:00
Anoop Jose 1 Reputation point Microsoft Employee
commented 2024-03-28T05:09:40.7233333+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
2 answers

How to Convert a date column from UTC to PST in COPY activity of ADF?

Hello Team, I have created one column on the fly in the **"Source" ,**in the copy activity.Now the date is appearing in UTC format. But i need to display in PST Time.So i am trying to convert from UTC to PST: @convertTimeZone(utcnow(), 'UTC',…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-20T08:16:25.5666667+00:00
Rohit Kulkarni 676 Reputation points
answered 2024-03-28T05:08:06.0666667+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
2 answers

dataset refresh issue using azure data factory

Cannot create Sql Source. Please double check the connection string, stored procedure are set with correct format. Error: The value of the property 'Value' is invalid for the stored procedure parameter 'MinTimestamp'.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-23T09:01:13.1133333+00:00
Dhruti Vyasa 0 Reputation points
answered 2024-03-28T05:00:00.5566667+00:00
AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
1 answer

Using a System-Assigned Managed Identity to Trigger a Logic App in a Different Subscription from Azure Data Factory

Can a system-assigned managed identity be used to connect to a resource in a different subscription? I want to trigger a logic app, which is in a different subscription, from my Azure Data Factory using a managed identity. Is this possible, and if so,…

Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
2,770 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-28T04:24:39.42+00:00
Ahwan Mishra 120 Reputation points
answered 2024-03-28T04:48:58.97+00:00
Vahid Ghafarpour 16,290 Reputation points
1 answer

How to schedule trigger when pipeline execution time greater than trigger time?

Trigger created to run every 15 min on daily basis. In case first trigger takes more than 15 minutes to complete and second trigger starts without completing first trigger. How to handle this scenario? Which trigger (Schedule / Tumbling window) needs to…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-28T02:26:17.7866667+00:00
Anuganti Suresh 140 Reputation points
edited an answer 2024-03-28T04:27:49.44+00:00
Harishga 2,790 Reputation points Microsoft Vendor
0 answers

Dynamic Data Masking with Azure Data Factory

How can I mask specific columns when copying data from an on-premises source to Azure Data Lake Gen 2 using Azure Data Factory? Is it possible to use one data flow and a control table containing the table and column names to be masked for this purpose?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-28T03:35:35.2266667+00:00
Tushar Hatwar 0 Reputation points
commented 2024-03-28T04:13:34.14+00:00
Nandan Hegde 29,796 Reputation points MVP
0 answers

Thank You to the Microsoft Q&A Community Champions

We would like to celebrate this quarter's Community Champions for their great contribution to the community on Microsoft Q&A! Microsoft Q&A Community Champions program recognizes external technology experts who contribute to the Microsoft Q&A…

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,112 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,011 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
6,685 questions
Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
19,089 questions
asked 2024-03-26T05:26:08.93+00:00
tbgangav-MSFT 10,371 Reputation points
commented 2024-03-28T04:09:41.9+00:00
tbgangav-MSFT 10,371 Reputation points
0 answers

Is this possible to deploy alerts and metrics from one env to another env

Hi, WE have github action CICD process to deploy the ADF pipeline using ARM template. However we have to deploy alerts and metrics created in one env to another env. The ARM template not capturing the alerts and metrics. so what would be the possible…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-26T15:36:33.3933333+00:00
Umesh Narayanan 0 Reputation points
edited a comment 2024-03-28T03:19:08.7833333+00:00
Harishga 2,790 Reputation points Microsoft Vendor
2 answers

Azure Data Factory to DataVerse BypassCustomPluginExecution

Hi. I have a CopyActivity, to upsert records to DataVerse environment. I need to bypass custom logic, to have better performance. https://learn.microsoft.com/en-us/power-apps/developer/data-platform/bypass-custom-business-logic?tabs=webapi How Can I set…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2023-02-07T18:52:30.22+00:00
Diego Gaston Arrighi 35 Reputation points
answered 2024-03-28T03:10:26.3233333+00:00
Thales Y X LUO 0 Reputation points
1 answer

Connection between Delta Lake of DataFactory and HDInsight Spark

Hi Everyone, Below is the my design: I am using ADF to ingest the data from Oracle DB to Raw Layer. Then I am using ADF DataFlow to create and load data from RawLayer to DeltaLake. PFA. Now I want to use HDinsight Spark cluster for further…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-26T06:56:39.2833333+00:00
Vaibhav 45 Reputation points
commented 2024-03-28T03:07:15.74+00:00
Harishga 2,790 Reputation points Microsoft Vendor
1 answer

how to convert the time from epoch or Unix Time in ADF activity

how to convert the time from epoch or Unix Time in ADF activity like set,lookup etc.. ex: i have one folder in S3 bucket but the folder name is in Unix time or epoch time ex:17908765 i need to convert the folder to datetime/date format how can i achieve?…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-25T07:08:56.7+00:00
commented 2024-03-28T03:00:15.38+00:00
1 answer One of the answers was accepted by the question author.

How to implement check points in my existing design to avoid duplicates in sinkin ADF

Hi, In adf one of the data flow is failed due to transist error in the middle of execution.the data flow is having insert ,update and delete operation based on scd2 scenario.To avoid the transist failures we have applied retry option.But while retrying…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-22T12:19:32.9666667+00:00
Sarag 61 Reputation points
accepted 2024-03-27T22:53:28.1033333+00:00
Sarag 61 Reputation points
0 answers

Operation on target Databricks_Transformation_Operation failed: Invalid access token.

I encountered an error in my Azure Data Factory pipeline during the copy data activity completion. Subsequently, when transitioning to the transformation stage in a Databricks notebook, I encountered the following error:. Interestingly, the pipeline was…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-27T11:20:20.56+00:00
Vaibhav Patil 20 Reputation points
commented 2024-03-27T20:06:09.81+00:00
BhargavaGunnam-MSFT 25,071 Reputation points Microsoft Employee
0 answers

Change Data Capture for On-premises SQL Server

Hi, I want to get on-premises SQL Server data into Microsoft Fabric by using change data capture feature in Azure Data Factory Dataflow, what are the steps i need to follow(I tried SHIR but data flow doesn't support ).I followed the below doc but unable…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-27T09:45:38.4433333+00:00
Ramakiran Gadde 0 Reputation points
edited a comment 2024-03-27T18:25:08.6466667+00:00
BhargavaGunnam-MSFT 25,071 Reputation points Microsoft Employee
1 answer

Cannot publish pipelines in Azure data factory.

A user that is set up as an owner, contributor and data factory contributor for a data factory resource cannot publish pipelines. Is there a specific role or setting required for this? DevOps is not involved, just directly publishing a pipeline.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-08T11:58:47.9266667+00:00
Brian Sullivan 5 Reputation points
commented 2024-03-27T16:11:11.0233333+00:00
Tjomme Vergauwen 0 Reputation points
2 answers

Copy Activity allows HTTP Post to get Excel but when debugging it fails

So I am trying to import an excel file from a website that uses a HTTP Post to pull the file. I have configured the linked service, dataset and then the Copy Activity. In the copy activity I am able to specifiy it use a Post method. When I click…

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,336 questions
asked 2024-03-18T22:51:22.9666667+00:00
Steve Dunker 20 Reputation points
answered 2024-03-27T15:53:06.4266667+00:00
Steve Dunker 20 Reputation points