Filtering data for last 24 months in Mapping Data Flows
How can I filter data using Mapping Data Flows for the last 24 months, starting from the max date and going back? The date column is in the mm/dd/yyyy format.
How to load distinct values from parquet file to azure sql database using copy data activity in ADF,I don't want to use Dataflows
I wrote the sql query in pre-copy script in SINK of copy data activity,I am getting error as below ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following…
Azure Data Factory Fuzzy logic implementation for multiple tables in the SQL DB
I am trying to build a solution using Azure Data Factory to implement the Fuzzy Logic for two data source which is stored in Synapse SQL DB. for i.e. Table 1 and Table 2 which has identical structures. I am trying to use ADF to build a pipelines using…
ADF Deployment
Hi, I have a ADF dev instance which has all pipelines, dataflows , triggers which are parameterized. If I deploy this to UAT and prod, how ADF will behave? Where can I specify the parameters for these instances? how it will affect the main branch…
How to call a REST API end point in ADF using service principal authentication?
I have an ADF workspace in MSFT tenant and I need to invoke a REST API hosted in AME tenant. REST API needs AME based token to accept any request. I have an AME Serice principal which used in the linked service for REST API. I created a REST API source…
How to Convert a date column from UTC to PST in COPY activity of ADF?
Hello Team, I have created one column on the fly in the **"Source" ,**in the copy activity.Now the date is appearing in UTC format. But i need to display in PST Time.So i am trying to convert from UTC to PST: @convertTimeZone(utcnow(), 'UTC',…
dataset refresh issue using azure data factory
Cannot create Sql Source. Please double check the connection string, stored procedure are set with correct format. Error: The value of the property 'Value' is invalid for the stored procedure parameter 'MinTimestamp'.
Using a System-Assigned Managed Identity to Trigger a Logic App in a Different Subscription from Azure Data Factory
Can a system-assigned managed identity be used to connect to a resource in a different subscription? I want to trigger a logic app, which is in a different subscription, from my Azure Data Factory using a managed identity. Is this possible, and if so,…
How to schedule trigger when pipeline execution time greater than trigger time?
Trigger created to run every 15 min on daily basis. In case first trigger takes more than 15 minutes to complete and second trigger starts without completing first trigger. How to handle this scenario? Which trigger (Schedule / Tumbling window) needs to…
Dynamic Data Masking with Azure Data Factory
How can I mask specific columns when copying data from an on-premises source to Azure Data Lake Gen 2 using Azure Data Factory? Is it possible to use one data flow and a control table containing the table and column names to be masked for this purpose?
Thank You to the Microsoft Q&A Community Champions
We would like to celebrate this quarter's Community Champions for their great contribution to the community on Microsoft Q&A! Microsoft Q&A Community Champions program recognizes external technology experts who contribute to the Microsoft Q&A…
Is this possible to deploy alerts and metrics from one env to another env
Hi, WE have github action CICD process to deploy the ADF pipeline using ARM template. However we have to deploy alerts and metrics created in one env to another env. The ARM template not capturing the alerts and metrics. so what would be the possible…
Azure Data Factory to DataVerse BypassCustomPluginExecution
Hi. I have a CopyActivity, to upsert records to DataVerse environment. I need to bypass custom logic, to have better performance. https://learn.microsoft.com/en-us/power-apps/developer/data-platform/bypass-custom-business-logic?tabs=webapi How Can I set…
Connection between Delta Lake of DataFactory and HDInsight Spark
Hi Everyone, Below is the my design: I am using ADF to ingest the data from Oracle DB to Raw Layer. Then I am using ADF DataFlow to create and load data from RawLayer to DeltaLake. PFA. Now I want to use HDinsight Spark cluster for further…
how to convert the time from epoch or Unix Time in ADF activity
how to convert the time from epoch or Unix Time in ADF activity like set,lookup etc.. ex: i have one folder in S3 bucket but the folder name is in Unix time or epoch time ex:17908765 i need to convert the folder to datetime/date format how can i achieve?…
How to implement check points in my existing design to avoid duplicates in sinkin ADF
Hi, In adf one of the data flow is failed due to transist error in the middle of execution.the data flow is having insert ,update and delete operation based on scd2 scenario.To avoid the transist failures we have applied retry option.But while retrying…
Operation on target Databricks_Transformation_Operation failed: Invalid access token.
I encountered an error in my Azure Data Factory pipeline during the copy data activity completion. Subsequently, when transitioning to the transformation stage in a Databricks notebook, I encountered the following error:. Interestingly, the pipeline was…
Change Data Capture for On-premises SQL Server
Hi, I want to get on-premises SQL Server data into Microsoft Fabric by using change data capture feature in Azure Data Factory Dataflow, what are the steps i need to follow(I tried SHIR but data flow doesn't support ).I followed the below doc but unable…
Cannot publish pipelines in Azure data factory.
A user that is set up as an owner, contributor and data factory contributor for a data factory resource cannot publish pipelines. Is there a specific role or setting required for this? DevOps is not involved, just directly publishing a pipeline.
Copy Activity allows HTTP Post to get Excel but when debugging it fails
So I am trying to import an excel file from a website that uses a HTTP Post to pull the file. I have configured the linked service, dataset and then the Copy Activity. In the copy activity I am able to specifiy it use a Post method. When I click…