Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
1 answer
One of the answers was accepted by the question author.
Latest folder or file extraction (Azure Data factory)
I have 7 years of data stored in separate date folders as partitions or I have 7 years of files in the directory. Requirement is to read the latest file or folder from the directory and then process further. I was able to achieve this using…
Azure Data Factory
asked 2021-06-24T10:55:20.957+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 66%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
Balasubramanian, N (Nagarajan)
21
Reputation points
commented 2021-07-02T09:56:57.667+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
How to converts string data to Base64 using azure data factory
How to converts string data to Base64 using data factory
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-23T10:47:13.167+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 66%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Praveen Kumar
21
Reputation points
accepted 2021-07-02T07:50:59.483+00:00
Praveen Kumar
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Data factory - ForEach Activity error - Next on empty iterator
Hello , I have created a pipeline with two nested foreach activities. The first one is extracting a list of files to load. The second loop is iterating through each file to join them together. In other words, I have a object with two…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-29T14:29:42.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 47%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Demarbre Sebastien
96
Reputation points
accepted 2021-07-02T07:26:09.337+00:00
Demarbre Sebastien
96
Reputation points
1 answer
One of the answers was accepted by the question author.
How to save and load ML model with Azure Data Factory
I have an Azure Data factory that receives data from a service bus and then I want to classify my data with an ML model. Is there any solution to save and load the ML model on the Azure Data Factory pipeline? For your information, I want to use cloud…
Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,350 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-14T08:03:31.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 43%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Mohsen Akhavan
831
Reputation points
accepted 2021-07-02T06:03:46.513+00:00
Mohsen Akhavan
831
Reputation points
1 answer
One of the answers was accepted by the question author.
How to identify and copy the most recently added files in Azure Data Factory when there are multiple sub-folders?
The way folders are structured in DF is something like this: Parentfolder/Subfolder1/Subfolder12/Subfolder13/File1 Parentfolder/Subfolder2/Subfolder22/Subfolder23/File2 Parentfolder/Subfolder3/Subfolder32/Subfolder33/File3 The Goal is create a…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-05-02T18:44:21.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 43%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
AHMAD, SYED [Deloitte]
21
Reputation points
commented 2021-07-02T05:01:39.05+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 31%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Roohiya Dudekula
96
Reputation points
0 answers
While ADF pipelines are running we are facing intermittent key_vault issue
Hi All, Hope you are all doing well! We are facing an intermittent key_vault issue while the ADF pipelines are running. The error it will show is "Failed to fetch secret from the key_vault" and we are not getting any specific pattern for…
Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,456 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-29T15:02:33.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 67%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENN%3C/text%3E%3C/svg%3E)
Nikki Nik
1
Reputation point
commented 2021-07-02T04:46:38.06+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
1 answer
Azure pricing calcultor help?
I'm looking to create a backup of a few storage containers using a copy activity and a delete to remove the old data in one pipeline once the new has been written. I'm struggling to understand the pricing calculator, am I just paying for the x amount…
Azure Storage
Azure Storage
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,548 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-29T08:43:45.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 98%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
JH-AM
41
Reputation points
commented 2021-07-02T04:45:10.983+00:00
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
5 answers
One of the answers was accepted by the question author.
Failed to access snowflake remote file error 2200
when try to query data from Snowflake into an Azure Data Explorer using Azure Data Factory connector. There is an error message as following: …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 72%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
0 answers
Copy data from Microsoft Access succeeds first pipeline run, fails all later runs
Using a self hosted Integration runtime installed on an Azure VM Windows server 2019, and the Microsoft Access accdb ODBC Driver. When I execute the first debug run of an activity after starting the Integration Runtime, the copy activities succeed, on…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-22T12:25:39.157+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 84%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJC%3C/text%3E%3C/svg%3E)
Joshua Christian
1
Reputation point
commented 2021-07-01T23:16:51.78+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,491
Reputation points Microsoft Employee
Moderator
1 answer
ADF Pipeline for Dynamics CRM on different domain
We have requirement to build an ADF pipeline which is on one domain and target - Dynamics CRM on another domain. Is this cause any issue while pipline execution.
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-22T10:24:48.463+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 57.00000000000001%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sarvesh Pandey
141
Reputation points
commented 2021-07-01T23:14:55.147+00:00
HimanshuSinha-msft
19,491
Reputation points Microsoft Employee
Moderator
0 answers
Apparantly failed Parallel Execution of Multiple Pipelines from Foreach Activity in Azure Data Factory
I have an ADF (Controller)pipeline which i use to train ML models on Azure Databricks. I need to train models on multiple datasets. So i have created another ADF ( Worker) pipeline within which i use Azure Databricks Activity to train a model using…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-17T04:17:10.053+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 15%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESU%3C/text%3E%3C/svg%3E)
Sushant Upadhyay
1
Reputation point
commented 2021-07-01T23:07:08.67+00:00
HimanshuSinha-msft
19,491
Reputation points Microsoft Employee
Moderator
2 answers
Add new field to USQL aggregation Task
We have a ADF Pipeline with 2 USQL Tasks, one processes the appinsights data of the day before and the second one aggregates all the processed appinsights data of the current year per Day for Power Bi Reporting. Due to business requirements we need…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-11T09:01:46.043+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 30%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHH%3C/text%3E%3C/svg%3E)
Haschtschek, Harald
1
Reputation point
commented 2021-07-01T23:05:36.477+00:00
HimanshuSinha-msft
19,491
Reputation points Microsoft Employee
Moderator
1 answer
How to leverage CheckMD5 to ensure today's file looks different from yesterday's file
I am working in Azure Synapse Analytics and am wanting to ensure today's file is different from yesterday's file in my pipeline. My naming convention for files is filename1YYYYMMDD.csv. I want to leverage MD5 to check to ensure each file is not the same,…
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,563 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
5,385 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-10T20:52:01.503+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 45%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Moore, Payton E
101
Reputation points
commented 2021-07-01T23:04:42.1+00:00
HimanshuSinha-msft
19,491
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
azure data factory get metadata not filtering by last modified date
I'm trying to find if a file exists in blob storage using the get metadata function, but only if has been modified in the last 5 minutes.. I am using the following settings: This sends the below json: `{ "dataset": { …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 7.000000000000001%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
0 answers
Azure Data factory convert string to decimal with precision shifting
Hello All, I am new to Azure Data factory. I am trying to convert a given string to decimal using the toDecimal(). There is column called xyz with values 253612375871010...1601... now i have to use the above value as expression to another column. …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-28T13:36:24.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 60%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
SrinivasMurthy D
1
Reputation point
commented 2021-07-01T06:34:34.777+00:00
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
1 answer
Dynamically changing self hosted IR's via Advance JSON
Hi, I have 2 self hosted integration runtimes in different data centres across different countries, I am trying to switch self hosted IR "on fly" using Advance JSON properties of Linked Service, I am able to pass all parameters with success…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-28T09:08:44.997+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 95%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Ahsan Khawaja
1
Reputation point
commented 2021-07-01T06:33:02.373+00:00
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
ADF Unable to write to BLOB Storage using Private end Point
Can any one please help on this, we are stuck on this and this is in production. We are getting this error message in ADF while writing to the BLOB (ADLS Gen2) storage.( "Failure happened on 'Sink' side. Error…
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
3,210 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-18T19:47:25.787+00:00
Gurvinder Kandhola
46
Reputation points
commented 2021-07-01T02:53:13.13+00:00
PRADEEPCHEEKATLA
90,736
Reputation points
Moderator
2 answers
Azure Data Factory: Copy a file from SharePoint to Blob
Hi, I am trying this method (https://learn.microsoft.com/en-us/azure/data-factory/connector-sharepoint-online-list#copy-file-from-sharepoint-online) but I get the error below when trying to copy a excel file from SharePoint to Azure Blob: Failure…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-03-25T14:06:41.773+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 28.000000000000004%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETT%3C/text%3E%3C/svg%3E)
Taariq Toffar
1
Reputation point
commented 2021-06-30T13:59:43.663+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 77%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sahoo, Priyaranjan
1
Reputation point
1 answer
One of the answers was accepted by the question author.
Azure Data Factory Trigger Run status shows as "Succeeded" for failed pipeline execution
I have a daily trigger scheduled in the azure data factory. The pipeline encountered an error upon execution. but the trigger run is showing "Succeeded" status. I am expecting the trigger run should marks status as "Failed" instead…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2020-05-08T22:45:02.593+00:00
KranthiPakala-MSFT
46,647
Reputation points Microsoft Employee
Moderator
commented 2021-06-30T08:18:20.163+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 9%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFW%3C/text%3E%3C/svg%3E)
Flemming Weyers
1
Reputation point
1 answer
One of the answers was accepted by the question author.
Failed to get the secret from key vault
I get the below issue when running a pipeline in ADF. "Message": "'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to get the secret from key vault, secretName: <<Secret_Name>>, secretVersion:…
Azure Key Vault
Azure Key Vault
An Azure service that is used to manage and protect cryptographic keys and other secrets used by cloud apps and services.
1,456 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,674 questions
asked 2021-06-24T06:42:35.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 83%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESV%3C/text%3E%3C/svg%3E)
Sukumar Vinnakota
316
Reputation points
accepted 2021-06-30T06:18:41.887+00:00
Sukumar Vinnakota
316
Reputation points