Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,534 questions
1 answer
Lookup Different Source Query Based on Pipeline Input Param Value
Hello, I have a pipeline with a parameter where I need to perform a Lookup on the source query based on the input and then process that data. I cannot use a Switch Activity as I am utilizing a ForEach Activity for further data processing. Is it possible…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 10%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
0 answers
ADF pipeline sink not showing any directory while browsing.
I have created a azure pipeline to connect azure blob storage to sftp server. The linked services are created and tested successfully with proper IP whitelisting. But when I am creating the azue pipeline and configuring the sink, I get connection…
asked 2024-09-05T07:56:32.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 35%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERV%3C/text%3E%3C/svg%3E)
Ridhima Verma
25
Reputation points
commented 2024-09-09T15:47:02.62+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
30,576
Reputation points Microsoft Employee
0 answers
Moving data from Snowflake to Azure SQL DB using Copy Activity
Hi All, I am using COPY activity to move data from Snowflake to Azure SQL DB. First of all, this data movement is not allowed without intermediate staging of the data in ADLS. I would love to avoid this intermediate step. If this is how it will work then…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 27%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
1 answer
Load different json files
How to copy 3 json files with different data without union as it has different columns... Without adding columns mannually due to many columns { "id": 10000, "id": null, "data": { "one":…
asked 2024-09-09T11:51:32.6733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
750
Reputation points
answered 2024-09-09T14:12:24.0666667+00:00
Amira Bedhiafi
22,691
Reputation points
1 answer
How can we synchronize data from Azure Cosmos DB to Snowflake using Azure Data Factory (ADF) with a Change Data Capture (CDC) mechanism
We have data stored in Cosmos DB NoSQL and need to migrate it to Snowflake using Azure Data Factory (ADF) with a Change Data Capture (CDC) approach. Our objective is to perform CRUD operations based on CDC to handle all data changes. This is our…
asked 2024-09-09T12:33:15.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 91%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Sowndar Kumaresan
0
Reputation points
answered 2024-09-09T13:26:31.97+00:00
Vinodh247
16,831
Reputation points
3 answers
Data Factory Copy function is not copying table correctly. Columns order is not correct.
I have a simple pipeline with 1 single copy function. It copies the calendar table from salesforce to a sql server database. This works perfect in our dev data factory. We are using dev salesforce and dev sql server. We moved this to our test data…
asked 2024-08-12T11:41:37.5266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 55.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBD%3C/text%3E%3C/svg%3E)
Blasko, Dan W
20
Reputation points
answered 2024-09-09T13:06:48.0233333+00:00
Blasko, Dan W
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Copy last modified file in copy activity
Hi how to copy last modified-2 hours copy from folder using copy activity from blob storage.. Pls share screenshot
1 answer
Integration runtimes are not auto-updating
Our Azure Data Factory integration runtimes are not auto-updating even though we have them set to auto-update. We checked the troubleshooting items listed, and none seem to apply. auto-update is turned on we are able to access the download center…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 47%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Microsoft Azure databricks all purpose cluster issue with jobs failure again and again
Jobs within the all_purpose Databricks Cluster are failing with "the spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached" In the event log it says "Event_type=DRIVER_NOT_RESPONDING…
asked 2024-09-05T12:12:00.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 3%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Prasant Kumar Das
20
Reputation points
accepted 2024-09-09T11:26:54.66+00:00
Prasant Kumar Das
20
Reputation points
0 answers
Azure Data Factory - Cannot open nested pipelines by double clicking on them.
Hi Everyone, My apologies if this is not logged/asked in the correct place, please point me to the correct location if needed. I am having a strange but frustrating issue with data factory. When working on pipelines, specifically nested pipelines, I…
asked 2024-09-06T08:28:15.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 64%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHV%3C/text%3E%3C/svg%3E)
Herman van Eeden
1
Reputation point
commented 2024-09-09T10:59:09.7566667+00:00
Herman van Eeden
1
Reputation point
1 answer
Fabric Data fetching API
I have created an azure app from which I need to pull user data present on their fabric workspaces (lakehouses, warehouses etc.). Fabric APIs like : GET…
asked 2024-09-05T14:50:50.1266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 63%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Shubham
0
Reputation points
commented 2024-09-09T10:21:10.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
14,310
Reputation points Microsoft Vendor
1 answer
For Each Activity to process CSV Files
I need to design a pipeline in Azure Data Factory to process multiple CSV files that are stored in an Azure Blob Storage container. Each file contains transaction data, and I need to perform the following tasks for each file: Load the data from each CSV…
asked 2024-09-09T09:32:51.9033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 53%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DataCoder
140
Reputation points
answered 2024-09-09T10:10:53.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 46%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
NIKHILANETHIKUNTA
1,510
Reputation points Microsoft Vendor
0 answers
Unable to View Business Central Tables/Entities in ADF Linked Service
I am able to successfully connect to Business Central using the Dynamics 365 connector in Azure Data Factory (ADF). However, when trying to list the available entities, I am not seeing the expected Business Central tables or APIs. Instead, I’m getting…
asked 2024-09-05T19:17:12.3733333+00:00
Vaibhav Patil
235
Reputation points
commented 2024-09-09T09:13:36.0866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
10,085
Reputation points Microsoft Vendor
2 answers
Azure data factory and secure access
What are the secure options available to prevent ADF and ADLS from cyber threats and other security vulnerabilities besides accessing the services through private link and express routes and using vnet and private endpoints? Please suggest a holistic…
asked 2024-08-30T10:58:08.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
200
Reputation points
commented 2024-09-09T07:34:03.2166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
7,131
Reputation points Microsoft Vendor
1 answer
Dynamic Mapping of JSON Sink Data in ADF
Hello Microsoft Community, I hope you're all doing well. I'm currently working on a project in Azure Data Factory and need some advice! We are successfully able to preview source data from the Salesforce Linked Service and sink data using the REST…
asked 2024-09-06T12:06:58.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 13%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVJ%3C/text%3E%3C/svg%3E)
Vinit Joshi
5
Reputation points
commented 2024-09-09T07:10:03.2466667+00:00
NIKHILANETHIKUNTA
1,510
Reputation points Microsoft Vendor
1 answer
How do I get the current date in an ADF Data Flow Sink action filename?
This is in Azure Data Factory. I have a data flow which has a sink to output to a .CSV file. I'd like to make the filename have the date like this: "out_20240906.csv". On the Sink action there is a "Filename" field where I can enter…
asked 2024-09-06T12:17:32.2633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 77%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECR%3C/text%3E%3C/svg%3E)
Chuck Roberts
85
Reputation points
commented 2024-09-09T07:08:38.4733333+00:00
NIKHILANETHIKUNTA
1,510
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Need to create pipeline to execute subfolder file
Hi Team, I wanted to create a pipeline with XML dataset , ADLS gen2 storage is in hierarchy as - Rootfolder>subfolder>IN>OUT>STOCK>2024>07>31 --> Inside 31 folder i have XML file likewise in monthly folder i have datewise folder…
asked 2024-09-02T13:06:46.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 1%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Samriddhi Gupta
20
Reputation points
commented 2024-09-09T05:48:58.35+00:00
Samriddhi Gupta
20
Reputation points
1 answer
adf postgresql linked service connector settings
We need to update our ADF postgresql connector to the newer version from the legacy version and in order to make it work we've had to add some settings. Can anyone tell me if these settings could have a negative impact? Our pipelines were failing before…
1 answer
getting CLUSTER_CREATION_TIMED_OUT issue
Hi, my jobs are failing without spark session is starting and getting timed out issue. all my pools are XXL configuration but inside pipeline calling Small or medium.
asked 2024-09-06T22:45:27+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 99%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVP%3C/text%3E%3C/svg%3E)
Venkateswarlu Pottapalli
0
Reputation points
commented 2024-09-09T04:51:32.9833333+00:00
Vinodh247
16,831
Reputation points
2 answers
Data factory pipeline execution fails on SqlParameter
Since a couple weeks our data factory pipelines sometimes fail due to below error message. We are running for approx 1,5 year in production and never experienced this error. So it is really strange why this happens. Also it does not happen each run,…
asked 2023-04-05T06:35:29.6633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 43%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jordi
5
Reputation points
answered 2024-09-07T19:31:45.4033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 78%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vladyslav Kabachnyi
0
Reputation points