Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,205 questions
2 answers
ADF Until activity slow to terminate
I have a number of ADF pipelines that I'm using to download historic transactional data from various external REST APIs. Because of the data volumes involved, the pipelines I've created run for many hours. The API pagination mechanisms are such that I…
asked 2024-07-25T14:58:37.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 64%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJW%3C/text%3E%3C/svg%3E)
Jez Walters
20
Reputation points
answered 2024-07-26T10:00:26.1666667+00:00
Jez Walters
20
Reputation points
1 answer
One of the answers was accepted by the question author.
How can i get the outbound IP address for Azure Data Factory ?
I need to find the outbound IP of ADF, I see in properties tab it doesn't have IP address list. Is there a way I can get this list ?
asked 2021-05-18T08:29:10.667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 96%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Shubham Shukla
21
Reputation points
edited a comment 2024-07-26T09:44:58.0066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 32%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESW%3C/text%3E%3C/svg%3E)
Sacher, Wolf-Rüdiger
21
Reputation points
0 answers
Installing IBMi Access Driver on Azure Data Factory Integration Runtime
Hello, I've been trying to install the Windows version of the IBM i Access Driver on a Data Factory Integration Runtime through adding a script (main.cmd) and the desired installer in a Blob Container This script seems to work perfectly locally but when…
asked 2024-07-24T07:19:18.0966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 97%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Alexandru Moise
0
Reputation points
commented 2024-07-26T09:11:56.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,620
Reputation points Microsoft Vendor
1 answer
load data from ADLS to Unity Catalog tables using Azure Data Factory?
What is the best way to load data from ADLS to Unity Catalog tables using Azure Data Factory?
asked 2024-07-24T16:08:45.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 20%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
chandrasekhar munagala
21
Reputation points
commented 2024-07-26T09:08:22.26+00:00
Smaran Thoomu
12,620
Reputation points Microsoft Vendor
1 answer
We have a storage event trigger in adf kicking off pipeline to process file too early?
We have a Storage event trigger, that is set to kick off a Data Factory Pipeline(To process files from blob container folder path ) onto a sql server database when a file is dropped on to the location. The DataFactory Pipeline is a simple pipeline with…
asked 2024-07-24T13:38:51.8033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 41%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Karkal,Prajna
0
Reputation points
commented 2024-07-26T08:33:57.9866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 81%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Amrinder Singh
4,910
Reputation points Microsoft Employee
5 answers
Script Activity not supporting multi-instruction with new Snowflake connector
I have a pipeline, in Azure Data Factory that has an "Activity Script" that executes 3 instructions against Snowflake. The script was working fine until I have updated the Snowflake connection in order to remove the legacy Snowflake connection…
asked 2024-03-21T08:23:45.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 40%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DavidR
40
Reputation points
answered 2024-07-26T01:37:28.9666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 21%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMF%3C/text%3E%3C/svg%3E)
Marco Flores
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to Fix the InternalServerError in Executing a ADF Pipeline?
I have a foreach loop in my Azure Data Factory pipeline with three activities. The first activity uses a Self-Hosted Integration Runtime (SHIR) to copy data from an On-premises SQL Server to Azure Data Lake Storage Gen2 (ADLS2). The second activity is a…
asked 2024-07-22T17:39:46.4166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 95%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENP%3C/text%3E%3C/svg%3E)
Nitin Patil
20
Reputation points
accepted 2024-07-25T19:29:48.0333333+00:00
Nitin Patil
20
Reputation points
0 answers
Power Query activity in Azure Data Factory - Not supported in region
I've done multiple Bing and Google searches on variations of the following error but haven't found anything helpful. Is a Power Query activity really not supported in the region specified? Is there an article somewhere that explains where this activity…
asked 2022-02-25T03:45:58.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 85%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWP%3C/text%3E%3C/svg%3E)
Walter Pelowski
76
Reputation points
commented 2024-07-25T18:15:19.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 47%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJP%3C/text%3E%3C/svg%3E)
Jacob Prouty
0
Reputation points
1 answer
Inserting Error Logs from Data Factory into Databricks Delta Table (Catalog) - Streaming vs. Batch
Hi everyone, I'm working on a solution to capture and store error logs generated by my Azure Data Factory pipelines. My goal is to insert these logs into a Databricks Delta Table (catalog) for further analysis and troubleshooting. I'm considering two…
asked 2024-07-22T23:40:52.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 70%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEA%3C/text%3E%3C/svg%3E)
Elliot Alderson
0
Reputation points
commented 2024-07-25T16:58:22.1366667+00:00
Smaran Thoomu
12,620
Reputation points Microsoft Vendor
3 answers
SQL Server wont connect to data factory
I recently set up a SQL server and database. However, when I try to connect it to data factory using SQL Authentication, i get the following error Cannot connect to SQL Database. Please contact SQL server team for further support. Server:…
asked 2024-07-19T12:52:59.1733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 4%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Sage Desiderio
0
Reputation points
commented 2024-07-25T16:47:49.56+00:00
Smaran Thoomu
12,620
Reputation points Microsoft Vendor
1 answer
Facing an issue to install dbt-core==1.6.0 and dbt-snowflake==1.6.0
Happens that we’re now facing an issue to install dbt-core==1.6.0 and dbt-snowflake==1.6.0. We found out that ADF UI appears to enable only packages from apache-airflow-providers, and not any that can be installed with pip. (The ones listed here…
asked 2024-07-12T04:30:08.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 50%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Siemens Healthineers Madivalappa_Azure Account
0
Reputation points
commented 2024-07-25T16:41:04.4866667+00:00
Smaran Thoomu
12,620
Reputation points Microsoft Vendor
1 answer
Efficient Log Handling and Data Retention in Azure Data Factory and Databricks
I need to create a solution to send logs from Azure Data Factory to the Databricks Unity Catalog. I'm considering the following structure: Whenever an activity run results in either failure or success, the corresponding log will be sent to Azure Logic…
asked 2024-07-17T19:44:42.0666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 31%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Hanna
220
Reputation points
commented 2024-07-25T15:34:53.78+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
555
Reputation points Microsoft Vendor
1 answer
Azure Synapse Analytics Data Flow: How do the Table Actions and Update Methods work & can you conditionally Truncate with 1 Data Flow?
I am working in Azure Synapse Analytics and making a dataflow, but I'm struggling with understanding some things. (I'm using Azure Data Lake Storage Gen 2 and Delta file for my sink) For the sink activity: what is the difference between the update…
asked 2024-07-18T21:18:12.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 94%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Joshua
0
Reputation points
commented 2024-07-25T15:32:47.1866667+00:00
Chandra Boorla
555
Reputation points Microsoft Vendor
1 answer
Question regarding for each loop in ADF
I'm trying to create a new pipeline in ADF and my requirement is as follows. I have a table 'abc' in my database which contains a list of table names in it. Now, I want to execute a stored procedure which is written in such a way that it checks the data…
asked 2024-07-23T04:50:32.59+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 3%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
Kenaniah Santhosh
6
Reputation points
commented 2024-07-25T15:28:36.0333333+00:00
Chandra Boorla
555
Reputation points Microsoft Vendor
7 answers
Put Blob REST API Content-Length error
I'm trying to write the XML response obtained from a REST API GET request to a file in Azure Blob Storage, using Azure Data Factory. I'd prefer to use the Copy activity to achieve this, but it doesn't support XML sinks. I've therefore been forced into…
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,644 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,205 questions
Azure Startups
Azure Startups
Azure: A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.Startups: Companies that are in their initial stages of business and typically developing a business model and seeking financing.
240 questions
asked 2024-05-07T10:06:05.8133333+00:00
Jez Walters
20
Reputation points
answered 2024-07-25T15:23:41.3466667+00:00
Jez Walters
20
Reputation points
1 answer
Datetime2 error when querying parquet file produced from Synapse copy activity from OData Business Central Source (data seems to be changed from 0001-01-01 to 0000-12-30)
I have a pipeline running a copy activity to copy data from Business Central OData source to parquet file everyday. Everything works fine before 16/07/2024 but problem arises starting from 17/07/2024. The parquet file no longer be able to query from…
asked 2024-07-24T15:13:17.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 10%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVC%3C/text%3E%3C/svg%3E)
Victor Chu
0
Reputation points
edited a comment 2024-07-25T15:11:18.5566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
29,276
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to process this JSON structure
Hi, i get this structure back from a web activity. Showing only a part { "Response": "[\r\n {\r\n "CreationDate": "2024-01-07T00:00:00Z",\r\n "Operation": "ViewReport",\r\n "UserId":…
asked 2024-07-23T15:27:30.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 92%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPV%3C/text%3E%3C/svg%3E)
Poel van der, RE (Ron)
451
Reputation points
accepted 2024-07-25T14:21:14.5566667+00:00
Poel van der, RE (Ron)
451
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory Writing to Business Central Purchase Header and Purchase Lines
Hello @Anonymous I am working on posting data to Business Central using ADF from CSV files to Purchase Header and Purchase Lines tables. I am having the following error on trying to post the data. The sink is a REST. I have checked my permissions on…
asked 2024-07-24T13:59:56.7166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 28.000000000000004%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKO%3C/text%3E%3C/svg%3E)
Kayode Ogidan
85
Reputation points
commented 2024-07-25T13:41:51.87+00:00
Chandra Boorla
555
Reputation points Microsoft Vendor
1 answer
How to handle an excel sheet(only 1 tab) with multiple datasets with different columns and ranges
Hi Team, Could you help me out with below Scenario of ADF? I have a single Excel file with just one tab, as displayed below with different columns and that tab contains 2 datasets. We want to move this data to 2 different SQL tables using ADF/data flow.
asked 2024-07-24T12:35:32.0766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 56.00000000000001%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Spandana Mogadala
0
Reputation points
commented 2024-07-25T06:26:30.5166667+00:00
Spandana Mogadala
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Delimited Text(CSV) dataset escape/quote character
Hi, I am facing an issue with delimited text(CSV) dataset, whne my data contains comma. So I added quote character in configuration. But this double quotes character is applied for most of the column values, even though the column value doesnt contain…
asked 2024-07-24T10:52:13.6533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Santhi Dhanuskodi
225
Reputation points
commented 2024-07-25T06:18:57.83+00:00
Santhi Dhanuskodi
225
Reputation points