Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,397 questions
1 answer
Temp table in adf
How to use temp table in adf for sql server without creating store procedure
asked 2024-08-15T21:51:25.5166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
670
Reputation points
answered 2024-08-16T12:20:08.52+00:00
Amira Bedhiafi
21,591
Reputation points
0 answers
Error in SOQL Query with Salesforce V2 dataset
Hi, I'm attempting to use the SOQL Query option to filter columns in the Salesforce dataset, but I'm running into the following error. Any help would be appreciated. Thanks!
asked 2024-08-08T12:42:16.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 94%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Roxan Sadia
0
Reputation points
commented 2024-08-16T00:04:29.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 46%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
NIKHILANETHIKUNTA
635
Reputation points Microsoft Vendor
1 answer
Data Factory to pull information from Snowflake using external stage to Blob Storage
We are creating an ADF pipeline to pull information from Snowflake to Blob Storage. Reading the help article https://learn.microsoft.com/en-us/azure/data-factory/connector-snowflake?tabs=data-factory I am getting some pre-requisites to use this process…
asked 2024-08-07T18:33:18.5833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 9%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE%3C/text%3E%3C/svg%3E)
Enrique
0
Reputation points
commented 2024-08-16T00:03:33.5333333+00:00
NIKHILANETHIKUNTA
635
Reputation points Microsoft Vendor
1 answer
Business Central Connection with Azure Data Factory
Hello, is there a way to connect Business Central to Azure Data Lake? I am only seeing one for NAV/AX and D365 CRM Apps.
asked 2024-08-07T19:38:39.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 18%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAH%3C/text%3E%3C/svg%3E)
Andrew Hairston Cloud
0
Reputation points
commented 2024-08-16T00:02:21.19+00:00
NIKHILANETHIKUNTA
635
Reputation points Microsoft Vendor
1 answer
Convert query in adf
Hi, how to convert the query of temp table and variable in adf sql query.. Am unable to convert.. Tried cte --/* DECLARE @Period VARCHAR(20)='202406', @Facility NVARCHAR(20)='', @ShowFacility BIT=0, @GroupBy…
0 answers
ADF and external source data handle
I require that the business has more than 60 packages(sources are ERP and other applications), which must be refactored into ADF pipelines and moved to Azure Storage services. The issue is all these packages have complex multi-step logic with…
asked 2024-08-14T11:52:39.4333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
120
Reputation points
edited a comment 2024-08-15T19:11:47.3866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT
29,721
Reputation points Microsoft Employee
2 answers
Error using Azure Data Factory to Copy data (Using Upsert) from Azure Blob to Azure SQL database
Hi all, I keep getting this error when I perform an upsert (under copy activity) with this error code: Failure happened on 'Sink' side. 'Type=System.NullReferenceException,Message=Object reference not set to an instance of an…
asked 2024-05-23T23:34:36.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 0%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERL%3C/text%3E%3C/svg%3E)
Rome Lin
5
Reputation points
commented 2024-08-15T17:59:24.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 8%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBG%3C/text%3E%3C/svg%3E)
Bleakley, Geoff
0
Reputation points
1 answer
One of the answers was accepted by the question author.
How to Work with UTC Timestamps in Azure Data Factory
I need assistance in writing a Data Flow expression in Azure Data Factory to generate the current UTC timestamp in the format yyyy-MM-ddTHH:mm:ss.SSS. Additionally, I would like to understand how to create expressions that allow me to both subtract and…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 53%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
0 answers
Storing JSON in Azure SQL Table
How can I store JSON responses received from a web activity within Azure Data Factory in an Azure SQL table? The JSON output contains special characters that need to be preserved in the table. I have tried passing the JSON as a string to the stored…
asked 2024-08-14T07:09:39.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 90%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Sodi, Deepthika
0
Reputation points
commented 2024-08-15T16:04:40.7266667+00:00
Bhargava-MSFT
29,721
Reputation points Microsoft Employee
2 answers
ReferenceError: Cannot access 'ne' before initialization when creating Airflow environment setup in Azure Data Factory.
I am trying to create the Airflow environment with Data Factory through the web. I specified just the required default parameters, but I am facing ReferenceError: Cannot access 'ne' before initialization. I don't know what it's trying to call under the…
asked 2024-08-07T13:34:46.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 33%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kateryna Barabash
10
Reputation points
commented 2024-08-15T15:08:53.1366667+00:00
Kateryna Barabash
10
Reputation points
1 answer
SFTP Requirement
Hello Team, I want to access the files from the SFTP drive and read the the file and upload into azure sql database via azure data factory pipeline. I tired to find the good document. But not able to find. Please any one can share the suitable link to…
asked 2024-08-12T07:44:06.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rohit Kulkarni
686
Reputation points
commented 2024-08-15T14:25:06.18+00:00
Bhargava-MSFT
29,721
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Configure self-hosted integration runtime over private network
Hi, As per my understanding self-hosted integration runtime uses https over public internet to transfer data between on-premise and Azure cloud services(Blob/Database). Please correct me if I am wrong, and also advise if its possible to configure…
asked 2021-07-25T17:47:09.903+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 41%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
saurabh
356
Reputation points
commented 2024-08-15T13:02:11.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 10%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYD%3C/text%3E%3C/svg%3E)
Yordan Dimov
45
Reputation points
1 answer
ADF storage event trigger pipeline is not triggered when file is SFTP to Azure Blob Storage.
Hi, I have a file coming from on-prem app through SFTP to Azure Blob Storage container. I set up an event storage trigger on my pipeline. However the pipeline is not getting started when the file (blob) created on Azure storage container through SFTP. On…
asked 2023-09-25T13:31:34.3466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 53%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Shrivastava, Ravi-ZB
0
Reputation points
commented 2024-08-15T12:49:09.1566667+00:00
Paul Hernandez
651
Reputation points Microsoft Employee
1 answer
Business Central Connector for Azure Data Factory
Is there a connector that can be configured to allow data to be extracted from Business Central into Azure Data Factory? Please do not confuse my query for AX or Finance and Operations.
asked 2024-08-08T22:04:24.4033333+00:00
Andrew Hairston Cloud
0
Reputation points
commented 2024-08-15T11:09:42.4966667+00:00
Andrew Hairston Cloud
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Linked Services for Salesforce in Azure Data Factory
I am having issue creating linked Service in Azure Data Factory to Salesforce. I get following error Unable to establish connection to the Salesforce environment using provided client id and client secret, please verify the details. Error message:…
asked 2024-08-12T21:37:58.96+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 52%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENG%3C/text%3E%3C/svg%3E)
Neha Ganatra
20
Reputation points
commented 2024-08-14T19:01:57.8933333+00:00
Neha Ganatra
20
Reputation points
1 answer
we deliver a reporting solution from D365 Finance and Operations using “Export to Datalake” functionality, ADF, and a on-prem SQL Server to, customer have asked us to come up with a solution to swap the “Export to Datalake” functionality.
we deliver a reporting solution from D365 Finance and Operations using “Export to Datalake” functionality, ADF, and a on-prem SQL Server to, customer have asked us to come up with a time estimate to swap the “Export to Datalake” functionality. what is…
asked 2024-08-12T07:32:11.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 22%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sampreeth
0
Reputation points
commented 2024-08-14T16:55:13.8966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
13,650
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Why isn't source considered while determining location of IR for copy activity in ADF?
This is an excerpt from the Azure IR documentation: For copy activity, a best effort is made to automatically detect your sink data store's location, then use the IR in either the same region, if available, or the closest one in the same geography,…
asked 2024-08-07T17:48:12.6433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 67%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Dhruv Singla
25
Reputation points
accepted 2024-08-14T16:06:09.2666667+00:00
Dhruv Singla
25
Reputation points
0 answers
Azure Data Factory Salesforce Connector and Salesforce Bulk API v2.0 support
As per documentation: The Salesforce connector is built on top of the Salesforce REST/Bulk API. When copying data from Salesforce, the connector automatically chooses between REST and Bulk APIs based on the data size – when the result set is large,…
asked 2021-12-01T20:25:33.47+00:00
Javier Callico
11
Reputation points
commented 2024-08-14T15:32:28.97+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 99%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
Venkoba Rao Jujaray
0
Reputation points
1 answer
How to write a dynamic expression in Flatten transformation for "Unroll by" and "column mapping"
I am developing a generic data flow to read several JSON files, flatten them and load into the parquet files. Following are the two sample JSON files. I want to select all the attributes under the array Value. Appreciate your inputs. {…
asked 2023-12-28T11:11:26.0233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 40%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
Narayanaswamy, M (Manjunath)
15
Reputation points
answered 2024-08-14T14:58:54.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 28.000000000000004%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDM%3C/text%3E%3C/svg%3E)
Deepak Madeti
0
Reputation points
1 answer
Distinct from blob storage
Hi, how to read distinct data from blob storage without using data flow ie table1---- company code(columnname) p1, p2, p1,p5, p2.. Pls share screenshot
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)