Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,606 questions
5 answers
Why is the public ip range download blocked?
Hey, I use a script to download the public ip ranges from azure (https://www.microsoft.com/en-us/download/details.aspx?id=56519). If I view that URL in my browser, it displays and I am able to download the JSON. However, if I run my script that would…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 81%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
0 answers
ADF Snowflake V2 connector throws GenericAdoNetReadError during Copy activity
The problem We're using a Copy activity in ADF to move data from a storage account into a Snowflake database using the V2 Snowflake connector. But just after the data has been copied, the Copy activity fails and throws the following error: Operation on…
asked 2024-05-06T10:11:42.4733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 60%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERF%3C/text%3E%3C/svg%3E)
Raphael Fritz
0
Reputation points
edited the question 2024-05-06T10:12:43.56+00:00
Raphael Fritz
0
Reputation points
1 answer
I have a CDC connector on ADF connected to an SAP BW system. Need to read data only of a particular country from the ODQ into ADF. How can I send a selection filter through the CDC connector to the SAP BW system so that only the filtered records are read?
I have tried the Optimize on source option on data flow with the conditions attached on the screenshot and I know that these conditions help with partitioning but I need to have some filters applied on the ODQ and then filtered data flowing into the…
asked 2024-04-01T05:14:39.2766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 38%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Pooja Kulkarni
0
Reputation points
commented 2024-05-06T09:24:26.31+00:00
AnnuKumari-MSFT
31,061
Reputation points Microsoft Employee
2 answers
In Azure Data Factory, is it possible to define link service and dataset with dynamic type?
In Data Factory, is it possible to define link service and dataset with dynamic type? Since the legacy Salesforce connector will soon be deprecated, I want to create a single link service and dataset to support both the legacy and new connector instead…
asked 2024-04-22T19:49:53.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 81%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Choi, Liam (he/him/his)
0
Reputation points
answered 2024-05-06T09:07:01.78+00:00
AnnuKumari-MSFT
31,061
Reputation points Microsoft Employee
0 answers
Reading 1 million Records from Single Excel in Blob storage + Azure functions/ADF/Cosmos Change feed/WebJob
Hi Everyone- I have a use case of reading 1 million records in a single Excel from a Blob storage. Basically when someone uploads this file to Blob, it should start processing. Please let me know the best option to choose from below for the same. …
asked 2024-05-06T03:15:08.2033333+00:00
Mathew James
316
Reputation points
edited the question 2024-05-06T08:39:47.7633333+00:00
MayankBargali-MSFT
68,656
Reputation points
1 answer
SAP latency data
Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…
asked 2024-04-23T07:29:53.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
165
Reputation points
commented 2024-05-06T08:36:05.2166667+00:00
AnnuKumari-MSFT
31,061
Reputation points Microsoft Employee
2 answers
How can use Lookup to get data from multiple salesforce Object in ADF
I need to select Some data from Multiple salesforce object to another database. Currently I am trying pipeline for this because Dataflow is not supporting Salesforce. But the problem is While creating source/dataset it is showing only object and don't…
asked 2024-04-23T08:15:29.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 96%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECV%3C/text%3E%3C/svg%3E)
Chandran, Vinitha
40
Reputation points
commented 2024-05-06T08:32:02.6866667+00:00
AnnuKumari-MSFT
31,061
Reputation points Microsoft Employee
1 answer
Azure Data Factory - Generate custom Guid and copy to SQL table
I need to generate a unique Guid which contains only numeric and of length 20, for every row in the table. Currently, I am running one lookup activity (name - FetchIDs) to run the Stored procedure. Output of the lookup activity is as shown below { …
asked 2024-04-24T04:52:39.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 46%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
SR
40
Reputation points
commented 2024-05-06T08:30:53.2433333+00:00
AnnuKumari-MSFT
31,061
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Design strategy for Data lake
Hi friends, we need to design an Azure data lake from scratch for a solution that has complex, multiple data sources from Databases, Rest APIs, and custom applications, and the Data lake solution should be scalable and high-performance in terms of data…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 15%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Data Factory to Salesforce using JWT
Hi All, We've been set with the requirement of authenticating with Salesforce via JWT. For the most part, this is set up and we can get an access token returned, however, how do we access Salesforce once we have the token? Does it have to be through a…
asked 2024-05-02T04:49:26.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 30%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Ryan Abbey
0
Reputation points
edited a comment 2024-05-06T08:21:58.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
3,580
Reputation points Microsoft Vendor
0 answers
How can we connect Blackboard DDA (a SaaS using PostgreSQL to store data ) via Azure Synapse analytics using SSL certificate for authentication in SHIR for Private Vnet enabled networks.
Blackboard DDA is a SaaS application which uses PostgreSQL as DB to store data. The DB uses a SSL certificate key to authenticate and we have private Vnet enabled. How to create a linked service for this DB and where do we store certificate in SHIR. Are…
asked 2024-05-03T05:41:52.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 47%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDJ%3C/text%3E%3C/svg%3E)
Dravya Jain
0
Reputation points
edited a comment 2024-05-06T07:16:25.6166667+00:00
Harishga
3,580
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
Schema Drift Issue in Dataflow with Sink-Delta
Hi Team, I was testing the schema drift option for my delta sink and am getting error while running my dataflow when added one new column to existing source schema. Pipeline ran successfully when I didn't change the schema. Error: I have selected Allow…
0 answers
Split single column data into multiple columns in datafactory
Issue: I have some csv files in sftp location . I was planning to use copyactivity to load files from sftp location to datalake for archiving as well as upserting in deltalake. However, I am getting error while reading from source, as some file has…
asked 2024-05-03T18:07:32.02+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vaibhav
105
Reputation points
commented 2024-05-06T06:40:51.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
9,845
Reputation points Microsoft Vendor
0 answers
Copy activity is success but no data in the sink source
Hello, I have created a data pipeline that copies data from MongoDB to a SQL Azure data source. The problem is that 1 time out of 2, no data is copied even though the pipeline is successful. I have to restart the flow manually a dozen times before any…
asked 2024-05-05T07:54:17.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 33%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Morgane Lachuer
0
Reputation points
commented 2024-05-06T06:29:40.9166667+00:00
Smaran Thoomu
9,845
Reputation points Microsoft Vendor
3 answers
One of the answers was accepted by the question author.
Connect to Linked Server Database using Azure Data Factory
Hi! I have a database that is a linked server of another. I can connect and copy data of the main database in ADF. But is it possible to also copy data from the linked server database? If yes, how would I do this?
asked 2024-05-02T09:28:02.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 26%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKO%3C/text%3E%3C/svg%3E)
Keren Osler
60
Reputation points
accepted 2024-05-06T06:19:21.2933333+00:00
Keren Osler
60
Reputation points
1 answer
Activity stuck in queue status
Starting from today morning, I am facing problem that any pipeline run, by trigger or manually or in debug mode, doesn't start. To be precise, pipeline starts ("In progress", but first activity stays in "Queue" status). Doesn't matter…
asked 2024-04-30T07:54:01.1866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 13%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGE%3C/text%3E%3C/svg%3E)
Gużewski, Jacek
55
Reputation points
commented 2024-05-06T06:05:52.1466667+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
0 answers
Azure Data Factory - Copy data activity failed: Failure happened on 'Source' side
Operation on target Copy data1 failed: Failure happened on 'Source' side. ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path:…
asked 2024-04-23T05:58:01.1533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 43%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Aman Pahariya
0
Reputation points
commented 2024-05-06T06:03:45.2+00:00
Aman Pahariya
0
Reputation points
1 answer
When we deploy ADF,we are getting "TooManyFactoryUpdaterequests" in Create or Replace GlobalParameter through Azure Devops
When we deploy ADF, we are getting "TooManyFactoryUpdaterequests" in Create or Replace GlobalParameter using CI/CD Azure devops
asked 2024-05-01T16:52:46.6233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 57.99999999999999%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Ramesh Hariharan
0
Reputation points
answered 2024-05-06T05:58:34.3133333+00:00
Ramesh Hariharan
0
Reputation points
1 answer
Azure Data Factory pipelines stuck on first activity for 7+ hours
Hi, I just woke up to monitor all the runs, and suddenly saw that all of my runs were stuck on the first activity, or not even starting the first activity some for more than 7+ hours. This happened over-night as yesterday everything was fine and I didn't…
asked 2024-04-30T10:04:21.0433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 7.000000000000001%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMG%3C/text%3E%3C/svg%3E)
Marcus Ghebrehiwet
0
Reputation points
commented 2024-05-06T05:57:13.2566667+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee
0 answers
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
asked 2024-05-01T12:30:52.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
commented 2024-05-06T05:51:32.2633333+00:00
PRADEEPCHEEKATLA-MSFT
77,751
Reputation points Microsoft Employee