Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,639 questions
3 answers
How to fix the Pipeline which is running as expected in ADF dev, post which I have published the pipeline (and associated objects) to ADF Prod it fails.
Hello Team, I have build ADF pipeline to load data from source (SQL server) to ADLS. Pipeline is running as expected in ADF dev, post which I have published the pipeline (and associated objects) to ADF Prod. But while running the pipeline (Debug),…
asked 2024-04-15T05:16:08.8066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 70%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
Bhosle, Nikhil
0
Reputation points
answered 2024-05-07T10:39:34.9833333+00:00
Bhosle, Nikhil
0
Reputation points
2 answers
Azure Data Factory Error- when trying to connect to Azure SQL Database. How to resolve? Why happened?
Spark job failed: { "text/plain":…
asked 2024-05-04T18:37:19.4233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 28.000000000000004%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFG%3C/text%3E%3C/svg%3E)
Fatemeh Ghorbani Sini
0
Reputation points
answered 2024-05-07T09:40:55.01+00:00
AnnuKumari-MSFT
31,151
Reputation points Microsoft Employee
1 answer
An csv file with 200 columns in adls. fetch only list of columns from config table. config table is in Azure SQL. config table has only 10 col names
An csv file with 200 columns in adls. fetch only the list of columns from the config table. config table is in Azure SQL.config table has only 10 column names. how to create pipeline with this scenario in Azure datafactory?
asked 2024-05-03T07:59:15.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 38%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E1P%3C/text%3E%3C/svg%3E)
19B01A04F3 P
0
Reputation points
commented 2024-05-07T09:30:50.7866667+00:00
AnnuKumari-MSFT
31,151
Reputation points Microsoft Employee
0 answers
How to parse nested json array of document in ADF data flow
Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …
asked 2024-05-02T15:44:57.8633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
45
Reputation points
commented 2024-05-07T08:14:09.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
6,080
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Azure Data Factory - Generate custom Guid and copy to SQL table
I need to generate a unique Guid which contains only numeric and of length 20, for every row in the table. Currently, I am running one lookup activity (name - FetchIDs) to run the Stored procedure. Output of the lookup activity is as shown below { …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 46%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Rest API End Condition in Copy Activity is not working
I'm working on a paginated get requestin adf. As soon as an EndCondtion is set, the request does not work anymore. I'm using pagination rules in order to loop the request until all data is loaded. The range rule works fine but as soon as I add the…
asked 2022-05-30T09:37:59.527+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 43%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Philipp Kempter
56
Reputation points
commented 2024-05-07T03:04:31.4366667+00:00
Gerardo Arevalo
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Syntax error in the metadata-driven control table script generated by the copy data wizard.
ADF ControlTable Script.txt In ADF I have selected the Metadata-driven Copy Task in the Ingestion wizard. My source database is Azure SQL (WorldWideImporters), and my destination database is also Azure SQL (AdventureWorks2014). I'm using the destination…
asked 2024-05-03T05:07:37.8666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 88%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWB%3C/text%3E%3C/svg%3E)
Warren Briggs
20
Reputation points
accepted 2024-05-06T22:45:05.9366667+00:00
Warren Briggs
20
Reputation points
0 answers
I am using the azure policy to whitelist the domain for outbound connectivity from Azure Data Factory to other services. But facing issues in connectivity due to throttling applied on policy.
I am using the azure policy (https://learn.microsoft.com/en-us/azure/data-factory/configure-outbound-allow-list-azure-policy) which is applied at resource group level. This policy is working as expected and is only allowing outbound connectivity to the…
asked 2023-11-27T13:36:07.8833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 75%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHD%3C/text%3E%3C/svg%3E)
Harshwardhan Deshmukh
5
Reputation points
commented 2024-05-06T22:12:53.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 99%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Shahzad
0
Reputation points
0 answers
How do I use the Script activity in ADF, so it uses Azure Databricks SQL Warehouse
I want to be able to use ADF Script activity to execute SQL statements on the Azure Databricks SQL warehouses (including the serverless kind). https://learn.microsoft.com/en-us/azure/data-factory/transform-data-using-script Azure Databricks SQL…
asked 2024-05-01T12:21:53.01+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
commented 2024-05-06T19:45:40.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
26,331
Reputation points Microsoft Employee
1 answer
How to translate database content with Azure Translator by ADF or Synapse notebook?
There is an Azure Database table. Some of columns need to be translated from one language to another into additional columns. Such as from English to Spanish, or Portuguese to English, etc. I am exploring how I can use ADF or Synapse notebook to…
asked 2024-04-22T15:17:46.4133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 49%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dataholic
80
Reputation points
edited the question 2024-05-06T19:34:05.7933333+00:00
BhargavaGunnam-MSFT
26,331
Reputation points Microsoft Employee
0 answers
ADF Pipeline using Global parameter fails when triggered but succeed when debugged
I'm working with ADF, using global parameters, and facing the issue. It works fine if I debug the pipeline. But, if I trigger the pipeline (manual or scheduled), it fails saying the property doesn't exist and it shows the name of the global parameter I'm…
asked 2024-05-06T15:44:03.9633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 46%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E0%3C/text%3E%3C/svg%3E)
09251703
0
Reputation points
edited a comment 2024-05-06T18:43:53.8333333+00:00
BhargavaGunnam-MSFT
26,331
Reputation points Microsoft Employee
0 answers
Reading 1 million Records from Single Excel in Blob storage + Azure functions/ADF/Cosmos Change feed/WebJob
Hi Everyone- I have a use case of reading 1 million records in a single Excel from a Blob storage. Basically when someone uploads this file to Blob, it should start processing. Please let me know the best option to choose from below for the same. …
asked 2024-05-06T03:15:08.2033333+00:00
Mathew James
316
Reputation points
commented 2024-05-06T17:45:13.89+00:00
Mathew James
316
Reputation points
3 answers
How do I copy data from a public sharepoint URL to Microsoft Blob Storage?
Hi everyone! I am new to using Azure services. My co-author at Stanford has uploaded several files (in csv format) to a public link. This link starts as: office365stanford-my.sharepoint.com/personal It has 40 files, ranging from a few 100 MB to 100 GB. I…
asked 2024-04-08T16:28:15.86+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 12%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENN%3C/text%3E%3C/svg%3E)
Namrata Narain
0
Reputation points
answered 2024-05-06T17:43:10.1166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 94%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JohnyBenz
301
Reputation points
2 answers
One of the answers was accepted by the question author.
When we deploy ADF,we are getting "TooManyFactoryUpdaterequests" in Create or Replace GlobalParameter through Azure Devops
When we deploy ADF, we are getting "TooManyFactoryUpdaterequests" in Create or Replace GlobalParameter using CI/CD Azure devops
asked 2024-05-01T16:52:46.6233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 57.99999999999999%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Ramesh Hariharan
20
Reputation points
accepted 2024-05-06T16:49:23.3+00:00
Ramesh Hariharan
20
Reputation points
1 answer
No MediaTypeFormatter is available to read an object of type 'ClusterInfo' from content with media type 'text/html'.
Hi, in azure data factory I wanted to test the connection of our new Databricks Workspace URL and ClusterID. I parameterized the linked service (LS) to global parameters databricksURL and databricksClusterID. When I want to test the connection…
asked 2022-10-18T12:40:54.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 12%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAE%3C/text%3E%3C/svg%3E)
admin-eGu
16
Reputation points
commented 2024-05-06T13:21:47.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 80%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECD%3C/text%3E%3C/svg%3E)
Ács Dániel
25
Reputation points
5 answers
Why is the public ip range download blocked?
Hey, I use a script to download the public ip ranges from azure (https://www.microsoft.com/en-us/download/details.aspx?id=56519). If I view that URL in my browser, it displays and I am able to download the JSON. However, if I run my script that would…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 81%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
1 answer
I have a CDC connector on ADF connected to an SAP BW system. Need to read data only of a particular country from the ODQ into ADF. How can I send a selection filter through the CDC connector to the SAP BW system so that only the filtered records are read?
I have tried the Optimize on source option on data flow with the conditions attached on the screenshot and I know that these conditions help with partitioning but I need to have some filters applied on the ODQ and then filtered data flowing into the…
asked 2024-04-01T05:14:39.2766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 38%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Pooja Kulkarni
0
Reputation points
commented 2024-05-06T09:24:26.31+00:00
AnnuKumari-MSFT
31,151
Reputation points Microsoft Employee
2 answers
In Azure Data Factory, is it possible to define link service and dataset with dynamic type?
In Data Factory, is it possible to define link service and dataset with dynamic type? Since the legacy Salesforce connector will soon be deprecated, I want to create a single link service and dataset to support both the legacy and new connector instead…
asked 2024-04-22T19:49:53.3066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 81%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECL%3C/text%3E%3C/svg%3E)
Choi, Liam (he/him/his)
0
Reputation points
answered 2024-05-06T09:07:01.78+00:00
AnnuKumari-MSFT
31,151
Reputation points Microsoft Employee
1 answer
SAP latency data
Hi Expert, how to we can load the data from modified data in updated or insert fields in databricks using ADF or data bricks on trigger level instead of loading multiple times example: table updated or inserted with new records how table change and…
asked 2024-04-23T07:29:53.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
165
Reputation points
commented 2024-05-06T08:36:05.2166667+00:00
AnnuKumari-MSFT
31,151
Reputation points Microsoft Employee
2 answers
How can use Lookup to get data from multiple salesforce Object in ADF
I need to select Some data from Multiple salesforce object to another database. Currently I am trying pipeline for this because Dataflow is not supporting Salesforce. But the problem is While creating source/dataset it is showing only object and don't…
asked 2024-04-23T08:15:29.11+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 96%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECV%3C/text%3E%3C/svg%3E)
Chandran, Vinitha
40
Reputation points
commented 2024-05-06T08:32:02.6866667+00:00
AnnuKumari-MSFT
31,151
Reputation points Microsoft Employee