Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,535 questions
1 answer
One of the answers was accepted by the question author.
best practice
Hi, What is the best practice to execute the appropriate pipeline(s) with parameters based on the file(s) being landed in container A? Thank you
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Custom Event Trigger for multiple synapse pipelines
Hi everyone, We have a process in place which sends and event to an event grid once a staging table is loaded. The payload of the event includes the table name. The next step is to develop a trigger which executes the correspondent pipeline to move…
asked 2022-01-17T13:10:19.34+00:00
Paul Hernandez
631
Reputation points Microsoft Employee
commented 2022-01-21T17:18:26.467+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
ADF Pipeline concurrency for different pipelines
Hello Team, I am having two different pipelines. Both the pipelines are configured to process data and insert in to a same table. So when these pipelines run parallelly, sometimes the first pipeline already inserts the data and the second pipeline has…
asked 2022-01-17T10:30:59.37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 16%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Monica Manoharan
66
Reputation points
commented 2022-01-21T16:54:24.167+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Latest file extraction by scanning multiple folders and copy to azure sql database
Hi, I have a folder structure in ADLS Gen2 in container source that goes as below: /table1/2021/01/20/file1.parquet /table1/2021/02/20/file2.parquet /table1/2022/01/10/file3.parquet /table1/2022/01/12/file4.parquet …
asked 2022-01-14T09:29:29.293+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 32%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Anmol Ganju
166
Reputation points
commented 2022-01-21T16:48:38.797+00:00
Anmol Ganju
166
Reputation points
1 answer
Read XML content as a parametar
HI All , I have one XML file .... I need to read the file and wants to pass the total data into one variable as a single column .....is it possible to do that in ADF ? Thanks in Advance
asked 2022-01-12T16:42:12.837+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 17%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
AYUB SHAIK
46
Reputation points
commented 2022-01-21T16:39:52.423+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
Unable to launch the ADFv2
I have tried launching adfv2 both in safari and chrome(normal/incognito) browsers but nothing worked for me. Also I verified cookies,it’s fine and deleted browsing data. After these many attempts still issue persists. Kindly suggest . I have tried…
asked 2022-01-12T06:11:58.477+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 7.000000000000001%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Sukanya Tamidalapati
1
Reputation point
commented 2022-01-21T16:39:16+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Write to cosmos DB using ADF
Hi Experts, I need to create a pipeline with source as "Azure SQL DB" and the target / sink being cosmos db. There might be few nested objects. Can someone provide some links with sample pipelines for the same? Thanks
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 53%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENC%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 57.99999999999999%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
Capture Pipeline error messages
Hello, My requirement is to capture the error message of pipeline execution in Data factory and store it into a table. Currently I'm able to do it activity wise only ('@{activity('Copy data').output.errors[0].Message}'), which captures the message for…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 48%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E3%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Is Data factory really needs cluster configuration for running? Can we run datafactory without any cluster configuration ?
Hi team, I am doing PAAS based project using data factory. We need push values from excel to mysql. For this do we really need cluster configuration or it will run with in azure without any cost apart from Data factory execution cost. Help and…
asked 2022-01-21T05:13:24.63+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 39%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETS%3C/text%3E%3C/svg%3E)
Thiyagarajan S
41
Reputation points
commented 2022-01-21T07:16:30.8+00:00
Thiyagarajan S
41
Reputation points
1 answer
One of the answers was accepted by the question author.
Release pipelines
Hello, In the development environment, I create sample pipelines to test, etc. When I decide to deploy to production using the CI/CD release pipelines, what happens to the sample pipelines in dev? will they get migrated to production as well? if so,…
2 answers
One of the answers was accepted by the question author.
ELT frameworks (Azure)
Generally speaking, which Azure data platform is most typically utilized for data processing in an ELT framework? Azure Data Factory or Azure Databricks...
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 64%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
ADF to schedule databricks streaming job
Hello, We are using Azure Data Factory (ADF) to schedule databricks streaming job. Requirement: The streaming job may need to run all the time. If it is cancelled or failed, then we need to automatically restart it. Problem: Currently we…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 82%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETT%3C/text%3E%3C/svg%3E)
0 answers
Select cluster option is changing after deplymento to stage and production in ADF
In Dev we have created linked service for databricks with choosing "existing interactive cluster" option . But after deployment to stage and production the configuration is changing to "New job cluster". What could be the issue? …
asked 2022-01-14T12:49:24.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 50%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDA%3C/text%3E%3C/svg%3E)
Dharappanavar, AkshayKumar
1
Reputation point
commented 2022-01-21T02:24:26.833+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF: Unable to load json file >100 MB into a database column
Hi, I have a requirement to load the json files (ADF Logs) into our database via Azure Data Factory (Copy Activity). The datatype of the column is "[nvarchar]". There will be around 20 files/day and the file at 21:00 hrs is having size more…
asked 2022-01-17T12:02:54.65+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 36%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Pashikanti Ranadheer
21
Reputation points
accepted 2022-01-20T18:23:14.51+00:00
Pashikanti Ranadheer
21
Reputation points
1 answer
Querying a data source - Azure Data Factory
Hi, I'm implementing some ADF pipelines. I need to query the data source (a relational database) in order to understand the right data filters before the data ingestion. Is it possible (f.e. at linked service and/or data set)? Thanks
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 41%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
Mapping parquet format datasets to Azure Datawarehouse
Hi, I have a pipeline that copies all the files (*.parquet) present in different folders into different tables using foreach loop, Now I have particular set of columns defined for each destination tables, for example Table A contains 100 columns, while…
asked 2022-01-18T15:45:39.47+00:00
Anmol Ganju
166
Reputation points
commented 2022-01-20T17:40:50.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
1 answer
How to connect to multiple servers to run the same query dynamically in Azure data factory or in Azure
I have a requirement to connect to multiple servers to run the same complex sql query and should combine the results of all those outputs as a single result set and should pass that to the next component for further processing. Could you please let me…
asked 2022-01-18T06:15:32.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 64%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
SUDHEER MADDU
1
Reputation point
commented 2022-01-20T15:50:28.597+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 63%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
svijay-MSFT
5,201
Reputation points Microsoft Employee
2 answers
Downloading multiple file types from dynamic url in ADF
I'm fetching a JSON object, { id:"1123123", "download_url":"https://file-examples-com.github.io/uploads/2017/02/file_example_XLS_10.xls", "status":1} { id:"1123123",…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 34%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
1 answer
Dynamically retrieving name of storage account in Azure Data Factory
I have multiple pipelines in different environments (dev, test, stage etc...). Is there a way that I can dynamically retrieve the name of the storage account (the name tells you the environment it's running in) in azure data factory so that I can make…
asked 2022-01-10T16:19:07.737+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 38%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Michael Vu
26
Reputation points
commented 2022-01-20T07:15:15.377+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee
1 answer
How to accomplish pagination with "rel = next"
For a REST api endpoint, the client data feed, gives us the pagination value in the header. Please can you advise on how to accomplish this. The relative URL for the dataset is https://clientname.com/api/v2/tickets?updated_since=2019-01-19 When the above…
asked 2022-01-02T20:49:24.403+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 42%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
sam nick
286
Reputation points
commented 2022-01-20T07:11:09.45+00:00
AnnuKumari-MSFT
30,676
Reputation points Microsoft Employee