Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,606 questions
1 answer
One of the answers was accepted by the question author.
executing pipeline dynamically
Hello, There are files such as follows: pipelineName.DatasetName.txt The metadata activity reads this filename and gets the pipelinename and assigns it to a set variable i.e. vPipelineName. Lets say the pipelinename is pRunContacts Question: …
1 answer
Some Activity Runs are Acquiring Compute after Already Acquiring Integration Runtime
I am running a pipeline in Azure Data Factory and while running this pipeline there are multiple Data Flows that need to be executed. The first data flow executes and acquires compute for the Integration Runtime. Although once the next data flow begins…
asked 2022-01-27T18:57:37.937+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 30%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECH%3C/text%3E%3C/svg%3E)
CJ Hillbrand
1
Reputation point Microsoft Employee
commented 2022-02-02T00:54:50.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
File download using HTTP connector
Hello, I'm downloading a file using HTTP connector from a URL similar to as below: https://{website}/FileView2.aspx?IDFile=26a883a7-afb4-4d98-95e8-29cc63638c28 The link points to a zip file. The file downloads with a proper name in the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 74%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
1 answer
Column 'ABC' does not allow DBNull.Value, but no null values in data source.
How does one troubleshoot this issue? "Operation on target XYZ CSV failed: Failure happened on 'Sink' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed. Please…
asked 2022-01-28T21:18:31.567+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 47%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mike McGuinness
26
Reputation points
commented 2022-02-01T20:45:23.477+00:00
Mike McGuinness
26
Reputation points
1 answer
Azure data factory REST API throwing error
I am trying to fetch data through data factory using REST API OAuth2ClientCredentails, but I am getting error : {"error_description":"access_denied","error":"server_error"}, where as its working through postman and…
asked 2021-10-29T04:51:23.067+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 30%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVT%3C/text%3E%3C/svg%3E)
Vikas Tiwari
766
Reputation points
commented 2022-02-01T18:05:02.83+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,031
Reputation points
1 answer
Publish Error via Azure Data Factory after Terraform Deployment Pointing to existing GitHub Repo
I have an error below when I tried to publish from Azure Data Factory after It is deployed via Terraform. Publishing error: Invalid references or dependencies found. This is likely due to publishing outside of Git mode or editing and deleting …
asked 2022-01-27T12:39:52.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 79%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
PingpongSet
36
Reputation points
commented 2022-02-01T16:35:17.393+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
2 answers
Why do we use app registration from Azure Active Directory ?
I'm fairly new to microsoft tools. I was working to copy file from sharepoint to adls. While going through it, I saw we had to app register in AAD. Why do we do that?
asked 2022-01-21T21:12:37.197+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 54%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
Pravash
21
Reputation points
commented 2022-02-01T16:23:26.68+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
Databricks notebook output exception log to Azure data factory
Are there anyway that the Databricks notebook output exception log to Azure data factory ? Like set variable and then log to Azure Log analytics ?
asked 2022-01-19T14:58:29.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 40%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sakuraime
2,321
Reputation points
commented 2022-02-01T16:18:03.647+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
SQL Query in CSV File in blob storage
Hello Team, I am having 3 queries in sql server . I need to run all the 3 queries and need to copy the data into 3 different files into Azure blob storage.(FiileName1,FileName2,FileName3) please advise me how it can be done. Regards Rohit
asked 2022-01-28T15:54:46.027+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rohit Kulkarni
676
Reputation points
commented 2022-02-01T16:12:04.543+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 63%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
svijay-MSFT
5,201
Reputation points Microsoft Employee
1 answer
Load data from XML file to Azure SQL DB using Azure data factory
I am trying to load the XML file (adf.xml) from blob storage to Azure SQL DB From the attached file (mappingadfxml1.pdf), I am only looking for the columns that are checked and loaded into the respective columns in Azure SQL DB. I have also…
asked 2022-01-10T15:58:13.763+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 89%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
amikm
11
Reputation points
commented 2022-02-01T16:09:01.493+00:00
ShaikMaheer-MSFT
37,896
Reputation points Microsoft Employee
1 answer
Unrecognized extension in ADLS
Hi, I'm downloading files from HTTP using Binary dataset in Copy Activity and Saving the file in ADLS using Binary Dataset. I'm using Binary dataset because the HTTP url(source) will have different file types dynamically. I'm getting the filename is url…
asked 2022-01-24T06:44:04.627+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 34%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Madhan
1
Reputation point
commented 2022-02-01T11:43:28+00:00
AnnuKumari-MSFT
31,061
Reputation points Microsoft Employee
1 answer
Azure data factory copy large number of files from ADLS to blob
Hi, I want to perform a copy activity from Azure data lake store to azure blob storage by compressing the files. I have a folder in ADLS which has to be compressed to a zip file and stored in a blob. The throughput I achieve is very less. I could…
asked 2022-01-23T08:26:35.853+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 34%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Arun S
1
Reputation point
commented 2022-02-01T10:50:56.9+00:00
svijay-MSFT
5,201
Reputation points Microsoft Employee
1 answer
Passing the output of copy activity to a parameter in Azure Data Factory
Hi, I'm using Copy Activity to call a HTTP server for fetching the data from a URL and storing the data in SQL database. At the same time I must also use this fetched data to perform further operations in same pipeline. So right now I'm storing data in…
asked 2022-01-20T09:25:01.777+00:00
Madhan
1
Reputation point
commented 2022-02-01T10:49:08.727+00:00
svijay-MSFT
5,201
Reputation points Microsoft Employee
2 answers
convert string to float in azure data factory in copy activity
I have a csv file in mapping all source column ha string type and in destination some column have float ,varchar ,int type. When i run the pipeline activity fail and show error. Error message: ErrorCode=TypeConversionFailure,Exception occurred…
asked 2021-12-31T07:31:59.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 37%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Jamshed Salik
1
Reputation point
commented 2022-02-01T10:45:03.807+00:00
svijay-MSFT
5,201
Reputation points Microsoft Employee
0 answers
Azure Data Factory / Self-Hosted Integration Runtime / ODBC / Connection String In a Parameter
Hey, Here's what we're trying to do: Got an ADF Self-Hosted Integration runtime, hosted in a Windows VM with an ODBC Driver installed Trying to pass ODBC Driver connection string as a parameter JSON for the Linked service looks like this: { …
asked 2022-01-14T13:32:51.537+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 83%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vladimir
1
Reputation point
commented 2022-02-01T10:43:41.237+00:00
svijay-MSFT
5,201
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF Self-Hosted IR performance issue.
Hello guys. The situation: I have 2 Virtual Machines with 2 installed SQL Servers. On both of them were installed Self-Hosted IRs for the correct connection between ADF and SQL servers. When I tried to run some Copy component that took around 100…
asked 2022-01-18T11:27:21.813+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 15%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mykola Sukhovii
21
Reputation points
commented 2022-02-01T08:28:20.64+00:00
Mykola Sukhovii
21
Reputation points
0 answers
Flowlet using join
Hi Team, I have one scenario, I am creating a flowlet with 2 inputs and1 join and 1 select component. When I am trying to use the created flowlet in a dataflow, i am not able to see the data in data preview tab. Getting below error : DF-COMP-003…
asked 2021-12-21T10:55:49.737+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 25%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
Kirti Bhudhar
31
Reputation points
commented 2022-02-01T06:31:18.363+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Azure Data Factory - Designer not picking up changes in environments from SSIS Catalog
I have added an additional Environment to my project deployed in the SSIS Catalog. However the ADF Designer will not pick it up, nor will it execute the package if I put the name of the Environment in manually. I have tried: - Restarting the SSIS IR…
asked 2021-12-22T00:07:29.957+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 21%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
James Beresford
0
Reputation points
answered 2022-01-31T22:44:31.893+00:00
James Beresford
0
Reputation points
2 answers
One of the answers was accepted by the question author.
Error while publishing: Cannot read property 'linkedService' of undefined
When trying to publish an ADF from master to adf_publish I am getting an error: Error while publishing: Cannot read property 'linkedService' of undefined There are no other details shown. I tried looking through the master branch and do not see…
asked 2020-10-01T18:40:13.957+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 45%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
John Aherne
516
Reputation points
commented 2022-01-31T21:07:17.437+00:00
MartinJaffer-MSFT
26,031
Reputation points
1 answer
data factory pipeline visual
Hi, If I do not have the azure account, is there a way to see the visuals of the pipeline if I only have received the .json file of it? Thanks
asked 2022-01-31T07:21:37.613+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
arkiboys
9,641
Reputation points
answered 2022-01-31T17:54:04.257+00:00
MartinJaffer-MSFT
26,031
Reputation points