Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,845 questions
1 answer
How to Remove Slashes in Set Variable Activity for use in Body of Web Activity - PLEASE
HELLO! I have the following code in the Web Activity (and a SET VARIABLE ACTIVITY) and I need to get rid of the slashes \ How can I do this? I have tried everything. My Web Activity Body will work if I can get these out. I have worked hours on this....…
asked 2020-12-08T05:39:33.443+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 49%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mike Kiser
1,536
Reputation points
commented 2020-12-17T22:04:20.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
SQL SERVER Source table create point in table of 70 gb table
Hi Guys, We want to create snapshot of an table everyday in azure synapse on sQldw, what would be best approach for daily load of 70GB table from on premise sql,oracle or sap hana, we want load the data in mill seconds max in minutes. Hint -Similar…
asked 2020-12-08T02:30:26.787+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 48%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Dondapati, Navin
281
Reputation points
commented 2020-12-17T22:03:19.62+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
How to get the one pipeline output into another pipeline
Hi, i am new azure data factory, How to get the one pipeline output into another pipeline in azure data factory. Thanks, Madhavi
asked 2020-12-06T10:10:43.257+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 46%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
madhavi
1
Reputation point
commented 2020-12-17T22:01:55.17+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
0 answers
Error when connecting to a second database in the same server IP using Azure Integration runtime
I have 2 database in the same Server, and they both have different logins. I can connect to 1 of them through Azure Integration Runtime without any issues, but when I try to create a new linked server for the second database I get an error that my…
asked 2020-12-05T06:18:33.257+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 82%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWI%3C/text%3E%3C/svg%3E)
www.intensa.com.mx
1
Reputation point
commented 2020-12-17T21:59:57.18+00:00
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
1 answer
How can we disable generation of ARM Templates?
Whenever we publish ADF pipeline changes, the publish process invokes some sort of ARM Template generation that fails because we have too many parameters, then most the time succeeds completion. We would prefer to disable this automatic generation of…
asked 2020-12-05T21:03:20.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 66%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EME%3C/text%3E%3C/svg%3E)
Michael Erickson
6
Reputation points
commented 2020-12-17T13:21:24.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Converting String to DateTime in Body of ADF Web Activity.....URGENT PLEASE
Hi! Here is my issue: I have a datetime value in my DB which has to be passed as a datatime value to the REST API. Please see the Body of the PATCH call: All fields are working except for the following : "EffectiveStart":…
asked 2020-12-12T02:40:38.627+00:00
Mike Kiser
1,536
Reputation points
commented 2020-12-17T13:14:46.8+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Get metadata filtering using a filename
Greetings I am trying to use get metadata activity to get only specific files with a date I want in my dataset from an FTP storage. File example is H_ODP_20201027_1232_00.CSV. I want to dynamically get files with certain date when I use the Tumbling…
asked 2020-12-14T06:14:00.113+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 96%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETC%3C/text%3E%3C/svg%3E)
Tinashe Chinyati
221
Reputation points
commented 2020-12-17T13:09:59.183+00:00
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
Validation activity - checking the existence of a single file
I have a validation activity. In this activity, I want to check if a particular blob file exists or not. I provide the container name, directory name and the file name as shown below. This timesout even if the file exists. However, if I delete…
asked 2020-12-08T07:57:59.073+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 6%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan
941
Reputation points Microsoft Employee
commented 2020-12-16T18:54:20.913+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,771
Reputation points Microsoft Employee
1 answer
ADF - load .csv files one at a time
Hello, Can you please guide me to a documentation or video which teaches me how to use ADF to loop through files i..e .csv files in blob storage and one at a time to load them into destination i.e. azure sql server or container... And to be able to…
asked 2020-12-03T07:09:16.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
arkiboys
9,686
Reputation points
commented 2020-12-16T18:50:47.193+00:00
Saurabh Sharma
23,771
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
ADF Copy Data Activity - Sudden Intermittent ADLS Connectivity Issue
Starting yesterday we're suddenly seeing these intermittent failures with Copy Data activities that copy data to Azure Data Lake Storage Gen1. It has happened both on 12/2/2020 and 12/3/2020 shortly after midnight (Central). Error code 2200 Failure…
asked 2020-12-03T16:41:19.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 3%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJB%3C/text%3E%3C/svg%3E)
Jared Buschkopf
211
Reputation points
accepted 2020-12-15T21:42:23.543+00:00
Jared Buschkopf
211
Reputation points
1 answer
One of the answers was accepted by the question author.
CSV comma issue in Sink
Hi, My source file is having a column value like Naperville, IL Please note that the value Naperville, IL is a single column value. I want this value to be written in a csv file. When I try to write this data in csv (sink), the column value is…
asked 2020-12-11T12:17:45.007+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Gowri Shankar
226
Reputation points
commented 2020-12-15T21:12:42.527+00:00
Nasreen Akter
10,766
Reputation points
2 answers
One of the answers was accepted by the question author.
CSV comma issue in sink (Azure DW)
Hi, I want to move my file from azure blob to azure DW. In azure blob, one of the table having column value like Naperville, IL. I want this value to be written in to azure dw. But when I try to write it into DW using (polybase copy), I…
asked 2020-12-14T08:12:48.383+00:00
Gowri Shankar
226
Reputation points
accepted 2020-12-15T10:34:53.22+00:00
Gowri Shankar
226
Reputation points
0 answers
Ingest into Azure Data Explorer failing through Azure Data Factory pipeline
I have created a Azure Data Factory pipeline to copy data from On- Premise Sql server to a Kusto cluster.I have created a service principal and given it admin permission on the Kusto database in Azure Data Explorer.Also used this service principal for…
asked 2020-11-18T07:10:29.487+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 52%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Ashutosh Rai (Tata Consultancy Services Ltd)
1
Reputation point
commented 2020-12-15T08:29:30.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 71%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVB%3C/text%3E%3C/svg%3E)
Vladik Branevich
1
Reputation point Microsoft Employee
1 answer
Calling Sentiment API in Azure Data Factory
Hi, I have a table that has all the customer feedbacks. I want to be able to call Cognitive Analytic Services (for sentiment analysis) API during ETL in Azure Data Factory and save the sentiment result in a table. Do you know how I can do it in ADF?…
asked 2020-12-02T03:37:14.333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 12%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
kayeesp2020
1
Reputation point
answered 2020-12-15T06:05:12.32+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
0 answers
Bad Request - Invalid linked service reference
I have a linked service connecting to an SFTP. The linked service also references another linked service connecting to my key vault so I can reference the secret that contains the password for the SFTP connection. When I execute a copy activity that uses…
asked 2020-12-09T17:14:32.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 22%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Dayton, Nate
1
Reputation point
commented 2020-12-15T00:09:01.447+00:00
Saurabh Sharma
23,771
Reputation points Microsoft Employee
1 answer
can we set global parameters with diffrent values, for the same pipeline across diffrent enviroments Such as (Dev , Qa, Prod)
Hi Team , 1)we have adf pipeline and we have dataset pointing to different locations(location path passed as parameter) in different environment, for same pipeline, can we over ride this location parameters using global paraments, so that it will…
asked 2020-12-03T11:12:04.55+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 99%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER1%3C/text%3E%3C/svg%3E)
Raviyaramati-1837
1
Reputation point
commented 2020-12-14T15:48:42.88+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
ADF Web Activity Response Codes
HellO! I am now able to concat and pass the Body of a Web Activity in Azure Data Factory; now, our requirements state that I have to capture the REST API Response Code, such as 200, 400, 500 etc. I can't find one article that says how to do this so…
asked 2020-12-07T00:11:54.583+00:00
Mike Kiser
1,536
Reputation points
commented 2020-12-14T15:45:16.71+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
tumbling-window-trigger-dependency(Azure Data Factory)
Hi Team, How to create dependent pipelines in your Azure Data Factory using tumbling window trigger.? I created the triggers and assigned dependency ,also activated the triggers but triggers not executing the pipelines by given start date and…
asked 2020-12-07T12:24:30.447+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 98%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
pankaj chaturvedi
86
Reputation points
commented 2020-12-14T15:44:00.697+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Flow splitting dataset into multiple tables
Hi team. I'm utilizing Azure Data Flow to read in an excel file with multiple tabs of data. One of the tabs contains three sets of data with the same schema, that is required to be broken out into separate datasets. The data is formatted similar to…
asked 2020-12-03T05:22:41.17+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 68%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
HansJKiamzon
56
Reputation points
commented 2020-12-14T07:25:53.123+00:00
HansJKiamzon
56
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory : How to pass DataPath as a parameter to Azure ML Pipeline activity?
Hello All, How to pass a Datapath as a parameter in Azure ML Pipeline activity? More details here : Have opened an issue here : https://github.com/Azure/Azure-DataFactory/issues/216 Thanks.
asked 2020-09-10T20:53:11.99+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 68%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESN%3C/text%3E%3C/svg%3E)
Sriram Narayanan
21
Reputation points
commented 2020-12-14T05:56:59.913+00:00
HarithaMaddi-MSFT
10,136
Reputation points