Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
2 answers
One of the answers was accepted by the question author.
DataFactory - drop table if schema changes
Hello, Refering to Auto-Create table on Copy activity. When the table is already on the database and there is a missmatch on the table schema the pipe fails. This is normal function. I should delete the actual table on my DB and then the table will…
Azure Data Factory
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 74%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Getting Error In Lookup while executing Stored Procedure For Snowflake
CREATE OR REPLACE PROCEDURE schema.merge_proc(SRC_TABLE varchar, tgt_table varchar, unique_columns varchar) RETURNS FLOAT not null LANGUAGE JAVASCRIPT EXECUTE AS CALLER AS $$ function getColumnsToBeInserted(difference) { difference_column = "";…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-07T07:55:25.787+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 48%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Aditya Raj
286
Reputation points
accepted 2021-08-12T05:33:18.097+00:00
Aditya Raj
286
Reputation points
1 answer
How to get .zip data from the web into my DataFactory
Hello everyone, first of all i am realativly new to the world of azure. I think it's best if I explain to you first what exactly I'm planning or what my project is. I'm trying to save data from a certain website, which is in a zip format, into…
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,563 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-07-29T09:25:46.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 71%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJT%3C/text%3E%3C/svg%3E)
Jonas Trumpfheller
21
Reputation points
answered 2021-08-11T22:22:42.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,156
Reputation points
1 answer
One of the answers was accepted by the question author.
Parse Error In Lookup For Stored Procedure In ADF
I am trying to run a stored procedure from lookup inside adf. CREATE OR REPLACE PROCEDURE merge_proc(src_table TEXT, tgt_table TEXT, unique_columns TEXT) RETURNS float not null LANGUAGE JAVASCRIPT EXECUTE AS CALLER AS $$ var row_count = 0; …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-06T10:49:43.53+00:00
Aditya Raj
286
Reputation points
accepted 2021-08-11T10:56:27.253+00:00
Aditya Raj
286
Reputation points
1 answer
How to Increase Web page response in Web activity of the Azure data Factory?
Hello Team, We are using the web activity in azure data factory. The maximum web page response we can receive it 4 MB and if it is exceeding this the pipeline is getting failed. So, we are doing the web pagination and adjusting the count to get the…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2020-12-17T09:35:15.127+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 77%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dheeraj
6
Reputation points
commented 2021-08-11T09:54:37.64+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 30%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDE%3C/text%3E%3C/svg%3E)
DELAHAIE Enguerran (EXT TALAN)
1
Reputation point
0 answers
Azure Data Factory run date format changed
Hello Experts, We have two ADF instances and we are facing some weird issue, we are using two additional parameters in copy activity and one of the parameter is Created_Date = utcnow() which basically pipeline run date. Earlier this was returning in…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-06T09:42:16.303+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 57.99999999999999%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECU%3C/text%3E%3C/svg%3E)
Chayan Upadhyay
96
Reputation points
commented 2021-08-11T08:34:03.943+00:00
Chayan Upadhyay
96
Reputation points
1 answer
One of the answers was accepted by the question author.
How to use a output of a databricks activity in future activity?
I have a Databricks activity in ADF and I pass the output with the below code: dbutils.notebook.exit(message_json) Now, I want to use this output for the next Databrick activity. As my search, I think add the last output into base…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,553 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-07-28T09:50:22.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 43%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Mohsen Akhavan
831
Reputation points
accepted 2021-08-11T06:24:03.093+00:00
Mohsen Akhavan
831
Reputation points
5 answers
One of the answers was accepted by the question author.
error handling
Hello, In my pipeline, there are several activities as follows: 1- copy activity to transfer on-prem data to blobstorage 2- on success of step1, a dataflow is called 3- on failure of step1, a copy activity is called to populate errorlog in blob…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Combine the varying number of rows to one row in Data Factory in Azure
I am so new in Azure Data Factory, and I have a problem solving this problem. I wanted to combine varying number of rows in to one row base on the substring of the first column , I know that I can use collect() but the problem is I don't have a key…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-05T16:18:07.877+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 40%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
Gurusvasti
21
Reputation points
accepted 2021-08-11T05:50:04.677+00:00
Gurusvasti
21
Reputation points
1 answer
Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation
I am trying to create a Linked Service and then a dataset for connecting to ADLS Gen1 to use it as a Sink in ADF Copy. I am using Service Principal as the Authentication method. Initially I was getting the below error while creating the Linked Service…
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,563 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-07-27T04:31:10.477+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 6%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUC%3C/text%3E%3C/svg%3E)
Utsav Chanda
76
Reputation points
commented 2021-08-10T21:04:37.893+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,491
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
ingesting all .txt file from on prem file share to ADLS Gen2 using ADF V2
What should I put in Value text box below screen shot to move all *.txt files in this share \tmdwmart\Stage\Dev\BOB in a dataset, so I can move them to ADLS container using destintion
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 38%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Limitations of Azure Data Factory resources
Hi Team, We have a requirement, where we need to execute 90+ migrations at the same time that includes 90 different source data servers and 90 different target data server's. Can you please suggest on following... How many maximum pipelines…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-10T14:57:57.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 57.00000000000001%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH0%3C/text%3E%3C/svg%3E)
HemantSudehely-0557
266
Reputation points
commented 2021-08-10T16:01:19.267+00:00
MarkKromer-MSFT
5,226
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
create trigger - publish first
Hi, The data factory is in devops repository... I would like to schedule the data factory pipeline. While I am in the master branch, I would like to create a trigger to schedule the run but it says I have to publish first. So in the master branch…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
3 answers
One of the answers was accepted by the question author.
Difference on Azure Data factory Microsoft.DataFactory/datafactories/ and Microsoft.DataFactory/factories/ permissions
We need to create custom roles for Azure Data Factories. However, there are two groups for permissions for the ADF, Microsoft.DataFactory/datafactories/* and Microsoft.DataFactory/factories/* For example, according to Azure document below, the…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 72%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERX%3C/text%3E%3C/svg%3E)
1 answer
Azure Data Factory Data Flow send to rest not working
Hi, trying to send data from a data flow to a rest API endpoint and I get the following error: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason:…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-06T15:17:24.237+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 13%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Mohammed Laher
1
Reputation point
commented 2021-08-10T11:58:02.167+00:00
Mohammed Laher
1
Reputation point
2 answers
How to exclude a column having null values in ADF
Hi, I am having a input file in which some of the columns will be having null values, column name will be there but the values are null, is there any way to exclude those in output. for example : file will be having , emp, emp_name, address, pin code…
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-09T16:12:53.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 45%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJS%3C/text%3E%3C/svg%3E)
Johny, Sijo
21
Reputation points
answered 2021-08-10T07:06:43.673+00:00
Johny, Sijo
21
Reputation points
1 answer
One of the answers was accepted by the question author.
failure activity
Hello, The copy activity is connected to a dataflow How is it possible to create a failure activity after the copy activity so that if the copy activity fails, a .parquet file is populated with some error information? For example Thank you
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
1 answer
Folder and filename use of wildcards
Hi I'm using the metadata driven copy in ADF. I'm copying several tables to parquet files but want to append the date to the file. What can I achieve this? How are wildcards used under this functionalities?
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-02T13:55:33.743+00:00
Adalberto García Espinosa
86
Reputation points
commented 2021-08-10T05:11:26.863+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
0 answers
Sharepoint API Registeration
Hello Team, After the API Registration for the SharePoint I have created the Source dataset .But not able to access the csv files in the SharePoint : The Preview data marked in red is disabled. Please let me know how to enable this option. …
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-07-30T10:51:49.033+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rohit Kulkarni
731
Reputation points
commented 2021-08-10T05:10:21.347+00:00
ShaikMaheer-MSFT
38,551
Reputation points Microsoft Employee
Moderator
1 answer
One of the answers was accepted by the question author.
Fan in and Fan out in ADF
Hi All, is it possible for any Parnellism in ADF by using Fan in and Fan out feature in Azure data factory? If yes what is the use case for it. Please explain. Thanks a lot.
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,670 questions
asked 2021-08-04T12:35:00.827+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 12%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Samy Abdul
3,376
Reputation points
accepted 2021-08-10T05:03:07.25+00:00
Samy Abdul
3,376
Reputation points