Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,845 questions
0 answers
Cannot register Azure Integration Runtime
Hello, I am trying to register a self hosted Azure integration runtime. This used to work before, but suddenly the existing runtime could not connect anymore. I already: Reinstalled the local runtime installation Removed integration runtime in Azure…
asked 2020-08-17T15:13:09.473+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 57.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EER%3C/text%3E%3C/svg%3E)
Erik Rans
6
Reputation points
commented 2020-08-27T20:20:57.057+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
Wanted to check any excel file/any file is available in File share Main and multiple layers subfolder
hello all, I want to check in ADF that, is there any file available in Fileshare main folder and multiple layers of hierarchical sub folder. Based on that I want to form m I f else condition.
asked 2020-08-24T04:17:16.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 37%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVP%3C/text%3E%3C/svg%3E)
Vishal Parikh
1
Reputation point
commented 2020-08-27T06:29:26.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Use Azure Data Factory to copy files and place a csv of files copied
Hello, I am trying to implement the following flow in an Azure Data Factory pipeline: Copy files from an SFTP to a local folder. Create a comma separated file in the local folder with the list of files and their sizes. The first step was easy…
asked 2020-08-16T11:29:48.23+00:00
Erik Rans
6
Reputation points
commented 2020-08-27T06:27:01.027+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Using reg ex to map inputs columns to one output column
I have need to ingest data from csv files into sql, however the column name for a particular value could differ, for example name could come as name or fullname. Assuming my database table has column called name how can I map the input columns using reg…
asked 2020-08-23T14:52:14.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 65%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
Kedarnath
1
Reputation point
commented 2020-08-27T06:25:02.337+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Copy from CosmosDB to CosmosDB Error: "Request size is too large" using Data Factory
I am using the basic Copy wizard in ADF v2. I have a source Cosmosdb in one Subscription, and moving to a new Cosmosdb in a new subscription Database and Containers are configured the same. I have one container that copies 212 of 216 documents and…
asked 2020-08-18T05:18:59.823+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 30%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Keith Miller
1
Reputation point
commented 2020-08-27T06:24:01.037+00:00
HarithaMaddi-MSFT
10,136
Reputation points
4 answers
One of the answers was accepted by the question author.
Remove-AzureRmDataFactoryPipeline is not deleting ADF pipeline present under GIT branch
I have a requirement to delete 50 pipelines from my ADF. I tried powershell way using the below command :- Remove-AzureRmDataFactoryV2Pipeline -Name "XYZ_DEF" -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName I am…
asked 2020-08-19T06:49:48.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 70%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
RATH, ARUN
26
Reputation points
commented 2020-08-26T21:55:18.82+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,041
Reputation points
2 answers
One of the answers was accepted by the question author.
Auto create table error in copy activity for sql to sql pipeline
Hi, I'm getting an error trying to preview data in the sink process. I selected the auto create table option for the source data which should automatically create a table in my sink sql datawarehouse but its not. Here's the error I get when trying to…
asked 2020-08-11T19:46:19.147+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Brighter Bites
21
Reputation points
accepted 2020-08-25T18:30:35.997+00:00
Brighter Bites
21
Reputation points
4 answers
How to convert binary data retrieved via HTTP into a parsable type
I am connecting to an HTTP source (SOAP API) via POST, and retrieving Binary data. How can I get this data into a parsable format that can be used to copy to a data warehouse? The goal is to use this data for PowerBI reports. Thank you
asked 2020-07-24T00:24:56.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 57.99999999999999%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECS%3C/text%3E%3C/svg%3E)
Cossever, Shane
1
Reputation point
commented 2020-08-25T16:52:34.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 46%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Shane Cossever
1
Reputation point
4 answers
One of the answers was accepted by the question author.
Excel Mapping Data Flow - Date Time Text Values are loading month as minute
When importing an XLSX file in to the mapping data flow (EDIT:Tested with both inline and dataset and same happens), a column containing date times is loading with the incorrect text value. It looks like the process has attempted to parse the date time,…
asked 2020-07-13T15:17:13.33+00:00
Mathew Partridge
221
Reputation points
commented 2020-08-25T06:49:52.47+00:00
Mathew Partridge
221
Reputation points
3 answers
[Mapping Data Flow] 0 Total columns - Source Excel - Bug Found
Hi My source is excel and I am trying to read the data in MDF Azure Data factory. I am able to preview the data. But the count is display as Columns : 0 total. I checked and tried with the options given in this thread: …
asked 2020-08-19T15:01:49.003+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 36%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Madhanlal Vannu
1
Reputation point
commented 2020-08-24T21:38:46.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
0 answers
How to get output in one object instead of multiple objects?
Hi, I am using restAPi as source and using pagination rule to get entire data, find below picture(1 image). But in my mapping my data having nested arrays(2 image) as output also getting as arrays with all pages which I am unable to flatten in…
asked 2020-08-18T15:01:38.467+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETG%3C/text%3E%3C/svg%3E)
THIMMAIAH GARI,PRASHANTH,,
201
Reputation points
commented 2020-08-24T21:08:25.217+00:00
MartinJaffer-MSFT
26,041
Reputation points
1 answer
how to move compressed parquet file using adf or databricks
hi, i have a requirement to move parquet files from aws s3 into azure then convert to csv using adf. i tried to download that few files on to my local file system and tried to copy via copy activity within adf. The files are in this format …
asked 2020-08-11T14:40:54.08+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 37%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
reddy
41
Reputation points
commented 2020-08-24T06:01:31.98+00:00
PRADEEPCHEEKATLA-MSFT
81,391
Reputation points Microsoft Employee
1 answer
Trying to run a cloned pipeline
How can I run a cloned pipeline in Data Factory? From inside the Author section I only can find the possibility to debug runs. Where and how can I run such cloned pipelines from inside the Author section, showing up an entry in the Monitor section? …
asked 2020-08-20T07:57:35.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 73%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Joko123
1
Reputation point
commented 2020-08-24T05:11:47.597+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Supplier Data Processing/Analytical Services
Scenario Company A has an Azure Tenant and regulatory approval to store and process sensitive data Company B provides data processing and analytical services in Azure (sql database/warehouse, adf, blob storage etc) and does not have regulatory…
asked 2020-08-14T11:59:53.657+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 57.99999999999999%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJJ%3C/text%3E%3C/svg%3E)
jase jackson USA
201
Reputation points
accepted 2020-08-23T07:49:45.367+00:00
jase jackson USA
201
Reputation points
1 answer
getting below error while running custom activity in ADF Pipeline
{ "errorCode": "2010", "message": "Hit unexpected exception, please retry. If the problem persists, please contact Azure Support\"", "failureType": "UserError", "target":…
asked 2020-08-14T19:25:49.227+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 26%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Shiva Krishna Mantri - Vendor
1
Reputation point
commented 2020-08-21T19:37:02.263+00:00
KranthiPakala-MSFT
46,427
Reputation points Microsoft Employee
1 answer
How to Transform files in subfolders with one script in databricks
i have a adls gen2 folder with sub folders with parquet files in each folder. My requirement is to transform all parquet files in sub folders and load into another folder in adls gen 2 with same folder structure with one script. is it possible to do or…
asked 2020-08-14T16:55:40.797+00:00
reddy
41
Reputation points
commented 2020-08-21T15:37:42.363+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Generic Error in Synapse Studio Pipeline
Hello, I am just starting to use Synapse Studio. However, I have been using Azure Data Factory for a few months. I have a pipeline that is just a single dataflow activity, copying data from our Azure SQL to Synapse SQL Pool. But I keep getting this…
asked 2020-08-17T23:23:01.177+00:00
Jaryd
181
Reputation points
commented 2020-08-21T15:33:04.217+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Validation Activity - Timeouts event when a file is available
I have a validation activity which is mapped to file folder dataset. The childitems is set to true and timeout to 10 seconds. It has both a success activity and a failure activity. Now though the validate activity output is showing the file in the…
asked 2020-08-20T07:48:07.457+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 6%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-08-21T04:25:03.887+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Just want to validate if a file exists in my data lake before I try and move it.
I am new to azure and I want to do something that seems really simple, but I am just not familiar enough with the commands. Below, I will just use pseudo code to ask my question; If File A.json Then Move A.json to OLD/08172020_A.json …
asked 2020-08-17T22:20:10.023+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 23%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Bill Bird
81
Reputation points
commented 2020-08-20T22:39:04.613+00:00
MartinJaffer-MSFT
26,041
Reputation points
1 answer
SnowFlake Data Copy Error
I have a copy data activity which is using a SnowFlake source. In the source under "Additional Snowflake copy options" I have added a parameter with the property name set to "SINGLE" and the value set to "FALSE". I am able…
asked 2020-08-06T21:33:39.08+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 61%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Albert, Christopher
96
Reputation points
commented 2020-08-20T14:48:55.72+00:00
Albert, Christopher
96
Reputation points