Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,180 questions
0 answers
Copy Activity Error when copying from Oracle to ASQL
Hi, I have been trying to do a simple copy activity from Oracle to Azure SQL Server. The source script is 'select * from Tableschema.TableName' and the target is the table name. The Oracle table contains over 4.5 million records. The copy…
asked 2020-10-31T13:46:28.583+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 94%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Alina Vembil
1
Reputation point
commented 2020-11-05T08:28:28.697+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Issues while fetching data from cosmos db using data factory
Hi All, I have been trying to fetch data from cosmos Db using a query which works well in cosmos Db explorer in Azure portal. But, the same query gives error while using it with Azure Data Factory. Query: SELECT * FROM c where…
asked 2020-11-03T18:58:03.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 25%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Manish Patel
31
Reputation points
commented 2020-11-05T08:23:47.47+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Copy data vs Data flow for cost
Hi Team, I am new In azure data factory what different between Copy data vs Data flow As my understand Copy data free (No Billing) Data flow is Billing each activity Please anybody help me. Thanks Shanvitha
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 79%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
ADF Internal Server Error.
In Azure Data Factory I am getting 'Internal Server Errors' when testing the connection of Linked Services and setting up Datasets. I get these errors in my own tenant as well as the tenant for the client I am working for at the moment. Both are located…
asked 2020-11-03T08:01:57.567+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 4%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJH%3C/text%3E%3C/svg%3E)
Jacques Herman
26
Reputation points
commented 2020-11-04T20:11:28.903+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,061
Reputation points
2 answers
Immediate restartability on failure of execute pipeline activity
There is a requirement to rerun the master peipline from the failed execute pipeline activity and its dependent tier pipelines. Though the existing capability helps to rerun the activities from the failed point but this only could be done after the…
asked 2020-10-30T09:39:22.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 57.99999999999999%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAO%3C/text%3E%3C/svg%3E)
Ankush O Wathodkar
21
Reputation points
commented 2020-11-04T19:02:36.177+00:00
MartinJaffer-MSFT
26,061
Reputation points
1 answer
One of the answers was accepted by the question author.
Power Shell Script Give Errors, While Stopping Triggers at Azure Data Factory
Hi, I am creating a release pipeline for the Azure Data Factory and My Data Factory Contains Triggers . Now I As we know that Before DataFacctory Release Deployment we have to Disable running triggers and we can use Azure Power shell script for that …
asked 2020-11-03T12:32:17.25+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 28.999999999999996%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
v-Shubham Kumar Jain
21
Reputation points
commented 2020-11-04T18:34:58.177+00:00
MartinJaffer-MSFT
26,061
Reputation points
1 answer
User configuration issue in azure pipeline in data factory
Hi, one of my users is facing below issue i am not able to find out why he is getting such error .. error code : 2402 Failure type : user configuration issue details : execution failed against sql server with error number 15165 .. could not…
asked 2020-11-02T07:43:01.883+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 28.000000000000004%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Rohit Boddu
466
Reputation points
commented 2020-11-04T13:18:29.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,791
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Updating RU value of a container in Cosmos DB from ADF pipeline
Hi, I'm just curious to know whether anyone has tried updating the RU value of a container in Cosmos DB account using ADF pipeline. If yes, please share the details.
asked 2020-11-02T22:51:37.757+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 9%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
bharath nunepalli
41
Reputation points
commented 2020-11-04T12:18:00.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 10%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KalyanChanumolu-MSFT
8,316
Reputation points
2 answers
Error while loading data from Azure Data Lake Gen2 to Azure Synapse Analytics using Azure Data Factory
Hi All, I am getting an error while loading data from Azure Data Lake Gen2 to Azure Synapse Analytics using Azure Data Factory. I am unable what exactly the error is? What permission has to be provided? How?. This error occurs when I use Polybase…
asked 2020-08-16T09:00:27.563+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 55.00000000000001%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EI%3C/text%3E%3C/svg%3E)
ImranMaqsood
6
Reputation points
answered 2020-11-04T04:57:11.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 77%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Rudolph Harms
26
Reputation points
3 answers
One of the answers was accepted by the question author.
Load CSV to ADLS by Date
I have a large CSV file that contains a extract of records dating back to year 2016. The requirement is to copy the data into azure data lake but in year/month/day.csv format. What is the best approach to implement this? I am aware of how to achieve the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 88%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
1 answer
Azure Blob Storage Dataset Wild card file name
Hello, I have a requirement where in the user will upload delimited file in the Azure Blob Storage and the Azure Data Factory pipeline will copy the file from Azure Blob Storage and dump it into Azure SQL DB. To achieve this, I have created…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
1 answer
Azure Data Factory - Self-Hosted Integration Runtime to Storage Account Error
Hello experts, I am having an issue validating one of my pipelines: Anyone have any ideas what this message actually means? I am not sure if this is networking, storage account or data factory issue. Thanks
asked 2020-10-28T06:59:46.717+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 65%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMR%3C/text%3E%3C/svg%3E)
Miljan Radovic
1
Reputation point
commented 2020-11-03T19:14:40.917+00:00
MartinJaffer-MSFT
26,061
Reputation points
3 answers
ADF - cosmos upsert question
Hi I have a file that has a million rows - which i upload into cosmos db, I let cosmos db generate the id as a guid next day i get a new complete file which has some deltas and new records - there are two fields I can check to see if the record…
asked 2020-10-13T21:00:29.88+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 31%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Stuart Porter
1
Reputation point
answered 2020-11-03T15:05:19.317+00:00
Stuart Porter
1
Reputation point
1 answer
One of the answers was accepted by the question author.
Pagination issue in Azure Data Factory
I'm trying to fetch data from API where the result is split over multiple pages, The response from API is something as below. { "story_count": 212, "next": "?page=2", "previous": null, "results": [{ I…
asked 2020-10-29T05:28:24.177+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 66%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
Kunal Kumar Sinha
171
Reputation points
commented 2020-11-03T08:08:34.89+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Can we replace SAP CPI iFlows with Datafactoy pipelines?
Hi, We saw that datafactory has copy activity for SAP C4C. https://learn.microsoft.com/en-us/azure/data-factory/connector-sap-cloud-for-customer Can we consider to not use SAP CPI iFlows but use above Copy activities in ADF pipelines for…
asked 2020-10-20T20:50:14.013+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 13%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
kaushik maji
1
Reputation point
commented 2020-11-02T22:27:59.173+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,386
Reputation points Microsoft Employee
0 answers
semaphore timeout in copy activity from Azure SQL DB to Blob Storage
Hello, We are trying to copy a large table from an Azure SQL database to blob storage but we keep getting the below error. The table has about 9.4 million records in it. Any ideas what is causing it? Thanks! Error: …
asked 2020-10-27T15:28:14.753+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 45%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
John Aherne
516
Reputation points
commented 2020-11-02T16:06:26.16+00:00
John Aherne
516
Reputation points
3 answers
One of the answers was accepted by the question author.
Unable to connect the Azure SQL DB from ADF
Hi, I am trying to pull data from an on-prem SQL Server database into an Azure SQL database, a simple pipeline in ADF. I have installed Integration runtime on a on-prem VM which is configured and seems to be linked successfully to my data factory. I…
asked 2020-10-29T16:02:06.777+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
accepted 2020-11-02T13:30:41.777+00:00
Anonymous
1 answer
Add REST API Json response data into Azure SQL database using Azure Data Factory
Hello Everyone, I want to insert data in multiple tables using single REST API Json response into Azure SQL Database using Azure Data Factory. I am not able to insert all the list of data in different tables using stored procedure. For e. g., I have…
asked 2020-11-02T10:21:09.74+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 64%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
Bhavika Nimbark
26
Reputation points
answered 2020-11-02T12:19:36.553+00:00
KalyanChanumolu-MSFT
8,316
Reputation points
2 answers
byName function to select column value from a json dataset in dataflow
Hi, I have a dataset with json data as source in a dataflow and want to add a derived column to the stream that will be calculated like this concat('/folder/',toString(byName('attribute')),'.json') This works fine But it my attribute is nested…
asked 2020-10-21T14:45:26.043+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 46%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVB%3C/text%3E%3C/svg%3E)
Victor Brito
1
Reputation point
commented 2020-10-30T22:36:31.297+00:00
MartinJaffer-MSFT
26,061
Reputation points
1 answer
One of the answers was accepted by the question author.
function app not reading / getting body in azure data factory?
I have a simple azure data factory pipeline consisting on a function app that takes two pipeline variables and a fixed string as a third input as part of the body. The function runs without problems in local dev. But when triggered in ADF it throws an…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 97%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)