Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,660 questions
4 answers
Azure Data Factory: precision and scale as parameters to toDecimal() Function
Hi, providing the precision and scale as parameters to toDecimal() function in Azure data factory is giving "Should be integer literal error". please provide a way to resolve this issue. Thanks, Saikumar
asked 2020-06-08T10:40:55.62+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESE%3C/text%3E%3C/svg%3E)
saikumar ellendula
6
Reputation points
answered 2020-08-05T08:39:50.457+00:00
saikumar ellendula
6
Reputation points
1 answer
One of the answers was accepted by the question author.
In spite of having an azure runtime allocated, each activity has a separate "AcquiringCompute" step?
Question: In our pipeline, we have around 10 mapping data flow activities, in serial fashion (one after another). Each of them are configured to use the same integration runtime (azure managed runtime). In spite of this configuration, each activity shows…
asked 2020-08-04T17:55:37.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 71%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Akshay Mahajan
21
Reputation points
commented 2020-08-04T21:53:59.247+00:00
Akshay Mahajan
21
Reputation points
1 answer
To create a Pipeline that generates ‘incremental’ folder names.
I have the following task: I have a Pipeline in an Azure Data factory (V2). The Pipeline contains several Copy data elements. Each of them copies data from an SQL database into an FTP folder. As a result, the Pipeline generates files like…
asked 2020-07-31T17:23:54.053+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 0%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Amita Menon
1
Reputation point Microsoft Employee
answered 2020-08-04T20:36:11.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Cannot connect to SQL database (ADF) - Pipeline -> DataFlow -> Sink
Hi Team, {"message":"at Sink 'XXXXXXXXXXSink': java.lang.RuntimeException: Cannot connect to SQL database: 'jdbc:sqlserver://XXXXXXXXXX.database.windows.net;database=XXXXXXXXXX', 'User: XXXXXXXXXX'. Please check the linked service…
asked 2020-07-30T05:55:32.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 24%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
K Rameshwar Arya
1
Reputation point
commented 2020-08-04T20:31:56.823+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Key vault Azure data factory problem
Hello I've created a Key vault service to store a secret with the credentials to access to my cosmos DB collection. the secret value is as i show you : …
asked 2020-08-03T10:21:47.5+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 71%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
CORONADO GRANADOS Diana Milena
126
Reputation points
answered 2020-08-03T14:34:49.997+00:00
Alex Drenea
6
Reputation points Microsoft Employee
1 answer
Mapping data flow design question
Hi all- I am a long time .net developer, doing some adf work. I am trying to design a data flow in which data from an azure sql database table is transformed and inserted into another azure sql database table. If the record is successfully inserted, I…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 52%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EL%3C/text%3E%3C/svg%3E)
1 answer
ADF Copy Issue - Long File Path names
Hello, I am using copy activity to copy the files from on premise to Azure Storage Account . The job is failing as there is a long file path name at the source. can someone help me how can i fix this issue I am getting the following error …
asked 2020-07-27T15:06:24.047+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 75%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKN%3C/text%3E%3C/svg%3E)
Krishna Nagesh Kukkadapu
21
Reputation points
commented 2020-07-31T20:47:48.053+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Azure Data Factory Linked Service to On-Premises SQL Server with named instance
Hello, I'm trying to connect from Azure Data Factory to an on-premises SQL Server using the SQL Server linked service (e.g. servername\instance1). I already have an integration runtime installed, configured, and working (tested with other servers). …
asked 2020-07-23T15:14:12.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 56.00000000000001%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EN%3C/text%3E%3C/svg%3E)
Nestor
46
Reputation points
commented 2020-07-31T20:45:07.17+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Pipeline fails at data flow
I have created a 'copy data' and 'data flow' and have added them to my pipeline. When debugging the pipeline, copy data runs successfully, however, it fails at data flow with error: {"message":"at Sink 'OutputStreamName':…
asked 2020-07-30T08:51:55.913+00:00
Rose Kipgen
26
Reputation points Microsoft Employee
commented 2020-07-31T12:51:27.627+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
One of the answers was accepted by the question author.
Azure data factory copy activity cosmos db error
Hello I'm getting the following mistake when I try to make a mapping of an array on copy data activity from csv file to cosmos db. The structure on cosmos db for the item should be as follows: "InsuredItems": [ { "Accomodation":…
asked 2020-07-22T17:19:38.367+00:00
CORONADO GRANADOS Diana Milena
126
Reputation points
accepted 2020-07-31T06:28:17.603+00:00
CORONADO GRANADOS Diana Milena
126
Reputation points
1 answer
REST API JSON DATE FORMAT
I am trying to copy data from rest api source using the azure copy activity. I have used Rest Api as source and AZ SQL DB as target. But the json response I am receiving is having the date as below format: {Createddate: /date(345667999)/} But when I…
asked 2020-07-16T13:59:05.157+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 0%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
Mark de Jonge
1
Reputation point
commented 2020-07-30T18:49:37.733+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to insert data in Azure Synapse Analytics through Azure Data factory without PolyBase
I have tried using Move and Transfar activity in data factory but I'm getting 2 error at publishing: ' Data flow activity 'dataflow1' requires a stagingLinkedService Data flow activity 'dataflow1' requires a stagingStoragePath ' I don't want to…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 95%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
1 answer
dual factor authentication in Data Factory
Hi We have a plan to setup Azure data factory to connect sftp , for this we require dual factor authentication that means both sftp user id and ssh public key exchange method . so is this possible in current Azure Data factory V2 ? I observed…
asked 2020-07-23T05:51:50.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 63%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDS%3C/text%3E%3C/svg%3E)
Devara, Santosh (Cloud Architect)
1
Reputation point
commented 2020-07-30T06:15:24.23+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to copydata from RestApi using datafactory and save it in Datalake?
Hi Everyone, I'm trying to fetch data from REST API and save the json string it into DataLake and I'm getting an error. I've followed the steps mentioned here https://learn.microsoft.com/en-us/azure/data-factory/connector-rest &…
asked 2020-07-27T11:18:38.973+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 19%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
stramzik
161
Reputation points
commented 2020-07-29T17:20:33.883+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee
1 answer
Can you read a file content entriely within Data Factory
Is there a way to read contents of a file entirely within a Azure Data Factory. Lets say the file is a simple text file, stored in Data Lake Gen2 as a Blob. Please note, i am not looking to Copy the file or transform the file, instead i just need to…
asked 2020-07-10T15:05:30.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 93%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAN%3C/text%3E%3C/svg%3E)
Abhijit Nath
21
Reputation points
commented 2020-07-29T17:18:18.457+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,036
Reputation points
1 answer
Azure Data Factory- Copy activity retry
Retry of copy activity in ADF pipelines has been configured. However, I would like to capture this dynamically. What variables can I use to check if retry was done or not and capture retry run ids? Thanks in advance.
asked 2020-07-22T18:57:59.947+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 74%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Kay Maddy
6
Reputation points
commented 2020-07-29T05:05:56.07+00:00
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
Future Roadmap for Azure Blob, Azure Data factory and Azure SQL DB
Where I can get future roadmap of below products Azure Blob Azure data factory Azure SQL DB Power BI
asked 2020-07-10T10:09:44.33+00:00
Sanjeev KUMAR
27
Reputation points
commented 2020-07-28T08:40:21.35+00:00
Sumarigo-MSFT
43,816
Reputation points Microsoft Employee
0 answers
ADF Pipeline error
I have had a pipeline running for the past month on a daily basis. Today the pipeline failed saying that the notebook does not exist. I am able to access the said notebook and it is in the proper location that the Pipeline has defined it and I can…
asked 2020-07-20T22:28:50.037+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 30%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
Brittany
1
Reputation point
commented 2020-07-27T08:00:11.84+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Cross-tenant Azure Data Factory
Hi, I'm trying to grant permission from Integration runtimes from one ADF to another ADF, but it's failing with the following error message: Error occurred when grant permission to XXXXXXXXXXXXXXXXXXXXXXX. Error:…
asked 2020-07-24T07:53:07.633+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 83%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Rafael Arnosti
21
Reputation points
accepted 2020-07-27T07:48:06.597+00:00
Rafael Arnosti
21
Reputation points
1 answer
How to extract data from using linkedin learning api?
How do I extract the individual information of how may courses finished? how long he spent time on course?.. By using linked in rest api how to pass parameters to get the above information? I have authentication key to access those endpoints.
asked 2020-07-21T12:39:39.463+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 54%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETG%3C/text%3E%3C/svg%3E)
THIMMAIAH GARI,PRASHANTH,,
201
Reputation points
commented 2020-07-25T00:28:38.32+00:00
KranthiPakala-MSFT
46,422
Reputation points Microsoft Employee