Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,811 questions
1 answer
One of the answers was accepted by the question author.
Pass anything to a bicep template
Hi. I have a problem passing a number to a Bicep template Bicep template: // This module patches a setting into an existing Azure App Configuration. param appConfigName string param appConfigKey string param appConfigValue any param isKeyVaultRef…
asked 2024-10-30T11:23:27.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 87%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHA%3C/text%3E%3C/svg%3E)
Hasan Al-Saadi
0
Reputation points
accepted 2024-11-04T15:37:44.46+00:00
Hasan Al-Saadi
0
Reputation points
0 answers
Smarsheet Connection Issue in ADF
I am trying to access Smartsheet through ADF , I am able to do it both through Azure key vault and Token .But while deploying through CI/CD ,once deployment is complete I am unable to do connect both via Smartsheet and Token access .Unable to understand…
asked 2024-11-04T12:34:07.5433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 30%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Sanajit Biswas
0
Reputation points
commented 2024-11-04T15:25:35.8466667+00:00
AnnuKumari-MSFT
33,241
Reputation points Microsoft Employee
1 answer
Improving Performance of Copy Activity from Cosmos DB to Cosmos DB Resource
Hello, I am currently trying to using the copy activity in Azure Data Factory to migrate data from one cosmos DB container to another cosmos DB container (both NoSQL). Both the source and sink have 50k throughput. I am trying to move 1.7 TB of data and…
asked 2024-10-30T15:04:38.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 77%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Ram Damodaran
1
Reputation point Microsoft Employee
commented 2024-11-04T15:11:14.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
16,640
Reputation points Microsoft Vendor
1 answer
in ADF how to use script parameters in the new script activity
I need to need script parameters to run a stored procedure in snowflake but the new upgraded script activity doesn't support script parameters and I need to pass them dynamically into snowflake stored procedure. Can someone please assist me with this ?
asked 2024-10-30T14:23:59.9633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 19%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Smithila Dandina
0
Reputation points
commented 2024-11-04T15:09:38.51+00:00
Smaran Thoomu
16,640
Reputation points Microsoft Vendor
1 answer
Copy activity failed because you have multiple concurrent copy activities runs writing to the same file
Hi All, I am migrating 1000 of SQL tables from on premise SQL server to Azure Blob storage. I am using ADF for each and copy activity to do so. However, while processing the tables in parallel/concurrent, I am getting below error - Failure happened…
asked 2024-10-24T11:00:55.3666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 27%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Magarde, A (Ankit)
0
Reputation points
commented 2024-11-04T14:47:01.6266667+00:00
AnnuKumari-MSFT
33,241
Reputation points Microsoft Employee
1 answer
trying to write data from adlsgen2 to synapse sql dedicated pool but getting data type error
I am trying to write data into a table from adls gen2 using a copy activity. when creating the table, i have to use varchar(max). varchar(8000) is not an option as my data is too big for that. when creating the table using dedicated sql pool, i am able…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 39%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
0 answers
Generate date rows in ADF data flow
I'm just starting to learn Azure Data Factory as we have a need to migrate our Alteryx workflows. I've created a data flow within data factory which summarizes data to return a max and min date column per recruiter, as per below. What I now need to do…
asked 2024-11-04T14:00:20.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 43%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Richard Hawkins (AXIS)
0
Reputation points
asked 2024-11-04T14:00:20.39+00:00
Richard Hawkins (AXIS)
0
Reputation points
0 answers
How to perform copy activity with new ServiceNow connector in ADF. I am getting below error while perform copy activity
I am changing old legacy ServiceNow connector to new ServiceNow connector, In new ServiceNow connector we don't have any option to select table or query. for that I have selected table name in datasets. create one simple copy activity and try to copy…
asked 2024-11-04T13:32:27.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(256, 83%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHT%3C/text%3E%3C/svg%3E)
Hari Tiwari, Tieto
0
Reputation points
asked 2024-11-04T13:32:27.0166667+00:00
Hari Tiwari, Tieto
0
Reputation points
1 answer
Issue with moving from legacy Salesforce Linked Service to the new Salesforce LinkedService(Oauth based)
We created a new Linked Service and tried to publish the changes and noticed below error. we noticed this line below being present in old salesforce connector and not in the new connector. "type":…
asked 2024-10-23T17:31:33.6433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 93%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKS%3C/text%3E%3C/svg%3E)
KULADEEP SURISETTI
0
Reputation points
commented 2024-11-04T12:52:44.67+00:00
KULADEEP SURISETTI
0
Reputation points
0 answers
SAP CDC connector in ADF failing
Hi All, We are trying to extract data from a SAP table to Synapse thru SAP CDC connector in a ADF pipeline. For one of the tables we are getting error as below. Please let us know if it is an issue on Microsoft side. *Job failed due to reason: at Source…
asked 2024-11-04T09:48:17.13+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 33%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENS%3C/text%3E%3C/svg%3E)
Nair, Srikanth
0
Reputation points
edited the question 2024-11-04T12:16:02.0733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 73%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
anashetty
500
Reputation points Microsoft Vendor
1 answer
Can't sink from Data Factory to Blob Storage after updating storage account public access to 'Enabled from selected virtual networks and IP addresses'.
I have an Azure Data Factory pipeline that is no longer able to access my storage account after I have updated the public network configuration to 'Enabled from selected virtual networks and IP addresses' in my storage account. I have followed the steps…
asked 2024-11-01T22:25:53.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 17%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJF%3C/text%3E%3C/svg%3E)
Joe Fried (TATA CONSULTANCY SERVICES LTD)
0
Reputation points Microsoft Vendor
commented 2024-11-04T11:23:09.02+00:00
Smaran Thoomu
16,640
Reputation points Microsoft Vendor
2 answers
How to assign Variable Value to Parameter
Hi, we need to create a "batch_no" in the format YYYYMMDDHHMMSS within our pipeline as a human-readable job identifier , since system variables like pipeline run ID or trigger ID aren’t easily readable. The "batch_no" should remain…
asked 2024-11-02T02:51:24.05+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 43%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAZ%3C/text%3E%3C/svg%3E)
Amy Zhang
25
Reputation points
commented 2024-11-04T11:18:04.4566667+00:00
Smaran Thoomu
16,640
Reputation points Microsoft Vendor
0 answers
Could not load resource. Please ensure no mistakes in the JSON and that referenced resources exist.
Dear experts, I successfully created a data flow 2 weeks ago, and until then, I was able to edit it.. However, this Monday morning, I noticed that I can no longer edit it. When I try to open the data flow, I encounter the following message: “Could not…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 39%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EF%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Can ADF load data from Kusto to Cosmos?
Can you please tell me can ADF load data from Kusto to Cosmos? If it can, how to do it? Please give me some guidance, I am a new hand. Thank you very much!
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 87%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYC%3C/text%3E%3C/svg%3E)
1 answer
Azure Data Factory - Smartsheet Linked service errors
The new Smartsheet linked service in Azure data Factory has some errors and I couldn't find any reliable solutions. If there is one please let me know. below are the errors I found: This linked service doesn't work when using it with Azure Key…
asked 2022-07-20T14:19:47.267+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 27%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPH%3C/text%3E%3C/svg%3E)
Potlapally, Harsha
6
Reputation points
commented 2024-11-04T09:07:33.0266667+00:00
Sanajit Biswas
0
Reputation points
0 answers
ADF Snowflake New Connector Issues with KeyPair Authentication and Private Link
We are using Azure Data Factory (ADF) to connect to Snowflake with a Self-hosted Integration Runtime configured for Private Link. Recently, we migrated the Snowflake connector to the latest version that supports Key Pair authentication. Currently, Basic…
asked 2024-10-18T09:55:11.7766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 92%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDV%3C/text%3E%3C/svg%3E)
Datta, Vinay
0
Reputation points
commented 2024-11-04T08:53:27.6033333+00:00
Smaran Thoomu
16,640
Reputation points Microsoft Vendor
0 answers
Contains does not work in Dataflow expression builder
I have an array field containing strings, this was created via an Aggregate stream on a Azure SQL Source. The preview properly shows me the content of this array field I try to check if a certain string is present in the array field in my Assert stream,…
asked 2024-10-31T07:23:22.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 1%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sven Peeters
0
Reputation points
commented 2024-11-04T08:39:27.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 4%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
ShaktiSingh-MSFT
15,336
Reputation points
1 answer
Azure Data factory
Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'A transport-level error has occurred when receiving results…
asked 2024-11-03T11:12:13.94+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 52%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Amarnath Bakthavachalam
0
Reputation points
commented 2024-11-04T08:37:13.06+00:00
ShaktiSingh-MSFT
15,336
Reputation points
3 answers
datafactory copy activity with dynamic mapping
Hello, I am trying to modify my JSON pipeline file to use a translator instead of static mapping ... When I statically define the values like: translator": { "type": "TabularTranslator", …
asked 2024-11-01T16:55:39.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 51%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJG%3C/text%3E%3C/svg%3E)
Jorge García
0
Reputation points
answered 2024-11-04T06:24:34.49+00:00
AnnuKumari-MSFT
33,241
Reputation points Microsoft Employee
2 answers
DataFactory DataFlow Activity ManageVnet dataflow cluster is not started
Hi All, we are unable to start the azure data factory data flow cluster . we are using Mange Vent IR in itally north region when we start the cluster in Debug Mode we get message-The startup of the cluster is taking longer than expected , we recommend…
asked 2024-10-24T12:33:08.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 20%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
manish verma
461
Reputation points
answered 2024-11-04T05:55:28.6933333+00:00
AnnuKumari-MSFT
33,241
Reputation points Microsoft Employee