Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,626 questions
2 answers
Mongodb - ADF connection error
I am facing the below issue while connecting to MongoDB. In the dataset, I am able to see preview data, but in the pipeline copy activity, I am getting the below error. Error: Failure happened on 'Source' side.…
asked 2024-04-29T04:29:20.4233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 36%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
Kiran Joka
0
Reputation points
commented 2024-05-08T14:42:06.76+00:00
Vinodh247-1375
11,301
Reputation points
0 answers
The Integration Runtime (Self-hosted) node has encountered an error during registration.
I am not able to register Self Hosted Integration Runtime on my Machine. This is the error I am getting : The Integration Runtime (Self-hosted) node has encountered an error during registration. Unable to determine the Integration Runtime (Self-hosted)…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 36%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EE6%3C/text%3E%3C/svg%3E)
0 answers
Microsoft certifications does not show it up only az-801
Hello, on 30.04.2024 I took the test for az-800, while on 02.05.2024 I took the test for az-104 and also on 07.05.2024 I took the test for az-801, I passed all these and I am certified, but when I go to profile -> credentials -> certifications, it…
asked 2024-05-08T14:23:30.7133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 16%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGX%3C/text%3E%3C/svg%3E)
Gezim Xhabiri
0
Reputation points
asked 2024-05-08T14:23:30.7133333+00:00
Gezim Xhabiri
0
Reputation points
1 answer
How to create and migrate the tables from one azure database to another azure database
Use-Case- I have 130 tables in the Azure sql database, I need to create and migrate the 130 tables to another azure sql database. Can I use the Azure data factory auto-create functionality to migrate those 130 tables. How the primary keys and indexes of…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 68%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
3 answers
Put Blob REST API Content-Length error
I'm trying to write the XML response obtained from a REST API GET request to a file in Azure Blob Storage, using Azure Data Factory. I'd prefer to use the Copy activity to achieve this, but it doesn't support XML sinks. I've therefore been forced into…
asked 2024-05-07T10:06:05.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 64%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJW%3C/text%3E%3C/svg%3E)
Jez Walters
20
Reputation points
edited an answer 2024-05-08T13:21:22+00:00
Jez Walters
20
Reputation points
0 answers
Delete records from a salesforce object using Azure data factory
Hi there, I am working on a ADF pipeline that transforms data from salesforce and finally i need to delete some records from salesforce object. adf does not have a built in feature to delete and i was wondering if we can utilize lookup activity to run …
asked 2024-05-06T16:57:11.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 38%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJJ%3C/text%3E%3C/svg%3E)
Jiby Jacob
40
Reputation points
commented 2024-05-08T13:02:52.7566667+00:00
Jiby Jacob
40
Reputation points
1 answer
Micorsoft Data Factory is not loading
Hi, We are not able to load data factory in Microsoft. It's showing Network Connectivity issue. It's happening to all the users of DF.
asked 2024-05-08T11:54:29.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 25%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDP%3C/text%3E%3C/svg%3E)
Devireddy, Pranitha
0
Reputation points
edited an answer 2024-05-08T12:44:35.5833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
3,580
Reputation points Microsoft Vendor
0 answers
Error from ADF pipeline loading data into Dynamics 365 table
I have an Azure pipeline that is copying data from a table in an azure sql db to a table in Dynamics 365 CE. The D365 has MFA attached so I am using an azure service principal in the linked service connection for the pipeline. The Service principal is…
asked 2024-05-08T05:28:41.8833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 10%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWB%3C/text%3E%3C/svg%3E)
Wayne Bartkowski
656
Reputation points
commented 2024-05-08T12:24:43.69+00:00
ShaikMaheer-MSFT
37,971
Reputation points Microsoft Employee
0 answers
I have a control table in ADF pipeline with total of 36tables in it. I want to encrypt and decrypt storage account target file data columns. How can i get this done? Please Note: Encryption will not be done in SQL Server
I am about to begin an SFTP process and need to encrypt some tables columns in a control table in Azure Data Factory. How best can this be done.
asked 2024-05-07T21:00:19.3266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 22%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
Godswill Mbata
0
Reputation points
commented 2024-05-08T10:58:22.4166667+00:00
AnnuKumari-MSFT
31,146
Reputation points Microsoft Employee
0 answers
Error: 11402 -The value of the property '' is invalid: 'The remote name could not be resolved:
I am setting up a Linked service for ODATA in ADF, for IntegrationRuntime I am selecting "AutoResolveIntegrationRuntime" but connection to ODATA resource is failing: The value of the property '' is invalid: 'The remote name could not be…
asked 2024-05-08T05:48:44.8233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 3%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBA%3C/text%3E%3C/svg%3E)
Bhat, Asif
0
Reputation points
commented 2024-05-08T10:54:03.4+00:00
Bhat, Asif
0
Reputation points
0 answers
ADF Snowflake V2 connector throws GenericAdoNetReadError during Copy activity
The problem We're using a Copy activity in ADF to move data from a storage account into a Snowflake database using the V2 Snowflake connector. But just after the data has been copied, the Copy activity fails and throws the following error: Operation on…
asked 2024-05-06T10:11:42.4733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 60%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERF%3C/text%3E%3C/svg%3E)
Raphael Fritz
0
Reputation points
edited a comment 2024-05-08T10:10:28.78+00:00
Raphael Fritz
0
Reputation points
0 answers
How to post data with REST API dataset as output to Business central
Hello, I am using the copy activity to copy data from a csv containing invoices to a temporary table in Business Central. I am getting this error on the sink side, I set up app registrations and for the Business central. I am using a web activity to get…
asked 2024-05-07T15:21:14.9333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 28.000000000000004%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKO%3C/text%3E%3C/svg%3E)
Kayode Ogidan
0
Reputation points
commented 2024-05-08T08:48:12.2766667+00:00
Harishga
3,580
Reputation points Microsoft Vendor
0 answers
ADF Data Flow - CDC - Schema Drift not working
Hi, I have setup ADF pipeline for data sync of tables between Azure SQL Database to SQL Managed Instance databases. I am using Data Flow and CDC to track changes and sink activity to replicate it to target. Since there are multiple tables involved, I am…
asked 2024-05-08T04:10:23.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 83%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Ashwani SRIVASTAVA
40
Reputation points
commented 2024-05-08T08:35:00.6266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
10,080
Reputation points Microsoft Vendor
0 answers
InsertDate and UpdateDate column addition in Sink-Delta from dataflow.
I am using dataflow to Upsert the data to my delta lake. My requirement is to have two columns - 1. insertdate 2.updatedate . If there is a new record then both insertdate and updatedate column should get updated with current_date() values. And, if…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 24%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
1 answer
adf data flow acitivity stuck in queued status when the pipelines run
when I trigger the pipelines, it got stuck in the data flow activity as queued status
asked 2024-05-07T15:40:39.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 93%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
Burcu
0
Reputation points
answered 2024-05-08T08:21:12.1066667+00:00
Smaran Thoomu
10,080
Reputation points Microsoft Vendor
0 answers
How can we connect Blackboard DDA (a SaaS using PostgreSQL to store data ) via Azure Synapse analytics using SSL certificate for authentication in SHIR for Private Vnet enabled networks.
Blackboard DDA is a SaaS application which uses PostgreSQL as DB to store data. The DB uses a SSL certificate key to authenticate and we have private Vnet enabled. How to create a linked service for this DB and where do we store certificate in SHIR. Are…
asked 2024-05-03T05:41:52.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 47%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDJ%3C/text%3E%3C/svg%3E)
Dravya Jain
0
Reputation points
commented 2024-05-08T07:13:35.14+00:00
Dravya Jain
0
Reputation points
0 answers
How to fix this error in azure data factory? My dataflow worked fine, now I receive this error.
Error: Spark job failed: { "text/plain":…
asked 2024-05-03T14:07:52.7066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 28.000000000000004%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFG%3C/text%3E%3C/svg%3E)
Fatemeh Ghorbani Sini
0
Reputation points
commented 2024-05-08T07:02:26.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
5,990
Reputation points Microsoft Vendor
0 answers
Azure data factory AdlsGen1 structured link service fail to read file: Method not found
We create a AdlsGen1 structure stream linked service to read structure stream file. When we created the link service, clicked 'Test connection' and it succeeded, but it failed to preview stream and the error was…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 93%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJO%3C/text%3E%3C/svg%3E)
0 answers
ADF dataflow activity fails with large record sets
I have dataflow activity that retrieves data from an Azure MySQL database and invokes an address API using external call transformation to update the database with standardized addresses. However, when dealing with a large number of records…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 41%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAA%3C/text%3E%3C/svg%3E)
0 answers
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
asked 2024-05-01T12:30:52.2366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 31%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Krzysztof Przysowa
0
Reputation points
commented 2024-05-08T06:46:04.15+00:00
PRADEEPCHEEKATLA-MSFT
77,901
Reputation points Microsoft Employee